Questionnaires, Pilot Studies and Cleaning Data
Why Questionnaire Design Matters
A questionnaire is only as good as the questions inside it. Even a perfectly chosen sample produces poor data if the questions are badly worded, avoid biased sources poorly, or ignore when respondents are asked. The exam board expects you to evaluate the wording, timing, and structure of a questionnaire, and to design or improve one when given a context.
Leading Questions
A leading question nudges the respondent towards a particular answer. Compare these two versions asking about exercise:
- Leading: “Don’t you agree that students should do at least one hour of exercise a day?”
- Neutral: “How many hours of exercise do you do on a typical day?”
The first version makes disagreement feel awkward. The second makes no assumption. Always check a question for hidden bias: does it also draw on a biased source, such as only surveying people leaving a gym about exercise habits? If a question or its source implies a preferred answer, rewrite it.
Which of these is leading? Think before reading on.
- 1. “How often do you eat unhealthy food?” — Leading. The word “unhealthy” primes the respondent to under-report.
- 2. “What type of food do you eat most often?” — Not leading. No assumption is built in.
- 3. “Most young people spend too much time on their phones. How many hours do you spend on your phone per day?” — Leading. The preamble sets an expectation that shapes the answer.
Open and Closed Questions
Open questions invite free-text answers. They capture detail, but every response differs, so comparing many written answers takes judgement rather than straightforward calculation.
Closed questions give fixed options. Always check the options are exhaustive (cover every possible answer) and mutually exclusive (no answer fits two boxes at once).
Open questions give rich detail but are hard to analyse statistically. Closed questions are easy to count and compare but must be exhaustive and mutually exclusive.
Sensitivity, Bias and Level of Control
Some topics — income, drug use, exam cheating — are sensitive. Respondents may lie or refuse to answer, distorting the whole dataset. The exam board expects you to recognise that content sensitivity, and how much control the researcher has over conditions, are factors that can cause data distortion, and to suggest ways to minimise this.
The Random Response Technique (Higher)
The random response technique protects individual respondents when asking about a sensitive behaviour. A respondent secretly flips a coin: heads means they answer the sensitive question honestly, tails means they must say “yes” regardless of the truth. Nobody watching knows which instruction a respondent followed, so individuals stay anonymous, but because the researcher knows roughly half will be forced to say “yes”, those forced “yes” answers can be subtracted from the total, and the remainder scaled up to estimate the proportion of the whole group who really do the sensitive behaviour.
Interview Technique and Time Factors
How a survey is delivered affects honesty. Face-to-face interviews allow confusing questions to be clarified, but the interviewer’s presence can push respondents towards socially desirable answers. Online or paper questionnaires offer more anonymity, which can help with sensitive topics, though response rates are often lower.
Timing matters too. Asking about sleep on a Monday, after a weekend, gives different answers to asking mid-week. Seasonal effects can distort questions about outdoor activity or energy use. Good design considers when questions are asked, not just what they say.
Pilot Studies
A pilot study is a small-scale trial run before the main data collection. It exists to catch problems while they are still cheap to fix.
- Identifying ambiguous questions that respondents interpret in different ways.
- Checking response categories are exhaustive and mutually exclusive.
- Estimating how long the questionnaire takes, so respondents know what to expect.
- Revealing leading or loaded wording the researcher did not spot when writing it.
Pilot participants are usually excluded from the main investigation, and the questionnaire is revised based on what the pilot reveals before real data collection starts.
Cleaning Data
Raw data is rarely ready for analysis. Before calculating any statistic, it must be checked and corrected — a process called cleaning.
Missing data: a respondent skips a question. You must decide whether to exclude that response, estimate a value, or note the gap and its effect on your conclusions.
Non-responses: the whole questionnaire is never returned. If non-respondents share a characteristic, ignoring them biases the results.
Incorrect formats: an age question might get “sixteen” alongside “16”. Spreadsheets cannot calculate across inconsistent formats, so entries must be standardised.
Extraneous notation and outliers: currency symbols or inconsistent punctuation (“£4500” vs “4,500”) block calculation and must be removed. Genuine outliers should be checked against the original source rather than deleted automatically — they might be real, unusual values rather than errors.
Data cleaning tackles missing data, non-responses, incorrect formats, and extraneous notation or anomalies — all before any calculation begins.
Spotting Errors With Row and Column Totals
One practical cleaning technique checks whether values in a row or column are internally consistent. If a stated total does not match the sum of the individual entries, one entry has been recorded incorrectly.
Worked example: a table records books read by three Year 9 classes in Term 1. If the printed column total is 120 but the three class entries are 42, 38 and 45 — summing to 125 — one of those three entries is wrong and needs investigating against the original records.
An incomplete response is a returned questionnaire with some blank answers — perhaps a question felt too personal. A non-response is a questionnaire never returned at all.
- Incomplete responses can sometimes be rescued for the questions that were answered.
- Non-responses produce no usable data from that respondent at all.
- If either group shares a characteristic (e.g. all skip the income question), the results are biased.
Control Groups, Matched Pairs and Extraneous Variables (Higher)
When comparing a treatment group against a control group, the control group receives no treatment, so it acts as a baseline. Without one, you cannot know whether any change is due to the treatment or would have happened anyway.
Extraneous variables are factors, other than the one being tested, that could still affect the outcome. If testing whether music aids concentration, room temperature and time of day are extraneous variables that should be kept the same across both groups.
One way to control extraneous variables is to use matched pairs: participants are paired up so each pair is similar in traits that might affect the result (e.g. similar prior test scores), then one member of each pair goes into the control group and the other into the treatment group. This reduces the chance that a difference in outcome is really just a difference between the two groups of people.