Questionnaires, Pilot Studies and Cleaning Data
Designing a Good Questionnaire
A questionnaire is only as good as the questions it contains. Even with a perfect sample, a poorly designed questionnaire will produce data that is biased, incomplete, or simply not measuring what you intended. The specification identifies several key features to consider: leading questions, avoiding biased sources, the choice of open or closed questions, interview technique, and time factors.
Leading Questions
A leading question nudges the respondent towards a particular answer. Consider these two versions:
- Leading: “Don’t you agree that students should do at least one hour of exercise a day?”
- Neutral: “How many hours of exercise do you do on a typical day?”
The first version makes it socially awkward to disagree. Respondents who exercise very little may inflate their answers to avoid seeming lazy. The second makes no assumption. Always ask: does this question imply there is a right or preferred answer? If yes, rewrite it.
Which of these is leading? Think before reading on.
- 1. “How often do you eat unhealthy food?” — Leading. The word “unhealthy” primes the respondent to under-report.
- 2. “What type of food do you eat most often?” — Not leading. No assumption is built in.
- 3. “Most young people spend too much time on their phones. How many hours do you spend on your phone per day?” — Leading. The preamble sets an expectation that shapes the answer.
Open and Closed Questions
Open questions invite free-text answers. The response might be detailed and revealing, but every answer is different — comparing many different written opinions requires careful judgement rather than straightforward statistical analysis.
Closed questions provide fixed options. The data is straightforward to count and compare. Always check that closed-question options are exhaustive (they cover all possible answers) and mutually exclusive (no answer fits more than one option).
Open questions: rich detail but hard to analyse statistically. Closed questions: easy to count and compare but may not capture every possible view. Good questionnaires often use both types.
Interview Technique and Time Factors
The way an interview is conducted can affect responses. Face-to-face interviews allow clarification of confusing questions, but the interviewer’s presence can cause socially desirable answers. Telephone or online questionnaires offer more anonymity, which may encourage more honest responses on sensitive topics, though response rates are often lower.
Time factors matter too. Asking about sleep habits on a Monday (after a weekend) will produce different answers from asking mid-week. Seasonal effects can skew data about outdoor activities, mental wellbeing, or energy consumption. A good questionnaire design considers when the questions will be asked and whether timing could bias the results.
Pilot Studies
A pilot study is a small-scale practice run — you administer the questionnaire to a small group before the main data collection. It serves several purposes:
- Identifying ambiguous or unclear questions that respondents interpret differently.
- Checking that response categories are exhaustive and mutually exclusive.
- Estimating how long the questionnaire takes, so respondents can be informed.
- Revealing leading or loaded language the researcher did not notice when writing the questions.
The pilot participants are typically not included in the main investigation. After the pilot, the questionnaire is revised before real data collection begins. Skipping the pilot is a common planning mistake — it is far cheaper to fix problems before 500 responses are collected than afterwards.
Cleaning Data
Raw data is rarely perfect. Before analysis begins, the dataset must be cleaned — checked and corrected for errors and inconsistencies. The specification identifies several types of problem that require data cleaning:
Missing data: Some respondents skip questions. A missing value cannot be included in calculations — you must decide whether to exclude the incomplete response, impute a value, or note the gap and acknowledge the effect on your results.
Non-responses: The entire questionnaire is not returned. If non-respondents share a characteristic, excluding them introduces bias.
Incorrect formats: A question asking for age might receive “sixteen” (text) alongside numerical entries. Spreadsheet software cannot calculate with inconsistent formats, so these must be standardised.
Extraneous notation: Currency symbols, spaces, or units entered inconsistently (“£4500” vs “4500” vs “4,500”) will prevent calculations. These must be removed or standardised before analysis.
Three main data cleaning problems: (1) missing data — blank cells where responses were skipped; (2) incorrect formats — inconsistent data entry; (3) extraneous notation — symbols or punctuation that prevent calculation.
Spotting Errors Using Row and Column Totals
One practical cleaning technique is to check whether values in a row or column are internally consistent. If the row total for a group is 45 but the three individual entries sum to 50, one entry is wrong. The specification notes that students may be asked to use row or column totals to identify a wrongly recorded value.
A worked example: a table records the number of books read by students in three year groups over two terms. If Year 9’s Term 1 column total is 120 but the individual class entries are 42, 38, and 45 (summing to 125), one of these three entries has been incorrectly recorded and needs investigating.
Control Groups and Extraneous Variables (Higher)
When an investigation compares two groups — for example, students who received a new study intervention versus those who did not — the group that does not receive the treatment is the control group. Without a control group, you cannot know whether any improvement is due to the intervention or would have happened anyway.
Extraneous variables — factors that could influence the outcome but are not the variable being tested — must be identified and held constant where possible. For example, if testing whether music helps concentration, room temperature and time of day are extraneous variables that should be kept the same across both groups.
An incomplete response is one where a participant returns the questionnaire but leaves some questions blank — perhaps because a question was unclear or felt too personal. A non-response is where the questionnaire is not returned at all.
Both create gaps in the data. Incomplete responses might be rescued for the questions that were answered, whereas non-responses produce no usable data. In both cases, if those affected share a characteristic, the results are biased.