Survey Research Training and Consultancy

Good examples of poor survey questions

Example 1: From a local council postal survey

The uncleaned data showed a large percentage of parents answering “no” to Q10 and continuing on to mark childcare in Q11! Fortunately this had a clear pattern. These parents marked the “Informal” category. These parents clearly did not think that family, friends and neighbours counted as childcare. This is a fault in the design of the simple question about childcare use.

Q10 Do you use childcare?Y/N
Q11 If yes, what type of childcare to you use? 
 Informal, e.g. Family / Friends / Neighbours1
Childminder, Self-employed carers based in their own homes2
Home Based Child Carer, Employed by parents and based in parents home3
Nanny / Aupair, Employed by parents and based in parents home4
Crèche, Occasional care for parents to access work, training or one off events5
Pre-school Play Group, Sessions of play and/or education for children aged 2-5yrs6
Day Nursery, To provide care and education for children aged 6 weeks to 5 yrs7
Breakfast / Before School Club, Safe place to play before school starts for children aged 3-14yrs8
After School Club, Safe place to play after school finishes for children aged 3-14yrs9
Holiday Play Scheme / Club, Safe place to play during school holidays for children aged 3-14yrs10
Specialist care for ages 15, 16, 17 - children with a disability only11
Other, please state12

Example 2:

From my 2011 AAPOR presentation based on ESRC SDMI Mixed Modes and Measurement Error data (grant number RES-175-25-0007) with colleagues Michelle Gray and Margaret Blake (National Centre for Social Research) and Steven Hope (University College London). Other core project team members include: Gerry Nicolaas (National Centre for Social Research) and Peter Lynn and Annette Jäckle, (Institute for Social and Economic Research). Below respondents have been randomly assigned to two conditions.

8 Category Versions3 Category Versions

FM75. Which of these best describes your home? Would you say a . . . (READ OUT) . . .

Detached house1
Semi-detached house2
Terraced house3
Flat in a block of flats5
Flat in a house6
Or other?8

FM75. Which of these best describes your home? Would you say a . . . (READ OUT) . . .

Flat or maisonette2
Or other ?3

When the 8 category version is collapsed into the same 3 categories as the 3 category version, you don't get the same results. Cognitive interviewing suggested that 'type of dwelling' was a confusing concept when this would not have been expected:

  • What the [Blip] difference is there between a maisonette and a flat and a block of flats, a flat and a house? (Male, postgraduate degree, employed, high income, White British
  • Regarding a maisonette. Household member: I'll call it a duplex, yeah. Respondent: Well, it's what they call it in the South. (Male, postgraduate degree, employed, high income, White British)
  • R answered 'flat', but the interviewer observed it as semi-detached house. R said it had to be very large to be called a house (Female, higher education below degree level, employed, medium income, other ethnicity).

Example 3: From Housing Association postal questionnaire

2. How happy are you with the area you live in?

Very satisfied
Fairly satisfied
Neither satisfied nor dissatisfied
Fairly unsatisfied
Very unsatisfied
Don't know
  • Mismatch between stem and answer categories - Happiness vs. Satisfaction
  • Uses “dissatisfied” and “unsatisfied”
  • "Area" is ambiguous
  • "You" is ambiguous - Singular (meaning just the respondent) or plural (meaning the whole household)
  • A reference period would help
  • Too general to be useful
  • Etc, etc.
 Photo heading: Copyright © 2013 The Survey Coach.    All rights reserved.