Survey Pre-Testing#
This page covers how to pre-test survey questions before fielding the study. It pairs well with Survey Questionnaire.
In an ideal survey, each respondent interprets the questions in the way we intended. But in reality, survey questions are misunderstood. Participants may find the answers hard to recall, difficult to estimate, and struggle to map their answer to the choices we provide. They frequently interpret words in different ways, and may hesitate to report answers due to social-desirability bias. Additionally, while writing questionnaires we may inadvertently miss key answer choices or accidentally include technical words that our respondents do not understand. Pre-testing surveys and interview protocols can help prevent these and other measurement errors and ensure that self-report survey and interview measurements are as accurate as possible. The main goals of pre-testing are to make sure that respondents correctly understand and interpret questions.
Cognitive Interviews#
Based on: Link, Link2, Link3, Link4, Link5 Link6
Cognitive interviewing is meant to identify and analyze sources of response error in survey questionnaires by focusing on the cognitive processes respondents use to answer questions on a survey or questionnaire.
- Conventionally, cognitive interviewing involves conducting face-to-face (f2f) interviews with small sample sizes of five to 30 respondents.
- Semi-structured, in-depth interviews are conducted on the basis of an interview protocol which contains the questions to be tested in the cognitive interview and the techniques to be adopted, in particular think-aloud and follow-up questions (probing).
- Probing is used to elicit information about how respondents interpret questions or define specific terms and how respondents arrive at their answers.
- Probing questions are administered either immediately after the subject has answered the survey question (concurrent) or at the end of the cognitive interview (retrospective)
Think Aloud#
Based on:
By using thinking aloud, you ask survey respondents to answer questions while continuously thinking out loud — that is, simply verbalizing their thoughts as they move through the survey questoinnaire.
- “Simply” ought to be in quotes, because it’s not that simple for most people to keep up a running monologue. You typically have to prompt respondents to keep them talking.
Probing Questions#
Based on:
Probes can be grouped into comprehension, elaborative and specific probes. Picking the right questions is important. Examples:
- Comprehension Probes
- What does the term X mean to you?
- What, to you is, X?
- Elaborative Probes
- Could you please explain your answer a little further?
- Why did you answer X?
- Specific Probes
- What kinds of implementation challenges did you think of when answering this question?
- Why do you say that you think it is very important that developers write secure code?
- Paraphrasing
- Can you repeat the question I just asked in your own words?
- Confidence Judgement
- How sure are you that you have 19 years of experience in writing Python code?
- Recall
- How do you remember that you have 19 years of experience in writing Python code?
- How did you come up with your answer?
- General
- How did you arrive at the answer?
- Was that easy or hard to answer?
- I noticed that you hesitated. Tell me what you are thinking.
- Tell me more about that.
Web Probing#
Based on: Link5
An alternative to conducting f2f cognitive interviews in the lab is to transfer the probing procedure into an online questionnaire, a method called online or web probing. Here, for the questions to be tested, open and closed probing questions are developed and then implemented into an online questionnaire. In the concurrent probing format:
- respondents first answer a survey question
- and after clicking on the next button receive one or more probes on the next survey page.
Advantages re Cognitive Interviews
- No cognitive interviewer required
- Recruiting respondents in a quicker and more cost-effective
- Realization of larger sample sizes
- Allows researchers to quantify their pretest findings (Ref)
- Enhances the radius of the regional accessibility
- Rules out any interviewer effects; increases the reliability and comparability of the results (Ref)
Disadvantages re Cognitive Interviews
- Due to the absence of the interviewer, no one can probe for more information, follow up on incomplete answers or provide clarification of the tasks
- Probing is restricted to the scripted questions previously programmed and implemented into the Web survey.
- No one can motivate the respondents during completion of the Web survey to answer the (open) probing questions thoughtfully and elaborately