Survey Questionnaire#

This page covers questionnaire design, consent forms, and reusable question blocks for survey studies. For pre-testing and sampling, see the sibling pages in this section.

Questionnaire Writing#

A survey consists of different questions about participant beliefs, behaviors or opinions. Questions can take different forms, the following being the most common:

• Closed Answer: Participants can choose one or more items from a precompiled list of answer options. The results are better structured and these can be answered faster, but the list of answers might not be exhaustive. Can introduce ordering bias, so answer order should be randomized.
• Open-Ended: Participants are not offered any answer options but can freely describe their answer in a text field. Participants have better control over their answers, but researchers need to go through all provided answers, usually using coding techniques. This takes more time and effort for both participants as well as researchers.
• Likert Scales: Participants are given a set of points that usually describe ranges such as “Very secure - somewhat secure - neither secure nor insecure - somewhat insecure - very insecure”. A Likert scale should consist of 5-10 points: With less points, the answers might not provide a clear picture as there are not enough nuances, with more points, differences begin to vanish. Likert scales should be balanced, i.e. have the same number of points on both sides of a neutral middle ground.
  • Odd number of points: Includes a true neutral option, which is desired by participants with e.g. no opionion on the topic.
  • Even number of points: Has no true neutral position, generates stronger opinions as participants are forced to choose a side.

A survey should be as short as possible and communicate a precise estimate of its length before participants start. Researchers should aim for duration of at most 15-20 minutes. A longer survey will tire participants, and if a shorter duration was communicated, this can be perceived as deceit. Both might lead to increased drop-out rates.

Survey Context#

Aspects that influence a survey outside of its questions; e.g. the question order, sponsors, introductory texts and more can strongly influence how participants respond. For example:

• Asking questions about privacy before a privacy coding study can prime participants and cause them to behave more privacy-aware than usual
• Asking personal questions before a privacy-related question block can lead to more privacy-concerned responses
• Telling participants that their privacy behavior might lead to them reporting more desireable behavior
  • Asking questions on different topics can help diffuse this effect.
• Being sponsored by an university might seem more trust-worthy and lead to more sincere answers
  • It can be beneficial to be vague or even lie about e.g. the true purpose behind a study. Be sure to debrief participants and not to violate ethical standards.

Bias#

Due to the nature of surveys, researchers need to be very careful when designing surveys. There are a multitude of biases that can be introduced with the presentation of a survey, the question phrasing or even the order of questions and answers. A few examples for this are:

• Ordering bias: The first or last answer options are more likely to be chosen, which is why their order should be randomized.
• Social desirability: Participants tend to answer in a way they deem socially acceptable or “the correct answer”. This is especially true for sensitive topics. Careful phrasing can help mitigate this bias.
• Framing bias: Answers can be influenced by the way they (or the survey) are presented, e.g. if they are positively or negatively connotated. Using neutral wordings can help against this bias.
• Sampling bias: Not all individuals from the target population had an equal chance of being selected, which often leads to wrong assumptions about the population. Can be reduced by adjusting the sampling strategy.

When designing a survey, it is important to keep possible sources for bias in mind and to weigh out options - sometimes, introducing a bias can be justified (e.g. if the answers are not as important as the answer behavior, if there were countermeasures against bias, if there is previous research supporting the experiment design choices). In these cases, researchers should always be acknowledging, report possible limitations and be able to defend their research decisions.

Choice of Words#

“[…] for example, the word ‘usually’ was interpreted in 24 different ways by one study’s participants.” - [1]

As there is often no direct contact to participants, both text sections within the survey and the questions need to be phrased carefully to avoid biases and misunderstandings. There are two common main problems:

• Different participants have different understandings of terms, questions or answer choices, which heavily influences the validity of the survey: If we don’t know what participants had in mind while answering, we cannot compare their answers! Besides technical terms participants might be unfamiliar with, this can also affect common words such as “usually” if there is no precise definition for them.
  • Pre-testing can cover up unclear phrasing and other error sources.
• Researchers using technical or abstract terms due to being more familiar with the topic at hand. This can be harder to avoid as there are often no common alternatives and participants often do not read provided explanations.
  • Try to use more familiar terms where possible
  • Use examples to clarify what you mean
  • Tools such as QUAID can draw attention to common pitfalls within the survey.

Avoid double-barred questions: Questions can include more than one question, e.g. “Do you believe that your employer should require you to update your computer and change your password every six months?” asks for both updates and passwords. For participants, it might be hard to answer both at the same time (which makes it hard to evaluate!), so examples like this should be split in two different questions.

Sensitive Topics#

Surveys can include topics that are regarded as sensitive - e.g. because participants are supposed to keep answers secret, because the topic is very personal and/or disclosing information can lead to serious consequences, or because there is social stigma against certain answers. This can include topics such as sexuality, religion or political views, but also security-related behavior such as password choices. As all data from surveys is self-reported, participants chose to not answere sensitive questions truthfully, due to e.g. lack of trust or because they want to answer “correctly” with a socially acceptable answer. There are some ways we can try to mitigate this:

• Phrase questions indirectly and not about the participants themselves (“Do your friends behave securely?” vs “Do you behave securely”).
• Use forgiving phrasing that signals that every answer is valid (“People have various reasons to choose not to update their phone. Which of these best describes your behavior?”)
• Balance questions by including both a positive and negative answer option (“If there is a serious fuel shortage this winter, do you think there should or should not be a law requiring people to lower the heat in their homes?”)
• It’s considered unethical to not include “a way out” for participants who don’t want to answer a question. This can be achieved by e.g. adding a “I don’t know” or “Prefer not to disclose” option.

A subset of this are demographic questions. As they are not only sensitive but can also highly influence the following questions, they should be put towards the end of the survey. Since demographical questions are usually easy to answer, they also fit very well at the end when participants are possibly already tired from previous questions.

Quality Assurance#

Participants might be careless in their responding, inconsistent, or otherwise distracted. To ensure high quality survey data, this should be considered during the survey design. There are several ways to prevent or detect such behavior. For example, it is possible to include attention check questions in a survey, and discard responses with failed attention checks.

As this is a whole subtopic for itself, the following literature might give some hints:

• Dealing with Careless Responding in Survey Data: Prevention, Identification, and Recommended Best Practices

Further Reading#

• Self-Reports: How the Questions Shape the Answers
• Rating Scales: Numeric Values May Change the Meaning of Scale Labels

Consent Forms#

The primary purpose of the informed consent process is to protect both the participant and you. A consent form is a legal document that ensures an ongoing communication process between you and your study participants. Before beginning data collection, participants need to give their consent. We require consent forms for all types of data collection that include humans, e.g., interview studies, online and lab studies, and surveys. Below you find a consent form template and examples from previous studies.

Every user study needs a consent form! Make sure to provide a consent form for every user study. Only begin data collection after participants gave their consent!

Legacy Examples: The consent form examples below are older templates from other universities or past projects. For new studies, use the NC State IRB guidance and provided consent form templates instead.

See the Consent Forms section in NC State’s IRB guidance.

Consent Form Examples#

Special Blocks#

• Disclaimer if we use IP filtering:

  Warning! This survey uses a protocol to check that you are responding from inside the U.S. and not using a proxy or Virtual Private Network (VPN) to hide your country. In order to take this survey, please turn off your proxy/VPN if you are using one. Failure to do this might prevent you from completing the HIT.

• If we want to use the data for teaching purposes:

  We would like to use your responses (in an anonymized version) for teaching purposes. This option is voluntary.

  • I agree that my anonymized responses may be used for teaching purposes.

Question Blocks#

Example question blocks from established scales and our previous surveys.

Misc#

Revoking Permissions#

If you run a survey that doesn’t collect participant data, you want to give them a way to opt-out of the survey. Common approach is to provide them with their survey ID and a contact email:

Demographics#

List of potential demographics questions (you probably don’t want all at the same time):

The above gender question is taken from How to do better with gender on surveys: a guide for HCI researchers by Spiel et al. and could be referenced in our papers.

Highly sensitive questions, talk to somebody who knows what they are doing beforehand:

Experts#

Tasks#

Task-specific questions, ask for each task:

Usability#

System Usability Scale (SUS)#

Taken from: usability.gov

The System Usability Scale (SUS) provides a reliable tool for measuring the usability. It consists of a 10 item questionnaire with five response options for respondents; from Strongly agree to Strongly disagree. It allows you to evaluate a wide variety of products and services, including hardware, software, mobile devices, websites and applications.

SUS Benefits#

• has become an industry standard
• very easy scale to administer to participants
• can be used on small sample sizes with reliable results
• is valid: it can effectively differentiate between usable and unusable systems

Considerations when using SUS#

• the scoring system is somewhat complex
• there is a temptation, when you look at the scores, since they are on a scale of 0-100, to interpret them as percentages, they are not
• the best way to interpret your results involves “normalizing” the scores to produce a percentile ranking
• SUS is not diagnostic - its use is in classifying the ease of use of the site, application or environment being tested

Interpreting Scores#

Interpreting scoring can be complex. The participant’s scores for each question are converted to a new number, added together and then multiplied by 2.5 to convert the original scores of 0-40 to 0-100. Though the scores are 0-100, these are not percentages and should be considered only in terms of their percentile ranking.

Based on research, a SUS score above a 68 would be considered above average and anything below 68 is below average, however the best way to interpret your results involves “normalizing” the scores to produce a percentile ranking.

Security#

SeBis#

Based on:

• Scaling the Security Wall - Developing a Security Behavior Intentions Scale (SeBIS) (original paper)
• Behavior Ever Follows Intention? A Validation of the Security Behavior Intentions Scale (SeBIS) (validation study)

Secure Software-Development Self-Efficacy SSD-SES#

Based on:

• Building and Validating a Scale for Secure Software Development Self-Efficacy

The Questions#

Privacy#

IUIPC#

Based on:

• Internet Users’ Information Privacy Concerns
  IUIPC

  Please rate your agreement or disagreement with the following statements. Options: {Strongly agree, Agree, Somewhat agree, Neutral, Somewhat disagree, Disagree, Strongly disagree}

  1. Control
     • Consumer online privacy is really a matter of consumers’ right to exercise control and autonomy over decisions about how their information is collected, used, and shared.
     • Consumer control of personal information lies at the heart of consumer privacy.
     • I believe that online privacy is invaded when control is lost or unwillingly reduced as a result of a marketing transaction.
  2. Awareness
     • Companies seeking information online should disclose the way the data are collected, processed, and used.
     • A good consumer online privacy policy should have a clear and conspicuous disclosure.
     • It is very important to me that I am aware and knowledgeable about how my personal information will be used.
  3. Collection
     • It usually bothers me when online companies ask me for personal information.
     • When online companies ask me for personal information, I sometimes think twice before providing it.
     • It bothers me to give personal information to so many online companies.
     • I’m concerned that online companies are collecting too much personal information about me.

Westin#

Based on:

• Privacy Indexes: A Survey of Westin’s Studies

• Would a privacy fundamentalist sell their DNA for $1000…if nothing bad happened as a result? The Westin categories, behavioral intentions, and consequences

  Westin

  Participants are asked to score the following 3 items with one of five responses that range from Strongly Agree to Strongly disagree:

  • Statement: Please rate your agreement to the following statements: [Likert items: Strongly agree; agree; neutral; disagree; strongly disagree.]
    1. Consumers have lost all control over how personal information is collected and used by companies.
    2. Most businesses handle the personal information they collect about consumers in a proper and confidential way.
    3. Existing laws and organizational practices provide a reasonable level of protection for consumer privacy today.

• Fundamentalist: Fundamentalists are generally distrustful of organizations that ask for their personal information, worried about the accuracy of computerized information and additional uses made of it, and are in favor of new laws and regulatory actions to spell out privacy rights and provide enforceable remedies. They generally choose privacy controls over consumer-service benefits when these compete with each other.

  • Agree (strongly or somewhat) with “Consumers…”; AND
  • Disagree (strongly or somewhat) with “Most businesses…”; AND
  • Disagree (strongly or somewhat) with “Existing laws…”.

• Pragmatist: They weigh the benefits to them of various consumer opportunities and services, protections of public safety or enforcement of personal morality against the degree of intrusiveness of personal information sought and the increase in government power involved. They look to see what practical procedures for accuracy, challenge and correction of errors the business organization or government agency follows when consumer or citizen evaluations are involved. They believe that business organizations or government should “earn” the public’s trust rather than assume automatically that they have it. And, where consumer matters are involved, they want the opportunity to decide whether to opt out of even non-evaluative uses of their personal information as in compilations of mailing lists.

  • Disagree (strongly or somewhat) with “Consumers…”; AND
  • Agree (strongly or somewhat) with “Most businesses…”; AND
  • Agree (strongly or somewhat) with “Existing laws…”.

• Unconcerned: The Unconcerned are generally trustful of organizations collecting their personal information, comfortable with existing organizational procedures and uses are ready to forego privacy claims to secure consumer-service benefits or public-order values and not in favor of the enactment of new privacy laws or regulations.

  • Anyone who is not a fundamentalist or unconcerned

References#

• Based on: Link