How to Survey#

One key technique for understanding and improving how human decision making affects information security and privacy is the gathering of self-reported data from users. This data is typically gathered via survey and interview studies, and serves to inform the broader security and privacy community about user needs, behaviors, and beliefs. The quality of this data, and the validity of subsequent research results, depends on the choices researchers make when designing their experiments.

In the following, we provide an overview of these best practices in the context of studying usability for security and privacy. They can be largely grouped into questionnaire writing, pre-testing and sampling. After data collection survey questionnaire results need to be analyzed using qualtitative and quantitative methods.

Questionnaire Writing#

A survey consists of different questions about participant beliefs, behaviors or opinions. Questions can take different forms, the following being the most common:

• Closed Answer: Participants can choose one or more items from a precompiled list of answer options. The results are better structured and these can be answered faster, but the list of answers might not be exhaustive. Can introduce ordering bias, so answer order should be randomized.

• Open-Ended: Participants are not offered any answer options but can freely describe their answer in a text field. Participants have better control over their answers, but researchers need to go through all provided answers, usually using coding techniques. This takes more time and effort for both participants as well as researchers.

• Likert Scales: Participants are given a set of points that usually describe ranges such as “Very secure - somewhat secure - neither secure nor insecure - somewhat insecure - very insecure”. A Likert scale should consist of 5-10 points: With less points, the answers might not provide a clear picture as there are not enough nuances, with more points, differences begin to vanish. Likert scales should be balanced, i.e. have the same number of points on both sides of a neutral middle ground.

  • Odd number of points: Includes a true neutral option, which is desired by participants with e.g. no opionion on the topic.
  • Even number of points: Has no true neutral position, generates stronger opinions as participants are forced to choose a side.

A survey should be as short as possible and communicate a precise estimate of its length before participants start. Researchers should aim for duration of at most 15-20 minutes. A longer survey will tire participants, and if a shorter duration was communicated, this can be perceived as deceit. Both might lead to increased drop-out rates.

Quality Assurance#

Participants might be careless in their responding, inconsistent, or otherwise distracted. To ensure high quality survey data, this should be considered during the survey design. There are several ways to prevent or detect such behavior. For example, it is possible to include attention check questions in a survey, and discard responses with failed attention checks.

As this is a whole subtopic for itself, the following literature might give some hints:

• Dealing with Careless Responding in Survey Data: Prevention, Identification, and Recommended Best Practices

Bias#

Due to the nature of surveys, researchers need to be very careful when designing surveys. There are a multitude of biases that can be introduced with the presentation of a survey, the question phrasing or even the order of questions and answers. A few examples for this are:

• Ordering bias: The first or last answer options are more likely to be chosen, which is why their order should be randomized.
• Social desirability: Participants tend to answer in a way they deem socially acceptable or “the correct answer”. This is especially true for sensitive topics. Careful phrasing can help mitigate this bias.
• Framing bias: Answers can be influenced by the way they (or the survey) are presented, e.g. if they are positively or negatively connotated. Using neutral wordings can help against this bias.
• Sampling bias: Not all individuals from the target population had an equal chance of being selected, which often leads to wrong assumptions about the population. Can be reduced by adjusting the sampling strategy.

When designing a survey, it is important to keep possible sources for bias in mind and to weigh out options - sometimes, introducing a bias can be justified (e.g. if the answers are not as important as the answer behavior, if there were countermeasures against bias, if there is previous research supporting the experiment design choices). In these cases, researchers should always be acknowledging, report possible limitations and be able to defend their research decisions.

Survey Context#

Aspects that influence a survey outside of its questions; e.g. the question order, sponsors, introductory texts and more can strongly influence how participants respond. For example:

• Asking questions about privacy before a privacy coding study can prime participants and cause them to behave more privacy-aware than usual
• Asking personal questions before a privacy-related question block can lead to more privacy-concerned responses
• Telling participants that their privacy behavior might lead to them reporting more desireable behavior
  • Asking questions on different topics can help diffuse this effect.
• Being sponsored by an university might seem more trust-worthy and lead to more sincere answers
  • It can be beneficial to be vague or even lie about e.g. the true purpose behind a study. Be sure to debrief participants and not to violate ethical standards.

Choice of Words#

“[…] for example, the word ‘usually’ was interpreted in 24 different ways by one study’s participants.” - [1]

As there is often no direct contact to participants, both text sections within the survey and the questions need to be phrased carefully to avoid biases and misunderstandings. There are two common main problems:

• Different participants have different understandings of terms, questions or answer choices, which heavily influences the validity of the survey: If we don’t know what participants had in mind while answering, we cannot compare their answers! Besides technical terms participants might be unfamiliar with, this can also affect common words such as “usually” if there is no precise definition for them.
  • Pre-testing can cover up unclear phrasing and other error sources.
• Researchers using technical or abstract terms due to being more familiar with the topic at hand. This can be harder to avoid as there are often no common alternatives and participants often do not read provided explanations.
  • Try to use more familiar terms where possible
  • Use examples to clarify what you mean
  • Tools such as QUAID can draw attention to common pitfalls within the survey.

Avoid double-barred questions: Questions can include more than one question, e.g. “Do you believe that your employer should require you to update your computer and change your password every six months?” asks for both updates and passwords. For participants, it might be hard to answer both at the same time (which makes it hard to evaluate!), so examples like this should be split in two different questions.

Sensitive Topics#

Surveys can include topics that are regarded as sensitive - e.g. because participants are supposed to keep answers secret, because the topic is very personal and/or disclosing information can lead to serious consequences, or because there is social stigma against certain answers. This can include topics such as sexuality, religion or political views, but also security-related behavior such as password choices. As all data from surveys is self-reported, participants chose to not answere sensitive questions truthfully, due to e.g. lack of trust or because they want to answer “correctly” with a socially acceptable answer. There are some ways we can try to mitigate this:

• Phrase questions indirectly and not about the participants themselves (“Do your friends behave securely?” vs “Do you behave securely”).
• Use forgiving phrasing that signals that every answer is valid (“People have various reasons to choose not to update their phone. Which of these best describes your behavior?”)
• Balance questions by including both a positive and negative answer option (“If there is a serious fuel shortage this winter, do you think there should or should not be a law requiring people to lower the heat in their homes?”)
• It’s considered unethical to not include “a way out” for participants who don’t want to answer a question. This can be achieved by e.g. adding a “I don’t know” or “Prefer not to disclose” option.

A subset of this are demographic questions. As they are not only sensitive but can also highly influence the following questions, they should be put towards the end of the survey. Since demographical questions are usually easy to answer, they also fit very well at the end when participants are possibly already tired from previous questions.

Established Questions: It is advisable to use already established questions, especially for topics such as demographics - many questions have already been phrased and tested. A pre-compiled list of questions can be found.

Further Reading#

• Self-Reports: How the Questions Shape the Answers
• Rating Scales: Numeric Values May Change the Meaning of Scale Labels

Questionnaire Pre-Testing#

In an ideal survey, each respondent interprets the questions in the way we intended. But in reality, survey questions are misunderstood. Participants may find the answers hard to recall, difficult to estimate, and struggle to map their answer to the choices we provide. They frequently interpret words in different ways, and may hesitate to report answers due to social-desirability bias. Additionally, while writing questionnaires we may inadvertently miss key answer choices or accidentally include technical words that our respondents do not understand. Pre-testing surveys and interview protocols can help prevent these and other measurement errors and ensure that self-report survey and interview measurements are as accurate as possible. The main goals of pre-testing are to make sure that respondents correctly understand and interpret questions.

Cognitive Interviews#

Based on: Link, Link2, Link3, Link4, Link5 Link6

Cognitive interviewing is meant to identify and analyze sources of response error in survey questionnaires by focusing on the cognitive processes respondents use to answer questions on a survey or questionnaire.

• Conventionally, cognitive interviewing involves conducting face-to-face (f2f) interviews with small sample sizes of five to 30 respondents.
• Semi-structured, in-depth interviews are conducted on the basis of an interview protocol which contains the questions to be tested in the cognitive interview and the techniques to be adopted, in particular think-aloud and follow-up questions (probing).
  • Probing is used to elicit information about how respondents interpret questions or define specific terms and how respondents arrive at their answers.
  • Probing questions are administered either immediately after the subject has answered the survey question (concurrent) or at the end of the cognitive interview (retrospective)

Think Aloud#

Based on:

• Cognitive Interviewing and Think Aloud Methods
• Thinking Aloud: The #1 Usability Tool

By using thinking aloud, you ask survey respondents to answer questions while continuously thinking out loud — that is, simply verbalizing their thoughts as they move through the survey questoinnaire.

• “Simply” ought to be in quotes, because it’s not that simple for most people to keep up a running monologue. You typically have to prompt respondents to keep them talking.

Probing Questions#

Based on:

• What Do Our Respondents Think We’re Asking? Using Cognitive Interviewing to Improve Medical Education Surveys

Probes can be grouped into comprehension, elaborative and specific probes. Picking the right questions is important. Examples:

• Comprehension Probes
  • What does the term X mean to you?
  • What, to you is, X?
• Elaborative Probes
  • Could you please explain your answer a little further?
  • Why did you answer X?
• Specific Probes
  • What kinds of implementation challenges did you think of when answering this question?
  • Why do you say that you think it is very important that developers write secure code?
• Paraphrasing
  • Can you repeat the question I just asked in your own words?
• Confidence Judgement
  • How sure are you that you have 19 years of experience in writing Python code?
• Recall
  • How do you remember that you have 19 years of experience in writing Python code?
  • How did you come up with your answer?
• General
  • How did you arrive at the answer?
  • Was that easy or hard to answer?
  • I noticed that you hesitated. Tell me what you are thinking.
  • Tell me more about that.

Web Probing#

Based on: Link5

An alternative to conducting f2f cognitive interviews in the lab is to transfer the probing procedure into an online questionnaire, a method called online or web probing. Here, for the questions to be tested, open and closed probing questions are developed and then implemented into an online questionnaire. In the concurrent probing format:

1. respondents first answer a survey question
2. and after clicking on the next button receive one or more probes on the next survey page.

Advantages re Cognitive Interviews

• No cognitive interviewer required
• Recruiting respondents in a quicker and more cost-effective
• Realization of larger sample sizes
• Allows researchers to quantify their pretest findings (Ref)
• Enhances the radius of the regional accessibility
• Rules out any interviewer effects; increases the reliability and comparability of the results (Ref)

Disadvantages re Cognitive Interviews

• Due to the absence of the interviewer, no one can probe for more information, follow up on incomplete answers or provide clarification of the tasks
• Probing is restricted to the scripted questions previously programmed and implemented into the Web survey.
• No one can motivate the respondents during completion of the Web survey to answer the (open) probing questions thoughtfully and elaborately

Sampling#

Survey results depend a lot on who responds to them - and therefore, they way participants are chosen or invited is very important to the whole process. It is advisable to first specify the target population (i.e. who we want to know more about), then think about how these persons can be reached best: If the target are computer science students, a convenience sample at the local university is usually sufficient. If we want to draw conclusions on the general public, sampling methods such as Crowdsourcing are a better fit. We describe a few of the most common sampling methods below - given that none of these are perfect, researchers must make tradeoffs to balance data quality with available resources (e.g. money to pay participants).

Sampling Methods#

• Convenience Sampling: Try to acquire participants from easy sources such as university students (if you’re researching at a university) or personal contacts. This is usually easy and cheap, but results in non-representative participants from similar backgrounds.
• Snowball Sampling: Similar to convenience sampling, but includes participants recruiting their friends who recruit their friends and so on. Can yield a slightly more representative sample and a larger participant pool, but will usually still result in participants that resemble each other.
• Crowdsourcing: Use crowdsourcing platforms such as Amazon MTurk or Prolific. These cover up a more representative portion of the general public that can also be recruited for behavioral experiments, but the costs are higher. Statistics on platform users or previous research should be consulted to get insights in available users as there might be still some bias (e.g. MTurk workers tend to be younger, better educated and more tech-savvy than the general public).
• Social Media: Use social media platforms such as Twitter to reach participants. This yields higher representativeness than Snowball Sampling, and depending on the topic, data can already be accessible without directly contacting users (as profiles are often public and can already include e.g. demographical information). Possible bias include that only a small portion of the general public is actively using e.g. Twitter, and that information on these profiles is self-reported and can be wrong.
• Online Census Platforms: Online Platforms that are specialized in putting together representative samples (or samples based on demographic criteria of the researchers choosing). This should yield much better representativeness than other methods and researchers can define their sample very well. However, this is more expensive than Crowdsourcing, and as over 90% of invitees do not respond to surveys, there is still a response bias present.
• Probabilistic Samples: Sampling method where every person in the population has a non-zero chance of being drawn at random. This is the sampling method that leads to the best representativeness, but is also very hard to achieve. Tools such as Google Consumer Surveys (GCS) or KnowledgePanel can be used, but they are even more expensive than Online Census Platforms. They are also often very limited, as for example GCS only allows up to 10 question items per survey, including demographics.

Consent Forms#

The primary purpose of the informed consent process is to protect both the participant and you. A consent form is a legal document that ensures an ongoing communication process between you and your study participants. Before beginning data collection, participants need to give their consent. We require consent forms for all types of data collection that include humans, e.g., interview studies, online and lab studies, and surveys. Below you find a consent form template and examples from previous studies.

Every user study needs a consent form! Make sure to provide a consent form for every user study. Only begin data collection after participants gave their consent!

For Interviews: For (oral) interviews, the requirements are somewhat less strict, as you probably already got the participants consent. Take a look at the consent form examples below as a general structure for your interview intro.

Some ideas that should appear in the intro:

• “I am going to interview you about X, this will take about Y minutes”
• " This is completly anonymous, and your answers will only be used for X"
• “Are you okay with me recording this interview? This recording will only be accessed by X and deleted after Y.”
• “Are you ready to start the interview?”

Consent Form Protocol#

Please follow the protocol below whenever you need to create a consent form.

TODO: NC State form

Special Blocks#

• Disclaimer if we use IP filtering:

  Warning! This survey uses a protocol to check that you are responding from inside the U.S. and not using a proxy or Virtual Private Network (VPN) to hide your country. In order to take this survey, please turn off your proxy/VPN if you are using one. Failure to do this might prevent you from completing the HIT.

• If we want to use the data for teaching purposes:

  We would like to use your responses (in an anonymized version) for teaching purposes. This option is voluntary.

  • I agree that my anonymized responses may be used for teaching purposes.

Consent Form Examples#

Question Blocks#

Example question blocks from established scales and our previous surveys.

Demographics#

List of potential demographics questions (you probably don’t want all at the same time):

The above gender question is taken from How to do better with gender on surveys: a guide for HCI researchers by Spiel et al. and could be referenced in our papers.

Highly sensitive questions, talk to somebody who knows what they are doing beforehand:

Experts#

Tasks#

Task-specific questions, ask for each task:

Usability#

System Usability Scale (SUS)#

Taken from: usability.gov

The System Usability Scale (SUS) provides a reliable tool for measuring the usability. It consists of a 10 item questionnaire with five response options for respondents; from Strongly agree to Strongly disagree. It allows you to evaluate a wide variety of products and services, including hardware, software, mobile devices, websites and applications.

SUS Benefits#

• has become an industry standard
• very easy scale to administer to participants
• can be used on small sample sizes with reliable results
• is valid: it can effectively differentiate between usable and unusable systems

Considerations when using SUS#

• the scoring system is somewhat complex
• there is a temptation, when you look at the scores, since they are on a scale of 0-100, to interpret them as percentages, they are not
• the best way to interpret your results involves “normalizing” the scores to produce a percentile ranking
• SUS is not diagnostic - its use is in classifying the ease of use of the site, application or environment being tested

The Scale#

Interpreting Scores#

Interpreting scoring can be complex. The participant’s scores for each question are converted to a new number, added together and then multiplied by 2.5 to convert the original scores of 0-40 to 0-100. Though the scores are 0-100, these are not percentages and should be considered only in terms of their percentile ranking.

Based on research, a SUS score above a 68 would be considered above average and anything below 68 is below average, however the best way to interpret your results involves “normalizing” the scores to produce a percentile ranking.

UMUX#

TODO

Cognitive Dimensions (API Usability)#

TODO

Security#

SeBis#

Based on:

• Scaling the Security Wall - Developing a Security Behavior Intentions Scale (SeBIS) (original paper)
• Behavior Ever Follows Intention? A Validation of the Security Behavior Intentions Scale (SeBIS) (validation study)

Secure Software-Development Self-Efficacy SSD-SES#

Based on:

• Building and Validating a Scale for Secure Software Development Self-Efficacy

The Questions#

Privacy#

IUIPC#

Based on:

• Internet Users’ Information Privacy Concerns

The Questions#

Interpretation#

TODO

Westin#

Based on:

• Privacy Indexes: A Survey of Westin’s Studies
• Would a privacy fundamentalist sell their DNA for $1000…if nothing bad happened as a result? The Westin categories, behavioral intentions, and consequences

The Questions#

Interpretation#

TODO

• Fundamentalist: Fundamentalists are generally distrustful of organizations that ask for their personal information, worried about the accuracy of computerized information and additional uses made of it, and are in favor of new laws and regulatory actions to spell out privacy rights and provide enforceable remedies. They generally choose privacy controls over consumer-service benefits when these compete with each other.
  • Agree (strongly or somewhat) with “Consumers…”; AND
  • Disagree (strongly or somewhat) with “Most businesses…”; AND
  • Disagree (strongly or somewhat) with “Existing laws…”.
• Pragmatist: They weigh the benefits to them of various consumer opportunities and services, protections of public safety or enforcement of personal morality against the degree of intrusiveness of personal information sought and the increase in government power involved. They look to see what practical procedures for accuracy, challenge and correction of errors the business organization or government agency follows when consumer or citizen evaluations are involved. They believe that business organizations or government should “earn” the public’s trust rather than assume automatically that they have it. And, where consumer matters are involved, they want the opportunity to decide whether to opt out of even non-evaluative uses of their personal information as in compilations of mailing lists.
  • Disagree (strongly or somewhat) with “Consumers…”; AND
  • Agree (strongly or somewhat) with “Most businesses…”; AND
  • Agree (strongly or somewhat) with “Existing laws…”.
• Unconcerned: The Unconcerned are generally trustful of organizations collecting their personal information, comfortable with existing organizational procedures and uses are ready to forego privacy claims to secure consumer-service benefits or public-order values and not in favor of the enactment of new privacy laws or regulations.
  • Anyone who is not a fundamentalist or unconcerned

References#

• Based on: Link