Interviews

Interviews#

Resources and guides for conducting research interviews in usable security and human-centered security.

Start with Pre-study, continue with Study after IRB approval, and use Reporting when writing the paper.

  • IF it is your first time conducting interviews, consider reading through the whole page.
  • IF you are short on time or revisiting interviews, check out the Cheat Sheet section for a high-level overview.
  • IF you are looking for a specific aspect, check out the Table of Contents on the right side.
  • ELSE check out the Examples section for inspiration from real-world interview papers.

Interview Pages#

  • Interviews: Pre-study covers preparation before data collection, including data protection, interview guides, piloting, recruiting, and consent collection.
  • Interviews: Study covers conducting interviews, data collection, transcription, and data analysis.
  • Interviews: Reporting covers writing interview methods and results, quotes, demographics, reporting numbers, and saturation.

Examples#

See also the group drive for past IRB example applications and consent forms: https://drive.google.com/drive/folders/11rJkniObDSS1tXbCuHUC3euDCRDBaJPn?usp=drive_link

  • Context Matters: Qualitative Insights into Developers’ Approaches and Challenges with Software Composition Analysis, USENIX 2025, PDF, Artifacts
    • PopulationIndustry Devs AreaTooling
  • It’s like flossing your teeth: On the Importance and Challenges of Reproducible Builds for Software Supply Chain Security, IEEE S&P 2023, PDF
    • PopulationReproducible Builds Professionals AreaBuilds
  • “Always Contribute Back”: A Qualitative Study on Security Challenges of the Open Source Supply Chain, IEEE S&P 2023, PDF, Artifacts
    • PopulationIndustry Devs AreaDependencies
  • Committed to Trust: A Qualitative Study on Security & Trust in Open Source Software Projects, IEEE S&P 2022, PDF, Artifacts
    • PopulationOpen Source Devs AreaOS Security

Approach#

The general approach for scientific interviews differs somewhat by type (see below), but generally, the following steps are taken (correspond to sections on this page):

  1. Sync on Data Protection plan
  2. Create Interview Guide Draft
  3. Set up recruitment pipeline
  4. Important: Get IRB approval (generally requires the artifacts from the steps above)
  5. Do Pilot testing
  6. Somewhat overlapping:
  7. Report Results

See also the PPQ for early project planning.

Data Protection: Data protection is especially relevant for interviews, as you can’t control participant answers and most collected data (including simple voice recordings) are considered personal information.

Give the data protection section a close consideration.

Cheat Sheet#

High-level overview if short on time or revisiting interviews, see sections below for more in-depth insights.

Interview Cheat Sheet

Structure:#

  1. Opener & Introduction
  2. Explain purpose and research context of interview
  3. Encourage natural interview flow, let participant speak, guide if necessary
  4. Ask probing questions to gain deeper insights
  5. Debrief & end the interview

Interview Types:#

  • Unstructured / Open / Exploratory:
    • Only initial question(s) might be planned
    • Interviewer develops interview based on answers
  • Structured:
    • Rigid script, branches only on pre-planed splits
    • Similar to a (guided) survey
  • Semi-Structured Interviews:
    • Guiding questions with optional follow-ups

Usable security / HCS mainly conducts semi-structured interviews.

  • Allows for some quasi-quant data (unlike unstructured)
  • Good coverage of everything we want to know (unlike unstructured)
  • Can follow-up or skip (unlike fully structured)

Good Interview Questions#

  • Build your interview guide around research questions, alignment matrix to see that they match
  • Go from broad to specific (Intro, general questions, specific questions, outro)
  • Allow participants to mention concepts themselves before you ask about them. (e.g. allow them to bring up security concerns)
  • Mostly open questions (instead of questions that can be answered with yes/no)
  • Clear questions
  • Applicable (ask for thoughts / experiences that they can answer)
  • Unbiased questions
  • Pilot them multiple times so you know your guide more or less by heart & are prepared for responses

Interview Goals#

  • The goal of the interviews is to collect data about people’s perceptions / challenges / ideas / … in their own words.

Priming and Leading#

  • It is important that the interviewer does not convey anything to the participant about what they know or believe about how the topic works/should work.
  • This means that the interviewer must pay careful attention to the language each participant uses during the interview, and refer to the same concepts the participant talks about using the same kinds of words as the participant, avoiding to indicate vocabulary of their own.
  • The more we guide their responses, the more we will be collecting data about something they’re only thinking about because we asked them to think about it. We want to know what they think about this, not what they think about what WE think about this.

Focus Points#

  • Unprompted mentions of privacy, security, ethics, fairness considerations.
  • Company pressures of requirements, deadlines, things that get cut due to them.
  • Challenges they face.
  • How other departments that may be involved in security or oversight interact with them.
  • Reasons why defenses they know are not used.
  • Feelings of confusion, defeat, carelessness, etc.

Misc#

  • Video interviews are great for non-verbal communication (nodding at the participant etc.). Also keeps the transcript cleaner.
  • Co-interviewers are nice for backup but also because it may make it easier to last through the interviews! Or interview experience & subject matter expertise can be matched up. Clarify how co-interviewer can ask questions (raise hands, end of section, backchannel via zoom text messages…); clarify who asks the first question if you equally co-interview; usually the person who asks a question follows up; speak up in cases of follow-ups. You can debrief together!
  • Debriefing: Brain-dump on advisor, or leaving many comments on transcript. Talk about or write down what the most insightful or interesting parts of the interview were. Summary may also be helpful later in the project for onboarding other people or choosing transcripts to read together or in detail, or for “which are so different that they may be good for codebooks”.
  • What to do if interviewee is dishonest: Whatever is least painful. Usually end interview in a coordinated way, but if you interviewed them for a bit, pay them too.

Setup#

  • Be there early, have waiting room
  • Limit amount of appointments in calendar
  • Make sure tech setup work (zoom, mic, camera, obs, etc)
  • Make sure you have recently taken a bio break
  • Bring water

Interview Types#

Most of our usable security research involves semi-structured interviews:

Semi-Structured Interview#

See also: Our Interview Guide section for examples of semi-structured questionnaires.
Often structured as a few sections with very general main question, each followed by more specific sub-questions to include not covered aspects. E.g.,

  • Main question: “Can you tell us a bit about your project”
  • If not covered by participant as follow-ups: “How many people are involved”, “When was the project created”, etc.

Challenges for semi-structured interviews include:

  • Higher mental load for the interviewer: you need to keep track (and understand) participants’ answers to decided which follow-ups to ask, often even across multiple interview sections.
  • During the interview design phase, you need to weight interview flow vs. what you want to report in the paper (to many follow-ups and it turns into a super long structured interview, which leaves less room for quotes and tires the participant).

Structured Interview#

Typically based on the same research logic as questionnaires: Standardized ways of asking questions are thought to lead to answers that can be compared across participants and possibly quantified.

  • Interviewers are supposed to “read questions exactly as worded to every respondent and are trained never to provide information beyond what is scripted in the questionnaire.”
  • Commonly used in CATIs (where call center agents do the interviews based on a survey template).

Unstructured Interview#

At the other end of the continuum lie interviews that have little preset structure. Can only start with one initial question and then continue by discretion of the interviewer.

  • Not that common in usable security research.

Other Variations#

  • Group Interviews: (Smallish) group of participants discusses high-level questions
    • Setup usually semi-structured (fixed high-level questions / discussion starters)
    • Unique group dynamics
    • Potential biases through peer interactions
  • CATI Interviews: (Computer Assisted Telephone Interviews)
    • Interviewer supported by computer system (i.e., they click through a survey script and fill in participants’ answers)
    • Less flexible than interviews, but easier to scale
    • Usually done through service providers (call centers) that work with provided contact lists + scripts
    • Example paper: https://dwermke.com/publications/2021-conf-usenix-huaman/
  • Walkthrough Interviews: Interviewer accompanies participant through task or process
    • Asking questions about actions, thoughts, and decisions
    • Common in usability testing
    • Can also be post-task (retrospective think-aloud) by going through a video or similar
  • Self-Interviewing: Respondent conducts interview themselves with a recording device or written responses.
    • Useful for:
      • Remote locations without internet access
      • Longitudinal studies (e.g., one self-interview every day for a month)
      • Anonymity or self-reflection is important
  • Dyadic Interviews: Similar to group interviews but focused on (relationship between) two participants
    • Involves two participants interacting with each other while being interviewed by a researcher.
    • Used to explore relationships, collaborations, or conflicts between participants.
    • Example: Developers’ perceptions and challenges after using a pair programming approach for some time

References#

References and helpful resources:

  • Interviewing As Qualitative Research: A Guide for Researchers in Education And the Social Sciences by Irving Seidman, Teachers College Press (2006).
  • Qualitative Interviewing: Understanding Qualitative Research by Svend Brinkmann, Oxford University Press (2013).
Previous Methods Pre-study Next