Interviews: Pre-study#

This page covers the work before data collection: data protection, guide design, IRB filing, recruiting, consent, and pilot logistics.

Data Protection#

Data protection and privacy is very important in interview studies. Stick to the following points when doing interview studies to ensure that you will not violate data protection and privacy.

Everyone in the project (all student assistants, all PhDs, all collaborators) should be introduced to data protection and privacy in the project and all its processes during onboarding — before actually collecting the sensitive data.

Data Storage

See also article on VeraCrypt for creating encrypted folders.

• Store contact data and similar data (e.g., data needed for payment of participants) separate from the actual interview data (recordings, transcripts, etc.).
• Assign a unique ID on a per interview basis, e.g., use simple numbers (00, 01, 02, …) or longer identifiers (GER01, GER02, …, US01, US02, …) and maintain a mapping in a secure place with corresponding contact data.
• Store all the data in safe encrypted places (BOTH encryption at rest and in transit). Never store the data publicly for any uninvolved people or in unencrypted form. Know where the data is stored.
• Do not send any data in unencrypted form (e.g., clear text emails).

GDPR requirements:

• Do not use cloud storage by third parties (e.g., Google Drive or Dropbox).
• Do not use any online office suite (e.g., Google Docs or Office 365).

Transcription

• Remove all personally identifiable information (PII) of the participants as well as information about companies or any other sensitive information/unique identifiers. Replace it by some non sensitive information/description while keeping the important contextual information (e.g., "Robert" → [participant], "Sascha" → [Co-Author], "my colleague Jason" → my [coworker], "Facebook" → [large internet company], product names "WhatsApp" → [smartphone messenger app], etc.).
• If some transcription service is used, ensure the transfer of interview recordings and transcripts in encrypted form.
• After transcription: Authors read the whole transcript to ensure that all data is adequately anonymized/pseudonymized. If a position has been forgotten, this will be corrected.
• When the transcript is finalized, the recordings are no longer needed and therefore should be deleted immediately.

Publication

• Never publish the interview recordings or transcripts, as the content will probably have enough context information to unblind and deanonymize the involved participants, companies, etc. - despite anonymization/pseudonymization. (Treat transcripts the same way as recordings.)
• It is ok to publish very small parts of the transcribed interviews, e.g., quotes, as long as they contain no PII and will not allow deanonymization/unblinding. Double check this when you are going to use a quote from the already anonymized transcripts.
• If you want to publish information about the interviews (e.g., for replication of your study) you could publish information about recruiting (texts, emails, social media posts), consent forms, code books, example codes, example quotes. See also Replication & Artifacts.
• Be precise when reporting how PII was handled since anonymization is not deidentification(!)
  • Anonymization is defined as:

  The act of permanently and completely removing personal identifiers from data, such as converting personally identifiable information into aggregated data. (Source)

  • We are usually only able to de-identify, especially in the case of interviews, so make sure to report it as such!

Interview Guide#

If you are conducting an interview, using an interview guide is important for structuring your questions, planning out how you’ll pose your questions to the interviewee, and keeping your questions consistent throughout multiple interviews.

Study Planning#

For early study planning before drafting the guide, see the PPQ.

Examples:

• Some with full pre-recording stuff like context and consent in them:
  • https://publications.teamusec.de/2022-oakland-sec-oss/files/interview-guide.pdf
  • https://publications.teamusec.de/2023-oakland-oss-consumers/pdf/interview-guide.pdf
• Reproducible builds interviews as an example for more tool-focused: https://teamusec.de/pdf/conf-oakland-fourne23-appendix.pdf
• Implementing Crypto standards as a very recent example: https://osf.io/yehqw

Format#

You can write your interview guide in any text document and even print it out for interviews.

For semi-structured interviews, a good structural approach is to write down the general question, check the provided follow-ups during the answer, and then specifically ask for the missed follow-ups.

Some features that help with this approach are:

• Checkboxes for sections, questions, and follow-up questions so that you or a shadow interviewer can keep track during interviews.
• Question IDs for easier referencing during calls and later publication.
• Color coded headlines for easier visibility during interviews.
• Summary word in front of follow ups for faster search when scanning the document during the interview.
• Page Breaks for different sections to avoid having to turn pages during a question.
• Highlight Branching questions with keywords and visuals, e.g., IF and IF NOT.

A guide with all of those suggestions included could look somewhat like this:

Structure#

As for general structure, including introduction and outro in the interview guide is a good practice to keep interviews more consistent between sessions.

1. Write down the larger research questions of the study. Outline the broad areas of knowledge that are relevant to answering these questions.
2. Develop questions within each of these major areas, shaping them to fit particular kinds of respondents. The goal here is to tap into their experiences and expertise.
3. Ask “how” questions rather than “why” questions to get stories of process rather than acceptable “accounts” of behavior.
4. Develop probes that will elicit more detailed and elaborate responses to key questions. The more detail, the better!
5. Think about the logical flow of the interview. What topics should come first? What follows more or less “naturally”? This may take some adjustment after several interviews.

A guide structure for semi-structured interviews could look like follows:

1. Preamble
   • Greeting
   • Not Judging
   • Consent (only if not gathered before)
2. Main Part
   • Ice Breaker, Building Rapport, Encouraging Questions
   • Demographics (project etc.)
   • Main Questions (usually more general -> more specific)
   • Thoughts & Opinions
3. Outro
   • Debrief
   • Payment

1. Preamble#

• Greeting. Give the participant a short intro about who we are and what the interview is about.
  • Introduce yourself and any other attendees of the call
  • Provide overview and context for the conducted research
  • Ask if they have any questions
  • Ask if they are okay with being recorded
  • Only then start recording and interview
• Not Judging: Towards start of interview:
  • “Before we begin, I want to emphasize that this interview is not about judging your answers or performance. We’re just interested in learning about your experiences and perspectives, so feel free to share openly. There are no right or wrong responses in this interview.”
• Consent. Allow the participant to consent to this study: provide background on how their data will be handled and answer any possible questions. (Sometimes moved to a pre-survey)
  • If you plan on recording the interview, get clear agreement, only then start the recording, and requery if they are okay with being recorded to also have it in the recording.

2. Main Part#

• Ice Breaker. Begin the interview with a “warm-up” question — something that the respondent can answer easily and at some length (though not too long).
  • It does not have to pertain directly to what you are trying to find out (although it might), but this initial rapport-building will put you more at ease with one another and thus will make the rest of the interview flow more smoothly.
  • General demographics (role, position, experience, …) are a good candidate for easy rapport-building questions.
  • Example: “Can you tell us about yourself / your project”
• Difficult questions should be asked toward the end of the interview, when rapport has been established.
• The Thoughts & Opinions should provide some closure for the interview, and leave the respondent feeling empowered, listened to, or otherwise glad that they talked to you.
  • Good candidates are outlook (“Where do you see X in 5 years?”) or improvement (“What would you personally change?”) questions.
  • Example: “If you could make one change to improve the security of X, what would that be?”

3. Outro#

• Debrief Provide the participant with some closure. It can be a good idea to include a feedback section:
  • Anything the participant wants to mention but wasn’t asked yet?
  • Any feedback they have about the interview? (can be on the recording, else take notes).
  • Clearly state when you are turning of the recording.
  • (Off recording) Any company / project / person they think would be a good fit to interview (especially when snowball sampling)
• Payment
  • If you plan on paying the participant, ask for the corresponding contact info, clearly state if the payment might take some processing time on our end.

Writing Questions#

Part of the challenge of conducting an effective interview is writing the right interview questions. Effective interview questions will have the following traits.

Interview questions should be:

Simple Questions#

Keep the questions simple, both in length and structure. Longer questions will be forgotten in a call, complex structure leads to confused participants.

Open-ended Questions#

Give the participant to express their thoughts. Generally avoid yes/no questions unless intentionally required (follow-up, demographic, interview split).

Clear#

Craft each question with simple, clear prose. Avoid confusion about how to understand terms the question itself.

• Prepare definitions (as text in guide appendix) for terms you can’t avoid.

Unbiased#

Avoid making any judgmental assumptions about the subject of research or of the respondent.

• Don’t assume certain answers to be right
• Don’t judge the respondent (bias potential)

References for Writing Questions#

• Writing Effective Interview Questions by Lumen Learning.
• Strategies for Qualitative Interviews by Havard Sociology.

Piloting#

Before starting the actual interviews, it is a good idea to pilot your interview guide and process.

There are two main types of piloting:

1. Internal Piloting: Conduct interviews with colleagues or friends that are somewhat familiar with the topic.
   • Can be done without IRB approval (data not used for publication)
   • Do not include internal pilot data in analysis or reporting.
   • Helps to get a feeling for the interview guide and process, rough time estimates
   • Allows to identify unclear questions or other issues
   • Also use these for training interview skills and testing technical setup (recording, video, transcribing etc.)
   • Generally framed as a normal interview, but you can also choose to stop and discuss things during the interview (especially for early pilots)
2. External Piloting: Conduct interviews with actual participants from the target population.
   • Requires IRB approval (and probably payment etc.)
   • Basically the same as actual interviews, but intended to test the interview guide and process
   • Can be included in the actual study (and counted as main participants) if no major changes are required to the guide or process

Artifacts for IRB#

Before filing, collect:

• Interview guide
• Recruitment text
• Consent form
• Landing page or pre-survey
• Compensation details
• Data protection plan

Submitting IRB#

For filing an IRB protocol, see –> NC State’s IRB.

Recruiting Participants#

Recruiting participants for interviews can be challenging, especially if you are looking for a specific population.

Interviews are generally not representative, so you can combine multiple channels to reach a more diverse set (in terms of the population) of participants. Mixed approaches are common, e.g., starting with direct contact and snowball sampling, and then using social media posts to reach a broader audience.

Recruitment Channels#

For more recruitment options, see –> Recruitment Channels. This page is in drafts, so treat it as working notes rather than finalized guidance.

Common Channels#

There are multiple channels that can be used to recruit participants for interviews, depending on the target population:

• Direct Contact: If you have access to a list of potential participants (e.g., through previous studies, conferences, or professional networks), you can reach out to them directly via email or phone.
• Snowball Sampling: Ask connections (LinkedIn, audiences etc.) and current participants to refer other potential participants who might be interested in the study.
• Networks: Use platforms like Twitter, LinkedIn, Reddit, or specialized forums to post about your study and invite participants.
  • Developer Waterholes: If you are looking for developers, consider reaching out to communities on GitHub, Stack Overflow, or specific technology forums.
  • Professional Groups: Contact professional organizations or groups related to your target population.
  • Conferences and Meetups: Attend relevant conferences, meetups, or webinars to network and recruit participants.
• Freelance Platforms: Use platforms like Upwork or Fiverr to find participants willing to take part in interviews for compensation.

Invites#

2024 Crypto Forum Post:

Subject: Interview Study regarding Cryptographic Standards and their Implementation

Dear All,

We are researchers from the CISPA Helmholtz Center for Information Security, the Max Planck Institutefor Security and Privacy, and the University of Paderborn from Germany.
In coordination with the PQCTeam at NIST, we would like to pitch our current research project.
We are conducting an interview studyto investigate developers' experiences of implementing cryptographic standards. You can find moreinformation about this study at https://research.teamusec.de/2023-secure-crypto-standards/.

If you have any level of experience implementing cryptography, or interesting opinions regarding thecryptographic standardization process, we would greatly appreciate being able to interview you foraround an hour.

To select an interview slot, you can respond via mail to hua...@sec.uni-hannover.de or book a slotdirectly at https://calendly.com/2023-crypto-standard-interviews/schedule.

Best Regards

2023 LinkedIn:

Hi xyz,

Sorry for cold-messaging you about this. I work together with a team of researchers from George Washington University on studying the ethical impacts of privacy and security software.
We would like to interview you about your take on ethics, and consideration of ethics during the development of security and privacy software. You can choose to obtain a $80 gift card as a thank you from us for your participation in this study.

You can find more details on our landing page for the survey: https://gwusec.seas.gwu.edu/ethicalimpactproject/ 

We are thankful that you took the time to read through this message and sorry again for bothering you this way. 

Many thanks, 
Dominik

2023 Reproducible Builds Cold Call:

Title: "Interview: Reproducible Builds Project Approaches and Security Impacts"
Dear [FULL NAME],

We are a group of researchers and are contacting you due to your involvement in [PROJECT]. We are interested in this software projects' involvement and experiences with reproducible builds and would love to interview you about it.

We are not trying to sell anything, you & the project would be treated completely anonymously. We hope your answers and our findings will help with further improving the reproducible builds effort (which might benefit your projects).

If an interview sounds interesting to you, feel free to check our landing page for this research with more information: https://research.teamusec.de/2022-interviews-reproducible/

We are very sorry to take up your valuable time with this email. Regrettably, cold emailing is an approach to reach a more diverse set of projects. If you do not want to participate, please accept our deepest apologies and simply ignore this email.

Best regards,
[AUTHOR]

[SIGNATURE]

2023 Reproducible Builds Established:

Hi [NAME],

We are contacting you due to a recommendation from [RECOMMENDER]. We are a group of researchers interested in reproducible open source software. We are interested in this software projects' involvement and experiences with reproducible builds and would love to interview you about it.

If an interview sounds interesting to you, feel free to check our landing page for this research with more information: https://research.teamusec.de/2022-interviews-reproducible/
If you do not want to participate, please accept our deepest apologies for wasting your valuable time and simply ignore this email.

Best regards,
[AUTHOR]

[SIGNATURE]

2022 Open Source Email:

Dear [name],

We are a group of researchers and contacting you due to your involvement
in the [project name] project on GitHub. 
We are interested in how popular and active open source project 
communities tackle trust & security and would love to interview 
you about it.

We are not trying to sell anything, you & the project would be 
treated completely anonymously. 
We hope your answers and our findings will help with further 
improving security & trust procedures in the open-source community.

If an interview sounds interesting to you, feel free to check out 
our landing page for this research with more information: 
[landing page URL]

Simply respond to this email with your preferred time slots and 
we will work something out.
Lastly, we are very sorry to take up your valuable time with this email. 
Regrettably, cold emailing is an approach to reach a more diverse set 
of projects. If you do not want to participate, please accept our 
deepest apologies and simply ignore this email, 
we will not contact you again.

Best regards,
[researcher]

2022 Open Source channels:

Hi [project name] community,

We are a group of researchers from [affiliations...]  and 
we are interested in how open source projects tackle 
trust & security and would love to interview someone 
from the community about it.

Our Landing page: [landing page URL]

You can message me directly here on [platform] if you 
have any questions or want to schedule an interview!

Thanks for your time,
[researcher]

Landing Page#

A landing web page with some further information and sign up link for interviews can be a good idea to (a) keep invite emails short and to the point and (b) allow invitees to pass on a link in case they know better suited participants (other maintainer, internal developer groups etc).

Examples:

• https://s3c2.org/studies
• https://research.teamusec.de/2023-secure-crypto-standards/
• https://research.teamusec.de/2023-crypto-api-design/

Preparing Consent Forms#

For NC State consent templates and submission requirements, see –> NC State’s IRB.

Consent Collection#

A consent form is a legal document that ensures an ongoing communication process between you and your study participants. The primary purpose of the informed consent process is to protect both the participant and you.

Before beginning data collection, participants need to give their consent. Consent forms are generally required by IRBs for all types of data collection that include humans, e.g., interview studies, online and lab studies, and surveys. See also NC State’s IRB.

Every user study needs a consent form! Make sure to provide a consent form for every user study. Only begin data collection after participants gave their consent!

Common collection methods:

• As quick async survey before the interview (preferred by US IRBs, provides consent list)
• At beginning of interview (before recording); outline consent, answer questions, get consent, get consent “Yes” again on recording.

Note: Being able to collect consent right in the interview call is more of a German thing. A US IRB likely prefers collecting the consent before the interview. Best approach is usually sending the participant an email with the link to a short form (Google Forms, Qualtrics, …), which shows the consent form and collects email + signature/submit of the participant.

After Approval#

After IRB approval, finalize scheduling, recruitment, consent collection, and pilot interview logistics.