Survey Sampling#

This page covers how to choose and reach participants for survey studies. See also Survey Questionnaire for questionnaire design and Recruitment Channels for broader participant sourcing guidance.

Survey results depend a lot on who responds to them - and therefore, they way participants are chosen or invited is very important to the whole process. It is advisable to first specify the target population (i.e. who we want to know more about), then think about how these persons can be reached best: If the target are computer science students, a convenience sample at the local university is usually sufficient. If we want to draw conclusions on the general public, sampling methods such as Crowdsourcing are a better fit. We describe a few of the most common sampling methods below - given that none of these are perfect, researchers must make tradeoffs to balance data quality with available resources (e.g. money to pay participants).

Sampling Methods#

• Convenience Sampling: Try to acquire participants from easy sources such as university students (if you’re researching at a university) or personal contacts. This is usually easy and cheap, but results in non-representative participants from similar backgrounds.
• Snowball Sampling: Similar to convenience sampling, but includes participants recruiting their friends who recruit their friends and so on. Can yield a slightly more representative sample and a larger participant pool, but will usually still result in participants that resemble each other.
• Crowdsourcing: Use crowdsourcing platforms such as Amazon MTurk or Prolific. These cover up a more representative portion of the general public that can also be recruited for behavioral experiments, but the costs are higher. Statistics on platform users or previous research should be consulted to get insights in available users as there might be still some bias (e.g. MTurk workers tend to be younger, better educated and more tech-savvy than the general public).
• Social Media: Use social media platforms such as Twitter to reach participants. This yields higher representativeness than Snowball Sampling, and depending on the topic, data can already be accessible without directly contacting users (as profiles are often public and can already include e.g. demographical information). Possible bias include that only a small portion of the general public is actively using e.g. Twitter, and that information on these profiles is self-reported and can be wrong.
• Online Census Platforms: Online Platforms that are specialized in putting together representative samples (or samples based on demographic criteria of the researchers choosing). This should yield much better representativeness than other methods and researchers can define their sample very well. However, this is more expensive than Crowdsourcing, and as over 90% of invitees do not respond to surveys, there is still a response bias present.
• Probabilistic Samples: Sampling method where every person in the population has a non-zero chance of being drawn at random. This is the sampling method that leads to the best representativeness, but is also very hard to achieve. Tools such as Google Consumer Surveys (GCS) or KnowledgePanel can be used, but they are even more expensive than Online Census Platforms. They are also often very limited, as for example GCS only allows up to 10 question items per survey, including demographics.