Draft: This page is still a draft and will be extended in the future. Be aware that some information might be missing or incomplete.

Recruitment Channels#

Most of our research requires recruiting participants and conducting studies, experiments, surveys, etc. with them. This page should give you a brief overview on different recruitment channels we use, our experiences with them, and what to consider when using them.

Recently, there was some workshop on recruitment for empirical software engineering research. (It might give some hints or inspiration, but is only a workshop so be cautious adapting advice from there.)

Our group also published a paper on choosing recruitment channels.

Freelancers#

Upwork#

Upwork is a online platform that brings together freelancers and clients. This includes but is not limited to IT professionals (developers, DevOps, security, architects).

General Process#

General process for using Upwork:

1. Create an Upwork job post.
2. Wait for freelancers that “apply” to your post. Additionally, you can also search for potential participants and invite them to your job.
3. Check out applications and answer any questions that the Upworker might have.
4. If you decide to recruit the Upworker, make an “offer”. This includes to already deposit the money in Upwork escrow.
5. The Upworker has to accept the offer. At this point the contract is established.
6. Conduct your study/experiment/survey/interview with the Upworker. At this stage you should send links to a survey, make an appointment for the interview, etc.
7. After the Upworker finished the study completely, you can release the payment (for the “milestone” of your study).
8. Reimbursement: Collect all receipts and invoices from Upwork and the credit card and prepare reimbursement.

Some hints on Upwork are available in a small workshop paper.

Terms of Service#

In general, adhere to Upwork’s Terms of Service and especially the User Agreement. This is important to protect both, the participants and us. Keep in mind that the Upworkers may do this for a living and violating any of Upwork’s regulations may put them at risk for losing their account and income. And we also do not want to get our account banned.

Most important, adhere to the Non-Circumvention paragraph. This mainly includes the following aspects:

• 7.2 Communicating through the site; not sharing contact details

Prior to establishing a contract with the Upworker you must not share any contact information with the Upworker (and vice versa). All communication at this stage has to be handled via Upwork. This includes not providing any adresses, emails, names, etc. in the Upwork job post. In our experience, some Upworkers may not want to share their contact details (e.g., email), even if the contract is already established.

If we understand the paragraph correct, it should be okay to share contact information after hiring and establishing the contract. But keep in mind, that you should keep your identity a secret until an Upworker took your job. Having a GDPR-compliant screening questionnaire, where in general all participating institutions are listed, could become a problem. Upwork gives the possibility to add questions to the job post that participants must answer if the apply for the job (that can be used as a “mini” screening).

• 7.1 Making Payments Through Upwork

You are required to use Upwork at least for two years as the only way/platform to collaborate with the Upworker (i.e., conduct studies). This means that you are not allowed to pay outside of Upwork during this time.

Other things to keep in mind#

• Fast communication: Upworkers are professionals and many reply quite fast, do the same.
• Upworkers might want to chat or ask for details.
• Upworkers may ask for a review (which is public on their profile). We should therefore make an effort and write a decent, professional, and not too short review (at least if the Upworkers ask for it). If an Upworker later want’s feedback after the contract was already ended, the feedback is locked. In that the Upworker has to enabled us to change feedback.
• We can also get reviews and want to get good reviews. Having bad reviews might prevent pontential Upworkers from participating in our studies!
  • Older posts won’t get much attention; make sure to create a new post after some time. (Always remove/deactivate the old post, Upwork disallows duplicates.)

Direct Recruiting#

In addition to the possibility of waiting for applicants who apply to the advertised post, it is possible actively send other Upworkers an invitation message and ask them to apply for job.

• The Upwork service comes with a huge set of filters for finding software professionals. This offers you the possibility to search for Freelancers with a specific set of skills, experiences, geolocation, etc.
• You also have the possibility to look at the upworkers’ last jobs and their received payments to estimate, whether your job’s compensation matches their hourly wages.
• Per job post only 15 invites could be active at the same time. If a Upworker rejects your invite you will get your invite back and you could start inviting another one. You are also able to withdraw an active invite. If an Upworker accepts your invite and applies for the job, you will loose one of the 15 invites.
• Upwork gives you also the option to upgrade you job post (for around $30). This will remove the restriction of having only 15 invites.

Strengths#

• Upworkers seem to be very motivated in general. Especially the “high price” upworkers turned out to be skilled and experienced.

Weaknesses#

• When you are paying only small amounts, it is possible that this is not competitive enough to get experienced participants or participants from “high price countries” (e.g., US, UK, Germany).
• The platform is not made for mass recruiting, as it is not a platform for research. (Typical Upwork use case is a client hiring one or two freelancers for a job.) This means, that hiring and payments may need more manual work as it cannot be automated very well.
• Payment handling is a little complex, as you cannot get a single invoice for all participants. Instead, you get 4-5 PDFs per Upworker (receipt for credit payment in escrow, invoice for the Upwokr fee, invoice of the Upworker, receipt about paying the invoice from escrow). Please note, that you also need the credit card’s receipt for successful reimbursement.

Freelancer.com#

Freelancer.com is a another platform for hiring freelancers, similar to Upwork.

Crowd Workers#

Amazon Mechanical Turk#

Data Quality: In recent years, MTurk had some problems with quality. Discuss whether this platform is the right choice for your experiment.

Amazon Mechanical Turk (MTurk) is a crowdsourcing marketplace that makes it easier for individuals and businesses to outsource their processes and jobs to a distributed workforce who can perform these tasks (“HITs”) virtually.

We generally run MTurk HITs with a Qualtrics survey as “backend”. To link back a (finished) survey with specific MTurk IDs (for payment or rejection), you can place a randomly generated number at the end of the survey (and store it with the results). The MTurkers have to enter to finish the MTurk HIT. More information & random number setup:

• https://blog.mturk.com/getting-great-survey-results-from-mturk-and-qualtrics-be1704ff9786

Screener#

For some survey, you might want to run a screener survey (pre-survey) before your main survey on MTurk. E.g., to filter for iPhone users or participants that use a specific messenger.

1. Create screener survey with few questions in Qualtrics.
   • With a RandomNumber at survey end that participants need to enter at MTurk.
   • Store this number in the survey results so we can link Qualtrics answers with MTurk accounts.
2. Create MTurk HIT with screener, pay all valid participants a small amount for participating (reject failed attention checks later).
3. Create custom HIT (filter: MTurk IDs of participants that passed screener) for the actual survey.
   • Again with a RandomNumber at the end so we can link accounts for payment etc.