Amazon Mechanical Turk (MTurk)

Much of our proposed research hinges on using Amazon Mechanical Turk (MTurk, for short). In case you are unfamiliar, MTurk is a crowdsourcing platform for Human Intelligence Tasks (HITs).

Let's slow down and dissect this a bit. Crowdsourcing simply means getting something done by using a crowd of people... usually online. HITs refer to tasks that can't be easily automated by a computer. Classic examples of HITs include tagging images, writing descriptions of web sites, filling out surveys, and much more. Fields such as computer science, psychology, and behavioral economics now commonly used crowdsourced HITs on MTurk for academic research. I also mentioned the word platform, which in this case is just a fancy website that helps efficiently connect people who need work done with people willing to do the work.

Our study requires MTurk HITs because information about journal editorial boards is stored all over the internet in a variety of formats, including XML, HTML, .pdf documents, plain text, spreadsheets, and more. Simply put, the only way to get all the information is to look it up, journal by journal, and type all of the information into a consistent format. This sort of task is ideally suited to MTurk. Once we have an established database of journal editors, we need to figure out their genders. Again, this requires crowd intelligence. (Gender is a delicate subject; please read our approach in this lab note.)

How exactly does MTurk work? The MTurk ecosystem consists of requesters and workers. Requesters are the people who post HITs that they would like completed. Along with the description of the HIT, the requester posts a fee that the worker will earn. Workers see lists of HITs online and can choose which ones to complete. When the work is complete, the requester can approve or reject the work. If the work is approved, the worker gets paid.

You might be wondering if requesters can abuse the system by rejecting legitimate work, hence denying the worker of their rightful wage. In theory, yes, this can happen. However, there is a rich online social community of workers who share information. Requesters who act in bad faith will very quickly earn a bad reputation and be ostracized. Workers will no longer choose to complete their HITs, making the MTurk platform useless for these requesters.

Similarly, you might be wondering if workers can abuse the system by submitting work that is bogus. In theory, yes, they can. However, the requester always retains the right to reject the work. Furthermore, the requester can set conditions that workers must meet to even undertake the HIT. In our own research, we only allow workers who have completed at least 1000 HITs (so they are experienced with the platform) and who have at least a 99% approval rating on past work (so they have a demonstrated track record).

To further ensure valid data, we require that each of our HITs be completed by multiple workers.

We anticipate somewhere around 14,000 editors for the 600 mathematical sciences journals in our study. Because the data set is large and because we are performing multiple HITs to validate our data, it is a pricy endeavor, which is why we need your generous support.