Design Study-Step by Step explanation

Under ‘Basic Info’ you'll find:

  • Project Name – This is the name of your project that is not visible to Workers. It will be visible on the dashboard, and will allow you to easily differentiate between multiple HITs with the same name.

Under ‘Describe HIT’ you'll find:

  • Title – This is the name of your HIT as it will appear to Workers on Mechanical Turk.
  • Description – A brief description of the Hit as it will appear to Workers on Mechanical Turk.
  • Keywords – Including keywords will make it easier for Workers to find your HIT.

Under ‘Setup HIT and Payment’ you'll find:

  • Workers payment per survey – How much each participant will be paid per HIT.
  • Estimated time to complete HIT – This estimated time lets potential workers know approximately how long the HIT will take. This estimated time to complete the HIT will be appended to the Description field, described above. 
  • For example, if the HIT Description is “Short personality survey”, and the Estimated Time to Complete HIT is “7 minutes”, then the Description will appear on Mechanical Turk as “Short personality survey, (~ 7 minutes)”
  • Total Number of Survey Participants – The total number of participants that you want to take your HIT.
  • Time allotted per assignment – This is the maximum amount of time that a worker can spend on a HIT. If a worker does not submit a HIT within the time allotted per assignment they cannot complete the HIT and cannot get paid. 
  • Note that participants differ as to how fast they work. You should thus provide workers with ample time to complete their HIT. However do not make the time allotted per assignment overly long, since this will deter workers from taking your HIT, as they will assume that the HIT will take a long time.
  • HIT expired in – The HIT will expire and will no longer be visible to workers after this time.
  • Survey Hyperlink – Copy and paste the link to your study in this field. Click on Preview Link in New Tab button to view your study.
  • Automatic Worker Bonus – Amount you will bonus each worker who is approved 
  • There will then be 3 checkboxes on the bottom of the page: 
  • ‘The HIT will only be visible to workers who qualify. This must be checked to accurately measure the Bounce rate’ – See question below for more details.
  • ‘Display the Median Hourly Rate (including guaranteed bonus) to Workers’ – Check this option if you want to show Workers before they accept your HIT, the median hourly rate of pay.
  • ‘A few demographic questions will be added to your HIT and worker gender consistency score is added to the CSV download’ – See question below for more details.
Under ‘How Workers are Approved’ you'll find:


  • Secret code – This is an optional feature, which allows you to automatically approve and reject workers based on whether or not they correctly enter the Secret Code. You can choose to have a fixed completion code which is a code that's provided to you, or you can also set a custom completion code that you manually enter. There is also an option to set up a dynamic completion code for Qualtrics studies ONLY. This means that each survey taker will be given a unique secret code to enter. You can either manually or automatically approve HIT assignments. 


(Why do I need a secret code?

There is nothing stopping a Worker from accepting your HIT on MTurk and clicking Submit without ever actually having looked at your HIT. For this reason requesters routinely include quality assurance checks. The Secret Code is one such quality assurance test. At some point in the study, for example at the very end, you can show the Worker your Secret Code (for example HGTR4EFG) and indicate to the Worker that they should copy this code and paste it on MTurk Prior to submitting the HIT. Here is an example text: “The study Secret Code is HGTR4EFG. Please enter this code on Amazon Turk before submitting this HIT.” This will guarantee that, at the very least, the Worker reached the end of your survey and saw the last question.

How does the Secret Code work?

When the worker completes the HIT having entered the correct Secret Code, TurkPrime will automatically approve the HIT. This will save you the time to go through each workers and manually check whether they entered the Secret Code. However, if a worker does not enter the Secret Code correctly, they will be automatically rejected by TurkPrime. Being rejected means that a Worker will not get paid for the HIT, and that the rejection will now be part of that Worker’s permanent MTurk record.

Can I reverse a rejection for a rejected a worker?

Yes. After a study is launched you can check the progress by clicking on Results at the top of the TurkPrime homepage. On the DashBoard page you will be able to see the status of each study. A number of buttons also provide options for further actions that can be taken. To reverse a rejection for a specific Worker click on Reverse Rejections. In the Worker ID field paste the Worker ID of the workers whose rejections you want to reverse. You can include a message that the Worker will receive in their email notifying them that their HIT had been approved, with the rejection reversed.
What if I don’t want to automatically approve and reject workers?
The use of the Secret Code is optional. To deactivate the Secret Code feature simply check off the box that appears below the Secret Code field. You will then be able to manually approve and reject Workers.))

  • Auto pay workers – Time after which all Workers will be automatically approved. This field is relevant only when the Secret Code option is not selected


Under ‘MicroBatch’ you'll find:

  • options to break up your survey into small HITs
  • Set auto restart time
  • Set the number of participants per micro batch 


Under ‘Worker Requirements’ you'll find:


  • Exclude -- The Exclude feature can be extremely useful for requesters who wish to exclude Workers who have participated in previous HITs. Launching a HIT while excluding participants from previous HITs will open your launched HIT to all Mechanical Turk Workers, but will not allow the excluded Workers to take your HIT.
((How do I use the Exclude feature?

When setting up your study in the Design a Linked Survey page, go to Worker Requirements. All of the HITs that you have previously launched will appear in the Exclude Workers Who Completed These Surveys field. Click on a study that you want to exclude. To exclude multiple studies, click on each study while holding the Ctrl key.

Can I exclude only some of the Workers from a study?
  

Yes. Simply paste all of the Worker IDs in the Exclude Workers with These WorkerIds field.))

  • Include -- The Include feature can be extremely useful for requesters who wish to run a HIT with only the Workers who have taken a previous HIT, or multiple previous HITS. Launching a HIT while including participants from previous HITs will open your launched HIT to only the Workers who have taken those HITs in the past, but will not allow any other Workers to take that HIT.
((How do I use the Include feature?

When setting up your study in the Design a Linked Survey page, go to Worker Requirements. All of the HITs that you have previously launched will appear in the Allow Workers Who Completed These Surveys field. Click on a study that you want to include. To include multiple studies, click on each study while holding the Ctrl key. When including multiple studies use the All option to include Workers who have taken all of these studies, and use the Any option to include Workers who have taken any of the studies.

Can I include only some of the Workers from a study?
  

Yes. Simply paste all of the Worker IDs in the Include Workers with These WorkerIds field.
What is the purpose of excluding or including TurkPrime or Amazon WorkerIds and where can I find the the Ids? When looking through your CSV file, it will show the Ids of all the people who participated in the survey. You may want to exclude or include people with specific answering tendencies, people who have taken the survey before, or people who provided insufficient data. ))


  • Worker Level – Mechanical Turk -- provides the option of using “experts”. Selecting Any Level will make your study visible to all Mechanical Turk Workers. Selecting the Master options will only make your HIT visible to Workers that MTurk preselects based on their criteria. Note that MTurk charges more for Master workers.
  • Location – Set or restrict the locations that you want your Survey to be distributed in.
  • HIT approval rate -- Each time a participant completes a HIT the requester has to either approve or reject that HIT. Rejections become part of a Worker’s permanent record. A high rejection rate indicates that Requesters consistently reject a Worker, and may be used as an index of overall work quality. Use the right and left sliders to select the range of approval rates for your Hits.
  • Number of HITs approved – When a Worker completes a HIT and a requester approves that HIT, the approved HIT becomes part of that Worker’s permanent record. MTurk keeps track of the number of approved HITs. Use the right and left sliders to select the range of approved HITs rates for your HIT. The number of approved HIT is an indicator of a Workers experience. Thus you may want to use only Workers who have completed many HITs if you want experienced Workers, or you may want to use only Workers who have a few or no HITs if you are looking for novice Workers. Note – The HIT approve rate does not become operational, until the number of approved HITs is at least 100, for survey HITs on MTurk.

Feedback and Knowledge Base