About This Project
Survey Methodology has shown when sending out a survey, the title of the survey alters not only response rate but what type of people take the survey. This project examines how online study titles on online platforms can alter treatment effects. We replicate online experiments with known treatment heterogeneity using titles designed to attract participants showing strong, weak, or neutral treatment effects. We aim to show online study titles can alter replicability.
Ask the ScientistsJoin The Discussion
What is the context of this research?
Survey titles influence participants and demographics. With the rise of online studies on platforms like mTurk or Prolific, where study titles are rarely reported, this project investigates the unexplored impact of titles on treatment effects and replicability. Recognizing treatment heterogeneity in experiments, we aim to show how titles can significantly affect both participant response and study replicability through treatment heterogeneity. For treatment effects that are moderated by participant characteristics (e.g. sex, political orientation, personality), launching the same study with different titles could draw different participants, altering participation and replicability.
What is the significance of this project?
One possible reason for unsuccessful replication could be due to characteristics of the participants. As titles of online experiments may draw people with different characteristics, the size of some treatment effects may be stronger or weaker depending simply on the title of the study.
Knowing this will change the way studies are reported in scientific publications. It may become a requirement to publish the title of the study when reporting how a study was run, so future researchers can ensure a similar recruitment strategy.
What are the goals of the project?
We will run 4 studies that have previously been shown to have treatment heterogeneity and were run using best practices. Each study will be run with three different titles, one aiming to recruit participants who show a high treatment effect, one aimed to recruit those with a small treatment effect, and an inert title.
So, for example, If an experiment has been shown to have higher treatment effects for do not believe in global warming, we will run the study under three titles that would be similar to: 1) A study on Is Global Warming a Hoax? (high treatment effect); 2) A Study on the Need for Action to Stop Climate Change (low treatment effect); 3) A study on the Weather Near You (inert title).
We will pursue a Registered Report to ensure publication regardless of outcome.
Budget items are entirely for conducting data collection (and fees). We will replicate four studies with a known treatment heterogeneity (e.g. treatment is stronger for people with certain views or from certain political parties or certain personality characteristics).
Each of the four studies have treatment heterogeneity that have been replicated in recent preregistered replications. We use the same sample size as the preregistered replications to run the overall studies. All of our studies are preregistered prior to data collection as well.
This study pursues a Registered Report, which involves submitting the Introduction & Methods before data is collected. If accepted, the study will be published regardless of outcome. The first step in this project is to write the Stage 1 Registered Report (3 months), respond to reviews (2 months), submit IRB documents (1 week), program the four studies (8 weeks), collect data (1 week each title for each study, so 12 weeks in total), write the Stage 2 results and discussion (2 months).
Oct 24, 2023
Mar 04, 2024
Write Stage 1 Registered Report & Submit
May 20, 2024
Respond to Reviewers
Jun 17, 2024
Submit IRB documents
Aug 12, 2024
Program 4 studies
Meet the Team
I am an Experimental Psychologist who studies Metascience, Prejudice and Social Cognition, and Cognitive Development.
I am a student at Central Connecticut state university, majoring in psychology. I am a research assistant in Dr. Protzko's research lab.
Nothing posted yet.
- $10Total Donations
- $10.00Average Donation