Do study titles alter treatment effects?

Backed by David Lang
Raised of $5,197 Goal
Ended on 12/08/23
Campaign Ended
  • $10
  • 1%
  • Finished
    on 12/08/23

About This Project

Survey Methodology has shown when sending out a survey, the title of the survey alters not only response rate but what type of people take the survey. This project examines how online study titles on online platforms can alter treatment effects. We replicate online experiments with known treatment heterogeneity using titles designed to attract participants showing strong, weak, or neutral treatment effects. We aim to show online study titles can alter replicability.

Ask the Scientists

Join The Discussion

What is the context of this research?

Survey titles influence participants and demographics. With the rise of online studies on platforms like mTurk or Prolific, where study titles are rarely reported, this project investigates the unexplored impact of titles on treatment effects and replicability. Recognizing treatment heterogeneity in experiments, we aim to show how titles can significantly affect both participant response and study replicability through treatment heterogeneity. For treatment effects that are moderated by participant characteristics (e.g. sex, political orientation, personality), launching the same study with different titles could draw different participants, altering participation and replicability.

What is the significance of this project?

One possible reason for unsuccessful replication could be due to characteristics of the participants. As titles of online experiments may draw people with different characteristics, the size of some treatment effects may be stronger or weaker depending simply on the title of the study.

Knowing this will change the way studies are reported in scientific publications. It may become a requirement to publish the title of the study when reporting how a study was run, so future researchers can ensure a similar recruitment strategy.

What are the goals of the project?

We will run 4 studies that have previously been shown to have treatment heterogeneity and were run using best practices. Each study will be run with three different titles, one aiming to recruit participants who show a high treatment effect, one aimed to recruit those with a small treatment effect, and an inert title.

So, for example, If an experiment has been shown to have higher treatment effects for do not believe in global warming, we will run the study under three titles that would be similar to: 1) A study on Is Global Warming a Hoax? (high treatment effect); 2) A Study on the Need for Action to Stop Climate Change (low treatment effect); 3) A study on the Weather Near You (inert title).

We will pursue a Registered Report to ensure publication regardless of outcome.


Please wait...

Budget items are entirely for conducting data collection (and fees). We will replicate four studies with a known treatment heterogeneity (e.g. treatment is stronger for people with certain views or from certain political parties or certain personality characteristics).
Each of the four studies have treatment heterogeneity that have been replicated in recent preregistered replications. We use the same sample size as the preregistered replications to run the overall studies. All of our studies are preregistered prior to data collection as well.

Endorsed by

Project Timeline

This study pursues a Registered Report, which involves submitting the Introduction & Methods before data is collected. If accepted, the study will be published regardless of outcome. The first step in this project is to write the Stage 1 Registered Report (3 months), respond to reviews (2 months), submit IRB documents (1 week), program the four studies (8 weeks), collect data (1 week each title for each study, so 12 weeks in total), write the Stage 2 results and discussion (2 months).

Oct 24, 2023

Project Launched

Mar 04, 2024

Write Stage 1 Registered Report & Submit

May 20, 2024

Respond to Reviewers

Jun 17, 2024

Submit IRB documents

Aug 12, 2024

Program 4 studies

Meet the Team

John Protzko
John Protzko
Assistant Professor


Central Connecticut State University
View Profile
Samruddhi Marathe
Samruddhi Marathe


Central Connecticut State University
View Profile

Team Bio

John Protzko

I am an Experimental Psychologist who studies Metascience, Prejudice and Social Cognition, and Cognitive Development.

Samruddhi Marathe

I am a student at Central Connecticut state university, majoring in psychology. I am a research assistant in Dr. Protzko's research lab.

Lab Notes

Nothing posted yet.

Additional Information

Project Backers

  • 1Backers
  • 1%Funded
  • $10Total Donations
  • $10.00Average Donation
Please wait...