Is it just me? On the generalisability of effects in psychology

Raised of $5,000 Goal
Ended on 9/30/23
Campaign Ended
  • $0
  • 0%
  • Finished
    on 9/30/23

About This Project

In much of psychological research, expected effects are not shown by everybody, but just by a subset of participants. We aim to identify whether this subset of participants consists of the same people across theoretically-unrelated psychological effects (i.e., whether psychology effects are driven by a small portion of the population). If so, this may have profound implications for how we understand research findings in psychology and beyond.

Ask the Scientists

Join The Discussion

What is the context of this research?

In psychology, there are many different types of measures and procedures. Psychologists assume that effects seen in these procedures apply to at least a majority of the population. In recent research, Jamie and his colleagues have found that this was not the case in one such procedure. When asked to report on attitudes to different topics (e.g., politics), they found that about 15% of people tended to show effects in general, regardless of the attitude topic being assessed; the rest of the people involved never showed effects. Similar subset effects have been documented for several other procedures too. We want to test whether the people who drive effects are the same people across different, unrelated tasks.

What is the significance of this project?

Although many psychologists are aware that a lot of their measures show these subset effects, we often just ignore them or consider them part of random variation. If we demonstrate that effects in various measures are driven by the same people, then we will effectively demonstrate that much of psychology has only been studying a subset of the population. This would open up a whole new range of questions. Why do only these people show an effect? How can we also study the portion of participants who don't? Are some measures more susceptible to this problem than others? Because this issue is potentially applicable to many different fields of psychology, and because it may threaten the validity of our measures, this project will be of great significance to the discipline.

What are the goals of the project?

The proposed experiment is the first step in a bigger research agenda we plan. We will get participants to complete popular measures from various areas of psychology (e.g., an evaluative conditioning paradigm [learning research], an aggression priming paradigm [media effects research], an implicit measure of attitudes [social psychology], a mood induction task [emotion research]). These will be measures which (i) are widely used in their field, (ii) are used regularly in online research, (iii) do not take much time to complete, and (iv) whose results are assumed to apply to most of the population being studied. We will then examine (i) whether scores in these unrelated tasks are correlated, and (ii) whether the participants who respond most strongly are consistent across tasks.


Please wait...

Participation will take an hour to complete. We require 375 participants. At a rate of $12 per hour, this amounts to $4500 in total. We add $500 to account for's processing fees.

Endorsed by

This is an extremely interesting project on the cutting edge of psychological meta-research. Their elegant and clever approach allows Jamie and Malte to ask a fascinating question about the landscape of psychological responses by simply going one step beyond the typical level of analysis. The answer to this question could have far-reaching implications for the discipline. Jamie's and Malte's profound expertise in psychological measurement and meta-science is perfectly suited for this research line and I'm very excited to see what they'll find.
I am thrilled about this project because it elevates the discussions regarding the generalizability of psychology (a topic much debated recently) to the next level. The proposed design is elegant and holds the promise of elucidating a phenomenon that has thus far confounded researchers, with potential implications for nearly every field within psychology. Malte and Jamie are both outstanding experimentalists -- if anybody can answer vexing questions about who responds to psychological manipulations, it's the two of them.

Project Timeline

Assuming funding is acquired by October 1st 2023, we will aim to finalise the programming of our candidate measures (which we will be thinking about in the meantime anyway) by the beginning of October. We will finalise the preregistration and analytic approach by the end of that week. Data collection can begin the week after; since collection will be done online, we can quickly move to analysing the data and then writing the paper (which should take about 3 weeks).

Aug 31, 2023

Project Launched

Oct 02, 2023

Finalise measures for use

Oct 06, 2023

Finalise analytic strategy, analytic code, and preregistration

Oct 16, 2023

Collect data

Oct 17, 2023

Run analysis

Meet the Team

Jamie Cummins
Jamie Cummins
Postdoctoral Research Scientist


University of Bern
View Profile
Malte Elson
Malte Elson
Associate Professor


University of Bern
View Profile

Team Bio

Jamie works as a postdoc for Malte, who is professor of the psychology of digitalisation at University of Bern. Jamie has a background in the psychology of learning and social psychology, whereas Malte has a background in media effects research. Both Malte and Jamie have conducted extensive research into measurement properties of psychological measures. Combined, we have both the scope and depth needed to fulfil this project and communicate it to diverse audiences of psychologists.

Jamie Cummins

I am a psychologist and meta-scientist. My research in general is unified by one common theme: an interest in how effective measures in psychology are at telling us about individual persons, rather than groups of people. I do this using conventional and nonconventional measurement approaches.

Malte Elson

I am an associate professor of the Psychology of Digitalisation at the University of Bern. My work focuses on meta-science and behavior research methods, technology use and media effects, and human factors in privacy and IT security. My meta-scientific interests include issues related to standardisation of research, mechanisms of academic peer review, and error detection and correction.

Lab Notes

Nothing posted yet.

Additional Information

We strongly believe in the principles of open and transparent science. Our study and planned analysis will be preregistered, and any deviations from the preregistered protocol will be explicitly stated. All of our experimental materials, processing and analysis code, and data will be made freely and openly available via the Open Science Framework.

Project Backers

  • 0Backers
  • 0%Funded
  • $0Total Donations
  • $0Average Donation
Please wait...