About This Project
Scientific grant review consists of experts assessing the approach and impact of a given proposal. The process is thus an implicit exercise in forecasting – reviewers are betting on which projects are likely to accomplish their goals and produce valuable outcomes.
Explicit and grounded forecasts could produce better funding decisions, but this approach has yet to be widely tested. We aim to launch a pilot forecasting challenge to build understanding of forecasting in scientific review.
Ask the ScientistsJoin The Discussion
What is the context of this research?
Merit review of scientific proposals is ubiquitous across the federal R&D enterprise. But the review process is arguably too conservative, focusing on feasibility over potential impact and limiting the uptake of new research models. Focused research organizations (FROs) – standalone, time-bound, non-profit organizations designed to tackle projects not incentivized by academia or industry – are a relatively untested model with high potential value. FROs aim to produce public goods: processes, tools, and datasets that unlock new methods and accelerate the pace of research.
Federal funders are tentatively curious about this model, but are often wary of (a) the likelihood of proposals achieving their goals and (b) the value of funding one large project rather than several small projects.
What is the significance of this project?
Forecasting can be a valuable tool for assessing the risk-reward profile of novel research models. Because of their bounded time span and technical milestones, FROs are particularly well-suited to forecasting.
The intersection of FROs and forecasting has the potential to yield more than the sum of its parts. By piloting this approach to review, we can (a) generate quantitative predictions of the feasibility and potential impact of real proposals, which funders can use to inform decision-making; (b) provide actionable insight to proposal authors regarding their approach; and (c) obtain metascientific understanding of how forecasting does – or does not – work in a scientific review context.
Follow-up research could entail a formal study comparing forecasting to traditional peer review.
What are the goals of the project?
FAS and Metaculus will run a pilot forecasting tournament on five high-quality FRO proposals sourced by partners at Convergent Research. The forecasting tournament will ask participants to make quantitative forecasts in response to three questions on the probability of success, the social impact, and the scientific impact, of achieving the technical milestones in each proposal. We will recruit and compensate subject matter experts and professional forecasters to participate in the tournament, which will be hosted on Metaculus' platform. After the tournament, we will use the forecasts to calculate the expected value of achieving each technical milestone and categorize the proposals based on their risk-reward profile. The results will be published in a public report.
This budget will enable us to provide financial compensation to subject matter experts for participating in the forecasting tournament. Since we would be asking subject matter experts to dedicate time and effort to evaluating a FRO proposal and making forecasts, we expect that it will be difficult to recruit participants without compensation. Our partner organization Metaculus will match this amount with $5,000 to fund professional forecasters to participate in the forecasting tournament.
We plan to work on the design of the forecasting tournament and recruit participants throughout the summer of 2023. We will launch the forecasting tournament in mid-September 2023 and close the tournament in mid-December 2023. A report detailing the results and learnings from this pilot will be published in January 2024.
Sep 28, 2023
Launch the online forecasting tournament on Metaculus
Sep 28, 2023
Dec 31, 2023
Close the forecasting tournament
Jan 15, 2024
Extract and analyze forecasting data from the tournament
Jan 31, 2024
Produce public-facing and funder-facing reports describing the results and implications of forecasts
Meet the Team
Alice Wu works on metascience and climate policy at the Federation of American Scientists. Prior, she conducted graduate research on solar cell technologies at the Massachusetts Institute of Technology, where she received her Master's in Electrical Engineering.
Jordan Dworkin is the metascience program lead at the Federation of American Scientists, where he aims to improve scientific funding, institutions, and incentive structures through experimentation. Before joining FAS, he was an assistant professor of clinical biostatistics at Columbia University and the New York State Psychiatric Institute, where his research touched on statistical methodology, medical imaging, and mental health.
Peter Mühlbacher is a research scientist at Metaculus, where he works on aggregation algorithms and forecasting analyses, among other things. Prior, he spent some time in Vienna and Cambridge before obtaining his PhD on probabilistic methods for quantum spin systems from Warwick.
- $5,605Total Donations
- $1,868.33Average Donation