Why science has a credibility problem — and how to address it
Summary
Brian Nosek, director of the Center for Open Science (COS) and co‑founder of Project Implicit, discusses findings from the SCORE project that probe reproducibility and replicability in the social sciences. The study attempted to reproduce results from original datasets and to replicate findings with new data. Reproduction succeeded for about 53% of tested papers; replication succeeded for about 49% by the primary criterion. Nosek argues the problem is not necessarily deliberate wrongdoing but structural: publication incentives favour novel, positive outcomes over transparent, rigorous practice. He outlines tools and reforms that work (OSF, Registered Reports) and pilots to shift evaluation earlier in the research lifecycle.
Key Points
- SCORE tested 145 papers for reproducibility (same data, same analysis) and 164 for replication (new data): ~53% reproduced, ~49% replicated by the main statistical criterion.
- Many reproduction failures stemmed from insufficient reporting and lack of shared code, not necessarily incorrect results.
- Current academic incentives reward publication outcomes over transparency, which undermines reproducibility and trust.
- Effective reforms include Registered Reports, the Open Science Framework (OSF) and transparency guidelines that make methods and code available earlier.
- COS is piloting a ‘Lifecycle Journal’ approach to integrate evaluation alongside research rather than at the end.
- Emerging tools and services (scite.ai, ERROR, AI checks) can provide credibility indicators, but no single metric will suffice.
Content summary
The interview traces Nosek’s path from Project Implicit to leading COS and describes SCORE, a large meta‑research effort to assess how often social‑science findings stand up to scrutiny. The project shows roughly half of tested studies either cannot be exactly reproduced from shared materials or fail to replicate with fresh data. Nosek emphasises that better sharing of code and clearer methods would dramatically improve reproduction rates. He highlights Registered Reports as a practical success because they change what earns publication — focusing on questions and rigour rather than flashy results. COS aims to update infrastructure (OSF) to provide more structure and to run pilots that evaluate research continuously through its lifecycle. Finally, Nosek notes complementary efforts — payment schemes to find errors, citation‑based support/contradiction metrics and AI‑driven checks — as part of a broader ecosystem for trustworthiness.
Context and relevance
This piece matters if you rely on social‑science evidence, design research or set policy informed by academic work. It situates SCORE within a growing wave of meta‑research showing uncertainty is more common than readers expect and points to concrete, tested reforms that institutions and journals can adopt. The article links to broader debates about reproducibility across disciplines, publishing incentives and the role of open infrastructure and new indicators in restoring public and professional trust.
Why should I read this?
Short version: it explains why half the studies you might trust could be shakier than they look — and gives practical fixes that actually work. If you read or use research, this saves you time by highlighting what to look for (sample size, shared code, Registered Reports) and what reforms are likely to stick.
Author style
Punchy: the interview cuts to the chase — this isn’t alarmism, it’s a clear call to rewire incentives and practice. If you work in research, publishing or policy, the examples and proposed fixes here make it essential reading.