An Australian education scholar has proposed a rapid assessment tool to flag weak or misleading research used in policy debates, saying current mechanisms to rebut dubious claims are too slow to shape public discussion.
Index aims to speed up scrutiny of policy evidence
Professor Pat Thomson of the University of Adelaide set out a five‑tier “policy bullshit index” in the Journal of Education Policy, according to Times Higher Education. The scale runs from level 0 (no bullshit) to level 4 (extreme bullshit) and is intended to provide a quick, practical test of whether think‑tank reports, advocacy briefs and government proposals actually meet basic standards of evidence.
Thomson told Times Higher Education that many policy documents adopt the trappings of academic research while failing to satisfy core evidentiary standards. She argued that traditional rebuttal routes — formal peer review and meticulous fact‑checking — often take too long to influence a debate in an era of rapid news cycles and soundbite politics.
“The incentive structure of the policy ecosystem rewards bullshit,” Thomson writes.
The proposed index is not meant to replace rigorous review. Instead, Thomson said it asks a set of straightforward questions that a knowledgeable reader could apply in a day, rather than the weeks required for full fact‑checking. The method shares features with fact‑checking and peer review, but emphasises speed and accessibility so that critical scholars and informed journalists can more readily call out weak claims.
- Five levels: level 0 (no problems) to level 4 (extreme problems)
- Designed to be applied rapidly — about 24 hours rather than several weeks
- Based on a 19‑question checklist that probes whether evidence supports claims
Implications for South African education debate
The proposal has clear resonance for South Africa, where policy discussions on schooling, curriculum change, matric outcomes and funding frequently rely on reports from think‑tanks, consultancy studies and commissioned reviews. Rapid, transparent checks of the quality of evidence could help teachers, parents and policymakers separate robust recommendations from those that merely look convincing.
Carefully applied, such an index could support several practical outcomes in the South African context:
- Stronger media coverage: journalists could use a short checklist to qualify claims made in policy documents before they reach large audiences.
- Better legislative scrutiny: parliamentary portfolio committees and provincial education oversight bodies could adopt elements of the index when evaluating proposals.
- Teacher and school leadership capacity: professional bodies might train educators to recognise when a report’s recommendations outstrip its evidence.
| Index level | Interpretation (summary) |
|---|---|
| 0 | No evident mismatch between claims and evidence |
| 1–2 | Some weaknesses or gaps that warrant caution |
| 3–4 | Significant or extreme divergence between claims and supporting evidence |
Thomson acknowledged that most research has limitations and that transparent hedging is part of good scholarship. Her frustration, as reported, stems from the way hedged academic findings are sometimes repackaged into assertive policy prescriptions without appropriate caveats. The index is offered as a tool to make that repackaging easier to detect.
For South African education stakeholders, the challenge will be translating a methodological idea into everyday practice without eroding trust in legitimate research. Rapid checks can help, but they must be paired with capacity building so teachers, officials and journalists can apply the questions reliably and consistently.
Times Higher Education reported the proposal; the Journal of Education Policy published Thomson’s full note.
