When the Evidence Disagrees

Learning objectives

What this chapter asks you to be able to do.

• Explain why an evidence body must be organized around a claim or question rather than citation count.
• Distinguish systematic review discipline from automatic validity, and identify how search, screening, inclusion, and exclusion decisions shape a synthesis.
• Evaluate whether findings are comparable in construct, estimand, population, intervention or exposure, comparison, setting, and time horizon before pooling them.
• Explain how shared datasets, samples, instruments, methods, model families, institutions, and source ancestry create evidence dependence.
• Interpret meta-analysis conceptually, including weighted pooling, fixed-effect and random-effects logics, heterogeneity, uncertainty, and sensitivity without treating a pooled estimate as a universal answer.
• Distinguish contradiction from heterogeneity, non-comparability, random variation, measurement disagreement, methodological failure, and unresolved uncertainty.
• Use triangulation and mixed-method synthesis without forcing qualitative, causal, administrative, computational, historical, and institutional evidence onto one numerical scale.
• Explain how publication, outcome-reporting, language, database, platform, and other selection processes shape the visible evidence body.
• Analyze expert disagreement and evidence hierarchies without treating consensus as either infallible or irrelevant.
• Identify aggregation rules, confidence labels, and narrative summaries as forms of procedural power.
• Evaluate AI-assisted evidence synthesis as a governed research role whose outputs require verification, lineage awareness, and human methodological responsibility.
• Produce Decision Dossier XII: an Evidence Synthesis and Disagreement Record that preserves comparability, independence, disagreement, and reopening conditions.

Chapter summary

The argument in inspectable form.

Key terms

Concepts to carry forward.

evidence body

A set of evidence units organized around a specified question or claim, including their relationships, quality, scope, uncertainty, and dependence rather than merely a list of documents.

synthesis question

The precise question an evidence review is intended to answer, including relevant construct, population, setting, time horizon, and intervention or exposure where applicable.

systematic review

A review using explicit, reproducible methods for question definition, searching, selection, appraisal, extraction, and synthesis. Systematic process does not guarantee an unbiased or correct conclusion.

eligibility criteria

Pre-specified rules governing which evidence is included or excluded from a review.

evidence unit

The unit treated as contributing a finding to synthesis: for example a study, estimate, case interpretation, dataset analysis, or other defined evidentiary contribution.

evidence lineage

Relationships showing how findings depend on shared data, samples, instruments, methods, institutions, models, sources, or derivations.

evidence dependence

Correlation or shared vulnerability among apparently separate findings because they rely on common observations, assumptions, methods, institutions, or analytical pipelines.

replication

A new study aimed at the same scientific question using new data; forms differ, and a replication can vary methods or settings.

reanalysis

A new analysis of previously collected data. It may test robustness or alternative specifications but does not automatically provide independent empirical confirmation.

comparability

The degree to which evidence units address the same or sufficiently related construct, estimand, population, exposure, comparison, setting, and time horizon for a proposed synthesis.

meta-analysis

Statistical combination of results from two or more studies when their quantitative findings are suitable for meaningful combination.

fixed-effect logic

A meta-analytic logic in which the included effect estimates are treated as estimating one common underlying effect for purposes of the model.

random-effects logic

A meta-analytic logic in which studies are treated as estimating different but related effects distributed around an average.

heterogeneity

Variation among study findings; it may reflect real differences in effects, populations, implementation, measurement, design, bias, or other sources.

publication bias

Distortion arising when the probability that results become visible in the published literature is related to their direction, magnitude, significance, or other characteristics.

selection bias at evidence-body level

Distortion caused by which studies, outcomes, languages, databases, reports, datasets, or analyses become available to or admitted into a synthesis.

triangulation

Reasoning across evidence generated through different observations, assumptions, methods, or perspectives to examine convergence, divergence, complementarity, and rival explanations.

qualitative synthesis

A family of approaches for integrating or interpreting findings across qualitative studies while preserving context, interpretation, and conceptual relationships.

mixed-method synthesis

Integration of different evidence forms so that each contributes to the part of the public question it is suited to address, without assuming all findings share one scale.

evidence-strength conclusion

A bounded judgment about how strongly an evidence body supports a specified claim, including dependence, quality, applicability, uncertainty, and disagreement.

living review

An evidence synthesis designed to be updated as relevant new evidence becomes available under explicit update methods and triggers.

minority interpretation

A credible dissenting interpretation preserved alongside a synthesis when disagreement remains methodologically or substantively important.

synthesis claim

The bounded proposition produced after evaluating the included evidence body, including what is supported, where it applies, and what remains unresolved.

aggregation fragility

A condition in which reasonable changes to aggregation rules, evidence inclusion, dependence treatment, or analytical assumptions materially change the synthesis result.

Review & discussion

Questions for seminar, revision, or assessment.

1. Why can six papers provide fewer than six independent confirmations? Give two different lineage mechanisms.
2. What is the difference between a systematic review being transparent and being unbiased?
3. Construct two studies on the same political topic that should not be pooled because they estimate different causal quantities.
4. Why is a reanalysis of the same dataset different from an independent replication? What can the reanalysis still teach us?
5. Explain fixed-effect and random-effects meta-analysis in conceptual terms. Why does neither model repair an invalid construct?
6. Give a case in which a pooled average is mathematically useful and a case in which heterogeneity is the more policy-relevant result.
7. How can publication bias, language bias, and platform-access bias each change an evidence body through different mechanisms?
8. What would count as genuinely independent triangulation between qualitative and quantitative evidence? What would only look independent?
9. Why should a qualitative synthesis not simply count how many studies mention a theme?
10. Using Asterbridge, distinguish a contradictory finding from a non-comparable finding.
11. Under what conditions should a minority expert interpretation remain visible even when most experts agree?
12. When can an evidence hierarchy be useful, and what makes it dangerous when generalized across claim types?
13. Identify three ways an aggregation rule can exercise political power without making a final public decision.
14. How could an AI system create false consensus while accurately summarizing every included paper?
15. What validation evidence would you require before allowing an LLM to exclude records during a consequential evidence review?
16. Why is a living review not simply a review that is rewritten whenever new information appears?
17. Does current NousPolis evidence governance merely record dependence, or does it require dependence to affect evidence strength? What remains underspecified?
18. What would a dedicated EvidenceBody or SynthesisClaim object need to preserve that the current core ontology does not force into one record?
19. Which is more dangerous for a public synthesis: a wrong pooled estimate or a polished narrative that collapses different estimands into one conclusion? Defend your answer.
20. Write a bounded Asterbridge conclusion that preserves one strong finding, one heterogeneous finding, one disputed finding, and one non-comparable evidence contribution.
21. What should the council learn from the statement “the evidence disagrees in an informative way”? When would that statement be evasive instead of rigorous?

Further reading

Continue into the literature.