Ten meta-analysis mistakes, and how to avoid them

Most flawed meta-analyses are not flawed in the arithmetic. They are flawed in what went into it: a question too loose to pool, a search that missed studies, a number read wrongly, a trial counted twice. The statistics then give a precise answer to the wrong question.

These ten mistakes are the ones to design out at the planning stage, because each is cheaper to prevent than to find later.

Ten mistakes

A question too broad to pool: studies of different patients, treatments or outcomes are averaged into a number that describes none of them. Write a narrow PICO before you search, starting from the PICO guide.
No protocol, or changing it after seeing the data: switching outcomes or analyses once the results are visible can make a result look better than it is. Register a protocol and record every change.
One person doing every step: single-reviewer abstract screening carries an estimated 13% risk of missing a relevant study, and one extractor's slip goes unnoticed. Have a second reviewer for the key steps.
A thin search: one database, English only, no registers and no reference-list checking. Studies with significant results are easier to find, so a thin search leans towards positive results. Use several sources, have an information specialist review the strategy, and follow citations.
Counting the same participants twice: several reports of one trial treated as separate studies, or a shared control group used in two comparisons. Link reports to trials, and split or combine shared arms properly.
Misreading the numbers: a standard error taken as a standard deviation, the wrong arm or time point, or a scale where higher is worse treated as one where higher is better. Check every number that enters the analysis against the paper, whether you or a tool extracted it.
Choosing the model after seeing the result: switching between fixed-effect and random-effects, or leaning on I-squared alone, to get the answer you wanted. Decide in the protocol, and report a prediction interval with a random-effects model.
Ignoring risk of bias: pooling weak studies with strong ones gives a precise estimate of a biased answer. Assess every study with a suitable tool, and test whether the result changes without the high-risk ones.
Funnel plot tests on too few studies: tests for funnel plot asymmetry have too little power with fewer than 10 studies, so the Cochrane Handbook advises against using them. Say that you could not assess small-study effects reliably, rather than report a test that cannot tell.
Claiming more than the evidence allows: reading an association in observational studies as a cause, or ignoring the certainty of the evidence. Grade the certainty with GRADE and write the conclusions at that strength.

Where AI makes these mistakes easier

Misreading the numbers: an extraction tool can read a figure confidently from the wrong row. See AI data extraction for meta-analysis.
A thin search: a polished AI-written search string can still miss studies. See AI and the search strategy.
One person doing every step: AI as the only other pair of eyes is not independent unless it is validated and reported.
Claiming more than the evidence allows: fluent text reads as certain. Check each claim against the data.

Let SynthGap run the first pass

Enter an area. SynthGap proposes candidate topics, checks each against existing systematic reviews, and lists the primary studies behind it. Start on the Free plan: no card needed, and every run shows every topic it finds.

How to use this list

Go through it when you write the protocol and again before you write up. Each item corresponds to something that a reviewer or an editor can check in minutes, and the protocol checklist in meta-analysis protocol checklist puts the fixes into your plan.

Sources

The figures and rules on this page come from these sources.

Gartlehner G, Affengruber L, Titscher V, et al. Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial. Journal of Clinical Epidemiology 2020;121:20-28.
Sterne JAC, Sutton AJ, Ioannidis JPA, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ 2011;343:d4002.
Cochrane Handbook for Systematic Reviews of Interventions, chapter 13: assessing risk of bias due to missing evidence in a meta-analysis.

Questions

Which meta-analysis mistake is the worst?

No list ranks them reliably, but the ones at the start, a loose question, no protocol and a thin search, are the hardest to repair afterwards, because every later step builds on them.

How do I avoid counting a trial twice?

Link every report to its trial by registration number, authors, sites, dates and sample sizes. Extract from the most complete report and use the others only for details. For a multi-arm trial with a shared control group, split the control group or combine the arms, as the Cochrane Handbook describes.

Can I trust I-squared?

Use it as a rough guide, not a rule. It depends on the size of the studies, and a low value does not mean the studies agree. Look at the forest plot, at tau-squared and, with a random-effects model, at the prediction interval.

Is a meta-analysis of two studies a mistake?

Not by itself, since it can technically be done. The question is whether the two studies are similar enough and the result useful. The guide on how many studies you need gives the thresholds.

Keep reading

General methodological guidance, not medical advice. SynthGap's suggestions are AI-generated from the published literature, so verify them before registering a protocol.