AI data extraction for meta-analysis: what it gets right and the errors to catch

A meta-analysis lives or dies on its numbers. For each study you need the group sizes and the result in each arm, in a form that can be combined, for the right outcome and time point. Extracting them by hand is slow and error-prone, which is why AI extraction is attractive.

It can work. A 2024 proof-of-concept study asked a large language model for 160 data elements from study reports and found about 96% agreement with human extraction, the most common error being a missed item. Other evaluations report lower and more variable accuracy, and the errors that matter most in a meta-analysis are the ones that look right.

What you need from each study

For a continuous outcome: the number of participants, the mean and the standard deviation in each group, or a mean difference with its confidence interval. For a binary outcome: events and totals in each group, or an odds ratio or risk ratio with its confidence interval. Add the definition of the outcome, the time point, the analysis population and the study design.

Errors to look for

SD or SE: a standard error read as a standard deviation, or the other way round, makes a study look far more or less precise than it is, and it can dominate the pooled result.
Wrong arm, time point or outcome: papers report several of each, and a model can take a value from the neighbouring row.
Counts and percentages: a percentage taken as a count, or a count taken from the wrong denominator.
Which analysis: intention-to-treat, per-protocol, adjusted and unadjusted estimates mixed within one meta-analysis.
Units and direction: mg/dL against mmol/L, or a scale where higher means worse in one trial and better in another.
The same trial twice: several reports of one trial extracted as separate studies, or a shared control group counted in two comparisons.
Data the model never saw: results in a supplement, in a figure, or in a table saved as an image inside the PDF.

Let SynthGap run the first pass

Enter an area. SynthGap proposes candidate topics, checks each against existing systematic reviews, and lists the primary studies behind it. Start on the Free plan: no card needed, and every run shows every topic it finds.

A way to check every number

Design the form first: decide the fields, the definitions and the unit of each before any tool runs.
Let the tool fill the form, with the page or table where it found each value.
Have a person check every value that enters a pooled analysis against the paper. Sample the descriptive fields if you must.
Where the tool and the person disagree, go back to the paper and record who was right. That also measures the tool on your topic.
Keep the log: the tool, version, date, prompt, and the corrections that were made.

Where it struggles

Cochrane's standards for intervention reviews ask for outcome data to be extracted by two people working independently. One person checking a tool's work is not the same thing, so say in your methods which of the two you did.

Scanned or badly formatted PDFs, where text and tables are read wrongly.
Complex designs, such as crossover, cluster, multi-arm and repeated-measures trials, where the unit of analysis needs thought.
Results given only in figures, where any value is an estimate read from a plot.
Outcomes reported in several ways within one paper.

Uploading papers

Check the publisher's terms before you upload full texts to a third-party service, and the service's own terms before you upload unpublished or confidential material. The RAISE recommendations list copyright, licensing and confidentiality among the legal points to check.

Reporting

PRISMA 2020, item 9, asks how many reviewers collected data from each report, whether they worked independently and, if applicable, details of the automation tools used. Name the tool and its version and describe your checks. See how to report AI use in a systematic review.

Sources

The figures and rules on this page come from these sources.

Gartlehner G, Kahwati L, Hilscher R, et al. Data extraction for evidence synthesis using a large language model: a proof-of-concept study. Research Synthesis Methods 2024;15:576-589.
Konet A, Thomas I, Gartlehner G, et al. Performance of two large language models for data extraction in evidence synthesis. Research Synthesis Methods 2024;15:818-824.
Flemyng E, Noel-Storr A, Macura B, et al. Position statement on artificial intelligence (AI) use in evidence synthesis across Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence 2025. Campbell Systematic Reviews 2025;21:e70074.
Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71.
Cochrane. Methodological Expectations of Cochrane Intervention Reviews (MECIR), standards C45 and C46 (extracting data).

Questions

Is AI data extraction accurate enough for a meta-analysis?

Sometimes. One 2024 proof-of-concept study reported about 96% agreement on 160 data elements, but other evaluations found lower accuracy, and a single wrong number can move a pooled result. It is accurate enough only if you check every number that enters the analysis.

Should the AI extract first, or check a person's extraction?

Either order can work. What matters is that a person has compared every number that reaches the pooled analysis with the paper, and that you report which approach you used.

Can AI extract data from figures?

Some tools can estimate values from a plot, but the values are approximate. Say so in the methods, and consider a sensitivity analysis that leaves those values out.

Do I need to report the use of AI for extraction?

Yes. Extraction is one of the judgements that the 2025 position statement says to declare, and PRISMA 2020 item 9 asks for details of automation tools used to collect data.

Keep reading

General methodological guidance, not medical advice. SynthGap's suggestions are AI-generated from the published literature, so verify them before registering a protocol.