AI screening for systematic reviews: how it works and how to check it

Screening titles and abstracts is the biggest block of routine work in a systematic review. In a typical search only a few records in every hundred are finally included, and a person has to read the rest to find out which. AI screening tries to cut that reading without losing relevant studies.

There are three kinds, with different risks. This guide explains each, how to test one on your own records before you rely on it, why recall is the number that matters, and what to write in your methods.

Three kinds of AI screening

Priority screening (active learning): you screen, the tool learns from your decisions and moves likely includes to the front, and you stop when relevant records have dried up. ASReview is an open-source example, and several screening platforms offer something similar.
Classifiers: a model trained to recognise a kind of record, for example randomised trials, used to filter a set of records before screening.
Language models: you give a model your eligibility criteria and each record, and it answers include or exclude with a reason. It is flexible and quick to set up, but its accuracy varies with the model, the prompt and the topic.

Recall is the number that matters

Screening can go wrong in two ways. Including too much costs reading time later. Excluding a relevant study cannot be undone, because nobody reads that record again. So the figure to measure is recall: of the studies that should be included, what share did the process keep?

Evaluations of ranking tools often report the work saved when 95% of the relevant records have been found, which is a useful benchmark, although you set your own target in the protocol. For language models, published sensitivities range from under half of the relevant records to nearly all, so a model that worked on someone else's review may not work on yours.

Let SynthGap run the first pass

Enter an area. SynthGap proposes candidate topics, checks each against existing systematic reviews, and lists the primary studies behind it. Start on the Free plan: no card needed, and every run shows every topic it finds.

Test it on your own records

Take a random sample of your search results, a few hundred records.
Screen the sample by hand, ideally with two people, and record the includes.
Run the tool on the same sample. For a classifier or a language model, count how many of those includes it kept. For a ranking tool, count how many records you would have had to read to reach your target share of the includes.
Look at the ones it missed. If they share a pattern, such as one type of outcome or study design, change the criteria or do not use the tool.
Write down the result, and the threshold you accepted, before you start the real screening.

Rules to fix in advance

Stopping rule: for priority screening, decide when to stop, for example after a long run of consecutive irrelevant records, and check a random sample of what was left unscreened.
Role of the tool: whether it only orders the records or also excludes some, and if it excludes, who checks what it excluded.
Disagreements: how a conflict between a person and the tool is settled, and by whom.
Updates: whether the same settings and version will be used if you run the search again later.

AI as the second screener

Many reviews, particularly rapid ones, have only one person screening. The 2025 joint position statement of Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence cites an estimated 13% risk that a single screener wrongly excludes a relevant study, and says that using AI as a second reviewer could help to reduce it. That is acceptable only with a justification, a person overseeing the process and transparent reporting. Cochrane's standards for its own intervention reviews still require two people working independently when deciding eligibility.

What to report

PRISMA 2020, item 8, asks you to say how many reviewers screened each record, whether they worked independently and, if applicable, details of the automation tools used. Name the tool and its version, the date, the settings or the prompt, the stopping rule and the result of your test. How to report AI use in a systematic review has the full list.

Sources

The figures and rules on this page come from these sources.

Flemyng E, Noel-Storr A, Macura B, et al. Position statement on artificial intelligence (AI) use in evidence synthesis across Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence 2025. Campbell Systematic Reviews 2025;21:e70074.
Gartlehner G, Affengruber L, Titscher V, et al. Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial. Journal of Clinical Epidemiology 2020;121:20-28.
Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71.
van de Schoot R, de Bruin J, Schram R, et al. An open source machine learning framework for efficient and transparent systematic reviews. Nature Machine Intelligence 2021;3:125-133.
Lieberum JL, Toews M, Metzendorf MI, et al. Large language models for conducting systematic reviews: on the rise, but not yet ready for use - a scoping review. Journal of Clinical Epidemiology 2025;181:111746.
Cochrane. Methodological Expectations of Cochrane Intervention Reviews (MECIR), standards C39 (selecting studies) and C46 (outcome data).

Questions

How accurate is AI screening?

It depends on the tool, the topic and the setting. Ranking tools can bring most relevant records to the front, and published evaluations of language models range from missing about half of the relevant records to finding nearly all. Test on your own records before relying on any figure.

Can I use AI instead of two human screeners?

For a Cochrane intervention review, no: two people working independently is the standard for eligibility decisions. In other reviews, particularly rapid ones, AI as a second reviewer is being proposed, but it must be justified, overseen and reported, and your journal may have its own rules.

How many records do I need before AI screening is worth it?

As a rule of thumb, the larger the set, the more it saves. A few hundred records are usually quicker to screen by hand than to validate a tool on. With several thousand records the saving can be large.

Does AI screening work for full texts?

Some tools and evaluations cover full-text screening, but the evidence is thinner than for titles and abstracts, and the decisions are harder because eligibility often turns on details in the methods. Keep a person responsible for full-text decisions.

Keep reading

General methodological guidance, not medical advice. SynthGap's suggestions are AI-generated from the published literature, so verify them before registering a protocol.