How to choose an AI tool for a systematic review or meta-analysis: twelve questions

Lists of 'the best AI tools for systematic reviews' are easy to find, and many are written by a vendor that ranks its own product first. A better way to choose is to ask the same questions of every tool, including ours.

The twelve questions below follow the 2025 position statement of Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence. It asks synthesists to be critical of a tool's evaluations, and it asks developers to publish how a tool works, how it was tested and where it is weak.

The twelve questions

Step: which step of the review does it cover, and is the output a ranking, a suggestion or a decision?
Judgements: does it make or suggest judgements about eligibility, risk of bias, extraction or certainty? If so, you must validate it and report it.
Evidence: has it been tested on systematic reviews like yours, in your field and with your kind of records?
Independence: were the tests done by people who do not sell the tool, and are they published in full?
Misses: does it report recall, the share of relevant studies it missed, and not only overall accuracy?
Sources: can you see where each claim comes from, for example through a link to the study, so that you can check it?
Versions: is the version stated, is there a change log, and can you get the same result again?
Control: can you override it, set thresholds and stopping rules, and see what it excluded?
Export: can you export decisions, settings and logs in a form you can report and archive?
Data: where do your texts and queries go, is any of it used for training, and do the terms allow you to upload licensed full texts or unpublished work?
Cost and continuity: what does it cost per review, and what happens to your project if the tool changes its price or closes?
Interests: who builds it, who funds it, and do you or your co-authors have any interest in it? The position statement asks authors to report such interests.

Red flags

A promise of a complete systematic review in minutes.
Accuracy given as a single figure, with no description of the data it was measured on.
No way to see the source of a claim, or references that you cannot open.
No export of decisions and logs.
Terms that do not say what happens to the text you upload.

Let SynthGap run the first pass

Enter an area. SynthGap proposes candidate topics, checks each against existing systematic reviews, and lists the primary studies behind it. Start on the Free plan: no card needed, and every run shows every topic it finds.

A general chatbot instead of a specialised tool

You can use a general chatbot for drafts, but it adds duties: you validate it for each use, record the version and the prompts, and check every output. A specialised tool with published validation can be easier to justify, although the responsibility stays with you either way. Can ChatGPT do a meta-analysis? goes through the failure modes.

SynthGap and the twelve questions

SynthGap is built for the first stage only: choosing and testing a topic. It makes suggestions rather than decisions, and it shows the studies behind each topic so that you can open them. Its suggestions are AI-generated and need verifying, as its AI output and research disclaimer says. Please apply the twelve questions to it as you would to any other tool.

Sources

The figures and rules on this page come from these sources.

Flemyng E, Noel-Storr A, Macura B, et al. Position statement on artificial intelligence (AI) use in evidence synthesis across Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence 2025. Campbell Systematic Reviews 2025;21:e70074.

Questions

Should I trust a tool's own accuracy figures?

Treat them as a starting point. Ask what data they were measured on, whether the test was independent and whether recall is reported. The 2025 position statement says that independent evaluations should be considered whenever they are available.

Is a free tool as safe as a paid one?

Price says nothing about validation. Ask the same twelve questions of either, and read the terms on data use.

Can I use a general chatbot instead of a specialised tool?

You can, but you take on the validation, the version record and the checking yourself. For judgement tasks such as screening and extraction, a tool with published evaluations is easier to justify.

Where do I record which tool I chose and why?

In the protocol, as part of the methods for the step it covers, and later in the methods section of the review. The guide to reporting AI use lists what to include.

Keep reading

General methodological guidance, not medical advice. SynthGap's suggestions are AI-generated from the published literature, so verify them before registering a protocol.