Can ChatGPT do a meta-analysis?

A chatbot can help with parts of a meta-analysis. It cannot be the meta-analyst. It can explain a method, draft analysis code and tidy a manuscript. Asked to find the studies, extract the numbers or produce the pooled result by itself, it can return something that looks finished and is partly or wholly wrong.

The same strengths and weaknesses apply to ChatGPT, Claude, Gemini and other assistants, so this guide treats them together. The working rule is simple: let the model draft, and let validated tools and people decide.

What a chatbot does well

Explaining: it can describe fixed-effect and random-effects models, heterogeneity or a forest plot in plain words, and answer follow-up questions at any hour.
Drafting code: it can write a first version of an R script using the metafor package, or a Stata command, for you to read, run and correct.
Brainstorming: it can suggest synonyms for a search, fields for an extraction form or subgroup ideas, which you then test.
Reading help: given one paper, it can summarise it or point you to the table where an outcome is reported.
Editing: it can improve the language and structure of your own text.

Where it goes wrong

Invented references: in a 2023 test, 55% of the references written by GPT-3.5 and 18% of those written by GPT-4 did not exist. Newer models are likely to do better, but you cannot tell a real reference from a made-up one without checking it.
Wrong numbers: a standard error read as a standard deviation, the wrong arm or time point, a value from a different outcome, a percentage taken as a count.
Arithmetic in prose: unless it runs code that you can inspect, a pooled estimate in a chat reply may not come from any calculation.
Answers that change: the same prompt can give different results on different days, so a method that depends on it cannot be reproduced.
Missing studies: it cannot show you what a search did not return, and it may describe a topic as untouched when a review exists.
Going along with you: a leading question tends to get a confirming answer, so ask it to argue against your topic as well.

Let SynthGap run the first pass

Enter an area. SynthGap proposes candidate topics, checks each against existing systematic reviews, and lists the primary studies behind it. Start on the Free plan: no card needed, and every run shows every topic it finds.

A safe way to use it

Do the thinking first: write the question as a PICO and the protocol yourself, starting from the PICO guide.
Use the chatbot for drafts: search strings, extraction form fields, analysis code, plain-language text.
Check every output against a primary source: open the paper and confirm the number or the reference.
Run the statistics in validated software with a script you keep. See meta-analysis software.
Log the model, its version, the date and the prompts you used.
Declare the use as your journal asks. See how to report AI use in a systematic review.

Prompts that keep you in charge

Ask for drafts and code, not for final answers. Two examples to adapt:

Code: 'Write R code using the metafor package that reads log odds ratios and their standard errors from a CSV file, pools them with a random-effects model and draws a forest plot. Do not calculate the result yourself.'
Check: 'Here is the methods section of a trial. List every number I would need to enter for a mean-difference meta-analysis, and tell me which ones are unclear or missing.'

How SynthGap differs from a chatbot

A chatbot answers from what it remembers, or from whatever a search tool returns to it. SynthGap uses tools to search an index of the published literature and a citation index of systematic reviews, and it shows the studies behind every suggestion so that you can open and check them. It does not run your search, extract your data or pool your results either, and its suggestions are AI-generated, so verify them in the same way.

Sources

The figures and rules on this page come from these sources.

Walters WH, Wilder EI. Fabrication and errors in the bibliographic citations generated by ChatGPT. Scientific Reports 2023;13:14045.
Gartlehner G, Kahwati L, Hilscher R, et al. Data extraction for evidence synthesis using a large language model: a proof-of-concept study. Research Synthesis Methods 2024;15:576-589.
International Committee of Medical Journal Editors. Use of AI by authors. Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals.

Questions

Can ChatGPT run the statistics for a meta-analysis?

It can write the code, and some versions can run code and show you the result. Treat the code as a draft, read it, run it yourself in R or Stata and keep the script. A pooled estimate that arrives only as text in a chat reply cannot be checked.

Can I upload papers and have it extract the data?

It can try, and one 2024 proof-of-concept study found high agreement with human extraction, but accuracy varies and every number still needs checking. Also check the publisher's terms before you upload full texts to a third-party service, and the service's data terms before you upload unpublished work.

Do I have to say I used ChatGPT?

If it made or suggested judgements, such as eligibility, data extraction or risk of bias, you should report it, and the ICMJE recommendations ask authors to disclose the use of AI in the submitted work. Check the policy of your journal.

Can ChatGPT be a co-author?

No. The ICMJE recommendations say that chatbots cannot be listed as authors, because they cannot take responsibility for the accuracy, integrity and originality of the work.

Keep reading

General methodological guidance, not medical advice. SynthGap's suggestions are AI-generated from the published literature, so verify them before registering a protocol.