GUIDE
AI-Assisted Systematic Review Screening: Tools, Workflow, and Validation
Faster screening is the easy part. Whether it holds up to a peer reviewer depends on what you validated, what you disclosed, and what you left as a human decision.
Published October 14, 2026
AI-assisted screening speeds up title/abstract triage on large record sets. It does not, on its own, make that triage defensible - that comes from piloting it, reporting it, and keeping a human decision in the loop. This guide covers the actual workflow, not a feature list.
See the screening workflowTwo separate questions get asked whenever AI screening comes up, and most content answers only the first one: does it save time (usually yes, on a large record set) and will a journal or peer reviewer accept it (depends entirely on what you did to validate and report it). The second question is the one that actually determines whether using AI helps or hurts your review.
What an AI screening suggestion actually is
A well-built AI screening tool reads a study’s title/abstract (or full text) against your eligibility criteria and returns three things: a suggested call (include, exclude, or maybe), a confidence score, and a short written reason. That is advisory context next to the study, not a decision the study has already received. In EvidenceFlow, a study can be scored one at a time or in a batch across every unscored study in a project, and a deeper-reasoning mode trades speed for more careful reasoning on the ambiguous cases - but no study moves to included or excluded until a human reviewer enters that decision separately. Nothing is auto-excluded before a human sees it.
That last point matters for reporting: PRISMA 2020’s flow diagram has a specific line for “records excluded by automation tools”, for tools that remove records before a human screens them. A tool that only suggests, and never excludes on its own, has nothing to report on that line - every record still goes through independent human screening either way.
What has to stay a human decision
Blind, independent dual review - two reviewers deciding without seeing each other’s calls, then resolving disagreements - is about catching where two humans read the eligibility criteria differently. An AI suggestion sitting alongside that process is not a substitute for the second reviewer unless you have specifically validated it as one (see below) and reported that you did. Treat the suggestion the way you’d treat a well-informed research assistant’s first pass: useful to look at first, not a vote that counts on its own.
Piloting and validating before you trust it
Confidence scores describe the model’s certainty about its own answer, not how often that answer is actually right for your review. The only way to know the real agreement rate is to check it directly:
- Pull a sample of studies you’ve already screened by hand - include some clear includes, clear excludes, and some genuinely borderline calls near your eligibility criteria, not only the easy cases.
- Run the AI suggestion against that same sample and compare it to the human decision, study by study.
- Report the agreement rate - and look specifically at disagreements on borderline studies, since that is where a tool is most likely to differ from a careful human reader.
- Decide, from that sample, how much weight the suggestion earns in your actual screening workflow - informing a single reviewer’s first pass is a lower bar than standing in for a second one.
This is the “pilot or calibrate” step the November 2025 joint position statement from Cochrane, the Campbell Collaboration, JBI, and the Collaboration for Environmental Evidence describes - done once per review, before relying on a tool at scale, not a one-time industry certification that applies to every review afterward.
What to put in your methods section
The position statement is specific about what “transparent reporting” of AI use actually means in practice:
| Report this | Why |
|---|---|
| Tool name and version, and the dates it was used | A reader needs to know exactly what ran, since model behavior changes between versions. |
| Which stages of the review it affected | Screening suggestions, extraction auto-fill, and risk-of-bias assistance are different claims with different risks. |
| Why you judged it methodologically sound for this review | Your validation-sample agreement rate from the step above is the evidence for this. |
| Known limitations and any conflicts of interest with the tool | Including commercial relationships with the tool's vendor, if any. |
PRISMA 2020 Item 8 (selection process) already asks whether automation tools were used in screening - this reporting satisfies that item with the detail the 2025 statement expects, rather than a one-line mention that a tool was used.
Can AI act as the second screener?
This is the actual sticking point researchers raise, and the honest answer is more open than most informal discussion assumes. The position statement notes that single-reviewer abstract screening can miss roughly 13% of relevant studies, and discusses AI as a possible second screener specifically to reduce that risk - it does not treat AI-as-second-screener as automatically disqualifying. What it does require is that you piloted the tool against human decisions on your own review (above), that independent evaluation is considered where it exists, and that human oversight stays central to the decision to use it this way at all. An unvalidated suggestion standing in, unreported, for an independent human opinion is what the statement is written to prevent - a validated, disclosed one used deliberately is a judgment call the statement leaves to the review team.
References
Flemyng E, Noel-Storr A, Macura B, et al. Position statement on artificial intelligence (AI) use in evidence synthesis across Cochrane, the Campbell Collaboration, JBI, and the Collaboration for Environmental Evidence 2025. JBI Evidence Synthesis. 2025;23(11):2162-2166. doi.org/10.11124/JBIES-25-00480. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. doi.org/10.1136/bmj.n71.
Continue your review workflow
- Set your eligibility criteria and PICO before screening starts - see the free protocol template.
- Review the full dual-screening workflow in the title/abstract screening guide.
- Document the final counts, including anything excluded by automation, with a PRISMA 2020 flow diagram.
- Start a free EvidenceFlow review to screen with optional AI-suggested decisions alongside independent human review, on the same project.
See how AI suggestions sit alongside human screening
Every study still gets an independent human decision - the suggestion is context, not a vote.
AI-suggested decisions (with confidence and reasoning) are visible alongside screening and export with the decision Excel sheet - exactly what a validation-sample comparison needs.
FAQ
Can AI replace the second reviewer in dual screening?
The November 2025 Cochrane/Campbell/JBI/CEE position statement doesn't rule this out - it notes single-reviewer abstract screening can miss around 13% of relevant studies and discusses AI as a possible second screener to reduce that risk. But it conditions this on piloting the tool against human decisions on your own review first, and transparently reporting what you did. An unvalidated AI suggestion standing in for an independent second human opinion is not what the statement describes.
What exactly do I have to disclose if I use AI screening?
Per the position statement: the tool's name and version, which stages of the review it affected, why you judged it methodologically sound for your use, and its known limitations. PRISMA 2020 Item 8 (selection process) already asks whether automation tools were used in screening - this is that same disclosure, with more detail.
Does AI-suggested screening count as a PRISMA automation exclusion?
Only if the tool removes records before a human sees them. PRISMA 2020's flow diagram has a specific line for "records excluded by automation tools" for that case. A tool that only suggests a decision alongside a human reviewer - never excluding anything on its own - has nothing to report on that line, but the use of the tool itself still belongs in your methods.
How do I validate an AI screening tool before trusting it?
Run it against a sample of studies you (or your team) have already screened by hand - ideally studies whose true include/exclude status you're confident in, including some near your eligibility boundary, not just obvious cases. Compare the AI's suggestion to the human decision on that sample and report the agreement rate. That's the "pilot or calibrate" step the position statement describes, done once per review, before you rely on the tool at scale.
Is a confidence score the same as accuracy?
No. A confidence score reflects the model's certainty about its own answer, not how often that answer matches the right one. The only way to know actual accuracy for your review is to check a sample of the model's suggestions against real human decisions - which is exactly what validating on a sample is for.