How to remove duplicate references in a systematic review
Short answer
Import every database export into one library, match records on identifiers (DOI, PMID) and on normalized title, authors, year and journal, review uncertain pairs by hand, keep the most complete record of each pair, and record how many duplicates you removed. That number goes into the identification section of your PRISMA 2020 flow diagram.
Step by step
- 1
Export complete records from every source
Export results in a structured format (RIS is the most portable) with all available fields: title, authors, year, journal, volume, pages, DOI, PMID and abstract. Keep the raw export files and note how many records each database returned; you will need those numbers for reporting.
- 2
Put all records in one place
Import everything into a single library or tool before screening. Deduplicating within one database and then across databases separately is error-prone; one combined pass is simpler and easier to report.
- 3
Match on identifiers first
Records that share a DOI or a PubMed ID are almost always the same publication. Identifier matching is fast and precise, but it is not enough on its own: many records, especially from regional databases and conference abstracts, have no DOI.
- 4
Match on normalized bibliographic fields
Compare titles after normalizing case, punctuation and accents, together with first author, year and journal. Databases format the same article differently (abbreviated journal names, translated titles, page ranges), so exact string matching misses duplicates while overly loose matching merges different papers.
- 5
Review uncertain pairs by hand
Check pairs where the match is not certain: same title but different year, conference abstract and full article, erratum and original, or a translated version. Keep the record with the most complete metadata and abstract.
- 6
Record and report the count
Write down how many records were identified per source, how many duplicates were removed, and how they were removed (tool, settings and manual checks). PRISMA 2020 asks for duplicate records removed before screening as a separate count.
Duplicate records are not the same as multiple reports
Deduplication removes copies of the same record. It does not resolve the case where one study is published in several reports, such as a main trial paper, a secondary analysis and a conference abstract. The Cochrane Handbook recommends linking multiple reports of the same study and treating the study as the unit of analysis, which you usually do later, during screening and data extraction.
How Prolytiq does this step
In Prolytiq you import RIS or CSV exports into the project and the deduplication step flags likely duplicate pairs using similarity across bibliographic fields. You can resolve each pair manually or let clear-cut pairs be resolved automatically, and the counts carry through to the project's PRISMA flow.
See the workflow →Frequently asked questions
Can I just use my reference manager to remove duplicates?
You can, and many reviews do, but automatic deduplication in reference managers can miss duplicates or merge different records depending on the settings. Whatever tool you use, check uncertain pairs by hand and report the method.
Should I deduplicate before or after screening?
Before. Removing duplicates first saves screening effort and keeps the counts in the PRISMA 2020 flow diagram consistent.
Methodological sources
- Cochrane Handbook for Systematic Reviews of Interventions, chapter 4: Searching for and selecting studies
- PRISMA 2020 statement (Page et al., BMJ 2021)
- Bramer WM et al. De-duplication of database search results for systematic reviews in EndNote. Journal of the Medical Library Association, 2016
Run your next systematic review in Prolytiq
From research question to publishable chart, with every decision logged and exportable for peer review.