what is systematic review research

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

Knowledge Base

Methodology

Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

Formulate a research question
Develop a protocol
Search for all relevant studies
Apply the selection criteria
Extract the data
Synthesize the data
Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

Academic style
Vague sentences
Style consistency

See an example

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Prevent plagiarism. Run a free check.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
Access to databases and journal archives. Often, your educational institution provides you with access.
Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

They minimize research bias by considering all available evidence and evaluating each study for bias.
Their methods are transparent , so they can be scrutinized by others.
They’re thorough : they summarize all available evidence.
They can be replicated and updated by others.

Systematic reviews also have a few cons .

They’re time-consuming .
They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

Allow you to more effectively communicate your research to other researchers and practitioners
Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

Population(s) or problem(s)
Intervention(s)
Comparison(s)

You can rearrange these four components to write your research question:

What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

Type of study design(s)
The population of patients with eczema
The intervention of probiotics
In comparison to no treatment, placebo , or non-probiotic treatment
The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
Randomized control trials, a type of study design

Their research question was:

What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

Background information : Provide the context of the research question, including why it’s important.
Research objective (s) : Rephrase your research question as an objective.
Selection criteria: State how you’ll decide which studies to include or exclude from your review.
Search strategy: Discuss your plan for finding studies.
Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
Handsearch: Conference proceedings and reference lists of articles
Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

Abstract : A summary of the review
Introduction : Including the rationale and objectives
Methods : Including the selection criteria, search method, data extraction method, and synthesis method
Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
Discussion : Including interpretation of the results and limitations of the review
Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

Student’s t -distribution
Normal distribution
Null and Alternative Hypotheses
Chi square tests
Confidence interval
Quartiles & Quantiles
Cluster sampling
Stratified sampling
Data cleansing
Reproducibility vs Replicability
Peer review
Prospective cohort study

Research bias

Implicit bias
Cognitive bias
Placebo effect
Hawthorne effect
Hindsight bias
Affect heuristic
Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other academic texts , with an introduction , a main body, and a conclusion .

An annotated bibliography is a list of source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a paper .

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved September 3, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

Systematic Reviews

What is a Systematic Review?

A systematic review is an evidence synthesis that uses explicit, reproducible methods to perform a comprehensive literature search and critical appraisal of individual studies and that uses appropriate statistical techniques to combine these valid studies.

Key Characteristics of a Systematic Review:

Generally, systematic reviews must have:

a clearly stated set of objectives with pre-defined eligibility criteria for studies
an explicit, reproducible methodology
a systematic search that attempts to identify all studies that would meet the eligibility criteria
an assessment of the validity of the findings of the included studies, for example through the assessment of the risk of bias
a systematic presentation, and synthesis, of the characteristics and findings of the included studies.

A meta-analysis is a systematic review that uses quantitative methods to synthesize and summarize the pooled data from included studies.

Additional Information

How-to Books
Beyond Health Sciences

Cochrane Handbook For Systematic Reviews of Interventions Provides guidance to authors for the preparation of Cochrane Intervention reviews. Chapter 6 covers searching for reviews.
Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care From The University of York Centre for Reviews and Dissemination: Provides practical guidance for undertaking evidence synthesis based on a thorough understanding of systematic review methodology. It presents the core principles of systematic reviewing, and in complementary chapters, highlights issues that are specific to reviews of clinical tests, public health interventions, adverse effects, and economic evaluations.
Cornell, Sytematic Reviews and Evidence Synthesis Beyond the Health Sciences Video series geared for librarians but very informative about searching outside medicine.
<< Previous: Getting Started
Next: Levels of Evidence >>
Getting Started
Levels of Evidence
Locating Systematic Reviews
Searching Systematically
Developing Answerable Questions
Identifying Synonyms & Related Terms
Using Truncation and Wildcards
Identifying Search Limits/Exclusion Criteria
Keyword vs. Subject Searching
Where to Search
Search Filters
Sensitivity vs. Precision
Core Databases
Other Databases
Clinical Trial Registries
Conference Presentations
Databases Indexing Grey Literature
Web Searching
Handsearching
Citation Indexes
Documenting the Search Process
Managing your Review

Research Support

Last Updated: Aug 14, 2024 11:07 AM
URL: https://guides.library.ucdavis.edu/systematic-reviews

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings
My Bibliography
Collections
Citation manager

Save citation to file

Email citation, add to collections.

Create a new collection
Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

Search in PubMed
Search in NLM Catalog
Add to Search

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses

Affiliations.

1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected].
2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom.
3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected].
PMID: 30089228
DOI: 10.1146/annurev-psych-010418-102803

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Keywords: evidence; guide; meta-analysis; meta-synthesis; narrative; systematic review; theory.

PubMed Disclaimer

LinkOut - more resources

Full text sources.

Ingenta plc
Ovid Technologies, Inc.

Other Literature Sources

scite Smart Citations

Miscellaneous

NCI CPTAC Assay Portal
Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

Knowledge Base
Methodology
Systematic Review | Definition, Examples & Guide

Systematic Review | Definition, Examples & Guide

Published on 15 June 2022 by Shaun Turney . Revised on 18 July 2024.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesise all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question ‘What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?’

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

What is a systematic review, systematic review vs meta-analysis, systematic review vs literature review, systematic review vs scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce research bias . The methods are repeatable , and the approach is formal and systematic:

Formulate a research question
Develop a protocol
Search for all relevant studies
Apply the selection criteria
Extract the data
Synthesise the data
Write and publish a report

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesising all available evidence and evaluating the quality of the evidence. Synthesising means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Prevent plagiarism, run a free check.

Systematic reviews often quantitatively synthesise the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesise results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarise and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimise bias by using transparent and repeatable methods.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
If you’re doing a systematic review on your own (e.g., for a research paper or thesis), you should take appropriate measures to ensure the validity and reliability of your research.
Access to databases and journal archives. Often, your educational institution provides you with access.
Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

They minimise research b ias by considering all available evidence and evaluating each study for bias.
Their methods are transparent , so they can be scrutinised by others.
They’re thorough : they summarise all available evidence.
They can be replicated and updated by others.

Systematic reviews also have a few cons .

They’re time-consuming .
They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

Allow you to more effectively communicate your research to other researchers and practitioners
Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

Population(s) or problem(s)
Intervention(s)
Comparison(s)

You can rearrange these four components to write your research question:

What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fourth component, the type of study design . In this case, the acronym is PICOT .

Type of study design(s)
The population of patients with eczema
The intervention of probiotics
In comparison to no treatment, placebo , or non-probiotic treatment
The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
Randomised control trials, a type of study design

Their research question was:

What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

Background information : Provide the context of the research question, including why it’s important.
Research objective(s) : Rephrase your research question as an objective.
Selection criteria: State how you’ll decide which studies to include or exclude from your review.
Search strategy: Discuss your plan for finding studies.
Analysis: Explain what information you’ll collect from the studies and how you’ll synthesise the data.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
Grey literature: Grey literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of grey literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of grey literature.
Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
Handsearch: Conference proceedings and reference lists of articles
Grey literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

You should apply the selection criteria in two phases:

Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarise what you did using a PRISMA flow diagram .

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
Your judgement of the quality of the evidence, including risk of bias .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomised into the control and treatment groups.

Step 6: Synthesise the data

Synthesising the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesising the data:

Narrative ( qualitative ): Summarise the information in words. You’ll need to discuss the studies and assess their overall quality.
Quantitative : Use statistical methods to summarise and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analysed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

Abstract : A summary of the review
Introduction : Including the rationale and objectives
Methods : Including the selection criteria, search method, data extraction method, and synthesis method
Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
Discussion : Including interpretation of the results and limitations of the review
Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

To familiarise yourself with the current state of knowledge on your topic
To ensure that you’re not just repeating what others have already done
To identify gaps in knowledge and unresolved problems that your research can address
To develop your theoretical framework and methodology
To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Turney, S. (2024, July 17). Systematic Review | Definition, Examples & Guide. Scribbr. Retrieved 3 September 2024, from https://www.scribbr.co.uk/research-methods/systematic-reviews/

Is this article helpful?

Shaun Turney

Other students also liked, what is a literature review | guide, template, & examples, exploratory research | definition, guide, & examples, what is peer review | types & examples.

Systematic Reviews and Meta Analysis

Getting Started
Guides and Standards
Review Protocols
Databases and Sources
Randomized Controlled Trials
Controlled Clinical Trials
Observational Designs
Tests of Diagnostic Accuracy
Software and Tools
Where do I get all those articles?
Collaborations
EPI 233/528
Countway Mediated Search
Risk of Bias (RoB)

Systematic review Q & A

What is a systematic review.

A systematic review is guided filtering and synthesis of all available evidence addressing a specific, focused research question, generally about a specific intervention or exposure. The use of standardized, systematic methods and pre-selected eligibility criteria reduce the risk of bias in identifying, selecting and analyzing relevant studies. A well-designed systematic review includes clear objectives, pre-selected criteria for identifying eligible studies, an explicit methodology, a thorough and reproducible search of the literature, an assessment of the validity or risk of bias of each included study, and a systematic synthesis, analysis and presentation of the findings of the included studies. A systematic review may include a meta-analysis.

For details about carrying out systematic reviews, see the Guides and Standards section of this guide.

Is my research topic appropriate for systematic review methods?

A systematic review is best deployed to test a specific hypothesis about a healthcare or public health intervention or exposure. By focusing on a single intervention or a few specific interventions for a particular condition, the investigator can ensure a manageable results set. Moreover, examining a single or small set of related interventions, exposures, or outcomes, will simplify the assessment of studies and the synthesis of the findings.

Systematic reviews are poor tools for hypothesis generation: for instance, to determine what interventions have been used to increase the awareness and acceptability of a vaccine or to investigate the ways that predictive analytics have been used in health care management. In the first case, we don't know what interventions to search for and so have to screen all the articles about awareness and acceptability. In the second, there is no agreed on set of methods that make up predictive analytics, and health care management is far too broad. The search will necessarily be incomplete, vague and very large all at the same time. In most cases, reviews without clearly and exactly specified populations, interventions, exposures, and outcomes will produce results sets that quickly outstrip the resources of a small team and offer no consistent way to assess and synthesize findings from the studies that are identified.

If not a systematic review, then what?

You might consider performing a scoping review . This framework allows iterative searching over a reduced number of data sources and no requirement to assess individual studies for risk of bias. The framework includes built-in mechanisms to adjust the analysis as the work progresses and more is learned about the topic. A scoping review won't help you limit the number of records you'll need to screen (broad questions lead to large results sets) but may give you means of dealing with a large set of results.

This tool can help you decide what kind of review is right for your question.

Can my student complete a systematic review during her summer project?

Probably not. Systematic reviews are a lot of work. Including creating the protocol, building and running a quality search, collecting all the papers, evaluating the studies that meet the inclusion criteria and extracting and analyzing the summary data, a well done review can require dozens to hundreds of hours of work that can span several months. Moreover, a systematic review requires subject expertise, statistical support and a librarian to help design and run the search. Be aware that librarians sometimes have queues for their search time. It may take several weeks to complete and run a search. Moreover, all guidelines for carrying out systematic reviews recommend that at least two subject experts screen the studies identified in the search. The first round of screening can consume 1 hour per screener for every 100-200 records. A systematic review is a labor-intensive team effort.

How can I know if my topic has been been reviewed already?

Before starting out on a systematic review, check to see if someone has done it already. In PubMed you can use the systematic review subset to limit to a broad group of papers that is enriched for systematic reviews. You can invoke the subset by selecting if from the Article Types filters to the left of your PubMed results, or you can append AND systematic[sb] to your search. For example:

"neoadjuvant chemotherapy" AND systematic[sb]

The systematic review subset is very noisy, however. To quickly focus on systematic reviews (knowing that you may be missing some), simply search for the word systematic in the title:

"neoadjuvant chemotherapy" AND systematic[ti]

Any PRISMA-compliant systematic review will be captured by this method since including the words "systematic review" in the title is a requirement of the PRISMA checklist. Cochrane systematic reviews do not include 'systematic' in the title, however. It's worth checking the Cochrane Database of Systematic Reviews independently.

You can also search for protocols that will indicate that another group has set out on a similar project. Many investigators will register their protocols in PROSPERO , a registry of review protocols. Other published protocols as well as Cochrane Review protocols appear in the Cochrane Methodology Register, a part of the Cochrane Library .

Next: Guides and Standards >>
Last Updated: Sep 4, 2024 4:04 PM
URL: https://guides.library.harvard.edu/meta-analysis

10 Shattuck St, Boston MA 02115 | (617) 432-2136

| |
Copyright © 2020 President and Fellows of Harvard College. All rights reserved.

Jump to navigation

Bahasa Malaysia

What are systematic reviews?

Watch this video from Cochrane Consumers and Communication to learn what systematic reviews are, how researchers prepare them, and why they’re an important part of making informed decisions about health - for everyone.

Cochrane evidence, including our systematic reviews, provides a powerful tool to enhance your healthcare knowledge and decision making. This video from Cochrane Sweden explains a bit about how we create health evidence and what Cochrane does.

Search our Plain Language Summaries of health evidence
Learn more about Cochrane and our work

1.2.2 What is a systematic review?

A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made (Antman 1992, Oxman 1993) . The key characteristics of a systematic review are:

a clearly stated set of objectives with pre-defined eligibility criteria for studies;

an explicit, reproducible methodology;

a systematic search that attempts to identify all studies that would meet the eligibility criteria;

an assessment of the validity of the findings of the included studies, for example through the assessment of risk of bias; and

a systematic presentation, and synthesis, of the characteristics and findings of the included studies.

Many systematic reviews contain meta-analyses. Meta-analysis is the use of statistical methods to summarize the results of independent studies (Glass 1976). By combining information from all relevant studies, meta-analyses can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review (see Chapter 9, Section 9.1.3 ). They also facilitate investigations of the consistency of evidence across studies, and the exploration of differences across studies.

Help us improve our Library guides with this 5 minute survey . We appreciate your feedback!

UOW Library
Key guides for researchers

Systematic Review

What is a systematic review.

Five other types of systematic review
How is a literature review different?
Search tips for systematic reviews
Controlled vocabularies
Grey literature
Transferring your search
Documenting your results
Support & contact

A systematic review is an authoritative account of existing evidence using reliable, objective, thorough and reproducible research practices.

It is a method of making sense of large bodies of information and contributes to the answers to questions about what works and what doesn't.

Systematic reviews map areas of uncertainty and identify where little or no relevant research has been done, but where new studies are needed.

It is a good idea to familiarise yourself with the systematic review process before beginning your review. You can do this by searching for other systematic reviews to look at as examples, by reading a glossary of commonly used terms , and by learning how to distinguish between types of systematic review.

Characteristics of a systematic review

Some characteristics, or features, of systematic reviews are:

Clearly stated set of objectives with pre-defined eligibility criteria
Explicit, reproducible methodology
A systematic search that attempts to identify all studies that would meet the eligibility criteria
Assesses the validity of the findings, for example assessing the risk of bias
Systematic presentation and synthesis of the findings of the included studies. (Cochrane Handbook for Systematic Reviews of Interventions, 2008, p. 6).

Watch this video from the Cochrane Library for more information about systematic reviews.

Previous: Introduction to systematic reviews
Next: Five other types of systematic review
Last Updated: Jun 20, 2024 12:04 PM
URL: https://uow.libguides.com/systematic-review

Insert research help text here

LIBRARY RESOURCES

Library homepage

Library SEARCH

A-Z Databases

STUDY SUPPORT

Academic Skills Centre

Referencing and citing

Digital Skills Hub

MORE UOW SERVICES

UOW homepage

Student support and wellbeing

IT Services

On the lands that we study, we walk, and we live, we acknowledge and respect the traditional custodians and cultural knowledge holders of these lands.

Library Services

UCL LIBRARY SERVICES

Guides and databases
Library skills
Systematic reviews

What are systematic reviews?

Types of systematic reviews
Formulating a research question
Identifying studies
Searching databases
Describing and appraising studies
Synthesis and systematic maps
Software for systematic reviews
Online training and support
Live and face to face training
Individual support
Further help

Systematic reviews are a type of literature review of research that require equivalent standards of rigour to primary research. They have a clear, logical rationale that is reported to the reader of the review. They are used in research and policymaking to inform evidence-based decisions and practice. They differ from traditional literature reviews in the following elements of conduct and reporting.

Systematic reviews:

use explicit and transparent methods
are a piece of research following a standard set of stages
are accountable, replicable and updateable
involve users to ensure a review is relevant and useful.

For example, systematic reviews (like all research) should have a clear research question, and the perspective of the authors in their approach to addressing the question is described. There are clearly described methods on how each study in a review was identified, how that study was appraised for quality and relevance and how it is combined with other studies in order to address the review question. A systematic review usually involves more than one person in order to increase the objectivity and trustworthiness of the reviews methods and findings.

Research protocols for systematic reviews may be peer-reviewed and published or registered in a suitable repository to help avoid duplication of reviews and for comparisons to be made with the final review and the planned review.

History of systematic reviews to inform policy (EPPI-Centre)
Six reasons why it is important to be systematic (EPPI-Centre)
Evidence Synthesis International (ESI): Position Statement Describes the issues, principles and goals in synthesising research evidence to inform policy, practice and decisions

A 'non-systematic review' might use some of the same methods as systematic reviews, such as systematic approaches to identify studies or quality appraise the literature. There may be times when this approach can be useful. In a student dissertation, for example, there may not be the time to be fully systematic in a review of the literature if this is only one small part of the thesis. In other types of research, there may also be a need to obtain a quick and not necessarily thorough overview of a literature to inform some other work (including a systematic review). Another example, is where policymakers, or other people using research findings, want to make quick decisions and there is no systematic review available to help them. They have a choice of gaining a rapid overview of the research literature or not having any research evidence to help their decision-making.

Just like any other piece of research, the methods used to undertake any literature review should be carefully planned to justify the conclusions made.

Finding out about different types of systematic reviews and the methods used for systematic reviews, and reading both systematic and other types of review will help to understand some of the differences.

Typically, a systematic review addresses a focussed, structured research question in order to inform understanding and decisions on an area. (see the Formulating a research question section for examples).

Sometimes systematic reviews ask a broad research question, and one strategy to achieve this is the use of several focussed sub-questions each addressed by sub-components of the review.

Another strategy is to develop a map to describe the type of research that has been undertaken in relation to a research question. Some maps even describe over 2,000 papers, while others are much smaller. One purpose of a map is to help choose a sub-set of studies to explore more fully in a synthesis. There are also other purposes of maps: see the box on systematic evidence maps for further information.

Reporting standards specify minimum elements that need to go into the reporting of a review. The reporting standards refer mainly to methodological issues but they are not as detailed or specific as critical appraisal for the methodological standards of conduct of a review.

A number of organisations have developed specific guidelines and standards for both the conducting and reporting on systematic reviews in different topic areas.

PRISMA PRISMA is a reporting standard and is an acronym for Preferred Reporting Items for Systematic Reviews and Meta-Analyses. The Key Documents section of the PRISMA website links to a checklist, flow diagram and explanatory notes. PRISMA is less useful for certain types of reviews, including those that are iterative.
eMERGe eMERGe is a reporting standard that has been developed for meta-ethnographies, a qualitative synthesis method.
ROSES: RepOrting standards for Systematic Evidence Syntheses Reporting standards, including forms and flow diagram, designed specifically for systematic reviews and maps in the field of conservation and environmental management.

Useful books about systematic reviews

Systematic approaches to a successful literature review

An introduction to systematic reviews

Cochrane handbook for systematic reviews of interventions

Systematic reviews: crd's guidance for undertaking reviews in health care.

Finding what works in health care: Standards for systematic reviews

Systematic Reviews in the Social Sciences

Meta-analysis and research synthesis.

Research Synthesis and Meta-Analysis

Doing a Systematic Review

Literature reviews.

What is a literature review?
Why are literature reviews important?
<< Previous: Systematic reviews
Next: Types of systematic reviews >>
Last Updated: Aug 2, 2024 9:22 AM
URL: https://library-guides.ucl.ac.uk/systematic-reviews

Systematic Review

Library Help
What is a Systematic Review (SR)?
Steps of a Systematic Review
Framing a Research Question
Developing a Search Strategy
Searching the Literature
Managing the Process
Meta-analysis
Publishing your Systematic Review

Introduction to Systematic Review

Introduction
Types of literature reviews
Other Libguides
Systematic review as part of a dissertation
Tutorials & Guidelines & Examples from non-Medical Disciplines

A "high-level overview of primary research on a focused question" utilizing high-quality research evidence through:

Source: Kysh, Lynn (2013): Difference between a systematic review and a literature review. [figshare]. Available at:

Depending on your learning style, please explore the resources in various formats on the tabs above.

For additional tutorials, visit the SR Workshop Videos from UNC at Chapel Hill outlining each stage of the systematic review process.

Know the difference! Systematic review vs. literature review

It is common to confuse systematic and literature reviews as both are used to provide a summary of the existent literature or research on a specific topic. Even with this common ground, both types vary significantly. Please review the following chart (and its corresponding poster linked below) for a detailed explanation of each as well as the differences between each type of review.

Source: Kysh, L. (2013). What’s in a name? The difference between a systematic review and a literature review and why it matters. [Poster] Retrieved from .

Check the website from UNC at Chapel Hill,

Types of literature reviews along with associated methodologies

JBI Manual for Evidence Synthesis . Find definitions and methodological guidance.

- Systematic Reviews - Chapters 1-7

- Mixed Methods Systematic Reviews - Chapter 8

- Diagnostic Test Accuracy Systematic Reviews - Chapter 9

- Umbrella Reviews - Chapter 10

- Scoping Reviews - Chapter 11

- Systematic Reviews of Measurement Properties - Chapter 12

Systematic reviews vs scoping reviews -

Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information and Libraries Journal , 26 (2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x

Gough, D., Thomas, J., & Oliver, S. (2012). Clarifying differences between review designs and methods. Systematic Reviews, 1 (28). htt p s://doi.org/ 10.1186/2046-4053-1-28

Munn, Z., Peters, M., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review ? Guidance for authors when choosing between a systematic or scoping review approach. BMC medical research methodology, 18 (1), 143. https://doi.org/10.1186/s12874-018-0611-x. Also, check out the Libguide from Weill Cornell Medicine for the differences between a systematic review and a scoping review and when to embark on either one of them.

Sutton, A., Clowes, M., Preston, L., & Booth, A. (2019). Meeting the review family: Exploring review types and associated information retrieval requirements . Health Information & Libraries Journal , 36 (3), 202–222. https://doi.org/10.1111/hir.12276

Temple University. Review Types . - This guide provides useful descriptions of some of the types of reviews listed in the above article.

UMD Health Sciences and Human Services Library. Review Types . - Guide describing Literature Reviews, Scoping Reviews, and Rapid Reviews.

Whittemore, R., Chao, A., Jang, M., Minges, K. E., & Park, C. (2014). Methods for knowledge synthesis: An overview. Heart & Lung: The Journal of Acute and Critical Care, 43 (5), 453–461. https://doi.org/10.1016/j.hrtlng.2014.05.014

Differences between a systematic review and other types of reviews

Armstrong, R., Hall, B. J., Doyle, J., & Waters, E. (2011). ‘ Scoping the scope ’ of a cochrane review. Journal of Public Health , 33 (1), 147–150. https://doi.org/10.1093/pubmed/fdr015

Kowalczyk, N., & Truluck, C. (2013). Literature reviews and systematic reviews: What is the difference? Radiologic Technology , 85 (2), 219–222.

White, H., Albers, B., Gaarder, M., Kornør, H., Littell, J., Marshall, Z., Matthew, C., Pigott, T., Snilstveit, B., Waddington, H., & Welch, V. (2020). Guidance for producing a Campbell evidence and gap map . Campbell Systematic Reviews, 16 (4), e1125. https://doi.org/10.1002/cl2.1125. Check also this comparison between evidence and gaps maps and systematic reviews.

Rapid Reviews Tutorials

Rapid Review Guidebook by the National Collaborating Centre of Methods and Tools (NCCMT)

Hamel, C., Michaud, A., Thuku, M., Skidmore, B., Stevens, A., Nussbaumer-Streit, B., & Garritty, C. (2021). Defining Rapid Reviews: a systematic scoping review and thematic analysis of definitions and defining characteristics of rapid reviews. Journal of clinical epidemiology , 129 , 74–85. https://doi.org/10.1016/j.jclinepi.2020.09.041

Image: by WeeblyTutorials

under the tab on the left side menu.

Müller, C., Lautenschläger, S., Meyer, G., & Stephan, A. (2017). Interventions to support people with dementia and their caregivers during the transition from home care to nursing home care: A systematic review . International Journal of Nursing Studies, 71 , 139–152. https://doi.org/10.1016/j.ijnurstu.2017.03.013
Bhui, K. S., Aslam, R. W., Palinski, A., McCabe, R., Johnson, M. R. D., Weich, S., … Szczepura, A. (2015). Interventions to improve therapeutic communications between Black and minority ethnic patients and professionals in psychiatric services: Systematic review . The British Journal of Psychiatry, 207 (2), 95–103. https://doi.org/10.1192/bjp.bp.114.158899
Rosen, L. J., Noach, M. B., Winickoff, J. P., & Hovell, M. F. (2012). Parental smoking cessation to protect young children: A systematic review and meta-analysis . Pediatrics, 129 (1), 141–152. https://doi.org/10.1542/peds.2010-3209

Scoping Review

Hyshka, E., Karekezi, K., Tan, B., Slater, L. G., Jahrig, J., & Wild, T. C. (2017). The role of consumer perspectives in estimating population need for substance use services: A scoping review . BMC Health Services Research, 171-14. https://doi.org/10.1186/s12913-017-2153-z
Olson, K., Hewit, J., Slater, L.G., Chambers, T., Hicks, D., Farmer, A., & ... Kolb, B. (2016). Assessing cognitive function in adults during or following chemotherapy: A scoping review . Supportive Care In Cancer, 24 (7), 3223-3234. https://doi.org/10.1007/s00520-016-3215-1
Pham, M. T., Rajić, A., Greig, J. D., Sargeant, J. M., Papadopoulos, A., & McEwen, S. A. (2014). A scoping review of scoping reviews: Advancing the approach and enhancing the consistency . Research Synthesis Methods, 5 (4), 371–385. https://doi.org/10.1002/jrsm.1123
Scoping Review Tutorial from UNC at Chapel Hill

Qualitative Systematic Review/Meta-Synthesis

Lee, H., Tamminen, K. A., Clark, A. M., Slater, L., Spence, J. C., & Holt, N. L. (2015). A meta-study of qualitative research examining determinants of children's independent active free play . International Journal Of Behavioral Nutrition & Physical Activity, 12 (5), 121-12. https://doi.org/10.1186/s12966-015-0165-9

Videos on systematic reviews


This video lecture explains in detail the steps necessary to conduct a systematic review (44 min.)	Here's a brief introduction to how to evaluate systematic reviews (16 min.)

Systematic Reviews: What are they? Are they right for my research? - 47 min. video recording with a closed caption option.

More training videos on systematic reviews:

from Yale University

(approximately 5-10 minutes each)

with Margaret Foster

(approximately 55 min each)

Books on Systematic Reviews

Books on Meta-analysis

University of Toronto Libraries - very detailed with good tips on the sensitivity and specificity of searches.
Monash University - includes an interactive case study tutorial.
Dalhousie University Libraries - a comprehensive How-To Guide on conducting a systematic review.

Guidelines for a systematic review as part of the dissertation

Guidelines for Systematic Reviews in the Context of Doctoral Education Background by University of Victoria (PDF)
Can I conduct a Systematic Review as my Master’s dissertation or PhD thesis? Yes, It Depends! by Farhad (blog)
What is a Systematic Review Dissertation Like? by the University of Edinburgh (50 min video)

Further readings on experiences of PhD students and doctoral programs with systematic reviews

Puljak, L., & Sapunar, D. (2017). Acceptance of a systematic review as a thesis: Survey of biomedical doctoral programs in Europe . Systematic Reviews , 6 (1), 253. https://doi.org/10.1186/s13643-017-0653-x

Perry, A., & Hammond, N. (2002). Systematic reviews: The experiences of a PhD Student . Psychology Learning & Teaching , 2 (1), 32–35. https://doi.org/10.2304/plat.2002.2.1.32

Daigneault, P.-M., Jacob, S., & Ouimet, M. (2014). Using systematic review methods within a Ph.D. dissertation in political science: Challenges and lessons learned from practice . International Journal of Social Research Methodology , 17 (3), 267–283. https://doi.org/10.1080/13645579.2012.730704

UMD Doctor of Philosophy Degree Policies

Before you embark on a systematic review research project, check the UMD PhD Policies to make sure you are on the right path. Systematic reviews require a team of at least two reviewers and an information specialist or a librarian. Discuss with your advisor the authorship roles of the involved team members. Keep in mind that the UMD Doctor of Philosophy Degree Policies (scroll down to the section, Inclusion of one's own previously published materials in a dissertation ) outline such cases, specifically the following:

" It is recognized that a graduate student may co-author work with faculty members and colleagues that should be included in a dissertation . In such an event, a letter should be sent to the Dean of the Graduate School certifying that the student's examining committee has determined that the student made a substantial contribution to that work. This letter should also note that the inclusion of the work has the approval of the dissertation advisor and the program chair or Graduate Director. The letter should be included with the dissertation at the time of submission. The format of such inclusions must conform to the standard dissertation format. A foreword to the dissertation, as approved by the Dissertation Committee, must state that the student made substantial contributions to the relevant aspects of the jointly authored work included in the dissertation."

by CommLab India

Cochrane Handbook for Systematic Reviews of Interventions - See Part 2: General methods for Cochrane reviews
Systematic Searches - Yale library video tutorial series
Using PubMed's Clinical Queries to Find Systematic Reviews - From the U.S. National Library of Medicine
Systematic reviews and meta-analyses: A step-by-step guide - From the University of Edinsburgh, Centre for Cognitive Ageing and Cognitive Epidemiology

by Vinova

Bioinformatics

Mariano, D. C., Leite, C., Santos, L. H., Rocha, R. E., & de Melo-Minardi, R. C. (2017). A guide to performing systematic literature reviews in bioinformatics . arXiv preprint arXiv:1707.05813.

Environmental Sciences

Collaboration for Environmental Evidence. 2018. Guidelines and Standards for Evidence synthesis in Environmental Management. Version 5.0 (AS Pullin, GK Frampton, B Livoreil & G Petrokofsky, Eds) www.environmentalevidence.org/information-for-authors .

Pullin, A. S., & Stewart, G. B. (2006). Guidelines for systematic review in conservation and environmental management. Conservation Biology, 20 (6), 1647–1656. https://doi.org/10.1111/j.1523-1739.2006.00485.x

Engineering Education

Borrego, M., Foster, M. J., & Froyd, J. E. (2014). Systematic literature reviews in engineering education and other developing interdisciplinary fields. Journal of Engineering Education, 103 (1), 45–76. https://doi.org/10.1002/jee.20038

Public Health

Hannes, K., & Claes, L. (2007). Learn to read and write systematic reviews: The Belgian Campbell Group . Research on Social Work Practice, 17 (6), 748–753. https://doi.org/10.1177/1049731507303106
McLeroy, K. R., Northridge, M. E., Balcazar, H., Greenberg, M. R., & Landers, S. J. (2012). Reporting guidelines and the American Journal of Public Health’s adoption of preferred reporting items for systematic reviews and meta-analyses . American Journal of Public Health, 102 (5), 780–784. https://doi.org/10.2105/AJPH.2011.300630
Pollock, A., & Berge, E. (2018). How to do a systematic review. International Journal of Stroke, 13 (2), 138–156. https://doi.org/10.1177/1747493017743796
Institute of Medicine. (2011). Finding what works in health care: Standards for systematic reviews . https://doi.org/10.17226/13059
Wanden-Berghe, C., & Sanz-Valero, J. (2012). Systematic reviews in nutrition: Standardized methodology . The British Journal of Nutrition, 107 Suppl 2, S3-7. https://doi.org/10.1017/S0007114512001432

Social Sciences

Bronson, D., & Davis, T. (2012). Finding and evaluating evidence: Systematic reviews and evidence-based practice (Pocket guides to social work research methods). Oxford: Oxford University Press.
Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide . Malden, MA: Blackwell Pub.
Cornell University Library Guide - Systematic literature reviews in engineering: Example: Software Engineering
Biolchini, J., Mian, P. G., Natali, A. C. C., & Travassos, G. H. (2005). Systematic review in software engineering . System Engineering and Computer Science Department COPPE/UFRJ, Technical Report ES, 679 (05), 45.
Biolchini, J. C., Mian, P. G., Natali, A. C. C., Conte, T. U., & Travassos, G. H. (2007). Scientific research ontology to support systematic review in software engineering . Advanced Engineering Informatics, 21 (2), 133–151.
Kitchenham, B. (2007). Guidelines for performing systematic literature reviews in software engineering . [Technical Report]. Keele, UK, Keele University, 33(2004), 1-26.
Weidt, F., & Silva, R. (2016). Systematic literature review in computer science: A practical guide . Relatórios Técnicos do DCC/UFJF , 1 .

by Day Translations

Resources for your writing

Academic Phrasebank - Get some inspiration and find some terms and phrases for writing your research paper
Oxford English Dictionary - Use to locate word variants and proper spelling
<< Previous: Library Help
Next: Steps of a Systematic Review >>
Last Updated: Aug 26, 2024 12:37 PM
URL: https://lib.guides.umd.edu/SR

Log in using your username and password

Search More Search for this keyword Advanced search
Latest content
Current issue
Write for Us
BMJ Journals

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

A high-quality systematic review is described as the most reliable source of evidence to guide clinical practice. The purpose of a systematic review is to deliver a meticulous summary of all the available primary research in response to a research question. A systematic review uses all the existing research and is sometime called ‘secondary research’ (research on research). They are often required by research funders to establish the state of existing knowledge and are frequently used in guideline development. Systematic review findings are often used within the …

Competing interests None.

Read the full text or download the PDF:

A-Z Publications

Annual Review of Psychology

Volume 70, 2019, review article, how to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses.

Andy P. Siddaway 1 , Alex M. Wood 2 , and Larry V. Hedges 3
View Affiliations Hide Affiliations Affiliations: 1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected] 2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom 3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected]
Vol. 70:747-770 (Volume publication date January 2019) https://doi.org/10.1146/annurev-psych-010418-102803
First published as a Review in Advance on August 08, 2018
Copyright © 2019 by Annual Reviews. All rights reserved

Article metrics loading...

Full text loading...

Literature Cited

APA Publ. Commun. Board Work. Group J. Artic. Rep. Stand. 2008 . Reporting standards for research in psychology: Why do we need them? What might they be?. Am. Psychol . 63 : 848– 49 [Google Scholar]
Baumeister RF 2013 . Writing a literature review. The Portable Mentor: Expert Guide to a Successful Career in Psychology MJ Prinstein, MD Patterson 119– 32 New York: Springer, 2nd ed.. [Google Scholar]
Baumeister RF , Leary MR 1995 . The need to belong: desire for interpersonal attachments as a fundamental human motivation. Psychol. Bull. 117 : 497– 529 [Google Scholar]
Baumeister RF , Leary MR 1997 . Writing narrative literature reviews. Rev. Gen. Psychol. 3 : 311– 20 Presents a thorough and thoughtful guide to conducting narrative reviews. [Google Scholar]
Bem DJ 1995 . Writing a review article for Psychological Bulletin. Psychol . Bull 118 : 172– 77 [Google Scholar]
Borenstein M , Hedges LV , Higgins JPT , Rothstein HR 2009 . Introduction to Meta-Analysis New York: Wiley Presents a comprehensive introduction to meta-analysis. [Google Scholar]
Borenstein M , Higgins JPT , Hedges LV , Rothstein HR 2017 . Basics of meta-analysis: I 2 is not an absolute measure of heterogeneity. Res. Synth. Methods 8 : 5– 18 [Google Scholar]
Braver SL , Thoemmes FJ , Rosenthal R 2014 . Continuously cumulating meta-analysis and replicability. Perspect. Psychol. Sci. 9 : 333– 42 [Google Scholar]
Bushman BJ 1994 . Vote-counting procedures. The Handbook of Research Synthesis H Cooper, LV Hedges 193– 214 New York: Russell Sage Found. [Google Scholar]
Cesario J 2014 . Priming, replication, and the hardest science. Perspect. Psychol. Sci. 9 : 40– 48 [Google Scholar]
Chalmers I 2007 . The lethal consequences of failing to make use of all relevant evidence about the effects of medical treatments: the importance of systematic reviews. Treating Individuals: From Randomised Trials to Personalised Medicine PM Rothwell 37– 58 London: Lancet [Google Scholar]
Cochrane Collab. 2003 . Glossary Rep., Cochrane Collab. London: http://community.cochrane.org/glossary Presents a comprehensive glossary of terms relevant to systematic reviews. [Google Scholar]
Cohn LD , Becker BJ 2003 . How meta-analysis increases statistical power. Psychol. Methods 8 : 243– 53 [Google Scholar]
Cooper HM 2003 . Editorial. Psychol. Bull. 129 : 3– 9 [Google Scholar]
Cooper HM 2016 . Research Synthesis and Meta-Analysis: A Step-by-Step Approach Thousand Oaks, CA: Sage, 5th ed.. Presents a comprehensive introduction to research synthesis and meta-analysis. [Google Scholar]
Cooper HM , Hedges LV , Valentine JC 2009 . The Handbook of Research Synthesis and Meta-Analysis New York: Russell Sage Found, 2nd ed.. [Google Scholar]
Cumming G 2014 . The new statistics: why and how. Psychol. Sci. 25 : 7– 29 Discusses the limitations of null hypothesis significance testing and viable alternative approaches. [Google Scholar]
Earp BD , Trafimow D 2015 . Replication, falsification, and the crisis of confidence in social psychology. Front. Psychol. 6 : 621 [Google Scholar]
Etz A , Vandekerckhove J 2016 . A Bayesian perspective on the reproducibility project: psychology. PLOS ONE 11 : e0149794 [Google Scholar]
Ferguson CJ , Brannick MT 2012 . Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychol. Methods 17 : 120– 28 [Google Scholar]
Fleiss JL , Berlin JA 2009 . Effect sizes for dichotomous data. The Handbook of Research Synthesis and Meta-Analysis H Cooper, LV Hedges, JC Valentine 237– 53 New York: Russell Sage Found, 2nd ed.. [Google Scholar]
Garside R 2014 . Should we appraise the quality of qualitative research reports for systematic reviews, and if so, how. Innovation 27 : 67– 79 [Google Scholar]
Hedges LV , Olkin I 1980 . Vote count methods in research synthesis. Psychol. Bull. 88 : 359– 69 [Google Scholar]
Hedges LV , Pigott TD 2001 . The power of statistical tests in meta-analysis. Psychol. Methods 6 : 203– 17 [Google Scholar]
Higgins JPT , Green S 2011 . Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0 London: Cochrane Collab. Presents comprehensive and regularly updated guidelines on systematic reviews. [Google Scholar]
John LK , Loewenstein G , Prelec D 2012 . Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23 : 524– 32 [Google Scholar]
Juni P , Witschi A , Bloch R , Egger M 1999 . The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 282 : 1054– 60 [Google Scholar]
Klein O , Doyen S , Leys C , Magalhães de Saldanha da Gama PA , Miller S et al. 2012 . Low hopes, high expectations: expectancy effects and the replicability of behavioral experiments. Perspect. Psychol. Sci. 7 : 6 572– 84 [Google Scholar]
Lau J , Antman EM , Jimenez-Silva J , Kupelnick B , Mosteller F , Chalmers TC 1992 . Cumulative meta-analysis of therapeutic trials for myocardial infarction. N. Engl. J. Med. 327 : 248– 54 [Google Scholar]
Light RJ , Smith PV 1971 . Accumulating evidence: procedures for resolving contradictions among different research studies. Harvard Educ. Rev. 41 : 429– 71 [Google Scholar]
Lipsey MW , Wilson D 2001 . Practical Meta-Analysis London: Sage Comprehensive and clear explanation of meta-analysis. [Google Scholar]
Matt GE , Cook TD 1994 . Threats to the validity of research synthesis. The Handbook of Research Synthesis H Cooper, LV Hedges 503– 20 New York: Russell Sage Found. [Google Scholar]
Maxwell SE , Lau MY , Howard GS 2015 . Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?. Am. Psychol. 70 : 487– 98 [Google Scholar]
Moher D , Hopewell S , Schulz KF , Montori V , Gøtzsche PC et al. 2010 . CONSORT explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 340 : c869 [Google Scholar]
Moher D , Liberati A , Tetzlaff J , Altman DG PRISMA Group. 2009 . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339 : 332– 36 Comprehensive reporting guidelines for systematic reviews. [Google Scholar]
Morrison A , Polisena J , Husereau D , Moulton K , Clark M et al. 2012 . The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. Int. J. Technol. Assess. Health Care 28 : 138– 44 [Google Scholar]
Nelson LD , Simmons J , Simonsohn U 2018 . Psychology's renaissance. Annu. Rev. Psychol. 69 : 511– 34 [Google Scholar]
Noblit GW , Hare RD 1988 . Meta-Ethnography: Synthesizing Qualitative Studies Newbury Park, CA: Sage [Google Scholar]
Olivo SA , Macedo LG , Gadotti IC , Fuentes J , Stanton T , Magee DJ 2008 . Scales to assess the quality of randomized controlled trials: a systematic review. Phys. Ther. 88 : 156– 75 [Google Scholar]
Open Sci. Collab. 2015 . Estimating the reproducibility of psychological science. Science 349 : 943 [Google Scholar]
Paterson BL , Thorne SE , Canam C , Jillings C 2001 . Meta-Study of Qualitative Health Research: A Practical Guide to Meta-Analysis and Meta-Synthesis Thousand Oaks, CA: Sage [Google Scholar]
Patil P , Peng RD , Leek JT 2016 . What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspect. Psychol. Sci. 11 : 539– 44 [Google Scholar]
Rosenthal R 1979 . The “file drawer problem” and tolerance for null results. Psychol. Bull. 86 : 638– 41 [Google Scholar]
Rosnow RL , Rosenthal R 1989 . Statistical procedures and the justification of knowledge in psychological science. Am. Psychol. 44 : 1276– 84 [Google Scholar]
Sanderson S , Tatt ID , Higgins JP 2007 . Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int. J. Epidemiol. 36 : 666– 76 [Google Scholar]
Schreiber R , Crooks D , Stern PN 1997 . Qualitative meta-analysis. Completing a Qualitative Project: Details and Dialogue JM Morse 311– 26 Thousand Oaks, CA: Sage [Google Scholar]
Shrout PE , Rodgers JL 2018 . Psychology, science, and knowledge construction: broadening perspectives from the replication crisis. Annu. Rev. Psychol. 69 : 487– 510 [Google Scholar]
Stroebe W , Strack F 2014 . The alleged crisis and the illusion of exact replication. Perspect. Psychol. Sci. 9 : 59– 71 [Google Scholar]
Stroup DF , Berlin JA , Morton SC , Olkin I , Williamson GD et al. 2000 . Meta-analysis of observational studies in epidemiology (MOOSE): a proposal for reporting. JAMA 283 : 2008– 12 [Google Scholar]
Thorne S , Jensen L , Kearney MH , Noblit G , Sandelowski M 2004 . Qualitative meta-synthesis: reflections on methodological orientation and ideological agenda. Qual. Health Res. 14 : 1342– 65 [Google Scholar]
Tong A , Flemming K , McInnes E , Oliver S , Craig J 2012 . Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med. Res. Methodol. 12 : 181– 88 [Google Scholar]
Trickey D , Siddaway AP , Meiser-Stedman R , Serpell L , Field AP 2012 . A meta-analysis of risk factors for post-traumatic stress disorder in children and adolescents. Clin. Psychol. Rev. 32 : 122– 38 [Google Scholar]
Valentine JC , Biglan A , Boruch RF , Castro FG , Collins LM et al. 2011 . Replication in prevention science. Prev. Sci. 12 : 103– 17 [Google Scholar]
Article Type: Review Article

Most Read This Month

Most cited most cited rss feed, job burnout, executive functions, social cognitive theory: an agentic perspective, on happiness and human potentials: a review of research on hedonic and eudaimonic well-being, sources of method bias in social science research and recommendations on how to control it, mediation analysis, missing data analysis: making it work in the real world, grounded cognition, personality structure: emergence of the five-factor model, motivational beliefs, values, and goals.

Official websites use .gov

A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS

A lock ( ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Systematic Reviews

Describes what is involved with conducting a systematic review of the literature for evidence-based public health and how the librarian is a partner in the process.

Several CDC librarians have special training in conducting literature searches for systematic reviews. Literature searches for systematic reviews can take a few weeks to several months from planning to delivery.

Fill out a search request form here or contact the Stephen B. Thacker CDC Library by email [email protected] or telephone 404-639-1717.

Campbell Collaboration

Cochrane Collaboration

Eppi Centre

Joanna Briggs Institute

McMaster University

PRISMA Statement

Systematic Reviews – CRD’s Guide

Systematic Reviews of Health Promotion and Public Health Interventions

The Guide to Community Preventive Services

Look for systematic reviews that have already been published.

To ensure that the work has not already been done.
To provides examples of search strategies for your topic

Look in PROSPERO for registered systematic reviews.

Search Cochrane and CRD-York for systematic reviews.

Search filter for finding systematic reviews in PubMed

Other search filters to locate systematic reviews

A systematic review attempts to collect and analyze all evidence that answers a specific question. The question must be clearly defined and have inclusion and exclusion criteria. A broad and thorough search of the literature is performed and a critical analysis of the search results is reported and ultimately provides a current evidence-based answer to the specific question.

Time: According to Cochrane , it takes 18 months on average to complete a Systematic Review.

The average systematic review from beginning to end requires 18 months of work. “…to find out about a healthcare intervention it is worth searching research literature thoroughly to see if the answer is already known. This may require considerable work over many months…” ( Cochrane Collaboration )

Review Team: Team Members at minimum…

Content expert
2 reviewers
1 tie breaker
1 statistician (meta-analysis)
1 economist if conducting an economic analysis
*1 librarian (expert searcher) trained in systematic reviews

“Expert searchers are an important part of the systematic review team, crucial throughout the review process-from the development of the proposal and research question to publication.” ( McGowan & Sampson, 2005 )

*Ask your librarian to write a methods section regarding the search methods and to give them co-authorship. You may also want to consider providing a copy of one or all of the search strategies used in an appendix.

The Question to Be Answered: A clearly defined and specific question or questions with inclusion and exclusion criteria.

Written Protocol: Outline the study method, rationale, key questions, inclusion and exclusion criteria, literature searches, data abstraction and data management, analysis of quality of the individual studies, synthesis of data, and grading of the evidience for each key question.

Literature Searches: Search for any systematic reviews that may already answer the key question(s). Next, choose appropriate databases and conduct very broad, comprehensive searches. Search strategies must be documented so that they can be duplicated. The librarian is integral to this step of the process. Before your librarian creates a search strategy and starts searching in earnest you should write a detailed PICO question , determine the inclusion and exclusion criteria for your study, run a preliminary search, and have 2-4 articles that already fit the criteria for your review.

What is searched depends on the topic of the review but should include…

At least 3 standard medical databases like PubMed/Medline, CINAHL, Embase, etc..
At least 2 grey literature resources like Clinicaltrials.gov, COS Conference Papers Index, Grey Literature Report, etc…

Citation Management: EndNote is a bibliographic management tools that assist researchers in managing citations. The Stephen B. Thacker CDC Library oversees the site license for EndNote.

To request installation: The library provides EndNote to CDC staff under a site-wide license. Please use the ITSO Software Request Tool (SRT) and submit a request for the latest version (or upgraded version) of EndNote. Please be sure to include the computer name for the workstation where you would like to have the software installed.

EndNote Training: CDC Library offers training on EndNote on a regular basis – both a basic and advanced course. To view the course descriptions and upcoming training dates, please visit the CDC Library training page .

For assistance with EndNote software, please contact [email protected]

Vendor Support and Services: EndNote – Support and Services (Thomson Reuters) EndNote – Tutorials and Live Online Classes (Thomson Reuters)

Getting Articles:

Articles can be obtained using DocExpress or by searching the electronic journals at the Stephen B. Thacker CDC Library.

IOM Standards for Systematic Reviews: Standard 3.1: Conduct a comprehensive systematic search for evidence

The goal of a systematic review search is to maximize recall and precision while keeping results manageable. Recall (sensitivity) is defined as the number of relevant reports identified divided by the total number of relevant reports in existence. Precision (specificity) is defined as the number of relevant reports identified divided by the total number of reports identified.

Issues to consider when creating a systematic review search:

All concepts are included in the strategy
All appropriate subject headings are used
Appropriate use of explosion
Appropriate use of subheadings and floating subheadings
Use of natural language (text words) in addition to controlled vocabulary terms
Use of appropriate synonyms, acronyms, etc.
Truncation and spelling variation as appropriate
Appropriate use of limits such as language, years, etc.
Field searching, publication type, author, etc.
Boolean operators used appropriately
Line errors: when searches are combined using line numbers, be sure the numbers refer to the searches intended
Check indexing of relevant articles
Search strategy adapted as needed for multiple databases
Cochrane Handbook: Searching for Studies See Part 2, Chapter 6

A step-by-step guide to systematically identify all relevant animal studies

Materials listed in these guides are selected to provide awareness of quality public health literature and resources. A material’s inclusion does not necessarily represent the views of the U.S. Department of Health and Human Services (HHS), the Public Health Service (PHS), or the Centers for Disease Control and Prevention (CDC), nor does it imply endorsement of the material’s methods or findings. HHS, PHS, and CDC assume no responsibility for the factual accuracy of the items presented. The selection, omission, or content of items does not imply any endorsement or other position taken by HHS, PHS, and CDC. Opinion, findings, and conclusions expressed by the original authors of items included in these materials, or persons quoted therein, are strictly their own and are in no way meant to represent the opinion or views of HHS, PHS, or CDC. References to publications, news sources, and non-CDC Websites are provided solely for informational purposes and do not imply endorsement by HHS, PHS, or CDC.

Research Consultation
How-to Guides
People Search
Library Databases
Research Guides
Find Periodicals
Google Scholar
Library Map
Interlibrary Loan - ILL
Research & Instruction
Computers and Laptops
Services For...
Reserve a Room
APSU Records Management
Library Collections
Digital Collections
Special Collections
Veterans' Oral History
Government Resources
Subject Librarians
Library Information
Woodward Library Society
New @ Woodward Library
Celebration of Scholarship
Join Woodward Library Society

Systematic Reviews: What is a systematic review?

What is a systematic review?
How do you perform a systematic review?
Tools & Resources

What is a Systematic Review?

In the health-related professions, systematic reviews are considered the most reliable resources. They are studies that make up the top level of the evidence-based information pyramid, and as a result, they are the most sought after information for questions about health. Systematic reviews are not quite the same as literature reviews. Literature reviews, also known as narrative reviews, attempt to find all published materials on a subject, whereas systematic reviews try to find everything that focuses on answering a specific question. Since systematic reviews are generally associated with health related fields, their main objective is to ensure the results of the review provide accurate evidence that answers relevant questions. If you are looking for information about literature reviews, please check the library's guide on the topic here .

Why use systematic reviews to answer questions?

When looking for answers to health questions, systematic reviews are considered the best resources to use for evidence-based information. The predefined protocols, the amount of information reviewed, the evaluation process involved, and the efforts to eliminate bias are all a part of what makes health professionals consider systematic reviews to be the highest level of evidence based information available. As a part of the process, systematic reviews tend to look at and evaluate all the randomized controlled trials, or all the cohort studies, for their specific topic. By looking at and evaluating a vast amount of comparable studies, a systematic review is able to provide answers that have a much stronger level of evidence than any individual study.

Why perform a systematic review?

These reviews collect large amounts of information that fit within the predetermined parameters, so performing a systematic review is an excellent way to develop expertise on a topic. Setting up the criteria, searching for the information, and evaluating the information found, gives the reviewer an extremely strong understanding of the process needed to create a review as well as how to evaluate its various elements. Creating a systematic review gives the reviewer an opportunity to further the discussion on a topic. In the health fields, performing and then publishing these reviews provides more evidence on topics that can be used for making decisions in a clinical environment.

How to find a systematic review

There are a number of databases that focus on health related resources, and most of them search through journals that include systematic reviews. In these cases, you can include the words “systematic review” and the results will include entries that have the words “systematic review” in them somewhere. Many of these results will be systematic reviews; however, some of the results may include these words, but are not systematic reviews. A few databases that are used by researchers have added in limitation features that make it easier to find systematic reviews, and ensure a specific article/document/publication type are found. Here are three examples of databases and how to limit their search results to systematic reviews:

Scroll down the main page and in the middle column, click on Clinical Queries. In the provided search box type in your search terms. The middle row of results is systematic reviews. You can type your search terms in on the main page, and limit the results after they are listed. To do this, find the section for “article types” on the left side of the page and click “customize”. On the list that pops up, clear all the check boxes except the one for “systematic reviews”, then click “Show”. When the main page is displayed again, click on “systematic reviews” to reset the search using that limit.
In CINAHL (EBSCO version) it is easy to limit search results to systematic reviews from the initial search page. Scroll down to find “Publication types” on the right side of the page. In the Publication Types box, search through the list until you find systematic reviews and highlight it. After this has been done, add in any other limits needed and then search as usual.
This is the easiest of the three to use for finding systematic reviews. On the main search page, type in the terms, run the search and wait for the results page. On the results page, the right side has a limit that says “systematic review”. Click on it and the results will all be systematic reviews.

Types of systematic reviews

These are a few of the various types of systematic reviews.

Critical review

Critical reviews are often thought to be the same as a literature review. They involve a comprehensive review of a topic that involves a high level of analysis that seeks to identify the most important aspects of a subject.

The overview is a look at the literature available on the topic. They may or may not have an analytical component that provides a synthesis of the information found, but this depends on how systematic the review has been.

Qualitative systematic review

This type of study looks at as many qualitative studies and tries to find themes among these studies that lie across the range of reviews. The information is then organized into a narrative that is frequently accompanied by a conceptual model.

Rapid review

The number of studies included in a rapid review are dictated by time constraints. Most often these reviews are conducted to determine what is known about policies or practices, and then determine how much these are based on evidence.

Scoping review

These reviews are intended to quickly assess the availability of research on a specific topic, and then determine whether the evidence provided in these research articles is sufficient and appropriate,

Umbrella review

Umbrella reviews research broad topic, and look at a wide range of reviews that have been done for a number of potential interventions. It then triest to establish the pros and cons of each treatment and what the potential results would be.

Subject Guide

Next: How do you perform a systematic review? >>
Last Updated: Jun 30, 2023 2:03 PM
URL: https://libguides.apsu.edu/systematic-reviews

Search Menu
Sign in through your institution
Advance articles
Editor's Choice
100 years of the AJE
Collections
Author Guidelines
Submission Site
Open Access Options
About American Journal of Epidemiology
About the Johns Hopkins Bloomberg School of Public Health
Journals Career Network
Editorial Board
Advertising and Corporate Services
Self-Archiving Policy
Dispatch Dates
Journals on Oxford Academic
Books on Oxford Academic

Article Contents

Assessing the certainty of the evidence in systematic reviews: importance, process, and use.

Article contents
Figures & tables
Supplementary Data

Romina Brignardello-Petersen, Gordon H Guyatt, Assessing the Certainty of the Evidence in Systematic Reviews: Importance, Process, and Use, American Journal of Epidemiology , 2024;, kwae332, https://doi.org/10.1093/aje/kwae332

Permissions Icon Permissions

When interpreting results and drawing conclusions, authors of systematic reviews should consider the limitations of the evidence included in their review. The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach provides a framework for the explicit consideration of the limitations of the evidence included in a systematic review, and for incorporating this assessment into the conclusions. Assessments of certainty of evidence are a methodological expectation of systematic reviews. The certainty of the evidence is specific to each outcome in a systematic review, and can be rated as high, moderate, low, or very low. Because it will have an important impact, before conducting certainty of evidence, reviewers must clarify the intent of their question: are they interested in causation or association. Serious concerns regarding limitations in the study design, inconsistency, imprecision, indirectness, and publication bias can decrease the certainty of the evidence. Using an example, this article describes and illustrates the importance and the steps for assessing the certainty of evidence and drawing accurate conclusions in a systematic review.

publication bias
grade approach

Email alerts

Citing articles via, looking for your next opportunity.

Recommend to your Library

Affiliations

Online ISSN 1476-6256
Print ISSN 0002-9262
About Oxford Academic
Publish journals with us
University press partners
What we publish
New features
Open access
Institutional account management
Rights and permissions
Get help with access
Accessibility
Advertising
Media enquiries
Oxford University Press
Oxford Languages
University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

Cookie settings
Cookie policy
Privacy policy
Legal notice

This Feature Is Available To Subscribers Only

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Something old, new, and borrowed . Rise of the systematic reviews

Published: 24 August 2024

Cite this article

Gheorghe-Gavrilă Hognogi 1 &
Ana-Maria Pop ORCID: orcid.org/0000-0001-9958-1391 1

126 Accesses

Explore all metrics

Systematic reviews and other types of literature reviews are more prevalent in clinical medicine than in other fields. The recurring need for improvement and updates in these disciplines has led to the Living Systematic Review (LSR) concept to enhance the effectiveness of scientific synthesis efforts. While LSR was introduced in 2014, its adoption outside clinical medicine has been limited, with one exception. However, it is anticipated that this will change in the future, prompting a detailed exploration of four key dimensions for LSR development, regardless of the scientific domain. These dimensions include (a) compliance with FAIR principles, (b) interactivity to facilitate easier access to scientific knowledge, (c) public participation for a more comprehensive review, and (d) extending the scope beyond mere updates to living systematic reviews. Each field needs to establish clear guidelines for drafting literature reviews as independent studies, with discussions centring around the central theme of the Living Systematic Review.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

Get 10 units per month
Download Article/Chapter or eBook
1 Unit = 1 Article or 1 Chapter
Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Explore related subjects

Artificial Intelligence

Aguilar Gómez, F., & Bernal, I. (2023). FAIR EVA: Bringing institutional multidisciplinary repositories into the FAIR picture. Scientific Data . https://doi.org/10.1038/s41597-023-02652-8

Article Google Scholar

Akl, E. A., Meerpohl, J. J., Elliott, J., Kahale, L. A., & Schünemann, H. J. (2017). Living systematic reviews: 4. Living guideline recommendations. Journal of Clinical Epidemiology, 91 , 47–53. https://doi.org/10.1016/j.jclinepi.2017.08.009

Amaral, O. B. (2022). To fix peer review, break it into stages. Nature, 611 , 637. https://doi.org/10.1038/d41586-022-03791-5

Breuer, C., Meerpohl, J. J., & Siemens, W. (2022). From standard systematic reviews to living systematic reviews. Zeitschrift fur Evidenz, Fortbildung und Qualitat Im Gesundheitswesen, 176 , 76–81. https://doi.org/10.1016/j.zefq.2022.11.007

Dajani, R. (2023). Scientists in diaspora are a powerful resource for their home countries. Nature . https://doi.org/10.1038/d41586-023-03300-2

Eisen, M. B., Akhmanova, A., Behrens, T. E., Harper, D. M., Weigel, D., & Zaidi, M. (2020). Implementing a “publish, then review” model of publishing. eLife, 9 , e64910. https://doi.org/10.7554/eLife.64910

Elliott, J. H., Synnot, A., Turner, T., Simmonds, M., Akl, E. A., McDonald, S., Salanti, G., Meerpohl, J., MacLehose, H., Hilton, J., Tovey, D., Shemilt, I., Thomas, J., Agoritsas, T., Hilton, J., Perron, C., Akl, E., Hodder, R., Pestridge, C., …, Pearson, L. (2017). Living systematic review: 1. Introduction—The why, what, when, and how. Journal of Clinical Epidemiology, 91 , 23–30. https://doi.org/10.1016/j.jclinepi.2017.08.010

Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins, J. P. T., Mavergames, C., & Gruen, R. L. (2014). Living systematic reviews: An emerging opportunity to narrow the evidence-practice gap. PLoS Medicine . https://doi.org/10.1371/journal.pmed.1001603

Enck, P. (2018). Living systematic reviews, not only for clinical (placebo) research. Journal of Clinical Epidemiology, 98 , 153–153. https://doi.org/10.1016/j.jclinepi.2018.01.001

Gerber, L. R. (2023). Bridging the gap between science and policy for a sustainable future. Nature Water . https://doi.org/10.1038/s44221-023-00145-x

Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26 (2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x

Griebler, U., Dobrescu, A., Ledinger, D., Klingenstein, P., Sommer, I., Emprechtinger, R., Persad, E., Gadinger, A., Trivella, M., Klerings, I., & Nussbaumer-Streit, B. (2023). Evaluation of the interim Cochrane rapid review methods guidance—A mixed-methods study on the understanding of and adherence to the guidance. Research Synthesis Methods . https://doi.org/10.1002/jrsm.1656

Hill, J. E., Harris, C., & Clegg, A. (2023). Methods for using Bing’s AI-powered search engine for data extraction for a systematic review. Research Synthesis Methods, 15 (2), 347–353. https://doi.org/10.1002/jrsm.1689

Kahale, L. A., Piechotta, V., & McKenzie, J. E. (2022). Extension of the PRISMA 2020 statement for living systematic reviews (LSRs): Protocol [version 2; peer review: 1 approved]. F1000Research . https://doi.org/10.12688/f1000research.75449.2

Macdonald, H., Loder, E., & Abbasi, K. (2020). Living systematic reviews at The BMJ. BMJ, 370 , m2925. https://doi.org/10.1136/bmj.m2925

Marshall, I. J., & Wallace, B. C. (2019). Toward systematic review automation: A practical guide to using machine learning tools in research synthesis. Systematic Reviews . https://doi.org/10.1186/s13643-019-1074-9

Norström, A., V., Cvitanovic, C., Löf, M. F., West, S., Wyborn, C., Balvanera, P., Bednarek, A. T., Bennett, E. M., Biggs, R., Bremond, A., Campbell, B. M., Canadell, J. G., Carpenter, S. R., Folke, C., Fulton, E. A., Gaffney, O., Gelcich, S., Jouffray, J.-B., Leach, M., …, Österblom, H. (2020). Principles for knowledge co-production in sustainability research. Nature Sustainability, 3 , 182–190. https://doi.org/10.1038/s41893-019-0448-2

Paul, M., & Leeflang, M. M. (2023). Living systematic reviews: Aims and standards. Clinical Microbiology and Infection, 30 (3), 265–266. https://doi.org/10.1016/j.cmi.2023.08.005

Polonioli, A. (2019). A plea for minimally biased naturalistic philosophy. Synthese, 196 , 3841–3867. https://doi.org/10.1007/s11229-017-1628-0

Polonioli, A. (2020). In search of better science: On the epistemic costs of systematic reviews and the need for a pluralistic stance to literature search. Scientometrics, 122 , 1267–1274. https://doi.org/10.1007/s11192-019-03333-3

Riley, S. P., Swanson, B. T., Shaffer, S. M., Flowers, D. W., Cook, C. E., & Brismée, J. M. (2023). Why do ‘Trustworthy’ living systematic reviews matter? Journal of Manual & Manipulative Therapy, 31 (4), 215–219. https://doi.org/10.1080/10669817.2023.2229610

Ripberger, J., Bell, A., Fox, A., Forney, A., Livingston, W., Gaddie, C., Silva, C., & Jenkins-Smith, H. (2022). Communicating probability information in weather forecasts: Findings and recommendations from a living systematic review of the research literature. Weather, Climate, and Society, 14 (2), 481–498. https://doi.org/10.1175/WCAS-D-21-0034.1

Roche, D. G., Kruuk, L. E. B., Lanfear, R., & Binning, S. A. (2015). Public data archiving in ecology and evolution: How well are we doing? PLoS Biology, 13 (11), e1002295. https://doi.org/10.1371/journal.pbio.1002295

Saulnier, K. M., Bujold, D., Dyke, S. O. M., Dupras, C., Beck, S., Bourque, G., & Joly, Y. (2019). Benefits and barriers in the design of harmonized access agreements for international data sharing. Scientific Data . https://doi.org/10.1038/s41597-019-0310-4

Schimidt, L., Mohamed, S., Meader, N., Bacardit, J., & Craig, D. (2023). Automated data analysis of unstructured grey literature in health research: A mapping review. Research Synthesis Methods, 15 (2), 178–197. https://doi.org/10.1002/jrsm.1692

Siemieniuk, R. A., Bartoszko, J. J., Zeraatkar, D., Kum, E., Qasim, A., Martinez, J. P. D., Izcovich, A., Lamontagne, F., Han, M. A., Agarwal, A., Agoritsas, T., Azab, M., Bravo, G., Chu, D. K., Couban, R., Devji, T., Escamilla, Z., Foroutan, F., Gao, Y., …, Brignardello-Petersen, R. (2020). Drug treatments for Covid-19: Living systematic review and network meta-analysis BMJ, 370 , m3536. https://doi.org/10.1136/bmj.m2980

Simmonds, M., Salanti, G., McKenzie, J., & Elliott, J. (2017). Living systematic reviews: 3. Statistical methods for updating meta-analyses. Journal of Clinical Epidemiology, 91 , 38–46. https://doi.org/10.1016/j.jclinepi.2017.08.008

Siontis, K. C., & Ioannidis, J. P. A. (2018). Replication, duplication, and waste in a quarter million systematic reviews and meta-analyses. Circulation. Cardiovascular Quality and Outcomes, 11 (12), e005212. https://doi.org/10.1161/CIRCOUTCOMES.118.005212

Thibault, R. T., Amaral, O. B., Argolo, F., Bandrowski, A. E., Davidson, A. R., & Drude, N. I. (2023). Open Science 2.0: Towards a truly collaborative research ecosystem. PLoS Biology, 21 (10), e3002362. https://doi.org/10.1371/journal.pbio.3002362

Thomas, J., Noel-Storr, A., Marshall, I., Wallace, B., McDonald, S., Mavergames, C., Glasziou, P., Shemilt, I., Synnot, A., Turner, T., & Elliott, J. (2017). Living systematic reviews: 2. Combining human and machine effort. Journal of Clinical Epidemiology, 91 , 31–37. https://doi.org/10.1016/j.jclinepi.2017.08.011

Thorp, H. H. (2023). Correction is courageous. Science, 382 , 743–743. https://doi.org/10.1126/science.adm8205

Turk, V. (2023). Protect the ‘right to science’ for people and the planet. Nature . https://doi.org/10.1038/d41586-023-03332-8

Turner, T., Lavis, J. N., Grimshaw, J. M., Green, S., & Elliott, J. (2023). Living evidence and adaptive policy: Perfect partners? Health Research Policy and Systems . https://doi.org/10.1186/s12961-023-01085-4

Uttley, L., Quintana, D. S., Montgomery, P., Carroll, C., Page, M. J., Falzon, L., Sutton, A., & Moher, D. (2023). The problems with systematic reviews: A living systematic review. Journal of Clinical Epidemiology, 156 , 30–41. https://doi.org/10.1016/j.jclinepi.2023.01.011

Vallet, A., Locatelli, B., Valdivia-Díaz, M., Vallet, A., Locatelli, B., Valdivia-Díaz, M., Conde, Y. Q., Matencio García, G., Criales, A. R., Huamanñahui, F. V., Criales, S. R., Makowski, D., & Lavorel, S. (2023). Knowledge coproduction to improve assessments of nature’s contributions to people. Conservation Biology . https://doi.org/10.1111/cobi.14182

van Noorden, R. (2023). How big is science’s fake-paper problem? Nature, 623 , 466–467. https://doi.org/10.1038/d41586-023-03464-x

Wilkinson, M., Dumontier, M., Aalbersberg, I., Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., ..., Mons, B. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3 , 160018. https://doi.org/10.1038/sdata.2016.18

Download references

Acknowledgements

The authors would like to thank both reviewers for their useful and precious suggestions.

Author information

Authors and affiliations.

Universitatea Babeș-Bolyai, Facultatea de Geografie, Centrul de Geografie Regională, Str. Clinicilor 5-7, 400006, Cluj-Napoca, Romania

Gheorghe-Gavrilă Hognogi & Ana-Maria Pop

You can also search for this author in PubMed Google Scholar

Contributions

Gheorghe-Gavrilă Hognogi—substantial contribution to conception and design, acquisition and interpretation of data, writing the comment, revision; Ana-Maria Pop—interpretation of data.

Corresponding author

Correspondence to Gheorghe-Gavrilă Hognogi .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Hognogi, GG., Pop, AM. Something old, new, and borrowed . Rise of the systematic reviews. Scientometrics (2024). https://doi.org/10.1007/s11192-024-05133-w

Download citation

Received : 07 February 2024

Accepted : 02 August 2024

Published : 24 August 2024

DOI : https://doi.org/10.1007/s11192-024-05133-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Living systematic reviews
Scientific knowledge
FAIR principles
Systematic review
Find a journal
Publish with us
Track your research

Literature Reviews

What is a Literature Review?

Steps for Creating a Literature Review
Providing Evidence / Critical Analysis
Challenges when writing a Literature Review
Systematic Literature Reviews

A literature review is an academic text that surveys, synthesizes, and critically evaluates the existing literature on a specific topic. It is typically required for theses, dissertations, or long reports and serves several key purposes:

Surveying the Literature : It involves a comprehensive search and examination of relevant academic books, journal articles, and other sources related to the chosen topic.
Synthesizing Information : The literature review summarizes and organizes the information found in the literature, often identifying patterns, themes, and gaps in the current knowledge.
Critical Analysis : It critically analyzes the collected information, highlighting limitations, gaps, and areas of controversy, and suggests directions for future research.
Establishing Context : It places the current research within the broader context of the field, demonstrating how the new research builds on or diverges from previous studies.

Types of Literature Reviews

Literature reviews can take various forms, including:

Narrative Reviews : These provide a qualitative summary of the literature and are often used to give a broad overview of a topic. They may be less structured and more subjective, focusing on synthesizing the literature to support a particular viewpoint.
Systematic Reviews : These are more rigorous and structured, following a specific methodology to identify, evaluate, and synthesize all relevant studies on a particular question. They aim to minimize bias and provide a comprehensive summary of the existing evidence.
Integrative Reviews : Similar to systematic reviews, but they aim to generate new knowledge by integrating findings from different studies to develop new theories or frameworks.

Importance of Literature Reviews

Foundation for Research : They provide a solid background for new research projects, helping to justify the research question and methodology.

Identifying Gaps : Literature reviews highlight areas where knowledge is lacking, guiding future research efforts.

Building Credibility : Demonstrating familiarity with existing research enhances the credibility of the researcher and their work.

In summary, a literature review is a critical component of academic research that helps to frame the current state of knowledge, identify gaps, and provide a basis for new research.

The research, the body of current literature, and the particular objectives should all influence the structure of a literature review. It is also critical to remember that creating a literature review is an ongoing process - as one reads and analyzes the literature, one's understanding may change, which could require rearranging the literature review.

Paré, G. and Kitsiou, S. (2017) 'Methods for Literature Reviews' , in: Lau, F. and Kuziemsky, C. (eds.) Handbook of eHealth evaluation: an evidence-based approach . Victoria (BC): University of Victoria.

Perplexity AI (2024) Perplexity AI response to Kathy Neville, 31 July.

Royal Literary Fund (2024) The structure of a literature review. Available at: https://www.rlf.org.uk/resources/the-structure-of-a-literature-review/ (Accessed: 23 July 2024).

Library Services for Undergraduate Research (2024) Literature review: a definition . Available at: https://libguides.wustl.edu/our?p=302677 (Accessed: 31 July 2024).

An overview of methodological approaches in systematic reviews

Prabhakar veginadu.

1 Department of Rural Clinical Sciences, La Trobe Rural Health School, La Trobe University, Bendigo Victoria, Australia

Hanny Calache

2 Lincoln International Institute for Rural Health, University of Lincoln, Brayford Pool, Lincoln UK

Akshaya Pandian

3 Department of Orthodontics, Saveetha Dental College, Chennai Tamil Nadu, India

Mohd Masood

Associated data.

APPENDIX B: List of excluded studies with detailed reasons for exclusion

APPENDIX C: Quality assessment of included reviews using AMSTAR 2

The aim of this overview is to identify and collate evidence from existing published systematic review (SR) articles evaluating various methodological approaches used at each stage of an SR.

The search was conducted in five electronic databases from inception to November 2020 and updated in February 2022: MEDLINE, Embase, Web of Science Core Collection, Cochrane Database of Systematic Reviews, and APA PsycINFO. Title and abstract screening were performed in two stages by one reviewer, supported by a second reviewer. Full‐text screening, data extraction, and quality appraisal were performed by two reviewers independently. The quality of the included SRs was assessed using the AMSTAR 2 checklist.

The search retrieved 41,556 unique citations, of which 9 SRs were deemed eligible for inclusion in final synthesis. Included SRs evaluated 24 unique methodological approaches used for defining the review scope and eligibility, literature search, screening, data extraction, and quality appraisal in the SR process. Limited evidence supports the following (a) searching multiple resources (electronic databases, handsearching, and reference lists) to identify relevant literature; (b) excluding non‐English, gray, and unpublished literature, and (c) use of text‐mining approaches during title and abstract screening.

The overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process, as well as some methodological modifications currently used in expedited SRs. Overall, findings of this overview highlight the dearth of published SRs focused on SR methodologies and this warrants future work in this area.

1. INTRODUCTION

Evidence synthesis is a prerequisite for knowledge translation. 1 A well conducted systematic review (SR), often in conjunction with meta‐analyses (MA) when appropriate, is considered the “gold standard” of methods for synthesizing evidence related to a topic of interest. 2 The central strength of an SR is the transparency of the methods used to systematically search, appraise, and synthesize the available evidence. 3 Several guidelines, developed by various organizations, are available for the conduct of an SR; 4 , 5 , 6 , 7 among these, Cochrane is considered a pioneer in developing rigorous and highly structured methodology for the conduct of SRs. 8 The guidelines developed by these organizations outline seven fundamental steps required in SR process: defining the scope of the review and eligibility criteria, literature searching and retrieval, selecting eligible studies, extracting relevant data, assessing risk of bias (RoB) in included studies, synthesizing results, and assessing certainty of evidence (CoE) and presenting findings. 4 , 5 , 6 , 7

The methodological rigor involved in an SR can require a significant amount of time and resource, which may not always be available. 9 As a result, there has been a proliferation of modifications made to the traditional SR process, such as refining, shortening, bypassing, or omitting one or more steps, 10 , 11 for example, limits on the number and type of databases searched, limits on publication date, language, and types of studies included, and limiting to one reviewer for screening and selection of studies, as opposed to two or more reviewers. 10 , 11 These methodological modifications are made to accommodate the needs of and resource constraints of the reviewers and stakeholders (e.g., organizations, policymakers, health care professionals, and other knowledge users). While such modifications are considered time and resource efficient, they may introduce bias in the review process reducing their usefulness. 5

Substantial research has been conducted examining various approaches used in the standardized SR methodology and their impact on the validity of SR results. There are a number of published reviews examining the approaches or modifications corresponding to single 12 , 13 or multiple steps 14 involved in an SR. However, there is yet to be a comprehensive summary of the SR‐level evidence for all the seven fundamental steps in an SR. Such a holistic evidence synthesis will provide an empirical basis to confirm the validity of current accepted practices in the conduct of SRs. Furthermore, sometimes there is a balance that needs to be achieved between the resource availability and the need to synthesize the evidence in the best way possible, given the constraints. This evidence base will also inform the choice of modifications to be made to the SR methods, as well as the potential impact of these modifications on the SR results. An overview is considered the choice of approach for summarizing existing evidence on a broad topic, directing the reader to evidence, or highlighting the gaps in evidence, where the evidence is derived exclusively from SRs. 15 Therefore, for this review, an overview approach was used to (a) identify and collate evidence from existing published SR articles evaluating various methodological approaches employed in each of the seven fundamental steps of an SR and (b) highlight both the gaps in the current research and the potential areas for future research on the methods employed in SRs.

An a priori protocol was developed for this overview but was not registered with the International Prospective Register of Systematic Reviews (PROSPERO), as the review was primarily methodological in nature and did not meet PROSPERO eligibility criteria for registration. The protocol is available from the corresponding author upon reasonable request. This overview was conducted based on the guidelines for the conduct of overviews as outlined in The Cochrane Handbook. 15 Reporting followed the Preferred Reporting Items for Systematic reviews and Meta‐analyses (PRISMA) statement. 3

2.1. Eligibility criteria

Only published SRs, with or without associated MA, were included in this overview. We adopted the defining characteristics of SRs from The Cochrane Handbook. 5 According to The Cochrane Handbook, a review was considered systematic if it satisfied the following criteria: (a) clearly states the objectives and eligibility criteria for study inclusion; (b) provides reproducible methodology; (c) includes a systematic search to identify all eligible studies; (d) reports assessment of validity of findings of included studies (e.g., RoB assessment of the included studies); (e) systematically presents all the characteristics or findings of the included studies. 5 Reviews that did not meet all of the above criteria were not considered a SR for this study and were excluded. MA‐only articles were included if it was mentioned that the MA was based on an SR.

SRs and/or MA of primary studies evaluating methodological approaches used in defining review scope and study eligibility, literature search, study selection, data extraction, RoB assessment, data synthesis, and CoE assessment and reporting were included. The methodological approaches examined in these SRs and/or MA can also be related to the substeps or elements of these steps; for example, applying limits on date or type of publication are the elements of literature search. Included SRs examined or compared various aspects of a method or methods, and the associated factors, including but not limited to: precision or effectiveness; accuracy or reliability; impact on the SR and/or MA results; reproducibility of an SR steps or bias occurred; time and/or resource efficiency. SRs assessing the methodological quality of SRs (e.g., adherence to reporting guidelines), evaluating techniques for building search strategies or the use of specific database filters (e.g., use of Boolean operators or search filters for randomized controlled trials), examining various tools used for RoB or CoE assessment (e.g., ROBINS vs. Cochrane RoB tool), or evaluating statistical techniques used in meta‐analyses were excluded. 14

2.2. Search

The search for published SRs was performed on the following scientific databases initially from inception to third week of November 2020 and updated in the last week of February 2022: MEDLINE (via Ovid), Embase (via Ovid), Web of Science Core Collection, Cochrane Database of Systematic Reviews, and American Psychological Association (APA) PsycINFO. Search was restricted to English language publications. Following the objectives of this study, study design filters within databases were used to restrict the search to SRs and MA, where available. The reference lists of included SRs were also searched for potentially relevant publications.

The search terms included keywords, truncations, and subject headings for the key concepts in the review question: SRs and/or MA, methods, and evaluation. Some of the terms were adopted from the search strategy used in a previous review by Robson et al., which reviewed primary studies on methodological approaches used in study selection, data extraction, and quality appraisal steps of SR process. 14 Individual search strategies were developed for respective databases by combining the search terms using appropriate proximity and Boolean operators, along with the related subject headings in order to identify SRs and/or MA. 16 , 17 A senior librarian was consulted in the design of the search terms and strategy. Appendix A presents the detailed search strategies for all five databases.

2.3. Study selection and data extraction

Title and abstract screening of references were performed in three steps. First, one reviewer (PV) screened all the titles and excluded obviously irrelevant citations, for example, articles on topics not related to SRs, non‐SR publications (such as randomized controlled trials, observational studies, scoping reviews, etc.). Next, from the remaining citations, a random sample of 200 titles and abstracts were screened against the predefined eligibility criteria by two reviewers (PV and MM), independently, in duplicate. Discrepancies were discussed and resolved by consensus. This step ensured that the responses of the two reviewers were calibrated for consistency in the application of the eligibility criteria in the screening process. Finally, all the remaining titles and abstracts were reviewed by a single “calibrated” reviewer (PV) to identify potential full‐text records. Full‐text screening was performed by at least two authors independently (PV screened all the records, and duplicate assessment was conducted by MM, HC, or MG), with discrepancies resolved via discussions or by consulting a third reviewer.

Data related to review characteristics, results, key findings, and conclusions were extracted by at least two reviewers independently (PV performed data extraction for all the reviews and duplicate extraction was performed by AP, HC, or MG).

2.4. Quality assessment of included reviews

The quality assessment of the included SRs was performed using the AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews). The tool consists of a 16‐item checklist addressing critical and noncritical domains. 18 For the purpose of this study, the domain related to MA was reclassified from critical to noncritical, as SRs with and without MA were included. The other six critical domains were used according to the tool guidelines. 18 Two reviewers (PV and AP) independently responded to each of the 16 items in the checklist with either “yes,” “partial yes,” or “no.” Based on the interpretations of the critical and noncritical domains, the overall quality of the review was rated as high, moderate, low, or critically low. 18 Disagreements were resolved through discussion or by consulting a third reviewer.

2.5. Data synthesis

To provide an understandable summary of existing evidence syntheses, characteristics of the methods evaluated in the included SRs were examined and key findings were categorized and presented based on the corresponding step in the SR process. The categories of key elements within each step were discussed and agreed by the authors. Results of the included reviews were tabulated and summarized descriptively, along with a discussion on any overlap in the primary studies. 15 No quantitative analyses of the data were performed.

From 41,556 unique citations identified through literature search, 50 full‐text records were reviewed, and nine systematic reviews 14 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 were deemed eligible for inclusion. The flow of studies through the screening process is presented in Figure 1 . A list of excluded studies with reasons can be found in Appendix B .

An external file that holds a picture, illustration, etc.
Object name is JEBM-15-39-g001.jpg

Study selection flowchart

3.1. Characteristics of included reviews

Table 1 summarizes the characteristics of included SRs. The majority of the included reviews (six of nine) were published after 2010. 14 , 22 , 23 , 24 , 25 , 26 Four of the nine included SRs were Cochrane reviews. 20 , 21 , 22 , 23 The number of databases searched in the reviews ranged from 2 to 14, 2 reviews searched gray literature sources, 24 , 25 and 7 reviews included a supplementary search strategy to identify relevant literature. 14 , 19 , 20 , 21 , 22 , 23 , 26 Three of the included SRs (all Cochrane reviews) included an integrated MA. 20 , 21 , 23

Characteristics of included studies

Author, year	Search strategy (year last searched; no. databases; supplementary searches)	SR design (type of review; no. of studies included)	Topic; subject area	SR objectives	SR authors’ comments on study quality
Crumley, 2005	2004; Seven databases; four journals handsearched, reference lists and contacting authors	SR; = 64	RCTs and CCTs; not specified	To identify and quantitatively review studies comparing two or more different resources (e.g., databases, Internet, handsearching) used to identify RCTs and CCTs for systematic reviews.	Most of the studies adequately described reproducible search methods, expected search yield. Poor quality in studies was mainly due to lack of rigor in reporting selection methodology. Majority of the studies did not indicate the number of people involved in independently screening the searches or applying eligibility criteria to identify potentially relevant studies.
Hopewell, 2007	2002; eight databases; selected journals and published abstracts handsearched, and contacting authors	SR and MA; = 34 (34 in quantitative analysis)	RCTs; health care	To review systematically empirical studies, which have compared the results of handsearching with the results of searching one or more electronic databases to identify reports of randomized trials.	The electronic search was designed and carried out appropriately in majority of the studies, while the appropriateness of handsearching was unclear in half the studies because of limited information. The screening studies methods used in both groups were comparable in most of the studies.
Hopewell, 2007	2005; two databases; selected journals and published abstracts handsearched, reference lists, citations and contacting authors	SR and MA; = 5 (5 in quantitative analysis)	RCTs; health care	To review systematically research studies, which have investigated the impact of gray literature in meta‐analyses of randomized trials of health care interventions.	In majority of the studies, electronic searches were designed and conducted appropriately, and the selection of studies for eligibility was similar for handsearching and database searching. Insufficient data for most studies to assess the appropriateness of handsearching and investigator agreeability on the eligibility of the trial reports.
Horsley, 2011	2008; three databases; reference lists, citations and contacting authors	SR; = 12	Any topic or study area	To investigate the effectiveness of checking reference lists for the identification of additional, relevant studies for systematic reviews. Effectiveness is defined as the proportion of relevant studies identified by review authors solely by checking reference lists.	Interpretability and generalizability of included studies was difficult. Extensive heterogeneity among the studies in the number and type of databases used. Lack of control in majority of the studies related to the quality and comprehensiveness of searching.
Morrison, 2012	2011; six databases and gray literature	SR; = 5	RCTs; conventional medicine	To examine the impact of English language restriction on systematic review‐based meta‐analyses	The included studies were assessed to have good reporting quality and validity of results. Methodological issues were mainly noted in the areas of sample power calculation and distribution of confounders.
Robson, 2019	2016; three databases; reference lists and contacting authors	SR; = 37	N/R	To identify and summarize studies assessing methodologies for study selection, data abstraction, or quality appraisal in systematic reviews.	The quality of the included studies was generally low. Only one study was assessed as having low RoB across all four domains. Majority of the studies were assessed to having unclear RoB across one or more domains.
Schmucker, 2017	2016; four databases; reference lists	SR; = 10	Study data; medicine	To assess whether the inclusion of data that were not published at all and/or published only in the gray literature influences pooled effect estimates in meta‐analyses and leads to different interpretation.	Majority of the included studies could not be judged on the adequacy of matching or adjusting for confounders of the gray/unpublished data in comparison to published data.
					Also, generalizability of results was low or unclear in four research projects
Morissette, 2011	2009; five databases; reference lists and contacting authors	SR and MA; = 6 (5 included in quantitative analysis)	N/R	To determine whether blinded versus unblinded assessments of risk of bias result in similar or systematically different assessments in studies included in a systematic review.	Four studies had unclear risk of bias, while two studies had high risk of bias.
O'Mara‐Eves, 2015	2013; 14 databases and gray literature	SR; = 44	N/R	To gather and present the available research evidence on existing methods for text mining related to the title and abstract screening stage in a systematic review, including the performance metrics used to evaluate these technologies.	Quality appraised based on two criteria‐sampling of test cases and adequacy of methods description for replication. No study was excluded based on the quality (author contact).

SR = systematic review; MA = meta‐analysis; RCT = randomized controlled trial; CCT = controlled clinical trial; N/R = not reported.

The included SRs evaluated 24 unique methodological approaches (26 in total) used across five steps in the SR process; 8 SRs evaluated 6 approaches, 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 while 1 review evaluated 18 approaches. 14 Exclusion of gray or unpublished literature 21 , 26 and blinding of reviewers for RoB assessment 14 , 23 were evaluated in two reviews each. Included SRs evaluated methods used in five different steps in the SR process, including methods used in defining the scope of review ( n = 3), literature search ( n = 3), study selection ( n = 2), data extraction ( n = 1), and RoB assessment ( n = 2) (Table 2 ).

Summary of findings from review evaluating systematic review methods

Key elements	Author, year	Method assessed	Evaluations/outcomes (P—primary; S—secondary)	Summary of SR authors’ conclusions	Quality of review

Excluding study data based on publication status	Hopewell, 2007	Gray vs. published literature	Pooled effect estimate	Published trials are usually larger and show an overall greater treatment effect than gray trials. Excluding trials reported in gray literature from SRs and MAs may exaggerate the results.	Moderate
	Schmucker, 2017	Gray and/or unpublished vs. published literature	P: Pooled effect estimate	Excluding unpublished trials had no or only a small effect on the pooled estimates of treatment effects. Insufficient evidence to conclude the impact of including unpublished or gray study data on MA conclusions.	Moderate
			S: Impact on interpretation of MA
Excluding study data based on language of publication	Morrison, 2012	English language vs. non‐English language publications	P: Bias in summary treatment effects	No evidence of a systematic bias from the use of English language restrictions in systematic review‐based meta‐analyses in conventional medicine. Conflicting results on the methodological and reporting quality of English and non‐English language RCTs. Further research required.	Low
			S: number of included studies and patients, methodological quality and statistical heterogeneity

Resources searching	Crumley, 2005	Two or more resources searching vs. resource‐specific searching	Recall and precision	Multiple‐source comprehensive searches are necessary to identify all RCTs for a systematic review. For electronic databases, using the Cochrane HSS or complex search strategy in consultation with a librarian is recommended.	Critically low
Supplementary searching	Hopewell, 2007	Handsearching only vs. one or more electronic database(s) searching	Number of identified randomized trials	Handsearching is important for identifying trial reports for inclusion in systematic reviews of health care interventions published in nonindexed journals. Where time and resources are limited, majority of the full English‐language trial reports can be identified using a complex search or the Cochrane HSS.	Moderate
	Horsley, 2011	Checking reference list (no comparison)	P: additional yield of checking reference lists	There is some evidence to support the use of checking reference lists to complement literature search in systematic reviews.	Low
			S: additional yield by publication type, study design or both and data pertaining to costs

Reviewer characteristics	Robson, 2019	Single vs. double reviewer screening	P: Accuracy, reliability, or efficiency of a method	Using two reviewers for screening is recommended. If resources are limited, one reviewer can screen, and other reviewer can verify the list of excluded studies.	Low
			S: factors affecting accuracy or reliability of a method
		Experienced vs. inexperienced reviewers for screening		Screening must be performed by experienced reviewers
		Screening by blinded vs. unblinded reviewers		Authors do not recommend blinding of reviewers during screening as the blinding process was time‐consuming and had little impact on the results of MA
Use of technology for study selection	Robson, 2019	Use of dual computer monitors vs. nonuse of dual monitors for screening	P: Accuracy, reliability, or efficiency of a method	There are no significant differences in the time spent on abstract or full‐text screening with the use and nonuse of dual monitors	Low
			S: factors affecting accuracy or reliability of a method
		Use of Google translate to translate non‐English citations to facilitate screening		Use of Google translate to screen German language citations
	O'Mara‐Eves, 2015	Use of text mining for title and abstract screening	Any evaluation concerning workload reduction	Text mining approaches can be used to reduce the number of studies to be screened, increase the rate of screening, improve the workflow with screening prioritization, and replace the second reviewer. The evaluated approaches reported saving a workload of between 30% and 70%	Critically low
Order of screening	Robson, 2019	Title‐first screening vs. title‐and‐abstract simultaneous screening	P: Accuracy, reliability, or efficiency of a method	Title‐first screening showed no substantial gain in time when compared to simultaneous title and abstract screening.	Low
			S: factors affecting accuracy or reliability of a method

Reviewer characteristics	Robson, 2019	Single vs. double reviewer data extraction	P: Accuracy, reliability, or efficiency of a method	Use two reviewers for data extraction. Single reviewer data extraction followed by the verification of outcome data by a second reviewer (where statistical analysis is planned), if resources preclude	Low
			S: factors affecting accuracy or reliability of a method
		Experienced vs. inexperienced reviewers for data extraction		Experienced reviewers must be used for extracting continuous outcomes data
		Data extraction by blinded vs. unblinded reviewers		Authors do not recommend blinding of reviewers during data extraction as it had no impact on the results of MA
Use of technology for data extraction		Use of dual computer monitors vs. nonuse of dual monitors for data extraction		Using two computer monitors may improve the efficiency of data extraction
		Data extraction by two English reviewers using Google translate vs. data extraction by two reviewers fluent in respective languages		Google translate provides limited accuracy for data extraction
		Computer‐assisted vs. double reviewer extraction of graphical data		Use of computer‐assisted programs to extract graphical data
Obtaining additional data		Contacting study authors for additional data		Recommend contacting authors for obtaining additional relevant data

Reviewer characteristics	Robson, 2019	Quality appraisal by blinded vs. unblinded reviewers	P: Accuracy, reliability, or efficiency of a method	Inconsistent results on RoB assessments performed by blinded and unblinded reviewers. Blinding reviewers for quality appraisal not recommended	Low
			S: factors affecting accuracy or reliability of a method
	Morissette, 2011	Risk of bias (RoB) assessment by blinded vs. unblinded reviewers	P: Mean difference and 95% confidence interval between RoB assessment scores	Findings related to the difference between blinded and unblinded RoB assessments are inconsistent from the studies. Pooled effects show no differences in RoB assessments for assessments completed in a blinded or unblinded manner.	Moderate
			S: qualitative level of agreement, mean RoB scores and measures of variance for the results of the RoB assessments, and inter‐rater reliability between blinded and unblinded reviewers
	Robson, 2019	Experienced vs. inexperienced reviewers for quality appraisal	P: Accuracy, reliability, or efficiency of a method	Reviewers performing quality appraisal must be trained. Quality assessment tool must be pilot tested.	Low
			S: factors affecting accuracy or reliability of a method
		Use of additional guidance vs. nonuse of additional guidance for quality appraisal		Providing guidance and decision rules for quality appraisal improved the inter‐rater reliability in RoB assessments.
Obtaining additional data		Contacting study authors for obtaining additional information/use of supplementary information available in the published trials vs. no additional information for quality appraisal		Additional data related to study quality obtained by contacting study authors improved the quality assessment.
RoB assessment of qualitative studies		Structured vs. unstructured appraisal of qualitative research studies		Use of structured tool if qualitative and quantitative studies designs are included in the review. For qualitative reviews, either structured or unstructured quality appraisal tool can be used.

There was some overlap in the primary studies evaluated in the included SRs on the same topics: Schmucker et al. 26 and Hopewell et al. 21 ( n = 4), Hopewell et al. 20 and Crumley et al. 19 ( n = 30), and Robson et al. 14 and Morissette et al. 23 ( n = 4). There were no conflicting results between any of the identified SRs on the same topic.

3.2. Methodological quality of included reviews

Overall, the quality of the included reviews was assessed as moderate at best (Table 2 ). The most common critical weakness in the reviews was failure to provide justification for excluding individual studies (four reviews). Detailed quality assessment is provided in Appendix C .

3.3. Evidence on systematic review methods

3.3.1. methods for defining review scope and eligibility.

Two SRs investigated the effect of excluding data obtained from gray or unpublished sources on the pooled effect estimates of MA. 21 , 26 Hopewell et al. 21 reviewed five studies that compared the impact of gray literature on the results of a cohort of MA of RCTs in health care interventions. Gray literature was defined as information published in “print or electronic sources not controlled by commercial or academic publishers.” Findings showed an overall greater treatment effect for published trials than trials reported in gray literature. In a more recent review, Schmucker et al. 26 addressed similar objectives, by investigating gray and unpublished data in medicine. In addition to gray literature, defined similar to the previous review by Hopewell et al., the authors also evaluated unpublished data—defined as “supplemental unpublished data related to published trials, data obtained from the Food and Drug Administration or other regulatory websites or postmarketing analyses hidden from the public.” The review found that in majority of the MA, excluding gray literature had little or no effect on the pooled effect estimates. The evidence was limited to conclude if the data from gray and unpublished literature had an impact on the conclusions of MA. 26

Morrison et al. 24 examined five studies measuring the effect of excluding non‐English language RCTs on the summary treatment effects of SR‐based MA in various fields of conventional medicine. Although none of the included studies reported major difference in the treatment effect estimates between English only and non‐English inclusive MA, the review found inconsistent evidence regarding the methodological and reporting quality of English and non‐English trials. 24 As such, there might be a risk of introducing “language bias” when excluding non‐English language RCTs. The authors also noted that the numbers of non‐English trials vary across medical specialties, as does the impact of these trials on MA results. Based on these findings, Morrison et al. 24 conclude that literature searches must include non‐English studies when resources and time are available to minimize the risk of introducing “language bias.”

3.3.2. Methods for searching studies

Crumley et al. 19 analyzed recall (also referred to as “sensitivity” by some researchers; defined as “percentage of relevant studies identified by the search”) and precision (defined as “percentage of studies identified by the search that were relevant”) when searching a single resource to identify randomized controlled trials and controlled clinical trials, as opposed to searching multiple resources. The studies included in their review frequently compared a MEDLINE only search with the search involving a combination of other resources. The review found low median recall estimates (median values between 24% and 92%) and very low median precisions (median values between 0% and 49%) for most of the electronic databases when searched singularly. 19 A between‐database comparison, based on the type of search strategy used, showed better recall and precision for complex and Cochrane Highly Sensitive search strategies (CHSSS). In conclusion, the authors emphasize that literature searches for trials in SRs must include multiple sources. 19

In an SR comparing handsearching and electronic database searching, Hopewell et al. 20 found that handsearching retrieved more relevant RCTs (retrieval rate of 92%−100%) than searching in a single electronic database (retrieval rates of 67% for PsycINFO/PsycLIT, 55% for MEDLINE, and 49% for Embase). The retrieval rates varied depending on the quality of handsearching, type of electronic search strategy used (e.g., simple, complex or CHSSS), and type of trial reports searched (e.g., full reports, conference abstracts, etc.). The authors concluded that handsearching was particularly important in identifying full trials published in nonindexed journals and in languages other than English, as well as those published as abstracts and letters. 20

The effectiveness of checking reference lists to retrieve additional relevant studies for an SR was investigated by Horsley et al. 22 The review reported that checking reference lists yielded 2.5%–40% more studies depending on the quality and comprehensiveness of the electronic search used. The authors conclude that there is some evidence, although from poor quality studies, to support use of checking reference lists to supplement database searching. 22

3.3.3. Methods for selecting studies

Three approaches relevant to reviewer characteristics, including number, experience, and blinding of reviewers involved in the screening process were highlighted in an SR by Robson et al. 14 Based on the retrieved evidence, the authors recommended that two independent, experienced, and unblinded reviewers be involved in study selection. 14 A modified approach has also been suggested by the review authors, where one reviewer screens and the other reviewer verifies the list of excluded studies, when the resources are limited. It should be noted however this suggestion is likely based on the authors’ opinion, as there was no evidence related to this from the studies included in the review.

Robson et al. 14 also reported two methods describing the use of technology for screening studies: use of Google Translate for translating languages (for example, German language articles to English) to facilitate screening was considered a viable method, while using two computer monitors for screening did not increase the screening efficiency in SR. Title‐first screening was found to be more efficient than simultaneous screening of titles and abstracts, although the gain in time with the former method was lesser than the latter. Therefore, considering that the search results are routinely exported as titles and abstracts, Robson et al. 14 recommend screening titles and abstracts simultaneously. However, the authors note that these conclusions were based on very limited number (in most instances one study per method) of low‐quality studies. 14

3.3.4. Methods for data extraction

Robson et al. 14 examined three approaches for data extraction relevant to reviewer characteristics, including number, experience, and blinding of reviewers (similar to the study selection step). Although based on limited evidence from a small number of studies, the authors recommended use of two experienced and unblinded reviewers for data extraction. The experience of the reviewers was suggested to be especially important when extracting continuous outcomes (or quantitative) data. However, when the resources are limited, data extraction by one reviewer and a verification of the outcomes data by a second reviewer was recommended.

As for the methods involving use of technology, Robson et al. 14 identified limited evidence on the use of two monitors to improve the data extraction efficiency and computer‐assisted programs for graphical data extraction. However, use of Google Translate for data extraction in non‐English articles was not considered to be viable. 14 In the same review, Robson et al. 14 identified evidence supporting contacting authors for obtaining additional relevant data.

3.3.5. Methods for RoB assessment

Two SRs examined the impact of blinding of reviewers for RoB assessments. 14 , 23 Morissette et al. 23 investigated the mean differences between the blinded and unblinded RoB assessment scores and found inconsistent differences among the included studies providing no definitive conclusions. Similar conclusions were drawn in a more recent review by Robson et al., 14 which included four studies on reviewer blinding for RoB assessment that completely overlapped with Morissette et al. 23

Use of experienced reviewers and provision of additional guidance for RoB assessment were examined by Robson et al. 14 The review concluded that providing intensive training and guidance on assessing studies reporting insufficient data to the reviewers improves RoB assessments. 14 Obtaining additional data related to quality assessment by contacting study authors was also found to help the RoB assessments, although based on limited evidence. When assessing the qualitative or mixed method reviews, Robson et al. 14 recommends the use of a structured RoB tool as opposed to an unstructured tool. No SRs were identified on data synthesis and CoE assessment and reporting steps.

4. DISCUSSION

4.1. summary of findings.

Nine SRs examining 24 unique methods used across five steps in the SR process were identified in this overview. The collective evidence supports some current traditional and modified SR practices, while challenging other approaches. However, the quality of the included reviews was assessed to be moderate at best and in the majority of the included SRs, evidence related to the evaluated methods was obtained from very limited numbers of primary studies. As such, the interpretations from these SRs should be made cautiously.

The evidence gathered from the included SRs corroborate a few current SR approaches. 5 For example, it is important to search multiple resources for identifying relevant trials (RCTs and/or CCTs). The resources must include a combination of electronic database searching, handsearching, and reference lists of retrieved articles. 5 However, no SRs have been identified that evaluated the impact of the number of electronic databases searched. A recent study by Halladay et al. 27 found that articles on therapeutic intervention, retrieved by searching databases other than PubMed (including Embase), contributed only a small amount of information to the MA and also had a minimal impact on the MA results. The authors concluded that when the resources are limited and when large number of studies are expected to be retrieved for the SR or MA, PubMed‐only search can yield reliable results. 27

Findings from the included SRs also reiterate some methodological modifications currently employed to “expedite” the SR process. 10 , 11 For example, excluding non‐English language trials and gray/unpublished trials from MA have been shown to have minimal or no impact on the results of MA. 24 , 26 However, the efficiency of these SR methods, in terms of time and the resources used, have not been evaluated in the included SRs. 24 , 26 Of the SRs included, only two have focused on the aspect of efficiency 14 , 25 ; O'Mara‐Eves et al. 25 report some evidence to support the use of text‐mining approaches for title and abstract screening in order to increase the rate of screening. Moreover, only one included SR 14 considered primary studies that evaluated reliability (inter‐ or intra‐reviewer consistency) and accuracy (validity when compared against a “gold standard” method) of the SR methods. This can be attributed to the limited number of primary studies that evaluated these outcomes when evaluating the SR methods. 14 Lack of outcome measures related to reliability, accuracy, and efficiency precludes making definitive recommendations on the use of these methods/modifications. Future research studies must focus on these outcomes.

Some evaluated methods may be relevant to multiple steps; for example, exclusions based on publication status (gray/unpublished literature) and language of publication (non‐English language studies) can be outlined in the a priori eligibility criteria or can be incorporated as search limits in the search strategy. SRs included in this overview focused on the effect of study exclusions on pooled treatment effect estimates or MA conclusions. Excluding studies from the search results, after conducting a comprehensive search, based on different eligibility criteria may yield different results when compared to the results obtained when limiting the search itself. 28 Further studies are required to examine this aspect.

Although we acknowledge the lack of standardized quality assessment tools for methodological study designs, we adhered to the Cochrane criteria for identifying SRs in this overview. This was done to ensure consistency in the quality of the included evidence. As a result, we excluded three reviews that did not provide any form of discussion on the quality of the included studies. The methods investigated in these reviews concern supplementary search, 29 data extraction, 12 and screening. 13 However, methods reported in two of these three reviews, by Mathes et al. 12 and Waffenschmidt et al., 13 have also been examined in the SR by Robson et al., 14 which was included in this overview; in most instances (with the exception of one study included in Mathes et al. 12 and Waffenschmidt et al. 13 each), the studies examined in these excluded reviews overlapped with those in the SR by Robson et al. 14

One of the key gaps in the knowledge observed in this overview was the dearth of SRs on the methods used in the data synthesis component of SR. Narrative and quantitative syntheses are the two most commonly used approaches for synthesizing data in evidence synthesis. 5 There are some published studies on the proposed indications and implications of these two approaches. 30 , 31 These studies found that both data synthesis methods produced comparable results and have their own advantages, suggesting that the choice of the method must be based on the purpose of the review. 31 With increasing number of “expedited” SR approaches (so called “rapid reviews”) avoiding MA, 10 , 11 further research studies are warranted in this area to determine the impact of the type of data synthesis on the results of the SR.

4.2. Implications for future research

The findings of this overview highlight several areas of paucity in primary research and evidence synthesis on SR methods. First, no SRs were identified on methods used in two important components of the SR process, including data synthesis and CoE and reporting. As for the included SRs, a limited number of evaluation studies have been identified for several methods. This indicates that further research is required to corroborate many of the methods recommended in current SR guidelines. 4 , 5 , 6 , 7 Second, some SRs evaluated the impact of methods on the results of quantitative synthesis and MA conclusions. Future research studies must also focus on the interpretations of SR results. 28 , 32 Finally, most of the included SRs were conducted on specific topics related to the field of health care, limiting the generalizability of the findings to other areas. It is important that future research studies evaluating evidence syntheses broaden the objectives and include studies on different topics within the field of health care.

4.3. Strengths and limitations

To our knowledge, this is the first overview summarizing current evidence from SRs and MA on different methodological approaches used in several fundamental steps in SR conduct. The overview methodology followed well established guidelines and strict criteria defined for the inclusion of SRs.

There are several limitations related to the nature of the included reviews. Evidence for most of the methods investigated in the included reviews was derived from a limited number of primary studies. Also, the majority of the included SRs may be considered outdated as they were published (or last updated) more than 5 years ago 33 ; only three of the nine SRs have been published in the last 5 years. 14 , 25 , 26 Therefore, important and recent evidence related to these topics may not have been included. Substantial numbers of included SRs were conducted in the field of health, which may limit the generalizability of the findings. Some method evaluations in the included SRs focused on quantitative analyses components and MA conclusions only. As such, the applicability of these findings to SR more broadly is still unclear. 28 Considering the methodological nature of our overview, limiting the inclusion of SRs according to the Cochrane criteria might have resulted in missing some relevant evidence from those reviews without a quality assessment component. 12 , 13 , 29 Although the included SRs performed some form of quality appraisal of the included studies, most of them did not use a standardized RoB tool, which may impact the confidence in their conclusions. Due to the type of outcome measures used for the method evaluations in the primary studies and the included SRs, some of the identified methods have not been validated against a reference standard.

Some limitations in the overview process must be noted. While our literature search was exhaustive covering five bibliographic databases and supplementary search of reference lists, no gray sources or other evidence resources were searched. Also, the search was primarily conducted in health databases, which might have resulted in missing SRs published in other fields. Moreover, only English language SRs were included for feasibility. As the literature search retrieved large number of citations (i.e., 41,556), the title and abstract screening was performed by a single reviewer, calibrated for consistency in the screening process by another reviewer, owing to time and resource limitations. These might have potentially resulted in some errors when retrieving and selecting relevant SRs. The SR methods were grouped based on key elements of each recommended SR step, as agreed by the authors. This categorization pertains to the identified set of methods and should be considered subjective.

5. CONCLUSIONS

This overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process. Limited evidence was also identified on some methodological modifications currently used to expedite the SR process. Overall, findings highlight the dearth of SRs on SR methodologies, warranting further work to confirm several current recommendations on conventional and expedited SR processes.

CONFLICT OF INTEREST

The authors declare no conflicts of interest.

Supporting information

APPENDIX A: Detailed search strategies

ACKNOWLEDGMENTS

The first author is supported by a La Trobe University Full Fee Research Scholarship and a Graduate Research Scholarship.

Open Access Funding provided by La Trobe University.

Veginadu P, Calache H, Gussy M, Pandian A, Masood M. An overview of methodological approaches in systematic reviews . J Evid Based Med . 2022; 15 :39–54. 10.1111/jebm.12468 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]

IMAGES

The Systematic Review Process
Introduction to systematic reviews
Systematic review protocols and registration
How to Conduct a Systematic Review
Systematic reviews
A Step by Step Guide for Conducting a Systematic Review

VIDEO

Everything about Systematic Review| Meaning| Steps
Pathways to Work Evidence Clearinghouse: A Resource for TANF and Employment Programs
Statistical Procedure in Meta-Essentials
🎓Achieve Research Excellence with Manuscriptedit || Systematic Review & Meta Analysis Mastery 📚✨
INTRODUCTION TO SYSTEMATIC REVIEW AND RESEARCH PARADIGMS
Introduction to Systematic Review of Research

COMMENTS

Systematic Review
A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer. Example: Systematic review. In 2008, Dr. Robert Boyle and his colleagues published a systematic review in ...
Systematic reviews: Structure, form and content
Topic selection and planning. In recent years, there has been an explosion in the number of systematic reviews conducted and published (Chalmers & Fox 2016, Fontelo & Liu 2018, Page et al 2015) - although a systematic review may be an inappropriate or unnecessary research methodology for answering many research questions.Systematic reviews can be inadvisable for a variety of reasons.
Systematic review
A systematic review is a scholarly synthesis of the evidence on a clearly presented topic using critical methods to identify, define and assess research on the topic. [1] A systematic review extracts and interprets data from published studies on the topic (in the scientific literature), then analyzes, describes, critically appraises and summarizes interpretations into a refined evidence-based ...
Introduction to systematic review and meta-analysis
It is easy to confuse systematic reviews and meta-analyses. A systematic review is an objective, reproducible method to find answers to a certain research question, by collecting all available studies related to that question and reviewing and analyzing their results. A meta-analysis differs from a systematic review in that it uses statistical ...
What is a Systematic Review?
an explicit, reproducible methodology. a systematic search that attempts to identify all studies that would meet the eligibility criteria. an assessment of the validity of the findings of the included studies, for example through the assessment of the risk of bias. a systematic presentation, and synthesis, of the characteristics and findings of ...
How to Do a Systematic Review: A Best Practice Guide for ...
Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question.
Systematic Review
A systematic review is a type of review that uses repeatable methods to find, select, and synthesise all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer. Example: Systematic review. In 2008, Dr Robert Boyle and his colleagues published a systematic review in ...
Systematic Reviews and Meta-Analysis: A Guide for Beginners
Systematic reviews involve the application of scientific methods to reduce bias in review of literature. The key components of a systematic review are a well-defined research question, comprehensive literature search to identify all studies that potentially address the question, systematic assembly of the studies that answer the question, critical appraisal of the methodological quality of the ...
Introduction to Systematic Reviews
A systematic review identifies and synthesizes all relevant studies that fit prespecified criteria to answer a research question (Lasserson et al. 2019; IOM 2011).What sets a systematic review apart from a narrative review is that it follows consistent, rigorous, and transparent methods established in a protocol in order to minimize bias and errors.
Getting Started
A systematic review is guided filtering and synthesis of all available evidence addressing a specific, focused research question, generally about a specific intervention or exposure. The use of standardized, systematic methods and pre-selected eligibility criteria reduce the risk of bias in identifying, selecting and analyzing relevant studies.
What are systematic reviews?
Watch on. Cochrane evidence, including our systematic reviews, provides a powerful tool to enhance your healthcare knowledge and decision making. This video from Cochrane Sweden explains a bit about how we create health evidence and what Cochrane does. About Cochrane.
Systematic reviews: Structure, form and content
In recent years, there has been an explosion in the number of systematic reviews conducted and published (Chalmers & Fox 2016, Fontelo & Liu 2018, Page et al 2015) - although a systematic review may be an inappropriate or unnecessary research methodology for answering many research questions.Systematic reviews can be inadvisable for a variety of reasons.
1.2.2 What is a systematic review?
A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made (Antman ...
Guides: Systematic Review: What is a systematic review?
A systematic review is an authoritative account of existing evidence using reliable, objective, thorough and reproducible research practices. It is a method of making sense of large bodies of information and contributes to the answers to questions about what works and what doesn't. Systematic reviews map areas of uncertainty and identify where ...
What are systematic reviews?
Sometimes systematic reviews ask a broad research question, and one strategy to achieve this is the use of several focussed sub-questions each addressed by sub-components of the review. Another strategy is to develop a map to describe the type of research that has been undertaken in relation to a research question. Some maps even describe over ...
What is a Systematic Review (SR)?
Systematic Reviews in the Social Sciences by Roberts, H., & Petticrew, M. Such diverse thinkers as Lao-Tze, Confucius, and U.S. Defense Secretary Donald Rumsfeld have all pointed out that we need to be able to tell the difference between real and assumed knowledge. The systematic review is a scientific tool that can help with this difficult task.
What is a systematic review?
A high-quality systematic review is described as the most reliable source of evidence to guide clinical practice. The purpose of a systematic review is to deliver a meticulous summary of all the available primary research in response to a research question. A systematic review uses all the existing research and is sometime called 'secondary research' (research on research).
How to Do a Systematic Review: A Best Practice Guide ...
Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to ...
Systematic Reviews and Meta-analysis: Understanding the Best Evidence
Systematic review and meta-analysis is a way of summarizing research evidence, which is generally the best form of evidence, and hence positioned at the top of the hierarchy of evidence. Systematic reviews can be very useful decision-making tools for primary care/family physicians.
CDC Library
Systematic Reviews. Describes what is involved with conducting a systematic review of the literature for evidence-based public health and how the librarian is a partner in the process.
Systematic Reviews: What is a systematic review?
Literature reviews, also known as narrative reviews, attempt to find all published materials on a subject, whereas systematic reviews try to find everything that focuses on answering a specific question. Since systematic reviews are generally associated with health related fields, their main objective is to ensure the results of the review ...
Assessing the Certainty of the Evidence in Systematic Reviews
The certainty of the evidence is specific to each outcome in a systematic review, and can be rated as high, moderate, low, or very low. Because it will have an important impact, before conducting certainty of evidence, reviewers must clarify the intent of their question: are they interested in causation or association.
Something old, new, and borrowed . Rise of the systematic reviews
Systematic reviews and other types of literature reviews are more prevalent in clinical medicine than in other fields. The recurring need for improvement and updates in these disciplines has led to the Living Systematic Review (LSR) concept to enhance the effectiveness of scientific synthesis efforts. While LSR was introduced in 2014, its adoption outside clinical medicine has been limited ...
LSBU Library: Literature Reviews: What is a Literature Review?
Integrative Reviews: Similar to systematic reviews, but they aim to generate new knowledge by integrating findings from different studies to develop new theories or frameworks. Importance of Literature Reviews. Foundation for Research: They provide a solid background for new research projects, helping to justify the research question and ...
Topical Anti‐Inflammatory Treatments for Eczema: A Cochrane Systematic
Funding: This systematic review and network meta-analysis was funded by the National Institute for Health and Care Research (NIHR) through a Research for Patient Benefit grant to Dr Robert Boyle (NIHR201993) and a Systematic Review Programme Grant to Cochrane Skin at the Centre of Evidence Based Dermatology. The views expressed are those of the authors and not necessarily those of the NIHR or ...
Mobile phones are not linked to brain cancer, according to a major
This new systematic review of human observational studies is based on a much larger data set compared to what the IARC examined in 2011. It includes more recent and more comprehensive studies.
Study designs: Part 7
Study designs: Part 7 - Systematic reviews. In this series on research study designs, we have so far looked at different types of primary research designs which attempt to answer a specific question. In this segment, we discuss systematic review, which is a study design used to summarize the results of several primary research studies.
Identifying the Causes and Effects of Decision Fatigue through a
The aim of this systematic review is to investigate the causes and effects of decision fatigue from the existing literature that can be generalized across different organizational domains. A comprehensive literature search in three databases identified 589 articles on decision fatigue.
Group schema therapy for personality disorders: Systematic review
Objective:There are significant temporal and financial barriers for individuals with personality disorders (PD) receiving evidence-based psychological treatments. Emerging research indicates Group Schema Therapy (GST) may be an accessible, efficient, and cost-effective PD intervention, however, there has been no synthesis of the available evidence to date. This review therefore aimed to ...
An overview of methodological approaches in systematic reviews
1. INTRODUCTION. Evidence synthesis is a prerequisite for knowledge translation. 1 A well conducted systematic review (SR), often in conjunction with meta‐analyses (MA) when appropriate, is considered the "gold standard" of methods for synthesizing evidence related to a topic of interest. 2 The central strength of an SR is the transparency of the methods used to systematically search ...

Have a language expert improve your writing

Systematic Review | Definition, Example & Guide

Table of contents

Receive feedback on language, structure, and formatting

Prevent plagiarism. Run a free check.

Step 1: Formulate a research question

Step 2: Develop a protocol

Step 3: Search for all relevant studies

Step 4: Apply the selection criteria

Step 5: Extract the data

Step 6: Synthesize the data

Step 7: Write and publish a report

Cite this Scribbr article

Is this article helpful?

Shaun Turney

Systematic Reviews

Key Characteristics of a Systematic Review:

Additional Information

Research Support

Save citation to file

Add to My Bibliography

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses

Similar articles

LinkOut - more resources

Other Literature Sources

Miscellaneous

Have a language expert improve your writing

Systematic Review | Definition, Examples & Guide

Table of contents

Prevent plagiarism, run a free check.

Step 1: Formulate a research question

Step 2: Develop a protocol

Step 3: Search for all relevant studies

Step 4: Apply the selection criteria

Step 5: Extract the data

Step 6: Synthesise the data

Step 7: Write and publish a report

Cite this Scribbr article

Is this article helpful?

Shaun Turney

Systematic Reviews and Meta Analysis

Systematic review Q & A

Is my research topic appropriate for systematic review methods?

If not a systematic review, then what?

Can my student complete a systematic review during her summer project?

How can I know if my topic has been been reviewed already?

What are systematic reviews?

1.2.2 What is a systematic review?

Systematic Review

Characteristics of a systematic review

UCL LIBRARY SERVICES

What are systematic reviews?

On this page

Useful books about systematic reviews

Systematic approaches to a successful literature review

An introduction to systematic reviews

Cochrane handbook for systematic reviews of interventions

Finding what works in health care: Standards for systematic reviews

Systematic Reviews in the Social Sciences

Research Synthesis and Meta-Analysis

Doing a Systematic Review

Systematic Review

Introduction to Systematic Review

Scoping Review

Qualitative Systematic Review/Meta-Synthesis

Log in using your username and password

You are here

Statistics from Altmetric.com

Read the full text or download the PDF:

Annual Review of Psychology

Most Read This Month

Systematic Reviews

Systematic Reviews: What is a systematic review?

What is a Systematic Review?

Why use systematic reviews to answer questions?

Why perform a systematic review?

How to find a systematic review

Types of systematic reviews

Subject Guide

Article Contents