objectives of literature review in research methodology

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

The objective of a literature review

Questions to Consider

B. In some fields or contexts, a literature review is referred to as the introduction or the background; why is this true, and does it matter?

The elements of a literature review • The first step in scholarly research is determining the “state of the art” on a topic. This is accomplished by gathering academic research and making sense of it. • The academic literature can be found in scholarly books and journals; the goal is to discover recurring themes, find the latest data, and identify any missing pieces. • The resulting literature review organizes the research in such a way that tells a story about the topic or issue.

The literature review tells a story in which one well-paraphrased summary from a relevant source contributes to and connects with the next in a logical manner, developing and fulfilling the message of the author. It includes analysis of the arguments from the literature, as well as revealing consistent and inconsistent findings. How do varying author insights differ from or conform to previous arguments?

Language in Action

A. How are the terms “critique” and “review” used in everyday life? How are they used in an academic context?

In terms of content, a literature review is intended to:

• Set up a theoretical framework for further research • Show a clear understanding of the key concepts/studies/models related to the topic • Demonstrate knowledge about the history of the research area and any related controversies • Clarify significant definitions and terminology • Develop a space in the existing work for new research

The literature consists of the published works that document a scholarly conversation or progression on a problem or topic in a field of study. Among these are documents that explain the background and show the loose ends in the established research on which a proposed project is based. Although a literature review focuses on primary, peer -reviewed resources, it may begin with background subject information generally found in secondary and tertiary sources such as books and encyclopedias. Following that essential overview, the seminal literature of the field is explored. As a result, while a literature review may consist of research articles tightly focused on a topic with secondary and tertiary sources used more sparingly, all three types of information (primary, secondary, tertiary) are critical.

The literature review, often referred to as the Background or Introduction to a research paper that presents methods, materials, results and discussion, exists in every field and serves many functions in research writing.

Adapted from Frederiksen, L., & Phelps, S. F. (2017). Literature Reviews for Education and Nursing Graduate Students. Open Textbook Library

Review and Reinforce

Two common approaches are simply outlined here. Which seems more common? Which more productive? Why? A. Forward exploration 1. Sources on a topic or problem are gathered. 2. Salient themes are discovered. 3. Research gaps are considered for future research. B. Backward exploration 1. Sources pertaining to an existing research project are gathered. 2. The justification of the research project’s methods or materials are explained and supported based on previously documented research.

Media Attributions

• 2589960988_3eeca91ba4_o © Untitled blue is licensed under a CC BY (Attribution) license

Sourcing, summarizing, and synthesizing:  Skills for effective research writing  Copyright © 2023 by Wendy L. McBride is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Get science-backed answers as you write with Paperpal's Research feature

What is a Literature Review? How to Write It (with Examples)

A literature review is a critical analysis and synthesis of existing research on a particular topic. It provides an overview of the current state of knowledge, identifies gaps, and highlights key findings in the literature. 1 The purpose of a literature review is to situate your own research within the context of existing scholarship, demonstrating your understanding of the topic and showing how your work contributes to the ongoing conversation in the field. Learning how to write a literature review is a critical tool for successful research. Your ability to summarize and synthesize prior research pertaining to a certain topic demonstrates your grasp on the topic of study, and assists in the learning process. 

Table of Contents

• What is the purpose of literature review? 
• a. Habitat Loss and Species Extinction: 
• b. Range Shifts and Phenological Changes: 
• c. Ocean Acidification and Coral Reefs: 
• d. Adaptive Strategies and Conservation Efforts: 
• How to write a good literature review 
• Choose a Topic and Define the Research Question: 
• Decide on the Scope of Your Review: 
• Select Databases for Searches: 
• Conduct Searches and Keep Track: 
• Review the Literature: 
• Organize and Write Your Literature Review: 
• Frequently asked questions 

What is a literature review?

A well-conducted literature review demonstrates the researcher’s familiarity with the existing literature, establishes the context for their own research, and contributes to scholarly conversations on the topic. One of the purposes of a literature review is also to help researchers avoid duplicating previous work and ensure that their research is informed by and builds upon the existing body of knowledge.

What is the purpose of literature review?

A literature review serves several important purposes within academic and research contexts. Here are some key objectives and functions of a literature review: 2  

• Contextualizing the Research Problem: The literature review provides a background and context for the research problem under investigation. It helps to situate the study within the existing body of knowledge. 
• Identifying Gaps in Knowledge: By identifying gaps, contradictions, or areas requiring further research, the researcher can shape the research question and justify the significance of the study. This is crucial for ensuring that the new research contributes something novel to the field. 
• Understanding Theoretical and Conceptual Frameworks: Literature reviews help researchers gain an understanding of the theoretical and conceptual frameworks used in previous studies. This aids in the development of a theoretical framework for the current research. 
• Providing Methodological Insights: Another purpose of literature reviews is that it allows researchers to learn about the methodologies employed in previous studies. This can help in choosing appropriate research methods for the current study and avoiding pitfalls that others may have encountered. 
• Establishing Credibility: A well-conducted literature review demonstrates the researcher’s familiarity with existing scholarship, establishing their credibility and expertise in the field. It also helps in building a solid foundation for the new research. 
• Informing Hypotheses or Research Questions: The literature review guides the formulation of hypotheses or research questions by highlighting relevant findings and areas of uncertainty in existing literature. 

Literature review example

Let’s delve deeper with a literature review example: Let’s say your literature review is about the impact of climate change on biodiversity. You might format your literature review into sections such as the effects of climate change on habitat loss and species extinction, phenological changes, and marine biodiversity. Each section would then summarize and analyze relevant studies in those areas, highlighting key findings and identifying gaps in the research. The review would conclude by emphasizing the need for further research on specific aspects of the relationship between climate change and biodiversity. The following literature review template provides a glimpse into the recommended literature review structure and content, demonstrating how research findings are organized around specific themes within a broader topic. 

Literature Review on Climate Change Impacts on Biodiversity:

Climate change is a global phenomenon with far-reaching consequences, including significant impacts on biodiversity. This literature review synthesizes key findings from various studies: 

a. Habitat Loss and Species Extinction:

Climate change-induced alterations in temperature and precipitation patterns contribute to habitat loss, affecting numerous species (Thomas et al., 2004). The review discusses how these changes increase the risk of extinction, particularly for species with specific habitat requirements. 

b. Range Shifts and Phenological Changes:

Observations of range shifts and changes in the timing of biological events (phenology) are documented in response to changing climatic conditions (Parmesan & Yohe, 2003). These shifts affect ecosystems and may lead to mismatches between species and their resources. 

c. Ocean Acidification and Coral Reefs:

The review explores the impact of climate change on marine biodiversity, emphasizing ocean acidification’s threat to coral reefs (Hoegh-Guldberg et al., 2007). Changes in pH levels negatively affect coral calcification, disrupting the delicate balance of marine ecosystems. 

d. Adaptive Strategies and Conservation Efforts:

Recognizing the urgency of the situation, the literature review discusses various adaptive strategies adopted by species and conservation efforts aimed at mitigating the impacts of climate change on biodiversity (Hannah et al., 2007). It emphasizes the importance of interdisciplinary approaches for effective conservation planning. 

How to write a good literature review

Writing a literature review involves summarizing and synthesizing existing research on a particular topic. A good literature review format should include the following elements. 

Introduction: The introduction sets the stage for your literature review, providing context and introducing the main focus of your review. 

• Opening Statement: Begin with a general statement about the broader topic and its significance in the field. 
• Scope and Purpose: Clearly define the scope of your literature review. Explain the specific research question or objective you aim to address. 
• Organizational Framework: Briefly outline the structure of your literature review, indicating how you will categorize and discuss the existing research. 
• Significance of the Study: Highlight why your literature review is important and how it contributes to the understanding of the chosen topic. 
• Thesis Statement: Conclude the introduction with a concise thesis statement that outlines the main argument or perspective you will develop in the body of the literature review. 

Body: The body of the literature review is where you provide a comprehensive analysis of existing literature, grouping studies based on themes, methodologies, or other relevant criteria. 

• Organize by Theme or Concept: Group studies that share common themes, concepts, or methodologies. Discuss each theme or concept in detail, summarizing key findings and identifying gaps or areas of disagreement. 
• Critical Analysis: Evaluate the strengths and weaknesses of each study. Discuss the methodologies used, the quality of evidence, and the overall contribution of each work to the understanding of the topic. 
• Synthesis of Findings: Synthesize the information from different studies to highlight trends, patterns, or areas of consensus in the literature. 
• Identification of Gaps: Discuss any gaps or limitations in the existing research and explain how your review contributes to filling these gaps. 
• Transition between Sections: Provide smooth transitions between different themes or concepts to maintain the flow of your literature review. 

Conclusion: The conclusion of your literature review should summarize the main findings, highlight the contributions of the review, and suggest avenues for future research. 

• Summary of Key Findings: Recap the main findings from the literature and restate how they contribute to your research question or objective. 
• Contributions to the Field: Discuss the overall contribution of your literature review to the existing knowledge in the field. 
• Implications and Applications: Explore the practical implications of the findings and suggest how they might impact future research or practice. 
• Recommendations for Future Research: Identify areas that require further investigation and propose potential directions for future research in the field. 
• Final Thoughts: Conclude with a final reflection on the importance of your literature review and its relevance to the broader academic community. 

Conducting a literature review

Conducting a literature review is an essential step in research that involves reviewing and analyzing existing literature on a specific topic. It’s important to know how to do a literature review effectively, so here are the steps to follow: 1  

Choose a Topic and Define the Research Question:

• Select a topic that is relevant to your field of study. 
• Clearly define your research question or objective. Determine what specific aspect of the topic do you want to explore? 

Decide on the Scope of Your Review:

• Determine the timeframe for your literature review. Are you focusing on recent developments, or do you want a historical overview? 
• Consider the geographical scope. Is your review global, or are you focusing on a specific region? 
• Define the inclusion and exclusion criteria. What types of sources will you include? Are there specific types of studies or publications you will exclude? 

Select Databases for Searches:

• Identify relevant databases for your field. Examples include PubMed, IEEE Xplore, Scopus, Web of Science, and Google Scholar. 
• Consider searching in library catalogs, institutional repositories, and specialized databases related to your topic. 

Conduct Searches and Keep Track:

• Develop a systematic search strategy using keywords, Boolean operators (AND, OR, NOT), and other search techniques. 
• Record and document your search strategy for transparency and replicability. 
• Keep track of the articles, including publication details, abstracts, and links. Use citation management tools like EndNote, Zotero, or Mendeley to organize your references. 

Review the Literature:

• Evaluate the relevance and quality of each source. Consider the methodology, sample size, and results of studies. 
• Organize the literature by themes or key concepts. Identify patterns, trends, and gaps in the existing research. 
• Summarize key findings and arguments from each source. Compare and contrast different perspectives. 
• Identify areas where there is a consensus in the literature and where there are conflicting opinions. 
• Provide critical analysis and synthesis of the literature. What are the strengths and weaknesses of existing research? 

Organize and Write Your Literature Review:

• Literature review outline should be based on themes, chronological order, or methodological approaches. 
• Write a clear and coherent narrative that synthesizes the information gathered. 
• Use proper citations for each source and ensure consistency in your citation style (APA, MLA, Chicago, etc.). 
• Conclude your literature review by summarizing key findings, identifying gaps, and suggesting areas for future research. 

The literature review sample and detailed advice on writing and conducting a review will help you produce a well-structured report. But remember that a literature review is an ongoing process, and it may be necessary to revisit and update it as your research progresses. 

Frequently asked questions

A literature review is a critical and comprehensive analysis of existing literature (published and unpublished works) on a specific topic or research question and provides a synthesis of the current state of knowledge in a particular field. A well-conducted literature review is crucial for researchers to build upon existing knowledge, avoid duplication of efforts, and contribute to the advancement of their field. It also helps researchers situate their work within a broader context and facilitates the development of a sound theoretical and conceptual framework for their studies.

Literature review is a crucial component of research writing, providing a solid background for a research paper’s investigation. The aim is to keep professionals up to date by providing an understanding of ongoing developments within a specific field, including research methods, and experimental techniques used in that field, and present that knowledge in the form of a written report. Also, the depth and breadth of the literature review emphasizes the credibility of the scholar in his or her field.  

Before writing a literature review, it’s essential to undertake several preparatory steps to ensure that your review is well-researched, organized, and focused. This includes choosing a topic of general interest to you and doing exploratory research on that topic, writing an annotated bibliography, and noting major points, especially those that relate to the position you have taken on the topic. 

Literature reviews and academic research papers are essential components of scholarly work but serve different purposes within the academic realm. 3 A literature review aims to provide a foundation for understanding the current state of research on a particular topic, identify gaps or controversies, and lay the groundwork for future research. Therefore, it draws heavily from existing academic sources, including books, journal articles, and other scholarly publications. In contrast, an academic research paper aims to present new knowledge, contribute to the academic discourse, and advance the understanding of a specific research question. Therefore, it involves a mix of existing literature (in the introduction and literature review sections) and original data or findings obtained through research methods. 

Literature reviews are essential components of academic and research papers, and various strategies can be employed to conduct them effectively. If you want to know how to write a literature review for a research paper, here are four common approaches that are often used by researchers.  Chronological Review: This strategy involves organizing the literature based on the chronological order of publication. It helps to trace the development of a topic over time, showing how ideas, theories, and research have evolved.  Thematic Review: Thematic reviews focus on identifying and analyzing themes or topics that cut across different studies. Instead of organizing the literature chronologically, it is grouped by key themes or concepts, allowing for a comprehensive exploration of various aspects of the topic.  Methodological Review: This strategy involves organizing the literature based on the research methods employed in different studies. It helps to highlight the strengths and weaknesses of various methodologies and allows the reader to evaluate the reliability and validity of the research findings.  Theoretical Review: A theoretical review examines the literature based on the theoretical frameworks used in different studies. This approach helps to identify the key theories that have been applied to the topic and assess their contributions to the understanding of the subject.  It’s important to note that these strategies are not mutually exclusive, and a literature review may combine elements of more than one approach. The choice of strategy depends on the research question, the nature of the literature available, and the goals of the review. Additionally, other strategies, such as integrative reviews or systematic reviews, may be employed depending on the specific requirements of the research.

The literature review format can vary depending on the specific publication guidelines. However, there are some common elements and structures that are often followed. Here is a general guideline for the format of a literature review:  Introduction:   Provide an overview of the topic.  Define the scope and purpose of the literature review.  State the research question or objective.  Body:   Organize the literature by themes, concepts, or chronology.  Critically analyze and evaluate each source.  Discuss the strengths and weaknesses of the studies.  Highlight any methodological limitations or biases.  Identify patterns, connections, or contradictions in the existing research.  Conclusion:   Summarize the key points discussed in the literature review.  Highlight the research gap.  Address the research question or objective stated in the introduction.  Highlight the contributions of the review and suggest directions for future research.

Both annotated bibliographies and literature reviews involve the examination of scholarly sources. While annotated bibliographies focus on individual sources with brief annotations, literature reviews provide a more in-depth, integrated, and comprehensive analysis of existing literature on a specific topic. The key differences are as follows: 

References 

• Denney, A. S., & Tewksbury, R. (2013). How to write a literature review.  Journal of criminal justice education ,  24 (2), 218-234. 
• Pan, M. L. (2016).  Preparing literature reviews: Qualitative and quantitative approaches . Taylor & Francis. 
• Cantero, C. (2019). How to write a literature review.  San José State University Writing Center . 

Paperpal is an AI writing assistant that help academics write better, faster with real-time suggestions for in-depth language and grammar correction. Trained on millions of research manuscripts enhanced by professional academic editors, Paperpal delivers human precision at machine speed.  

Try it for free or upgrade to  Paperpal Prime , which unlocks unlimited access to premium features like academic translation, paraphrasing, contextual synonyms, consistency checks and more. It’s like always having a professional academic editor by your side! Go beyond limitations and experience the future of academic writing.  Get Paperpal Prime now at just US$19 a month!

Related Reads:

• Empirical Research: A Comprehensive Guide for Academics 
• How to Write a Scientific Paper in 10 Steps 
• Life Sciences Papers: 9 Tips for Authors Writing in Biological Sciences
• What is an Argumentative Essay? How to Write It (With Examples)

6 Tips for Post-Doc Researchers to Take Their Career to the Next Level

Self-plagiarism in research: what it is and how to avoid it, you may also like, measuring academic success: definition & strategies for excellence, what is academic writing: tips for students, why traditional editorial process needs an upgrade, paperpal’s new ai research finder empowers authors to..., what is hedging in academic writing  , how to use ai to enhance your college..., ai + human expertise – a paradigm shift..., how to use paperpal to generate emails &..., ai in education: it’s time to change the..., is it ethical to use ai-generated abstracts without....

Research Methods

• Getting Started
• Literature Review Research
• Research Design
• Research Design By Discipline
• SAGE Research Methods
• Teaching with SAGE Research Methods

Literature Review

• What is a Literature Review?
• What is NOT a Literature Review?
• Purposes of a Literature Review
• Types of Literature Reviews
• Literature Reviews vs. Systematic Reviews
• Systematic vs. Meta-Analysis

Literature Review  is a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works.

Also, we can define a literature review as the collected body of scholarly works related to a topic:

• Summarizes and analyzes previous research relevant to a topic
• Includes scholarly books and articles published in academic journals
• Can be an specific scholarly paper or a section in a research paper

The objective of a Literature Review is to find previous published scholarly works relevant to an specific topic

• Help gather ideas or information
• Keep up to date in current trends and findings
• Help develop new questions

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Helps focus your own research questions or problems
• Discovers relationships between research studies/ideas.
• Suggests unexplored ideas or populations
• Identifies major themes, concepts, and researchers on a topic.
• Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
• Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches.
• Indicates potential directions for future research.

All content in this section is from Literature Review Research from Old Dominion University 

Keep in mind the following, a literature review is NOT:

Not an essay 

Not an annotated bibliography  in which you summarize each article that you have reviewed.  A literature review goes beyond basic summarizing to focus on the critical analysis of the reviewed works and their relationship to your research question.

Not a research paper   where you select resources to support one side of an issue versus another.  A lit review should explain and consider all sides of an argument in order to avoid bias, and areas of agreement and disagreement should be highlighted.

A literature review serves several purposes. For example, it

• provides thorough knowledge of previous studies; introduces seminal works.
• helps focus one’s own research topic.
• identifies a conceptual framework for one’s own research questions or problems; indicates potential directions for future research.
• suggests previously unused or underused methodologies, designs, quantitative and qualitative strategies.
• identifies gaps in previous studies; identifies flawed methodologies and/or theoretical approaches; avoids replication of mistakes.
• helps the researcher avoid repetition of earlier research.
• suggests unexplored populations.
• determines whether past studies agree or disagree; identifies controversy in the literature.
• tests assumptions; may help counter preconceived ideas and remove unconscious bias.

As Kennedy (2007) notes*, it is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the original studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally that become part of the lore of field. In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews.

Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are several approaches to how they can be done, depending upon the type of analysis underpinning your study. Listed below are definitions of types of literature reviews:

Argumentative Review      This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to to make summary claims of the sort found in systematic reviews.

Integrative Review      Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.

Historical Review      Few things rest in isolation from historical precedent. Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review      A review does not always focus on what someone said [content], but how they said it [method of analysis]. This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection and data analysis, and helps highlight many ethical issues which we should be aware of and consider as we go through our study.

Systematic Review      This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyse data from the studies that are included in the review. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?"

Theoretical Review      The purpose of this form is to concretely examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review help establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

* Kennedy, Mary M. "Defining a Literature."  Educational Researcher  36 (April 2007): 139-147.

All content in this section is from The Literature Review created by Dr. Robert Larabee USC

Robinson, P. and Lowe, J. (2015),  Literature reviews vs systematic reviews.  Australian and New Zealand Journal of Public Health, 39: 103-103. doi: 10.1111/1753-6405.12393

What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters . By Lynn Kysh from University of Southern California

Systematic review or meta-analysis?

A  systematic review  answers a defined research question by collecting and summarizing all empirical evidence that fits pre-specified eligibility criteria.

A  meta-analysis  is the use of statistical methods to summarize the results of these studies.

Systematic reviews, just like other research articles, can be of varying quality. They are a significant piece of work (the Centre for Reviews and Dissemination at York estimates that a team will take 9-24 months), and to be useful to other researchers and practitioners they should have:

• clearly stated objectives with pre-defined eligibility criteria for studies
• explicit, reproducible methodology
• a systematic search that attempts to identify all studies
• assessment of the validity of the findings of the included studies (e.g. risk of bias)
• systematic presentation, and synthesis, of the characteristics and findings of the included studies

Not all systematic reviews contain meta-analysis. 

Meta-analysis is the use of statistical methods to summarize the results of independent studies. By combining information from all relevant studies, meta-analysis can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review.  More information on meta-analyses can be found in  Cochrane Handbook, Chapter 9 .

A meta-analysis goes beyond critique and integration and conducts secondary statistical analysis on the outcomes of similar studies.  It is a systematic review that uses quantitative methods to synthesize and summarize the results.

An advantage of a meta-analysis is the ability to be completely objective in evaluating research findings.  Not all topics, however, have sufficient research evidence to allow a meta-analysis to be conducted.  In that case, an integrative review is an appropriate strategy. 

Some of the content in this section is from Systematic reviews and meta-analyses: step by step guide created by Kate McAllister.

• << Previous: Getting Started
• Next: Research Design >>
• Last Updated: Aug 21, 2023 4:07 PM
• URL: https://guides.lib.udel.edu/researchmethods

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Dissertation
• What is a Literature Review? | Guide, Template, & Examples

What is a Literature Review? | Guide, Template, & Examples

Published on 22 February 2022 by Shona McCombes . Revised on 7 June 2022.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research.

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarise sources – it analyses, synthesises, and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Why write a literature review, examples of literature reviews, step 1: search for relevant literature, step 2: evaluate and select sources, step 3: identify themes, debates and gaps, step 4: outline your literature review’s structure, step 5: write your literature review, frequently asked questions about literature reviews, introduction.

• Quick Run-through
• Step 1 & 2

When you write a dissertation or thesis, you will have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and scholarly context
• Develop a theoretical framework and methodology for your research
• Position yourself in relation to other researchers and theorists
• Show how your dissertation addresses a gap or contributes to a debate

You might also have to write a literature review as a stand-alone assignment. In this case, the purpose is to evaluate the current state of research and demonstrate your knowledge of scholarly debates around a topic.

The content will look slightly different in each case, but the process of conducting a literature review follows the same steps. We’ve written a step-by-step guide that you can follow below.

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research objectives and questions .

If you are writing a literature review as a stand-alone assignment, you will have to choose a focus and develop a central question to direct your search. Unlike a dissertation research question, this question has to be answerable without collecting original data. You should be able to answer it based only on a review of existing publications.

Make a list of keywords

Start by creating a list of keywords related to your research topic. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list if you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can use boolean operators to help narrow down your search:

Read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

To identify the most important publications on your topic, take note of recurring citations. If the same authors, books or articles keep appearing in your reading, make sure to seek them out.

You probably won’t be able to read absolutely everything that has been written on the topic – you’ll have to evaluate which sources are most relevant to your questions.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models and methods? Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• How does the publication contribute to your understanding of the topic? What are its key insights and arguments?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible, and make sure you read any landmark studies and major theories in your field of research.

You can find out how many times an article has been cited on Google Scholar – a high citation count means the article has been influential in the field, and should certainly be included in your literature review.

The scope of your review will depend on your topic and discipline: in the sciences you usually only review recent literature, but in the humanities you might take a long historical perspective (for example, to trace how a concept has changed in meaning over time).

Remember that you can use our template to summarise and evaluate sources you’re thinking about using!

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It’s important to keep track of your sources with references to avoid plagiarism . It can be helpful to make an annotated bibliography, where you compile full reference information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

You can use our free APA Reference Generator for quick, correct, consistent citations.

Prevent plagiarism, run a free check.

To begin organising your literature review’s argument and structure, you need to understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly-visual platforms like Instagram and Snapchat – this is a gap that you could address in your own research.

There are various approaches to organising the body of a literature review. You should have a rough idea of your strategy before you start writing.

Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarising sources in order.

Try to analyse patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organise your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text, your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

If you are writing the literature review as part of your dissertation or thesis, reiterate your central problem or research question and give a brief summary of the scholarly context. You can emphasise the timeliness of the topic (“many recent studies have focused on the problem of x”) or highlight a gap in the literature (“while there has been much research on x, few researchers have taken y into consideration”).

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, make sure to follow these tips:

• Summarise and synthesise: give an overview of the main points of each source and combine them into a coherent whole.
• Analyse and interpret: don’t just paraphrase other researchers – add your own interpretations, discussing the significance of findings in relation to the literature as a whole.
• Critically evaluate: mention the strengths and weaknesses of your sources.
• Write in well-structured paragraphs: use transitions and topic sentences to draw connections, comparisons and contrasts.

In the conclusion, you should summarise the key findings you have taken from the literature and emphasise their significance.

If the literature review is part of your dissertation or thesis, reiterate how your research addresses gaps and contributes new knowledge, or discuss how you have drawn on existing theories and methods to build a framework for your research. This can lead directly into your methodology section.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your  dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2022, June 07). What is a Literature Review? | Guide, Template, & Examples. Scribbr. Retrieved 22 April 2024, from https://www.scribbr.co.uk/thesis-dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, how to write a dissertation proposal | a step-by-step guide, what is a theoretical framework | a step-by-step guide, what is a research methodology | steps & tips.

Objectives and Positioning of [Systematic] Literature Reviews

• First Online: 11 August 2022

Cite this chapter

• Rob Dekkers 4 ,
• Lindsey Carey 5 &
• Peter Langhorne 6  

1872 Accesses

When undertaking literature reviews, it is important to know why you have to do these and how. In 2012 a doctoral student told that his supervisor had advised for the literature review: ‘Just do it!’ Whereas this advice holds true to a certain extent, it does not provide a true starting point for appropriate and thorough literature reviews. Better options for starting are to know how literature reviews are embedded in research processes and how they are related to different approaches for the design of research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Some call it also ‘critiquing review’, which is slightly confusing. A case in point is Xiao and Watson ( 2017 , p. 10), although the source they refer to, i.e. Paré et al. ( 2015 ), does not use the word ‘critique’ in any manner.

The term ‘research paradigm’ instead of ‘research philosophy’ has been adopted in this book, following the thoughts of Kuhn ( 1962 , p. 23), who describes a paradigm as ‘accepted model or pattern’, a reflection of a particular discourse and a philosophical position relating to the nature of social phenomena and social structures. In this sense, a paradigm directs research efforts, it serves to reassert itself to the exclusion of other paradigms and to articulate the theories it already established. Paradigms could be interpreted as prescriptive and as requiring particular research methods and excluding others. Feilzer ( 2010 , p. 7) derives from Kuhn ( 1962 , p. 24) and Mills ( 1959 ), a paradigm can constrain intellectual curiosity and creativity, blind researchers to aspects of social phenomena, or even new phenomena and theories, and limit sociological imagination.

The terms constructivism and interpretivism are often confused. In this writing constructivism is taken as gravitating towards an ontological perspective and interpretivism as an epistomological perspective for this research paradigm.

At the time of writing this book, the editors of the journal Innovations. Journal of Innovation Economics & Management have lodged a formal objection to the publishers of the second publication. This note is written with the sole purpose to indicate how different types of plagiarism may occur.

Balabkins NW (2010) Joseph A. Schumpeter: not guilty of plagiarism but of “Infelicities of Attribution”. In: Aronson JR, Parmet HL, Thornton RJ (eds) Variations in economic analysis—essays in Honor of Eli Schwartz. Springer, New York

Google Scholar  

Bergdahl E (2019) Is meta-synthesis turning rich descriptions into thin reductions? A criticism of meta-aggregation as a form of qualitative synthesis. Nurs Inq 26(1):e12273. https://doi.org/10.1111/nin.12273

Article   Google Scholar  

Boughzala Y (2020) Vers une approche collective de l’innovation sociale: le rôle joué par l’entrepreneuriat social en Tunisie. [Towards a collective approach of social innovation: the case of the social entrepreneurship in Tunisia]. Innovations 62(2):161–189. https://doi.org/10.3917/inno.062.0161

Creswell JW (2014) Research design: qualitative, quantitative, and mixed methods approaches, 4th edn. Sage, Los Angeles

Dekkers R (2017) Applied systems theory, 2nd edn. Springer, Cham

Book   Google Scholar  

Eisenhardt KM (1989) Building theories from case study research. Acad Manag Rev 14(4):532–550. https://doi.org/10.5465/amr.1989.4308385

Feilzer MY (2010) Doing mixed methods research pragmatically: implications for the rediscovery of pragmatism as a research paradigm. J Mixed Methods Res 4(1):6–16. https://doi.org/10.1177/1558689809349691

Fridhi B (2021) Social entrepreneurship and social enterprise phenomenon: toward a collective approach to social innovation in Tunisia. J Innov Entrepren 10(1):14. https://doi.org/10.1186/s13731-021-00148-6

Fusar-Poli P, Radua J (2018) Ten simple rules for conducting umbrella reviews. Evid Based Mental Health 21(3):95–100. https://doi.org/10.1136/ebmental-2018-300014

Grant S, Mayo-Wilson E, Hopewell S, Macdonald G, Moher D, Montgomery P (2013a) Developing a reporting guideline for social and psychological intervention trials. J Exp Criminol 9(3):355–367. https://doi.org/10.1007/s11292-013-9180-5

Grant S, Montgomery P, Hopewell S, Macdonald G, Moher D, Mayo-Wilson E (2013b)Developing a reporting guideline for social and psychological intervention trials. Res Soc Work Pract 23(6):595–602. https://doi.org/10.1177/1049731513498118

Green BN, Johnson CD, Adams A (2006) Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J Chiropr Med 5(3):101–117. https://doi.org/10.1016/S0899-3467(07)60142-6

Hartley J (2003) Improving the clarity of journal abstracts in psychology: the case for structure. Sci Commun 24(3):366–379. https://doi.org/10.1177/1075547002250301

Kuhn TS (1962) The structure of scientific revolutions. University of Chicago Press, Chicago

Kühnle H, Dekkers R (2012) Some thoughts on interdisciplinarity in collaborative networks’ research and manufacturing sciences. J Manuf Technol Manag 23(8):961–975. https://doi.org/10.1108/17410381211276826

Mayo-Wilson E, Grant S, Hopewell S, Macdonald G, Moher D, Montgomery P (2013a) Developing a reporting guideline for social and psychological intervention trials. Trials 14(1):242. https://doi.org/10.1186/1745-6215-14-242

Mayo-Wilson E, Montgomery P, Hopewell S, Macdonald G, Moher D, Grant S (2013b) Developing a reporting guideline for social and psychological intervention trials. Br J Psychiatry 203(4):250–254. https://doi.org/10.1192/bjp.bp.112.123745

Mills CW (1959) The sociological imagination. Oxford University Press, New York

Montgomery P, Mayo-Wilson E, Hopewell S, Macdonald G, Moher D, Grant S (2013) Developing a reporting guideline for social and psychological intervention trials. Am J Public Health 103(10):1741–1746. https://doi.org/10.2105/ajph.2013.301447

Newman MEJ (2003) The structure and function of complex networks. SIAM Rev 45(2):167–256

Paré G, Trudel M-C, Jaana M, Kitsiou S (2015) Synthesizing information systems knowledge: a typology of literature reviews. Inform Manag 52(2):183–199. https://doi.org/10.1016/j.im.2014.08.008

Pugh S (1981) Concept selection: a method that works. Paper presented at the international conference on engineering design, Rome

Reinert H, Reinert ES (2006) Creative destruction in economics: Nietzsche, Sombart, Schumpeter. In: Backhaus JG, Drechsler W (eds) Friedrich Nietzsche (1844–1900): economy and society. Springer, New York, NY, pp 55–85

Chapter   Google Scholar  

Runyan WM (1982) In defense of the case study method. Am J Orthopsychiatry 52(3):440–446. https://doi.org/10.1111/j.1939-0025.1982.tb01430.x

Salgado EG, Dekkers R (2018) Lean product development: nothing new under the sun? Int J Manag Rev 20(4):903–933. https://doi.org/10.1111/ijmr.12169

Strauss AC, Corbin J (1998) Basics of qualitative research techniques and procedures for developing grounded theory. Sage, London

Wallace M, Wray A (2006) Critical reading and writing for postgraduates. Sage Publications, London

Ward A, Liker JK, Cristiano JJ, Sobek DK II (1995) The second Toyota paradox: how delaying decisions can make better cars faster. Sloan Manag Rev 36(3):43–61

Xiao Y, Watson M (2017) Guidance on conducting a systematic literature review. J Plan Educ Res 39(1):93–112. https://doi.org/10.1177/0739456X17723971

Download references

Author information

Authors and affiliations.

University of Glasgow, Glasgow, UK

Rob Dekkers

Glasgow Caledonian University, Glasgow, UK

Lindsey Carey

Peter Langhorne

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Rob Dekkers .

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this chapter

Dekkers, R., Carey, L., Langhorne, P. (2022). Objectives and Positioning of [Systematic] Literature Reviews. In: Making Literature Reviews Work: A Multidisciplinary Guide to Systematic Approaches. Springer, Cham. https://doi.org/10.1007/978-3-030-90025-0_2

Download citation

DOI : https://doi.org/10.1007/978-3-030-90025-0_2

Published : 11 August 2022

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-90024-3

Online ISBN : 978-3-030-90025-0

eBook Packages : Education Education (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 5. The Literature Review
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

• Give a new interpretation of old material or combine new with old interpretations,
• Trace the intellectual progression of the field, including major debates,
• Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
• Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

• Place each work in the context of its contribution to understanding the research problem being studied.
• Describe the relationship of each work to the others under consideration.
• Identify new ways to interpret prior research.
• Reveal any gaps that exist in the literature.
• Resolve conflicts amongst seemingly contradictory previous studies.
• Identify areas of prior scholarship to prevent duplication of effort.
• Point the way in fulfilling a need for additional research.
• Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE : Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews."  Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I.  Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

• Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness -- which of the author's theses are most convincing or least convincing?
• Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  Development of the Literature Review

Four Basic Stages of Writing 1.  Problem formulation -- which topic or field is being examined and what are its component issues? 2.  Literature search -- finding materials relevant to the subject being explored. 3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1.  Roughly how many sources would be appropriate to include? 2.  What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3.  Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4.  Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5.  Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

• Current Situation : Information necessary to understand the current topic or focus of the literature review.
• Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
• History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
• Standards : Description of the way in which you present your information.
• Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

• Sources in your literature review do not clearly relate to the research problem;
• You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
• Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
• Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
• Does not describe the search procedures that were used in identifying the literature to review;
• Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
• Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

• Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research?  Does it forge a new path? Or, does is merely add more of the same thing being said?
• Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
• Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

• << Previous: Theoretical Framework
• Next: Citation Tracking >>
• Last Updated: Apr 24, 2024 10:51 AM
• URL: https://libguides.usc.edu/writingguide

Types of Literature Review

There are many types of literature review. The choice of a specific type depends on your research approach and design. The following types of literature review are the most popular in business studies:

Narrative literature review , also referred to as traditional literature review, critiques literature and summarizes the body of a literature. Narrative review also draws conclusions about the topic and identifies gaps or inconsistencies in a body of knowledge. You need to have a sufficiently focused research question to conduct a narrative literature review

Systematic literature review requires more rigorous and well-defined approach compared to most other types of literature review. Systematic literature review is comprehensive and details the timeframe within which the literature was selected. Systematic literature review can be divided into two categories: meta-analysis and meta-synthesis.

When you conduct meta-analysis you take findings from several studies on the same subject and analyze these using standardized statistical procedures. In meta-analysis patterns and relationships are detected and conclusions are drawn. Meta-analysis is associated with deductive research approach.

Meta-synthesis, on the other hand, is based on non-statistical techniques. This technique integrates, evaluates and interprets findings of multiple qualitative research studies. Meta-synthesis literature review is conducted usually when following inductive research approach.

Scoping literature review , as implied by its name is used to identify the scope or coverage of a body of literature on a given topic. It has been noted that “scoping reviews are useful for examining emerging evidence when it is still unclear what other, more specific questions can be posed and valuably addressed by a more precise systematic review.” [1] The main difference between systematic and scoping types of literature review is that, systematic literature review is conducted to find answer to more specific research questions, whereas scoping literature review is conducted to explore more general research question.

Argumentative literature review , as the name implies, examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. It should be noted that a potential for bias is a major shortcoming associated with argumentative literature review.

Integrative literature review reviews , critiques, and synthesizes secondary data about research topic in an integrated way such that new frameworks and perspectives on the topic are generated. If your research does not involve primary data collection and data analysis, then using integrative literature review will be your only option.

Theoretical literature review focuses on a pool of theory that has accumulated in regard to an issue, concept, theory, phenomena. Theoretical literature reviews play an instrumental role in establishing what theories already exist, the relationships between them, to what degree existing theories have been investigated, and to develop new hypotheses to be tested.

At the earlier parts of the literature review chapter, you need to specify the type of your literature review your chose and justify your choice. Your choice of a specific type of literature review should be based upon your research area, research problem and research methods.  Also, you can briefly discuss other most popular types of literature review mentioned above, to illustrate your awareness of them.

[1] Munn, A. et. al. (2018) “Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach” BMC Medical Research Methodology

  John Dudovskiy

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 05 March 2024

A systematic review of the methodology of trade-off analysis in agriculture

• Timo S. Breure   ORCID: orcid.org/0000-0001-5695-8064 1 ,
• Natalia Estrada-Carmona   ORCID: orcid.org/0000-0003-4329-5470 2 ,
• Athanasios Petsakos   ORCID: orcid.org/0000-0003-0224-4087 3 ,
• Elisabetta Gotor   ORCID: orcid.org/0000-0003-0533-3077 3 ,
• Boris Jansen   ORCID: orcid.org/0000-0002-4493-1734 4 &
• Jeroen C. J. Groot   ORCID: orcid.org/0000-0001-6516-5170 1  

Nature Food volume  5 ,  pages 211–220 ( 2024 ) Cite this article

4781 Accesses

33 Altmetric

Metrics details

• Agroecology
• Ecological modelling
• Ecosystem services
• Environmental impact
• Sustainability

Trade-off analysis (TOA) is central to policy and decision-making aimed at promoting sustainable agricultural landscapes. Yet, a generic methodological framework to assess trade-offs in agriculture is absent, largely due to the wide range of research disciplines and objectives for which TOA is used. In this study, we systematically reviewed 119 studies that have implemented TOAs in landscapes and regions dominated by agricultural systems around the world. Our results highlight that TOAs tend to be unbalanced, with a strong emphasis on productivity rather than environmental and socio-cultural services. TOAs have mostly been performed at farm or regional scales, rarely considering multiple spatial scales simultaneously. Mostly, TOAs fail to include stakeholders at study development stage, disregard recommendation uncertainty due to outcome variability and overlook risks associated with the TOA outcomes. Increased attention to these aspects is critical for TOAs to guide agricultural landscapes towards sustainability.

Similar content being viewed by others

Long-term evidence for ecological intensification as a pathway to sustainable agriculture

An interactive model to assess pathways for agriculture and food sector contributions to country-level net-zero targets

Ecological sensitivity within human realities concept for improved functional biodiversity outcomes in agricultural systems and landscapes

Contemporary agriculture should not only provide food, fibre, feed and fuel but also environmental and socio-economic benefits for rural communities and beyond 1 . To ensure that agriculture delivers multiple services while minimizing its negative impacts, society must be aware of the trade-offs and synergies that may arise. The nature of these trade-offs depends on location-specific natural, social and cultural conditions that place constraints on the inputs and outputs of an agricultural system. For example, market-based farmers are concerned with enhancing commodity production, whereas the priority of subsistence farmers lies with improving food security 2 . The global imperative to achieve the United Nations Sustainable Development Goals (SDGs) underscores the need to reduce the environmental impact of land use practices and strengthen equitable social outcomes at both landscape and community levels. However, achieving the SDGs might require sacrifices to primary productivity and economic revenues. Thus, to reconcile the demands of agriculture and inform decision-making, an analysis is required of potential trade-offs measured against agronomic, environmental, economic and social indicators 3 .

Trade-off analysis (TOA) was established as a concept to generate quantitative information on competing (trade-offs) or complementary (synergies) indicators that can be used to guide policy and decision-making 4 . A typical TOA project starts with three preparatory steps: formulation of the research question, identification of which indicators to assess, and formulation of hypotheses about the relationships between the indicators and the associated trade-offs and synergies. Subsequently, the management, policy or technological changes that affect the TOA indicators can be identified and included in the analysis framework. Then, the trade-offs and synergies under changing conditions or scenarios can be quantified and, finally, the results are communicated to relevant stakeholders to inform decision-making and policy 4 . Since its first implementation in the context of agriculture, a wide range of methods have been used to conduct TOAs, including optimization, simulations, qualitative, econometric and narrative-based approaches. In some cases, these approaches are deployed in a spatially explicit manner with the support of geographic information systems (GIS) 5 .

Although important advances have been made regarding TOA in agricultural contexts, researchers have expressed concerns about the scope and methodological limitations of published studies. These concerns relate to the limited transfer of the academic knowledge generated by TOA into decision- and policy-making due to the inability to take into account social and cultural factors 6 , the sparsity of multi- and cross-scale assessments 3 , 5 , 6 , 7 , and the limited representation of uncertainty 8 , 9 and risk analysis 5 .

The concerns reported in the literature on the limitations of TOA analysis can indeed have important implications. First, failure to recognize the importance of scale (spatial, temporal, jurisdictional and legislative) in TOA may lead to erroneous inferences on how the relationships between trade-offs and indicators develop across scales. Multiple scales can be analysed without interactions between them or a cross-scale analysis can be performed that accounts for interactions between scales 10 . Furthermore, adverse effects appearing outside the TOA case study area (off-site effects) may offset any gains stemming from a TOA-informed policy 11 . Second, recognition of social interactions and cultural values is needed to assure representation of beneficiaries and non-beneficiaries relevant to the topic at hand, that is, distributional justice 9 , 10 . Representation among stakeholders and their involvement in the design and implementation of a TOA can increase the legitimacy of its findings, assure that the data used are relevant to the context and thus enhance adoption of a study’s findings 12 . Third, validation and acknowledgement of uncertainty in both data and model estimates increase the robustness of a TOA and can facilitate risk-based decision-making 13 , 14 , 15 .

Previous literature reviews on TOA in agriculture adopted a ‘storytelling’ approach, where key studies were selected from the literature to discuss research trends. However, given the wide scope of TOAs applied in the context of agriculture, a systematic review could reveal the variety of approaches used and potential knowledge gaps, as well as the indicators that were studied and by which methods, ultimately facilitating the comparability of results.

Here we report on the TOA indicators, methodology and analysis used in 119 peer-reviewed articles. Descriptive statistics are used to characterize articles based on the extent to which they considered (1) indicators relevant to environmental and socio-economic services, (2) multiple spatial scales and their interactions, (3) the comprehensive involvement of stakeholders, and (4) the validity of trade-offs and recommendations in the context of associated uncertainties and risks (see Table 1 for further details). Finally, a cluster analysis shows which indicators were frequently studied together and which TOA methods were associated with each cluster.

The aim of this study was thus to provide an overview of the peer-reviewed literature on TOA in the context of agriculture using a systematic approach. For this purpose, we sought to define how trade-offs in agriculture are conceptualized, characterized and analysed in the TOA literature. Based on these findings, we have identified common gaps in the implementation of TOA.

The distribution of publication dates for the articles in the sample was mainly centred in the years 2015–2021 (Extended Data Fig. 1a ). Specifically, 73% of the articles were published after 2014, which indicates an increasing research effort directed towards TOAs in an agricultural context (Extended Data Fig. 1b ).

Common interrelationships and co-occurrences among TOA indicators

The articles examined included a median of 3.8 ± 1.9 (s.d.) TOA indicators, ranging from 1 to 10. Based on the cumulative distribution, 52% of the articles included three or fewer TOA indicators, while 90% included six or fewer TOA indicators (Extended Data Fig. 2a ). The most prevalent indicators across all articles were ‘profitability’ (57%, economic), ‘yield’ (44%, agronomic) and ‘water quantity’ (34%, sustainable resource management). The second most common set of indicators encompassed a selection of biophysical (for example, ‘water quality’ and ‘greenhouse gases’), agronomic (for example, ‘input efficiency’ and ‘land use efficiency’) and economic indicators (for example, ‘assets’), ranging between 13% and 21% (Fig. 1 ). The remaining TOA indicators were used less frequently and related to economic (that is, ‘labour productivity’ and ‘poverty’), human health (for example, ‘nutrition’, ‘health’ or ‘food security’) and agronomic (that is, ‘self-sufficiency’) aspects, representing a share of 5–6% (Fig. 1 ). Rarely considered TOA indicators (less than 5%) included ‘market supply or demand’ (economic), ‘yield stability’ (agronomic), ‘empowerment’ and ‘gender equity’ (both human health; Fig. 1 ).

Percentage of articles that include a TOA indicator (black dotted line and circles) and the share of each TOA method M1–M9 used to study that indicator (coloured bars). The prefixes of the TOA indicators refer to their class association (A, E, H, S) and number of occurrence within that class as provided in Table 1 . Table 1 also describes the TOA methods M1–M9.

The articles were grouped into 11 clusters, depending on which TOA indicators were assessed (left y -axis dendrogram in Fig. 2 ). These clusters show a dominant theme based on the co-occurrence of TOA indicators (right y- axis in Fig. 2 ). For example, in cluster 7, ‘poverty’ was studied in conjunction with ‘soil nutrients’, whereas in cluster 5, ‘poverty’ was studied in conjunction with ‘profitability’, ‘food security’ and ‘nutrition’. The clustering of articles by TOA indicator reveals which TOA indicators are often studied together. Indicators of ‘profitability’ and ‘yield’ were the most commonly used (Figs. 1 and 2 ) and were generally combined with case-specific environmental and social indicators (Fig. 2 ). This suggests that agronomic and economic viability are conditional for the exploration of improvements in agricultural system sustainability. The cluster with the largest number of articles (cluster 6, Fig. 2 ) concerned agricultural production and water quality. This highlights the strong focus on solving pressing issues related to pollution by surplus nutrients from fertilizers and manure.

The articles were clustered by TOA indicator (row-wise) and TOA indicator clusters (column-wise). The associations of articles with clusters are indicated by the colours and labels on the left of the figure; the colours are arbitrary. TOA indicator clusters (top x axis) are specified by colour, corresponding to the main indicator categories (legend in top left of the figure), and their name (bottom x axis). The matrix indicates whether a TOA indicator has been included in an article (red) or not (beige). The labels on the right list the main TOA indicators included in each cluster. GHG, greenhouse gases; SOC, soil organic carbon; supp./dem., supply or demand.

The clustering of TOA indicators (top x -axis dendrogram in Fig. 2 ) shows that for 50% of the indicators, the indicator closest in the dendrogram belongs to the same category (sustainable resource management, agronomic, economic or human health). In particular, four out of five human health indicators were studied in isolation from other indicators, forming closely paired branches (top x -axis dendrogram, orange colour, in Fig. 2 ).

The application of TOA methods varied across different TOA indicators and clusters. For example, the TOA indicators ‘labour productivity’, ‘empowerment’, ‘gender equity’ and ‘yield stability’ lacked cases involving spatially explicit methods (M1 or M8; Fig. 1 ). This same observation applies to the clusters in which these TOA indicators belong (Fig. 3 ). While the absence of spatially explicit methods for social indicators such as ‘empowerment’ and ‘gender equity’ is expected, given that their spatial dimension is often disregarded, it is worth noting that gender and empowerment may relate to the spatial distribution of fields and resources in the landscape. For instance, their distance from the location of the homestead or decision-making processes regarding the (distribution of) use and ownership of these resources. Clusters of articles associated with ‘yield’, ‘energy’, ‘biodiversity’ and ‘land use’ exhibited a high use of GIS (M8), qualitative (M6) and other (M9) methods, with fewer articles applying optimization methods (M3; Fig. 3 ). Lastly, an interesting anomaly is the ‘health’ indicator, where methods M1–M3, encompassing (spatially explicit) simulations and optimization methods, were conspicuously absent (Fig. 1 ).

Cluster associations are as per Fig. 2 and the number of articles within each cluster is given by n .

Frequency of criteria levels

The majority of TOAs were conducted at regional (65%) and farm (17%) scales, followed by field (7%) and national (6%) scales. The TOAs conducted at multi-country (4%) and global scales, along with ‘other’, accounted for only a small proportion of the analyses (Fig. 4a ). The spatial scales for TOAs differed from the scales at which modelling was performed or data were collected, with the farm and field scale contributing to a combined share of 48%. Of the articles considered, 12% implemented cross-scale analyses and 17% considered off-site effects (Fig. 4a ). Case study areas were predominantly delineated using administrative borders (54%), followed by biophysical delineation (24%), with 18% of the articles using both methods (Fig. 4b ).

a , Criteria related to the scale of the analysis. TOA: the spatial scale at which the TOA was conducted. The numbers refer to the spatial scales of field (1), farm (2), regional (3), national (4), multi-country (5) and global (6). Off-site: whether off-site effects have been considered in the TOA. Discipline: the spatial scale at which modelling or data collection was performed for a discipline. The numbers refer to the spatial scales detailed above for TOA. Cross-scale: whether aggregative (1), interactive (2) or no cross-scale modelling was performed (3). b , Criteria related to the TOA framework. TOA method: the methods used to perform the TOAs. The numbers refer to the TOA methods M1–M9 defined in Table 1 . System border: which boundaries were used to define the TOA case study area. Scenario: whether the article considered a scenario and, if so, which type of scenario. The numbers refer to the scenarios 1–8 defined in Table 1 . c , Criteria related to stakeholders. Type: whether local beneficiaries and non-beneficiaries, experts, government, farmers, distant beneficiaries and non-beneficiaries, academics, private organizations or environmental organizations were involved. Inclusion: whether the study included stakeholders. Implementation: whether stakeholders were involved in consultation, co-development, valuation or validation. d , Criteria related to TOA robustness: whether the article performed a validation, risk analysis or acknowledged uncertainty. e , The frequency (shown in the circles) for each spatial scale at which the modelling or data collection was performed for a given discipline. f , The frequency (shown in the circles) at which an article considered a given scenario in TOA for each spatial scale. The scenario numbers 1–8 are defined in Table 1 .

Including a scenario in the TOA allows investigation of the effect of a postulated event or driver on the TOA indicators. In our analysis, scenarios focusing on climate, behavioural or demographic change accounted for 14% of the articles, while scenarios involving alternative intensities of resource use constituted 37% of the articles. Scenarios were absent in 25% of the articles (Fig. 4b ). Over half of the articles included stakeholders in their analysis, with a relatively equal spread across stakeholder types, except for ‘distant beneficiaries and non-beneficiaries’, which were under-represented. Farmers and experts constituted a larger share (48%) compared to other categories (Fig. 4c ). Stakeholders were mainly involved in consultation and valuation, with co-development and validation implemented in less than 25% of the articles considered (Fig. 4c ). Overall, the robustness of the TOA results was not widely considered, as the criteria ‘uncertainty’ and ‘validation’ were logged for less than 50% of the articles. Articles incorporating risk analysis constituted 12% of the sample (Fig. 4d ).

Links between spatial scales and criteria

Of the articles considered, ‘livestock’, ‘fisheries’ and ‘forestry’ accounted for a relatively small share (16%) compared with ‘crop’, ‘economic’ and ‘environmental’ disciplines. For the livestock discipline, modelling and data collection were predominantly carried out at the farm scale, while for forestry, they were primarily conducted at the field or regional scale (Fig. 4e ). For the economic discipline, modelling and data collection were evenly distributed between the farm ( n  = 34) and regional ( n  = 34) scales (Fig. 4e ), in contrast to the overall share of these scales across all of the articles, where ‘regional’ constituted 65% and ‘farm’ constituted 17% of the articles (Fig. 4a ). In general, for a large share of the reviewed articles, data were collected and modelling was performed at the field and farm scales, but the TOA was conducted at the regional scale. These findings show that, before the TOA, some form of aggregation occurs in the majority of the reviewed articles. Regarding the spatial scale at which the TOA was conducted for articles including a scenario, two observations can be made. First, all of the scenarios (except the resource use scenario) were rarely studied at scales larger than the national scale. Second, the climate, behavioural and demographic change scenarios were almost exclusively studied at the regional scale (Fig. 4f ). These results show that few studies investigated how scenarios unfolding at smaller or larger scales affect the indicators at the TOA scale.

Multi-scale, cross-scale and robustness criteria

Figure 5 shows the percentage of articles that include a TOA indicator (black line, the same as shown in Fig. 1 ). The articles were then divided into subsets according to whether they included a cross-scale, multi-scale or robustness criterion. The coloured lines represent the percentage of articles in the subset that include a specific TOA indicator. With the exception of indicators rarely included in all articles (for example, those related to nutrition or health), most TOA indicators were present in articles adopting a cross-scale modelling framework (Fig. 5a ). These findings occur despite the overall low number of articles (<20%) reporting cross-scale analyses (Fig. 5a ). Notably, articles applying an interactive modelling framework did not include ‘water quality’, ‘soil erosion’, ‘soil organic carbon’ and ‘biodiversity’, despite these indicators having a relatively high frequency across all articles (Fig. 5a ).

The percentages of all reviewed articles and subsets of articles that include specific TOA indicators. a – c , The subsets comprise articles that included cross-scale ( a ), multi-scale ( b ) and robustness ( c ) criteria. In b , TOA refers to articles in which the TOA was conducted on multiple spatial scales, ‘Discipline’ refers to articles that considered multiple spatial scales for modelling or data collection, and ‘Off-site’ refers to articles in which effects outside the TOA case study area were considered.

Across all articles, 17% considered off-site effects (Fig. 4a ). Notably, the ‘poverty’ and ‘soil erosion’ indicators were under-represented in articles considering off-site effects (Fig. 5b ). Eight indicators were excluded in articles considering multiple spatial scales in modelling or data collection (‘discipline’ in Fig. 5b ). This finding is particularly striking for ‘biodiversity’, given that it constitutes a large share of spatially explicit TOA methods (Fig. 1 ).

Thirteen per cent of articles reported TOA on multiple spatial scales, with seven indicators excluded in these cases (‘TOA’ in Fig. 5b ). Among the excluded indicators, those related to human health dominated (except for ‘nutrition’). For certain indicators, these findings are to be expected. For instance, market supply or demand (economic) is irrelevant at low geographical scales (field and farm) as prices are determined at the regional (local), national or international scale. The articles that included a risk analysis showed stark contrasts between TOA indicators with respect to their representation relative to all articles. Economic and human health indicators were particularly over-represented, while ‘yield’, ‘input efficiency’ and a set of biophysical indicators were under-represented (Fig. 5c ). For articles in which uncertainty was acknowledged or validation was performed, no indicators were over- or under-represented relative to their inclusion across all articles (Fig. 5c ).

Limitations on the inclusion of TOA indicators

Recent reviews on TOA have stated that there is little to no representation of indicators related to social interactions, justice and gender issues in TOAs for agricultural systems 5 , 6 . These studies referred in particular to intra-household equity, asset ownership, health, education and nutrition. Our results also demonstrate that social and cultural TOA indicators are largely absent, mostly considered in isolation and studied by statistical approaches. These findings are probably a result of the limited data availability and the inability of TOA methods to include socio-cultural indicators for features and processes that are difficult to capture quantitatively 16 , 17 . We further note a similarly low frequency for the following indicators: food security, self-sufficiency and yield stability. These findings raise questions about the rationale behind the selection of TOA indicators. That is, the prevalent use of profitability and crop yield as primary indicators reflects the focus on profit and crop yield maximization in the literature 5 . The outcomes and priorities of a TOA depend on the chosen objectives and indicators. Alternative indicators might therefore facilitate a more comprehensive analysis of the delivery of environmental, economic and socio-cultural services from agriculture. One illustrative example is the metric ‘nutritional yield’, defined as “the number of hectares required to provide sufficient quantity to fulfil 100% of dietary reference intake for a nutrient for one adult” 2 . Nutritional yield thus allows the assessment of land use efficiency in both agronomic and social terms. Integrating nutritional yield into TOA in the context of subsistence agriculture could unveil the need for changes in farmers’ crop plans to balance food security and economic profitability objectives.

TOA methodologies

The formulation of research objectives, questions and methodology determines the information base that a TOA can provide 16 , 18 . Decisions regarding TOA objectives and methodology determine the degree to which scales, disciplines and indicators are compartmentalized. In addition, these decisions influence the range of interventions and scenarios explored for alternative agro-environmental management of land, resources and technologies 7 , 18 . The results of our analysis reveal associations between TOA methods and indicators, indicating common gaps, such as the absence of articles reporting the use of spatially explicit methods to study the indicators ‘human health’ and ‘yield stability’. Studying these indicators in a spatially explicit manner could allow for targeted land use planning at the local scale. For instance, Prestele and Verburg demonstrated that spatially explicit analysis of climate-smart agriculture adoptions unveils local-scale trade-offs affecting yield and soil carbon sequestration at an aggregated scale 19 . Our results also underscore expected patterns, with socio-economic indicators predominantly studied through statistical approaches and qualitative methods. These methods, static and based on existing datasets, differ from mechanistic models, which allow extrapolation and ex ante assessment under alternative future scenarios. Simulations based on mechanistic models hold the potential to explore scenarios that minimize trade-offs between indicators 3 , 7 . However, the validity of this kind of optimization depends on having sufficient understanding of relevant processes and feedbacks in the socio-environmental system 3 . For example, while crop models vary in their capacity to assess climate change impacts, they share common limitations, such as inadequate representation of low-intensity agricultural systems 20 . We found that a description of study limitations in the context of the TOA framework, for example, excluded aspects, was often absent. Ideally, models and associated uncertainties would be assessed in the design phase of the TOA. This could ensure the availability of adequate information for quantifying all desired parameters at the desired resolution, allowing the study to comprehensively represent the agricultural system. Such an approach is crucial to guide planning in future management decisions aligned with research objectives 17 .

Involvement of stakeholders and practical application of TOA results

One recurring concern in the literature is the frequent omission of stakeholders at the onset of the TOA, potentially limiting the practical application of TOA results 6 , 8 . Our findings partially support these concerns, given that co-development with stakeholders was observed in only 10% of the articles. However, making a definitive statement on equal representation among stakeholders proved challenging as there was generally an absence of a systematic inventory outlining the relevance of different stakeholders to the decision-making process based on their interests and influence 21 . Our analysis shows that farmers and experts were the primary stakeholders included in the articles. Nonetheless, the omission of distant beneficiaries and non-beneficiaries is noteworthy as they are likely to be relevant to the decision-making process in numerous cases, especially when off-site effects are considered in TOAs conducted on multiple scales.

Multi- and cross-scale analysis

Depending on the research objectives, the TOA literature underscores the importance of acknowledging processes across scales and including them in research 3 , 6 , 7 , 8 , 9 , 22 . In many of the articles, data were collected or modelling was performed at field and farm scales, yet the TOA was conducted at the regional scale. This highlights an opportunity for multi-scale TOA analysis, potentially enhancing the relevance of TOA studies to policy. For example, bilevel optimization is a promising approach to facilitating nested decision-making processes at different scales. In this approach, the solution at the higher level (for example, larger spatial scale) depends on the solution at the lower level (for example, smaller spatial scale). Bostian et al. demonstrated the application of this methodology in recognizing multiple spatial scales inherent to non-point pollution regulation 23 . However, the restricted application of cross-scale analysis in our sample (12%) shows the limited extent to which TOA in agriculture captures the hierarchical nature of social, cultural, environmental, economic and agronomic processes.

Furthermore, 17% of the articles considered effects outside the TOA case study area, considering off-site effects in a diverse array of subjects, including transnational emission permits, water trading and increased demand for scarce resources, anticipated to influence their shadow prices 24 , 25 , 26 . However, off-site effects might have feedbacks, such as dependencies between alternative production systems within a supply chain 27 . In such cases, the delineation of the system boundary must be considered in the context of these feedbacks to ensure their inclusion within the system. In cases where off-site effects do not have feedbacks, these can be classified as ‘teleconnections’, denoting processes whose cause and effect are widely separated 28 . A case in point is a study of the water quality of the Danube River, in which distant beneficiaries and non-beneficiaries, represented by an international committee, were considered in the TOA 29 . The results also show that climate, behavioural and demographic scenarios were rarely assessed at lower or higher scales (compared to the regional scale). This underscores that the extent to which these scales are relevant to TOA is understudied and merits further research. For example, generic methods, such as the carbon 30 or water 31 footprint, can provide a broad assessment of which off-site effects at larger scales are relevant to TOA outcomes. These approaches may facilitate the inclusion of underlying causes, the involvement of more inclusive stakeholders and account for leakage effects, such as the expansion of agricultural lands beyond the TOA case study area 32 .

Ideally, a TOA methodological framework is conceptualized such that (1) it recognizes multi- and cross-scale interactions where applicable, (2) the system boundary aligns with substantiated biophysical and relevant socio-institutional boundaries, and (3) it recognizes the heterogeneity in which scales and associated consequences are perceived as well as valued by different stakeholders 10 .

Robustness of TOA results

The risk associated with TOA extends across spatial, temporal and jurisdictional scales, carrying implications for the dissemination of TOA results 13 . The under-representation of ‘yield’ in articles considering risk analysis highlights the dichotomy between yield and profitability as the most prominent indicators. That is, risk analysis appears to be mainly associated with the economic domain 5 . However, it is important to recognize that the evaluation of risk and the formulation of relevant strategies (risk aversion, mitigation or offsetting) are critical for farmers adopting system transformations, such as alternative forms of land use to mitigate inputs and associated greenhouse gas emissions. Integrating risk into TOA enables the study of the policies and incentives necessary for achieving whole-system transformations towards sustainable agricultural practices 13 , 14 . Decision-making under uncertainty becomes interpretable when recommendations are accompanied by an assessment of associated risks. Ideally, these risks are context-specific. For example, Hochman et al. provided TOA results on crop rotations alongside a minimum risk threshold quantified as the highest gross margin for the poorest 20% of years 33 .

While a moderate number of the articles considered uncertainty, only a few articles quantified changes in trade-offs as a function of uncertainty. The inclusion of stochastic components and the associated uncertainty inherent in biological systems could facilitate a more realistic description of outcomes, proving valuable for decision-making 13 , 15 . Varying input data or model parameterization within an expected range could reveal the sensitivity of results. For instance, when climate scenarios are used, realizations of these scenarios can be used to assess the stochasticity of the objectives for which the TOA is implemented 34 . This approach enables the acknowledgement of both the frequency and pattern of stochastic events, including extreme weather events, and their impact on TOA outcomes. Consequently, an analysis of the adaptability of a farming system would not solely rely on optimal solutions given the mean output but would also account for associated variability and unexpected events 15 . However, it is crucial to contextualize the effect of stochasticity. For example, the relative impact of model or parameter uncertainty on optimization outcomes has been shown to vary depending on the prioritization of objectives and site conditions 35 .

Limitations of this study

An important limitation of our review lies in the use of ‘trade-off analysis’ as a single term in our Web of Science search string. There are research areas that address trade-offs and synergies across various disciplines, scales and methods without explicitly using the term ‘trade-off analysis’ to describe their research objectives. Examples include the ‘food–energy–water nexus’ literature 36 , as well as research under the auspices of the Agricultural Model Intercomparison and Improvement Project (AgMIP) ( https://agmip.org/ ) and the Food, Agriculture, Biodiversity, Land-Use and Energy (FABLE) Consortium 37 . Both AgMIP and FABLE are particularly concerned with the relevance of TOA to policy. AgMIP explicitly states the use of “multiple scenarios and models to assess and probabilistically manage risk” 38 . Given the focus of these studies on global and regional assessments, we anticipate that our findings for those spatial scales could be affected. Indeed, the identified gaps in TOA implementation need to be viewed in the context of our sample, which mostly comprises studies in which modelling or data analysis was performed up to the regional level and TOA at the regional scale.

The method used to log the occurrence of pre-set criteria not only affects the variance within a criterion but also influences its abundance. For example, Sanon et al. included a large number of TOA indicators that were all classified under ‘biodiversity’ 29 . Thus, binary criteria logging does not capture the intensity with which a criterion is considered, a well-known phenomenon in the field of ecology 39 . This limitation may have resulted in the underestimation of both the intensity with which certain TOA indicators and their classes have been studied (Fig. 1 ) and the total number of TOA indicators considered per article (Extended Data Fig. 2 ).

Conclusions

Based on our analysis, it is possible to identify some actions that would increase the contribution of TOAs to SDG-aligned agricultural landscapes.

For instance, future studies should include multi- and cross-scale effects when relevant to the research objectives. We have identified an opportunity for multi-scale analysis, given that many studies aggregated farm- or field-scale data before performing TOA at a regional scale. As the inclusion of multiple scales, indicators and methods may in some cases reduce the generalizability of results and make them more context-specific, an alternative would be to discuss the anticipated implications of multi- and cross-scale effects on the study findings.

Furthermore, the relevance of TOA to society and policy can be improved by formulating research objectives such that TOA indicators lie within the scope of frameworks such as the SDGs. The most frequent indicators were biophysical or informed by profit maximization theory (for example, profitability and yield). However, indicators relevant to human well-being, security and farm resilience (for example, empowerment, nutrition and yield stability) occurred less frequently. To aid the interpretation of TOA results, the rationale behind the TOA methodology that is used to assess indicators should be listed together with a critical review of how the agricultural system under study is represented and what is excluded as a consequence.

In the reviewed articles, the most consulted stakeholders were farmers and experts, stakeholder co-development and validation were rare, and scenarios were predominantly based on resource use with little consideration of off-site effects. These findings suggest that TOAs mostly explore alternative management across a set of farms rather than policies and incentives that would facilitate whole landscape and food system transformations.

Agricultural policy- and decision-making carry an inherent risk. TOAs will become more operational when they evaluate associated risks and list strategies to manage these risks. This process could promote the robustness of quantified trade-offs with respect to the associated uncertainty of data and variability in outcomes. Finally, an inventory of stakeholders that are relevant to the decision-making process and their respective roles in the study would provide legitimacy of results. While this element has already been recognized in the literature 12 , 29 , some of the shortcomings that we have identified here would probably occur less frequently, particularly the lack of stakeholder inclusion and the over-representation of specific stakeholder types and methods of stakeholder engagement.

Closer adherence to these guidelines could enhance the relevance of TOA to the scientific community, policy-makers and farmers.

We followed the approach of Lautenbach et al. and Seppelt et al. in their systematic review of the literature on ecosystem services 9 , 22 . The generic structure involved (1) the identification, screening and selection of relevant peer-reviewed literature from a global repository, (2) formulation of the criteria against which to evaluate each article (Table 1 and Supplementary Table 1 ), and (3) descriptive statistics and cluster analysis to assess common interrelationships between criteria and identify knowledge gaps.

We used the following search string “ALL=agricultur* AND (“trade off* analysis” OR “trade-off* analysis” OR “tradeoff* analysis”)” in the Web of Science (on 14 September 2021) to identify peer-reviewed articles in English reporting TOA. We found 153 articles with publication dates spanning from 1993 to 2021. We excluded studies that mentioned the existence of trade-offs but did not assess relationships between indicators. For this reason, review and opinion papers were considered off-topic and were excluded from the search results. Furthermore, methodological papers that did not involve a case study were also excluded, leading to a total sample of 119 articles.

We selected criteria based on current TOA research 5 , 6 , 7 , 8 , 9 , 16 , 22 and recorded information on these criteria that were relevant to the conceptualization, characterization and analysis of trade-offs in agriculture (research objective 1). Briefly, the criteria included the type of TOA methods used, the spatial scales at which the analyses were performed and/or data collected, the indicators assessed in the TOA, which stakeholder types were included as well as how the stakeholders were engaged in the case study, whether the case study included alternative scenarios and of what type, how the case study area was delineated, whether effects outside the case study area were considered, and whether the case study acknowledged and accounted for uncertainty, validated results or performed a risk analysis. To assess whether cross-scale analyses were performed in case studies, we adopted the definition of Kanter et al., who distinguished between model frameworks that aggregate outputs at lower scales to use as inputs at higher scales (aggregative) and model frameworks that have submodels operating at different spatial and temporal resolutions (interactive) 6 . Thus, whereas an aggregative model framework follows a sequential approach, an interactive model framework performs analysis across scales simultaneously, allowing for interactions between scales and emergent indicators at higher levels. Furthermore, descriptive information was collected for three criteria: the agricultural system(s) studied, agricultural activities and knowledge gaps reported in the discussion section of the article. All of the criteria are listed in Table 1 with a generic description. We refer the reader to Supplementary Table 1 for more detailed information on the criteria. Based on these criteria, knowledge gaps were then assessed through descriptive statistics and cluster analysis (research objective 2).

The decision of which TOA indicators to include is a major methodological decision in TOA as it determines which interrelations are considered and analysed, and therefore which trade-offs and synergies can be identified. We anticipated thematic clusters of TOA indicators based on the discipline, scale, geography and method considered. To identify co-occurrences between TOA indicators, we performed hierarchical Ward clustering to group articles by TOA indicators as well as the TOA indicators themselves based on the Jaccard similarity coefficient 40 . Through the use of the Jaccard similarity metric, we accounted for the double-zero problem. Namely, the absence of a TOA indicator in two articles does not indicate a similarity, whereas its presence does 9 . For the clustering of articles by the TOA indicators used, the number of clusters to be retained was decided by the ‘elbow’ method based on the Mantel correlation between the data for each cluster and the raw distance matrix 40 . For the clustering of TOA indicators, the dendrogram was not cut to visualize common co-occurrences for all of the TOA indicators.

Criteria were logged in a Microsoft Office Excel (2021) spreadsheet (Supplementary Data 1 ). The data collected during this systematic review were further analysed and visualized in R (ref. 41 ). Data handling, visualizations and analysis were performed using the following R packages: tidyverse 42 , dendextend 43 , cluster 44 , vegan 45 and pheatmap 46 .

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Data availability

The dataset created has been made available as extended data.

Code availability

The code created for data handling, analysis and visualizations is available on request.

Renting, H. et al. Exploring multifunctional agriculture. A review of conceptual approaches and prospects for an integrative transitional framework. J. Environ. Manag. 90 , 112–123 (2009).

Article   Google Scholar  

DeFries, R. et al. Synergies and trade-offs for sustainable agriculture: nutritional yields and climate-resilience for cereal crops in Central India. Glob. Food Sec. 11 , 44–53 (2016).

Seppelt, R., Lautenbach, S. & Volk, M. Identifying trade-offs between ecosystem services, land use, and biodiversity: a plea for combining scenario analysis and optimization on different spatial scales. Curr. Opin. Environ. Sustain. 5 , 458–463 (2013).

Stoorvogel, J. J., Antle, J. M., Crissman, C. C. & Bowen, W. The tradeoff analysis model: integrated bio-physical and economic modeling of agricultural production systems. Agric. Syst. 80 , 43–66 (2004).

Antle, J. M. & Valdivia, R. O. Trade-off analysis of agri-food systems for sustainable research and development. Q Open https://doi.org/10.1093/qopen/qoaa005 (2021).

Kanter, D. R. et al. Evaluating agricultural trade-offs in the age of sustainable development. Agric. Syst. 163 , 73–88 (2018).

Groot, J. C. J. et al. On the contribution of modelling to multifunctional agriculture: learning from comparisons. J. Environ. Manag. 90 , 147–160 (2009).

Klapwijk, C. J. et al. Analysis of trade-offs in agricultural systems: current status and way forward. Curr. Opin. Environ. Sustain. 6 , 110–115 (2014).

Lautenbach, S. et al. Blind spots in ecosystem services research and challenges for implementation. Reg. Environ. Change 19 , 2151–2172 (2019).

Cash, D. W. et al. Scale and cross-scale dynamics: governance and information in a multilevel world. Ecol. Soc . https://www.jstor.org/stable/26265993 (2006).

Gibson, R. B. Sustainability assessment: basic components of a practical approach. Impact Assess. Proj. Apprais. 24 , 170–182 (2006).

Cash, D. W. et al. Knowledge systems for sustainable development. Proc. Natl Acad. Sci. USA 100 , 8086–8091 (2003).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Anderson, J. R. Risk in rural development: challenges for managers and policy makers. Agric. Syst. 75 , 161–197 (2003).

Hardaker, J. B., Lien, G., Anderson, J. R. & Huirne, R. B. Coping with Risk in Agriculture: Applied Decision Analysis (Cabi, 2015).

Uusitalo, L., Lehikoinen, A., Helle, I. & Myrberg, K. An overview of methods to evaluate uncertainty of deterministic models in decision support. Environ. Model. Softw. 63 , 24–31 (2015).

Thornton, P. K. et al. A framework for priority-setting in climate smart agriculture research. Agric. Syst. 167 , 161–175 (2018).

Jones, J. W. et al. Toward a new generation of agricultural system data, models, and knowledge products: state of agricultural systems science. Agric. Syst. 155 , 269–288 (2017).

Article   PubMed   PubMed Central   Google Scholar  

Morrison-Saunders, A. & Pope, J. Conceptualising and managing trade-offs in sustainability assessment. Environ. Impact Assess. Rev. 38 , 54–63 (2013).

Prestele, R. & Verburg, P. H. The overlooked spatial dimension of climate‐smart agriculture. Glob. Change Biol. 26 , 1045–1054 (2020).

Article   ADS   Google Scholar  

Ewert, F. et al. Crop modelling for integrated assessment of risk to food production from climate change. Environ. Model. Softw. 72 , 287–303 (2015).

Norström, A. V. et al. Principles for knowledge co-production in sustainability research. Nat. Sustain. 3 , 182–190 (2020).

Seppelt, R., Dormann, C. F., Eppink, F. V., Lautenbach, S. & Schmidt, S. A quantitative review of ecosystem service studies: approaches, shortcomings and the road ahead. J. Appl. Ecol. 48 , 630–636 (2011).

Bostian, M., Whittaker, G., Barnhart, B., Fare, R. & Grosskopf, S. Valuing water quality tradeoffs at different spatial scales: an integrated approach using bilevel optimization. Water Res. Econ. 11 , 1–12 (2015).

Google Scholar  

Popp, A. et al. The economic potential of bioenergy for climate change mitigation with special attention given to implications for the land system. Environ. Res. Lett . https://doi.org/10.1088/1748-9326/6/3/034017 (2011).

Hayha, T., Franzese, P. P., Paletto, A. & Fath, B. D. Assessing, valuing, and mapping ecosystem services in Alpine forests. Ecosyst. Serv. 14 , 12–23 (2015).

Maraseni, T., An-Vo, D. A., Mushtaq, S. & Reardon-Smith, K. Carbon smart agriculture: an integrated regional approach offers significant potential to increase profit and resource use efficiency, and reduce emissions. J. Clean. Prod . https://doi.org/10.1016/j.jclepro.2020.124555 (2021).

Modongo, O. & Kulshreshtha, S. N. Economics of mitigating greenhouse gas emissions from beef production in western Canada. Agric. Syst. 162 , 229–238 (2018).

Kinzig, A. P. in The Princeton Guide to Ecology (eds Levin, S. A. et al.) 573–670 (Princeton Univ. Press, 2012).

Sanon, S., Hein, T., Douven, W. & Winkler, P. Quantifying ecosystem service trade-offs: the case of an urban floodplain in Vienna, Austria. J. Environ. Manag. 111 , 159–172 (2012).

Wright, L. A., Kemp, S. & Williams, I. ‘Carbon footprinting’: towards a universally accepted definition. Carbon Manag. 2 , 61–72 (2011).

Chapagain, A. K. & Hoekstra, A. Y. The water footprint of Morocco and the Netherlands: global water use as a result of domestic consumption of agricultural commodities. Ecol. Econ. 64 , 109–118 (2007).

Lambin, E. F. & Meyfroidt, P. Global land use change, economic globalization, and the looming land scarcity. Proc. Natl Acad. Sci. USA 108 , 3465–3472 (2011).

Hochman, Z. et al. Cropping system yield gaps can be narrowed with more optimal rotations in dryland subtropical Australia. Agric. Syst. 184 , 102896 (2020).

Karner, K., Schmid, E., Schneider, U. A. & Mitter, H. Computing stochastic Pareto frontiers between economic and environmental goals for a semi-arid agricultural production region in Austria. Ecol. Econ . https://doi.org/10.1016/j.ecolecon.2021.107044 (2021).

Holzkämper, A., Klein, T., Seppelt, R. & Fuhrer, J. Assessing the propagation of uncertainties in multi-objective optimization for agro-ecosystem adaptation to climate change. Environ. Model. Softw. 66 , 27–35 (2015).

D’Odorico, P. et al. The global food–energy–water nexus. Rev. Geophys. 56 , 456–531 (2018).

FABLE Consortium. Pathways to Sustainable Land-Use and Food Systems. 2020 Report of the FABLE Consortium (International Institute of Applied Systems Analysis and Sustainable Development Solutions Network, 2020); https://doi.org/10.22022/ESM/12-2020.16896

Agricultural Model Intercomparison and Improvement Project. Approach. Track 2: Climate change multi-model assessment AgMIP https://agmip.org/approach-4/ (2023).

Royle, J. A. & Nichols, J. D. Estimating abundance from repeated presence–absence data or point counts. Ecology 84 , 777–790 (2003).

Legendre, P. & Legendre, L. Numerical Ecology (Elsevier, 2012).

R Core Team. R: A language and environment for statistical computing. R package version 4.1.2. https://www.R-project.org/ (2021).

Wickham, H. et al. Welcome to the tidyverse. J. Open Source Softw . https://doi.org/10.21105/joss.01686 (2019).

Galili, T. dendextend: an R package for visualizing, adjusting, and comparing trees of hierarchical clustering. Bioinformatics 31 , 3718–3720 (2015).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Maechler, M., Rousseeuw, P., Struyf, A., Hubert, M. & Hornik, K. cluster: Cluster analysis basics and extensions. R package version 2.1.2 (2021).

Oksanen, J. et al. vegan: Community ecology package. R package version 2.5-7 https://CRAN.R-project.org/package=vegan (2020).

Kolde, R. pheatmap: Pretty heatmaps. R package version 1.0.12 https://CRAN.R-project.org/package=pheatmap (2019).

Download references

Acknowledgements

We acknowledge R. Seppelt for his comments on the initial methodology. This work was made possible by the CGIAR Research Program on Roots, Tubers and Bananas (RTB) and the One CGIAR Initiatives ‘Nexus Gains—Realizing Multiple Benefits Across Water, Energy, Food and Ecosystems’ and ‘Nature Positive Solutions’, together with all of the donors who supported this research through their contributions to the CGIAR and One CGIAR Fund. For a list of One CGIAR Fund donors, please see http://www.cgiar.org/our-funders . This research was partly funded by the United States Agency for International Development (USAID; AID-BFS-G-11-00002) as part of the US government’s Feed the Future Initiative. The contents of this Article are the responsibility of the producing organizations and do not necessarily reflect the opinion of USAID or the US government.

Author information

Authors and affiliations.

Farming Systems Ecology, Wageningen University and Research, Wageningen, The Netherlands

Timo S. Breure & Jeroen C. J. Groot

Bioversity International, Montpellier, France

Natalia Estrada-Carmona

Bioversity International, Rome, Italy

Athanasios Petsakos & Elisabetta Gotor

Institute for Biodiversity and Ecosystem Dynamics, University of Amsterdam, Amsterdam, The Netherlands

Boris Jansen

You can also search for this author in PubMed   Google Scholar

Contributions

T.S.B. conceived and designed the study, led and performed the review and data analyses, interpretations and writing. N.E.-C. contributed to the study’s design, interpretations and writing. A.P., E.G. and B.J. contributed to interpretations and writing. J.C.J.G. contributed to the study’s design, analysis, interpretations and writing.

Corresponding author

Correspondence to Jeroen C. J. Groot .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Peer review

Peer review information.

Nature Food thanks John Antle and Emma Stephens for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended data fig. 1 articles per year of publication..

Number of articles by publication year ( a ) and its cumulative distribution ( b ).

Extended Data Fig. 2 Figures on the number of trade-off analysis (TOA) indicators considered.

Cumulative distribution of articles per number of TOA indicators included within an article ( a ). Frequency (%) of the number of TOA indicators included within an article, color-coded by cluster as specified in Fig. 2 in the main text ( b ).

Supplementary information

Supplementary information.

Supplementary Table 1, Figs. 1–9 and a list of articles included in the systematic review.

Reporting Summary

Supplementary data 1.

Criteria assessed in the systematic review. This file was used to perform the analysis and create the figures.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Breure, T.S., Estrada-Carmona, N., Petsakos, A. et al. A systematic review of the methodology of trade-off analysis in agriculture. Nat Food 5 , 211–220 (2024). https://doi.org/10.1038/s43016-024-00926-x

Download citation

Received : 25 August 2022

Accepted : 15 January 2024

Published : 05 March 2024

Issue Date : March 2024

DOI : https://doi.org/10.1038/s43016-024-00926-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: Anthropocene newsletter — what matters in anthropocene research, free to your inbox weekly.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Wiley-Blackwell Online Open

An overview of methodological approaches in systematic reviews

Prabhakar veginadu.

1 Department of Rural Clinical Sciences, La Trobe Rural Health School, La Trobe University, Bendigo Victoria, Australia

Hanny Calache

2 Lincoln International Institute for Rural Health, University of Lincoln, Brayford Pool, Lincoln UK

Akshaya Pandian

3 Department of Orthodontics, Saveetha Dental College, Chennai Tamil Nadu, India

Mohd Masood

Associated data.

APPENDIX B: List of excluded studies with detailed reasons for exclusion

APPENDIX C: Quality assessment of included reviews using AMSTAR 2

The aim of this overview is to identify and collate evidence from existing published systematic review (SR) articles evaluating various methodological approaches used at each stage of an SR.

The search was conducted in five electronic databases from inception to November 2020 and updated in February 2022: MEDLINE, Embase, Web of Science Core Collection, Cochrane Database of Systematic Reviews, and APA PsycINFO. Title and abstract screening were performed in two stages by one reviewer, supported by a second reviewer. Full‐text screening, data extraction, and quality appraisal were performed by two reviewers independently. The quality of the included SRs was assessed using the AMSTAR 2 checklist.

The search retrieved 41,556 unique citations, of which 9 SRs were deemed eligible for inclusion in final synthesis. Included SRs evaluated 24 unique methodological approaches used for defining the review scope and eligibility, literature search, screening, data extraction, and quality appraisal in the SR process. Limited evidence supports the following (a) searching multiple resources (electronic databases, handsearching, and reference lists) to identify relevant literature; (b) excluding non‐English, gray, and unpublished literature, and (c) use of text‐mining approaches during title and abstract screening.

The overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process, as well as some methodological modifications currently used in expedited SRs. Overall, findings of this overview highlight the dearth of published SRs focused on SR methodologies and this warrants future work in this area.

1. INTRODUCTION

Evidence synthesis is a prerequisite for knowledge translation. 1 A well conducted systematic review (SR), often in conjunction with meta‐analyses (MA) when appropriate, is considered the “gold standard” of methods for synthesizing evidence related to a topic of interest. 2 The central strength of an SR is the transparency of the methods used to systematically search, appraise, and synthesize the available evidence. 3 Several guidelines, developed by various organizations, are available for the conduct of an SR; 4 , 5 , 6 , 7 among these, Cochrane is considered a pioneer in developing rigorous and highly structured methodology for the conduct of SRs. 8 The guidelines developed by these organizations outline seven fundamental steps required in SR process: defining the scope of the review and eligibility criteria, literature searching and retrieval, selecting eligible studies, extracting relevant data, assessing risk of bias (RoB) in included studies, synthesizing results, and assessing certainty of evidence (CoE) and presenting findings. 4 , 5 , 6 , 7

The methodological rigor involved in an SR can require a significant amount of time and resource, which may not always be available. 9 As a result, there has been a proliferation of modifications made to the traditional SR process, such as refining, shortening, bypassing, or omitting one or more steps, 10 , 11 for example, limits on the number and type of databases searched, limits on publication date, language, and types of studies included, and limiting to one reviewer for screening and selection of studies, as opposed to two or more reviewers. 10 , 11 These methodological modifications are made to accommodate the needs of and resource constraints of the reviewers and stakeholders (e.g., organizations, policymakers, health care professionals, and other knowledge users). While such modifications are considered time and resource efficient, they may introduce bias in the review process reducing their usefulness. 5

Substantial research has been conducted examining various approaches used in the standardized SR methodology and their impact on the validity of SR results. There are a number of published reviews examining the approaches or modifications corresponding to single 12 , 13 or multiple steps 14 involved in an SR. However, there is yet to be a comprehensive summary of the SR‐level evidence for all the seven fundamental steps in an SR. Such a holistic evidence synthesis will provide an empirical basis to confirm the validity of current accepted practices in the conduct of SRs. Furthermore, sometimes there is a balance that needs to be achieved between the resource availability and the need to synthesize the evidence in the best way possible, given the constraints. This evidence base will also inform the choice of modifications to be made to the SR methods, as well as the potential impact of these modifications on the SR results. An overview is considered the choice of approach for summarizing existing evidence on a broad topic, directing the reader to evidence, or highlighting the gaps in evidence, where the evidence is derived exclusively from SRs. 15 Therefore, for this review, an overview approach was used to (a) identify and collate evidence from existing published SR articles evaluating various methodological approaches employed in each of the seven fundamental steps of an SR and (b) highlight both the gaps in the current research and the potential areas for future research on the methods employed in SRs.

An a priori protocol was developed for this overview but was not registered with the International Prospective Register of Systematic Reviews (PROSPERO), as the review was primarily methodological in nature and did not meet PROSPERO eligibility criteria for registration. The protocol is available from the corresponding author upon reasonable request. This overview was conducted based on the guidelines for the conduct of overviews as outlined in The Cochrane Handbook. 15 Reporting followed the Preferred Reporting Items for Systematic reviews and Meta‐analyses (PRISMA) statement. 3

2.1. Eligibility criteria

Only published SRs, with or without associated MA, were included in this overview. We adopted the defining characteristics of SRs from The Cochrane Handbook. 5 According to The Cochrane Handbook, a review was considered systematic if it satisfied the following criteria: (a) clearly states the objectives and eligibility criteria for study inclusion; (b) provides reproducible methodology; (c) includes a systematic search to identify all eligible studies; (d) reports assessment of validity of findings of included studies (e.g., RoB assessment of the included studies); (e) systematically presents all the characteristics or findings of the included studies. 5 Reviews that did not meet all of the above criteria were not considered a SR for this study and were excluded. MA‐only articles were included if it was mentioned that the MA was based on an SR.

SRs and/or MA of primary studies evaluating methodological approaches used in defining review scope and study eligibility, literature search, study selection, data extraction, RoB assessment, data synthesis, and CoE assessment and reporting were included. The methodological approaches examined in these SRs and/or MA can also be related to the substeps or elements of these steps; for example, applying limits on date or type of publication are the elements of literature search. Included SRs examined or compared various aspects of a method or methods, and the associated factors, including but not limited to: precision or effectiveness; accuracy or reliability; impact on the SR and/or MA results; reproducibility of an SR steps or bias occurred; time and/or resource efficiency. SRs assessing the methodological quality of SRs (e.g., adherence to reporting guidelines), evaluating techniques for building search strategies or the use of specific database filters (e.g., use of Boolean operators or search filters for randomized controlled trials), examining various tools used for RoB or CoE assessment (e.g., ROBINS vs. Cochrane RoB tool), or evaluating statistical techniques used in meta‐analyses were excluded. 14

2.2. Search

The search for published SRs was performed on the following scientific databases initially from inception to third week of November 2020 and updated in the last week of February 2022: MEDLINE (via Ovid), Embase (via Ovid), Web of Science Core Collection, Cochrane Database of Systematic Reviews, and American Psychological Association (APA) PsycINFO. Search was restricted to English language publications. Following the objectives of this study, study design filters within databases were used to restrict the search to SRs and MA, where available. The reference lists of included SRs were also searched for potentially relevant publications.

The search terms included keywords, truncations, and subject headings for the key concepts in the review question: SRs and/or MA, methods, and evaluation. Some of the terms were adopted from the search strategy used in a previous review by Robson et al., which reviewed primary studies on methodological approaches used in study selection, data extraction, and quality appraisal steps of SR process. 14 Individual search strategies were developed for respective databases by combining the search terms using appropriate proximity and Boolean operators, along with the related subject headings in order to identify SRs and/or MA. 16 , 17 A senior librarian was consulted in the design of the search terms and strategy. Appendix A presents the detailed search strategies for all five databases.

2.3. Study selection and data extraction

Title and abstract screening of references were performed in three steps. First, one reviewer (PV) screened all the titles and excluded obviously irrelevant citations, for example, articles on topics not related to SRs, non‐SR publications (such as randomized controlled trials, observational studies, scoping reviews, etc.). Next, from the remaining citations, a random sample of 200 titles and abstracts were screened against the predefined eligibility criteria by two reviewers (PV and MM), independently, in duplicate. Discrepancies were discussed and resolved by consensus. This step ensured that the responses of the two reviewers were calibrated for consistency in the application of the eligibility criteria in the screening process. Finally, all the remaining titles and abstracts were reviewed by a single “calibrated” reviewer (PV) to identify potential full‐text records. Full‐text screening was performed by at least two authors independently (PV screened all the records, and duplicate assessment was conducted by MM, HC, or MG), with discrepancies resolved via discussions or by consulting a third reviewer.

Data related to review characteristics, results, key findings, and conclusions were extracted by at least two reviewers independently (PV performed data extraction for all the reviews and duplicate extraction was performed by AP, HC, or MG).

2.4. Quality assessment of included reviews

The quality assessment of the included SRs was performed using the AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews). The tool consists of a 16‐item checklist addressing critical and noncritical domains. 18 For the purpose of this study, the domain related to MA was reclassified from critical to noncritical, as SRs with and without MA were included. The other six critical domains were used according to the tool guidelines. 18 Two reviewers (PV and AP) independently responded to each of the 16 items in the checklist with either “yes,” “partial yes,” or “no.” Based on the interpretations of the critical and noncritical domains, the overall quality of the review was rated as high, moderate, low, or critically low. 18 Disagreements were resolved through discussion or by consulting a third reviewer.

2.5. Data synthesis

To provide an understandable summary of existing evidence syntheses, characteristics of the methods evaluated in the included SRs were examined and key findings were categorized and presented based on the corresponding step in the SR process. The categories of key elements within each step were discussed and agreed by the authors. Results of the included reviews were tabulated and summarized descriptively, along with a discussion on any overlap in the primary studies. 15 No quantitative analyses of the data were performed.

From 41,556 unique citations identified through literature search, 50 full‐text records were reviewed, and nine systematic reviews 14 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 were deemed eligible for inclusion. The flow of studies through the screening process is presented in Figure  1 . A list of excluded studies with reasons can be found in Appendix B .

Study selection flowchart

3.1. Characteristics of included reviews

Table  1 summarizes the characteristics of included SRs. The majority of the included reviews (six of nine) were published after 2010. 14 , 22 , 23 , 24 , 25 , 26 Four of the nine included SRs were Cochrane reviews. 20 , 21 , 22 , 23 The number of databases searched in the reviews ranged from 2 to 14, 2 reviews searched gray literature sources, 24 , 25 and 7 reviews included a supplementary search strategy to identify relevant literature. 14 , 19 , 20 , 21 , 22 , 23 , 26 Three of the included SRs (all Cochrane reviews) included an integrated MA. 20 , 21 , 23

Characteristics of included studies

SR = systematic review; MA = meta‐analysis; RCT = randomized controlled trial; CCT = controlled clinical trial; N/R = not reported.

The included SRs evaluated 24 unique methodological approaches (26 in total) used across five steps in the SR process; 8 SRs evaluated 6 approaches, 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 while 1 review evaluated 18 approaches. 14 Exclusion of gray or unpublished literature 21 , 26 and blinding of reviewers for RoB assessment 14 , 23 were evaluated in two reviews each. Included SRs evaluated methods used in five different steps in the SR process, including methods used in defining the scope of review ( n  = 3), literature search ( n  = 3), study selection ( n  = 2), data extraction ( n  = 1), and RoB assessment ( n  = 2) (Table  2 ).

Summary of findings from review evaluating systematic review methods

There was some overlap in the primary studies evaluated in the included SRs on the same topics: Schmucker et al. 26 and Hopewell et al. 21 ( n  = 4), Hopewell et al. 20 and Crumley et al. 19 ( n  = 30), and Robson et al. 14 and Morissette et al. 23 ( n  = 4). There were no conflicting results between any of the identified SRs on the same topic.

3.2. Methodological quality of included reviews

Overall, the quality of the included reviews was assessed as moderate at best (Table  2 ). The most common critical weakness in the reviews was failure to provide justification for excluding individual studies (four reviews). Detailed quality assessment is provided in Appendix C .

3.3. Evidence on systematic review methods

3.3.1. methods for defining review scope and eligibility.

Two SRs investigated the effect of excluding data obtained from gray or unpublished sources on the pooled effect estimates of MA. 21 , 26 Hopewell et al. 21 reviewed five studies that compared the impact of gray literature on the results of a cohort of MA of RCTs in health care interventions. Gray literature was defined as information published in “print or electronic sources not controlled by commercial or academic publishers.” Findings showed an overall greater treatment effect for published trials than trials reported in gray literature. In a more recent review, Schmucker et al. 26 addressed similar objectives, by investigating gray and unpublished data in medicine. In addition to gray literature, defined similar to the previous review by Hopewell et al., the authors also evaluated unpublished data—defined as “supplemental unpublished data related to published trials, data obtained from the Food and Drug Administration  or other regulatory websites or postmarketing analyses hidden from the public.” The review found that in majority of the MA, excluding gray literature had little or no effect on the pooled effect estimates. The evidence was limited to conclude if the data from gray and unpublished literature had an impact on the conclusions of MA. 26

Morrison et al. 24 examined five studies measuring the effect of excluding non‐English language RCTs on the summary treatment effects of SR‐based MA in various fields of conventional medicine. Although none of the included studies reported major difference in the treatment effect estimates between English only and non‐English inclusive MA, the review found inconsistent evidence regarding the methodological and reporting quality of English and non‐English trials. 24 As such, there might be a risk of introducing “language bias” when excluding non‐English language RCTs. The authors also noted that the numbers of non‐English trials vary across medical specialties, as does the impact of these trials on MA results. Based on these findings, Morrison et al. 24 conclude that literature searches must include non‐English studies when resources and time are available to minimize the risk of introducing “language bias.”

3.3.2. Methods for searching studies

Crumley et al. 19 analyzed recall (also referred to as “sensitivity” by some researchers; defined as “percentage of relevant studies identified by the search”) and precision (defined as “percentage of studies identified by the search that were relevant”) when searching a single resource to identify randomized controlled trials and controlled clinical trials, as opposed to searching multiple resources. The studies included in their review frequently compared a MEDLINE only search with the search involving a combination of other resources. The review found low median recall estimates (median values between 24% and 92%) and very low median precisions (median values between 0% and 49%) for most of the electronic databases when searched singularly. 19 A between‐database comparison, based on the type of search strategy used, showed better recall and precision for complex and Cochrane Highly Sensitive search strategies (CHSSS). In conclusion, the authors emphasize that literature searches for trials in SRs must include multiple sources. 19

In an SR comparing handsearching and electronic database searching, Hopewell et al. 20 found that handsearching retrieved more relevant RCTs (retrieval rate of 92%−100%) than searching in a single electronic database (retrieval rates of 67% for PsycINFO/PsycLIT, 55% for MEDLINE, and 49% for Embase). The retrieval rates varied depending on the quality of handsearching, type of electronic search strategy used (e.g., simple, complex or CHSSS), and type of trial reports searched (e.g., full reports, conference abstracts, etc.). The authors concluded that handsearching was particularly important in identifying full trials published in nonindexed journals and in languages other than English, as well as those published as abstracts and letters. 20

The effectiveness of checking reference lists to retrieve additional relevant studies for an SR was investigated by Horsley et al. 22 The review reported that checking reference lists yielded 2.5%–40% more studies depending on the quality and comprehensiveness of the electronic search used. The authors conclude that there is some evidence, although from poor quality studies, to support use of checking reference lists to supplement database searching. 22

3.3.3. Methods for selecting studies

Three approaches relevant to reviewer characteristics, including number, experience, and blinding of reviewers involved in the screening process were highlighted in an SR by Robson et al. 14 Based on the retrieved evidence, the authors recommended that two independent, experienced, and unblinded reviewers be involved in study selection. 14 A modified approach has also been suggested by the review authors, where one reviewer screens and the other reviewer verifies the list of excluded studies, when the resources are limited. It should be noted however this suggestion is likely based on the authors’ opinion, as there was no evidence related to this from the studies included in the review.

Robson et al. 14 also reported two methods describing the use of technology for screening studies: use of Google Translate for translating languages (for example, German language articles to English) to facilitate screening was considered a viable method, while using two computer monitors for screening did not increase the screening efficiency in SR. Title‐first screening was found to be more efficient than simultaneous screening of titles and abstracts, although the gain in time with the former method was lesser than the latter. Therefore, considering that the search results are routinely exported as titles and abstracts, Robson et al. 14 recommend screening titles and abstracts simultaneously. However, the authors note that these conclusions were based on very limited number (in most instances one study per method) of low‐quality studies. 14

3.3.4. Methods for data extraction

Robson et al. 14 examined three approaches for data extraction relevant to reviewer characteristics, including number, experience, and blinding of reviewers (similar to the study selection step). Although based on limited evidence from a small number of studies, the authors recommended use of two experienced and unblinded reviewers for data extraction. The experience of the reviewers was suggested to be especially important when extracting continuous outcomes (or quantitative) data. However, when the resources are limited, data extraction by one reviewer and a verification of the outcomes data by a second reviewer was recommended.

As for the methods involving use of technology, Robson et al. 14 identified limited evidence on the use of two monitors to improve the data extraction efficiency and computer‐assisted programs for graphical data extraction. However, use of Google Translate for data extraction in non‐English articles was not considered to be viable. 14 In the same review, Robson et al. 14 identified evidence supporting contacting authors for obtaining additional relevant data.

3.3.5. Methods for RoB assessment

Two SRs examined the impact of blinding of reviewers for RoB assessments. 14 , 23 Morissette et al. 23 investigated the mean differences between the blinded and unblinded RoB assessment scores and found inconsistent differences among the included studies providing no definitive conclusions. Similar conclusions were drawn in a more recent review by Robson et al., 14 which included four studies on reviewer blinding for RoB assessment that completely overlapped with Morissette et al. 23

Use of experienced reviewers and provision of additional guidance for RoB assessment were examined by Robson et al. 14 The review concluded that providing intensive training and guidance on assessing studies reporting insufficient data to the reviewers improves RoB assessments. 14 Obtaining additional data related to quality assessment by contacting study authors was also found to help the RoB assessments, although based on limited evidence. When assessing the qualitative or mixed method reviews, Robson et al. 14 recommends the use of a structured RoB tool as opposed to an unstructured tool. No SRs were identified on data synthesis and CoE assessment and reporting steps.

4. DISCUSSION

4.1. summary of findings.

Nine SRs examining 24 unique methods used across five steps in the SR process were identified in this overview. The collective evidence supports some current traditional and modified SR practices, while challenging other approaches. However, the quality of the included reviews was assessed to be moderate at best and in the majority of the included SRs, evidence related to the evaluated methods was obtained from very limited numbers of primary studies. As such, the interpretations from these SRs should be made cautiously.

The evidence gathered from the included SRs corroborate a few current SR approaches. 5 For example, it is important to search multiple resources for identifying relevant trials (RCTs and/or CCTs). The resources must include a combination of electronic database searching, handsearching, and reference lists of retrieved articles. 5 However, no SRs have been identified that evaluated the impact of the number of electronic databases searched. A recent study by Halladay et al. 27 found that articles on therapeutic intervention, retrieved by searching databases other than PubMed (including Embase), contributed only a small amount of information to the MA and also had a minimal impact on the MA results. The authors concluded that when the resources are limited and when large number of studies are expected to be retrieved for the SR or MA, PubMed‐only search can yield reliable results. 27

Findings from the included SRs also reiterate some methodological modifications currently employed to “expedite” the SR process. 10 , 11 For example, excluding non‐English language trials and gray/unpublished trials from MA have been shown to have minimal or no impact on the results of MA. 24 , 26 However, the efficiency of these SR methods, in terms of time and the resources used, have not been evaluated in the included SRs. 24 , 26 Of the SRs included, only two have focused on the aspect of efficiency 14 , 25 ; O'Mara‐Eves et al. 25 report some evidence to support the use of text‐mining approaches for title and abstract screening in order to increase the rate of screening. Moreover, only one included SR 14 considered primary studies that evaluated reliability (inter‐ or intra‐reviewer consistency) and accuracy (validity when compared against a “gold standard” method) of the SR methods. This can be attributed to the limited number of primary studies that evaluated these outcomes when evaluating the SR methods. 14 Lack of outcome measures related to reliability, accuracy, and efficiency precludes making definitive recommendations on the use of these methods/modifications. Future research studies must focus on these outcomes.

Some evaluated methods may be relevant to multiple steps; for example, exclusions based on publication status (gray/unpublished literature) and language of publication (non‐English language studies) can be outlined in the a priori eligibility criteria or can be incorporated as search limits in the search strategy. SRs included in this overview focused on the effect of study exclusions on pooled treatment effect estimates or MA conclusions. Excluding studies from the search results, after conducting a comprehensive search, based on different eligibility criteria may yield different results when compared to the results obtained when limiting the search itself. 28 Further studies are required to examine this aspect.

Although we acknowledge the lack of standardized quality assessment tools for methodological study designs, we adhered to the Cochrane criteria for identifying SRs in this overview. This was done to ensure consistency in the quality of the included evidence. As a result, we excluded three reviews that did not provide any form of discussion on the quality of the included studies. The methods investigated in these reviews concern supplementary search, 29 data extraction, 12 and screening. 13 However, methods reported in two of these three reviews, by Mathes et al. 12 and Waffenschmidt et al., 13 have also been examined in the SR by Robson et al., 14 which was included in this overview; in most instances (with the exception of one study included in Mathes et al. 12 and Waffenschmidt et al. 13 each), the studies examined in these excluded reviews overlapped with those in the SR by Robson et al. 14

One of the key gaps in the knowledge observed in this overview was the dearth of SRs on the methods used in the data synthesis component of SR. Narrative and quantitative syntheses are the two most commonly used approaches for synthesizing data in evidence synthesis. 5 There are some published studies on the proposed indications and implications of these two approaches. 30 , 31 These studies found that both data synthesis methods produced comparable results and have their own advantages, suggesting that the choice of the method must be based on the purpose of the review. 31 With increasing number of “expedited” SR approaches (so called “rapid reviews”) avoiding MA, 10 , 11 further research studies are warranted in this area to determine the impact of the type of data synthesis on the results of the SR.

4.2. Implications for future research

The findings of this overview highlight several areas of paucity in primary research and evidence synthesis on SR methods. First, no SRs were identified on methods used in two important components of the SR process, including data synthesis and CoE and reporting. As for the included SRs, a limited number of evaluation studies have been identified for several methods. This indicates that further research is required to corroborate many of the methods recommended in current SR guidelines. 4 , 5 , 6 , 7 Second, some SRs evaluated the impact of methods on the results of quantitative synthesis and MA conclusions. Future research studies must also focus on the interpretations of SR results. 28 , 32 Finally, most of the included SRs were conducted on specific topics related to the field of health care, limiting the generalizability of the findings to other areas. It is important that future research studies evaluating evidence syntheses broaden the objectives and include studies on different topics within the field of health care.

4.3. Strengths and limitations

To our knowledge, this is the first overview summarizing current evidence from SRs and MA on different methodological approaches used in several fundamental steps in SR conduct. The overview methodology followed well established guidelines and strict criteria defined for the inclusion of SRs.

There are several limitations related to the nature of the included reviews. Evidence for most of the methods investigated in the included reviews was derived from a limited number of primary studies. Also, the majority of the included SRs may be considered outdated as they were published (or last updated) more than 5 years ago 33 ; only three of the nine SRs have been published in the last 5 years. 14 , 25 , 26 Therefore, important and recent evidence related to these topics may not have been included. Substantial numbers of included SRs were conducted in the field of health, which may limit the generalizability of the findings. Some method evaluations in the included SRs focused on quantitative analyses components and MA conclusions only. As such, the applicability of these findings to SR more broadly is still unclear. 28 Considering the methodological nature of our overview, limiting the inclusion of SRs according to the Cochrane criteria might have resulted in missing some relevant evidence from those reviews without a quality assessment component. 12 , 13 , 29 Although the included SRs performed some form of quality appraisal of the included studies, most of them did not use a standardized RoB tool, which may impact the confidence in their conclusions. Due to the type of outcome measures used for the method evaluations in the primary studies and the included SRs, some of the identified methods have not been validated against a reference standard.

Some limitations in the overview process must be noted. While our literature search was exhaustive covering five bibliographic databases and supplementary search of reference lists, no gray sources or other evidence resources were searched. Also, the search was primarily conducted in health databases, which might have resulted in missing SRs published in other fields. Moreover, only English language SRs were included for feasibility. As the literature search retrieved large number of citations (i.e., 41,556), the title and abstract screening was performed by a single reviewer, calibrated for consistency in the screening process by another reviewer, owing to time and resource limitations. These might have potentially resulted in some errors when retrieving and selecting relevant SRs. The SR methods were grouped based on key elements of each recommended SR step, as agreed by the authors. This categorization pertains to the identified set of methods and should be considered subjective.

5. CONCLUSIONS

This overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process. Limited evidence was also identified on some methodological modifications currently used to expedite the SR process. Overall, findings highlight the dearth of SRs on SR methodologies, warranting further work to confirm several current recommendations on conventional and expedited SR processes.

CONFLICT OF INTEREST

The authors declare no conflicts of interest.

Supporting information

APPENDIX A: Detailed search strategies

ACKNOWLEDGMENTS

The first author is supported by a La Trobe University Full Fee Research Scholarship and a Graduate Research Scholarship.

Open Access Funding provided by La Trobe University.

Veginadu P, Calache H, Gussy M, Pandian A, Masood M. An overview of methodological approaches in systematic reviews . J Evid Based Med . 2022; 15 :39–54. 10.1111/jebm.12468 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]