mixed method research design limitations

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Taking a critical stance towards mixed methods research: A cross-disciplinary qualitative secondary analysis of researchers’ views

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Validation, Writing – original draft, Writing – review & editing

* E-mail: [email protected] , [email protected]

Affiliation Department of Psychology and Education, Universitat Oberta de Catalunya, Barcelona, Spain

Roles Conceptualization, Methodology, Validation, Writing – review & editing

Affiliation Department of Education, Universidad del Norte, Barranquilla, Colombia

Roles Conceptualization, Formal analysis, Methodology, Validation, Writing – review & editing

Affiliation Department of Management, Universidad de Alicante, Alicante, Spain

Roles Conceptualization, Validation, Writing – review & editing

Affiliation Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre), University College London, London, United Kingdom

Roles Conceptualization, Supervision, Writing – review & editing

Affiliation Centre d’Estudis Sociologics sobre la Vida Quotidiana i el Treball (Sociological Research Centre on Everyday Life and Work—QUIT), Universitat Autonoma de Barcelona, Barcelona, Spain

• Sergi Fàbregues, 
• Elsa Lucia Escalante-Barrios, 
• José Francisco Molina-Azorin, 
• Quan Nha Hong, 
• Joan Miquel Verd

• Published: July 9, 2021
• https://doi.org/10.1371/journal.pone.0252014
• Peer Review
• Reader Comments

Recent growth and institutionalization in the field of mixed methods research has provided fertile ground for a wide range of thoughtful criticism of how this research approach has been developed and conceptualized by some members of the mixed methods community. This criticism reflects the increasing maturity of the field as well as the different theoretical perspectives and methodological practices of researchers in different disciplines. While debates addressing these criticisms are likely to lead to valuable insights, no empirical studies have been carried out to date that have investigated researchers’ critical views on the development and conceptualization of mixed methods research. This study examines the criticisms of the mixed methods field raised by a cross-national sample of researchers in education, nursing, psychology, and sociology. We carried out a secondary analysis of semi-structured interviews with 42 researchers and identified 11 different criticisms, which we classified in four domains: essence of mixed methods, philosophy, procedures, and politics. The criticisms related to the procedures domain were equally distributed among the four disciplines, while those related to the essence, philosophy and politics domains were more common among sociologists. Based on our findings, we argue that the divergence of views on foundational issues in this field reflects researchers’ affiliation to different communities of practice, each having its own principles, values, and interests. We suggest that a greater awareness of this divergence of perspectives could help researchers establish effective collaboration and anticipate potential challenges when working with researchers having different methodological approaches.

Citation: Fàbregues S, Escalante-Barrios EL, Molina-Azorin JF, Hong QN, Verd JM (2021) Taking a critical stance towards mixed methods research: A cross-disciplinary qualitative secondary analysis of researchers’ views. PLoS ONE 16(7): e0252014. https://doi.org/10.1371/journal.pone.0252014

Editor: Adam T. Perzynski, The MetroHealth System and Case Western Reserve University, UNITED STATES

Received: July 24, 2020; Accepted: May 7, 2021; Published: July 9, 2021

Copyright: © 2021 Fàbregues et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: Data cannot be shared publicly as participants did not give consent for their transcripts to be shared in this manner. Since the consent statement approved by the Institutional Review Board (IRB) of the Universitat Autònoma de Barcelona, and signed by the participants, did not include the provision that data would be made publicly available, we do not have participant consent to share this data. Also, the content is sensitive, and participants could be identified. Requests for anonymized data can be made to the Principal Investigator of the study, Sergi Fàbregues ( [email protected] ) or the Institutional Review Board (IRB) of the Universitat Autònoma de Barcelona ( [email protected] ).

Funding: The authors received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Since the 1990s, mixed methods research as a distinct methodology has seen vigorous institutionalization [ 1 , 2 ] with the launch of a specialized journal ( Journal of Mixed Methods Research ) in 2007, the establishment of the Mixed Methods International Research Association (MMIRA) in 2013, and the publication of a stream of handbooks, textbooks, and articles on the philosophy and practice of mixed methods. Mixed methods research is increasingly used in a wide range of disciplines, particularly in the social, educational, behavioral and health sciences [ 3 – 6 ]. Several scholars have argued that mixed methods inquiry can help researchers arrive at a more complete understanding of research problems, develop more robust quantitative instruments, and integrate several worldviews in a single research study [ 7 , 8 ].

A clear indication of the institutionalization of mixed methods research as a field is the expansion of the mixed methods community. Tashakkori, Johnson and Teddlie [ 9 ] have characterized this community as a group of scholars who share similar backgrounds, methodological orientations, philosophical assumptions, and views on research and practice. As distinct from researchers using only qualitative or quantitative approaches, these scholars often share similarities in training, research background, and professional affiliation. Those authors’ view of the mixed methods community is consistent with Thomas Kuhn’s preferred definition of paradigms, as cited by Morgan [ 10 ]. In disagreement with the view of paradigms as incompatible epistemological stances, Morgan [ 10 ], following Kuhn [ 11 ], advanced a more integrative notion of paradigms focused on shared beliefs and joint actions in a community of researchers. Denscombe [ 12 ] took this perspective one step further by proposing the notion of communities of practice , an idea originally developed by the educationalist Étienne Wenger. This notion brings us closer to a definition of paradigms as shared beliefs able to accommodate the diversity of perspectives and approaches that currently exists within the mixed methods community [ 12 , 13 ].

While the institutionalization of mixed methods as a field has helped to formalize and clarify research practices [ 14 ], this trend has also led to some criticism of the ways in which this approach has been developed and conceptualized by some members of the mixed methods community [ 15 , 16 ]. The critiques of the mixed methods field have already been summarized in two overviews published in the early 2010s. In the second edition of the Handbook , Tashakkori and Teddlie [ 17 ] outlined four frequent criticisms raised in the mixed methods literature, including overreliance on typologies and the higher status of quantitative versus qualitative research. One year later, Creswell [ 18 ] described some of the same criticisms in a summary of 11 key controversies. The emergence of this criticism testifies to the increasing maturity of the field and its progress towards what Creswell and Plano Clark [ 7 ] define as the period of reflection and refinement in mixed methods research. In their view, the mixed methods community should be honored that it has attracted critical attention and it should stimulate debate around the issues raised. Therefore, it is crucially important to address these criticisms in greater detail because such engagement will lead to valuable insights that could lay the basis for further discussion needed to ensure the healthy development of the field. Furthermore, addressing those criticisms is essential to enhance researchers’ understanding of the complexity of the mixed methods field and to provide them with the awareness needed to deal with tensions that might emerge when working in teams with researchers subscribing to different methodological viewpoints [ 19 , 20 ].

Most of the criticisms of mixed methods so far have been formulated in the literature by several scholars specialized in theoretical aspects of methodology. However, it would be useful to find out whether other researchers share these criticisms and whether they may have formulated others. Scholars with an interest in mixed methods research come from different academic disciplines that embody different theoretical and methodological perspectives. As Plano Clark and Ivankova [ 3 ] argue, these differences are highly likely to influence the ways in which those scholars view mixed methods as well as the questions they might raise regarding current ideas in the field. Therefore, it would be useful to examine the ways in which researchers’ criticisms differ according to discipline.

The aim of the present study is to examine the criticisms of the mixed methods field raised by a cross-national sample of 42 researchers working in the disciplines of education, nursing, psychology, and sociology. We report a secondary analysis of semi-structured interviews originally conducted to describe how researchers operationalize and conceptualize the quality of mixed methods research [ 14 ]. The contribution of this article is twofold: (a) it is, to our knowledge, the first study based on an empirical approach to examining researchers’ critical views on the development and conceptualization of mixed methods research, and (b) it enhances our understanding of the ways in which these critical views may be associated with different academic disciplines.

This article reports a secondary analysis of data originally collected in a multiple-case study of the quality of mixed methods based on semi-structured interviews with researchers in the disciplines of education, nursing, psychology, and sociology. In line with Heaton’s [ 21 ] definition of secondary analysis — which he calls supplementary analysis — as an “in-depth investigation of an emergent issue or aspect of the data which was not considered or fully addressed in the primary study” [ 21 ], in this article we re-analyze the original interview data in order to address the following two research questions (RQs): (RQ1) What criticisms of the mixed methods field are made by researchers in education, nursing, psychology, and sociology? and (RQ2) What differences and similarities can be identified in the criticisms reported by researchers working in different disciplines? In the following subsections, we provide a brief description of the sampling and data collection methods used in the original study, and of the procedures used in the secondary analysis of data. A more detailed explanation of procedures followed in the original study can be found in Fàbregues, Paré, and Meneses [ 22 ].

The original study

Sampling and recruitment of participants..

The disciplines of education, nursing, psychology, and sociology were selected for four main reasons: (1) professionals working in these disciplines contribute a relatively high proportion of mixed methods empirical articles and other methodological publications, (2) a considerable number of prevalence studies and methodological reviews on the use of mixed methods in subfields of these disciplines have been published, and (3) these disciplines are characterized by their clear disciplinary boundaries, and this characteristic offers the possibility of gaining useful comparative insights [ 6 ]. Criterion and maximum variation sampling were used to select the researchers who participated in the study [ 23 ]. In the criterion sampling, participants fulfilled two inclusion criteria: (1) they had carried out research primarily in one of the four disciplines mentioned above, and (2) they had contributed to at least one methodological publication on mixed methods research. Participant identification started with a systematic search for methodological publications on mixed methods published in English during or after 2003. Selected publications fulfilled the definitions of mixed methods suggested by either Creswell and Tashakkori [ 24 ] or Johnson, Onwuegbuzie, and Turner [ 25 ]. A number of characteristics of the first authors’ profiles were extracted, including the field of expertise, the country of affiliation, and the job title. To ensure heterogeneity of the sample, maximum variation sampling was applied to authors meeting the two inclusion criteria. An iterative approach was used to recruit 11 participants for each discipline. Sample size was based on recommendations found in the literature [ 26 ]. Potential participants were contacted using a prioritized list until a total of 44 participants had been recruited.

Data collection.

Data collection involved semi-structured interviews. Questions from the interview guide were focused on the following topics: (a) participants’ research background and methodological expertise, (b) participants’ conceptualization of mixed methods research, and (c) how participants perceived the quality of the mixed methods approach in practice. Interviews were conducted using Skype, telephone and, in two cases, e-mail correspondence. Interviews were audio-recorded and transcribed, and average interview length was 49 minutes. We carried out a member-checking process by sending back to participants the transcriptions and summaries of key points of the interviews to confirm that the data accurately represented their views. At this stage, we also gave participants an opportunity to clarify or expand the statements they made during the interview.

Trustworthiness.

Four strategies were used to enhance the trustworthiness of the original study. First, as explained above, all participants member-checked their transcribed interviews and summaries to confirm accuracy. Second, peer-debriefing was carried out during data collection by one researcher working together with another researcher familiar with mixed methods research who was not included in the sample. Third, an audit trail was used to record the decisions made during the study and to help researchers to reflect on the influence on the study findings of their own assumptions and disciplines. Fourth, the decisions taken during the analysis and interpretation of the data, as well as the disagreements arising during this stage, were discussed by the researchers until a consensus was reached.

The secondary data analysis

While the original study aimed to examine researchers’ views on the conceptualization and operationalization of the quality of mixed methods research, the aim of this secondary analysis of the same data is to examine researchers’ critical views of commonly accepted concepts and practices in the mixed methods field. Ethical approval for secondary data analysis was included in the ethics application for the original study, which was approved by the Institutional Review Board of the Universitat Autònoma de Barcelona. Participants signed an informed consent before the interview. Of the 44 researchers who participated in the original study, two did not consent to the subsequent use of their interview data for a secondary analysis. Therefore, the information provided by these two researchers was not used in the present study. Table 1 shows the characteristics of the 42 participants finally included in this secondary analysis.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0252014.t001

Qualitative content analysis as described by Graneheim & Lundman [ 27 ] was used to carry out the secondary analysis of the interview data. This form of analysis is especially appropriate when, as in this study, researchers are interested in systematically describing only the topics of interest indicated by the research questions, and not in obtaining a holistic overview of all of the data [ 28 ]. The data analysis was carried out in three stages using NVivo 12 for Mac (QSR International Pty Ltd, Victoria, Australia). In the first stage, the interview transcripts were read thoroughly in order to extract the sections of text in which the participants raised criticisms of the mixed methods field. These sections of text constituted the unit of analysis. In the second stage, the extracted sections were divided into meaning units, which were subsequently condensed, abstracted and labelled with codes. Each code included a description of the meaning of the code, an indicator to identify its presence in the data, and an example of a passage coded as belonging to that code. In the third stage, the codes were compared for similarities and differences, and clustered into several categories. The underlying meaning of these categories was then examined and formulated into themes. These themes represented the study participants’ criticisms of mixed methods as a field, which were the focus of RQ1. Decisions made in this phase of the study, along with any disagreements, were discussed by the researchers until a consensus was reached.

In order to answer RQ2, a multiple correspondence analysis [ 29 – 31 ] was carried out. This technique is a non-inferential form of statistical analysis designed to analyze the multivariate association of categorical variables by generating a representation of the underlying structure of a dataset. Since the statistical requirements of multiple correspondence analysis (i.e., sampling, linearity, and normality) are highly flexible, this method is especially suited for examining qualitative interview data transformed into quantitative data [ 32 ]. The output of the multiple correspondence analysis is a scatterplot representing the spatial grouping of categories and participants. The distances between plotted points represent the degree of similarity in the patterns of participants’ responses. Multiple correspondence analysis was used to examine the relationship between the participants’ discipline and the themes relating to criticisms. To perform this analysis, we used the NVivo “matrix coding query” function to generate a matrix in which binary codes related to criticisms identified in the qualitative content analysis (the mention or failure to mention the criticism) were displayed in the columns, while the 42 participants were displayed in the rows. The matrix output was exported to XLSTAT Version 2018.1 (Addinsoft, Paris, France), which was used to perform the analysis, using the binary codes for the criticisms as active variables and the participants’ discipline as supplementary variables. Following the recommendations of Bazeley [ 32 ], after carrying out the multiple correspondence analysis, we checked the results against the qualitative data to verify the interpretation of the statistical analysis.

RQ1; What criticisms of the mixed methods field are made by researchers in education, nursing, psychology, and sociology?

Eleven criticisms of how some members of the mixed methods community have developed and conceptualized this research approach were identified in 27 of the 42 interviews included in this secondary analysis. These criticisms were then grouped into the four domains used by Creswell [ 33 ] to map the landscape of mixed methods research: (1) the essence of mixed methods research (definitions and nomenclature), (2) philosophy (philosophical assumptions and paradigmatic stances), (3) procedures ( methods and techniques for carrying out mixed methods research), and (4) politics (justification of the use of mixed methods research). Each of these four domains and the corresponding criticisms are discussed in the following sections with the support of verbatim quotes from the interviews. Table 2 shows the criticisms for each domain and the number of participants making each criticism.

https://doi.org/10.1371/journal.pone.0252014.t002

Domain 1: The essence of mixed methods research.

Criticism 1 : The accepted definition of mixed methods research takes into account only the mixing of both quantitative and qualitative methods . Some participants objected that the most common definition of mixed methods research that usually prevails in the literature conceives the approach as being limited to the use of quantitative and qualitative methods. These participants believe that the field should adopt a broader definition that would also include the mixing of methods within the same tradition in a single design, that is, the combination of two or more quantitative methods or qualitative methods:

“(…) for me mixed methods is not only mixing qualitative and quantitative methods , but it could also be qualitative + qualitative or quantitative + quantitative methods” (Sociologist #4).

These participants argued that the current definition of mixed methods implies that mixing the two distinct families of methods is often the only appropriate approach while in fact this definition obviates the contingent nature of research. Certain research questions might be better answered by using a combination of methods from a single tradition. Furthermore, combining two methods from the same tradition can be as valuable and as challenging as combining two methods from different traditions. One participant used the term “pressure” to describe the feeling that he was obliged to mix quantitative and qualitative methods even when this approach was not the most appropriate one:

“ … combining methods isn’t just a matter of combining quantitative and qualitative methods . You can combine different methods that are both qualitative or both quantitative and that’s , that’s valuable in it- itself , and I am worried about the kind of pressure to combine quantitative and qualitative as if that would always be appropriate” (Sociologist #2).

Criticism 2 : The terminology used in mixed methods reflects a lack of agreement among its proponents . Several participants noted the lack of clear agreement on the terminology generally used to describe the concepts and procedures that pertain to mixed methods research. They also cited a tendency to use multiple definitions for the same term and different terms to refer to similar notions. One participant cited as problematic the use of several different terms (e.g., legitimation, validity, rigor) to refer to the quality of mixed methods research:

“I would like to see a word that’s used by as many people as possible to describe that [quality]… . But , you know , I , I just think if we , everybody continues to use different terms , that could be problematic” (Educationalist #8).

According to this participant, while synonymous terms might add some precision when used to describe the complexities of implementing mixed methods research, their use can also generate confusion, especially among reviewers, editors and researchers who are trying to familiarize themselves with the field:

“…it just gets to the point where if everyone has a different definition , then how useful is that ? And that gets confusing for those who review manuscripts , or editors , when people are using in different ways that exact same term” (Educationalist #8).

Participants suggested two possible reasons for this lack of agreement. First, the tendency among some scholars to consider that mixed methods researchers should be able to use whatever terminology they may find convenient. Second, the desire of some authors to claim priority for the terminology that defines a particular method or typology. In order to resolve this lack of agreement, participants suggested that members of the mixed methods community should work towards building a greater consensus on terminology:

“There needs to be a common language” (Educationalist #9).

Criticism 3 : Mixed methods research is not a new type of methods practice . Some participants noted the tendency in the literature to present mixed methods as a new type of research practice that emerged during the past three decades. They pointed out that the use of mixed methods has a prior history that considerably predates the time when it became formalized as a research field. These participants cited examples of studies in sociology by Jahoda and Zeisel (Marienthal study of unemployment) and fieldwork in anthropology by Margaret Mead, both dating from the early 20th century. While these studies had an influence on methodology in the social sciences on account of the ways in which they creatively combined multiple quantitative and qualitative data sources, they have been generally overlooked in the mixed methods literature:

“I don’t particularly think that [mixed methods research has allowed us to answer research questions which were left unanswered in the past] but what I do think is that , you know , do remember as well that mixed methods research does actually have a long history in Sociology” (Sociologist #7).

Domain 2: Philosophy.

Criticism 4 : Mixed methods research is not a third paradigm . A considerable number of participants argued against the idea of characterizing mixed methods research as a third paradigm. They found two major faults with this characterization. First, it relies on the idea of mixed methods research as an approach that is distinct from quantitative and qualitative methodologies. In the view of these participants, mixed methods approaches do not rely on singular elements that are distinct in their nature, philosophy, or procedures:

“So no , I think , ultimately , I’m probably , I’m not really convinced that is a distinct methodology (…) So I worry when , when the idea of something that’s very special about mixed methods is given a lot , is given too much primacy” (Sociologist #3). “I don’t think it’s helpful to see it as a separate approach in terms of actually conducting , you know , planning and conducting , the research…I certainly think it’s stretching it to see it as a different , as a separate paradigm…I think the whole idea of ‘paradigm’ is a little bit difficult” (Sociologist #5).

Second, the conceptualization of mixed methods as a paradigm presupposes a strong link between epistemology and method, that is, the identification of the use of mixed methods with a particular epistemological or ontological view, whereas, in fact, these are separate entities. Attaching epistemological and ontological assumptions to mixed methods research would weaken its functionality and creative potential:

“…if we restrict mixed methods to only one paradigm then we’re bottlenecking mixed methods into a certain area , and we restrict the functionality of it” (Nurse, #10).

Criticism 5 : Current discussions of mixed methods research conceive quantitative and qualitative research as separate paradigms . Related to the previous criticism, a number of participants noted that current conceptualizations of mixed methods take for granted the nature of the quantitative and qualitative approaches, conceiving them as separate paradigms based on particular philosophical assumptions, thus reinforcing the conventional divide between them and accentuating their differences:

“…the whole purpose , of course , of mixed methods is that it’s , that’s a paradigm , but I’m not convinced it is because it still draws on those conventional traditional paradigms…I find that’s likely less helpful because again it starts from the assumption that there is a strong division between qualitative and quantitative research” (Educationalist #2).

These participants stated that the mixed methods literature may have uncritically incorporated the methodological “rules” (conventions) that were dominant in the 1980s by associating qualitative research with the constructivist paradigm and quantitative research with the positivist paradigm. This linkage between philosophy and method may have been a result of the process of formalization of the methodology carried out by the “second generation of mixed methods researchers” (from 1980s to present), while the “first generation” (i.e., from the 1900s to 1980s) might not have had a philosophical problem:

“I think the biggest problem that mixed methods research is in right now is having adopted , without reflection the rules that were established in the mid 80’s on , on paradigms in quali and quanti…we have these pillars , these quali-quanti pillars and we’re working on these rooms…All these classical studies [from the first generation] had no problems in doing quali-quanti , it was only the attempt to formalize it which has actually created these , these problems” (Sociologist #6).

According to this participant, the association of the quantitative and qualitative approaches with particular epistemological stances contradicts the very nature of the mixed methods approach: if such philosophical and methodological differences between quantitative and qualitative research really existed, then the integration of the two approaches would not be possible:

“…the big problem with having adopted this [association] , on the one hand , it actually makes mixed methods impossible . So , it is not possible within one single design to argue that your da- , that there is a single and objectifiable reality out there , on the other hand , and there are multiple or no reality , there’s no reality” (Sociologist #6).

In the view of another participant, part of the mixed methods literature may have accentuated the differences between the two methodologies by representing their characteristics in different columns in a table, while ignoring the existence of methods that incorporate features of both approaches (e.g., survey containing both open and closed ended questions or qualitative studies that include descriptive statistics):

“I know what’s been recently suggested in the literature (…) I’m not even sure that I would say that we should have drawn a line between qualitative and quantitative as firmly as we have . A lot of the qualitative work that I do includes descriptive statistics” (Educationalist #7).

Criticism 6 : Superficiality of pragmatism . Some participants argued that authors in the mixed methods community sometimes characterize the notion of pragmatism in a superficial way by reducing it to merely eclecticism and confusing it with “practicalism”. In this way, these authors advocate a “what works” approach which may be useful when justifying the integration of the quantitative and qualitative methods, but this attitude distorts the nature of pragmatism by failing to consider its underlying theoretical and philosophical assumptions:

“…they tend to think that pragmatism is just the practicalities , and it’s just the technicalities…” (Educationalist #2).

One participant noted that the feebly argued debates on pragmatism to date may have led the mixed methods community to undervalue the important contribution this paradigm has made to the philosophical basis of empirical inquiry:

“I’ve probably never in my life seen such weak debates on pragmatism as I have in mix- , in the mixed methods debate . I mean if I think of this fabulous contribution that , that pragmatism as a philosophical discipline has made” (Sociologist #6).

Moreover, another participant observed that many researchers in the field have acquired their knowledge of pragmatism mainly from the descriptions of the mixed methods paradigm found in the literature, whereas a sound basis for pragmatism in mixed methods research practice would require consulting the seminal papers on pragmatism, such as those by John Dewey, Charles Sanders Pierce or William James:

“… from what I’ve read anything about pragmatism that’s in a mixed methods paper does tend to be superficial…you have to go right back to the original authors of pragmatism and I think sometimes when we speak about pragmatism in mixed methods research , students particularly ten- tend , maybe just read some articles in pragmatism and think they know about this , but I think it is important to go right back to Dewey and James and Pierce” (Nurse, #6).

Criticism 7 : Mixed methods research aligns with positivism . A few participants noted that some members of the mixed methods community tend to accord a higher status to the quantitative component because they consider that it is more objective and more closely embodies the scientific method. In their view, some researchers regard the qualitative component as mainly a supplement to the quantitative component. Consequently, researchers may fail to appreciate the added value that may be gained by using mixed methods research:

“qualitative research [is often used] to almost to kind of flesh out the , the , the quantitative aspects , so it’s a kind of embellishment rather than seeing it as something that might challenge some of the quantitative findings or might contribute to , to ultimately rephrasing the research question or to reanalyzing the , the quantitative data” (Sociologist #5).

Domain 3: Procedures.

Criticism 8 : Limitations of typologies . A number of participants criticized the tendency of some authors to present mixed methods designs and procedures from a typological perspective. Typologies are used in the mixed methods literature as classifications of methodological features, such as the timing and priority of the quantitative and qualitative components and the stage at which integration is carried out [ 7 , 9 ]. In the view of those participants, typologies are presented in the literature in a way that is excessively mechanical and prescriptive, unnecessarily simplifying the process of carrying out a mixed methods study by suggesting that a successful implementation of a mixed methods design can be carried out only by following a predefined set of steps:

“I’m arguing against approaches that I think are too sort of mechanical in the sense of laying out : ‘Ok , here’s categories A , B , C , D and E , and here are the rules for applying them . And if you just follow the rules , then you’ll be ok’” (Educationalist #1).

Participants noted that this approach entails four problems. First, in order to adapt their approach to the research questions that they need to answer, researchers may need to modify the guidelines suggested in the literature. Therefore, guidelines for the use of mixed methods designs should be only “guiding principles” that are adaptable to varying circumstances and able to take into account the interactions between the different elements of the design. One of these participants stated the following:

“I think , I mean , I started off by using…the sort of prescription…and it’s only when you start to get delve more into mixed methods… So , it’s , it’s really , I think [they should be] just guiding principles” (Nurse, #5).

Second, the typologies may curtail the creativity of researchers by restricting them to a series of predefined models that are considered the “correct” ways of combining quantitative and qualitative methods. As expressed by the following participant:

“…researchers are using mixed methods in such creative ways , it’s like , it’s just , when you read these designs and they can be just so , so different and they just don’t fit into , you know , the typologies” (Nurse, #3).

Third, rather than being empirically generated by examining how mixed methods research is actually carried out in practice, these typologies are the highly formalized result of a list of ideal designs formulated by mixed methods theorists, as noted by this participant:

“There were basically two different approaches [to the development of typologies] and the one that was most common was the sort of develop very formal systems…The opposite of that was Bryman who went out and interviewed qualitative researchers about what they did…he talked to people about what they really did rather than coming up with formal systems” (Sociologist #1).

Fourth, the existing typologies are too extensive, which makes them difficult for inexperienced researchers to apply, as we can see in the following quote:

“…there must be forty or fifty different designs associated with mixed methods and I think it’s , you know , I think that’s confusing to people and it’s … in some way I think it becomes irrelevant” (Psychologist #9).

Criticism 9 : Procedures described in the literature are not aligned with mixed methods practice . Some participants mentioned occasional discrepancies between the procedures explained in textbooks and articles and the implementation and reporting of mixed methods in practice, which may not always conform to published guidelines and typologies.

“you , you open any textbook…and the rules that are proposed there are broken every day very successfully by researchers who , who actually conduct the research…the practice and the debate need to run parallel and they probably , right now I think they are a bit too separate from each other” (Sociologist #3).

Participants attributed this disjuncture to the fact that a few influential authors probably lacked sufficient practical experience in using mixed methods. These authors may have tended to suggest methodological guidelines “from their desk” without testing them in practice or reviewing the empirical work of other researchers:

“…there might be a gap that , that a lot of researchers talk about using mixed methods , but I’m not sure if they actually do it in practice” (Psychologist #2).

Furthermore, participants also noted that in some cases those authors may have placed greater emphasis on the philosophical and theoretical basis of mixed methods than on describing the techniques involved in implementing mixed methods research:

“…people spend far too much time talking about epistemology , most of those discussions are actually very simple , but people make them very complicate…I think…technical questions about how you work with the data and what it means [are more important]” (Educationalist #6).

Domain 4: Politics.

Criticism 10 : Mixed methods research is not better than monomethod research . Some participants pointed out a tendency among some members of the mixed methods community to consider this type of research to be inherently superior to monomethod research:

“…thinking about the , the papers that I’ve reviewed have been for the kind of applied end journals…I think the main issue for me has been in terms of , you know , the purpose of using mixed methods; that I think there’s a tendency to slip into thinking that more is necessarily better” (Sociologist #9).

Participants noted that to attribute a higher status to mixed methods research is wrong because this view could lead to the oversimplification of other approaches, which would undermine their prestige. Furthermore, participants argued that a mixed methods approach is not always the best research option and a fully integrated design may not be the most appropriate. What really determines the suitability of an approach or a design is the research question of a study, so that a monomethod design is sometimes the most appropriate.

Criticism 11 : Homogenization of mixed methods research . A few participants criticized a tendency in the mixed methods field to homogenize terminology and procedures. In their view, some members of the community have tried to develop a “mixed methods way of doing things” which would be acceptable to all researchers and would require them to write in a particular way using particular terminologies and strategies:

“They’re , they’re trying to develop a language , they’re trying to develop an approach , a strategy that , that is going to be acceptable by all mixed methods researchers , which really is , is unacceptable” (Educationalist #2).

This attitude towards homogenization of mixed methods research could hinder the advancement of the field since it promotes a uniform approach, suppresses intellectual disputes and ignores the diversity of approaches and attitudes regarding mixed methods found in the literature. As one participant argued, to find space for legitimate difference in the field is very difficult due to the protectionist attitude of some prominent authors who are interested in propagating their own ideas rather than incorporating the ideas of other authors:

“I fear that there’s , among , among those who have some prominence , there are some who would be very eager to protect their own turf and not wanting to come together for some kind of joint effort” (Educationalist #3).

RQ2: What differences and similarities can be identified in the criticisms reported by researchers working in different disciplines?

Of the 27 participants who raised criticisms, ten were sociologists, eight were educationalists, five were nurses, and four were psychologists. Multiple correspondence analysis was used to analyze the differences, depending on their discipline, in the types of criticisms the participants raised. Fig 1 shows the multiple correspondence analysis map for the first two axes. Highly associated categories are plotted near to one another on the basis of their loading to the corresponding axes, while the least associated categories are plotted far from one another. Therefore, the axes should be interpreted based on the grouping seen on the map of the relative positions of the categories, which are expressed by the magnitude of the coordinates. The measures known as eigenvalues indicate how much of the categorical information is explained by each dimension. Higher eigenvalues indicate a greater amount of variance of the variables in that dimension [ 29 , 30 ].

Note: The Yes label indicates that the criticism was mentioned, while the No label indicates that the criticism was not mentioned.

https://doi.org/10.1371/journal.pone.0252014.g001

On Axis 1, which accounts for most of the variation in the data (eigenvalue of 92.4%), the yes categories of the criticisms associated with the domains of philosophy, politics and the essence of mixed methods are plotted on the right-hand side of the map, while the no categories of the same domains are on the left-hand side of the map. As shown in the figure, sociologists were more likely to formulate criticisms associated with the three abovementioned domains while psychologists and nurses were less likely to formulate criticisms associated with any of these three domains. Educationalists were less inclined than sociologists to formulate criticisms associated with those domains, as indicated by the proximity of the education category to the centroid (the center of the axis). Finally, the fact that the yes and no categories of the procedures domain are plotted on Axis 2, which has very low explanatory power (eigenvalue of 7.6%), reveals that participants in the four disciplines showed no relevant differences in formulating this criticism.

Summary of findings

The aim of this study, based on a secondary analysis of interview data, was to describe the criticisms of mixed methods as a field raised by a sample of researchers in the disciplines of education, nursing, psychology, and sociology. Overall, the findings revealed a number of criticisms related to several different issues. These criticisms were initially identified by the first author in the data analysis phase of the original study, which was focused on the conceptualization and operationalization of quality in mixed methods research. The criticisms were particularly relevant since they were unprompted, that is, they were spontaneously given by the participants in response to questions about another subject, rather than to questions about criticisms. Furthermore, the original study was based on a broad and diverse sample of participants; it included a few researchers from the disciplines of sociology and education known for their criticisms as well as a considerable number of researchers whose publications seemed to be neutral on the subject of criticisms of the mixed methods field. To this latter group of researchers, the interviews afforded an opportunity to express their disagreement with some predominant notions in the mixed methods field which they otherwise might not have published. Another key element of this study is the examination of the relationship between the discipline of the participants and the frequency and type of criticisms they made. This has been possible since the sample is relatively balanced in terms of the number of participants from each of the four disciplines included in the study.

In response to RQ1, participants raised a total of eleven unprompted critical remarks, categorized in the following four domains: the essence of mixed methods (three criticisms), philosophy (four criticisms), procedures (two criticisms), and politics (two criticisms). Nine of the eleven critical comments have been previously mentioned in the literature. For instance, on the essence of mixed methods domain, some authors have alluded to problems such as the narrow definition of mixed methods research [ 15 , 34 – 36 ], the lack of agreement on the terminology used [ 16 , 37 ] and the inappropriateness of considering mixed methods a new methodology [ 35 , 38 , 39 ]. In the philosophical domain, several authors have criticized the dominance of a positivist approach to mixed methods research in some disciplines [ 37 , 40 , 41 ] while some authors have pointed out that considering as a separate or distinct paradigm can lead to an artificial separation of the quantitative and qualitative approaches [ 42 – 45 ]. In the procedures domain, a number of authors cited the problems inherent in conceptualizing mixed methods designs typologically, since such a view is restrictive and unable to reflect the variety of mixed methods designs used in practice [ 38 , 46 , 47 ]. Finally, in the politics domain, a few authors have criticized a tendency, in some of the literature, to homogenize the field [ 48 ], while others have critically noted the occasional adoption of a universalist position based on the idea that the mixed methods approach is inherently superior to monomethod research [ 44 , 47 ]. However, we also identified two criticisms not previously mentioned in the literature: the excessively superficial characterizations of pragmatism (criticism 6) found occasionally in the mixed methods literature and the description of procedures that are not necessarily in line with research practice (criticism 9).

Regarding criticism 6, some participants noted a tendency on the part of some researchers in the field to cite, when writing about pragmatism, what other mixed methods researchers had written about this paradigm rather than citing foundational writings, such as those by John Dewey, William James, or Charles Sanders Peirce. In fact, none of the most influential and most frequently cited textbooks on mixed methods research in the four disciplines we studied cites any work by key authors in the pragmatist tradition. Therefore, it could be useful to learn whether these authors’ highly synthetic explanation of foundational knowledge leads inexperienced researchers to only weakly engage with this paradigm, or, on the other hand, whether this simplification might help them to grasp the basic principles of pragmatism more quickly while leading them to consult first-hand the foundational writings.

With respect to criticism 9, participants reported that researchers do not always follow in practice all of the procedures described in mixed methods textbooks. This disjuncture between textbook guidance and research practice has been described in several methodological reviews of the use of mixed methods in the four disciplines we included in our study. Features such as explicitly stating the mixed methods design used, reporting mixed methods research questions, or explicitly stating the limitations associated with the use of a mixed methods design are regarded by some authors as key characteristics of mixed methods studies (Creswell and Plano Clark, 2018; Plano Clark and Ivankova, 2016; Onwuegbuzie and Corrigan, 2014; O’Cathain et al, 2008). However, Bartholomew & Lockard (2018) reported that very few of the studies included in their review of the use of mixed methods in psychotherapy explicitly stated the mixed methods design used (13%) or reported mixed methods research questions (29%). Additionally, Bressan et al. (2017) and Irvine et al. (2021), in their reviews on mixed methods in nursing, found that most of the studies they included failed to report the limitations associated with the use of a mixed methods design. Therefore, it could be of great interest to study whether the omission of these characteristic features of mixed methods studies reflects the researchers’ view that these features are unimportant, or whether they are unfamiliar with reporting standards. Identifying this latter criticism is a particularly relevant finding of this study, since the intimate context of the interview might have led to the expression of subjective judgments that otherwise might not have come to light (i.e., the participant’s perception that some authors may not habitually carry out empirical research).

With respect to RQ2, we found relevant differences in the type of criticisms raised across disciplines. In fact, one of our key findings is that criticisms in the procedures domain were equally distributed across the four disciplines, while criticisms in the essence of mixed methods, philosophy and politics domains were clearly more common in sociology. First, these findings are consistent with statements made by Plano Clark and Ivankova [ 3 ] regarding the ways in which the sociocultural context of researchers — including the discipline in which they work — can shape their beliefs, knowledge and even experiences with regards to mixed methods. Indeed, the greater number of criticisms made by sociologists categorized in three of the four domains shows how disciplinary conventions might affect how researchers think about mixed methods and judge the acceptability of certain predominant conceptualizations. According to several authors [ 49 – 51 ], critique is a foundational and distinct feature of the discipline of sociology. Therefore, the generalized tendency among sociologists to question traditional assumptions about the order of the world and to detach themselves from predominant belief systems and ideologies might help to explain why many sociologists in our sample criticized ideas such as the conceptualization of quantitative and qualitative research as separate entities and the consideration of mixed methods as inherently better than monomethod research.

Theoretical implications of the study

The findings of our study highlight several differences in opinion in the mixed methods field previously identified by authors such as Greene [ 52 ], Tashakkori and Teddlie [ 17 ], Leech [ 53 ] and Maxwell, Chmiel, and Rogers [ 54 ], among others. Those authors showed that, owing to differences in philosophical and theoretical stances and their adherence to different research cultures, researchers in this field sometimes disagree on foundational issues such as nomenclature, the need for consensus, and the definition of mixed methods research. In our study, the participants, particularly those in the field of sociology, made several criticisms about how some foundational and philosophical aspects of mixed methods research have been conceptualized by the mixed methods community, including how mixed methods has been defined and accorded status as a third paradigm. Furthermore, our findings showed contradictory criticisms formulated by the participants as a group: while some researchers criticized the lack of a consensus in the field on the terminology used to describe mixed methods research (criticism 2), others criticized a tendency by some authors to homogenize terminology (criticism 11).

This divergence of views is consonant with the notion of communities of practice suggested by Denscombe [ 12 ]. Departing from Kuhn’s notion of paradigms as “shared beliefs among the members of a specialty area” (as cited by Morgan [ 10 ]), in Denscombe’s view, the broader mixed methods community is a paradigm encompassing a conglomerate of multiple research communities shaped by the principles, values and interests prevailing in their disciplines and research orientations. In line with that author’s view that methodological decisions and viewpoints “will be shaped by a socialization process involving the influence of peers” [ 12 ], our findings suggest that the disciplinary community of our participants is likely to have informed their criticisms of the mixed methods field. Although this divergence of views suggests that complete agreement and unhindered communication among researchers is not possible [ 55 ], Ghiara [ 56 ] argued that, in Kuhn’s view, some form of communication is always possible; and furthermore, conflicting viewpoints can be reconciled to a certain extent. Similarly, Johnson [ 57 ] has argued that this diversity of critical voices, rather than being a problem, merely indicates that “reality is likely plural”—there is no single set of ontological assumptions underlying mixed methods—and, furthermore, knowledge is articulated on the basis of “multiple standpoints and strategies for learning about our world” that can be reconciled to some degree. An example of the healthy coexistence of divergent viewpoints within the mixed methods community can be found in the Journal of Mixed Methods Research , the leading journal in the field, described in its webpage as a “primary forum for the growing community of international and multidisciplinary scholars of mixed methods research”. The journal publishes a wide range of manuscripts, including articles revealing approaches to mixed methods research that rest on divergent foundational and philosophical perspectives.

The desire for inclusion of divergent viewpoints should not lead researchers to ignore the challenges posed by this divergence. Due to the interdisciplinary nature of mixed methods research, Curry et al. [ 58 ] argue that mixed methods teams often include researchers with different methodological backgrounds and propensities. Occasionally, these differences may pose challenges for establishing effective collaboration and for efficiently integrating research methods. A greater awareness of multiple perspectives on mixed methods research, including divergent critical views like those reported in our study, could help researchers better anticipate difficulties that might present themselves in the course of working with researchers who hold differing viewpoints. Furthermore, as Maxwell, Chmiel, and Rogers [ 54 ] have suggested, a better understanding on the part of mixed methods researchers of the perspectives of others in the field who embrace a differing approach should facilitate the process of integrating quantitative and qualitative methods in studies where different ontological positions coincide. In a similar vein, such an understanding could also help overcome a form of methodological tokenism described by Hancock, Sykes and Verma [ 59 ]. This can occur when mixed methods researchers fail to attend to, and therefore align, the distinct epistemological and ontological premises that underlie the methodological orientations that are integrated in a mixed methods design.

Furthermore, the recognition of mixed methods researchers’ divergence of views should be an integral part in any effort to design and implement a curriculum for mixed methods research. Plano Clark and Ivankova [ 13 ] have pointed out that any lack of clarity concerning the existing disagreements about foundational elements of the mixed methods approach could be confusing to researchers inexperienced in this field. Therefore, it is essential that courses and workshops on mixed methods research take note of these criticisms. This last point is particularly important since the topic of critical viewpoints is not included in any of the mixed methods syllabus exemplars published in the literature, including those by Earley [ 60 ], Christ [ 61 ], and Ivankova and Plano Clark [ 13 ].

Limitations, strengths, and possibilities for future research

Our findings are subject to a few limitations. First of all, the interviews were carried out by Skype and telephone. While these two forms of communication allowed us to interview participants residing in various locations around the world, they limited the possibilities for building the sort of rapport that might have encouraged some participants to elaborate more in their responses. To minimize this limitation, participants were given the opportunity in the member-checking phase to add additional insights to their initial statements. Second, the transferability of our findings is limited by our decision to include only four disciplines while excluding the views of researchers working in other disciplines that also have a high prevalence of mixed methods studies, such as medicine, business, and information science. From our findings alone, it is not possible to infer how frequent these criticisms are and what types of criticisms may be more prevalent in each of the disciplines. Third, multiple correspondence analysis is an exploratory method not appropriate for testing hypotheses or statistical significance. In other words, the method is designed to describe associations between categorical variables rather than to make predictions about a population [ 31 ]. Therefore, in light of the limitations of multiple correspondence analysis for drawing deeper inferences, the findings regarding RQ2 should be considered provisional and subject to further investigation. Finally, since this study is based on a secondary analysis, the interview questions did not specifically prompt the participants to bring up criticisms since the questions were focused on participants’ views related to the quality of mixed methods research. If we had specifically prompted participants to report their own criticisms, it is likely that more critical opinions would have been gathered.

While the use of secondary data entailed certain limitations, it also conferred some advantages. Participants’ critical statements were entirely spontaneous since they were not explicitly solicited. This spontaneity probably helped to reduce the social desirability bias, that is, a presumed tendency for respondents to dissimulate their own critical views in a way that might seem professionally and socially more acceptable. Furthermore, despite the limitations of multiple correspondence analysis, this method allowed us to generate a parsimonious visual representation of the underlying patterns of relationships among the criticisms and the disciplines. This representation helped us to improve our interpretation of the qualitative findings and to identify useful leads for carrying our further analysis. A further strength is that the study included a broad sample of participants in terms of their geographic location, academic position, seniority, and methodological expertise. This diversity probably also afforded us access to a wider range of views. Finally, a key strength of our study is that, to our knowledge, it is the first empirical study that has addressed the topic of criticisms of mixed methods as a field.

This study represents a step towards a better understanding of some current criticisms of mixed methods research. However, further research will be needed to confirm and expand our findings. Future research based on a larger and more diverse sample of mixed methods researchers could extend the scope of our research questions and help researchers generalize from our findings. Such studies might help us discover whether researchers from other disciplines share the criticisms made by the participants in our study and whether those researchers harbor other criticisms of their own. Other analytical tools could be used to examine in greater detail how the circumstances and attributes of researchers — including their disciplinary background, methodological expertise and paradigmatic viewpoints — influence the way these scholars formulate their criticisms of the mixed methods field. Finally, future research is needed to examine the critical accounts of researchers less experienced in methodological writing and probably less exposed to current theoretical debates and developmental issues in the field.

Acknowledgments

The authors would like to acknowledge the help of Dick Edelstein in editing the final manuscript. The first draft of this paper was written while the first author was a visiting scholar in the Mixed Methods Program of the University of Michigan, directed by Michael D. Fetters, John W. Creswell, and Timothy D. Guetterman. This author would like to thank these three scholars for their support and, particularly Michael D. Fetters for his invaluable mentorship over the past few years.

• 1. Hesse-Biber S. Navigating a turbulent research landscape: Working the boundaries, tensions, diversity, and contradictions of multimethod and mixed methods inquiry. In: S Hesse-Biber, Johnson RB, editors. The Oxford handbook of multimethod and mixed methods research inquiry. New York, NY: Oxford University Press; 2015. p. xxxiii–liii.
• View Article
• Google Scholar
• 3. Plano Clark VL, Ivankova NV. Mixed methods research: A guide to the field. Thousand Oaks, CA: Sage; 2016.
• PubMed/NCBI
• 6. Ivankova NV, Kawamura Y. Emerging trends in the utilization of integrated designs in the social, behavioral, and health sciences. In: Tashakkori A, Teddlie C, editors. The SAGE handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks, CA: Sage; 2010. p. 581–612.
• 7. Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. 3rd ed. Thousand Oaks, CA: Sage; 2018.
• 8. Fetters MD. The mixed methods research workbook: Activities for designing, implementing, and publishing projects. Thousand Oaks, CA: Sage; 2020.
• 9. Tashakkori A, Johnson RB, Teddlie C. Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. 2nd ed. Thousand Oaks, CA: Sage; 2021.
• 11. Kuhn TS. The structure of scientific revolutions. 4th ed. Chicago: The University of Chicago Press; 2012.
• 17. Tashakkori A, Teddlie C. Epilogue: Current developments and emerging trends in integrated research methodology. In: Tashakkori A, Teddlie C, editors. The SAGE handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks, CA: Sage; 2010. p. 803–26.
• 18. Creswell JW. Controversies in mixed methods research. In: Denzin NK, Lincoln Y, editors. The SAGE handbook of qualitative research. 4th ed. Thousand Oaks, CA: Sage; 2011. p. 269–83.
• 21. Heaton J. Reworking qualitative data. London: Sage; 2004.
• 23. Patton MQ. Qualitative research & evaluation methods. 4th ed. Thousand Oaks, CA: Sage; 2014.
• 28. Schreier M. Qualitative content analysis in practice. London: Sage; 2012.
• 29. Greenacre M. Correspondence analysis in practice. 3rd ed. Boca Raton, FL: Chapman & Hall/CRC; 2017.
• 30. Le Roux B, Rouanet H. Multiple correspondence analysis. Thousand Oaks, CA: Sage; 2010.
• 31. Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate data analysis: A global perspective. 7th ed. Upper Saddle River, NJ: Pearson Education; 2010.
• 32. Bazeley P. Integrating analyses in mixed methods research. London: Sage; 2018.
• 33. Creswell JW. Mapping the developing landscape of mixed methods research. In: Tashakkori A, Teddlie C, editors. The SAGE handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks, CA: Sage; 2010. p. 45–68.
• 39. Pelto PJ. Mixed methods in ethnographic research: Historical perspectives. New York, NY: Routledge; 2017.
• 43. Niglas K. The multidimensional model of research methodology: An integrated set of continua. In: Tashakkori A, Teddlie C, editors. The SAGE handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks, CA: Sage; 2010. p. 215–36.
• 52. Greene JC. Mixed methods in social inquiry. San Francisco, CA: Jossey-Bass; 2007.
• 53. Leech NL. Interviews with the early developers of mixed methods research. In: Tashakkori A, Teddlie C, editors. The SAGE handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks: Sage; 2010. p. 253–72.
• 54. Maxwell JA, Chmiel M, Rogers SE. Designing integration in multimethod and mixed methods research. In: Hesse-Biber S, Johnson RB, editors. The Oxford handbook of multimethod and mixed methods research inquiry. New York: Oxford University Press; 2015. p. 223–39.

Research design: qualitative, quantitative, and mixed methods approaches / sixth edition

• Published: 15 November 2023
• Volume 58 , pages 1011–1013, ( 2024 )

Cite this article

• James P. Takona   ORCID: orcid.org/0009-0001-4591-8136 1  

4226 Accesses

Explore all metrics

This review examines John W. Creswell and David Creswell’s sixth edition, which covers the most popular research methods, offering readers a comprehensive understanding and practical guidance in qualitative, quantitative, and mixed methods. The review includes observations on existing drawbacks, gaps, and ideas on potential areas for improvement in the book. The book is an excellent entry point for understanding the three broad research designs. It stands out for incorporating various methods and empowering researchers to effectively align them with specific research questions, objectives, and philosophical underpinnings. However, it could be further refined by incorporating newer research approaches and expanding practical aspects such as data collection, sampling strategies, and data analysis techniques. With these improvements, the sixth edition could further solidify its position as a comprehensive and accessible guide adeptly catering to researchers, educators, and students. Despite the book’s many strengths, there are opportunities for refinement in future editions, incorporating newer approaches to research designs and expanding practical aspects such as data collection, sampling strategies, and data analysis techniques. This review highlights that, with these suggested improvements, future editions could not only maintain but also enhance the text’s comprehensive and accessible nature, further solidifying its status as a vital resource for researchers, educators, and student.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Research Design and Methodology

Research Methods and Approach to Analysis Within Chapters

Types of Research Designs

Author information, authors and affiliations.

Coppin State University, Maryland, USA

James P. Takona

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to James P. Takona .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Takona, J.P. Research design: qualitative, quantitative, and mixed methods approaches / sixth edition. Qual Quant 58 , 1011–1013 (2024). https://doi.org/10.1007/s11135-023-01798-2

Download citation

Published : 15 November 2023

Issue Date : February 2024

DOI : https://doi.org/10.1007/s11135-023-01798-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative research
• Quantitative research
• Mixed methods
• Practical guidance
• Research process
• Research designs
• Find a journal
• Publish with us
• Track your research

• News & Highlights

• Publications and Documents
• Postgraduate Education
• Browse Our Courses
• C/T Research Academy
• K12 Investigator Training
• Translational Innovator
• SMART IRB Reliance Request
• Biostatistics Consulting
• Regulatory Support
• Pilot Funding
• Informatics Program
• Community Engagement
• Diversity Inclusion
• Research Enrollment and Diversity
• Harvard Catalyst Profiles

Community Engagement Program

Supporting bi-directional community engagement to improve the relevance, quality, and impact of research.

• Getting Started
• Resources for Equity in Research
• Community-Engaged Student Practice Placement
• Maternal Health Equity
• Youth Mental Health
• Leadership and Membership
• Past Members
• Study Review Rubric
• Community Ambassador Initiative
• Implementation Science Working Group
• Past Webinars & Podcasts
• Policy Atlas
• Community Advisory Board

For more information:

Mixed methods research.

According to the National Institutes of Health , mixed methods strategically integrates or combines rigorous quantitative and qualitative research methods to draw on the strengths of each. Mixed method approaches allow researchers to use a diversity of methods, combining inductive and deductive thinking, and offsetting limitations of exclusively quantitative and qualitative research through a complementary approach that maximizes strengths of each data type and facilitates a more comprehensive understanding of health issues and potential resolutions.¹ Mixed methods may be employed to produce a robust description and interpretation of the data, make quantitative results more understandable, or understand broader applicability of small-sample qualitative findings.

Integration

This refers to the ways in which qualitative and quantitative research activities are brought together to achieve greater insight. Mixed methods is not simply having quantitative and qualitative data available or analyzing and presenting data findings separately. The integration process can occur during data collection, analysis, or in the presentation of results.

¹ NIH Office of Behavioral and Social Sciences Research: Best Practices for Mixed Methods Research in the Health Sciences

Basic Mixed Methods Research Designs 

View image description .

Five Key Questions for Getting Started

• What do you want to know?
• What will be the detailed quantitative, qualitative, and mixed methods research questions that you hope to address?
• What quantitative and qualitative data will you collect and analyze?
• Which rigorous methods will you use to collect data and/or engage stakeholders?
• How will you integrate the data in a way that allows you to address the first question?

Rationale for Using Mixed Methods

• Obtain different, multiple perspectives: validation
• Build comprehensive understanding
• Explain statistical results in more depth
• Have better contextualized measures
• Track the process of program or intervention
• Study patient-centered outcomes and stakeholder engagement

Sample Mixed Methods Research Study

The EQUALITY study used an exploratory sequential design to identify the optimal patient-centered approach to collect sexual orientation data in the emergency department.

Qualitative Data Collection and Analysis : Semi-structured interviews with patients of different sexual orientation, age, race/ethnicity, as well as healthcare professionals of different roles, age, and race/ethnicity.

Builds Into : Themes identified in the interviews were used to develop questions for the national survey.

Quantitative Data Collection and Analysis : Representative national survey of patients and healthcare professionals on the topic of reporting gender identity and sexual orientation in healthcare.

Other Resources:

  Introduction to Mixed Methods Research : Harvard Catalyst’s eight-week online course offers an opportunity for investigators who want to understand and apply a mixed methods approach to their research.

Best Practices for Mixed Methods Research in the Health Sciences [PDF] : This guide provides a detailed overview of mixed methods designs, best practices, and application to various types of grants and projects.

Mixed Methods Research Training Program for the Health Sciences (MMRTP ): Selected scholars for this summer training program, hosted by Johns Hopkins’ Bloomberg School of Public Health, have access to webinars, resources, a retreat to discuss their research project with expert faculty, and are matched with mixed methods consultants for ongoing support.

Michigan Mixed Methods : University of Michigan Mixed Methods program offers a variety of resources, including short web videos and recommended reading.

To use a mixed methods approach, you may want to first brush up on your qualitative skills. Below are a few helpful resources specific to qualitative research:

• Qualitative Research Guidelines Project : A comprehensive guide for designing, writing, reviewing and reporting qualitative research.
• Fundamentals of Qualitative Research Methods – What is Qualitative Research : A six-module web video series covering essential topics in qualitative research, including what is qualitative research and how to use the most common methods, in-depth interviews, and focus groups.

View PDF of the above information.

• Open access
• Published: 30 April 2024

Identifying barriers to outpatient appointment attendance in patient groups at risk of inequity: a mixed methods study in a London NHS trust

• Beatrice Sung 1   na1 ,
• Fiona O’Driscoll 2   na1 ,
• Alice Gregory 2 ,
• Kate Grailey 2 ,
• Hannah Franklin 3 ,
• Sharon Poon 3 ,
• Anna Lawrence-Jones 4 , 5 ,
• Leila Shepherd 2 ,
• Clare McCrudden 2 ,
• Bob Klaber 3 , 6 ,
• Chris Pavlakis 2 ,
• Ara Darzi 4 &
• Sarah Huf 4  

BMC Health Services Research volume  24 , Article number:  554 ( 2024 ) Cite this article

248 Accesses

6 Altmetric

Metrics details

There is significant health inequity in the United Kingdom (U.K.), with different populations facing challenges accessing health services, which can impact health outcomes. At one London National Health Service (NHS) Trust, data showed that patients from deprived areas and minority ethnic groups had a higher likelihood of missing their first outpatient appointment. This study’s objectives were to understand barriers to specific patient populations attending first outpatient appointments, explore systemic factors and assess appointment awareness.

Five high-volume specialties identified as having inequitable access based on ethnicity and deprivation were selected as the study setting. Mixed methods were employed to understand barriers to outpatient attendance, including qualitative semi-structured interviews with patients and staff, observations of staff workflows and interrogation of quantitative data on appointment communication. To identify barriers, semi-structured interviews were conducted with patients who missed their appointment and were from a minority ethnic group or deprived area. Staff interviews and observations were carried out to further understand attendance barriers. Patient interview data were analysed using inductive thematic analysis to create a thematic framework and triangulated with staff data. Subthemes were mapped onto a behavioural science framework highlighting behaviours that could be targeted. Quantitative data from patient interviews were analysed to assess appointment awareness and communication.

Twenty-six patients and 11 staff were interviewed, with four staff observed. Seven themes were identified as barriers – communication factors, communication methods, healthcare system, system errors, transport, appointment, and personal factors. Knowledge about appointments was an important identified behaviour, supported by eight out of 26 patients answering that they were unaware of their missed appointment. Environmental context and resources were other strongly represented behavioural factors, highlighting systemic barriers that prevent attendance.

This study showed the barriers preventing patients from minority ethnic groups or living in deprived areas from attending their outpatient appointment. These barriers included communication factors, communication methods, healthcare the system, system errors, transport, appointment, and personal factors. Healthcare services should acknowledge this and work with public members from these communities to co-design solutions supporting attendance. Our work provides a basis for future intervention design, informed by behavioural science and community involvement.

Peer Review reports

Health inequity, defined as avoidable and unfair differences in health access, outcomes and experience between groups or populations, is a major issue faced by patients globally [ 1 , 2 ] and in the United Kingdom (U.K.) [ 3 ]. This inequity is long-standing and widespread, for example with racial inequity resulting in Black women having a four-fold increase in maternal mortality rates and Asian women having a two-fold increase, compared to white women [ 4 ]. Inequity has also been demonstrated in data arising during the COVID-19 pandemic, with the risk of dying once diagnosed with COVID-19 higher among people living in the most deprived areas compared to the least [ 5 ]. In current national policy from National Health Service (NHS) England [ 6 ] and government [ 7 ], there is a clear focus on reducing health inequalities, a term often used interchangeably with health inequity [ 6 , 7 , 8 ]. Health equality is often presented as synonymous with health equity, where people with equal health needs are treated equally [ 9 ]. However, health equity identifies the requirement to treat people with unequal needs differently, for them to have an equal health outcome [ 9 ]. NHS England’s Core20PLUS5 framework outlines several population groups that should be considered when designing actions to reduce healthcare inequalities, such as the most deprived 20% of the population and ethnic minority communities [ 6 ].

First outpatient appointments are a vital way of accessing secondary care and managing complex health problems, acting as a gateway into diagnostics, care pathways and treatments [ 10 ]. Recording whether a patient “Did Not Attend” (DNA) their appointment is a key measure for health services to track patient pathways and service utilisation [ 11 ]. Appointments that are reported as DNAs reflect those that are missed without the patient cancelling or rescheduling beforehand. A focus on reducing the rate of those who did not attend new outpatient appointments has the potential to improve health outcomes by providing patients with earlier diagnosis and access to specialist expertise and treatments [ 11 , 12 , 13 , 14 ]. A patient’s history of missing an outpatient appointment is a factor that may contribute to future non-attendance [ 15 ], showing the significance of this healthcare interaction. Overall, this will ensure long-term management of chronic diseases and improve health outcomes.

One aspect of health equity is the ability of different groups or populations to be supported in accessing a range of healthcare services, such as outpatient appointments. It has been shown that some patient groups are more likely to not attend an outpatient appointment, with national data sources showing that patients from more deprived areas [ 16 ] and minority ethnic groups [ 17 , 18 ] were more likely to not attend their appointments. This inequity means certain populations may struggle to access healthcare due to various barriers, despite initiatives to improve service utilisation. These efforts include NHS Trusts proactively contacting patients from minority ethnic groups and deprived areas with offers of support for travel costs or parking before their appointment [ 17 ] and offering parents of children living in the most deprived areas free transport to appointments [ 19 ]. Existing literature has highlighted the barriers in attending outpatient appointments that can lead to non-attendance. A systematic review of studies reporting reasons for non-attendance found that “forgetfulness” was a common cause, as were health-related barriers, timing and compatibility issues, administrative errors and miscommunication, financial issues, and perceived negative outcomes of attending [ 20 ]. However, the impact of each barrier is reported variably – for example, forgetting to attend an appointment contributes to between 8 and 45% of DNAs [ 20 ]. This suggests that non-attendance is complex, and the barriers that different individuals face can depend on a multitude of factors. Communities facing inequity may experience different barriers or the same barriers but to varying extents. The literature suggests that patients from more socioeconomically deprived backgrounds experience more transport-related issues [ 20 ] compared to those from less deprived backgrounds, which may be due to the cost or reliability of public transport. To help individuals at risk of inequity access to healthcare services, the specific barriers they face should be investigated. Corresponding solutions should be co-designed with people from these communities to ensure interventions are tailored to their experiences and circumstances. This is required given the evidence that a “one size fits all” approach to supporting patients attend their appointments is not suitable [ 21 ].

Data from one NHS Trust in the U.K. highlighted the presence of inequity in their outpatient attendance rates. In preliminary work, this Trust found that there was a statistically significant difference in proportion of DNA rates for outpatient appointments based upon both deprivation and ethnicity in 11 clinical specialties. Patients whose ethnicity was recorded as “Black”, “mixed” or “other” Footnote 1 or who lived in an area scoring quintile 1 on the Index of Multiple Deprivation (IMD) (most deprived), were up to 50% more likely to not attend their outpatient appointment compared to patients with “White – British” ethnicity or who lived in the least deprived areas (IMD quintile 5).

These data created the rationale for this study – to explore the barriers to attendance in these patient groups identified as being more likely to DNA an outpatient appointment. Given the existing evidence showing that many barriers to attendance are related to wider healthcare and systemic issues, this was also evaluated [ 11 , 20 ].

Specific aims were to:

Understand the barriers to attendance at outpatient appointments faced by patients at risk of inequity based on ethnicity or the deprivation of where they live.

Explore the systemic factors that might contribute to DNAs.

Assess patient appointment awareness and the communication methods received by patients.

Study design

This project was undertaken as a mixed methods service evaluation in line with the outcomes for the Health Research Authority (HRA) decision tool [ 23 ] and received the relevant approvals from the clinical governance team at the study site (reference number 830). Qualitative methods, including semi-structured interviews and observation were used to identify barriers to attendance and explore systemic factors. The interview discussion guides included primarily semi-structured questions that were flexible and facillitated the gathering of rich insights and experiences. Structured, closed questions method were additionally incorporated to quantitatively evaluate appointment awareness and communication methods for patients who had missed their appointment. The study is reported in line with the Standards for Reporting Qualitative Research (SRQR) [ 24 ].

Study setting

Five clinical specialties within outpatient services at the study site were selected as the study setting, along with the Patient Service Centre (PSC). The five clinical specialties were ophthalmology, gastroenterology, colorectal surgery, plastic surgery, and ophthalmology; chosen due to the fact they were high-volume specialties and had a statistically significant disparity in outpatient attendance when considering ethnicity and deprivation. The PSC provides all centralised outpatient appointment booking services and receives inbound patient appointment queries and was therefore included given its importance in the outpatient appointment journey.

Participants

Patient participants included those who had not attended a recent first appointment at one of the five specialties, and whose ethnicity was recorded in the electronic healthcare record (EHR) as ‘Black’, ‘mixed’ or ‘other’ [ 25 ] or whose address had the highest level of deprivation (IMD quintile 1). Participants were identified using appointment data provided by the Trust Business Intelligence (BI) team. Staff participants worked in the outpatient department of one of the five clinical specialties or the PSC. They were identified based on their role and its relevance to the project aims. All staff interviewed had administrative, operational or coordinator roles in an outpatient department, and were involved in coordinating and booking an outpatient appointment for a patient. Staff and patients were recruited using a purposive sampling technique, to ensure the participant groups were reflective of the target populations.

Data collection

Data collection was carried out from March to April 2023. Two separate semi-structured discussion guides (provided as additional files) were developed based on the literature review and using the theoretical domains framework (TDF) as a behavioural science framework [ 26 ]. A behavioural science framework was chosen to underpin the interview discussions, given the study’s focus on understanding the determinants of patient behaviours with respect to them attending their outpatient appointment. It was anticipated that this understanding could be utilised in future work to design bespoke interventions that support attendance at outpatient appointments for groups facing inequity. The TDF is a behavioural science framework that combines different theories of behavioural change and provide researchers with 14 ‘domains’ which summarise different factors influencing behaviour. These domains cover different influences of behaviour, such as knowledge, skills and social identity [ 26 ]. The TDF was used to develop the discussion guide to focus the questions on potential behaviours affecting attendance. The discussion guides were reviewed by the research team and developed further as part of the team’s public engagement and involvement work. Feedback from a member of the project’s Public Steering Group (PSG) was incorporated. The PSG member (MH) suggested changes such as explaining the purpose of the phone calls at the beginning of the call with the patient before their details are confirmed, to address power dynamics for those with a history of trauma or negative lived experiences with authorities. Interview findings were used to iterate the discussion guides during the data collection period.

All patient interviews were conducted in English by team members using a hospital phone to contact patients on the phone number(s) recorded on the EHR. Attempted contact was made up to 3 times and all calls were made during working hours of 9am – 5pm, Monday – Friday. Each patient interview was carried out by an interviewer (AG) in a private, quiet room at the NHS Trust, with another member of the research team present acting as note-taker (FO) and transcribing notes verbatim. Verbal informed consent was gained prior to carrying out the interview after explaining the purpose of the study. The interviews were planned to last approximately 10 to 15 min given their unscheduled nature for participants.

To further explore the systemic factors that might contribute to DNAs, staff interviews were carried out either on video call or in-person at the staff member’s hospital location of work and lasted 30 min to one hour. Verbal informed consent was gained prior to carrying out the interview or observation. Where staff were also observed, this was carried out in-person during the interviews and varied depending on the staff role. For staff working in the clinical specialties, observations were carried out by asking them to demonstrate administrative processes and tasks described during the interview that are involved in coordinating and booking a patient’s appointment. Staff working in the PSC were observed by listening to staff respond to incoming patient calls and observing them carrying out their work in booking or rescheduling a patient’s appointment. For most staff interviews, there was one interviewer (AG) and one note-taker (FO) transcribing notes, except for the staff observations at the PSC where both researchers (AG and FO) interviewed and observed separate staff members due to the individual nature of their work receiving inbound patient calls. Recruitment for both patient and staff interviews ended when thematic saturation was reached as determined by the researchers, being the point at which no new themes were being identified through ongoing data capture [ 27 ].

To provide a quantitative analysis of whether patients were aware of their missed appointment, at the start of each interview they were asked whether they were aware of the appointment they had missed. If the participant was aware of the appointment, they were then asked what communication method they had received to notify them.

Data analysis

To analyse patient interview data and identify barriers to attendance, reflexive thematic analysis following an inductive approach was chosen. As outlined below, our thematic analysis process followed a six-phase approach: familiarisation with the dataset, generation of codes, construction of themes by collating codes, and then review, finalisation and naming of themes [ 28 ]. All patient interview notes were reviewed by one researcher to become familiar with the dataset (FO). The initial codes were generated by the researcher and themes were constructed by comparing codes and the corresponding data with each other. 10% of patient interview notes were analysed by a second team member (CM) and compared with the initial analysis. An initial framework with themes and subthemes was developed to represent the barriers and issues experienced by patients at risk of inequity based on their ethnicity or the deprivation of where they live.

Following the development of initial themes and subthemes, all staff interview notes were reviewed and coded by one researcher (FO). These data were analysed in a deductive approach with the context of the initial framework to explore the wider systemic issues that may contribute to DNAs. The additional insights gained from interviewing staff about barriers and wider issues added to and expanded the framework. This data triangulation contributed to the development of a comprehensive framework that summarises the issues affecting patients experiencing inequity alongside the wider systemic issues that may face patients. The process was recursive, and the last stage of the six-step process to finalise and name the themes was carried out through review and iteration with the project team. Any disagreement was discussed and resolved within the study team as part of the analysis process.

Quantitative analysis was performed for specific questions that were asked to all patients interviewed, to assess whether they had been aware of the missed appointment and if so, what method of communication they had received. This quantitative analysis involved calculating the percentage of patients interviewed who had been aware of the appointment, and the percentage of patients who received each communication type.

Behavioural science frameworks

As discussed above, the TDF was chosen as a behavioural science framework to provide a theoretical basis to the reasons contributing to missed appointments. It aims to provide a lens to explain the different influences on human behaviour, such as cognitive, social and environmental [ 29 ]. The themes were mapped onto the relevant 14 TDF domains in a deductive approach, to identify behaviour change components.

Researcher characteristics and reflexivity

The research team was comprised of a multidisciplinary team of designers, behavioural science researchers, policy fellows and patient and public involvement specialists. This diversity of thought and experience led to strong collaboration and communication from various team members throughout the study. The involvement of a public involvement specialist ensured the championing of the patient, public and carer perspective throughout the study, such as recruiting a public steering group (PSG) who were from minority ethnic groups or living in deprived areas to influence the study. Members of the PSG provided feedback on the discussion guides and reviewed project outputs. The researchers who conducted and analysed the interviews were aware of the potential of introducing bias through their own experiences and perspectives. To mitigate this, the discussion guides were used during the interviews and team input and reflections were sought throughout the thematic analysis and framework development process.

Patient demographics

A total of 26 patients who had missed their first outpatient appointment at one of the five clinical specialties, or a family member, were interviewed. Twenty-two (84.6%) patients interviewed had their ethnicity recorded in the EHR as “Black”, “mixed” or “other”. Nine (34.6%) patients lived in an area with the highest level of deprivation (IMD quintile 1). For six patients, the interview was conducted with a family member who managed their healthcare, such as a child or grandchild. In three of these cases, the family member also translated directly to the patient to allow them to participate. Further detail on these patient demographics is shown in Table  1 .

Initially, interviews were planned to last 10 to 15 min with patients. However, given the nature of these unscheduled calls, the team found that patients were often not available or unprepared to talk in-depth about appointment barriers, as they may be at work or carrying out other responsibilities and unable to take the call. To overcome this challenge, the data collection approach was modified to include an initial screening call to inform patients about the interview and decide a time when they would be called back for a longer interview. This allowed the patients to prepare for the interview and schedule it for a time that was suitable for them. One of these scheduled interviews lasted 30 min for which the patient was reimbursed for their time as per best practice guidelines [ 30 ].

Staff demographics

Eleven staff members from four of the five clinical specialities and the Patient Service Centre were interviewed, as shown in Table  2 . It was not possible to interview staff from the ophthalmology service due to staff unavailability during the study. Of these 11 staff participants, two members from cardiology and two members from the PSC were shadowed to gain a more detailed understanding of the day-to-day appointment booking process. Interviews lasted 30 min to one hour.

Thematic analysis

The thematic analysis and triangulation of all qualitative data led to the development of seven high-level themes, each with specific subthemes (Table  3 ). These themes represent the barriers and systemic issues that patients at risk of inequity, based on their ethnicity or deprivation, may face that can result in missed first outpatient appointments. After the initial coding, a second researcher reviewed 10% of patient interview notes. Most of the codes generated and compared by each team member were consistent, with any discrepancies discussed and added or removed. There were disagreements in the research group around naming of the subthemes and these were resolved through group discussion. For example, insights and quotes related to the subtheme ‘conflicting information’ were initially included within the ‘unclear or insufficient information’ subtheme, but through discussion it was determined that this was a separate subtheme. The TDF domains were mapped onto each subtheme, which highlighted the aspects of behaviour change that could support individuals to attend their appointments. The broad range of barriers and systemic issues identified in Table  3 reinforced the existing literature by demonstrating the variety of factors that influence the ability of patients to attend.

Theme 1: communication factors

Patients identified barriers they faced which limited the effectiveness of communication about their outpatient appointments. Even if patients received the communication on time, the lack of or unclear information provided within the communication modality meant that patients did not have the information needed to attend their appointments. The focus on support from family members to help with appointment communication was also evident.

“They told me that it was the second time they tried to arrange the appointment for me, but I didn’t receive anything for the first appointment. I don’t know – I didn’t receive anything​.” Patient 11; Ethnicity – Other; IMD – 2 . “Sometimes people use their number for more than one occasion. They don’t give you the details in the text message. I use the phone for my mum and dad – I don’t know who the appointment is for.” Patient 14; Ethnicity – Other; IMD – 2 . “My dad gets the letters, but sometimes he can be forgetful. I get all the texts and phone calls” – Daughter of Patient 5; Ethnicity – Black; IMD – 1 .

Staff interviews corroborated patient insights, by highlighting the provision of unclear or insufficient information and adding that there is an inability within the system infrastructure to provide tailored and accessible communication about appointments. Staff described how these issues result in patients struggling to access or understand appointment communications. Language was highlighted as an issue related to written and phone communications, such as the inability to send communication in different languages and the lack of provision of translators for phone communications with patients.

The automated letter that is sent from [name of hospital] doesn’t have the specific information needed for certain scans, such as instructions on fasting – Staff members 1 and 2 . So much information provided in the letters and patients cannot pull out the key information – Staff members 1 and 2 . The appointment information provided is not in any other language than English. Even if on the patient record their spoken language is not English, the documents cannot be sent in that language – Staff members 1 and 2 .

Theme 2: communication method

Patients identified issues related to specific communication methods used by healthcare services to provide information about their outpatient appointment, specifically phone, email, and letter reminders. Phone communication was a challenge as patients may not have access to a charged and working phone, although calls or texts close to the time of the appointment were seen as helpful as reminders. While some patients preferred letters as they did not require digital literacy skills or equipment, others found that they were easy to misplace or had issues with delivery. Emails were preferred by individuals with strong digital literacy skills as a source of immediate information, but others had setting up email reminders and access challenges.

“I like the letters, sometimes I lose my telephone, or it’s charging – sometimes there’s a problem with the telephone​” – Patient 12; Ethnicity – Asian; IMD – 1 . “The day before the appointment they ring me and say my appointment is the next day. I also get a text… To be honest with you, it’s got to be 2 or 3 days before the appointment so I can make space for that. I must admit I do forget” – Patient 24; Ethnicity – Black; IMD – 2 . “I have issues with the post to do with where I live. It’s not just the post that’s unreliable – because it’s a shared post box so they don’t always get to me” – Patient 21; Ethnicity – Black; IMD – 2 . “They’ve done an email for me – I haven’t used it myself, I’ve written it down somewhere. They have to explain to me how to use” – Patient 24; Ethnicity – Black; IMD – 2 .

Systemic issues identified by staff members related to how text reminders are sent to patients from an NHS phone number that they cannot call back, which is frustrating for patients as they cannot make enquiries or reschedule directly. Letters were most frequently mentioned by staff as a failed communication method, highlighting that patients often say that they have not received the appointment letter.

Texts come from an NHS number that they can text back on but can’t ring. When they do ring it back they just get a general voice message that says they’ve gotten a call from [name of Trust] – Staff member 11 . There is the option to print an automated letter when the appointment is made and this request is sent to [name of hospital] where it’s printed. However, the service doesn’t trust this system as often the patients say that they don’t receive these letters. Therefore, the service staff usually print their own letter – Staff members 1 and 2 . Postal strikes mean sometimes people don’t get a letter – staff member 5

Theme 3: healthcare system

The wider healthcare system, referring to the way in which healthcare services are set-up and organised, beyond the control of the outpatient services at the study site, was identified as a significant source of barriers by patients. Appointment delays and service capacity issues meant that several patients had chosen to seek private healthcare services, without being removed from the NHS waiting list. Several patients noted their negative perceptions of the outpatient experience and wider NHS services. Patients identified the lack of action or communication by their general practitioner (GP) to facilitate their outpatient appointment as an additional issue.

“If you’ve not seen someone for 2 years you need to ask the person if they’ve been seen elsewhere. I’ve been seen privately – that’s another person off the waiting list” – Patient 4; Ethnicity – Black; IMD – 2 . “No reminder text – nothing, it was a shambles. [name of Trust] is a big Trust, they should be doing this – sending a reminder​” – Patient 19; Ethnicity – Black; IMD – 4 . “Referred by the GP, they didn’t give any guidance about what to expect – didn’t communicate anything with that regard” – Patient 4; Ethnicity – Black; IMD – 2 . “My GP doesn’t do anything for me. After one and a half years they sent me to MRI. Then they refer me to the hospital and nothing. The NHS haven’t done anything for me” – Patient 14; Ethnicity – Other; IMD – 2 .

Theme 4: system error

Despite some patients intending to attend their appointment, rescheduling, and even been present waiting for their video or phone appointment, errors within the administrative system led to some patients being incorrectly recorded as DNAs. In certain cases, this led to patients being discharged from the outpatient service without having the opportunity to attend their first appointment. Other patients said that the missed appointment had been cancelled by the hospital beforehand.

“I was aware but I didn’t receive any call from them. I rang them and they said they’d give me a call on the day but I didn’t receive any call ​” – Patient 26 talking about a phone appointment; Ethnicity – Black; IMD – 3 . “They kept on pushing appointment forward, the appointment was on video. Eventually the date they gave me I joined the link, was waiting in the lobby for ages and no one came. My experience has not been great” – Patient 4; Ethnicity – Black; IMD – 2 . “Yeah, I was aware of the appointment, but I was also aware that the appointment was booked for the 31st March and so I didn’t attend. It was initially booked for the 9th and was changed to the 31st. I took the day off work just to attend the appointment on the 31st. On the 10th March I got a letter addressed to my GP and myself that I’ve been discharged from the service because I didn’t attend the appointment. But actually the appointment was rebooked for a different date on the 31st March. It’s bad that it was rebooked and I was discharged” – Patient 19; Ethnicity – Black; IMD – 4 .

These system errors were also highlighted in staff interviews, where staff shared that they could occur due to the complex administrative process behind appointment booking and communications.

Sometimes the clinics make a mistake and discharge a patient when they have only DNA’d once – Staff member 3 talking about the Trust policy, which states that patients may miss an appointment twice .

Theme 5: transport

Patients reported challenges when travelling to appointments, including issues with the patient transport service, public transport options, and general access needs. Difficulties related to the existing patient transport service included early pick-up times that interfered with their health needs or caring responsibilities, issues with getting collected by the service and delays in being transported home. While some patients preferred to use public transport, access needs meant that for others this was not possible.

“Either you have to go hours before your appointment and then hang around or coming back is even worse. I go with my dad – I take him and bring him back. I go past the patient lounge and it’s ridiculous – it’s just not an option and I would like it to be” – Daughter of Patient 5 talking about the patient transport service; Ethnicity – Black; IMD – 1 . “I’m 60 now, but I cannot use the underground any more. When I go on the underground I can’t breathe – It doesn’t agree with me at all so I have to use the buses. I have to leave in time because of roadworks and that… can’t get the train or the tube. I’ve got a freedom pass” – Patient 24; Ethnicity – Black; IMD – 2 .

Staff found it challenging to support patients with the patient transport service, as it must be booked directly by the patient through an external organisation. The study site has specialist national services, so they noted that this is particularly an issue for patients travelling long distances. They also highlighted that some patients struggle to attend early morning appointments as free transport passes only work within certain times.

Sometimes a transport request is in the referral letter but all we can do is give the transport number. The patient has to book it themselves – Staff member 3 . [Name of Trust] has some specialist cardiology services that are national, such as the national pulmonary hypertension service. This means that patients travel very long distances to attend appointments and may not be able to organise transport or are not eligible for the patient transport service – Staff members 1 and 2 . Bus pass doesn’t start working until 9.30am – staff member 3

Theme 6: appointment factors

Many patients had preferences for their outpatient appointment time and type, due to health needs, work, or caring responsibilities. However, this often is not reflected in the appointment given and as a result they cannot attend their scheduled appointment. Patients also reported that the appointment booking processes were inflexible or inaccessible, thus it was difficult for some patients to book or reschedule an appointment that suited their needs. Different appointment formats, such as video calls, intended to offer more flexibility were not deemed appropriate or useful by some patients.

“Generally I just receive an appointment, which is not always ideal for me – I don’t get to choose the appointment time​… Sometimes they send me an appointment for 10 in the morning and I can’t go because I’m with my mum” – Daughter of Patient 5; Ethnicity – Black; IMD – 1 . “The problem right now, the letters that come there’s no direct number to call. Most of the time when I ring it goes to a central place. They can ring me but I can’t ring them, can’t get to speak to any of them” – Patient 24; Ethnicity – Black; IMD – 2 . “If it’s video it’s not helpful – I don’t want to take time off work to attend a video consultation” – Patient 4; Ethnicity – Black; IMD – 2 .

Staff shared that they thought appointments provided over the phone and during the weekend supported patients to attend. However, they acknowledged that there are difficulties for patients attending weekend appointments when other outpatient support services are not available.

There are lots of weekend clinics now which should give more time options for people to choose from – may be easier to attend – Staff members 1 and 2 . More weekend clinics ongoing, but the call centre isn’t open at the weekend. So, if someone is sick over the weekend and can’t make their appointment, they can’t cancel and will show up as a DNA – Staff member 11 .

Theme 7: personal factors

Many personal factors were reported by patients that led to challenges in attending their appointments. Although some patients wanted to attend, they were forced to miss their appointment as it was scheduled at times when they had existing responsibilities or commitments. Unexpected circumstances which prevented patients from attending appointments also arose and the inability to reschedule, as highlighted above, meant that the appointment was not utilised. Several patients were sick and unable to attend or reported issues with taking time off work to attend a hospital appointment. Some participants struggled to remember to attend their appointments without reminders or keep track of multiple appointments.

“I was aware but I couldn’t make it – I had family issues​” – Patient 13; Ethnicity – Black; IMD – 1 . “I can’t go because I’m with my mum… I need someone to sit with my mum when I go” – Daughter of Patient 5; Ethnicity – Black; IMD – 1 . “Two people were sacked from the company and I couldn’t take a day off from work” – Patient 11; Ethnicity – Other; IMD − 2 .

Staff supplemented these insights, noting that certain specialties predominantly have older patient groups who may find it challenging to remember or attend hospital appointments, particularly if they are attending multiple services or sites.

In cardiology a lot of the patients are older, this means that they may forget about their appointments or often have a lot of appointments to keep track of – Staff members 1 and 2 . If a patient has two appointments – one in morning and one in afternoon, they may just decide not to go to both. Alternately, people usually have multiple diagnostics appointments and sometimes they are not able to be booked into them all at the same time – Staff members 1 and 2 .

Staff observations

Two members of the cardiology team and two members of the Patient Service Centre were shadowed as part of their interview. Data gathered during the observation were recorded alongside the interview notes to create a broader understanding of the systemic issues affecting outpatient attendance. Patients called the PSC for multiple reasons, such as to reschedule their appointment due to personal issues or to reschedule an appointment that they had missed without being aware, linking to the “personal factors” and “appointment factors” themes. By observing PSC staff take inbound calls, insights were gained about the appointment booking and rescheduling process, such as how many manual and complex processes are involved. These processes involve many individuals and departments and can be challenging for the PSC to coordinate on behalf of the patient given that they are highly manual and rely on human input, which related to the “healthcare system” and “system errors” themes.

Quantitative analysis

Eighteen of the 26 patients (69.2%) interviewed said they had been aware of the appointment they missed. When asked about how they had been made aware of the appointment, nine received a letter, two received a text, one received an email, two received both a letter and a text and one received an email and a text. Three patients did not answer the question, or their answers was unclear. Eight (30.8%) patients responded that they were not aware of the missed appointment.

Key findings

This study highlighted the key barriers contributing to DNAs in the patient groups most likely to be affected by inequity of access due to their ethnicity or deprivation. These barriers included communication factors and methods, the healthcare system, system errors, transport, appointment, and personal factors. While these barriers were identified primarily through patient interviews, they were triangulated with and supported by qualitative data obtained through discussions and observations of healthcare staff. The quantitative analysis of patient awareness about their missed appointment helped showcase certain barriers, such as issues with receiving appointment communications.

The study highlights the importance of identifying barriers for patient groups experiencing inequity as they may experience unique challenges that must be addressed to support appointment attendance, such as negative cultural perceptions of NHS services or language needs. The onus of this is on healthcare services to adapt their practices, tailor communication methods and provide support to specific communities. While some barriers faced by individuals can be influenced by practical offers such as reimbursement of travel costs or interpretation services, it is imperative that tailored, accessible hospital communications to patients about upcoming appointments are implemented.

Evidence in context of existing research

This work builds on the literature focusing on patient groups experiencing inequity, to identify the barriers preventing patients from attending scheduled outpatient appointments [ 20 ]. Our study findings are consistent with previous literature suggesting that the reasons behind missed appointments are complex and often intersect with and compound each other and that there are specific issues affecting populations experiencing inequity [ 20 ]. Patient forgetfulness is commonly cited as a primary reason for non-attendance [ 31 , 32 ] along with transport issues [ 33 , 34 ]. Compared to other studies [ 35 ], our findings had less emphasis on barriers related to associated cost of attending and loss of earning potential, which may have been due to the nature of the questions in our interview guide. However, our work builds upon this to further elaborate on these specific barriers and identify systemic issues, for example healthcare system errors, that can result in the incorrect recording of a missed appointment. Given its use in implementation research [ 26 ], the TDF adds evidence to the existing knowledge base by aiming to provide practical, implementation support when appropriate solutions are designed for barriers in the future. This study involved interviewing patients and staff from outpatient services in a London Trust, therefore providing evidence for a specific region and for communities who are often underrepresented in research.

Study strengths

There are multiple strengths that supported the conduct of this study and the validity of its findings. The dual study aims focusing on patient and staff perspectives ensured that a holistic and accurate representation of the patient barriers and systemic factors to missed appointments were identified. The process of completing interviews with staff and patients in parallel along with staff shadowing allowed barriers raised by patients to be understood in greater detail by investigating them from the system perspective with healthcare staff. Asking patients questions about appointment awareness at the same time as the interview allowed for quantitative data to be gathered in parallel. The research team’s responsiveness to adapt the patient interview process ensured that interviews were carried out at a time suitable to the patient and gave them adequate time to prepare.

Using the TDF to develop the discussion guide and map onto the framework of barriers and issues allowed the research team to understand potential supports required in the future to address these factors. While some TDF domains, such as memory or knowledge, can be addressed through co-designing practical interventions that address these barriers, many of the issues related to the wider environmental context and resources available to patients. This provides evidence that certain barriers are outside the control of individual patients, and, therefore, future healthcare and wider societal systems are required to change to support patients, particular from underserved groups, to attend appointments.

Study limitations

The present work has several limitations. Despite our study population including patients from minority ethnic groups, no translation services were offered to patients during the phone interviews. This meant that individuals unable to be interviewed through English were excluded from the study. As a result, fewer individuals currently facing language barriers when interacting with healthcare services were interviewed and their experiences not actively included.

A similar structural limitation of the study was carrying out the interviews during conventional working hours. This meant that many of those at work would not have answered the phone when called, and their perspectives were missed. As a result, our findings may have lacked perspectives from those who work at jobs or industries with traditional hours, and instead captured perspectives of those who are retired, unemployed, or work flexible hours.

An inherent selection bias in our work is the issue with poor quality contact information for patients in NHS systems. While some patients we attempted to contact were found to have inactivated phone numbers, others may have not answered the phone due to changing phone numbers and the EHR not being updated. Two studies on text message reminders for cervical screening showed that 36.4% and 38% of patients had an up-to-date mobile phone number recorded in their clinical record, with this varying by age and IMD subgroups of participants [ 36 ]. As a result, we were unable to interview patients who had incorrect contact information recorded, and therefore who were also more likely to have not received information about their missed outpatient appointment. In addition, data on other patient demographics such as age and education level were not collected, which may have provided information on potential confounding factors for some of the barriers identified.

Due to the study being carried out in the chosen five clinical specialties, we anticipate that findings are transferrable to similar outpatient clinical specialties experiencing the same pressures and with similar patient cohorts. However, the study setting is a large NHS Trust in London’s city centre, meaning that the findings may not be transferable to other settings, such as those in rural environments or healthcare providers outside the NHS.

Opportunities for future work

Several opportunities exist to replicate this work in other settings and develop future solutions to the barriers identified. By repeating this work in other specialties or study settings, this would allow more specific insights to be developed for populations or environments. To gain additional insights, further studies could include interpretation services or carry out interviews in local community spaces. In addition, further work should be performed to co-design interventions with public members to address the barriers identified in this study. This would ensure that acceptable and feasible interventions are developed alongside healthcare services to support individuals facing inequity attend their appointments. In this future work, public members should be involved from the beginning to address potential limitations. This work, and future initiatives, are supported by the NHS goals of reducing health inequalities [ 37 ], particularly in healthcare provision [ 38 ].

Patient groups experiencing inequity of outpatient access based on their ethnicity or the deprivation of where they live were found through patient interviews to face a variety of barriers to attending their appointments, such as communication needs and preferences, transport, appointment booking processes and systemic issues such as negative perceptions of wider NHS and health services. This was supported by staff interviews and observation to expand on the causes of missed appointments from a service perspective. Several patients were not aware of the appointment they had missed, highlighting barriers related to appointment awareness. Health services need to provide additional tailored support to individuals to attend their outpatient appointments, as opposed to a “one size fits all” approach. The application of a behavioural science framework provides a strong basis for co-designing future interventions to address these barriers.

Data availability

The discussion guides used in this study are provided as supplementary information files. The data generated or analysed during this study are available from the corresponding author on reasonable request.

Ethnicity data collected by NHS organisations uses national definitions where a limited number of ethnicity categories are provided as options. For example, the high-level category “other” includes two ethnicity categories – “other ethnic groups – Chinese” and “other ethnic groups – any other ethnic group” [ 22 ].

Abbreviations

Coronavirus Disease 2019

did not attend

general practitioner

electronic healthcare record

Health Research Authority

index of multiple deprivation

National Health Service

Patient Service Centre

public steering group

theoretical domain framework

United Kingdom

Anderko L. Achieving health equity on a global scale through a community-based, public health framework for action. J Law Med Ethics. 2010;38(3):486–9.

Article   PubMed   Google Scholar  

Global Health Council. Global health equity. 2022.

Marmot M, Allen J, Boyce T, Goldblatt P, Morrison J. Health Equity in England: the Marmot Review ten years on. London: Institute of Health Equity; 2020.

Google Scholar  

Knight M, Bunch K, Felker A, Patel R, Kotnis R, Kenyon S et al. Saving lives, improving mothers’ Care lessons learned to inform maternity care from the UK and Ireland Confidential enquiries into maternal deaths and morbidity 2019-21 2023 October.

Public Health England. COVID-19: review of disparities in risks and outcomes. 2020 August.

NHS England. Core20PLUS5 (adults)– an approach to reducing healthcare inequalities [cited 2023 January 3]. https://www.england.nhs.uk/about/equality/equality-hub/national-healthcare-inequalities-improvement-programme/core20plus5/#:~:text=Core20PLUS5%20is%20a%20national%20NHS,clinical%20areas%20requiring%20accelerated%20improvement .

Health and Care Act. 2022, Parliament of the United Kingdom, 2021-22 Sess. (2022).

McCartney G, Popham F, McMaster R, Cumbers A. Defining health and health inequalities. Public Health. 2019;172:22–30.

Article   CAS   PubMed   Google Scholar  

Raine R, Or Z, Prady S. Challenges, solutions and future directions in the evaluation of service innovations in health care and public health. Southampton, U.K.: Health Services and Delivery Research; 2016 May.

Royal College of Physicians. Modern outpatient care: Principles and practice for patient-centred outpatient care. 2023 April.

NHS England. Reducing did not attends (DNAs) in outpatient services 2023 [updated August 15 2023; cited 2023 November 6]. https://www.england.nhs.uk/long-read/reducing-did-not-attends-dnas-in-outpatient-services/ .

Aggarwal A, Davies J, Sullivan R. Nudge and the epidemic of missed appointments. J Health Organ Manag. 2016;30(4):558–64.

Hwang AS, Atlas SJ, Cronin P, Ashburner JM, Shah SJ, He W, et al. Appointment no-shows are an independent predictor of subsequent quality of care and resource utilization outcomes. J Gen Intern Med. 2015;30(10):1426–33.

Article   PubMed   PubMed Central   Google Scholar  

AlRowaili O, Ahmed M, Areabi AE. Factors associated with No-Shows and rescheduling MRI appointments. BMC Health Serv Res. 2016;16(1):679.

Article   Google Scholar  

Dashtban M, Li W. Predicting non-attendance in hospital outpatient appointments using deep learning approach. Health Syst (Basingstoke). 2022;11(3):189–210.

Campbell K, Millard A, McCartney G, McCullough S. Who is least likely to attend? An analysis of outpatient appointment DNA data in NHS Greater Glasgow & Clyde. NHS Health Scotland; 2015.

Abeyratne RNHS, England. 2022 July 4 2022. [cited 2023 November 17]. https://www.england.nhs.uk/blog/narrowing-inequalities-in-waiting-lists-in-leicester/ .

Q Exchange. Addressing health inequalities by co-developing change to reduce missed appointments 2023 [cited 2023 November 17]. https://q.health.org.uk/idea/2023/addressing-health-inequalities-by-co-developing-change-to-reduce-missed-appointments/ .

NHS England. Free transport reduces ‘was not brought’ rates for children at Midlands trust [cited 2023 November 17]. https://www.england.nhs.uk/about/equality/equality-hub/case-studies/free-transport-reduces-was-not-brought-rates-for-children-at-midlands-trust/ .

McLean S, Gee M, Booth A, Salway S, Nancarrow S, Cobb M, et al. Health Services and Delivery Research. Targeting the use of Reminders and notifications for Uptake by populations (TURNUP): a systematic review and evidence synthesis. Southampton (UK): NIHR Journals Library; 2014.

Wilson R, Winnard Y. Causes, impacts and possible mitigation of non-attendance of appointments within the National Health Service: a literature review. J Health Organ Manag. 2022;36(7):892–911.

NHS Digital. Ethnicity 2022 [updated November 23 2022; cited 2024 January 2]. https://digital.nhs.uk/data-and-information/data-collections-and-data-sets/data-sets/mental-health-services-data-set/submit-data/data-quality-of-protected-characteristics-and-other-vulnerable-groups/ethnicity .

Health Research Authority. Is my study research? 2017 [updated October 2022; cited 2023 November 28]. https://www.hra-decisiontools.org.uk/research/about.html .

O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014;89(9).

NHS England. NHS Data Model and Dictionary - Ethnic Category 2023 [updated June 28 2023; cited 2023 November 7]. https://www.datadictionary.nhs.uk/data_elements/ethnic_category.html .

Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7(1):37.

Lowe A, Norris AC, Farris AJ, Babbage DR. Quantifying thematic saturation in qualitative data analysis. Field Methods. 2018;30(3):191–207.

Braun V, Clarke V. Successful qualitative research: a practical guide for beginners. 1 ed. Sage Publications Ltd; 2013. p. 400.

Atkins L, Francis J, Islam R, O’Connor D, Patey A, Ivers N, et al. A guide to using the theoretical domains Framework of behaviour change to investigate implementation problems. Implement Sci. 2017;12(1):77.

National Institute for Health and Care Research. Payment guidance for researchers and professionals 2022 [updated July 2023; cited 2023 November 2]. https://www.nihr.ac.uk/documents/payment-guidance-for-researchers-and-professionals/27392 .

Mitchell AJ, Selmes T. Why don’t patients attend their appointments? Maintaining engagement with psychiatric services. Adv Psychiatr Treat. 2007;13(6):423–34.

Kumthekar A, Johnson B. AB1105 barriers to appointment compliance and the effect of reminder phone calls on Clinic Show rate at an academic Lupus Clinic in the Underserved Community. Ann Rheum Dis. 2015;74(Suppl 2):1270.

Bove AM, Gough ST, Hausmann LRM. Providing no-cost transport to patients in an underserved area: impact on access to physical therapy. Physiother Theory Pract. 2019;35(7):645–50.

PubMed   Google Scholar  

Syed ST, Gerber BS, Sharp LK. Traveling towards disease: transportation barriers to health care access. J Community Health. 2013;38(5):976–93.

Lin S, Shou BL, Bibee K. Understanding barriers to medical appointment keeping: a case-control study of predictive factors for no-shows and same-day cancellations in dermatology clinics in an academic medical center in the United States. JAAD Int. 2023;11:189–92.

Huf S, Kerrison RS, King D, Chadborn T, Richmond A, Cunningham D, et al. Behavioral economics informed message content in text message reminders to improve cervical screening participation: two pragmatic randomized controlled trials. Prev Med. 2020;139:106170.

NHS England. The NHS Long Term Plan. 2019 January.

NHS England. Our approach to reducing healthcare inequalities [cited 2024 January 3]. https://www.england.nhs.uk/about/equality/equality-hub/national-healthcare-inequalities-improvement-programme/our-approach-to-reducing-healthcare-inequalities/ .

Download references

Acknowledgements

The research team would like to thank the Patient Service Centre, cardiology, colorectal surgery, ophthalmology, gastroenterology, and plastic surgery teams for their support in carrying out the study. We would also like to thank Michelle Dixon for her support during the study.

This study received funding from Imperial College London’s UKRI Impact Acceleration Account (IAA). The UKRI IAA funding sources for this project are NIHR Imperial BRC and MRC. The wider project has also received funding from ICHT.

Author information

Beatrice Sung and Fiona O’Driscoll contributed equally to this work.

Authors and Affiliations

University of Bath, Bath, UK

Beatrice Sung

Helix Centre, Institute of Global Health Innovation, Imperial College London, Room 1035/7, QEQM Wing, St Mary’s Campus, London, W2 1NY, UK

Fiona O’Driscoll, Alice Gregory, Kate Grailey, Leila Shepherd, Clare McCrudden & Chris Pavlakis

Imperial College Healthcare NHS Trust, London, UK

Hannah Franklin, Sharon Poon & Bob Klaber

Institute of Global Health Innovation, Imperial College London, London, UK

Anna Lawrence-Jones, Ara Darzi & Sarah Huf

National Institute for Health and Care Research (NIHR) North West London Patient Safety Research Collaboration (PSRC), London, UK

Anna Lawrence-Jones

Imperial College London, London, UK

You can also search for this author in PubMed   Google Scholar

Contributions

FO and AG carried out the patient and staff interviews. FO performed the initial thematic analysis and supplementary data analysis. CM carried out the second review of interview notes. Most authors (FO, AG, KG, HF, SP, ALJ, LS, CMC, BK and SH) contributed to the overall conduct and analysis of the study. BS and FO were major contributors in writing the manuscript. All authors (BS, FO, AG, KG, HF, SP, ALJ, LS, CMC, BK, CP, AD and SH) read and approved the final manuscript.

Corresponding author

Correspondence to Fiona O’Driscoll .

Ethics declarations

Ethics approval and consent to participate.

This study did not require ethics approvals as per the National Health Service Health Research Authority decision tool ( www.hra-decisiontools.org.uk ) and was carried out as a service evaluation. The service evaluation protocol and discussion guides were approved by the clinical audit team at the NHS Trust involved (reference number 830) before the study was commenced. Informed consent was obtained from all patients and/or their family member coordinating their care with the NHS Trust to participant in the interview after the purpose and format was explained to them. This consent was recorded by the researcher in the transcribed interview notes. Informed consent was also obtained from all staff participants.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Sung, B., O’Driscoll, F., Gregory, A. et al. Identifying barriers to outpatient appointment attendance in patient groups at risk of inequity: a mixed methods study in a London NHS trust. BMC Health Serv Res 24 , 554 (2024). https://doi.org/10.1186/s12913-024-10947-8

Download citation

Received : 05 January 2024

Accepted : 03 April 2024

Published : 30 April 2024

DOI : https://doi.org/10.1186/s12913-024-10947-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Health inequity
• Health inequality
• Appointment
• Behavioural science
• Public involvement
• Ethnic minorities
• Deprivation
• Underserved groups

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

• Open supplemental data
• Reference Manager
• Simple TEXT file

People also looked at

Original research article, a community health worker led approach to cardiovascular disease prevention in the uk—spices-sussex (scaling-up packages of interventions for cardiovascular disease prevention in selected sites in europe and sub-saharan africa): an implementation research project.

• 1 Department of Primary Care and Public Health, Brighton and Sussex Medical School, Brighton, United Kingdom
• 2 Department of Disease Control and Environmental Health, Makerere University, Kampala, Central Region, Uganda
• 3 Faculty of Medicine and Health Sciences, University of Antwerp, Antwerp, Belgium

Background: This paper describes a UK-based study, SPICES-Sussex, which aimed to co-produce and implement a community-based cardiovascular disease (CVD) risk assessment and reduction intervention to support under-served populations at moderate risk of CVD. The objectives were to enhance stakeholder engagement; to implement the intervention in four research sites and to evaluate the use of Voluntary and Community and Social Enterprises (VCSE) and Community Health Worker (CHW) partnerships in health interventions.

Methods: A type three hybrid implementation study design was used with mixed methods data. This paper represents the process evaluation of the implementation of the SPICES-Sussex Project. The evaluation was conducted using the RE-AIM framework.

Results: Reach: 381 individuals took part in the risk profiling questionnaire and forty-one women, and five men participated in the coaching intervention. Effectiveness: quantitative results from intervention participants showed significant improvements in CVD behavioural risk factors across several measures. Qualitative data indicated high acceptability, with the holistic, personalised, and person-centred approach being valued by participants. Adoption: 50% of VCSEs approached took part in the SPICES programme, The CHWs felt empowered to deliver high-quality and mutually beneficial coaching within a strong project infrastructure that made use of VCSE partnerships. Implementation: Co-design meetings resulted in local adaptations being made to the intervention. 29 (63%) of participants completed the intervention. Practical issues concerned how to embed CHWs in a health service context, how to keep engaging participants, and tensions between research integrity and the needs and expectations of those in the voluntary sector. Maintenance: Several VCSEs expressed an interest in continuing the intervention after the end of the SPICES programme.

Conclusion: Community-engagement approaches have the potential to have positively impact the health and wellbeing of certain groups. Furthermore, VCSEs and CHWs represent a significant untapped resource in the UK. However, more work needs to be done to understand how links between the sectors can be bridged to deliver evidence-based effective alternative preventative healthcare. Reaching vulnerable populations remains a challenge despite partnerships with VCSEs which are embedded in the community. By showing what went well and what did not, this project can guide future work in community engagement for health.

1 Introduction

Cardiovascular disease (CVD) is among the most prevalent, costly to treat, and deadly medical issues in the world ( 1 ). As part of the continual effort to combat CVD, greater emphasis is being placed on prevention. This often takes the form of behavioural or lifestyle change, focusing on the reduction of risk factors (e.g., hypertension, poor diet, obesity). Reducing these risk factors using evidence-based interventions not only works to lower rates of CVD, but also impacts rates of a variety of other medical issues, including susceptibility to severe COVID-19 infection ( 2 ), many common Noncommunicable Diseases (NCDs) including Type 2 diabetes and a wide range of cancers ( 3 ). Furthermore these preventative interventions are less expensive than reactionary care and can lower the treatment burden on strained medical systems ( 4 ).

Community-Based Participatory Research (CBPR) and Community Engagement (CE) have grown increasingly popular as potential methods to engender sustainable, long-term change in communities—particularly those communities under-served by existing medical systems and/or those at heightened risk of CVD ( 5 ). One's behaviour is influenced by their environment and the community they live in, meaning that tapping into a community's resources can be effective in changing lifestyle behaviour as well as having impacts on the wider community ( 6 , 7 ). The use of community-based practices fits within the growing South-North collaboration that this project joins as part of an international collaboration known as “Scaling-up Packages of Interventions for Cardiovascular disease prevention in selected sites in Europe and Sub-Saharan Africa: An implementation research project” (SPICES). In the Low- and Middle-Income countries (LMIC) there is evidence for the successful implementation of evidence-based community-based interventions in increasing knowledge of, and changing behaviour related to, CVD ( 8 ) however their use in the Global North is less well tested or understood ( 9 ).

In the UK, the flagship intervention to address preventative health issues is the National Health Service's (NHS) Health Check initiative, which is free to individuals ages 40–76 and which assesses risk for long term health conditions including CVD ( 10 ). Following initial assessment by a health professional, patients are advised on a course of action which often includes some degree of preventative prescribing to address behavioural risk factors ( 11 ). Just under half of eligible individuals accepts a first health-checks appointment (44.2%)—it is associated with increased detection of CVD risk, but uptake is skewed by several demographic factors (principally, age, gender, and socio-demographics), and it has struggled to create change in underserved groups ( 12 , 13 ). Marginalised coastal communities in Sussex face overall below-average healthy-life expectancy ( 14 ). This, alongside heightening inequality and the impact of COVID-19, has left some communities in Sussex significantly deprived in terms of access and engagement with health services ( 15 ). People in these communities experience transgenerational poverty, precarity, and lifestyle behaviours ingrained into the communities that lead many to be at higher risk for CVD. CBPR and CE models have the potential to lead to improved health and health behaviours among disadvantaged populations if designed properly and implemented through effective community consultation and participation ( 16 ).

CBPR and CE offer the chance to bring lessons from effective programmes in the Global South and apply them to programmes in the Global North. Community-based strategies to promote evidence-based preventative health interventions using Community Health Workers (CHWs) are often more established in the Global South where more tightly knit communities and established community health programmes fulfil a range of public health needs ( 17 , 18 ). CHWs interventions are a form of “task-sharing” intervention in which responsibility and power is shared between professional health workers and communities which have been proposed to effectively manage non-communicable disease risk ( 19 ). Lay Community Health Workers are individuals who are trained to perform of health-related functions but lack a formal professional health education. They can provide links between local communities and health care institutions thereby building and on and developing the social capital that already exists in communities ( 20 ). Although there is plenty of evidence communicating the importance and usefulness of these methods (the “what”), there remains a lack of attention given to how to do it . This article joins the work and voices attempting to begin filling that lacuna.

Within the literature on CBPR and CE, a handful of common themes emerge. The first is a push for human-centred research design ( 21 , 22 ). Yardley et al. ( 23 ) focused on this idea in their “person-based” approach to digital health interventions, where they recommended a “focus on understanding and accommodating the perspectives of the people who will use the intervention” ( 24 ). Hopkins and Rippon's ( 25 ) “asset-based” approach to CE interventions recommends recognising and adapting to the need, wants, and strengths already present in the community. Particularly the strengths, or “assets” already present in the community provide an opportunity for projects to use those assets. Such an implementation approach requires flexibility and adaptability, as well as deep involvement with the community. The second theme builds on the first, with the idea that not only should project design be person-centred, but those participants and other stakeholders in the community should be involved at every level of project planning through co-design. Yardley et al. ( 23 ) included this as a key element of their paper, writing that people from the target population should be involved in project development as well as at every stage of the intervention. Similarly, Berrera et al. ( 26 ) emphasise the need to adapt all projects to the cultural context of the community. This insight speaks to the third theme, continuous evaluation ( 27 ). As the needs of the community will be ever-shifting, so must the project adapt to those needs continually. Instead of designated periods of evaluation, a shift to continual processes of qualitative evaluation is called for to identify and adjust to the needs of the community. These processes require elevated levels of trust and participation from the community, which has its own challenges. Trust especially takes significant time and resources to develop and is an under-studied area of community engagement ( 28 ).

The SPICES-Sussex project was carried out from January 2019 and aimed to answer the following overarching research question: How can Community Health Workers (CHW) CVD prevention interventions, that have been used in the Global South, be developed, and implemented in a Global North setting and what barriers and enablers exist to their implementation? The project began with a situational analysis which included an exploration of the views and experiences of the local community with regards to CVD health and Community Health Workers and early stakeholder mapping of the research sites which was carried out between 2019 and 2020 ( 29 , 30 ).

The primary aim of the current paper is to provide a comprehensive examination of the project's implementation including complementary mixed methods analyses according to the Reach Effectiveness-Adoption Implementation and Maintenance (RE-AIM) framework ( 31 ). The secondary aims of the project are to inform future CE projects what worked (and did not work) for our project and to tie insights from our project to broader discussions in the discipline. The project is based on a protocol published in 2020 prior to the onset of COVID and was conducted through the period of the COVID-19 pandemic ( 29 ). Subsequently, several aspects of the original protocol were adapted to make implementation feasible within the constraints of this period (see Supplementary Appendix 2 ).

2.1 Study design

The project uses a type 3 hybrid implementation design ( 29 ) meaning that the primary aim of the research was to determine utility of an implementation intervention/strategy whilst the secondary aim was to assess clinical outcomes associated with the implementation trial. This means that we focused on understanding what barriers and enablers existed for the project's implementation and the context within which it operated. Effectiveness of the intervention remained important, however we were primarily interested in how and why it did (or did not) work. The project was carried out at four geographic research sites within Sussex (see Section 2.3 ) and implementation was conducted on an iterative basis from research site to research site broadly following the Medical Research Council's (MRC) framework for the development and implementation of complex interventions ( 32 ). The research team developed and then began delivering the intervention at each site before moving onto the next. At each site the following stages were carried out: (1) Development: this included stakeholder mapping, formation of implementation partners, and codesign/local adaptation of the intervention [covered in the study's pre-implementation paper ( 30 )]; (2) Implementation: this included the delivery of the CHW intervention at the research sites and collection of mixed method data pertaining to effectiveness and stakeholder experiences, and (3) Evaluation: this included the analysis of the mixed method data in line with the MRC guidance on analysis complex interventions.

2.2 Research site and voluntary and community sector enterprise partner selection

Four study sites were selected across East Sussex by identifying Middle Layer Super Output Area (MSOA) postcodes with high levels of deprivation according to the Indices of Multiple Deprivation (IMD) ( 33 ). Selection of the research sites was based on the pre-implementation community mapping phase of the project ( 30 ). Following on from CBPR practices, VCSEs and Volunteer Coordinators (VCs) were recruited to co-design and deliver the implementation strategy at each of the research sites.

VCSEs organisations were recruited as partners at each research site. The intervention was primarily run through these organisations and a paid staff member was recruited at each organisation. Their responsibilities included, CHW management, and participant recruitment. They also had a role in local adaptation activities. VCSE organisations were eligible to take part in the organisations if they were based in the research site, if they had interests and existing activities that aligned with the project's goals (CVD risk reduction and community development), and if they had existing experience of volunteer recruitment and management.

2.3 Community health worker recruitment and training

The aim was for each site to recruit a pool of five to eight CHWs. As part of this each site was asked for input into local CHW recruitment flyers, which were shared on VCSE websites and social media pages and shared on social media via existing CHWs at the VCSEs. CHWs were recruited through intermediary organisation recruitment via the VCSE partner organisation. The project was also advertised at a Virtual Volunteer Fair. Local contacts and existing volunteer pools at the VCSEs meant that the target number of CHWs was rapidly recruited at each site. CHWs were eligible to take part in the intervention if they were over 18 years of age, if they lived within the research site (determined by postcode), and if they had some kind of pre-existing relationship with the VCSE partner organisation (i.e., as a volunteer).

Potential CHWs who expressed an interest in the project were invited to attend an induction to the project, and then the local adaptation co-design meeting. Those who decided they would like to become a CHW then went on to receive five online, group training sessions (each of which lasted for 2 h, 10 h in total): an introductory session, a session covering project policies, heart health and the structure of the intervention, and three sessions on behaviour change techniques. These training sessions were developed and delivered by an external organisation (National Centre for Behaviour Change) specifically for the project after a consultation and planning process with the research team. Before the onset of the intervention at each site CHWs made various recommendations in the local adaptation meetings on the design of the training programme. These included providing information on listening techniques, engaging, and managing resistance, providing simple health information, using accessible language, using different starting points depending on the CHW's background knowledge and experience, training on conducting coaching virtually, and providing a training handbook. A Volunteer coordinator (VC) was recruited at each site. This VC was a trained and experience health coach (KFS) and provided training support and guidance through monthly group training support sessions in addition to the initial 10 h training block the received prior to the intervention onset. These monthly training and support sessions were organised into specific themes and agendas that were set with the CHW participants.

2.4 Local adaptation

Elements of the evidence-based intervention were tailored to the individuals and their community in the stakeholder-mapping phase using qualitative interviews, workshops, and focus groups with a range of stakeholders across the study site ( 30 ). Further rounds of local adaptation were carried out with VCs and CHWs at each of the research sites to tailor to individuals and their community context through iterative co-design workshops ( 34 ). CHWs and VCSE also agreed on a “volunteer charter” during the co-design session. This was a list of principles, behaviours, and practices upon which guided interactions between research staff, CHWs, VCSE and participants. The charter was designed to ensure that the practices of the project aligned with the principles of the CHW and partner organisation.

2.5 Participant recruitment and screening questionnaire

Participants (who received coaching) were eligible to take part in the eligibility screening if they lived in, or adjacent to, the study site's postcode and if they were aged eighteen or older. Participant recruitment was also based on intermediary organisation recruitment, community outreach, paid social media advertisement (through Meta™), gatekeeper and snowball sampling. Gatekeeper recruitment was conducted when interacting with a relevant statutory or non-statutory service provider (i.e., a fitness/weight loss group leader) and involved asking them to recommend the intervention to their members or to recommend participants who may be interested in taking part. Snowball sampling involved asking participants who participated in the study to sending email invitations to their social group. A social media recruitment strategy was undertaken to recruit people from the local area to the risk profiling survey to supplement the community-based recruitment through the VCSE partners. Social media was conducted on Facebook via paid advertisement in four waves of recruitment which took place over 1–2 weeks at each site. The advert targeted people who were 35+ and over and to people with 5 km of each research site. Messages were changed regularly from a list of recruitment messages drafted with CHWs during co-design sessions. Additionally, CHWs and VCSE participants were asked to send recruitment emails to any social or professional networks they thought would be interested in taking part. We did not record where participants were recruited.

Screening and risk profiling for the CVD coaching was carried out using the validated non-laboratory based INTERHEART questionnaire, presented online, for all participants that expressed an interest in the study ( 35 ). This questionnaire assessed modifiable and non-modifiable CVD risk factors and categorised participants as either “Low,” “Moderate.” Or “High” risk. See the protocol paper for further information on the INTERHEART risk profiling; for more information on the screening questionnaire, see the study protocol paper ( 29 , 35 ). Questionnaire data were collected and managed using Research Electronic Data Capture (REDCap) electronic data capture tools hosted at the University of Antwerp ( 36 ). Participants were considered to be eligible for the intervention if they were aged eighteen or over, if they lived within the research site (determined by postcode), and if they were categorised as “Moderate” risk of CVD according to the INTERHEART questionnaire. High risk participants were not included as their needs were considered to be too high for a pilot study involving CHWs. Eligible participants were then emailed by the research team with an invitation to take part in the CVD coaching intervention. After recruitment for the intervention was closed for each site, an online questionnaire survey was sent to eligible participants to gather information the reasons for not accepting the invitation to the intervention. Open response questions were used which the research team later categorised into codes.

2.6 The CVD prevention coaching intervention

The coaching intervention was based on motivational interviewing techniques which are promoted by the European commission on cardiovascular disease prevention in clinical practice ( 37 ) and which include techniques such as Open questions, Affirmation, Reflective listening and Summary reflections (OARS) ( 38 , 39 ). The use of these Behaviour Change Techniques (BCTs) used during the intervention were based on five target behaviours highlighted by the World Health Organisation including: reduce/cease smoking, increase moderate physical activity, reduce the fat, salt, and sugar content of the diet, increase fibre, oily fish (or alternatives), fruit, and vegetable content of the diet, reduce sedentary hours. The intervention involved six, one-hour long coaching sessions between participants and CHWs which were delivered every two weeks. Participants were also considered to have completed the intervention if they only completed three sessions and then notified the team of their withdrawal from the intervention.

The study team included two participant co-ordinators (PCs) who managed the participant journey through the intervention, sending welcome emails, questionnaires, and invitations to post-intervention interviews, and co-ordination between participants and CHWs to book coaching sessions. Reminders of appointments were also sent to CHWs and participants one week and two days before the session. Participants and CHWs were matched, based on gender preference and availability, and supported throughout the coaching intervention the PCs. CHWs were provided with guidance, resources, and signposting information throughout the intervention but were also given the flexibility to deliver the coaching in a way that suited them and their participant(s). Initially, counselling and goalsetting were based on their individual item INTERHEART assessment scores. Participants and CHWs were then encouraged to create an action plan with appropriate goal setting for the behaviours they wanted to change (e.g., diet, exercise habits). The goals were set in relation to when, where, and how they would undertake the behaviour, e.g., when the physical activity will be performed, where it will be performed, how often it will be performed (i.e., in a group or using specific equipment). CHWs helped participants to analyse any factors which might influence their ability to achieve the goals and to generate strategies which could help them overcome these barriers using problem solving. Full details of the participant journey through the intervention are given in Supplementary Appendix 2 in the Supplementary Material . All coaching was conducted virtually using Zoom™ to host and monitor coaching sessions and Microsoft OneDrive to store, recruit, and communicate written and visual resources with CHWs and participants. Monitoring in Zoom calls was called out by the PCs who checked whether both the participant began and ended the coaching session. If either the participants or CHW did not join, the PC could join the call to help the attendee. Feedback was obtained from the participant about the coaching session through emails after the session and by inviting participants to a follow-up interview after the intervention (see qualitative evaluation).

2.7 Evaluation

The evaluation was underpinned by the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework ( 31 ) which allows for an understanding of the multifaceted and interactive effects of personal, social, and environmental factors that determine behaviour; and for identifying behavioural and organisational leverage points and intermediaries for health promotion within organisations and communities. RE-AIM has been used to evaluate programs and setting in public health and community settings and is thought to be particularly useful when evaluate interventions in “real-world” settings ( 40 , 41 ). It has also been used to evaluate public health interventions which make use of community health workers in community-based setting ( 42 – 44 ). Results are made up of quantitative measures from the participant questionnaires, qualitative interviews with the participants, the CHWs, VCSE partners, and the research team. Primary quantitative outcome measures included implementation measures such as uptake and engagement and the pre/post changes to the self-report CVD behavioural questions which included the following three questionnaires: (1) the INTERHEART CVD risk questionnaire collected during the screening process was used as the baseline and collected again after completion of the intervention. (2) Physical activity levels were measured using the International Physical Activity Questionnaire (IPAQ) ( 45 ). The IPAQ is an internationally validated instrument to capture information about weekly physical activity habits, behaviours, and routines. (3) Diet was assessed using a 20-item questionnaire based on a modified version of the UK Diet and Diabetes Questionnaire ( 46 ), a brief food frequency questionnaire designed to assess conformity to healthy eating guidelines, and to assist in the setting of dietary goals. It was used to estimate the number of portions eaten daily or weekly of fruit and vegetables, oily fish (or alternative), and foods high in fat, salt, and sugar, what proportion of the time wholegrain cereal products were chosen, weekly units of alcohol consumed and the frequency of binge drinking. Due to the small sample sizes and non-parametric data used in this study, Wilcoxon Sign test was used to evaluate for differences in continuous variables whilst McNemar's test was used for binary categorical data. The pre-intervention assessment of the primary outcome measures was sent to participants before they participated in the intervention (no participant could begin the intervention without completing the baseline measures). Post intervention primary outcome measures were collected after their participant in the intervention was completed.

Focus groups and one-to-one interviews were conducted with four groups of stakeholders: (1) VCSE partners; (2) CHWs; (3) members of the research team, (4) participants in the intervention. Individual interviews were conducted with VCs, members of the research team, and participants, while data from the CHWs was collected in focus groups. Discussion guides for VCs, CHWs and members of the research team all included questions on the respondents' role within the project, the process of community engagement, barriers, and facilitators the implementation process, recommendations for the future and sustainability. Discussion guides for participant interviews included questions on how and why participants became involved in the project, their experience of the health coaching, and their views on the impact and usefulness of the project. Interviews and focus groups were conducted online using Zoom or MS Teams. The analysis was conducted by TGJ, IR, and RD and using qualitative framework analysis based on the components of the RE-AIM framework. Following data collection interviews and focus groups were transcribed by a professional transcription service and TGJ, IR, and RD familiarised themselves with the full set of data. They then undertook line-by-line coding of the data in NVivo using descriptive primary codes which were then interlinked with secondary codes. These secondary codes were then organised under the five elements of the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation, Maintenance). The analysis was interpreted, findings were synthesised with reference to the stakeholder group and theme descriptions were produced with supplementary illustrative quotes.

The Reach of the intervention was assessed through recruitment rates for the VCSE partners, CHWs and Intervention participants and qualitative data collected from the VCSE partners, and the research team was used to understand barriers and facilitators to recruitment. Effectiveness was assessed during the primary outcome measures and barriers and facilitators to effectiveness were assessed through qualitative interviews with the participants and CHWs. Adoption was at the setting level was determined through assessment of the retention of VCSE partners and qualitatively through interviews with VCSE partners and the research team. At the individual level, Adoption was assessed through CHW retention rates and qualitatively assessed through interviews with the research team and the CHWs. Implementation was assessed qualitatively through interviews with the intervention participants focusing on intervention fidelity. Maintenance was assessed at the setting level qualitatively through interviews with VCSE partners and the research team and through a report of the status of the intervention after 6 months. No individual level maintenance data is reported. A description of the data sources which contributed to each component of the RE-AIM framework is listed in Table 1 .

Table 1 . Description of data source used to evaluate the SPICES-Sussex intervention for each of the RE-AIM components.

Ethical approval for this research was obtained from Brighton and Sussex Medical School's Research Governance and Ethics Committee (R-GEC) (application reference: ER/DG241/17BSMS9E3G/1). This ethics review covered the methods described herein, key research materials, and recruitment and consent protocols for both intervention participants and staff/CHW interviews. Due to the changes imposed on the project by COVID-19 (see Supplementary Appendix 2 ) and because of minor adaptations from research site to research sites; several minor amendments were made (final application reference: ER/BSMS9E3G/6).

Informed consent was obtained in three ways from study participants depending on the nature of their participation. (1) Online screening questionnaire: these participants were presented with an approved information sheet on the first page of the online screening questionnaire, they were then provided with an Informed Consent Form (ICF) which they had to sign with a digital signature. (2) Intervention participants: just prior to participation and data collection participants met with a research staff member to review the information sheet and to sign the ICF if they agreed to participate, consent was sought again for those intervention participants who took part in a post-intervention interview. (3) CHW and research staff members: participants were sent the information sheet and consent form several days before their interview and were asked to sign and return the ICF prior to their interview appointment.

3.1 Participant characteristics

Risk profiling data was collected from 381 participants (Females: 310, Males: 71; mean (SD) age = 58 (12.39) years. Forty-Six participants began the intervention (39 Females, 7 Males; age = 58 (11.94) years. Sixteen participants took part in one-to-one interviews at the end of the intervention (thirteen females and two males, aged 32–67 years). Seven members of the research team (6 females, 1 male), and four VCSE partners (3 females and 1 male) took part in the research team interviews. Four focus groups with a total of thirteen participants (10 females and 3 males) were conducted with CHWs from each of the research sites. Thirteen participants (no gender data collected) took part in the post-intervention questionnaire for non-participants.

3.2 Analytical framework

The remainder of these findings are organised into RE-AIM dimensions with various quantitative and qualitative methods used to evidence each dimension, see Table 1 for a description of each of the data sources. Table 2 summarises concordance and discordance with expectations of the intervention [as described in the study protocol ( 29 )] in line with the RE-AIM framework. Supplementary Appendix 3 summarises changes to the study design from the published study protocol . Throughout this section participant codes are used to attribute quotations and references to specific terminology to a respondent. The codes identify the respondent as either a member of the Research team (RT), VCSE partners (VCSE), Community Health Worker (CHW) or Participants (PP). For VCSEs, CHWs and PPs references to their sites are also made (EB, HA, NH, HG). All codes refer to gender and (F/M), and their number within each respondent category.

Table 2 . Description of concordance/discordance with our pre-implementation expectations of the SPICES intervention for each of the RE-AIM components ( 29 ).

4.1 Recruitment of voluntary and community sector enterprise partners

A community-mapping exercise was carried out during the pre-implementation phase of the project ( 30 ) in which three partner organisations were identified across three research sites in East Sussex (Hastings, East Brighton, and Newhaven). All these organisations were volunteer based community organisations with a focus on local community development and improving health, with the Hastings organisations being focused on health and wellbeing. During the intervention set up phase, the East Brighton organisations dropped out of the study due to the impact of Covid-19 whilst the Hastings, and Newhaven organisations were carried forward to deliver the intervention. The East Brighton organisations helped the research team to develop links with a health and wellbeing organisation that was associated with a local General Practice (GP) clinic in East Brighton. Finally, a fourth research site was identified in West Hove and a final VCSE partner was identified. This organisation was a local community development organisation for the area. In total four VCSE organisations were partnered with across four research sites. In each site a VC was recruited from the partner organisation to deliver the intervention with the research team.

4.2 Community health worker recruitment

The research team and VCSE partners recruited 38 individuals who attended the introductory CHW meetings (Gender: 27 females and 11 males, NH n  = 7, EB n  = 13, HG n  = 10, HA n  = 8). Twenty-seven of these individuals completed the full training for CHWs (20 females and 7 males; NH n  = 5, EB n  = 9, HG n  = 7, HA n  = 6).

4.3 Participant recruitment

Social media recruitment had a wider reach to potential participants compared with gatekeeper recruitment, however, several participants did not complete the REDCap screening questions, had a poor understanding of the study, or were not part of the study's target population. VCSE gatekeepers yielded poor recruitment results apart from when a newsletter with a particularly large reach was used. Social media was the primary strategy for recruiting participants to the study. In total the messages reached 13,086 individuals across four waves of recruitment and of these 472 (3.6%) engaged with post by clicking on the survey link. Of those who clicked on the link 80% were female and 20% were male.

The INTERHEART screening data is shown in Figure 1 and Supplementary Appendix 1 for all those who completed the screening questionnaire ( N  = 381), participants who started the intervention and then withdrew ( N  = 17), and participants who completed the intervention and on whom we have full data ( n  = 27). Of the CVD risk factors measured by the INTERHEART screening tool, the two most prevalent were stress (reported by 61% of those screened, 56% of those who started the intervention, and 78% of those who went on to complete), and physical inactivity (reported by 55%, 81% and 64% respectively).

Figure 1 . Primary outcome measures for the “Reach” and “Effectiveness” components of the SPICES-Sussex intervention. ( A ) The proportion of “Low”, “Medium”, and “High” risk participants identified during the Interheart risk profiling questionnaire; ( B ) the mean Interheart score pre and post intervention for those who completed the intervention, p value from paired t -tests; ( C ) shows the % change regularly of dietary behaviours from pre/post intervention UKDDQ score, within-group t -tests; ( D ) the change in the % of intervention participants classified as having either low or medium/high activity levels pre and post intervention, p value from McNemar's test.

Forty-six participants took part in the CVD coaching intervention across the four research sites, all of whom completed the pre-intervention quantitative questionnaires. Sixty-three percent completed the full coaching intervention, and one participant withdrew from the project after three months. We had full data for twenty-seven of twenty-nine participants who completed the full 6-month coaching intervention (note: these participants have been removed from Supplementary Appendix 1 , n  = 2), Participants' characteristics are summarised in Table 3 . Several participants withdrew (37%), reasons given for withdrawing were: ill health/poor mental health/ill health in the family (13%); the intervention was considered a poor fit for the participant/did not meet their expectations/they did not need the intervention (9%); other commitments got in the way/they were too busy with their normal lives (7%); repeated non-attendance at planned coaching sessions from the CHW (4%); did not get on well with CHW (2%), language issues (2%).

Table 3 . Facilitators and barriers to reach and recruitment.

Due to low initial recruitment rates, the recruitment areas were expanded and included more affluent adjacent areas. The proportion of those completing the screening questionnaire and of those who went on to start the intervention who were in the target population (i.e., had an address with a postcode and IMD in the most deprived three deciles) was 30% in both cases. Despite recruitment being gender neutral and without gender/sex related parameters on social media our risk profiling questionnaire recruited far more women than men (77% female, 23% male, see Supplementary Appendix 1 ). This issue was carried forward to the main intervention in which only five of the forty-six who initially took part in the study were male.

4.4 Reasons for non-participation

Reasons given for not participating included missing or not receiving an invitation to take part ( n  = 4), lack of time due to responsibilities and commitments ( n  = 4), not feeling like the intervention was a good fit for them and their circumstances ( n  = 2), not being happy with the CHW allocated to them ( n  = 2), being reluctant to take part in online activities due to a lack of privacy at home ( n  = 1). When asked what would have made them more likely to participate the most common response was more clarity/detail on what was involved ( n  = 3).

4.5 Facilitators and barriers to reach

Intervention participants referred to several intervention components that functioned as facilitators or barriers to the reach of the intervention. These barriers and facilitators were organised into themes which include: (1) Experience of CHW recruitment; (2) The value of community partnerships; (3) The experience of the risk profiling questionnaire; (4) Impacts of COVID-19. These barriers and facilitators are described in more detail in Table 3 and illustrative quotes are provided.

5 Effectiveness

5.1 primary outcomes measures.

For those participants who completed the intervention, the before and after measures of cardiovascular risk, diet, physical activity, and readiness to change were compared (see Figure 1 and Supplementary Appendix 1 ). Mean INTERHEART score fell significantly from 11.7 to 9.9, taking the mean to within the low-risk range. There were also significant improvements in the self-reported dietary measures including: an increase in the proportion of time wholegrain foods were chosen, and the daily portions of fruit and vegetables eaten, and decreases in the consumption of fatty, salty, and sugary food. No changes were observed in the consumption of oily fish. Self-reported levels of physical inactivity also dropped over the course of the intervention with the proportion of those classified in the “low” physical activity category falling from 40% to 7%. Additionally, the self-reported levels of participants' “readiness to change” during the intervention increased from 3.6 to 4.5, which indicates increased levels of motivation as a result of the intervention.

5.2 Participant reported facilitators and barriers to the effectiveness

Intervention participants referred to several intervention components that functioned as facilitators or barriers to the effectiveness of the intervention. These barriers and facilitators were organised into themes which include: (1) accountability—the ways CHWs kept participants accountable about their health behaviours; (2) connection and community—the importance of making human connections with the CHWs and feelings of community togetherness; (3) judgement-free—the importance of a judgement-free intervention experience; (4) motivation and support—the coaching role that the intervention took in the lives of participants; (5) personalisation—the feeling that the intervention was adapted to their own needs and experiences; (6) reflection—the value of reflecting on experiences during the coaching intervention; (7) self-efficacy—the ways in which CHWs made participants feel in control of their health behaviours; (8) gradual or modest impact—the feeling that the intervention largely lead to modest impacts (9) generic or inappropriate advice—the feeling that the information provided during the coaching was too generic, obvious, or inappropriate to their needs. These barriers and facilitators are described in more detail in Table 4 and illustrative quotes are provided.

Table 4 . Facilitators and barriers to intervention effectiveness.

6.1 Retention of voluntary and community sector enterprise partnerships (setting level)

Of the six VCSE organisation engaged with during the pre-implementation phase of the project, four went on to be VCSE partner organisations during the implementation phase. Disruption and staff pressures resulting from the COVID-19 pandemic were a significant barrier to recruiting partner VCSEs, with two organisations who had been involved in initial discussions deciding not to proceed for this reason. Furthermore, interruptions to communication caused by COVID-19 and research team changes led to a loss of trust and engagement in some cases. One organisation which had a group of people ready to volunteer at the beginning of the project later withdrew as this group had fragmented due to COVID-19-related delays and substantial staffing changes that took place just prior to the implementation phase between 2019 and 2020. Other factors impacting on VCSE recruitment included the availability of funding, and issues with recruiting staff to the VC role. After one of the VCSE partners dropped out of the study just prior to the implementation phase, the same organisation linked the research team with another organisation who eventually functioned as VCSE partners for the implementation phase. The need to develop trust, and having the time to achieve this, was stated by several members of the research team as being key to recruiting partner VCSEs. Quality of communication was also felt to be especially important.

6.2 Facilitators of voluntary and community sector enterprise partnerships (setting level)

VCSES and research team members referred to several intervention components that functioned as facilitators or barriers to setting level adoption. These barriers and facilitators were organised into themes which include: (1) Trust—the importance of developing trust with community- partners; (2) Local Knowledge—the value of local knowledge and to delivering appropriate community care; (3) Local Skills—the value of the skills and experiences in local communities to delivering the intervention. These barriers and facilitators are described in more detail in Table 5 and illustrative quotes are provided.

Table 5 . Facilitators and barriers to intervention setting level adoption.

6.3 Retention of community health workers (individual level)

Of the twenty-seven CHWs who were recruited and trained to be a part of the intervention, twenty-one went on to deliver one or more session as an active CHW (Gender: 15 females and 6 males NH n  = 5, EB n  = 6 HG n  = 5, HA n  = 5). Each of these CHWs completed the intervention with at least one participant and the maximum number of participants who completed the intervention with one CHW was three.

6.4 Community health worker training needs feedback (individual level)

After training sessions in our first site, a short questionnaire was conducted with CHWs who attended the training in the formof one-to-one discussions with the training coordinator and the research team. Questions were asked about the anticipated barriers that CHWs thought they would face during the coaching as well as key training needs. Anticipated barriers and challenges during the project included: a sense of mistrust amongst participants, issues of poverty and deprivation, triggers, and sensitivities to the experiences of participants (i.e., trauma or addition triggers). The key training needs identified included: the sharing of personal stories to empower participants, how to set achievable health goals, preparing CHWs with tools to challenge the participant in a supportive way, improving CHW confidence, and advice on how to communicate CVD risk to participants in a straightforward way.

6.5 Community health worker facilitators and barriers to adoption (individual level)

CHWs referred to several intervention components that functioned as facilitators or barriers to the adoption of the intervention at the individual CHW level. These barriers and facilitators were organised into themes which include: (1) Local adaptation and Codesign Sessions; (2) CHW motivation for participating; (3) CHW experiences of the training; (4) CHW experience of the support provided to them. These barriers and facilitators are described in more detail in Table 6 and illustrative quotes are provided.

Table 6 . Facilitators and barriers to intervention individual level adoption.

7 Implementation

7.1 participant retention and fidelity.

Overall, 48% ( n  = 51) of those eligible ( n  = 106) to take part in the intervention agreed to do so and provided consent, of those 90% ( n  = 46) attended their first CHW coaching session and completed the baseline questionnaire. Of those who completed their first session 63% ( n  = 29) completed three sessions and 45% completed six sessions. For the 46 participants that began the intervention there were 276 planned program contacts of which 183 (66%) were completed. Retention and attendance data are summarised in Supplementary Appendix 2 . No data was collected on the amount of time each participant spent in their coaching session.

7.2 Participant facilitators and barriers to implementation

Intervention participants referred to several intervention components that functioned as facilitators or barriers to the implementation of the intervention. These barriers and facilitators were organised into themes which include: (1) expectations of the coaching intervention, (2) the virtual coaching sessions; (3) holistic and flexible, (4) length of the coaching session, (5) administrative support, (6) past experiences, (6) mental health. These barriers and facilitators are described in more detail in Table 7 and illustrative quotes are provided.

Table 7 . Facilitators and barriers to intervention implementation .

8 Maintenance

8.1 status of the intervention after six months.

Six months after the intervention's funding period ended the program was being continued at two of the sites. One of the sites continued it as volunteer opportunity and peer support program which was covered by their existing funding for peer support programs. A second site was awarded funding from the National Health Service to continue the intervention. The latter's findings will be reported as a program evaluation in the future.

8.2 Facilitators and barriers to maintenance

Interviewees referred to several intervention components that functioned as facilitators or barriers to the maintenance of the intervention at the setting level. These barriers and facilitators were organised into themes which include: (1) continuity of the intervention; (2) funding; (3) infrastructure These barriers and facilitators are described in more detail in Table 8 and illustrative quotes are provided.

Table 8 . Facilitators and barriers to maintenance.

9 Discussion

SPICES Sussex developed strategies to implement effective community-based CVD risk reduction interventions based on behaviour change coaching with CHWs by partnering with and leveraging the experience and influence of VCSE in four underserved communities in East Sussex, UK. Despite issues with recruitment and challenges associated with COVID-19 as well as other logistic, management, and research design challenges, the project showed clear markers of success. Participants experienced the interventions positively and many made gradual, and sometimes substantial, lifestyle changes. The quantitative results showed significant reduction in participants' CVD risk after taking part in the interventions. We think these successes were due to implementing our interventions in a flexible, personalised, and holistic way, which empowered CHWs to use their skills and experiences to aid participants. These results demonstrate how CHWs-led and community-based preventative CVD interventions could be implemented, such as those seen widely across the Global South ( 17 , 18 ). They also support a “person-based” and “asset-based” approach to community-based implementation design ( 23 , 25 ) in which the strengths and assets of communities and their members are used to promote health and wellbeing.

9.1 Intervention design

The SPICES-Sussex project used community-engagement and community health worker approaches to improve CVD health that are based on practices developed and tested in Kampala, Uganda ( 47 ). As part of the SPICES consortium these practices were adapted to several global north (UK, France, Belgium) and global south settings (South Africa). In the global south social public health approaches have long advocated for the decentralisation of healthcare to community partners and for a greater focus on prevention ( 48 ). Community-based public health practices such as task-sharing are often utilised in low-resource health systems in low-and middle income countries by recruiting and training community health workers to deliver low-intensity health intervention such as health coaching and signposting ( 49 ). In global south SPICES settings, there was greater buy-in to community-based interventions from governments and much of the trust building, and infrastructure for community health workers already exists ( 50 ). These settings, including the SPICES sites that influenced the Sussex site, often rely on voluntary or unpaid volunteers to conduct public health work in order to lower cost and to make use of existing social networks.

In resource-rich global north settings, healthcare is far more institutionalised and focused on secondary care and the infrastructure for community-based and participatory interventions is far less well developed. In the UK, most health interventions must adhere to the institutional demands of the National Health Service which presents a range of resource intensive training, recruitment, safeguarding, and management practices. There is much less history of CHWs in the UK; the role of these workers is not well understood or well defined outside of the third/voluntary sector despite recent calls for their use during the COVID-19 pandemic ( 51 ). This squeezed landscape for community-based intervention and the lack of familiarity with the role makes the development and implementation of these interventions challenging. In the global north there are increasing challenges to the volunteer nature of CHWs with researchers calling for compensation, capacity building, or payment of members of the public involved in intervention delivery of research and health interventions on moral and efficacy grounds ( 52 , 53 ). In our study, the decision not to pay CHW was made as a result of us following the SPICES approach developed in the Global South ( 17 ) and because the VCSE organisations we partnered with all had existing unpaid community volunteer programs. In our post-intervention qualitative evaluation interviews, participants and CHWs both discussed the value of paying CHWs. Furthermore, the drop in CHWs and the small number of participants they were able to take on implies that the lack of payment impacted the degree to which CHWs were able to engage with the project and therefore impacted the intervention's effectiveness and sustainability. In the UK, the NIHR now recommends that members of the public who are involved in research are properly reimbursed for their involvement and provide frequently updated guidelines on how to do so ( 54 ). In the future we argue that public health intervention that make use of CHWs should reimburse and pay them in some way for their involvement.

Community volunteers with low levels of training (10 h core training plus ongoing support), such as those used during this study, are not well-suited to complex cases or acute needs that required specialised support. In our findings, participants complained of generic, inappropriate, or obvious advice from the CHWs. Participants did not seem to prioritize the knowledge or expertise of the CHWs and instead valued the personalised, holistic, and supportive relationships that were offered by the CHWs. Participants in the intervention reported having good knowledge of what they needed to do to improve their health but struggled to do it in practice. Therefore, this kind of intervention may be well-suited to providing emotional and social support to people at risk of CVD who know what they “should” do but need a support and judgement free support mechanism to make changes.

Interviews with participants revealed a tension in the study linked to the use of an individualistic lifestyle change intervention situated within a community-based and participatory study. The study design did not address community-level, socio-economic, or environmental issues known to be vital when addressing CVD health ( 55 ). Tengland ( 56 ) argues that an individualistic lens of behavioural change can limit understandings of a person's CVD health. The result can be too narrow, as the “secondary” effects of their wider environmental conditions (i.e., powerlessness, lack of control, or lack of hope), are not considered. They further suggest that interventions should focus more on the attainment of instrumental goals, such as increased real opportunities in life. For community-based projects to grow further, they should seek to become multi-faceted by combining individualistic interventions with environment/community activities such as community education ( 57 ).

The frequency with which mental health issues were raised in discussions was notable. Those who took part in the screening reported high levels of stress and depression, and rates were even higher amongst participants taking part in the programme, furthermore, participants reported lower levels of stress and depression at the end. This may show that this type of intervention is particularly well suited to people with mental health concerns for whom talking to someone can make a real difference. This was also observed in the SPICES consortium partner sites including Brest (France) ( 58 ), and Antwerp (Belgium) ( 59 ). Most non-specialist or non-clinical people do not think of their health siloed into CVD, mental health, digestive health etc ( 60 ). Instead, one's health is perceived holistically, and mental health is often the most prominent barrier and facilitator to behaviour change.

9.2 Implementation strategy

We adopted a type 3 hybrid implementation study which focused primarily on implementation factors rather than evaluation, dropping the randomisation approach and embracing flexible more emergent iterative development and growth perspective, co-design, and contextual/place-based factors. A rigid evaluation linear approach as required for a type 1/2 design, which was initially planned, caused tensions with the community-based, participatory, and “emergent” aspects of the project and (2) the pressures imposed on our voluntary sector partners by the pandemic meant that adhering to a rigid randomisation approach was less realistic ( 7 ). The planned approach placed power in the hands of the research team which negatively affected our stakeholder relationships, and a rigid adherence to study protocols would have meant we could not effectively adapt strategies or interventions to context.

Instead we adopted a type 3 approach, which has been used to assess a wide range of preventative health and eHealth interventions which operate in communities based on participatory principles ( 61 ). In their systematic review of such strategies to implement interventions, Haldane et al. ( 62 ) highlight the importance of building mutually beneficial and trust-based relationships particularly with marginalised stakeholders, and stress the importance of developing strategies and interventions contextually whilst reporting and acting on lessons learnt throughout the project. Wildman et al. ( 63 ) argue that successful community-based projects require extensive community input, learning and adaptation captured from existing programmes to facilitate the replicability of programmes in other community contexts. With the more flexible type three approach we were able to make local adaptation to meet the need and priorities of the local community and local VCSE partner organisation thereby listening to the voices of those who are involved. This iterative approach to intervention design is similar to the “scaling-out” approach suggested by Aaron et al. ( 64 ) which advocates iterative roll-out and local adaptation in place of simply “copy and pasting” interventions across context. In reality, during SPICES-Sussex the local adaption became less flexible as the intervention became more well-developed as the internal factors became more institutionalised within the research team. However, the principle of meeting the needs and priorities of the local VCSE organisation were maintained from site to site and the team sought input from local organisations where possible.

We do not know whether the changes observed will be maintained due to the short follow up period, both at an individual level or a setting level ( 65 ), and the research lacks an economic appraisal. The short follow-up period was forced on the research team because of delays to the project caused by COVID-19 which meant our funding period was not long enough to conduct a follow up assessment. An economic appraisal was not considered appropriate because the development approach taken during the study meant that any economic appraisal was not likely to reflect real-world roll-out. In the future we would advocate for greater scaling-out to include a larger sample and an economic appraisal.

9.3 Recruitment and retention

The impacts of the restrictions placed on the people, organisations, and communities involved in this research due to COVID-19 were extensive and wide-ranging. The per-implementation phase of the research began in January of 2020 with the recruitment of an implementation team and participant recruitment was due to begin in April 2020. Following the outbreak of COVID-19 in the UK, recruitment was stopped from 16 March 2020 to 1 October 2020. By June 2020, a decision was made to fully move to remote delivery of the coaching intervention using video conferencing services.

Research recruitment and retention were near constant challenges, and all activities were significantly impacted by the Covid restrictions. We believe that the use of the INTERHEART tool, presented on the REDCap platform, acted as a barrier to recruitment as evidenced through the follow sources: (1) Over 650 participants attempted to begin the screening questionnaire and our records show those who did not complete it stopped towards the beginning or mid-way through the questions, particularly when they were asked to measure their waist/hip circumference, (2) of the 380 participants who completed the survey only approximately 100 were eligible for the intervention meaning we were selecting from a very limited pool of participants, (3) many of the participants in the per implementation interviews mentioned finding the screening tool to be “clunky” or “annoying” to use. Its overly “medical” focus, as a basis for lifestyle discussions may not have been engaging for the target audience.

Our initial recruitment strategy was to rely heavily on our VCSE partners to act as gatekeepers for recruitment, a practice commonly seen in participatory research methods ( 66 ). Whilst the VCSE partners were adept in the recruitment and management of CHWs and in the development of practices and policies, they did not seem to have the reach or access for the recruitment of large numbers of potential participants. Our experience aligns with that of Williams ( 67 ), who states that VCSE and end users' relationships are often smaller in number but deep, based on trust and protection, and covered by a range of risk related policies. Instead, we relied heavily on the use of paid for social media adverts for recruitment due to our ethics restrictions. Much like the experience of other researchers who used these tools, we found that they were low cost and reached large numbers of people but engagement with the screening and risk profiling and participant recruitment was low ( 68 ). In future studies, it may be more suitable to use social media as an adjunct to mixed recruitment strategies which make use of community outreach, primary care recruitment, and media outreach ( 69 , 70 ).

The study sample was heavily skewed towards middle-aged females and much of the sample was not considered to be from vulnerable or low socio-economic groups. Furthermore, males are under-represented in both the risk profiling and intervention samples which represents a divergence with our planned recruitment targets in which we aimed for a more representative sample. The difficulties in recruiting men and vulnerable and other “seldom heard” populations to life style interventions are well-recognised ( 71 , 72 ). Recommended strategies to improve male participation in community-based interventions include engaging with male-friendly spaces, workplace-based interventions, and incorporating activity-based programming, social-support, and group activities ( 73 , 74 ). Some of these elements were suggested during the planning phase of SPICES but were not feasible due to COVID restrictions ( 30 ).

9.4 Project infrastructure

We made the key decision to bring VCSE organisations into the research team with paid roles to foster stronger community/research partnerships as promoted by CBPR researchers ( 75 ) and the NHS's PPIE (Public and Patient Involvement and Engagement) initiatives ( 76 ). Our research shows that the VCSE sector is an untapped resource within primary and community care that has a great deal of expertise, compassion, and enthusiasm to offer health provision ( 77 ). To facilitate this community-based project, we focused on the concept of trust building throughout the intervention as described by Christopher et al. ( 78 ).

VCSE partnerships brought knowledge and expertise of their local communities, policies/practices of volunteer management and, critically, perspectives of the motivations and drivers for CHWs and communities. CHWs were empowered to bring their own skills and abilities to the intervention through an asset based and flexible project development which included them in the co-design of the project ( 79 ). The strategies we used to implement the interventions were not prescriptive and did not force CHWs to follow a set of strict guidelines. This led to a highly personalised, flexible, and reflective experience for CHWs. However, our experience highlights potential problems with relying on unpaid volunteers to deliver complex interventions, including issues with volunteer commitment, attendance and drop out.

Our research highlights the importance of infrastructure when managing CHWs and partnering with VCSE sector organisations. We developed a bespoke behaviour change training course for CHWs, a range of CHW risk appraisal and mitigation policies with our VCSE partners, and a dedicated team of participant and CHW support and management coordinators. Clear protocols were developed and followed for the recruitment, onboarding, matching, and hosting of participant coaching sessions whilst CHWs were provided with multiple channels of regular communication and continuous training and feedback opportunities. We support calls for project managers, VCSEs, primary care providers, and community members to be more explicitly involved in the design and development of interventions which affect and include communities ( 80 ).

In this study, the research team also experienced issues of positionality throughout the project whereby the lines between implementor, community worker, and evaluator were blurred. Coulter et al. ( 81 ) have pointed out that research that includes CHWs in the design and delivery of interventions commonly experience a tension between fidelity of the intervention protocol and community expectations, needs, and norms. We also experienced differing goals between academic and community partners (including CHWs), where academic partners prioritized data collection and community partners prioritized funding, sustainability, and policy. This can be likened to the experience of Furman et al. ( 82 ) who discussed how community partners were hesitant to endorse their research due to conflicts with on-the-ground realities of the community members they served.

9.5 Recommendations

During this project the research team, VCSE partners, and CHWs constantly learnt lessons and were quick to make adaptations to their approach based on feedback from a range of stakeholders and capturing all of these in this paper would be an impossible task. However, several key insights can be drawn from our collective experience and evaluation of the project. They include:

1. Environmental issues are larger and more complex than any coaching intervention based on individualistic changes can hope to remedy.

2. The voluntary and community sector has a range of strengths and assets based on local experience and knowledge developed over significant periods of time that can be used for CVD prevention. However, the sector is highly under-resourced and spread thinly across a wide range of priorities. Individual VCSE partner organisations do not always have enough reach to facilitate recruitment.

3. Community engagement works best if it is built into a project early on through co-design and resources and time should be allocated to this activity.

4. CHWs bring significant advantages during the delivery of community-based interventions. They are trusted peers, they bring their own skills and experience, and they can benefit from the intervention alongside the participants.

5. Strategies to encourage the participation of men should be specifically considered during the planning phase.

6. Virtual coaching interventions are acceptable to participants, and in many cases preferable to participants, due to their flexibility and ease of use.

7. The issue of mental health must be addressed even when working with unrelated health public conditions.

8. A strong project infrastructure, made up of well-trained support/administrative staff, is essential when delivering community-based interventions.

9. CHWs should be paid or reimbursed for their involvement in research and public health interventions. Falling to do so is looked down on my stakeholders and has impacts on sustainability and effectiveness.

10. The Global North can look to innovations in the Global South for examples of success for community-based interventions, however, proper contextual or situational analyses must be conducted to understand the needs and priorities of target communities.

9.6 Conclusion

This study demonstrates the feasibility of a CHW-led preventative health interventions could be implemented with overseen and unheard communities in the UK. It highlights the wealth of untapped resources that exist with VCSE and CHWs and suggests how a beneficial community-based service could be set up to run alongside and support NHS Health Checks, to reduce the incidence of CVD. The aim was to empower CHWs to discuss health with people in their communities based on behaviour change principles. We have set out what worked well and what did not, to facilitate development of future community-based interventions in the Global North. We believe that the community-based approach need not be restricted to CVD risk reduction, and that it could easily be applied to low level mental health conditions, diabetes, or other preventable NCDs. If CHWs are confident, well supported, and well-trained, they will have the skills and ability to contribute to improving the health and wellbeing of people in their communities. The benefits do not only extend to patients but also to CHWs and to the VCSE partners involved. We believe our project shows how these interventions can become a supplementary tool that links primary care services with the VCSE sector.

Data availability statement

The datasets presented in this study can be found in the University of Sussex's data repository through the following link: https://sussex.figshare.com/ ; (doi: 10.25377/sussex.25569084).

Ethics statement

The studies involving humans were approved by Brighton and Sussex Medical School Research Governance and Ethics Committee (RGEC). The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

First Author (First Authorship): Thomas Grice-Jackson Second Authors (Equal Contribution): Imogen Rogers, Elizabeth Ford, Robert Dickson Third Authors (Equal Contribution): Kat-Frere Smith, Katie Goddard, Linda Silver, Catherine Topham Fourth Author (Equal Contribution): Papreen Nahar, Geofrey Musinguzi, Hilde Bastiens. Senior Author (Senior Authorship): Harm Van Marwijk. All authors contributed to the article and approved the submitted version.

This project was funded as part of an EU Commission Horizon. CORDIS (The Community Research and Development Information Service (CORDIS) Grant agreement number: 733356.

Acknowledgments

We thank the following voluntary and community sector organisations for their partnerships whilst designing and delivering this project: Active Hastings, Wellsbourne Healthcare Community Interest Company, Sussex Community Development Association, the Crew Club, and the Hangleton and Knoll project. We thank the National Centre for Behaviour Change for their contribution to the development and delivery of the Community Health Workers training. We thank all members of the SPICES consortium and European Commission who provide consultation and advice throughout the project. Finally, we thank all our Community Health Workers for giving up their time for this project. They were central to every part of this work and their contribution is greatly appreciated. We would also like to thank the editorial and reviewer team assigned to this manuscript. Their contributions improved the quality of our manuscript presentation, structure, and discussion.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frhs.2024.1152410/full#supplementary-material

1. World Health Organisation. Cardiovascular Diseases Fact Sheet. (2022) (cited Nov 7, 2022). Cardiovascular diseases Fact Sheet. Available online at: https://www.who.int/health-topics/cardiovascular-diseases#tab=tab_1

Google Scholar

2. Matsushita K, Ding N, Kou M, Hu X, Chen M, Gao Y, et al. The relationship of COVID-19 severity with cardiovascular disease and its traditional risk factors: a systematic review and meta-analysis. Glob Heart . (2020) 15(1). doi: 10.5334/gh.814

PubMed Abstract | Crossref Full Text | Google Scholar

3. National Institute for Health and Care Excellence. Prevention of Cardiovascular disease (PH25)—Review Proposal (2014).

4. Marsh K, Phillips CJ, Fordham R, Bertranou E, Hale J. Estimating cost-effectiveness in public health: a summary of modelling and valuation methods. Health Econ Rev . (2012) 2(1):1–6. doi: 10.1186/2191-1991-2-17

5. Brush BL, Mentz G, Jensen M, Jacobs B, Saylor KM, Rowe Z, et al. Success in long-standing community-based participatory research (CBPR) partnerships: a scoping literature review. Health Educ Behav . (2020) 47(4):556–68. doi: 10.1177/1090198119882989

6. Bogart LM, Uyeda K. Community-based participatory research: partnering with communities for effective and sustainable behavioral health interventions. Health Psychol . (2009) 28(4):391–3. doi: 10.1037/a0016387

7. Thomas P. Collaborating for Health . London: Routledge (2017).

8. Hassen HY, Bowyer M, Gibson L, Abrams S, Bastiaens H. Level of cardiovascular disease knowledge, risk perception and intention towards healthy lifestyle and socioeconomic disparities among adults in vulnerable communities of Belgium and England. BMC Public Health . (2022) 22(1):1–9. doi: 10.1186/s12889-021-12274-7

9. Hassen HY, Ndejjo R, Musinguzi G, Van Geertruyden JP, Abrams S, Bastiaens H. Effectiveness of community-based cardiovascular disease prevention interventions to improve physical activity: a systematic review and meta-regression. Prev Med . (2021) 153:106797. doi: 10.1016/j.ypmed.2021.106797

10. Robson J, Dostal I, Sheikh A, Eldridge S, Madurasinghe V, Griffiths C, et al. The NHS health check in England: an evaluation of the first 4 years. BMJ Open . (2016) 6(1):e008840. doi: 10.1136/bmjopen-2015-008840

11. Office for Improvement and Disapities, UK Government. NHS Health Checks: Applying All Our Health. (2022). Available online at: https://www.gov.uk/government/publications/nhs-health-checks-applying-all-our-health/nhs-health-checks-applying-all-our-health (Accessed January 25, 2023).

12. Martin A, Saunders CL, Harte E, Griffin SJ, MacLure C, Mant J, et al. Delivery and impact of the NHS health check in the first 8 years: a systematic review. Br J Gen Pract . (2018) 68(672):e449–59. doi: 10.3399/bjgp18X697649

13. Tanner L, Kenny RPW, Still M, Ling J, Pearson F, Thompson K, et al. NHS health check programme: a rapid review update. BMJ Open . (2022) 12(2):e052832. doi: 10.1136/bmjopen-2021-052832

14. Agarwal S, Jakes S, Essex S, Page SJ, Mowforth M. Disadvantage in English seaside resorts: a typology of deprived neighbourhoods. Tour Manag . (2018) 69:440–59. doi: 10.1016/j.tourman.2018.06.012

Crossref Full Text | Google Scholar

15. UK Chief Medical Officer. Health in Coastal Communities. UK Chief Medical Officer; (2021). p. 259.

16. Cyril S, Smith BJ, Possamai-Inesedy A, Renzaho AM. Exploring the role of community engagement in improving the health of disadvantaged populations: a systematic review. Glob Health Action . (2015) 8(1):29842. doi: 10.3402/gha.v8.29842

17. Ndejjo R, Hassen HY, Wanyenze RK, Musoke D, Nuwaha F, Abrams S, et al. Community-based interventions for cardiovascular disease prevention in low-and middle-income countries: a systematic review. Public Health Rev . (2021) 42. doi: 10.3389/phrs.2021.1604018

18. Khetan AK, Purushothaman R, Chami T, Hejjaji V, Mohan SKM, Josephson RA, et al. The effectiveness of community health workers for CVD prevention in LMIC. Glob Heart . (2017) 12(3):233–243. 6. doi: 10.1016/j.gheart.2016.07.001

19. Joshi R, Peiris D. Task-sharing for the prevention and control of non-communicable diseases. Lancet Glob Health . (2019) 7(6):e686–7. doi: 10.1016/S2214-109X(19)30161-5

20. Adams C. Toward an institutional perspective on social capital health interventions: lay community health workers as social capital builders. Sociol Health Illn . (2020) 42(1):95–110. doi: 10.1111/1467-9566.12992

21. Van Velsen L, Illario M, Jansen-Kosterink S, Crola C, Di Somma C, Colao A, et al. A community-based, technology-supported health service for detecting and preventing frailty among older adults: a participatory design development process. J Aging Res . (2015) 2015:9–18. doi: 10.1155/2015/216084

22. Michie S, Abraham C, Whittington C, McAteer J, Gupta S. Effective techniques in healthy eating and physical activity interventions: a meta-regression. Health Psychol . (2009) 28(6):690. doi: 10.1037/a0016136

23. Yardley L, Morrison L, Bradbury K, Muller I. The person-based approach to intervention development: application to digital health-related behavior change interventions. J Med Internet Res . (2015) 17(1):e4055. doi: 10.2196/jmir.4055

24. Person-Based Approach. The Person-Based Approach For Developing Health Interventions. (2022). Available online at: https://www.personbasedapproach.org/#:∼:text=What%20is%20the%20Person%2DBased,and%20experiences%20of%20intervention%20users (Accessed January 25, 2023).

25. Hopkins T, Rippon S. Head, Hands and Heart: Asset-Based Approaches in Health Care . London: Health Foundations (2015).

26. Barrera M, Castro FG, Strycker LA, Toobert DJ. Cultural adaptations of behavioral health interventions: a progress report. J Consult Clin Psychol . (2013) 81(2):196–205. doi: 10.1037/a0027085

27. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al.. process evaluation of complex interventions: medical research council guidance. Br Med J . (2015) 350. doi: 10.1136/bmj.h1258

28. Brown A, Ramsay N, Milo M, Moore M, Hossain R. How research-based theatre is a solution for community engagement and advocacy at regional medical campuses: the health and equity through advocacy, research, and theatre (HEART) program. Can Med Educ J . (2018) 9(1):e6. doi: 10.36834/cmej.42191

29. Nahar P, Van Marwijk H, Gibson L, Musinguzi G, Anthierens S, Ford E, et al. A protocol paper: community engagement interventions for cardiovascular disease prevention in socially disadvantaged populations in the UK: an implementation research study. Glob Health Res Policy . (2020) 5:1–9. doi: 10.1186/s41256-020-0131-1

30. Grice-Jackson T, Rogers I, Ford E, Van Marwijk H, Topham C, Musinguzi G, et al. The pre-implementation phase of a project seeking to deliver a community-based CVD prevention intervention (SPICES-Sussex): a qualitative study exploring views and experience relating to intervention development. Health Promot Pract . (2023):15248399231182139. doi: 10.1177/15248399231182139

31. Glasgow RE, McKay HG, Piette JD, Reynolds KD. The RE-AIM framework for evaluating interventions: what can it tell us about approaches to chronic illness management? Patient Educ Couns . (2001) 44(2):119–27. doi: 10.1016/S0738-3991(00)00186-5

32. Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al.. A new framework for developing and evaluating complex interventions: update of medical research council guidance. Br Med J . (2021) 374:1–11. doi: 10.1136/bmj.n2061

33. Ministry of Housing, Communities & Local Government UK Government. English Indices of Deprivation 2019. (2019) (cited Nov 7, 2022). Available online at: https://www.gov.uk/government/statistics/english-indices-of-deprivation-2019

34. Palmer VJ. The participatory zeitgeist in health care: it is time for a science of participation. J Particip Med . (2020) 12(1):e15101. doi: 10.2196/15101

35. McGorrian C, Yusuf S, Islam S, Jung H, Rangarajan S, Avezum A, et al. Estimating modifiable coronary heart disease risk in multiple regions of the world: the INTERHEART modifiable risk score. Eur Heart J . (2011) 32(5):581–9. doi: 10.1093/eurheartj/ehq448

36. Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O’Neal L, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform . (2019) 95:103208. doi: 10.1016/j.jbi.2019.103208

37. Piepoli MF, Hoes AW, Agewall S, Albus C, Brotons C, Catapano AL, et al. Guidelines: editor’s choice: 2016 European guidelines on cardiovascular disease prevention in clinical practice: the sixth joint task force of the European society of cardiology and other societies on cardiovascular disease prevention in clinical practice (constituted by representatives of 10 societies and by invited experts) developed with the special contribution of the European association for cardiovascular prevention & rehabilitation (EACPR). Eur Heart J . (2016) 37(29):2315. doi: 10.1093/eurheartj/ehw106

38. Levounis P, Arnaout B, Marienfeld C. Motivational interviewing for clinical practice . Arlington: American Psychiatric Association Publishing (2017). p. 15–35.

39. Miller WR, Rollnick S. Motivational Interviewing: Helping People Change . New York: Guilford press (2012).

40. Harden SM, Smith ML, Ory MG, Smith-Ray RL, Estabrooks PA, Glasgow RE. RE-AIM in clinical, community, and corporate settings: perspectives, strategies, and recommendations to enhance public health impact. Front Public Health . (2018) 6:71. doi: 10.3389/fpubh.2018.00071

41. Kwan BM, McGinnes HL, Ory MG, Estabrooks PA, Waxmonsky JA, Glasgow RE. RE-AIM in the real world: use of the RE-AIM framework for program planning and evaluation in clinical and community settings. Front Public Health . (2019) 7:345. doi: 10.3389/fpubh.2019.00345

42. Glasgow RE, Askew S, Purcell P, Levine E, Warner ET, Stange KC, et al. Use of RE-AIM to address health inequities: application in a low-income community health center-based weight loss and hypertension self-management program. Transl Behav Med . (2013) 3(2):200–10. doi: 10.1007/s13142-013-0201-8

43. Onono M, Abdi M, Opondo I, Okung’u J, Asadhi E, Nyamai R, et al. Using the RE-AIM framework to evaluate the implementation of integrated community case management in Kenya. Acta Paediatr . (2018) 107:53–62. doi: 10.1111/apa.14662

44. Wippold GM, Garcia KA, Frary SG, Griffith DM. Community health worker interventions for men: a scoping review using the RE-AIM framework. Health Educ Behav . (2023) 51(1):10901981231179498. doi: 10.1177/1090198123117949

45. Hagströmer M, Oja P, Sjöström M. The international physical activity questionnaire (IPAQ): a study of concurrent and construct validity. Public Health Nutr . (2006) 9(6):755–62. doi: 10.1079/PHN2005898

46. England CY, Thompson JL, Jago R, Cooper AR, Andrews RC. Development of a brief, reliable and valid diet assessment tool for impaired glucose tolerance and diabetes: the UK diabetes and diet questionnaire. Public Health Nutr . (2017) 20(2):191–9. doi: 10.1017/S1368980016002275

47. Musinguzi LK, Turinawe EB, Rwemisisi JT, de Vries DH, Mafigiri DK, Muhangi D, et al. Linking communities to formal health care providers through village health teams in rural Uganda: lessons from linking social capital. Hum Resour Health . (2017) 15:1–16. doi: 10.1186/s12960-016-0177-9

48. Katz E, Chikwenhere Y, Essien E, Olirus Owilli A, Westerhaus M. Rethinking global health from south and north: a social medicine approach to global health education. Glob Public Health . (2023) 18(1):2191685. doi: 10.1080/17441692.2023.2191685

49. Anand TN, Joseph LM, Geetha AV, Chowdhury J, Prabhakaran D, Jeemon P. Task-sharing interventions for cardiovascular risk reduction and lipid outcomes in low-and middle-income countries: a systematic review and meta-analysis. J Clin Lipidol . (2018) 12(3):626–42. doi: 10.1016/j.jacl.2018.02.008

50. Ndejjo R, Wanyenze RK, Nuwaha F, Bastiaens H, Musinguzi G. Barriers and facilitators of implementation of a community cardiovascular disease prevention programme in Mukono and Buikwe districts in Uganda using the consolidated framework for implementation research. Implement Sci . (2020) 15:1–17. doi: 10.1186/s13012-020-01065-0

51. Haines A, de Barros EF, Berlin A, Heymann DL, Harris MJ. National UK programme of community health workers for COVID-19 response. Lancet . (2020) 395(10231):1173–5. doi: 10.1016/S0140-6736(20)30735-2

52. Ballard M, Odera M, Bhatt S, Geoffrey B, Westgate C, Johnson A. Payment of community health workers. Lancet Glob Health . (2022) 10(9):e1242. doi: 10.1016/S2214-109X(22)00311-4

53. Hanson K, Brikci N, Erlangga D, Alebachew A, De Allegri M, Balabanova D, et al. The lancet global health commission on financing primary health care: putting people at the centre. Lancet Glob Health . (2022) 10(5):e715–72. doi: 10.1016/S2214-109X(22)00005-5

54. NIHR. UK Standards for Public Involvement. (2024) (cited Jan 9, 2024). Available online at: https://sites.google.com/nihr.ac.uk/pi-standards/home

55. Schmid TL, Pratt M, Howze E. Policy as intervention: environmental and policy approaches to the prevention of cardiovascular disease. Am J Public Health . (1995) 85(9):1207–11. doi: 10.2105/AJPH.85.9.1207

56. Tengland PA. Behavior change or empowerment: on the ethics of health-promotion goals. Health Care Anal . (2016) 24(1):24–46. doi: 10.1007/s10728-013-0265-0

57. Pennant M, Davenport C, Bayliss S, Greenheld W, Marshall T, Hyde C. Community programs for the prevention of cardiovascular disease: a systematic review. Am J Epidemiol . (2010) 172(5):501–16. doi: 10.1093/aje/kwq171

58. Le Goff D, Barais M, Perraud G, Derriennic J, Aujoulat P, Guillou-Landreat M, et al. Innovative cardiovascular primary prevention population-based strategies: a 2-year hybrid type 1 implementation randomised control trial (RCT) which evaluates behavioural change conducted by community champions compared with brief advice only from the SPICES project (scaling-up packages of interventions for cardiovascular disease prevention in selected sites in Europe and sub-Saharan Africa). BMC Public Health . (2021) 21(1):1–9. doi: 10.1186/s12889-021-11443-y

59. Aerts N, Anthierens S, Van Bogaert P, Peremans L, Bastiaens H. Prevention of cardiovascular diseases in community settings and primary health care: a Pre-implementation contextual analysis using the consolidated framework for implementation research. Int J Environ Res Public Health . (2022) 19(14):8467. doi: 10.3390/ijerph19148467

60. Collyer TA, Smith KE. An atlas of health inequalities and health disparities research:“how is this all getting done in silos, and why?”. Soc Sci Med . (2020) 264:113330. doi: 10.1016/j.socscimed.2020.113330

61. Van Velsen L, Wentzel J, Van Gemert-Pijnen JE. Designing eHealth that matters via a multidisciplinary requirements development approach. JMIR Res Protoc . (2013) 2(1):e2547. doi: 10.2196/resprot.2547

62. Haldane V, Chuah FL, Srivastava A, Singh SR, Koh GC, Seng CK, et al. Community participation in health services development, implementation, and evaluation: a systematic review of empowerment, health, community, and process outcomes. PLoS One . (2019) 14(5):e0216112. doi: 10.1371/journal.pone.0216112

63. Wildman JM, Valtorta N, Moffatt S, Hanratty B. ‘What works here doesn’t work there’: the significance of local context for a sustainable and replicable asset-based community intervention aimed at promoting social interaction in later life. Health Soc Care Community . (2019) 27(4):1102–10. doi: 10.1111/hsc.12735

64. Aarons GA, Sklar M, Mustanski B, Benbow N, Brown CH. “Scaling-out” evidence-based interventions to new populations or new health care delivery systems. Implement Sci . (2017) 12:1–13. doi: 10.1186/s13012-016-0533-0

65. Luten KA, Reijneveld SA, Dijkstra A, de Winter AF. Reach and effectiveness of an integrated community-based intervention on physical activity and healthy eating of older adults in a socioeconomically disadvantaged community. Health Educ Res . (2016) 31(1):98–106. doi: 10.1093/her/cyv064

66. Goodman A, Gatward R. Who are we missing? Area deprivation and survey participation. Eur J Epidemiol . (2008) 23(6):379–87. doi: 10.1007/s10654-008-9248-0

67. Williams P. “It all sounds very interesting, but we’re just too busy!”: exploring why “gatekeepers” decline access to potential research participants with learning disabilities. Eur J Spec Needs Educ . (2020) 35(1):1–14. doi: 10.1080/08856257.2019.1687563

68. McRobert CJ, Hill JC, Smale T, Hay EM, Van der Windt DA. A multi-modal recruitment strategy using social media and internet-mediated methods to recruit a multidisciplinary, international sample of clinicians to an online research study. PLoS One . (2018) 13(7):e0200184. doi: 10.1371/journal.pone.0200184

69. Khatri C, Chapman SJ, Glasbey J, Kelly M, Nepogodiev D, Bhangu A, et al. Social media and internet driven study recruitment: evaluating a new model for promoting collaborator engagement and participation. PloS one . (2015) 10(3):e0118899. doi: 10.1371/journal.pone.0118899

70. UyBico SJ, Pavel S, Gross CP. Recruiting vulnerable populations into research: a systematic review of recruitment interventions. J Gen Intern Med . (2007) 22(6):852–63. doi: 10.1007/s11606-007-0126-3

71. Shaghaghi A, Bhopal RS, Sheikh A. Approaches to recruiting “hard-to-reach”populations into research: a review of the literature. Health Promot Perspect . (2011) 1(2):86. doi: 10.5681/hpp.2011.009

72. Pagoto SL, Schneider KL, Bodenlos JS, Appelhans BM, Whited MC, Ma Y, et al. Association of post-traumatic stress disorder and obesity in a nationally representative sample. Obesity . (2012) 20(1):200–5. doi: 10.1038/oby.2011.318

73. Oliffe JL, Rossnagel E, Bottorff JL, Chambers SK, Caperchione C, Rice SM. Community-based men’s health promotion programs: eight lessons learnt and their caveats. Health Promot Int . (2020) 35(5):1230–40. doi: 10.1093/heapro/daz101

74. Bell OJ, Flynn D, Clifford T, West D, Stevenson E, Avery L. Identifying behavioural barriers and facilitators to engaging men in a community-based lifestyle intervention to improve physical and mental health and well-being. Int J Behav Nutr Phys Act . (2023) 20(1):1–14. doi: 10.1186/s12966-022-01404-y

75. Springer MV, Skolarus LE. Community-based participatory research: partnering with communities. Stroke . (2019) 50(3):e48–50. doi: 10.1161/STROKEAHA.118.024241

76. Holmes L, Cresswell K, Williams S, Parsons S, Keane A, Wilson C, et al. Innovating public engagement and patient involvement through strategic collaboration and practice. Res Involvement Engagem . (2019) 5(1):1–12. doi: 10.1186/s40900-019-0160-4

77. Ragsdell G. Voluntary sector organisations: untapped sources of lessons for knowledge management. Proceedings of the International Conference on Intellectual Capital, Knowledge Management & Organizational Learning (2013). p. 349–55

78. Christopher S, Watts V, McCormick AKHG, Young S. Building and maintaining trust in a community-based participatory research partnership. Am J Public Health . (2008) 98(8):1398–406. doi: 10.2105/AJPH.2007.125757

79. Bainbridge L, Lunt N. Place, strengths and assets: a case study of how local area coordination is supporting individuals and families under conditions of austerity. Br J Social Work . (2021) 51(4):1354–73. doi: 10.1093/bjsw/bcab041

80. Di Maddaloni F, Davis K. Project manager’s perception of the local communities’ stakeholder in megaprojects. An empirical investigation in the UK. Int J Proj Manag . (2018) 36(3):542–65. doi: 10.1016/j.ijproman.2017.11.003

81. Coulter K, Ingram M, McClelland DJ, Lohr A. Positionality of community health workers on health intervention research teams: a scoping review. Front Public Health . (2020) 8:208. doi: 10.3389/fpubh.2020.00208

82. Furman L, Matthews L, Davis V, Killpack S, O’Riordan MA. Breast for success: a community–academic collaboration to increase breastfeeding among high-risk mothers in Cleveland. Prog Community Health Partnersh . (2016) 10(3):341–53. doi: 10.1353/cpr.2016.0041

Keywords: community based participatory research, implementation research, RE-AIM (reach, effectiveness, adoption, implementation and maintenance), cardiovascular disease, community health workers (CHW)

Citation: Grice-Jackson T, Rogers I, Ford E, Dickinson R, Frere-Smith K, Goddard K, Silver L, Topham C, Nahar P, Musinguzi G, Bastiaens H and Van Marwijk H (2024) A community health worker led approach to cardiovascular disease prevention in the UK—SPICES-Sussex (scaling-up packages of interventions for cardiovascular disease prevention in selected sites in Europe and Sub-saharan Africa): an implementation research project. Front. Health Serv. 4:1152410. doi: 10.3389/frhs.2024.1152410

Received: 27 January 2023; Accepted: 20 March 2024; Published: 7 May 2024.

Reviewed by:

© 2024 Grice-Jackson, Rogers, Ford, Dickinson, Frere-Smith, Goddard, Silver, Topham, Nahar, Musinguzi, Bastiaens and Van Marwijk. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Thomas Grice-Jackson [email protected]

This article is part of the Research Topic

Hybrid Effectiveness-Implementation Trial Designs: Critical Assessments, Innovative Applications, and Proposed Advancements

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Methodology
• Mixed Methods Research | Definition, Guide, & Examples

Mixed Methods Research | Definition, Guide, & Examples

Published on 4 April 2022 by Tegan George . Revised on 25 October 2022.

Mixed methods research combines elements of quantitative research and qualitative research in order to answer your research question . Mixed methods can help you gain a more complete picture than a standalone quantitative or qualitative study, as it integrates benefits of both methods.

Mixed methods research is often used in the behavioral, health, and social sciences, especially in multidisciplinary settings and complex situational or societal research.

• To what extent does the frequency of traffic accidents ( quantitative ) reflect cyclist perceptions of road safety ( qualitative ) in Amsterdam?
• How do student perceptions of their school environment ( qualitative ) relate to differences in test scores ( quantitative ) ?
• How do interviews about job satisfaction at Company X ( qualitative ) help explain year-over-year sales performance and other KPIs ( quantitative ) ?
• How can voter and non-voter beliefs about democracy ( qualitative ) help explain election turnout patterns ( quantitative ) in Town X?
• How do average hospital salary measurements over time (quantitative) help to explain nurse testimonials about job satisfaction (qualitative) ?

Table of contents

When to use mixed methods research, mixed methods research designs, benefits of mixed methods research, disadvantages of mixed methods research, frequently asked questions about mixed methods research.

Mixed methods research may be the right choice if your research process suggests that quantitative or qualitative data alone will not sufficiently answer your research question. There are several common reasons for using mixed methods research:

• Generalisability : Qualitative research usually has a smaller sample size , and thus is not generalisable . In mixed methods research, this comparative weakness is mitigated by the comparative strength of ‘large N’, externally valid quantitative research.
• Contextualisation: Mixing methods allows you to put findings in context and add richer detail to your conclusions. Using qualitative data to illustrate quantitative findings can help ‘put meat on the bones’ of your analysis.
• Credibility: Using different methods to collect data on the same subject can make your results more credible. If the qualitative and quantitative data converge, this strengthens the validity of your conclusions. This process is called triangulation .

As you formulate your research question , try to directly address how qualitative and quantitative methods will be combined in your study. If your research question can be sufficiently answered via standalone quantitative or qualitative analysis, a mixed methods approach may not be the right fit.

Keep in mind that mixed methods research doesn’t just mean collecting both types of data; you need to carefully consider the relationship between the two and how you’ll integrate them into coherent conclusions. Mixed methods can be very challenging to put into practice, so it’s a less common choice than standalone qualitative or qualitative research.

Prevent plagiarism, run a free check.

There are different types of mixed methods research designs . The differences between them relate to the aim of the research, the timing of the data collection , and the importance given to each data type.

As you design your mixed methods study, also keep in mind:

• Your research approach ( inductive vs deductive )
• Your research questions
• What kind of data is already available for you to use
• What kind of data you’re able to collect yourself.

Here are a few of the most common mixed methods designs.

Convergent parallel

In a convergent parallel design, you collect quantitative and qualitative data at the same time and analyse them separately. After both analyses are complete, compare your results to draw overall conclusions.

• On the qualitative side, you analyse cyclist complaints via the city’s database and on social media to find out which areas are perceived as dangerous and why.
• On the quantitative side, you analyse accident reports in the city’s database to find out how frequently accidents occur in different areas of the city.

In an embedded design, you collect and analyse both types of data at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.

This is a good approach to take if you have limited time or resources. You can use an embedded design to strengthen or supplement your conclusions from the primary type of research design.

Explanatory sequential

In an explanatory sequential design, your quantitative data collection and analysis occurs first, followed by qualitative data collection and analysis.

You should use this design if you think your qualitative data will explain and contextualise your quantitative findings.

Exploratory sequential

In an exploratory sequential design, qualitative data collection and analysis occurs first, followed by quantitative data collection and analysis.

You can use this design to first explore initial questions and develop hypotheses. Then you can use the quantitative data to test or confirm your qualitative findings.

‘Best of both worlds’ analysis

Combining the two types of data means you benefit from both the detailed, contextualised insights of qualitative data and the generalisable, externally valid insights of quantitative data. The strengths of one type of data often mitigate the weaknesses of the other.

For example, solely quantitative studies often struggle to incorporate the lived experiences of your participants, so adding qualitative data deepens and enriches your quantitative results.

Solely qualitative studies are often not very generalisable, only reflecting the experiences of your participants, so adding quantitative data can validate your qualitative findings.

Method flexibility

Mixed methods are less tied to disciplines and established research paradigms. They offer more flexibility in designing your research, allowing you to combine aspects of different types of studies to distill the most informative results.

Mixed methods research can also combine theory generation and hypothesis testing within a single study, which is unusual for standalone qualitative or quantitative studies.

Mixed methods research is very labour-intensive. Collecting, analysing, and synthesising two types of data into one research product takes a lot of time and effort, and often involves interdisciplinary teams of researchers rather than individuals. For this reason, mixed methods research has the potential to cost much more than standalone studies.

Differing or conflicting results

If your analysis yields conflicting results, it can be very challenging to know how to interpret them in a mixed methods study. If the quantitative and qualitative results do not agree or you are concerned you may have confounding variables , it can be unclear how to proceed.

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to test a hypothesis by systematically collecting and analysing data, while qualitative methods allow you to explore ideas and experiences in depth.

In mixed methods research , you use both qualitative and quantitative data collection and analysis methods to answer your research question .

Data collection is the systematic process by which observations or measurements are gathered in research. It is used in many different contexts by academics, governments, businesses, and other organisations.

Triangulation in research means using multiple datasets, methods, theories and/or investigators to address a research question. It’s a research strategy that can help you enhance the validity and credibility of your findings.

Triangulation is mainly used in qualitative research , but it’s also commonly applied in quantitative research . Mixed methods research always uses triangulation.

These are four of the most common mixed methods designs :

• Convergent parallel: Quantitative and qualitative data are collected at the same time and analysed separately. After both analyses are complete, compare your results to draw overall conclusions. 
• Embedded: Quantitative and qualitative data are collected at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.
• Explanatory sequential: Quantitative data is collected and analysed first, followed by qualitative data. You can use this design if you think your qualitative data will explain and contextualise your quantitative findings.
• Exploratory sequential: Qualitative data is collected and analysed first, followed by quantitative data. You can use this design if you think the quantitative data will confirm or validate your qualitative findings.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

George, T. (2022, October 25). Mixed Methods Research | Definition, Guide, & Examples. Scribbr. Retrieved 6 May 2024, from https://www.scribbr.co.uk/research-methods/mixed-methods/

Is this article helpful?

Tegan George

• Open access
• Published: 04 May 2024

Delivering clinical tutorials to medical students using the Microsoft HoloLens 2: A mixed-methods evaluation

• Murray Connolly 1 ,
• Gabriella Iohom 1 ,
• Niall O’Brien 2 ,
• James Volz 2 ,
• Aogán O’Muircheartaigh 3 ,
• Paschalitsa Serchan 3 ,
• Agatha Biculescu 3 ,
• Kedar Govind Gadre 3 ,
• Corina Soare 1 ,
• Laura Griseto 3 &
• George Shorten 1  

BMC Medical Education volume  24 , Article number:  498 ( 2024 ) Cite this article

215 Accesses

12 Altmetric

Metrics details

Mixed reality offers potential educational advantages in the delivery of clinical teaching. Holographic artefacts can be rendered within a shared learning environment using devices such as the Microsoft HoloLens 2. In addition to facilitating remote access to clinical events, mixed reality may provide a means of sharing mental models, including the vertical and horizontal integration of curricular elements at the bedside. This study aimed to evaluate the feasibility of delivering clinical tutorials using the Microsoft HoloLens 2 and the learning efficacy achieved.

Following receipt of institutional ethical approval, tutorials on preoperative anaesthetic history taking and upper airway examination were facilitated by a tutor who wore the HoloLens device. The tutor interacted face to face with a patient and two-way audio-visual interaction was facilitated using the HoloLens 2 and Microsoft Teams with groups of students who were located in a separate tutorial room. Holographic functions were employed by the tutor. The tutor completed the System Usability Scale, the tutor, technical facilitator, patients, and students provided quantitative and qualitative feedback, and three students participated in semi-structured feedback interviews. Students completed pre- and post-tutorial, and end-of-year examinations on the tutorial topics.

Twelve patients and 78 students participated across 12 separate tutorials. Five students did not complete the examinations and were excluded from efficacy calculations. Student feedback contained 90 positive comments, including the technology’s ability to broadcast the tutor’s point-of-vision, and 62 negative comments, where students noted issues with the audio-visual quality, and concerns that the tutorial was not as beneficial as traditional in-person clinical tutorials. The technology and tutorial structure were viewed favourably by the tutor, facilitator and patients. Significant improvement was observed between students’ pre- and post-tutorial MCQ scores (mean 59.2% Vs 84.7%, p  < 0.001).

Conclusions

This study demonstrates the feasibility of using the HoloLens 2 to facilitate remote bedside tutorials which incorporate holographic learning artefacts. Students’ examination performance supports substantial learning of the tutorial topics. The tutorial structure was agreeable to students, patients and tutor. Our results support the feasibility of offering effective clinical teaching and learning opportunities using the HoloLens 2. However, the technical limitations and costs of the device are significant, and further research is required to assess the effectiveness of this tutorial format against in-person tutorials before wider roll out of this technology can be recommended as a result of this study

Peer Review reports

Introduction

Clinical tutorials which include encounters with real patients are recognised as integral elements in medical education [ 1 , 2 , 3 ]. Sir William Osler famously stated that “medicine is learned by the bedside and not in the classroom.” [ 4 ] However, many medical schools are facing challenges in delivering clinical education to students in an environment where there are increasing numbers of students, a limited number of patients and tutors, and increased scrutiny regarding the costs and environmental impacts of travel [ 5 , 6 , 7 , 8 ]. The COVID-19 pandemic also had a significant impact on in-person medical education in many countries, where students’ access to patients was severely curtailed [ 9 , 10 ]..

The argument that medical education requires interactive tutorials on actual patients is supported by various educational theories. Bandura’s Social Learning Theory and Social Cognitive Theory propose that students learn via attention, retention, reproduction and motivation [ 11 , 12 ]. This supports the need for direct observation and modelling of relevant clinical role-models participating in doctor-patient interactions [ 13 , 14 ]..

The Constructivist theory is based on the premise that the act of learning is based on a process which connects new knowledge to pre-existing knowledge [ 15 , 16 ]. Vertical Integration in medical education involves the integration of aspects of the curriculum across time, namely the integration of basic sciences and clinical sciences [ 17 , 18 , 19 ]..

Providing medical education within these frameworks, prioritising student exposure to direct interactions with clinicians and patients, and vertical integration of curriculum material, in situations where physical access to patients may be limited by numbers, logistics or infection control concerns poses a significant challenge to medical schools around the world. Utilising technology to facilitate the delivery of clinical education remotely may present a solution to these issues.

The broadcast of bedside tutorials to a remote location can be delivered using a “third-person” perspective, via a fixed or mobile broadcasting device, or using a first-person perspective, via a device mounted on the tutor. Devices which provide a first-person perspective are typically head-mounted-display devices (HMDs). The capabilities of these devices range widely, from basic two-way communication with a remote location, to devices with Augmented Reality (AR) and Mixed Reality (MR) functions which allow the integration of holographic artefacts into tutorials.

Augmented reality (AR) is a virtual environment that allows the user to view both their physical environment and virtual elements in real-time. Mixed Reality (MR) is an extension of AR which allows the real and holographic elements to interact [ 20 , 21 ]..

The use of AR and MR are expanding in many industries including healthcare, education, engineering, and manufacturing [ 22 , 23 , 24 ]. MR investigated in a variety of settings pertaining to medical education. Many early studies focused on teaching relevant anatomy, and more recently studies have evaluated the use of MR in procedural training, and its use in streaming of clinical ward-rounds to medical students [ 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 ]..

Head-mounted-display devices which offer MR experiences are growing in number and capability [ 34 ].The Microsoft HoloLens2 is one such device which enables the creation of an immersive Mixed Reality environment and can superimpose holographic images onto the user’s surroundings.

The HoloLens 2 has a number of specific capabilities which can be utilised in the virtual delivery of in-person clinical tutorials.The device can facilitate educationally effective, three-way communication between students, tutors and patients, as well as facilitating the incorporation of mixed reality elements into tutorials. The MR capabilities may provide a means of sharing holographic artefacts such as images and diagrams, which can allow the vertical and horizontal integration of curricular elements at the bedside.Utilisation of the MR capabilities of the device may improve student experiences and learning, in particular through instructional scaffolding (e.g rendering cell, organ or system pathways proximate to a patient) [ 35 ] Given the device’s connectivity capabilities, students can be in a separate geographical location to the patient and tutor. This has the potential to decrease student travel requirements and enables the delivery of tutorials to students in multiple different locations simultaneously [ 36 ]. The tutorial can also be delivered to a greater number of students than would be practical in a traditional bedside clinical tutorial environment. This can decrease the burden on both tutors and patients in comparison to multiple smaller group sessions. Finally, infection control risks are reduced as only the tutor enters patients’ environments.

Study goals

There is little published research to date which robustly evaluates the use of the HoloLens in replicating bedside tutorials while also incorporating mixed reality elements into the tutorials. The aims of this study are to evaluate the use of the Microsoft HoloLens 2 device to deliver a tutorial on preoperative anaesthetic history and upper airway examination to medical students in a remote location, while incorporating MR holograms in the tutorial delivery. Specific objectives include evaluating the feasibility of delivering tutorials with the HoloLens device, assessing the learning efficacy of these tutorials, and assessing student, tutor, facilitator, and patient perspectives of the tutorials.

This study was approved by the Clinical Research Ethic Committee of the Cork Teaching Hospitals, and the University College Cork Research and Postgraduate Affairs Committee. All participants including students, patients, tutor and technical facilitator provided written informed consent prior to inclusion in the study.

Study population

University College Cork medical students from two cohorts, third year Graduate-Entry and fourth year Direct-Entry medical students attending a tertiary referral teaching hospital for a clinical attachment with the Department of Anaesthesia and Intensive Care Medicine were invited to participate in the study. Both groups are in their second-last year of medical training, and thus have completed modules and examinations in basic medical sciences and clinical practice in the preceding years, with a maximum of 1 week experience in the field of anaesthesia [ 37 , 38 , 39 ]. Patients attending Cork University Hospital for scheduled surgery were selected and approached for consent by tutors according to clinical relevance. All participants were 18 years or over and were deemed capable of providing consent. Each student provided information on their age, gender and previous third-level qualifications.

Tutorial Sturcture

A one-hour tutorial focusing on completing a preoperative history and focused assessment of the upper airway was developed by MC (adjunct clinical lecturer), GI (Senior Clinical Lecturer) and GS (Professor) in line with the University curriculum’s learning objectives. (Fig. 1 ) Tutorials were delivered on a weekly basis to groups of third year Graduate Entry and fourth year Direct Entry medical students across the 2021–2022 academic year.

Preoperative Anaesthetic History and Focused Preoperative Assessment of the upper airway tutorial structure

All tutorials were delivered by one tutor (MC) and assisted by a technical facilitator (NOB), both males aged in their thirties, who enabled the connection between the site of the clinical encounter and nearby tutorial room. The tutor had no prior experience with the HoloLens 2 device or other AR HMDs prior to participation in this study; the facilitator had significant experience in its use. The tutor was given a period of familiarisation with the device which included using the Microsoft “HoloLens Tips” app, which provides a structured tutorial on the various hand gestures used to control the device, as well as a number of practice calls in order to test the network and audiovisual equipment in the tutorial room [ 40 ]. This familiarisation period totalled approximately 3 hours.

During the tutorial, the tutor (MC) interacted with a patient (face to face) in the pre-or postoperative units and remotely with a small group of [ 6 , 7 , 8 , 9 , 10 ] students in a nearby tutorial room. The remote interaction occurred via Hololens 2 worn by the tutor, institutional Wi-Fi (Eduroam), and Microsoft Teams.He demonstrated and explained the techniques of preoperative history taking and preoperative upper airway assessment.

Throughout the patient assessment the tutor interacted both with the patient and with the students as if conducting an in-person tutorial, providing additional information, asking the students pertinent questions, and expanding on the findings of the patient’s history and physical examination. Students communicated with the patient by asking questions via the tutor.

Resources employed

Resources necessary to provide the tutorials via the HoloLens included capital costs of the HoloLens device (€3500) and microphone (€88) as well as annual licence costs of €275 per user ( n  = 4). Human resources employed in developing the tutorials and trialling equipment included approximately 20 hours of training, remote assistance (Microsoft) and collaboration between the tutor (MC), Professor (GS) and facilitator (NOB), as well as 5 hours input from the Senior Clinical Lecturer (GI).

Internet connectivity

An internet connection of at least 1.5mpbs of bandwidth is recommended by Microsoft for best audio, visual and content sharing experience [ 41 ]. Secure, password protected wireless internet access via the University institutional network (Eduroam) was utilised by both tutor and students.

In most tutorials, broadcasts were hosted by an MSI running the Windows 10 operating system, audio was amplified using a Bose SoundLink Mini portable speaker and video was screened via a HDMI cable to a 36″ monitor. In one tutorial students accessed the tutorial via their personal smartphones or laptops. In order to bypass the noise cancellation technology within the HoloLens an external microphone (Saramonic SmartMic+UC L/weight Smartphone Mic USB-C) and 3.5 mm earphone were used.

Dynamics 365 Remote Assist application was used, in-tandem with Microsoft Teams, to host each video call. This connection allowed the students to see the tutors field of vision and hear both the tutor and patient. Hand gestures including the “hand-ray”, “air-tap”, “air-tap and hold” and “start-gesture” were used to control the HMD and manipulate the holographic artefacts. Relevant holographic artefacts were superimposed during the tutorial. This included the insertion of diagramatic representaions of the Mallampati scoring system and Thyromental Distance during the airway assessment portion of the tutorial [Fig. 2 (a) and (b)]. The holographic pointer and “drawing” functions were used by the tutor to highlight relevant upper airway structures and emphasise information on the holographic diagrams [Fig. 2 (c) and (d)].

a Assessment of Mallampati Score. b Assessment of Thyromental Distance. c Identification of thyroid cartilage using holographic pointer. d Illustration of holographic “drawing” function

Assessment of tutor perceptions

Immediately after completion of the first tutorial, the tutor completed a System Usability Scale assessment and on completion of the last tutorial, the tutor and facilitator summarised their perceptions of using the HMD.

Assessment of student perceptions

Immediately after completion of the tutorial, students completed a modified Evaluation of Technology-Enhanced Learning Materials: Learner Perceptions (ETELM-LP) questionnaire in order to assess their perceptions of the tutorial, which incorporated a seven-point Likert Scale and open questions [ 42 ]. Cronbach’s Alpha was calculated after exclusion of question 1 and reverse scoring of questions 13 and 15.

Three students also took part in semi-stuctured interviews via Microsoft Teams. Researchers undertook this study from an interpretive approach [ 43 ]. The interviews were conducted by JV, and followed a template of questions and corresponding probes from which the interviewer expanded as appropriate [Additional file 1 ]. The template served as a foundation from which the interviewer expanded as appropriate. The interviews were recorded and transcribed. Analysis of the interview transcripts and questionnare responses was performed using Dedoose Qualitative Research Software Version 4.3.Qualitative data from interviews and feedback questionnaires were coded thematically in alignment with Clarke and Braun’s suggestions for qualitative analysis [ 44 ]. Following the initial thematic coding, researchers conducted a content analysis to strengthen the interpretation of results. Illustrative quotes were chosen based on the representativeness of the theme or subtheme and the clarity of their intrinsic interpretation. In alignment with current literature, the quotes selected were determined to be illustrative of the point, reflective of patterns observed, and relatively succinct [ 45 ]..

Assessment of patient perceptions

On completion of the tutorials, patients were also asked to complete a mixed quantitative and qualitative questionnaire in order to assess their perceptions of the tutorial.

Assessment of learning efficacy

We carried out a prospective non-comparative study of tutorial efficacy. Students completed a pre-tutorial Multiple Choice Question (MCQ) examination to assess baseline knowledge [Additional file 2 ], and a post-tutorial MCQ two to 3 days later [Additional file 3 ]. Students then completed an end-of-year assessment two to 5 months later consisting of a data interpretation exam and an Objective Structured Clinical Examination (OSCE) which focused on preoperative history taking and preoperative assessment of the upper airway respectively [Additional files 4 and 5 ]. These examinations were written by an investigator and the University Senior Clinical Lecturer in line with University standards. Examination results were converted to percentages and the data interpretation and OSCE results were combined to give a total End-of-Year result.

The Chi-Squared test was used to compare direct-entry and graduate-entry student demographics. Welch’s two-sample t-Test assuming unequal variances was used to compare student group ages. The Shapiro Wilk and Kolmogorov-Smirnov Tests were used to assess to normality of distribution of student assessment scores for data sets less than 50 and greater than 50 respectively. The Mann-Whitney U Test was performed to compare group performance in assessments and overall student performance between the pre- and post-tutorial examinations, and between the post-tutorial and End-of-Year scores. Cohen’s d was calculated for the pre and post-tutorial MCQ scores to assess effect size.

Twelve tutorials were completed involving 12 separate patients and 78 students. Four students did not complete the post-tutorial MCQ and one did not complete the End-of-Year assessments due to illness related absences. These students were excluded from efficacy calculations. Baseline characteristics of the student participants are summarised in Table 1 . As expected the graduate-entry students was a significantly older cohort (graduate-entry median age 26 vs direct-entry mean of 22). Mean age of patient participants was 43.25, with an SD of 16.48, and a range of 18–64.

Feasibility

We found that it was feasible to use the HoloLens2 to facilitate weekly bedside tutorials on live patients in a busy, tertiary referral teaching hospital. No tutorials were cancelled or postponed due to technology-related issues. Of note, in order to improve the audio quality of the patient’s voice, it was neccessary to add the USB microphone, which is not routinely supplied with the HoloLens 2. The tutorials were also dependent on secure Wi-Fi access for both tutor and students, the presence of a tutorial facilitator to control the equipment at the student end, and access to a quiet space to examine the patient.

Tutor feedback

The sole tutor (MC) completed the System Usability Scale score, which was 72.5 (a score > 68 is deemed above average). The tutor (MC) stated that the HoloLens 2 was found to be comfortable to wear, the visor was unobtrusive and did not interfere with interaction with the patient or impede visualisation of clinical signs. The interaction with the device via hand gestures was relatively smooth and intuitive after the intial familiarisation period and the MR functions including the insertion of holographic diagrams, pointing, drawing and highlighting were useful. The holographic artefacts were visible throughout the tutorials at a “brightness” setting of seven out of 10.

Occasionally when talking to the students via the HMD, it was not clear to the patient if the tutor was talking to the patient or to the students. Utilising a structured pattern of speech such as “I am now talking to the students” was found to be useful to overcome this issue.

Facilitator feedback

The technical facilitator (NOB) found that the set-up of the live broadcast to the students was akin to that of a video presentation and that the learning curve for hosting the tutorials was short as the Dynamic 365 Remote Assist application was quite similar to general videoconferencing software. He noted that patient proximity to the tutor was essential to ensure adequate audio quality and referenced an example where a supine patient was farther from the device than normal and that patient responses had to be repeated by the tutor. Backgound noise was noted as a “minor issue and transient in nature”, and the technical facilitator accepted that a certain amount of background noise was unavoidable in an active hospital ward.

Student feedback

Quantitative student feedback via the modified ETELM-LP questionnaire is summarised in Fig. 3 . Results are presented as (mean, SD) and refer to a seven-point Likert scale. Students had little experience in MR prior to the tutorial (1.7, 1.29). They found the audio and visual quality was clear and that the MR elements of the tutorial were useful. Most agreed the tutorial approximated a live patient encounter (5.69, 1.26), was more beneficial than a PowerPoint-based tutorial, and were neutral when asked if it was as beneficial as a live clinical encounter (5, 1.69). They did not agree that the tutorial structure required inappropriately high technology skill levels on the part of the students, nor that the MR elements served as a distraction. Most agreed that they would like MR to be incorporated into further tutorials (6.05, 1). Cronbach’s Alpha, excluding question 1 was calculated as 0.86, displaying good internal consistency.

Student Modified ETELM-LP Scores. 7 point Likert scale with 7 as strongly agree and 1 as strongly disagree. Presented as Mean +/− 1 Standard Deviation

Student qualitative feedback results

Analysis of written and verbal feedback from 78 students identified 90 specific positive excerpts and 62 negatives (Table 2 ). Positive feedback included the technology’s ability to broadcast the tutor’s point-of-vision, the inclusion of holographic artefacts, and the remote nature of the tutorial. Negative feedback included issues with the audio-visual stream quality, the fact that students were not able to individually carry out the practical examination, and 11 students expressed concerns that the tutorial was not as useful as traditional in-person bedside clinical tutorials.

Three students participated in semi-structured interviews. The limited sense of “presence” and interaction with the patient were identified as limitations to the format by all three interviewees. With respect to the physical examination one student explained he would have preferred to “experience it yourself, and have a look and feel and touch”. Specific mention was made of the value of combining broadcast (patient) and rendered (schematics) images, “The adding of the images … right next to the patient was really, really helpful”. This may indicate the potential to employ this format to support vertical and horizontal integration of curricular elements. All three interviewed students reported either a six or seven (on a verbal scale of 1–7) when asked to recommend this technology for inclusion in the medical curriculum.

Patient feedback

Quantitative feedback data from patient questionnaires is summarised in Fig. 4 . Most patients had little experience with MR in the past (mean, SD: 1.75, 1.48) apart from one patient who scored 6. All agreed that the communication with the tutor was clear, that they felt safe, that the experience was enjoyable and that they would participate in a similar session in the future. Six of seven expressed that it was preferable to both small (5 or less) and large group in-person tutorials. Most patients did not agree that the HoloLens served as a distraction or made them uncomfortable.

Patient Feedback Questionnaire Results. 7 point Likert scale with 7 as strongly agree and 1 as strongly disagree. Presented as Mean +/− 1 Standard Deviation

Five patients gave qualitative feedback. Positive comments included that “it is good to see that you are moving on with new technology”, “it was well explained beforehand so I was very comfortable” and “it was fantastic to teach students when they can’t be at the bedside. Very unobtrusive”. One patient commented that “sometimes not sure if he [the tutor] was talking to me or the students” and another commented that “it would be lovely to see who I was talking to [the student group]”.

Learning efficacy

Student examination scores are sumarised in Table 3 and Fig. 5 . Student assessment scores were not normally distrubuted. A statistically significant improvement was observed between overall students’ pre and post tutorial MCQ scores (mean 59.2% Vs 84.7%, p  < 0.001). Cohen’s d was 0.612, indicating a medium effect size. There was a statistically significant difference in student performance between the post tutorial MCQ and the composite End-of-Year scores (84.7% Vs 82.2%, p  < 0.05). There were no statistically significant differences found between the graduate-entry and direct-entry students for any individual examination.

Boxplot of overall student assessment scores

Mixed Reality headsets offer several novel capabilities which can facilitate remote education and vertical and horizontal integration of curriculum elements, particularly when aligned with appropriate educational theories such as Constructivism and Social Cognitive Theory. A large number of studies have focused on applying the technology in surgical and anatomical subject fields [ 46 ]. However, there are significant gaps in the evidence base, particularly studies specific to anaesthesiology, clinical exam, and addressing the provision of interactive tutorials to remote locations. Our study has demonstrated that it is feasible and effective to use the Microsoft HoloLens 2, incorporating its Mixed Reality functions to provide a live bedside tutorial on anaesthetic preoperative assessment to students situated in a remote location. Feedback from students, patients and the tutor were generally positive. Quantitative feedback from students regarding the audio-visual quality was mainly positive, however technical issues were noted, and preference for in-person tutorials was expressed by a minority of students.

Mill et al. previously examined the feasibility of the HoloLens 2 in broadcasting medical ward rounds [ 26 ]. While papers such as that by Mill et al. demonstrated the feasibility of utilizing the HoloLens 2 HMD to stream educational ward-rounds, they did not utilize the MR functions of the HMD, nor assess the learning efficacy of the device [ 26 ]. This study incorporates both quantitative and qualitative feedback from multiple sources, namely students, patients, the tutor, and tutorial facilitator. We believe this demonstrates a robust examination of the perceptions of the relevant stakeholders involved in the provision of clinical tutorials to medical students. Our findings that the tutorials were feasible, agreeable to both patients and students, and that students had occasional audio-visual difficulties are consistent with those of Mill et al. Our study additionally demonstrates that incorporation of holographic artefacts is both feasible and regarded by the tutor and students as useful, and that the tutorials provide effective knowledge acquisition.

Our tutorial format aimed to reproduce some of the educationally relevant components of an in-person tutorial. Other suggested structures advocate streaming video of the physician as opposed to the physician’s point-of-view [ 47 ]. The HoloLens 2 device allows the students to view the tutor’s field of vision which we argue is superior, and student feedback reflected this. This viewpoint allows students to appreciate in real time the clinical signs demonstrated during the clinical examination and correlate these with the holographic diagrammatic examples used. The MR environment provides an ideal setting to facilitate vertical integration in real time by displaying holographic artefacts of anatomical, physiological and pathological information, as well as patient specific data such as radiological imaging or lab results while interacting with a patient. Furthermore, delivering tutorials remotely reduces infection-control concerns and allows delivery to greater numbers of students in multiple locations.

Preserving patient confidentiality is essential in medical practice and education. In our study, both the HMD and devices at the student end were connected to secure institutional Wi-Fi and accessed via University accounts. Also, access to the audio-visual stream was controlled by the technical facilitator, and the students were located in a supervised tutorial room. It would be essential to control both access to the tutorial and the environment to which it is broadcast to maintain confidentiality.

Limitations

Our study design has a number of limitations. It is non-comparative, and thus we are unable to draw conclusions regarding the relative learning experience or efficacy associated with tutorials delivered via the HoloLens device and the more traditional in-person bedside tutorials. Additionally, the different assessment methods between the MCQs and end of year examinations make direct measurement of knowledge retention difficult. The number of patients involved in the study was relatively small, and thus interpretation of both quantitative and qualitative data must be viewed in this context, and the generalisability of the data is low. The feedback from the tutor and tutorial facilitator must be viewed in the context that they were study investigators.

There are a number of limitations specific to research involving the HoloLens. Common limitations in studying the learning effects of the HoloLens in tested roles include the absence of validated measures and comprehensive evaluation instruments. Unlike other technologies, there are no benchmarks, datasets, or standard standardized protocols to specifically evaluate augmented reality systems, experiences, and methodologies [ 48 , 49 , 50 ]. Although the viewpoint offered to the students by the HoloLens allows the students to appreciate what the tutor is demonstrating, one drawback to this is that the focus of attention is primarily controlled by the tutor, and thus it is difficult for the tutorial to challenge the students to select the relevant areas to attend to. Depending on the tutorial topic and structure, an ideal virtual format may provide three perspectives: the tutors view, a third person view of the clinical encounter, and where applicable, an instrument’s view.

Regarding the generalisability of our study to other tutorial topics, the appreciation of clinical signs which would require palpation or auscultation would be beyond the current capabilities of the HoloLens 2 and therefore, careful tutorial design and topic selection is necessary.

Our results demonstrate the feasibility of facilitating remote bedside tutorials on preoperative anaesthetic assessment using the HoloLens 2. The tutorial structure was found to be agreeable to students, patients, and tutors. Provision of tutorials in the format described in this study may be an option for situations where students’ access to live bedside tutorials are limited. However, further research is required to characterise the role, potential and limitations of incorporating Mixed Reality into clinical medical education in a broader context. Poor audio-visual quality and lack of hands-on practice were found to be the most frequent issues identified in our study and may be significant limitations to the use of this technology in wider medical education. There are significant costs involved in developing the infrastructure and expertise necessary to provide tutorials in this format. Prior to this technology being adopted by educational institutions, we recommend the completion studies to compare the learning efficacy of MR facilitated remote tutorials and traditional in-person bedside tutorials.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

• Augmented Reality

Evaluation of Technology-Enhanced Learning Materials: Learner Perceptions

Head-Mounted Display

Interquartile Range

Multiple Choice Question

• Mixed Reality

Objective Structured Clinical Examination

Standard Deviation

Keifenheim KE, Teufel M, Ip J, Speiser N, Leehr EJ, Zipfel S, et al. Teaching history taking to medical students: a systematic review. BMC Med Educ. 2015;15(1):159.

Article   Google Scholar  

Wessely S. Making doctors: an institutional apprenticeship. BMJ. 1998;316(7132):713.

Dornan T, Tan N, Boshuizen H, Gick R, Isba R, Mann K, et al. How and what do medical students learn in clerkships? Experience based learning (ExBL). Adv Health Sci Educ Theory Pract. 2014;19(5):721–49.

Stone MJ. The wisdom of sir William Osler. Am J Cardiol. 1995;75(4):269–76.

Walsh K. E-learning in medical education: the potential environmental impact. Educ Prim Care. 2018;29(2):104–6.

Sherpa JR, Donahue L, Tsai J, Nguemeni Tiako MJ. The planetary benefit of suspending USMLE step 2 CS: estimating carbon emissions associated with US medical Students' travel to testing centers. Yale J Biol Med. 2023;96(2):185–8.

Sharma D, Rizzo J, Nong Y, et al. Virtual Learning Decreases the Carbon Footprint of Medical Education. Dermatol Ther (Heidelb). 2024;14(4):853–9. https://doi.org/10.1007/s13555-024-01120-4 .

Medical Schools Council. The expansion of medical student numbers in the United Kingdom. London: MSC; 2021. Available at https://www.medschools.ac.uk/media/2899/the-expansion-of-medical-student-numbers-in-the-unitedkingdom-msc-position-paper-october-2021.pdf . Accessed April 2024.

Harries AJ, Lee C, Jones L, Rodriguez RM, Davis JA, Boysen-Osborn M, et al. Effects of the COVID-19 pandemic on medical students: a multicenter quantitative study. BMC Med Educ. 2021;21(1):14.

Choi B, Jegatheeswaran L, Minocha A, Alhilani M, Nakhoul M, Mutengesa E. The impact of the COVID-19 pandemic on final year medical students in the United Kingdom: a national survey. BMC Med Educ. 2020;20(1):206.

Bandura A. Social learning theory. Oxford: Prentice-Hall; 1977.

Irby DM. Clinical teaching and the clinical teacher. J Med Educ. 1986;61(9 Pt 2):35–45.

Google Scholar  

Horsburgh J, Ippolito K. A skill to be worked at: using social learning theory to explore the process of learning from role models in clinical settings. BMC Med Educ. 2018;18(1):156.

Park J, Woodrow SI, Reznick RK, Beales J, MacRae HM. Observation, reflection, and reinforcement: surgery faculty members' and residents' perceptions of how they learned professionalism. Acad Med. 2010;85(1):134–9.

Dennick R. Constructivism: reflections on twenty five years teaching the constructivist approach in medical education. Int J Med Educ. 2016;7:200–5.

Steffe LP, Gale J, editors. Constructivism in education. 1st ed. New York: Routledge; 1995.

Kulasegaram KM, Martimianakis MA, Mylopoulos M, Whitehead CR, Woods NN. Cognition before curriculum: rethinking the integration of basic science and clinical learning. Acad Med. 2013;88(10):1578–85.

Wijnen-Meijer M, van den Broek S, Koens F, ten Cate O. Vertical integration in medical education: the broader perspective. BMC Med Educ. 2020;20(1):509.

Brauer DG, Ferguson KJ. The integrated curriculum in medical education: AMEE guide no. 96. Medical Teacher. 2015;37(4):312–22.

Azuma RT. A survey of augmented reality. Presence: teleoperat virt environ. 1997;6(4):355–85.

Hu HZ, Feng XB, Shao ZW, Xie M, Xu S, Wu XH, et al. Application and Prospect of mixed reality Technology in Medical Field. Curr Med Sci. 2019;39(1):1–6.

Garzón J, Pavón J, Baldiris S. Systematic review and meta-analysis of augmented reality in educational settings. Virt Real. 2019;23(4):447–59.

Hantono BS, Nugroho LE, Santosa PI, editors. Meta-Review of Augmented Reality in Education. 2018 10th International Conference on Information Technology and Electrical Engineering (ICITEE); 2018 24–26 July 2018.

Hanna MG, Ahmed I, Nine J, Prajapati S, Pantanowitz L. Augmented Reality Technology Using Microsoft HoloLens in Anatomic Pathology. Arch Pathol Lab Med. 2018;142(5):638-644. https://doi.org/10.5858/arpa.2017-0189-OA .

Eckert M, Volmerg JS, Friedrich CM. Augmented reality in medicine: systematic and bibliographic review. JMIR Mhealth Uhealth. 2019;7(4):e10967.

Mill T, Parikh S, Allen A, Dart G, Lee D, Richardson C, et al. Live streaming ward rounds using wearable technology to teach medical students: a pilot study. BMJ Simul Technol Enhanc Learn. 2021;7(6):494–500.

Lia H, Paulin G, Yi N, Haq H, Emmanuel S, Ludig K, Keri Z, Lasso A, Yeo C, Andrews J, Fichtinger G. HoloLens in suturing training. Proc. SPIE 10576, Medical Imaging 2018: Image-Guided Procedures, Robotic Interventions, and Modeling, 2018;1057628. https://doi.org/10.1117/12.2293934 .

Carmigniani J, Furht B, Anisetti M, Ceravolo P, Damiani E, Ivkovic M. Augmented reality technologies, systems and applications. Multimed Tools Appl. 2011;51(1):341–77.

Silvero Isidre A, Friederichs H, Müther M, Gallus M, Stummer W, Holling M. Mixed reality as a teaching tool for medical students in neurosurgery. Med Int. 2023;59(10)

Kolecki R, Pręgowska A, Dąbrowa J, Skuciński J, Pulanecki T, Walecki P, et al. Assessment of the utility of mixed reality in medical education. Translat Res Anatom. 2022;28:100214.

Robinson BL, Mitchell TR, Brenseke BM. Evaluating the use of mixed reality to teach gross and microscopic respiratory anatomy. Med Sci Educ. 2020;30(4):1745–8.

Maniam P, Schnell P, Dan L, Portelli R, Erolin C, Mountain R, et al. Exploration of temporal bone anatomy using mixed reality (HoloLens): development of a mixed reality anatomy teaching resource prototype. J Vis Commun Med. 2020;43(1):17–26.

Kumar N, Pandey S, Rahman E. A novel three-dimensional interactive virtual face to facilitate facial anatomy teaching using Microsoft HoloLens. Aesth Plast Surg. 2021;45(3):1005–11.

Cipresso P, Giglioli IAC, Raya MA, Riva G. The Past, Present, and Future of Virtual and Augmented Reality Research: A Network and Cluster Analysis of the Literature. Front Psychol. 2018;9:2086. https://doi.org/10.3389/fpsyg.2018.02086 .

Quarles J, Lampotang S, Fischler I, Fishwick P, Lok B. Scaffolded learning with mixed reality. Comput Graph. 2009;33(1):34–46.

Palumbo A. Microsoft HoloLens 2 in medical and healthcare context: state of the art and future prospects. Sensors (Basel). 2022;22(20)

University College Cork. Book of Modules 2023. [Internet]. Cork. Cited March 2024. Available from: https://ucc-ie-public.courseleaf.com/modules /.

University College Cork School of Medicine. Medicine (Graduate Entry) (MB, BCh, BAO Degrees) - CK791 Programme Requirements 2023. [Internet]. Cork. Cited March 2024. Available from: https://ucc-iepublic.courseleaf.com/programmes/bmbbg/#programmerequirementstext .

University College Cork School of Medicine. Medicine (MB, BCh, BAO Degrees) - CK701 Programme Requirements 2023. Internet. CorkCited March 2024. Available from: https://ucc-ie-public.courseleaf.com/programmes/bmbb/#programmerequirementstext .

Microsoft Corporation. HoloLens Tips 2023. Internet. [Internet]. Cited March 2024. Available from: https://www.microsoft.com/en-us/p/hololens-tips/9pd4cxkklc47?activetab=pivot:overviewtab .

Microsoft Corporation. Infrastructure Guidelines for HoloLens. Microsoft Learn. [Internet]. Cited October 2022. Available from: https://learn.microsoft.com/en-us/hololens/hololens-commercial-infrastructure .

Cook DA, Ellaway RH. Evaluating technology-enhanced learning: A comprehensive framework. Med Teach. 2015;37(10):961–70.

Thompson Burdine J, Thorne S, Sandhu G. Interpretive description: A flexible qualitative methodology for medical education research. Med Educ. 2021;55(3):336–43.

Clarke V, Braun V. Teaching thematic analysis: overcoming challenges and developing strategies for effective learning. Psychol. 2013;26:120–3.

Lingard L. Beyond the default colon: effective use of quotes in qualitative research. Perspect Med Educ. 2019;8(6):360–4.

Tang YM, Chau KY, Kwok APK, Zhu T, Ma X. A systematic review of immersive technology applications for medical practice and education - trends, application areas, recipients, teaching contents, evaluation methods, and performance. Educ Res Rev. 2022;35:100429.

Pennell CE, Kluckow H, Chen SQ, Wisely KM, Walker BL. Live-streamed ward rounds: a tool for clinical teaching during the COVID-19 pandemic. Med J Aust. 2020;213(7):306–8.e1.

Herron J. Augmented reality in medical education and training. J Electron Resour Med Libr. 2016;13:1–5.

Gerup J, Soerensen CB, Dieckmann P. Augmented reality and mixed reality for healthcare education beyond surgery: an integrative review. Int J Med Educ. 2020;11:1–18.

Barsom EZ, Graafland M, Schijven MP. Systematic review on the effectiveness of augmented reality applications in medical training. Surg Endosc. 2016;30(10):4174–83.

Download references

Acknowledgements

The authors would like to acknowledge the assistance from members of the UCC College of Medicine and Health, including Dr. Colm O’Tuathaigh, Dr. Gabriella Rizzo, Dr. Pat Henn and Professor Paula O’Leary, as well as Ms. Michelle Donovan in the UCC Centre for Digital Education.

This study received funding and research support through the UCC Learning Analytics LITE programme, which is funded through the Strategic Alignment of Teaching and Learning Enhancement fund. The UCC Learning Analytics LITE programme provided logistical and research support in study design and funds were used to hire assistance in data interpretation.

This study also received funding from the UCC College of Medicine and Health which was utilised to purchase the HoloLens 2 Device and associated licences.

Author information

Authors and affiliations.

Cork University Hospital and University College Cork, Cork, Ireland

Murray Connolly, Gabriella Iohom, Corina Soare & George Shorten

University College Cork, Cork, Ireland

Niall O’Brien & James Volz

Cork University Hospital, Cork, Ireland

Aogán O’Muircheartaigh, Paschalitsa Serchan, Agatha Biculescu, Kedar Govind Gadre & Laura Griseto

You can also search for this author in PubMed   Google Scholar

Contributions

MC lead the design of the study, carried out the tutorials, analysed both quantitative and qualitative data and was the primary author of the manuscript. GI contributed to the design of the study, the student examinations contributed to writing the manuscript. NOB contributed to the technical and logistical design of the study and acted as technical facilitator for the tutorials and contributed to manuscript composition. JV designed, completed and analysed the semi-structured student interviews and contributed to manuscript composition. AOM, PS, AB, LG and supervised and analysed student examination data. KGG analysed student demographic data and student examination data. CS contributed to initial evaluation of the HoloLens device and tutorial design. GS played a central role in study design and completion and was a major contributor in manuscript composition.

Corresponding author

Correspondence to Murray Connolly .

Ethics declarations

Ethics approval and consent to participate.

This study was approved by the Clinical Research Ethic Committee of the Cork Teaching Hospitals, and the University College Cork Research and Postgraduate Affairs Committee. All methods were carried out in accordance with guidelines and regulations as set out by the ethics and research committees. All participants provided informed consent to participate in the study.

Consent for publication

All participants including students, patients, tutor and technical facilitator provided written informed consent prior to inclusion in the study. Participants who’s identifiable images are included provided informed consent for publication of identifiable information/ images in an open access journal.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1., additional file 2., additional file 3., additional file 4., additional file 5., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Connolly, M., Iohom, G., O’Brien, N. et al. Delivering clinical tutorials to medical students using the Microsoft HoloLens 2: A mixed-methods evaluation. BMC Med Educ 24 , 498 (2024). https://doi.org/10.1186/s12909-024-05475-2

Download citation

Received : 09 March 2023

Accepted : 26 April 2024

Published : 04 May 2024

DOI : https://doi.org/10.1186/s12909-024-05475-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Medical Education

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]