quality improvement projects and clinical research studies

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Adv Pract Oncol
• v.12(4); 2021 May

Quality Improvement Projects and Clinical Research Studies

Every day, I witness firsthand the amazing things that advanced practitioners and nurse scientists accomplish. Through the conduct of quality improvement (QI) projects and clinical research studies, advanced practitioners and nurse scientists have the opportunity to contribute exponentially not only to their organizations, but also towards personal and professional growth.

Recently, the associate editors and staff at JADPRO convened to discuss the types of articles our readership may be interested in. Since we at JADPRO believe that QI projects and clinical research studies are highly valuable methods to improve clinical processes or seek answers to questions, you will see that we have highlighted various QI and research projects within the Research and Scholarship column of this and future issues. There have also been articles published in JADPRO about QI and research ( Gillespie, 2018 ; Kurtin & Taher, 2020 ). As a refresher, let’s explore the differences between a QI project and clinical research.

Quality Improvement

As leaders in health care, advanced practitioners often conduct QI projects to improve their internal processes or streamline clinical workflow. These QI projects use a multidisciplinary team comprising a team leader as well as nurses, PAs, pharmacists, physicians, social workers, and program administrators to address important questions that impact patients. Since QI projects use strategic processes and methods to analyze existing data and all patients participate, institutional review board (IRB) approval is usually not needed. Common frameworks, such as Lean, Six Sigma, and the Model for Improvement can be used. An attractive aspect of QI projects is that these are generally quicker to conduct and report on than clinical research, and often with quantifiable benefits to a large group within a system ( Table 1 ).

Clinical Research

Conducting clinical research through an IRB-approved study is another area in which advanced practitioners and nurse scientists gain new knowledge and contribute to scientific evidence-based practice. Research is intended for specific groups of patients who are protected from harm through the IRB and ethical principles. Research can potentially benefit a larger group, but benefits to participants are often unknown during the study period.

Clinical research poses many challenges at various stages of what can be a lengthy process. First, the researcher conducts a review of the literature to identify gaps in existing knowledge. Then, the researcher must be diligent in their self-reflection (is this phenomenon worth studying?) and in developing the sampling and statistical methods to ensure validity and reliability of the research ( Higgins & Straub, 2006 ). A team of additional researchers and support staff is integral to completing the research and disseminating findings. A well-designed clinical trial is worth the time and effort it takes to answer important clinical questions.

So, as an advanced practitioner, would a QI project be better to conduct than a clinical research study? That depends. A QI project uses a specific process, measures, and existing data to improve outcomes in a specific group. A research study uses an IRB-approved study protocol, strategic methods, and generates new data to hopefully benefit a larger group.

In This Issue

Both QI projects and clinical research can provide evidence to base one’s interventions on and enhance the lives of patients in one way or another. I hope you will agree that this issue is filled with valuable information on a wide range of topics. In the following pages, you will learn about findings of a QI project to integrate palliative care into ambulatory oncology. In a phenomenological study, Carrasco explores patient communication preferences around cancer symptom reporting during cancer treatment.

We have two excellent review articles for you as well. Rogers and colleagues review the management of hematologic adverse events of immune checkpoint inhibitors, and Lemke reviews the evidence for use of ginseng in the management of cancer-related fatigue. In Grand Rounds, Flagg and Pierce share an interesting case of essential thrombocythemia in a 15-year-old, with valuable considerations in the pediatric population. May and colleagues review practical considerations for integrating biosimilars into clinical practice, and Moore and Thompson review BTK inhibitors in B-cell malignancies.

• Higgins P. A., & Straub A. J. (2006). Understanding the error of our ways: Mapping the concepts of validity and reliability . Nursing Outlook , 54 ( 1 ), 23–29. 10.1016/j.outlook.2004.12.004 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Gillespie T. W. (2018). Do the right study: Quality improvement projects and human subject research—both valuable, simply different . Journal of the Advanced Practitioner in Oncology , 9 ( 5 ), 471–473. 10.6004/jadpro.2018.9.5.1 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Kurtin S. E., & Taher R. (2020). Clinical trial design and drug approval in oncology: A primer for the advanced practitioner in oncology . Journal of the Advanced Practitioner in Oncology , 11 ( 7 ), 736–751. 10.6004/jadpro.2020.11.7.7 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]

Quality Improvement Projects and Clinical Research Studies

• PMID: 34123473
• PMCID: PMC8163249
• DOI: 10.6004/jadpro.2021.12.4.1

Publication types

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• How to get started in...

How to get started in quality improvement

Linked opinion.

The benefits of QI are numerous and the challenges worth overcoming

Read the full collection

• Related content
• Peer review
• Bryan Jones , improvement fellow 1 ,
• Emma Vaux , consultant nephrologist 2 ,
• Anna Olsson-Brown , research fellow 3
• 1 The Health Foundation, London, UK
• 2 Royal Berkshire NHS Foundation Trust. Reading, UK
• 3 Department of Molecular and Clinical Pharmacology, The Institute of Translational Medicine, University of Liverpool, Liverpool, UK
• Correspondence to B Jones bryan.jones{at}health.org.uk

What you need to know

Participation in quality improvement can help clinicians and trainees improve care together and develop important professional skills

Effective quality improvement relies on collaborative working with colleagues and patients and the use of a structured method

Enthusiasm, perseverance, good project management skills, and a willingness to explain your project to others and seek their support are key skills

Quality improvement ( box 1 ) is a core component of many undergraduate and postgraduate curriculums. 1 2 3 4 5 Numerous healthcare organisations, 6 professional regulators, 7 and policy makers 8 recognise the benefits of training clinicians in quality improvement.

Defining quality improvement 1

Quality improvement aims to make a difference to patients by improving safety, effectiveness, and experience of care by:

Using understanding of our complex healthcare environment

Applying a systematic approach

Designing, testing, and implementing changes using real time measurement for improvement

Engaging in quality improvement enables clinicians to acquire, assimilate, and apply important professional capabilities 7 such as managing complexity and training in human factors. 1 For clinical trainees, it is a chance to improve care 9 ; develop leadership, presentation, and time management skills to help their career development 10 ; and build relationships with colleagues in organisations that they have recently joined. 11 For more experienced clinicians, it is an opportunity to address longstanding concerns about the way in which care processes and systems are delivered, and to strengthen their leadership for improvement skills. 12

The benefits to patients, clinicians, and healthcare providers of engaging in quality improvement are considerable, but there are many challenges involved in designing, delivering, and sustaining an improvement intervention. These range from persuading colleagues that there is a problem that needs to be tackled, through to keeping them engaged once the intervention is up and running as other clinical priorities compete for their attention. 13 You are also likely to have competing priorities and will need support to make time for quality improvement. The organisational culture, such as the extent to which clinicians are able to question existing practice and try new ideas, 14 15 16 also has an important bearing on the success of the intervention.

This article describes the skills, knowledge, and support needed to get started in quality improvement and deliver effective interventions.

What skills do you need?

Enthusiasm, optimism, curiosity, and perseverance are critical in getting started and then in helping you to deal with the challenges you will inevitably face on your improvement journey.

Relational skills are also vital. At its best quality improvement is a team activity. The ability to collaborate with different people, including patients, is vital for a project to be successful. 17 18 You need to be willing to reach out to groups of people that you may not have worked with before, and to value their ideas. 19 No one person has the skills or knowledge to come up with the solution to a problem on their own.

Learning how systems work and how to manage complexity is another core skill. 20 An ability to translate quality improvement approaches and methods into practice ( box 2 ), coupled with good project and time management skills, will help you design and implement a robust project plan. 27

Quality improvement approaches

Healthcare organisations use a range of improvement methods, 21 22 such as the Model for Improvement , where changes are tested in small cycles that involve planning, doing, studying, and acting (PDSA), 23 and Lean , which focuses on continually improving processes by removing waste, duplication, and non-value adding steps. 24 To be effective, such methods need to be applied consistently and rigorously, with due regard to the context. 25 In using PDSA cycles, for example, it is vital that teams build in sufficient time for planning and reflection, and do not focus primarily on the “doing.” 26

Equally important is an understanding of the measurement for improvement model, which involves the gradual refinement of your intervention based on repeated tests of change. The aim is to discover how to make your intervention work in your setting, rather than to prove it works, so useful data, not perfect data, are needed. 28 29 Some experience of data collection and analysis methods (including statistical analysis tools such as run charts and statistical process control) is useful, but these will develop with increasing experience. 30 31

Most importantly, you need to enjoy the experience. It is rare that a clinician can institute real, tangible change, but with quality improvement this is a real possibility, which is both empowering and satisfying. Finally, don’t worry about what you don’t know. You will learn by doing. Many skills needed to implement successful quality improvement will be developed as you go; this is a fundamental feature of quality improvement.

How do you get started?

The first step is to recruit your improvement team. Start with colleagues and patients, 32 but also try to bring in people from other professions, including non-clinical staff. You need a blend of skills and perspectives in your team. Find a colleague experienced in quality improvement who is willing to mentor or supervise you.

Next, identify a problem collaboratively with your team. Use data to help with this (eg, clinical audits, registries of data on patients’ experiences and outcomes, and learning from incidents and complaints) ( box 3 ). Take time to understand what might be causing the problem. There are different techniques to help you (process mapping, five whys, appreciative inquiry). 35 36 37 Think about the contextual factors that are contributing to the problem (eg, the structure, culture, politics, capabilities and resources of your organisation).

Clinical audit and quality improvement

Quality improvement is an umbrella term under which many approaches sit, clinical audit being one. 33 Clinical audit is commonly used by trainees to assess clinical effectiveness. Confusion of audit as both a term for assurance and improvement has perhaps limited its potential, with many audits ending at the data collection stage and failing to lead to improvement interventions. Learning from big datasets such as the National Clinical Audits in the UK is beginning to shift the focus to a quality improvement approach that focuses on identifying and understanding unwanted variation in the local context; developing and testing possible solutions, and moving from one-off change to multiple cycles of change. 34

Next, develop your aim using the SMART framework: Specific (S), Measurable (M), Achievable (A), Realistic (R), and Timely (T). 38 This allows you to assess the scale of the intervention and to pare it down if your original idea is too ambitious. Aligning your improvement aim with the priorities of the organisation where you work will help you to get management and executive support. 39

Having done this, map those stakeholders who might be affected by your intervention and work out which ones you need to approach, and how to sell it to them. 40 Take the time to talk to them. It will be appreciated and increases the likelihood of buy in, without which your quality improvement project is likely to fail irrespective of how good your idea is. You need to be clear in your own mind about the reasons you think it is important. Developing an “elevator pitch” based on your aims is a useful technique to persuade others, 38 remembering different people are hooked in for different reasons.

The intervention will not be perfect first time. Expect a series of iterative changes in response to false starts and obstacles. Measuring the impact of your intervention will enable you to refine it. 28 Time invested in all these aspects will improve your chances of success.

Right from the start, think about how improvement will be embedded. Attention to sustainability will mean that when you move to your next job your improvement efforts, and those of others, and the impact you have collectively achieved will not be lost. 41 42

What support is needed?

You need support from both your organisation and experienced colleagues to translate your skills into practice. Here are some steps you can take to help you make the most of your skills:

Find the mentor or supervisor who will help identify and support opportunities for you. Signposting and introduction to those in an organisation who will help influence (and may hinder) your quality improvement project is invaluable

Use planning and reporting tools to help manage your project, such as those in NHS Improvement’s project management framework 27

Identify if your local quality improvement or clinical audit team may be a source of support and useful development resource for you rather than just a place to register a project. Most want to support you.

Determine how you might access (or develop your own) local peer to peer support networks, coaching, and wider improvement networks (eg, NHS networks; Q network 43 44 )

Use quality improvement e-learning platforms such as those provided by Health Education England or NHS Education for Scotland to build your knowledge 45 46

Learn through feedback and assessment of your project (eg, via the QIPAT tool 47 or a multi-source feedback tool. 48 49

Quality improvement approaches are still relatively new in the education of healthcare professionals. Quality improvement can give clinicians a more productive, empowering, and educational experience. Quality improvement projects allow clinicians, working within a team, to identify an issue and implement interventions that can result in true improvements in quality. Projects can be undertaken in fields that interest clinicians and give them transferable skills in communication, leadership, project management, team working, and clinical governance. Done well, quality improvement is a highly beneficial, positive process which enables clinicians to deliver true change for the benefit of themselves, their organisations, and their patients.

Quality improvement in action: three doctors and a medical student talk about the challenges and practicalities of quality improvement

This box contains four interviews by Laura Nunez-Mulder with people who have experience in quality improvement.

Alex Thompson, medical student at the University of Cambridge, is in the early stages of his first quality improvement project

We are aiming to improve identification and early diagnosis of aortic dissections in our hospital. Our supervising consultant suspects that the threshold for organising computed tomography angiography for a suspected aortic dissection is too high, so to start with, my student colleague and I are finding out what proportion of CT angiograms result in a diagnosis of aortic dissection.

I fit the project around my studies by working on it in small chunks here and there. You have to be very self motivated to see a project through to the end.

Anna Olsson-Brown, research fellow at the University of Liverpool, engaged in quality improvement in her F1 year, and has since supported junior doctors to do the same. This extract is adapted from her BMJ Opinion piece ( https://blogs.bmj.com/bmj/ )

Working in the emergency department after my F1 job in oncology, I noticed that the guidelines on neutropenic sepsis antibiotics were relatively unknown and even less frequently implemented. A colleague and I devised a neutropenic sepsis pathway for oncology patients in the emergency department including an alert label for blood tests. The pathway ran for six months and there was some initial improvement, but the benefit was not sustained after we left the department.

As an ST3, I mentored a junior doctor whose quality improvement project led to the introduction of a syringe driver prescription sticker that continues to be used to this day.

My top tips for those supporting trainees in quality improvement:

Make sure the project is sufficiently narrow to enable timely delivery

Ensure regular evaluation to assess impact

Support trainees to implement sustainable pathways that do not require their ongoing input.

Amar Puttanna, consultant in diabetes and endocrinology at Good Hope Hospital, describes a project he carried out as a chief registrar of the Royal College of Physicians

The project of which I am proudest is a referral service we launched to review medication for patients with diabetes and dementia. We worked with practitioners on the older adult care ward, the acute medical unit, the frailty service, and the IT teams, and we promoted the project in newsletters at the trust and the Royal College of Physicians.

The success of the project depended on continuous promotion to raise awareness of the service because junior doctors move on frequently. Activity in our project reduced after I left the trust, though it is still ongoing and won a Quality in Care Award in November 2018.

Though this project was a success, not everything works. But even the projects that fail contain valuable lessons.

Mark Taubert, consultant in palliative medicine and honorary senior lecturer for Cardiff University School of Medicine, launched the TalkCPR project

Speaking to people with expertise in quality improvement helped me to narrow my focus to one question: “Can videos be used to inform both staff and patients/carers about cardiopulmonary resuscitation and its risks in palliative illness?” With my team I created and evaluated TalkCPR, an online resource that has gone on to win awards (talkcpr.wales).

The most challenging aspect was figuring out which tools might get the right information from any data I collected. I enrolled on a Silver Improving Quality Together course and joined the Welsh Bevan Commission, where I learned useful techniques such as multiple PDSA (plan, do, study, act) cycles, driver diagrams, and fishbone diagrams.

Education into practice

In designing your next quality improvement project:

What will you do to ensure that you understand the problem you are trying to solve?

How will you involve your colleagues and patients in your project and gain the support of managers and senior staff?

What steps will you take right from the start to ensure that any improvements made are sustained?

How patients were involved in the creation of this article

The authors have drawn on their experience both in partnering with patients in the design and delivery of multiple quality improvement activities and in participating in the Academy of Medical Royal Colleges Training for Better Outcomes Task and Finish Group 1 in which patients were involved at every step. Patients were not directly involved in writing this article.

Sources and selection material

Evidence for this article was based on references drawn from authors’ academic experience in this area, guidance from organisations involved in supporting quality improvement work in practice such as NHS Improvement, The Health Foundation, and the Institute for Healthcare Improvement, and authors’ experience of working to support clinical trainees to undertake quality improvement.

Competing interests: The BMJ has judged that there are no disqualifying financial ties to commercial companies.

The authors declare the following other interests: none.

Further details of The BMJ policy on financial interests is here: https://www.bmj.com/about-bmj/resources-authors/forms-policies-and-checklists/declaration-competing-interests

Contributors: BJ produced the initial outline after discussions with EV and AOB. AO-B produced a first complete draft, which EV reworked and expanded. BJ then edited and finalised the text, which was approved by EV and AO-B. The revisions in the resubmitted version were drafted by BJ and edited and approved by EV and AO-B. BJ is responsible for the overall content as guarantor.

Provenance and peer review: This article is part of a series commissioned by The BMJ based on ideas generated by a joint editorial group with members from the Health Foundation and The BMJ, including a patient/carer. The BMJ retained full editorial control over external peer review, editing, and publication. Open access fees and The BMJ’s quality improvement editor post are funded by the Health Foundation.

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/ .

• ↵ Academy of Medical Royal Colleges (AoMRC). Quality improvement: training for better outcomes. March 2016. http://www.aomrc.org.uk/reports-guidance/quality-improvement-training-better-outcomes/
• Bethune R ,
• Woodhead P ,
• Van Hamel C ,
• Teigland CL ,
• Blasiak RC ,
• Wilson LA ,
• Meyerhoff KL ,
• ↵ Jones B, Woodhead T. Building the foundations for improvement—how five UK trusts built quality improvement capability at scale within their organisations. The Health Foundation. February 2015. https://www.health.org.uk/publication/building-foundations-improvement
• ↵ General Medical Council (GMC). Generic professional capabilities framework. May 2017. https://www.gmc-uk.org/-/media/documents/generic-professional-capabilities-framework-0817_pdf-70417127.pdf
• ↵ NHS improvement (NHSI). Developing people—improving care A national framework for action on improvement and leadership development in NHS-funded services. December 2016. https://improvement.nhs.uk/resources/developing-people-improving-care/
• ↵ The Health Foundation. Involving junior doctors in quality improvement: evidence scan. September 2011. https://www.health.org.uk/publication/involving-junior-doctors-quality-improvement
• ↵ Zarkali A, Acquaah F, Donaghy G, et al. Trainees leading quality improvement. A trainee doctor’s perspective on incorporating quality improvement in postgraduate medical training. Faculty of Medical Leadership and Management. March 2016. https://www.fmlm.ac.uk/sites/default/files/content/resources/attachments/FMLM%20TSG%20Think%20Tank%20Trainees%20leading%20quality%20improvement.pdf
• Hillman T ,
• ↵ Bohmer R. The instrumental value of medical leadership: Engaging doctors in improving services. The King’s Fund. 2012. https://www.kingsfund.org.uk/sites/default/files/instrumental-value-medical-leadership-richard-bohmer-leadership-review2012-paper.pdf
• ↵ Dixon-Woods M, McNicol S, Martin G. Ten challenges in improving quality in healthcare: lessons from the Health Foundation's programme evaluations and relevant literature. BMJ Qual Saf 2012;1e9. doi: 10.1136/bmjqs-2011-000760 OpenUrl Abstract / FREE Full Text
• Brault MA ,
• Linnander EL ,
• Carroll JS ,
• Edmondson AC
• Mannion R ,
• Richter A ,
• McPherson K ,
• Headrick L ,
• ↵ Lucas B, Nacer H. The habits of an improver. Thinking about learning for improvement in health care. The Health Foundation. October 2015. https://www.health.org.uk/sites/health/files/TheHabitsOfAnImprover.pdf
• Greenhalgh T
• ↵ The Health Foundation. Quality Improvement made simple: what everyone should know about quality improvement. The Health Foundation. 2013. https://www.health.org.uk/publication/quality-improvement-made-simple
• ↵ Boaden R, Harvey G, Moxham C, Proudlove N. Quality improvement: theory and practice in healthcare. NHS Institute for Innovation and Improvement. 2008. https://www.england.nhs.uk/improvement-hub/publication/quality-improvement-theory-practice-in-healthcare/
• ↵ Institute for Healthcare Improvement (IHI). IHI resources: How to improve. IHI. 2018 http://www.ihi.org/resources/Pages/HowtoImprove/default.aspx
• ↵ Lean Enterprise Institute. What is lean? Lean Enterprise Institute. 2018. https://www.lean.org/WhatsLean/
• ↵ Bate P, Robert G, Fulop N, Øvretveit J, Dixon-Woods M. Perspectives on context. A selection of essays considering the role of context in successful quality improvement. The Health Foundation. 2014. https://www.health.org.uk/sites/health/files/PerspectivesOnContext_fullversion.pdf
• ↵ Improvement NHS. (NHSI) Quality, Service Improvement and Redesign Tools. Project management an overview. September 2017. https://improvement.nhs.uk/resources/project-management-overview/
• ↵ Clarke J, Davidge M, James L. The how-to guide for measurement for improvement. NHS Institute for Innovation and Improvement 2009. https://www.england.nhs.uk/improvement-hub/wp-content/uploads/sites/44/2017/11/How-to-Guide-for-Measurement-for-Improvement.pdf
• Nelson EC ,
• Splaine ME ,
• Batalden PB ,
• ↵ Improvement NHS. (NHSI) Quality, Service Improvement and Redesign Tools. Run charts. January 2018. https://improvement.nhs.uk/resources/run-charts/
• ↵ Improvement NHS. (NHSI) Quality, Service Improvement and Redesign Tools. Statistical process control tool. May 2018. https://improvement.nhs.uk/resources/statistical-process-control-tool/
• Cornwell J ,
• Purushotham A ,
• Sturmey G ,
• Burgess R ,
• ↵ Royal College of Physicians. Unlocking the potential. Supporting doctors to use national clinical audit to drive improvement. April 2018. https://www.rcplondon.ac.uk/projects/outputs/unlocking-potential-supporting-doctors-use-national-clinical-audit-drive
• ↵ Improvement NHS. (NHSI) Quality, Service Improvement and Redesign Tools: conventional process mapping. January 2018. https://improvement.nhs.uk/resources/process-mapping-conventional-model/
• ↵ Institute for Healthcare Improvement (IHI) 5 Whys: Finding the root cause. IHI tool. 2018. http://www.ihi.org/resources/Pages/Tools/5-Whys-Finding-the-Root-Cause.aspx
• Scottish Social Services Council (SSSC) Appreciative Inquiry Resource Pack
• ↵ Improvement NHS. (NHSI) Quality, Service Improvement and Redesign Tools: Developing your aims statement. January 2018. https://improvement.nhs.uk/resources/aims-statement-development/
• Pannick S ,
• Sevdalis N ,
• Athanasiou T
• ↵ Improvement NHS. (NHSI) Quality, Service Improvement and Redesign Tools: Stakeholder Analysis. January 2018. https://improvement.nhs.uk/documents/2169/stakeholder-analysis.pdf
• ↵ Maher L, Gustafson D, Evans A. Sustainability model and guide. NHS Institute for Innovation and Improvement. February 2010. http://webarchive.nationalarchives.gov.uk/20160805122935/http:/www.nhsiq.nhs.uk/media/2757778/nhs_sustainability_model_-_february_2010_1_.pdf
• Networks NHS
• ↵ Community Q. The Health Foundation. 2018. https://q.health.org.uk/
• ↵ Health Education England. e-learning for healthcare. https://www.e-lfh.org.uk/programmes/research-audit-and-quality-improvement/
• ↵ Scotland Quality Improvement Hub NHS. QI e-learning. http://www.qihub.scot.nhs.uk/education-and-learning-xx/qi-e-learning.aspx
• ↵ Joint Royal Colleges of Physicians Training Board. Quality Improvement Assessment Tool (QIPAT). 2017. https://www.jrcptb.org.uk/documents/may-2012-quality-improvement-assessment-tool-qipat
• ↵ Joint Royal Colleges of Physicians Training Board. Quality improvement assessment tool. May 2017. https://www.jrcptb.org.uk/documents/may-2012-quality-improvement-assessment-tool-qipat
• ↵ Joint Royal Colleges of Physicians Training Board. Multi-source feedback. August 2014. https://www.jrcptb.org.uk/documents/multi-source-feedback-august-2014 .

• Research article
• Open access
• Published: 02 March 2020

Costs and economic evaluations of Quality Improvement Collaboratives in healthcare: a systematic review

• Lenore de la Perrelle   ORCID: orcid.org/0000-0001-9239-0728 1 , 2 ,
• Gorjana Radisic 1 , 2 ,
• Monica Cations 1 , 2 ,
• Billingsley Kaambwa 3 ,
• Gaery Barbery 4 &
• Kate Laver 1 , 2  

BMC Health Services Research volume  20 , Article number:  155 ( 2020 ) Cite this article

23k Accesses

35 Citations

15 Altmetric

Metrics details

In increasingly constrained healthcare budgets worldwide, efforts to improve quality and reduce costs are vital. Quality Improvement Collaboratives (QICs) are often used in healthcare settings to implement proven clinical interventions within local and national programs. The cost of this method of implementation, however, is cited as a barrier to use. This systematic review aims to identify and describe studies reporting on costs and cost-effectiveness of QICs when used to implement clinical guidelines in healthcare.

Multiple databases (CINAHL, MEDLINE, PsycINFO, EMBASE, EconLit and ProQuest) were searched for economic evaluations or cost studies of QICs in healthcare. Studies were included if they reported on economic evaluations or costs of QICs. Two authors independently reviewed citations and full text papers. Key characteristics of eligible studies were extracted, and their quality assessed against the Consolidated Health Economic Evaluation Reporting Standards (CHEERS). Evers CHEC-List was used for full economic evaluations. Cost-effectiveness findings were interpreted through the Johanna Briggs Institute ‘three by three dominance matrix tool’ to guide conclusions. Currencies were converted to United States dollars for 2018 using OECD and World Bank databases.

Few studies reported on costs or economic evaluations of QICs despite their use in healthcare. Eight studies across multiple healthcare settings in acute and long-term care, community addiction treatment and chronic disease management were included. Five were considered good quality and favoured the establishment of QICs as cost-effective implementation methods. The cost savings to the healthcare setting identified in these studies outweighed the cost of the collaborative itself.

Conclusions

Potential cost savings to the health care system in both acute and chronic conditions may be possible by applying QICs at scale. However, variations in effectiveness, costs and elements of the method within studies, indicated that caution is needed. Consistent identification of costs and description of the elements applied in QICs would better inform decisions for their use and may reduce perceived barriers. Lack of studies with negative findings may have been due to publication bias. Future research should include economic evaluations with societal perspectives of costs and savings and the cost-effectiveness of elements of QICs.

Trial registration

PROSPERO registration number: CRD42018107417 .

Peer Review reports

A significant challenge facing health care settings is how to implement proven clinical interventions in practice in a cost-effective manner [ 1 ]. Scarce resources, including lack of time and staff are often cited as barriers to implementation [ 2 , 3 ]. A recent review of medical research shows health savings from broad research translation, significantly outweigh the cost of delivering them [ 3 ] but the field of economic evaluation of implementation strategies is still developing [ 4 ]. Decisions to use particular implementation methods can be better informed by identifying cost-benefits of methods in addition to health outcomes [ 5 , 6 ].

Methods of knowledge translation have been tested with mixed results. For example, clinical practice guidelines aim to translate research into practice and improve the quality of care and health outcomes for people. However, studies have shown that the dissemination of guidelines alone is insufficient to effect change in routine clinical practice [ 7 ]. Education and training of clinicians, the development of champions of change in organisations, and audit and feedback mechanisms have been trialled to improve adherence to guidelines [ 8 ]. However, these strategies lead to only modest effects in quality improvement [ 8 ]. A recent review found that while multifaceted strategies are more effective, costs associated with components were difficult to discern and cost-effectiveness was not explicitly evaluated [ 9 ]. Knowledge translation approaches which are tailored to an organisation can be successful but may lack transferability to other settings [ 10 , 11 , 12 ]. QICs have been adapted from manufacturing industry [ 13 ] for use across multiple settings by the US Institute for Healthcare Improvement (IHI) [ 14 ]. A QIC is a multifaceted approach to implementation of evidence-based practices, clinical guidelines or improved methods for quality and safety. Typically, they draw participants from multiple healthcare organisations to learn, apply and share improvement methods over a year or more. Teams are supported by experts who coach participants to test strategies adapted to their own setting. By collaborating, participants learn more effectively, spread improvement ideas and benchmark their progress against other organisations [ 14 , 15 ]. Common components of QICs include face to face training sessions focussing on healthcare improvement and quality improvement methods, telephone meetings, feedback and the use of process improvement methods [ 13 ]. QICs have been used in healthcare systems in several countries to improve implementation outcomes [ 15 , 16 , 17 ]. They are adaptable within complex healthcare systems and offer a way to scale up implementation across many different organisations. However, inconsistent results, multiple elements and perceived cost of establishing, conducting and sustaining a collaborative are barriers to their use [ 17 , 18 , 19 ]. Wells and colleagues recently identified 64 QICs reporting effectiveness measures that met their inclusion criteria [ 15 ]. They found that 73% of these collaboratives reported significant results in diverse settings such as hospitals, health clinics and nursing homes. Improvement was associated with targeted clinical practice related to infection control, management of chronic conditions or prevention of falls, wounds or pain management [ 15 ]. While these improvements were associated with cost savings, only four studies reported on cost-effectiveness outcomes [ 15 ]. They identified gaps in design, reporting and assessment of costs which limited the information on cost-effectiveness. The costs of establishing a QIC can be significant, including personnel to recruit and coordinate activities, development of materials and education, the time spent by all participants involved in the collaborative and expenses associated with face to face meetings [ 17 ].

With increasing pressure on the healthcare system to deliver evidence-based practice with scarce resources, there is a need to evaluate the cost-effectiveness of healthcare improvement and knowledge translation strategies. Economic evaluation can assess implementation strategies to guide decisions about the choice of strategy providing value for money.

The aim of this systematic review was to identify and describe studies that report on the costs and cost-effectiveness of QICs to inform strategies to implement clinical guideline recommendations in healthcare.

The protocol for this systematic review was developed in advance and was registered with PROSPERO on 7 September 2018; registration number CRD42018107417.

Eligibility criteria

Studies were included in this review if they reported on initiatives that comprised healthcare clinicians across teams, professions, or organisations involved in a QIC or a quality improvement team with the aim of improving practice over time. Quality improvement teams were included if they included the most common components of QICs as identified by Nadeem et al. [ 13 ]. Studies were included if the collaboratives used multi-modal interventions, such as training, developing implementation plans, trying out a practice improvement, seeking advice from experts and people with lived experience and reviewing plans over time to improve practice [ 15 ]. We included quantitative studies that used full economic evaluation (i.e. cost-effectiveness, cost-utility analysis, cost-benefit analysis, cost-consequences analysis); partial economic evaluations (i.e. cost analyses, cost descriptions, cost outcome descriptions, cost minimisation studies); and randomised trials reporting estimates of resource use or costs associated with implementation or improvement. We excluded systematic reviews, study protocols, conference proceedings, editorials and commentary papers, effectiveness analyses with no analysis of costs, burden of disease studies, and cost of illness studies. The primary outcome of interest was the cost-effectiveness or cost-benefit of the use of elements of QICs to implement improvement in healthcare or adherence to clinical guidelines. A secondary outcome was costs associated with QICs.

Search strategy and study selection

Five electronic databases were searched on 19 November 2018 (CINAHL, Medline, PsycINFO, EconLit, ProQuest (Health and Medicine: Social Sciences subsets only)). Embase was searched on 20 August 2019. Websites of large organisations interested in healthcare improvement such as the Institute of Healthcare Improvement (IHI, USA) and government bodies such as National Health and Medical Research Council (Australia), National Health Services and the National Institute for Health and Care Excellence (UK) and the European Network of Health Economic Evaluation Databases were searched for grey literature. Reference lists of included studies were scanned for potentially eligible studies. Studies were limited to English language, but no time limits were imposed on the search strategies. Research librarians with expertise in systematic reviews assisted with the development of the search strategies. The search strategy was developed for MEDLINE using medical subject search headings (MeSH) and text words and then adapted for use with the other databases. The strategy combined terms relating to quality improvement, collaborative, guidelines implementation and cost, cost-benefit or economic analysis. The search strategy for MEDLINE is attached (Additional file  1 ). Results are reported per the Preferred Reporting Items of Systematic Reviews and Meta-Analyses (PRISMA) guidelines [ 20 ].

Two authors (LdlP and GR) independently screened titles and abstracts based on the inclusion criteria detailed in the review protocol. Full texts of studies identified by abstract and title screen as having met the inclusion criteria were obtained and reviewed independently (LdlP and GR). Differences between reviewer’s results were resolved by discussion and when necessary in consultation with a third review author (MC) .

Data extraction

One author (LdlP) extracted data using a modified version of the Joanna Briggs Institute (JBI) Data Extraction form for Economic Evaluations [ 21 ]. Another author (GR) checked the extraction for accuracy. Data was extracted about the study method, evaluation design, participants, intervention used, comparator, outcomes, prices and currency used for costing, time period of analysis, setting, tools used to measure outcomes and authors conclusions. This information was presented descriptively and summarised in Table 1 (Additional file  2 ) . Both costs of care resulting from improved care and costs of establishing QICs were identified. Cost components were standardised by converting currency and year to US dollars for 2018 through the Eurostat-OECD data base and manual on purchasing power parities for Euros and The World Bank GDP deflator data base for United States dollar values [ 22 , 23 ].

Risk of bias assessment

Two checklists were used to critically appraise the studies due to the difference in design of studies included. The 24 item Health Economic Evaluation Reporting Standards (CHEERS) checklist was used to determine methodological quality of all the included studies as it applies to any form of economic evaluation [ 24 ]. This is presented in Table 2a (Additional file  3 ). The Evers CHEC-List [ 25 ] was also used to assess the full economic evaluations and is included as Table 2b (Additional file  4 ) [ 26 ]. A score of one point was assigned to each positive response, zero to a negative response or for items that did not apply. A summary score is calculated at the bottom of each table with a maximum score of 24 and 19 respectively. This scoring provides an indication of total items present for each study.

Assessment of generalizability

The currency and year of studies was converted to US dollars for 2018 using the Eurostat-OECD purchasing power parities data base for Euros and the World Bank deflator data base for US dollar updates. This provided an option to compare results but due to the varied type of studies and focus on the implementation method rather than the healthcare intervention, a full transferability assessment was not conducted.

Data synthesis

Included studies were subjected to data extraction by the author (LdlP) and information was synthesised to interpret the findings of full and partial economic evaluations and cost analysis studies. The Johanna Briggs Institute (JBI) ‘three by three dominance ranking matrix tool’ was used to interpret findings [ 27 ] and was checked by another author (GR) for consistency. Any inconsistencies were resolved by discussion and by consultation with a third review author (BK). This tool assists in drawing conclusions about the results of studies in terms of both cost and effectiveness (health benefits). It classifies results as favoured, unclear or rejected in favour of the comparator. An intervention was favoured if relative to its comparator it either (i) was cheaper but more effective, (ii) was cheaper but just as effective or (iii) cost the same but was more effective. An intervention was rejected if, relative to its comparator, it either (i) was more expensive and less effective, (ii) was more expensive and just as effective or (iii) cost the same but was less effective. A judgement would have to be made about all other scenarios based on other criteria [ 27 ]. For instance, an intervention would be favoured if it was more expensive and more effective than a comparator provided the associated incremental cost-effectiveness ratio (ICER) was below the threshold used for assessing cost-effectiveness e.g. €80,000 per quality adjusted life years (QALY) in the Netherlands [ 28 ].

Study selection

The search identified 8505 citations and after removing duplicates, 3481 titles and abstracts were reviewed. Twenty-two full text reviews revealed eight papers that met the inclusion criteria. PRISMA flowchart at Fig.  1 describes the process of selection [ 29 ].

PRISMA flowchart describing the process of study selection

Overview of studies

Table 1 (Additional file 2 ) presents the overview of characteristics of the studies included in this review. Most studies describe the costs of establishing a collaborative to improve quality in healthcare and compared costs to outcomes. Five of the included studies involved full economic analyses using cost-effectiveness analysis (CEA) or cost utility analysis (CUA) [ 30 , 31 , 32 , 33 , 34 ], whereas three studies were cost analyses [ 35 , 36 , 37 ]. All studies were set in multi-centre healthcare settings, hospitals, long term care or community clinics, and related to diverse health conditions such as Parkinson’s disease, diabetes, obstetrics, neonatal intensive care, hip fractures, pressure ulcers, cardiac care or addiction treatment. All included clinicians working either nationally or across multiple states.

Methodological quality

Table 2a (Additional file 3 ) summarises the methodological quality of the studies included in this review.

Cost effectiveness study conducted by Broughton et al. [ 30 ], and cost utility studies by Schouten et al. [ 33 ], Makai et al. [ 32 ] and Huang et al. [ 34 ] were considered high quality, complying with most of the items on CHEERS checklist [ 24 ]. Item 12 related to valuation of preferences for outcomes was not addressed in these studies [ 24 ]. A cost analysis by Bloem et al. [ 35 ] and a cost effectiveness study by Gustafson and colleagues [ 31 ] were of moderate quality. They did not address item 13, related to estimating costs via a model-based evaluation, items 15 and 16, the choice of model or assumptions or item 20, how uncertainty was addressed. The cost analysis by Rogowski et al. [ 36 ] was rated low quality on CHEERS checklist and the cost study by Dranove et al. [ 37 ] study was considered lowest quality as less than half of all items were addressed. Using the Evers CHEC-List [ 25 ], the full economic evaluations [ 30 , 31 , 32 , 33 , 34 ] were rated good quality.

Conflicts of interest and uncertainties in data were addressed by five studies [ 30 , 32 , 33 , 34 , 35 ]. An incremental cost-effectiveness ratio (ICER) was not applicable for the cost analyses [ 35 , 36 , 37 ] and future costs were not directly considered for those studies.

Table 3 (Additional file  5 ) provides a three by three dominance ranking matrix (JBI DRM) tool to assist in interpreting the cost-effectiveness results of the studies included [ 27 ]. In this review, five studies were classified as favoured interventions (strong dominance) [ 30 , 31 , 33 , 35 , 36 ], two as unclear [ 32 , 34 ] and one rejected [ 37 ]. Bloem et al. [ 35 ], Broughton et al. [ 30 ] and Schouten et al. [ 33 ] all showed reduced costs and improved health outcomes and are most favoured interventions. The studies by Gustafson et al. [ 31 ] and Rogowski et al. [ 36 ] show reduced costs for equally effective processes which are next favoured interventions. The study by Makai et al. [ 32 ] reported increased costs and reduced pressure ulcers while Huang et al. [ 34 ] reported that the improvements in Diabetes care were not cost-effective. These results are uncertain because while the interventions were more expensive but also cost effective, most scenarios analysed yielded ICERs that were above the traditionally accepted thresholds of €80,000/QALY [ 32 ] and US$100,000/QALY [ 34 ]. They therefore need to be assessed against specific priorities for health improvements and expenditure. In a cost analysis, Dranove et al. [ 37 ] were unable to identify cost savings or health improvements as a result of quality improvement expenditure and the comparator is favoured in this case.

Effectiveness and cost-effectiveness

Clinical effectiveness.

Five studies [ 30 , 32 , 33 , 34 , 35 ] reported positive clinical outcomes as a result of using a QIC approach. In studies involving people with chronic health conditions, quality improvements led to reduced mortality risk and reduction in associated health events [ 33 , 35 ]. For example, adherence to guidelines for Parkinson’s disease care achieved via the collaboratives produced improved outcomes, such as reduction in hip fractures, fewer hospital admissions, lower mortality risk and fewer disease related complications [ 35 ]. Quality improvement in diabetes care [ 33 , 34 ] resulted in reduced scores for diabetes risk for cardiovascular disease events and mortality, reduced lifetime incidence of complications and improved life expectancy for both men and women. In both acute and critical care, the improvements led to reduced associated illness but differed in relation to the effect on mortality risk [ 30 , 36 ]. In obstetric care, establishment of a QIC resulted in reduced post-partum haemorrhage, reduced mortality and increased numbers of births in clinics [ 30 ]. In neonatal intensive care, a QIC achieved reductions in infections in critically ill pre-term babies and reduced surgical interventions but no significant difference in mortality was found [ 36 ]. Residents in long term care had reduced incidence of pressure ulcers and slightly improved quality of life as a result of a QIC [ 32 ].

Gustafson and colleagues tested the effectiveness of four different elements of a QIC in the context of addiction treatment clinics [ 31 ]. This study compared clinic level coaching, group telephone calls to clinicians, face to face learning sessions and a combination of these elements to see which methods were more effective. This study did not collect patient outcomes but focussed on three primary process outcomes: waiting time, retention of patients and annual numbers of new patients. These process outcomes were chosen, as the link between treatment programs and patient outcomes was considered weak [ 31 ]. Significant improvements in waiting time and number of new patients were identified for two of the interventions: coaching and the combination of all three elements. A combination of all elements was found to be more costly than coaching alone although it was similarly effective [ 31 ]. Dranove and colleagues found no direct links between the clinical outcomes for patients of hospitals studied and the amount they spent on general quality improvement activities [ 37 ].

Cost-effectiveness and cost savings

Five studies [ 30 , 31 , 33 , 35 , 36 ] reported favourable cost findings from the use of QICs. These were related to savings in the health care system and did not consider broader costs and benefits such as lost productivity, non-medical patient costs and carer time. These studies are considered here in relation to cost effectiveness and cost savings achieved for the use of QICs across a range of health conditions and countries. Values provided below are conversions to US$ for 2018 [ 22 , 23 ] where the price year was provided.

• Cost-effectiveness

Within the context of diabetes care in the Netherlands [ 33 ], the QIC was found to be cost-effective. For the large populations of people who live with diabetes there are significant medical costs related to medicines and cardio-vascular disease [ 33 , 34 ]. The incremental costs per quality adjusted life year (QALY) of US$1550–1714 compared favourably with other published studies on diabetes [ 33 ]. With a cost of about US$19 per patient for the QIC over 2 years, the cost-effectiveness was reported to be significant. In the US, a diabetes care improvement in public health clinics [ 34 ] found lower incidence of complications but the cost of individual improvements in care varied and all interventions but the use of an Angiotensin-converting enzyme (ACE) inhibitor, were not cost-effective [ 34 ].

The cost effectiveness study examining obstetric and newborn care in Niger [ 30 ] found the cost per normal delivery reduced, with a similar decrease in both numbers and costs of deliveries with post-partum haemorrhage [ 30 ]. The cost of the QIC was calculated to be US$2.84 per delivery. The incremental cost-effectiveness was US$335 per disability-adjusted life year (DALY) averted and the study concluded that if other obstetric clinics used the collaborative approach, substantive cost savings could be achieved [ 30 ].

In long term care [ 32 ], reduction in incidence of non-severe pressure ulcers using a QIC approach increased costs of care in the short term. Cost-effectiveness in the longer term was unclear due to small effects on quality of life in nursing home populations near the end of life, and the difficulty in sustaining trained staff to continue to prevent pressure ulcers. As a preventable condition however, quality improvement in the prevention and care of pressure ulcers for a vulnerable population was a worthy goal [ 32 ].

A comparison of four different approaches to implementing QICs (in the context of addiction treatment) identified cost-effective elements [ 31 ]. This study found that while both coaching and a combination of interventions were equally effective in reducing waiting times and increasing numbers of new patients there were significant differences in costs of the interventions. They found the estimated cost per clinic for a coaching intervention was US$2878 (no year) compared to US$7930 (no year) for the combination of interventions. They concluded that the coaching intervention was substantially more cost-effective [ 31 ].

Cost analyses

A cost analysis of ParkinsonNet [ 35 ] showed annual cost savings of US$449 per patient by avoiding or delaying complications or high cost treatments of Parkinson’s disease. The cost per patient per annum was around US$30. However, based on a population of 40,000 people with Parkinson’s disease in The Netherlands, they predicted a national cost saving of over US$17.4 million per annum as a result of the quality improvement [ 35 ].

In the costly area of neonatal intensive care, a cost analysis study [ 36 ] reported significant cost savings per infant were achieved. While costs varied, the average savings per hospital in the post intervention year was US$2.3 million for an average cost of $68,206 per hospital in resources to undertake the QIC [ 36 ].

Finally, the study of costs to improve quality of care in hospitals in United States [ 37 ], found a wide variety in expenditures on quality improvement activities which were not correlated with condition specific costs. Differences in costs were not statistically significant. They presumed that a lack of consensus about the purpose of quality improvement efforts at the time, led to this variation in costs and disconnection with outcomes [ 37 ].

Costs of care

The costs of clinical treatment were measured in most studies and included clinic visits or treatment provided in hospital such as ventilation, surgery and medications, complications or infections [ 30 , 32 , 33 , 34 , 35 , 36 ]. Costs were extracted from hospital bills, medical claims and records maintained by clinicians. Some studies used estimations of costs to form their data, or surveyed managers to identify costs from budgets [ 30 , 33 ]; one used weekly diaries of activities and applied hourly costs for personnel time [ 36 ]. Costs of care were not reported in two studies [ 31 , 37 ].

Costs of establishing QICs

The most common costs identified were: program management costs for the QIC coordinators, time of the participating clinicians in face to face meetings, education sessions, collecting data, travel costs, conference calls, data analysis costs, overhead costs and some capital costs. The cost of developing evidence-based guidelines was included in the ParkinsonNet study to give a complete cost of start-up of the network [ 35 ]. Four studies provided a cost per patient of establishment of the QIC. These included US$3.67 per infant delivery [ 30 ], US$30 per person with Parkinson’s disease [ 35 ], US$19 per person with diabetes in Europe [ 33 ] and US$130 per patient with diabetes in USA [ 34 ]. Dranove et al. reported a wide variation in costs of quality improvement activities between hospitals with the highest costs attributed to meetings [ 37 ]. All reported costs are presented in Table 4 (Additional file  6 ).

There is a need for larger scale and more rapid translation of evidence-based interventions into practice [ 34 ]. However, the cost associated with research translation is an important consideration for constrained health care budgets. QICs have been used widely in diverse healthcare settings and have been effective in improving outcomes for patients [ 38 ] although the costs of the collaboratives may be a barrier to their use [ 35 ]. This review sought to identify and describe studies that report on the costs and cost-effectiveness of QICs in healthcare settings. Although a recent systematic review of QICs identified 64 studies on effectiveness, only four reported on cost-effectiveness [ 15 ]. We identified eight studies that reported on costs or cost-effectiveness of QICs. This included the four studies identified in the review by Wells et al. so updated that aspect of the review [ 15 ]. Our results confirm that the consideration of costs of QICs has not been reported in many studies. This may be because of the difficulty in defining costs associated with QICs over time and in different contexts [ 38 , 39 ]. It may be that costs are small in comparison to operating costs or funded separately to the health system and of less importance for research [ 40 ].

Five of the eight studies in this review showed that QICs were cost-effective in implementing clinical guidelines [ 30 , 31 , 33 , 35 , 36 ]. They identified cost savings and improvement in health outcomes for patients in both acute care and chronic condition management. The costs associated with the QIC appeared low in relation to savings across large populations or for reducing the need for high cost treatments [ 36 , 41 ]. These studies calculated the cost of the QIC per patient for the duration of the intervention which provided useful data compared to overall outcomes and savings achieved. Where smaller populations are treated with high cost interventions, the cost per patient for the QICs would be expected to be higher.

These studies were conducted in different countries or across states, with different infrastructure costs and resources. It would be difficult to generalize the costs of the QICs across such different countries and conditions. However, they used a similar process to engage clinicians and modify practices locally. This indicated that the QIC methodology was adapted to different conditions with similar set up structures needed. An investment in QICs was needed and the costs per person could be best spread across large populations of people with a condition or where high cost treatments can be reduced [ 38 ].

One study evaluated which element of the QIC intervention was more cost-effective [ 23 ]. This demonstrated that differences that can be achieved in both effectiveness and cost by the choice of how education or support was provided to clinicians. Only one study found no correlation between health outcomes and the costs of quality improvement activities in hospitals [ 26 ].

Although most of the studies captured only medical costs, most considered that societal effects of health improvements may increase the cost-effectiveness due to improved quality of life (QoL). For treatment of chronic conditions, improved care is likely to result in long term cost savings, however QoL in long term care populations was more difficult to measure [ 32 ]. Schouten et al. [ 22 ] found that a wide range of disease risk control was achieved in diabetes treatment. They suggested that outcomes of other chronic conditions may be improved through a QIC approach and the societal effects may also be higher when considering better quality of life outcomes. Bloem et al. [ 23 ] similarly identified the potential for improvement of cost-effectiveness of healthcare for other chronic disorders. They also reported the need to structure funding sources and medical insurance related to improvements in health outcomes.

Rogowski et al. [ 24 ] identified the potential for higher cost savings for expensive health interventions and at least short-term sustainability of QICs. Widespread adoption of the interventions may increase costs of interventions but Rogowski et al. considered that expected savings and benefits would offset these [ 24 ]. The potential for higher cost savings and effectiveness through a wider use or broader scale of QICs is a pertinent aspect of these studies for healthcare budgets.

The establishment of collaboratives was shown to require considerable investment in the initial phases of the improvements, which then decreased over time of the collaborative process. QICs were funded in most studies by national agencies with specialist healthcare improvement staff involved in developing the collaborative, engaging participants and providing education, guidance and support for the duration. Only one study identified the relative cost-effectiveness of different combinations of elements of a QIC [ 31 ]. This suggests an opportunity to improve cost-effectiveness of QICs by selecting key elements for uses.

Despite increasing acknowledgement of the importance of patient and public involvement, there was no involvement of members of the public or patients reported in these studies. Costs were spread across state and national healthcare systems to scale up improvements for low per clinic or patient cost. One study included the external cost of developing guidelines in the assessment of cost-effectiveness [ 35 ] which provided an additional insight into the costs of developing or adapting international guidelines to national conditions. In most cases the clinical guidelines were developed separately to implementation in healthcare services and funded separately. Despite this inclusion of the cost of developing guidelines, the use of the QIC was shown to be cost-effective [ 35 ].

The identified costs of the QIC had similar elements across the five studies showing cost-effectiveness [ 30 , 31 , 33 , 35 , 36 ]. Costs were highest for the initial development of collaboratives, face to face meetings and travel for participants, and for multi-factored interventions. While most studies used similar components of QICs as described by Nadeem et al. [ 13 ] and IHI [ 14 ], only one study compared the costs of different elements of the QIC [ 31 ]. There is an opportunity to consider which elements of QICs contribute to cost effectiveness and in which setting they may be useful. One study included the cost of development of guidelines and a maintenance cost for an ongoing collaborative [ 35 ]. This provides a wider consideration of all set up costs for quality improvement and the costs to maintain the collaborative beyond a research study. The local infrastructure costs varied widely in four studies [ 31 , 34 , 36 , 37 ] which made the cost assessments difficult to compare within and between studies. Inclusions and exclusions of costs varied between studies which also made comparisons between studies difficult. It would be of use to identify common costs to consider when budgeting for QICs and to allow for local differences in infrastructure.

The value of these studies shows that savings can be made to healthcare for quality improvements, the real set up costs and how to assess benefit. Caution in interpreting results is needed as the studies varied in what was included and costed and the perspective from which assessment of cost effectiveness was judged. Similarly, few studies of cost effectiveness of QICs were identified suggesting that studies with negative results may not have been published.

A strength of this review is the rigorous and systematic method used to identify studies and synthesise data. A comprehensive search strategy was developed and used in a range of databases. Our search of the grey literature was an important step given the variety of ways in which healthcare improvements are reported. The use of both the CHEERS checklist [ 24 ] and Evers CHEC-List [ 25 ] to assess the mixed designs found most studies to be of good to medium quality. The main limitations of the review are that only studies published in English were considered and we did not search trial registers. The few papers identified may reflect a publication bias or may indicate economic evaluations of QICs have not been conducted.

Few cost analyses or cost-effectiveness studies have been identified to assess the costs and benefits of QICs to translate research and knowledge into practice. Most that are included in this review show cost savings or improvement in healthcare process and patient outcomes across acute, long term care and chronic conditions. Judgement is required in relation to the priority given to healthcare improvement from a societal perspective compared to the cost of QICs. The potential to scale up knowledge translation through QICs and to improve cost-effectiveness based on these studies is suggested. The costs of QICs need to be factored into translation of improvements, and their costs or cost-effectiveness evaluated to identify savings to healthcare budgets and benefits to society. A detailed break-down of costs of QICs may assist in identifying elements of greatest cost and alternatives that may be effective for cost savings to the quality improvement process.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

European Euros

Angiotensin-converting enzyme inhibitor

Cognitive Decline Partnership Centre

Cost Effectiveness Analysis

Cost Effectiveness Ratio

Consolidated Health Economic Evaluation Reporting Standards statement

Cost Utility Analysis

Disability Adjusted Life Year

Dominance Ranking Matrix

European Quality Group 5 dimensions 3 levels measure of quality of life

Consensus on Health Economic Criteria checklist by Silvia Evers et al.

Incremental Cost Effectiveness Ratio

Institute for Healthcare Improvement

Johanna Briggs Institute

Medical subject search headings

National Health and Medical Research Council, Australia

Preferred Reporting Items for a Systematic Review and Meta-Analysis of Diagnostic Test Accuracy studies

Quality Adjusted Life Year

Quality Improvement

Quality Improvement Collaborative/s

Quality of Life

United Kingdom of Great Britain; US/USA: United States/United States of America

Greenhalgh T, Howick J, Maskrey N. Evidence based medicine a movement in crisis? BMJ. 2014;348(7963):7.

Google Scholar  

Brown V, Fuller J, Ford D, Dunbar J. The enablers and barriers for the uptake, use and spread of primary health care Collaboratives in Australia. Herston QLD: APHCRI Centre of research Excellence in Primary Health Care Microsystems; 2014. p. 2014.

KPMG. Economic Impact of Medical Research in Australia. Melbourne: KPMG; 2018. p. 2018.

Roberts, SLE, Healey A, Sevdalis N. Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review. Implement Sci. 2019;14(1):72.

Dalziel K, Segal L, Mortimer D. Review of Australian health economic evaluation - 245 interventions: what can we say about cost effectiveness? Cost Eff Resour Alloc. 2008;6(1):9.

Article   Google Scholar  

Hoomans T, Severens J. Economic evaluation of implementation strategies in health care. Implement Sci. 2014;9(1):168.

Grol R. Successes and Failures in the Implementation of Evidence-Based Guidelines for Clinical Practice. Med Care. 2001;39(8 Suppl 2):II46–54.

CAS   PubMed   Google Scholar  

Grimshaw JM, Schünemann HJ, Burgers J, Cruz AA, Heffner J, Metersky M, et al. Disseminating and implementing guidelines: article 13 in integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9(5):298–303.

Chan Wiley V, Pearson Thomas A, Bennett Glen C, Cushman William C, Gaziano Thomas A, Gorman Paul N, et al. ACC/AHA special report: clinical practice guideline implementation strategies: a summary of systematic reviews by the NHLBI implementation science work group: a report of the American College of Cardiology/American Heart Association task force on clinical practice guidelines. J Am Coll Cardiol. 2017;69(8):1076.

Article   CAS   Google Scholar  

Greenhalgh T. The research traditions. In: Greenhalgh T, Robert G, Bate P, Macfarlane F, Kyriakidou O, (Editors). Diffusion of innovations in health service organisations:a systematic literature review. Malden: Blackwell; 2005.p 48–82.

Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients' care. Lancet. 2003;362(9391):1225–30.

Glasgow R, Vinson C, Chambers D, Khoury M, Kaplan R, Hunter C. National Institutes of Health approaches to dissemination and implementation science: current and future directions. Am J Public Health. 2012;102(7):1274–81.

Nadeem E, Olin SS, Hill LC, Hoagwood KE, Horwitz SM. Understanding the components of quality improvement Collaboratives: a systematic literature review. Milbank Q. 2013;91(2):354–94.

Institute for Healthcare Improvement. The Breakthrough Series: IHI's Collaborative Model for Achieving Breakthrough Improvement. Boston: Institure for Healthcare Improvement; 2003. p. 2003.

Wells S, Tamir O, Gray J, Naidoo D, Bekhit M, Goldmann D. Are quality improvement collaboratives effective? A systematic review. BMJ Qual Safety. 2018;27(3):226.

Ovretveit J, Gustafson D. Evaluation of quality improvement programmes. (Quality Improvement Research). Qual Safety Health Care. 2002;11(3):270.

Schouten L, Hulscher M, van Everdingen J, Huijsman R, Grol R. Evidence for the impact of quality improvement collaboratives: systematic review. Br Med J. 2008;336(7659):1491.

Chin MH. Quality improvement implementation and disparities: the case of the health disparities collaboratives. Med Care. 2010;48(8):668–75.

Ovretveit J, Bate P, Cleary P, Cretin S, Gustafson D, McInnes K, et al. Quality collaboratives: lessons from research. Qual Safety Health Care. 2002;11(4):345–51.

McInnes MDF, Moher D, Thombs BD, McGrath TA, Bossuyt PM and thePRISMA-DTA Group. Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy studies: The PRISMA-DTA Statement, PRISMA reporting guideline for diagnostic test accuracy Studies. JAMA. 2018;319(4):388–96.

Aromataris E, Munn Z, (Editors) Joanna Briggs Institute Reviewer's Manual. Adelaide, SA: The Johanna Briggs Institute; 2017.

OECD, Eurostat. Eurostat-OECD Methodological Manual on Purchasing Power Parities (2012 Edition) 2012.

The World Bank. GDP Deflator (base year varies by country-United States) [Data Base]. 2019 [Data base of deflator values for US Dollars by year]. Available from: https://data.worldbank.org/indicator/NY.GDP.DEFL.ZS?end=2018&locations=US&most_recent_year_desc=false&start=2000 .

Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. BMJ (Clinical research ed). 2013;346(mar25 1):f1049.

Evers S, Goossens M, de Vet H, van Tulder M, Ament A. Criteria list for assessment of methodological quality of economic evaluations: consensus on health economic criteria. J of Inter Tech of Health Care. 2005;21(2):240–5.

Higgins J, Green S, (Editors). Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 ed. Chichester: Wiley; 2011.

Gomersall SJ, Jadotte TY, Xue TY, Lockwood TS, Riddle TD, Preda TA. Conducting systematic reviews of economic evaluations. Int J Evidence-Based Healthcare. 2015;13(3):170–8.

College voor Zorgverzekeringen (CVZ). Guidelines for pharmacoeconomic research in the Netherlands, updated version. 2006 ed. Dieman: College voor Zorgverzekeringen; 2006.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–9.

Broughton E, Saley Z, Boucar M, Alagane D, Hill K, Marafa A, et al. Cost-effectiveness of a quality improvement collaborative for obstetric and newborn care in Niger. Int J Health Care Qual Assurance (09526862). 2013;26(3):250–61.

Gustafson DH, Quanbeck AR, Robinson JM, Ford JH 2nd, Pulvermacher A, French MT, et al. Which elements of improvement collaboratives are most effective? A cluster-randomized trial. Addiction (Abingdon, England). 2013;108(6):1145–57.

Makai P, Koopmanschap M, Bal R, Nieboer AP. Cost-effectiveness of a pressure ulcer quality collaborative. Cost Eff Resour Alloc. 2010;8:11.

Schouten LM, Niessen LW, van de Pas JW, Grol RP, Hulscher ME. Cost-effectiveness of a quality improvement collaborative focusing on patients with diabetes. Med Care. 2010;48(10):884–91.

Huang ES, Zhang Q, Brown SES, Drum ML, Meltzer DO, Chin MH. The Cost-Effectiveness of Improving Diabetes Care in U.S. Federally Qualified Community Health Centers. Health Serv Res. 2007;42(6p1):2174–93.

Bloem BR, Rompen L, de Vries NM, Klink A, Munneke M, Jeurissen P. ParkinsonNet: a low-cost health care innovation with a systems approach from the Netherlands. Health Aff. 2017;36(11):1987–96.

Rogowski JA, Horbar JD, Plsek PE, Baker LS, Deterding J, Edwards WH, et al. Economic implications of neonatal intensive care unit collaborative quality improvement. Pediatrics. 2001;107(1):23–9.

Dranove D, Reynolds KS, Gillies RR, Shortell SS, Rademaker AW, Huang CF. The cost of efforts to improve quality. Med Care. 1999;37(10):1084–7.

Franx G. Quality improvement in mental healthcare: the transfer of knowledge into practice, vol. 2012. Utrecht: The Trimbos Instituut; 2012.

Ovretveit J. Does improving quality save money? A review of evidence of which improvements to quality reduce costs to health service providers. London: The Health Foundation; 2009.

Chen LM, Rein MS, Bates DW. Costs of quality improvement: a survey of four acute care hospitals. Jt Comm J Qual Patient Saf. 2009;35(11):544–50.

PubMed   Google Scholar  

Sathe NA, Nocon RS, Hughes B, Peek ME, Chin MH, Huang ES. The costs of participating in a diabetes quality improvement collaborative: variation among five clinics. Jt Comm J Qual Patient Saf. 2016;42(1):18–24.

PubMed   PubMed Central   Google Scholar  

Download references

Acknowledgements

Librarians Nikki Lee and Shannon Brown developed search strategies for six data bases.

This study was provided by the National Health and Medical Research Council (NHMRC) Partnership Centre on Dealing with Cognitive and Related Functional Decline in Older People (CDPC) (grant no. GNT 9100000) and a NHMRC Boosting Dementia Research Grant (APP1135667). KL is supported by a National Health and Medical Research Council Dementia Research Development Fellowship.

Author information

Authors and affiliations.

Department of Rehabilitation, Aged and Extended Care, Flinders University, Bedford Park SA, GPO Box 2100, Adelaide, 5001, South Australia

Lenore de la Perrelle, Gorjana Radisic, Monica Cations & Kate Laver

Cognitive Decline Partnership Centre, the University of Sydney, Hornsby Ku-Ring-Gai Hospital, Hornsby, NSW, Australia

Health Economics, College of Medicine and Public Health, Flinders University, Bedford Park, SA, Australia

Billingsley Kaambwa

Health Services Management, School of Medicine, Griffith University, Southbank, Qld, Australia

Gaery Barbery

You can also search for this author in PubMed   Google Scholar

Contributions

KL conceptualised the review, obtained research funding, reviewed and edited the drafts and final manuscript. LdlP developed the PROSPERO registration, search strategies with assistance of librarians, screened titles, abstracts and full articles, extracted and synthesised data, drafted, reviewed and edited final manuscript. GR screened titles, abstracts and full articles, checked extraction and synthesis of data and reviewed the draft. BK reviewed and edited the drafts, checked extraction and synthesis of data and provided expert advice. MC and GB reviewed and edited the drafts and provided advice on methods and style. All authors read and approved final manuscript.

Corresponding author

Correspondence to Lenore de la Perrelle .

Ethics declarations

Ethics approval and consent to participate.

Not applicable

Consent for publication

Competing interests.

MC has been employed in the last 5 years to assist with data collection for Alzheimer’s disease drug trials funded by Janssen and Merck. LdlP, GR, BK, GB and KL declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1..

Medline Search Strategy using medical subject search headings (MeSH) and text words to search for studies and adapted to search other data bases.

Additional file 2.

Table 1 Overview of studies data extraction: a modified version of JBI data extraction form describing nine aspects of each of the eight studies included in the review.

Additional file 3.

Table 2a CHEERS Checklist of included economic evaluation studies: A completed checklist of 24 items used to assess the methodological quality of all included studies in the review.

Additional file 4.

Table 2b Evers Chec-List of quality of full economic evaluations only: A completed checklist of 19 items to assess the methodological quality of full economic evaluations included in the review.

Additional file 5.

Table 3 JBI Dominance Ranking Matrix: a three by three dominance ranking matrix (DRM) tool to classify the cost-effectiveness results of the included studies as dominant and favoured, unclear or rejected.

Additional file 6.

Table 4 Costs of aspects of Quality Improvement Collaboratives in the selected studies: a comparison of costs of QICs between eight selected studies by 4 main aspects of cost of QIC

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

de la Perrelle, L., Radisic, G., Cations, M. et al. Costs and economic evaluations of Quality Improvement Collaboratives in healthcare: a systematic review. BMC Health Serv Res 20 , 155 (2020). https://doi.org/10.1186/s12913-020-4981-5

Download citation

Received : 25 July 2019

Accepted : 12 February 2020

Published : 02 March 2020

DOI : https://doi.org/10.1186/s12913-020-4981-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Quality improvement
• Collaborative
• Implementation
• Economic evaluation

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

• Submissions
• Advertising

Quality Improvement Projects and Clinical Research Studies

Editor-in-Chief Beth Faiman celebrates the contributions of advanced practitioners in quality improvement projects and clinical research studies, and compares the characteristics of each type of study.

Disclaimer » Advertising

• HealthyChildren.org
• Facebook Icon
• Twitter Icon
• LinkedIn Icon

Navigating the Path to Healthcare Equity: Insights From a Quality Improvement Initiative

Editor’s Note : Dr. Alex Eaton (he/him) is a first-year resident physician in pediatrics at The Boston Combined Residency Program at Boston Children's Hospital and Boston Medical Center. He is interested in medical education and health disparities research, specifically pediatric pain management in the setting of historic practices of race-based medicine. Alex is planning to pursue a fellowship specializing in pediatric critical care.   -Rachel Y. Moon, MD, Associate Editor, Digital Media, Pediatrics

Achieving healthcare equity remains a paramount challenge in the US healthcare system. Despite concerted efforts, disparities that hinder the delivery of quality care to all individuals, especially children of minoritized racial or ethnic groups, persist. Recognizing this, a recent initiative conducted by Asha Payne, MD, Katharine Moore, MHA, and colleagues at Children’s National Hospital aimed to address these clinical disparities. In their data-driven quality improvement project conducted under the guidance of a multidisciplinary working group, they pursued two key objectives:

1) to enhance/improve completion rates of racial demographic data documentation across the institution and

2) to identify, understand, and address disparities in clinical care.

In their article, published this week in Pediatrics ( 10.1542/peds.2023-063096 ), the authors share the team’s experiences, challenges, and lessons learned, which serve as exemplars for others committed to advancing healthcare equity.

Goal 1: Enhancing Completeness of Racial Demographic Data

Recognizing the old adage that "what gets measured gets improved," the group first explored the electronic health record to tackle the lack of completeness in the racial demographic data. One must have a robust and accurate data-based representation of a population to properly understand the problem. While we may understand disparity on a structural level, organization of anecdotal shortcomings into useful action steps requires hard data. Through a systematic approach, the research team engaged leadership and enhanced technology to encourage staff to consistently capture racial designations. These efforts resulted in significant improvement with racial demographic data completeness, reaching 95.3% across the institution.

Goal 2: Identifying Clinical Disparities

To identify clinical disparities, the authors adopted a collaborative approach by seeking input from diverse stakeholders (such as clinicians, staff, community members, and hospital leadership). Through facilitated discussions, they prioritized specific areas; for example, making sure that printed discharge instructions are in a patient’s preferred language. Further examples are shared within the article and point to the complex interplay of social determinants of health that impact Children’s National’s (and any hospital’s) ability to meet a fundamental goal: providing high value care to every single patient seen throughout the institution.

Outcomes and Lessons Learned

Children’s National’s journey has yielded valuable insights that can inform similar initiatives elsewhere. Organizational transparency (as evidenced by numerous public forums and soliciting the input of community leaders) emerged as a crucial factor, as did fostering engagement and trust among stakeholders. Progress within this realm relies heavily on institutional support, both in terms of resources and leadership commitment. Finally, the authors found that to make a long-lasting difference, it is essential to embed diversity, equity, and inclusion programs throughout healthcare systems.

Conclusion and Future Directions

Looking ahead, the focus of this team extends beyond data completeness to leveraging the data into addressing systemic challenges and social determinants of health. In the quest for healthcare equity, each institution's journey is unique, yet institutions can learn from the experience of other institutions. By fostering a culture of transparency, garnering institutional support, and embracing collaborative approaches, we can collectively strive toward a future in which disparities in healthcare are effectively addressed.

Copyright © 2024 American Academy of Pediatrics

Advertising Disclaimer »

Affiliations

• Pediatrics On Call
• Pediatrics Open Science
• Hospital Pediatrics
• Pediatrics in Review
• AAP Grand Rounds
• Latest News
• Pediatric Care Online
• Red Book Online
• Pediatric Patient Education
• AAP Toolkits
• AAP Pediatric Coding Newsletter

First 1,000 Days Knowledge Center

Institutions/librarians, group practices, licensing/permissions, integrations, advertising.

• Privacy Statement | Accessibility Statement | Terms of Use | Support Center | Contact Us
• © Copyright American Academy of Pediatrics

This Feature Is Available To Subscribers Only

Sign In or Create an Account

• Schools & Colleges
• Undergraduate Programs
• Graduate Programs
• Dual-Degree Programs
• Online Graduate Programs
• Professional Development & Continuing Education
• Academic Support
• Research & Scholarship
• Undergraduate
• Admitted Students Next Steps
• International Students
• Financial Aid & Cost
• Four-Year Guarantee
• Community Involvement Program
• Our Campuses
• Student Outcomes
• Community Impact
• Diversity, Equity and Inclusion
• Sustainability
• President Callahan
• Administrative Offices
• University Leadership
• History & Mission
• Activities & Programs
• Housing & Dining
• Student Services
• Career Services
• Equity & Inclusion
• Safety & Wellness

Innovative Clinical and Outcomes Research

• Thomas J. Long School of Pharmacy
• Support the School
• Interactions Magazine
• Annual Report
• Continuing Education
• How to Apply
• App Deadlines
• Prerequisites
• Technical Standards
• Tuition & Financial Aid
• Admitted PharmD Students
• Pre-Pharmacy Advantage Program
• BS in Pharmaceutical Science
• MS in Regulatory Science
• Doctor of Pharmacy
• Pharmaceutical and Chemical Sciences
• PharmD Outcomes Data
• Jie Du Center
• Flow Therapy
• U.S. Air Force
• Pharmacy Practice
• Pharmaceutics & Medicinal Chemistry
• Physiology & Pharmacology
• Preceptor Corner
• Noteworthy Submissions
• Medicare Part D Outreach Clinics
• Community Impact - Medicare Part D
• Diabetes Care Clinics

The Innovative Clinical and Outcomes Research (iCOR) team is a hub of cutting-edge clinical research at the Thomas J. Long School of Pharmacy. iCOR provides opportunities for students to collaborate with peers, faculty, fellows and industry pharmacists on high-impact research projects. The team is led by accomplished faculty members who are leading experts in their fields and is supported by an impressive network of successful alumni who are now leaders in industry.

As trailblazers in the medical device and dietary supplement space, the iCOR team spearheads innovative research projects and clinical trials to deliver consequential findings that impact patient care and the scientific community. We engage in innovative projects which use existing data or data collected by recruiting more than 100 volunteers. We focus on studies which can be conducted rapidly, between six months and two years. Completed projects have ranged from proof-of-concepts to studies validating causality. iCOR conducts research in collaboration with the Fellowship in Industry Program, which includes partnerships with Flow Therapy , Genentech and the United States Air Force .

Where we’re going

Building on extensive research and previous work, iCOR continues to pioneer innovative research through a robust pipeline of ongoing research programs, including research on continuous glucose monitoring devices and enhanced external counter pulsation treatments. Through continuous evaluation of the current industry landscape, iCOR’s research focus stays ahead of the curve. In addition, the team seeks to continue to strategically develop working relationships with leading pharmaceutical and biotechnology companies.

Media impressions

Posters/publications

Clinical trials

We strive to advance patient care through innovative research. Our goal is to train students and prepare them to become leaders in the pharmaceutical industry by providing rich learning experiences, leadership opportunities and mentorship.

What started as a passion project for one student to pursue the pharmaceutical industry pathway has grown into a track for students to excel in this arena. As iCOR has grown over the years, the scope of the projects conducted by its members have also expanded. The work of our students, faculty and alumni has been nationally recognized through numerous publications and presentations. Our projects illustrate the classic interplay between science and scientists, a cat-and-mouse game in the pursuit of optimizing human health.

Sachin A. Shah, PharmD, FACC, FAHA

Kate M. O’Dell, PharmD, BCPS

Allen Shek, PharmD

Nancy N. Nguyen, PharmD, BCPS, AAHIVP, FCSHP

Class of 2022

• Jason Hong , PharmD - Clinical Science Fellow at Genentech
• Jessica Kaye , PharmD - Oncology Drug Safety Fellow at Seagen
• Kimmy Dovan , PharmD - Regulatory Affairs: Advertising and Promotion Fellow at Merck
• Ricky Philipossian , PharmD - Clinical Development Fellow at Pfizer
• Lena Tieu - Global Medical Information Fellow at Gilead
• Reina Marie Sanz - Regulatory Affairs Fellow at Novartis
• Huy Pham - PharmD/ PhD Candidate at University of the Pacific

Class of 2021

• Brandon Tran , PharmD - Global Clinical Operations Oncology Fellow at Genentech
• Anh Nguyen , PharmD - Clinical Research and Drug Development Fellow at UNC
• Brittany Tran , PharmD - Quantitative Clinical Pharmacology Fellow at Daiichi Sankyo
• Elizabeth Lindemann , PharmD - Biologics Associate at Regeneron
• Katherine Hsu , PharmD - US Marketing & Global Commercial Strategy, Rare Diseases at Sanofi Genzyme
• Simran Randhawa , PharmD - Global Medical Affairs: Inflammation & Immunology Fellow at Pfizer

"iCOR is filled with opportunities to explore a career within the pharmaceutical industry. Having the opportunity to learn about the drug development process, conducting clinical trials and analyzing clinical data challenged my critical and clinical knowledge throughout my experience within iCOR. It helped to cement my passion for research and my continued desire to to make an impact in health care."

— Huy Pham ’22 , PharmD/PhD Candidate University of the Pacific

"iCOR had a profound impact on my journey through pharmacy school. The collaborative relationship between faculty and students of iCOR creates an environment that combines critical thinking and science, and ultimately maximizes patient outcomes. I got the opportunity to conduct literature review, design and execute clinical trials, and summarize the resulting data in manuscripts. It helped me realize my passion for pursuing a career in clinical development within the pharmaceutical industry."

— Ricky Philipossian ’22 , PharmD Clinical Development Fellow at Pfizer

Energy Drink

Journal of Diabetes Science and Technology, May 2019 Journal of the American Heart Association, April 2017

Continuous Glucose Monitoring (CGM) Studies

SAGE Journals: Digital Health, November, 2020 Journal of Diabetes Science and Technology, May 2019

Enhanced External Counterpulsation (EECP)

American Journal of Cardiovascular Disease, December 2020 American Heart Association Journals: Circulation, November 2019

Purple Carrot Meal Kits

American Heart Association Journals: Circulation, November 2020

Study shows glucose levels can be different between left and right arm

In a novel study, Pacific researchers found the selection of either the right or left arm for continuous glucose monitoring device placement could potentially affect the glucose readings collected by the devices.

Research shows health benefits of plant-based meal kits

Results of a clinical trial showed a statistically significant reduction in both “bad” cholesterol and body weight when volunteers ate dinners made from a plant-based meal kit service compared to a non-plant-based meal kit.

Research team tests hangover product

The team conducted a randomized, double-blind, placebo-controlled study to determine if The Hangover Secret, a dietary supplement mix, reduced hangover symptoms.

Accepted fellowships or residencies

Alumni in the pharmaceutical industry

“iCOR gave me the opportunity to continually challenge my critical thinking and broaden my perspective. My understanding of drug development deepened through hands-on experience with designing and executing the first virtual clinical trial at Pacific. I witnessed firsthand how vital research is in impacting and improving patient care. The skills and lessons I learned from the supportive mentorship and unique experiences in iCOR will stay with me throughout my career.”

— Kimmy Dovan ’22 , PharmD Regulatory Affairs: Advertising and Promotion Fellow at Merck

[email protected]

• Open access
• Published: 10 May 2024

Community-based participatory-research through co-design: supporting collaboration from all sides of disability

• Cloe Benz   ORCID: orcid.org/0000-0001-6950-8855 1 ,
• Will Scott-Jeffs 2 ,
• K. A. McKercher   ORCID: orcid.org/0000-0003-4417-585X 3 ,
• Mai Welsh   ORCID: orcid.org/0000-0002-7818-0115 2 , 4 ,
• Richard Norman   ORCID: orcid.org/0000-0002-3112-3893 1 ,
• Delia Hendrie   ORCID: orcid.org/0000-0001-5022-5281 1 ,
• Matthew Locantro 2 &
• Suzanne Robinson   ORCID: orcid.org/0000-0001-5703-6475 1 , 5  

Research Involvement and Engagement volume  10 , Article number:  47 ( 2024 ) Cite this article

820 Accesses

Metrics details

As co-design and community-based participatory research gain traction in health and disability, the challenges and benefits of collaboratively conducting research need to be considered. Current literature supports using co-design to improve service quality and create more satisfactory services. However, while the ‘why’ of using co-design is well understood, there is limited literature on ‘ how ’ to co-design. We aimed to describe the application of co-design from start to finish within a specific case study and to reflect on the challenges and benefits created by specific process design choices.

A telepractice re-design project has been a case study example of co-design. The co-design was co-facilitated by an embedded researcher and a peer researcher with lived experience of disability. Embedded in a Western Australian disability organisation, the co-design process included five workshops and a reflection session with a team of 10 lived experience and staff participants (referred to as co-designers) to produce a prototype telepractice model for testing.

The findings are divided into two components. The first describes the process design choices made throughout the co-design implementation case study. This is followed by a reflection on the benefits and challenges resulting from specific process design choices. The reflective process describes the co-designers’ perspective and the researcher’s and organisational experiences. Reflections of the co-designers include balancing idealism and realism, the value of small groups, ensuring accessibility and choice, and learning new skills and gaining new insights. The organisational and research-focused reflections included challenges between time for building relationships and the schedules of academic and organisational decision-making, the messiness of co-design juxtaposed with the processes of ethics applications, and the need for inclusive dissemination of findings.

Conclusions

The authors advocate that co-design is a useful and outcome-generating methodology that proactively enables the inclusion of people with disability and service providers through community-based participatory research and action. Through our experiences, we recommend community-based participatory research, specifically co-design, to generate creative thinking and service design.

Plain language summary

Making better services with communities (called co-design) and doing research with communities (e.g. community-based participatory research) are ways to include people with lived experience in developing and improving the services they use. Academic evidence shows why co-design is valuable, and co-design is increasing in popularity. However, there needs to be more information on how to do co-design. This article describes the process of doing co-design to make telepractice better with a group of lived experience experts and staff at a disability organisation. The co-design process was co-facilitated by two researchers – one with a health background and one with lived experience of disability. Telepractice provides clinical services (such as physiotherapy or nursing) using video calls and other digital technology. The co-design team did five workshops and then reflected on the success of those workshops. Based on the groups’ feedback, the article describes what worked and what was hard according to the co-designers and from the perspective of the researchers and the disability organisation. Topics discussed include the challenge of balancing ideas with realistic expectations, the value of small groups, accessibility and choice opportunities and learning new skills and insights. The research and organisational topics include the need to take time and how that doesn’t fit neatly with academic and business schedules, how the messiness of co-design can clash with approval processes, and different ways of telling people about the project that are more inclusive than traditional research. The authors conclude that co-design and community-based participatory research go well together in including people with lived experience in re-designing services they use.

Peer Review reports

Introduction

Co-design has the potential to positively impact co-designers and their community, researchers, and organisations. Co-design is defined as designing with, not for, people [ 1 ] and can reinvigorate business-as-usual processes, leading to new ideas in industry, community and academia. As co-design and community-based participatory research gain traction, the challenges and benefits of collaborative research between people with lived experience and organisations must be considered [ 2 ].

Disability and healthcare providers previously made decisions for individuals as passive targets of an intervention [ 3 ]. By contrast, the involvement of consumers in their care [ 4 ] has been included as part of accreditation processes [ 4 ] and shown to improve outcomes and satisfaction. For research to sufficiently translate into practice, consumers and providers should be involved actively, not passively [ 4 , 5 ].

Approaches such as community-based participatory research promote “a collaborative approach that equitably involves community members, organisational representatives and researchers in all aspects of the research process” [ 6 ] (page 1). This approach originated in public health research and claims to empower all participants to have a stake in project success, facilitating a more active integration of research into practice and decreasing the knowledge to practice gap 6 . Patient and public involvement (PPI) increases the probability that research focus, community priorities and clinical problems align, which is increasingly demanded by research funders and health systems [ 7 ].

As community-based participatory research is an overarching approach to conducting research, it requires a complementary method, such as co-production, to achieve its aims. Co-production has been attributed to the work of Ostrom et al. [ 8 ], with the term co-design falling under the co-production umbrella. However, co-design can be traced back to the participatory design movement [ 9 ]. The term co-production in the context of this article includes co-planning, co-discovery, co-design, co-delivery, and co-evaluation [ 10 ]. Within this framework, the concept of co-design delineates the collaborative process of discovery, creating, ideating and prototyping to design or redesign an output [ 11 ]. The four principles of co-design, as per McKercher [ 1 ], are sharing power, prioritising relationships, using participatory means and building capacity [ 1 ]. This specific method of co-design [ 1 ] has been used across multiple social and healthcare publications [ 10 , 12 , 13 , 14 ].

A systematic review by Ramos et al. [ 15 ] describes the benefits of co-design in a community-based participatory-research approach, including improved quality and more satisfactory services. However, as identified by Rahman et al. [ 16 ], the ‘ why ’ is well known, but there is limited knowledge of ‘ how ’ to co-design. Multiple articles provide high-level descriptions of workshops or briefly mention the co-design process [ 13 , 17 , 18 , 19 ]. Pearce et al. [ 5 ] include an in-depth table of activities across an entire co-creation process, however within each part i.e., co-design, limited descriptions were included. A recent publication by Marwaa et al. [ 20 ] provides an in-depth description of two workshops focused on product development, and Tariq et al. [ 21 ] provides details of the process of co-designing a research agenda. Davis et al. [ 11 ] discuss co-design workshop delivery strategies summarised across multiple studies without articulating the process from start to finish. Finally, Abimbola et al. [ 22 ] provided the most comprehensive description of a co-design process, including a timeline of events and activities; however, this project only involved clinical staff and did not include community-based participation.

As “We know the why, but we need to know the how-to” [ 16 ] (page 2), of co-design, our primary aim was to describe the application of co-design from start to finish within a specific case study. Our secondary aim was to reflect on the challenges and benefits created by specific process design choices and to provide recommendations for future applications of co-design.

Overview of telepractice project

The case study, a telepractice redesign project, was based at Rocky Bay, a disability support service provider in Perth, Australia [ 23 ]. The project aimed to understand the strengths and pain points of telepractice within Rocky Bay. We expanded this to include telepractice in the wider Australian disability sector. The project also aimed to establish potential improvements to increase the uptake and sustainability of Rocky Bay’s telepractice service into the future. Rocky Bay predominantly serves people under the Australian National Disability Insurance Scheme (NDIS) [ 24 ] by providing a variety of services, including allied health (e.g. physiotherapy, dietetics, speech pathology, etc.), nursing care (including continence and wound care), behaviour support and support coordination [ 23 ]—Rocky Bay services metropolitan Perth and regional Western Australia [ 23 ].

The first author, CB, predominantly conducted this research through an embedded researcher model [ 25 ] between Curtin University and Rocky Bay. An embedded researcher has been defined as “those who work inside host organisations as members of staff while also maintaining an affiliation with an academic institution” [ 25 ] (page 1). They had some prior contextual understanding which stemmed from being a physiotherapist who had previously delivered telehealth in an acute health setting. A peer researcher, WSJ, with lived experience of disability, worked alongside CB. They had no previous experience in research or co-design, this was their first paid employment and they had an interest in digital technology. Peer Researcher is a broad term describing the inclusion of a priority group or social network member as part of the research team to enhance the depth of understanding of the communities to which they belong [ 26 ]. Including a peer researcher in the team promoted equity, collective ownership, and better framing of the research findings to assist with connecting with people with lived experience. These outcomes align with key components of community-based participatory research and co-design [ 27 , 28 , 29 , 30 ].

Person-first language was used as the preference of experts with lived experience who contributed to this research to respect and affirm their identity. However, we respect the right to choose and the potential for others to prefer identity-first language [ 31 ].

A summary of the structure of the phases completed before co-design workshops are represented in Fig.  1 below. Ethical approval for the project was received iteratively before each phase on the timeline (Fig.  1 ) from the Curtin Human Research Ethics Committee (HRE2021-0731). The reporting of this article has been completed in line with the Guidance for Reporting Involvement of Patients and the Public (GRIPP2) checklist [ 7 ].

Summary of telepractice co-design project structure [ 1 ]

Here, we present an outline of the chosen research methods with descriptions of each process design choice and supporting reasons and examples specific to the study. The format is in chronological order, with further details of each step provided in Appendix 1 (Supplementary Material 1).

Methods and results

Process of co-production and preparation for co-design.

Co-production was chosen as the planning method for the study, as the inclusion of community members (Rocky Bay Lived experience experts and Staff) in each step of the research process would increase buy-in and make the research more likely to meet their needs [ 5 ]. An example of co-planning (part of co-production) includes the study steering committee, with a lived experience expert, clinician and project sponsor representatives collaborating on the selection of study aim, methods and recruitment processes. Another example of co-planning, co-design, and co-delivery was recruiting a peer researcher with disability, who worked with the embedded researcher throughout the study design and delivery.

The second process design choice was to attempt to build safe enough conditions for community participation, as people who feel unsafe or unwelcome are less likely to be able to participate fully in the research [ 1 ]. Building conditions for safety was applied by repeatedly acknowledging power imbalances, holding space for community input, and anticipating and offering accessibility adjustments without judgment.

Getting started

Understanding and synthesising what is already known about telepractice experiences and learning from lived experience was prioritised as the first step in the process. We paired a scoping review of the literature with scoping the lived experiences of the community [ 32 ]. Our reasoning was to understand whether the findings aligned and, secondly, to learn what had already been done and to ask what was next, rather than starting from the beginning [ 1 ]. Examples of strategies used in this step included interviewing clinicians and service provider Managers across Australia to establish how they implemented telepractice during the pandemic and understand their views of what worked and what did not. The second learning process occurred onsite at Rocky Bay, with people with lived experience, clinicians and other support staff, whom the embedded researcher and peer researcher interviewed to understand experiences of telepractice at Rocky Bay.

The authors presented the interview findings during focus groups with Rocky Bay participants to share the learnings and confirm we had understood them correctly. The groups were divided into staff and lived experience cohorts, allowing for peer discussions and sharing of common experiences. This helped build relationships and a sense of familiarity moving into the workshop series.

Co-design workshops

This section outlines specific components of the co-design workshop preparation before describing each of the five workshops and the final reflection session.

Staff and community co-designers

Two process design choices were implemented to form the co-design group. The first was to prioritise lived experience input as there are generally fewer opportunities for lived experience leadership in service design [ 16 ], and because the disability community have demanded they be included where the focus impacts them [ 33 ]. To acknowledge the asymmetry of power between community members, people with lived experience of disability and professionals, we ensured the co-design group had at least the same number of lived experience experts as staff.

The second priority for the co-design group was to include people for whom involvement can be difficult to access (e.g. people who are isolated for health reasons and cannot attend in-person sessions, people who live in supported accommodation, part-time staff, and people navigating the dual-role of staff member while disclosing lived experience). It was important to learn from perspectives not commonly heard from and support equity of access for participants [ 4 ].

Workshop series structure

When structuring the workshop series, lived experience co-designers nominated meeting times outside standard work hours to reduce the impact of co-design on work commitments and loss of income while participating. The workshops were designed to be delivered as a hybrid of in-person and online to give co-designers a choice on how they wanted to interact. The workshops were designed as a series of five sequential 90-minute workshops, where co-designers voted for the first workshop to be predominantly in-person and the remainder of the workshops online. Some co-designers chose to attend the initial session in person to build rapport. However, the virtual option remained available. The subsequent online sessions reduced the travel burden on co-designers, which the co-designers prioritised over further face-to-face meetings.

Workshop facilitators

To maintain familiarity and ensure predictability for co-designers, the workshops were co-facilitated by the embedded researcher and peer researcher. The co-facilitators built on relationships formed through previous interactions (interviews and focus groups), and each facilitator represented part of the co-designer group as a clinician or a person with disability. An extra support person was tasked with supporting the co-designers with disability to break down tasks and increase the accessibility of activities. The reason for selecting the support person was that they could contribute their skills as a school teacher to support the communication and completion of activities, and they had no previous experience with disability services to influence the co-designers opinions. This role was adapted from the provocateur role described by McKercher [ 1 ].

Pre-workshop preparations

To prepare for the workshops, each co-designer was asked to complete a brief survey to ensure the co-facilitators understood co-designers collect preferences and needs ahead of the session to enable preparation and make accommodations. The survey included pronouns, accessibility needs and refreshment preferences. Following the survey, the co-facilitators distributed a welcome video; the peer researcher, a familiar person, was videoed explaining what to expect, what not to expect and expected behaviours for the group to support a safe environment [ 1 ]. This process design choice was made to allow co-designers to alleviate any potential anxieties due to not having enough information and to increase predictability.

Workshop resources and supports

As the first workshop was in-person, specific process choices were made to ensure co-designers felt welcome and to uphold the dignity of co-designers with lived experience [ 34 ]. Examples of process design choices include facilitating transport and parking requests, providing easy access to the building and room, making a sensory breakout room available and having the peer researcher waiting at the entrance to welcome and guide people to the workshop room.

After reaching the workshop room, all co-designers received an individualised resource pack to equalise access to workshop materials, aiming again to balance power in a non-discriminatory way [ 11 ]. The resource pack included name tags with pronouns, individualised refreshments, a fidget toy [ 35 ] whiteboard markers and a human bingo activity described in a later section. An easy-to-apply name tag design was selected after consulting a co-designer with an upper limb difference. Further details on the resource packs are included in Appendix 1 (Supplementary Material 1).

Enabling different kinds of participation

We provided non-verbal response cards to each co-designer as communication preferences vary significantly within the disability community. The cards were intended to benefit any co-designer who struggled to use the response buttons on MS teams. The co-facilitators co-created the Yes, No, and In-the-middle response cards (Fig.  2 ) and were guided by recommendations by Schwartz and Kramer [ 29 ]. They found that people with intellectual disability were more likely to respond “yes” if the negative option included a frowning face or red-coloured images, as choosing these types of alternatives was perceived as being negative or would cause offence [ 29 ].

Non-verbal response cards

A summary of the structure and purpose of each of the five workshops is shown in Fig.  3 , followed by a more in-depth discussion of the strategies employed in each workshop.

Outline of workshop and group structures

Workshop 1: the beginning

Human Bingo was the first workshop activity, as it aimed to support relationship building in an inclusive way for both in-person and online attendees. The activity asked each co-designer to place a name in each worksheet box of someone who fit the described characteristic of that square(for example, someone who likes cooking). To include the two online attendees, laptops were set up with individual videocall streams and noise cancelling headphones enabling the online co-designers to interact one-on-one with others during the activities.

The second activity used The Real Deal cards by Peak Learning [ 36 ] to ask the co-designers to sort cards to prioritise the top five experiences and feelings they would want in a future version of telepractice. This activity aimed to set initial priorities for the redesign of telepractice [ 1 ]. Small groups with a mix of lived experience experts and staff were tasked with negotiating and collaborating to produce their top five desired experiences and feelings for future service success.

A follow-up email was sent after the session to thank co-designers, provide closure, invite feedback and let co-designers know what to expect from the next session.

Workshop 2: mapping the journey

In the second workshop, held online, the co-facilitators explained the journey mapping process and showed a draft of how the visual representation would likely look (Fig.  4 ). As the first step, co-designers were tasked with completing a series of activities to analyse lived experience interview data on the current experience of telepractice for lived experience experts. Small mixed groups were created, prioritising the needs of the lived experience experts to have staff who would be the best fit in supporting them to work through the task [ 1 ]. The small groups were allocated interview quotes corresponding to the steps of a customer journey through telepractice and asked to identify strengths, challenges and emotions associated with the current Telepractice service journey at Rocky Bay [ 1 ]. Further details on the journey map analysis are described in Appendix 1 (Supplementary Material 1) and in a published article co-authored by the co-designers (Benz et al. [ 37 ]).

Draft journey map visualisation

After workshop two, the embedded researcher drafted a journey map by compiling the co-designer group responses to the analysis activity, which was then circulated for feedback and confirmation. The completed journey map is published with further details on the process in an article co-authored with the co-designers, Benz et al. [ 37 ].

Workshop 3: ideas for addressing pain points

For the third workshop, the co-facilitators selected activities to be completed separately by lived experience and staff co-designers. The lived experience expert activity involved exploring preferences for improving pain points identified through the journey map. The lived experience expert activity was facilitated by the peer researcher and support person and included questions such as, how would it be best to learn how to use telepractice? Visual prompt cards were shared to support idea creation, where lived experience expert co-designers could choose any option or suggest an alternative (Fig.  5 ).

Option cards for Lived experience expert co-designer workshop activity

Simultaneously, the staff co-designers completed a parallel activity to address pain points from a service delivery point of view. These pain points were identified in the clinical and non-clinical staff interviews and from the journey map summary of lived experience expert interviews (analysed in Workshop 2). Staff co-designers completed a mind map based on service blueprinting guidelines by Flowers and Miller [ 38 ]. The activity used service blueprinting to identify a list of opportunities for improvement, with four prompts for co-designers to commence planning the actions required to implement these improvements. The foci of the four prompts were roles, policies, technology and value proposition [ 38 ] (described further in Appendix 1 (Supplementary Material 1)). Each of the four prompts were completed for the ten proposed opportunities for improvement to draft plans for future telepractice service delivery.

Workshop 4: story telling and generation of future state solutions

In the fourth workshop, we introduced the concept of prototyping [ 39 ] as a designerly way to test co-designers’ ideas for improving telepractice according to desirability, feasibility and viability with a wider audience of lived experience experts and staff. The co-designers helped to plan the prototyping, and accessibility was a key consideration in selecting a prototype, as the group were conscious of the target audience.

Creating the prototype was collaborative, allowing co-designers to produce an output representing their ideas. They selected a video storyboard prototype with a staff and customer version formatted similarly to a children’s book. It included cartoon animations completed on PowerPoint, voiceover narration, closed captioning and an introductory explanation from two co-designers.

After workshop four, the co-designers collaborated on the customer and staff prototypes during the two weeks between workshops four and five, with support and input from the facilitators. The prototype files were co-produced, with different co-designers working on the visual aspects, the script for the main audio narration and the introductory explanation.

Workshop 5: finishing the story

The co-design group reviewed the draft prototypes in the final workshop, with specific attention paid to the story’s cohesiveness.

The feedback questionnaire was then created to be completed by viewers outside of the co-design group after engaging with either the staff or the customer prototype. The survey allowed Rocky Bay customers and staff to contribute ideas. Following thoughtful discussions, consensus was reached by all co-designers on the final survey questions (Appendix 2 (Supplementary Material 1)).

A reflection activity concluded the final workshop, allowing co-designers to provide feedback on the co-design process, elements for improvement and aspects they valued in participating in the project. Their reflections on the benefits and challenges of co-design in this study are included in the section Co-designer’s perspectives of the workshop series , with the reflection questions included in Appendix 3 (Supplementary Material 1).

Post prototype reflection session

The prototype feedback responses were reviewed with co-designers in a final reflection session. The group then discussed adaptations to the implementation plan for proposal to Rocky Bay. Following the survey discussion, co-designers reviewed proposed service principles for the new telepractice implementation recommendations. These principles aim to align any future decisions in the implementation and service provision stages of the telepractice project with the intentions of the co-designers. An additional reflection activity was completed, specific to the telepractice proposal they had produced and the prototyping process. Feedback relevant to subsequent discussions of the challenges and benefits of co-design is included in the following section: Co-designer’s perspectives of the workshop series , with the reflection prompts in Appendix 3 (Supplementary Material 1).

Benefits and challenges

Learnings derived from completing a study of this kind are complex. However, it is necessary to reflect on which strategies used in the project were beneficial and which strategies created challenges - anticipated and unexpected. These reflections are discussed in two sections, the first being the challenges and benefits reflected upon by co-designers. The second set of reflections relates to organisational and research project-level benefits and challenges from the perspective of clinical department managers and researchers involved in the project.

Co-designer’s perspectives of the workshop series

Co-designers were positive overall about the workshop series. Responses to a prompt for one-word descriptors of their experience included “captivating, innovative, fulfilling, exciting, insightful, helpful, eye-opening and informative ” .

Co-designing as a team

A foundational strategy implemented in this project was the intentional collaboration of lived experience experts with staff; this linked to the co-design principle of prioritising relationships and sharing power. Multiple reflections commented on feeling like a team and that having diverse perspectives across the group was beneficial.

It was especially interesting to hear the perspective of clinicians (for us, the other side of Telepractice). [Lived experience expert Co-designer]

Additionally, the combination of facilitators, including an embedded researcher with an allied health clinical background, a peer researcher with lived experience and a support person with strengths in breaking down tasks, provided different facets of support and task modelling to the co-designers throughout the process.

Balancing idealism and realism

There is an inherent challenge in collaboration between lived experience experts and service providers, whereby co-designers formulate ideas for service improvement and then, in good faith, propose required changes to be implemented. Strategies to support imagination and idealism while being honest about the constraints of what can be delivered were implemented in the context of this project. This was essential to reinforce to co-designers that their contributions and ideas are valid while tempering their hopes with the truth that organisational change is challenging and funding for change is limited. Co-designers were encouraged to be cognisant of ideas that would require high investment (cost and time) and which ideas faced fewer barriers to implementation. This strategy did not prevent the ideation of changes and prioritising what mattered most to them, and co-designers felt it was beneficial in adding a level of consideration regarding what investments they deemed necessary versus those that would be nice to have. For example, having a person to call for help was viewed as necessary, while a nice to have was more advanced technological features.

I feel that the prototype is useful; however, I worry that nothing will be carried over to the Rocky Bay Service. I feel like more customers will want to access telepractice, and Rocky Bay now needs to start the implementation process to ensure that telepractice is utilised, including processes, education and training. [Clinician Co-designer]

The value of small groups

Working in small groups was another beneficial strategy, aiming to create a more hospitable environment for co-designers to voice their thoughts. The small groups varied across activities and workshops, with facilitators intentionally pairing groups that would best support the lived experience of expert co-designers completing activities. As described in the workshop sections, some activities suited mixed groups, whereas others suited lived experience expert and staff-specific groups. Two reflective comments demonstrated the benefit of the small groups, one from a clinician who reflected on supporting a fellow co-designer:

I found that in our group, all of us had a say; however, [Lived Experience Co-designer name] was a bit overwhelmed at times, so I tried to support her with that. [Clinician Co-designer]

And a lived experience expert co-designer additionally reflected:

The breakout rooms were a very good idea. It can be quite intimidating speaking in front of the main group. I found it much easier to participate in the smaller groups . [Lived experience expert Co-designer]

The second session included an unplanned whole group activity, which challenged co-designers. Co-designers reflections of this experience demonstrate the benefits of smaller groups:

I did feel that at the end when the whole group did the task, there wasn’t as much collaboration as there were quite a few more assertive participants, so the quieter ones just sat back. [Clinician Co-designer]

Accessibility and choice

A challenge navigated throughout the workshop series with a diverse group of co-designers was meeting their varying individual health and other needs. This required responding in sensitive, non-judgemental, and supportive ways to encourage co-designers to engage fully. Examples of support include the presence of a support person and adaption of resource packs for co-designers who have difficulty swallowing (re: refreshments), as well as the previously mentioned non-verbal response cards and accessible name tags.

Accessibility supports were also provided for the peer researcher during facilitation activities, including pre-written scripts to provide clarity when explaining tasks to the co-design group, written reminders and regular check-ins. A lived experience expert co-designer reflected that it was beneficial that they could tell the peer researcher was nervous but appreciated that he was brave and made them feel like they did not need to be perfect if the peer researcher was willing to give it a go.

When facilitating the sessions, the embedded researcher and peer researcher identified that the workshops were long and, at times, mentally strenuous. One co-designer requested “more breaks during each session” . Breaks were offered frequently; however, upon reflection, we would schedule regular breaks to remove the need for co-designers to accept the need for a break in front of the group. The instructions for each activity were visual, verbal and written and given at the start of a task. However, once the co-designers were allocated to breakout rooms, they could no longer review the instructions. Many co-designers suggested that having the instructions in each breakout room’s chat window would have been a valuable visual reminder.

One thing I think might of helped a little is having the instructions in the chat as I know I that I listened but couldn’t recall some of the instructions for the group task. [Lived experience expert Co-designer]

Learning new skills and gaining new insight

The co-designers considered that the benefits of working together included learning new skills and widening their understanding of research, the services they provide or use, and the differences between the priorities of lived experience experts and staff. Two lived experience experts commented that the opportunity to learn collaboration skills and create cartoons using PowerPoint were valuable skills for them to utilise in the future. One clinician reflected that the process of co-design had improved their clinical practice and increased their use of telepractice:

My practice is 100% better. I am more confident in using telepractice and more confident that, as a process, it doesn’t reduce the impact of the service- in some ways, it has enhanced it when customers are more relaxed in their own environments. I have not seen my stats, but my use of telepractice has increased significantly, too. [Clinician Co-designer]

The management co-designer acknowledged that although ideas across the group may be similar, prioritisation of their importance can vary dramatically:

Whilst all the feedback and potential improvements were very similar, some things that I viewed as not an issue, was very different to a customer’s perspective. [Management Co-designer]

Overall, the workshop series challenged co-designers. However, the provision of a supportive and accessible environment resulted in mutual benefits for the research, organisation, and co-designers themselves. The strategy for facilitating the workshops was to pose challenges, support the co-designers in rising to meet them, and take into account their capabilities if provided with the right opportunity. A lived experience expert co-designer summarised the effectiveness of this strategy:

I found the activities to be challenging without being too difficult. Each activity provided enough guidance and structure to encourage interesting group discussions and make collaboration easy. [Lived experience expert Co-designer]

Research and organisational reflections of benefits and challenges of co-design

A significant challenge in completing this project was that building foundational relationships and trust takes time. While the authors view this trust as the foundation on which community-based participatory research and co-design are built, they note the direct tension of the time needed to develop these foundational relationships with the timeline expectations of academic and organisational decision-making. The flexibility required to deliver a person-centred research experience for the co-designers resulted in regular instances when timeline extensions were required to prioritise co-designer needs over efficiency. The result of prioritising co-designer needs over research timeline efficiency was an extended timeline that was significantly longer than expected, which sometimes created a disconnect between the flexibility of co-design and the rigidity in traditional academic and organisational processes.

The impacts of a longer-than-expected timeline for completion of the co-design process included financial, project scope, and sponsorship challenges. The project’s initial scope included a co-implementation and co-evaluation phase; however, due to the three-year time constraint, this was modified to conclude following the prototyping process. Whilst the three-year period set expectations for project sponsors and other collaborators from Rocky Bay, the wider context for the project varied significantly and rapidly over this period. This included two changes in Rocky Bay supervisor and one change in Rocky Bay project sponsor. Additionally, one of the academic supervisors left Curtin. This challenge indicates that the project would benefit from key role succession planning.

The peer researcher role was beneficial in providing an opportunity for a person with lived experience to join the study in a strength-based role and experience academic and business processes. However, challenges arose with the timeline extensions, which required this part-time, casual role to be extended by seven months. While the contract extension posed budgetary challenges, the role was viewed as vital to the completion of the project.

While an essential component of research, particularly involving vulnerable populations, ethical approvals proved challenging due to the non-traditional research methods involved in co-design. It was evident to the authors that while the ethics committee staff adhered to their processes, they were bound by a system that did not have adequate flexibility to work with newer research methods, such as co-design. Multiple methods in this study were heavily integrated into the community, including embedded research, peer research and co-design.

The present ethics process provided a comprehensive review focusing on planned interactions within research sessions (e.g. interviews and workshops). Unfortunately, this failed to account for a wider view, including the initial co-production prior to ethical application and anecdotal interactions that occurred regularly in the organic co-design process. In addition to the repeated submissions required to approve the sequential study format, these interactions created a significant workload for the research team and ethics office. These challenges were compounded by the need to navigate Rocky Bay’s organisational processes and changing business needs within ethical approval commitments.

In the authors’ opinion, prioritising the inclusion of lived experience experts in co-creating outputs to disseminate findings was beneficial. The co-creation enabled an authentic representation of the study to audiences regarding community-based participatory research and co-design method implementation. For example, the presentation of a panel discussion at a conference in which the peer researcher could prerecord his responses to questions as his preferred method of participation. All posters presented by the project were formatted to be accessible to lay consumers and were collaboratively produced, with the additional benefit of the posters being displayed across Rocky Bay hubs for customers and staff to gain study insights.

Due to the co-design method’s dynamic nature, some budgetary uncertainty was challenging to navigate. However, financial and non-financial remuneration for all non-staff participants in the project was prioritised. As previously discussed, the position of peer researcher was a paid role; additionally, all lived experience expert participants were remunerated at a rate of AUD 30/hour in the form of gift cards. The carer representative on the steering committee recommended using gift cards to avoid income declaration requirements from government benefits people may receive. Non-financial remuneration for the valuable time and contribution of the co-designer group included co-authorship on an article written regarding the Journey Map they produced (Benz et al. [ 37 ]) and acknowledgement in any other appropriate outputs. The implementation proposal provided to Rocky Bay included recommendations for continued inclusion and remuneration of co-designers.

Setting a new bar for inclusion

Another benefit to reflect upon, which may be the most significant legacy of the project, was setting the precedence for the inclusion of people with disability in decision-making roles in future projects and research conducted by the University and Rocky Bay. After this project commenced, other Rocky Bay clinical projects have similarly elevated the voices of lived experience in planning and conducting subsequent quality improvement initiatives.

I’m lucky enough to have been part of a lot of projects. But I guess I probably haven’t been a part of continuous workshops, pulling in all perspectives of the organisation perfectly… So, collaboration and getting insight from others I haven’t usually was a very unique experience, and I definitely found value if this were to continue in other projects. [Manager Co-designer]

In summary, the findings from using a co-design method for the telepractice research study produced a series of benefits and presented the researchers with multiple challenges. The findings also addressed a literature gap, presenting in-depth descriptive methods to demonstrate how co-design can be applied to a specific case.

Drawn from these findings, the authors identified six main points which form the basis of this discussion. These include (1) the fact that the necessary time and resources required to commit to co-design process completion adequately were underestimated at the outset, (2) there is a need to support the health, well-being and dignity of lived experience expert participants, (3) academic ethical processes have yet to adapt to address more participatory and integrated research methods, (4) strategies used to foster strong collaborative relationships across a diverse group were valued by all participants, (5) better delineation between terminologies such as co-design and community-based participatory research or patient and public involvement would improve the clarity of research methods and author intent and, (6) broader non-traditional impacts that participatory research can create should be better quantified and valued in the context of research impact. Each point will now be discussed in further detail.

In underestimating the time and resources required to complete the telepractice study, a scope reduction was required. This scope reduction removed the study’s originally planned co-implementation and co-evaluation phases. While Harrison et al. [ 40 ] and Bodden and Elliott [ 41 ] advocate for more frequent and comprehensive evaluation of co-designed initiatives, the authors acknowledge that this became no longer feasible within the study constraints. A growing body of literature indicates expected timelines for completed co-production projects from co-planning to co-evaluation. An example by Pearce et al. [ 5 ] indicated that a timeline of five years was reasonable. In contrast, a more limited co-design process was completed with a shorter timeline by Tindall et al. [ 13 ]. Although neither of these articles were published when this study commenced, they are complementary in building an evidence base for future research to anticipate an adequate timeline.

While co-design and other co-production processes are resource and time-intensive, the investment is essential to prioritise the health and other needs of potentially vulnerable population groups in the context of an imbalance of power [ 42 ]. In exploring the concept of dignity for people with disability, Chapman et al. [ 34 ] indicated that recognising the right to make decisions and proactively eliminating or minimising barriers to inclusion are key to protecting dignity. Community participation in decision-making processes such as this study can result in messy and unpredictable outcomes. However, the onus must be placed on policymakers, organisations, and academia to acknowledge this sufficiently rather than demand conformity [ 15 ].

The authors posit that the study would have benefited from an alternative ethics pathway, which may provide additional required flexibility while upholding the rigour of the ethical review process. The increasing frequency of participatory research studies indicates that challenges experienced by the authors of this study are unlikely to be isolated. Lloyd [ 43 ] described challenges regarding information gathered in-between, before and after structured research sessions, reflecting that they relied on personal judgement of the intent to consent for research use. Similarly, Rowley [ 44 ] reflected on the ethical complexities of interacting with families and respecting their confidentiality within the context of being integrated within an organisation. While these studies were co-production in child protection and education, the ethical challenges of their reflections parallel those experienced in the telepractice study. The risks posed by inadequate ethical support in these contexts are that increased poor ethical outcomes will occur, especially in the in-between times of co-design. Therefore, an ethics pathway that involves more frequent brief liaisons with a designated ethics representative to update project progress and troubleshoot ethical considerations may better support researchers to safeguard study participants.

We believe the decision to complete a sequential workshop series with a consistent group of diverse co-designers, led by co-facilitators, was a strength of the co-design process implemented in the telepractice re-design project. The group worked together across a series of workshops, which enabled them to build solid working relationships. Pearce et al. [ 5 ], Rahman et al. [ 16 ] and Tindall et al. [ 13 ] also demonstrated a collaborative whole-team approach to co-design. By contrast, studies that involved separate workshops with different cohorts or multiple of the same workshop did not demonstrate strong collaboration between co-designers [ 18 , 19 , 20 ]. Nesbitt et al. [ 19 ] explicitly highlighted that they would improve their method by completing sequential workshops with a continuous cohort. Stephens et al. [ 45 ] found that small mixed groups were not sufficient to support the participation of people with disability, indicating that the choice to intentionally balance groups to meet the lived experience expert co-designer’s needs may have been an impacting factor on our success.

A lack of clarity in the terminology used in co-design and community-based participatory practice was identified during the completion of this study. We found that co-design frequently meant either a collaborative design process or good participatory practices [ 46 ]. When viewing the structure of the telepractice re-design project, the overarching research approach was community-based participatory-research, and the method was co-design [ 9 ]. The delineation between the overarching approach and methods clarifies the misappropriation of the term co-design with the intent of meaning public participation [ 46 ] rather than the joint process of creative thinking and doing to design an output [ 11 ]. The use of the two-level structure appears more prominent in the United Kingdom, whereas Fox et al. [ 47 ] systematic review assessing public or patient participants identified that 60% of studies originated from the United Kingdom, compared to the next highest 16% for Canada or 4% from Australia and the United States. To improve clarity and reduce confusion about the terminology used, the authors advocate for greater awareness and implementation of the delineation between the concepts of a community-based-participatory-research/patient or public involvement approach versus the co-design method.

An example of co-design being used where alternate terms such as community-based participatory processes (or research) may be more relevant was the most recent amendment to the act governing the NDIS under which this project resided [ 48 ]. The term co-design could be interpreted as an intent to collaborate with people with disability for equitable involvement in all aspects of the NDIS [ 48 ]. It is proposed that the differentiation of these terms would assist in clarifying the intent of the study and dissuade inaccurate expectations of community involvement or design processes.

Implementing community-based participatory research has demonstrated the potential to create an impact that expands further than the original aim of the study. The skills learned by co-designers, the learning of the research team in collaboration with people with disability, the engagement and skill-building of a peer researcher with lived experience, the organisations who engaged in the co-design process and the academic and lay people who engaged with research outputs, all carry a piece of the impact of the co-design process. Rahman et al. [ 16 ] contend that co-design processes positively impact communities. In the context of this study, the peer researcher was included in the National Disability Insurance Agency’s quarterly report as an example of strength-based employment opportunities, which significantly positively impacted his career prospects [ 49 ]. This project provided skills for people with disability that they value and improved the clinical practice of clinician co-designers, which echoes the conclusions of Ramos et al. [ 15 ], who described that participants felt valued and experienced improved self-esteem. There is additional intent from the authors to positively impact disability providers and academia, to advocate for greater collaboration, and to provide open-access publications to provide a stronger evidence base for co-design in clinical practice and service delivery.

Strengths and limitations

The study provides reflective evidence to support the challenges and benefits experienced during the implementation of the study. However, a limitation in the project’s design was the exclusion of outcome measures to assess the impact of process design choices directly. Stephens et al. [ 45 ] completed targeted outcome measures correlating to accessibility adaptations in co-design and conceded that the variability of findings and individual needs reduced the usefulness of these measures.

The reduction of project scope enabled the completion of the study within the limitations of budgeting and timeline restrictions. Although the scope of the project had some flexibility, there were limitations to how far this could be extended as resources were not infinite, and staffing changes meant that organisational priorities changed. Including implementation and evaluation would have improved the study’s rigour. However, Rocky Bay now has the opportunity to implement internally without potential research delays and restrictions.

The blended and flexible approach to the co-design process was a strength of the study as it met the co-designers needs and maximised the project’s potential inclusivity. This strength has the potential to positively impact other studies that can modify some of the process design choices to suit their context and increase inclusivity [ 11 ]. It is believed that the messiness of co-design is important in meeting the needs and context of each individual study; therefore, no two co-design processes should look the same.

The authors concede that the inclusion of a cohort of people with disability and clinical staff does not represent the entirety of their communities, and their proposed changes may cause some parts of the disability community to experience increased barriers [ 50 ]. It is important to note that while the co-designers who participated in this project provided initial design developments, future opportunities remain to iterate the proposed telepractice service and continue to advocate for equitable access for all.

Recommendations for future studies

Recommendations from this study fall into two categories: recommendations for those intending to utilise the described methods and recommendations for future avenues of research inquiry. For those intending to implement the methods, the primary recommendations are to build ample time buffers into the project schedule, implement key role succession planning and set remuneration agreements at the outset, and work together as partners with the mindset that all contributors are creative [ 51 ] with important expertise and invaluable insights if supported appropriately.

Regarding avenues for future inquiry, we recommend investigating a more dynamic and flexible ethics process that may utilise more frequent short consultations to respond to ethical considerations during the emergent co-design and participatory research.

In the authors’ opinion, supported by co-designers experiences, co-design is a useful and outcome-generating methodology that can proactively enable the inclusion of people with disability and service providers in a community-based participatory research approach. The process is both time and resource-intensive; however, in our opinion, the investment is justified through the delivery of direct research benefits and indirect wider community benefits. We advocate for using community-based participatory-research/processes paired with co-design to generate creative thinking within service design processes. Through co-design processes, we recommend collaborating with a single diverse group of co-designers who have the time and space to build trusting working relationships that enable outputs representative of the group consensus.

Data availability

The dataset supporting the conclusions of this article is predominantly included within the article (and its additional files). However, due to the small number of co-designers reflecting upon the research, despite deidentification, there is a reasonable assumption of identification; therefore, the reflection activity response supporting data is not available.

Abbreviations

Australian Dollar

Guidance for Reporting Involvement of Patients and the Public 2 Checklist

Human Research Ethics Committee

Doctor of Philosophy

Patient and Public Involvement

Microsoft Teams

National Disability Insurance Scheme

McKercher KA. Beyond Sticky Notes doing co-design for Real: mindsets, methods, and movements. 1 ed. Sydney, NSW: Beyond Sticky Notes; 2020. p. 225.

Google Scholar  

Mullins RM, Kelly BE, Chiappalone PS, Lewis VJ. No-one has listened to anything I’ve got to say before’: co-design with people who are sleeping rough. Health Expect. 2021;24(3):930–9. https://doi.org/10.1111/hex.13235 .

Article   PubMed   PubMed Central   Google Scholar  

Ekman I, Swedberg K, Taft C, Lindseth A, Norberg A, Brink E, et al. Person-centered Care — Ready for Prime Time. Eur J Cardiovasc Nurs. 2011;4248–51. https://doi.org/10.1016/j.ejcnurse.2011.06.008 . [cited 3/9/2022];10.

National Commission on Safety and Quality in Healthcare. Partnering with Consumers Standard. Australia: National Commission on Safety and Quality in Healthcare. 2021. https://www.safetyandquality.gov.au/standards/nsqhs-standards/partnering-consumers-standard .

Pearce T, Maple M, McKay K, Shakeshaft A, Wayland S. Co-creation of new knowledge: good fortune or good management? Res Involv Engagem. 2022;8(1):1–13. https://doi.org/10.1186/s40900-022-00394-2 .

Article   Google Scholar  

Bordeaux BC, Wiley C, Tandon SD, Horowitz CR, Brown PB, Bass EB. Guidelines for writing manuscripts about community-based participatory research for peer-reviewed journals. Prog Community Health Partnersh. 2007;1(3):281–8. https://doi.org/10.1353/cpr.2007.0018 .

Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. Res Involv Engagem. 2017;3(1):1–11. https://doi.org/10.1186/s40900-017-0062-2 .

Ostrom E, Baugh W, Guarasci R, Parks R, Whitaker G. Community Organization and the Provision of Police Services. Sage; 1973.

Masterson D, Areskoug Josefsson K, Robert G, Nylander E, Kjellström S. Mapping definitions of co-production and co-design in health and social care: a systematic scoping review providing lessons for the future. Health Expect. 2022;25(3):902–13. https://doi.org/10.1111/hex.13470 .

Bibb J. Embedding lived experience in music therapy practice: Towards a future of co-designed, co-produced and co-delivered music therapy programs in Australia. Australian Journal of Music Therapy [Journal Article]. 2022 [cited 2023/08/21];33(2):25–36. https://doi.org/10.3316/informit.829441047529429 .

Davis A, Gwilt I, Wallace N, Langley J. Low-contact Co-design: considering more flexible spatiotemporal models for the co-design workshop. Strategic Des Res J. 2021;14(1):124–37. https://doi.org/10.4013/sdrj.2021.141.11 .

Claborn KR, Creech S, Whittfield Q, Parra-Cardona R, Daugherty A, Benzer J. Ethical by design: engaging the community to co-design a Digital Health Ecosystem to Improve Overdose Prevention efforts among highly vulnerable people who use drugs. Front Digit Health [Original Research]. 2022;4:1–13. https://doi.org/10.3389/fdgth.2022.880849 .

Tindall RM, Ferris M, Townsend M, Boschert G, Moylan S. A first-hand experience of co‐design in mental health service design: opportunities, challenges, and lessons. Int J Ment Health Nurs. 2021;30(6):1693–702. https://doi.org/10.1111/inm.12925 .

Article   PubMed   Google Scholar  

Wahlin DW, Blomkamp DE. Making global local: global methods, local planning, and the importance of genuine community engagement in Australia. Policy Des Pract. 2022;5(4):483–503. https://doi.org/10.1080/25741292.2022.2141489 .

Ramos M, Forcellini FA, Ferreira MGG. Patient-centered healthcare service development: a literature review. Strategic Des Res J. 2021;14(2):423–37. https://doi.org/10.4013/sdrj.2021.142.04 .

Rahman A, Nawaz S, Khan E, Islam S. Nothing about us, without us: is for us. Res Involv Engagem. 2022;8(1):1–10. https://doi.org/10.1186/s40900-022-00372-8 .

Harrison R, Manias E, Ellis L, Mimmo L, Walpola R, Roxas-Harris B, et al. Evaluating clinician experience in value-based health care: the development and validation of the Clinician experience measure (CEM). BMC Health Serv Res. 2022;22:1–11. https://doi.org/10.1186/s12913-022-08900-8 .

Kerr JAS, Whelan M, Zelenko O, Harper-Hill K, Villalba C. Integrated Co-design: a model for co-designing with multiple stakeholder groups from the ‘Fuzzy’ front-end to Beyond Project Delivery. Int J Des. 2022;16(2):1–17. https://doi.org/10.57698/v16i2.06 .

Nesbitt K, Beleigoli A, Du H, Tirimacco R, Clark RA. User experience (UX) design as a co-design methodology: lessons learned during the development of a web-based portal for cardiac rehabilitation. Eur J Cardiovasc Nurs. 2022;21(2):178–83. https://doi.org/10.1093/eurjcn/zvab127 .

Marwaa MN, Guidetti S, Ytterberg C, Kristensen HK. Using experience-based co-design to develop mobile/tablet applications to support a person-centred and empowering stroke rehabilitation. Res Involv Engagem. 2023;9(1):1–17. https://doi.org/10.1186/s40900-023-00472-z .

Tariq S, Grewal EK, Booth R, Nat B, Ka-Caleni T, Larsen M, et al. Lessons learned from a virtual community-based Participatory Research project: prioritizing needs of people who have diabetes and experiences of homelessness to co-design a participatory action project. Res Involv Engagem. 2023;9(1):1–11. https://doi.org/10.1186/s40900-023-00456-z .

Abimbola S, Li C, Mitchell M, Everett M, Casburn K, Crooks P, et al. On the same page: co-designing the logic model of a telehealth service for children in rural and remote Australia. Digit Health. 2019;5:2055207619826468–2055207619826468. https://doi.org/10.1177/2055207619826468 .

Rocky Bay. Rocky Bay Annual Report FY 2021–2022. Perth. 2022. https://www.rockybay.org.au/wp-content/uploads/2022/12/Rocky-Bay-Annual-Report-21-22.pdf .

National Disability Insurance Agency. What is the NDIS? [Internet]. 2021 [updated 14.08.2021. https://www.ndis.gov.au/understanding/what-ndis .

Reen G, Page B, Oikonomou E. Working as an embedded researcher in a healthcare setting: a practical guide for current or prospective embedded researchers. J Eval Clin Pract. 2022;28(1):93–8. https://doi.org/10.1111/jep.13593 .

Bell S, Aggleton P, Gibson A. Peer Research in Health and Social Development 1st Edition ed. London: Routledge; 2021. p. 286.

Book   Google Scholar  

Curran T, Jones M, Ferguson S, Reed M, Lawrence A, Cull N, et al. Disabled young people’s hopes and dreams in a rapidly changing society: a co-production peer research study. Disabil Soc. 2021;36(4):561–78. https://doi.org/10.1080/09687599.2020.1755234 .

Kelly B, Friel S, McShane T, Pinkerton J, Gilligan E. I haven’t read it, I’ve lived it! The benefits and challenges of peer research with young people leaving care. Qualitative Social work: QSW: Res Pract. 2020;19(1):108–24. https://doi.org/10.1177/1473325018800370 .

Schwartz AE, Kramer JM. Inclusive approaches to developing content valid patient-reported outcome measure response scales for youth with intellectual/developmental disabilities. Br J Learn Disabil. 2021;49(1):100–10. https://doi.org/10.1111/bld.12346 .

Webb P, Falls D, Keenan F, Norris B, Owens A, Davidson G, et al. Peer researchers’ experiences of a co-produced research project on supported decision-making. Res Involv Engagem. 2022;8(1):1–10. https://doi.org/10.1186/s40900-022-00406-1 .

People with Disability Australia. PWDA Language Guide: A guide to language about disability. Sydney, Australia. 2021. https://pwd.org.au/wp-content/uploads/2021/12/PWDA-Language-Guide-v2-2021.pdf .

Peters MDJGC, McInerney P, Munn Z, Tricco AC, Khalil H. Chapter 11: Scoping Reviews (2020 version). In: Aromataris E MZ, editor. JBI Manual for Evidence Synthesis, JBI, 2020: JBI; 2020.

Australian Broadcasting Commission. ‘My purpose is changing perceptions’: Australian of the Year Dylan Alcott’s speech in full [Internet]. 2022 [cited 17.08.2023]. https://www.abc.net.au/news/2022-01-26/dylan-alcott-australian-of-the-year-speech-in-full/100783308 .

Chapman K, Dixon A, Ehrlich C, Kendall E. Dignity and the importance of acknowledgement of Personhood for people with disability. Qual Health Res. 2024;34(1–2):141–53. https://doi.org/10.1177/10497323231204562 .

Flattery S. Stim Joy: Using Multi-Sensory Design to Foster Better Understanding of the Autistic Experience: ProQuest Dissertations Publishing; 2023.

Peak Learning. The Real Deal [Internet]. 2023 [cited 6.10.2023]. https://www.peaklearning.com/trd/ .

Benz C, Scott-Jeffs W, Revitt J, Brabon C, Fermanis C, Hawkes M, et al. Co-designing a telepractice journey map with disability customers and clinicians: partnering with users to understand challenges from their perspective. Health Expect. 2023;1–11. https://doi.org/10.1111/hex.13919 .

Flowers E, Miller ME. Your Guide to Blueprinting The Practical Way. 1 ed. USA: Practical By Design 2022. 134 p. pp. 1-134.

Blomkvist J. Benefits of Service Level Prototyping. Des J. 2016;19(4):545–64. https://doi.org/10.1080/14606925.2016.1177292 .

Harrison R, Ní Shé É, Debono D, Chauhan A, Newman B. Creating space for theory when codesigning healthcare interventions. J Eval Clin Pract. 2023;29(4):572–5. https://doi.org/10.1111/jep.13720 .

Bodden S, Elliott J. Finding space for Shared futures. Edinb Archit Res. 2022;37:90–104.

Page K. Ethics and the co-production of knowledge. Public Health Research & Practice. 2022:1–5. https://www.phrp.com.au/issues/june-2022-volume-32-issue-2/ethics-and-co-production/ .

Lloyd J. Life in a lanyard: developing an ethics of embedded research methods in children’s social care. J Children’s Serv. 2021;16(4):318–31. https://doi.org/10.1108/JCS-12-2019-0047 . [cited 2023/12/05];.

Rowley H. Going beyond procedure:engaging with the ethical complexities of being an embedded researcher. Manage Educ. 2014;28(1):19–24. https://doi.org/10.1177/0892020613510119 .

Stephens L, Smith H, Epstein I, Baljko M, McIntosh I, Dadashi N, et al. Accessibility and participatory design: time, power, and facilitation. CoDesign. 2023;1–17. https://doi.org/10.1080/15710882.2023.2214145 .

Gardner G, McKercher KA. But is it co-design? And if it is, so what? 2021. https://healthvoices.org.au/issues/nov-2021/but-is-it-co-design-and-if-it-is-so-what .

Fox G, Lalu MM, Sabloff T, Nicholls SG, Smith M, Stacey D, et al. Recognizing patient partner contributions to health research: a systematic review of reported practices. Res Involv Engagem. 2023;9(1):1–30. https://doi.org/10.1186/s40900-023-00488-5 .

National Disability Insurance Agency. 2022 NDIS legislation amendments Australia; 2022. https://www.ndis.gov.au/news/7975-2022-ndis-legislation-amendments-july-update .

National Disability Insurance Agency. Report to disability ministers for Q4 of Y10 Summary Part A Australia. 2023. https://www.ndis.gov.au/about-us/publications/quarterly-reports .

Lid IM. Universal Design and disability: an interdisciplinary perspective. Disabil Rehabil. 2014;36(16):1344–9. https://doi.org/10.3109/09638288.2014.931472 .

Sanders E, Stappers PJ. Co-creation and the New landscapes of Design. CoDesign. 2008;4:5–18. https://doi.org/10.1080/15710880701875068 .

Download references

Acknowledgements

The authors acknowledge the contribution of Rocky Bay as the industry partner of this project and would like to thank the Co-designers of this project, without whom none of this was possible. The research team would also like to thank Katie Harris for her time and support throughout the workshop series, which were invaluable to the completion of the project and the formation of the published study.

The article forms part of a PhD project funded by the first author, CB’s Australian Government Research Training Program (RTP) scholarship.

Author information

Authors and affiliations.

School of Population Health, Curtin University, Bentley, Australia

Cloe Benz, Richard Norman, Delia Hendrie & Suzanne Robinson

Rocky Bay, Mosman Park, WA, Australia

Will Scott-Jeffs, Mai Welsh & Matthew Locantro

Beyond Sticky Notes, Sydney, Australia

K. A. McKercher

Therapy Focus, Bentley, Australia

Deakin Health Economics, Institute for Health Transformation, Deakin University, Melbourne, Australia

Suzanne Robinson

You can also search for this author in PubMed   Google Scholar

Contributions

CB and MW liaised with the steering committee and conceived the study and structure. SR, DH and RN guided the protocol development and ethics approval. KAM provided methodological support to the project and subject matter expertise. CB and WJS completed participant recruitment, facilitation of workshops and data collection. KAM and CB ideated the format and content of the article. CB completed data analysis and wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved of the final version of the manuscript.

Corresponding author

Correspondence to Cloe Benz .

Ethics declarations

Ethical approval and consent.

The study was approved by the Curtin University Human Research Ethics Committee (ID# HRE2021-0731), and all participants provided written informed consent before engaging in any research activity.

Consent for publication

Not applicable.

Competing interests

Cloe Benz, Richard Norman, Delia Hendrie & Suzanne Robinson do not have any competing interests to declare. Will Scott-Jeffs, Matthew Locantro and Mai Welsh, for all or part of the study period were employed by Rocky Bay a Not-For-Profit Disability Service provider who function as the industry partner for the project. K.A. McKercher is the author of a co-design method book referenced in the article. McKercher also runs a business that helps people co-design.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1:

Appendix 1–3

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Benz, C., Scott-Jeffs, W., McKercher, K.A. et al. Community-based participatory-research through co-design: supporting collaboration from all sides of disability. Res Involv Engagem 10 , 47 (2024). https://doi.org/10.1186/s40900-024-00573-3

Download citation

Received : 13 November 2023

Accepted : 12 April 2024

Published : 10 May 2024

DOI : https://doi.org/10.1186/s40900-024-00573-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Community-based participatory-research
• Telepractice
• Lived experience
• Embedded researcher
• Digital health
• Patient and public involvement

Research Involvement and Engagement

ISSN: 2056-7529

• General enquiries: [email protected]

• Open access
• Published: 14 May 2024

A phase I/II clinical trial of ex-vivo expanded human bone marrow derived allogeneic mesenchymal stromal cells in adult patients with perianal fistulizing Crohn’s Disease

• Shekhar Swaroop 1   na1 ,
• Sudheer Kumar Vuyyuru 1   na1 ,
• Bhaskar Kante 2 ,
• Peeyush Kumar 1 ,
• Sandeep Kumar Mundhra 1 ,
• Umang Arora 1 ,
• Ankur Goyal 3 ,
• Devasenathipathy Kandasamy 3 ,
• Raju Sharma 3 ,
• Kavirajan Kabilan 3 ,
• Saurabh Kedia 1 ,
• Nihar Ranjan Dash 4 &
• Vineet Ahuja 1  

Stem Cell Research & Therapy volume  15 , Article number:  140 ( 2024 ) Cite this article

7 Altmetric

Metrics details

Perianal fistulas (PF) affect one-third patients with Crohn’s disease (CD) with limited therapeutic options. There is dearth of literature on safety and efficacy of bone marrow-derived mesenchymal stromal cells (BMSCs) in this population.

An open-label, phase I/II, single-arm study was conducted involving local administration of human allogeneic bone marrow-derived mesenchymal stromal cells in perianal fistula of patients with Crohn’s disease refractory to standard therapies. Clinical severity and biomarkers were assessed at baseline and periodically until week 104 , and MRI at week 24 and 104. Primary and secondary objectives were to assess safety and efficacy respectively. Fistula remission was complete closure of fistula openings with < 2 cm perianal collection on MRI, and fistula response was decrease in drainage by ≥ 50%. Change in perianal disease activity index, quality-of-life and Van Assche index on MRI over time was assessed using mixed-effect linear regression model.

Ten patients (male:8, mean age:27.4 ± 12.0years) were recruited. Self-resolving procedure-related adverse events occurred in three patients, with no follow-up adverse events. In intention to treat analysis at week 24, two patients (20%) achieved fistula remission and seven (70%) had fistula response. At week 52, two (20%) patients were in remission and seven (70%) maintained response. At 104 weeks, two (20%) patients maintained response and one (10%) was in remission. Statistically significant decrease in perianal disease activity index ( P  = 0.008), Van Assche Index ( P  = 0.008) and improvement in quality-of-life ( P  = 0.001) were observed over time.

Conclusions

Allogeneic BMSCs are safe and effective for the treatment of perianal fistulizing CD with significant improvement in clinical severity and radiological healing.

Trial registration

The study was prospectively registered on Clinical trials registry – India (CTRI), CTRI/2020/01/022743 on 14 January 2020, http://ctri.nic.in .

Crohn’s disease (CD) is a chronic, multifactorial, immune mediated disease of the gastrointestinal (GI) tract characterized by stricturing and penetrating complications. Perianal fistula is one of the debilitating complications associated with considerable morbidity in patients with CD. Although, varying prevalence was reported in studies from different geographical regions across the world, approximately one-fifth of patients with CD are affected by perianal fistula at the time of diagnosis and one-third at 10 years following diagnosis [ 1 ]. Effective treatment options for treating perianal fistulizing CD are limited. Although various surgical techniques are available for the treatment of perianal CD, medical therapy remains cornerstone in the management to achieve and maintain remission [ 2 ]. However, despite the availability of multiple advanced medical therapies such as biologics and oral small molecules for the management of luminal CD, anti-TNF therapy was the only biological therapy that was systematically evaluated in phase 3 randomized controlled trials (RCTs) primarily designed for patients with perianal CD and remains preferred treatment of choice [ 3 ].

Mesenchymal stem cell (MSCs) therapy has shown to be safe and effective in patients with perianal CD in various studies with sustained long-term response and is a valuable addition to the existing therapeutic armamentarium for the management of perianal CD [ 4 , 5 , 6 ]. MSCs are multi-potent, spindle-like cells that possess the ability to self-renew as well as to differentiate into cartilage, bone and fat tissues in vitro [ 7 ]. MSCs exhibit unique immunomodulatory properties by suppressing T cell activation and proliferation, dendritic cell differentiation, maturation and function, B cell function, and natural killer cell proliferation [ 8 ]. MSCs can be allogenic or autologous and can be obtained from various tissues, such as adipose tissue and bone marrow [ 9 , 10 ]. A large phase 3 RCT demonstrated statistically significant fistula response with human adipose tissue-derived MSCs (AMSCs) compared to placebo which led to approval of MSCs by European Medical Agency (EMA) as an orphan indication [ 6 ]. However, MSCs have been still undergoing evaluation in other regions of the world. Unlike AMSCs, efficacy of Bone marrow derived MSCs (BMSCs) has not been adequately investigated especially in Asia [ 11 , 12 , 13 , 14 ]. To the best of our knowledge, there are only four studies with small sample size available which evaluated safety and efficacy of BMSCs in adult patients with perianal fistulizing CD and none were conducted in Asian population [ 15 , 16 , 17 , 18 ]. Patients in Asian countries are genetically distinct with difference in gene polymorphisms which could potentially affect disease phenotype and response to therapy [ 19 ]. Majority of the studies evaluating efficacy of MSCs were conducted in European countries and these findings may not be directly applicable to individuals with perianal fistulizing CD in Asian populations. Therefore, it is crucial to assess safety of efficacy of MSCs which could potentially be useful in this population where there is limited availability of advanced therapies. Hence this phase I/II trial was undertaken to assess the safety and efficacy of local administration of human BMSCs in adult patients with perianal fistulizing CD.

Study design

An open label, single arm study was conducted for a total duration of 104 weeks (2 years), with the primary objective to assess the safety of local administration of adult human bone marrow derived, cultured, pooled, allogeneic mesenchymal stromal cells (BMSCs) in patients with perianal fistulizing CD. The secondary objective was to evaluate their efficacy by clinical and radiological assessments. The study was conducted in compliance with the protocol, the ethical principles that have their origin in the Declaration of Helsinki, the International Conference on Harmonization (ICH) consolidated Guideline E6 for Good Clinical Practice (GCP) (CPMP/ICH/135/95) and in accordance to “Guidelines for Stem Cell Research and Therapy” by Department of Biotechnology and Indian Council of Medical Research (ICMR), 2017, Schedule-Y and ICH-GCP and as per the recommendations of the Cellular Biology Based Therapeutic Drug Evaluation Committee (CBBTDE). The trial protocol was approved by institutional ethics committees and institutional stem cell committee (Ref No-IC-SCR/94/19) and is registered under clinical trials registry – India (CTRI No. CTRI/2020/01/022743). The confidentiality of all patients taking part in the study was preserved in accordance with GCP and local regulations. All patients provided written as well as audio-visual consent for participation in the study. The study recruitment began in February 2020 and completed in June 2022. Due to novel Coronavirus (SARS CoV-2) pandemic, for some follow up visits, patients could not visit hospital and hence the assessment was done telephonically for those visits. (Supplementary Table 1 )

Patient selection

Eligible patients were of either sex, aged between 18 and 65 years, with complex perianal fistulae associated with CD of at least 3 months duration, an active draining fistula with a maximum of 1 internal opening and a maximum of 2 external openings, that was refractory to medical (antibiotics, immunomodulators, or biologics) or surgical therapy. Patients were excluded if they had Crohn’s Disease Activity Index (CDAI) score more than or equal to 220 points, received steroids within 1 month prior to enrolment, treatment naïve fistulas, perianal abscess larger than 2 cm in diameter on magnetic resonance imaging (MRI) of the pelvis, presence of proctitis, anal canal stricture and fistulas other than perianal fistulas. Crohn’s disease was diagnosed as per the ECCO guidelines and were classified into various phenotypes using Montreal classification which includes age at onset, location and behaviour of the disase [ 20 , 21 ] The investigations used for diagnosis included CT Enterography, ileocolonoscopy, and biopsy from abnormal mucosa. CT enterography was done for evaluation of small bowel in all patients. MR Pelvis was done for evaluation of the perianal fistula and presence of any perianal abscess or collection.

For the evaluation of perianal disease activity, PDAI score was used which includes variables like fistula discharge, pain/ restriction of activities, restriction of sexual activity, type of perianal disease, and degree of induration [ 22 ]. For the evaluation of luminal activity, CDAI score was used which includes the following variables: number of liquid stools, abdominal pain, general well-being, presence of extraintestinal complications, use of antidiarrheal drugs, presence of abdominal mass, body weight, and haematocrit [ 22 ].

For the evaluation of quality of life, a questionnaire comprising 5 questions and visual analogue scale was used and was rated from 0 to 100 with 0 being the worst control and 100 being the best control [ 23 ]. For the evaluation of radiological response, Van Assche index which included six MRI pelvis parameters: number of fistula tracts, fistula location and extension, T2 hyperintensity of the tract, presence or absence of collections and rectal wall involvement [ 24 ].

Investigational Medical product (IMP)

STEMPEUCEL® is a suspension of 25 million ex vivo expanded, adult human bone marrow derived, cultured, pooled, allogeneic mesenchymal stromal cells (MSCs) formulated in CS5 medium and CZ vials. These MSCs were manufactured by Stempeutics Research Pvt. Ltd, Bengaluru, Karnataka, India, and registered as an Investigational medical product (IMP). The IMP was transported from the laboratory to the operating theatres of All India Institute of Medical Sciences (AIIMS), New Delhi, India in cryovial, in a temperature-controlled transport container containing a dry shipper that was stored in the transport container at -185 °C to -196 °C.

Administration of mesenchymal stromal cells

Before scheduling the administration, pre-medication with intravenous injection of 100 mg hydrocortisone and 45.5 mg of pheniramine maleate was administered, and administration of BMSCs was completed within 60 min of administration of first premedication. BMSC injections were administered locally through the perianal route under spinal anaesthesia. 75 million cells (15 mL cell suspension containing 5 × 10 6 cells/mL) were provided through intralesional injection. Fistula tract was curetted, and internal opening was identified before administration of BMSCs. Internal opening was closed with absorbable sutures and 5 ml of cell suspension containing 25 million cells was injected at internal fistula opening. Remaining 10 ml cell suspension containing approximately 50 million cells was injected with 20 gauge long hypodermic needle along the walls of fistula tract so that it should produce a 2 mm bleb. Only one session of MSCs administration was done. Administration of BMSCs is depicted in Fig.  1 . Patients were admitted in the hospital for 48 h after administration to monitor for acute local or systemic side effects.

Administration of stem cells in perianal fistula

Baseline screening and follow up

The screening visit included clinical examination, vitals recording, blood tests, sigmoidoscopy or colonoscopy, and an MRI pelvis. BMSC injection was administered within 2 weeks of the screening visit. Crohn’s disease activity index (CDAI), Perianal Disease Activity Index (PDAI), quality of life (assessed as visual analogue scale), and evaluation of adverse events were assessed for all participants at baseline as well as at 2, 6, 12, 18, 24, 52 and 104 weeks. MRI Pelvis and sigmoidoscopy or colonoscopy were repeated at 24 weeks and 104 weeks during follow up. Details of follow up visit is provided in Supplementary Table 1 .

Outcome measures

An adverse event was defined as any untoward medical occurrence in a patient administered IMP and which did not necessarily have a causal relationship with treatment at weeks 0, 12, 24, 52, 104 and has been graded as per Common terminology criteria for adverse events (CTCAE) V5. Fistula remission was defined as complete closure of all external openings and no collections larger than 2 cm on pelvic MRI at weeks 24 and 104. Fistula response was defined as closure of more than 50% of all openings or a decrease in fistula discharge by ≥ 50%. Change in PDAI, and quality-of-life was assessed at weeks 24, 52 and 104. Quality-of-life was assessed as a visual analogue scale (VAS) ranging from 0 to 100 (worst to best) [ 23 ]. Change in Van Assche index (VAI) was assessed at week 24 and 104 weeks [ 24 ].

Statistical analysis

Statistical analysis was performed using standard methods. Continuous variables that were normally distributed were expressed as mean ± standard deviation(SD), otherwise expressed as median (range). Categorical data were presented as proportions. Changes in the four outcome measures (PDAI, QOL score, CDAI and Van Assche Index score) over time was assessed using mixed-effect linear regression model. Wilcoxon sign rank test was used to compare median measurements of these four outcomes for each pair of follow-up duration. A p-value < 0.05 was considered statistically significant. Statistical analysis was performed using Stata v14 (StataCorp, Texas, USA).

Role of the funding source

This study was supported and funded by Stempeutics Research Pvt. Ltd, Bengaluru, Karnataka. The funder of the study had no role in the data collection, data analysis, data interpretation, or writing of the report.

Baseline characteristics

Ten CD patients with actively draining perianal fistula (eight males, mean age − 27.4 ± 12.0 years) were recruited after satisfying eligibility criteria. Median disease duration was 7.5 (IQR: 2.5–21.0) years. All patients failed medical therapy and six patients failed both medical and surgical therapies prior to recruitment. None of the patients were on anti-TNF therapy at the time of recruitment. All 10 patients received multiple courses of antibiotics for variable duration ranging from 1 month to 10 years without any response in terms of fistula healing. Commonly used antibiotics were Ciprofloxacin, Ofloxacin, Metronidazole, and Satranidazole. Seven out of 10 patients also had received immunomodulators in the form of Azathioprine for a duration ranging from 6 months to 6 years. Eight patients also have been on multiple courses of steroids including budesonide and prednisolone. Biological therapy with Infliximab was received by 5 patients and 1 patient received both infliximab and adalimumab. Details of previous therapy can be found in the Table  1 . Majority had colonic involvement (5/10) and none of the patients had non-perianal fistulae. One patient had exclusive perianal involvement without significant bowel involvement. All patients who were recruited had received multiple courses of antibiotics in the past. Baseline characteristics of the patients are shown in Table  1 . Clinical course of patients during the study is summarized in a Swimmer’s plot in Fig.  2 .

Swimmer’s plot: Outcome measures over time

On clinical assessment at baseline, eight out of ten patients had one external opening and two patients had two external openings. The median PDAI score at baseline was 9 (IQR:7–9), the median IBD-QOL (VAS) score was 30 (IQR: 20–30) and the median CDAI score was 66 (IQR: 50–102). (Table  1 )

On radiological assessment at baseline, four patients had inter-sphincteric, one patient had extra-sphincteric, two patients had trans-sphincteric, and three patients had supra-sphincteric location of fistula. Seven patients had presence of collections which were less than 2 cm. Seven patients had moderate and three had mild T2 hyperintensity on MRI. Three patients had rectal wall thickening despite no evidence of active proctitis on sigmoidoscopy. Mean Van Assche Index was 15.1 ± 5.4. (Table  2 )

Safety of BMSCs

Three out of the ten patients had periprocedural adverse events in the form of post spinal headache in two patients which was considered unrelated to the BMSCs injection. One patient had perianal ecchymoses and urinary retention following procedure which resolved without requiring any medical of surgical intervention. Seven patients had perianal pain which required analgesics. All events were mild in severity. One patient with no response to stem cell therapy, on dual biological therapy (Infliximab and Ustekinumab) developed left iliac fossa abscess at 100 weeks and was treated with antibiotics and drainage of abscess. Adverse events during procedure and follow up are shown in Tables  3 and 4 .

Efficacy of BMSCs

Clinical assessment.

A total of ten patients were recruited, one of whom was lost to follow up after week 24. Remaining nine patients had completed 52 weeks of follow up and five patients had completed week 104 follow up.

At week 24, two (20%) patients achieved fistula remission and seven (70%) achieved fistula response and none of these patients received any concomitant biological therapy or surgical drainage. Among the three patients who did not achieve fistula response at week 24 were managed with different therapeutic strategies. The first patient was started on biological with Infliximab at 26 weeks and subsequently upgraded to a combination of Infliximab and Ustekinumab at 95 weeks. The second patient was started on Azathioprine at 98 weeks and Ustekinumab at 126 weeks. The third patient underwent seton placement and fistulectomy at 24 weeks. Among the seven patients who achieved response at 24 weeks, 1 patient was started on Azathioprine at 52 weeks with a course of antibiotics. One patient required the addition of a course of antibiotics during follow-up. The rest of 5 patients did not receive any concomitant therapy (antibiotics, immunomodulators, steroids or biologicals) during follow up. On per protocol analysis at week 24, two patients (20%) achieved fistula remission and seven (70%) fistula response. At week 52, one patient who had remission at 24 weeks relapsed, one patient maintained remission, one more patient achieved remission, hence two out of nine (22%) patient were in remission and seven out of nine (78%) maintained response. At 104 weeks, two out of six (33%) patients maintained response and one (17%) patient maintained remission. On intention to treat analysis at week 24, two patients (20%) achieved fistula remission and seven (70%) fistula response. At week 52, two out of ten (20%) patient were in remission and seven out of ten (70%) maintained response. At 104 weeks, two out of ten (20%) patients maintained response and one (10%) patient was in remission. Outcome measures are shown in Tables  5 and 6 ; Fig.  2 for the study population.

Patient reported outcome measures

Median PDAI (IQR) scores at baseline, 24 weeks, 52 weeks, and 104 weeks were 9 (7.0–9.0), 4 (3.0–4.0), 4 (3.0–5.0) and 5.5 (3.0–7.0) respectively ( P  = 0.008). Median IBD QOL (VAS) (IQR) scores at baseline, 24 weeks, 52 weeks, and 104 weeks were 30 (20–30),70 (50–80), 60 (50–80) and 70 (50–80) respectively ( P  = 0.001). Median CDAI (IQR) scores at baseline, 24 weeks, 52 weeks, and 104 weeks were 66 (50–102), 74 (44–158), 74 (60–99), and 79 (31–227) ( P  = 0.251). (Table  7 ; Fig.  3 ) One patient had worsening of luminal activity, indicated by an increasing CDAI score, and requiring use of Infliximab at week 52. Evolution per patient of PDAI, IBD-QOL (VAS) score, and CDAI over time is depicted in Fig.  4 .

Change in various scores from baseline to weeks 104

Evolution per patient of the PDAI score, Van Assche Index, CDAI score, QOL score

Radiological assessment

The mean VAI score at baseline was 15.3 ± 5.4 which showed statistically significant decline over time with VAI score of 14.4 ± 5.1 at 24–52 weeks, and 9.5 ± 4.0 at 104 weeks ( P  = 0.008). Seven out of ten patients had pronounced T2 hyperintensity of the fistulous tract at baseline, five out of eight had pronounced T2 hyperintensity at 24–52 weeks and none of the patients had T2 hyperintense fistulous tract at 104 weeks (Table  2 ; Fig.  5 ).

MRI image of a patient in remission ( a ) At baseline - a linear fistulous tract is seen which is hyperintense on T2W sequence, ( b ) at 104 weeks – the fistulous tract becomes hypointense on T2W suggestive of fibrosis

Efficacy and safety of adipose derived MSCs in perianal fistulizing CD has been documented in various studies, and they have shown promise in the management of this complex condition [ 15 , 16 , 17 , 18 ]. However, there is lack of robust data on efficacy of BMSCs. In the present phase I/II study we have demonstrated that BMSCs are safe in patients with CD having complex perianal fistulae after failure of conventional medical and surgical therapies. Apart from safety, BMSCs were also effective in achieving fistula remission and response, and corresponding improvement in quality of life.

Local administration of MSCs is considered safe without any significant increased risk of adverse effects compared to placebo across clinical trials. In a large RCT comparing local administration of AMSCs (ADMIRE-CD), 17% of patients receiving MSCs developed treatment related adverse events compared to 29% in placebo arm at week 24 [ 6 ]. The most common adverse event was perianal abscess, and it was considered to be unrelated to MSCs but instead due to manipulation of perianal tissues. In the long-term follow up of same study, seven out of 40 patients had treatment emergent adverse events through 104 weeks [ 4 ]. Studies on BMSCs also demonstrated no increase in adverse events. In a placebo-controlled trial assessing local administration of allogenic BMSCs with three different doses of stem cells compared with placebo, no serious adverse events were reported except for one perianal abscess event in each group and one patient with positive family history of colorectal cancer receiving MSCs developed caecal carcinoma which was considered unlikely to be a result of stem cell therapy [ 15 ]. In our study, in consistence with results of previously published studies, we did not observe serious adverse events in patients receiving BMSCs. Similarly, a recently published study performed in paediatric patients with perianal fistulizing CD in seven participants did not demonstrate any serious adverse events [ 17 ].

As far as efficacy is concerned, local administration of MSCs demonstrated statistically significant clinical as well as radiological improvement across various studies. In ADMIRE-CD trial, patients randomized to AMSCs arm achieved combined clinical and radiological remission in 50% of patients when compared to 34% in placebo arm at 24 weeks. High response rates in placebo arm could be because of surgical treatment received in placebo arm along with placebo [ 6 ]. Long term follow up also demonstrated sustained remission [ 4 ]. Studies on BMSCs showed varying fistula healing ranging from 20 to 83% [ 15 , 16 , 17 , 18 ]. In our study 70% of patients experienced fistula response and 20% achieved remission at 24 weeks.

Although, all types of MSCs are presumed to have similar properties, several studies have demonstrated considerable differences in immunomodulatory properties [ 25 , 26 ]. Comparative studies on different types of MSCs demonstrated notable differences at molecular level as well as in clinical efficacy between AMSCs and BMSCs [ 27 , 28 , 29 , 30 ]. This suggests that there could be potential therapeutic differences between AMSCs and BMSCs in the management of perianal fistulizing CD which needs to be further explored. Genetic and phenotypic differences in inflammatory bowel disease between Western and Asian population may also influence the efficacy of MSCs [ 31 , 32 , 33 ]. Therefore, our study is a valuable addition to the existing limited literature.

Limitations of the study

First, our study is a single center study with small sample size and majority being males, limiting generalizability of results. Moreover, the genetic background of patients was similar. However, prospective long-term follow-up for two years demonstrated safety of MSCs. Secondly, there was no control arm, therefore comparative efficacy with standard of care was not possible. We did not include patients with more than two external openings, therefore, results of our study many not be applicable to patients with multiple fistula tracts and external openings. All patients did not undergo surgical drainage/seton placement prior to stem cell administration, hence the response when combined with drainage could not be assessed. Furthermore we did not explore the mechanistic property of mesenchymal stromal cells which would involve the measurements of inflammatory cytokines in the serum, rectal tissue and perianal fistula scraping. Lastly, a single dose of MSCs was administered as was the practice in previous clinical trials of MSCs [ 6 , 18 ]; a repeat injection in those who achieved partial response or inadequate response may be required to achieve optimal response. In a recent paediatric study repeat injection of BMSCs after 3 months, if there was no response, led to complete clinical and radiological healing in 83% of patients [ 17 ].

The mechanistic aspects of BMDSc like inflammatory cytokines, change in microbiota were not assessed, which would have made our conclusion more strong.

To conclude, our study findings demonstrated that allogeneic BMSCs are safe and effective in patients with perianal fistulizing CD refractory to conventional therapy.

Data availability

The data can be made available and shared upon reasonable request from the corresponding author depending on the nature of the request and its intended use.

Abbreviations

All India Institute of Medical Sciences

adipose tissue-derived mesenchymal stem cells

bone marrow-derived mesenchymal stem cells

Cellular Biology Based Therapeutic Drug Evaluation Committee

Crohn’s Disease

Crohn’s Disease Activity Index

Coronavirus disease

Clinical trials registry – India

European Medical Agency

Good Clinical Practice

Gastrointestinal

inflammatory bowel disease quality of life

International Conference on Harmonization

Indian Council of Medical Research

Investigational Medical Product

magnetic resonance imaging

mesenchymal stem cells

mesenchymal stem cell therapy

Perianal disease activity index

Randomised control trials

Van Assche index

Brochard C, Rabilloud ML, Hamonic S, Bajeux E, Pagenault M, Dabadie A, et al. Natural history of Perianal Crohn’s Disease: long-term follow-up of a Population-based cohort. Clin Gastroenterol Hepatol. 2022;20(2):e102–10.

Article   PubMed   Google Scholar  

Vuyyuru SK, Sahu P, Kedia S, Kante B, Kumar P, Ranjan MK, et al. Long-term outcomes in perianal fistulizing Crohn’s disease in a resource-limited setting: a cohort analysis. Indian J Gastroenterol. 2020;39(5):435–44.

Sands BE, Anderson FH, Bernstein CN, Chey WY, Feagan BG, Fedorak RN, et al. Infliximab Maintenance Therapy for Fistulizing Crohn’s Disease. N Engl J Med. 2004;350(9):876–85.

Article   CAS   PubMed   Google Scholar  

Garcia-Olmo D, Gilaberte I, Binek M, D´Hoore AJL, Lindner D, Selvaggi F, et al. Follow-up study to evaluate the long-term safety and efficacy of Darvadstrocel (mesenchymal stem cell treatment) in patients with Perianal Fistulizing Crohn’s Disease: ADMIRE-CD phase 3 Randomized Controlled Trial. Dis Colon Rectum. 2022;65(5):713–20.

Article   PubMed   PubMed Central   Google Scholar  

Garcia-Olmo D, Herreros D, Pascual I, Pascual JA, Del-Valle E, Zorrilla J, et al. Expanded adipose-derived stem cells for the treatment of Complex Perianal Fistula: a phase II clinical trial. Dis Colon Rectum. 2009;52(1):79–86.

Panés J, García-Olmo D, Van Assche G, Colombel JF, Reinisch W, Baumgart DC, et al. Expanded allogeneic adipose-derived mesenchymal stem cells (Cx601) for complex perianal fistulas in Crohn’s disease: a phase 3 randomised, double-blind controlled trial. Lancet. 2016;388(10051):1281–90.

Bernardo ME, Fibbe WE. Mesenchymal stromal cells: sensors and switchers of inflammation. Cell Stem Cell. 2013;13(4):392–402.

Nauta AJ, Fibbe WE. Immunomodulatory properties of mesenchymal stromal cells. Blood. 2007;110(10):3499–506.

Pontikoglou C, Deschaseaux F, Sensebé L, Papadaki HA. Bone marrow mesenchymal stem cells: Biological properties and their role in Hematopoiesis and hematopoietic stem cell transplantation. Stem Cell Rev Rep. 2011;7(3):569–89.

Schreml S, Babilas P, Fruth S, Orsó E, Schmitz G, Mueller MB, et al. Harvesting human adipose tissue-derived adult stem cells: resection versus liposuction. Cytotherapy. 2009;11(7):947–57.

Panés J, García-Olmo D, Van Assche G, Colombel JF, Reinisch W, Baumgart DC, et al. Long-term efficacy and safety of Stem Cell Therapy (Cx601) for Complex Perianal Fistulas in patients with Crohn’s Disease. Gastroenterology. 2018;154(5):1334–e13424.

Cho YB, Lee WY, Park KJ, Kim M, Yoo HW, Yu CS. Autologous adipose tissue-derived stem cells for the treatment of Crohn’s Fistula: A Phase I Clinical Study. Cell Transpl. 2013;22(2):279–85.

Article   Google Scholar  

Lee WY, Park KJ, Cho YB, Yoon SN, Song KH, Kim DS, et al. Autologous adipose tissue-derived stem cells treatment demonstrated favorable and sustainable therapeutic effect for Crohn’s fistula. Stem Cells. 2013;31(11):2575–81.

de la Portilla F, Alba F, García-Olmo D, Herrerías JM, González FX, Galindo A. Expanded allogeneic adipose-derived stem cells (eASCs) for the treatment of complex perianal fistula in Crohn’s disease: results from a multicenter phase I/IIa clinical trial. Int J Colorectal Dis. 2013;28(3):313–23.

Molendijk I, Bonsing BA, Roelofs H, Peeters KCMJ, Wasser MNJM, Dijkstra G, et al. Allogeneic bone marrow–derived mesenchymal stromal cells promote Healing of Refractory Perianal Fistulas in patients with Crohn’s Disease. Gastroenterology. 2015;149(4):918–e9276.

Reenaers C, Gillard RP, Coimbra C, Gillard RM, Meunier P, Lechanteur C, et al. Clinical and MRI evolution after local injection of bone marrow-derived mesenchymal stem cells in Perianal Fistulae in Crohn’s Disease: results from a prospective Monocentric Study. J Crohns Colitis. 2023;17(5):728–37.

Lightner AL, Otero-Pineiro A, Reese J, Ream J, Nachand D, Adams AC et al. A Phase I Study of Ex Vivo Expanded Allogeneic Bone Marrow–Derived Mesenchymal Stem Cells for the Treatment of Pediatric Perianal Fistulizing Crohn’s Disease. Inflamm Bowel Dis. 2023;izad100.

Ciccocioppo R, Bernardo ME, Sgarella A, Maccario R, Avanzini MA, Ubezio C, et al. Autologous bone marrow-derived mesenchymal stromal cells in the treatment of fistulising Crohn’s disease. Gut. 2011;60(6):788–98.

Shi HY, Levy AN, Trivedi HD, Chan FKL, Ng SC, Ananthakrishnan AN. Ethnicity influences phenotype and outcomes in inflammatory bowel disease: a systematic review and Meta-analysis of Population-based studies. Clin Gastroenterol Hepatol. 2018;16(2):190–e19711.

Maaser C, Sturm A, Vavricka SR, Kucharzik T, Fiorino G, Annese V et al. ECCO-ESGAR Guideline for Diagnostic Assessment in IBD Part 1: Initial diagnosis, monitoring of known IBD, detection of complications.

Satsangi J, Silverberg MS, Vermeire S, Colombel JF. The Montreal classification of inflammatory bowel disease: controversies, consensus, and implications. Gut. 2006;55(6):749–53.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Sostegni R, Daperno M, Scaglione N, Lavagna A, Rocca R, Pera A. Crohn’s disease: monitoring disease activity. Aliment Pharmacol Ther. 2003.

Bodger K, Ormerod C, Shackcloth D, Harrison M, on behalf of the IBD Control Collaborative. Development and validation of a rapid, generic measure of disease control from the patient’s perspective: the IBD-Control questionnaire. Gut. 2014;63(7):1092–102.

Assche GV, Coremans G, Penninckx F, Rutgeerts P. Magnetic resonance imaging of the effects of Infliximab on Perianal Fistulizing Crohn’s Disease. 2003;98(2).

Melief SM, Zwaginga JJ, Fibbe WE, Roelofs H. Adipose tissue-derived multipotent stromal cells have a higher Immunomodulatory Capacity Than their bone marrow-derived counterparts. Stem Cells Transl Med. 2013;2(6):455–63.

Blanco B, Herrero-Sánchez M, del Rodríguez-Serrano C, García-Martínez C, Blanco ML, Muntión JF. Immunomodulatory effects of bone marrow versus adipose tissue-derived mesenchymal stromal cells on NK cells: implications in the transplantation setting. Eur J Haematol. 2016;97(6):528–37.

Jeon YJ, Kim J, Cho JH, Chung HM, Chae JI. Comparative analysis of human mesenchymal stem cells derived from bone marrow, Placenta, and adipose tissue as sources of cell therapy: P ROTEOMIC V ALIDATION OF M ESENCHYMAL S TEM C ELLS. J Cell Biochem. 2016;117(5):1112–25.

Pomatto M, Gai C, Negro F, Cedrino M, Grange C, Ceccotti E, et al. Differential Therapeutic Effect of Extracellular vesicles derived by bone marrow and adipose mesenchymal stem cells on Wound Healing of Diabetic Ulcers and correlation to their cargoes. Int J Mol Sci. 2021;22(8):3851.

Nammian P, Asadi-Yousefabad SL, Daneshi S, Sheikhha MH, Tabei SMB, Razban V. Comparative analysis of mouse bone marrow and adipose tissue mesenchymal stem cells for critical limb ischemia cell therapy. Stem Cell Res Ther. 2021;12(1):58.

Hoang DH, Nguyen TD, Nguyen HP, Nguyen XH, Do PTX, Dang VD, et al. Differential Wound Healing Capacity of Mesenchymal Stem Cell-Derived Exosomes originated from bone marrow, adipose tissue and umbilical cord under serum- and Xeno-Free Condition. Front Mol Biosci. 2020;7:119.

Park SC, Jeen YT. Genetic Studies of Inflammatory Bowel Disease-Focusing on Asian Patients. 2019.

Lui RNS, Ng SC. The same intestinal inflammatory disease despite different genetic risk factors in the East and West? Inflamm Intest Dis. 2016;1(2):78–84.

Walker DG, Inflamm. BOWEL Dis. 2011;106.

Download references

Acknowledgements

We would like to thank Stempeutics Research Pvt. Ltd, Bengaluru, Karnataka for supporting and funding the project.

This study was supported and funded by Stempeutics Research Pvt. Ltd, Bengaluru, Karnataka. (SRPL/PAF/18–19/002)

Author information

Shekhar Swaroop and Sudheer Kumar Vuyyuru are joint first authors.

Authors and Affiliations

Department of Gastroenterology, AIIMS, New Delhi, India

Shekhar Swaroop, Sudheer Kumar Vuyyuru, Peeyush Kumar, Sandeep Kumar Mundhra, Umang Arora, Saurabh Kedia & Vineet Ahuja

Department of Medical Gastroenterology, KIMS Hospitals, Hyderabad, India

Bhaskar Kante

Department of Radiodiagnosis and Interventional Radiology, AIIMS, New Delhi, India

Ankur Goyal, Devasenathipathy Kandasamy, Raju Sharma & Kavirajan Kabilan

Department of Gastrointestinal Surgery, AIIMS, New Delhi, India

Nihar Ranjan Dash

You can also search for this author in PubMed   Google Scholar

Contributions

VA made the initial proposal. VA, SK, and SKV designed the study and developed the protocol. VA, SKV, SS, BK, PK, and SKM were involved in the subject recruitment. VA, SKV, SS, BK, PK, SKM, and NRD were involved in the management of the patients. NRD did the administration of the stem cell. SS, SKV, BK, PK, and SKM were involved in the data collection. SS, SKV, BK, PK, SKM, RS, AG, DK, and KK were involved in the interpretation of the data. RS, AG, DK, and KK did the radiological image analysis. VA and UA did the final analysis. SS and SKV drafted the manuscript. VA, UA, and SK critically revised the manuscript. All authors contributed to conducting the trial. All authors revised the report and read and approved the final version before submission.

Corresponding authors

Correspondence to Nihar Ranjan Dash or Vineet Ahuja .

Ethics declarations

Ethics approval and consent to participate.

The study was approved by Institutional ethics committees and institutional stem cell committee of All India Institute of Medical Sciences, New Delhi, India (Project title: An open-label, single-arm, investigator initiated phase I/II study to assess the safety and efficacy of local administration of stempeucel® (adult human bone marrow derived, cultured, pooled, allogeneic mesenchymal stromal cells) in patients with perianal fistulizing Crohn’s disease; Reference No: IC-SCR/94/19; date of approval: 6th May 2019).

Consent for publication

Audio-visual and written consent was taken from each patient for publication of all the information generated from the study.

Audio-visual and written consent was taken for participation in the study from each patient.

Conflict of interest

The authors have no financial disclosures or conflicts of interest to declare.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Swaroop, S., Vuyyuru, S.K., Kante, B. et al. A phase I/II clinical trial of ex-vivo expanded human bone marrow derived allogeneic mesenchymal stromal cells in adult patients with perianal fistulizing Crohn’s Disease. Stem Cell Res Ther 15 , 140 (2024). https://doi.org/10.1186/s13287-024-03746-9

Download citation

Received : 11 November 2023

Accepted : 26 April 2024

Published : 14 May 2024

DOI : https://doi.org/10.1186/s13287-024-03746-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Perianal CD

Stem Cell Research & Therapy

ISSN: 1757-6512

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]