use of digital technology in education literature review

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Open access
• Published: 07 July 2023

Digital transformation and digital literacy in the context of complexity within higher education institutions: a systematic literature review

• Silvia Farias-Gaytan   ORCID: orcid.org/0000-0001-5858-5900 1 ,
• Ignacio Aguaded 2 &
• Maria-Soledad Ramirez-Montoya 1  

Humanities and Social Sciences Communications volume  10 , Article number:  386 ( 2023 ) Cite this article

9293 Accesses

12 Citations

3 Altmetric

Metrics details

• Science, technology and society

The incessant changes in technology generate new products and services, presenting multiple opportunities for the complex educational environment. Consequently, higher education institutions must be attentive to these changes to ensure that students have the knowledge and skills necessary for the work environment. This research aimed to identify studies related to digital transformation and digital literacy in higher education institutions through a systematic study of literature. The search resulted in 830 articles published in the Scopus and Web of Science databases from 2015 to 2022. Quality questions, inclusion and exclusion criteria were applied where 202 articles were selected for the study. The results show (a) interest of educational institutions in empirical studies where technologies are incorporated for didactic purposes, (b) challenges of opportunity in training programs to develop digital competences of teachers and students, (c) little interest in the development of media literacy, (d) the methodological aspects of the studies allow exploring new perspectives of digital transformation in higher education. This article may be of interest to academics, decision-makers and trainers of future professionals to introduce educational technology into learning processes in line with the complex demands of the world of work and society.

Similar content being viewed by others

Education reform and change driven by digital technology: a bibliometric study from a global perspective

Exploring factors influencing pre-service teacher’s digital teaching competence and the mediating effects of data literacy: empirical evidence from China

A bibliometric analysis of knowledge mapping in Chinese education digitalization research from 2012 to 2022

Introduction.

At the end of the twentieth century, the emergence of the internet led to organizations’ digital transformation from analogous to digital information (“digitization”), followed by the incorporation of information technologies into business processes (“digitalization”) (Verhoef et al., 2019 ). Several authors make no distinction between digitalization and digital transformation (Hess et al., 2016 ; Tratkowska, 2020 ; Xiao, 2020 ). Verhoef et al. (2021) propose that digital transformation goes further; its impact generates new business models and value creation. Organizations’ various areas are influenced and committed to change to remain relevant (Anderson and Ellerby, 2018 ). For this study, the term “digital transformation” (DT) was used.

Digital transformation goes beyond just incorporating technologies. An example of this is to consider that digital technologies and automation demand that the workforce develop digital skills and human-centered skills (Digital Transformation Expert Panel, 2021 ), which impacts aspects such as culture, processes, as well as the strategy of the organization (Fischer et al., 2020 ) consequently the organization must make the necessary adjustments for its effective implementation. These impacts reach all business lines including higher education.

Higher education institutions, in particular, must be attentive to the changes in the environment and society to ensure that students have the knowledge and skills demanded. Morin ( 2019 , 2020 ) invites us to think of complexity as a challenge of contemporary thinking, which requires a reform of our way of thinking, since classical scientific thinking was previously built on three foundations: order, separability and reason, but developments in science have undermined these foundations. In this sense, high-level competences such as reasoning for complexity become indispensable in the formation of critical, systemic, scientific and innovative thinking (Ramírez-Montoya et al., 2022 ; Vázquez-Parra et al., 2022 ). Complex environments require active (Patiño et al., 2023 ), collaborative (Romero-Rodríguez et al., 2022 ), open education (Suárez-Brito et al., 2022 ) and digital technology systems (George-Reyes et al., 2023 ; Ponce et al., 2022 ). Because of this, education systems around the world have made various efforts to address the influence of digital technologies and DT, such as UNESCO’s ‘Working Group on Education on Digital Skills and Work’ (UNESCO, 2017 ), the “Bologna Digital 2020” report in Europe (Rampelt et al., 2019 ), the “Outline of China’s National Plan for Medium and Long-term Educational Reform and Development (2010–2020)” of the Chinese government (Xiao, 2020 ), and the Digital Educational Agenda ADE.mx in Mexico (SEP, 2019 ). Likewise, this transformation has triggered the development of topics of interest that intertwine education with technology as proposed by González-Pérez et al. ( 2019 ) (Table 1 ):

Currently, skills performed in digital environments have been added to the basic skills performed in analog environments. Digital literacy involves mastering software and hardware, development, analysis, and interaction with digital content (Chetty et al., 2018 ). Skills such as problem-solving and applying technology were derived from digital technologies (UNESCO, 2017 ), and are considered essential for workers to adapt to digital transformation (Digital Transformation Expert Panel, 2021 ). As new technology becomes available to users, it demands from them continuous learning to remain relevant.

Due to the above, it is worthwhile to research the use and impact of technologies in the educational field on the delivery of content, pedagogical practices, and evaluation and management of learning (Williamson and Hogan, 2020 ), as well as its impact on users, teachers and students. Systematic studies of related literature are scarce, during this investigation, we found four reviews ranging from 2020 to 2021; they focused on the development of digital skills of students (Starkey, 2020 ), or university professors (Bilbao Aiastui et al., 2021 ), on digital competence assessment processes and methods in higher education (Sillat et al., 2021 ), and one focused on media literacy (Manca et al., 2021 ). This study contributes to the subject by integrating digital transformation practices in education, as well as studies on digital competencies of students and teachers, which are key roles of higher education institutions.

This article aims to identify recent studies (2015–2022) related to the issues of digital transformation and digital literacy in higher education institutions through a systematic study of literature. The study seeks to answer what educational trends higher education institutions are using, as well as what studies they have carried out in this regard, and the opportunities they have identified to advance in digital transformation and digital literacy. This study can serve as a basis for higher education institutions interested in exploring educational innovations to identify these implementations and their outcomes and seek inter-institutional collaborations with common interests.

Methodology

The study was conducted through a systematic literature review (SLR) based on the guidelines proposed by Kitchenham and Charters ( 2007 , p. 11), “a means to identify, evaluate and interpret relevant research on a particular topic". The phases to carry out the study were adapted from Kitchenham et al. ( 2010 ) and are described as follows:

Phase 1 Planning: The research starts from the objective of analyzing studies related to the topics of digital transformation and digital literacy in higher education institutions. A series of questions were defined to guide the review; these questions were derived from the integration of elements that would contribute to identify trends in digital transformation, research methods and instruments used in assessing such practices, as well as opportunities for future research; such findings would be useful to other researchers interested in the subject (Kitchenham and Charters, 2007 ) (Table 2 ).

Phase 2 Execution: The articles were selected using inclusion criteria such as the publication period between 2015 and 2022, studies in higher education institutions, focus on students and professors, and empirical research or mixed studies. Articles not arbitrated or published in languages other than Spanish and English were excluded (Table 3 ).

The search was conducted based on the above criteria in the Scopus and WoS databases (Table 4 ). 202 studies met the specified criteria (Fig. 1 ).

The flowchart presents the process of classifying the articles based on inclusion and exclusion criteria and the resulting number of articles. The flowchart was adapted from Moher et al. ( 2009 ).

Phase 3 Results: The results of each research question were analyzed to determine the educational trends higher education institutions incorporate, the studies they have carried out in this regard, and the opportunities to advance in digital transformation.

Results are presented based on the research questions. For data analysis, Excel and Power BI were used. The database is available at the following link: https://doi.org/10.6084/m9.figshare.21972170.v2 .

RQ1 What are the trends and topics addressed in the articles?

The trends identified were determined based on the emerging themes of educational technology by González-Pérez et al. ( 2019 ), highlighting digital pedagogies (166 articles), which “link pedagogical and technological supports to adapt to each area of knowledge” (González-Pérez et al., 2019 , p. 189). Examples include implementing the “blended learning” strategy (Power and Kannara, 2016 ; Tang and Chaw, 2016 ; Wang et al., 2022 ) and studies on digital skills (Ting, 2015 ; Tømte et al., 2015 ; Torres-Gastelú et al., 2019 ) and media competencies (Koc and Barut, 2016 ; Jormand et al., 2022 ) Second place went to adaptive technologies (21 articles) that “introduce systems that adapt to the needs of society and encourage learning” (González-Pérez et al., 2019 , p. 189). Examples are the use of Web 2.0 tools (Sichel et al., 2019 ), e-portfolio (Carl and Strydom, 2017 ), e-Learning (Divya and Mohamed Haneefa, 2018 ; Feriady et al., 2020 ), adaptive systems (Murray and Pérez, 2015 ), and social networks (Amaro-Jiménez et al., 2016 ; Robles Moral and Fernández Díaz, 2021 ).

To a lesser extent, the rest of the trends were found in 6 articles on technological models (Andrew et al., 2018 ; Bond et al., 2018 ; Kör et al., 2017 ) and open technologies (Cronin, 2017 ; Paskevicius and Irvine, 2019 ; Spieler et al., 2020 ). Finally, there were articles on disruptive technologies that use extended reality resources (Astudillo Torres, 2019 ; Bucea-Manea-Ţoniş et al., 2020 ) and smart technologies for mobile learning (Pinto Molina et al., 2019 ) (Fig. 2 ).

The rectangles show the proportion and number of published articles classified according to specific emerging issues in the use of educational technology as proposed by González-Pérez et al. ( 2019 ).

The analysis of the author’s keywords highlighted the issue of digital competence and digital literacy (de Ovando Calderón and Jara, 2019 ; Liu et al., 2020 ; Oria, 2020 ) and, to a lesser extent, digital teaching and media literacy (Tetep and Suparman, 2019 ; Sánchez-Caballé and Esteve-Mon, 2022 ) Also notable were keywords regarding technology in these topics (Roa Banquez et al., 2021 ; Rodríguez-Hoyos et al., 2021 ) (Fig. 3 ).

Main keywords identified in the reviewed articles.

RQ2 What are the trends in research methods observed in the articles?

Studies on digital literacy and digital transformation increased in the last three years; in 2022, it rose 53% compared to the previous year. The most commonly used research method (56%) was quantitative (Guillén-Gámez and Peña, 2020 ; Kim et al., 2018 ; Miguel-Revilla et al., 2020 ). Qualitative methods were found in similar proportions (Kajee, 2018 ; Önger and Çetin, 2018 ), and mixed methods (Pozos Pérez and Tejada Fernández, 2018 ; Techataweewan and Prasertsin, 2018 ) (Fig. 4 ).

Number of published articles during 2015–2022 classified by research method, qualitative, quantitative or mixed method.

Also, the highest number of articles was found in Spain, which represents 32% of the total, and shows an interest in digital transformation and digital literacy issues in higher education institutions; followed by Turkey with ten, the United States with nine, and Chile, China and Mexico with seven research papers each (Fig. 5 ).

Proportion of published articles distributed by country.

RQ3 What are the main findings in digital transformation and digital literacy?

The principal findings center on studies on the level of digital skills, and use of educational technology (Fig. 6 ). The most significant number of articles (121) focuses on digital competency (Blayone, 2018 ; Hong and Kim, 2018 ; Torres-Coronas and Vidal-Blasco, 2015 ; Zhao et al., 2021 ). The use of educational technology involves 2.0 technologies (Novakovich, 2016 ), virtual communities (Robin Sullivan et al., 2018 ), online education, or e-Learning (Aznar Díaz et al., 2019 ; Hamutoğlu et al., 2019 ; Gumede and Badriparsad, 2022 ). Regarding media literacy, it was found in eight articles (Altamirano Galván, 2021 ; Brown et al., 2016 ; Koc and Barut, 2016 ; Jormand et al., 2022 ; Leier and Gruber, 2021 ; Olivia-Dumitrina et al., 2019 ; Reyna and Meier, 2018 ; Robles Moral and Fernández Díaz, 2021 ). Two additional issues identified were environmental protection (Amador-Alarcón et al., 2022 ) and educational process (Makarova et al., 2021 ) both of interest to today’s situation faced by higher education institutions.

Trends, topics and main findings from the reviewed articles.

RQ4 What are the authors’ recommendations for future studies? And RQ5 What are the opportunities identified in the studies?

By correlating these two questions, we identified four opportunities regarding digital literacy and digital transformation (Fig. 7 ); first, that higher education institutions have training programs for both students and teachers to help them develop digital skills (Igbo and Imo, 2020 ; Martzoukou et al., 2020 ; Sandí Delgado, 2020 ), media skills (López-Meneses et al., 2020 ; Reyna and Meier, 2018 ; Romero-Rodriguez et al., 2016 ), and critical thinking (Kocak et al., 2021 ; Nagel et al., 2022 ; Vetter and Sarraf, 2020 ). Second, that the development of skills requires to enhance learning design by incorporating new didactic strategies, and educational technologies in academic programs (Boulton, 2020 ; del Prete and Almenara, 2020 ; Foster, 2020 ; Liesa-Orús et al., 2020 ; McGrew et al., 2019 ), and that the impact of these changes improves learning (Castellanos et al., 2017 ; Dafonte-Gómez et al., 2018 ; Sosa Díaz and Palau Martín, 2018 ).

Frequency of recommendations and opportunities for future studies.

On the other hand, methodological recommendations for future studies included incorporating new instruments and variables to collect more information (Kamardeen and Samaratunga, 2020 ; Khalil and Srinivasan, 2019 ; Varga-Atkins, 2020 ; Vetter and Sarraf, 2020 ). Others pointed to increasing the sample size (Amhag et al., 2019 ; Kolodziejczyk et al., 2020 ; Munoz-Repiso and del Pozo, 2016 ; Pozo-Sánchez et al., 2020 ). To a lesser extent, longitudinal studies were recommended to test the models used (He et al., 2018 ; Johnston, 2020 ). In addition, we found that 28% of the studies did not include recommendations, and 31% did not include opportunities for future studies.

RQ6 What are the stated limitations in digital literacy studies involving digital transformation?

The limitations indicated in the studies refer primarily to the small sample size (45%) (Arango et al., 2020 ; Romero-Tena et al., 2020 ; Tugtekin and Koc, 2020 ). To a lesser extent, limitations were found with the instrument used to carry out the study (Heuling et al., 2021 ; Nikou and Aavakare, 2021 ; Sánchez-Caballé and Esteve-Mon, 2022 ). Problems with the technology used was another limitation highlighted in eight studies (Castellano, 2016 ; Pozo-Sánchez et al., 2020 ). Finally, seven studies reported limitation regarding its feasibility (Dafonte-Gómez et al., 2018 ; Fázik and Steinerová, 2020 ; Kerr et al., 2019 ) and one on the low response obtained (Myyry et al., 2022 ); 36% of the studies did not include limitations (Fig. 8 ).

Frequency of limitations found in the reviewed articles. The figure does not include data from articles that did not specify the limitations (36%).

Incorporating educational trends and new technologies in the educational environment has highlighted the need to continue developing skills that allow their adoption by teachers and students. The interest in digital pedagogies and the study of digital competencies were relevant trends among higher education institutions aiming to use adaptive, intelligent, open, or disruptive technologies and technological models (Fig. 2 ). The transition from the analog to the digital world in both processes and products of organizations is part of their journey towards digital transformation (Hess et al., 2016 ; Tratkowska, 2020 ). It also includes organizational and cultural changes among users and operators (Anderson and Ellerby, 2018 ). However, we must point out that technology is not the end in itself but should be a means to facilitate learning.

Therefore, studies employing the scientific method where the benefit can be determined are relevant, and those that examine areas of opportunity by adopting technologies in the learning process. In the last three years, empirical studies on incorporating educational innovations in teaching practice in higher education institutions increased, most applying mainly quantitative methods (Figs. 4 and 5 ). Spain is the country that stands out with the most studies (64). In some cases, the impetus for these efforts has come from the establishment of educational strategies at the national (SEP, 2019 ; Xiao, 2020 ) and regional level (Rampelt et al., 2019 ). These studies denote international interest in the influence of digital transformation, and digital literacy on the educational process.

Digital technology skills and knowledge are hallmarks of the twenty-first-century generations. Digital literacy and educational technology accounted for 95% of the study findings, and only 4% focused on media literacy. Required job competencies include software and hardware skills, critical thinking, information analysis, and the ability to create and communicate content (Chetty et al., 2018 ; Silva et al., 2021 ; UNESCO, 2017 ). “Workers who can combine ‘human’ skills like empathy, cooperation and negotiation with cognitive skills such as problem-solving, will thrive in an economy that increasingly relies on both types of skill” (Digital Transformation Expert Panel, 2021 ). As the work environment and education continue to evolve along new technologies.

In addition to the conceptual components, the methodological aspects of the studies allow exploring new perspectives of digital transformation in higher education. In the studies reviewed, 44% of the recommendations concerned using new instruments, and exploring new variables, while 56% were about sample size increase and longitudinal studies (Fig. 7 ). Although they have not been conceived or designed for the educational field, the technologies are embedded today in the learning process (González-Pérez et al., 2019 ). Studies on their adoption allow testing and validating methodologies and instruments to have reliable data for their implementation (García-Ruiz et al., 2014 ). Though used simultaneously by teachers and students, the adoption of technology may require the implementation of different strategies or approaches to meet the needs of each group.

The ability to learn and unlearn is being tested by constantly introducing technologies into human activities. The opportunities reported by the studies coincide with the need for institutions to have training programs to develop skills for larger groups (27%). In the case of students, other topics of interest are the use of technology, enriched learning experiences, and security and privacy issues (Fig. 7 ). Organizations’ digital transformation strategy must consider the training of their members and their users because the skills required for the job become increasingly specialized (Anderson and Ellerby, 2018 ; Hess et al., 2016 ; Verhoef et al., 2019 ). In order to get the best out of educational technology, users are required to have a minimum level of digital literacy (Kerr et al., 2019 ). Higher education institutions are a fertile place to continue studies on digital transformation and the development of digital literacy of their members.

Therefore, empirical studies on the experiences and challenges faced by higher education institutions in adopting technologies in the learning process and strategies implemented to train teachers and students are relevant. The limitations reported in the studies focused on methodological issues, with the sample size being the most crucial aspect to consider (45%). These studies were carried out in groups managed by the researcher, making it difficult to project the results. The systematic literature review methodology emphasizes the analysis of variables to answer research questions so that similarities and differences among studies can be identified (Kitchenham et al., 2010 ; Kitchenham and Charters, 2007 ). Inter-institutional collaboration can contribute to achieving results that help find joint strategies to promote the adoption of educational innovations and the development of competencies in both teachers and students.

Limitations

This study was limited to trends in higher education institutions in a specific period of time (2015–2022). Another limitation was the selection of two databases, Scopus and Web of Science, which although they include high-impact journals, articles from other databases were not considered; future research can continue the timeline and include other systems and databases.

Conclusions

The digital transformation of higher education institutions goes beyond its impact on administrative and operational processes. The study showed that teachers have incorporated educational trends, new pedagogies and technologies for didactic purposes, and this has highlighted the need to develop the level of digital literacy of both teachers and students. Higher education institutions, as trainers of future professionals, must acknowledge the need for digital transformation and act upon to develop strategies so students and teachers are prepared for the demands of the workplace.

The pandemic spurred the urgency of developing digital skills for both teachers and students. Technologies they used for socializing and leisure became necessary tools for study and work. Higher education institutions are conducting studies on their experiences of adopting educational technologies and the impact on their users. Although related empirical studies on media literacy were scarce, since it is linked to the use of technology, future studies have an opportunity to assess how it develops in the following years. These should examine teachers’ and students’ performance, their critical capacity as media users, and content creators.

The development of teachers’ digital competencies involves not only the mastery of technology but also the improvement of their teaching practice with the appropriate pedagogical use of technology to contribute to student learning. There are opportunities for higher education institutions in measuring digital competencies to find strengths and weaknesses to focus their training programs. The same applies to students, who should be provided with the relevant training for the development of digital skills and prevent the lack of these from becoming an obstacle to their performance in the classroom.

This study aimed to identify the state of digital transformation and digital literacy in higher education institutions and their impact on students and teachers. Digital transformation and new technologies are generating complex environments that demand the development of digital and high-level skills. Technological progress provides opportunities to enhance the learning process. Research must continue to assess the performance and students’ learning gains. This study can serve as a basis for higher education institutions interested in exploring educational innovations to identify these implementations and their outcomes and seek inter-institutional collaborations with common interests.

Data availability

The datasets generated during and/or analyzed in the current study are available in Figshare repository: https://doi.org/10.6084/m9.figshare.21972170 .

Altamirano Galván SG (2021) Perfil de alfabetización mediática de estudiantes y docentes de educación superior. CPU-e, Revista de Investigación Educativa 32:88–110. https://doi.org/10.25009/cpue.v0i32.2735

Article   Google Scholar  

Amador-Alarcón MP, Torres-Gastelú CA, Lagunes-Domínguez A, Medina-Cruz H, Argüello-Rosales CA (2022) Perceptions of environmental protection of university students: a look through digital competences in Mexico. Sustainability 14(18):11141. https://doi.org/10.3390/su141811141

Amaro-Jiménez C, Hungerford-Kresser H, Pole K (2016) Teaching with a technological twist: exit tickets via Twitter in literacy classrooms. J Adolesc Adult Lit 60(3):305–313. https://doi.org/10.1002/jaal.572

Amhag L, Hellström L, Stigmar M (2019) Teacher educators’ use of digital tools and needs for digital competence in higher education. J Digit Learn Teach Educ 35(4):203–220. https://doi.org/10.1080/21532974.2019.1646169

Anderson C, Ellerby W (2018) Digital Maturity Model. Achieving digital maturity to drive growth. In: Deloitte (Issue February). https://www2.deloitte.com/content/dam/Deloitte/global/Documents/Technology-Media-Telecommunications/deloitte-digital-maturity-model.pdf

Andrew M, Taylorson J, Langille DJ, Grange A, Williams N (2018) Student attitudes towards technology and their preferences for learning tools/devices at two universities in the UAE. J Inf Technol Educ Res 17:309–344. https://doi.org/10.28945/4111

Arango DAG, Fernández JEV, Carrillo JAO, Rojas ÓAC, Villa CFH (2020) Dimensions of digital competence in university teachers: Relational analysis based on components | Dimensiones de competencia digital en docentes universitarios: Análisis relacional basado en componentes. RISTI-Revista Iberica de Sistemas e Tecnologias de Informacao 2020(E28):945–960

Google Scholar  

Astudillo Torres MP (2019) Aplicación de la realidad aumentada en las prácticas educativas universitarias. RELATEC: Revista Latinoamericana de Tecnología Educativa 18(2):203–218. https://doi.org/10.17398/1695-288X.18.2.203

Aznar Díaz I, Cáceres Reche MP, Romero Rodríguez JM (2019) Digital competence of an E-learning tutor: an emerging model of good teaching practices in ICT. Texto Livre 12(3):49–68. https://doi.org/10.17851/1983-3652.12.3.49-68

Bilbao Aiastui E, Arruti Gómez A, Carballedo Morillo R (2021) A systematic literature review about the level of digital competences defined by DigCompEdu in higher education. Aula Abierta 50(4):841–850. https://doi.org/10.17811/rifie.50.4.2021.841-850

Blayone T (2018) Reexamining digital-learning readiness in higher education: Positioning digital competencies as key factors and a profile application as a readiness tool. Int J E-Learn Corp Gov Healthcare High Educ 17(4):425–451

Bond M, Marín VI, Dolch, Bedenlier S, Zawacki-Richter O (2018) Digital transformation in German higher education: student and teacher perceptions and usage of digital media. Int J Educ Technol High Educ 15(1). https://doi.org/10.1186/s41239-018-0130-1

Boulton P (2020) Digitally proficient but disconnected from the outdoor world? A reflection on pedagogies used in an Early Years degree in higher education. J Appl Res High Educ 13(1):195–210. https://doi.org/10.1108/JARHE-03-2019-0066

Brown C, Czerniewicz L, Noakes T (2016) Online content creation: looking at students’ social media practices through a Connected Learning lens. Learn Media Technol 41(1):140–159. https://doi.org/10.1080/17439884.2015.1107097

Bucea-Manea-Ţoniş R, Bucea-Manea-Ţoniş R, Simion VE, Ilic D, Braicu C, Manea N (2020) Sustainability in higher education: The relationship between work-life balance and XR e-learning facilities. Sustainability (Switzerland), 12(14). https://doi.org/10.3390/su12145872

Carl A, Strydom S (2017) e-Portfolio as reflection tool during teaching practice: The interplay between contextual and dispositional variables. South Afr J Educ 37(1). https://doi.org/10.15700/saje.v37n1a1250

Castellano J (2016) Advanced media english–a modern PrOCALL course. CALL-EJ 17(1):52–66

Castellanos A, Sánchez C, Calderero JF (2017) Nuevos modelos tecnopedagógicos. Competencia digital de los alumnos universitarios. Revista Electrónica de Investigación Educativa 19(1):1–9. https://doi.org/10.24320/redie.2017.19.1.1148

Chetty K, Qigui L, Gcora N, Josie J, Wenwei L, Fang C (2018). Bridging the digital divide: measuring digital literacy. Economics. https://doi.org/10.5018/economics-ejournal.ja.2018-23

Cronin C (2017) Openness and praxis: exploring the use of open educational practices in higher education. Int Rev Res Open Dist Learn 18(5):15–34. https://doi.org/10.19173/irrodl.v18i5.3096

Dafonte-Gómez A, García-Crespo O, Ramahi-García D (2018) ‘Flipped learning’ y competencia digital: diseño tecnopedagógico y percepción del alumnado universitario. Index Comunicación 8(2):275–294. http://plataformarevistascomunicacion.org/2018/11/articulo-flipped-learning-competencia-digital-diseno-tecnopedagogico-percepcion-del-alumnado-universitario/

de Ovando Calderón JS, Jara VJ (2019) Digital competence of health sciences teachers of a Chilean university Pixel-Bit, Revista de Medios y Educacion 56:193–211. https://doi.org/10.12795/pixelbit.2019.i56.10

del Prete A, Almenara JC (2020) Use of the virtual learning environment among higher education teaching staff: a gender analysis | El uso del Ambiente Virtual de Aprendizaje entre el profesorado de educación superior: Un análisis de género. Revista de Educacion a Distancia, 20(62). https://doi.org/10.6018/RED.400061

Digital Transformation Expert Panel. (2021) The Learning Country. Digital Transformation Skills Strategy. https://www.digitalskillsformation.org.au/wp-content/uploads/2021/05/Digital-Transformation-Skills-Strategy-010521.pdf?v=2

Divya P, Mohamed Haneefa K (2018) Digital reading competency of students: A study in universities in Kerala. DESIDOC J Libr Inf Technol 38(2):88–94. https://doi.org/10.14429/djlit.38.2.12233

Fázik J, Steinerová J (2020) Technologies, knowledge and truth: the three dimensions of information literacy of university students in Slovakia. J Doc https://doi.org/10.1108/JD-05-2020-0086

Feriady M, Nurkhin A, Mahmud N, Setiani R, Astuti DP (2020) Influence of organizational suport and digital literacy on lecturer acceptance of e-learning in indonesia: a modification of technologi acceptance model. Int J Sci Technol Res 9(1):2229–2233

Fischer M, Imgrund F, Janiesch C, Winkelmann A (2020) Strategy archetypes for digital transformation: defining meta objectives using business process management. Inf Manage. https://doi.org/10.1016/j.im.2019.103262

Foster B (2020) Information literacy beyond librarians: a data/methods triangulation approach to investigating disciplinary IL teaching practices. Evid Based Libr Inf Pract 15(1):20–37. https://doi.org/10.18438/EBLIP29635

García-Ruiz R, Ramírez-García A, Rodríguez-Rosell M (2014) Educación en alfabetización mediática para una nueva ciudadanía prosumidora. Comunicar XXII(43):15–23. https://doi.org/10.3916/C43-2014-01

George-Reyes CE, Ramírez-Montoya MS, López-Caudana EO (2023) Imbrication of the Metaverse in the complexity of education 4.0: Approach from an analysis of the literatura [Imbricación del Metaverso en la complejidad de la educación 4.0: Aproximación desde un análisis de la literatura]. Pixel-Bit. Revista de Medios y Educación 66:199–237. https://doi.org/10.12795/pixelbit.97337

González-Pérez LI, Ramírez-Montoya MS, García-Peñalvo FJ (2019) Innovación educativa en estudios sobre el desarrollo y uso de la tecnología: un mapeo sistemático. In: Ramírez Montoya MS, & Valenzuela González JR (eds.), Innovación educativa: tendencias globales de investigación e implicaciones prácticas (Primera, pp. 137–160). Ediciones OCTAEDRO, S. L. https://octaedro.com/libro/innovacion-educativa-tendencias-globales-de-investigacion-e-implicaciones-practicas/

Guillén-Gámez FD, Peña MP (2020) Univariate analysis of digital competence in physical education: an empirical study | Análisis Univariante de la Competencia Digital en Educación Física: Un estudio empírico. Retos 37:326–332

Gumede L, Badriparsad N (2022) Online teaching and learning through the students’ eyes –Uncertainty through the COVID-19 lockdown: a qualitative case study in Gauteng province, South Africa. Radiography 28(1):193–198. https://doi.org/10.1016/j.radi.2021.10.018

Article   CAS   PubMed   Google Scholar  

Hamutoğlu NB, Savaşçı M, Sezen-Gültekin G (2019) Digital literacy skills and attitudes towards e-learning. J Educ Fut, August, 93–107. https://doi.org/10.30786/jef.509293

He T, Zhu C, Questier F (2018) Predicting digital informal learning: an empirical study among Chinese University students. Asia Pac Educ Rev 19(1):79–90. https://doi.org/10.1007/s12564-018-9517-x

Hess T, Matt C, Benlian A, Wiesböck F (2016) Options for formulating a digital transformation strategy. MIS Q Executive 15(2):103–119. https://www.researchgate.net/publication/291349362

Heuling LS, Wild S, Vest A (2021) Digital competences of prospective engineers and science teachers: a latent profile and correspondence analysis. Int J Educ Math Sci Technol 9(4):760–782. https://doi.org/10.46328/ijemst.1831

Hong AJ, Kim HJ (2018) College students’ Digital Readiness for Academic Engagement (DRAE) Scale: scale development and validation. Asia-Pac Educ Res 27(4):303–312. https://doi.org/10.1007/s40299-018-0387-0

Igbo HU, Imo NT (2020) Digital libraries and access to information in Nigerian Federal Universities: The impact of technology variables. J Inf Knowl Manage 19(2). https://doi.org/10.1142/S0219649220500136

Johnston N (2020) The shift towards digital literacy in Australian University libraries: developing a digital literacy framework. J Aust Libr Inf Asso 69(1):93–101. https://doi.org/10.1080/24750158.2020.1712638

Article   MathSciNet   Google Scholar  

Jormand H, Bashirian S, Barati M, Rezapur-Shahkolai F, Babamiri M (2022) Evaluation of a web-based randomized controlled trial educational intervention based on media literacy on preventing substance abuse among college students, applying the integrated social marketing approach: a study protocol. Trials 23(1):1006. https://doi.org/10.1186/s13063-022-06913-6

Article   PubMed   PubMed Central   Google Scholar  

Kajee L (2018) Teacher education students engaging with digital identity narratives. S Afr J Educ 38(2). https://doi.org/10.15700/saje.v38n2a1501

Kamardeen I, Samaratunga M (2020) Digiexplanation driven assignments for personalising learning in construction education. Constr Econ Build 20(3):103–123. https://doi.org/10.5130/AJCEB.v20i3.7000

Kerr J, Dale VH, Gyurko F (2019) Evaluation of a MOOC Design Mapping Framework (MDMF): Experiences of academics and learning technologists. Electron J E-Learn 17(1):38–51. http://www.ejel.org/volume17/issue1/p38

Khalil R, Srinivasan V (2019) Massive open online courses/ MOOCS: a gateway to enrich e-learning in management education. Int J Innov Creat Change 7(5):112–124

Kim HJ, Hong AJ, Song H-D (2018) The relationships of family, perceived digital competence and attitude, and learning agility in sustainable student engagement in higher education. Sustainability (Switzerland), 10(12). https://doi.org/10.3390/su10124635

Kitchenham B, Pretorius R, Budgen D, Brereton OP, Turner M, Niazi M, Linkman S (2010) Systematic literature reviews in software engineering-a tertiary study. Inf Softw Technol 52(8):792–805. https://doi.org/10.1016/j.infsof.2010.03.006

Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. https://doi.org/10.1145/1134285.1134500

Koc M, Barut E (2016) Development and validation of New Media Literacy Scale (NMLS) for university students. Comput Hum Behav 63:834–843. https://doi.org/10.1016/j.chb.2016.06.035

Kocak O, Coban M, Aydin A, Cakmak N (2021) The mediating role of critical thinking and cooperativity in the 21st century skills of higher education students. Think Ski Creat 42:100967. https://doi.org/10.1016/j.tsc.2021.100967

Kolodziejczyk I, Gibbs P, Nembou C, Sagrista MR (2020) Digital skills at divine word university, Papua New Guinea. IAFOR J Educ 8(2):107–124. https://doi.org/10.22492/ije.8.2.06

Kör H, Erbay H, Engin M, Dünder E (2017) An examination of the correlation between science and technology attitudes scale, frequency of smartphone usage scale and lifelong learning scale scores using the structural equation model. J Baltic Sci Educ 16(1):86–99

Leier V, Gruber A (2021) Team New Zealand-Sweden-Germany: a joint venture exploring language learning in digital spaces. JALT CALL J 17(3):298–324. https://doi.org/10.29140/jaltcall.v17n3.410

Liesa-Orús M, Latorre-Cosculluela C, Vázquez-Toledo S, Sierra-Sánchez V (2020) The technological challenge facing higher education professors: perceptions of ICT tools for developing 21st Century skills. Sustainability (Switzerland), 12(13). https://doi.org/10.3390/su12135339

Liu ZJ, Tretyakova N, Fedorov V, Kharakhordina M (2020) Digital literacy and digital didactics as the basis for new learning models development. Int J Emerg Technol Learn 15(14):4–18. https://doi.org/10.3991/ijet.v15i14.14669

Article   CAS   Google Scholar  

López-Meneses E, Sirignano FM, Vázquez-Cano E, Ramírez-Hurtado JM (2020) University students’ digital competence in three areas of the DigCom 2.1 model: a comparative study at three European universities. Australas J Educ Technol 36(3):69–88. https://doi.org/10.14742/AJET.5583

Makarova O, Ldokova G, Egorova R (2021) Analysis of students’ views of the quality of pedagogical education in Russia. Int J Educ Math Sci Technol 9(3):462–481. https://doi.org/10.46328/ijemst.1624

Manca S, Bocconi S, Gleason B (2021) “Think globally, act locally”: a glocal approach to the development of social media literacy. Comput Educ 160:104025. https://doi.org/10.1016/j.compedu.2020.104025

Martzoukou K, Fulton C, Kostagiolas P, Lavranos C (2020) A study of higher education students’ self-perceived digital competences for learning and everyday life online participation. J Doc 76(6):1413–1458. https://doi.org/10.1108/JD-03-2020-0041

McGrew S, Smith M, Breakstone J, Ortega T, Wineburg S (2019) Improving university students’ web savvy: an intervention study. Br J Educ Psychol 89(3):485–500. https://doi.org/10.1111/bjep.12279

Article   PubMed   Google Scholar  

Miguel-Revilla D, Martínez-Ferreira JM, Sánchez-Agustí M (2020) Assessing the digital competence of educators in social studies: an analysis in initial teacher training using the TPACK-21 model. Australas J Educ Technol 36(2):1–12. https://doi.org/10.14742/ajet.5281

Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group (2009) Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med 6(7):1–6. https://doi.org/10.1371/journal.pmed.1000097

Morin E (2019) Pensar la Complejidad. Crisis ymetamorfosis. Universitat de Valencia

Morin E (2020) Cambiemos de Via. Lecciones de la pandemia. Paidos

Munoz-Repiso A, del Pozo M (2016) Analysis of the digital competences of graduates of university degrees to be a teacher. Revista Latinoamericana de Tecnologia Educativa-RELATEC 15(2):155–168. https://doi.org/10.17398/1695-288X.15.2.155

Murray MC, Pérez J (2015) Informing and performing: a study comparing adaptive learning to traditional learning. Inform Sci 18(1):111–125. https://doi.org/10.28945/2165

Myyry L, Kallunki V, Katajavuori N, Repo S, Tuononen T, Anttila H, Kinnunen P, Haarala-Muhonen A, Pyörälä E (2022) COVID-19 accelerating academic teachers’ digital competence in distance teaching. Front Educ 7. https://doi.org/10.3389/feduc.2022.770094

Nagel M-T, Zlatkin-Troitschanskaia O, Fischer J (2022) Validation of newly developed tasks for the assessment of generic Critical Online Reasoning (COR) of university students and graduates. Front Educ 7. https://doi.org/10.3389/feduc.2022.914857

Nikou S, Aavakare M (2021) An assessment of the interplay between literacy and digital technology in higher education. Educ Inf Technol 26(4):3893–3915. https://doi.org/10.1007/s10639-021-10451-0

Novakovich J (2016) Fostering critical thinking and reflection through blog-mediated peer feedback. J Comput Assist Learn 32(1):16–30. https://doi.org/10.1111/jcal.12114

Olivia-Dumitrina N, Casanovas M, Capdevila Y (2019) Academic writing and the internet: Cyber-plagiarism amongst university students. J New Approach Educ Res 8(2):112–125. https://doi.org/10.7821/naer.2019.7.407

Önger S, Çetin T (2018) An investigation into digital literacy views of social studies preservice teachers in the context of authentic learning. Rev Int Geogr Educ Online 8(1):109–124

Oria B (2020) Edmodo como herramienta de aprendizaje telecolaborativo online en el aula de inglés. Encuentro. 28:49–70

Paskevicius M, Irvine V (2019) Open education and learning design: Open pedagogy in praxis. J Interact Media Educ 2019(1). https://doi.org/10.5334/jime.512

Patiño A, Ramírez-Montoya MS, Buenestado-Fernández M (2023) Active learning and education 4.0 for complex thinking training: analysis of two case studies in open education. Smart Learn Environ 10(8). https://doi.org/10.1186/s40561-023-00229-x

Pinto Molina M, Gómez-Hernández J-A, Sales D, Cuevas-Cerveró A, Fernández Pascual R, Caballero Mariscal D, Guerrero-Quesada DJ, Navalón C (2019) Aprender y enseñar competencias digitales en un entorno móvil: avances de una investigación aplicada a profesorado y alumnado universitario de Ciencias Sociales. Revista Ibero-Americana de Ciencia Da Informacao 12(2):585–596. https://doi.org/10.26512/10.26512/rici.v

Ponce P, Ramirez R, Ramirez-Montoya MS, Molina A, MacCleery B, Ascanio M (2022) From understanding a simple DC motor to developing an electric vehicle AI controller rapid prototype using MATLAB-Simulink, real-time simulation and complex thinking. Front Educ 7:941972. https://doi.org/10.3389/feduc.2022.941972

Power J, Kannara V (2016) Best-practice model for technology enhanced learning in the creative arts. Res Learn Technol 24. https://doi.org/10.3402/rlt.v24.30231

Pozos Pérez KV, Tejada Fernández J (2018) Competencias digitales en docentes de educación superior: niveles de dominio y necesidades formativas. Revista Digital de Investigación En Docencia Universitaria 12(2):59–87. https://doi.org/10.19083/ridu.2018.712

Pozo-Sánchez S, López-Belmonte J, Fuentes-Cabrera A, Moreno-Guerrero A-J (2020) Incidence of retro-innovation in higher education. Radio and television as complementary tools when using the educational model known as flipped learning. Form Univ 13(3):139–146. https://doi.org/10.4067/S0718-50062020000300139

Ramírez-Montoya MS, Castillo-Martínez IM, Sanabria-Zepeda JC, Miranda J (2022) Complex thinking in the framework of education 4.0 and open innovation—a systematic literature review. J Open Innov Technol Market Complex 8(4). https://doi.org/10.3390/joitmc8010004

Rampelt F, Orr D, Knoth A (2019) Bologna Digital 2020–White Paper on Digitalisation in the European Higher Education Area. Publisher: Geschäftsstelle Hochschulforum Digitalisierung beim Stifterverband für die Deutsche Wissenschaft e.V

Reyna J, Meier P (2018). Using the Learner-Generated Digital Media (LGDM) framework in tertiary science education: a pilot study. Educ Sci 8(3). https://doi.org/10.3390/educsci8030106

Roa Banquez K, Viviana Rojas Torres CG, González Rincón LJ, Ortiz Ortiz EG (2021) El docente en la era 4.0: una propuesta de formación digital que fortalezca el proceso de enseñanza y aprendizaje. Revista Virtual Universidad Católica Del Norte 63:126–160. https://doi.org/10.35575/rvucn.n63a6

Robin Sullivan R, Neu V, Yang F (2018) Faculty development to promote effective instructional technology integration: a qualitative examination of reflections in an online community. Online Learn J 22(4):341–359. https://doi.org/10.24059/olj.v22i4.1373

Robles Moral FJ, Fernández Díaz M (2021) Future primary school teachers’ digital competence in teaching science through the use of social media. Sustainability 13(5):2816. https://doi.org/10.3390/su13052816

Rodríguez-Hoyos C, Fueyo Gutiérrez A, Hevia Artime I (2021) Competencias digitales del profesorado para innovar en la docencia universitaria. Analizando el uso de los dispositivos móviles. Pixel-Bit, Revista de Medios y Educación 61:71–97. https://doi.org/10.12795/pixelbit.86305

Romero-Rodriguez L, Torres-Toukoumidis D, Pérez-Rodríguez M, Aguaded I (2016) Analfanauts and fourth screen: lack of infodiets and media and information literacy in Latin American University Students. Fonseca-J Commun 12:11–25. https://core.ac.uk/download/pdf/60674744.pdf

Romero-Rodríguez JM, Ramirez-Montoya, MS, Glasserman-Morales LD, Ramos Navas-Parejo M (2022) Collaborative online international learning between Spain and Mexico: a microlearning experience to enhance creativity in complexity. Educ+Train. https://doi.org/10.1108/ET-07-2022-0259

Romero-Tena R, Barragán-Sánchez R, Llorente-Cejudo C, Palacios-Rodríguez A (2020) The challenge of initial training for early childhood teachers. A cross sectional study of their digital competences. Sustainability (Switzerland), 12(11). https://doi.org/10.3390/su12114782

Sánchez-Caballé A, Esteve-Mon FM (2022) Digital teaching competence of university teachers: a comparative study at two European universities. Australas J Educ Technol 50–61. https://doi.org/10.14742/ajet.7408

Sandí Delgado JC (2020) Desarrollo de competencias digitales en el profesorado a través de juegos serios: un estudio de caso aplicado en la Universidad de Costa Rica (UCR). E-Ciencias de La Información. https://doi.org/10.15517/eci.v10i2.38946

SEP (2019) Agenda Digital Educativa ADE.mx. In: Secretaría de Educación Pública. https://infosen.senado.gob.mx/sgsp/gaceta/64/2/2020-02-05-1/assets/documentos/Agenda_Digital_Educacion.pdf

Sichel CE, Javdani S, Ueberall S, Liggett R (2019) Leveraging youths’ digital literacies: the E-Responder social media violence interruption model and pilot evaluation. J Prevent Intervent Community 47(2):76–89. https://doi.org/10.1080/10852352.2019.1582145

Sillat LH, Tammets K, Laanpere M (2021) Digital competence assessment methods in higher education: a systematic literature review. Educ Scie 11(8):402. https://doi.org/10.3390/educsci11080402

Silva MB, Borges G, Fantin M, Almeida M, Aguaded I (2021) Media competence in children aged 9 to 12 in Brazilian settings. Revista Brasileira de Ciências Da Comunicação 44(1):21–45. http://portcom.intercom.org.br/revistas/index.php/revistaintercom/article/view/3487/2499

Sosa Díaz MJ, Palau Martín RF (2018) Flipped classroom para adquirir la competencia digital docente: una experiencia didáctica en la Educación Superior. Pixel-Bit Revista de Medios y Educación 52:37–54. https://doi.org/10.12795/pixelbit.2018.i52.03

Spieler B, Grandl M, Ebner M, Slany W (2020) Bridging the gap: a computer science Pre-MOOC for first semester students making with kids view project. Electron J E-Learn. https://doi.org/10.34190/EJEL.20.18.3.004

Starkey L (2020) A review of research exploring teacher preparation for the digital age. Camb J Educ 50(1):37–56. https://doi.org/10.1080/0305764X.2019.1625867

Suárez-Brito P, López-Caudana EO, Baena-Rojas JJ, Ramírez-Montoya MS (2022) Eliciting complex thinking through open educational resource projects. J Soc Stud Educ Res 13(4):56–77. https://jsser.org/index.php/jsser/article/view/4472

Tang CM, Chaw LY (2016) Digital literacy: A prerequisite for effective learning in a blended learning environment? Electron J E-Learn 14(1):54–65

Techataweewan W, Prasertsin U (2018) Development of digital literacy indicators for Thai undergraduate students using mixed method research. Kasetsart J Soc Sci 39(2):215–221. https://doi.org/10.1016/j.kjss.2017.07.001

Tetep, Suparman A (2019) Students’ digital media literacy: effects on social character. Int J Recent Technol Eng 8(2 Special Issue 9):394–399. https://doi.org/10.35940/ijrte.B1091.0982S919

Ting Y-L (2015) Tapping into students’ digital literacy and designing negotiated learning to promote learner autonomy. Internet High Educ 26:25–32. https://doi.org/10.1016/j.iheduc.2015.04.004

Tømte C, Enochsson A-B, Buskqvist U, Kårstein A (2015) Educating online student teachers to master professional digital competence: The TPACK-framework goes online. Comput Educat 84:26–35. https://doi.org/10.1016/j.compedu.2015.01.005

Torres-Coronas T, Vidal-Blasco M-A (2015) Students and employers perception about the development of digital skills in higher education | Percepción de estudiantes y empleadores sobre el desarrollo de competencias digitales en la educación superior. Revista de Educacion 367:63–89. https://doi.org/10.4438/1988-592X-RE-2015-367-283

Torres-Gastelú CA, Cordero-Guzmán DM, Soto-Ortíz JL, Mory-Alvarado A (2019) Influence of factors about the manifestation of digital citizenship. Prisma Soc 26:27–49

Tratkowska K (2020) Digital transformation: theoretical backgrounds of digital change. Manage Sci 24(4):32–37. https://doi.org/10.15611/ms.2019.4.05

Tugtekin EB, Koc M (2020) Understanding the relationship between new media literacy, communication skills, and democratic tendency: Model development and testing. New Media Soc 22(10):1922–1941. https://doi.org/10.1177/1461444819887705

UNESCO (2017) Working group on education: Digital skills for life and work. In: Broadband commission for sustainable development

Varga-Atkins T (2020) Beyond description: In search of disciplinary digital capabilities through signature pedagogies. Res Learn Technol 28:1–19. https://doi.org/10.25304/rlt.v28.2467

Vázquez-Parra JC, Cruz-Sandoval M, Carlos-Arroyo M (2022) Social entrepreneurship and complex thinking: a bibliometric study. Sustainability 14(20). https://doi.org/10.3390/su142013187

Verhoef PC, Broekhuizen T, Bart Y, Bhattacharya A, Qi Dong J, Fabian N, Haenlein M (2019) Digital transformation: a multidisciplinary reflection and research agenda. J Bus Res https://doi.org/10.1016/j.jbusres.2019.09.022

Vetter MA, Sarraf KS (2020) Assessing the Art + feminism Edit-a-thon for Wikipedia literacy, learning outcomes, and critical thinking. Interact Learn Environ https://doi.org/10.1080/10494820.2020.1805772

Wang HL, Almeida S, Frino B, Wijayawardena K, Rauf A, Hardie G (2022) Dialogue matters a lot: autoethnographic reflections of an Australian teaching team managing first-year undergraduate students. Int J Manage Educ 20(3):100699. https://doi.org/10.1016/j.ijme.2022.100699

Williamson B, Hogan A (2020). Commercialisation and privatisation in / of education in the context of Covid-19. In: Educ Int Res (Issue July). Education International. https://codeactsineducation.wordpress.com/2020/07/14/evolution-global-education-industry-during-pandemic/

Xiao J (2020) Digital transformation in higher education: critiquing the five-year development plans (2016-2020) of 75 Chinese Universities. Dist Educ 40(4):515–533

Zhao Y, Sánchez Gómez MC, Pinto Llorente AM, Zhao L (2021) Digital competence in higher education: students’ perception and personal factors. Sustainability 13(21):12184. https://doi.org/10.3390/su132112184

Download references

Acknowledgements

The authors would like to thank the support from Tecnologico de Monterrey through the “Challenge-Based Research Funding Program 2022”. Project ID # I003-IFE001-C2-T3–T. Also, academic support from Writing Lab, Institute for the Future of Education, Tecnologico de Monterrey, México.

Author information

Authors and affiliations.

Tecnologico de Monterrey, Monterrey, Mexico

Silvia Farias-Gaytan & Maria-Soledad Ramirez-Montoya

University of Huelva, Huelva, Spain

Ignacio Aguaded

You can also search for this author in PubMed   Google Scholar

Contributions

Three authors contributed to the content of the article, conceptualizing the approach: IA, M-SR-M; supporting the study theoretically and methodologically: SF-G, M-SR-M; and discussing the data: SF-G, IA.

Corresponding author

Correspondence to Silvia Farias-Gaytan .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Informed consent

Additional information.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Farias-Gaytan, S., Aguaded, I. & Ramirez-Montoya, MS. Digital transformation and digital literacy in the context of complexity within higher education institutions: a systematic literature review. Humanit Soc Sci Commun 10 , 386 (2023). https://doi.org/10.1057/s41599-023-01875-9

Download citation

Received : 31 August 2022

Accepted : 20 June 2023

Published : 07 July 2023

DOI : https://doi.org/10.1057/s41599-023-01875-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Digital proficiency: assessing knowledge, attitudes, and skills in digital transformation, health literacy, and artificial intelligence among university nursing students.

• Ebtsam Aly Abou Hashish
• Hend Alnajjar

BMC Medical Education (2024)

Digital Literacy, Sustainable Development and Radiation Regulation: Policy and Information Systems Implications

• Farid Gasmi
• Paul Noumba Um

Information Systems Frontiers (2024)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Skip to main content
• Accessibility help

Information

We use cookies to collect anonymous data to help us improve your site browsing experience.

Click 'Accept all cookies' to agree to all cookies that collect anonymous data. To only allow the cookies that make the site work, click 'Use essential cookies only.' Visit 'Set cookie preferences' to control specific cookies.

Your cookie preferences have been saved. You can change your cookie settings at any time.

Literature Review on the Impact of Digital Technology on Learning and Teaching

This literature review was commissioned by the Scottish Government to explore how the use of digital technology for learning and teaching can support teachers, parents, children and young people in improving outcomes and achieving our ambitions for education in Scotland

Digital learning and raising attainment

Key findings

There is conclusive evidence that digital equipment, tools and resources can, where effectively used, raise the speed and depth of learning in science and mathematics for primary and secondary age learners. There is indicative evidence that the same can be said for some aspects of literacy, especially writing and comprehension. Digital technologies appear to be appropriate means to improve basic literacy and numeracy skills, especially in primary settings.

The effect sizes are generally similar to other educational interventions that are effective in raising attainment, though the use of digital learning has other benefits. Also, the extent of the effect may be dampened by the level of capability of teachers to use digital learning tools and resources effectively to achieve learning outcomes. More effective use of digital teaching to raise attainment includes the ability of teachers to identify how digital tools and resources can be used to achieve learning outcomes and adapting their approach, as well as having knowledge and understanding of the technology. This applies in all schools.

Where learners use digital learning at home as well as school for formal and non-formal learning activities these have positive effects on their attainment, because they have extended their learning time. This is particularly important for secondary age learners.

The assessment framework, set out in Annex 2 , identifies a number of educational benefits that digital learning and teaching has the potential to help learners aged 5 to 18 to realise, through the opportunity to learn in different ways, access more sources of information, and be tested and get feedback differently. In terms of raising attainment, these benefits include short term outcomes, such as having a greater feeling of control over learning and more confidence to practise a skill, through to medium term outcomes such as faster acquisition of knowledge and skills, and improved impacts in terms of learners achieving higher exam or test results where digital technology has been used.

In this section, the impact of digital technology on children's attainment in a range of areas is discussed, followed by the impact on aspects of numeracy, literacy and science learning.

Raising children's attainment

There is a substantial body of research that has examined the impact of digital tools and resources on children's attainment in a range of areas.

Higgins et al (2012) provide a summary of research findings from studies with experimental and quasi-experimental designs, which have been combined in meta-analyses to assess the impact of digital learning in schools. Their search identified 48 studies which synthesised empirical research of the impact of digital tools and resources on the attainment of school age learners (5-18 year olds).

They found consistent but small positive associations between digital learning and educational outcomes. For example, Harrison et al (2004) identified statistically significant findings, positively associating higher levels of ICT use with school achievement at each Key Stage in England, and in English, maths, science, modern foreign languages and design technology. Somekh et al (2007) identified a link between high levels of ICT use and improved school performance. They found that the rate of improvement in tests in English at the end of primary education was faster in ICT Test Bed education authorities in England than in equivalent comparator areas. However, Higgins et al note that while these associations show, on average, schools with higher than average levels of ICT provision also have learners who perform slightly higher than average, it may be the case that high performing schools are more likely to be better equipped or more prepared to invest in technology or more motivated to bring about improvement.

Higgins et al report that in general analyses of the impact of digital technology on learning, the typical overall effect size is between 0.3 and 0.4 - just slightly below the overall average for researched interventions in education (Sipe & Curlette, 1997; Hattie, 2008) and no greater than other researched changes to teaching to raise attainment, such as peer tutoring or more focused feedback to learners. The range of effect sizes is also very wide (-0.03 to 1.05),which suggests that it is essential to take into account the differences between technologies and how they are used.

Table 4: Summary of meta-analyses published between 2000 and 2012 (in Higgins et al 2012)

In an earlier meta-analysis, Liao et al (2007), considered the effects of digital tools and resources on elementary school learners' achievement in Taiwan. Synthesizing research comparing the effects of digital learning (equipment, tools and resources) with traditional instruction on elementary school learners' achievement, they considered quantitative and qualitative information from 48 studies including over 5,000 learners. Of the 48 studies, 44 (92%) showed positive effects in favour of a computer assisted intervention, while four (8%) were negative and favoured a traditional instruction method. Nearly 60% of the studies examined the effects of computer aided instruction for teaching mathematics or science. Another 11% of the studies concentrated on the teaching of reading and language. They found an overall positive effect size across all the studies of 0.45 (study-weighted grand mean), which is considered to be a moderate effect, with a wide range of effect sizes (from 0.25 to 2.67).

No significant differences were found between subject areas, and the authors suggest that digital learning has the potential to be implemented in many different subject areas. They found that the two subjects that showed the highest effects were reading and languages, which had a high positive effect size of 0.7. Studies using computer simulations also had higher effects. The authors suggest this may be because simulations can provide learners with the opportunity to engage in a learning activity which could not be replicated in a classroom.

More qualitative studies have identified how improvements in attainment are achieved. From a wide study of primary and secondary schools in England that were early adopters in using digital learning and teaching, Jewitt et al (2011) concluded that:

• Using digital resources provided learners with more time for active learning in the classroom;
• Digital tools and resources provided more opportunity for active learning outside the classroom, as well as providing self-directed spaces, such as blogs and forums, and access to games with a learning benefit;
• Digital resources provided learners with opportunities to choose the learning resources;
• The resources provided safer spaces for formative assessment and feedback.

The sections below focus on specific key areas of attainment: literacy, numeracy, and science learning.

There is a large body of research that has examined the impact of digital equipment, tools and resources on children's literacy. The effects are generally positive, though not as large as the effects found where digital learning is used to improve numeracy, and consistent in finding that ICT helps improve reading and writing skills, as well as developing speaking and listening skills.

Effect of context

Archer and Savage (2014) undertook a meta-analysis to reassess the outcomes presented in three previous meta-analyses considering the impact of digital learning on language and literacy learning: Slavin et al (2008 and 2009) and Torgenson and Zhu (2003). Overall they found a relatively small average positive effect size of 0.18, with a few of the studies having a negative effect and three studies showing moderate to large effect sizes. The authors found that programmes with a small number of participants tended to show larger effect sizes than larger programmes but that not all were statistically significant.

Archer and Savage sought to understand whether the context within which the digital tool or resource was used has an impact on outcomes. In particular, they examined whether training and support given to the teachers or other staff delivering the programme had an impact. The authors found that training and support could be identified in around half of the studies and that it did appear to have a positive impact on the effectiveness of the literacy intervention, with the average effect size rising to 0.57. The authors conclude that this indicates the importance of including implementation factors, such as training and support, when considering the relative effectiveness of digital learning and teaching.

Effect on specific literacy skills

In their meta-analysis, Higgins et al (2012) found that digital learning has a greater impact on writing than on reading or spelling. For example, Torgenson and Zhu (2003) reviewed the impact of using digital technology on the literacy competences of 5-16 year-olds in English and found effect sizes on spelling (0.2) and reading (0.28) much lower than the high effect size for writing (0.89).

In their meta-analysis of studies investigating the effects of digital technology on primary schools in Taiwan, Laio et al (2007) considered studies over a range of curriculum areas; 11 of which addressed the effects of using digital learning in one or more literacy competence. They found no significant differences in effect size between the different subject areas, suggesting the potential for digital technology to raise outcomes is equal across different subjects. However, they did note that the two areas that showed the highest effect sizes (over 0.7) were reading and comprehension.

Effect of specific digital tools and resources

Somekh et al (2007) evaluated the Primary School Whiteboard Expansion ( PSWB ) project in England. They found that the length of time learners were taught with interactive whiteboards ( IWB s) was a major factor in learner attainment at the end of primary schooling, and that there were positive impacts on literacy (and numeracy) once teachers had experienced sustained use and the technology had become embedded in pedagogical practice. This equated to improvements at Key Stage 2 writing (age 11), where boys with low prior attainment made 2.5 months of additional progress.

Hess (2014) investigated the impact of using e-readers and e-books in the classroom, among 9-10 year olds in the USA . The e-books were used in daily teacher-led guided reading groups, replacing traditional print books in these sessions. Teachers also regularly used the e-readers in sessions where the class read aloud, and e-readers were available to learners during the school day for silent reading. The study found a significant difference in reading assessment scores for the group using the e-readers. Scores improved for both male and female learners and the gap between males and females decreased.

The use of digital tools and resources also appears to affect levels of literacy. Lysenko and Abrami (2014) investigated the use of two digital tools on reading comprehension for elementary school children (aged 6-8) in Quebec, Canada. The first was a multimedia tool which linked learning activities to interactive digital stories. The tool included games to engage learners in reading and writing activities, and instructions were provided orally to promote listening comprehension. The second tool was a web-based electronic portfolio in which learners could create a personalised portfolio of their reading and share work with peers, teachers and parents to get feedback. The authors found that in classes where both tools were used together during the whole school year learners performed significantly better both in vocabulary and reading comprehension (with medium-level effect sizes) than learners in classes where the tools were not part of English language instruction.

Rosen and Beck-Hill (2012) reported on a study programme that incorporated an interactive core curriculum and a digital teaching platform. At the time of their report it was available for 9-11 year old learners in English language, arts and mathematics classes in Dallas, Texas. The online platform contained teaching and learning tools. Learners were assessed using standardised tests administered before the programme and after a year's participation. The results of increased achievement scores demonstrated that in each of the two school year groups covered, the experimental learners significantly outperformed the control learners in reading and maths scores. In observations in classrooms that used the programme, the researchers observed higher teacher-learner interaction, a greater number and type of teaching methods per class, more frequent and complex examples of differentiation processes and skills, more frequent opportunities for learner collaboration, and significantly higher learner engagement. The authors report that the teaching pedagogy observed in the classrooms differed significantly from that observed in more traditional classrooms. The teachers following the programme commented that the digital resources made planning and implementing 'differentiation' more feasible. This is differentiation of teaching in terms of content, process, and product, to reflect learners' readiness, interests, and learning profile, through varied instructional and management strategies.

Effect of the amount and quality of digital technology use

The uses of digital technology and access to it appear to be critical factors. Lee et al (2009) analysed how in the US 15-16 year-old learners' school behaviour and standardised test scores in literacy are related to computer use. Learners were asked how many hours a day they typically used a computer for school work and for other activities. The results indicated that the learners who used the computer for one hour a day for both school work and other activities had significantly better reading test scores and more positive teacher evaluations for their classroom behaviours than any other groups [5] . This was found while controlling for socio-economic status, which has been shown to be a predictor of test scores in other research. The analysis used data from a national 2002 longitudinal study, and it is likely that learners' usage of computers has increased and changed since that time.

Biagi and Loi (2013), using data from the 2009 Programme for International Student Assessment ( PISA ) and information on how learners used digital technology at school and at home (both for school work and for entertainment), assessed the relationship between the intensity with which learners used digital tools and resources and literacy scores. They examined uses for: gaming activities (playing individual or collective online games), collaboration and communication activities (such as linking with others in on-line chat or discussion forums), information management and technical operations (such as searching for and downloading information) and creating content, knowledge and problem solving activities (such as using computers to do homework or running simulations at school). These were then compared to country specific test scores in reading. The authors found a positive and significant relationship between gaming activity and language attainment in 11 of the 23 countries studied. For the other measures, where relationships existed and were significant, they tended to be negative.

The more recent PISA data study ( OECD , 2015, using 2012 results) also found a positive relationship between the use of computers and better results in literacy where it is evident that digital technology is being used by learners to increase study time and practice [6] . In addition, it found that the effective use of digital tools is related to proficiency in reading.

There is a large body of research which has examined the impact of digital equipment, tools and resources on children's numeracy skills and mathematical competences throughout schooling. Higgins et al (2012) found from their meta-analysis that effect sizes of tested gains in knowledge and understanding tend to be greater in mathematics and science than in literacy. The key benefits found relate to problem solving skills, practising number skills and exploring patterns and relationships (Condie and Monroe, 2007), in addition to increased learner motivation and interest in mathematics.

Effect on specific numeracy skills

Li and Ma's (2010) meta-analysis of the impact of digital learning on school learners' mathematics learning found a generally positive effect. The authors considered 46 primary studies involving a total of over 36,000 learners in primary and secondary schools. About half of the mathematics achievement outcomes were measured by locally-developed or teacher-made instruments, and the other half by standardized tests. Almost all studies were well controlled, employing random assignment of learners to experimental or control conditions.

Overall, the authors found that, on average, there was a high, significantly positive effect of digital technology on mathematics achievement (mean effect size of 0.71), indicating that, in general, learners learning mathematics with the use of digital technology had higher mathematics achievement than those learning without digital technology. The authors found that:

• Although the difference was small, younger school learners (under 13 years old) had higher attainment gains than older secondary school learners;
• Gains were more positive where teaching was more learner-centred than teacher-centred. In this regard, the authors differentiate between traditional models, where the teacher tends to teach to the whole class, and a learner-centred teaching model which is discovery-based (inquiry-oriented) or problem-based (application-oriented) learning;
• Shorter interventions (six months or less) were found to be more effective in promoting mathematics achievement than longer interventions (between six and 12 months). It is suggested that such gains in mathematics achievement are a result of the novelty effects of technology, as suggested in other research, and as learners get familiar with the technology the novelty effects tend to decrease;
• The authors found no significant effects from different types of computer technology on mathematics achievement. Whether it was used as communication media, a tutorial device, or exploratory environment, learners displayed similar results in their mathematics achievement;
• Equally, the authors found no significant relationship between the effect of using digital technology and the characteristics of learners included in the samples for studies, such as gender, ethnicity, or socio-economic characteristics.

The studies by Lee et al (2009) and Biagi and Loi (2013) found similar results for mathematics as they did for reading and literacy in relation to the use of digital equipment. Learners who used a computer at least one hour a day for both school work and other activities had significantly better mathematics test scores and more positive teacher evaluations for their classroom behaviour in mathematics classes than those who did not use the computer. Biagi and Loi (2013) found a significant positive relationship between intensity of gaming activity and maths test scores in 15 countries out of the 23 studied. As with language, the authors found that learners' total use of digital technologies was positively and significantly associated with PISA test scores for maths in 18 of the 23 countries studied.

Studies have found that using digital equipment for formal learning is also associated with increases in learners' motivation for learning mathematics. House and Telese (2011 and 2012) found that:

• For learners aged 13 and 14 in South Korea, for example, those who expressed high levels of enjoyment at learning mathematics, more frequently used computers in their mathematics homework. However, learners who more frequently played computer games and used the internet outside of school tended to report that they did not enjoy learning mathematics;
• Learners in the USA and Japan aged 13 and 14 who showed higher levels of algebra achievement also used computers more at home and at school for school work. Those who used computers most for other activities had lower test scores. In each of the USA and Japan they found that overall computer usage which included use for school work was significantly related to improvements in test scores.

Somekh et al (2007) found that, once the use of IWB s was embedded, in Key Stage 1 mathematics (age 7) in England, high attaining girls made gains of 4.75 months, enabling them to catch up with high attaining boys. In Key Stage 2 mathematics (age 11), average and high attaining boys and girls who had been taught extensively with the IWB made the equivalent of an extra 2.5 to 5 months' progress over the course of two years.

Digital tools and resources can also increase some learners' confidence in mathematics as well as their engagement in new approaches to learning and their mathematical competences. Overcoming learners' anxieties about mathematics and their competence in specific aspects of the subject are common concerns in teaching mathematics which hampers their ability to learn (reported in Huang et al 2014).

Huang et al (2014) researched the outcomes, in Taiwan, from a computer game simulating the purchase of commodities, from which 7 and 8 year-old primary school learners can learn addition and subtraction, and apply mathematical concepts. The model combined games-based learning with a diagnosis system. When the learner made a mistake, the system could detect the type of mistake and present corresponding instructions to help the learner improve their mathematical comprehension and application. The authors compared two learning groups: both used the game-based model but one without the diagnostic, feedback element. They found that the learning achievement post-test showed a significant difference and also that the mathematics anxiety level of the two learner groups was decreased by about 3.5%.

Passey (2011) found that among over 300 schools in England using Espresso digital resources, those that had been using them over a longer period made significantly greater increases in end of primary school numeracy test results than schools which were recent users.

Science learning

Effects on science knowledge and skills

In their meta-analysis, Laio et al (2007) considered 11 studies looking at the impact of digital technology on science learning. These had a moderate average effect size of 0.38 and generally had positive effects. Condie and Monroe (2007) identified that digital learning made science more interesting, authentic and relevant for learners and provided more time for post-experiment analysis and discussion.

In their study of the PISA data, Biagi and Loi (2013) found a significant positive relationship between learners' total use of digital equipment and science test scores in 21 of the 23 countries they studied. They also found evidence of a significant positive relationship between the intensity of using gaming activity and science scores in 13 of the 23 countries they studied. Somekh et al (2007) found that in primary school science all learners, except high attaining girls, made greater progress when given more exposure to IWB s, with low attaining boys making as much as 7.5 months' additional progress.

Effects of specific digital tools and resources

Digital tools and resources generally have a positive effect on learners' science learning. This can be seen from a number of studies assessing outcomes for learners in different stages of education.

Hung et al (2012) explored the effect of using multi-media tools in science learning in an elementary school's science course in Taiwan. Learners were asked to complete a digital storytelling project by taking pictures with digital cameras, developing the story based on the pictures taken, producing a film based on the pictures by adding subtitles and a background, and presenting the story. From the experimental results, the authors found that this approach improved the learners' motivation to learn science, their attitude, problem-solving capability and learning achievements. In addition, interviews found that the learners in the experimental group enjoyed the project-based learning activity and thought it helpful because of the digital storytelling aspect.

Hsu et al (2012) investigated the effects of incorporating self-explanation principles into a digital tool facilitating learners' conceptual learning about light and shadow with 8-9 year old learners in Taiwan. While they found no difference in the overall test scores of the experimental and control groups, they found a statistically significant difference in retention test scores. Those learners who had paid more attention to the self-explanation prompts tended to outperform those in the control group.

Anderson and Barnett's (2013) study, in the US , examined how a digital game used by learners aged 12-13 increased their understanding of electromagnetic concepts, compared to learners who conducted a more traditional inquiry-based investigation of the same concepts. There was a significant difference between the control and experimental groups in gains in knowledge and understanding of physics concepts. Additionally, learners in the experimental group were able to give more nuanced responses about the descriptions of electric fields and the influence of distance on the forces that change experience because of what they learnt during the game.

Güven and Sülün (2012) considered the effects of computer-enhanced teaching in science and technology courses on the structure and properties of matter, such as the periodical table, chemical bonding, and chemical reactions, for 13-14 year olds in Turkey. Their proposition was that computer-enhanced teaching can instil a greater sense of interest in scientific and technological developments, make abstract concepts concrete through simulation and modelling, and help to carry out some dangerous experiments in the classroom setting. They found a significant difference in achievement tests between the mean scores of the group of learners who were taught with the computer-enhanced teaching method and the control group who were taught with traditional teaching methods.

Belland (2009) investigated the extent to which a digital tool improved US middle school children's ability to form scientific arguments. Taking the premise that being able to construct and test an evidence-based argument is critical to learning science, he studied the impact of using a digital problem based learning tool on 12-14 year olds. Learners worked in small groups and were asked to develop and present proposals for spending a grant to investigate an issue relating to the human genome project. Those in the experimental group used an online system which structured the project into stages of scientific enquiry. The system prompted the learners to structure and organise their thinking in particular ways: by prompting the learners individually, sharing group members' ideas, tasking the group to form a consensus view, and prompting the group to assign specific tasks among themselves.

Using pre- and post- test scores to assess the impact on learners' abilities to evaluate arguments, Belland found a high positive effect size of 0.62 for average-achieving learners compared to their peers in the control group. No significant impacts were found for higher or lower-achieving learners. Belland suggests that for high-achieving learners, this may be because they already have good argument making skills and are already able to successfully structure how they approach an issue and gather evidence. The study also used qualitative information to consider how the learners used the digital tool and compared this to how learners in the control group worked. The author found that in the experimental group they made more progress and were more able to divide tasks up between them, which saved time. They also used the tool more and the teacher less to provide support.

Kucukozer et al (2009) examined the impact of digital tools on teaching basic concepts of astronomy to 11-13 year old school children in Turkey. Learners were asked to make predictions about an astronomical phenomenon such as what causes the seasons or the phases of the moon. A digital tool was used to model the predictions and display their results. The learners were then asked to explain the differences and the similarities between their predictions and their observations. In the prediction and explanation phase the learners worked in groups to discuss their ideas and come to a conclusion. In the observation phase they watched the 3D models presented by their teacher. Thereafter, they were asked to discuss and make conclusions about what they had watched. The authors found that instruction supported by observations and the computer modelling was significantly effective in bringing about better conceptual understanding and learning on the subject.

Ingredients of success

Where studies examine the process that brings about positive results from digital learning and teaching compared to traditional approaches, it is evident that these are more likely to be achieved where digital equipment, tools and resources are used for specific learning outcomes and built into a teaching model from the outset. This broadly supports Higgins et al's (2012) conclusions that:

• Digital technology is best used as a supplement to normal teaching rather than as a replacement for it;
• It is not whether technology is used (or not) which makes the difference, but how well the technology is applied to support teaching and learning by teachers;
• More effective schools and teachers are more likely to use digital technologies effectively than other schools.

Differences in effect sizes and the extent that learners achieve positive gains in attainment are ascribed by most authors of the studies above to:

• The quality of teaching and the ability of teachers to use the digital equipment and tools effectively for lessons;
• The preparation and training teachers are given to use equipment and tools;
• The opportunities teachers have to see how digital resources can be used and pedagogies adapted (Rosen and Beck-Hill, 2012; Belland, 2009).

Teachers have to adapt to learner-centred approaches to learning if they are to use digital tools and resources (Li and Ma, 2010).

As well as ensuring digital tools and resources are supporting learning goals, success appears to also be linked to some other factors:

• The availability of equipment and tools within schools (and at home);
• How learners use digital equipment. Higgins et al (2012) found that collaborative use of technology (in pairs or small groups) is usually more effective than individual use, though some learners - especially younger children - may need guidance in how to collaborate effectively and responsibly;
• The extent that teaching continues to innovate using digital tools and resources (Higgins et al, 2012).

Fullan (2013) suggested four criteria that schools should meet if their use of digital technology to support increased attainment is to be successful. These were that systems should be engaging for learners and teachers; easy to adapt and use; ubiquitous - with access to the technology 24/7; and steeped in real life problem solving.

Fullan and Donnelly (2013) developed these themes further, proposing an evaluation tool to enable educators to systematically evaluate new companies, products and school models, using the context of what they have seen as necessary for success. Questions focus on the three key criteria of pedagogy (clarity and quality of intended outcome, quality of pedagogy and the relationship between teacher and learner, and quality of assessment platform and functioning); system change (implementation support, value for money, and whole system change potential) and technology (quality of user experience/model design, ease of adaptation, and comprehensiveness and integration).

Email: Catriona Rooke

There is a problem

Thanks for your feedback

Your feedback helps us to improve this website. Do not give any personal information because we cannot reply to you directly.

Advertisement

Literature Review on the Factors Affecting Primary Teachers’ Use of Digital Technology

• Original research
• Open access
• Published: 27 July 2018
• Volume 25 , pages 115–128, ( 2020 )

Cite this article

You have full access to this open access article

• Marthese Spiteri   ORCID: orcid.org/0000-0002-0299-1878 1 &
• Shu-Nu Chang Rundgren 1  

43k Accesses

120 Citations

2 Altmetric

Explore all metrics

Digital technology is widely available in schools; however, results from international studies indicate that they are not effective toward students’ educational achievement. Teachers need to realise the potential of digital technology in their daily practises and use them well. However, teachers need training and guidelines to develop their expertise when using technology for teaching and learning. Failure to do so might result in students lacking the necessary coping skills for their future life in the information age. This literature review aimed to find out what factors affect primary teachers’ use of digital technology in their teaching practices, so as to suggest better training, which will eventually lead to a more guided and relevant use of technology in education. After applying the concept map to the data from the selected studies, four influencing factors were identified: teachers’ knowledge, attitudes and skills, which are also influenced by and influence the school culture. From these findings, recommendations on teacher training with technology and suggestions for further research are given.

Similar content being viewed by others

A systematic review of primary school teachers’ experiences with digital technologies curricula

Exploring factors influencing pre-service teacher’s digital teaching competence and the mediating effects of data literacy: empirical evidence from China

Attitudes of Preservice Teachers About the Use of Digital Technologies and the Use of Digital Technologies in Primary Education

Avoid common mistakes on your manuscript.

1 Introduction

The use of digital technology by teachers from early years in primary education makes learning a more familiar experience for students today. Using digital technology is also seen as the application of information and communication technology (ICT) by researchers and teachers in the field of education, where ICT is defined as “forms of technology that are used to transmit, process, store, create, display, share or exchange information by electronic means” (UNESCO 2007 , p. 1). Consequently, teachers use digital resources to enhance learning by preparing lessons via powerpoint presentations and word document, or create communication channels for students and parents through social media and e-mails. Research has shown that teachers’ ability to use technology to plan and implement student-centred learning activities and effective communication with parents can enhance children’s learning (OECD 2010 ; Wake and Whittingham 2013 ; UNESCO 2011 ). Teachers’ use of digital technology is also recognized as important for children’s future employment and participation in society (European Parliament and the Council 2006 ; Leu et al. 2004 ; UNESCO 2011 ). However, although we are living in a technology-dominated society, the school might be the only place for some children to use digital technology since they have different family backgrounds and cultures (OECD 2010 ). Still, such technology savvy students must be appreciated and this requires new attitudes from the teachers such as to learn with and from the students, and further, to know how to facilitate learning with technology (UNESCO 2011 ; Wake and Whittingham 2013 ).

Nearly all teachers in europe use ICT to prepare lessons and in schools, its availability was for four out of five students (EU 2013 ). However, availability of digital technology and increased use by the teachers has not resulted in progress in relation to students’ educational achievement (OECD 2014 ). This shows that teachers need guided training on how to use digital technologies and take the right decisions especially when they change so quickly (Mishra and Koehler 2006 ). There are various differences among teachers, not only in their digital skills (Liang et al. 2010 ; Wang et al. 2011 ) and knowledge on ICT (Aesaert et al. 2013 ), but also on their attitudes towards the use of technology in their practise (Kim and Keller 2011 ; Lemon and Garvis 2016 ; Wake and Whittingham 2013 ; Wastiau et al. 2013 ).

2 Methodology

In this review, a synthesis of studies related to the use of digital technology was conducted to illustrate the factors affecting technology integration and to develop the definition of a digitally competent teacher. As of august 2016, the keywords “ICT”, “primary teacher” and “technology integration” were searched in three electronic databases: springer link, jstor and ebscohost since these are three of the most common academic databases ( https://en.wikipedia.org/wiki/list_of_academic_databases_and_search_engines ) which researchers use and were chosen to provide an over-view of the focus concerning teachers’ use of digital technology presented in this article. In total, the search generated 947 studies, the abstract, introduction and conclusion of each article were read. After eliminating duplicate studies, a total of 409 studies remained. The inclusion criteria for selecting the studies for this analysis were that (a) the subjects were pre-service or in-service teachers, (b) research included primary education, (c) it was an empirical research and (d) it was published in a peer-reviewed journal. After this procedure, 27 studies were selected as shown in Table  1 , which served as the source of data for our analysis presented in this article.

2.1 Findings

The review covered studies at primary school level, and included three methodologies; quantitative, qualitative and mixed methods. The percentage of quantitative studies was 40.8% where information was sourced from surveys. 37.0% were qualitative studies where data was collected from online communication, classroom observations, project based and inquiry learning activities, teachers’ reflections and evaluation on activities, interviews, one-to-one discussion meetings, focus groups and informal conversations. The mixed methods studies accounted for 22.2%. After reading the articles, a concept map was developed (Fig.  1 ) to categorize the factors which affect primary teachers’ use of digital technology.

The emerging four factors affecting primary teachers’ technology integration

A total of four main areas were identified including the school culture, teachers’ knowledge, attitude and skills. Descriptions of the results found in the studies, characteristics and classification are shown in Table  2 .

2.2 School Culture

Teachers’ knowledge, attitudes and skills were both influenced by and influenced the school culture, since there was a reciprocal relationship between the school culture and the teacher. A school culture empowering quality teacher training allowed primary teachers to work collaboratively, reflect on the process and share the new knowledge (Hsu and Kuan 2013 ; Tondeur et al. 2016 ). It was suggested that in primary schools such learning opportunities must be provided for the teachers (Getenet et al. 2016 ). Working on local projects with digital technology, contributed towards teachers’ training and innovation (Tondeur et al. 2016 ) especially when adequate resources were available and feedback was provided during workshops on lesson design and teacher instruction (Getenet et al. 2016 ).

Furthermore, the school culture effected the teachers’ attitudes towards technology integration (Apeanti 2016 ). When the teachers were respected and valued for their work, they were motivated to use technology more often (Tondeur et al. 2016 ). Findings indicated that for a supportive school culture in primary education, digitally competent leaders, technical help and encouragement were required to integrate technology (Kim and Keller 2011 ; Omwenga and Nyabero 2016 ; Tezci 2011 ). Hsu and kuan ( 2013 ) found that the amount of time allocated to training and the teachers’ perceived support from the school, were the two most influential factors to technology integration. Further, when teachers collaborated and shared their projects more ideas were developed (Tondeur et al. 2016 ).

2.3 Teachers’ Knowledge

In the category of teachers’ knowledge, various areas such as teachers’ knowledge on themselves, on the students and on technology itself were identified. Teachers’ knowledge was related to what, how and why technology was used.

Knowledge on how to integrate technology in the classroom was reported by various researchers (Gu et al. 2013 ; Mishra and Koehler 2006 ; Orlando and Attard 2016 ). It was not enough to provide primary teachers with new technological tools; they also needed to know how to use them and the strategies for teaching purpose to meet the various needs of the students. For example, during digital story telling students were given the opportunity to safely share their stories, when using different digital approaches to express themselves (Duveskog et al. 2012 ).

Gu et al. ( 2013 ) found that there were differences between how teachers and students used technology and how they perceived its importance. Consequently, this knowledge could help teachers prepare more motivating lessons with adequate resources, considering also the affordances of multimodal activity that could be beneficial in reaching the digitally native students (Lenters and Winters 2013 ; Wake and Whittingham 2013 ). Besides, the new generation of teachers are themselves the digital natives, and could better understand and communicate with these students (Orlando and Attard 2016 ).

Mishra and Koehler ( 2006 ) further illustrated the teachers’ knowledge on the use of technology for the teaching purpose, in the technological, pedagogical and content knowledge (TPACK) framework, where effective technology integration occurs at the intersection area, of teachers’ technology knowledge (TK), pedagogy knowledge (PK) and content knowledge (CK). TPACK was not about developing expertise in individual technologies, but rather a mind set to help teachers plan effective technology integration, within the areas of technology, pedagogy and content (Dalton 2012 ). Research differentiated between knowledge on traditional curricula and curricula with technology; the latter were more complex and varied, and allowed for innovation in the subject content presented in the classroom (Aesaert et al. 2013 ).

It was found that one of the important factors to integrate technology was the teachers’ readiness to use it, when novice teachers experienced higher readiness than veterans (Inan and Lowther 2010 ). However, the use of technology was not influenced by the teachers’ age but by the number of years in service, where teachers with less than five years teaching experience, used technology less than those with longer service (Gu et al. 2013 ).

2.4 Teachers’ Attitude

Teachers’ attitudes toward the use of digital technology, in primary education were found to be related to teachers’ confidence, beliefs and self-efficacy, and with a significant relation to school culture.

Studies indicated that initially elementary teachers did not feel confident when teaching with technology and that their self-efficacy beliefs improved with time, when they observed and worked with their colleagues (Al-awidi and Alghazo 2012 ; Wake and Whittingham 2013 ). Technology was looked upon as a tool to help teachers deliver a better lesson, but with experience, it was considered for the educational development of the students (Wake and Whittingham 2013 ). Further training preservice teachers with explicit instructions, fostered positive changes in their beliefs and behaviours towards technology integration (Rehmat and Bailey 2014 ). Research on pre-service teachers’ self-efficacy beliefs could give insight on their confidence to integrate technology and allow for better pre-service teacher training (Lemon and Garvis 2016 ) while on the other hand effective technology training could contribute towards developing teachers’ positive attitudes and perceptions (Apeanti 2016 ). It was noted that novice primary teachers experienced device conflict since they were still learning how to use technology in their teaching practice (Orlando and Attard 2016 ) which indicated that they were not experts in technology integration (Wake and Whittingham 2013 ).

In a study conducted between two pre-service teachers’ cohorts in 2006 and 2012, barak ( 2014 ) found that teachers’ aptitudes towards the use of technology changed. In the first cohort, teachers depicted digital technologies as inefficient tools that weakened the teachers’ authority and brought about distractions in the classroom. On the other hand, the second teachers’ cohort indicated that digital technologies were beneficial to exploit teaching and learning experiences (Barak 2014 ). Consequently, in a recent study, primary school teachers showed great enthusiasm when using blogs to teach a foreign language (Al-Qallaf and Al-Mutairi 2016 ). It was observed that students were more motivated, worked independently and wrote longer sentences with fewer spelling and grammar mistakes (Al-Qallaf and Al-Mutairi 2016 ).

Generally, teachers’ attitudes and confidence in using technology did not depend only on its availability, as confident teachers exploited what technology was available for the benefits of the students (Wastiau et al. 2013 ). Teachers’ confidence and belief that technology was important for students’ learning were the main factors, which contributed towards technology integration, and also affected the students’ confidence to use it (Al-awidi and Alghazo 2012 ; Wastiau et al. 2013 ). However, Tezci ( 2011 ) concluded that having a computer and access to the internet were perceived by the teachers as influencing factors in enhancing the school culture towards technology integration.

2.5 Teachers’ Skills

Primary school teachers’ skills were mainly related to information management and visual literacy, to enhance their teaching practices. It was argued that teachers must consider multimodal activity for reading and writing activity (Wake and Whittingham 2013 ).

When using technology with fifth-grade students, teachers lacked the visual literacy (Wang et al. 2011 ) and the skill to choose the best information provided on the internet (Al-Qallaf and Al-Mutairi 2016 ). This was also evident after inquiring on sixth graders’ use of blogs, ms power point (ppt) and the internet; it was also found that students lacked the skills to assess information, take notes and synthesize the information (Al-qallaf and Al-mutairi 2016 ; Wang et al. 2011 ). Furthermore, in chile, Brun ( 2014 ) found that teachers only used a few digital resources mostly projectors and computers, where the ‘traditional’ teaching and learning methods were applied. Sun et al. ( 2014 ) stated that the way teachers interacted with the students, when giving instructions and asking questions with technology, influenced the students’ understanding of new concepts and encouraged more collaborative inquiry. It was suggested that in order to move away from the traditional ways of teaching and learning, teachers must apply inquiry activities, such as project based learning and problem based learning, which are more child-centred and constructivist in their approach (Tondeur et al. 2016 ).

The integration of technology challenged the teachers’ traditional methods of teaching and developed new skills such as applying the constructivist approach to teaching, learning, and orchestration , where the teacher fulfilled various roles and systematically organised different activities with technology, depending on students’ needs (Wake and Whittingham 2013 ). Nevertheless, teaching methods were noted to be evolving rather than in revolution with traditional teaching methods and depended on the type of digital technology being used (Orlando and Attard 2016 ). At primary level a distinction was noted between fixed and mobile technology, such as the interactive whiteboard (IWB) and ipads, where the former could be used with traditional ways of teaching, but the latter, due to their mobility required different classroom management and changes in teachers’ and students’ roles (Orlando and Attard 2016 ). Further, Anastasiades and Vitalaki ( 2011 ) found that teachers who daily-integrated digital technology in their practices found it easier to promote safety issues related to the internet by discussing the topic with the students.

3 Conclusion and Recommendations

Teachers found it difficult to adapt to new digital tools continuously, especially when previous lessons worked well, and to accept that some students might be more skilful in using a new digital technology than themselves (Morsink et al. 2010 ). Ultimately, it was found that preservice training in technology ensured better skilled teachers, with the right attitudes to develop digital technology in the school curriculum (Aesaert et al. 2013 ; Lemon and Garvis 2016 ).

This review of 27 articles is timely to highlight the importance of teachers’ professional development in the use of digital technology and how it can be sustainably developed during their school practices. As illustrated, teachers required not only the skill to use digital technology but also the right attitudes and the knowledge on how to apply these skills. It was revealed that the application of digital competence to primary school teacher’s professional development is in line with Ferrari ( 2013 ), where she indicated that for an effective use of digital technology, a citizen required digital competence (DC) which include the knowledge, the skills and the right attitude to use technology in five areas, namely to manage information, to communicate, to create content, for safety and to solve problems. Thus, like any other citizens, for effective use of technology, primary teachers need to apply such DC areas in their practices. These DC areas could be the indicators to measure the effective use of technology since analysing these areas could give evidence on how teachers are performing with technology. Table  3 illustrates the four factors affecting the teachers’ use of digital technology; the school culture, the teachers’ knowledge, attitudes and skills, which are cross-linked with the areas of DC.

Recommendations from the primary years’ teachers’ perspectives are discussed and suggested as guidelines for the teachers’ professional development in the sustainable use of digital technology in schools.

4 To Manage Information

When using digital technology, teachers need to know how to manage information . The right questions need to be asked and best sources of information searched, and then the data obtained is evaluated, synthesized and communicated to others (Leu et al. 2004 ). Teachers need to be able to teach their students how to search for information, which includes making use of various search engines and reading various articles, and then critically evaluate the results (Kinzer 2010 ). When teachers apply this strategy for information management in class, students learn how to think critically about information found on the internet. Research indicated that students lacked this skill (Wang et al. 2011 ) and it is the responsibility of the teacher to teach this area of DC. Knowing how to manage information will allow better life choices and safer social environment. In this digital environment, teacher training must consider multimodal ways of interacting with information since it had changed from print to multiple modes; including images, sound, video clips, text and kinaesthetic (Kress 2003 ; Lenters and Winters 2013 ; Wake and Whittingham 2013 ).

4.1 To Communicate

Studies indicated that the school culture was considered an important factor in technology integration, especially when the school management team offered encouragement and technical help to the teachers (Tondeur et al. 2016 ; Tezci 2011 ). When the teachers communicated and shared their teaching material, they felt confident and secure since their innovative approaches were accepted (Tondeur et al. 2016 ). Teachers felt their work was worthwhile when they were contributing to the local needs thus promoting a better school culture (Duveskog et al. 2012 ). Teachers should be encouraged to share their work to gain and give feedback and others can learn from their experiences. Various means can be used such as learning platforms, mobile phones and the internet.

In this review, various studies considered teachers’ communication and working together as a requirement for quality teacher training (Tondeur et al. 2016 ). Communicating with the students’ parents or guardians offers a great opportunity for teachers, to better design lessons tailored to students’ needs and activities initiated at school could be further discussed at home. This interest will foster more sharing between students’ different backgrounds and more inclusion especially where there is a language barrier. Teacher training in this DC area of communication could be useful since teachers can construct new knowledge, reflect on the process, give, and receive feedback. Through reflection, teachers could critically examine their work, understand new conceptions of constructivist teaching and learning, and accept new roles of teaching from an instructive to a more constructive approach (Brun ( 2014 ; Tondeur et al. 2016 ). In this environment, students can also provide feedback to their teachers and which is a learning opportunity (Wake and Whittingham 2013 ).

4.2 To Create New Content

Various studies mentioned the importance of creating and constructing new knowledge when using digital technology (Anastasiades and Vitalaki 2011 ; Sun et al. 2014 ). Further TPACK was considered a type of knowledge, which expert teachers applied when using technology, and involved the interplay of the three areas of technology, pedagogy and content knowledge. Developing teachers’ TPACK can result in creation and innovation in teaching and learning since technology changes how the teacher teaches and eventually the content as well (Mishra and Koehler 2006 ). Teacher training needs to acknowledge that content knowledge is always changing since information on the internet changes continuously and teachers need to adapt their pedagogical instruction. In this constructivist environment, the teacher is learning with students and develops the curriculum as she or he gains insight from the students (Duveskog et al. 2012 ; Tondeur et al. 2016 ). Teacher training on inquiry and pbl can be recommended for further teacher training in technology integration.

When using technology for teaching and learning, primary years’ teachers must be aware of legal frameworks to act ethically and responsibly. They can overcome concerns over internet safety when they are provided with the right information and given some ideas of how they can safely integrate technology (Anastasiades and Vitalaki 2011 ). The school management can filter unwanted websites, however if we want to protect the students from bad experiences on the internet, teachers must educate them. School management can organise talks with all those involved within the school community and make explicit the school’s ICT policies. Knowing these boundaries, everyone can use technology more confidently.

4.4 To Solve Problems

Evaluating and problem solving in a digital environment requires the teacher to recognize the difficulties related to a problem and subsequently assess the information to solve the problem and share the conclusions with others (Leu et al. 2004 ). Several studies indicated how teachers could make use of various digital activities to encourage problem solving; some of the mentioned activities were computer simulations, scenarios, blogs and inquiry activities (Al-Qallaf and Al-Mutairi 2016 ; Morsink et al. 2010 ; Tondeur et al. 2016 ). Training in this area is beneficial since students are already familiar with simulations through digital games and this could encourage learning. Training preservice teachers to solve problems with technology ensured better skilled teachers with the right attitudes to develop the curriculum later on in their profession (Al-Awidi and Alghazo 2012 ; Kim and Keller 2011 ; Wake and Whittingham 2013 ).

Since technology is continuously evolving, training with new tools must continuously be provided and this is quite challenging for the teachers, as they need to continuously adapt their teaching to new digital tools. Several studies in this review highlighted that teachers need the knowledge, the skills and the right attitudes to use technology (Barak 2014 ; Morsink et al. 2010 ). Teachers need to have the disposition to experiment with new technologies to capture the interests of all the students in the class (Kinzer 2010 ). This will result in more inquiry and innovation in learning (Sun et al. 2014 ). As stated by Dalton ( 2012 ) the teacher must reflect on his or her own strengths and interests, activities that she or he is already comfortable with and then develop the lessons with the use of digital technology. This requires time and collaborative training and feedback and a supportive school culture.

*Aesaert, K., Vanderlinde, R., Tondeur, J., & van Braak, J. (2013). The content of educational technology curricula: a cross-curricular state of the art. Education Technology Research and Development, 61 (1), 131–151.

Article   Google Scholar  

*Al-Awidi, H. M., & Alghazo, I. M. (2012). The effect of student teaching experience on preservice elementary teachers’ self-efficacy beliefs for technology integration in the UAE. Educational Technology Research and Development, 60 (5), 923–941.

*Al-Qallaf, C. L., & Al-Mutairi, A. S. F. (2016). Digital literacy and digital content supports learning. Electronic Library, 34 (3), 522–547.

*Anastasiades, P. S., & Vitalaki, E. (2011). Promoting Internet safety in Greek primary schools: The teacher’s role. Journal of Educational Technology & Society, 14 (2), 71–80.

Google Scholar  

*Apeanti, W. O. (2016). Contributing factors to pre-service mathematics teachers’ e-readiness for ICT integration. International Journal of Research in Education and Science (IJRES), 2 (1), 223–238.

*Barak, M. (2014). Closing the gap between attitudes and perceptions about ICT-enhanced learning among pre-service STEM teachers. Journal of Science Education and Technology, 23 (1), 1–14.

*Brun, M. (2014). Learning to become a teacher in the 21st century: ICT integration in initial teacher education in Chile. Journal of Educational Technology & Society, 17 (3), 222–238.

Dalton, B. (2012). Multimodal composition and the common core standards. The Reading Teacher, 66 (4), 333–339.

*Duveskog, M., Tedre, M., Sedano, C. I., & Sutinen, E. (2012). Life planning by digital storytelling in a primary school in rural Tanzania. Journal of Educational Technology & Society, 15 (4), 225–237.

European Parliament and the Council. (2006). Recommendation of the European Parliament and the Council of 18th December 2006 on key competences for lifelong learning. Official Journal of the European Union , L394/310.

European Union. (2013). Survey of schools: ICT in education . Benchmarking access, use and attitudes to technology in Europe’s schools. Belgium: European Union.

Ferrari, A. (2013). DIGCOMP: A framework for developing and understanding digital competence in Europe . Seville: Joint Research Centre.

*Getenet, S. T., Beswick, K., & Callingham, R. (2016). Professionalizing in-service teachers’ focus on technological pedagogical and content knowledge. Education and Information Technology, 21 (1), 19–34.

*Gu, X., Zhu, Y., & Guo, X. (2013). Meeting the “digital natives”: Understanding the acceptance of technology in classrooms. Journal of Educational Technology & Society, 16 (1), 392–402.

*Hsu, S., & Kuan, P. (2013). The impact of multilevel factors on technology integration: The case of Taiwanese grade 1–9 teachers and schools. Educational Technology Research and Development, 61 (1), 25–50.

*Inan, F., & Lowther, D. (2010). Factors affecting technology integration in K-12 classrooms: A path model. Educational Technology Research and Development, 58 (2), 137–154.

*Kim, C., & Keller, J. (2011). Towards technology integration: The impact of motivational and volitional email messages. Educational Technology Research and Development, 59 (1), 91–111.

Kinzer, Ch. K. (2010). Considering literacy and policy in the context of digital environments. Language Arts , 88 (1), 51–61.

Kress, C. (2003). Literacy in the New Media Age . London: Routledge.

Book   Google Scholar  

*Lemon, N., & Garvis, S. (2016). Pre-service teacher self-efficacy in digital technology. Teachers and Teaching, 22 (3), 387–408.

*Lenters, K., & Winters, K. (2013). Fracturing writing spaces: Multimodal storytelling ignites process writing. The Reading Teacher, 67 (3), 227–237.

Leu, D. J., Jr., Kinzer, C. K., Coiro, J. L., & Cammack, D. W. (2004). Towards a theory of new literacies emerging from the Internet and other information and communication technologies. In R. B. Ruddell & N. J. Unrau (Eds.), Theoretical models and processes of reading (5th ed., pp. 1570–1611). Newark, NJ: International Reading Association.

*Liang, L., Ebenezer, J., & Yost, D. (2010). Characteristics of pre-service teachers’ online discourse: The study of local streams. Journal of Science Education and Technology, 19 (1), 69–79.

Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108 (6), 1017–1054.

*Morsink, P., Hagerman, M. S., Heintz, A., Boyer, D. M., Harris, B., Kereluik, K., et al. (2010). Professional development to support TPACK technology integration: The initial learning trajectories of thirteen fifth- and sixth-grade educators. The Journal of Education, 191 (2), 3–16.

OECD. (2010). Are the new millennium learners making the grade? Technology use and educational performance in PISA . Derby: Centre for Educational Research and Innovation.

OECD. (2014). PISA 2012 results in focus. What 15 - year - olds know and what they can do with what they know. OECD: Paris.

*Omwenga, E., Nyabero, C., & Okioma. (2016). Assessing the influence of the PTTC Principal’s competency in ICT on the teachers’ integration of ICT in teaching Science in PTTCs in Nyanza Region, Kenya. 6 (35).

*Orlando, J., & Attard, C. (2016). Digital natives come of age: the reality of today’s early career teachers using mobile devices to teach mathematics. Mathematics Education Research Journal, 28 (1), 107–121.

*Rehmat, A., & Bailey, J. (2014). Technology integration in a science classroom: Preservice teachers’ perceptions. Journal of Science Education and Technology, 23 (6), 744–755.

*Sun, D., Looi, C., & Xie, W. (2014). Collaborative inquiry with a web-based science learning environment: When teachers enact it differently. Journal of Educational Technology & Society, 17 (4), 390–403.

*Tezci, E. (2011). Turkish primary school teachers’ perceptions of school culture regarding ICT integration. Educational Technology Research and Development, 59 (3), 429–443.

*Tondeur, J., Forkosh-Baruch, A., Prestridge, S., Albion, P., & Edirisinghe, S. (2016). Responding to challenges in teacher professional development for ICT integration in education. Educational Technology & Society, 19 (3), 110–120.

UNESCO. (2007). The UNESCO ICT in Education Programme . Bangkok: UNESCO.

UNESCO. (2011). UNESCO ICT Competency Framework for Teachers . Paris: UNESCO.

*Wachira, P., & Keengwe, J. (2010). Technology integration barriers: Urban school mathematics teachers’ perspectives. Journal of Science Education and Technology, 20 (1), 17–25.

*Wake, D., & Whittingham, J. (2013). Teacher candidates’ perceptions of technology supported literacy practices. Contemporary Issues in Technology and Teacher Education, 13 (3), 175–206.

*Wang, C., Ke, Y., Wu, J., & Hsu, W. (2011). Collaborative action research on technology integration for science learning. Journal of Science Education and Technology, 21 (1), 125–132.

*Wastiau, P., Blamire, R., Kearney, C., Quittre, V., Van de Gaer, E., & Monseur, C. (2013). The use of ICT in education: A survey of schools in Europe. European Journal of Education, 48 (1), 11–27.

References marked with an asterisk (*) indicate the articles used in the review.

Download references

Author information

Authors and affiliations.

Department of Education, Stockholm University, 10691, Stockholm, Sweden

Marthese Spiteri & Shu-Nu Chang Rundgren

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Marthese Spiteri .

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Spiteri, M., Chang Rundgren, SN. Literature Review on the Factors Affecting Primary Teachers’ Use of Digital Technology. Tech Know Learn 25 , 115–128 (2020). https://doi.org/10.1007/s10758-018-9376-x

Download citation

Published : 27 July 2018

Issue Date : March 2020

DOI : https://doi.org/10.1007/s10758-018-9376-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Primary teacher
• Teacher training
• Digital competence
• Technology integration
• School culture
• Find a journal
• Publish with us
• Track your research

Literature Review on Impact of Digital Technology to Learning and Thinking and Implications to Pedagogy

6 Pages Posted: 6 Apr 2022

Princess Gerbie Durante

Bulacan State University

Date Written: April 6, 2020

The study aims to investigate how digital technology affects the brain and its impact to the way digital natives learn and think through an extensive literature review. Moreover, results of the literature review shall be used to develop a conceptual framework on the impact of digital technology to learning and thinking and its implications to pedagogy. Ultimately, the paper highlights the need for teachers to be well-equipped on how to use digital technology in the classroom together with direct instruction to scaffold skills needed to develop interpersonal skills, communicative proficiency and deep thinking with 21st century skills of skimming and scanning. The framework to be developed illustrates a balance in the use of brain-based research and digital technology to achieve a classroom environment where digital natives will be able fully develop their brain and at the same time address the requirements of the 21st century world in which they belong. The conceptual paper further addressed how technology can be used to mediate learning and how teachers evaluate and use technology in the classroom.

Keywords: Brain Research, Digital Technology, Digital natives, 21st century skill

JEL Classification: Y20, I20, Y50, Y80

Suggested Citation: Suggested Citation

Princess Gerbie Durante (Contact Author)

Bulacan state university ( email ).

Capitol Compound Bgry Guinhawa Mc Arthur Hway Malolos City, Bulacan 3000 Philippines

Do you have a job opening that you would like to promote on SSRN?

Paper statistics, related ejournals, social sciences education ejournal.

Subscribe to this fee journal for more curated articles on this topic

Educational Technology, Media & Library Science eJournal

Educational impact & evaluation research ejournal.

Subscribe to this free journal for more curated articles on this topic

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Springer Nature - PMC COVID-19 Collection

Impacts of digital technologies on education and factors influencing schools' digital capacity and transformation: A literature review

Stella timotheou.

1 CYENS Center of Excellence & Cyprus University of Technology (Cyprus Interaction Lab), Cyprus, CYENS Center of Excellence & Cyprus University of Technology, Nicosia-Limassol, Cyprus

Ourania Miliou

Yiannis dimitriadis.

2 Universidad de Valladolid (UVA), Spain, Valladolid, Spain

Sara Villagrá Sobrino

Nikoleta giannoutsou, romina cachia.

3 JRC - Joint Research Centre of the European Commission, Seville, Spain

Alejandra Martínez Monés

Andri ioannou, associated data.

Data sharing not applicable to this article as no datasets were generated or analysed during the current study.

Digital technologies have brought changes to the nature and scope of education and led education systems worldwide to adopt strategies and policies for ICT integration. The latter brought about issues regarding the quality of teaching and learning with ICTs, especially concerning the understanding, adaptation, and design of the education systems in accordance with current technological trends. These issues were emphasized during the recent COVID-19 pandemic that accelerated the use of digital technologies in education, generating questions regarding digitalization in schools. Specifically, many schools demonstrated a lack of experience and low digital capacity, which resulted in widening gaps, inequalities, and learning losses. Such results have engendered the need for schools to learn and build upon the experience to enhance their digital capacity and preparedness, increase their digitalization levels, and achieve a successful digital transformation. Given that the integration of digital technologies is a complex and continuous process that impacts different actors within the school ecosystem, there is a need to show how these impacts are interconnected and identify the factors that can encourage an effective and efficient change in the school environments. For this purpose, we conducted a non-systematic literature review. The results of the literature review were organized thematically based on the evidence presented about the impact of digital technology on education and the factors that affect the schools’ digital capacity and digital transformation. The findings suggest that ICT integration in schools impacts more than just students’ performance; it affects several other school-related aspects and stakeholders, too. Furthermore, various factors affect the impact of digital technologies on education. These factors are interconnected and play a vital role in the digital transformation process. The study results shed light on how ICTs can positively contribute to the digital transformation of schools and which factors should be considered for schools to achieve effective and efficient change.

Introduction

Digital technologies have brought changes to the nature and scope of education. Versatile and disruptive technological innovations, such as smart devices, the Internet of Things (IoT), artificial intelligence (AI), augmented reality (AR) and virtual reality (VR), blockchain, and software applications have opened up new opportunities for advancing teaching and learning (Gaol & Prasolova-Førland, 2021 ; OECD, 2021 ). Hence, in recent years, education systems worldwide have increased their investment in the integration of information and communication technology (ICT) (Fernández-Gutiérrez et al., 2020 ; Lawrence & Tar, 2018 ) and prioritized their educational agendas to adapt strategies or policies around ICT integration (European Commission, 2019 ). The latter brought about issues regarding the quality of teaching and learning with ICTs (Bates, 2015 ), especially concerning the understanding, adaptation, and design of education systems in accordance with current technological trends (Balyer & Öz, 2018 ). Studies have shown that despite the investment made in the integration of technology in schools, the results have not been promising, and the intended outcomes have not yet been achieved (Delgado et al., 2015 ; Lawrence & Tar, 2018 ). These issues were exacerbated during the COVID-19 pandemic, which forced teaching across education levels to move online (Daniel, 2020 ). Online teaching accelerated the use of digital technologies generating questions regarding the process, the nature, the extent, and the effectiveness of digitalization in schools (Cachia et al., 2021 ; König et al., 2020 ). Specifically, many schools demonstrated a lack of experience and low digital capacity, which resulted in widening gaps, inequalities, and learning losses (Blaskó et al., 2021 ; Di Pietro et al, 2020 ). Such results have engendered the need for schools to learn and build upon the experience in order to enhance their digital capacity (European Commission, 2020 ) and increase their digitalization levels (Costa et al., 2021 ). Digitalization offers possibilities for fundamental improvement in schools (OECD, 2021 ; Rott & Marouane, 2018 ) and touches many aspects of a school’s development (Delcker & Ifenthaler, 2021 ) . However, it is a complex process that requires large-scale transformative changes beyond the technical aspects of technology and infrastructure (Pettersson, 2021 ). Namely, digitalization refers to “ a series of deep and coordinated culture, workforce, and technology shifts and operating models ” (Brooks & McCormack, 2020 , p. 3) that brings cultural, organizational, and operational change through the integration of digital technologies (JISC, 2020 ). A successful digital transformation requires that schools increase their digital capacity levels, establishing the necessary “ culture, policies, infrastructure as well as digital competence of students and staff to support the effective integration of technology in teaching and learning practices ” (Costa et al, 2021 , p.163).

Given that the integration of digital technologies is a complex and continuous process that impacts different actors within the school ecosystem (Eng, 2005 ), there is a need to show how the different elements of the impact are interconnected and to identify the factors that can encourage an effective and efficient change in the school environment. To address the issues outlined above, we formulated the following research questions:

a) What is the impact of digital technologies on education?

b) Which factors might affect a school’s digital capacity and transformation?

In the present investigation, we conducted a non-systematic literature review of publications pertaining to the impact of digital technologies on education and the factors that affect a school’s digital capacity and transformation. The results of the literature review were organized thematically based on the evidence presented about the impact of digital technology on education and the factors which affect the schools’ digital capacity and digital transformation.

Methodology

The non-systematic literature review presented herein covers the main theories and research published over the past 17 years on the topic. It is based on meta-analyses and review papers found in scholarly, peer-reviewed content databases and other key studies and reports related to the concepts studied (e.g., digitalization, digital capacity) from professional and international bodies (e.g., the OECD). We searched the Scopus database, which indexes various online journals in the education sector with an international scope, to collect peer-reviewed academic papers. Furthermore, we used an all-inclusive Google Scholar search to include relevant key terms or to include studies found in the reference list of the peer-reviewed papers, and other key studies and reports related to the concepts studied by professional and international bodies. Lastly, we gathered sources from the Publications Office of the European Union ( https://op.europa.eu/en/home ); namely, documents that refer to policies related to digital transformation in education.

Regarding search terms, we first searched resources on the impact of digital technologies on education by performing the following search queries: “impact” OR “effects” AND “digital technologies” AND “education”, “impact” OR “effects” AND “ICT” AND “education”. We further refined our results by adding the terms “meta-analysis” and “review” or by adjusting the search options based on the features of each database to avoid collecting individual studies that would provide limited contributions to a particular domain. We relied on meta-analyses and review studies as these consider the findings of multiple studies to offer a more comprehensive view of the research in a given area (Schuele & Justice, 2006 ). Specifically, meta-analysis studies provided quantitative evidence based on statistically verifiable results regarding the impact of educational interventions that integrate digital technologies in school classrooms (Higgins et al., 2012 ; Tolani-Brown et al., 2011 ).

However, quantitative data does not offer explanations for the challenges or difficulties experienced during ICT integration in learning and teaching (Tolani-Brown et al., 2011 ). To fill this gap, we analyzed literature reviews and gathered in-depth qualitative evidence of the benefits and implications of technology integration in schools. In the analysis presented herein, we also included policy documents and reports from professional and international bodies and governmental reports, which offered useful explanations of the key concepts of this study and provided recent evidence on digital capacity and transformation in education along with policy recommendations. The inclusion and exclusion criteria that were considered in this study are presented in Table ​ Table1 1 .

Inclusion and exclusion criteria for the selection of resources on the impact of digital technologies on education

To ensure a reliable extraction of information from each study and assist the research synthesis we selected the study characteristics of interest (impact) and constructed coding forms. First, an overview of the synthesis was provided by the principal investigator who described the processes of coding, data entry, and data management. The coders followed the same set of instructions but worked independently. To ensure a common understanding of the process between coders, a sample of ten studies was tested. The results were compared, and the discrepancies were identified and resolved. Additionally, to ensure an efficient coding process, all coders participated in group meetings to discuss additions, deletions, and modifications (Stock, 1994 ). Due to the methodological diversity of the studied documents we began to synthesize the literature review findings based on similar study designs. Specifically, most of the meta-analysis studies were grouped in one category due to the quantitative nature of the measured impact. These studies tended to refer to student achievement (Hattie et al., 2014 ). Then, we organized the themes of the qualitative studies in several impact categories. Lastly, we synthesized both review and meta-analysis data across the categories. In order to establish a collective understanding of the concept of impact, we referred to a previous impact study by Balanskat ( 2009 ) which investigated the impact of technology in primary schools. In this context, the impact had a more specific ICT-related meaning and was described as “ a significant influence or effect of ICT on the measured or perceived quality of (parts of) education ” (Balanskat, 2009 , p. 9). In the study presented herein, the main impacts are in relation to learning and learners, teaching, and teachers, as well as other key stakeholders who are directly or indirectly connected to the school unit.

The study’s results identified multiple dimensions of the impact of digital technologies on students’ knowledge, skills, and attitudes; on equality, inclusion, and social integration; on teachers’ professional and teaching practices; and on other school-related aspects and stakeholders. The data analysis indicated various factors that might affect the schools’ digital capacity and transformation, such as digital competencies, the teachers’ personal characteristics and professional development, as well as the school’s leadership and management, administration, infrastructure, etc. The impacts and factors found in the literature review are presented below.

Impacts of digital technologies on students’ knowledge, skills, attitudes, and emotions

The impact of ICT use on students’ knowledge, skills, and attitudes has been investigated early in the literature. Eng ( 2005 ) found a small positive effect between ICT use and students' learning. Specifically, the author reported that access to computer-assisted instruction (CAI) programs in simulation or tutorial modes—used to supplement rather than substitute instruction – could enhance student learning. The author reported studies showing that teachers acknowledged the benefits of ICT on pupils with special educational needs; however, the impact of ICT on students' attainment was unclear. Balanskat et al. ( 2006 ) found a statistically significant positive association between ICT use and higher student achievement in primary and secondary education. The authors also reported improvements in the performance of low-achieving pupils. The use of ICT resulted in further positive gains for students, namely increased attention, engagement, motivation, communication and process skills, teamwork, and gains related to their behaviour towards learning. Evidence from qualitative studies showed that teachers, students, and parents recognized the positive impact of ICT on students' learning regardless of their competence level (strong/weak students). Punie et al. ( 2006 ) documented studies that showed positive results of ICT-based learning for supporting low-achieving pupils and young people with complex lives outside the education system. Liao et al. ( 2007 ) reported moderate positive effects of computer application instruction (CAI, computer simulations, and web-based learning) over traditional instruction on primary school student's achievement. Similarly, Tamim et al. ( 2011 ) reported small to moderate positive effects between the use of computer technology (CAI, ICT, simulations, computer-based instruction, digital and hypermedia) and student achievement in formal face-to-face classrooms compared to classrooms that did not use technology. Jewitt et al., ( 2011 ) found that the use of learning platforms (LPs) (virtual learning environments, management information systems, communication technologies, and information- and resource-sharing technologies) in schools allowed primary and secondary students to access a wider variety of quality learning resources, engage in independent and personalized learning, and conduct self- and peer-review; LPs also provide opportunities for teacher assessment and feedback. Similar findings were reported by Fu ( 2013 ), who documented a list of benefits and opportunities of ICT use. According to the author, the use of ICTs helps students access digital information and course content effectively and efficiently, supports student-centered and self-directed learning, as well as the development of a creative learning environment where more opportunities for critical thinking skills are offered, and promotes collaborative learning in a distance-learning environment. Higgins et al. ( 2012 ) found consistent but small positive associations between the use of technology and learning outcomes of school-age learners (5–18-year-olds) in studies linking the provision and use of technology with attainment. Additionally, Chauhan ( 2017 ) reported a medium positive effect of technology on the learning effectiveness of primary school students compared to students who followed traditional learning instruction.

The rise of mobile technologies and hardware devices instigated investigations into their impact on teaching and learning. Sung et al. ( 2016 ) reported a moderate effect on students' performance from the use of mobile devices in the classroom compared to the use of desktop computers or the non-use of mobile devices. Schmid et al. ( 2014 ) reported medium–low to low positive effects of technology integration (e.g., CAI, ICTs) in the classroom on students' achievement and attitude compared to not using technology or using technology to varying degrees. Tamim et al. ( 2015 ) found a low statistically significant effect of the use of tablets and other smart devices in educational contexts on students' achievement outcomes. The authors suggested that tablets offered additional advantages to students; namely, they reported improvements in students’ notetaking, organizational and communication skills, and creativity. Zheng et al. ( 2016 ) reported a small positive effect of one-to-one laptop programs on students’ academic achievement across subject areas. Additional reported benefits included student-centered, individualized, and project-based learning enhanced learner engagement and enthusiasm. Additionally, the authors found that students using one-to-one laptop programs tended to use technology more frequently than in non-laptop classrooms, and as a result, they developed a range of skills (e.g., information skills, media skills, technology skills, organizational skills). Haßler et al. ( 2016 ) found that most interventions that included the use of tablets across the curriculum reported positive learning outcomes. However, from 23 studies, five reported no differences, and two reported a negative effect on students' learning outcomes. Similar results were indicated by Kalati and Kim ( 2022 ) who investigated the effect of touchscreen technologies on young students’ learning. Specifically, from 53 studies, 34 advocated positive effects of touchscreen devices on children’s learning, 17 obtained mixed findings and two studies reported negative effects.

More recently, approaches that refer to the impact of gamification with the use of digital technologies on teaching and learning were also explored. A review by Pan et al. ( 2022 ) that examined the role of learning games in fostering mathematics education in K-12 settings, reported that gameplay improved students’ performance. Integration of digital games in teaching was also found as a promising pedagogical practice in STEM education that could lead to increased learning gains (Martinez et al., 2022 ; Wang et al., 2022 ). However, although Talan et al. ( 2020 ) reported a medium effect of the use of educational games (both digital and non-digital) on academic achievement, the effect of non-digital games was higher.

Over the last two years, the effects of more advanced technologies on teaching and learning were also investigated. Garzón and Acevedo ( 2019 ) found that AR applications had a medium effect on students' learning outcomes compared to traditional lectures. Similarly, Garzón et al. ( 2020 ) showed that AR had a medium impact on students' learning gains. VR applications integrated into various subjects were also found to have a moderate effect on students’ learning compared to control conditions (traditional classes, e.g., lectures, textbooks, and multimedia use, e.g., images, videos, animation, CAI) (Chen et al., 2022b ). Villena-Taranilla et al. ( 2022 ) noted the moderate effect of VR technologies on students’ learning when these were applied in STEM disciplines. In the same meta-analysis, Villena-Taranilla et al. ( 2022 ) highlighted the role of immersive VR, since its effect on students’ learning was greater (at a high level) across educational levels (K-6) compared to semi-immersive and non-immersive integrations. In another meta-analysis study, the effect size of the immersive VR was small and significantly differentiated across educational levels (Coban et al., 2022 ). The impact of AI on education was investigated by Su and Yang ( 2022 ) and Su et al. ( 2022 ), who showed that this technology significantly improved students’ understanding of AI computer science and machine learning concepts.

It is worth noting that the vast majority of studies referred to learning gains in specific subjects. Specifically, several studies examined the impact of digital technologies on students’ literacy skills and reported positive effects on language learning (Balanskat et al., 2006 ; Grgurović et al., 2013 ; Friedel et al., 2013 ; Zheng et al., 2016 ; Chen et al., 2022b ; Savva et al., 2022 ). Also, several studies documented positive effects on specific language learning areas, namely foreign language learning (Kao, 2014 ), writing (Higgins et al., 2012 ; Wen & Walters, 2022 ; Zheng et al., 2016 ), as well as reading and comprehension (Cheung & Slavin, 2011 ; Liao et al., 2007 ; Schwabe et al., 2022 ). ICTs were also found to have a positive impact on students' performance in STEM (science, technology, engineering, and mathematics) disciplines (Arztmann et al., 2022 ; Bado, 2022 ; Villena-Taranilla et al., 2022 ; Wang et al., 2022 ). Specifically, a number of studies reported positive impacts on students’ achievement in mathematics (Balanskat et al., 2006 ; Hillmayr et al., 2020 ; Li & Ma, 2010 ; Pan et al., 2022 ; Ran et al., 2022 ; Verschaffel et al., 2019 ; Zheng et al., 2016 ). Furthermore, studies documented positive effects of ICTs on science learning (Balanskat et al., 2006 ; Liao et al., 2007 ; Zheng et al., 2016 ; Hillmayr et al., 2020 ; Kalemkuş & Kalemkuş, 2022 ; Lei et al., 2022a ). Çelik ( 2022 ) also noted that computer simulations can help students understand learning concepts related to science. Furthermore, some studies documented that the use of ICTs had a positive impact on students’ achievement in other subjects, such as geography, history, music, and arts (Chauhan, 2017 ; Condie & Munro, 2007 ), and design and technology (Balanskat et al., 2006 ).

More specific positive learning gains were reported in a number of skills, e.g., problem-solving skills and pattern exploration skills (Higgins et al., 2012 ), metacognitive learning outcomes (Verschaffel et al., 2019 ), literacy skills, computational thinking skills, emotion control skills, and collaborative inquiry skills (Lu et al., 2022 ; Su & Yang, 2022 ; Su et al., 2022 ). Additionally, several investigations have reported benefits from the use of ICT on students’ creativity (Fielding & Murcia, 2022 ; Liu et al., 2022 ; Quah & Ng, 2022 ). Lastly, digital technologies were also found to be beneficial for enhancing students’ lifelong learning skills (Haleem et al., 2022 ).

Apart from gaining knowledge and skills, studies also reported improvement in motivation and interest in mathematics (Higgins et. al., 2019 ; Fadda et al., 2022 ) and increased positive achievement emotions towards several subjects during interventions using educational games (Lei et al., 2022a ). Chen et al. ( 2022a ) also reported a small but positive effect of digital health approaches in bullying and cyberbullying interventions with K-12 students, demonstrating that technology-based approaches can help reduce bullying and related consequences by providing emotional support, empowerment, and change of attitude. In their meta-review study, Su et al. ( 2022 ) also documented that AI technologies effectively strengthened students’ attitudes towards learning. In another meta-analysis, Arztmann et al. ( 2022 ) reported positive effects of digital games on motivation and behaviour towards STEM subjects.

Impacts of digital technologies on equality, inclusion and social integration

Although most of the reviewed studies focused on the impact of ICTs on students’ knowledge, skills, and attitudes, reports were also made on other aspects in the school context, such as equality, inclusion, and social integration. Condie and Munro ( 2007 ) documented research interventions investigating how ICT can support pupils with additional or special educational needs. While those interventions were relatively small scale and mostly based on qualitative data, their findings indicated that the use of ICTs enabled the development of communication, participation, and self-esteem. A recent meta-analysis (Baragash et al., 2022 ) with 119 participants with different disabilities, reported a significant overall effect size of AR on their functional skills acquisition. Koh’s meta-analysis ( 2022 ) also revealed that students with intellectual and developmental disabilities improved their competence and performance when they used digital games in the lessons.

Istenic Starcic and Bagon ( 2014 ) found that the role of ICT in inclusion and the design of pedagogical and technological interventions was not sufficiently explored in educational interventions with people with special needs; however, some benefits of ICT use were found in students’ social integration. The issue of gender and technology use was mentioned in a small number of studies. Zheng et al. ( 2016 ) reported a statistically significant positive interaction between one-to-one laptop programs and gender. Specifically, the results showed that girls and boys alike benefitted from the laptop program, but the effect on girls’ achievement was smaller than that on boys’. Along the same lines, Arztmann et al. ( 2022 ) reported no difference in the impact of game-based learning between boys and girls, arguing that boys and girls equally benefited from game-based interventions in STEM domains. However, results from a systematic review by Cussó-Calabuig et al. ( 2018 ) found limited and low-quality evidence on the effects of intensive use of computers on gender differences in computer anxiety, self-efficacy, and self-confidence. Based on their view, intensive use of computers can reduce gender differences in some areas and not in others, depending on contextual and implementation factors.

Impacts of digital technologies on teachers’ professional and teaching practices

Various research studies have explored the impact of ICT on teachers’ instructional practices and student assessment. Friedel et al. ( 2013 ) found that the use of mobile devices by students enabled teachers to successfully deliver content (e.g., mobile serious games), provide scaffolding, and facilitate synchronous collaborative learning. The integration of digital games in teaching and learning activities also gave teachers the opportunity to study and apply various pedagogical practices (Bado, 2022 ). Specifically, Bado ( 2022 ) found that teachers who implemented instructional activities in three stages (pre-game, game, and post-game) maximized students’ learning outcomes and engagement. For instance, during the pre-game stage, teachers focused on lectures and gameplay training, at the game stage teachers provided scaffolding on content, addressed technical issues, and managed the classroom activities. During the post-game stage, teachers organized activities for debriefing to ensure that the gameplay had indeed enhanced students’ learning outcomes.

Furthermore, ICT can increase efficiency in lesson planning and preparation by offering possibilities for a more collaborative approach among teachers. The sharing of curriculum plans and the analysis of students’ data led to clearer target settings and improvements in reporting to parents (Balanskat et al., 2006 ).

Additionally, the use and application of digital technologies in teaching and learning were found to enhance teachers’ digital competence. Balanskat et al. ( 2006 ) documented studies that revealed that the use of digital technologies in education had a positive effect on teachers’ basic ICT skills. The greatest impact was found on teachers with enough experience in integrating ICTs in their teaching and/or who had recently participated in development courses for the pedagogical use of technologies in teaching. Punie et al. ( 2006 ) reported that the provision of fully equipped multimedia portable computers and the development of online teacher communities had positive impacts on teachers’ confidence and competence in the use of ICTs.

Moreover, online assessment via ICTs benefits instruction. In particular, online assessments support the digitalization of students’ work and related logistics, allow teachers to gather immediate feedback and readjust to new objectives, and support the improvement of the technical quality of tests by providing more accurate results. Additionally, the capabilities of ICTs (e.g., interactive media, simulations) create new potential methods of testing specific skills, such as problem-solving and problem-processing skills, meta-cognitive skills, creativity and communication skills, and the ability to work productively in groups (Punie et al., 2006 ).

Impacts of digital technologies on other school-related aspects and stakeholders

There is evidence that the effective use of ICTs and the data transmission offered by broadband connections help improve administration (Balanskat et al., 2006 ). Specifically, ICTs have been found to provide better management systems to schools that have data gathering procedures in place. Condie and Munro ( 2007 ) reported impacts from the use of ICTs in schools in the following areas: attendance monitoring, assessment records, reporting to parents, financial management, creation of repositories for learning resources, and sharing of information amongst staff. Such data can be used strategically for self-evaluation and monitoring purposes which in turn can result in school improvements. Additionally, they reported that online access to other people with similar roles helped to reduce headteachers’ isolation by offering them opportunities to share insights into the use of ICT in learning and teaching and how it could be used to support school improvement. Furthermore, ICTs provided more efficient and successful examination management procedures, namely less time-consuming reporting processes compared to paper-based examinations and smooth communications between schools and examination authorities through electronic data exchange (Punie et al., 2006 ).

Zheng et al. ( 2016 ) reported that the use of ICTs improved home-school relationships. Additionally, Escueta et al. ( 2017 ) reported several ICT programs that had improved the flow of information from the school to parents. Particularly, they documented that the use of ICTs (learning management systems, emails, dedicated websites, mobile phones) allowed for personalized and customized information exchange between schools and parents, such as attendance records, upcoming class assignments, school events, and students’ grades, which generated positive results on students’ learning outcomes and attainment. Such information exchange between schools and families prompted parents to encourage their children to put more effort into their schoolwork.

The above findings suggest that the impact of ICT integration in schools goes beyond students’ performance in school subjects. Specifically, it affects a number of school-related aspects, such as equality and social integration, professional and teaching practices, and diverse stakeholders. In Table ​ Table2, 2 , we summarize the different impacts of digital technologies on school stakeholders based on the literature review, while in Table ​ Table3 3 we organized the tools/platforms and practices/policies addressed in the meta-analyses, literature reviews, EU reports, and international bodies included in the manuscript.

The impact of digital technologies on schools’ stakeholders based on the literature review

Tools/platforms and practices/policies addressed in the meta-analyses, literature reviews, EU reports, and international bodies included in the manuscript

Additionally, based on the results of the literature review, there are many types of digital technologies with different affordances (see, for example, studies on VR vs Immersive VR), which evolve over time (e.g. starting from CAIs in 2005 to Augmented and Virtual reality 2020). Furthermore, these technologies are linked to different pedagogies and policy initiatives, which are critical factors in the study of impact. Table ​ Table3 3 summarizes the different tools and practices that have been used to examine the impact of digital technologies on education since 2005 based on the review results.

Factors that affect the integration of digital technologies

Although the analysis of the literature review demonstrated different impacts of the use of digital technology on education, several authors highlighted the importance of various factors, besides the technology itself, that affect this impact. For example, Liao et al. ( 2007 ) suggested that future studies should carefully investigate which factors contribute to positive outcomes by clarifying the exact relationship between computer applications and learning. Additionally, Haßler et al., ( 2016 ) suggested that the neutral findings regarding the impact of tablets on students learning outcomes in some of the studies included in their review should encourage educators, school leaders, and school officials to further investigate the potential of such devices in teaching and learning. Several other researchers suggested that a number of variables play a significant role in the impact of ICTs on students’ learning that could be attributed to the school context, teaching practices and professional development, the curriculum, and learners’ characteristics (Underwood, 2009 ; Tamim et al., 2011 ; Higgins et al., 2012 ; Archer et al., 2014 ; Sung et al., 2016 ; Haßler et al., 2016 ; Chauhan, 2017 ; Lee et al., 2020 ; Tang et al., 2022 ).

Digital competencies

One of the most common challenges reported in studies that utilized digital tools in the classroom was the lack of students’ skills on how to use them. Fu ( 2013 ) found that students’ lack of technical skills is a barrier to the effective use of ICT in the classroom. Tamim et al. ( 2015 ) reported that students faced challenges when using tablets and smart mobile devices, associated with the technical issues or expertise needed for their use and the distracting nature of the devices and highlighted the need for teachers’ professional development. Higgins et al. ( 2012 ) reported that skills training about the use of digital technologies is essential for learners to fully exploit the benefits of instruction.

Delgado et al. ( 2015 ), meanwhile, reported studies that showed a strong positive association between teachers’ computer skills and students’ use of computers. Teachers’ lack of ICT skills and familiarization with technologies can become a constraint to the effective use of technology in the classroom (Balanskat et al., 2006 ; Delgado et al., 2015 ).

It is worth noting that the way teachers are introduced to ICTs affects the impact of digital technologies on education. Previous studies have shown that teachers may avoid using digital technologies due to limited digital skills (Balanskat, 2006 ), or they prefer applying “safe” technologies, namely technologies that their own teachers used and with which they are familiar (Condie & Munro, 2007 ). In this regard, the provision of digital skills training and exposure to new digital tools might encourage teachers to apply various technologies in their lessons (Condie & Munro, 2007 ). Apart from digital competence, technical support in the school setting has also been shown to affect teachers’ use of technology in their classrooms (Delgado et al., 2015 ). Ferrari et al. ( 2011 ) found that while teachers’ use of ICT is high, 75% stated that they needed more institutional support and a shift in the mindset of educational actors to achieve more innovative teaching practices. The provision of support can reduce time and effort as well as cognitive constraints, which could cause limited ICT integration in the school lessons by teachers (Escueta et al., 2017 ).

Teachers’ personal characteristics, training approaches, and professional development

Teachers’ personal characteristics and professional development affect the impact of digital technologies on education. Specifically, Cheok and Wong ( 2015 ) found that teachers’ personal characteristics (e.g., anxiety, self-efficacy) are associated with their satisfaction and engagement with technology. Bingimlas ( 2009 ) reported that lack of confidence, resistance to change, and negative attitudes in using new technologies in teaching are significant determinants of teachers’ levels of engagement in ICT. The same author reported that the provision of technical support, motivation support (e.g., awards, sufficient time for planning), and training on how technologies can benefit teaching and learning can eliminate the above barriers to ICT integration. Archer et al. ( 2014 ) found that comfort levels in using technology are an important predictor of technology integration and argued that it is essential to provide teachers with appropriate training and ongoing support until they are comfortable with using ICTs in the classroom. Hillmayr et al. ( 2020 ) documented that training teachers on ICT had an important effecton students’ learning.

According to Balanskat et al. ( 2006 ), the impact of ICTs on students’ learning is highly dependent on the teachers’ capacity to efficiently exploit their application for pedagogical purposes. Results obtained from the Teaching and Learning International Survey (TALIS) (OECD, 2021 ) revealed that although schools are open to innovative practices and have the capacity to adopt them, only 39% of teachers in the European Union reported that they are well or very well prepared to use digital technologies for teaching. Li and Ma ( 2010 ) and Hardman ( 2019 ) showed that the positive effect of technology on students’ achievement depends on the pedagogical practices used by teachers. Schmid et al. ( 2014 ) reported that learning was best supported when students were engaged in active, meaningful activities with the use of technological tools that provided cognitive support. Tamim et al. ( 2015 ) compared two different pedagogical uses of tablets and found a significant moderate effect when the devices were used in a student-centered context and approach rather than within teacher-led environments. Similarly, Garzón and Acevedo ( 2019 ) and Garzón et al. ( 2020 ) reported that the positive results from the integration of AR applications could be attributed to the existence of different variables which could influence AR interventions (e.g., pedagogical approach, learning environment, and duration of the intervention). Additionally, Garzón et al. ( 2020 ) suggested that the pedagogical resources that teachers used to complement their lectures and the pedagogical approaches they applied were crucial to the effective integration of AR on students’ learning gains. Garzón and Acevedo ( 2019 ) also emphasized that the success of a technology-enhanced intervention is based on both the technology per se and its characteristics and on the pedagogical strategies teachers choose to implement. For instance, their results indicated that the collaborative learning approach had the highest impact on students’ learning gains among other approaches (e.g., inquiry-based learning, situated learning, or project-based learning). Ran et al. ( 2022 ) also found that the use of technology to design collaborative and communicative environments showed the largest moderator effects among the other approaches.

Hattie ( 2008 ) reported that the effective use of computers is associated with training teachers in using computers as a teaching and learning tool. Zheng et al. ( 2016 ) noted that in addition to the strategies teachers adopt in teaching, ongoing professional development is also vital in ensuring the success of technology implementation programs. Sung et al. ( 2016 ) found that research on the use of mobile devices to support learning tends to report that the insufficient preparation of teachers is a major obstacle in implementing effective mobile learning programs in schools. Friedel et al. ( 2013 ) found that providing training and support to teachers increased the positive impact of the interventions on students’ learning gains. Trucano ( 2005 ) argued that positive impacts occur when digital technologies are used to enhance teachers’ existing pedagogical philosophies. Higgins et al. ( 2012 ) found that the types of technologies used and how they are used could also affect students’ learning. The authors suggested that training and professional development of teachers that focuses on the effective pedagogical use of technology to support teaching and learning is an important component of successful instructional approaches (Higgins et al., 2012 ). Archer et al. ( 2014 ) found that studies that reported ICT interventions during which teachers received training and support had moderate positive effects on students’ learning outcomes, which were significantly higher than studies where little or no detail about training and support was mentioned. Fu ( 2013 ) reported that the lack of teachers’ knowledge and skills on the technical and instructional aspects of ICT use in the classroom, in-service training, pedagogy support, technical and financial support, as well as the lack of teachers’ motivation and encouragement to integrate ICT on their teaching were significant barriers to the integration of ICT in education.

School leadership and management

Management and leadership are important cornerstones in the digital transformation process (Pihir et al., 2018 ). Zheng et al. ( 2016 ) documented leadership among the factors positively affecting the successful implementation of technology integration in schools. Strong leadership, strategic planning, and systematic integration of digital technologies are prerequisites for the digital transformation of education systems (Ređep, 2021 ). Management and leadership play a significant role in formulating policies that are translated into practice and ensure that developments in ICT become embedded into the life of the school and in the experiences of staff and pupils (Condie & Munro, 2007 ). Policy support and leadership must include the provision of an overall vision for the use of digital technologies in education, guidance for students and parents, logistical support, as well as teacher training (Conrads et al., 2017 ). Unless there is a commitment throughout the school, with accountability for progress at key points, it is unlikely for ICT integration to be sustained or become part of the culture (Condie & Munro, 2007 ). To achieve this, principals need to adopt and promote a whole-institution strategy and build a strong mutual support system that enables the school’s technological maturity (European Commission, 2019 ). In this context, school culture plays an essential role in shaping the mindsets and beliefs of school actors towards successful technology integration. Condie and Munro ( 2007 ) emphasized the importance of the principal’s enthusiasm and work as a source of inspiration for the school staff and the students to cultivate a culture of innovation and establish sustainable digital change. Specifically, school leaders need to create conditions in which the school staff is empowered to experiment and take risks with technology (Elkordy & Lovinelli, 2020 ).

In order for leaders to achieve the above, it is important to develop capacities for learning and leading, advocating professional learning, and creating support systems and structures (European Commission, 2019 ). Digital technology integration in education systems can be challenging and leadership needs guidance to achieve it. Such guidance can be introduced through the adoption of new methods and techniques in strategic planning for the integration of digital technologies (Ređep, 2021 ). Even though the role of leaders is vital, the relevant training offered to them has so far been inadequate. Specifically, only a third of the education systems in Europe have put in place national strategies that explicitly refer to the training of school principals (European Commission, 2019 , p. 16).

Connectivity, infrastructure, and government and other support

The effective integration of digital technologies across levels of education presupposes the development of infrastructure, the provision of digital content, and the selection of proper resources (Voogt et al., 2013 ). Particularly, a high-quality broadband connection in the school increases the quality and quantity of educational activities. There is evidence that ICT increases and formalizes cooperative planning between teachers and cooperation with managers, which in turn has a positive impact on teaching practices (Balanskat et al., 2006 ). Additionally, ICT resources, including software and hardware, increase the likelihood of teachers integrating technology into the curriculum to enhance their teaching practices (Delgado et al., 2015 ). For example, Zheng et al. ( 2016 ) found that the use of one-on-one laptop programs resulted in positive changes in teaching and learning, which would not have been accomplished without the infrastructure and technical support provided to teachers. Delgado et al. ( 2015 ) reported that limited access to technology (insufficient computers, peripherals, and software) and lack of technical support are important barriers to ICT integration. Access to infrastructure refers not only to the availability of technology in a school but also to the provision of a proper amount and the right types of technology in locations where teachers and students can use them. Effective technical support is a central element of the whole-school strategy for ICT (Underwood, 2009 ). Bingimlas ( 2009 ) reported that lack of technical support in the classroom and whole-school resources (e.g., failing to connect to the Internet, printers not printing, malfunctioning computers, and working on old computers) are significant barriers that discourage the use of ICT by teachers. Moreover, poor quality and inadequate hardware maintenance, and unsuitable educational software may discourage teachers from using ICTs (Balanskat et al., 2006 ; Bingimlas, 2009 ).

Government support can also impact the integration of ICTs in teaching. Specifically, Balanskat et al. ( 2006 ) reported that government interventions and training programs increased teachers’ enthusiasm and positive attitudes towards ICT and led to the routine use of embedded ICT.

Lastly, another important factor affecting digital transformation is the development and quality assurance of digital learning resources. Such resources can be support textbooks and related materials or resources that focus on specific subjects or parts of the curriculum. Policies on the provision of digital learning resources are essential for schools and can be achieved through various actions. For example, some countries are financing web portals that become repositories, enabling teachers to share resources or create their own. Additionally, they may offer e-learning opportunities or other services linked to digital education. In other cases, specific agencies of projects have also been set up to develop digital resources (Eurydice, 2019 ).

Administration and digital data management

The digital transformation of schools involves organizational improvements at the level of internal workflows, communication between the different stakeholders, and potential for collaboration. Vuorikari et al. ( 2020 ) presented evidence that digital technologies supported the automation of administrative practices in schools and reduced the administration’s workload. There is evidence that digital data affects the production of knowledge about schools and has the power to transform how schooling takes place. Specifically, Sellar ( 2015 ) reported that data infrastructure in education is developing due to the demand for “ information about student outcomes, teacher quality, school performance, and adult skills, associated with policy efforts to increase human capital and productivity practices ” (p. 771). In this regard, practices, such as datafication which refers to the “ translation of information about all kinds of things and processes into quantified formats” have become essential for decision-making based on accountability reports about the school’s quality. The data could be turned into deep insights about education or training incorporating ICTs. For example, measuring students’ online engagement with the learning material and drawing meaningful conclusions can allow teachers to improve their educational interventions (Vuorikari et al., 2020 ).

Students’ socioeconomic background and family support

Research show that the active engagement of parents in the school and their support for the school’s work can make a difference to their children’s attitudes towards learning and, as a result, their achievement (Hattie, 2008 ). In recent years, digital technologies have been used for more effective communication between school and family (Escueta et al., 2017 ). The European Commission ( 2020 ) presented data from a Eurostat survey regarding the use of computers by students during the pandemic. The data showed that younger pupils needed additional support and guidance from parents and the challenges were greater for families in which parents had lower levels of education and little to no digital skills.

In this regard, the socio-economic background of the learners and their socio-cultural environment also affect educational achievements (Punie et al., 2006 ). Trucano documented that the use of computers at home positively influenced students’ confidence and resulted in more frequent use at school, compared to students who had no home access (Trucano, 2005 ). In this sense, the socio-economic background affects the access to computers at home (OECD, 2015 ) which in turn influences the experience of ICT, an important factor for school achievement (Punie et al., 2006 ; Underwood, 2009 ). Furthermore, parents from different socio-economic backgrounds may have different abilities and availability to support their children in their learning process (Di Pietro et al., 2020 ).

Schools’ socioeconomic context and emergency situations

The socio-economic context of the school is closely related to a school’s digital transformation. For example, schools in disadvantaged, rural, or deprived areas are likely to lack the digital capacity and infrastructure required to adapt to the use of digital technologies during emergency periods, such as the COVID-19 pandemic (Di Pietro et al., 2020 ). Data collected from school principals confirmed that in several countries, there is a rural/urban divide in connectivity (OECD, 2015 ).

Emergency periods also affect the digitalization of schools. The COVID-19 pandemic led to the closure of schools and forced them to seek appropriate and connective ways to keep working on the curriculum (Di Pietro et al., 2020 ). The sudden large-scale shift to distance and online teaching and learning also presented challenges around quality and equity in education, such as the risk of increased inequalities in learning, digital, and social, as well as teachers facing difficulties coping with this demanding situation (European Commission, 2020 ).

Looking at the findings of the above studies, we can conclude that the impact of digital technologies on education is influenced by various actors and touches many aspects of the school ecosystem. Figure  1 summarizes the factors affecting the digital technologies’ impact on school stakeholders based on the findings from the literature review.

Factors that affect the impact of ICTs on education

The findings revealed that the use of digital technologies in education affects a variety of actors within a school’s ecosystem. First, we observed that as technologies evolve, so does the interest of the research community to apply them to school settings. Figure  2 summarizes the trends identified in current research around the impact of digital technologies on schools’ digital capacity and transformation as found in the present study. Starting as early as 2005, when computers, simulations, and interactive boards were the most commonly applied tools in school interventions (e.g., Eng, 2005 ; Liao et al., 2007 ; Moran et al., 2008 ; Tamim et al., 2011 ), moving towards the use of learning platforms (Jewitt et al., 2011 ), then to the use of mobile devices and digital games (e.g., Tamim et al., 2015 ; Sung et al., 2016 ; Talan et al., 2020 ), as well as e-books (e.g., Savva et al., 2022 ), to the more recent advanced technologies, such as AR and VR applications (e.g., Garzón & Acevedo, 2019 ; Garzón et al., 2020 ; Kalemkuş & Kalemkuş, 2022 ), or robotics and AI (e.g., Su & Yang, 2022 ; Su et al., 2022 ). As this evolution shows, digital technologies are a concept in flux with different affordances and characteristics. Additionally, from an instructional perspective, there has been a growing interest in different modes and models of content delivery such as online, blended, and hybrid modes (e.g., Cheok & Wong, 2015 ; Kazu & Yalçin, 2022 ; Ulum, 2022 ). This is an indication that the value of technologies to support teaching and learning as well as other school-related practices is increasingly recognized by the research and school community. The impact results from the literature review indicate that ICT integration on students’ learning outcomes has effects that are small (Coban et al., 2022 ; Eng, 2005 ; Higgins et al., 2012 ; Schmid et al., 2014 ; Tamim et al., 2015 ; Zheng et al., 2016 ) to moderate (Garzón & Acevedo, 2019 ; Garzón et al., 2020 ; Liao et al., 2007 ; Sung et al., 2016 ; Talan et al., 2020 ; Wen & Walters, 2022 ). That said, a number of recent studies have reported high effect sizes (e.g., Kazu & Yalçin, 2022 ).

Current work and trends in the study of the impact of digital technologies on schools’ digital capacity

Based on these findings, several authors have suggested that the impact of technology on education depends on several variables and not on the technology per se (Tamim et al., 2011 ; Higgins et al., 2012 ; Archer et al., 2014 ; Sung et al., 2016 ; Haßler et al., 2016 ; Chauhan, 2017 ; Lee et al., 2020 ; Lei et al., 2022a ). While the impact of ICTs on student achievement has been thoroughly investigated by researchers, other aspects related to school life that are also affected by ICTs, such as equality, inclusion, and social integration have received less attention. Further analysis of the literature review has revealed a greater investment in ICT interventions to support learning and teaching in the core subjects of literacy and STEM disciplines, especially mathematics, and science. These were the most common subjects studied in the reviewed papers often drawing on national testing results, while studies that investigated other subject areas, such as social studies, were limited (Chauhan, 2017 ; Condie & Munro, 2007 ). As such, research is still lacking impact studies that focus on the effects of ICTs on a range of curriculum subjects.

The qualitative research provided additional information about the impact of digital technologies on education, documenting positive effects and giving more details about implications, recommendations, and future research directions. Specifically, the findings regarding the role of ICTs in supporting learning highlight the importance of teachers’ instructional practice and the learning context in the use of technologies and consequently their impact on instruction (Çelik, 2022 ; Schmid et al., 2014 ; Tamim et al., 2015 ). The review also provided useful insights regarding the various factors that affect the impact of digital technologies on education. These factors are interconnected and play a vital role in the transformation process. Specifically, these factors include a) digital competencies; b) teachers’ personal characteristics and professional development; c) school leadership and management; d) connectivity, infrastructure, and government support; e) administration and data management practices; f) students’ socio-economic background and family support and g) the socioeconomic context of the school and emergency situations. It is worth noting that we observed factors that affect the integration of ICTs in education but may also be affected by it. For example, the frequent use of ICTs and the use of laptops by students for instructional purposes positively affect the development of digital competencies (Zheng et al., 2016 ) and at the same time, the digital competencies affect the use of ICTs (Fu, 2013 ; Higgins et al., 2012 ). As a result, the impact of digital technologies should be explored more as an enabler of desirable and new practices and not merely as a catalyst that improves the output of the education process i.e. namely student attainment.

Conclusions

Digital technologies offer immense potential for fundamental improvement in schools. However, investment in ICT infrastructure and professional development to improve school education are yet to provide fruitful results. Digital transformation is a complex process that requires large-scale transformative changes that presuppose digital capacity and preparedness. To achieve such changes, all actors within the school’s ecosystem need to share a common vision regarding the integration of ICTs in education and work towards achieving this goal. Our literature review, which synthesized quantitative and qualitative data from a list of meta-analyses and review studies, provided useful insights into the impact of ICTs on different school stakeholders and showed that the impact of digital technologies touches upon many different aspects of school life, which are often overlooked when the focus is on student achievement as the final output of education. Furthermore, the concept of digital technologies is a concept in flux as technologies are not only different among them calling for different uses in the educational practice but they also change through time. Additionally, we opened a forum for discussion regarding the factors that affect a school’s digital capacity and transformation. We hope that our study will inform policy, practice, and research and result in a paradigm shift towards more holistic approaches in impact and assessment studies.

Study limitations and future directions

We presented a review of the study of digital technologies' impact on education and factors influencing schools’ digital capacity and transformation. The study results were based on a non-systematic literature review grounded on the acquisition of documentation in specific databases. Future studies should investigate more databases to corroborate and enhance our results. Moreover, search queries could be enhanced with key terms that could provide additional insights about the integration of ICTs in education, such as “policies and strategies for ICT integration in education”. Also, the study drew information from meta-analyses and literature reviews to acquire evidence about the effects of ICT integration in schools. Such evidence was mostly based on the general conclusions of the studies. It is worth mentioning that, we located individual studies which showed different, such as negative or neutral results. Thus, further insights are needed about the impact of ICTs on education and the factors influencing the impact. Furthermore, the nature of the studies included in meta-analyses and reviews is different as they are based on different research methodologies and data gathering processes. For instance, in a meta-analysis, the impact among the studies investigated is measured in a particular way, depending on policy or research targets (e.g., results from national examinations, pre-/post-tests). Meanwhile, in literature reviews, qualitative studies offer additional insights and detail based on self-reports and research opinions on several different aspects and stakeholders who could affect and be affected by ICT integration. As a result, it was challenging to draw causal relationships between so many interrelating variables.

Despite the challenges mentioned above, this study envisaged examining school units as ecosystems that consist of several actors by bringing together several variables from different research epistemologies to provide an understanding of the integration of ICTs. However, the use of other tools and methodologies and models for evaluation of the impact of digital technologies on education could give more detailed data and more accurate results. For instance, self-reflection tools, like SELFIE—developed on the DigCompOrg framework- (Kampylis et al., 2015 ; Bocconi & Lightfoot, 2021 ) can help capture a school’s digital capacity and better assess the impact of ICTs on education. Furthermore, the development of a theory of change could be a good approach for documenting the impact of digital technologies on education. Specifically, theories of change are models used for the evaluation of interventions and their impact; they are developed to describe how interventions will work and give the desired outcomes (Mayne, 2015 ). Theory of change as a methodological approach has also been used by researchers to develop models for evaluation in the field of education (e.g., Aromatario et al., 2019 ; Chapman & Sammons, 2013 ; De Silva et al., 2014 ).

We also propose that future studies aim at similar investigations by applying more holistic approaches for impact assessment that can provide in-depth data about the impact of digital technologies on education. For instance, future studies could focus on different research questions about the technologies that are used during the interventions or the way the implementation takes place (e.g., What methodologies are used for documenting impact? How are experimental studies implemented? How can teachers be taken into account and trained on the technology and its functions? What are the elements of an appropriate and successful implementation? How is the whole intervention designed? On which learning theories is the technology implementation based?).

Future research could also focus on assessing the impact of digital technologies on various other subjects since there is a scarcity of research related to particular subjects, such as geography, history, arts, music, and design and technology. More research should also be done about the impact of ICTs on skills, emotions, and attitudes, and on equality, inclusion, social interaction, and special needs education. There is also a need for more research about the impact of ICTs on administration, management, digitalization, and home-school relationships. Additionally, although new forms of teaching and learning with the use of ICTs (e.g., blended, hybrid, and online learning) have initiated several investigations in mainstream classrooms, only a few studies have measured their impact on students’ learning. Additionally, our review did not document any study about the impact of flipped classrooms on K-12 education. Regarding teaching and learning approaches, it is worth noting that studies referred to STEM or STEAM did not investigate the impact of STEM/STEAM as an interdisciplinary approach to learning but only investigated the impact of ICTs on learning in each domain as a separate subject (science, technology, engineering, arts, mathematics). Hence, we propose future research to also investigate the impact of the STEM/STEAM approach on education. The impact of emerging technologies on education, such as AR, VR, robotics, and AI has also been investigated recently, but more work needs to be done.

Finally, we propose that future studies could focus on the way in which specific factors, e.g., infrastructure and government support, school leadership and management, students’ and teachers’ digital competencies, approaches teachers utilize in the teaching and learning (e.g., blended, online and hybrid learning, flipped classrooms, STEM/STEAM approach, project-based learning, inquiry-based learning), affect the impact of digital technologies on education. We hope that future studies will give detailed insights into the concept of schools’ digital transformation through further investigation of impacts and factors which influence digital capacity and transformation based on the results and the recommendations of the present study.

Acknowledgements

This project has received funding under Grant Agreement No Ref Ares (2021) 339036 7483039 as well as funding from the European Union’s Horizon 2020 Research and Innovation Program under Grant Agreement No 739578 and the Government of the Republic of Cyprus through the Deputy Ministry of Research, Innovation and Digital Policy. The UVa co-authors would like also to acknowledge funding from the European Regional Development Fund and the National Research Agency of the Spanish Ministry of Science and Innovation, under project grant PID2020-112584RB-C32.

Data availability statement

Declarations.

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

• Archer K, Savage R, Sanghera-Sidhu S, Wood E, Gottardo A, Chen V. Examining the effectiveness of technology use in classrooms: A tertiary meta-analysis. Computers & Education. 2014; 78 :140–149. doi: 10.1016/j.compedu.2014.06.001. [ CrossRef ] [ Google Scholar ]
• Aromatario O, Van Hoye A, Vuillemin A, Foucaut AM, Pommier J, Cambon L. Using theory of change to develop an intervention theory for designing and evaluating behavior change SDApps for healthy eating and physical exercise: The OCAPREV theory. BMC Public Health. 2019; 19 (1):1–12. doi: 10.1186/s12889-019-7828-4. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Arztmann, M., Hornstra, L., Jeuring, J., & Kester, L. (2022). Effects of games in STEM education: A meta-analysis on the moderating role of student background characteristics. Studies in Science Education , 1-37. 10.1080/03057267.2022.2057732
• Bado N. Game-based learning pedagogy: A review of the literature. Interactive Learning Environments. 2022; 30 (5):936–948. doi: 10.1080/10494820.2019.1683587. [ CrossRef ] [ Google Scholar ]
• Balanskat, A. (2009). Study of the impact of technology in primary schools – Synthesis Report. Empirica and European Schoolnet. Retrieved 30 June 2022 from: https://erte.dge.mec.pt/sites/default/files/Recursos/Estudos/synthesis_report_steps_en.pdf
• Balanskat, A. (2006). The ICT Impact Report: A review of studies of ICT impact on schools in Europe, European Schoolnet. Retrieved 30 June 2022 from:  https://en.unesco.org/icted/content/ict-impact-report-review-studies-ict-impact-schools-europe
• Balanskat, A., Blamire, R., & Kefala, S. (2006). The ICT impact report.  European Schoolnet . Retrieved from: http://colccti.colfinder.org/sites/default/files/ict_impact_report_0.pdf
• Balyer, A., & Öz, Ö. (2018). Academicians’ views on digital transformation in education. International Online Journal of Education and Teaching (IOJET), 5 (4), 809–830. Retrieved 30 June 2022 from  http://iojet.org/index.php/IOJET/article/view/441/295
• Baragash RS, Al-Samarraie H, Moody L, Zaqout F. Augmented reality and functional skills acquisition among individuals with special needs: A meta-analysis of group design studies. Journal of Special Education Technology. 2022; 37 (1):74–81. doi: 10.1177/0162643420910413. [ CrossRef ] [ Google Scholar ]
• Bates, A. W. (2015). Teaching in a digital age: Guidelines for designing teaching and learning . Open Educational Resources Collection . 6. Retrieved 30 June 2022 from: https://irl.umsl.edu/oer/6
• Bingimlas KA. Barriers to the successful integration of ICT in teaching and learning environments: A review of the literature. Eurasia Journal of Mathematics, Science and Technology Education. 2009; 5 (3):235–245. doi: 10.12973/ejmste/75275. [ CrossRef ] [ Google Scholar ]
• Blaskó Z, Costa PD, Schnepf SV. Learning losses and educational inequalities in Europe: Mapping the potential consequences of the COVID-19 crisis. Journal of European Social Policy. 2022; 32 (4):361–375. doi: 10.1177/09589287221091687. [ CrossRef ] [ Google Scholar ]
• Bocconi S, Lightfoot M. Scaling up and integrating the selfie tool for schools' digital capacity in education and training systems: Methodology and lessons learnt. European Training Foundation. 2021 doi: 10.2816/907029,JRC123936. [ CrossRef ] [ Google Scholar ]
• Brooks, D. C., & McCormack, M. (2020). Driving Digital Transformation in Higher Education . Retrieved 30 June 2022 from: https://library.educause.edu/-/media/files/library/2020/6/dx2020.pdf?la=en&hash=28FB8C377B59AFB1855C225BBA8E3CFBB0A271DA
• Cachia, R., Chaudron, S., Di Gioia, R., Velicu, A., & Vuorikari, R. (2021). Emergency remote schooling during COVID-19, a closer look at European families. Retrieved 30 June 2022 from  https://publications.jrc.ec.europa.eu/repository/handle/JRC125787
• Çelik B. The effects of computer simulations on students’ science process skills: Literature review. Canadian Journal of Educational and Social Studies. 2022; 2 (1):16–28. doi: 10.53103/cjess.v2i1.17. [ CrossRef ] [ Google Scholar ]
• Chapman, C., & Sammons, P. (2013). School Self-Evaluation for School Improvement: What Works and Why? . CfBT Education Trust. 60 Queens Road, Reading, RG1 4BS, England.
• Chauhan S. A meta-analysis of the impact of technology on learning effectiveness of elementary students. Computers & Education. 2017; 105 :14–30. doi: 10.1016/j.compedu.2016.11.005. [ CrossRef ] [ Google Scholar ]
• Chen, Q., Chan, K. L., Guo, S., Chen, M., Lo, C. K. M., & Ip, P. (2022a). Effectiveness of digital health interventions in reducing bullying and cyberbullying: a meta-analysis. Trauma, Violence, & Abuse , 15248380221082090. 10.1177/15248380221082090 [ PubMed ]
• Chen B, Wang Y, Wang L. The effects of virtual reality-assisted language learning: A meta-analysis. Sustainability. 2022; 14 (6):3147. doi: 10.3390/su14063147. [ CrossRef ] [ Google Scholar ]
• Cheok ML, Wong SL. Predictors of e-learning satisfaction in teaching and learning for school teachers: A literature review. International Journal of Instruction. 2015; 8 (1):75–90. doi: 10.12973/iji.2015.816a. [ CrossRef ] [ Google Scholar ]
• Cheung, A. C., & Slavin, R. E. (2011). The Effectiveness of Education Technology for Enhancing Reading Achievement: A Meta-Analysis. Center for Research and reform in Education .
• Coban, M., Bolat, Y. I., & Goksu, I. (2022). The potential of immersive virtual reality to enhance learning: A meta-analysis. Educational Research Review , 100452. 10.1016/j.edurev.2022.100452
• Condie, R., & Munro, R. K. (2007). The impact of ICT in schools-a landscape review. Retrieved 30 June 2022 from: https://oei.org.ar/ibertic/evaluacion/sites/default/files/biblioteca/33_impact_ict_in_schools.pdf
• Conrads, J., Rasmussen, M., Winters, N., Geniet, A., Langer, L., (2017). Digital Education Policies in Europe and Beyond: Key Design Principles for More Effective Policies. Redecker, C., P. Kampylis, M. Bacigalupo, Y. Punie (ed.), EUR 29000 EN, Publications Office of the European Union, Luxembourg, 10.2760/462941
• Costa P, Castaño-Muñoz J, Kampylis P. Capturing schools’ digital capacity: Psychometric analyses of the SELFIE self-reflection tool. Computers & Education. 2021; 162 :104080. doi: 10.1016/j.compedu.2020.104080. [ CrossRef ] [ Google Scholar ]
• Cussó-Calabuig R, Farran XC, Bosch-Capblanch X. Effects of intensive use of computers in secondary school on gender differences in attitudes towards ICT: A systematic review. Education and Information Technologies. 2018; 23 (5):2111–2139. doi: 10.1007/s10639-018-9706-6. [ CrossRef ] [ Google Scholar ]
• Daniel SJ. Education and the COVID-19 pandemic. Prospects. 2020; 49 (1):91–96. doi: 10.1007/s11125-020-09464-3. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Delcker J, Ifenthaler D. Teachers’ perspective on school development at German vocational schools during the Covid-19 pandemic. Technology, Pedagogy and Education. 2021; 30 (1):125–139. doi: 10.1080/1475939X.2020.1857826. [ CrossRef ] [ Google Scholar ]
• Delgado, A., Wardlow, L., O’Malley, K., & McKnight, K. (2015). Educational technology: A review of the integration, resources, and effectiveness of technology in K-12 classrooms. Journal of Information Technology Education Research , 14, 397. Retrieved 30 June 2022 from  http://www.jite.org/documents/Vol14/JITEv14ResearchP397-416Delgado1829.pdf
• De Silva MJ, Breuer E, Lee L, Asher L, Chowdhary N, Lund C, Patel V. Theory of change: A theory-driven approach to enhance the Medical Research Council's framework for complex interventions. Trials. 2014; 15 (1):1–13. doi: 10.1186/1745-6215-15-267. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Di Pietro G, Biagi F, Costa P, Karpiński Z, Mazza J. The likely impact of COVID-19 on education: Reflections based on the existing literature and recent international datasets. Publications Office of the European Union; 2020. [ Google Scholar ]
• Elkordy A, Lovinelli J. Competencies, Culture, and Change: A Model for Digital Transformation in K12 Educational Contexts. In: Ifenthaler D, Hofhues S, Egloffstein M, Helbig C, editors. Digital Transformation of Learning Organizations. Springer; 2020. pp. 203–219. [ Google Scholar ]
• Eng TS. The impact of ICT on learning: A review of research. International Education Journal. 2005; 6 (5):635–650. [ Google Scholar ]
• European Commission. (2020). Digital Education Action Plan 2021 – 2027. Resetting education and training for the digital age. Retrieved 30 June 2022 from  https://ec.europa.eu/education/sites/default/files/document-library-docs/deap-communication-sept2020_en.pdf
• European Commission. (2019). 2 nd survey of schools: ICT in education. Objective 1: Benchmark progress in ICT in schools . Retrieved 30 June 2022 from: https://data.europa.eu/euodp/data/storage/f/2019-03-19T084831/FinalreportObjective1-BenchmarkprogressinICTinschools.pdf
• Eurydice. (2019). Digital Education at School in Europe , Luxembourg: Publications Office of the European Union. Retrieved 30 June 2022 from: https://eacea.ec.europa.eu/national-policies/eurydice/content/digital-education-school-europe_en
• Escueta, M., Quan, V., Nickow, A. J., & Oreopoulos, P. (2017). Education technology: An evidence-based review. Retrieved 30 June 2022 from  https://ssrn.com/abstract=3031695
• Fadda D, Pellegrini M, Vivanet G, Zandonella Callegher C. Effects of digital games on student motivation in mathematics: A meta-analysis in K-12. Journal of Computer Assisted Learning. 2022; 38 (1):304–325. doi: 10.1111/jcal.12618. [ CrossRef ] [ Google Scholar ]
• Fernández-Gutiérrez M, Gimenez G, Calero J. Is the use of ICT in education leading to higher student outcomes? Analysis from the Spanish Autonomous Communities. Computers & Education. 2020; 157 :103969. doi: 10.1016/j.compedu.2020.103969. [ CrossRef ] [ Google Scholar ]
• Ferrari, A., Cachia, R., & Punie, Y. (2011). Educational change through technology: A challenge for obligatory schooling in Europe. Lecture Notes in Computer Science , 6964 , 97–110. Retrieved 30 June 2022  https://link.springer.com/content/pdf/10.1007/978-3-642-23985-4.pdf
• Fielding, K., & Murcia, K. (2022). Research linking digital technologies to young children’s creativity: An interpretive framework and systematic review. Issues in Educational Research , 32 (1), 105–125. Retrieved 30 June 2022 from  http://www.iier.org.au/iier32/fielding-abs.html
• Friedel, H., Bos, B., Lee, K., & Smith, S. (2013). The impact of mobile handheld digital devices on student learning: A literature review with meta-analysis. In Society for Information Technology & Teacher Education International Conference (pp. 3708–3717). Association for the Advancement of Computing in Education (AACE).
• Fu JS. ICT in education: A critical literature review and its implications. International Journal of Education and Development Using Information and Communication Technology (IJEDICT) 2013; 9 (1):112–125. [ Google Scholar ]
• Gaol FL, Prasolova-Førland E. Special section editorial: The frontiers of augmented and mixed reality in all levels of education. Education and Information Technologies. 2022; 27 (1):611–623. doi: 10.1007/s10639-021-10746-2. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Garzón J, Acevedo J. Meta-analysis of the impact of Augmented Reality on students’ learning gains. Educational Research Review. 2019; 27 :244–260. doi: 10.1016/j.edurev.2019.04.001. [ CrossRef ] [ Google Scholar ]
• Garzón, J., Baldiris, S., Gutiérrez, J., & Pavón, J. (2020). How do pedagogical approaches affect the impact of augmented reality on education? A meta-analysis and research synthesis. Educational Research Review , 100334. 10.1016/j.edurev.2020.100334
• Grgurović M, Chapelle CA, Shelley MC. A meta-analysis of effectiveness studies on computer technology-supported language learning. ReCALL. 2013; 25 (2):165–198. doi: 10.1017/S0958344013000013. [ CrossRef ] [ Google Scholar ]
• Haßler B, Major L, Hennessy S. Tablet use in schools: A critical review of the evidence for learning outcomes. Journal of Computer Assisted Learning. 2016; 32 (2):139–156. doi: 10.1111/jcal.12123. [ CrossRef ] [ Google Scholar ]
• Haleem A, Javaid M, Qadri MA, Suman R. Understanding the role of digital technologies in education: A review. Sustainable Operations and Computers. 2022; 3 :275–285. doi: 10.1016/j.susoc.2022.05.004. [ CrossRef ] [ Google Scholar ]
• Hardman J. Towards a pedagogical model of teaching with ICTs for mathematics attainment in primary school: A review of studies 2008–2018. Heliyon. 2019; 5 (5):e01726. doi: 10.1016/j.heliyon.2019.e01726. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Hattie J, Rogers HJ, Swaminathan H. The role of meta-analysis in educational research. In: Reid AD, Hart P, Peters MA, editors. A companion to research in education. Springer; 2014. pp. 197–207. [ Google Scholar ]
• Hattie J. Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Routledge. 2008 doi: 10.4324/9780203887332. [ CrossRef ] [ Google Scholar ]
• Higgins S, Xiao Z, Katsipataki M. The impact of digital technology on learning: A summary for the education endowment foundation. Education Endowment Foundation and Durham University; 2012. [ Google Scholar ]
• Higgins, K., Huscroft-D’Angelo, J., & Crawford, L. (2019). Effects of technology in mathematics on achievement, motivation, and attitude: A meta-analysis. Journal of Educational Computing Research , 57(2), 283-319.
• Hillmayr D, Ziernwald L, Reinhold F, Hofer SI, Reiss KM. The potential of digital tools to enhance mathematics and science learning in secondary schools: A context-specific meta-analysis. Computers & Education. 2020; 153 (1038):97. doi: 10.1016/j.compedu.2020.103897. [ CrossRef ] [ Google Scholar ]
• Istenic Starcic A, Bagon S. ICT-supported learning for inclusion of people with special needs: Review of seven educational technology journals, 1970–2011. British Journal of Educational Technology. 2014; 45 (2):202–230. doi: 10.1111/bjet.12086. [ CrossRef ] [ Google Scholar ]
• Jewitt C, Clark W, Hadjithoma-Garstka C. The use of learning platforms to organise learning in English primary and secondary schools. Learning, Media and Technology. 2011; 36 (4):335–348. doi: 10.1080/17439884.2011.621955. [ CrossRef ] [ Google Scholar ]
• JISC. (2020). What is digital transformation?.  Retrieved 30 June 2022 from: https://www.jisc.ac.uk/guides/digital-strategy-framework-for-university-leaders/what-is-digital-transformation
• Kalati, A. T., & Kim, M. S. (2022). What is the effect of touchscreen technology on young children’s learning?: A systematic review. Education and Information Technologies , 1-19. 10.1007/s10639-021-10816-5
• Kalemkuş, J., & Kalemkuş, F. (2022). Effect of the use of augmented reality applications on academic achievement of student in science education: Meta-analysis review. Interactive Learning Environments , 1-18. 10.1080/10494820.2022.2027458
• Kao C-W. The effects of digital game-based learning task in English as a foreign language contexts: A meta-analysis. Education Journal. 2014; 42 (2):113–141. [ Google Scholar ]
• Kampylis P, Punie Y, Devine J. Promoting effective digital-age learning - a European framework for digitally competent educational organisations. JRC Technical Reports. 2015 doi: 10.2791/54070. [ CrossRef ] [ Google Scholar ]
• Kazu IY, Yalçin CK. Investigation of the effectiveness of hybrid learning on academic achievement: A meta-analysis study. International Journal of Progressive Education. 2022; 18 (1):249–265. doi: 10.29329/ijpe.2022.426.14. [ CrossRef ] [ Google Scholar ]
• Koh C. A qualitative meta-analysis on the use of serious games to support learners with intellectual and developmental disabilities: What we know, what we need to know and what we can do. International Journal of Disability, Development and Education. 2022; 69 (3):919–950. doi: 10.1080/1034912X.2020.1746245. [ CrossRef ] [ Google Scholar ]
• König J, Jäger-Biela DJ, Glutsch N. Adapting to online teaching during COVID-19 school closure: Teacher education and teacher competence effects among early career teachers in Germany. European Journal of Teacher Education. 2020; 43 (4):608–622. doi: 10.1080/02619768.2020.1809650. [ CrossRef ] [ Google Scholar ]
• Lawrence JE, Tar UA. Factors that influence teachers’ adoption and integration of ICT in teaching/learning process. Educational Media International. 2018; 55 (1):79–105. doi: 10.1080/09523987.2018.1439712. [ CrossRef ] [ Google Scholar ]
• Lee, S., Kuo, L. J., Xu, Z., & Hu, X. (2020). The effects of technology-integrated classroom instruction on K-12 English language learners’ literacy development: A meta-analysis. Computer Assisted Language Learning , 1-32. 10.1080/09588221.2020.1774612
• Lei, H., Chiu, M. M., Wang, D., Wang, C., & Xie, T. (2022a). Effects of game-based learning on students’ achievement in science: a meta-analysis. Journal of Educational Computing Research . 10.1177/07356331211064543
• Lei H, Wang C, Chiu MM, Chen S. Do educational games affect students' achievement emotions? Evidence from a meta-analysis. Journal of Computer Assisted Learning. 2022; 38 (4):946–959. doi: 10.1111/jcal.12664. [ CrossRef ] [ Google Scholar ]
• Liao YKC, Chang HW, Chen YW. Effects of computer application on elementary school student's achievement: A meta-analysis of students in Taiwan. Computers in the Schools. 2007; 24 (3–4):43–64. doi: 10.1300/J025v24n03_04. [ CrossRef ] [ Google Scholar ]
• Li Q, Ma X. A meta-analysis of the effects of computer technology on school students’ mathematics learning. Educational Psychology Review. 2010; 22 (3):215–243. doi: 10.1007/s10648-010-9125-8. [ CrossRef ] [ Google Scholar ]
• Liu, M., Pang, W., Guo, J., & Zhang, Y. (2022). A meta-analysis of the effect of multimedia technology on creative performance. Education and Information Technologies , 1-28. 10.1007/s10639-022-10981-1
• Lu Z, Chiu MM, Cui Y, Mao W, Lei H. Effects of game-based learning on students’ computational thinking: A meta-analysis. Journal of Educational Computing Research. 2022 doi: 10.1177/07356331221100740. [ CrossRef ] [ Google Scholar ]
• Martinez L, Gimenes M, Lambert E. Entertainment video games for academic learning: A systematic review. Journal of Educational Computing Research. 2022 doi: 10.1177/07356331211053848. [ CrossRef ] [ Google Scholar ]
• Mayne J. Useful theory of change models. Canadian Journal of Program Evaluation. 2015; 30 (2):119–142. doi: 10.3138/cjpe.230. [ CrossRef ] [ Google Scholar ]
• Moran J, Ferdig RE, Pearson PD, Wardrop J, Blomeyer RL., Jr Technology and reading performance in the middle-school grades: A meta-analysis with recommendations for policy and practice. Journal of Literacy Research. 2008; 40 (1):6–58. doi: 10.1080/10862960802070483. [ CrossRef ] [ Google Scholar ]
• OECD. (2015). Students, Computers and Learning: Making the Connection . PISA, OECD Publishing, Paris. Retrieved from: 10.1787/9789264239555-en
• OECD. (2021). OECD Digital Education Outlook 2021: Pushing the Frontiers with Artificial Intelligence, Blockchain and Robots. Retrieved from: https://www.oecd-ilibrary.org/education/oecd-digital-education-outlook-2021_589b283f-en
• Pan Y, Ke F, Xu X. A systematic review of the role of learning games in fostering mathematics education in K-12 settings. Educational Research Review. 2022; 36 :100448. doi: 10.1016/j.edurev.2022.100448. [ CrossRef ] [ Google Scholar ]
• Pettersson F. Understanding digitalization and educational change in school by means of activity theory and the levels of learning concept. Education and Information Technologies. 2021; 26 (1):187–204. doi: 10.1007/s10639-020-10239-8. [ CrossRef ] [ Google Scholar ]
• Pihir, I., Tomičić-Pupek, K., & Furjan, M. T. (2018). Digital transformation insights and trends. In Central European Conference on Information and Intelligent Systems (pp. 141–149). Faculty of Organization and Informatics Varazdin. Retrieved 30 June 2022 from https://www.proquest.com/conference-papers-proceedings/digital-transformation-insights-trends/docview/2125639934/se-2
• Punie, Y., Zinnbauer, D., & Cabrera, M. (2006). A review of the impact of ICT on learning. Working Paper prepared for DG EAC. Retrieved 30 June 2022 from: http://www.eurosfaire.prd.fr/7pc/doc/1224678677_jrc47246n.pdf
• Quah CY, Ng KH. A systematic literature review on digital storytelling authoring tool in education: January 2010 to January 2020. International Journal of Human-Computer Interaction. 2022; 38 (9):851–867. doi: 10.1080/10447318.2021.1972608. [ CrossRef ] [ Google Scholar ]
• Ran H, Kim NJ, Secada WG. A meta-analysis on the effects of technology's functions and roles on students' mathematics achievement in K-12 classrooms. Journal of computer assisted learning. 2022; 38 (1):258–284. doi: 10.1111/jcal.12611. [ CrossRef ] [ Google Scholar ]
• Ređep, N. B. (2021). Comparative overview of the digital preparedness of education systems in selected CEE countries. Center for Policy Studies. CEU Democracy Institute .
• Rott, B., & Marouane, C. (2018). Digitalization in schools–organization, collaboration and communication. In Digital Marketplaces Unleashed (pp. 113–124). Springer, Berlin, Heidelberg.
• Savva M, Higgins S, Beckmann N. Meta-analysis examining the effects of electronic storybooks on language and literacy outcomes for children in grades Pre-K to grade 2. Journal of Computer Assisted Learning. 2022; 38 (2):526–564. doi: 10.1111/jcal.12623. [ CrossRef ] [ Google Scholar ]
• Schmid RF, Bernard RM, Borokhovski E, Tamim RM, Abrami PC, Surkes MA, Wade CA, Woods J. The effects of technology use in postsecondary education: A meta-analysis of classroom applications. Computers & Education. 2014; 72 :271–291. doi: 10.1016/j.compedu.2013.11.002. [ CrossRef ] [ Google Scholar ]
• Schuele CM, Justice LM. The importance of effect sizes in the interpretation of research: Primer on research: Part 3. The ASHA Leader. 2006; 11 (10):14–27. doi: 10.1044/leader.FTR4.11102006.14. [ CrossRef ] [ Google Scholar ]
• Schwabe, A., Lind, F., Kosch, L., & Boomgaarden, H. G. (2022). No negative effects of reading on screen on comprehension of narrative texts compared to print: A meta-analysis. Media Psychology , 1-18. 10.1080/15213269.2022.2070216
• Sellar S. Data infrastructure: a review of expanding accountability systems and large-scale assessments in education. Discourse: Studies in the Cultural Politics of Education. 2015; 36 (5):765–777. doi: 10.1080/01596306.2014.931117. [ CrossRef ] [ Google Scholar ]
• Stock WA. Systematic coding for research synthesis. In: Cooper H, Hedges LV, editors. The handbook of research synthesis, 236. Russel Sage; 1994. pp. 125–138. [ Google Scholar ]
• Su, J., Zhong, Y., & Ng, D. T. K. (2022). A meta-review of literature on educational approaches for teaching AI at the K-12 levels in the Asia-Pacific region. Computers and Education: Artificial Intelligence , 100065. 10.1016/j.caeai.2022.100065
• Su J, Yang W. Artificial intelligence in early childhood education: A scoping review. Computers and Education: Artificial Intelligence. 2022; 3 :100049. doi: 10.1016/j.caeai.2022.100049. [ CrossRef ] [ Google Scholar ]
• Sung YT, Chang KE, Liu TC. The effects of integrating mobile devices with teaching and learning on students' learning performance: A meta-analysis and research synthesis. Computers & Education. 2016; 94 :252–275. doi: 10.1016/j.compedu.2015.11.008. [ CrossRef ] [ Google Scholar ]
• Talan T, Doğan Y, Batdı V. Efficiency of digital and non-digital educational games: A comparative meta-analysis and a meta-thematic analysis. Journal of Research on Technology in Education. 2020; 52 (4):474–514. doi: 10.1080/15391523.2020.1743798. [ CrossRef ] [ Google Scholar ]
• Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of Educational research, 81 (1), 4–28. Retrieved 30 June 2022 from 10.3102/0034654310393361
• Tamim, R. M., Borokhovski, E., Pickup, D., Bernard, R. M., & El Saadi, L. (2015). Tablets for teaching and learning: A systematic review and meta-analysis. Commonwealth of Learning. Retrieved from: http://oasis.col.org/bitstream/handle/11599/1012/2015_Tamim-et-al_Tablets-for-Teaching-and-Learning.pdf
• Tang C, Mao S, Xing Z, Naumann S. Improving student creativity through digital technology products: A literature review. Thinking Skills and Creativity. 2022; 44 :101032. doi: 10.1016/j.tsc.2022.101032. [ CrossRef ] [ Google Scholar ]
• Tolani-Brown, N., McCormac, M., & Zimmermann, R. (2011). An analysis of the research and impact of ICT in education in developing country contexts. In ICTs and sustainable solutions for the digital divide: Theory and perspectives (pp. 218–242). IGI Global.
• Trucano, M. (2005). Knowledge Maps: ICTs in Education. Washington, DC: info Dev / World Bank. Retrieved 30 June 2022 from  https://files.eric.ed.gov/fulltext/ED496513.pdf
• Ulum H. The effects of online education on academic success: A meta-analysis study. Education and Information Technologies. 2022; 27 (1):429–450. doi: 10.1007/s10639-021-10740-8. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Underwood, J. D. (2009). The impact of digital technology: A review of the evidence of the impact of digital technologies on formal education. Retrieved 30 June 2022 from: http://dera.ioe.ac.uk/id/eprint/10491
• Verschaffel, L., Depaepe, F., & Mevarech, Z. (2019). Learning Mathematics in metacognitively oriented ICT-Based learning environments: A systematic review of the literature. Education Research International , 2019 . 10.1155/2019/3402035
• Villena-Taranilla R, Tirado-Olivares S, Cózar-Gutiérrez R, González-Calero JA. Effects of virtual reality on learning outcomes in K-6 education: A meta-analysis. Educational Research Review. 2022; 35 :100434. doi: 10.1016/j.edurev.2022.100434. [ CrossRef ] [ Google Scholar ]
• Voogt J, Knezek G, Cox M, Knezek D, ten Brummelhuis A. Under which conditions does ICT have a positive effect on teaching and learning? A call to action. Journal of Computer Assisted Learning. 2013; 29 (1):4–14. doi: 10.1111/j.1365-2729.2011.00453.x. [ CrossRef ] [ Google Scholar ]
• Vuorikari, R., Punie, Y., & Cabrera, M. (2020). Emerging technologies and the teaching profession: Ethical and pedagogical considerations based on near-future scenarios  (No. JRC120183). Joint Research Centre. Retrieved 30 June 2022 from: https://publications.jrc.ec.europa.eu/repository/handle/JRC120183
• Wang LH, Chen B, Hwang GJ, Guan JQ, Wang YQ. Effects of digital game-based STEM education on students’ learning achievement: A meta-analysis. International Journal of STEM Education. 2022; 9 (1):1–13. doi: 10.1186/s40594-022-00344-0. [ CrossRef ] [ Google Scholar ]
• Wen X, Walters SM. The impact of technology on students’ writing performances in elementary classrooms: A meta-analysis. Computers and Education Open. 2022; 3 :100082. doi: 10.1016/j.caeo.2022.100082. [ CrossRef ] [ Google Scholar ]
• Zheng B, Warschauer M, Lin CH, Chang C. Learning in one-to-one laptop environments: A meta-analysis and research synthesis. Review of Educational Research. 2016; 86 (4):1052–1084. doi: 10.3102/0034654316628645. [ CrossRef ] [ Google Scholar ]

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Artificial intelligence as a disruptive technology—a systematic literature review.

1. Introduction

2. materials and methods.

• First exclusion: ○ Document types—the Editorial Materials and Meeting Abstracts were removed (WoS—38, S—42), leaving 124 (WoS) + 142 (S) = 266 papers; ○ All publishers with only 1 article, as we considered that they did not have a serious approach toward this topic, were removed (WoS—20, S—23), leaving 104 (WoS) + 119 (S). Further, at this stage, the intermediary results (1) were merged into the same file, resulting in 223 articles.
• Second exclusion: ○ With the support of EndNote (used for reference management), it was possible to identify duplicate records (196) originating from the two databases and retain only 1 entry (98). In this manner, we obtained the intermediary results (2), with a total of 125 references.
• Third exclusion: ○ The remaining list was evaluated for relevance based on title, keyword, and abstract analysis, and the articles that did not fit the purpose of the research were eliminated (−28), leaving a total of 97 papers included in the study.

3.1. AI as a Disruptive Technology in Healthcare (Medicine)

3.1.1. disruptive features in the applications to surgery, 3.1.2. disruptive features in the applications to healthcare, 3.2. ai as a disruptive technology in business—logistics and transportation and the labor market, 3.2.1. logistics, 3.2.2. labor market, 3.3. ai as a disruptive technology in agriculture, 3.3.1. smart farming, 3.3.2. digital twins, 3.3.3. the fourth industrial revolution (4ir), 3.4. ai as a disruptive technology in education, 3.5. ai as a disruptive technology with respect to urban development—society, smart cities, and smart government, 3.5.1. disruptive technology’s impact on society, 3.5.2. smart cities, 3.5.3. smart government, 4. discussion and conclusions.

• Enhanced diagnosis, as AI algorithms can examine a large number of medical data to help clinicians make more accurate diagnoses, thus minimizing the possibility of misdiagnosis;
• Personalized medicine, since by using a patient’s particular medical history and genetic data, AI can aid the development of individualized treatment approaches;
• Superior patient outcomes, as AI may be used to track patients, anticipate future health difficulties, and alert medical professionals to take preventative action before significant health issues arise;
• Expedite drug development, because AI can analyze massive volumes of data to hasten the process of developing new drugs and bringing them to market;
• Improved clinical trials, due to the fact that data from clinical trials may be analyzed using AI algorithms, thus assisting in the selection of the most efficient therapies and enhancing patient results.
• The development of AI in healthcare creates ethical issues, such as the issue of responsibility in situations of misdiagnosis or treatment suggestions;
• Limited clinical validity poses a serious problem, because in certain complicated medical situations, AI algorithms may not be as accurate as human specialists and may not be completely verified for assessing all medical disorders;
• Healthcare professionals and patients who are suspicious about the accuracy and dependability of the technology can be resistant to the adoption of AI in the industry.
• For improved supply chain management, AI may aid routing, scheduling, and delivery optimization, which lowers transportation costs and increases delivery times;
• Transportation safety may be improved by using AI to track and improve driver behavior, reduce collisions, and increase road safety;
• AI can enhance logistics efficiency, as it may be used to improve inventory management, optimize storage and picking procedures, and expedite warehouse operations;
• AI is transforming the labor sector by replacing many old manual jobs while also opening up new career prospects in programming and data analysis;
• AI may improve customer experience as it can be used to offer updates on tracking and delivery in real-time, thereby reducing wait times and raising satisfaction;
• AI may aid the maximization of fuel use and the cutting of emissions through effective vehicle scheduling and routing and thus contribute to minimized environmental impacts;
• Many laborious and repetitive tasks will be automated, which may result in fewer jobs and employment possibilities, particularly in sectors such as logistics and transportation;
• As the demand for more high-skilled positions in AI and data analysis increases and fewer low-skilled occupations are automated, the rising usage of AI may worsen already-existing income discrepancies;
• The widespread usage of autonomous cars may result in substantial social and cultural changes, such as the loss of individual driving abilities and the demise of the automobile culture.
• Improved agricultural yields and less waste are possible with the use of AI, which may help farmers optimize planting, irrigation, and fertilization;
• Better resource management may help farmers conserve energy, water, and other resources while decreasing waste and enhancing sustainability;
• Enhanced food safety can be enforced by tracking the whole food production chain from farm to table, while AI can assist in the identification and prevention of food-borne diseases;
• AI can provide real-time analysis of crop, soil, and weather variables, thus enabling farmers to make educated decisions;
• Predictive maintenance may reduce downtime and boost production by predicting when machines and equipment need maintenance.
• AI systems are not immune to technical glitches or malfunctions, and the agricultural sector might suffer significantly as a result, leading to crop losses and possible food shortages;
• The usage of AI in agriculture may have unforeseen environmental effects, including increased pesticide and herbicide use, degraded soil, and the loss of biodiversity.
• A decrease in dropout rates and improved student results due to AI’s ability to detect students’ areas of need and offer focused support;
• Education that is customized to each student’s requirements, interests, and learning preferences may be achieved by using AI to deliver personalized learning experiences for students;
• Improved assessment and feedback due to AI’s ability to automate, enhance, and optimize the grading and feedback process and provide students faster, more precise, and more thorough feedback on their work;
• Lifelong learning is possible because of AI, which can help people continue to learn and advance their expertise.
• Education quality may suffer due to the usage of AI in the classroom when human interaction, creativity, and critical thinking abilities are substituted by automated procedures;
• A lack of critical thinking abilities may be precipitated by AI because the use of AI-powered tools and resources may lessen the necessity for critical thinking and problem-solving abilities, which may retard the development of these skills among students;
• The dependence on technology due to an overreliance on AI in the classroom may result in a lack of creativity, independence, and decision-making abilities, which will reduce students’ capacity to think and work independently.
• An increase in transparency, as by using AI to render governmental processes more open and accountable, individuals will be able to better understand how choices are made;
• Enhanced fraud detection, since AI may be used to identify and stop corruption and fraud in government systems, thus increasing public confidence in these organizations;
• Better resource allocation, because governmental organizations may use AI to more effectively direct resources, including money and staff, to the areas where they are most needed;
• The introduction of predictive analytics, as through the use of AI, government agencies may employ predictive analytics to proactively address prospective concerns before they become problems.
• Privacy issues—Government entities frequently deploy AI algorithms that rely on substantial volumes of personal data, which raises privacy concerns regarding how these data are gathered, kept, and used;
• Lack of transparency—AI technologies employed by government agencies may be opaque, making it difficult for the public to comprehend how and why choices are being made;
• The employment of AI in governmental affairs may result in greater control and surveillance, which may have detrimental effects on free expression and civil rights;
• When an AI system utilized by a government errs or causes harm, it may be challenging to pinpoint the culprit, which results in a lack of accountability.

Author Contributions

Data availability statement, conflicts of interest.

• Bower, L.J.; Christensen, M.C. Disruptive technologies: Catching the wave. Har. Buss. Rev. 1995 , 73 , 43–53. [ Google Scholar ]
• Christensen, C.M.; Bower, J.L. Customer power, strategic investment, and the failure of leading firms. Strateg. Manag. J. 1996 , 17 , 197–218. [ Google Scholar ]
• Christensen, C.M. The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail ; Harvard Business School Press: Boston, MA, USA, 1997; 179p. [ Google Scholar ]
• O’Connor, S.; Sidorko, P. Chapter 2—The complexities of our informational environment. In Imagine Your Library’s Future ; Chandos Publishing: Oxford, UK, 2010; pp. 33–60. [ Google Scholar ]
• Laukyte, M. Disruptive Technologies and the Sport Ecosystem: A Few Ethical Questions. Philosophies 2020 , 5 , 24. [ Google Scholar ] [ CrossRef ]
• Jekov, B.; Petkova, P.; Parusheva, Y.; Shoikova, E. Disruptive Technologies—Artificial Intelligence and Blockchain in Education. In Proceedings of the 11th Annual International Conference of Education, Research and Innovation (ICERI), Seville, Spain, 7–9 November 2022; pp. 6784–6793. [ Google Scholar ]
• Bongomin, O.; Ocen, G.G.; Nganyi, E.O.; Musinguzi, A.; Omara, T. Exponential Disruptive Technologies and the Required Skills of Industry 4.0. J. Eng. 2020 , 2020 , 4280156. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Hernández, R. World Standards Day 2018 Puts the Spotlight on the Fourth Industrial Revolution. Available online: https://www.iso.org/news/ref2333.html (accessed on 19 January 2023).
• Bird, K. Four Trends Will Impact ISO’s Future Strategy. Available online: https://www.iso.org/news/ref2436.html (accessed on 19 January 2023).
• Bublitz, F.M.; Oetomo, A.; Sahu, K.S.; Kuang, A.; Fadrique, L.X.; Velmovitsky, P.E.; Nobrega, R.M.; Morita, P.P. Disruptive Technologies for Environment and Health Research: An Overview of Artificial Intelligence, Blockchain, and Internet of Things. Int. J. Environ. Res. Public Health 2019 , 16 , 3847. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Chang, N.; Zhang, Y.; Lu, D.; Zheng, X.; Xue, J. Is a Disruptive Technology Disruptive? The Readiness Perspective Based on TOE. In Proceedings of the 2020 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM), Singapore, 14–17 December 2020; pp. 893–897. [ Google Scholar ]
• Cartaxo, B.; Pinto, G.; Soares, S. The role of rapid reviews in supporting decision-making in software engineering practice. In Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering—EASE’18, Christchurch, New Zealand, 28–29 June 2018. [ Google Scholar ]
• Christensen, C.M.; McDonald, R.; Altman, E.J.; Palmer, J.E. Disruptive innovation: An intellectual history and directions for future research. J. Manag. Stud. 2018 , 55 , 1043–1078. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Popescul, D.; Radu, L.D.; Păvăloaia, V.D.; Georgescu, M.R. Psychological Determinants of Investor Motivation in Social Media-Based Crowdfunding Projects: A Systematic Review. Front. Psychol. 2020 , 11 , 588121. [ Google Scholar ] [ CrossRef ]
• Radu, L.D. Disruptive Technologies in Smart Cities: A Survey on Current Trends and Challenges. Smart Cities 2020 , 3 , 1022–1038. [ Google Scholar ] [ CrossRef ]
• VOSViewer. Visualizing Scientific Landscapes. Available online: https://www.vosviewer.com/features/highlights (accessed on 20 July 2022).
• Basmmi, A.B.M.N.; Abd Halim, S.; Saadon, N.A. Comparison of web services for sentiment analysis in social networking sites. In Proceedings of the IOP Conference Series: Materials Science and Engineering, Tangerang, Indonesia, 18–20 November 2020. [ Google Scholar ]
• Abdullah, N.S.D.; Zolkepli, I.A. Sentiment Analysis of Online Crowd Input towards Brand Provocation in Facebook, Twitter, and Instagram. In Proceedings of the International Conference on Big Data and Internet of Thing—BDIOT2017, London, UK, 20–22 December 2017. [ Google Scholar ]
• Byrne, M.; O’Malley, L.; Glenny, A.M.; Pretty, I.; Tickle, M. Assessing the reliability of automatic sentiment analysis tools on rating the sentiment of reviews of NHS dental practices in England. PLoS ONE 2021 , 16 , e0259797. [ Google Scholar ] [ CrossRef ]
• Saura, J.R.; Reyes-Menendez, A.; Alvarez-Alonso, C. Do Online Comments Affect Environmental Management? Identifying Factors Related to Environmental Management and Sustainability of Hotels. Sustainability 2018 , 10 , 3016. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Pahuriray, A.V.; Basanta, J.D.; Arroyo, J.C.T.; Delima, A.J.P. Flexible Learning Experience Analyzer (FLExA): Sentiment Analysis of College Students through Machine Learning Algorithms with Comparative Analysis using WEKA. Int. J. Emerg. Technol. Adv. Eng. 2022 , 12 , 1–15. [ Google Scholar ] [ CrossRef ]
• Stoiber, C.; Walchshofer, C.; Pohl, M.; Potzmann, B.; Grassinger, F.; Stitz, H.; Streit, M.; Aigner, W. Comparative evaluations of visualization onboarding methods. Vis. Inform. 2022 , 6 , 34–50. [ Google Scholar ] [ CrossRef ]
• Zhuang, Y. Emotional analysis of sentences based on machine learning. In Big Data Analytics for Cyber-Physical System in Smart City ; Advances in Intelligent Systems and Computing; Atiquzzaman, M., Yen, N., Xu, Z., Eds.; Springer: Singapore, 2020; pp. 813–820. [ Google Scholar ] [ CrossRef ]
• Elangovan, K.; JasmineRani, L.P.; Karthikeyan, M.P.; Therasa, M. Analysis of Social Network with Ontology and Deep Sentiment Durability Detection (SSD) Model for Green Community. J. Green Eng. 2020 , 10 , 2661–2677. [ Google Scholar ]
• Monkeylearn. No-Code Text Analytics. Available online: https://monkeylearn.com (accessed on 20 July 2022).
• Contreras, D.; Wilkinson, S.; Alterman, E.; Hervás, J. Accuracy of a pre-trained sentiment analysis (SA) classification model on tweets related to emergency response and early recovery assessment: The case of 2019 Albanian earthquake. Nat. Hazards 2022 , 113 , 403–421. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Contreras, D.; Wilkinson, S.; Balan, N.; James, P. Assessing post-disaster recovery using sentiment analysis: The case of L’Aquila. Earthq. Spectra 2022 , 38 , 81–108. [ Google Scholar ] [ CrossRef ]
• Sadriu, S.; Nuci, K.P.; Imran, A.S.; Uddin, I.; Sajjad, M. An Automated Approach for Analysing Students Feedback Using Sentiment Analysis Techniques. In Mediterranean Conference on Pattern Recognition and Artificial Intelligence ; Springer International Publishing: Cham, Switzerland, 2021; pp. 228–239. [ Google Scholar ]
• Bredava, A. A Guide to Sentiment Analysis: What Is It and How Does It Work? Available online: https://awario.com/blog/sentiment-analysis (accessed on 15 January 2023).
• Ryman-Tubb, N.F.; Krause, P.; Garn, W. How Artificial Intelligence and machine learning research impacts payment card fraud detection: A survey and industry benchmark. Eng. Appl. Artif. Intell. 2018 , 76 , 130–157. [ Google Scholar ] [ CrossRef ]
• Aria, M.; Cuccurullo, C. Bibliometrix: An R-tool for comprehensive science mapping analysis. J. Informetr. 2017 , 11 , 959–975. [ Google Scholar ] [ CrossRef ]
• International Organization for Standardization. ISO Standardization Foresight Framework—Trend Report 2022. Available online: https://www.iso.org/files/live/sites/isoorg/files/store/en/PUB100470.pdf (accessed on 19 January 2023).
• Dal Mas, F.; Piccolo, D.; Cobianchi, L.; Edvinsson, L.; Presch, G.; Massaro, M.; Skrap, M.; Vajana, A.F.D.; D’Auria, S.D.S.; Bagnoli, C. The Effects of Artificial Intelligence, Robotics, and Industry 4.0 Technologies. Insights from the Healthcare Sector. In Proceedings of the European Conference on the Impact of Artificial Intelligence and Robotics (ECIAIR), EM Normandie Business Sch, Oxford, UK, 31 Octorber–1 November 2022; pp. 88–95. [ Google Scholar ]
• Manickam, P.; Mariappan, S.A.; Murugesan, S.M.; Hansda, S.; Kaushik, A.; Shinde, R.; Thipperudraswamy, S.P. Artificial Intelligence (AI) and Internet of Medical Things (IoMT) Assisted Biomedical Systems for Intelligent Healthcare. Biosensors 2022 , 12 , 562. [ Google Scholar ] [ CrossRef ]
• Kelly, J.T.; Collins, P.F.; McCamley, J.; Ball, L.; Roberts, S.; Campbell, K.L. Digital disruption of dietetics: Are we ready? J. Hum. Nutr. Diet. 2021 , 34 , 134–146. [ Google Scholar ] [ CrossRef ]
• Joda, T.; Yeung, A.W.K.; Hung, K.; Zitzmann, N.U.; Bornstein, M.M. Disruptive Innovation in Dentistry: What It Is and What Could Be Next. J. Dent. Res. 2021 , 100 , 448–453. [ Google Scholar ] [ CrossRef ]
• Ahmad, P.; Alam, M.; Aldajani, A.; Alahmari, A.; Alanazi, A.; Stoddart, M.; Sghaireen, M. Dental Robotics: A Disruptive Technology. Sensors 2021 , 21 , 3308. [ Google Scholar ] [ CrossRef ]
• McBee, M.P.; Wilcox, C. Blockchain Technology: Principles and Applications in Medical Imaging. J. Digit. Imaging 2020 , 33 , 726–734. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Rasouli, J.J.; Shao, J.N.; Neifert, S.; Gibbs, W.N.; Habboub, G.; Steinmetz, M.P.; Benzel, E.; Mroz, T.E. Artificial Intelligence and Robotics in Spine Surgery. Glob. Spine J. 2021 , 11 , 556–564. [ Google Scholar ] [ CrossRef ] [ PubMed ] [ Green Version ]
• Dorweiler, B.; Wegner, M.; Salem, O.; Murtaja, A.; Schäfers, J.F.; Oberhuber, A. Innovation, disruptive technologies and transformation in vascular surgery. Gefasschirurgie 2022 , 27 , 561–568. [ Google Scholar ] [ CrossRef ]
• Mohanty, K.; Subiksha, S.; Kirthika, S.; Bh, S.; Sokkanarayanan, S.; Bose, P.; Sathiyanarayanan, M.; IEEE. Opportunities of Adopting AI-Powered Robotics to Tackle COVID-19. In Proceedings of the International Conference on COMmunication Systems and NETworkS (COMSNETS), Bangalore, India, 5–9 January 2022; pp. 703–708. [ Google Scholar ]
• Jabarulla, M.Y.; Lee, H.N. A Blockchain and Artificial Intelligence-Based, Patient-Centric Healthcare System for Combating the COVID-19 Pandemic: Opportunities and Applications. Healthcare 2021 , 9 , 1019. [ Google Scholar ] [ CrossRef ]
• Saraswat, D.; Bhattacharya, P.; Verma, A.; Prasad, V.K.; Tanwar, S.; Sharma, G.; Bokoro, P.N.; Sharma, R. Explainable AI for Healthcare 5.0: Opportunities and Challenges. IEEE Access 2022 , 10 , 84486–84517. [ Google Scholar ] [ CrossRef ]
• Mesko, B.; Hetenyi, G.; Gyorffy, Z. Will artificial intelligence solve the human resource crisis in healthcare? BMC Health Serv. Res. 2018 , 18 , 545. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Maliha, G.; Gerke, S.; Cohen, G.; Parikh, R.B. Artificial Intelligence and Liability in Medicine: Balancing Safety and Innovation. Milbank Q. 2021 , 99 , 629–647. [ Google Scholar ] [ CrossRef ]
• Khatab, Z.; Yousef, G.M. Disruptive innovations in the clinical laboratory: Catching the wave of precision diagnostics. Crit. Rev. Clin. Lab. Sci. 2021 , 58 , 546–562. [ Google Scholar ] [ CrossRef ]
• Brunelle, F.; Brunelle, P. Artificial Intelligence and Medical Imaging: Definition, State of the Art and Perspectives. Bull. Acad. Natl. Med. 2019 , 203 , 683–687. [ Google Scholar ]
• Garbuio, M.; Lin, N. Artificial Intelligence as a Growth Engine for Health Care Startups: Emerging Business Models. Calif. Manag. Rev. 2019 , 61 , 59–83. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Prakash, S.; Balaji, J.N.; Joshi, A.; Surapaneni, K.M. Ethical Conundrums in the Application of Artificial Intelligence (AI) in Healthcare—A Scoping Review of Reviews. J. Pers. Med. 2022 , 12 , 1914. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Mesko, B. Future Directions of Digital Health. In Digital Health: Scaling Healthcare to the World ; Rivas, H., Wac, K., Eds.; Springer: Berlin/Heidelberg, Germany, 2018; pp. 339–363. [ Google Scholar ] [ CrossRef ]
• Ojo, A. Next Generation Government—Hyperconnected, Smart and Augmented. In Proceedings of the 20th IFIP WG 5.5 Working Conference on Virtual Enterprises (PRO-VE), Turin, Italy, 23–25 September 2019; pp. 285–294. [ Google Scholar ]
• Ljepava, N. AI-Enabled Marketing Solutions in Marketing Decision Making: AI Application in Different Stages of Marketing Process. TEM J. 2022 , 11 , 1308–1315. [ Google Scholar ] [ CrossRef ]
• Wiegandt, D. Blockchain and Smart Contracts and the Role of Arbitration. J. Int. Arbitr. 2022 , 39 , 671–690. [ Google Scholar ] [ CrossRef ]
• Dong, C.W.; Akram, A.; Andersson, D.; Arnas, P.O.; Stefansson, G. The impact of emerging and disruptive technologies on freight transportation in the digital era: Current state and future trends. Int. J. Logist. Manag. 2021 , 32 , 386–412. [ Google Scholar ] [ CrossRef ]
• Noor, A. Adoption of Blockchain Technology Facilitates a Competitive Edge for Logistic Service Providers. Sustainability 2022 , 14 , 15543. [ Google Scholar ] [ CrossRef ]
• Ugochukwu, N.A.; Goyal, S.B.; Rajawat, A.S.; Islam, S.M.N.; He, J.; Aslam, M. An Innovative Blockchain-Based Secured Logistics Management Architecture: Utilizing an RSA Asymmetric Encryption Method. Mathematics 2022 , 10 , 4670. [ Google Scholar ] [ CrossRef ]
• Zondervan, N.A.; Tolentino-Zondervan, F.; Moeke, D. Logistics Trends and Innovations in Response to COVID-19 Pandemic: An Analysis Using Text Mining. Processes 2022 , 10 , 2667. [ Google Scholar ] [ CrossRef ]
• Hong, Z.F.; Zhang, H.Y.; Gong, Y.M.; Yu, Y.G. Towards a multi-party interaction framework: State-of-the-art review in sustainable operations management. Int. J. Prod. Res. 2022 , 60 , 2625–2661. [ Google Scholar ] [ CrossRef ]
• Yu, H. Modeling a remanufacturing reverse logistics planning problem: Some insights into disruptive technology adoption. Int. J. Adv. Manuf. Technol. 2022 , 123 , 4231–4249. [ Google Scholar ] [ CrossRef ]
• Jia, W.; Wang, S.; Xie, Y.; Chen, Z.; Gong, K. Disruptive technology identification of intelligent logistics robots in AIoT industry: Based on attributes and functions analysis. Syst. Res. Behav. Sci. 2022 , 39 , 557–568. [ Google Scholar ] [ CrossRef ]
• Cukier, W. Disruptive processes and skills mismatches in the new economy Theorizing social inclusion and innovation as solutions. J. Glob. Responsib. 2019 , 10 , 211–225. [ Google Scholar ] [ CrossRef ]
• Oosthuizen, R.M. The Fourth Industrial Revolution—Smart Technology, Artificial Intelligence, Robotics and Algorithms: Industrial Psychologists in Future Workplaces. Front. Artif. Intell. 2022 , 5 , 913168. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Khatri, S.; Pandey, D.K.; Penkar, D.; Ramani, J. Impact of Artificial Intelligence on Human Resources. In Data Management, Analytics and Innovation ; Advances in Intelligent Systems and Computing; Sharma, N., Chakrabarti, A., Balas, V., Eds.; Springer: Singapore, 2019. [ Google Scholar ]
• Burden, K. Impact of disruptive technologies on sourcing and outsourcing transactions. Comput. Law Secur. Rev. 2018 , 34 , 886–889. [ Google Scholar ] [ CrossRef ]
• Miklosik, A.; Evans, N. Impact of Big Data and Machine Learning on Digital Transformation in Marketing: A Literature Review. IEEE Access 2020 , 8 , 101284–101292. [ Google Scholar ] [ CrossRef ]
• Koizumi, S. The Light and Shadow of the Fourth Industrial Revolution. In Innovation beyond Technology: Science for Society and Interdisciplinary Approaches ; Lechevalier, S., Ed.; Springer: Berlin/Heidelberg, Germany, 2019; pp. 63–86. [ Google Scholar ] [ CrossRef ]
• Rosales, M.A.; Magsumbol, J.A.V.; Palconit, M.G.B.; Culaba, A.B.; Dadios, E.P. Artificial Intelligence: The Technology Adoption and Impact in the Philippines. In Proceedings of the IEEE 12th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management (HNICEM), Manila, Philippines, 3–7 December 2020. [ Google Scholar ]
• Anitha, J.; Saranya, N. Cassava Leaf Disease Identification and Detection Using Deep Learning Approach. Int. J. Comput. Commun. Control 2022 , 17 , 1–7. [ Google Scholar ] [ CrossRef ]
• Ortola, A. Unlocking the full potential of artificial intelligence in PH. Philippine Daily Inquirer , 24 June 2019. [ Google Scholar ]
• Karimanzira, D.; Rauschenbach, T. An intelligent management system for aquaponics. at-Automatisierungstechnik 2021 , 69 , 345–350. [ Google Scholar ] [ CrossRef ]
• Jacobs, M.; Remus, A.; Gaillard, C.; Menendez, H.M.; Tedeschi, L.O.; Neethirajan, S.; Ellis, J.L. ASAS-NANP symposium: Mathematical modeling in animal nutrition: Limitations and potential next steps for modeling and modelers in the animal sciences. J. Anim. Sci. 2022 , 100 , skac132. [ Google Scholar ] [ CrossRef ]
• De Bernardi, P.; Azucar, D.; Forliano, C.; Franco, M. Innovative and Sustainable Food Business Models. In Innovation in Food Ecosystems: Entrepreneurship for a Sustainable Future ; Springer: Berlin/Heidelberg, Germany, 2020; pp. 189–221. [ Google Scholar ] [ CrossRef ]
• Neethirajan, S.; Kemp, B. Digital Twins in Livestock Farming. Animals 2021 , 11 , 1008. [ Google Scholar ] [ CrossRef ]
• Spanaki, K.; Sivarajah, U.; Fakhimi, M.; Despoudi, S.; Irani, Z. Disruptive technologies in agricultural operations: A systematic review of AI-driven AgriTech research. Ann. Oper. Res. 2022 , 308 , 491–524. [ Google Scholar ] [ CrossRef ]
• Kamble, S.; Gunasekaran, A.; Sharma, R. Modeling the blockchain enabled traceability in agriculture supply chain. Int. J. Inf. Manag. 2020 , 52 , 101967. [ Google Scholar ] [ CrossRef ]
• Wamba, S.F.; Queiroz, M.M. Blockchain in the operations and supply chain management: Benefits, challenges and future research opportunities. Int. J. Inf. Manag. 2020 , 52 , 102064. [ Google Scholar ] [ CrossRef ]
• Moore, W.B.; Felo, A. The evolution of accounting technology education: Analytics to STEM. J. Educ. Bus. 2022 , 97 , 105–111. [ Google Scholar ] [ CrossRef ]
• Brito, C.R.; Ciampi, M.M.; Sluss, J.J.; Santos, H.D. Trends in Engineering Education: A Disruptive View for not so far Future. In Proceedings of the 18th International Conference on Information Technology Based Higher Education and Training (ITHET), Magdeburg, Germany, 26–27 September 2019. [ Google Scholar ]
• Ghani, A. Engineering education at the age of Industry 5.0—Higher education at the crossroads. World Trans. Eng. Technol. Educ. 2022 , 20 , 112–117. [ Google Scholar ]
• Bühler, M.M.; Jelinek, T.; Nübel, K. Training and Preparing Tomorrow’s Workforce for the Fourth Industrial Revolution. Educ. Sci. 2022 , 12 , 782. [ Google Scholar ] [ CrossRef ]
• Chaka, C. Is Education 4.0 a Sufficient Innovative, and Disruptive Educational Trend to Promote Sustainable Open Education for Higher Education Institutions? A Review of Literature Trends. Front. Educ. 2022 , 7 , 226. [ Google Scholar ] [ CrossRef ]
• Choi, T.-M.; Kumar, S.; Yue, X.; Chan, H.-L. Disruptive Technologies and Operations Management in the Industry 4.0 Era and Beyond. Prod. Oper. Manag. 2022 , 31 , 9–31. [ Google Scholar ] [ CrossRef ]
• Bengoechea, J.; Bell, A. The Impact of 21st Century Technology in Higher Education: The Role of Artificial Intelligence. Int. J. Educ. Pedagog. Sci. 2022 , 16 , 493–496. [ Google Scholar ]
• Miao, F.; Wayne, H.; Ronghuai, H.; Hui, Z. AI and Education: A Guidance for Policymakers ; UNESCO Publishing: Paris, France, 2021. [ Google Scholar ]
• Pornpongtechavanich, P.; Eambunnapong, K.; Daengsi, T.; Nilsook, P. Critical success factors for smart-professional disruptor in university. Int. J. Eval. Res. Educ. 2022 , 11 , 1696–1703. [ Google Scholar ] [ CrossRef ]
• Molenaar, I. Towards hybrid human-AI learning technologies. Eur. J. Educ. 2022 , 57 , 632–645. [ Google Scholar ] [ CrossRef ]
• Holmes, W.; Tuomi, I. State of the art and practice in AI in education. Eur. J. Educ. 2022 , 53 , 542–570. [ Google Scholar ] [ CrossRef ]
• Zeeshan, K.; Hämäläinen, T.; Neittaanmäki, P. Internet of Things for Sustainable Smart Education: An Overview. Sustainability 2022 , 14 , 4293. [ Google Scholar ] [ CrossRef ]
• Ciolacu, M.; Svasta, P.M.; Berg, W.; Popp, H.; IEEE. Education 4.0 for Tall Thin Engineer in a Data Driven Society. In Proceedings of the 23rd IEEE International Symposium for Design and Technology in Electronic Packaging (SIITME), Constanta, Romania, 26–29 October 2022; pp. 438–443. [ Google Scholar ]
• Ahmad, T. Scenario based approach to re-imagining future of higher education which prepares students for the future of work. High. Educ. Ski. Work-Based Learn. 2019 , 10 , 217–238. [ Google Scholar ] [ CrossRef ]
• Yang, K.; Shi, Y.M.; Zhou, Y.; Yang, Z.P.; Fu, L.Q.; Chen, W. Federated Machine Learning for Intelligent IoT via Reconfigurable Intelligent Surface. IEEE Netw. 2020 , 34 , 16–22. [ Google Scholar ] [ CrossRef ]
• Kasinathan, P.; Pugazhendhi, R.; Elavarasan, R.M.; Ramachandaramurthy, V.K.; Ramanathan, V.; Subramanian, S.; Kumar, S.; Nandhagopal, K.; Raghavan, R.R.V.; Rangasamy, S.; et al. Realization of Sustainable Development Goals with Disruptive Technologies by Integrating Industry 5.0, Society 5.0, Smart Cities and Villages. Sustainability 2022 , 14 , 15258. [ Google Scholar ] [ CrossRef ]
• Sgantzos, K.; Grigg, I. Artificial Intelligence Implementations on the Blockchain. Use Cases and Future Applications. Future Internet 2019 , 11 , 170. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Corchado, J.M.; Trabelsi, S. Advances in Sustainable Smart Cities and Territories. Electronics 2022 , 11 , 1280. [ Google Scholar ] [ CrossRef ]
• Xu, Y. Programmatic Dreams: Technographic Inquiry into Censorship of Chinese Chatbots. Soc. Media + Soc. 2018 , 4 , 2056305118808780. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Zweispace. Zweispace Won the Best Startup Award for Real Estate and Construction Batch at the Plug and Play Summer Summit 2020. Available online: https://en.zweispace.co.jp/2020/08/04/zweispace-won-the-best-startup-award-for-real-estate-and-construction-batch-at-the-plug-and-play-summer-summit-2020/ (accessed on 4 August 2022).
• Ortega-Fernández, A.; Martín-Rojas, R.; García-Morales, V. Artificial Intelligence in the Urban Environment: Smart Cities as Models for Developing Innovation and Sustainability. Sustainability 2020 , 12 , 7860. [ Google Scholar ] [ CrossRef ]
• Schintler, L.A.; McNeely, C.L. Artificial intelligence, institutions, and resilience: Prospects and provocations for cities. J. Urban Manag. 2022 , 11 , 256–268. [ Google Scholar ] [ CrossRef ]
• Fu, G.; Jin, Y.; Sun, S.; Yuan, Z.; Butler, D. The role of deep learning in urban water management: A critical review. Water Res. 2022 , 223 , 118973. [ Google Scholar ] [ CrossRef ]
• Ullah, F. Smart Tech 4.0 in the Built Environment: Applications of Disruptive Digital Technologies in Smart Cities, Construction, and Real Estate. Buildings 2022 , 12 , 1516. [ Google Scholar ] [ CrossRef ]
• Ullah, F.; Sepasgozar, S.M.E.; Wang, C.X. A Systematic Review of Smart Real Estate Technology: Drivers of, and Barriers to, the Use of Digital Disruptive Technologies and Online Platforms. Sustainability 2018 , 10 , 3142. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Ghobakhloo, M. Industry 4.0, digitization, and opportunities for sustainability. J. Clean. Prod. 2020 , 252 , 119869. [ Google Scholar ] [ CrossRef ]
• Khanna, A.; Sah, A.; Bolshev, V.; Jasinski, M.; Vinogradov, A.; Leonowicz, Z.; Jasiński, M. Blockchain: Future of e-Governance in Smart Cities. Sustainability 2021 , 13 , 11840. [ Google Scholar ] [ CrossRef ]
• McLean, J.; Mackenzie, R. Digital justice in Australian visa application processes? Altern. Law J. 2019 , 44 , 291–296. [ Google Scholar ] [ CrossRef ]
• Ishengoma, F.R.; Shao, D.; Alexopoulos, C.; Saxena, S.; Nikiforova, A. Integration of artificial intelligence of things (AIoT) in the public sector: Drivers, barriers and future research agenda. Digit. Policy Regul. Gov. 2022 , 24 , 449–462. [ Google Scholar ] [ CrossRef ]
• Rapanyane, M.B.; Sethole, F.R. The rise of artificial intelligence and robots in the 4th Industrial Revolution: Implications for future South African job creation. Contemp. Soc. Sci. 2020 , 15 , 489–501. [ Google Scholar ] [ CrossRef ]
• Mesko, B. The role of artificial intelligence in precision medicine. Expert Rev. Precis. Med. Drug Dev. 2017 , 2 , 239–241. [ Google Scholar ] [ CrossRef ]
• Brougham, D.; Haar, J. Smart Technology, Artificial Intelligence, Robotics, and Algorithms (STARA): Employees’ perceptions of our future workplace. J. Manag. Organ. 2018 , 24 , 239–257. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Frey, C.B.; Osborne, M.A. The future of employment: How susceptible are jobs to computerisation? Technol. Forecast. Soc. Chang. 2017 , 114 , 254–280. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Păvăloaia, V.-D.; Necula, S.-C. Artificial Intelligence as a Disruptive Technology—A Systematic Literature Review. Electronics 2023 , 12 , 1102. https://doi.org/10.3390/electronics12051102

Păvăloaia V-D, Necula S-C. Artificial Intelligence as a Disruptive Technology—A Systematic Literature Review. Electronics . 2023; 12(5):1102. https://doi.org/10.3390/electronics12051102

Păvăloaia, Vasile-Daniel, and Sabina-Cristiana Necula. 2023. "Artificial Intelligence as a Disruptive Technology—A Systematic Literature Review" Electronics 12, no. 5: 1102. https://doi.org/10.3390/electronics12051102

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

This paper is in the following e-collection/theme issue:

Published on 23.7.2024 in Vol 26 (2024)

Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review

Authors of this article:

• Moustafa Laymouna 1, 2, 3 , MD, MSc   ; 
• Yuanchao Ma 2, 3, 4, 5 , MScA   ; 
• David Lessard 2, 3, 4 , PhD   ; 
• Tibor Schuster 1 , PhD   ; 
• Kim Engler 2, 3, 4 , PhD   ; 
• Bertrand Lebouché 1, 2, 3, 4 , MD, PhD  

1 Department of Family Medicine, Faculty of Medicine and Health Sciences, McGill University, Montreal, QC, Canada

2 Centre for Outcomes Research and Evaluation, Research Institute of the McGill University Health Centre, Montreal, QC, Canada

3 Infectious Diseases and Immunity in Global Health Program, Research Institute of McGill University Health Centre, Montreal, QC, Canada

4 Chronic and Viral Illness Service, Division of Infectious Disease, Department of Medicine, McGill University Health Centre, Montreal, QC, Canada

5 Department of Biomedical Engineering, Polytechnique Montréal, Montreal, QC, Canada

Corresponding Author:

Bertrand Lebouché, MD, PhD

Centre for Outcomes Research and Evaluation

Research Institute of the McGill University Health Centre

D02.4110 – Glen Site, 1001 Decarie Blvd

Montreal, QC, H4A 3J1

Phone: 1 514 843 2090

Email: [email protected]

Background: Chatbots, or conversational agents , have emerged as significant tools in health care, driven by advancements in artificial intelligence and digital technology. These programs are designed to simulate human conversations, addressing various health care needs. However, no comprehensive synthesis of health care chatbots’ roles, users, benefits, and limitations is available to inform future research and application in the field.

Objective: This review aims to describe health care chatbots’ characteristics, focusing on their diverse roles in the health care pathway, user groups, benefits, and limitations.

Methods: A rapid review of published literature from 2017 to 2023 was performed with a search strategy developed in collaboration with a health sciences librarian and implemented in the MEDLINE and Embase databases. Primary research studies reporting on chatbot roles or benefits in health care were included. Two reviewers dual-screened the search results. Extracted data on chatbot roles, users, benefits, and limitations were subjected to content analysis.

Results: The review categorized chatbot roles into 2 themes: delivery of remote health services , including patient support, care management, education, skills building, and health behavior promotion, and provision of administrative assistance to health care providers . User groups spanned across patients with chronic conditions as well as patients with cancer; individuals focused on lifestyle improvements; and various demographic groups such as women, families, and older adults. Professionals and students in health care also emerged as significant users, alongside groups seeking mental health support, behavioral change, and educational enhancement. The benefits of health care chatbots were also classified into 2 themes: improvement of health care quality and efficiency and cost-effectiveness in health care delivery . The identified limitations encompassed ethical challenges, medicolegal and safety concerns, technical difficulties, user experience issues, and societal and economic impacts.

Conclusions: Health care chatbots offer a wide spectrum of applications, potentially impacting various aspects of health care. While they are promising tools for improving health care efficiency and quality, their integration into the health care system must be approached with consideration of their limitations to ensure optimal, safe, and equitable use.

Introduction

In the dynamic landscape of IT and digital communication, chatbots—known as conversational agents —stand at the forefront, revolutionizing interactions between technology and human users. Chatbots are computer programs designed to simulate conversation through text, image, audio, or video messaging with human users on platforms such as websites, smartphone apps, or stand-alone computer software [ 1 - 47 ]. Originating from the concept ChatterBot , coined in 1994 [ 48 ], chatbots have undergone substantial evolution in their functionality and application.

The evolution of chatbots represents a significant technological leap, transitioning from reliance on predefined, rule-based scripted conversations to the sophisticated use of natural language processing and artificial intelligence (AI). By leveraging natural language processing and AI, chatbots have become capable of understanding and appropriately responding to user requests [ 49 , 50 ]. Their versatility has facilitated applications in a variety of sectors such as education, e-commerce, finance, news, health care, and entertainment. Popular instances of these applications include Amazon’s Alexa [ 51 ], Apple’s Siri [ 52 ], Google Assistant [ 53 ], Microsoft’s Cortana [ 54 ], and Samsung’s Bixby [ 55 ].

A notable advancement in the field of chatbots has been the integration of generative AI and large language models (LLMs) such as ChatGPT [ 56 - 58 ]. They have the capability to generate human-like text, enabling more natural and informative interactions [ 56 - 58 ]. However, their application in health care is still emerging. The risk of misinformation and errors is a significant concern [ 59 , 60 ], particularly in health care where accuracy is critical. The one-size-fits-all approach of LLMs may not align well with the nuanced needs of patient-centered care in the health sector [ 59 ].

The promise of chatbots in health care is considerable, offering potential for more efficient, cost-effective, and high-quality care [ 61 - 65 ], as well as their broad spectrum of uses and acceptability [ 66 , 67 ]. The use of chatbots to access and deliver health care services seems to be on the rise [ 23 , 68 - 70 ], granting them multiple potential roles in prevention, diagnosis, and support with care and treatment, with possible impacts on the whole health care system.

Despite the potential benefits, health care chatbots face unique challenges [ 71 - 74 ]. The need for highly specialized and context-sensitive advice is paramount. Generic responses from current chatbot models often overlook individual health profiles and local health contexts, which are crucial for patient care [ 75 ].

While a wide range of health care chatbot reviews have been conducted—demonstrating the versatility of chatbots in areas such as genetic cancer risk assessment [ 44 ]; oncological care [ 9 , 11 , 24 , 25 ]; sexual and reproductive health [ 35 , 45 ]; preconception, pregnancy, and postpartum health [ 36 ]; support for smoking cessation [ 38 ]; management of weight [ 39 ] and chronic conditions [ 6 , 9 , 20 , 40 ]; vaccine communication [ 26 ]; and broader health care acceptability [ 27 ]—these reviews often exhibit significant limitations in scope and depth. They tend to concentrate narrowly on specific applications such as rehabilitation for neurological conditions [ 28 ], mental health support [ 4 , 8 , 12 - 17 , 29 , 30 , 41 , 42 ], health behavior change [ 31 - 33 , 37 ], the language used in health communication by chatbots [ 43 ], and the use of chatbots in the COVID-19 public health response [ 44 ], leading to a fragmented understanding of chatbots’ roles in health care; for instance, while some reviews [ 3 , 7 ] offer insights, they do not encompass a comprehensive evaluation of the broader implications of chatbots, particularly in diverse contexts. By contrast, other reviews [ 5 , 30 ] concentrate extensively on technical aspects and AI algorithms [ 24 , 25 , 75 , 76 ]; yet, this focus tends to overshadow a detailed exploration of the impact these technologies have on health care outcomes.

This approach has left significant gaps in the literature. There is an evident need for an integrative overview that thoroughly analyzes the varied roles of chatbots across different health care applications, capturing new trends and advancements. Furthermore, the interactions and benefits of health care chatbots for diverse demographic groups, especially those who are underrepresented, are underexplored. There is also a conspicuous absence of a deeper understanding of the potential benefits and practical limitations of health care chatbots in various contexts.

Therefore, the objectives of this review are to bridge these existing knowledge gaps. Our review aims to provide a comprehensive exploration of chatbots’ functional roles, analyze the specific populations they serve, and examine in detail their potential and reported benefits, as well as the limitations of these innovative tools in health care. This endeavor will offer a more holistic and nuanced understanding of chatbots in the health care sector, addressing critical areas overlooked in previous studies.

Design and Search Strategy

This study is a rapid review, which refers to an accelerated, resource-efficient process of knowledge synthesis through streamlining or omitting specific methods associated with more traditional review processes [ 77 - 79 ]. Hence, a rapid review assesses what is already known in a given area within a relatively short period.

Our search strategy, detailed in Textbox 1 , was developed in collaboration with a health sciences librarian and performed within the MEDLINE and Embase databases on February 5, 2022. Recognizing the dynamic nature of our study field, we conducted 2 subsequent updates to our search: the first on April 22, 2022, and the second on October 30, 2023. The strategy also included searches within reference lists and websites (eg, Google Scholar) for relevant material. We exported our search records to EndNote (Clarivate).

Our search was limited to records published in English, as suggested by the Cochrane rapid reviews guide [ 80 ], from 2017 to 2023. This time frame was chosen based on preliminary searches that indicated that the largest number of relevant articles was published during this period [ 81 ]. Furthermore, it allowed us to focus on chatbots incorporating more recent technological advancements. No limitations were set based on the study population.

Our rapid review adheres to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, as depicted in Figure 1 [ 82 ].

Search terms

• user-computer interface/or (Chatbot* or chat bot* or User-Computer Interface* or (conversational adj2 (agent* or assistant*)).mp
• Limit 1 to yr = “2017 - Current”
• Limit 2 to English

Study Selection

We included primary research studies that used text- or voice-based tailored chatbots as interventions within the health care system or as a means to deliver interventions. These studies report original data on the roles and benefits of chatbots in the health care setting.

Studies not meeting the inclusion criteria were excluded, as were studies reporting any of the following: engineering or computer science data, preintervention data about future initiatives such as protocols, and studies in the preintervention or predevelopment phase. We also excluded interventions based solely on nonbehavioral actions such as gestures and facial expressions without text or voice interaction, interactions with an actual robot (as opposed to a conversational interface), and virtual reality chatbots. In addition, abstracts lacking sufficient details were excluded.

Data Extraction and Synthesis

Two reviewers (ML and YM) dual-screened 15% of the titles and abstracts and full texts to calculate the percentage agreement and interrater reliability, using Cohen κ [ 83 ]. Any discrepancies were resolved through discussion. ML conducted all remaining screenings. Data extraction was performed using Microsoft Office 365 (Excel and Word), capturing key study characteristics, including title, authors, month and year of publication, journal, study design, chatbot users, the chatbot’s medical specialty, whether the chatbot uses AI or is animated, and country of origin. In addition, we extracted information about the roles of chatbots, their benefits to health care, and their limitations. We categorized the source data into empirical and indicative data. This distinction reflects the 2-fold impact of contributions to the field: the actual findings demonstrate concrete evidence about the roles, users, benefits, and limitations of existing chatbots, while the authors’ discussion extends the conversation beyond current applications, providing perspective on the potential impacts, challenges, and future directions of health care chatbots, thus more comprehensively rounding out our assessment.

To synthesize these diverse pieces of information, relevant data underwent content analysis to generate subcategories, categories, and overarching themes [ 84 ].

While our research centers on chatbots, we have chosen to use the number of studies, rather than the chatbots themselves, as the basis for presenting most of our results. This approach accounts for the diverse adaptations to the identified chatbots across different contexts. Many of the chatbots we studied were modified to serve varied roles; cater to different user groups; and, in some cases, were given entirely different names in separate studies, as indicated in the Results section. Importantly, we noticed that a given study could contribute to multiple categories, indicating the flexible and interconnected characteristics of chatbot roles, users, benefits, or limitations. By focusing on the individual studies, we capture a more detailed and context-specific understanding of each chatbot’s functionality and versatility, which would be obscured if we merely counted each chatbot once, regardless of its various adaptations.

Database Searches

Our search yielded 3672 records (databases: n=3146, 85.68%; reference searches: n=21, 0.57%; and other websites: n=3, 0.08%). After removing 526 (14.32%) duplicates from the 3672 records, 3122 (85.02%) records remained for title and abstract screening. During this screening phase, we achieved a 97% agreement rate and a Cohen κ value of 0.85, indicating substantial agreement beyond chance. Subsequently, of the 3146 records, 327 (10.39%) full texts were reviewed [ 85 - 245 ] ( Figure 1 ), with 94% agreement and a Cohen κ value of 0.88 among the reviewers. Interrater reliability between the 2 reviewers, covering both the screening and final study inclusion as well as the data extraction process, ranged from 64% to 81%, indicating strong agreement [ 83 ]. This ensures the reliability and validity of the study selection and data extraction phases of our review.

After reviewing the 327 full texts, we ultimately included 161 (49.2%) studies that reported the roles and benefits of chatbots. All 161 studies reported on the roles of chatbots, 157 (97.5%) mentioned their benefits, and 157 (97.5%) addressed their limitations. Each study also reported on the user group or groups of focus that the chatbot was designed to assist.

Origins of the Included Studies

More than a quarter of the studies originated from the United States (46/161, 28.6%; Figure 2 ). China (15/161, 9.3%), Australia (10/161, 6.2%), Japan (9/161, 5.6%), and Spain (7/161, 4.3%) followed. Of the 161 studies, Italy, Switzerland, the United Kingdom, Singapore, Brazil, and South Korea each contributed 6 (3.7%), France and the Netherlands each contributed 4 (2.5%), while New Zealand, Greece, Russia, Norway, Malaysia, India, Senegal, Peru, Portugal, Canada, Latvia, South Africa, Indonesia, Argentina, Thailand, Saudi Arabia, Germany, and Austria each contributed 1 (0.6%) study. Notably, some studies were multinational; for instance, 1 (0.6%) of the 161 studies included Switzerland, Austria, and Germany; another included Northern Ireland, the Republic of Ireland, Scotland, Sweden, and Finland; yet another included Thailand, China, and Singapore; another study included India, North America, and the United Kingdom; a study included Finland, Denmark, and the Netherlands; another included Norway and Switzerland; and an additional study included the Netherlands and Scotland. Collectively, these 7 multinational studies account for 4.3% of the 161 included studies.

In our review of 161 studies, certain chatbots were the focus of multiple studies, particularly in the United States, Australia, South Korea, Switzerland, New Zealand, and Singapore; for instance, 2 specific chatbots were each the subject of 4 (2.5%) of the 161 studies (Gabby [ 94 , 99 , 101 , 114 ] and Woebot [ 86 , 92 , 119 , 173 ]). In addition, 11 chatbots were each studied twice (Todaki [ 90 , 102 ], GAMBOT [ 97 , 133 ], Laura [ 98 , 121 ], Vik [ 108 , 186 ], Termbot [ 151 , 195 ], ChatPal [ 158 , 168 ], a chatbot in a virtual ward [ 180 , 194 ], Corowa-kun [ 181 , 197 ], Dokbot [ 189 , 192 ], BotMaria [ 193 , 205 ], and COUCH [ 236 , 237 ]). Among these, a unique situation was observed in 5 (3.1%) of the 161 studies where the same original chatbot was presented under 5 different names [ 89 , 104 , 107 , 124 , 240 ]. These studies often shared several coauthors, indicating a common origin but with adaptations for different populations and roles. However, it is important to note that not all studies with mutual coauthors clearly indicated a shared origin of the chatbots.

Chatbot Roles

All studies stated the role or roles of the chatbot used, with at least 1 role per study. Our analysis yielded 14 subcategories of primary roles (presented in italics), grouped into 5 categories, which were organized into 2 overarching themes, as summarized in Table 1 .

Theme 1: Delivery of Remote Health Services

This theme refers to health services offered at a distance as an alternative or complement to the usual on-site modes of care delivery. It includes 3 categories and 7 subcategories of roles, with 158 (98.1%) of the 161 studies contributing to this theme.

Patient Support and Care Management

This category refers to the facilitation of medical consultations or the delivery of advice or support by providing counseling or treatment advice, triaging patients’ complaints, and fostering self-management and monitoring.

Overall, 103 (65.2%) of the 158 studies contributed to this category. Of these 103 studies, 46 (44.7%) mentioned using chatbots for mental health support , 26 (25.2%) reported providing counseling and treatment advice through chatbots, while 22 (21.4%) included chatbot use for improving self-management or monitoring for chronic conditions . Furthermore, of the 103 studies, 14 (13.6%) described chatbot use for triaging, screening, risk assessment, and referral ; 8 (7.8%) studies each reported chatbot use for self-care and monitoring for COVID-19 symptoms and rehabilitation guidance ; whereas 7 (6.8%) studies used chatbots to provide reminders .

Education and Skills Building

This category included the dissemination of educational material or medical information or skills development material (eg, exercising and using a medical device) for users, including patients, health care providers, or nursing and medical students.

In all, 41 (25.9%) of the 158 studies contributed to this category. Of these 41 studies, 23 (56%) reported promoting health literacy of the targeted population with the chatbot, 12 (29%) reported using chatbots in medical education and clinical skills for health care professionals and medical students , and psychoeducation was reported by 5 (12%) studies to enhance mental well-being.

Health Behavior Promotion

This category included the promotion of healthy lifestyles such as physical activity, a healthy diet, or stress management. Of the 158 studies, 39 (24.7%) contributed to this category. Of these 39 studies, healthy lifestyle behavior was encouraged through the chatbot in 30 (77%), while 6 (15%) reported self-monitoring for health behavior change as a chatbot role.

Theme 2: Provision of Administrative Assistance to Health Care Providers

This theme refers to all types of administrative work carried out by the chatbots, grouped within 2 categories—health-related administrative tasks and research purposes—with 9 (5.6%) of the 161 studies contributing to this theme.

Health-Related Administrative Tasks

This category included the completion of health care providers’ routine administrative work, such as data collection (eg, medical history taking), data entry, or transferring data to patients’ medical records. Of the 9 studies, 6 (67%) reported using the chatbot for data collection and storage in patient electronic medical records and charts, as well as for patient-reported outcome data, which could be captured by chatbots to replace collection by health care providers.

Research Purposes

This category refers to chatbot use for the completion of research-related work such as participant recruitment, the consent process, or data collection through surveys. Of the 9 studies, 3 (33%) contributed to this category, reporting the use of chatbots for participants’ recruitment and data collection through a self-administered questionnaire, in addition to obtaining electronic consent from individuals to participate in the study.

Chatbot Users

All 161 studies specified the intended chatbot user population. The content analysis yielded 21 subcategories of chatbot users (presented in italics), grouped into 8 broader categories of users, as summarized in Table 2 .

Lifestyle and General Well-Being Enthusiasts

This category, with 69 (42.9%) of the 161 studies, addressed individuals aiming to improve or maintain their health and well-being. Of these 69 studies, 44 (64%) focused on healthy adults (adults who are in good health, without any significant or chronic medical conditions). General public (16/69, 23%) targeted the broader and more inclusive population that encompasses all segments of the population, regardless of their health status. Lifestyle-improvement seekers , encompassing 9 (13%) of the 69 studies, included individuals motivated to change their lifestyle.

Health Condition–Focused Groups

This category, comprising 46 (28.6%) of the 161 studies, included patients with specific health conditions across 4 subcategories. Of these 46 studies, individuals seeking mental health support, the largest subcategory with 23 (50%) studies, referred to adults with conditions such as attention-deficit and panic symptoms. Patients with chronic conditions (10/46, 22%) focused on individuals with conditions such as irritable bowel syndrome and hypertension. Patients with cancer (7/46, 15%) targeted those with breast cancer and those at risk for hereditary cancer. Recovering patients (6/46, 13%) focused on patients in various stages of recovery.

Demographic and Family-Centric Groups

Addressing specific demographic groups and family dynamics, this category comprised 15.5% (25/161) of the included studies. Women (14/25, 56%) focused on women’s health issues. Parents and children (7/25, 28%) centered on the health issues of children and adolescents. Families (4/25, 16%) looked at family dynamics and health.

Unlike age-based groups that are defined solely by the age of individuals, demographic and family-centric groups consider a wider range of factors, including gender, family roles, and the interplay of relationships within a family unit.

Age-Based User Groups

With 23 (14.3%) of the 161 studies, this category targeted specific age groups or life stages. Older adults (11/23, 48%) focused on older adults and age-related health concerns. Young seekers of mental health support (8/23, 35%) focused on mental health support for young adults. Children (4/23, 17%) targeted health issues specific to children.

Underserved Populations

With 22 (13.7%) of the 161 studies, this category focused on inclusive and accessible health care. Culturally diverse groups (14/22, 64%) targeted ethnic and cultural groups. Individuals with disabilities (8/22, 36%) focused on the unique health care needs of people with disabilities.

Health Care Professionals and Students

Encompassing 15 (9.3%) of the 161 studies, this category targeted health care professionals and students. Medical and nursing students (8/15, 53%) covered educational aspects for students in medical and nursing fields. Health care professionals (7/15, 47%) focused on training and professional development with this group.

Health-Related–Behavior-Change Seekers

Comprising 15 (9.3%) of the 161 studies, this category focused on behavioral health and lifestyle changes. Behavioral change seekers (8/15, 53%) included studies on individuals seeking to change health-related behaviors. Individuals in addiction recovery (7/15, 47%) targeted those dealing with addictions.

Educational and Skills Enhancement Seekers

Comprising 15 (9.3%) of the 161 studies, this category involved the use of chatbots for educational purposes. Nonmedical professionals (8/15, 53%) focused on skills enhancement for various professionals. Health care training users (7/15, 47%) were concerned about chatbots being used to train health care professionals.

While the health care professionals subcategory within the health care professionals and students category focuses on the professional development and training of individuals in the health care field, the educational and skills enhancement seekers category addresses a broader spectrum of users, including nonmedical professionals, and emphasizes the role of chatbots as a tool for educational purposes across various sectors.

Health Care Chatbot Benefits

Most of the studies (157/161, 97.5%) described the benefits of using chatbots in health care. The content analysis yielded 7 different subcategories of benefits (presented in italics), grouped into 5 categories, which were organized into 2 broad themes, as summarized in Table 3 .

Theme 1: Improvement of Health Care Quality

This theme refers to the processes of enhancing the standards, personalization, and accessibility of health care services delivered to the targeted chatbot users. It included 6 subcategories grouped into 2 categories of benefits, with 121 (77.1%) of the 157 studies contributing to the overarching theme.

Improvement in Health Outcomes and Patient Management

Of the 121 studies in this category, 65 (53.7%) addressed the benefits of chatbots to improve health outcomes and patient management. Of these 65 studies, 42 (65%) reported on improved mental health and well-being , 15 (23%) reported on enhanced self-management , and 8 (12.3%) reported on improved physical health as outcomes of using chatbots.

Personalization Through Patient-Centered and Equitable Care

Of the 121 studies, 62 (51.2%) reported on promoting personalization through patient-centered and equitable care. Chatbot personalization refers to customizing its interactions, content, and functionalities to suit individual needs and preferences, ensuring that it provides relevant, user-specific advice and support, enhancing its effectiveness and user experience. Health equity refers to minimizing disparities and inequality based on the social determinants of health, including differences between groups in terms of socioeconomic factors, gender, and ethnicity [ 246 ]. Patient-centered care addresses patients’ specific health care needs and concerns, improving the quality of personal, professional, and organizational relationships and aiding patients to actively participate in their own care [ 247 , 248 ].

Of the 62 studies, 60 (97%) discussed chatbot use benefits in terms of increased accessibility and reach of health care by helping engage diverse populations to access health services for minor health concerns that do not require emergency visits, with convenience and 24/7 availability.

Moreover, 16 (26%) of the 62 studies discussed using a chatbot to achieve engaged and satisfied users . In these studies, user acceptance was assessed by measuring the users’ positive feedback and their willingness to use the chatbot. This was often gauged through surveys or user feedback sessions after the interaction. The studies also highlighted that friendly interactions facilitated by the chatbot could enhance self-disclosure, further contributing to user satisfaction and engagement.

Of the 62 studies, 4 (6%) described chatbot use benefits for supported groups considered vulnerable and reduced biases in health care delivery , particularly for groups considered marginalized (eg, Black women and older users) facing stigma in health care settings and for people with low technological literacy.

Theme 2: Efficiency and Cost-Effectiveness in Health Care Delivery

This theme refers to chatbot use as favoring efficient care for targeted users. Providing efficient care means producing desired results with minimal or no waste of time, costs, materials, or personnel [ 249 ]. Three categories of benefits contributed to this overarching theme.

Optimization of Resources

In all, 75 (47.8%) of the 157 studies indicated reduced administrative or financial burdens for the health care system through chatbots because they can help relieve the burden of managing chronic health conditions, staffing shortage, and overwhelmed primary care settings. These studies indicated that chatbots could provide saved time and cost of health interventions , especially compared to other routine interventions.

Scalability of Health Interventions

Of the 157 studies, 4 (2.5%) indicated the feasibility of using chatbots for the implementation of large-scale health interventions to capture and assess large-scale public health situations, providing evidence for researchers and policy makers. The studies also addressed the significance of user data collected during the COVID-19 pandemic to evaluate the public health situation and aid decision-making by policy makers, public health authorities, and researchers.

Data Quality and Research Support

Of the 157 studies, 4 (2.5%) pointed out the benefits of enhancing data collection and clinical research quality by chatbots, providing timely, consistent, and standardized data collection, reducing human error, increasing patient engagement, and assisting in recruiting a diverse participant pool.

Health Care Chatbot Limitations

Most of the studies (157/161, 97.5%) identified specific limitations of chatbots in health care, presented as 12 subcategories grouped into 5 categories, as summarized in Table 4 .

Challenges in User Experience and Overreliance

A total of 157 (97.5%) of the 161 studies contributed to this category, addressing the tendency of overconfidence and overreliance among users who overestimate the capabilities of chatbots or rely excessively on them for health care needs, as noted in 154 (98.1%) studies. Overconfidence in chatbots can lead to users substituting professional medical advice with chatbot suggestions, while overreliance might result in users neglecting other essential aspects of health care or disregarding the need for human health care professional intervention. This subcategory highlights the importance of maintaining a balanced perspective on the capabilities and limitations of chatbots in health care contexts.

In addition, this category encompasses the usability and accessibility issues related to the ease with which users can interact with chatbots and the extent to which these chatbots are accessible to a diverse range of users, as referred to in most of the studies (135/157, 86%). It includes considerations of user interface design, the intuitiveness of chatbot interactions, the chatbots’ adaptability to different user needs, and their accessibility to individuals with varying levels of technology savviness or disabilities. Challenges in this category can lead to user dissatisfaction, reduced effectiveness of the chatbot, and potentially lower engagement with the health care service it provides.

Technical Challenges

This category refers to the broad spectrum of technological difficulties encountered in the design, development, and implementation of these systems, with 32 (20.1%) of the 157 studies contributing to it. This category underscores the need for sophisticated technology that can handle the nuances of health care communication and patient interaction while being accessible and practical for real-world application.

It includes the complexity of effective language and communication processing , as noted in 24 (75%) of the 32 studies, to ensure accurate and relevant medical information, as well as the chatbot’s ability to understand and respond to a range of user inputs, including those related to emotional states and complex health care queries.

The limitations extend to challenges in empathy and personal connection , which refer to the difficulties chatbots face in simulating human conversations and establishing rapport with users. This is a critical aspect in health care settings where patient trust and comfort are paramount, as highlighted in 17 (53%) of the 32 studies.

In addition, this category involves considering the challenges with resource allocation and cost efficiency of developing and maintaining these systems to ensure that they are not only technologically advanced but also financially viable and sustainable, as indicated in 2 (6%) of the 32 studies.

Medicolegal and Safety Concerns

With 6 (3.8%) of the 157 contributing studies, this category includes regulatory and legal issues encompassing the implications of chatbot advice and overall patient safety, as highlighted in 3 (50%) studies. These issues include chatbots’ compliance with health care regulations and patient privacy laws, liability for misdiagnosis or inadequate advice, and the need for specific regulatory guidelines for their development and application.

Furthermore, challenges extend to concerns about content and information quality , such as the medical accuracy of information provided by chatbots (eg, the potential for misdiagnosis) and the reliability of medical content. It also concerns limitations tied to the chatbot’s challenges in emergency response and expertise capabilities. Each of these subcategories was noted in 2 (33%) of the 6 studies.

Societal and Economic Challenges

This category refers to the wider implications of health care chatbots on the broader societal context and the economy, with 5 (3.2%) of the 157 contributing studies. It covers the influence of social, political, and economic factors on the adoption and effectiveness of chatbots in different communities.

It includes social, economic, and political challenges and considerations, as noted in all 5 studies. This subcategory scrutinizes the challenges arising from the integration of chatbots into the health care system, such as potential shifts in social norms, and the influence on economic policies and political decision-making in health care.

This category also includes issues of inequality in accessibility , as highlighted in 4 (80%) of the 5 studies. This subcategory delves into the challenges related to unequal access to chatbot technology. It focuses on how chatbots might inadvertently exacerbate existing disparities in health care, particularly for groups considered underprivileged, thereby highlighting the need for equitable distribution and accessibility of these technologies.

Ethical Challenges

This category deals with the ethical implications of using chatbots in health care, with 3 (1.9%) of the 157 studies contributing to it. It includes patient privacy and confidentiality concerns related to the use of patient data. This category also includes ethical and safety concerns encompassing the need to maintain transparency with users about the chatbot being a nonhuman agent and ensuring ethical standards in patient interactions. Each of these 2 subcategories was discussed in 2 (67%) of the 3 studies.

Principal Findings

This rapid review revealed that chatbot roles in health care are diverse, ranging from patient support to administrative tasks, and they show great promise in improving health care accessibility, especially for groups considered marginalized. It also highlighted critical gaps in the literature, which are addressed in the following subsections.

Global Trends in Chatbot Research Indicate Its Predominance in Higher-Income Countries and Opportunities in Lower-Income Regions

With 35 countries represented by the studies in this review, the topic is clearly of global interest. However, more than a quarter of the included studies (46/161, 28.6%) originated from the United States, with the remainder conducted in high- or upper–middle-income countries across North America, Europe, and parts of Asia [ 250 ]. The concentration of chatbot research in high-income countries reflects underlying disparities with low- or lower–middle-income countries, particularly in parts of Africa, South America, and certain regions in Asia, in terms of technology access and health care investment. This gap highlights the need for more research focused on these regions, considering their unique digital infrastructure and resource challenges, to democratize health technology and address chronic conditions and health literacy [ 20 , 251 - 254 ].

Chatbots Have Varied Roles in the Enhancement of Health Care Delivery and User-Centric Services

Our review underscores the transformative roles of chatbots in health care, particularly in delivering remote health services and enhancing patient support, care management, and mental health support. Consistent with previous literature [ 254 - 257 ], our findings affirm chatbots’ potential to improve health care accessibility and patient management. The findings’ emphasis on education and skills building, particularly to enhance health literacy (which aligned with past literature [ 255 , 258 ]) and to support behavioral change (also highlighted by past research [ 255 ]), aligns with the growing need for patient empowerment in health care. The administrative efficiency of chatbots, noted in our review, resonates with previous findings [ 23 , 35 , 255 , 258 ] on the importance of resource optimization in health care settings.

Our findings indicate that chatbots also play a key role in facilitating clinical research, consistent with past work [ 259 ], a potential that needs further exploration, especially considering AI’s evolving role in health care [ 72 , 259 - 262 ].

The Diverse User Base of Chatbots Shows Their Potential to Support Equity and Bridge the Access Gap in Health Care Services

Our analysis indicates a broad and diverse user base for health care chatbots. From individuals focused on general well-being to those with specific health conditions, chatbots have been designed to cater to a wide array of needs. Notably, their use by demographic and family-centric groups and their accessibility to underserved populations underline the inclusive capacity of chatbots and their role in enhancing health care access and equity, especially for groups considered marginalized, in line with existing research [ 12 , 67 , 255 , 263 - 265 ].

In addition, our findings show the significant use of chatbots in mental health support for various age groups, reflecting the pressing need for accessible mental health services highlighted by others [ 4 , 8 , 12 - 17 , 29 , 30 ].

Furthermore, chatbots have emerged as tools for reducing stigma [ 12 , 265 ], linking users to health services [ 266 - 268 ], and protecting sensitive information [ 269 ]. Their empathetic and multilingual capabilities, as seen in our results [ 107 , 111 , 112 , 120 , 122 , 126 - 128 , 132 ] and past literature [ 270 - 276 ], are vital to reach diverse populations. They are particularly critical in light of the digital divide and the need for inclusive and accessible health care solutions [ 254 , 258 , 263 , 277 , 278 ].

The Use of AI in Chatbots Is a Promising but Still Evolving Field

The studies included in our review show a substantial number of AI-based chatbots, with fewer relying on non-AI platforms. AI in health care is recognized for its potential to improve health outcomes and the quality of life globally [ 260 ]. Given advances in machine learning and AI, expanding the scope of chatbots is expected to cause a mutation in their role in the health care system to assist clinicians and potentially take over some of their duties [ 72 , 261 , 262 ]. The synergy between big data and AI, coupled with the increasing availability of data in health care, suggests that AI-based chatbots could effectively use extensive health care data [ 259 , 279 ]. This aligns with 1 (0.6%) of the 161 included studies [ 94 ], which discusses the use of collected data as a key benefit of chatbots. However, ethical considerations such as data privacy and algorithmic biases must be addressed for responsible AI deployment, crucial for maintaining trust and fairness [ 73 ].

Studies included in this review indicate that using avatars in these chatbots to simulate social behaviors can enhance user engagement and trust. This form of chatbot technology is particularly appealing in patient interactions and medical education to establish trust and therapeutic alliances between health care professionals and patients and to improve the communication skills of medical students and health care professionals [ 118 , 123 , 130 , 131 , 280 ].

Balancing AI’s benefits to enhance data use and user interactions with its ethical concerns, including data privacy and algorithmic bias, is crucial for its implementation, shaping the future of patient care and medical education in an innovative and ethically sound way.

Despite the Potential Revolutionary Roles of Chatbots in Health Care, Critical Challenges and Limitations Exist

This review stresses that despite chatbots’ roles and benefits, their use comes with various challenges, including ethical, technical, medicolegal, and user experience concerns, as also discussed in past literature [ 3 - 5 , 23 , 25 , 30 , 72 , 74 , 95 , 281 , 282 ].

While the studies included in our review have highlighted chatbot use to address minor health concerns and provide off-hour information, there is a noticeable gap in evaluating their technical limitations, especially in complex health care scenarios, as underscored by past literature [ 3 - 5 , 23 , 25 , 30 , 72 , 74 , 95 , 281 , 282 ]. This raises concerns about patient safety and the accuracy of health management, emphasizing the need for comprehensive assessment and iterative improvement of chatbot technologies [ 22 , 25 , 68 , 72 , 95 , 254 , 283 ].

The findings in our review indicate the regulatory and ethical landscape for chatbots as another area of concern. This agrees with past studies highlighting the need for ethical use, data privacy, and transparent communication about chatbots’ capabilities and limitations [ 4 , 73 , 74 , 254 , 281 , 284 , 285 ]. The absence of specific laws and regulations addressing health care chatbot use introduces risks around liability and medicolegal issues [ 72 , 286 , 287 ]. These challenges are further complicated by ethical dilemmas, such as privacy and confidentiality in nonanonymous interactions [ 71 , 72 , 288 , 289 ] and safety concerns in medical emergencies due to limited chatbot expertise [ 72 ].

Technical issues identified by this review, including difficulty in language processing and a lack of empathic response, can lead to trust issues and increased clinical workload and align with past literature [ 3 - 5 , 68 , 72 , 73 , 280 , 290 ]. Overreliance on chatbots for self-diagnosis and health care decisions may lead to misjudgments, potentially exacerbating health issues [ 4 , 68 , 73 ]. In addition, the financial motives of private companies in the health sector raise ethical concerns about the primary purpose and application of health chatbots [ 73 ]. The requirement for sophisticated AI technology also implies increased demands on human resource expertise and storage services, potentially escalating costs [ 73 , 287 ].

Our results indicate that chatbots serve a wide range of populations from various groups in terms of age, gender, ethnicity, and socioeconomic and educational status due to their promising acceptability and usability [ 291 ]. However, the digital divide [ 292 - 294 ], algorithmic ethical concerns [ 295 ], and the potential misuse of chatbots in replacing established health services [ 296 ] present risks. These factors, along with social, economic, and political influences [ 297 ], could inadvertently widen health disparities, highlighting the importance of inclusive and equitable chatbot development and deployment.

The discussion on health care chatbots is fundamentally about their potential and promise, grounded in our exploration of current studies and developments. These digital tools could significantly enhance health care access, service quality, and efficiency. However, realizing their full potential hinges on addressing challenges such as ethical AI use, data privacy, and integration with health care systems.

Efforts moving forward should concentrate on incorporating AI responsibly and designing chatbots that cater to all user demographics, ensuring equitable health care access. Collaboration across technology, health care, and policy sectors is crucial to establish ethical guidelines and confirm chatbots’ efficacy and safety. Successfully navigating these challenges will enable chatbots to fulfill their promising role in health care, contributing to a more accessible and patient-focused system.

Limitations

This review, while insightful, is not without its limitations. Although rapid and systematic reviews are often considered comparable in their conclusions, each methodology has its own set of constraints [ 289 , 298 ]. Specifically, this rapid review was limited by a noncomprehensive search strategy that included only 2 databases. In addition, the inclusion criteria were restricted by date and language, which potentially led to the exclusion of some pertinent studies. Another limitation was the concentration of screening and analysis tasks on a single reviewer (ML), which might have introduced bias or overlooked nuances in the data. Moreover, a formal quality appraisal of the included studies was not conducted due to the descriptive nature of this review. Consequently, this limitation may affect the depth of understanding and the strength of the conclusions drawn.

One critical aspect of our methodology was the combination of empirical findings and opinion-based data from the discussions in the included studies. We did not distinguish between these 2 types of data but rather treated them as a unified source of information. This approach, while allowing for a comprehensive overview of chatbots in health care, might have led to a potential bias in favor of chatbot benefits because both empirical results and positive speculative insights were reported together. However, this potential bias is somewhat mitigated by our consistent reporting of the challenges associated with chatbots, as identified in the included studies. By presenting both the potential benefits and the challenges, we aimed to offer a balanced view, reducing the likelihood of a 1-sided interpretation favoring chatbot benefits.

In addition, this review might have overestimated the results due to the dependence on the discussion sections of each study, which may have overcounted the results and miscounted those that may have disagreed or contradicted the results of these included studies. However, this did not significantly impact the study’s aim to provide an exploratory and descriptive overview of health care chatbots, mapping out the landscape of their applications in health care. In such a context, a broad, inclusive approach that captures diverse opinions and trends is more important than precise quantification.

Moreover, one of the potential limitations of this review is the exclusion of generative AI and LLMs such as ChatGPT. However, among the studies we reviewed, a standout comparison involved a health care chatbot, specialized in medical terminology, and ChatGPT. This unique comparison serves to highlight the advanced capabilities of LLMs such as ChatGPT in enhancing the delivery and accuracy of remote health services [ 59 , 75 ]. Nonetheless, a significant challenge persists in guaranteeing the contextual relevance and appropriateness of chatbot responses, particularly in intricate medical scenarios [ 59 , 60 ]. In addition, the personalization of health care interactions and the precision of information provided by these AI-driven systems are critical areas necessitating extensive future research and rigorous evaluation of their outputs [ 59 , 60 , 299 ].

Finally, the results were presented solely as a narrative summary [ 77 ], which might limit the breadth of perspectives and interpretations that a more diverse methodological approach could have provided. Nevertheless, the inclusion of both benefits and challenges in our reporting suggests that the review may not be significantly biased toward a positive portrayal of chatbots, providing a more nuanced understanding of their role in health care.

Conclusions

This review underscores the significant potential of chatbots in health care, evident in their diverse roles, benefits, and user populations. In addition, it explores the current limitations and challenges of chatbot development and implementation in health care. Finally, it underscores significant research gaps in the field. As such, this review aims to contribute to academic discourse on this important topic and offer insights into the effective design, implementation, and investigation of chatbots in health care.

Acknowledgments

BL is supported by 2 career awards—a Senior Salary Award from Fonds de recherche du Québec–Santé (311200) and the Lettre d’Entente (LE) 250 from Québec’s Ministry of Health for researchers in family medicine—and holds a Canadian Institutes for Health Research Strategy for Patient-Oriented Research Mentorship Chair in Innovative Clinical Trials. ML is supported by a Graduate Excellence Fellowship Award from the Department of Family Medicine at McGill University and a McGill Centre for Viral Diseases studentship. YM was supported by a doctoral scholarship from the Fonds de recherche du Québec–Nature et Technologies given in partnership with the Strategy for Patient-Oriented Research Support Unit of Québec. YM is supported by a postgraduate scholarship–doctoral program given through the Natural Sciences and Engineering Research Council of Canada.

Authors' Contributions

ML and YM dual-screened 15% of the titles and abstracts and the full texts. ML performed all other screenings. ML analyzed and interpreted the results and wrote the manuscript. DL reviewed the results. All authors reviewed and edited the manuscript and approved the final version.

Conflicts of Interest

None declared.

• Kocaballi AB, Berkovsky S, Quiroz JC, Laranjo L, Tong HL, Rezazadegan D, et al. The personalization of conversational agents in health care: systematic review. J Med Internet Res. Nov 07, 2019;21(11):e15360. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Luo TC, Aguilera A, Lyles CR, Figueroa CA. Promoting physical activity through conversational agents: mixed methods systematic review. J Med Internet Res. Sep 14, 2021;23(9):e25486. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Laranjo L, Dunn AG, Tong HL, Kocaballi AB, Chen J, Bashir R, et al. Conversational agents in healthcare: a systematic review. J Am Med Inform Assoc. Sep 01, 2018;25(9):1248-1258. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vaidyam AN, Wisniewski H, Halamka JD, Kashavan MS, Torous JB. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. Can J Psychiatry. Jul 21, 2019;64(7):456-464. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Milne-Ives M, de Cock C, Lim E, Shehadeh MH, de Pennington N, Mole G, et al. The effectiveness of artificial intelligence conversational agents in health care: systematic review. J Med Internet Res. Oct 22, 2020;22(10):e20346. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bin Sawad A, Narayan B, Alnefaie A, Maqbool A, Mckie I, Smith J, et al. A systematic review on healthcare artificial intelligent conversational agents for chronic conditions. Sensors (Basel). Mar 29, 2022;22(7):2625. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tudor Car L, Dhinagaran DA, Kyaw BM, Kowatsch T, Joty S, Theng YL, et al. Conversational agents in health care: scoping review and conceptual analysis. J Med Internet Res. Aug 07, 2020;22(8):e17158. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vaidyam AN, Linggonegoro D, Torous J. Changes to the psychiatric chatbot landscape: a systematic review of conversational agents in serious mental illness: changements du paysage psychiatrique des chatbots: une revue systématique des agents conversationnels dans la maladie mentale sérieuse. Can J Psychiatry. Apr 16, 2021;66(4):339-348. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Geoghegan L, Scarborough A, Wormald JC, Harrison CJ, Collins D, Gardiner M, et al. Automated conversational agents for post-intervention follow-up: a systematic review. BJS Open. Jul 06, 2021;5(4):zrab070. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Allouch M, Azaria A, Azoulay R. Conversational agents: goals, technologies, vision and challenges. Sensors (Basel). Dec 17, 2021;21(24):8448. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bibault JE, Chaix B, Nectoux P, Pienkowsky A, Guillemasse A, Brouard B. Healthcare ex Machina: are conversational agents ready for prime time in oncology? Clin Transl Radiat Oncol. May 2019;16:55-59. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Abd-Alrazaq AA, Alajlani M, Alalwan AA, Bewick BM, Gardner P, Househ M. An overview of the features of chatbots in mental health: a scoping review. Int J Med Inform. Dec 2019;132:103978. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pacheco-Lorenzo MR, Valladares-Rodríguez SM, Anido-Rifón LE, Fernández-Iglesias MJ. Smart conversational agents for the detection of neuropsychiatric disorders: a systematic review. J Biomed Inform. Jan 2021;113:103632. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Provoost S, Lau HM, Ruwaard J, Riper H. Embodied conversational agents in clinical psychology: a scoping review. J Med Internet Res. May 09, 2017;19(5):e151. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rampioni M, Stara V, Felici E, Rossi L, Paolini S. Embodied conversational agents for patients with dementia: thematic literature analysis. JMIR Mhealth Uhealth. Jul 16, 2021;9(7):e25381. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gaffney H, Mansell W, Tai S. Conversational agents in the treatment of mental health problems: mixed-method systematic review. JMIR Ment Health. Oct 18, 2019;6(10):e14166. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bérubé C, Schachner T, Keller R, Fleisch E, V Wangenheim F, Barata F, et al. Voice-based conversational agents for the prevention and management of chronic and mental health conditions: systematic literature review. J Med Internet Res. Mar 29, 2021;23(3):e25933. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chew HS. The use of artificial intelligence-based conversational agents (Chatbots) for weight loss: scoping review and practical recommendations. JMIR Med Inform. Apr 13, 2022;10(4):e32578. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kramer LL, Ter Stal S, Mulder BC, de Vet E, van Velsen L. Developing embodied conversational agents for coaching people in a healthy lifestyle: scoping review. J Med Internet Res. Feb 06, 2020;22(2):e14058. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Schachner T, Keller R, V Wangenheim F. Artificial intelligence-based conversational agents for chronic conditions: systematic literature review. J Med Internet Res. Sep 14, 2020;22(9):e20701. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Reger GM, Norr AM, Gramlich MA, Buchman JM. Virtual standardized patients for mental health education. Curr Psychiatry Rep. Jul 15, 2021;23(9):57. [ CrossRef ] [ Medline ]
• Safi Z, Abd-Alrazaq A, Khalifa M, Househ M. Technical aspects of developing chatbots for medical applications: scoping review. J Med Internet Res. Dec 18, 2020;22(12):e19127. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Abd-Alrazaq A, Safi Z, Alajlani M, Warren J, Househ M, Denecke K. Technical metrics used to evaluate health care chatbots: scoping review. J Med Internet Res. Jun 05, 2020;22(6):e18301. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wang A, Qian Z, Briggs L, Cole AP, Reis LO, Trinh QD. The use of chatbots in oncological care: a narrative review. Int J Gen Med. May 1, 2023;16:1591-1602. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Xu L, Sanders L, Li K, Chow JC. Chatbot for health care and oncology applications using artificial intelligence and machine learning: systematic review. JMIR Cancer. Nov 29, 2021;7(4):e27850. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Passanante A, Pertwee E, Lin L, Lee KY, Wu JT, Larson HJ. Conversational AI and vaccine communication: systematic review of the evidence. J Med Internet Res. Oct 03, 2023;25:e42758. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wutz M, Hermes M, Winter V, Köberlein-Neu J. Factors influencing the acceptability, acceptance, and adoption of conversational agents in health care: integrative review. J Med Internet Res. Sep 26, 2023;25:e46548. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hocking J, Oster C, Maeder A, Lange B. Design, development, and use of conversational agents in rehabilitation for adults with brain-related neurological conditions: a scoping review. JBI Evid Synth. Feb 01, 2023;21(2):326-372. [ CrossRef ] [ Medline ]
• He Y, Yang L, Qian C, Li T, Su Z, Zhang Q, et al. Conversational agent interventions for mental health problems: systematic review and meta-analysis of randomized controlled trials. J Med Internet Res. Apr 28, 2023;25:e43862. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Abd-Alrazaq AA, Rababeh A, Alajlani M, Bewick BM, Househ M. Effectiveness and safety of using chatbots to improve mental health: systematic review and meta-analysis. J Med Internet Res. Jul 13, 2020;22(7):e16021. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lin X, Martinengo L, Jabir AI, Ho AH, Car J, Atun R, et al. Scope, characteristics, behavior change techniques, and quality of conversational agents for mental health and well-being: systematic assessment of apps. J Med Internet Res. Jul 18, 2023;25:e45984. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Aggarwal A, Tam CC, Wu D, Li X, Qiao S. Artificial intelligence-based chatbots for promoting health behavioral changes: systematic review. J Med Internet Res. Feb 24, 2023;25:e40789. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Martinengo L, Jabir AI, Goh WW, Lo NY, Ho MH, Kowatsch T, et al. Conversational agents in health care: scoping review of their behavior change techniques and underpinning theory. J Med Internet Res. Oct 03, 2022;24(10):e39243. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Webster EM, Ahsan MD, Perez L, Levi SR, Thomas C, Christos P, et al. Chatbot artificial intelligence for genetic cancer risk assessment and counseling: a systematic review and meta-analysis. JCO Clin Cancer Inform. Sep 2023;7:e2300123. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mills R, Mangone ER, Lesh N, Mohan D, Baraitser P. Chatbots to improve sexual and reproductive health: realist synthesis. J Med Internet Res. Aug 09, 2023;25:e46761. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chua JY, Choolani M, Chee CY, Chan YH, Lalor JG, Chong YS, et al. Insights of parents and parents-to-be in using chatbots to improve their preconception, pregnancy, and postpartum health: a mixed studies review. J Midwifery Womens Health. Feb 03, 2023;68(4):480-489. [ CrossRef ] [ Medline ]
• Singh B, Olds T, Brinsley J, Dumuid D, Virgara R, Matricciani L, et al. Systematic review and meta-analysis of the effectiveness of chatbots on lifestyle behaviours. NPJ Digit Med. Jun 23, 2023;6(1):118. [ FREE Full text ] [ CrossRef ] [ Medline ]
• He L, Balaji D, Wiers RW, Antheunis ML, Krahmer E. Effectiveness and acceptability of conversational agents for smoking cessation: a systematic review and meta-analysis. Nicotine Tob Res. Jun 09, 2023;25(7):1241-1250. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Noh E, Won J, Jo S, Hahm DH, Lee H. Conversational agents for body weight management: systematic review. J Med Internet Res. May 26, 2023;25:e42238. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Thoumrungroje P, Chainarong A, Namwaing P, Pitiruangsit L, Sittichanbuncha Y, Ngamjarus C, et al. Chatbot intervention in asthma and obstructive sleep apnea: a systematic review. J Med Assoc Thai. 2023;106:S134. [ FREE Full text ] [ CrossRef ]
• Lim SM, Shiau CW, Cheng LJ, Lau Y. Chatbot-delivered psychotherapy for adults with depressive and anxiety symptoms: a systematic review and meta-regression. Behav Ther. Mar 2022;53(2):334-347. [ CrossRef ] [ Medline ]
• Ogilvie L, Prescott J, Carson J. The use of chatbots as supportive agents for people seeking help with substance use disorder: a systematic review. Eur Addict Res. Aug 30, 2022;28(6):405-418. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Shan Y, Ji M, Xie W, Qian X, Li R, Zhang X, et al. Language use in conversational agent-based health communication: systematic review. J Med Internet Res. Jul 08, 2022;24(7):e37403. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Amiri P, Karahanna E. Chatbot use cases in the COVID-19 public health response. J Am Med Inform Assoc. Apr 13, 2022;29(5):1000-1010. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Balaji D, He L, Giani S, Bosse T, Wiers R, de Bruijn GJ. Effectiveness and acceptability of conversational agents for sexual health promotion: a systematic review and meta-analysis. Sex Health. Oct 2022;19(5):391-405. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Christopherjames JE, Saravanan M, Thiyam DB, Surendhar S PA, Sahib MY, Ganapathi MV, et al. Natural language processing based human assistive health conversational agent for multi-users. In: Proceedings of the 2021 2nd International Conference on Electronics and Sustainable Communication Systems. 2021. Presented at: ICESC '21; August 4-6, 2021:1414-1420; Coimbatore, India. URL: https://ieeexplore.ieee.org/document/9532913 [ CrossRef ]
• Imtiaz MT, Kennington C. Incremental unit networks for distributed, symbolic multimodal processing and representation. In: Proceedings of the 13th International Conference, DHM 2022, Held as Part of the 24th HCI International Conference on Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management for Health, Operations Management, and Design. 2022. Presented at: HCII '22; June 26-July 1, 2022:344-363; Virtual Event. URL: https://link.springer.com/chapter/10.1007/978-3-031-06018-2_24 [ CrossRef ]
• Mauldin ML. Chatterbots, Tinymuds, and the Turing test: entering the Loebner prize competition. In: Proceedings of the 25th AAAI National Conference on Artificial Intelligence. 1994. Presented at: AAAI '94; August 1-4, 1994:16-21; Seattle, WA. URL: https://dl.acm.org/doi/abs/10.5555/2891730.2891733
• Nuruzzaman M, Hussain OK. A survey on chatbot implementation in customer service industry through deep neural networks. In: Proceedings of the 2018 IEEE 15th International Conference on e-Business Engineering. 2018. Presented at: ICEBE '18; October 12-14, 2018:54-61; Xi'an, China. URL: https://ieeexplore.ieee.org/document/8592630 [ CrossRef ]
• Kumar VM, Keerthana A, Madhumitha M, Valliammai S, Vinithasri V. Sanative chatbot for health seekers. Int J Eng Comput Sci. Mar 29, 2016;5(03):16022-16025. [ FREE Full text ] [ CrossRef ]
• Alexa developer portal. Amazon. 2014. URL: https://www.developer.amazon.com/en-US/alexa/ [accessed 2022-12-22]
• Siri. Apple. 2010. URL: https://www.apple.com/siri/ [accessed 2022-12-22]
• Google Assistant, your own personal Google. Google. 2016. URL: https://assistant.google.com/ [accessed 2022-12-22]
• Cortana- your personal productivity assistant. Microsoft. 2014. URL: https://www.microsoft.com/en-us/cortana/ [accessed 2022-12-22]
• Bixby: apps and services. Samsung. 2017. URL: https://www.samsung.com/in/apps/bixby/ [accessed 2022-12-22]
• Introducing ChatGPT. OpenAI. URL: https://openai.com/blog/chatgpt [accessed 2022-12-22]
• Ouyang L, Wu J, Jiang X, Almeida D, Wainwright CL, Mishkin P, et al. Training language models to follow instructions with human feedback. arXiv. Preprint posted online March 4, 2022. [ FREE Full text ]
• OpenAI. GPT-4 technical report. arXiv. Preprint posted online March 15, 2023. 2023. [ FREE Full text ]
• Thirunavukarasu AJ, Ting DS, Elangovan K, Gutierrez L, Tan TF, Ting DS. Large language models in medicine. Nat Med. Aug 17, 2023;29(8):1930-1940. [ CrossRef ] [ Medline ]
• Thirunavukarasu AJ, Hassan R, Mahmood S, Sanghera R, Barzangi K, El Mukashfi M, et al. Trialling a large language model (ChatGPT) in general practice with the applied knowledge test: observational study demonstrating opportunities and limitations in primary care. JMIR Med Educ. Apr 21, 2023;9:e46599. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Huisman L, van Duijn SM, Silva N, van Doeveren R, Michuki J, Kuria M, et al. A digital mobile health platform increasing efficiency and transparency towards universal health coverage in low- and middle-income countries. Digit Health. Apr 11, 2022;8:20552076221092213. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Osipov VS, Skryl TV. Impact of digital technologies on the efficiency of healthcare delivery. In: Marques G, Bhoi AK, de Albuquerque VH, Hareesha KS, editors. IoT in Healthcare and Ambient Assisted Living. Singapore, Singapore. Springer; 2021:243-261.
• Jones SP, Patel V, Saxena S, Radcliffe N, Ali Al-Marri S, Darzi A. How Google's 'ten things we know to be true' could guide the development of mental health mobile apps. Health Aff (Millwood). Sep 2014;33(9):1603-1611. [ CrossRef ] [ Medline ]
• Chandrashekar P. Do mental health mobile apps work: evidence and recommendations for designing high-efficacy mental health mobile apps. Mhealth. Mar 2018;4:6. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tanielian T, Jaycox LH. Invisible Wounds of War: Psychological and Cognitive Injuries, Their Consequences, and Services to Assist Recovery. Santa Monica, CA. RAND Corporation; 2008.
• Nadarzynski T, Miles O, Cowie A, Ridge D. Acceptability of artificial intelligence (AI)-led chatbot services in healthcare: a mixed-methods study. Digit Health. Aug 21, 2019;5:2055207619871808. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Abd-Alrazaq AA, Alajlani M, Ali N, Denecke K, Bewick BM, Househ M. Perceptions and opinions of patients about mental health chatbots: scoping review. J Med Internet Res. Jan 13, 2021;23(1):e17828. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kretzschmar K, Tyroll H, Pavarini G, Manzini A, Singh I, NeurOx Young People’s Advisory Group. Can your phone be your therapist? Young people's ethical perspectives on the use of fully automated conversational agents (Chatbots) in mental health support. Biomed Inform Insights. Mar 05, 2019;11:1178222619829083. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cheng Y, Jiang H. AI‐Powered mental health chatbots: examining users’ motivations, active communicative action and engagement after mass‐shooting disasters. J Conting Crisis Man. Sep 29, 2020;28(3):339-354. [ CrossRef ]
• Boucher EM, Harake NR, Ward HE, Stoeckl SE, Vargas J, Minkel J, et al. Artificially intelligent chatbots in digital mental health interventions: a review. Expert Rev Med Devices. Dec 31, 2021;18(sup1):37-49. [ CrossRef ] [ Medline ]
• Luxton DD, Anderson SL, Anderson M. Ethical issues and artificial intelligence technologies in behavioral and mental health care. In: Luxton DD, editor. Artificial Intelligence in Behavioral and Mental Health Care. Washington, DC. Academic Press; 2016:255-276.
• Denecke K, Abd-Alrazaq A, Househ M. Artificial intelligence for chatbots in mental health: opportunities and challenges. In: Househ M, Borycki E, Kushniruk A, editors. Multiple Perspectives on Artificial Intelligence in Healthcare: Opportunities and Challenges. Cham, Switzerland. Springer; 2021:115-128.
• Parviainen J, Rantala J. Chatbot breakthrough in the 2020s? An ethical reflection on the trend of automated consultations in health care. Med Health Care Philos. Mar 04, 2022;25(1):61-71. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Palanica A, Flaschner P, Thommandram A, Li M, Fossat Y. Physicians' perceptions of chatbots in health care: cross-sectional web-based survey. J Med Internet Res. Apr 05, 2019;21(4):e12887. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chang Y, Wang X, Wang J, Wu Y, Yang L, Zhu K, et al. A survey on evaluation of large language models. ACM Trans Intell Syst Technol. Mar 29, 2024;15(3):1-45. [ CrossRef ]
• Mohanta B, Das P, Patnaik S. Healthcare 5.0: a paradigm shift in digital healthcare system using artificial intelligence, IOT and 5G communication. In: Proceedings of the 2019 International Conference on Applied Machine Learning. 2019. Presented at: ICAML '19; May 25-26, 2019:191-196; Bhubaneswar, India. URL: https://ieeexplore.ieee.org/document/8989306 [ CrossRef ]
• Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, et al. A scoping review of rapid review methods. BMC Med. Sep 16, 2015;13(1):224. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hamel C, Michaud A, Thuku M, Skidmore B, Stevens A, Nussbaumer-Streit B, et al. Defining Rapid Reviews: a systematic scoping review and thematic analysis of definitions and defining characteristics of rapid reviews. J Clin Epidemiol. Jan 2021;129:74-85. [ CrossRef ] [ Medline ]
• Tricco AC, Langlois EV, Straus SE. Rapid reviews to strengthen health policy and systems: a practical guide. World Health Organization. 2017. URL: https://iris.who.int/bitstream/handle/10665/258698/9789241512763-eng.pdf?sequence=1 [accessed 2024-04-29]
• Garritty C, Gartlehner G, Nussbaumer-Streit B, King VJ, Hamel C, Kamel C, et al. Cochrane rapid reviews methods group offers evidence-informed guidance to conduct rapid reviews. J Clin Epidemiol. Feb 2021;130:13-22. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Denecke K, May R. Developing a technical-oriented taxonomy to define archetypes of conversational agents in health care: literature review and cluster analysis. J Med Internet Res. Jan 30, 2023;25:e41583. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. Mar 29, 2021;372:n71. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McHugh ML. Interrater reliability: the Kappa statistic. Biochem Med (Zagreb). 2012;22(3):276-282. [ FREE Full text ] [ Medline ]
• Elo S, Kyngäs H. The qualitative content analysis process. J Adv Nurs. Apr 18, 2008;62(1):107-115. [ CrossRef ] [ Medline ]
• Nazareth S, Hayward L, Simmons E, Snir M, Hatchell KE, Rojahn S, et al. Hereditary cancer risk using a genetic chatbot before routine care visits. Obstet Gynecol. Dec 01, 2021;138(6):860-870. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Prochaska JJ, Vogel EA, Chieng A, Baiocchi M, Maglalang DD, Pajarito S, et al. A randomized controlled trial of a therapeutic relational agent for reducing substance misuse during the COVID-19 pandemic. Drug Alcohol Depend. Oct 01, 2021;227:108986. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Söderström A, Shatte A, Fuller-Tyszkiewicz M. Can intelligent agents improve data quality in online questiosnnaires? A pilot study. Behav Res Methods. Oct 05, 2021;53(5):2238-2251. [ CrossRef ] [ Medline ]
• Anan T, Kajiki S, Oka H, Fujii T, Kawamata K, Mori K, et al. Effects of an artificial intelligence-assisted health program on workers with neck/shoulder pain/stiffness and low back pain: randomized controlled trial. JMIR Mhealth Uhealth. Sep 24, 2021;9(9):e27535. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stasinaki A, Büchter D, Shih CH, Heldt K, Güsewell S, Brogle B, et al. Effects of a novel mobile health intervention compared to a multi-component behaviour changing program on body mass index, physical capacities and stress parameters in adolescents with obesity: a randomized controlled trial. BMC Pediatr. Jul 09, 2021;21(1):308. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jang S, Kim JJ, Kim SJ, Hong J, Kim S, Kim E. Mobile app-based chatbot to deliver cognitive behavioral therapy and psychoeducation for adults with attention deficit: a development and feasibility/usability study. Int J Med Inform. Jun 2021;150:104440. [ CrossRef ] [ Medline ]
• Hunt M, Miguez S, Dukas B, Onwude O, White S. Efficacy of Zemedy, a mobile digital therapeutic for the self-management of irritable bowel syndrome: crossover randomized controlled trial. JMIR Mhealth Uhealth. May 20, 2021;9(5):e26152. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Prochaska JJ, Vogel EA, Chieng A, Kendra M, Baiocchi M, Pajarito S, et al. A therapeutic relational agent for reducing problematic substance use (Woebot): development and usability study. J Med Internet Res. Mar 23, 2021;23(3):e24850. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Echeazarra L, Pereira J, Saracho R. TensioBot: a chatbot assistant for self-managed in-house blood pressure checking. J Med Syst. Mar 15, 2021;45(4):54. [ CrossRef ] [ Medline ]
• Gardiner P, Bickmore T, Yinusa-Nyahkoon L, Reichert M, Julce C, Sidduri N, et al. Using health information technology to engage African American women on nutrition and supplement use during the preconception period. Front Endocrinol (Lausanne). Jan 19, 2020;11:571705. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fan X, Chao D, Zhang Z, Wang D, Li X, Tian F. Utilization of self-diagnosis health chatbots in real-world settings: case study. J Med Internet Res. Jan 06, 2021;23(1):e19928. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Maeda E, Miyata A, Boivin J, Nomura K, Kumazawa Y, Shirasawa H, et al. Promoting fertility awareness and preconception health using a chatbot: a randomized controlled trial. Reprod Biomed Online. Dec 2020;41(6):1133-1143. [ CrossRef ] [ Medline ]
• So R, Furukawa TA, Matsushita S, Baba T, Matsuzaki T, Furuno S, et al. Unguided ChatBot-delivered cognitive behavioural intervention for problem gamblers through messaging app: a randomised controlled trial. J Gambl Stud. Dec 11, 2020;36(4):1391-1407. [ CrossRef ] [ Medline ]
• Gong E, Baptista S, Russell A, Scuffham P, Riddell M, Speight J, et al. My diabetes coach, a mobile app-based interactive conversational agent to support type 2 diabetes self-management: randomized effectiveness-implementation trial. J Med Internet Res. Nov 05, 2020;22(11):e20322. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jack BW, Bickmore T, Yinusa-Nyahkoon L, Reichert M, Julce C, Sidduri N, et al. Improving the health of young African American women in the preconception period using health information technology: a randomised controlled trial. Lancet Digit Health. Sep 2020;2(9):e475-e485. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yoneoka D, Kawashima T, Tanoue Y, Nomura S, Ejima K, Shi S, et al. Early SNS-based monitoring system for the COVID-19 outbreak in Japan: a population-level observational study. J Epidemiol. Aug 05, 2020;30(8):362-370. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bickmore T, Zhang Z, Reichert M, Julce C, Jack B. Promotion of preconception care among adolescents and young adults by conversational agent. J Adolesc Health. Aug 2020;67(2S):S45-S51. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Oh J, Jang S, Kim H, Kim JJ. Efficacy of mobile app-based interactive cognitive behavioral therapy using a chatbot for panic disorder. Int J Med Inform. Aug 2020;140:104171. [ CrossRef ] [ Medline ]
• Anthony CA, Rojas EO, Keffala V, Glass NA, Shah AS, Miller BJ, et al. Acceptance and commitment therapy delivered via a mobile phone messaging robot to decrease postoperative opioid use in patients with orthopedic trauma: randomized controlled trial. J Med Internet Res. Jul 29, 2020;22(7):e17750. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kramer JN, Künzler F, Mishra V, Smith SN, Kotz D, Scholz U, et al. Which components of a smartphone walking app help users to reach personalized step goals? Results from an optimization trial. Ann Behav Med. Jun 12, 2020;54(7):518-528. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bennion MR, Hardy GE, Moore RK, Kellett S, Millings A. Usability, acceptability, and effectiveness of web-based conversational agents to facilitate problem solving in older adults: controlled study. J Med Internet Res. May 27, 2020;22(5):e16794. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Piao M, Ryu H, Lee H, Kim J. Use of the healthy lifestyle coaching ChatBot app to promote stair-climbing habits among office workers: exploratory randomized controlled trial. JMIR Mhealth Uhealth. May 19, 2020;8(5):e15085. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hauser-Ulrich S, Künzli H, Meier-Peterhans D, Kowatsch T. A smartphone-based health care chatbot to promote self-management of chronic pain (SELMA): pilot randomized controlled trial. JMIR Mhealth Uhealth. Apr 03, 2020;8(4):e15806. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bibault JE, Chaix B, Guillemassé A, Cousin S, Escande A, Perrin M, et al. A chatbot versus physicians to provide information for patients with breast cancer: blind, randomized controlled noninferiority trial. J Med Internet Res. Nov 27, 2019;21(11):e15787. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Greer S, Ramo D, Chang YJ, Fu M, Moskowitz J, Haritatos J. Use of the chatbot "Vivibot" to deliver positive psychology skills and promote well-being among young people after cancer treatment: randomized controlled feasibility trial. JMIR Mhealth Uhealth. Oct 31, 2019;7(10):e15018. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fadhil A, Wang Y, Reiterer H. Assistive conversational agent for health coaching: a validation study. Methods Inf Med. Jun 22, 2019;58(1):9-23. [ CrossRef ] [ Medline ]
• Abdullah AS, Gaehde S, Bickmore T. A tablet based embodied conversational agent to promote smoking cessation among veterans: a feasibility study. J Epidemiol Glob Health. Dec 2018;8(3-4):225-230. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Auriacombe M, Moriceau S, Serre F, Denis C, Micoulaud-Franchi JA, de Sevin E, et al. Development and validation of a virtual agent to screen tobacco and alcohol use disorders. Drug Alcohol Depend. Dec 01, 2018;193:1-6. [ CrossRef ] [ Medline ]
• King AC, Campero I, Sheats JL, Castro Sweet CM, Garcia D, Chazaro A, et al. Testing the comparative effects of physical activity advice by humans vs. computers in underserved populations: the COMPASS trial design, methods, and baseline characteristics. Contemp Clin Trials. Oct 2017;61:115-125. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gardiner PM, McCue KD, Negash LM, Cheng T, White LF, Yinusa-Nyahkoon L, et al. Engaging women with an embodied conversational agent to deliver mindfulness and lifestyle recommendations: a feasibility randomized control trial. Patient Educ Couns. Sep 2017;100(9):1720-1729. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hajna S, Sharp SJ, Cooper AJ, Williams KM, van Sluijs EM, Brage S, et al. Effectiveness of minimal contact interventions: an RCT. Am J Prev Med. Mar 2021;60(3):e111-e121. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ali R, Hoque E, Duberstein P, Schubert L, Razavi SZ, Kane B, et al. Aging and engaging: a pilot randomized controlled trial of an online conversational skills coach for older adults. Am J Geriatr Psychiatry. Aug 2021;29(8):804-815. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Reger GM, Norr AM, Rizzo AS, Sylvers P, Peltan J, Fischer D, et al. Virtual standardized patients vs academic training for learning motivational interviewing skills in the US Department of Veterans Affairs and the US Military: a randomized trial. JAMA Netw Open. Oct 01, 2020;3(10):e2017348. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lorenz N, Heim E, Roetger A, Birrer E, Maercker A. Randomized controlled trial to test the efficacy of an unguided online intervention with automated feedback for the treatment of insomnia. Behav Cogn Psychother. May 2019;47(3):287-302. [ CrossRef ] [ Medline ]
• Fitzpatrick KK, Darcy A, Vierhile M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Ment Health. Jun 06, 2017;4(2):e19. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Abujarad F, Ulrich D, Edwards C, Choo E, Pantalon MV, Jubanyik K, et al. Development and usability evaluation of VOICES: a digital health tool to identify elder mistreatment. J Am Geriatr Soc. Jun 21, 2021;69(6):1469-1478. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Baptista S, Wadley G, Bird D, Oldenburg B, Speight J, My Diabetes Coach Research Group. Acceptability of an embodied conversational agent for type 2 diabetes self-management education and support via a smartphone app: mixed methods study. JMIR Mhealth Uhealth. Jul 22, 2020;8(7):e17038. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hirsh AT, Miller MM, Hollingshead NA, Anastas T, Carnell ST, Lok BC, et al. A randomized controlled trial testing a virtual perspective-taking intervention to reduce race and socioeconomic status disparities in pain care. Pain. Oct 10, 2019;160(10):2229-2240. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kim YJ, DeLisa JA, Chung YC, Shapiro NL, Kolar Rajanna SK, Barbour E, et al. Recruitment in a research study via chatbot versus telephone outreach: a randomized trial at a minority-serving institution. J Am Med Inform Assoc. Dec 28, 2021;29(1):149-154. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kowatsch T, Schachner T, Harperink S, Barata F, Dittler U, Xiao G, et al. Conversational agents as mediating social actors in chronic disease management involving health care professionals, patients, and family members: multisite single-arm feasibility study. J Med Internet Res. Feb 17, 2021;23(2):e25060. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Loveys K, Sagar M, Broadbent E. The effect of multimodal emotional expression on responses to a digital human during a self-disclosure conversation: a computational analysis of user language. J Med Syst. Jul 22, 2020;44(9):143. [ CrossRef ] [ Medline ]
• Loveys K, Sagar M, Pickering I, Broadbent E. A digital human for delivering a remote loneliness and stress intervention to at-risk younger and older adults during the COVID-19 pandemic: randomized pilot trial. JMIR Ment Health. Nov 08, 2021;8(11):e31586. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ly KH, Ly AM, Andersson G. A fully automated conversational agent for promoting mental well-being: a pilot RCT using mixed methods. Internet Interv. Dec 2017;10:39-46. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Morse KE, Ostberg NP, Jones VG, Chan AS. Use characteristics and triage acuity of a digital symptom checker in a large integrated health system: population-based descriptive study. J Med Internet Res. Nov 30, 2020;22(11):e20549. [ FREE Full text ] [ CrossRef ] [ Medline ]
• O'Rourke SR, Branford KR, Brooks TL, Ives LT, Nagendran A, Compton SN. The emotional and behavioral impact of delivering bad news to virtual versus real standardized patients: a pilot study. Teach Learn Med. Aug 22, 2020;32(2):139-149. [ CrossRef ] [ Medline ]
• Schario ME, Bahner CA, Widenhofer TV, Rajaballey JI, Thatcher EJ. Chatbot-assisted care management. Prof Case Manag. 2022;27(1):19-25. [ CrossRef ] [ Medline ]
• Dhinagaran DA, Sathish T, Soong A, Theng YL, Best J, Tudor Car L. Conversational agent for healthy lifestyle behavior change: web-based feasibility study. JMIR Form Res. Dec 03, 2021;5(12):e27956. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Troitskaya O, Batkhina A. Mobile application for couple relationships: results of a pilot effectiveness study. Fam Process. Jun 14, 2022;61(2):625-642. [ CrossRef ] [ Medline ]
• So R, Emura N, Okazaki K, Takeda S, Sunami T, Kitagawa K, et al. Guided versus unguided chatbot-delivered cognitive behavioral intervention for individuals with moderate-risk and problem gambling: a randomized controlled trial (GAMBOT2 study). Addict Behav. Feb 2024;149:107889. [ CrossRef ] [ Medline ]
• An J, Ferrante JM, Macenat M, Ganesan S, Hudson SV, Omene C, et al. Promoting informed approaches in precision oncology and clinical trial participation for Black patients with cancer: community-engaged development and pilot testing of a digital intervention. Cancer (Forthcoming). Oct 13, 2023. [ CrossRef ] [ Medline ]
• Coelho J, Pecune F, Levavasseur Y, De Sevin E, D'incau E, Sagaspe P, et al. From improved sleep regularity to reduced sleep complaints and mental health conditions: a population-based interventional study using a smartphone-based virtual agent. Sleep. Sep 08, 2023;46(9):zsad165. [ CrossRef ] [ Medline ]
• Siglen E, Vetti HH, Augestad M, Steen VM, Lunde Å, Bjorvatn C. Evaluation of the rosa chatbot providing genetic information to patients at risk of hereditary breast and ovarian cancer: qualitative interview study. J Med Internet Res. Sep 01, 2023;25:e46571. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Griffin AC, Khairat S, Bailey SC, Chung AE. A chatbot for hypertension self-management support: user-centered design, development, and usability testing. JAMIA Open. Oct 2023;6(3):ooad073. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nadarzynski T, Lunt A, Knights N, Bayley J, Llewellyn C. "But can chatbots understand sex?" Attitudes towards artificial intelligence chatbots amongst sexual and reproductive health professionals: an exploratory mixed-methods study. Int J STD AIDS. Oct 03, 2023;34(11):809-816. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Alkoudmani RM, Ooi GS, Tan ML. Implementing a chatbot on Facebook to reach and collect data from thousands of health care providers: PharmindBot as a case. J Am Pharm Assoc (2003). Sep 2023;63(5):1634-42.e3. [ CrossRef ] [ Medline ]
• Morato JE, do Nascimento JW, Roque GD, de Souza RR, Santos IC. Development, validation, and usability of the Chatbot ESTOMABOT to promote self-care of people with intestinal ostomy. Comput Inform Nurs. Dec 01, 2023;41(12):1037-1045. [ CrossRef ] [ Medline ]
• Al-Hilli Z, Noss R, Dickard J, Wei W, Chichura A, Wu V, et al. A randomized trial comparing the effectiveness of pre-test genetic counseling using an artificial intelligence automated chatbot and traditional in-person genetic counseling in women newly diagnosed with breast cancer. Ann Surg Oncol. Oct 11, 2023;30(10):5990-5996. [ CrossRef ] [ Medline ]
• Hsu MH, Chen YH. Personalized medical terminology learning game: guess the term. Games Health J. Apr 01, 2024;13(2):84-92. [ CrossRef ] [ Medline ]
• Escobar-Viera CG, Porta G, Coulter RW, Martina J, Goldbach J, Rollman BL. A chatbot-delivered intervention for optimizing social media use and reducing perceived isolation among rural-living LGBTQ+ youth: development, acceptability, usability, satisfaction, and utility. Internet Interv. Dec 2023;34:100668. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cheng CI, Lin WJ, Liu HT, Chen YT, Chiang CK, Hung KY. Implementation of artificial intelligence chatbot in peritoneal dialysis nursing care: experience from a Taiwan medical center. Nephrology (Carlton). Dec 12, 2023;28(12):655-662. [ CrossRef ] [ Medline ]
• Wang Z, Chan PS, Fang Y, Yu FY, Ye D, Zhang Q, et al. Chatbot-delivered online intervention to promote seasonal influenza vaccination during the COVID-19 pandemic: a randomized clinical trial. JAMA Netw Open. Sep 05, 2023;6(9):e2332568. [ FREE Full text ] [ CrossRef ] [ Medline ]
• de Queiroz DA, Passarello RS, de Moura Fé VV, Rossini A, da Silveira EF, de Queiroz EA, et al. A wearable chatbot-based model for monitoring colorectal cancer patients in the active phase of treatment. Healthc Anal. Dec 2023;4:100257. [ FREE Full text ] [ CrossRef ]
• Chow JS, Blight V, Brown M, Glynn V, Lane B, Larkin A, et al. Curious thing, an artificial intelligence (AI)-based conversational agent for COVID-19 patient management. Aust J Prim Health. Aug 2023;29(4):312-318. [ CrossRef ] [ Medline ]
• Marrs JC, Orlando ST, Saseen JJ, Novins-Montague S, Sandy LC, Waughtal J, et al. Description of patient questions received by clinical pharmacists in the Nudge Study. Am J Health Syst Pharm. Sep 07, 2023;80(18):1247-1254. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rainey JP, Blackburn BE, McCutcheon CL, Kenyon CM, Campbell KJ, Anderson LA, et al. A multilingual chatbot can effectively engage arthroplasty patients who have limited english proficiency. J Arthroplasty. Jul 2023;38(7 Suppl 2):S78-S83. [ CrossRef ] [ Medline ]
• El Ayadi AM, Singh P, Duggal M, Kumar V, Kaur J, Sharma P, et al. Feasibility and acceptability of Saheli, a WhatsApp Chatbot, on COVID-19 vaccination among pregnant and breastfeeding women in rural North India. BMJ Innov. Aug 22, 2023;9(4):195-206. [ CrossRef ]
• Hsu MH. Mastering medical terminology with ChatGPT and Termbot. Health Educ J. Sep 04, 2023. [ FREE Full text ] [ CrossRef ]
• Mahlknecht A, Engl A, Piccoliori G, Wiedermann CJ. Supporting primary care through symptom checking artificial intelligence: a study of patient and physician attitudes in Italian general practice. BMC Prim Care. Sep 04, 2023;24(1):174. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Xu J, Yang L, Guo M. Designing and evaluating an emotionally responsive virtual patient simulation. Simul Healthc (Forthcoming). May 18, 2023. [ CrossRef ] [ Medline ]
• Gupta S, Gupta SS, McMath K, Sugandh S. Enhancing complex wound care by leveraging artificial intelligence: an artificial intelligence chatbot software study. Wounds. Aug 2023;35(8):E265-E267. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Beavers J, Schell RF, VanCleave H, Dillon RC, Simmons A, Chen H, et al. Evaluation of inpatient medication guidance from an artificial intelligence chatbot. Am J Health Syst Pharm. Dec 05, 2023;80(24):1822-1829. [ CrossRef ] [ Medline ]
• Larizza C, Bosoni P, Quaglini S, Chasseur M, Bevolo V, Zuccotti G, et al. V-care: An application to support lifestyle improvement in children with obesity. Int J Med Inform. Sep 2023;177:105140. [ CrossRef ] [ Medline ]
• Suharwardy S, Ramachandran M, Leonard SA, Gunaseelan A, Lyell DJ, Darcy A, et al. Feasibility and impact of a mental health chatbot on postpartum mental health: a randomized controlled trial. AJOG Glob Rep. Aug 2023;3(3):100165. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Potts C, Lindström F, Bond R, Mulvenna M, Booth F, Ennis E, et al. A multilingual digital mental health and well-being chatbot (ChatPal): pre-post multicenter intervention study. J Med Internet Res. Jul 06, 2023;25:e43051. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Babington-Ashaye A, de Moerloose P, Diop S, Geissbuhler A. Design, development and usability of an educational AI chatbot for people with haemophilia in Senegal. Haemophilia. Jul 22, 2023;29(4):1063-1073. [ CrossRef ] [ Medline ]
• Taylor JJ, Subramanian A, Freitas A, Ferreira DM, Dickinson CM. What do individuals with visual impairment need and want from a dialogue-based digital assistant? Clin Exp Optom. Aug 29, 2023;106(6):656-665. [ CrossRef ] [ Medline ]
• Santa-Cruz J, Moran L, Tovar M, Peinado J, Cutipe Y, Ramos L, et al. Mobilizing digital technology to implement a population-based psychological support response during the COVID-19 pandemic in Lima, Peru. Glob Ment Health (Camb). Jul 28, 2022;9:355-365. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chun-Hung L, Guan-Hsiung L, Wu-Chuan Y, Yu-Hsin L. Chatbot-assisted therapy for patients with methamphetamine use disorder: a preliminary randomized controlled trial. Front Psychiatry. Jul 7, 2023;14:1159399. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mane HY, Channell Doig A, Marin Gutierrez FX, Jasczynski M, Yue X, Srikanth NP, et al. Practical guidance for the development of Rosie, a health education question-and-answer chatbot for new mothers. J Public Health Manag Pract. 2023;29(5):663-670. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cho K, Foo YM, Dalziel B, Hu W. Chatbot-mediated learning of cardiac auscultation. Intern Med J. Dec 17, 2022;52(12):2176-2180. [ CrossRef ] [ Medline ]
• Bruijnes M, Kesteloo M, Brinkman WP. Reducing social diabetes distress with a conversational agent support system: a three-week technology feasibility evaluation. Front Digit Health. Jun 13, 2023;5:1149374. [ FREE Full text ] [ CrossRef ] [ Medline ]
• van der Schyff EL, Ridout B, Amon KL, Forsyth R, Campbell AJ. Providing self-led mental health support through an artificial intelligence-powered chat bot (Leora) to meet the demand of mental health care. J Med Internet Res. Jun 19, 2023;25:e46448. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Guo N, Luk TT, Wu YS, Guo Z, Chu JC, Cheung YT, et al. Effect of mobile interventions with nicotine replacement therapy sampling on long-term smoking cessation in community smokers: a pragmatic randomized clinical trial. Tob Induc Dis. Mar 24, 2023;21(March):44-13. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Booth F, Potts C, Bond R, Mulvenna M, Kostenius C, Dhanapala I, et al. A mental health and well-being chatbot: user event log analysis. JMIR Mhealth Uhealth. Jul 06, 2023;11:e43052. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Anmella G, Sanabra M, Primé-Tous M, Segú X, Cavero M, Morilla I, et al. Vickybot, a chatbot for anxiety-depressive symptoms and work-related burnout in primary care and health care professionals: development, feasibility, and potential effectiveness studies. J Med Internet Res. Apr 03, 2023;25:e43293. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fabian KE, Foster KT, Chwastiak L, Turner M, Wagenaar BH. Adapting a transdiagnostic digital mental health intervention for use among immigrant and refugee youth in Seattle: a human-centered design approach. Transl Behav Med. Nov 05, 2023;13(11):867-875. [ CrossRef ] [ Medline ]
• Massa P, de Souza Ferraz DA, Magno L, Silva AP, Greco M, Dourado I, et al. A transgender chatbot (Amanda Selfie) to create pre-exposure prophylaxis demand among adolescents in Brazil: assessment of acceptability, functionality, usability, and results. J Med Internet Res. Jun 23, 2023;25:e41881. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tapolin FM, Liaskos J, Zoulias E, Mantas J. A conversational web-based chatbot to disseminate COVID-19 advisory information. Stud Health Technol Inform. Jun 29, 2023;305:483-486. [ CrossRef ] [ Medline ]
• Durden E, Pirner MC, Rapoport SJ, Williams A, Robinson A, Forman-Hoffman VL. Changes in stress, burnout, and resilience associated with an 8-week intervention with relational agent "Woebot". Internet Interv. Sep 2023;33:100637. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pereira DS, Falcão F, Nunes A, Santos N, Costa P, Pêgo JM. Designing and building OSCEBot® for virtual OSCE - performance evaluation. Med Educ Online. Dec 22, 2023;28(1):2228550. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Brinsley J, Singh B, Maher CA. A digital lifestyle program for psychological distress, wellbeing and return-to-work: a proof-of-concept study. Arch Phys Med Rehabil. Nov 2023;104(11):1903-1912. [ CrossRef ] [ Medline ]
• Matheson EL, Smith HG, Amaral AC, Meireles JF, Almeida MC, Linardon J, et al. Using chatbot technology to improve Brazilian adolescents' body image and mental health at scale: randomized controlled trial. JMIR Mhealth Uhealth. Jun 19, 2023;11:e39934. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Walters NL, Lindsey-Mills ZT, Brangan A, Savage SK, Schmidlen TJ, Morgan KM, et al. Facilitating family communication of familial hypercholesterolemia genetic risk: assessing engagement with innovative chatbot technology from the IMPACT-FH study. PEC Innov. Dec 2023;2:100134. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jackson-Triche M, Vetal D, Turner EM, Dahiya P, Mangurian C. Meeting the behavioral health needs of health care workers during COVID-19 by leveraging chatbot technology: development and usability study. J Med Internet Res. Jun 08, 2023;25:e40635. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nehme M, Schneider F, Perrin A, Sum Yu W, Schmitt S, Violot G, et al. The development of a chatbot technology to disseminate post-COVID-19 information: descriptive implementation study. J Med Internet Res. Jun 05, 2023;25:e43113. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ko SQ, Chua CM, Koh SH, Lim YW, Shorey S. Experiences of patients and their caregivers in a virtual ward in Singapore: a descriptive qualitative study. Int J Med Inform. Sep 2023;177:105111. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kobayashi T, Tomoi H, Nishina Y, Harada K, Tanaka K, Sasaki S, et al. Effect of a mobile app chatbot and an interactive small-group webinar on COVID-19 vaccine intention and confidence in Japan: a randomised controlled trial. BMJ Glob Health. May 29, 2023;8(5):e010370. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lee KY, Dabak SV, Kong VH, Park M, Kwok SL, Silzle M, et al. Effectiveness of chatbots on COVID vaccine confidence and acceptance in Thailand, Hong Kong, and Singapore. NPJ Digit Med. May 25, 2023;6(1):96. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sabour S, Zhang W, Xiao X, Zhang Y, Zheng Y, Wen J, et al. A chatbot for mental health support: exploring the impact of Emohaa on reducing mental distress in China. Front Digit Health. May 4, 2023;5:1133987. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wang Q, Peng S, Zha Z, Han X, Deng C, Hu L, et al. Enhancing the conversational agent with an emotional support system for mental health digital therapeutics. Front Psychiatry. Apr 17, 2023;14:1148534. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Beatty C, Malik T, Meheli S, Sinha C. Evaluating the therapeutic alliance with a free-text CBT conversational agent (Wysa): a mixed-methods study. Front Digit Health. Apr 11, 2022;4:847991. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bézie A, Morisseau V, Rolland R, Guillemassé A, Brouard B, Chaix B. Using a chatbot to study medication overuse among patients suffering from headaches. Front Digit Health. Mar 17, 2022;4:801782. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Moilanen J, van Berkel N, Visuri A, Gadiraju U, van der Maden W, Hosio S. Supporting mental health self-care discovery through a chatbot. Front Digit Health. Mar 7, 2023;5:1034724. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Burger F, Neerincx MA, Brinkman WP. Using a conversational agent for thought recording as a cognitive therapy task: feasibility, content, and feedback. Front Digit Health. Jul 19, 2022;4:930874. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Soni H, Ivanova J, Wilczewski H, Bailey A, Ong T, Narma A, et al. Virtual conversational agents versus online forms: patient experience and preferences for health data collection. Front Digit Health. Oct 13, 2022;4:954069. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Crovari P, Pidò S, Pinoli P, Bernasconi A, Canakoglu A, Garzotto F, et al. GeCoAgent: a conversational agent for empowering genomic data extraction and analysis. ACM Trans Comput Healthc. Oct 15, 2021;3(1):1-29. [ CrossRef ]
• Dwyer T, Hoit G, Burns D, Higgins J, Chang J, Whelan D, et al. Use of an artificial intelligence conversational agent (Chatbot) for hip arthroscopy patients following surgery. Arthrosc Sports Med Rehabil. Apr 2023;5(2):e495-e505. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wilczewski H, Soni H, Ivanova J, Ong T, Barrera JF, Bunnell BE, et al. Older adults' experience with virtual conversational agents for health data collection. Front Digit Health. Mar 15, 2023;5:1125926. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Montenegro JL, da Costa CA, Janssen LP. Evaluating the use of chatbot during pregnancy: a usability study. Healthc Anal. Nov 2022;2:100072. [ CrossRef ]
• Ko SQ, Kumar SK, Jacob J, Hooi BM, Soo M, Nashi N, et al. Technology-enabled virtual ward for COVID management of the elderly and immunocompromised in Singapore: a descriptive cohort. BMC Infect Dis. Feb 21, 2023;23(1):102. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hsu MH, Chan TM, Yu CS. Termbot: a chatbot-based crossword game for gamified medical terminology learning. Int J Environ Res Public Health. Feb 26, 2023;20(5):4185. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Epalte K, Tomsone S, Vētra A, Bērziņa G. Patient experience using digital therapy "Vigo" for stroke patient recovery: a qualitative descriptive study. Disabil Rehabil Assist Technol. Feb 06, 2023;18(2):175-184. [ CrossRef ] [ Medline ]
• Kobayashi T, Nishina Y, Tomoi H, Harada K, Tanaka K, Matsumoto E, et al. Corowa-kun: a messenger app chatbot delivers COVID-19 vaccine information, Japan 2021. Vaccine. Jul 30, 2022;40(32):4654-4662. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chin H, Lima G, Shin M, Zhunis A, Cha C, Choi J, et al. User-chatbot conversations during the COVID-19 pandemic: study based on topic modeling and sentiment analysis. J Med Internet Res. Jan 27, 2023;25:e40922. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Albers N, Hizli B, Scheltinga BL, Meijer E, Brinkman WP. Setting physical activity goals with a virtual coach: vicarious experiences, personalization and acceptance. J Med Syst. Jan 30, 2023;47(1):15. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kaywan P, Ahmed K, Ibaida A, Miao Y, Gu B. Early detection of depression using a conversational AI bot: a non-clinical trial. PLoS One. Feb 3, 2023;18(2):e0279743. [ FREE Full text ] [ CrossRef ] [ Medline ]
• He Y, Yang L, Zhu X, Wu B, Zhang S, Qian C, et al. Mental health chatbot for young adults with depressive symptoms during the COVID-19 pandemic: single-blind, three-arm randomized controlled trial. J Med Internet Res. Nov 21, 2022;24(11):e40719. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Castilla E, Escobar JJ, Villalonga C, Banos O. HIGEA: an intelligent conversational agent to detect caregiver burden. Int J Environ Res Public Health. Nov 30, 2022;19(23):16019. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chen T, Chen L, Li S, Du J, Su H, Jiang H, et al. Virtual digital psychotherapist app-based treatment in patients with methamphetamine use disorder (Echo-APP): single-arm pilot feasibility and efficacy study. JMIR Mhealth Uhealth. Jan 31, 2023;11:e40373. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dosovitsky G, Bunge E. Development of a chatbot for depression: adolescent perceptions and recommendations. Child Adolesc Ment Health. Feb 11, 2023;28(1):124-127. [ CrossRef ] [ Medline ]
• Montenegro JL, da Costa CA, da Rosa Righi R, Farias ER, Matté LB. Development and validation of conversational agent to pregnancy safe-education. J Med Syst. Jan 10, 2023;47(1):7. [ CrossRef ] [ Medline ]
• Ntinga X, Musiello F, Keter AK, Barnabas R, van Heerden A. The feasibility and acceptability of an mHealth conversational agent designed to support HIV self-testing in South Africa: cross-sectional study. J Med Internet Res. Dec 12, 2022;24(12):e39816. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rodriguez-Arrastia M, Martinez-Ortigosa A, Ruiz-Gonzalez C, Ropero-Padilla C, Roman P, Sanchez-Labraca N. Experiences and perceptions of final-year nursing students of using a chatbot in a simulated emergency situation: a qualitative study. J Nurs Manag. Nov 18, 2022;30(8):3874-3884. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Han JW, Park J, Lee H. Analysis of the effect of an artificial intelligence chatbot educational program on non-face-to-face classes: a quasi-experimental study. BMC Med Educ. Dec 01, 2022;22(1):830. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Singh E, Bompelli A, Wan R, Bian J, Pakhomov S, Zhang R. A conversational agent system for dietary supplements use. BMC Med Inform Decis Mak. Jul 07, 2022;22(Suppl 1):153. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Andriani SP, Adhyanacarira P, Fuad A, Pertiwi AA. Comparison of non-AI and AI-enabled mHealth platforms for COVID-19 self screening in Indonesia. Stud Health Technol Inform. Jun 29, 2022;295:226-229. [ CrossRef ] [ Medline ]
• Olano-Espinosa E, Avila-Tomas JF, Minue-Lorenzo C, Matilla-Pardo B, Serrano ME, Martinez-Suberviola FJ, et al. Dejal@ Group. Effectiveness of a conversational chatbot (Dejal@bot) for the adult population to quit smoking: pragmatic, multicenter, controlled, randomized clinical trial in primary care. JMIR Mhealth Uhealth. Jun 27, 2022;10(6):e34273. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rizzato Lede DA, Inda D, Rosa JM, Zin Y, Tentoni N, Médici MM, et al. Tana, a healthcare chatbot to help patients during the COVID-19 pandemic at a university hospital in Argentina. Stud Health Technol Inform. Jun 06, 2022;290:301-303. [ CrossRef ] [ Medline ]
• Denecke K, Lombardo P, Nairz K. Digital medical interview assistant for radiology: opportunities and challenges. Stud Health Technol Inform. May 16, 2022;293:39-46. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ben-Shabat N, Sharvit G, Meimis B, Joya DB, Sloma A, Kiderman D, et al. Assessing data gathering of chatbot based symptom checkers - a clinical vignettes study. Int J Med Inform. Dec 2022;168:104897. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fitzsimmons-Craft EE, Chan WW, Smith AC, Firebaugh ML, Fowler LA, Topooco N, et al. Effectiveness of a chatbot for eating disorders prevention: a randomized clinical trial. Int J Eat Disord. Mar 28, 2022;55(3):343-353. [ CrossRef ] [ Medline ]
• Pithpornchaiyakul S, Naorungroj S, Pupong K, Hunsrisakhun J. Using a chatbot as an alternative approach for in-person toothbrushing training during the COVID-19 pandemic: comparative study. J Med Internet Res. Oct 21, 2022;24(10):e39218. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yang LW, Ng WY, Lei X, Tan SC, Wang Z, Yan M, et al. Development and testing of a multi-lingual natural language processing-based deep learning system in 10 languages for COVID-19 pandemic crisis: a multi-center study. Front Public Health. Feb 13, 2023;11:1063466. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Suárez A, Adanero A, Díaz-Flores García V, Freire Y, Algar J. Using a virtual patient via an artificial intelligence chatbot to develop dental students' diagnostic skills. Int J Environ Res Public Health. Jul 18, 2022;19(14):8735. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ogawa M, Oyama G, Morito K, Kobayashi M, Yamada Y, Shinkawa K, et al. Can AI make people happy? The effect of AI-based chatbot on smile and speech in Parkinson's disease. Parkinsonism Relat Disord. Jun 2022;99:43-46. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vasileiou MV, Maglogiannis IG. The health chatbots in telemedicine: intelligent dialog system for remote support. J Healthc Eng. Oct 6, 2022;2022:1-12. [ CrossRef ]
• Danieli M, Ciulli T, Mousavi SM, Silvestri G, Barbato S, Di Natale L, et al. Assessing the impact of conversational artificial intelligence in the treatment of stress and anxiety in aging adults: randomized controlled trial. JMIR Ment Health. Sep 23, 2022;9(9):e38067. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Luk TT, Lui JH, Wang MP. Efficacy, usability, and acceptability of a chatbot for promoting COVID-19 vaccination in unvaccinated or booster-hesitant young adults: pre-post pilot study. J Med Internet Res. Oct 04, 2022;24(10):e39063. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Martinengo L, Lum E, Car J. Evaluation of chatbot-delivered interventions for self-management of depression: content analysis. J Affect Disord. Dec 15, 2022;319:598-607. [ CrossRef ] [ Medline ]
• Shah J, DePietro B, D'Adamo L, Firebaugh ML, Laing O, Fowler LA, et al. Development and usability testing of a chatbot to promote mental health services use among individuals with eating disorders following screening. Int J Eat Disord. Sep 18, 2022;55(9):1229-1244. [ FREE Full text ] [ CrossRef ] [ Medline ]
• De Marchi F, Serioli M, Collo A, Belotti EG, Alloatti F, Biroli G, et al. A telehealth intervention for nutritional counseling in amyotrophic lateral sclerosis patients. J Clin Med. Jul 23, 2022;11(15):4286. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rabinowitz AR, Collier G, Vaccaro M, Wingfield R. Development of RehaBot-a conversational agent for promoting rewarding activities in users with traumatic brain injury. J Head Trauma Rehabil. 2022;37(3):144-151. [ CrossRef ] [ Medline ]
• Alturaiki AM, Banjar HR, Barefah AS, Alnajjar SA, Hindawi S. A smart chatbot for interactive management in beta thalassemia patients. Int J Telemed Appl. May 11, 2022;2022:9734518-9734513. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Terblanche N, Molyn J, de Haan E, Nilsson VO. Comparing artificial intelligence and human coaching goal attainment efficacy. PLoS One. Jun 21, 2022;17(6):e0270255. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nam KH, Kim DY, Kim DH, Lee JH, Lee JI, Kim MJ, et al. Conversational artificial intelligence for spinal pain questionnaire: validation and user satisfaction. Neurospine. Jun 2022;19(2):348-356. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nißen M, Rüegger D, Stieger M, Flückiger C, Allemand M, V Wangenheim F, et al. The effects of health care chatbot personas with different social roles on the client-chatbot bond and usage intentions: development of a design codebook and web-based study. J Med Internet Res. Apr 27, 2022;24(4):e32630. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Larbi D, Denecke K, Gabarron E. Usability testing of a social media chatbot for increasing physical activity behavior. J Pers Med. May 20, 2022;12(5):828. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ritchie JB, Frey LJ, Lamy JB, Bellcross C, Morrison H, Schiffman JD, et al. Automated clinical practice guideline recommendations for hereditary cancer risk using chatbots and ontologies: system description. JMIR Cancer. Jan 31, 2022;8(1):e29289. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hope DL, Grant GD, Rogers GD, King MA. Virtualized gamified pharmacy simulation during COVID-19. Pharmacy (Basel). Mar 26, 2022;10(2):41. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Munsch N, Gruarin S, Nateqi J, Lutz T, Binder M, Aberle JH, et al. Symptoms associated with a COVID-19 infection among a non-hospitalized cohort in Vienna. Wien Klin Wochenschr. May 13, 2022;134(9-10):344-350. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Minutolo A, Damiano E, De Pietro G, Fujita H, Esposito M. A conversational agent for querying Italian patient information leaflets and improving health literacy. Comput Biol Med. Feb 2022;141:105004. [ CrossRef ] [ Medline ]
• Hurmuz MZ, Jansen-Kosterink SM, Beinema T, Fischer K, Op den Akker H, Hermens HJ. Evaluation of a virtual coaching system eHealth intervention: a mixed methods observational cohort study in the Netherlands. Internet Interv. Mar 2022;27:100501. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Beinema T, Op den Akker H, Hurmuz M, Jansen-Kosterink S, Hermens H. Automatic topic selection for long-term interaction with embodied conversational agents in health coaching: a micro-randomized trial. Internet Interv. Mar 2022;27:100502. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Liu H, Peng H, Song X, Xu C, Zhang M. Using AI chatbots to provide self-help depression interventions for university students: a randomized trial of effectiveness. Internet Interv. Mar 2022;27:100495. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Morgan KM, Hamilton JG, Symecko H, Kamara D, Jenkins C, Lester J, et al. Targeted BRCA1/2 population screening among Ashkenazi Jewish individuals using a web-enabled medical model: an observational cohort study. Genet Med. Mar 2022;24(3):564-575. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dhinagaran DA, Martinengo L, Ho MH, Joty S, Kowatsch T, Atun R, et al. Designing, developing, evaluating, and implementing a smartphone-delivered, rule-based conversational agent (DISCOVER): development of a conceptual framework. JMIR Mhealth Uhealth. Oct 04, 2022;10(10):e38740. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Schmidlen T, Jones CL, Campbell-Salome G, McCormick CZ, Vanenkevort E, Sturm AC. Use of a chatbot to increase uptake of cascade genetic testing. J Genet Couns. Oct 26, 2022;31(5):1219-1230. [ CrossRef ] [ Medline ]
• Wang WT, Tan N, Hanson JA, Crubaugh CA, Hara AK. Initial experience with a COVID-19 screening chatbot before radiology appointments. J Digit Imaging. Oct 13, 2022;35(5):1303-1307. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rathnayaka P, Mills N, Burnett D, De Silva D, Alahakoon D, Gray R. A mental health chatbot with cognitive skills for personalised behavioural activation and remote health monitoring. Sensors (Basel). May 11, 2022;22(10):3653. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Miura C, Chen S, Saiki S, Nakamura M, Yasuda K. Assisting personalized healthcare of elderly people: developing a rule-based virtual caregiver system using mobile chatbot. Sensors (Basel). May 18, 2022;22(10):3829. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Asensio-Cuesta S, Blanes-Selva V, Conejero A, Portolés M, García-Gómez M. A user-centered chatbot to identify and interconnect individual, social and environmental risk factors related to overweight and obesity. Inform Health Soc Care. Jan 02, 2022;47(1):38-52. [ CrossRef ] [ Medline ]
• Braveman P. Health disparities and health equity: concepts and measurement. Annu Rev Public Health. Apr 01, 2006;27(1):167-194. [ CrossRef ] [ Medline ]
• Epstein RM, Street RL. The values and value of patient-centered care. Ann Fam Med. Mar 14, 2011;9(2):100-103. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Shaller D. Patient-centered care: what does it take? The Commonwealth Fund. 2007. URL: https:/​/www.​commonwealthfund.org/​publications/​fund-reports/​2007/​oct/​patient-centered-care-what-does-it-take [accessed 2024-04-29]
• Efficient definition and meaning. Merriam-Webster. 2022. URL: https://www.merriam-webster.com/dictionary/efficient [accessed 2024-04-29]
• The world by income and region. The World Bank. 2022. URL: https://datatopics.worldbank.org/world-development-indicators/the-world-by-income-and-region.html [accessed 2024-04-29]
• Primary health care. World Health Organization. URL: https://www.who.int/health-topics/primary-health-care [accessed 2024-04-29]
• Frangoudes F, Hadjiaros M, Schiza EC, Matsangidou M, Tsivitanidou O, Neokleous K. An overview of the use of chatbots in medical and healthcare education. In: Proceedings of 8th International Conference, LCT 2021, Held as Part of the 23rd HCI International Conference, HCII 2021 on Learning and Collaboration Technologies: Games and Virtual Environments for Learning. 2021. Presented at: HCII '22; July 24-29, 2021:170-184; Virtual Event. URL: https://link.springer.com/chapter/10.1007/978-3-030-77943-6_11 [ CrossRef ]
• Thakre K, Rothe PR, Kukade S, Shinde P, Madame K. Health care chatbot using NLP and flask. Ijraset Int J Res Appl Sci Eng Technol. May 31, 2022;10(5):2632-2654. [ CrossRef ]
• Sheth A, Yip HY, Shekarpour S. Extending patient-chatbot experience with internet-of-things and background knowledge: case studies with healthcare applications. IEEE Intell Syst. Jul 1, 2019;34(4):24-30. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Roca S, Sancho J, García J, Alesanco Á. Microservice chatbot architecture for chronic patient support. J Biomed Inform. Feb 2020;102:103305. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vryoni V, Bailey S. Chatbots in healthcare: connecting patients to information: emerging health technologies. Canadian Agency for Drugs and Technologies in Health. 2024. URL: https://www.ncbi.nlm.nih.gov/books/NBK602381/ [accessed 2024-04-29]
• Jadczyk T, Wojakowski W, Tendera M, Henry TD, Egnaczyk G, Shreenivas S. Artificial intelligence can improve patient management at the time of a pandemic: the role of voice technology. J Med Internet Res. May 25, 2021;23(5):e22959. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fadhil A. Beyond patient monitoring: conversational agents role in telemedicine and healthcare support for home-living elderly individuals. arXiv. Preprint posted online March 3, 2018. [ FREE Full text ]
• Bidve V, Virkar A, Raut P, Velapurkar S. NOVA-a virtual nursing assistant. Indones J Electr Eng Comput Sci. Apr 01, 2023;30(1):307-315. [ CrossRef ]
• Stone P, Brooks R, Brynjolfsson E, Calo R, Etzioni O, Hager G, et al. Artificial intelligence and life in 2030: the one hundred year study on artificial intelligence. arXiv. Preprint posted online October 31, 2022. [ FREE Full text ] [ CrossRef ]
• Nishida T, Nakazawa A, Ohmoto Y, Mohammad Y. Conversational Informatics: A Data-Intensive Approach with Emphasis on Nonverbal Communication. Cham, Switzerland. Springer; 2014.
• Wolters MK, Kelly F, Kilgour J. Designing a spoken dialogue interface to an intelligent cognitive assistant for people with dementia. Health Informatics J. Dec 26, 2016;22(4):854-866. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tzelios C, Contreras C, Istenes B, Astupillo A, Lecca L, Ramos K, et al. Using digital chatbots to close gaps in healthcare access during the COVID-19 pandemic. Public Health Action. Dec 21, 2022;12(4):180-185. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vijayarani M, Balamurugan G. Chatbot in mental health care. Indian J Psychiatry Nurse. 2019;16(2):126-128. [ CrossRef ]
• Peng ML, Wickersham JA, Altice FL, Shrestha R, Azwa I, Zhou X, et al. Formative evaluation of the acceptance of HIV prevention artificial intelligence chatbots by men who have sex with men in Malaysia: focus group study. JMIR Form Res. Oct 06, 2022;6(10):e42055. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Koulouri T, Macredie RD, Olakitan D. Chatbots to support young adults’ mental health: an exploratory study of acceptability. ACM Trans Interact Intell Syst. Jul 20, 2022;12(2):1-39. [ CrossRef ]
• Ahmed A, Hassan A, Aziz S, Abd-Alrazaq AA, Ali N, Alzubaidi M, et al. Chatbot features for anxiety and depression: a scoping review. Health Informatics J. Jan 24, 2023;29(1):14604582221146719. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mehta A, Virkar S, Khatri J, Thakur R, Dalvi A. Artificial intelligence powered chatbot for mental healthcare based on sentiment analysis. In: Proceedings of the 2022 5th International Conference on Advances in Science and Technology. 2022. Presented at: ICAST '22; December 2-3, 2022:185-189; Mumbai, India. URL: https://ieeexplore.ieee.org/document/10039548 [ CrossRef ]
• Lucas GM, Gratch J, King A, Morency LP. It’s only a computer: virtual humans increase willingness to disclose. Comput Human Behav. Aug 2014;37:94-100. [ CrossRef ]
• Trost MJ, Chrysilla G, Gold JI, Matarić M. Socially-assistive robots using empathy to reduce pain and distress during peripheral IV placement in children. Pain Res Manag. Apr 09, 2020;2020:7935215-7935217. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Suzuki Y, Galli L, Ikeda A, Itakura S, Kitazaki M. Measuring empathy for human and robot hand pain using electroencephalography. Sci Rep. Nov 03, 2015;5:15924. [ CrossRef ] [ Medline ]
• Chita-Tegmark M, Ackerman JM, Scheutz M. Effects of assistive robot behavior on impressions of patient psychological attributes: vignette-based human-robot interaction study. J Med Internet Res. May 19, 2019;21(6):e13729. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bulla C, Parushetti C, Teli A, Aski S, Koppad S. A review of AI based medical assistant chatbot: research and applications of web development and design. Res Appl Web Develop Des. 2020;3(2):1-4. [ FREE Full text ]
• Liu B, Sundar SS. Should machines express sympathy and empathy? Experiments with a health advice chatbot. Cyberpsychol Behav Soc Netw. Oct 2018;21(10):625-636. [ CrossRef ] [ Medline ]
• Kao CH, Chen CC, Tsai YT. Model of multi-turn dialogue in emotional chatbot. In: Proceedings of the 2019 International Conference on Technologies and Applications of Artificial Intelligence. 2019. Presented at: TAAi '19; November 21-23, 2019:1-5; Kaohsiung, Taiwan. URL: https://ieeexplore.ieee.org/document/8959855 [ CrossRef ]
• Purushotham K, Priya KS, Jayasree K, Sekhar GS, Jaswanth N, Kumar KK. Automated conversation chatbot for multiple languages for hospitals. Int J Adv Res Sci Commun Technol. Jan 10, 2023;3(1):209-217. [ FREE Full text ] [ CrossRef ]
• Hernández-Neuta I, Neumann F, Brightmeyer J, Ba Tis T, Madaboosi N, Wei Q, et al. Smartphone-based clinical diagnostics: towards democratization of evidence-based health care. J Intern Med. Jan 12, 2019;285(1):19-39. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Iancu I, Iancu B. Interacting with chatbots later in life: a technology acceptance perspective in COVID-19 pandemic situation. Front Psychol. Jan 16, 2022;13:1111003. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, et al. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol. Dec 21, 2017;2(4):230-243. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Brown JE, Halpern J. AI chatbots cannot replace human interactions in the pursuit of more inclusive mental healthcare. SSM Mental Health. Dec 2021;1:100017. [ CrossRef ]
• Whitby B. The ethical implications of non-human agency in health care. In: Proceedings of the 2014 Machine Ethics in the Context of Medical and Care Agents. 2014. Presented at: MEMCA '14; April 1-4, 2014:1-4; London, UK. URL: https://doc.gold.ac.uk/aisb50/AISB50-S17/AISB50-S17-Whitby-Paper.pdf
• Chaix B, Guillemassé A, Nectoux P, Delamon G, Brouard B. Vik: a chatbot to support patients with chronic diseases. Health. 2020;12(07):804-810. [ CrossRef ]
• Srilakshmi V, Prathima V. Disease prediction and diagnosis using machine learning. Int J Res Appl Sci Eng Technol. Jul 31, 2022;10(7):1594-1597. [ CrossRef ]
• Jovanovic M, Baez M, Casati F. Chatbots as conversational healthcare services. IEEE Internet Comput. May 2021;25(3):44-51. [ CrossRef ]
• Murtarelli G, Gregory A, Romenti S. A conversation-based perspective for shaping ethical human–machine interactions: the particular challenge of chatbots. J Bus Res. May 2021;129:927-935. [ CrossRef ]
• Følstad A, Araujo T, Law EL, Brandtzaeg PB, Papadopoulos S, Reis L, et al. Future directions for chatbot research: an interdisciplinary research agenda. Computing. Oct 19, 2021;103(12):2915-2942. [ CrossRef ]
• Cameron G, Cameron D, Megaw G, Bond RR, Mulvenna MD, O'Neill S, et al. Best practices for designing chatbots in mental healthcare: a case study on iHelpr. In: Proceedings of the 32nd International BCS Human Computer Interaction Conference. 2018. Presented at: HCI '18; July 4-6, 2018:1-5; Belfast, UK. URL: https://dl.acm.org/doi/10.14236/ewic/HCI2018.129 [ CrossRef ]
• Stiefel S. 'The chatbot will see you now': mental health confidentiality concerns in software therapy. SSRN Journal. Preprint posted online July 19, 2018. [ FREE Full text ] [ CrossRef ]
• Watt A, Cameron A, Sturm L, Lathlean T, Babidge W, Blamey S, et al. Rapid versus full systematic reviews: validity in clinical practice? ANZ J Surg. Nov 21, 2008;78(11):1037-1040. [ CrossRef ] [ Medline ]
• Kandpal P, Jasnani K, Raut R, Bhorge S. Contextual chatbot for healthcare purposes (using deep learning). In: Proceedings of the 2020 4th World Conference on Smart Trends in Systems, Security and Sustainability. 2020. Presented at: WorldS4 '20; July 27-28, 2020:625-634; London, UK. URL: https://ieeexplore.ieee.org/document/9210351 [ CrossRef ]
• Pernencar C, Saboia I, Dias JC. How far can conversational agents contribute to IBD patient health care-a review of the literature. Front Public Health. Jun 30, 2022;10:862432. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ramsetty A, Adams C. Impact of the digital divide in the age of COVID-19. J Am Med Inform Assoc. Jul 01, 2020;27(7):1147-1148. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lorence DP, Park H, Fox S. Racial disparities in health information access: resilience of the Digital Divide. J Med Syst. Aug 7, 2006;30(4):241-249. [ CrossRef ] [ Medline ]
• Cornejo Müller A, Wachtler B, Lampert T. Digital divide-social inequalities in the utilisation of digital healthcare. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. Feb 08, 2020;63(2):185-191. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tett G. When algorithms reinforce inequality. Financial Times. 2018. URL: https://www.ft.com/content/fb583548-0b93-11e8-839d-41ca06376bf2 [accessed 2024-04-29]
• Fiske A, Henningsen P, Buyx A. Your robot therapist will see you now: ethical implications of embodied artificial intelligence in psychiatry, psychology, and psychotherapy. J Med Internet Res. May 09, 2019;21(5):e13216. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kim HW, Kankanhalli A. Investigating user resistance to information systems implementation: a status quo bias perspective. MIS Q. 2009;33(3):567-582. [ CrossRef ]
• Best L, Stevens A, Colin-Jones D. Rapid and responsive health technology assessment: the development and evaluation process in the South and West region of England. J Clin Eff. 1997;2(2):51-56. [ CrossRef ]
• Gravel J, D’Amours-Gravel M, Osmanlliu E. Learning to fake it: limited responses and fabricated references provided by ChatGPT for medical questions. Mayo Clin Proc Digit Health. Sep 2023;1(3):226-234. [ FREE Full text ] [ CrossRef ]

Abbreviations

Edited by Y-H Lin; submitted 02.02.24; peer-reviewed by J Xie, R Forsyth; comments to author 18.03.24; revised version received 07.04.24; accepted 12.04.24; published 23.07.24.

©Moustafa Laymouna, Yuanchao Ma, David Lessard, Tibor Schuster, Kim Engler, Bertrand Lebouché. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 23.07.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.