21st century skills using technology to research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Front Psychol

A Review of Research on Technology-Supported Language Learning and 21st Century Skills

Associated data.

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Modern society needs people to be equipped with 21st century skills (e.g., critical thinking, creativity, communication, digital literacy, or collaboration skills). For this reason, teaching and learning nowadays should promote not only students' knowledge acquisition in various learning contexts but also their 21st century skills, and language learning context is no exception. This study reviewed research on technology-supported language learning and 21st century skills. The reason is that earlier studies reviewed only articles related to language learning supported by technology and mostly focused on languages, language skills and technologies used. That is to say, 21st century skills were not considered in earlier review studies. The present study selected and reviewed 34 articles published between 2011 and 2022 (February) and focused on the following dimensions: (1) research focus such as language skills and 21st century skills; (2) theoretical foundations; (3) technologies; (4) learning activities; (5) methodology; and (6) findings. The present research found that reviewed studies had focused most frequently on such language skills as speaking and writing and on such 21st century skills as communication and collaboration. The social constructivism theory was often used by scholars to base their studies on. Facebook, Google Docs, and Moodle were popular technologies in reviewed studies to facilitate language and 21st century skills. Scholars in reviewed studies reported that technology-supported language learning activities provided learners with good learning experiences and enhanced their learning motivation, engagement, and confidence. However, some challenges that learners faced during learning activities were also reported. Based on the results of the review, this study made several recommendations for stakeholders such as educators and researchers in the field.

Introduction

It is important that our students not only acquire new knowledge when they learn, but also develop skills, such as problem-solving, social cooperation, creativity, and so on, in order to apply newly learned knowledge to the real world. Such knowledge and skills will help them adapt to modern society and will enhance their competitiveness (Shadiev et al., 2022a , b ). Many countries have put forward the 21st century skills framework to carry out education reform (Lin et al., 2020 ), and one of them was proposed by the Partnership for 21st Century (P21). The P21 (Partnership for 21st Century Skills, 2008 ) provided a detailed conceptual framework and listed three types of skills: (1) learning and innovation skills (critical thinking and problem solving, creativity and innovation, and communication and collaboration), (2) digital literacy (information literacy, media literacy, and information and communication technologies (ICT) literacy), and (3) career and life skills (flexibility and adaptability, initiative and self-direction, social and cross-cultural interaction, productivity and accountability, and leadership and responsibility). The essence of these skills is that they are key skills that learners will need for their social and professional life in the future. These skills also emphasize the ability of learners to use and transfer knowledge and solve problems in complex situations, so they can achieve deep levels of individual learning as well as lifelong learning (Shadiev et al., 2022a ).

Developing students' 21st century skills needs to be implemented in all disciplines, and foreign language learning is no exception (Shadiev et al., In Press ). This matter has been addressed in the documents related to Asia Pacific Economic Cooperation ( 2004 ). Furthermore, researchers have carried out related studies, and pointed out the advantages of technology in developing both language skills and 21st century skills (Shadiev et al., In Press ). For example, Suzanne ( 2014 ) pointed out that when developing learners' reading skills, they deepened the learners' understanding of reading content, and also developed critical thinking skills. García-Sánchez and Burbules ( 2016 ) have found that students' skills such as problem solving, collaboration, listening and speaking improved after they completed online collaborative tasks. Srebnaja and Stavicka ( 2018 ) also pointed out that, in language learning projects supported by WebQuest, students' creativity, collaboration, and speaking skills have been developed. In the study by Chiang ( 2020 ), the digital storytelling activity was designed which promoted language learners' writing skills as well as their digital literacy skills.

A theoretical foundation to support technology supported language learning and development of 21st century skills can be built on various theories. The most relevant can be considered as second language acquisition theory, socio-cultural theory, and constructivism theory. For example, second language acquisition theory states that language acquisition is a process of input, absorption, and output. Language acquisition is acquired through exposure to contexts, understanding discourse, and then using language in natural communicative contexts (Krashen, 1985 ). According to socio-cultural theory, learning is a social phenomenon; it emphasizes the social nature of learning and argues that the development of learners' abilities arises from interpersonal interactions (Lantolf, 2000 ). Constructivism theory suggests that learning is a process in which a learner actively constructs meaning. That is, learners generate meaning and construct understanding based on prior knowledge and experience, often in the context of socio-cultural interactions. Constructivism theory emphasizes the social and contextual nature of learning (Vygotsky, 1978 ). Over the years, scholars have created technology-supported learning environments for language learning and 21st century skills development based on these theories. Such environments provide students with authentic learning materials, support social interaction, and facilitate their creative expression and construction of meaning actively using the target language.

Some related review studies already exist in the field. For example, Shadiev and Yang ( 2020 ) reviewed 398 articles related to technology-assisted language learning published in 10 Social Science Citation Index (SSCI) journals. The dimensions analyzed in their study included target language, language skills, technologies, and research findings. Shadiev and Yang ( 2020 ) found that the most commonly used language was English, followed by Chinese. The most targeted language skills were writing, speaking, and vocabulary acquisition. Digital games and online videos were the most commonly used technologies in these reviewed studies. In addition, most of the reviewed studies reported positive impacts of technology applications on language learning. Zhang and Zou ( 2020 ) reviewed 57 articles on technology applications for language learning that were published between 2016 and 2019 in 10 SSCI journals. The types of technology, the purpose of technology use, and the effectiveness of the technologies were reviewed by the authors. Zhang and Zou ( 2020 ) found that mobile learning, multimedia learning and socialization, voice to text recognition, text to speech recognition, and digital game-based learning were the most frequently investigated types of technology in the literature. The purposes for their use mainly covered four areas, including promoting practice, providing teaching content, promoting interaction, and reconstructing teaching methods. Scholars have claimed that technologies have positive effects on language learning. Goksu et al. ( 2020 ) reviewed 310 articles in 10 journals in the field of technology-assisted language learning. In addition, they evaluated a metadata set of 469 articles in the Web of Science database through bibliometric mapping. The review focused on the types, purposes, and effectiveness of the latest technologies on language learning. Goksu et al. ( 2020 ) found that most studies used quantitative research methods and were carried out with participants at higher academic levels. In addition, most studies focused on language skills as well as learning motivation and learner perceptions. Shadiev et al. ( 2017 ) studied 37 articles published in the top 10 SSCI journals related to educational technology from 2007 to 2016 (March). Scholars took mobile language learning in a real environment as the research object and summarized the results from four perspectives: journal publishing trends, language learning, research focus, and research methods. The results showed that the journal publishing trend was increasing. The most common research focus was cognition and language learner proficiency. The results also showed that mobile technology was positively perceived and accepted by students in most of these studies, and the technology was also found to have a positive impact on the students' language skills.

By exploring these review studies, the present review research found that 21st century skills were not considered in these earlier studies at all because scholars mainly focused on language skills. Therefore, several important aspects (e.g., theoretical foundations used to support the studies, methodology, and types of learning activities that promote language skills and 21st century skills) were ignored. These aspects are important for stakeholders in the design and implementation of language teaching and learning for 21st century skills development. In order to fill this gap in the literature, the present study was carried out, and the following research questions were addressed:

• What language skills and 21st century skills did the researchers focus on in the reviewed studies?
• What theories were used as a foundation in reviewed studies?
• What technologies were used to promote language skills and 21st century skills?
• What learning activities were used in the reviewed studies?
• What were the methodological characteristics of the reviewed studies?
• What research findings were obtained in the reviewed studies?

Research Method

The present study is a systematic review. The study used preferred reporting items for systematic reviews and meta-analyses (PRISMA) for the electronic search. PRISMA is considered as a set of programs that facilitates researchers to prepare and report various systematic evaluations and meta-analyses (Moher et al., 2009 ). According to scholars, PRISMA has been widely and successfully applied in educational research. In addition to PRISMA, this review followed the general guidelines for searching and selecting research articles proposed by Avgousti ( 2018 ), Shadiev and Yang ( 2020 ), and Shadiev and Yu ( In Press ). The search and selection processes are shown in Figure 1 . Articles were found through a search on the Web of Science database and Peer-Reviewed Instructional Materials Online Database (PRIMO). According to Kukulska-Hulme and Viberg ( 2018 ), PRIMO is a search tool and it contains several databases such as ERIC and Scopus. For this reason, PRIMO features a very comprehensive collection of full-text articles and bibliographic records, and it has been used by many researchers in their systematic reviews and meta-analyses to find relevant literature.

The search and selection process.

Based on general recommendations from previous review studies (Guan, 2014 ; Duman et al., 2015 ), this review used keywords such as 21st century skills, language learn * , and technology. 21st century skills were also included to widen the search results (e.g., creativity and innovation, critical thinking, problem solving, communication, collaboration, digital literacy, information literacy, media literacy, ICT literacy, flexibility and adaptability, initiative and self-direction, social and cross-cultural interaction, productivity and accountability, leadership and responsibility). This review used these terms in different combinations to search articles.

A total of 9,162 articles were found from the search. This review narrowed down the selection of research articles based on the following criteria (see Figure 1 ): articles that were (1) published during 2011–2022 (February); (2) published in English; and (3) focused on technology-supported language learning and 21st century skills. Two researchers screened each article individually and excluded articles from the study that did not focus on technology-supported language learning and 21st century skills. The researchers discussed any discrepancies in their selection results until an agreement was reached. At the end of the selection process, 34 empirical studies were chosen for the review.

This review proposed an analytical framework (see Figure 2 ) to answer the research questions of the study and to better understand the research design of the reviewed studies and findings. This review also used this framework to help us better review articles and regarded it as the basis for coding the content of reviewed studies. This review used the open coding method to carry out content analysis (Creswell, 2002 ) which can enable us to segment research content and to form categories relevant to the phenomenon of interest. The analytical dimensions included the following (see Figure 2 ): (1) language skills and 21st century skills—skills related to language learning and 21st century skills, (2) technology—the tools and devices participants used for language learning, (3) learning activities supported by technology to cultivate 21st century skills and language skills, (4) theoretical foundation—theories, models or hypothesis involved in research, (5) methodology—participants' academic level and major, research duration, sample size, data collection tools, and research design, and (6) findings—results reported in research.

Analytical framework.

Two researchers were involved in the coding process. They read articles and coded content according to the above coding scheme. After that, they categorized codes into categories and identified attributes for each category. If there were any differences in coding, the researchers re-examined an article to resolve differences, and then finally completed the coding phase. Interrater reliability was measured using Cohen's kappa coefficient and the result was high (k = 0.886).

The present study starts this section with the results related to publication year, languages, and participants. Figure 3 shows the distribution of articles published in the past 10 years. Most studies were published in 2019 ( n = 8), and no articles were published in 2012. From the figure, it can also be seen that the research trend in this field is on the rise. Figure 4 demonstrates the frequency at which different languages were the focus in the reviewed studies. 29 studies focused on English. There were also studies focused on Chinese ( n = 2), Ukrainian ( n = 1), Japanese ( n = 1), and Spanish ( n = 1). As shown in Figure 5 , undergraduates were the most common academic level ( n = 17), and there was a relatively low number of studies conducted on junior high school ( n = 5), senior high school ( n = 2), and primary school ( n = 1) academic levels. As shown in Figure 6 , researchers were more willing to involve students who were majoring in the fields of education ( n = 9), management ( n = 4), or engineering ( n = 4).

Distribution of research year.

Distribution of languages.

Distribution of educational level.

Distribution of participants' major.

Research Focus

This section presents the results related to research focus of reviewed articles. As can be seen from Figure 7 (and from Appendix 1 ), researchers carried out technology-assisted language learning studies and focused on the development of listening, speaking, reading, writing, grammar, and vocabulary skills. Among these skills, speaking skills ( n = 20) received considerable attention from researchers, followed by writing skills ( n = 19) and vocabulary ( n = 13). Reading ( n = 5) skills received less interest from researchers.

Distribution of language skills.

According to Figure 8 (and Appendix 1 ), researchers pointed out that technology-supported language learning can also promote 21st century skills. These skills relate to the following three categories: 4C (communication, collaboration, critical thinking, and creativity), digital literacy, and career and life skills. The most common skills that scholars targeted were communication ( n = 15) and collaboration ( n = 15), followed by critical thinking ( n = 10) and social and cross-cultural interaction ( n = 10). Problem solving ( n = 5) skills have received the least attention from researchers.

Distribution of 21st century skills.

Theoretical Foundation

This section focuses on theoretical foundation in the reviewed articles. As shown in Appendix 2 , a total of 16 theories were used. The most used theory was the social constructivism theory ( n = 9), followed by Byram's intercultural competence model ( n = 3), project-based learning ( n = 2), content based instruction ( n = 2), task based approach to language teaching ( n = 2), and sociocultural theory ( n = 2). The rest of theories were used only once.

As shown in Appendix 3 , a total of 52 technologies were used in reviewed studies. This review grouped them into eight categories: Social tools ( n = 20), Creative tools ( n = 19), Collaboration tools ( n = 13), Learning management system ( n = 9), Multimedia materials ( n = 5), Classroom interactive tools ( n = 4), Presentation tools ( n = 2), Wearable devices ( n = 1). Among the most commonly used technologies were Facebook ( n = 4), Google Docs ( n = 4), Moodle ( n = 4), followed sequentially by Skype ( n = 3), Padlet ( n = 3), WhatsApp ( n = 2), YouTube ( n = 2), Blogs ( n = 2), Google Drive ( n = 2), and Wiki ( n = 2). The other 40 technologies have only been used once, i.e., Windows Movie Maker, Live On, Edmodo, Kahoot, and Prezi. In addition, one study involved a virtual reality production tool (EduVenture) and a wearable device (Google Cardboard).

Learning Activity

As shown in Appendix 4 , in reviewed studies, scholars designed the following five main types of learning activities: (1) collaborative task-based language learning ( n = 9); (2) learning activities based on online communication ( n = 9); (3) creative work-based language learning ( n = 8); (4) adaptive learning activities ( n = 4); and (5) learning activities based on multimedia materials ( n = 4).

Methodology

This section presents methodological details of reviewed studies, such as sample size, research duration, data collection tools and research design.

As shown in Figure 9 , the most common sample size was from 11 to 30 participants ( n = 11), followed by sample sizes between 61 and 90 ( n = 8) and between 31 and 7 ( n = 7). Only two studies selected a sample size between 1 and 10. The sample size of two studies was >151. As shown in Figure 10 , most of research duration was between 3 and 6 months ( n = 10). There were 12 studies that did not state any research duration.

Sample size distribution.

Research duration distribution.

As shown in Appendix 5 , the most common data collection method was questionnaires ( n = 17), followed by tests ( n = 15) and interviews ( n = 13). Two data collection methods were used only 2 times, they were scales ( n = 2) and rubric ( n = 2).

As shown in Appendix 6 , research designs related to technology-supported language learning and 21st century skills were categorized into three main categories, namely quasi-experimental research ( n = 14), case studies ( n = 12), and action research ( n = 8).

As shown in Appendix 7 , various findings were reported in reviewed studies. In addition, that learners' language skills acquisition and 21st century skills, technology-supported language learning activities provided learners with good learning experiences, enhanced motivation and engagement, and improved self-confidence. In reviewed studies, some scholars reported about challenges faced by students during learning activities; they included challenges from technology, from their own competence, challenges of collaborating with others and self-attitude.

Language Skills

Regarding language skills, researchers have focused on improving learners' speaking, writing and vocabulary skills more. This shows that researchers are more concerned with the improvement of learners' skills related to language output. Researchers who reviewed studies on technology-supported language learning from 2014 to 2019 came to the same conclusion (Shadiev and Yang, 2020 ). However, the present study showed that reading skills received the least attention, while previous studies noted that grammar skills received less attention. This revealed that researchers are now beginning to pay more attention to previously neglected skills and are beginning to focus on the role of technology-supported language learning in facilitating learners' grammar skills. For example, Lai ( 2017 ) noted that grammar skills improved when learners completed activities to create vocabulary lists and greeting cards using multimedia resources. Jung et al. ( 2019 ) noted that students' grammar skills improved as they corrected each other's pronunciation and grammatical errors through video chat. Jamalai and Krish ( 2021 ) found that students' grammar skills improved through online forum discussions and knowledge sharing.

21st Century Skills

In terms of 21st century skills, communication and collaboration have received the most attention from researchers. It is probably because the 21st century society is more globalized and along with the increased complexity of related work, interpersonal communication and cooperation are being enhanced. The 21st century society emphasizes teamwork skills, and therefore scholars focus on collaborative and communication skills. Problem-solving skills have received little attention, and no researcher focused on career and life skills. In the face of the evolving and changing society of the future, problem-solving skills are among the core 21st century skills, emphasizing learners' ability to define problems, think critically, and solve problems. For example, scholars in reviewed studies have focused on learners' problem-solving skills in virtual technology-supported language learning (Chen et al., 2021 ).

Based on the results, this study has several recommendations for educators and researchers. First, input skills are an important component of language skills and an indispensable way for learners to develop output skills (Harmer, 2007 ). The present study suggests that researchers can focus on learners' input skills supported by technology, such as listening and reading. Second, problem-solving skills and career and life skills also deserve attention; therefore, future studies try to explore the effects of technology-supported language learning on these skills.

Theories Related to Instructional Design

The most commonly used instructional design theory in reviewed studies was social constructivism theory. The results of this research are consistent with those of previous review studies of technology-supported language learning (Parmaxi and Zaphiris, 2017 ). According to this theory (Vygotsky, 1978 ), knowledge is not a set of “facts” but rather a synthesis of information that is actively constructed and evolving in the learner's mind. The teacher does not “give” knowledge to the learner, but the learner should acquire knowledge actively. Learners' knowledge evolves as they process old and new information, as well as their experiences. The researchers designed collaborative, creative, and communicative activities based on a social constructivism perspective to encourage learners to input the target language and output the target language in a meaningful context. At the same time, researchers have used various learning and teaching activities to promote students' collaboration, communication, creativity, critical thinking and digital literacy skills (Yang et al., 2013 , 2014 , 2022 ; Lai, 2017 ; Sevilla-Pavón and Nicolaou, 2017 ; Huang, 2021 ).

Other researchers have also used theories based on learner-centered pedagogies such as problem-based or project-based theories. These pedagogies are all used to promote student-directed learning, adaptive learning, and personalized assessment. Learning theories were used to design activities that provided learners with opportunities for language input and output, e.g., to learn new knowledge and then apply it to the real world by creating own content. This allows learners to acquire language skills and develop 21st century skills such as communication, collaboration, and problem solving (Arnó-Macià and Rueda-Ramos, 2011 ; Yang et al., 2013 , 2014 ; Srebnaja and Stavicka, 2018 ).

Theories Related to Language Learning

Researchers have also designed learning activities based on theories related to foreign language learning, such as task-based language teaching, content-based instruction, and output-input theory. For example, digital story creation activities and integrated cross-cultural communication activities designed by the researcher are in line with these theoretical perspectives, in which learners have access to the target language through social tools and partner communication. The ability to use creative and collaborative tools to complete target-language based tasks also contributes to the acquisition of language skills and 21st century skills development, such as social and cross-cultural interaction, communication skills (Lewis and Schneider, 2015 ; Tseng, 2017 ).

Theories Related to Measuring Learning Outcomes

Since language learning is closely related to culture, scholars have designed foreign language courses based on cross-cultural communication, where learners acquired both language skills and cultural knowledge. Further, there are theories that have been used by scholars to assess and measure learners' outcomes. For example, researchers have focused on learners' intercultural competence along with their language skills and utilized the Byram' ICC model and the developmental model of intercultural sensitivity to measure their cross-cultural knowledge acquisition and skills development (Bennett, 1986 ; Byram, 1997 ). In addition, the Keller' ARCS motivational model (Keller, 1987 ) has been used by researchers to measure learners' perceived attention, relevance, confidence, and satisfaction in technology-supported language learning environments.

This review analyzed the theoretical foundation that was used by those few studies that focused on non-English languages such as Chinese, Ukrainian, Japanese, and Spanish. This review found that learning theories used by scholars in these studies were diverse. They were related to instructional design (e.g., social constructivism), language learning (e.g., language output and input), and cross-cultural learning (e.g., intercultural sensitivity).

Based on the findings, several suggestions for educators and researchers are proposed. First, the theories mentioned by researchers are instruction-related theories, language learning-related theories, and measurement-related theories; they were used to guide the design of technology-supported language learning activities that focus both on the acquisition of language skills and on the 21st century skills. These theories can be useful to inform the design of appropriate language learning activities for educators and researchers in the future. Second, this review found that many researchers did not indicate what theories were used in their studies. Theoretical foundations are important for the instructional design, language learning or measuring activities, so such information should be clearly indicated so that other researchers can gain a deeper understanding of them.

Eight Technologies With Different Functions

Based on the literature review, this study grouped technologies into eight categories based on their functions: (1) collaborative tools (e.g., Google Docs or Padlet) for supporting learners to collaborate on a task through co-editing and information sharing; (2) social tools (e.g., Facebook or Skype) for supporting learners to communicate and share content remotely or synchronously using text, audio and video; (3) creative tools (e.g., Photo Story or Adobe Spark) to support learners in creating work, such as digital stories or videos; (4) learning management system (e.g., Moodle) to integrate learning activities and learning resources for adaptive online learning; (5) classroom interaction tools (e.g., Quizlet or Kahoot) to support question-answering, polling, and other activities in the classroom; (6) multimedia materials are some audio and video resources on the web or multimedia textbooks; (7) presentation tools (e.g., PowerPoint) are used to support learners to present their learning content digitally; (8) wearable devices (e.g., Google Glass) to support learners to view or interact with content in virtual reality learning environments.

Most Commonly Used Technologies

Facebook (social tool), Google Docs (collaboration tool), and Moodle technologies (learning management system) were used the most in previous studies to facilitate language and 21st century skills. The study further analyzed which technologies are most often used by researchers to promote 21st century skills. Appendix 8 demonstrates these most commonly used tools. The study found that Facebook (social tool), Google Docs (collaboration tool) and Moodle (learning management system) were also the tools most often used by researchers to promote communication, collaboration and critical thinking, social and cross-cultural interaction skills. This indicates that scholars valued such 21st century skills as collaboration and communication among students in technology-supported language learning activities. For example, Sevy-Biloon and Chroman ( 2019 ) used social and collaborative tools (e.g., Google Docs, Facebook, etc.) to support communication between students from different cultural backgrounds and their results showed that students' speaking skills, social and cross-cultural interaction, and communication skills were promoted. Moodle is popular among researchers because this learning management system not only supports learners' adaptive and inquiry-based learning, but also helps teachers share learning resources with learners, design learning activities, and manage learners' learning progress (García-Sánchez and Burbules, 2016 ). For example, Yang et al. ( 2014 ) designed a language learning activity based on the Moodle platform that asked students to complete reading and writing tasks in the system to promote the development of reading, writing skills and critical thinking. In addition, researchers most often used Google Docs (collaboration tool), Prezi (presentation tool), Windows Movie Maker, Photo Story3 (creative tools) and Blogs (social tool) to support students' creativity and innovation skills, problem-solving skills, and ICT literacy. And only two studies have used films (multimedia materials) and blogs (social tool) to support students' media literacy.

Experienced Challenges of Using Technology

Scholars reported that technologies pose some challenges for learners. For example, students were not experienced to use technology and had no trainings before learning activities; then they complained about problems to use technology during learning (Lai, 2017 ). Students were also confused about the layout of the platform and noted that there were incompatibilities and connectivity issues with learning devices (Hosseinpour et al., 2019 ). When communicating remotely, students pointed out that there were problems with the network and they were not able to connect and participate in learning process (Mohamadi Zenouzagh, 2018 ; Jung et al., 2019 ).

The Distribution of Technology in non-English Language Studies and Different Theories

This review also analyzed technologies that were used by those few studies that focused on non-English languages. This review found that, in general, scholars in these studies used such technologies as creative tools (Adobe Spark), collaboration tools (Google Docs), and social tool (Facebook) to present multimedia content to learners and support collaborative, creative and communicative learning activities (Valdebenito and Chen, 2019 ).

With regard to the distribution of technology in theory. Social constructivism theory was the most commonly cited theory in reviewed research and scholars used various technologies such as learning management systems (e.g., Moodle), creative tools (e.g., iMovie) or social tools (e.g., Facebook) to support constructivism-based learning activities. That is, interactive and collaborative learning activities were designed for students to learn new knowledge and then apply it to construct meaning in authentic contexts.

Based on the results of this study, several recommendations for educators and researchers were proposed. First, it is recommended that learning activities supported by technologies are designed based on appropriate theoretical foundation. Second, teachers are encouraged to conduct appropriate technology training for students beforehand so that they become familiar with technology tools. Third, teachers and researchers should test learning tools with students in advance in order to identify any possible technical problems, and provide timely support during learning process.

Learning Activities Used to Promote Language Skills and 21st Century Skills

This section describes what technologies are used in each type of learning activity and how they contribute to the development of learners' language skills and 21st century skills. In addition, it offers relevant suggestions to researchers and educators.

Adaptive Language Learning Activities on Learning Platforms

As shown in Table 1 , in the reviewed study, researchers used the following tools: Moodle, Google classroom, Quantum leap, and WebQuest, to develop adaptive language learning activities on learning platforms. These tools are used to integrate different types of instructional resources and diverse language learning activities to provide learners with adaptive learning materials that meet their learning needs. Students can ask questions and receive feedback from other students or teachers, and take control of their own learning progress.

Adaptive language learning activities on learning platforms.

For example, Arnó-Macià and Rueda-Ramos ( 2011 ) designed tasks for reading, listening, and speaking practice in Quantum leap platform. Researchers have designed listening tasks in Moodle platform; students were required to analyze, evaluate, and summarize content after listening (Yang et al., 2013 , 2014 ). Srebnaja and Stavicka ( 2018 ) designed WebQuests-based speaking and writing tasks.

All of these studies noted that learners' performance in speaking, listening, reading, writing, and grammar improved after completing the computer-assisted adaptive language learning tasks. In addition, students' critical thinking skills were developed.

Collaborative Task-Based Language Learning Activities

As shown in Table 2 , the following tools were used by researchers for the development of collaborative-based language learning activities: (1) collaboration tools: Google Docs, Google Drive, Wiki, Edmodo, and E-writing forum. These collaborative tools have the following functions: sharing, collaborative editing, cloud storage, synchronized display, and help students freely share information in various formats (e.g., text, images, videos, web links, audio recordings, music, etc.) on the platform so that they can exchange ideas and collaborate on editing content; (2) creative tools: Adobe Spark, to support students' expression of ideas; (3) social tools: Blogs or WordPress, to support students in reading and commenting on each other's work.

Collaboration-based language learning activities.

Collaboration-based language learning activities are those in which students work in groups to solve problems and complete tasks proposed by the teacher, such as asking students to provide an essay or present their ideas in other ways (e.g., a solution, a report, and a performance). For example, Amir et al. ( 2011 ) asked students to work in groups to publish six articles based on different topics over the course of 14 weeks, and one of the tasks required students to find and discuss software about computer-assisted writing.

Mohamadi Zenouzagh ( 2018 ) designed a collaborative writing activity based on the E-writing platform. Valdebenito and Chen ( 2019 ) designed a collaborative activity on the theme of “food and culture” in which students first had to use Google Maps to identify geographic areas related to the content, then use a Google Doc to record their ideas, and finally use video production tools such as Adobe Spark to express their ideas and share them on the WordPress platform. Huh and Lee ( 2020 ) designed a creative learning English collaborative activity in which students first used a mobile app to learn how to spell words, then the group took the words they learned and expressed them through the role play and song. Lai ( 2017 ) designed different collaborative tasks, for example, students needed to use the ThingLink tool to create vocabulary lists and greeting cards related to the topic, which were then shared on the Padlet platform and discussed. In addition, students were required to use HomeStyler to collaboratively design a dream home and use some vocabulary related to “location” to describe the design of their home.

Girgin and Cabaroglu ( 2021 ) designed an English learning project that integrates Web 2.0 technology and flipped classroom, and students used Padlet to watch videos in class. In grammar classes, students used Kahoot, Quizlet, Quizizz, Animoto, Powtoon, and Poster MyWall to answer grammar questions. In vocabulary and reading classes, students used tools such as Mind Meister, Voki, Canva, Cram, Go Animate and Story-bird to create mind maps, as well as create digital stories, which can be presented and shared. Chen et al. ( 2021 ) used virtual reality technology to design language learning activities. Learners were required to first watch a virtual reality scene and think about how to solve the problem based on a series of guiding questions provided by the teacher. Then students role-played in English to create a virtual reality video of the problem being solved.

The results of the abovementioned studies showed that collaborative-based language learning activities facilitated the development of learners' language skills. The researchers noted that collaborative problem-solving language learning activities provided learners with a large number of writing tasks, such as writing reports, essays, or creating storylines and designing works. The process of sharing with each other enabled to point out grammatical errors (Amir et al., 2011 ; Mohamadi Zenouzagh, 2018 ; Hosseinpour et al., 2019 ). When learners used multimedia resources to create vocabulary lists and greeting cards, their vocabulary and grammar skills were also improved (Lai, 2017 ).

At the same time, students' critical thinking was developed as they gave each other's critical and constructive comments (Valdebenito and Chen, 2019 ; Zou and Xie, 2019 ; Girgin and Cabaroglu, 2021 ). In addition, students completed tasks in small groups which promoted the development of communication and collaboration skills during discussions with each other (Amir et al., 2011 ; García-Sánchez and Burbules, 2016 ; Lai, 2017 ; Mohamadi Zenouzagh, 2018 ; Hosseinpour et al., 2019 ; Zou and Xie, 2019 ; Girgin and Cabaroglu, 2021 ). The process of students voicing digital content promoted the development of speaking skills (Huh and Lee, 2020 ). In the process of creating digital works, digital literacy was developed (García-Sánchez and Burbules, 2016 ; Valdebenito and Chen, 2019 ). Chen et al. ( 2021 ) pointed out that learners learn contextually in an immersive learning environment, and solving real problems through virtual reality technology improved learners' vocabulary as well as promoted their problem-solving skills.

Creative Work-Based Language Learning Activities

As shown in Table 3 , in reviewed studies, language learning activities based on creative works consisted of two main categories: creating digital stories or videos. The main models for this type of learning activity were as follows: students communicated in groups about how to create a digital story or video, then collected and processed relevant information, after that created a digital story, and finally shared content and communicated with each other about it.

Creative work-based language learning activities.

The researchers chose different tools to support such learning process, e.g., (1) creating digital stories, i.e., Photo Story3, Windows Movie Maker, or iMovie; (2) creating video scripts in collaboration, i.e., Google Docs or Google Drive; (3) presenting digital stories, i.e., Prezi or PPT; (4) sharing digital stories and communicating, i.e., Google+ forums, Facebook, Instagram, WhatsApp, Google Classroom, and classroom management systems.

The researcher noted that digital storytelling promoted language skills, specifically, the process of writing story scripts promoted students' writing and vocabulary skills (Thang et al., 2014 ; Sevilla-Pavón and Nicolaou, 2017 ; Kulsiri, 2018 ; Yalçin and Öztürk, 2019 ; Chiang, 2020 ). It also promoted 21st century skills. Researchers mentioned three approaches for creating digital stories or videos such as free-writing, rewriting the ending of the story, and specifying the theme, and in this open-ended work creation process, students' sense of creativity, problem-solving skills, and digital literacy were developed (Thang et al., 2014 ; Sevilla-Pavón and Nicolaou, 2017 ; Kulsiri, 2018 ; Yalçin and Öztürk, 2019 ; Yang et al., 2022 ). Regarding the creation of digital stories on a specific theme, the researcher asked learners to design a new country, and students needed to understand a range of elements including different countries and cultures, such as national characteristics, language, national policies, climate and life. As a result, students' social and cross-cultural skills were improved. In addition, critical thinking was facilitated as students developed different ideas and perspectives as they evaluated each other's digital stories (Sevilla-Pavón and Nicolaou, 2017 ). Finally, students developed their communication and collaboration skills when working in groups (Thang et al., 2014 ; Sevilla-Pavón and Nicolaou, 2017 ; Kulsiri, 2018 ; Yalçin and Öztürk, 2019 ; Mirza, 2020 ; Huang, 2021 ).

Language Learning Activities Based on Multimedia Learning Materials

As shown in Table 4 , language learning activities based on multimedia materials involved such tools as (1) web-based learning management system, e.g., EDpuzzle; (2) social tool, e.g., YouTube; and (3) multimedia textbooks. All of them provided multimedia resources for students. There were also (4) collaboration tools, e.g., Padlet and Google docs, which supported learners to share ideas with each other.

Language learning activities based on learning multimedia materials.

Scholars have designed a variety of language learning activities based on multimedia materials, but the topics and learning tasks of the multimedia materials involved in these studies differed. For example, Tseng ( 2017 ) asked learners to watch a video on the topic of cultural differences, and then students gave oral presentations and reflections to present their views on cultural differences. Zou and Xie ( 2019 ) asked students to watch a video on EDpuzzle, then to discuss in groups, negotiate and compare answers, to share their output to the Padlet platform, and finally submit their reports in Google docs. Nikitova et al. ( 2020 ) asked students to watch videos from multimedia textbooks with different English contexts and then simulated learners' role play activities. Aristizábal-Jiménez ( 2020 ) asked learners to watch YouTube videos, analyze the structure and content of video content, and then create posters to present and share their ideas.

The researcher noted that language learning activities based on multimedia materials promoted learners' language skills and 21st century skills. Specifically, learners' listening skills were promoted after watching the videos (Tseng, 2017 ). Culturally relevant content in videos and culture-based communication among peers promoted students' social and cross-cultural interaction skills (Tseng, 2017 ). Learners actively used dictionaries and discussed grammar while completing tasks to make the information easier to understand, which also promoted students' vocabulary and grammar skills (Aristizábal-Jiménez, 2020 ). In addition, working in groups to complete tasks promoted speaking, writing, grammar, and vocabulary skills. This was also beneficial to develop students' problem solving, collaboration, critical thinking, and communication skills (Aristizábal-Jiménez, 2020 ; Nikitova et al., 2020 ).

Language Learning Activities Based on Online Communication

As shown in Table 5 , the researchers designed online communication-based language learning activities. Most of them were cross-cultural communication activities to support cross-cultural communication between students from different cultural backgrounds. In terms of technology, the researchers mainly used social tools to support textual or video communication, e.g., Facebook, Skype, and WhatsApp. In addition, researchers have utilized learning management systems to support students to view learning resources uploaded by teachers.

Language learning activities based on online communication.

The design of cross-cultural communication activities followed the same pattern—exposure to cross-cultural knowledge, reflection on cross-cultural differences, and cross-cultural exchange. For example, Calogerakou and Vlachos ( 2011 ) had students from two countries to watch movies and compare culture presented in movies with their own culture. Then students had to post comments on a blog and discuss their ideas. Chen and Yang ( 2016 ) asked students to share culturally specific folklore stories with their partners and to make videos of the stories to send to their partners. In addition, students were asked to perform a puppet show via videoconference. All of these were for students to learn about cultural similarities and differences. Chen and Yang ( 2014 ) designed a discussion activity based on cultural themes; for example, students discussed movies that involved culturally different content, and then students shared their opinions on Wiki. Lewis and Schneider ( 2015 ) asked learners to interact with native Spanish-speaking students and discuss cultural topics such as “local living conditions” and “how to celebrate holidays.” Learners were then asked to write a mini-biography or travel brochure for their study partner to demonstrate the cultural knowledge they gained during the exchange. Özdemir ( 2017 ) asked students to watch YouTube videos and discuss them based on cross-cultural questions prepared by the teacher. Sevy-Biloon and Chroman ( 2019 ) designed an intercultural exchange program in which students from Ecuador and the United States were randomly paired and then engaged in a cultural exchange based on the theme of the language course. Jung et al. ( 2019 ) asked students from different cultural backgrounds to discuss cultural topics, including “happiness factors, family, and food,” and finally, students reflected on the discussion, exchanged proverbs with each other, and then presented cultural differences. They reflected on their experiences in a reflective journal. Jamalai and Krish ( 2021 ) designed an online discussion activity, in which learners were required to engage in online discussions based on topics posted by teachers in a forum.

The results showed that students' speaking, vocabulary, writing, reading, and grammar skills improved when communicating through text and speech because students double-checked vocabulary spelling and grammar. Students identified errors they made when communicating using text and speech and corrected them to ensure that others understood their intended meaning (Calogerakou and Vlachos, 2011 ; Chen and Yang, 2014 , 2016 ; Lewis and Schneider, 2015 ; Özdemir, 2017 ; Hirotani and Fujii, 2019 ; Jung et al., 2019 ; Sevy-Biloon and Chroman, 2019 ; Jamalai and Krish, 2021 ). In addition, students' listening skills improved after watching YouTube videos (Özdemir, 2017 ).

At the same time, students' communication process using social tools developed the ability to use writing software, electronic dictionaries, and collect information on the Internet, and therefore media literacy was improved (Calogerakou and Vlachos, 2011 ). All studies point to the development of cultural interaction skills after students interacted and exchanged different cultural perspectives with partners (Calogerakou and Vlachos, 2011 ; Chen and Yang, 2014 , 2016 ; Lewis and Schneider, 2015 ; Özdemir, 2017 ; Hirotani and Fujii, 2019 ; Jung et al., 2019 ; Sevy-Biloon and Chroman, 2019 ). Communication (Chen and Yang, 2014 ; Lewis and Schneider, 2015 ; Hirotani and Fujii, 2019 ) and collaboration skills were also developed (Chen and Yang, 2014 ) in reviewed studies.

This review also analyzed learning activities that were used by those few studies that focused on non-English languages. This review found that most learning activities designed in these studies were online cross-cultural communicative activities. This shows that the primary goal of these learning projects was to develop students' foreign language and intercultural communication skills.

Based on the findings of the reviewed literature, the five types of language learning activities supported by technology had a positive impact on students' language skills as well as their 21st century skills development. Moreover, this review found that these learning activities followed similar pattern. The common pattern for language learning activities based on culture-related communication was exposure to cross-cultural knowledge, reflection on cross-cultural differences, and cross-cultural exchange. The common pattern of language learning activities for creative works was as follows: students communicated in groups about how to create a work (such as digital story or video), then collected and processed relevant information, created a work, and then shared content and communicated with each other about it. These patterns could provide suggestions for researchers and teachers to design similar instructional activities that target development of language skills and 21st century skills in the future.

Second, this review found that researchers designed similar instructional activities, but the research focus was different. For example, in the adaptive language learning activities on learning platforms, researchers focused on the development of students' speaking skills and lacked attention to reading skills. And in the collaborative task-based language learning activities, researchers have focused more on writing and vocabulary skills, collaboration, and communication skills, and lacked attention to listening skills. In creative writing-based language learning activities, researchers focused more on speaking and writing skills as well as creative and communication skills.

Research Duration, Participants' Academic Level, and Sample Size

The most common study samples were small ones with participants range from 11 to 30 ( n = 11) and medium samples with range between 61 and 90 ( n = 8) participants. Research durations were mostly between 3 and 6 months ( n = 10). Small sample size was acknowledged as a limitation in some studies (Hirotani and Fujii, 2019 ; Zou and Xie, 2019 ). The possible reason for this is that most of the studies were based on small classroom settings. In the reviewed studies, the most common academic level of participants was undergraduate level. There were 12 studies that did not specify research duration. Regarding this finding, there is a lack of attention in previous retrospective studies (Guan, 2014 ; Duman et al., 2015 ; Persson and Nouri, 2018 ).

Data Collection

Most of the studies collected both quantitative and qualitative data, which can help researchers to draw conclusions from different perspectives. Quantitative data included tests, scales, and rubrics; qualitative data included student's work, open-ended questions, student feedback, interviews, student chat transcripts, student reflections, teacher journals, and observations. One of the most common forms of quantitative data collection is a test ( n = 15), involving student language tests (tests of English speaking and listening) and tests of 21st century skills (critical thinking and creative thinking). The most common method of qualitative data collection was interview ( n = 13), where the researcher usually designed an interview outline and then asked learners questions to understand their learning experiences, attitudes, motivations, and challenges in the learning process. In addition, researchers have extensively used questionnaires ( n = 17), including both closed-ended and open-ended questions, to collect both quantitative and qualitative data. For example, the researchers used questionnaires to investigate learners' perceptions of technology-supported language learning, including effectiveness, usefulness, and students' perceptions of developing intercultural communicative competence and language skills through online discussions (Jung et al., 2019 ).

Based on the above findings, the recommendations of the present study for researchers and teachers are as follow. First, researchers could consider studies with longer time spans and collect data from bigger number of participants to investigate students' development over time and have generalizable conclusions. Second, researchers can collect multiple types of data, focus on students' learning processes and outcomes, and then interpret findings from different perspectives.

Research Design

There are a variety of research designs for reviewed studies on technology-supported language learning and 21st century skills. The most common are quasi-experimental studies. Such studies are characterized by using pre- and post-tests to measure changes in participants' language skills, 21st century skills and other learning outcomes and attitudes before and after participation in learning activities. In quasi-experimental studies, participants are not randomly assigned to an experimental or control group (Persson and Nouri, 2018 ; Huang, 2021 ). These findings are consistent with other reviews on technology-supported language learning (Persson and Nouri, 2018 ). The present study suggests that educators and researchers can use the three research methods mentioned above to validate their studies in future.

Positive Learning Experiences

In this section, the study discusses findings from reviewed studies and recommendations for educators and researchers. In reviewed studies, in addition to finding that technology-supported learning activities promoted learners' language skills and 21st century skills, researchers also found that these technologies led to positive learning experiences, which resulted in better learning outcomes. For example, learning through multimedia textbooks, collaborative blog-based writing activities, smartphone-based video filming activities and language learning projects based on intercultural exchange all increased students' motivation (Amir et al., 2011 ; García-Sánchez and Burbules, 2016 ; Sevy-Biloon and Chroman, 2019 ; Aristizábal-Jiménez, 2020 ; Huang, 2021 ). For example, Hosseinpour et al. ( 2019 ) noted that through collaborative writing activities, learners' motivation and self-confidence levels were increased. Mirza ( 2020 ) argued that through digital storytelling-based learning activities, students gained more confidence. Researchers have also looked at the different learning performance of students due to individual differences in abilities or their characteristics. Yang et al. ( 2014 ) found that in terms of writing, significant differences were found between “basic” and “low-intermediate” learners as a result of the difference in ability. Yalçin and Öztürk ( 2019 ) found that girls had a more negative attitude toward technology than boys.

Challenges Faced by Students

While many studies pointed to positive student attitudes toward technology-supported learning activities (Arnó-Macià and Rueda-Ramos, 2011 ; Girgin and Cabaroglu, 2021 ), several studies highlighted challenges that students faced when using technology for learning. Challenges from technology, with some learners finding it difficult to use in learning activities or being confused about the layout of mobile applications were mentioned. Students also noted problems with device incompatibility and poor network quality and speed when using technology. Self-competence challenges, with learners noting that learning tasks were difficult for them, for example, insufficient time to complete learning tasks, lack of research skills, or language skills needed to complete tasks, were reported. Difficulties in finding an interesting topic and choosing the right tools to create their work were also reported in reviewed studies. Challenges of collaborating with others, with some learners noting that they encounter uncoordinated teamwork, uneven distribution of work and unequal student contributions in collaborative tasks, were mentioned by scholars. Self-attitudes, as noted by learners who felt anxious about video chatting when they were communicating remotely, as well as fear of having their writing errors discovered by their partners when communicating in text, were reported in reviewed studies.

Based on the above findings, the present study recommends to educators and researchers, in addition to focusing on the impact of technology-supported learning activities on learners' language skills and 21st century skills, it is also important to focus on students' perceptions of technology, motivation, engagement, and confidence. This is because positive learning experiences can lead to better learning outcomes (Sevy-Biloon and Chroman, 2019 ; An et al., 2021 ). Regarding the technological challenges that students encounter in the learning process, it is recommended that they be addressed through advance trainings and through providing students with appropriate technological services during learning activities. Self-competence challenges can be addressed by designing collaborative tasks in which students with higher levels of competence can help students with lower levels of competence to complete the task. Regarding the challenges in collaborative activities, it is recommended that teachers and researchers design learning activities with clear rules for collaborative division of labor and rules regarding how learning performance of every learner will be evaluated. With regard to alleviating negative student attitudes, it is recommended that teachers design diverse teaching strategies and scaffolds to give students assistance during learning activities.

This study reviewed articles on technology-supported language learning and 21st century skills published from 2011 to 2022 (February) in terms of (a) research focus; (b) theoretical foundations; (c) technology; (d) learning activities; (e) methodology and (f) findings. The results indicate that research on technology-supported language learning and 21st century skills have shown an upward trend in the overall research in the covered time period, with most of the research focusing on English and the majority of participants in these studies majored in education.

Secondly, in terms of research focus, most of the researchers focused on learners' speaking skills (27.40%), followed by writing (26.03%) and vocabulary skills (17.81%). In terms of 21st century skills, most researchers focused on communication (20.83%), collaboration (20.83%), critical thinking (13.89%), and social and cross-cultural interaction skills (13.89%). In terms of theoretical foundations, social constructivist learning theory was most often adopted by researchers. In terms of technology, tools that support learners' creativity and socialization are often utilized by researchers, e.g., Facebook or Google Docs. In terms of learning activities, researchers have designed the following five types of learning activities to support learners' language learning and 21st century skills: (1) collaborative task-based language learning activities; (2) language learning activities based on online communication; (3) creative work-based language learning activities (4) adaptive language learning activities based on learning platforms; and (5) language learning activities based on multimedia learning materials. The results of reviewed studies indicate that these learning activities supported by technology are effective in promoting the development of learners' different language skills and 21st century skills. Finally, in terms of methodology, most of the studies had a sample of 11–30, the most common study period was 3–6 months, the data collection method often used by researchers was questionnaires, the most common method to collect quantitative data was tests, and the most common method to collect qualitative data was interviews.

In contrast to traditional paper and pencil-based learning, technologies used by researchers in reviewed studies allowed learners to improve language learning outcomes and 21st century skills through individual and collaborative learning activities. Some reported advantages are learning with technologies without the constraints of time and space, technologies enable personalized learning, technologies create authentic learning environments that provides adaptive learning content, helps create multimedia content actively, allows social interaction such as sharing, giving or receiving feedback, and reflecting on learning more efficiently.

Based on the above findings, recommendations for researchers and educators in this study include: (1) In terms of language skills, in addition to focusing on output skills, input skills (reading, listening) also deserve attention from researchers. In terms of 21st century skills, learners' problem-solving skills and career and life skills also need more attention from researchers in the future; (2) Advanced technology training for learners to familiarize them with technology and its effective usage as well as teachers need to check in advance for possible technology problems, such as network problems. These suggestions can help teachers address the technological barriers that learners encounter in the learning process; (3) The use of various theoretical approaches, such as instructional design-related theories and language learning-related theories, is important for the rational design of instructional activities that promote learners' language and 21st century skills; (4) Researchers and educators can follow the general model of conducting the five types of instructional activities summarized above to design instructional activities. In addition, it is recommended that researchers and educators use variety of technologies and design different instructional activities to promote learners' language and 21st century skills. It is also important to be aware of the challenges that students may encounter in terms of technology, learning activity tasks, peer collaboration and self-attitudes when implementing learning activities; (5) Teachers and educators could involve more participants and consider longer time spans in future studies to focus on more learners' development and to collect diverse quantitative and qualitative data to explain students' learning processes and outcomes.

There are few limitations to this study. Articles reviewed in this study were sourced from PRIMO and Web of Science databases, and some conference papers, books and dissertations were excluded. For this reason, this study reviewed smaller number of articles. Future studies may consider this limitation and address it by including more relevant sources.

Data Availability Statement

Author contributions.

RS and XW contributed to the conception, designed the work, collected the data, analyzed, and interpreted data. XW drafted the work and RS substantively revised it. RS was responsible for correspondence. All authors approved the submitted version and agreed both to be personally accountable for the author's own contributions and to the accuracy the work.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

1 Articles reviewed in this study.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2022.897689/full#supplementary-material

• * . Amir Z., Ismail K., Hussin S. (2011). Blogs in language learning: maximizing students' collaborative writing . Procedia Soc. Behav. Sci. 18 , 537–543. 10.1016/j.sbspro.2011.05.079 [ CrossRef ] [ Google Scholar ]
• An Z., Wang C., Li S., Gan Z., Li H. (2021). Technology-assisted self-regulated English language learning: associations with English language self-efficacy, English enjoyment, and learning outcomes . Front. Psychol. 11 :558466. 10.3389/fpsyg.2020.558466 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• * . Aristizábal-Jiménez Y. (2020). Fostering talk as performance in an EFL class through the critical analysis of youtubers' content . Profile Issues Teach. Professional Dev. 22 , 181–195. 10.15446/profile.v22n2.82510 [ CrossRef ] [ Google Scholar ]
• * . Arnó-Macià E., Rueda-Ramos C. (2011). Promoting reflection on science, technology, and society among engineering students through an EAP online learning environment . J. English Acad. Purposes 10 , 19–31. 10.1016/j.jeap.2010.12.004 [ CrossRef ] [ Google Scholar ]
• Asia Pacific Economic Cooperation (2004). 2004 APEC Ministerial Meeting. Available online at: https://www.apec.org/Meeting-Papers/Annual-Ministerial-Meetings/2004/2004_amm (accessed June 11, 2022).
• Avgousti M. I. (2018). Intercultural communicative competence and online exchanges: a systematic review . Computer Assisted Lang. Learn. 31 , 819–853. 10.1080/09588221.2018.1455713 [ CrossRef ] [ Google Scholar ]
• Bennett M. (1986). A developmental approach to training for intercultural sensitivity . Int. J. Intercultural Relat. 10 , 179–196. 10.1016/0147-1767(86)90005-2 [ CrossRef ] [ Google Scholar ]
• Byram M. (1997). Teaching and Assessing Intercultural Communicative Competence . Clevedon: Multilingual Matters. [ Google Scholar ]
• * . Calogerakou C., Vlachos K. (2011). Films and Blogs: an authentic approach to improve the writing skill-an intercultural project-based framework in the Senior High State School . Res. Papers Lang. Teach. Learn. 2 , 98–110. [ Google Scholar ]
• * . Chen C. H., Hung H. T., Yeh H. C. (2021). Virtual reality in problem-based learning contexts: effects on the problem-solving performance, vocabulary acquisition and motivation of English language learners . J. Computer Assisted Learn. 37 , 851–860. 10.1111/jcal.12528 [ CrossRef ] [ Google Scholar ]
• * . Chen J. J., Yang S. C. (2014). Fostering foreign language learning through technology-enhanced intercultural projects . Lang. Learn. Technol. 18 , 57–75. [ Google Scholar ]
• * . Chen J. J., Yang S. C. (2016). Promoting cross-cultural understanding and language use in research-oriented Internet-mediated intercultural exchange . Computer Assisted Lang. Learn. 29 , 262–288. 10.1080/09588221.2014.937441 [ CrossRef ] [ Google Scholar ]
• * . Chiang M. H. (2020). Exploring the effects of digital storytelling: a case study of adult L2 writers . IAFOR J. Educ. 8 , 65–82. 10.22492/ije.8.1.04 [ CrossRef ] [ Google Scholar ]
• Creswell J. W. (2002). Educational Research: Planning, Conducting, and Evaluating Quantitative. Upper Saddle River, NJ: Prentice Hall. [ Google Scholar ]
• Duman G., Orhon G., Gedik N. (2015). Research trends in mobile assisted language learning from 2000 to 2012 . ReCALL 27 , 197–216. 10.1017/S0958344014000287 [ CrossRef ] [ Google Scholar ]
• * . García-Sánchez M. S., Burbules N. C. (2016). Learning technologies and EFL teamwork . Revista de Lenguas para Fines Específicos 22 , 100–115. 10.20420/rlfe.2016.0092 [ CrossRef ] [ Google Scholar ]
• * . Girgin P., Cabaroglu N. (2021). Web 2.0 supported flipped learning model: EFL students' perceptions and motivation . Cukurova Univ. Faculty Educ. J. 50 , 858–876. 10.14812/cuefd.944217 [ CrossRef ] [ Google Scholar ]
• Goksu I., Ozkaya E., Gunduz A. (2020). The content analysis and bibliometric mapping of CALL journal . Computer Assisted Lang. Learn. 1–31. 10.1080/09588221.2020.1857409 [ CrossRef ] [ Google Scholar ]
• Guan S. (2014). Internet-based technology use in second language learning: a systematic review . Int. J. Cyber Behav. Psychol. Learn. 4 , 69–81. 10.4018/ijcbpl.2014100106 [ CrossRef ] [ Google Scholar ]
• Harmer J. (2007). The Practice of English Language Teaching . London: Longman. [ Google Scholar ]
• * . Hirotani M., Fujii K. (2019). Learning proverbs through telecollaboration with Japanese native speakers: facilitating L2 learners' intercultural communicative competence . Asian Pacific J. Second Foreign Lang. Educ. 4 , 1–22. 10.1186/s40862-019-0067-5 [ CrossRef ] [ Google Scholar ]
• * . Hosseinpour N., Biria R., Rezvani E. (2019). Promoting academic writing proficiency of Iranian EFL learners through blended learning . Turkish Online J. Distance Educ. 20 , 99–116. 10.17718/tojde.640525 [ CrossRef ] [ Google Scholar ]
• * . Huang H. W. (2021). Effects of smartphone-based collaborative vlog projects on EFL learners' speaking performance and learning engagement . Austral. J. Educ. Technol. 37 , 18–40. 10.14742/ajet.6623 [ CrossRef ] [ Google Scholar ]
• * . Huh K., Lee J. (2020). Fostering creativity and language skills of foreign language learners through SMART learning environments: evidence from fifth-grade Korean EFL learners . TESOL J. 11 , e489. 10.1002/tesj.489 [ CrossRef ] [ Google Scholar ]
• * . Jamalai M., Krish P. (2021). Fostering 21st century skills using an online discussion forum in an English for specific purpose course . Malaysian J. Learn. Instruct. 18 , 219–240. 10.32890/mjli2021.18.1.9 [ CrossRef ] [ Google Scholar ]
• * . Jung Y., Kim Y., Lee H., Cathey R., Carver J., Skalicky S. (2019). Learner perception of multimodal synchronous computer-mediated communication in foreign language classrooms . Lang. Teach. Res. 23 , 287–309. 10.1177/1362168817731910 [ CrossRef ] [ Google Scholar ]
• Keller J. M. (1987). Development and use of the ARCS model of instructional design . J. Instruct. Dev. 10 , 2–10. 10.1007/BF02905780 [ CrossRef ] [ Google Scholar ]
• Krashen S. D. (1985). The Input Hypothesis: Issues and Implications . London: Longman. [ Google Scholar ]
• Kukulska-Hulme A., Viberg O. (2018). Mobile collaborative language learning: state of the art . Br. J. Educ. Technol. 49 , 207–218. 10.1111/bjet.12580 [ CrossRef ] [ Google Scholar ]
• * . Kulsiri S. (2018). Students' perceptions of a student-produced video project in the General English language course at Srinakharinwirot University, Thailand . Arab World Eng. J. 4 , 40–54. 10.24093/awej/call4.4 [ CrossRef ] [ Google Scholar ]
• * . Lai A. (2017). Implementing online platforms to promote collaborative learning in Chinese language classrooms . J. Technol. Chin. Lang. Teach. 8 , 39–52. [ Google Scholar ]
• Lantolf J. (2000). Sociocultural Theory and Language Learning . Oxford: OUP. [ Google Scholar ]
• * . Lewis T. N., Schneider H. (2015). Integrating international video chat into the foreign language curriculum . Int. J. Comput. Assisted Lang. Learn. Teach. 5 , 72–84. 10.4018/IJCALLT.2015040105 [ CrossRef ] [ Google Scholar ]
• Lin L., Shadiev R., Hwang W. Y., Shen S. S. (2020). From knowledge and skills to digital works: an application of design thinking in the information technology course . Think. Skill Creat. 36 , 100646. 10.1016/j.tsc.2020.100646 [ CrossRef ] [ Google Scholar ]
• * . Mirza H. S. (2020). Improving university students' english proficiency with digital storytelling . Int. Online J. Educ. Teach. 7 , 84–94. [ Google Scholar ]
• * . Mohamadi Zenouzagh Z. (2018). Multidimensional analysis of efficacy of multimedia learning in development and sustained development of textuality in EFL writing performances . Educ. Information Technol. 23 , 2969–2989. 10.1007/s10639-018-9754-y [ CrossRef ] [ Google Scholar ]
• Moher D., Liberati A., Tetzlaff J., Altman D. G., PRISMA Group * (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement . Ann. Internal Med. 151 , 264–269. 10.7326/0003-4819-151-4-200908180-00135 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• * . Nikitova I., Kutova S., Shvets T., Pasichnyk O., Matsko V. (2020). “Flipped learning” methodology in professional training of future language teachers . Euro. J. Educ. Res. 9 , 19–31. 10.12973/eu-jer.9.1.19 [ CrossRef ] [ Google Scholar ]
• * . Özdemir E. (2017). Promoting EFL learners' intercultural communication effectiveness: a focus on Facebook . Comput. Assisted Lang. Learn. 30 , 510–528. 10.1080/09588221.2017.1325907 [ CrossRef ] [ Google Scholar ]
• Parmaxi A., Zaphiris P. (2017). Web 2.0 in Computer-Assisted Language Learning: a research synthesis and implications for instructional design and educational practice . Interact. Learn. Environ. 25 , 704–716. 10.1080/10494820.2016.1172243 [ CrossRef ] [ Google Scholar ]
• Partnership for 21st Century Skills (2008). P21 Framework Definitions Document . Available online at: http://www.21stcenturyskills.org (accessed February 28, 2022).
• Persson V., Nouri J. (2018). A systematic review of second language learning with mobile technologies . Int. J. Emerg. Technol. Learn. 13 , 188–210. 10.3991/ijet.v13i02.8094 [ CrossRef ] [ Google Scholar ]
• * . Sevilla-Pavón A., Nicolaou A. (2017). Online intercultural exchanges through digital storytelling . Int. J. Comput. Assisted Lang. Learn. Teach. 7 , 44–58. 10.4018/IJCALLT.2017100104 [ CrossRef ] [ Google Scholar ]
• * . Sevy-Biloon J., Chroman T. (2019). Authentic use of technology to improve EFL communication and motivation through international language exchange video chat . Teach. English Technol. 19 , 44–58. [ Google Scholar ]
• Shadiev R., Hwang W.-Y., Ghinea G. (2022a). Guest editorial: Creative learning in authentic contexts with advanced educational technologies . Educ. Technol. Soc. 25 , 76–79. Available online at: https://www.jstor.org/stable/48660125 [ Google Scholar ]
• Shadiev R., Hwang W. Y., Huang Y. M. (2017). Review of research on mobile language learning in authentic environments . Comput. Assist. Lang. Learn. 30 , 284–303. 10.1080/09588221.2017.1308383 [ CrossRef ] [ Google Scholar ]
• Shadiev R., Wang X., Liu T.Y., Yang M. (In Press). Improving students' creativity in familiar versus unfamiliar mobile-assisted language learning environments. Interact. Learn. Environ. 10.1080/10494820.2021.2023891 [ CrossRef ] [ Google Scholar ]
• Shadiev R., Yang M. (2020). Review of studies on technology-enhanced language learning and teaching . Sustainability. 12 , 524. 10.3390/su12020524 [ CrossRef ] [ Google Scholar ]
• Shadiev R., Yi S., Dang C. Sintawati W. (2022b). Facilitating students' creativity, innovation and entrepreneurship in a telecollaborative project . Front. Psychol. 13 , 887620. 10.3389/fpsyg.2022.887620 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Shadiev R., Yu J. T. (In Press). Review of research on computer-assisted language learning with a focus on intercultural education. Comput. Assist. Lang. Learn. 10.1080/09588221.2022.2056616 [ CrossRef ] [ Google Scholar ]
• * . Srebnaja J., Stavicka A. (2018). Web-based projects to develop transversal skills in secondary school . Hum. Technol. Qual. Educ. 2018 , 25–34. 10.22364/htqe.2018.03 [ CrossRef ] [ Google Scholar ]
• Suzanne N. (2014). “Critical and effective reading to build the characters as active readers,” in International Conference on Languages and Arts (Padang: ), 47–352. [ Google Scholar ]
• * . Thang S. M., Sim L. Y., Mahmud N., Lin L. K., Zabidi N. A., Ismail K. (2014). Enhancing 21st century learning skills via digital storytelling: voices of Malaysian teachers and undergraduates . Procedia Soc. Behav. Sci. 118 , 489–494. 10.1016/j.sbspro.2014.02.067 [ CrossRef ] [ Google Scholar ]
• * . Tseng C. T. H. (2017). Teaching “Cross-Cultural Communication” through content based instruction: curriculum design and learning outcome from EFL learners' perspectives . English Lang. Teach. 10 , 22–34. 10.5539/elt.v10n4p22 [ CrossRef ] [ Google Scholar ]
• * . Valdebenito M., Chen Y. (2019). Technology as enabler of learner autonomy and authentic learning in chinese language acquisition: a case study in higher education . J. Technol. Chin. Lang. Teach. 10, 61. [ Google Scholar ]
• Vygotsky L. S. (1978). Mind in Society: The Development of Higher Psychological Processes . Cambridge, MA: Harvard University Press. [ Google Scholar ]
• * . Yalçin O. B., Öztürk E. (2019). “The effects of digital storytelling on the creative writing skills of literature students based on their gender,” in ICGR 2019 2nd International Conference on Gender Research . (Rome: Academic Conferences and Publishing Limited; ), 59. [ Google Scholar ]
• * . Yang Y. T. C., Chen Y. C., Hung H. T. (2022). Digital storytelling as an interdisciplinary project to improve students' English speaking and creative thinking . Comput. Assist. Lang. Learn. 35 , 840–862. 10.1080/09588221.2020.1750431 [ CrossRef ] [ Google Scholar ]
• * . Yang Y. T. C., Chuang Y. C., Li L. Y., Tseng S. S. (2013). A blended learning environment for individualized English listening and speaking integrating critical thinking . Comput. Educ. 63 , 285–305. 10.1016/j.compedu.2012.12.012 [ CrossRef ] [ Google Scholar ]
• * . Yang Y. T. C., Gamble J. H., Hung Y. W., Lin T. Y. (2014). An online adaptive learning environment for critical-thinking-infused English literacy instruction . Br. J. Educ. Technol. 45 , 723–747. 10.1111/bjet.12080 [ CrossRef ] [ Google Scholar ]
• Zhang R., Zou D. (2020). Types, purposes, and effectiveness of state-of-the-art technologies for second and foreign language learning . Comput. Assist. Lang. Learn. 35 , 696–742. 10.1080/09588221.2020.1744666 [ CrossRef ] [ Google Scholar ]
• * . Zou D., Xie H. (2019). Flipping an English writing class with technology-enhanced just-in-time teaching and peer instruction . Interact. Learn. Environ. 27 , 1127–1142. 10.1080/10494820.2018.1495654 [ CrossRef ] [ Google Scholar ]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Published: 09 May 2022

Teaching of 21st century skills needs to be informed by psychological research

• Samuel Greiff   ORCID: orcid.org/0000-0003-2900-3734 1 &
• Francesca Borgonovi   ORCID: orcid.org/0000-0002-6759-4515 2  

Nature Reviews Psychology volume  1 ,  pages 314–315 ( 2022 ) Cite this article

268 Accesses

3 Citations

3 Altmetric

Metrics details

• Problem solving
• Science, technology and society

The technological advancements and globalization of the 21st century require a broad set of skills beyond traditional subjects such as mathematics, reading, and science. Research in psychological science should inform best practice and evidence-based recommendations for teaching these skills.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Behavioral patterns in collaborative problem solving: a latent profile analysis based on response times and actions in PISA 2015

• , Florian Krieger
•  …  Samuel Greiff

Large-scale Assessments in Education Open Access 13 November 2023

Access options

Subscribe to this journal

Receive 12 digital issues and online access to articles

55,14 € per year

only 4,60 € per issue

Buy this article

• Purchase on Springer Link
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Frank, M. R. et al. Toward understanding the impact of artificial intelligence on labor. Proc. Natl Acad. Sci. USA 116 , 6531–6539 (2019).

Article   Google Scholar  

OECD. Skills Outlook 2021: Learning for Life (OECD Publishing, 2021).

OECD. PISA 2015 Results: Collaborative Problem Solving (OECD Publishing, 2017).

Autor, D. H. Skills, education, and the rise of earnings inequality among the “other 99 percent”. Science 344 , 843–851 (2014).

Hattie, J. A. C. & Donoghue, G. M. Learning strategies: a synthesis and conceptual model. NPJ Sci. Learn. 1 , 16013 (2016).

Sonnleitner, P., Brunner, M., Keller, U. & Martin, R. Differential relations between facets of complex problem solving and students’ immigration background. J. Edu. Psy. 106 , 681–695 (2014).

Graesser, A. C., Sabatini, J. P. & Li, H. Educational psychology is evolving to accommodate technology, multiple disciplines, and twenty-first century skills. Annu. Rev. Psychol. 73 , 547–574 (2022).

National Academy of Sciences. Investigating the Influence of Standards. A Framework for Research in Mathematics , Science and Technology Education (National Academies Press, 2022).

Markovitz, D. The Meritocracy Trap: How America’s Foundational Myth Feeds Inequality, Dismantles the Middle Class, and Devours the Elite (Penguin, 2020).

Download references

Author information

Authors and affiliations.

Department of Behavioural and Cognitive Sciences, University of Luxembourg, Esch-sur-Alzette, Luxembourg

• Samuel Greiff

Social Research Institute, University College London, London, UK

Francesca Borgonovi

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Samuel Greiff .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Greiff, S., Borgonovi, F. Teaching of 21st century skills needs to be informed by psychological research. Nat Rev Psychol 1 , 314–315 (2022). https://doi.org/10.1038/s44159-022-00064-w

Download citation

Published : 09 May 2022

Issue Date : June 2022

DOI : https://doi.org/10.1038/s44159-022-00064-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

• Florian Krieger

Large-scale Assessments in Education (2023)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Integrating 21st century skills into education systems: From rhetoric to reality

Subscribe to the center for universal education bulletin, ramya vivekanandan rv ramya vivekanandan senior education specialist, learning assessment systems - gpe secretariat.

February 14, 2019

This is the third post in a series about  education systems alignment in teaching, learning, and assessing 21st century skills .

What does it mean to be a successful learner or graduate in today’s world? While in years past, a solid acquisition of the “three Rs” (reading, writing, and arithmetic) and mastery in the core academic subjects may have been the measure of attainment, the world of the 21 st century requires a radically different orientation. To participate effectively in the increasingly complex societies and globalized economy that characterize today’s world, students need to think critically, communicate effectively, collaborate with diverse peers, solve complex problems, adopt a global mindset, and engage with information and communications technologies, to name but just a few requirements. The new report from Brookings, “ Education system alignment for 21st century skills: Focus on assessment ,” illuminates this imperative in depth.

Recognizing that traditional education systems have generally not been preparing learners to face such challenges, the global education community has increasingly talked about and mobilized in favor of the changes required. This has resulted in a suite of initiatives and research around the broad area of “21st century skills,” which culminated most notably with the adoption of Sustainable Development Goal 4 and the Education 2030 agenda, including Target 4.7, which commits countries to ensure that learners acquire knowledge and skills in areas such as sustainable development, human rights, gender equality, global citizenship, and others.

In this landscape, Global Partnership for Education (GPE) has a core mandate of improving equity and learning by strengthening education systems. GPE supports developing countries, many of which are affected by fragility and conflict, to develop and implement robust education sector plans. Depending on the country, GPE implementation grants support a broad range of activities including teacher training, textbook provision, interventions to promote girls’ education, incentives for marginalized groups, the strengthening of data and learning assessment systems, early childhood education, and many other areas.

This work is buttressed by thematic work at the global level, including in the area of learning assessment. The strengthening of learning assessment systems is a strategic priority for GPE because of its relevance to both improving learning outcomes and ensuring effective and efficient education systems, which are two of the three key goals of the GPE strategic plan for the 2016-2020 period . The work on learning assessment includes the Assessment for Learning (A4L) initiative, which aims to strengthen learning assessment systems and to promote a holistic measurement of learning.

Under A4L, we are undertaking a landscape review on the measurement of 21st century skills, using a definition derived from Binkley et. al . and Scoular and Care :

“21st century skills are tools that can be universally applied to enhance ways of thinking, learning, working and living in the world. The skills include critical thinking/reasoning, creativity/creative thinking, problem solving, metacognition, collaboration, communication and global citizenship. 21st century skills also include literacies such as reading literacy, writing literacy, numeracy, information literacy, ICT [information and communications technologies] digital literacy, communication and can be described broadly as learning domains.”

Using this lens, the landscape review examines the research literature, the efforts of GPE partners that have been active in this space, and data collected from a sample of countries in sub-Saharan Africa and Asia in regard to the assessment of these skills. These research efforts were led by Brookings and coordinated by the UNESCO offices in Dakar and Bangkok. As another important piece of this work, we are also taking stock of the latest education sector plans and implementation grants of these same countries (nine in sub-Saharan Africa and six in Asia), to explore the extent to which the integration of 21st century skills is reflected in sector plans and, vitally, in their implementation.

Though the work is in progress, the initial findings provide food for thought. Reflecting the conclusions of the new report by Brookings, as well as its earlier breadth of work on skills mapping, a large majority of these 15 countries note ambitious objectives related to 21st century skills in their education sector plans, particularly in their vision or mission statements and/or statements of policy priorities. “Skills” such as creativity and innovation, critical thinking, problem-solving, decisionmaking, life and career skills, citizenship, personal and social responsibility, and information and communications technology literacy were strongly featured, as opposed to areas such as collaboration, communication, information literacy, and metacognition.

However, when we look at the planned interventions noted in these sector plans, there is not a strong indication that countries plan to operationalize their intentions to promote 21st century skills. Not surprisingly then, when we look at their implementation grants, which are one of the financing instruments through which education sector plans are implemented, only two of the 15 grants examined include activities aimed at promoting 21st century skills among their program components. Because the GPE model mandates that national governments determine the program components and allocation of resources for these within their grant, the bottom line seems to echo the findings of the Brookings report: vision and aspiration are rife, but action is scarce.

While the sample of countries studied in this exercise is small (and other countries’ education sector plans and grants may well include integration of 21st century skills), it’s the disconnect between the 15 countries’ policy orientation around these skills and their implementation that is telling. Why this gap? Why, if countries espouse the importance of 21st century skills in their sector plans, do they not concretely move to addressing them in their implementation? The reasons for this may be manifold, but the challenges highlighted by the Brookings report in terms of incorporating a 21 st century learning agenda in education systems are indeed telling. As a field, we still have much work to do to understand the nature of these skills, to develop learning progressions for them, and to design appropriate and authentic assessment of them. In other words, it may be that countries have difficulty in imagining how to move from rhetoric to reality.

However, in another perspective, there may be a challenge associated with how countries (and the broader education community) perceive 21st century skills in general. In contexts of limited resources, crowded curricula, inadequately trained teachers, fragility, weak governance, and other challenges that are characteristic of GPE partner countries, there is sometimes an unfortunate tendency to view 21st century skills and the “basics” as a tradeoff. In such settings, there can be a perception that 21st century skills are the concern of more advanced or higher-income countries. It is thus no wonder that, in the words of the Brookings report, “a global mobilization of efforts to respond to the 21CS [21st century skills] shift is non-existent, and individual countries struggle alone to plan the shift.”

This suggests that those who are committed to a holistic view of education have much work to do in terms of research, sharing of experience, capacity building, and advocacy around the potential and need for all countries, regardless of context, to move in this direction. The Brookings report makes a very valuable contribution in this regard. GPE’s landscape review, which will be published this spring, will inform how the partnership thinks about and approaches 21st century skills in its work and will thereby provide a complementary perspective.

Related Content

Helyn Kim, Esther Care, Alvin Vista

January 30, 2019

Tserennadmid Nyamkhuu, Jun Morohashi

February 5, 2019

Global Education K-12 Education

Global Economy and Development

Center for Universal Education

Modupe (Mo) Olateju, Grace Cannon

April 15, 2024

Brad Olsen, John McIntosh

April 3, 2024

Darcy Hutchins, Emily Markovich Morris, Laura Nora, Carolina Campos, Adelaida Gómez Vergara, Nancy G. Gordon, Esmeralda Macana, Karen Robertson

March 28, 2024

• Open access
• Published: 25 November 2019

Developing student 21 st Century skills in selected exemplary inclusive STEM high schools

• Stephanie M. Stehle   ORCID: orcid.org/0000-0003-4017-186X 1 &
• Erin E. Peters-Burton 1  

International Journal of STEM Education volume  6 , Article number:  39 ( 2019 ) Cite this article

121k Accesses

96 Citations

Metrics details

There is a need to arm students with noncognitive, or 21 st Century, skills to prepare them for a more STEM-based job market. As STEM schools are created in a response to this call to action, research is needed to better understand how exemplary STEM schools successfully accomplish this goal. This conversion mixed method study analyzed student work samples and teacher lesson plans from seven exemplary inclusive STEM high schools to better understand at what level teachers at these schools are engaging and developing student 21 st Century skills.

We found of the 67 lesson plans collected at the inclusive STEM high schools, 50 included instruction on 21 st Century skills. Most of these lesson plans designed instruction for 21 st Century skills at an introductory level. Few lesson plans encouraged multiple 21 st Century skills and addressed higher levels of those skills. Although there was not a significant difference between levels of 21 st Century skills by grade level, there was an overall trend of higher levels of 21 st Century skills demonstrated in lesson plans designed for grades 11 and 12. We also found that lesson plans that lasted three or more days had higher levels of 21 st Century skills.

Conclusions

These findings suggest that inclusive STEM high schools provide environments that support the development of 21 st Century skills. Yet, more can be done in the area of teacher professional development to improve instruction of high levels of 21 st Century skills.

Introduction

School-aged students in the USA are underperforming, particularly in science, technology, engineering, and mathematics (STEM) subjects. National Assessment of Educational Progress (U.S. Department of Education, 2015a ) scores show that in science, only 34% of 8th graders are performing at or above proficiency and 12th grade students at or above proficient US students drop to 22%. Similarly, mathematics scores show 33% of 8th graders and 22% of 12th graders were at or above proficiency (U.S. Department of Education, 2015a ). Additionally, the US mathematics scores for the Programme for International Student Assessment (PISA) for 2015 were lower than the scores for 2009 and 2012 (Organisation for Economic Co-operation and Development; OECD, 2018 ). US students not only underachieve in mathematics and science, but are also not engaging successfully in engineering and technology. At the secondary level, there are relatively few students in the USA that take engineering (2%) and computer science (5.7%) (National Science Board, 2016 ). The NAEP technology and engineering literacy (TEL) assessment found that for technology and engineering literacy, only 43% of 8th graders were at or above the proficiency level (U.S. Department of Education, 2015b ). This consistent trend of underperformance has focused many national, state, and local efforts to improve student experiences in integrated STEM subjects (cf. President’s Council of Advisors on Science and Technology, 2010 ; Texas Education Association ( n.d. ) for school-aged students and beyond.

The efforts for improvement in STEM teaching in K-12 environments have yielded a slight increase in the enrollment of STEM majors recently (National Science Board, 2016 ). However, roughly half of students who declare a STEM major when entering college either switch majors or drop out of college (National Science Board, 2016 ). One approach to helping students persist in undergraduate education is a stronger foundation in content knowledge, academic skills, and noncognitive skills (Farrington et al., 2012 ). Academic skills, including analysis and problem solving skills, allow students to engage with content knowledge at higher levels of cognition. Noncognitive skills, including study skills, time management, and self-management, assist students in optimizing their ability to gain content knowledge and use their academic skills to solve problems. Students who possess these skills have high-quality academic behaviors, characterized by a pursuit of academic goals despite any setbacks (Farrington et al., 2012 ).

Because academic skills, noncognitive skills, and content knowledge have fluid definitions and may not be directly observable, for the purposes of this study we used 21 st Century skills consisting of knowledge construction, real-world problem solving, skilled communication, collaboration, use of information and communication technology for learning, and self-regulation (Partnership for 21 st Century Learning, 2016 ). Graduates who possess 21 st Century skills are sought out by employers (National Research Council, 2013 ). In the environment of rapid advancements in technology and globalization, employees need to be flexible and perpetual learners in order to keep up with new developments (Bybee, 2013 ; Johnson, Peters-Burton, & Moore, 2016 ). There is a need to ensure that students who graduate the K-12 system are adept in 21 st Century skills so that they can be successful in this new workforce landscape (Bybee, 2013 ).

Not only do 21 st Century skills help students be successful in all areas of formal school, these skills are also necessary for a person to adapt and thrive in an ever changing world (Partnership for 21 st Century Learning, 2016 ). One movement embracing the need for the development of student 21 st Century skills is the proliferation of inclusive STEM high schools (ISHSs), schools that serve all students regardless of prior academic achievement (LaForce et al., 2016 ; Lynch et al., 2018 ). ISHSs promote student research experiences by using inquiry-based curricular models to scaffold independent learning and encourage personal responsibility (Tofel-Grehl & Callahan, 2014 ). The goal for ISHSs to facilitate this type of student-centered learning is to build students’ 21 st Century skills such as adaptability, communication, problem solving, critical thinking, collaboration, and self-management (Bybee, 2013 ; Johnson et al., 2016 ; LaForce et al., 2016 ). Although there has been some evidence that not all ISHSs are advantageous in offering STEM opportunities (Eisenhart et al., 2015 ), there is an accumulation of evidence that ISHSs can increase college and career readiness for students from groups who are typically underrepresented in STEM careers (Erdogan & Stuessy, 2015 ; Means, Wang, Viki, Peters, & Lynch, 2016 ). As the number of inclusive STEM schools continue to increase across the USA, there is a need to understand the ways these schools successfully engage students in 21 st Century skills. The purpose of this paper is to systematically analyze teacher-constructed lessons and student work from seven exemplar ISHSs in order to better understand how teachers are engaging and developing student 21 st Century skills.

Specifically, this study looked at the extent to which teachers at these exemplar ISHSs ask students to practice the 21 st Century skills and at the level of student performance of the following categories: (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of information and communication technology (ICT) for learning, and (f) self-regulation (SRI International, n.d. -a; SRI International, n.d. -b). An examination of the lesson plans and student work products at exemplar ISHSs provides insight into effective development of student 21 st Century skills in a variety of contexts.

Conceptual framework

In an attempt to clearly define the skills, content knowledge and literacies that students would need to be successful in their future endeavors, the Partnership for 21 st Century Learning (P21; 2016) created a framework that includes (a) life and career skills; (b) learning and innovation skills; (c) information, media, and technology skills; and (d) key subjects (Partnership for 21 st Century Learning, 2016 ). The first three parts of the framework, (a) life and career skills, (b) learning and innovation skills, and (c) information, media, and technology skills, describe proficiencies or literacies students should develop and can be integrated and developed in any academic lesson. The fourth piece, key subjects, suggests 21 st Century interdisciplinary themes or content to engage students in authentic study (Partnership for 21 st Century Learning, 2016 ).

Due to the need to build 21 st Century skills, this study focused on the teaching and learning of (a) learning and innovation skills; (b) information, media, and technology skills; and (c) life and career skills at exemplar ISHSs. In order to operationalize and measure the three categories, we searched for instruments that measured the learning of 21 st Century skills. Microsoft, in collaboration with SRI Education, developed two rubrics that are designed to assess the extent to which 21 st Century skills are present in lessons and the extent to which students demonstrate the skills from these lessons (SRI International, n.d. -a; SRI International, n.d. -b). The 21 st Century Learning Design Learning Activity Rubric examined the proficiency of teacher lesson plans for the development of 21 st Century skills while the 21 st Century Learning Design Student Work Rubric examined the level of competency for each 21 st Century skill. Although the rubrics did not align exactly with the P21 Framework, we felt that there was enough alignment with the categories that the rubrics would be useful in measuring the extent to which lessons in ISHSs taught 21 st Century skills and the extent to which students demonstrated these skills. The rubrics had the same categories for lesson assessment and student work assessment: (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of ICT for learning, and (f) self-regulation in teacher lesson plans and student work samples (SRI International, n.d. -a; SRI International, n.d. -b). Table 1 shows how the categories assessed in the two rubrics align with the categories in the P21 Framework. Further, as we reviewed the literature on these categories, a model of their relationship emerged. Our literature review discusses the individual categories followed by the conceptual model of how these categories work together in 21 st Century skill development.

• Knowledge construction

Knowledge construction occurs when students create new knowledge themselves rather than reproducing or consuming information (Prettyman, Ward, Jauk, & Awad, 2012 ; Shear, Novais, Means, Gallagher, & Langworthy, 2010 ). When students participate in knowledge construction rather than reproduction, they build a deeper understanding of the content. Learning environments that are designed for knowledge construction promote self-regulated and self-directed learners as well as building grit (Carpenter & Pease, 2013 ).

Although knowledge construction helps students to build deep understandings and skills to be self-directed and resilient learners, many students are unfamiliar with this approach to learning and frequently need scaffolding to take on joint responsibility of learning (Carpenter & Pease, 2013 ; Peters, 2010 ). When transitioning to a more student-centered learning environment that supports knowledge construction, the teacher becomes more of a facilitator rather than a lecturer (McCabe & O’Connor, 2014 ). A student-centered learning environment encourages students to shift from a paradigm of expecting one convergent answer and toward deeper meaning-making when learning (Peters, 2010 ). Knowledge construction anchors the development of 21 st Century skills because students need to be able to have background knowledge in order to perform the skills in an authentic context.

• Real-world problem solving

Sometimes called project-based learning (Warin, Talbi, Kolski, & Hoogstoel, 2016 ), real-world problem solving is characterized by students working to solve problems that have no current solution and where the students can implement their own approach (Shear et al., 2010 ). When solving a real-world problem, students work to identify the problem, propose a solution for a specific client, test the solution, and share their ideas (Prettyman et al., 2012 ; Warin et al., 2016 ). The design aspect of the process encourages students to be creative and learn from failures (Carroll, 2015 ). When using real-world problem solving, students develop knowledge in a meaningful way (White & Frederiksen, 1998 ), must regulate their cognition and behavior in a way to reach their goals (Brown, Bransford, Ferrara, & Campione, 1983 ; Flavell, 1987 ), and gain experience defending their choices through evidence and effective communication skills (Voss & Post, 1988 ).

Teachers can develop real-world problem solving skills in their students by modeling inquiry after research actual scientist are involved in, using databases with real-life data, and evaluating evidence from current events (Chinn & Malhortra, 2002 ). Designing real-world problem scenarios for the classroom provide a framework by which students can engage in 21 st Century learning and can help to encourage a more positive attitude towards STEM careers (Williams & Mangan, 2016 ). Together, knowledge construction and real-world problem solving create the foundation from which students can engage in self-regulation, collaboration, and communication.

• Self-regulation

Self-regulation is a key 21 st Century skill for independent learners. Students who are self-regulated plan their approach to problem solving, monitor their progress, and reflect on their work given feedback (Shear et al., 2010 ; Zimmerman, 2000 ). During the self-regulation process, a student motivates himself or herself to control impulses in order to efficiently solve problems (Carpenter & Pease, 2013 ; English & Kitsantas, 2013 ). Fortunately, these skills are teachable; however, students need time to accomplish regulatory tasks and guidance for the key processes of reflection and revision (Zimmerman, 2000 ). Therefore, long-term projects give a more appropriate time frame than short-term projects to hone these regulatory skills.

Students have different levels of self-regulation (English & Kitsantas, 2013 ) and teachers may need to integrate strategies and ways of monitoring students into lessons (Bell & Pape, 2014 ; English & Kitsantas, 2013 ). Incorporating self-regulated learning strategies helps students to stay engaged and deal with any adversity that may come up in the process (Boekaerts, 2016 ; Peters & Kitsantas, 2010 ). A tangible way teachers can support student self-regulation is by using Zimmerman’s ( 1998 ) four-stage model of self-regulated learning support: modeling, emulation, self-control, and self-regulation (Peters, 2010 ). First, teachers explicitly model the target learning strategy that the student should acquire, pointing out key processes (modeling). Second, teachers can provide students with verbal or written support for the key processes of the learning strategy while the student attempts to emulate the modeling from the teacher (emulation). Once students can roughly emulate the learning strategy, the teacher can fade support and have the student try to do the learning strategy on their own (self-control). After students attempt it on their own, the teacher provides feedback to the student to help them improve their attempted learning strategy (self-regulation). When a student can successfully perform the learning strategy on their own, they have become self-regulated in that aspect of their learning. Students who have mastered self-regulated learning have the ability to be proactive in knowledge building and in problem solving, which are characteristics that STEM industry employers value.

• Collaboration

Collaboration occurs when students take on roles and interact with one another in groups while working to produce a product (Shear et al., 2010 ). Collaborative interactions include taking on leadership roles, making decisions, building trust, communicating, reflecting, and managing conflicts (Carpenter & Pease, 2013 ). Students who collaborate solve problems at higher levels than students who work individually because students respond to feedback and questions to create solutions that better fit the problem (Care, Scoular, & Griffin, 2016 ). Collaboration is an important skill to enhance knowledge building and problem solving. Conversations among peers can support student self-regulated learning through modeling of verbalized thinking.

• Skilled communication

“Even the most brilliant scientific discovery, if not communicated widely and accurately, is of little value” (McNutt, 2013 , p. 13). For the purpose of this paper, skilled communication is defined as types of communication used to present or explain information, not discourse communication. Skilled communicators present their ideas and demonstrate how they use relevant evidence (Shear et al., 2010 ). An important part of being able to communicate successfully is the ability to connect a product to the needs of a specific audience or client (Warin et al., 2016 ). In doing so, the students need to take into account both the media they are using and the ideas they are communicating so that it is appropriate for the audience (Claro et al., 2012 ; van Laar, van Deursen, van Dijk, & de Haan, 2017 ). Like collaboration, skilled communication is a necessary process to successfully employ knowledge construction and real-world problem solving.

Use of information and communication technology for learning

When students use information and communication technology (ICT) for learning, they are designing, creating, representing, evaluating, or improving a product, not merely demonstrating their knowledge (Koh, Chai, Benjamin, & Hong, 2015 ). In doing so, they need to choose how and when to use the ICT as well as know how to recognize credible online resources (Shear et al., 2010 ). The effective use of ICT requires self-regulation in order to use these tools independently and to keep up with technological advances. Given the continuous advancements in technology, it is essential that students know how to manage and communicate information in order to solve problems (Ainley, Fraillon, Schulz, & Gebhardt, 2016 ).

Conceptual Model of 21 st Century Skills

The six 21 st Century skills presented above are critical for students to develop to prepare for both college (National Science Board, 2016 ) and the future employment (Bybee, 2013 ; Johnson et al., 2016 ). Twenty-first century skills do not exists in isolation. By building one skill, others are reinforced. For example, knowledge construction and real-world problem solving can be enhanced by self-regulation. Likewise, collaboration requires skilled communication to build knowledge and solve problems. These skills coalesce to build the necessary toolkit for students who can learn on their own. Figure 1 shows a working hypothesis of how these six skills, (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of ICT for learning, and (f) self-regulation, interact to foster lifelong learning for student.

Working hypothesis of how 21 st Century skills work together to build a 21 st Century student

Knowledge construction and real-world problem solving are the keystones of the model and typically represent the two main goals of student-centered lessons. Knowledge construction is the conceptual formation while real-world problem solving represents the process skills that students are expected to develop. Knowledge construction and real-world problem solving feed each other in a circular fashion. Knowledge construction is built through the inquiry process of real-world problem solving. At the same time, real-world problem solving requires new knowledge to be constructed in order to solve the problem at hand. The connection between knowledge construction and real work problem solving is mediated by collaboration and communication.

While communication and collaboration allow a student to work with others to build their conceptual knowledge and work toward a solution to their real-world problem, self-regulation is an internal process that occurs simultaneously. The student’s self-regulation guides the student’s individual connections, reflections, and revisions between knowledge construction and real-world problem solving.

Information and communication technology provides tools for the students to facilitate communication and collaboration as well as other 21 st Century skills. ICT helps to simplify and assist the communication and collaboration for groups of students. ICT can help streamline the process of analysis and record keeping as well as facilitating the sharing ideas with others. It allows students to more easily document their progress and express their ideas for later reflection. Although ICT is not directly connected with other elements in the model, the use of ICT allows for the learning process to be more efficient.

The six 21 st Century skills addressed in this study, (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of ICT for learning, and (f) self-regulation, are important facets of STEM education. This study documented the extent to which each of the 21 st Century skills were present in both lesson plans and in student work at seven exemplar ISHSs. Given that the schools in the study were highly regarded, understanding the structure and student outcomes of lessons could provide a model for teachers and teacher educators. With that in mind, the study was driven by the following research questions:

To what extent do teacher lesson plans at exemplar ISHSs exhibit 21 st Century learning practices as measured by the 21 st Century Learning Design Learning Activity and Student Work Rubrics?

Do teacher lesson plans and student work samples from exemplar ISHSs show differences in rubric scores by grade level?

During the analysis of these questions, a third research question emerged regarding the duration of lessons. The question and rationale can be found in the data analysis section.

This study is part of a larger multiple instrumental case study of eight exemplar ISHSs. The larger study (Opportunity Structures for Preparation and Inspiration in STEM; OSPrI) examined the common features of successful ISHSs (Lynch et al., 2018 ; Lynch, Peters-Burton, & Ford, 2014 ). OSPrI identified 14 critical components (CC; Table 2 ) that successful ISHSs possess (Behrend et al., 2016 ; Lynch et al., 2015 ; Lynch, Means, Behrend, & Peters-Burton, 2011 ; Peters-Burton, Lynch, Behrend, & Means, 2014 ). Three of the 14 critical components involve the application of 21 st Century skills in the classroom. This study addresses these three critical components: (a) CC1: STEM focused curriculum for all, (b) CC2: reform instructional strategies and project-based learning, and (c) CC3: integrated, innovative technology use.

Cross-case analysis of the eight schools found similarities in how the schools addressed two specific critical components: CC1: college-prep, STEM focused curriculum for all and CC2: reform instructional strategies and project-based learning. From these two critical components, curriculum and instruction, four themes emerged: (a) classroom-related STEM opportunities, (b) cross-cutting school level STEM learning opportunities, (c) school-wide design for STEM opportunities, and (d) responsive design (Peters-Burton, House, Han, & Lynch, 2018 ). The theme of classroom-related STEM opportunities was characterized by the expectation that teachers act as designers of the curriculum and look beyond the typical textbook for resources. While designing the curriculum, teachers took a mastery learning approach and provided students multiple opportunities to master the material. Through the use of collaborative group projects, summative projects, culminating projects, and interdisciplinary studies, the schools demonstrated a cross-cutting school level approach to the STEM learning. School-wide STEM opportunities included a rigorous curriculum, incorporating engineering classes and/or engineering design thinking, emphasizing connections between the curriculum and real-world examples, as well as building strong collaboration between teachers. Finally, these ISHSs had systems such as data-driven decision making and supports for incoming ninth graders built into their schools as a responsive design. In summary, these schools worked to improve students’ 21 st Century skill such as collaboration, problem solving, information and media literacy, and self-directed learning (Lynch et al., 2018 ).

Research design

This study was designed as a conversion mixed methods approach (Tashakkori & Teddlie, 2003 ) in that qualitative data were transformed into quantitative data using established rubrics. Document analysis was used as a tool to identify occasions of evidence within lessons plans and student work products related to the identified 21 st Century skills (Krippendorff, 2012 ). In this conversion approach, the 21 st Century skill demonstrated qualitatively in the documents was scored using the rubrics, ergo integrating qualitative and quantitative methods in the analysis.

Participating schools

The eight exemplar ISHSs for this study came from the same quintain as used by the OSPrI project (Lynch et al., 2018 ). Because this origin project was a cross-case analysis and the IRB did not allow for school to school comparison, the data collected from individual schools was aggregated as one data source. Protocol for inclusion in the OSPrI study was that the school had no academic admission requirements, self-identified as a STEM school, was in operation for grades 9 through 12, and intentionally recruited students typically underrepresented in STEM. For more information on the demographics of the schools and the selection process, see Lynch et al., 2018 . Of the eight schools that were in the original OSPrI project, seven provided teacher lesson plans and/or student work samples during the school visit. All schools have given permission to use their actual names. The sample size from each school was inconsistent, therefore, we treated the data set as one combined group that included all seven schools.

Data sources

Student work samples and teacher lesson plans were collected during OSPrI site visits to the seven schools, which were each visited once between 2012 and 2014. Researchers requested paper copies of typical lesson plans and student work that resulted in an average performance from the lesson plan that was observed at all eight ISHSs during the site visits. Because this was a convenience sample, not all teachers submitted lesson plans, and only a few teachers submitted the student work products related to those lessons. Unfortunately, few parents consented to release student work products. As a result, 67 teacher lesson plans and 29 student work samples were collected from seven of the eight schools. We decided to keep the student work products in the descriptive portion of the analysis, but not the inferential analysis in the study because this is a unique opportunity to gain even a small insight into student work from STEM schools that were considered exemplary and served students who are typically underrepresented in STEM. Table 3 describes the content matter and grade level(s) associated collected teacher lesson plan and corresponding student work product.

Each teacher lesson plan was analyzed using the 21 st Century Learning Design (21CLD) Learning Activity Rubric and each student work product was analyzed using the 21 st Century Learning Design Student Work Rubric (SRI International, n.d.-a; SRI International, n.d.-b). These instruments were found to be valid and reliable for use in high school classrooms, and Shear et al., 2010 reports the details of the development and validation of the rubrics. Although the student work products were related to the teacher lesson plans, they were analyzed independently according to the protocol of the 21CLD rubrics. The 21CLD Activity Rubric and the 21CLD Student Work Rubric were designed by Microsoft Partner’s in Learning with a collaboration between ITL Research and SRI International (SRI International, n.d.-a; SRI International, n.d.-b). These two 21CLD rubrics were the result of a multi-year project synthesizing research-based practices that promote 21 st Century skills (Shear et al., 2010 ). The rubrics, each 44-pages in length, are available online for public use ( https://education.microsoft.com/GetTrained/ITL-Research ). The 21CLD rubrics assess teacher lesson plans or student work products on six metrics aligned with 21 st Century skills: (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of ICT for learning, and (f) self-regulation (SRI International, n.d.-a; SRI International, n.d.-b). Collaboration, knowledge construction, and use of ICT score ratings range from one to five while real-world problem solving, self-regulation, and skilled communication score ratings range from one to four.

Data analysis

The teacher lessons and student work samples were assessed on (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of ICT for learning, and (f) self-regulation using the 21CLD Learning Activity and the 21CLD Student Work Rubrics respectively. Examples of excerpts from teacher lesson plans and student work products for each category can be found in Table 4 . Two raters were used to establish interrater reliability. Both raters have a background as secondary science teachers and were trained on the use of the rubric. One rater has a terminal degree in education and the other rater is a doctoral student in education. The two raters met and discussed the rubric scores until the interrater reliability was 100%. Once consensus scores were established, tests for assumptions, descriptive, and inferential statistics were run.

During the analysis of research questions one and two, unique trends of short-term and long-term lesson plans were noted. From this, a third research question emerged from the analysis:

Are there differences in the 21 CLD Learning Activity scores of short-term lessons and long-term lessons?

The 21CLD Learning Activity and the 21CLD Student Work Rubrics required a lesson to be long-term order to assess self-regulation. The rubric defined long-term as “if students work on it for a substantive period of time” (SRI International, n.d.-a, p. 32). From our reading of the lesson plans, lessons that were scheduled for three or more days met the criterion of a substantive period of time, while lesson that were scheduled for 1 or 2 days did not meet this criterion. For the purposes of this study, we decided to refine the definition of long-term to be a lesson lasting three or more class periods and a short-term lesson lasting less than three class periods. The analyses for all research questions separated lessons into long-term and short-term in order to clarify the category of self-regulation.

The data were checked for normality, skewness, and outliers; only the teacher lesson plans met all assumptions for an ANOVA (comparison of grade levels) and t test (long-term versus short-term). Due to the small number of student work samples collected (see Table 6 ), the data related to student work did not meet the assumptions needed to run a t test therefore was not included in this analysis.

Overall rubric scores

To answer the first research question, a descriptive analysis was run for each of the six categories on the rubric and the total score (found in Tables 5 and 6 ). The average score for all teacher lesson plans was less than 2 for all six categories (out of a total of 4 or 5). Likewise, overall student work sample averages scored below 2 except on the category of Knowledge Construction. Table 6 also shows the median score for long-term student work sample categories to better describe central tendencies of the data. Figure 2 shows the distribution of total rubric scores for all teacher lesson plans. Seventeen of the 67 lessons scored a 6, the lowest possible score. Only 16 of the 67 lessons scored higher than 13 points, half of the total possible points. Out of those 16 scoring over 50%, only three lessons scored 20 points or more out of the possible 27.

Distribution of total 21CLD rubric scores for all lessons

Figure 3 illustrates the quantity of 21 st Century skills found in each lesson. Nearly 75% of the teacher lesson plans included at least one 21 st Century skill in the lesson and 67% addressed two or more 21 st Century skills. Although most of the lessons at the ISHSs introduced multiple 21 st Century skills, the overall scores for the quality were low.

Distribution of number of 21 st Century skills addressed in a lesson

21 st Century learning by grade

To answer the second research question, an ANOVA was conducted to compare lesson scores by grade level. There were no statistically significant differences between grade level scores for the total rubric score. Data were separated into short-term and long-term lessons by rubric category. There were no significant differences in short-term lessons by grade level (Fig. 4 ). However, there were significant differences across grades for long-term lessons. Total rubric score for grade 12 lessons were significantly higher than grade 9 ( p = 0.023) and grade 11 ( p = 0.032). Difference in total rubric scores for grade 12 lessons were approaching significance with grade 10 ( p = 0.063). As seen in Fig. 5 , category scores for long-term learning activities have small differences in 9th, 10th, and 11th grades but peaks noticeably in 12th grade. The exception to this trend is use of ICT which peaks in 11th grade.

The average rubric metric scores for short-term lessons, sorted by grade level for the lesson

The average rubric metric scores for long-term lessons, sorted by grade level for the lesson

Long-term versus short-term assignments

To answer the second research question, a t test with Bonferroni correction was performed to compare long-term and short-term lessons for each of the categories. A statistically significant difference was found between short-term ( N = 35) and long-term ( N = 32) lessons on total score, knowledge construction, use of ICT, self-regulation, and skilled communication (Table 7 ). The effect sizes for these categories as calculated by Hedges g (Lakens, 2013 ) were all above 0.8 indicated large effect size (Table 7 ). In all of those categories, long-term lessons scored higher than short-term lessons (Table 5 ). The category of real-world problem solving was approaching statistical significance with the t-score not showing significance [ t = − 2.67, p = .001] but a statistically significant confidence interval [− 1.23, 0.003] and a medium effect size (Table 7 ).

• 21 st Century skills

Overall, the teacher lesson plans collected at the ISHSs showed evidence of addressing 21 st Century skills. Nearly 75% of the lessons included at least one 21 st Century skill with 67% addressing two or more. Although the majority of lessons addressed multiple 21 st Century skills, the rubric scores for these lessons were low because they addressed these skills at a minimal level. For example, a minimal level of collaboration would be instructions to form a group. A high level of collaboration would include defining roles, explicit instructions on how to share responsibility, and evidence of interdependence. Only five lessons showed evidence of multiple 21 st Century skills implemented at the highest level, as measured by the 21CLD Learning Activity Rubric.

While assessing the lesson plans, we noted that more explicit instructions in the teacher lesson plans would have resulted in higher rubric scores. Placing students in groups, structuring peer feedback, and having students design a final project for a particular audience are three small changes not seen frequently in the lesson plans that are articulated in the Lesson Plan rubrics to encourage multiple 21 st Century skills. When students work in groups, they improve their collaboration and communication skills while constructing knowledge and solving problems (Care et al., 2016 ; Shear et al., 2010 ). When teachers incorporate peer feedback into their lesson, students engage in collaboration. Peer feedback also gives students the opportunity to revise their work based on feedback, increasing self-regulation (Shear et al., 2010 ; Zimmerman, 2000 ). When students design their final project for a specific target audience, rather than simply displaying their knowledge for the teacher, they work on their skilled communication processes (Claro et al., 2012 ; van Laar et al., 2017 ; Warin et al., 2016 ). In summary, placing students in groups, structuring peer feedback, and having students design a final project for a particular audience provides opportunities for students to practice 21 st Century skills.

When lessons addressed more than one 21 st Century skill, they usually demonstrated the use of collaboration or communication in real-world problem solving and knowledge construction (Care et al., 2016 ; Carpenter & Pease, 2013 ). Thirty-three lesson plans in which real-world problem solving or knowledge construction was evident, 31 showed evidence of collaboration or communication. Similarly, 13 of the 18 student work samples showed evidence of collaboration or communication when real-world problem solving or knowledge construction was practiced. The results from the indirect measures of the rubric build support for a conceptual model connecting the components of 21 st Century skills (Fig. 1 ). There was some evidence demonstrating the support that collaboration and communication have for knowledge construction and real-world problem solving.

The findings of this study point to the likelihood of self-regulation being connected to other 21 st Century skills. Each time self-regulation was present in a teacher lesson plan, there was evidence of at least one other 21 st Century skill in that lesson. Seventeen of the 23 lesson plans addressing self-regulation included at least three other 21 st Century skills, showing evidence that self-regulation is a skill that is related to knowledge construction and real-world problem solving. Our findings reflect the findings of other researchers, in that self-regulation guides the students’ individual connections, reflections, and revisions between knowledge construction and real-world problem solving (Brown et al., 1983 ; Carpenter & Pease, 2013 ; Flavell, 1987 ; Shear et al., 2010 ).

Evidence from the lessons showed that there was no consistent connection to the use of ICT and the presence of the other 21 st Century skills. ICT was seen in both low-scoring lessons as the sole 21 st Century skill, as well as in high-scoring lessons in tandem with multiple other 21 st Century skills. As in our model, technology is a tool to help facilitate but is not necessary in the development of the other 21 st Century skills (Koh et al., 2015 ; Shear et al., 2010 ). After examining the data, our model remained unchanged for all 21 st Century skills and their relationship to each other.

Grade level differences

Overall, there were no statistically significant differences in the total 21CLD scores across grade levels. This is consistent with the missions of the ISHSs in this study to shift responsibility for learning to the students by weaving 21 st Century skills throughout high school grade levels (Lynch et al., 2017 ). When looking at trends in long-term projects, there was a jump in total 21CLD score for 12th grade. Again, this aligns with the participating schools’ goals of creating an environment where students have a more independent learning experience during their senior year internships, college classes, and specialized programs CC1 (Lynch et al., 2018 ). This is consistent with the goal of many of the schools to have the students work independently during their senior year either by taking college classes, completing an internship, or taking a career specific set of classes.

Short-term vs. long-term lessons

The data showed that long-term lesson planning had significantly higher scores on the rubric as compared to the short-termed lessons. This difference is consistent with the literature regarding the need for students to have time to develop and practice skills (Lynch et al., 2017 ; NGSS Lead States, 2013 ). The extended time allows students to monitor and reflect on their progress while working toward self-regulation of the skill (Carpenter & Pease, 2013 ; English & Kitsantas, 2013 ). To truly become self-regulated, students need repeated supported attempts to be able to do it on their own (Zimmerman, 2000 ).

Although not significant, collaboration was the only rubric metric where the short-term lessons averaged a higher collaboration score than the long-term lessons. Evidence from the lessons show students worked in pairs or groups, but infrequently shared responsibility, made decisions together, or worked interdependently. This leads to the possibility that incorporating the higher levels of collaborations is difficult, even in long-term projects. In addition, evaluating the higher levels of collaboration is difficult to make based solely on documents. Observations would be required to evaluate how the students within the group were interacting with one another.

Limitations

Because this study used data collected as part of a larger study, there were several limitations. The work collected is a snapshot of the work students were doing at the time of the observation and does not allow for a clear longitudinal look at student growth over time. As stated before, the small student work sample limited what we were able to do with the analysis.

By only analyzing paper copies of the student work, it was not possible to determine a true collaboration score for many of the projects. Higher levels of collaboration such as sharing responsibility, making decisions together, and working interdependently require observation or more detailed notes from the students or teachers. Some lessons may have scored higher in the metric of collaboration had the student interactions been observed or noted.

This study confirmed the presence of all identified 21 st Century skills in the lesson plans at the selected exemplar ISHSs serving underrepresented students in STEM: (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of information and communication technology (ICT) for learning, and (f) self-regulation. In light of the patterns that emerged from the rubrics, we posit that in the lesson plans communication and collaboration are the core 21st Century skills that facilitate knowledge construction and real-world problem solving, while student self-regulation creates efficiencies resulting in improved knowledge construction and real-world problem solving. We also saw in the lesson plans that ICT provides tools to support communication and reflection which leads to knowledge construction and real-world problem solving. To further develop knowledge about how 21 st Century skills addressed in lesson plans help to support student work, our model can be a hypothesized starting point to investigate interactions.

While teachers were successful at including 21 st Century skills into lessons, very few lessons practiced higher levels of those skills. This could be an indication that high levels of 21 st Century skills are difficult to teach explicitly at the high school level. Future studies may investigate why teachers are not frequently incorporating higher level 21 st Century skills into their lessons to answer questions as to whether teachers feel that (a) they need more training on incorporating 21 st Century skills, (b) students need more practice and scaffolding to build up to higher levels of 21 st Century skills, or (c) they need more time for long-term projects to work on the higher level skills.

The use of the 21CLD rubric is a tangible way for teachers to self-assess the level of 21 st Century skills in their lessons. Self-evaluation helps encourage reflection, promote professional growth, and recommendations for new aspects of lessons (Akram & Zepeda, 2015 ; Peterson & Comeaux, 1990 ). This can also help teachers make the instructions for the development of 21 st Century skills more explicit in their lesson. In conducting a self-evaluation, teachers may realize that they do not have a deep understanding of the characteristics of 21 st Century skills. If teachers are new to incorporating these skills into their lessons, the teachers may need time to learn the skills themselves before they can incorporate them into their lessons (Yoon et al., 2015 ). Further studies may examine how teachers use the 21CLD rubric to improve their lesson.

Students need time to grapple with and learn new skills (Lynch et al., 2017 ; NGSS Lead States, 2013 ). While we were able to see evidence of higher rubric scores for 21 st Century skills for 12th grade students in the lesson plans, due to the convenience sampling of lesson plans and student work samples, we were not able to look at how students’ 21 st Century skills were built over time. There is a desire to better understand how ISHSs successfully develop these skills. This includes how schools incorporate and build the 21 st Century skills (a) within multiple lessons in one course, (b) across multiple classes over the course of a school year, and (c) throughout the students’ entire high school sequence. Future research may look at a longitudinal study that follows one student’s work over an entire school year to see how the 21CLD scores change. In addition, future studies may also look at how the short-term projects build the skills needed for the students to incorporate higher levels of 21 st Century skills in long-term projects.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

21 st Century Learning Design

Critical component

Information and communication technology

Inclusive STEM high school

National Assessment of Educational Progress

Next-generation science standards

Opportunity Structures for Preparation and Inspiration in STEM

Partnership for 21 st Century Learning

Programme for International Student Assessment

Science, technology, engineering, and mathematics

Technology and engineering literacy

Ainley, J., Fraillon, J., Schulz, W., & Gebhardt, E. (2016). Conceptualizing and measuring computer and information literacy in cross-national contexts. Applied Measurement in Education, 29 , 291–309. https://doi.org/10.1080/08957347.2016.1209205 .

Article   Google Scholar  

Akram, M., & Zepeda, S. J. (2015). Development and validation of a teacher self-assessment instrument. Journal of Research and Reflections in Education, 9 (2), 134–148.

Google Scholar  

Behrend, T. S., Peters-Burton, E. E., Hudson, C., Matray, S., Ford, M., & Lynch, S. J. (2016). STEM High School Inventory. [Measurement instrument]. Retrieved from https://ospri.research.gwu.edu/sites/ospri.research.gwu.edu/files/downloads/CC%20Inventory_FINAL.pdf .

Bell, C. V., & Pape, S. J. (2014). Scaffolding the development of self-regulated learning in mathematics classrooms. Middle School Journal, 45 (4), 23–32.

Boekaerts, M. (2016). Engagement as an inherent aspect of the learning process. Learning and Instruction, 43 , 76–83. https://doi.org/10.1016/j.learninstruc.2016.02.001 .

Brown, A. L., Bransford, J., Ferrara, R., & Campione, J. (1983). Learning, remembering, and understanding. In P. H. Musen (Ed.), Handbook of child psychology (Vol. III, pp. 77–166). New York: Wiley.

Bybee, R. W. (2013). The case for STEM education . Arlington: NSTA press.

Care, E., Scoular, C., & Griffin, P. (2016). Assessment of collaborative problem solving in education environments. Applied Measurement in Education, 29 , 250–264. https://doi.org/10.1080/08957347.2016.1209204 .

Carpenter, J. P., & Pease, J. S. (2013). Preparing students to take responsibility for learning: The role of non-curricular learning strategies. Journal of Curriculum & Instruction, 7 (2), 38–55. https://doi.org/10.3776/joci.2013.v7n2p38-55 .

Carroll, M. (2015). Stretch, dream, and do—A 21 st century design thinking & STEM journey. Journal of Research in STEM Education, 1 (1), 59–70.

Chinn, C. A., & Malhortra, B. A. (2002). Epistemologically authentic inquiry in schools: A theoretical framework for evaluating inquiry tasks. Science Education, 86 , 175–218. https://doi.org/10.1002/sce.10001 .

Claro, M., Preiss, D. D., San Martin, E., Jara, I., Hinostroza, J. E., Valenzuela, S., Cortes, F., & Nussbaun, M. (2012). Assessment of 21 st century ICT skills in Chile: Test design and results from high school level students. Computers & Education, 59 , 1042–1053. https://doi.org/10.1016/j.compedu.2012.04.004 .

Eisenhart, M., Weis, L., Allen, C. D., Cipollone, K., Stich, A., & Dominguez, R. (2015). High school opportunities for STEM: Comparing inclusive STEM-focused and comprehensive high schools in two US cities. Journal of Research in Science Teaching, 52 (6), 763–789. https://doi.org/10.1002/tea.21213 .

English, M. C., & Kitsantas, A. (2013). Supporting student self-regulated learning in problem- and project based learning. Interdisciplinary Journal of Problem-Based Learning, 7 (2), 127–150. https://doi.org/10.7771/1541-5015.1339 .

Erdogan, N., & Stuessy, C. (2015). Examining the role of inclusive STEM schools in the college and career readiness of students in the United States: A multi-group analysis on the outcome of student achievement. Educational Sciences: Theory & Practice, 15 (6), 1517–1529. https://doi.org/10.12738/estp.2016.1.0072 .

Farrington, C. A., Roderick, M., Allensworth, E., Nagaoka, J., Keyes, T. S., Johnson, D. W., & Beechum, N. O. (2012). Teaching adolescents to become learners. The role of noncognitive factors in shaping school performance: A critical literature review . Chicago: University of Chicago Consortium on Chicago School Research.

Flavell, J. H. (1987). Speculations about the nature and development of metacognition. In F. Weinert & U. R. Kluwe (Eds.), Metacognition, motivation, and understanding (pp. 21–29). Hillsdale: Erlbaum.

Johnson, C. C., Peters-Burton, E. E., & Moore, T. J. (Eds.). (2016). STEM road map: A framework for integrated STEM education . New York: Routledge.

Koh, J. H. L., Chai, C. S., Benjamin, W., & Hong, H. Y. (2015). Technological pedagogical content knowledge (TPACK) and design thinking: A framework to support ICT lesson design for 21st century learning. Asia-Pacific Education Researcher (Springer Science & Business Media B.V.), 24 (3), 535–543. https://doi.org/10.1007/s40299-015-0237-2 .

Krippendorff, K. H. (2012). Content analysis: An introduction to its methodology (3rd ed.). Los Angeles: SAGE Publications, Inc..

LaForce, M., Noble, E., King, H., Century, J., Blackwell, C., Holt, S., Ibrahim, A., & Loo, S. (2016). The eight essential elements of inclusive STEM high schools. International Journal of STEM Education, 3 (21), 1–11. https://doi.org/10.1186/s40594-016-0054-z .

Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4 , 863. https://doi.org/10.3389/fpsyg.2013.00863 .

Lynch, S. J., House, A., Peters-Burton, E., Behrend, T., Means, B., Ford, M., Spillane, N., Matray, S., Moore, I., Coyne, C., Williams, C., & Corn, J. (2015). A Logic model that describes and explains eight exemplary STEM-focused high schools with diverse student populations . Washington DC: George Washington University OSPrI Project Retrieved from http://ospri.research.gwu.edu .

Lynch, S. J., Means, B., Behrend, T., & Peters-Burton, E. (2011). Multiple instrumental case studies of inclusive STEM-focused high schools: Opportunity Structures for Preparation and Inspiration (OSPrI). Retrieved from http://ospri.research.gwu.edu

Lynch, S. J., Peters-Burton, E. E., Behrend, T., House, A., Ford, M., Spillane, N., Matray, S., Han, E., & Means, B. (2018). Understanding inclusive STEM high schools as opportunity structures for underrepresented students: Critical components. Journal of Research in Science Teaching., 55 (5), 712–748. https://doi.org/10.1002/tea.21437 .

Lynch, S. J., Peters-Burton, E. E., & Ford, M. (2014). Building STEM opportunities for all. Educational Leadership, 72 (4), 54–60.

Lynch, S. J., Spillane, N., House, A., Peters-Burton, E., Behrend, T., Ross, K. M., & Han, E. M. (2017). A policy-relevant instrument case study of an inclusive STEM-focused high school: Manor New Tech High. International Journal of Education in Mathematics, Science and Technology, 5 (1), 1–20. https://doi.org/10.18404/ijemst.75656 .

McCabe, A., & O’Connor, U. (2014). Student-centered learning: The role and responsibility of the lecturer. Teacher in Higher Education, 19 (4), 350–359. https://doi.org/10.1080/13562517.2013.860111 .

McNutt, M. (2013). Improving scientific communication. Science, 342 , 13. https://doi.org/10.1126/science.1246449 .

Means, B., Wang, H., Viki, Y., Peters, V. L., & Lynch, S. J. (2016). STEM-focused high schools as a strategy for enhancing readiness for postsecondary STEM programs. Journal of Research in Science Teaching, 53 (5), 709–736. https://doi.org/10.1002/tea.21313 .

National Research Council. (2013). Monitoring progress toward successful K-12 STEM education: A nation advancing? Washington, DC: National Academies Press. https://doi.org/10.17226/13509 .

Book   Google Scholar  

National Science Board. (2016). Science and engineering indicators 2016. (NSB-2016-1) . Arlington: National Science Foundation.

NGSS Lead States. (2013). Next generation science standards: For states, by states . Washington, DC: The National Academies Press.

Organisation for Economic Co-operation and Development. (2018). PISA 2015 results in focus. Retrieved from https://www.oecd.org/pisa/pisa-2015-results-in-focus.pdf

Partnership for 21st Century Learning. (2016). Framework for 21st century learning. Retrieved from www.p21.org/about-us/p21-framework .

Peters, E. E. (2010). Shifting to a student-centered science classroom: An exploration of teacher and student changes in perceptions and practices. Journal of Science Teacher Education, 21 (3), 329–349. https://doi.org/10.1007/s10972-009-9178-z .

Peters, E. E., & Kitsantas, A. (2010). The effect of nature of science metacognitive prompts on science students’ content and nature of science knowledge, metacognition, and self-regulatory efficacy. School Science and Mathematics, 110 , 382–396. https://doi.org/10.1111/j.1949-8594.2010.00050.x .

Peters-Burton, E. E., House, A., Han, E., & Lynch, S. (2018). Curriculum and instruction at inclusive STEM high schools. Journal of Research in STEM Education, 4 (2), 193–212.

Peters-Burton, E. E., Lynch, S. J., Behrend, T. S., & Means, B. B. (2014). Inclusive STEM high school design: 10 critical components. Theory into Practice, 53 (1), 64–71. https://doi.org/10.1080/00405841.2014.862125 .

Peterson, P. L., & Comeaux, M. A. (1990). Evaluating the systems: Teachers’ perspectives on teacher evaluation. Educational Evaluation and Policy Analysis, 12 (1), 3–24. https://doi.org/10.3102/01623737012001003 .

President’s Council of Advisors on Science and Technology. (2010). Prepare and inspire: K-12 education in science, technology, engineering, and math (STEM) for America’s future . Washington, DC: Executive Office of the President.

Prettyman, S. S., Ward, C. L., Jauk, D., & Awad, G. (2012). 21st century learners: Voices of students in a one-to-one STEM environment. Journal of Applied Learning Technology, 2 (4), 6–15.

Shear, L., Novais, G., Means, B., Gallagher, L., & Langworthy, M. (2010). ITL research design . Menlo Park: SRI International Retrieved from https://www.sri.com/sites/default/files/publications/itl_research_design_15_nov_2010.pdf .

SRI International. (n.d.-a). 21CLD learning activity rubrics. Retrieved from https://education.microsoft.com/GetTrained/ITL-Research

SRI International. (n.d.-b). 21CLD student work rubrics. Retrieved from https://education.microsoft.com/GetTrained/ITL-Research

Tashakkori, A., & Teddlie, C. (2003). Handbook of mixed methods in social & behavioral research . Thousand Oaks: Sage.

Texas Education Association (n.d.). Texas science, technology, engineering and mathematics initiative (T-STEM). Retrieved from https://tea.texas.gov/T-STEM/

Tofel-Grehl, C., & Callahan, C. M. (2014). STEM high school communities: Common and differing features. Journal of Advanced Academics, 25 (3), 237–271. https://doi.org/10.1177/1932202X14539156 .

U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. (2015a). The nation’s report card: 2015 mathematics and reading assessments. (NCES No. 2015136). Retrieved from https://www.nationsreportcard.gov/reading_math_2015/#?grade=4

U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. (2015b). The nation’s report card: Technology and engineering literacy. (NCES No. 2016119). Retrieved from https://www.nationsreportcard.gov/tel_2014/

van Laar, E., van Deursen, A. J. A. M., van Dijk, J. A. G. M., & de Haan, J. (2017). The relation between 21 st -century skills and digital skills: A systematic literature review. Computers in Human Behavior, 77 , 577–588. https://doi.org/10.1016/j.chb.2017.03.010 .

Voss, J. F., & Post, T. A. (1988). On the solving of ill-structured problems. In M. T. H. Chi, R. Glaser, & M. J. Farr (Eds.), The nature of expertise (pp. 261–285). Hillsdale: Lawrence Erlbaum.

Warin, B., Talbi, O., Kolski, C., & Hoogstoel, F. (2016). Multi-role project (MRP): A new project-based learning method for STEM. IEEE Transactions on Education, 59 (2), 137–146. https://doi.org/10.1109/TE.2015.2462809 .

White, B. Y., & Frederiksen, J. R. (1998). Inquiry, modeling, and metacognition: Making science accessible to all students. Cognition and Instruction, 16 (1), 3–18.

Williams, P. J., & Mangan, J. (2016). The effectiveness of using young professionals to influence STEM career choices of secondary school students. Journal of Research in STEM Education, 2 (1), 2–18.

Yoon, S. A., Anderson, E., Koehler-Yom, J., Klopfer, E., Sheldon, J., Wendel, D., Schoenfeld, I., Scheintaub, H., Oztok, M., & Evans, C. (2015). Designing curriculum and instruction for computer-supported complex systems teaching and learning in high school science classrooms. Journal of Research in STEM Education, 1 (1), 4–14.

Zimmerman, B. J. (1998). Developing self-fulfilling cycles of academic regulation: An analysis of exemplary instructional models. In D. H. Schunk & B. J. Zimmerman (Eds.), Self-regulated learning: From teaching to self-reflective practice (pp. 1–19). New York: The Guilford Press.

Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 13–39). San Diego: Academic Press.

Chapter   Google Scholar  

Download references

Acknowledgments

Publication of this article was funded in part by the George Mason University Libraries Open Access Publishing Fund.

This work was conducted by the OSPrI research project, with Sharon Lynch, Tara Behrend, Erin Peters-Burton, and Barbara Means as principal investigators. Funding for OSPrI was provided by the National Science Foundation (DRL 1118851). Any opinions, findings, conclusions, or recommendations are those of the authors and do not necessarily reflect the position or policy of endorsement of the funding agency.

Author information

Authors and affiliations.

George Mason University, Fairfax, USA

Stephanie M. Stehle & Erin E. Peters-Burton

You can also search for this author in PubMed   Google Scholar

Contributions

Both authors contributed equally to this manuscript. Both authors read and approved the final manuscript.

Corresponding author

Correspondence to Stephanie M. Stehle .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Stehle, S.M., Peters-Burton, E.E. Developing student 21 st Century skills in selected exemplary inclusive STEM high schools. IJ STEM Ed 6 , 39 (2019). https://doi.org/10.1186/s40594-019-0192-1

Download citation

Received : 21 December 2018

Accepted : 04 October 2019

Published : 25 November 2019

DOI : https://doi.org/10.1186/s40594-019-0192-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• STEM schools

Advertisement

Teachers’ AI digital competencies and twenty-first century skills in the post-pandemic world

• Cultural and Regional Perspectives
• Published: 21 February 2023
• Volume 71 , pages 137–161, ( 2023 )

Cite this article

• Davy Tsz Kit Ng   ORCID: orcid.org/0000-0002-2380-7814 1 ,
• Jac Ka Lok Leung 2 ,
• Jiahong Su 1 ,
• Ross Chi Wui Ng 1 &
• Samuel Kai Wah Chu 1  

27k Accesses

37 Citations

29 Altmetric

Explore all metrics

The pandemic has catalyzed a significant shift to online/blended teaching and learning where teachers apply emerging technologies to enhance their students’ learning outcomes. Artificial intelligence (AI) technology has gained its popularity in online learning environments during the pandemic to assist students’ learning. However, many of these AI tools are new to teachers. They may not have rich technical knowledge to use AI educational applications to facilitate their teaching, not to mention developing students’ AI digital capabilities. As such, there is a growing need for teachers to equip themselves with adequate digital competencies so as to use and teach AI in their teaching environments. There are few existing frameworks informing teachers of necessary AI competencies. This study first explores the opportunities and challenges of employing AI systems and how they can enhance teaching, learning and assessment. Then, aligning with generic digital competency frameworks, the DigCompEdu framework and P21’s framework for twenty-first century learning were adapted and revised to accommodate AI technologies. Recommendations are proposed to support educators and researchers to promote AI education in their classrooms and academia.

Similar content being viewed by others

Students’ voices on generative AI: perceptions, benefits, and challenges in higher education

Cecilia Ka Yuk Chan & Wenjie Hu

Artificial intelligence in education: Addressing ethical challenges in K-12 settings

Selin Akgun & Christine Greenhow

Impacts of digital technologies on education and factors influencing schools' digital capacity and transformation: A literature review

Stella Timotheou, Ourania Miliou, … Andri Ioannou

Avoid common mistakes on your manuscript.

Opportunities and challenges brought by the pandemic

The COVID-19 pandemic has caused a significant shift to online/blended teaching and learning that educators tried to incorporate new technologies in their classrooms (Ng et al., 2020 ; Sartika et al., 2021 ; Whalley et al., 2021 ). Among these, artificial intelligence in education (AIED) technology has gained its popularity during the pandemic. Studies have started discussions on how AI reshapes education to reduce teachers’ workload by automating some non-teaching related tasks, enhancing data analysis and optimizing online teaching (Kexin et al., 2020 ). AIED has traditionally referred to intelligent tutors that help automate students’ learning and track their progress so that teachers can offer personalized assistance. More recently, AI-driven tools have become more teacher-focused and help teachers identify effective pedagogies based on students’ learning data, automate operational tasks, generate assessments, automate grading and feedback which greatly save teachers’ time and enhance efficiencies (Chaudhry & Kazim et al., 2022 ). Some studies argued that AI technology can effectively promote students’ personalized learning (Ahmad et al., 2022 ), advance their knowledge acquisition and motivate students’ learning using intelligent agents (Chen et al., 2020a , 2020b ; Hwang et al., 2020 ). However, without addressing the issue of teachers’ roles and competencies, the claimed benefits could be questionable. Therefore, it is important to consider how teachers’ competencies change in an AI context (Markauskaite et al., 2022 ).

A major part of teachers’ responsibilities is to create meaningful learning environments to deepen students’ learning experiences and boost their capacities. However, teachers may not be digitally ready to use AI-driven educational applications for teaching and learning purposes (e.g., Ally, 2019 ; Seo et al., 2021 ). They may lack technological experience to conduct data analysis, or to set rules to automatically generate assignments and feedback for students via AI-driven tools (Seo et al., 2021 ). Challenges such as AI-based misunderstanding, misleadingness, limitations, and hidden ethical issues behind different platforms have been identified (Akgun & Greenhow, 2021 ; Sijing & Lan, 2018 ). There is a broad consensus that teacher education is an important factor influencing student achievement which could lead to higher social and economic expectations (OECD, 2005 ). Instructional and theoretical frameworks are useful to set as a reference for teachers to identify necessary AI competencies to facilitate their teaching (Chiu, 2021 ; Ng et al., 2022a ).

Although recent research effort has been put to address the need of fostering students’ AI competencies (e.g., Kong et al., 2021 ; Ng et al., 2021a , b ; Xia et al., 2022 ), rarely do studies reveal what digital competencies teachers need to become ready in an AI-driven learning environment. The pandemic has led to an unprecedented use of technology for education. It has also turned a crisis into an opportunity, and catalyzed a shift to AI-driven digital teaching/learning (Green et al., 2020 ; Moorhouse, 2020 ). AI has become one of the most essential and popular technologies adopted by university educators to process and analyze big data for distant learners (Aljarrah et al., 2021 ; European Commission, 2022 ). To develop teachers’ competencies to adapt AI-driven teaching and learning tools and approaches, educators need to timely update their skills and knowledge so as to create suitable learning environments for their students (Williamson & Eynon, 2020 ).

This conceptual paper uses research examples as evidence to identify what types of AI competencies should be emphasized from both theoretical and empirical perspectives. First, it is important for educators to be aware of the opportunities and challenges when employing AIED technology (European Commission, 2022 ). Therefore, this article first explores the opportunities and challenges associated with AIED technology and identifies needs related to AI competencies. Second, the identified AI competencies are synthesized for better alignment with existing competence frameworks. In doing so, the new proposed framework could bring significant value to teachers by equipping themselves with necessary AI competencies so as to facilitate better teaching, learning and assessments in the post-pandemic context.

This paper will guide readers through the conceptualization, integration, and applications of the DigCompEdu framework, and revise P21’s framework for twenty-first century learning to accommodate AI technologies. This article extends the proposal of European Commission ( 2022 )’s guidelines that views AI as an important twenty-first century competency.

Motivation for teachers’ AI digital competency

Ai turns online teaching crisis to opportunities.

Due to the Great Online Transition (GOT), instructors have tried to respond to the pandemic and make changes in response to the global remote and online teaching (Howard et al., 2022 ). Among digital technologies, the use of AIED technology for online learning has become more popular than before to help educators to address teaching challenges such as social isolation, heavy workload and students’ lack of motivation (Westera et al., 2020 ; Zhu et al., 2020 ). During the pandemic, school administrators and teachers have been seeking better alternatives to improve their teaching, enhance students’ interactions (Ahuja & Nair, 2021 ; Kexin et al., 2020 ), and facilitate administration (Chen et al., 2020a , 2020b ). For example, intelligent tutoring systems propose automatic personalized suggestions and tasks according to learners’ profiles (Cao et al., 2021 ; Kochmar et al., 2020 ). AI-driven learning platforms record students’ behavior and interaction sequences for educators to further analyze to understand their progress in their online learning (Tang et al., 2021 ), and recommend personalized learning resources for students (Whalley et al., 2021 ).

These benefits could facilitate teachers to address various online teaching challenges (e.g., learning diversity, motivation problem, social interaction) during the pandemic. AI technologies provide students with learning opportunities that facilitate teachers and students with interactive, personalized and just-in-time feedback (Dizon, 2017 ). AIED technology helps cater for different learners’ needs on a case-by-case basis, addresses their needs during the online lessons (Chiu, 2021 ), and supports learners to overcome their learning disabilities and accommodate their learning styles (Ouherrou et al., 2019 ).

Overall, the rapid shift to online education during the COVID-19 pandemic has accelerated the incorporation of AI technologies into education systems around the world (Hwang et al., 2022 ). AIED applications offer effective support for teachers and students to facilitate their teaching and learning in different subject areas (e.g., language, medical education) (Ahuja & Nair, 2021 ; Liang et al., 2021 ). The way how educators teach and how students learn has been dramatically influenced by AI technologies to enhance students’ learning outcomes, achievements and attitudes in the post-pandemic world.

For many teachers, it may be their first time to use AIED technologies for teaching. They may not have rich experience of using AIED technologies to teach, and they face various challenges such as technological difficulties, communication and collaboration problems when using these novel-to-teacher technologies (Kim et al., 2022 ). Teachers who are more capable of using AI-driven technologies tend to adapt more towards the digital transformation, and facilitate their teaching and administrative work (Huang, 2021 ).

To support educators with such digital transformation, it is necessary to understand what challenges they face when using AIED technologies so as to prepare them with adequate digital competencies to overcome these difficulties. They need to choose suitable AI-driven tools to connect their subject knowledge, and facilitate their teaching and classroom management. For example, voice-driven AI such as Siri is a good choice to teach language and lifelike interactional skills. Online examination software with facial recognition is applied to conduct tests and examinations in a fair way and reduce plagiarism issues in online learning settings (Pandey et al., 2020 ). These technologies provide great potential for online educators to address their teaching challenges. However, they may be novel to educators, and not all of them are familiar with these technologies, which require teachers’ technological knowledge and skills. Since AI brings opportunities to online teaching, there is a need to update teachers’ digital competency to facilitate their teaching and work (Ng et al., 2022b ; Zhang & Aslan, 2021 ).

Opportunities of using AIED for online teaching

AI-driven technologies bring opportunities for enhancing students’ learning experience through intelligent tutoring, individualized learning and recommendation systems (e.g., Hwang et al., 2022 ; Zawacki-Richter et al., 2019 ). In the educational field, it is estimated that 47% of learning management tools will be enabled with AI capabilities by 2024 (Schmelzer, 2019 ). AI-driven systems can develop custom learning profiles for each student and customize their learning journeys and materials based on their needs, ability, preferred mode of learning, and experience (Fu et al., 2020 ). However, to many educators, it is their first experience to use AI technologies in an online learning environment, and they may not have adequate knowledge and skills to manipulate these AI applications (Guerrero-Roldan et al., 2021 ). This provides reasons why educators need to equip themselves with related digital competencies so that they can teach and learn effectively in online learning environments.

Many other publications highlight opportunities of using AI to conduct online teaching. For example, Seo et al. ( 2021 ) proposed that AI could facilitate teachers’ generation of repetitive questions, and offer students learner-instruction connections, just-in-time personalized support and meaningful automatic communication for classmates. Torda ( 2020 ) used Sophya AI, an artificial intelligence system, as an example to reach larger audiences easily and enable more condensed access to experts. The AI-driven system could enhance instructor/machine-student interaction through the chat features. Moreover, AI elements provide high computational power, which simulates authentic environments (e.g., plant and animal growth as time passes) and acts like humans as non-player characters. Educators or instructional designers could program the pre-defined rules for learners to learn with virtual agents (tutors, peers and tutees) (Hwang & Chien, 2022 ). Torda ( 2020 ) incorporated simulations and 3D space for clinical practice and skill development into AI-driven systems. Another study by Ratten ( 2020 ) adopted AI technologies to create a sense of online community that strengthens bonds among students and enables them to express their knowledge in different digital formats. The study proposed that AI could be a meaningful teaching tool complementing existing pedagogical methods and providing a bridge between reality and simulated business environments. Havenga ( 2020 ) adopted AI robotics to introduce computer science concepts for engineering students via online collaborative problem-based learning. However, these features require teachers to connect the digital tools to content knowledge and pedagogy, and customize the settings before teaching so as to compromise the learners’ continuity in the courses, personalize their learning, and encourage students to express and gain knowledge via AI-driven technologies.

On top of empowering students to engage in AI learning environments, teachers who can use AI tools can enhance their teaching efficiency and handle their routine administrative tasks (e.g., grading, repetitive paperwork). AI assists teachers in designing instructional content that suits students’ needs to personalize students’ learning through task automation. For example, Yang et al. ( 2022 ) embedded a predictive model into the AI-driven system to facilitate teachers’ assessment and analysis of learners’ performance to detect the potential at-risk learners at early stages. It offers timely cognitive support in the course and effectively enhances students’ academic performance. With data analysis, teachers can adjust their teaching strategies and prepare appropriate learning materials to customize students’ learning throughout the courses. Teachers can adjust their teaching strategies to address the changing learning situations and learning goals from either educators or online learning platforms. Likewise, Whitelock-Wainwright et al. ( 2021 ) found that teachers could make good use of learning analytics to offer students automatic feedback to improve course outcomes. Abu-Dalbouh ( 2021 ) applied a data mining strategy to predict students' online learning performance during the pandemic to enable higher education educators to improve course outcomes, plan ahead to enhance student performance, provide better understanding of student enrolment structure in the course, and update and improve their decision, training, policies and methods for students. However, not all educators have the technological background and experience to operate different AI-driven equipment and systems. There is a need for educators to equip themselves with AI competencies to support them.

AIED technologies offer teachers new features and functionalities (e.g., chat function, personalized support, automatic communication, learning analytics) to facilitate their teaching. Teachers who are capable of using them for teaching can enhance their teaching effectiveness (Healy & Blade, 2020 ; Whitelock-Wainwright et al., 2021 ), motivate students’ learning, raise students’ self-efficacy, promote their self-regulation (Seo et al., 2021 ; Guerrero-Roldan et al., 2021 ), and help students interact with other learners in AI-driven environments (Torda, 2020 ). Teachers need to timely grasp the opportunity to develop their AI digital competency so as to enrich students with better (online) learning experiences.

Challenges of using AIED for online teaching

Teachers may not be familiar with these novel technologies to facilitate their teaching in terms of technical and other broader aspects (e.g., communication, collaboration, multidisciplinary skills). When designing an AI-driven learning environment, teachers may face a number of challenges such as technical difficulties in enabling students to use AI applications and compile algorithms (Vazhayil et al., 2019 ), as well as insufficient funding, immature AI curricula, lack of tools or evaluation methods (Ng et al., 2022c ).

First, AI-driven platforms offer a new way of creation and delivery of instructional content (Ratten, 2020 ). Teachers are now challenged by digital transformation to meet new requirements that have not been part of the traditional repertoire of expectations for effective teaching during their professional teacher training. They feel challenged when meeting complex demands and new trends (e.g., online learning, AI education) in their classrooms. In the previous section, AI was found to facilitate teaching and administrative work; however, it turns out that teachers need to get rid of various technical problems and need additional time and resources to adapt to these AI technologies (e.g., Hwang et al., 2020 ; Luan et al., 2020 ). Teachers may not get ready with technological knowledge and skills that are essential for such digital transformation. Studies found that technical difficulties could seriously reduce the quality of teachers’ delivery of content, instructional design, and assessments (e.g., Seo et al., 2021 ; Torda, 2020 ). To ensure well-qualified teachers in AI-enhanced classroom, studies suggested that teachers would need to equip themselves with AI-related technological skills to facilitate students’ knowledge acquisition and expression (Healy & Blade, 2020 ), and interact with learners using AI technologies (e.g., chatbots, automatic feedback) (Guerrero-Roldan et al., 2021 ; Whitelock-Wainwright et al., 2021 ). Therefore, teacher competencies have become necessary to enhance students’ AI-driven online learning. Teachers need to upgrade their skills and knowledge, and connect the tools to content knowledge and pedagogy through continuous professional training (Kim et al., 2021 ) such as technical support, guidelines, and teacher education programs (Chiu & Chai, 2020 ; Luan et al., 2020 ). These can help teachers become well-prepared to reduce socialization gaps, technical issues, and barriers that prevent AI systems from achieving their intended goals.

Apart from technical skills, developing a positive leadership attitude and ethical mindset is important for educators to use AIED technologies for teaching. Some teachers worry that AI could replace them (Selwyn, 2019 ), and they feel negative about relying on AI interpretation to understand students’ social interaction cues (Seo et al., 2021 ). In fact, researchers have pointed out that AIED technologies may be a “black box” that teachers may not know the working mechanism behind how AI provides such judgements and recommendations for learners (Pereira et al., 2021 ). Several potential risks and conflicts such as privacy concerns, changes in power structures, and excessive control have been identified between students and teachers due to such misunderstanding or misleadingness (Seo et al., 2021 ). Seo et al. ( 2021 ) warned that AI could give unreliable recommendations which may negatively impact students’ performance, especially when teachers solely rely on AI-driven technologies to predict and assess students’ learning outcomes. AI-driven platforms can probably misunderstand users and offer misleading suggestions for learners (Seo et al., 2021 ). These platforms could probably be trained and developed by certain groups of learners and may not be universal to all. In this way, educators should know the ethical concerns and limitations behind the AI-driven technologies. For example, AI systems should not offer standardized support for all students, and student learning outcomes and social interaction should not merely rely on AI interpretation. Lastly, the design of such platforms may not be human-centered (or even student-centered) enough, so it may cause students’ discomfort because features such as eye tracking or facial expression analysis feel like surveillance to students (Seo et al., 2021 ). Overall, instead of worrying that AI will replace teachers’ roles of socialization and mentoring in a physical learning environment one day (Torda, 2020 ), teachers need to become AI literate and learn about the ethical concerns, limitations and human-centered design behind AI technologies to facilitate students’ learning process.

AI competencies

The term ‘digital competencies’ refers to a set of skills that everyone needs to live, learn and work in a society where people need to communicate and access relevant information through digital technologies such as internet platforms, social media and mobile devices (Falloon, 2020 ; Ng et al., 2021a , 2021b ; Ng, 2012 ). In recent years, AI technologies such as robotics, chatbots, and smart devices have become common in our daily life. However, people may not understand the technologies, principles and ethical concerns behind (Ng et al., 2021a , 2021b ). With more age-appropriate technologies, educators begin to design meaningful curriculum and pedagogy to develop students’ related knowledge, skills and attitude to facilitate their learning, living and working (Su et al., 2022 ). AI competency has become one of the important twenty-first century technological skills nowadays. With AI competencies, people can critically evaluate AI technologies, communicate and collaborate effectively with AI, and use AI as a tool online, at home, and in the workplace (Long & Magerko, 2020 ).

Recent studies and reports have proposed important digital competencies to guide what abilities and skills people should gain. The Five Big Ideas in AI serve as a K–12 education framework to articulate what all K-12 should learn about AI in terms of four foundational concepts (perceptions, representations, reasoning, learning, natural interaction and societal impact) (Touretzky et al., 2019a , 2019b ). Long and Magerko ( 2020 ) proposed 16 competencies that people should learn: recognizing AI, understanding AI, interdisciplinary skills, distinguishing general and narrow AI, identifying strengths and weaknesses of AI, imagining future applications of AI and their societal impacts, knowledge representations, decision making, understanding machine learning, recognizing the roles of human in AI, data competency, learning from data, critically interpreting data, higher-level reasoning of AI, sensors, and ethical concerns behind. The two proposed sets of competencies form the basis of what AI competencies students should learn, such that educators can design instructions and assessments around these competencies. Taking a step further, Ng et al. ( 2021b ) categorized the necessary AI competencies into four cognition domains (i.e., know and understand; use and apply; evaluate and create; and ethical issues) to foster students’ AI knowledge from low to high thinking skills, inspired by Bloom’s Taxonomy. For example, “recognizing and understanding AI” suggested by Long and Magerko ( 2020 ) was categorized to the level of “know and understand”; “critically interpreting data” would be related to the level of “evaluate and create”; while societal impacts” by Touretzky et al., ( 2019a , 2019b ) was grouped into “ethical issues”. This model helps teachers understand what AI competencies that students need and develop models of learning to implement meaningful instructional design and pedagogies to enhance students’ learning outcomes.

Teachers’ AI digital competency

The aforementioned discussions suggest some opportunities and challenges of educators’ use of AI for teaching. There is a lack of frameworks or guidelines to inform educators what particular digital competencies are necessary to help students become empowered learners. As suggested by Ng et al. ( 2021b ), only few existing studies discuss how teacher education programs could strengthen teachers’ AI digital competency to use AI for teaching, learning and assessment. Ng et al. ( 2022d ) further suggested a set of teacher’s AI competencies, including using basic applications, managing information, creating learning content, and connecting their students via technology. Xu ( 2020 ) proposed that developing AI digital competency is important for educators. Teachers who know how to use AI may replace the teachers who do not know how because AI can empower teachers and promote their role transformation, which greatly improves the efficiency of management and the level of decision-making (Ng et al., 2022b ; Vazhayil et al., 2019 ). Markauskaite et al. ( 2022 ) suggested that educators need to integrate new digital technologies and support learning to meet educational standards through digital technologies, engage in professional learning to build competencies, and gain experience using AI-enabled tools. Moreover, they should learn how to use appropriate AI-driven technologies such as adaptive learning systems and intelligent agents to facilitate their daily teaching management and practices to collaborate with different parties (e.g., parents, colleagues), enhance personalized learning to understand students’ learning progress and needs, and conduct various tasks such as offering automatic feedback, self-diagnosing, and promoting online collaboration among learners (Cavalcanti et al., 2021 ). On top of using AIED technologies, they also need to update their pedagogical and content knowledge on AI and learn how to develop suitable pedagogies (e.g., collaborative learning, problem-based learning), digital resources, learning materials and assessments to empower learners (Vazhayil et al., 2019 ). This is consistent with Ng et al. ( 2021b )’s review that revises the Technological, Pedagogical and Content Knowledge (TPACK) framework to inform teachers’ competencies and understanding of how AI can design their teaching and learning. The TPACK framework has been adopted in research of teachers’ technology integration and offers a nuanced perspective on teachers’ digital competencies via multiple types of knowledge (Koehler et al., 2013 ; Scherer et al., 2023 ). Content knowledge describes teachers’ own knowledge of the subject matter. Pedagogical Knowledge describes teachers’ knowledge of their teaching and learning practices, processes, and approaches. Technological Knowledge describes teachers’ knowledge of, and ability to use, various technologies, technological tools, and digital resources (Falloon, 2020 ).

Educational frameworks for educators

Educators play a leading role in realizing the potential benefits of using AI in education. It is important that teachers and school leaders are aware of and appreciate the opportunities and challenges of employing AI systems mentioned in previous sections, and how they can enhance teaching, learning and assessment. To develop teachers’ AI competencies, the European Framework for the Digital Competence of Educators (DigCompEdu) is complemented in Section  “ DigCompEdu framework ”, and revised and adapted the P21's frameworks for twenty-first century learning in Section “ P21’s framework for 21st century learning ”.

DigCompEdu framework

As teachers face rapidly changing demands, educators need to acquire a more sophisticated set of competencies than before (European Commission, 2022 ), especially when using digital technologies to help students become digitally competent. The DigCompEdu offers a sound framework that provides a guideline for educators to help educators implement tools and design their learning programs. It is an educator-specific competency framework that defines and describes teachers’ key competencies, and proficiency levels which provide a general reference to support the development of educator-specific digital competencies (Caena & Redecker, 2019 ). This model includes a wide range of components organized within six major areas: (1) professional engagement, (2) digital resources, (3) teaching and learning, (4) assessment, (5) empowering learners, (6) facilitating learners’ digital competency (European Commission, 2022 ) (see Fig.  1 ).

DigCompEdu framework for teachers’ AI competency (European Commission, 2022 )

Professional engagement

Teachers’ digital competency is important for enhancing teaching and facilitating their professional interactions with colleagues, learners, parents and other parties (Redecker, 2017 ). With digital affordances of AI technologies, teachers should consider different AI-driven tools and systems to help them develop and improve organizational communication strategies. AI could enhance organizational communication with other teachers, and enable teachers to share and exchange knowledge, teaching experiences and pedagogies (Bryant et al., 2020 ; Elnaggar & Arelhi, 2021 ).

Digital resources

Educators are currently confronted with a wealth of AI-driven learning resources they can use for teaching (e.g., Archambault et al., 2022 ; Liu et al., 2021 ). First, AI can support teachers to manage teaching resources, facilitate their teaching, as well as source, create and share resources to fit learning goals, needs and teaching style (Archambault et al., 2022 ). For example, AI recommendation engines can support teachers to recommend specific learning activities and resources based on students’ preferences, progress and needs (Klašnja-Milićević et al., 2015 ). Moreover, there are many free and open-source learning resources and tools online (e.g., Code.org, Teachable Machine, Microsoft AI900 learning resources) (Ng & Chu, 2021 ). Teachers should identify, select, modify and build on these existing AI resources and technologies for teaching and learning. They need to think about how to incorporate these resources according to different specific learning goals, learning environment, pedagogy, and learner group, when designing digital resources and planning their use.

Teaching and learning

When discussing how digital technologies can help teaching and learning, the DigCompEdu suggests four major elements, namely (1) teaching, (2) guidance, (3) collaborative learning, and (4) self-regulated learning (Vazhayil et al., 2019 ). It is believed the combination of these elements can prepare educators for getting ready for AI teaching and learning.

First, to plan for and implement digital technologies in the teaching process, teachers need to manage and design their interventions and develop pedagogical approaches wisely (Leung et al., 2021 ). With various AI technologies to support instruction, teachers need to restructure the lessons, activities and learning content to best support learning objectives. Among the pedagogical approaches, Ng et al. ( 2022c ) suggested that collaborative learning, project-based learning and learning with game elements for secondary students were the top three effective approaches to facilitate students to solve authentic problems. Second, to offer timely and targeted guidance and assistance, AI helps teachers to respond promptly to learners’ questions and doubts. For example, intelligent agents and chatbots could provide personalized learning through the use of natural language processing to offer students timely guidance and feedback (Tisdell, 2018 ; Zawacki-Richter et al., 2019 ).

Third, collaborative learning is important for students to solve problems, complete tasks or create products with joint effort (Linden et al., 2000 ). Nowadays, AI technologies should help teachers to foster and enhance students’ collaboration, communication, and knowledge co-construction. Zheng et al. ( 2019 ) adopted a machine learning classification to understand students’ contributions and comment patterns on a virtual learning environment to support students’ collaborative learning in STEM education. Fourth, recent research has drawn attention to how AI technologies permit more adaptive support and guidance for learners. These adaptive systems help students develop self-regulated learning which refers to a set of learning abilities (e.g., goal setting, self-monitoring, self-instruction, self-reinforcement) for students to understand and control their learning environment (Panadero et al., 2017 ). AI can support self-regulated learning processes; for example, by enabling learners to plan, monitor and reflect on their own learning, and making learning progress (Kay & Kummerfeld, 2019 ). Online learning platforms offer a mechanism and interfaces that enable learners to scrutinize and control their learning data and models, and support their meta-cognitive processes (Kay & Kummerfeld, 2019 ).

When integrating AI technologies with assessment practices, teachers need to consider how AI can enhance existing assessment strategies. AI can support teachers in creating innovative assessment approaches (Chassignol et al., 2018 ; Chen et al., 2020a , 2020b ). For example, AI-driven writing assistants can evaluate and grade students’ written work automatically, and identify features such as word usage, grammar and sentence structures to grade and provide feedback (Ramesh & Sanampudi, 2021 ). Chatbots can serve as virtual teacher assistants to ask students questions with simple instruction and provide students with directions with a range of questions (Smutny & Schreiberova, 2020 ). AI can help teachers to understand learners’ learning behavior by analyzing and interpreting data generated from AI systems. This helps teachers make better decisions and refine their learning interventions. For example, AI-driven learning analytics dashboard applications are applied to visualize learning patterns such as utilization, satisfaction and learning achievement (Verbert et al., 2013 ), and identify the feedback from students and teachers (Sedrakyan et al., 2020 ). Overall, teachers should learn how to utilize various types of AI technologies to direct monitoring learner progress, facilitate provision of feedback and allow themselves to assess and adapt their teaching strategies.

Empowering learners

AI technologies have the potential for supporting learner-centered pedagogical strategies, classroom differentiation and personalized learning to engage students in their learning process (Ouyang & Jiao, 2021 ). The use of AI in differentiated learning enables personalized learning, which was not possible in the past when teaching large classes (Renz & Hilbig, 2020 ). It enables teachers to understand the students’ learning strategies, background, progress and academic interests (Ouyang & Jiao, 2021 ). It addresses students’ diverse learning needs, by allowing them to advance at different levels and speeds and follow their learning pathways. Second, it can help reduce learners’ gap due to inequality issues, promote and ensure accessibility for all learners, including those with special educational needs. AI can help ensure accessibility to learning resources and activities. For example, NWEA (a Microsoft project) makes mathematics assessment more accessible for students with vision disabilities, which can exclude students from higher-level STEM careers (Microsoft, 2022 ). Other examples remove accessibility barriers through image and facial recognition for students with visual impairment, lip-reading recognition for students with hearing impairment, and real-time captioning and translations for students with hearing impairment and those who do not speak the language (Martinez, 2022 ).

Facilitating learners’ AI competency

Educators enable learners to creatively and responsibly use AI technologies for information, communication, content creation and problem solving. The DigCompEdu framework proposes that educators need to equip themselves with five competencies: (1) information and media literacy skills, (2) digital communication and collaboration, (3) digital content creation, (4) responsible use of AI, and (5) digital problem solving.

First, information and media literacy skills are important for educators who need to incorporate AI into learning activities and assessments to fulfill students’ information needs (e.g., find resources in AI-driven environments; organize, analyze and interpret information using AI). Second, educators need to enable students to effectively use AI for communication and collaboration. When students want to share their files publicly online, teachers need to be aware that the materials can be used to train AI on social media platforms (e.g., family photos, comments), which may lead to privacy issues (Westerlund, 2019 ). Third, AI can automatically create digital content (e.g., texts, news, essays, images) using existing digital content as its source (Salminen et al., 2019 ). The AI-generated content may be indistinguishable from human creations. Educators should incorporate learning activities and assessments for students to create content through creative writing (Clark et al., 2018 ), music composition (Lopez-Rincon et al., 2018 ) and stylising painting (Hertzmann, 2018 ). Third, it is important for teachers to be aware of the ethical concerns behind AI systems. Teachers need to take measures to ensure students’ psychological and social well-being (e.g., self-image, self-efficacy) while using AI technologies. They need to recognize potential risks, ethical and safety concerns when using AI technologies for teaching, learning and assessment. They also need to remind their students of these issues. Recent ethical guidelines have been published to advise educators to pay attention to the ethical use of AI and data in teaching, learning, and assessment (European Commission, 2022 ; Holmes et al., 2022 ). Finally, AI can facilitate teachers’ work and enable them to solve teaching problems, and empower learners to be creative problem solvers (European Commision, 2022). Teachers need to enhance their pedagogical and technological competencies to design appropriate learning environments for students to solve authentic problems using AI with their classmates.

P21’s framework for twenty-first century learning

This section investigates what teacher competencies are desired for effective online teaching with AI technology using the P21’s framework for twenty-first century learning. Based on the model, four key competencies that teachers have been summarized. The key competencies of teachers should not only focus on acquisition of basic AI knowledge and skills, but also cultivation of the qualities necessary for the teacher’s adaptation to, survival in and control over future society as well as teachers’ lifelong professional development.

The P21’s framework for twenty-first century learning is famous for educators and business leaders to illustrate the knowledge and skills that they need to succeed in working, learning and living (National Research Council, 2012 ; Ng et al., 2022a ). The framework is useful for teachers to establish effective learning standards and assessments, curriculum and instruction, professional development and learning environments. This paper does not focus on the digital competency of technological/ computer science knowledge. Instead, it aims to focus on broader digital competencies (e.g., instructional communication, collaboration) that support teachers to conduct online teaching using AI technologies. From this perspective, teachers should not be limited to knowing and using AI applications for empowering their students and preparing related teaching resources. The framework requires a further update on the other competencies about AI competency such as applying knowledge to different disciplines, demonstrating creativity and life and career skills, and communicating and collaborating with their students.

Core subjects, 3Rs and twenty-first century themes

Mastering key subject areas, 3Rs (reading, writing, arithmetic), and twenty-first century themes is essential for students to succeed in work and life (Kay & Greenhill, 2011 ). AI technologies have been used in applications for learning in almost every discipline such as language and arts (Liang et al., 2021 ), and STEM related areas (Zawacki-Richter et al., 2019 ). This extends to twenty-first century themes including global awareness (Kong et al., 2021 ), environmental, financial, civic, and health competency (Guerrero-Roldan et al., 2021 ), as well as the 3R competency (i.e., reading, writing and arithmetic) (Kandlhofer et al., 2016 ). Universities could educate the teacher workforce to integrate AI applications throughout students’ lifecycle, to harness opportunities of using AI technologies in their fields to empower their students, and be aware of ethical implications and risks (Zawacki-Richter et al., 2019 ).

Teachers could use different AI tools (e.g., virtual laboratories, music pieces generation, chatbots) to enable students to express their subject (or even multidisciplinary) understanding and reach higher cognitive levels such as creativity, collaboration and communication. In this way, teachers have increased accessibility to empower their students to express knowledge in multi/interdisciplinary themes and subjects such as music (Miranda, 2021 ) and language learning (Liang et al., 2021 ). In higher education, universities across the globe have started to design AI learning programmes for students from diverse educational backgrounds (e.g., radiology, architecture majors) to develop their foundational AI knowledge, and equip themselves with skills and mindset to solve authentic problems using AI applications (Kong et al., 2021 ; Ng et al., 2022a , 2022b , 2022c , 2022d ). Some courses focus on algorithms and programming for science and engineering students who need to learn underlying computer science concepts behind AI technologies (Long & Magerko, 2020 ).

Regarding the 3Rs, many AI technologies can help students to read, write and calculate in their learning cycle. For example, AI-driven book recommendation systems could suggest reading materials, practices and assignments according to learners’ reading habits (Wu & Peng, 2017 ). When writing essays, students could use AI writing assistants and content generators to paraphrase texts, reduce grammatical errors and become faster writers (Lin & Chang, 2020 ; Nazari et al., 2021 ). Moreover, some applications (e.g., Wolfram Alpha, Symbolab) allow students to input photos of mathematical formulae to generate steps to solve complex problems by clicking a button. Suitable tools could help students solve problems and empower their learning. However, plagiarism problems may occur especially when students use these applications to complete their assignments and examinations (Francke & Alexander, 2019 ).

Learning and innovation skills

The skill sets include creativity and innovation, critical thinking and problem solving, communication skills and collaboration (Van Laar et al., 2017 ). Recent studies have suggested the use of AIED technologies to enhance learners' learning and innovation skills, which are essential to enhance their working and learning efficiency. Demchenko et al. ( 2021 ) identified a digital transformation in legal education that teachers need to equip their students with AI competencies. There is a need for law schools to innovate and form stronger interdisciplinary collaboration with AI experts to enhance their effective professional and everyday activities such as using AI-based tools to aid human judgement and identifying algorithmic bias (Yang et al., 2022 ). In business education, AI has a wide range of use in business to complete authentic tasks such as aggregating business data, managing customer relationships and predicting future trends. Business educators need to update their knowledge and enable students to integrate AI into their workplace and create a new user experience for their clients (Uzialko, 2022 ; Williamson & Eynon, 2020 ). These knowledge and skills are useful for teachers and students to become professionals/leaders in their knowledge fields to implement complex cognitive and decision-making tasks, and adapt to present scenarios.

Information, media and technology skills

Educators need to prepare themselves for becoming digital ready so that they will be able to teach students related skills such as information, media, and ICT competencies (Gleason & Von Gillern, 2018 ). Especially in recent years, teachers and students need to adapt to digital transformation and develop related technological skills. In AI-driven classrooms, teachers need to manipulate different AI-enhanced systems to design assessments, and examine students’ performance using their historical and current data using the adaptive learning system (Guerrero‑Roldán et al., 2021 ). In another study, teachers adopted an automatic mode in an AI-driven service called IBM RXN to enable students to draw target molecules, and generate chemical reactions and structure representations (Healy & Blade, 2020 ). Kostopoulos et al. ( 2021 ) introduced the use of an AI-driven system called DevOps to equip smart city professionals and educators with adequate technological skills to visualize urban innovation in an 11-week online course. These examples show that university educators from different disciplines need to equip themselves with technological skills to enable their students to express ideas, solve problems and manipulate AI-driven applications so that they will be ready to work in AI-driven environments.

Life and career skills

Life and career skills are important for preparing students for engaging as citizens in a dynamic global community and facing different challenges and opportunities in the workplace. Students need to develop a positive mindset, positive attitudes and other competencies (e.g., flexibility, adaptability, self-direction, social skills, productivity, responsibility) to navigate complex life and work environments (Van Laar et al., 2017 ). First, studies have suggested that AI has the potential for transforming youth employment, and students need to develop relevant skills to adapt to this change. For example, Singh et al. ( 2020 ) suggested that AI profiling would move away from merely collecting information about formal qualifications to a more holistic approach of capturing skills and life experiences. Educators need to upgrade their students to fit the future job market. Second, improving students’ self-efficacy and self-regulation is important when using AI-driven systems to support students’ online learning since these systems seldom consist of a physical teacher to monitor their learning (Guerrero‑Roldán et al., 2021 ). Third, Cetindamar et al. ( 2022 ) highlighted four sets of workplace capabilities associated with AI: technological skills (e.g., data collection, analytics, ethics, security), work-related skills (e.g., decision making, critical thinking, teamwork), human–machine interaction (e.g., situation assessments, affordance analysis, adaptive expertise), and learning-related capabilities (e.g., lifelong learning, self-learning ability). Other studies have also highlighted the importance of life and career skills such as problem-solving (Mohammed et al., 2021 ), emotional intelligence, judgment, service orientation, negotiating and cognitive flexibility (Webber-Youngman, 2017 ), as well as communication and teamwork skills (Seo et al., 2021 ) in the fourth industrial revolution. Teachers can enable their students to become adaptive thinkers who equip themselves with technological literacies to solve problems, think critically, lead their teammates and implement reflective practice (Li & Du, 2017 ). With these life and workplace skills, students become more digitally ready to contribute to their fields and companies after graduation.

The four essential digital competencies inform how university policymakers design their educational standards for their countries/regions and schools, and provide related professional development for teachers. At the classroom level, teachers use them as guides to design suitable curricula and instruction, and create positive learning environments. Educational standards could define what AI competencies are essential to possess for university students. Such standards serve as a basis of educational reforms and digital transformation across countries/regions. Nation-wide standards can identify necessary learning outcomes such as enhancing student competitiveness, equipping students with futuristic skills, fulfilling job demands in the AI industry, and raising children to become responsible citizens. To implement the standards in classrooms, educators play an important role in making decisions on what learning elements (e.g., assessments, curriculum, instruction) should be included in the curriculum guide and learning environments. However, teachers may not be familiar with novel technology. As such, professional development is important for supporting educators to design appropriate learning activities that achieve teaching goals and learning outcomes. In this way, universities, professional organizations and companies could offer the guidelines and standards to research and develop appropriate materials, tools and platforms to support teachers and students via the standard-based AI education.

The pandemic has led to an unprecedented use of technology for education and training purposes. Among technologies, AI has presented opportunities to improve the quality and quantity of teaching, support the digitalisation of pedagogy, and inclusive remote learning, as well as resolve different online learning problems such as social isolation and motivation (Cao et al., 2021 ; Havenga, 2020 ). Online applications and systems such as search engines, chatbots (Smutny & Schreiberova, 2020 ), smart assistants (Dizon, 2017 ), language translation, online games and simulations are now equipped with AI capabilities (Martinez, 2022 ). AI promotes new ways for students to learn and apply their knowledge. It helps diagnose students’ learning problems and offer immediate assistance so as to meet learning needs of individual learners. Moreover, data generated from these applications are helpful for supporting students’ online behavior and providing feedback and recommendations to facilitate personalized learning (Pereira et al., 2021 ). AI enables teachers to analyze students’ behaviors, performance and characteristics. However, teachers may not be familiar with these novel technologies. There is a need to timely discuss what digital competencies are important for educators and school leaders to facilitate their teaching, learning and assessment (European Commision, 2022 ).

According to Ng et al. ( 2021b )’s review, four cognition domains (i.e., know and understand; use and apply; evaluate and create; and ethical issues) are proposed to support AI competencies according to Bloom’s taxonomy. They further suggested the use of the TPACK model to support how teachers develop appropriate pedagogies, content knowledge and technologies for K-16 education (Ng et al., 2021a , 2022b ). In this article, two existing generic digital competency frameworks (i.e., DigCompEdu and P21’s framework) are discussed to add values to Ng et al. ( 2021b )’s four cognition domains of AI competency and their revised TPACK model to prepare teachers’ adequate AI competencies for their teaching/learning.

Ng et al. ( 2021b ) focused mainly on the four cognitive domains from knowing to creating AI, and AI ethics that help educators to choose their content in their curricula for specific learning goals. The P21’s framework is consistent with these proposed aspects that AI competency include technological skills, and other higher-order learning and innovation skills (i.e., critical thinking, communication, creativity and collaboration). However, it further adds other critical and non-technical competencies such as interdisciplinary abilities, life and career skills to conceptualize AI competency. In fact, schools are seeking teachers and students with developed skills such as self-regulation, learning to learn, self-motivation, technological savvy, and time management (McGunagle & Zizka, 2020 ). Educators need to strengthen themselves and their students’ a wider set of twenty-first century skills and abilities to become digitally competent, instead of merely obtaining technological skills, and learning and innovation skills. Other career and soft skills such as project management, leadership, professionalism are also important capabilities to facilitate their teaching (Caena & Redecker, 2019 ). In other words, when considering Ng et al. ( 2021b )’s four cognition domains and P21’s framework, a broader set of AI literacy skills are suggested (e.g., socio-emotional, behavioral, cognitive learning, life and career aspects).

The DigCompEdu framework is wider in scope when considering educators’ AI competencies as knowledge, skills and attitudes, with the skill domain dominating over the knowledge domain. However, the learning and teaching dimension in the DigCompEdu framework is narrower in scope compared with the Ng et al. ( 2021b )’s revised TPACK model, as it focuses more on the pedagogical dimension. It does not emphasize on the specificities and constraints of different subjects (i.e., the content knowledge), and technological knowledge which DigCompEdu assumes the two elements are described in other guidelines such as learning and teaching, digital resources and empowering learners (Caena & Redecker, 2019 ). However, DigCompEdu outlines more specific actions to foster teachers to develop a high performing AI and online education. For example, teachers should learn the positive and negative impacts of AI and data use in education, and understand the basics of AI and learning analytics to enhance their professional engagement. Also, they need to know how to design and choose appropriate digital resources, understand how AI works and use various AI systems to implement their instructional design. After that, to empower students’ learning, addressing learners’ diverse learning needs is important. Teachers need to know different ways personalized learning systems can adapt students’ behavior, and explain how AI technologies can benefit all students, independent of their learning backgrounds. They make adjustments related to pedagogy, content and technologies to meet learners’ needs and learning goals. Therefore, it is important for teachers to equip themselves with necessary TPACK knowledge and learn how pedagogies that underpin a given AI system. They also need to know how the AI system addresses the learning goals and the ethical issues behind. Finally, assessment is crucial to understand students’ learning progress using AI. The assessment should take care of students' different cognitive domains. Instead of merely examining the basic understanding of AI, there is a need to examine higher levels of thinking skills such as collaboration, communication, and creativity. Overall, the DigCompEdu framework proposes specific actions that support the TPACK model.

The three models provide guidelines from different perspectives that help teachers to design instructional support and learning/teaching content to reach students’ learning goals and empower their competencies. The suggested elements of the three models have the potential to incorporate into a conceptual framework for future AI competency instructional design. The four key domains are illustrated as follows:

“Teacher professional engagement” illustrates the abilities to enhance teaching and facilitate their professional interactions with stakeholders;

“Instructional design” comprises three teachers’ inputs that they need to obtain adequate knowledge to design appropriate pedagogies, and technologies and assessment tools to pursue students’ learning goals;

“Content choices” across disciplines comprise four cognitive domains measuring students’ knowledge and skills achievement from lower (know and understand AI) to high-order thinking skills (evaluate and create AI), as well as AI ethics. The learning content can be designed to help foster students’ AI lIteracy across different subject areas (e.g., mathematics, science, language).

“Learning competencies” comprise a set of students’ knowledge, skills and values and affect (e.g., life and career skills, learning and innovation skills, technological skills) that develop students a broad set of AI competencies.

This article proposes a new framework by extending the existing versions of the two frameworks (i.e., P21, DigCompEdu) and Ng et al. ( 2021b )’s model to include AI and achieve the latest learning standards with regard to digital upskilling of the population. The new framework suggests that AI competency should not merely include technological related competency (e.g., attitudes, skills, knowledge). On top of it, it moves towards a more holistic understanding that recognizes a non-technical, critical and complex competency that young people need to learn to manipulate AI technologies ethically, safely and wisely (Ng et al., 2022c ; European Commission, 2022 ). Towards a broader picture, teachers should not view AI competency as an independent technological domain but an avenue for developing other non-technical skill sets such as life and career skills, multidisciplinary skills, learning and innovation skills. Second, it offers a revised model for teachers to develop meaningful interventions to foster their students’ AI competencies via four processes: (1) teachers’ professional engagement; (2) instructional support; (3) content choices across disciplines; and (4) students’ learning competencies (see Fig. 2 ). It iss hoped that the examples and discussion in this article will inform educators of the necessary AI competencies to support their teaching, learning and assessment in the post-pandemic world.

Instructional design framework for AI literacy/competence education

Recommendations

Measures are suggested to promote AI digital skills by providing training programs, offering personal development paths, retraining specialists in different fields, and providing grant support for educational projects (Demchenko et al., 2021 ). Studies have proposed that educators should provide guidance that helps learners maintain long-term learning motivation, and improves their digital competencies to learn independently, thereby improving the quality of learning and making learning a spontaneous behavior (Xu, 2021 ). Recommendations are suggested to help teachers develop their digital competencies:

Professional development, teacher training programs, guidelines and technical support are necessary to empower teachers to develop AI knowledge, skills and mindsets to use the teaching tools effectively.

Schools should upgrade their infrastructure and digital equipment to enhance effective digital capacity and development.

A lot of online technologies are used in education to complement AI-driven learning experiences such as Metaverse, blockchain, cloud computing and big data. Teachers should always update their knowledge and learn the potential of using these technologies in their classrooms to be digital competent.

Teachers should not merely focus on technological knowledge and skills. Instead, they need to develop other important skill sets such as life and career skills, multidisciplinary skills, learning and innovation skills, as well as ethical mindsets and risks.

More digital competency frameworks should be proposed to inform the key competencies for educators to establish effective learning standards and assessments, curriculum and instruction, professional development and learning environments. The EduCompEdu and P21’s framework for twenty-first century learning are two of the models that provide guidelines for teachers to develop necessary digital competencies for AI-driven learning environments.

Conclusions and recommendations

The pandemic has catalyzed a significant shift to more AI-enhanced teaching/learning in higher education. Educational institutions and universities have conducted digital transformations that include machine learning and artificial intelligence (Bygstad et al., 2022 ). However, teachers may not be AI competent enough to manipulate AIED technologies to facilitate their teaching, learning and assessment. Although they know about the existence of these AIED services, they may not understand the ethical concerns and limitations behind (Ng et al., 2021b , 2022d ), not to mention other non-technical competencies such as collaboration, teamwork, decision making, communication, and multidisciplinary skills, (Cetindamar et al., 2022 ; Demchenko et al., 2021 ). A set of challenges for teachers such as technical difficulties, ethical concerns and limitations behind when using AI tools in their learning, have been identified.

To respond to the pandemic and adapt to changes in response to the trend of online/blended learning, a set of knowledge and skills are proposed for teachers based on the EduCompEdu framework and the P21’s framework for 21st Century Learning. Evidence is suggested to show what digital competencies are important for university educators to equip themselves with necessary technological skills to enable their students to express ideas, solve problems and manipulate AI-driven applications so that they are ready for working in AI-driven environments. First, teachers need to learn technological and operational skills to access AI devices and software, and work with other colleagues for teaching purposes. They also need to equip themselves with other working skills such as data analysis (Kexin et al., 2020 ), assessment and evaluation using AI-driven technologies (Sánchez-Prieto et al., 2020 ). On top of these technical skills, there is a need to include broader digital competencies such as ethical concerns, teacher identity, attitudes and mindsets as components of teacher education in AI competency education (e.g., Akgun & Greenhow, 2021 ; Seo et al., 2021 ).

Although this conceptual paper provides the foundations for building a theoretical basis for teachers’ AI digital competency, several limitations are identified. First, the discussion is based on existing literature rather than experimental and empirical study, the proposed model is less fact-based compared with empirical research, and follow-up research is necessary to support its reliability and validity. Second, the model is subjective since the model is adapted and generated from existing studies instead of practical experimentations. However, most of the existing studies only touch on the emerging ideas (teachers’ AI digital competency). A timely paper is necessary to explore the conceptual underpinnings and accumulate the evidence to introduce a frame to facilitate how educators conduct their digital transformation using AI smoothly. Future research could address these limitations. Researchers can design meaningful AI competency programs with reference to the EduCompEdu, P21’s and Ng et al. ( 2021b )’s frameworks for teachers with experimental and empirical design. Through evidence-based research, quantitative and qualitative evidence could then support the refinement of the frameworks and programs and suggest useful recommendations for teachers.

Abu-dalbouh, H. M. (2021). Application of decision tree algorithm for predicting students’ performance via online learning during coronavirus pandemic. Journal of Theoretical and Applied Information Technology , 99 (19), 4546–4556.

Google Scholar  

Ahmad, S. F., Alam, M. M., Rahmat, M. K., Mubarik, M. S., & Hyder, S. I. (2022). Academic and administrative role of artificial intelligence in education. Sustainability, 14 (3), 1101.

Article   Google Scholar  

Ahuja, V., & Nair, L. V. (2021). Artificial intelligence and technology in COVID era: a narrative review. Journal of Anaesthesiology, Clinical Pharmacology, 37 (1), 28.

Akgun, S., & Greenhow, C. (2021). Artificial intelligence in education: Addressing ethical challenges in K-12 settings. AI and Ethics . https://doi.org/10.1007/s43681-021-00096-7

Aljarrah, A., Ababneh, M., Karagozlu, D., & Ozdamli, F. (2021). Artificial Intelligence techniques for distance education: A systematic literature review. Tem Journal-Technology Education Management Informatics . https://doi.org/10.18421/TEM104-18

Ally, M. (2019). Competency profile of the digital and online teacher in future education. International Review of Research in Open and Distributed Learning . https://doi.org/10.19173/irrodl.v20i2.4206

Archambault, L., Leary, H., & Rice, K. (2022). Pillars of online pedagogy: A framework for teaching in online learning environments. Educational Psychologist, 57 (3), 178–191.

Bryant, J., Heitz, C., Sanghvi, S., & Wagle, D. (2020). How artificial intelligence will impact K-12 teachers . Retrieved November 11, 2022, from https://www.mckinsey.com/~/media/McKinsey/Industries/Public%20and%20Social%20Sector/Our%20Insights/How%20artificial%20intelligence%20will%20impact%20K%2012%20teachers/How-artificial-intelligence-will-impact-K-12-teachers.pdf

Bygstad, B., Øvrelid, E., Ludvigsen, S., & Dæhlen, M. (2022). From dual digitalization to digital learning space: Exploring the digital transformation of higher education. Computers & Education, 182 , 104463.

Caena, F., & Redecker, C. (2019). Aligning teacher competence frameworks to 21st century challenges: The case for the European Digital Competence Framework for Educators (Digcompedu). European Journal of Education, 54 (3), 356–369.

Cao, J., Yang, T., Lai, I. K. W., & Wu, J. (2021). Student acceptance of intelligent tutoring systems during COVID-19: The effect of political influence. The International Journal of Electrical Engineering & Education . https://doi.org/10.1177/00207209211003270

Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y. S., Gašević, D., & Mello, R. F. (2021). Automatic feedback in online learning environments: A systematic literature review. Computers and Education: Artificial Intelligence, 2 , 100027.

Cetindamar, D., Kitto, K., Wu, M., Zhang, Y., Abedin, B., & Knight, S. (2022). Explicating AI literacy of employees at digital workplaces. IEEE Transactions on Engineering Management . https://doi.org/10.1109/TEM.2021.3138503

Chassignol, M., Khoroshavin, A., Klimova, A., & Bilyatdinova, A. (2018). Artificial Intelligence trends in education: A narrative overview. Procedia Computer Science, 136 , 16–24.

Chaudhry, M. A., & Kazim, E. (2022). Artificial Intelligence in Education (AIEd): A high-level academic and industry note 2021. AI and Ethics, 2 (1), 157–165.

Chen, L., Chen, P., & Lin, Z. (2020b). Artificial intelligence in education: A review. Ieee Access, 8 , 75264–75278.

Chen, X., Xie, H., Zou, D., & Hwang, G. J. (2020a). Application and theory gaps during the rise of artificial intelligence in education. Computers and Education: Artificial Intelligence, 1 , 100002.

Chiu, T. K. (2021). A holistic approach to the design of artificial intelligence (AI) education for K-12 schools. TechTrends, 65 (5), 796–807.

Chiu, T. K., & Chai, C. S. (2020). Sustainable curriculum planning for artificial intelligence education: A self-determination theory perspective. Sustainability, 12 (14), 5568.

Clark, E., Ross, A. S., Tan, C., Ji, Y., & Smith, N. A. (2018). Creative writing with a machine in the loop: Case studies on slogans and stories. In 23rd International Conference on Intelligent User Interfaces (pp. 329–340).

Demchenko, M. V., Gulieva, M. E., Larina, T. V., & Simaeva, E. P. (2021). Digital transformation of legal education: Problems, risks and prospects. European Journal of Contemporary Education, 10 (2), 297–307.

Dizon, G. (2017). Using intelligent personal assistants for second language learning: A case study of Alexa. Tesol Journal, 8 (4), 811–830.

Elnaggar, O., & Arelhi, R. (2021). Quantification of Knowledge Exchange within Classrooms: An AI-based Approach. In The European Conference on Education (pp. 1–11).

European Commission (2022). Ethical guidelines on the use of artificial intelligence (AI) and data in teaching and learning for Educators. Retrieved November 11, 2022, from https://education.ec.europa.eu/news/ethical-guidelines-on-the-use-of-artificial-intelligence-and-data-in-teaching-and-learning-for-educators

Falloon, G. (2020). From digital literacy to digital competence: The teacher digital competency (TDC) framework. Educational Technology Research and Development, 68 (5), 2449–2472.

Francke, E., & Alexander, B. (2019). The potential influence of artificial intelligence on plagiarism a higher education perspective. In Proc European Conference on the Impact of Artificial Intelligence and Robotics. EM Normandie Business School, Oxford (pp. 131–140).

Fu, S., Gu, H., & Yang, B. (2020). The affordances of AI-enabled automatic scoring applications on learners’ continuous learning intention: An empirical study in China. British Journal of Educational Technology, 51 (5), 1674–1692.

Gleason, B., & Von Gillern, S. (2018). Digital citizenship with social media: Participatory practices of teaching and learning in secondary education. Journal of Educational Technology & Society, 21 (1), 200–212.

Green, W., Anderson, V., Tait, K., & Tran, L. T. (2020). Precarity, fear and hope: Reflecting and imagining in higher education during a global pandemic. Higher Education Research & Development, 39 (7), 1309–1312.

Guerrero-Roldán, A. E., Rodríguez-González, M. E., Bañeres, D., Elasri-Ejjaberi, A., & Cortadas, P. (2021). Experiences in the use of an adaptive intelligent system to enhance online learners’ performance: A case study in Economics and Business courses. International Journal of Educational Technology in Higher Education, 18 (1), 1–27.

Havenga, M. (2020). COVID-19: Transition to online problem-based learning in robotics-challenges, opportunities, and insights. In Int. Symp. Proj. Approaches Eng. Educ (Vol. 10, pp. 339–346).

Healy, E. F., & Blade, G. (2020). Tips and tools for teaching organic synthesis online. Journal of Chemical Education, 97 (9), 3163–3167.

Hertzmann, A. (2018). Can computers create art? Arts, 7 (2), 1–25.

Holmes, W., Porayska-Pomsta, K., Holstein, K., Sutherland, E., Baker, T., Shum, S. B., Santos, O. C., Rodrigo, M. T., Cukurova, M., Bittencourt, I. I., & Koedinger, K. R. (2022). Ethics of AI in education: Towards a community-wide framework. International Journal of Artificial Intelligence in Education, 32 (3), 504–526.

Howard, S., Tondeur, J., Hutchison, N., Scherer, R., & Siddiq, F. (2022). A t (r) opical journey: Using text mining to explore teachers’ experiences in the Great Online Transition. In  Society for Information Technology & Teacher Education International Conference  (pp. 823–828). Association for the Advancement of Computing in Education (AACE).

Huang, X. (2021). Aims for cultivating students’ key competencies based on artificial intelligence education in China. Education and Information Technologies, 26 (5), 5127–5147.

Hwang, G. J., & Chien, S. Y. (2022). Definition, roles, and potential research issues of the metaverse in education: An artificial intelligence perspective. Computers and Education: Artificial Intelligence, 3 , 100082.

Hwang, G. J., Tu, Y. F., & Tang, K. Y. (2022). AI in online-learning research: Visualizing and interpreting the journal publications from 1997 to 2019. International Review of Research in Open and Distributed Learning, 23 (1), 104–130.

Hwang, G. J., Xie, H., Wah, B. W., & Gašević, D. (2020). Vision, challenges, roles and research issues of artificial intelligence in education. Computers and Education: Artificial Intelligence, 1 , 100001.

Kandlhofer, M., Steinbauer, G., Hirschmugl-Gaisch, S., & Huber, P. (2016). Artificial intelligence and computer science in education: From kindergarten to university. In  2016 IEEE Frontiers in Education Conference (FIE)  (pp. 1–9). IEEE.

Kay, J., & Kummerfeld, B. (2019). From data to personal user models for life-long, life-wide learners. British Journal of Educational Technology, 50 (6), 2871–2884.

Kay, K., & Greenhill, V. (2011). Twenty-first century students need 21st century skills. Bringing schools into the 21st century (pp. 41–65). Springer.

Chapter   Google Scholar  

Kexin, L., Yi, Q., Xiaoou, S., & Yan, L. (2020). Future education trend learned from the Covid-19 pandemic: Take artificial intelligence online course as an example. In 2020 International Conference on Artificial Intelligence and Education (ICAIE) (pp. 108–111). IEEE.

Kim, J., Lee, H., & Cho, Y. H. (2022). Learning design to support student-AI collaboration: perspectives of leading teachers for AI in education. Education and Information Technologies . https://doi.org/10.1007/s10639-021-10831-6

Kim, S., Jang, Y., Choi, S., Kim, W., Jung, H., Kim, S., & Kim, H. (2021). Analyzing teacher competency with TPACK for K-12 AI education. KI-Künstliche Intelligenz, 35 (2), 139–151.

Klašnja-Milićević, A., Ivanović, M., & Nanopoulos, A. (2015). Recommender systems in e-learning environments: A survey of the state-of-the-art and possible extensions. Artificial Intelligence Review, 44 (4), 571–604.

Kochmar, E., Vu, D. D., Belfer, R., Gupta, V., Serban, I. V., & Pineau, J. (2020). Automated personalized feedback improves learning gains in an intelligent tutoring system. In International Conference on Artificial Intelligence in Education (pp. 140–146). Springer, Cham.

Koh, J. H. L. (2019). TPACK design scaffolds for supporting teacher pedagogical change. Educational Technology Research and Development, 67 (3), 577–595.

Kong, S. C., Cheung, W. M. Y., & Zhang, G. (2021). Evaluation of an artificial intelligence literacy course for university students with diverse study backgrounds. Computers and Education: Artificial Intelligence, 2 , 100026.

Kostopoulos, G., Panagiotakopoulos, T., Kotsiantis, S., Pierrakeas, C., & Kameas, A. (2021). Interpretable models for early prediction of certification in MOOCs: A case study on a MOOC for Smart city professionals. IEEE Access, 9 , 165881–165891.

Koehler, M. J., Mishra, P., & Cain, W. (2013). What is technological pedagogical content knowledge (TPACK)? Journal of Education, 193(3), 13–19.

Lee, C. J., & Kim, C. (2014). An implementation study of a TPACK-based instructional design model in a technology integration course. Educational Technology Research and Development, 62 (4), 437–460.

Leung, J. K. L., Chu, S. K. W., Pong, T. C., Ng, D. T. K., & Qiao, S. (2021). Developing a framework for blended design-based learning in a first-year multidisciplinary design course. IEEE Transactions on Education, 65 (2), 210–219.

Li, D., & Du, Y. (2017). Artificial intelligence with uncertainty . CRC Press.

Book   Google Scholar  

Liang, J. C., Hwang, G. J., Chen, M. R. A., & Darmawansah, D. (2021). Roles and research foci of artificial intelligence in language education: an integrated bibliographic analysis and systematic review approach. Interactive Learning Environments . https://doi.org/10.1080/10494820.2021.1958348

Lin, M. P. C., & Chang, D. (2020). Enhancing post-secondary writers’ writing skills with a chatbot. Journal of Educational Technology & Society, 23 (1), 78–92.

Linden, J., Erkens, G., Schmidt, H., & Renshaw, P. (2000). Collaborative learning. New learning (pp. 37–54). Springer.

Liu, T., Gao, Z., & Guan, H. (2021). Educational information system optimization for artificial intelligence teaching strategies. Complexity . https://doi.org/10.1155/2021/5588650

Long, D., & Magerko, B. (2020, April). What is AI literacy? Competencies and design considerations. In Proceedings of the 2020 CHI conference on human factors in computing systems (pp. 1–16).

Lopez-Rincon, O., Starostenko, O., & Ayala-San Martín, G. (2018, February). Algoritmic music composition based on artificial intelligence: A survey. In 2018 International Conference on Electronics, Communications and Computers (pp. 187–193). IEEE.

Luan, H., Geczy, P., Lai, H., Gobert, J., Yang, S. J., Ogata, H., Baltes, J., Guerra, R., Li, P., & Tsai, C. C. (2020). Challenges and future directions of big data and artificial intelligence in education. Frontiers in Psychology, 11 , 580820.

Markauskaite, L., Marrone, R., Poquet, O., Knight, S., Martinez-Maldonado, R., Howard, S., Tondeur, J., De Laat, M., Buckingham Shum, S., Gašević, D., & Siemens, G. (2022). Rethinking the entwinement between artificial intelligence and human learning: What capabilities do learners need for a world with AI? Computers and Education: Artificial Intelligence, 3 , 100056. https://doi.org/10.1016/j.caeai.2022.100056

Martinez, C. (2022). Artificial intelligence and accessibility: Examples of a technology that serves people with disabilities. Retrieved November 11, 2022, from https://www.inclusivecitymaker.com/artificial-intelligence-accessibility-examples-technology-serves-people-disabilities/

McGunagle, D., & Zizka, L. (2020). Employability skills for 21st-century STEM students: the employers’ perspective. Higher Education, Skills and Work-Based Learning, 10 , 591–203.

Microsoft. (2022). AI for accessibility. Retrieved November 11, 2022, from https://www.microsoft.com/en-us/ai/ai-for-accessibility

Miranda, E. R. (2021). Handbook of artificial intelligence for music: Foundations, advanced approaches, and developments for creativity . Springer Nature.

Mohammed, Z., Arafa, A., Atlam, E. S., El-Qerafi, N., El-Shazly, M., Al-Hazazi, O., & Ewis, A. (2021). Psychological problems among the university students in Saudi Arabia during the COVID-19 pandemic. International Journal of Clinical Practice, 75 (11), e14853.

Moorhouse, B. L. (2020). Adaptations to a face-to-face initial teacher education course ‘forced’online due to the COVID-19 pandemic. Journal of Education for Teaching, 46 (4), 609–611.

National Research Council. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century . National Academies Press.

Nazari, N., Shabbir, M. S., & Setiawan, R. (2021). Application of Artificial Intelligence powered digital writing assistant in higher education: Randomized controlled trial. Heliyon, 7 (5), e07014.

Ng, D. T. K., & Chu, S. K. W. (2021). Motivating Students to Learn AI Through Social Networking Sites: A Case Study in Hong Kong. Online Learning, 25 (1), 195–208.

Ng, D. T. K., Lee, M., Tan, R. J. Y., Hu, X., Downie, J. S., & Chu, S. K. W. (2022c). A review of AI teaching and learning from 2000 to 2020. Education and Information Technologies . https://doi.org/10.1007/s10639-022-11491-w

Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021a). Conceptualizing AI literacy: An exploratory review. Computers and Education: Artificial Intelligence, 2 , 100041.

Ng, D. T., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021b). AI literacy: definition, teaching, evaluation and ethical issues. Proceedings of the Association for Information Science and Technology, 58 (1), 504–509.

Ng, D. T. K., Leung, J. K. L., Su, M. J., Yim, I. H. Y., Qiao, M. S., & Chu, S. K. W. (2022a). The Landscape of AI Literacy. AI literacy in K-16 classrooms (pp. 31–60). Springer International Publishing.

Ng, D. T. K., Leung, J. K. L., Su, M. J., Yim, I. H. Y., Qiao, M. S., & Chu, S. K. W. (2022b). AI literacy from educators’ perspectives. AI literacy in K-16 classrooms (pp. 131–139). Springer International Publishing.

Ng, D. T. K., Luo, W., Chan, H. M. Y., & Chu, S. K. W. (2022d). Using digital story writing as a pedagogy to develop AI literacy among primary students. Computers and Education: Artificial Intelligence, 3 , 100054.

Ng, T. K., Reynolds, R., Chan, M. Y. H., Li, X. H., & Chu, S. K. W. (2020). Business (teaching) as usual amid the COVID-19 pandemic: A case study of online teaching practice in Hong Kong. Journal of Information Technology Education. Research, 19 , 775.

Ng, W. (2012). Can we teach digital natives digital literacy? Computers & Education, 59 (3), 1065–1078.

OECD. (2005). Teachers matter: Attracting, developing and retaining effective teachers. Retrieved November 11, 2022, from https://www.oecd.org/education/school/34990905.pdf

Ouherrou, N., Elhammoumi, O., Benmarrakchi, F., & El Kafi, J. (2019). Comparative study on emotions analysis from facial expressions in children with and without learning disabilities in virtual learning environment. Education and Information Technologies, 24 (2), 1777–1792.

Ouyang, F., & Jiao, P. (2021). Artificial intelligence in education: The three paradigms. Computers and Education: Artificial Intelligence, 2 , 100020.

Panadero, E., Andrade, H., & Brookhart, S. (2018). Fusing self-regulated learning and formative assessment: A roadmap of where we are, how we got here, and where we are going. The Australian Educational Researcher, 45 (1), 13–31.

Pandey, A. K., Kumar, S., Rajendran, B., & Bindhumadhava, B. S. (2020, November). E-parakh: Unsupervised online examination system. In  2020 IEEE region 10 conference (TENCON)  (pp. 667–671). IEEE.

Pereira, F. D., Fonseca, S. C., Oliveira, E. H., Cristea, A. I., Bellhäuser, H., Rodrigues, L., Oliveira, D. B., Isotani, S., & Carvalho, L. S. (2021). Explaining individual and collective programming students’ behavior by interpreting a black-box predictive model. IEEE Access, 9 , 117097–117119.

Ramesh, D., & Sanampudi, S. K. (2021). An automated essay scoring systems: a systematic literature review. Artificial Intelligence Review, 55 , 1–33.

Ratten, V. (2020). Coronavirus (Covid-19) and the entrepreneurship education community. Journal of Enterprising Communities: People and Places in the Global Economy, 14 (5), 753–764.

Redecker, C. (2017). European framework for the digital competence of educators: DigCompEdu . Joint Research Centre. Retrieved November 11, 2022, from  https://ideas.repec.org/p/ipt/iptwpa/jrc107466.html

Renz, A., & Hilbig, R. (2020). Prerequisites for artificial intelligence in further education: Identification of drivers, barriers, and business models of educational technology companies. International Journal of Educational Technology in Higher Education, 17 (1), 1–21.

Salminen, J., Yoganathan, V., Corporan, J., Jansen, B. J., & Jung, S. G. (2019). Machine learning approach to auto-tagging online content for content marketing efficiency: A comparative analysis between methods and content type. Journal of Business Research, 101 , 203–217.

Sánchez-Prieto, J. C., Cruz-Benito, J., Therón Sánchez, R., & García Peñalvo, F. J. (2020). Assessed by machines: Development of a TAM-based tool to measure AI-based assessment acceptance among students. International Journal of Interactive Multimedia and Artificial Intelligence, 6 (4), 80.

Sartika, F., Ritonga, M., Lahmi, A., Rasyid, A., & Febriani, S. R. (2021). Online learning in the low internet area, planning, strategies and problems faced by students during the Covid-19 period. Artificial intelligence for COVID-19 (pp. 413–421). Springer.

Scherer, R., Siddiq, F., Howard, S. K., & Tondeur, J. (2023). The more experienced, the better prepared? New evidence on the relation between teachers’ experience and their readiness for online teaching and learning. Computers in Human Behavior , 139 , 107530.

Schmelzer, R. (2019). AI applications in education. Retrieved November 11, 2022, from https://www.forbes.com/sites/cognitiveworld/2019/07/12/ai-applications-in-education/?sh=df2d36262a38

Sedrakyan, G., Malmberg, J., Verbert, K., Järvelä, S., & Kirschner, P. A. (2020). Linking learning behavior analytics and learning science concepts: Designing a learning analytics dashboard for feedback to support learning regulation. Computers in Human Behavior, 107 , 105512.

Selwyn, N. (2019). Should robots replace teachers?I and the future of education . Wiley.

Seo, K., Tang, J., Roll, I., Fels, S., & Yoon, D. (2021). The impact of artificial intelligence on learner–instructor interaction in online learning. International Journal of Educational Technology in Higher Education, 18 (1), 1–23.

Sijing, L., & Lan, W. (2018). Artificial intelligence education ethical problems and solutions. In 2018 13th International Conference on Computer Science & Education (ICCSE) (pp. 1–5). IEEE.

Singh, S., Molina-Naar, M., & Ehlers, S. (2020). Policies for professionalisation in adult learning and education: A comparative study from India. Colombia and Denmark. Andragoške Studije, 2 , 33–61.

Smutny, P., & Schreiberova, P. (2020). Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Computers & Education, 151 , 103862.

Su, J., Zhong, Y., & Ng, D. T. K. (2022). A meta-review of literature on educational approaches for teaching AI at the K-12 levels in the Asia-Pacific region. Computers and Education Artificial Intelligence . https://doi.org/10.1016/j.caeai.2022.100065Get

Tang, K. Y., Chang, C. Y., & Hwang, G. J. (2021). Trends in artificial intelligence-supported e-learning: A systematic review and co-citation network analysis (1998–2019). Interactive Learning Environments . https://doi.org/10.1080/10494820.2021.1875001

Tisdell, C. C. (2018). Pedagogical alternatives for triple integrals: Moving towards more inclusive and personalized learning. International Journal of Mathematical Education in Science and Technology, 49 (5), 792–801.

Torda, A. (2020). How COVID-19 has pushed us into a medical education revolution. Internal Medicine Journal, 50 (9), 1150–1153.

Touretzky, D., Gardner-McCune, C., Martin, F., & Seehorn, D. (2019b). Envisioning AI for K-12: What should every child know about AI?. In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, No. 01, pp. 9795–9799).

Touretzky, D., Gardner-McCune, C., Breazeal, C., Martin, F., & Seehorn, D. (2019a). A year in K-12 AI education. AI Magazine, 40 (4), 88–90.

Uzialko, A. (2022). How artificial intelligence will transform businesses. Retrieved November 11, 2022, from https://www.businessnewsdaily.com/9402-artificial-intelligence-business-trends.html

Van Laar, E., Van Deursen, A. J., Van Dijk, J. A., & De Haan, J. (2017). The relation between 21st-century skills and digital skills: A systematic literature review. Computers in Human Behavior, 72 , 577–588.

Vazhayil, A., Shetty, R., Bhavani, R. R., & Akshay, N. (2019, December). Focusing on teacher education to introduce AI in schools: Perspectives and illustrative findings. In 2019 IEEE tenth international conference on Technology for Education (T4E) (pp. 71–77). IEEE.

Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning analytics dashboard applications. American Behavioral Scientist, 57 (10), 1500–1509.

Wang, X., Pang, H., Wallace, M. P., Wang, Q., & Chen, W. (2022b). Learners’ perceived AI presences in AI-supported language learning: a study of AI as a humanized agent from community of inquiry. Computer Assisted Language Learning . https://doi.org/10.1080/09588221.2022.2056203

Webber-Youngman, R. C. W. (2017). Life skills needed for the 4th industrial revolution. Journal of the Southern African Institute of Mining and Metallurgy, 117 (4), iv–v.

Westera, W., Prada, R., Mascarenhas, S., Santos, P. A., Dias, J., Guimarães, M., Georgiadis, K., Nyamsuren, E., Bahreini, K., Yumak, Z., Christyowidiasmoro, C., Dascalu, M., Gutu-Robu, G., & Ruseti, S. (2020). Artificial intelligence moving serious gaming: Presenting reusable game AI components. Education and Information Technologies, 25 (1), 351–380.

Westerlund, M. (2019). The emergence of deepfake technology: A review. Technology Innovation Management Review, 9 (11), 39–52.

Whalley, B., France, D., Park, J., Mauchline, A., & Welsh, K. (2021). Towards flexible personalized learning and the future educational system in the fourth industrial revolution in the wake of Covid-19. Higher Education Pedagogies, 6 (1), 79–99.

Whitelock-Wainwright, A., Tsai, Y. S., Drachsler, H., Scheffel, M., & Gašević, D. (2021). An exploratory latent class analysis of student expectations towards learning analytics services. The Internet and Higher Education, 51 , 100818.

Williamson, B., & Eynon, R. (2020). Historical threads, missing links, and future directions in AI in education. Learning, Media and Technology, 45 (3), 223–235.

Wu, J. Y., & Peng, Y. C. (2017). The modality effect on reading literacy: Perspectives from students’ online reading habits, cognitive and metacognitive strategies, and web navigation skills across regions. Interactive Learning Environments, 25 (7), 859–876.

Xia, Q., Chiu, T. K., Lee, M., Sanusi, I. T., Dai, Y., & Chai, C. S. (2022). A self-determination theory (SDT) design approach for inclusive and diverse artificial intelligence (AI) education. Computers & Education, 189 , 104582.

Xu, L. (2020). The dilemma and countermeasures of AI in educational application. In 2020 4th International Conference on Computer Science and Artificial Intelligence (pp. 289–294).

Xu, B. (2021). Artificial intelligence teaching system and data processing method based on big data. Complexity, 2021 , 1–11.

Yang, G., Ouyang, Y., Ye, Z., Gao, R., & Zeng, Y. (2022). Social-path embedding-based transformer for graduation development prediction. Applied Intelligence . https://doi.org/10.1007/s10489-022-03268-y

Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education–where are the educators? International Journal of Educational Technology in Higher Education, 16 (1), 1–27.

Zhang, K., & Aslan, A. B. (2021). AI technologies for education: Recent research & future directions. Computers and Education: Artificial Intelligence, 2 , 100025.

Zheng, J., Xing, W., & Zhu, G. (2019). Examining sequential patterns of self-and socially shared regulation of STEM learning in a CSCL environment. Computers & Education, 136 , 34–48.

Zhu, M., Bonk, C. J., & Doo, M. Y. (2020). Self-directed learning in MOOCs: Exploring the relationships among motivation, self-monitoring, and self-management. Educational Technology Research and Development, 68 (5), 2073–2093.

Download references

Author information

Authors and affiliations.

Faculty of Education, The University of Hong Kong, Hong Kong, China

Davy Tsz Kit Ng, Jiahong Su, Ross Chi Wui Ng & Samuel Kai Wah Chu

Division of Integrative Systems and Design, The Hong Kong University of Science and Technology, Hong Kong, China

Jac Ka Lok Leung

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Davy Tsz Kit Ng .

Ethics declarations

Conflict of interest.

The authors have no conflicts of interest to disclose. The manuscript has not been published previously and is not being simultaneously submitted elsewhere. There are no any real or potential conflicts of interest that could be seen as having an influence on the research.

Ethical approval

No reproduction of copyrighted material is evident in this manuscript hence there is no need to apply for any necessary permission.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Ng, D.T.K., Leung, J.K.L., Su, J. et al. Teachers’ AI digital competencies and twenty-first century skills in the post-pandemic world. Education Tech Research Dev 71 , 137–161 (2023). https://doi.org/10.1007/s11423-023-10203-6

Download citation

Accepted : 22 January 2023

Published : 21 February 2023

Issue Date : February 2023

DOI : https://doi.org/10.1007/s11423-023-10203-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• AI education
• AI literacy
• Digital competency
• Twenty-first century skills
• Find a journal
• Publish with us
• Track your research

SYSTEMATIC REVIEW article

A review of research on technology-supported language learning and 21st century skills.

• School of Education Science, Nanjing Normal University, Nanjing, China

Modern society needs people to be equipped with 21st century skills (e.g., critical thinking, creativity, communication, digital literacy, or collaboration skills). For this reason, teaching and learning nowadays should promote not only students' knowledge acquisition in various learning contexts but also their 21st century skills, and language learning context is no exception. This study reviewed research on technology-supported language learning and 21st century skills. The reason is that earlier studies reviewed only articles related to language learning supported by technology and mostly focused on languages, language skills and technologies used. That is to say, 21st century skills were not considered in earlier review studies. The present study selected and reviewed 34 articles published between 2011 and 2022 (February) and focused on the following dimensions: (1) research focus such as language skills and 21st century skills; (2) theoretical foundations; (3) technologies; (4) learning activities; (5) methodology; and (6) findings. The present research found that reviewed studies had focused most frequently on such language skills as speaking and writing and on such 21st century skills as communication and collaboration. The social constructivism theory was often used by scholars to base their studies on. Facebook, Google Docs, and Moodle were popular technologies in reviewed studies to facilitate language and 21st century skills. Scholars in reviewed studies reported that technology-supported language learning activities provided learners with good learning experiences and enhanced their learning motivation, engagement, and confidence. However, some challenges that learners faced during learning activities were also reported. Based on the results of the review, this study made several recommendations for stakeholders such as educators and researchers in the field.

Introduction

It is important that our students not only acquire new knowledge when they learn, but also develop skills, such as problem-solving, social cooperation, creativity, and so on, in order to apply newly learned knowledge to the real world. Such knowledge and skills will help them adapt to modern society and will enhance their competitiveness ( Shadiev et al., 2022a , b ). Many countries have put forward the 21st century skills framework to carry out education reform ( Lin et al., 2020 ), and one of them was proposed by the Partnership for 21st Century (P21). The P21 ( Partnership for 21st Century Skills, 2008 ) provided a detailed conceptual framework and listed three types of skills: (1) learning and innovation skills (critical thinking and problem solving, creativity and innovation, and communication and collaboration), (2) digital literacy (information literacy, media literacy, and information and communication technologies (ICT) literacy), and (3) career and life skills (flexibility and adaptability, initiative and self-direction, social and cross-cultural interaction, productivity and accountability, and leadership and responsibility). The essence of these skills is that they are key skills that learners will need for their social and professional life in the future. These skills also emphasize the ability of learners to use and transfer knowledge and solve problems in complex situations, so they can achieve deep levels of individual learning as well as lifelong learning ( Shadiev et al., 2022a ).

Developing students' 21st century skills needs to be implemented in all disciplines, and foreign language learning is no exception ( Shadiev et al., In Press ). This matter has been addressed in the documents related to Asia Pacific Economic Cooperation (2004) . Furthermore, researchers have carried out related studies, and pointed out the advantages of technology in developing both language skills and 21st century skills ( Shadiev et al., In Press ). For example, Suzanne (2014) pointed out that when developing learners' reading skills, they deepened the learners' understanding of reading content, and also developed critical thinking skills. García-Sánchez and Burbules (2016 ) have found that students' skills such as problem solving, collaboration, listening and speaking improved after they completed online collaborative tasks. Srebnaja and Stavicka (2018 ) also pointed out that, in language learning projects supported by WebQuest, students' creativity, collaboration, and speaking skills have been developed. In the study by Chiang (2020 ), the digital storytelling activity was designed which promoted language learners' writing skills as well as their digital literacy skills.

A theoretical foundation to support technology supported language learning and development of 21st century skills can be built on various theories. The most relevant can be considered as second language acquisition theory, socio-cultural theory, and constructivism theory. For example, second language acquisition theory states that language acquisition is a process of input, absorption, and output. Language acquisition is acquired through exposure to contexts, understanding discourse, and then using language in natural communicative contexts ( Krashen, 1985 ). According to socio-cultural theory, learning is a social phenomenon; it emphasizes the social nature of learning and argues that the development of learners' abilities arises from interpersonal interactions ( Lantolf, 2000 ). Constructivism theory suggests that learning is a process in which a learner actively constructs meaning. That is, learners generate meaning and construct understanding based on prior knowledge and experience, often in the context of socio-cultural interactions. Constructivism theory emphasizes the social and contextual nature of learning ( Vygotsky, 1978 ). Over the years, scholars have created technology-supported learning environments for language learning and 21st century skills development based on these theories. Such environments provide students with authentic learning materials, support social interaction, and facilitate their creative expression and construction of meaning actively using the target language.

Some related review studies already exist in the field. For example, Shadiev and Yang (2020) reviewed 398 articles related to technology-assisted language learning published in 10 Social Science Citation Index (SSCI) journals. The dimensions analyzed in their study included target language, language skills, technologies, and research findings. Shadiev and Yang (2020) found that the most commonly used language was English, followed by Chinese. The most targeted language skills were writing, speaking, and vocabulary acquisition. Digital games and online videos were the most commonly used technologies in these reviewed studies. In addition, most of the reviewed studies reported positive impacts of technology applications on language learning. Zhang and Zou (2020) reviewed 57 articles on technology applications for language learning that were published between 2016 and 2019 in 10 SSCI journals. The types of technology, the purpose of technology use, and the effectiveness of the technologies were reviewed by the authors. Zhang and Zou (2020) found that mobile learning, multimedia learning and socialization, voice to text recognition, text to speech recognition, and digital game-based learning were the most frequently investigated types of technology in the literature. The purposes for their use mainly covered four areas, including promoting practice, providing teaching content, promoting interaction, and reconstructing teaching methods. Scholars have claimed that technologies have positive effects on language learning. Goksu et al. (2020) reviewed 310 articles in 10 journals in the field of technology-assisted language learning. In addition, they evaluated a metadata set of 469 articles in the Web of Science database through bibliometric mapping. The review focused on the types, purposes, and effectiveness of the latest technologies on language learning. Goksu et al. (2020) found that most studies used quantitative research methods and were carried out with participants at higher academic levels. In addition, most studies focused on language skills as well as learning motivation and learner perceptions. Shadiev et al. (2017) studied 37 articles published in the top 10 SSCI journals related to educational technology from 2007 to 2016 (March). Scholars took mobile language learning in a real environment as the research object and summarized the results from four perspectives: journal publishing trends, language learning, research focus, and research methods. The results showed that the journal publishing trend was increasing. The most common research focus was cognition and language learner proficiency. The results also showed that mobile technology was positively perceived and accepted by students in most of these studies, and the technology was also found to have a positive impact on the students' language skills.

By exploring these review studies, the present review research found that 21st century skills were not considered in these earlier studies at all because scholars mainly focused on language skills. Therefore, several important aspects (e.g., theoretical foundations used to support the studies, methodology, and types of learning activities that promote language skills and 21st century skills) were ignored. These aspects are important for stakeholders in the design and implementation of language teaching and learning for 21st century skills development. In order to fill this gap in the literature, the present study was carried out, and the following research questions were addressed:

1. What language skills and 21st century skills did the researchers focus on in the reviewed studies?

2. What theories were used as a foundation in reviewed studies?

3. What technologies were used to promote language skills and 21st century skills?

4. What learning activities were used in the reviewed studies?

5. What were the methodological characteristics of the reviewed studies?

6. What research findings were obtained in the reviewed studies?

Research Method

The present study is a systematic review. The study used preferred reporting items for systematic reviews and meta-analyses (PRISMA) for the electronic search. PRISMA is considered as a set of programs that facilitates researchers to prepare and report various systematic evaluations and meta-analyses ( Moher et al., 2009 ). According to scholars, PRISMA has been widely and successfully applied in educational research. In addition to PRISMA, this review followed the general guidelines for searching and selecting research articles proposed by Avgousti (2018) , Shadiev and Yang (2020) , and Shadiev and Yu (In Press) . The search and selection processes are shown in Figure 1 . Articles were found through a search on the Web of Science database and Peer-Reviewed Instructional Materials Online Database (PRIMO). According to Kukulska-Hulme and Viberg (2018) , PRIMO is a search tool and it contains several databases such as ERIC and Scopus. For this reason, PRIMO features a very comprehensive collection of full-text articles and bibliographic records, and it has been used by many researchers in their systematic reviews and meta-analyses to find relevant literature.

Figure 1 . The search and selection process.

Based on general recommendations from previous review studies ( Guan, 2014 ; Duman et al., 2015 ), this review used keywords such as 21st century skills, language learn * , and technology. 21st century skills were also included to widen the search results (e.g., creativity and innovation, critical thinking, problem solving, communication, collaboration, digital literacy, information literacy, media literacy, ICT literacy, flexibility and adaptability, initiative and self-direction, social and cross-cultural interaction, productivity and accountability, leadership and responsibility). This review used these terms in different combinations to search articles.

A total of 9,162 articles were found from the search. This review narrowed down the selection of research articles based on the following criteria (see Figure 1 ): articles that were (1) published during 2011–2022 (February); (2) published in English; and (3) focused on technology-supported language learning and 21st century skills. Two researchers screened each article individually and excluded articles from the study that did not focus on technology-supported language learning and 21st century skills. The researchers discussed any discrepancies in their selection results until an agreement was reached. At the end of the selection process, 34 empirical studies were chosen for the review.

This review proposed an analytical framework (see Figure 2 ) to answer the research questions of the study and to better understand the research design of the reviewed studies and findings. This review also used this framework to help us better review articles and regarded it as the basis for coding the content of reviewed studies. This review used the open coding method to carry out content analysis ( Creswell, 2002 ) which can enable us to segment research content and to form categories relevant to the phenomenon of interest. The analytical dimensions included the following (see Figure 2 ): (1) language skills and 21st century skills—skills related to language learning and 21st century skills, (2) technology—the tools and devices participants used for language learning, (3) learning activities supported by technology to cultivate 21st century skills and language skills, (4) theoretical foundation—theories, models or hypothesis involved in research, (5) methodology—participants' academic level and major, research duration, sample size, data collection tools, and research design, and (6) findings—results reported in research.

Figure 2 . Analytical framework.

Two researchers were involved in the coding process. They read articles and coded content according to the above coding scheme. After that, they categorized codes into categories and identified attributes for each category. If there were any differences in coding, the researchers re-examined an article to resolve differences, and then finally completed the coding phase. Interrater reliability was measured using Cohen's kappa coefficient and the result was high (k = 0.886).

The present study starts this section with the results related to publication year, languages, and participants. Figure 3 shows the distribution of articles published in the past 10 years. Most studies were published in 2019 ( n = 8), and no articles were published in 2012. From the figure, it can also be seen that the research trend in this field is on the rise. Figure 4 demonstrates the frequency at which different languages were the focus in the reviewed studies. 29 studies focused on English. There were also studies focused on Chinese ( n = 2), Ukrainian ( n = 1), Japanese ( n = 1), and Spanish ( n = 1). As shown in Figure 5 , undergraduates were the most common academic level ( n = 17), and there was a relatively low number of studies conducted on junior high school ( n = 5), senior high school ( n = 2), and primary school ( n = 1) academic levels. As shown in Figure 6 , researchers were more willing to involve students who were majoring in the fields of education ( n = 9), management ( n = 4), or engineering ( n = 4).

Figure 3 . Distribution of research year.

Figure 4 . Distribution of languages.

Figure 5 . Distribution of educational level.

Figure 6 . Distribution of participants' major.

Research Focus

This section presents the results related to research focus of reviewed articles. As can be seen from Figure 7 (and from Appendix 1 ), researchers carried out technology-assisted language learning studies and focused on the development of listening, speaking, reading, writing, grammar, and vocabulary skills. Among these skills, speaking skills ( n = 20) received considerable attention from researchers, followed by writing skills ( n = 19) and vocabulary ( n = 13). Reading ( n = 5) skills received less interest from researchers.

Figure 7 . Distribution of language skills.

According to Figure 8 (and Appendix 1 ), researchers pointed out that technology-supported language learning can also promote 21st century skills. These skills relate to the following three categories: 4C (communication, collaboration, critical thinking, and creativity), digital literacy, and career and life skills. The most common skills that scholars targeted were communication ( n = 15) and collaboration ( n = 15), followed by critical thinking ( n = 10) and social and cross-cultural interaction ( n = 10). Problem solving ( n = 5) skills have received the least attention from researchers.

Figure 8 . Distribution of 21st century skills.

Theoretical Foundation

This section focuses on theoretical foundation in the reviewed articles. As shown in Appendix 2 , a total of 16 theories were used. The most used theory was the social constructivism theory ( n = 9), followed by Byram's intercultural competence model ( n = 3), project-based learning ( n = 2), content based instruction ( n = 2), task based approach to language teaching ( n = 2), and sociocultural theory ( n = 2). The rest of theories were used only once.

As shown in Appendix 3 , a total of 52 technologies were used in reviewed studies. This review grouped them into eight categories: Social tools ( n = 20), Creative tools ( n = 19), Collaboration tools ( n = 13), Learning management system ( n = 9), Multimedia materials ( n = 5), Classroom interactive tools ( n = 4), Presentation tools ( n = 2), Wearable devices ( n = 1). Among the most commonly used technologies were Facebook ( n = 4), Google Docs ( n = 4), Moodle ( n = 4), followed sequentially by Skype ( n = 3), Padlet ( n = 3), WhatsApp ( n = 2), YouTube ( n = 2), Blogs ( n = 2), Google Drive ( n = 2), and Wiki ( n = 2). The other 40 technologies have only been used once, i.e., Windows Movie Maker, Live On, Edmodo, Kahoot, and Prezi. In addition, one study involved a virtual reality production tool (EduVenture) and a wearable device (Google Cardboard).

Learning Activity

As shown in Appendix 4 , in reviewed studies, scholars designed the following five main types of learning activities: (1) collaborative task-based language learning ( n = 9); (2) learning activities based on online communication ( n = 9); (3) creative work-based language learning ( n = 8); (4) adaptive learning activities ( n = 4); and (5) learning activities based on multimedia materials ( n = 4).

Methodology

This section presents methodological details of reviewed studies, such as sample size, research duration, data collection tools and research design.

As shown in Figure 9 , the most common sample size was from 11 to 30 participants ( n = 11), followed by sample sizes between 61 and 90 ( n = 8) and between 31 and 7 ( n = 7). Only two studies selected a sample size between 1 and 10. The sample size of two studies was >151. As shown in Figure 10 , most of research duration was between 3 and 6 months ( n = 10). There were 12 studies that did not state any research duration.

Figure 9 . Sample size distribution.

Figure 10 . Research duration distribution.

As shown in Appendix 5 , the most common data collection method was questionnaires ( n = 17), followed by tests ( n = 15) and interviews ( n = 13). Two data collection methods were used only 2 times, they were scales ( n = 2) and rubric ( n = 2).

As shown in Appendix 6 , research designs related to technology-supported language learning and 21st century skills were categorized into three main categories, namely quasi-experimental research ( n = 14), case studies ( n = 12), and action research ( n = 8).

As shown in Appendix 7 , various findings were reported in reviewed studies. In addition, that learners' language skills acquisition and 21st century skills, technology-supported language learning activities provided learners with good learning experiences, enhanced motivation and engagement, and improved self-confidence. In reviewed studies, some scholars reported about challenges faced by students during learning activities; they included challenges from technology, from their own competence, challenges of collaborating with others and self-attitude.

Language Skills

Regarding language skills, researchers have focused on improving learners' speaking, writing and vocabulary skills more. This shows that researchers are more concerned with the improvement of learners' skills related to language output. Researchers who reviewed studies on technology-supported language learning from 2014 to 2019 came to the same conclusion ( Shadiev and Yang, 2020 ). However, the present study showed that reading skills received the least attention, while previous studies noted that grammar skills received less attention. This revealed that researchers are now beginning to pay more attention to previously neglected skills and are beginning to focus on the role of technology-supported language learning in facilitating learners' grammar skills. For example, Lai (2017 ) noted that grammar skills improved when learners completed activities to create vocabulary lists and greeting cards using multimedia resources. Jung et al. (2019 ) noted that students' grammar skills improved as they corrected each other's pronunciation and grammatical errors through video chat. Jamalai and Krish (2021 ) found that students' grammar skills improved through online forum discussions and knowledge sharing.

21st Century Skills

In terms of 21st century skills, communication and collaboration have received the most attention from researchers. It is probably because the 21st century society is more globalized and along with the increased complexity of related work, interpersonal communication and cooperation are being enhanced. The 21st century society emphasizes teamwork skills, and therefore scholars focus on collaborative and communication skills. Problem-solving skills have received little attention, and no researcher focused on career and life skills. In the face of the evolving and changing society of the future, problem-solving skills are among the core 21st century skills, emphasizing learners' ability to define problems, think critically, and solve problems. For example, scholars in reviewed studies have focused on learners' problem-solving skills in virtual technology-supported language learning ( Chen et al., 2021 ).

Based on the results, this study has several recommendations for educators and researchers. First, input skills are an important component of language skills and an indispensable way for learners to develop output skills ( Harmer, 2007 ). The present study suggests that researchers can focus on learners' input skills supported by technology, such as listening and reading. Second, problem-solving skills and career and life skills also deserve attention; therefore, future studies try to explore the effects of technology-supported language learning on these skills.

Theories Related to Instructional Design

The most commonly used instructional design theory in reviewed studies was social constructivism theory. The results of this research are consistent with those of previous review studies of technology-supported language learning ( Parmaxi and Zaphiris, 2017 ). According to this theory ( Vygotsky, 1978 ), knowledge is not a set of “facts” but rather a synthesis of information that is actively constructed and evolving in the learner's mind. The teacher does not “give” knowledge to the learner, but the learner should acquire knowledge actively. Learners' knowledge evolves as they process old and new information, as well as their experiences. The researchers designed collaborative, creative, and communicative activities based on a social constructivism perspective to encourage learners to input the target language and output the target language in a meaningful context. At the same time, researchers have used various learning and teaching activities to promote students' collaboration, communication, creativity, critical thinking and digital literacy skills ( Yang et al., 2013 , 2014 , 2022 ; Lai, 2017 ; Sevilla-Pavón and Nicolaou, 2017 ; Huang, 2021 ).

Other researchers have also used theories based on learner-centered pedagogies such as problem-based or project-based theories. These pedagogies are all used to promote student-directed learning, adaptive learning, and personalized assessment. Learning theories were used to design activities that provided learners with opportunities for language input and output, e.g., to learn new knowledge and then apply it to the real world by creating own content. This allows learners to acquire language skills and develop 21st century skills such as communication, collaboration, and problem solving ( Arnó-Macià and Rueda-Ramos, 2011 ; Yang et al., 2013 , 2014 ; Srebnaja and Stavicka, 2018 ).

Theories Related to Language Learning

Researchers have also designed learning activities based on theories related to foreign language learning, such as task-based language teaching, content-based instruction, and output-input theory. For example, digital story creation activities and integrated cross-cultural communication activities designed by the researcher are in line with these theoretical perspectives, in which learners have access to the target language through social tools and partner communication. The ability to use creative and collaborative tools to complete target-language based tasks also contributes to the acquisition of language skills and 21st century skills development, such as social and cross-cultural interaction, communication skills ( Lewis and Schneider, 2015 ; Tseng, 2017 ).

Theories Related to Measuring Learning Outcomes

Since language learning is closely related to culture, scholars have designed foreign language courses based on cross-cultural communication, where learners acquired both language skills and cultural knowledge. Further, there are theories that have been used by scholars to assess and measure learners' outcomes. For example, researchers have focused on learners' intercultural competence along with their language skills and utilized the Byram' ICC model and the developmental model of intercultural sensitivity to measure their cross-cultural knowledge acquisition and skills development ( Bennett, 1986 ; Byram, 1997 ). In addition, the Keller' ARCS motivational model ( Keller, 1987 ) has been used by researchers to measure learners' perceived attention, relevance, confidence, and satisfaction in technology-supported language learning environments.

This review analyzed the theoretical foundation that was used by those few studies that focused on non-English languages such as Chinese, Ukrainian, Japanese, and Spanish. This review found that learning theories used by scholars in these studies were diverse. They were related to instructional design (e.g., social constructivism), language learning (e.g., language output and input), and cross-cultural learning (e.g., intercultural sensitivity).

Based on the findings, several suggestions for educators and researchers are proposed. First, the theories mentioned by researchers are instruction-related theories, language learning-related theories, and measurement-related theories; they were used to guide the design of technology-supported language learning activities that focus both on the acquisition of language skills and on the 21st century skills. These theories can be useful to inform the design of appropriate language learning activities for educators and researchers in the future. Second, this review found that many researchers did not indicate what theories were used in their studies. Theoretical foundations are important for the instructional design, language learning or measuring activities, so such information should be clearly indicated so that other researchers can gain a deeper understanding of them.

Eight Technologies With Different Functions

Based on the literature review, this study grouped technologies into eight categories based on their functions: (1) collaborative tools (e.g., Google Docs or Padlet) for supporting learners to collaborate on a task through co-editing and information sharing; (2) social tools (e.g., Facebook or Skype) for supporting learners to communicate and share content remotely or synchronously using text, audio and video; (3) creative tools (e.g., Photo Story or Adobe Spark) to support learners in creating work, such as digital stories or videos; (4) learning management system (e.g., Moodle) to integrate learning activities and learning resources for adaptive online learning; (5) classroom interaction tools (e.g., Quizlet or Kahoot) to support question-answering, polling, and other activities in the classroom; (6) multimedia materials are some audio and video resources on the web or multimedia textbooks; (7) presentation tools (e.g., PowerPoint) are used to support learners to present their learning content digitally; (8) wearable devices (e.g., Google Glass) to support learners to view or interact with content in virtual reality learning environments.

Most Commonly Used Technologies

Facebook (social tool), Google Docs (collaboration tool), and Moodle technologies (learning management system) were used the most in previous studies to facilitate language and 21st century skills. The study further analyzed which technologies are most often used by researchers to promote 21st century skills. Appendix 8 demonstrates these most commonly used tools. The study found that Facebook (social tool), Google Docs (collaboration tool) and Moodle (learning management system) were also the tools most often used by researchers to promote communication, collaboration and critical thinking, social and cross-cultural interaction skills. This indicates that scholars valued such 21st century skills as collaboration and communication among students in technology-supported language learning activities. For example, Sevy-Biloon and Chroman (2019 ) used social and collaborative tools (e.g., Google Docs, Facebook, etc.) to support communication between students from different cultural backgrounds and their results showed that students' speaking skills, social and cross-cultural interaction, and communication skills were promoted. Moodle is popular among researchers because this learning management system not only supports learners' adaptive and inquiry-based learning, but also helps teachers share learning resources with learners, design learning activities, and manage learners' learning progress ( García-Sánchez and Burbules, 2016 ). For example, Yang et al. (2014 ) designed a language learning activity based on the Moodle platform that asked students to complete reading and writing tasks in the system to promote the development of reading, writing skills and critical thinking. In addition, researchers most often used Google Docs (collaboration tool), Prezi (presentation tool), Windows Movie Maker, Photo Story3 (creative tools) and Blogs (social tool) to support students' creativity and innovation skills, problem-solving skills, and ICT literacy. And only two studies have used films (multimedia materials) and blogs (social tool) to support students' media literacy.

Experienced Challenges of Using Technology

Scholars reported that technologies pose some challenges for learners. For example, students were not experienced to use technology and had no trainings before learning activities; then they complained about problems to use technology during learning ( Lai, 2017 ). Students were also confused about the layout of the platform and noted that there were incompatibilities and connectivity issues with learning devices ( Hosseinpour et al., 2019 ). When communicating remotely, students pointed out that there were problems with the network and they were not able to connect and participate in learning process ( Mohamadi Zenouzagh, 2018 ; Jung et al., 2019 ).

The Distribution of Technology in non-English Language Studies and Different Theories

This review also analyzed technologies that were used by those few studies that focused on non-English languages. This review found that, in general, scholars in these studies used such technologies as creative tools (Adobe Spark), collaboration tools (Google Docs), and social tool (Facebook) to present multimedia content to learners and support collaborative, creative and communicative learning activities ( Valdebenito and Chen, 2019 ).

With regard to the distribution of technology in theory. Social constructivism theory was the most commonly cited theory in reviewed research and scholars used various technologies such as learning management systems (e.g., Moodle), creative tools (e.g., iMovie) or social tools (e.g., Facebook) to support constructivism-based learning activities. That is, interactive and collaborative learning activities were designed for students to learn new knowledge and then apply it to construct meaning in authentic contexts.

Based on the results of this study, several recommendations for educators and researchers were proposed. First, it is recommended that learning activities supported by technologies are designed based on appropriate theoretical foundation. Second, teachers are encouraged to conduct appropriate technology training for students beforehand so that they become familiar with technology tools. Third, teachers and researchers should test learning tools with students in advance in order to identify any possible technical problems, and provide timely support during learning process.

Learning Activities Used to Promote Language Skills and 21st Century Skills

This section describes what technologies are used in each type of learning activity and how they contribute to the development of learners' language skills and 21st century skills. In addition, it offers relevant suggestions to researchers and educators.

Adaptive Language Learning Activities on Learning Platforms

As shown in Table 1 , in the reviewed study, researchers used the following tools: Moodle, Google classroom, Quantum leap, and WebQuest, to develop adaptive language learning activities on learning platforms. These tools are used to integrate different types of instructional resources and diverse language learning activities to provide learners with adaptive learning materials that meet their learning needs. Students can ask questions and receive feedback from other students or teachers, and take control of their own learning progress.

Table 1 . Adaptive language learning activities on learning platforms.

For example, Arnó-Macià and Rueda-Ramos (2011 ) designed tasks for reading, listening, and speaking practice in Quantum leap platform. Researchers have designed listening tasks in Moodle platform; students were required to analyze, evaluate, and summarize content after listening ( Yang et al., 2013 , 2014 ). Srebnaja and Stavicka (2018 ) designed WebQuests-based speaking and writing tasks.

All of these studies noted that learners' performance in speaking, listening, reading, writing, and grammar improved after completing the computer-assisted adaptive language learning tasks. In addition, students' critical thinking skills were developed.

Collaborative Task-Based Language Learning Activities

As shown in Table 2 , the following tools were used by researchers for the development of collaborative-based language learning activities: (1) collaboration tools: Google Docs, Google Drive, Wiki, Edmodo, and E-writing forum. These collaborative tools have the following functions: sharing, collaborative editing, cloud storage, synchronized display, and help students freely share information in various formats (e.g., text, images, videos, web links, audio recordings, music, etc.) on the platform so that they can exchange ideas and collaborate on editing content; (2) creative tools: Adobe Spark, to support students' expression of ideas; (3) social tools: Blogs or WordPress, to support students in reading and commenting on each other's work.

Table 2 . Collaboration-based language learning activities.

Collaboration-based language learning activities are those in which students work in groups to solve problems and complete tasks proposed by the teacher, such as asking students to provide an essay or present their ideas in other ways (e.g., a solution, a report, and a performance). For example, Amir et al. (2011 ) asked students to work in groups to publish six articles based on different topics over the course of 14 weeks, and one of the tasks required students to find and discuss software about computer-assisted writing.

Mohamadi Zenouzagh (2018 ) designed a collaborative writing activity based on the E-writing platform. Valdebenito and Chen (2019 ) designed a collaborative activity on the theme of “food and culture” in which students first had to use Google Maps to identify geographic areas related to the content, then use a Google Doc to record their ideas, and finally use video production tools such as Adobe Spark to express their ideas and share them on the WordPress platform. Huh and Lee (2020 ) designed a creative learning English collaborative activity in which students first used a mobile app to learn how to spell words, then the group took the words they learned and expressed them through the role play and song. Lai (2017 ) designed different collaborative tasks, for example, students needed to use the ThingLink tool to create vocabulary lists and greeting cards related to the topic, which were then shared on the Padlet platform and discussed. In addition, students were required to use HomeStyler to collaboratively design a dream home and use some vocabulary related to “location” to describe the design of their home.

Girgin and Cabaroglu (2021 ) designed an English learning project that integrates Web 2.0 technology and flipped classroom, and students used Padlet to watch videos in class. In grammar classes, students used Kahoot, Quizlet, Quizizz, Animoto, Powtoon, and Poster MyWall to answer grammar questions. In vocabulary and reading classes, students used tools such as Mind Meister, Voki, Canva, Cram, Go Animate and Story-bird to create mind maps, as well as create digital stories, which can be presented and shared. Chen et al. (2021 ) used virtual reality technology to design language learning activities. Learners were required to first watch a virtual reality scene and think about how to solve the problem based on a series of guiding questions provided by the teacher. Then students role-played in English to create a virtual reality video of the problem being solved.

The results of the abovementioned studies showed that collaborative-based language learning activities facilitated the development of learners' language skills. The researchers noted that collaborative problem-solving language learning activities provided learners with a large number of writing tasks, such as writing reports, essays, or creating storylines and designing works. The process of sharing with each other enabled to point out grammatical errors ( Amir et al., 2011 ; Mohamadi Zenouzagh, 2018 ; Hosseinpour et al., 2019 ). When learners used multimedia resources to create vocabulary lists and greeting cards, their vocabulary and grammar skills were also improved ( Lai, 2017 ).

At the same time, students' critical thinking was developed as they gave each other's critical and constructive comments ( Valdebenito and Chen, 2019 ; Zou and Xie, 2019 ; Girgin and Cabaroglu, 2021 ). In addition, students completed tasks in small groups which promoted the development of communication and collaboration skills during discussions with each other ( Amir et al., 2011 ; García-Sánchez and Burbules, 2016 ; Lai, 2017 ; Mohamadi Zenouzagh, 2018 ; Hosseinpour et al., 2019 ; Zou and Xie, 2019 ; Girgin and Cabaroglu, 2021 ). The process of students voicing digital content promoted the development of speaking skills ( Huh and Lee, 2020 ). In the process of creating digital works, digital literacy was developed ( García-Sánchez and Burbules, 2016 ; Valdebenito and Chen, 2019 ). Chen et al. (2021 ) pointed out that learners learn contextually in an immersive learning environment, and solving real problems through virtual reality technology improved learners' vocabulary as well as promoted their problem-solving skills.

Creative Work-Based Language Learning Activities

As shown in Table 3 , in reviewed studies, language learning activities based on creative works consisted of two main categories: creating digital stories or videos. The main models for this type of learning activity were as follows: students communicated in groups about how to create a digital story or video, then collected and processed relevant information, after that created a digital story, and finally shared content and communicated with each other about it.

Table 3 . Creative work-based language learning activities.

The researchers chose different tools to support such learning process, e.g., (1) creating digital stories, i.e., Photo Story3, Windows Movie Maker, or iMovie; (2) creating video scripts in collaboration, i.e., Google Docs or Google Drive; (3) presenting digital stories, i.e., Prezi or PPT; (4) sharing digital stories and communicating, i.e., Google+ forums, Facebook, Instagram, WhatsApp, Google Classroom, and classroom management systems.

The researcher noted that digital storytelling promoted language skills, specifically, the process of writing story scripts promoted students' writing and vocabulary skills ( Thang et al., 2014 ; Sevilla-Pavón and Nicolaou, 2017 ; Kulsiri, 2018 ; Yalçin and Öztürk, 2019 ; Chiang, 2020 ). It also promoted 21st century skills. Researchers mentioned three approaches for creating digital stories or videos such as free-writing, rewriting the ending of the story, and specifying the theme, and in this open-ended work creation process, students' sense of creativity, problem-solving skills, and digital literacy were developed ( Thang et al., 2014 ; Sevilla-Pavón and Nicolaou, 2017 ; Kulsiri, 2018 ; Yalçin and Öztürk, 2019 ; Yang et al., 2022 ). Regarding the creation of digital stories on a specific theme, the researcher asked learners to design a new country, and students needed to understand a range of elements including different countries and cultures, such as national characteristics, language, national policies, climate and life. As a result, students' social and cross-cultural skills were improved. In addition, critical thinking was facilitated as students developed different ideas and perspectives as they evaluated each other's digital stories ( Sevilla-Pavón and Nicolaou, 2017 ). Finally, students developed their communication and collaboration skills when working in groups ( Thang et al., 2014 ; Sevilla-Pavón and Nicolaou, 2017 ; Kulsiri, 2018 ; Yalçin and Öztürk, 2019 ; Mirza, 2020 ; Huang, 2021 ).

Language Learning Activities Based on Multimedia Learning Materials

As shown in Table 4 , language learning activities based on multimedia materials involved such tools as (1) web-based learning management system, e.g., EDpuzzle; (2) social tool, e.g., YouTube; and (3) multimedia textbooks. All of them provided multimedia resources for students. There were also (4) collaboration tools, e.g., Padlet and Google docs, which supported learners to share ideas with each other.

Table 4 . Language learning activities based on learning multimedia materials.

Scholars have designed a variety of language learning activities based on multimedia materials, but the topics and learning tasks of the multimedia materials involved in these studies differed. For example, Tseng (2017 ) asked learners to watch a video on the topic of cultural differences, and then students gave oral presentations and reflections to present their views on cultural differences. Zou and Xie (2019 ) asked students to watch a video on EDpuzzle, then to discuss in groups, negotiate and compare answers, to share their output to the Padlet platform, and finally submit their reports in Google docs. Nikitova et al. (2020 ) asked students to watch videos from multimedia textbooks with different English contexts and then simulated learners' role play activities. Aristizábal-Jiménez (2020 ) asked learners to watch YouTube videos, analyze the structure and content of video content, and then create posters to present and share their ideas.

The researcher noted that language learning activities based on multimedia materials promoted learners' language skills and 21st century skills. Specifically, learners' listening skills were promoted after watching the videos ( Tseng, 2017 ). Culturally relevant content in videos and culture-based communication among peers promoted students' social and cross-cultural interaction skills ( Tseng, 2017 ). Learners actively used dictionaries and discussed grammar while completing tasks to make the information easier to understand, which also promoted students' vocabulary and grammar skills ( Aristizábal-Jiménez, 2020 ). In addition, working in groups to complete tasks promoted speaking, writing, grammar, and vocabulary skills. This was also beneficial to develop students' problem solving, collaboration, critical thinking, and communication skills ( Aristizábal-Jiménez, 2020 ; Nikitova et al., 2020 ).

Language Learning Activities Based on Online Communication

As shown in Table 5 , the researchers designed online communication-based language learning activities. Most of them were cross-cultural communication activities to support cross-cultural communication between students from different cultural backgrounds. In terms of technology, the researchers mainly used social tools to support textual or video communication, e.g., Facebook, Skype, and WhatsApp. In addition, researchers have utilized learning management systems to support students to view learning resources uploaded by teachers.

Table 5 . Language learning activities based on online communication.

The design of cross-cultural communication activities followed the same pattern—exposure to cross-cultural knowledge, reflection on cross-cultural differences, and cross-cultural exchange. For example, Calogerakou and Vlachos (2011 ) had students from two countries to watch movies and compare culture presented in movies with their own culture. Then students had to post comments on a blog and discuss their ideas. Chen and Yang (2016 ) asked students to share culturally specific folklore stories with their partners and to make videos of the stories to send to their partners. In addition, students were asked to perform a puppet show via videoconference. All of these were for students to learn about cultural similarities and differences. Chen and Yang (2014 ) designed a discussion activity based on cultural themes; for example, students discussed movies that involved culturally different content, and then students shared their opinions on Wiki. Lewis and Schneider (2015 ) asked learners to interact with native Spanish-speaking students and discuss cultural topics such as “local living conditions” and “how to celebrate holidays.” Learners were then asked to write a mini-biography or travel brochure for their study partner to demonstrate the cultural knowledge they gained during the exchange. Özdemir (2017 ) asked students to watch YouTube videos and discuss them based on cross-cultural questions prepared by the teacher. Sevy-Biloon and Chroman (2019 ) designed an intercultural exchange program in which students from Ecuador and the United States were randomly paired and then engaged in a cultural exchange based on the theme of the language course. Jung et al. (2019 ) asked students from different cultural backgrounds to discuss cultural topics, including “happiness factors, family, and food,” and finally, students reflected on the discussion, exchanged proverbs with each other, and then presented cultural differences. They reflected on their experiences in a reflective journal. Jamalai and Krish (2021 ) designed an online discussion activity, in which learners were required to engage in online discussions based on topics posted by teachers in a forum.

The results showed that students' speaking, vocabulary, writing, reading, and grammar skills improved when communicating through text and speech because students double-checked vocabulary spelling and grammar. Students identified errors they made when communicating using text and speech and corrected them to ensure that others understood their intended meaning ( Calogerakou and Vlachos, 2011 ; Chen and Yang, 2014 , 2016 ; Lewis and Schneider, 2015 ; Özdemir, 2017 ; Hirotani and Fujii, 2019 ; Jung et al., 2019 ; Sevy-Biloon and Chroman, 2019 ; Jamalai and Krish, 2021 ). In addition, students' listening skills improved after watching YouTube videos ( Özdemir, 2017 ).

At the same time, students' communication process using social tools developed the ability to use writing software, electronic dictionaries, and collect information on the Internet, and therefore media literacy was improved ( Calogerakou and Vlachos, 2011 ). All studies point to the development of cultural interaction skills after students interacted and exchanged different cultural perspectives with partners ( Calogerakou and Vlachos, 2011 ; Chen and Yang, 2014 , 2016 ; Lewis and Schneider, 2015 ; Özdemir, 2017 ; Hirotani and Fujii, 2019 ; Jung et al., 2019 ; Sevy-Biloon and Chroman, 2019 ). Communication ( Chen and Yang, 2014 ; Lewis and Schneider, 2015 ; Hirotani and Fujii, 2019 ) and collaboration skills were also developed ( Chen and Yang, 2014 ) in reviewed studies.

This review also analyzed learning activities that were used by those few studies that focused on non-English languages. This review found that most learning activities designed in these studies were online cross-cultural communicative activities. This shows that the primary goal of these learning projects was to develop students' foreign language and intercultural communication skills.

Based on the findings of the reviewed literature, the five types of language learning activities supported by technology had a positive impact on students' language skills as well as their 21st century skills development. Moreover, this review found that these learning activities followed similar pattern. The common pattern for language learning activities based on culture-related communication was exposure to cross-cultural knowledge, reflection on cross-cultural differences, and cross-cultural exchange. The common pattern of language learning activities for creative works was as follows: students communicated in groups about how to create a work (such as digital story or video), then collected and processed relevant information, created a work, and then shared content and communicated with each other about it. These patterns could provide suggestions for researchers and teachers to design similar instructional activities that target development of language skills and 21st century skills in the future.

Second, this review found that researchers designed similar instructional activities, but the research focus was different. For example, in the adaptive language learning activities on learning platforms, researchers focused on the development of students' speaking skills and lacked attention to reading skills. And in the collaborative task-based language learning activities, researchers have focused more on writing and vocabulary skills, collaboration, and communication skills, and lacked attention to listening skills. In creative writing-based language learning activities, researchers focused more on speaking and writing skills as well as creative and communication skills.

Research Duration, Participants' Academic Level, and Sample Size

The most common study samples were small ones with participants range from 11 to 30 ( n = 11) and medium samples with range between 61 and 90 ( n = 8) participants. Research durations were mostly between 3 and 6 months ( n = 10). Small sample size was acknowledged as a limitation in some studies ( Hirotani and Fujii, 2019 ; Zou and Xie, 2019 ). The possible reason for this is that most of the studies were based on small classroom settings. In the reviewed studies, the most common academic level of participants was undergraduate level. There were 12 studies that did not specify research duration. Regarding this finding, there is a lack of attention in previous retrospective studies ( Guan, 2014 ; Duman et al., 2015 ; Persson and Nouri, 2018 ).

Data Collection

Most of the studies collected both quantitative and qualitative data, which can help researchers to draw conclusions from different perspectives. Quantitative data included tests, scales, and rubrics; qualitative data included student's work, open-ended questions, student feedback, interviews, student chat transcripts, student reflections, teacher journals, and observations. One of the most common forms of quantitative data collection is a test ( n = 15), involving student language tests (tests of English speaking and listening) and tests of 21st century skills (critical thinking and creative thinking). The most common method of qualitative data collection was interview ( n = 13), where the researcher usually designed an interview outline and then asked learners questions to understand their learning experiences, attitudes, motivations, and challenges in the learning process. In addition, researchers have extensively used questionnaires ( n = 17), including both closed-ended and open-ended questions, to collect both quantitative and qualitative data. For example, the researchers used questionnaires to investigate learners' perceptions of technology-supported language learning, including effectiveness, usefulness, and students' perceptions of developing intercultural communicative competence and language skills through online discussions ( Jung et al., 2019 ).

Based on the above findings, the recommendations of the present study for researchers and teachers are as follow. First, researchers could consider studies with longer time spans and collect data from bigger number of participants to investigate students' development over time and have generalizable conclusions. Second, researchers can collect multiple types of data, focus on students' learning processes and outcomes, and then interpret findings from different perspectives.

Research Design

There are a variety of research designs for reviewed studies on technology-supported language learning and 21st century skills. The most common are quasi-experimental studies. Such studies are characterized by using pre- and post-tests to measure changes in participants' language skills, 21st century skills and other learning outcomes and attitudes before and after participation in learning activities. In quasi-experimental studies, participants are not randomly assigned to an experimental or control group ( Persson and Nouri, 2018 ; Huang, 2021 ). These findings are consistent with other reviews on technology-supported language learning ( Persson and Nouri, 2018 ). The present study suggests that educators and researchers can use the three research methods mentioned above to validate their studies in future.

Positive Learning Experiences

In this section, the study discusses findings from reviewed studies and recommendations for educators and researchers. In reviewed studies, in addition to finding that technology-supported learning activities promoted learners' language skills and 21st century skills, researchers also found that these technologies led to positive learning experiences, which resulted in better learning outcomes. For example, learning through multimedia textbooks, collaborative blog-based writing activities, smartphone-based video filming activities and language learning projects based on intercultural exchange all increased students' motivation ( Amir et al., 2011 ; García-Sánchez and Burbules, 2016 ; Sevy-Biloon and Chroman, 2019 ; Aristizábal-Jiménez, 2020 ; Huang, 2021 ). For example, Hosseinpour et al. (2019 ) noted that through collaborative writing activities, learners' motivation and self-confidence levels were increased. Mirza (2020 ) argued that through digital storytelling-based learning activities, students gained more confidence. Researchers have also looked at the different learning performance of students due to individual differences in abilities or their characteristics. Yang et al. (2014 ) found that in terms of writing, significant differences were found between “basic” and “low-intermediate” learners as a result of the difference in ability. Yalçin and Öztürk (2019 ) found that girls had a more negative attitude toward technology than boys.

Challenges Faced by Students

While many studies pointed to positive student attitudes toward technology-supported learning activities ( Arnó-Macià and Rueda-Ramos, 2011 ; Girgin and Cabaroglu, 2021 ), several studies highlighted challenges that students faced when using technology for learning. Challenges from technology, with some learners finding it difficult to use in learning activities or being confused about the layout of mobile applications were mentioned. Students also noted problems with device incompatibility and poor network quality and speed when using technology. Self-competence challenges, with learners noting that learning tasks were difficult for them, for example, insufficient time to complete learning tasks, lack of research skills, or language skills needed to complete tasks, were reported. Difficulties in finding an interesting topic and choosing the right tools to create their work were also reported in reviewed studies. Challenges of collaborating with others, with some learners noting that they encounter uncoordinated teamwork, uneven distribution of work and unequal student contributions in collaborative tasks, were mentioned by scholars. Self-attitudes, as noted by learners who felt anxious about video chatting when they were communicating remotely, as well as fear of having their writing errors discovered by their partners when communicating in text, were reported in reviewed studies.

Based on the above findings, the present study recommends to educators and researchers, in addition to focusing on the impact of technology-supported learning activities on learners' language skills and 21st century skills, it is also important to focus on students' perceptions of technology, motivation, engagement, and confidence. This is because positive learning experiences can lead to better learning outcomes ( Sevy-Biloon and Chroman, 2019 ; An et al., 2021 ). Regarding the technological challenges that students encounter in the learning process, it is recommended that they be addressed through advance trainings and through providing students with appropriate technological services during learning activities. Self-competence challenges can be addressed by designing collaborative tasks in which students with higher levels of competence can help students with lower levels of competence to complete the task. Regarding the challenges in collaborative activities, it is recommended that teachers and researchers design learning activities with clear rules for collaborative division of labor and rules regarding how learning performance of every learner will be evaluated. With regard to alleviating negative student attitudes, it is recommended that teachers design diverse teaching strategies and scaffolds to give students assistance during learning activities.

This study reviewed articles on technology-supported language learning and 21st century skills published from 2011 to 2022 (February) in terms of (a) research focus; (b) theoretical foundations; (c) technology; (d) learning activities; (e) methodology and (f) findings. The results indicate that research on technology-supported language learning and 21st century skills have shown an upward trend in the overall research in the covered time period, with most of the research focusing on English and the majority of participants in these studies majored in education.

Secondly, in terms of research focus, most of the researchers focused on learners' speaking skills (27.40%), followed by writing (26.03%) and vocabulary skills (17.81%). In terms of 21st century skills, most researchers focused on communication (20.83%), collaboration (20.83%), critical thinking (13.89%), and social and cross-cultural interaction skills (13.89%). In terms of theoretical foundations, social constructivist learning theory was most often adopted by researchers. In terms of technology, tools that support learners' creativity and socialization are often utilized by researchers, e.g., Facebook or Google Docs. In terms of learning activities, researchers have designed the following five types of learning activities to support learners' language learning and 21st century skills: (1) collaborative task-based language learning activities; (2) language learning activities based on online communication; (3) creative work-based language learning activities (4) adaptive language learning activities based on learning platforms; and (5) language learning activities based on multimedia learning materials. The results of reviewed studies indicate that these learning activities supported by technology are effective in promoting the development of learners' different language skills and 21st century skills. Finally, in terms of methodology, most of the studies had a sample of 11–30, the most common study period was 3–6 months, the data collection method often used by researchers was questionnaires, the most common method to collect quantitative data was tests, and the most common method to collect qualitative data was interviews.

In contrast to traditional paper and pencil-based learning, technologies used by researchers in reviewed studies allowed learners to improve language learning outcomes and 21st century skills through individual and collaborative learning activities. Some reported advantages are learning with technologies without the constraints of time and space, technologies enable personalized learning, technologies create authentic learning environments that provides adaptive learning content, helps create multimedia content actively, allows social interaction such as sharing, giving or receiving feedback, and reflecting on learning more efficiently.

Based on the above findings, recommendations for researchers and educators in this study include: (1) In terms of language skills, in addition to focusing on output skills, input skills (reading, listening) also deserve attention from researchers. In terms of 21st century skills, learners' problem-solving skills and career and life skills also need more attention from researchers in the future; (2) Advanced technology training for learners to familiarize them with technology and its effective usage as well as teachers need to check in advance for possible technology problems, such as network problems. These suggestions can help teachers address the technological barriers that learners encounter in the learning process; (3) The use of various theoretical approaches, such as instructional design-related theories and language learning-related theories, is important for the rational design of instructional activities that promote learners' language and 21st century skills; (4) Researchers and educators can follow the general model of conducting the five types of instructional activities summarized above to design instructional activities. In addition, it is recommended that researchers and educators use variety of technologies and design different instructional activities to promote learners' language and 21st century skills. It is also important to be aware of the challenges that students may encounter in terms of technology, learning activity tasks, peer collaboration and self-attitudes when implementing learning activities; (5) Teachers and educators could involve more participants and consider longer time spans in future studies to focus on more learners' development and to collect diverse quantitative and qualitative data to explain students' learning processes and outcomes.

There are few limitations to this study. Articles reviewed in this study were sourced from PRIMO and Web of Science databases, and some conference papers, books and dissertations were excluded. For this reason, this study reviewed smaller number of articles. Future studies may consider this limitation and address it by including more relevant sources.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Author Contributions

RS and XW contributed to the conception, designed the work, collected the data, analyzed, and interpreted data. XW drafted the work and RS substantively revised it. RS was responsible for correspondence. All authors approved the submitted version and agreed both to be personally accountable for the author's own contributions and to the accuracy the work.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2022.897689/full#supplementary-material

* Amir, Z., Ismail, K., and Hussin, S. (2011). Blogs in language learning: maximizing students' collaborative writing. Procedia Soc. Behav. Sci. 18, 537–543. doi: 10.1016/j.sbspro.2011.05.079

CrossRef Full Text | Google Scholar

An, Z., Wang, C., Li, S., Gan, Z., and Li, H. (2021). Technology-assisted self-regulated English language learning: associations with English language self-efficacy, English enjoyment, and learning outcomes. Front. Psychol. 11:558466. doi: 10.3389/fpsyg.2020.558466

PubMed Abstract | CrossRef Full Text | Google Scholar

* Aristizábal-Jiménez, Y. (2020). Fostering talk as performance in an EFL class through the critical analysis of youtubers' content. Profile Issues Teach. Professional Dev. 22, 181–195. doi: 10.15446/profile.v22n2.82510

* Arnó-Macià, E., and Rueda-Ramos, C. (2011). Promoting reflection on science, technology, and society among engineering students through an EAP online learning environment. J. English Acad. Purposes 10, 19–31. doi: 10.1016/j.jeap.2010.12.004

Asia Pacific Economic Cooperation (2004). 2004 APEC Ministerial Meeting. Available online at: https://www.apec.org/Meeting-Papers/Annual-Ministerial-Meetings/2004/2004_amm (accessed June 11, 2022).

Google Scholar

Avgousti, M. I. (2018). Intercultural communicative competence and online exchanges: a systematic review. Computer Assisted Lang. Learn. 31, 819–853. doi: 10.1080/09588221.2018.1455713

Bennett, M. (1986). A developmental approach to training for intercultural sensitivity. Int. J. Intercultural Relat. 10, 179–196. doi: 10.1016/0147-1767(86)90005-2

Byram, M. (1997). Teaching and Assessing Intercultural Communicative Competence . Clevedon: Multilingual Matters.

* Calogerakou, C., and Vlachos, K. (2011). Films and Blogs: an authentic approach to improve the writing skill-an intercultural project-based framework in the Senior High State School. Res. Papers Lang. Teach. Learn. 2, 98–110.

* Chen, C. H., Hung, H. T., and Yeh, H. C. (2021). Virtual reality in problem-based learning contexts: effects on the problem-solving performance, vocabulary acquisition and motivation of English language learners. J. Computer Assisted Learn. 37, 851–860. doi: 10.1111/jcal.12528

* Chen, J. J., and Yang, S. C. (2014). Fostering foreign language learning through technology-enhanced intercultural projects. Lang. Learn. Technol. 18, 57–75.

* Chen, J. J., and Yang, S. C. (2016). Promoting cross-cultural understanding and language use in research-oriented Internet-mediated intercultural exchange. Computer Assisted Lang. Learn. 29, 262–288. doi: 10.1080/09588221.2014.937441

* Chiang, M. H. (2020). Exploring the effects of digital storytelling: a case study of adult L2 writers. IAFOR J. Educ. 8, 65–82. doi: 10.22492/ije.8.1.04

Creswell, J. W. (2002). Educational Research: Planning, Conducting, and Evaluating Quantitative. Upper Saddle River, NJ: Prentice Hall.

Duman, G., Orhon, G., and Gedik, N. (2015). Research trends in mobile assisted language learning from 2000 to 2012. ReCALL 27, 197–216. doi: 10.1017/S0958344014000287

* García-Sánchez, M. S., and Burbules, N. C. (2016). Learning technologies and EFL teamwork. Revista de Lenguas para Fines Específicos 22, 100–115. doi: 10.20420/rlfe.2016.0092

* Girgin, P., and Cabaroglu, N. (2021). Web 2.0 supported flipped learning model: EFL students' perceptions and motivation. Cukurova Univ. Faculty Educ. J. 50, 858–876. doi: 10.14812/cuefd.944217

Goksu, I., Ozkaya, E., and Gunduz, A. (2020). The content analysis and bibliometric mapping of CALL journal. Computer Assisted Lang. Learn. 1–31. doi: 10.1080/09588221.2020.1857409

Guan, S. (2014). Internet-based technology use in second language learning: a systematic review. Int. J. Cyber Behav. Psychol. Learn. 4, 69–81. doi: 10.4018/ijcbpl.2014100106

Harmer, J. (2007). The Practice of English Language Teaching . London: Longman.

* Hirotani, M., and Fujii, K. (2019). Learning proverbs through telecollaboration with Japanese native speakers: facilitating L2 learners' intercultural communicative competence. Asian Pacific J. Second Foreign Lang. Educ. 4, 1–22. doi: 10.1186/s40862-019-0067-5

* Hosseinpour, N., Biria, R., and Rezvani, E. (2019). Promoting academic writing proficiency of Iranian EFL learners through blended learning. Turkish Online J. Distance Educ. 20, 99–116. doi: 10.17718/tojde.640525

* Huang, H. W. (2021). Effects of smartphone-based collaborative vlog projects on EFL learners' speaking performance and learning engagement. Austral. J. Educ. Technol. 37, 18–40. doi: 10.14742/ajet.6623

* Huh, K., and Lee, J. (2020). Fostering creativity and language skills of foreign language learners through SMART learning environments: evidence from fifth-grade Korean EFL learners. TESOL J. 11, e489. doi: 10.1002/tesj.489

* Jamalai, M., and Krish, P. (2021). Fostering 21st century skills using an online discussion forum in an English for specific purpose course. Malaysian J. Learn. Instruct. 18, 219–240. doi: 10.32890/mjli2021.18.1.9

* Jung, Y., Kim, Y., Lee, H., Cathey, R., Carver, J., and Skalicky, S. (2019). Learner perception of multimodal synchronous computer-mediated communication in foreign language classrooms. Lang. Teach. Res. 23, 287–309. doi: 10.1177/1362168817731910

Keller, J. M. (1987). Development and use of the ARCS model of instructional design. J. Instruct. Dev. 10, 2–10. doi: 10.1007/BF02905780

Krashen, S. D. (1985). The Input Hypothesis: Issues and Implications . London: Longman.

Kukulska-Hulme, A., and Viberg, O. (2018). Mobile collaborative language learning: state of the art. Br. J. Educ. Technol. 49, 207–218. doi: 10.1111/bjet.12580

* Kulsiri, S. (2018). Students' perceptions of a student-produced video project in the General English language course at Srinakharinwirot University, Thailand. Arab World Eng. J. 4, 40–54. doi: 10.24093/awej/call4.4

* Lai, A. (2017). Implementing online platforms to promote collaborative learning in Chinese language classrooms. J. Technol. Chin. Lang. Teach. 8, 39–52.

Lantolf, J. (2000). Sociocultural Theory and Language Learning . Oxford: OUP.

* Lewis, T. N., and Schneider, H. (2015). Integrating international video chat into the foreign language curriculum. Int. J. Comput. Assisted Lang. Learn. Teach. 5, 72–84. doi: 10.4018/IJCALLT.2015040105

Lin, L., Shadiev, R., Hwang, W. Y., and Shen, S. S. (2020). From knowledge and skills to digital works: an application of design thinking in the information technology course. Think. Skill Creat. 36, 100646. doi: 10.1016/j.tsc.2020.100646

* Mirza, H. S. (2020). Improving university students' english proficiency with digital storytelling. Int. Online J. Educ. Teach. 7, 84–94.

* Mohamadi Zenouzagh, Z. (2018). Multidimensional analysis of efficacy of multimedia learning in development and sustained development of textuality in EFL writing performances. Educ. Information Technol. 23, 2969–2989. doi: 10.1007/s10639-018-9754-y

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., and PRISMA Group* (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann. Internal Med. 151, 264–269. doi: 10.7326/0003-4819-151-4-200908180-00135

* Nikitova, I., Kutova, S., Shvets, T., Pasichnyk, O., and Matsko, V. (2020). “Flipped learning” methodology in professional training of future language teachers. Euro. J. Educ. Res. 9, 19–31. doi: 10.12973/eu-jer.9.1.19

* Özdemir, E. (2017). Promoting EFL learners' intercultural communication effectiveness: a focus on Facebook. Comput. Assisted Lang. Learn. 30, 510–528. doi: 10.1080/09588221.2017.1325907

Parmaxi, A., and Zaphiris, P. (2017). Web 2.0 in Computer-Assisted Language Learning: a research synthesis and implications for instructional design and educational practice. Interact. Learn. Environ. 25, 704–716. doi: 10.1080/10494820.2016.1172243

Partnership for 21st Century Skills (2008). P21 Framework Definitions Document . Available online at: http://www.21stcenturyskills.org (accessed February 28, 2022).

Persson, V., and Nouri, J. (2018). A systematic review of second language learning with mobile technologies. Int. J. Emerg. Technol. Learn. 13, 188–210. doi: 10.3991/ijet.v13i02.8094

* Sevilla-Pavón, A., and Nicolaou, A. (2017). Online intercultural exchanges through digital storytelling. Int. J. Comput. Assisted Lang. Learn. Teach. 7, 44–58. doi: 10.4018/IJCALLT.2017100104

* Sevy-Biloon, J., and Chroman, T. (2019). Authentic use of technology to improve EFL communication and motivation through international language exchange video chat. Teach. English Technol. 19, 44–58.

Shadiev, R., Hwang, W.-Y., and Ghinea, G. (2022a). Guest editorial: Creative learning in authentic contexts with advanced educational technologies. Educ. Technol. Soc. 25, 76–79. Available online at: https://www.jstor.org/stable/48660125

Shadiev, R., Hwang, W. Y., and Huang, Y. M. (2017). Review of research on mobile language learning in authentic environments. Comput. Assist. Lang. Learn. 30, 284–303. doi: 10.1080/09588221.2017.1308383

Shadiev, R., Wang, X., Liu, T.Y., and Yang, M. (In Press). Improving students' creativity in familiar versus unfamiliar mobile-assisted language learning environments. Interact. Learn. Environ. doi: 10.1080/10494820.2021.2023891

Shadiev, R., and Yang, M. (2020). Review of studies on technology-enhanced language learning and teaching. Sustainability. 12, 524. doi: 10.3390/su12020524

Shadiev, R., Yi, S., and Dang C. Sintawati, W. (2022b). Facilitating students' creativity, innovation and entrepreneurship in a telecollaborative project. Front. Psychol. 13, 887620. doi: 10.3389/fpsyg.2022.887620

Shadiev, R., and Yu, J. T. (In Press). Review of research on computer-assisted language learning with a focus on intercultural education. Comput. Assist. Lang. Learn. doi: 10.1080/09588221.2022.2056616

* Srebnaja, J., and Stavicka, A. (2018). Web-based projects to develop transversal skills in secondary school. Hum. Technol. Qual. Educ. 2018, 25–34. doi: 10.22364/htqe.2018.03

Suzanne, N. (2014). “Critical and effective reading to build the characters as active readers,” in International Conference on Languages and Arts (Padang), 47–352.

* Thang, S. M., Sim, L. Y., Mahmud, N., Lin, L. K., Zabidi, N. A., and Ismail, K. (2014). Enhancing 21st century learning skills via digital storytelling: voices of Malaysian teachers and undergraduates. Procedia Soc. Behav. Sci. 118, 489–494. doi: 10.1016/j.sbspro.2014.02.067

* Tseng, C. T. H. (2017). Teaching “Cross-Cultural Communication” through content based instruction: curriculum design and learning outcome from EFL learners' perspectives. English Lang. Teach. 10, 22–34. doi: 10.5539/elt.v10n4p22

* Valdebenito, M., and Chen, Y. (2019). Technology as enabler of learner autonomy and authentic learning in chinese language acquisition: a case study in higher education. J. Technol. Chin. Lang. Teach. 10, 61.

Vygotsky, L. S. (1978). Mind in Society: The Development of Higher Psychological Processes . Cambridge, MA: Harvard University Press.

* Yalçin, O. B., and Öztürk, E. (2019). “The effects of digital storytelling on the creative writing skills of literature students based on their gender,” in ICGR 2019 2nd International Conference on Gender Research . (Rome: Academic Conferences and Publishing Limited), 59.

* Yang, Y. T. C., Chen, Y. C., and Hung, H. T. (2022). Digital storytelling as an interdisciplinary project to improve students' English speaking and creative thinking. Comput. Assist. Lang. Learn. 35, 840–862. doi: 10.1080/09588221.2020.1750431

* Yang, Y. T. C., Chuang, Y. C., Li, L. Y., and Tseng, S. S. (2013). A blended learning environment for individualized English listening and speaking integrating critical thinking. Comput. Educ. 63, 285–305. doi: 10.1016/j.compedu.2012.12.012

* Yang, Y. T. C., Gamble, J. H., Hung, Y. W., and Lin, T. Y. (2014). An online adaptive learning environment for critical-thinking-infused English literacy instruction. Br. J. Educ. Technol. 45, 723–747. doi: 10.1111/bjet.12080

Zhang, R., and Zou, D. (2020). Types, purposes, and effectiveness of state-of-the-art technologies for second and foreign language learning. Comput. Assist. Lang. Learn. 35, 696–742. doi: 10.1080/09588221.2020.1744666

* Zou, D., and Xie, H. (2019). Flipping an English writing class with technology-enhanced just-in-time teaching and peer instruction. Interact. Learn. Environ. 27, 1127–1142. doi: 10.1080/10494820.2018.1495654

* ^ Articles reviewed in this study.

Keywords: language learning, 21st century skills, technology, review, development

Citation: Shadiev R and Wang X (2022) A Review of Research on Technology-Supported Language Learning and 21st Century Skills. Front. Psychol. 13:897689. doi: 10.3389/fpsyg.2022.897689

Received: 17 March 2022; Accepted: 13 June 2022; Published: 07 July 2022.

Reviewed by:

Copyright © 2022 Shadiev and Wang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Rustam Shadiev, rustamsh@gmail.com

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Don't Miss Out!

Subscribe to the E-Bulletin for regular updates on research, free resources, solutions, and job postings from WestEd.

Subscribe Now

• Resources Home
• New Releases
• Top Downloads
• Webinar Archives
• Best Sellers
• Login / Create Account

Technology-Based Assessments for 21st Century Skills: Theoretical and Practical Implications from Modern Research

Edited by Michael C. Mayrath, Jody Clarke-Midura, Daniel H. Robinson, Gregory Schraw

Description

Solving problems creatively and collaborating effectively have always been two key workforce skills. A third skill just as vital in the 21st century job market? Using technology with ease.

Teaching students all three skills is essential for both economic survival and success, and assessment is a fundamental component. Yet, measuring these skills can be tricky due to the interacting factors associated with higher order thinking and multifaceted communication.

Advances in assessment theory, education psychology, and technology create an opportunity to create new methods of measuring students’ 21st century skills with validity, reliability, and scalability.

In this book, leading scholars from a variety of disciplines present the latest research on how to best measure complex knowledge, skills, and abilities using technology-based assessments.

Contributors discuss the theoretical and practical implications from their research and outline their visions for the future of technology-based assessments.

WestEd’s Edys Quellmalz, Barbara Buckley, Jodi Davenport , Mark Loveland , and Matt Silberglitt contribute a key chapter on 21st century dynamic assessment.

Resource Details

Product information, related resources, how can simulations be components of balanced state science assessment systems.

Simulations are signaling a new era in science assessment. This brief explains why and recommends two models that policymakers may consider incorporating into state science assessment systems.

Stay Connected

Subscribe to the E-Bulletin and receive regular updates on research, free resources, solutions, and job postings from WestEd.

Your download will be available after you subscribe, or choose no thanks .

Ask a question, request information, make a suggestion, or sign up for our newsletter.

• WestEd Bulletin
• Insights & Impact
• Equity in Focus
• Areas of Work
• Charters & School Choice
• Comprehensive Assessment Solutions
• Culturally Responsive & Equitable Systems
• Early Childhood Development, Learning, and Well-Being
• Economic Mobility, Postsecondary, and Workforce Systems
• English Learner & Migrant Education Services
• Justice & Prevention
• Learning & Technology
• Mathematics Education
• Resilient and Healthy Schools and Communities
• School and District Transformation
• Special Education Policy and Practice
• Strategic Resource Allocation and Systems Planning
• Supporting and Sustaining Teachers
• Professional Development
• Research & Evaluation
• How We Can Help
• Reports & Publications
• Technical Assistance
• Technical Assistance Services
• Policy Analysis and Other Support
• R&D Alert
• Board of Directors
• Equity at WestEd
• WestEd Pressroom
• WestEd Offices
• Work with WestEd

Work at WestEd