An Analysis of University Students' Needs for Learning Support Functions of Learning Management System Augmented with Artificial Intelligence Technology

Abstract

The aim of this study is to identify intelligent learning support functions in Learning Management System (LMS) to support university student learning activities during the transition from face-to-face classes to online learning. To accomplish this, we investigated the perceptions of students on the levels of importance and urgency toward learning support functions of LMS powered with Artificial Intelligent (AI) technology and analyzed the differences in perception according to student characteristics. As a result of this study, the function that students considered to be the most important and felt an urgent need to adopt was to give automated grading and feedback for their writing assignments. The functions with the next highest score in importance and urgency were related to receiving customized feedback and help on task performance processed as well as results in the learning progress. In addition, students view a function to receive customized feedback according to their own learning plan and progress and to receive suggestions for improvement by diagnosing their strengths and weaknesses to be both vitally important and urgently needed. On the other hand, the learning support function of LMS, which was ranked as low importance and urgency, was a function that analyzed the interaction between professors and students and between fellow students. It is expected that the results of this student needs analysis will be helpful in deriving the contents of learning support functions that should be developed as well as providing basic information for prioritizing when applying AI technology to implement learner-centered LMS in the future.

1. Introduction

With the advent of the 4th industrial revolution era, interest in Learning Management Systems (LMS) is rapidly increasing, as awareness of the importance of online teaching and learning to improve the quality of university education has been spreading due to the impact of the COVID-19 pandemic since 2019. LMS are systems for efficiently supporting and managing teaching and learning utilizing web-based instructional materials and announcements, evaluation, bulletin boards, discussions, and video lectures. In other words, it means a sort of web-based platform that integratedly manages various aspects of teaching and learning for teachers and students [1] [2].

Early research on LMS focused mostly on designing and implementing functions that could be provided for classes from the point of view of the system developer rather than the user, such as the learners and instructors. For example, these are studies on the functions that the system can provide, such as posting notices on bulletin boards, providing learning materials or video lectures, checking attendance, and providing grades [3]. These studies focus on the functions from the perspective of the instructor using the LMS, that is, functions that focus on the delivery and operation of learning content [4]. Due to the shift from an instructor-centered to a learner-centered educational paradigm, there is growing demand for learner-centered LMS functions to enhance learning such as self-directed learning modules for learners [5].

The transformation from the previous instructor-centered, one-way, content delivery LMS to learner-centered LMS is taking place in academia and industry. The functions available to learners in representative LMSs widely used in education, such as Blackboard Learn, Moodle, Khan Academy, Edmodo, and iTunes U, are offering customized content provision, management of learning history and interaction, author content, interaction management, and support for mobile learning environments, updates, and cloud computing services [6]. In addition to fundamental and basic functions of LMS that support learning activities such as assignments, assessments, surveys, and discussions, other functions like team-projects, video lectures for synchronous online classes, report plagiarism check, LMS functions of content authoring tools, chat rooms, wikis, and peer assessment/review are gradually being made and released [7]. The learning activity support functions implemented in learner-centered LMS will be gradually developed to provide an open, integrated, and flexible learning environment that supports the overall process of learning, such as allowing various learning activities, content, accommodation of various learning methods, learning evaluation, as well as learning design, and it will lead to personalized, level-differentiated and customized learning in consideration of individual learner’s situations [6] [8]. In designing and developing LMS, a ‘learning effect’ aspect beyond the ‘learning management’ aspects should be considered. Furthermore, how to provide actual ‘learning activities’ should be the ultimate aim.

Learners have different preferences in an LMS environment based on different personal traits and learning styles [9]. It is important to offer functions to support various learning activities and types within LMS. In an LMS, online courses rely on the analysis of students’ actions as well as the utilizing content and procedures. AI has various applications that make student activities more flexible and help students achieve better learning outcomes by evaluating, storing, and analyzing data [10]. The introduction of AI technology allows intelligent as well as new services to be incorporated into the existing LMS [11]. Analyzing log data related to students' learning activities, predicting their cognitive, social, and emotional engagement and presenting differentiated interventions and prescriptions according to their characteristics and style will further develop the learning support functions of LMS. In order to develop an effective LMS using AI techniques, instructional interventions through appropriate learning log data collection and analysis, that is, learning analytics, is essential [12] [13]. Learners should be grouped according to their different levels and types of participation, or by the achievement they have attained and provided with different LMS functions to better support their performance and maintain successful learning. An adaptive LMS environment continually updates student profiles and delivers these profile changes to teachers [14]. It is necessary to design and develop functions that can support self-directed or self-regulated learning customized to the students by upgrading the current LMS to an adaptive and intelligent LMS by applying such AI technology.

Therefore, the purpose of this study is to understand the needs of users for the learner-centered intelligent LMS learning (activity) support functions by assessing importance-urgency levels and by using text-mining methods. Based on the results of this study, intelligent LMS design guidelines that meet user needs can be inferred.

2. Research Method

2.1 Participants

The participants were 192 undergraduates (57.3% female, 42.7% male) from various disciplines at a metropolitan university in the Republic of Korea. They were contacted by email and asked to participate in the online survey. The profile of participants is presented in Table 1.

Table 1. Participant demographics

2.2 Measure

A questionnaire was developed as shown in Table 2 to investigate the perception of learning support functions in LMS among university students to which AI technology is applied. The questionnaire was revised and supplemented through continuous discussion by a research team composed of experts who have research expertise in Educational Technology and LMS. The LMS learning support function augmented with AI technology contained a total of 22 functions in the five categories drawing reference from previous studies: learning plan, learning progress, learning evaluation, learning reflection, and learning environment [15] [16] [17].

Table 2. Importance and Urgency for Learning Support Functions of LMS applied AI technology

First, in the learning plan section, a function was proposed to continuously revise and improve the learning plan by providing customized feedback and information while monitoring the progress of the learning goals set by the student. Second, in the learning progress section, a function was proposed that provides individually tailored learning materials, analyzed individual or team learning activities, helped AI tutors in the task performance process, and provided customized feedback on performance results. Third, in the evaluation section, automated scoring and feedback provision for descriptive evaluation items, customized feedback on evaluation results, and a function to analyze the contents of peer evaluation were proposed. Fourth, in the learning reflection section, the function of analyzing learning results by subject and suggesting improvements was proposed. Lastly, in the learning environment section, a function to set a personalized background or user interface was proposed.

The questionnaire consisted of a total of 44 items asking about the importance and urgency of 22 function items (3 learning plan items, 13 learning progress items, 3 learning evaluation items, 2 learning reflection items, and 1 learning environment item) rated on a 5-point Likert scale. While importance has a critical influence in setting high-priority goals [18], urgency has a significant impact on time-related decisions [19] [20]. The reliability coefficient (Cronbach α) of the questionnaire was 0.93.

2.3 Procedure

To examine the importance and urgency of the LMS learning support function augmented with AI technology according to the characteristics of college students (gender, grade level, major), and to examine the demand for the LMS learning support function to which AI technology was applied, the following analysis methods were used.

First, the mean of importance and urgency for each function was analyzed to find out the status of importance and urgency for the LMS learning support function augmented with AI technology among university students. Based on the overall mean, the functions that were recognized to be in need of development first because both importance and urgency responses were high, as well as functions that were found to be a lower priority or were judged to not be necessary because of low importance and urgency were analyzed. There were no functions found to be of low importance but high urgency or high importance but low urgency.

Second, the SPSS statistics program was used to statistically verify the difference in the perception of importance and urgency of the LMS learning support function augmented with AI technology according to the characteristics of college students (gender, grade level, major). Differences according to gender were analyzed by an independent t-test to satisfy the normality test, but the Kruskal-Wallis test, a rank-based nonparametric test, was implemented as the differences according to grade and major did not satisfy the normality test.

Third, in order to analyze the additional requirements for the LMS learning support function augmented with AI technology, a frequency analysis of the words presented in narrative response contents was conducted through text mining. For this purpose, the KoNLP library for Korean text data processing of the R statistical program was used.

3. Results

3.1 Analysis of the importance and urgency of the LMS learning support function augmented with AI technology

Table 2 shows the importance and urgency of the LMS learning support function augmented with 22 AI technologies in 5 areas presented in the questionnaire. The total mean of the importance and urgency measures were 4.19 and 3.93, respectively. In Fig. 1, functions showing both high importance and urgency based on each mean value are shown in red, and functions showing both low importance and urgency are shown in black. Recognizing sections where both importance and urgency are high, there are a total of 15 functions that need to be developed first. In the learning planning area (A1, A2, A3), all 3 functions correspond, and in the learning progress area, 8 functions (B1, B2, B3, B7, B8, B9, B10, B11) were applicable. In the learning evaluation area, two functions (C1, C2) correspond, in the learning reflection area, both functions (D1, D2) correspond, and in the learning environment area, there is no function matching the results. On the other hand, a total of 7 functions were found to be either subordinated or having no need for development because both importance and urgency were recognized as low. There are five functions in the learning progress section (B4, B5, B6, B12, B13), one function in the learning evaluation section (C3), and a function in the learning environment section (E1). However, there are no functions in the learning plan and learning reflection sections. In other words, while students recognize that the utilization of AI technology is important and must be applied quickly in all functions of learning planning and learning reflection, and in some functions of learning progress and learning evaluation, they think that application of AI technology in in the learning environment is neither important nor urgent.

Fig. 1. Means for Importance and Urgency

3.2 Analysis of differences in perception of importance and urgency of developing LMS functions with AI technology applied according to student characteristics

3.2.1 Gender

As a result of analyzing differences in perception of LMS learning support functions augmented with AI technology according to the gender of college students (Refer to Table 3), statistically significant differences were found in the learning evaluation and environment. Female students perceived the learning evaluation both of higher importance and urgency than male students. In the learning environment, there was no significant difference in importance, but in regard to urgency, it was found that female students perceived that the relevant functions should be provided as quickly as possible compared to male students.

Table 3. Differences in perception of importance and urgency by gender

*p<0.05; ***p<0.001

3.2.2 Grade level

As a result of analyzing differences in perception of LMS learning support functions augmented with AI technology according to grade level of college students (Refer to Table 4), no statistically significant difference was found in all sections . Namely, it can be considered that the difference in grades does not affect the degree of recognition of the importance or urgency of applying AI technology.

Table 4. Differences in perception of importance and urgency by grade level

3.2.3 Major

As a result of analyzing differences in perception by classifying the majors of the students who participated in the survey into three major categories (Humanities and Social Science, Science and Engineering, as well as Art, Music, and Physical Education) (Refer to Table 5), statistically significant differences in all sectionss did not appear. That is to say, it can be considered that the difference in major does not affect the degree of recognition of the importance or urgency of applying AI technology.

Table 5. Differences in perception of importance and urgency by major

3.3 Analysis of needs for LMS learning support function augmented with AI technology

As a result of a frequency analysis of sentences that students freely responded to in the descriptive form to describe the functions that they feel are also necessary, the top twenty most frequently appearing words, among the LMS learning support functions to which AI technology is applied, are presented in Table 6. A total of 557 words were used by the students, and the frequency of the word support function was the highest at 54, followed by learning (learner, student), assignment, lecture (class, course, subject), evaluation, and information. This can be seen that the LMS function that students require the application of AI technology are to provide information, feedback on the task progress, or learning evaluation results. In addition, it was found that the function of adjusting the learning time or guiding the learning activities by using AI technology for the learning plan is required.

Table 6. Result of the Frequency Analysis on additional demand

4. Discussion and Implications

Recently, as online and blended learning has expanded due to the influence of the COVID-19 pandemic, the role of LMS to support and manage the teaching and learning process has become more important. In the future, it will be necessary to provide intelligent LMS functions tailored to student learning activities. In this study, a basic study for the implementation of learner-centered intelligent LMS, the needs of college students with LMS learning support functions augmented with AI technology were analyzed. Discussion and implications of the study results are the following.

First, among the learning support functions that students want to be implemented in the LMS using AI technology, the function that students considered to be the most important but also felt an urgent need for in the evaluation was a function to give automated grading and feedback for their writing (C1). Whereas existing LMS mostly only present the scoring results for multiple choice or short-answer type evaluation questions. In a new LMS utilizing AI technology, it is necessary to develop a learning support functions to provide intelligent and personalized feedback, along with scoring results for descriptive evaluation items. This function will alleviate the burden of evaluating students' narrative responses by the instructor in large class teachinng, and will enable quantitative and qualitative activation of descriptive evaluation that can promote higher-order thinking of students [21].

Second, the functions with the highest score of importance and urgency in the learning progress section were related to receiving customized feedback and help on task performance processes and results (B9, B10, B11). In other words, students can see the results of qualitative analysis of their submitions, whether as assignments, reports, artifacts submitted by students, uploaded files, comments, or assignment scores and feedback (B9) and their desire to receive customized feedback based on the results of their assignments (B11). Since most LMS only present quantitative analysis presented as statistical values, there are many limitations in providing feedback according to the individual learning context and situation of students. Therefore, a function that can give customized feedback based on high-quality qualitative analysis of the content collected during the learning process through the implementation of AI technology in the future should be developed and utilized in LMS using eudcational big data. In addition, the function to request and receive help when students encounter difficulties in the process of performing a task or completing assignments (B10) was also ranked important and was desired to be implemented as soon as possible. In the educational environment of universities, where it is not easy to get help from real-time instructors, it is necessary to develop a function that helps students solve a variety of problems through AI technology as soon as possible so that better LMS can be implemented.

Third, in the learning plan and learning reflection section, students consider functions to receive adaptive feedback according to the learning progress in comparison to their learning plan (A2), as well as receive suggestions for improvement by diagnosing their strengths and weaknesses (D2) to be the most important and functions that need to be developed urgently. Existing LMS are focused on instructor centered functions to manage the learning of the students. However, now that the utilization of AI technology within LMS is possible, development of AI-based LMS focused on Learning Analytics is critical to the implementation of learning support functions that help students in self-directed learning, from the lesson planning to self-reflection and finally improving the learning process and results.

Fourth, the learning support function of LMS, which showed low importance and urgency, was a function for analyzing interactions between professors and students as well as between students. Such result may be due to the lack of practical interaction using LMS. If an LMS function that analyzes various interactions by utilizing AI technology is implemented in the future, it will be an opportunity to promote interpersonal interactions in an online educational environment.

Fifth, the LMS function of allowing personalized backgrounds and user interfaces (UI) / user experience (UX) for students in the learning environment was not perceived as important overall or urgently needed, but female students perceived the urgency to be higher than male students. This implies that the requirements for the learning environment may differ depending on the characteristics of the learner such as gender. Therefore, the function of the changing the learning environment should be applied to the LMS function development process to which AI technology is applied by deriving factors according to the characteristics of the learner through future research.

5. Conclusions, Limitations, and Future Research

This study is meaningful and significant in that it examines AI technology-based LMS learning support functions and explores future development plans based on a student needs analysis. In the online learning environment that will continue to expand in the future, the self-regualted learning processes that students plan and implement on their own is very important, but there are problems in that most learners are not ready for self-regulated learning [22]. In particular, it was reported that students did not properly recognize the value of a task when they were given an online task, and their learning motivation was lower due to emtional burden and cognitive overload [23].

The results of applying an AI-based approach to data sets collected from student interactions with the LMS could be used to develop predictive models that predict potential students at risk [24] . Identification of at-risk students helps increase memory and retention rate by intervening in a just-in-time manner in the learning process to improve their academic progress and performance [12]. The learning analytics approach contributes to the knowledge area as it utilized a variety of AI techniques to improve the accuracy of predictive model to select best performing predictive model [24]. In the results of Jho’s [25] study, exploring the learner achievement prediction model in the online education environment, the performance level of a learning task was presented as a major variable predicting the achievement of online lectures. According to the results of the student needs analysis, it is urgent to develop LMS learning support functions that reduces the burden of online learning tasks by providing help in real time by utilizing AI technology and providing customized feedback that can suuport self-directed learning. The learners will take advantage of the AI-based LMS to gain the adaptive learning support tools and materials by predictive model [26] [27] [28]. Thus, LMS using AI could offer a personalized intelligent learning environment to respond to diverse students with different learning styles and activities.

On the other hand, in this study, the difference in requirements for the LMS functions with the AI technology in only three aspects (gender, grade level, and major) of learner characteristics was analyzed for 192 college students, which can be said to have a relatively small sample size. It is necessary to be careful that the interpretation of this result is broadly adopted. In addition, since the area of evaluating their perceptions was only limited to 22 functions in a total of five categories (learning plan, learning progress, learning evaluation, learning reflection, and learning environment). Therefore, further investigation on the demand for AI application in consideration of various learner characteristics and LMS functions should be conducted in the future.

If learning support functions in LMS utilizing AI technology derived from this study are developed, it will be possible to lead change and innovation in the current educational environment. For example, the creation of an automated scoring function for descriptive questions in addition to scoring traditional multiple-choice questions that only evaluate the students' final level of proficiency, can lead to a process-oriented assessment that diagnoses the learning situation of students, evaluates the learning process and offers instructional interventions designed to prevent dropout. Furthermore, the learning support function that quantitatively and qualitatively analyzes the student-student interactions can provide adaptive feedback and guidance to check the collaborative learning process and improve student interactions. Lastly, the results of the student needs analysis presented in this study provide basic information for deriving and prioritizing the learning support functions that need to be developed when AI technology is used to implement learner-centered intelligent LMS in the future.