I. Introduction
MOOC (Massive Open Online Course), a public online open course, has contributed to popularizing higher education and lifelong education at home and abroad. In the case of Korea, the Korean Massive Open Online Course (K-MOOC) operation project has been promoted since October 2015 in order to open the excellent lectures of universities to the general public. It started with 27 courses and 56,000 course registrations in 2015 due to the merit of pursuing equal opportunities for education so that anyone can receive quality education anytime, anywhere, free from time and place restrictions, age restrictions, tuition burden, and admission requirements. As a result, the number of courses offered and the number of applicants has increased to 999 courses and 522,000 cases in 2020 [1].
However, K-MOOC lacks contents that meet the needs of learners and the completion rate is low (23.9%, as of August 2020). Also the educational and administrational basis for recognizing university students’ course completion as credits is insufficient. It is pointed that the system for efficient operation is not constructed [2]. Due to COVID-19, the number of course registrations in March and April 2020 (178,687 cases) was on the increase 78 percent, compared to the number in March and April 2019 (100,534 cases). In spite of the quantitative growth, it has been pointed out that the courses opened at MOOCs often use instructional methods that just convey knowledge and theory, which is insufficient to motivate learners and difficult to anticipate high educational effect [2].
According to Gartner's hype cycle, which shows the maturity level of technological development for MOOCs, MOOCs were judged to be in the trough of disillusionment after going through the technological nascent stage in 2011 [3]. In terms of business model, Gartner estimates that MOOCs services are not progressing in the form of replacing existing higher education institutions, contrary to the initial forecast. So MOOCs are devalued enough to be excluded from the education sector's hype cycle [4].
Questions are constantly being raised as to why the educational innovation using MOOCs, which was expected to replace the existing university education system, provide equal educational opportunities, and reduce educational costs, is being delayed. So far K-MOOC has reached its limits due to development-oriented approach, emphasis on credit linkage, video-oriented lectures, and lack of publicity [1]. Due to the relatively small number of courses compared to the number of courses offered by global MOOCs such as Coursera, Edx, and Udacity, a development-oriented approach that increases the number of courses developed with participating universities has been taken. So the learner motivation approach has not received attention. In addition, as long-term courses of 15 weeks or longer that can be recognized for credit, are mainly operated in order to solve the problem of the consistently low completion rate, the learning threshold is increased for adult learners. Since MOOCs consist of mainly videos and quizzes, they focused on imparting knowledge and had a limited ability to improve practical competency.
The limitations and problems of MOOCs mentioned above are basically judged to be caused by a supplier-centered thinking (or approach) at the national level. MOOCs still did not deviate from content delivery-centered education of suppliers. When designing education, education is not accepted as a service that enables learners to create value. As a result, it was difficult to overcome the abstraction of ‘education’ and to establish specific measures for learners to have an optimal educational experience [5]. In addition, since educational services have specific service targets and goals, they help to focus on educational activities, that is, learners' educational service experiences. It can contribute to achieving customized individualized education [5,6].
In resolving the problems faced by MOOCs, the educational service perspective that advocates a systemic and ecological approach for creating shared value between learners and instructors and solving educational problems can be beneficial [3], [6]. This view is in line with the core aim of the OECD 2030 Project, which has been promoted since 2015 by the OECD for educational innovation of nurturing talents suitable for the future society. This project emphasizes the concept of co-agency along with student agency and teacher agency. The perspective of co-agency seeks to achieve classroom innovation in which student agency is realized through close interaction with parents, local communities, and the state as well as students and teachers for a rich learning experience environment.
Because quality is viewed through different lenses, the meaning of quality varies according to context and actors (government, institution, instructor, learner, etc.). As the initiative towards MOOCs is shifting to learners, learners' perceptions, behaviors, and experiences should be the basis in the process of evaluating the quality of MOOCs [7].
The educational service perspective also holds the view that value is created by close interaction between multiple stakeholders surrounding education as well as teachers and learners [6]. In a word, it goes beyond value-in-exchange from the supplier's point of view and expands the approach to value-in-use, value-in-experience, and value-in-life [8].
As academic efforts to evaluate MOOC quality, there have been researches including development of quality control benchmark tool for MOOC [9], development of MOOC course quality certification and approval guidelines [7,10], MOOC quality control checklist development [11], etc. Various related studies have been conducted to improve the quality level of MOOCs. Rosewell and Jansen (2014)[9] developed OpenupEd Quality Label, a quality management tool for European MOOC portal OpenupEd (www.openuped.eu). For the evaluation of MOOCs quality, Gu and colleagues (2021)[12] suggested seven areas for evaluating MOOCs quality; quality of information, system quality, service quality, confirmation, perceived usefulness, satisfaction, and willingness to continue. CoL (Commonwealth of Learning) (2016)[10] presented eight areas as a checklist for MOOCs approval: course outline, learning outcomes, evaluation, content validity, learners, educational engineering, course resources, and learner support resources. Most of the related studies focused on quality control from the perspective of the education provider. There was a study [3] that attempted to propose a theoretical proposal for the MOOCs quality management from the perspective of education consumers, but it had limitations in that content validity and statistical validity were not verified.
Therefore, it is required that an evaluation model of MOOCs quality should consider the integrated perspectives of education providers and consumers so that learners' educational experiences can be optimized in the teaching and learning process using MOOCs. The purpose of this study is to derive an evaluation model of MOOCs quality that can diagnose the value creation of education based on co-agency from the perspective of education service, and to develop a reliable and valid evaluation tool of MOOCs quality based on the model. Research questions are as follows: (a) What is the theoretical model for evaluating MOOCs from the perspective of education service? (b) What is the final evaluation model revised by statistical validation?
II. Education Service and MOOCs
1. MOOCs as an Education Service
Although MOOCs at home and abroad have grown rapidly, led by the private sector and the state since 2012 and 2015, respectively, they have fundamental limitations in not improving the completion rate (15% in 2012 and 24% or less in 2015), which is the practical effects of education. The serious absence of teaching methods, the homogenization and depersonalization of education, and the excessive influence of private companies on academic disciplines are mentioned as problems in MOOCs and especially, educational limitations with the quality of instructional design including learner motivation and supports have been pointed out as the main cause of low completion rates [13]. Those problems are based on the perspective that education is a product from the supplier's point of view, and an approach based on the goods dominant logic that values are created through value-in-exchange with learning consumers. Developers and providers in MOOCs take the position of providing a kind of educational product, and instructors also focus on providing well-designed videos content to learners.
The concept of education quality is an unclear and controversial [14]. However, the definition of higher education quality can be evaluated by college students with education services which higher education institutions provide. Students’learning experiences in engaging with the different services comprise service quality, because they are the main stakeholders in higher education institutions [15].
In this respect, a new interdisciplinary approach, education service science, can be proposed as a useful way to solve the limitations faced by MOOCs. Education service science goes beyond the abstract ideology of the word 'education' and focuses on practical activities on the premise of specific activities and objects of 'education service' and can specify the goals and results to be achieved [5]. Therefore, when looking at MOOCs as 'education service' rather than 'education' [5], it is possible to focus on designing for learners to have optimal educational experiences online. Education service science focuses on the education service system and education service-oriented logic as its core constructs. It approaches the education service with a service-oriented logic, and reconstructs the current education service system. The ultimate goal is to derive education service innovation and to create the value of learning experience. Value creation in education is created not through simple exchange activities. It is created for learners to use and experience education service. The educational value of MOOC is also determined by the learning experience of learners rather than the delivery of educational content itself.
MOOC content developers and operating faculty should design education services so that learners can have an optimal online learning experiences. For this, it is important to understand the characteristics of educational services. Education services have the characteristics of intangibility, inseparability, heterogeneity, and perishability [6,13]. Since education services have no form (intangibility), it is necessary to design teaching and learning so that learners, who are education consumers, can experience the value of the service. And education services cannot be separated from the delivered educational content (inseparability). Therefore, at each stage of education, it is necessary to focus on managing touch-points with learners by providing experiences of interaction. Even if an instructor provides the same education service, each learner perceives different experiences (heterogeneity). A systemic or integrated approach is required to overcome the heterogeneity of educational experiences for these education services. Education services should be designed by considering each system level (e.g. organization-program-individual) in an integrated way. Education services are destroyed (perishability) as soon as they are delivered to learning customers. Therefore, education services should be designed to be memorable education experiences [8], that is, experiences to co-create values through mutual cooperation among instructors, learners, and related stakeholders [3,6].
2. A Potential Model for MOOCs Quality Evaluation from the Perspective of Education
In order to develop a potential model for MOOCs quality evaluation from the point of view of education services, previous researches related to MOOCs evaluation and literature on education services were analyzed. And considering four methods (visualization, touch-point management with learners, systemic approach, creating memorable experiences) that overcome the limitations of the aforementioned fundamental characteristics (intangibility, indivisibility, heterogeneity, and perishability) of education services [13,16], we researchers set providing value experience, providing sustainable experience, providing integrated access opportunity, and providing opportunity for value creation as key factors for MOOCs quality evaluation from the point of view of education services, as shown in Figure 1.
Fig. 1. Potential Model for MOOCs Quality Evaluation at the Perspective of Education Service
The MOOCs quality evaluation criteria from the perspective of educational service consists of four factors: value experience, sustainable experience, integrated access opportunity, and opportunity for value creation. They are four interrelated domains.
The ‘value experience’ evaluates MOOCs focusing on the strategy to overcome the intangibility of MOOCs education services, and consists of sub-factors of reality and reliability. ‘Reality’ refers to the deep connection of MOOCs contents with real life, dealing with the competencies required in real life and letting learners experience the problem-solving process beyond the simple transfer of knowledge [17,18]. ‘Reliability’ refers to the belief that high-quality MOOC education services can be provided. It evaluates the professor's expertise in the content of the online class, and the perceived value compared to the time and effort which are invested [19,20,21].
The ‘sustainable experience’ is an evaluation area focusing on the strategy to tide over the inseparability of MOOCs education services, dealing with sub-factors of certainty and reactivity. 'Certainty' means predictable and stable support of education services, by giving evaluation scores appropriate for online learning activities and conducting online classes to maintain the optimal level of learning experience before, during, and after class [22]. ‘Reactivity’ refers to the degree of individual interest paid to learners, by showing equal interests to all learners and providing appropriate and immediate feedback on learners’ questions, presentations, discussions, and activities [22]).
The ‘integrated access opportunity’ evaluates MOOCs focusing on strategy to overcome the heterogeneity of MOOCs education services, and consists of sub-factors of organizational environment systematization, program structure, and systematization of learner support. ‘Organizational environment systematization’ refers to the systematic provision of online class environments at a macro level, by providing class environments suitable for online classes and maintaining the appropriateness of online content quality and sound quality [22]. 'Program structure' evaluated the systematic provision of teaching and learning activities at the meso level, by guiding students to have multiple perspectives, and maintaining the consistency among the learning goal, content & method, and evaluation of online classes, conducting a learner-centered learning activity class [21,22]. 'Systematization of learner support' refers to the provision of learner motivational strategies at the micro level, providing opportunities for students to take their own initiative in the learning process, encouraging students to have self-confidence [17,22]
The final area 'opportunity of value creation' focuses on tiding over the perishability of education services, and co-agency is a sub-factor. ‘Co-agency’ evaluates providing opportunities for positive mutual cooperation of instructor-student and student-student, and providing classes that are linked to various fields such as the local community and industry [23,24].
Table 1. Theoretical Foundation of MOOCs Quality Evaluation at the Perspective of Education Service
III. Method
1. Participants
The 138 college students and graduates completing the online survey had basic understandings of MOOCs as learners who had taken classes related to distance education theory and practice, or participated in MOOC-based classes. The data of 136 were analyzed, except 2 insincere answers. The evaluation group was composed of 45 males (33.3%) and 90 females (66.7%)1), and 112 college students (82.4%) and 24 graduates (17.6%). Online survey was conducted, checking content validity (5-point scale) of 18 items which are developed on the potential model for MOOCs quality evaluation from the perspective of education service.
2. Instruments
The potential model for MOOCs quality evaluation comprised 18 items; Value experience (4 items), Sustainable experience (4 items), Integrated access opportunity (7 items), Opportunity for value creation (3 items). A 5-point Likert-type scale was used. The Cronbach’s alpha statistics for the potential model were .94. The detailed item structure and reliability analysis results are shown in <Table 2>.
Table 2. Structure of Tool for Evaluating MOOCs Quality at the perspective of Education Service
3. Procedures and data analysis
In order to derive a potential model for MOOCs quality evaluation and to validate the model from the perspective of education service, an evaluation criteria framework was developed based on a literature review. 18 items were developed based on the potential model for MOOCs quality evaluation. The importance (5-point scale) of 18 items was evaluated by 5 experts with Ph.D. in education to secure the content validity. According to the diagnosis results of Content Validity Index (CVI)2) suggested by Fehring (1987) [25], 4 items (B2, B3, C3, C5) were considered as non-major characteristics. Those items were revised and supplemented. After data cleaning, 136 surveys were used for exploratory factor analysis (principal component analysis. varimax rotation) and reliability analysis that confirmed the fitness of the potential model.
IV. Results
As a result of analyzing the data collected from the online survey about the importance of items for MOOCs quality evaluation, the descriptive statistics are as shown in <Table 3>. The average value of all questions was 4.2, and the average value of each question was located between 3.88 and 4.46. Therefore, as a whole, each item could be said to be good as an item to evaluate the quality of MOOCs.
Table 3. Mean and Standard Deviation for the variables
As a result of exploratory factor analysis, KMO (Kaiser-Mayer-Olkin) values was .96, and the Bartlett sphericity test value was also statistically significant even at a reliability level of 99%, indicating that common factors exist between items. In order to investigate whether the items belonging to each evaluation criterion are grouped into four factors, the principle component analysis method was used and the varimax method was adopted as the rotation method. The explanatory variance ratio of each factor was 69.1%, and the factor loading of all items was .49 or higher (see <Table 4>). In addition, as a result of comprehensively considering the eigenvalues and screening test results, the items for each evaluation criterion could be explained as four factors (see <Table 5>).
Table 4. Proportion of variance per eigenvalue
Table 5. Result of Factor Analysis for Four-Factor Model
Table 6. Result of Factor Analysis for Sub-factors Model of 1st Factor (2nd EFA)
Table 7. Result of Factor Analysis for Sub-factors Model of 2nd Factor(2nd EFA)
Table 8. Result of Factor Analysis for Sub-factors Model of 3rd Factor(2nd EFA)
Table 9. Result of Factor Analysis for Subfactors Model of 4rd Factor(2nd EFA)
Reliability analysis was performed to determine whether the items selected as the result of exploratory factor analysis showed the degree of internal fit as the items constituting each scale. The overall reliability value (Cronbach α) was .938, which was very high, and the reliability value for each domain was also between .638 and .926, indicating a high level of reliability (see <Table 10>).
Table 10. Structure of Final Model
V. Conclusions
This study intended to provide educational implications for improving the quality of MOOCs from the perspective of education service as an innovation of education. It tried to take an interdisciplinary approach of education service focusing on co-creation of value [5]. For tiding over the fatal limitation of present MOOCs with low completion rate, it is requested that the new approach that education is service, should be considered.
The purpose of this study was to develop an evaluation tool for MOOCs quality from the perspective of education service and to verify its statistical validity and reliability. A potential model for MOOCs quality evaluation was derived based on the four factors; value experience, sustainable experience, integrated access opportunity, and opportunity for value creation. After developing 18 evaluation items based on this model, exploratory factor analysis and reliability analysis were conducted using the data collected through an online survey asking for the importance of the items. As a result of the analysis, it was confirmed that the developed evaluation tool had statistical validity and reliability. The final model for evaluating the quality of MOOCs from the perspective of education service, was composed of systemic learning experience, value experience, co-creation of value experience, and high order learning experience. This is the result that proves the rationality of the development and application of MOOC's quality evaluation tool that aims to achieve co-creation of value through close interaction between learners as consumers for education and instructors as providers for education beyond the existing education provider-centered perspective.
Suggestions and conclusions based on these research results are as follows. First, the evaluation tool for MOOCs quality from the perspective of education service that comprehensively reflects both the perspectives of education service providers and consumers, will be available as a self-diagnosis tool for designers and instructors when designing, developing, and operating MOOCs. In other words, the criterion of the evaluation tool for MOOCs whether they provide learners with participatory experience, learning experience, and value experience, can be helpful in designing MOOCs from the perspective of education service.
Second, for learning success in MOOCs, learner’s experience should be a main factor in designing MOOCs. The final confirmed model consisted of four factors named as systemic learning experience, value experience, co-creation of value experience, and high order learning experience. The effectiveness of the learner’s experience is focused rather than the viewpoint of resource efficiency [3]. When designing MOOCs, therefore, the designer should focus on the learner’s viewpoint enabling the learner to have an optimal experience in the teaching and learning process as well as the provider's point of view, which focuses on delivering the teaching and learning content [3,26]. The education service breaks the boundaries between the supplier and the consumer and places importance on the relationship between their interactions. During the learning experience, the education service customers invest time and effort to perform high-quality learning tasks with enthusiasm, learn with peers under the guidance of instructors, and actively participate in the organization’s education system, such as using resources and support [3]. The evaluation tool’s items for MOOCs quality from the perspective of education service are supported by the result of Aldowah and his colleagues’study [27] ; six core factors that directly influenced student dropout in MOOCs academic skills and abilities, prior experience, course design, feedback, social presence, and social support.
There were some limitations in this study. This tool did not consider both of two distinct types of MOOCs such as cMOOC or connectivist MOOC and xMOOC or eXtended MOOC. It treated mainly with cMOOCs which are based on principles of connectivist pedagogy rather than xMOOCs, the content-based or professor centric MOOCs, which reflect a more traditional learning approach of knowledge duplication through video presentations and short quizzes and tests [13]. In addition, the fact that some of the students who participated in this study did not directly experience the MOOC class has limitations in interpreting the research results. Future research could perform formative research of an instructional guideline about how MOOCs may be designed according to the criterion of an evaluation tool for MOOCs from the perspective of education science.
To innovate the education service which is called as MOOCs, the model for MOOCs quality evaluation focusing on the essential characteristics of the service (intangibility, inseparability, heterogeneity, perishability) will be helpful in making learners take optimal experiences and stay long in MOOCs. A service or product that impresses consumers makes them purchase the service, and an educational service that impresses the learner leads them to immerse themselves in the learning process and achieve learning success.
ACKNOWLEDGEMENT
This research was supported by the Korea Nazarene University Research Grants in 2023
Appendix
References
- Ministry of Education. Learner centered K-MOOC 2.0: 2021 Operational plan of MOOC, 2021.
- I.S.Cho. Present status and improvement tasks for K-MOOC. NARS Current Issues & Analysis, 171, National Assembly Research Service., 2020.
- S.W. Kim. What brings about the success of MOOCs in the perspective of education service? In MOOC (Massive Open Online Courses) (chapter 1, pp.3-16). London; Intechopen. [Internet]. 2022. DOI:10.5772/intechopen.99053
- Ministry of Science and ICT. The effect of ICT on education. Issues & Analysis, Vol. 85,
- H.Y. Jang, P.G. Baek, S.W. Kim, J.S. Hong. A Theoretical Exploration for Establishing the Discipline of 'Education Service Science'. Journal of Corporate Education and Talent Research, Vol. 20, No. 4, pp.125-147, 2018. https://doi.org/10.46260/kslp.20.4.5
- S.W. Kim, H.K. Lee. Validation of a tool evaluating distance teaching competencies for higher education in the perspective of education service. The Journal of Korean Education, Vol. 48, No. 3, pp.153-174, 2021. DOI:10.22804/jke.2021.48.3.001.
- N. Hood, A. Littlejohn. MOOC Quality: The Need for New Measures. Journal of Learning for Development. Vol. 3, No. 3, pp.28-42,
- L. Cantone, P. Testa, T.Marrone. Service-Dominant Logic: Inward and Outward Views. In Handbook of Service Science, Volume II. Service Science: Research and Innovations in the Service Economy(pp.675~710). Springer, Cham, 2019. DOI:10.1007/978-3-319-98512-1_30
- J. Rosewell, D. Jansen. The OpenupEd quality label: Benchmarks for MOOCs. INNOQUAL: The International Journal for Innovation and Quality in Learning, Vol. 2, No. 3, pp.88-100, 2014.
- CoL (Commonwealth of Learning). Guidelines for quality assurance and accreditation of MOOCs. Commonwealth of Learning, 2016. http://oasis.col.org/bitstream/handle/11599/2362/2016__Guidelines-QAA-MOOCs.pdf?sequence=6&isAllowed=y
- M. Yepes-Baldo, M. Romeo, C. Martin, M. A. Garcia, G. Monzo, A. Besoli. Quality indicators: Developing "MOOCs" in the European higher education area. Educational Media International, Vol. 53, No. 3, pp.184-197, 2016. DOI: 10.1080/09523987.2016. 1236998
- W. Gu, Y. Xu, S. Zeng-Jun. Does MOOC quality affect users' continuance intention?. Sustainability, Vol. 13, No. 22, 12536, 2021. DOI:10.3390/su132212536
- S.W. Kim. MOOCs in Higher Education, In Dragan Cvetkovic (ed.). Virtual Learning (chapter 8, pp. 121-135), IntechOpen, 2016. DOI: 10.5772/66137.
- Y.C. Cheng, M.M. Tam. Multi-model of quality in education. Quality Assurance in Education, Vol. 5, No. 1, pp.22-31, 1997. DOI:10.1108/09684889710156558
- J. Jancey, B. Burns. Institutional factors and the postgraduate student experience. Quality Assurance in Education, Vol. 21, No. 3, pp.311-322, 2013. DOI:10.1108/QAE-Nov-2011-0069
- J.S. Hong, H.Y. Jang. Combining Education Service and Design for the Improvement of Educational Activities: Exploring the Methodologies of Education Service Design. Journal of Corporate Education and Talent Research, Vol. 21, No. 3, pp.53-70, 2019. DOI:10.46260/KSLP.21.3.3
- R. Conrad, A. Donaldson. Engaging the online learner: Activities and resources for creative instruction. San Francisco: Jossey-Bass, 2004.
- S.M. Kosslyn, B. Nelson. Building the intentional University. MIT, 2017.
- P.G. Baek, S.W. Kim. Analysis of Innovation Cases in the University From the Viewpoint of Education Service. Journal of Corporate Education and Talent Research, Vol. 21, No. 3, pp.155~177, 2019. DOI:10.46260/KSLP.21.3.7
- B.S. Cho. Future education, Olin. Seoul: Threechairs, 2017..
- H.W. Park. An innovation case of higher education: Franklin W. Olin College of Engineering, Educational Development, Vol. 44, No. 3, pp.56-61, 2017.
- M. Simonson, S. Smaldino, M. Albright, S. Zvacek. Teaching and learning at a distance: Foundations of distance education, (2nd ed.). Upper Saddle River, NJ: Prentice Hall, 2003.
- R.M. Palloff, K. Pratt. Building learning communities in cyberspace: Effective strategies for classroom. San Francisco, CA: Jossey-Bass, 1999.
- N. Dabbagh, B. Bannan-Ritland. Online learning: Concepts, strategies, and application. Columbus, OH: Merrill/Prentice-Hall, 2005.
- H. J. Fehring. Methods to validate nursing diagnosis. Heart and Lung, Vol. 16, No. 6, pp.625-629, 1987.
- R. Johnston, G. Clark, M. Shulver. Service operations management (4th ed.). Harlow: Pearson, 2012.
- H. Aldowah, H. Al-Samarraie, A.I. Alzahrani, N. Alalwan, (2020). Factors affecting student dropout in MOOCs: a cause and effect decision-making model. Journal of Computing in Higher Education, Vol. 32, pp.429-454, 2020. DOI:10.1007/s12528-019-09241-y.