Educational Technology International

The Korean Society for Education Technology (한국교육공학회)
권호 표지
DOI QR코드

DOI QR Code

Using topic modeling-based network visualization and generative AI in online discussions, how learners' perception of usability affects their reflection on feedback

  • Received : 2024.02.29
  • Accepted : 2024.04.21
  • Published : 2024.04.30
Download PDF
⟨ Previous Next ⟩

Abstract

This study aims to analyze the impact of learners' usability perceptions of topic modeling-based visual feedback and generative AI interpretation on reflection levels in online discussions. To achieve this, we asked 17 students in the Department of Korean language education to conduct an online discussion. Text data generated from online discussions were analyzed using LDA topic modeling to extract five clusters of related words, or topics. These topics were then visualized in a network format, and interpretive feedback was constructed through generative AI. The feedback was presented on a website and rated highly for usability, with learners valuing its information usefulness. Furthermore, an analysis using the non-parametric Mann-Whitney U test based on levels of usability perception revealed that the group with higher perceived usability demonstrated higher levels of reflection. This suggests that well-designed and user-friendly visual feedback can significantly promote deeper reflection and engagement in online discussions. The integration of topic modeling and generative AI can enhance visual feedback in online discussions, reinforcing the efficacy of such feedback in learning. The research highlights the educational significance of these design strategies and clears a path for innovation.

Keywords

Acknowledgement

This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea(NRF-2020S1A5A2A01040909).

References

  1. Abawajy, J. (2012). Analysis of asynchronous online discussion forums for collaborative learning. International Journal of Education and Learning, 1(2), 11-21.
  2. Abdul Rabu, S., & Badlishah, N. (2020). Levels of students' reflective thinking skills in a collaborative learning environment using Google docs. TechTrends, 64, 533-541. https://doi.org/10.1007/s11528-020-00504-5
  3. Blei, D. (2012). Probabilistic topic models. Communications of the ACM, 55(4), 77-84. https://doi.org/10.1145/2133806.2133826
  4. Blei, D., & Lafferty, J. (2006). Correlated topic models. Advances in neural information processing systems, 18, 147.
  5. Chang, J., Gerrish, S., Wang, C., Boyd-Graber, J., & Blei, D. (2009). Reading tea leaves: How humans interpret topic models. Advances in Neural Information Processing Systems, 22.
  6. Cho, Y., Park, H., Kim, J., Seok, Y., & Lee, S. (2015). Exploring roles of feedback to facilitate online discussion, Asian Journal of Education, 16(2), 289-313. https://doi.org/10.15753/AJE.2015.16.2.011
  7. Dabbebi, I., Iksal, S., Gilliot, J. M., May, M., & Garlatti, S. (2017). Towards adaptive dashboards for learning analytic: An approach for conceptual design and implementation. In 9th international conference on computer supported education (pp. 120-131).
  8. De Wever, B., Schellens, T., Valcke, M., & Van Keer, H. (2006). Content analysis schemes to analyze transcripts of online asynchronous discussion groups: A review. Computers & education, 46(1), 6-28. https://doi.org/10.1016/j.compedu.2005.04.005
  9. Dillon, A., &Morris, M. G. (1996). User acceptance of new information technology: theories and models. In M. Williams (ed.) Annual Review of Information Science and Technology (Vol. 31, pp. 3-32). Informtion Today.
  10. DiMaggio, P., Nag, M, & Blei, D. (2013). Exploiting affinities between Topic modeling and the sociological perspective on culture, Poetics, 41(6), 570-606. https://doi.org/10.1016/j.poetic.2013.08.004
  11. Fung, Y. (2004). Collaborative online learning: Interaction patterns and limiting factors. Open Learning: The Journal of Open, Distance and e-Learning, 19, 135-149. https://doi.org/10.1080/0268051042000224743
  12. Grun, B., & Hornik, K. (2011). Topic models: An R package for fitting topic models. Journal of statistical software, 40, 1-30.
  13. Hewitt, J. (2005). Toward an understanding of how threads die in asynchronous computer conferences. Journal of the Learning Sciences, 14(4), 567-589. https://doi.org/10.1207/s15327809jls1404_4
  14. Im, K., & Jin, S. (2021). Students' reflection levels by feedback type in asynchronous online group discussion activities, Korean Journal of Educational Research, 59(4), 31-62.
  15. Jang, M., & Lee, H. (2023). Analyzing learners' intention to use topic modeling-based visual feedback: Focusing on reflection activities in online discussion learning, The Korean Journal of Educational Methodology Studies, 35(1), 155-174.
  16. Jeon, M., Kwon, K., & Bae, H. (2023). Effects of different graphic organizers in asynchronous online discussions. Educational technology research and development, 71(2), 689-715. https://doi.org/10.1007/s11423-022-10175-z
  17. Jin, S., & Yoo, M. (2018). Dashboard development and usability test to support contents analysis of team-based online discussion, Journal of Educational Technology, 34(3), 587-616. https://doi.org/10.17232/KSET.34.3.587
  18. Joksimovic, S., Gasevic, D., Kovanovic, V., Riecke, B., & Hatala, M. (2015). Social presence in online discussions as a process predictor of academic performance. Journal of Computer Assisted Learning, 31(6), 638-654. https://doi.org/10.1111/jcal.12107
  19. Jyothi, S., McAvinia, C., & Keating, J. (2012). A visualization tool to aid exploration of students' interactions in asynchronous online communication. Computers & Education, 58(1), 30-42.
  20. Kember, D., McKay, J., Sinclair, K., & Wong, F. (2008). A four-category scheme for coding and assessing the level of reflection in written work. Assessment & Evaluation in Higher Education, 33(4), 369-379. https://doi.org/10.1080/02602930701293355
  21. Kim, H. (2023). A Developmental Study on a Dashboard-based Chatbot for Reflection on Online Discussion, [Master's thesis, Seoul National University].
  22. Kline, T. (2005). Psychological testing: A practical approach to design and evaluation. Sage publications.
  23. Kolb, D. (2014). Experiential learning: Experience as the source of learning and development. FT press.
  24. Kwon, K., & Park, S. (2017). Effects of discussion representation: Comparisons between social and cognitive diagrams. Instructional Science, 45, 469-491. https://doi.org/10.1007/s11251-017-9412-6
  25. Lao, T. (2002). A description of the experiences, perceptions, and attitudes of professors and graduate students about teaching and learning in a web-enhanced learning environment at a Southwest border institution. New Mexico State University.
  26. Lee, D., Yi, H. (2021). Exploring methods for determining the appropriate number of topics in LDA: Focusing on perplexity and harmonic mean method, Journal of Educational Evaluation, 34(1), 1-30.
  27. Lim, J., Oh, Y., & Ahn, M. (2021). An analysis of research trends on maker education using LDA-based topic modeling, Journal of Korean Association for Educational Information and Media, 27(3), 1189-1219. https://doi.org/10.15833/KAFEIAM.27.3.1189
  28. Lim, K., Kim, H., Choi, J., & Kim, Y. (2020). Usability test of learner dashboard: Visualization of multi-dimensional interaction information, Journal of Korean Association for Educational Information and Media, 26(2), 395-424. https://doi.org/10.15833/KAFEIAM.26.2.395
  29. Lim, K., Park, H., & Kim, H. (2014). Effects of social network-based visual feedback on learning in online discussion, Journal of Educational Technology, 30(3), 443-466. https://doi.org/10.17232/KSET.30.3.443
  30. Lim, S., & Kim, E. (2017). The design of dashboard for instructor feedback support based on learning analytics, The Journal of Korean Association of Computer Education, 20(6), 1-15. https://doi.org/10.32431/KACE.2017.20.6.001
  31. Lin, K., Hong, H., & Chai, C. (2014). Development and validation of the knowledge-building environment scale. Learning and Individual Differences, 30, 124-132. https://doi.org/10.1016/j.lindif.2013.10.018
  32. Malheiro, S., Morgado, L., & Mendes, A. (2008). Analysis of engaged online collaborative discourse: A methodological approach. In A. J. Mendes, I. Pereira, & R. Costa (Eds.), Computers and education: Towards educational change and innovation (pp. 33-43). UK: Springer.
  33. Matsuzaw, Y., Oshima, J., Oshima, R., Niihara, Y., & Sakai, S. (2011). KBDeX: A platform for exploring discourse in collaborative learning. Procedia-Social and Behavioral Sciences, 26, 198-207. https://doi.org/10.1016/j.sbspro.2011.10.576
  34. Mezirow, J. (1978). Perspective transformation. Adult education, 28(2), 100-110. https://doi.org/10.1177/074171367802800202
  35. Mezirow, J. (1991). Transformative dimensions of adult learning. Jossey-Bass.
  36. Mimno, D., Wallach, H., Talley, E., Leenders, M., & McCallum, A. (2011, July). Optimizing semantic coherence in topic models. Proceedings of the 2011 conference on empirical methods in natural language processing (pp. 262-272).
  37. Nussbaum, E., & Schraw, G. (2007). Promoting argument-counterargument integration in students' writing. The Journal of Experimental Education, 76(1), 59-92. https://doi.org/10.3200/JEXE.76.1.59-92
  38. Park, J., Park, H., Jung, S., & Lim, K. (2022). An analysis of research trends on personalized learning: Using TF-IDF, LDA based text mining, The Korean Journal of Educational Methodology Studies, 34(4), 711-739.
  39. Park, Y., & Jo, I. (2019). Factors that affect the success of learning analytics dashboards. Educational Technology Research and Development, 67, 1547-1571. https://doi.org/10.1007/s11423-019-09693-0
  40. Rohfeld, R., & Hiemstra, R. (1995). Moderating discussions in the electronic classroom. Computer Mediated Communication and the Online Classroom, 3, 91-104.
  41. Rohloff, T., Sauer, D., & Meinel, C. (2019, December). Student perception of a learner dashboard in MOOCs to encourage self-regulated learning. In 2019 IEEE international conference on engineering, technology and education (TALE) (pp. 1-8). IEEE.
  42. Roskos, K., Vukelich, C., & Risko, V. (2001). Reflection and learning to teach reading: A critical review of literacy and general teacher education studies. Journal of Literacy Research, 33(4), 595-635. https://doi.org/10.1080/10862960109548127
  43. Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 30.
  44. Smith, D., & Smith, K. (2014). Case for 'Passive' learning-the 'Silent' community of online learners. European Journal of Open, Distance and E-Learning, 17(2), 86-99. https://doi.org/10.2478/eurodl-2014-0021
  45. Sun, L., & Vassileva, J. (2006). Social visualization encouraging participation in online communities. In Groupware: Design, Implementation, and Use: 12th International Workshop, CRIWG 2006, Medina del Campo, Spain, September 17-21, 2006. Proceedings 12 (pp. 349-363). Springer.
  46. Syed, S., & Spruit, M. (2018, January). Selecting priors for latent Dirichlet allocation. In 2018 IEEE 12th International Conference on Semantic Computing (ICSC) (pp. 194-202). IEEE.
  47. Tudge, J., & Rogoff, B. (1999). Peer influences on cognitive development: Piagetian and Vygotskian perspectives. In P. Lloyd, & C. Fernyhough (Eds.), Lev Vygotsky: Critical assessments (pp. 32-56). Routledge.
  48. Dennen, V. (2005). From message posting to learning dialogues: Factors affecting learner participation in asynchronous discussion. Distance education, 26(1), 127-148. https://doi.org/10.1080/01587910500081376
  49. Wise, A., Speer, J., Marbouti, F., & Hsiao, Y. (2013). Broadening the notion of participation in online discussions: Examining patterns in learners' online listening behaviors. Instructional Science, 41(2), 323-343. https://doi.org/10.1007/s11251-012-9230-9
  50. Yilmaz, F., & Keser, H. (2016). The impact of reflective thinking activities in e-learning: A critical review of the empirical research. Computers & Education, 95, 163-173. https://doi.org/10.1016/j.compedu.2016.01.006
  51. Yoo, M. (2017). Development of Visualization Principles for Online Discussion Activities based on Learning Analytics, [Doctoral dissertation, Seoul National University].