Educational Technology International

The Korean Society for Education Technology (한국교육공학회)
권호 표지
DOI QR코드

DOI QR Code

Impacts of Badges and Leaderboards on Academic Performance: A Meta-Analysis

  • Received : 2022.08.11
  • Accepted : 2022.10.21
  • Published : 2022.10.30
Download PDF
⟨ Previous Next ⟩

Abstract

As technological changes continue to accelerate every day, meeting the needs of a shifting educational landscape requires leaving an exclusively "in-person" education behind. Gamified learning environments should be carefully designed in light of conflicting studies to suit students' needs. The purpose of this meta-analysis is to draw conclusive results regarding the application of the most commonly used game elements in education, i.e., badges and leaderboards, through a comprehensive analysis of their impact on academic performance in online learning. Review Manager (RevMan 5.4) was used to analyze eligible studies selected from Emerald, SAGE, ERIC, EBSCO, and ProQuest between January 2011 and January 2022. Analyzing 37 studies found that using leaderboards and badges in online education enhanced academic performance when compared to traditional learning without gamification (SMD = 0.39). The badge-only intervention showed a larger effect size (SMD = 0.33) than the leaderboard-only intervention (SMD = 0.27). Badges and leaderboards together exhibited a larger effect size (SMD = 0.48) than individual game elements (SMD = 0.40). The impact of the game elements on academic performance was greater in the humanities (SMD = 0.51) than in STEM fields (SMD = 0.32) and was greater for K-12 students (SMD = 0.63) than for college students (SMD = 0.31). This study contributes to a timely discussion of the use of badges and leaderboards in COVID-19 online learning trends and provides relevant data for designing integrations of online education and gamification models.

Keywords

References

  1. Ahmad, A., Zeshan, F., Khan, M., Marriam, R., Ali, A., & Samreen, A. (2020). The impact of gamification on learning outcomes of computer science majors. ACM Transactions on Computing Education, 20(2), 1-25.
  2. Andrade, F. R., Mizoguchi, R., & Isotani, S. (2016). The bright and dark sides of gamification. Intelligent Tutoring Systems, 9864, 176-186.
  3. Auvinen, T., Hakulinen, L., & Malmi, L. (2015). Increasing students' awareness of their behavior in online learning environments with visualizations and achievement badges. IEEE Transactions on Learning Technologies, 8(3), 261-273. https://doi.org/10.1109/TLT.2015.2441718
  4. Balci, S., Secaur, J. M., & Morris, B. J. (2022). Comparing the effectiveness of badges and leaderboards on academic performance and motivation of students in fully versus partially gamified online physics classes. Education and Information Technologies, 27(6), 8669-8704. https://doi.org/10.1007/s10639-022-10983-z
  5. Barata, G., Gama, S., Jorge, J., & Goncalves, D. (2014). Identifying student types in a gamified learning experience. International Journal of Game-Based Learning, 4(4), 19-36. https://doi.org/10.4018/ijgbl.2014100102
  6. Baydas, O., & Cicek, M. (2019). The examination of the gamification process in undergraduate education: A scale development study. Technology, Pedagogy and Education, 28(3), 269-285. https://doi.org/10.1080/1475939x.2019.1580609
  7. Bernik, A., Bubas, G., & Radosevic, D. (2015). A pilot study of the influence of gamification on the effectiveness of an e-Learning course. 26th Central European Conference on Information and Intelligent Systems (CECIIS 2015), 73-79.
  8. Cakiroglu, U., & Guler, M. (2021). Enhancing statistical literacy skills through real life activities enriched with gamification elements: An experimental study. E-Learning and Digital Media, 18(5), 441-459. https://doi.org/10.1177/2042753020987016
  9. Chans, G. M., & Portuguez Castro, M. (2021). Gamification as a strategy to increase motivation and engagement in higher education chemistry students. Computers, 10(10), 132. https://doi.org/10.3390/computers10100132
  10. Chen, C.-C., Huang, C., Gribbins, M., & Swan, K. (2018). Gamify online courses with tools built into your learning management system (LMS) to enhance self-determined and active learning. Online Learning, 22(3), 41-54.
  11. Christy, K. R., & Fox, J. (2014). Leaderboards in a virtual classroom: A test of stereotype threat and social comparison explanations for women's math performance. Computers & Education, 78, 66-77. https://doi.org/10.1016/j.compedu.2014.05.005
  12. Cohen, J. (2013). Statistical power analysis for the behavioral sciences. Taylor and Francis.
  13. Crabtree, J., & Rutland, A. (2001). Self-evaluation and social comparison amongst adolescents with learning difficulties. Journal of Community & Applied Social Psychology, 11(5), 347-359. https://doi.org/10.1002/casp.634
  14. Denny, P., McDonald, F., Empson, R., Kelly, P., & Petersen, A. (2018). Empirical support for a causal relationship between gamification and learning outcomes. Proceedings of the 2018 CHI conference on human factors in computing systems, 1-13.
  15. Dichev, C., & Dicheva, D. (2017). Gamifying education: What is known, what is believed and what remains uncertain: A critical review. International Journal of Educational Technology in Higher Education, 14(1), 9. https://doi.org/10.1186/s41239-017-0042-5
  16. Ding, L. (2019). Applying gamifications to asynchronous online discussions: A mixed methods study. Computers in Human Behavior, 91(February), 1-11. https://doi.org/10.1016/j.chb.2018.09.022
  17. Dominguez, A., Saenz-de-Navarrete, J., de-Marcos, L., Fernandez-Sanz, L., Pages, C., & Martinez-Herraiz, J.-J. (2013). Gamifying learning experiences: Practical implications and outcomes. Computers & Education, 63, 380-392. https://doi.org/10.1016/j.compedu.2012.12.020
  18. Fanfarelli, J. R., & McDaniel, R. (2019). Designing effective digital badges: Applications for learning. Routledge.
  19. Festinger, L. (1954). A theory of social comparison processes. Human Relations, 7(2), 117-140. https://doi.org/10.1177/001872675400700202
  20. Furdu, I., Tomozei, C., & Kose, U. (2017). Pros and cons gamification and gaming in classroom. Broad Research in Artificial Intelligence and Neuroscience, 8(2), 56-62.
  21. Ghasia, M., Machumu, H., & de Smet, E. (2019). Micro-credentials in higher education institutions: An exploratory study of its place in Tanzania. The International Journal of Education and Development Using Information and Communication Technology, 15(1), 219-230.
  22. Goosby, C. L. (2020). Using Digital Badges to Impact First Grade Math Achievement: A Quasi-experimental Study of Digital Badges and the North Carolina Early Numeracy Skills Indicators (Publication No. 28318393) [doctoral dissertation, Northcentral University] PQDT Open.
  23. Goudeau, S., Sanrey, C., Stanczak, A., Manstead, A., & Darnon, C. (2021). Why lockdown and distance learning during the COVID-19 pandemic are likely to increase the social class achievement gap. Nature Human Behaviour, 5(10), 1273-1281. https://doi.org/10.1038/s41562-021-01212-7
  24. Greitzer, F. L., Kuchar, O. A., & Huston, K. (2007). Cognitive science implications for enhancing training effectiveness in a serious gaming context. Journal on Educational Resources in Computing, 7(3), 2. https://doi.org/10.1145/1281320.1281322
  25. Gunduz, A. Y., & Akkoyunlu, B. (2020). Effectiveness of gamification in flipped learning. SAGE Open, 10(4), 215824402097983. https://doi.org/10.1177/2158244020979837
  26. Hakulinen, L., & Auvinen, T. (2014). The effect of gamification on students with different achievement goal orientations. Proceedings of the 2014 international conference on teaching and learning in computing and engineering, Kuching, Malaysia, 9-16.
  27. Hakulinen, L., Auvinen, T., & Korhonen, A. (2015). The effect of achievement badges on students' behavior: An empirical study in a university- level computer science course. International Journal of Emerging Technologies in Learning, 10(1), 18-30. https://doi.org/10.3991/ijet.v10i1.4221
  28. Hamari, J. (2017). Do badges increase user activity? A field experiment on the effects of gamification. Computers in Human Behavior, 71, 469-478. https://doi.org/10.1016/j.chb.2015.03.036
  29. Hanus, M. D., & Fox, J. (2015). Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance. Computers and Education, 80, 152-161. https://doi.org/10.1016/j.compedu.2014.08.019
  30. Heinert, S. W., Quasim, N., Ollmann, E., Socarras, M., & Suarez, N. (2021). Engaging youth through digital badges to promote health in underserved communities. Health promotion practice, 22(5), 631-637. https://doi.org/10.1177/1524839920934798
  31. Foon, H. K., & Ki, L. C. (2019). Using digital badges and leader-boards in primary school math lessons: Beneficial or merely new wine in old bottles?. In W. Ma, W. Chan & C. Cheng (Eds.), Shaping the future of education, communication and technology. Educational communications and technology yearbook (pp. 71-90). Springer.
  32. Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327(7414), 557-560. https://doi.org/10.1136/bmj.327.7414.557
  33. Homer, R., Hew, K. F., & Tan, C. Y. (2018). Comparing digital badges-and-points with classroom token systems: Effects on elementary school ESL students' classroom behavior and English learning. Journal of Educational Technology & Society, 21(1), 137-151.
  34. Huang, B., & Hew, K. F. (2015). Do points, badges and leaderboard increase learning and activity: A quasi-experiment on the effects of gamification. Proceedings of the 23rd International Conference on Computers in Education, China, 275-280.
  35. Huang, R., Ritzhaupt, A. D., Sommer, M., Zhu, J., Stephen, A., Valle, N., Hampton, J., & Li, J. (2020). The impact of gamification in educational settings on student learning outcomes: A meta-analysis. Educational Technology Research and Development, 68(4), 1875-1901. https://doi.org/10.1007/s11423-020-09807-z
  36. Jagust, T., Boticki, I., & So, H.-J. (2018). Examining competitive, collaborative and adaptive gamification in young learners' math learning. Computers & Education, 125, 444-457. https://doi.org/10.1016/j.compedu.2018.06.022
  37. Jensen, J. C., Lee, E. A., & Seshia, S. A. (2013). Virtualizing cyber-physical systems: Bringing CPS to online education. Proceedings of the First Workshop on CPS Education (CPS-Ed), 1-5.
  38. Kapsalis, G. D., Galani, A., & Tzafea, O. (2020). Kahoot! as a formative assessment tool in foreign language learning: A case study in greek as an L2. Theory and Practice in Language Studies, 10(11), 1343-1350. https://doi.org/10.17507/tpls.1011.01
  39. Kivunja, C. (2014). Theoretical perspectives of how digital natives learn. International Journal of Higher Education, 3(1), 94-109.
  40. Kim, J., & Castelli, D. M. (2021). Effects of gamification on behavioral change in education: A meta-analysis. International Journal of Environmental Research and Public Health, 18(7), 3550. https://doi.org/10.3390/ijerph18073550
  41. Kim, J. T., & Lee, W. H. (2015). Dynamical model for gamification of learning (DMGL). Multimedia Tools and Applications, 74(19), 8483-8493. https://doi.org/10.1007/s11042-013-1612-8
  42. Kim, J., & Kim, S. (2018). A comparative study on the differences in cultural attributes of gamification between Korea and Austria. Journal of the Korea Industrial Information Systems Research, 23(1), 109-122.
  43. Kusuma, G. P., Wigati, E. K., Utomo, Y., & Suryapranata, L. K. (2018). Analysis of gamification models in education using MDA framework. Procedia Computer Science, 135, 385-392. https://doi.org/10.1016/j.procs.2018.08.187
  44. Kyewski, E., & Kramer, N. C. (2018). To gamify or not to gamify? An experimental field study of the influence of badges on motivation, activity, and performance in an online learning course. Computers & Education, 118(1), 25-37. https://doi.org/10.1016/j.compedu.2017.11.006
  45. Lam, Y. W., Hew, K. F., & Chiu, K. F. (2018). Improving argumentative writing: Effects of a blended learning approach and gamification. Language learning & technology, 22(1), 97-118.
  46. Landers, R. N., Bauer, K. N., & Callan, R. C. (2017). Gamification of task performance with leaderboards: A goal setting experiment. Computers in Human Behavior, 71, 508-515. https://doi.org/10.1016/j.chb.2015.08.008
  47. Legaki, N. Z., Xi, N., Hamari, J., Karpouzis, K., & Assimakopoulos, V. (2020). The effect of challenge-based gamification on learning: An experiment in the context of statistics education. International Journal of Human-Computer Studies, 144, 102496. https://doi.org/10.1016/j.ijhcs.2020.102496
  48. Looyestyn, J., Kernot, J., Boshoff, K., Ryan, J., Edney, S., & Maher, C. (2017). Does gamification increase engagement with online programs? A systematic review. PloS One, 12(3), e0173403. https://doi.org/10.1371/journal.pone.0173403
  49. Mah, D. K. (2016). Learning analytics and digital badges: Potential impact on student retention in higher education. Technology, Knowledge and Learning, 21(3), 285-305. https://doi.org/10.1007/s10758-016-9286-8
  50. Manzano-Leon, A., Camacho-Lazarraga, P., Guerrero, M. A., Guerrero-Puerta, L., Aguilar-Parra, J. M., Trigueros, R., & Alias, A. (2021). Between level up and game over: A systematic literature review of gamification in education. Sustainability, 13(4), 2247. https://doi.org/10.3390/su13042247
  51. Margolis, H., & Mccabe, P. P. (2006). Improving self-efficacy and motivation: What to do, what to say. Intervention in School and Clinic, 41(4), 218-227. https://doi.org/10.1177/10534512060410040401
  52. Maslow, A. H. (1954). Motivation and Personality. Harper & Row Publishers.
  53. Mazarakis, A., & Brauer, P. (2022). Gamification is working, but which one exactly? Results from an experiment with four game design elements. International Journal of Human-Computer Interaction, 1-16.
  54. McGonigal, J. (2011). Reality is broken: Why games make us better and how they can change the world. Penguin Press.
  55. Mekler, E. D., Bruhlmann, F., Tuch, A. N., & Opwis, K. (2017). Towards understanding the effects of individual gamification elements on intrinsic motivation and performance. Computers in Human Behavior, 71, 525-534. https://doi.org/10.1016/j.chb.2015.08.048
  56. Morris, B. J., Dragovich, C., Todaro, R., Balci, S., & Dalton, E. (2019). Comparing badges and learning goals in low- and high-stakes learning contexts. Journal of Computing in Higher Education, 31(3), 573-603. https://doi.org/10.1007/s12528-019-09228-9
  57. Newby, T. J., & Cheng, Z. (2020). Instructional digital badges: Effective learning tools. Educational Technology Research and Development, 68(3), 1053-1067. https://doi.org/10.1007/s11423-019-09719-7
  58. Nieto-Escamez, F. A., & Roldan-Tapia, M. D. (2021). Gamification as online teaching strategy during COVID-19: A mini-review. Frontiers in Psychology, 12, 648552. https://doi.org/10.3389/fpsyg.2021.648552
  59. Ongoro, C. A., & Mwangoka, J. W. (2019). Effects of digital games on enhancing language learning in Tanzanian preschools. Knowledge Management & E-Learning, 11(3), 325-344.
  60. Ozdener, N. (2018). Gamification for enhancing Web 2.0 based educational activities: The case of pre-service grade school teachers using educational Wiki pages. Telematics and Informatics, 35(3), 564-578. https://doi.org/10.1016/j.tele.2017.04.003
  61. Poondej, C., & Lerdpornkulrat, T. (2019). Gamification in e-learning: A Moodle implementation and its effect on student engagement and performance. Interactive Technology and Smart Education, 17(1), 56-66. https://doi.org/10.1108/itse-06-2019-0030
  62. Pursell, D. P. (2009). Adapting to student learning styles: Engaging students with cell phone technology in organic chemistry instruction. Journal of Chemical Education, 86(10), 1219. https://doi.org/10.1021/ed086p1219
  63. Ritzhaupt, A. D., Huang, R., Sommer, M., Zhu, J., Stephen, A., Valle, N., Hampton, J., & Li, J. (2021). A meta-analysis on the influence of gamification in formal educational settings on affective and behavioral outcomes. Educational Technology Research and Development, 69(5), 2493-2522. https://doi.org/10.1007/s11423-021-10036-1
  64. Sailer, M., Hense, J. U., Mayr, S. K., & Mandl, H. (2017). How gamification motivates: An experimental study of the effects of specific game design elements on psychological need satisfaction. Computers in Human Behavior, 69, 371-380. https://doi.org/10.1016/j.chb.2016.12.033
  65. Sailer, M., & Homner, L. (2020). The gamification of learning: A meta-analysis. Educational Psychology Review, 32(1), 77-112. https://doi.org/10.1007/s10648-019-09498-w
  66. Sailer, M., & Sailer, M. (2021). Gamification of inclass activities in flipped classroom lectures. British Journal of Educational Technology, 52(1), 75-90. https://doi.org/10.1111/bjet.12948
  67. Saleem, A. N., Noori, N. M., & Ozdamli, F. (2021). Gamification applications in E-learning: A literature review. Technology, Knowledge and Learning, 27, 1-21.
  68. Sarkar, N., Ford, W., & Manzo, C. (2017). Engaging digital natives through social learning. Systemics, Cybernetics and Informatics, 15(2), 1-4.
  69. Schardt, C., Adams, M. B., Owens, T., Keitz, S., & Fontelo, P. (2007). Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC medical informatics and decision making, 7(1), 1-6. https://doi.org/10.1186/1472-6947-7-1
  70. Seaborn, K., & Fels, D. I. (2015). Gamification in theory and action: A survey. International Journal of Human-Computer Studies, 74, 14-31. https://doi.org/10.1016/j.ijhcs.2014.09.006
  71. Silpasuwanchai, C., Ma, X., Shigemasu, H., & Ren, X. (2016). Developing a comprehensive engagement framework of gamification for reflective learning. Proceedings of the 2016 ACM Conference on Designing Interactive Systems, 459-472.
  72. Stetson-Tiligadas, S. M. (2016). The impact of digital achievement badges on undergraduate learner motivation (Publication No. 10037460) [Doctoral dissertation, Capella University] PQDT Open.
  73. Strmecki, D., Bernik, A., & Radosevic, D. (2015). Gamification in E-learning: Introducing gamified design elements into E-learning Systems. Journal of Computer Science, 11(12), 1108-1117. https://doi.org/10.3844/jcssp.2015.1108.1117
  74. Tahir, F., Mitrovic, A., & Sotardi, V. (2022). Investigating the causal relationships between badges and learning outcomes in SQL-Tutor. Research and Practice in Technology Enhanced Learning, 17(1), 7. https://doi.org/10.1186/s41039-022-00180-4
  75. Teasley, S. D. (2017). Student facing dashboards: One size fits all?. Technology, Knowledge and Learning, 22(3), 377-384. https://doi.org/10.1007/s10758-017-9314-3
  76. Tsay, C. H.-H., Kofinas, A., & Luo, J. (2018). Enhancing student learning experience with technology-mediated gamification: An empirical study. Computers & Education, 121, 1-17. https://doi.org/10.1016/j.compedu.2018.01.009
  77. Turan, Z., Avinc, Z., Kara, K., & Goktas, Y. (2016). Gamification and education: Achievements, cognitive loads, and views of students. International journal of emerging technologies in learning, 11(7).
  78. Uanhoro, J., & Young, S. S. (2022). Investigation of the effect of badges in the online homework system for undergraduate general physics course. Education Sciences, 12(3), 217. https://doi.org/10.3390/educsci12030217
  79. Ugur-Erdogmus, F., & Cakir, R. (2022). Effect of gamified mobile applications and the role of player types on the achievement of students. Journal of Educational Computing Research, 60(4), 1063-1080. https://doi.org/10.1177/07356331211065679
  80. Uz Bilgin, C., & Gul, A. (2019). Investigating the effectiveness of gamification on group cohesion, attitude, and academic achievement in collaborative learning environments. TechTrends, 64(1), 124-136. https://doi.org/10.1007/s11528-019-00442-x
  81. Von Stumm, S. (2017). Socioeconomic status amplifies the achievement gap throughout compulsory education independent of intelligence. Intelligence, 60, 57-62. https://doi.org/10.1016/j.intell.2016.11.006
  82. Watson-Huggins, J. (2018). An experimental study on the effects of a gamified software intervention in mathematics achievement among sixth grade students (Publication No. 13820081) [Doctoral dissertation, Nova Southeastern University]. PQDT Open.
  83. Werbach, K., & Hunter, D. (2012). For the win: How game thinking can revolutionize your business. Wharton School Press.
  84. Xu, J., Lio, A., Dhaliwal, H., Andrei, S., Balakrishnan, S., Nagani, U., & Samadder, S. (2021). Psychological interventions of virtual gamification within academic intrinsic motivation: A systematic review. Journal of Affective Disorders, 293, 444-465. https://doi.org/10.1016/j.jad.2021.06.070
  85. Zainuddin, Z. (2018). Students' learning performance and perceived motivation in gamified flipped-class instruction. Computers & Education, 126, 75-88. https://doi.org/10.1016/j.compedu.2018.07.003
  86. Zhou, T., Hu, Z., Zhou, Z., Guo, X., & Sha, J. (2013). Genome-wide analysis of human hotspot intersected genes highlights the roles of meiotic recombination in evolution and disease. BMC Genomics, 14(1).
  87. Zimmerling, E., Hollig, C. E., Sandner, P. G., & Welpe, I. M. (2019). Exploring the influence of common game elements on ideation output and motivation. Journal of Business Research, 94, 302-312. https://doi.org/10.1016/j.jbusres.2018.02.030
  88. Zichermann, G., & Cunningham, C. (2011). Gamification by design: Implementing game mechanics in web and mobile apps. O'Reilly Media, Inc.