Browse > Article
http://dx.doi.org/10.30693/SMJ.2021.10.4.80

YOLO based Optical Music Recognition and Virtual Reality Content Creation Method  

Oh, Kyeongmin (조선대학교 컴퓨터공학과)
Hong, Yoseop (조선대학교 컴퓨터공학과)
Baek, Geonyeong (조선대학교 컴퓨터공학과)
Chun, Chanjun (조선대학교 컴퓨터공학과)
Publication Information
Smart Media Journal / v.10, no.4, 2021 , pp. 80-90 More about this Journal
Abstract
Using optical music recognition technology based on deep learning, we propose to apply the results derived to VR games. To detect the music objects in the music sheet, the deep learning model used YOLO v5, and Hough transform was employed to detect undetected objects, modifying the size of the staff. It analyzes and uses BPM, maximum number of combos, and musical notes in VR games using output result files, and prevents the backlog of notes through Object Pooling technology for resource management. In this paper, VR games can be produced with music elements derived from optical music recognition technology to expand the utilization of optical music recognition along with providing VR contents.
Keywords
Deep Learning; Virtual Reality; Optical Music Recognition(OMR); Computer Vision;
Citations & Related Records
연도 인용수 순위
  • Reference
1 J. Calvo-Zaragoza and D. Rizo "End-to-end neural optical music recognition," Applied Sciences, vol. 8, no. 4, Apr., 2018.
2 J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: unified, real-time object detection," arXiv:1506.02640v5, May, 2016.
3 J. Redmon and A. Farhadi, "YOLO9000: Better, faster, stronger," arXiv:1612.08242v1, Dec., 2016.
4 J. Redmon and A. Farhadi, "YOLOv3: An incremental improvement," arXiv:1804.02767v1, Apr., 2018..
5 D. H. Pruslin, "Automatic recognition of sheet music", Sc.D. Dissertation, Massachusetts Institute of Technology, 1966.
6 R. Girshick, "Fast R-CNN," arXiv:1504.08083, Sep., 2015.
7 L. Tuggener, I. Elezi, J. Schmidhuber, M. Pelillo, and T. Stadelmann, "DeepScores -- A dataset for segmentation, detection and classification of tiny objects," arXiv:1804.00525v2, May, 2018.
8 W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, "SSD: Single shot multibox detector," arXiv:1512.02325v5, Dec., 2016.
9 S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNN: Towards real-time object detection with region proposal networks," arXiv:1506.01497v3, Jan., 2016.
10 J. Hajic. and P. Pecina, "In search of a dataset for handwritten optical music recognition: introducing MUSCIMA++," arXiv:1703.04824v1, Mar., 2017.
11 이승환, "VR 저널리즘의 발전과 미래," 한국애니메이션학회 - 애니메이션연구, 제13권, 제1호, pp. 43-65, 2017년 3월
12 배장은, "국내외 게임 산업 동향분석을 통한 가상 현실 기반의 기능성 게임 발전 방안," 디지털디자인학연구, 제14권, 제3호, pp. 737-748, 2014년 7월
13 정우정, "VR 시장 변화에 따른 VR 콘텐츠 문제점 연구," 2017 한국디자인학회 봄 국제학술대회논문집, pp. 200-201, 2017년 6월
14 이은석, "가상현실 콘텐츠를 위한 고도필드 렌더링 가속화," 한국정보과학회 2020 한국컴퓨터종합학술대회 논문집, pp. 1142-1144, 2020년 7월
15 양준석, "'오브젝트 풀링' 기법을 이용한 가비지콜렉터에 대한 최적화," 한국정보과학회 2016년 동계학술대회 논문집, pp. 1910-1912, 2016년 12월
16 김혜영, "효율적인 온라인 게임 서버를 위한 객체 풀링 기법에 관한 연구," 한국게임학회 논문지, 제9권, 제6호, pp. 163-170, 2009년 12월
17 J. Novotny and J. Pokorny, "Introduction to optical music recognition: Overview and practical challenges," in Proc. of 15th Annual International Workshop on Databases, Texts, Specifications, and Objects (DATESO), pp. 65-76, Apr., 2015.
18 J. Hajic., "Handwritten optical music recognition", Ph.D. Thesis Proposal, Charles University Institute of Formal and Applied Linguistics, 2017.