KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

EAR: Enhanced Augmented Reality System for Sports Entertainment Applications

  • Mahmood, Zahid (Electrical and Computer Engineering from COMSATS Institute of Information Technology) ;
  • Ali, Tauseef (Faculty of Computer Science, Mathematics, and Engineering, University of Twente) ;
  • Muhammad, Nazeer (Department of Mathematics, COMSATS Institute of Information Technology) ;
  • Bibi, Nargis (Faculty of Computer Science, Fatima Jinnah Women University) ;
  • Shahzad, Imran (Electrical and Computer Engineering from COMSATS Institute of Information Technology) ;
  • Azmat, Shoaib (Electrical and Computer Engineering from COMSATS Institute of Information Technology)
  • Received : 2016.10.09
  • Accepted : 2017.09.03
  • Published : 2017.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Augmented Reality (AR) overlays virtual information on real world data, such as displaying useful information on videos/images of a scene. This paper presents an Enhanced AR (EAR) system that displays useful statistical players' information on captured images of a sports game. We focus on the situation where the input image is degraded by strong sunlight. Proposed EAR system consists of an image enhancement technique to improve the accuracy of subsequent player and face detection. The image enhancement is followed by player and face detection, face recognition, and players' statistics display. First, an algorithm based on multi-scale retinex is proposed for image enhancement. Then, to detect players' and faces', we use adaptive boosting and Haar features for feature extraction and classification. The player face recognition algorithm uses boosted linear discriminant analysis to select features and nearest neighbor classifier for classification. The system can be adjusted to work in different types of sports where the input is an image and the desired output is display of information nearby the recognized players. Simulations are carried out on 2096 different images that contain players in diverse conditions. Proposed EAR system demonstrates the great potential of computer vision based approaches to develop AR applications.

Keywords

References

  1. Bielli S, Harris C, "A mobile augmented reality system to enhance live sporting events," in Proc. of 6th Augmented Human International Conference ACM, USA, pp. 141-144, 2015.
  2. Yang M and Liao W, "Computer-assisted culture learning in an online augmented reality environment based on free-hand gesture interaction," IEEE Transactions on Learning Technologies, vol. 7, no. 2, pp. 107-117, 2014. https://doi.org/10.1109/TLT.2014.2307297
  3. Billinghurst M and Dunser A, "Augmented reality in the classroom," IEEE Comput., vol. 45, no. 7, pp. 56-63, 2012.
  4. Kuzmanovic et-al, "Hybrid broadcast broadband TV implementation in java based applications on digital TV devices," IEEE Trans. Consumer Electroics, vol. 58, no. 3, pp. 1056-1062, 2012. https://doi.org/10.1109/TCE.2012.6311356
  5. Kim et-al, "Design and implementation for interactive augmented broadcasting system," IEEE Transactions on broadcasting, vol. 60, no. 2, pp. 217-226, 2014. https://doi.org/10.1109/TBC.2013.2295478
  6. Cavallaro R, Hybinette, M, White, and T. Balch, "Augmenting live broadcast sports with 3-D tracking information," IEEE Multimedia, vol. 18, no. 4, pp. 38-47, 2011. https://doi.org/10.1109/MMUL.2011.61
  7. Mahmood Z, Ali T, and Khan S U, "Effects of pose and image resolution on automatic face recognition," IET Biometrics, Vol. 5, No. 2, pp. 111-119, 2016. https://doi.org/10.1049/iet-bmt.2015.0008
  8. Mahmood Z, Ali T, Khattak S, and Khan S U, "A comparative study of baseline algorithms of face recognition," in Proc. of 12th International Conference on Frontiers of Information Technology (FIT), Pakistan, pp. 263-268, 2014.
  9. Han J and Farin D, "A real-time augmented reality system for sports broadcast video enhancement," in Proc. of MM'07, Augsburg, Germany, pp. 1-4, 2007.
  10. Liang et-al. "Video2cartoon: generating 3-D cartoon from broadcast soccer video," in Proc. of ACM Multimedia, pp.217-218, 2005.
  11. Yu X, Yan X, Chi T, and Cheong L, "Inserting 3-D projected virtual Content into broadcast tennis Video," in Proc. of ACM Multimedia, pp.619-622, 2006.
  12. Matsui et-al, "Soccer image sequence computed by a virtual camera," in Proc. of CVPR, pp. 860-865, 1998.
  13. Hervas R, Bravo J, and Fontecha J, "An assistive navigation system based on augmented reality and context awareness for people with mild cognitive impairments," IEEE Journal Of Biomedical And Health Informatics, vol. 18, no. 1, pp. 368-374, 2014. https://doi.org/10.1109/JBHI.2013.2266480
  14. Cavallaro et-al. "Augmenting live broadcast sports with 3D tracking information," IEEE MultiMedia, pp. 38-47, 2011.
  15. Jungong H and Farin D, "A real-time augmented-reality system for sports broadcast video enhancement," in Proc. of 15th International Conference on Multimedia, ACM, 2007.
  16. Xiangzeng et-al. "Tennis ball tracking using a two-layered data association approach," IEEE Transactions on Multimedia, vol. 17, no. 2, pp. 145-156, 2015. https://doi.org/10.1109/TMM.2014.2380914
  17. Hanzra B and Rossi R, "Automatic cameraman for dynamic video acquisition of football match," in Proc. of 2nd International Conference on Image Information Processing (ICIIP), pp. 142-147, 2013.
  18. Liu et-al. "Automatic player detection, labeling and tracking in broadcast soccer video," Pattern Recognition Letters, pp. 103-113, 2009.
  19. Venkateshan, Kishore, A, Shekar, and Saha S, "Baseball hand tracking from monocular video," Advances in Computing, Communications and Informatics (ICACCI), pp. 953-961, 2013.
  20. Hu et-al. "Robust camera calibration and player tracking in broadcast basketball video," IEEE Transactions on Multimedia, vol. 13, no. 2, pp. 266-279, 2011. https://doi.org/10.1109/TMM.2010.2100373
  21. Mackowiak, et-al, "A complex system for football player detection in broadcasted video," in Proc. of International Conference on Signals and Electronic Systems (ICSES), pp. 119-122, 2010.
  22. Xing et-al, "Multiple player tracking in sports video: A dual-mode two-way Bayesian inference approach with progressive observation modeling," IEEE Transactions on Image Processin, pp. 1652-1667, 2011.
  23. Gross et-al, "Multi-pie", Image Vis. Comput, vol. 28, no. 5, pp. 807-813, 2010. https://doi.org/10.1016/j.imavis.2009.08.002
  24. Mahmood et-al, "Automatic player detection and identification for sports entertainment applications," Pattern Analysis and Applications, vol. 18, no. 4, pp. 971-982, 2015. https://doi.org/10.1007/s10044-014-0416-4
  25. Mahmood Z, Ali T, and Khattak S, "Automatic player detection and recognition in images using AdaBoost," in Proc. of 9th International Bhurban Conference on Applied Sciences and Technology (IBCAST), pp. 64-69, 2012.
  26. Setty S, Srinath N, and Hanumantharaju M, "Development of multiscale retinex algorithm for medical image enhancement based on multi-rate sampling," in Proc. of International Conference on Signal Processing, Image Processing and Pattern Recognition (ICSIPR), pp. 1-6, 2013.
  27. Poursaberi A, Svetlana N, and Gavrilova M, "Modified multiscale vesselness filter for facial feature detection," in Proc. of 4th International Conference on Emerging Security Technologies, pp. 21-24, 2013.
  28. Changyan X, Staring M, and Wang Y, "Multiscale Bi-Gaussian filter for adjacent curvilinear structures detection with application to vasculature images," IEEE Transactions on Image Processing, vol. 22, no. 1, pp. 174-188, 2013. https://doi.org/10.1109/TIP.2012.2216277
  29. Mahmood et al., "A color image enhancement technique using multiscale retinex," in Proc. of 11th International Conference on Frontiers of Information Technology (FIT), pp. 119-124, 2013.
  30. Viola P and Jones M, "Robust real-time face detection," Int. J. Comput. Vis., vol. 57, pp. 137-154, 2004. https://doi.org/10.1023/B:VISI.0000013087.49260.fb
  31. Kuo, Chorng S, Lin C, and Peng C, "Using AdaBoost method for face detection and pedestrian-flow evaluation of digital signage,"in Proc. of International Symposium on Computer, Consumer and Control (IS3C), pp. 90-93, 2014.
  32. Shylaja S, Balasubramanya K, and Natarajan S, "Dimensionality reduction techniques for face recognition," Department of Information Science and Engineering, PEC Institute of Technology.
  33. Turk M and Pentland A, "Eigenfaces for recognition," Journal of cognitive neuroscience, vol. 3, no. 1, pp. 71-86, 1991. https://doi.org/10.1162/jocn.1991.3.1.71
  34. Ahonen, Timo, Hadid A, and Pietikainen M, "Face description with local binary patterns: Application to face recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 12, pp. 2037-2041, 2006. https://doi.org/10.1109/TPAMI.2006.244
  35. Belhumeur P N, Hespanha J P, and Kriegman D, "Eigenfaces vs. fisherfaces: Recognition using class specific linear projection," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711-720, 1997. https://doi.org/10.1109/34.598228
  36. Lu J, Plataniotis K, Venetsanopoulos A, and Li S, "Ensemble-based discriminant learning with boosting for face recognition," IEEE Transactions on Neural Networks, vol. 17, no. 1, pp. 166- 178, 2006. https://doi.org/10.1109/TNN.2005.860853
  37. Mahmood et-al, "A parallel framework for object detection and recognition for secure vehicle parking," in Proc. of 17th International Conference on High Performance Computing and Communications (HPCC), USA, pp. 892-895, 2015.
  38. Gao Q B, and Wang Z Z, "Center-based nearest neighbor classifier," The Journal of the Pattern Recognition Society, vol. 40, pp. 346-349, 2007. https://doi.org/10.1016/j.patcog.2006.06.033
  39. Sun B, Tao W, and Chen W, "Luminance based MSR for color image enhancement," International Congress on Image and Signal Processing, pp. 358-362, 2008.
  40. Jiao Z and Xu B, "An image enhancement approach using retinex and YIQ," in Proc. of International Conference on Information Technology and Computer Science (ITCS), pp. 476-479, 2009.
  41. Ghimire D and Lee J, "Nonlinear transfer function-based local approach for color image enhancement," IEEE Transactions on Consumer Electronics, vol. 57, no. 2, pp. 858-865, 2011. https://doi.org/10.1109/TCE.2011.5955233
  42. Cun L, Yann, Bengio Y, and Hinton G, "Deep learning," Nature, 521.7553. pp. 436-444, 2015. https://doi.org/10.1038/nature14539
  43. Parkhi, Omkar M, Vedaldi A, and Zisserman A, "Deep Face Recognition," BMVC, vol. 1. no. 3, pp. 1-12, 2015.
  44. Abdi H and Williams L J, "Principal component analysis," Computational statistics, vol. 2, no. 4, pp. 433-459, 2010. https://doi.org/10.1002/wics.101
  45. Crane R, "Simplified approach to image processing: classical and modern techniques in C," Prentice Hall PTR, pp. 13-23, 1996.

Cited by

  1. Mobile cloud based-framework for sports applications vol.30, pp.4, 2019, https://doi.org/10.1007/s11045-019-00639-6