Journal of Korea Multimedia Society (한국멀티미디어학회논문지)

Korea Multimedia Society (한국멀티미디어학회)
권호 표지
DOI QR코드

DOI QR Code

Automatic Mobile Screen Translation Using Object Detection Approach Based on Deep Neural Networks

심층신경망 기반의 객체 검출 방식을 활용한 모바일 화면의 자동 프로그래밍에 관한 연구

  • Yun, Young-Sun (Dept. of Computer, Communications, and Unmanned Tech., Hannam University) ;
  • Park, Jisu (Dept. of Information and Communication Eng., Hannam University) ;
  • Jung, Jinman (Dept. of Computer, Communications, and Unmanned Tech., Hannam University) ;
  • Eun, Seongbae (Dept. of Computer, Communications, and Unmanned Tech., Hannam University) ;
  • Cha, Shin (Dept. of Computer, Communications, and Unmanned Tech., Hannam University) ;
  • So, Sun Sup (School of Computer Engineering, Kongju National University)
  • Received : 2018.08.30
  • Accepted : 2018.10.16
  • Published : 2018.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

Graphical user interface(GUI) has a very important role to interact with software users. However, designing and coding of GUI are tedious and pain taking processes. In many studies, the researchers are trying to convert GUI elements or widgets to code or describe formally their structures by help of domain knowledge of stochastic methods. In this paper, we propose the GUI elements detection approach based on object detection strategy using deep neural networks(DNN). Object detection with DNN is the approach that integrates localization and classification techniques. From the experimental result, if we selected the appropriate object detection model, the results can be used for automatic code generation from the sketch or capture images. The successful GUI elements detection can describe the objects as hierarchical structures of elements and transform their information to appropriate code by object description translator that will be studied at future.

Keywords

MTMDCW_2018_v21n11_1305_f0001.png 이미지

Fig. 1. Overall system and focused research module (boxed area) on this paper.

MTMDCW_2018_v21n11_1305_f0002.png 이미지

Fig. 2. Comparison of typical UI design screen on Android and iOS mobile environments.

MTMDCW_2018_v21n11_1305_f0003.png 이미지

Fig. 3. Annotation process for sketch image using LabelImg [21].

MTMDCW_2018_v21n11_1305_f0004.png 이미지

Fig. 4. Detected GUI elements for sketch image and captured edge image.

MTMDCW_2018_v21n11_1305_f0005.png 이미지

Fig, 5. Detected GUI elements for hand drawing (sketch) image.

Table 1. Comparison of related works

MTMDCW_2018_v21n11_1305_t0001.png 이미지

Table 2. Target GUI elements and corresponding dis-tribution and rankings in [20]

MTMDCW_2018_v21n11_1305_t0002.png 이미지

Table 3. Distribution of GUI elements

MTMDCW_2018_v21n11_1305_t0003.png 이미지

Table 4. Comparison of key layers on YOLOv2 and YOLOv3 models

MTMDCW_2018_v21n11_1305_t0004.png 이미지

Table 5. Comparison of Mean Average Precisions for YOLOv2 and YOLOv3

MTMDCW_2018_v21n11_1305_t0005.png 이미지

References

  1. Graphical user interface, https://en.wikipedia.org/wiki/Graphical_user_interface (accessed Aug., 20, 2018).
  2. Macintosh, https://en.wikipedia.org/wiki/Macintosh (accessed Aug., 20, 2018).
  3. Microsoft Windows, https://en.wikipedia.org/wiki/Microsoft_Windows (accessed Aug., 20, 2018).
  4. Android developers, https://developer.android.com/ (accessed Aug., 22, 2018).
  5. iOS12 Apple developer, https://developer.apple.com/ios/ (accessed Aug., 22, 2018).
  6. Tizen delelopers, https://developer.tizen.org/ (accessed Aug., 22, 2018).
  7. T. Parag, C. Bahlmann, V. Shet, and M. Singh, "A Grammar for Hierarchical Object Description in Logic Programs," Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 33-38, 2012.
  8. T. Nguyen and C. Csallner, "Reverse Engineering Mobile Application User Interfaces with REMAUI (T)," Proceeding of IEEE/ACM International Conference on Automated Software Engineering, pp. 248-259. 2015.
  9. H. Pham, T. Nguyen, P. Vu, and T. Nguyen, "Toward Mining Visual Log of Software," arXiv e-prints arXiv:1610.08911, 2016.
  10. T. Beltramelli, "pix2code: Generating Code from a Graphical User Interface Screenshot," arXiv e-prints arXiv:1705.07962, 2017.
  11. S. Hochreiter and J. Schmidhuber, "Long Short-term Memory," Neural Computation, Vol 9. No. 8. pp. 1735-1780, 1997. https://doi.org/10.1162/neco.1997.9.8.1735
  12. Y. Jeong, I. Ansari, J. Shim, and J. Lee, "A Car Plate Area Detection System Using Deep Convolution Neural Network," Journal of Korea Multimedia Society, Vol. 20, No. 8, pp. 1166-1174, 2017. https://doi.org/10.9717/KMMS.2017.20.8.1166
  13. R. Cirshick, J. Donahue, T. Darrell, J. Malik, "Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation," arXiv e-prints arXiv:1311.2524v5, 2014.
  14. R. Girshick, "Fast R-CNN," arXiv e-prints arXiv:1504.08083v2, 2015.
  15. S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNNN: Towards Real-Time Object Detection with Region Proposal Networks," arXiv e-prints arXiv:1506.01497v3, 2016.
  16. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You Only Look Once: Unified, Real-Time Object Detection," arXiv e-prints arXiv:1506.02640v4, 2016.
  17. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, et al., "SSD: Single Shot Multibox Detector," arXiv e-prints arXiv:1512.02325v5, 2016.
  18. J. Redmon and A. Farhadi, "YOLO9000: Better, Faster, Stronger," arXiv e-prints arXiv:1612.02842, 2016.
  19. J. Redmon and A. Farhadi, "YOLOv3: An Incremental Improvement," arXiv e-prints arXiv: 1804.02767, 2018.
  20. S. Kim, J. Park, J. Jung, S. Eun, Y.-S. Yun, S. So, et al., “Identifying UI Widgets of Mobile Applications from Sketch Images,” Journal of Engineering and Applied Sciences, Vol. 13, No. 6, pp. 1561-1566, 2018.
  21. Tzutalin, LabelImg, Git code (2015). https://github.com/tzutalin/labelImg (accessed Mar., 7, 2018).
  22. J. Redmon, Darknet: Open Source Neural Networks in C, http://pjreddie.com/darknet/, 2013. (accessed Mar., 7, 2017).
  23. Marvis, https://github.com/marvis/pytorchyolo2, 2017. (accessed Mar., 7, 2018).
  24. Y.-S. Yun, https://github.com/andy-yun/pytorch-0.4-yolov3, 2018. (accessed Jun., 1, 2018).
  25. Common Objects in Context, http://cocodataset.org/#home (accessed Aug., 22, 2018)

Cited by

  1. 효율적인 화면요소 배치에 관한 연구 vol.23, pp.2, 2018, https://doi.org/10.9717/kmms.2020.23.2.274