Browse > Article
http://dx.doi.org/10.9717/kmms.2018.21.11.1305

Automatic Mobile Screen Translation Using Object Detection Approach Based on Deep Neural Networks  

Yun, Young-Sun (Dept. of Computer, Communications, and Unmanned Tech., Hannam University)
Park, Jisu (Dept. of Information and Communication Eng., Hannam University)
Jung, Jinman (Dept. of Computer, Communications, and Unmanned Tech., Hannam University)
Eun, Seongbae (Dept. of Computer, Communications, and Unmanned Tech., Hannam University)
Cha, Shin (Dept. of Computer, Communications, and Unmanned Tech., Hannam University)
So, Sun Sup (School of Computer Engineering, Kongju National University)
Publication Information
Journal of Korea Multimedia Society / v.21, no.11, 2018 , pp. 1305-1316 More about this Journal
Abstract
Graphical user interface(GUI) has a very important role to interact with software users. However, designing and coding of GUI are tedious and pain taking processes. In many studies, the researchers are trying to convert GUI elements or widgets to code or describe formally their structures by help of domain knowledge of stochastic methods. In this paper, we propose the GUI elements detection approach based on object detection strategy using deep neural networks(DNN). Object detection with DNN is the approach that integrates localization and classification techniques. From the experimental result, if we selected the appropriate object detection model, the results can be used for automatic code generation from the sketch or capture images. The successful GUI elements detection can describe the objects as hierarchical structures of elements and transform their information to appropriate code by object description translator that will be studied at future.
Keywords
Automatic Programming; Gui Elements Detection; Sketch Image to Code; Deep Neural Networks; Object Detection;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 Microsoft Windows, https://en.wikipedia.org/wiki/Microsoft_Windows (accessed Aug., 20, 2018).
2 Graphical user interface, https://en.wikipedia.org/wiki/Graphical_user_interface (accessed Aug., 20, 2018).
3 Macintosh, https://en.wikipedia.org/wiki/Macintosh (accessed Aug., 20, 2018).
4 Android developers, https://developer.android.com/ (accessed Aug., 22, 2018).
5 iOS12 Apple developer, https://developer.apple.com/ios/ (accessed Aug., 22, 2018).
6 Tizen delelopers, https://developer.tizen.org/ (accessed Aug., 22, 2018).
7 T. Parag, C. Bahlmann, V. Shet, and M. Singh, "A Grammar for Hierarchical Object Description in Logic Programs," Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 33-38, 2012.
8 T. Nguyen and C. Csallner, "Reverse Engineering Mobile Application User Interfaces with REMAUI (T)," Proceeding of IEEE/ACM International Conference on Automated Software Engineering, pp. 248-259. 2015.
9 H. Pham, T. Nguyen, P. Vu, and T. Nguyen, "Toward Mining Visual Log of Software," arXiv e-prints arXiv:1610.08911, 2016.
10 T. Beltramelli, "pix2code: Generating Code from a Graphical User Interface Screenshot," arXiv e-prints arXiv:1705.07962, 2017.
11 S. Hochreiter and J. Schmidhuber, "Long Short-term Memory," Neural Computation, Vol 9. No. 8. pp. 1735-1780, 1997.   DOI
12 Y. Jeong, I. Ansari, J. Shim, and J. Lee, "A Car Plate Area Detection System Using Deep Convolution Neural Network," Journal of Korea Multimedia Society, Vol. 20, No. 8, pp. 1166-1174, 2017.   DOI
13 J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You Only Look Once: Unified, Real-Time Object Detection," arXiv e-prints arXiv:1506.02640v4, 2016.
14 R. Cirshick, J. Donahue, T. Darrell, J. Malik, "Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation," arXiv e-prints arXiv:1311.2524v5, 2014.
15 R. Girshick, "Fast R-CNN," arXiv e-prints arXiv:1504.08083v2, 2015.
16 S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNNN: Towards Real-Time Object Detection with Region Proposal Networks," arXiv e-prints arXiv:1506.01497v3, 2016.
17 W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, et al., "SSD: Single Shot Multibox Detector," arXiv e-prints arXiv:1512.02325v5, 2016.
18 J. Redmon and A. Farhadi, "YOLO9000: Better, Faster, Stronger," arXiv e-prints arXiv:1612.02842, 2016.
19 J. Redmon and A. Farhadi, "YOLOv3: An Incremental Improvement," arXiv e-prints arXiv: 1804.02767, 2018.
20 S. Kim, J. Park, J. Jung, S. Eun, Y.-S. Yun, S. So, et al., “Identifying UI Widgets of Mobile Applications from Sketch Images,” Journal of Engineering and Applied Sciences, Vol. 13, No. 6, pp. 1561-1566, 2018.
21 Tzutalin, LabelImg, Git code (2015). https://github.com/tzutalin/labelImg (accessed Mar., 7, 2018).
22 J. Redmon, Darknet: Open Source Neural Networks in C, http://pjreddie.com/darknet/, 2013. (accessed Mar., 7, 2017).
23 Marvis, https://github.com/marvis/pytorchyolo2, 2017. (accessed Mar., 7, 2018).
24 Y.-S. Yun, https://github.com/andy-yun/pytorch-0.4-yolov3, 2018. (accessed Jun., 1, 2018).
25 Common Objects in Context, http://cocodataset.org/#home (accessed Aug., 22, 2018)