• Journal of Korea Multimedia Society
• /
• v.21 no.11
• /
• pp.1305-1316
• /
• 2018

Graphical user interface(GUI) has a very important role to interact with software users. However, designing and coding of GUI are tedious and pain taking processes. In many studies, the researchers are trying to convert GUI elements or widgets to code or describe formally their structures by help of domain knowledge of stochastic methods. In this paper, we propose the GUI elements detection approach based on object detection strategy using deep neural networks(DNN). Object detection with DNN is the approach that integrates localization and classification techniques. From the experimental result, if we selected the appropriate object detection model, the results can be used for automatic code generation from the sketch or capture images. The successful GUI elements detection can describe the objects as hierarchical structures of elements and transform their information to appropriate code by object description translator that will be studied at future.

• International Journal of Contents
• /
• v.15 no.4
• /
• pp.8-15
• /
• 2019

This paper presents a solution of the 'Quick, Draw! Doodle Recognition Challenge' hosted by Google. Doodles are drawings comprised of concrete representational meaning or abstract lines creatively expressed by individuals. In this challenge, a doodle is presented as a sequence of sketches. From the view of at the sketch level, to learn the pattern of strokes representing a doodle, we propose a sequential model stacked with multiple convolution layers and Long Short-Term Memory (LSTM) cells following the attention mechanism [15]. From the view at the image level, we use multiple models pre-trained on ImageNet to recognize the doodle. Finally, an ensemble and a post-processing method using the minimum cost flow algorithm are introduced to combine multiple models in achieving better results. In this challenge, our solutions garnered 11th place among 1,316 teams. Our performance was 0.95037 MAP@3, only 0.4% lower than the winner. It demonstrates that our method is very competitive. The source code for this competition is published at: https://github.com/ngxbac/Kaggle-QuickDraw.

• The Journal of the Institute of Internet, Broadcasting and Communication
• /
• v.20 no.3
• /
• pp.89-95
• /
• 2020

Recently, studies are being conducted to recognize a sketch image of a GUI (Graphical User Interface) based on a deep learning and to make it into a code implemented in an application. UI / UX designers can communicate with developers through storyboards when developing mobile applications. However, UI / UX designers can create different widgets for ambiguous widgets. In this paper, we propose an automatic UI detection method using symbol markers to improve the accuracy of DNN (Deep Neural Network) based UI identification. In order to evaluate the performance with or without the symbol markers, their accuracy is compared. In order to improve the accuracy according to of the symbol marker, the results are analyzed when the shape is a circle or a parenthesis. The use of symbol markers will reduce feedback between developer and designer, time and cost, and reduce sketch image UI false positives and improve accuracy.