• Journal of IKEEE
• /
• v.22 no.3
• /
• pp.693-699
• /
• 2018

CNN based object recognitions have reported splendid results. However, the recognition of mobile apps raises an interesting challenge that recognition performance of similar widgets is not consistent. In order to improve the performance, we propose a noble method utilizing relations between input widgets. The recognition process flows from the Faster R-CNN based recognition to enhancement using a relation recognizer. The relations are represented as vector translation between objects in a relation space. Experiments on 323 apps show that our method significantly enhances the Faster R-CNN only approach.

• The Journal of the Institute of Internet, Broadcasting and Communication
• /
• v.20 no.3
• /
• pp.89-95
• /
• 2020

Recently, studies are being conducted to recognize a sketch image of a GUI (Graphical User Interface) based on a deep learning and to make it into a code implemented in an application. UI / UX designers can communicate with developers through storyboards when developing mobile applications. However, UI / UX designers can create different widgets for ambiguous widgets. In this paper, we propose an automatic UI detection method using symbol markers to improve the accuracy of DNN (Deep Neural Network) based UI identification. In order to evaluate the performance with or without the symbol markers, their accuracy is compared. In order to improve the accuracy according to of the symbol marker, the results are analyzed when the shape is a circle or a parenthesis. The use of symbol markers will reduce feedback between developer and designer, time and cost, and reduce sketch image UI false positives and improve accuracy.