Realization of a Motion-based Interactive System Using Extraction of Real-time Search Terms

Abstract

The purpose of this research is to realize interactive art based on user's motions information using real time internet search terms. For this purpose, real-time search terms and related news information were extracted from three domestic and foreign portal sites, and the extracted information was used to generate content for interaction with the user. For interaction between the generated content and the user, a motion-based interactive technology that optimizes the intentions and experiences of the user was developed. A motion-based interactive system can be used to develop an immersive interface that induces user interest.

1. INTRODUCTION

Nowadays, as the web has become more influential, it has been exercising its influence on all sections of society such as politics, economy, society and culture. As the mobile-based search showed a rapid increase in addition to the PC-based search, web search is becoming routinized.

One of the characteristics of web search is a simple search pattern. At the beginning of the web, the search was the only service provided by the portal. At that time, companies are focused on improving the performance of its search engine. However, increasing the number of portals caused intense competition among them. Also the number of Internet user increased and Web-enabled technology was developed. As a result, type of service provided by the portal has changed gradually [1].

The representative service is a real-time search terms service being treated like basic features of the current portal site. Real-time search terms service is a service that relays the ranking search terms by the number of searching during certain hours of the day. This represents the changing information desires of Internet users in real time.

Real-time search terms are made to be determined by the number of searches entered into a search engine, do not use artificial intervention or adjustment by default [2].

Since query consisting of one or two terms is used to search for a suitable document out of massive document repository, ambiguity occurs and limits exist in accurately displaying the user needs. To resolve such problem, most of the domestic and foreign search portals have introduced methods for recommending search terms. Real-time search terms reflect topics a number of internet users are interested in based on the search terms being input the most at the current time. Accordingly, a number of researches on real-time search terms have been performed [3], [4].

Real-time search terms are exposed on top of the initial screen shown by the portal, and they unconsciously acquire interest of the users in advance. Accordingly, the content design using real-time search terms in this research holds values sufficient enough to be used as a factor for increasing immersion and maximizing participation of the user through visualizing search terms which induce interest of the modern people.

The purpose of this research is to collect and extract real-time search terms from the three domestic and foreign portal sites to build virtual contents, and to develop interactive user behavior-sensitized interface which uses the natural user interface(NUI) such as one’s motions for controlling the built digital contents.

We use information on motions of the user as information for making changes to contents. The recognition sensor was used to recognize, trace and analyze behavior of the user, and a research was performed on the technology for applying acquired information to the controlling of virtual contents. Interaction between contents and user can be processed through the use of non-contact motion recognition sensor based on natural motions of the user without help of any mobile device.

For efficient development of this research, researches on query as well as cases of interactive artworks using user’s motions will be simply examined in the Section 2, and the contents modeling and interactive methods proposed in this research will be observed in the Section 3. Methods for utilization in actual exhibitions will be analyzed in the Section 4, and the conclusion will be specified in final.

 

2. RELEVANT RESEARCHES

2.1 Real-time Search Terms

As internet indicated rapid development, the web started to take over the role of providing information of which TV and newspaper used to be in charge in the past, and the web has become the most popular media providing information in real life. The web has been receiving fervent response in that it infinitely produces articles dealing with the current issues in real-time, and in that it also reflects changes in the public interests in real-time. The most common method for users to acquire information on the internet is to acquire information by directly typing in preferred search terms on searching sites. However, a number of internet users steadily show interest in social issues or concerns of others that they were not aware of, in addition to their active way of acquiring information.

In this age of web, it is easy to find out what other people are interested in through the popular search term service. Since a number of people are searching on the internet, on what issue people are mostly interested in can be understood by finding out the search term people typed in the most. Accordingly, major searching sites and various portals provide the popular search term priority service.

Recently, the fact that foreign searching sites such as Google in addition to domestic portal sites such as Naver and Daum are competing to provide information such as ‘real-time popular search terms’ and ‘searching progress per search term’ signifies that displaying what other people are interested in and displaying what the hottest social issues are hold more important significance rather than just simply providing direct answers to the users’ questions. ‘Real-time popular search term’ along with the web 2.0 is the most representative example of collective intelligence being emphasized. An individual may assume current issues the public is interested in through such service, and may be provided with the information of the social issue with a single term [5].

Real-time search terms are determined through the keywords the users type into the searching engine. Of course, it has been pointed out for a few years that the real-time search terms listed in various portal sites are being partially manipulated to create issues or to advertise particular sites. However, the process of finding trends through search terms listed in searching sites is an interesting field of research.

In Korean PC/mobile search market, Naver scored mid- 70% market share and continues to hold its 1st position. Google and Daum followed by going back and forth from second and third positions. Naver features real-time search term priority service which analyzes the search terms typed in the most by the users in real-time and displays the analyzed search terms updated in every five seconds on the right side of the search result screen. The real-time search term priority service includes separate collection of popular search terms from diverse services to allow users to become aware of the current popular search terms per category such as knowledge iN, blog and news in addition to the combined search terms. The real-time search term priority service is known as ‘real-time suddenly increased search term’ in Naver, and as “real-time searches’ in Daum.

In Jan. 2016, Naver started a new service called "Lab data" to provide information about trends in the search. "Data Lab" is composed of 11 categories (entertainment, books, movies, theater, cars, games, single men, single women, housewives, students, youth).

DAUM classify the real-time search terms into four categories (Issue, news, sports, weather), and provides seven real-time search terms each category.

Google provides popular search term service through a separate site known as ‘Google Trends’. Popular search terms per nation can be confirmed in the ‘popular rapidly increased research terms’ menu in the Google Trends. However, for the case of ‘Google Trends’, the range of issues is excessively massive due to its global awareness, and its wide interval between updates makes it difficult to view popular search terms in real-time [6].

2.2 Interactive Art Using Body Motions

A number of interactive works using a camera to trace motions of the user and playing contents according to the previously set scenario are being presented. The following figures show three interactive artworks.

In ‘TextRain’ designed by Camil Utterback and Romy Achituv, the participants can use their body to play around with or pull out texts falling down like rain through the imaging device. The participants move around or stand still in front of a large screen and view themselves reflected on the screen like a mirror. On this screen, the texts fall down and land on the body of the user like rain or snow. The texts react to motions of the user, thereby being lifted up in the air or dropped on the ground [7].

Fig. 1.Interactive artwork ‘TextRain’

‘Bubbles’ co-designed by Wolfgang Muench and Kiyosh Furukawa. The air bubbles sensitively react to shadows of the user, and the gathered bubbles bounce and pop when the shadow touches the air bubbles [8].

Fig. 2.Interactive artwork ‘Bubbles’

Daniel C. Howe’s interactive artwork, ‘Text.Curtain’ explores relationships between poetic text and ludic play via an interactively evolving recombinant text. Projected on a wallsize screen, ‘Text.Curtain’ presents a physics-based 'springmass' interface that organically responds to the interactions of multiple simultaneous users. As the piece is disrupted and letters wash back and forth, a granular synthesis engine provides real-time aural feedback. Tension is created through the simultaneous desire of users to both disrupt the existing text via 'play' and to 'read' the piece as it evolves and recombines in response. As a user approaches the piece they are presented with fourteen lines. Depending on the version, either video motion-tracking (as below) or multiple track-balls (as above,) measure users' interactions, allowing disruption of the letters and lines via movement [9].

Fig. 3.Interactive artwork ‘Text.Curtain’

In this research, different from the static contents displayed in the works introduced above, dynamic contents being updated based on the reflection of concerns of the modern people through a connection with the internet was used to induce active interest of the user. The depth camera was used as a medium for interaction between built digital contents and user, to build motion-based interface in which the location of the contents being played according to joint motions of the user and the reaction occurring when hit change in real-time.

 

3. REALIZATION OF INTERACTIVE SYSTEM

3.1 System Overview

Under hypothesis that the real-time popular search terms have values as a factor reflecting concerns of the current public, we extracted real-time search terms from the following three domestic and foreign portal sites: ‘real-time rapidly increasing search terms’ from Naver, ‘real-time issue’ from Naver, and real-time search terms from ‘Google Trends’ of Google.

The following Fig. 4 shows the flow of the proposed system. The flow of the system can largely be divided into three stages. In the initial stage, the system extracts and collects lists of search terms from the web for indexing and saving. In the second stage, the depth camera is used to recognize skeleton information of the user. In the final stage, the image is controlled in real-time according to motions of the recognized user.

Fig. 4.Designing of a reaction model by using real-time search terms

3.2 Data Visualization

The system proposed in this study is composed of two tasks. One is to extract real-time search terms from the three main portal sites, the other is to visualize them. Extracting real-time search terms are used to generate list of real-time search terms after sorting.

As shown in Table 1, the number of collected real-time search terms in our system is 158 and the list of real-time search terms is periodically updated.

Table 1.The number of collected real-time search terms

The following Fig. 5 shows the process of extracting real-time queries from three portals.

Fig. 5.Real-time search term extraction flow chart

For this process, HTML of each portal consisting of real-time search terms needs to go through parsing in order to extract real-time search terms. For example, as shown in Fig. 6, the list of real-time search terms from Naver is surrounded by and tags. And Fig. 7 shows the list of real-time search terms from Daum.

Fig. 6.List of real-time search terms from Naver

Fig. 7.List of real-time search terms from Daum

After saving the extracted information in the database, information of the built database becomes visualized. In case the extracted information is long, it is necessary to divide long sentences through processing courses of morphological analysis and others. However, information extracted from the sites we selected as the experiment subjects consisted of short sentences, and additional processing courses were not required.

It is necessary to separately designate updating time for the built database information. Since updating time for the real-time search terms from each site varies, we set the updating time to 1 hour as a result of considering various conditions. Of course, such decision can be controversial since it was subjectively made based on statistics.

3.3 Motion Recognition Technology

Motion recognition method using a camera to extract a particular color from the inputted image and detects a spectator [10], [11]. This method has the disadvantage maven mostly on and a limit to the color for the detection light of the spectator. And a number of problems caused by the recognition sensitively changes depending on the surrounding environment, such as lighting. Therefore, it is important to set up a reasonably structured environment.

With the recent emergence of the motion recognition sensor can be more free from the influence of the tracking light [12], [13]. Motion Sensor such as Microsoft's Kinect have the advantage that high recognition rate by tracking human joint information.

In this research, a Kinect sensor from Microsoft was used to develop an interactive model using motions of the user. Kinect consists of infrared projector, RGB camera, depth recognition sensor and multi-array microphone, and it can be used for 3D motion capturing of overall human shape, for face recognition and for voice recognition. Kinect transmits and receives 30 frames of 640x480 dpi image information per second, and each frame contains 3D coordinates for 20 joints. Infrared projector of the Kinect emits infrared light in pixel level. Infrared camera collects reflection of the emitted infrared pixels. Collected information is then used along with information recognized through RGB camera as information on location and joint of the user [14].

The system becomes activated when the approach of the user is recognized within the viewing angle of Kinect. The Kinect sensor traces information of 20 joints of the user approaching within the range of visible distance to figure out the moving line, and the figured out information is used as information for controlling contents.

To recognize texts forming virtual contents touching the body of the user, we set a collision region list and named it ‘Bodyhat List’. For the utmost sensitivity in expressing vibration of texts, the size of collision region must be minimized. We set the size of collision region to 2-pixel in length, and to the maximum speed the texts can be dropped down at in width. Locations of the texts coming into the collision region are randomly changed to bring out the shaking effect. The following figure shows the collision region we set.

Fig. 8.Setting collision region by using skeleton information

 

4. UTILIZATION IN EXHIBITIONS

To utilize the motion-recognition-based image control system and digital contents designed in this research, we built a physical model of interactive system which involves beam project, Kinect sensor and sound system.

Fig. 9 shows a configuration of the system installed in exhibitions. We were installed on the wall facing the sensor to track the motion of the user, and our content was played by rear projection method.

Fig. 9.System installation diagram

Fig. 10 shows that a user interacts with his motion. In this figure we can see that the real-time search terms were shaking in the collision region is come down.

Fig. 10.Result of user interaction

We used Kinect for Windows SDK 1.7 and Processing (Processing 3.0.1) to realize proposed interactive contents.

 

5. CONCLUSION

In this research, real-time search terms and related news information were extracted from domestic and foreign portal sites, the extracted information was used for modeling virtual contents, and the interactive system based on such virtual contents and motions of the user was built. Virtual contents that we designed were able to grab attention of the user by visually expressing concerns of the modern people, and such designs acted as motives resulting in active interactions.

In addition, motion-based interactive technology in which intention and experience of the user are optimized showed its value as essential technology directly applicable to diverse digital content industries such as game, education and entertainment in addition to exhibition of culture and art.

In addition to real-time search terms, major search engines providing a service to recommend extended/related search terms also provided users with convenient search regarding query extension. Such service, for provision of effective information to the users, allows diverse comparative searches by providing terms having high relatedness to the initial query.

In the future, we have plans to use extended/related search term services provided by a number of search engines to manufacture contents considering differences in queries and interested fields preferred by each user. Such interactive contents reflecting changes in interested fields as time passes in addition to search terms typed in by the user are estimated to increase satisfaction of the users. In addition, we are looking forward to expanding the field of our study to the field of social search through constructing a social network based on possibility of establishing communities in the similar interested fields.