Design and Implementation of MongoDB-based Unstructured Log Processing System over Cloud Computing Environment (클라우드 환경에서 MongoDB 기반의 비정형 로그 처리 시스템 설계 및 구현)
-
- Journal of Internet Computing and Services
- /
- v.14 no.6
- /
- pp.71-84
- /
- 2013
Log data, which record the multitude of information created when operating computer systems, are utilized in many processes, from carrying out computer system inspection and process optimization to providing customized user optimization. In this paper, we propose a MongoDB-based unstructured log processing system in a cloud environment for processing the massive amount of log data of banks. Most of the log data generated during banking operations come from handling a client's business. Therefore, in order to gather, store, categorize, and analyze the log data generated while processing the client's business, a separate log data processing system needs to be established. However, the realization of flexible storage expansion functions for processing a massive amount of unstructured log data and executing a considerable number of functions to categorize and analyze the stored unstructured log data is difficult in existing computer environments. Thus, in this study, we use cloud computing technology to realize a cloud-based log data processing system for processing unstructured log data that are difficult to process using the existing computing infrastructure's analysis tools and management system. The proposed system uses the IaaS (Infrastructure as a Service) cloud environment to provide a flexible expansion of computing resources and includes the ability to flexibly expand resources such as storage space and memory under conditions such as extended storage or rapid increase in log data. Moreover, to overcome the processing limits of the existing analysis tool when a real-time analysis of the aggregated unstructured log data is required, the proposed system includes a Hadoop-based analysis module for quick and reliable parallel-distributed processing of the massive amount of log data. Furthermore, because the HDFS (Hadoop Distributed File System) stores data by generating copies of the block units of the aggregated log data, the proposed system offers automatic restore functions for the system to continually operate after it recovers from a malfunction. Finally, by establishing a distributed database using the NoSQL-based Mongo DB, the proposed system provides methods of effectively processing unstructured log data. Relational databases such as the MySQL databases have complex schemas that are inappropriate for processing unstructured log data. Further, strict schemas like those of relational databases cannot expand nodes in the case wherein the stored data are distributed to various nodes when the amount of data rapidly increases. NoSQL does not provide the complex computations that relational databases may provide but can easily expand the database through node dispersion when the amount of data increases rapidly; it is a non-relational database with an appropriate structure for processing unstructured data. The data models of the NoSQL are usually classified as Key-Value, column-oriented, and document-oriented types. Of these, the representative document-oriented data model, MongoDB, which has a free schema structure, is used in the proposed system. MongoDB is introduced to the proposed system because it makes it easy to process unstructured log data through a flexible schema structure, facilitates flexible node expansion when the amount of data is rapidly increasing, and provides an Auto-Sharding function that automatically expands storage. The proposed system is composed of a log collector module, a log graph generator module, a MongoDB module, a Hadoop-based analysis module, and a MySQL module. When the log data generated over the entire client business process of each bank are sent to the cloud server, the log collector module collects and classifies data according to the type of log data and distributes it to the MongoDB module and the MySQL module. The log graph generator module generates the results of the log analysis of the MongoDB module, Hadoop-based analysis module, and the MySQL module per analysis time and type of the aggregated log data, and provides them to the user through a web interface. Log data that require a real-time log data analysis are stored in the MySQL module and provided real-time by the log graph generator module. The aggregated log data per unit time are stored in the MongoDB module and plotted in a graph according to the user's various analysis conditions. The aggregated log data in the MongoDB module are parallel-distributed and processed by the Hadoop-based analysis module. A comparative evaluation is carried out against a log data processing system that uses only MySQL for inserting log data and estimating query performance; this evaluation proves the proposed system's superiority. Moreover, an optimal chunk size is confirmed through the log data insert performance evaluation of MongoDB for various chunk sizes.
Ancestral Burial Ground on the Inwangsan Mountain is a ten-panel folding screen with images and postscripts. Commissioned by Bak Gyeong-bin (dates unknown), this screen was painted by Jo Jung-muk (1820-after 1894) in 1868. The postscripts were written by Hong Seon-ju (dates unknown). The National Museum of Korea restored this painting, which had been housed in the museum on separate sheets, to its original folding screen format. The museum also opened the screen to the public for the first time at the special exhibition Through the Eyes of Joseon Painters: Real Scenery Landscapes of Korea held from July 23 to September 22, 2019. Ancestral Burial Ground on the Inwangsan Mountain depicts real scenery on the western slopes of Inwangsan Mountain spanning present-day Hongje-dong and Hongeun-dong in Seodaemun-gu, Seoul. In the distance, the Bukhansan Mountain ridges are illustrated. The painting also bears place names, including Inwangsan Mountain, Chumohyeon Hill, Hongjewon Inn, Samgaksan Mountain, Daenammun Gate, and Mireukdang Hall. The names and depictions of these places show similarities to those found on late Joseon maps. Jo Jung-muk is thought to have studied the geographical information marked on maps so as to illustrate a broad landscape in this painting. Field trips to the real scenery depicted in the painting have revealed that Jo exaggerated or omitted natural features and blended and arranged them into a row for the purposes of the horizontal picture plane. Jo Jung-muk was a painter proficient at drawing conventional landscapes in the style of the Southern School of Chinese painting. Details in Ancestral Burial Ground on the Inwangsan Mountain reflect the painting style of the School of Four Wangs. Jo also applied a more decorative style to some areas. The nineteenth-century court painters of the Dohwaseo(Royal Bureau of Painting), including Jo, employed such decorative painting styles by drawing houses based on painting manuals, applying dots formed like sprinkled black pepper to depict mounds of earth and illustrating flowers by dotted thick pigment. Moreover, Ancestral Burial Ground on the Inwangsan Mountain shows the individualistic style of Jeong Seon(1676~1759) in the rocks drawn with sweeping brushstrokes in dark ink, the massiveness of the mountain terrain, and the pine trees simply depicted using horizontal brushstrokes. Jo Jung-muk is presumed to have borrowed the authority and styles of Jeong Seon, who was well-known for his real scenery landscapes of Inwangsan Mountain. Nonetheless, the painting lacks an spontaneous sense of space and fails in conveying an impression of actual sites. Additionally, the excessively grand screen does not allow Jo Jung-muk to fully express his own style. In Ancestral Burial Ground on the Inwangsan Mountain, the texts of the postscripts nicely correspond to the images depicted. Their contents can be divided into six parts: (1) the occupant of the tomb and the reason for its relocation; (2) the location and geomancy of the tomb; (3) memorial services held at the tomb and mysterious responses received during the memorial services; (4) cooperation among villagers to manage the tomb; (5) the filial piety of Bak Gyeong-bin, who commissioned the painting and guarded the tomb; and (6) significance of the postscripts. The second part in particular is faithfully depicted in the painting since it can easily be visualized. According to the fifth part revealing the motive for the production of the painting, the commissioner Bak Gyeongbin was satisfied with the painting, stating that "it appears impeccable and is just as if the tomb were newly built." The composition of the natural features in a row as if explaining each one lacks painterly beauty, but it does succeed in providing information on the geomantic topography of the gravesite. A fair number of the existing depictions of gravesites are woodblock prints of family gravesites produced after the eighteenth century. Most of these are included in genealogical records and anthologies. According to sixteenth- and seventeenth-century historical records, hanging scrolls of family gravesites served as objects of worship. Bowing in front of these paintings was considered a substitute ritual when descendants could not physically be present to maintain their parents' or other ancestors' tombs. Han Hyo-won (1468-1534) and Jo Sil-gul (1591-1658) commissioned the production of family burial ground paintings and asked distinguished figures of the time to write a preface for the paintings, thus showing off their filial piety. Such examples are considered precedents for Ancestral Burial Ground on the Inwangsan Mountain. Hermitage of the Recluse Seokjeong in a private collection and Old Villa in Hwagae County at the National Museum of Korea are not paintings of family gravesites. However, they serve as references for seventeenth-century paintings depicting family gravesites in that they are hanging scrolls in the style of the paintings of literary gatherings and they illustrate geomancy. As an object of worship, Ancestral Burial Ground on the Inwangsan Mountain recalls a portrait. As indicated in the postscripts, the painting made Bak Gyeong-bin "feel like hearing his father's cough and seeing his attitudes and behaviors with my eyes." The fable of Xu Xiaosu, who gazed at the portrait of his father day and night, is reflected in this gravesite painting evoking a deceased parent. It is still unclear why Bak Gyeong-bin commissioned Ancestral Burial Ground on the Inwangsan Mountain to be produced as a real scenery landscape in the folding screen format rather than a hanging scroll or woodblock print, the conventional formats for a family gravesite paintings. In the nineteenth century, commoners came to produce numerous folding screens for use during the four rites of coming of age, marriage, burial, and ancestral rituals. However, they did not always use the screens in accordance with the nature of these rites. In the Ancestral Burial Ground on the Inwangsan Mountain, the real scenery landscape appears to have been emphasized more than the image of the gravesite in order to allow the screen to be applied during different rituals or for use to decorate space. The burial mound, which should be the essence of Ancestral Burial Ground on the Inwangsan Mountain, might have been obscured in order to hide its violation of the prohibition on the construction of tombs on the four mountains around the capital. At the western foot of Inwangsan Mountain, which was illustrated in this painting, the construction of tombs was forbidden. In 1832, a tomb discovered illegally built on the forbidden area was immediately dug up and the related people were severely punished. This indicates that the prohibition was effective until the mid-nineteenth century. The postscripts on the Ancestral Burial Ground on the Inwangsan Mountain document in detail Bak Gyeong-bin's efforts to obtain the land as a burial site. The help and connivance of villagers were necessary to use the burial site, probably because constructing tombs within the prohibited area was a burden on the family and villagers. Seokpajeong Pavilion by Yi Han-cheol (1808~1880), currently housed at the Los Angeles County Museum of Art, is another real scenery landscape in the format of a folding screen that is contemporaneous and comparable with Ancestral Burial Ground on the Inwangsan Mountain. In 1861 when Seokpajeong Pavilion was created, both Yi Han-cheol and Jo Jung-muk participated in the production of a portrait of King Cheoljong. Thus, it is highly probable that Jo Jung-muk may have observed the painting process of Yi's Seokpajeong Pavilion. A few years later, when Jo Jungmuk was commissioned to produce Ancestral Burial Ground on the Inwangsan Mountain, his experience with the impressive real scenery landscape of the Seokpajeong Pavilion screen could have been reflected in his work. The difference in the painting style between these two paintings is presumed to be a result of the tastes and purposes of the commissioners. Since Ancestral Burial Ground on the Inwangsan Mountain contains the multilayered structure of a real scenery landscape and family gravesite, it seems to have been perceived in myriad different ways depending on the viewer's level of knowledge, closeness to the commissioner, or viewing time. In the postscripts to the painting, the name and nickname of the tomb occupant as well as the place of his surname are not recorded. He is simply referred to as "Mister Bak." Biographical information about the commissioner Bak Gyeong-bin is also unavailable. However, given that his family did not enter government service, he is thought to have been a person of low standing who could not become a member of the ruling elite despite financial wherewithal. Moreover, it is hard to perceive Hong Seon-ju, who wrote the postscripts, as a member of the nobility. He might have been a low-level administrative official who belonged to the Gyeongajeon, as documented in the Seungjeongwon ilgi (Daily Records of Royal Secretariat of the Joseon Dynasty). Bak Gyeong-bin is presumed to have moved the tomb of his father to a propitious site and commissioned Ancestral Burial Ground on the Inwangsan Mountain to stress his filial piety, a conservative value, out of his desire to enter the upper class. However, Ancestral Burial Ground on the Inwangsan Mountain failed to live up to its original purpose and ended up as a contradictory image due to its multiple applications and the concern over the exposure of the violation of the prohibition on the construction of tombs on the prohibited area. Forty-seven years after its production, this screen became a part of the collection at the Royal Yi Household Museum with each panel being separated. This suggests that Bak Gyeong-bin's dream of bringing fortune and raising his family's social status by selecting a propitious gravesite did not come true.