• Title/Summary/Keyword: extracting methods

Search Result 947, Processing Time 0.025 seconds

A Study of The Medical Classics in the '$\bar{A}yurveda$' ('아유르베다'($\bar{A}yurveda$)의 의경(醫經)에 관한 연구)

  • Kim, Ki-Wook;Park, Hyun-Kuk;Seo, Ji-Young
    • Journal of Korean Medical classics
    • /
    • v.20 no.4
    • /
    • pp.91-117
    • /
    • 2007
  • Through a simple study of the medical classics in the '$\bar{A}yurveda$', we have summarized them as follows. 1) Traditional Indian medicine started in the Ganges river area at about 1500 B. C. E. and traces of medical science can be found in the "Rigveda" and "Atharvaveda". 2) The "Charaka" and "$Su\acute{s}hruta$(妙聞集)", ancient texts from India, are not the work of one person, but the result of the work and errors of different doctors and philosophers. Due to the lack of historical records, the time of Charaka or $Su\acute{s}hruta$(妙聞)s' lives are not exactly known. So the completion of the "Charaka" is estimated at 1st${\sim}$2nd century C. E. in northwestern India, and the "$Su\acute{s}hruta$" is estimated to have been completed in 3rd${\sim}$4th century C. E. in central India. Also, the "Charaka" contains details on internal medicine, while the "$Su\acute{s}hruta$" contains more details on surgery by comparison. 3) '$V\bar{a}gbhata$', one of the revered Vriddha Trayi(triad of the ancients, 三醫聖) of the '$\bar{A}yurveda$', lived and worked in about the 7th century and wrote the "$A\d{s}\d{t}\bar{a}nga$ $A\d{s}\d{t}\bar{a}nga$ $h\d{r}daya$ $sa\d{m}hit\bar{a}$ $samhit\bar{a}$(八支集)" and "$A\d{s}\d{t}\bar{a}nga$ Sangraha $samhit\bar{a}$(八心集)", where he tried to compromise and unify the "Charaka" and "$Su\acute{s}hruta$". The "$A\d{s}\d{t}\bar{a}nga$ Sangraha $samhit\bar{a}$" was translated into Tibetan and Arabic at about the 8th${\sim}$9th century, and if we generalize the medicinal plants recorded in each the "Charaka", "$Su\acute{s}hruta$" and the "$A\d{s}\d{t}\bar{a}nga$ Sangraha $samhit\bar{a}$", there are 240, 370, 240 types each. 4) The 'Madhava' focused on one of the subjects of Indian medicine, '$Nid\bar{a}na$' ie meaning "the cause of diseases(病因論)", and in one of the copies found by Bower in 4th century C. E. we can see that it uses prescriptions from the "BuHaLaJi(布哈拉集)", "Charaka", "$Su\acute{s}hruta$". 5) According to the "Charaka", there were 8 branches of ancient medicine in India : treatment of the body(kayacikitsa), special surgery(salakya), removal of alien substances(salyapahartka), treatment of poison or mis-combined medicines(visagaravairodhikaprasamana), the study of ghosts(bhutavidya), pediatrics(kaumarabhrtya), perennial youth and long life(rasayana), and the strengthening of the essence of the body(vajikarana). 6) The '$\bar{A}yurveda$', which originated from ancient experience, was recorded in Sanskrit, which was a theorization of knowledge, and also was written in verses to make memorizing easy, and made medicine the exclusive possession of the Brahmin. The first annotations were 1060 for the "Charaka", 1200 for the "$Su\acute{s}hruta$", 1150 for the "$A\d{s}\d{t}\bar{a}nga$ Sangraha $samhit\bar{a}$", and 1100 for the "$Nid\bar{a}na$", The use of various mineral medicines in the "Charaka" or the use of mercury as internal medicine in the "$A\d{s}\d{t}\bar{a}nga$ Sangraha $samhit\bar{a}$", and the palpation of the pulse for diagnosing in the '$\bar{A}yurveda$' and 'XiZhang(西藏)' medicine are similar to TCM's pulse diagnostics. The coexistence with Arabian 'Unani' medicine, compromise with western medicine and the reactionism trend restored the '$\bar{A}yurveda$' today. 7) The "Charaka" is a book inclined to internal medicine that investigates the origin of human disease which used the dualism of the 'Samkhya', the natural philosophy of the 'Vaisesika' and the logic of the 'Nyaya' in medical theories, and its structure has 16 syllables per line, 2 lines per poem and is recorded in poetry and prose. Also, the "Charaka" can be summarized into the introduction, cause, judgement, body, sensory organs, treatment, pharmaceuticals, and end, and can be seen as a work that strongly reflects the moral code of Brahmin and Aryans. 8) In extracting bloody pus, the "Charaka" introduces a 'sharp tool' bloodletting treatment, while the "$Su\scute{s}hruta$" introduces many surgical methods such as the use of gourd dippers, horns, sucking the blood with leeches. Also the "$Su\acute{s}hruta$" has 19 chapters specializing in ophthalmology, and shows 76 types of eye diseases and their treatments. 9) Since anatomy did not develop in Indian medicine, the inner structure of the human body was not well known. The only exception is 'GuXiangXue(骨相學)' which developed from 'Atharvaveda' times and the "$A\d{s}\d{t}\bar{a}nga$ Sangraha $samhit\bar{a}$". In the "$A\d{s}\d{t}\bar{a}nga$ Sangraha $samhit\bar{a}$"'s 'ShenTiLun(身體論)' there is a thorough listing of the development of a child from pregnancy to birth. The '$\bar{A}yurveda$' is not just an ancient traditional medical system but is being called alternative medicine in the west because of its ability to supplement western medicine and, as its effects are being proved scientifically it is gaining attention worldwide. We would like to say that what we have researched is just a small fragment and a limited view, and would like to correct and supplement any insufficient parts through more research of new records.

  • PDF

A Study of The Medical Classics in the '$\bar{A}yurveda$' (아유르베다'($\bar{A}yurveda$) 의경(醫經)에 관한 연구)

  • Kim, Kj-Wook;Park, Hyun-Kuk;Seo, Ji-Young
    • The Journal of Dong Guk Oriental Medicine
    • /
    • v.10
    • /
    • pp.119-145
    • /
    • 2008
  • Through a simple study of the medical classics in the '$\bar{A}yurveda$', we have summarized them as follows. 1) Traditional Indian medicine started in the Ganges river area at about 1500 B. C. E. and traces of medical science can be found in the "Rigveda" and "Atharvaveda". 2) The "Charaka(閣羅迦集)" and "$Su\acute{s}hruta$(妙聞集)", ancient texts from India, are not the work of one person, but the result of the work and errors of different doctors and philosophers. Due to the lack of historical records, the time of Charaka(閣羅迦) or $Su\acute{s}hruta$(妙聞)s' lives are not exactly known. So the completion of the "Charaka" is estimated at 1st$\sim$2nd century C. E. in northwestern India, and the "$Su\acute{s}hruta$" is estimated to have been completed in 3rd$\sim$4th century C. E. in central India. Also, the "Charaka" contains details on internal medicine, while the "$Su\acute{s}hruta$" contains more details on surgery by comparison. 3) '$V\bar{a}gbhata$', one of the revered Vriddha Trayi(triad of the ancients, 三醫聖) of the '$\bar{A}yurveda$', lived and worked in about the 7th century and wrote the "$Ast\bar{a}nga$ $Ast\bar{a}nga$ hrdaya $samhit\bar{a}$ $samhit\bar{a}$(八支集) and "$Ast\bar{a}nga$ Sangraha $samhit\bar{a}$(八心集)", where he tried to compromise and unify the "Charaka" and "$Su\acute{s}hruta$". The "$Ast\bar{a}nga$ Sangraha $samhit\bar{a}$" was translated into Tibetan and Arabic at about the 8th$\sim$9th century, and if we generalize the medicinal plants recorded in each the "Charaka", "$Su\acute{s}hruta$" and the "$Ast\bar{a}nga$ Sangraha $samhit\bar{a}$", there are 240, 370, 240 types each. 4) The 'Madhava' focused on one of the subjects of Indian medicine, '$Nid\bar{a}na$' ie meaning "the cause of diseases(病因論)", and in one of the copies found by Bower in 4th century C. E. we can see that it uses prescriptions from the "BuHaLaJi(布唅拉集)", "Charaka", "$Su\acute{s}hruta$". 5) According to the "Charaka", there were 8 branches of ancient medicine in India : treatment of the body(kayacikitsa), special surgery(salakya), removal of alien substances(salyapahartka), treatment of poison or mis-combined medicines(visagaravairodhikaprasamana), the study of ghosts(bhutavidya), pediatrics(kaumarabhrtya), perennial youth and long life(rasayana), and the strengthening of the essence of the body(vajikarana). 6) The '$\bar{A}yurveda$', which originated from ancient experience, was recorded in Sanskrit, which was a theorization of knowledge, and also was written in verses to make memorizing easy, and made medicine the exclusive possession of the Brahmin. The first annotations were 1060 for the "Charaka", 1200 for the "$Su\acute{s}hruta$", 1150 for the "$Ast\bar{a}nga$ Sangraha $samhit\bar{a}$", and 1100 for the "$Nid\bar{a}na$". The use of various mineral medicines in the "Charaka" or the use of mercury as internal medicine in the "$Ast\bar{a}nga$ Sangraha $samhit\bar{a}$", and the palpation of the pulse for diagnosing in the '$\bar{A}yurveda$' and 'XiZhang(西藏)' medicine are similar to TCM's pulse diagnostics. The coexistence with Arabian 'Unani' medicine, compromise with western medicine and the reactionism trend restored the '$\bar{A}yurveda$' today. 7) The "Charaka" is a book inclined to internal medicine that investigates the origin of human disease which used the dualism of the 'Samkhya', the natural philosophy of the 'Vaisesika' and the logic of the 'Nyaya' in medical theories, and its structure has 16 syllables per line, 2 lines per poem and is recorded in poetry and prose. Also, the "Charaka" can be summarized into the introduction, cause, judgement, body, sensory organs, treatment, pharmaceuticals, and end, and can be seen as a work that strongly reflects the moral code of Brahmin and Aryans. 8) In extracting bloody pus, the "Charaka" introduces a 'sharp tool' bloodletting treatment, while the "$Su\acute{s}hruta$" introduces many surgical methods such as the use of gourd dippers, horns, sucking the blood with leeches. Also the "$Su\acute{s}hruta$" has 19 chapters specializing in ophthalmology, and shows 76 types of eye diseases and their treatments. 9) Since anatomy did not develop in Indian medicine, the inner structure of the human body was not well known. The only exception is 'GuXiangXue(骨相學)' which developed from 'Atharvaveda' times and the "$Ast\bar{a}nga$ Sangraha $samhit\bar{a}$". In the "$Ast\bar{a}nga$ Sangraha $samhit\bar{a}$"'s 'ShenTiLun(身體論)' there is a thorough listing of the development of a child from pregnancy to birth. The '$\bar{A}yurveda$' is not just an ancient traditional medical system but is being called alternative medicine in the west because of its ability to supplement western medicine and, as its effects are being proved scientifically it is gaining attention worldwide. We would like to say that what we have researched is just a small fragment and a limited view, and would like to correct and supplement any insufficient parts through more research of new records.

  • PDF

Image Watermarking for Copyright Protection of Images on Shopping Mall (쇼핑몰 이미지 저작권보호를 위한 영상 워터마킹)

  • Bae, Kyoung-Yul
    • Journal of Intelligence and Information Systems
    • /
    • v.19 no.4
    • /
    • pp.147-157
    • /
    • 2013
  • With the advent of the digital environment that can be accessed anytime, anywhere with the introduction of high-speed network, the free distribution and use of digital content were made possible. Ironically this environment is raising a variety of copyright infringement, and product images used in the online shopping mall are pirated frequently. There are many controversial issues whether shopping mall images are creative works or not. According to Supreme Court's decision in 2001, to ad pictures taken with ham products is simply a clone of the appearance of objects to deliver nothing but the decision was not only creative expression. But for the photographer's losses recognized in the advertising photo shoot takes the typical cost was estimated damages. According to Seoul District Court precedents in 2003, if there are the photographer's personality and creativity in the selection of the subject, the composition of the set, the direction and amount of light control, set the angle of the camera, shutter speed, shutter chance, other shooting methods for capturing, developing and printing process, the works should be protected by copyright law by the Court's sentence. In order to receive copyright protection of the shopping mall images by the law, it is simply not to convey the status of the product, the photographer's personality and creativity can be recognized that it requires effort. Accordingly, the cost of making the mall image increases, and the necessity for copyright protection becomes higher. The product images of the online shopping mall have a very unique configuration unlike the general pictures such as portraits and landscape photos and, therefore, the general image watermarking technique can not satisfy the requirements of the image watermarking. Because background of product images commonly used in shopping malls is white or black, or gray scale (gradient) color, it is difficult to utilize the space to embed a watermark and the area is very sensitive even a slight change. In this paper, the characteristics of images used in shopping malls are analyzed and a watermarking technology which is suitable to the shopping mall images is proposed. The proposed image watermarking technology divide a product image into smaller blocks, and the corresponding blocks are transformed by DCT (Discrete Cosine Transform), and then the watermark information was inserted into images using quantization of DCT coefficients. Because uniform treatment of the DCT coefficients for quantization cause visual blocking artifacts, the proposed algorithm used weighted mask which quantizes finely the coefficients located block boundaries and coarsely the coefficients located center area of the block. This mask improves subjective visual quality as well as the objective quality of the images. In addition, in order to improve the safety of the algorithm, the blocks which is embedded the watermark are randomly selected and the turbo code is used to reduce the BER when extracting the watermark. The PSNR(Peak Signal to Noise Ratio) of the shopping mall image watermarked by the proposed algorithm is 40.7~48.5[dB] and BER(Bit Error Rate) after JPEG with QF = 70 is 0. This means the watermarked image is high quality and the algorithm is robust to JPEG compression that is used generally at the online shopping malls. Also, for 40% change in size and 40 degrees of rotation, the BER is 0. In general, the shopping malls are used compressed images with QF which is higher than 90. Because the pirated image is used to replicate from original image, the proposed algorithm can identify the copyright infringement in the most cases. As shown the experimental results, the proposed algorithm is suitable to the shopping mall images with simple background. However, the future study should be carried out to enhance the robustness of the proposed algorithm because the robustness loss is occurred after mask process.

KNU Korean Sentiment Lexicon: Bi-LSTM-based Method for Building a Korean Sentiment Lexicon (Bi-LSTM 기반의 한국어 감성사전 구축 방안)

  • Park, Sang-Min;Na, Chul-Won;Choi, Min-Seong;Lee, Da-Hee;On, Byung-Won
    • Journal of Intelligence and Information Systems
    • /
    • v.24 no.4
    • /
    • pp.219-240
    • /
    • 2018
  • Sentiment analysis, which is one of the text mining techniques, is a method for extracting subjective content embedded in text documents. Recently, the sentiment analysis methods have been widely used in many fields. As good examples, data-driven surveys are based on analyzing the subjectivity of text data posted by users and market researches are conducted by analyzing users' review posts to quantify users' reputation on a target product. The basic method of sentiment analysis is to use sentiment dictionary (or lexicon), a list of sentiment vocabularies with positive, neutral, or negative semantics. In general, the meaning of many sentiment words is likely to be different across domains. For example, a sentiment word, 'sad' indicates negative meaning in many fields but a movie. In order to perform accurate sentiment analysis, we need to build the sentiment dictionary for a given domain. However, such a method of building the sentiment lexicon is time-consuming and various sentiment vocabularies are not included without the use of general-purpose sentiment lexicon. In order to address this problem, several studies have been carried out to construct the sentiment lexicon suitable for a specific domain based on 'OPEN HANGUL' and 'SentiWordNet', which are general-purpose sentiment lexicons. However, OPEN HANGUL is no longer being serviced and SentiWordNet does not work well because of language difference in the process of converting Korean word into English word. There are restrictions on the use of such general-purpose sentiment lexicons as seed data for building the sentiment lexicon for a specific domain. In this article, we construct 'KNU Korean Sentiment Lexicon (KNU-KSL)', a new general-purpose Korean sentiment dictionary that is more advanced than existing general-purpose lexicons. The proposed dictionary, which is a list of domain-independent sentiment words such as 'thank you', 'worthy', and 'impressed', is built to quickly construct the sentiment dictionary for a target domain. Especially, it constructs sentiment vocabularies by analyzing the glosses contained in Standard Korean Language Dictionary (SKLD) by the following procedures: First, we propose a sentiment classification model based on Bidirectional Long Short-Term Memory (Bi-LSTM). Second, the proposed deep learning model automatically classifies each of glosses to either positive or negative meaning. Third, positive words and phrases are extracted from the glosses classified as positive meaning, while negative words and phrases are extracted from the glosses classified as negative meaning. Our experimental results show that the average accuracy of the proposed sentiment classification model is up to 89.45%. In addition, the sentiment dictionary is more extended using various external sources including SentiWordNet, SenticNet, Emotional Verbs, and Sentiment Lexicon 0603. Furthermore, we add sentiment information about frequently used coined words and emoticons that are used mainly on the Web. The KNU-KSL contains a total of 14,843 sentiment vocabularies, each of which is one of 1-grams, 2-grams, phrases, and sentence patterns. Unlike existing sentiment dictionaries, it is composed of words that are not affected by particular domains. The recent trend on sentiment analysis is to use deep learning technique without sentiment dictionaries. The importance of developing sentiment dictionaries is declined gradually. However, one of recent studies shows that the words in the sentiment dictionary can be used as features of deep learning models, resulting in the sentiment analysis performed with higher accuracy (Teng, Z., 2016). This result indicates that the sentiment dictionary is used not only for sentiment analysis but also as features of deep learning models for improving accuracy. The proposed dictionary can be used as a basic data for constructing the sentiment lexicon of a particular domain and as features of deep learning models. It is also useful to automatically and quickly build large training sets for deep learning models.

Query-based Answer Extraction using Korean Dependency Parsing (의존 구문 분석을 이용한 질의 기반 정답 추출)

  • Lee, Dokyoung;Kim, Mintae;Kim, Wooju
    • Journal of Intelligence and Information Systems
    • /
    • v.25 no.3
    • /
    • pp.161-177
    • /
    • 2019
  • In this paper, we study the performance improvement of the answer extraction in Question-Answering system by using sentence dependency parsing result. The Question-Answering (QA) system consists of query analysis, which is a method of analyzing the user's query, and answer extraction, which is a method to extract appropriate answers in the document. And various studies have been conducted on two methods. In order to improve the performance of answer extraction, it is necessary to accurately reflect the grammatical information of sentences. In Korean, because word order structure is free and omission of sentence components is frequent, dependency parsing is a good way to analyze Korean syntax. Therefore, in this study, we improved the performance of the answer extraction by adding the features generated by dependency parsing analysis to the inputs of the answer extraction model (Bidirectional LSTM-CRF). The process of generating the dependency graph embedding consists of the steps of generating the dependency graph from the dependency parsing result and learning the embedding of the graph. In this study, we compared the performance of the answer extraction model when inputting basic word features generated without the dependency parsing and the performance of the model when inputting the addition of the Eojeol tag feature and dependency graph embedding feature. Since dependency parsing is performed on a basic unit of an Eojeol, which is a component of sentences separated by a space, the tag information of the Eojeol can be obtained as a result of the dependency parsing. The Eojeol tag feature means the tag information of the Eojeol. The process of generating the dependency graph embedding consists of the steps of generating the dependency graph from the dependency parsing result and learning the embedding of the graph. From the dependency parsing result, a graph is generated from the Eojeol to the node, the dependency between the Eojeol to the edge, and the Eojeol tag to the node label. In this process, an undirected graph is generated or a directed graph is generated according to whether or not the dependency relation direction is considered. To obtain the embedding of the graph, we used Graph2Vec, which is a method of finding the embedding of the graph by the subgraphs constituting a graph. We can specify the maximum path length between nodes in the process of finding subgraphs of a graph. If the maximum path length between nodes is 1, graph embedding is generated only by direct dependency between Eojeol, and graph embedding is generated including indirect dependencies as the maximum path length between nodes becomes larger. In the experiment, the maximum path length between nodes is adjusted differently from 1 to 3 depending on whether direction of dependency is considered or not, and the performance of answer extraction is measured. Experimental results show that both Eojeol tag feature and dependency graph embedding feature improve the performance of answer extraction. In particular, considering the direction of the dependency relation and extracting the dependency graph generated with the maximum path length of 1 in the subgraph extraction process in Graph2Vec as the input of the model, the highest answer extraction performance was shown. As a result of these experiments, we concluded that it is better to take into account the direction of dependence and to consider only the direct connection rather than the indirect dependence between the words. The significance of this study is as follows. First, we improved the performance of answer extraction by adding features using dependency parsing results, taking into account the characteristics of Korean, which is free of word order structure and omission of sentence components. Second, we generated feature of dependency parsing result by learning - based graph embedding method without defining the pattern of dependency between Eojeol. Future research directions are as follows. In this study, the features generated as a result of the dependency parsing are applied only to the answer extraction model in order to grasp the meaning. However, in the future, if the performance is confirmed by applying the features to various natural language processing models such as sentiment analysis or name entity recognition, the validity of the features can be verified more accurately.

A Two-Stage Learning Method of CNN and K-means RGB Cluster for Sentiment Classification of Images (이미지 감성분류를 위한 CNN과 K-means RGB Cluster 이-단계 학습 방안)

  • Kim, Jeongtae;Park, Eunbi;Han, Kiwoong;Lee, Junghyun;Lee, Hong Joo
    • Journal of Intelligence and Information Systems
    • /
    • v.27 no.3
    • /
    • pp.139-156
    • /
    • 2021
  • The biggest reason for using a deep learning model in image classification is that it is possible to consider the relationship between each region by extracting each region's features from the overall information of the image. However, the CNN model may not be suitable for emotional image data without the image's regional features. To solve the difficulty of classifying emotion images, many researchers each year propose a CNN-based architecture suitable for emotion images. Studies on the relationship between color and human emotion were also conducted, and results were derived that different emotions are induced according to color. In studies using deep learning, there have been studies that apply color information to image subtraction classification. The case where the image's color information is additionally used than the case where the classification model is trained with only the image improves the accuracy of classifying image emotions. This study proposes two ways to increase the accuracy by incorporating the result value after the model classifies an image's emotion. Both methods improve accuracy by modifying the result value based on statistics using the color of the picture. When performing the test by finding the two-color combinations most distributed for all training data, the two-color combinations most distributed for each test data image were found. The result values were corrected according to the color combination distribution. This method weights the result value obtained after the model classifies an image's emotion by creating an expression based on the log function and the exponential function. Emotion6, classified into six emotions, and Artphoto classified into eight categories were used for the image data. Densenet169, Mnasnet, Resnet101, Resnet152, and Vgg19 architectures were used for the CNN model, and the performance evaluation was compared before and after applying the two-stage learning to the CNN model. Inspired by color psychology, which deals with the relationship between colors and emotions, when creating a model that classifies an image's sentiment, we studied how to improve accuracy by modifying the result values based on color. Sixteen colors were used: red, orange, yellow, green, blue, indigo, purple, turquoise, pink, magenta, brown, gray, silver, gold, white, and black. It has meaning. Using Scikit-learn's Clustering, the seven colors that are primarily distributed in the image are checked. Then, the RGB coordinate values of the colors from the image are compared with the RGB coordinate values of the 16 colors presented in the above data. That is, it was converted to the closest color. Suppose three or more color combinations are selected. In that case, too many color combinations occur, resulting in a problem in which the distribution is scattered, so a situation fewer influences the result value. Therefore, to solve this problem, two-color combinations were found and weighted to the model. Before training, the most distributed color combinations were found for all training data images. The distribution of color combinations for each class was stored in a Python dictionary format to be used during testing. During the test, the two-color combinations that are most distributed for each test data image are found. After that, we checked how the color combinations were distributed in the training data and corrected the result. We devised several equations to weight the result value from the model based on the extracted color as described above. The data set was randomly divided by 80:20, and the model was verified using 20% of the data as a test set. After splitting the remaining 80% of the data into five divisions to perform 5-fold cross-validation, the model was trained five times using different verification datasets. Finally, the performance was checked using the test dataset that was previously separated. Adam was used as the activation function, and the learning rate was set to 0.01. The training was performed as much as 20 epochs, and if the validation loss value did not decrease during five epochs of learning, the experiment was stopped. Early tapping was set to load the model with the best validation loss value. The classification accuracy was better when the extracted information using color properties was used together than the case using only the CNN architecture.

A Proposal of a Keyword Extraction System for Detecting Social Issues (사회문제 해결형 기술수요 발굴을 위한 키워드 추출 시스템 제안)

  • Jeong, Dami;Kim, Jaeseok;Kim, Gi-Nam;Heo, Jong-Uk;On, Byung-Won;Kang, Mijung
    • Journal of Intelligence and Information Systems
    • /
    • v.19 no.3
    • /
    • pp.1-23
    • /
    • 2013
  • To discover significant social issues such as unemployment, economy crisis, social welfare etc. that are urgent issues to be solved in a modern society, in the existing approach, researchers usually collect opinions from professional experts and scholars through either online or offline surveys. However, such a method does not seem to be effective from time to time. As usual, due to the problem of expense, a large number of survey replies are seldom gathered. In some cases, it is also hard to find out professional persons dealing with specific social issues. Thus, the sample set is often small and may have some bias. Furthermore, regarding a social issue, several experts may make totally different conclusions because each expert has his subjective point of view and different background. In this case, it is considerably hard to figure out what current social issues are and which social issues are really important. To surmount the shortcomings of the current approach, in this paper, we develop a prototype system that semi-automatically detects social issue keywords representing social issues and problems from about 1.3 million news articles issued by about 10 major domestic presses in Korea from June 2009 until July 2012. Our proposed system consists of (1) collecting and extracting texts from the collected news articles, (2) identifying only news articles related to social issues, (3) analyzing the lexical items of Korean sentences, (4) finding a set of topics regarding social keywords over time based on probabilistic topic modeling, (5) matching relevant paragraphs to a given topic, and (6) visualizing social keywords for easy understanding. In particular, we propose a novel matching algorithm relying on generative models. The goal of our proposed matching algorithm is to best match paragraphs to each topic. Technically, using a topic model such as Latent Dirichlet Allocation (LDA), we can obtain a set of topics, each of which has relevant terms and their probability values. In our problem, given a set of text documents (e.g., news articles), LDA shows a set of topic clusters, and then each topic cluster is labeled by human annotators, where each topic label stands for a social keyword. For example, suppose there is a topic (e.g., Topic1 = {(unemployment, 0.4), (layoff, 0.3), (business, 0.3)}) and then a human annotator labels "Unemployment Problem" on Topic1. In this example, it is non-trivial to understand what happened to the unemployment problem in our society. In other words, taking a look at only social keywords, we have no idea of the detailed events occurring in our society. To tackle this matter, we develop the matching algorithm that computes the probability value of a paragraph given a topic, relying on (i) topic terms and (ii) their probability values. For instance, given a set of text documents, we segment each text document to paragraphs. In the meantime, using LDA, we can extract a set of topics from the text documents. Based on our matching process, each paragraph is assigned to a topic, indicating that the paragraph best matches the topic. Finally, each topic has several best matched paragraphs. Furthermore, assuming there are a topic (e.g., Unemployment Problem) and the best matched paragraph (e.g., Up to 300 workers lost their jobs in XXX company at Seoul). In this case, we can grasp the detailed information of the social keyword such as "300 workers", "unemployment", "XXX company", and "Seoul". In addition, our system visualizes social keywords over time. Therefore, through our matching process and keyword visualization, most researchers will be able to detect social issues easily and quickly. Through this prototype system, we have detected various social issues appearing in our society and also showed effectiveness of our proposed methods according to our experimental results. Note that you can also use our proof-of-concept system in http://dslab.snu.ac.kr/demo.html.