• Title/Summary/Keyword: Structured Data

Search Result 4,007, Processing Time 0.033 seconds

Job-related analysis and visualization using big data distributed processing system (빅데이터를 활용한 직업관련 분석 및 시각화)

  • Choi, Dong-Cheol;Choi, Nakjin;Kim, Min-Seok;Park, Jun-wook;Lee, Jun-Dong
    • Proceedings of the Korean Society of Computer Information Conference
    • /
    • 2020.07a
    • /
    • pp.249-251
    • /
    • 2020
  • 본 논문에서는 코로나바이러스감염증19 사태가 국내 취업시장에 어떠한 영향을 미쳤는지에 대해 알아보기 위하여 빅데이터를 활용한 직업 관련 분석 및 시각화를 수행하였다. 빅데이터를 위한 기본 자료는 통계청 자료와 워크넷 Open API를 활용하였으며, 빅데이터 처리 과정을 거쳐 결과값을 예측을 시도하였다. 2020년도 워크넷 Open API를 통해 고용수와 통계청 자료를 통해 비교 분석 및 시각화를 실시하였고, 08년~20년 취업자수를 통해 시계열 분석 및 예측을 진행해 앞으로의 횡보를 예상해보았다. 분석한 결과 19년, 20년도를 비교 분석했을 때에는 크게 차이가 나지 않았다. 추가적으로 시계열 분석기법을 활용해 보았을 때 매년 고용수는 전체적으로 증가하고 4월에는 감소, 7월에는 증가하는 추세가 나왔다. 코로나바이러스감염증19 사태로 인해 공공기관과 언택트 시대에 따른 화상회의나 재택근무로 인해 운수·통신 취업률은 상승한다는 결과값이 도출되었고, 자영업이나 서비스 직업 등은 다른 직종에 비해 큰 감소를 보여줬으나 국가 경제 활성화에 따른 고용수는 점차 증가할 것이라 예측된다.

  • PDF

Research of Knowledge Management and Reusability in Streaming Big Data with Privacy Policy through Actionable Analytics (스트리밍 빅데이터의 프라이버시 보호 동반 실용적 분석을 통한 지식 활용과 재사용 연구)

  • Paik, Juryon;Lee, Youngsook
    • Journal of Korea Society of Digital Industry and Information Management
    • /
    • v.12 no.3
    • /
    • pp.1-9
    • /
    • 2016
  • The current meaning of "Big Data" refers to all the techniques for value eduction and actionable analytics as well management tools. Particularly, with the advances of wireless sensor networks, they yield diverse patterns of digital records. The records are mostly semi-structured and unstructured data which are usually beyond of capabilities of the management tools. Such data are rapidly growing due to their complex data structures. The complex type effectively supports data exchangeability and heterogeneity and that is the main reason their volumes are getting bigger in the sensor networks. However, there are many errors and problems in applications because the managing solutions for the complex data model are rarely presented in current big data environments. To solve such problems and show our differentiation, we aim to provide the solution of actionable analytics and semantic reusability in the sensor web based streaming big data with new data structure, and to empower the competitiveness.

Domain Adaptation for Opinion Classification: A Self-Training Approach

  • Yu, Ning
    • Journal of Information Science Theory and Practice
    • /
    • v.1 no.1
    • /
    • pp.10-26
    • /
    • 2013
  • Domain transfer is a widely recognized problem for machine learning algorithms because models built upon one data domain generally do not perform well in another data domain. This is especially a challenge for tasks such as opinion classification, which often has to deal with insufficient quantities of labeled data. This study investigates the feasibility of self-training in dealing with the domain transfer problem in opinion classification via leveraging labeled data in non-target data domain(s) and unlabeled data in the target-domain. Specifically, self-training is evaluated for effectiveness in sparse data situations and feasibility for domain adaptation in opinion classification. Three types of Web content are tested: edited news articles, semi-structured movie reviews, and the informal and unstructured content of the blogosphere. Findings of this study suggest that, when there are limited labeled data, self-training is a promising approach for opinion classification, although the contributions vary across data domains. Significant improvement was demonstrated for the most challenging data domain-the blogosphere-when a domain transfer-based self-training strategy was implemented.

Encoding of XML Elements for Mining Association Rules

  • Hu Gongzhu;Liu Yan;Huang Qiong
    • The Journal of Information Systems
    • /
    • v.14 no.3
    • /
    • pp.37-47
    • /
    • 2005
  • Mining of association rules is to find associations among data items that appear together in some transactions or business activities. As of today, algorithms for association rule mining, as well as for other data mining tasks, are mostly applied to relational databases. As XML being adopted as the universal format for data storage and exchange, mining associations from XML data becomes an area of attention for researchers and developers. The challenge is that the semi-structured data format in XML is not directly suitable for traditional data mining algorithms and tools. In this paper we present an encoding method to encode XML tree-nodes. This method is used to store the XML data in Value Table and Transaction Table that can be easily accessed via indexing. The hierarchical relationship in the original XML tree structure is embedded in the encoding. We applied this method to association rules mining of XML data that may have missing data.

  • PDF

A Case Study on Data Educational Program for Non-major Trainees

  • Hyemi Um
    • Journal of Information Technology Applications and Management
    • /
    • v.30 no.5
    • /
    • pp.159-170
    • /
    • 2023
  • Due to technological advancements, the data industry is growing, leading to a demand for data professionals in the market. Job seekers interested in data-related positions include not only those with relevant majors but also non-majors. Therefore, this study aims to identify effective educational methods for non-majors lacking data knowledge and skills to develop both data and business competencies. This research focuses on 28 trainees who participated in the "Data Youth Campus" program conducted by K-institution. The program spanned 10 weeks and was structured into three phases: fundamentals, practical training, and projects, systematically enhancing trainees' capabilities. The effectiveness of the curriculum and trainee management was verified by measuring and analyzing improvement of competencies and satisfaction with the program. This study offers valuable insights for the design and implementation of data education programs tailored to non-majors.

Iterative integrated imputation for missing data and pathway models with applications to breast cancer subtypes

  • Linder, Henry;Zhang, Yuping
    • Communications for Statistical Applications and Methods
    • /
    • v.26 no.4
    • /
    • pp.411-430
    • /
    • 2019
  • Tumor development is driven by complex combinations of biological elements. Recent advances suggest that molecularly distinct subtypes of breast cancers may respond differently to pathway-targeted therapies. Thus, it is important to dissect pathway disturbances by integrating multiple molecular profiles, such as genetic, genomic and epigenomic data. However, missing data are often present in the -omic profiles of interest. Motivated by genomic data integration and imputation, we present a new statistical framework for pathway significance analysis. Specifically, we develop a new strategy for imputation of missing data in large-scale genomic studies, which adapts low-rank, structured matrix completion. Our iterative strategy enables us to impute missing data in complex configurations across multiple data platforms. In turn, we perform large-scale pathway analysis integrating gene expression, copy number, and methylation data. The advantages of the proposed statistical framework are demonstrated through simulations and real applications to breast cancer subtypes. We demonstrate superior power to identify pathway disturbances, compared with other imputation strategies. We also identify differential pathway activity across different breast tumor subtypes.

Selection and Allocation of Point Data with Wavelet Transform in Reverse Engineering (역공학에서 웨이브렛 변황을 이용한 점 데이터의 선택과 할당)

  • Ko, Tae-Jo;Kim, Hee-Sool
    • Journal of the Korean Society for Precision Engineering
    • /
    • v.17 no.9
    • /
    • pp.158-165
    • /
    • 2000
  • Reverse engineering is reproducing products by directly extracting geometric information from physical objects such as clay model wooden mock-up etc. The fundamental work in the reverse engineering is to acquire the geometric data for modeling the objects. This research proposes a novel method for data acquisition aiming at unmanned fast and precise measurement. This is come true by the sensor fusion with CCD camera using structured light beam and touch trigger sensor. The vision system provides global information of the objects data. In this case the number of data and position allocation for touch sensor is critical in terms of the productivity since the number of vision data is very huge. So we applied wavelet transform to reduce the number of data and to allocate the position of the touch probe. The simulated and experimental results show this method is good enough for data reduction.

  • PDF

The Study of Chronic Kidney Disease Classification using KHANES data (국민건강영양조사 자료를 이용한 만성신장질환 분류기법 연구)

  • Lee, Hong-Ki;Myoung, Sungmin
    • Proceedings of the Korean Society of Computer Information Conference
    • /
    • 2020.01a
    • /
    • pp.271-272
    • /
    • 2020
  • Data mining is known useful in medical area when no availability of evidence favoring a particular treatment option is found. Huge volume of structured/unstructured data is collected by the healthcare field in order to find unknown information or knowledge for effective diagnosis and clinical decision making. The data of 5,179 records considered for analysis has been collected from Korean National Health and Nutrition Examination Survey(KHANES) during 2-years. Data splitting, referred as the training and test sets, was applied to predict to fit the model. We analyzed to predict chronic kidney disease (CKD) using data mining method such as naive Bayes, logistic regression, CART and artificial neural network(ANN). This result present to select significant features and data mining techniques for the lifestyle factors related CKD.

  • PDF

A study on the policy of de-identifying unstructured data for the medical data industry (의료 데이터 산업을 위한 비정형 데이터 비식별화 정책에 관한 연구)

  • Sun-Jin Lee;Tae-Rim Park;So-Hui Kim;Young-Eun Oh;Il-Gu Lee
    • Convergence Security Journal
    • /
    • v.22 no.4
    • /
    • pp.85-97
    • /
    • 2022
  • With the development of big data technology, data is rapidly entering a hyperconnected intelligent society that accelerates innovative growth in all industries. The convergence industry, which holds and utilizes various high-quality data, is becoming a new growth engine, and big data is fused to various traditional industries. In particular, in the medical field, structured data such as electronic medical record data and unstructured medical data such as CT and MRI are used together to increase the accuracy of disease prediction and diagnosis. Currently, the importance and size of unstructured data are increasing day by day in the medical industry, but conventional data security technologies and policies are structured data-oriented, and considerations for the security and utilization of unstructured data are insufficient. In order for medical treatment using big data to be activated in the future, data diversity and security must be internalized and organically linked at the stage of data construction, distribution, and utilization. In this paper, the current status of domestic and foreign data security systems and technologies is analyzed. After that, it is proposed to add unstructured data-centered de-identification technology to the guidelines for unstructured data and technology application cases in the industry so that unstructured data can be actively used in the medical field, and to establish standards for judging personal information for unstructured data. Furthermore, an object feature-based identification ID that can be used for unstructured data without infringing on personal information is proposed.

What Roles should Population-based Cancer Registries be Playing in the 21st Century? Reflections on the Asian Cancer Registry Forum, Bangkok, February 2014

  • Roder, David
    • Asian Pacific Journal of Cancer Prevention
    • /
    • v.15 no.5
    • /
    • pp.1895-1896
    • /
    • 2014
  • Cancer registries have fundamental roles in cancer surveillance, research, and health services planning, monitoring and evaluation. Many are now assuming a broader role by contributing data for health-service management, alongside data inputs from other registries and administrative data sets. These data are being integrated into de-identified databases using privacy-protecting data linkage practices. Structured pathology reporting is increasing registry access to staging and other prognostic descriptors. Registry directions need to vary, depending on local need, barriers and opportunities. Flexibility and adaptability will be essential to optimize registry contributions to cancer control.