• Title/Summary/Keyword: Large Scale Data

Search Result 2,796, Processing Time 0.046 seconds

Adaptive Priority Queue-driven Task Scheduling for Sensor Data Processing in IoT Environments (사물인터넷 환경에서 센서데이터의 처리를 위한 적응형 우선순위 큐 기반의 작업 스케줄링)

  • Lee, Mijin;Lee, Jong Sik;Han, Young Shin
    • Journal of Korea Multimedia Society
    • /
    • v.20 no.9
    • /
    • pp.1559-1566
    • /
    • 2017
  • Recently in the IoT(Internet of Things) environment, a data collection in real-time through device's sensor has increased with an emergence of various devices. Collected data from IoT environment shows a large scale, non-uniform generation cycle and atypical. For this reason, the distributed processing technique is required to analyze the IoT sensor data. However if you do not consider the optimal scheduling for data and the processor of IoT in a distributed processing environment complexity increase the amount in assigning a task, the user is difficult to guarantee the QoS(Quality of Service) for the sensor data. In this paper, we propose APQTA(Adaptive Priority Queue-driven Task Allocation method for sensor data processing) to efficiently process the sensor data generated by the IoT environment. APQTA is to separate the data into job and by applying the priority allocation scheduling based on the deadline to ensure that guarantee the QoS at the same time increasing the efficiency of the data processing.

Development of a Converter for Visualizing SEDRIS (SEDRIS 합성 환경 데이터 가시화를 위한 변환기 개발)

  • Kang, Yuna;Kim, Hyungki;Han, Soonhung;Kim, Man Kyu
    • Korean Journal of Computational Design and Engineering
    • /
    • v.18 no.3
    • /
    • pp.189-199
    • /
    • 2013
  • The need for reusing synthetic environment data that are employed in the field of modeling and simulation has recently been rising. SEDRIS (Synthetic Environment Data Representation & Interchange Specification) is a standard to exchange synthetic environment data, and is the specification utilized in various military simulations of the Pentagon for representing and exchanging 3D data. SEDRIS represents environmental areas based on a data model; it can represent wind speed, wind directions, weather changes, the information of buildings, as well as terrain data. In some situations, however, the synthetic environment data stored in SEDRIS format should be converted to various visualization formats. First, because SEDRIS is a form of a super-set, it is necessary to verify whether large scale SEDRIS files are stored successfully through visualization. Second, the synthetic environment data should be visualized in some visualization programs for the simulation results to provide an immersive and realistic sense. In this study, we have developed converters for converting SEDRIS data to various visualization formats and visualized the converted results.

Standard-based Integration of Heterogeneous Large-scale DNA Microarray Data for Improving Reusability

  • Jung, Yong;Seo, Hwa-Jeong;Park, Yu-Rang;Kim, Ji-Hun;Bien, Sang Jay;Kim, Ju-Han
    • Genomics & Informatics
    • /
    • v.9 no.1
    • /
    • pp.19-27
    • /
    • 2011
  • Gene Expression Omnibus (GEO) has kept the largest amount of gene-expression microarray data that have grown exponentially. Microarray data in GEO have been generated in many different formats and often lack standardized annotation and documentation. It is hard to know if preprocessing has been applied to a dataset or not and in what way. Standard-based integration of heterogeneous data formats and metadata is necessary for comprehensive data query, analysis and mining. We attempted to integrate the heterogeneous microarray data in GEO based on Minimum Information About a Microarray Experiment (MIAME) standard. We unified the data fields of GEO Data table and mapped the attributes of GEO metadata into MIAME elements. We also discriminated non-preprocessed raw datasets from others and processed ones by using a two-step classification method. Most of the procedures were developed as semi-automated algorithms with some degree of text mining techniques. We localized 2,967 Platforms, 4,867 Series and 103,590 Samples with covering 279 organisms, integrated them into a standard-based relational schema and developed a comprehensive query interface to extract. Our tool, GEOQuest is available at http://www.snubi.org/software/GEOQuest/.

Data Analysis and Design Method for automatically generating Office Data of Switching System (교환 시스템의 국 데이터 자동 생성을 위한 데이터 분석 및 설계 방법)

  • Chung, Chang-Shin;Jung, Soon-Key
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.28 no.4B
    • /
    • pp.316-322
    • /
    • 2003
  • The office data for telecommunication switching systems included system configuration data, processing data, maintenance and administration data on a very large scale. Those data are dependent on functions of the system and the place of system installation. The effect of errors of office data is very serious. In order to reduce time and effort on the system development phase and to enhance system reliability, in this paper we proposed a data analysis and design method for automatically generating office data that are dependent on installation capability and system configuration of the swiching office.

Land cover classification based on the phonology of Korea using NOAA-AVHRR

  • Kim, Won-Joo;Nam, Ki-Deock;Park, Chong-Hwa
    • Proceedings of the KSRS Conference
    • /
    • 1999.11a
    • /
    • pp.439-442
    • /
    • 1999
  • It is important to analyze the seasonal change profiles of land cover type in large scale for establishing preservation strategy and environmental monitoring. Because the NOAA-AVHRR data sets provide global data with high temporal resolution, it is suitable for the land cover classification of the large area. The objectives of this study were to classify land cover of Korea, to investigate the phenological profiles of land cover. The NOAA-AVHRR data from Jan. 1998 to Dec. 1998 were received by Korea Ocean Research & Development Institute(KORDI) and were used for this study. The NDVI data were produced from this data. And monthly maximum value composite data were made for reducing cloud effect and temporal classification. And the data were classified using the method of supervised classification. To label the land cover classes, they were classified again using generalized vegetation map and Landsat-TM classified image. And the profiles of each class was analyzed according to each month. Results of this study can be summarized as follows. First, it was verified that the use of vegetation map and TM classified map was available to obtain the temporal class labeling with NOAA-AVHRR. Second, phenological characteristics of plant communities of Korea using NOAA-AVHRR was identified. Third, NDVI of North Korea is lower on Summer than that of South Korea. And finally, Forest cover is higher than another cover types. Broadleaf forest is highest on may. Outline of covertype profiles was investigated.

  • PDF

Temperature Data Visualization for Condition Monitoring based on Wireless Sensor Network (무선 센서 네트워크 기반의 상태 모니터링을 위한 온도 데이터 시각화)

  • Seo, Jung-Hee
    • The Journal of the Korea institute of electronic communication sciences
    • /
    • v.15 no.2
    • /
    • pp.245-252
    • /
    • 2020
  • Unexpected equipment defects can cause a huge economic losses in the society at large. Although condition monitoring can provide solutions, the signal processing algorithms must be developed to predict mechanical failures using data acquired from various sensors attached to the equipment. The signal processing algorithms used in a condition monitoring requires high computing efficiency and resolution. To improve condition monitoring on a wireless sensor network(WSN), data visualization can maximize the expressions of the data characteristics. Thus, this paper proposes the extraction of visual feature from temperature data over time using condition monitoring based on a WSN to identify environmental conditions of equipment in a large-scale infrastructure. Our results show that time-frequency analysis can visually track temperature changes over time and extract the characteristics of temperature data changes.

A Comparative Study on Discrimination Issues in Large Language Models (거대언어모델의 차별문제 비교 연구)

  • Wei Li;Kyunghwa Hwang;Jiae Choi;Ohbyung Kwon
    • Journal of Intelligence and Information Systems
    • /
    • v.29 no.3
    • /
    • pp.125-144
    • /
    • 2023
  • Recently, the use of Large Language Models (LLMs) such as ChatGPT has been increasing in various fields such as interactive commerce and mobile financial services. However, LMMs, which are mainly created by learning existing documents, can also learn various human biases inherent in documents. Nevertheless, there have been few comparative studies on the aspects of bias and discrimination in LLMs. The purpose of this study is to examine the existence and extent of nine types of discrimination (Age, Disability status, Gender identity, Nationality, Physical appearance, Race ethnicity, Religion, Socio-economic status, Sexual orientation) in LLMs and suggest ways to improve them. For this purpose, we utilized BBQ (Bias Benchmark for QA), a tool for identifying discrimination, to compare three large-scale language models including ChatGPT, GPT-3, and Bing Chat. As a result of the evaluation, a large number of discriminatory responses were observed in the mega-language models, and the patterns differed depending on the mega-language model. In particular, problems were exposed in elder discrimination and disability discrimination, which are not traditional AI ethics issues such as sexism, racism, and economic inequality, and a new perspective on AI ethics was found. Based on the results of the comparison, this paper describes how to improve and develop large-scale language models in the future.

The MS Card Data Transfer System using Bluetooth Protocol (블루투스를 이용한 마그네틱 카드 정보 전송 시스템)

  • 강형원;김영길
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2003.05a
    • /
    • pp.435-438
    • /
    • 2003
  • The MS card data transfer system using blue-tooth protocol ran communicate the MS card data wirelessly and does not take an extra communication expense which is a weakness point of existing wireless communication system. This Blue-tooth system, which has excellent security and no extra communication expense, can efficiently communicate data of the place ,where can be solved with small scale wireless network, such as the theme-park or gasoline-station. Existing wireless communication system compose network using wireless-LAN protocol which has extra communication expense, or with RF protocol which has poor security. But this system suitable for LAN because it has not extra communication expense and it has excellent security cause frequency-hopping of Blue-tooth protocol. The MS card data transfer system using blue-tooth protocol has low power, high performance RISC processor and large scale 16-gray graphic LCD which is suitable for portable unit. The MS card data transfer system can efficiently control depot for a long time because it has low power, excellent security and no extra communication expense.

  • PDF

Modeling, Discovering, and Visualizing Workflow Performer-Role Affiliation Networking Knowledge

  • Kim, Haksung;Ahn, Hyun;Kim, Kwanghoon Pio
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.8 no.2
    • /
    • pp.691-708
    • /
    • 2014
  • This paper formalizes a special type of social networking knowledge, which is called "workflow performer-role affiliation networking knowledge." A workflow model specifies execution sequences of the associated activities and their affiliated relationships with roles, performers, invoked-applications, and relevant data. In Particular, these affiliated relationships exhibit a stream of organizational work-sharing knowledge and utilize business process intelligence to explore resources allotting and planning knowledge concealed in the corresponding workflow model. In this paper, we particularly focus on the performer-role affiliation relationships and their implications as organizational and business process intelligence in workflow-driven organizations. We elaborate a series of theoretical formalisms and practical implementation for modeling, discovering, and visualizing workflow performer-role affiliation networking knowledge, and practical details as workflow performer-role affiliation knowledge representation, discovery, and visualization techniques. These theoretical concepts and practical algorithms are based upon information control net methodology for formally describing workflow models, and the affiliated knowledge eventually represents the various degrees of involvements and participations between a group of performers and a group of roles in a corresponding workflow model. Finally, we summarily describe the implications of the proposed affiliation networking knowledge as business process intelligence, and how worthwhile it is in discovering and visualizing the knowledge in workflow-driven organizations and enterprises that produce massively parallel interactions and large-scaled operational data collections through deploying and enacting massively parallel and large-scale workflow models.

A Study of Spatial Interpolation Impact on Large Watershed Rainfall Considering Elevation (고도를 고려한 공간보간기법이 대유역 강우량 산정시 미치는 영향 연구)

  • Jung, Hyuk;Shin, Hyung-Jin;Park, Jong-Yoon;Jung, In-Kyun;Kim, Seong-Joon
    • Journal of The Korean Society of Agricultural Engineers
    • /
    • v.53 no.6
    • /
    • pp.23-29
    • /
    • 2011
  • This study was conducted to identify the effect of lapse rate application according to elevation on the estimation of large scale watershed rainfall. For the Han river basin (26,018 $km^2$), the 11 years (2000-2010) daily rainfall data from 108 AWS (Automatic Weather Station) were collected. Especially, the 11 heavy rain and typhoon events from 2004 to 2009 were selected for trend analysis. The elevation effect by IDW (Inverse Distance Weights) interpolation showed the change up to +62.7 % for 1,200~1,600m elevation band. The effect based on 19 subbasins of WAMIS (Water Resources Management Information System) water resources unit map, the changes of IDW and Thiessen were -8.0 % (Downstream of Han river)~ +19.7 % (Upstream of Namhan river) and -5.7 %~+15.9 % respectively. It showed the increase trend as the elevation increases. For the 11 years rainfall data analysis, the lapse rate effect of IDW and Thiessen showed increase of 9.7 %~15.5 % and 6.6 %~9.6 % respectively.