• Title/Summary/Keyword: 토큰화 처리

Search Result 43, Processing Time 0.017 seconds

A Comparative study on the Effectiveness of Segmentation Strategies for Korean Word and Sentence Classification tasks (한국어 단어 및 문장 분류 태스크를 위한 분절 전략의 효과성 연구)

  • Kim, Jin-Sung;Kim, Gyeong-min;Son, Jun-young;Park, Jeongbae;Lim, Heui-seok
    • Journal of the Korea Convergence Society
    • /
    • v.12 no.12
    • /
    • pp.39-47
    • /
    • 2021
  • The construction of high-quality input features through effective segmentation is essential for increasing the sentence comprehension of a language model. Improving the quality of them directly affects the performance of the downstream task. This paper comparatively studies the segmentation that effectively reflects the linguistic characteristics of Korean regarding word and sentence classification. The segmentation types are defined in four categories: eojeol, morpheme, syllable and subchar, and pre-training is carried out using the RoBERTa model structure. By dividing tasks into a sentence group and a word group, we analyze the tendency within a group and the difference between the groups. By the model with subchar-level segmentation showing higher performance than other strategies by maximal NSMC: +0.62%, KorNLI: +2.38%, KorSTS: +2.41% in sentence classification, and the model with syllable-level showing higher performance at maximum NER: +0.7%, SRL: +0.61% in word classification, the experimental results confirm the effectiveness of those schemes.

Study on the Connection with Public Authentication and Bio Authentication (공인인증서와 바이오인증 연계를 위한 연구)

  • Ryu, Gab-Sang
    • Journal of Internet of Things and Convergence
    • /
    • v.1 no.1
    • /
    • pp.39-44
    • /
    • 2015
  • Organization is increasing the authorizing process to use public certificate and bio information. Certificate, has evolved to be able to parallel distributes the bio authentication and portable bio-authentication device. Authentication using an individual's PC and smart devices continue to generalize, while convenience for authentication is increased by comparison Study on cooperation with the security at the network level's a weak situation. If ask authentication method through the cooperation of the public certificate and bio information work with current network access control, there is a possibility to develop a more powerful security policy. by cooperation weaknesses against vulnerable personal authentication techniques on security token in a reliable and secure personal authentication techniques, such as bio-recognition, Bio Information for identification and to prevent exposing a methodology suggest to validate whether or not to carry out in this paper. In addition, organize the scenario that can work with the 802.1x network authentication method, and presented a proposal aimed at realization.

Operation and Analysis of Network for Multivendor PLC Group Management in the Wall Paper Process (발포 벽지 공정에서 이기종 PLC 그룹 관리를 위한 네트워크 운영과 해석)

  • Gang, Seong-Deok;Lee, Dong-Chun;Kim, Jeong-Ho;Lee, Sang-Beom
    • The Transactions of the Korea Information Processing Society
    • /
    • v.3 no.2
    • /
    • pp.283-294
    • /
    • 1996
  • To operate the automatic devices of manufacturing process more effectively and to solve the needs of the resource sharing, network technology is applied to the unit control devices located in common manufacturing zone and operated by connecting them. This paper introduces deign and test operation of cell system for the management of multivendor PLC by applying to wall paper process based on the manufacturing standardization of CIM. The analysis of designed multivendor PLC group is performed by selecting the parameters such as variation of data packet size and node number of PLC ladder logic program and variation by analyzing the corresponding variation values of token rotation time and waiting time for the system operation. For the method of the analysis,the equation is considering the overhead such as indicated packet service time and transmission safety margin for PLC network connection in M/G/1 queue model,and it is applied to the cell system and PLC group manage-ment for operation .Through the experiment,it is found that the realtime processing is possible with expanded and better result than the resuit obtained by Jayasumana.Browaka with PLC scan time considered,response lower limit of 10-20 msec. data packet size not more than 50 bytes.and the number of nodes less than 40.

  • PDF