• Proceedings of the Korea Information Processing Society Conference
• /
• 2023.05a
• /
• pp.530-531
• /
• 2023

딥러닝에서 지식 증류 기법은 큰 모델의 지식을 작은 모델로 전달하여 작은 모델의 성능을 개선하는 방식이다. 지식 증류 기법은 모델 경량화, 학습 속도 향상, 학습 정확도 향상 등에 활용될 수 있는데, 교사 모델이라 불리는 큰 모델은 일반적으로 학습된 딥러닝 모델을 사용한다. 본 연구에서는 학습된 딥러닝 모델 대신에 수치 기반 시뮬레이션 모델을 사용함으로써 어떠한 효과가 있는지 검증하였으며, 수치 모델을 활용한 기상 예측 모델에서의 지식 증류는 기존 단독 딥러닝 모델 학습 대비 더 작은 학습 횟수(epoch)에서도 동일한 에러 수준(RMSE)까지 도달하여, 학습 속도 측면에서 이득이 있음을 확인하였다.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2023.11a
• /
• pp.524-525
• /
• 2023

설명 가능한 심층 강화 학습 지식 증류 방법(ERL-KD)이 제안하였다. 이 방법은 모든 하위 에이전트로부터 점수를 수집하며, 메인 에이전트는 주 교사 네트워크 역할을 하고 하위 에이전트는 보조 교사 네트워크 역할을 한다. 글로벌 최적 솔루션은 샤플리 값과 같은 해석 가능한 방법을 통해 얻어진다. 또한 유사도 제약이라는 개념을 도입하여 교사 네트워크와 학생 네트워크 간의 유사도를 조정함으로써 학생 네트워크가 자유롭게 탐색할 수 있도록 유도한다. 실험 결과, 학생 네트워크는 아타리 2600 환경에서 대규모 교사 네트워크와 비슷한 성능을 달성하는 것으로 나타났다.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2024.05a
• /
• pp.592-595
• /
• 2024

엄청난 성능에도 불구하고, 심층 신경망은 예측결과에 대한 설명이 없는 블랙 박스로 작동한다는 비판을 받고 있다. 이러한 불투명한 표현은 신뢰성을 제한하고 모델의 대한 과학적 이해를 방해한다. 본 연구는 여러 개의 교사 신경망으로부터 설명 중심의 학생 신경망으로 지식 증류를 통해 해석 가능성을 향상시키는 것을 제안한다. 구체적으로, 인간이 정의한 개념 활성화 벡터 (CAV)를 통해 교사 모델의 개념 민감도를 방향성 도함수를 사용하여 계량화한다. 목표 개념에 대한 민감도 점수에 비례하여 교사 지식 융합을 가중치를 부여함으로써 증류된 학생 모델은 양호한 성능을 달성하면서 네트워크 논리를 해석으로 집중시킨다. 실험 결과, ResNet50, DenseNet201 및 EfficientNetV2-S 앙상블을 7 배 작은 아키텍처로 압축하여 정확도가 6% 향상되었다. 이 방법은 모델 용량, 예측 능력 및 해석 가능성 사이의 트레이드오프를 조화하고자 한다. 이는 모바일 플랫폼부터 안정성이 중요한 도메인에 걸쳐 믿을 수 있는 AI 의 미래를 여는 데 도움이 될 것이다.

• International journal of advanced smart convergence
• /
• v.4 no.1
• /
• pp.154-161
• /
• 2015

A novel process has successfully been developed by overcoming major difficulties through the elimination of number of process steps involved in the Classical Transesterification reaction during the preparation of Fatty Acid Methyl/Ethyl Ester (FAME.FAEE) called biodiesel. The Classical process with cost intensive process steps such as the utilization of excess alcohol, needing downstream distillation for the recovery and reutilization of excess alcohol/cosolvent, unrecoverable homogenous catalyst which consumes vast quantity of fresh distilled water during the purification of the product and downstream waste water treatment before its safe disposal to the surface water body. The Novel Process FAME/FAEE is produced from any vegetable oil irrespective of edible or inedible variety using sonication energy. The novelty of the finding is the use of only theoretical quantity of alcohol along with a co-solvent and reduced quantity of homogeneous catalyst. Under this condition neither the homogeneous catalyst goes to the FAME layer nor is the distillation needed. The same ester also has been prepared in high pressure high temperature reactor without using catalyst at sub critical temperature. The quality of prepared biodiesel without involving any purification step meets the ASTM standards. Blended Biodiesel with Common Diesel Fuel (CDF) and FAME is prepared, characterized and used as fuel in the Kirloskar make CI Engines. The evaluation of the engine performance result of pure CDF, B05 biodiesel, B10 biodiesel of all types of biodiesel prepared by using the feedstock of Soybean (Glycine max) and Karanja (Pongamia pinnate) oil along with their mixed oil provides useful information such as brake power, brake thermal efficiency, brake specific fuel consumption, etc, and established it as ideal fuel for unmodified CI engine.

• The KIPS Transactions:PartA
• /
• v.13A no.3 s.100
• /
• pp.231-240
• /
• 2006

In this paper, wireless internet proxy server clusters are used for the wireless internet because their caching, distillation, and clustering functions are helpful to overcome the limitations and needs of the wireless internet. TranSend was proposed as a clustering based wireless internet proxy server but it has disadvantages; 1) its scalability is difficult to achieve because there is no systematic way to do it and 2) its structure is complex because of the inefficient communication structure among modules. In our former research, we proposed the CD-A structure which can be scalable in a systematic way but it also has disadvantages; its communication structure among modules is partly complex. In this paper, we proposed a consolidated scheme which has a systematic scalability and an efficient communication structure among modules. We performed experiments using 16 PCs and experimental results show 196% and 40% performance improvement of the proposed system compared to the TranSend and the CD-A system, respectively.

• ETRI Journal
• /
• v.41 no.2
• /
• pp.242-253
• /
• 2019

We devise a layer-wise hint training method to improve the existing hint-based knowledge distillation (KD) training approach, which is employed for knowledge transfer in a teacher-student framework using a residual network (ResNet). To achieve this objective, the proposed method first iteratively trains the student ResNet and incrementally employs hint-based information extracted from the pretrained teacher ResNet containing several hint and guided layers. Next, typical softening factor-based KD training is performed using the previously estimated hint-based information. We compare the recognition accuracy of the proposed approach with that of KD training without hints, hint-based KD training, and ResNet-based layer-wise pretraining using reliable datasets, including CIFAR-10, CIFAR-100, and MNIST. When using the selected multiple hint-based information items and their layer-wise transfer in the proposed method, the trained student ResNet more accurately reflects the pretrained teacher ResNet's rich information than the baseline training methods, for all the benchmark datasets we consider in this study.

• Journal of KIISE:Information Networking
• /
• v.31 no.1
• /
• pp.101-111
• /
• 2004

As different from wired internet, wireless internet has limitations due to the following characteristics; low bandwidth, frequent disconnection, low computing power, small screen in user terminal, and user mobility. Also, wireless internet server should be scalable to handle a large scale traffic due to rapidly growing users. Wireless proxy servers are used for the wireless internet because their caching and transcoding functions are helpful to overcome the above limitation. TranSend was proposed as a clustering based wireless proxy server but its scalability is difficult to achieve because there is no systematic way to do it. In this Paper. we proposed a clustering based wireless internet proxy server which can be scalable in a systematic way. We performed experiments using 16 PCs and experimental results show 32.17% performance improvement of the proposed system compared to TranSend system.

• Proceedings of the Korean Information Science Society Conference
• /
• 2005.07a
• /
• pp.313-315
• /
• 2005

사회적으로 큰 관심의 대상이 되고 있는 무선 인터넷은 유선 인터넷과 달리 기술 환경과 그 특성상 여러 가지 제약점들을 가지고 있다. 대역폭이 낮고, 접속이 빈번하게 끊기며, 단말기내의 컴퓨팅 파워가 낮고 화면이 작다. 또한 사용자의 이동성 문제와 네트워크 프로토콜, 보안 등에서 아직 기술적으로 부족한 부분을 보이고 있다. 그리고 급속도로 증가하는 수요에 따라 무선 인터넷 서버는 대용량 트래픽을 처리할 수 있는 확장성이 요구되어지고 있다. 이에 본 논문에서는 무선 인터넷 프록시 서버 클러스터를 사용하여 앞에서 언급된 무선 인터넷의 문제와 요구들을 캐싱(Caching), 압축(Distillation) 및 플러스터(Clustering)를 통하여 해결하려고 한다. TranSend는 클러스터링 기반의 무선 인터넷 프록시 서버로 제안된 것이나 시스템적인(Systematic) 방법으로 확장성을 보장하지 못하고 불필요한 모듈간의 통신구조로 인해 복잡하다는 단점을 가진다. 기존 연구에서 시스템적인 방법으로 확장성을 보장하는 CD-A라는 구조를 제안하였으나 이 역시 모듈간의 불필요한 통신 구조를 가진다는 단점을 있다. 이에 본 논문에서는 시스템적인 확장성과 단순한 구조를 가지는 새로운 클러스터링 기반의 무선 인터넷 프록시 서버를 제안한다. 16대의 컴퓨터를 사용하여 실험을 수행하였고 실험 결과 TranSend 시스템과 CD-A 시스템에 비해 각각 $216\%,\;40\%$의 성능 향상을 보였다.

• Journal of Intelligence and Information Systems
• /
• v.28 no.3
• /
• pp.45-55
• /
• 2022

Recently, large-scale language models (LPLM) have been shown state-of-the-art performances in various tasks of natural language processing including intent classification. However, fine-tuning LPLM requires much computational cost for training and inference which is not appropriate for dialog system. In this paper, we propose compressed intent classification model for multi-agent in low-resource like CPU. Our method consists of two stages. First, we trained sentence encoder from LPLM then compressed it through knowledge distillation. Second, we trained agent-specific adapter for intent classification. The results of three intent classification datasets show that our method achieved 98% of the accuracy of LPLM with only 21% size of it.

• Journal of KIISE:Information Networking
• /
• v.32 no.3
• /
• pp.415-426
• /
• 2005

Wireless internet, which becomes a hot social issue, has limitations due to the following characteristics, as different from wired internet. It has low bandwidth, frequent disconnection, low computing power, and small screen in user terminal. Also, it has technical issues to Improve in terms of user mobility, network protocol, security, and etc. Wireless internet server should be scalable to handle a large scale traffic due to rapidly growing users. In this paper, wireless internet proxy server clusters are used for the wireless Internet because their caching, distillation, and clustering functions are helpful to overcome the above limitations and needs. TranSend was proposed as a clustering based wireless internet proxy server but it has disadvantages; 1) its scalability is difficult to achieve because there is no systematic way to do it and 2) its structure is complex because of the inefficient communication structure among modules. In our former research, we proposed the All-in-one structure which can be scalable in a systematic way but it also has disadvantages; 1) data sharing among cache servers is not allowed and 2) its communication structure among modules is complex. In this paper, we proposed its improved scheme which has an efficient communication structure among modules and allows data to be shared among cache servers. We performed experiments using 16 PCs and experimental results show 54.86$\%$ and 4.70$\%$ performance improvement of the proposed system compared to TranSend and All-in-one system respectively Due to data sharing amount cache servers, the proposed scheme has an advantage of keeping a fixed size of the total cache memory regardless of cache server numbers. On the contrary, in All-in-one, the total cache memory size increases proportional to the number of cache servers since each cache server should keep all cache data, respectively.