• Journal of Korea Multimedia Society
• /
• v.25 no.7
• /
• pp.895-902
• /
• 2022

Building segmentation using satellite imagery such as EO (Electro-Optical) and SAR (Synthetic-Aperture Radar) images are widely used due to their various uses. EO images have the advantage of having color information, and they are noise-free. In contrast, SAR images can identify the physical characteristics and geometrical information that the EO image cannot capture. This paper proposes a learning framework for efficient building segmentation that consists of a teacher-student-based privileged knowledge distillation and deformable convolution block. The teacher network utilizes EO and SAR images simultaneously to produce richer features and provide them to the student network, while the student network only uses EO images. To do this, we present objective functions that consist of Kullback-Leibler divergence loss and knowledge distillation loss. Furthermore, we introduce deformable convolution to avoid pixel-level noise and efficiently capture hard samples such as small and thin buildings at the global level. Experimental result shows that our method outperforms other methods and efficiently captures complex samples such as a small or narrow building. Moreover, Since our method can be applied to various methods.

• Proceedings of the Korean Society of Computer Information Conference
• /
• 2021.07a
• /
• pp.313-315
• /
• 2021

Timely and accurate diagnosis of lung diseases using Chest X-ray images has been gained much attention from the computer vision and medical imaging communities. Although previous studies have presented the capability of deep convolutional neural networks by achieving competitive binary classification results, their models were seemingly unreliable to effectively distinguish multiple disease groups using a large number of x-ray images. In this paper, we aim to build an advanced approach, so-called Ensemble Knowledge Distillation (EKD), to significantly boost the classification accuracies, compared to traditional KD methods by distilling knowledge from a cumbersome teacher model into an ensemble of lightweight student models with parallel branches trained with ground truth labels. Therefore, learning features at different branches of the student models could enable the network to learn diverse patterns and improve the qualify of final predictions through an ensemble learning solution. Although we observed that experiments on the well-established ChestX-ray14 dataset showed the classification improvements of traditional KD compared to the base transfer learning approach, the EKD performance would be expected to potentially enhance classification accuracy and model generalization, especially in situations of the imbalanced dataset and the interdependency of 14 weakly annotated thorax diseases.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2024.05a
• /
• pp.711-714
• /
• 2024

In the dynamic landscape of modern machine learning, Federated Learning (FL) has emerged as a compelling paradigm designed to enhance privacy by enabling participants to collaboratively train models without sharing their private data. Specifically, Distillation-based Federated Learning, like Federated Learning with Model Distillation (FedMD), Federated Gradient Encryption and Model Sharing (FedGEMS), and Differentially Secure Federated Learning (DS-FL), has arisen as a novel approach aimed at addressing Non-IID data challenges by leveraging Federated Learning. These methods refine the standard FL framework by distilling insights from public dataset predictions, securing data transmissions through gradient encryption, and applying differential privacy to mask individual contributions. Despite these innovations, our survey identifies persistent vulnerabilities, particularly concerning the susceptibility to logit inversion attacks where malicious actors could reconstruct private data from shared public predictions. This exploration reveals that even advanced Distillation-based Federated Learning systems harbor significant privacy risks, challenging the prevailing assumptions about their security and underscoring the need for continued advancements in secure Federated Learning methodologies.

• Journal of the Korea Society of Computer and Information
• /
• v.27 no.10
• /
• pp.29-41
• /
• 2022

Recently, with the development of deep learning technology, a variety of huge models with excellent performance have been devised by pre-training massive amounts of text data. However, in order for such a model to be applied to real-life services, the inference speed must be fast and the amount of computation must be low, so the technology for model compression is attracting attention. Knowledge distillation, a representative model compression, is attracting attention as it can be used in a variety of ways as a method of transferring the knowledge already learned by the teacher model to a relatively small-sized student model. However, knowledge distillation has a limitation in that it is difficult to solve problems with low similarity to previously learned data because only knowledge necessary for solving a given problem is learned in a teacher model and knowledge distillation to a student model is performed from the same point of view. Therefore, we propose a heterogeneous knowledge distillation method in which the teacher model learns a higher-level concept rather than the knowledge required for the task that the student model needs to solve, and the teacher model distills this knowledge to the student model. In addition, through classification experiments on about 18,000 documents, we confirmed that the heterogeneous knowledge distillation method showed superior performance in all aspects of learning efficiency and accuracy compared to the traditional knowledge distillation.

• Journal of the Korean Society for information Management
• /
• v.24 no.3
• /
• pp.201-218
• /
• 2007

This study proposes a topic distillation algorithm that ranks the relevant sites selected from retrieved web pages, and evaluates the performance of the algorithm. The algorithm calculates the topic score of a site using its hierarchical structure. The TREC .GOV test collection and a set of TREC-2004 queries for topic distillation task are used for the experiment. The experimental results showed the algorithm returned at least 2 relevant sites in top ten retrieval results. We peformed an in-depth analysis of the relevant sites list provided by TREC-2004 to find out that the definition of topic distillation was not strictly applied in selecting relevant sites. When we re-evaluated the retrieved sites/sub-sites using the revised list of relevant sites, the performance of the proposed algorithm was improved significantly.

• Journal of the Korea Institute of Information and Communication Engineering
• /
• v.25 no.9
• /
• pp.1244-1253
• /
• 2021

In this paper, we propose a method for predicting in advance whether pedestrians will enter the hazardous area after the current time using the pedestrian trajectory prediction method and an efficient simplification method of the trajectory prediction network. In addition, we propose a method to apply KD(Knowledge Distillation) to a small network for real-time operation in an embedded environment. Using the correlation between predicted future paths and hazard zones, we determined whether to enter or not, and applied efficient KD when learning small networks to minimize performance degradation. Experimentally, it was confirmed that the model applied with the simplification method proposed improved the speed by 37.49% compared to the existing model, but led to a slight decrease in accuracy. As a result of learning a small network with an initial accuracy of 91.43% using KD, It was confirmed that it has improved accuracy of 94.76%.

• Membrane and Water Treatment
• /
• v.13 no.1
• /
• pp.51-62
• /
• 2022

Vacuum-assisted air gap membrane distillation (V-AGMD) has the potential to achieve higher flux and productivity than conventional air gap membrane distillation (AGMD). Nevertheless, there is not much information on technical aspects of V-AGMD operation. Accordingly, this study aims to analyze the effect of membrane deformation on flux in V-AGMD operation. Experiments were carried out using a bench-scale V-AGMD system. Statistical models were applied to understand the flux behaviors. Statistical models based on MLR, GNN, and MLFNN techniques were developed to describe the experimental data. Results showed that the flux increased by up to 4 times with the application of vacuum in V-AGMD compared with conventional AGMD. The flux in both AGMD and V-AGMD is affected by the difference between the air gap pressure and the saturation pressure of water vapor, but their dependences were different. In V-AGMD, the membranes were found to be deformed due to the vacuum pressure because they were not fully supported by the spacer. As a result, the deformation reduced the effective air gap width. Nevertheless, the rejection and LEP were not changed even if the deformation occurred. The flux behaviors in V-AGMD were successfully interpreted by the GNN and MLFNN models. According to the model calculations, the relative impact of the membrane deformation ranges from 10.3% to 16.1%.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2023.05a
• /
• pp.479-481
• /
• 2023

Generative Adversarial Networks (GANs) have facilitated substantial improvement in single-image super-resolution (SR) by enabling the generation of photo-realistic images. However, the high memory requirements of GAN-based SRs (mainly generators) lead to reduced performance and increased energy consumption, making it difficult to implement them onto resource-constricted devices. In this study, we propose an efficient and compressed architecture for the SR-GAN (generator) model using the model compression technique Knowledge Distillation. Our approach involves the transmission of knowledge from a heavy network to a lightweight one, which reduces the storage requirement of the model by 58% with also an increase in their performance. Experimental results on various benchmarks indicate that our proposed compressed model enhances performance with an increase in PSNR, SSIM, and image quality respectively for x4 super-resolution tasks.

• Journal of Broadcast Engineering
• /
• v.26 no.4
• /
• pp.419-428
• /
• 2021

Shadow removal plays a key role for the pre-processing of image processing techniques such as object tracking and detection. With the advances of image recognition based on deep convolution neural networks, researches for shadow removal have been actively conducted. In this paper, we propose a novel method for shadow removal, which utilizes self attention distillation to extract semantic features. The proposed method gradually refines results of shadow detection, which are extracted from each layer of the proposed network, via top-down distillation. Specifically, the training procedure can be efficiently performed by learning the contextual information for shadow removal without shadow masks. Experimental results on various datasets show the effectiveness of the proposed method for shadow removal under real world environments.

• Journal of Broadcast Engineering
• /
• v.27 no.3
• /
• pp.378-390
• /
• 2022

With the recent development of deep composite multiplication neural network learning, deep learning techniques applied to single-image super-resolution have shown good results, and the strong expression ability of deep networks has enabled complex nonlinear mapping between low-resolution and high-resolution images. However, there are limitations in applying it to real-time or low-power devices with increasing parameters and computational amounts due to excessive use of composite multiplication neural networks. This paper uses blocks that extract hierarchical characteristics little by little using information distillation and suggests the Recursive Distillation Super Resolution Network (RDSRN), a lightweight network that improves performance by making more accurate high frequency components through high frequency residual purification blocks. It was confirmed that the proposed network restores images of similar quality compared to RDN, restores images 3.5 times faster with about 32 times fewer parameters and about 10 times less computation, and produces 0.16 dB better performance with about 2.2 times less parameters and 1.8 times faster processing time than the existing lightweight network CARN.