• Journal of the Institute of Electronics Engineers of Korea CI
• /
• v.46 no.2
• /
• pp.65-74
• /
• 2009

In optimizing the neural network structure, there are two methods of the pruning scheme and the constructive scheme. In this paper we use the pruning scheme to optimize neural network structure, and the genetic algorithm to find out its optimum node pruning. In the conventional researches, the input and hidden layers were optimized separately. On the contrary we attempted to optimize the two layers simultaneously by encoding two layers in a chromosome. The offspring networks inherit the weights from the parent. For teaming, we used the existing error back-propagation algorithm. In our experiment with various databases from UCI Machine Learning Repository, we could get the optimal performance when the network size was reduced by about $8{\sim}25%$. As a result of t-test the proposed method was shown better performance, compared with other pruning and construction methods through the cross-validation.

• The Journal of the Korea Contents Association
• /
• v.7 no.1
• /
• pp.58-64
• /
• 2007

In pattern recognition field, there are many optimization problems having exponential search spaces. To solve of sequential search algorithms seeking sub-optimal solutions have been used. The algorithms have limitations of stopping at local optimums. Recently lots of researches attempt to solve the problems using genetic algorithms. This paper explains the huge search spaces of typical problems such as feature selection, classifier ensemble selection, neural network pruning, and clustering, and it reviews the genetic algorithms for solving them. Additionally we present several subjects worthy of noting as future researches.

• Journal of the Korea Institute of Information and Communication Engineering
• /
• v.23 no.1
• /
• pp.55-62
• /
• 2019

Convolutional neural networks (CNNs) show high performance in the computer vision, but they require an enormous amount of operations, making them unsuitable for some resource- or energy-starving environments like the embedded environments. To overcome this problem, there have been much research on accelerators or pruning of CNNs. The previous pruning schemes have not considered the architecture of CNN accelerators, so the accelerators for the pruned CNNs have some inefficiency. This paper proposes a new pruning scheme, Conv-XP, which considers the architecture of CNN accelerators. In Conv-XP, the pruning is performed following the 'X' or '+' shape. The Conv-XP scheme induces a simple architecture of the CNN accelerators. The experimental results show that the Conv-XP scheme does not degrade the accuracy of CNNs, and that the accelerator area can be reduced by 12.8%.

• Journal of the Korean Institute of Telematics and Electronics B
• /
• v.33B no.1
• /
• pp.187-194
• /
• 1996

In this paper, we propose regression neural networks based on regression trees. We map regression trees into three layered feedforward networks. We put multi feature split functions in the first layer so that the networks have a better chance to get optimal partitions of input space. We suggest two supervised learning algorithms for the network training and test both in single feature split and multifeature split functions. In experiments, the proposed regression neural networks is proved to have the better learning performance than those of the single feature split regression trees and the single feature split regression networks. Furthermore, we shows that the proposed learning schemes have an effect to prune an over-grown tree without degrading the learning performance.

• Proceedings of the Korean Society of Broadcast Engineers Conference
• /
• 2019.11a
• /
• pp.44-45
• /
• 2019

최근 CNN 등 인공신경망은 최근 이미지 분류, 객체 인식, 자연어 처리 등 다양한 분야에서 뛰어난 성능을 보이고 있다. 그러나, 대부분의 분야에서 보다 더 높은 성능을 얻기 위해 사용한 인공신경망 모델들은 파라미터 수 및 연산량 등이 방대하여, 모바일 및 IoT 디바이스 같은 연산량이나 메모리가 제한된 환경에서 추론하기에는 제한적이다. 따라서 연산량 및 모델 파라미터 수를 압축하기 위한 딥러닝 경량화 알고리즘이 연구되고 있다. 본 논문에서는 임베디트 보드에서의 압축된 CNN 모델의 성능을 검증한다. 인공지능 지원 맞춤형 칩인 QCS605 를 내장한 임베디드 보드에서 카메라로 입력한 영상에 대해서 원 CNN 모델과 압축된 CNN 모델의 분류 성능과 동작속도 비교 분석한다. 본 논문의 실험에서는 CNN 모델로 MobileNetV2, VGG16 을 사용했으며, 주어진 모델에서 가지치기(pruning) 기법, 양자화, 행렬 분해 등의 인공신경망 압축 기술을 적용하였을 때 원래의 모델 대비 추론 시간 및 분류의 정확도 성능을 분석하고 인공신경망 압축 기술의 유용성을 확인하였다.

• Journal of the Institute of Convergence Signal Processing
• /
• v.22 no.3
• /
• pp.99-103
• /
• 2021

Recently, convolutional neural networks (CNNs) have been widely utilized due to their outstanding performance in various computer vision fields. However, due to their computational-intensive and high memory requirements, it is difficult to deploy CNNs on hardware platforms that have limited resources, such as mobile devices and IoT devices. To address these limitations, a neural network compression research is underway to reduce the size of neural networks while maintaining their performance. This paper proposes a CNN compression technique that dynamically adjusts the thresholds of pruning, one of the neural network compression techniques. Unlike the conventional pruning that experimentally or heuristically sets the thresholds that determine the weights to be pruned, the proposed technique can dynamically find the optimal thresholds that prevent accuracy degradation and output the light-weight neural network in less time. To validate the performance of the proposed technique, the LeNet was trained using the MNIST dataset and the light-weight LeNet could be automatically obtained 1.3 to 3 times faster without loss of accuracy.

• Journal of the Korea Institute of Information and Communication Engineering
• /
• v.24 no.1
• /
• pp.141-144
• /
• 2020

Convolutional neural networks (CNNs) show high performance in computer vision tasks including object detection, but a lot of weight storage and computation is required. In this paper, a pruning scheme is applied to CNNs for object detection, which can remove much amount of weights with a negligible performance degradation. Contrary to the previous ones, the pruning scheme applied in this paper considers the base accelerator architecture. With the consideration, the pruned CNNs can be efficiently performed on an ASIC or FPGA accelerator. Even with the constrained pruning, the resulting CNN shows a negligible degradation of detection performance, less-than-1% point degradation of mAP on VOD0712 test set. With the proposed scheme, CNNs can be applied to objection dtection efficiently.

• Electronics and Telecommunications Trends
• /
• v.38 no.5
• /
• pp.12-22
• /
• 2023

The development of neural networks is evolving towards the adoption of transformer structures with attention modules. Hence, active research focused on extending the concept of lightweight neural network algorithms and hardware acceleration is being conducted for the transition from conventional convolutional neural networks to transformer-based networks. We present a survey of state-of-the-art research on lightweight neural network algorithms and hardware architectures to reduce memory usage and accelerate both inference and training. To describe the corresponding trends, we review recent studies on token pruning, quantization, and architecture tuning for the vision transformer. In addition, we present a hardware architecture that incorporates lightweight algorithms into artificial intelligence processors to accelerate processing.

• Proceedings of the Korean Society of Broadcast Engineers Conference
• /
• 2020.07a
• /
• pp.662-663
• /
• 2020

최근 MPEG 에서는 인공신경망 모델을 다양한 딥러닝 프레임워크에서 상호운용 가능한 포맷으로 압축 표현할 수 있는 NNR(Compression of Neural Network for Multimedia Content Description and Analysis) 표준화를 진행하고 있다. 본 논문에서는 MPEG-NNR 에서 CNN 모델을 압축하기 위한 지역 비선형 양자화(Local Non-linear Quantization: LNQ) 기법을 제시한다. 제안하는 LNQ 는 균일 양자화된 CNN 모델의 각 계층의 가중치 행렬 블록 단위로 추가적인 비선형 양자화를 적용한다. 또한, 제안된 LNQ 는 가지치기(pruning)된 모델의 경우 블록내의 영(zero) 값의 가중치들은 그대로 전송하고 영이 아닌 가중치만을 이진 군집화를 적용한다. 제안 기법은 음성 분류를 위한 CNN 모델(DCASE Task)의 압축 실험에서 기존 균일 양자화를 대비 동일한 분류 성능에서 약 1.78 배 압축 성능 향상이 있음을 확인하였다.

• Journal of Broadcast Engineering
• /
• v.25 no.2
• /
• pp.200-207
• /
• 2020

Recently, deep neural networks such as CNN are showing excellent performance in various fields such as image classification, object recognition, visual quality enhancement, etc. However, as the model size and computational complexity of deep learning models for most applications increases, it is hard to apply neural networks to IoT and mobile environments. Therefore, neural network compression algorithms for reducing the model size while keeping the performance have been being studied. In this paper, we apply few compression methods to CNN models and evaluate their performances in the embedded environment. For evaluate the performance, the classification performance and inference time of the original CNN models and the compressed CNN models on the image inputted by the camera are evaluated in the embedded board equipped with QCS605, which is a customized AI chip. In this paper, a few CNN models of MobileNetV2, ResNet50, and VGG-16 are compressed by applying the methods of pruning and matrix decomposition. The experimental results show that the compressed models give not only the model size reduction of 1.3~11.2 times at a classification performance loss of less than 2% compared to the original model, but also the inference time reduction of 1.2~2.21 times, and the memory reduction of 1.2~3.8 times in the embedded board.