Comparison of Deep Learning Frameworks: About Theano, Tensorflow, and Cognitive Toolkit

The deep learning framework is software designed to help develop deep learning models. Some of its important functions include "automatic differentiation" and "utilization of GPU". The list of popular deep learning framework includes Caffe (BVLC) and Theano (University of Montreal). And recently, Microsoft's deep learning framework, Microsoft Cognitive Toolkit, was released as open-source license, following Google's Tensorflow a year earlier. The early deep learning frameworks have been developed mainly for research at universities. Beginning with the inception of Tensorflow, however, it seems that companies such as Microsoft and Facebook have started to join the competition of framework development. Given the trend, Google and other companies are expected to continue investing in the deep learning framework to bring forward the initiative in the artificial intelligence business. From this point of view, we think it is a good time to compare some of deep learning frameworks. So we compare three deep learning frameworks which can be used as a Python library. Those are Google's Tensorflow, Microsoft's CNTK, and Theano which is sort of a predecessor of the preceding two. The most common and important function of deep learning frameworks is the ability to perform automatic differentiation. Basically all the mathematical expressions of deep learning models can be represented as computational graphs, which consist of nodes and edges. Partial derivatives on each edge of a computational graph can then be obtained. With the partial derivatives, we can let software compute differentiation of any node with respect to any variable by utilizing chain rule of Calculus. First of all, the convenience of coding is in the order of CNTK, Tensorflow, and Theano. The criterion is simply based on the lengths of the codes and the learning curve and the ease of coding are not the main concern. According to the criteria, Theano was the most difficult to implement with, and CNTK and Tensorflow were somewhat easier. With Tensorflow, we need to define weight variables and biases explicitly. The reason that CNTK and Tensorflow are easier to implement with is that those frameworks provide us with more abstraction than Theano. We, however, need to mention that low-level coding is not always bad. It gives us flexibility of coding. With the low-level coding such as in Theano, we can implement and test any new deep learning models or any new search methods that we can think of. The assessment of the execution speed of each framework is that there is not meaningful difference. According to the experiment, execution speeds of Theano and Tensorflow are very similar, although the experiment was limited to a CNN model. In the case of CNTK, the experimental environment was not maintained as the same. The code written in CNTK has to be run in PC environment without GPU where codes execute as much as 50 times slower than with GPU. But we concluded that the difference of execution speed was within the range of variation caused by the different hardware setup. In this study, we compared three types of deep learning framework: Theano, Tensorflow, and CNTK. According to Wikipedia, there are 12 available deep learning frameworks. And 15 different attributes differentiate each framework. Some of the important attributes would include interface language (Python, C ++, Java, etc.) and the availability of libraries on various deep learning models such as CNN, RNN, DBN, and etc. And if a user implements a large scale deep learning model, it will also be important to support multiple GPU or multiple servers. Also, if you are learning the deep learning model, it would also be important if there are enough examples and references.

딥러닝 프레임워크의 비교: 티아노, 텐서플로, CNTK를 중심으로

딥러닝 프레임워크의 대표적인 기능으로는 '자동미분'과 'GPU의 활용' 등을 들 수 있다. 본 논문은 파이썬의 라이브러리 형태로 사용 가능한 프레임워크 중에서 구글의 텐서플로와 마이크로소프트의 CNTK, 그리고 텐서플로의 원조라고 할 수 있는 티아노를 비교하였다. 본문에서는 자동미분의 개념과 GPU의 활용형태를 간단히 설명하고, 그 다음에 logistic regression을 실행하는 예를 통하여 각 프레임워크의 문법을 알아본 뒤에, 마지막으로 대표적인 딥러닝 응용인 CNN의 예제를 실행시켜보고 코딩의 편의성과 실행속도 등을 확인해 보았다. 그 결과, 편의성의 관점에서 보면 티아노가 가장 코딩 하기가 어렵고, CNTK와 텐서플로는 많은 부분이 비슷하게 추상화 되어 있어서 코딩이 비슷하지만 가중치와 편향을 직접 정의하느냐의 여부에서 차이를 보였다. 그리고 각 프레임워크의 실행속도에 대한 평가는 '큰 차이는 없다'는 것이다. 텐서플로는 티아노에 비하여 속도가 느리다는 평가가 있어왔는데, 본 연구의 실험에 의하면, 비록 CNN 모형에 국한되었지만, 텐서플로가 아주 조금이지만 빠른 것으로 나타났다. CNTK의 경우에도, 비록 실험환경이 달랐지만, 실험환경의 차이에 의한 속도의 차이의 편차범위 이내에 있는 것으로 판단이 되었다. 본 연구에서는 세 종류의 딥러닝 프레임워크만을 살펴보았는데, 위키피디아에 따르면 딥러닝 프레임워크의 종류는 12가지가 있으며, 각 프레임워크의 특징을 15가지 속성으로 구분하여 차이를 특정하고 있다. 그 많은 속성 중에서 사용자의 입장에서 볼 때 중요한 속성은 어떤 언어(파이썬, C++, Java, 등)로 사용가능한지, 어떤 딥러닝 모형에 대한 라이브러리가 잘 구현되어 있는지 등일 것이다. 그리고 사용자가 대규모의 딥러닝 모형을 구축한다면, 다중 GPU 혹은 다중 서버를 지원하는지의 여부도 중요할 것이다. 또한 딥러닝 모형을 처음 학습하는 경우에는 사용설명서가 많은지 예제 프로그램이 많은지 여부도 중요한 기준이 될 것이다.