Browse > Article
http://dx.doi.org/10.3837/tiis.2019.04.001

A Review of Deep Learning Research  

Mu, Ruihui (College of Computer and Information, Hohai University)
Zeng, Xiaoqin (College of Computer and Information, Hohai University)
Publication Information
KSII Transactions on Internet and Information Systems (TIIS) / v.13, no.4, 2019 , pp. 1738-1764 More about this Journal
Abstract
With the advent of big data, deep learning technology has become an important research direction in the field of machine learning, which has been widely applied in the image processing, natural language processing, speech recognition and online advertising and so on. This paper introduces deep learning techniques from various aspects, including common models of deep learning and their optimization methods, commonly used open source frameworks, existing problems and future research directions. Firstly, we introduce the applications of deep learning; Secondly, we introduce several common models of deep learning and optimization methods; Thirdly, we describe several common frameworks and platforms of deep learning; Finally, we introduce the latest acceleration technology of deep learning and highlight the future work of deep learning.
Keywords
Deep learning; machine learning; artificial intelligence; learning model; neural network; optimization method;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Zhang J, Shan S, Kan M, Chen X, "Coarse-to-fine auto-encoder networks for real-time face alignment," in Proc. of European Conference on Computer Vision. Springer International Publishing, pp.1-16, 2014.
2 Hinton G E, "Training products of experts by minimizing contrastive divergence," Neural computation, vol.14, no.8, pp.1771-1800, 2002.   DOI
3 Hinton G E, "A Practical Guide to Training restricted Boltzmann machines," Momentum, vol.9, no.1, pp.599-619, 2012.
4 Krizhevsky A, Hinton G E, "Using very deep autoencoders for content-based image retrieva," in Proc. of ESANN, pp.262-270, 2011.
5 Zou W Y, Ng A Y, Yu K, "Unsupervised learning of visual invariance with temporal coherence," in Proc. of Workshop on Deep Learning and Unsupervised Feature Learning, pp.362-370, 2011.
6 Krizhevsky A, Sutskever I, Hinton G E, "ImageNet classification with deep convolutional neural networks," Communications of the ACM, vol.60, no.6, pp.84-90, 2017.   DOI
7 Hinton G E, Ranzato M, "Modeling pixel means and covariances using factorized third-order Boltzmann machines," in Proc. of Computer Vision and Pattern Recognition Conference. pp.2551-2558, 2010.
8 Glorot X, Bengio Y, "Understanding the difficulty of training deep feedforward neural networks," Journal of Machine Learning Research, vol.9, pp.249-256, Jan 2010.
9 Rousseau A, Attik M, Schwenk H, "Large pruned or continuous space language models on a GPU for statistical machine translation," in Proc. of the NAACL-HLT 2012 Workshop: Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, Association for Computational Linguistics, pp.11-19, June 2012.
10 Karlen M, Collobert R, Weston J, "Natural Language Processing (Almost) from Scratch," Journal of Machine Learning Research, vol.12, no.1, pp.2493-2537, 2011.
11 Kumar A, Irsoy O, Ondruska P, "Ask me anything: Dynamic memory networks for natural language processing," in Proc. of International Conference on Machine Learning. New York, NJ, USA: ACM, pp.1378-1387, 2016.
12 Deoras A, Kombrink, SMikolov T, "Empirical evaluation and combination of advanced language modeling techniques," in Proc. of Conference of the International Speech Communication Association. pp.605-608, 2011.
13 Vincent P, Janvin C,Bengio Y, "A neural probabilistic language model," Journal of Machine Learning Research, vol.3, no.6, pp.1137-1155,2003.
14 BAHDANAU D, CHO K, BENGIO Y, "Neural machine translation by jointly learning to align and translate," CORR, pp.473-485, 2014.
15 Weiss D, Alberti C, Collins M, "Structured training for neural network transition-based parsing," Eprint Arxiv, vol.6, no.3, pp.125-133, 2015.
16 Lample G, Ballesteros M, Subramanian S, "Neural architectures for named entity recognition," Eprint Arxiv, vol.6, no.4, pp.86-93, 2016.
17 He L, Lee K, Lewis M, "Deep semantic role labeling: What works and what's next," in Proc. of Annual Meeting of the Association for Computational Linguistics. New York, NJ, USA: ACM, pp.473-483, 2017.
18 Severyn A, Moschitti A, "Twitter sentiment analysis with deep convolutional neural networks," in Proc. of Research on Development in Information Retrieval. New York, NJ, USA: ACM, pp.959-962, 2015.
19 Gantz J, Reinsel D, "The digital universe in 2020: Big data, bigger digital shadows, and biggest growth in the far east," IDC iView: IDC Analyze the future, pp.1-16, 2012.
20 Marz N, Warren J, "Big Data: Principles and best practices of scalable real time data systems," Greenwich, USA: Manning Publications Co, 2015.
21 LeCun Y, Bengio Y, Hinton G, "Deep learning," Nature, vol.521, pp.436-444, 2015.   DOI
22 Peng Y, Zhu W, Zhao Y, "Cross-media analysis and reasoning: advances and directions," Frontiers of Information Technology & Electronic Engineering, vol.18, no.1, pp.44-57, Feb.2017.   DOI
23 Dong C., Loy C. C., He K., Tang X, "Image Super-Resolution Using Deep Convolutional Networks," IEEE Transactions on Pattern Analysis and Machine Intelligence, vo.38, no.2, pp. 295-307, 2015.   DOI
24 Denker J S, Boser B, Lecun Y, "Backpropagation applied to handwritten zip code recognition," Neural Computation, vol.1, no.4, pp.541-551, 1989.   DOI
25 Ren S., He K., Girshick R., Sun J, "Faster R-CNN: Towards real-time object detection with region proposal networks," in Proc. of the Advances in Neural Information Processing Systems, Montreal, Canada, pp.91-99, 2015.
26 Wang X, Wang Y, "Improving content-based and hybrid music recommendation using deep learning," in Proc. of 22nd ACM International Conference on Multimedia, Orlando, USA, pp.627-636, 2014.
27 Courville A, Bergstra J, Bengio Y, "A spike and slab restricted Boltzmann machine," in Proc. of 14th International Conference on Artificial Intelligence and Statistics, pp.233-241, 2011.
28 Memisevic R, Hinton G, "Unsupervised learning of image transformations," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp.1-8, 2007.
29 Hinton G E, Osindero S, Teh Y W, "A fast learning algorithm for deep belief nets," Neural Computation, vol.18, no.7, pp.1527-1554, 2006.   DOI
30 Bengio Y, "Learning deep architectures for AI," Foundations and Trends(R) in Machine Learning, vol.2, no.1, pp.1-127, 2009.   DOI
31 Goodfellow Ian J, Pouget-Abadie Jean, Mirza Mehdi, "Generative Adversarial Networks," Eprint arXiv:1406.2661, vol.6, no.3, pp.125-133, 2014.
32 Karpathy A, Li F F, "Deep visual-semantic alignments for generating image descriptions," IEEE Trans on Pattern Analysis& Machine Intelligence, vol.39, no.4, pp.656-664, 2017.
33 Zhou X, Dong D, Wu H, "Multi-view response selection for human-computer conversation," in Proc. of Empirical Methods in Natural Language Processing. New York, NJ, USA: ACM, pp.372-381, 2016.
34 He Y X, Sun S T, Niu F F, "A deep learning model enhanced with emotion semantics for microblog sentiment analysis," Chinese Journal of Computers, vol.40, no.4, pp.773-790, 2017.
35 Gehring J, Auli M, Grangier D, "Convolutional sequence to sequence learning," Eprint Arxiv, vol.3, no.2, pp.86-93, 2017.
36 Vinyals O, Fortunato M, Jaitly N, "Pointer networks," in Proc. of Neural Information Processing Systems. Cambridge, USA: MIT Press, pp.2692-2700, 2015.
37 Rumelhart D, Williams R, Hinton G, "Learning representations by back-propagating errors," Nature, vol.323, pp. 533-538, 1986.   DOI
38 Bourlard H, Kamp Y, "Auto-association by multi-layer perceptron and singular value decomposition," Biological Cybernetics, vol.59, no.4, pp.291-294, 1988.   DOI
39 Mikolov T, Le Q V, "Distributed Representations of Sentences and Documents," Eprint arXiv:1405.4053, vol.4, pp.1172-1188, 2014.
40 Kim Y, "Convolutional Neural Network for Sentence Classification," Eprint arXiv:1408.5882, vol.6, no.3, pp.105-112, 2015.
41 Hinton G E, Salakhutdinov R R, "Reducing the dimensionality of data with neural networks," Science, Vol.313(5786), pp.504-507, 2006.   DOI
42 Vincent P, Larochelle H, Bengio Y, "Extracting and composing robust features with denoising autoencoders," in Proc. of International Conference, pp.1096-1103, 2008.
43 Sejnowski T J, Hinton G E, "Learning and relearning in Boltzmann machines," Parallel distributed processing: explorations in the microstructure of cognition, MIT Press, vol.1, pp.45-76, 1986.
44 Le Q V, Ngiam J, Coates A, "On optimization methods for deep learning," in Proc. of International Conference on Machine Learning, pp.265-272, 2011.
45 Wang Z., Liu D., Yang J., Han W., Huang T, "Deep Networks for Image Super-Resolution with Sparse Prior," in Proc. of the International Conference on Computer Vision, Santiago, Chile, pp.370-278, 2015.
46 Yu K, Yang J, Gong Y, "Linear spatial pyramid matching using sparse coding for image classification," in Proc. of 2009 IEEE on Computer Vision and Pattern Recognition(CVPR), Miami, FL, USA, pp. 1794-1801, 2009.
47 Xia R., Pan Y., Lai H., Liu C., Yan S, "Supervised Hashing for Image Retrieval via Image Representation Learning," in Proc. of the Association for the Advancement of Artificial Intelligence, Quebec City, Canada, pp.2156-2162, 2014.
48 Mao Q, Dong M, Huang Z, "Learning salient features for speech emotion recognition using convolutional neural networks," IEEE Transactions on Multimedia, vol.6, no.8, pp.2203-2213, 2014.
49 Felix W, Jurgen G, Martin W, "Feature enhancement by deep LSTM networks for ASR in reverberant multisource environments," Computer Speech & Language, vol.28, no.4, pp.888-902, 2014.   DOI
50 ZEN H, SENIOR A, SCHUSTER M, "Statistical parametric speech synthesis using deep neural networks," in Proc. of 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, NJ: IEEE, pp.7962-7966, May 2013.
51 MOHAMED A R, DAHL G E, HINTON G, "Acoustic modeling using deep belief networks," IEEE transactions on audio, speech, and language processing, vol.20, no.1, pp.14-22, 2012.   DOI
52 CHO K, van MERRIENBOER B, GULCEHRE C, "Learning phrase representations using RNN encoder decoder for statistical machine translation," CORR, pp.1406-1416, 2014.
53 Rifai S, Vincent P, Muller X, "Contractive auto-encoders: Explicit invariance during feature extraction," in Proc. of the 28th International Conference on Machine Learning, pp.833-840, 2011.
54 Taylor G W, Fergus R, Zeiler M D, "Adaptive deconvolutional networks for mid and high-level feature learning," in Proc. of IEEE International Conference on Computer Vision, pp.2018-2025, 2011.