[KSCI] Korea Science Citation Index Service

http://dx.doi.org/10.3837/tiis.2021.06.002

Micro-Expression Recognition Base on Optical Flow Features and Improved MobileNetV2

Xu, Wei (College of Computer Science and Information Engineering, Guangxi Normal University)
Zheng, Hao (School of Information and Engineering, Nanjing XiaoZhuang University)
Yang, Zhongxue (School of Information and Engineering, Nanjing XiaoZhuang University)
Yang, Yingjie (Centre for Computational Intelligence, De Montfort University)

Publication Information

KSII Transactions on Internet and Information Systems (TIIS) / v.15, no.6, 2021 , pp. 1981-1995 More about this Journal

Abstract

When a person tries to conceal emotions, real emotions will manifest themselves in the form of micro-expressions. Research on facial micro-expression recognition is still extremely challenging in the field of pattern recognition. This is because it is difficult to implement the best feature extraction method to cope with micro-expressions with small changes and short duration. Most methods are based on hand-crafted features to extract subtle facial movements. In this study, we introduce a method that incorporates optical flow and deep learning. First, we take out the onset frame and the apex frame from each video sequence. Then, the motion features between these two frames are extracted using the optical flow method. Finally, the features are inputted into an improved MobileNetV2 model, where SVM is applied to classify expressions. In order to evaluate the effectiveness of the method, we conduct experiments on the public spontaneous micro-expression database CASME II. Under the condition of applying the leave-one-subject-out cross-validation method, the recognition accuracy rate reaches 53.01%, and the F-score reaches 0.5231. The results show that the proposed method can significantly improve the micro-expression recognition performance.

Keywords

Micro-expression; MobileNetV2; optical flow; SVM;

Citations & Related Records

Reference

1	M. B. Nejad, and M. E. Shiri, "A new enhanced learning approach to automatic image classification based on SALP swarm algorithm," Computer Systems Science and Engineering, vol. 34, no.2, pp. 91-100, 2019. DOI
2	M. Duan, K. Li, K. Li, and Q. Tian, "A Novel Multi-task Tensor Correlation Neural Network for Facial Attribute Prediction," ACM Transactions on Intelligent Systems and Technology, vol. 12, no. 1, pp. 1-22, Feb. 2021. DOI
3	W. Fang, F. Zhang, Y. Ding, and J. Sheng, "A new sequential image prediction method based on LSTM and DCGAN," Computers, Materials & Continua, vol. 64, no. 1, pp. 217-231, 2020. DOI
4	X. Wu, C. Luo, Q. Zhang, J. Zhou, H. Yang, and Y. Li, "Text detection and recognition for natural scene images using deep convolutional neural networks," Computers, Materials & Continua, vol. 61, no. 1, pp. 289-300, 2019. DOI
5	O. B. Sezer and A. M. Ozbayoglu, "Financial trading model with stock bar chart image time series with deep convolutional neural networks," Intelligent Automation & Soft Computing, vol. 26, no.2, pp. 323-334, 2020.
6	D. Patel, X. Hong, and G. Zhao, "Selective deep features for micro-expression recognition," in Proc. of 2016 23rd international conference on pattern recognition, pp. 2258-2263, 2016.
7	S. T. Liong, J. See, R. C. W. Phan, A. C. Le Ngo, Y. H. Oh, and K. Wong, "Subtle expression recognition using optical strain weighted features," in Proc. of Asian conference on computer vision, pp. 644-657, 2014.
8	G. Zhang, X. Sun, J. Liu, and J. Chu, "Research on TV-L1 Optical Flow Model for Image Registration Based on Fractional-order Difierentiation," Acta Automatica Sinca, vol. 43, no. 12, pp. 2213-2224, 2017.
9	Y. S. Gan, S. T. Liong, W. C. Yau, Y. C. Huang, and L. K. Tan, "Off-apexnet on micro-expression recognition system," Signal Processing: Image Communication, vol. 74, pp. 129-139, 2019. DOI
10	B. Horn, and B. G. Schunck, "Determining optical flow," Artificial intelligence, 17(1-3), pp. 185-203, 1981. DOI
11	X. Shen, Q. Wu, and X. Fu, "Effects of the duration of expressions on the recognition of microexpressions," Journal of Zhejiang University Science B, vol. 13, no. 3, pp. 221-230, Mar. 2012. DOI
12	Y. H. Oh, A. C. Le Ngo, J. See, S. T. Liong, R. C. W. Phan and H. C. Ling, "Monogenic Riesz wavelet representation for micro-expression recognition," in Proc. of 2015 IEEE International Conference on Digital Signal Processing (DSP), pp. 1237-1241, 2015.
13	T. Pock, M. Urschler, C. Zach, R. Beichel, and H. Bischof, "A Duality Based Algorithm for TV-L1-Optical-Flow Image Registration," in Proc. of International Conference on Medical Image Computing and Computer-assisted Intervention, Springer, Berlin, Heidelberg, pp. 511-518, 2007.
14	S. J. Wang, W. J. Yan, X. Li, G. Zhao, C. G. Zhou, X. Fu, M. Yang, and J. Tao, "Micro-Expression Recognition Using Color Spaces," IEEE Transactions on Image Processing, vol. 24, no. 12, pp. 6034-6047, Dec. 2015. DOI
15	A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, "Mobilenets: Efficient convolutional neural networks for mobile vision applications," arXiv preprint arXiv:1704.04861, 2017.
16	M. Peng, C. Wang, T. Chen, G. Liu, and X. Fu, "Dual temporal scale convolutional neural network for micro-expression recognition," Frontiers in psychology, vol. 8, Oct. 2017.
17	Q. Wu, X. Shen, and X. Fu, "The machine knows what you are hiding: an automatic microexpression recognition system," in Proc. of international conference on affective computing and intelligent Interaction, Springer, Berlin, Heidelberg, pp. 152-162, 2011.
18	A. C. Le Ngo, J. See and R. C. W. Phan, "Sparsity in Dynamics of Spontaneous Subtle Emotions: Analysis and Application," IEEE Transactions on Affective Computing, vol. 8, no. 3, pp. 396-411, 1 July-Sept. 2017. DOI
19	L. I. Rudin, S. Osher, and E. Fatemi, "Nonlinear total variation based noise removal algorithms," Physica D: nonlinear phenomena, vol. 60, no. 1-4, pp. 259-268, 1992. DOI
20	P. Ekman and W. V. Friesen, "Nonverbal leakage and clues to deception," Psychiatry, vol. 32, no. 1, pp. 88-106, 1969. DOI
21	M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. C. Chen, "MobileNetV2: Inverted Residuals and Linear Bottlenecks," in Proc. of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2018.
22	W. J. Yan, X. Li, S. J. Wang, G. Zhao, Y. J. Liu, Y. H. Chen, and X. Fu, "CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation," PLoS ONE, vol. 9, no. 1, p. e86041, Jan. 2014. DOI
23	F. Xu, J. Zhang and J. Z. Wang, "Microexpression identification and categorization using a facial dynamics map," IEEE Transactions on Affective Computing, vol. 8, no. 2, pp. 254-267, 1 April-June 2017. DOI
24	P. Zhang, X. Ben, R. Yan, C. Wu, and C. Guo, "Micro-expression recognition system," Optik, vol. 127, no. 3, pp. 1395-1400, Feb. 2016. DOI
25	G. Zhao and M. Pietikainen, "Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 6, pp. 915-928, June 2007. DOI
26	Y. Wang, J. See, R. C. W. Phan, and Y. H. Oh, "LBP with Six Intersection Points: Reducing Redundant Information in LBP-TOP for Micro-expression Recognition," in Proc. of Asian conference on computer vision, Springer, Cham, pp. 525-537, 2014.
27	J. Li, Y. Lv, B. Ma, M. Yang, C. Wang and Y. Zheng, "Video source identification algorithm based on 3D geometric transformation," Computer Systems Science and Engineering, vol. 35, no.6, pp.513-521, 2020. DOI
28	E. A. Haggard and K. S. Isaacs, "Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy," Methods of research in psychotherapy, pp. 154-165, 1966.
29	O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and F. Li, "Imagenet large scale visual recognition challenge," International journal of computer vision, vol. 115, no. 3, pp. 211-252, Apr. 2015. DOI
30	M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, "The Pascal Visual Object Classes Challenge: A Retrospective," International Journal of Computer Vision, vol. 111, no. 1, pp. 98-136, Jun. 2015. DOI
31	Y. Liu, J. Zhang, W. Yan, S. Wang, G. Zhao, and X. Fu, "A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition," IEEE Transactions on Affective Computing, vol. 7, no. 4, pp. 299-310, 1 Oct.-Dec. 2016. DOI
32	A. Asthana, S. Zafeiriou, S. Cheng, and M. Pantic, "Robust discriminative response map fitting with constrained local models," in Proc. of 2013 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2013.
33	G. Ravikanth, K. V. N. Sunitha, and B. E. Reddy, "Location related signals with satellite image fusion method using visual image integration method," Computer Systems Science and Engineering, vol. 35, no.5, pp. 385-393, 2020. DOI
34	W. Fang, L. Pang and W. N. Yi, "Survey on the application of deep reinforcement learning in image processing," Journal on Artificial Intelligence, vol. 2, no. 1, pp. 39-58, 2020. DOI
35	K. Zhu, N. Zhang, Q. Zhang, S. Ying and X. Wang, "Software defect prediction based on nonlinear manifold learning and hybrid deep learning techniques," Computers, Materials & Continua, vol. 65, no. 2, pp. 1467-1486, 2020. DOI
36	M. Verma, S. K. Vipparthi, G. Singh, and S. Murala, "LEARNet: Dynamic imaging network for micro expression recognition," IEEE Transactions on Image Processing, vol. 29, pp. 1618-1627, 2020. DOI
37	T. Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick, "Microsoft coco: Common objects in context," in Proc. of European conference on computer vision, Springer, Cham, pp. 740-755, 2014.
38	S. Wang, W. Yan, X. Li, G. Zhao and X. Fu, "Micro-expression Recognition Using Dynamic Textures on Tensor Independent Color Space," in Proc. of 2014 22nd International Conference on Pattern Recognition, pp. 4678-4683, 2014.
39	C. Zach, T. Pock, and H. Bischof, "A Duality Based Approach for Realtime TV-L1 Optical Flow," in Proc. of Joint pattern recognition symposium, Springer, Berlin, Heidelberg, pp. 214-223, 2007.