Browse > Article
http://dx.doi.org/10.13089/JKIISC.2020.30.6.1053

A Method of Detection of Deepfake Using Bidirectional Convolutional LSTM  

Lee, Dae-hyeon (Korea University)
Moon, Jong-sub (Korea University)
Publication Information
Journal of the Korea Institute of Information Security & Cryptology / v.30, no.6, 2020 , pp. 1053-1065 More about this Journal
Abstract
With the recent development of hardware performance and artificial intelligence technology, sophisticated fake videos that are difficult to distinguish with the human's eye are increasing. Face synthesis technology using artificial intelligence is called Deepfake, and anyone with a little programming skill and deep learning knowledge can produce sophisticated fake videos using Deepfake. A number of indiscriminate fake videos has been increased significantly, which may lead to problems such as privacy violations, fake news and fraud. Therefore, it is necessary to detect fake video clips that cannot be discriminated by a human eyes. Thus, in this paper, we propose a deep-fake detection model applied with Bidirectional Convolution LSTM and Attention Module. Unlike LSTM, which considers only the forward sequential procedure, the model proposed in this paper uses the reverse order procedure. The Attention Module is used with a Convolutional neural network model to use the characteristics of each frame for extraction. Experiments have shown that the model proposed has 93.5% accuracy and AUC is up to 50% higher than the results of pre-existing studies.
Keywords
Deepfake; LSTM; Attention module; Artificial intelligence; Time distribution;
Citations & Related Records
Times Cited By KSCI : 2  (Citation Analysis)
연도 인용수 순위
1 K. Li, Z. Wu, K.-C. Peng, J. Ernst, and Y. Fu, "Tell me where to look: Guided attention inference network," arXiv preprint arXiv:1802.10171, 2018.
2 E. Sabir, J. Cheng, A. Jaiswal, W. AbdAlmageed, I. Masi, and P. Natarajan, "Recurrent Convolutional Strategies for Face Manipulation Detection in Videos," in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019.
3 W.S. Hu, H.C. Li, L. Pan, W. Li, R. Tao, and Q. Du, "Feature extraction and classification based on spatial-spectral convlstm neural network for hyperspectral images," arXiv preprint arXiv:1905.03577, 2019.
4 Y. Li and S. Lyu, "Exposing deepfake videos by detecting face warping artifacts," arXiv preprint arXiv:1811.00656, 2018.
5 P. Korshunov and S. Marcel, "Deepfakes: a new threat to face recognition? assessment and detection," arXiv preprint arXiv:1812.08685, 2018.
6 T.T. Nguyen et al., "Deep Learning for Deepfakes Creation and Detection," arXiv preprints, arXiv:1909.11573, 2019.
7 Q. Liu, F. Zhou, R. Hang, and X. Yuan, "Bidirectional-convolutional LSTM based spectral-spatial feature learning for hyperspectral image classification," Remote Sens, vol. 9, no. 12, p. 1330, 2017.   DOI
8 J. Deng, J. Guo, and S. Zafeiriou, "ArcFace: Additive angular margin loss for deep face recognition," arXiv:1801.07698, 2018.
9 H. Fukui, T. Hirakawa, T. Yamashita, and H. Fujiyoshi, "Attention branch network: Learning of attention mechanism for visual explanation," In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
10 A. Hanson, K. Pnvr, S. Krishnagopal, and L. Davis, "Bidirectional convolutional LSTM for the detection of violence in videos," Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 11130 LNCS, pp. 280-295, 2019.
11 S. Woo, J. Park, J. Y. Lee, and I. So Kweon, "Cbam: Convolutional block attention module," In Proceedings of the European Conference on Computer Vision (ECCV), pages 3-19, 2018.
12 Y. Li, P. Sun, H. Qi, and S. Lyu, "Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics," in IEEE Conf. on Computer Vision and Patten Recognition (CVPR), Seattle, WA, United States, 2020.
13 S. Xingjian, Z. Chen, H. Wang, D. Yeung, W. Wong, and W. Woo, "Convolutional LSTM network: A machine learning approach for precipitation nowcasting," In Neural Information Processing Systems, 2015.
14 O.M. Parkhi, A. Vedaldi, A. Zisserman, "Deep face recognition." In Proceedings of the British Machine Vision, vol. 1, no. 3, p. 6, 2015.
15 I. Amerini, L. Galteri, R. Caldelli, and A. Bimbo, "Deepfake Video Detection through Optical Flow based CNN," in Proc. IEEE/CVF International Conference on Computer Vision, 2019.
16 D. Guera and EJ. Delp, "Deepfake video detection using recurrent neural networks," In AVSS, 2018.
17 M. Tan and Q.V. Le, "EfficientNet: Rethinking model scaling for convolutional neural networks," In International Conference on Machine Learning, 2019.
18 D.A. Pitaloka, A. Wulandari, T. Basaruddin, and D.Y. Liliana, "Enhancing cnn with preprocessing stage in automatic emotion recognition," Procedia Computer Science, vol. 116, pp. 523-529, 2017.   DOI
19 A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Niebner. "Faceforensics++: Learning to detect manipulated facial images." arXiv preprint arXiv:1901.08971, 2019.
20 H. Dang, F. Liu, J. Stehouwer, X. Liu, and A. Jain, "On the Detection of Digital Face Manipulation," in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
21 C. Ding and D. Tao, "Robust face recognition via multimodal deep face representation," IEEE TMM, 17(11):2049-2058, 2015.
22 L. Chen, H. Zhang, J. Xiao, L. Nie, J. Shao, W. Liu, and T. Chua, "SCACNN: Spatial and channel-wise attention in convolutional networks for image captioning," In CVPR, 2017.
23 S. Suwajanakorn, S.M. Seitz, and I. Kemelmacher-Shlizerman, "Synthesizing Obama: learning lip sync from audio," ACM Transactions on Graphics (TOG), 36(4), 2017.
24 Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, "Deepface: Closing the gap to human-level performance in faceverification," In CVPR, 2014.
25 R. Tolosana, R. Vera-Rodriguez, J. Fierrez, A. Morales, and J. OrtegaGarcia, "DeepFakes and beyond: A survey of face manipulation and fake detection," Inf. Fusion, 2020.
26 A. Singh, A. S. Saimbhi, N. Singh an d M. Mittal, "DeepFake Video Detection: A Time-Distributed Approach," SN Computer Science, 2020.
27 C. Szegedy, V. Vanhoucke, S. Iofe, J. Shlens, Z. Wojna, "Rethinking the inception architecture for computer vision," In Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. p. 2818-26.
28 K. Dale, K. Sunkavalli, MK. Johnson, D. Vlasic, W. Matusik, H. Pfster, "Video face replacement," In Proceedings of the 2011 SIGGRAPH Asia conference. 2011. p. 1-10
29 J. Thies, M. Zollhofer, M. Stamminger, C. Theobalt, and Matthias Niebner, "Face2Face: Real-time face capture and reenactment of RGB videos," In CVPR, 2016.
30 F. Chollet, "Xception: Deep learning with depthwise separable convolutions," In Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. p. 1251-8.
31 J. Hu, L. Shen, G. Sun, "Squeeze-and-excitation networks," In Proceedings of the IEEE conference on computer vision and pattern recognition. 2018. p. 7132-41.
32 K. He, X. Zhang, S. Ren, J. Sun, "Deep residual learning for image recognition," In Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. p. 770-8.
33 DP. Kingma and M. Welling, "Auto-encoding variational bayes," In ICLR, 2014.
34 D.J. Rezende, S. Mohamed, and D. Wierstra, "Stochastic backpropagation and approximate inference in deep generative models," In ICML, 2014.
35 I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville and Y. Bengio, "Generative adversarial nets," In NeurIPS, 2014.
36 ZAO.https://apps.apple.com/cn/app/zao/id1465199127. Accessed: 2019-09-16.
37 FakeApp.https://www.malavida.com/en/soft/fakeapp/, Acessed Nov 4, 2019.
38 S. Bianco, R. Cadene, L. Celona, andP. Napoletano, "Benchmark analysis ofrepresentative deep neural network architectures," IEEE Access, vol. 6, 2018.
39 Ajder, H, Patrini, G, Cavalli, F, et al.(2019) The state of DeepFakes: landscape, threats, and impact. Deeptrace Labs, September. Available at: https://regmedia.co.uk/2019/10/08/deepfake_report.pdf
40 Min-seo Kim and Jong-sub Moon, "Speaker Verification Model Using Short-Time Fourier Transform and RecurrentNeural Network," Jonornal of The Korea Institute of information Security &Cryptology, 29(6), pp. 1393-1401, Feb.2019
41 V.J. Reddi et al., "Mlperf inference benchmark," in arXiv preprint arXiv:1911.02549, 2019.
42 B. Dolhansky, R. Howes, B. Pflaum, N. Baram, and C. C. Ferrer, "The deepfake detection challenge (DFDC) preview dataset," arXiv preprint arXiv:1910.08854, 2019.