KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

A Comprehensive Study on Key Components of Grayscale-based Deepfake Detection

  • Seok Bin Son (Department of Electrical and Computer Engineering, Korea University) ;
  • Seong Hee Park (Department of Computer Engineering, Hongik University) ;
  • Youn Kyu Lee (Department of Computer Engineering, Hongik University)
  • Received : 2023.08.18
  • Accepted : 2024.07.28
  • Published : 2024.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

Advances in deep learning technology have enabled the generation of more realistic deepfakes, which not only endanger individuals' identities but also exploit vulnerabilities in face recognition systems. The majority of existing deepfake detection methods have primarily focused on RGB-based analysis, offering unreliable performance in terms of detection accuracy and time. To address the issue, a grayscale-based deepfake detection method has recently been proposed. This method significantly reduces detection time while providing comparable accuracy to RGB-based methods. However, despite its significant effectiveness, the "key components" that directly affect the performance of grayscale-based deepfake detection have not been systematically analyzed. In this paper, we target three key components: RGB-to-grayscale conversion method, brightness level in grayscale, and resolution level in grayscale. To analyze their impacts on the performance of grayscale-based deepfake detection, we conducted comprehensive evaluations, including component-wise analysis and comparative analysis using real-world datasets. For each key component, we quantitatively analyzed its characteristics' performance and identified differences between them. Moreover, we successfully verified the effectiveness of an optimal combination of the key components by comparing it with existing deepfake detection methods.

Keywords

Acknowledgement

This work was supported in part by the National Research Foundation of Korea (NRF) Grant funded by the Korean Government (MSIT) under Grant RS-2022-00165648, in part by the 2023 Hongik University Research Fund, and in part by the BrainLink Program funded by the Ministry of Science and ICT through the National Research Foundation of Korea under Grant RS-2023-00237308.

References

  1. J. A. Costales et al., "The Impact of Blockchain Technology to Protect Image and Video Integrity from Identity Theft using Deepfake Analyzer," in Proc. of 2023 International Conference on Innovative Data Communication Technologies and Application (ICIDCA), pp.730-733, 2023.
  2. M. Westerlund, "The Emergence of Deepfake Technology: A Review," Technology Innovation Management Review, vol.9, no.11, pp.40-53, Nov. 2019.
  3. M. Albahar et al., "Deepfakes: Threats and Countermeasures Systematic Review," Journal of Theoretical and Applied Information Technology, vol.97, no.22, pp.3242-3250, Nov. 2019. .
  4. M. Kim et al., "FReTAL: Generalizing Deepfake Detection using Knowledge Distillation and Representation Learning," in Proc. of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp.1001-1012, 2021.
  5. A. Ismail et al., "A New Deep Learning-Based Methodology for Video Deepfake Detection Using XGBoost," Sensors, vol.21, no.16, Aug. 2021.
  6. D. Wodajo et al., "Deepfake Video Detection Using Convolutional Vision Transformer," arXiv:2102.11126, 2021.
  7. J. Liu et al., "A lightweight 3D convolutional neural network for deepfake detection," International Journal of Intelligent Systems, vol.36, no.9, pp.4990-5004, Jun. 2021.
  8. Y. Xu et al., "Supervised Contrastive Learning for Generalizable and Explainable DeepFakes Detection," in Proc. of IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW), pp.379-389, 2022.
  9. P. Sebastian et al., "The effect of colour space on tracking robustness," in Proc. of 3rd IEEE Conference on Industrial Electronics and Applications, pp.2512-2516, 2008.
  10. M. Podpora et al., "YUV vs RGB - Choosing a Color Space for Human-Machine Interaction," in Proc. of Federated Conference on Computer Science and Information Systems, vol.3, pp.29-34, 2014.
  11. X. Chang et al., "DeepFake Face Image Detection based on Improved VGG Convolutional Neural Network," in Proc.of 39th Chinese Control Conference (CCC), pp.7252-7256, 2020.
  12. P. Saikia et al., "A Hybrid CNN-LSTM model for Video Deepfake Detection by Leveraging Optical Flow Features," in Proc. of International Joint Conference on Neural Networks (IJCNN), pp.1-7, 2022.
  13. V. V. V. N. S Vamsi et al., "Deepfake detection in digital media forensics," Global Transitions Proceedings, vol.3, no.1, pp.74-79, Jun. 2022.
  14. S. B. Son et al., "A Measurement Study on Gray Channel-based Deepfake Detection," in Proc. of International Conference on Information and Communication Technology Convergence (ICTC), pp.428-430, 2021.
  15. S. McCloskey et al., "Detecting GAN-Generated Imagery Using Saturation Cues," in Proc. of IEEE International Conference on Image Processing (ICIP), pp.4584-4588, 2019.
  16. A. Pishori et al., "Detecting Deepfake Videos: An Analysis of Three Techniques," arXiv:2007.08517, 2020.
  17. S. B. Son et al., "A Comparative Study on Deepfake Detection using Gray Channel Analysis," Journal of Korea Multimedia Society, vol.24, no.9, pp.1224-1241, 2021. .
  18. C. Kanan et al., "Color-to-Grayscale: Does the Method Matter in Image Recognition?," PLoS ONE, vol.7, no.1, Jan. 2012. Artical (CrossRef Link).
  19. S. Yohanandan et al., "Saliency Preservation in Low-Resolution Grayscale Images," in Proc. of Computer Vision - ECCV 2018, pp.237-254, 2018.
  20. P. Korshunov et al., "DeepFakes: a New Threat to Face Recognition? Assessment and Detection," arXiv:1812.08685, 2018.
  21. A. Rossler et al., "FaceForensics++: Learning to Detect Manipulated Facial Images," in Proc. of IEEE/CVF International Conference on Computer Vision (ICCV), pp.1-11, 2019.
  22. Y. Li et al., "Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics," in Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.3204-3213, 2020.
  23. S. Ramachandran et al., "An Experimental Evaluation on Deepfake Detection using Deep Face Recognition," in Proc. of International Carnahan Conference on Security Technology (ICCST), pp.1-6, 2021.
  24. A. Aghasanli et al., "Interpretable-through-prototypes deepfake detection for diffusion models," in Proc. of IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp.467-474, 2023. .
  25. T. Wang et al., "Noise Based Deepfake Detection via Multi-Head Relative-Interaction," in Proc. of the AAAI Conference on Artificial Intelligence, vol.37, no.12, pp.14548-14556, 2023.
  26. M. S. Rana et al., "Deepfake Detection: A Systematic Literature Review," IEEE Access, vol.10, pp.25494-25513, Feb. 2022.
  27. P. Yu et al., "A Survey on Deepfake Video Detection," IET Biometrics, vol.10, no.6, pp.607-624, 2021.
  28. A. M. Naser, "Color to grayscale image conversion based dimensionality reduction with Stationary Wavelet transform," in Proc. of Al-Sadeq International Conference on Multidisciplinary in IT and Communication Science and Applications (AIC-MITCSA), pp.1-5, 2016.
  29. S. Bezryadin et al., "Brightness Calculation in Digital Image Processing," in Proc. of International Symposium on Technologies for Digital Photo Fulfillment, pp.10-15, 2007.
  30. P. Ambalathankandy et al., "Warm-cool color-based high-speed decolorization: an empirical approach for tone mapping applications," Journal of Electronic Imaging, vol.30, no.4, Aug. 2021.
  31. M. E. Paramasivam et al., "Perceptually Weighted Color-to-Grayscale Conversion for Images with Non-Uniform Chromatic Distribution using Multiple Regression," ICTACT Journal on Image and Video Processing, vol.11, no.2, pp.2325-2330, Nov. 2020.
  32. E. Sirisha et al., "Image Retrieval Comparisons Using Color Models," International Journal of Advanced Research in Computer Science, vol.4, no.9, pp.115-118, 2013.
  33. S. Macedo, G. Melo, and J. Kelner, "A Comparative Study of Grayscale Conversion Techniques Applied to SIFT Descriptors," SBC Journal on Interactive Systems, vol.6, no.2, pp.30-36, 2015.
  34. L. Zhang et al., "Color-to-Gray Conversion Based on Boundary Points," in Proc. of 11th International Conference on Communications, Circuits and Systems (ICCCAS), pp.241-245, 2022.
  35. E. Karantoumanis et al., "Computational comparison of image preprocessing techniques for plant diseases detection," in Proc. of 7th South-East Europe Design Automation, Computer Engineering, Computer Networks and Social Media Conference (SEEDA-CECNSM), pp.1-5, 2022.
  36. S. Tabrizchi et al., "AppCiP: Energy-Efficient Approximate Convolution-in-Pixel Scheme for Neural Network Acceleration," IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol.13, no.1, pp.225-236, Mar. 2023.
  37. V. Ganchovska et al., "Converting Color to Grayscale Image Using LabVIEW," in Proc. of International Conference Automatics and Informatics (ICAI), pp.320-323, 2022.
  38. C. H. Yeh et al., "Lightweight Deep Neural Network for Joint Learning of Underwater Object Detection and Color Conversion," IEEE Transactions on Neural Networks and Learning Systems, vol.33, no.11, pp.6129-6143, Nov. 2022.
  39. S. M. Newhall et al., "Final Report of the O.S.A. Subcommittee on the Spacing of the Munsell Colors," Journal of the Optical Society of America, vol.33, no.7, pp.385-418, 1943.
  40. S. H. Hong et al., "Image interpolation using interpolative classified vector quantization," Image and Vision Computing, vol.26, no.2, pp.228-239, Feb. 2008.
  41. R. Durall et al., "Unmasking DeepFakes with simple Features," arXiv:1911.00686, 2019. .
  42. B, Dolhansky et al., "The Deepfake Detection Challenge (DFDC) Dataset," arXiv:2006.07397, 2020.
  43. L. Jiang et al., "DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection," in Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.2886-2895, 2020.
  44. P. Kwon et al., "KoDF: A Large-scale Korean DeepFake Detection Dataset," in Proc. of IEEE/CVF International Conference on Computer Vision (ICCV), pp.10724-10733, 2021.
  45. B. Zi et al., "WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection," in Proc. of MM '20: Proceedings of the 28th ACM International Conference on Multimedia, pp.2382-2390, 2020.
  46. Y. Wang et al., "HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping," in Proc. of the Thirtieth International Joint Conference on Artificial Intelligence, pp.1136-1142, 2021.
  47. MarekKowalski, MarekKowalski/FaceSwap:3D face swapping implemented in Python. https://github.com/MarekKowalski/FaceSwap
  48. Shaoanlu, shaoanlu/faceswap-GAN:A denoising autoencoder + adversarial losses and attention mechanisms for face swapping. https://github.com/shaoanlu/faceswap-GAN
  49. C. Sanderson et al., "Multi-Region Probabilistic Histograms for Robust and Scalable Identity Inference," in Proc. of Advances in Biometrics: Third International Conferences, ICB 2009, pp.199-208, 2009.
  50. K. Simonyan et al., "Very Deep Convolutional Networks for Large-Scale Image Recognition," in Proc. of 3rd International Conference on Learning Representations, ICLR 2015, 2015.
  51. A. K. Dubey et al., "Automatic facial recognition using VGG16 based transfer learning model," Journal of Information and Optimization Sciences, vol.41, no.7, pp.1589-1596, Sep. 2020.
  52. I. Amerini et al., "Deepfake Video Detection through Optical Flow Based CNN," in Proc. of IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp.1205-1207, 2019.
  53. A. Hamza et al., "Deepfake Audio Detection via MFCC Features Using Machine Learning," IEEE Access, vol.10, pp.134018-134028, Dec. 2022.
  54. F. Ding et al., "Anti-Forensics for Face Swapping Videos via Adversarial Training," IEEE Transactions on Multimedia, vol.24, pp.3429-3441, 2022.
  55. F. Chollet, "Xception: Deep Learning with Depthwise Separable Convolutions," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1800-1807, 2017.
  56. H. Ji et al., "Robust video denoising using low rank matrix completion," in Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp.1791-1798, 2010.
  57. A. Karpathy et al., "Large-Scale Video Classification with Convolutional Neural Networks," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1725-1732, 2014.
  58. S. Yeung et al., "End-to-End Learning of Action Detection from Frame Glimpses in Videos," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.2678-2687, 2016.
  59. D. Min et al., "Depth Video Enhancement Based on Weighted Mode Filtering," IEEE Transactions on Image Processing, vol.21, no.3, pp.1176-1190, 2012.
  60. Opencv, opencv/resize.cpp at 4.x · opencv/opencv. https://github.com/opencv/opencv/blob/4.x/module s/imgproc/src/resize.cpp
  61. OpenCV: Geometric Image Transformations. https://docs.opencv.org/4.x/da/d54/group__imgproc__transform.html
  62. L. Li et al., "Advancing High Fidelity Identity Swapping for Forgery Detection," in Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.5073-5082, 2020.
  63. Z. Sun et al., "Improving the Efficiency and Robustness of Deepfakes Detection Through Precise Geometric Features," in Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.3608-3617, 2021.
  64. Y. Zhu et al., "One Shot Face Swapping on Megapixels," in Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.4832-4842, 2021.
  65. Z. Chen et al., "MagDR: Mask-Guided Detection and Reconstruction for Defending Deepfakes," in Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.9010-9019, 2021.
  66. X. Dong et al., "Protecting Celebrities from DeepFake with Identity Consistency Transformer," in Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.9458-9468, 2022.
  67. X. Wang et al., "DeepFake Disrupter: The Detector of DeepFake Is My Friend," in Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.14900-14909, 2022.
  68. L. Chen et al., "Self-supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection," in Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.18689-18698, 2022.
  69. K. Shiohara et al., "Detecting Deepfakes with Self-Blended Images," in Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.18699-18708, 2022.
  70. T. Le et al., "OpenForensics: Large-Scale Challenging Dataset For Multi-Face Forgery Detection And Segmentation In-The-Wild," in Proc. of IEEE/CVF International Conference on Computer Vision (ICCV), pp.10097-10107, 2021.
  71. Z. Cai et al., "MARLIN: Masked Autoencoder for facial video Representation LearnINg," in Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.1493-1504, 2023.
  72. B. Huang et al., "Implicit Identity Driven Deepfake Face Swapping Detection," in Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.4490-4499, 2023.
  73. K. Narayan et al., "DF-Platter: Multi-Face Heterogeneous Deepfake Dataset," in Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.9739-9748, 2023.
  74. N. Larue et al., "SeeABLE: Soft Discrepancies and Bounded Contrastive Learning for Exposing Deepfakes," in Proc. of IEEE/CVF International Conference on Computer Vision (ICCV), pp.20954-20964, 2023.
  75. Z. Yan et al., "UCF: Uncovering Common Features for Generalizable Deepfake Detection," in Proc. of IEEE/CVF International Conference on Computer Vision (ICCV), pp.22355-22366, 2023.
  76. Y. Xu et al., "TALL: Thumbnail Layout for Deepfake Video Detection," in Proc. of IEEE/CVF International Conference on Computer Vision (ICCV), pp.22601-22611, 2023.
  77. D. Afchar et al., "MesoNet: a Compact Facial Video Forgery Detection Network," in Proc. of IEEE International Workshop on Information Forensics and Security (WIFS), pp.1-7, 2018.
  78. Y. Li et al., "Exposing DeepFake Videos By Detecting Face Warping Artifacts," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019.
  79. A. Gunes et al., "Optimizing the color-to-grayscale conversion for image classification," Signal, Image and Video Processing, vol.10, no.5, pp.853-860, 2016.
  80. R. Mukherjee et al., "Object Detection Under Challenging Lighting Conditions Using High Dynamic Range Imagery," IEEE Access, vol.9, pp.77771-77783, 2021.
  81. H. K. Leung et al., "A Deep-Learning-Based Vehicle Detection Approach for Insufficient and Nighttime Illumination Conditions," Applied Sciences, vol.9, no.22, Nov. 2019.
  82. F. Mehmood et al., "Object detection mechanism based on deep learning algorithm using embedded IoT devices for smart home appliances control in CoT," Journal of Ambient Intelligence and Humanized Computing, pp.1-17, Mar. 2019.
  83. J. Zhao et al., "High-Resolution Image Classification Integrating Spectral-Spatial-Location Cues by Conditional Random Fields," IEEE Transactions on Image Processing, vol.25, no.9, pp.4033-4045, Sep. 2016.
  84. D. Guera et al., "Deepfake Video Detection Using Recurrent Neural Networks," in Proc. of 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp.1-6, 2018.
  85. C. M. Yu et al., "Detecting Deepfake-Forged Contents with Separable Convolutional Neural Network and Image Segmentation," arXiv:1912.12184, 2019.
  86. I. Ahmad et al., "Color-to-grayscale algorithms effect on edge detection - A comparative study," in Proc. of International Conference on Electronics, Information, and Communication (ICEIC), pp.1-4, 2018.
  87. H. M. Bui et al., "Using grayscale images for object recognition with convolutional-recursive neural network," in Proc. of IEEE Sixth International Conference on Communications and Electronics (ICCE), pp.321-325, 2016.