Browse > Article
http://dx.doi.org/10.11627/jksie.2021.44.4.193

Development of Dataset Evaluation Criteria for Learning Deepfake Video  

Kim, Rayng-Hyung (Department. of Industiral & Management Enginering, Hanbat National University)
Kim, Tae-Gu (Department. of Industiral & Management Enginering, Hanbat National University)
Publication Information
Journal of Korean Society of Industrial and Systems Engineering / v.44, no.4, 2021 , pp. 193-207 More about this Journal
Abstract
As Deepfakes phenomenon is spreading worldwide mainly through videos in web platforms and it is urgent to address the issue on time. More recently, researchers have extensively discussed deepfake video datasets. However, it has been pointed out that the existing Deepfake datasets do not properly reflect the potential threat and realism due to various limitations. Although there is a need for research that establishes an agreed-upon concept for high-quality datasets or suggests evaluation criterion, there are still handful studies which examined it to-date. Therefore, this study focused on the development of the evaluation criterion for the Deepfake video dataset. In this study, the fitness of the Deepfake dataset was presented and evaluation criterions were derived through the review of previous studies. AHP structuralization and analysis were performed to advance the evaluation criterion. The results showed that Facial Expression, Validation, and Data Characteristics are important determinants of data quality. This is interpreted as a result that reflects the importance of minimizing defects and presenting results based on scientific methods when evaluating quality. This study has implications in that it suggests the fitness and evaluation criterion of the Deepfake dataset. Since the evaluation criterion presented in this study was derived based on the items considered in previous studies, it is thought that all evaluation criterions will be effective for quality improvement. It is also expected to be used as criteria for selecting an appropriate deefake dataset or as a reference for designing a Deepfake data benchmark. This study could not apply the presented evaluation criterion to existing Deepfake datasets. In future research, the proposed evaluation criterion will be applied to existing datasets to evaluate the strengths and weaknesses of each dataset, and to consider what implications there will be when used in Deepfake research.
Keywords
Deepfake; Dataset; Video; Evaluation criteria; AHP;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 Kim, T.G., Kim, D.H., Lee, D.J., Kim, K.T., and Moon, S.D.S., Development of Checklist to Promote Commercialization of R&D for Railway Transportation using AHP method, Journal of the Korea Management Engineers Society, 2017, Vol. 22, No. 1, pp. 77-87.   DOI
2 National Information Society Agency, Guideline for artificial intelligence learning data quality management, 2020.
3 Yun, C.H., Shin, H.Y., Choo, S.Y., and Kim, J.I., An evaluation study on artificial intelligence data validation methods and open-source frameworks, Journal of Korea Multimedia Society, 2021, Vol. 24, No. 10, pp. 1403-1413.   DOI
4 Ramadhani, K.N. and Munir, R., A Comparative Study of Deepfake Video Detection Method, In 2020 3rd International Conference on Information and Communications Technology (ICOIACT), IEEE, 2020, November, pp. 394-399.
5 Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., and Niessner, M., Faceforensics++: Learning to detect manipulated facial images, In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 1-11.
6 Saaty, T.L., Axiomatic foundation of the analytic hierarchy process, Management Science, 1986, Vol. 32, No. 7, pp. 841- 855.   DOI
7 Thies, J., Zollhofer, M., and Niessner, M., Deferred neural rendering, Image synthesis using neural textures. ACM Transactions on Graphics (TOG), 2019, Vol. 38, No. 4, pp. 1-12.
8 Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., and Niessner, M., Face2face: Real-time face capture and reenactment of rgb videos, In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2387-2395.
9 Verdoliva, L., Media Forensics and Deepfakes: An Overview, IEEE Journal of Selected Topics in Signal Processing, 2020, Vol. 14, No. 5, pp. 910-932.   DOI
10 Tayseer, M., Mohammad, J., Ababneh, M., Al-Zoube, A., and Elhassan, A., Digital Forensics and Analysis of Deepfake Videos, In 11th International Conference on Information and Communication Systems (ICICS), 2020.
11 Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., and Ortega-Garcia, J., Deepfakes and beyond: A survey of face manipulation and fake detection, Information Fusion, 2020, Vol. 64, pp. 131-148.   DOI
12 Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., and Ferrer, C.C., The Deepfake detection challenge (dfdc) dataset, arXiv preprint arXiv:2006.07397, 2020.
13 Dordevic, M., Milivojevic, M., and Gavrovska, A., Deepfake video analysis using SIFT features, In 2019 27th Telecommunications Forum (TELFOR), IEEE, 2019, November, pp. 1-4.
14 Jiang, L., Li, R., Wu, W., Qian, C., and Loy, C. C., Deeperforensics-1.0: A large-scale dataset for real-world face forgery detection, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 2889-2898.
15 Karras, T., Aila, T., Laine, S., and Lehtinen, J., Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
16 Kim, H. S. A study on the data quality management evaluation model. Journal of the Korea Convergence Society, 2020, Vol. 11, No. 7, pp. 217-222.   DOI
17 KOREA Data Agency, Guideline for data quality management(Ver 2.1), 2006.
18 Korshunov, P., and Marcel, S., Vulnerability assessment and detection of Deepfake videos, In 2019 International Conference on Biometrics (ICB), IEEE, 2019, June, pp. 1-6.
19 Kwon, P., You, J., Nam, G., Park, S., and Chae, G., KoDF: A Large-scale Korean Deepfake Detection Dataset, arXiv preprint arXiv:2103.10094, 2021.
20 Le, T. N., Nguyen, H. H., Yamagishi, J., and Echizen, I. OpenForensics: Large-Scale Challenging Dataset For Multi-Face Forgery Detection And Segmentation In-The-Wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10117-10127.
21 Li, Y., and Lyu, S., Exposing Deepfake videos by detecting face warping artifacts, ArXiv Preprint ArXiv:1811.00656, 2018.
22 Feng, K., Wu, J., and Tian, M., A Detect method for Deepfake video based on full face recognition, In 2020 IEEE International Conference on Information Technology, Big Data and Artificial Intelligence (ICIBA), IEEE, 2020, November, Vol. 1, pp. 1121-1125.
23 Li, Y., Yang, X., Sun, P., Qi, H., and Lyu, S., Celeb-df: A large-scale challenging dataset for Deepfake forensics, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 3207-3216.
24 Mahfoudi, G., Tajini, B., Retraint, F., Morain-Nicolier, F., Dugelay, J.L., and Marc, P.I.C., DEFACTO: Image and face manipulation dataset, In 2019 27th European Signal Processing Conference (EUSIPCO), IEEE, 2019, September, pp. 1-5.
25 Matern, F., Riess, C., and Stamminger, M., Exploiting visual artifacts to expose Deepfakes and face manipulations, In 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW), IEEE, 2019, January, pp. 83-92.
26 Park, G. and Kim, C.J., Quality Characteristics of Public Open Data, Journal of Digital Convergence, 2015, Vol. 13, No. 10, pp. 135-146.   DOI
27 Wang, R., Juefei-Xu, F., Ma, L., Xie, X., Huang, Y., Wang, J., and Liu, Y., Fakespotter: A simple yet robust baseline for spotting ai-synthesized fake faces, ArXiv Preprint ArXiv:1909.06122, 2019.
28 Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., and Catanzaro, B., High-resolution image synthesis and semantic manipulation with conditional gans, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 8798-8807.
29 Yang, J.Y. and Lee, S.R., A Study on the Priority-Gap Measurement of Performance Factors Before and After Introduction of Electronic Price Information System in Retail Stores using IT-BSC and AHP, Information Systems Review, 2020, Vol. 22, No. 2, pp. 53-76.   DOI
30 Guarnera, L., Giudice, O., and Battiato, S., Fighting Deepfake by exposing the convolutional traces on images, IEEE Access, 2020, Vol. 8, pp. 165085-165098.   DOI
31 Dolhansky, B., Howes, R., Pflaum, B., Baram, N., and Ferrer, C. C., The Deepfake detection challenge (dfdc) preview dataset, arXiv preprint arXiv:1910.08854., 2019.
32 Dufour, N. and Gully, A., Contributing data to deep-fake detection research, 2019. URL https://ai.googleblog.com/2019/09/contributing-data-to-Deepfake-detection.html.
33 Youm, G.Y. and Kim, M.C., SAR image target recognition research trend using deep learning, The Journal of The Korean Institute of Communication Sciences, 2017, Vol. 34, No. 7, pp. 31-39.
34 Yu, P., Xia, Z., Fei, J., and Lu, Y., A Survey on Deepfake Video Detection, IET Biometrics, 2016, October.
35 Zi, B., Chang, M., Chen, J., Ma, X., and Jiang, Y.G., WildDeepfake: A challenging real-world dataset for Deepfake detection, In Proceedings of the 28th ACM International Conference on Multimedia, 2020, October, pp. 2382-2390.
36 UCI Machine Learning Repository, https://archive.ics.uci.edu/ml/datasets.php.
37 Kaggle, https://www.kaggle.com/.