KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

Automated Facial Wrinkle Segmentation Scheme Using UNet++

  • Hyeonwoo Kim (School of Electrical Engineering, Korea University) ;
  • Junsuk Lee (School of Electrical Engineering, Korea University) ;
  • Jehyeok, Rew (Department of Data Science, Duksung Women's University) ;
  • Eenjun Hwang (School of Electrical Engineering, Korea University)
  • Received : 2024.03.27
  • Accepted : 2024.06.03
  • Published : 2024.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

Facial wrinkles are widely used to evaluate skin condition or aging for various fields such as skin diagnosis, plastic surgery consultations, and cosmetic recommendations. In order to effectively process facial wrinkles in facial image analysis, accurate wrinkle segmentation is required to identify wrinkled regions. Existing deep learning-based methods have difficulty segmenting fine wrinkles due to insufficient wrinkle data and the imbalance between wrinkle and non-wrinkle data. Therefore, in this paper, we propose a new facial wrinkle segmentation method based on a UNet++ model. Specifically, we construct a new facial wrinkle dataset by manually annotating fine wrinkles across the entire face. We then extract only the skin region from the facial image using a facial landmark point extractor. Lastly, we train the UNet++ model using both dice loss and focal loss to alleviate the class imbalance problem. To validate the effectiveness of the proposed method, we conduct comprehensive experiments using our facial wrinkle dataset. The experimental results showed that the proposed method was superior to the latest wrinkle segmentation method by 9.77%p and 10.04%p in IoU and F1 score, respectively.

Keywords

Acknowledgement

This work was partly supported by National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. RS-2023-00252257), and the Ministry of Trade, Industry and Energy(MOTIE) and Korea Institute for Advancement of Technology(KIAT) through the International Cooperative R&D program (Project No. P0017192).

References

  1. P. K. Mukherjee, N. Maity, N. K. Nema, and B. K. Sarkar, "Bioactive compounds from natural resources against skin aging," Phytomedicine, vol.19, no.1, pp.64-73, 2011.
  2. P. Quatresooz, L. Thirion, C. Pierard-Franchimont, and G.E. Pierard, "The riddle of genuine skin microrelief and wrinkles," International Journal of Cosmetic Science, vol.28, no.6, pp.389-395, 2006.
  3. O. F. Osman, R. M. I. Elbashir, I. E. Abbass, C. Kendrick, M. Goyal, and M. H. Yap, "Automated assessment of facial wrinkling: A case study on the effect of smoking," in Proc. of 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp.1081-1086, 2017.
  4. M. H. Yap, N. Batool, C. C. Ng, M. Rogers, and K. Walker, "A Survey on Facial Wrinkles Detection and Inpainting: Datasets, Methods, and Challenges," IEEE Transactions on Emerging Topics in Computational Intelligence, vol.5, no.4, pp.505-519, 2021.
  5. G. O. Cula, P. R. Bargo, A. Nkengne, and N. Kollias, "Assessing facial wrinkles: automatic detection and quantification," Skin Research and Technology, vol.19, no.1, pp.e243-e251, 2013.
  6. C. C. Ng, M. H. Yap, N. Costen, and B. Li, "Wrinkle Detection Using Hessian Line Tracking," IEEE Access, vol.3, pp.1079-1088, 2015.
  7. A. Alrabiah, M. Alduailij, and M. Crane, "Computer-based Approach to Detect Wrinkles and Suggest Facial Fillers," International Journal of Advanced Computer Science and Applications (IJACSA), vol.10, no.9, pp.319-325, 2019.
  8. N. Batool and R. Chellappa, "Fast detection of facial wrinkles based on Gabor features using image morphology and geometric constraints," Pattern Recognition, vol.48, no.3, pp.642-658, 2015.
  9. C. C. Ng, M. H. Yap, N. Costen, and B. Li, "Automatic Wrinkle Detection Using Hybrid Hessian Filter," in Proc. of Computer Vision --ACCV 2014, 12th Asian Conference on Computer Vision, vol.9005, pp.609-622, 2015.
  10. M. H. Yap, J. Alarifi, C. C. Ng, N. Batool, and K. Walker, "Automated Facial Wrinkles Annotator," in Proc. of Computer Vision - ECCV 2018 Workshops, vol.11132, pp.676-680, 2019.
  11. H. Kim, C. Kim, H. Kim, S. Cho, and E. Hwang, "Panoptic blind image inpainting," ISA Transactions, vol.132, pp.208-221, 2023.
  12. H. Kim, H. Kim, J. Shim, and E. Hwang, "A robust kinship verification scheme using face age transformation," Computer Vision and Image Understanding, vol.231, 2023.
  13. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," Communications of the ACM, vol.60, no.6, pp.84-90, 2017.
  14. H. Li, K. Wu, H. Cheng, C. Gu, and X. Guan, "Nasolabial Folds Extraction based on Neural Network for the Quantitative Analysis of Facial Paralysis," in Proc. of 2018 2nd International Conference on Imaging, Signal Processing and Communication (ICISPC), pp.54-58, 2018.
  15. S. Umirzakova and and T. K. Whangbo, "Nasolabial Wrinkle Segmentation Based on Nested Convolutional Neural Network," in Proc. of 2021 International Conference on Information and Communication Technology Convergence (ICTC), pp.483-485, 2021.
  16. S. Kim, H. Yoon, J. Lee, and S. Yoo, "Semi-automatic Labeling and Training Strategy for Deep Learning-based Facial Wrinkle Detection," in Proc. of 2022 IEEE 35th International Symposium on Computer-Based Medical Systems (CBMS), pp.383-388, 2022.
  17. H. Yoon, S. Kim, J. Lee, and S. Yoo, "Deep-Learning-Based Morphological Feature Segmentation for Facial Skin Image Analysis," Diagnostics, vol.13, no.11, 2023.
  18. Z. Zhou, M. M. Rahman Siddiquee, N. Tajbakhsh, and J. Liang, "UNet++: A Nested U-Net Architecture for Medical Image Segmentation," in Proc. of Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, DLMIA ML-CDS 2018, vol.11045, pp.3-11, 2018.
  19. W.R. Crum, O. Camara, and D.L.G. Hill, "Generalized Overlap Measures for Evaluation and Validation in Medical Image Analysis," IEEE Transactions on Medical Imaging, vol.25, no.11, pp.1451-1461, 2006.
  20. T. Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, "Focal Loss for Dense Object Detection," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.42, no.2, pp.318-327, 2020.
  21. D. Gabor, "Theory of communication. Part 1: The analysis of information," Journal of the Institution of Electrical Engineers -Part III: Radio and Communication Engineering, vol.93, no.26, pp.429-441, 1946.
  22. R. Girshick, "Fast R-CNN," in Proc. of 2015 IEEE International Conference on Computer Vision (ICCV), pp.1440-1448, 2015.
  23. C. Peng, X. Zhang, G. Yu, G. Luo, and J. Sun, "Large Kernel Matters - Improve Semantic Segmentation by Global Convolutional Network," in Proc. of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1743-1751, 2017.
  24. D. Bradley and G. Roth, "Adaptive Thresholding using the Integral Image," Journal of Graphics Tools, vol.12, no.2, pp.13-21, 2007.
  25. O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional Networks for Biomedical Image Segmentation," in Proc. of Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015, vol.9351, pp.234-241, 2015.
  26. M. Sanchez, G. Triginer, C. Ballester, L. Raad, and E. Ramon, "Photorealistic Facial Wrinkles Removal," in Proc. of 16th Asian Conference on Computer Vision - ACCV 2022 Workshops, vol.13848, pp.117-133, 2023.
  27. T. Karras, S. Laine, and T. Aila, "A Style-Based Generator Architecture for Generative Adversarial Networks," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.43, no.12, pp.4217-4228, 2021.
  28. Y. Kartynnik, A. Ablavatski, I. Grishchenko, and M. Grundmann, "Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs," arXiv:1907.06724, 2019.
  29. I. Grishchenko, A. Ablavatski, Y. Kartynnik, K. Raveendran, and M. Grundmann, "Attention Mesh: High-fidelity Face Mesh Prediction in Real-time," arXiv:2006.10962, 2020.
  30. C. Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu, "Deeply-Supervised Nets," in Proc. of the Eighteenth International Conference on Artificial Intelligence and Statistics, PMLR, vol.38, pp.562-570, 2015.
  31. S. Xie, R. Girshick, P. Dollar, Z. Tu, and K. He, "Aggregated Residual Transformations for Deep Neural Networks," in Proc. of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.5987-5995, 2017.
  32. P. Iakubovskii, "Segmentation models PyTorch," GitHub repository, 2019.
  33. D. P. Kingma and J. Ba, "Adam: A Method for Stochastic Optimization," in Proc. of the 3rd International Conference for Learning Representations, San Diego, 2015.