Browse > Article
http://dx.doi.org/10.5909/JBE.2020.25.7.1042

Interactive Pixel-unit AR Lip Makeup System Using RGB Camera  

Nam, Hyeongil (Department of Computer Science, Hanyang University)
Lee, Jeongeun (Department of Computer Science, Hanyang University)
Park, Jong-Il (Department of Computer Science, Hanyang University)
Publication Information
Journal of Broadcast Engineering / v.25, no.7, 2020 , pp. 1042-1051 More about this Journal
Abstract
In this paper, we propose an AR (Augmented Reality) lip makeup using bare hands interactively using an RGB camera. Unlike previous interactive makeup studies, this interactive lip makeup system is based on an RGB camera. Also, the system controls the makeup area in pixels, not in polygon-units. For pixel-unit controlling, the system also proposed a 'Rendering Map' that can store the relative position of the touched hand relative to the lip landmarks. With the map, the part to be changed in color can be specified in the current frame. And the lip color of the corresponding area is adjusted, even if the movement of the face changes in the next frame. Through user experiments, we compare quantitatively and qualitatively our makeup method with the conventional polygon-unit method. Experimental results demonstrate that the proposed method enhances the quality of makeup with a little sacrifice of computational complexity. It is confirmed that natural makeup similar to the actual lip makeup is possible by dividing the lip area into more detailed areas. Furthermore, the method can be applied to make the face makeup of other areas more realistic.
Keywords
AR Makeup; pixel-unit make up; Rendering Map; Interactive Makeup;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Dlib C++Library. http://dlib.net/ (accessed Sep 20, 2020).
2 Z. Cao, G. Hidalgo,T. Simon, S. Wei and Y. Sheikh, "Open-Pose: realtime multi-person 2D pose estimation using Part Affinity Fields", Proceeding of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Hawaii, USA, pp.7291-7299, July 2017.
3 T. Li, R. Qian, C. Dong, S. Liu, Q. Yan, W. Zhu and L. Lin, "BeautyGAN: Instance-level Facial Makeup Transfer with Deep Generative Adversarial Network," Proceedings of ACM Multimedia Conference(MM '18). pp 645-653. Seoul, Republic of Korea, October 2018.
4 P. Aarabi, "Method and system for simulated product evaluation via personalizing advertisements based on portrait images," US Patent 9,275,400 B2, to MODIFACE INC., Patent and Trademark Office, Washington D.C.,, 2016.
5 SNOW application. https://www.snowcorp.com/ko/(accessed Sep 20, 2020)
6 A. Borges and C. Morimoto "A Virtual Makeup Augmented Reality System," Proceeding of 21st Symposium on Virtual and Augmented Reality (SVR), pp. 34-42. Rio de Janeiro, Brazil, 2019.
7 B. Treepong, P. Wibulpolprasert and S. Hasegawa, "The Development of an Augmented Virtuality for Interactive Face Makeup System," Proceeding of Advances in Computer Entertainment Technology (ACE), London, UK, pp. 614-625. 2017.
8 A. Rahman, T. Tran, S. Hossain, "Augmented Rendering of Makeup Features in a Smart Interactive Mirror System for Decision Support in Cosmetic Products Selection." Proceeding of the IEEE/ACM 14th International Symposium on Distributed Simulation and Real Time Applications, Fairfax, Virginia, USA, pp203-206, 2010.
9 R. Bao, H. Yu, S. Li and B. LI, "Automatic makeup based on generative adversarial nets," Proceeding of the International Conference on Internet Multimedia Computing and Service, New York, USA, pp.1-5, 2018.
10 Business Apps & Solutions | PERFECT Application. https://www.perfectcorp.com/business. (accessed Sep 20 2020).
11 S. Kristina, R. Tobias, H. Matthias, T. Thorsten, B. Volker and S. Hans-Peter, "Computer-Suggested Facial Makeup," Computer Graphics Forum, Vol. 30, No.2, pp.485-492, April 2011.   DOI
12 D. Guo and T. Sim, "Digital face makeup by example," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, Miami, USA, pp. 73-79, 2009.
13 A. Nishimura and I. Siio, "iMake: computer-aided eye makeup," Proceedings of the 5th Augmented Human International Conference, Kobe, Japan, pp.1-2, 2014.
14 C. Li, K. Zhou and S. Lin, "Simulating makeup through physics-based manipulation of intrinsic image layers." Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, pp.4621-4629, 2015.
15 A. Bermano, M. Billeter, D. Iwai and A. Grundhofer, "Makeup Lamps: Live Augmentation of Human Faces via Projection," Computer Graphics Forum, Vol.36, No.2, pp.311-323, May 2017.   DOI
16 E. Iwabuchi, M. Nakagawa and I. Siio, "Smart Makeup Mirror: Computer-Augmented Mirror to Aid Makeup Application," Proceeding of Human-Computer Interaction. Interacting in Various Application Domains (HCI), San Diego, CA, USA, pp.495-503, 2009.
17 B. Evangelista, H. Meshkin, H. Kim, A. Aburto, B. Rubinstein and A. Ho, "Realistic AR makeup over diverse skin tones on mobile," Proceeding of SIGGRAPH Asia, Tokyo, Japan, pp.1-2, 2018.
18 B. Treepong, H. Mitake and S. Hasegawa. "Makeup Creativity Enhancement with an Augmented Reality Face Makeup System," Computers in Entertainment, Vol.16, No.4. pp.1-17, November 2018.
19 G. Oztel and S. Kazan, "Virtual Makeup Application Using Image Processing Methods," International Journal of Scientific Engineering and Applied Science, Vol.1, No.5, pp.401-404, August 2015.
20 D. Almeida, P. Guedes, M. Silva, A. Silva, J. Lima and V. Teichrieb, "Interactive Makeup Tutorial Using Face Tracking and Augmented Reality on Mobile Devices," Proceeding of 2015 XVII Symposium on Virtual and Augmented Reality, Sao Paulo, Brazil, pp.220-226, 2015.
21 Z. Ye et al., "Method of virtual makeup achieved by facial tracking," US Patent 9,224,248 B2, to ULSee Inc., Patent and Trademark Office, Washington D.C., 2015.
22 M. Choe et al., "Makeup supporting methods for creating and applying a makeup guide content to makeup user's face on a real-time basis," US Patent 10,083,345 B2 to Myongsu CHOE, Patent and Trademark Office, Washington D.C., 2018.
23 Y. Goto, "Makeup simulation system, makeup simulation apparatus, makeup simulation method, and makeup simulation program," US Patent 8,107,672 B2, to Shiseido Company, Ltd., Patent and Trademark Office, Washington D.C., 2012.
24 C. Cheng et al., "Systems and methods for interactive virtual makeup experience," US Patent 2018/0165855 A1, to Perfect Corp., Patent and Trademark Office, Washington D.C., 2018.