Browse > Article
http://dx.doi.org/10.3745/JIPS.02.0125

Face Sketch Synthesis Based on Local and Nonlocal Similarity Regularization  

Tang, Songze (Dept. of Criminal Science and Technology, Nanjing Forest Police College)
Zhou, Xuhuan (Dept. of Criminal Science and Technology, Nanjing Forest Police College)
Zhou, Nan (Dept. of Criminal Science and Technology, Nanjing Forest Police College)
Sun, Le (School of Computer and Software, Nanjing University of Information Science and Technology)
Wang, Jin (School of Computer & Communication Engineering, Changsha University of Science & Technology)
Publication Information
Journal of Information Processing Systems / v.15, no.6, 2019 , pp. 1449-1461 More about this Journal
Abstract
Face sketch synthesis plays an important role in public security and digital entertainment. In this paper, we present a novel face sketch synthesis method via local similarity and nonlocal similarity regularization terms. The local similarity can overcome the technological bottlenecks of the patch representation scheme in traditional learning-based methods. It improves the quality of synthesized sketches by penalizing the dissimilar training patches (thus have very small weights or are discarded). In addition, taking the redundancy of image patches into account, a global nonlocal similarity regularization is employed to restrain the generation of the noise and maintain primitive facial features during the synthesized process. More robust synthesized results can be obtained. Extensive experiments on the public databases validate the generality, effectiveness, and robustness of the proposed algorithm.
Keywords
Face Sketch Synthesis; Local Similarity; Nonlocal Similarity; Patch Representation;
Citations & Related Records
Times Cited By KSCI : 2  (Citation Analysis)
연도 인용수 순위
1 Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, vol. 521, no. 7553, pp. 436-444, 2015.   DOI
2 P. Isola, J. Y. Zhu, T. Zhou, and A. A. Efros, "Image-to-image translation with conditional adversarial networks," in Proceedings of the IEEE conference on Computer Vision And Pattern Recognition, Honolulu, HI, 2017, pp. 1125-1134.
3 C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, et al., "Photo-realistic single image super-resolution using a generative adversarial network," in Proceedings of the IEEE Conference on Computer Vision And Pattern Recognition, Honolulu, HI, 2017, pp. 4681-4690.
4 C. Dong, C. C. Loy, K. He, and X. Tang, "Image super-resolution using deep convolutional networks," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 295-307, 2015.   DOI
5 L. Zhang, L. Lin, X. Wu, S. Ding, and L. Zhang, "End-to-end photo-sketch generation via fully convolutional representation learning," in Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, Shanghai, China, 2015, pp. 627-634.
6 I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," Advances in Neural Information Processing Systems, vol. 27, pp. 2672-2680, 2014.
7 A. Dosovitskiy and T. Brox, "Generating images with perceptual similarity metrics based on deep networks," Advances in Neural Information Processing Systems, vol. 29, pp. 658-666, 2016.
8 N. Wang, W. Zha, J. Li, and X. Gao, "Back projection: an effective postprocessing method for GAN-based face sketch synthesis," Pattern Recognition Letters, vol. 107, pp. 59-65, 2018.   DOI
9 X. Tang and X. Wang, "Face photo recognition using sketch," in Proceedings of IEEE International Conference on Image Processing, Rochester, NY, 2002.
10 Q. Liu, X. Tang, H. Jin, H. Lu, and S. Ma, "A nonlinear approach for face sketch synthesis and recognition," in Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Diego, CA, 2005, pp. 1005-1010.
11 S. T. Roweis and L. K. Saul, "Nonlinear dimensionality reduction by locally linear embedding," Science, vol. 290, no. 5500, pp. 2323-2326, 2000.   DOI
12 Y. Song, L. Bao, Q. Yang, and M. H. Yang, "Real-time exemplar-based face sketch synthesis," in Computer Vision-ECCV 2014. Cham: Springer, 2014, pp. 800-813.
13 N. Wang, X. Gao, and J. Li, "Random sampling for fast face sketch synthesis," Pattern Recognition, vol. 76, pp. 215-227, 2018.   DOI
14 S. Tang, L. Xiao, P. Liu, L. Huang, N. Zhou, and Y. Xu, "Pansharpening via sparse regression," Optical Engineering, vol. 56, no. 9, article no. 093105, 2017.
15 X. Gao, N. Wang, D. Tao, and X. Li, "Face sketch-photo synthesis and retrieval using sparse representation," IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 8, pp. 1213-1226, 2012.   DOI
16 J. Wright, Y. Ma, J. Mairal, G. Sapiro, T. S. Huang, and S. Yan, "Sparse representation for computer vision and pattern recognition," Proceedings of the IEEE, vol. 98, no. 6, pp. 1031-1044, 2010.   DOI
17 R. Qi, Y. Zhang, and H. Li, "Block sparse signals recovery via block backtracking-based matching pursuit method," Journal of Information Processing Systems, vol. 13, no. 2, pp. 360-369, 2017.   DOI
18 L. Chang, M. Zhou, Y. Han, and X. Deng, "Face sketch synthesis via sparse representation," in Proceedings of the 20th International Conference on Pattern Recognition, Istanbul, Turkey, 2010, pp. 2146-2149.
19 X. Gao, J. Zhong, J. Li, and C. Tian, "Face sketch synthesis algorithm based on E-HMM and selective ensemble," IEEE Transactions on Circuits and Systems for Video Technology, vol. 18, no. 4, pp. 487-496, 2008.   DOI
20 S. Wang, L. Zhang, Y. Liang, and Q. Pan, "Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, 2012, pp. 2216-2223.
21 X. Wang and X. Tang, "Face photo-sketch synthesis and recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 11, pp. 1955-1967, 2008.   DOI
22 C. Li, S. Zhao, K. Xiao, and Y. Wang, "Face recognition based on the combination of enhanced local texture feature and DBN under complex illumination conditions," Journal of Information Processing Systems, vol. 14, no. 1, pp. 191-204, 2018.   DOI
23 H. Zhou, Z. Kuang, and K. Y. K. Wong, "Markov weight fields for face sketch synthesis," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, 2012, pp. 1091-1097.
24 W. Zhang, X. Wang, and X. Tang, "Lighting and pose robust face sketch synthesis," in Computer Vision- ECCV2010. Heidelberg: Springer, 2010, pp. 420-433.
25 C. Peng, X. Gao, N. Wang, D. Tao, X. Li, and J. Li, "Multiple representations-based face sketch-photo synthesis," IEEE Transactions on Neural Networks and Learning Systems, vol. 27, no. 11, pp. 2201-2215, 2016.   DOI
26 A. Buades, B. Coll, and J. M. Morel, "A non-local algorithm for image denoising," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, San Diego, CA, 2005, pp. 60-65.
27 Z. Zha, X. Zhang, Q. Wang, Y. Bai, Y. Chen, L. Tang, and X. Liu, "Group sparsity residual constraint for image denoising with external nonlocal self-similarity prior," Neurocomputing, vol. 275, pp. 2294-2306, 2018.   DOI
28 A. Martinez and R. Benavente, "The AR face database," 1998; http://www.cat.uab.cat/Public/Publications/1998/MaB1998/CVCReport24.pdf.
29 D. Liu, B. Wen, Y. Fan, C. C. Loy, and T. S. Huang, "Non-local recurrent network for image restoration," Advances in Neural Information Processing Systems, vol. 31, pp. 1673-1682, 2018.
30 W. Zhang, X. Wang, and X. Tang, "Coupled information-theoretic encoding for face photo-sketch recognition," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, 2011, pp. 513-520.
31 K. Messer, J. Matas, J. Kittler, J. Luettin, and G. Maitre, "XM2VTSDB: the extended M2VTS database," in Proceedings of the 2nd International Conference on Audio and Video-Based Biometric Person Authentication, Washington, DC, 1999.
32 P. J. Phillips, H. Moon, S. A. Rizvi, and P. J. Rauss, "The FERET evaluation methodology for face-recognition algorithms," IEEE Transactions on Pattern Analysis and Machine intelligence, vol. 22, no. 10, pp. 1090-1104, 2000.   DOI
33 L. Zhang, L. Zhang, X. Mou, and D. Zhang, "FSIM: a feature similarity index for image quality assessment," IEEE Transactions on Image Processing, vol. 20, no. 8, pp. 2378-2386, 2011.   DOI
34 X. Li and X. Cao, "A simple framework for face photo-sketch synthesis," Mathematical Problems in Engineering, vol. 2012, article no. 910719, 2016.
35 R. G. Uhl and N. da Vitoria Lobo, "A framework for recognizing a facial image from a police sketch," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, CA, 1996, pp. 586-593.
36 N. Wang, X. Gao, L. Sun, and J. Li, "Anchored neighborhood index for face sketch synthesis," IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, no. 9, pp. 2154-2163, 2017.   DOI
37 J. Wang, H. Bao, W. Zhou, Q. Peng, and Y. Xu, "Automatic image-based pencil sketch rendering," Journal of Computer Science and Technology, vol. 17, no. 3, pp. 347-355, 2002.   DOI