Journal of IKEEE (전기전자학회논문지)

Institute of Korean Electrical and Electronics Engineers (한국전기전자학회)
권호 표지
DOI QR코드

DOI QR Code

Recent advances in sketch based image retrieval: a survey

스케치 기반 이미지 검색의 최신 연구 동향

  • Sehong Oh (Dept. of Artificial Intelligence and Data Science, Korea Military Academy) ;
  • Ho-Sik Seok (Dept. of Artificial Intelligence and Data Science, Korea Military Academy)
  • 오세홍 ;
  • 석호식
  • Received : 2024.06.10
  • Accepted : 2024.06.25
  • Published : 2024.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

A sketch is an intuitive means to express information, but compared to actual images, it has the problem of being highly abstract, diverse, and sparse. Recent advances in deep learning models have made it possible to discover features that are common to images and sketches. In this paper, we summarize recent trends in sketch-based image retrieval (SBIR) but it is not limited to SBIR. Besides SBIR, we also introduce sketch-based image recognition and generation studies. Zero-shot learning enables models to recognize categories not encountered during training. Zero-shot SBIR methods are also discussed. Commonly used free-hand sketch datasets are summarized and retrieval performance based on these datasets is reported.

스케치는 직관적으로 정보를 표현할 수 있는 수단이나 실제 이미지와 비교하였을 때 추상화가 심하고 동일한 대상에 대한 스케치 임에도 변화가 매우 크다는 문제점이 있다. 따라서 스케치에 기반한 이미지 인식에 도움이 되는 특성을 표현하기 어려웠는데, 딥뉴럴 모델의 발전은 이미지와 스케치라는 두 도메인에 공통으로 존재하는 특성을 발견할 수 있게 해 주었다. 본 논문에서는 스케치에 기반한 이미지 검색 기술의 최신 연구 동향을 소개한다. 이미지 검색 외에 스케치에 기반한 이미지 인식과 이미지 생성의 연구 동향도 함께 요약하였으며, 스케치 기반 연구에서 주로 활용되는 데이터셋 및 해당 데이터셋에 기반하여 측정한 검색 성능을 포함하였다.

Keywords

Acknowledgement

This study was supported by research fund of Korea Military Academy. (Future Strategy and Technology Research Institute). (RN: 24-AI-Center-03).

References

  1. A. Sain, A. K. Bhunia, Y. Yang, T. Xiang, and Y.-Z. Song, "StyleMeUp: towards style-agnostic sketch-based image retrieval," in Proc. of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021), 2021. DOI: 10.48550/arXiv.2103.15706
  2. S. Koley, A. K. Bhunia, A. Sain, P. N. Chowdhury, T. Xiang, Yi-Zhe Song, "Picture that sketch: photorealistic image generation from abstract sketches," in Proc. of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023), 2023. DOI: 10.48550/arXiv.2303.11162
  3. P. Xu, T. M. Hospedales, Q. Yin, Y.-Z. Song, T. Xiang, and L. Wang, "Deep learning for freehand sketch: a survey," IEEE Trans. Pattern Anal. Mach. Intell., Vol.45, no.1, pp.285-312, 2022. DOI: 10.48550/arXiv.2001.02600
  4. R. Hu and J. Collomosse, "A performance evaluation of gradient field HOG descriptor for sketch based image retrieval," CVIU, vol.117, no.7, pp.790-806, 2013. DOI: 10.1016/j.cviu.2013.02.005
  5. S. K. Yelamarthi, S. K. Reddy, A. Mishra, and A. Mittal, "A zero-shot framework for sketch based image retrieval," in Proc. the 15th European Conference on Computer Vision (ECCV2018), 2018. DOI: 10.48550/arXiv.1807.11724
  6. F. Lin, M. Li, D. Li, T. Hospedales, Y.-Z. Song, Y. Qi, "Zero-shot everything sketch-based image retrieval, and in explainable style," in Proc. of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023), 2023. DOI: 10.48550/arXiv.2303.14348
  7. H. Zhang, S. Liu, C. Zhang, W. Ren, R. Wang, and X. Cao, "SketchNet: sketch classification with web images," in Proc. of 2016 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2016), 2016. DOI: 10.1109/CVPR.2016.125
  8. X. Zhang, Y. Huanga, Q. Zoua, Y. Pei, R. Zhanga, and S. Wang, "A Hybrid convolutional neural network for sketch recognition," Pattern Recognit. Lett., vol.130, pp.73-82, 2020. DOI: 10.1016/j.patrec.2019.01.006
  9. K. Zhang, W. Luo, L. Ma, and H. Li, "Cousin network guided sketch recognition via latent attribute warehouse," in Proc. of the AAAI Conference on Artificial Intelligence (AAAI2019), pp. 9203-9210, 2019. DOI: https://doi.org/10.1609/aaai.v33i01.33019203
  10. G. Jain, S. Chopra, S. Chopra, and A. S. Parihar, "TransSketchNet: attention-based sketch recognition using transformers," in Proc. of the 24th European Conference on Artificial Intelligence (ECAI 2020), 2020.
  11. L. Li, C. Zou, Y. Zheng, and Q. Su, "Sketch-R2CNN: An Attentive Network for Vector Sketch Recognition," IEEE Trans. Vis. Comput. Graph., vol.27, no.9, pp.3745-3754, 2021.
  12. A. Tripathi, R. R. Dani, A. Mishra, and A. Chakraborty, "Sketch-Guided Object Localization in Natural Images," in Proc. of the 16th European Conference on Computer Vision (ECCV2020), 2020. DOI: 10.48550/arXiv.2008.06551
  13. P. N. Chowdhury, A. K. Bhunia, A. Sain, S. Koley, T. Xiang, and Y.-Z. Song, "What can human sketches do for object detection?," in Proc. of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023), 2023. DOI: 10.48550/arXiv.2303.15149
  14. S. Woo, S.-Y. Jeon, J. Park, M. Son, S. Lee, and C. Kim, "Sketch-based video object localization," in Proc. of IEEE/CVF Winter Conference on Applications of Computer Vision (WACV2024), 2024. DOI: 10.48550/arXiv.2304.00450
  15. J. Song, K. Pang, Y.-Z. Song, T. X., and T. M Hospedales, "Learning to Sketch with Shortcut Cycle Consistency," in Proc. of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2018), 2018. DOI: 10.48550/arXiv.1805.00247
  16. J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, "Unpaired image-to-image translation using cycle-consistent adversarial networks," in Proc. of 2017 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2017), 2017. DOI: 10.48550/arXiv.1703.10593
  17. D. Ha and D. Eck, "A neural representation of sketch drawings," in Proc. of the 6th International Conference on Learning Representations (ICLR2018), 2018.
  18. F. Huang and J. F. Canny, "Sketchforme: Composing Sketched Scenes from Text Descriptions for Interactive Applications," in Proc. of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST19), 2019. DOI: 10.48550/arXiv.1904.04399
  19. X. Xing, C. Wang, H. Zhou, J. Zhang, Q. Yu, and D. Xu, "DiffSketcher: text guided vector sketch synthesis through latent diffusion models," in Proc. of the 28th Annual Conference on Neural Information Processing Systems (NeurIPS 2024), 2024. DOI: 10.48550/arXiv.2306.14685
  20. K. Sasaki and T. Ogata, "Adaptive drawing behavior by visuomotor learning using recurrent neural networks," IEEE Trans. Cogn. Dev. Syst., vol.11, no.1, pp.119-128, 2019. DOI: 10.1109/TCDS.2018.2868160
  21. S. Ge, V. Goswami, C. L. Zitnick, and D. Parikh, "Creative sketch generation," in Proc. of International Conference on Learning Representations (ICLR2021), 2021. DOI: 10.48550/arXiv.2011.10039
  22. A. Das, Y. Yang, T. Hospedales, T. Xiang, and Y.-Z. Song, "Cloud2Curve: generation and vectorization of parametric sketches," in Proc. of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021), 2021.
  23. Q. Wang, H. Deng, Y. Qi, D. Li, and Y.-Z. Song, "SketchKnitter: vectorized sketch generation with diffusion models," in Proc. of International Conference on Learning Representations (ICLR2023), 2023.
  24. T. Karras, S. Laine, and T. Aila, "A style-based generator architecture for generative adversarial networks," in Proc. of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019), 2019. DOI: 10.48550/arXiv.1812.04948
  25. K. He, X. Zhang, S. Ren, and Ji. Sun, "Deep residual learning for image recognition," in Proc. of 2016 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2016), 2016.
  26. L. Gao, S.-Y. Chen, K. Jiang, C.-P. Li, Y.-K. Lai, and H. Fu, "SketchFaceNeRF: sketch-based facial generation and editing in neural radiance fields," ACM Trans. Graph., vol.42, no.4, pp.1-17, 2023. DOI: 10.1145/3592100
  27. T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, "Analyzing and improving the image quality of stylegan," in Proc. of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2020), 2020. DOI Bookmark: 10.1109/CVPR42600.2020.00813
  28. E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J Guibas, J. Tremblay, S. Khamis, et al., "Efficient geometry-aware 3D generative adversarial networks," in Proc. of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022), 2022. DOI: 10.48550/arXiv.2112.07945
  29. A. Voynov, K. Aberman, and D. Cohen-Or, "Sketch-guided text-to-image diffusion models," in Proc. of Special Interest Group on Computer Graphics and Interactive Techniques (SIGGRAPH '23), 2023.
  30. A. Mikaeili, O. Perel, M. Safaee, D. Cohen-Or, and A. Mahdavi-Amiri, "SKED: sketch-guided text-based 3D editing," in Proc. 2023 IEEE/CVF International Conference on Computer Vision (ICCV 2023), 2023. DOI: 10.48550/arXiv.2303.10735
  31. Y. Li and W. Li, "A survey of sketch-based image retrieval," Mach. Vis. Appl., vol.29, pp. 1083-1100, 2018.
  32. F. Yang, N. A. Ismail, Y. Y. Pang, V. R. Kebande, A. AL-Dhaqm, AND T. W. Koh, "A systematic literature review of deep learning approaches for sketch-based image retrieval: datasets, metrics, and future directions," IEEE Access, vol.12, pp.14847-14869, 2024. DOI: 10.1109/ACCESS.2024.3357939
  33. A. Creswell and A. A. Bharath, "Adversarial training for sketch retrieval," in Proc. of the 14th European Conference on Computer Vision (ECCV 2016), 2016. DOI: 10.1007/978-3-319-46604-0_55
  34. J. Song, Q. Yu, Y.-Z. Song, T. Xiang, and T. M. Hospedales, "Deep spatial-semantic attention for fine-grained sketch-based image retrieval," in Proc. of 2017 IEEE International Conference on Computer Vision (ICCV2017), 2017. DOI: 10.1109/ICCV.2017.592
  35. L. Liu, F. Shen, Y. Shen, X. Liu, and L. Shao, "Deep Sketch Hashing: Fast Free-Hand Sketch-Based Image Retrieval," in Proc. of 2017 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2017), 2017. DOI: 10.48550/arXiv.1703.05605
  36. L. Guo, J. Liu, Y. Wang, Z. Luo, W. Wen, and H. Lu, "Sketch-based image retrieval using generative adversarial networks," in Proc. of the 25th ACM International Conference on Multimedia (MM17), 2017. DOI: 10.1145/3123266.3127939
  37. T. Portenier, Q. Hu, P. Favaro, and M. Zwicker, "SmartSketcher: sketch-based image retrieval with dynamic semantic re-ranking," in Proc. of the Symposium on Sketch-Based Interfaces and Modeling(SBIM'17), 2017. DOI: 10.1145/3092907.3092910
  38. D. Yu, Y. Liu, Y. Pang, Z. Li, and H. Li, "A multi-layer deep fusion convolutional neural network for sketch based image retrieval," Neuro-computing, vol.296, pp.23-32, 2018. DOI: 10.1016/j.neucom.2018.03.031
  39. T. Bui, L. Ribeiro, M. Ponti, and J. Collomosse, "Sketching out the details: Sketch-based image retrieval using convolutional neural networks with multi-stage regression," Comput. Graph., vol.71, pp.77-87, 2018. DOI: 10.1016/j.cag.2017.12.006
  40. H. Fei, J. Cheng, Z. Yuejie, W. Kangnian, Z. Tao, and F. Weiguo, "Sketch-based image retrieval with deep visual semantic descriptor," Pattern Recognit., vol.76, pp.537-548, 2018. DOI: 10.1016/j.patcog.2017.11.032
  41. H. Zhang, P. She, Y. Liu, J. Gan, X. Cao, and H. Foroosh, "Learning structural representations via dynamic object landmarks discovery for sketch recognition and retrieval," IEEE Trans. Image Process. vol.28, no.9, pp.4486-4499, 2019. DOI: 10.1109/TIP.2019.2910398
  42. F. Huang, J. F. Canny, and J. Nichols, "Swire: sketch-based user interface retrieval," in Proc. of the 2019 CHI Conference on Human Factors in Computing Systems (CHI'19), 2019. DOI: 10.1145/3290605.3300334
  43. M. Bhattarai, D. Oyen, J. Castorena, L. Yang, and B. Wohlberg, "Diagram image retrieval using sketch-based deep learning and transfer learning," in Proc. of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020. DOI: 10.1109/CVPRW50498.2020.00095
  44. C. Bai, J. Chen, Q. Ma, P. Hao, and S. Chen, "Cross-domain representation learning by domain-migration generative adversarial network for sketch based image retrieval," J. Vis. Commun. Image R. vol.71, 102835, 2020. DOI: 10.1016/j.jvcir.2020.102835
  45. Z. Zhang, Y. Zhang, R. Feng, T. Zhang, and W. Fan, "Zero-shot sketch-based image retrieval via graph convolution network," in Proc. of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20), 2020. DOI: 10.1609/aaai.v34i07.6993
  46. A. Fuentes and J. M. Saavedra, "Sketch-QNet: a quadruplet convnet for color sketch-based image retrieval," in Proc. of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021. DOI: 10.48550/arXiv.2104.11130
  47. P. Lu, G. Huang, H. Lin, W. Yang, G. Guo, and Y. Fu, "Domain-aware SE network for sketch-based image retrieval with multiplicative euclidean margin softmax," in Proc. of the 29th ACM International Conference on Multimedia (MM '21), 2021. DOI: 10.48550/arXiv.1812.04275
  48. J. Xu, H. Sun, Q. Qi, J. Wang, C. Ge, L. Zhang, and J. Liao, "DLA-Net for FG-SBIR: dynamic local aligned network for fine-grained sketch-based image retrieval," in Proc. of the 29th ACM International Conference on Multimedia (MM '21), 2021. DOI: 10.1145/3474085.3475705
  49. A. K. Bhunia, S. Koley, A. F. Khilji, A. Sain, P. N. Chowdhury, T. Xiang, and Y.-Z. Song, "Sketching without worrying: noise-tolerant sketch-based image retrieval," in Proc. of 2022 IEEE/ CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022), 2022.
  50. Dai, X. Tang, Y. Liu, S. Xia, and G. Wang, "Multi-granularity association learning for on-the-fly fine-grained sketch-based image retrieval," Knowl.-Based Syst., vol.253, no.109447, 2022. DOI: 10.1016/j.knosys.2022.109447
  51. Z. Ling, Z. Xing, J. Li, and L. Niu, "Multi-level region matching for fine-grained sketch-based image retrieval," in Proc. of the 30th ACM International Conference on Multimedia (MM '22), 2022. DOI:10.1145/3503161.3548147
  52. X. Liu, X. Cheng, H. Chen, H. Yu, and G. Zhao, "Differentiable auxiliary learning for sketch re-identifcation," in Proc. of the Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24), 2024. DOI:10.1609/aaai.v38i4.28165
  53. S. Koley, A. K. Bhunia, A. Sain, P. N. Chowdhury, T. Xiang, and Y.-Z. Song, "How to handle sketch-abstraction in sketch-based image retrieval?," accepted in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024), 2024. DOI: 10.48550/arXiv.2403.07203
  54. Q. Liu, L. Xie, H. Wang, and A. L. Yuille, "Semantic-aware knowledge preservation for zero-shot sketch-based image retrieval," in Proc. of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019), 2019. DOI: 10.48550/arXiv.1904.03208
  55. O. Tursun, S. Denman, S. Sridharan, E. Goan, and C. Fookes, "An efficient framework for zero-shot sketch-based image retrieval," Pattern Recognit., vol.126, 108528, 2022. DOI: 10.1016/j.patcog.2022.108528
  56. U. Chaudhuri, R. Chavan, B. Banerjee, A. Dutta, and Z. Akata, "BDA-SketRet: Bi-level domain adaptation for zero-shot SBIR," Neurocomputing, vol.514, pp.245-255, 2022. DOI: 10.1016/j.neucom.2022.09.104
  57. L. S. F. Ribeiro and M. A. Ponti, "Sketch-an-anchor: sub-epoch fast model adaptation for zero-shot sketch-based image retrieval," arXiv: 2303.16769, 2023.
  58. A. Chaudhuri, A. K. Bhunia, Y.-Z. Song, and A. Dutta, "Data-free sketch-based image retrieval," in Proc. of 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023), 2023. DOI: 10.48550/arXiv.2303.07775
  59. H. Thakur and S. Chattopadhyay, "Active learning for fine-grained sketch-based image retrieval," in Proc. of the 34th British Machine Vision Conference (BMVC2023), 2023. DOI: 10.48550/arXiv.2309.08743
  60. M. Eitz, K. Hildebrand, T. Boubekeur, and M. Alexa, "Sketch-based image retrieval: benchmark and bag-of-features descriptors," IEEE Trans. Vis. Comput. Graphics., vol.17, no.11, pp.1624-1636, 2011. DOI: 10.1109/TVCG.2010.266
  61. M. Eitz, J. Hays, and M. Alexa, "How do humans sketch object?," ACM Trans. Graph., vol. 31, no.4, pp.1-10, 2012.
  62. H. Zhang, S. Liu, C. Zhang, W. Ren, R. Wang, and X. Cao, "SketchNet: sketch classification with web images," n Proc. of 2016 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2016), 2016. DOI: 10.1109/CVPR.2016.125
  63. P. Sangkloy, N. Burnell, C. Ham, and J. Hays, "The sketchy database: learning to retrieve badly drawn bunnies," ACM Trans. Graph., vol.35, no.4, pp.1-12, 2016. DOI: 10.1145/2897824.29259
  64. D. Ha and E. Eck, "A neural representation of sketch drawings," in Proc. of 2018 International Conference on Learning Representations (ICLR 2018), 2018.
  65. Y. Wang, F. Huang, Y. Zhang, R. Feng, T. Zhang, and W. Fan, "Deep cascaded cross-modal correlation leaning for fine-grained sketch-based image retrieval," Pattern Recognit., vol.100, no. 107148, 2020. DOI: 10.1016/j.patcog.2019.107148
  66. Q. Yu, J. Song, Y.-Z. Song, T. Xiang, and T. H. Hospedales, "Fine-grained instance-level sketch-based image retrieval," Int. J. Comput. Vis., vol. 129, no.2, pp.484-500, 2021. DOI: 10.1007/s11263-020-01382-3
  67. L. S. Ferraz Ribeiro, T. Bui, J. Collomosse, and M. Ponti, "Scene designer: a unified model for scene search and synthesis from sketch," in Proc. of 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2021.
  68. X. Zhang, M. Shen, X. Li, and F. Feng, "A deformable CNN-based triplet model for fine-grained sketch-based image retrieval," Pattern Recognit. vol.125, 108508, 2022. DOI: 10.1016/j.patcog.2021.108508