Browse > Article
http://dx.doi.org/10.33851/JMIS.2022.9.2.75

No-Reference Image Quality Assessment based on Quality Awareness Feature and Multi-task Training  

Lai, Lijing (School of Software, Nanchang Hangkong University)
Chu, Jun (School of Software, Nanchang Hangkong University)
Leng, Lu (School of Software, Nanchang Hangkong University)
Publication Information
Journal of Multimedia Information System / v.9, no.2, 2022 , pp. 75-86 More about this Journal
Abstract
The existing image quality assessment (IQA) datasets have a small number of samples. Some methods based on transfer learning or data augmentation cannot make good use of image quality-related features. A No Reference (NR)-IQA method based on multi-task training and quality awareness is proposed. First, single or multiple distortion types and levels are imposed on the original image, and different strategies are used to augment different types of distortion datasets. With the idea of weak supervision, we use the Full Reference (FR)-IQA methods to obtain the pseudo-score label of the generated image. Then, we combine the classification information of the distortion type, level, and the information of the image quality score. The ResNet50 network is trained in the pre-train stage on the augmented dataset to obtain more quality-aware pre-training weights. Finally, the fine-tuning stage training is performed on the target IQA dataset using the quality-aware weights to predicate the final prediction score. Various experiments designed on the synthetic distortions and authentic distortions datasets (LIVE, CSIQ, TID2013, LIVEC, KonIQ-10K) prove that the proposed method can utilize the image quality-related features better than the method using only single-task training. The extracted quality-aware features improve the accuracy of the model.
Keywords
Deep Learning; No-Reference Image Quality Assessment; Multiple Task Learning; Score Prediction;
Citations & Related Records
연도 인용수 순위
  • Reference
1 J. Xu, P. Ye, Q. Li, H. Du, Y. Liu, and D. Doermann, "blind image quality assessment based on high order statistics aggregation," IEEE Trans Image Process, vol. 25, no. 9, pp. 4444-4457, 2016.   DOI
2 F. Li, Y. Zhang, and P. C. Cosman, "MMMNet: An end-to-end multi-task deep convolution neural network with multi-scale and multi-hierarchy fusion for blind image quality assessment," IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 12, pp. 4798-4811, 2021.   DOI
3 J. Ma, J. Wu, L. Li, W. Dong, X. Xie, and G. Shi, et al.,"Blind image quality assessment with active inference," IEEE Transactions on Image Processing, vol. 30, pp. 3650-3663, 2021.   DOI
4 J. Wu, J. Ma, F. Liang, W. Dong, G. Shi, and W. Lin, "End-to-end blind image quality prediction with cascaded deep neural network," IEEE Transaction on Image Process, vol. 29, pp. 7414-7426, 2020.   DOI
5 B. Yan, B. Bare, and W. Tan, "Naturalness-aware deep no-reference image quality assessment," IEEE Transactions on Multimedia, vol. 21, no. 10, pp. 2603-2615, Oct. 2019.   DOI
6 H. Zhu, L. Li, J. Wu, W. Dong, and G. Shi, "MetaIQA: Deep meta-learning for no-reference image quality assessment," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2020, pp. 14131-14140.
7 S. Yang, Q. Jiang, W. Lin, and Y. Wang, "SGDNet: An end-to-end saliency-guided deep neural network for no-reference image quality assessment," ACM International Conference on Multimedia Association for Computing Machinery, Nice, France, 2019, pp.1383-1391.
8 J. Gu, H. Cai, H. Chen, X. Ye, J. S. Ren, and C. Dong, "PIPAL: A large-scale image quality assessment dataset for perceptual image restoration," in European Conference on Computer Vision, Springer, Cham, 2020, pp. 633-651.
9 L. Zhang, L. Zhang, and A. C. Bovik, "A feature-enriched completely blind image quality evaluator," in IEEE Transactions on Image Processing, vol. 24, no. 8, pp. 2579-2591, 2015.   DOI
10 K. He, X. Zhang, S. Ren, and J. Sun, "Delving deep into rectifiers: Surpassing human-level performance on Imagenet Classification," Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1026-1034.
11 D. Ghadiyaram and A. C. Bovik, "Massive online crowdsourced study of subjective and objective picture quality," IEEE Transactions on Image Processing, vol. 25, no. 1, pp. 372-387, 2016.   DOI
12 H. Z. Nafchi, A. Shahkolaei, R. Hedjam, and M. Cheriet, "Mean deviation similarity index: Efficient and reliable full-reference image quality evaluator," IEEE Access, vol. 4, pp. 5579-5590, 2016.   DOI
13 W. Xue, L. Zhang, X. Mou, and A. C. Bovik, "Gradient magnitude similarity deviation: A highly efficient perceptual image quality index," IEEE Transactions on Image Processing, vol. 23, no. 2, pp. 684-695, 2014.   DOI
14 N. Ponomarenko, L. Jin, O. Ieremeiev, V. Lukin, K. Egiazarianet, and J. Astola, et al., "Image Database Tid2013: Peculiarities, results and perspectives," Signal Processing: Image Communication, vol. 30, pp. 57-77, 2015.   DOI
15 G. Zhai and X. Min. "perceptual image quality assessment: A survey," Science China. Information Sciences, vol. 63, no. 11, pp.1-52, 2020.
16 S. Li, W. Ren, F. Wang, I. B. Araujo, E. K. Tokuda, and R. H. Junior, et al., "A comprehensive benchmark analysis of single image deraining: Current challenges and future perspectives," International Journal of Computer Vision, vol. 129, no. 4, pp. 1301-1322, 2021.   DOI
17 D. Singh and V. Kumar, "A comprehensive review of computational dehazing techniques," Archives of Computational Methods in Engineering, vol. 26, no. 5, pp. 1395-1413, 2019.   DOI
18 W. Wang, Y. Yang, X. Wang, W, Wang, and J, Li, "Development of convolutional neural network and its application in image classification: A survey, "Optical Engineering, vol. 58, no. 4, p. 040901, 2019.
19 M. Cheon, S. J. Yoon, B. Kang, and J. Lee, "Perceptual image quality assessment with transformers," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, pp. 433-442.
20 X. Liu, J. an De Weijer, and A. D. Bagdanov, "RankIQA: Learning from rankings for no-reference image quality assessment," IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 2017, pp. 1040-1049.
21 W. Zhang, K. Ma, J. Yan, D. Deng, and Z. Wang, "Blind image quality assessment using a deep bilinear convolutional neural network," IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 1, pp. 36-47, 2020.   DOI
22 M. Everingham, L. Vangool, C. K. I. Williams, J. Winn, and A. Zisserman," The PASCAL visual object classes (VOC) challenge," International Journal of Computer Vision, vol. 88, no. 2, pp. 303-338, 2010.   DOI
23 S. Bosse, D. Maniry, K. R. Mller, T. Wiegand, and W. Samek, "Deep neural networks for no-reference and full-reference image quality assessment," IEEE Transactions on Image Processing, vol. 27, no. 1, pp. 206- 219, Jan. 2018.   DOI
24 L. Kang, P. Ye, Y. Li, and D. Doermann, "Convolutional neural networks for no-reference image quality assessment," in IEEE Conference on Computer Vision and Pattern Recognition, Columbus, USA, 2014, pp. 1733-1740.
25 D. Liang, X. Gao, W. Lu, and J. Li, "Deep blind image quality assessment based on multiple instance regression," Neurocomputing, vol. 431, pp. 78-89, 2021.   DOI
26 H. Lin, V. Hosu, and D. Saupe, DeepFL-IQA: Weak Supervision for Deep IQA Feature Learning, http://arxiv.org/abs/2001.08113, 2020.
27 K. Ma, Z. Duanmu, Q. Wu, Z. Wang, H. Yong, and H. Li, et al., "Waterloo exploration database: New challenges for image quality assessment models," IEEE Trans. Image Process, vol. 26, no. 2, pp. 1004-1016, 2017.   DOI
28 J. H. Lee, Y. W. Lee, D. Jun, and B. G. Kim, "Efficient color artifact removal algorithm based on high-efficiency video coding (HEVC) for high-dynamic range video sequences," IEEE Access (IEEE), vol. 8, pp. 64099-64111, 2020.   DOI
29 A. Mittal, G. S. Muralidhar, J. Ghosh, and A. C. Bovik, "Blind image quality assessment without human training using latent quality factors," in IEEE Signal Process Letters, vol. 19, no. 2, pp. 75-78, 2012.   DOI
30 D. Ghadiyaram and A. C. Bovik, "Perceptual quality prediction on authentically distorted images using A bag of features approach," Journal of Vision, vol. 17, no. 1, pp. 32-32, 2017.   DOI
31 H. O. Shahreza, A. Amini, and H. Behroozi, "No-reference image quality assessment using transfer learning," in2018 9th International Symposium on Telecommunications (IST). IEEE, 2018, pp. 637-640.
32 Y. Liu, P. Sun, N. Wergeles, and Y. Shang, "A survey and performance evaluation of deep learning methods for small object detection," Expert Systems with Applications, vol. 172, no. 4, p. 114602, 2021.   DOI
33 G. Ciaparrone, Fl. Sanchez, S. Tabik, L. Troianoc, R. Tagliaferria, and F. Herrerab, "Deep learning in video multi-object tracking: A survey," Neurocomputing, vol. 381, pp. 61-88, 2019.   DOI
34 X. Xie, Y. Zhang, Wu J, G. Shi, and W. Dong, "Bag-ofwords feature representation for blind image quality assessment with local quantized pattern," Neurocomputing, vol. 266, pp. 176-187, 2017.   DOI
35 J. Gu, H. Cai, C. Dong, J. S. Ren, R. Timofte, and Y. Gong, et al., "NTIRE 2021 challenge on perceptual image quality assessment," in Proceedings of the IEEE/ CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 677-690.
36 H. R. Sheikh, M. F. Sabir, and A.C. Bovik, "A statistical evaluation of recent full reference image quality assessment algorithms," IEEE Transactions on Image Processing, vol. 15, no. 11, pp. 3440-3451, 2006.   DOI
37 H. Chen, X. He, L. Qing, Y. Wu, C. Ren, and R. E. Sheriff, et al.," Real-world single image super-resolution: A brief review," Information Fusion, vol. 79, pp. 124-145, 2022.   DOI
38 Z. Wang, E. P. Simoncelli, and A. C. Bovik, "Multiscale structural similarity for image quality assessment," in The Thirty-Seventh Asilomar Conference on Signals, Systems & Computers, CA, USA, 2003, vol. 2, pp. 1398-1402. 2003.
39 L. Zhang, Y. Shen, and H. Li, "VSI: A visual saliency-induced index for perceptual image quality assessment," IEEE Transactions on Image Processing, vol. 23, no. 10, pp. 4270-4281, 2014.   DOI
40 L. Zhang, L. Zhang, X. Mou, and D. Zhang, "FSIM: A feature similarity index for image quality assessment," IEEE Transactions on Image Processing, vol. 20, no. 8, pp. 2378-2386, 2011.   DOI
41 E. C. Larson and D. M. Chandler, "Most apparent distortion: Full-reference image quality assessment and the role of strategy," Journal of Electronic Imaging, vol. 19, no. 1, pp. 011006:1-011006:21, 2010.
42 V. Hosu, H. H. Lin, T. Sziranyi, and D. Saupe, "Koniq10k: An ecologically valid database for deep learning of blind image quality assessment," IEEE Transactions on Image Processing, vol. 29, pp. 4041-4056, 2020.   DOI
43 Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: From error visibility to structural similarity," IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, 2004.   DOI
44 P. Ye, J. Kumar, L. Kang, and D. Doermann, "Unsupervised feature learning framework for no-reference image quality assessment," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Providence, Rhode Island, pp. 1098-1105, 2012.