Browse > Article
http://dx.doi.org/10.9717/kmms.2020.24.1.021

Context-Awareness Cat Behavior Captioning System  

Chae, Heechan (InfoValleyKorea)
Choi, Yoona (Dept of Computer and Information Science, Korea University)
Lee, Jonguk (Dept of Computer and Convergence Software, Korea University)
Park, Daihee (Dept of Computer and Convergence Software, Korea University)
Chung, Yongwha (Dept of Computer and Convergence Software, Korea University)
Publication Information
Journal of Korea Multimedia Society / v.24, no.1, 2021 , pp. 21-29 More about this Journal
Abstract
With the recent increase in the number of households raising pets, various engineering studies have been underway for pets. The final purpose of this study is to automatically generate situation-sensitive captions that can express implicit intentions based on the behavior and sound of cats by embedding the already mature behavioral detection technology of pets as basic element technology in the video capturing research. As a pilot project to this end, this paper proposes a high-level capturing system using optical-flow, RGB, and sound information of cat videos. That is, the proposed system uses video datasets collected in an actual breeding environment to extract feature vectors from the video and sound, then through hierarchical LSTM encoder and decoder, to identify the cat's behavior and its implicit intentions, and to perform learning to create context-sensitive captions. The performance of the proposed system was verified experimentally by utilizing video data collected in the environment where actual cats are raised.
Keywords
Cat Behavior Monitoring; Cat Behavior Captioning System; Context-Awareness; Deep Learning;
Citations & Related Records
연도 인용수 순위
  • Reference
1 M. Lee, J. Kang, and S. Lim, "Design of YOLO-Based Removable System for Pet Monitoring," Journal of the Korea Institute of Information and Communication Engineering, Vol. 24, No. 1, pp. 22-27, 2020.   DOI
2 X. Wang, Y. Wang, and W. Wang, "Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning," arXiv preprint arXiv:1804.05448, 2018.
3 J. Careira and Z. Andrew, "Quo Vadis, Action Recognition? a New Model and the Kinetics Dataset," The IEEE Conference on Computer Vision and Pattern Recognition, pp. 629-6308, 2017.
4 S. Hershey, S. Chaudhuri, D.P. Elis, J.F. Gemmeke, A. Jansen, R.C. More, M. Plakal, D. Plat, et al., "CNN Architectures for Large-Scale Audio Classification," 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 131-135, 2017.
5 D. Bahdanau, K. Cho, and Y. Bengio, "Neural Machine Translation by Jointly Learning to Align and Translate," arXiv preprint arXiv: 1409.0473, 2014.
6 Q. Zhou and H. Wu, "NLPat IEST 2018: BiLSTM-Attention and LSTM-Attention via Soft Voting in Emotion Classification," Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pp. 189-194, 2018.
7 Z. Huang, W. Yu, and K. Yu, "Bidirectional LSTM-CRF Models for Sequence Tagging," arXiv preprint arXiv:1508.01991, 2015.
8 M. Hong, H. Ahn, O. Atif, J. Lee, D. Park, and Y. Chung, "Field-Applicable Pig Anomaly Detection System Using Vocalization for Embedded Board Implementations," Applied Sciences, Vol. 10, No. 19, 6991, 2020.   DOI
9 J. Seo, H. Ahn, D. Kim, S. Lee, Y. Chung, and D. Park, "EmbeddedPigDet: Fast and Accurate Pig Detection for Embedded Board Implementations," Applied Sciences, Vol. 10, No. 8, 2878, 2020.   DOI
10 Y. Choi, O. Atif, J. Lee, D. Park, and Y. Chung, "Noise-Robust Sound-Event Classification System with Texture Analysis," Symmetry, Vol. 10, No. 9, 402, 2018.   DOI
11 Y. Pandeya, D. Kim, and J. Lee, "Domestic Cat Sound Classification Using Learned Features from Deep Neural Nets," Applied Sciences, Vol. 8, 1949, 2018.   DOI
12 Pet Trend Report 2020. https://blog.opensurvey.co.kr/trendreport/companionanimal-2020 (accessed April 13, 2020).
13 P. Kumpulainen, A. Valldeoriola, S. Somppi, H. Tornqvist, H. vaataja, P. Majaranta, et al., "Dog Activity Classification with Movement Sensor Placed on the Collar," The Fifth International Conference on Animal-Computer Interaction, Vol. 4, pp. 1-6, 2018.
14 Y. Kim, J. Sa, Y, Chung, D. Park, and S. Lee, "Resource-Efficient Pet Dog Sound Events Classification Using LSTM-FCN Based on Time-Series Data," Sensors, Vol. 18, No. 11, 4019, 2018.   DOI
15 S. Lee and I. Kim, "Video Captioning with Visual and Semantic Features," Journal of Information Processing Systems, Vol. 14, No. 6, pp. 1318-1330, 2018.   DOI
16 S. Park, J. Kim, and D. Kim, "A Study on Classification Performance Analysis of Convolutional Neural Network Using Ensemble Learning Algorithm," Journal of Korea Multimedia Society, Vol. 22, No. 6, pp. 665-675, 2019.   DOI
17 P. Pan, Z. Xu, Y. Yang, F. Wu, and Y. Zhuang, "Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning," The IEEE Conference on Computer Vision and Pattern Recognition, pp. 1029-1038, 2016.
18 C. Ladha, N. Hammerla, E. Hughes, P. Olivier, and T. Ploetz, "Dog's Life: Wearable Activity Recognition for Dogs," Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 415-418, 2013.
19 L. Nanni, G. Maguolo, and M. Paci, "Data Augmentation Approaches for Improving Animal Audio Classification," Ecological Informatics, Vol. 57, 101084, 2020.   DOI
20 J. Lee, Y. Choi, Y. Chung, and D. Park, "Sound Noise-Robust Porcine Wasting Diseases Detection and Classification System Using Convolutional Neural Network," Journal of Korean Institute of Information Technology, Vol. 16, No. 5, pp. 1-13, 2018.
21 H. Park, S. Bhattacharjee, P. Deekshitha, C. Kim, and H. Choi, "A Study on Deep Learning Binary Classification of Prostate Pathological Images Using Multiple Image Enhancement Technique," Journal of Korea Multimedia Society, Vol. 23, No. 4, pp. 539-548, 2020.
22 K. Jo, S. Jung, and C. Sim, "A Study of Shiitake Disease and Pest Image Analysis Based on Deep Learning," Journal of Korea Multimedia Society, Vol. 23, No. 1, pp. 50-57, 2020.
23 Raspberry Pi Official Website, https://projects.raspberrypi.org/en/projects/getting-started-with-picamera (accessed October 11, 2020).
24 X. Chen, H. Fang, T.Y. Lin, R. Vedantam, and S. Gupta, "Microsoft COCO Captions: Data Collection and Evaluation Server," arXiv preprint arXiv:1504.00325, 2015.
25 J.B Pam, "Cat Wise: America's Favorite Cat Expert Answers Your Cat Behavior Questions," Penguin Books, 2016
26 J.B Pam, "Think Like a Cat: How to Raise a Wel-Adjusted Cat-Not a Sour Pus," Pengui Books, 2017.