Browse > Article
http://dx.doi.org/10.9717/kmms.2016.19.8.1260

Automatic Extraction Techniques of Topic-relevant Visual Shots Using Realtime Brainwave Responses  

Kim, Yong Ho (Dept. of Mass Communications, College of Humanities and Social Sciences, Pukyong National University)
Kim, Hyun Hee (Dept. of Library and Information Science, College of Humanities, Myongji University)
Publication Information
Journal of Korea Multimedia Society / v.19, no.8, 2016 , pp. 1260-1274 More about this Journal
Abstract
To obtain good summarization algorithms, we need first understand how people summarize videos. 'Semantic gap' refers to the gap between semantics implied in video summarization algorithms and what people actually infer from watching videos. We hypothesized that ERP responses to real time videos will show either N400 effects to topic-irrelevant shots in the 300∼500ms time-range after stimulus on-set or P600 effects to topic-relevant shots in the 500∼700ms time range. We recruited 32 participants in the EEG experiment, asking them to focus on the topic of short videos and to memorize relevant shots to the topic of the video. After analysing real time videos based on the participants' rating information, we obtained the following t-test result, showing N400 effects on PF1, F7, F3, C3, Cz, T7, and FT7 positions on the left and central hemisphere, and P600 effects on PF1, C3, Cz, and FCz on the left and central hemisphere and C4, FC4, P8, and TP8 on the right. A further 3-way MANOVA test with repeated measures of topic-relevance, hemisphere, and electrode positions showed significant interaction effects, implying that the left hemisphere at central, frontal, and pre-frontal positions were sensitive in detecting topic-relevant shots while watching real time videos.
Keywords
Video Summarization; Topic-Relevance; EEG; ERP; N400 Effect; P600 Effect; Visual Abstraction;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 Y. Song, G. Marchionini, and C.Y. Oh, "What are the Most Eye-catching and Ear-catching Features in the Video? Implications for Video Summarization," Proceeding of International Conference of World Wide Web, pp. 911-920. 2010.
2 A.G. Money and H. Agius, "Analysing User Physiological Responses for Affective Video Summarization," Displays, Vol. 30, No. 2, pp. 59-70, 2009.   DOI
3 A.G. Money and H. Agius, "Video Summarization: A Conceptual Framework and Survey," Journal of Visual Communication and Image Representation, Vol. 19, No. 2, pp. 121-143, 2008.   DOI
4 Z. Lu and K. Grauman, "Story-Driven Summarization for Egocentric Video," The IEEE Conference on Computer Vision and Pattern Recognition, pp. 2714-2721, 2013.
5 W. Ren and Y. Zhu, "A Video Summarization Approach Based on Machine Learning," Proceeding of IEEE Xplore Abstract, pp. 450-453, 2008.
6 A. Porselvi and S. Gunasundari, "Survey on Web Page Visual Summarization," Proceeding of International Conference on Information Systems and Computing, pp.26-32, 2008
7 S. Lu, M.R. Jyu, and I. King, "Semantic Video Summarization Using Mutual Reinforcement Principle and Shot Arrangement Patterns," Proceedings of the 11th International Multimedia Modelling Conference, 2005. http://www.cse.cuhk.edu.hk/-lyu/paper_pdf/MMM2005.pdf
8 A.W.M. Smeulders, M. Worring, S. Satini, A. Gupta, and R. Jain, "Content-based Image Retrieval at the End of the Early Years," IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 22, No. 12, pp. 1349-1380, 2000.   DOI
9 P.N. Johnson-Laird, Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness, Harvard University Press, Cambridege, 1983.
10 S. Haramati, M. Soroker, Y. Dudai, and D.A. Levy, "The Posterior Parietal Cortex in Recognition Memory: A Neuropsychological Study," Neuropsychologia, Vol. 46, pp. 1756-1766, 2008.   DOI
11 W.C. West and P.J. Holcomb, "Event-related Potentials during Discourse-level Semantic Integration of Complex Pictures," Cognitive Brain Research, Vol. 13, pp. 363-375, 2002.   DOI
12 W.B. McPherson and P.J. Holcomb, "An Electrophysiological Investigation of Semantic Priming with Pictures of Real Objects," Psychophysiology, Vol. 36, pp. 53-65, 1999.   DOI
13 J. Sassenhagen, M. Schlesewsky, and I. Bornkessel-Schlesewsky, "The P600-as-P3 Hypothesis Revisited: Single-trial Analyses Reveal that the Late EEG Positivity Following Linguistically Deviant Material is Reaction Time Aligned," Brain and Language, Vol. 137, pp. 29-39, 2014.   DOI
14 A. Kim and L. Osterhout, "The Independence of Combinatory Semantic Processing: Evidence from Event-related Potentials," Journal of Memory and Language, Vol. 52, pp. 205-225, 2005.   DOI
15 H. Brouwer, H. Fitz, J. Hoeks, "Getting Real about Semantic Illusions: Rethinking the Functional Role of the P600 in Language Comprehension," Brain Research 1446, 127-143. 2012.   DOI
16 N. Cohn, R. Jackendoff, P.J. Holcomb, and G.R. Kuperberg, "The Grammar of Visual Narrative: Neural Evidence for Constituent Structure in Sequential Image Comprehension," Neurophychologia, Vol. 64, pp. 63-70, 2014.   DOI
17 D.M. Goldenholz, S.P. Ahlfors, M.S. Hamalaainen, D. Sharon, M. Ishitobi, L.M. Vaina, et al., "Mapping the Signal- To-Noise-Ratios of Cortical Sources in Magnetoencephalography and Electroencephalography, HHS Public Access Author Manuscript," Human Brain Mapping, Vol. 30, No. 4, pp. 1077-1086, 2009.   DOI
18 M.J.A. Eugster, T. Ruotsalo, M.M. Spape, I. Kosunen, O. Barral, N. Ravaja, G. Jacucci, and S. Kaski, "Predicting Term-Relevance from Brain Signals," Proceeding of ACM Special Interest Group on Information Retrieval, 2014. https://www.cs.helsinki.fi/u/jacucci/eugster14sigir.pdf
19 S.E. Barrett and M.D. Rugg, "Event-related Potentials and the Semantic Matching of Pictures," Journal of Brain and Cognition, Vol. 14, No. 2, pp. 201-212, 1990.   DOI
20 X. Zhu, A.B. Goldberg, M. Eldawy, C.R. Dyer, and B. Strock, "A Text-to-Picture Synthesis System for Augmenting Communication," Proceeding of Association for the Advancement of Artificual Intelligence, pp. 1590-1595, 2007.
21 M. Allegreti, Y. Moshfeghi, M. Hadjigeorgieva, F.E. Pollick, J.M. Jose, and G. Pasi, "When Relevance Judgement is Happening? An EEG-based Study," Proceeding of ACM Special Interest Group on Information Retrieval, 2015. http://dl.acm.org/citation.cfm?id=2767811
22 J.M. Zacks, N.K. Speer, J.M. Swallow, and C. J. Maley, "The Brain's Cutting Room Floor: Segmentation of Narrative Cinema," Frontiers in Human Neuroscience, Vol. 4, Article 168, pp. 1-15. 2010.   DOI
23 D. Friedman and R. Johnson, "Event-Related Potential(ERP) Studies of Memory Encoding and Retrieval: A Selective Review," Microscopy Research and Technique, Vol. 51, pp. 6-28, 2000.   DOI
24 L. Nyberg, R. Habib, A.R. McIntosh, and E. Tulving, "Reactivation of Encoding-related Brain Activity during Memory Retrieval," Proceedings of the National Academy of Sciences of the United States of America, Vol. 97, No. 20, pp. 11120-11124, 2000.   DOI
25 J.M. Zacks, S. Braver, M.A. Sherian, D.I. Donaldson, A.Z. Snyder, J.M. Ollinger, R.L. Buckner, and M.E. Raichle, "Human Brain Activity Time-locked to Perceptual Event Boundaries," Nature Neuroscience, Vol. 4, pp. 651-655, 2001.   DOI
26 M. Lee, H. Kim, H, Kang, EEG-based Analysis of Auditory Stimulations Generated from Watching Disgust-Eliciting Videos, Journal of Korea Multimedia Society, Vol. 19, No. 4, pp. 756-764 2016.   DOI
27 A. Kok, "Event-related-potential (ERP) Reflections of Mental Resources: A Review and Synthesis," Biological Psychology, Vol. 45, pp. 19-56, 1997.   DOI
28 H.J. Neville, M. Kutas, G. Chesney, and A.L. Schmidt, "Event-Related Brain Potentials during Initual Encoding and Recognition Memory of Congruous and Incongruous Words," Journal of Memory and Language, Vol. 25, pp. 75-92, 1986.   DOI
29 A.D. Wagner, D.L. Schacter, M. Rotte, W. Koutstaal, A. Maril, A.M. Dale, B.R. Rosen, and R.L. Buckner, "Experiences as Predicted by Brain Activity Building," Science, Vol. 281, No. 5380, pp. 1188-1191, 1998.   DOI