한국문헌정보학회지 (Journal of the Korean Society for Library and Information Science)

한국문헌정보학회 (Korean Society For Library And Information Science)
권호 표지
DOI QR코드

DOI QR Code

뇌파정보를 활용한 영상물 요약 알고리즘 설계와 평가

Design and Evaluation of Video Summarization Algorithm based on EEG Information

  • 투고 : 2018.10.16
  • 심사 : 2018.11.13
  • 발행 : 2018.11.30
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

본 연구는 비디오 스킴의 자동 생성을 위한 비디오 요약 알고리즘을 제안하고 이를 평가하였다. 제안된 알고리즘은 ERP(Event Related Potentials) 기반의 주제 적합성 모형, MMR(Maximal Marginal Relevance) 기법 및 판별분석기법을 사용하여 구현하였다. 제안한 ERP/MMR 기반 알고리즘을 이용하여 구성한 비디오 스킴의 품질과 유용성을 내재적 및 외재적 평가를 통해서 검증하였다. 내재적 및 외재적 평가에서 ERP/MMR 방법들의 평가 점수들은 각각 경쟁 기준으로 사용한 SBD(Shot Boundary Detection) 방법의 평가 점수 보다 유의미한 차이를 보이며 높게 나왔다. 그러나 이 두 평가에서 ERP/MMR(${\lambda}=0.6$) 방법의 평가 점수와 ERP/MMR(${\lambda}=1.0$) 방법의 평가 점수 간에 통계적으로 유의미한 차이는 없는 것으로 나타났다.

We proposed a video summarization algorithm based on an ERP (Event Related Potentials)-based topic relevance model, a MMR (Maximal Marginal Relevance), and discriminant analysis to generate a semantically meaningful video skim. We then conducted implicit and explicit evaluations to evaluate our proposed ERP/MMR-based method. The results showed that in the implicit and explicit evaluations, the average scores of the ERP / MMR methods were statistically higher than the average score of the SBD (Shot Boundary Detection) method used as a competitive baseline, respectively. However, there was no statistically significant difference between the average score of ERP/MMR (${\lambda}=0.6$) method and that of ERP/MMR (${\lambda}=1.0$) method in both assessments.

키워드

MHJBB6_2018_v52n4_91_f0001.png 이미지

<그림 1> 주제 적합성 모형

MHJBB6_2018_v52n4_91_f0002.png 이미지

<그림 2> 비디오 3의 ERP/MMR(λ=0.6) 점수(1)

MHJBB6_2018_v52n4_91_f0003.png 이미지

<그림 3> 비디오 3의 ERP/MMR(λ=0.6) 점수(2)

MHJBB6_2018_v52n4_91_f0004.png 이미지

<그림 4> 비디오 3에 대한 비디오 스킴들

<표 1> 실험에 사용된 비디오 목록

MHJBB6_2018_v52n4_91_t0001.png 이미지

<표 2> ERP/MMR(λ=0.6), ERP/MMR(λ=1.0) 및 SBD 방법의 내재적 평가 결과

MHJBB6_2018_v52n4_91_t0002.png 이미지

<표 3> ERP/MMR(λ=0.6), ERP/MMR(λ=1.0) 및 SBD 방법의 외재적 평가 결과

MHJBB6_2018_v52n4_91_t0003.png 이미지

참고문헌

  1. Kwon, Jun Soo. 2000. "The Use of Event-related Potentials in the Study of Cognitive Functions." Journal of Cognitive Science, 1(1): 79-98.
  2. Kim, Hyun-Hee and Kim, Yong-Ho. 2016. "Automatic Extraction Techniques of Topic-Relevant Visual Shots using Real Time Brainwave Responses: ERP N400 and P600 Hypotheses Test." Journal of Korea Multimedia Society, 19(8): 1260-1274. https://doi.org/10.9717/kmms.2016.19.8.1260
  3. Chung, Young-Mee. 2012. Research in Information Retrieval. Seoul: Yonsei University Press.
  4. Allegretti, M. et al. 2015. "When Relevance Judgement is Happening?: An EEG-based study." In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 719-722). New York: ACM.
  5. Burmester, J., Spalek, K. and Wartenburger, I. 2014. "Context Updating during Sentence Comprehension: The Effect of Aboutness Topic." Brain and Language, 137: 62-76. https://doi.org/10.1016/j.bandl.2014.08.001
  6. Carbonell, J. and Goldstein, J. 1998. "The Use of MMR, Diversity-based Reranking for Reordering Documents and Producing Summaries." In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 335-336). New York: ACM.
  7. Chen, F., Delannay, D. and De Vleeschouwer, C. 2011. "An Autonomous Framework to Produce and Distribute Personalized Team-sport Video Summaries: A Basketball Case Study." IEEE Transactions on Multimedia, 13(6): 1381-1394. https://doi.org/10.1109/TMM.2011.2166379
  8. Eugster, M. J. et al. 2016. "Natural Brain-information Interfaces: Recommending Information by Relevance Inferred from Human Brain Signals." Scientific Reports, 6(38580), 1-10. https://doi.org/10.1038/s41598-016-0001-8
  9. Evans, W. J., Cui, L. and Starr, A. 1995. "Olfactory Event-related Potentials in Normal Human Subjects: Effects of Age and Gender." Electroencephalography and Clinical Neurophysiology, 95(4): 293-301. https://doi.org/10.1016/0013-4694(95)00055-4
  10. Hu, W. et al. 2011. "A Survey on Visual Content-based Video Indexing and Retrieval." IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 41(6): 797-819. https://doi.org/10.1109/TSMCC.2011.2109710
  11. IBM Research Corp. 2016. Morgan | IBM Creates First Movie Trailer by AI. [online] [cited 2018. 8. 3.]
  12. Kim, H. H. and Kim, Y. H. 2010. "Toward a Conceptual Framework of Key-frame Extraction and Storyboard Display for Video Summarization." Journal of the Association for Information Science and Technology, 61(5): 927-939.
  13. Lin, C. 2004. "ROUGE: A Package for Automatic Evaluation of Summaries." In Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004), Barcelona, Spain.
  14. Luck, S. J. 2014. An Introduction to the Event-related Potential Technique. Cambridge: MIT press.
  15. Mayer, R. E. 2009. Multimedia Learning. New York: Cambridge University Press.
  16. Mehmood, I. et al. 2016. "Divide-and-conquer based Summarization Framework for Extracting Affective Video Content." Neurocomputing, 174: 393-403. https://doi.org/10.1016/j.neucom.2015.05.126
  17. Mishra, R. et al. 2015. "Real time and Non Real time Video Shot Boundary Detection using Dual Tree Complex Wavelet Transform." 2015 International Conference on Industrial Instrumentation and Control (ICIC, pp. 1495-1500). New York: IEEE.
  18. Mostafa, J. and Gwizdka, J. 2016. "Deepening the Role of the User: Neuro-Physiological Evidence as a Basis for Studying and Improving Search." In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (pp. 63-70). New York: ACM.
  19. Mudrik, L. et al. 2014. "Synchronous Contextual Irregularities Affect Early Scene Processing: Replication and Extension." Neuropsychologia, 56: 447-458. https://doi.org/10.1016/j.neuropsychologia.2014.02.020
  20. Nakano, H. et al. 2014. "Electrophysiological Response to Omitted Stimulus in Sentence Processing." NeuroReport, 25(14): 1169-1174. https://doi.org/10.1097/WNR.0000000000000250
  21. Over, P., Smeaton, A. F. and Awad, G. 2008. "The TRECVid 2008 BBC Rushes Summarization Evaluation." In Proceedings of the 2nd ACM TRECVid Video Summarization Workshop (pp. 1-20). New York: ACM.
  22. Schumacher, P. B. and Hung, Y. C. 2012. "Positional Influences on Information Packaging: Insights from Topological Fields in German." Journal of Memory and Language, 67(2): 295-310. https://doi.org/10.1016/j.jml.2012.05.006
  23. Seoane, L. F., Gabler, S. and Blankertz, B. 2015. "Images from the Mind: BCI Image Evolution based on Rapid Serial Visual Presentation of Polygon Primitives." Brain-Computer Interfaces, 2(1): 40-56. https://doi.org/10.1080/2326263X.2015.1060819
  24. Sitnikova, T. et al. 2008. "Two Neurocognitive Mechanisms of Semantic Integration during the Comprehension of Visual Real-world Events." Journal of Cognitive Neuroscience, 20(11): 2037-2057. https://doi.org/10.1162/jocn.2008.20143
  25. Tavassolipour, M., Karimian, M. and Kasaei, S. 2014. "Event Detection and Summarization in Soccer Videos using Bayesian Network and Copula." IEEE Transactions on Circuits and Systems for Video technology, 24(2): 291-304. https://doi.org/10.1109/TCSVT.2013.2243640
  26. Wang, L. and Schumacher, P. B. 2013. "New is not always Costly: Evidence from Online Processing of Topic and Contrast in Japanese." Frontiers in Psychology, 4: 363.
  27. Xu, R. et al. 2014. "Enhanced low-latency Detection of Motor Intention from EEG for Closed-loop Brain-computer Interface Applications." IEEE Transactions on Biomedical Engineering, 61(2): 288-296. https://doi.org/10.1109/TBME.2013.2294203
  28. Yang, J. et al. 2012. "Channel Selection and Classification of Electroencephalogram Signals: An Artificial Neural Network and Genetic Algorithm-based Approach." Artificial Intelligence in Medicine, 55(2): 117-126. https://doi.org/10.1016/j.artmed.2012.02.001
  29. Zhu, X. et al. 2007. "A Text-to-picture Synthesis System for Augmenting Communication." In Proceedings of the 22nd National Conference on Artificial Intelligence-Volume 2 (pp. 1590-1595). Menlo Park, CA: AAAI Press.