Journal of the Korean Society for Library and Information Science (한국문헌정보학회지)

Korean Society For Library And Information Science (한국문헌정보학회)
권호 표지
DOI QR코드

DOI QR Code

Understanding Topical Relevance of Multimedia based on EEG Techniques

뇌파측정기술(EEG)에 기초한 멀티미디어 자료의 주제 적합성에 관한 연구

  • Received : 2016.07.25
  • Accepted : 2016.08.18
  • Published : 2016.08.30
Download PDF
⟨ Previous Next ⟩

Abstract

This study proposed two topical relevance models, simple and complex models, using EEG/ERP techniques. In the simple model regarding simple search tasks, N300 and P3b components are used. The N300 is specific to the semantic processing of pictures and the P3b reflects mechanisms involved in the decision about whether an external stimulus matches or does not match an internal representation of a specific category. In the complex model regarding complex search tasks, on the other hand, N400 and P600 components are used. The N400 reflects activation of an amodel system that integrates both image-based and conceptual representations into a context, whereas the P600 is related to complex cognitive processes. Our research results can be used as a source to design an EEG-based interactive multimedia system.

본 연구는 뇌파측정기술(EEG/ERP)을 적용하여 멀티미디어 자료의 주제 적합성의 단순 모형과 복합 모형을 제안하였다. 즉, 단순 탐색을 처리할 때의 주제 적합성 판단 과정은 단순 모형으로 기술하고, 복합 탐색을 처리할 때의 주제 적합성 판단 과정은 복합 모형으로 설명한다. 단순 모형은 이미지의 의미 분석을 기술하는 N300과 작업 분류에 대한 판단을 할 때 발생하는 P3b 요소들을 이용하였다. 한편 복합 모형은 이미지 기반 표현과 개념적 표현을 하나의 문맥으로 통합시킬 때 나타나는 N400과 복잡한 인지 과정을 기술하는 P600 요소들을 이용하였다. 제안된 적합성 모형들은 EEG 측정 정보에 기초한 멀티미디어 정보시스템의 상호 작용 인터페이스 설계의 기초 자료로 활용될 수 있을 것으로 생각한다.

Keywords

References

  1. 김현희. 2015. 뇌파측정기술(EEG)과 판별분석을 이용한 영상물의 키프레임 자동 분류 방안 연구. 정보관리학회지, 32(3): 377-396. Kim, Hyun-Hee. 2015. "Toward a Key-Frame Automatic Extraction Method for Video StoryboardSurrogates based on Users' EEG Signals and Discriminant Analysis." Journal of the KoreanSociety for Information Management, 32(3): 377-396. https://doi.org/10.3743/KOSIM.2015.32.3.377
  2. 박명관, 나윤주. 2012. 통사-의미 역학의 신경언어학적 연구. 생성문법연구, 22(1): 157-183. Park, Myuung-Kwan and Na, Yoonjoo. 2012. "A Neurolinguistic Study of Syntax-SemanticsDynamics." Studies in Generative Grammar, 22(1): 157-183.
  3. Allegretti, M. et al. 2015. "When Relevance Judgement Is Happening?: An EEG-based Study." In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2015), August 9-13th, 2015, Santiago, Chile:PUC Extension Center: 719-722.
  4. Baddeley, A. 2007. Working Memory, Thought and Action. Oxford: Oxford University Press.
  5. Barrett, S. and Rugg, M. 1990. "Event-Related Potentials and the Semantic Matching of Pictures." Brain and Cognition, 14(2): 201-212. https://doi.org/10.1016/0278-2626(90)90029-N
  6. Graben, P. B., Gerth, S. and Vasishth, S. 2008. "Towards Dynamical System Models of Language-Related Brain Potentials." Cognitive Neurodynamics, 2(3): 229-55. https://doi.org/10.1007/s11571-008-9041-5
  7. Chen, Z. and Xu, Y. 2005. "User-Oriented Relevance Judgment: A Conceptual Model." In Proceedings of the 38th Hawaii International Conference on System Sciences, January 6th, 2005, Big Island, HI: Hilton Waikoloa Village Resort: 1-11.
  8. Choi, Y. and Rasmussen, E. 2002. Users' Relevance Criteria in Image Retrieval in American History. Information Processing & Management, 38(5): 695-726. https://doi.org/10.1016/S0306-4573(01)00059-0
  9. Cosijna, E. and Ingwersena, P. 2000. "Dimensions of Relevance." Information Processing & Management, 36(4): 533-550. https://doi.org/10.1016/S0306-4573(99)00072-2
  10. Corbetta, M. and Shulman, G. 2002. "Control of Goal-Directed and Stimulus-Driven Attention in the Brain." Nature Reviews Neuroscience, 3(3): 201-215. https://doi.org/10.1038/nrn755
  11. Donchin, E. 1981. "Surprise!...Surprise?." Psychophysiology, 18(5): 493-513. https://doi.org/10.1111/j.1469-8986.1981.tb01815.x
  12. Donchin, E. and Coles, M. G. 1988. "Is the P300 Component a Manifestation of Context Updating." Behavioral and Brain Sciences, 11(3): 357-427. https://doi.org/10.1017/S0140525X00058027
  13. Eugster, M. et al. 2014. "Predicting Term-Relevance from Brain Signals." In Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR 2014). July 6-11th, 2014, Broadbeach, QLD: Gold Coast Convention and Exhibition Centre: 425-434.
  14. Federmeier, K. D. and Laszlo, S. 2009. "Time for Meaning: Electrophysiology Provides Insights into the Dynamics of Representation and Processing in Semantic Memory." In B. H. Ross. Psychology of Learning and Motivation, 51. Burlington: Academic Press, 1-44.
  15. Friederici, A. D. 1995. "The Time Course of Syntactic Activation during Language Processing:A Model based on Neuropsychological and Neurophysiological Data." Brain and Language, 50: 259-281. https://doi.org/10.1006/brln.1995.1048
  16. Harter, S. P. 1992. "Psychological Relevance and Information Science." Journal of the American Society for Information Science and Technology, 43(9): 602-615. https://doi.org/10.1002/(SICI)1097-4571(199210)43:9<602::AID-ASI3>3.0.CO;2-Q
  17. Huang, X. and Soergel, D. 2006. "An Evidence Perspective on Topical Relevance Types and Its Implications for Exploratory and Task-based Retrieval." Information Research, 12(1):281.
  18. Kaan, E. and Swaab, T. 2003. "Repair, Revision, and Complexity in Syntactic Analysis:An Electrophysiological Differentiation." Journal of Cognitive Neuroscience, 15(1): 98-110. https://doi.org/10.1162/089892903321107855
  19. Kim, A. and Sikos, L. 2011. "Conflict and Surrender during Sentence Processing: An ERP Study of Syntax-Semantics Interaction." Brain and Language, 118(1-2): 15-22. https://doi.org/10.1016/j.bandl.2011.03.002
  20. Koelstra, S., Muehl, C. and Patras, I. 2009. "EEG Analysis for Implicit Tagging of Video Data." In Proceedings of the 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, ACII. September 10-12th, 2009, Amsterdam: 1-14.
  21. Kok, A. 1997. "Event-Related-Potential (ERP) Reflections of Mental Resources: A Review and Synthesis." Biological Psychology, 45: 19-56. https://doi.org/10.1016/S0301-0511(96)05221-0
  22. Kok, A. 2001. "On the Utility of P3 Amplitude as a Measure of Processing Capacity." Psychophysiology, 38: 557-577. https://doi.org/10.1017/S0048577201990559
  23. Komaki, D., Hara, T. and Nishio, S. 2012. "How Does Mobile Context Affect People's Web Search Behavior?: A Diary Study of Mobile Information Needs and Search Behaviors." In Proceedings of 2012 IEEE 26th International Conference on Advanced Information Networking and Applications, March 26-29th, 2012, Fukuoka: Fukuoka Institute of Technology: 245-252.
  24. Kuperberg, G. et al. 2007. "The Role of Animacy and Thematic Relationships in Processing Active English Sentences: Evidence from Event-Related Potentials." Brain and Language, 100(3): 223-237. https://doi.org/10.1016/j.bandl.2005.12.006
  25. Kutas, M. and Federmeier, K. D. 2011. "Thirty Years and Counting: Finding Meaning in the N400 Component of the Event-Related Brain Potential (ERP)." Annual Review of Psychology, 62: 621-647. https://doi.org/10.1146/annurev.psych.093008.131123
  26. Kutas, M. and Hillyard, S. A. 1980. "Reading Senseless Sentences: Brain Potentials Reflect Semantic Incongruity." Science, 207: 203-208. https://doi.org/10.1126/science.7350657
  27. Maillard, L. et al. 2011. "From Perception to Recognition Memory: Time Course and Lateralization of Neural Substrates of Word and Abstract Picture Processing." Journal of Cognitive Neuroscience, 23(4): 782-800. https://doi.org/10.1162/jocn.2010.21434
  28. McPherson, W. and Holcomb, P. 1999. "An Electrophysiological Investigation of Semantic Priming with Pictures of Real Objects." Psychophysiology, 36: 53-65. https://doi.org/10.1017/S0048577299971196
  29. Mehravari, A. S. et al. 2015. "Effects of Grammaticality and Morphological Complexity on the P600 Event-Related Potential Component." PLoS ONE, 10(10): 1-16.
  30. Mizzaro, S. 1997. "Relevance: The Whole History." Journal of the American Society for Information Science and Technology, 48(9): 810-832. https://doi.org/10.1002/(SICI)1097-4571(199709)48:9<810::AID-ASI6>3.0.CO;2-U
  31. Moshfeghi, Y. et al. 2013. "Understanding Relevance: An fMRI Study." In Serdyukov, P. et al. Advances in Information Retrieval. Heidelberg: Springer.
  32. Mostafa, J. and Gwizdka, J. 2016. "Deepening the Role of the User: Neuro-Physiological Evidence as a Basis for Studying and Improving Search." In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval. March 13-17th, 2016, North Carolina: Hampton Inn and Suites Chapel Hill-Carrboro: 63-70.
  33. Neville, H. et al. 1986. "Event-Related Brain Potentials during Initial Encoding and Recognition Memory of Congruous and Incongruous Words." Journal of Memory and Language, 25:75-92. https://doi.org/10.1016/0749-596X(86)90022-7
  34. Osterhout, L. and Holcomb, P. 1992. "Event-Related Brain Potentials Elicited by Syntactic Anomaly." Journal of Memory and Language, 31(6): 785-806. https://doi.org/10.1016/0749-596X(92)90039-Z
  35. Polich, J. 2007. "Updating P300: An Integrative Theory of P3a and P3b." Clinical Neurophysiology, 118(10): 2128-2148. https://doi.org/10.1016/j.clinph.2007.04.019
  36. Saracevic, T. 1996. Relevance Reconsidered. In Proceedings of the Second Conference on Conceptions of Library and Information Science (CoLIS2). October 14-17rh, 1996, Copenhagen:Royal School of Librarianship: 201-218.
  37. Saracevic, T. 2007. "Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science: Part II: Nature and Manifestations of Relevance." Journal of the American Society for Information Science and Technology, 58(13): 1915-1933. https://doi.org/10.1002/asi.20682
  38. Singer, G., Danilov, D. and Norbisrath, U. 2012. "Complex Search: Aggregation, Discovery, and Synthesis." Proceedings of the Estonian Academy of Sciences, 61(2): 89-106. https://doi.org/10.3176/proc.2012.2.02
  39. Singer G., Norbisrath, U. and Lewandowski, D. 2013. "Ordinary Search Engine Users Carrying Out Complex Search Tasks." Journal of Information Science, 39(3): 346-358. https://doi.org/10.1177/0165551512466974
  40. Tang, R. and Solomon, P. 1998. "Toward an Understanding of the Dynamics of Relevance Judgment: An Analysis of One Person's Search Behavior." Information Processing and Management, 34(2/3): 237-256. https://doi.org/10.1016/S0306-4573(97)00081-2
  41. van de Meerendonk, N. et al. 2010. "Monitoring in Language Perception: Mild and Strong Conflicts Elicit Different ERP Patterns." Journal of Cognitive Neuroscience, 22(1): 67-82. https://doi.org/10.1162/jocn.2008.21170
  42. Verleger, R. 1988. "Event-Related Potentials and Cognition: A Critique of the Context Updating Hypothesis and an Alternative Interpretation of P3." Behavioral and Brain Sciences, 11: 343-356. https://doi.org/10.1017/S0140525X00058015
  43. Verleger, R., Jaskowski, P. and Wascher, E. 2005. "Evidence for an Integrative Role of P3b in Linking Reaction to Perception." Journal of Psychophysiology, 19: 165-181. https://doi.org/10.1027/0269-8803.19.3.165
  44. Wagner, A. et al. 1998. "Experiences as Predicted by Brain Activity Building." Science, 281(5380): 1188-1191. https://doi.org/10.1126/science.281.5380.1188
  45. West, W. and Holcomb, P. 2002. "Event-Related Potentials during Discourse-Level Semantic Integration of Complex Pictures." Cognitive Brain Research, 13: 363-375. https://doi.org/10.1016/S0926-6410(01)00129-X