KIISE Transactions on Computing Practices (정보과학회 컴퓨팅의 실제 논문지)

Korean Institute of Information Scientists and Engineers (한국정보과학회)
권호 표지
DOI QR코드

DOI QR Code

Multi-Modal Wearable Sensor Integration for Daily Activity Pattern Analysis with Gated Multi-Modal Neural Networks

Gated Multi-Modal Neural Networks를 이용한 다중 웨어러블 센서 결합 방법 및 일상 행동 패턴 분석

  • Received : 2016.04.25
  • Accepted : 2016.11.16
  • Published : 2017.02.15
⟨ Previous Next ⟩

Abstract

We propose a new machine learning algorithm which analyzes daily activity patterns of users from multi-modal wearable sensor data. The proposed model learns and extracts activity patterns using input from wearable devices in real-time. Inspired by cue integration of human's property, we constructed gated multi-modal neural networks which integrate wearable sensor input data selectively by using gate modules. For the experiments, sensory data were collected by using multiple wearable devices in restaurant situations. As an experimental result, we first show that the proposed model performs well in terms of prediction accuracy. Then, the possibility to construct a knowledge schema automatically by analyzing the activation patterns in the middle layer of our proposed model is explained.

본고에서는 다중 웨어러블 센서 데이터로부터 사용자의 일상 생활 행동 패턴을 분석할 수 있는 새로운 기계학습 모델을 제안한다. 제안하는 모델은 다중 웨어러블 센서 데이터를 효과적으로 학습하기 위하여 사람이 다중 센서 정보를 처리하는 방법을 적용한 새로운 신경망 모델이다. 제안하는 Gated multi-modal neural netoworks는 계층적 신경망 구조를 가지고 있으며 Gate 모듈을 통해 각 센서 데이터를 선택적으로 결합하여 처리하는 특징을 가진다. 실험을 위해 다중 웨어러블 장치를 착용하고 일상 생활 중 한 가지인 레스토랑에서의 행동 센서 데이터를 수집하였다. 실험 결과로서, 제시하는 모델을 이용하여 실제 웨어러블 센서 데이터를 분석하였을 때 분류 정확도가 비교적 정확하고 빠르게 처리할 수 있음을 확인하였다. 또한 모델의 중간 계층에서의 노드의 활성화 패턴 분석을 통해 자동으로 일상생활 패턴을 추출할 수 있고 이를 이용하여 지식 스키마를 생성할 수 있음을 확인하였다.

Keywords

Acknowledgement

Supported by : 한국연구재단, 정보통신기술진흥센터, 한국산업기술평가관리원, 국방과학연구소

References

  1. B.-T. Zhang, "Ontogenesis of agency in machines: A multidisciplinary review," AAAI 2014 Fall Symposium on The Nature of Humans and Machines: A Multidisciplinary Discourse, Arlington. 2014.
  2. Bruno, Barbara, et al., "Analysis of human behavior recognition algorithms based on acceleration data," Robotics and Automation (ICRA), 2013 IEEE International Conference on. IEEE, 2013.
  3. Jacob, Robert JK, and Keith S. Karn, "Eye tracking in human-computer interaction and usability research: Ready to deliver the promises," Mind 2.3 (2003): 4.
  4. Trommershauser, Julia, Konrad Kording, and Michael S. Landy, eds. Sensory cue integration, Oxford University Press, 2011.
  5. Ngiam, Jiquan, et al., "Multimodal deep learning," Proc. of the 28th international conference on machine learning (ICML-11), 2011.
  6. Srivastava, Nitish, and Ruslan R. Salakhutdinov, "Multimodal learning with deep boltzmann machines," Advances in neural information processing systems, 2012.
  7. K. Ramirez-Amaro, M. Beetz, and G. Cheng, "Transferring skills to humanoid robots by extracting semantic representations from observations of human activities," Artificial Intelligence, 2015.
  8. J. Biagioni and J. Krumm, "Days of our lives: Assessing day similarity from location traces," User Modeling, Adaptation, and Personalization, pp. 89-101, Springer, 2013.
  9. F.-T. Sun, Y.-T. Yeh, H.-T. Cheng, C. Kuo, and M.L. Griss, "Nonparametric discovery of human routines from sensor data," Proc. Pervasive Computing and Communications (PerCom-2014), pp. 11-19, 2014.
  10. C.-Y. Lee, D. H. Kwak, B.-J. Lee, and B.-T Zhang, "Event cognition-based daily activity prediction using wearable sensors," Journal of KIISE, 43(7):781-785, 2016. https://doi.org/10.5626/JOK.2016.43.7.781
  11. Abelson, R., and Schank, R. C, Scripts, plans, goals and understanding, An inquiry into human knowledge structures, Psychology Press, 1977.
  12. Newell, A. Shaw, J. C. and Simon, H. A, "Empirical explorations of the logic theory machine: a case study in heuristic," western joint computer conference: Techniques for reliability, 218-230. ACM, 1957.
  13. Newell, A., Simon, H., et al., "The logic theory machine-a complex information processing system," Information Theory, IRE Transactions on 2(3):61-79, 1956. https://doi.org/10.1109/TIT.1956.1056797
  14. Hinton, Geoffrey, A practical guide to training restricted Boltzmann machines, Momentum 9.1 (2010):926.
  15. Freund, Yoav, and David Haussler, Unsupervised learning of distributions of binary vectors using two layer networks, Computer Research Laboratory [University of California, Santa Cruz], 1994.
  16. Subramanya, Amarnag, et al., Recognizing activities and spatial context using wearable sensors, arXiv preprint arXiv:1206.6869 (2012).
  17. Dargie, Waltenegus, "Analysis of time and frequency domain features of accelerometer measurements," Computer Communications and Networks, 2009. ICCCN 2009. Proceedings of 18th Internatonal Conference on. IEEE, 2009.
  18. Peper, Erik, et al., "Is there more to blood volume pulse than heart rate variability, respiratory sinus arrhythmia, and cardiorespiratory synchrony?," Biofeedback 35.2 (2007).
  19. Fleureau, Julien, Philippe Guillotel, and Izabela Orlac, "Affective benchmarking of movies based on the physiological responses of a real audience," Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on. IEEE, 2013.