Journal of the Korea Computer Graphics Society (한국컴퓨터그래픽스학회논문지)

Korea Computer Graphics Society (한국컴퓨터그래픽스학회)
권호 표지
DOI QR코드

DOI QR Code

Interface Application of a Virtual Assistant Agent in an Immersive Virtual Environment

몰입형 가상환경에서 가상 보조 에이전트의 인터페이스 응용

  • Giri Na (Department of Computer Engineering, Graduate School, Hansung University) ;
  • Jinmo Kim (Department of Computer Engineering, Graduate School, Hansung University)
  • 나기리 (한성대학교 일반대학원 컴퓨터공학과) ;
  • 김진모 (한성대학교 일반대학원 컴퓨터공학과)
  • Received : 2023.10.30
  • Accepted : 2023.12.27
  • Published : 2024.03.01
Download PDF
⟨ Previous Next ⟩

Abstract

In immersive virtual environments including mixed reality (MR) and virtual reality (VR), avatars or agents, which are virtual humans, are being studied and applied in various ways as factors that increase users' social presence. Recently, studies are being conducted to apply generative AI as an agent to improve user learning effects or suggest a collaborative environment in an immersive virtual environment. This study proposes a novel method for interface application of a virtual assistant agent (VAA) using OpenAI's ChatGPT in an immersive virtual environment including VR and MR. The proposed method consists of an information agent that responds to user queries and a control agent that controls virtual objects and environments according to user needs. We set up a development environment that integrates the Unity 3D engine, OpenAI, and packages and development tools for user participation in MR and VR. Additionally, we set up a workflow that leads from voice input to the creation of a question query to an answer query, or a control request query to a control script. Based on this, MR and VR experience environments were produced, and experiments to confirm the performance of VAA were divided into response time of information agent and accuracy of control agent. It was confirmed that the interface application of the proposed VAA can increase efficiency in simple and repetitive tasks along with user-friendly features. We present a novel direction for the interface application of an immersive virtual environment through the proposed VAA and clarify the discovered problems and limitations so far.

본 연구는 혼합현실과 가상현실을 포함하는 몰입형 가상환경에서 OpenAI의 ChatGPT를 활용한 가상 보조 에이전트의 인터페이스 응용에 관한 새로운 방법을 제안한다. 제안하는 응용 방법은 사용자의 질의에 응답하는 정보 에이전트와 사용자의 요구에 맞춰 가상 객체, 환경 등을 제어하는 제어 에이전트로 구성된다. 이를 위해, Unity 3D 엔진, OpenAI, 그리고 가상현실과 혼합현실 사용자 참여를 위한 패키지 및 개발 도구를 통합하는 개발환경을 설정한다. 그리고 음성 입력으로부터 질문 쿼리에서 답변 쿼리, 또는 제어 요구 쿼리에서 제어 스크립트로 생성으로 연결되는 작업 흐름을 설정한다. 이를 기반으로 혼합현실, 가상현실 체험 환경을 직접 제작하고 에이전트의 성능 확인을 위한 실험을 정보 에이전트의 반응 시간, 제어 에이전트의 정확도로 나누어 진행하였다. 결과적으로 제안하는 인터페이스 응용을 통해 사용자 친화적이고 단순하고 반복적인 작업에서의 효율을 높이는데 유용할 수 있음을 확인하였다. 우리는 새롭게 제안하는 인터페이스를 통해 몰입형 가상환경에서 인터페이스로의 응용에 관한 새로운 방향성을 제시하고 발견된 문제점과 현재까지의 한계점을 분명히 밝힌다.

Keywords

Acknowledgement

본 연구는 한성대학교 학술연구비 지원과제임 (김진모, Jinmo Kim).

References

  1. Maloney D, and Freeman G. "Falling asleep together: What makes activities in social virtual reality meaningful to users," In Proceedings of the Annual Symposium on Computer-Human Interaction in Play, pp. 510-521, 2020.
  2. Kim J. "VIVR: Presence of immersive interaction for visual impairment virtual reality," IEEE Access vol. 8, pp. 196151-196159, 2020. https://doi.org/10.1109/ACCESS.2020.3034363
  3. Cho Y, Kang J, Jeon J, Park J, Kim M, and Kim J. "X-person asymmetric interaction in virtual and augmented realities," Computer Animation and Virtual Worlds, vol. 32, no. 5, e1985, 2021.
  4. S. Hong, G. Na, Y. Cho and J. Kim, "A Study on Movement Interface in Mobile Virtual Reality," Journal of the Korea Computer Graphics Society, vol. 27, no. 3, pp. 55-63, 2021. https://doi.org/10.15701/kcgs.2021.27.3.55
  5. Huang A, Knierim P, Chiossi F, Chuang L L, and Welsch R. "Proxemics for human-agent interaction in augmented reality," In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1-13, 2022.
  6. A. von der Putten, N. Kramer, J. Gratch and S. Kang, ""It doesn't matter what you are!" Explaining social effects of agents and avatars," Computers in Human Behavior, vol. 26, issue 6, pp. 1641-1650, 2010. https://doi.org/10.1016/j.chb.2010.06.012
  7. Kyrlitsias C, and Michael-Grigoriou D. "Social interaction with agents and avatars in immersive virtual environments: A survey," Frontiers in Virtual Reality, vol. 2, 2022.
  8. Oh C S, Bailenson J N, and Welch G F. "A systematic review of social presence: Definition, antecedents, and implications," Frontiers in Robotics and AI, vol. 5, 2018.
  9. Pelechano N, Stocker C, Allbeck J, and Badler N., "Being a part of the crowd: Towards validating vr crowds using presence," 7th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 2008), 2008.
  10. Park M, Cho Y, Na G, and Kim J. "Application of virtual avatar using motion capture in immersive virtual environment," International Journal of Human-Computer Interaction, 2023.
  11. Park M. and Kim J. "Application of Immersive Virtual Environment Through Virtual Avatar Based On Rigid-body Tracking," Journal of the Korea Computer Graphics Society, vol. 29, no. 3, pp. 69-77, 2023. https://doi.org/10.15701/kcgs.2023.29.3.69
  12. Narang S, Best A, and Manocha D. "Simulating movement interactions between avatars & agents in virtual worlds using human motion constraints," In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 9-16, 2018.
  13. Holz T, Dragone M, Martin A, Duffy B, and O'Hare G. "Mixed reality agents as museum guides," In 2nd Agent-Based Systems for Human Learning, 2006
  14. Topsakal O, and Topsakal E. "Framework for a foreign language teaching software for children utilizing ar, voicebots and chatgpt(large language models)," The Journal of Cognitive Systems, vol. 7, no. 2, pp. 33-38, 2022. https://doi.org/10.52876/jcs.1227392
  15. Jones B, Xu Y, Hood M A, Kader M S, and Eghbalzadeh H. "Using generative ai to produce situated action recommendations in augmented reality for high-level goals," In GenAICHI, 2023.
  16. Numan N, Giunchi D, Congdon B, and Steed A. "Ubiq-genie: Leveraging external frameworks for enhanced social vr experiences," In 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 497-501, 2023
  17. OpenAI, "OPENAI: Chatgpt," [Internet] Available: https://chat.openai.com/, 2023.
  18. Altundas S. "Openai unity package," [Internet] Available: https://github.com/srcnalt/OpenAI-Unity/, 2023.
  19. Takahashi K. "Aicommand-chatgpt integration with unity editor," [Internet] Available: https://github.com/keijiro/AICommand, 2023.
  20. Microsoft, "MICROSOFT: Mixed reality toolkit," [Internet] Available:https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk2/?view= mrtkunity-2022-05, 2022
  21. Tamulur, "Spiritual chat with chatgpt npcs in virtual reality," [Internet] Available: https://youtu.be/7xA5K7fRmig, 2023.
  22. Lund A.,"Measuring usability with the use questionnaire," Usability Interface, vol. 8, no. 2, pp. 3-6, 2001.