• Journal of Broadcast Engineering
• /
• v.25 no.5
• /
• pp.665-671
• /
• 2020

MPEG-I is actively working on standardization on the coding of immersive video which provides up to 6 degree of freedom (6DoF) in terms of viewpoint. 3DoF+ video, which provides motion parallax to omnidirectional view of 360 video, renders a view at any desired viewpoint using multiple view videos acquisitioned in a limited 3D space covered with upper body motion at a fixed position. The MPEG-I visual group is developing a test model called TMIV (Test Model for Immersive Video) in the process of development of the standard for 3DoF+ video coding. In the TMIV, the redundancy between a set of input view videos is removed, and several atlases are generated by packing patches including the remaining texture and depth regions into frames as compact as possible, and coded. This paper presents an atlas generation method that removes small-sized blocks in the atlas for more efficient 3DoF+ video coding. The proposed method shows a performance improvement of BD-rate bit savings of 0.7% and 1.4%, respectively, in natural and graphic sequences compared to TMIV.

• ETRI Journal
• /
• v.44 no.1
• /
• pp.38-50
• /
• 2022

This paper presents a study on the use of encoder-derived features in decoder-side depth estimation. The scheme of multiview video encoding does not require the transmission of depth maps (which carry the geometry of a three-dimensional scene) as only a set of input views and their parameters are compressed and packed into the bitstream, with a set of features that could make it easier to estimate geometry in the decoder. The paper proposes novel recursive block splitting for the feature extraction process and evaluates different scenarios of feature-driven decoder-side depth estimation, performed by assessing their influence on the bitrate of metadata, quality of the reconstructed video, and time of depth estimation. As efficient encoding of multiview sequences became one of the main scopes of the video encoding community, the experimental results are based on the "geometry absent" profile from the incoming MPEG Immersive video standard. The results show that the quality of synthesized views using the proposed recursive block splitting outperforms that of the state-of-the-art approach.

• ETRI Journal
• /
• v.44 no.1
• /
• pp.62-72
• /
• 2022

In this study, we propose a multi-GPU-based 8KVR stitching system that operates in real time on both local and cloud machine environments. The proposed system first obtains multiple 4 K video inputs, decodes them, and generates a stitched 8KVR video stream in real time. The generated 8KVR video stream can be downloaded and rendered omnidirectionally in player apps on smartphones, tablets, and head-mounted displays. To speed up processing, we adopt group-of-pictures-based distributed decoding/encoding and buffering with the NV12 format, along with multi-GPU-based parallel processing. Furthermore, we develop several algorithms such as equirectangular projection-based color correction, real-time CG overlay, and object motion-based seam estimation and correction, to improve the stitching quality. From experiments in both local and cloud machine environments, we confirm the feasibility of the proposed 8KVR stitching system with stitching speed of up to 83.7 fps for six-channel and 62.7 fps for eight-channel inputs. In addition, in an 8KVR live streaming test on the 5G MEC/cloud, the proposed system achieves stable performances with 8 K@30 fps in both indoor and outdoor environments, even during motion.

• Electronics and Telecommunications Trends
• /
• v.34 no.6
• /
• pp.156-163
• /
• 2019

As a primitive immersive video technology, a three degrees of freedom (3DoF) $360^{\circ}$ video can currently render viewport images that are dependent on the rotational movements of the viewer. However, rendering a flat $360^{\circ}$ video, that is supporting head rotations only, may generate visual discomfort especially when objects close to the viewer are rendered. 3DoF+ enables head movements for a seated person adding horizontal, vertical, and depth translations. The 3DoF+ $360^{\circ}$ video is positioned between 3DoF and six degrees of freedom, which can realize the motion parallax with relatively simple virtual reality software in head-mounted displays. This article introduces the standardization trends for the 3DoF+ video in the MPEG-I visual group.

• Journal of Broadcast Engineering
• /
• v.27 no.6
• /
• pp.848-859
• /
• 2022

This paper presents a novel 360-degree video streaming system for large-scale immersive displays and its ongoing implementation. Recent VR systems aim to provide a service for a single viewer on HMD. However, the proposed 360-degree video streaming system enables multiple viewers to explore immersive contents on a large-scale immersive display. The proposed 360-degree video streaming system is being developed in 3 research phases, with the final goal of providing 6DoF. Currently, the phase 1: implementation of the 3DoF 360-degree video streaming system prototype is finished. The implemented prototype employs subpicture-based viewport-dependent streaming technique, and it achieved bit-rate saving of about 80% and decoding speed up of 543% compared to the conventional viewport-independent streaming technique. Additionally, this paper demonstrated the implemented prototype on UCSB AlloSphere, the large-scale instrument for immersive media art exhibition.

• Journal of Internet Computing and Services
• /
• v.22 no.2
• /
• pp.51-58
• /
• 2021

Three degrees of freedom plus(3DoF+) and six degrees of freedom(6DoF) system, which supports a user's movements in graphical and natural scene-based virtual reality, requires multiple high-quality and high-resolution videos to provide immersive media. Previous video quality assessment methods are not appropriate for the 3DoF+ and 6DoF system assessment because different types of artifacts occur in these systems which are not shown in the traditional video compression. This paper provides the performance assessments of several quality assessment methods in 3DoF+ system. Furthermore, this paper presents a super metric, which combines multiple quality assessment methods, thereby it showed a higher correlation coefficient with the subjective quality assessment than the previous methods. Experimental results on 3DoF+ immersive video showed 0.4513 gain on correlation coefficient with subjective quality assessment compared to that of peak signal-to-noise ratio(PSNR).

• Proceedings of the IEEK Conference
• /
• 2006.06a
• /
• pp.685-686
• /
• 2006

Recently, various AR-based product design methodologies have been introduced. In this paper, we propose technologies for enhancing robust augmentation and immersive realization of virtual objects. A robust augmentation technology is developed for various lighting conditions and a partial solution is proposed for the hand occlusion problem that occurs when the virtual objects overlay the user' hands. It provides more immersive or natural images to the users. Finally, vibratory haptic cues by page motors as well as button clicking force feedback by modulating pneumatic pressures are proposed while interacting with virtual widgets. Also our system reduces gabs between modeling spaces and user spaces. An immersive game-phone model is selected to demonstrate that the users can control the direction of the car in the racing game by tilting a tangible object with the proposed augmented haptic and robust non-occluded visual feedback. The proposed methodologies will be contributed to the immersive realization of the conventional AR system.

• The Journal of Korean Institute of Communications and Information Sciences
• /
• v.37 no.6A
• /
• pp.497-502
• /
• 2012

In this paper, we present an immersive videoconferencing system that enables gaze correction between users in the internet protocol TV (IPTV) environment. The proposed system synthesizes the gaze corrected images using the depth estimation and the virtual view synthesis algorithms as one of the most important techniques of 3D video system. The conventional processes, however, causes several problems, especially temporal inconsistency of a depth video. This problem leads to flickering artifacts discomforting viewers. Therefore, in order to reduce the temporal inconsistency problem, we exploit the joint bilateral filter which is extended to the temporal domain. In addition, we apply an outlier reduction operation in the temporal domain. From experimental results, we have verified that the proposed system is sufficient to generate the natural gaze-corrected image and realize immersive videoconferencing.

• Journal of IKEEE
• /
• v.24 no.2
• /
• pp.529-535
• /
• 2020

As the main technology of the 4th industrial revolution, immersive 360-degree video contents are drawing attention. The market size of immersive 360-degree video contents worldwide is projected to increase from $6.7 billion in 2018 to approximately $70 billion in 2020. However, most of the immersive 360-degree video contents are distributed through illegal distribution networks such as Webhard and Torrent, and the damage caused by illegal reproduction is increasing. Existing 2D video industry uses copyright filtering technology to prevent such illegal distribution. The technical difficulties dealing with immersive 360-degree videos arise in that they require ultra-high quality pictures and have the characteristics containing images captured by two or more cameras merged in one image, which results in the creation of distortion regions. There are also technical limitations such as an increase in the amount of feature point data due to the ultra-high definition and the processing speed requirement. These consideration makes it difficult to use the same 2D filtering technology for 360-degree videos. To solve this problem, this paper suggests a feature point extraction and identification technique that select object identification areas excluding regions with severe distortion, recognize objects using deep learning technology in the identification areas, extract feature points using the identified object information. Compared with the previously proposed method of extracting feature points using stitching area for immersive contents, the proposed technique shows excellent performance gain.

• Proceedings of the Korean Society of Broadcast Engineers Conference
• /
• 2022.06a
• /
• pp.253-256
• /
• 2022

실사 영상 기반의 메타버스 환경을 구축하기 위한 다수의 카메라를 통한 영상 취득 및 부호화, 전송 기술이 활발히 연구되고 있고, 이를 위해 영상 압축 표준화 단체인 moving picture experts group (MPEG) 에서는 MPEG immersive video (MIV) 표준을 개발하였다. 하지만, 현재 널리 사용되는 가상 현실 영상을 스트리밍 가능한 장비의 연산 능력으로는 MIV 기반 몰입형 영상을 스트리밍 시 복호기 동기화 문제가 발생할 수 있다. 따라서 본 논문은 저사양 및 고사양 장비에서 적응적으로 복호기 개수를 조절 가능한 geometry packing 기법을 MIV 의 참조 소프트웨어인 test model for immersive video (TMIV)에서 구현한다. 제안하는 패킹 기법은 지오메트리 영상을 패킹하여 텍스쳐 영상과 같은 높이를 가지도록 한 후 각각 단일 서브픽쳐 (subpicture) 로 부호화한다. 이후 부호화된 서브픽쳐들에 적응적으로 비트스트림 병합이 적용되어 장비의 복호기 사양에 대응한다.