• The Journal of Korean Institute of Communications and Information Sciences
• /
• v.38C no.10
• /
• pp.875-880
• /
• 2013

When we generate 3D video that provides immersive and realistic feeling to users, depth information of the scene is essential. Since the resolution of the depth map captured by a depth sensor is lower than of the color image, we need to upsample the low-resolution depth map for high-resolution 3D video generation. In this paper, we propose a depth upsampling method using depth-discontinuity information. Using the high-resolution color image and the low-resolution depth map, we detect depth-discontinuity regions. Then, we define an energy function for the depth map upsampling and optimize it using the belief propagation method. Experimental results show that the proposed method outperforms other depth upsampling methods in terms of the bad pixel rate.

• Journal of Broadcast Engineering
• /
• v.17 no.6
• /
• pp.933-944
• /
• 2012

In this paper, we propose a new method to convert a low-resolution depth map into its high-resolution one called distance transform-based bilateral upsampling. Since the proposed method controls the spatial domain weighting function based on distance transform values of the depth map, it increases the input depth map resolution while preserving edge sharpness. The proposed method is composed of three main steps: distance transform, spatial weighting control, and image interpolation. Experimental results show that our method outperforms the conventional bilateral upsampling in terms of the quality of output depth maps.

• Journal of Broadcast Engineering
• /
• v.19 no.1
• /
• pp.31-43
• /
• 2014

A depth map is an important component for stereoscopic image generation. Since the depth map acquired from a depth camera has a low resolution, upsamling a low-resolution depth map to a high-resolution one has been studied past decades. Upsampling methods are evaluated by objective evaluation tools such as PSNR, Sharpness Degree, Blur Metric. As well, the subjective quality is compared using virtual views generated by DIBR (depth image based rendering). However, works on the analysis of the relation between depth map upsampling and stereoscopic images are relatively few. In this paper, we investigate the relationship between subjective evaluation of stereoscopic images and objective performance of upsampling methods using cross correlation and linear regression. Experimental results demonstrate that the correlation of edge PSNR and visual fatigue is the highest and the blur metric has lowest correlation. Further, from the linear regression, we found relative weights of objective measurements. Further we introduce a formulae that can estimate 3D performance of conventional or new upsampling methods.

• Journal of Broadcast Engineering
• /
• v.26 no.3
• /
• pp.283-295
• /
• 2021

This paper proposes a method for generating dense depth map using information of color images and depth map generated based on lidar based on self-organizing map. The proposed depth map upsampling method consists of an initial depth prediction step for an area that has not been acquired from LiDAR and an initial depth filtering step. In the initial depth prediction step, stereo matching is performed on two color images to predict an initial depth value. In the depth map filtering step, in order to reduce the error of the predicted initial depth value, a self-organizing map technique is performed on the predicted depth pixel by using the measured depth pixel around the predicted depth pixel. In the process of self-organization map, a weight is determined according to a difference between a distance between a predicted depth pixel and an measured depth pixel and a color value corresponding to each pixel. In this paper, we compared the proposed method with the bilateral filter and k-nearest neighbor widely used as a depth map upsampling method for performance comparison. Compared to the bilateral filter and the k-nearest neighbor, the proposed method reduced by about 6.4% and 8.6% in terms of MAE, and about 10.8% and 14.3% in terms of RMSE.

• Journal of Broadcast Engineering
• /
• v.21 no.6
• /
• pp.957-964
• /
• 2016

Acquisition of reliable depth maps is a critical requirement in many applications such as 3D videos and free-viewpoint TV. Depth information can be obtained from the object directly using physical sensors, such as infrared ray (IR) sensors. Recently, Time-of-Flight (ToF) range camera including KINECT depth camera became popular alternatives for dense depth sensing. Although ToF cameras can capture depth information for object in real time, but are noisy and subject to low resolutions. Recently, filter-based depth up-sampling algorithms such as joint bilateral upsampling (JBU) and noise-aware filter for depth up-sampling (NAFDU) have been proposed to get high quality depth information. However, these methods often lead to texture copying in the upsampled depth map. To overcome this limitation, we formulate a convex optimization problem using higher order regularization for depth map upsampling. We decrease the texture copying problem of the upsampled depth map by using edge weighting term that chosen by the edge information. Experimental results have shown that our scheme produced more reliable depth maps compared with previous methods.

• Journal of the Institute of Electronics Engineers of Korea SP
• /
• v.48 no.6
• /
• pp.8-17
• /
• 2011

Recently, fusion camera systems that consist of depth sensors and color cameras have been widely developed with the advent of a new type of sensor, time-of-flight (TOF) depth sensor. The physical limitation of depth sensors usually generates low resolution images compared to corresponding color images. Therefore, the pre-processing module, such as camera calibration, three dimensional warping, and hole filling, is necessary to generate the high resolution depth map that is placed in the image plane of the color image. However, the result of the pre-processing step is usually inaccurate due to errors from the camera calibration and the depth measurement. Therefore, in this paper, we present a depth map upsampling method robust these errors. First, the confidence of the measured depth value is estimated by the interrelation between the color image and the pre-upsampled depth map. Then, the detailed depth map can be generated by the modified kernel regression method which exclude depth values having low confidence. Our proposed algorithm guarantees the high quality result in the presence of the camera calibration errors. Experimental comparison with other data fusion techniques shows the superiority of our proposed method.

• Journal of Broadcast Engineering
• /
• v.25 no.6
• /
• pp.965-972
• /
• 2020

Depth maps have distance information of objects. They play an important role in organizing 3D information. Color and depth images are often simultaneously obtained. However, depth images have lower resolution than color images due to limitation in hardware technology. Therefore, it is useful to upsample depth maps to have the same resolution as color images. In this paper, we propose a novel method to upsample depth map by shifting the pixel position instead of compensating pixel value. This approach moves the position of the pixel around the edge to the center of the edge, and this process is carried out in several steps to restore blurred depth map. The experimental results show that the proposed method improves both quantitative and visual quality compared to the existing methods.

• Journal of the Korea Institute of Information and Communication Engineering
• /
• v.21 no.7
• /
• pp.1371-1378
• /
• 2017

In this paper, we present an upsampling technique for depth map image using selective bilateral weights and a color weight using laplacian function. These techniques prevent color texture copy problem, which problem appears in existing upsamplers uses bilateral weight. First, we construct a high-resolution image using the bicubic interpolation technique. Next, we detect a color texture region using pixel value differences of depth and color image. If an interpolated pixel belongs to the color texture edge region, we calculate weighting values of spatial and depth in $3{\times}3$ neighboring pixels and compute the cost value to determine the boundary pixel value. Otherwise we use color weight instead of depth weight. Finally, the pixel value having minimum cost is determined as the pixel value of the high-resolution depth image. Simulation results show that the proposed algorithm achieves good performance in terns of PSNR comparison and subjective visual quality.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.12 no.7
• /
• pp.3217-3238
• /
• 2018

In this paper, a depth image up-sampling method is put forward by using pixel classifying and jointed bilateral filtering. By analyzing the edge maps originated from the high-resolution color image and low-resolution depth map respectively, pixels in up-sampled depth maps can be classified into four categories: edge points, edge-neighbor points, texture points and smooth points. First, joint bilateral up-sampling (JBU) method is used to generate an initial up-sampling depth image. Then, for each pixel category, different refinement methods are employed to modify the initial up-sampling depth image. Experimental results show that the proposed algorithm can reduce the blurring artifact with lower bad pixel rate (BPR).

• Proceedings of the Korean Society of Broadcast Engineers Conference
• /
• 2020.11a
• /
• pp.205-207
• /
• 2020

This paper proposes a novel deep learning-based method to upsample a depth map. Most conventional methods estimate high-resolution depth map by modifying pixel value of given depth map using high-resolution color image and low-resolution depth map. However, these methods cause under- or over-shooting problems that restrict performance improvement. To overcome these problems, the proposed method iteratively performs grid warping scheme which shifts pixel values to restore blurred image for estimating high-resolution depth map. Experimental results show that the proposed method improves both quantitative and visual quality compared to the existing method.