Realistic Scenes Reproduction Based on Total Variation

Abstract

In order to completely record all the information of realistic scenes, high dynamic range (HDR) images have been widely used in virtual reality, photography and computer graphics. A simple yet effective tone mapping method based on total variation is proposed so as to reproduce realistic scenes on low dynamic range (LDR) display devices. The structural component and texture component are obtained using total variation model in logarithmic domain. Then, the dynamic range of the structural component is compressed with an adaptive arcsine function. The texture component is processed by Taylor series. Finally, we adjust the saturation component using sigmoid function and restore the color information. Experimental results demonstrate that our method outperforms existing methods in terms of quality and speed.

1. Introduction

Real-world scenes have a wide dynamic range, which may be as high as 10¹⁰:1[1]. High dynamic range (HDR) images record the full dynamic range of realistic scenes, which can provide more image details and better overall appearance than low dynamic range (LDR) images [2]. HDR technology has developed rapidly in the last decade [3]. However, the dynamic range of digital display devices is limited, which is about 102 :1. Conventional digital devices can only display LDR images due to dynamic range mismatch [4,5]. Although HDR displays have been well developed in recent years, LDR displays still have a large consumer market due to low cost. If HDR images are displayed directly on LDR display devices, many image details will be lost. In order to reproduce the HDR images on LDR display devices, HDR images need to be converted to LDR images. Tone mapping operators can complete that conversion process and can preserve image information as much as possible. In the past few years, a lot of tone mapping methods have been presented, but most of them can not produce satisfactory images. Many methods can not maintain good global and local contrast at the same time, which will lead to the loss of visual information. Some algorithms introduce halo artifacts and color distortion. These will result in unsatisfactory tone mapped images.

Total variation (TV) model is widely used in image processing due to the anisotropic diffusion, which can preserves image structure and texture features. TV model is appropriate for image decomposition, which can avoid halo artifacts and preserve visual information as much as possible. In our paper, we propose a simple yet effective tone mapping method for HDR realistic scenes reproduction on LDR display devices. We employ TV model to obtain structural component and texture component of HDR images. An adaptive arcsine function is applied to compress the global dynamic range of the structural component. After that, the texture component is processed by Taylor series. A novel color restoration method for tone mapping is also proposed. Subjective and objective comparisons show that our method produces appealing images.

The organization of this paper is as follows. Section 2 summarizes the related work on tone mapping. Section 3 introduces the proposed method in detail. Section 4 presents subjective and objective comparisons. Finally, section 5 summarizes this paper.

2. Related Work

Tone mapping includes global mapping and local mapping [6]. The global mapping uses the same function to process the whole HDR images. The mapping function can be a linear or nonlinear function. In the past decades, scholars have proposed many global mapping algorithms. Qiu et al. [7] proposed a learning-based method to map HDR images to LDR images. In the literature, the authors consider the mapping process as a quantitative problem and achieved the tone mapping operator by an adaptive conscience learning strategy. Mantiuk et al. [8] introduced a global mapping method based on minimal visible contrast distortion. Jung et al. [9] utilized perceptual quantization to map HDR images to LDR images. This method represents human visual perception by enhancing contrast. Lee et al. [10] developed an asymmetric sigmoid curve for global dynamic range compression and introduced luminance histogram to enhance contrast. Khan et al. [11] used luminance histogram to obtain tone mapped outputs. Above-mentioned global tone mapping operators are fast and achieve good overall dynamic range compression, but they cause loss of local details.

Local mapping can preserve the details of realistic scenes well. Many scholars have conducted in-depth research on local mapping. Kuang et al. [12] developed a new tone mapping method based on color appearance model (iCAM06). HDR images are decomposed into two layers by fast bilateral filter. Tone compression is only applied to the base layer. Then, detail layer is enhanced by photoreceptor light adaptation function. Although this algorithm enhances the visibility of image details, it causes color distortion. Pouli et al.[13] presented a tone mapping method based on progressive histogram reshaping technique and introduced a color transfer method for output images. Fang et al. [14] proposed an improved tone mapping method using hierarchical model. They used the simulated camera response function to compress the dynamic range and enhanced the local details using guided image filter. Paris et al. [15] used the Laplaciian pyramid to decompose HDR images into multiple scales and compressed large-scale variations by defining a point-wise function. Lee et al. [16] employed image decomposition and Retinex theory to process HDR images. They only compressed the base layer. Qiao et al. [17] improved the local contrast using localized gamma correction and utilized weight determined to adjust color saturation. Liang et al. [18] employed a hybrid decomposition method to reduce halo artifacts and over-enhancement in output images. Barai et al. [19] decomposed HDR images based on saliency region information and added two layers back together with constant weights. Gu et al. [20] proposed a multi-scale model using an adaptive filter and reduced the dynamic range of each detail layer. Shibata et al. [21] introduced a tone mapping method based on the intensity-range constraint. This method can preserve more details, but it causes unnatural overall appearance. Ok and Lee [22] compressed local contrast based on adaptive reference values and acquired the output images based on objective quality metrics. Li et al. [23] employed guided filtering and Sigmoid function to compress the dynamic range and achieved desirable results. Rana et al. [24] introduced a deep learning method for tone mapping operator based on generative adversarial network. This method produces halo artifacts and overly saturated colors due to the limited training data. 

Above-mentioned local tone mapping operators can preserve more details in bright as well as dark regions, but they cause halo artifacts and are computationally expensive.

3. Proposed Algorithm

How to better reproduce HDR realistic scenes on LDR display devices. This section introduces a simple yet effective tone mapping method for realistic scenes reproduction. Fig. 1 shows the whole process in detail. First, we transform HDR images from RGB color space to HSV color space. Second, value channel is converted to logarithmic domain using a logarithm function. Third, structural component and texture component are obtained using TV model. Fourth, an arcsine function is used to compress global dynamic range. The texture information is fused by Taylor series. Finally, we adjust the saturation channel and restore color information.

Fig. 1. The overview of our proposed algorithm

3.1 Color Space Conversion

In RGB color space, tone mapping operations can easily lead to color distortion because luminance and chromatic components are correlated. In contrast, luminance and chromatic components are independent in HSV color space, which will facilitate tone mapping operations. Meanwhile, the HSV color space is close to the perceptual characteristics of the human visual system [25]. Thus, HDR images are converted into HSV color space for improving the quality of tone mapped images. First, R，G and B channels are normalized to R' , G' and B' respectively. Then, the conversion process can be expressed through equations (1)-(3) [25].

\(H=\left\{\begin{array}{ll} 0 & \text { if } \max \left(R^{\prime}, G^{\prime}, B^{\prime}\right)=\min \left(R^{\prime}, G^{\prime}, B^{\prime}\right) \\ 60^{\circ} \times \frac{G^{\prime}-B^{\prime}}{\max \left(R^{\prime}, G^{\prime}, B^{\prime}\right)-\min \left(R^{\prime}, G^{\prime}, B^{\prime}\right)} & \text { if } R^{\prime}=\max \left(R^{\prime}, G^{\prime}, B^{\prime}\right) \\ 60^{\circ} \times \frac{B^{\prime}-R^{\prime}}{\max \left(R^{\prime}, G^{\prime}, B^{\prime}\right)-\min \left(R^{\prime}, G^{\prime}, B^{\prime}\right)}+120^{\circ} & \text { if } \mathrm{G}^{\prime}=\max \left(R^{\prime}, G^{\prime}, B^{\prime}\right) \\ 60^{\circ} \times \frac{R^{\prime}-G^{\prime}}{\max \left(R^{\prime}, G^{\prime}, B^{\prime}\right)-\min \left(R^{\prime}, G^{\prime}, B^{\prime}\right)}+240^{\circ} & \text { if } \mathrm{B}^{\prime}=\max \left(R^{\prime}, G^{\prime}, B^{\prime}\right) \end{array}\right.\)       (1)

\(S=\left\{\begin{array}{lr} 0 & \text { if } \max \left(R^{\prime}, G^{\prime}, B^{\prime}\right)=0 \\ \frac{\max \left(R^{\prime}, G^{\prime}, B^{\prime}\right)-\min \left(R^{\prime}, G^{\prime}, B^{\prime}\right)}{\max \left(R^{\prime}, G^{\prime}, B^{\prime}\right)} & \text { otherwise } \end{array}\right.\)       (2)

\(V=\max \left(R^{\prime}, G^{\prime}, B^{\prime}\right)\)       (3)

where R' , G' and B' represent the normalized values of three channels in RGB color space respectively. H, S and V represent hue, saturation and value in HSV space, respectively.

3.2 Structure-Texture Decomposition

The perceived brightness of human visual system approximates logarithm function. Before structure-texture decomposition, we convert the value channel to logarithm domain. The logarithm function can be given as follows.

\(V_{\log }=\frac{\log (1+250 V)}{\log (1+250)}\)       (4)

where V_log is the value channel in logarithm domain,. From the equation (4), it is very obvious that the V_log is in the interval [0,1].

TV model is widely used for image denoising because of its effectiveness in preserving edge information [26]. Due to the anisotropic diffusion, TV model preserves image structure and texture features while removing image noise. In recent years, the TV model has been widely applied in many other image processing fields, such as pattern recognition, image reconstruction and super-resolution. In this paper, TV model based on Rudin-Osher-Fatemi method is developed in tone mapping operator. The structure component is obtained based on TV regularization which can be expressed as follows.

\(I_{s}=\arg \min _{I_{s}}\left(\left\|I_{s}-V_{\log }\right\|_{2}^{2}+\mu\left|\nabla I_{s}\right|\right)\)       (5)

where I_s is the structure component containing large objects in HDR images, µ is the regulation parameter. In this paper, the regulation parameter is set to µ=0.1. \(\|\cdot\|_{2}\) denotes the 2-norm. ∇ denotes the gradient operator. \(\left|\nabla I_{s}\right|\) is given as follows.

\(\left|\nabla I_{s}\right|=\left|\frac{\partial I_{s}}{\partial x}\right|+\left|\frac{\partial I_{s}}{\partial y}\right|\)       (6)

where x and y represent the horizontal and vertical directions respectively.

The equation (5) is solved using half-quadratic splitting scheme [26]. After obtaining the structure component Is , the texture component is obtained by calculating the difference between V_log and I_s . The texture component is given by

\(I_{T}=V_{\log }-I_{s}\)       (7)

where I_T is the texture component containing the fine details (high-frequency).

3.3 Dynamic Range Compression

After aforementioned operation, the dynamic range compression is only applied to the structure component which have a high dynamic range. We employ an arcsine function to compress the dynamic range of HDR images, which reflects human visual characteristics. The arcsine function is described as follows.

\(V_{g}=0.5+\frac{1}{\pi} \arcsin \left[F_{C} \times\left(I_{s}-0.5\right)\right]\)       (8)

where V_g is the global compression output, F_c is a compression factor. Considering that the dynamic range of different scenes varies greatly, the compression factor is defined as

\(F_{c}=0.8+\overline{I_{s}}\)       (9)

where \(\overline{I_{s}}\) is the average of the structure component.

3.4 Texture Processing and Value Channel Recomposition

After the dynamic range compression, I_T needs to be added back to V_g . From the equation (8), the dynamic range compression function can be written as \(f(x)=0.5+\frac{1}{\pi} \arcsin \left[F_{C} \times(x-0.5)\right]\). This function f (⋅) is not linear, thus the texture component cannot be simply added to the luminance component V_g using the same operation. In other words, \(f\left(V_{\log }\right) \neq f\left(I_{s}\right)+f\left(I_{T}\right)\) ,where \(V_{\log }=I_{s}+I_{T}\) . In order to obtain desirable images, we should find an adjustment factor λ to make it approximately satisfy the following condition: \(f\left(V_{\log }\right)=f\left(I_{s}\right)+\lambda \cdot I_{T}\). Since the Taylor series \(f(x) \approx f\left(x_{0}\right)+f^{\prime}\left(x_{0}\right)\left(x-x_{0}\right)\) approximately satisfy the condition, thus we can obtain the adjustment factor \(\lambda=f^{\prime}\left(x_{0}\right)=f^{\prime}\left(I_{s}\right)\) . The value channel recomposition can be expressed as follows.

\(V_{l d r}=f\left(V_{\log }\right) \approx f\left(I_{s}\right)+f^{\prime}\left(I_{s}\right) I_{T}=V_{g}+f^{\prime}\left(I_{s}\right) I_{T}\)       (10)

where V_ldr is the output of the value channel recomposition.

3.5 Color Restoration

After completing the dynamic range compression and value channel recomposition, the relationship between the value and the saturation has changed. To make LDR images look more vivid, the saturation channel needs to be adjusted in HSV space. We employ a sigmoid function to adjust saturation channel because human visual adaptation have sigmoid nonlinear feature and sigmoid function can overcome the loss of naturalness in tone mapped images[27]. The adjustment function of the saturation channel is defined as follows.

\(S_{l d r}=\frac{1}{1+\exp [-3 \times(2 S-1)]}\)       (11)

where S _ldr is the saturation output, S is the saturation of the input HDR image in HSV color space. Hue channel remains unchanged before and after tone mapping. After above processing, the hue component H, S_ldr component and V_ldr component are converted back to RGB color space.

Finally, about 1% of extremely dark or bright pixels are cut for improving image visual effect and reducing noise.

4. Experimental Results and Analysis

We implement and test our algorithm in this part. The experiments are performed on MATLAB (R2016a) environment using a PC with Intel Core i7 (3.5GHz, 64 bit) processor and 16GB RAM. Several HDR images are used in our experiments. We compare the results of our algorithm with other existing results produced by iCAM06 [12], Fang et al. [14], Shibata et al. [21] , Li et al. [23], and Rana et al. [24].

4.1 Subjective Comparison

Figs. 2,3, 4 and 5 show the comparison results between the proposed method and other previous works on the different datasets.

The results produced by iCAM06 have good overall contrast. However, iCAM06 leads to color distortion and the whole image is yellowish [see Figs. 2(a) and 5(a)]. iCAM06 method loses a lot of details in dark regions because the local contrast is low [see Figs. 3(a) and 5(a)]. Fang et al.’s algorithm causes loss of details in both dark regions (e.g., the trunk in Fig. 2(b)) and bright regions (e.g., the scenes outside the window in Fig. 4(b)). The images look unnatural due to low color saturation [see Figs. 4(b) and 5(b)]. Li et al.’s algorithm obtains good global contrast, but it causes color oversaturation [see Fig. 4(c)] and the images lose some visual details in dark regions, for example the roof region of the building in Fig. 3(c) and inside the office in Fig. 4(c). Shibata et al.’s algorithm represents more details. However, the images are over-enhanced and color distortion is serious [see Figs. 3(d) and 5(d)]. The results produced by Rana’s method have good local contrast, but they look unnatural due to overly saturated colors [see Figs. 2(e) and 3(e)]. The method also produces halo artifacts around high bright regions in some scenes [see Figs. 4(e) and 5(e)]. By contrast, the proposed method preserves more visual details in dark as well as bright regions. For example, the trunk is clearly visible (in Fig. 2(f) ) and the lights are also clear (in Fig. 5(f) ). Moreover, the proposed method shows vivid and natural visual appearance [see Figs. 2(f), 3(f) and 4(f)].

Fig. 2. Comparison of experimental output for treeUnil​​​​​​​

Fig. 3. Comparison of experimental output for memorial​​​​​​​

Fig. 4. Comparison of experimental output for small office​​​​​​​

Fig. 5. Comparison of experimental output for rend04​​​​​​​

4.2 Objective Comparison

Subjective evaluation is susceptible to human emotional factors. In order to objectively evaluate the performance of the proposed algorithm, we used two popular metrics to conduct an in-depth evaluation. These two metrics are information entropy and tone mapping image quality index (TMQI) respectively. Information entropy describes the information amount contained in an image. The larger the information entropy, the more the image details. The information entropy is defined as 

\(H=-\sum_{z=0}^{255} p_{z} \log _{2}\left(p_{z}\right)\)       (12)

where p_z is the probability of pixel gray value z .

TMQI [28] is used to comprehensively evaluates the structural information and naturalness of tone mapped results. The range of TMQI is between 0 and 1. The larger the value of TMQI, the better the image quality. TMQI is defined as

\(Q=k S^{\alpha}+(1-k) N^{\beta}\)       (13)

where the parameters are discussed in reference [28].

Many realistic scenes are used for objective evaluation, including bright scenes, dark scenes, indoor scenes and outdoor scenes. The test results of different realistic scenes are shown in Table 1 and Table 2 respectively. In Table 1, the information entropy values of the proposed method are greater than other compared algorithms in most scenes, which show that the proposed method retains more image details. In Table 2, TMQI values of the proposed method are also greater than other compared algorithms in most scenes, which show that our method obtains high quality tone mapped images. According to the test results, we can note that our method outperforms existing methods in most cases.

Table 1. The evaluated results of information entropy​​​​​​​

Table 2. The evaluated results of TMQI​​​​​​

4.3 Computational Efficiency

Our method has a low computation cost. Table 3 shows the comparison results of execution time. All test results are obtained on the MATLAB environment. From Table 3, we can know that the proposed method is faster than the compared algorithms.

Table 3. Comparison of computation efficiency (unit: second)​​​​​​​

5. Conclusion

This paper have presented a high quality tone mapping algorithm for realistic scenes reproduction. In HSV color space, value channel is converted to logarithm domain. We employ TV model to obtain the structural component and texture component. An adaptive arcsine function and Taylor series are used to process the two components respectively. A novel color restoration method for tone mapping is also proposed in our paper. We compare our method with previous works in subjective and objective assessments. Experimental results demonstrate that our method exhibits good visual appearance and preserves more scene details. Meanwhile, the proposed method has low computation cost.