A Tone Mapping Algorithm Based on Multi-scale Decomposition

Abstract

High dynamic range (HDR) images can present the perfect real scene and rich color information. A commonly encountered problem in practical applications is how to well visualize HDR images on standard display devices. In this paper, we propose a multi-scale decomposition method using guided filtering for HDR image tone mapping. In our algorithm, HDR images are directly decomposed into three layers：base layer, coarse scale detail layer and fine detail layer. We propose an effective function to compress the base layer and the coarse scale detail layer. An adaptive function is also proposed for detail adjustment. Experimental results show that the proposed algorithm effectively accomplishes dynamic range compression and maintains good global contrast as well as local contrast. It also presents more image details and keeps high color saturation.

1. Introduction

The dynamic range of the scene is defined as the ratio between the highest and the lowest luminance value. The dynamic range of real scene may be as high as 108:1, but currently available low dynamic range (LDR) capturing device is only 102:1 [1]. HDR images have higher dynamic ranges than LDR images, so as to capture the wide luminance variations in real scenes [2]. HDR images can present the perfect real scene in details and rich color information. With the development of the relative techniques, HDR images have been widely used in many kinds of fields, such as computer vision, virtual reality, photography, visual effects and the video game industry [3,4]. At present, the dynamic range of common display devices is very limited and much lower than HDR image. A commonly encountered problem in practical applications is how to well visualize HDR images on standard display devices.

In order to realize the goal, we have to compress the dynamic range of HDR images. The mapping techniques from the HDR images to the LDR display devices are called tone mapping or tone reproduction [5]. The ultimate goal of tone mapping is to have the same visual perception between displayed image on LDR devices and reality scene. The perceptual match diagram of ideal dynamic range compression is shown in Fig. 1 [6].

Fig. 1.The perceptual match diagram of ideal dynamic range compression

In this paper, we propose a new tone mapping algorithm based on multi-scale decomposition. The paper is organized as follows: Section 2 presents an overview of previous works about tone mapping. Section 3 describes the proposed algorithm in detail. In section 4, we present experimental results and compare them with previous works. Finally, we conclude this paper in section 5.

 

2. Related Work

Aiming to better preserve characteristics of HDR images, many tone mapping operators (TMOs) have been developed over the past few years. TMOs are divided into two types: global tone mapping and local tone mapping. Global tone mapping algorithm uses a mapping function to compress the dynamic range of the whole image. The method is simple and efficient, but it can not preserve fine details. Local TMOs select different operational parameters to compress dynamic range according to different scene, which can maintain good global contrast and local details but are computationally expensive.

Ward et al. [7] proposed linear mapping algorithm. It can not effectively compress dynamic range. Larson et al. [8] proposed an approach using histogram adjustment. It requires a lot of input parameters. Reinhard et al. [9] used automatic dodging-and-burning to complete TMOs. Durand et al. [10] proposed an approach using bilateral filter to decompose the HDR images into base layer and detail layer. The method only compresses the base layer contrast but preserves detail layer. It still suffers from halo artifacts. Yee et al. [11] presented the segmentation based operators. The details can be preserved but the method is only applied to small luminance range scenes. Drago et al. [12] proposed adaptive logarithmic tone mapping to accomplish dynamic range compression. The algorithm achieves good dynamic range compression, but it can not always preserve fine details. Kuang et al. [13] proposed iCAM06 algorithm based on color appearance model and bilateral filter. The algorithm has achieved good image quality, but contrast is not high and images have a color cast. Paris et al. [14] presented a tone mapping approach based on multi-scale decomposition with the Laplacian pyramid. It considerably improves the contrast of the details,but it still suffers from halo artifacts and keeps low color saturation. Chae et al. [15] improved iCAM06 algorithm, but images still have a color cast. Murofushi et al.[16] proposed a tone mapping method based on integer data operation. This algorithm improves computational efficiency. Lee et al. [5] proposed an approach based on sub-band decomposed multi-scale Retinex (SD-MSR). This algorithm can only improve local contrast.

 

3. Proposed Algorithm

In this section, we describe the proposed algorithm in detail. The goal of our algorithm is to present more image details, keep high color saturation and improve tone mapping image quality. The flowchart of proposed algorithm is shown in Fig. 2.

Fig. 2.The flowchart of proposed algorithm

3.1 Extraction of Luminance Channel

The purpose of tone mapping is to compress the dynamic range of HDR image. The luminance channel of HDR image contains information about dynamic range. So we calculate the luminance channel of HDR image. The luminance channel is discussed in [17].The function is described in equation (1).

where L is the luminance value of HDR image, R, G and B are red, green and blue channels of HDR image respectively.

3.2 Image Multi-scale Decomposition

Once the luminance component of HDR image is obtained, it is decomposed by using the edge-preserving filtering. Guided filtering [18] is based on a local linear model, which possesses an edge-preserving smoothing property. Guided filtering can avoid ringing artifacts since it will not blur strong edges in the process of image decomposition. In this paper, the guided filtering is applied for image decomposition.

3.2.1 Guided Filtering

Firstly, we define a guidance image P and an output image Q. The guided filtering assumes that the filtering output Q is a linear transformation of the guidance image P in a local window wj centered at pixel j [18].

where wj is a square window of a radius r, cj and dj are constant linear coefficients in wj. The value of output image Qi in equation (2) will be different when it is computed in different windows. In order to solve this problem, it is an effective strategy to average all the possible values of coefficients cj and dj. The filtering output is calculated as follows [18]:

here, are the average coefficients of all windows, |w| is the number of pixels in local window.

In this paper, Gr,ε(I, P) is used to represent the guided filtering operation, where r and ε decide the filter size and blur degree of the guided filtering respectively.

3.2.2 Multi-scale Decomposition

Traditionally, the edge-preserving filter is used to decompose the HDR images into two layers: base layer and detail layer. The base layer no longer only contains low frequency band, it also has some coarse scale details (high frequency) information such as textures.When we greatly compress the dynamic range of the base layer, it will result in loss of coarse scale details. This will seriously affect final image quality. In order to preserve more visual details, the coarse scale details are progressively extracted from the base layer. Therefore, HDR image is decomposed into three layers: base layer, coarse scale detail layer and fine detail layer. The base layer can retain high-contrast features like edges. The coarse scale detail layer preserves some edges and coarse details, while the fine detail layer can perfectly reserve low-contrast details like textures.

Iteratively applying guided filtering to the output image Q will generate a multi-scale decomposition. While iterating, the filter size r and blur degree parameter ε are increasing, which result in progressive coarsening. The pixel intensity calculations are performed in the logarithm domain because the logarithm of luminance approximates the perceived lightness [19]. The input luminance value is scaled into range [0, 1] before multi-scale decomposition. When the guidance image is identical to the filtering input I, we can calculate the fine detail layer Ifd as follows:

where GFout is the output of the guided filtering, Ifd is the fine detail layer, guidance image is set as P1 = I. In order to get the base layer and the coarse scale detail layer, the guided filtering is applied to GFout again.

where Ibase is the base layer, Icd is the coarse scale detail layer, guidance image is set as P2 = I.

3.3 Tone Compression

Through the above process, HDR image is decomposed into three layers. Tone compression is applied to the base layer and the coarse scale detail layer which have large dynamic range. According to the theory that human visual adaptation have sigmoid nonlinear feature [20], we propose an effective sigmoid function for dynamic range compression. The function is described in equation (8). The input luminance value is scaled into range [0, 1] before tone compression.

where x is the input luminance value, y is the output of tone compression, a and b are the luminance adjustment factor and the contrast adjustment factor respectively.

Fig. 3 shows the compression curve variations with different parameter a when parameter b is set to b = 8. The compression curve moves to right with the increase of a.

Fig. 3.The compression curve variation with different parameter a

An example with different parameter a is given in Fig. 4. It shows that output image becomes dark with the increase of a. If the luminance adjustment factor a is too small, the output image will be very bright. If the luminance factor a is too big, the output image will be very dark. So parameter a controls the brightness of the output image.

Fig. 4.Output image brightness variation with different parameter a when the parameter b is set to b = 8

Fig. 5 shows the compression curve with different b when parameter a is set to a = 1.2. The slope of the curves are gradually increasing along with the increasing of b. If the slope of compression curve is too large, it may cause artifacts enhancement. So it is important to determine the appropriate b. An example with different parameter b is given in Fig. 6. It shows that image maintains high local contrast when contrast adjustment factor b is small, but global contrast is low. On the contrary, image maintains high global contrast when contrast adjustment factor b is big, but local contrast is low. So parameter b controls the contrast of the output image.

Fig. 5.The compression curve variation with different parameter b

Fig. 6.Output image contrast variation with different parameter b when the parameter a is set to a = 1.2.

3.3.1 The Base Layer Compression

The base layer contains large scale variations which have high dynamic range. In order to display the HDR image on standard devices, we apply compression function to compress the dynamic range of the base layer. In order to obtain good local contrast, we set b = 2.2 in this part. The tone compression image of the base layer is shown in equation (9).

where Lb is the luminance value of the base layer , Lbtc is tone compression output of the base layer. The principal characteristic of output image is that the brightness of output image is adaptive adjustment. The computation of the brightness factor a is given in equation (10).

3.3.2 The Coarse Scale Detail Layer Compression

The coarse scale detail layer preserves some edges and coarse details. It still has large dynamic range. In order to improve visual effects, we apply the identical compression function shown in equation (8) to compress dynamic range, but parameter a and b are different from the base layer. In order to obtain good global contrast, we set b = 7.5 in this part. The compression function is given in equation (11).

where Lcdtc is tone compression output of the coarse scale detail layer, Lcd is luminance value of the coarse scale detail layer.

The brightness factor a is an adaptive adjustment parameter. The computation of the brightness factor a is given in equation (12).

3.4 Adaptive Detail Adjustment

When HDR image is reproduced on common display devices, the loss of image details will be severe. In order to reproduce HDR image perfectly, detail adjustment will be very necessary. According to Stevens effect [21], more details are discerned with the increase of image brightness. On the contrary, we can discern fewer details with the decrease of image brightness. Therefore, the detail enhancement is necessary when the image brightness is reduced after dynamic range compression. On the contrary, details attenuation operation is necessary when the image becomes bright after dynamic range compression. If we enhance the details in this case, it will cause false contour and halo artifacts that are common for local tone mapping operation. Therefore, we propose an adaptive detail adjustment function. The function is described in equation (13).

where Detailis the output of detail adjustment , ∆L is luminance difference before and after tone compression. The ∆L is calculated as follows:

∆L is scaled into range [-1, 1].

3.5 Generation of Tone Mapping Image

After processing the base layer, the coarse scale detail layer and fine detail layer, we can obtain the new luminance image Lout. The Lout is given in equation (15).

After obtaining Lout, we restore the image color information using the method described in [17]. The restored image Iout is given in equation (16).

where Lin and Lout are luminance value before and after tone mapping respectively, Rin, Gin and Bin are red, green and blue channels of HDR image respectively. Rout, Gout and Bout are red, green and blue channels of restored image Iout respectively. γ is used to control the color saturation.

 

4. Experimental Results and Analysis

The experiments were performed on MATLAB(R2012a) environment using a computer with Intel Core i5 (3.5GHz, 64 bit) processor and 4GB RAM. We selected different HDR images which had a wide range of content such as indoor and outdoor scenes with different luminance conditions. For the experimental validation of our method, we compared our approach with other representative methods such as Reinhard et al. ’s algorithm [9] , Durand et al. ’s method [10], iCAM06 [13], He et al.’s algorithm[18] and Paris et al.’s method [14].

4.1 Analysis of Free Parameters

The parameters for our proposed algorithm are divided into two groups: parameters for image decomposition and parameters for tone compression. r1, r2, ε1 and ε2 are the parameters for image decomposition and are related to filter size and blur degree of the guided filtering respectively. The output image becomes blur with the increase of filter size and blur degree. In order to obtain the fine detail layer, r1 and1ε should be small. So we set r1 = 3 and ε1 = 0.1 for the first decomposition. In order to progressively extract coarse scale details from the base layer, r2 and ε2 should be larger than the first decomposition. If r2 and ε2 are too large, the filtered output will be over smoothed. The tone mapping results are not desirable . So we use r2 = 15 and ε3 = 0.3 for the second decomposition. The luminance adjustment factor a and contrast adjustment factor b are the parameters for tone compression. These two parameters have been discussed in the section 3.3. Parameter γ is used to control the color saturation. As discussed in reference [17], useful value for γ falls into the range 0.4 ~ 0.6. In our experiments, the number is set to 0.6.

4.2 Subjective Assessment

Many well-known HDR test images are selected for subjective assessment. Some examples of our method results and the results of other five methods are shown in Fig. 7 to Fig. 10.

Fig. 7.Tone mapping results for memorial church image

From Fig. 7 to Fig. 10, we can observe that Reinhard et al.’s algorithm results in overall dark appearance [see Figs. 7(a), 8(a) and 9(a)]. It maintains low local contrast and can not show details in the dark and bright regions such as both sides of the lamp box [see Fig. 8(a)] and top window area of memorial church [see Fig. 7(a)]. So Reinhard et al.’s algorithm results in loss of visual details. Durand et al.’s algorithm can maintain good contrast, but it also results in loss of details in extreme bright region such as the lamp box in Fig. 8(b). It also leads to gradient reversal artifacts [see Fig. 9(b)]. He et al.’s algorithm maintains high global contrast,but it still leads to loss of details such as the lamp box [see Fig. 8(d)] and the ceiling of belgium house [see Fig. 10(d)]. iCAM06 and Paris et al.’s algorithm can not show details in the dark regions of the image such as the left part of the ceiling in Fig. 7. iCAM06 can preserve global details, but image local contrast is not high and images have a color cast. Paris et al.’s algorithm considerably improves the contrast of the details, but it makes a dull impression due to low color saturation. The proposed algorithm effectively accomplishes dynamic range compression and preserves more visual details in both the dim and bright areas. The results of our algorithm look natural and clean globally. They also maintain high global contrast as well as local contrast and keep high color saturation. We can also observe that the images in Fig. 7(f), Fig. 8(f), Fig. 9(f) and Fig. 10(f) have higher definition than others.

Fig. 8Tone mapping results for desk image

Fig. 9.Tone mapping results for rend04 image

Fig. 10.Tone mapping results for belgium house image

Finally, in order to demonstrate that our algorithm works well on a broad range of high dynamic range images, Fig. 11 shows many tone mapping results using our new algorithm. These results demonstrate that our algorithm preserves more visual details, maintains good contrast and looks natural globally.

Fig. 11.Tone mapping results using our algorithm

4.3 Objective Assessment

In order to assess the results, we adopt two objective assessment metrics. The first one is image information entropy [22] . The second is tone mapping image quality index (TMQI) [23]. Image information entropy is an assessment metric which shows the richness of image detail information. The computation of image information entropy is given in equation (17).

where pz is the probability of pixel gray value z.

TMQI combines the structural fidelity measure and the statistical naturalness measure. It is used to evaluate the overall quality of tone mapping images. The computation of TMQI is given in equation (18).

here,0 ≤ k ≤ 1 is used to adjust the relative importance of the two components. α and β determine the sensitivity of the two components. S is the structural fidelity measure, N is the statistical naturalness measure.

where the parameters are discussed in [23].

Table 1 indicates the assessment results of image information entropy for ten test images. From the numerical results, we can note that the proposed algorithm preserves more image detail information than compared operators. Table 2 indicates the assessment results of TMQI. We can note that the proposed algorithm is better than compared algorithms in terms of the tone mapping images quality. From the results of Table 1 and Table 2, we can know that the images reproduced by our algorithm have higher images quality and contain richer details than the compared algorithms.

Table 1.The evaluated results of image information entropy

Table 2.The evaluated results of TMQI

4.4 Computational Efficiency

We addressed a tone mapping algorithm based on guided filtering and an effective compression function. Our algorithm has a linear time complexity O(n), where n is the number of input image pixels. Table 3 shows the comparison of execution time. We implemented our algorithm using Matlab. All experiments were performed on a PC with 3.5GHz CPU and 4GB RAM.

Table 3.Comparison of the execution time (unit: second)

 

5. Conclusion

In this paper, we have presented a tone mapping algorithm to display HDR images based on guided filtering which is used for multi-scale image decomposition. An effective compression function and an adaptive detail adjustment function are proposed for HDR image tone mapping. We compare our algorithm with previous works in subjective assessment and objective assessment. Experimental results demonstrate that the proposed approach is good at compressing the dynamic range and the reproduce images are appealing.