A Variational Framework for Single Image Dehazing Based on Restoration

Abstract

The single image dehazing algorithm in existence can satisfy the demand only for improving either the effectiveness or efficiency. In order to solve the problem, a novel variational framework for single image dehazing based on restoration is proposed. Firstly, the initial atmospheric scattering model is transformed to meet the kimmel's Retinex variational model. Then, the green light component of image is considered as an input of the variational framework, which is generated by the sensitivity of green wavelength. Finally, the atmospheric transmission map is achieved by multi-resolution pyramid reduction to improve the visual effect of the results. Experimental results demonstrate that the proposed method can remove haze effectively with less memory consumption.

1. Introduction

As one of the most important topics and basic issues in image processing, single image dehazing aims at two aspects. One is creating visually-pleasing images for human visual perception, and the other is improving the interpretability of information in images for computer vision serving and for the pre-processing tasks. Thus, advanced techniques for single image dehazing are in urgent need. The existing papers can be divided into two categories.

One scheme based on image enhancement technique aims at improving the visual effect of image directly, such as gamma correction [1], histogram equalization [2], and Retinex [3]. This scheme is fast and simple but has strong pertinence and can hardly adjust all image characteristics to a proper range simultaneously, according to human vision system. The other one is based on image restoration technique. Modeling around atmospheric transmission and environmental luminance make it possible to solve the problem caused by the atmospheric scattering model which has the ill-posedness, with the strong prior or assumption. In terms of the solving methods, the second scheme can be further divided into two groups: one is based on the statistical prior and the other on the objective assumption.

The first group is originated from CVPR 2009, when He [4]’s Dark Channel Prior (DCP) achieved the best paper in the conference and made significant progresses in single image haze removal. His method does not rely on significant variance on transmission or surface shading in the input image and the output image is less affected by halos than before. It works for most outdoor scenes, except when the object is inherently similar to the airlight over a large location and no shadow is cast on the object. Then, Codruta [5] produces a semi-inverse of the image in 2010, based on the analysis of hue disparity between the original image and its semi-inverse, which improves the haze removal performance greatly. This method proposes a simple estimation of the airlight constant and the transmission map, but may cause the phenomenon of excessive enhancement. Therefore, it’s hard to find the appropriate statistical prior for the former; the other group is reconsidered in recent years. Carr [6] assumes that objects which appear towards the top of the image are usually further away, based on a projection model from 3D to 2D. This soft constraint is compatible with the alpha-expansion optimization technique and can be used to improve the robustness of any single image dehazing technique, but can’t adapt to the dynamic range of human vision system. In 2011, Gibson [7] assumes that the RGB channels in haze-free image are based on the elliptical distribution. This approach has a visually compelling result but may not performwell on heavy hazy images and tend to be fail in prospect.

By analyzing the recent dehazing algorithms based on image restoration [8], we found that most algorithms only consider the statistical prior or objective assumption of degraded image separately; however, settling of the atmospheric scattering model with prior and assumption simultaneously is more important, and more powerful.

In this paper, we propose a novel prior “sensitivity of green wavelength”, and use an important assumption “variational framework” for our approach at the same time. The sensitivity of green wavelength is based on the statistics of hazy outdoor images. We find that, in the green light component of image of hazy outdoor images, there are more details and distant objects than others in the results. Based on the variational framework, it shows that the transmission map estimation problem can be formulated as an efficient multi-resolution algorithm, which is proposed to exploit the spatial correlation. Combining the atmospheric scattering model and the Retinex algorithm, we can produce a good transmission map directly and recover a high-quality haze-free image.

 

2. Related Work

In this section, we will briefly summarize the atmospheric scattering model and the Retinex algorithm, which are the foundation of our approach. In order to find out the applicable model with the most potential for real-time processing without sacrificing too much of the image quality, we draw attention to the models in existence. Among these, the McCartney’s model is widely used to describe the formation of a degraded image, which can be showed as follows [9]:

where I(x, y) denotes the observed degraded image, J(x, y) denotes the scene radiance, which represents original appearance of image, A denotes the global atmospheric light, which is mostly recognized as the pixel with the highest intensity in the image, t(x, y) is the atmospheric transmission map. Then the problem is simplified as how to estimate the latent image J(x, y) from the hazy input I(x, y) when no additional information about depth and airlight is given, as it is an abnormal equations.

As the computational model, the Retinex algorithm considers that visual perception by human eyes does not depend on the light reflected by the object, but correlates with integrated reflectance. The formation of an input image I can be decomposed into two parts: the reflectance image R and the illumination image L, such that at each pixel I(x, y) =R(x, y) × L(x, y). The benefits of such decomposition include the ability to remove illumination effects of back/front air-lighting, and lead to improve the brightness, contrast and high dynamic range of image. Retinex algorithms can convert the three components to logarithmic domain, with the multiplication changing into addition, which is: i=l+ r; i=log(I), l=log(L), r=log(R).

Recovering the illumination from a given image is known to be a mathematically ill-posed problem. In order to alleviate this problem, additional assumptions on the unknowns are required. Kimmel’s Retinex variational framework to the ill-posed problem is introduced and compared to other state-of-the-art methods, which relies on the following assumptions about the illumination image [10]:

(1) The illumination is spatially smooth.

(2) Since R is restricted to the unit interval, we can add the constraint L ≥ S. As the log function is monotone, we also have l ≥ s.

(3) The illumination image is close to the input image. It minimizes a penalty term of the form dist (l, s), e.g., the L2 norm (l − s) 2.

(4) The illumination continues smoothly as a constant beyond the image boundaries.

The connection between the atmospheric scattering model and the Retinex algorithm was found in the next section. Collecting all the above assumptions into one expression, the penalty function is defined as Eq.(5).

 

3. Our Approach

3.1 The Variational Framework

The key to our approach is that it combines the best of both factors: the effectiveness of variational framework with the rapidity of multi-resolution pyramid reduction, and the sensitivity of the green light component. Our approach can be decomposed into three steps: the establishment of the variational framework, the estimation of our variational framework’s input, and the solution of our variational framework.

Rearrangeing Eq.(1), we find the following expression:

In natural scene, the brightest pixel could be from a white car or a white building, so we assume A as the mean of the top 0.6% brightest pixels in the hazy image [11]. In order to keep Eq.(2) non-negative, we reverse it:

where IA(x, y) = A-I(x, y), JA(x, y) = A-J(x, y). Then Eq.(2) is converted to logarithmic domain, with the multiplication changing into addition:

where iA=In(IA(x, y)), jA=In(JA(x, y)), t’=In(t(x, y)). According to Kimmel’s Retinex algorithm, the inverse-reflectance image jA is considered as the same meaning of the illumination image. Then we brought about the variational framework via the following penalty function to estimate the transmission [10]:

where Ω is the support of the image, ∂Ω is its boundary, is the normal to the boundary, α and β are free non-negative real parameters. Three penalty terms force t’ to be smooth, t’ and iA to be proximal, and jA to be smooth respectively.

Global spatial smoothness of the image is the basic assumption in the atmospheric transmission map. Especially, the third item in Eq.(5) assures the inverse-reflectance’s smoothness. However, this Bayesian penalty expression is bound to lose detail and edge information. Because such |∇t'|2 as the smoothness’ realization focuses much on decreasing the gradient norm, and will impose bad effects causing discontinuous jump in jA. As it is forced upon in Kimmel’s algorithm, it will blur jA and cause artificial halos. So another basic assumption should be considered that an item reflects the jA’s boundary based on gradient norm. And we deem that information in gradient domain of the reflectance jA and the given image iA has a high correlation. Hence, we modify the penalty function of Eq.(5) by replacing the third item with a new nonlinear diffusion one -γ|∇(t' - iA)|2|∇iA|2 to preserve the detail-and-edge information of jA. Combined with the correlation between jA and iA, the modified energy function is formulated as follows:

In the intrinsical Retinex algorithm, Eq.(6) will be executed three times during every simulation of the RGB image processing. Therefore, three different t’ will be got as there are three different color channels. Here we first assume that the input of Eq.(6) is one and only, and we will present a way to estimate it in the next.

3.2 The Sensitivity of Green Wavelength

Human visual system (HVS) has specific response sensitivity to the small interval of light wave length. Fig. 1.(a) shows the segment of wavelength where the HSV has its maximum sensitivity. In figure one, the green curve represents the sensitivity for photonic vision, and the blue one for scotopic vision. Since we can see the much higher sensitivity to luminous efficiency of the scotopic vision compared to the photopic vision, both of them have maximum sensitivity from green-blue wavelength for red and blue perception, in which the combined overall sensitivity lies at about from 505nm to 555nm. Fig. 1.(b) shows the symmetric forward-scattered intensity from particle of aerosol in the incident light beam: the blue wavelength will tangentially be scattered more into 90° (270° respectively) direction relative to the incident light in the plane of observation, and the red wavelength will be scattered into forward (0°) in the plane of observation. With the angle increasing from 0° to 90°, the less intensity will be scattered. Meanwhile, the light wavelength ranges from red to blue.

Fig. 1.The sensitivity of green wavelength. (a) Photopic and scotopic response of the HVS [12], (b) Angular patterns of forward-scattered intensity from particle of aerosol [13]

Due to the response of green wavelength and the intensity of forward-scattered, the green light component of image is assumed as the input of Eq.(6), which not only has satisfied efficiency (reduced the numbers of the estimation of transmission map from three to one) but also corresponds with the statistical prior. We make all the three color channels of image (including: red, green, and blue light component) as the input of Eq.(6). Different haze removal results can be found in Fig. 2.

Fig. 2.Haze removal results. Top: example images. Bottom: histogram of red, green, and blue light component of image. (a) Input hazy image. (b) Haze removal result with the input of red light component. (c) Haze removal result with the input of green light component. (d) Haze removal result with the input of blue light component

As shown above, haze removal result with the input of green light component is more valid, and its histogram is more balanced than others (see Fig.2.(c)).

3.3 The Multi-resolution Pyramid Reduction

Finally, the energy functional Eq. (6) is solved via the Euler equation, which is given by:

where the third item provides nonlinear diffusion process. Eq.(7) is the conductive coefficient or edge-stopping function and dependent on the image. The normal numerical process of Eq.(7) employs the Projected Normalized Steepest Descent (PNSD) with a NSD iteration format. It can get an ideal t’, but converges slowly. So, we apply the regularized P_M diffusion for pyramid construction.

P_M diffusion is introduced with the second-order edge-stopping function, which tends to cause blocky effects and false edges [14]. Thus, more attention is put on the correlation of gradient information between iA and jA in our method. We deem that the edges and details of the jA derive from iA, replace the so-called anisotropic function by the norm of iA, which can not only preserve the edge information, but also eliminate the blocky effects caused by edge effects. Results show that the performance is in proximity to the fourth-order PDEs but with less complexity apparently. Therefore, our P_M diffusion function is defined:

where Gσ is the Gaussian kernel with the variance of σ. We choose the second edge-stopping function: g = 1/(1 + |∇t'|2/k2) and other parameters are K=10，σ2=0.1.

The numerical process of Eq.(7) can be solved as follows: (1) The initial condition is set to the maximum of the image. When updating the next resolution layer, the result is up scaled(2:1 ratio) by pixel replication in the neighborhood. By the multi-resolution solution based on regularized P_M, we can obtain more edged and vivid images than precious pyramid construction. The results are shown in Fig. 3, from which we can see more edge and texture information in every layer of the regularized P_M. (2) For better visual effect, linear diffusion takes place only one time in every iteration to assume illumination, which is from the top-level to the bottom-level with high resolution. In our experiment, the total iteration time is set to 10. (3) Then we add the constraints in every iteration as: max(t’j, t’’j), where t’j is the image after down-sampling, t’’j is the image after iteration.

Fig. 3.Two pyramid constructions. Right: the Gaussian sampling. Left: our regularized P_M sampling.

After we get the one and only t’, the latent scene radiance J(x, y) will be solved as follows:

 

4. Experimental Results

In order to validate the performance of our approach, two groups of experiments are established. Firstly, the difference between two variational frameworks is compared via testing synthetic images. The different performance is shown in Fig. 4. More sharp edges can be obtained using the nonlinear diffusion introduced in our algorithm than the Kimmel’s.

Fig. 4.Synthetic images to test performance. Top: example synthetic images. Bottom: the 3D mesh of the synthetic images. (a) Original Image. (b) Kimmel’s Retinex (c) Our variational framework.

Then, the proposed algorithm is compared with both DCP and Kimmel’s Retinex for natural images in Fig. 5 - Fig. 8. The results show that our algorithm could present more vivid high-contrast restored image, and obtain more colorful results without color distortion. Especially, the proposed algorithm achieves wider dynamic range compression in the dark regions. Meanwhile, it takes the same time as Kimmel’s Retinex, which is approximately 12.5% of DCP’s. Besides subjective evaluation, the typical objective evaluation values are in list in Table 1. Our results and comparison are generated on a PC with Intel Dual-Core E5300 2.6GHz CPU with 2GB memory, Matlab 2013a.

Fig. 5.Natural images to test performance (Aerial image). (a) Input image. (b) DCP. (c) Kimmel’s Retinex. (d) Our algorithm.

Fig. 6.Natural images to test performance (Trees). (a) Input image. (b) DCP. (c) Kimmel’s Retinex. (d) Our algorithm.

Fig. 7.Natural images to test performance (City) [15]. (a) Input image. (b) DCP. (c) Kimmel’s Retinex. (d) Our algorithm.

Fig. 8.Natural images to test performance (Peak). (a) Input image. (b) DCP. (c) Kimmel’s Retinex. (d) Our algorithm.

Table 1.Objective evaluations.

According to Table 1, The “Mean” stands for the mean luminance of the image. The “Variance” stands for the contrast of the image. The “PDCP” is the proportion of the number of pixels, whose luminance is lower than 25 in our DCP image. A closer value of mean to 125 indicates a better performance; meanwhile with the increase of the variance and PDCP, we will have a better result of image dehazing. With common standards for global and local contrast, the proposed algorithm gives the better results than others’. The experiments witness that subjective evaluation is consistent with the objective.

 

5. Conclusion

In summary, we present a variational framework for signal image dehazing based on restoration, and mainly focus on the efficiency by adopting the regularized P_M for the multi-resolution pyramid construction. With the adaptive adjustment of the variational framework, experimental results give the better visible restored images with wider dynamic range and vivid color. The impact on the proposed approach by the issue of noise would be considered in our future work.