An Adaptive Iterative Algorithm for Motion Deblurring Based on Salient Intensity Prior

Abstract

In this paper, an adaptive iterative algorithm is proposed for motion deblurring by using the salient intensity prior. Based on the observation that the salient intensity of the clear image is sparse, and the salient intensity of the blurred image is less sparse during the image blurring process. The salient intensity prior is proposed to enforce the sparsity of the distribution of the saliency in the latent image, which guides the blind deblurring in various scenarios. Furthermore, an adaptive iteration strategy is proposed to adjust the number of iterations by evaluating the performance of the latent image and the similarity of the estimated blur kernel. The negative influence of overabundant iterations in each scale is effectively restrained in this way. Experiments on publicly available image deblurring datasets demonstrate that the proposed algorithm achieves state-of-the-art deblurring results with small computational costs.

1. Introduction

Accumulation of photons in the sensor during a given exposure time is the basic principle of photography. In general, the better quality of the final image depends on more photons reach the sensor to reduce the photonic noise. However, during the exposure process, the movement of objects or camera shake generates blur because the photons are accumulated in neighboring pixels. The mathematical model of the blur process can be formulated as a convolution operation:

B = I ⊗ K + N ,        (1)

where I is the latent image, K is the blur kernel, N is introduced random noise, B is the blurred image, and ⊗ is the convolution operator. Blind image deblurring is a classical inverse problem that aims to recover a sharp latent image I and a blur kernel K from a blurred image B .

Impressive progress has been made in estimating clear images in recent years, including both direct [1][2] and iterative [3][4][5][6] blind deblurring methods. While the iterative kernel estimation is easily stuck into the local minima, it may lead to an unsatisfactory kernel and some artifacts in the deblurred result.

Multi-scale iterative framework and sparse priors contribute much to the success of accurately estimating the latent image and blur kernel. Specifically, a mixture of Gaussians approximation is used to fit the image gradient’s the heavy-tailed distribution [7]. Then, the sparse priors are exploited for estimating both the latent image and the blur kernel [8]. In the multi-scale iterative framework, the blur kernel is obtained from the given blurred image and the estimated latent image. And then the given blurred image and the blur kernel are used to estimate the latent image. The iterative process can alternatively optimize the motion blurred kernel and the latent image.

Deblurring methods which based on sparse gradient priors are not efficient in estimating clear latent images, especially for methods formulated within the MAP (maximum a posterior) framework [9]. To solve this problem, some methods seek to detect sharp image structures with high efficiency [10][11]. The shock filter is used to create a sharpened latent image map for kernel estimation. Meanwhile, the bilateral filter and edge thresholding are used to suppress small details and noise in the iterative process [10]. The accurate kernel estimation of iterative methods mainly depends on the latent image estimation, which is characterized by restroring the salient structures and suppress noise in smooth regions. A texture-removal strategy is proposed to guide the large-scale structures selection [11]. Similarly, salient region segmentation is used to extract sharp foreground regions from the background, which help the algorithm to partially estimate the blur kernel and perform deconvolution throughout the entire image without introducing artifacts [12].

To benefit the kernel estimation, a unified model is proposed to leverage an unnatural L₀ sparse representation which seeks the gradient sparsity close L₀ to suppress the pernicious small-amplitude structures [13][14]. However, this model developed for natural blurred images does not perform well in some specific scenarios, such as faces, text, and low illumination images. Based on the fact that in the clear text images, the text and the background regions have nearly uniform intensity values, the intensity prior and gradient prior are proposed for text image deblurring [15]. Considering that the dark channel is less sparse during the image blurring process, the dark channel prior [16] is proposed to enforce the sparsity of the dark channels in latent images. The bright channel prior is then proposed to deblur images with bright channel pixels [17]. The method based on the extreme channels prior achieves efficient restorations by leveraging both the dark and the bright information. However, the superior performances of these algorithms [15][16][17] are achieved at the cost of higher computational complexity.

In this paper, a novel algorithm based on salient intensity prior is introduced to accurately detect useful structures from complex backgrounds. Since the natural image gradient follows a heavy-tailed distribution, the sparse prior for latent images is thus provided. The saliency values of image intensity also follow a heavy-tailed distribution. Meanwhile, compared with the salient intensity of a clean image, the salient intensity of a blurred image is less sparse. During the blurring process, a low-saliency pixel is averaged with neighboring high-saliency pixels which lead its saliency increases. Therefore, the sparse distribution of the saliency can be used to restore the salient structures in the blurred image. We propose a new regularization term to enforce the sparsity of the distribution of saliency in the latent image. In the deblurring process, the regularization term facilitates the recovery of clean images in various scenarios with small computational costs.

In the multi-scale iterative framework, the number of iterations in each scale is usually fixed. Considering that the number of iterations depends on many factors, such as the complexity of the blur kernel, the regions with useful structures in a blurred image and the quality of the initial kernel computed in previous scale, a fixed number of iterations is not suitable for each scale. In this paper, an adaptive iteration strategy is proposed to adjust the number of iterations by evaluating the performance of the latent image and the similarity of the estimated kernel. The performance of the latent image is evaluated by gradient cepstrum analysis. Similar to an autocorrelation map [1], the cepstrum converts the two-dimensional convolution into an addition operation, which is helpful to separate the information of the latent image and the blur kernel from a given blurred image [2]. The adaptive iteration strategy effectively restrains the negative influence of overabundant iterations, especially in deblurring images with complex motion blurred kernels. Moreover, the computational costs are reduced significantly in this way.

Extensive experimental results on real blurred images demonstrate that the proposed algorithm performs well in both quantitative and visual evaluations against the state-of-the-art methods.

2. Motion Deblurring Based on Salient Intensity Prior

The iterative process of image deblurring uses a coarse-to-fine manner. The success of many image deblurring methods depend on the restoration of intermediate images with sharp edges which is important for kernel estimation. However, the assumption that blurred images have sharp edges not always hold, especially when deblurring low-illumination images, as shown in the middle column of Fig. 1. Therefore, the intensity prior and gradient prior are used in recent work. However, the distributions of the intensity of a text image, low-illumination image and natural image are quite different. Compared with the dark channel [16] and bright channel [17], the method based on salient region detection can detect the salient structures more accurately in various scenarios. In this section, we describe the salient intensity prior and then its role in motion deblurring.

 

Fig. 1. Given the blurred images (top), a method based on salient region detection is used to compute the histograms and detect the salient region (second row). The image saliency follows a heavy-tailed distribution (third row). The regions with sharp edges or salient structures detected by this method are more accurate than the dark channel (fourth row) and bright channel (bottom).

2.1 Salient Intensity Prior

By determining the most useful and informative part of a scene, salient region detection is widely used in many computer vision applications to decrease the computational complexity.

The saliency map is built upon the intensity contrast between image pixels. The saliency value of a pixel at location p in latent image I is defined as follows [19]:

\(\operatorname{Sal}\left(I_{p}\right)=\sum_{\forall I_{i} \in I}\left\|I_{p}-I_{i}\right\|\) ,        (2)

where \(\|\cdot\|\) represents the distance metric of the intensity and the value of I_i is in the range of 0-1. Then, the saliency values are normalized and sorted to detect the most salient region and the least salient region.

As shown in the second row of Fig. 1, salient region detection accurately suppresses the insignificant structures in blurred images and highlights the salient objects. Compared with the dark channel and bright channel, salient region detection can reflect the regions with salient structures from the complex background more accurately, which can guide the latent image recovery in the deblurring process.

Similar with the image gradient, image saliency is also sparse, as shown in the third row of Fig. 1. The salient object detection can be reformulated to a sparsity pursuit problem because salient object can be viewed as the sparse but strong “outliers” within an image [20]. Fig. 2 shows that the saliency values of both blurred images and clear images follow a heavy-tailed distribution. Furthermore, the blurring process tends to increase the saliency of low-saliency pixels by weighted averaging of the pixels in the neighborhood. Therefore, the distribution of the salient intensity in a blurred image is less sparse than that in a clear image. This property not only holds for text images, but it also holds for low-illumination images and natural images. Thus, the sparsity of the salient intensity can be used to restore the salient structures in the blurred image.

 

Fig. 2. Saliency histograms of both blurred and clear images. Blurred images have fewer pixels with zero salient intensity than clear ones, which means that the distribution of the salient intensity in a blurred image is less sparse than that in a clear image.

2.2 Motion Deblurring Model

A new regularization term \(\|S(I)\|_{0}\) is introduced to measure the sparsity of the salient intensity. S( I ) is the saliency map of image I , and it can be defined as follows:

\(S\left(I_{p}\right)=\left\{\begin{array}{cl} \operatorname{Sal}\left(I_{p}\right), & I_{p} \geq I_{S a l_{\min }} \\ -\operatorname{Sal}\left(I_{p}\right), & I_{p}<I_{S a l_{\min }} \end{array}\right.\)       (3)

where \(I_{S a l_{\min }}\) is the intensity of the least salient pixel.

In the motion deblurring process, \(\|S(I)\|_{0}\) can enforce the sparsity of the salient intensity in a latent image. Based on the MAP framework [13][16], the motion deblurring problem can be formulated as follows:

\(\min _{I, K}\|I \otimes K-B\|_{2}^{2}+\gamma\|K\|_{2}^{2}+\sigma\|S(I)\|_{0}+\lambda\|\nabla I\|_{0}\) ,       (4)

where the second term is the prior of the blur kernel K . The third term is the prior of the salient intensity of the latent image, which plays an important role in preserving the salient structures. The fourth term is the prior of the image gradient, which is used to retain large gradients and remove smooth structures, and γ, σ and λ are the weight parameters.

Because it is difficult to solve Eq. (4) directly, a classical motion deblurring method uses an alternating minimization algorithm [14]. The blur kernel K and the latent image I are estimated alternatively by fixing each other. Then, the solution of Eq. (4) becomes two sub-problems:

\(\min _{I}\|I \otimes K-B\|_{2}^{2}+\sigma\|S(I)\|_{0}+\lambda\|\nabla I\|_{0}\) ,       (5)

and

\(\min _{K}\|I \otimes K-B\|_{2}^{2}+\gamma\|K\|_{2}^{2}\) .       (6)

2.3 Estimating Latent Image

Because the L₀ regularization term is computationally intractable, Eq. (5) is solved based on a half-quadratic splitting technique [14]. By introducing the auxiliary variables s and \(g=\left(g_{h}, g_{v}\right)^{\top}\) that correspond to S( I ) and the image gradients, respectively, Eq. (5) can be written as follows:

\(\min _{l, u, g}\|I \otimes K-B\|_{2}^{2}+\alpha\|S(I)-s\|_{2}^{2}+\beta\|\nabla I-g\|_{2}^{2}+\sigma\|s\|_{0}+\lambda\|g\|_{0}\),        (7)

where α and β are the penalty parameters. Eq. (7) can be solved by minimizing I , s and g separately while fixing one another.

Then, the latent image I can be solved by the following:

\(\min _{I}\|I \otimes K-B\|_{2}^{2}+\alpha\|S(I)-s\|_{2}^{2}+\beta\|\nabla I-g\|_{2}^{2}\).          (8)

The closed-form solution for this least squares minimization problem is as follows:

\(I=\mathcal{F}^{-1}\left(\frac{\overline{\mathcal{F}(K)} \mathcal{F}(B)+\alpha \mathcal{F}(s)+\beta F_{G}}{\mathcal{F}(K) \mathcal{F}(K)+\alpha+\beta \overline{\mathcal{F}(\nabla)} \mathcal{F}(\nabla)}\right)\),         (9)

where \(\mathcal{F}(\cdot)\) and \(\mathcal{F}^{-1}(\cdot) \) denote the fast Fourier transform (FFT) and the IFFT, respectively, \(\overline{\mathcal{F}(\cdot)}\) is the complex conjugate operator, \(F_{G}=\overline{\mathcal{F}\left(\nabla_{h}\right)} \mathcal{F}\left(g_{h}\right)+\overline{\mathcal{F}\left(\nabla_{v}\right)} \mathcal{F}\left(g_{v}\right)\) , and \(\nabla_{h}\) and ∇_v denote the horizontal and vertical differential operators, respectively.

When the latent image I is given, s and g can be solved by the following:

\(\min _{u, g} \alpha\|S(I)-s\|_{2}^{2}+\beta\|\nabla I-g\|_{2}^{2}+\sigma\|s\|_{0}+\lambda\|g\|_{0}\).              (10)

To improve the efficiency of the iteration process, s and g are solved together in one iteration rather than in separate iterations.

Eq. (10) is a pixel-wise minimization problem. The solutions of s and g can be obtained by the following [13]:

\(s=\left\{\begin{array}{c} {S(I),|S(I)|^{2} \geq \frac{\sigma}{\alpha}} \\ {0, \quad \text { otherwise }} \end{array}\right.\),             (11)

and

\(g=\left\{\begin{array}{ll} {\nabla I,} & {|\nabla I|^{2} \geq \frac{\lambda}{\beta}} \\ {0,} & {\text { otherwise }} \end{array}\right.\).            (12)

Compared with other methods based on the image intensity prior, the proposed method based on the salient intensity prior can separate the salient structures of blurred images earlier in the iterative process. Furthermore, the salient intensity prior also has computational costs advantages.

2.4 Estimating Blur Kernel

When latent image I is given, the blur kernel estimation is also a least squares problem. The blur kernel K is optimized by employing the fast deblurring method based on the image gradient [10][13]:

\(\min _{K}\|\nabla I \otimes K-\nabla B\|_{2}^{2}+\gamma\|K\|_{2}^{2}\).          (13)

Similar to traditional methods [10][13], the closed-form solution of this minimization problem is as follows:

\(K=\mathcal{F}^{-1}\left(\frac{\overline{\mathcal{F}(\nabla I)} \mathcal{F}(\nabla B)}{\mathcal{F}(\nabla I) \mathcal{F}(\nabla I)+\gamma}\right)\).       (14)

After the blur kernel K is obtained, the negative elements are set to zero. Then, the kernel is normalized. In the multi-scale iterative deblurring framework, the kernel estimation is also carried out in a coarse-to-fine manner.

2.5 Image Restoration

The estimated latent image is not the final deblurred result because of the lack of details. So based on the estimated blur kernel, the final deblurred image is restored by non-blind deconvolution algorithms. In this paper, the low-illumination images and other images are restored by non-blind deconvolution algorithms in [21] and [13][15], respectively.

3. Motion Deblurring Optimized by Adaptive Iteration Strategy

Considering the multi-scale iterative framework, the number of iterations depends on many factors, such as the complexity of the blur kernel, the regions with useful structures in the blurred image and the quality of the initial kernel that was computed in the previous scale, a fixed number of iterations is not suitable for each scale. In this section, an adaptive iteration strategy is proposed to adjust the number of iterations in each scale.

3.1 Cepstrum of Latent Image Gradient

Cepstrum was originally developed for characterizing the seismic echoes resulting from earthquakes and bomb explosions. It is the result of the inverse Fourier transform of the logarithm of the estimated spectrum of a signal. Similar to the autocorrelation map [1], in the cepstrum domain, the two-dimensional convolution is converted into an addition operation, which is helpful to separate the information of the latent image and the blur kernel from a given blurred image [2].

The blur kernel actually describes the path of the random camera shaking or the movement of objects during the camera exposure process. The cepstrum of the blurred image gradient is neither sensitive to the effect of the random path [2] nor to the random noise which produced in the blurring process. Thus, the cepstrum of the blurred image gradient accurately reflects the information of the blur kernel:

\(C_{\nabla B}=\mathcal{F}^{-1}(|\log (1+|\mathcal{F}(\nabla B)|)|\).       (15)

According to the property of the cepstrum, the cepstrum of the blur process is expressed as follows:

\(C_{\nabla B}=C_{\nabla K}+C_{\nabla I}\) ,       (16)

where C_∇K and C_∇I are the cepstrum of ∇K and ∇I , respectively. Because of the difference between the frequency components of ∇K and ∇I , the information of ∇K can be extracted from the cepstrum of ∇B .

As shown in the left column of Fig. 3, compared with the cepstrum of the clear image gradient, the cepstrum of the blurred image gradient contains the information of the blur kernel, which is symmetric about the origin. The right two columns show the change of the latent image and the corresponding cepstrum of the gradient in one image scale. Compared with the latent image shown in Fig. 3 (c), Fig. 3 (d) contains significant residual blur, which means that overabundant iterations will degrade the latent image. Meanwhile, the difference of the cepstrum between the latent image gradient and the blurred image gradient decreases. Thus, the performance of the latent image can be evaluated by the mean difference between the cepstrum of the intermediate latent image gradient and the cepstrum of the blurred image gradient:

\(\overline{E_{\nabla K}^{n}}=\frac{1}{M} \sum_{i, j}\left|C_{\nabla B}-C_{\nabla I}^{n}\right|\) ,       (17)

where M is the pixel number of the blur kernel K , and C_∇B and \(C_{\nabla I}^{n}\) are the cepstrum of the blurred image gradient and the cepstrum of the intermediate latent image gradient, respectively.

 

Fig. 3. The cepstrum (bottom) of the gradient of the blurred image, the ground truth and the intermediate latent images (top) in one image scale. It is clear that overabundant iterations will degrade the performance of the blur kernel. The latent image of the 40^th iteration contains significant residual blur.

3.2 Adaptive Iteration Strategy

To reduce unnecessary computational costs, the similarity of the kernel is proposed to evaluate the subtle changes in the estimated blur kernel in the iterative process:

\(\overline{\rho_{K}^{n}}=\frac{1}{M} \sum_{i, j}\left|K^{n}-K^{n-1}\right|\) ,       (18)

\(\overline{E_{\nabla K}}\) and \(\overline{\rho_{K}}\) can accurately guide the blur kernel estimation in the iteration process. In this way, the negative influence of overabundant iterations in each scale is effectively restrained. In the last two or three scales, the adaptive iteration strategy will perform more effectively, which largely reduces the computational costs of the motion deblurring algorithm. Algorithm 1 shows the main steps of the adaptive iteration process on one image scale.

Algorithm1: Motion deblurring algorithm on one image scale.

 

The intensity weight σ and the gradient weight λ are penalty parameters, which can separate details from background in a coarse-to-fine manner. In the beginning of the whole deblurring algorithm, these two parameters are initialized as follows:  

\(\left\{\begin{array}{l} {\sigma=0.005 k^{2}} \\ {\lambda=0.25 k^{-2}} \end{array}\right.\) ,       (19)

where k is the kurtosis of blurred image. As shown in Fig. 1, the kurtosis of the text image and the low-illumination image are higher than natural image, which means that these images contain more intensity information than gradient information. Moreover, high intensity weights and low gradient weights are suitable for deblurring these high-contrast images. In the adaptive iteration process, ε is a penalty parameter which that restrains the unfavorable factors from the earlier iterations.

4. Experimental Results

In this section, the proposed algorithm is compared with state-of-the-art image deblurring methods on two publicly available image deblurring datasets [9] [18]. And then the proposed algorithm is evaluated by using low-illumination images, text images and face images. The results of the state-of-the-art algorithms are all generated by their executable programs or source codes. The code of our adaptive iterative algorithm based on salient intensity prior is provided in Ref. [22].

4.1 Synthetic Dataset

To better verify the effectiveness of the proposed algorithm, two image benchmark datasets [9][18] are used for quantitative evaluations. The evaluation protocols defined for these two datasets are followed for a fair comparison. We first test using the dataset from [18], which consists of 48 images with 4 ground truth images and 12 kernels, including several large kernels. We compare the proposed algorithm with the algorithms of [7], [8], [10], [11], [21],

[16] and [17]. The PSNR value is computed by comparing each restored image with 199 clear images captured along the camera’s motion trajectory. As shown in Table 1, the proposed algorithm achieves the highest average PSNR among all the algorithms evaluated. The first row of Fig. 4 shows the results on a challenging blurred image with large-scale blur kernels. Although [11] and [16] are able to address overall deblurring performance, their deblurred results also contain some ringing artifacts. The deblurred results of the proposed algorithm contain clearer details and fewer artifacts in comparison with other deblurring methods. It can also be seen in the third row of Fig. 4.

Table 1 Comparison of average PSNR for dataset [18]

 

 

Fig. 4. Visual comparisons and PSNR of the state-of-the-art algorithms on Blurry2_8 and Blurry3_7 from the dataset in [18]. The results generated by the proposed algorithm are visually more pleasing.

Then, the proposed algorithm is compared with the algorithms of [10], [13], [15], and [17] on the dataset by [9], this dataset includes 4 images and 8 blur kernels. Fig. 5 indicates that the proposed algorithm performs better than other algorithms on this dataset in terms of the cumulative error ratio.

 

Fig. 5. Comparisons in terms of the cumulative error ratio

The time consumed by the proposed algorithm is compared with other algorithms from [16] and [17], which are based on the intensity prior and the gradient prior. For a fair comparison, all the algorithms are implemented on a laptop with an Intel Core i7-6700HQ processor and 4GB memory. As shown in Table 2, the processing of the blur kernel estimation of our algorithm for dataset [18] is much faster than the other algorithms. The main reason is that, compared to the dark channel prior [16] and the extreme channel prior [17], the computational costs of the salient intensity prior are much lower, as shown in Table 3. Furthermore, the adaptive iterative strategy effectively restrains the negative influence of overabundant iterations in each scale, especially in the last two or three scales.

Table 2. Comparison of average processing time of blur kernel estimation for dataset [18]  

 

Table 3. Comparison of average processing time of intensity priors of an 800×800 gray scale image  

 

4.2 Real Images

For most deblurring methods, low-illumination blurred images are particularly challenging because they often have saturated pixels which makes high quality kernel estimation very difficult. As shown in Fig. 6, compared with the other three methods [15][16][17], the result generated by the proposed algorithm has clearer details and fewer artifacts, especially in the regions enclosed in red boxes. Fig. 7 shows a visual example from the project dataset of [16], where the results of other state-of-the-art algorithms contain significant ringing artifacts [16] or residual blur [15][17]. In low-illumination image deblurring, the proposed algorithm based on the salient intensity prior can effectively separate the salient structures (blurred high-light point) from the dark background. Thus, the estimated blur kernel is more accurate than other algorithms, which is propitious to generating results with less ringing artifacts or residual blurred artifacts.

 

Fig. 6. Results of low-illumination image.

 

Fig. 7. Results of number plate image.

Fig. 8 shows that our algorithm performs well on a complex blurred text image against the other algorithms which based on the image intensity prior and the image gradient prior. As shown in the top row of Fig. 8, the proposed algorithm generates clearer results. Furthermore, compared to the results generated by other algorithms [15][16][17], both the result and the latent image of the proposed algorithm contain less ring artifacts. The latent images show that the method based on the salient intensity prior can extract regions with salient structures more accurately, which is helpful for image restoration.

 

Fig. 8. Results (top) and latent images (bottom) of text image.

Blurred face images are also troublesome for most state-of-the-art algorithms. Because blurred face images contain fewer sharp edges or salient textures, the kernel estimation of most motion deblurring algorithms cannot make full use of the image gradient. The left column of Fig. 9 shows a real blurred face image that is obtained from the dataset of [23]. The method of [16] cannot generate accurate image intensity in the region of the left eyelash, while the method of [15] cannot generate a clear result, as shown in Fig. 9. Compared with other algorithms, the results of the proposed algorithm accurately recover the fine details of the original latent images.

 

Fig. 9. Results of face image.

5. Conclusion

We present a novel algorithm for motion deblurring by leveraging a salient intensity prior to benefit the latent image restoration. An adaptive iterative strategy is also proposed to adjust the number of iterations in each scale, which makes the algorithm perform robustly and fast.

Extensive experimental results on real blurred images demonstrate that the proposed algorithm performs well in both quantitative and visual evaluations against the state-of-the-art methods with small computational costs.