Adaptive Object-Region-Based Image Pre-Processing for a Noise Removal Algorithm

Abstract

A pre-processing system for adaptive noise removal is proposed based on the principle of identifying and filtering object regions and background regions. Human perception of images depends on bright, well-focused object regions; these regions can be treated with the best filters, while simpler filters can be applied to other regions to reduce overall computational complexity. In the proposed method, bright region segmentation is performed, followed by segmentation of object and background regions. Noise in dark, background, and object regions is then removed by the median, fast bilateral, and bilateral filters, respectively. Simulations show that the proposed algorithm is much faster than and performs nearly as well as the bilateral filter (which is considered a powerful noise removal algorithm); it reduces computation time by 19.4 % while reducing PSNR by only 1.57 % relative to bilateral filtering. Thus, the proposed algorithm remarkably reduces computation while maintaining accuracy.

1. Introduction

As digital imaging techniques proliferate, there is demand for high-quality images from sources ranging from digital broadcasting systems to smartphones. However, the continuing miniaturization and increasing resolution of CMOS image sensors (CIS) comeswith the problem of increasing noise. A great deal of research has focused on noise removal techniques [1].

Noise removal algorithms aim to remove image noise while maximally maintaining edges, texture, and color. Noisy images can be symbolized as follows:

where o(x,y) represents the noisy image, i(x,y) represents the noise-free natural image, and n(x,y) represents the noise; x and y represent the two-dimensional pixel coordinates. In the context of Eq. (1), the purpose of the noise removal algorithm can be thought of as estimating the image as close to i(x,y) as possible by removing n(x,y) from o(x,y).

Among the numerous possible approaches, this paper proposes an effective algorithm for pre-processing based on region-based noise removal. The region-based noise removal strategy targets the human visual system, which gives more attention to well-focused objects and bright regions rather than blurred and dark regions [2]. Accordingly, the region-based noise removal method segments a noised image into three types of regions: well-focused regions that include objects, bright regions without objects, and dark regions. According to the characteristics of each region, one of three noise removal algorithms is applied: a highly accurate algorithm with high computational complexity is applied to well-focused regions, a moderately accurate algorithm with moderate computational complexity is applied to bright regions, and a low-accuracy algorithm with low computational complexity is applied to dark regions. Thus, the computational complexity and processing time can be reduced without loss of noise removal accuracy as perceived by the human visual system.

Our research aims to develop efficient methods for the pre-processing system of a noise removal algorithm; to this end, the present work employs a novel method based on adaptive higher-order statistics (HOS).

In Section 2, we review previous related work in noise removal and segmentation. Section 3 describes the proposed algorithm, focusing on pre-processing filters for effective segmentation. Section 4 presents simulations in which the pre-processing filter is applied to database images and nature images and evaluated in terms of computation speed and noise removal accuracy.

 

2. Related Work

2.1 Image Segmentation

There are many approaches for segmenting images. Among them, the higher-order statistics (HOS) method is widely used to determine which regions are focused well or poorly. This method gives outstanding results for Gaussian noisy images [3–4]. The HOS method accentuates the difference between neighboring pixels using fourth-order statistical analysis on the red, blue, and green channels of input images, which are calculated as follows for each channel (using the green channel as an example).

where η(x,y) are the neighboring pixels centered at (x,y), IG(s,t) is the green channel of the input image, is the mean value of the neighboring pixels, Nη is the size of set η(x,y), and DSF is a downscaling factor experimentally set to 300. are the fourth-order statistics on the green channel, and HOS(x,y) is the final result representing the disparity value.

According to previous research, HOS methods suffer from some limitations. One issue is that current implementations of the HOS method do not consider the distance between neighboring pixels. Thus, we can improve the effectiveness by using the distance between pixels as in the bilateral filter.

2.2 Noise Reduction

Noise reduction is a widely researched topic in image processing. In this paper, we do not improve the accuracy of the noise reduction algorithm but instead propose a pre-processing system to improve the efficiency of noise reduction by means of state-of-the-art algorithms. In this section, we introduce the three noise reduction algorithms used in the proposed system.

where F[I]p is the restored value of pixel p, σs is the standard deviation for the spatial distance of neighboring pixels, σr is the standard deviation in the tonal distance between neighboring pixels, G represents the Gaussian function, Ip and Iq are the pixel intensities of pixels p and q, respectively, S represents the set of neighboring pixels centered at p, and Wp is the sum of the weights within the block. The bilateral filter is computationally complex but very useful for preserving edge information; thus, it is best applied to well-focused and object regions.

where F[I]p is the restored value of pixel p, Iq is the intensity value of pixel q, and S represents the neighboring pixel centered at pixel p. dq and vq are the spatial weight and tonal weight of pixel q, respectively, which are defined as follows:

where gq is the spatial distance, fq is the tonal distance, and c is an adjustable constant experimentally set to 8. The shapes of these weights are similar to that of the Gaussian function and can replace it without great loss of accuracy.

 

3. Proposed Algorithm

The core algorithm proposed in this paper is the pre-processing filter that segments images into three different regions for effective image noise removal; this strategy has been researched extensively because of its ability to improve the accuracy and speed [8].

The proposed pre-processing filter segments images into three types of regions related to the perception of the human visual system, and a different noise-removal algorithm is applied to each type as appropriate. The three region types used in the filter are dark regions, background regions, and object regions. First, object edges are not noticeable in dark regions, so preserving edges during noise removal is unnecessary. Second, noise is more conspicuous in background regions but does not attract the viewer’s main attention. Third, the object region attracts the most attention, and noise in this region has the worst impact on the human visual system; accordingly, clear noise removal and preservation of edge information is key. This characteristic-based region classification improves the noise-removal algorithm by reducing its computational cost and concentrating on the regions that are most likely to receive careful attention.

In this paper, we apply the median filter to the dark region, the fast bilateral filter to the background region, and the bilateral filter to the object region (Fig. 1).

Fig. 1.Block diagram of the proposed algorithm.

3.1 Image Segmentation by Luminance

Image segmentation by luminance classifies image segments as dark regions and bright regions. In the proposed algorithm, dark regions are those in which it is too dark to recognize objects; dark regions are well-suited to the median filter, since it inexpensively removes noise and its tendency to degrade image quality is less important in these regions. Segmentation based on luminance level can be described as follows:

where σ(i,j) represents the standard deviation within each block, represents the mean within each block, c(i,j) is the coefficient of each block, v(i, j) is the luminance intensity, x(i,j) represents the pixel intensity on pixel (i,j), α is the adjustment coefficient (experimentally set to 1.1), TB is the threshold value (set to 1/9 of the mean value), and L(i,j) is a binary classification in which 1 represents a bright region and 0 represents a dark region. This simple algorithm segments dimly lit regions easily and with low computational complexity.

3.2 Image Segmentation by Object and Background

The proposed segmentation method for object and background is based on the HOS [3] presented above in Eq. (2). The HOS refers to the disparity between each pixel and its neighboring pixels; however, this method has a weakness in that the mean does not contain sufficient information to disentangle objects in complex, noisy conditions. We solve this problem by supplementing the image gradient and spatial-tonal distances with a bilateral filter.

First, spatial distance weights represent similarity in location among pixels; closer points have higher weights and farther points have lower weights. Second, tonal distance weights represent the similarity in intensity among pixels; cognate pixels have low disparity and are given high weights. Third, gradient weights are calculated from the disparity between each pixel and its top, bottom, left, and right neighboring pixels; if these gradients are less than a predefined threshold, the pixel is judged to be in the same region as its neighboring pixels, and the results of the segmentation algorithm can be applied to the pixels as a group. The benefits of this algorithm are reduction in the computational complexity and improvement in the accuracy of segmentation because error points of HOS due to noise are eliminated. These three weights are implemented as follows:

where wsd(s,t) and wtd(s,t) are the spatial and tonal distance weights between pixel (x,y) and its neighboring pixel (s,t), respectively. S is the set of neighboring pixels, and wG(x,y) is the binary weight of image gradients on pixel (x,y). TG is the threshold value; we set this value to 20 according to experimental calculations, but it can be changed in accordance with images.

The proposed HOS method using the weights mentioned above is implemented as follows:

where mR(2)(x,y), mG(2)(x,y), and mB(2)(x,y) are the proposed second-order statistics on the red, green, and blue channels, respectively; wG(x,y), wsd(s,t), and wtd(s,t) are the weights mentioned above; IR(s,t) and IR(x,y) are the intensity values on the (s,t) and (x,y) pixels, respectively; η(x,y) is the set of neighboring pixels centered at (x,y); and C is an adjustment constant set to 100 in this paper. ASOS(x,y) is the result of the proposed adaptive second-order statistics (ASOS) and is highest in the object region.

The binary ASOS value classifies image regions as object or background regions [9]; here we use the convention that 1 represents the high-ASOS object region and 0 represents the low-ASOS background region.

 

4. Experimental Classification Results and Analysis

Simulations were carried out on the Tampere Image Database 2008 (TID 2008) [10], a collection of low-depth-of-field images and nature images. TID 2008 contains 25 different images with various scenes, ensuring that simulations assess the accuracy of the proposed noise-removal algorithm in a variety of conditions. The images with low depth of field emphasize the performance of the proposed ASOS because their object and background regions are distinct. The nature images are taken without any effects and represent conditions found in a wide variety of images, thereby testing the algorithm’s general utility.

Our first simulation compared the noise reduction performance of our proposed ASOS to that of the original HOS on a noise-free image, an image with artificially added noise, and a naturally noisy image (Fig. 2). These simulations were intended to test the performance of the proposed method in segmenting object and background regions; the ideal results are better performance by the proposed segmentation algorithm on the noisy images.

Fig. 2.Object/background segmentation by HOS and the proposed method. The first row is a noise-free image, the second row is the same image with artificially added noise, and the third row is a naturally captured image.

Fig. 3.PSNR Comparison: ASOS vs. HOS with increasing noise level.

This comparison verifies that the proposed method more accurately segments the images into object and background areas than the original HOS method for both the artificially noisy image and naturally noisy image, verifying its applicability as a pre-processing filter to improve the accuracy and efficiency of the region-based noise removal algorithm. In addition, we simulate the noise immunity of ASOS with the increasing noise level images shown in Fig. 3. The figure above shows that the PSNR results of ASOS decreased more slowly than those of HOS, which indicates that ASOS is less sensitive to noise than HOS. PSNR results are calculated based on HOS and ASOS results for a noiseless image, respectively.

Fig. 4.Image, PSNR, and computation time results of the proposed algorithm (first row), the bilateral filter (second row), and the fast bilateral filter (third row).

Table 1 describes the numerical verification of the simulation results for LENA, BABOON, park, and flower in Fig. 5. As shown in Table 1, the proposed algorithm greatly reduces the computation time without loss of accuracy compared to the bilateral filter and fast bilateral filter. However, some of the results take too much computation time compared to the fast bilateral filter. Therefore, we need to focus on reducing the computational time of the proposed segmentation algorithm in order to reduce the time gap between the proposed method and the fast bilateral filter in future work.

In addition, we simulated a naturally captured image like that in Fig. 6. With the results of the simulation, we verify that the proposed algorithm can be applied to the general images without low depth of field. The enlarged images in Fig. 6 show that the proposed algorithm can effectively maintain object edges with elimination of the background region. Faces on the enlarged images demonstrate that the proposed algorithm has the same results as the bilateral filter, and the wall surface shows the same results as the fast bilateral filter.

Table 1.Performance measurement of the proposed algorithm

Fig. 5.Test images.

Fig. 6.Noise removal from a low-light image.

Lastly, we compared the proposed algorithm with famous state-of-the-art algorithms. One is a current fast bilateral filter using trigonometric range kernels [11], and the other is a current region-considered denoising algorithm via gradient histogram preservation [12]. Both algorithms have similarities to our approaches to fast denoising. Therefore, we demonstrate that the proposed algorithm works well with this simulation.

Fig. 7 shows that the proposed algorithm is more accurate than the traditional fast bilateral filter; however, the traditional region-considered denoising algorithm is surprisingly accurate. With regard to accuracy, the traditional region-considered denoising algorithm is superior, and the proposed algorithm shows better performance that the current fast bilateral filter. Nevertheless, with regard to computational time, the region-considered algorithm is extremely slow compared to other algorithms. As a result, the proposed algorithm is competitive with regard to computation time and accuracy.

Fig. 7.Comparison with state-of-the-art algorithms.

 

5. Conclusion

We proposed a pre-processing filter for noise removal which segments the image into object and background regions and applies different noise removal algorithms to each region based on its characteristics. Simulations demonstrated that the proposed algorithm reduces computational times relative to the bilateral filter while performing nearly as well in terms of PSNR. However, the proposed algorithm still requires relatively long computational times compared to the fast bilateral filter. Future work will focus on reducing the computational time of the segmentation.