No-reference Sharpness Index for Scanning Electron Microscopy Images Based on Dark Channel Prior

Abstract

Scanning electron microscopy (SEM) image can link with the microscopic world through reflecting interaction between electrons and materials. The SEM images are easily subject to blurring distortions during the imaging process. Inspired by the fact that dark channel prior captures the changes to blurred SEM images caused by the blur process, we propose a method to evaluate the SEM images sharpness based on the dark channel prior. A SEM image database is first established with mean opinion score collected as ground truth. For the quality assessment of the SEM image, the dark channel map is generated. Since blurring is typically characterized by the spread of edge, edge of dark channel map is extracted. Then noise is removed by an edge-preserving filter. Finally, the maximum gradient and the average gradient of image are combined to generate the final sharpness score. The experimental results on the SEM blurred image database show that the proposed algorithm outperforms both the existing state-of-the-art image sharpness metrics and the general-purpose no-reference quality metrics.

1. Introduction

 Scanning electron microscopy (SEM) has been widely applied to display themicrostructures in many fields, such as medicinal chemistry, medical science and genetics, etc. Typically, various types of distortions directly affect the quality of the SEM images during the process of SEM imaging [1-2], e.g. blurring, noise, astigmatism, contrast and brightness. Among them, blurring is one of the most common distortions that affects the quality perception. Therefore, the sharpness assessment of the SEM images is of greatsignificance. Image quality assessment (IQA) contains subjective evaluation and objective evaluation. The former is the most accurate method because human is the ultimate receiverof images. However, the subjective evaluation is time-consuming. For objective IQA, it can be divided into full reference (FR) [3-5], reduced reference (RR) [6-8] and no reference (NR)[9-13]. FR methods need the complete information of the original images. RR methods require parts of the original information, and NR methods operate without any reference information.

 Since the reference information of images are difficult to obtain, NR methods are highly desired. NR methods can be classified into two categories according to the type of distortion. The first category is designed for the specific type of distortion. As aforementioned, blurring is the most common distortion, so only the sharpness assessment approaches are summarized as follows. Ferzli et al. [14] proposed the just noticeable blur (JNB) algorithm. Through the combination of the concept of JNB with the probability sum model, the algorithm was used to predict the relative amount of the blurred image with different contents. In [15], Bahramiet al. defined the maximum local variation (MLV) for each pixel as the maximum intensity difference between the pixel and its 8-neighborhood, and the standard deviation of the MLV distribution was defined as the sharpness score. In [16], Li et al. proposed a blind image blurevaluation (BIBLE) algorithm that the utilization of discrete Tchebichef moments formeasuring the variation of image shape caused by image blur distortion. The second category of NR quality metrics are for general distortions. In [17], Saad et al. used the Bayesianinference mode to predict the quality of image. The feature was based on natural scenestatistical model of image discrete cosine transform coefficients. Zhang et al. [18] proposed aquality evaluation algorithm using log-derivative statistics of natural scenes. Statistical features related to image quality were extracted at two image scales in both spatial and frequency domain. Liu et al. [19] proposed a spatial-spectral entropy-based quality (SSEQ) index model to predict the quality of the image. Although these algorithms have achieved great advances on natural images, they can not effectively evaluate the quality of SEM images, which are different from natural images.

 The SEM images are grayscale images, they look gloomy on the background and show the surfaces of objects. So the content on SEM image can give rise to low intensities in the dark channel [20]. In addition, it is found that is sensitive to the changes toblurred images caused by the blur process [21]. The blurred images have fewer dark channeland different blurred images have obvious differences in the dark channel maps. Based on these observations, we propose a NR sharpness index for SEM images based on dark channel prior. Due to the lack of SEM image database, we first build a database and collectsubjective scores of SEM images through user study. Because blurred image is typically characterized by the spread of edges, the edge of dark channel maps are extracted. Then anedge-preserving operator is employed to remove noise effect but preserve the edge information based on the weighted least squares (WLS) framework. Finally, the maximum gradient and the average gradient are combined to generate the sharpness score of the SEM image. The experimental results on the SEM blurred image database demonstrate that the proposed method outperforms the state-of-the-art metrics.

2. Database establishment and subjective experiment

 Image databases play an important role in understanding how people perceive the quality of images, and they are crucial for designing and evaluating IQA models in terms of the consistency with human subjective evaluations. For SEM, the quality evaluation algorithm which can guide the selection of imaging parameters is highly desired. To the best of our knowledge, such a database for SEM images is still lacking. Therefore, we need to build adatabase to test the performance of the methods. In this section, we build a SEM database, followed by a subjective test to collect quality scores.

2.1. Database establishment

 All SEM images were taken using a scanning electron microscope from the Advanced Analysis and Computation Center in China University of Mining and Technology. The SEM image database contains 50 groups of images with diversified contents, including ants, metals, coal particles, colloids, etc. Fig. 1 shows eight clear SEM images of differentsubstances. For each image, there are 3 blurred distorted versions, the size of the image is 1024 * 884, a total of 150 images. The blurring degrees of SEM images are related toscanning time. We can get SEM images with different blurring degrees by adjusting differents canning time.

 

Fig. 1. Clear SEM images of 8 different substances

2.2. Subjective experiment

 The subjective experiment is conducted using a PC of Intel (R) Pentium (R) CPU G6452.9GHz. The image is displayed on a LCD with a resolution of 1920*1080. The experimentis conducted in a normal light indoor workplace. Thirty subjects with no knowledge of image processing participate in the subjective experiment. Besides, an interactive platform is designed to display the test images. Meanwhile, subjects also learned the principles of platform operation and criterion of image quality rating. The perceive image quality of the experiment is divided into five levels [22], corresponding to consecutive 1-5 points, where 5 is the best and the 1 is the worst. The subjects click to select the perceived image quality score. Thirty subjects rate all 650 images. After completing the subjective experiment, each 2532 test image has 30 subjective scores from 30 subjects. In order to reduce the error of experimental results, five outliers are removed from 30 ratings for each image, leaving only 25 valid ratings. Meanwhile, the average value of the 25 valid data represents the subjective experimental result of the image. We use 150 blurred images and the corresponding meanopinion score (MOS) values to prepare further study. Fig. 2 shows an original image and the corresponding images with different levels of blur.

 

Fig. 2. The original SEM image and the corresponding images with different levels of blurring: (a) MOS=5.0000; (b) MOS=4.3200; (c) MOS=3.1739; (d) MOS=1.8366.

3. Proposed SEM image sharpness metric

 Due to the very narrow electron beam, SEM images have a large depth of field producing acharacteristic of three-dimensional appearance for understanding the surface structure of asample. According to the characteristics of SEM images described above and the ability of to captures the changes to blurred images [21], the dark channel maps arefirst generated. The study has shown that the human visual system has multi-channel characteristics [23]. Each layer of images owns individual features, and high frequency partmainly provides the edge and contour information of the image. The essence of image blurring is that the high frequency component of the image is attenuated. Therefore, the image edge can reflect the blurring degree. Based on the above points, the edge of dark channel maps are extracted. Then an edge-preserving smoothing filter is used to removenoise while enhancing the edge information based on the weighted least squares (WLS). According to the characteristics of the human visual system, the maximum gradient and the average gradient are integrated to generate the final quality score of the SEM images. Fig. 3 shows the flowchart of the proposed metric.

 

Fig. 3. The flowchart of the proposed metric

3.1. Dark channel

 SEM is the interaction of electrons and material to obtain the surface morphology of the sample. Pan [21] et al. found that some pixels that in clean image are not dark when averaged with neighboring high intensity pixels during the blur process. And the dark channel [20] maps are different with different blur degrees of SEM images.

 For an image I, the dark channel is defined as:

\(D(I)(x)=\min _{y \in \Omega(x)}\left(\min _{c \in\{x, g, b\}} I^{c}(y)\right),\)       (1)

where x and y are the positions of the pixels and is the center of the image block at x, which I^c is the c-th color channel. As the SEM images are grayscale images, the equation [21] can be simplified as:

\(D(I)(x)=\min _{y \in \Omega(x)}(I(y)),\)       (2)

 

Fig. 4. The original blurred image and dark channel maps with three different block sizes

 Fig. 4 shows the original blurred image and the dark channel maps calculated using threedifferent block sizes. For the small block size, the image is not changed in comparison with the original image. By contrast, too big block size tends to generated dark channel maps with much less information maintained. In this work, the size of image block is empirically set to 15* 15.

3.2. Edge detection

 In the process of image quality evaluation, edge features can be used to improve the accuracy of image quality evaluation. Therefore, the method of extracting edge features for SEM image quality evaluation can greatly reduce the amount of calculation and improve the efficiency of calculation. In addition, when the image distortion is heavy, the edge feature is more easily detected. Fig. 5 shows different blur levels of the SEM images and Fig. 6 shows dark channel maps that the 15*15 block size of three blur degrees. Through observation from Fig. 6, it can be seen that the texture is simple and the edge is more prominent.

 

Fig. 5. Different blur levels of the SEM images

 

Fig. 6. Dark channel maps that the 15*15 block size of three blur degrees

 After dark channel processing, the Sobel edge operator is used to filter the horizontal and vertical direction of the image, producing the edge map.

 The Sobel operator is defined as:

\(G=\left|G_{x}\right|+\left|G_{y}\right|,\)       (3)

where G_x and G_y are defined as:

\(G_{x}=\left[\begin{array}{lll} -1 & 0 & 1 \\ -2 & 0 & 2 \\ -1 & 0 & 1 \end{array}\right] * I\)       (4)

\(G_{y}=\left[\begin{array}{ccc} 1 & 2 & 1 \\ 0 & 0 & 0 \\ -1 & -2 & -1 \end{array}\right] * I\)       (5)

3.3. Edge preserving smoothing filter based on weighted least squares

 The Weighted Least Squares Framework (WLS) [24] is a non-linear, edge-preserving smoothing method that captures the details of various scales by multi-scale edge-preserving decomposition. The edge preserving smoothing filter based on weighted least squares is used to remove the noise while maintaining the strength of the edge.

 For one input images G, on the one hand, we want the target image U to be as close aspossible to G, while U should be as smooth as possible except for some places where theedge gradient changes a lot in G. Formally, we have:

\(F_{\lambda}(g)=\left(I+\lambda L_{g}\right) U=G,\)      (6)

where \(L_{g}=D_{x}^{T} B_{x} D_{x}+D_{y}^{T} B_{y} D_{y}\). D_x and D_y are discrete differentiation operators. B_x and B_y are smoothness weights, the smoothness is required to be performed in a spatially varying manner which depending on U. λ is the balance factor between the data item and the smoothed item. λ controls the smoothness and increases the value of the result in the progressive smoothed image.

 Fig. 7 shows the image edge of different blurring degree after dark channel processing. It can be seen from the figure that there are much noise in the image, which seriously affects the quality of the image. Fig. 8 shows the images after the edge-preserving filtering. It can beseen from Fig. 7 and Fig. 8 that the noise is weakened and the edge information remains intact.

 

Fig. 7. The image edge of different blurring degree after dark channel processing

 

Fig. 8. The images after the edge-preserving filter

3.4. Quality score

 Zhan et al. [25] found that the maximum gradient can be effective for image sharpnessassessment. And the average gradient reflects the sharpness of the global content of the image. To eliminate the influence of the image content, the maximum gradient and the average gradient are combined to generate the final sharpness score.

 The maximum gradient and the average gradient of the image are defined as:

\(M G=\max (G(i, j)),\)       (7)

\(A G=\left(\sum_{i, j} G(i, j) /(w \times h)\right),\)       (8)

where MG and AG are the maximum gradient and the average gradient respectively, w and wand h are the size of the image.

 Finally, the SEM image sharpness score is defined as:

\(I Q A=M G \times A G^{-\alpha}\),       (9)

 In this work, α is empirically set to 0.4366.

4. Experimental results and analysis

4.1. Experiment Settings

 To evaluate the performance of the proposed method, we adopt three commonly used criteria, including Pearson linear correlation coefficients (PLCC), Spearman rank order correlation coefficients (SRCC) and root mean square error (RMSE) [26]. PLCC and RMSE measurethe accuracy of algorithm prediction, while SRCC is used to measure the prediction monotonicity. The objective quality scores and subjective scores are mapped to the samescale [27] by nonlinear fitting function:

\(f(v)=\tau_{1}\left(\frac{1}{2}-\frac{1}{1+e^{\tau_{2}\left(v-\tau_{3}\right)}}\right)+\tau_{4} v+\tau_{5}\),       (10)

where τ_i (i=1,2,…5) are the fitted parameters.

4.2. Performance Evaluation

 1) Comparison with NR-IQA sharpness metrics: In this part, we compare the performance of the proposed method with eleven existing no-reference image sharpness metrics on the SEM blurred image database, which are JNB [14], CPBD [28], Marz. [29], LPC [30], MLV[15], BIBLE [16], S3 [31], SPARISH [32], FISH [33], ARISM [34], SVC [35]. Table 1 summarizes the experimental results, where the best performance values are marked in bold face.

Table 1. Performance of the proposed method and state-of-the-art sharpness metrics

 

 It can be seen from Table 1, the performances of proposed algorithm is better than otherexisting sharpness/blur algorithms in terms of monotonicity and prediction accuracy. This proves that SEM images are unnatural images. The quality evaluation methods based onnatural images cannot evaluate the quality of SEM images very well. Therefore, the method of this paper which is designed for the characteristics of SEM images has achieved the best performance.

 2) Comparison with general-purpose NR-IQA metrics: Several common NR image quality metrics proposed in the previous literature can predict the image quality without knowing thespecific distortion type. So we further compare the proposed algorithm with the state-of-the-art general-purpose NR quality models on the SEM blurred image database, including BRISQUE [36], DESIQUE [18], BIQI [37], DIIVINE [38], BLLINDS-II [17], NIQE[39], SSEQ [19], QAC [40]. Table 2 summarizes the experimental results, where the best performance values are marked in boldface.

 It can be seen from Table 2 that no matter the prediction accuracy or monotonicity, the proposed algorithm achieves the best performance among all compared algorithms. This also proves that the SEM images are unnatural images, and the general methods based on naturalimages are not suitable for SEM images.

Table 2. Proposed method and the state-of-the-art general algorithms

 

 3) In order to compare the statistical significance of the proposed method and the existing metrics, we further conduct the F-test [41]. Fig. 9 shows the F statistics of the compared metrics against the proposed method. During the experiment, 95% confidence level is used for the F-test. Table 3 and Table 4 are the statistical performance results of the sharpnessimage quality metrics and the general-purpose image metrics with the proposed method, respectively. The value of 1 indicates that the proposed method performs statistically betterthan other methods, and 0 indicate that the performance of the proposed method is statistically equivalent to the other methods. It can be seen from the table that the performance of proposed method is obviously superior to all the no-reference sharpnessindicators and the five general-purpose NR quality metrics. The remaining three general-purpose NR metrics are statistically equivalent to the performance of this method.

 

Fig. 9. The F statistics of the compared metrics against proposed method. (a) NR-IQA sharpnessmetrics with proposed metrics; (b) General-purpose NR-IQA metrics with proposed metrics.

Table 3. Statistical performance results of the sharpness image quality metrics with proposed method

 

Table 4. Statistical performance results of the general-purpose image metrics with proposed method

 

4.3. Impact of dark channel block sizes

 In order to evaluate the impact of dark channel block sizes on the performance, we have tested different block sizes, ranging from 3*3 to 27*27. Table 5 lists the results of PLCC, SRCC and RMSE with different block sizes. It can be seen from the table that the performance varies with the sizes of the block. Moreover, we can see that the best results are obtained when the block size is 15*15. Therefore, 15*15 block size is used in this paper.

Table 5. The results of PLCC, SRCC and RMSE with different block sizes

 

5. Conclusion

 In this paper, due to the captures the changes to blurred images, we have proposed a no-reference sharpness index of SEM images based on . The SEM image database was built and a subjective experiment has been done to collect the ground truth of subjective scores. Experimental results on SEM blurred image database have shown that the proposed method is superior to the existing state-of-the-art algorithms. Although the proposed method has achieved notable improvement, it has the followinglimitations. First, the database size is limited, which needs to be enlarged. Second, it is specifically designed for the sharpness evaluation of the SEM images. However, in practice, the distortions introduced to the SEM images are diverse. That is to say the SEM image may be multiply-distorted. Therefore, designing a quality metric that is effective for the evaluation of the multiply-distorted SEM images is desirable. In the future, we will dedicate to not only constructing a large database but also designing the effective metric for the SEM images with multiple distortions.