Near-infrared face recognition by fusion of E-GV-LBP and FKNN

Abstract

To solve the problem of face recognition with complex changes and further improve the efficiency, a new near-infrared face recognition algorithm which fuses E-GV-LBP and FKNN algorithm is proposed. Firstly, it transforms near infrared face image by Gabor wavelet. Then, it extracts LBP coding feature that contains space, scale and direction information. Finally, this paper introduces an improved FKNN algorithm which is based on spatial domain. The proposed approach has brought face recognition more quickly and accurately. The experiment results show that the new algorithm has improved the recognition accuracy and computing time under the near-infrared light and other complex changes. In addition, this method can be used for face recognition under visible light as well.

1. Introduction

In recent years, face recognition as a classic pattern recognition subject, has been widely studied and applied [1]. To solve the problem of face recognition with illumination interference, Ziqing Li, Professor of the Chinese Academy of Sciences, and his research team firstly proposed a face recognition method based on Near-Infrared (NIR) technology [2]. This method had been used in the 2008 Beijing Olympic Games. Besides, it had received concerns and recognition from international counterpart experts. Comparing with the traditional visible face recognition, the near-infrared face recognition has obvious advantages. However, it also has limitations. For example, there are lots of visible light image that can not re-use. Then, we have to re-build the near-infrared face database. So the recognition is not flexible and need to improve the real-time face recognition. Therefore, the success of near-infrared face recognition depends on a reasonable strategy. That means we need a feature extraction [3] method robust to multi-criteria and a faster classifier [4].

Comparing with the global feature extraction algorithm, local appearance feature extraction algorithm is more stable for illumination, expression and angle changes. Gabor [5] and Local Binary Pattern [6] are two representative characteristics. Some researcher use Adaboost algorithm to select Gabor features and reduce dimension [7]. However, due to high time complexity of Adaboost and high dimensions of the initial Gabor feature, the selection process is time consuming. In short, this feature selection method is not suitable for face recognition. The Local Binary Pattern was first proposed by Ojala [8] et al. It is a texture operator with stronger identification and simple calculation. LBP can be used to analyze the real-time image. But the range of application is limited. Besides, the widely used classification methods are decision tree, neural network, KNN, SVM and Bayesian. Among them, the K-nearest neighbor (KNN) classification algorithm [9], with simple and accurate auto classification, has been widely applied. When the KNN algorithm classified the sample with many properties, the efficiency would be greatly reduced. That is due to a large quantity of computing. Then, many researchers have focus on how to improve the deficiency of KNN.

Based on above problems, we presented a new algorithm which is based on NIR face recognition. That fuses Effective Gabor Wavelet LBP (E-GV-LBP) feature extraction method [10] and the improved Fast KNN (FKNN) classification. Firstly, it transforms the pre-processed NIR image by the multi-scale, multi direction Gabor wavelet. To extract more useful Magnitude and Phase features, LBP algorithm builds the relational model by common space, scale and direction information. Secondly, it cascades the extracted sub-block feature histograms. Afterward, there generates an enhanced histogram. Finally, according to the aggregation of feature vectors, it calculates the spatial distribution of each kind of feature vectors. That is called Fast KNN algorithm which could quickly classify the enhanced histogram. The experiment on PolyU NIR [11] face database shows that the new algorithm’s efficiency is significantly better than the original algorithm. In addition, based on the visible light face database such as Yale [12], ORL [13] and CMU PIE [14], the recognition efficiency has also been improved. It shows that the new algorithm can solve the problem of universality.

 

2. The E-GV-LBP for feature extraction

2.1 Gabor wavelet

Gabor wavelet is a group of complex function system generated by the scaling and angle rotating. It has good localization and direction selectivity characteristics. When used Gabor wavelet filters, the image could be extracted partial changes. Gabor usually takes two-dimensional wavelet for transform. The kernel function is defined as follow.

where μ and v respectively is the direction and scale of the kernel function. z = (x, y) and the wave vector is defined as follows.

where kv = kmax / fv , kmax = π / 2 , f = and ϕμ = πμ/8 .

From the above equation, Gabor filter set is generated by a series of wave vectors kμ,v . Besides, kμ,v has self-similarity. Usually, the Gabor kernel function has five scales v ∈ {0,1,2,3,4}, eight directions μ = {0,1,2,3,4,5,6,7} and one parameter σ = 2π [15].

The input image is respectively connected with each filter clusters by convolution. Then, corresponding convolution images are amplitude images A and phase images ϕ. The calculation is shown as follows.

where Creal and Cimag respectively is real component and imaginary component of the Gabor filtered image.

Therefore, a single face image is filtered by multi-scale Gabor which will be a group of phase and amplitude face set. Among that, the phase information is changing with the cyclical illumination and robust to changeable light. Similarly, the amplitude information is relatively smooth and stable [16] as well.

2.2 E-GV-LBP algorithm

In recent years, most studies [17-19] have proved that combining Gabor with Local Binary Pattern (LBP) is an effective face recognition method. LBP is an operator for local visual characterization. The main idea of LBP is encoding the image binary pixel by threshold. In the 3×3 neighborhood, it encodes the binary threshold result by center pixel and corresponding neighborhoods.

In Fig. 1, it selects eight points around the neighborhood. When the value is less than the center pixel marked 0 and more than the center pixel marked 1. Then, there generates a binary code and translates to decimal mode. The result is ranged from 0 to 255. Based on the algorithm, E-GV-LBP [10] codes the image space, scale and orientation information at the same time. The method is shown in Fig. 2.

Fig. 1.The encoding computation of LBP operator

Fig. 2.The Formulation of E-GV-LBP

For the center pixel Ic, I0 and I4 are the orientation neighboring pixels, I2 and I6 are the scale neighboring ones, I1, I3, I5 and I7 are the neighboring pixels in spatial domains. The coding method of E-GV-LBP operator is similar to LBP.

where S(Ip - Ic) is a threshold function and is defined as follows.

With the change of space, it would generate different Gabor face. In case of that problem, E-GV-LBP algorithm codes image space, scale and orientation information at the same time. Then, it gets amplitude and phase Gabor feature set. So, it has a large dimension of feature vectors. The histogram has been proven that could effectively reduce the feature dimension and more reliable. The detail calculation will be introduced in the experimental analysis.

 

3. Improved fast KNN classification

3.1 KNN algorithm

K-nearest neighbor (KNN) algorithm was advanced by Cover and Hart [20] in 1968. It is one of the simplest machine learning algorithms. Besides, it is also a mature theoretical method. The procedure of face recognition application is divided into four steps. Firstly, calculate the distance between the unclassified sample and various types of samples. Next, choose the nearest K samples from the classified samples. Then, do some statistical analysis for the top K samples classification category. Finally, the unclassified sample belongs to the top classification.

Sometimes, a certain kind of the sample number is much larger than the others. That is called sample imbalance. In that case, it might lead to more other samples in one sample’s K-nearest neighborhood. Then, we could take the method with weigh to avoid the above problem. In addition, to obtain a K-nearest neighbor, the algorithm must calculate the distance with each sample in database. Due to the larger amount of computation, such as the large training sample attributes, the algorithm efficiency would be reduced. Currently, the widely used methods to solve this kind of problem is to edit for sample points in advance and removing the little effect sample. Nevertheless, it is more prone to be an error classification for small samples. Consequently, the ordinary KNN algorithm is not adaptive for variable samples. The following paper introduces a new classification algorithm. That is based on spatial domain and an improved algorithm called fast K-nearest neighbor (FKNN).

3.2 FKNN classification algorithm based on the spatial distribution

From the perspective of spatial distribution, face feature vectors will show an extent of aggregation. That is to say, the majority feature vectors from one person will be distributed in the same area. The feature vectors from different person will be distributed in different areas. In machine learning, support vector machines [21] are supervised learning models with associated learning algorithms that analyze data and recognize patterns, are used for classification and regression analysis. An SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. However, the SVM is difficult to implement for large-scale training sample and more difficult to solve multi-classification problems.

Based on the above theory, we will introduce a new classification algorithm which one has the self-adaptive capacity. When processing a large number of samples, the SVM algorithm calculation will cost a lot of machine memory and computing time. So, the new method just does some statistic for the intra-class vector distribution in training process and measures the inter-class distribution in testing process. Then we could recognize the large-scale samples. Due to that, it is the evolution of the SVM. Besides, to improve the efficiency of calculation greatly and solve the multi-classification problems, it uses the KNN algorithm as the distance measurement method. There is classified multi-class at the same time and the calculation faster than KNN. Overall, it is defined as the Fast K-nearest neighbor (FKNN) classification. The calculation is shown as follows.

(a) Constructing the spatial distribution for each class samples

The spatial distribution domain takes a circle as the unit. So, each class sample’s center position of the circle domain in space is defined as the center feature vector. Based on each sample’s size and the whole spatial distribution, it has respectively calculated the outer and inner radius. Finally, it has formed two concentric circles distribution domain with different radii. The calculation is shown as follows.

In Fig. 3 , O(X) is the center feature vector of class X. S is an ascending order vector set. The set contains the distance between O(X) and whole vector space. dist(vi, O(X)) is the distance between vi and O(X) . tI and tO are defined as the discrimination threshold for inner and outer radius. If the threshold t is unsuitable or the spatial distribution has smaller training samples, the inner radius rI may be greater than the outer radius rO . For computing the distribution domain of different size samples and ensuring that reasonableness, it would exchange the value of rI and rO .

Fig. 3.Fast KNN Algorithm

(b) Locating the position of vector u in feature space

The distance method is used in the commonly classifier. Such as Hamming distance, cosine distance and Euclidean distance. In this paper, it calculates the Euclidean distance and the calculation is shown as follows.

where X = (x1,x2,...,xn), Y = (y1,y2,...,yn) and n is the dimension of the sample faces. In this paper, X and Y are sample feature vectors.

(c) Judging and classifying based on the relationship between u and the according spatial distribution

From the above step, it will determine the sample u spatial location. As shown in Fig. 4(a), if the sample u (red dots) distributes in the space with inner radius, u belongs to this class. Then, the classification is over. In Fig. 4(b), if the sample u (red dots) distributes out of the space with outer radius, u doesn’t belong to it. Then, it eliminates this category in the next k nearest neighbor domain. The classification is going on. Otherwise, if the sample u (red dots) distributes between the outer radius and inner radius area as Fig. 4(c), further classify with KNN algorithm. Until determinate the sample u, the classification is finished.

Fig. 4.The distribution of vector u and spatial domain

When construct the various spatial domain, the inner and outer radius depend on the distribution performance of each sample. That is said, Compared with SVM, FKNN classification algorithm is robust to large-scale and multi-classification recognition. Because of the representation ability for samples actual spatial distribution, FKNN is suitable for different samples. In the FKNN classification, if the unclassified vector u is directly determine as one class, there will reduce the computing time of KNN algorithm. On the other hand, if the vector u locate in some outer radius spaces, there will narrow the search area range of u. Compare with KNN, FKNN algorithm has been significantly improved the computation efficiency. In addition, the sample vector within the inner radius region has an entirely correct classification. The vector out of the outer radius region has been excluded some uncorrelated classes. It means that the vector does not belong to the class. In other words, the recognition accuracy of FKNN algorithm has non-negative impact.

Generally speaking, FKNN is a classification algorithm with self-adaptive capacity. In the next chapter will analysis that the FKNN algorithm could improve the recognition accuracy and reduce the computing time complexity.

 

4. Experimental Analysis of Face Recognition

To verify the efficiency of this face recognition algorithm, we used the face database to do the experiments. The experiment mainly to classify samples based on near-infrared face database. We selected the optimal parameters for each algorithm. Then, it analyzed the algorithm ability of characterization and classification in detail. In addition, in order to further verify the algorithm performance and the application range with optimal parameters, we also had a simulation experiment for visible face database Yale, ORL and CMU PIE.

In experiments, the image firstly was pre-processed and then was extracted the E-GV-LBP features on sub-block. As mentioned above, it would get 40 amplitude and phase E-GV-LBP faces respectively. To avoid the amount of feature data, the results are shown by the statistics histogram. H is a statistical value of the corresponding pixel. Then the dimension reduction and the calculation are shown as follows.

where fi (x, y) is the encoded value of coordinate point (x, y) in number i feature image. The n means that the total image number and is assigned as 40 in this paper. I (.) ∈ {0,1} is a Boolean function.

Finally, it got the enhanced statistical histogram and identifies the unclassified sample based on FKNN algorithm. The above face recognition experiment analysis is shown as follows.

4.1 Face recognition based on near-infrared light

Near-infrared image is constituted by the electromagnetic waves. The light is between the visible and middle infrared light. NIR has preserved more detail information than the thermal middle infrared image and recognizable. Besides, the NIR image with gray information is easier process than colorful visible image. Based on the above characteristics, NIR image has better adaptability for ambient light face recognition.

The recognition experiment samples are from the Hong Kong Polytechnic University near-infrared (PolyU NIR) face database.(HTTP://www4.comp.polyu.edu.hk/~biometrics/polyudb_face.htm). Portion images of this database are shown in Fig. 5. The face database contains 350 classes. One class has 100 samples including posture, facial expression, focus, scale, time and other changes. The total image number is 35000. To evaluate different methods performance on PolyU NIR database, there are three types of experiments. Each of experiment contains a training set, a target (Gallery) set and a query (Probe) set. In Exp#1, the used images include frontal face images as well as images with expression variations, scale changes (include blurring), time difference, etc. In Exp#2, it adds more faces captured in uncontrolled conditions to make the test more challenging. In Exp#3, it focuses on the images with high pose variations and excludes the images with expression, scale and time variations. Being unify pre-processed, the image size is 64×64. In order to achieve a better near-infrared face recognition, following content will select and analysis the parameters of the new algorithm.

Fig. 5.A part of faces from PolyU NIR database

As Fig. 6(a) shows, for effectively combining local and global image texture information, this paper cascades sub-block local features and forms the enhanced histogram. However, the inappropriate sub-block will lost part of spatial information or lead the method sensitive to local variations. That means it will reduce recognition rate. Therefore, selecting the appropriate block size for PolyU NIR facial image recognition is necessary. In Fig. 6(b), there is the changing relationship between non-overlapping block and recognition results. The result shows that the recognition rate is increasing with the sub-block and the recognition time is decreasing with the sub-block. When the sub-block size is larger than the 8×8, the recognition time has an increasing trend. Therefore, this paper selects the 8×8 sub-block for experiments.

Fig. 6(a)Feature extraction for block face

Fig. 6(b)The efficiency of each block image

When classify the histogram feature by FKNN algorithm, it needs to calculate the outer and inner radius for spatial domain. However, to a certain extent, the radius depends on the parameter t. The Fig. 7 shows the quantitative comparison of threshold parameter t and corresponding recognition time. According to the definition of threshold t, Fig. 7(a) is the FKNN algorithm recognition time under the inner radius parameter in [0,1] interval. Fig. 7(b) is the recognition time under the outer radius parameters in [0, 0.05] interval. For the function of quickly recognition, it chooses = 0.95, = 0.02 as FKNN threshold parameters. In Fig. 7, it has a fastest recognition time.

Fig. 7.FKNN threshold t and the corresponding recognition efficiency（s）

In addition, the literature 11 puts forward DBC recognition algorithm for PolyU NIR face database. It also bases on Gabor wavelet feature extraction. According to the above optimal parameters, Fig. 8 is the comparison of new algorithm and DBC recognition rate. In Fig. 8, all other conditions remain the same except the algorithm.

Fig. 8.Algorithm of face recognition rate based on PolyU NIR (%)

By the comparing result, this recognition algorithm for near infrared face is more attractive. Then, we would further analyze and validate the performance of this new algorithm. It combines the LGBP algorithm [22] and E-GV-LBP algorithm with KNN and FKNN by pairwise. The corresponding experiment result is shown in Table 1 .

Table 1.Algorithms recognition result (recognition rate % and recognition time s)

By different combination algorithm, the E-GV-LBP algorithm has a higher recognition than the LGBP algorithm and has the robust to variable light. That is not only in Exp1 with facial positive changes but also in Exp2, Exp3. Meanwhile, the integration with FKNN algorithm is faster than KNN. In summary, this paper proposes a new near-infrared face recognition algorithm and the algorithm has improved recognition performance.

4.2 Face recognition based on the visible light

Experiment I: the face recognition base on Yale face database. Yale database was founded by the visual and curb Center of Yale University. It contains 165 images from 15 volunteers including 11 different variations. Samples include glasses occlusion, Illumination and expression changes. Images are processed uniform size 64×64. It is sequentially selected one to six images as training samples and recognized the remaining images. According to the above selected parameters, the accuracy and time are the average value of twenty times recognition. The result is shown in Table 2.

Table 2.Recognition results based on Yale database (recognition rate % and recognition times)

From the experiment results, we could summarize the following conclusions. For the face recognition with visible light changes, glasses occlusion and variable expression, the proposed algorithm has certain robustness. As can be seen in the Table 2, the new algorithm has an obvious advantage on recognition rate with the increasing number of training samples. In addition, compared with four fusion methods calculate time, the new algorithm has the shortest time and integrally shows a decreasing trend. In short, as the training sample set of the rich, the improved recognition algorithm is more accurately and quickly than before. Due to the method is based on the accurately spatial distribution for various samples.

Experiment II: the face recognition is based on ORL face database. ORL face database was founded by AT & T Laboratories of Cambridge University. It includes 400 images from 40 people under the postures, facial expressions and facial occlusion changes. Then, it is processed uniform size 66×66. In this experiment, there are five groups from one to five images as training samples and recognized the remaining images. According to the above selected parameters, the accuracy and time was the average value of twenty times recognition. The result is shown in Table 3.

Table 3.Recognition results based on ORL database (recognition rate % and recognition times)

From the experiment results, we could summarize the following conclusions. The proposed algorithm also has robustness for visible light face recognition with gesture, facial expression and variable light. Besides, it has a more obviously advantage of rapidly recognition. Comparing the recognition rate of Experiment II with Experiment I, the E-GV-LBP algorithm is also robust to gesture changes. In addition, comparing the first group with other groups, the new algorithm has a straightly rising recognition rate. So do the fusion method of LGBP and FKNN. That means FKNN is suitable for various training samples. With the higher recognition accuracy and the shorter computing time, there are further validate of the new algorithm applicability for diversity face recognition.

Experiment III: the face recognition is based on CMU PIE face database. CMU PIE face database contains 68 volunteers and one person with twenty-one kinds of changes. The images are processed unified size 64×64. In this paper, we used CMU PIE face database to further validate the robustness of illumination recognition. Meanwhile, the above experiments show that the algorithm is more suitable for variable recognition. Therefore, this experiment increases the number of training samples until the recognition results stabilized. As showing in Table 4, there are the selected odd-number training samples and corresponding recognition time. Then, as showing in Fig. 9, it further verifies the stability of the recognition algorithm. The sequentially number is the training face samples. And we have recognized the remaining samples. According to the above selected parameters, the accuracy and time are the average value of twenty times recognition.

Table 4.CMU PIE database recognition times ( s)

Fig. 9.Algorithm recognition rate based on CMU PIE database(%)

From the experiment results, we can summarize the following conclusion. The face recognition algorithm is robust to visible light and remains stable more faster. When in illumination changes sample sets, the spatial distribution has strong ability of distinguish between various samples. Then, the method by fuse the FKNN is more effective. As showing in Fig. 9, the algorithm with FKNN has a faster trend to steady recognition rate. Overall, for the diversity of the face recognition under illumination change, this new algorithm can quickly local the spatial distribution and accurately recognize it.

 

5. Summary and conclusions

This paper presented a fusion of E-GV-LBP and FKNN algorithm for near-infrared face recognition method. The algorithm firstly took multi-scale Gabor wavelet to transform image and code the LBP structure model by space, scale and orientation information at the same time. Then, it calculated the samples outer and inner radius to build the spatial domain. Finally, to verify the efficiency, it used FKNN method which is based on the spatial distribution to classify the enhanced histogram. The FKNN is a new self-adaptive classification algorithm. In this paper, it presented an experiment for near-infrared face database and an experiment program for visible face database. That not only proved the algorithm performance, but also analyzed the algorithm in various respects. The experiment results show that the new recognition method is robust to near-infrared and visible light face. Besides, the face recognition computation is faster than other algorithms. Especially, for the various training samples recognition, we believe that the new algorithm has more advantages. According to the conclusion of this paper, we will continue to optimize the algorithm and focus on combining infrared light with visible light for face recognition and video analysis [23].