Video Palmprint Recognition System Based on Modified Double-line-single-point Assisted Placement

Abstract

Palmprint has become a popular biometric modality; however, palmprint recognition has not been conducted in video media. Video palmprint recognition (VPR) has some advantages that are absent in image palmprint recognition. In VPR, the registration and recognition can be automatically implemented without users' manual manipulation. A good-quality image can be selected from the video frames or generated from the fusion of multiple video frames. VPR in contactless mode overcomes several problems caused by contact mode; however, contactless mode, especially mobile mode, encounters with several revere challenges. Double-line-single-point (DLSP) assisted placement technique can overcome the challenges as well as effectively reduce the localization error and computation complexity. This paper modifies DLSP technique to reduce the invalid area in the frames. In addition, the valid frames, in which users place their hands correctly, are selected according to finger gap judgement, and then some key frames, which have good quality, are selected from the valid frames as the gallery samples that are matched with the query samples for authentication decision. The VPR algorithm is conducted on the system designed and developed on mobile device.

Ⅰ. INTRODUCTION

Accurate user authentication and authorization control are the key functions on Internet. The possessions for authentication can be stolen, broken, or forged; the passwords for authentication can be forgotten or attacked. Biometric recognition realizes authentication based on the inherent physical or behavioral characteristics of human beings [1]; which overcomes the drawbacks of the traditional authentication technologies [2].

Palmprint has rich discriminative features, including main lines, wrinkles, ridges and minutiae points. In addition, palmprint recognition can have low equipment requirements, fast speed, and high accuracy; therefore, it is widely deemed as an important biometric modality.

Video palmprint recognition (VPR) has some advantages that are absent in image palmprint recognition. In VPR, the registration and recognition can be automatically implemented without users’ manual manipulation. A good-quality image can be selected from the video frames or generated from the fusion of multiple video frames [3]. VPR in contactless mode overcomes the problems caused by contact mode.

● Health risk: Due to the health and personal safety, it is unhygienic for users to contact the identical sensors or devices. Contact acquisition definitely increases the risk of infectious diseases, such as COVID-19.

● Low acquisition flexibility: Contact acquisition suppresses the users’ acceptance, flexibility and comfortability.

● Surface contamination: The surface of contact sensors can be contaminated easily, especially in harsh, dirty, and outdoor environments. The surface contamination of contact sensors is likely to degrade the quality of the subsequent acquired biometric images. In addition, the biometric features can be retained on the surface of contact sensors, which increases the risk of leaking biometric privacy.

● Resistance from traditional customs: Some conservative nations/nationalities resist the users of different genders to contact the identical devices.

Although VPR in contactless mode overcomes the aforementioned problems, it encounters with several revere challenges such as complex background, various illuminations, uncontrollable hand placement (pose and location)[4].

This paper develops practical VPR algorithm and system. The main contributions are summarized as follows.

(1) Double-line-single-point (DLSP) assisted placement technique can overcome the challenges as well as effectively reduce the localization error and computation complexity. Modified DLSP (MDLSP) technique is developed to reduce the invalid area in the frames.

(2) The valid frames, in which users place their hands correctly, are selected according to finger gap judgement, and then some key frames, which have good quality, are selected from the valid frames as the gallery samples that are matched with the query samples for authentication decision.

(3) The VPR algorithm is conducted on the system designed and developed on mobile device.

The rest of this paper is organized as follows. Section 2 revisits the related works. Section 3 specifies the methodology. Section 4 are the experiments and discussions. Finally, the conclusions are drawn in Section 5.

Ⅱ. RELATED WORKS

2.1. Preprocessing

The purpose of palmprint image preprocessing is typically to accurately segment, localize and crop the region of interest (ROI) [5]. Palmprint recognition can be divided into contact mode, contactless mode and mobile mode. Mobile mode is a special case of contactless mode, which can be considered as the most difficult contactless mode. The preprocessing on contact mode is easier than those on the other two modes. Table 1 summarizes the palmprint preprocessing methods in contactless mode.

Table 1 Contactless palmprint preprocessing.

2.2. Feature Extraction

Palmprint recognition methods can be briefly divided into five categories [16], namely coding-based methods, structure-based methods, subspace-based methods [17], statistics-based methods [18], deep-learning/machine- learning based methods. Fusion technologies are also been used in palmprint recognition [19,20]. Because coding-based methods can be free from training and have low storage and computational complexity, they are suitable for edge computing devices such as mobile phones with low-performance hardware. Therefore, coding-based methods, including PalmCode [21], OrdinalCode[22], FusionCode[23], CompCode[24], RLOC[25], BOCV[26], E-BOCV[27], DCC[28], DRCC[28], DOC[29], are employed in our VPR system.

Ⅲ. METHODOLOGY

The key frame selection in VPR system is shown as in Figure 1. The valid frames, in which users place their hands correctly, are first selected according to finger gap judgement, and then the key frames, which have good quality, are selected from the valid frames according to quality judgement. Some key frames at registration are used as the gallery samples; while each key frame at authentication is used as a query sample and matched with the gallery samples until the authentication request is approved.

Fig. 1. Key frame selection in VPR system.

3.1. Assisted Placement

DLSP assisted placement technique can overcome the challenges as well as effectively reduce the localization error and computation complexity. However, the assisted lines in DLSP are oblique, so there are large invalid area. In addition, the preprocessing is not easy conducted along oblique directions. Thus MDLSP technique is developed to reduce the invalid area in the frames and facilitate computation. Figure 2 shows the assisted placement interface for left hand in VPR system. The interface can be mirror flipped for right hand. The invalid area is very small, and the computation can be easily conducted along horizontal and vertical directions.

Fig. 2. MDLSP interface.

In MDLSP, there are two horizontal assisted lines (long line CD, short line AB) and one assisted point B (the right end point of line segment AB) on the screen. The point O in the upper left corner is the origin of the coordinate system. The positive directions of the X-axis and Y-axis are horizontally rightward and vertically downward, respectively. The positions of the assisted graphics are defined as follows.

The image is a horizontal screen preview on mobile devices. The width and height are W and H, respectively. When the palm surface is parallel to the interface surface, the upper and lower boundaries of the palm are approximately parallel to AB and CD, respectively. Let the two end points of the upper boundary be A(𝑥_𝐴, 𝑦_𝐴) and B(𝑥_𝐵, 𝑦_𝐵), respectively.

\(x_{A}=\frac{7}{15} H, y_{A}=\frac{1}{15} W\).       (1)

\(x_{B}=\frac{9}{15} H, y_{B}=\frac{1}{15} W.\)       (2)

Let the two end points of the lower boundary be C(𝑥_𝐶, 𝑦_𝐶), D(𝑥_𝐷, 𝑦_𝐷).

\(x_{C}=\frac{4}{15} H, y_{C}=\frac{14}{15} W\).       (3)

\(x_{D}=\frac{9}{15} H, y_{D}=\frac{14}{15} W\).       (4)

The distance between line segment AB and CD is L.

\(L=y_{D}-y_{B}\).       (5)

L determines the distance from the palm surface to the camera. A user needs to keep his/her palm surface and the lens at a proper distance.

3.2. Hand Placement

A user should place his/her hand according to the following rules.

(1) The four fingers (index, middle, ring and little fingers) are naturally brought together, and the thumb is spread out.

(2) The upper and lower boundary lines of the palm should be aligned with and tangent to the two assisted lines.

(3) Point B should be aligned with the intersection of the upper boundary of the index finger and the bottom line of the index finger.

3.3. Valid Frame selection

The valid frames are selected according to the finger gap judgement [14]. The finger gap is the sub-region of hand, i.e., the shaded rectangle in Figure 2. The location of shaded rectangle is:

Fig. 2. MDLSP interface.

\(\left\{\begin{array}{c} x:\left[x_{B}, x_{B}+L \times \frac{2}{5}\right] \\ y:\left[y_{B}+L / 10, y_{B}+L \times 4 / 5\right] \end{array} .\right.\)       (6)

If the dark rectangle has the finger gap, i.e., the finger gap appears at the correct location in one frame, which demonstrates that the user places his/her hand correctly according to the assisted placement, then this frame is considered as the valid frame. The finger gap processing is shown in Figure 3 and its horizontal integral projection is shown in Figure 4.

Fig. 3. Finger gap processing.

Fig. 4. Horizontal integral projection of finger gap region.

3.4. ROI Localization and Cropping

The shaded area in Figure 5 is the ROI. The location of ROI is:

Fig. 5. ROI localization and cropping.

\(\left\{\begin{array}{l} x:\left[x_{B}-L \times 4 / 5, x_{B}-L / 10\right] \\ y:\left[y_{B}+L / 10, y_{B}+L \times 4 / 5\right] \end{array}\right.\).       (7)

3.5. Key Frame Selection

For the image evaluation without reference, the grayscale variance product is used to evaluate the quality of valid frames, which is defined as:

\(\begin{array}{c} G(f)=\sum_{y} \sum_{x}|f(x, y)-f(x+1, y)| \times \mid f(x, y)- \\ f(x, y+1) \mid . \end{array}\)       (8)

In order to reduce the computation complexity, only the quality of ROI is evaluated. Valid frames have different qualities, as shown in Figure 6. The valid frames, whose gray variance products are higher than β, are selected as the key frames.

Fig. 6. The gray-scale variance product values of valid frames.

3.6. Registration

The system inputs the gallery video, and the ROI set is \(\mathrm{Rr}=\{\mathrm{r} 1, \mathrm{r} 2, \ldots, \mathrm{rn}-1, \mathrm{rn}\}\). The template set generated from ROI set is \(\mathrm{Tr}=\{\mathrm{t} 1, \mathrm{t} 2, \ldots, \mathrm{tn}-1, \mathrm{tn}\}\). Assume that the number of gallery templates is k, the registration distance judgment threshold is hr, and the gallery template set is \(\mathrm{Tr}^{\prime}=\left\{t_{1}^{\prime}\right., \left.t_{2}^{\prime}, \ldots, t_{k}^{\prime}\right\}\), the gallery templates are generated by the following registration algorithm.

Registration algorithm

3.7. Verification

During authentication, the system inputs the authentication video, and the template set of the ROI is \(\mathrm{Tv}=\{\mathrm{t} 1, \mathrm{t} 2, \ldots, \mathrm{tn}-1, \mathrm{tn}\}\), the authentication distance threshold is h_v, then the authentication algorithm is:

Verification algorithm

Ⅳ. EXPERIMENTAL RESULTS

4.1. Database

The hand video capture application is shown in Figure 7. We capture totally the videos of 100 left and right hands of 50 persons, each palm has 5 videos. Each video lasts about 6 seconds and its size is about 10MB. The ROIs are cropped from valid frames, as shown in Figure 8.

Fig. 7. Hand video capture APP interface.

Fig. 8. ROI cropping.

4.2. Accuracy

Equal error rate (EER) and decidability index d' are used to evaluate the accuracy. d' is defined as:

\(d^{\prime}=\frac{\left|\mu_{1}-\mu_{2}\right|}{\sqrt{\frac{\sigma_{1}^{2}+\sigma_{1}^{2}}{2}}}\)       (9)

where μ1 and μ2 are the mathematical expectations of intra-class and inter-class normalized Hamming distances, respectively. σ1 and σ2 are the standard deviations of intra-class and inter-class normalized Hamming distances, respectively.

Table 2 shows the EER and d'. Figure 9 shows the ROC curves. The experimental results demonstrate the effectiveness of our VPR algorithm and system.

Table 2. Verification accuracy.

Fig. 9. ROC curves.

Ⅴ. CONCLUSIONS AND FUTURE WORKS

Palmprint recognition in video media is a novel technology. This paper modifies DLSP technique to reduce the invalid area in the frames. In addition, the valid frames are selected according to finger gap judgement, and then some key frames are selected from the valid frames according to image quality as the gallery samples that are matched with the query samples for authentication decision. The VPR algorithm is conducted on the system designed and developed on mobile device. In future, we will further modify the assisted technique or develop other state-of-the-art assisted techniques, and employ other recognition methods, such as deep learning, to improve accuracy.