Plant Species Identification based on Plant Leaf Using Computer Vision and Machine Learning Techniques

Abstract

Plants are very crucial for life on Earth. There is a wide variety of plant species available, and the number is increasing every year. Species knowledge is a necessity of various groups of society like foresters, farmers, environmentalists, educators for different work areas. This makes species identification an interdisciplinary interest. This, however, requires expert knowledge and becomes a tedious and challenging task for the non-experts who have very little or no knowledge of the typical botanical terms. However, the advancements in the fields of machine learning and computer vision can help make this task comparatively easier. There is still not a system so developed that can identify all the plant species, but some efforts have been made. In this study, we also have made such an attempt. Plant identification usually involves four steps, i.e. image acquisition, pre-processing, feature extraction, and classification. In this study, images from Swedish leaf dataset have been used, which contains 1,125 images of 15 different species. This is followed by pre-processing using Gaussian filtering mechanism and then texture and color features have been extracted. Finally, classification has been done using Multiclass-support vector machine, which achieved accuracy of nearly 93.26%, which we aim to enhance further.

I. INTRODUCTION

Plants are a fundamental part of life on our planet. They give us oxygen to breathe, food, medicine and plenty of other things which make our lives worth living. They are the backbone of all life [1]. However, identifying the plants correctly is out of reach of an ordinary person as it requiresspecialized knowledge, and only the experts of botanical background are able to pull off this task. Moreover, even botanists do not have knowledge of all the existing plants in this world for there is an unlimited number of plantspecies. Hence, the task of plant identification is limited to a very small number of people. However, plant species knowledge is necessary for various purposes such asidentifying a new or rare species, balancing of the ecosystem, medicinal purposes, agricultural industry, etc. [1]. To be able to achieve these objectives, automation of plant species identification is a necessity [2]. There are enormous plant species in the world, which is nearly 390,000 [3] in number, and each year, new species are reported in different parts of the world [4]. Plants are very different from one another, hence requiring in-depth taxonomic knowledge to identify and assign them to aparticular species. Many activities, such as studying the flora of a particular area, investigation of the endangered species, discovering new plant species depends profoundly upon precise and accurate identification skills. With this, the need for automated identification of plant species isincreasing, but unfortunately, the number of plant systematics experts are limited.

In manual identification, botanists use specific defined characteristics of a plant as a key for identification, whichis helpful in identifying plant species. The identification keys involve features such as ‘shape,’ ‘texture,’ ‘color’ and ‘venation ’ of an unknown plant. When thoroughly examined, these characteristics eventually lead to the desired species. Moreover, identifying a plant species from a natural site demands extra ordinary taxonomical expertise, which is beyond the capacity of any ordinary person. Thus, conventional plant species identification methods seem impractical for ordinary people and are a challenge for professional taxonomists as well. Even for the expertbotanists, species identification is often a laborious task.

Manual identification is often time-consuming and inefficient [5], even the expert taxonomists take a considerable amount of time to identify a plant species. Since the traditional identification methods are strenuous, there arises a need to automate the process of species identification. As a result, researchers have tried to develop automated plant species identification and classification systems which can serve the purpose of species recognition to some extent. A few of these are discussed in the nextsection.

In this era of the digital world, smartphones and digital cameras are available to everyone and are found in abundance. As a result of this technological advancement, digital images have become an indispensable element of several fields, which include face recognition, plant recognition, and health informatics, computer vision [6]. In addition, technological development in the field of image processing and the tool-boxes available to implement it hasaimed to automate the process of species identification. There have been some successful attempts to automate the process of species identification. One of the most successful attempts has been made by the authors in [7], who developed the largest of its kind plant species recognition system called Leafsnap. It is the very first mobile application developed, and it efficiently performs on the real-world plant images. Despite all the efforts made in computer vision and machine learning, automated plant species identification still faces numerous challenges since plants species are present in huge number, and they have avery similar representation of shape and color.

Automatic plant identification typically involves four steps viz. image acquisition, image pre-processing, feature extraction, and classification [2]. In our study, we have proposed a plant identification system which automatically classifies plant images by extracting color, texture features from the input image. Apart from the abovementioned four fundamental steps, we have also included image segmentation before feature extraction to obtain better classification accuracy. Image segmentation is an important step in image processing, but it hasn’t been widely used in the previous studies carried out in this area. For classification, used multiclass support vector machine (MSVM) as it is a robust model, and it uses a rule-based environment to solve the given problem. The system is then evaluated with Swedish leaf dataset for classification results.

The rest of the paper has been organized as follows: Section 2 gives a detailed literature review of the studies conducted in the automatic plant identification area with a comparison table. Section 3 describes in detail the steps involved proposed methodology, i.e. image acquisition, preprocessing, segmentation, feature extraction, and classification with necessary figures. This is followed by section 4 that gives the results of the implementation of the proposed method. Conclusion of the paper is discussed insection 5.

 

II. LITERATURE REVIEW

In the past decade a lot of research has been done in order to develop efficient and robust plant identification systems.

Wu et al. [5] have proposed on of the earliest plantidentification system. In their scheme, they have created their own dataset named Flavia, which has been used by various other researchers as standard dataset for their work. It consists of 1907 leaf images of 32 different plant species. In their study, they extracted 5 basic geometric and 12 digital morphological features based on shape and ve instructure from the leaf images. Further, principal component analysis (PCA) was used to reduce the dimensions of input vector to be fed to the probabilistic neural network (PNN) for classification. They used a three-layered PNN which achieved an average accuracy of 90.32%.

Hossain et al. [8] extracted a set of unique featured called “Leaf Width Factor (LWF)” with 9 other morphological features using the Flavia dataset. These features were thenused as inputs to PNN for classification of leaf shapefeatures. A total of 1200 leaf images were used to train thenetwork and then PNN was tested using 10-fold cross validation, which achieved maximum accuracy of 94% at 8^th fold. The average accuracy attained was 91.41%.

Wang et al. [9] proposed a robust method for leaf image classification by using both global and local features. They used shape context (SC) and SIFT (Scale Invariant Feature Transform) as global and local features respectively. K-nearest neighbor (k-NN) was used to perform classification on ICL dataset which achieved an overall accuracy of 91.30%.

Authors in [10] developed a scheme which extracted 12 common digital morphological shape and vein features derived from 5 basic features. They implemented both k-NN and support vector machine (SVM) which attained anaccuracy of 78% and 94.5% respectively when tested on Flavia dataset.

Pham et al. [11] in their computer-aided plantidentification system compared the performance of twof eature descriptors i.e. histogram of oriented gradients (HOG) and Hu moments. For classification, they selected SVM due to its ability to work with high dimensional data. They obtained accuracy of 25.3% for Hu moments and 84.68% for HOG when tested with 32 species of Flaviadataset.

Mouine et al. [12], in their study introduced new multiscale shape-based approach for leaf image classification. They studied four multiscale triangular shapedescriptors viz. Triangle area representation (TAR), Triangle side length representation (TSL), Triangle oriented angles (TOA) and Triangle side lengths and anglerepresentation (TSLA). They tested their system on fourimage datasets: Swedish, Flavia, ImageCLEF 2011 and Image CLEF 2012. With Swedish dataset they computed classification rate as 96.53%, 95.73%, 95.20% and 9 0.4% for TSLA, TSL, TOA and TAR respectively using 1-NN.

Authors in [13] proposed a method for plantidentification using Intersecting Cortical Model (ICM) and used SVM as the classifier. This study used both shape and texture features viz. Entropy Sequence (EnS) and Centre Distance Sequence (CDS). They attained accuracy of 97.82% with Flavia dataset, 95.87% with ICL1 and 94.21% with ICL2 (where ICL1 and ICL2 are subsets of ICL dataset).

Ghasab et al. [14] are one of the very few authors who have implemented a combination of shape, color, textureand vein feature descriptors. They applied ant colony optimization (ACO) as a feature decision-making algorithm, which helped obtain the best discriminant features. They attained an accuracy of 96.25% with Flavia dataset using SVM as the classifier.

Aakif et al [15] proposed an algorithm which used Artificial Neural Network (ANN) with back propagation. An input vector of morphological features, Fourierdescriptors (FD) were fed into the ANN which resulted in a classification accuracy of 96% for their own dataset. They further verified the efficiency testing on Flavia and ICL datasets and attained accuracy of 96% for both the datasets.

Authors in [16] developed an algorithm which extracts around 15 shape features and applies feature normalization and dimensionality reduction. For classification, SVM has been implemented and an aggregate accuracy of 87.40% was attained when tested on Flavia dataset.

Begue et al [17] developed a system using their owndataset including images of leaves from 24 differentmedicinal plants. They extracted shape-based features fromeach leaf image. A number of classifiers (k-NN, naïve bayes, SVM, neural network and random forest) were employed, out of which random forest classifier attained highestaccuracy of 90.1%.

Amlekar et al [18] developed a method that performs classification by automatically extracting shape features. Classification has been performed using feed forward back propagation neural network. This method was then tested on ICL dataset and attained accuracy of 99% for training images and 96% for testing images.

In the existing literature, majority of the studies have used shape feature descriptors for feature extraction as it is considered the most discriminative feature in plantidentification. Feature extraction is one of the most significant steps in image processing, thus features describing various aspects of plant leaf must be taken into account before final classification of plant image. Moreover, texture and color features can better describe a leaf image in cases where the leaves are tampered or not fully grown. In our study, we have extracted the best possible set oftexture and color features for classification. A comparison of techniques used in existing literature is given in table 1.

 

Table 1. Comparison of existing literature.

 

III. PROPOSED SYSTEM

The flow of operation of the proposed system is shown in figure 1. The details of each step are discussed in the subsequent sub-sections.

 

Fig. 1. Flow of operation of the system.

 

3.1. Image acquisition

The first step in the process of identification is to acquirethe image of the plant. The image taken can be of the entireplant, leaf, flower, stem or even the fruits [2]. Authors in [3]suggest that there are three categories of images based on how the image is acquired, viz. ‘scans’, ‘pseudo-scans’, and ‘photos. In scan and pseudo-scan categories, the leafimages are taken by the method of scanning and photography respectively i.e. the images are captured infront of a plain background indoors. For the third category, the images are of plants are captured in naturalenvironment. Scans and pseudo-scans images are largely used by researchers as they are easy to examine [2]. Typically, the leaves selected are simple, fully grown and not tampered. These are then imaged in the lab under properlighting conditions. The scans and pseudo-scans simplify the classification task as the image is taken against a plainbackground. Some of the available standard datasets are Swedish dataset (15 species of leaves), Flavia dataset (32 species of leaves), ICL dataset (220 plant species), etc. Majority studied have worked on images from these threedatasets (refer table 1). In our study we have used Swedish dataset [19] which contains 75 images each of 15 species of plants, which makes a total of 1,125 images. The dataset is available in public domain and can be downloaded from the official website (http://www.cvl. isy. liu. se/en/research/datasets/swedish-leaf/). It contains images of plant leaves which are in . tiffformat. Table 2 gives the names of the 15 species and one image each from all the species.

 

3.2. Image pre-processing

Image pre-processing is an important step as it helps to enhance the quality of image for further processing. This step is necessary as an image inherently contains noise and this may result in lower classification accuracy. It isperformed to remove the noise that hampers the identification process and handle the degraded data. A series of operations are followed to improve the image of the leaf which include, converting the RGB image tograyscale, then from grayscale to binary, followed by smoothing, filtering etc [20]. Pre-processing mechanismused in this paper contains noise handling along withresizing operation and image enhancement.

 

Table 2. Plant species in Swedish Dataset.

 

3.2.1. Gaussian Filtering

To handle the noise in the images, this study hasemployed Gaussian Filtering which is also sometimescalled Gaussian Smoothing. It is a linear filter whichreduces the noise or redundant information in the image. Formula to apply gaussian filter is given in equation 1.

\(G_{smoothened_{image}}=\frac{1}{2\pi a^2}e^{(a^2+b^2)\frac{1}{2a^2}}\)       (1)

‘α’ is standard deviation, ’a’ is the distance from horizontalaxes and ‘b’ is the distance of origin from vertical axes.

 

3.2.2. Resizing

After handling noise, resizing operation has been performed. In our study, the images have been resized to [300 × 400]. Resizing is done using equation 2.

\(Resized_{G}=Resize \left(G_{\text {Smoothened }_{\text {image }}},[300,400]\right)\)       (2)

 

3.2.3. Image enhancement

Since we are working on color images, image enhancement becomes an essential step to perform. Moreover, the next step involves color image segmentation for which the image contrast and texture needs to beenhanced to obtain better results. Image enhancementremoves any redundant pixels present in the color image before performing segmentation [21]. In our study we have enhanced the contrast of the image by contrast stretching which improves the contrast in an image by expanding the dynamic range of intensity values it contains. This step is followed by contrast adjustment which saturates the top one percent and bottom one percent of all pixel values [22].

 

Figure 2 below shows an enhanced image.

 

Fig. 2. Image enhancement.

 

3.3. Image Segmentation

Image segmentation is an important step and critical step for image analysis and is basically performed to extract theregion of interest (ROI). It is a process in which each pixelof the image is individually processed and is grouped together with other pixels in the image which share the same attributes and an image divided into various segments is obtained as the output of the segmentation step [23]. Inother words, it is process in which each pixel is assigned alabel based on certain characteristics and the pixels that share the similar characteristics are grouped together. The images are generally segmented into different parts (orsegments) on the basis texture, color, gray level, pixelintensity value etc [24]. Segmentation plays a significantrole as partitioning of images into several parts makesimage analysis much easier and manageable. Segmentation is widely used in various application areas such as content-based retrieval, object recognition, medicinal imaging etc.

 

Fig. 3. Segmentation performed by K-means clustering.

Edge-based method, region-based method, clustering method, watershed method, are some of the segmentationtechniques which are widely used in these areas of work. In the past few decades, many studies have performed segmentation on gray-scale images. However, in our study we are dealing with RGB images which require color-based segmentation for further processing of the image. Since color images carry a lot of information within them, thus processing a color image as it is, reduces the efficiency. With the help of image processing toolbox of MATLABR2018b, we have implemented color based segmentation by applying k-means clustering technique and three different clusters were generated (one each for ‘R’, ‘G’, ‘B’). Thesethree images thus obtained can be used individually for further processing of the input image in the subsequent steps. Figure 3 illustrates an example of the segmentation performed in our system. Out of the three images generated, the selected image is then converted to gray-scale and binary for extraction of certain features which will be discussed in the next section. Figures 4 and 5 illustrate the gray-scale and binary images respectively of the segmented imagerepresented by the caption ‘cluster 2’ in figure 3.

 

Fig. 4. Gray- scale image.

 

Fig. 5. Binary image.

 

3.4. Feature Extraction

After performing pre-processing and segmenting the image into desired region of interest, feature extraction isperformed. It is regarded as one of the most important steps in image processing and pattern analysis. Feature extraction can also be considered dimensionality reduction process. An image inherently contains a lot of information, all of which cannot be processed as it may contain redundant data and such huge amount of data requires large amount of computation power and memory [25]. Hence, feature extraction is performed to reduce the number of variables for further processing of the image. Choosing the right setof features to optimally describe the image thus becomesvery important. In our study, we have used a combination of texture and color features to represent the image.

 

3.4.1. Texture Features

Texture analysis is very significant in many areas suchas medical imaging, image retrieval. Texture as a term inimage processing defines various properties of images suchas smoothness, coarseness, regularity etc. It represents thespatial distribution of the grey-levels of the pixels of adigital image in a neighbourhood.

There are four methods of extracting texture features viz. statistical, structural, model-based and transform-based. In our study, we have used statistical method which characterises texture by the statistical properties of the grey-level image. Statistical methods can be classified as first order (one pixel), second order (two pixels) and higherorder (three or more pixels). The first order statistics (or histogram-based features), calculate texture features from the individual pixel irrespective of the relationship of the pixel with its neighbours. Second order statistics take into account the pixels that occur relative to each other [26]. We have used GLCM (Grey-Level Co-occurrence Matrix) fortexture feature extraction which is one of the most studied second order statistics. GLCM considers the spatial relation of pixels and extracts texture features by creating a matrix by calculating how often a pixel with grey-level value ‘i’ occurs in a specific spatial relation to grey-level value ‘j’[27]. In other words, it considers relationship between twopixels at a time called the reference pixel and neighbourpixel. Haralick [28] derived 14 features from GLCM.

We have used five features viz. ‘contrast’, ‘correlation ’,‘energy ’, ‘entropy’, and ‘homogeneity’. The formulas of these features are given in table 3 where ‘i’ and ‘j’ arespatial coordinates of the function ‘Pi,j’ , ‘N’ is grey tone and ‘σx’, ‘σy’ represent the standard deviation of x and y coordinates of the image.

 

3.4.2. Color Features

In the segmentation phase, the input image was divided into three different color channels. Color features are extracted individually from all three images generated as anoutput of segmentation phase. The color features extracted in this paper can also be referred to as color-based texturefeatures as we have extracted features (mean, S.D., kurtosis, skewness) based on first order statistics from colored imagerather than the usual grey-scale. The formulas of thesefeatures are given in table 4 where ‘xi’ represents the individual pixel and ‘N’ is the number of pixels.

 

Table 3. Formulas for texture features.

 

Table 4. Formulas for color features.

The above steps have been implemented in MATLABR2018b environment. The features extracted for an imageare then stored in the feature database for subsequent classification of the images to their desired species.

 

3.5. Classification

Classification in our work, typically means to assign acertain plant species to the image based on the feature setextracted. In other words, classification is a process ofidentifying the class label of a new input image on the basis of the prior knowledge (training dataset). For our study, we have used a supervised classification technique in which thelabels of the classes (here, plant species) are already known and the new data input is assigned to one of the labels.

 

3.5.1. SVM Classifier

Support vector machine is one of the most effective androbust technique used for classification. It incorporatessupervised learning techniques which are implemented forclassification and regression [29]. SVM was originally developed by Vapnik [30] and has been widely used by researches in the area of image processing [ls] due to its ability to maximise predictive accuracy and tendency to avoid over-fitting of data [31].

Typically, SVM is a binary classifier that classifies datainto 2 classes. Classification by SVM is performed by constructing a hyperplane (or set of hyperplanes) in a n-dimensional space (where ‘n’ is the number of features) that distinctly classifies input data points. An optimal hyperplane is the one that achieves maximum margin between positive and negatives classes [16]. SVM classifieris built by employing a kernel function, which transforms the input data into higher dimensional feature space and a hyperplane which optimally separates 2 classes is thusconstructed [32]. Since in this study, number of classes (i.e. plant species) are more than two, we have used Multiclass-SVM (MSVM). The MSVMs are generally implemented by combining binary SVMs [33]. The MSVM used in this study implements ‘one-vs-all’ approach in which the ith SVM is trained such that the samples of ith class arespecified as ‘positive’ and the rest ‘negative’.

 

IV. RESULTS

The proposed methodology was tested on Swedish dataset which contains 1,125 images of 15 different species (table 2 gives the names of all the species). The datasettypically contains single leaf images of plants. The leaves are mostly in good shape and fully grown. A very few (almost negligible) leaves are distorted or slightly deformed. The dataset shows high intra-class similarity as well asinter-class similarity in a very few cases.

For this study, the images of the dataset are resized to [300 ×400]. The input image is first processed to removeany inappropriate data or noise contained in the image by filtering and contrast enhancement. This step is necessarysince the images in the dataset are colored and an RGB image contains redundant information which need not be processed. Pre-processing, in this study, includes filtering and image enhancement (as shown in figure 2). After pre-processing, the model employs color-based segmentation tosegment the image into three clusters by applying k-meansclustering technique (result of this step is as shown in figure 3). Feature extraction phase extracts second order GLCM features (‘energy’, ‘entropy’, ‘contrast’, ‘correlation ’,‘ homogeneity ’) which specify the details about the texture of the leaf image. To make classification more efficient, four color features (‘mean’, ‘standard deviation’, ‘kurtosis ’,‘skewness ’) are extracted along with five texture features.Majority of the previous studied have used only shapefeatures [8, 11, 12,15-18] for plant identification. This study however, emphasises on texture and color features because shape features cannot always correctly identify aplant. For instance, while working on plant images taken innatural environment, the leaves can often be damaged or not fully grown. In such cases, shape feature extraction can prove inefficient and unreliable. In the last step of the proposed methodology, the features are used to train the SVM classifier using ‘one-vs-all” approach. Multiclass-SVM was trained using 70% of the images and 30% images were utilised for testing.

 

Table 5. Values of the features extracted of the testing images

Five images from testing sets of each species were used to test the overall accuracy of the system. Table 5 gives the results of feature extraction for one image each from the testing sets of all the species. The GLCM texture features represent the relationship between a pair of pixels of a gray-scale image: ‘Contrast’ measures the intensity of a pixel and the neighbour pixel over the entre image; ‘ Correlation ’represents the similarity between two neighbour pixels;‘ Energy ’ depicts the uniformity of the image;‘ Homogeneity ’ measures the closeness of the distribution of GLCM elements [26]; ‘Entropy’ is the measure ofuncertainty in a gray-scale image. Entropy is generally inversely proportional to energy. For e.g. the image belonging to the species Ulmus glabra has the highestentropy value of 2.06 and the corresponding energy value is the lowest i.e. 0.64. The color features are first orderstatistics and consider only the individual pixels: Mean is the average of all pixels in the image which can also betermed as average color of the image [34]; Standard Deviation is measure of the deviation or variation from the mean value; Skewness is the degree of asymmetry of the color distribution; Kurtosis specifies the shape of the distribution. The classifier achieved accuracy as high as 94.19% for two classes (Alnus incana and Sorbus aucuparia) and the lowest accuracy obtained was 92.25% for Populustremula. The system achieved an aggregate accuracy of 93.26%. The accuracy values achieved for individual species are shown in figure 6. Classification accuracy is expressed as given in equation 3:

\(𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 ={ {𝑇𝑃+𝑇𝑁}\over {𝑇𝑃+𝑇𝑁+𝐹𝑃+𝐹𝑁}} \)                (3)

where TP is True Positive, TN is True Negative, FP is False Positive and FN is False Negative.

 

Fig. 6. Accuracies of individual species achieved with SVM Classifier

 

V. CONCLUSION

This paper has proposed an automatic plant speciesidentification approach which is employed using computervision and machine learning techniques to classify plantleaf images. The study has been conducted in phases like image pre-processing, image segmentation, feature extraction and finally classification of the image. A combination of texture and color features (5 and 4 respectively) were extracted and then SVM classifier was used for classification. The system was tested on Swedish dataset and attained an average accuracy of 93.26%. the model could automatically classify 15 different plantspecies. Texture and color feature space performed satisfactorily well in comparison to the methods which work only on morphological shape features. Also, SVM as a classifier performed considerably well when compared to PNN, k-NN. The proposed method is very easy to implement and efficient. Although the model achieved anaccuracy of more than 90%, it still lags in comparison tomethods implementing neural networks or deep learning techniques. In future, we aim to overcome this limitation and achieve higher accuracy by extracting much morecultivated features of all types (shape, texture, color and vein) and implementing improved classifier or a hybrid of classifiers. Finally, the objective is to make the idea of automatic plant species identification more realistic by working on live dataset.