A computer vision-based approach for behavior recognition of gestating sows fed different fiber levels during high ambient temperature

Abstract

The objectives of this study were to evaluate convolutional neural network models and computer vision techniques for the classification of swine posture with high accuracy and to use the derived result in the investigation of the effect of dietary fiber level on the behavioral characteristics of the pregnant sow under low and high ambient temperatures during the last stage of gestation. A total of 27 crossbred sows (Yorkshire × Landrace; average body weight, 192.2 ± 4.8 kg) were assigned to three treatments in a randomized complete block design during the last stage of gestation (days 90 to 114). The sows in group 1 were fed a 3% fiber diet under neutral ambient temperature; the sows in group 2 were fed a diet with 3% fiber under high ambient temperature (HT); the sows in group 3 were fed a 6% fiber diet under HT. Eight popular deep learning-based feature extraction frameworks (DenseNet121, DenseNet201, InceptionResNetV2, InceptionV3, MobileNet, VGG16, VGG19, and Xception) used for automatic swine posture classification were selected and compared using the swine posture image dataset that was constructed under real swine farm conditions. The neural network models showed excellent performance on previously unseen data (ability to generalize). The DenseNet121 feature extractor achieved the best performance with 99.83% accuracy, and both DenseNet201 and MobileNet showed an accuracy of 99.77% for the classification of the image dataset. The behavior of sows classified by the DenseNet121 feature extractor showed that the HT in our study reduced (p < 0.05) the standing behavior of sows and also has a tendency to increase (p = 0.082) lying behavior. High dietary fiber treatment tended to increase (p = 0.064) lying and decrease (p < 0.05) the standing behavior of sows, but there was no change in sitting under HT conditions.

INTRODUCTION

Heat stress (HT) has become a frequent and serious problem for swine productivity and welfare in the swine industry [1]. Pig thermoregulation relies on heat dissipation through the respiratory pathway (gasping) rather than via the sweat glands due to the thick subcutaneous fat, which inhibits sensible heat loss [2,3]. Various strategies such as behavioral and metabolic changes are initiated to minimize heat generation when it is difficult for pigs to maintain euthermia in response to high ambient temperatures [1]. In general, one of the major adaptations for correcting metabolic heat production in response to high ambient temperature is reducing voluntary feed intake (VFI), which minimizes metabolic heat production [4]. However, in commercial farms, reduction in VFI, induced by high ambient temperature, of the pregnant sows may not help maintain euthermia throughout the regulation of metabolic heat production owing to the limited VFI for preventing obesity. Moreover, limited VFI can trigger certain behaviors induced by hunger, and restrictions on behavior due to the stall can increase sow stress as well as heat stress. To prevent this problem, studies have been conducted on the level of fiber in sow feed. Dietary fiber is related to the physicochemical properties of diets and increases the bulk and water-holding capacity of feed, thereby allowing it to expand in the gastrointestinal tract of the sow [5]. Dietary fiber can be used to prevent constipation and increase the satiety of sows [5,6]. The satiety of sows is reflected in stereotypic behaviors such as non-feeding oral activities and physical activities [7]. Long periods of sitting or standing inactive could indicate poor welfare, whereas lying may reflect improved wellbeing [8]. Therefore, the assessment of behavioral characteristics is used to evaluate the stress and welfare of sows [5]. The productivity and welfare of swine are adversely affected by heat stress, but manual observation and classification of animal stress-related behaviors are laborious, invasive, time-consuming, and dependent on expert scorers. Automated monitoring and recognition of swine behavior can be a more effective way to overcome these hindrances and improve farm animal management compared with manual observations.

The classical approaches to pose estimation are based on manual visual observation by the farmer, based on behavioral patterns as swine typically show a variety of behavioral responses to environmental stimuli and stress. This management method may contribute to improved welfare when appropriate measures are taken [9]. However, traditional manual observation methods are time- and labor-consuming and rely heavily on the operator. Computational analysis for evaluating behavior data is an area of increased research interest because of its importance in health-related diagnosis issues and management planning.

Recently, with the rapid development of deep learning, automated and non-intrusive animal monitoring systems have drawn attention to the potential of significantly reducing labor consumption and improving the collection and classification of behavioral information to support management decisions. In a study by Yang [10], the performance of a fully convolutional network (FCN) [11] was evaluated to automatically recognize the nursing behaviors of sows. They proposed a method that exploits Spatio-temporal relations using FCN and oriented optical flow to automatically recognize nursing behaviors from daily behavioral videos of lactating sows [10]. Zheng et al. [12] introduced a deep learning detector, with faster regions and a convolutional neural network (R-CNN) [13] to automatically identify five postures of sows in loose pens. Chen et al. [14] proposed a deep learning method based on convolutional neural networks (CNNs) [15,16] and long short-term memory for the early detection of aggressive episodes of pigs.

In this study, we constructed a swine image dataset under real farm conditions. The acquisition of real farm images is influenced by conditions and parameters such as variable distances, poor resolution, and non-ideal illumination. The quality of images depends on several factors such as shadows, noises, artifacts, and reflections, which hinder data acquisition with reliability and robustness for swine posture analysis. Therefore, a computer-aided diagnosis system that can analyze real farm condition images may be considered for swine behavioral pattern classification. Therefore, the objectives of this study were to evaluate the effect of the automatic classification system of eight well established and state-of-the-art artificial intelligence techniques using a swine image dataset to predict behaviors by applying an appropriate classifier to the pre-processed data and to apply the derived result in investigating the effect of dietary fiber level on the behavioral characteristics of the pregnant sow under high ambient temperatures during the last stage of gestation.

MATERIALS AND METHODS

The protocol for the current study was approved by the Institutional Animal Care and Use Committee of Kangwon National University, Chuncheon, Korea (KW-170519-1).

Animals and management

The experiment was conducted at a swine farm located at Kangwon National University (Chuncheon, Korea). During the last days of gestation (days 90 to 114), a total of 27 crossbred sows (Yorkshire × Landrace; average body weight [BW], 192.2 ± 4.8 kg) were assigned to three treatment groups, in a randomized complete block design, based on the body weight (kg). Each treatment had 10 replicates with one sow. The sows in group 1 were fed diets containing 3% fiber (LF) under neutral ambient temperature (NT); the sows in group 2 were fed diets containing 3% fiber under HT; and the sows in group 3 were fed diet containing 6% fiber (HF) under HT. Two isoenergetic and isonitrogenous diets in a mash were formulated for gestation period (Table 1). Diets weighing 2.2 kg were provided to each sow per day at 08:00 and 16:00 until farrowing. The sows were given ad libitum access to water throughout the gestation period. They were housed in gestation stalls (2,000 × 600 × 1,000 mm) from 90 to 100 d of gestation. At day 100 of gestation, the sows were moved into two types of conventional farrowing houses: farrowing house A was an open house with facilitated farrowing crates (2,000 × 600 × 1,000 mm); farrowing house B was a closed house with facilitated farrowing crates (2,000 × 600 × 1,000 mm) and a cooling pad. The average temperatures in the gestation house (days 90 to 100 of gestation), farrowing house A (days 100 to 114 of gestation), and farrowing house B (days 100 to 114 of gestation) were 29.7 ± 2.4℃, 28.2 ± 1.1℃, and 21.4 ± 1.8℃, respectively.

Table 1. Formula and chemical composition of lactation sow diets (as-fed basis)

¹⁾Supplied per kilogram of vitamin premix: 12,000,000 IU vitamin A, 2,400,000 IU vitamin D3, 132,000 IU vitamin E, 1,500 mg vitamin K3, 3,000 mg vitamin B1, 11,250 mg vitamin B2, 3,000 mg vitamin B6, 45 mg vitamin B12, 36,000 mg pantothenic acid, 30,000 mg niacin, 600 mg biotin, 4,000 mg folic acid.

²⁾Supplied per kilogram of mineral premix: 80,000 mg Fe, 170 mg Co, 8,500 mg Cu, 25,000 mg Mn, 95,000 mg Zn, 140 mg I, 150 mg Se.

³⁾Calculated values.

DDGS, dried distiller’s grains with solubles; TCP, tricalcium phosphate; MET, methionine; CYS, cysteine; ME, metabolizable energy.

Video acquisition

To build image datasets for the behavioral observation of sows, the videos were recorded and collected as top-down view images. Cameras were installed above the center of the swine pen on the ceiling at a height of 2.20 m, relative to the floor. The lens on the camera pointed directly downwards the pen to monitor their activity over time. The videos were recorded in JPEG format with 1, 920 × 1, 080 pixels at 30 fps, between 11:00 am and 1:30 pm during the last gestation period in the natural state, without special preparations to enhance the lighting or to improve the distinguishability of the animals from the background. Videos were converted to image sequences, and the images were used for the development of digital image processing. Only one image per second was captured for substantial savings in storage space.

Datasets and labeling

The dataset contained 11, 850 images, and the data were split into 7, 556 training images, 2, 517 validation images, and 1, 776 testing images. The images were categorized into four classes including 2, 439 images which were created for lying left, 2, 650 for laying right, 4, 458 for sitting, and 2, 277 for standing. The computations were performed using a Python 3.7 on an Intel(R) Core (TM) i7-8700 K CPU @ 3.70 GHz, 16 GB RAM, NVIDIA GeForce RTX 2080 with 8GB RAM on a 64-bit Windows 10 operating system. The CNN and long short-term memory (LSTM) architectures were implemented in TensorFlow 1.13.1 and Keras 2.2.4, respectively.

Definition of the sow behaviors

In this study, four postures were defined for the behavior classification. The sows’ behaviors covered in this analysis included lying left, lying right, sitting, and standing. The definitions of the four classes of sow postures are shown in Table 2, and their corresponding examples are shown in Fig. 1.

Table 2. Definition and description of different sow behaviors recorded in a farrowing crate

Fig. 1. Representation of swine posture categorization in four classes.

Architectures

Here, we used four deep learning architectures: InceptionV3, DenseNet, MobileNet, and VGG net models. InceptionV3 is the third generation of Google’s Inception CNN and consists of 159 layers in total. The prominent feature of the inception module is the use of three convolution sizes of filters (1 × 1, 3 × 3, and 5 × 5) instead of using one type of kernel, and a bottleneck layer for nonlinear dimensionality reduction. Moreover, the presence of a pooling layer is used for dimension reduction within the module. DenseNet is one of the recent discoveries in neural networks for visual object recognition that, by summation operations, connect each layer to every other layer in a feed-forward fashion. DenseNet is able to achieve maximum information flow between layers in the network. Feature maps extracted from each layer are reused as inputs for the subsequent layers. Compared with the previous image recognition network, it solves the problem of gradient disappearance of a deep network, strengthens the propagation of features, encourages feature reuse, and reduces model parameters. The MobileNet architecture, designed by the Google research team for object recognition on mobile devices, consists of a depth-wise separable convolution and 1 × 1 point-wise, convolution layer. The performance was evaluated on the ImageNet dataset [17] and achieved the same level of accuracy as VGG16 [18] with 32 times fewer parameters and was 27 times less computationally intensive. Depth-wise convolution uses a single spatial filter for each input feature map, and pointwise convolution (1 × 1) is applied to cross-channel patterns. The VGG network architecture is the CNN model introduced by Simonyan and Zisserman [18] for the ILSVRC-2014 challenge. This network is characterized by its simplicity, using only 3 × 3 convolutional layers stacked on top of each other in increasing depth. Reducing the volume size is handled by max pooling. Two fully connected layers, each with 4, 096 nodes, are then followed by a softmax classifier.

Evaluation criteria

To estimate the prediction performance of individual models, four commonly used evaluation metrics were applied: accuracy, sensitivity, specificity, and F1-score. Sensitivity is the measure of swine posture labels that are correctly classified and expressed in the form as:

\(\text { Sensitivity }=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}} \times 100\)

where true positive (TP) is the number of images that are correctly predicted and false-negative (FN) is the number of images that are incorrectly predicted.

Specificity is the measure of swine posture labels that are successfully classified and is expressed as:

\(\text { Specificity }=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}} \times 100\)

where true negative (TN) is the number of negative images that are correctly predicted and false positive (FP) is the number of negative images that are incorrectly predicted.

Accuracy is used to show the number of correctly classified swine postures divided by the total number of swine postures and is defined as:

\(\text { Accuracy }=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}} \times 100 .\)

F1-score, known as F-measure, is defined as the weighted average of precision and recall and is obtained by

\(\text { F1-Socre }=2 \times \frac{\text { Re call } \times \text { Pr ecision }}{\text { Recall }+\text { Pr ecision }}\)

Statistical analysis

The percentage of the number of image results classified by architectures from individual sows was pulled and was considered the experimental unit in all statistical analyses to compare sow behavior classification. Analysis of variance (ANOVA) was performed using the SAS program (SAS 2012; SAS Institute, Cary, NC, USA), and the differences among the treatments were determined using Tukey’s test. Probability values less than 0.05 were considered significant.

RESULTS AND DISCUSSION

Loss and accuracy comparison

Total loss and accuracy are two indicators routinely used to evaluate the fitting effect and the convergence of training and validation processes when using deep learning methods. A learning curve can be used to visualize a cumulative evaluation of a classifier’s learning performance over time (epochs) when using training and validation datasets. It noticeably signifies how well fitted (under-fit or over-fit) a particular classifier or any machine learning model is during the model training phase.

A comparative assessment was conducted to determine the training behavior of different models. Fig. 2 illustrates the learning curves (loss and accuracy plots) for the eight individual networks that were evaluated using training and hold-out validation datasets. The results from the training and validation loss for various classifiers ensured the learning and generalization ability of the classifiers on training and hold-out validation datasets. Fig. 2 shows that all eight individual algorithm models converged in the training and validation processes of the rock structure classification. Additionally, a minimum loss value signifies the best results, which means that the training and validation datasets were learned by the classifier with fewer errors (Figs. 2a, b). In contrast, a maximum accuracy value signifies the best results, indicating more learning ability by the classifier model. This kind of dual learning curve helps in evaluating and selecting a suitable classifier model with an optimized loss and maximum classification accuracy in both the training and validation sets. Fig. 3 illustrates the relationship between the prediction accuracy and the total loss, which shows that the training and validation loss remains very close, indicating that the model is not overfitting to the training data. These graphs illustrate the deep architectures of how and what happened during these phases.

Fig. 2. Loss function and accuracy curves of individual architectures. a) DenseNet121, b) DenseNet201, c) InceptionResNetV2, d) InceptionV3, e) MobileNet, f) VGG16, g) VGG19, and h) Xception.

Fig. 3. Confusion matrix of swine behavior image classification. a) DenseNet121, b) DenseNet201, c) InceptionResNetV2, d) InceptionV3, e) MobileNet, f) VGG16, g) VGG19, and h) Xception.

Behavior classification by deep convolutional neural network (CNN) architectures

Deep feature extraction based on transfer learning started by investigating the effect of eight different deep CNN architectures in extracting deep features from input images and achieving optimal prediction on the unseen test data. The swine posture dataset was trained on eight different CNN architectures, including DenseNet121, DenseNet201, InceptionResNetV2, InceptionV3, MobileNet, VGG16, VGG19, and Xception, individually. DenseNet121 significantly outperformed the other individual architectures on the provided dataset with an accuracy of 99.83%, a sensitivity of 100%, a specificity of 100%, and an F1_score of 99.83% (Table 3). Compared with the remaining network models, the DenseNet201and MobileNet architectures individually provided better performance classification with accuracies of 99.77% and 99.77%, respectively. However, VGG16 is the only model that obtained the lowest accuracy of 99.54% (Table 3). The VGG16 also had poor performance (99.55%) on the F1-score. Therefore, the DenseNet family and MobileNet CNN architectures achieve high performance in terms of accuracy in the detection and classification of swine posture as compared to the other five architectures. It also shows that the DenseNet family network model can extract more distinguishing features with better robustness and generalization than the other models.

Table 3. Classification results from pre-trained deep CNN models

¹⁾The best result.

²⁾Represents the second-best result of the respective category.

Confusion matrix

In multi-class classification tasks, most of the evaluation metrics are computed based on a “confusion matrix”. The confusion matrices can be used to precisely sum up the performance of the proposed classification model. In addition to the comparison of accuracy and loss, for a more detailed analysis, the quantitative results for eight individual architectures were compared in the form of confusion matrices (Fig. 3). The confusion matrix demonstrates the performance of the classifier in terms of the number of swine posture images that have been correctly classified as well as the misclassified cases for each class. Densenet121 was able to correctly classify 1, 773 images out of 1, 776 images and hence provided a better classification of swine posture than the others. In conclusion, DenseNet121 is the best learner, and its counterpart, DenseNet201, and MobileNet, are the second-best learners. It is clear that these three architectures surpass other architecture models in terms of accuracy, sensitivity, specificity, and F1_score, which implies that these models are able to extract more discriminatory features from the swine posture images, resulting in a higher detection rate (Table 3). In swine posture image classification on the test set, however, Vgg16 incorrectly recognized eight images, in which seven images of “sitting” were misclassified as “lying right”. In the confusion matrix of swine behavior classification using different deep CNN models, the values on the main diagonal represent all correctly classified instances. The row under each confusion matrix shows the rate of accuracy achieved for each predicted class and misclass predictions.

Sow behavior

The processed results of behavior classification using the Densenet121 model are shown in Table 4. When comparing the ambient temperature (neutral vs. high) with the same dietary treatment (CF 3%), the frequency of standing of the sows in the HT treatment group was higher (p < 0.05) than that of sows in the NT treatment group and the frequency of lying behavior of sows tended to increase (p = 0.082) under the HT. However, the ambient temperature did not significantly affect the frequency of sitting and lying on the right.

Table 4. Effect of ambient temperature and dietary fiber levels on sow’s behavior frequency (%) classified by Densenet121

¹⁾Sum of left and right lying.

NT, neutral ambient temperature; LF, low dietary fiber (3%); HT, high ambient temperature; HF, high dietary fiber (3%).

Under the HT, the increase in dietary fiber significantly decreased (p < 0.05) the frequency of standing of the sows and tended to increase (p = 0.064) the lying behavior, whereas the frequency of sitting was not affected by the dietary treatment. Comparing the NT + LF and HT + HF treatments, the frequency of lying of sows in HT + HF treatment was higher (p < 0.05) than that of sows in HT + HF treatment. However, the frequency of standing of sows in HT + HF treatment was lower than that of sows in NE + LF treatment.

In this study, the behavioral analysis was performed during the last stage of the gestation of sows, and the frequency of lying behavior in all treatments occupied a larger proportion of time as compared to that of sitting and standing. Zhang et al. [19] evaluated the behavior in sows of different parities and the welfare of sows in intensive farming systems and reported that the lying behavior increased in the latter stages of pregnancy and decreased before farrowing. Improving the confined environment would improve the comfort of sows and reduce physiological and mental stress [20]. Sitting or standing for long periods of time indicates poor welfare, whereas lying reflects improved well-being [8]. However, considering the increase in lying behavior of sows under HT conditions as a parameter would be controversial as it may not adequately reflect the welfare of sows. The HT in our study reduced the standing behavior of sows and had the tendency to increase lying behavior. Our results were contrary to previous reports on the sow’s comfort under HT conditions. The sows’ behavioral changes at HT were decreased and sows spent most of their time lying due to the heat increment resulting from their movement [21]. However, HT and metabolic heat increment of sows increase with the presence of some facility (floor or crate) in contact with sow’s skin. This increases the sow’s temperature regulation characteristics (emitting metabolic heat through the skin) so that the frequency of posture change increases due to the instinct to find a cool place [22]. These conflicting results disprove the need for more research into investigating sow behavior under HT.

In our study, the high dietary fiber treatment tended to increase the lying and decrease the standing behavior of sows, but there was no change in sitting. In the present study, the ventral lying was included in sitting, and ‘lying’ represented only lateral lying (left and right side); thus, sitting behavior might not have been affected by the treatments. Changes in the frequency of the sows’ behavior regarding the level of dietary fiber can be associated with satiety. Dietary fibers are composed of plant polysaccharides and lignin that are not hydrolyzed by endogenous enzymes in the gastrointestinal tract, increasing the passage rate and altering the digestion and absorption times of nutrients [23]. The dietary fiber increases the concentration of non-esterified fatty acids in the blood by supplying a constant concentration of glucose to the pregnant sows for a long time, thereby increasing satiety [23]. Our findings reflect the satiety of the sows, and the integrated results show its positive effects on sows before farrowing. However, dietary fiber increases the internal body temperature due to fermentation in the gut [24]. It is hypothesized that high dietary fiber induces additional heat stress in pregnant sows. However, in our study, the sows were fed the same amount of feed with adequate nutrients, using restricted feeding, and there was no additional supply other than the suggested fiber. Therefore, a possible explanation for our overall behavioral results could be the hypothesis that the sows reduced metabolic heat generation by minimizing movement to adapt to the HT. Further studies are required to evaluate the occurrence of heat stress and sows’ health due to the fermentation of fiber during late stages of pregnancy.

CONCLUSION

In this paper, a comparative study was presented for swine posture classification during the last stage of gestation, generating a farm image dataset containing various poses of swine. Preprocessing methods such as illumination correction, contrast enhancement, and artifact removal were used to improve image quality and obtain a better generalization ability. Due to the limitation of the behavioral pattern of sows during the late stages of gestation, four behavior patterns—lying left, lying right, sitting, and standing—were applied for classification. Various standard evaluation metrics such as specificity, sensitivity, accuracy, and F1-score were employed to evaluate the obtained results. The diagnostic accuracy of several deep-learning models was tested in this work, and it was observed that the DenseNet neural network model gave excellent results regarding the accuracy and was superior to other structural models in classification performance, thus achieving a more accurate classification of the swine image dataset. The results of this study suggest that the application of CNN-based computer-aided diagnostic methods integrated with image processing and machine learning methods can reliably classify different behavioral postures under different commercial farm conditions and therefore assist in the interpretation of swine behavioral patterns and that this has the potential to become a key approach in commercial swine farming. A new strategy for the investigation of sow’s behavior was successfully applied in our study to determine how ambient temperature and dietary fiber levels affected the behavior of sows during the last stage of gestation. The HT in our study reduced the standing behavior of sows and had a tendency to increase lying behavior. High dietary fiber tended to increase lying and decrease the standing behavior of sows, but there was no change in sitting under HT condition. Therefore, standing behavior can be presented as an influential indicator of heat stress in sow welfare.

Competing interests

No potential conflict of interest relevant to this article was reported.

Funding sources

This research was funded by Rural Development Administration, Korea (grant number: PJ014796012020).