A LightGBM and XGBoost Learning Method for Postoperative Critical Illness Key Indicators Analysis

Abstract

Accurate prediction of critical illness is significant for ensuring the lives and health of patients. The selection of indicators affects the real-time capability and accuracy of the prediction for critical illness. However, the diversity and complexity of these indicators make it difficult to find potential connections between them and critical illnesses. For the first time, this study proposes an indicator analysis model to extract key indicators from the preoperative and intraoperative clinical indicators and laboratory results of critical illnesses. In this study, preoperative and intraoperative data of heart failure and respiratory failure are used to verify the model. The proposed model processes the datum and extracts key indicators through four parts. To test the effectiveness of the proposed model, the key indicators are used to predict the two critical illnesses. The classifiers used in the prediction are light gradient boosting machine (LightGBM) and eXtreme Gradient Boosting (XGBoost). The predictive performance using key indicators is better than that using all indicators. In the prediction of heart failure, LightGBM and XGBoost have sensitivities of 0.889 and 0.892, and specificities of 0.939 and 0.937, respectively. For respiratory failure, LightGBM and XGBoost have sensitivities of 0.709 and 0.689, and specificity of 0.936 and 0.940, respectively. The proposed model can effectively analyze the correlation between indicators and postoperative critical illness. The analytical results make it possible to find the key indicators for postoperative critical illnesses. This model is meaningful to assist doctors in extracting key indicators in time and improving the reliability and efficiency of prediction.

1. Introduction

The occurrence of critical illness in perioperative patients will not only increase the medical expenses of patients, but also affect the Rehab Results (FCR) of patients [1, 2], and even lead to the death of patients. The study of Khuri et al. [3] showed that the median survival time of patients would be reduced by 69% if critical adverse events occurred within 30 days after the operation, and the long-term consequences of short-term surgical complications had a profound impact on the life expectancy and quality of life of survivors [4]. Preoperative and intraoperative risk prediction of critical illness, immediate capture of the incidence of critical illness for patients, and targeted measures can greatly reduce the pain and mortality of patients, avoid missing the best time for treatment, and avoid excessive use of drugs, which is conducive to the rational allocation of hospital resources.

For the prediction of critical illness risk, various monitoring indicators of patients are the key points to prediction. The quality and selection of the indicators have an important impact on the accuracy and reliability of the results for prediction. However, the samples in the dataset of critical illness are not complete, and there are various redundant and repetitive phenomena. At the same time, the indicators in the dataset are not necessarily related to critical illness, and the indicators of patients required for different illnesses are inconsistent when making treatment decisions. For doctors and clinicians, excessive patient test indicators will also interfere with real-time diagnosis and treatment decisions and affect the accuracy of diagnosis. Therefore, an efficient method is needed to analyze and sort out these data, remove irrelevant and redundant indicators, and select key indicators, so as to improve the efficiency and accuracy of critical illness prediction and real-time diagnosis and treatment decisions of doctors.

At present, machine learning has been widely used in the medical field by researchers [5-10]. Bendi Venkata Ramana et al. [11] used support vector machine, C4.5 decision tree, BP neural network to complete diagnosis classification based on textual dataset of liver. Patricio Miguel et al. [12] used logistic regression, support vector machine, and random forest classification algorithms to predict the presence of breast cancer based on blood sample data. Aljaaf et al. [13] proposed a multi-level risk assessment of developing heart failure based on C4.5 decision tree. Otoom et al [14] presented a system to analyze and monitor coronary arteries, their data set had 76 features and only 13 features were used. Studies by Morelli V et al. [15] and Ursini F et al. [16] had shown that inflammatory factors, blood lipids and uric acid can be used as indicators for the evaluation and prediction of cardiovascular adverse events in patients with diabetes after surgery. Zhang et al. [17] adopted a classification scheme based on a one-class kernel principle component analysis (KPCA) model ensemble that has been proposed for the classification of medical images, and the accuracy rate of recognition in the UCI breast cancer dataset (diagnostic) was 92%. Spanhol et al. [18] used fusion rules to combine different convolutional neural networks for classification, and the average recognition rate on the BreakHis dataset [19] reached 83.2%. Han et al. [20] designed a structured deep-learning model combined with data enhancement methods, the accuracy of their work on the BreakHis dataset was 96%. These studies demonstrate the positive role of machine learning methods in the prediction of disease. Janez Demsar et al. [21] used the RELIEFF method combined with machine learning methods such as Naive Bayes to prove that a small number of features may carry enough information to build a reasonable and accurate prediction model. Prerna Sharma et al. [22] used the improved gray wolf algorithm to select features and make predictions for Perkins, achieving an estimated accuracy of 94.83%. Filipe Lucini et al. [23] used data mining methods combined with machine learning algorithms such as support vector machines, to predict the future hospitalization and discharge of patients. Since a single classifier cannot make the diagnosis of all diseases, MadhuSudana Rao Nalluri et al. [24] proposed a hybrid classifier parameter optimization diagnosis system based on three evolutionary algorithms, support vector machines, and multilayer perceptron to achieve the diagnosis for mixture of diseases. The above researches show that artificial intelligence is a kind of effective method to predict the risk of illness and the features in the dataset are not need to be fully used in the prediction. However, most of the data features used in the prediction of the above studies were extracted using different methods for specific illnesses or relying on the experience of the physician. In 2018, L. Nelson Sanchez-Pinto et al. [25] analyzed 8 different feature selection methods for the variable selection methods of machine learning models currently used in clinical diagnosis. Their results showed that the regression-based feature selection method seems to achieve better parsimony in clinical prediction on smaller data sets, while tree-based methods perform better on larger data sets. L. Nelson Sanchez-Pinto et al. studied the effectiveness of two types of feature extraction algorithms in different-size data sets, and their work is instructive. For a range of critical illnesses, an effective method is still needed to analyze their association with preoperative and intraoperative indicators of patients.

In view of the characteristics of various medical indicators and different formats of critical illness patients, this paper proposed a critical illness indicator analysis model based on a supervised feature selection method. This model used a light gradient boosting machine (LightGBM) [26] and Shapley additive explanation (SHAP) values [27] to analyze the correlation between the indicators and critical illness and select the corresponding key indicators, so as to facilitate real-time diagnosis and treatment by doctors and improve the accuracy of prediction for critical illness.

2. The Proposed Model

This section mainly describes the key indicators analysis model for the prediction of critical illness. There are four parts in the proposed model and the overall flow of this model is shown in Fig. 1. Data extraction and merging, missing values processing and single unique values processing make up the first part. Collinear indicators processing is the second part. Analysis of the importance of indicators is the third part. Key indicators selection is the last part.

Fig. 1. Flowchart of the proposed method.

2.1 Data preprocessing

2.1.1 Data extraction and merging

To analyze the correlation between indicators and critical illness, the preoperative and intraoperative test data should be extracted and combined correctly. Since patients were not tested for all indicators during each preoperative examination, there were some missing values in various preoperative test tables. In the preoperative samples, the proposed model took 14 days as the threshold and completes the examination information of each patient through (1), where e_{i, j}, e_{k, j} represent the examination values of the indicator j detected of a patient at time i and k. Threshold set to 14 days was the result after a discussion with doctors. Then, the preoperative information of patients in each table was combined with reference to the medical record number and the operation time of patients in the surgical information table.

\(\begin{aligned}\mathrm{e}_{\mathrm{i}, \mathrm{j}}=\left\{\begin{array}{ll}\text { null } & i-k>14 \text { days } \\ \mathrm{e}_{\mathrm{k}, \mathrm{j}} & i-k \leq 14 \text { days }\end{array}\right.\end{aligned}\)       (1)

The intraoperative monitoring data of patients were time series. To combine these data with preoperative test data, the statistical values such as mean values, variance, standard deviation, max values, min values, kurtosis and skewness of indicators in the intraoperative monitoring data were calculated. Then the intraoperative monitoring data were represented by the statistic values.

After extracting the preoperative and intraoperative test data, the integrated examination data were generated by combining the two kinds of data with reference to the medical record number and the operation time of patients. Then, the test dataset has been extracted.

2.1.2 Missing value processing

Although the preoperative test dataset has been filled in the previous part, a large number of missing values still exist in the test dataset, which is caused by incomplete data records and different test indicators for different patients. The amount of missing value affects the contribution of this indicator for the prediction of critical illness and the indicators with a large number of missing values are invalid for classification. The proposed model statistically analyzed the missing values of all indicators. Firstly, the proportion of missing values in each indicator was calculated through (2), where m_i represents the proportion of missing values for indicator i, MissNum is the number of missing values in indicator i, TotalNum is the total number of values in indicator i, namely the number of samples in the test dataset. Then, a threshold M_T of missing value was set (as shown in Table 2, M_T =0.90). If the proportion of an indicator was higher than the threshold, the indicator was removed.

\(\begin{aligned}m_{i}=\frac{\text { MissNum }_{i}}{\text { TotalNum }}\end{aligned}\)       (2)

2.1.3 Single unique value processing

n addition to missing values, some indicators only have one kind of values in the test dataset. The cause of this problem is that the values of some test indicators for the patients are the same. For example, all patients in the test dataset may not have used a certain drug in the past. These indicators are the same for all patients, regardless of whether the patients are sick or not. Therefore, these indicators have no contribution to the prediction of critical illness.

To overcome this problem, the proposed model calculated the number of different values for all indicators in the datasets. Then the indicators which only have a single unique value were removed.

2.2 Correlation analysis

Due to the large number of indicators in the test dataset, there will inevitably be some correlation between different indicators. However, two indicators with high correlation have the same contribution for predicting critical illness. Therefore, it is necessary to remove indicators with higher correlation to improve the predictive efficiency of critical illness.

In addition to continuous test indicators, there are also non-continuous indicators such as gender. Therefore, the proposed method analyzed the correlation between all the indicators in the datasets by Spearman correlation coefficient. First, the values of the two indicators that need to calculate the correlation were sorted, and the order of each value in the sorting was recorded. Then the recorded sequences of the two indicators were subtracted to obtain the vector d. Finally, the correlation coefficient of the two indicators is calculated according to (3), where d_i is the value of d at position i, and n is the number of elements in the indicator, namely the number of samples in the dataset. To select highly correlated pairs of indicators, a threshold was set and the pairs of indicators with a correlation coefficient greater than the threshold was selected. Afterward, one of the indicators in each pair of the selected indicators pairs was removed.

\(\begin{aligned}\rho=1-\frac{6 \times \sum_{i=1}^{n} d_{i}^{2}}{n \times\left(n^{2}-1\right)}\end{aligned}\)       (3)

2.3 Importance analysis

To analyze the correlation between the key indicators and critical illness, the crucial part was estimating the importance of each indicator for the prediction of critical illness. The importance of the indicator represents how much this indicator contributes to the prediction.

In this part, the proposed model used LightGBM [26] as the classifier to predict critical illness. LightGBM was proposed by Guolin Ke et al. at 2017. It was an implementation of Gradient Boosting Decision Tree (GBDT) [28]. This algorithm used two novel techniques: Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB), which make LightGBM outperform XGBoost [29] and SGB [30] in terms of computational speed and memory consumption.

The objective function of LightGBM in the proposed model is shown as (4), where t is the current iteration, n is the total number of samples, y_i is the label of x_i, f_t represents the CART decision tree in the t iteration, f_t(x_i) is the prediction of x_i in the t iteration, L is the loss function and Ω is the regularization term.

𝑂bj^𝑡 = Σⁿ_i=1𝐿(𝑦_𝑖, 𝑓_𝑡(𝑥_𝑖)) + 𝛺(𝑓_𝑡)       (4)

Since the model was trained to predict critical illness, the contribution of each feature to the model was equivalent to the contribution of each indicator to preoperative critical illness prediction. Therefore, the importance of each feature in the model can be taken as the importance of each indicator for preoperative prediction of critical illness after training the model. In the proposed model, SHAP values was used with an explainer for LightGBM and analyze the contribution of indicators to the prediction. Through the explainer, each indicator was assigned an important value.

2.4 Key indicators selection

The importance of indicators was used to select the key indicators in this part. Firstly, the indicators were sorted in descending order of their importance. Secondly, the importance of all indicators was normalized by (5), where v_i is the importance of indicator i, N_i is the normalized importance of indicator i, m is the number of indicators in the test dataset. Thirdly, the cumulative sum (cumsum) of the normalized indicators was calculated by (6), where N_k is the cumsum of indicator k.

Finally, a threshold S_T was set and the indicators with cumsum values lower than the threshold was selected as the rough key indicators. The parameters are shown in Table 2. If the number of the rough key indicators was more than 25, the rough key indicators were set as the key indicator set. If the number of the rough key indicators was less than 25, the first 25 indicators in the indicator series were set as the key indicators.

\(\begin{aligned}N_{i}=\frac{v_{i}}{\sum_{i=1}^{m} v_{i}}\end{aligned}\)       (5)

𝐶_𝑘 = ∑^𝑘_𝑖=1 𝑁_𝑖       (6)

3. Experiment and Results

3.1 Description of the experiment

The data of two critical illnesses were collected from preoperative and intraoperative clinical features and laboratory results of patients in the southwest hospital in China. Diagnoses of heart failure and respiratory failure were derived from database entries. After data preprocessing, the number of samples for heart failure and respiratory failure is shown in Table 1. It is obvious that the number of negative samples is greater than the number of positive samples in each type of critical illness. In response to this problem, the downsampling method which called random undersampling was used to reduce the number of negative samples to the same as the number of positive samples. The parameters of the proposed model are shown in Table 2.

Table 1. Number of samples

Table 2. Experiment parameters

The test dataset was split into training samples and test samples in a ratio of 4:1. The indicators of each patient were used as the features of each sample. The labels of patients with critical illness are set to 1 and the labels of patients without critical illness are set to 0. In the experiment, all the importance of indicators and the results of prediction are taken from the average value of the classifiers after 10 runs.

3.2 Results of the experiment

The number of indicators in the samples of heart failure and respiratory failure are 197 and 189 respectively. After data preprocessing, the number of indicators removed from the samples of heart failure and respiratory failure are 52 and 77 respectively. Fig. 2 shows the normalized importance for top 20 indicators of the two critical illnesses. It can be seen that the contribution of the top 20 indicators for the prediction of heart failure is lower than the contribution of the top 20 indicators for the prediction of respiratory failure. Fig. 3 shows the graph for cumulative indicators importance of heart failure and respiratory failure. The number of indicators of the heart failure with cumsum values lower than S_T is 39. Thus, the top 39 indicators are used as the key indicators of heart failure. The number of the indicators of the respiratory failure with cumsum values lower than S_T is 35. Hence, the top 35 indicators are used as the key indicators of respiratory failure. The two kinds of key indicators are shown in Table 3.

Fig. 2. Normalized importance for top20 indicators

Fig. 3. Cumulative indicators importance of heart failure. The value of the ordinate for the intersection of the vertical dotted line and the curve in the three figures is S_T. The vertical dotted line divides the indicators into two parts, and the left part represents the number of indicators with cumsum values lower than S_T.

Table 3. Key indicators of two critical illnesses

To verify the effectiveness of the key indicators, it was used to generate multiple indicators of subsets. The number of subsets is the number of indicators in the key indicators. The first subset contains only the most important indicators of the key indicators. The second set contains the top two most important metrics from the key indicators. And so on, the last set contains all indicators in the key indicators. These indicators subsets were used to generate data subsets from the test dataset. Then, the data subsets were sorted in ascending order by the number of indicators included. The AUC (Area Under Curve) of these subsets in predicting critical illness was analyzed in turn by this order. The classifier used in the analysis was LightGBM.

The AUC of prediction using data subsets generated by the key indicators of heart failure and respiratory failure are shown in Fig. 4. It can be seen that, with the increase of indicators number, the AUC for the prediction of heart failure and respiratory failure keep rising and have reached the convergence around 0.96 and 0.92, respectively. Fig. 5 and Fig. 6 show the ROC curve and P-R curve for the prediction of heart failure and respiratory failure. For the two kinds of critical illnesses, the area under the dotted line is both slightly greater than that under the solid line. This indicates that the performance of the prediction using key indicators is better than the performance of the prediction using all indicators.

Fig. 4. AUC of the data subsets

Fig. 5. ROC curve for the prediction of the two critical illnesses. The red curve and green curve are the results of prediction using XGBoost and LightGBM respectively. The dotted line and solid line represent the results of prediction using all indicators and the key indicators respectively.

Fig. 6. P-R curve for the prediction of the two critical illnesses. The red curve and green curve are the results of prediction using XGBoost and LightGBM respectively. The dotted line and solid line represent the results of prediction using all indicators and the key indicators respectively.

In order to further verify the effect of the key indicators, LightGBM and XGBoost were used as classifiers to analyze the prediction using all indicators and the key indicator respectively. The predicted results using all indicators and the key indicators of the two kinds of critical illnesses were shown in Table 4 and Table 5. The results in Table 4 and Table 5 were taken from the 10-fold cross validation. HFA, HFK, RFA and RFK represent all indicators of heart failure, key indicators of heart failure, all indicators of respiratory failure and key indicators of respiratory failure respectively. The classifiers used in Table 4 and Table 5 are LightGBM and XGBoost respectively. These results include the accuracy, AUC, f1_score, sensitivity and specificity of the prediction for the heart failure and respiratory failure. For the heart failure, the experimental results showed that the prediction results using key indicators are similar to those using all indicators in accuracy and specificity, and slightly higher in AUC, f1_score and sensitive than those using all indicators. For the respiratory, the experimental results show that the prediction results using the key indicators are higher than the prediction results using all indicators, especially in terms of AUC, f1_score and sensitive.

Table 4. Predicting results using LightGBM

Table 5. Predicting results using XGBoost

These results suggest that the key indicators extracted from the model can effectively predict the critical illness.

4. Discussion

Machine learning algorithms which use distance measurement are sensitive to data with missing values. Therefore, the proposed model in this paper chooses LightGBM to analyze the importance of indicators.

The prediction performance of these critical illnesses using the key indicators extracted by the proposed model from the two kinds of critical illnesses are convergent. This shows that the key indicators extracted by the model are effective. At the same time, for the samples of heart failure and respiratory, the performance of key indicators extracted from the model in the prediction of critical illness was slightly higher than the performance of using all indicators. These experimental results show that the key indicators can replace all indicators to make an effective prediction of critical illness. This indicates that the model successfully removes redundant indicators from the dataset and avoids the influence of these redundant indicators on the prediction performance of critical illness.

However, the key indicators extracted from the model still have some shortcomings. To ensure the validity of patient samples, the model does not fill in the missing values in the examined data of patients. Although the model has removed some indicators with a lot of missing values, there still exist indicators with missing values in the dataset. Due to the influence of missing values and other factors of the data, the importance of some indicators that should be significant for the critical illness according to doctors' experience may not be ranked in the first few in the important analysis of the model. However, some common indicators have no missing values, so the importance obtained in the model is higher than the actual importance of them. For the model, the future work needs to analyze the impact of missing values and other factors on the importance of indicators, combined with the doctor's prior knowledge of the importance of the indicators, so that the analysis results obtained by the model are closer to their real correlation with critical illness.

For the model's generalizability, it is possible to obtain different results when using the same modeling approach on data from different centers. This can occur due to several factors: differences in data distribution, sample size and quality, variations across centers. In the future study, the model's generalization and versatility need to be improved through proper data and model handling techniques, leading to more consistent predictions across various data centers.

5. Conclusion

This paper proposes a key indicator analysis model of postoperative critical illness based on machine learning. The model includes four modules: data preprocessing, analysis of the correlation in indicators, analysis of the importance for indicators and selection of key indicators. The missing value, single value, correlation and importance of indicators are counted and calculated in the first three modules, and the key indicators in the data are selected based on these results in the lase module. In this paper, two kinds of samples for critical illnesses from surgical patients were used in the experiment. The prediction accuracy of postoperative critical illness obtained by using the key indicators extracted from the model is slightly higher than those obtained by using all the indicators. The experimental results show that the model presented in this paper can effectively analyze the correlation of the indicators with postoperative critical illnesses and extract the key indicators of postoperative critical illnesses.

Funding

The work was supported by Chongqing Municipal Natural Science Foundation (NO: CSTB2022NSCQ-MSX0894), Youth Innovation Promotion Association of Chinese Academy of Sciences (No. 2020377), National Natural Science Foundation project(82070630), Chongqing Talent Plan Project (CQYC202103080) and Chongqing Graduate Education Teaching Reform Research Project (Key subject: yjg202039).