Malware Detection with Directed Cyclic Graph and Weight Merging

Abstract

Malware is a severe threat to the computing system and there's a long history of the battle between malware detection and anti-detection. Most traditional detection methods are based on static analysis with signature matching and dynamic analysis methods that are focused on sensitive behaviors. However, the usual detections have only limited effect when meeting the development of malware, so that the manual update for feature sets is essential. Besides, most of these methods match target samples with the usual feature database, which ignored the characteristics of the sample itself. In this paper, we propose a new malware detection method that could combine the features of a single sample and the general features of malware. Firstly, a structure of Directed Cyclic Graph (DCG) is adopted to extract features from samples. Then the sensitivity of each API call is computed with Markov Chain. Afterward, the graph is merged with the chain to get the final features. Finally, the detectors based on machine learning or deep learning are devised for identification. To evaluate the effect and robustness of our approach, several experiments were adopted. The results showed that the proposed method had a good performance in most tests, and the approach also had stability with the development and growth of malware.

1. Introduction

The concepts of malware could include all binary files that would injure the computing system intentionally. In recent years, malware have more diversities in techniques and destination. Nowadays, many malware adopt the way that mix with multiple attack methods to reach the target, and most of them take obfuscation or encryption method to disguise themselves and avoid detection.

Contract to the development of malware techniques, most of the defense systems still use static analysis as their primary measurement, which is mainly based on signature matching [1]. Some defense systems have developed dynamic analysis, including sensitive behaviors, access to critical privileges, network analysis, and key process monitor as their assistant method [2], [3]. However, all of these methods are mainly focused on specific malware or malware classes so that they are limited when meeting new types or variants of malware. Besides, they are also weak to the anti-detection techniques, which would let the detectors be deceived by disguised malware and cause damage. All of the situations indicate that developing a new method of detection is essential.

In recent years, some researchers have focused on the dynamic detection of malware based on Application Programming Interface (API) call sequences, and have proved their effects [4], [5], [6]. API is a set of instructions used to program software applications. All programs can interact with the Windows API and access pre-defined functions by invoking API calls to make use of facilities provided involving base services to access resources such as system files, processes, threads, devices, and advanced services [7]. The API calls of malware reflect its high-level functionality and can be used to understand its overall behavior. API call sequence analysis is, therefore, an effective method of malware analysis [8].

In this work, we propose a novel approach for feature extraction and detection. The approach extract features from the specific sample and combines the feature with the general characteristics of malware. Besides, we adopt Principal Components Analysis (PCA) for reducing dimensions and consumption of resources. Moreover, some models based on machine or deep learning are devised as detectors to identify the features. The main contributions of our work are listed as follows:

• We propose a novel method for feature extraction from samples with a solid structure. The structure is based on the invoke relationship between API and could maintain most information on the API call sequences.

• To get general features of malware, we design a new weighting model based on Markov Chain, the model can be trained with numerous known malware to decide the weight of each invokes caused in malware. Then the features extracted from a specific sample will be combined with general features of whole malware so that the newly generated features can maintain the characteristics of the sample itself while increasing the generality of the features to defend the disguises and variants of malware.

• We adopt PCA to reduce consumption and improve the performance of the approach. Afterward, we use several models based on machine learning and devise a few other models based on neural networks for detection. The purpose of using various models is to prove the universality of our approach and compare the performance among different models.

• We evaluate the proposed method with a series of experiments and diverse datasets. The results present that our approaches are effective with most detectors and have robustness when meeting the development. However, some problems are also found in the experiments, which need further study.

The remainder of the paper is organized as follows. Some related works are reviewed in Section 2. Our method of feature extraction and detection are presented in Section 3 and 4. The details of experiments and results are shown in Section 5, and the whole work is concluded in Section 6.

2. Related Work

In recent years, static detections of malware have faced severe challenges from the anti-detection technique evolved subsequently, especially encryption and disguise. Therefore, more researchers have concentrated on dynamic detection techniques. One most efficient of them is API call sequences analysis.

Generally, the researches about API-based malware detection are mainly focused on the method of feature extraction such as n-grams, features, and flow graphs, and some researchers combined two of the above of the analysis to propose new approaches. Most of them got good results. Naval et al. [9] proposed a new strategy adopted both graphs and n-grams to generate a unique model called Ordered System-Call Graph (OSCG). Then the graph was transformed into a Feature Vector Table (FVT) using semantically relevant path extraction and was processed with an ensemble-based algorithm. The results showed the accuracy of detection could get up to 95%.

Detections based on Markov chains have also been adopted and developed by some researchers. Onwuzurike et al. [10] proposed a new method named MaMaDroid based on static analysis of API and Markov chains of the call graph to detect malicious Android apps. Ficco [11] improved the approach by exploiting dynamic analysis system calls instead of static. The developed approach was applied in IoT malware detection and get an F-measure up to 89%.

With the development of deep learning, some researchers also attempted to apply the technique of neural networks to malware identification. For example, Ganesh et al. [12] extracted features from APK samples and created a 12x12 vector image, then the image was detected by CNN models. The results showed an accuracy of 93%. However, detection approaches based on deep learning methods also are confronted with risks, specifically attack to deep learning models. Chen et al. [13] proposed an adversarial way to attack deep learning detectors based on Jacobian-based Saliency Map Attack (JSMA) and optimal perturbations onto Android APK. Then the approach was evaluated with MaMaDroid and Derbin. The results showed that the attacking method could attack of MaMaDroid and Derbin effectively. The research verified the seriousness of adversarial attacks towards the detection methods based on deep learning, which should be valued in all related researches.

3. Data Processing

In our work, a new detection method is proposed that consists of feature extraction, weighting, data compression, and detectors based on machine learning and deep learning. In this section, the implementation of data processing will be presented, and the details of the detection method would be put in the next section.

3.1 Weighted Directed Cycle Graph

Directed Acyclic Graph (DAG) is a continuous graph with a finite set of points S and a set of directed edges E, and any edge e from vertex s_i to another vertex s_j (s_i,s_j ∈ S), and any vertex cannot return itself and the directed edge sequence [14]. DAG can represent the set of possibilities that satisfied Markov property, which clarifies that the probability of transitioning from a state to another only depends on the current state.

DAG has been widely used in many fields. However, some graph nodes may access themselves. In this case, DAG is not practical and needs to be represented by Directed Cyclic Graph (DCG).

In the calling of software API, an API may call other APIs many times, or it may call itself. Therefore, we use DCG diagram to describe the calling relationship of software API. In the DCG graph of API, we define that the edges of DCG graph will be weighted according to the calling relationship of API. In other words, the weight of the directed edge between S_i and S_j is the number of times API S_i calls API S_j. In particular, the weight of the directed edge between S_i and S_i is also the structure of S_i calling S_i. As shown in Fig. 1, it describes the calling relationship between vertex (API) and vertex.

Fig. 1. Sturcture of DCG

We use adjacency matrix to describe a DCG graph called by API. Both rows and columns are used to represent API. The value of i row and j column in the matrix represents the number of times API S_i calls API S_j, which represents the weight of DCG edges. The adjacency matrix of DCG graph shown in Fig. 1 can be represented by Table 1, the weight of the edge in the graph corresponds to the value of the matrix one by one.

Table 1. Adjacent matrix of the DCG in Fig. 1

3.2 Weighting from Markov Chain

Markov chain is a set of discrete random variables with Markov property[15]. Markov chain is widely used in the classification and processing of sequential data, especially in the dynamic detection of malicious code [16], [17].

In our method, in order to identify and quantify the sensitivity of API calls, we calculate the weight of each call from one API to another based on Markov chain. Therefore, we assume that the API call is independent of the previous process and do not consider the sequence of API calls when calculating the weight. Therefore, when we consider the API calling process of the sample, we only consider the number of each API calling each other, not its sequence. In this way, the API calls related to time series can be represented by DCG diagram, and then the weight can be calculated. The specific calculation process is shown in Algorithm 1.

Algorithm 1 weight calculation algorithm based on Markov chain

When we calculate the weight, firstly, we need a Data Set which only contains malicious code and is rich enough to cover the general characteristics of malicious code. Then, all APIs in the Data Set are numbered (S₁ to S_M), and the corresponding DCG graph and adjacency matrix of samples are generated based on Markov chain. We define the DCG graph based on Markov chain as "weighted graph", denote it as matrix M, and generate its weight graph. In the weight graph, the weight W_ij of edge e_ij is calculated as follows:

\(w_{i, j}=\frac{N\left(e_{i, j}\right)}{\sum_{k=1}^{n} N\left(e_{i, k}\right)}\)       (1)

where \(N\left(e_{i, j}\right)\) means the number of calls that from API s_i to s_j, and \(\sum_{k=1}^{n} N\left(e_{i, k}\right)\) refers to the sum of calls that invoked from s_i. The nature of the weight in Markov Chain is the probability of the event's occurrence. Hence the sum of the weight invoked from an API must be 1:

\(\sum_{k=1}^{n} w_{i, k}=1\)          (2)

In the weighting stage, the weight of each sample is calculated first. After the weight of all samples is calculated, the weight matrix of each sample is accumulated, and the final weight graph is normalized by using the above method again to obtain the adjacency matrix of the final weight graph. The adjacency matrix can express the importance of API call and the characteristics of samples.

3.3 Feature extraction of detection samples

After getting the weight graph of malicious code Data Set, we need to use it to extract the feature graph of detection samples. The specific process is as follows:

Firstly, the DCG graph of the detection sample is generated.

Secondly, the DCG graph and the weight graph are used for point multiplication to obtain the feature graph of the detection sample.

The process of feature map extraction is shown in Fig. 2. From the above two steps, the DCG graph of detection samples and the weight graph of malicious samples can be obtained respectively. Both of them are represented by adjacency matrix. It is not difficult to find that the dimensions of the two matrices are the same, which are m × m dimension matrices. In feature extraction, we multiply the two matrices, that is, multiply the corresponding elements, and get the feature graph of the detection sample, which is also m × m dimension matrix. Similarly, the elements of the adjacency matrix of the feature graph are normalized, and the processing formula is as follows:

Fig. 2. Merge process of DCG and weighting graph

\(W_{i j}=w_{i j} \bullet n_{i j}\)          (3)

Where n_ij is the value of the i row j column in the adjacency matrix M of the sample to be detected, that is, the number of times e_i is called. w_ij is the value of ith row jth column in the weight matrix W. In the figure, the calculation principle is represented by a simple example, and the feature graph extracted from the figure is represented as an adjacency matrix, as shown in Table 2.

Table 2. Adjacent matrix of the Merged graph

3.3 Principal Components Analysis

To improve the performance of the evaluation and reduce time consumption, PCA technique is adopted to centralize the information and reduce the dataset volume.

PCA is a prevalent statistic processing technique for data analysis and preprocessing, which has been applied widely in data processing and mining [18], [19], [20]. The main propose of PCA is to reduce the dimension of analyzed data while maintaining the most information [21]. Generally, PCA can transform an m-dimension dependent variable into an n-dimension independent one (m < n) under the premise of keeping most of the information, and the transformed variables are called the principal components (PCS).

Considering a data set X with the number of data M and the dimension N, so that X = [ x , x , ..., x , ..., x ]^T and x = [ v , v , ..., v ] . The PCA process could be presented as following steps:

• Before calculation, each variable \(v_{i, j}\), j = 1, 2, ..., N of the vector \(x_{i}\) was rescaled as

\(\mu_{i}=\frac{1}{M} \sum_{j=1}^{m} v_{i, j}\)        (4)

\(\theta_{i, j}=v_{i, j}-\mu_{i}\)           (5)

each variable \(v_{i, j}\) would be replaced with \(\theta_{i, j}\), and the vector x_i finish the rescalation. 

• After the rescalation, the covariance matrix is further calculated and eigenvalue decomposition is performed as:

\(R=X^{T} \cdot X=P \cdot \Lambda \cdot P^{T}\)       (6)

where P is the matrix formed by the eigenvector, Λ is the eigenvalue matrix and is arranged as follows by size:

\(R=(1-N-1) X^{T} \cdot X=P \cdot \Lambda \cdot P^{T}\)      (7)

where \(P P^{T}=P^{T} P=I_{M}\).

\(R=\left[P_{q} P_{M-q}\right]\left[\begin{array}{cc} \Lambda_{q} & 0 \\ 0 & \Lambda_{M-q} \end{array}\right]\left[\begin{array}{c} P_{q}^{T} \\ P_{M-q}^{T} \end{array}\right]\)      (8)

where Λ is the eigenvalues matrix of R with decreasing order.

4. Detection Method

In this section, several detection methods based on machine learning and deep learning are proposed.

4.1 Machine Learning Approaches

For the detectors that based on machine learning, Support Vector Machine (SVM), Decision Tree (DT), Random Forests (RF), and Naive Bayes (NB) are adopted to evaluate our approaches. All those models are supervised learning models, which need to be trained with lots of labeled data to obtain a model for classification. Notably, the principals of each model are very different, so that they will have distinct results in malware detection.

Support Vector Machine is a valid binary classifier that is applied widely in the task of two-class identification [22]. The target of SVM is to get a hyperplane that could identify a binary class with maximum margin from support vectors so that the hyperplane could work well in classify new input data.

The principal of Decision Tree is to build a tree that consists of a non-leaf node for representing attribute and leaf-node for labeling [23]. The method can learn the characteristics from the training data by using higher information gain, and the attributes will be used for classification.

Random Forest is a combination of several self-determining decision tree [24]. Each tree in the method will process independently, and all results of the trees will be collected and finally voted for the ultimate result. Because the outcome of Random Forest is based on various trees, it usually performs better than a single Decision Tree.

Naive Bayes is a classification method based on Bayes theorem, which assumes that the features of the data are independent among them. The model calculates the conditional probability of each feature in the training phase, and the possibility will be adopted for computing posterior probability and then classification.

4.2 Deep Learning Approaches

Besides the models of machine learning, some models based on CNN and RNN are also used to detect malware. Each model is customized to fit the detection that effective for our method.

The structure of CNN is shown in Fig. 3. The input layer of the model is a matrix generated from PCA. The output size of each layer is marked above it. C₁ is convolution layer, which contains 16 convolution cores with a size of 5 × 5 and a step size of 1; C₃ is convolution layer, which contains 32 convolution cores with a size of 5 × 5 and a step size of 1; S2 and S4 are pooling layers with the same size of 2 × 2; the number of neurons in the full connection layer is 128 and 100; the final output layer is softmax, the output is a two-dimensional variable for classification.

Fig. 3. Sturcture of CNN

The architecture of RNN is shown in Fig. 4. RNN is a sequence-based model that output a h_t at each step t, and each state of current step will depend on both current input x_t and previous state h_i-1 [26], [27].

Fig. 4. Sturcture of RNN

In this model, we combine a full connection layer, a Relu unit and an RNN cell as a network structure at one time. The full connection layer and Relu unit are used to extract sample features, RNN is used to identify the attributes of data.

5. Experimental Classification Results and Analysis

5.1 Datasets and Environment

Dataset: In this work, we collected a dataset consists of 13624 samples, which have 6686 malware and 6938 benign samples. The detailed statistics were presented in Table 3. The malicious samples came from VirusTotal and VirusShare, and the benign samples came from system programs as well as the Internet. The benign dataset was split into five parts evenly to fit the numbers of malware dataset for each year. In our experiments, 10-fold cross-validation method is used to train and verify the model. The method is as follows: first, all samples are randomly divided into 10 subsets, each time 9 subsets are selected as training samples, and another subset is taken as test samples. In this way, the model can be trained ten times, and the remaining validation sample can be used to evaluate the accuracy of the model after training. Finally, average the ten test results as the final test results. For weighting models, an API log dataset was adopted for training. The dataset for weighting included 62307 malware samples that were obtained from VirusShare and selected randomly so that the generality of the weighting model could be guaranteed.

Table 3. Datasets for Evaluation

Environments: The experiments were performed on a workstation with Ubuntu 18.04 system. To monitor and extract the call sequences of each sample, a Cuckoo sandbox was deployed on the workstation as the running environment of the samples.

5.2 Experiment Steps

A. Weighting and Generating of Graph

In this phase, the extracted call sequences were numbered firstly. Then the sequences were transformed into the DCG. The index of the DCG presented the corresponding API and the value in each cell referred to the appearance number of the API invoked by the previous API. After generating DCG, the graph mixed with the weighting graph, so that a weighted DCG with a unique value for each edge was created.

In the weighting phase, firstly, the weighting graph was trained from the dataset. The initial weighting graph had 1609 rows and columns after the training. Then the weighting graph was used to generate merged graphs for detection.

B. Detection and Evaluation

After data processing, the final data was detected with the proposed models. To evaluate the performance of each model, the index of accuracy, precision, recall, and F1-score were adopted.

Accuracy is a standard metric that measures the exactitude of prediction. Precision refers to the number of predicted positive samples that are really positive. Recall is the number of positive examples in the sample that are predicted as positive. F1-Score is a comprehensive measure index of the classification model. All of these indexes could be computed as the following equations:

\(\text { Accuracy }=\frac{T P+T N}{T P+T N+F P+F N}\)       (10)

\(\text { Precision }=\frac{T P}{T P+F P}\)       (11)

\(\text { Recall }=\frac{T P}{T P+F N}\)       (12)

\(F 1-\text { Score }=\frac{2 \times \text { Precision } \times \text { Recall }}{\text { Precision }+\text { Recall }}\)       (13)

where TP is True Positive, which refers to the number of positive samples that are predicted as positive. FN is False Negative, namely the number of positive samples that are predicted as negative. FP is False Positive, which means the number of negative samples that are predicted as positive. TN is True Negative, which is the number of negative samples that are predicted as negative. In our work, malicious samples were labeled as positive, and benign samples were labeled as negative.

Besides, the receiver operating characteristic (ROC) curve was also be implemented as a measure to evaluate the models. The curve used False Positive Rate (FPR) as the X-axis and True Positive Rate (TPR) as the Y-axis. Both values could be computed as (14) and (15). Moreover, the Area Under Curve (AUC) was calculated to estimate the overall accuracy of the models [28].

\(T P R=\frac{T P}{T P+F N}\)       (14)

\(F P R=\frac{F P}{F P+T N}\)       (15)

5.3 Detection Result and Analysis

Table 4 shows the evaluation results of the model based on the datasets of different years. The results show that both machine learning model and deep learning model have achieved good prediction results, and the performance of each model is relatively close, which does not show that the model is the best. It shows that our feature extraction method is effective, and some main machine learning models based on this feature extraction method have good detection results, which shows that our proposed feature extraction method is universal.

Table 4. Evaluation of the models with different datasets

It can be seen that with the increase of years, the prediction accuracy of the model shows a downward trend. We think that this is because with the increase of years, the design level of malicious code is getting higher and higher, and it has more powerful anti detection ability, which is a big challenge to our model. In the future research, we will try to improve the self optimization ability of the model to solve the problem.

Fig. 5 showed the ROC curves of the evaluation, where DT, RF, and NB are the abbreviation of Decision Tree, Random Forest, and Naive Bayes. The AUC of each model was presented in the legend of the figure. It could be seen that most models had a good effect in evaluation, except Naive Bayes that had a poor performance and RNN that performed very unstably in the detection.

Fig. 5. ROC Curve of evaluation

In summary, the result indicated that our approach had a good effect on the general malware detection with the most models based on machine or deep learning, which could prove the effectiveness of our approach.

6. Discussion and Conclusion

In this work, we propose a new malware detection approach. The approach mainly consists of the generation of Directed Cyclic Graph and the weighting based on the Markov Chain. To improve the performance and accuracy of the method, Primal Component Analysis is applied to reduce the volume of data as well as retain the primary information related to the detection. Finally, some models based on machine learning and deep learning are adopted to evaluate the effect of our approach. The result shows that the method had an excellent performance in most of the detection, with the highest accuracy of 96.73%. The AUC of most models could keep above 90% except NB, which could confirm the robustness and universality of the approach.

Inevitably, our method also has some problems and limitations, which need to be further studied. First of all, we can see that our method needs PCA method to reduce the dimension, so as to meet the requirements of fixed input of the model, and it is not adaptive when facing the malicious code with large differences. In the future work, our research will focus on the adaptive learning of features and models, so that the features of malware detection can adapt to the development of malware and related technologies, without human interference. In addition, in view of the increasing trend of adversarial attacks based on machine learning model, we will further study the related defense measures.