9.2.1 Software Defect Prediction Based on Deep Learning

Singh et al. [15] used public data set AR1 for predicting fault proneness of modules. They compared logistic regression technique with six ML classifiers (DT, group method of data handling polynomial method, artificial neural network (ANN), gene expression programming, SVM, and cascade correlation network). The performance was compared by computing the area under the curve using Receiver Operating Characteristic (ROC) analysis where it was concluded that the value generated by DT was 0.865 which outperformed regression and other ML techniques. Dejaeger et al. [16] considered 15 distinct Bayesian Network (BN) classifiers and comparison was done with ML techniques. For the purpose of feature selection, Markov blanket principle was used. AUC and H-measure was tested using statistical framework of Demšar. The result showed that simple and comprehensible networks having a smaller number of nodes can be constructed using BN classifiers other than the NB classifier. Kumar et al. [17] experimented on 30 open-source projects to build a ML-based model for software fault prediction model using Least Square SVM (LSSVM). They applied 10 distinct feature selection techniques. Their prediction model was only appropriate for project with faulty classes less than the threshold value. B. Twala [18] performed software fault prediction on four NASA public data sets using DT, SVM, K-Nearest Neighbor, and NB. He concluded that NB classifier was most robust and DT classifier the most accurate.

9.2.2 Software Defect Prediction Based on Deep Features

S. Wang et al. [19] proposed a representation learning algorithm using Deep Belief Network (DBN) which helps in learning semantic program representation directly from source code. They worked on 10 open-source projects and showed that directly learned semantic features considerably improve both within- and cross-project defect prediction (WPDP and CPDP). On an average, WPDP was improved by 14.2% in F1, 11.5% in recall, and 14.7% in precision. CPDP approach beats TCA+ having traditional features by 8.9% in F1. Miholca et al. [20] proposed HYGRAR, a non-linear hybrid supervised classification method for software fault prediction. HYGRAR combined relational association rule mining and ANNs to distinguish between faulty and non-faulty software objects. For experiment purposes, they used 10 open-source data sets and validated the outstanding performance of the HYGRAR classifier. J. Li et al. [21] proposed a framework called Defect Prediction via CNN (DP-CNN) that used deep learning in order to effectively generate features. On the bases of program’s Abstract Syntax Trees (ASTs), they initially extracted token vectors and then encoded them as numerical vectors with the help of the process of word embedding and word mapping. Then, these numerical vectors were fed into CNN that automatically learnt structural and semantic program features. After this, for perfect software fault prediction, they combined traditional hand-crafted features with the learnt features. The experiment was conducted on seven open source project data. The measurement was done on the basis of F-measure. The final results showed that DP-CNN improves the state-of-the-art method by 12%. Cong Pan et al. [22] proposed an improved CNN model for WPDP and compared their results to existing CNN results and an empirical study. Their experiment was based on a 30-repetition holdout validation and a 10 * 10 cross-validation. Experimental results showed that their improved CNN model was comparable to the existing CNN model, and it outperformed the state-of-the-art ML models significantly for WPDP. Furthermore, they defined hyperparam-eter instability and examined the threat and opportunity it presents for deep learning models on defect prediction.

9.2.3 Deep Learning in Software Engineering

Apart from software defect prediction, deep learning models have been used in software maintenance [23], code clone detection [24], defect detection [25], and other areas. Guo et al. [23] used a Recurrent Neural Network (RNN) model in software maintenance to create links between requirements, design, source code, test cases, and other artifacts. Li et al. [24] proposed a deep learning-based clone detection approach. In their paper, they used ASTs tokens to represent method-level code clones and non-clones to train a classifier, and classifier used to detect code clones. Their methods accomplished similar performance with low time cost. Nguyen et al. [25] in defect prediction used DNN for bug localization. Their model aim was to solve lexical mismatch problem, and pointed out that the terms used in bug report are different from the terms and code tokens used in source files. Their model achieved 70% accuracy with five recommended files.

Other software engineering areas influenced by deep learning are source code organization [26], run-time behavior analysis [27], feature position [28], vulnerability analysis [29], and code novelist identification [30].

9.3.1 Software Defect Prediction

Figure 9.1 presents a typical file level defect prediction process based on ML concepts, which is adopted by many researchers in most recent studies. From Figure 9.1, the first step of the procedure is to extract program modules (i.e., source files) from the repositories. The second step is to cate-gorized program modules defect (i.e., buggy) or no-defect (i.e., clean). The categorization is based on post-release defects collected from PROMISE repository (i.e., NASA Metrics Data Program). The third step is to extract features from the categorized program modules to form training instances. The features consist of code and design metrics. The fourth step is to build a classification model and use training instances to train the model. In our case, we had selected three ML models (RF, DT, and NB) as well as DNN for comparative analysis with our proposed NCNN model. The last step is to feed new program feature instances into the trained classifier to predict whether a source file is defective or non-defective (Buggy/Clean).

9.3.2 Convolutional Neural Network

CNN is a special kind of neural network which is used to process data and has a recognized, grid-like topology [31] such as one-dimensional (1D) time series data and 2D image data. CNN has been extremely successful in practical applications, with speech recognition [32], image classification [33], and natural language processing [34, 35]. In our work, we influenced our proposed model to extract features from software repositories (i.e., source files).

Figure 9.2 demonstrates a basic CNN architecture. It consists of convo-lutional layers, pooling layers, and a simple fully connected network, along with a dense network. Neuron units are also connected to all neuron units of its neighboring layers. Even neural units connected to these two layers are sparsely connected, which is determined by kernel size and pooling size. The architecture represents the two features of CNN: sparse connectivity and shared weights, which allows CNN to capture local structural information of inputs.

The sparse connectivity property means that each neuron is connected to only a limited number of other neurons, and in CNN, it is controlled by kernel size and pooling size. From Figure 9.2, if we take node V₃ and kernel size 3, it only affects three nodes in convolutional layer, i.e., h₁, h₂, and h₃, whereas node h₄ is not affected by V₃. Each subset acts as a local filter

connecting to the next layer in CNN which produce strong responses to a spatially local input pattern. To find the output to the next layer, each local filter will multiply by outputs from the previous layer, then add a bias and perform a non-linear transformation. From Figure 9.2, it is observed that the i^th neuron in the m^th layer (convolutional layer) as h_i^m, the weights of the i^th neuron in the (m – 1)^th layer as W_i^m–1, the bias in the (m– 1)^th layer as b^m–1, and, in our work, we use rectified linear units (RELUs) which were recently shown to give a better performance in many neural network classifications tasks. The out can be calculated as follows:

(9.1)

To produce values from output layer, Softmax activation function is used which is also a type of sigmoid function. Softmax normalizes each neuron’s output to a range 1 and 0. It is non-linear in nature. It is usually used when we are trying to handle multiple classes.

The mathematical expression for Softmax activation function is as follows:

(9.2)

where

z: The input vector to the softmax function.

zi: All the zi values are the elements of the input vector to the softmax function, and they can take any real value, positive, zero, or negative.

e^zi: The standard exponential function is applied to each element of the input vector. This gives a positive value above 0, which will be very small if the input was negative, and very large if the input was large.

: The term on the bottom of the formula is the normalization term. It ensures that all the output values of the function will sum to 1 and each be in the range (0, 1), thus constituting a valid probability distribution.

K: The number of classes in the multi-class classifier.

Shared weight means filter shares the same parameterization (weight vector and bias). From Figure 9.2, connecting the input layer and convo-lutional layer indicated by all the solid black lines which share the same parameters. The same is also true for the blue sparse-dotted lines and the orange dense-dotted lines. This concept of share weights allows a CNN to capture features that are self-determining their positions and efficiently diminish model capacity.

Another significant concept of CNN is max-pooling, which partition the output vector into several non-overlapping subregions, and outputs the maximum value of each sub-region. This keen concept of reducing the dimensionality of in-between representations provides additional robustness to our defect prediction.

The efficacy of CNNs largely works in an empirical manner, with the researcher tuning her models as per the application domain and the data available. Thus, parameter tuning is a key to train a successful CNN. We will discuss in Section 9.4 how to set these parameters in our proposed model.

9.4.1 Data Set Description

There are a number of open-source data sets available online for the analysis of defect prediction models. For the study, four NASA system data sets (KC1, PC1, PC2, and KC3) are selected from PROMISE repository [13] which is freely available as public data sets. The selected data sets are of different sizes and different number of set of metrics, i.e., KC1, have 22 attributes with 2,109 instances, PC1 have 22 attributes with 1,109 instances, PC2 have 37 attributes with 745 instances, and KC3 have 40 attributes with 194 instances. These data sets contain software metrics like Halstead and McCabe metrics and a Boolean variable that indicates defect or no-defect proneness of a module. Table 9.1 displays characteristics of the NASA data sets (PC1, PC2, KC1, and KC3).

The WEKA (Waikato Environment for Knowledge Analysis) tool was used for the statistical output processing of data sets. WEKA is open-source software that gives the user the power of pre-processing, implementation of well-known ML algorithms, and conception of their data so that one can develop ML techniques and apply them to real-world data problems. The data was analysed i.e., the accuracy of different data sets was calculated using various classifiers, namely, RF, DT, NB [12], and DNN [14]. The results of these classifiers were then compared with the results generated by our proposed NCNN model.

Moreover, in this data set, we are provided with 21 traditional defect prediction features for each source file, including Lines of Code (LOC), McCabe complexity measures, Halstead base measure, and derive measure. The 21 traditional features are carefully extracted from PROMISE Software Engineering Repository [13]. We list the detailed description about the 21 features in Table 9.2.

9.4.2 Building Novel Convolutional Neural Network (NCNN) Model

This section presents our proposed NCNN model based on 1D CNN model. The overall network architecture of NCNN is shown in Figure 9.3. NCNN model consisted of two convolutional layers, two max-pooling layers to extract global pattern, a flattening layer and two dense layers to generate deep features and help better simplification, and, finally, a convo-lutional linear sequential model classifier to predict whether a source file was defect.

Other details about NCNN architecture are mentioned below:

• For the proposed NCNN model’s modeling, Python 3.5.2 is used. With the help of Keras pre-processing (version: 1.1.2), which is a neural network library written in Python and which is also capable of running on top of TensorFlow (version: 2.3.1), the NCNN related results were generated. The experiment was executed using the system having 64-bit operating system, x64 processor with 16 GB RAM.
• For predicting software defects convolutional linear sequential model classifier is proposed which is implemented in Keras. After pre-processing of labeled source files, we split data set into training set and test set with split ratio [75:25]. We fed our training data to our NCNN model, both weights and bias are fixed, and then for each file in the test set, we fed it into defect prediction model to get our prediction results. The obtained result is in the form of 0 and 1 and based on which we predicted a source file as defective or non-defective. If the result was above 0.5, then it is considered as defective, otherwise it was considered as non-defective.

• Since we use two convolutional layers, two max-pooling layers, two dense layers, and a flattening, as increasing more depth of deep models, we might get better outcomes.
• Input layers and hidden layers used a RELU activation function and last layer which used the SoftMax function for classification.
• 9 We used Adam optimizer, as an optimization function in order to update the weight of the network after every single iteration. We used binary cross-entropy as the loss function.

9.4.3 Evaluation Parameters

This section presents the effectiveness of our NCNN by comparing its F-measure and accuracy on defect prediction with other state-of-the-art methods such as RF, DT, and NB [12] and DNN [14]. We also explain some basic terminologies associated with software defect prediction. A training set refers to a set of instances used to train a model, whereas a test set refers to a set of instances used to evaluate the learned model. When applying defect prediction, the training set and the test set come from the same source file. In the field of ML and, specifically, the problem of statistical classification, a confusion matrix, also known as an error matrix is used. A confusion matrix is a summary of prediction results on a classification problem. The number of correct and incorrect predictions are summarized with count values and broken down by each class. This is the key to the confusion matrix. The confusion matrix shows the ways in which classification model is confused when it makes predictions. It gives us insight not only into the errors being made by a classifier but more importantly the types of errors that are being made. Figure 9.4 shows the description regarding the confusion matrix.

Class 1: False

Class 2: True

Where the above terms are defined as follows:

1. 1. True: Observation is true.
2. 2. False: Observation is not true.
3. 3. True-Positive (TP): Observation is true, and is predicted to be true.
4. 4. False-Negative (FN): Observation is true, but is predicted false.

1. 5. True-Negative (TN): Observation is false, and is predicted to be false.
2. 6. False-Positive (FP): Observation is false, but is predicted true.

The other evaluation parameters used in this research work are TP Rate (TPR), TN Rate (TNR), FN Rate (FNR), and FP Rate (FPR), precision, recall, F-measure (also called F1-score), and accuracy.

TPR is when it is actually positive, how often does it predict positive?

TNR is when it is actually negative, how often does it predict negative?

FPR is when it is actually negative, how often does it predict positive?

FNR is proportion of positive which yield negative test outcomes with the test?

Precision (P) measures the number of positive class predictions that belong to the positive class.

Recall measures the number of positive class predictions made out of all positive examples in the data set.

F-measure offers a single score that balances both the concerns of precision and recall in one number.

Whereas the accuracy is the total number of correct predictions divided by the total number of predictions made for a data set.

9.4.4 Results and Analysis

For the final analysis, we computed the performance measures for all the five classification techniques used in the study. The results were based on the values of precision, recall, F-measure, and accuracy. Mentioned below are the tables and graphs of our study for four NASA system data sets (KC1, PC1, PC2, and KC3) that are selected from PROMISE repository [13] which is freely available as public data sets.

The observations from the tables and graphs are as follows:

• For better performance, TPR and TNR should be high, and FNR and FPR should be low. From Table 9.7, it is observed that NCNN classifier predicts TPR and TNR greater than all ML classifier (RF, DT, and NB) and DNN classifier in the data sets KC1, KC2, and PC2 except in PC1 TPR of NCNN is lower than NB classifier but greater than RF, DT, and DNN classifiers but TNR of NCNN in case of PC1 is greater than all other classifiers. It is also observed that NCNN predicts FNR and FPR well; it is lower than all the data sets except in KC3 and PC1 where FPR of DNN is lower than NCNN and FNR of NB is lower than NCNN. This observation predicts that NCNN model outperformed in confusion rate analysis for the all data sets. This shows that NCNN model is not in underfit or overfit.
• From Figure 9.5, it is found that NCNN predicts more faults than RF, DT, NB, and DNN (162 faults in case of KC1 data set, 33 faults in case of KC3 data set, and 14 faults in case of PC2 data set), but in case of PC1 data set, NCNN predicts more faults than DNN, RF, and DT but lesser than NB as NB predicts 23 faults and NCNN predicts 20 faults.
• Figure 9.6 represents model accuracy and model loss for the data sets (KC1, KC3, PC1, and PC2). Accuracy and loss are two well-known metrics in ML and neural networks. Splitting data set will use at every epoch and used Adam optimizer, as an optimization function in order to update the weight of the network. Binary cross-entropy used as the loss function, so that we can get better model performance. Accuracy is a method for measuring a classification model’s performance and naturally expressed as a percentage. It is the count of predictions where the predicted value is equal to the true value and a loss function taken into consideration the probabilities of a prediction based on how much the prediction varies from the true value. This gives us a clearer view how well the model is performing. The lower the loss the better a model to the training data. The loss is calculated on training and validation and its interpretation is how well the model is doing for the data sets. Unlike accuracy, loss is not a percentage. From the plot of model accuracy, we can see that the model could possibly be trained a little more as the trend for accuracy on KC1 data set, KC3 data set, and PC2 data set is rising for the middle and last few epochs except in PC1 data set where, in the middle and last epochs, it slightly moves up and down. From the plot of model loss, we can see that the model has comparable performance on both train and validation data sets (labeled test). If these equivalent plots start to depart constantly, then it might be a sign to stop training at an earlier epoch. For the data sets KC1, KC3, and PC2 model loss has comparable performance on both train and validation data sets except in PC1 data set where in the last few epochs it drastically increasing.

  

  

• To evaluate prediction accuracy F-measure (also called a F1-score), precision, and recall were used. Usually, there are adjustment between precision and recall. As for example, by predicting all the test files as defective, we will get a recall values as 1 and a very low precision. So, F-measure is the better prediction performance representation which is a compound of precision and recall and falls in the range [0, 1].
• Figure 9.7 shows the performance comparison in terms of F-measure, recall, and precision of different models for soft-ware defect prediction for the data sets (KC1, KC3, PC1, and PC2). From Figure 9.7, it is observed that F-measure of NCNN is 5% higher than DNN, 9.3% higher than NB, 7.4% higher than DT, and 6.5% higher than RT for the data set KC1. For the data set KC3, F-measure of NCNN is 1% higher than DNN, 10.9% higher than NB, 8.1% higher than DT, and 8.5% higher than RF. For the data set PC1, F-measure of NCNN is 1.8% higher than NB and equal to RF and DT but 1% less than DNN. The reason may be that the training set of PC1 is relatively small. Similarly, for the data set PC2 F-measure of NCNN is 1% higher than DNN, 3.9% higher than NB, 1% higher than DT, and 1% higher than RF. Almost in all the data sets, NCNN outperformed the other state-of-the-art models of ML and DNN in terms of F-measure for software defect prediction.
• From Tables 9.3 to 9.6 and Figure 9.8, we examined the accuracy value for the data sets KC1, KC3, PC1, and PC2. From analysis, we found that NCNN has higher accuracy for the data set KC1 (88.76), for the data set KC3 (96.97), and for the data set PC2 (98.71). For the data set PC1, NCNN has less accuracy (93.16) than DNN (94.65) and RF (93.68) but higher than DT and NB.

• From the above analysis, it reflects that the proposed NCNN model outperforms well for all four NASA system data sets from PROMISE repository [13] with other ML classifier and neural network classifier in terms of F-measure, recall, precision, and accuracy and through confusion matrix analysis and confusion rate analysis for the software defect prediction.