How Threshold-Moving Technique May Change the Performance of Different Machine Learning Models in Crash Severity Prediction Problems

To predict crash severity using Machine Learning (ML) models, dealing with imbalanced classification problems could be inevitable. Threshold-moving can address such problems. Based on a review of the literature, this technique seems to be underutilized. Also, the issue of comparing the performance of different machine learning models in the prediction of crash severity seems to be an open one. Thus, this research focuses on comparing the performance of Random Forest (RF), Logistic Regression (LR) and Naïve Bayes (NB) models by analyzing the trade-off between accuracy and recall for the minority class (both measures change as a result of thresholding). The minority class in our problem is fatal and serious injuries crashes. We use a state-wide crash database from California which contains ۱۴۳۳۱۰ records in order to address this issue. Various thresholds are used in the comparison, which are determined by Receiver Operating Characteristic Curves (ROC) and Precision-Recall Curves. There are three thresholds chosen for this study: ۰.۰۵, ۰.۱۰, and ۰.۱۵. Based on the results, the LR with a threshold of ۰.۱, the RF with ۲۵۰ trees and the Bernoulli Naive Bayes (BNB) with a threshold of ۰.۰۵ are the best models. In addition, LR outperforms the rest of these three models. After threshold moving is employed, even simple models such as the LR can outperform more complicated ones like the RF in this paper, contradicting several previous studies in which the RF is found to be the best model.

کلیدواژه ها:

Crash Severity Prediction ، Threshold-moving Technique ، Random forest ، logit ، Naïve Bayes Models