Abstract

Statistical model validation (SMV) evaluates the accuracy of a computational model’s predictions. In SMV, hypothesis testing is used to determine the validity or invalidity of a prediction, based on the value of a statistical validation metric that quantifies the difference between the predicted and observed results. Errors in hypothesis testing decisions are troublesome when evaluating the accuracy of a computational model, since an invalid model can be used in practical engineering design activities and incorrect results in these settings may lead to safety issues. The overall goal of this paper is to provide a recommendation for statistical validation metrics appropriate for accurate SMV. For this, this paper compares various statistical validation metrics to highlight those that show the less errors in hypothesis testing. The resulting work provides a statistical validation metric that is sensitive to a discrepancy in the mean or variance of the two distributions from predictions and observations. In particular, this study considers unknown parameters in a prediction model, which represent the primary source of invalidity, to understand the ability of statistical validation metrics to deal with Type II errors in hypothesis testing. Statistical validation metrics examined in this study include Kullback–Leibler divergence, area metric with U-pooling, Bayes factor, likelihood, probability of separation, and the probability residual. Overall, this paper provides a summary of feasible statistical validation metrics, along with related scientific discussion.