A surrogate-assisted Bayesian framework for uncertainty-aware validation benchmarks

Abstract

Over the last century, computational modeling in geoscience, especially in porous media research, has witnessed tremendous improvement. After decades of development, the state-of-the-art simulators can now solve coupled partial differential equations governing the complex subsurface multiphase flow system within a practically large spatial and temporal domain. Given the importance of computational modeling, quality assessment of these models in light of the purpose of a given simulation is of paramount importance to engineering designers and managers, public officials, and those affected by the decisions based on the predictions. Users and developers of computational simulations deal with a challenging question: How should confidence in modeling and simulation be critically assessed? Validation is one of the primary methods for building and quantifying confidence in modeling and simulation. It investigates the degree to which a model accurately represents reality from the perspective of the intended application of the model. Usually, this comparison between model outputs and experimental data constitutes plotting the model results against data on the same axes to provide a visual assessment of agreement or lack thereof. While comparisons between model and data are at the heart of any validation procedure, there are several concerns with such naive comparisons. First, these comparisons tend to provide qualitative rather than quantitative assessments and are clearly insufficient as a basis for making decisions regarding model validity. Second, naive comparisons often disregard or only partly account for existing uncertainties in the experimental observations or the model input parameters. Third, such comparisons can not reveal whether the model is appropriate for the intended purposes, as they mainly focus on the agreement in the observable quantities. These pitfalls give rise to the need for an uncertainty-aware framework that includes a validation metric. This metric shall provide a measure for comparison of the system response quantities of an experiment with the ones from a computational model while accounting for uncertainties in both in a rigorous way. To address this need, we developed a statistical framework incorporating a probabilistic modeling technique using a fully Bayesian approach. The dissertation aims to help modelers perform uncertainty aware model validation benchmarks. A two-stage Bayesian multi-model framework is discussed for modeling tasks where a set of models are at hand. To make this framework applicable for computationally demanding models, it is extended to a surrogate-assisted framework, keeping the computational costs at a reasonable level. Moreover, correction factors were introduced to compensate for the surrogate error in the Bayesian hypothesis testing and Bayesian model selection, as using surrogate representations instead of the full-fidelity computational models introduces additional errors to the validation metrics. In this dissertation, I show how the Bayesian formalism could be materialized by employing the concept of polynomial chaos expansion to achieve more accurate surrogates with a sparse representation and account for the uncertainty in the surrogate’s predictions. I also highlight how such surrogate models could be constructed with as few simulations as the computational budget allows. To this end, sequential adaptive sampling strategies are discussed, in which one attempts to augment the initial design iteratively. By doing so, informative regions in the parameter space are adequately explored. These regions are more likely to provide valuable information on the behavior of the original model responses. Using a sequential sampling strategy avoids the waste of computational resources, as opposed to the so-called one-shot designs. A series of benchmark studies are conducted to investigate the predictive capabilities of different sparsity and sequential adaptive sampling methods. Moreover, I introduce BayesValidRox, an open-source, object-oriented Python package that provides an automated workflow for surrogate-based sensitivity analysis, Bayesian calibration, and validation of computational models with a modular structure. The uncertainty-aware validation framework was applied to a range of cases in the field of subsurface hydro-system modeling, mainly to flow and transport in porous media, such as flow simulation models in fractured porous media, coupling free flow and porous medium flow, and microbially induced calcite precipitation. However, this validation framework can be transferred to other disciplines in which models are used for prediction.