Credibility, Validity and Testing of Dynamic Simulation Models

Abstract

Also called ‘model validity’ testing, model credibility evaluation has always been a controversial issue in any modeling methodology. We briefly discuss why this important notion is so controversial. To this end, we classify major types of models, particularly as they impact the notion of model credibility: i- Purely statistical forecasting (black box) models, and ii- Causal-descriptive policy (transparent) models. We then focus on what makes causal-descriptive model credibility and evaluation unique and quite difficult, compared to short-term forecasting models. One important result is that causal-descriptive model credibility consists of two different aspects: structural and behavioral. In most simulation modeling (particularly system dynamics policy simulation), establishing structure credibility must strictly precede behavior credibility; the latter has no value without the former. We thus discuss Structural tests and output behavior tests for dynamic simulation models separately. Structure tests can further be classified into direct and indirect structure tests. We place special emphasis on indirect structure tests. We also provide a quick overview of recent model testing software developed in SESDYN Laboratory at Boğaziçi University. Finally we discuss model credibility in broader context and some implementation issues.