The role of validation in optimization models for forest management

Abstract

Key message

A validation convention can be established for forest management optimization models. It consists of (1) the delivery of face validation, (2) performing at least one other validation technique, and (3) an explicit discussion of how the optimization model fulfills the stated purpose. Validation by potential users or external experts is of high importance.

Context

Optimization modeling has long assisted the management of forest ecosystems, but the credibility of these models has always been debated with criticisms concerning data quality, failures to include relevant processes in the scope of models, and the inclusion of unrealistic assumptions. Validation is widely considered to be crucial to establishing the credibility of models in general, but how to validate optimization models in particular represents a permanent question generally in operations research.

Aims

We aim to synthesize practical recommendations for the development of validation frameworks in the optimization modeling for forest management.

Methods

We selected a sample of 46 studies devoted to optimization models to be applied in practice, analysed the contents with respect to validation, and provided a critical review.

Results

We (1) clarified the meaning and usage of different validation-related terms that are commonly encountered in the literature, (2) identified and categorised the various methods and frameworks that are used to demonstrate model credibility, and (3) derived organizing principles that helped to suggest improvements in validation frameworks.

Conclusions

A practical validation convention can be established and we suggest the convention to consist of three stages. By providing structured and consistent information about validation processes, researchers in forest management optimization can better demonstrate the credibility of their work to readers and potential users.

1 Introduction

Despite the high potential of the decision support systems to help inform environmental and natural resource management, these tools are seldom adopted by end users (McIntosh et al. 2011; Schuwirth et al. 2019). There is a broad agreement that the validation of environmental models is urgently needed to enhance their utility and applicability in managerial and policy-making practice (Rykiel 1996; Getz et al. 2018; Eker et al. 2018; Schuwirth et al. 2019). But there is still confusion about what validation means exactly to decision-makers and modelers, including what it should involve, and how it should be performed for specific types of problems (Kleindorfer et al. 1998; Huang et al. 2003; Eker et al. 2018; Janova et al. 2019). The question of how to ensure the credibility of modeling results has been a persistent problem since the early days of operations research (OR) when decision support systems began to be developed (Gass 1983; Goodall 1972; Caswell 1976). Validation has been defined in the literature as “the process by which scientists assure themselves and others that a theory or model is a description of the selected phenomena that is adequate for the uses to which it will be put” (Miser 1993; Landry et al. 1983; Barlas 1996). Positivism and relativism are often mentioned as two key philosophical directions influencing modern understandings of validation (Barlas and Carpenter 1990; Kleindorfer et al. 1998). The positivist validation viewpoint reflects the natural sciences practice; hence, it focuses on an accurate representation of reality and requires quantitative evidence demonstrating congruence between model predictions and empirical observation. The relativist validation approaches value model usefulness more than the representation of accuracy (Eker et al. 2018). According to this view, any given model is equally valid and invalid given that validity is a matter of opinion (Barlas and Carpenter 1990; Kleindorfer et al. 1998). Consequently, there is a need for a dialog between model developers and other stakeholders such that the validation process emerges as a “a matter of social conversation rather than objective confrontation” (Barlas and Carpenter 1990). In decision science, both philosophical viewpoints are accepted, and a number of approaches to validation combine elements from both schools (Eker et al. 2018). Some authors suggest that models can only be invalidated (Holling 1978; McCarl 1984), especially if political or social content is involved (Gass 1983). Such complex problems, when questioned deeply enough, can be doubted regarding the representation of the original real-world problem, i.e., on close examination, one can almost certainly question the sufficiency of the representativeness of the model with respect to the real system (Gass 1983). Overall the validity of decision support models is a longstanding unresolved question that has generated much discussion in the OR literature and beyond (Gass 1983; Miser 1993; Eker et al. 2018).

In the case of simulation models, something like a philosophy of validation has already been established. Kleijnen (1995), for example, conceptualized validation in terms of comparing simulated results against empirical data through statistical tests and procedures such as regression analysis. Optimization models are generally less concerned with what will happen than with what ought to happen, i.e., “Out of many possible courses of action, which one is the best?” Hence, getting the real-world data for validation is rarely straightforward, and often even impossible. Some optimization issues do, of course, deal with well-defined technical problems, including ones where solution-testing has a relatively low cost in terms of time and resources. Optimization models concerned with managing inventory or designing packing systems, for instance, might be able to demonstrate credibility by following a validation philosophy much like that used in simulation modeling: implement the recommended solution as a pilot experiment in a small portion of the system, and compare its performance with that of the existing business-as-usual process (Gass 1983). However, for complex natural resources optimization problems, this is usually not possible: in forestry, decision-making deals with timescales measured in decades or centuries, potentially across entire landscapes that cannot be replicated, and involves manipulating highly complex natural systems, under deeply uncertain sets of economic, social, and political assumptions. This type of problem is fundamentally-and often extremely-squishy. Practical guidance for the validation of optimization models is virtually absent in the literature and this type of validation is itself a persistent theoretical problem. The main reason for this is the impossibility of comparison with the “correct solution” and thus the impossibility of using commonly proposed data-driven procedures.

The environmental literature often interprets validation of models strictly in terms of the degree of agreement between the model and the real system (Goodall 1972; Rykiel 1996) and the most validation provided in recent environmental modeling literature is based on data-driven approaches (e.g. Eker et al. 2018) that are usually not applicable in the case of optimization models. Moreover, many ecological models do not provide any form of validation to ensure that the results are appropriate for the inferences made (Krausman 2020). Forest management is a fairly conservative field, where willingness to adopt new measures or tools is often quite limited (Brunette et al. 2020; Janová et al. 2022). This tendency might well be related to the squishiness of the problems it must navigate, which could conceivably favor sticking with tried-and-true heuristics. Developing a more effective and systematic process for validating new models and methods could help modelers overcome decision makers’ mistrust.

To help clarify the role of validation in the forest management decision support context and to contribute to an improved understanding of how to validate environmental optimization models in practice, this review examines a sample of optimization studies and aims to identify organizing principles that can help overcome the longstanding conceptual and practical challenges mentioned above. First, we aim to deliver an understanding of the different terms used, and to identify and categorize the various methods and frameworks used in the literature to demonstrate the reliability of optimization models. Second, we synthesize these results to assess the evidence for validation frameworks applicable to forest management optimization modeling and suggest characteristics of a validation convention for environmental optimization models.

2 Theory

In this section, we provide a terminological summary based on the literature review. We follow the established general procedure of developing the decision support system (DSS) (Sargent 1984; Gass 1983): 1. Real-world problem statement, 2. Conceptual model development, 3. Computerized model development, 4 Solving the computerized model, see Fig. 1. By “stakeholders” we mean model developers, scientists, and users of the models.

Fig. 1

Diagram representing the position of conceptual and operational validation, and computerized model verification

2.1 Verification

Verification refers to the process of demonstrating that the modeling formalism is correct (Rykiel 1996). Computerized model verification (see Fig. 1) reveals how well a digital implementation of a model represents the conceptual model upon which it was based (Gass 1983; Kleijnen 1995; Sargent 2013). In the literature, the term “verification” is often understood to refer to computerized model verification (Janova et al. 2019) only focusing on debugging a computerized model and/or on demonstrating the technical correctness and calibration of the mathematical model (Shugart 1984; Rykiel 1996).

In a broader context, the terms verification and validation have a certain overlap in the literature. Conceptual validity (see Fig. 1) means that the theories and assumptions underlying the conceptual model are justifiable, and that the mathematical logic of the model is reasonable for the model’s intended purpose (Rykiel 1996; Sargent 2013). In a similar context (Gass 1983) uses the term technical validity.

In our proposed approach, we consider computerized model verification, and conceptual/technical validity to be the key components of verification: that is, the process of evaluating “the formal appropriateness and correctness of the conceptual model logic” to ensure the “technical correctness of the subsequent computerized model” (Janova et al. 2019, p. 917). Defined in this way, verification and validation are not overlapping concepts, but discrete and complementary processes.

Verification can be narrowly understood as referring to computerized model verification only in the case that a standard, routinely used conceptual model with an established computerized model has been adopted for building the decision support model (Borgonovo et al. 2018; Hinder and Mason 2017; Janova et al. 2019). But complete verification (i.e. including both computerized and conceptual model validation) is a must for decision support models dealing with non-standard new problems requiring original conceptual model development (Gass 1983; Balci 1994; Sargent 2013; Robinson 2002).

2.2 Operational validation

Conceptual validity is an endorsement of the scientific content of the model, but it does not guarantee that the model will provide recommendations that are sufficiently accurate or precise for a specific use-case (Rykiel 1984). Operational validation, on the other hand, is concerned with how well the model fulfills the intended purpose of the decision support system within the domain of its applicability (Gass 1983; Landry et al. 1983; Sargent 1984), see Fig. 1. It is a pragmatic approach because it focuses on the performance of the model regardless of the mathematical inner structure of the model (Rykiel 1996). Therefore, verification (including conceptual validation) and operational validation must both be established to confirm overall model validity. Gass (1983) introduces the term model accreditation for this complete process (verification and validation). Operational validation is one of the most under-studied and least-discussed components of validation science. Assessing operational validation is a permanent open problem (Gass 1983).

2.3 Data validation

Data validation ensures that when building the model we are using appropriate data of sufficient standard (Gass 1993; Sargent 2013; Rykiel 1996). To assess data quality, control should be an inherent part of standard decision model development process, and should be routinely provided by model developers. The data must be adequate to estimate model parameters (Krausman 2020; Getz et al. 2018). Any model should consider the underlying sampling design of the data and any biases of data repositories used (Kramer-Schadt et al. 2013). This step is often neglected in practice when introducing new models (Krausman 2020).

2.4 Credibility

We say that the decision support model is credible if the decision maker has sufficient confidence to use it in real situations (Sargent 1984; Holling 1978). According to Rykiel (1996), credibility is established when there is “a sufficient degree of belief in the validity of a model to justify its use for research and decision making”. Rykiel (1996) stresses that credibility is a subjective, qualitative judgment that cannot be approached in quantitative sense. Therefore, the decision support model may be credible only for the particular user or group of users, but not necessarily for other stakeholders. Smith (1993) developed a validation approach based on collecting, and then thoroughly discussing, arguments for and against the credibility of a given model and its results. The model is considered valid if the pros outweigh the cons.

2.5 Accreditation, evaludation

The terms evaludation (Augusiak et al. 2014) and accreditation (Gass 1983) have been introduced to describe the complex processes of model quality control and credibility establishment. These terms are intended to cover the complete structure of computerized model verification, conceptual and operational model validation and data validation, but have not yet been routinely adopted in the scientific literature. Rather, the term “validation” itself is often-albeit inconsistently-used in this broad sense.

2.6 Validation procedures

Some previous work has sought to establish some form of quantifiable certification, which would serve the decision makers as a user-friendly tool for assessing model credibility. Gass (1993), for instance, suggest assigning a numerical score to each component of model development, verification, and validation, then using the weighted composite score to guide model accreditation decisions. However, the authors themselves raise doubts about the general applicability of such an approach due the difficulty in defining a priori standardized criteria for classes of models with common features. Currently, the general view is that attempting to define a standardized quantitative criterion for model evaluation (e.g. a statistical test or index), is either infeasible or inappropriate for decision support models (Augusiak et al. 2014) given the sheer diversity of structure, scope, and application.

Rykiel (1996) formulate a theoretical concept as a procedure demonstrating that model outputs meet the performance standards required for the model’s purpose. These standards can be customized to the model developer’s specifications and can take various forms depending on characteristics of a given decision support model. Another option involves establishing a validation convention for the particular class of problems that would determine the performance standards. In this case, as a prerequisite for initiating the validation process it would first be necessary to specify : (1) the purpose of the model, (2) the convention criteria the model must meet to be acceptable for use, and (3) the context in which the model is intended to operate.

Here we list some of the most common validation procedures mentioned in the literature.

• 1. Face validation provides initial information on a model’s realism. It consists of presenting a model’s recommendation (often using graphical visualization of partial solutions) and asking experts familiar with the real-world problem to use their professional judgment to evaluate whether those recommendations are plausible given their knowledge of the system in question (Halachmi et al. 2001; Balci 1994; Oral and Kettani 1993; Zadnik Stirn 1990; Sargent 2013).

• 2. Cross-validation uses part of the existing real data to guide the development of the conceptual model, while keeping a subset of that data separate so that it can be used to test the model’s predictive power (Landry et al. 1983; Richardson 1978; Sargent 2013). Cross validation is often mentioned as an important device for validating ecological models (Krausman 2020; Rykiel 1996). This approach is, however, particularly relevant for predictive models. Cross validation is not generally appropriate when developing optimization models, which are often deployed in scenarios when real-world data is not available (e.g. due to the counterfactual baselines that are typically involved when performing prescriptive, rather than predictive, modeling).

• 3. Comparisons to other models interprets validation in terms of consistency with the results produced by other comparable (but ideally established and well-validated) models (Landry et al. 1983; Sargent 2013). In the ecological literature, this form of validation is arguably the principal means by which new models are assessed (Cess et al. 1990; Rykiel 1996; Janova and Hampel 2016).

• 4. Experimental validation, also known as proof of concept, involves designing a smaller or more controlled implementation to explore the model’s performance and to demonstrate the feasibility, viability, and applicability of a new concept. For instance, the recommendations produced by a decision support model might (in theory) be tested in a working environment (Gass 1983). This approach is applied in a number of different industries. However, it is often not possible in the forest management context, mostly due to time and resource constraints. In simulation modeling, validation might be delivered by developing a small-scale experiment of a real situation and comparing the observed data to the results of simulation (Rykiel 1996).

• 5. Expert validation compares the model’s results to the predictions or recommendations of experts (Landry et al. 1983).

• 6. Sensitivity analysis The parameters with the largest potential to generate major changes in a given model’s behavior should be estimated with the highest accuracy (Rykiel 1996). Sensitivity analysis quantifies how models respond to changes in their parameters (Landry et al. 1983; Richardson 1978; Sargent 2013). Comparing these responses to the expected behavior of the real system can reveal disparities between the parameters to which the real system is sensitive and those to which the model is sensitive (Rykiel 1996). Sensitivity analysis is considered an important instrument to validate the models (Razavi et al. 2021).

• 7. Extreme-condition testing is designed to determine whether or not a model’s behavior outside normal operating conditions is bounded reasonably (Rykiel 1996; Sargent 1984). When the context of a model’s applicability is well defined, extreme conditions may be out of its domain. In this case, the model may be considered valid even if it produces unreasonable results in edge cases or under extreme conditions.

• 8. Model development in cooperation with the user can considerably improve the practical utility of the final product (Wisdom et al. 2020). When the user not only defines the problem, but also engages with the model development process by, e.g., defining the weights of different objectives, specifying key constraints, and discussing underlying assumptions, then such user-developer cooperation may be considered a type of validation. Because this process can greatly improve users’ understanding of (and ability to critically interpret) model results, it can greatly increase the likelihood of the model being adopted into use and applied correctly in practice. Bringing public into the process of developing forest management policies may provide also considerable improvements for credibility of the models’ solutions and can facilitate the acceptance and implementation of the complex decision support models (Zadnik Stirn 2006).

3 Materials and methods

3.1 Paper selection

We queried the Scopus database for peer-reviewed research papers dealing with optimization models developed for forest management decision making. Searching the title, keywords, and abstract of English-language articles published from 1990-2021 in the areas of Environmental Sciences, Agricultural and Biological Sciences, Social Sciences, and Mathematics and Economics for the terms (“optimization model” or “mathematical programming”) and “forest*” and (“decision*” or “management”), we obtained 336 items. We repeated the search using the terms “forest management”, “optimization”, “validation” or “verification”, and obtained 22 more papers, which we included for an initial sample of 358 items. We manually screened all 358 titles and abstracts to evaluate their relevance to the topic. We selected only papers that were clearly designed with the main objective of developing mathematical optimization models. Particularly, papers with missing quantitative models and primarily theoretical studies on new optimization techniques and method improvements (e.g. novel heuristic algorithms) that did not mention the ambition of applications to particular real-world problems were excluded, as were studies dedicated purely to simulation modeling (i.e. without a decision-making component). Further, to keep the problems homogeneous with respect to long-term planning we considered only forest planning problems on stand-, forest- and landscape-levels including road planning problems. Therefore we excluded tree-level optimizations and timber processing optimizations. The resulting number of papers was 46 (see Fig. 2). The selected papers are dealing with various types of problems: mostly harvest scheduling and rotation, followed by other topics encompassing reforestation, afforestation or species composition, land use planning, and also forest roads planning (for complete overview see Appendix).

Fig. 2

Overview of the paper selection process. Dashed box: eliminated items

3.2 Methods

This review is designed to identify potential organizing principles to support the development of more systematic validation processes for the optimization models in forest management. It is designed to respond, for example, to Bettinger et al. (2009) call for an agreement on validation standards between authors and users/reviewers, and Rykiel (1996) suggestion that disagreements over the meaning of validation can only be resolved by establishing a clear convention. Before starting a validation process, three issues in particular must be clear: (1) the purpose of the model, (2) the context in which the model should be applied, and (3) the performance criteria of the validation concept (Rykiel 1996). Therefore, within each paper, we identify:

• 1. The intended purpose of developing the model. Typically, the reason for developing the model is declared in the aim of the paper at the end of the introductory section. In each paper, we found the formulation of the aim and listed the stated intended purpose for the particular model development.

• 2. The context in which the model may be used. The context of model usage answers the questions

  • Q1 Is there a real-world proof-of-concept?

  • Q2 What are the limitations?

  • Q3 How should the model results be understood and used in practice?

  • Q4 Can the model be applied to problems other than the one for which it was originally developed?

  In our approach, if all four questions are answered in the paper, then we say that the context of model usage is explicitly mentioned. These papers clearly state specific circumstances under which the model may be applied and present and discuss assumptions and shortcomings in detail. If only some of the questions are answered, or if all are answered but not adequately, then we say that the context of model usage is partially mentioned. We say that the context of model usage is not mentioned if the questions Q2,Q3,Q4 are not answered at all.

• 3. The validation evidence or discussion. Because there is no particular place assigned to communicate this information in standard journal formats, we collected it through a careful reading of the full text of each article in our sample. The validation evidence or discussion refers to the sections of the paper that are designed to report the performance of the model with respect to its intended purpose and/or deliver credibility to the model. As the term “validation” is seldom used in the literature, we search for any procedures and discussions in the paper that are aimed at these activities. Based on the terminology formulated in the preceding section, we classify the credibility-delivering procedures in terms of the four categories: (1) computerized model verification, (2) conceptual model validation, (3) data validation, (4) operational model validation. We focus specifically on conceptual model validation and operational model validation, for which we note down the particular validation procedures used. Note that computerized model verification has become a standard part of model development that is often not explicitly mentioned by authors Janova (2012); Janova et al. (2019). Similarly, although data validation should be an inherent part of the model development, this step is not necessarily performed when presenting the model (Krausman 2020).

4 Results and discussion

Of the 46 papers in our sample, 24, 17 and 5 papers dealt with optimization problems on the forest, stand and landscape levels, respectively. The conceptual models were based on linear programming in 10, on another mathematical programming technique (mixed integer programming, nonlinear programming, multiple criteria optimization or stochastic programming) in 19, on dynamic optimization in 9, on heuristics in 6, and on other methods in 2 papers. A detailed list of the papers with findings on the particular model purpose, problem type and modelling type can be found in Appendix Table 4.

4.1 Model purpose

All papers in our sample stated the purpose for which the model was developed. This information was typically placed prominently in the abstract and/or aims and motivation in the introduction. Typically, the purpose was to develop a particular mathematical model which enables the representation of the nature of the decision problem. The purpose of each model development is listed in Appendix Table 4.

Among the common operations research purposes focusing on solving particular decision problems in forest management, we identified two specific groups of purposes:

• 1. Model refinement purpose (8 papers)— to improve or adjust the model in order to achieve greater accuracy or fidelity, typically to address particularly complex aspects of real decision problems through optimization. These include incorporating climate change uncertainty into harvest scheduling (Garcia-Gonzalo et al. 2016), presenting a model for considering risk from storm damage (López-Andújar Fustel et al. 2021), finding a harvest schedule that addresses conflicting objectives (Álvarez-Miranda et al. 2018), incorporating carbon sequestration as complementary objective (Díaz-Balteiro and Romero 2003), and also (Amrouss et al. 2017; Arias-Rodil et al. 2017; Diaz-Balteiro et al. 2009; Flisberg et al. 2021).

• 2. Scenario development purpose (9 papers)— to deliver by optimization scenarios additional viewpoints for a certain situations and in this way to support managerial decisions, i.e., to enable the decision maker to have not only one but a pool of policies (Álvarez-Miranda et al. 2019); to help managers better understand the effects of different options on ecosystem services (Bagdon et al. 2016); and to investigate the role of different agroforestry systems in hypothetical farm portfolios (Gosling et al. 2020). Coulter et al. (2006) solved two separate optimization problems with different objectives to provide the decision maker the freedom of choice of optimal solution. Other papers written with a scenario development purpose include (Borges et al. 2010; Botequim et al. 2021; Gutierréz et al. 2006; Nghiem and Tran 2016; Raymer et al. 2009). For these purposes, the aim of developing the model was not to provide an unambiguous optimal solution to the problem to be implemented in practice, but rather to provide optimal scenarios for varying parameters and/or objectives and/or assumptions and expand the information base available to the decision-maker.

These purposes are interesting in part because they represent certain evolution in the operations research problem statement of solving a practical decision problem. Particularly, studies whose purpose is model refinement often involve highly complex target problems and the gradual work of bringing them to represent reality as faithfully as possible. Studies with scenario development purposes, in turn, outline an interesting new direction for the use of optimization models: their creators used these to simulate scenarios of what optimal decision-making should look like in different situations and what the implications of this decision-making would be for the controlled systems. This might be called “optimization-sensitivity” analysis, which may have a higher added value for the decision maker than finding a single optimal solution to the decision problem. Recently, a simulation-optimization technique has been used to predict real deforestation (Knoke et al. 2023), which may be another line of optimization model purpose development.

A model’s complexity depends on its purpose (Getz et al. 2018) and is closely linked to its usefulness in decision support (Tolk et al. 2022). The papers in our sample typically described models with the purpose of increasing the coverage of the problems’ complexity, which tends to produce models which are, correspondingly, more mathematically complex. But as discussed in Williams (2011), even simple models with low fidelity may have considerable strategic value when comparing alternative management options, or for formulating policy when implemented in adaptive management. From the model-user viewpoint, the discussion and particular presentation of comparisons of low-fidelity models with those with high fidelity could be of importance for the future evolution of operations research in the natural resource context.

4.2 Model context

We identified 8 papers (Chung et al. 2008; Church et al. 2000; Liu et al. 2017; Messerer et al. 2020; Nhantumbo et al. 2001; Raymer et al. 2009; Rytwinski and Crowe 2010; Wei and Murray 2015) mentioning the context of the model usage explicitly by discussing the model functioning, including the strengths and weaknesses of its application in practice, and by providing an understanding of the results in a real-world context. In 6 papers the model context was not mentioned at all. In the remaining 32 papers, the context was partially mentioned in various ways and with varying degrees of detail (see typical examples in Table 1 ).

Table 1 Examples of “partially mentioned” model context

Explaining the context of model usage is a necessary condition for model implementation. However, as there is no universal standard for elaborating on the context, researchers deal with this point by applying a variety of standards. provided a complete explanation of context of model usage. We find that incomplete information decreases the model understanding which results in possibly lower credibility of the model for the potential user.

Since models are custom-made for specific situations, localities, and/or assumptions, their application to other problems is often complex. In most cases, to explain the context of model usage, authors only mentioned the conditions for which the model was created and the limitations that result from its possible application to the real problem at hand. It would be beneficial for the authors to comment on the generalizability- and/or universality- aspect in their papers to aid readers in understanding the model’s potential for broader applicability. Note that the generalizability of some models can also be found at another level, where the models’ results contribute theoretically to the general understanding of climate change impacts on optimal decision-making or policymaking, for example.

4.3 Validation

All the papers we analyzed were carefully developed from a technical standpoint. The authors provided detailed descriptions of model design, and the results were thoroughly presented and discussed, with conclusions drawn regarding their applicability in forest management. However, an explicitly formulated validation framework for model development was mostly absent from the papers, and a variety of methods were used to establish credibility on different levels of detail.

Conceptual model validation should address why particular logic and theories were used when building the mathematical model, and authors should provide evidence that these are reasonable for the model’s intended purpose. While the studies in our sample typically do describe and justify underlying assumptions as described above, the choice of a particular mathematical model is not always supported by any arguments. Thirteen papers neglected to mention why a particular optimization approach was selected over other established alternative approaches. Additionally the appropriateness of the model developed is sometimes not discussed in case the authors use computational tool previously developed for solving the given problem (Keleş 2010; Rytwinski and Crowe 2010). In the remainder of the sample, conceptual validation is mostly provided by noting (typically in the introduction, literature review, or methods sections) information summarized in Table 2.

Table 2 Summary of how authors deliver conceptual model validation

Generally, researchers build on the long history of the application of operational research methods in natural resources management, referring to earlier research and current standards for use of the methods they employ to address a given problem. The use of some optimization methods has become so standardized in forest management decision support science that conceptual model validation may not be performed at all or is typically delivered by mentioning that the model is commonly used in similar problems. However, in more complex problems, there are typically multiple mathematical approaches that can be used to address the issue at hand. To improve conceptual model validation, authors could discuss the various methods available and present the pros and cons of their chosen approach to a given problem. Such a discussion would be beneficial to readers and potential model users as it would provide a deeper understanding of the model’s underlying philosophy.

From validation-related-expressions text analysis of all papers (see Table 3) it follows that terms connected with credibility, usability, applicability, verification or validation have seldom been used in the papers. More often, terms connected to efficiency or model evaluation were discussed. But these terms dealt with validation topic only in several cases.

Table 3 Overall frequency of use of validation-related terms in the text of articles

Nevertheless, we did identify multiple attempts to evaluate model performance, although the authors used varying terminology (see Figs. 3 and 4). Except for the seven papers where we identified no attempt for operational validation, all the others performed some form of face analysis. That is, in most papers we found a combination of face validation with another (more sophisticated) operational validation approach(es). Operational validation was typically not provided in research that primarily involved adapting and applying an established model to new conditions.

Fig. 3

Distribution of operational validation techniques in our sample: the number of papers in which a specific single technique is used for operational validation (regardless of whether only one or more techniques are used). 7 articles did not perform any operational validation at all

Fig. 4

Distribution of face validation and face validation in combinations with other operational validation technique(s) in our sample : the number of papers in which face validation alone or face validation in combination with other technique(s) are used for operational validation. 7 articles did not perform any operational validation at all

For model refinement studies, the intended purpose would be fulfilled even by model development itself. However, in our review, these models were supported by operational validation techniques with no difference from other models. Generally, the operational validation attempts we identified aimed to ensure the functioning of the model and did not particularly connect this performance to the intended purpose of the model. This can be considered the main shortcoming of the presented attempts, since an explicit evaluation of the fulfillment of the original purpose of the model should be a main subject of the validation.

The collection of approaches based on plausibility checks can be considered a form of face validation. A common way to demonstrate model credibility in the reviewed papers is to apply it to a real-world decision problem as (e.g. in Bagdon et al. 2009; Diaz-Balteiro et al. 2016; Keleş 2010). If reasonable results are obtained, then this is taken as evidence of the model’s effectiveness (Wei and Murray 2015) or potential (Álvarez-Miranda et al. 2018; Botequim et al. 2021), and capability (Álvarez-Miranda et al. 2019) or feasibility (Nhantumbo et al. 2001). For instance, Gutierréz et al. (2006) use a case study “to test and validate” their model, while (Moreira et al. 2013) include a case study specifically to “test their model’s mathematical formulation”. Similarly, López-Andújar Fustel et al. (2021) use a case study “to validate the basic functioning of their optimization model and to determine whether the model was able to fulfill its intended purpose”. Díaz-Balteiro and Romero (2003), in turn, discuss the practical applicability of modeled solutions and argue that their modeling framework should be considered sound because (i) it does not make identifiable computational errors, (ii) it allows users not only to obtain a solution, but also aids in understanding trade-offs between conflicting criteria.

Often the attempts for face validation were limited to the statement that the model was validated by the application and/or that it performs well. True face validation should be provided by external experts or professionals; providing face validation only in the form of an author’s assertion, without any further discussion on performance criteria, does not seem to us a sufficient standard. This type of discussion does not provide much additional information to the reader or potential model-user and as such, it is not a means of improving the credibility of the model.

Sensitivity analysis is another common way that researchers attempt to demonstrate the performance and credibility of their models, e.g., Najafi and Richards (2013). Maness and Farrell (2004) analyze cross-sensitivity between the indicators in their model, discuss its characteristics and limitations in detail, and clearly show what conclusions forest companies can draw from its results. The purpose of including this kind of detailed, quantitative analysis is ostensibly to demonstrate model credibility to potential users. In addition to performing cross-model comparisons, Matthies and Valsta (2016) stresses the importance of evaluating the sensitivity of model results and management recommendations to the input values.

Several papers do not attempt to demonstrate the effectiveness, capabilities, or credibility of their models beyond performing sensitivity analysis (e.g., Nghiem and Tran 2005; Quintero-Méndez and Jerez-Rico 2019; Wam et al. 2016).

Collaborative co-development processes with users and/or experts represent another approach to operational validation. For Flisberg et al. (2021), validation is a primary objective, and achieving it required concrete measures for which the authors obtained specific financial support. The validation process was carried out with the help of experienced forest company staff, who validated the model’s usability and implemented its results. Notably, criteria weights were informed by expert feedback. Similarly, Kašpar et al. (2015) used a survey of forest professionals to set criteria weights in their multi-objective model. Gosling et al. (2020) view model credibility as contingent on the validity of background information obtained through a series of stakeholder consultations. By performing in-depth interviews with farmers, the researchers aim to access not only empirical expertise but also incorporate subjective preferences into decision making. Although (Borges et al. 1997) do not explicitly mention terms related to validation, they do describe a model co-development process that integrated input from decision-makers, which they consider to be a prerequisite for a successful use of mathematical programming in forest management. Similarly, Serrano-Ramírez et al. (2021) incorporate owners’ judgments and preferences into the process of formulating a model that is designed to generate a range of alternative management scenarios for forest owners, enabling them to make an informed judgment about which scenario is most attractive.

Many authors established the credibility of their model primarily by presenting a comparison with other models (e.g., in Chung et al. 2016; Matthies and Valsta 1992; Parkatti et al. 2019; Valsta 2008). Amrouss et al. (2017) explicitly mention their intention to validate the model and performs validation via comparison to a complete information decision model, which was tested under disrupting scenarios. Liu et al. (2017) suggest a mathematical programming model with a heuristics solution technique and provides thorough comparisons. In discussing their models, Borges et al. (2010); Schroder et al. (2016); Sacchelli and Bernetti (2019) mention the consistency of their modeling outputs with past research. Garcia-Gonzalo et al. (2016) compare the performance of their stochastic optimization model to that of a parallel deterministic model, citing the superiority of the stochastic model’s performance as a “demonstration of the value” of the new model. Tahvonen and Rämö (2016); Tahvonen et al. (2010) compare their results against those produced by earlier versions of the model, and contextualize these comparisons with a detailed theoretical discussion. Similarly, Hennigar and MacLean (2010) compares spruce budworm outbreaks scenarios to scenarios suggested by other authors to prove the credibility of underlying structures used in the optimization. Valle-Carrión et al. (2021) compares the results of new optimization model to two others: the first, official, developed by other authors and another developed together with the new model to mimic the current management practices. Messerer et al. (2020) compared the model results with those well-established and provided justifications for unexpected findings regarding the role of the “mild” form of uncertainty in forest management optimization modelling. Rytwinski and Crowe (2010) evaluated model’s solutions by comparing them to those found using a heuristic which mimics a conventional approach. The term “validation" is not used elsewhere; however, the authors mentioned in the conclusion that the model was validated.

Some researchers acknowledge the importance of validation but lack the resources or capabilities to carry out a validation process themselves, choosing instead to postpone the validation to future research. Solberg and Haight (1991) felt that they could not effectively validate their model due to a lack of empirical data, not least because they viewed their normative results as beyond the scope of contemporary empirical knowledge. At the same time they performed a sensitivity analysis, however they did not consider it as a validation of the model. Thus, they opted to postpone validation, suggesting that future research could compare their modeling results with data from either existing research plots or to-be-established plots designed specifically to test some of their modeling assumptions about stand-level responses to thinning. Hence, validation is sometimes postponed and left to future research when more empirical information will be available from the plots. However, although realistic, such an approach do not support the model with sufficient credibility. Therefore, the authors should seek to provide immediate operational validation in papers that present original models.

In our study, 7 papers did not provide any operational validation attempt at all. When a model is published in refereed journals, the reviewers may (in the opinion of the readers and the authors) play the role of experts who evaluate the model and thereby provide something like face validation, i.e. by recommending that a manuscript should be accepted, reviewers communicate that they have the impression that the model and its results are reasonable. This might mean that once published, the results are generally considered valid in the scientific and model-user community. In our opinion, the review process can be considered as a true face validation; however, this face validation is not sufficient to fulfill the essence of full operational validation provided by model developer. Hence, for the model development to be complete and credible, the validation part is essential in the original paper, irrespective of the quality or stringency of journal review processes.

Most of the papers we reviewed performed some kind of face validation, and in many cases this was accompanied by other (more sophisticated) techniques with varying levels of rigor and detail. In our sample, operational validation is most commonly performed by way of comparisons with other models. This approach is not only approachable for researchers but also provides a good understanding of the model’s performance and functioning. Bettinger et al. (2009) confirms the high value of validation by comparison with other models. Their research on the use of validation when developing heuristic approaches for supporting forest planning identified six different levels of validation, ranging from none to comparison with the mathematical programming solution of the problem, with the latter being the most valuable. Combining multiple validation approaches seems to be a more promising way of improving model understanding and demonstrating credibility. Combinations methods (e.g. sensitivity analysis, face validation, and/or comparisons with other models) can help build readers’ understanding of a model’s philosophy and functioning. In accordance with suggestions made in Wisdom et al. (2020), we also found that developing and/or evaluating models with experts or end-users can be highly valuable. In this way, the model keeps in close contact with the real problem, increasing its legitimacy and transparency (Linkevicius et al. 2019); furthermore, cooperation with the user might be considered a prerequisite for successful model usage in forest management (Borges et al. 1997). Validation by experiment was not used at all in the papers we reviewed. However, using simulation experiments to evaluate the functionality and rationale of the optimization model could be a promising contribution to model validation procedures (Janova et al. 2019).

Fig. 5

Diagram outlining a potential validation convention for optimization models

4.4 Towards validation convention in optimization modelling

The evidence we reviewed indicates that that most authors of forest management optimization studies provide some form of operational validation for their models. However, conceptual model validation appears to be obsolete, as optimization techniques are now well-established and there are often standardized tools available for given decision problems. Nevertheless, more attention should be paid to describing the context in which the model may be used, including the particular circumstances of the application and any model limitations, assumptions, and overall applicability. Providing such information would enhance the understanding and credibility of the model, as well as help potential users assess whether it is appropriate for their particular needs.

Attempts at operational validation differ in scope and detail, but the existing evidence suggests that validating optimization models in forest management is feasible. The complex validation convention visualized in Fig. 5 offers an approach to establishing model credibility and ensuring congruence with its intended purpose:

• 1. Delivery of face validation at least by the authors but preferably by external experts.

• 2. Performing at least one other validation technique as a mandatory part of validation, most approachable techniques for the model developer are sensitivity analysis and comparisons with other models, involvement of the external expert is beneficial at this stage.

• 3. Careful, critical, and explicit discussion of how the optimization model fulfills its stated purpose, involvement of the end user for external evaluation of the level of fullfilling th purpose of the model is highly recommended.

Hence, it is possible to establish a more standardized validation convention based on combining commonly-used techniques in a mutually-reinforcing way. For this to be achievable in practice, it is necessary for model developers to combine validation techniques for which they have data and sufficient supporting evidence; at the same time, these techniques must provide the user with meaningful insight into, and understanding of, the model itself. Our review suggests that viable techniques include face validation, sensitivity analysis, comparisons with other models, and collaborative co-development with users and/or external experts with a special status and importance. A modeller’s own validation may be useful, but it comes with a high potential for bias in favor of one’s own model (Huang et al. 2003). Third party validation is thus very important component of model validation (Gass 1977; Huang et al. 2003). This is represented in the papers we reviewed by development the model with the user or by expert validation. Although this validation type has only been done in a few isolated cases, it has a large potential to convincingly demonstrate model credibility to academic readers and end users alike.

Face validation, in particular, is a direct, relatively accessible, and commonly-used technique for verifying model functioning, and should be included in any application. To ensure a more robust validation procedure, at least one other technique should also be employed. Sensitivity analysis and comparisons to other models have been found to be the most approachable for authors in this context. Moreover, comparisons to the related models is considered theoretically the principal means of model evaluation (Cess et al. 1990). This is especially true for optimization models, for which it is usually not possible to use e.g. cross-validation. To fulfill the theoretical definition of validation, the authors should also carefully, critically, and explicitly discuss the model results and overall performance with reference to the model’s intended purpose. An integral part of the formulation of the model purpose must be a precise determination of the context of model usage, which was rather neglected in most of the papers we analyzed.

If validation were to become a common part of the presentation of optimization models in forest management, we suggest that authors include a validation section after the results section, enabling readers to quickly and easily access important information about the model’s validation process.

5 Conclusion

This review aimed to evaluate whether it might be feasible to design a standardized validation convention for forest management optimization modelling. Based on our analysis of forest management optimization models and comparisons with historical and current literature, we conclude that the answer is yes. We outline a validation convention consisting of (1) the delivery of face validation at least by the authors (but preferably by external experts), (2) performing at least one other validation technique as a mandatory part of validation, and (3) a careful, critical, and explicit discussion of how the optimization model fulfills its stated purpose. Cooperating with with users or professionals during the model development and/or evaluation processes may be particularly beneficial for establishing model credibility and facilitating adoption into practice. By providing structured and consistent information about validation processes, researchers in forest management optimization (and other OR applications where cross-validation is typically not feasible) can better demonstrate the credibility of their work to readers and potential users. The validation convention was derived based on the experience with optimization models for stand- forest- and landscape-level problems, however, it offers a good starting point for establishing the validation concept for optimization models in environmental modeling in general.

We found that optimization models in forest management serve both to solve specific problems, but also as tools designed to support scenario analysis by providing decision-makers with a comprehensive picture of their problem space. This novel aspect of optimization modeling in forest management planning and policymaking probably merits more in-depth research.

Finally, we suggest that presenting and discussing comparisons of simpler, low-fidelity models with their newer, high-fidelity counterparts could be important for the future development of natural resources operations research science, because relatively simple models may have considerable strategic value for decision-makers comparing alternative management options or as a support for formulating policies.

Appendix

Table 4 The papers included in the review: Purpose of the model, the forest management decision problem type and level, the mathematical type of the model (LP stands for linear programming)