A Normative Methodology for Research & Development Alignment and Decision-Making in the Arctic

: ERDC’s current portfolio development process introduces ill-defined and low technology readiness level (TRL) concepts before processing them through the USACE Project Portfolio Management (PFM) System to determine the list of projects for lab resourcing. This research will present a scalable decision model before the PFM, to ensure that projects are aligned with higher level strategies and priorities. This would minimize wasted time and resources by the research community before Research and Development (R&D) projects are approved using the PFM. The multi-attribute decision aid coupled with a Kendall Tau analysis provides mechanisms to ensure alignment. The goal is to eliminate ideas that do not fall under the strategic objectives set by the White House and the Department of Defense and make sure all projects are aligned with these priorities. The model illustrates how the PFM misaligns with the more normative design created for ill-defined projects.


Introduction 1.1 Background
The ERDC director published his "Top 10 USACE R&D Priorities" in 2021 (USACE, 2021).Two of the priorities listed were "Revolutionize and Accelerate Decision-Making" and "Protect and Defend the Arctic."An opportunity to address both priorities surfaced following discussion with USACE leaders.The Cold Regions Research and Engineering Laboratory (CRREL) focuses primarily on developing emerging technologies that could increase capabilities for the United States in the Arctic region.CRREL is only one of eight labs, but it is the ideal test example for a scalable model because of its growing strategic relevance in protecting the Arctic and diversity of research and development (R&D) initiatives.The Portfolio Management (PFM) cone illustrates how new and innovative ideas evolve into concepts, proposals, and finally, project execution.Stakeholder analysis confirmed a desire to design a new decision methodology for the far left side of the PFM cone's 'Ideas and leads' as depicted in Figure 1.This methodology would then prioritize and align initial ideas to resources for further development ahead of the PFM process and in order to best address Arctic R&D portfolio strategic needs.

Literature Review
The Arctic region contains eight countries geographically: Denmark, Finland, Russia, Canada, United States, Sweden, Greenland, and Iceland, with China claiming territorial ties (Lee, 2019).The region has increased in interest to the surrounding nations due to its estimated reserves of 90 billion barrels of oil and, roughly 1,699 trillion cubic feet of natural gas, as well as its potential to provide expedited shipping lanes (Long, 2018).This increased interest stems from the decrease in ice coverage, which has been averaging 4.7% per decade, and the development of new technology to both open shipping lanes and harvest untouched resources (Long, 2018).Competition in the region has emphasized technological needs and capabilities in the Arctic region, both immediate and in the long term.The Center for Strategic and International Studies (CSIS) has been cataloging Arctic activity since September of 2020.Russian activity is more prominent in the Arctic, compared to NATO forces and allies, which is seen through installations, technologies, and physical presence in and around the area.
With numerous needs in the Arctic to stay competitive with surrounding nations, USACE is seeking to implement a decision support tool that will create a prioritized project portfolio to reduce uncertainty, predict outcomes, and accelerate mission delivery.The current method for establishing a project portfolio within USACE is through a heuristic-centered Project Portfolio Management (PFM) decision model which addresses mature R&D project proposals.The implementation of a more objective-based approach would reduce possible biases in scoring projects, so the portfolio would more accurately align to meet the Army and Nation's strategic 'ends' in the Arctic.Considering stakeholders and the system itself through an interview with lead strategists, directors, and integrators, there is space for an additional decision-making model to validate the existing PFM methodology and provide feedback using open-source and existing project data.

Systems Thinking
Systems Thinking represents a framework to gain better understanding of the Arctic's complexities and is an initial step in the Systems Decision Process (SDP) which guided the methodology development (Parnell et al, 2011).Systems Thinking eased the understanding and interpretation of how a decision-making process might best enhance R&D project prioritization as part of a portfolio.It also created the opportunity to identify possible desired end states, subject to environmental or conditional changes, of key stakeholders within the Army, USACE, and the laboratories that specifically provide results and technologies for the Arctic.
Figure 2 displays a mental model created with Plectica software which illuminates the distinctions of processes, systems within the model, relationships between the processes and model, and different perspectives; this approach to systems thinking is DSRP framework (Cabrera, 2015).This mental model unwraps the numerous contributing factors that influence key strategies and that contribute to the R&D processes in the Arctic.For example, competition from other countries, collected data, and priorities are identified by higher authorities and affect the labs and their portfolios, including CRREL.The result is a deeper understanding of the complexities of the Arctic R&D needs in the categories of Civil Works, Environmental Quality/Installations, Geospatial Research and Engineering, and Military Engineering (Berman, 2021).Understanding that national priorities and agencies shape the process and contribute to the actual work done in laboratories was a key insight as a result of systems thinking and diagramming.
Proceedings of the Annual General Donald R. Keith Memorial Conference West Point, New York, USA April 28, 2022 ISBN: 97819384962-2-6 161 A Regional Conference of the Society for Industrial and Systems Engineering

The Project
The initial established focus of the project was to accelerate research and development to protect and defend the Arctic.Initially, the end state was to recreate or improve the current Portfolio Management tool (PFM) in use by ERDC.However, a significant event that progressed the project definition was the site visit where three team members went to CRREL in Hanover, New Hampshire.The visit clarified perceptions of the project authorization entities, the scope of the project, and the process of the portfolio ranking system currently in place.Further discussions identified gaps in the current process and methods for new project ideas.This meeting led to a revised problem statement: Build scalable decision model to prioritize initial ideas in the Arctic R&D portfolio that align with Army/ERDC and National/DoD Arctic strategies, ensure the fit to national priorities, addresses threat factors of foreign involvement, supports positive climate change, and evaluates impacts on natural resources.
Discussions with stakeholders about the project's life cycle illuminated the desire to screen and prioritize project ideas before full conceptualization by associating a score with each one.The PFM operates in a space of the decision-making process which occurs after the proposal of ideas.At this point, sunk costs or premature commitments to projects may affect the subjective scores assigned to each project.Therefore, if a model can facilitate the filtering of projects much earlier in the process, where ideas have yet to materialize into concepts and proposals, then this can help key leaders understand where their resources may be allocated adequately.After creating the outline of this model, data from open sources helped to quantify enemy capabilities, climate, and natural resources in the Arctic for the model.Excel was chosen for the prototype model dashboard due to its wide-spread acceptance across USACE and ERDC commands and laboratories.ERDC provided an example of a project portfolio to run through the model and to validate the results.

Process Methodology and Decision Model
The decision methodology and value model is shown in Figure 3.The input for the methodology is a list of ideas that could emanate from ERDC labs, Army Futures Command, Congressional sponsors, and others for approval and development.Each R&D idea would include a description corresponding to a specific initial questionnaire capturing key information including the title, sponsor, laboratory that will take on the project, goal or end state of the project, problem the idea addresses, an initial cost estimate, expected timeline, and strategic objectives the idea falls under.Next, a team of experts conducts an Proceedings of the Annual General Donald R. Keith Memorial Conference West Point, New York, USA April 28, 2022 ISBN: 97819384962-2-6 162 A Regional Conference of the Society for Industrial and Systems Engineering initial screening of each idea.This team includes a strategist from USACE headquarters, a strategist from ERDC, and a subject matter expert from CRREL.This team examines each project and either sends the idea on to the next step in the model or ends the development of the idea for the time being.For the decision model, the team must first consider if the idea supports the Department of Defense or Army Arctic strategies.If it does, they examine the project to see if the CRREL has the capabilities to complete the project.They consider the manpower, technology, and equipment in a Go/No-Go matrix.For an idea to receive a "Go," each respective capability must earn a "Go."If the team deems that the idea can be completed at the laboratory, they consider if the project would fit in a timeline relevant to the strategy.They compare the expected project duration to the projected need of the client to see if the requirement and feasibility are aligned.If the idea is realistic according to all stated requirements, it is evaluated to see how much it aligns with higher priorities and core competencies.Each R&D idea is evaluated through two value model functional criteria and five objectives which capture its alignment to strategic objectives and relevance to the Arctic's current environment.An additive value model (Equation 1) then produces the total value score for each project.The values of x, y, l, m, and n are unique global weights (Figure 3) that are developed using a pairwise comparison.The pairwise comparison enables the project evaluation team to evaluate relative importance of each variable to produce a global weight by assigning higher values to ideas that are the most important.Finally, the scores from each of the five criteria are multiplied by their respective weight of importance and added together in Equation 1to result in a final project score for Project A, which is denoted as QA.
The first value measure, or metric consideration is the ERDC and USACE strategies.Each idea is considered individually and assigned a percent fit score, A A in Equation 1, based on correspondence to total identified ERDC and USACE objectives.The second value measure consideration is the National Arctic Strategy and the Army War College Key Strategic Issue List.This step results in a percent fit score, B A in Equation 1, for every idea, which is measured from 0-100% with 100% being the ideal condition.Assessing R&D ideas against the Arctic environment employs three value measures which are combined under one function, and as additional data is identified, additional value measures could be employed.The first metric for the Arctic environment is enemy presence, which is measured by the number of overflights in the Arctic area and in time will be expanded to include maritime, icebreaker, and military installations.A linear regression of number of overflights in the Arctic region was created, but in time this forecast approach to future threats might be better served with moving averages or exponential smoothing algorithms.The predicted idea completion date is used to predict the number of future Russian overflights provided the regression forecast.This value is then normalized on a scale of 1-100.This value is EnemyA in Equation 1.The second component, climate change, is measured by the average square kilometer of ice coverage in the Arctic.Applying the same approach, and with the same recognition of linear trend forecasting, a linear regression of average ice coverage in terms of date was derived from data that covered the years 1979-2021.The raw score is normalized, and this is the value associated to Climate A in Equation 1.The third component of emergent capabilities in the Arctic is Natural Resources.This Proceedings of the Annual General Donald R. Keith Memorial Conference West Point, New York, USA April 28, 2022 ISBN: 97819384962-2-6 163 A Regional Conference of the Society for Industrial and Systems Engineering data comes from the ratio of Russian to friendly crude oil drawn from the Arctic.The linear regression of the ratio of Russian to friendly crude oil provides a score based on the projected end date of the proposed project.This score is normalized to the value associated with Resources A in Equation 1.
The output is a rank ordered list from greatest value to least of all the project ideas and their corresponding PFM scores.The list will then be divided into two groups based on the expected implementation timeline.Short term projects are less than 7 years and long-term projects are greater.A cut is made in each list based on how many projects of each timeline the lab can support and which ones should proceed to the lab for further development in each category.

Sensitivity and Validating the Model
A method for comparing the results of the Additive Value Model to the PFM is a Kendall's Tau correlation coefficient (Figure 4).The outcome of the additive value model for Equation 1 produced a project score for each project.In addition, ERDC and CRREL experts delivered their projected order of most important ideas given the PFM objectives.These results are displayed in Figure 4. To compare the two, Kendall's Tau seeks to quantify the strength and association between the proposed normative model and the existing descriptive PFM that is currently in use.The equation takes the difference between concordant and discordant pairs and divides it by the number of possible combinations between the two lists.A concordant pair refers to the number of observed ranks below a particular rank which are larger than that rank.Discordant pairs are the number of observed ranks below a particular rank which are smaller in value than that rank.This equation produces a tau value, or a significance level discussed in section 3 of this paper.A significance level of -1 shows complete disagreement while a significance level of 1 shows perfect agreement.A tau value of 0 indicates no correlation in the two rank order lists.

Findings
The first critical finding is that the ERDC R&D portfolio needs a method for accepting projects to ensure alignment with higher priorities.Implementation of this scalable screening decision support tool provides CRREL, and other ERDC laboratories with insight on fundable ideas early in the decision-making process prior to each project's conceptualization and potential execution.Furthermore, ERDC experts found that this decision support tool provides ERDC leaders with an acceptable method of ensuring alignment of project portfolios to the strategic objectives of the Army, the Department of Defense, and USACE, thus contributing to savings of valuable time and resources for the development of projects that better fit the scope of each ERDC laboratory.The model's scalability to other portfolios beyond the Arctic was noted as a strength.
Secondly, based on the Kendall's Tau value from Figure 4, the assessment of the initial model as it relates to the PFM is that it does not accurately predict how the PFM will rank projects.The tau value calculated from the PFM data and the additive value model shows a significance level of -0.3231.This indicates that there is a weak negative correlation between the two project idea rankings.This could inform the ERDC R&D decision-making bodies of inherent cognitive biases that might exist in the current PFM decision method and thus could lead to a convergence of the processes to merge the more descriptive PFM approach and this proposed normative analytical model.
A third finding in Figure 5 is reflected in a sensitivity analysis tornado diagram.A mean value for each variable created the baseline project score.Then, considering the lowest and highest value for a specific variable, Equation 1 produced a project Proceedings of the Annual General Donald R. Keith Memorial Conference West Point, New York, USA April 28, 2022 ISBN: 97819384962-2-6 164 A Regional Conference of the Society for Industrial and Systems Engineering score while holding the four other variables constant at the respective mean value.The tornado diagram reflects "Enemy Threats" has the largest effect on project score when keeping all other variables constant at the mean value.The Army Strategy alignment also has a considerably higher impact on the outcome of a project score.Additional sensitivity analysis using 1 or 2-way tables would be performed on global weights developed by the pairwise comparison given ERDC inputs; for this prototype, the data itself was examined at for its sensitivities.

Conclusions and Future Work
USACE strategists and lead ERDC researchers observed the developed model and considered it to be scalable, usable and useful.They communicated that they are actively posturing to integrate a further refined model.In addition, they validated the importance of objectively assigning values to projects early in the development process to gauge the relative impact of each project idea and take ideas out of the pool if they are not aligned to the objectives of the Nation, Department of Defense, the Army, USACE, and ERDC.This proof-of-concept model design, enabled by a portfolio of actual projects from Congress, validated previous PFM decisions resulting from the determinations of ERDC upon additional iterations of PFM scoring.
Future work could include the creation of new criteria for Objective 3 and scaling the model for use in another lab.There is a wealth of data, including maritime threats in addition to the overflight threats which could be utilized as additional criteria in the additive value model.ERDC also has access to classified data that was not available during development of the model.Alternative or additional measures of climate change like ocean levels, temperature, and ice melt could add value to the model.Other data sources that include the number of training exercises, unauthorized maritime navigation, and drone usage could also contribute to projecting enemy capabilities.Likewise, data that could better reflect natural resource value include natural gas usage, mineral extraction, and fishery harvesting.
The methodology is easily adjustable to align to the needs of any laboratory.It is scalable in its screening questionnaire, the pairwise comparison, and value model metrics.While there is a suggestion in the model for the makeup of the team that conducts the initial screening and objective scoring, this could be changed if deemed necessary.If the model was utilized in a different laboratory, the ERDC strategic objectives in Objective 1 could be changed to describe the strategy of the new lab and their area of responsibility.Additionally, weights of the scores in the additive value model can also contribute to the scalability of the proposed model.It can, and should, be adjusted to take world conditions into account.

Figure 2 .
Figure 2. Plectica DSRP Mental Model of the Arctic R&D Environment