Performance evaluation framework of Wyoming connected vehicle pilot deployment program: summary of Phase 2 pre-deployment efforts and lessons learned


Purpose
This paper aims to present a summary of the performance measurement and evaluation plan of the Wyoming connected vehicle (CV) Pilot Deployment Program (WYDOT Pilot).


Design/methodology/approach
This paper identified 21 specific performance measures as well as approaches to measure the benefits of the WYDOT Pilot. An overview of the expected challenges that might introduce confounding factors to the evaluation effort was outlined in the performance management plan to guide the collection of system performance data.


Findings
This paper presented the data collection approaches and analytical methods that have been established for the real-life deployment of the WYDOT CV applications. Five methodologies for assessing 21 specific performance measures contained within eight performance categories for the operational and safety-related aspects. Analyses were conducted on data collected during the baseline period, and pre-deployment conditions were established for 1 performance measures. Additionally, microsimulation modeling was recommended to aid in evaluating the mobility and safety benefits of the WYDOT CV system, particularly when evaluating system performance under various CV penetration rates and/or CV strategies.


Practical implications
The proposed performance evaluation framework can guide other researchers and practitioners identifying the best performance measures and evaluation methodologies when conducting similar research activities.


Originality/value
To the best of the authors’ knowledge, this is the first research that develops performance measures and evaluation plan for low-volume rural freeway CV system under adverse weather conditions. This paper raised some early insights into how CV technology might achieve the goal of improving safety and mobility and has the potential to guide similar research activities conducted by other agencies.



Introduction
Over the past few years, the U.S. Department of Transportation (USDOT) has been leading several efforts to stimulate the adoption of connected vehicle (CV) technology with the goal of improving the safety, mobility and productivity of the users of the nation's transportation system (FHWA, 2018). CV technology is a broad term to describe the applications and the systems that leverage dedicated shortrange communications (DSRC) for vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I) and infrastructure-to-vehicle (I2V) communication to broadcast real-time traveler information messages (TIMs) between CVs and/or Transportation Management Centers (TMCs) (Liu and Khattak, 2016;Shladover, 2018;Park et al., 2018). In September of 2015, the 402-mile of Interstate 80 (I-80) in Wyoming (WYDOT Pilot) was selected by the USDOT as one of three CV pilot deployment sites (WYDOT, 2018). The other two pilot deployment sites were New York City (NYCDOT Pilot) and Tampa City (THEA Pilot). Among the three CV pilot deployment sites, NYCDOT Pilot aims to improve the safety of travelers and pedestrians at tightly-spaced urban intersections through the deployment of V2V and V2I and Infrastructure-to-Pedestrian (IVP) CV technologies (NYCDOT, 2018). These applications provide drivers with alerts so that the driver can take action to avoid a crash or reduce the severity of injuries or damage to vehicles and infrastructure. The THEA Pilot deploys a variety of V2V and V2I applications to relieve congestion, reduce collisions and prevent wrong way entry at the exit. THEA also plans to use CV technology to enhance pedestrian safety, speed bus operations and reduce conflicts between street cars, pedestrians and passenger cars at locations with high volumes of mixed traffic (THEA, 2018). Unlike the other two CV Pilot sites that focus on urban area, Wyoming Department of Transportation (WYDOT) Pilot focuses on the needs of the commercial vehicle operator on I-80 in the State of Wyoming and will develop CV applications that use V2I and V2V connectivity to support a flexible range of services from advisories including roadside alerts, parking notifications and dynamic travel guidance.
The Wyoming I-80 corridor is a major freight and passenger corridor that is subject to frequent and severe weather events, which affects the safety and mobility of the corridor. The truck volume ranges between 30 and 55 per cent of the total annual traffic volume and comprises as much as 70 per cent of the seasonal traffic volume. Furthermore, its elevation is all above 6,000 feet, with the highest point reaching 8,640 feet (2,633 m) above mean sea level. Drivers on I-80 can experience high winds, heavy snow, blowing snow, low visibility from blizzard conditions and icy road surface (see Figure 1), which resulted in numerous road closures each winter season. These conditions also directly contributed to higher than normal incident rates, often resulting in multivehicle crashes and significant economic loss, especially during the winter months (WYDOT, 2018). In addition, alternate routes are limited and can add significant extra time to the trip. Furthermore, the mix of passenger vehicles and commercial vehicles can result in catastrophic crashes. For example, during elevation changes, slow-moving commercial vehicles can cause drivers of passenger vehicles to take risky maneuvers they normally would not take, such as driving too closely, cutting off other vehicles to pass, or speeding to make up for lost times.
With these concerns, the WYDOT Pilot aims to reduce the number of blow-over incidents and adverse weather related incidents (including secondary incidents) in the corridor in order to improve safety and reduce incident-related delays (Gopalakrishna, 2016a). Specifically, the WYDOT Pilot Deployment Program include the following scopes (Gopalakrishna, 2016b): Deploy about 75 roadside units (RSUs) with DSRC that are able to transmit advisories and alerts to equipped vehicles along I-80.
Develop several CV applications that will enable communication to drivers of alerts and advisories regarding various road conditions. These CV applications mainly include: forward collision warning (FCW), distress notification (DN), situation awareness (SA), work zone warning (WZW), spot weather impact warning (SWIW).
Equip and operate a set of vehicles that are expected to be regular users of I-80, with on-board units (OBU) with DSRC connectivity. Figure 2 illustrated the existing communication and traffic control devices along Wyoming I-80 corridor, and the DSRC locations that will be deployed on the corridor.
Systems and applications developed in the WYDOT CV Piot Deployment Program are expected to enable CV drivers to have improved awareness of potential hazards when driving on I-80. Specifically, this pilot expects to support TMC staff to implement more effective traffic control strategies on the corridor especially during periods of adverse weather and when work zones are present. Through the anticipated outcomes of the pilot, fleet managers will be able to make better decisions regarding their freight operations on I-80, truckers will be made aware of conditions downstream and will be provided guidance on parking options as they travel the corridor. Moreover, automobile travelers will receive improved road condition and incident information through various existing and new information outlets.
An evaluation of the impacts of the CV Pilot is central to the USDOT's strategic goals (Vasudevan and Asare, 2016); in this regard, each CV Pilot site developed a Performance Measurement and Evaluation Support Plan during Phase 1 -Concept Development phase of CV Pilot Deployment Program (Galgano et al., 2016;THEA, 2016;Kitchener et al., 2016). This paper summarized the performance measurements, confounding factors that may affect system performance, system evaluation efforts for baseline conditions, and system evaluation plan for post-deployment conditions of the WYDOT Pilot. The remainder of this paper is organized as follows: Section 2 presents a review of state-of-the-art regarding performance evaluation of CV systems. Section 3 illustrates the pilot on-board CV applications developed by the WYDOT Pilot as well as expected benefits. Sections 4 to 6 describe the detailed performance measures identified by the WYDOT Pilot team, potential confounding factors that might affect the performance of WYDOT CV system, and approaches for performance evaluation, respectively. Section 7 details the methods for collecting traffic performance data as well as preliminary analysis of pre-deployment operation conditions. Section 8 discusses the potential challenges that might be involved in the system performance evaluation and the Figure 1 An illustration of Wyoming I-80 under winter adverse weather condition (Mack, 2018) potential of using microsimulation modeling for performance evaluation. Finally, Section 9 concludes the preliminary findings from the pre-deployment phase of WYDOT Pilot as well as insights into post-deployment phase.

Literature review
As mentioned before, in addition to the WYDOT Pilot, the NYCDOT Pilot and THEA Pilot also developed macroscopiclevel performance evaluation framework for their CV systems. Both of NYCDOT and THEA Pilots proposed site-specified performance measures, confounding factors that may affect system performance, the data collection process, and the experimental design and analytical processes that will be undertaken to evaluate the impacts. Specifically, the NYCDOT Pilot considered speed compliance and reductions in vehicle to vehicle, vehicle to pedestrian and vehicle to infrastructure crashes as the primary performance measures (Galgano et al., 2016). Potential confounding factors mainly include traffic demand variations, weather condition, accident and incidents, updates of traffic signal timing, short-term and long-term work zones, planned special events, economic related conditions, and the use of alternative traffic modes such as shared vehicles, bike, and public transit (Galgano et al., 2016). Similarly, the THEA Pilot concerned both safety (including wrong-way entries, pedestrian safety, streetcartrolley conflicts) and traffic operation (including morning peak hour queues, bus rapid transit signal priority optimization, signal coordination and traffic progression) as performance measures, with weather, special events, city constructions, system reliability, and participant-specific features as confounding factors (THEA, 2016).
At microscopic-level, the mobility and safety benefits of CV technology are mostly stemmed from the changes of CV drivers' route choice as well as driving behavior to the received real-time CV alerts. In current practice, being limited by the low number of CVs in a real-world setting, the majority of existing CV performance evaluation research were based on simulation modeling.
In term of mobility benefits assessment, Olia et al. (2016) developed a microsimulation modeling framework to quantitatively assess the potential impacts of CV on mobility.
Specifically, this research examined the effects of providing real-time routing guidance and advisory warning messages to CVs. The results demonstrated the potential of CVs to improve mobility at the network-wide level, and market penetration of CVs proportionally affected the performance of the traffic network. Mahmassani (2016) evaluated the impacts of CVs on traffic flow and operations. A framework that utilizes different models with technology-appropriate assumptions was developed to simulate different vehicle types with distinct communication capabilities. The stability analysis of the resulting traffic stream behavior using the developed framework was presented for different market penetration rates of connected and autonomous vehicles (CAVs). The analysis revealed that CAVs could improve string stability. In addition to stability, the effects of these technologies on throughput were explored, suggesting substantial potential throughput increases under certain penetration scenarios. Wang et al. (2017aWang et al. ( , 2017b employed Travel Time Reliability (TTR) to assess the mobility benefits of CV system. Five potential confounding factor that might affect system performance were considered: congestion level, CV penetration rate, CV drivers' compliance rate, release delay time, and following rate. Simulation results indicated that penetration and compliance rates have a positive effect on TTR, the average improvement rate was found to be about 77 per cent and 73 per cent with the increase of penetration and compliance rate, respectively. Bahaaldin et al. (2017) employed VISSIM microsimulation to evaluate the impacts of CVs on no-notice evacuations based on a case study of a 24 km freeway segment of St. Louis, MO. The vehicle communication capabilities were modeled using the VISSIM's COM interface, and the microsimulation model was calibrated using field collected travel time data. System performance with CV penetration rates from 0 to 30 percent were evaluated; results indicated that significant reductions in total delays when CVs reached a penetration rate of 30 percent. Tian et al. (2018) provided an in-depth survey on the performance measurement evaluation of CAV applications. This research summarized three typical performance measures: mobility, safety, and environmental benefits, and analyzed the potential tradeoffs and co-benefits of the three performance measures.
For safety performance assessment, the most commonly used methods have been the Surrogate Measure of Safety (SMoS) based methods. The concept of SMoS, along with the Surrogate Safety Assessment Model (SSAM) were first introduced by Gettman and Head (2003). Among various SMoS used in the literature, time-to-collision (TTC) speed variance and headway variance were found to be efficient SMoS. To date, there has been a number of studies that adopted SMoS for traffic safety assessment in a CV environment. Generally speaking, the traffic performance data were collected either from existing roadway system with CVs running on the road or based on microsimulation modeling. Olia et al. (2013) attempted to quantify potential safety benefits of deploying a CV system through microscopic traffic simulation modeling. PARAMICS was used to model CVs, construction zones and incidents associated with work zones. The result of this research clearly demonstrated the effectiveness of CV systems to improve network safety. The percentage of CVs within the network was found to be the most significant factor to increase network safety (up to 50 per cent improvements). This could be explained by re-routing to alternate routes and increased driver awareness. Paikari et al. (2013) explored a low cost modeling approach to provide advisory speed and re-routing guidance in V2V and V2I systems for improving safety and mobility on freeways. The study tested fifteen scenarios differentiated by the V2V percentage penetration (0 per cent, 10 per cent, 20 per cent, 30 per cent, and 40 per cent), and demand loading (60 per cent, 80 per cent, and 100 per cent) implicitly representing peak and off-peak traffic. It was found that CV technology can enhance traffic safety on freeways, if the percentage of CVs is significant (e.g. 30-40 per cent) and when it is accompanied by advisory speed reflected on VMSs not only upstream but also downstream of the incident location. Genders and Raviza (2016) evaluated the potential safety benefits of deploying a CV system on a traffic network in the presence of a work zone. The modeled CV system in the study uses V2V communication to share information about work zone links and link travel times. Vehicles which receive work zone information will also modify their driving behavior by increasing awareness and decreasing aggressiveness. Improved Time-to-Collision (ITTC) was used as a SMoS to assess the safety of the network. Various market penetrations of CVs were utilized along with three different behavior models to account for the uncertainty in driver response to CV information. The results showed that network safety was strongly correlated with the behavior model used; conservative models yielded conservative changes in network safety. The results also showed that market penetrations of CVs under 40 per cent contribute to a safer traffic network, while market penetrations above 40 per cent decrease network safety. The decrease in safety when rerouting more than 40 per cent of traffic on a work zone could be explained by the fact that more traffic diverted to other alternate routes resulting in more exposure to higher traffic volumes and increased crash risks. Fyfe and Sayed (2017) combined VISSIM and SSAM with the application of the Cumulative Travel Time (CTT) algorithm to evaluate the safety under CV environment. The study showed a 40 per cent reduction of rear-end conflict frequency at a signalized intersection with the application of CV. Yang et al. (2017) investigated the impact of CV on mitigating secondary crash risk through a simulation-based modeling framework that enables V2V communication. A 4-mile highway section was modeled in PARAMICS; a Modified Time-to-Collision (MTTC) was proposed as the SMoS to capture vehicular conflicts as a proxy for secondary crash risk upstream of a primary crash site. Simulation results showed that CVs can be a viable way to reduce the risk of secondary crashes, and the benefits increased with an increasing market penetration of CVs. Wang et al. (2017aWang et al. ( , 2017b proposed a simulation-based performance evaluation framework for testing and evaluating Advanced Driver Assistance Systems (ADAS) using lane departure events collected from the University of Michigan Transportation Research Institute (UMTRI) Safety Pilot Model Deployment naturalistic driving database. Statistical results indicated that the proposed framework could capture the impacts of Lane Departure Correction (LDC) systems on driver behavior and has the potential to evaluate other ADAS as well as autonomous vehicles. Nair et al. (2018) developed an enhanced microsimulation model to simulate vehicle collisions, based on which, to quantify the safety benefits of a CV application -Road Hazard Warning. Safety performance was measured by Time Integrated Time-to-Collision (TIT) and number of secondary crashes. Eventually, a linear relationship was drawn between CV market penetration rate and safety improvement. Rahman et al. (2018) employed the standard deviation of speed, the standard deviation of headway, and rear-end crash risk index (RCRI) as SMoS in a microsimulation environment to assess the safety effectiveness of CV technologies. Simulation results indicated that CV significantly improved traffic safety in fog conditions as market penetration rates of CV increase. Similarly, Abdulsattar et al. (2018) presented an agent-based modeling and simulation framework to evaluate the safety performance impacts of CV technologies in work zone under various CV market penetration rates and traffic demand levels. TTC and Time Exposed Time-to-Collision (TET) were used as SMoS for safety evaluation, and concluded that the higher the traffic flow rate, the higher CV market penetration level is needed to show improvement in the safety performance of the work zone section.

Pilot on-board connected vehicle applications
The pilot developed five on-board CV applications that will provide key information to the drivers of equipped vehicles (Gopalakrishna et al., 2016a): FCW, I2V SA, DN, WZW, and SWIW. Through the on-board CV applications, WYDOT hopes to improve on road messaging on the corridor, especially when adverse weather and work zones are present.

Forward collision warning
FCW is a V2V communication-based safety feature that issues a warning to the driver of the CV in case of an impending frontend collision with a CV ahead in traffic in the same lane and direction of travel on both straight and curved geometry roadways. FCW will help drivers avoid front-to-rear vehicle collisions in the forward path of travel. The conception diagram of FCW CV warning is illustrated in Figure 3.

I2V situational awareness
This application enables relevant downstream road condition information including weather alerts, speed restrictions, vehicle restrictions, road conditions, incidents, parking, and road closures to be broadcasted from a RSU and received by the CVs. Such information is useful to CVs that are not fully equipped with weather sensors or to CVs approaching or entering areas with hazardous conditions. The conception diagram of SA CV warning is illustrated in Figure 4.

Distress notification
This application enables CVs to communicate a distress status when the vehicle's sensors detect an event that might require assistance from emergency services or the vehicle operator manually initiates a distress status. The vehicle generates and broadcasts a distress message to the nearest RSU. When an RSU is not within communication range, the message is received by CVs that are in the vicinity and forwarded to an RSU that transmits the message to the WYDOT CV System. Additionally, the DN received by nearby CVs is broadcast to notify oncoming vehicles that a distressed vehicle is ahead. The conception diagram of DN CV warning is illustrated in Figure 5.

Work zone warnings
This application provides information about the conditions that exist in a work zone a vehicle is approaching. This capability provides approaching vehicles with information about work zone activities that could present unsafe conditions for the vehicle, such as obstructions in the vehicle's travel lane, lane closures, lane shifts, speed reductions or vehicles entering/ exiting the work zone. The conception diagram of WZW CV warning is illustrated in Figure 6.
3.1.5 Spot weather impact warning SWIW is a special case of SA. Similar to SA, this application enables relevant road condition information, such as fog or icy roads, to be broadcasted from a RSU and received by the CV. This application, however, is distinct from SA in that it provides more localized information (i.e. at the segment level instead of area wide or region wide). The conception diagram of SWIW CV warning is illustrated in Figure 7.

Expected benefits
The expected benefits of the project revolve around objectives of improving safety, mobility and productivity of the users of I-80 in Wyoming. These benefits are directly dependent on the CV applications that are being developed during the WYDOT Pilot. Table I summarizes the expected benefits of the CV applications to various users of the WYDOT CV system.
The benefits are expected to be realized by both connected and non-CVs as much of the WYDOT CV system's improved information will also be channeled through their existing traveler information systems (i.e. 511, 511 phone application, websites, dynamic message signs etc.).

Performance measures
The WYDOT Pilot team has identified 21 performance measures incorporated within 8 performance categories. The major performance categories represent the primary activities and outcomes of the WYDOT CV pilot system. These categories focus on improvements to efficiency, safety and mobility. Quantitative and qualitative measures were proposed to evaluate the WYDOT CV project with a focus on understanding the extent and impact of the benefits described above. For each performance measure, targets have been identified. Table II summarizes the proposed performance measures and targets, which will guide the evaluation of the WYDOT Pilot project. An overview of the architecture of the performance measure categories is illustrated in Figure 8.

Overview of confounding factors
The WYDOT Pilot is unique among the three pilot sites in its project extent (over 400 miles of mostly rural Interstate) and its focus on improving safety during severe weather events. To provide an accurate evaluation of the CV deployment program, it is critical to identify confounding factors and isolate their impacts so that performance improvements are neither overstated nor understated. This section identifies confounding  Improved safety through (near) real-time notification of unsafe work zones at specific points on the downstream roadway SWIW Improved safety through (near) real-time notification of unsafe conditions or road closure at specific points on the downstream roadway factors that may affect the ability to successfully implement the evaluation designs described herein. The confounding factors identified by the Wyoming team are listed below:

CV Technology penetration rate
The pilot demonstration testing will be challenged by the relatively few number of vehicles that will be capable of receiving direct information from the infrastructure or other vehiclesespecially in comparison to the number of total vehicles on I-80 (which has a range of average daily traffic volumes between 10,000 and 20,000). The technology can be shown to work; however, measuring the benefits with such a small sample size will be limiting.

Freight and passenger vehicle demand
The corridor is heavily used by commercial freight trucks, making system performance highly dependent on freight demand. This affects both the total number of freight trucks and the percentage of trucks in the overall traffic flow in the corridor. Freight demand along the corridor is mostly impacted by the national economy since earlier research has shown that more than 90 per cent of the truck traffic along I-80 neither originates nor is destined for locations within Wyoming. Fluctuations in freight traffic could be caused by changes in goods movement demands, economic conditions, fuel prices, or heavier than normal construction seasons, all of which are major variables in the logistic decisions made by fleet managers. Number of road sections with at least one reported road condition per hour pre and post CV Pilot (coverage) 3 Average refresh time of road condition reports in each section pre and post CV Pilot (latency) 4 Pikalert TM generated motorist alert warnings (MAWs) that were accepted by TMC operators Improved Ability of the TMC to Generate Alerts and Advisories 5 Number of messages sent from the TMC that are received by the RSU Effectively Disseminate and Receive I2V and V2I Messages 6 Number of messages sent and received between the RSU and WYDOT fleet vehicle's OBU (when vehicles are in the vicinity of a RSU) 7 CVs that likely took action following receipt of an alert. "Parked, Reduced speed, Came to a stop safely, Exited" 8 Commercial vehicle managers are satisfied with information provided by the TMC (compare before and after CV Pilot). "Road conditions, Road weather forecasts, Parking information) Improved Information to Commercial Vehicle Fleets 9 Number of operational changes made by fleet managers due to information from TMC (compare before and after CV Pilot). "Routing, Timing, Parking availability, Canceled trips" 10 Commercial vehicle drivers' benefits experienced due to CV technology during major incidents and events on I-80 11 Number of V2V messages properly received in surrounding vehicles from sending vehicle (WYDOT fleet vehicles in vicinity of each other) Effectively Transmitted V2V Messages

12
CVs that likely took action following receipt of a V2V alert. "Parked, Reduced speed, Came to a stop safely, Exited" 13 Number of emergency notifications that are first received in the TMC from CVs (compared to alternate traditional methods, such as 911 caller)

14
Total vehicles traveling at no more than 5 mph over the posted speed (compare before and after CV Pilot)

15
Total vehicles traveling within 1/-10 mph of 85th percentile speed (compare before and after CV Pilot) 16 Speed of applicable CVs are closer to posted speed when compared to non-CVs 17 Number of CVs involved in a crash. "Initial crashes, and Secondary crashes" Reduced Vehicle Crashes 18 Reduction of the number of vehicles involved in a crash (compare a multi-year average before and after CV Pilot) 19 Reduction of total and truck crash rates within a work zone area (compare a multiyear average before and after CV Pilot) 20 Reduction of total and truck crash rates along the corridor (compare a multi-year average before and after CV Pilot) 21 Reduction of critical (fatal or incapacitating) total and truck crash rates in the corridor (compare a multi-year average before and after CV Pilot) For instance, high fuel prices can also result in fleet managers reducing the maximum speed allowed for its drivers, construction along a corridor may push drivers to choose alternate routes or avoid travel altogether if the impacts of construction are viewed as too large.

Connected vehicle Technology adoption
New technology involving a change in the way people do things is always challenging. For the being evaluated CV system, there are myriad agency personnel that are affected: Transportation Management Center (TMC) staffs, snowplow drivers, commercial vehicle truck drivers, commercial vehicle company dispatch center personnel, etc. How these stakeholders adopt the new technology and information that will now be available is unknownthe lack of technology adoption and information use may affect the evaluation outcomes.

Weather condition variability
The variability of weather events and entire winter weather seasons presents challenges to analyzing pre-and post-system implementation data. Ideally, the evaluation would compare data during similar weather eventsthis is not always possible. Improve road weather conditions reports received into TMC

Vehicle/Roadside Alerts & Advisories Outcomes
Improve ability of the TMC to generate alerts and advisories

New information use
System capabilities are limited by how much information can realistically be given to drivers. There is a multitude of possible information that could be provided including roadway and weather condition, speed limits, warnings, incidents ahead, detours, parking opportunities, etc. This could cause an information overload situation or drivers misunderstanding the messages. In a project stakeholder meeting it was suggested that the information must be simple to understand and easily delivered. This issue has been a significant design challenge.
Additionally, even though this information will be made available to commercial vehicle companies and drivers, it is not known the extent to which the new information will be fully utilized by these groups.

Safety data availability
Reported crashes in Wyoming are stored in a statewide database maintained by WYDOT regardless of the agency that has jurisdiction of the roadway where the crash occurred. For the I-80 corridor, most of the crash records are generated by the Wyoming Highway Patrol (WHP). Crash reports are required for any crashes involving an injury or for property damage in excess of $1,000. If an officer is not sent to the crash scene, the drivers are required to notify the law enforcement agency nearest to the scene. Crash reporting forms used by the WHP follow the Model Minimum Uniform Crash Criteria (MMUCC) guidelines.

Limited duration of evaluation activities
The WYDOT pilot is focusing on providing information and improving safety during inclement weather which occurs mostly during the winter months. The CV Pilot Demonstration will begin in October 2018 and continue through November 2019. This will allow for only one winter season of data collection, analysis, and reporting. Baseline data collection and documentation occurred during Phase 2 (2016-2017 winter season). Another concern is whether the appropriate weather conditions exist to conduct the evaluation during the evaluation period as designed. These issues may limit the amount and content of evaluation data during the designated evaluation period.

Availability of speed sensing in the corridor
Currently, radar speed sensors are spaced approximately 6-7 miles apart in the VSL corridors. These speed sensors are located along with the VSL signs and RWIS sensors to take advantage of the power and communication infrastructure available at these sites. The WYDOT Pilot team acknowledges that knowing the vehicle speeds in between the VSL signs would be helpful to ensuring a consistent and correct speed to enhance safety and to determine if the speed adherence impacts are sustained past the location where the speed signs are posted.

Reliability of roadside and in-vehicle systems
The reliability of roadside and in-vehicle systems is not known. This includes DSRC equipment to transmit and receive information to and from vehicles from the roadside to vehicle, vehicle to roadside, and vehicle to vehicle. The winter environment can be extreme in Wyoming including severe cold, high winds, and significant snowfall. Additionally, the equipment that will be installed in trucks will be subjected to a harsh noise and vibration environment that may impact their operational life and capabilities. If the reliability of this equipment is less than expected, the data collection activities to support system performance measurement could be compromised.

Performance evaluation approaches
Evaluation designs incorporate both quantitative and qualitative approaches to make assessments of specific performance measures. The design also provides an evaluation structure that can help control for the potential effects of confounding factors. It is important to seek to control such effects in order to be able to say that the WYDOT Pilot project itself was responsible, or at least primarily responsible, for the identified outcomes. The following five evaluation design types are anticipated for use during the evaluation activities.

Before-after study
This approach quantitatively compares traffic performance data under baseline conditions (pre-deployment) with data during the WYDOT Pilot demonstration (post-deployment).
The before-after evaluation design requires the establishment of a baseline to document conditions before the CV Pilot project elements were deployed. As mentioned earlier, the baseline has been established during Phase 2 (2016-2017 winter season) and will define the benchmarks from which future conditions with the CV Pilot project elements in place can be compared.

With-without study
This technique quantitatively compares traffic performance data from conditions where CV Pilot project elements are in place and used (experimental) with conditions where CV Pilot project elements are not present (control). These comparisons would be made during the same time, location and conditions. An advantage of the with-without design is the ability to effectively control for variability in weather conditions and other confounding factors that would equally affect two different situations, one of which would experience the CV Pilot deployments and the other would not. The differences in outcomes will be observed between these two situations, and those differences will be attributed to the effect of the CV Pilot elements. Fewer performance measures identified will utilize this method. Examples of data needed to support the withwithout analyses include speed of connected vs non-CVs, crashes of connected vs non-CVs, and driver behavior in terms of actions taken following receipt of an alert (whether from the TMC or another CV).

System performance evaluation
This evaluation will quantitatively assess how well the system worked to provide information, alerts, and advisories (V2I and V2V). The two previous evaluation approaches (before-after and with-without) do not apply to the evaluation of these measures. The system performance evaluation approach will collect the necessary data to assess how well the system performed against the expectations (targets). Examples of data needed to support this approach include number of automated alerts/advisories sent and number that were actually received, including V2I and V2V applications.

Behavior assessment
This evaluation measure specifically addresses behaviors or actions that result from alerts being received by drivers of CVs. Possible behaviors could include speed reductions, coming to a stop safely, parking the truck in the event of a closure, or exiting around an incident or closure if available. Additionally, this evaluation design will apply to operational changes made by fleet managers in response to information received from the TMC.

Qualitative assessment
This assessment is a descriptive approach to evaluate a particular strategy implementation, a qualitative assessment evaluation seeks to identify what worked well and what did not and derive lessons from the experience.
For the before-after, with-without, and system performance evaluation designs, qualitative surveys or staff interviews obtained from key informants are useful in supplementing the quantitative data normally collected during the evaluation period and aiding in interpreting evaluation results. An advantage of this approach is that it is tightly focused on a particular (or several) CV Pilot deployment and can track the cause-effect relationships as the use of these deployments yield desired outcomes. The data are primarily derived from readily available sources and surveys and/or interviews with key project stakeholders such as TMC operators, WYDOT maintenance personnel, fleet managers, CV drivers, etc.

Data collection
Descriptions of the pre-deployment data collected to support the analyses of the aforementioned performance measurements are provided below by grouping the data into six categories.

Category 1: road condition reports
The WYDOT TMC collects and stores all field maintenance reported road conditions by day/time and location. Special software was written to extract the data required during weather events. WYDOT rates the overall impact (low, moderate, high) to the traveler by various road conditions, weather conditions, advisories, and restrictions. Data collected in this category supports the calculation of the following performance measures: PM # 1. Number of road condition reports per road section/day (quantity); PM # 2. Number of road section with at least one reported road condition per hour (coverage); and PM # 3. Average refresh time of road condition reported in each section (latency).

Category 2: commercial vehicle operator surveys
The data collected to support the establishment of the predeployment conditions in this category was from a survey executed by WYDOT. The survey was sent to all subscribers of the Commercial Vehicle Operator Portal (CVOP). The portal provides specific weather forecast information that helps commercial vehicle operators make decisions or preparations for upcoming truck trips. Data collected in this category supports the calculation of the following performance measures: PM # 8. Commercial vehicle managers are satisfied with the information provided by the TMC; and PM # 9. Number of operational changes made by fleet managers due to information from TMC. 7.1.3 Category 3: commercial vehicle baseline driver surveys Data collected in this category supports the establishment of a baseline for the following performance measure: PM # 10. Commercial vehicle drivers' benefits experienced due to CV technology during major incidents and events on I-80.
As part of the training program, commercial vehicle drivers were asked a set of questions to establish baseline conditions. The questions were grouped into the following categories: Experience driving a commercial vehicle on I-80; Familiarization and understanding of technological tools to support safe driving within the I-80 corridor, including CV technologies; How traveler information is acquired and used to ensure safe and efficient driving within the I-80 corridor;

Impressions of safety and use of technology; and
Responses to various severe weather conditions when driving within the I-80 corridor.

Category 4: speed
Individual speed data comes from 74 of the 88 Wavetronix speed radar devices. Location of the speed sensors, as well as weather stations and VSL corridors are illustrated in Figure 8. Various sensors descriptors including a unique ID, device name, route milepost, and the location of the speed sensor installation will be used for matching lane numbers to direction of travel. Data collected in this category supports the calculation of the following performance measures: PM#14. Total vehicles traveling at no more than 5 mph over the posted speed (compare before and after CV Pilot); and PM#15. Total vehicles traveling within 1/-10 mph of 85th percentile speed (compare before and after CV Pilot).

Category 5: crash
Crash records for the state of Wyoming are maintained by the WYDOT's Highway Safety Program. The crash data for the baseline safety performance measures covers the time period from January 2010 to December 2017. Data collected in this category supports the calculation of the following performance measures: PM#18. Reduction of the number of vehicles involved in a crash (compare a 5-year average before Pilot to CV Pilot data); and PM#19. Reduction of total and truck crash rates within a work zone area (compare a 5-year average before Pilot to CV Pilot data); PM#20. Reduction of total and rates of truck crash along the corridor (compare a 5-year average before Pilot to CV Pilot data); and PM#21. Reduction of critical (fatal or incapacitating) total and truck crash rates in the corridor (compare a 5year average before Pilot to CV Pilot data).
7.1.6 Category 6: supportive data Other supportive data sources such as weather, VSL, road closures, work zones, traffic volumes, and dynamic messages were found to be important to the performance measure analyses. In some cases, this supportive data were necessary to incorporate with the primary data sources to verify the performance of developed methodologies.
Road Weather Information System (RWIS) sensor data were collected from ten priority stations (from 50 total along I-80). RWIS data are currently used to account for weather in the speed-related performance measures. VSL data came from 66 Variable Speed Limit signs located along the I-80 corridor. The VSL dataset were organized by month and offer information regarding device ID, milepost range, location, default speed setting and current posted speed in 5-minute intervals. Roadway closures along the project corridor are relatively common occurrences due to weather and crash events. A road closure database is maintained by the WYDOT TMC. For the baseline analysis period, road closure data from October 2016 to May 2017 plus October and November of 2017 were analyzed. Work zone data were collected from WYDOT's database called the Construction Console. The database documented information about the construction project number, location, and start and end dates. In order to determine the effects of traffic volumes on various performance measures and to calculate the crash rates for different corridor segments, the traffic volumes for the 402-mile project corridor had to be determined. Information on traffic volumes on WYDOT facilities can be obtained from the annual WYDOT Vehicle Miles Book. Dynamic Message Sign (DMS) data came from 40 I-80 DMSs. The DMS sign data are only used to verify conditions on the roadway.

Preliminary analysis of pre-deployment data
The primary focus of the WYDOT Pilot is to improve safety on the I-80 corridor in Wyoming. The analysis of historical and current speed adherence and crash data presented herein provides some early insights into how CV technology may achieve this goal of improved safety. For instance: During this baseline data collection period for all weather conditions, about 14.2 per cent of vehicles are currently traveling 5 mph above the post speed (speed adherence is good) and a 29.6 per cent of the vehicles are traveling outside a 1/-10 mph buffer (speed variation is moderate). For certain severe storm conditions, like ice and high winds, the compliance rate drops to 53.4 per cent and the speed buffer to 45 per cent. These conditions can translate or contribute to the number of crashes and crash severity. It is anticipated that an improvement in these values through CV-technologies to improve SA (TIM messages) regarding posted speeds, especially in VSL areas. Additionally, the VSL systems and DMS will have more accurate and timely information based on improved and expanded data collected by CVs.
A total of 1,310 crashes were recorded from October 2016 through May 2017. Weather conditions existing during the crashes included clear (48 per cent) and snowing (21 per cent). Road conditions existing during the crashes included ice/frost (39 per cent), dry (36 per cent) and snow (15 per cent). It is expected that CV-enabled technologies can help to reduce the number of crashes during all conditions. FCW can help avoid a crash in any condition. SWIW can alert a driver to poor weather or road conditions resulting in an avoided crash. Improved driver SA through TIM messages can also result in an avoided crash, especially during inclement weather and hazardous road conditions. Historically, about 30 per cent of crashes on I-80 are multi-vehicle crashes, which include some events with tens of vehicles involved. One of the key goals of WYDOT Pilot is to reduce the number of secondary crashes by using CV technologies to alert drivers of a crash ahead so they can stop earlier or otherwise avoid becoming a crash victim. Further, these crashes can be the reason a section of I-80 need to be closed. It is anticipated that implementation of CV applications such as FCW, DN, WZW, and in-vehicle TIM messages have the potential to reduce the number of vehicles in a crash by warning the driver of a crash just ahead.
Since 2010, 553 critical injury crashes have resulted from crashes on I-80. Of those, 132 fatal crashes occurred. Through implementation of CV technologies mentioned above, it is expected that the WYDOT Pilot have the potential to significantly reduce these numbers either by drivers avoiding a crash altogether or speeds being reduced during a crash.

Challenges and insights into future works
When assessing the performance of CV systems, the traditional "Before-After" analysis approach is usually challenged by the relatively few number of CVs that are capable of receiving information from the infrastructure or other CVs. Mainly, statistical analysis performed utilizing historical data is the core of the aforementioned traditional approaches. Being in an early development phase of CV, assessment of the obtained mobility and safety benefits is challenging and still unclear. Enough data to evaluate the system performance in the after implementation period are not available, which hinders the use of traditional performance evaluation methodologies. Considering the challenges involved in collecting traffic performance data under adverse weather conditions and the costs of in-lab data processing, it is critical to determine the appropriate amount of naturalistic driving data needed to understand changes of driver behaviors and provide a reliable and accurate performance assessment results Wang et al. (2017aWang et al. ( , 2017b In additional, penetration rates are considered among the key factors affecting the benefits obtained from the CV system as well as it affect its safety evaluation. In order to adopt the traditional methodologies in the evaluation process, a significant proportion of vehicles on the roadways should be CV. Being in an early phase of the era of CV, penetration rates are still low. This also would delay the ability to use traditional evaluation processes. In addition, variability of weather events in Wyoming presents extra challenges to the analysis of Preand Post-system implementation data. Ideally, the performance assessment should compare data during similar weather events, which is not always possible. With these considerations, the WYDOT CV performance evaluation team also proposed using microscopic traffic simulation modeling to aid in investigating the mobility and safety effectiveness of WYDOT CV system. Microsimulation modeling has been proven as a cost-efficient method in assessing both mobility and safety performance of CV systems (Smith and Razo, 2016). In comparison with the traditional approaches that require a large sample size of traffic performance data under CV environment, a well calibrated microsimulation model has the ability to accurately and quickly test various CV strategies and directly output the system performance with very low marginal cost. Another advantage of microsimulation approach is the ability to extrapolate various levels of CV technology penetration rates. A microsimulation model has the ability to effectively control for variability in weather conditions, passenger and freight vehicle demand, and other confounding factors. In addition, since a traffic collision or conflict is a small-probability event in a real-world setting, traditional empirical data analysis approach requires a relatively long period to collect ample sample for data analysis. In comparison, microsimulation modeling employs surrogate of safety measures to assess the number, type, severity and locations of simulated conflicts. Since a microsimulation model has the ability to control the confounding factors that may affect system performance, this provides a more credible environment for the comparison of system performance between the Pre-and Post-deployment periods. More importantly, microsimulation modeling allows for conducting large number of simulation runs to test system performance under various CV penetration rates as well as weather scenarios and demand levels. Through such a sensitivity analysis, it is possible to identify confounding factors that affect system performance and potential strategies for managing them.
Nevertheless, to generate an accurate simulation results, the driver behavior parameters in the microsimulation models need to be calibrated using field collected traffic performance data (Rakha et al., 2009;Songchitruksa et al., 2016;Khavas et al., 2017;Hammit et al., 2019). With consideration of the unique traffic flow, terrain, and winter weather features in Wyoming, additional research is required to identify driver behavior under adverse weather conditions. The other needed research is to calibrate the impact of CV on microsimulation parameters. This requires an integration of the CV pilot data, including the actual CV penetration rate during the post-deployment data collection period, driver compliance rates to CV alerts, speeds of connected and non-CVs. The driver behavior data mainly include car-following headway, standstill distance, minimum front to rear headway, lane-changing gap, acceleration and deceleration rates, waiting time before diffusion, etc. In addition to collect naturalistic driver behavior data under adverse weather conditions and/or in a CV environment, an alternative method for calibrating CV microsimulation models is using driver behavior data generated from high-fidelity driving simulators (Chen et al., 2019; Yang et al., 2019), which can overcome the limitation of the field data-based model calibration approach.

Concluding remarks
The WYDOT Pilot aimed at creating a connected rural freeway environment to promote traffic safety and mobility along a 402mile I-80 freeway corridor in Wyoming. Evaluating the effectiveness and benefits of WYDOT CV system is crucial for improving the system performance (Raddaoui and Ahmed, 2019). In this regard, this paper first discussed the 21 specific performance measures contained within 8 performance categories, which represent the major expected benefits and outcomes of the WYDOT Pilot to guide the traffic and system performance data collection activities during the postdeployment stage. These benefits are focused on major improvements in safety, mobility and agency efficiency. Such as improved speed adherence and reduced speed variation will compare the vehicle speeds versus posted speed, and reduced vehicle crashes will measure the reduction in vehicle crashes and crash rates during pre-and post-CV-deployment periods. Road weather conditions reports will measure the quantity, coverage and latency of road weather condition reports from the WYDOT field maintenance forces. Improved Information to commercial vehicle fleet managers will measure fleet manager's satisfaction with TMC information and actions taken due to receipt of that information.
In addition, this paper presented the data collection approaches and analytical methods that have been established for the real-life deployment of the WYDOT CV applications. Five methodologies for assessing the operational and safetyrelated performance measures were proposed. Analyses were conducted on data collected during the baseline period, and pre-deployment conditions were established for eleven performance measures. Additionally, microsimulation modeling were recommended to aid in evaluating the mobility and safety benefits of the WYDOT CV system, particularly, when evaluating system performance under various CV penetration rates and/or CV strategies. The proposed performance evaluation framework can guide other researchers and practitioners identifying the best performance measures and evaluation methodologies when conducting similar research activities.
Currently, the University of Wyoming research team conducted a driving simulator study to provide early insights about the impacts of CV warnings on driver behavior (Raddaoui et al., 2019;Yang et al., 2019). It was concluded that both average and variance of speed under CV scenarios were significantly lower than baseline scenario. These effects of CV warnings can bring potential safety benefits, since a lower mean and variance of speed may help in reducing the risk of crashes under adverse weather conditions.
In addition to the potential direct benefits such as improving mobility and safety, another advantage of CV technology is it can assist transportation managers to develop efficient and cost-effective traffic management strategies for various traffic and weather events in a timely manner. Examples of this potential include but not limited to: development of weatherresponsive variable speed limit (VSL) algorithms (Hammit et al., 2017), real-time identification of traffic operation status (Fountoulakis et al., 2017), and identification of high-risk locations (Xie et al., 2019). Nevertheless, in reality there is a trade-off between mobility and safety benefits (Tian et al., 2018), such as a lower VSL tends to reduce the risk of traffic crash, while it will also bring a longer delay. This further emphasized the significance of using microsimulation modeling for performance assessment, since the market penetration rate of CV should be large enough to ensure that sufficient data will be collected to support for the decision-making process for transportation operators at the transportation management center.