Decision Making and Underperformance in Competitive Environments: Evidence from the National Hockey League

We find evidence of suboptimal decisions leading to underperformance in a policy experiment where two teams of professionals compete in a tournament (National Hockey League shootout) performing a task (penalty shot) sequentially. Before an exogenous policy change, home teams had to perform the task second in the sequence. After the policy change, home teams were given the choice to lead or to follow in the sequence. Home teams should move first only when this is optimal, and this should lead them to winning the tournament more often. We find that after given the choice, home teams most of the time choose to move first in the sequence, and this results in a lower winning frequency for them. Contrary to what economic theory would predict, we find that an expanded choice set can lead to worse outcomes for the agents.

change. Before the exogenous policy change, in the season 2005/06, the NHL rules of the game stipulated that the away teams had to always start the shootout, and perform the task first in the sequence. After the policy change, in the seasons from 2006/07 to 2011/12 home teams were given the choice to decide who will start the shootout. This exogenous expansion of the choice set of home teams allows us to study decision making and performance.
These data are interesting because they feature professionals that are familiar with the tournament and the task, and that have substantial incentives to excel. The shootout was introduced in the 2005/06 season as a method of determining the winning team if a match is still drawn following overtime. In a shootout both teams take penalty shots until a winner is determined, while the opposing team's goalkeeper tries to stop the penalty shots from being scored. As a method of determining the winner of a game, the shootout has been widely used in other North-American and International competitions. The penalty shot task is used during regular time to punish specific infractions. Furthermore our dataset is fully comprehensive and includes all the shootouts in the NHL in a short period of time. Therefore we have a tournament that is repeated a substantial number of times by the same highly skilled professionals with incentives to excel. 1 The most interesting aspect of our policy experiment is that after the policy change home teams enjoy an expanded choice set. If home teams are taking the rational decision, they should move first only when this is optimal, and this should lead them to win the tournament more often after the policy change.
The main finding of this paper is that in fact after given the choice, home teams most of the time move first in the sequence, and this results in lower winning frequency for them. In other words, we find that a larger choice set causally leads to different choices and to worse performance. Therefore, a greatly expanded choice set leads to worse outcomes for home teams.
The second finding of this paper is that it matters whether home teams start the shootout or not. We can reject the null hypothesis that the order does not matter for winning the shootout. Unfortunately, our data does not have randomization on the sequence order, therefore we can not attribute our findings of decreased performance purely to the sequence order. Other factors could be driving both the choice 1. We argue that strategic considerations about letting the other team win play an insignificant role in our data. One potential concern would be that teams might be interested in deliberately losing games as soon as they are out of the playoff race. The rationale behind this strategy would be to benefit from a better position in the draft of new players entering the League in the next season. The order of the draft pick is determined by a combination of last season's standing and a lottery system. For example, the last team in the table has a 25% probability of gaining the right to the first pick. Although this is still the highest possible probability of getting the first pick in the draft, it reduces the benefits from losing on purpose. More importantly, if a team wanted to lose on purpose it would never wait for the shootout. The reason is that a loss following regular time is still rewarded with a point, while a loss in regular time is not rewarded with any points. In any case, the robustness checks described in Section IV show that the issue of losing on purpose is not a reason for concern in our NHL data. of whether to start the shootout and the outcome of the shootout. E.g., it could be a selection effect -weaker teams choose to start and lose more often. In face of these selection problems, in the remaining of the paper we explore different explanations and highlight which explanations are consistent with the data. Our results are consistent with an explanation based on overconfidence in a setting where there is psychological pressure from lagging behind in the tournament. In a seminal paper Svenson (1981) documented that 93% of a US sample and 69% of a Swedish sample of drivers believe to be more skillful drivers than the median driver. Park and Santos-Pinto (2010) show that agents are overconfident about their own ability in poker and chess tournaments. Apesteguia and Palacios-Huerta (2010) document an advantage of shooting first in the context of soccer penalty shootouts. In their random assignment data, teams starting the shootout win with 60% probability. This result has been challenged by Kocher, Lenz, and Sutter (2012) using a different dataset and by Feri, Innocenti and Pin (2013) in a field experiment. 2 Using survey data, Apesteguia and Palacios-Huerta (2010) also document that coaches and managers prefer to go first in the shootout, with the explicit objective of putting opponents under pressure.
We argue that in hockey shootouts there is an advantage of shooting second. In soccer, shooting first is nearly identical to scoring first, as the unconditional probability of scoring a goal from a penalty kick is very high, about 72%. 3 In hockey however, there is an important distinction between shooting first and scoring first, as the chance of realizing a goal from a penalty shot is relatively low, about 33% (see Table 1). If psychological pressure from 2. We discuss how these results relate to our interpretation in Section V. 3. Calculated from the data Apesteguia and Palacios-Huerta (2010) provided for replication at http:// www.aeaweb.org/articles.php?doi=10.1257/aer.100.5.2548. lagging in the partial score is present in hockey shootouts, the advantage should fall on the team that shoots second. But if home teams are overconfident regarding their ability to score first, they may start the shootout too often with the objective of getting the opponent's shooter under pressure. Once they fail to materialize this advantage, this releases the opponent's team from psychological pressure and leads home teams starting the shootout to underperform. We look at shot by shot data and study shooter performance conditional on the shooter's team being in advantage, disadvantage or in a situation of potential advantage in the shootout. The first two are defined as a positive and a negative partial score, respectively. We define potential advantage as a situation where the shootout is tied, but the shooter's team has one more shot to take than the goalie's team. We find no significant effects on the probability of scoring from being in disadvantage. We do find statistically significant positive effects on the probability of scoring from being in advantage in the shootout -which could justify the desire to move first. We also find statistically significant positive coefficients from being in a situation of potential advantage. These correlations are consistent with the explanation we propose.
Finally, we consider a number of other potential explanations for our data: negative selection in terms of quality of home teams that get to the shootout, choice inducing choking, losing on purpose and learning. We can reject that home teams getting to the shootout are of less observable quality relative to away teams after the policy change. We find no direct evidence of increased choking for home team shooters following the policy change, nor evidence that losing on purpose is driving our results. Regarding learning, if we drop the first two seasons following the policy change, when the probability of starting the shootout is around 65%, our results are even stronger.
We proceed as follows. Section II describes what a shootout is, and our data. Section III describes our test of rationality. The results of our analysis of the NHL policy experiment, as well as the analysis of potential explanations are in Section IV. Section V concludes.

II.1. The shootout and the penalty shot
The shootout in the NHL consists of a series of three alternating shots per team. The team with a higher score following these three shots is the winner. If one team scores the first two shots, and the other team misses the first two shots, the third shot is not performed as it would not change the outcome of the shootout.
If after each team has taken three shots the score remains tied, the shootout proceeds to a "sudden death" format. In the "sudden death" rounds, if one team scores and the other team does not, the team that scored wins and this completes the shootout. If both teams do not score or if both teams do score in the "sudden death" round, they proceed to the next "sudden death" round, and this process in principle can continue ad infinitum.
The coach of each team determines the shooters and the order they shoot. No player may shoot twice before everyone eligible has shot. The rules state that the home team has the choice of shooting first or second (NHL Official Rules, 2011). The exception is season 2005/06, when the home team always had to shoot second (NHL Official Rules, 2005).
The procedure for a penalty shot is exactly the same as the procedure for a long standing foul in hockey during regular time. Each penalty shot involves two players: a shooter and a goalkeeper. The shooter takes the puck from the center face-off spot and attempts to score while keeping the puck in motion, while the goalkeeper attempts to cover his goal from the puck. Each penalty shot takes around 3 to 5 seconds to be completed. It is a well defined task with immediately observed outcomes. 4

II.2. Data
We use data from the National Hockey League (www.nhl.com). The dataset comprises 1138 shootouts over seven seasons. Table 1 summarizes the data on shootouts and games. What we can gather from Table 1 is that shootouts are very important in hockey, as a large fraction of games are decided through a shootout. We can also observe that the unconditional probability of being successful in a penalty shot is about 33%.
There are two main advantages of using this dataset relative to other tournaments data. First, we have 1138 shootouts, and this is considerably larger than previous studies using sports natural, quasi-natural and policy experiments. Second, this dataset is fully comprehensive in that it includes all the shootouts in the NHL to date. Because the data spans a relatively short time period -only 7 years -a large proportion of the agents involved in the shootouts does not change in the data. This means we have a tournament, the shootout, that is repeated a 4. The Penalty shot rules: . . . place the puck on the center face-off spot and the player taking the shot will, on the instruction of the Referee (by blowing his whistle), play the puck from there and shall attempt to score on the goalkeeper. The puck must be kept in motion towards the opponents goal line and once it is shot, the play shall be considered complete. No goal can be scored on a rebound of any kind (an exception being the puck off the goal post or crossbar, then the goalkeeper and then directly into the goal), and any time the puck crosses the goal line or comes to a complete stop, the shot shall be considered complete.

DECISION MAKING AND UNDERPERFORMANCE IN COMPETITIVE ENVIRONMENTS
large number of times by the same highly skilled professionals, that have incentives to excel. 5

III. SETTING UP THE TEST OF RATIONALITY
In this section we describe the test of rationality. Let us denote by pb the probability of a home team winning the tournament before the policy change when NHL regulation stipulated that home teams had to shoot second. Let us denote by pa the probability of a home team winning the tournament after the policy change when NHL regulation gave home teams the right to choose whether to start or not. As discussed in the introduction, the expansion of the home team choice set implies that the null hypothesis of rationality versus the alternative of irrational home team behavior after the policy change is where the equality pa = pb holds if and only if the choice of whether to start a shootout or not is irrelevant for winning the tournament. In words, if home teams act rationally, they are more likely to win the shootout after the policy change when they enjoy expanded choice set. If home teams act irrationally, they might make themselves worse off under the expanded choice set after the policy change, because they might choose a suboptimal option (to start first in the shootout when this is undesirable) which was not available before the policy change.
The null hypothesis of rationality is a composite null hypothesis. Standard statistical hypothesis testing is ill equipped to handle such situations. Therefore we will proceed as is customary in the literature by considering a worse scenario and thereby testing : : To this end, let the random variable W be the number of times the home team wins. Let N be the number of observations, Wo be the observed number of times the home team has won the shootout after the policy change, and pa = pb under ′ H o , be the probability of winning a shootout after the policy change. Then .
gives us the probability of observing as bad as documented in the first row of Table 2 or worse performance of home teams after the policy change under ′ H o . As the binomial cumulative distribution function in the above expression is decreasing in pb this will in turn give us a conservative upper bound on Pr[W ≤ Wo] under Ho, i.e., it will give us a conservative test of rationality of home team behavior. Table 2 collects the results of this test in column 4.
In column 5 of Table 2 we present the results of a two-sided binomial test for the null hypothesis that the probability of a home team winning is 50%. Note that given that we do not have random assignment of the shootout order there is no reason a priori to expect the winning probability to be 50%. Yet the probability of a home team winning equal to 50% is an interesting no effect null hypothesis.

IV.1. Rationality test
Focus first on the row "Seasons 2006/07-2011/12" of Table 2. This is the test of rationality carried out on all seasons where the home team has the choice of : holds true, is very low, only 0.001. In other words, we are able to reject the null hypothesis of rationality Ho : pa ≥ pb at least at the 0.1% significance level. Remember that 0.001 is a conservative upper bound for the p-value of the test Ho : pa ≥ pb versus HA : pa < pb.
In the last two rows of Table 2 we decompose home teams according to their choice of whether to shoot first in the shootout. Table 1 shows that home teams overwhelmingly choose to shoot first: they shoot first 76% of the times they are allowed to choose. We find that the winning probability of a home team that shoots first is 47.68%. This is smaller than the winning probability in the season when home teams were forced to shoot second of 51.72%, and the difference is significant at the 1.4% significance level. Perhaps surprisingly, home teams deciding to shoot second do even worse: they win only 43.7% of the shootouts, and this is significantly smaller than 51.72% at the 1% significance level. 6

IV.2. No effect test
Turn now to column 5 of Table 2. Our results for the first season, a sample of 145 shootouts without selection, appear to support the view that there is no significant difference between shooting first or second. However our results for the rest of the seasons where the home teams have the choice to start or not, show that the null hypothesis that the home team winning probability is equal to 50% can be rejected at the 4% significance level using a two-sided binomial test, i.e., the two sided p-value of the No Effect Test is 0.04.

IV.3. Potential explanations
In this subsection we explore some possible explanations for the results documented in the previous subsections. Our exploration here is not causal, given the non-randomized nature of our data. We propose in Section I that overconfidence in the presence of psychological pressure can be an explanation for our results. We highlight that in hockey it is hard to score a penalty shot, and a team going first will find itself frequently failing to materialize the shot, thus releasing the 6. Following the policy change only a small fraction of home teams choose to shoot second. This result could reflect selection on the teams choosing not to start the shootout. Another explanation would be the existence of heterogeneity in the incidence of psychological effects in competitive environments, as documented by Feri et al. (2013) in a field experiment. If a large mass of home teams that would win the shootout going second, decide instead to start the shootout and lose, we should observe a decrease in the probability of home teams winning in both sets of teams. opponent team from psychological pressure. We can use our data to see if these effects are present. We also investigate explanations based on home team quality, choice induced choking and learning.

IV.3.1. Overconfidence and psychological pressure
To investigate this explanation we explore shot-level data. We model the probability of scoring a penalty shot as a function of a full set of dummies that capture whether the shooter's team is in Advantage (A), Disadvantage (DA), or Tied in the shootout at the moment of the shot. We also divide Ties into two situations: one where the number of shots taken by both teams prior to the current shot is the same (Neuter), and one where the number of shots taken by the opponent's team is larger (Potential Advantage). The latter situation is interesting because it comprises a situation of Potential Advantage (PA) where a team following in the sequence can benefit from an opponent's miss. To control for pressure associated with game deciding shots, we divide shot situations into game deciding shots (DS) and non-game deciding shots (NDS). We define game deciding shots as shots for which the outcome decides the game immediately. We control also for whether the shot was taken by a shooter from the home team or by a shooter from an away team, in each of the categories described before. Finally, for each specification, we add round and shooter fixed effects. Round fixed effects remove the time invariant effect of a shot being taken at a specific round, where round #1 is defined as shots 1 and 2, round #2 as shots 3 and 4, and so on and so forth. Shooter fixed effects control for the time invariant effect of a shot being taken by a specific individual. Controlling for these is important to capture the effect of later rounds and shooter ability on the probability of scoring. Table 3 collects the results of the linear probability model for different regression specifications on the full sample of shots. In the Supplementary Information (SI), we present results for the Logit and Probit models with the same specifications as in Table 3, and the results are similar. In all of them the dependent variable corresponds to a dummy variable that takes the value of 1 if the outcome of the penalty shot is a goal, and 0 otherwise. In column (1) we focus on the categories that capture the partial result of the shootout: Advantage, Disadvantage and Potential Advantage. The omitted category is ties where teams have taken the same number of shots, referred as the Neuter category. Relative to the omitted category, we can see that being in Advantage is related to a higher probability of scoring the shot, significant at the 5% level. All other categories are not significantly different from the omitted category. In column (4) we add round and shooter fixed effects, and the results are similar. (The omitted category in column (4) is ties where teams have taken the same number of shots (the Neuter category) in the first round.) In column (2) we divide the categories defined above between Deciding shots (DS) and Non-deciding shots (NDS). Note that we do not include the dummy for non-game deciding shot directly. This is because the omitted category of ties with the same number of shots is by definition a non-game deciding shot. When we divide shots between deciding and non-deciding we observe that being in advantage in a non-deciding shot is associated with 6.4 percentage points higher probability of scoring relative to the omitted category, and this is significant at the 10% level. Interestingly, being in Advantage in decisive shots is not associated with a higher probability of scoring relative to the omitted category. We observe also that potential advantage in non-deciding shots is now associated with a 3.8 percentage points larger probability of scoring relative to the omitted category. Once again, being in disadvantage is not associated with the probability of scoring relative to the omitted category. As expected, the categories for game deciding shots are associated with lower probabilities of scoring the shot. In column (5) we add round and shooter fixed effects, and the results are similar.

DECISION MAKING AND UNDERPERFORMANCE IN COMPETITIVE ENVIRONMENTS
Finally, we investigate how these relationships depend on whether a shooter belongs to the Home Team (HT). Column (3) includes a dummy capturing if the shooter is from the home team, and the omitted category is now Neuter, and the shot is taken by a shooter from the Away Team (AT). Interestingly, the positive associations between scoring and being in advantage and potential advantage for non-game deciding shots hold only for home team shooters. Column (6) shows that the result is robust to the introduction of shooter and round fixed effects. This result suggests that releasing a shooter from pressure has a larger effect on shooters that belong to the home teams, a result that is consistent with Dohmen (2008), who shows that soccer shooters from home teams choke under friendly pressure.

IV.3.2. Quality, choice induced choking, losing on purpose and learning
Another explanation we consider is whether following the policy change the quality of teams getting to the shootout changed. This could be a consequence of different strategies prior to the shootout. Given that home teams can now choose whether to take the penalty shot first or second, home teams of lower quality could try to take the game into the shootout to benefit from this advantage. If the lower home team quality effect still dominates any potential advantage from choosing the order in the shootout, we would observe a decrease in the performance of home teams, independently of their choice. Although team quality is ultimately unobservable, we can use their overall league table standing prior to the shootout as a proxy, and look for a decrease in the quality of home teams following the policy change. In fact, we find a significant increase in the mean quality of home teams, from 17th place to 14th place (mean classifications rounded, lower values mean higher classification; two-sample Wilcoxon ranksum (Mann-Whitney) test: p-value = 0.0043). Interestingly, we find also an increase in the average quality of away teams getting to the shootout, from 16th place to 14th place (mean classifications rounded, two-sample Wilcoxon ranksum (Mann-Whitney) test: p-value = 0.03). If we look at the difference between home and away teams' classification, there is no significant difference following the policy change (two-sample Wilcoxon rank-sum (Mann-Whitney) test: p-value = 0.5513). 7 We also directly test for choking under friendly pressure. Dohmen (2008) finds that in German soccer penalties shooters from the home team are more likely to choke, where choking is defined as failing to score a penalty shot because the shot fails to hit the goal. This represents a situation where, arguably, the intervention of the goalie was immaterial. If the choice of whether to go first or second puts players from the home teams under differential friendly pressure, this could explain our results of reduced performance for home teams. If these effects are present in our data, the policy change should have led to an increase in the relationship between choking and having a shooter from the home team.
We test the above hypothesis in Table 4. All shooters choke more in the choice regime after the policy change when home teams decide who starts the shootout, relative to the first no-choice season. However, home team shooters do not choke more, relative to away team shooters. More importantly, the hypothesis that decreased home teams performance after the policy change is due to increased home team choking after the policy change is refuted by the data. The interaction term between choice regime (after policy change seasons) and home team shooter dummies is negative and insignificant. Therefore if anything, home team shooters choke less relative to away team shooters in the choice regime seasons compared to how much they choked relative to away team shooters in the no-choice seasons.
In Footnote 1 we discussed the validity of our data as a tournament model, where both teams have full incentives to succeed. There are some incentives for teams not making it to the playoffs to lose games on purpose, which could bias our results. As a robustness check, we drop the last month of regular seasons (75 shootouts), as well as the last two months (293 shootouts) and redo the rationality test presented in Table 1. Our results are in line with previous results for the 7. We can also use this quality measure to study if there is observable heterogeneity of home teams choosing or not choosing to start the shootout. We find no significant difference in quality between home teams choosing to go first or to go second (two-sample Wilcoxon rank-sum (Mann-Whitney) test: p-value = 0.3302). The same is true for away teams (two-sample Wilcoxon rank-sum (Mann-Whitney) test: p-value = 0.3592) -there is no significant difference in observable quality between away teams that have been chosen to shoot first, and away teams that have been chosen to shoot second.
full sample, and strongly significant. 8 We also drop observations in the last month of shootouts involving teams that ultimately did not make it to the playoffs, and results are unchanged. 9 Finally, we discuss learning. Looking at Table 1 it is possible to see that the first two seasons following the introduction of the choice for home teams are also the seasons with a closer split between home teams going first and home teams going second. Interestingly, they are also among the seasons with a higher winning frequency for home teams. In fact, our rationality test is not rejected for the first two seasons with choice, only for the latter seasons. Following these two seasons home teams choose to start much more often, and performance significantly decreases. Figure 1 plots winning and starting frequencies for home teams by month for each season. The main thing to note in this figure is that there is no 8. Omitting the last month of the regular season, the winning probability for home teams in the first season is 50.0%, compared to 46.5% in the seasons with choice. This probability is different at the 5% level: one sided p-value = 0.018. Omitting the last two months these probabilities are 50.5% and 45.1%, respectively, and they are different at the 1% level: one sided p-value= 0.002. 9. More concretely, we redo the test omitting the shootouts played in the last month of the season that involve a home or away team that ultimately did not make it to the playoff. Our results are unchanged and significant at the 1% level. The winning probability for home teams in the first season is 50.4%, compared to 46.5% in the seasons with choice. This probability is different at the 5% level: one sided p-value = 0.009. clear trend in the winning probabilities for home teams within the season when the shootout was introduced (2005-06) nor for both winning probabilities and the probability to start the shootout within the season when the choice was introduced (2006-07). This suggests that learning by home teams did not play a major role in these two seasons.

V. DISCUSSION AND CONCLUSION
In this paper we study a policy experiment that resulted from a change in the National Hockey League (NHL) rules regulating shootouts. In the first season in which shootouts were introduced as a means to determine the winner in case the hockey match is still drawn after overtime, home teams had to shoot second. In the rest of the seasons, the NHL changed the rule and gave home teams the right to choose whether to start first or second. We use this policy experiment to devise a test of whether home teams behave rationally. We are able to reject the null hypothesis that home teams act rationally at least at the 0.1% significance level. We are also able to reject the null hypothesis that the home teams winning probability in the seasons when they choose whether to start or not is equal to 50% at the 4% significance level, versus the two sided alternative that the home teams winning probability is different from 50%.
We discuss one interpretation of our findings: overconfidence leading home teams to start too often in an environment where psychological pressure from lagging behind is present. Contrary to what is argued in Apesteguia and Palacios-Huerta (2010), our results suggest that agents are not perfectly aware of the interaction between psychological pressure and low scoring probabilities (in hockey the unconditional probability of scoring penalty shot is 33%, versus 72% in soccer) and do not respond optimally to it. Due to the nature of our data, the exploration of the mechanisms behind our findings is not causal. Understanding the relationship between tasks with different success probabilities, decision making, psychological pressure and performance in dynamic tournaments remains an exciting field for empirical and experimental work.
Finally, our paper also relates to the existence of negative psychological pressure from lagging behind in competitive environments. Recent work by Kocher et al. (2012) and Feri et al. (2013) questions the existence of this phenomenon. Kocher et al. (2012) use a super-sample of the Apesteguia and Palacios-Huerta (2010) soccer data to show that there is no statistically significant difference between shooting first or second in soccer. Feri et al. (2013) use data from a field experiment where they can control for individual heterogeneity. They find that lagging behind does not affect negatively second-mover's scoring probability. They show that there is a positive effect as the second-mover's scoring probability improves significantly when free throws are worthy, with heterogeneity playing a role.
Our results for the first season, a sample of 145 shootouts without selection, appear to support the view that there is no significant difference between shooting first or second. However our results for the rest of the seasons where the home teams have the choice to start or not, show that the null hypothesis that the home team winning probability is equal to 50% can be rejected at the 4% significance level. Therefore, our interpretation of our results is that first, in hockey it matters whether a team starts the shootout or not. Second, in hockey home teams do not understand correctly the implications of starting first and behave irrationally, thereby performing worse when given the choice of starting first when compared to a situation when they were not given this choice.

SUPPORTING INFORMATION
Additional Supporting Information may be found in the online version of this article at the publisher's web-site: Table SI1. Shot by shot analysis -Logit conditional marginal effects Table SI2. Shot by shot analysis -Probit conditional marginal effects SUMMARY We find evidence of suboptimal decisions leading to underperformance in a policy experiment where two teams of professionals compete in a tournament (National Hockey League shootout) performing a task (penalty shot) sequentially. Before an exogenous policy change, home teams had to perform the task second in the sequence. After the policy change, home teams were given the choice to lead or to follow in the sequence. Home teams should move first only when this is optimal, and this should lead them to winning the tournament more often. We find that after given the choice, home teams most of the time choose to move first in the sequence, and this results in a lower winning frequency for them. Contrary to what economic theory would predict, we find that an expanded choice set can lead to worse outcomes for the agents.