Disentangling the impact of mean reversion in estimating policy response with dynamic panels

2.1 Policy setting

The mechanism provides an incentive proportional to measured quality and has been applied to discharges in the inpatient prospective payment system at acute-care Medicare hospitals since 2013.^[3] The scheme reduced Medicare’s base payment^[4] to each hospital by a factor α t which equaled 0.01 in 2013. The amount of the reduction was increased annually by 0.0025 in 2014–2017 and has remained flat at 0.02 since 2017. Note that α is the parameter of the reform intensity, varying over time, and α = 0 would correspond to a counterfactual setting with the absence of the reform.

The accumulated saving from reduction in base payment is redistributed across hospitals according to an adjustment coefficient, which is computed as a linear function of the composite quality measure: 1 + κ t m i t 100 − 1 ⋅ α t , where i is the index of a hospital, t indicates year, and m i t is the hospital’s total performance score (TPS), ( 0 ≤ m i t ≤ 100 ). A hospital is rewarded in period t + 2 if the adjustment coefficient based on m i t is above one and is penalized otherwise. The quality incentive scheme is budget-neutral and the value of the slope κ t is chosen to ensure budget neutrality, so that hospitals with value of T P S above the empirical mean gained under the reform. In the first years of the reform κ t was close to 2, so hospitals with value of the composite quality measure above 50 were winners from the incentive scheme.

The TPS is a weighted sum of scores for measures in several domains: timely implementation of recommended medical interventions (clinical process of care), quality of healthcare as perceived by patients (patient experience of care), survival rates for AMI, heart failure and pneumonia patients and other proxies for outcome of care, healthcare-associated infections and other measures of safety of care, and spending per beneficiary as a measure of efficiency of care.^[5]

A hospital’s intertemporal incentive in Medicare’s scheme is based on the expectation that the quality payments will continue over a long term, so the hospital’s executives and physicians realize that demand is proportionate to quality and that their current policies toward quality of care will influence future reimbursement [46,73].

2.2 Autoregressive process and quality convergence

The evolution of the measured quality constitutes a process with serial correlation. If the process for the measured quality is stationary, then it may be treated as an autoregressive process m t − μ ( θ ) = ϕ 1 ( m t − 1 − μ ( θ ) ) + ⋯ + ϕ p ( m t − p − μ ( θ ) ) + ε t . Here μ ( θ ) = E ( m t ∣ θ ) denotes the mean value of the measured quality for a hospital with type θ . As the absolute values of the reciprocals of the roots of the characteristic equation of AR( p ) processes are less than one, the maximum absolute value of these reciprocals (denoted λ ) may be used as the measure of persistence for the process of measured quality [74].

Using definitions in [29], we can disentangle a permanent component in m t , which is related to economic impact of pay-for-performance from a transient component (a pure dynamic effect), which may be referred to as “mean reversion” or “regression toward the mean” [30].

The reason for the phenomenon of mean reversion is the existence of the random error ε t in the measured quality m t . Indeed, in the absence of ε t the process quickly converges to its mean μ ( θ ) and does not exhibit mean reversion because it always sits at the mean. The random error in the measured quality is largely attributed to imprecision in quality measurement: it is hard to reveal true quality using observable proxies. Another reason is random variation in true quality, which may be explained by the fact that patients do not always comply with the prescribed treatment [62]. Combined with the fact that hospitals make an intertemporal decision in respect of the quality-based reimbursement, the random error leads to the autoregressive form of measured quality m t .

The autoregressive specification can be taken as equivalent to convergence of the measured quality toward the value μ ( θ ) and λ is associated with the speed of quality convergence. The persistence parameter λ essentially describes how quickly the effect of any unexpected shock in value of the dependent fades over time. For example, consider a simple AR(1) process with 0 < λ < 1 and the conditional mean E ( m t ∣ m t − 1 , θ ) = μ ( θ ) + λ ( m t − 1 − μ ( θ ) ) . Here the expected value of current measured quality E m t is closer to the mean value μ ( θ ) than is the value of the measured quality in the previous period, i.e., m t − 1 . The expression for E ( m t ∣ m t − 1 , θ ) becomes more complicated for AR( p ) processes with p > 1 , but λ can still be used as a measure of persistence of the process.

The hospital receives higher profits for improvement of performance under higher values of α than under lower values of α . This, combined with the serial correlation between performance in consecutive periods, implies a direct association between the persistence parameter λ and α . Higher values of λ imply a lower rate of convergence of quality and hence a weaker effect of mean reversion.

2.3 Expected outcomes of the reform and time profiles of the quality measure

3.1 Specification

The dependent variable y i t is the TPS of hospital i in year t . The value of y i t is used for remuneration of Medicare hospitals at time t + 2 , so we employ the second-order dynamic panel,^[6]

where z i t are hospital time-varying characteristics, u i are individual hospital effects (in particular, they incorporate the altruistic effects), the size of quality incentives α t varies in different years and enters the equation multiplied by the share of Medicare discharges s i t , which indicates that the quality incentives apply only to treatment of Medicare patients, and d t is a set of dummy variables which capture external time effects (effects unrelated to hospital decisions). The following restrictions are used to identify the constant term ϕ 0 : the sum of the coefficients for components of d t is normalized to zero, and the expected value E ( u i ) = 0 . Hospital time-varying characteristics are disproportionate share index, casemix index, number of hospital beds,^[7] physician-to-bed ratio, and nurse-to-bed ratio. The posterior analysis of the effect of quality incentives deals with hospital grouping according to the time-invariant characteristics, which could not be incorporated in the empirical specification with fixed effects: geographic region where the hospital is located, public ownership, urban location, and teaching status.

We use two hospital control variables which affect quality improvement and allow us to mitigate potential biases, which might occur if the pay-for-performance effect is identified based only on the variation of α in time. The HRRP penalty captures the impact of a simultaneously adopted incentive program with similar incentives. Moreover, the readmission reduction program targets improvement of quality measures which are components of TPS.^[8] The binary variable for successful attestation of meaningful usage of EHR accounts for the effect of another compulsory program, which provides bonuses to attested hospitals. The variable controls for the fixed cost incurred by a hospital to improve its quality through installing and using health information technology systems.

Eq. (1) can be estimated using the generalized method of moments: the [2] and [12] methodology for dynamic panel data. Examples of use of the methodology in health economics include analysis of the quality of care at Medicare’s hospitals in [56], study of the length of stay at Japanese hospitals in [10], investigation of labor supply by Norwegian physicians in [4], and of health status of individuals in the US in [57].

The first set of moment conditions in GMM comes from the approach of [2] and [12]. We take the first difference of the right-hand side and left-hand side of Eq. (1):

Since ε i t cannot be predicted using the information available at period t − 1 , ε i t is uncorrelated with any variable known at time t − 1 , t − 2 etc. Therefore, Δ ε i t is uncorrelated with any variable known at time t − 2 , t − 3 etc. Hence, the following set of moment conditions can be imposed to estimate the model parameters in Eq. (2), see [2] and [12]:

where e i t is the regression residual and Z i t is any variable known at time t .^[9]

Another set of moment conditions comes from [12] for the level Eq. (1): u i + ε i t has to be uncorrelated with Δ Z i t − 1 for any stationary variable Z i t , where Z i t − 1 is known at time t − 1 .

So Z i t includes lagged values of predetermined and endogenous variables (the first set of moment conditions) and differenced predetermined and endogenous variables (the second set of the moment conditions). All moment conditions are formulated separately for different years, so the number of observations for asymptotics equals the number of hospitals.^[10]

More specifically, lagged value of TPS and other hospital control variables in z i t (beds, physician-to-bed and nurse-to-bed ratios, HRRP penalty, and the binary variable for hospital EHR attestation) are taken as predetermined and do not require the use of instruments in estimations. Casemix and the disproportionate share index are assumed to be endogenous: we rely on the empirical evidence of manipulation by hospitals of patient diagnoses (i.e., with casemix) and reluctance to admit low-income patients under quality-incentive schemes [17,23,28]. We assume that the Medicare share is endogenous, too: the fact may be explained by demand-side response from Medicare patients to publicly reported hospital quality [44,53,72].

It should be noted that the use of dynamic panel data methodology requires justification on economic grounds. This is because the approach uses lags and lagged differences as instruments, and there are potential problems with using lags as instruments even though they pass the Arrelano-Bond tests. Specifically, lags may prove to be weak and invalid [7]: the weakness may occur when lags are distant [59], and invalidity happens due to overfitting of the endogenous variable under large T [66]. However, neither of these problems (weakness and invalidity) are likely to be present in our analysis since we restrict our instruments only by the first appropriate lag.

The validity of instruments is assessed through statistics of the Arellano-Bond test. We employ [80] robust standard errors for estimation.^[11] But formal tests are insufficient for establishing the causal relationship in models, which use an instrumental variable approach [1,7]. Accordingly, it is necessary to provide an economic justification for the assumption of the exclusion restriction of the instruments, i.e., that the instruments are exogenous and impact the dependent variable through no channels other than the endogenous variable and, possibly, also through exogenous covariates. An example of such justification on theoretical grounds can be found in [6], who uses lags of GDP and lags of the inflation rate as instruments for GDP and inflation. Another way of arguing for the exclusion restriction is given in [38], which estimates per capita output in various countries as a function of social infrastructure. Owing to endogeneity of social infrastructure, variables related to exposure to Western culture are used as instruments, and there is a discussion of the absence of any direct channels through which these variables could impact a country’s per capita output.

We follow the latter approach to provide an economic justification for the validity of instruments in the dynamic panel data model for the composite quality measure at Medicare hospitals. Our arguments below, which advocate the applicability of lagged first differences as instruments for the level Eq. (1) and first lagged levels as instruments for the difference Eq. (2), are based on the plausible assumption of a short adjustment period in the values of the dependent variable. Specifically, we assume that hospital managers take prompt action upon learning the TPS in year t , so that adjustment is observed in the next period and is not delayed until a more remote future. This assumption is supported by interviews with hospital managers [21,46,37,55,73,77], which show real-time assessment of performance of hospital personnel and immediate feedback initiatives aimed at correcting possible lack of quality. For instance, at Medicare hospitals which participated in the pilot pay-for-performance program, “progress reports were routinely delivered to hospital leadership and regional boards” ([37], p. 45S). Hospital-specific and physician-specific compliance reports were collected at least every 1.5 months on average, and the results of these reports were delivered to individual physicians once in 5 months on average at both top-performing and bottom-performing hospitals ([77], Table 4, p. 837). As regards nationwide implementation of pay-for-performance at Medicare hospitals, the TPS is calculated annually, but values for the quality dimensions of the TPS are made publicly available on a quarterly basis.^[12] Frequent announcements of quality scores make it possible to expedite quality adjustment at each hospital and improve the value of the TPS within a year. For instance, the survey of hospital CEOs, physicians, nurses, and board members showed that, since implementation of the value-based purchasing program, “data were shared with their board and discussed at least quarterly with senior leadership” ([55], p. 435).

As regards our formal analysis, Eq. (1) has TPS as a dependent variable and its first and second lags as explanatory variables. Δ y t − 1 is used as an instrument for y t − 1 . We assume that the change in TPS from period t − 2 to t − 1 , i.e., Δ y t − 1 , which is observed at a hospital at t − 1 , is immediately followed by the hospital’s action in period t − 1 . So the instrument Δ y t − 1 affect the dependent variable y t through the endogenous variable y t − 1 , i.e., through improved quality in period t − 1 (and potentially also through the predetermined variable y t − 2 , i.e., quality adjustment may start as early as in period t − 2 ) but not through other channels. Without the short adjustment period, these other channels might have included some postponed effects which only come into effect in period t . Note that the equation has hospital control variables, and we follow the empirical literature on the US Medicare reform by treating some of them as endogenous. One of such variables, the share of Medicare patients, reflects the desire of the regulator to sign contracts with the hospital to treat Medicare patients, and it is a function of the hospital’s quality enhancing efforts [46]. Our empirical strategy relies on the fact that Δ x t − 1 is an excludable instrument for x t . It is, indeed, plausible to assume that increase of quality efforts from period t − 2 to period t − 1 results in positive value of Δ s t − 1 (where s t denotes the share of Medicare patients) and impacts the value of the TPS in the period t . A similar argument applies to another endogenous control variable – casemix – which reflects the share of patients with complicated diagnoses. If we ignore potential dumping of patients by hospitals, hospitals are interested in treating patients with complicated diagnoses, since compensation in the system of diagnosis-related groups is higher for severe cases. But patient demand responds to public reports on hospital quality [20,42,44], so the share of Medicare cases becomes a function of hospital quality.

Another equation is (2) and it models first differences, i.e., changes in quality. The dependent variable is Δ y t and it is a function of the endogenous variable Δ y t − 1 , a predetermined variable Δ y t − 2 , and the difference in the values of hospital control variables Δ x t . The instrument for Δ y t − 1 is y t − 2 and the instrument for each endogenous hospital control variable is x t − 2 . Following the above logic about prompt response of TPS to its values in the previous period, we presume that y t − 2 will affect the change in the value of the TPS from period t − 2 to period t − 1 . So y t − 2 impacts y t through Δ y t − 1 (and potentially even through the predetermined variable Δ y t − 2 ) but not through other channels (i.e., not through processes that occur as late as in period t ). Similarly, upon learning the value of x t − 2 , hospitals speedily adjust their quality to change Δ x t and it affects Δ y t .

Note that [56] used similar arguments in discussing applicability of the dynamic panel data model to analyze in-hospital mortality and the complication rate, which are used as measures of hospital quality in US Medicare hospitals. They write: “We believe our approach is appropriate because (i) changes to in-hospital mortality and complications should be immediately affected by changes in staffing levels, not after a long adjustment period, and (ii) the influence of the past is incorporated through the lagged value of the dependent variable.” (p. 296, Footnote 3).

A related study applying dynamic panel data models to hospital performance indicators deals with average length of stay at Japanese acute-care hospitals that plan to introduce a prospective payment system [10]. The variable is treated in Japan as a proxy for hospital efficiency. It is regularly monitored and analyzed by the regulator and by hospital management, with feedback actions by hospital personnel in response to annual updates on levels of the variable [9,10,43,45,75]. Accordingly, the assumption of a short adjustment period for the length of stay is likely to hold at Japanese hospitals and the use of lagged levels and lagged differences as instruments is justified.

Note that potential violations of the exclusion restriction may occur in instances where the quality measure requires long periods to adjust. In such instances, causal impact of the Medicare reform on the quality of care cannot be established [1,7].

We note other limitations of our approach. First, the analysis deals with the composite quality measure. While quality-related efforts of a hospital and the TPS composite quality measure are multi-dimensional, we do not touch upon multi-tasking in the empirical estimations. Our approach considers a one-dimensional effort, a one-dimensional true quality, and its measurable proxy.^[13]

Second, we do not touch on the rules for computing the scores of each dimension of the composite measure or on aggregation of dimension scores. It is important to note that Medicare uses whichever is highest, improvement points or achievement points, as the score for each dimension. The choice between achievement and improvement points stimulates low-performing hospitals, and the uniform formula assumes that all groups of hospitals have equal margin for improvement. A minor exception is protection of hospitals above the benchmark value of the 95th percentile of a corresponding measure score: these hospitals receive 10 points for their achievement on a [ 0 , 10 ] scale, while the maximum number of points for improvement by any hospital is 9.^[14]

Third, weighting of scores across domains is another feature of the design of the incentive mechanism which is not analyzed in our article. So the dichotomous variables for annual periods in the empirical specification capture time effects unrelated to Medicare’s value-based purchasing as well as time effects not associated with the size of incentives but potentially linked to changes in other elements of the reform design (i.e., changes in weights).

Finally, conventional policy evaluation using a control group of hospitals is not possible because quality measures for non-Medicare hospitals are not available.^[15] The empirical part of the article therefore focuses solely on pay-for-performance hospitals and identifies the effect of quality incentives based on variation in α t and the share of Medicare patients in the hospital s i t . Variation in α t plays the role of the dummy for treatment/pre-treatment periods, and variations in s i t act similar to the dummy for the treatment/control groups.

3.2 Multivariate dependence of the quality variable

4.1 Data sources and variables

The analysis uses data for Medicare hospitals in 2011–2019 from several sources. We use Hospital Compare data archives (January 2021 update) for quality measures, hospital ownership, and geographic location. The medical school affiliation of a hospital, the number of hospital beds, nurses, and physicians come from Provider of Service files. Other hospital control variables are taken from the Final Rules, which are Medicare’s annual documents on reimbursement rates in the inpatient prospective payment system. Specifically, we use information from the Impact Files, which accompany the Final Rules and estimate the impact of the reimbursement mechanism on hospital characteristics. The variables taken from the Impact Files are the share of Medicare’s discharges, ownership, and urban location.

Patient characteristics are also taken from the Impact Files. The casemix variable reflects the relative weight of each DRG in financial terms and is adjusted for transfers of patients between hospitals.^[16] Casemix makes it possible to control for the composition of patient cases taking account of the objective link between severity of illness and hospital resources. The disproportionate-share index accounts for the share of low-income patients and makes it possible to proxy a patient’s income.

To account for other major channels of quality improvement by Medicare hospitals over the observed time period, we use the data for two programs run by the Centers for Medicare and Medicaid Services. One of them is the HRRP, which applies to Medicare hospitals since fiscal year 2013 and penalizes them for excess readmissions. Specifically, the payment reduction which may equal from 0 to 3% is applied to hospital’s Medicare remuneration, higher values of the percentage for the penalty represent more excess readmissions at the hospital. Using the HRRP Supplemental data files, which accompany annual Final Rules on acute inpatient PPS (June 2020 update), we find the HRRP penalty for 2013–2019 and use it as one of the control variables in the empirical analysis.

We also consider the EHR Incentive Program, which was in force since 2011. The program establishes hospital attestation on the use of EHR. The adoption of quality-improving information technology requires substantial fixed cost, so the binary variable for hospital attestation within EHR makes it possible to control for the fixed cost in the empirical analysis. The EHR promotion program consists of three stages (sequentially introduced in 2011, 2014, and 2017). Using data from The Eligible Hospitals Public Use Files on the EHR incentive program (February 2020 update), we set the EHR attestation dummy equal to one if the hospital passed its attestation for the given year at any stage. Owing to non-availability of data on the third stage of the program, we extend the second stage data from year 2016 to years 2017–2019. Use of an attestation dummy lets us control for the fact of incurring the fixed cost of quality-improvement efforts. Owing to the small size of the non-EHR group (only 8–10% of the sample), we do not analyze whether quality goes up faster in the group of the hospitals (for instance, we do not interact the attestation dummy with α ).

4.2 Sample

The non-anonymous character of the data sources allows us to merge them by year and hospital name. Our analysis focuses on acute-care Medicare’s hospitals, as the pay-for-performance incentive contract applies exclusively to this group. We restrict the sample by considering only hospitals with share of Medicare cases greater than 5%.

The specification with second-order lag enables estimation of the fitted values of TPS t and the values of μ t only starting 2013. Accordingly, we can evaluate the impact of the incentive contract on quality improvement in 2013–2019. There are 2,984 hospitals in our sample for 2013–2019, which make TPSN observations (Table 1).

4.3 Flow of quality and evidence of mean reversion

Descriptive analysis of the values of TPS offers suggestive evidence in support of some of the main hypotheses generated by the model. Specifically, we focus on the flow of hospitals between quintiles of TPS in different years. The Sankey diagrams in Figures 1 and 2 use the width of arrows as the intensity of flow rates and demonstrates how hospitals change their position in quintiles of the composite quality measure after the introduction of pay-for-performance (e.g., from 2012 to 2013).

Figure 1

Flow of hospitals between quintiles of TPS in 2012–2013.

Figure 2

Flow of hospitals between quintiles of TPS in 2018–2019.

As can be inferred from Figure 1, there is considerable movement of hospitals between quintiles. For instance, consider hospitals which in 2012 belonged to the fifth quintile of TPS (quintile with the highest performance). Fewer than half of these hospitals remained in the fifth quintile of TPS in 2013, and the rest saw a decline of their position relative to other hospitals by moving to quintiles one through four. Similar tendencies are observed for hospitals in any other given quintile of TPS in 2012: only a small share of hospitals continue to belong to the same quintile in the subsequent year. This can be viewed as graphic support for the phenomenon of mean reversion since hospitals would rarely change their quintile from year to year in the absence of mean reversion.

It is plausible to assume that mean reversion becomes weaker when there is an increase of α . Figure 2 supports this prediction. It shows the flow of hospitals between quintiles of TPS from 2018 to 2019, when the value of α was 0.02. Compared to Figure 1 with α equal to 0.01 in 2013, the flows in 2018–2019 are much weaker than the flows in 2012–2013, so hospitals change their position in quintiles less often.