Effect of Performance Feedback on Community Health Workers’ Motivation and Performance in Madhya Pradesh, India: A Randomized Controlled Trial

Background Small-scale community health worker (CHW) programs provide basic health services and strengthen health systems in resource-poor settings. This paper focuses on improving CHW performance by providing individual feedback to CHWs working with an mHealth program to address malnutrition in children younger than 5 years. Objective The paper aims to evaluate the immediate and retention effects of providing performance feedback and supportive supervision on CHW motivation and performance for CHWs working with an mHealth platform to reduce malnutrition in five districts of Madhya Pradesh, India. We expected a positive impact on CHW performance for the indicator they received feedback on. Performance on indicators the CHW did not receive feedback on was not expected to change. Methods In a randomized controlled trial, 60 CHWs were randomized into three treatment groups based on overall baseline performance ranks to achieve balanced treatment groups. Data for each treatment indicator were analyzed with the other two treatments acting as the control. In total, 10 CHWs were lost to follow-up. There were three performance indicators: case activity, form submissions, and duration of counseling. Each group received weekly calls to provide performance targets and discuss their performance on the specific indicator they were allocated to as well as any challenges or technical issues faced during the week for a 6-week period. Data were collected for a further 4 weeks to assess intertemporal sustained effects of the intervention. Results We found positive and significant impacts on duration of counseling, whereas case activity and number of form submissions did not show significant improvements as a result of the intervention. We found a moderate to large effect (Glass’s delta=0.97, P=.004) of providing performance feedback on counseling times in the initial 6 weeks. These effects were sustained in the postintervention period (Glass’s delta=1.69, P<.001). The counseling times decreased slightly from the intervention to postintervention period by 2.14 minutes (P=.01). Case activity improved for all CHWs after the intervention. We also performed the analysis by replacing the CHWs lost to follow-up with those in their treatment groups with the closest ranks in baseline performance and found similar results. Conclusions Calls providing performance feedback are effective in improving CHW motivation and performance. Providing feedback had a positive effect on performance in the case of duration of counseling. The results suggest that difficulty in achieving the performance target can affect results of performance feedback. Regardless of the performance information disclosed, calls can improve performance due to elements of supportive supervision included in the calls encouraging CHW motivation.

Performanc e ij =β 0 + β 1 Treat i + β 2 Afte r j + β 3 (Treat i * Afte r j )+ϵ ij (1) We estimated three different specifications for this model with three different dependent variables for each of the performance indicators being assessed, namely case activity, form submissions number, and duration of counseling. For each of the specifications Performance ij is the performance of the community nutrition expert i in week j. Treat i is 1 if the community nutrition expert belongs to the treatment group and is 0 otherwise. After j is 1 if the data is from the intervention or post-intervention period, and 0 if it is from the baseline period. Treat i *After j is 1 if Treat and After are both 1 and 0 otherwise. β 1 gives the difference in performance between the treatment and control groups in the baseline period (for difference in performance between the treatment and control in the intervention/post intervention period we can add β 1 and β 3 ), β 2 gives the difference between the baseline and intervention/post intervention period for the control group (for difference between the baseline and intervention periods for the treatment group we can add β 2 and β 3 ) , and β 3 gives us the double difference estimate, which is the estimated average effect of the treatment while accounting for baseline differences in performance and any change in performance overtime. ϵ ij captures the error while β 0 gives us the constant. Our hypothesis is that β 3 will be significantly different than 0 for all three performance indicators.
We will estimate equation one using data only from the intervention period, weeks one to six, when the community nutrition experts were receiving calls, as well as using data from the intervention and post intervention periods, weeks one to ten to look for sustained effects of providing performance feedback on their performance.

Comparing each treatment against other two treatments as pooled controls: Cross Partial Effects
Performanc e ij =β 0 + β 1 Treat i + β 2 Afte r j + β 3 (Treat i * Afte r j )+ϵ ij (2) After estimating the effect of performance feedback on the relevant indicator in equation 1, we estimate cross partial effects in equation 2, which is the effect of providing feedback on one indicator (x) on the performance in the other two indicators. That is to say we estimate the cross partial effects of providing feedback on case activity, on performance in duration of counseling, and form submissions, for each of the three indicators. The coefficients are to be interpreted similarly as in equation 1. Here too β 3 gives us the double difference estimates, which is our average treatment effect, the effect of providing performance feedback on one indicator on the performance in the other two indicators.
We run two different specifications for each of the three equations as in the first model

Comparing the Three Groups Head-to-Head
Where T i F indicates that the community nutrition expert belongs to the Form Submissions treatment, T i C indicates that the community nutrition expert belongs to the Case Activity treatment, and T i D indicatates that the community nutrition expert belongs to the Duration of Counseling treatment.
In equation three we estimate the treatment effects of each treatment head to head against the other two treatments. The variables included in the equation are the same as in equation 1. The difference is that treatment dummies for two out of three treatments are included in the regression (in equation 1, we only include one treatment dummy, and treat the other two treatment groups as a pooled control group), and we omit the treatment group for the indicator being analyzed (i.e. if case activity is the response variable, then treatment group case activity is omitted from the regression) to compare the performance of the three treatment groups head to head on the different indicators. The three interaction terms are perfectly collinear, hence we drop the same indicator as the response variable from our model. We will repeat the estimation for each of the three indicators.
There are two null hypothesis: 1) β 1 = β 2 =0 , where we hope not to reject the null and find no baseline differences in the three treatment groups, and 2) β 3 =β 4 =0, where we hope to reject the null and find that receiving feedback on a certain indicator e.g. Case Activity, affects performance on that indicator (case activity) differently than performance on the other two indicators (form submissions or duration of counseling). If the feedback is more effective in increasing performance on the other two indicators than on the feedback indicator, the coefficients for β 3 and β 4 will be negative.

Heterogeneous Effects
To test for heterogeneous effects of the treatment, i.e. call intensity, we interact the number of calls received with the treatment and period to obtain the effects of the number of calls received by the community nutrition expert in the treatment group. We estimate equation four.
Performance ij = β 0 + β 1 Treat + β 2 After+ β 3 (Treat * After)+ β 4 calls+ β 5 (calls * After )+ β 6 (calls * After * Treat )+ E i The coefficient of interest is β 6 which gives the heterogeneous impact of the treatment based on treatment intensity, i.e. Number of calls received in the intervention period. The sum of β 6 and β 3 give the double difference estimator measuring overall treatment effects where call intensity is also factored in. Here too, we run two different specifications, first limiting the data to the intervention period, and second including data from the post-intervention period to look for sustained effects of the treatment.