Can surveys of women accurately track indicators of maternal and newborn care? A validity and reliability study in Kenya

Background Tracking progress on maternal and newborn survival requires accurate information on the coverage of essential interventions. Despite widespread use, most indicators measuring maternal and newborn intervention coverage have not been validated. This study assessed the ability of women delivering in two Kenyan hospitals to recall critical elements of care received during the intrapartum and immediate postnatal period at two time points: hospital discharge and 13–15 months following delivery. Methods Women’s reports of received care were compared against observations by trained third party observers. Indicators selected for validation were either currently in use or have the potential to be included in population–based surveys. We used a mixed–methods approach to validate women’s reporting ability. We calculated individual–reporting accuracy using the area under the receiver operating curve (AUC), population–level accuracy using the inflation factor (IF), and compared the accuracy of women’s reporting at baseline and follow–up. We also assessed the consistency of women’s reporting over time. We used in–depth interviews with a sub–set of women (n = 20) to assess their understanding of key survey terms. Results Of 606 women who participated at baseline and agreed to follow–up, 515 were re–interviewed. Thirty–eight indicators had sufficient sample size for validation analysis; ten met criteria for high or moderate reporting accuracy (0.60<AUC) alone and ten met criteria for low population–level bias alone (0.75<IF<1.25). There was a significant decline in the individual level reporting accuracy between baseline and follow–up for ten indicators. Seven indicators had moderate or higher (0.4≤rphi) consistency between self–reports at baseline and follow–up. Four indicators met all criteria at follow–up: support person was present during the birth, episiotomy, caesarean section, and low birthweight infant (<2500 g). Conclusion The few indicators that women reported accurately at baseline were consistently recalled with accuracy at 13–15 months follow–up. Although there is deterioration in women’s recall in some indicators over time, the extent of deterioration does not appreciably compromise reporting accuracy for indicators with high baseline validity. Indicators related to initial client assessment and the immediate postnatal period have generally low accuracy and poor reporting consistency over time.

. Survey indicators and questions.
Indicator Interview question Inclusion in Global Health Initiative a Initial Client Assessment Takes blood pressureinitial examination Did someone take your blood pressure, at or near your admission to the facility?

HIV status checked
During your time at the health facility for the birth of your baby, did someone ask you what your HIV status was?
Receives HIV test While you were at the health facility for the birth of your baby, did anyone test you for HIV?
Providers washes hands with soap and water or uses antiseptic before any initial examination Did the health provider(s) wash his/her hands with soap and water or use antiseptic before examining you?

Provider Respectful Care
Encourages/assists woman to ambulate during labor Were you allowed to get up and walk around while you were in labor?
Allowed to drink liquids/ eat Were you allowed to drink liquids or eat any food while you were in labor?
Encourages/assists woman to assume different positions in labor Did a health care provider ask you what position you preferred during labor?
Allowed to have a support Were you allowed to have a companion with you ENAP: Birth companion of choice and skilled

Indicator
Interview question Inclusion in Global Health Initiative a person present during labor and delivery during your labor? [Or] Were you allowed to have a companion with you during your delivery? attendant at birth. (Number of births attended by a doctor/ nurse/ midwife and where the mother had a birth companion of choice) A support person/companion is present during birth Did you have a companion with you during labor? [Or] Did you have a companion with you during delivery?

First Stage of Labor
Induces labor with a uterotonic (IV, IM, tablet) (1) For the birth of your baby, did your labor start spontaneously or did someone do something to bring on your labor? [Yes, No] (2) For your birth of your baby, what was done to bring on your labor? [Injection in thigh/buttocks, IV line inserted in arm, Given tablets to put under tongue, tablets inserted in birth canal, or tablets inserted in rectum] Augments labor with a uterotonic (IV, IM, tablet) (1) For your delivery, was anything done to speed up or to strengthen your labor? (Yes, No) (2) What was done to speed up or to strengthen your labor? [Injection in thigh/buttocks, IV line inserted in arm, Given tablets to put under tongue, Tablets inserted in birth canal, or Tablets  Main provider deliverynurse/midwife Who was the main provider assisting you during delivery?

Second & Third Stage Labor
Uterotonic administered in 1-3 minutes following delivery (injection, IV medication, tablets) (women who had vaginal delivery) Just after the delivery of your baby In the first few minutes after the delivery of your baby, did anyone give you an injection in your thigh or buttock?
[Or] … did anyone give you medication intravenously (through a tube in your arm)? [Or] ...did anyone give you tablets to swallow or hold in your mouth? [Or] ...did anyone place tablets in your rectum?
Uterotonic administered after delivery of placenta (women who had vaginal delivery) After the delivery of your baby, in the first few minutes after the delivery of the placenta, did anyone give you an injection in your thigh? [Or] ...did anyone give you tablets to take or hold in your mouth?
Method of uterotonic postbirth-IM injection (women who had vaginal delivery) After the delivery of your baby, in the first few minutes after the delivery of the placenta, did anyone give you an injection in your thigh?

Interview question Inclusion in Global Health Initiative a Episiotomy
For your delivery, shortly before you delivered your baby, did anyone cut the opening of your vagina (episiotomy) to make more room for the baby's head?
Cesarean section Was your baby delivered by cesarean, that is, did they cut your belly open to take the baby out? 3 essential elements of newborn care (immediately dried, skin to skin, breastfed) (1) Was your baby dried off with a towel or cloth immediately after his/her birth, within a few minutes of delivery? [AND] (2) Did someone place the baby on your chest, against your skin, immediately after delivery of the baby? [AND] 3) Did you breastfeed your baby within the first hour after delivery?
ENAP: Proportion of newborns who received all four elements of essential newborn care: immediate and thorough drying, immediate skin-toskin contact, delayed cord clamping, initiation of breastfeeding in the first hour.

Immediate Postnatal Care-Mother
Uterine massage after delivery of placenta (women who had vaginal delivery) After the delivery of the placenta, did the birth attendant firmly massage your lower abdomen to help make your womb contract (become firm)?
In first examination postdelivery, did provider ask or check for bleeding?
In your first physical examination/check after delivery, did a health provider look for or ask you about bleeding?
In first examination postdelivery, did provider examine perineum?
In your first physical examination/check after delivery, did a health provider do a perineal exam?
In first examination postdelivery, did provider take temperature?
In your first physical examination/check after delivery, did someone take your temperature?
In first examination postdelivery, did provider take blood pressure?
In your first physical examination/check after delivery, did someone take your blood pressure?

Indicator
Interview question Inclusion in Global Health Initiative a In first examination postdelivery, did provider check for involution?
In your first physical examination/check after delivery, did a health provider check your belly to see if your womb was becoming firm after the birth of your baby?

Maternal and Newborn Morbidity
Low birthweight infant (<2500g) What was your baby's weight? [weight given in grams] WHO100: Incidence of low birthweight among newborns (less than 2500 g) DHS & MICS Survey Question: How much did (NAME) weigh?

Complicationshemorrhage
Did you experience any of the following complications during or within the first few hours after your delivery? [Excessive bleeding] EPMM: Address all causes of maternal mortality, reproductive and maternal morbidities and related disabilities. Includes emphasis on documenting major causes and complications that lead to maternal death in every country.

Complications-prolonged labor
Did you experience any of the following complications during or within the first few hours after your delivery? [Long labor, more than 12 hours]

Complications-none
Did you experience any of the following complications during or within the first few hours after your delivery? [None] Complications-yes (to any)    Received pain relief medication 506 0.38 a Analysis performed for indicators for which there were at least 5 counts per cell of two-by-two table comparing women's self-reports at follow-up and baseline.