Case study used to guide the development of effectiveness principles for Principles-Focused Evaluation

Principles-Focused Evaluation (P-FE) allows for the evaluation of contextual adaptations necessary to successfully guide social interventions in dynamic, complex contexts. In this type of participatory, utilisation-focused evaluation, effectiveness principles need to be developed at the start of the project and are used to guide decision-making during project implementation. At the end of implementation, these principles are evaluated. The evaluator’s role in this type of developmental evaluation is collaborative and ongoing (Patton 2018).


Introduction
and outcomes. A purpose of the effectiveness principle is to Ensure usefulness throughout the presentation (Patton 2018). Thus existing values and principles need to be constructed as effectiveness principles which can be evaluated.
Principles-Focused Evaluation was developed by Michael Quinn Patton (2018), and in theory, it seems like a useful methodology to use within the South African context. This is because P-FE allows innovators to move away from authoritarian, one-size-fits-all type of programmes towards a less-prescriptive application, which is inclusive of the needs and rights of beneficiaries (Easterly 2014), whilst at the same time maintaining the necessary levels of accountability.
Social innovators working with complex issues are often guided by principles as much as they are by theories of change, logic models and other plans. These principles may be implicit or explicit and they usually form an intrinsic part of the intervention strategy. In these instances, principles reflect deeply held values and beliefs and provide an insight into ethical reasoning. At the start of P-FE, principles need to be extracted, clarified and articulated, so that they become useful, actionable, adaptable and can guide decision-making. To do this, principles need to be worded in such a way that they can be evaluated (Patton 2018).

Aims and objectives
In presenting this retrospective, exploratory case study of the successfully implemented Saving Lives at Birth (SLB) project, the authors illustrate how values can be extracted from existing project documents, reports and clarifying interviews. Through the application of the GUIDE framework (Patton 2018) these values are then articulated as effectiveness principles and used as a framework for further projects.
The rigorous extraction of principles at the start of the project was intended to increase the evaluators understanding of the P-FE process and allow for quicker understanding of organisational norms. At the same time we aimed to generate a meaningful starting point for the development of effectiveness principles during a stakeholder workshop.
It is expected that by reporting the practical application of P-FE and highlighting some of the early challenges, the authors will initiate a conversation within the African evaluation community, and by doing so facilitate an in-depth understanding for themselves and others of what appears to be an important methodology for our context.

Method and design
A pragmatic paradigm informs this retrospective, exploratory case study of a successfully implemented project (the SLB Project) guided by the methodology described by Yin (2014). The case study was undertaken to ensure rigour whilst developing effectiveness principles intended to guide the implementation of another project (Kistefos Primary Mental Health Pilot Project) in six Department of Health (EC DoH) Primary Health Clinics located in two semi-rural districts within Buffalo City, Eastern Cape Province, South Africa.

Data sources
Data were collected during the programme by means of evaluation document (report) analysis and clarification interviews with the programme staff.

Data analysis
All documents were systematically analysed using an inductive reasoning process to identify implicit and explicit values or principles that informed choices and changes in programme implementation. When these were unclear, the staff members were interviewed so that the evaluator could understand the rationale behind the decisions taken. Later the GUIDE framework (Patton 2018) was used to guide the articulation of the effectiveness principles from the values and principles identified during the initial analysis.

Context
The programme reported in this case study was implemented in 13 DoH primary health clinics, located within under-resourced, remote rural communities, in the OR Tambo District (Nyandeni Municipality) of the Eastern Cape province.

Background
Mental disorders (MD) currently affect around 450 million people globally and is a leading cause of ill health and disability worldwide. Mental health disorders are increasing, and about 75% of people affected do not seek help, mainly because they fear stigma, discrimination and also because of their inability to access care. The poor are the most affected, as they are exposed to the greatest number of risk factors and usually lack the resources to access care (World Health Organization (WHO) 2001). Pregnant women, babies and adolescents are especially vulnerable. Common MD, namely anxiety and depression are implicated in a wide range of social and developmental concerns, including cognitive and emotional development, non-communicable diseases, addictions and human immunodeficiency virus (HIV) (Woody et al. 2017;WHO 2001).
The South African Mental Health Care Act (Act No. 17 of 2002) and the Mental Health Care Amendment Act (Act No. 12 of 2014) provide the framework for mental health care in South Africa, but the application of policy has proved to be challenging and, except few exceptions, has not been effectively integrated into the primary health care systems (Draper 2009).
The Small Projects Foundation (SPF, https://www.spf.org. za) based in East London is a non-government organisation (NGO) established in 1988. It provides developmental support for the poor and disadvantaged communities to improve the physical, social and economic quality of life of both individuals and the community. In the past, many SPF interventions were aimed at supporting the primary health clinics to develop effective management structures and provide training and simple strategies to improve services. In one such intervention, the SLB Project, the SPF partnered with the Eastern Cape DoH and various funding agencies to successfully develop a comprehensive model for the holistic care of children and families affected by HIV/AIDS, living in remote areas of the Eastern Cape. The SLB Project has several elements, including a mental health component, which holds important lessons for the development of the Kistefos Primary Mental Health Project.

Case study: The Saving Lives at Birth Project
The SLB Project ran from 2014 to 2018 and was funded by the Positive Action for Children Fund, implemented by the SPF together with the Eastern Cape DoH and the fund was managed by the One to One Children's Fund. The project aimed to reduce the mother-to-child transmission (MTCT) of HIV in one of South Africa's poorest rural areas, from 29.6% to 2%. In addition, the SLB Project aimed to improve the uptake and follow-up of HIV counselling and testing (HCT), ensure that any child born with HIV benefitted from early infant diagnosis (EID) and anti-retroviral therapy (ART). Project strategies included stimulating community mobilisation, introducing integrated clinic registers, updating referral and tracing systems, and improving clinic responsiveness and accountability. Also, community health workers (CHWs) were trained to conduct community-based outreach programme and provide health education to encourage pregnant women, new mothers and family members to be tested for HIV and adhere to ART and prevention of mother-to-child transmission (PMTCT).

Background to the Saving Lives at Birth Project
UNAIDS (2019) estimate almost 38 million around the world live with HIV. Around 54% of these HIV-infected people live in Eastern and Southern Africa, and amongst them 37% live in South Africa (USAID Global Report 2017). Childhood HIV usually occurs when the virus is transmitted from an HIVpositive woman to her child during pregnancy, birth or breastfeeding. The risk of this type of transmission, called MTCT, is between 15% and 45% for mothers who do not receive any treatment. However, with PMTCT, the risk can be reduced to well below 5% (WHO 2019). In 2010, the highest prevalence (29.6%) of HIV amongst pregnant women in South Africa was reported in the OR Tambo District Mental health is an often a neglected but very important factor that impacts HIV (Freeman et al. 2007). This is because there is a complex, bi-directional relationship between MD and HIV, which increases the risk for both. For example, people with a common MD like anxiety and depression, and those who use substances, have a greater risk of contracting HIV. Those who already have HIV are at a greater risk of developing an MD and also of abusing substances (Jallow et al. 2017). The reasons for the latter are thought to be related to stigma and the stress associated with having a condition carrying severe risk and which requires life-long adherence to medication (Jallow et al. 2017). In addition to increasing the risk of contracting HIV, MD impacts on other critical issues such as disclosure of HIV status, readiness to start ART, accessing clinic services, adherence to treatment (Jallow et al. 2017) and PMTCT.

Population
The SLB Project was implemented in 13 DoH primary health clinics in an under-resourced, remote rural area of the OR Tambo District (Nyandeni Municipality) in the Eastern Cape.

Ethical considerations
The ethical clearance for the Principle-Focused Evaluation of the Kistefos Primary Mental Health Care Project was obtained from the Human Sciences Research Council (HREC) Ethics Committee (REC 10/20/03/19) on 12 July 2019, however this article reports on a review of project reports from another Small Projects Foundation project, which informed the development of principles for the evaluation.

Findings
The SLB Project was complex with a wide reach, which was deemed necessary for the long-term sustainability and success of the intervention. Initially, the community were mobilised, trained and elected onto clinic committees to increase clinic accountability and responsiveness to community needs, to improve patient-centred care and to ensure compliance to national guidelines. Access to HIV testing and treatment for the whole community was improved, and all pregnant women were encouraged to make early use of antenatal services, get tested for HIV and, if positive, to start treatment at the earliest. Integrated clinic registers and a new referral and tracing system were introduced to ensure the effective follow-up of patients who missed their appointments. Balanced scorecards (Kaplan & Norton 1992) were introduced, updated regularly and jointly developed by the community and the clinic. These balanced scorecards were used to determine community priorities and ensure accountability of clinics. Also, babies born with HIV were diagnosed and treated early, and CHWs were trained and deployed to increase community awareness and understanding about HIV as well as other health problems. Training in maternal mental health was a late inclusion to the SLB Project, and the programme staff, clinic staff and CHWs were sensitised of the problems these disorders could cause for mothers, unborn babies, infants and older children. Routine, integrated mental health screening was introduced, the need for empathic support and care was highlighted and networks for effective referral was developed. This final aspect is of particular relevance to the Kistefos Primary Mental Health Project.

Maternal mental health
The Perinatal Mental Health Project (PMHP, https://pmhp. za.org), a project based in the University of Cape Town (UCT), which is situated within the Centre for Public Mental Health (CPMH), Faculty of Health Sciences, was contracted to introduce maternal mental health services to the district. The SPF trainers initially completed the Bettercare (https:// bettercare.co.za) Maternal Mental Health distance learning programme (Field et al. 2019). This was followed by a 3-day experiential workshop aimed at reinforcing aspects of the distance learning programme as well as providing opportunity to learn how to screen, engage empathically and if necessary refer mothers who may need more specialised care. The SPF trainers then trained the CHWs in aspects of maternal mental health, empathic care and screening for depression using the Whooley questions (Bosanquet 2015). The importance of accurate data collection and capture was highlighted. To support the CHW training and data capture processes, a facilitator training guide and project-specific CHW information rulers, with screening prompts, were prepared by the PMHP.
It was considered important for all CHWs providing empathic care to steady themselves emotionally to limit care provider compassion fatigue and burnout. Accordingly, a peer-based package of care, developed by the PMHP (Nyamekela4Care), was introduced, which intended to provide a structure for monthly CHW peer-support meetings. This package proved difficult to implement because of logistical, time and cost constraints and was substantially adapted by the SPF into a more focused training system, named Skills to Care. This emerged out of an expressed need for further training by the CHWs, who were already meeting monthly for reporting and debriefing. Skills to Care was devised and tested in partnership with Bettercare, which modified some of their existing distance learning material to meet the training needs of these CHWs.

Programme impact
At the end of the funding period, the SLB Project had achieved a greater impact than the initial programme objectives. During the course of the project, 8393 women accessed antenatal care services (ANC), and almost 88% of them tested for HIV. Of those who tested positive, 98% benefitted from PMTCT interventions. In addition, women started attending clinics earlier in their pregnancy, which allowed for better access to PMTCT. Before the intervention, only 30% of the women attended before 20 weeks of pregnancy; this increased to 65% by 2018. Ninety-five per cent of pregnant women in the area are now routinely tested early for HIV. Those who tested positive were placed on ART, which has led to the reduction in MTCT to 0.2%, and almost all HIV-exposed infants are placed on ART. Also, 95% of female clinic defaulters are traced, and almost 9000 community members were reached through home visits, health talks and outreach campaigns. Of the 3650 women who were screening using the Whooley questions, only 136 (3.73%) were identified as depressed and were managed within the clinic or the community or referred for more specialised care. A referral network was established, and 54 women were referred for more specialised psychological care. These numbers are much lower than anticipated given international data and the current understanding of the rates of anxiety and depression in low-and middle-income communities. Whilst this may be a true reflection, it was thought more likely to be indicative of the problems with the application of the screening tool and CHW training, or the data capture process or the existence of high levels of stigma in the community. Depression in low and middle-income (LMIC) communities (Turner 2016;Andersson 2013). Patton (2018) suggests that existing documents should be examined to look for patterns in the values, expressed norms and concerns raised, as well as any insights highlighted and effective ways of working within the community that informed any adaptations during programme implementation. This process was carried out systematically using the method of inductive reasoning similar to coding qualitative data. A list of principles extracted from the SLB Project reports and documents in this way are listed below:

Recognising and clarifying principles
• Project aims must be clear, but not restrictive, and guided by evolving community needs (e.g. MTCT of HIV was reduced and early treatment of HIV-exposed babies was ensured whilst achieving other unforeseen, but beneficial outcomes). • Involve beneficiaries in the planning and ensure longterm sustainability by strengthening community knowledge (e.g. community dialogues, CHW training). • Provide simple tools and sufficient support to enable ongoing community engagement and ensure service provider accountability (e.g. balanced scorecard, clinic committees and CHW rulers). • Partner with recognised organisations with previously tested models, which can be modified if necessary to meet local needs (e.g. Bettercare and the PMHP). • Be responsive to local needs and be prepared to innovate when existing models do not fit the current situation (e.g. Skills to Care CHW training, which evolved from the PMHP package). • If results seem too good to be true, it is important to review the entire process (e.g. mental health screening data).

Developing effectiveness principles
Once identified, the principles and values were appraised and articulated according to the GUIDE framework (Patton 2018) so they were phrased in a way which could be evaluated and applied contextually.
The GUIDE framework is applied to principles in a similar way in which the SMART framework is applied to goals and objectives. A more detailed explanation follows.

Principles must provide guidance (G)
To guide on what to do, how to think and act, effectiveness principles need to be prescriptive and should provide a clear direction. The wording is important, as it must direct action and show what will be effective: Do this … to be effective (Patton 2018). For example, two of the SLB-extracted principles can be reworded to include action verbs as: • be clear where you want to go, but be flexible about the route • when unsure about how to proceed, ask the beneficiaries.

Principles should be useful (U)
Principles must be helpful when the need arises to make choices, take decisions and manage problems. To meet this criterion, the principle must be clear (easily understood), feasible, relevant in any situation and able to point towards the desired result (Patton 2018). The authors think that the two principles articulated in the previous section provide guidance and are also useful, although the word 'beneficiaries' in the second example is restrictive. There may be situations when it is necessary to consult beneficiaries or other parties directly involved in the project like CHWs, community committees, clinic staff, etc., about the best way to proceed. This second example is widely applicable for it provides direction, as this context points, towards gaining further clarity on more precise working from project stakeholders during a participant workshop! At this point, the principle can be reworded as below, so that it is clear and unambiguous to all parties: • When unsure about how to proceed, ask those directly affected.

Principles need to be inspirational (I)
To be inspirational, principles must be grounded in values and beliefs that matter and are deeply held within the organisation and community. Inspiring principles provide an anchor for complex interventions, especially where outcomes are unpredictable and mostly uncontrollable. For example, the principle Involve beneficiaries in the planning to ensure long-term sustainability by strengthening community knowledge (e.g. community dialogues). It is rooted in the deeply held belief that communities have an essential voice in all aspects of developmental processes, as they need to be empowered to incorporate change in their lives and be benefitted in uniquely meaningful ways. The principle above provides guidance and is useful, but it must be reworded to make it shorter, more succinct and catchier to be inspirational. Where the evaluator is unable to think of an inspirational principle, this again can be directed to stakeholders, who will then identify more strongly with the principle. A shorter, but not particularly inspirational, version of the principle is as follows: • Knowledge must to be understood and integrated to ensure sustainable change.

Principles should also be developmental (D)
To be considered developmental, a principle must be adaptable and applicable across many different complicated and changing situations over long periods (Patton 2018). For example, two of the principles listed above -namely, partner with recognised organisations who have previously tested models, which can be modified for local needs and be responsive to local needs and be prepared to innovate when existing models do not fit the current situation -could be reworded as: • partner with established organisations, but be prepared to innovate • be responsive, flexible and guided by feedback and local needs.

Principles must be evaluable (E)
In P-FE, the principles that guide the innovation must also be able to be evaluated to determine if they were meaningful to the context in which they were applied, if they were followed and used to guide choices and decisions, and if they led to the desired outcome. Data must be collectable to support these judgements. Effectiveness principles must, therefore, be written in a way that makes it clear as to how they can be evaluated. So, one of the principles identified earlier reads provide simple tools and sufficient support to enable the community to engage with and hold service providers accountable. By way of example, in the SLB Project, one could evaluate or judge this principle by determining whether the tools used were simple, whether sufficient support was provided and whether they allowed the community to become involved (e.g. how many members are on the clinic committee, how often they meet).
With the review of completed scorecards, the evaluator can determine if negotiated goals were met.
The list of effectiveness principles developed in this way was used during a workshop with key stakeholders and programme staff to supplement the values they identified as necessary to guide the implementation of the Kistefos Primary Mental Health Project. Some principles were combined with others and reworded to resonate with all the stakeholders. The final list is not included in this article.

Discussion
The extraction and development of effectiveness principles for a P-FE is an essential starting point. However, the process of formulating high-quality effectiveness principles is more difficult than it sounds; it takes time and effort to develop principles that are succinct and specific. The authors found Patton's (2018) book outlining the methodology interesting and informative; however, in spite of the many checklists included in the book, this process was difficult for a novice in P-FE to apply. There are only minimal sources of explanation, other than a few YouTube webinars by Patton, so there is no means of gaining an alternative perspective. The authors therefore embarked on a process of learning-by-doing for this project and intended that sharing this process will be helpful to colleagues who may otherwise be daunted by the prospect of trying this new methodology. The real test, however, will be how well these principles can be applied during programme implementation and also the outcome of the final evaluation.
A rigorous process was followed to extract principles for the existing documents and reports for this case study. However, in the end, some were similar to 'lessons-learnt', which is often included at the end of many evaluation reports. Inductive coding did allow for the identification and highlighting of previously unseen values, which otherwise may have been missed. The entire process was very useful for the external evaluator to gain substantive knowledge and insights into the organisation, their underlying values and also the project being developed.
Principles-Focused Evaluation appears to be a useful methodology to use within the African context, as it may be able to provide clear guidance about the integration of policy within our multiple, complex and dynamic communities. The need to adapt policy for practice in different contexts, especially where resources vary and in situations where needs and understanding are evolving, can be overwhelming. Also, the authors found that developing these principles together with stakeholders, who are not first-language English speakers, was a powerful unifying process.

Conclusion
The development of effectiveness principles is essential for P-FE. Effectiveness principles have to be meaningful to the intended users and must provide a shared language and vision for the project. The process of identifying and articulating effectiveness principles for the start of this developmental evaluation was a useful way to ensure that the evaluator and all stakeholders have a clear and thorough understanding of the project and are fully engaged with the process.