Transformational Impact Rating System: a method to assess the impact potential of hospitals in low-resource settings

Background: Christian mission hospitals provide critical assistance to communities that need their services the most, yet they often lack the financial and human resources needed to realize their full potential for community impact. Existing charity rating organizations rate few charities and lack transparency for donors, and limited resources exist to help volunteers find organizations that would benefit from their services the most. The objective of this article is to develop a transparent Transformational Impact Rating System (TIRS) that can be used by funding agencies and volunteer directing agencies to assess the potential for transformational impact of any hospital anywhere in the world. Methods: Research to inform the rating system was conducted via a comprehensive review of existing charity rating systems and literature, in-depth interviews, and surveys. TIRS was developed by identifying, selecting, categorizing, and allocating weights to variables. TIRS was implemented via a custom-built website to enable future testing and evolution. Results: A rating system was developed that generates an overall hospital score and scores in each of five categories (regional impact potential, financial health, operational performance, external relations, and volunteer experience) based on data collected from the hospital and surveys of donors, volunteers, staff, patients, and the local community. Discussion: TIRS in its current state is a proof of concept that demonstrates data-driven decision-making for funding and staffing hospitals with high potential for transformational impact. TIRS can be modified for future applications. Conclusion: TIRS provides a proof-of-concept system through which all hospitals in low-resource settings can make their impact potential visible to donors and volunteers. If TIRS (with suggested modifications) is used by those interested in helping mission hospitals, invisible hospitals that are struggling in remote areas will be more likely to receive much-needed resources. TIRS can help direct declining funds and manpower to hospitals that have the highest potential for transformational impact


Introduction
This project was motivated by the experience of two doctors who restarted the then closed-down Makunda Christian Leprosy and General Hospital 1 in 1992 and transformed the hospital into a model of high quality, accessible healthcare in a low-resource setting. 2 The hospital is located in a remote rural part of Assam, India.Over the next 30 years, the hospital grew into a large, 200-bed institution with poor-centric strategies that brought high capacity utilization to its services.In 2019-20 (the last fiscal year unaffected by COVID- 19), the hospital saw 126,382 outpatients, admitted 15,297 inpatients, performed 8,710 surgeries, and conducted 6,750 deliveries.In addition, the Makunda Christian Higher Secondary School, started in 2004, taught 1,300 students from local communities and upgraded its School of Nursing to a College of Nursing.A branch hospital was also started in the neighboring state of Tripura (the first Christian mission hospital in that state).
The early years at Makunda were difficult. 1There were no patients, no external financial support, local opposition, staff rebellion, and the need to provide high quality services safely despite numerous constraints.Since electrical generators had been sold by previous staff to pay salaries, the hospital staff lived without electricity in their homes for the first 14 years, performed surgeries like cholecystectomy without electrocautery, and gave anesthesia with an Epstein-Mackintosh-Oxford apparatus as oxygen and electricity were difficult to procure.Further, due to multiple perceived strains on personal and professional life, it was difficult to recruit and retain professional staff, resulting in heavy workloads on the remaining staff.Although the hospital is doing well 30 years later, the high levels of stress the staff endured might have been avoided and greater impact achieved for patients earlier if additional financial support and manpower were available during the early years.
Hospitals serving the neediest populations across the world face similar scenarios to Makunda in its early years. 3When these hospitals are in the early stages of development and at their most vulnerable, they are invisible to donors and committed staff and volunteers.Donor and human resource directing agencies send support to institutions that have already established impact many years after their establishment.Faced with financial and workload stress, founding staff at hospitals earlier in their impact journey may decide to abandon the work, although hospitals in these locations of great need may have the highest potential for transformational impact.
The impact of healthcare institutions can be incremental or transformational.Healthcare impact is incremental when the hospital's establishment improves safety, quality, costs (including indirect costs such as travel), and convenience to people who are already accessing healthcare services.Healthcare impact is transformational when the services are designed to enable people who have never accessed modern healthcare services to do so.Further, in many countries in the developing world, healthcare access and outcomes vary tremendously between and within their states.Hospitals that are located in the neediest areas and adopt best practices that enable them to run most effectively and efficiently, as well as improve the personal and professional lives of staff, have the highest level of potential for transformational impact.While impact may take many years to realize, assessing the potential for impact in the early years when these institutions are at their most vulnerable stages of growth may provide access to resources to accelerate their growth.Relatively small investments of funds, manpower, and infrastructure in hospitals with high potential for transformational impact should result in disproportionately large improvements in healthcare indices.
The research objective of this study was to develop a system to evaluate the potential for transformational impact of hospitals that may otherwise remain invisible.The Transformational Impact Rating System (TIRS) rates hospitals based on location and performance factors.The resulting scores can assist donors and potential volunteers and staff in finding hospitals that fit their search criteria and have the highest potential for transformational impact.The increased visibility provided by TIRS could result in increased investment in these hospitals and, therefore, improvements in healthcare indices in the surrounding communities.

Methods
A team of students from the Wharton School of Business conducted the research and produced an initial report on the rating system upon which this article is based. 4Research to inform the rating system was conducted using three primary methods: 1) comprehensive review of existing charity rating systems, 2) in-depth interviews, and 3) surveys.The rating system was developed and implemented as a website for future testing.

Review of Existing Systems
A comprehensive review of existing charity rating systems was conducted to identify the strengths and limitations of currently available rating systems.][7][8][9][10][11][12][13][14][15] Methods adopted by each organization were evaluated, including the charities it chooses to rate, the factors it considers in determining a rating, the level of transparency it provides into the rating, and its reliance on information from other charity rating systems.Table 1 summarizes the methods of existing charity rating systems.

In-depth Interviews
A total of 12 in-depth interviews were conducted, including 7 representatives of donation management and financial certification organizations and 5 representatives of volunteer and staff sourcing organizations (see Supplemental Table S-1 for details).At the beginning of each interview, interviewees were informed of the purpose of the research and that their responses would be aggregated to inform the project.Semi-structured interviews focused on specific factors that donors and volunteers might consider when choosing to contribute or not contribute to a charitable organization and how they might utilize a rating system (see Supplemental

Surveys
A survey was conducted to evaluate which factors were most important to volunteers in deciding where to volunteer.Survey recipients were identified through interviews with organizations that coordinate volunteers and prior volunteers at Makunda Hospital.The survey asked respondents to rate 28 factors identified through the research as relevant to volunteers on a scale from 1 (not important at all) to 10 (extremely important).In addition, the survey asked respondents to provide any other factors they considered important.A total of 32 responses were received (Supplemental Table S-3 summarizes the results).

Rating System
Based on steps 1 through 3 above, an impact rating system was developed to score the transformational impact potential of charitable organizations, specifically focusing on hospitals in low-resource settings.More than 150 variables were identified that could be used to measure impact potential; of these variables, 93 independent variables were selected that repeatedly appeared in the research, were previously validated, were important to donors' and volunteers' decisionmaking based on interviews and surveys, and could be objectively measured.Each variable was given a weight from 1 to 10, with regional and local factors, factors that appeared in the research more frequently, factors that were previously validated, and factors rated as more important in surveys receiving a higher weight.Levels from 1 (low impact potential) to 5 (high impact potential) were defined to measure each variable based on published benchmarks and research 2, as well as the interview findings. (1) ariables were categorized into one of five categories that emerged: regional impact potential, financial health, operational performance, external relations, or volunteer experience. Varbles within the three largest categories (financial health, operational performance, and volunteer experience) were further categorized into subcategories.Using the predefined levels and weights, a hospital receives an integer score from 1-5 for each variable, and weighted averages produce subcategory, category, and overall scores.The final variables included in the rating system with their definitions, categories, and weights are shown in Supplemental Table S-4, and variable levels are shown in Supplemental Table S-5.

Website Implementation
The rating system was implemented via a website (www.transformationalimpact.org) to enable future testing and evolution by simplifying data collection.In the future, the website could also be used to improve the ability of donors and volunteers to find hospitals that have obtained an impact score.To prioritize access and speed to implementation, a mobile-friendly website that could be accessed via any device was chosen over a mobile app, which would need to be designed for multiple operating systems.
First, data collection templates were prepared to define specific data elements that would be collected from the hospital, donors, volunteers, staff, patients, the local community, and publicly available sources.For the first iteration, the hospital data collection template was implemented on the website by allowing a user to sign in via Google account, add their hospital to the system, and answer questions for each variable by selecting one of the predefined levels for each variable.A sample of the questions is shown in Figure 1.All questions are required.(The user may select "No Data"; however, selecting this option results in a score of 0 for that variable to disincentivize hospitals from reporting "No Data" in areas where they may be underperforming.)The website then calculates the hospital's score as described in Results, below.Users may check their overall rating before submitting their data and allowing their score to be publicly displayed.Information that can be populated from public sources is pulled in via web scraping. (1)A table cross referencing each variable with the associated sources can be provided upon request to the authors.Mayner, Ismavel, From the home page of the website, visitors can see hospitals that have been rated, their overall scores, and their locations on a map.By selecting a hospital, visitors are directed to the hospital's page which includes a description of the hospital, causes it supports, its overall score, and scores for each category and subcategory, as shown in Figure 2. Future iterations of the website could allow donors and volunteers to see the variables that are of interest to them, provide feedback on hospitals to which they have previously donated or at which they have volunteered, and see comments from other users.

Proof-of-Concept Testing
As a proof of concept, the rating system was initially tested using hypothetical data for two hospital archetypes (one that might be considered high impact and one that might be considered low impact) as well as with actual Makunda Hospital data.Data for these hospitals were input into the data collection templates and the resulting scores were examined to determine whether the scores accurately reflected the hospital's expected transformational impact potential.Levels were adjusted for some variables and some variables were removed based on the results of these tests.The final results of the tests were an overall score of 4.1 for the archetype of a hospital with relatively high transformational impact potential and room for improvement; an overall score of 2.6 for the archetype of a hospital with relatively low transformational impact potential but high resources and average performance; and an overall score of 4.6 for Makunda Hospital, which has been studied for its high transformational impact. 2

Data Collection
To obtain a TIRS impact score, data for each variable must first be collected for the hospital.The hospital provides information for 69 variables that evaluate the hospital's regional impact potential, financial health, operational performance, external relations, and volunteer experience.Although collecting data on so many variables could be burdensome for hospitals, many variables require qualitative assessment rather than quantitative calculation.In addition, all data is collected by selecting one of the five levels for each variable, such that understanding the range the data falls within is sufficient.Both of these functional choices reduce the burden of data collection for hospitals.
In addition to collecting data from the hospital, data is collected from surveys of donors, volunteers, staff, patients, and the local community.Volunteers are asked to rate two variables related to the hospital's commitment to its mission and the credibility of its leadership, six variables related to the logistics of the volunteer opportunity, and thirteen variables related to their satisfaction with their experience.Staff are asked to rate five variables related to the hospital's commitment to its mission, level of community engagement, culture of continuous improvement, the credibility of its leadership, and the staff's level of satisfaction.Staff are also asked to score three variables related to volunteer opportunities.Donors, patients, and the local community are asked to rate only three variables each, focusing on their satisfaction and the hospital's performance.In each survey, respondents may provide additional comments that can be shared with those viewing the hospital's score.
Finally, data that is publicly available from the internet is collected to rate some variables.For the first iteration of the rating system, the only variable that relies solely on data from the internet is whether the organization has previously been rated by another charity rating system.While country-level data on maternal mortality rate, infant mortality rate, population density, and crime rates are provided to the hospital for reference, hospitals select the levels for those local, non-modifiable factors due to lack of more granular publicly available data within a country.

Calculating the Score
Once data has been collected for each variable, the hospital receives an overall score from 1 to 5, where 5 represents a hospital with the highest possible impact potential.This score is simply a weighted average of each variable's rating (based on the levels and weights discussed in the Methods and detailed in Supplemental Tables S-4 and S-5).Variables for which no data is available receive a score of zero (rather than being excluded from the score) to encourage future collection of these data.
In addition to the overall score, hospitals receive a score in each of five categories: regional impact potential, financial health, operational performance, external relations, and volunteer experience.Category scores allow hospitals to understand on a broad level in which areas they are performing well and in which areas they could improve.These category scores are simple weighted averages of the ratings of the variables that fall within the category.In other words, specific categories are not given additional weight; the importance of the category was already considered when weighting individual variables.
Regional impact potential represents the hospital's ability to make an impact on the community based on local needs.This category largely consists of factors over which the hospital has no control.Financial health represents the extent to which the hospital uses its funds efficiently, manages its funds properly, and is sustainable.Operational performance is the largest category, including factors that measure how efficiently the hospital executes its mission, its level of governance, its transparency, the number of patients the hospital serves, and how well it serves its patients.External relations measures how well the hospital collaborates with other parties, such as donors, the community, and other programs.Finally, volunteer experience measures the quality of the experience, logistics, and satisfaction of prior volunteers.
Because financial health, operational performance, and volunteer experience encompass the most factors, each category is broken down further into subcategories.Financial health includes funding sources, financial efficiency, financial capacity, and accountability.Operational performance includes operational efficiency, governance, transparency, organization quality, volume of services, and care quality.Volunteer experience includes volunteer opportunity and volunteer Mayner, Ismavel, Shroff AUGUST 2024 -VOL 11 ISSUE 2 satisfaction.Hospitals receive a score in each subcategory (calculated as a weighted average of the variables included in the subcategory) so that they can identify more granular areas in which they can either continue to perform well or improve.
Figure 3 shows a diagram of the scoring hierarchy.

Using the Score
Once data has been collected and TIRS scores have been calculated, hospitals can view their overall score, category scores, and subcategory scores, as well as ratings for each individual variable.Hospitals can view which factors are modifiable (i.e., can be changed based on the hospital's performance) and which are non-modifiable (i.e., are based on local conditions).By examining the modifiable factors for which they scored the lowest, hospitals can review the criteria for obtaining a higher score and take actions to improve their rating (starting with the modifiable variables which are weighted most highly and / or require little effort to improve) with the goal of increasing their attractiveness to donors and volunteers.Hospitals may choose to update their data at least annually, or sooner if they make significant changes, to ensure donors and volunteers can view the most up-to-date score.
In the future, donors and volunteers could choose to contribute to hospitals with the highest TIRS scores that match their interests.Users could search the TIRS website based on TIRS scores and additional factors (noted at the end of Supplemental Table S-4), such as the geography in which the hospital is located, any specific causes the hospital supports (e.g., maternal health, leprosy), whether the hospital has a Christian mission, whether the hospital can accept foreign donations, and the ease with which volunteers can obtain visas.Hospitals that meet the user's search criteria would be displayed, and hospitals that have the highest overall impact score would be shown at the top.Users could select each hospital to view more information, such as category and subcategory scores, as well as any descriptive information the hospital has provided.

Score Differentiation
Many charity rating systems exist today, as summarized in Table 1.However, many organizations choose which charities to rate and do not accept requests from charities that ask to be rated.Even organizations that accept charities' requests to be evaluated may limit their evaluation to charities in America that have IRS filings, or they may have strict criteria that the charity must meet in order to be rated.For example, Charity Navigator only rates US-based 501(c)(3) tax-exempt organizations. 5As a result, small but high impact hospitals in needy parts of the world have no presence on these rating websites.In contrast, TIRS is available to any hospital that wishes to be rated, regardless of size, operating metrics, or location.
In terms of the target audience of these rating systems, nearly all (with the exception of Universal Giving) focus on donors and do not address specific factors of interest to volunteers.However, many high impact hospitals in low resource settings could benefit nearly or equally as much from volunteer doctors who can help them serve their communities as from monetary donations.To fill this gap, TIRS includes factors that are important to both donors and volunteers, so that volunteers can find hospitals where volunteering can have the greatest impact.
Additionally, existing rating systems ignore regional impact factors.They evaluate charities based on how efficiently they use their funding, how well they have sustained their programs and services over time, and their level of commitment to good governance, best practices, and openness with information.For instance, CharityWatch considers a charity that spends at least 75% of their funds on program expenses and that spends no more than $25 to raise each $100 to be highly efficient. 6onsequently, the ratings represent how well charities currently perform in terms of financial health, operational performance, transparency, and governance.However, current systems fail to consider the regional impact potential of organizations located in areas where the populations served have particularly great needs.Because regional impact factors measure the need of the community and directly measure an organization's potential for impact, TIRS includes these factors in addition to validated factors included in existing rating systems.

Discussion
Considering the strengths and weaknesses of existing rating systems, TIRS was developed to provide visibility to hospitals with high potential for transformational impact.To address the issue that small organizations and organizations outside of the U.S. are usually excluded from existing rating systems, TIRS will be open to all legally registered hospitals.Hospitals are not required to meet any criteria on revenue, length of operation, or otherwise.To fill in the gap of impact factors relevant to volunteers, TIRS captures information related to volunteer satisfaction and opportunities.Additionally, TIRS rates regional impact factors so that hospitals' potential for community impact can be measured along with their existing performance.TIRS is designed to enable donors and volunteers to find opportunities where their contributions would be most impactful.
TIRS in its current state is a proof of concept that demonstrates data-driven decisionmaking for funding and staffing hospitals with high potential for transformational impact.The system should continue to be improved and can be customized to meet specific needs.Potential applications and modifications include: • Refine and simplify TIRS.TIRS currently includes 93 variables, making data collection burdensome.Opportunity exists to test TIRS with the actual hospital data, simplify data collection by combining or reducing variables, and automate collection of publicly available data to improve the usability and accessibility of TIRS.customized for specific countries could use data ranges that are more relevant for a given geographical location.• Implement third-party verification.The present version of TIRS is based largely on hospitals' self-assessment, which introduces bias.Administrators can employ third-party evaluators to visit and periodically assess hospitals to verify the data provided and improve credibility.The scores of hospitals that have been evaluated by third parties can be indicated as "verified."• Provide hospitals with individual impact reports.Individual detailed reports, including which factors are modifiable and actionable recommendations, can help hospitals improve their score.Adoption of TIRS by major donors and human resource directing agencies coupled with individualized reports could be a powerful tool for institutional behavior change.• Implement additional data collection on the website.While TIRS collects data from surveys of donors, volunteers, staff, patients, and the local community, the website implementation of TIRS currently only collects hospital-reported data.Future versions of the website could implement these additional surveys.Furthermore, when donors and volunteers engage with a hospital, they could be invited to provide their assessment at the end of their engagement.• Improve website searchability.Tags could be added to hospital profiles so that hospitals can be searched based on location (e.g., Western Africa), service profile (e.g., maternal health), denominational background (e.g., Christian), or other factors that may be of interest to donors, volunteers, and staff.• Customize TIRS over time.The factors and weights for each factor can be customized as the internal and external environment in a particular situation or application evolve over time.Site administrators can be provided access to granular information from hospitals that allow fine-tuning of factors to suit emerging needs.• Create customized versions of TIRS for specific uses.TIRS can be used for other impact areas of interest, such as agricultural or educational institutions, by substituting healthcare-related factors for appropriate factors in these areas.

Conclusion
In the neediest parts of the world, healthcare investment by government entities is inadequate and interest from private donors and volunteers is insufficient.A primary challenge that hospitals in these areas face is limited access to financial and human capital.Many charity rating systems only rate a limited number of organizations or do not provide sufficient transparency into the ratings they provide.The objective of this project is to develop a rating system through which all hospitals in low-resource settings can make their impact potential visible to donors and volunteers looking to make the greatest difference with their money and time.
The TIRS rating system combines information on 93 variables across five categories to provide hospitals with an overall impact score (as well as scores by category and subcategory).To obtain a score, information is collected from a variety of sources, including the hospital, publicly available information, and surveys of donors, volunteers, staff, patients, and the community.Donors and volunteers can use these ratings to identify hospitals with high impact potential where their donations and time will make the greatest difference.In addition, hospitals can use detailed information from their scores to continue performing well and to improve in areas that could allow them to access additional resources.TIRS demonstrates data-driven identification of hospitals with the greatest potential for transformational impact, especially for donors and volunteers.If TIRS (with suggested modifications, such as simplification, customization, and verification) is used by those interested in helping mission hospitals, these hospitals will be more likely to adopt best Mayner, Ismavel, Shroff AUGUST 2024 -VOL 11 ISSUE 2 practices, and invisible hospitals struggling in remote areas will be more likely to receive much-needed resources.Ultimately, the goal of TIRS is to connect small, non-profit hospitals with appropriate resources to improve the lives of those who need their services most.TIRS and derived tools can be powerful instruments in the hands of entities and individuals that seek to revive sick mission hospitals across the globe and establish new ones by using data to direct resources to hospitals that have the highest potential for transformational impact.

Table 1 :
Methods of Existing Charity Rating Systems

•
Create customized versions of TIRS for specific geographies.Because the current scope of TIRS is global, some variables have very wide data ranges.For example, a maternal mortality rate of 600 currently receives a score of 5, indicating great need; however, this rate would be low for South Sudan and high for India.Versions Mayner, Ismavel, Shroff AUGUST 2024 -VOL 11 ISSUE 2

Table S -
3. Volunteer Survey Questions and Responses

Table S -
4.b.Hospital Page Description and Search CriteriaIf you would like, you may provide a story of an individual whose life your hospital has benefited.Personal stories often help donors and volunteers make a personal connection to a charitable organization.Does the hospital have any specific donor needs?If so, please explain, as a way for potential donors to find the hospital.If available, please provide a plan for future funds, including specific uses (e.g., new program or facility), amounts, priority, and alignment to the hospital's strategic plan.Does the hospital have any specific volunteer needs?If so, please explain, as a way for potential volunteers to find the hospital.If available, please provide a plan for volunteer work (e.g., specific procedures the volunteer will get to learn or perform, learning objectives).Does the hospital have any specific volunteer needs?If so, please explain, as a way for potential volunteers to find the hospital.If available, please provide a plan for volunteer work (e.g., specific procedures the volunteer will get to learn or perform, learning objectives).Mayner, Ismavel, Shroff it for volunteers to obtain a visa to work with the hospital?Please explain.Page Description Volunteer Salary Are volunteers able to receive a salary?If so, what is the average salary or a typical range of salaries?Page Description Attractions Are there any features of the hospital's location that you would like to highlight that might attract short-term volunteers, such as music, wildlife photography, nearby tourist attractions, etc.? Page Description Location Access Please describe the location in which the hospital is located and how difficult it would be for volunteers to travel there.support any specific causes (e.g., leprosy, cancer, maternal health)?Please list each as a way to allow donors and volunteers to search for the hospital.Is the hospital registered with the appropriate bodies required to receive foreign donations (e.g., FCRA in India)?Search Criteria Conditional Donations Can the hospital accommodate donations that come with conditions tied to them?Peer Reviewed: Submitted 23 June 2023, Revised 29 Jan 2024, Accepted 5 June 2024, Published 19 Aug 2024 Competing Interests: None declared AUGUST 2024 -VOL 11 ISSUE 2