An innovative medical and dental hygiene clinic for street youth: Results of a process evaluation

Highlights

• A focus on program fidelity, dose, reach, and satisfaction was informative.

• A mixed methods design strengthened the trustworthiness of results.

• The “hybrid model” of evaluation led to timely program improvements.

• Evaluation and implementation science can build on each other's knowledge base.

Abstract

Canada has a noteworthy reputation for high quality health care. Nonetheless, street youth are one of our most vulnerable yet underserved populations. Consequently, a medical and dental clinic was created in downtown Ottawa, Ontario to respond to their needs. The purpose of this study is to describe a process evaluation of the clinic during its first year of operation with a focus on program fidelity, dose, reach, and satisfaction. A mixed methods approach was used involving interviews with providers, focus groups with street youth, analysis of Electronic Medical Record (EMR) data, and supplemental information such as document reviews. The evaluation identified areas that were working well along with challenges to program implementation. Areas of concerns and possible solutions were presented to the management team that then helped to plan and make improvements to the clinic. Our evaluation design and working relationship with clinic management promoted the integration of real-time evidence into program improvements.

Introduction

It is estimated that some 150,000 youth live on the streets of Canada (DeMatteo et al., 1999) with about 100–250 of these youth living in Ottawa (Sourial, 2003). Commonly termed “street youth” or “homeless youth” this group has been defined as “young people who spend considerable amounts of time on the street, who live in marginal or precarious situations and who participate extensively in street lifestyle practices” (Kelly & Caputo, 2007, p. 728). Unfortunately, street youth live under unstable conditions and many do not receive health care services such as medical and dental care (Hudson et al., 2010, Klein et al., 2000, Lee et al., 1994Nabors et al., 2004, Sneller et al., 2008). For example, Klein et al. (2000) conducted a nationally representative interview survey of 640 sheltered youth and a purposive sample of 600 street youth, aged 12–21 years, in the United States (U.S.), and found that half of street youth and 36 percent of sheltered youth lacked a regular source of health care (p ≤ .05).

Few studies have examined the oral health of street youth. Lee et al. (1994) reported that only 22.4 percent of street youth in Toronto, Canada had seen a dentist in the previous year and that 40.8 percent had not seen one in the previous two years even though the majority of these youth were in need of dental care. More recently, Chi and Milgrom (2008) surveyed 55 homeless adolescent and young adults, aged 14–28 years, in Seattle, U.S., and discovered a number of unmet dental problems, most commonly sensitive teeth (52.6%), discolored teeth (48.6%), toothache (38.5%), or a broken tooth (37.8%).

The literature reveals numerous barriers to health care services for street youth. These include not knowing what is available (French et al., 2003, Rew et al., 2002), mistrust of health care providers (Christiani et al., 2008, Kelly and Caputo, 2007), and lack of health insurance (Ensign and Panke, 2002, Martins, 2008). Fortunately, some of these barriers can be overcome by community clinics sensitive to youth-related issues and that install a sense of trust in the youth (Geber, 1997, Kelly and Caputo, 2007). Ensign (2004) conducted interviews and focus groups with homeless youth who reported that the most helpful health care services were those who treated them with respect, had good communication, and provided continuity of care. Evaluation is an important tool to help ensure that such services are delivered in an appropriate manner.

Process evaluation is “a form of program evaluation designed to determine whether the program is delivered as intended to the target recipients” (Rossi, Freeman & Lipsey, 2004, p. 431) with at least three of many possible purposes, to (Grembowski, 2001, Issel, 2004, Rossi et al., 2004): (1) make adjustments to a program midstream to advance fidelity to its original plan and thereby increasing the likelihood of meeting stated outcomes; (2) clearly articulate what is in the “black box” referred to as “the program” so that if marked improvements or deficiencies occur you can link those back to particular program activities; (3) document the history and context of a program to inform others who might want to replicate the program. It has been applied in numerous health programs (Devine et al., 2012, Griffin et al., 2010, Verweij et al., 2011); however, few studies have been written on the process of implementing or “start-up” of community based clinics or programs to support street youth (Law & Shek, 2011).

Though touted as “the most frequent form of program evaluation” (Rossi, Freeman & Lipsey, 2004, p. 57), there is no “gold-standard” approach to process evaluation in terms of design, questions, or methods. Commonly, process evaluators are concerned about program fidelity or adherence to the original program design; dose in terms of both delivering and receiving the program in the subscribed intensity; reach or serving the intended target group; and satisfaction of core stakeholders such as clients and service providers (Fagan et al., 2008, Frazen et al., 2009, Griffin et al., 2010, Law and Shek, 2011, Saunders et al., 2005, Verweij et al., 2011). Some evaluators caution that it is not feasible to measure all areas of program implementation and thus one must be selective in choosing those aspects that provide the most relevant information particularly in relation to the key program components (Scheirer, 1994).

The purpose of this study is to describe a process evaluation of a medical and dental hygiene clinic for street youth with a focus on program fidelity, dose, reach and satisfaction. These were of particular interest to clinic management in their attempt to identify any problems that the clinic had with implementation so that these could be corrected midstream and thereby increase the likelihood of achieving its intended outcomes.