Online Reviews of Mental Health Treatment Facilities: Narrative Themes Associated With Positive and Negative Ratings

In the United States, approximately one in five people experience a mental illness each year, and less than 50% receive treatment (1). Although most evidence points to increasing provision of treatment over time, studies also show an increasing mental health burden both nationally and internationally (2–4). Factors contributing to the treatment gap include not only underfunding—in 2015, the mental health burden in the United States was 2.7 times greater than the proportion of health funds allocated to mental health (3)—but also patients’ engagement in their own care. Patients’ engagement affects treatment retention and therapeutic outcomes (5). Although engagement depends on patients’ perceptions of the quality of care they receive, few studies have analyzed factors associated with positive or negative experiences of mental health treatment (6, 7). Patients’ satisfaction is associated with fewer psychiatric symptoms and better quality of life (8).

The implementation of quality measures in mental health care lags behind that in other fields of medical care (9). The majority of the quality measures that have been endorsed by the U.S. National Quality Forum and used in major quality reporting programs, such as the Centers for Medicare and Medicaid Services’ Hospital Compare, are related to screening and assessment rather than patient-centered outcomes (10). According to the Patient-Centered Outcomes Research Institute, end users of research, including patients, should be involved in defining outcomes that are “meaningful and important to patients and caregivers” (11). Psychiatric hospitals included in Hospital Compare and evaluated through the Inpatient Psychiatric Facility Quality Reporting Program do not report star ratings, and patients who “received psychiatric or rehabilitative services” are excluded from the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) because the “instrument is not designed to address . . . the behavioral health issues pertinent to psychiatric patients” (12, 13). The Substance Abuse and Mental Health Services Administration (SAMHSA) collects data on mental health treatment facilities through the National Mental Health Services Survey (N-MHSS), but the survey does not include quality measures or patients’ satisfaction measures (14).

Online review sites, such as Yelp, Google, and Facebook, provide platforms for word-of-mouth communication about health care, including mental health services (15–23). Risks with using online review data include fraudulent reviews and overrepresentation of extreme opinions. Online reviews do, however, offer advantages: they allow for narrative reporting, are public, and are available to both patients and their support networks (15, 24). Studies have demonstrated that review ratings and narrative themes tend to correlate with both existing national surveys, such as HCAHPS and the National Survey of Substance Abuse Treatment Services, and, in some cases, with health outcomes (16, 21, 25). Review themes also provide insight into drivers of satisfaction, as measured by review ratings, that are not captured by standard surveys (16, 18, 19, 21, 22, 25).

To identify factors associated with patients’ or their families’ positive or negative experiences with mental health care, we analyzed Yelp reviews of mental health facilities registered in SAMHSA’s Treatment Locator. As of March 31, 2020, Yelp had 35 million average monthly mobile app users and 211 million reviews (26). Given Yelp’s extensive usership and the lack of standardized patient quality and satisfaction measures in U.S. mental health care, patients may turn to the site when selecting or evaluating a mental health facility. Understanding the factors that influence user reviews of facilities could inform areas of focus for reducing unmet mental health needs in the United States. The purpose of this analysis was threefold: to understand the extent to which Yelp is used to review mental health treatment facilities, to describe narrative themes in Yelp reviews of mental health treatment, and to identify narrative themes and facility services correlated with review ratings.

Methods

This study was considered exempt by the University of Pennsylvania Institutional Review Board. The Yelp data used in the study represent an academic data set generated directly by Yelp for research and include all U.S. facilities tagged as “health” facilities according to Yelp’s developer documentation (27). The data set is updated daily and extends beyond content available through the Yelp application programming interface. SAMHSA facility data and N-MHSS data are freely available for download from the SAMHSA website.

Study Sample

To evaluate online reviews for a validated group of facilities, we sought to match SAMHSA mental health facilities to their corresponding Yelp pages. This matching afforded the added advantage of allowing for comparison between Yelp review ratings and the N-MHSS data associated with each SAMHSA facility.

We downloaded all SAMHSA data and Yelp data in September 2019. Facilities were added to Yelp between March 2007 and September 2019. Facility data included name, location, and reviews, including both review star ratings (one to five stars) and narrative review content.

From SAMHSA’s Treatment Locator, we identified 10,191 unique mental health facilities, with listings last updated between February 2017 and September 2019. Facilities are eligible for registration in the Treatment Locator if they are funded by a state mental health agency, administered by the U.S. Department of Veterans Affairs, or licensed by a state agency or national organization to provide mental health treatment. Eligible facilities are identified through surveys of state mental health authorities and medical organizations. All included facilities complete the N-MHSS, and, according to the 2018 N-MHSS, the “survey universe” of mental health treatment facilities was 14,159 (14).

Each SAMHSA facility was matched to a corresponding Yelp page according to a probabilistic string-matching algorithm with a sensitivity of 88.9% and a specificity of 95.1% (28, 29). The list of SAMHSA mental health facilities successfully matched to Yelp pages included medical centers with psychiatric units. In a majority of such cases, the corresponding Yelp page was for the parent organization, and most of the reviews pertained to general medical care rather than mental health care. We used a term search strategy to identify and remove facilities with reviews including high proportions of terms specific to general medical care (rather than mental health care) (19). This step was validated through a combination of hand coding and topic modeling. (Further details are available in an online supplement to this article.)

Review Analysis

To understand Yelp usage over time, we calculated the number of new reviews and the number of facilities receiving a first review by yearly quarter. To evaluate review content, we used a machine learning–based natural language processing model called latent Dirichlet allocation. Latent Dirichlet allocation operates under the assumption that documents, in this case Yelp reviews, can be described by a prespecified number of topics. Topics are defined by distributions of commonly co-occurring words, and each review is then defined by a distribution of topics. A topic most strongly defined by the words “staff,” “helpful,” and “friendly” might then then be highly representative of a review such as, “Great place!! Front desk staff were very helpful.” Two authors (D.C.S., H.M.) independently reviewed the 10 words and reviews most strongly associated with each of the 30 latent Dirichlet allocation topics and, where review content was consistent, assigned the topics a theme. Discrepancies were adjudicated by a third reviewer (R.M.M.). This process resulted in identification of 13 consistent topics. This approach, and the ratio of meaningful topics to all latent Dirichlet allocation topics, was consistent with prior Yelp studies (16, 19, 21).

Survey Analysis

Patients’ ratings of mental health treatment facilities may be correlated with existing objective facility metrics, as has proven true in hospitals, substance use treatment facilities, and nursing homes (16, 21, 25). In mental health care, for instance, evidence from qualitative analyses suggests that patients have concerns specific to pharmacologic and inpatient treatment (30, 31). The N-MHSS is an annual survey conducted by SAMHSA and is the only national survey of both public and private mental health treatment facilities (14). The survey includes 18 service categories divided into 126 binary service codes and covers such information as facility type, special programming, and availability of emergency mental health services. To identify how narrative analysis of review content may support or augment data currently collected by SAMHSA, we sought to measure the correlation between N-MHSS survey items and Yelp review ratings.

Statistical Analysis

To compare Yelp star ratings with both N-MHSS services and latent Dirichlet allocation topics, we used Spearman’s rank-order correlation coefficient, which is suited to comparisons of nonparametric data. Star ratings are ordinal, ranging from 1 to 5, while latent Dirichlet allocation probability distributions are generally nonnormally distributed continuous variables between 0 and 1. N-MHSS services are binary variables. Both the correlations between N-MHSS services and star ratings and between star ratings and latent Dirichlet allocation probabilities were performed at the level of the Yelp review. Significance was defined by a Bonferroni-corrected p<0.001, corresponding to a corrected α=0.0000072 (32). All statistical analyses were performed by using R, version 3.6.1 (33).

Results

Study Sample

Of the 10,191 unique SAMHSA facilities, 2,403 (23.6%) were matched to Yelp “health” facilities. Of these matches, 1,383 (13.6%) remained after filtering by general medical terms (a flow diagram is available in the online supplement). As summarized in Table 1, the final sample differed meaningfully from SAMHSA mental health facilities without dedicated Yelp pages by geographic distribution, with overrepresentation of facilities located in western states, and on several survey items. Among service settings, hospital inpatient and partial hospitalization or day treatment were overrepresented, as was special programming for specific groups, such as young adults and veterans.

The facilities had a total of 8,133 reviews, corresponding to a mean of 5.88 reviews per facility (range 1–107, median=3). The mean±SD word count per review was 170.2±164.8. The distribution of review ratings was bimodal, with 57.5% one-star (lowest possible) and 26.5% five-star (highest possible) reviews. As shown in Figure 1, both the number of reviews and the number of facilities receiving a first review increased from 2006 to 2019.

Review Analysis

Of the 13 latent Dirichlet allocation topics, four were positively correlated with review ratings, seven were negatively correlated with review ratings, and two were not significantly correlated with review ratings, according to a Bonferroni-corrected p<0.001. The topics, redacted sample reviews, and correlations with review ratings are presented in Table 2. The strength of correlations was generally greater for positive topics, for example, “caring staff” (Spearman’s ρ=0.39), than for negative topics, such as “rude ancillary staff” (ρ=−0.14).

Reviews most represented by the generally positive topic “nonpharmacologic treatment modalities” (ρ=0.16) included mentions of “individual therapy,” “yoga,” “dietitian support,” “group therapy,” “dialectical behavior skills,” “family workshops,” and “trauma program(s).” Those for the generally negative topic “safety and abuse” (ρ=−0.14) mentioned specific concerns about “name calling” and being “violated . . . verbally,” “physical assault” by both patients and staff, “sexual assault,” “theft,” and “neglect.” Of the seven topics negatively correlated with review rating, two were related to clerical services: “billing and insurance” (ρ=−0.11) and “scheduling” (ρ=−0.09).

Survey Analysis

Facilities offered, on average, 39.3±10.9 of the 126 N-MHSS services. In total, 11 services were significantly correlated with five-star reviews, and 33 were significantly correlated with one-star reviews. Table 3 shows the 10 services with strongest positive and negative correlations to Yelp review ratings. Among facility types, “outpatient mental health facility” (ρ=0.13) and “residential treatment center for adults” (ρ=0.07) were positively correlated with review rating, whereas “psychiatric hospital” (ρ=−0.19) was negatively correlated with review rating.

As with the latent Dirichlet allocation topic “nonpharmacologic treatment modalities” (ρ=0.16), N-MHSS codes for special programs and for ancillary services—for example, programming for “lesbian, gay, bisexual or transgender clients” (ρ=0.10) and “housing services” (ρ=0.07)—were generally positively correlated with review rating. On the other hand, as with the latent Dirichlet allocation topic “billing and insurance,” N-MHSS codes for specific forms of payment and insurance, especially public insurance—for example, “Medicare” (ρ=−0.21)—were generally negatively correlated with review rating, as were both the latent Dirichlet allocation topic “pharmacotherapy” (ρ=−0.06) and the N-MHSS code “psychotropic medication” (ρ=−.15).

Discussion

This study had three main findings. First, individuals are increasingly reviewing mental health facilities online. The percentage of SAMHSA facilities identified on Yelp (13.6%) was between the percentage of hospitals (31.0%) and skilled nursing facilities (10.4%) identified on the site in prior studies (16, 25). The number of Yelp pages and reviews of mental health facilities is expected to continue to increase: in the 4 months between June and September 2019, 806 Yelp pages were added under the site’s “counseling and mental health” tag. It is noteworthy that facilities in western states and facilities offering inpatient treatment were overrepresented on Yelp. Yelp is based in California, which may explain the former, and large facilities may be both more likely to offer inpatient services and more likely to have Yelp pages. The bimodal distribution of online review ratings, with high proportions of one-star and five-star reviews, was consistent with distributions observed for other health facilities (16, 19). In the absence of formal outlets for patients to report their experiences of mental health care, we expect the volume of patients choosing to do so through more informal means, such as online reviews, to continue to grow.

Second, correlations between review ratings, narrative topics, and N-MHSS services were consistent with previous findings in both qualitative analyses of mental health care and analyses of online reviews. An analysis of online reviews of substance use treatment facilities in Pennsylvania identified the positive review theme “life-changing experiences” and the negative review theme “medication needs,” consistent with themes we identified as correlated positively (“life-saving treatment”) and negatively (“pharmacotherapy”) with online reviews of mental health facilities (21). In the study of substance use treatment facilities, the strength of the correlations reported between themes and review ratings was slightly larger than those reported here, perhaps because of a combination of fewer latent Dirichlet allocation topics and binary treatment of review ratings. Studies of inpatient psychiatric care point to patients’ concerns around safety, restrictions on freedom, communication, and stigma (34, 35). In our analysis, there were more negative comments associated with inpatient services than with outpatient services, and reviews with poor ratings were more likely to mention issues of “safety and abuse,” “communication with family,” and “rude ancillary staff.” Themes associated with more positive ratings, such as “therapeutic alliance with primary provider” and “caring staff,” also correspond to existing evidence: a study of 61 inpatients found that “feeling cared for” and “positive qualities of staff” were themes common to positive patient appraisals of care (34). In general, evidence suggests that patient satisfaction is perhaps most dependent on staff-patient communication (36). Communication with staff has been consistently identified as a theme in online reviews of health care, including in reviews for emergency departments, hospitals, and substance use treatment facilities (16, 18, 21). Among mental health care facility reviews, we found that topics related to communication, namely “caring staff” and “rude ancillary staff,” were those most strongly correlated with positive and negative review ratings, respectively.

Third, narrative topics identified in reviews include actionable items that may improve satisfaction with mental health care. Satisfaction is inherently important—how patients feel about the care they receive should matter to providers—but in mental health care in particular, it may also be associated with fewer psychiatric symptoms and better quality of life (8). Evidence regarding the relationship between satisfaction and health outcomes in medical care is mixed but generally supports positive correlations between patients’ evaluations of communication with both doctors and nurses and more objective measures of quality (25, 36, 37). Many negative reviews of mental health care facilities mentioned “rudeness” explicitly. While “rudeness” is not unique to mental health care, it may contribute to patients’ internalized stigma, which is associated with decreased help-seeking and disempowerment among those with mental illness (38, 39). Mental health care facilities could use the information from Yelp reviews to guide patient-centered interventions. For instance, negative reviews consistently mentioning rude staff could prompt clinic managers to organize targeted communication and antistigma training, with the potential to improve patient engagement in mental health care.

Limitations inherent to online reviews include that reviews skew toward extremes and that fake reviews may escape algorithmic detection. In addition, Yelp users are not representative of those needing mental health care. According to Yelp, U.S. users are almost equally distributed between the ages of 18–34 years, 35–54 years, and 55 years and older. Users skew heavily toward at least some college education (82%), and half of users have an income above $100,000 a year (26). According to a 2018 SAMHSA report, “any mental illness” was most common among those ages 18–49 years, and, among those ages 18 years and older, approximately 64% had at least some college education and 24% were living below the poverty line (1). Yelp is therefore at least overrepresentative of those with higher levels of education and income. Yelp notably does not report on the racial-ethnic demographics of its users, and several racial-ethnic groups in the United States, particularly Black and Latinx people, face persistent barriers to mental health care (40). In the future, dialogue with those who have lived experience of mental health treatment and those with specific barriers to care will be necessary to determine how well Yelp themes encompass factors contributing to patients’ satisfaction.

This study was further limited by a relatively low proportion of SAMHSA facilities with Yelp pages. Google and Facebook reviews are not accessible through application programming interfaces but may include facilities not found on Yelp (21). Subanalyses by facility type, for example, inpatient versus outpatient and adult versus child, were not conducted because of limitations in sample size. These analyses, and evaluations of emergency psychiatric care and of psychiatric telehealth communication, will be important as further online review data becomes available.

A limitation of natural language processing is that identified themes cannot always be easily addressed (23). For instance, despite a negative correlation between pharmacologic treatment and review ratings, in many cases, evidence supports the use of medication in treating mental illness. Knowledge of patients’ aversion to medication, however, can help set the foundation for therapeutic alliance.

Finally, although we were able to compare facility ratings to standard facility-level characteristics through N-MHSS data, there is no nationally reported facility rating system for mental health facilities in the United States (such as Hospital Compare for hospitals) against which to measure Yelp review ratings. Evidence from studies where such comparisons were possible point to correlations ranging from 0.09 to 0.50 (16, 20).

Conclusions

Online reviews are powerful in that they are unscripted, and, as such, the themes that arise from these review narratives are naturally patient centered. Our study’s findings regarding review ratings and the narrative content of reviews are consistent with both national survey data and the available literature. In the absence of the widespread adoption of validated patient-centered outcome measures in mental health, online reviews of mental health facilities have the potential to guide interventions in order to improve patient satisfaction.