RTAnswers online patient education materials deviate from recommended reading levels

Objective
Patients are turning to the Internet more often for cancer-related information. Oncology organizations need to ensure that appropriately written information is available for patients online. The aim of this study was to determine whether the readability of radiation oncology online patient information (OPI) provided by RTAnswers (RTAnswers.org, created by the American Society for Radiation Oncology) is written at a sixth-grade level as recommended by the National Institutes of Health (NIH), the U.S Department of Health and Human Services (HHS), and the American Medical Association (AMA).


Methods
RTanswers.org was accessed and online patient-oriented brochures for 13 specific disease sites were analyzed. Readability of OPI from RTAnswers was assessed using 10 common readability tests: New Dale-Chall Test, Flesch Reading Ease Score, Coleman-Liau Index, Flesch-Kinkaid Grade Level, FORCAST test, Fry Score, Simple Measure of Gobbledygook, Gunning Frequency of Gobbledygook, New Fog Count, and Raygor Readability Estimate.


Results
A composite grade level of readability was constructed using the 8 readability measures that provide a single grade-level output. The grade levels computed by each of these 8 tests were highly correlated (SI alpha = 0.98). The composite grade level for these disease site-specific brochures was 11.6 ± 0.83, corresponding to a senior in high school, significantly higher than the target sixth-grade level (p < 0.05) recommended by the NIH, HHS, and AMA.


Conclusion
Patient educational material provided by RTAnswers.org is written significantly above the target reading level. Simplifying and rewording this information could improve patients' understanding of radiation therapy and improve treatment adherence and outcomes.

PATIENT EDUCATION MATERIALS DEVIATE FROM RECOMMENDED READING LEVELS applied radiation oncology C ommunication and education are imperative to the physician-patient relationship.][3] Cancer patients often find information about treatments and the value of receiving a second opinion, and obtain support through the internet. 3,4Modern radiation therapy is highly personalized based on a complex interplay between patient characteristics, tumor characteristics, and previous treatments.Compared with other cancer treatments, radiation therapy is disproportionally associated with misconceptions and misunderstanding. 5Patients often leave a consultation trying to make sense of this information deluge and turn to the internet for answers. 6This use of online information can allow patients to more actively participate in their treatment. 7owever, it is imperative that the information be presented in a way that can be accurately comprehended by most patients.This is especially important if patients first seek information online before going to see a physician, as is increasingly becoming the case. 8ost of our knowledge regarding the literacy of Americans comes from the U.S. Department of Education's literacy surveys, conducted in 1982, 1992 and 2003.Most recently, the 2003 National Assessment of Adult Literacy (NAAL) assessed prose, document, and quantitative literacy in a representative sample of 19 000 adults (age 16 and older, including 1200 prisoners) from across the nation.This survey was the first to incorporate a component on health literacy, defined as "the ability of U.S. adults to use printed and written health-related information to function in society, to achieve one's goals, and to develop one's knowledge and potential."The NAAL demonstrated that 43% of American adults have basic or below basic literacy skills. 9Regarding health literacy specifically, over onethird of American adults have health literacy at or below the basic level, 10 and only 12% have proficient health literacy. 11Health literacy is a strong predictor of the health status of an in-dividual, 12 and those with poor health literacy demonstrate worse compliance with treatment recommendations and worse outcomes. 12,13Based on the U.S. literacy rate, the NIH, HHS, and AMA recommend that OPI be written applied radiation oncology at a sixth-grade level. 14These reported grade levels are derived from readability formulas, but do not necessarily indicate that an adult with a specific level of education will be able to read the text.Further, patients typically have a reading ability that is about 5 grades lower than the highest attained educational grade. 15,16To make informed healthcare decisions, patients must have access to both accurate and understandable information.
][19][20] This work has resulted in our cancer center revising online patient information.In this brief report, we hypothesized that radiation oncology patient information found on RTAnswers (RTAnswers.org,created by the American Society for Radiation Oncology [ASTRO]) is written at the recommended sixth-grade level and assessed the readability of this text using a panel of validated readability tests.

Methods and Materials
RTAnswers.org was accessed in May 2016, and OPI in the form of patient-oriented, disease-specific brochures for 13 disease sites was analyzed.Readability analysis was performed with Readability Studio version 2012.0 (Oleander Software, Hadapsar, India).Ten commonly used readability tests were employed: New Dale-Chall Test, Flesch Reading Ease Score, Coleman-Liau Index, Flesch-Kinkaid Grade Level, FORCAST test, Fry Score, Simple Measure of Gobbledygook (SMOG), Gunning Frequency of Gobbledygook (Gunning FOG), New Fog Count, and Raygor Readability Estimate.These tests are well validated and commonly used to assess readability. 21he definition of a reading level is made on the basis of completed school years in the American school system.A composite score was constructed using the 8 tests that provide a single grade-level output (all tests except New Dale-Chall and Flesch Reading Ease Score).
Statistical analyses were performed using SPSS (IBM Corporation, Armonk, New York).All measures were compared to a sixth-grade reading level by t-tests, as this is the grade level recommended by NIH, DHHS, and AMA. 22

Results
We analyzed the readability of 13 patient-oriented disease-specific brochures from RTAnswers.On all readability tests utilized, the readability of OPI from RTAnswers was significantly above target levels.The Raygor test, which analyzes both sentence length and the number of long words to derive a score, demonstrated a mean grade level of 12 (Figure 1A).Similarly, the Fry score, which incorporates sentence length and number of syllables per word, demonstrated a mean grade level of 13 (Figure 1B).The Flesch scale generates a score from 0-100 (with 100 FIGURE 2. The Flesch Reading Ease Score of online patient information (OPI).This test generates a score from 100 (very easy) to 0 (very difficult) with "plain English" corresponding to 60-70 (understood by most 13 to 15 year-olds).This test is a standard measurement of readability often used by U.S. government agencies.The mean score on this test was 47 (range 16-53), well below the target level and, therefore, well above the target level of readability.A composite measure of grade level was constructed using the 8 readability measures that provide a discrete grade level (above).These measures were strongly correlated and formed a reliable measure (SI alpha = 0.98).The Flesch Reading Ease and New Dale-Chall did not provide an output as a single grade but were consistent with these tests.

PATIENT EDUCATION MATERIALS DEVIATE FROM RECOMMENDED READING LEVELS
applied radiation oncology being the easiest level of readability and 60-70 corresponding to "plain English").This test focuses on words per sentence and syllables per word and is commonly used by U.S. government agencies to assess readability. 23The average score on the Flesch Reading Ease Scale was 47 (range 16-53) (Figure 2).
We then constructed a composite grade level of readability of the 13 RTAnswers brochures using the 8 readability measures that provide a single grade-level output.The readability of patient brochures for all disease sites are far from the target level as determined by these readability tests (Table 1).Furthermore, the grade level computed by each of these 8 tests was highly correlated (SI alpha = 0.98).When combined, the tests yielded a composite grade level of 11.6 ± 0.8, corresponding to a senior in high school, and significantly greater than the recommended sixth-grade level (p < 0.05).
Next, we demonstrate one example of a sentence from a bladder cancer brochure from RTAnswers that we have edited to improve the readability (Figure 3).This correction decreased the Flesch-Kincaid reading level from 17.6 to 8.9.We conclude that similar alterations can be made to all RTAnswers brochures and other OPI to improve readability.

Discussion
Cancer patients commonly use the internet to seek information about their diagnosis and treatment. 6One major barrier to effectively understanding and utilizing this online information is the readability of the material.Our results indicate that online patient educational materials on RTAnswers are significantly more complex than the recommended sixth-grade level.8][19] Furthermore, our findings are consistent with the readability of other online materials (eg, WebMD, NIH, Mayo Clinic) pertaining to the most common internal medicine diagnoses. 24This study builds on the work the work from Byun and Golden, who utilized the Flesch-Kincaid test to assess written ASTRO materials. 25owever, our analysis goes a step further by extracting and analyzing information from patient brochures by each disease site found online.
To improve patients' comprehension of radiation therapy and its role in their treatment, our analysis suggests that the language used in online patient information can be simplified to improve communication.Simple, easily understood language can reduce patient stress and anxiety, and improve the physician-patient relationship. 26,27To accomplish this, OPI provided by RTAnswers and academic cancer centers should use simple or well-known terminology, avoid medical or technical terms, use simple phrase and sentence structure, and incorporate feedback from nonmedical personnel into developing these brochures.An example of simplifying language is shown in Figure 3.This shows a sentence extracted from one of the brochures on bladder cancer and our interpretation to improve it.Similar techniques can be extrapolated to other OPI.
Improving readability of OPI can have a myriad of positive outcomes.Patients typically assess website accuracy based on its endorsement by a government agency or professional organization, their own perception of the website's reliability, and their ability to understand the information presented. 28,29Improving readability increases the likelihood that patients will follow recommendations.Furthermore, improving readability of OPI is also important for effective communication, a potential barrier to help overcome healthcare disparities.

Conclusion
The composite grade-level readability of OPI collected from RTAnswers

RTAnswers (Words 48, Flesch-Kincaid 17.6)
External beam radiation therapy (also called radiotherapy) involves a series of daily treatments to accurately deliver radiation to the bladder and pelvis.Research trials have shown that radiation and chemotherapy can permit some bladder cancer patients to have organ-preserving treatment that doesn't require complete removal of the bladder.
Revised RTAnswers (Words 60, Flesch-Kincaid 8.9) Bladder cancer can be treated using high power x-rays known as radiation.Radiation is focused on the bladder and pelvis.Therapy is delivered Monday thru Friday for several weeks in a row.Radiation may also be combined with chemotherapy.This can allow some patients to avoid surgery.Your doctor will discuss the risks and benefits of this "organ preserving" treatment.PATIENT EDUCATION MATERIALS DEVIATE FROM RECOMMENDED READING LEVELS applied radiation oncology was 11.6, corresponding to the senior year of high school.This was significantly greater than the target sixthgrade level.These differences may prevent understanding of OPI by the general public.The readability of OPI provided by RTAnswers can be improved to enhance patient understanding and improve outcomes.

FIGURE 1 .
FIGURE 1. Raygor and Fry Scores of RTAnswers online patient information.(A) The mean Raygor score was 12 (range 11-13) for all 13 disease site brochures assessed, well above the target grade level of 6 (green shading) proposed by the National Institutes of Health (NIH) and the U.S. Department of Health and Human Services (HHS).One text from RTAnswers was too difficult even for analysis and is not included in the average above.(B)The mean Fry score was 13 (range11-16) for all 13 disease site brochures assessed, exceeding the target grade level 6 (green shading) as recommended by the NIH and HHS.

FIGURE 3 .
FIGURE 3. Revising word choice and sentence structure to improve readability.(A) This figure shows a sentence from a patient brochure from RTAnswers in bladder cancer.The readability score of this paragraph is high and corresponds to at least college level.(B) By simplifying word choice and minimizing words per a sentence, we can improve the readability of the text.