Do Healthcare Professionals Comprehend Standardized Symbols Present on Medical Device Packaging?: An Important Factor in the Fight Over Label Space

Objective: Evaluate the ability of healthcare providers from the US to comprehend internationally standardized symbols placed on medical device packaging. Participants: Eighty‐six healthcare providers attending the Association of Surgical Technologists (AST) Instructors' Forum (Savannah, GA February 2014), the 46th Annual AST National Conference (Denver, CO May 2014), or members of the same organization that responded to a targeted email blast. Measures: We adapted the comprehension portion of ISO 9186‐1: 2007 to test 38 standardized symbols developed for the medical device industry. Participants were asked to provide an open‐ended response regarding the meaning of each symbol. Survey responses were categorized into five levels of comprehension: correct, wrong, opposite, don't know or no response. Symbols receiving response rates of ≥85% in the correct category were considered successful. Conversely, if responses categorized as opposite were ≥5%, symbols were considered ‘critically confusing.’ Main Results: Six of 38 symbols were classified as ‘successful’; five of the six had text (in English) imbedded within. Three out of the 38 were categorized as ‘critically confusing’; they were not only misunderstood, but, in fact, interpreted to mean the opposite of what was intended by ≥5% of participants surveyed. Conclusions: Given that the medical device industry in the US has requested permission from the US Food and Drug Administration (FDA) to use stand‐alone symbols to better harmonize with EU Directives, the exploration of healthcare providers' comprehension of the same is an important and timely topic. Our work suggests that symbols commonly incorporated into the labeling of medical devices may not be readily understood at present. As such, policy decisions should be carefully considered. Limitations: Although we provided participants with a general context of use (i.e. these symbols are used on medical devices), the specific part of the hospital, or type of procedure where the symbol would be found was not noted. Further research to evaluate symbol comprehension with specific context (e.g. IVD, general procedure, etc.) is recommended.


INTRODUCTION
Symbols are touted as an effective means of communication because of their high visual impact, ability to transcend language and the fact that they frequently save space (Davies, et al., 1998 1 ;Wolff & Wogalter, 1998 2 ;Liu, et al, 2005 3 ). These qualities are particularly important when communicating information that is required for the safe and effective use of products, such as drugs and medical devices. As global commerce becomes more prevalent, companies and regulators alike increasingly rely on pictorial symbols to convey important messages. The use of symbols is an especially attractive strategy when limited label space precludes the use of multiple languages (e.g. the European Union) or forces the use of text that is difficult to read and interpret. 4 While numerous studies have examined pictorial aids and symbols on processing of information related to medication labels (see 4,5 for reviews of the literature), a very limited number of studies have investigated symbol use and medical devices. 3,6,7 This has led some researchers to suggest 'there is currently insufficient empirical evidence on how practitioners (healthcare providers) utilize and view labeling' of medical devices. 8 Our own review of the literature suggests a paucity of work investigating comprehension rates of the symbols associated with medical devices. 3,7 The two available studies, both conducted outside of the US, suggest poor comprehension rates relating to standardized medical device symbols.
Liu, et al. 3 evaluated the ability of 20 German nurses and 13 Chinese doctors and nurses working in intensive care units (ICUs) to comprehend 16 symbols common to those units using methods prescribed by ISO 9186:2001. 9 Researchers concluded that symbol comprehension was poor for both countries, and that there was no evidence of a significant difference in rates of comprehension when the two populations were compared (p = 0.669). Researchers also tested the effect of symbol specificity on comprehension; symbols were classified as 'global,' those that are used widely in medical products, or 'special' those that are dominantly used in ICU environments. Researchers concluded that comprehension scores of global symbols were significantly better than the special symbols (p = 0.026) and that self-reported familiarity with symbol was highly correlated with comprehension score. This suggests that prior exposure has the potential to improve comprehension; as such, it is probable that training is a fruitful path to improved rates of comprehension. Using the ANSI Z535.3 criterion of ≥85% comprehension, the research team concluded that only three of the 16 symbols (18.75%) tested would be classified as acceptable; this was true for both countries.
Hermans, et al. 7 evaluated the ability of laboratory healthcare workers from four countries (Belgium, Democratic Republic of the Congo, Cambodia and Cuba) to comprehend 18 symbols (from ISO 15223 and EN 980) used on In Vitro Diagnostic Device (IVD) and Rapid Diagnostic Tests (RDT). To do so, the research team used methods described in ISO 9186. Two conditions were tested: stand-alone symbols and symbols presented in context (e.g. the symbol was displayed on a color photograph of a malaria RDT kit package). Participants were instructed to write the presumed meaning of each symbol as an open ended response. In accordance with ISO 3864 (1984), the research team found a symbol to be successful based on ≥67% of participant responses classified as 'correct.' Three out of the 18 symbols (16.7%) met this criteria in the stand alone context (i.e. symbol only). When 16 of the 18 were presented in context (as part of an RDT), five reached the 67% threshold (31.3%), although seven were correctly categorized significantly more than they had been as stand-alone symbols.
Authors concluded that participants scored poorly and that this was 'in contrast to the general perception of regulatory authorities;' EN 980:2008 describes IVD symbols as self-evident to healthcare professionals with no need for further explanation. The authors conclude, 'despite these claims, it is clear that actual comprehension of the IVD symbols does not reach the intended level'. The authors suggest a need for administrative and outreach procedures to improve acquaintance and performance for medical device symbols.
In the US, the Food and Drug Administration (FDA) is granted the authority to regulate the labeling of medical devices by the Federal Food Drug and Cosmetic Act ('Federal Food, Drug and Cosmetic Act (FD&C Act),'. 10 Section 502, subsection 352, of the Act requires clear and understandable labeling for FDA regulated products, including medical devices. As such, if required information on medical device labeling is not 'likely to be read and understood by the ordinary individual under customary conditions of use' (Section 502(c)), a device is deemed misbranded. With a few exceptions (e.g. Invitro Diagnostic Devices intended for professional use), existing, finalized, US regulations indicate that graphics, pictures or symbols that represent required information on medical devices must be accompanied by explanatory English text adjacent to the symbol. 11 However, recent provisions added to the FD&C (21 USC 360d(c)) ('Federal Food, Drug and Cosmetic Act (FD&C Act),' 1938) authorize the FDA to recognize international standards when these standards help to fulfill a regulatory requirement. Recent activities indicate that US FDA is interested in adopting this provision for the labeling of medical devices. Specifically, in April 2013, FDA published a proposed rule involving the use of standardized symbols for medical devices. If finalized, the rule will: (a) allow for the inclusion of stand-alone graphical representation of information or symbols, provided they are part of a standard developed by a nationally or internationally recognized standards development organization (SDO) and accompanied by a symbols glossary, and (b) authorize the use of the symbol statement 'Rx only' on the labeling of prescription devices (US Food and Drug Administration, 2013). 12 The proposal's primary intentions are to make medical device labeling 'more user-friendly by replacing small, difficult-to-read text with pictorial information, and to harmonize the labeling information of US and foreign regulatory bodies (e.g. European Commission)'. Symbol use is widespread in the European Union (EU), largely in response to the EU Commission's 1993 Medical Device Directive, which requires that text must be presented in multiple languages in order to be accessible to providers in multiple countries. Recognizing challenges related to label space, the EU Directive also indicates 'where appropriate this information should take the form of symbols,' and that symbols should conform to harmonized standards. When harmonized standards do not exist, symbols must be described in accompanying documentation (EU Council Directive 93/42/EEC, 1993).
This creates a disparate (and costly) situation where device manufacturers must develop unique labels separately for US and EU markets. Devices designed for European markets must be revised to be sold in US markets so that symbols are either: (a) removed from the label because they are not accompanied by text or, (b) explanatory text (English) must be added to make the device eligible for sale in US commerce. The proposed change in US regulation (US Food and Drug Administration, 2013) is predicated on the assertion that the US FDA believes that certain symbols contained in national or international standards are 'likely to be read and understood by the ordinary individual under customary conditions of purchase and use' [as dictated by the FD&C Act section 502(c)] ['Federal Food, Drug and Cosmetic Act (FD&C Act), ' 1938].
Work presented herein explores the validity of the assertion that the symbols will be understood. Our specific objective was to assess whether people in the US in perioperative roles (e.g. surgical technologists and peri-operative nurses) accurately interpret the meaning of stand-alone symbols from an internationally recognized standard of symbols for medical devices (ISO 15223). 13 If they cannot do so successfully (and the proposed rule were to pass), they would need to seek clarification from a symbols glossary for each unclear symbol in order to understand its meaning.

MATERIALS AND METHODS
All procedures were conducted in accordance with methods approved by the Biomedical Institutional Review Board (BIRB) under IRB #13-698 and were based on ISO 9186-1:2007. 13 Participants provided basic demographic information (e.g. gender, age, education, etc.), and their work experience was characterized with written responses to a survey (e.g. years in healthcare, job title, healthcare setting).
No time limit was imposed.

Participants
Participants were recruited and tested at conferences hosted by the Association of Surgical Technologists (AST) in Savannah, GA and Denver, CO, and through a targeted emailing of members of the same organization within a 30 mile radius of Lansing, MI; email respondents were tested in our laboratory. To participate subjects had to: be 18 or older, have no history of seizure (other portions of the test not reported here involved flashing images), be legally sighted and have worked as a healthcare professional, nursing or surgical technology student. Of the 86 participants recruited, 17 were male; 69, female. Eighty-four listed English as their native language. The average age of participants was 44 years old (ranging from 18 to 66, median: 47). On average, the subject population had 20 years of experience in healthcare (ranging from 1 to 51, median: 22). More than half (52.3%) reported working in acute care hospital settings, 25.6% taught surgical technology, 7% reported their work environments as ambulatory/day surgery centers, 4.7% indicated that they were currently students and the remaining respondents came from long term care facilities, public/community health or other settings of care.

Stimulus materials
Participants were provided with a single variant of each symbol (a referent in the standard's terms) in a printed booklet form. Booklets were produced in a single batch on a copier in black and white contrast with no more than six symbols per page (see Figure 1) in a table comprised of three rows and two columns.
All symbols contained a single variant (i.e. there were not multiple symbols that conveyed the same information). The booklet began with the following instructions, 'This study is intended to evaluate your comprehension level of medical device symbols used for commercially available medical devices. Please view the medical device symbols presented in the following table, and fill in the box "answer" with the meaning of each symbol. If you are not able to assign a meaning of a symbol(s), you can simply write "don't know" in the answer box. It is highly recommended to write whatever you think each symbol means.' Graphical symbols were printed such that they were within a square not less than 28 mm × 28 mm such that the graphical symbol would fill a square of this size if it were present. Thirty-eight symbols were taken from the internationally recognized standard, AAMI/ANSI/ISO 15223: 2007 A1: 2008 (see Table 1 and Figures 2 and 3). It is important to note that 38 symbols is well beyond the maximum of fifteen that is dictated by the ISO standard. That said, we believed this to be reasonable alteration to the standard because previous work with the participant population suggests that they open up to 200 packages per shift, with some respondents reporting as many as 50 for a single procedure 15 ; as such, it is not unreasonable to assume that they would need to be able to interpret the meaning of 38 symbols within a single procedure. No time pressure was placed on test participants during the course of testing.

Procedures
Categorization. Subject responses to each symbol were categorized in accordance with ISO categories. Namely: • 2b: Wrong and the response given is the opposite of the intended meaning • 3: The response given is 'Don't know' • 4: No response is given All participant responses were first appraised by a single researcher who reviewed and assessed them. The initial reviewer then met with one other reviewer and presented common responses that people provided for each and also provided the other researcher with the range of responses subjects used. The second reviewer then assessed the data by category. The review team convened and discussed discrepant responses until consensus was reached regarding the appropriate response category for each stimulus by participant. Responses in category one represented the tally of correct responses; those which matched the definition as indicated by the standard. Responses in category 2b were tallied as 'opposite.' Percentage by category code was calculated for each symbol by dividing the number responses in a given category by the total number of responses for each symbol (see Figure 3).
The ANSI Z535.3 recommendations suggest that labels with a successful comprehension level of 85% or greater are acceptable, and those with less than 10% comprehension are unacceptable (ANSI Z535.

1-5 2011, & Liu et al 3 ).
To classify the labels we tested as acceptable or unacceptable, we used the 'Means' procedure of the statistical software SAS 9.3 (SAS Ins., Cary, NC) and identified the 95% confidence interval around each label's mean percentage correct (Figure 4). We used the upper confidence levels as our cut off for both of these assessments. This method gives the labels 'the benefit of the doubt' for both thresholds. The shaded rows in Figure 2 depict 'acceptable symbols', based on this classification, and the symbols with unacceptable/low comprehension are presented in light or dark orange in Figure 4.
Four symbols (see Figures 4 and 5) did not receive a single correct response; these symbols appear in Figure 4 in dark orange shading and alone in Figure 5.
We also assessed the responses that were opposite of the intended meaning (coded as '2b'see Figure 6). In accordance with ANSI Z535.3, symbols were considered critically confusing when the rate of responses in the 2b category exceeded 5%. The percentage data in the 2b category (opposite) for tested symbols was analyzed statistically using the 'Means' procedure of the statistical software SAS (Version 9.3, SAS Institute, Cary, NC). Through the 'Means' data analysis, Lower Confidence Limit (LCL) and Upper Confidence Limit (UCL) at the 95% confidence level are reported in Figure 6. Because providing a completely opposite response has the potential for significant ramifications, we evaluated two ways, when their LCL exceeded 5% (shaded in red - Figure 6) and symbols whose UCL exceeded 5% (this would include both the red and the orange). Figure 4. Percentage of correct response category (Category 1) by symbol: Mean and UCL and LCL at 95%. CI Grey shading indicates symbol with acceptable rates of comprehension (Category 1 greater than or equal to 85%). Orange shading indicates symbols with unacceptable rates of comprehension (Category 1 UCL less than or equal to 10%). Red shading indicates symbols that were found to be critically confusing (more than 5% defined meaning OPPOSITE of intended definition-Category 2b).

Categorization of responses
The open-ended responses of participants regarding the meaning of the 38 standardized medical device symbols were categorized according to the criteria described in the Materials and Methods section. Complete results by category are presented in Figure 3.
Only six out of the tested 38 symbols (15.8%) passed the comprehension criterion specified by the ANSI standard (ANSI Z535. 1-5 2011(85% in category 1)), despite the fact that the UCL was applied in the interpretation of the data (providing the benefit of the doubt). Those symbols with UCLs exceeding 85% in the correct interpretation category are shaded in grey (see Figure 4) are considered to have acceptable comprehension.
Five out of six symbols that were indicated to have acceptable comprehension levels incorporated text within them. Only one, 'Caution: Consult accompanying documents,' did not incorporate text into the symbol itself. This calls to question the efficacy of recognized symbols that do not directly utilize text, and is relevant given the current FDA proposal which would allow for incorporation of standalone symbols (if accompanied by a legend and recognized by an international standard).
The percentage data of the correct responses (category 1) was also analyzed for unacceptable/low rates of comprehension. Using the UCL of the data as the threshold, there were six standard symbols which had an UCL at or below 10% in category 1. These were: 'In-vitro diagnostic medical device', 'Atmosphere pressure limitation', 'Sampling site', 'Fluid path', 'Liquid filter with pore size' and 'One-way valve' (see Figure 4 orange shading). Most of the symbols with poor comprehension levels were pictorial symbols which did not incorporate alphanumeric characters, with the exception of 'In-vitro diagnostic medical device', 'Liquid filter with pore size' and 'Drops per milliliter'. Orange shading indicates that symbols were found to be confusing (UCL exceeded 5%). Red shading indicates symbols that were found to be critically confusing (LCL exceeded 5%).
Four of the symbols did not receive a single correct response from study participants; these were: 'Sampling site', 'Fluid path', 'Liquid filter with pore size' and 'One-way valve' (see Figure 4 dark orange shading). These symbols were newly added to the ANSI/AAMI/ISO 15223, in 2008, but are not included in ISO 15223.
To further assess comprehension, we also examined the % of participant responses that were categorized as 2b, in other words, response coded to be opposite of the intended meaning. If the LCL of the 2b response category exceeded 5%, it was considered as a 'critically confusing symbol' with regard to comprehension. If the UCL of the 2b response category (wrong and opposite) exceeded 5%, we considered it a 'confusing symbol' with regard to comprehension. Three symbols, all relating to appropriate storage conditions, were identified as meeting the criteria of 'critically confusing' and five symbols were identified as 'confusing' (Figure 6 critically confusingin red; confusing in orange).

DISCUSSION
It is likely that many manufacturers will take advantage of the opportunity to gain label space by utilizing symbols from recognized standards if the proposed rule is enacted by the US FDA. This has the potential to yield significant savings for device manufacturers, who are currently forced to create two sets of labels (one for markets in the EU that do not require the addition of English when a symbol is used and another for the US where English is required to accompany symbols that are used).
Our work supports the work of others 3,7 that suggests that the comprehension level for internationally published symbols for labeling medical devices is quite poor. This was despite the fact that we recruited from an experienced pool of active members of the AST with an average of 20 years of experience in healthcare.
Only 6 out of 38 standard symbols (15.8%) we tested passed the 85% criterion (see Figure 4) despite the fact that we used the UCL as the differentiating factor. This poor comprehension result echoes those reported by Liu et al. (2004), who tested comprehension levels of medical device symbols with populations outside of the US and reported only 18.75% of symbols they tested were classified as acceptable using the same criteria, with no evidence of an effect of country (China vs Germany) on comprehension rates.
Although Hermans (2011) tested more explicit symbols (those used for IVD and RDT) using a very targeted population, laboratory healthcare workers from four countries (Belgium, Democratic Republic of the Congo, Cambodia and Cuba), they report similar scores for comprehension. In their study, three out of the 18 symbols (16.7%) met a 67% threshold in a stand-alone context (i.e. symbol only).
A common characteristic of successful symbols that we tested was the inclusion of supplementary text within the symbol (see Figure 4 grey shading). Conversely, most of the symbols that were correctly defined by less than 10% of respondents, defined as those with unacceptable comprehension rates, did not incorporate text within the symbols (see Figure 4 orange shading).
Perhaps most concerning is the fact that three of the symbols that we tested were categorized as 'critically confusing' according to the ANSI Z535.3 criteria, where users attribute an opposite meaning to the symbol (see Figure 6), and four symbols (see Figure 5) were not correctly defined by any of the participants that we tested.
We reviewed our results in light of those available in the other studies published. 3,7 Both of the other studies available only fully reported the category 1 data (percentage of correct answers). Liu primarily tested symbols taken from IEC 60878 TR Ed. 2.0: 2003 'Graphic symbols for electrical equipment in medical practice.' As such, our study and theirs had only two symbols in common, 'Do not reuse,' a 2 with a cross hatch through it, and Date of manufacture, a symbol that appears to be a manufacturing facility (see Figure 2). Hermans et al. tested symbols from ISO 15223 EN 980, and, thus, had more symbols for comparison. Figure 7 provides a synopsis of each study's results. Comprehension rates for our study were fairly consistent with results reported by the other two studies (see Figure 7). 70 D. C. SEO ET AL.

LIMITATION AND FUTURE WORK
Although we defined the general context of use, i.e. that the symbols would be used with medical devices, our comprehension testing was open-ended. That is, the specific part of the hospital, or type of medical device or procedure where the symbol would be found was not noted. Some literature 2,7,16 suggests that the presence of context (depicting the probable environment where a symbol would be seen) enhances the level of symbol comprehension. It would be worthwhile to evaluate symbol comprehension in a context-based test form (e.g. symbols being embedded on a real medical device label) in future research. Additionally, participants who were recruited for our symbol comprehension study were generalists (e.g. surgical technologists, registered nurses, etc.) in the healthcare industry. Twenty-two out of the tested 38 symbols were included in the FDA guidance: Use of symbols on labels and in labeling In-Vitro Diagnostic devices intended for professional use (2004), tests likely to be used by laboratory workers. Some specialized symbols (e.g. In Vitro Diagnostic medical device, Sterilized using aseptic processing techniques, etc.) used for In-Vitro Diagnostic devices may not be commonly used in the environments where our participants work. Nonetheless, results were similar to those reported by Hermans, et al. 7 who did tailor their test population to a specific environment and also utilized contextual testing.
We deviated from the published standard when we did not randomize the order of presentation of the symbols, but instead presented each participant with the same booklet. To test whether or not a run effect was present, we correlated order with mean performance to see if there was a trend through the data (toward improvement or decline). No evidence for a trend was noted (r = À.0.13; p À0.46). To further investigate for a possible effect, we assumed that the symbols were repeated measures from subjects. For example, the first symbol is the first measurement taken from the person and so on. Therefore, each subject had 38 measurements. We then tested to see if the subject's response changed over time. The results did not show any consistent changes from the first label to the last. Last, we assessed labels that are similar to each other and see if the order has an effect. For example, in the table there are three labels that are similar: Sterilized using ethylene oxide, Sterilized using aseptic processing techniques and Sterilized using steam or dry heat. The mean comprehension percent for these three symbols are 0.29, 0.62 and 0.28 respectivelywhich indicates no order effect.

CONCLUSIONS
In light of a very limited body of work (Liu et al., 2004 andHermans et al., 2011), all of which suggests poor comprehension rates for standard symbols, and the data presented here, policy changes regarding stand-alone symbols for medical device packaging and the potential implications of these changes should be carefully considered prior to enactment. Although we appreciate the challenges posed by an increasingly integrated, global economy, our work provides strong evidence that current stand alone symbols are not interpreted correctly by healthcare providers.

FINANCIAL SUPPORT AND CONFLICT OF INTEREST
Work presented here was partially funded by an Investigator Initiated Grant provided by the Blue Cross Blue Shield of Michigan Foundation; we are grateful for this financial support. Additional funds were provided by proceeds from the Healthcare Packaging Immersion Experience (HcPIE). A portion of Dr. Bix's salary is supported by the USDA National Institute of Food and Agriculture, Hatch project MICL02263. CR Bard and Teleflex Medical provided several cases of urinary catheters so that we could survey the symbols that were in commercial use. Dr. Bix has served as an expert witness in a case concerning the design of medical device trays. 72 D. C. SEO ET AL.