What Kind of Information About Marginal Donors Is Available Through Sources Other Than Health Care Professionals for Patients on the Waiting List for Organ Transplantation? (e15)

Background: The current organ shortage has necessitated expanding the criteria for potential donations to marginal donors (older or sick donors whose organs would have been considered unsuitable before). In France, physicians are not required to provide information to recipients about marginal donors except for hepatitis C or hepatitis B infection and non-heart-beating donations. We hypothesized that patients can be informed about these risks by other information sources than health care professionals, such as websites and patient associations. Objective: The objectives of the study were to identify the main health information sources of transplant patients other than health professionals and to evaluate the information provided by websites and associations to patients about the risks of transplantation from marginal donors. Methods: In this study, the information sources for kidney, liver, heart, and lung patients that had already received transplants or registered on waiting lists were identified by a survey in four transplant centers. Further, the information proposed by French and English language websites and patient associations were evaluated, respectively, by a systematic review of websites and a survey among the presidents of kidney, liver, heart, and lung patient associations. Results: For the first survey, (367/402) 91.3% responses were registered. Apart from health professionals identified as the principal information source (363/367) 98.9%, 19 liver and 28 heart patients searched for information on the websites, while 37 kidney and 42 lung patients were more informed by patients’ associations. Our two last surveys showed that information about marginal donors is accessible by websites and (10/34) 30% of associations. All of the 60 Internet documents evaluated on French language and English language websites proposed information about marginal donors. Otherwise, (52/65) 80% of these documents were dedicated to health professionals and contained specialized information, difficult to understand by patients. Certain associations, (20/34) 59%, provided information about the risks of transplantation. There were 45/115 patients considering associations as their main information source that were informed by an association’s website. However, only (5/22) 23% of associations communicated the risks of transplantation with patients through their websites. Interact J Med Res 2015 | vol. 4 | iss. 3 | e15 | p.2 http://www.i-jmr.org/2015/3/e15/ (page number not for citation purposes) Kamran et al INTERACTIVE JOURNAL OF MEDICAL RESEARCH


Expanded Criteria Donors
The current organ shortage has necessitated expanding the criteria for potential donation to donors whose organs would have been considered unsuitable before. Kidneys of marginal donors had been used for transplantation in France since 1998 [1] (French national agency of transplantation, ABM's, report). These donors are termed "marginal" donors, also referred to as "expanded" or "extended" criteria donors. However, these concepts are not clearly defined. "Expanded criteria" is the term most intended for kidney donors. The United Network for Organ Sharing (UNOS) [2] first described these criteria in the United States. They include age 60 or older, or between 50 and 59 with at least two of the following conditions: history of hypertension, creatinine level greater than 1.5 milligram/deciliter, and death caused by a cerebrovascular accident.
The results of our research on Google AdWords in July 2013 showed that the French people did not search some specialized keywords such as "marginal donor" or "expanded criteria donor" on the Google search engine, whereas they searched, on average, 590 times per month the keywords "risks of transplantation" [3]. This means that they want to know more about the risks of transplantation, but that they do not know the specialized keywords. In contrast, people of the United States searched, on average, 170 times per month the specialized keywords "expanded criteria donor" [3]. To find out the reasons of this difference, the main health information sources for patients, and information proposed by them, should be identified.
We thought that patients registered on the French national waiting list (NWL) might not be always informed about the risks and benefits associated with the transplantation of organs from marginal donors by the health care professionals in charge of their care. This hypothesis was supported by the results of a study in submission process that we performed among physicians responsible for transplant centers about the information proposed to patients concerning the risks and benefits associated with the transplantation of organs from marginal donors.
The current regulation makes it mandatory that physicians inform the potential recipient about a donor with a history of hepatitis C or hepatitis B infection [4], or a non-heart-beating donor, especially for a liver or kidney transplantation [5]. Additionally, in France, a law enacted in March 2002 [6] requires that patients be informed about every risk that might occur during a medical procedure. However, no laws, regulations, or instructions force physicians to provide information to recipients about the risks and benefits of organs available from other types of marginal donors.

Information Proposed to Patients About the Risks and Benefits of Marginal Donors
We only found two studies [7,8] focusing on information proposed to potential recipients about the risks and benefits associated with marginal donors. The authors supported the idea that patients should receive information and may take part in the decision-making process of whether or not to be transplanted with organs available from a marginal donor.
Adult patients could search for information sources other than health care professionals. According to the results of the "Health Online 2013" survey, realized by the Pew research center [9], adults from the United States got information, care, or support from: a doctor or another health care professional (70%), friends and family (60%), and other patients having the same health conditions (24%). During the past year, certain adults (35%) said that they have searched using the Internet for their or others' diagnoses. In France, the survey conducted for the National Council of the College of Physicians in 2010 [10] found that the main French health information sources were health care professionals (89%), Internet (64%), relatives (64%), and pharmacists of retail pharmacies (63%).
The hypothesis underlying this study was that apart from health care professionals, Internet websites and patient associations could be two main information sources for transplant patients. Indeed, using the Internet to find information became a current practice, particularly among young people. Furthermore, patients with chronic diseases such as those leading to transplantation often gather in patient associations. Therefore, the objectives of this study were to: (1) identify the main health information sources of transplant patients; (2) evaluate the use of information sources such as websites or patient associations by patients who had already received transplants or were on the NWL; (3) examine the information provided by patient associations and websites about the risks of transplantation from marginal donors; and (4) compare the information provided by websites in the French and English languages.

Survey Design
A cross-sectional survey using an anonymous electronic self-questionnaire was carried out from October 2013 to March 2014 among the presidents of kidney, liver, heart, and lung patient associations to examine the information proposed by these associations to patients about the risks associated with marginal donors. The link of the electronic questionnaire was sent to the presidents of associations in partnership with the ABM. The ABM is recognized as the medical, scientific, and ethical authority in the field of harvesting and transplant of organs, tissues, and cells in France.

Population and Setting
The included patient associations were, first, three national federations of kidney, liver, and heart-lung in Paris that federate regional associations (24 for kidney, 11 for liver, and 9 for heart-lung), and then independent associations (3 kidney associations and 1 association for cystic fibrosis). Regional associations were also questioned because their attitude regarding their activities can differ from the national attitude for cultural reasons.
The objectives of these associations are to inform and support patients and their families in the treatment process before or after transplantation, and help them to improve their quality of life.
The associations supporting only tissue or organ donation, or not receiving transplant patients, were excluded.

Data Collection
The questions were selected based on relevance to our study questions: "Which health information about risks related to transplantation from marginal donors are proposed by associations?", and "How could patients receive this information?". Before deployment, the presidents of three principal French federations of kidney, liver, and heart-lung associations reviewed the questions and were asked to give feedback on whether the questions were understandable for presidents of associations, and whether any questions seemed out of place. Their feedback was incorporated into the survey by 2 of the researchers.
To inform presidents about the context, the purpose, the length of time for the survey, the name of sponsors, and contact information, a leaflet was prepared.
The link of our electronic anonymous questionnaire on the "Survey Monkey" website and an information leaflet were sent by email to the presidents of patient associations. The questionnaire was posted on one page with 10 questions on the website of "Survey Monkey" that captured all of the responses.
The respondents were able to review and change their answers before final validation. Before access to the questionnaire, the Internet Protocol address (numerical label assigned to each computer) was verified by "Survey Monkey", and a visitor could not respond twice to our questionnaire.

Statistical Analysis
For all three studies, standard descriptive statistics were performed as appropriate.
In the first survey, a bivariate analysis was performed to identify differences between the information sources of patients in kidney, liver, heart, and lung transplantation. To analyze the answers to multiple choice questions, we chose to consider the distribution of the answers rather than the distribution of the patients.
In the second survey, a bivariate analysis was conducted between the five variables defined previously to search for contrasts. The documents appearing on two websites or dedicated to two organs were considered as two separate documents and counted twice.
Data were compared using chi-square test and Fisher's exact test as appropriate. The level of statistical significance was set at P<.05 using SPSS statistics software version 17.0.0.

Second Survey
In total, sixty documents were found on the first pages of the Google search using each keyword (30 written in French and 30 in English). There were five documents that were dedicated to both kidney and liver, so the total analyzed was 65.
There were (32/65) 49% and (13/65) 20% of documents that were dedicated, respectively, to kidney and to liver transplantation. There were two documents of 65 that were dedicated to lung transplantation, but we found no documents in the field of heart transplantation. There were (18/65) 28% of documents that did not determine a specific organ. Among this last category, (15/18) 83% were found using the keywords "non-heart-beating donor" in French or in English. Searching for "marginal donor", (6/11) 55% and (7/10) 70% of the documents were dedicated, respectively, to liver transplant on websites in the French language and to kidney transplant on websites in the English language. Searching for "expanded criteria donor", most of the documents were related to kidney transplant both in French and in English. No document was dedicated to heart transplantation.
The only keywords that allowed finding information about ethical aspects were "non-heart-beating donor" in English or in French.
Among the Internet documents, (52/65) 80% were intended for health care professionals and (13/65) 20% for patients, knowing that the documents dedicated to the general population were counted as documents dedicated to patients. The sources of these last documents were different in each language. The French language documents were proposed by a French association (4/7), the ABM (1/7), a Belgian association (1/7), and a Swiss foundation (1/7). The English language documents were proposed by American hospitals (3/6), Wikipedia (1/6), and an American university (1/6).No English language documents intended for patients appeared in the Google search for "marginal donor".
The documents published after 2009 were published more on French language websites (P=.001), dedicated to patients (P=.004), composed of general information (P=.003), and available to all Internet users (P=.024) than those published before.
There were (20/34) 59% of these associations that provided information to patients registered on the NWL about the risks of transplantation related to surgical procedures or to the risks associated to donors (ie, donor with hepatitis B or C, or marginal donor) or both (Figure 1 shows this).
The presidents of 12 associations did not answer the question asking for their ways to communicate with patients about the risks of transplantation. Among the 22 presidents who responded, (13/22) 59%, (8/22) 36%, (7/22) 32%, and (5/22) 23% of associations communicated with patients, respectively, by discussion groups involving patients that had already received transplants and patients registered on the NWL, patients' meetings in hospital, written communication, and websites.
There were (24/32) 75% of the presidents who confirmed that transplanted patients and patients registered on the NWL could share experiences either by mentoring or by punctual meeting.

Principal Findings
Through one study including three surveys, we identified the information sources of transplant patients other than health care professionals, evaluated the information proposed by websites and patients associations about the risks of transplantation from marginal donors, and compared the information proposed by websites according to language, French or English. Patients' knowledge about marginal donors was evaluated among kidney, liver, and heart transplant patients, and only a small proportion had heard about this type of donors.
For the first survey, 367/402 (91.3%) responses were registered. Apart from health professionals identified as the principal information source, 19 liver and 28 heart patients searched for information on the websites, while 37 kidney and 42 lung patients were informed by patients' associations. Our two last surveys showed that information about marginal donors is accessible by websites and (10/34) 30% of associations. All the 60 Internet documents evaluated in websites in the French and English language proposed information about marginal donors. Otherwise, (52/65) 80% of these documents were dedicated to health professionals and contained specialized information difficult to understand by patients.
To our knowledge, the surveys focusing on health information sources [9,10] used the samples of the general population, but not patients. In our study, 338 patients had already received transplants and 29 patients on the NWL were questioned about their own information sources.
We focused on information rarely proposed by health care professionals to patients and tried to evaluate the accessibility of this type of information using other sources. Therefore, information proposed by other sources is considered as an alternative. Other studies [15][16][17][18] assessed the quality of information proposed to patients by other sources than health care professionals and the impact of this information on the physician-patient relationship. These studies suggested that information proposed by other sources was complementary to that proposed by health care professionals.
A survey among 3867 renal patients from 36 countries [19] found that health care professionals were more frequently scored as giving helpful information than patient organizations, websites, or social media. Our study also found that physicians and health care professionals were the most important information sources for transplant patients. Nevertheless, when an interested patient felt they were not being sufficiently informed, he or she could search the information somewhere else.
The source of Internet documents intended for patients was different in French and English language websites. Transplant centers offered half of the English language documents. While French patients' associations prepared more than half of the French language documents intended for patients. Internet documents provided by health care professionals may be more trusted by patients than those provided by other sources. Our study found that physicians or other health care professionals informed all patients.
According to our three surveys, the ways in which the associations informed the patients about transplantation were not adapted to patients' behavior. On the one hand, (45/115) 39.1% of patients considering an association as their main information source were informed by the association's websites. However, only (5/22) 23% of the associations communicated the risks of transplantation with patients through their websites. On the other hand, more than half of the presidents of French patient associations stated that they mostly provided information to patients by "discussion groups involving already transplanted patients and patients registered on the NWL". Communication tools proposed by these associations may not be well adapted to the patients' demands. Nevertheless, only 15 patients considering associations as their main information source have already participated in a discussion group.
Accessibility of information proposed by other information sources than health care professionals depended on several factors: the knowledge of keywords by Internet users, the ability of searching in several languages, organ type and patient sociodemographic, and psychological characteristics.
The main information concerning marginal donors could be found by searching the specialized keywords. Searching simpler keywords, frequently used by patients (ie "transplantation" or "graft"), the risk communication was limited to "surgical risks" or "transplant rejection". So, to access to useful information about marginal donors, patients should know that specialized keywords exist and know them.
Structured therapeutic education and using specialized keywords about marginal donors may facilitate Internet searches for US citizens. In the United States, therapeutic education programs [2,[20][21][22] are organized for patients on the NWL and propose information about the risks of transplantation from marginal donors to patients. Moreover, the content of these programs are available on the websites of transplant centers. Therapeutic education about transplantation is not yet common in France. We found that only (51/367) 13.9% of French transplant patients have participated in a therapeutic education program. These programs are mostly organized for patients who had already received transplants and propose the information about post transplant care, particularly medication [23][24][25][26][27][28]. Additionally, the contents of these programs are not available on the websites. Therefore, bilingual patients who search for information both in English and in French on the websites may be more informed than others.
The type of organ is another factor influencing the type of information provided to patients. The Internet documents about marginal donors were often dedicated to kidney or liver transplantation, rarely to lung, and never to heart transplantation. In contrast, 44/51 patients (86.3%) participating in therapeutic education programs were heart or lung patients.
Only (47/280) 16.8% of kidney, liver, and heart patients have already heard about "marginal donors". Physicians of transplant centers informed most of these patients.
The most important information sources apart from health care professionals were, for heart and liver patients, websites in the French language, and transplant associations for kidney and lung patients. Using the Internet as a source for heath information continues to increase. However, kidney and lung patients may have a special opportunity to be informed by patient associations. Kidney and lung associations are among the oldest associations of patients. They are devoted to patient information on their disease and their treatment since their beginning, before and after transplantation. Lung patients, particularly those suffering from cystic fibrosis, and kidney patients, during their dialysis, usually have contact with patient associations.
A minority of active patients, more able to understand medical topics than others, could be more informed than other patients thanks to reading the documents dedicated to health care professionals on websites, discussions with patients in the associations that had already received transplants, and searching for information in other languages on the Internet.

Limitations
Our study had some limitations, especially concerning the first survey. First, most respondents had already received a transplant. This can be explained by the necessary recurrent outpatient visits for follow-up, while patients waiting for transplantation were supposed to come just once for pre transplant assessment. Second, the physicians of the lung transplant center chose to delete the question of our questionnaire concerning patients' knowledge on marginal donors. Providing information to patients about marginal donors remains a taboo subject. Health care professionals, particularly lung and heart physicians, do not want to talk about this with patients. Furthermore, the physicians of kidney and liver transplant patients are not really much more prone to speak about this topic with their patients. Therefore, the transparency about marginal donors by health authorities could help to break down this taboo.

Conclusions
Currently, patients want to be more informed by other information sources than health professionals, particularly by the websites. Patients could trust more websites if their physicians confirmed the reliability of information proposed by this source and informed patients about specialized keywords. Patients even expect physicians to recommend specific websites to them [29]. So the websites of universities, transplant centers, and associations should be improved also for dedicating the information for patients. It allows patients to have access to reliable information sources. Another conclusion of this study could be to improve the capacity of health professionals to communicate with patients, particularly by training the physicians in shared decision-making skills.

Introduction
The birth of a child can have great impact in any family system, let alone when it is the birth of a child with a disability or a facial difference such as a cleft lip and/or palate (CLP). In this situation, parents may not only have to adjust to the expected demands of parenthood but also manage challenges resulting from their child's congenital anomaly [1]. Parents anticipate and worry about countless stressors, including multiple reconstructive surgeries, feeding hurdles, dental agenesis and malalignment, facial-skeletal disharmony, speech impairment, strained social relationships, and compromised self-image for the child [2,3].
In addition to concerns regarding the physical complications, parents are often overwhelmed by the task of seeking information to better understand their child's condition, treatment, and management. One resource that has the potential for providing an abundance of information is the Internet, which is being increasingly used to answer questions and gain knowledge. A 2006 study by Fox [4] showed that over 100 million Americans used the Internet for health information searches in that year, with 8 million Americans searching the Web each day. She also observed that 514 individuals (25% of the sample investigated) reported feeling "overwhelmed" when acquiring online information on health-related topics [4]. This overwhelming sensation could be due to the vast number of resources, to problems with understanding the content of the website, and also its trustworthiness.
Comprehension is an important factor and one that is often overlooked as families are encouraged by health care professionals and those who provide other services to the family to turn to resources such as the Internet for support and information. Potentially useful information that could educate and improve parenting skills is often serving no practical purpose due to its readability level. In 2001, Berland [5] wrote "One must be able to comprehend the material in order to be able to utilize it." The utilization of the information to better care for a child with a cleft is ultimately the main goal.
According to Graber et al [6], the reading level of a person in the general population is usually lower than that of the final grade level he/she completed. Furthermore, those who navigate the Web in search of health-related information face another layer of difficulty: clinical terminology (ie, medical and dental). According to D'Alessandro et al [7], the US national reading level is in the range of 8th to 9th grade. They recommended that health-related websites and printed literature should aim for a 6th grade reading level. This recommendation has been adopted by the Center for Disease Control and Prevention (CDC), the American Medical Association (AMA), and the National Institutes of Health (NIH). Finally, they also concluded that even those individuals with higher readability levels prefer to read easier documents [7].
Parenting includes, but is not limited to, everything that supports the physiological and social development of a child besides the basic responsibility of providing shelter and food. Literature on parenting is vital to those wishing to have guidance with parenting practices or looking for answers on how to deal with health-related, social, or psychological issues when raising a child. Parenting practices are all the actions parents take to socialize children's behaviors and, as such, they primarily influence the shaping of children's behavioral adjustment.
Resources that teach and enhance parenting practices allow parents to promote ideal development of their child by optimizing their potential [8]. Parents also have the power to create an environment that rears a child in a desired direction [9]. A study conducted by Klein et al [10] showed that mothers, specifically of children with craniofacial anomalies, experienced higher levels of emotional and social adjustment in comparison to parents of unaffected children and, therefore, demonstrated greater need to have their parenting practices coached. Since CLP children are at higher risk to develop psychosocial adjustment problems, it is possible that their parents are using the Internet as a resource to avoid, combat, or decrease the frequency of such challenges [9]. As such, accessible and understandable literature on parenting can provide some stress relief and much needed guidance.
The purpose of this study is to determine the readability of information related to parenting a child with CLP available to the public via the Internet. Our hypothesis is that the readability of such materials is greater than the 6th grade level recommended by the CDC, AMA, and NIH.

Methods
This study reports the findings of a Web search conducted using the Google search engine in February, 2012 using the terms "parenting cleft lip and palate." A total of 1,980,000 links showed up in 0.39 seconds. The first 5 pages of results were analyzed based on the findings described by Jansen and Spink [11] who observed that most users explore the results displayed in the first page only. We expanded our analysis to include the first 5 pages to account for computer and display variances in font size and formatting. . After directly contacting the author of one of the books, it was learned that its reprints were no longer being published and we were thus unable to perform the readability test on it. Therefore, a total of 7 books were included. A thematic analysis of the content of each resource was performed so that patterns of information could be recorded. After familiarization with the data, initial codes were generated and generic themes emerged from the preliminary analysis. Lastly, a list with the most frequent themes (ie, author, country of origin, information specific on cleft, terms and definitions, etiology, team approach, feeding, surgery, orthodontics, speech, hearing, links and paths to request information, social support, as well as information on parenting practices) was created. Websites, books and booklets/factsheets had their content analyzed for the presence or absence of each theme. The data collection process is shown in Figure 1.
Readability for the 15 websites, 7 books for parents, and 8 booklets/factsheets was tested using the Flesch-Kincaid Grade Level, the Fog Scale Level, and the Simple Measure of Gobbledygook (SMOG) index. These tests were selected to be used in this study for the following reasons (1) they were readily accessible on the Internet and free of charge, (2) they have been used in sociology, healthcare, and publishing/media literature [6,7,[12][13][14][15][16], (3) they were fairly easy to use, and (4)  Website URLs were copied from an Excel spreadsheet into a browser using the latest available version of Microsoft Office Word software. Once the Web page was displayed, the text from that link was copied in its entirety. The text was copied into a text box available on the online readability calculator as previously described by Antonarakis and Kiliaridis [12]. The first and last 50 words of each chapter for all 7 books were typed out into a Microsoft Word document and later pasted into the text box available on the online readability calculator. The first and last 100 words of each booklet and factsheet were typed into a Microsoft Word document and each was analyzed separately using the method mentioned above.

Results
Only 16 websites (38%, 16/42) mentioned direct or indirect guidance for parenting practices. Of these, 15 (94%, 15/16) entered the readability analysis; the one remaining link was not used because it was protected against copying of information.
Overall, the analysis of the Fletch-Kincaid resulted in a mean score of 8.93, which correlated to a 9th grade reading level. The Fog scale gave a mean score of 11.50, which correlated with a "hard to read" readability level. Finally, the SMOG scale resulted in a mean score of 8.24, implying an 8th grade reading level.
All books written for parents included either direct (20%, 5/25) or indirect (12%, 3/25) parenting advice. One book could not be used since it was no longer published. The readability analysis for the books (n=7) resulted in a mean Fletch-Kincaid score of 9.76, which correlated to a 10th grade reading level. The Fog scale gave a mean score of 12.54, which correlated with "hard to read". Finally, the SMOG scale resulted in a mean score of 8.96, implying a 9th grade reading level.
Factsheets (n=3) and booklets (n=5) were analyzed because they contained parenting advice that was either direct (88%, 7/8) or indirect (12%, 1/8). The readability analysis for the booklets resulted in a mean Fletch-Kincaid score of 10.44, which correlated to a 10th grade reading level. The Fog scale gave a mean score of 14.54, which correlated to "difficult to read". Finally, the SMOG scale resulted in a mean score of 10.10, implying a 10th grade reading level.
Even though the booklets/factsheets had the greatest readability scores among the media resources analyzed, when we tested the mean readability difference among the 3 groups using Kruskal-Wallis, we could not find any statistically significant difference (at a power of 80%). This suggests that all 3 groups of media resources presented similar mean reading scores for the 3 tests (Fletch-Kincaid Grade Level, Fog Scale Level, and SMOG Index) ( Table 1) and were all considered "hard to read."

Principal Findings
The readability analysis of websites ranked in the top 5 pages of a Google search, as well as books and booklets/factsheets accessed through those links, was performed based on the models proposed by Antonarakis and Kiliaridis and Fitzsimmons et al [12,15]. Google was the search engine of choice because in 2008, Lewandowski [20] noted that users looking for health-related issues perceive it as the best search engine due to its ability to deliver a high ratio of relevant results and descriptions per search.
The Internet is a popular source of parenting information, as well as any consumer-oriented healthcare information, that is convenient and of relative easy access [15]. In agreement with Antonarakis and Kiliaridis [12], we believe that there is an urgent need to guide practitioners and those involved in CLP care towards the most useful, reliable, readable, and complete websites, so that they can direct patients seeking information to these sites. The World Health Organization (WHO) recognizes the problem related on any health topic [21] and has proposed the creation of and supervision of a ''health'' domain to impose standards of quality on all disclosed materials.
This study did not investigate the quality and/or utility of the information, rather it focused on the readability aspect of the information available to parents. We observed that, overall, the contents of the websites varied greatly in covered themes as well as in quantity. Of all the websites, 29 were loaded with medical technical information, while 13 were blogs and forums of lay people sharing their life experiences. This is a similar finding to Antonarakis and Kiliaridis [12] who also concluded that the information available to CLP families on the Internet is vast and highly variable. The consequence of such abundant and variable sources of information is yet to be determined.
However, all of these resources are only useful if the consumer understands their content [15].
With respect to readability, Antonarakis and Kiliaridis [12] found that website information on orthodontics for the CLP population is on average at the 8th to 9th grade level. Our research, which investigated the topic of "parenting practices," found the level to be slightly higher at a 9th to 10th grade. For instance, the Cleftline website [22], which is one of the most popular websites, has a reading level of 11th grade. On the other hand, the Specialchildren and Café Mom websites [23,24] had reading levels below 6th grade. Interestingly, Specialchildren is a website dedicated for parenting children with special needs, and was most likely designed with the goal of establishing clear communication with families. Café Mom is also a parenting website designed by a marketing corporation (CMI Marketing, Inc) which probably used effective communication strategies in its design.
Our findings for books, booklets and factsheets had a similar range of 9th to 10th grade. Most books written by parents for parents, such as "Children with Facial Difference: A Parent's Guide" had high reading levels (11th grade) [25]. However, 2 books written by parents for parents were exceptions: "Don't Despair Cleft Repair" and "An Unconditional Love" [26,27] had scores at the 6th grade level. Books written by experienced doctors, despite the fact that they are routinely recommended by health care professionals, were considered hard to read by an adult based on the readability scores, as compared to US national literacy averages. Dr Berkowitz' "The Cleft Palate Story" [28], for instance, had the reading level of at/or above college level, while Dr Moller's book, "Parent's Guide to Cleft Lip and Palate" [29], scored at an 11th grade level. Likewise, highly recommended and used booklets from the Cleft Palate Foundation scored high on the readability test. It is not uncommon to have these booklets readily available for families in outstanding cleft/craniofacial centers in the United States.
The most difficult one to read according to our study, and perhaps one of the most popular ones, titled "Toddlers and Preschoolers" [30], rated at/or above college level. In general, booklets/factsheets had higher reading scores and were not found to be statistically different than the average reading scores for books or websites. Considering these findings, the reading level difficulty poses a problem for a large percent of the population.
When authorship and reading level were analyzed together, it was observed that resources written by parents, especially by those who write well in English such as Terri Mauro (BA in Literature) from the Specialchildren website [23] and Karen Lipman, author of "Don't Despair Cleft Repair" [26] presented lower grade reading levels. Likewise, the book "An Unconditional Love" [27] written by the experienced mystery writer, Lorraine Barlett, was found to be at an "easy to read" level.
Basic reading level indicates skills necessary to perform everyday literacy activities, such as reading and comprehending information in simple documents, such as charts and forms. Below basic reading level indicates no more than the most simple and concrete literacy skills, such as locating easily identifiable information, and following written instructions in simple documents [31]. The average reading level for the American population [12,7] is 8th grade. It is important to differentiate between an individual's academic grade achieved and actual reading skill. Studies have demonstrated that one's reading level is usually lower than his/her highest accomplished academic grade [16,17]. Therefore, it is possible that most of the websites investigated in this study would not be consistent with the readability level of individuals with a high school diploma, which make up approximately 30% of the population (ie, around 42 million adult internet users in 2006) [4].
Parenting a child with CLP can be challenging because of the increased emotional, physical, and social considerations that exist related to the condition in different stages of the child's life [10]. It is expected and understandable that parents have a thirst for knowledge about their child's condition and the psychosocial adjustments needed as he/she grows. Knowledge has the potential to play a profound coping role throughout this entire process. Health care professionals are encouraged to provide parents with accurate written and oral information [18] in order to facilitate the learning and coping process. Although this is helpful, many families still turn to the Internet to address unanswered questions and concerns that arise throughout their child's treatment process [32]. Based on our findings, they are likely to face the challenge of understanding the material due to the difficult readability levels of the vast majority of media resources. In addition to general readability, adding the dimension of health-related vocabulary that is likely unfamiliar to the parent makes the text more challenging to comprehend [7]. As a result, parents do not acquire the guidance and knowledge they are seeking to incorporate in parenting practices, which could benefit their child's development.

Conclusions
Most resources tested presented with average reading scores above the US national's average literacy scores [31]. There is a vast amount of information available, especially with the growth and convenience of the Internet. However, this material may only be useful if patients are able to comprehend it [15,32]. The suggested reading level of information related to CLP should be at the 6th grade level [7]; endorsed by the CDC, AMA, and NIH. Our study found that only 4 resources (13%, 4/30) in compliance with this recommendation. The books "Don't Despair Cleft Repair" [26] and "An Unconditional Love" [27], written by parents of children with CLP, and the websites Specialchildren and Café Mom specialized in parenting practices and tips to raise children.
When considering the books, factsheets, booklets, and websites analyzed, the average readability level was between 8th and 11th grade. With the US national average at 8th grade, many parents are probably finding the text they read too difficult to comprehend. In agreement with Antonarakis et al [12], we believe that there is an urgent need to guide practitioners and those involved in CLP care towards the most useful, reliable, readable, and complete websites, so that they can direct patients seeking information to these sites.

Recommendations
There are multiple ways in which this useful material can become more readable and relevant for parents. Some recommendations are (1) the use of short sentences and avoid passive voice, (2) limit medical jargon, explain the root of medical terminology, and break down long medical words [33], (3) avoid ambiguous words, symbols, and quotation marks [13], (4) select familiar words and use them consistently [7,13], (5) use analogies that are familiar and culturally appropriate for the target audience [13,33], (6) instead of real numbers, when conveying statistics use words a such as ''half,'' or ''one third'' [13], (7) plan and test websites as well as booklets before releasing/publishing them, (8) use free readability tests available on the Internet to improve the readability level of a text from "hard to read" to the 6th grade level [13,5], (9) use illustrations, pictures, and/or simple drawings as an effective alternative to substitute complex words or terms [5,34], and (10) explain procedures, symptoms, and treatment modalities using plain language in conversation style (eg, making use of a plain language website [35]). Comprehensible material is a necessity to foster confidence and understanding of the anomaly while promoting effective parenting practices in families with children with CLP. It is imperative that organizations test the readability of the content in their websites prior to making them available to the general population.
Illustrations or pictures may also be useful in explaining a technique or self-care procedure to a patient. Key messages can be communicated in a manner that is not demeaning to individuals with low health literacy [31]. As providers develop consumer health materials, readability-assessment tools such as Gunning FOG, SMOG, or Flesch-Kincaid may assist them to edit the writing down to the appropriate reading level. This step provides a quality check to ensure that patient-education materials meet the United States Department of Health and Human Services (USDHHS) reading-level recommendation.

Background
The Internet has become a popular medium for consumers seeking health-related information [1]. The proportion of the population regularly accessing the Internet is large and growing: The Australian Bureau of Statistics reports that 83% of Australians were using the Internet in 2012 and 2013 compared to 76% in 2010 [2]. In 2014, the Internet was predominantly accessed via desktop computer (81%) compared with 19% who used mobile phones [3]. However, there was a 33% increase of people using their mobile phone to access the Internet from 2012 to 2013 [4]. Recent data suggest that searching for health and medical information was one of the top 15 reasons for accessing the Internet among Australians over 14 years of age [5]. In addition to websites, smartphone apps represent another increasingly popular source of health information [6]. A recent US consumer survey identified that one fifth of smartphone owners have downloaded a health app [7]. It is estimated that presently there are more than 100,000 health-related apps available and, with the growth of smartphone ownership, the use of health apps will continue to rise [8].
Increasingly, parents are turning to the Internet for information and support on how and what to feed infants and toddlers in different life stages [9] including infant feeding practices such as breastfeeding, formula feeding, introducing solids, and also the type of foods to introduce [10]. A Google Consumers Survey found that expecting parents conduct Internet searches twice as frequently as nonparents [11]. However, there are concerns regarding the quality of information provided on websites and apps about infant feeding as this may lead to the adoption of inappropriate practices [12].
There is evidence to show that many eating habits and preferences are formed in infancy and childhood and carried through to adulthood [13]. Because poor eating habits such as eating too many energy-dense foods or eating too few fruits and vegetables begin in early life, there is a key opportunity to support parents to get healthy eating established in early life [14,15]. Given this, it is important that the information provided to parents is continuously updated and consistent with the latest evidence-based infant and child feeding guidelines, such as the Infant Feeding Guidelines: Information for Health Workers available from the Australian government's National Health and Medical Research Council (NHMRC) [16]. This will ensure that parents have access to sources of information that are credible and of good quality.
Presently, there is little information on the quality of websites and apps accessible in Australia regarding infant feeding practices even though various tools are available for evaluation of the quality of Web-based health information. The evaluation of quality includes assessing the website content, credibility, currency, accuracy, reliability, readability, and design [17,18]. However, there is evidence that website developers rarely use these tools [19]. Several studies have evaluated the content of websites and apps focused on health issues such as asthma, pain self-management, and warfarin intake and suggest that the quality of the information and user-friendliness of these resources varied substantially [20][21][22]. The suitability of health information is also an important aspect to consider; in addition to predicting the appropriateness of the information in terms of content and literacy demands, this also measures graphics and layout and cultural specificity [23]. While health information is widely available on the Web, many individuals with poor health and low literacy may not find the information usable [24]. An overestimation of consumer ability to comprehend the information provided on the Internet may increase the risk of misunderstanding [25].

Objectives
Given the importance of health-related information targeting infancy and early childhood, conducting an analysis on infant feeding websites and apps is timely. This work will help identify appropriateness and suggest ways in which quality and usability can be improved. This is important if we are to effectively engage consumers around the uptake of healthy infant feeding practices. The aim of this systematic analysis, conducted between December 2013 and December 2014, was to critically evaluate 4 items: quality, comprehensibility, suitability, and readability of information available about infant feeding on websites and apps.

Websites
Infant feeding websites were identified using the Internet Explorer browser and Google and Bing search engines; selection was based on the most commonly used terms in Australia [26,27]. The key search terms used for websites included infant feeding, baby feeding, breast feeding, infant feeding schedule, infant formula, formula feeding, introducing solids, introducing baby solids, solids and fussy babies, and introducing solids schedule. These key terms were identified as the most frequently used terms by consumers on Google Trends [28]. A study reports that consumers seldom read beyond the first page of search results for online health information [29]; therefore, the first 30 websites in both of the search engines were screened. The screening of the websites was conducted by researcher LW using predefined inclusion and exclusion criteria. The websites were reviewed if they met the criteria. All websites were cross-checked by researcher ST. Any disagreements regarding which websites should be included in the study were discussed until consensus was reached.

Apps
Infant feeding apps were identified by performing searches in the digital application distribution platforms for the 2 largest smartphone operating systems: the App Store for iOS (Apple Inc) and Google Play for Android (a Linux-based system currently owned by Google). The search terms were modified slightly for the medium. Revised terms included infant feeding, baby feeding, breast feeding, formula feeding, bottle feeding, baby solids, baby food, and baby weaning. All of the apps yielded from the key terms were screened for eligibility as neither the App Store nor Google Play sorts the most commonly used apps by the number of downloads. The screening of iOS apps was conducted by researcher LW, and the screening of Android apps was completed by researcher ST, both using predefined inclusion and exclusion criteria. The apps were reviewed if they met the criteria. All apps were cross-checked by researcher ST. Any disagreements regarding the inclusion of apps in the study were discussed until consensus was reached.
Inclusion criteria for selecting websites and apps for this study included being written in the English language, targeted to parents of infants up to 1 year of age, and last updated after 2002. Websites were also restricted to those which originated from Australia so advice could be compared to the NHMRC's Infant Feeding Guidelines. This requirement did not apply to apps, however, as there are limited methods to restrict country of origin in app stores; to be included they needed to provide at least information on the Australian infant feeding guidelines. The websites and apps must include information on at least one of the following topics around healthy milk feeding behaviors (breast, expressed breast milk, formula feeding, frequency, timing, correct preparation, feeding on demand, nonnutritive feeding, repeated exposure, varied exposure, and reducing exposure to unhealthy food/beverages) or healthy solid food feeding behaviors (age of solid introduction, types of food introduced, repeated exposure, reducing exposure to unhealthy food/beverages). Additionally, websites that could not be accessed due to broken/dead links; apps that were not free; and electronic books, YouTube or other videos, audio files, news, podcasts, blogs, and PDF and Word documents were excluded.

Quality Assessment
Websites Two validated tools, the Health-Related Website Evaluation Form (HRWEF) [17] and the Quality Component Scoring System (QCSS) [18,30], were used to assess the quality of websites, as they each contain different criteria.
The HRWEF tool is currently used by the nongovernmental organization Health On the Net Foundation in their code of conduct (HONcode) [31] to certify the quality of an array of health-based websites. It assesses the quality of websites by evaluating the content, credibility, currency, accuracy, reliability, readability, and design of Web-based health information. The QCSS is a tool previously used for medical website evaluations [30,32]. The assessment criteria for this tool include purpose of the content; disclosure of authors/sponsors; currency; accuracy and reliability; accessibility and interactivity; readability; and graphics/layout of information [33,34]. The scoring systems of the tools are as follows: in the HRWEF a score of not applicable (0), disagree (1), or agree (2) and in the QCSS no information (0), partial information (1), or complete information (2). A final score assessing each item on both of the tools was calculated. Websites were rated as excellent for scores of 90% or higher, adequate for 75-89%, or poor for less than 75% with the HRWEF. With the QCSS tool, they were rated excellent for scores 80% or higher, very good for 70-79%, good for 60-69%, fair for 50-59%, or poor for less than 50%.

Apps
To our knowledge there were no published, validated tools available to evaluate the quality of apps. Given this, a quality assessment tool was developed by author ST (see Multimedia Appendix 1). Tools previously developed from other studies [20,21] did not comprehensively address the quality of apps; therefore, the new tool was based on items from the HRWEF tool used for websites [17] and tools used in previous studies [20,21]. The criteria used to measure the quality of apps included the description of the app, information about the developer, design and layout, navigation, interactivity, content and accessibility, and security and connectivity of the app. The scoring system used in this tool was attained from one of the studies in which the app quality tool was developed [21]. The scoring system included 29 items which either agreed (1) or disagreed (0) that the app met the criteria and 12 items that were scored as 3 if 100% of the app met the criteria, 2 if 50-99% of the app met the criteria, 1 if 1-49% of the app met the criteria, or 0 if the app did not meet the criteria at all. The final scoring system used was similar to that of the HRWEF tool [17], where a final score rated each app as excellent for a score of 90% or higher, adequate for 75-89%, or poor for less than 75% (see Multimedia Appendix 1). The QCSS tool was also used to measure the quality of the apps.

Comprehensiveness
Comprehensiveness was an item in the quality tools that assessed the accuracy and coverage of the content available on websites and apps. In addition, assessment criteria with 8 topics and 22 subtopics based on the Infant Feeding Guidelines [16] (see Multimedia Appendix 2, with scoring system derived from [35]) were developed to evaluate the consistency of the information provided. For each topic, accuracy was scored as either correct (+1), incorrect (−1), or absent (0) in turn measuring the amount of topics covered in each website and app. Completeness, the breadth of information provided on each topic, was measured as complete (2) or partially complete (1). A final score in the quality assessment tool included 3 if 100% of information was covered/accurate, 2 if 50% or more of information was covered/accurate, or 1 if less than 50% of information was covered/accurate.

Suitability of Information
The Suitability Assessment of Material (SAM) [23] is a validated instrument, which was used to evaluate the appropriateness of information on the websites and apps for the target audience relating to literacy level, cultural appropriateness, content, and layout. The scoring system used for each item measured included not suitable (0), adequate (1), or superior (2), and each website and app was given a final rating of superior (70-100%), adequate (40-69%), or not suitable (0-39%).

Readability
The term "readability" refers to the grade level of written text. Readability is an item that was measured with the website and app quality tools and the SAM instrument. Two readability tools were used to measure the content of websites and apps: the Flesch-Kincaid (F-K) [36] and Simple Measure of Gobbledygook (SMOG) [37]. Calculations for F-K were automatically performed using a readability statistics feature available on Word Professional version 2010 (Microsoft, Redmond, WA, USA) by pasting a block of writing from each website or app on the Word document and the reading ease and grade level were recorded. The same block of writing was pasted on an online SMOG calculator that automatically calculated the SMOG and F-K reading grade levels. The average level of reading of US and Canadian adults is between 7th and 8th grade [38,39]. In Australia, literacy competence is measured using the Adult Literacy and Lifeskills Survey, which uses a ranking scale from level 1 (lowest) to level 5 (highest) [40]. As the tools used to measure readability are American, the reading level of information provided could not be compared against the average reading level of Australians. Both the website and app quality assessment tools use a scoring system of agree (2) if the reading level is 8th grade or lower and disagree (1) if the reading level is 9th grade or higher. For the SAM instrument, the scoring was superior (5th grade or lower), adequate (6th to 8th grade), and not suitable (9th grade or higher).

Stage 1: Website and App Selection
Searches were performed between December 2013 and March 2014 and rerun in December 2014. In total, 600 websites from Google and Bing and 2884 apps from the app stores for were available for screening (Figure 1). After screening and based on the inclusion criteria, 44 websites and 46 apps were evaluated for the quality, comprehensibility, suitability, and readability of the information. Of the 44 websites, 8 were published by government entities, 10 were sponsored by commercial organizations, and 26 were noncommercial sites from education/nonprofit organizations or hospitals. Of the 46 apps, 2 had university and Australian Breastfeeding Association endorsements, 1 was commercial, and 43 were from noncommercial sites. A numbered list of websites and apps included in this study can be found in Multimedia Appendices 4 and 5, and a summary sheet of the scoring criteria for each evaluation tool can be found in Multimedia Appendix 3.

Websites
Using the HRWEF tool, the majority of the websites (27/44, 61%) received a poor rating. The median score was determined to be 65% and the interquartile range was 55-86% ( Figure 2). Seven of the websites scored an excellent (>90%) rating for quality, and 10 websites received scores of adequate. Four websites stated they subscribed to the HONcode principles.
The QCSS tool revealed that 66% (29/44) of websites were rated poor with a median score of 50% and interquartile range of 36-76%. Two websites were rated excellent, 2 were very good, 7 were good, 4 were fair, and the majority (29/44) was rated poor. Of the 44 websites, 11 reported on author qualifications. Nine of the websites reported that their authors were health care professionals (nutritionists/dieticians, doctors, or nurses/midwives); the authors of 2 websites had no medical expertise (1 was a journalist and 1 was a parent). In regards to the latest content update, 8 websites had not been recently updated to suit the latest infant feeding guidelines (2012) and 7 websites did not identify the date of last update.
Characteristic differences between high-and low-scoring websites varied across the quality items measured. Most websites rated "poor" failed to provide minimal coverage of infant feeding topics, provided inaccurate information, were written at unattainably high reading levels, had not been updated recently, or failed to provide author credentials and external links.

Apps
Using the quality assessment tool to measure the quality of apps, 78% (36/46) were rated poor quality, and the median score was 65% with an interquartile range of 58-71% (Figure 2). None of the apps scored excellent, and 10 apps scored adequate. Using the QCSS tool, 91% (42/46) apps were rated poor quality; the median score was 49% with an interquartile range of 41-60%. Four apps were rated fair and 42 were rated poor. Of the 46 apps, 10 reported author qualifications-4 were health professionals (nutritionists/dieticians and nurses) and 6 had no medical expertise. The country of origin for the apps was unidentifiable, but only apps written in American, Australian, and British English were selected. Five apps had not been updated to suit the latest guidelines.
Most apps rated poor had deficits in navigability, design, and color; readability; accessibility (text size and help and search options); and breadth of coverage.

Websites
Using the Infant Feeding Guidelines to assess the comprehensiveness, there were 2 websites that scored 100% for comprehensibility, where all 8 topics about infant feeding (see Multimedia Appendix 2) were included and covered, and the information provided was accurate. Two websites had the lowest comprehensibility score (5%). Inaccurate information about particular infant feeding practices was identified on 2 websites when compared to the guidelines.

Apps
Of the 46 apps, the highest score attained for comprehensibility was 78%, and 2 apps scored zero for comprehensibility. Two of the most commonly covered topics in both the websites and apps were Topic 1, encouraging, supporting, and promoting breastfeeding (29/44 and 30/46), and Topic 8, introduction to solids (37/44 and 30/46). Figure 3, there were very few websites that provided information on all of the subtopics of the infant feeding practices measured in this study. There were no apps that covered the breadth of each topic. Topic 6, breastfeeding in specific situations, was the least covered, with only 2% of websites and no apps covering this topic. Overall, websites covered a wider range of infant feeding topics and provided more extensive information about each topic than the apps, but the completeness of each topic is low.

Websites
Readability grades for all evaluated websites are shown in Table  2. While there was some variability in the actual readability grades attained, the average was consistent across each of the tools used.
The median readability grade for websites was measured as 9 (interquartile range 8-11) using the F-K test in Word and the online F-K calculator. There were 10 websites that were written at approximately 8th grade level or below, which meets the recommended level of written health information. The median SMOG readability grade level was measured as 10 (interquartile range 7-10). Using the SMOG formula, 16 of the websites were written at approximately 8th grade level or below.

Apps
As presented in Table 2, the median readability grade level was 8 (interquartile range 7-10) for apps using the F-K test in Word and the online F-K calculator. There were 14 apps that were written at approximately 8th grade level or below which meets the recommended level of written health information. The median SMOG readability grade levels for apps were measured as 7 (interquartile range 7-8). Using the SMOG formula, 20 of the apps were written at approximately 8th grade level or below.

Principal Findings
To our knowledge, this is the first systematic analysis to evaluate websites and smartphone apps providing information on infant feeding practices. This analysis examined the quality standards of information on infant feeding available to users. It also ascertained that there is a need for the development of reliable websites or apps about infant feeding practices that are accessible to health professionals and the general public.
This systematic analysis found that the majority of the websites and apps on infant feeding had poor quality ratings. In contrast, other studies which have evaluated health-related information from websites using similar tools reported adequate ratings for the majority of included websites [22,32]. Another study analyzing apps for the management of obesity using a developed tool rated the majority of apps as fair [41]. One reason resources regarding obesity treatment and infant feeding may be of poorer quality is that a broader group of interested parties, such as journalists and parents, may be involved in website/app development. This would contrast with medical conditions where we might expect expert input and consequent improvement in quality. In turn, this may impact a number of assessed items including credibility of the source, accuracy and coverage of the information, and use of references. Low quality scores were influenced by the number of authors lacking medical backgrounds developing these resources and also the lack of information about author credibility (missing in 75% of the websites and 78% of the apps). Website credibility is one way in which consumers can make a judgment about the quality of information posted on sites [42]. Without this information, consumers may access low quality sites with misleading and inaccurate information.
Commercial websites scored the lowest quality rating, a finding consistent with other studies [34,43]. This finding supports the proposition that commercially motivated sites may set different criteria for information provision and may not represent the existing evidence-based practices [34]. It is of interest that a British qualitative study analyzing maternal accounts of trust regarding healthy eating information sources reported that food manufacturers were the least trusted source for Web-based health information [44]. Regardless, to minimize the risk of consumers accessing websites that may have misleading or inaccurate information, we propose that website developers should use a tool such as HONcode in the early stages of development. Currently in Australia, only medical apps which are used as diagnostic or monitoring tools require approval from the Therapeutic Goods Administration. General health and well-being apps are not regulated [45]. We propose that health apps should also be examined for approval before becoming available to consumers.

Certifying Health Websites and Apps
Of note, 4 websites stated they subscribed to HONcode principles. Of these, 2 websites attained excellent quality scores. Therefore, using a tool such as HONcode provides a certified endorsement to indicate good quality and encourages website developers to maintain the quality standards of the organization. A qualitative study found that online health information seekers do not commonly evaluate the credibility of sources [46]. Participants lacked the skill to assess website credibility as there was no report of using the About Us section, disclaimer, or disclosure on the websites. The participants' perceived method to assess credibility was to eyeball the available source, design, and layout of the website, language used, ease of navigation. Given this, using a certified endorsement on websites has the potential to reduce the burden for consumers to search for good quality websites and apps [47].
Another benefit of using a certified endorsement organization to regulate the quality of websites and apps is to ensure that the information shared is constantly updated and in line with appropriate guidelines; more recently updated websites and apps scored higher in quality than those with earlier dates of revision. These findings are similar to a study that assessed smartphone apps around pharmacology education and reported that apps included in their study had not been updated for several years, and the reliability and accuracy of the content were questioned [48]. However, with the rapid growth of apps and constant update of app versions, there is a need to continuously assess and regulate these sources [48]. A study that examined the evolution of asthma-based apps found that the number of apps on asthma more than doubled over 2 years [49]. Although the study's findings reported no difference in the comprehensiveness of the information available in the newer apps, they did identify improvements in the features offered. Therefore, later versions of apps scored better due to the ease of navigation, updated content, and appropriate layout and graphics. Furthermore, using a certified endorsement may be a useful strategy for policy makers to regulate the information on health websites and apps before allowing it to become available to the public. Another policy innovation might include action by the NHMRC to provide an app with the release of every new Infant Feeding Guidelines, which could be made available to parents and health practitioners. This innovation would be potentially powerful as the people responsible for reviewing the evidence could contribute directly to the dissemination strategy (the app) thus reducing any problems in translating evidence into practice.
Another factor contributing to the poor quality of the websites and apps was the level of comprehensibility, including coverage of topics and the completeness of the information on each topic about infant feeding. Our study found that most websites did not cover a range of topics nor did they provide in-depth information about each topic. Similar findings were identified in a study that analyzed online information about dementia, where very few websites covered all topics [50]. Despite the efficiency that has been associated with using the Internet to find health information, websites that lack in information and do not cover a range of topics become a limitation and are no longer a reliable source [51]. Consumers then need to access various websites or apps to obtain information about a particular health subject. Therefore, website and app designers who do not include a range of topics around health information should consider including references that thoroughly cover topics not discussed [50]. In addition to using appropriate specific guidelines and tools to develop good quality websites and apps, they should consider assessing user requirements specific to health conditions and topics in order to meet user needs and expectations [52].

Adherence to Health Information Best Practice Principles
From the analysis of this study, 3 websites addressed the widest range of topics and attained high completeness scores, as they provided an appropriate level of detail consistent with the Australian Infant Feeding Guidelines. Only 4 websites provided incorrect information. These findings are consistent with other studies which have reported on the comprehensiveness of information related to guidelines [20,42]. Incorrect information provided in resources may have serious implications, as the layperson may not be familiar with the Infant Feeding Guidelines and might be misguided in the practice of infant feeding.
This study highlights that most of the websites and apps were written at a reading level of 12th grade. This analysis is consistent with other studies [22,53] and is an important finding given that, as previously noted, the average reading level has been reported to be between 7th and 8th grade [38,39]. It is crucial that app and website developers consider literacy levels of the general population as health-related information may be challenging for users with low literacy skills (poorly educated, culturally diverse background) [54]. It is particularly important given those with the least education and lower reading levels may benefit most from well-targeted information, advice, and support.
In our evaluation of the suitability of infant feeding information, we rated the majority of the websites superior or adequate, whereas most of the apps were rated as poor. Using the SAM criteria, poor graphics and low levels of cultural appropriateness were notably deficient. This finding supports a study [53] that reported from a review of Web-based information on osteoporosis that few websites were culturally appropriate. Australia is ethnically diverse, and Internet access is high across all social groups. Given this, culturally appropriate information should be presented across websites and apps [55]. A study evaluating health information on websites about cancer therapy [56] illustrated the difficulty of presenting information to all ethnic backgrounds. As infant feeding practices can vary with different cultural backgrounds (eg, diets, religious beliefs), it is important for website and app developers to consider identifying these aspects in the early stages of development.

Limitations and Strengths
There are a number of potential limitations of this study that need to be considered. First, the study was limited to evaluating websites and apps written in the English language and websites targeting the Australian population. Therefore, the findings may not be representative of websites and apps written in other languages or from other countries. Another limitation on this point is the fact that this study included only Australian websites while the apps were accepted regardless of the country of origin. Given this, it may have influenced the findings about the comprehensibility and accuracy of the content. There is a potential that the websites may have attained higher comprehensibility scores compared to apps, as the websites would most likely include information from the Australian guidelines compared to the apps. Another limiting factor which may have impacted quality scores of apps is that app development is in its infancy compared to website development. The fact that there is not yet a published quality tool to measure apps enforces the point that there is still much research that needs to be undertaken around health-related apps. Furthermore, Internet and smartphone apps are continuously updated, limiting the likelihood of receiving similar findings using the search terms from this study if it were replicated. To minimize this limitation, the author used Google Trends to identify commonly searched terms around infant feeding practices. Another limitation identified is that the subjective nature of some quality and suitability criteria may impact variability in scoring. Two researchers conducted searches for websites and apps and measured quality and suitability, but only one of the researchers cross-checked the websites and apps. An important strength of this study was the use of 2 different tools to measure the quality and readability of the websites and apps, a method which in turn enabled a comparison of the results.

Conclusion
It is evident that there are key areas for improvement to increase the utility of information related to infant feeding practices on websites and apps. A majority of websites and apps were of poor quality and had inappropriately high reading levels; few were given a good rating. There were no apps in this study which addressed all of the topics from the Australian Infant Feeding Guidelines. Government implementation of policy or certification systems such as HONcode would enable consumers to identify reliable and appropriate information. It would also would ensure that the readability level is appropriate for vulnerable populations. Involving users early in the development of health apps is advised as establishing ways to merge user requirements with evidence-based content to provide high-quality apps.

Acknowledgments
The research reported in this paper is a project of the Australian Primary Health Care Research Institute, which is supported by a grant from the Australian government Department of Health and Ageing. The information and opinions contained in it do not necessarily reflect the views or policies of the Australian Primary Health Care Research Institute or the Australian government Department of Health and Ageing. The authors would like to acknowledge the input of Louisa Wilson who assisted with the screening process and analysis of the websites and apps included in this study.

Conflicts of Interest
None declared.

Multimedia Appendix 1
Quality criteria assessment for smartphone apps.

Multimedia Appendix 2
Information guide sheet for content (accuracy and coverage).

Multimedia Appendix 3
Summary sheet of the scoring criteria for evaluation tools and items measured.

Multimedia Appendix 4
List of websites included in this systematic analysis.

Lack of Donors in German Transplantation System
The German transplantation system is in a crisis due to a lack of donor organs. About 12,000 patients are waiting for an organ graft [1]. Every year more than 1000 patients in Germany die because they cannot be supplied with an organ graft in time [2].
Organ donation rates in Germany decreased constantly over the last few years. In 2012, there were only 1046 deceased organ donors. These were 12.8% fewer donors compared to 2011, and it is the lowest number of organ donors since 2003 [3]. In 2013, this number decreased again to only 876 deceased organ donors [4]. The number of new registrations on the organ transplant waiting list increased from 8264 patients in 2004 to 10,106 patients in 2013 [5]. These two opposing developments are aggravating the lack of donor organs, creating a dramatic situation.

Information Campaigns Used to Increase Organ Donation Rates
Information campaigns are one of the main approaches to increase organ donation rates [6]. Organ donation information campaigns are dedicated to attract the attention of the targeted audience to the issue. Once the attention is directed toward organ donation, the distributed information material should help the recipients to make a balanced decision based on the presumingly increased personal level of specific knowledge. In November 2012, the German Transplant Act was updated to support this approach. Since then, health insurance funds are obliged to inform their members over 16 years of age about organ donation. The information has to be provided in an objective manner. The information material has to be distributed every two years including an organ donor card form. However, there is no obligation for the members to fill and sign an organ donor card [7].
Generally, educational information campaigns have the potential to improve the willingness to donate [8][9][10][11][12][13][14][15][16]. Still, the effect of this nationwide information campaign toward specific knowledge concerning organ donation needs to be quantified using an objective knowledge score. We wanted to know the answers to the following questions. What is the effect of conducting information campaigns by unsolicitedly distributing written information (flyers) to the population? Does distribution of written information lead to actual reading and processing of the information, and ultimately to an increased declarative knowledge of the participants? Does this distribution lead to an increase in organ donor card holders? Is an increase in knowledge leading to an increase in organ donor card holders? We raised the hypothesis: The willingness to sign a donor card rises due to the subsequent increase of specific knowledge by receiving the information material of the health insurance funds.

Survey
The aim of this study was to evaluate the influence of a nationwide information campaign on the willingness to sign a donor card. The information campaign was conducted by the health insurance agencies, as enforced by law. We conducted an open Internet survey from June 10 to July 18, 2013 using soscisurvey.de as the questionnaire tool. Our target population was the general population between 15 and 64 years of age. An institutional review board (IRB) approval was not necessary (decision of the IRB of the University Hospital Jena). The questionnaires were anonymous, and we did not save any personal data. On the first screen, participants were told that the questionnaire would take 10 minutes. We did tell the topic of the survey, but we did not tell its purpose to avoid bias. The investigators and their contact details have been displayed. The questionnaire design was based on the literature of Kuckartz, Porst, and Raab-Steiner [17][18][19]. The survey comprised 44 items. A maximum of 10 items per screen were distributed over 16 screens. The participants were able to change their answers through a "Back" button. There was no review step displaying a summary. We did not use cookies, and did not save the participants Internet protocol address. In conclusion, theoretically, participants were able to participate more than once. Questionnaires that terminated early were also analyzed. We did not exclude questionnaires due to atypical timestamps. We performed a pretest and distributed the questionnaire in the revised final version. The survey questionnaire is appended as multimedia appendix (see Multimedia Appendix 1).
The hyperlink to the survey was distributed via 202 Facebook groups of all kinds. To avoid bias, we did not use any organ donation group or thematically similar groups. In order to take advantage of the so-called "snowball effect" [20], we recommended the users to share the hyperlink via Facebook. A table of all Facebook groups is appended as multimedia appendix (see Multimedia Appendix 2). In addition, we used Facebook advertisement that was shown 141,366 times to different Facebook users. The hyperlink was also distributed using mailing lists of medical faculties in Germany.
To explore the specific knowledge concerning organ donation, five factual questions with different levels of difficulty were asked, see Table 1. The following response options were offered: two false answers, the correct answer, and "I don't know". To avoid bias, these four response options were mixed randomly for every single questionnaire. A sum of 0-5 correctly answered questions could be achieved resulting in the "specific knowledge score". This new variable was taken as a marker for the individual knowledge concerning organ donation.
At the time of the survey, some, but not all, health insurance agencies had already sent their information material to their members [21]. Therefore, it was possible to compare two different groups. We call participants prior to receiving information-material "uninformed participants". Participants after receiving the material are "informed participants". The group of organ donor card holders among "uninformed participants" was taken as the control group to explore the effect of the educational intervention. The relative difference in organ donor card holders was considered to be the effect of the information campaign. a The correct answers are underlined. The relative proportion of participants' responses is given for every question.

Statistics
Due to our distribution method, we assumed a distinct overrepresentation of participants of the medical sector. To minimize this bias, we weighted the percentage of the medical sector to realistic 9.52%. We calculated this percentage based on the following numbers: In 2012, 54,154,000 inhabitants age between 15 and 64 years (our target population) lived in Germany, of whom about 5,155,000 inhabitants worked in the medical sector [22,23].
Descriptive statistical analysis was carried out. We compared the different quantitative variables using Student's t test or Mann-Whitney U test, qualitative variables using chi-square test. P values < .05 were considered statistically significant.
Statistical analyses were performed with IBM SPSS Statistics 21.0 for Windows (IBM Corporation).

Overrepresentation of Academics, Participants of the Medical Sector, and Younger Participants
A total of 2484 participants took part in our survey. There were 65.81% (1594/2422) that were female. The youngest respondent was 14 and the oldest 77 years old (mean age 29.9, SD 11.0, median 26.0, interquartile range [22][23][24][25][26][27][28][29][30][31][32][33][34][35]. Participants from all educational levels were reached with our survey, albeit with an overrepresentation of high education compared with a statistic from the German Federal Statistical Office [24]. However, the statistical comparison of the epidemiological data did not reveal any significant difference between the "informed" and "uninformed" population ( Table 2). As expected, we found an overrepresentation of participants from the medical sector (62.49%, 1533/2453). If not stated otherwise, all values are presented based on a percentage of participants of the medical sector weighted to 9.52%. These results indicate that receiving information material leads to a higher percentage of organ donor card holders. Actually reading this material leads to an essential increase in the percentage.

Principal Findings
Our survey used the unique opportunity of conducting a study on a nationwide intervention without intervening by us.

Information Campaigns Lead to More Organ Donor Card Holders
The two groups of participants did not show any differences in age, gender, working sector, or level of education. Therefore, we attributed the observed difference in organ donor card holders in the "informed group" to the "uninformed group" to the educational intervention of the health insurance funds.
A study of Techniker Krankenkasse revealed that 31% of health insurance policyholders of this particular health insurance fund were donor card holders, compared to 21% among the general population. This health insurance fund was the only one that had sent information material to its members at that time [25]. This was a first hint that the information campaign of the health insurance funds was successful, and it matches our findings.
Our results indicate that receiving information material leads to a higher percentage of organ donor card holders. Actually reading this material leads to an essential increase in the percentage. Several publications indicate the potential of information campaigns to increase the number of organ donor card holders [8][9][10][11][12][13][14][15][16].
On the contrary, a study by Radunz et al did not show significant differences in the number of organ donor card holders after educational interventions with medical students. With 67% before the intervention, there were already a high proportion of donor card holders among the participants [26]. See Multimedia Appendix 3 for a table containing literature of educational interventions on organ donation and their results.

Greater Knowledge Concerning Organ Donation Leads to More Organ Donor Card Holders
We could also demonstrate that greater knowledge concerning organ donation was correlated to holding an organ donor card. Comparable to our results, several publications indicate that knowledge regarding organ donation was a significant factor for increased willingness to donate [27][28][29][30][31][32][33]. See Multimedia Appendix 4 for a table containing literature of the correlation between knowledge about organ donation and the willingness to donate.
We were able to demonstrate that participants with a medical background or working in the medical sector were more likely to hold an organ donor card than participants from other working sectors. A study on medical students by Gauher et al showed that the medical students were more likely to donate than other students due to their greater knowledge concerning organ donation [34]. Another study by McGlade and Pierscionek on student nurses found that improved knowledge leads to more positive discussion behavior of student nurses about organ donation [35]. Hobeika et al found contrary results. In a study with medical students and surgeons, they discovered that participants with less professional experience are more willing to agree to organ donation. Especially responders who had witnessed a procurement procedure showed more refusal to donate their organs [36].
Our findings demonstrate no significant correlation between the level of education and holding an organ donor card. Yilmaz found similar results [10], whereas Boulware et al found that participants with higher education level and more income were more willing to become an organ donor than participants with less education and income [37].

Information Campaign Did Not Lead to Greater Knowledge
Several publications indicate that education interventions have the potential to increase the specific knowledge concerning organ donation [14,[38][39][40].
Therefore, one could assume that the increase in the percentage of organ donor card holders was due to a greater knowledge because of the information campaigns. Our results show that this increase in knowledge was very slight, and it presumably was not decisive for the increase in the percentage of organ donor card holders. A discussion about the true reasons for this increase might be speculative. Most likely the key reason is that an organ donor card form was enclosed to the information material [7]. Offer of information and ease to fill the form were coming together and did facilitate the decision and the written documentation of this decision.

Limitations
It is possible that five factual questions were not sufficient to clarify the effect of the information material on specific knowledge concerning organ donation. Future examinations should verify the effect by using a questionnaire only containing factual questions. Our study indicates a basic level of 46.9% (281/599) organ donor card holders in our sample group. This is much more than in a representative previous study (21%) [25]. These different findings must not be related to an increase in the over-all willingness to donate organs, but may be explained by the self-selection bias. Even though we strictly refrained from mentioning the topic of the survey while distributing the hyperlink, people with more interest in organ donation were presumably more likely to participate. We used Facebook for distributing the hyperlink. This procedure is controversial because Facebook does not represent the whole population. Nevertheless, over 25 million Germans visit their Facebook profile every month. These are nearly half of all German Internet users [41]. Furthermore, Nelson as well as Fenner concluded that using social media sites such as Facebook was a successful way in recruiting participants for surveys [42,43]. Baltar and Brunet got the same conclusion, especially with the snowball sampling method using Facebook [20].