https://orcid.org/0000-0003-0031-5873
Figures
Abstract
Background
Open science practices are implemented across many scientific fields to improve transparency and reproducibility in research. Complementary, alternative, and integrative medicine (CAIM) is a growing field that may benefit from adoption of open science practices. The efficacy and safety of CAIM practices, a popular concern with the field, can be validated or refuted through transparent and reliable research. Investigating open science practices across CAIM journals by using the Transparency and Openness Promotion (TOP) guidelines can potentially promote open science practices across CAIM journals. The purpose of this study is to conduct an audit that compares and ranks open science practices adopted by CAIM journals against TOP guidelines laid out by the Center for Open Science (COS).
Methods
CAIM-specific journals with titles containing the words “complementary”, “alternative” and/or “integrative” were included in this audit. Each of the eight TOP criteria were used to extract open science practices from each of the CAIM journals. Data was summarized by the TOP guideline and ranked using the TOP Factor to identify commonalities and differences in practices across the included journals.
Results
A total of 19 CAIM journals were included in this audit. Across all journals, the mean TOP Factor was 2.95 with a median score of 2. The findings of this study reveal high variability among the open science practices required by journals in this field. Four journals (21%) had a final TOP score of 0, while the total scores of the remaining 15 (79%) ranged from 1 to 8.
Conclusion
While several studies have audited open science practices across discipline-specific journals, none have focused on CAIM journals. The results of this study indicate that CAIM journals provide minimal guidelines to encourage or require authors to adhere to open science practices and there is an opportunity to improve the use of open science practices in the field.
Citation: Ng JY, Lin B, Parikh T, Cramer H, Moher D (2024) Investigating the nature of open science practices across complementary, alternative, and integrative medicine journals: An audit. PLoS ONE 19(5): e0302655. https://doi.org/10.1371/journal.pone.0302655
Editor: Diego A. Forero, Fundación Universitaria del Área Andina, COLOMBIA
Received: December 19, 2023; Accepted: April 8, 2024; Published: May 3, 2024
Copyright: © 2024 Ng et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are included in this manuscript or posted on Open Science Framework: https://doi.org/10.17605/OSF.IO/S7G6P.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: CAIM, complementary, alternative, and integrative medicine; COS, Center for Open Science; TOP, Transparency and Openness Promotion; ASJC, All Science Journal Classification
Background
Open science is an emerging movement aimed at making scientific research and data more transparent and accessible. Researchers and those in the publishing community promote collaboration and reproducibility in research by implementing open science practices, which can include: open data, referring to readily available study data; open access, referring to accessible distribution or publication of data; and transparent peer review processes [1–5]. Issues with certain existing research processes, including limited access to research data, and low reproducibility, have led to the growing popularity of open science [6]. One of the leading organizations advocating for open science practices in research is the Center for Open Science (COS), a non-profit organization with a mission to increase transparency, integrity, and reproducibility of research. The COS created the 2015 Transparency and Openness Promotion (TOP) guidelines published in Science [7] as an incentive for researchers, journal funding models, and infrastructures to integrate open science practices into their scholarly work [8]. These guidelines comprise the TOP Factor and include open science standards such as data transparency and study preregistration [7]. The TOP Factor is a metric that quantitatively measures the extent to which the journals have adopted, require, and encourage the TOP guidelines in their publications [9]. The usage of open science practices provides many benefits for scholarly journals. For example, the public availability of study data allows for more transparent science and encourages the reuse of existing data to avoid unnecessary redundancy within the scholarly literature [10–12]. Nonetheless, there are still many barriers to implementing open science. Reduced scientific flexibility brought about by a higher focus on confirmatory rather than exploratory research, increased time commitments with preregistration, quality control, and peer review, and increased publishing costs may disincentivize researchers and journals from adopting open science practices [11,13,14]. However, the ability of open science to make the field more reliable and accessible, should drive journals to encourage and require open science practices as part of their publishing processes and, in turn, promote researchers to conduct research in line with these practices [15].
Open science practices specific to journals have been audited in various other fields of research, such as in psychology [16], the communication sciences [17] and the medical sciences [18]. The implementation of open science practices in complementary, alternative, and integrative medicine (CAIM) journals has not yet been investigated. CAIM is a field of medicine involving healthcare practices that are not generally used in conventional medical care [19]. Complementary therapies refer to those used in conjunction with conventional medicine, whereas alternative therapies are those used in place of conventional medicine [20]. More recently, integrative medicine has become an increasingly popular field which uses both conventional and complementary approaches to medicine to provide a more holistic form of care [19–21]. For the purpose of this study, all the terms described will be collectively referred to as CAIM. Though many CAIM therapies have been used for centuries, evidence-based practices are still relatively new to this field which stems from being a neglected area of original research that is performed by the relatively small subset of CAIM clinical practitioners [22–24]. Though some areas of CAIM (e.g., mindfulness and meditation research) [25,26] have a large evidence basis, the paucity of many areas of CAIM research is due to the lack of funding and financial incentive [22,27], inadequate research training of many CAIM practitioners and their uncertainty towards the fundamental tenets of the scientific process [24,28,29], and stigma from biomedical researchers around the supposed biological implausibility of CAIM therapies. Even when clinical evidence supports the biological plausibility of CAIM therapies, its efficacy is still often questioned [27]. Regardless, CAIM therapies remain a popular option among patients which justifies the need for improved research quality within this field [23,28,29]. A growing number of CAIM publications over the last several decades further highlights an increased interest in the field by researchers and practitioners [30–32]. The adoption of open science practices in CAIM can provide a greater availability of safety and efficacy profiles to deliver a higher quality of care for patients [33]. To look for trends in open science practices in CAIM journals, a journal audit can be conducted to investigate the degree of open science requirement in a journal’s manuscript submission guidelines. Thus, the purpose of this study is to conduct an audit that compares and ranks open science practices adopted by CAIM journals against TOP guidelines laid out by the Center for Open Science (COS).
Methods
Approach and open science statement
CAIM journals were identified and items from the TOP guidelines were used to extract data from each journal [34]. These items were assessed using the TOP rubric [35] to find the TOP Factor and determine which open science practices are implemented, encouraged, and required for their publications. The protocol, study material, and blank extraction form, were made available on Open Science Framework (OSF) [36].
Journal selection
A list of CAIM journals was obtained from Table 2 of Ng’s (2021) bibliometric analysis of CAIM journals [37]. Ng conducted a search on Scopus for CAIM journals. Compared to other databases such as Web of Science, Scopus includes more CAIM-categorized journals in its database [38]. The search was based on journals belonging to the category of “Complementary and Alternative Medicine” (Code 2707), of the All-Science Journal Classification (ASJC) [37].
Eligibility criteria
Literature on any CAIM topic can be found in a general CAIM journal, but general CAIM research is not likely to be published in journals on specific topics (e.g. homeopathy). Hence, the list obtained from Table 2 of Ng’s bibliometric analysis was modified to include only journals with the exact words “complementary”, “alternative”, and/or “integrative” in the journal title [37]. This step ensured the audit only included general CAIM journals rather than journals on specific topics in CAIM. The modified list was further narrowed to exclude journals that were discontinued, renamed, inaccessible via their website, incorporated into other journals already found in the list, or only published in print. In other words, the revised list only includes journals that are currently active and accessible at the time of data collection. Table 1 contains the comprehensive list of journals along with the exclusion criteria that were applied.
TOP factor
The TOP guidelines are a set of eight standards (Table 2) that journals are encouraged to adopt to enhance open science practices [7]. The TOP Factor is a metric that rates how well journals implement each of the TOP guidelines [9]. The TOP Factor is scored using the TOP rubric, a publicly available rubric created by the COS to guide journal assessment [35]. The TOP rubric includes four possible levels where journals can be ranked, from 0 to 3 [35]. Each TOP guideline has individual requirements for each score, with a higher score indicating stronger adherence to each guideline. For example, for data sharing, a score of 0 indicates that the journal only encourages or does not mention data sharing. A score of 1 indicates that the journal requires the author to disclose where data are publicly accessible. A score of 2 indicates that the journal mandates that authors share their data except for certain circumstances such as sensitive health data and proprietary data. A score of 3 implies that the journal not only requires data sharing, but also includes a verification process to ensure that the data is consistent with the reported findings in the published article [35]. The highest possible TOP Factor a journal can receive is 24.
Data extraction and assessment of journal practices
The first draft of the data extraction form was developed by BL and TP, which was reviewed by JYN with careful detail. The revised data extraction form was then circulated to HC and DM for their feedback and amended before beginning data extraction. The following items were extracted from the journals being evaluated in this study: website Uniform Resource Locator (URL), International Standard Serial Number (ISSN), Journal Citation Reports (JCR) impact factor, publisher, whether the journal is available in print or online, first year of print publication (if applicable), first year of online publication, and the specific text associated with the qualitative features as outlined by the items in the TOP guidelines.
The degree to which open science practices were required by journals was assessed using the TOP Factor [39]. Journals can be submitted for evaluation by the COS or journals can be self-evaluated using the TOP rubric and submitted to the COS for verification [34]. We self-assessed journals and submitted our evaluations to the COS through a Google form posted on their website [40]. Journals that were found to have already been assessed by the COS were used to pilot test the data extraction procedure [35]. A pilot data extraction procedure was performed by BL and TP on 5 journals in the list, of which 2 were previously assessed by COS. The purpose of the pilot extraction was to allow an opportunity for standardization among scorers before proceeding to the independent extractions. Most of the journals selected for the pilot had different publishers to account for anticipated differences in the requirement of open science practices. The extracted data was compared between BL and TP, then carefully reviewed by JYN. The remaining conflicting TOP ratings were resolved with consultation from HC and DM.
Data analysis and presentation
The CAIM journals’ open science practices were evaluated using the TOP rubric to identify the individual scores for each journal. CAIM journals were then ranked by their overall TOP Factor scores. To evaluate which TOP guidelines were most engaged with by the journals, scores for each TOP guideline were summed across all journals to find the TOP Factor.
Results
Description of journals
A total of 19 CAIM journals were included in this audit, two of which were previously assessed by the COS. A flow chart of the journal exclusion process can be found in Fig 1. Table 3 includes each journal’s URL and general information about their impact factors, whether they are available in print or online, and their first year of publication. In summary, the journals’ impact factors ranged from 1.449 to 4.473. Seventeen journals (89%) are available in print and online and two (11%) are only available online. The first year of online publication ranges from 1995 to 2022.
Overall TOP scores
Table 4 shows the TOP guideline scores and TOP Factor calculated for all 19 journals. The mean TOP Factor across all journals was 2.95, with a median score of 2 and standard deviation of 2.63. The journals with the highest overall TOP Factors were Evidence-based Complementary and Alternative Medicine (8), Complementary Medicine Research (7), Integrative Cancer Therapies (7), and Journal of Evidence-Based Integrative Medicine (7). Four journals (21%) had a TOP Factor of 0, including African Journal of Traditional, Complementary and Alternative Medicines, Chinese Journal of Integrative Medicine, Integrative Medicine, and Journal of Complementary and Integrative Medicine. Information about the specific text associated with each TOP guideline for each journal can be found in S1 Table or on OSF here: https://osf.io/jmkyn.
Data citation
Six journals (32%) provided no information regarding data citation practices and received a score of 0 in this category. Ten journals (53%) encouraged citation of datasets, though the practice was not required nor mandated for publication, receiving a score of 1. As noted in the TOP rubric, the word "should" was presumed as encouragement rather than a requirement [35]. Three journals (16%) required appropriate citation for all data obtained elsewhere, resulting in a score of 2. No journals requested enforced data citation as a condition for publication.
Data, analytical code, and research materials transparency
The majority of journals (68%) received scores of 0 for the data transparency, analytical code transparency, and research materials transparency categories. Of these, 8 journals (42%) had no mention of data sharing, while 5 (26%) only encouraged it.
Two journals, BMC Complementary Medicine and Therapies and Complementary Medicine Research (11%) required a Data Availability Statement outlining whether and where research, including code data, can be found. Another journal, Evidence-based Complementary and Alternative Medicine, required that "research published in the journal must be: as reproducible as possible–sharing underlying data, code, and supporting materials wherever able." [41] All three of these journals (16%) were ranked a score of 1 in both the data transparency and analytical code transparency categories. However, BMC Complementary Medicine and Therapies only encouraged rather than required sharing research materials upon request, which resulted in a score of 0 for research materials transparency, unlike the other two journals which received a score of 1.
One journal, Integrative Medicine Research, received a score of 2 in the data transparency category for mandating a Data Availability Statement and, where applicable, noting reasons why data may not be shared. This journal, however, only encouraged code and material sharing, receiving a score of 0 for both the analytical code transparency and research materials transparency categories.
Two journals, Integrative Cancer Therapies and Journal of Evidence-Based Integrative Medicine, (11%) required publicly available data as a condition of publication, receiving a full score of 3 for data transparency. Both of these journals did not include any specific information about code or material sharing, receiving a score of 0 for these latter categories.
Design & analysis transparency
Seven journals (37%) received a score of 0 in this category as four journals (21%) did not mention reporting guidelines at all and three journals (16%) only mentioned CONSORT, which does not encompass the majority of articles published within journals [35]. Seven journals (37%) received a score of 1 for encouraging use of reporting guidelines across several study types. As noted in the TOP rubric, the word "expect" was presumed as a requirement [35] Five journals (26%) required adherence to appropriate reporting guidelines for publication and received a score of 2. None of the journals enforced this adherence and hence were not allotted a full score of 3.
Study and analysis plan preregistration
Sixteen journals (84%) received a score of 0 for study preregistration. Preregistration, as defined by OSF, refers to the creation and storage of the research plan in a public repository at the start of the study [42]. Of these journals, five (26%) did not mention preregistration at all and 11 (58%) only mentioned information about preregistration for clinical trials. Three journals, Evidence-based Complementary and Alternative Medicine, European Journal of Integrative Medicine, and Complementary Therapies in Clinical Practice (16%), received a score of 1 for including a statement about article preregistration in their authorship guidelines.
Eighteen journals (95%) received a score of 0 for analysis plan preregistration due to no explicit mention of preregistration with an analysis plan. One journal, Complementary Medicine Research, (5%) received a score of 1 for including a statement about analysis plan preregistration in its authorship guidelines.
Discussion
Summary of results
The purpose of this study was to conduct an audit which investigates the nature of open science practices across CAIM journals. The mean TOP Factor across all journals was found to be 2.95, with a median score of 2 and standard deviation of 2.63. The range of TOP scores reflects the variability in open science practices within CAIM journals. Four journals (21%) had a final TOP score of 0, indicating no adherence to open science practices. The highest TOP factor was an 8, out of a maximum possible of 24. Across the audit, journals most frequently adhered to some requirements of data citation and design and analysis transparency, as evidenced by the fewest scores of zero in these categories. Journals had the lowest scores in the analysis plan preregistration (this is not a common termed used in medicine. Registration is a more established and used term) and replication categories. A common trend across several categories was encouragement rather than requirement or enforcement of open science practices. TOP scores of 0 and 1, corresponding to no mention or encouragement, were more common in all categories than scores of 2 and 3, corresponding to a mandate or condition of publication. Overall, our findings suggest that CAIM journals provide minimal guidelines to encourage or require authors to adhere to open science practices, as reflected by the average TOP Factor of 2.95.
This low usage of open science practices across journals is similar to findings in other disciplines and suggests that journals have an opportunity to improve research practices through regulating the use of open science practices. For example, research shows open science practices are not highly encouraged nor required by many communication sciences and disorders journals, with a mean TOP Factor of 1.4 [17]. A similar finding was seen in an audit of open science practices across health and medical science journals, where transparency and open science practices in these journals were not frequently encouraged or mandated [18]. However, the mean TOP Factor was found to be 7, which is higher than the mean for the CAIM journals included in the present study. Similar audits were conducted on pain journals and sleep and chronobiology journals, which found that there was relatively low journal engagement with open science standards in these fields with a median TOP Factors of 3.5 and 3, respectively [43,44]. Furthermore, an audit done on clinical psychology journals found that many journal recommendations, such as use of reporting guidelines, are not frequently enforced in journals [16]. In the field of CAIM, open science practices have the potential to increase the quality reproducibility of research, as better data transparency can promote replicability and peer review to ensure reliability and credibility of findings. Open science practices in CAIM may also expedite the research process through emphasis on preregistration, for example. With improved article quality, stronger experimental evidence, and greater dissemination of CAIM research, negative perceptions of the field may be addressed. Increased public and provider trust in the field may then allow more funding and resources to be allocated for CAIM research, allowing continual improvement of research quality in the field.
With a growing interest in CAIM from patients and practitioners, evident by the numerous patient trials each year, it is becoming increasingly important to ensure that the research quality within this field of medicine is improved and standardized [45]. As many patients use research to make decisions about their healthcare, it is critical for journals to hold submitted articles to a greater standard of open science practices to avoid patient misinformation and improve provider attitudes [46,47]. For example, a study found that many patients resorted to and trusted information on CAIM therapy from medical journals, in the form of clinical trials, compared to social networks or other online media [46]. Another study also noted patients used medical and lay publications in addition to the Internet for better understanding their CAIM options [48]. Additionally, greater journal implementation of open science practices can allow for an easier assessment of comprehensive and transparent reporting, particularly the methods and results. This likely provides increased credibility of CAIM evaluations, which is important to health care providers and patients. Various parts of the research lifecycle can also be made more accessible, allowing researchers to better build upon existing research data. While this audit focused on general CAIM journals, future research may investigate open science practices across journals on specific CAIM topics (e.g. homeopathy) to understand how research is conducted and presented between and across different CAIMs.
Given the potential benefits of open science practices for CAIM research, it is important to consider barriers to its usage and ways to overcome them. A potential reason for low TOP scores includes a low incentive for journals to implement the usage of open science practices due to the added time and resources to mandate the change [49]. Moreover, support from editors is crucial for promoting open science practices reforms within journals. Despite overall positive attitudes towards open science, a survey conducted by Naaman et al. identified that editors see the time and effort required to implement open science practices as a major barrier for open science promotion at the journal level [49]. For example, if data citation was mandated as a condition of publication, peer reviewers may not want to take additional time to ensure articles are published in accordance with the journal’s open science practices. Additionally, the current academic publishing culture favors publications, most of which report statistically positive results. [50]. Open science is a relatively new set of principles and practices which might explain why some authors have not yet integrated them into their research ecosystem. Publishers can potentially alleviate this barrier for both authors and editors by changing the default settings in popular manuscript submission systems to allow for more uniform data input in line with TOP (e.g., a mandatory field requiring input of a data availability statement) [51]. Such changes have been attempted in a study performed by Giofrè et al. that suggests that journal-specific submission guidelines could promote positive alterations in author practices [15]. This adjustment would require authors to include TOP items in the manuscript submission, requiring author adherence to open science practices and reducing the time and effort required from peer reviewers to check for usage of open science practices. Based on this discussion, a next step is to investigate how well CAIM authors adhere to open science practices encouraged or required by these CAIM journals. This type of study can allow comparison of standards set by journals with the actual implementation of open science practices by authors in the field.
Another barrier to the implementation of open science practices at a journal level stems from the frequent poor reporting of CAIM research [52]. When research is poorly reported, there is a decreased ability to draw conclusive results and make comparison with similar research studies, reducing research efficiency, validity, and replicability [52,53]. The multimodal treatment delivery of some types of CAIM are not well substantiated by research, often limiting its focus to one or two treatment interventions [53]. For example, while acupuncture needling is researched as the primary technique employed by licensed acupuncturists, their use of other interventions (e.g., cupping and massage) that are also employed in their multimodal treatment plans are not well researched [53]. Adapting research methods commonly used in Western biomedicine, such as clinical trials, for use in CAIM research in addition to using other research methods such as qualitative studies can improve its research basis [52,54]. Components of open science practice can support this research through the requirement of data sharing and a methodologically sound study design. Furthermore, there is low support for CAIM research from institutions, leading to its inadequate research infrastructure development and funding for research training in the field [54,55]. By training CAIM researchers in open science practices, which emphasize open data sharing and preregistration, its encouraged use can improve the rigor of CAIM study reporting and methodology [53]. Consequently, the number of inadequately designed studies can be reduced, thereby making research more efficient [53]. With improved research and open science training, CAIM researchers may be more inclined to use open science practices in research development, which also makes journal level open science practice changes more likely to be made. For example, in the field of economics, researcher attitudes are readily accepting of strong transparency standards, which has led many high-impact economics journals to also adopt strong open science requirements [7].
In addition to journals, it is also important to consider the usage of open science practices and barriers authors face with respect to its implementation in their fields, as they are primarily affected by open science mandates in journals. A survey study conducted on author usage of open science practices found that while many authors are familiar with the idea of open science, many lack knowledge on how to implement these practices [56]. In another survey on CAIM author attitudes about open science, funding was found to be a major barrier to applying open science practices in research [57]. Both of these findings can serve as starting points for investigating how journals can better facilitate the use of open science research practices. Other fields of research have also initiated collaborative efforts between different members of the scientific community to improve open science practices. For example, during an expert meeting organized by the European Health Psychology Society to improve the use of open science best practices in health psychology, it was suggested that members of the scientific community themselves, including researchers and editors, identify and work to resolve barriers that prevent adoption of open science practices in their own field [58]. Overall, further research focused on gathering and implementing feedback from CAIM experts as part of the ongoing improvement and implementation process of open science practices may improve the use and efficacy of open science practices in CAIM.
Furthermore, increased funding needs to be directed towards CAIM research to train and incentivize these practitioners to perform research in this field. While many research disciplines face challenges with limited funding, the allocation of funding for CAIM research is notably lower compared to other healthcare sectors [59]. The safety and efficacy of CAIM therapies have been long debated and uncertainty with findings in the field significantly contribute to the lack of financing allocated to CAIM research [22]. Failure to obtain CAIM research funding in the past has often been attributed to skepticism and low perceived priority of CAIM research by grant reviewers, as well as low institutional support of CAIM research [60]. Funding applications for CAIM research have also been seen to fall short in comparison to conventional medicine due to insufficient infrastructure and fewer university level researchers showing interest in the field [22,55]. As noted by Veziari et al., as compared to advocating for linear research funding models, creating multi-faceted funding programs, though previously limited to other disciplines, may be favorable for CAIM as well [61] For example, funding research programs that require interaction between sponsors, CAIM researchers, and users, which include practitioners and patients, allows the opportunity to identify and address misconceptions regarding quality of findings within the field [52,62]. Investing in CAIM research training can also promote research quality that meets a more comparable standard to conventional medical research, and, in turn, makes for more competitive funding applications [55]. The adoption of open science practices, itself, provides an opportunity to present transparent, reproducible, and evidence-based CAIM research, which can also incentivize funding agencies to invest in CAIM research [33].
Several fields of research have seen improvements in the quality of their research articles and other metrics after implementing open science practices. Studies in the field of psychology have focused on registered reports, which is a type of registration that requires a two-stage peer review process [63]. A protocol’s introduction, materials, and methods undergo a first stage of peer review prior to the execution of the methodology, and is peer reviewed a second time once the manuscript is completed, checking for adherence to the original methodology [63]. Soderberg et al. found that psychology research articles published with registered reports have significantly improved research thoroughness and reputability, as well as include a more rigorous methodology and analysis [64]. Additionally, Obels et al. found that compared to other articles, registered reports have higher rates of data and code sharing [65]. However, studies have also found that even when data sharing is required by journals, data reporting is often inadequate for reproducibility which shows that open data alone is unable to achieve the proposed benefits of data sharing [66,67]. To draw conclusions about the impact of open science in CAIM, more research is warranted.
Strengths and limitations
Our study has notable strengths. We sourced our subset of CAIM journals from a bibliometric analysis conducted by Ng, which was sourced from the Scopus Source List, a list of journals with established ASJC categories [37]. Furthermore, screening, data extraction, and TOP rubric assessment of the CAIM journals was conducted independently and in duplicate. All collected information was reviewed by all authors, and any discrepancies was resolved via consensus. Furthermore, comparing definitions against the TOP rubric provided a standardized reference for comparing data across different journals. The detailed and specific methodology also allows for the replicability of the study’s findings.
Regarding limitations, this study only includes CAIM journals that published online, which may not reflect the open science practices of CAIM journals that only publish in print format. Additionally, only journals that publish in the English language were included, thus our findings may not be representative of CAIM journals published in other languages. Furthermore, the audit relies on the information made publicly available by the included journals. If relevant data on open science practices was not readily accessible or inconsistently reported, the study’s findings may be incomplete or inaccurate. We also acknowledge that the audit may not capture the reasons behind the observed open science practices. Without additional qualitative data or information from journal editors, authors, or reviewers, it can be challenging to fully understand the motivations or barriers influencing open science practices in CAIM journals. Further, this audit represents a snapshot of open science practices at a specific point in time, and it can be expected that this information may change over time. Lastly, the TOP guidelines also do not address open science practices in regard to open access publishing models, which is a notable aspect of open science.
Conclusions
In conclusion, it was found that CAIM journals provide minimal guidelines to encourage or require authors to adhere to open science practices. This audit serves as a starting point to understand and improve the usage of open science practices in CAIM journals. The inclusion of greater open science practices in CAIM journals may serve to enhance the usability and replicability of research published in these journals. Knowledge of how open science practices are encouraged within CAIM journals can inspire initiatives that aim to make research within this field more accessible to researchers and healthcare providers.
Supporting information
S1 Table. Specific text associated with each of the eight TOP guidelines for the 19 CAIM journals.
https://doi.org/10.1371/journal.pone.0302655.s001
(XLSX)
Acknowledgments
We gratefully acknowledge Navila Asgar for her contributions and review of our study protocol.
References
- 1. Azevedo F, Parsons S, Micheli L, Strand J, Rinke EM, Guay S, et al. Introducing a Framework for Open and Reproducible Research Training (FORRT) 2019.
- 2.
National Research Council of Canada. Open science action plan: response to the Government of Canada Roadmap for Open Science 2021.
- 3.
Open Science—OECD n.d. https://www.oecd.org/sti/inno/open-science.htm (accessed December 2, 2022).
- 4.
https://plus.google.com/+UNESCO. UNESCO Recommendation on Open Science. UNESCO 2020. https://en.unesco.org/science-sustainable-future/open-science/recommendation (accessed December 2, 2022).
- 5. Crüwell S, van Doorn J, Etz A, Makel MC, Moshontz H, Niebaum JC, et al. Seven Easy Steps to Open Science. Zeitschrift Für Psychologie 2019;227:237–48.
- 6. McKiernan EC, Bourne PE, Brown CT, Buck S, Kenall A, Lin J, et al. How open science helps researchers succeed. eLife n.d.;5:e16800. pmid:27387362
- 7. Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD, Breckler SJ, et al. Promoting an open research culture. Science 2015;348:1422–5. pmid:26113702
- 8. l Me lor D. Transparency and Openness Promotion (TOP) Guidelines. Guidelines for Transparency and Openness Promotion (TOP) in Journal Policies and Practices “The TOP Guidelines” 2014. https://osf.io/9f6gx/ (accessed October 5, 2023).
- 9.
New Measure Rates Quality of Research Journals’ Policies to Promote Transparency and Reproducibility n.d. https://www.cos.io/about/news/new-measure-rates-quality-research-journals-policies-promote-transparency-and-reproducibility (accessed March 20, 2023).
- 10. Vicente-Saez R, Martinez-Fuentes C. Open Science now: A systematic literature review for an integrated definition. Journal of Business Research 2018;88:428–36.
- 11. Clark MP, Luce CH, AghaKouchak A, Berghuijs W, David CH, Duan Q, et al. Open Science: Open Data, Open Models, …and Open Publications? Water Resources Research 2021;57:e2020WR029480.
- 12. Wilkinson MD, Dumontier M, Aalbersberg IjJ, Appleton G, Axton M, Baak A, et al. The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data 2016;3:160018. pmid:26978244
- 13. Allen C, Mehler DMA. Open science challenges, benefits and tips in early career and beyond. PLOS Biology 2019;17:e3000246. pmid:31042704
- 14. Mills-Finnerty C. Barriers to reproducibility: misalignment of career incentives and open science best practices. Journal for Reproducibility in Neuroscience 2020;1:304.
- 15. Giofrè D, Cumming G, Fresc L, Boedker I, Tressoldi P. The influence of journal submission guidelines on authors’ reporting of statistics and use of open research practices. PLOS ONE 2017;12:e0175583. pmid:28414751
- 16. Nutu D, Gentili C, Naudet F, Cristea IA. Open science practices in clinical psychology journals: An audit study. Journal of Abnormal Psychology 2019;128:510–6. pmid:31368730
- 17. Schroeder SR, Gaeta L, El AM, Chow JC, Borders JC. Evaluating Research Transparency and Openness in Communication Sciences and Disorders Journals. Journal of Speech, Language, and Hearing Research n.d. pmid:36516469
- 18. Gardener AD, Hick EJ, Jacklin C, Tan G, Cashin AG, Lee H, et al. Open science and conflict of interest policies of medical and health sciences journals before and during the COVID-19 pandemic: A repeat cross-sectional study: Open science policies of medical journals. JRSM Open 2022;13:20542704221132139. pmid:36407750
- 19.
Complementary, Alternative, or Integrative Health: What’s In a Name? NCCIH n.d. https://www.nccih.nih.gov/health/complementary-alternative-or-integrative-health-whats-in-a-name (accessed October 19, 2022).
- 20. Ng JY, Boon HS, Thompson AK, Whitehead CR. Making Sense of “Alternative”, “Complementary”, “Unconventional” and “Integrative” Medicine: Exploring the Terms and Meanings Through a Textual Analysis. BMC Complementary and Alternative Medicine 2016;16:134. pmid:27206976
- 21. Ng JY, Dhawan T, Dogadova E, Taghi-Zada Z, Vacca A, Wieland LS, et al. Operational Definition of Complementary, Alternative, and Integrative Medicine Derived from a Systematic Search. BMC Complementary Medicine and Therapies 2022;22:104. pmid:35413882
- 22. Veziari Y, Leach MJ, Kumar S. Barriers to the conduct and application of research in complementary and alternative medicine: a systematic review. BMC Complementary and Alternative Medicine 2017;17:166. pmid:28335766
- 23. Gale N. The Sociology of Traditional, Complementary and Alternative Medicine. Sociology Compass 2014;8:805–22. pmid:25177359
- 24. Steel A, McEwen B. The need for higher degrees by research for complementary medicine practitioners. Australian Journal of Herbal Medicine 2014;26.
- 25.
Effects of Mindfulness on Psychological Health: A Review of Empirical Studies—PMC n.d. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3679190/ (accessed October 27, 2023).
- 26. Schlechta Portella CF, Ghelman R, Abdala V, Schveitzer MC, Afonso RF. Meditation: Evidence Map of Systematic Reviews. Frontiers in Public Health 2021;9. pmid:34926371
- 27.
Hoffer LJ. Complementary or alternative medicine: the need for plausibility n.d.
- 28. Jackson S, Scambler G. Perceptions of evidence-based medicine: traditional acupuncturists in the UK and resistance to biomedical modes of evaluation. Sociology of Health & Illness 2007;29:412–29. pmid:17470219
- 29. Leach MJ, Gillham D. Are complementary medicine practitioners implementing evidence based practice? Complementary Therapies in Medicine 2011;19:128–36. pmid:21641517
- 30. Barnes J, Abbot NC, Harkness EF, Ernst E. Articles on Complementary Medicine in the Mainstream Medical Literature: An Investigation of MEDLINE, 1966 Through 1996. Archives of Internal Medicine 1999;159:1721–5. pmid:10448774
- 31. Song E, Ang L, Lee MS. Increasing trends and impact of integrative medicine research: From 2012 to 2021. Integrative Medicine Research 2022;11:100884. pmid:36052205
- 32. Lewith G, Verhoef M, Koithan M, Zick SM. Developing CAM Research Capacity for Complementary Medicine. Evidence-Based Complementary and Alternative Medicine 2006;3:283–9. pmid:16786061
- 33. Foster ED, Deardorff A. Open Science Framework (OSF). Journal of the Medical Library Association: JMLA 2017;105:203–6.
- 34.
Science C for O. TOP Guidelines n.d. https://www.cos.io/initiatives/top-guidelines (accessed April 15, 2023).
- 35.
OSF. TOP factor rubric 2020. https://osf.io/t2yu5 (accessed April 15, 2023).
- 36. Ng J, Lin B, Parikh T, Cramer H, Moher D. PROTOCOL: Investigating the Nature of OpenScience Practices Across Complementary, Alternative, and Integrative Medicine (CAIM) Journals: An Audit 2023.
- 37. Ng JY. Insight into the characteristics of research published in traditional, complementary, alternative, and integrative medicine journals: a bibliometric analysis. BMC Complementary Medicine and Therapies 2021;21:185. pmid:34210316
- 38. Gusenbauer M, Haddaway NR. Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research Synthesis Methods 2020;11:181–217. pmid:31614060
- 39. Mayo-Wilson E, Grant S, Supplee L, Kianersi S, Amin A, DeHaven A, et al. Evaluating implementation of the Transparency and Openness Promotion (TOP) guidelines: the TRUST process for rating journal policies, procedures, and practices. Research Integrity and Peer Review 2021;6:9. pmid:34078479
- 40.
TOP Factor Submission. Google Docs n.d. https://docs.google.com/forms/d/e/1FAIpQLScBd3juqw-V8uGFMJYCAY0CVcoO5sNRq634qGng51oGNvrr4A/viewform?usp=embed_facebook (accessed April 15, 2023).
- 41.
Hindawi. Hindawi Evidence-Based Complementary and Alternative Medicine Author Guidelines. Hindawi n.d. https://www.hindawi.com/journals/ecam/guidelines/ (accessed August 26, 2023).
- 42. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proceedings of the National Academy of Sciences 2018;115:2600–6. pmid:29531091
- 43. Cashin AG, Bagg MK, Richards GC, Toomey E, McAuley JH, Lee H. Limited engagement with transparent and open science standards in the policies of pain journals: a cross-sectional evaluation. BMJ Evidence-Based Medicine 2021;26:313–9. pmid:31980469
- 44. Spitschan M, Schmidt MH, Blume C. Principles of open, transparent and reproducible science in author guidelines of sleep research and chronobiology journals. Wellcome Open Research 2021;5:172. pmid:33748432
- 45. Jonas WB, Eisenberg D, Hufford D, Crawford C. The Evolution of Complementary and Alternative Medicine (CAM) in the USA over the Last 20 Years. Complementary Medicine Research 2013;20:65–72. pmid:23727764
- 46.
Bridging the gap: decision-making processes of women with breast cancer using complementary and alternative medicine (CAM) | Supportive Care in Cancer n.d. https://link.springer.com/article/10.1007/s00520-007-0282-3#Sec3%20+%20https://journals-lww-com.libaccess.lib.mcmaster.ca/familyandcommunityhealth/fulltext/2003/01000/surviving_breast_cancer__the_role_of_complementary.3.aspx (accessed December 6, 2023).
- 47. Balneaves LG, Weeks L, Seely D. Patient Decision-Making about Complementary and Alternative Medicine in Cancer Management: Context and Process. Current Oncology 2008;15:94–100. pmid:18769576
- 48. Canales MK, Geller BM. Surviving Breast Cancer: The Role of Complementary Therapies. Family & Community Health 2003;26:11. pmid:12802124
- 49. Naaman K, Grant S, Kianersi S, Supplee L, Henschel B, Mayo-Wilson E. Exploring enablers and barriers to implementing the Transparency and Openness Promotion Guidelines: a theory-based survey of journal editors. Royal Society Open Science 2023. pmid:36756061
- 50. Hunter P. The reproducibility “crisis.” EMBO Reports 2017;18:1493–6. pmid:28794201
- 51. Aalbersberg IjJ, Appleyard T, Brookhart S, Carpenter T, Clarke M, Curry S, et al. Making Science Transparent By Default; Introducing the TOP Statement 2018.
- 52. Fischer FH, Lewith G, Witt CM, Linde K, von Ammon K, Cardini F, et al. High prevalence but limited evidence in complementary and alternative medicine: guidelines for future research. BMC Complementary and Alternative Medicine 2014;14:46. pmid:24499316
- 53. Nahin RL, Straus SE. Research into complementary and alternative medicine: problems and potential. BMJ 2001;322:161–4. pmid:11159578
- 54. Jonas WB. Building an evidence house: challenges and solutions to research in complementary and alternative medicine. Forschende Komplementarmedizin Und Klassische Naturheilkunde = Research in Complementary and Natural Classical Medicine 2005;12:159–67. pmid:15985781
- 55. Lewith G, Holgate S. CAM research and development. Complementary Therapies in Nursing and Midwifery 2000;6:19–24. pmid:11033649
- 56. Ng JY, Lin B, Kreuder L, Cramer H, Moher D. Open Science Practices Among Authors Published in Complementary, Alternative, and Integrative Medicine Journals: An International, Cross-Sectional Survey 2024:2024.02.26.24303402.
- 57. Ng JY, Santoro LJ, Cobey KD, Steel A, Cramer H, Moher D. Complementary, Alternative, and Integrative Medicine Researchers’ Practices and Perceived Barriers Related to Open Science: An International, Cross-Sectional Survey 2023:2023.10.24.23297458.
- 58. Kwasnicka D, ten Hoor GA, van Dongen A, Gruszczyńska E, Hagger MS, Hamilton K, et al. Promoting scientific integrity through open science in health psychology: results of the Synergy Expert Meeting of the European health psychology society. Health Psychology Review 2021;15:333–49. pmid:33198583
- 59. Veziari Y, Kumar S, Leach M. Barriers to the conduct and application of research among complementary and alternative medicine professions in Australia and New Zealand: A cross-sectional survey. Complementary Therapies in Medicine 2021;60:102752. pmid:34126172
- 60. Ernst E. Funding research into complementary medicine: the situation in Britain. Complementary Therapies in Medicine 1999;7:250–3. pmid:10709311
- 61. Veziari Y, Kumar S, Leach M. Addressing barriers to the conduct and application of research in complementary and alternative medicine: a scoping review. BMC Complementary Medicine and Therapies 2021;21:201. pmid:34266441
- 62. Arnott JC, Neuenfeldt RJ, Lemos MC. Co-producing science for sustainability: Can funding change knowledge use? Global Environmental Change 2020;60:101979.
- 63.
The AJE Team. Pre-registration vs. Registered Reports: What’s the difference? AJE 2022. https://www.aje.com/arc/pre-registration-vs-registered-reports/ (accessed September 13, 2023).
- 64. Soderberg CK, Errington TM, Schiavone SR, Bottesini J, Thorn FS, Vazire S, et al. Initial evidence of research quality of registered reports compared with the standard publishing model. Nature Human Behaviour 2021;5:990–7. pmid:34168323
- 65. Obels P, Lakens D, Coles NA, Gottfried J, Green SA. Analysis of Open Data and Computational Reproducibility in Registered Reports in Psychology. Advances in Methods and Practices in Psychological Science 2020;3:229–37.
- 66. Hardwicke TE, Mathur MB, MacDonald K, Nilsonne G, Banks GC, Kidwell MC, et al. Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition. Royal Society Open Science 2018;5:180448. pmid:30225032
- 67. Stockemer D, Koehler S, Lentz T. Data Access, Transparency, and Replication: New Insights from the Political Behavior Literature. PS: Political Science & Politics 2018;51:799–803.