https://orcid.org/0000-0002-4089-725X
Figures
Abstract
The translation gap between knowledge production and implementation into clinical practice and policy is an ongoing challenge facing researchers, funders, clinicians and policy makers globally. Research generated close to practice and in collaboration with end users is an approach that is recognised as an effective strategy to facilitate an improvement in the relevance and use of health research as well as building research capacity amongst end users. The Research Translation Projects (RTP) program funded by the Western Australian (WA) Department of Health facilitates clinical and academic collaboration through competitive funding of short-term research projects. Its aim is to improve healthcare practice while also finding efficiencies that can be delivered to the WA health system. A mixed methods approach was adopted to evaluate the research impact of the RTP program, at completion of the two-year funding period, across a range of impact domains through the adaptation and application of the Canadian Academy of Health Sciences’ (CAHS) framework for research impact. In addition, further analysis was undertaken to address specific objectives of the RTP program more closely, in particular research capacity building and collaboration and health system Inefficiencies targeted by the program. Social network analysis was applied to assess the extent and growth of collaboration across WA health organisations over time. Results indicated that the ‘bottom up’ approach to research translation has triggered modest, yet positive outcomes across impact domains including advancing knowledge, collaboration and capacity building as well as contributing to changes in policy and practice. Additionally, the projects identified opportunities by which inefficiencies in the health system can be addressed. Further work is required to better understand the pathways by which short-term outcomes can be translated into more long-term impacts and the mechanisms that trigger this process.
Citation: Mosedale A, Geelhoed E, Zurynski Y, Robinson S, Chai K, Hendrie D (2022) An impact review of a Western Australian research translation program. PLoS ONE 17(3): e0265394. https://doi.org/10.1371/journal.pone.0265394
Editor: Dylan A. Mordaunt, Illawarra Shoalhaven Local Health District, AUSTRALIA
Received: April 7, 2021; Accepted: March 2, 2022; Published: March 31, 2022
Copyright: © 2022 Mosedale et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data cannot be shared publicly because data belongs to the Western Australia Department of Health and was provided to us for the specific purpose of conducting the retrospective review.Data are available from the WA Department of Research Governance Committee (contact via https://ww2.health.wa.gov.au/Articles/N_R/Research-Governance-Office) for researchers who meet the criteria for access to confidential data.
Funding: This work is supported by the National Health and Medical Research Council Partnership Center for Health System Sustainability grant (ID: 9100002).
Competing interests: The authors have declared that no competing interests exist.
Background
The production of quality evidence to inform policy and practice in healthcare requires significant investment. However, implementation and translation of healthcare research into policy and practice is an ongoing challenge facing researchers, funders, clinicians and policy makers globally [1]. This translation gap between knowledge production and implementation into practice is widely recognised in the health research community. As a consequence of this gap, the health system does not meet its potential to provide patients with advances in technology and cost-effective healthcare programs and interventions. As a result, the return on the original investment in research could be brought into question [1, 2].
Research translation is a complex process, with many factors and conceptualisations to be considered. The ‘bench-to-bedside’ process of research translation whereby applying knowledge from basic sciences to produce new medicines, devices and treatment options for patients focuses on the interface between basic science and clinical medicine [3]. From a public health and health services research perspective, the emphasis is on the healthcare delivery systems and improving health through the translation of research into policy as well as practice [4]. In addition to the conceptualisation of research translation, terminology varies throughout the literature to include knowledge translation, evidence-based practice and knowledge to action [4]. Despite the semantics, all acknowledge a gap between the knowledge from research and its application to, policy and practice, and treatment options, and the importance of closing this gap. In addition, regardless of the perspective taken, there is agreement that this process is dynamic, involving interaction between researchers and end users, as well as the context of implementation [5]. One such complex interaction identified in the literature is the ‘top down’ process of research translation whereby researchers introduce pre-conceived practices or interventions to a system of care and ask clinicians to assist in its implementation, requiring adaptation to the local context and leading to poorly sustained interventions [6, 7]. The top-down approach comes with an expectation that management and resource structures are in place to enable clinicians to implement the change. A ‘bottom up’ approach may be the response needed to address the research translation gap, whereby research is generated close to practice in collaboration with end users. Collaborative research from a healthcare perspective, which is the organised and deliberate interactions and processes between researchers and end users (clinicians, decision makers and patients) is recognised as an effective strategy for increased relevance and use of health research as well as building research capacity amongst end users [8–11]. To facilitate this approach to research translation, research funders are enabling translational research projects and programs through unique funding opportunities. On a large scale, such funding programs have been seen in Canada, the United Kingdom and Australia [12].
A smaller scale example is the Research Translation Project (RTP) program, funded by the Western Australia Department of Health (WA Health). The RTP program is a competitive program, funding short-term research projects. Its aim is to improve healthcare practice and/or policy by investigating potential efficiencies that can be delivered to WA Health while maintaining and/or improving patient outcomes [13]. The RTP program encourages applications in the areas of clinical research, health services research, and public health research [13]. Projects must address relevant contemporary challenges faced by the publicly funded health system in WA by undertaking new research, proof-of-concept and/or pilot studies or local application and evaluation of research that has been applied elsewhere. Collaboration between disciplines and institutions is a key priority for successful funding. In addition, proposals that are likely to become the groundwork for further implementation, commercialisation or research grant applications to national funding bodies such as the National Health and Medical Research Council (NHMRC) are also strongly encouraged.
While funding organisations are increasingly focused on creating opportunities for research translation in the form of funding programs, less attention has been paid to the evaluation of such programs [14]. McLean, Graham [14] highlight the lack of evaluation of knowledge translation programs at the funding level and call for evaluation that identifies what works, for what type of funder and why.
While internal evaluation and monitoring of the RTP program has been undertaken, a more formal evaluation has the capacity to better inform ongoing investment. The measurement and assessment of returns on research investment is high on the policy agenda in Australia and across the globe [15]. Additional to accountability upwards to funders and downwards to users, the evaluation of research impact can also contribute to the development and strengthening of research dissemination and translation, inform ongoing funding allocation to bring about desired impact as well as manage and monitor the performance of research programs [16, 17]. The aim of this study is to evaluate the research impact of the RTP program across a range of impact domains consistent with the programs focus. Specific objectives of the study are to: (i) assess the research impact of research translation projects at the completion of funding against an adapted version of an existing evaluation framework, (ii) undertake social network analysis of collaboration across the WA health system as a result of the RTP program and (iii) identify sources of inefficiency addressed by projects and the proposed decision-making informed by the project outcomes.
Methods
A mixed methods approach was adopted to evaluate the RTP program based on document analysis. Research impact of the program was assessed along the spectrum of research translation through the adaptation of an existing framework [18]. In addition, further analysis was undertaken to address the objectives of the RTP program more closely, specifically with regard to research capacity building and collaboration as desired outcomes of the RTP program as well as areas of inefficiency targeted by projects funded under the RTP program.
Selection of the evaluation framework
Driven by a policy agenda that demands accountability, reduced waste and a focus on sustainable health systems, research impact has become a topical issue and, as such, there is growing interest in measurement internationally. We drew on the literature to identify a suitable conceptual framework to support the RTP program evaluation. A number of authors have attempted to summarise the use of research evaluation frameworks and tools presented in the literature through comparative studies and systematic reviews, including the “real world” impact of research, rather than just that which is applicable to academia [1, 16, 19, 20]. Cruz Rivera, Kyte [16] reported on twenty-four unique methodological frameworks for assessing health research impact. In recent reviews, both Cruz Rivera, Kyte [16] and Raftery, Hanney [19] have identified the Payback Framework as the most widely applied model for the evaluation of funded health research programs, a finding that is consistent with a review, conducted by Hanney, Buxton [21]. The Payback Framework developed by the Health Economics Research Group at Brunel University was originally designed to assess the return on investment from health service research [22]. The concept of “payback” as the foundation of the work of Buxton and Hanney [22] focuses on the benefit of research across a number of domains, such as knowledge, future research, research use and political and societal benefits. In recent years, the Payback Framework has been applied and adapted in the development of other methodological frameworks [16, 19].
The framework chosen here for the assessment of research impact of the RTP program is the Canadian Academy of Health Sciences’ (CAHS) framework [18], an adaption of the Payback Framework. The CAHS framework was selected because it provides a menu of defined and published preferred indicators and metrics for each category and subcategory of the framework; a unique advantage not provided by other frameworks. We chose indicators that are generic and could be applied across the four pillars of research outlined in the CAHS, namely biomedical research, applied clinical research, health services and policy research and population and public health research [18]. The adaptability of the CAHS framework across these four pillars of research is important given the heterogeneity of the research projects funded through the RTP program. Additionally, this evaluation of the RTP program is over a relatively short timeframe, with projects being assessed up to the end of a two-year funding period. The CAHS framework considers research impact along a pathway of translation, which allows outcomes that are feasible to assess in the short term to be evaluated. Finally, the CAHS framework was selected based on its previous use within the RTP program, for internal evaluations as well as in program documentation.
The CAHS framework is designed to capture health research impact across five main categories: 1) advancing knowledge, 2) building capacity 3) informing decision‐making, 4) health impacts, and 5) broad socio‐economic impacts. Given the short timeframe for the evaluation, research projects were only assessed across the first three categories. Research impact in categories 4 and 5 are unlikely to be achieved at the completion of a two-year project. As instructed by the framework developers, the set of indicators and metrics chosen for this evaluation were based on their ‘feasibility’ and ‘attractiveness’, primarily on account of data availability, timeliness, and attribution.
A key goal of the RTP program is to facilitate clinical and academic collaboration. Literature suggests that a benefit of clinical and academic collaboration is the development of research capacity amongst the clinical workforce, which has been argued to be fundamental to closing the evidence–practice gap [23]. In addition to using the CAHS framework for measuring research impact, the principles of capacity building proposed by Cooke and colleagues [23–25] were applied to assess the contribution of the RTP program to research capacity building across the WA health system. These principles are 1) skill development 2) developing research ‘close to practice’ 3) building linkages and partnerships 4) appropriate dissemination 5) continuity and sustainability and 6) establishing infrastructure.
In addition to facilitating clinical and academic collaboration the RTP program also seeks to improve efficiencies in the WA health system. Due to the analysis being undertaken in the short term, a framework to capture the types of system inefficiencies at the centre of each project was developed based on the work of Ravaghi, Afshari [26] on hospital inefficiencies and the World Health Organization on technical inefficiencies relating to health system inputs [27]. Four broad types of inefficiencies were identified: 1) inefficient use of procedures, investigation and equipment, 2) irrational use and supply of drugs and blood products, 3) inappropriate or costly workforce mix and 4) inappropriate hospital admissions and length of stay and suboptimal quality of care.
Data sources and extraction
The primary source of data was archival documents of the RTP program including application guidelines, grant applications and completion reports from researchers. Data extraction was undertaken using a systematic examination of completion reports and corresponding grant applications for RTP projects funded between 2011 and 2016. Each grant recipient is required to submit a completion report to provide an account of objectives achieved, whether or not the program/intervention was found to provide efficiencies to the health system and whether the funding had led to other benefits (such as changes in culture, capacity, new collaborations etc.). Completion reports also provide information regarding research outputs such as publications and presentations resulting from the funded research. Given the two-year funding period of the RTP program, the selection of projects funded between 2011 and 2016 provided sufficient time for project completion and submission of final completion reports, with an allowance of time for slippage. Projects were included in the review only where the final reports and corresponding applications were available. For the assessment of collaboration, applications for successfully funded projects were included beyond 2016 up until 2019. Given that the applications and reports are completed by the research teams, the data they contain is self-reported and the data quality is limited to the level to which the applications and reports are completed.
Data analysis
Several approaches were applied to analyse data extracted from the RTP documents, depending on the indicators and metrics used. These included content and thematic analysis and social network analysis. In the first instance, a content analysis against indicators and metrics selected from the CAHS research impact framework was undertaken for descriptive quantitative analysis (i.e. counts). Table 1 identifies the specific indicators and metrics extracted from each document as part of the content analysis.
Guided by Sarre and Cooke’s principles of research capacity building [25], further analysis of the collaboration between clinicians and academics was undertaken using social network analysis (SNA) and thematic analysis [25].
The SNA was conducted at the organisation level, identifying the organisations represented within each research team between 2011 and 2019. Organisations represented on each successfully funded project team were coded into categories (Table 2). Nodes in the network graph are used to represent the organisations while the connections (edges) are weighted (i.e. drawn thicker) based on the number of project collaborations. The data analysis and processing was developed using Python and NetworkX (https://networkx.org/), a module for network analysis, in addition to developing a Web application using a JavaScript library named vis.js (https://visjs.org/) for building interactive network visualisations.
Deductive thematic analysis was also undertaken to illustrate the contribution of the RTP program to capacity building. Direct quotes reflecting the principles of research capacity building developed by Cooke [24] were identified in RTP final reports and coded based on these principles. Finally, sources of inefficiency and proposed solutions were identified from information presented in the RTP final reports and coded based on the adapted framework developed from the work of Ravaghi, Afshari [26] and the World Health Organization [27].
Curtin University Human Research Ethics Committee (HREC) has approved this study. Approval number: HRE2020-0464.
Results
Research impact
Of the 59 projects funded between 2011 and 2016, the required documentation was available for 33 projects for which the research focus spanned across a spectrum of acute hospital-based research to community care, general practice as well as diagnostics and clinical education From a broader point of view the RTPs by nature of the grant guidelines are focused on quality improvement issues. In addition the cost savings perspective means that the research projects are largely quantitative, pre-post or before after studies. Project funding was for a term of two years with a maximum amount of $270 000. Fig 1 provides a summary of the short-term research impacts in terms of the traditional impact measures. The longer-term impacts, beyond 5 years, such as the effects on healthcare and risk factors and contributions to changing health, social and economic wellbeing were beyond the scope of this study. In relation to Advancing knowledge, 60 peer- reviewed publications were reported as having been published as a result of RTP funding, however, upon citation analysis only 16 publications acknowledged this funding. Of these 16, 81% (n = 13) were published in high impact (Q1) journals [28]. Twenty-one projects reported publications in draft at the time of completion. There were 122 conference presentations resulting from the 33 projects, and 10 projects reported media coverage. Other publications noted in the final reports included educational resources and unpublished theses.
(Adapted from https://www.cahs-acss.ca/wp-content/uploads/2011/09/ROI_FullReport.pdf).
Within the category of capacity building, content analysis revealed that across the traditional measure of capacity building, of the 33 projects, six PhD candidates used the projects for research contributing to their doctoral award. Nineteen projects gained additional research funding leveraged off the initial RTP funding, four of which were NHMRC grants.
Finally, in relation to influencing decision making, 14 projects reported a contribution to the implementation of new local practice guidelines or policy within the setting where the project was undertaken. Eight projects reported making contributions either to changes to policy and guidelines beyond the local setting.
Research capacity building and collaboration
Both thematic and social network analysis was undertaken to provide a deeper understanding of the extent of collaboration and research capacity development as a result of the RTP program.
Thematic analysis.
The review of narratives on research capacity developed as a result of the RTP program reflected capacity building in line with the six principles proposed by Cook (2005) (Table 3). Additionally, improved collaboration was a noticeable outcome of the RTP program as reported by investigators in terms of both clinical collaboration and research collaboration.
Social network analysis.
The extent of the collaboration network generated by the RTP program across the WA health system was depicted using SNA. Fig 2 represents the cumulative growth of the RTP collaboration network at time intervals 2011, 2015 and 2019. Each node on the diagram represents an organisation that has participated in the RTP program as part of a project team and each line represents a collaborative relationship. The size of the node is indicative of the number of projects related to each organisation.
Over this time period, 95 unique organisations were represented on funded projects. There was cumulative growth in the RTP collaborative network over time with an increasing number of organisations and collaborative relationships added in 2015 and 2019 compared with 2011, (Fig 2). To illustrate the collaboration between academics and service providers, academic organisations are represented as purple nodes and clinical service organisations as orange nodes. For details of organisations included in each group, refer to Table 2. It is important to note that two nodes, identified in green represent consumers or consumer representative organisation. In addition to an overall growth in the collaborative network, Fig 3 distinguishes organisations with a special interest in primary care (red nodes), highlighting their increased involvement (although small) of these organisations in the RTP network over time.
Sources of inefficiency.
An additional feature of the RTP program is its focus on efficiencies and generating cost saving. Report narratives provided by investigators suggested that, as a result of participating in the RTP program, clinicians and researchers developed a sense of resource awareness and an appreciation for undertaking economic evaluation as part of intervention research.
"Being involved in this study has provided internal staff with experience in clinical research and the value of analysis of cost effectiveness and considering new models of care to maximise the health dollar"
“The project has given researchers at the… a better understanding of health economics…”
Of the 33 projects assessed, 67% (n = 22) attempted to address service delivery inefficiencies. The majority of these primarily aimed to reduce inpatient hospital costs related with inappropriate hospital admissions and length of stay and suboptimal quality of care (Table 4). Strategies employed to address these inefficiencies included improved coordination of care, shifting care from inpatient to community or home-based care and implementation of care pathways for inpatient admissions. Six projects (n = 6) attempted to address inefficiencies in relation to tests and procedures, proposing new test or procedure technology that could deliver the same or better outcomes at a lower cost. Three projects focused on inefficiencies relating to the irrational use and supply of drugs or blood products. Examples included the implementation of new guidelines to improve dispensary efficiency and effectiveness, and the trial of an alternative, more cost-effective therapy. Finally, workforce substitution from medical staff to allied health staff was the focus of two projects (n = 2) to address inefficiencies related to an inappropriate or costly workforce mix.
Discussion
This study reports on the assessment of the RTP program against established research impact indicators including advancing knowledge, building capacity and informing decision making in line with the short-term domains of the CAHS framework, leaving the long term domains for future investigation. While common indicators of advancing knowledge and capacity building have been widely used and reported in the literature (e.g. publication count, additional funding etc.), we have applied more novel methods such as SNA to evaluate capacity building and collaboration in greater detail in addition to the traditional approaches.
Based on the findings from this study, the RTP program has made some notable contributions to a range of established research impact categories. Despite over-reporting by investigators, evidenced by the 60 peer review publications reported but only 16 explicitly attributed to the RTP grant funding, the program has demonstrated research output in the form of peer reviewed publications, conference presentations, media appearances and other publications such as educational resources, unpublished thesis and non-academic reports. It is recognised that counting publications is a simple way to measure research output and is a limited indicator of impact without an accepted benchmark for comparison [18]. Given the short time frame over which projects were assessed it was unrealistic to undertake a citation analysis at the completion of funding. In addition to the number of publications, the quality of publication output was of a high standard, as evidenced by the majority of publications appearing in high impact journals [28]. While the RTP program has demonstrated achievement across the 3 short term domains of the CHAS framework the research translation projects were not without issues. Several projects identified problems with recruitment, ability to access data, changes to the practice setting, changes to funding of services and movement of staff away from the study area, all of which impacted on the completion and quality of research.
A key aim of the RTP program is to integrate health professionals more effectively into clinical research activity that addresses clinical needs by providing practical solutions on the frontline of care. There was evidence within the project completion reports that the RTP program led to changes to policy or implementation of new practice guidelines within the local setting of the intervention and in some cases beyond the that setting, which is an illustration of its ability to contribute to evidence-based policy and practice. However, the extent of uptake of such practice guidelines and policy and the sustainability of such changes are not well understood.
Beyond the traditional measures of research output, the RTP program, with relatively modest funds, appears to have made some contribution to developing research capacity and collaboration across the WA health system. The social network analysis illustrates the breadth of organisations who are collaborating across the WA health system via the RTP program. While there is currently limited involvement of consumer groups and primary care, this appears to be growing over time which is an important finding given the commitment of the WA department of health to collaborate and engage more meaningfully with consumers and primary care through research, evaluation and health service improvement as outlined in the sustainable health review [29]. Increased collaboration across acute and academic organisations with primary care is also beneficial as we attempt to move away from hospital based care to preventative care.
This study has highlighted research capacity building as an added value of undertaking collaborative research. While collaboration is inevitable because of the nature of the projects, research capacity building can be seen as a by-product of the opportunity provided throughout the RTP program to undertake collaborative research. This was evidenced by specific reference to research capacity building principles in the report narratives. In addition, indicators for capacity building from the CAHS framework were also reported on, with the program contributing to a number of higher degrees and supporting the procurement of additional funding. Research capacity building is considered fundamental to closing the evidence–practice gap [23] and advancing both the individual and organisation’s ability to conduct, translate and sustain higher quality research and practice [30–32]. The development of skills and research knowledge as the first principle of research capacity building was a powerful message that came across throughout the majority of the reports assessed. The importance of programs such as the RTP program is therefore accentuated by the growing concerns for the research capacity of the clinical workforce and the competitiveness of clinician-researchers when it comes to securing funding for quality research [33–35].
In addition to increasing research skills and knowledge, collaboration and linkage was a notable benefit reported by investigators as a result of undertaking an RTP. The SNA effectively demonstrated the growth in the RTP collaborative network over time and the balance of collaboration between health service organisations and academic institutions. The notion of building capacity in research and facilitating collaboration is dependent on funding and support opportunities from funding like the RTP program. Given the benefits of clinician engagement in research outlined in this paper, initiatives to develop and facilitate such opportunities should be supported.
The evaluation focus was over a short time period, which meant only the research outputs and impacts that were present at the end of the funding period could be captured and long term impacts which are subject to significant time lags such as health impacts, and broad economic and social impacts where not able to be assessed [16]. While the broad economic impacts were unable to be assessed, we were able to identify the specific sources of inefficiencies that each of the RTPs where addressing to reduce waste in the system. The largest source of inefficiency to have been addressed was in service delivery where inappropriate hospital admissions, length of stay and suboptimal quality of care drive up healthcare costs. Other opportunities that were adopted to promote greater efficiency and a more sustainable heath system were the inefficient use of procedures, investigations and equipment, the irrational use and supply of medicines and blood products as well as inappropriate workforce mix. While the detail of economic analysis presented in the final reports made it difficult to estimate that tangible cost savings had been made, eligibility for RTP funding requires clinicians and researchers to actively seek more efficient ways of delivering healthcare across a range of inefficient practices without compromising health outcomes.
While this study has provided important insights into the contribution of the RTP program to indicators of research impact, capacity building and identifying opportunities for efficiency gains in the WA health system, insight into how research translation projects such as those funded by the RTP program achieve these outcomes has not been explored. Future research on the RTP program is planned, delving into the mechanisms that lead to positive outcomes such as research translation and its longer-term contribution to overall health system sustainability.
Limitations
The completeness of the final reports submitted by investigators varied greatly and, in many cases, completion reports were poorly done. With only 33 out of 59 projects able to be assessed, this in itself reflects waste to the system as there might have been important findings to improve efficiency that were missed. Incomplete reporting also potentially introduces a degree of bias into the study’s results. The self-reporting nature of the data collection may also be a limitation with the potential for self-reporting bias and overstating positive outcomes of the research as well as a bias towards completion with only those who felt they were successful bothering to complete the final report The short-term nature of the evaluation is a further limitation to capturing the outputs and impacts that may have arisen over time. It is widely acknowledged that research impacts on health and broader societal and economic outcomes is associated with significant time lags. There is therefore a need for further evaluation beyond the two-year funding period to better understand the extent of theses outcomes and impacts of the RTP program.
Conclusion
The RTP program has contributed to common measures of research impact such as advancing knowledge, building research capacity and collaboration and, to a lesser extent, contributing to few yet worthy changes to policy and practice. These outcomes have been achieved through clinician-initiated projects, in collaboration with academic researchers, and implemented within local health organisations in response to an opportunity to initiate change with a potential to generate efficiencies. This approach to initiating change is consistent with one of the enduring strategies put forward by the recent Western Australia Sustainable Health Review, which reiterates the notion of partnerships with clinicians, consumers and a wide range of partners being needed to develop, test and spread initiatives that deliver better patient care, and are vital to achieving a more sustainable health system [29].
While the research presented here highlights the short term research outputs and contributions to research capacity building and collaboration, it does not explore how these outcomes were achieved or the mechanisms which may contribute to the spread and uptake of research and innovations across the system in the longer term. In addition, a lens that looks beyond the two-year funding period would provide insight into research outcomes that are subject to time lags, such as changes in policy and practice and economic impacts at a system or societal level. Further research is required to better understand the mechanisms triggered by the RTP program to facilitate the spread and uptake of research innovation across the system over time. Understanding the ‘how’ will enable the recommendation made by the Sustainable Health Review to be implemented more effectively to achieve a more sustainable health system.
References
- 1. Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. Implement Sci. 2012;7(1):50–. pmid:22651257
- 2. Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JPA, et al. Biomedical research: increasing value, reducing waste. Lancet. 2014;383(9912):101–4. pmid:24411643
- 3. van der Laan AL, Boenink M. Beyond bench and bedside: disentangling the concept of translational research. Health Care Anal. 2015;23(1):32–49. pmid:23248053
- 4. Rychetnik L, Bauman A, Laws R, King L, Rissel C, Nutbeam D, et al. Translating research for evidence-based public health: key concepts and future directions. J Epidemiol Community Health. 2012;66(12):1187–92. pmid:22569750
- 5. Milat AJ, Li B. Narrative review of frameworks for translating research evidence into policy and practice. Public Health Res Pract. 2017;27(1). pmid:28243670
- 6. Blevins D, Farmer MS, Edlund C, Sullivan G, Kirchner JE. Collaborative research between clinicians and researchers: a multiple case study of implementation. Implement Sci. 2010;5(1):76–. pmid:20946658
- 7. Sahs JA, Nicasio AV, Storey JE, Guarnaccia PJ, Lewis-Fernández R. Developing Research Collaborations in an Academic Clinical Setting: Challenges and Lessons Learned. Community Ment Health J. 2016;53(6):647–60. pmid:27900495
- 8. Brazil K, Ozer E, Cloutier MM, Levine R, Stryer D. From theory to practice: improving the impact of health services research. BMC Health Serv Res. 2005;5(1):1–. pmid:15638931
- 9. Lomas J. Using ’linkage and exchange’ to move research into policy at a Canadian foundation. Health Aff (Millwood). 2000;19(3):236. pmid:10812803
- 10. Gagliardi AR, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci. 2016;11(1):38–. pmid:26988000
- 11. Misso ML, Ilic D, Haines TP, Hutchinson AM, East CE, Teede HJ. Development, implementation and evaluation of a clinical research engagement and leadership capacity building program in a large Australian health care service. BMC Med Educ. 2016;16(1):13.
- 12. Sibbald SL, Tetroe J, Graham ID. Research funder required research partnerships: a qualitative inquiry. Implement Sci. 2014;9(1):176–. pmid:25430813
- 13.
Government of Western Australia Department of Health. Research Translation Projects: Evidence for a sustainable health system 2019 [Available from: https://ww2.health.wa.gov.au/Articles/N_R/Research-Translation-Projects.
- 14. McLean RKD, Graham ID, Tetroe JM, Volmink JA. Translating research into action: an international study of the role of research funders. Health Res Policy Syst. 2018;16(1):44–. pmid:29793541
- 15. Deeming S, Reeves P, Ramanathan S, Attia J, Nilsson M, Searles A. Measuring research impact in medical research institutes: A qualitative study of the attitudes and opinions of Australian medical research institutes towards research impact assessment frameworks. Health Research Policy and Systems. 2018;16(1):<xocs:firstpage xmlns:xocs = ""/>. pmid:29548331
- 16. Cruz Rivera S, Kyte DG, Aiyegbusi OL, Keeley TJ, Calvert MJ. Assessing the impact of healthcare research: A systematic review of methodological frameworks. PLoS medicine. 2017;14(8):e1002370–e. pmid:28792957
- 17. Penfield T, Baker MJ, Scoble R, Wykes MC. Assessment, evaluations, and definitions of research impact: A review. Research evaluation. 2014;23(1):21–32.
- 18.
Panel on Return on Investment in Health Research. Making an Impact: A Preferred Framework and Indicators to Measure Returns on Investment in Health Research. Ottawa, ON, Canada Canadian Academy of Health Sciences; 2009.
- 19. Raftery J, Hanney S, Greenhalgh T, Glover M, Blatch-Jones AJHTA. Models and applications for measuring the impact of health research: Update of a systematic review for the Health Technology Assessment Programme. 2016;20.
- 20. Greenhalgh T, Raftery J, Hanney S, Glover M. Research impact: a narrative review. BMC medicine. 2016;14(78):78–. pmid:27211576
- 21. Hanney S, Buxton M, Green C, Coulson D, Raftery J. An assessment of the impact of the NHS Health Technology Assessment Programme. Health technology assessment (Winchester, England). 2007;11(53):iii. pmid:18031652
- 22. Buxton M, Hanney S. How Can Payback from Health Services Research Be Assessed? Journal of Health Services Research & Policy. 1996;1(1):35–43. pmid:10180843
- 23. Cooke J, Gardois P, Booth A. Uncovering the mechanisms of research capacity development in health and social care: a realist synthesis. Health Res Policy Syst. 2018;16(1):93–22. pmid:30241484
- 24. Cooke J. A framework to evaluate research capacity building in health care. BMC family practice. 2005;6(1). pmid:16253133
- 25. Sarre G, Cooke J. Developing indicators for measuring Research Capacity Development in primary care organizations: a consensus approach using a nominal group technique. Health Soc Care Community. 2009;17(3):244–53. pmid:19040697
- 26. Ravaghi H, Afshari M, Isfahani P, Bélorgeot VD. A systematic review on hospital inefficiency in the Eastern Mediterranean Region: sources and solutions. BMC Health Serv Res. 2019;19(1):830–20. pmid:31718648
- 27.
Chisholm D, Evans D. Improving health system efficiency as a means of moving towards universal coverage. Geneva, Switzerland World Health Organisation 2010.
- 28. Miranda R, Garcia-Carpintero E. Comparison of the share of documents and citations from different quartile journals in 25 research areas. Scientometrics. 2019;121(1):479–501.
- 29.
Sustainable Health Review: Final Report to the Western Australian Government. Department of Health, Western Australia; 2019.
- 30. Matus J, Walker A, Mickan S. Research capacity building frameworks for allied health professionals–a systematic review. BMC Health Serv Res. 2018;18(1):716–. pmid:30219065
- 31. Cooke J, Nancarrow S, Dyas J, Williams M. An evaluation of the ’Designated Research Team’ approach to building research capacity in primary care. BMC Fam Pract. 2008;9(1):37–. pmid:18588685
- 32. Bennett S, Whitehead M, Eames S, Fleming J, Low S, Caldwell E. Building capacity for knowledge translation in occupational therapy: learning through participatory action research. BMC Medical Education. 2016;16(1):257. pmid:27716230
- 33. Traill CL, Januszewski AS, Larkins R, Keech AC, Jenkins AJ. Time to research Australian physician‐researchers. Intern Med J. 2016;46(5):550–8. pmid:26909676
- 34. Eley DS. The clinician-scientist track: an approach addressing Australia’s need for a pathway to train its future clinical academic workforce. BMC Med Educ. 2018;18(1):227–. pmid:30285826
- 35. Levine R, Russ-Eft D, Burling A, Stephens J, Downey J. Evaluating health services research capacity building programs: Implications for health services and human resource development. Evaluation and program planning. 2013;37:1–11. pmid:23291786