Service evaluation of the implementation of a digitally-enabled care pathway for the recognition and management of acute kidney injury

Acute Kidney Injury (AKI), an abrupt deterioration in kidney function, is defined by changes in urine output or serum creatinine. AKI is common (affecting up to 20% of acute hospital admissions in the United Kingdom), associated with significant morbidity and mortality, and expensive (excess costs to the National Health Service in England alone may exceed £1 billion per year). NHS England has mandated the implementation of an automated algorithm to detect AKI based on changes in serum creatinine, and to alert clinicians. It is uncertain, however, whether ‘alerting’ alone improves care quality. We have thus developed a digitally-enabled care pathway as a clinical service to inpatients in the Royal Free Hospital (RFH), a large London hospital. This pathway incorporates a mobile software application - the “Streams-AKI” app, developed by DeepMind Health - that applies the NHS AKI algorithm to routinely collected serum creatinine data in hospital inpatients. Streams-AKI alerts clinicians to potential AKI cases, furnishing them with a trend view of kidney function alongside other relevant data, in real-time, on a mobile device. A clinical response team comprising nephrologists and critical care nurses responds to these AKI alerts by reviewing individual patients and administering interventions according to existing clinical practice guidelines. We propose a mixed methods service evaluation of the implementation of this care pathway. This evaluation will assess how the care pathway meets the health and care needs of service users (RFH inpatients), in terms of clinical outcome, processes of care, and NHS costs. It will also seek to assess acceptance of the pathway by members of the response team and wider hospital community. All analyses will be undertaken by the service evaluation team from UCL (Department of Applied Health Research) and St George’s, University of London (Population Health Research Institute).

We have thus developed a digitally-enabled care pathway as a clinical service to inpatients in the Royal Free Hospital (RFH), a large London hospital. This pathway incorporates a mobile software application -the "Streams-AKI" app, developed by DeepMind Health -that applies the NHS AKI algorithm to routinely collected serum creatinine data in hospital inpatients. Streams-AKI alerts clinicians to potential AKI cases, furnishing them with a trend view of kidney function alongside other relevant data, in real-time, on a mobile device. A clinical response team comprising nephrologists and critical care nurses responds to these AKI alerts by reviewing individual patients and administering interventions according to existing clinical practice guidelines.
We propose a mixed methods service evaluation of the implementation of this care pathway. This evaluation will assess how the care pathway meets the health and care needs of service users (RFH inpatients), in terms of clinical outcome, processes of care, and NHS costs. It will also seek to assess Acute kidney injury (AKI) is a sudden loss of kidney function, defined by a rise in serum creatinine or a fall in urine volume 1 . It has diverse causes, which include sepsis or acute infection, hypovolaemia or hypoperfusion, nephrotoxicity (from drugs or radiological contrast), obstruction of the renal tract, and primary renal diseases such as acute glomerulonephritis. In the United Kingdom, AKI affects up to 15% of hospital admissions and 20% of emergency admissions 2,3 . AKI may result in fluid overload, respiratory failure and metabolic derangements such as hyperkalaemia 4 , and is thus strongly associated with adverse outcomes including death 5 , prolonged hospitalisation 6 , requirement for renal replacement therapy 7 , and a need for high dependency or intensive care 8 . It is also associated with an increased lifetime risk of chronic kidney disease 9 . AKI is also expensive: the associated excess costs to the National Health Service (NHS) in England may exceed £1 billion per annum 3 .
Management of AKI involves four processes of care: timely recognition, general supportive care, therapy directed at the underlying cause, and management of complications. Across the NHS, there are substantial cross-pathway deficits in care 10 . Increasing awareness of these and of the clinical and economic impact of this condition has led to local, regional, national and global initiatives to try to prevent AKI from occurring, and to encourage timely and appropriate interventions to prevent progression and deliver more rapid recovery. Clinical practice guidelines for the management of AKI have now been developed 11 .
More recently, on the basis that the prompt and reliable identification of AKI cases to clinicians may trigger improved care, NHS England issued a national patient safety alert on "standardising the early identification of Acute Kidney Injury" 12 . This mandated the installation of a new detection algorithm in each NHS hospital, so that potential AKI incidents could be flagged to treating clinicians. The 'Think Kidneys' NHS England National programme has provided best practice examples of how AKI alerts may be clinically deployed 13 . However, simply alerting a clinician to the presence of a possible AKI incident may be insufficient to improve outcomes 14 . A much richer clinical dataset is required to help clinicians prioritise, diagnose and manage patients. The UK's National Institute for Health and Care Excellence (NICE) guidelines on AKI management suggest that patients with more severe AKI might benefit from care delivery by suitably expert clinicians, for example as part of a 'rapid referral' nephrology service 11 .
The Royal Free Hospital (RFH) will implement existing standards for best practice through deployment of a digitally-enabled care pathway as a core service to hospital inpatients. A key component of this is a mobile software application (Streams-AKI). This application will identify potential new cases of AKI by applying the NHS AKI algorithm to a live stream of serum creatinine data, providing real-time monitoring of kidney function. The application will provide real-time alerts of potential AKI cases, alongside other critical clinical data, to a clinical response team comprising nephrologists and critical care nurses. This response team will assess the data provided by the application, prioritise cases and then deliver investigations and therapies according to current best practice guidelines 11 .
We propose a service evaluation of the introduction of the digitally-enabled care pathway with respect to processes of care, patient outcomes, qualitative feedback from patients and staff, and NHS costs.

Aims and objectives
This service evaluation aims to assess the benefits of the implementation of the digitally-enabled AKI pathway from January 2017, with respect to clinical outcome, renal recovery, processes of care, and NHS costs. The evaluation will also assess the experience of service users (RFH in-patients who have been treated by the pathway), members of the clinical response team and the wider clinical community of RFH.

Service evaluation setting
The Streams-AKI app and clinical response team are deployed at a single hospital site within the Royal Free London Foundation Trust (RFLFT): the RFH, an 800-bed teaching hospital which also provides diverse specialist and tertiary services, including a dialysis unit and 34-bed intensive care unit with renal replacement therapy onsite. The outcomes of this evaluation will inform further development of the care pathway and its deployment in other RFLFT sites.
Following the implementation of the digitally-enabled care pathway, data from the RFH will be compared with data from the RFH prior to deployment in addition to pre-deployment and postdeployment data from a second hospital that is part of the RFLFT, Barnet General Hospital (BGH). BGH is a 450-bed district general hospital providing acute care (including onsite renal replacement therapy, a 21-bed ITU and on-site nephrology services), with similar arrangements for the care of AKI patients to that at the RFH prior to implementation of the digitally-enabled care pathway.
The RFH has onsite wireless internet networks available to clinicians, and was classified as a global digital exemplar (GDE) by NHS England in April 2016.
The AKI care pathway prior to implementation. Serum creatinine, an indicator of kidney function, is currently measured in the hospital laboratory. The current AKI detection algorithm in the laboratory information management system (which predates the

Amendments from Version 1
The article has been revised in response to two referee reports. We have clarified the timeline for clinical intervention, and the rationale behind using Barnet General Hospital as a comparator site. A section has been added to discuss the determination by the Office of the Information Commissioner. Finally, we have added a follow-up measurement of renal function to the list of secondary outcomes, and have appended the care protocol as a Supplementary Figure. See referee reports REVISED national algorithm) identifies potential AKI cases and presents a message for clinicians in both the hospital results system and electronic patient record. This message also flags the availability of clinical guidance and education via a link to a webpage displaying local guidelines (www.londonaki.net). Prior to implementation, such results were normally batch reviewed by nonspecialists at the end of the day and may have been seen several hours after the results first become available. Clinicians may have opted to review results earlier, but this process relied upon repeated accessing of the results systems, as clinicians did not know when results were ready. Where blood tests suggested AKI, this may have been communicated by telephone to the clinical teams responsible for the patient by the biochemistry laboratory. However, this process was cumbersome and may have been unreliable.
Patients develop AKI in multiple wards and settings. Early management of AKI was overwhelmingly delivered by nonspecialist, ward-based teams with primary clinical responsibility for the patient. Specialist nephrology review of kidney function blood tests or of patients with AKI only occurred if requested by the patient's responsible (or 'home') clinical team. This required the home team to assess kidney function results, assess the patient, decide to option a specialist review, contact the renal team via phone or pager systems and await a response. The renal team would then receive verbal referral information or would manually access results and other clinical data to prioritise the referral, managing information relating to multiple referrals with paper-based processes. The hospital's critical care nursing team did not receive automated referrals and were entirely reliant on being contacted by pager systems when ward staff were concerned that a patient was deteriorating.
The RFLFT deployed clinical guidelines and had an active AKI education programme to support clinical teams, but local audit showed that performance in managing AKI varied, and did not always consistently meet national standards 15 . Quality improvement in AKI care is a RFLFT organisational priority, and has driven the development of the care pathway and its proposed evaluation.
The digitally-enabled care pathway comprising the Streams-AKI app and the AKI clinical response team The digitally-enabled care pathway is the service whose implementation is being evaluated.

The Streams-AKI application.
Streams-AKI is a mobile application that is deployed on iPhone Operating System (iOS)-enabled smartphones. It processes routinely-collected demographic data (i.e. patient identifiers, location, responsible consultant, and responsible medical specialty), and also serum creatinine data in real time according to the nationally mandated NHS AKI algorithm, which grades patients' AKI stage from 1 to 3. When the algorithm identifies a case of AKI, a patient-specific notification is delivered directly to the clinician user's iOS device. In current clinical practice, clinicians must distinguish patients with clinically relevant changes in creatinine from those without, through review of current and historical blood tests, or elements of past medical history that indicate disease causality, complications or pre-existing risk. These routine data are therefore displayed in-app alongside the AKI alert to facilitate interpretation and clinical decision making. The Streams-AKI app is fully integrated with the existing RFH electronic health record system, and operates on Fast Healthcare Interoperability Resources (FHIR) interoperability standards. During implementation, it will be utilised alongside existing electronic health record software. Data security is ensured through the use of on-disk (AES256) and in-flight encryption (TLS v1.2) for all app data in compliance with NHS Digital information security guidelines. The app was first registered with the Medicines and Healthcare Products Regulatory Agency (MRHA) as a Class I, non-measuring, non-sterile medical device on 30/08/2016.
The clinical response team. At the RFLFT, Streams-AKI will be installed on RFLFT-owned iOS devices. These will be held by members of a dedicated clinical response team, consisting of a clinical lead nephrologist, a duty consultant nephrologist, a specialty nephrology registrar, and a critical care outreach nurse. Following in-app review of alerts, all patients determined to be suffering from clinically-relevant AKI will receive a prompt bedside review form the nephrology team. The timeline to intervention after alerts has not been standardized, recognising that 'real world' clinical pressures and judgement may require prioritisation of actions within available resource constraints. We have, instead, specified immediate review for patients with life threatening results (e.g. hyperkalaemia), and that other, less urgent, cases are viewed within hours. The Streams app allows clinicians to view results and triage patients for early review based on perceived clinical urgency. The timeframe for review will therefore depend on clinical prioritisation and other (clinically determined) variable demands.
Members of the clinical response team will then administer a standardized care protocol. The critical care outreach nurse will also receive alerts on more severe (stage 2 and 3) cases and will respond to the most severely unwell cases according to clinical judgement of patient risk. All interventions and future care requirements will be communicated to responsible clinicians verbally and through a standard written proforma (Supplementary Figure 1) entered into the patient record. Where necessary, the clinical response team will arrange a further review within 24 hours. The Streams-AKI app will further alert the team if the patient's AKI stage subsequently worsens and will also alert after 48 hours if a patient is still suffering from AKI, as determined by the national AKI algorithm. The team will respond to such follow up alerts according to best practice and clinical judgement.
All clinicians using the app first receive training. This comprises: -A detailed review of the Streams-AKI app -Introduction to the devices being used to host the app: RFLFTowned iPhones (Apple, Inc., Cupertino, Calif., USA) -An introduction to the other members of the response team -Review of the standardised digitally-enabled AKI care protocol.
Streams-AKI was deployed in January 2017. Following a pilot phase of 12 weeks to allow optimisation of the care pathway, an 18-week evaluation phase commenced, during which outcome data will be accrued (see Figure 1). For comparative analysis, data will be collected from the two hospital sites (RFH and BGH) at three time points: -One year before deployment (January to August 2016) -Immediately before deployment (September 2016 to January 2017) -During deployment (January to August 2017) An interim analysis is planned half way through the evaluation phase. Results from this may inform the duration of the service evaluation.
Evaluation sample. The service implementation evaluation will include data from all inpatients triggering an AKI alert as defined by the national AKI detection algorithm who are aged 18 or over.
Utilising a daily data feed from the RFLFT chronic dialysis database, AKI alerts for patients receiving kidney dialysis will be identified by the application itself and removed. Alerts for inpatients on the Acute Kidney or Critical Care Units will also be removed. Patients on an end-of-life pathway at the time of the AKI alert will be excluded.
Outcome measures. The evaluation of the pathway will employ a mixture of both quantitative and qualitative methods. The primary outcome will be recovery of renal function, which will be defined as a return to a creatinine level within 120% of the baseline (as defined by the National AKI algorithm) prior to discharge from hospital. Secondary outcomes will be categorised into four areas: processes of care, clinical outcomes, Trust-wide metrics, and NHS costs. Definitions of each outcome, and the sources of all data to be collected, are provided in Table 1-Table 4. Qualitative evaluation. During the 12-week pilot phase, the clinical response team was observed by a member of the service evaluation team, allowing key issues relating to both the technological and clinical aspects of the enhanced care pathway (including resource use) to be recorded. Semi-structured interviews were carried out with a selection of response team members (including nephrology consultants and specialty registrars, and critical care outreach nurses). The interviews explored whether the clinical response team members found that the new care pathway and the Streams-AKI application helped them provide best-practice care for patients, which aspects of the digitally-enabled pathway worked well or where they could be improved, adverse experiences or consequences of app use, and any unexpected indirect beneficial or adverse effects. Observational work and interviews carried out during the pilot phase were used to drive iterative improvements in the digitally-enabled care pathway prior to the beginning of the evaluation phase.
At the conclusion of the evaluation phase, a selection of doctors and nurses will be invited to a second series of semi-structured interviews. We will specifically target those responsible for the care of those patients who have been reviewed by the AKI response team during the evaluation. Participants will be purposively sampled to include a mixture of grades of clinicians (House Officers; Senior House Officers; Registrars; Consultants) and nurses (Staff Nurse; Charge Nurse). These will explore strengths and weaknesses of the digitally-enabled care pathway; how these map to perceived deficiencies in AKI care; how the new pathway affected the quality and equity of patient care; and how they feel the service could be improved. Approximately 20 interviews will be carried out. The exact number of health care professionals interviewed will be determined by the need to achieve sufficient diversity in elicited accounts for the questions to fully address all major themes, with no new issues arising at further interviews. Figure 1 outlines the timeframes for each phase of the service evaluation, as described above.

Statistical analysis plan and power calculation
We will use interrupted time series segmented regression analysis to examine the effect of the new AKI service on the weekly patient recovery rate for acute kidney injury within 30 days; recovery is defined as a return to creatinine level within 120% of baseline level. The primary dependent variable (recovery of renal function) will be modelled using a generalised linear model assuming a binomial distribution and using a logit link. This modelling approach will ensure that predicted values yielded from the model cannot fall outside the valid 0-100% range. We will also allow for autocorrelation in the model, which can be an issue with time series data. Using this approach, it will be possible to test for a change in level and/or regression slope following the implementation of the intervention.
Pathological and clinical endpoints will also be compared to those from a partner hospital (BGH) not deploying the digitallyenabled care pathway over the same time period, and from the preimplementation periods as specified above. BGH has been included as a comparison site to allow us to demonstrate that impacts attributed to the service being evaluated are not due to systemic changes in process locally or nationally (e.g. relating to the National awareness campaign, 'Think Kidneys'). Baseline care pathways are similar at BGH and RFH.
It is anticipated that the care pathway will be subsequently deployed in BGH, with such deployment informed by the results of this service evaluation. Although comparison with the BGH comparator site should negate any seasonal effects, this is based on the assumption that the effect of time is the same in both intervention and comparator sites. The inclusion of the second pre-intervention period (i.e. immediately preceding the intervention period) will allow us to assess whether this is a fair assumption.
For the economic analysis, unit costs for each of the components will be obtained from two sources. First, we will use tariffs from the NHS National Tariff Payment System. Second, we will use local tariffs at RFLFT sites. Costs will be calculated using both sets of unit costs. We will multiply resource use by unit costs for each patient/cost component and sum across patients to calculate total costs per patient. The output will be a patient-level dataset of total costs per patient before and after the introduction of the digitally-enabled care pathway. Any attributable cost savings will be balanced against the additional costs of the alerting system and care pathway. This will include estimates of the use of clinician time; an activity observation exercise will be carried out with response team members during the evaluation period.
A sensitivity analysis will be completed to assess the cost of care delivered to patients with AKI alerts that were discarded at clinician triage.
At the point of writing this service evaluation protocol, there are no published sample size calculations available for determining the number of timepoints needed for a well powered service evaluation using an interrupted time series design. In line with best practice in service evaluation, we therefore utilised simulations implementing the SIMSAM command in Stata (v14) (StataCorp LLC, College Station, Texas, USA) to establish the sample size needed. We simulated data containing weekly referral rates for four years prior to intervention, where the intervention will occur at 208 weeks. The average baseline recovery rate was assumed to be 0.51 (SD 0.08) which was determined using one year of pre-intervention data from the Royal Free Hospital, the site where the intervention will be implemented. One hundred observations (patients) or more per timepoint are encouraged 16 , which is a viable assumption based on historical data. A normally distributed random variable with mean of zero and standard deviation of 0.08 was generated to simulate the variation in recovery rate. The pre-intervention regression slope and the change in the effect of the intervention over time following the intervention were both assumed to have an odds ratio of one. The recovery rate was generated as a function of these effects, the average baseline referral rate and the random variable.
The number of timepoints (measured on a weekly basis) needed to detect an odds ratio of 1.15 for the intervention effect with 90% power assuming a significance level of 5%, determined by simulation, was 11 in total. This number of post-intervention timepoints increased to 32 if the size of effect to be detected has an odds ratio of 1.1, i.e. a 10% increase in the odds of recovery. The number of timepoints needed to detect an odds ratio of 1.1 for the intervention effect with 80% power assuming a significance level of 5%, determined by simulation, was 20 in total. In further analyses, the segmented regression model will be extended to include a second hospital site for comparison of the change in level or change in slope post intervention. This will be Barnet Hospital, which will not receive the intervention.
All other quantitative data collected (see Table 1) will be analysed using Stata. Data will be screened for normality and homogeneity of variance prior to analysis.
A series of interim analyses will be performed midway through the evaluation phase. These will include inter-and intra-operator analyses of variance for transcription of process of care data from patient notes, and initial modelling of primary outcomes. These analyses will be carried out by staff from University College London and St George's University of London.
Analysis of qualitative data. The semi-structured interviews will be digitally recorded and transcribed verbatim. We will analyse the data from a realist viewpoint 17 , but our model will be revised to fit emerging interpretations of the data. We will apply a framework approach 18 , whereby the initial transcripts will undergo scrutiny by one member of the service evaluation team (AC) in order to gain familiarity with the data and to identify key themes. These will be discussed and further scrutinised with another service evaluation team member (RR), and with emerging interpretations or questions will be shared and critically explored with the with the entire service evaluation team. Transcripts will be analysed in a systematic manner by applying the coding framework and by rearranging the data according to the thematic content. Generalisations that represent the total set and that address each of the objectives will be developed. During analysis, we will maintain a constant vigilance for deviant cases that may question the emerging thematic and conceptual relations.

Steering committee.
A steering committee for this service evaluation has been convened at RFLFT. This committee includes an independent chair with no relationship to the project, a patient service evaluation has also been presented to the UK's National Institute for Health Research (NIHR) North Thames Collaboration for Leadership in Applied Health Research and Care (CLAHRC) Patient and Public Involvement Panel. The service evaluation has the full support of the Royal Free Kidney Patients Association.
It is theoretically possible that implementation of the digitallyenabled care pathway could have unforeseen adverse consequences. These will be sought through clinical feedback at weekly Trust implementation meetings and monthly patient safety programme operational group meetings. Additionally, during the service evaluation, broad metrics of care quality and safety will be monitored by the Trust, and these will be reviewed by the independent steering committee at RFLFT for this evaluation (described above).

Dissemination of findings
Results will be presented to the renal, acute medicine and critical care departments, and to the RFLFT Patient Safety Committee and RFLFT Board. The findings will inform subsequent developments of the care pathway and will inform the RFLFT strategy for detecting and managing AKI. Data will also be presented on the Trust website via our patient safety portal and presented in lay language. Our findings will be shared at a learning event of the UCL Partners AKI Quality Improvement Collaborative, which has 9 participant Trusts as well as a London AKI Network event and as a case study to Think Kidneys (the NHS England National AKI Programme). We will publish in relevant scientific journals, and present at conferences.
Competing interests CL, HM, GR, and RR are paid clinical advisors to DeepMind. AC's clinical research fellowship is part-funded by DeepMind. DeepMind will remain independent from the collection and analysis of all data. HM co-holds a patent on a fluid delivery device which might ultimately help in preventing some (dehydrationrelated) cases of AKI occurring.

Grant information
The service evaluation will be conducted by NIHR-CLAHRC North Thames, based at UCL and directed by RR. RR is also an NIHR Senior Investigator. GR is funded in part by the NIHR University College London Hospitals Biomedical Research Centre. HM was similarly funded during project inception. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.
member and a nephrologist from a different NHS Trust. This committee will review the results of the interim analyses discussed above.
Ethics, information and clinical governance RFLFT uses Streams as part of its provision of care for patients.
To support this service, DeepMind Health processes Patient Identifiable Data. This is in line with the governance arrangements for all other clinical software applications, and this arrangement forms part of an information processing agreement with the Trust, which has been published on the DeepMind website. The digitallyenabled care pathway will be a new standard service at RFH, and under NHS guidance there are therefore no consent requirements for patients for the processing of their personally identifiable data for direct patient care functions.
A determination by the Office of the Information Commissioner (ICO) was published in July 2017. This primarily focused on the clinical safety testing phase of Streams-AKI that occurred between app development (using only synthetic data) and the full clinical implementation that is being evaluated. The ICO ruled that the processing of data for the clinical safety testing of the application did not amount to processing for direct care purposes and so did not fully comply with data protection law. Concerns were also raised that not enough was done to ensure that patients and the public were aware of the project prior to data processing. RFLFT undertook such clinical safety testing in the interests of patient safety, but has accepted the determination by the ICO and is in the process of completing her specified undertakings. The Trust statement on this determination is available on their website.
When undertaking the service evaluation, data will be transferred from RFLFT to the Department of Applied Research, University College London for analysis. Prior to transfer and analysis, all data will be de-identified, meaning that no consent will be required from patients for this purpose.
Plans for the evaluation of the digitally-enabled care pathway have been independently reviewed by the University College London Joint Research Office. They directed that this project falls under the remit of service evaluation, as per "Defining Research" guidance from the NHS Health Research Authority. As such, the service evaluation has been registered locally with the RFLFT Audit Lead and Medical Director. The service evaluation has the approval of the RFLFT Executive, RFLFT Board and Sub-Board Patient Safety Committee (including patient governor representatives and a non-Executive Chair and RFLFT Board Member). The Patient Safety Committee and RFLFT Board will receive reports on the results of the service evaluation. The

Supplementary Data
Supplementary Figure 1: the standardized Streams-AKI proforma. The outcomes of patient review (including suggested interventions and future care requirements) are communicated to responsible clinicians via a standard written proforma that is entered into the patient record.
Click here to access the data.

1.
as a primary measure, and if it has not, where in the process has this failed. However, I think the methodology and the inclusion of a number of secondary outcome definitions is sensible, useful and achievable in the context of the clinical problem. There is only really one other study that, unsuccessfully, tried to answer similar clinical questions (Wilson ) so the authors have minimal literature to guide them et al yet there approach is sound. The analysis will incorporate a before and after side to the study at the intervention hospital but there is also a second site control which again adds strength to the protocol.
This study cannot be replicated without access to the technology stream AKI app, however, should this trial show benefit, a wider distribution by the sponsor to a number of different hospital types for a multicentre study would seem logical.
I would however ask the authors to consider on further outcome measure metric. The literature is clear that renal function measured by serum creatinine at discharge following significant illness can over estimate renal function (Prowle CJASN). This may confound the interpretation of success by a d/c et al creatinine being within 20% of baseline. A series of follow up creatinine measures at 3 and potentially 6 months would be a more robust measure of AKI treatment success as the long term effects of AKI on renal function is potentially the most important success of early AKI intervention.
Finally, following recent media coverage in the UK in relation a court judgement regarding the sharing of patient data between the RFLFT and DeepMind, can the authors clarify for the benefit of all readers that this project remains legally and ethically sound.
I very much look forward to the results of this important study.

Are sufficient details of the methods provided to allow replication by others? Yes
Are the datasets clearly presented in a useable and accessible format? Yes We thank Dr. Kirwan for his helpful comments, to which we are happy to be able to respond: We agree that structured recording of follow-up creatinine at three and six months would be 1.
We agree that structured recording of follow-up creatinine at three and six months would be highly desirable. RFLFT does not currently have the infrastructure to implement long term hospital-based AKI follow-up for all patients with AKI exposed to the pathway under evaluation, nor do we have funding or resource for such work at this time. This may be a future development. However, we will present follow-up creatinine data from both intervention and comparator sites and time periods where these are readily available through existing clinical practice on the reviewer's recommendation. Though hospital-based, structured AKI follow up clinics are not part of the current service improvement it is quite possible that more reliable detection and more rigorous early management will result in more reliable AKI follow up either through referral to renal clinic, non-renal secondary care clinics or through primary care follow up. We will attempt to study this effect.
We fully understand the reviewer raising this point, and agree that it requires clarification. The determination by the Office of the Information Commissioner (ICO) primarily focused on the clinical safety testing phase of Streams-AKI that occurred between app development (using only synthetic data) and the full clinical implementation that is being evaluated. The ICO ruled that the processing of data for the clinical safety testing of the application did not amount to processing for direct care purposes and so did not fully comply with data protection law. Concerns were also raised that not enough was done to ensure that patients and the public were aware of the project prior to data processing. RFLFT undertook such clinical safety testing in the interests of patient safety, but has accepted the determination by the ICO and is in the process of completing her specified undertakings. The Trust statement on this determination is available on their website: https://www.royalfree.nhs.uk/patients-visitors/how-we-use-patient-information/information-commissioners-off No competing interests were disclosed.

3.
Overall, this is an exciting study with potentially profound implications for care of patients with AKI.

Is the study design appropriate for the research question? Yes
Are sufficient details of the methods provided to allow replication by others? Partly We thank Prof. Rosner for his insightful comments, to which we are happy to be able to respond: We have appended the care protocol for the intervention as a supplementary figure.
The timeline to intervention after alerts has not been standardized, recognising that 'real world' clinical pressures and judgement may require prioritisation of actions within available resource constraints. We have, instead, specified immediate review for patients with life threatening results (e.g. hyperkalaemia), and that other, less urgent, cases are viewed within hours. The Streams app allows clinicians to view results and triage patients for early review based on perceived clinical urgency. The timeframe for review will therefore depend on clinical prioritisation and other (clinically determined) variable demands.
Barnet General Hospital has been included as a comparison site to allow us to demonstrate that impacts attributed to the service being evaluated are not due to systemic changes in process locally or nationally (e.g. relating to the English national awareness campaign, 'Think Kidneys'). Resources do not allow us to implement the new care pathway simultaneously in Barnet and Hampstead sites, though we plan to implement the new service on the Barnet site as part of our staged service improvement plans. Learning from this evaluation will inform that process. We anticipate site-specific variation in outcomes relating to casemix, as Professor Rosner rightly points out. Baseline care pathways are, however, similar at Barnet and Hampstead sites: there is a legacy alert system in the results viewing platform, non-specialists undertake batch review of results, biochemistry staff phone out some AKI results, early care is administered by unsupervised non-specialist teams with discretionary escalation to expert input via bleep or phone referrals. We thus feel it is a useful site to compare process outcomes and time-series trends in both process and