Skip to main content
  • Research article
  • Open access
  • Published:

Integration of a nationally procured electronic health record system into user work practices

Abstract

Background

Evidence suggests that many small- and medium-scale Electronic Health Record (EHR) implementations encounter problems, these often stemming from users' difficulties in accommodating the new technology into their work practices. There is the possibility that these challenges may be exacerbated in the context of the larger-scale, more standardised, implementation strategies now being pursued as part of major national modernisation initiatives. We sought to understand how England's centrally procured and delivered EHR software was integrated within the work practices of users in selected secondary and specialist care settings.

Methods

We conducted a qualitative longitudinal case study-based investigation drawing on sociotechnical theory in three purposefully selected sites implementing early functionality of a nationally procured EHR system. The complete dataset comprised semi-structured interview data from a total of 66 different participants, 38.5 hours of non-participant observation of use of the software in context, accompanying researcher field notes, and hospital documents (including project initiation and lessons learnt reports). Transcribed data were analysed thematically using a combination of deductive and inductive approaches, and drawing on NVivo8 software to facilitate coding.

Results

The nationally led "top-down" implementation and the associated focus on interoperability limited the opportunity to customise software to local needs. Lack of system usability led users to employ a range of workarounds unanticipated by management to compensate for the perceived shortcomings of the system. These had a number of knock-on effects relating to the nature of collaborative work, patterns of communication, the timeliness and availability of records (including paper) and the ability for hospital management to monitor organisational performance.

Conclusions

This work has highlighted the importance of addressing potentially adverse unintended consequences of workarounds associated with the introduction of EHRs. This can be achieved with customisation, which is inevitably somewhat restricted in the context of attempts to implement national solutions. The tensions and potential trade-offs between achieving large-scale interoperability and local requirements is likely to be the subject of continuous debate in England and beyond with no easy answers in sight.

Peer Review reports

Background

The literature is littered with examples of "failed" electronic health record (EHR) implementations in healthcare settings, and particularly large-scale ones although these are, in the light of the significant anticipated benefits, increasingly pursued internationally [1–6]. A recurring theme in this literature is that of clinicians finding it difficult to integrate the use of information technology (IT) systems into their work practices and care provision [6–20]. As a result, end-users may either partially use (i.e. only use the parts that they perceive as useful), develop "workarounds" (which we define as behaviour employed by users to overcome a perceived limitation in a technical system), or avoid using the system altogether ("non-compliance") [21–23]. These coping strategies may in turn lead to the technology being used in ways other than originally intended or not being used at all, which can result in important disturbances to organisational functioning and worse still compromise patient safety [24].

This literature indicates that difficulty integrating technology with the work practices of healthcare staff is particularly likely in situations in which a new system is perceived as being imposed on users [21]. Integration of EHRs with user work practices is therefore a central concern for large-scale national ventures where system design often reflects a focus on interoperability (i.e. the ability meaningfully to exchange information between systems). A key challenge then is to design a system that is able to deal with the potentially multiple (at times conflicting) requirements of different stakeholders and within different user groups in an environment that is characterised by complexity and variability [19, 25]. It is however also important to use the technology to facilitate change and to implement an (agreed) change management programme on the back of this, rather than seeking simply to replicate existing processes.

As part of its national strategy, England procured three commercial EHR systems to be implemented in hospitals. These included Cerner Millennium, Lorenzo Regional Care (henceforth referred to as Lorenzo), and RiO. Lorenzo (described in Table 1) was planned to be implemented across the North, Midlands and Eastern region of England and was intended to be used in 219 hospitals. Our study drew on a subset of data obtained as part of a national evaluation of EHRs in English hospitals [26–28]. However, the focus of this work was a more in-depth exploration of the consequences of Lorenzo on both individual and collective work practices and stakeholder relationships. Our rationale for focusing on this particular type of software was that the system was implemented in consecutive releases, gradually replacing paper systems, which meant that changes in work practices could be traced over time and in more detail than would have been possible with a more rapid roll-out.

Table 1 Description of the properties of the nationally procured type of software that was the focus of our investigation

This work was informed by socio-technical principles and drew on Actor-Network Theory (ANT) [29, 30], which helped us to investigate how the centrally procured EHR software has played an active role in shaping social relationships. This fluidity and interrelated nature of social and technical processes is often neglected in the technological determinism that typically dominates IT implementation strategies and views humans as passive recipients of technological innovation [31]. We therefore explored how the new technology has shaped professional practice and what consequences these changes had for organisational functioning, record keeping and patient care.

Methods

Design

We conducted a longitudinal comparative case study undertaking field work between February 2009 and November 2010. The majority of data were collected in three "early adopter" National Health Service (NHS) specialist care providers (henceforth referred to as 'sites') implementing early releases of Lorenzo in England. We conceptualised these as case studies to take into account local contingencies [29, 30, 32–35]. Our focus was on exploring the micro-environment of implementation and adoption, but without neglecting the wider implementation context (i.e. the national implementation environment), which was found to impact heavily on local developments [27, 28].

Sampling

We purposefully sampled sites from among the first few to implement Lorenzo software. Sampling criteria included the range of different settings (e.g. mental health, community, acute) and the timeframes of implementation (i.e. it had to begin during our study period).

Within each case study site, implementation team members were initially approached for interviews and observations through the local head/director of IT. We then used snowball sampling, asking individual participants to recommend further interviewees, actively seeking a range of different viewpoints. Individual users were initially recruited with the help of IT managers, ward managers and heads of service (again asking for recommendations of further interviewees). In doing so, we sampled a range of staff groups including both junior and senior nurses and doctors, allied health professionals and administrative staff.

Data collection

As work practices are heavily context dependent, we focused on collecting qualitative data through interviews, observations and documents in the three selected sites. These data collection approaches helped us to gain an insight into local contingencies, as well as more general processes across settings and over time.

We conducted semi-structured interviews with users (i.e. those who were expected to use the technology within their everyday work including doctors, nurses, allied health professionals and administrative staff) and other hospital staff (e.g. managers, IT staff and clinical leads) in order to gain an insight into their experiences of using the new system. Particular issues explored included views and attitudes relating to the system and implementation strategy (if relevant over time), perceived challenges and changes to individual and collaborative working, and potential suggestions for improvements (see Table 2 for an indicative topic guide). The wide range of interviewees sampled allowed us to gain an insight into the consequences for individuals as well as for wider organisational functioning, which is dependent on effective collaborative working. The first round of interviews (Time 1) was conducted during early use, when sites had implemented the technology for between three months and one year. Where possible, interviews were conducted longitudinally, speaking to the same stakeholders around six months later (Time 2) in order to trace potential changes in attitudes and investigate the effects of more embedded use of the system. This temporal component also helped us to assess the local consequences of the continuously changing political and economic climate. This was particularly important in the context of the national implementation, as changes in national arrangements often had significant consequences for hospital staff. These included, for example, delays in implementation timelines (or conversely pressures to implement), and changes in strategic direction such as reduced software functionality [27].

Table 2 Sample topic guide employed in interviews

In order to gain an insight into the more informal work practices of users, interviews were complemented by a mixture of object/activity/person-oriented observations of the software in use with on-site questioning, where appropriate and convenient. This involved "following the thing" (i.e. the software) and the various ways in which it was used and talked about in the clinical setting (e.g. by observing activities that involved the software) [36].

Hospital-specific documents, including project initiation documents, 'lessons learned' documents and local evaluation reports, were also collected in order to investigate the more formal (i.e. official or desired by management) changes to work practices. These included documents drafted before the system was implemented, outlining planned changes to work practices. We used these as a comparator to actual observed changes.

In order to examine concurrent developments in the wider political and economic landscape, we conducted additional semi-structured interviews with stakeholders outside sites. Interviewees here included governmental stakeholders, developers and representatives from the independent sector.

Data handling and analysis

Transcribed interviews, observations, researcher field notes and documents were coded with the help of NVivo8 software [37], drawing on both deductive and inductive coding approaches [38]. We began by building a coding framework based on the existing literature surrounding technology implementation and associated challenges with integration into user work practices [6]. This formed the deductive part of our analysis and included dimensions of integration of the system with professional responsibilities, existing roles and routines, and changes in individual and collaborative working. The inductive part of the analysis involved recording of new themes emerging from the data. Evolving categories from these themes were fed back into subsequent data collection. In investigating how exactly the software shaped both individual and collective work practices (i.e. the micro-context), drawing on ANT proved particularly useful [39, 40]. In doing so, our focus was on the active impact of the new technology on work practices of users, as it was beginning to replace established paper processes in the organisations. These insights were then used to draw conclusions about the wider environment in which these practices were situated by a constant process of "zooming in" and "zooming out" [39, 40]. This approach helped us to build on the literature and gain a holistic picture of the implementation and adoption landscape whilst still allowing new themes to emerge. As the focus of the study was on exploring changes to user work practices, this category was investigated in more detail first within and then across cases and data sources as well as time points [41]. Triangulation was facilitated by using the query function in NVivo8, which allowed extracting data from different data sources and collection times, and drawing up thematic tables recording emerging issues in each data source, time-point, group of interviewee, and individual location.

We discussed findings in regular team meetings and kept a research journal capturing our emerging understanding of the cases. During the analysis process, we paid particular attention to reflecting on our own assumptions and the way we had reached our interpretations, seeking potential alternative explanations. This was achieved by testing whether these better explained the data, ensuring that findings did not occur due to chance, searching for evidence that may refute explanations, and by following up unexpected findings [41].

Ethics

The observational component of this study received ethical approval from the East London and the City Research Ethics Committee on the 2nd of April 2009. The interview component submitted for ethical review to the same ethics committee but was classed as a service evaluation on the 9th of October 2008 (08/H0703/112). To protect the anonymity of participants and Trusts, we have removed potential identifiers wherever possible.

Results

We collected data through interviews with 66 users and other hospital staff, carried out 38.5 hours of observations, collected 13 hospital-specific documents, and conducted interviews with 14 stakeholders outside the immediate hospital environment. A summary of data collected in each case study site is provided in Table 3. This also shows a breakdown of the number of different participating professions.

Table 3 Summary of data collected at each hospital

As a result of our inductive analysis, we were able to identify a number of important factors relating to the integration of Lorenzo with user work practices. These are outlined in detail in Table 4 and included the following main categories:

  • Software characteristics and associated consequences

  • Coping strategies employed by software users in different contexts

  • Direct and indirect knock-on effects.

Table 4 Emerging themes from our study

We elaborate on these themes below with supporting illustrative data (further data are available on request from the corresponding author).

Software characteristics and associated consequences

The software in question was nationally procured, which meant that organisations, although to some extent free to choose which 'releases' (i.e. versions) they wished to implement, were presented with a fixed set of core releases that allowed limited customisability at either an organisational or group/individual user level. This reflected the then government's central desire to see interoperability. This was seen to be particularly important in the context of a state-run national health service. However, the focus on interoperability resulted in poor alignment between local work processes and software specifications. Users thus frequently expressed concern that the software lacked fitness for purpose and/or usability.

"Two fundamental criticisms remain that the system is not, and what you see on the screen is not intuitive, in other words if you haven't been taught, haven't used it regularly enough to remember what all the sequences are, so if you're only dipping into it occasionally it is actually very, very difficult because it is, you cannot sort of automatically think well this is what I want to do next and look at it and say that's what I do ... the other criticism of it is the speed of the system that you don't, when you expect to move from one field to another it is not instant and that is a big concern in a system where one feels instinctively that it ought to be." (Interview, Consultant, Time 2)

The limited ability to customise the software meant that it was difficult to adapt to suit local needs, which often resulted in significant increases in users' workloads.

"Somebody rang me, 5-10 minutes on the phone, and it takes you, one time it took me over 50 minutes to do it all because it's a phone call, going downstairs to get the file cause you still have to have the file, finding the file, bringing that up then logging on, waiting, cause I just did this one thing and then putting it all on, trying to print it and it took 50 minutes and that was just a parent had rang me to say that they needed to cancel the session (laughs)." (Interview, Therapist, Time 2)

The nature of Lorenzo also meant that it had to be implemented in phased releases as increasing functionality became available. Consequently, paper and electronic systems had to be run in parallel initially in order to compensate for the limited functionality of the early releases. Acknowledging that this temporary arrangement may have increased workload for end-users yet further, local implementation teams suggested possible workarounds for users to cope with these parallel systems. These, for example, included the:

"...printing of Lorenzo notes and attaching these to paper files, creating written notes if there are issues with electronic notes (i.e. paper used as fallback)". (Source: local Deployment Verification Report)

Coping strategies employed by software users in different contexts

We in addition observed a number of more unintended workarounds and coping strategies employed by users struggling to accommodate software that was perceived to be of poor usability. Some users, particularly those with more autonomy such as senior consultants, resisted use altogether by insisting on using paper records.

"... medical staff sort of dig their heels in and then don't do it, do they and if they can get out of doing it they'll do it on paper..." (Interview, Administrative Staff, Time 2).

Others, notably those who could not avoid using the system, devised various ways to compensate for the increasing demands on their time and concerns about the shortcomings of the technology. The most commonly employed techniques here included using other systems to compensate (e.g. typing letters in Microsoft Word as the spell-check was perceived to be much faster than that embedded in the software and/or reverting to paper systems); partially using the system (e.g. not recording certain activities if they were not viewed as important); and using the system in ways other than intended by management (e.g. getting around compulsory boxes by cross-referencing).

"So you end up, because you can't cut and paste so you can't say look OK this is the same let's cut and paste it into here, you end up having to write it multiple, multiple times or you end up having to cross-reference and I actually think it is safer for the [patient] to have all the risk stuff clearly and concisely written in one place. The nature of the form means that one place isn't the form so I actually write it very clearly in the progress note or the assessment form and I always in the boxes that open up I just put 'please refer to [name of form]', which then increases the amount that the risk indicator is a tick box exercise." (Interview, Nurse, Time 2)

Over time, some of these difficulties of integrating Lorenzo within users' everyday work practices attenuated to some extent as they became more familiar with the system. This was particularly true in smaller-scale deployments in sites that had invested significant time and resources to adapt the software to fit with their everyday practices. In addition to more effective integration with work practices as a consequence of increased familiarity, stakeholders reported at follow-up interviews that, as they received system upgrades, technical performance had improved significantly, particularly in relation to speed.

"I think the speed difference is massive from when we first started to now, you hardly wait at all... A lot has changed... I know the steps to take to find stuff so it's not a problem to find stuff." (Interview, Allied Healthcare Professional, Time 2)

In addition, some of the interim workarounds, such as printing forms completed on Lorenzo and attaching these to paper files, attenuated when the whole service used Lorenzo. Over time, paper appeared to progressively lose its significance, increasingly being used as a "back-up system", with users employing compensating techniques such as, for example, performing another activity whilst waiting for the system (such as switching on the computer and making a cup of coffee); inputting less descriptive data as this helped to speed up data entry; or allocating extra time at the end of consultations for correcting spelling mistakes made whilst typing.

Direct and indirect knock-on effects

Coping strategies employed by users had several direct and indirect knock-on effects. These are considered below in relation to collaborative working, patients, paper records, managerial outputs and recording activity.

Collaborative working-hierarchical structures and communication

The new technology impacted on the ways in which the healthcare team interacted. This was in some instances seen as a positive consequence, but in other situations as a negative development. In relation to the former, some users felt that using the system helped to make communication more effective over longer distances.

"Yeah it is [more effective] because now we know that definitely everybody has access so things like, so last week we had a really urgent nail surgery on the Wednesday and I could actually book her in to have it on the Thursday knowing safely that her assessment was all there and I didn't have to rush off a set of notes and everything else it was all done. And that's only minor benefits for us but everyday where we use it more and more now we're paper free we just think of more things." (Interview, Allied Healthcare Professional, Time 2)

On the other hand, users also reported that the system changed the way the healthcare team interacted in negative ways. This was mainly expressed in relation to changing professional roles and responsibilities with an increased emphasis on administrative tasks. It was seen as particularly problematic by clinical staff who expressed concern that the resulting displacement of administrative duties on to their shoulders was detracting from their more pressing clinical responsibilities.

"It takes you much, much, much, much, much longer; it doesn't really help us at the moment. This version is absolutely useless to be honest and yeah, it's a waste of time, you're sitting in front of a computer and you should see basically patients and doing something and so instead of this you're typing in something, and you're kind of frustrated when you wait for something that took half a minute as a chest X-ray, it takes now at least 10 minutes..." (Interview, Junior Doctor, Time 1).

Similarly, the system tended to change professional responsibilities, often making existing hierarchies more visible. For example, it only allowed nurses who had special training to order X-rays. Other nurses had to ask the doctors to order these if necessary. In such cases, doctors then had to complete the form, which created more (unwelcome) work for them. Conversely, nurses felt that some of their professional autonomy had been eroded (although it has to be noted that officially most nurses were not allowed to order X-rays-they were doing it informally by completing the cards on behalf of a doctor).

"... for example you get a load of new doctors on the ward and you get a patient in with a specific condition you know automatically before the consultant even comes round what investigations that patient is to have, because there's a protocol so you know what they're going to have or you know, because you've experienced what they're going to have so you could in effect order them and we did do, you know, if someone came in with renal colic I would order, you would just do it because that's what you knew the patient was going to have." (Interview, Nurse, Time 1)

Time with patients

The introduction of the new system was perceived to reduce the time healthcare professionals could spend with patients, leaving clinical staff frustrated, as direct patient care was seen as more central to their role and professional identity. The increased time spent in front of the computer thus adversely impacted on job satisfaction.

"I mean the fact that there's no jobs in the NHS at the moment is the only reason why people would have stayed and morale has been, people are just not feeling job satisfaction because as I say when you should be seeing patients you're actually sitting at a screen that is going interminably slow." (Interview, Therapist, Time 2)

In some instances, the technology also reduced the perceived quality of the interaction with patients. Using computers whilst consulting was, for example, felt to impact on communication flow, rendering the consultation more formal and less engaging.

"It means that it's almost like a barrier to communication because you're having to take your eyes off that patient and break that communication to look down at a laptop or a computer and I don't think that's terribly professional..." (Interview, Allied Healthcare Professional, Time 1).

Other systems such as paper

It also became apparent that the introduction of the new system not only impacted on individuals, but also had an effect on paper records (which were, as mentioned above, still used in parallel). Here, paper was often found to be more distributed across geographical locations within healthcare settings. For example, different users would take paper files to their desks at different times to file the electronic print-outs whereas before the introduction of the system all paper records were held centrally in the reception area.

The paper comes out of a printer in his office but he [Therapist] explains that this printer is also shared, he signs the paper and explains that he "now have to find the paper file to put the paper copy into it", he says that sometimes he cannot find the paper file as other clinicians might have it... (Researcher Notes, Observation).

As a result, other users needed increasingly to "chase" paper records.

"Oh yes, I mean I've had one where I've said "I'm sure someone has seen this case" and of course it's just not on the file cause it's still with other people. And when it's things like medication and things like that, sometimes I'll just come in and I've not been in for a week and I just need to pick up the file and there is nothing there and I know she's seen them but it's not typed in which never happens with notes cause they just sit there and write it while the client is there. So it's very time consuming." (Interview, Therapist, Time 2)

The medical record itself

Similarly, data entry itself was impacted upon, this in the main manifesting as delays in transcribing the record with the introduction of the new system. This was due to the software being perceived as slow and as impacting on the communication flow with the patient. Notes were therefore often typed up at the end of the day or, in some instances, days after the consultation had taken place. Paper notes were used in the meantime as reminders.

"Well at the moment because we've sort of piloted it on the [name] wards we have tried to make an effort to use [name of system] as much as possible. I mean we were asked to try and request investigations live on the ward rounds using a portable computer but unfortunately as I said because of constraints of time it wasn't possible to use it that way, it just took too long because we, you know, we have a fair number of patients to see in a short space of time. And so the junior staff are making notes of who needs which investigation and then they're requesting them via [name of system] at the end of the ward round." (Interview, Consultant, Time 1)

As a result of this delayed data entry, the new computer system was found to be less up to-date than systems that were previously in place.

He [nurse] then puts the Smartcard [an electronic card used for user identification] into the keyboard and picks up a folder of patient notes on his desk, on top of the folder is a small notepad page with some scribbles on it, he then looks at his paper diary (like a book and full of scribbles) and looks up when he saw the patient, "ah 2 pm on Friday" (it's Monday today), he says that he jots down notes on paper "to jog my memory", it is a girl with an eating disorder... (Researcher Notes, Observation).

Managerial reports

Managerial outputs were also affected by the introduction of the new electronic system. When staff employed workarounds, this would in some cases result in inaccurate reports being generated by the system further down the line, impacting on managers' ability to track activity levels.

"... so you're getting a certain amount of well I'll just try anything that makes it look like its worked and then you get problems with back-end data because there's a diversification of the numbers of processes that people are using cause they are just desperate to try and get from A to B in a day and they'll try any route to get there that looks like it's working to them." (Interview, Manager, Time 2)

An interesting example given in this context was that users in outpatients would book appointments on the system using a route that was less laborious, but which meant that, although these appointments would show on their screens as booked, they would not show on the rest of the system as booked. As a result, managerial outputs became so unpredictable that they in a number of instances no longer reflected the reality of what actually happened.

Conceptualising overall consequences for work practices

ANT helped us to conceptualise changes in constellations between various network components including humans, paper and the technology. This is illustrated in Figure 1, which depicts existing work practices before the introduction of Lorenzo, planned changes in work practices with the introduction of Lorenzo, and actual changes in work practices resulting from the introduction of Lorenzo.

Figure 1
figure 1

An ANT-based diagrammatic presentation of the changed networks.

Discussion

Summary of main findings

The decision to nationally procure EHR systems, coupled with the deliberate decision to limit customisability, resulted in significant changes to user work practices. Users had to adapt to a system that they did not believe was fit for purpose. If they could not resist use, they tended to employ workarounds which were often unanticipated by management. These had several knock-on effects on other factors within the hospital, including hierarchical structures and patterns of communication within the multi-disciplinary healthcare team, the time and quality of interactions with patients, paper records (which became more distributed), the timeliness of the record itself, and managerial outputs (which became less predictable).

Strengths and limitations of this work

Drawing on a range of data sources has allowed us to triangulate our findings and gain an insight into both formal and informal work practices associated with both intended and unintended workarounds [42]. ANT proved useful in guiding our data collection by placing the technology at the heart of the investigation, helping to appreciate the fluidity of sociotechnical processes, and the active role of the technology on other factors around it, both human and inanimate. It however proved less useful in explaining changes in the technology resulting from the way it was received by users and organisations. Other theoretical lenses may help to address this issue, such as for example the Strong Structuration Theory and the Social Shaping of Technology [40, 43], which emphasise the mutually shaping relationship between structures and agents.

Our longitudinal design meant that we were able to explore certain changes and developments over time, but this was still, however, limited to early implementation and adoption. We therefore observed certain ways in which users accommodated the system over time, but despite spending many months in the field we did not have the opportunity to assess the more embedded use of the software. Given our limited funding, we were also unable to develop a detailed understanding of the way healthcare workers used paper systems by examining work practices before the introduction of the new system, although this has often been advocated as an important first step towards understanding how electronic systems are likely to affect individual and collaborative work [42, 44–46].

In addition, it has to be kept in mind that: 1) work practices are not static, but constantly changing/adapting (which complicates planning) [47]; and 2) work processes are contingent upon situations [48]. It may be, therefore, that over time and in different contexts users may have developed more (or indeed less) effective ways of using the technology. Consequently, despite our results being likely to be transferable to similar settings (i.e. sites implementing early releases of the same software), they may not apply to all care activities and situations.

Considering our findings in relation to the wider literature

Our results have shed light on the complex consequences and knock-on effects of a nationally procured EHR system for the work practices of a range of users. We illustrate these diagrammatically in Figure 2.

Figure 2
figure 2

Diagrammatic presentation of the desirable and undesirable consequences of Lorenzo for user work practices.

Usability of the system was perceived to be lacking, as the design of the "one-size-fits-all" technology limited tailoring to user needs. It is commonly acknowledged that increased customisation can facilitate the introduction of technology within user work practices [49–53], so the finding that the relatively inflexible nature of Lorenzo resulted in unintended workarounds is probably not surprising [21–23, 54].

Users needed to change their work processes to fit in with the technology; and this could be expected with the introduction of a new system as new routines need to be developed and existing work practices inevitably change to some extent [14]. However, issues with poor system usability resulted in clinicians employing informal workarounds to accommodate a system that was viewed as lacking fitness for purpose and increasing workloads and stress levels, as well as taking time away from patient care. When examining the literature on 'workarounds', most of the responses users in our study demonstrated can be thought of as "essential hindrance workarounds". That is, they were used to get around perceived problems in systems that were seen to impact on the more direct task of care provision. Most of the workarounds we identified were also viewed as essential as they were designed to save time that was needed for adequate patient care [22]. However, paradoxically these workarounds, designed to reduce the workload of users, also made the use of the system very specific to users' own contexts, thereby hampering the possibility of meaningful communication between interoperable systems being used in different contexts [55].

We have outlined several knock-on effects of these workarounds. Changes to existing professional responsibilities, hierarchical structures and ways of collaborative working of the multi-disciplinary healthcare team have been found in previous studies investigating the impact of new IT systems in healthcare settings [16, 56–59]. Similarly, the finding that new technology can adversely impact on the interactions with patients is reinforcing existing evidence [60]. However, the effects on other factors outside of the immediate environment of use (e.g. organisations and paper records) are indeed new findings, possibly because they are context dependent. These included paper (which became more distributed), managerial outputs (which were perceived to become less accurate), and the timeliness of the record (as data entry was reported as being increasingly delayed). Although previous studies have investigated the potential impact of 'workarounds' on the safety and quality of care, the mechanisms by which effects are observed are likely to vary across settings and may not be transferable to other IT applications [24, 61].

Our findings are worrying, considering the potential knock-on implications for the quality and safety of patient care (e.g. if an important record is not available when needed). It is therefore vital that these wider factors are considered when mapping anticipated changes to work practices, which needs to be routinely and regularly done by organisations preparing for changes associated with IT applications in healthcare [62, 63].

Considering the range of staff and contexts of IT use in healthcare settings, mapping of work practices needs to be local by definition. Likewise, emerging 'workarounds' are likely to reflect the idiosyncrasies of the individual setting and are therefore highly context-dependent. This is in itself at odds with a national implementation with limited system customisability. Here, work practices may be effectively mapped, but the reality of systems-in-use is likely to vary from planned patterns as informal and emergent work practices that characterise the dynamic and ever-changing hospital environment are often not taken into account [42, 64]. This, in turn, may result in a lack of attention being paid to consequences for other stakeholders. The national nature of the implementation means that the resulting mitigating actions are far too slow to be visible in a timely manner to users on the ground, limiting the potential for organisations to flexibly address unanticipated issues [61]. This is supported by the relative successes of small-scale EHR implementations characterised by extensive customisation of systems to suit local needs [65].

In relation to England's national IT strategy, a more participatory approach to development and implementation has repeatedly been advocated [6, 66, 67]. For example, Catwell and Sheikh have suggested that this should be characterised by early user involvement and development of a shared vision, formative evaluations and testing of prototypes to assess whether the system is perceived as usable, potential re-design so that it fits with users' needs, summative evaluation and benefits identification once implementation completed, and incorporation of issues identified along the way [67].

Engagement and user input in design is clearly important [6, 21, 66, 68–71], but in line with our findings, it is likely to be difficult to realise in the context of a centrally procured system that is intended for implementation on a national scale. This was found to be the case in relation to Lorenzo, which was intended to be "co-creational" by involving users in system design. However, the result of the focus on interoperability was limited local input and a resulting application that did in many instances not fulfill user and organisational needs. In addition, the organic development of engagement was hampered from the start as organisations and users were mandated to implement and use Lorenzo. Our experiences and the wider evidence-base suggest that a potential way forward may involve implementing a combination of standardised core solutions and customised elements, corresponding to particular users and environments (also known as 'configurational technology') [72, 73].

Implications for policy, practice and research

Our findings provide important insights relating to the impact of nationally procured systems on both formal and informal work practices of users and collaborative working. There is clearly a need to understand better how coordination is achieved without IT in order to produce a system that has the ability to facilitate this coordination or indeed revolutionise it by making processes more effective [42]. This is likely to vary across different settings and will warrant detailed small-scale local studies over longer periods of time in order to trace how work is structured with paper systems, as well as the initial, continuing and more embedded use of new EHR systems. As part of this work, it may be helpful to explore these issues from a range of theoretical angles. These may include deliberations surrounding the classification of an EHR system as a medical device and associated considerations surrounding usability testing including assessments of potentially negative consequences; more specific discussion surrounding effective leadership in organisations that have a high degree of autonomy; and exploration of ways in which evidence-based practice can be effectively promoted and integrated within individual workflows and within political strategies.

With regards to the impact on the quality and safety of care, there is a need to assess outcomes identified in our study quantitatively, in order to determine whether the mechanisms identified are indeed contributing to adverse patient outcomes. We do, however, acknowledge that such studies will be challenging to design as outcome measures may not necessarily be traceable to the system and comparators may be difficult to identify [74, 75].

The implications for practice emerging from our research are two-fold: firstly, there is a need to allow local work process mapping and responses to emergent practices and perceived inadequacies in the technological design; and secondly, it is vital not only to assess workarounds employed by local users, but the reasons behind their adoption in order to be in a position to reflect on other downstream consequences.

Conclusions

We have identified a number of important consequences resulting from key procurement, design and implementation decision in relation to plans to implement a national EHR software system in England. These consequences were observed in relation to healthcare staff, organisations, medical records and patients.

We have furthermore highlighted that assessments of workarounds and their consequences need to be locally undertaken and addressed. In the context of the particular implementation investigated in our work, this was inhibited by the focus on interoperability at the expense of customisability in national efforts. The question surrounding potential trade-offs between large-scale interoperability and local requirements is likely to be the subject of continuous debate in England and beyond with no easy answers in sight. Based on the findings from this study, we suggest that concerns relating to interoperability should not be prioritised over-and-above ensuring that essential local needs are met. This is because a failure to address these local needs not only risks the viability of the entire implementation effort, but it may also severely hinder successful adoption which may result in important adverse local consequences (as we have shown). In addition, unintended use of local systems may also impact on interoperability considerations in relation to sharing of data between different healthcare providers. This may be the case if data held within local systems does not accurately reflect reality, potentially resulting in inaccurate information being drawn on by other providers.

Informal work practices, and knock-on effects are expected to vary, but assessing and tracing changes and effects over time is likely to pay significant returns for the implementing organisation, users and the quality of patient care.

References

  1. Massaro TA: Introducing physician order entry at a major academic medical center: 1. Impact on organizational culture and behaviour. Acad Med. 1993, 68: 20-25. 10.1097/00001888-199301000-00003.

    Article  CAS  PubMed  Google Scholar 

  2. Sicotte C, Denis JL, Lehoux P: The computer based patient record: a strategic issue in process innovation. J Med Syst. 1998, 22: 431-443. 10.1023/A:1020674527409.

    Article  CAS  PubMed  Google Scholar 

  3. Southon G, Sauer C, Dampney K: Lessons from a failed information systems initiative: issues for complex organisations. Int J Med Inform. 1999, 55: 33-46. 10.1016/S1386-5056(99)00018-0.

    Article  CAS  PubMed  Google Scholar 

  4. Black A, Car J, Pagliari C, Anandan C, Cresswell K: The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med. 2011, 8: e1000387-10.1371/journal.pmed.1000387. doi:10.1371/journal.pmed.1000387

    Article  PubMed  PubMed Central  Google Scholar 

  5. Blumenthal D, Tavenner M: The meaningful use regulation for electronic health records. N Engl J Med. 2010, 363: 501-504. 10.1056/NEJMp1006114.

    Article  CAS  PubMed  Google Scholar 

  6. Cresswell K, Sheikh A: The NHS Care Record Service (NHS CRS): recommendations from the literature on successful implementation and adoption. Inform Prim Care. 2009, 17: 153-160.

    PubMed  Google Scholar 

  7. Beuscart-Zephir MC, Anceaux F, Crinquette V, Renard JM: Integrating users' activity modeling in the design and assessment of hospital electronic patient records: The example of anesthesia. Int J Med Inform. 2001, 64: 157-171. 10.1016/S1386-5056(01)00210-6.

    Article  CAS  PubMed  Google Scholar 

  8. Giuse DA, Kuhn KA: Health information systems challenges: The Heidelberg conference and the future. Int J Med Inform. 2003, 69: 105-114. 10.1016/S1386-5056(02)00182-X.

    Article  PubMed  Google Scholar 

  9. Halamka J, Aranow M, Ascenzo C, Bates DW, Berry K, Debor G: E-Prescribing Collaboration in Massachusetts: early experiences from regional prescribing projects. J Am Med Inform Assoc. 2006, 13: 239-244. 10.1197/jamia.M2028.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Lium JT, Tjora A, Faxvaag A: No paper, but the same routines: A qualitative exploration of experiences in two Norwegian hospitals deprived of the paper based medical record. BMC Med Inform Decis Mak. 2008, 8: 2-10.1186/1472-6947-8-2.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Ludwick DA, Doucette J: Adopting electronic medical records in primary care: Lessons learned from health information systems implementation experience in seven countries. Int J Med Inform. 2009, 78: 22-31. 10.1016/j.ijmedinf.2008.06.005.

    Article  CAS  PubMed  Google Scholar 

  12. Moen A: A nursing perspective to design and implementation of electronic patient record systems. J Biomed Inform. 2003, 36: 375-378. 10.1016/j.jbi.2003.09.019.

    Article  PubMed  Google Scholar 

  13. Pare G, Sicotte C, Jaana M, Girouard D: Prioritizing the risk factors influencing the success of clinical information system projects. A Delphi study in Canada. Methods Inf Med. 2008, 47: 251-259.

    CAS  PubMed  Google Scholar 

  14. Puffer MJ, Ferguson JA, Wright BC, Osborn J, Anshus AL, Cahill BP: Partnering with clinical providers to enhance the efficiency of an EMR. J Healthc Inf Manag. 2007, 21: 24-32.

    PubMed  Google Scholar 

  15. Rose J, Jones M, Truex D: The problem of agency re-visited. Scand J Inf Syst. 2005, 17: 133-152.

    Google Scholar 

  16. Scott JT, Rundall TG, Vogt TM, Hsu J: Kaiser Permanente's experience of implementing an electronic medical record: A qualitative study. Br Med J. 2005, 331: 1313-1316. 10.1136/bmj.38638.497477.68.

    Article  Google Scholar 

  17. Sicotte C, Pare G, Moreault M-P, Paccioni A: A risk assessment of two interorganizational clinical information systems. J Am Med Inform Assoc. 2006, 13: 557-566. 10.1197/jamia.M2012.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Sprague L: Electronic health records: How close? How far to go?. NHPF Issue Brief. 2004, 800: 1-17.

    PubMed  Google Scholar 

  19. Greenhalgh T, Potts H, Wong G, Bark P, Swinglehurst D: Tensions and paradoxes in electronic patient record research: A systematic literature review using the meta-narrative method. Milbank Q. 2009, 87: 729-788. 10.1111/j.1468-0009.2009.00578.x.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Schiff GD, Bates DW: Can electronic clinical documentation help prevent diagnostic errors?. N Engl J Med. 2010, 362: 1066-1069. 10.1056/NEJMp0911734.

    Article  CAS  PubMed  Google Scholar 

  21. Eason K: Local sociotechnical system development in the NHS National Programme for Information Technology. J Inf Technol. 2007, 22: 257-264. 10.1057/palgrave.jit.2000101.

    Article  Google Scholar 

  22. Ferneley E, Sobreperez P: Resist, comply or workaround? An examination of different facets of user engagement with information systems. Eur J Inf Syst. 2006, 15: 345-356. 10.1057/palgrave.ejis.3000629.

    Article  Google Scholar 

  23. Bjørn P, Burgoyne S, Crompton V, MacDonald T, Pickering B, Munro S: Boundary factors and contextual contingencies: configuring electronic templates for healthcare professionals. Eur J Inf Syst. 2009, 18: 428-441. 10.1057/ejis.2009.34.

    Article  Google Scholar 

  24. Koppel R, Wetterneck T, Telles JL, Karsh BT: Workarounds to barcode medication administration systems: Their occurrences, causes, and threats to patient safety. J Am Med Inform Assoc. 2008, 15: 408-423. 10.1197/jamia.M2616.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Sobreperez P: Technological frame incongruence, diffusion, and noncomplianc. Open IT-Based Innovation: Moving Towards Cooperative IT Transfer and Knowledge Diffusion. 2008, Springer Boston, 179-196.

    Chapter  Google Scholar 

  26. Cresswell K, Ali M, Avery A, Barber N, Cornford T, Crowe S: The long and winding road... an independent evaluation of the implementation and adoption of the National Health Service Care Records Service (NHS CRS) in secondary care in England. 2011, Available from: http://www.chs.med.ed.ac.uk/grantdocs/526%20-%20Final%20report%20v31st%20Mar%20FINAL.pdf (Last accessed: 28/11/2011)

    Google Scholar 

  27. Robertson A, Cresswell K, Takian A, Petrakaki D, Crowe S, Cornford T: Implementation and adoption of nationwide electronic health records in secondary care in England: qualitative analysis of interim results from a prospective national evaluation. Br Med J. 2010, 341: c4564-10.1136/bmj.c4564.

    Article  Google Scholar 

  28. Sheikh A, Cornford T, Barber N, Avery A, Takian A, Lichtner V: Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from a prospective national evaluation in "early adopter" hospitals (in press). Br Med J. 2011, 343: d6054-10.1136/bmj.d6054.

    Article  Google Scholar 

  29. Cresswell K, Worth A, Sheikh A: Actor-Network Theory and its role in understanding the implementation of information technology developments in healthcare. BMC Med Inform Decis Mak. 2010, 10: 67-10.1186/1472-6947-10-67.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Latour B: Reassembling the social: an introduction to Actor-Network Theory. 2005, Oxford: Oxford University Press

    Google Scholar 

  31. Smith M, Marx L: Does technology drive history? The dilemma of technological determinism. 1994, Cambridge: MIT Press

    Google Scholar 

  32. George A, Bennett A: Case studies and theory development in the social sciences. 2005, Massachusetts: MIT Press Cambridge

    Google Scholar 

  33. Miles M, Huberman A: Qualitative data analysis. 1984, London: Sage

    Google Scholar 

  34. Stake RE: The art of case study research. 1995, London: SAGE Publications

    Google Scholar 

  35. Yin R: Case study research, design and methods. 2009, Sage Publications

    Google Scholar 

  36. Bruni A: Shadowing software and clinical records: on the ethnography of non-humans and heterogeneous contexts. Organization. 2005, 12: 357-378. 10.1177/1350508405051272.

    Article  Google Scholar 

  37. 2008, QSR International Pty Ltd, NVivo qualitative data analysis software Version 9

  38. Mason J: Qualitative researching. 2002, London: Sage

    Google Scholar 

  39. McLean C, Hassard J: Symmetrical absence/symmetrical absurdity: critical notes on the production of Actor Network Theory. J Manag Stud. 2004, 41: 493-519. 10.1111/j.1467-6486.2004.00442.x.

    Article  Google Scholar 

  40. Williams R, Edge D: The social shaping of technology. Res Policy. 1996, 25: 865-899. 10.1016/0048-7333(96)00885-2.

    Article  Google Scholar 

  41. Miles MB, Huberman AM: Qualitative data analysis. 1994, Thousand Oaks: Sage

    Google Scholar 

  42. Symon G, Long K, Ellis J: The coordination of work activities: cooperation and conflict in a hospital context. Computer Supported Cooperative Work. 1996, 5: 1-31.

    Article  Google Scholar 

  43. Greenhalgh T, Stones R: Theorising big IT programmes in healthcare: Strong structuration theory meets actor-network theory. Soc Sci Med. 2010, 70: 1285-1294. 10.1016/j.socscimed.2009.12.034.

    Article  PubMed  Google Scholar 

  44. Østerlund C: Mapping medical work: documenting practices across multiple medical settings. J Center Inf Stud (Japan). 2004, 3: 35-43.

    Google Scholar 

  45. Osterlund K: Combining genres: how practice matters. Proceedings of the 39th Annual Hawaii International Conference. System Sciences, 2006. HICSS'06

  46. Østerlund C: Two Doctors' Documenting Practices: How the Indexical Centering of Medical Records Integrates the Encoding, Communication and Coordination of Patient Care. Paper presented at the second annual meeting of the Document Academy (DOCAM'04) at the School of Information Management and Systems, University of California, October 23, Berkeley. 2004

    Google Scholar 

  47. Weick KE, Quinn RE: Organizational change and development. Annu Rev Psychol. 1999, 50: 361-386. 10.1146/annurev.psych.50.1.361.

    Article  CAS  PubMed  Google Scholar 

  48. Bardram JE, Bossen C: A web of coordinative artifacts: collaborative work at a hospital ward. GROUP'05: Proceedings of the. 2005, New York, 168-176. international ACM SIGGROUP conference on supporting group work

    Chapter  Google Scholar 

  49. Rose AF, Schnipper JL, Park ER, Poon EG, Li Q, Middleton B: Using qualitative studies to improve the usability of an EMR. J Biomed Inform. 2005, 38: 51-60. 10.1016/j.jbi.2004.11.006.

    Article  PubMed  Google Scholar 

  50. Doebbeling BN, Chou AF, Tierney WM: Priorities and strategies for the implementation of integrated informatics and communications technology to improve evidence-based practice. J Gen Intern Med. 2006, 21: S50-S57. 10.1007/s11606-006-0275-9.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Ash JS, Fournier L, Stavri PZ, Dykstra R: Principles for a successful computerized physician order entry implementation. AMIA Annual Symposium Proceedings. 2003, 36-40.

    Google Scholar 

  52. Aarts J, Doorewaard H, Berg M: Understanding implementation: The case of a computerized physician order entry system in a large dutch university medical center. J Am Med Inform Assoc. 2004, 11: 207-216. 10.1197/jamia.M1372.

    Article  PubMed  PubMed Central  Google Scholar 

  53. Miranda D, Fields W, Lund K: Lessons learned during 15 years of clinical information system experience. Comput Nurs. 2001, 19: 147-151.

    CAS  PubMed  Google Scholar 

  54. Ash JS, Sittig DF, Dykstra RH, Guappone K, Carpenter JD, Seshadri V: Categorizing the unintended sociotechnical consequences of computerized provider order entry. Int J Med Inform. 2007, 76: S21-S27.

    Article  PubMed  Google Scholar 

  55. Pirnejad H, Bal R, Berg M: Building an inter-organizational communication network and challenges for preserving interoperability. Int J Med Inform. 2008, 24 (77): 818-27.

    Article  Google Scholar 

  56. Morrison C, Jones M, Blackwell A, Vuylsteke A: Electronic patient record use during ward rounds: a qualitative study of interaction between medical staff. Crit Care. 2008, 12: R148-10.1186/cc7134.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Yarbrough AK, Smith TB: Technology acceptance among physicians: a new take on TAM. Med Care Res Rev. 2007, 64: 650-672. 10.1177/1077558707305942.

    Article  PubMed  Google Scholar 

  58. Braithwaite J, Runciman WB, Merry AF: Towards safer, better healthcare: harnessing the natural properties of complex sociotechnical systems. Qual Saf Health Care. 2009, 18: 37-41. 10.1136/qshc.2007.023317.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  59. Eriksson-Zetterquist U, Lindberg K, Styhre A: When the good times are over: professionals encountering new technology. Hum Relat. 2009, 62: 1145-1170. 10.1177/0018726709334879.

    Article  Google Scholar 

  60. May C, Gask L, Atkinson T, Ellis N, Mair F, Esmail A: Resisting and promoting new technologies in clinical practice: the case of telepsychiatry. Soc Sci Med. 2001, 52: 1889-1901. 10.1016/S0277-9536(00)00305-1.

    Article  CAS  PubMed  Google Scholar 

  61. Spear SJ, Schmidhofer M: Ambiguity and workarounds as contributors to medical error. Ann Intern Med. 2005, 142: 627-630.

    Article  PubMed  Google Scholar 

  62. Davidson SM, Heineke J: Toward an effective strategy for the diffusion and use of clinical information systems. J Am Med Inform Assoc. 2007, 14: 361-367. 10.1197/jamia.M2254.

    Article  PubMed  PubMed Central  Google Scholar 

  63. Kelly CS: Perinatal computerized patient record and archiving systems: pitfalls and enhancements for implementing a successful computerized medical record. J Perinat Neonatal Nurs. 1999, 12: 1-14.

    Article  CAS  PubMed  Google Scholar 

  64. Cooper S: Technoculture and critical theory: in the service of the machine?. 2002, London: Routledge

    Book  Google Scholar 

  65. Permanente Kaiser: Connected for Health: using electronic health records to transform care delivery. 2010, San Francisco: Jossey-Bass

    Google Scholar 

  66. Brennan S: The biggest computer programme in the world ever! How's it going?. J Inf Technol. 2007, 22: 202-211. 10.1057/palgrave.jit.2000104.

    Article  Google Scholar 

  67. Catwell L, Sheikh A: Information technology (IT) system users must be allowed to decide on the future direction of major national IT initiatives. But the task of redistributing power equally amongst stakeholders will not be an easy one. Inform Prim Care. 2009, 17: 1-4.

    PubMed  Google Scholar 

  68. Mair FS, May C, Finch T, Murray E, Anderson G, Sullivan F: Understanding the implementation and integration of e-health services. J Telemed Telecare. 2007, 13: 36-37.

    Article  Google Scholar 

  69. Webster A: Innovative health technologies and the social: redefining health, medicine and the body. Curr Sociol. 2002, 50: 443-457. 10.1177/0011392102050003009.

    Article  Google Scholar 

  70. Patel N, Dittrich V: Deferred system's design: Developing context-aware information systems for dynamic environments. 2002, Gdansk, Poland: European Conference on Information Systems

    Google Scholar 

  71. Jakobs K, Procter R, Williams R: Standardisation and implementation of information technology. Proceedings of the International Resource Management Association. 2001

    Google Scholar 

  72. Stewart J, Williams R: The wrong trousers? Beyond the design fallacy: Social learning and the use. User involvement in innovation processes. Strategies and limitations from a socio-technical perspective. Edited by: Rohracher H. 2005, Munich: Profil-Verlag

    Google Scholar 

  73. Greenhalgh T, Macfarlane F, Maskrey N: Getting a better grip on research: the organizational dimension. InnovAiT. 2010, 3: 102-107. 10.1093/innovait/inp120.

    Google Scholar 

  74. Pagliari C: Implementing the National Programme for IT: what can we learn from the Scottish experience?. Inform Prim Care. 2005, 13: 105-111.

    PubMed  Google Scholar 

  75. Smaltz DH, Callander R, Turner M, Kennamer G, Wurtz H, Bowen A: Making sausage-effective management of enterprise-wide clinical IT projects. J Healthc Inf Manag. 2005, 19: 48-55.

    PubMed  Google Scholar 

Pre-publication history

Download references

Acknowledgements

We are very grateful to the participating sites for supporting this work, to all interviewees who kindly gave their time, and the helpful suggestions of two expert peer reviewers. AS and KC are currently funded to undertake an evaluation of the NHS Care Record Service (NHS CFHEP 005), data coding and structuring (NHS CFHEP 009) and the impact of IT on the clinician-patient relationship (NHS CFHEP 010). We gratefully acknowledge the contribution of our colleagues working on these evaluations. KC is supported by a studentship from the Medical Research Council. We acknowledge the support of the National Institute for Health Research, through the Comprehensive Clinical Research Network.

This work is independent research commissioned by the NHS Connecting for Health Evaluation Programme. The views expressed in this publication are those of the authors and not necessarily those of the NHS, the NHS Connecting for Health Evaluation Programme or the Department of Health.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Kathrin M Cresswell.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

KC is guarantor. AS and AW contributed to interpreting the evidence and drafting the manuscript. KC collected and analysed the data, and led the drafting of this paper. All authors read and approved the final manuscript.

Kathrin M Cresswell, Allison Worth and Aziz Sheikh contributed equally to this work.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Cresswell, K.M., Worth, A. & Sheikh, A. Integration of a nationally procured electronic health record system into user work practices. BMC Med Inform Decis Mak 12, 15 (2012). https://doi.org/10.1186/1472-6947-12-15

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/1472-6947-12-15

Keywords