-
PDF
- Split View
-
Views
-
Cite
Cite
Mary Patricia Tully, Judith A. Cantrill, The validity of explicit indicators of prescribing appropriateness, International Journal for Quality in Health Care, Volume 18, Issue 2, April 2006, Pages 87–94, https://doi.org/10.1093/intqhc/mzi084
Close - Share Icon Share
Abstract
Objective. To assess, from the perspective of UK hospital doctors, the content validity and operational validity of a set of 14 previously developed explicit indicators of the appropriateness of long-term prescribing started during a hospital admission.
Method. A combination of data extraction from medical records and qualitative interviews with a maximum variability sample of hospital doctors.
Participants. The indicators were applied to 132 new prescriptions, intended for long-term use, prescribed for 61 patients; 36 doctors, of various grades, were purposively selected for interview.
Results. Appropriate prescribing was viewed as prescribing that was indicated, necessary, evidence based (using a broad meaning of ‘evidence’) and of acceptable cost and risk-benefit ratio. These concepts applied to individual drugs for individual patients, rather than at a more general, public health level. Where drugs had failed an indicator, rationales were explored. Often, it was missing data in the medical notes that had resulted in the drug failing the indicator.
Conclusions. The 14 indicators were considered to have content validity, reflecting all aspects of appropriate prescribing discussed by the doctors. Their operational validity was less clear-cut, due to the lack of necessary data in the medical notes. This has implications for the use of explicit indicators for assessing prescribing appropriateness, as these hospital doctors did not consider that the data required for objective, systematic assessment of prescribing would ever be recorded in hospital medical notes.
The evaluation and assessment of health care quality are receiving worldwide attention [1–3]. The spiralling cost of drugs, with finite resources and frequent medication errors [4], has drawn attention to prescribing quality and appropriateness. Appropriate prescribing has been defined as ‘the outcome of the process of decision-making that maximises net individual health gains within the society’s available resources’ [5].
The commonest methods for assessing appropriate or inappropriate prescribing are explicit criteria and implicit professional judgement. The former uses a consensus panel to prepare a set of standards from the literature [6,7]. They are dependent not only upon available evidence but also upon the authors’ ability to routinely include new evidence. Implicit judgement uses health care professionals’ clinical acumen to assess the appropriate prescribing for individual patients [8,9]. Consequently, implicit judgements are subject to reviewer variations. However, implicit judgements take account of the context of individual patients and easily incorporate new evidence into the assessment. They may have greater validity but poorer reliability than explicit criteria [5].
Combining these assessment types has the potential to utilize the advantages of both. Explicit indicators require each prescription to be compared with a set of published standards (e.g. the Physicians’ Desk Reference [10] or the British National Formulary [11]), but within the context of the individual patient. Such standards are used for the Medication Appropriateness Index [12] and the prescribing appropriateness indicators [13], which have been validated for ambulatory care in the US and primary care in the UK. However, there has been no similar work using explicit indicators to evaluate prescribing appropriateness in secondary care in the UK. We have previously described the development and preliminary assessment of a set of explicit indicators for this purpose [14,15]. This study aimed to assess the content and operational validity of these indicators from the perspective of UK hospital doctors.
Method
A set of 14 explicit indicators (Table 1) were derived from the literature [12,13,16], applied to data in the hospital medical records about all newly prescribed long-term drugs for randomly selected patients, and suitably refined [14]. They were designed to assess prescribing started during a hospital admission, which was expected to be continued long-term in primary care. The explicit indicators require that drugs be prescribed within the standards set out in the British National Formulary [11] or, if the prescriber wished to deviate from those, that the rationale be documented in the patient’s medical record.
Indicators for long-term prescribing commenced in secondary care
| A | The indication for the drug is recorded in the inpatient medical record |
| B | The indication for the drug is recorded in the discharge summary |
| C | The indication for the drug in the discharge prescription or discharge summary is upheld in the British National Formulary |
| D | If the drug is part of a questionable high-risk therapeutic combination, the prescriber gives a reason in the discharge summary |
| E | If the duration of a drug is outside the range stated in the British National Formulary, the prescriber gives a reason in the discharge summary |
| F | If a drug is listed in the British National Formulary as ‘less suitable for prescribing’, the prescriber gives a reason for prescribing it in the discharge summary |
| G | The drug is considered to be efficacious for the condition, or the prescriber gives a reason for prescribing it in the discharge summary |
| H | The drug is in the hospital formulary or the prescriber gives a reason for prescribing it in the patient medical record |
| J | If the total daily dose is outside the range stated in the British National Formulary, the prescriber gives a reason for prescribing it in the discharge prescription or discharge summary |
| K | If the dosage frequency is outside the range stated in the British National Formulary , the prescriber gives a reason for prescribing it in the discharge summary |
| L | If a type A drug reaction occurs, there are details given of the reaction and recommended future monitoring in the patient medical record |
| M | If a type B drug reaction occurs, the prescriber gives details of the reaction and recommended action in both the inpatient record and the discharge summary |
| N | If a potentially hazardous drug–drug combination is prescribed, the prescriber gives details of the interaction and recommended action in both the inpatient record and the discharge summary |
| O | If a potentially hazardous drug–disease combination is prescribed, the prescriber gives a reason in the discharge summary |
| A | The indication for the drug is recorded in the inpatient medical record |
| B | The indication for the drug is recorded in the discharge summary |
| C | The indication for the drug in the discharge prescription or discharge summary is upheld in the British National Formulary |
| D | If the drug is part of a questionable high-risk therapeutic combination, the prescriber gives a reason in the discharge summary |
| E | If the duration of a drug is outside the range stated in the British National Formulary, the prescriber gives a reason in the discharge summary |
| F | If a drug is listed in the British National Formulary as ‘less suitable for prescribing’, the prescriber gives a reason for prescribing it in the discharge summary |
| G | The drug is considered to be efficacious for the condition, or the prescriber gives a reason for prescribing it in the discharge summary |
| H | The drug is in the hospital formulary or the prescriber gives a reason for prescribing it in the patient medical record |
| J | If the total daily dose is outside the range stated in the British National Formulary, the prescriber gives a reason for prescribing it in the discharge prescription or discharge summary |
| K | If the dosage frequency is outside the range stated in the British National Formulary , the prescriber gives a reason for prescribing it in the discharge summary |
| L | If a type A drug reaction occurs, there are details given of the reaction and recommended future monitoring in the patient medical record |
| M | If a type B drug reaction occurs, the prescriber gives details of the reaction and recommended action in both the inpatient record and the discharge summary |
| N | If a potentially hazardous drug–drug combination is prescribed, the prescriber gives details of the interaction and recommended action in both the inpatient record and the discharge summary |
| O | If a potentially hazardous drug–disease combination is prescribed, the prescriber gives a reason in the discharge summary |
Indicators for long-term prescribing commenced in secondary care
| A | The indication for the drug is recorded in the inpatient medical record |
| B | The indication for the drug is recorded in the discharge summary |
| C | The indication for the drug in the discharge prescription or discharge summary is upheld in the British National Formulary |
| D | If the drug is part of a questionable high-risk therapeutic combination, the prescriber gives a reason in the discharge summary |
| E | If the duration of a drug is outside the range stated in the British National Formulary, the prescriber gives a reason in the discharge summary |
| F | If a drug is listed in the British National Formulary as ‘less suitable for prescribing’, the prescriber gives a reason for prescribing it in the discharge summary |
| G | The drug is considered to be efficacious for the condition, or the prescriber gives a reason for prescribing it in the discharge summary |
| H | The drug is in the hospital formulary or the prescriber gives a reason for prescribing it in the patient medical record |
| J | If the total daily dose is outside the range stated in the British National Formulary, the prescriber gives a reason for prescribing it in the discharge prescription or discharge summary |
| K | If the dosage frequency is outside the range stated in the British National Formulary , the prescriber gives a reason for prescribing it in the discharge summary |
| L | If a type A drug reaction occurs, there are details given of the reaction and recommended future monitoring in the patient medical record |
| M | If a type B drug reaction occurs, the prescriber gives details of the reaction and recommended action in both the inpatient record and the discharge summary |
| N | If a potentially hazardous drug–drug combination is prescribed, the prescriber gives details of the interaction and recommended action in both the inpatient record and the discharge summary |
| O | If a potentially hazardous drug–disease combination is prescribed, the prescriber gives a reason in the discharge summary |
| A | The indication for the drug is recorded in the inpatient medical record |
| B | The indication for the drug is recorded in the discharge summary |
| C | The indication for the drug in the discharge prescription or discharge summary is upheld in the British National Formulary |
| D | If the drug is part of a questionable high-risk therapeutic combination, the prescriber gives a reason in the discharge summary |
| E | If the duration of a drug is outside the range stated in the British National Formulary, the prescriber gives a reason in the discharge summary |
| F | If a drug is listed in the British National Formulary as ‘less suitable for prescribing’, the prescriber gives a reason for prescribing it in the discharge summary |
| G | The drug is considered to be efficacious for the condition, or the prescriber gives a reason for prescribing it in the discharge summary |
| H | The drug is in the hospital formulary or the prescriber gives a reason for prescribing it in the patient medical record |
| J | If the total daily dose is outside the range stated in the British National Formulary, the prescriber gives a reason for prescribing it in the discharge prescription or discharge summary |
| K | If the dosage frequency is outside the range stated in the British National Formulary , the prescriber gives a reason for prescribing it in the discharge summary |
| L | If a type A drug reaction occurs, there are details given of the reaction and recommended future monitoring in the patient medical record |
| M | If a type B drug reaction occurs, the prescriber gives details of the reaction and recommended action in both the inpatient record and the discharge summary |
| N | If a potentially hazardous drug–drug combination is prescribed, the prescriber gives details of the interaction and recommended action in both the inpatient record and the discharge summary |
| O | If a potentially hazardous drug–disease combination is prescribed, the prescriber gives a reason in the discharge summary |
Further content and operational validity assessments were conducted in this study using two methods, data extraction from medical records and qualitative interviews. Approval for the programme of work was obtained from the relevant National Health Service Local Research Ethics Committee.
Data extraction from medical records
Patients discharged from medical or care of the elderly wards, at one large teaching hospital in England, were screened for eligibility. The inclusion criteria were that the patient was discharged on a drug that was commenced during this admission, the discharge summary did not recommend that the drug be discontinued within 3 months and the patients were suitable for interview (data not presented in this article). Post-discharge, the indicators were applied to these long-term prescriptions, using data from the patient records.
Interviews
Doctors working in general medical specialities (e.g. cardiology, endocrinology, or geriatrics) were invited to take part in one of two types of in-depth interviews. They were eligible for the first if they had either made or documented in the notes the decision to commence a drug for one of the eligible patients, where their prescription had failed one or more indicators. Additional doctors from the same departments were invited to participate in the second type of interviews, which discussed appropriate prescribing more generally, to widen the scope of the data. A maximum variability sample was sought in both cases.
The interviews were qualitative, lasting 30–60 minutes, audiotaped and transcribed in full. Doctors were not given a definition of appropriate prescribing but were asked how they would judge the appropriateness of prescribing decisions that they or their colleagues might make. Where applicable, they also discussed how their prescribing had been assessed by the indicators (a copy was provided), both specifically and as examples of what they considered ‘normal practice’.
Data Analysis
Data were recorded as to how each prescription had been judged by each of the explicit indicators. Each indicator used an ordinal scale, which was further subclassified as to whether each prescription was appropriate or inappropriate.
The interview data were analysed from an interpretivist perspective, assisted by NVivo software (QSR, La Trobe University, Melbourne, Australia). Themes were identified during a process of repeated reviewing of the transcripts and data tabulated, using the Framework method [17].
Content validity meant that the explicit indicators included all the relevant domains of appropriate prescribing. The indicators (Table 1) reflected the domains of indication (Indicators A–C), evidence for effectiveness (F, G), cost effectiveness (inclusion in the hospital formulary as a proxy, H), adverse reactions (L, M) to the drug, high risk or hazardous drug–drug or drug–disease combinations (D, N, O) and duration, dose and frequency of the prescription (E, J, K). Each indicator was judged to have content validity if the domain was discussed by the doctors as being part of how they would judge appropriate prescribing.
Operational validity meant that the indicators were capable of being implemented in practice. The operational definitions used data from the patients’ medical records (e.g. Appendix). The British National Formulary was used as the ‘gold standard’. This validity assessment used a combination of data extraction from medical records and qualitative interviews with the prescribing doctors. The indicators were judged to have operational validity if, when prescriptions were judged inappropriate by them, it was because they were actually inappropriate, rather than due to a problem with the data used to assess the prescriptions.
Results
Sixty-one patients met the inclusion criteria; the indicators were applied to 132 new long-term prescriptions (Table 2). Thirteen doctors (three consultants, two specialist registrars, six senior and two pre-registration house officers) were interviewed about 38 prescriptions for 13 patients. A further 23 doctors (eight pre-registration house officers and five each of senior house officers, specialist registrars, and consultants) were purposively selected for the more general interviews. Illustrative quotations from the interviewed doctors are provided in Table 3, which were selected for their relevance to the themes.
Assessment of 132 long-term prescriptions by indicators of appropriate prescribing
| Indicator . | Appropriate prescriptions . | Inappropriate prescriptions . | Inappropriate prescriptions discussed during interviews1 . | Inappropriate prescriptions after interviews . |
|---|---|---|---|---|
| Indication in inpatient medical record | 114 | 18 | 6 | 4 |
| Indication in discharge summary | 82 | 50 | 15 | 6 |
| Indication in prescription upheld in British National Formulary | 93 | 39 | 17 | 7 |
| Questionable high-risk combination | 130 | 2 | 0 | 0 |
| Duration outside range in British National Formulary | 122 | 10 | 1 | 0 |
| ‘Less suitable for prescribing’ in British National Formulary | 131 | 1 | 0 | 0 |
| Drug efficacious for indication | 111 | 21 | 6 | 5 |
| Drug in hospital formulary | 126 | 6 | 1 | 1 |
| Total daily dose outside range in British National Formulary | 121 | 11 | 4 | 2 |
| Dosage frequency outside British National Formulary range | 126 | 6 | 0 | 0 |
| Type A reaction occurred without action | 128 | 4 | 1 | 0 |
| Type B reaction occurred without action | 132 | 0 | 0 | 0 |
| Hazardous drug–drug combination | 98 | 34 | 14 | 8 |
| Hazardous drug–disease combination | 104 | 28 | 5 | 2 |
| Indicator . | Appropriate prescriptions . | Inappropriate prescriptions . | Inappropriate prescriptions discussed during interviews1 . | Inappropriate prescriptions after interviews . |
|---|---|---|---|---|
| Indication in inpatient medical record | 114 | 18 | 6 | 4 |
| Indication in discharge summary | 82 | 50 | 15 | 6 |
| Indication in prescription upheld in British National Formulary | 93 | 39 | 17 | 7 |
| Questionable high-risk combination | 130 | 2 | 0 | 0 |
| Duration outside range in British National Formulary | 122 | 10 | 1 | 0 |
| ‘Less suitable for prescribing’ in British National Formulary | 131 | 1 | 0 | 0 |
| Drug efficacious for indication | 111 | 21 | 6 | 5 |
| Drug in hospital formulary | 126 | 6 | 1 | 1 |
| Total daily dose outside range in British National Formulary | 121 | 11 | 4 | 2 |
| Dosage frequency outside British National Formulary range | 126 | 6 | 0 | 0 |
| Type A reaction occurred without action | 128 | 4 | 1 | 0 |
| Type B reaction occurred without action | 132 | 0 | 0 | 0 |
| Hazardous drug–drug combination | 98 | 34 | 14 | 8 |
| Hazardous drug–disease combination | 104 | 28 | 5 | 2 |
Column does not total 38, as several prescriptions failed more than one indicator.
Assessment of 132 long-term prescriptions by indicators of appropriate prescribing
| Indicator . | Appropriate prescriptions . | Inappropriate prescriptions . | Inappropriate prescriptions discussed during interviews1 . | Inappropriate prescriptions after interviews . |
|---|---|---|---|---|
| Indication in inpatient medical record | 114 | 18 | 6 | 4 |
| Indication in discharge summary | 82 | 50 | 15 | 6 |
| Indication in prescription upheld in British National Formulary | 93 | 39 | 17 | 7 |
| Questionable high-risk combination | 130 | 2 | 0 | 0 |
| Duration outside range in British National Formulary | 122 | 10 | 1 | 0 |
| ‘Less suitable for prescribing’ in British National Formulary | 131 | 1 | 0 | 0 |
| Drug efficacious for indication | 111 | 21 | 6 | 5 |
| Drug in hospital formulary | 126 | 6 | 1 | 1 |
| Total daily dose outside range in British National Formulary | 121 | 11 | 4 | 2 |
| Dosage frequency outside British National Formulary range | 126 | 6 | 0 | 0 |
| Type A reaction occurred without action | 128 | 4 | 1 | 0 |
| Type B reaction occurred without action | 132 | 0 | 0 | 0 |
| Hazardous drug–drug combination | 98 | 34 | 14 | 8 |
| Hazardous drug–disease combination | 104 | 28 | 5 | 2 |
| Indicator . | Appropriate prescriptions . | Inappropriate prescriptions . | Inappropriate prescriptions discussed during interviews1 . | Inappropriate prescriptions after interviews . |
|---|---|---|---|---|
| Indication in inpatient medical record | 114 | 18 | 6 | 4 |
| Indication in discharge summary | 82 | 50 | 15 | 6 |
| Indication in prescription upheld in British National Formulary | 93 | 39 | 17 | 7 |
| Questionable high-risk combination | 130 | 2 | 0 | 0 |
| Duration outside range in British National Formulary | 122 | 10 | 1 | 0 |
| ‘Less suitable for prescribing’ in British National Formulary | 131 | 1 | 0 | 0 |
| Drug efficacious for indication | 111 | 21 | 6 | 5 |
| Drug in hospital formulary | 126 | 6 | 1 | 1 |
| Total daily dose outside range in British National Formulary | 121 | 11 | 4 | 2 |
| Dosage frequency outside British National Formulary range | 126 | 6 | 0 | 0 |
| Type A reaction occurred without action | 128 | 4 | 1 | 0 |
| Type B reaction occurred without action | 132 | 0 | 0 | 0 |
| Hazardous drug–drug combination | 98 | 34 | 14 | 8 |
| Hazardous drug–disease combination | 104 | 28 | 5 | 2 |
Column does not total 38, as several prescriptions failed more than one indicator.
Quotations taken from qualitative interviews, selected for their relevance to the themes described in the text
| Quote number . | Interviewed doctor1 . | . |
|---|---|---|
| 1 | House officer 5 | ‘So, I think it’s [appropriate prescribing] much less of an issue than it used to be, because now there’s policies for most things…as long as you stick to them everything’s appropriate really’ (general interview) |
| 2 | Consultant 1 | ‘So within that whole context, it’s a cost effectiveness decision, but when I say cost, I am talking about financial cost, but I am also talking far more about the risks to the patient, the costs to the patient’ (general interview) |
| 3 | Consultant 5 | ‘And ahem, we don’t have enough information to know when you are, when it’s the fifth tablet, whether its really worth while having your sixth, which might make it difficult for you to remember to take your seventh, which might be of more benefit than, sort of, it’s a difficult thing’ (general interview) |
| 4 | Specialist registrar 2 | ‘That’s usually, that would be inappropriate – certainly everybody would say that was inappropriate if the kidney goes off. If it doesn’t, then it’s like you got away with it, or was it appropriate? Presumably not, ’cause there’s almost always alternatives’ (general interview) |
| 5 | House officer 7 | ‘Prescribing, em, something where a patient’s got a contraindication, for example giving aspirin to somebody who’s had a haemorrhagic stroke or giving something like Voltarol® [diclofenac] to someone who’s got bad asthma, I’d say it’s inappropriate’ (general interview) |
| 6 | Consultant 8 | ‘You can prescribe appropriately without writing anything in the notes, the problem is just then nobody knows how appropriate they were [laughs] they have to make up their own mind, you know, what your reasons were, so unless its barn door obvious, it’s best to write it in the notes’ (interview about specific prescription) |
| 7 | Senior house officer 8 | ‘When somebody had lots and lots of problems then something like ‘[her] iron was constipating’ would be understood, it would go unsaid’ (interview about specific prescription) |
| 8 | Senior house officer 1 | ‘They’ll be seen by one doctor who does all the changing and things like that and eventually you’ll come to do the discharge summary and its not completely obvious from the notes why things have been changed…you tell him it has been changed but you can’t always give information why’ (interview about specific prescription) |
| 9 | House officer 10 | ‘I couldn’t tell you what most of the drugs that I use are licensed for, to be honest. As I say, I like to use them on the basis of evidence but even a lot of things you get evidence for from studies aren’t not actually licensed for, for a while are they?’ (interview about specific prescription) |
| Quote number . | Interviewed doctor1 . | . |
|---|---|---|
| 1 | House officer 5 | ‘So, I think it’s [appropriate prescribing] much less of an issue than it used to be, because now there’s policies for most things…as long as you stick to them everything’s appropriate really’ (general interview) |
| 2 | Consultant 1 | ‘So within that whole context, it’s a cost effectiveness decision, but when I say cost, I am talking about financial cost, but I am also talking far more about the risks to the patient, the costs to the patient’ (general interview) |
| 3 | Consultant 5 | ‘And ahem, we don’t have enough information to know when you are, when it’s the fifth tablet, whether its really worth while having your sixth, which might make it difficult for you to remember to take your seventh, which might be of more benefit than, sort of, it’s a difficult thing’ (general interview) |
| 4 | Specialist registrar 2 | ‘That’s usually, that would be inappropriate – certainly everybody would say that was inappropriate if the kidney goes off. If it doesn’t, then it’s like you got away with it, or was it appropriate? Presumably not, ’cause there’s almost always alternatives’ (general interview) |
| 5 | House officer 7 | ‘Prescribing, em, something where a patient’s got a contraindication, for example giving aspirin to somebody who’s had a haemorrhagic stroke or giving something like Voltarol® [diclofenac] to someone who’s got bad asthma, I’d say it’s inappropriate’ (general interview) |
| 6 | Consultant 8 | ‘You can prescribe appropriately without writing anything in the notes, the problem is just then nobody knows how appropriate they were [laughs] they have to make up their own mind, you know, what your reasons were, so unless its barn door obvious, it’s best to write it in the notes’ (interview about specific prescription) |
| 7 | Senior house officer 8 | ‘When somebody had lots and lots of problems then something like ‘[her] iron was constipating’ would be understood, it would go unsaid’ (interview about specific prescription) |
| 8 | Senior house officer 1 | ‘They’ll be seen by one doctor who does all the changing and things like that and eventually you’ll come to do the discharge summary and its not completely obvious from the notes why things have been changed…you tell him it has been changed but you can’t always give information why’ (interview about specific prescription) |
| 9 | House officer 10 | ‘I couldn’t tell you what most of the drugs that I use are licensed for, to be honest. As I say, I like to use them on the basis of evidence but even a lot of things you get evidence for from studies aren’t not actually licensed for, for a while are they?’ (interview about specific prescription) |
Doctors’ training in UK: Consultant, specialist doctors, holding certificate of completion of specialist training; specialist registrar, undertaking higher specialist training (4–6 years, depending on the speciality); senior house officer, undertaking basic specialist training (usually 2–3 years); house officer, undertaking 1 year immediately post-graduation training (before registration with General Medical Council).
Quotations taken from qualitative interviews, selected for their relevance to the themes described in the text
| Quote number . | Interviewed doctor1 . | . |
|---|---|---|
| 1 | House officer 5 | ‘So, I think it’s [appropriate prescribing] much less of an issue than it used to be, because now there’s policies for most things…as long as you stick to them everything’s appropriate really’ (general interview) |
| 2 | Consultant 1 | ‘So within that whole context, it’s a cost effectiveness decision, but when I say cost, I am talking about financial cost, but I am also talking far more about the risks to the patient, the costs to the patient’ (general interview) |
| 3 | Consultant 5 | ‘And ahem, we don’t have enough information to know when you are, when it’s the fifth tablet, whether its really worth while having your sixth, which might make it difficult for you to remember to take your seventh, which might be of more benefit than, sort of, it’s a difficult thing’ (general interview) |
| 4 | Specialist registrar 2 | ‘That’s usually, that would be inappropriate – certainly everybody would say that was inappropriate if the kidney goes off. If it doesn’t, then it’s like you got away with it, or was it appropriate? Presumably not, ’cause there’s almost always alternatives’ (general interview) |
| 5 | House officer 7 | ‘Prescribing, em, something where a patient’s got a contraindication, for example giving aspirin to somebody who’s had a haemorrhagic stroke or giving something like Voltarol® [diclofenac] to someone who’s got bad asthma, I’d say it’s inappropriate’ (general interview) |
| 6 | Consultant 8 | ‘You can prescribe appropriately without writing anything in the notes, the problem is just then nobody knows how appropriate they were [laughs] they have to make up their own mind, you know, what your reasons were, so unless its barn door obvious, it’s best to write it in the notes’ (interview about specific prescription) |
| 7 | Senior house officer 8 | ‘When somebody had lots and lots of problems then something like ‘[her] iron was constipating’ would be understood, it would go unsaid’ (interview about specific prescription) |
| 8 | Senior house officer 1 | ‘They’ll be seen by one doctor who does all the changing and things like that and eventually you’ll come to do the discharge summary and its not completely obvious from the notes why things have been changed…you tell him it has been changed but you can’t always give information why’ (interview about specific prescription) |
| 9 | House officer 10 | ‘I couldn’t tell you what most of the drugs that I use are licensed for, to be honest. As I say, I like to use them on the basis of evidence but even a lot of things you get evidence for from studies aren’t not actually licensed for, for a while are they?’ (interview about specific prescription) |
| Quote number . | Interviewed doctor1 . | . |
|---|---|---|
| 1 | House officer 5 | ‘So, I think it’s [appropriate prescribing] much less of an issue than it used to be, because now there’s policies for most things…as long as you stick to them everything’s appropriate really’ (general interview) |
| 2 | Consultant 1 | ‘So within that whole context, it’s a cost effectiveness decision, but when I say cost, I am talking about financial cost, but I am also talking far more about the risks to the patient, the costs to the patient’ (general interview) |
| 3 | Consultant 5 | ‘And ahem, we don’t have enough information to know when you are, when it’s the fifth tablet, whether its really worth while having your sixth, which might make it difficult for you to remember to take your seventh, which might be of more benefit than, sort of, it’s a difficult thing’ (general interview) |
| 4 | Specialist registrar 2 | ‘That’s usually, that would be inappropriate – certainly everybody would say that was inappropriate if the kidney goes off. If it doesn’t, then it’s like you got away with it, or was it appropriate? Presumably not, ’cause there’s almost always alternatives’ (general interview) |
| 5 | House officer 7 | ‘Prescribing, em, something where a patient’s got a contraindication, for example giving aspirin to somebody who’s had a haemorrhagic stroke or giving something like Voltarol® [diclofenac] to someone who’s got bad asthma, I’d say it’s inappropriate’ (general interview) |
| 6 | Consultant 8 | ‘You can prescribe appropriately without writing anything in the notes, the problem is just then nobody knows how appropriate they were [laughs] they have to make up their own mind, you know, what your reasons were, so unless its barn door obvious, it’s best to write it in the notes’ (interview about specific prescription) |
| 7 | Senior house officer 8 | ‘When somebody had lots and lots of problems then something like ‘[her] iron was constipating’ would be understood, it would go unsaid’ (interview about specific prescription) |
| 8 | Senior house officer 1 | ‘They’ll be seen by one doctor who does all the changing and things like that and eventually you’ll come to do the discharge summary and its not completely obvious from the notes why things have been changed…you tell him it has been changed but you can’t always give information why’ (interview about specific prescription) |
| 9 | House officer 10 | ‘I couldn’t tell you what most of the drugs that I use are licensed for, to be honest. As I say, I like to use them on the basis of evidence but even a lot of things you get evidence for from studies aren’t not actually licensed for, for a while are they?’ (interview about specific prescription) |
Doctors’ training in UK: Consultant, specialist doctors, holding certificate of completion of specialist training; specialist registrar, undertaking higher specialist training (4–6 years, depending on the speciality); senior house officer, undertaking basic specialist training (usually 2–3 years); house officer, undertaking 1 year immediately post-graduation training (before registration with General Medical Council).
Content validity
The term ‘appropriate prescribing’ was used by the doctors in general conversation, with an implicit, rather than explicit, understanding of its meaning. A ‘recognized’ or ‘acceptable’ indication for the drug was the paramount issue for most doctors (Indicators A–C). Evidence of, or a theoretical possibility of, benefit was seen by many as a further prerequisite for appropriateness (Indicators F, G). An appropriate drug, therefore, had to be ‘needed’, and an inappropriate drug the opposite.
Clinical trials played an important role but were not the sole source of acceptable evidence (Indicators F, G). The blind application of such evidence, ‘taking generic advice and applying it irrationally to people throughout the population’ (Consultant 1), was also considered inappropriate. Phrases such as ‘pharmacological evidence’ were used to describe the doctor’s extrapolation of a drug’s action to a putative benefit where clinical trials were lacking.
Doctors of all grades raised the ‘cost’ of prescribing drugs (Indicator H). The junior doctors considered adherence to policies, such as the hospital formulary, which incorporated this ‘cost’ into drug selection, to be important (Table 3, quote 1). Acquisition cost was mentioned by two doctors but only as a final deciding factor in decision-making. The potential ‘cost’ was most often described in terms of clinical problems, such as adverse effects (indicators L, M), onerous monitoring or the impact on patients (Table 3, quote 2). Doctors working in care of the elderly and nephrology, where polypharmacy was common, mentioned drug–drug interactions in most detail (Indicators D, N) (Table 3, quote 3). Another questioned whether only interactions that had actually caused morbidity should be classified as inappropriate prescribing. He gave as an example the co-prescription of angiotensin converting enzyme inhibitors and non-steroidal anti-inflammatory drugs (Table 3, quote 4).
There was recognition of the risk–benefit relationship in drug–disease interactions (Indicator O) that had to be weighed up, consciously or unconsciously, for each prescribing decision (Table 3, quote 5). The doctors knew that patients may need drugs that were cautioned with concomitant diseases (such as amiodarone with thyroid disease), but these were chosen based ‘on the old adage of doing least harm or doing no harm’ (specialist registrar 2). ‘Continuing medication indefinitely’ and ‘lack of review of continued need’ (Indicator E) were further examples of inappropriate prescribing. The dose and frequency of the drugs were raised (Indicators J, K), usually for drugs with narrow therapeutic indices, such as phenytoin.
There were two main areas raised in the discussions of appropriate prescribing that were not assessed by the indicators. The senior doctors raised the issues of patient acceptability and ability to take the medication as being part of the overall concept of appropriateness. They considered these important aspects that addressed the needs and circumstances of the individual patient.
Operational validity
Thirty-eight prescriptions for 13 patients, which had failed at least one indicator (Table 2), were discussed in the interviews. Additional information was provided by the doctors during the interviews, which allowed judgements of some of the indicators to be revised. Although the interviewed doctors had been involved in the prescribing process, not all knew additional information that had not been documented in the medical record. Two main themes underpinned the prescriptions failing the indicators: documentation of information relating to prescribing decisions in the medical record and the suitability of the ‘gold standard’.
The indication per se was linked closely to the doctor’s views of appropriateness. However, the existence of an indication did not necessarily mean that documentation would exist to allow independent assessment (Table 3, quote 6). Many of the doctors ‘would like to think’ that the indication for a drug would always be in the medical record, but recognized that this was unlikely to be the case (Indicator A). Reasons included the lack of available time and the questionable necessity to record the information. In particular, drugs of low perceived risk (such as laxatives) or that replaced others in the same therapeutic class (such as omeprazole replacing rabeprazole) were unlikely to be documented.
Similarly, the rationale for prescribing new drugs would often be omitted from the discharge summary (Indicator B) if this was perceived as unnecessary, not considered a priority or not known to the doctor writing the prescription. The doctors often expected that the general practitioner could deduce prescribing rationales from other available information. Less important information would be excluded from the discharge summary, to avoid compromising the important content. For example, when a patient had been admitted with a myocardial infarction and subsequently diagnosed with diabetes, problems with additional treatment for iron deficiency anemia were considered less important (Table 3, quote 7).
The impact of shift working or transferring patients between clinical teams meant that the doctor who wrote the final discharge summary may well not have been present when initial prescribing decisions were made. Therefore, if the information was not documented, it could not be transferred into the discharge summary (Table 3, quote 8).
There was recognition of the usefulness but also the limitations of the British National Formulary in some specialist areas, where the drugs would be used in unusual ways for complex patients. Five prescriptions failed Indicator C, because the drugs were used for unlicensed indications, with no rationale in the discharge summary. The interviews highlighted how common this was in certain specialities, which reduced the perceived importance of information transfer. In addition, including such information in the discharge summary requires knowledge of the drug’s license. Junior doctors, who normally write the notes and discharge summaries, described a very limited knowledge of this (Table 3, quote 9).
Discussion
This study assessed the content and operational validity of a set of 14 explicit indicators of appropriate long-term prescribing in secondary care in the UK. Content validity was achieved, as all indicators assessed domains that were considered relevant to appropriate prescribing. Operational validity was not consistently achieved, as many prescriptions were incorrectly judged as inappropriate because of the lack of data in the medical record.
The indicators were used with 132 prescriptions for 61 patients; the interviews were with doctors in a single teaching hospital, working in medical or care of the elderly specialities. All of these may limit the generalizability of the findings. Doctors from other hospitals or specialities may have views on the meaning and assessment of appropriate prescribing that are not reflected here. However, data saturation was achieved for this group of doctors, with no additional themes appearing during later interviews.
Appropriate prescribing was viewed by the doctors as being indicated, necessary, evidence-based (using a broad meaning for ‘evidence’) and of acceptable cost and risk–benefit ratio. In addition, they saw appropriate and inappropriate prescribing as concepts that applied to individual prescriptions for individual patients, as proposed by Buetow and colleagues [5], rather than at a public health level. They did not class drugs or types of prescribing as inappropriate per se, as viewed by Beers and colleagues [7].
These explicit indicators can be criticized as having a purely clinical perspective. Appropriate prescribing is not merely about technical aspects but should also reflect the ways in which medicines are used by patients [18]. Sharpe and Faden [19] have contended that appropriateness is ‘fundamentally evaluative’ and that its determination is not just based on evidence but on ‘values assessment’ by all the major players. In the case of prescribing, that also includes the patient. Elderly patients consider practicality of use as an important aspect of appropriate prescribing [20]. The doctors in our study also discussed patient acceptability, but the lack of relevant data in the medical notes prevented the development of suitable explicit indicators [14].
Using the medical record was intended to facilitate assessing appropriate prescribing as explicitly as possible, minimizing the need for implicit judgements. However, medical records are known to be imperfect tools even for supporting patient care [21]. The doctors’ descriptions of appropriate prescribing did not suggest that these domains would, or even should, be explicitly documented. However, inappropriate prescribing can result from rational decision-making based on inadequate information about the patient [22]. Hence, there is an argument for a greater emphasis on training doctors in good prescribing documentation.
The doctors interviewed in this study described how they would extrapolate information from data in the medical records or not to include points they expected colleagues would already know. Detailed documentation in the medical record is contrary to normal clinical practice, where information must be quick to both write and read [23,24]. However, this ‘reading between the lines’ can often only be done by health-care professionals. For example, when non-clinically trained data analysts screened medical records for adverse drug events or medical errors, they had to pass 93% of cases to a study nurse for further review [25].
The use of these indicators is currently hampered by their operational validity, due to the lack of required data. However, the electronic patient record may provide a solution. For example, electronic prescribing software currently being implemented in UK hospitals can require the prescriber to select one of many predefined reasons for having modified or discontinued prescriptions (personal observation). Similar mandatory documentation of drug indication could be technically possible and would facilitate the use of these indicators for both clinical audit and research. It will be imperative to reassess the operational validity, however, as the way in which data are recorded electronically is likely to be different to how they are recorded on article [26].
The British National Formulary [11] was successfully used as the gold standard in this study. There were very few occasions when drugs were prescribed for indications or regimens that contravened that standard. Nonetheless, these hospital doctors commented on its limitations in a specialist health care environment. The flexibility of these indicators makes such deviations from the standard possible, but the burden of proof for the effectiveness lies with the prescriber. For such prescriptions to pass the indicators, justification needs to be documented in the patient’s medical record.
These indicators of appropriate prescribing move the research agenda from a more general consideration of appropriate drugs to a consideration of appropriate prescribing for the individual patient. Other work [6,27] in prescribing use explicit criteria founded on evidence-based literature review. Our work accepts the reality that the evidence base is lacking for much of prescribing and takes account of this in permitting the use of drugs both within and outside their licensed indication, provided the doctor is willing to provide justification. Thus, it may take greater account of the ‘complex and fluid reality’ in the implementation of evidence-based medicine for individual patients [28].
Appendix
Example of detailed operational definitions for one of the indicators of appropriate long-term prescribing
| Indicator A: The indication for the drug is recorded in the inpatient medical record . | . | |
|---|---|---|
| Judgement . | Explicit operational definition . | |
| Drug passes indicator | ||
| A1 The indication for the drug is explicitly recorded in the inpatient records | The indication is the sign, symptom, disease or condition for which the drug is prescribed. There is a statement by a prescriber in the inpatient record that the drug (or a member of a specified drug group) has been commenced and, in the same entry, a statement as to why the drug has been commenced. The statement may refer to the use of a protocol or regime in use, rather than state their contents. Example(s): ‘chest pain—isosorbide mononitrate’, ‘tender epigastrum—for PPI’, or ‘alcohol detoxification protocol’ | |
| A2 The indication for the drug is not explicitly recorded in the inpatient records, but evidence suggests an indication for which the drug would be suitable | There is evidence in the inpatient record of an indication for which the drug could be commenced (such as in a separate dated entry, in the nursing notes or in the laboratory test results). A statement by a prescriber that the drug (or a member of a specified drug group) has been commenced may or may not be included in the inpatient record. Example(s): ‘Cholesterol 5.8’ in margin of notes, but no mention of starting simvastatin or statins; ‘chest tightness this morning’ in nursing notes and ‘add nitrate’ in ward round notes; ‘paracetamol to reduce pyrexia’ in nursing notes only | |
| Drug fails indicator | ||
| A3 The drug replaces another of the same type, without explanation in the inpatient record | The drug has replaced another drug from the same subsection of the British National Formulary that the patient was previously prescribed (at any other time). There is no statement by a prescriber or report of a statement by a prescriber in the nursing notes as to why the substitution was necessary. There may or may not be a statement in the inpatient notes that the substitution has taken place. Example(s): lisinopril started and captopril (commenced during this admission) discontinued, without explanation; omeprazole prescribed at home and rabeprazole started on admission to hospital without explanation | |
| A4 The drug replaces another for the same indication, without explanation in the inpatient record | The drug has replaced another drug that the patient was previously prescribed (at any other time) for the same indication. There is no statement by a prescriber or report of a statement by a prescriber in the nursing notes as to why the substitution has taken place. There may or may not be a statement in inpatient notes that the substitution has taken place. If the indication is not known for the drug being replaced, then use criterion A5 instead. Example(s): lisinopril started and bendroflumethiazide (commenced for hypertension) discontinued, without explanation; cimetidine prescribed at home and rabeprazole started on admission to hospital without explanation | |
| A5 There is no explicitly recorded indication, or evidence to suggest a suitable indication for the drug, in the inpatient record | There is no statement by a prescriber in the inpatient record as to why the drug (or a member of a specified drug group) has been commenced. There is no evidence in the inpatient record of an indication for which the drug could be commenced (such as in a separate dated entry, in the nursing notes or in the laboratory test results). A statement by a prescriber that the drug (or a member of a specified drug group) has been commenced may or may not be included in the inpatient record. Example(s): quinine sulphate prescribed, with the only details in the record as a comment in nursing notes ‘will try to get Dr to prescribe quinine’ | |
| A6 Other | The drug does not fulfil any of the criteria A1–5. Give further details | |
| Indicator A: The indication for the drug is recorded in the inpatient medical record . | . | |
|---|---|---|
| Judgement . | Explicit operational definition . | |
| Drug passes indicator | ||
| A1 The indication for the drug is explicitly recorded in the inpatient records | The indication is the sign, symptom, disease or condition for which the drug is prescribed. There is a statement by a prescriber in the inpatient record that the drug (or a member of a specified drug group) has been commenced and, in the same entry, a statement as to why the drug has been commenced. The statement may refer to the use of a protocol or regime in use, rather than state their contents. Example(s): ‘chest pain—isosorbide mononitrate’, ‘tender epigastrum—for PPI’, or ‘alcohol detoxification protocol’ | |
| A2 The indication for the drug is not explicitly recorded in the inpatient records, but evidence suggests an indication for which the drug would be suitable | There is evidence in the inpatient record of an indication for which the drug could be commenced (such as in a separate dated entry, in the nursing notes or in the laboratory test results). A statement by a prescriber that the drug (or a member of a specified drug group) has been commenced may or may not be included in the inpatient record. Example(s): ‘Cholesterol 5.8’ in margin of notes, but no mention of starting simvastatin or statins; ‘chest tightness this morning’ in nursing notes and ‘add nitrate’ in ward round notes; ‘paracetamol to reduce pyrexia’ in nursing notes only | |
| Drug fails indicator | ||
| A3 The drug replaces another of the same type, without explanation in the inpatient record | The drug has replaced another drug from the same subsection of the British National Formulary that the patient was previously prescribed (at any other time). There is no statement by a prescriber or report of a statement by a prescriber in the nursing notes as to why the substitution was necessary. There may or may not be a statement in the inpatient notes that the substitution has taken place. Example(s): lisinopril started and captopril (commenced during this admission) discontinued, without explanation; omeprazole prescribed at home and rabeprazole started on admission to hospital without explanation | |
| A4 The drug replaces another for the same indication, without explanation in the inpatient record | The drug has replaced another drug that the patient was previously prescribed (at any other time) for the same indication. There is no statement by a prescriber or report of a statement by a prescriber in the nursing notes as to why the substitution has taken place. There may or may not be a statement in inpatient notes that the substitution has taken place. If the indication is not known for the drug being replaced, then use criterion A5 instead. Example(s): lisinopril started and bendroflumethiazide (commenced for hypertension) discontinued, without explanation; cimetidine prescribed at home and rabeprazole started on admission to hospital without explanation | |
| A5 There is no explicitly recorded indication, or evidence to suggest a suitable indication for the drug, in the inpatient record | There is no statement by a prescriber in the inpatient record as to why the drug (or a member of a specified drug group) has been commenced. There is no evidence in the inpatient record of an indication for which the drug could be commenced (such as in a separate dated entry, in the nursing notes or in the laboratory test results). A statement by a prescriber that the drug (or a member of a specified drug group) has been commenced may or may not be included in the inpatient record. Example(s): quinine sulphate prescribed, with the only details in the record as a comment in nursing notes ‘will try to get Dr to prescribe quinine’ | |
| A6 Other | The drug does not fulfil any of the criteria A1–5. Give further details | |
Example of detailed operational definitions for one of the indicators of appropriate long-term prescribing
| Indicator A: The indication for the drug is recorded in the inpatient medical record . | . | |
|---|---|---|
| Judgement . | Explicit operational definition . | |
| Drug passes indicator | ||
| A1 The indication for the drug is explicitly recorded in the inpatient records | The indication is the sign, symptom, disease or condition for which the drug is prescribed. There is a statement by a prescriber in the inpatient record that the drug (or a member of a specified drug group) has been commenced and, in the same entry, a statement as to why the drug has been commenced. The statement may refer to the use of a protocol or regime in use, rather than state their contents. Example(s): ‘chest pain—isosorbide mononitrate’, ‘tender epigastrum—for PPI’, or ‘alcohol detoxification protocol’ | |
| A2 The indication for the drug is not explicitly recorded in the inpatient records, but evidence suggests an indication for which the drug would be suitable | There is evidence in the inpatient record of an indication for which the drug could be commenced (such as in a separate dated entry, in the nursing notes or in the laboratory test results). A statement by a prescriber that the drug (or a member of a specified drug group) has been commenced may or may not be included in the inpatient record. Example(s): ‘Cholesterol 5.8’ in margin of notes, but no mention of starting simvastatin or statins; ‘chest tightness this morning’ in nursing notes and ‘add nitrate’ in ward round notes; ‘paracetamol to reduce pyrexia’ in nursing notes only | |
| Drug fails indicator | ||
| A3 The drug replaces another of the same type, without explanation in the inpatient record | The drug has replaced another drug from the same subsection of the British National Formulary that the patient was previously prescribed (at any other time). There is no statement by a prescriber or report of a statement by a prescriber in the nursing notes as to why the substitution was necessary. There may or may not be a statement in the inpatient notes that the substitution has taken place. Example(s): lisinopril started and captopril (commenced during this admission) discontinued, without explanation; omeprazole prescribed at home and rabeprazole started on admission to hospital without explanation | |
| A4 The drug replaces another for the same indication, without explanation in the inpatient record | The drug has replaced another drug that the patient was previously prescribed (at any other time) for the same indication. There is no statement by a prescriber or report of a statement by a prescriber in the nursing notes as to why the substitution has taken place. There may or may not be a statement in inpatient notes that the substitution has taken place. If the indication is not known for the drug being replaced, then use criterion A5 instead. Example(s): lisinopril started and bendroflumethiazide (commenced for hypertension) discontinued, without explanation; cimetidine prescribed at home and rabeprazole started on admission to hospital without explanation | |
| A5 There is no explicitly recorded indication, or evidence to suggest a suitable indication for the drug, in the inpatient record | There is no statement by a prescriber in the inpatient record as to why the drug (or a member of a specified drug group) has been commenced. There is no evidence in the inpatient record of an indication for which the drug could be commenced (such as in a separate dated entry, in the nursing notes or in the laboratory test results). A statement by a prescriber that the drug (or a member of a specified drug group) has been commenced may or may not be included in the inpatient record. Example(s): quinine sulphate prescribed, with the only details in the record as a comment in nursing notes ‘will try to get Dr to prescribe quinine’ | |
| A6 Other | The drug does not fulfil any of the criteria A1–5. Give further details | |
| Indicator A: The indication for the drug is recorded in the inpatient medical record . | . | |
|---|---|---|
| Judgement . | Explicit operational definition . | |
| Drug passes indicator | ||
| A1 The indication for the drug is explicitly recorded in the inpatient records | The indication is the sign, symptom, disease or condition for which the drug is prescribed. There is a statement by a prescriber in the inpatient record that the drug (or a member of a specified drug group) has been commenced and, in the same entry, a statement as to why the drug has been commenced. The statement may refer to the use of a protocol or regime in use, rather than state their contents. Example(s): ‘chest pain—isosorbide mononitrate’, ‘tender epigastrum—for PPI’, or ‘alcohol detoxification protocol’ | |
| A2 The indication for the drug is not explicitly recorded in the inpatient records, but evidence suggests an indication for which the drug would be suitable | There is evidence in the inpatient record of an indication for which the drug could be commenced (such as in a separate dated entry, in the nursing notes or in the laboratory test results). A statement by a prescriber that the drug (or a member of a specified drug group) has been commenced may or may not be included in the inpatient record. Example(s): ‘Cholesterol 5.8’ in margin of notes, but no mention of starting simvastatin or statins; ‘chest tightness this morning’ in nursing notes and ‘add nitrate’ in ward round notes; ‘paracetamol to reduce pyrexia’ in nursing notes only | |
| Drug fails indicator | ||
| A3 The drug replaces another of the same type, without explanation in the inpatient record | The drug has replaced another drug from the same subsection of the British National Formulary that the patient was previously prescribed (at any other time). There is no statement by a prescriber or report of a statement by a prescriber in the nursing notes as to why the substitution was necessary. There may or may not be a statement in the inpatient notes that the substitution has taken place. Example(s): lisinopril started and captopril (commenced during this admission) discontinued, without explanation; omeprazole prescribed at home and rabeprazole started on admission to hospital without explanation | |
| A4 The drug replaces another for the same indication, without explanation in the inpatient record | The drug has replaced another drug that the patient was previously prescribed (at any other time) for the same indication. There is no statement by a prescriber or report of a statement by a prescriber in the nursing notes as to why the substitution has taken place. There may or may not be a statement in inpatient notes that the substitution has taken place. If the indication is not known for the drug being replaced, then use criterion A5 instead. Example(s): lisinopril started and bendroflumethiazide (commenced for hypertension) discontinued, without explanation; cimetidine prescribed at home and rabeprazole started on admission to hospital without explanation | |
| A5 There is no explicitly recorded indication, or evidence to suggest a suitable indication for the drug, in the inpatient record | There is no statement by a prescriber in the inpatient record as to why the drug (or a member of a specified drug group) has been commenced. There is no evidence in the inpatient record of an indication for which the drug could be commenced (such as in a separate dated entry, in the nursing notes or in the laboratory test results). A statement by a prescriber that the drug (or a member of a specified drug group) has been commenced may or may not be included in the inpatient record. Example(s): quinine sulphate prescribed, with the only details in the record as a comment in nursing notes ‘will try to get Dr to prescribe quinine’ | |
| A6 Other | The drug does not fulfil any of the criteria A1–5. Give further details | |
Acknowledgements
This work was funded by a North West Region National Health Service Postdoctoral Fellowship.
References
Australian Council for Safety and Quality in Health Care.
The President’s Advisory Commission on Consumer Protection and Quality in the Health Care Industry.
Department of Health.
Department of Health.
Medical Economics.
Joint Formulary Committee.
Audit Commission.
British Medical Association. Doctors’ Training and Qualifications: http://www.bma.org.uk/ap.nsf/Content/doctorsqualAccessed21May2004.
