Dr. Walter Himmel (the ‘walking encyclopedia of EM’) gave a fantastic talk from North York General’s Emergency Medicine Update Conference in Toronto, which has been edited with key commentary and summaries into a digestible podcast. Dr. Himmel eloquently shows us, through absolutely stunning personal cases, how evidence based medicine can be appropriately or inappropriately applied in real practice, resulting in major outcome differences for your patients. He elucidates the importance of clinical experience, patient values and ED resources in helping apply the medical literature to your practice. He reviews the essence of critical appraisal, the hierarchy of evidence and how to keep up with the emergency medicine literature. The famous NINDS thrombolysis for stroke trial is distilled down to a few key considerations and the NEJM transfusion for upper GI bleed trial from last year is dissected, analyzed and applied to Dr. Himmel’s personal cases, to help us understand exactly how to avoid BARF – the Brainless Application of Research Findings.
Blog post and written summary prepared by Keerat Grewal, edited by Anton Helman July 2014
Cite this podcast as: Himmel, W, Helman, A. Evidence Based Medicine from NYGH EMU Conference 2014. Emergency Medicine Cases. July, 2014. https://emergencymedicinecases.com/episode-47-walter-himmel-evidence-based-medicine-nygh-emu-conference-2014/. Accessed [date].
Evidence Based Medicine (EBM)
Three Spheres of EBM1
David Sackett: An integration of factors are required to help improve patient outcomes when using EBM
1) Best research evidence 2) Clinical expertise 3) Patient values
Fig 1: Spheres of EBM
Hierarchy of Evidence (see Fig 2)
Strength of evidence increases from:
Expert opinion, case reports, cohort studies, RCTs (which attempt to minimize systematic error) and systematic reviews / meta-analysis (a combination of trials and the highest level of evidence)
Another model of levels of evidence is the 6S model2 (see Fig 3). The quality of evidence increases from single studies; synopses of single studies; syntheses (e.g. Cochrane reviews, can include systematic reviews); synopses of syntheses (e.g. summary of a systematic review); summaries (e.g. evidence-based textbooks, clinical practice guidelines); systems (fully developed clinical decision systems)
Fig 3: Levels of Evidence – 6S Model
EBM asks specific, useful, efficient questions (PICO):
P: who is the patient, population, or problem of interest
I: what is the intervention
C: what is the comparison
O: what are the outcomes
Minimizing Error in EBM
consider sources of error, or deviation from the truth, of evidence:
Systematic Error: is BIAS, or non-random error (e.g. failure to blind, unconcealed allocation, patient drop out, etc.
Random Error: due to chance, unavoidable; revealed by consideration of statistical concepts such as p-values and confidence intervals
Critical Appraisal of Evidence
When assessing evidence/literature, ask yourself questions about the study and the results. Avoid BARF (Brainless Application of Research Findings). Critical appraisal of a study can be based on the JAMA User’s Guide to Medical Literature3:
- Are the results valid? Is it free from bias?
Were the patients randomized? Were patients in the study groups similar? Was the study blinded? Was follow up complete?
- Are the outcomes important? Is there significant clinical importance to the patient?
What were the results? How large was the treatment effect? How precise were the results? Are all patient-oriented outcomes considered (harms and benefits)?
- Are the results relevant to my practice?
Were the study patients similar to your population (inclusion criteria)? Is the setting similar? Are the treatment benefits worth the potential harm and costs? What are your patient’s values? Is it feasible in your institution?
To improve your critical appraisal skills of landmark EM articles go to the Global EM Journal Club on the Academic Life in EM blog. Note that coming soon will be the JOURNAL JAM podcast, an EM Cases – ALiEM – Annals of EM collaboration, that summarizes the Global EM Journal Club and interviews the lead authors of the landmark EM articles.
For excellent critical appraisal tools- Ken Milne’s Skeptics Guide to EM BEEM tool kit.
Examples of Application of EBM
Tissue plasminogen activator for acute ischemic stroke (NINDS stroke study group, NEJM, 1995)4
Two part study, part 1: assessed whether TPA had clinical activity (resolution of symptoms, improvement in NIHSS); part 2: clinical outcomes assessed at 3 months
Randomized trial (TPA vs. placebo), blinded
Results: 26% of placebo arm had good function at 3 months (based on modified Rankin score); vs. 39% of the TPA arm. However, 6% of the TPA arm had intracerebral hemorrhage (with about half of those patients dying)
When critically appraising this article, it is important to note that patients in the placebo arm were sicker than the TPA arm (NIHHS 15 vs. 14, respectively). This suggests that randomization failed. Moreover, patients were not consecutively enrolled. Given the findings of this study, incorporation of patient values in administration of TPA is very important (i.e. ICH/mortality risk vs. functional recovery).
Go here for an in depth discussion with Walter Himmel and Daniel Selchen on Stroke Controversies including a review of the major thrombolysis trials, and here for our Journal Jam podcast dissecting the world’s literature on thrombolysis for stroke.
Transfusion strategies for acute upper gastrointestinal bleeding (Villaneuva et al., NEJM, 2013)5
N = 900 with UGIB (melena or hematemesis)
No active heart disease, no LGIB, no massive bleeding, no stroke history
Patients had access to gastroscopy within 6hrs
Patients were randomized into one of two groups:
1) restricted transfusion group: blood transfusion for Hb <70; or
2) liberal transfusion group: blood transfusion for Hb <90
If patient had SOB, was symptomatic, looked unwell, patient received blood transfusion regardless of Hb
Results: Mortality in restricted transfusion group was 5%, in liberal transfusion group, mortality was almost doubled. Adverse events in liberally transfused group was significantly higher than those in the restricted group.
When critically appraising this article, note that patients who were unwell, unstable or symptomatic were transfused as per clinical judgment, not according to the group they were randomized to. Moreover, patients in this study had access to gastroscopy within 6 hours, which may be difficult to achieve depending on your local resources. This is a good example of the need to carefully assess the details of patient enrolment (does your patient fit the precise enrolment criteria?) and whether the circumstances of a study are applicable to your Emergency Department (can you get a gastroscopy at your hospital within 6 hours?).
Go here for an in depth discussion with Walter Himmel, Katerina Pavenski & Jeannie Callum on Transfusions, Anticoagulants & Bleeding, where they discuss the upper GI bleed trial as well as transfusions in patients with coronary artery disease, transfusion reactions and more.
Dr. Helman and Dr. Himmel have no conflicts of interest to declare.
Key References
- Sackett, D. et al. Evidence based medicine: What it is and what it isn’t. 1996. BMJ, 312: 71. Free Full Text available at: http://www.bmj.com/content/312/7023/71
- DiCenso, a. et al. Accessing preappraised evidence: Fine-tuning the 5S into a 6S model. 2009. ACP Journal Club, 151(3): JC2-3. Abstract available at: http://www.ncbi.nlm.nih.gov/pubmed/19755349
- Guyatt, G. et al. JAMA User’s Guide to Medical Literature, 2nd edition. 2008. JAMA. Free Full Text: http://jamaevidence.com/resource/520
- NINDS Stroke Study Group. Tissue plasminogen activator for acute ischemic stroke.1995. NEJM, 333(24): 1581-1587. Free Full PDF available at: http://www.nejm.org/doi/full/10.1056/NEJM199512143332401
- Villaneuva, C. et al. Transfusion strategies for acute upper gastrointestinal bleeding. 2013. NEJM, 368(1): 11-21. Free Full PDF available at: http://www.nejm.org/doi/full/10.1056/NEJMoa1211801
[…] Don’t “BARF” — or, (ir)responsible application of evidence-based medicine (EBM). […]
[…] Don’t “BARF” — or, (ir)responsible focus of evidence-based medicine (EBM). […]
I wonder if the editors or Dr. Himmel would comment on the (what I have heard) are wide confidence intervals for the thrombolytics stroke trial and what confidence intervals mean in evidence based medicine.
Great questions:
Confidence intervals: One can never be sure of any result. Therefore, if an incidence of a phenomena in a study is X, X is really just the best estimate of the truth and it is called the point estimate. The estimate is given a range within which the “truth” rests. This range is called the confidence interval and is described as the point estimate + or – a certain number. There are complex mathematical formulas used to calculate this. There is no doubt that the bigger the population studied, the smaller the confidence interval. Small confidence intervals are usually good unless a study is looking at meaningless outcomes.
A 95% per cent confidence interval (this is usually used) does NOT mean that the phenomena measured is found in the range 95% of the time [a common misconception]. What it means is that the “truth” has a 95% probability that is somewhere in that range.
In the original NINDS trial the confidence intervals were large. However, the treated groups did not overlap the placebo group. This means that there was a 95% probability that the study revealed the truth (at least in the patients they studied, with the outcomes they measured.)
Sadly, biological phenomena are very complex, highly variable, and their study is very complex due to multiple confounding variables. As well, depending on what you measure, you will get different results. Some will argue that the only reliable variable to measure is death. Mathematically, this is probably true. However, this is a deceptive simplification. Since there are realities worse than death and since values play a major role in the treatment of patients, other outcomes than death must be measured. Identifying what makes a clinically meaningful outcome is the unanswerable question and is the source of endless debate. For example, is a modified Rankin score of 2( two) so much better than3 ( three) that it is worth taking a risk of possible death. For me the answer is yes. However, I would rather be dead than significantly disabled … at least at the moment. If I felt differently and valued life far more than quality of life, I would probably chose a different treatment plan if I had s stroke.
As well , one must also define clinically meaningful harms. Although evidence-based medicine is an improvement over opinion, it is not the perfect answer to a question regarding quality of life. In the extreme, we may be looking at meaningless outcomes (that are statistically significant but clinically meaningless) and claiming great victories. This is the reasons that it is essential to look at the primary and secondary outcomes and to determine if they are important to you and your patients.
It gets sort of complex. None the less, is a great improvement from the 1970’s.
[Just to complicate the mess, some are arguing that with the new analytics (like GOOGLE uses to determine what to advertise to you), randomized, placebo controlled studies may be supersede by computers that evaluate thousands of data points for each person to give “meaningful” data for each individual patient. This will probably happen in the next several decades. But, human values and meaning will always play a role as long as we are allowed to have feelings and opinions.
I could go on and on……
Walter
Happy for any other questions or comments.