WTBS 11 – Keeping Score: Providing Physician Feedback

This is Waiting to Be Seen 11 on EM Cases – Keeping Score: Providing Physician Feedback

Written by Amy Cheng and Howard Ovens, edited by Howard Ovens and Lucas Chartier, December 2016

Keeping Score: What Is the Best Way to Provide Physician Performance Feedback?

As a department chief I occasionally obtain data on individual physician performance in my group: patients seen per hour, revisit rates, consult rates, etc. I always check how my score compares with the rest of the group—not necessarily because of my competitive side, but mostly to reassure myself that I still measure up as I work part-time clinical and push ever closer to the usual retirement age. Young doctors, actually all curious and responsible physicians, likely feel the same way.

I find most of the members of my group crave objective feedback. When they learn they are an outlier in some important way, such as the pace of seeing patients, imaging and consult rates, or complaint rates, they usually want to understand why and learn how to bring themselves back to the middle of the curve. In general, consistency is a virtue in an ED; nurses and consultants appreciate knowing what to expect from us, and anything that increases consistency—order sets, checklists, protocols, and even physician scorecards—allows team members to anticipate what the next steps are and prepare for them, and thus act cohesively.

What does the evidence say about the true utility of performance feedback and scorecards? Do they meet a real need for information to guide self-improvement or just scratch our competitive itches? What do we know about the best way to provide feedback? In this month’s guest blog Dr. Amy Cheng, the Emergency Department Director of Quality Improvement at St. Michael’s Hospital in Toronto with an interest in physician performance feedback, reviews what’s known and comments on her own experiences.

—Dr. Howard Ovens

The Physician Scorecard: Using Physician Feedback to Improve Quality of Care

I still recall the anxiety I experienced during my first year of practice. On the drive home after every shift, I often wondered whether I had done the right thing or had done a good job. Had I made the right decision by sending that elderly patient with pneumonia home? Should I have done that CT head on the toddler who had fallen and hit his head? Had I seen patients quickly enough? What would my other, more experienced colleagues have done in similar situations?

As residents, we receive feedback after every shift. Feedback to trainees helps identify knowledge gaps and other areas for improvement. Once we become staff physicians, however, we receive little to no feedback. Instead, physicians are left to judge their performance based on their patients’ outcomes and benchmark their performance against that of their peers. Without sufficient data, it is hard for individual physicians to identify gaps in knowledge and performance to improve patient care. The lack of information about how I was doing added to my angst of being a new attending and made it difficult for me to know what priorities to follow in advancing my skills.

Emergency physicians provide episodic care to a diverse patient population. Receiving regular feedback on the quality of our care is extremely challenging: We have little follow-up of our patients and we see relatively few cases of individual diagnoses. Most of the feedback we receive is anecdotal, such as the dreaded “Remember the patient you handed over to me the other night?”

A recent Cochrane Review shows that physician audit and feedback can lead to small but important improvements in professional practice.[1] Groups ranging from individual emergency departments to physician regulatory bodies in the United States, the United Kingdom, and Canada are now providing physician feedback as part of physicians’ maintenance of certification processes, and as a way to evaluate and monitor the quality of patient care.[2-5]

I will present here how physician scorecards can be used for physician feedback and as a tool for quality improvement in emergency medicine.

Individual performance feedback

Feedback is defined as a summary of a physician’s clinical performance over a specific time period.[1] For feedback to be effective at the individual level, the focus should be on quality measures that are clinically relevant to and within the control of individual physicians. The collection of quality measures should also cover the breadth of clinical practice in domains, such as clinical processes, clinical outcomes, patient experiences, and resource utilization.[6] The focus of the feedback should be quality improvement and not solely performance evaluation, as feedback that is delivered with a punitive tone is less effective.[1]

Group performance data should be provided for comparison; small improvements in performance can result when peer comparison data are added to individual physician feedback, and benchmarking to group performance is important for personal goal setting.[7, 8] However, how the peer data should be presented is unclear. One study showed that comparing individual physicians to the top 10th percentile resulted in a higher level of performance than when individual physicians were compared with the peer average, while another found that identifying the top performers in a group quality educational round did not increase the effectiveness of the feedback.[9, 10]

The scorecard

Of the many methods used to deliver individual physician feedback, the “360” and the scorecard approach have recently received a lot of attention.

The 360 is a feedback method where multiple individuals from different roles (e.g., patients, nurses, other physician colleagues) evaluate the individual physician subjectively. The scorecard, on the other hand, is a concept borrowed from business strategic performance tools that allows objective and subjective data from multiple domains of a physician’s performance to be presented simultaneously. Examples of metrics on the emergency physician scorecard include ED length of stay, unexpected 72-hour return visit rate, and CT ordering rate. Patient satisfaction scores may also be added to the scorecard to provide a subjective evaluation of the physician’s performance. The specific data presented on the scorecard are usually determined by the leadership team, and are selected to be relevant to the medical specialty and the local context. It is important, however, to involve physicians’ input early to achieve group buy-in.

Depending on the metrics chosen, most of the scorecard data are abstracted from electronic medical records (EMRs) and reported on a pre-specified time period (e.g., quarterly), or in “real time” (as a clinical dashboard) depending on the robustness of the EMR and the outcomes measured.

Delivering physician feedback     

The method of feedback delivery is an important factor in the value of the data. The Cochrane meta-analysis mentioned previously suggests physician feedback and audit are most effective when the feedback is provided by a physician who is a supervisor or a more senior colleague; is delivered frequently (e.g., weekly); uses both a verbal and a written format; and offers support for improvement in the form of goal setting and action planning.[1] These are suggestions of course, and individual departments should consider pragmatic approaches when implementing their own feedback systems.

In their book Thanks for the Feedback, Harvard University researchers Douglas Stone and Sheila Heen caution against using a one-size-fits-all approach to delivering feedback. The feedback delivery method should be tailored to what the specific feedback recipient wants from the session (e.g., appreciation, coaching, evaluation) and the feedback giver/recipient relationship.[12]

New staff members generally need a balance of evaluation and coaching, as it is important for them to know where they stand before they can receive advice on how to improve. That balance may change over the course of a physician’s career. More senior physicians may just want to know that all their years of hard work are appreciated. When physicians who want appreciation are coached and evaluated instead, the feedback session may be taken as criticism and the physicians are left feeling underappreciated and with decreased morale.

The role of the chief in physician feedback

The chief often takes on the role of the feedback giver. It is important for the chief to balance the dual purposes of the feedback: quality improvement and evaluation. Instead of punishing the low performers, the chief should identify and help these individuals gain insights into their practices, set explicit goals, and construct action plans for improvement.

However, not all physician behaviours are amenable to improvement using physician feedback. The Cochrane Review cited earlier showed that feedback may be more suitable for influencing simple physician behaviours, such as reducing the number of prescriptions the doctor writes, rather than complex physician behaviours, such as increasing compliance with clinical guidelines in managing specific diseases.[1]

Designing a specific action strategy for improvement is also challenging, as not all physician behaviours allow for clear improvement solutions. Deficiencies in knowledge in a specific clinical area can be improved by attending courses and conferences, and deficiencies in a particular skill can be improved by shadowing “better performers.” At the Martini Klinik, a German centre that focuses on prostatectomies, surgeons who are slower and/or have higher complication rates are asked to scrub in with colleagues who are faster and/or have lower complication rates to improve their techniques.[13] In some emergency departments physicians who see the fewest patients per hour are asked to do a few shadow shifts with physicians who see the most patients per hour. The success of the shadowing solution, however, is highly dependent on the departmental culture and providers’ willingness to participate.

While many educational interventions are available to enhance physician practice (e.g., reminders, educational outreach, small group learning) the literature does not offer any helpful insights into which are most effective. Therefore, it is ultimately up to the chief and the physician being assessed to determine which intervention would be the most suitable.

Final thoughts on physician feedback

Physician feedback can lead to small improvements in professional practice.[1] The physician scorecard is a popular method for physician feedback due to its ability to simultaneously present data on multiple domains of a physicians’ practice. Characteristics that enhance the effectiveness of physician feedback include: using a feedback provider who is a supervisor or a senior physician colleague; delivering feedback in a manner that matches what the recipient wants from the feedback (appreciation, coaching, or evaluation); delivering feedback frequently; delivering feedback in verbal and written formats; and setting goals.

Despite its popularity in emergency medicine, the use of scorecards for quality improvement should be done cautiously. Most of the literature on the effectiveness of physician feedback has been on single outcomes on a specific clinical diagnosis, or on the reduction of specific laboratory or radiological testing. Little is known about the effect of physician scorecards and feedback on processes (e.g., emergency department length of stay) and safety measures (e.g., unexpected 72-hour emergency department return rates) that are commonly used in emergency medicine. Furthermore, while goal setting and action plans are critical during feedback delivery, little is known about the types of specific educational methods that can help achieve improvement.

As an emergency physician, I strongly support the use of scorecards because I value knowing how my performance compares with that of my peers. As the emergency department Director of Quality Improvement, however, I recognize that designing an effective feedback strategy is challenging due to the paucity of literature on this topic. As with many quality improvement and administrative initiatives, there is little sharing of lessons learned between departments, and we are often left to “reinvent the wheel.” I hope this blog will help start a dialogue on our experiences with scorecards and allow physicians to share lessons learned and challenges faced so that collectively we can improve both the physician feedback process and the care of all of our patients.

—Dr. Amy H.Y. Cheng is the Emergency Department Director of Quality Improvement at St. Michael’s Hospital in Toronto, Ontario, Canada.

A quick guide to developing a physician scorecard for your emergency department

1. Ask the physician group for help in selecting quality indicators that are meaningful to their practices

The following guiding principles can help:

• Indicators should be primarily under the control of individual physicians
• Reliable and timely data are readily accessible
• Evidence-based quality of care indicators are a good starting point for generating an initial list of metrics[14]

2. Consider peer comparisons

• Peer comparison data may help (e.g., group average or median)

3. Include a physician data check to ensure face validity of the data
4. Keep the individual physician data anonymous and have the chief deliver the feedback
5. Present data frequently

• Feedback should be presented frequently, but within the limits of practical constraints

6. Deliver feedback in both written and verbal formats
7. Ensure feedback sessions involve goal setting and action planning

Download a pdf of ‘A quick guide to developing a physician scorecard for your ED’ HERE

References

1. Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. The Cochrane database of systematic reviews. 2012;6.
2. American Board of Internal Medicine – Practice Assessment. 2014. Accessed July 25, 2014 from the world wide web. http://www.abim.org/maintenance-of-certification/requirements/practice-assessment/default.aspx
3. General Medical Council – Supporting information for apprisal and revalidation.
4. Physician Achievement Review (PAR). In: Alberta TCoPaSo, editor.
5. A Primary Care Performance Measurement Framework for Ontario: Report of the Steering Committee for the Ontario Primary Care Performance Measurement Initiative: Phase One. Toronto: Health Quality Ontario; 2014.
6. Shaller D, Kanouse D. Private “Performance Feedback” Reporting for Physicians: Guidance for Community Quality Collaboratives. Rockville, MD: Agency for Healthcare Research and Quality; 2012.
7. Sondergaard J, Adersen M, Vach K, Kragstrup J, Maclure M, Gram L. Detailed postal feedback about prescribing to asthma patients combined with a guideline statement showed no impact: a randomized controlled trial. European Journal of Clinical Pharmacology. 2002;58:127-32.
8. Wones R. Failure of low-cost audits with feedback to reduce laboratory test utilization. Medical Care 1987;25:78-82.
9. Kiefe C, Allison J, Williams O, Person S, Weaver M, Weissman N. Improving quality improvement: using achievable benchmarks for physician feedback: a randomized controlled trial. JAMA. 2001;285:2871-9.
10. Schneider A, Wensing M, Biessecker K, Quinzler R, Kaufmann-Kolle P, Szecsenyi J. Impact of quality circles for improvement of asthma care: results of a randomized controlled trial. Journal of evaluation in clinical practice. 2008;14:185-90.
11. Kaplan RS. Conceptual Foundations of the Balanced Scorecard. Harvard Business School; 2010.
12. Stone D, Heen S. Thanks for the Feedack: Penguin Group; 2014.
13. Porter ME, Deerberg-Wittram J, Marks C. Martini Klinik: Prostate Cancer Care. Harvard Business School Case 714-471: Harvard; 2014.
14. Schull MJ, Guttmann A, Leaver CAV, M, Hatcher CM, Rowe BH, Zwarenstein M, et al. Prioritizing performance measurement for emergency department care: consensus on evidence-based quality of care indicators. CJEM. 2011;13:300-9.