Why It Matters
What does it take to bring about improvement? Every evaluation of a QI project aims to answer this fundamental question.
The Model for Improvement comprises three questions that guide quality improvement (QI):
- What are we trying to accomplish?
- How will we know that a change is an improvement?
- What changes can we make that will result in improvement?
While the questions are simple, challenges in our daily work lives can prevent teams from answering them. Some may even wonder, “What’s the use? Patients are waiting and we don’t have time, so let’s get going!”
But we know from our own experience that bypassing them can lead to even more frustrating questions down the road: Did we achieve our goals? How effective was this intervention? Under what conditions? Were our predictions correct? What did we learn? Should we keep doing this? Should this intervention be abandoned, adapted, implemented, or scaled up?
The Impact of Unanswered Questions
For people on improvement teams, lacking answers to these questions can lower their sense of achievement and feel demoralizing. For the health care improvement community, lacking answers means losing valuable knowledge that could enhance others’ work. Most important, for patients, lacking answers means quality and safety improve at a slower pace. Ultimately, addressing the three Model for Improvement questions helps us explore a much bigger, albeit underlying question: What does it take to bring about improvement?
Every evaluation of a QI project aims to answer this fundamental question. As part of IHI’s Results and Evaluation team, we seek to give teams the tools to answer this question and to help them regularly reflect on their learning (also known as evaluation). We want to make sure that the project itself is undergoing continuous improvement.
Setting a Foundation for Ongoing Learning: Five Core Components
Gareth Parry, Senior Scientist at IHI, has led the way in changing IHI’s view of evaluation — promoting it as a tool for accelerating improvement. Building on his 2010 IHI innovation project and years of testing new systems internally, Gareth and other colleagues published Recommendations for Evaluation of Health Care Improvement Initiatives in 2013. Since then, we’ve continued to test these concepts in our own work. Learning from these tests, our Results and Evaluation team has identified five “core components” that we believe enable projects to demonstrate results and produce learning.
These include a(n):
- Aim statement (measurable goals)
- Content theory (driver diagram and/or change package)
- Execution theory (logic model)
- Measurement plan (data, results and learning)
- Dissemination plan (publication and communication)
(You can refer to a fuller explanation of the core components in a chart at the end of this post).
In order to evaluate a project, we must have an initial record of these components, however imperfect, before the improvement work begins. We often encourage our teams to, “Just get the first bad draft out!” From there, teams can work together to finalize their goals, how they plan to reach them, how they’ll know, and how they can share learning within their organization and across the larger QI field.
This process and documentation offers a snapshot of where the project started so a team can observe change over time. In addition, we must also consider how to observe changes over time and best utilize the core components.
Formative vs. Summative Evaluation
One approach involves waiting for the end of an initiative to answer what impact it had and collect learning to inform the next phase of improvement. This is known as summative evaluation. A long history of program evaluation and health services research is based on this before/after approach.
Another approach, known as formative evaluation, takes a different view of the role evaluation can play in improvement. This approach involves regularly reviewing progress toward goals, assessing what appears to be working and why, and using this knowledge to make adjustments that could accelerate improvement going forward. Formative evaluation methods are closely aligned with small-scale testing and practical learning — fundamental tenets of QI.
We are working diligently to build rapid-cycle, formative evaluation directly into our programs at IHI, and would like to help you do the same. Anyone, anywhere can take simple steps that enable a QI project to:
- Track progress towards medium- and long-term goals
- Learn what it takes to bring about improvement
- Make informed, evidence-based adjustments during an initiative to maximize improvement
Alexander (Sandy) Cohen, MPH, MSW, is a research associate at the Institute for Healthcare Improvement. Amy Reid, MPH, is a senior research associate at the Institute for Healthcare Improvement.
You may also be interested in: