Measurement & Quality Initiatives

Commonwealth Study Demonstrates "Public Reporting Matters"
2009 — 2010

With support from The Commonwealth Fund, researchers evaluated how publicly reporting healthcare performance information through Wisconsin Collaborative for Healthcare Quality (WCHQ) has impacted the quality of ambulatory care in Wisconsin.

The principle investigators were Geoffrey Lamb, MD of the Medical College of Wisconsin, Maureen Smith, MD, PhD of the University of Wisconsin Health Innovation Program, William Weeks, MD, MBA of The Dartmouth Institute for Health Policy and Clinical Practice, and Chris Queram of WCHQ.

This study investigated whether the public reporting of ambulatory care measures by WCHQ is associated with improvement in the delivery of recommended interventions and outcomes in Wisconsin.

The project study design has three main elements. First, the research team evaluated longitudinal quality results, reported by members of WCHQ, for a five-year time frame. The data in the study represented 13 quality measures related to diabetes, hypertension and preventive cancer screenings from 567 practice sites in 20 medical groups.

In addition to analyzing the reported data, members were surveyed to understand how public reporting of the measures affected their priorities for improvement and specific strategies they employed to achieve better performance.

And, lastly, the project compared the performance of WCHQ members with that of non-WCHQ-member physician organizations in Wisconsin as well as in two states (Iowa and South Dakota) that have no ambulatory public reporting programs.

By collecting and analyzing this data, the researchers addressed the following questions:

1. Has there been any improvement in performance on WCHQ reported measures?
2. How did the WCHQ members react to public reporting of measures?
3. Are there differences in quality improvement when WCHQ participation is compared to non-participation?

Has performance improved for the measures reported by WCHQ?

For WCHQ as a whole, there was significant overall improvement in 7 of the 8 measures that were implemented before 2005-2006 and, therefore, had at least 3 years of reporting. Figure 1 displays the year the measure was first reported, an indication of statistical significance, the number of years it took to get a significant improvement and the percent improvement.

Figure 1:

Year WCHQ measures were first reported, indication of statistical significance, number of years it took to get a significant improvement and percent improvement.

The study also evaluated the rate of improvement. Analysis of each member group demonstrated the following findings:

• No correlation between group size and rate of improvement
• Variable correlation with rate of improvement and the group decision to focus on that measure
• Strong correlation between the initial rank of a group compared to its peers and the subsequent rate of improvement (Figure 2)

Figure 2:

Relationship between rate of improvement and initial comparative ranking for Glycohemoglobin testing

In general, programs that were initially ranked quite low compared to their counterparts improved at a greater rate. This last finding, coupled with the demonstrated improvement in all but one of the measures, shows that "public reporting matters."

How did participating healthcare providers react to the public reporting of WCHQ measures?

The member survey demonstrated that it was common for WCHQ member organizations to focus on WCHQ measures during the study period. Every group reported formally giving priority for improvement to at least one WCHQ measure, in response to WCHQ reporting. Nine groups indicated their priorities were always or nearly always in response to WCHQ reporting, while seven showed a mix of responses, with five of those only occasionally choosing their priorities in response to WCHQ reporting.

While there are some missing data for the detailed questions on quality improvement implementation timing, the study obtained relatively complete information on whether WCHQ member clinics have implemented the activities asked about. Overall, there was a significant amount of activity in implementing systems and procedures to improve care quality and outcomes at the clinic level. The most common initiatives implemented by WCHQ members at care sites were adopting guidelines (85% - 87%) and patient reminders (76% - 82%). One-on-one diabetes education (81%) and providing diabetes data to primary providers (81%) were also very common.

Are there differences in quality improvement when WCHQ participation is compared to non-participation?

The Dartmouth Institute completed a comparative analysis of WCHQ member organizations versus the remainder of Wisconsin, Iowa/South Dakota and the rest of the United States, focusing on performance and rates of change for the diabetes related measures and mammography.

The study demonstrated that WCHQ member organizations outperformed the comparator groups, including the remainder of Wisconsin, nearby states of Iowa and South Dakota, and the rest of the United States in measures of glycohemoglobin testing, lipid testing in diabetics (Figure 3) and breast cancer screening (Figure 4) — all of which are publicly reported through WCHQ. The statistical data supporting the graphical displays can be found in Figure 5.

In each of these measures, there was a trend toward the rate of improvement during the study years being higher for WCHQ members, but this did not reach statistical significance. In contrast, Iowa and South Dakota patient populations were more likely to have received a diabetes related eye examination, which is not among WCHQ's publicly reported measures.

Patients of WCHQ members also tend to be somewhat more affluent and less likely to be on Medicaid than the comparison groups. This may create a bias in favor of better performance among WCHQ members. However, the performance of Iowa/South Dakota on the diabetic eye examination demonstrates that more is at play than just demographics.

Figure 3:

Performance on diabetes related measures for 2004-2007, using 20% Medicare sample (Adjusted for differences in age, gender, race and income)

Figure 4:

Performance on mammography screening for 2004-2007, using 20% Medicare sample (Adjusted for differences in age, gender, race and income)

Figure 5:

Statistical comparison between WCHQ and comparator groups

* P less than 0.05; Note that due to multiple comparisons a p value less than 0.017 required for significance.

** P less than 0.017 (multiple comparison adjusted significance value)

Conclusions

The three components of this study provide compelling evidence that public reporting of ambulatory measures led to sustained improved performance among WCHQ member organizations. Clearly this was not a randomized controlled study and, therefore, there are several potential weaknesses. The decision to join WCHQ is voluntary and, as such, the members are highly motivated. The decision by groups to focus on WCHQ measures for improvement and the apparently more rapid improvement among the lower performing practice groups strongly suggest that public reporting of comparative performance is a true driver for improvement. The findings are a nice example of the old management adage of "you manage what you measure."

This study clearly indicates the value of membership in an organization such as WCHQ, in public reporting of healthcare outcomes and working collaboratively to improve care.

Save/Print this page (PDF)

<< Back to Measurement & Quality Initiatives