February 18, 2006

Bronze Medal for BC's Health Performance

BC Health Minister George Abbott should read and attempt to understand the Conference Board's Report since he appears to be fond of quoting it. In question period on February 16th he said: "As we noted yesterday in question period, the Conference Board of Canada has found, after a very comprehensive examination, that British Columbia in fact has the best overall health care system in Canada." Not so fast, George! What the Conference Board did was arbitrarily average rankings of 70 measures (119 when some were broken down in greater detail). The Conference Board offered the following example of how it awarded its medals:

It assigned weights of 2 points for gold, 1 for silver and 0 for bronze, and then simply added the points for each of the measures. That means that life expectancy and infant mortality were given the same weight as exposure to environmental tobacco smoke. Second hand smoke is a big issue but it cannot begin to compare, let alone be given the same importance in a ranking of health systems, as infant mortality. To make matters worse, the methodology used by the Conference Board divided exposure to second hand smoke into three categories, at home, in public, and in cars, while infant mortality was divided into just two categories, male or female. That means exposure to second hand smoke received 50% greater weight than infant mortality.

While infant mortality and second hand smoke are used as examples here, any other pair-wise comparison could be selected to make the point that it is difficult, some would say impossible, to fairly aggregate indicators that measure completely different aspects of health or health-care. The World Health Organization's report on national health systems suffered from some of the same problems but at least it assigned lessor weights to things like wait times than it did to disability-adjusted life expectancy. The Conference Board's efforts treated wildly diverse indicators almost equally - as shown with smoking, some are included three times while others are included twice. Life expectancy by income is included for top, middle and low income brackets in the Report with no discussion of why that is better or worse than using income deciles.

The Conference Board's "study" divided the 70 overall measures into three broad categories, not for the purpose of assigning different weights to each category, but for the purpose of reporting separately on each of: 1) health status ranking, 2) health-care outcomes ranking, and 3) health-care utilization and performance ranking. It is in the third category that British Columbia finished second lowest. After reporting BC's low rating for patient satisfaction, the Conference Board's Report said: "These apparently contradictory findings bring into question the relationship between satisfaction rates and overall health services. Why are British Columbia's patient satisfaction scores low when it appears to have the best overall health performance in the country? It will take considerable research and effort to answer this question." Before applying for more research funding on that "difficult" question, the Conference Board might look at the glaring weaknesses of its methodology.

The Conference Board's 44 page Report devoted only three short paragraphs to "limitations of the methodology" and those paragraphs focused on difficulties related to using average indicators rather than distributions which might exposure differences between urban and rural areas. Nothing was said about the weakness of treating infant mortality with less weight than exposure to second hand smoke.

The data used in the Conference Board's flawed methodology comes from a federal-provincial agreement for all provinces to report in a consistent manner on a selection of measures. Those data can be found on the Health Canada website. Links to comparable provincial reports are available on a Statistics Canada website. The production of comparable health indicators is an important step for the analysis of health policy in Canada. Those data can be used or misused. It is arguable whether the Conference Board's Report would qualify for publication in respected peer reviewed journals. It is good for the provinces to report in a consistent manner, but ranking 119 indicators and adding them using arbitrary weights is worse than comparing apples and oranges; it is comparing pears and ball bearings.