This is a guest post by Sara Mayo, Social Planner with Social Planning & Research Council of Hamilton. This is on the eve of the first release of data from the National Household Survey, which replaced the long-form census in Canada. The original post can be found here.

SPRC reserves judgement on NHS data until questions are answered about data quality

By Sara Mayo, Social Planner

On May 8, Statistics Canada will be releasing the first data from the 2011 National Household Survey (NHS). Major new data releases are usually hotly anticipated and a busy time for social planners, but the Social Planning and Research Council of Hamilton has decided that we will not be commenting or reporting on the new NHS data at that time, including trends on Immigration, Aboriginals, Ethnic Origin, Language and Religion that will be released by Statistics Canada.

Since the news of the cancellation of the 2011 Census long form, the SPRC has been clear that the voluntary NHS is a poor replacement. The cancellation of the 2011 Census long form was a hasty decision by the federal government, without any notice or consultation with Canadians, and the negative effects will be felt for years to come. At a community forum on the Census that we organized  in September 2010, Code Red for Neighbourhood Data, organizations across Hamilton voiced their concerns that the cancellation of the long form would harm planning for new schools, immigration settlement services, poverty reduction strategies, youth programming at community centres, services for working parents, and neighbourhood improvements, among many others.

While NHS response rates have been better than the low benchmark of 50% set by Statistics Canada, they are still much lower than for the Census long form. 94% of households completed the Census long form in 2006, while the average NHS response rate for Canada was 68.9% (67.6% for Ontario). In the city of Hamilton, only 65.3% of households who received the survey answered it.

Statistics Canada’s own estimates show that NHS survey respondents are not representative of the population. Their analysis shows that raw NHS data from the Toronto Census Metropolitan Area overreports people earning a high income, those who are university educated, Canadians who work in the public administration industry, or in business, finance or administrative occupations and Chinese Canadians. Conversely, raw NHS data from the Toronto region undercounts the number of low income Canadians, the number without a high school diploma or less, those working in construction and transportation, Black Canadians, non-citizens, Aboriginals, and those who have moved in the last year. Accurate estimates of these populations are critical to social planning.

Our own analysis of response rates by Social Planning Network of Ontario communities (Table 1), shows a general trend whereby communities with low NHS response rates usually have higher poverty rates. The major outliers to this trend are the cities of Toronto, Markham, and Mississauga and the town of Richmond Hill, which all have higher than average poverty rates, yet higher than average NHS response rates. This may be explained in part by the higher NHS response rate of Chinese Canadians (who are a large portion of residents in these cities) as reported by Statistics Canada.

Table 1: Social Planning Network of Ontario Communities, NHS response rate and poverty levels

Data source: National Household Survey: Final response rates (Statistics Canada)

Statistics Canada has not yet indicated which mitigation strategies they have applied to adjust estimates to compensate for lower response rates among these critical groups. They have also not answered our questions about whether they will be releasing NHS data at lower levels of geography (such as census tracts and dissemination areas). Given the geographic concentration of many demographic groups with low response rates, we are quite concerned that neighbourhood-level data will be very unreliable.

We are especially dismayed that detailed information on data quality in the NHS will not be published until May 8^th. As part of a conference call with members of the Community Data Program on October 18 2012, Statistics Canada representatives informed data users that they would be publishing a paper on NHS data quality issues in December 2012. Requests to find out why that document has been delayed have not been answered. When it is finally released, we will examine this paper to better understand data quality issues with the NHS.

At the same time, we want to make clear that we continue to support Statistics Canada staff, and the agency as a whole,  who are facing budget cuts that undermine their ability to obtain reliable data about Canadians. High quality Statistics Canada is used in communities across the country to plan for the population’s needs and to improve quality of life.

Starting on May 8^th, the SPRC  will take the time to read Statistics Canada’s analysis of NHS data quality and mitigation strategies, and consult with other data users across the country about their judgment on the reliability and comparability of this data. We will update the community when we have formed an opinion on what uses of NHS data will be most prudent for Hamilton.