Canadian universities withdraw from Maclean's survey

The following is a letter that was sent to Maclean's magazine from university presidents across the country. It indicates their collective intention to withdraw from the magazine's annual university survey. Most of the country's leading research universities, including Dalhousie, are united in this action. The rationale for the decision is outlined below.

Also see:

• Questions and Answers
• Measuring Up: What university rankings do and don't tell us
  (Ottawa Citizen Op/Ed by David Naylor, President, University of Toronto)
• PDF version of letter [208 Kb]
  

To: Dalhousie Community From: Tom Traves Re: Canadian Universities Withdraw from Maclean's Survey I am circulating a letter from a group of Canadian university presidents that will be arriving at Maclean's magazine today indicating our collective intention to withdraw from the annual Maclean's university survey. Most of Canada's leading research universities are united in this action. The rationale for this decision is clear in the letter, but it boils down to our objection to the magazine's insistence on ranking universities based on arbitrary and deficient methodology. For many years, I have made this same point when the survey results appear. Maclean's provides a useful service in distributing detailed statistical information about different aspects of Canada's universities. However, when it lumps all these categories together into a single ranking, arbitrarily assigning more points to one category than another based on its own idiosyncratic judgment, it fundamentally misrepresents the character of every institution. For universities with a broad range of programs and degree options, such as Dalhousie, this is particularly objectionable. As the letter makes clear, the universities involved certainly are willing to discuss amendments to the magazine's approach, but to date, and over many years of discussion, these requests have not been accepted. Under the circumstances, my colleagues and I felt it necessary to take this step after careful consultation on campus and across the country. Tom Traves President Dalhousie University

August 14, 2006
 
Mr. Tony Keller
Managing Editor, Special Projects
MacleanÕs
11th floor
One Mount Pleasant Road
Toronto, ON  M4Y 2Y5

Dear Mr. Keller:

We regret to advise you that our universities will not be participating in the 2006 MacleanÕs questionnaire.  

We share MacleanÕs goal of providing good information for students and their families who are researching post-secondary education.  We also compliment you on your editorial coverage of the post-secondary sector.  Many of the articles in Maclean's have contributed to the national discussion about post secondary education, and have helped to frame studentsÕ choices.   Our concern relates specifically to Maclean's attempts to generate a global ranking of Canadian universities.

In various ways and for some years, many institutional spokespersons have expressed considerable reservations about the methodology used in the MacleanÕs university survey and the validity of some of the measures used.  Thus far, these serious concerns have gone largely unaddressed, and there is still no evidence that MacleanÕs intends to respond to them. 

We welcome public assessment of our work, and all our institutions devote significant resources to that end.  We already publish a great deal of data about our own institutions on-line and intend to publish more in future, ideally in the form of standardized datasets that will facilitate valid temporal and institutional comparisons.  However, it is truly hard for us to justify the investment of public funds required to generate customized data for your survey when those data are compiled in ways that we regard as over-simplified and arbitrary. 

Our concerns about MacleanÕs misuse of data in its rankings issue can be briefly recapitulated here. 

To begin with, the MacleanÕs rankings aggregate data from a range of variables related to the student body, class sizes, faculty, finances, library and reputation.  It is inappropriate to aggregate information across a range of programs at a large and multi-dimensional research university into a single ranking number. Consider how such an approach might pervert oneÕs understanding of a general hospital that is ranked #1 in obstetrics and #10 in cancer care. Averaging these rankings would result in this hospital being ranked Ò#5 overall”. For the patient seeking care in one of these areas, such a measure would be useless at best and misleading at worst. This is, effectively, the method that MacleanÕs applies to Canadian universities by its calculation of Òleague tables” based on the arbitrary assignment of weights to variables which, by themselves, are of questionable validity.  The variables selected by MacleanÕs also fail to capture the breadth of experiences students say are important in their university education such as, for example, extra-curricular activities or the opportunity for rich and diverse interactions with peers and faculty outside the classroom.

We are also concerned by MacleanÕs recent attempt to draw comparisons of student experience across incomparable surveys of student engagement, and Maclean's reliance on survey data with low response rates and all the associated response biases that arise from skewed profiles of respondents.  The responsible compilation and comparison of data is a core tenet of academic research. Several universities already show student survey data, in context, on their own web sites and question Maclean's decision to pull different kinds of data out of context and compare Òapples and oranges”. MacleanÕs treatment of these survey data, in our view, fails to give appropriate notice to these methodological limitations.

It is not just the MacleanÕs student survey that has suffered from low response rates.  Equally troubling is the fact that a clear majority of individuals who receive the Maclean's reputational survey do not respond.  This is a particular concern as the results of the reputational survey not only affect rankings in a significant way, but are given prominence separately by your magazine.

This is only a partial accounting of the methodological flaws in the MacleanÕs rankings.  In short, the ranking methodology used by MacleanÕs is oversimplified and arbitrary.  We do find it ironic that universities are being asked to subsidize and legitimize this flawed methodology, when many faculty, staff and students at our institutions are dedicated in their research to ensuring that data are collected rigorously and analyzed meticulously.

We remain open to the possibility of collaborating with MacleanÕs at some future date, particularly if we can agree on means to ensure that the data will be valid and the analyses truly informative.  Meanwhile, we will continue to publish data on our websites to facilitate informed student and family choice. 
 
Yours truly,

Tom Traves, Dalhousie University
Peter George, McMaster University
Michael Stevenson, Simon Fraser University
Indira Samarasekera, University of Alberta
Stephen Toope, University of British Columbia
Harvey Weingarten, University of Calgary
William Cade, University of Lethbridge
Emoke Szathmary, University of Manitoba
Luc Vinet, University of Montreal
Gilles Patry, University of Ottawa
David Naylor, University of Toronto