Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

League Tables as Policy Instruments

League Tables as Policy Instruments international higher education 2 international issues University (SJTU), CHE presents detailed survey data from League Tables as Policy thousands of teachers and students as well as third-party data dealing with the universities and technological institutes sepa- Instruments rately. One can look at the standing of each university or even Jamil Salmi and Alenoush Saroyan a specific academic subject, based on a particular indicator or set of indicators. Readers can even create their own ranking Jamil Salmi is coordinator of the World Bank Tertiary Education Thematic based on the indicators most relevant to them. The approach Group. Alenoush Saroyan is professor in the Department of Educational developed by CHE presents the additional advantage of avoid- and Counseling Psychology at McGill University, Montreal, Canada. The ing data biases linked to self-reporting by universities. findings, interpretations, and conclusions expressed in this article are entire- Austrian and Swiss universities have recently joined this exer- ly those of the authors and should not be attributed in any manner to the cise, accepting benchmarking against German universities. World Bank. E-mail: [email protected]. Pakistan n 1963, at the University of California at Berkeley, faculty The ranking of universities in Pakistan developed out of a Iand the administration objected when the radical student direct mandate given to the Higher Education Commission newspaper on campus, Cal Reporter, took the initiative to pub- (HEC) in 2002 to evaluate the universities in a way that pro- lish student evaluations of courses and professors. Despite this moted rapid and comprehensive development of the entire ter- initial resistance, student evaluations have become part of tiary education system, particularly to support the country's many universities' internal accountability mechanisms. place in the world economy. By comparing the inputs and out- Demands for increased accountability come not only from stu- puts of the nation's institutions, Pakistan has established a dents but also from other stakeholders—such as governments, mechanism for rewarding excellence and investing in the employers, and the public at large. Institutional rankings by strengthening of institutions that need improvement. The five league tables constitute one way of meeting the keen interest in information about the quality of universities. The expansion of league tables and ranking exercis- League tables use a wide range of indicators to measure the es has not gone unnoticed by the various stakehold- system's setup (input variables), function and efficiency ers, and the reaction they elicit is rarely benign. (process variables), and productivity and impact (output vari- ables)—relative to the performance of other universities and programs. The expansion of league tables and ranking exercises has main ranking criteria used by HEC (faculty qualifications, not gone unnoticed by the various stakeholders, and the reac- research output, students, facilities, and finances) are similar tion they elicit is rarely benign. In some countries (Argentina, to ranking indicators in many other nations. Nigeria, and India), the ranking exercise is undertaken as part The advisory committee overseeing this ranking exercise, of the accreditation process. With leagues tables becoming comprised of HEC officials and university representatives, had something of a growth industry, their accuracy, relevance, and to decide whether the results should be made public or not. usefulness have become issues of concern. Do they have any Under vehement protests from one of the leading vice-chancel- beneficial use for public policy, accountability, and consumer lors, whose (public) university had scored very low, the com- information? This article examines ways in which league mittee agreed not to publish the results. What HEC did instead tables have been used internationally to enhance quality. was to share key benchmarking data with each university, notably its relative position against each criterion used in the Germany rankings. For example, university X was told that, with respect In 1990, after the fall of the Berlin wall, teams of academics to the proportion of professors with a doctorate, it scored in the from the West German Science Council were given the task of lowest quartile compared to all universities in Pakistan. evaluating their counterparts at East German universities. It Despite the general outcry against the publication of the rank- was realized that in the absence of a tradition of evaluation in ings' results, this experience has forced the universities to take W data collection much more seriously est-German universities the mission would call for inventing . an appropriate methodology. More recently, the ranking exer- The Germany and Pakistan examples illustrate that, in cise conducted since 1998 by the Center for Higher Education countries without established evaluation or accreditation Development (Centrum für Hochschulentwicklung—CHE), mechanisms, rankings can be used effectively to monitor and an independent policy research agency, has become the first enhance quality. Finally, it is interesting to note that rankings comprehensive system providing a panorama of quality indica- are not used only by governments in their national context but tors in Germany. Instead of calculating a global ranking of also increasingly in an international perspective. In M ongolia institutions based on weighted indicators as do the Times and Qatar, for instance, the authorities have decided to restrict Higher Education Supplement and Shanghai Jiao Tong scholarships for studies abroad to students admitted in a high- international higher education international issues 3 ly ranked university. assessment test intended to compare the quality of undergrad- uate programs across universities. It could be described as a Public Initiatives ranking exercise categorizing university programs based on The press is often criticized for using rankings as a gimmick to the average score of their participating students. While the test boost sales. However, the mass media can play a genuine edu- (Provão) at first met with a lot of resistance, over time it cational role by making relevant information available to the became more accepted and increasingly influenced students in public, especially in countries lacking any form of quality their choice of tertiary institution. assurance mechanism. In Poland, when the transition to the Nigeria market economy started in the early 1990s and many private Similarly, in Nigeria, after the National Universities Council education institutions began to operate, there was a thirst for initiated a ranking of professional programs in 2001, even information about the quality of these institutions, which Perspektyvy magazine to initiate the going as far as closing down a number of programs, private- pushed the owner of the country's first ranking. Similarly, in Japan for many years the sector employers started to regain confidence in local universi- Asahi Shimbun newspaper annual ranking published by the ties and to hire graduates of the highest-ranked programs. fulfilled an essential quality assurance function, in the absence of any evaluation or accreditation agency. The Way Forward Governments and the public at large are increasingly preoccu- Colombia pied with the relative performance of tertiary education institu- Colombia was the first country in Latin America to set up a tions. Some countries are striving to have “world-class univer- national accreditation system in the mid-1990s, but the num- sities” that will spearhead the development of a knowledge- ber of programs reviewed by the new accreditation agency based economy. Others, faced with a shrinking student popu- remained relatively low in the first years because the accredita- lation, struggle to attract more fee-paying foreign students. tion law made the process voluntary and the most prestigious Just as scarcity, prestige, and excellence dominate the purchase universities, public and private, did not feel any compulsion to of consumer goods, students are also looking for indicators to participate. After the country's main newspaper, El Tiempo, identify the best universities. Thus, regardless of their contro- published in 2000 the full list of accredited programs to help versial nature and methodological shortcomings, university students choose among the various offerings, the universities rankings are unlikely to disappear. have felt increasing pressure to join the accreditation process. The following general recommendations, developed through our analysis of the recent international experience France with league tables, can help make the ranking exercise benefi- Another important merit of rankings is to stimulate public dis- cial to institutions, governments, students, parents and the cussions around critical issues affecting the tertiary education public at large. system. A good example is the debate that started in France Clarify what the ranking measures. Despite the ambiguities when the SJTU world rankings were published for the first surrounding the construct of quality, the rankers should speci- time. After observing that the best French university was fy what they do or do not measure, the goals, and the audiences ranked 65th, the daily paper Le Monde ran an article, “The for whom they do the ranking. The validity, reliability, and Great Misery of French Universities,” that started a widespread comprehensiveness of selected indicators can be discerned in discussion. One of the principal problems identified was the light of these specifications and by examining the scope of the fact that French universities are not allowed to select the most academic tasks (e.g., teaching, research, etc.) and the types of institutions being assessed. Furthermore, the rankers should make the raw data on which they base the ranking widely avail- The mass media can play a genuine educational able and the calculation process transparent so that their rank- ings can be verified independently. role by making relevant information available to Use multiple indicators and measures rather than a single, the public, especially in countries lacking any form weighted ranking. League tables should use a wide range of of quality assurance mechanism. indicators, placing greater emphasis on output and outcome indicators to ensure that every dimension of quality gets fac- academically qualified students, unlike the practice in more tored in the evaluation. Multiple sets of indicators will yield successful university systems in the United States, the United multiple scores rather than a global score, thus bringing to Kingdom, or Japan. The other important factor is the absolute light areas of strengths as well as areas of weaknesses. lack of competition among universities. Compare similar programs or institutions. Because of their methodological limitations, rankings are more meaningful Brazil when the unit of comparison is smaller. Ranking programs is In 1996 the Ministry of Education in Brazil introduced an thus preferable than ranking institutions. international higher education 4 international issues At the institutional level, use rankings for strategic planning and investors, managerial capacity, and other professional quality improvement purposes. Tertiary education institutions resources. that look at detailed ranking data for benchmarking purposes A second way universities can support economic develop- can use specific indicators to understand the determinants of ment is providing updated technical knowledge and skills in their performance and then work toward improving the quali- the workforce. Though less “sexy” or highlighted, universities ty of teaching and research. can play this important role at all levels. Not only do a small At the government level, use rankings to stimulate a culture of number of R&D scientists need their knowledge to be updated . In countries that do not yet have a well-established but also a large number of midlevel technical professionals quality quality assurance system, rankings can be used as a proxy for and even larger numbers of technical workers. One global quality. Similarly, at the international level, in the absence of a optoelectronics company in Japan had a well-developed inter- single global quality assurance agency, ranking systems take national network with cutting-edge scientists for its R&D activ- on some characteristics of a quality regulator for international ities. The company also appreciated its less visible ties with students. Use rankings as one of the instruments available to inform stake- Scientific discoveries leading to industrial innova- holders. Rather than being considered as the ultimate measure tions, particularly through academic spin-offs, pro- of quality and/or relevance, rankings should be complemented vide a classic image of universities contributing to by information on accreditation and labor market outcome data collected through surveys of employers and tracer surveys. economic development The results of league tables can also serve to generate a nation- al debate about long-term strategic priorities and policies for tertiary education. academics at local universities because it was these relation- ships that impacted a large number of company employees and helped them keep abreast of scientific progress. There is a complementary role in which academics under- take joint research, consultancy, or contract work with industry The Role of Higher Education to help address various technological problems. This interac- tion with industry allows academics to learn about industrial needs, but the relationships inform them equally about what is Institutions in Innovation and relevant. Universities can play a less direct yet important economic Economic Development role by setting forth the social, cultural, and intellectual tone of Sachi Hatakenaka a local area. Cultural events surrounding universities can make Sachi Hatakenaka is an independent consultant and researcher on higher a place more attractive to educated professionals and their education policy and management and is currently affiliated to the employers. The University of Pennsylvania goes further and Industrial Performance Center at the Massachusetts Institute of Technology makes an institutional commitment to community develop- for her research. E-mail: [email protected]. ment in its neighborhood. Other roles relevant to economic development include con- tributing to government and industrial decision making by everal decades of debate and experimentation are leading to participating in key committees and advisory boards and San understanding worldwide that many (but not all) univer- expanding local economic activities. sities can play diverse but important roles in innovation and economic development. Relevance as a University Mission In many countries, higher education institutions are or aspire Diverse Roles to be focused on academic research with little practical orienta- Scientific discoveries leading to industrial innovations, partic- tion. There is also a strong tendency for academic and research ularly through academic spin-offs, provide a classic image of drif t. Institutions with a practical orientation often become universities contributing to economic development (e.g., more academic, and teaching-focused institutions become Silicon Valley and Route 128). Many research institutions more research oriented. One reason for these changes has struggle to replicate this; universities may not have the culture been the dominant interpretation of “scientif ic autonomy,” or capability to do so. Yet some universities surround them- which was often interpreted to encourage isolation from exter- selves with a local industrial community that can absorb and nal stakeholders. make use of scientif T ic discoveries to accrue economic benefits. oday it is seen as possible for scientists to undertake fun- damental research while being motivated by practical rele- For spin-offs to be successful, the university must reside with- vance—rather like Pasteur, the French biologist known for his in a supportive ecosystem with technology-aware financial http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Higher Education Management Unpaywall

League Tables as Policy Instruments

Salmi, Jamil; Saroyan, Alenoush
Higher Education Management – Jun 1, 2007
Download PDF
3 pages

Loading next page...
 
/lp/unpaywall/league-tables-as-policy-instruments-5JgdFvWyOo

References

References for this paper are not available at this time. We will be adding them shortly, thank you for your patience.

Publisher
Unpaywall
ISSN
1013-851X
DOI
10.1787/hemp-v19-art10-en
Publisher site
See Article on Publisher Site

Abstract

international higher education 2 international issues University (SJTU), CHE presents detailed survey data from League Tables as Policy thousands of teachers and students as well as third-party data dealing with the universities and technological institutes sepa- Instruments rately. One can look at the standing of each university or even Jamil Salmi and Alenoush Saroyan a specific academic subject, based on a particular indicator or set of indicators. Readers can even create their own ranking Jamil Salmi is coordinator of the World Bank Tertiary Education Thematic based on the indicators most relevant to them. The approach Group. Alenoush Saroyan is professor in the Department of Educational developed by CHE presents the additional advantage of avoid- and Counseling Psychology at McGill University, Montreal, Canada. The ing data biases linked to self-reporting by universities. findings, interpretations, and conclusions expressed in this article are entire- Austrian and Swiss universities have recently joined this exer- ly those of the authors and should not be attributed in any manner to the cise, accepting benchmarking against German universities. World Bank. E-mail: [email protected]. Pakistan n 1963, at the University of California at Berkeley, faculty The ranking of universities in Pakistan developed out of a Iand the administration objected when the radical student direct mandate given to the Higher Education Commission newspaper on campus, Cal Reporter, took the initiative to pub- (HEC) in 2002 to evaluate the universities in a way that pro- lish student evaluations of courses and professors. Despite this moted rapid and comprehensive development of the entire ter- initial resistance, student evaluations have become part of tiary education system, particularly to support the country's many universities' internal accountability mechanisms. place in the world economy. By comparing the inputs and out- Demands for increased accountability come not only from stu- puts of the nation's institutions, Pakistan has established a dents but also from other stakeholders—such as governments, mechanism for rewarding excellence and investing in the employers, and the public at large. Institutional rankings by strengthening of institutions that need improvement. The five league tables constitute one way of meeting the keen interest in information about the quality of universities. The expansion of league tables and ranking exercis- League tables use a wide range of indicators to measure the es has not gone unnoticed by the various stakehold- system's setup (input variables), function and efficiency ers, and the reaction they elicit is rarely benign. (process variables), and productivity and impact (output vari- ables)—relative to the performance of other universities and programs. The expansion of league tables and ranking exercises has main ranking criteria used by HEC (faculty qualifications, not gone unnoticed by the various stakeholders, and the reac- research output, students, facilities, and finances) are similar tion they elicit is rarely benign. In some countries (Argentina, to ranking indicators in many other nations. Nigeria, and India), the ranking exercise is undertaken as part The advisory committee overseeing this ranking exercise, of the accreditation process. With leagues tables becoming comprised of HEC officials and university representatives, had something of a growth industry, their accuracy, relevance, and to decide whether the results should be made public or not. usefulness have become issues of concern. Do they have any Under vehement protests from one of the leading vice-chancel- beneficial use for public policy, accountability, and consumer lors, whose (public) university had scored very low, the com- information? This article examines ways in which league mittee agreed not to publish the results. What HEC did instead tables have been used internationally to enhance quality. was to share key benchmarking data with each university, notably its relative position against each criterion used in the Germany rankings. For example, university X was told that, with respect In 1990, after the fall of the Berlin wall, teams of academics to the proportion of professors with a doctorate, it scored in the from the West German Science Council were given the task of lowest quartile compared to all universities in Pakistan. evaluating their counterparts at East German universities. It Despite the general outcry against the publication of the rank- was realized that in the absence of a tradition of evaluation in ings' results, this experience has forced the universities to take W data collection much more seriously est-German universities the mission would call for inventing . an appropriate methodology. More recently, the ranking exer- The Germany and Pakistan examples illustrate that, in cise conducted since 1998 by the Center for Higher Education countries without established evaluation or accreditation Development (Centrum für Hochschulentwicklung—CHE), mechanisms, rankings can be used effectively to monitor and an independent policy research agency, has become the first enhance quality. Finally, it is interesting to note that rankings comprehensive system providing a panorama of quality indica- are not used only by governments in their national context but tors in Germany. Instead of calculating a global ranking of also increasingly in an international perspective. In M ongolia institutions based on weighted indicators as do the Times and Qatar, for instance, the authorities have decided to restrict Higher Education Supplement and Shanghai Jiao Tong scholarships for studies abroad to students admitted in a high- international higher education international issues 3 ly ranked university. assessment test intended to compare the quality of undergrad- uate programs across universities. It could be described as a Public Initiatives ranking exercise categorizing university programs based on The press is often criticized for using rankings as a gimmick to the average score of their participating students. While the test boost sales. However, the mass media can play a genuine edu- (Provão) at first met with a lot of resistance, over time it cational role by making relevant information available to the became more accepted and increasingly influenced students in public, especially in countries lacking any form of quality their choice of tertiary institution. assurance mechanism. In Poland, when the transition to the Nigeria market economy started in the early 1990s and many private Similarly, in Nigeria, after the National Universities Council education institutions began to operate, there was a thirst for initiated a ranking of professional programs in 2001, even information about the quality of these institutions, which Perspektyvy magazine to initiate the going as far as closing down a number of programs, private- pushed the owner of the country's first ranking. Similarly, in Japan for many years the sector employers started to regain confidence in local universi- Asahi Shimbun newspaper annual ranking published by the ties and to hire graduates of the highest-ranked programs. fulfilled an essential quality assurance function, in the absence of any evaluation or accreditation agency. The Way Forward Governments and the public at large are increasingly preoccu- Colombia pied with the relative performance of tertiary education institu- Colombia was the first country in Latin America to set up a tions. Some countries are striving to have “world-class univer- national accreditation system in the mid-1990s, but the num- sities” that will spearhead the development of a knowledge- ber of programs reviewed by the new accreditation agency based economy. Others, faced with a shrinking student popu- remained relatively low in the first years because the accredita- lation, struggle to attract more fee-paying foreign students. tion law made the process voluntary and the most prestigious Just as scarcity, prestige, and excellence dominate the purchase universities, public and private, did not feel any compulsion to of consumer goods, students are also looking for indicators to participate. After the country's main newspaper, El Tiempo, identify the best universities. Thus, regardless of their contro- published in 2000 the full list of accredited programs to help versial nature and methodological shortcomings, university students choose among the various offerings, the universities rankings are unlikely to disappear. have felt increasing pressure to join the accreditation process. The following general recommendations, developed through our analysis of the recent international experience France with league tables, can help make the ranking exercise benefi- Another important merit of rankings is to stimulate public dis- cial to institutions, governments, students, parents and the cussions around critical issues affecting the tertiary education public at large. system. A good example is the debate that started in France Clarify what the ranking measures. Despite the ambiguities when the SJTU world rankings were published for the first surrounding the construct of quality, the rankers should speci- time. After observing that the best French university was fy what they do or do not measure, the goals, and the audiences ranked 65th, the daily paper Le Monde ran an article, “The for whom they do the ranking. The validity, reliability, and Great Misery of French Universities,” that started a widespread comprehensiveness of selected indicators can be discerned in discussion. One of the principal problems identified was the light of these specifications and by examining the scope of the fact that French universities are not allowed to select the most academic tasks (e.g., teaching, research, etc.) and the types of institutions being assessed. Furthermore, the rankers should make the raw data on which they base the ranking widely avail- The mass media can play a genuine educational able and the calculation process transparent so that their rank- ings can be verified independently. role by making relevant information available to Use multiple indicators and measures rather than a single, the public, especially in countries lacking any form weighted ranking. League tables should use a wide range of of quality assurance mechanism. indicators, placing greater emphasis on output and outcome indicators to ensure that every dimension of quality gets fac- academically qualified students, unlike the practice in more tored in the evaluation. Multiple sets of indicators will yield successful university systems in the United States, the United multiple scores rather than a global score, thus bringing to Kingdom, or Japan. The other important factor is the absolute light areas of strengths as well as areas of weaknesses. lack of competition among universities. Compare similar programs or institutions. Because of their methodological limitations, rankings are more meaningful Brazil when the unit of comparison is smaller. Ranking programs is In 1996 the Ministry of Education in Brazil introduced an thus preferable than ranking institutions. international higher education 4 international issues At the institutional level, use rankings for strategic planning and investors, managerial capacity, and other professional quality improvement purposes. Tertiary education institutions resources. that look at detailed ranking data for benchmarking purposes A second way universities can support economic develop- can use specific indicators to understand the determinants of ment is providing updated technical knowledge and skills in their performance and then work toward improving the quali- the workforce. Though less “sexy” or highlighted, universities ty of teaching and research. can play this important role at all levels. Not only do a small At the government level, use rankings to stimulate a culture of number of R&D scientists need their knowledge to be updated . In countries that do not yet have a well-established but also a large number of midlevel technical professionals quality quality assurance system, rankings can be used as a proxy for and even larger numbers of technical workers. One global quality. Similarly, at the international level, in the absence of a optoelectronics company in Japan had a well-developed inter- single global quality assurance agency, ranking systems take national network with cutting-edge scientists for its R&D activ- on some characteristics of a quality regulator for international ities. The company also appreciated its less visible ties with students. Use rankings as one of the instruments available to inform stake- Scientific discoveries leading to industrial innova- holders. Rather than being considered as the ultimate measure tions, particularly through academic spin-offs, pro- of quality and/or relevance, rankings should be complemented vide a classic image of universities contributing to by information on accreditation and labor market outcome data collected through surveys of employers and tracer surveys. economic development The results of league tables can also serve to generate a nation- al debate about long-term strategic priorities and policies for tertiary education. academics at local universities because it was these relation- ships that impacted a large number of company employees and helped them keep abreast of scientific progress. There is a complementary role in which academics under- take joint research, consultancy, or contract work with industry The Role of Higher Education to help address various technological problems. This interac- tion with industry allows academics to learn about industrial needs, but the relationships inform them equally about what is Institutions in Innovation and relevant. Universities can play a less direct yet important economic Economic Development role by setting forth the social, cultural, and intellectual tone of Sachi Hatakenaka a local area. Cultural events surrounding universities can make Sachi Hatakenaka is an independent consultant and researcher on higher a place more attractive to educated professionals and their education policy and management and is currently affiliated to the employers. The University of Pennsylvania goes further and Industrial Performance Center at the Massachusetts Institute of Technology makes an institutional commitment to community develop- for her research. E-mail: [email protected]. ment in its neighborhood. Other roles relevant to economic development include con- tributing to government and industrial decision making by everal decades of debate and experimentation are leading to participating in key committees and advisory boards and San understanding worldwide that many (but not all) univer- expanding local economic activities. sities can play diverse but important roles in innovation and economic development. Relevance as a University Mission In many countries, higher education institutions are or aspire Diverse Roles to be focused on academic research with little practical orienta- Scientific discoveries leading to industrial innovations, partic- tion. There is also a strong tendency for academic and research ularly through academic spin-offs, provide a classic image of drif t. Institutions with a practical orientation often become universities contributing to economic development (e.g., more academic, and teaching-focused institutions become Silicon Valley and Route 128). Many research institutions more research oriented. One reason for these changes has struggle to replicate this; universities may not have the culture been the dominant interpretation of “scientif ic autonomy,” or capability to do so. Yet some universities surround them- which was often interpreted to encourage isolation from exter- selves with a local industrial community that can absorb and nal stakeholders. make use of scientif T ic discoveries to accrue economic benefits. oday it is seen as possible for scientists to undertake fun- damental research while being motivated by practical rele- For spin-offs to be successful, the university must reside with- vance—rather like Pasteur, the French biologist known for his in a supportive ecosystem with technology-aware financial

Journal

Higher Education Management – Unpaywall

Published: Jun 1, 2007

There are no references for this article.