On July 30, 2009, the Higher Education Evaluation & Accreditation Council of Taiwan (HEEACT) released the results of the “2009 Performance Ranking of Scientific Papers for World Universities” project. In order to explain the usage and limitations of this performance ranking system, HEEACT has compiled a list of frequently asked questions as follows:
1.What are the features of this performance ranking system?
2.What types of universities are the targets of evaluation for this performance ranking system?
3.What are the differences between the 2008 and the 2009 rankings?
4.How do you classify subject fields for the field based rankings?
5.Why are the 2009 indicators identical with the 2008 ones?
6.Why are there two overall performance based rankings?
7.Why can’t I find faculty number adjusted rankings in the field based rankings?
8.What distinguishes this performance ranking from the “Academic Ranking of World Universities” published by Shanghai Jiao Tong University?
9.What distinguishes this performance ranking from the “World University Rankings” by the UK’s “Times Higher Education Supplement”?
10.Why do the overall performance ranking and field based rankings use the same indicators?
11.How do the 2009 HEEACT field based rankings differ from those conducted by Shanghai Jiao Tong University?
12.How is the field based performance ranking different from the World University Rankings by the Times Higher Education Supplement?
13.Can this performance ranking system replace other university rankings?
14.Can this performance ranking indicate the overall academic capabilities of a university?
15.Does this performance ranking project take into consideration the size of the universities?
16.Why are papers from the humanities disciplines not calculated in this performance ranking project?
17.Why does the h-index indicator calculate performance data of the last two years only?
18.Some universities are presumed to rank higher than they actually do in the HEEACT ranking. Why is that?
19.Do universities with similar scores perform equally well in their scientific papers production?
20.Do lower ranks indicate poor academic performance of universities?
21.Why weren’t universities ranked beyond 200 categorized into two groups, 201-250 and 250-300, to clearly indicate those universities closer in scores?
22.Are the indicators used in the HEEACT ranking too quantitative and thereby failing to assess the qualitative performance?
23.Will universities focusing largely on humanities and social sciences fall behind in the ranking or fail to be included in the ranking at all?
24.Will a university with a medical school rank higher?
25.Are larger universities more likely to rank higher?
26.How should we interpret the HEEACT ranking results which might be very different from people’s expectations?
27.Why are there so many indicators? Can they be combined or simplified?
28.In compiling and analyzing scientific papers for the ranking process, was authority control used?
29.For universities that have consolidated or changed their names, were there adjustments in authority control?
30.For data retrieval, why did this project select WOS rather than Scopus?
31.Why is there such a huge difference between the scores of the first-place and second-place universities?
1.What are the features of this performance ranking system?
The features of this performance ranking system are that quality indicators account for 80% of the score; that it is the first ranking system employing h-index, an indicator reflecting both the quality and quantity of research papers; and that 55% of the score represents a university’ short-term progress in research. It should be noted that this performance ranking is based exclusively on the qualitative and quantitative performance of scientific papers. It does not assess the overall university performance in teaching, research, and administration. It also de-emphasizes performance indices used in other ranking systems that represent subjectively perceived reputations and extraordinary achievements. As such this ranking system serves as an objective and informative benchmarking tool for research universities in transitional and newly developed countries in assessing the achievement of science research.
[back to Top]
2.What types of universities are the targets of evaluation for this performance ranking system?
This performance ranking system is designed for research universities especially those of transitional and newly developed countries. The objective indicators used in this ranking system measure both long-term and short-term research performance of each university. The ranking serves as a good benchmark for world universities from which each can map its relative position among peer institutions. It also allows each research university to track its annual progress in terms of its output of scientific papers.
[back to Top]
3.What are the differences between the 2008 and the 2009 rankings?
The 2009 ranking differs from the 2008 ranking in the selection of universities for field based ranking. Using the same set of weighted indicators and the same subject categorization, the 2009 ranking expanded the pool of universities for field based performance ranking. In addition to the universities already included in the overall performance ranking, the 2009 field based ranking also added universities that had excelled in certain subject fields but were not included in the former. These universities were selected based on the ESI field based data and the lists of universities ranked by THES and ARWU.
[back to Top]
4.How do you classify subject fields for the field based rankings?
First, we used the subject categorization scheme of WOS as the basis for our subject classification and subsequent data analyses. We further referenced the subject categorization of the Shanghai Jiao Tong University and Times Higher Education Supplement rankings and consulted with different subject area researchers. The resulted subject categorization used for this project is comprised of six major categories as follows,
(1) AGE: Agricultural Sciences, Plant & Animal Science, Environment/Ecology.
(2) MED: Clinical Medicine, Psychiatry.
(3) ENG: Computer Science, Engineering, Materials Sciences.
(4) LIFE: Biology & Biochemistry, Microbiology, Immunology, Neuroscience & Behavior, Pharmacology & Toxicology, Molecular Biology & Genetics.
(5) SCI: Chemistry, Geosciences, Mathematics, Physics, Space Science, Psychology.
(6) SOC: Social Sciences, General, Economics & Business.
[back to Top]
5.Why are the 2009 indicators identical with the 2008 ones?
In order to benchmark the performance of universities and trace their progresses, consistency in assessment methodology is a must for long term, continuous ranking programs.
[back to Top]
6.Why are there two overall performance based rankings?
The 2009 ranking offers two sets of ranking results indicating the overall performance of the universities. The first set is based on the original scores of the universities, and the second is the ranking result where the original scores have been further adjusted by university faculty numbers. After the 2008 ranking was released much criticism has called for the use of faculty numbers to adjust the original scores. However, although the faculty number is a significant factor affecting the scientific papers performance of a university, incorporating that factor in score calculation is practically infeasible at the current stage. First, each university defines its full-time faculty differently; second and the most important of both, not all faculty number information of the universities is available and/or is provided on the equal basis. Thus, although this year HEEACT has decided to offer the faculty numbers adjusted ranking per request of audience, readers are reminded that the faculty number information used for the adjusted calculation was drawn from various sources, and thus it does not really constitute a concrete basis for cross institution or cross domain comparisons. HEEACT considers the ranking based on the original scores as the official result of the 2009 ranking project.
[back to Top]
7.Why can’t I find faculty number adjusted rankings in the field based rankings?
As explained above, due to infeasibility in obtaining accurate and comparable faculty number information for the adjustment. Instead, we conducted subject disciplines based rankings to resolve the disciplinary differences in ranking. Thus no adjusted ranking is offered for the field based rankings.
[back to Top]
8.What distinguishes this performance ranking from the “Academic Ranking of World Universities” published by Shanghai Jiao Tong University?
The “Academic Ranking of World Universities” published by Shanghai Jiao Tong University uses certain indicators that measures extraordinary research achievement, including the number of Nobel laureates affiliated with that institution, the number of highly cited scholars, and numbers of papers in Nature or Science. However, these types of research achievement are usually beyond most universities in the world and therefore offer limited use for those average institutions. The HEEACT ranking, instead, uses a set of indicators that are sensitive to short-term scientific papers excellence which are achievable for many universities. A university’s annual progress will easily result in a change of ranking in this system, thus offering timely, accurate, and informative indication for university evaluation.
[back to Top]
9.What distinguishes this performance ranking from the “World University Rankings” by the UK’s “Times Higher Education Supplement”?
The “World University Rankings” by the Times Higher Education Supplement emphasizes on peer review and employer review, weighing up to 50%. The results could easily be influenced by subjective views, especially in its questionnaire-type ranking and point allotment, which would easily turn this into an assessment of a university’s reputation and not its actual performance. HEEACT’s performance ranking, in contrast, utilizes more objective methods and statistics to conduct its university ranking.
[back to Top]
10.Why do the overall performance ranking and field based rankings use the same indicators?
Ranking programs often use a same set of indicators in rankings for consistency in data as well as to enhance the comparability of the results. For example, Shanghai Jiao Tong University’s rankings employ the same indicators both for overall performance and field based rankings. In our case using the same set of indicators in both the overall performance and field based rankings ensures the comparability of cross-discipline and/or cross-institution performances. Our indicators are applicable to and effective for both university-based and field based rankings.
[back to Top]
11.How do the 2009 HEEACT field based rankings differ from those conducted by Shanghai Jiao Tong University?
The HEEACT ranking system differs from the Shanghai Jiao Tong University rankings in the indicators used for field based rankings. In addition, the HEEACT ranking also differs from the Shanghai Jiao Tong University ranking in its subject categorization. The Shanghai Jiao Tong University ranking divides subject disciplines into five major areas: clinical medicine, engineering, life sciences and agricultural sciences, natural sciences, and social sciences. The HEEACT ranking separate agricultural and life sciences areas and classifies the subject disciplines into six major categories.
[back to Top]
12.How is the field based performance ranking different from the World University Rankings by the Times Higher Education Supplement?
In addition to the use of different indicators, this ranking system differs from the Times Higher Education Supplement’s ranking mainly in the categories of subject fields. Whereas the Times analyzes the rankings in five fields - arts and humanities, life sciences and biomedicine, natural sciences, social sciences and technology, this study conducts field based rankings in six broad subject categories – agriculture and environment sciences, clinical medicine, engineering, computing and technology, life sciences, social sciences and natural sciences. And this study did not include the field of arts and humanities as part of rankings.
[back to Top]
13.Can this performance ranking system replace other university rankings?
No. This ranking system evaluates performances of scientific papers only. The indicators are designed to compare the quality and quantity of each university’s scientific papers (including sciences and social sciences) from both the long-term and short-term perspectives. The ranking does not indicate universities’ overall performances in teaching, research, and administrative activities.
[back to Top]
14.Can this performance ranking indicate the overall academic capabilities of a university?
Partially. A university’s academic capabilities can be demonstrated in a variety of research activities and output such as its monographic publications, granted projects, patents, collaboration with industries, etc. Scientific papers production is only a part of a university’s research capabilities; however, it is a major one. Except in the disciplines of humanities, the quality and quantity of scientific papers are well-received indicators for assessing the academic capability of a university.
[back to Top]
15.Does this performance ranking project take into consideration the size of the universities?
Yes. The number of papers from a university is naturally related to the size and history of that institution. This performance ranking system has used certain indicators such as the average citations per paper to balance the influences from university size and history. Even with the precaution the overall ranking results suggest that our ranking methodology is rather sensitive to the factor of university size. To offer an alternative view on the ranking the 2008 project offers an additional ranking in which the original ranking is adjusted by faculty numbers. However, readers are reminded that the adjusted ranking employs complicated sources for faculty number information, and therefore the results must be treated with caution.
[back to Top]
16.Why are papers from the humanities disciplines not calculated in this performance ranking project?
In humanities disciplines, the research tends to take on regional characteristics, and research output is often published in non-English journals or via monographic publications. Since the A&HCI indexes mainly papers in English journals, it only weakly represents the worldwide humanities researchers’ scholarly performance and achievement. For the reason this performance ranking project excludes humanities disciplines from the analyses.
[back to Top]
17.Why does the h-index indicator calculate performance data of the last two years only?
H-index is a highly sensitive indicator. The number of papers published in two years is usually sufficient for h-index analyses (for example in 2005-2006 Harvard University produced 28,951 papers). Results from several recent studies also confirmed that two years for h-index is sufficient for evaluations of institution. To determine the effective range of years for h-index performance measurement, we took a sample of 47 universities including Harvard and Tokyo University and analyzed their annual h-index performances within the ranges of 2 (2005-2006) to 11 (1996-2006) years. The result shows that the h-index values of 2 years and of 11 years are highly correlated (.967), thus supporting our use of 2-year h-index values in this ranking project.
[back to Top]
18.Some universities are presumed to rank higher than they actually do in the HEEACT ranking. Why is that?
The HEEACT ranking is based exclusively on the quantitative and qualitative performance of scientific papers. Its results can naturally differ from those of other ranking programs which evaluate different aspects of university performances. Further, the HEEACT ranking system is rather sensitive to certain factors including university size, whether the university has a medical school, and how large the social sciences departments account for the entirety of the university. Situations in these aspects can affect universities’ ranks in the HEEACT system and yield unexpected results.
[back to Top]
19.Do universities with similar scores perform equally well in their scientific papers production?
Yes. One feature of the HEEACT ranking is that the score of the first-place university stands out among all universities. As such the score differences among all other universities are rather small. In other words, the scores are relative and indicative rather than absolute and rigid. A university with a slightly lower score is not necessarily inferior to its peer institutions with higher scores.
[back to Top]
20.Do lower ranks indicate poor academic performance of universities?
Not necessarily. Scientific papers performance constitutes only a part of a university’s overall academic performance although it is a significant one. Any interpretation of poorer academic performance or reputation due to a university’s lower rank in the HEEACT system is over inference. As explained elsewhere, certain factors such as university size, whether a university has a medical school, and how large the social sciences disciplines account for the entirety of the university’s scholarly research can significantly affect the HEEACT rankings. These factors must be taken into consideration when one interprets the ranking results.
[back to Top]
21.Why weren’t universities ranked beyond 200 categorized into two groups, 201-250 and 250-300, to clearly indicate those universities closer in scores?
We would encounter problems with setting the range of each group, if we had indicated those universities closer in scores simply into two ranking groups. What might ensue is that there might be only a slight difference in scores between the university at the last place of the first group and the university at the first place of the second group, yet these two universities are ranked into different groups. Since it would be difficult to determine how many universities should be included in a ranking group, and slight score differences should not be ranked into two groups, thereby HEEACT’s performance ranking project did not indicate additional ranking by groups.
[back to Top]
22.Are the indicators used in the HEEACT ranking too quantitative and thereby failing to assess the qualitative performance?
No. In contrast the HEEACT ranking highly emphasizes the qualitative performance of scientific papers. Indicators measuring qualitative performance weight up to 80%. In other words, although the HEEACT ranking employs exclusively objective statistical data, it conceptually assesses the quantity and quality of each university’s scientific papers.
[back to Top]
23.Will universities focusing largely on humanities and social sciences fall behind in the ranking or fail to be included in the ranking at all?
Very likely. The ranking is based exclusively on the journal papers indexed in SCI and SSCI databases. Humanities papers (A&HCI database) are outside the scope of the ranking. Further, a large gap between SCI and SSCI exists in terms of the numbers of journals included. SCI indexes 7,565 journals, while SSCI indexes 2,306. The discrepancy in the numbers of the indexed journal naturally results in less favorable ranking for those universities specializing in humanities and social sciences.
[back to Top]
24.Will a university with a medical school rank higher?
Yes. Papers published in medical sciences journals and their citations are significantly higher than many other disciplines. Between 1997 and 2007, 1,952,646 clinical medicine papers were published, and they were cited 22,430,015 times; while in the fields of engineering 748,139 papers were published and were cited 2,734,738 times. The huge discrepancy in papers production and citations between the medical sciences and other disciplines naturally leads to favorable rankings for universities with medical school. To avoid high rankings caused by subject discipline differences, starting from 2008, the HEEACT ranking system also offers ranking results by subject disciplines.
[back to Top]
25.Are larger universities more likely to rank higher?
Yes. The ranking has attempted to neutralize negative influences from university size through the use of indicators such as the average citations per paper. However, the ranking was still prone to favorable representation of larger institutions. Recognizing the potential biases resulted from university size, this year HEEACT offers an additional ranking which incorporates faculty numbers in the calculation of scores. As one can see, the University of Washington at Seattle has approximately 4800 faculty members, 40,000 students. It ranks the fourth in the primary ranking. When faculty number is taken into account in score calculation, however, it ranks 13th. This institution offers an example of how factors of university and faculty sizes may affect the overall ranking.
[back to Top]
26.How should we interpret the HEEACT ranking results which might be very different from people’s expectations?
The readers are reminded again that the HEEACT ranking is not a reputation ranking or a university ranking. It is based exclusively on objectively obtained data and measures scientific papers performance only. Thus the results can very likely differ from people’s subjective perceptions about how well certain universities should do in relation to other universities. Although there might be gaps between the HEEACT ranking results and readers’ expectations, the relative positions of the universities from the same country in HEEACT rankings are generally consistent with social expectations.
[back to Top]
27.Why are there so many indicators? Can they be combined or simplified?
In this ranking system, each indicator represents different criteria in measuring the performance of scientific papers. Although the incorporation of short-term indicators increases the complexity of ranking, it enhances the sensitivity of the ranking methodology and is able to prioritize universities with recent progress in research. Therefore, combining or simplifying these indicators may compromise the quality of the ranking results. Furthermore, all 8 indicators passed the significance test on regression, signifying the necessity for these indicators.
[back to Top]
28.In compiling and analyzing scientific papers for the ranking process, was authority control used?
Yes. Generally speaking, for this ranking system, project staff conducts authority control on the various forms of a university name to ensure the accuracy and completeness of data. For university systems with several campuses, this project differentiates each campus by labeling the city name of the said campus. For example, the various campuses of University of Texas are marked by their locations - Austin, M. D. Anderson Cancer Center, Dallas, Southwestern Medical Center, etc. - as basis for analysis.
[back to Top]
29.For universities that have consolidated or changed their names, were there adjustments in authority control?
Universities often change names, consolidate or reorganize. That is why each year before conducting ranking analysis, project staff verifies the universities selected as targets to ensure accuracy and objectivity of the rankings. A case in point was the Royal Veterinary and Agricultural University. The school became a part of University of Copenhagen as of January 2007. Prior to that, project staff had already made appropriate adjustments in authority control in the database.
[back to Top]
30.For data retrieval, why did this project select WOS rather than Scopus?
This ranking system adopts the data from WOS for its higher emphasis on scientific papers and its careful selection of publications. Also, the quality of database is more consistent. Moreover, recent studies indicate only slight differences of study results between WOS and Scopus. Therefore, this project shall continue using WOS database.
[back to Top]
31.Why is there such a huge difference between the scores of the first-place and second-place universities?
The huge gap between the first two universities in the ranking result is a prevalent phenomenon in academic rankings. For example, in the 2008 Shanghai Jiao Tong University ranking the difference between the first two institutions’ scores reaches approximately 30 points. Some ranking systems may try to lessen the gap in scores. For instance, The Times ranking has used the Z-scores to adjust the original scores in order to reduce the gap. In order to truthfully present the result of score calculation the HEEACT ranking does not use any point adjustment techniques.
The first-placed Harvard University is large in size (with a student body of 20,000 and faculty of 3,369) and has a medical school with great performance record (it published 42,126 papers between 1997 and 2007, which received 1,048,531 citations); it also has exceptional performance in terms of papers quantity and quality in most of the academic disciplines. Thus among the eight indicators used in the 2008 HEEACT ranking, Harvard ranks highest in all indicator expect the average number of citations. The second-placed Johns Hopkins University is also a large university (with a student body of 20,000 and faculty of 3,370) and with a medical school performing outstandingly (the school published 24,274 papers from 1997-2007, which received 552,123 citations). However, its quantitative and qualitative performance of papers examined from long-term and short-term perspectives fell significantly behind Harvard. There are also significant gaps in highly cited papers and high-impact journal papers between the two universities. All the phenomena contributed to the great difference in the two universities’ scores.
[back to Top]