On Oct. 19, 2011, the Higher Education Evaluation & Accreditation Council of Taiwan (HEEACT) released the results of the “2011 Performance Ranking of Scientific Papers for World Universities.” In order to explain the usage and limitations of this performance ranking system, HEEACT has compiled a list of frequently asked questions as follows:
1.What are the features of this ranking system?
2.What types of universities are the targets of evaluation for this performance ranking system?
3.Why do you select the database of WOS and ESI of Thomson Reuters?
4.What are the differences between the rankings of 2011 and the previous ones?
5.Why there are two types of performance based rankings? Which is the standard ranking?
6.What does distinguish the HEEACT performance ranking from the “Academic Ranking of World Universities” published by Shanghai Jiao Tong University?
7.What does distinguish this performance ranking from the “THE-QS World University Rankings” by the Times Higher Education and Quacquarelli Symonds of UK?
8.Can this performance ranking system replace other university rankings?
9.Can this performance ranking indicate the overall academic capability of a university?
10.Does this performance ranking project take into consideration for the size of the universities?
11.Why are papers of the disciplines of humanity not calculated in this performance ranking project?
12.Why does the h-index indicator only calculate the data performance of the last two years?
13.Some universities are presumed to rank higher than they actually performed in the ranking of HEEACT. What the reasons caused this phenomenon?
14.As if several universities are with similar scores in the ranking, does it mean that they are both well performed in the research performance among the evaluation?
15.Why there is such a huge difference between the scores of the first-place and second-place universities?
16.Do lower ranks indicate that a university has poorer academic performance?
17.Why do not categorize the universities ranked beyond 200 into two groups of 201-250 and 250-300, for a clearer indication of those universities with closer scores?
18.Are the indicators taken in the HEEACT ranking partial to quantitative measurement and thereby failing to assess the qualitative performance?
19.Will universities which mainly focus on humanities and social sciences, fall behind in the ranking or be excluded from the ranking?
20.Will a university with medical school be ranked higher?
21.Are the universities with larger size ranked higher?
22.How should we interpret the HEEACT ranking results which might be very different from expectations?
23.Why are there various indicators? Can they be combined or simplified?
24.Have the institutions been authoritatively complied and analyzed while calculated the ranking through the scientific papers?
25.For the universities have been renamed or unified, are the authority control be adjusted?
26.How to calculate the number of articles of high impact journals?
27.Do the English-speaking countries have advantages on the ranking performance since English journals are mainly selected in the database?
1.What are the features of this ranking system?
The features of this ranking system are with a quality indicator which accounts for 80% of the entire score. Our ranking is the first system employed through h-index, an indicator reflecting both the quality and quantity of research papers; 55% of the score represents a short-term progress of university in the performance of research. It should be noted that this ranking is exclusively based on the qualitative and quantitative performance of scientific papers. It does not evaluate the overall performance of university in teaching, research or administration. It also de-emphasizes the performance indices applied in other ranking systems which reflect only subjectively perceived reputations and extraordinary achievements. As our ranking system serves as an objective and informative benchmarking tool, it is a rather justified ranking system for the research universities of traditional or newly developed countries for assessing its achievement on science research.
[back to Top]
2.What types of universities are the targets of evaluation for this performance ranking system?
This performance ranking system is designed for research universities especially in those of transitional and newly developed countries. The objective indicators of this ranking system measure both long-term and short-term research performance of each university. The ranking serves as a good benchmark for world universities which each can map its relative position among the peer institutions. It also allows each research university to track its annual progress in terms of its output of scientific papers.
[back to Top]
3.Why do you select the database of WOS and ESI of Thomson Reuters?
The database of WOS and ESI of Thomson Reuters are selected because of its good and consistent quality of data. The databases of WOS and ESI are provided with the characteristics as follows:
(1) The data has covered the longest time span and widest coverage among the peer products: The earliest articles indexed in the database dated back to 1900, and citations traced back to 1800. By February, 2011, the database has contained more than 40 million article records from 11,456 journals.
(2) The data is above a high quality: WOS employs strict procedures of evaluation to select and indexes only the core journals from the basis of given scientific discipline. WOS reviews over than 2,000 journals each year, and only 10%-12% are selected into inclusion; the selected journals of WOS are examined through continuous reviews additionally.
(3) The rest of well-known ranking programs also take the data from Thomson Reuters, e.g., Academic Ranking of World Universities by Shanghai Jiao Tong University and the THE-QS World University Rankings by Times Higher Education and QS.
[back to Top]
4.What are the differences between the rankings of 2011 and the previous ones?
The same set of quality and quantity of indicators provided in previous rankings is applied to the HEEACT 2011 rankings. However, we only provide the rankings of top 500 universities in overall performance. The ranking of fields will be separately announced in the future.
[back to Top]
5.Why there are two types of performance based rankings? Which is the standard ranking?
The ranking offers two sets of ranking results with the overall performance of universities. The first set is based on the original scores of universities; the second one is calculated via original scores which has been further adjusted by the numbers of university faculty. HEEACT has received many suggestions concerning the factor of faculty size and the ranking results after the performance ranking of 2007. Therefore, HEEACT has started offering the adjusted ranking based on the amount of faculty since 2008. However, the ranking based on the original scores is still considered as the official data of annual ranking project. The information of faculty number is used for calculation adjustment which is drawn from various of sources, thus it does not really constitute a concrete basis of cross institution and cross domain comparisons.
[back to Top]
6.What does distinguish the HEEACT performance ranking from the “Academic Ranking of World Universities” published by Shanghai Jiao Tong University?
The “Academic Ranking of World Universities” published by Shanghai Jiao Tong University which takes certain indicators for the measurement of extraordinary research achievement, including the amount of Nobel laureates affiliated with the institution, the number of highly cited scholars, and the sum of papers in Nature or Science. However, these types of research achievement are usually beyond some specific universities in the world, therefore it might bring the limited effect on the average institutions. The HEEACT ranking is achievable for many universities as it’s associated with a set of indicators which are sensitive to the excellence of short-term scientific papers. An annual progress of university will easily result in a change of the ranking of this system, as the evaluation of university is timely modified through the accurate and informative indication.
[back to Top]
7.What does distinguish this performance ranking from the “THE-QS World University Rankings” by the Times Higher Education and Quacquarelli Symonds of UK?
“THE-QS World University Rankings” emphasizes on peer review which is account for 50% of the ratio of score. The results can be easily influenced by subjective views, especially in its questionnaire-type ranking and the allotment of point, which would turn the result into an assessment of university reputation but not the actual performance. On the contrary, the ranking performance of HEEACT utilizes more objective methods and statistics to conduct its university ranking.
[back to Top]
8.Can this performance ranking system replace other university rankings?
No. This ranking system only evaluates performances of scientific papers. The indicators are designed to compare the quality and quantity of scientific papers of each university (including sciences and social sciences) from both the long-term and short-term perspectives. The ranking does not indicate overall performances of universities in teaching, research, and administrative activities.
[back to Top]
9.Can this performance ranking indicate the overall academic capability of a university?
Only a partial ranking can indicate the academic capability of a university. The amount of thesis not the only basis for demonstrating a variety of research activities, the output such as monographic publications, granted projects, patents, collaboration related industries as well as may evaluate the performance ranking. The production of scientific papers is only a sort of research capability of a university; however, it is the major one. Except the disciplines of humanities, the quality and quantity of scientific papers are frequent-used indicators for assessing the academic capability of a university.
[back to Top]
10.Does this performance ranking project take into consideration for the size of the universities?
Yes. The amount of papers is tightly related to the size and history of the institution itself. This performance ranking system has used certain indicators such as the average citations per paper to balance the influence from the size and history of one university. Our ranking methodology is rather sensitive to the factor of university size even with the precaution of the overall ranking results. In the purpose of offering an impartial view on the ranking, our project also provides an additional ranking as a reference, in which the original ranking has adjusted by the numbers of faculty.
[back to Top]
11.Why are papers of the disciplines of humanity not calculated in this performance ranking project?
Due to the regional characteristics in the disciplines of humanity, and the research output is often published in non-English journals or monographic publications. Since English journals are mainly included in the indices of A&HCI, it only represents some particular scholarly performance and achievement of worldwide humanities researchers. For the reason above, this performance ranking project excludes the disciplines of humanity from the analyses.
[back to Top]
12.Why does the h-index indicator only calculate the data performance of the last two years?
H-index is a highly sensitive indicator. The amount of papers published in two years is usually sufficient for the analyses of h-index (For an instance, a figure of 28,951 papers published by Harvard University in 2005-2006). The results of several recent studies also confirmed that two years of h-index is sufficient for the evaluations of institution. To determine an effective range of years for the measurement of h-index performance, we took a sample of an amount of 47 universities including Harvard and Tokyo University, and analyzed their annual h-index performances with the ranges of 2 years (2005-2006) and 11years (1996-2006). The result shows that the h-index values of 2 years and 11 years are highly correlated with a figure of .967. According to the result, we’ve decided to take the values of 2-year h-index for this ranking project.
[back to Top]
13.Some universities are presumed to rank higher than they actually performed in the ranking of HEEACT. What the reasons caused this phenomenon?
The HEEACT ranking is exclusively based on the quantitative and qualitative performance of scientific papers. Its results differ from those of other ranking programs which evaluate different aspects of university performances. The differences of indicators of evaluation, the scale of the university, the affiliation of hospital, and the ratio of the departments of social science of one institution are the factors that may influence the result of evaluation.
[back to Top]
14.As if several universities are with similar scores in the ranking, does it mean that they are both well performed in the research performance among the evaluation?
Yes. One feature of HEEACT ranking is that the score of the first-place university obviously stands out among all universities. As such the score differences among all other universities are rather small. In other words, the scores are relative and indicative rather than being absolute and rigid. A university with slightly lower score is not necessarily inferior to its peer institutions with higher scores. The close scores of school are not representative among the performance of its scientific papers.
[back to Top]
15.Why there is such a huge difference between the scores of the first-place and second-place universities?
The huge gap between the top two universities is a prevalent phenomenon in academic rankings. For instance, the difference between the scores of the top two institutions in the 2011 Academic Ranking of World Universities reaches 27.4 points. Several ranking systems may try to lessen the gap of scores. For instance, the world universities ranking of Times Higher Education has taken Z-scores to adjust the original scores in order to reduce the gap. In order to trustworthily presenting the result of HEEACT ranking, we’ve decided not to use any adjustment techniques.
As the first-ranked Harvard University has presented a great record of its performance of medical school (The medical school has published 41,895 papers over a span of 2000 to 2011, which received an amount of 1,182,589 citations); additionally, its other academic disciplines also show an exceptional performance in the quantity and quality of papers. Thus among the eight indicators used in the 2011 HEEACT ranking, Harvard ranks highest among all the indicator expect the average number of citations. The second-placed Johns Hopkins University has an outstanding performance with its school of medicine (The school of medicine has published 26,264 papers during 2000 to 2011, which received an amount of 552,123 citations). However, its quantitative and qualitative performances of papers examined from long-term and short-term perspectives have obviously fallen behind Harvard University. There are also large gaps in both of the performance of highly cited papers and high-impact journal papers between the scores of two universities.
[back to Top]
16.Do lower ranks indicate that a university has poorer academic performance?
Not completely. Since the performance of scientific papers only reflects part of academic performance of a university, a lower rank of university in HEEACT system doesn’t explain the entire academic performance or even the reputation of one university. Some specific factors which may lead the difference among the university performance, such as the size of university, the affiliation of medical school, and the ratio of social sciences disciplines of university. The factors above must be taken into consideration when one interprets the ranking results.
[back to Top]
17.Why do not categorize the universities ranked beyond 200 into two groups of 201-250 and 250-300, for a clearer indication of those universities with closer scores?
It is not easy to set a range of close interval. We would encounter problems with the setting of the range of each group, if we indicated those universities with closer scores into two ranking groups. There might be only a slight difference in scores between the university at the last place of the 1st group and the university at the first place of the 2nd group, yet these two universities are separated into different groups. Since it would be difficult to determine how many universities should be included in one ranking group, and the differences of slight scores should not be ranked into two groups, thereby HEEACT performance ranking project did not indicate additional ranking by groups.
[back to Top]
18.Are the indicators taken in the HEEACT ranking partial to quantitative measurement and thereby failing to assess the qualitative performance?
No. In contrast, the HEEACT ranking highly emphasizes the qualitative performance of scientific papers with objectivity. The indicators of measuring qualitative performance have weighted up to 80%. In other words, although the HEEACT ranking employs exclusively objective statistical data, it conceptually assesses both the quantity and quality of each university’s scientific papers.
[back to Top]
19.Will universities which mainly focus on humanities and social sciences, fall behind in the ranking or be excluded from the ranking?
Most likely. Since the ranking is based on the journal papers indexed in the database of SCI and SSCI. Humanities papers from A&HCI database are excluded from the scope of the ranking. Though the social science papers are included, the amount of the papers is still much lower due to the sum of journals of social science (SCI indexes 7,565 journals, SSCI merely indexes 2,306). The discrepancy in the numbers of indexed journal naturally results in less favorable ranking of those universities specializing in humanities and social sciences.
[back to Top]
20.Will a university with medical school be ranked higher?
Yes. Papers published in medical sciences journals and the higher citations than many other disciplines may cause a better ranking. For instance, 2,116,193 papers of clinical medicine have been published, and the papers have been cited 27,355,596 times. 817,334 papers in the fields of engineering have been published and been cited 3,887,615 times. The huge discrepancy in papers production and citations between medical sciences and other disciplines naturally leads university with medical school to a more outstanding ranking performance.
[back to Top]
21.Are the universities with larger size ranked higher?
Yes. The ranking has attempted to neutralize the negative influences from university size through the indicators such as the average of citations per paper. However, the ranking is still beneficially prone to larger institutions. To recognize the potential biases resulted from university size, HEEACT has offered an additional ranking as reference which incorporates faculty numbers in the calculation of scores.
[back to Top]
22.How should we interpret the HEEACT ranking results which might be very different from expectations?
To remind again that the HEEACT ranking is not a reputation ranking or a university ranking. It is exclusively based on objective data and the measurement of scientific papers performance thus the results would likely differ from subjective perceptions. Although there might be gaps between the results of HEEACT ranking and readers’ expectations, the relative positions of universities from the same country of HEEACT rankings are generally consistent with social expectations.
[back to Top]
23.Why are there various indicators? Can they be combined or simplified?
In this ranking system, each indicator represents different criteria of measuring the performance of scientific papers. Although the incorporation of short-term indicators has increased the complexity of ranking, it enhances the sensitivity of ranking methodology and is able to prioritize universities with recent progress in research. Therefore, combining or simplifying the indicators may compromise the quality of ranking results. Furthermore, all the 8 indicators passed the examination of regression, signifying the necessity for these indicators.
[back to Top]
24.Have the institutions been authoritatively complied and analyzed while calculated the ranking through the scientific papers?
Yes. Generally speaking, we have conducted authority control on various forms and names of a university to ensure the accuracy and completeness of data. The university system with several campuses is another case of authority control, this project has differentiated each campus by labeling the city name. For example, the various campuses of University of Texas are marked by their locations - Austin, M. D. Anderson Cancer Center, Dallas, Southwestern Medical Center, etc.
[back to Top]
25.For the universities have been renamed or unified, are the authority control be adjusted?
The worldwide universities have often been renamed, unified or reorganized. We have verifies the target universities to ensure the accuracy and objectivity of the rankings before conducting the analysis. For example, University Louis Pasteur, Marc Bloch University and Robert Schuman University have been unified into University of Strasbourg since January, 2009. Prior to that, we have already made appropriate adjustments of authority control in the database.
[back to Top]
26.How to calculate the number of articles of high impact journals?
We have selected the top 5% journal articles among the impact factor of every field in JCR database. In other words, we rearranged the journals by their impact factors from the highest to the lowest, and selected the top 5% as the high impact journals. The high impact journals will be re-calculated within the new sets of assembling journals.
Note: Impact factor (called IF for short) is an average number of citations received within one year per paper and the paper has published in that journal during two preceding years. If the impact factor of one journal is higher, its research article can bring a greater influence, which also reflects the journals has a better academic performance of one specific discipline.
[back to Top]
27.Do the English-speaking countries have advantages on the ranking performance since English journals are mainly selected in the database?
The English-speaking countries do have more advantages on the ranking since the HEEACT ranking selected the database of Thomson Reuters to analyze the qualitative and quantitative performance of scientific papers. According to the 2011 Ranking Performance of Scientific Papers for World Universities, the non-English-speaking countries have kept making progress in the recent five year, for example, France, German, the Netherlands and China. In contrast, the English-speaking countries have gradually been exceeded by the non-English-speaking countries. During the 5 years, 12 English-speaking universities are out of the top 500, and 105 English-speaking universities have been ranked backward. The result shows that non-English-speaking countries have continued publishing English articles, its performance has kept growing up.
[back to Top]
Note: The Tsinghua University and the Peking Union Medical College have been proven as non-affiliations. Therefore, HEAACT Ranking has removed the related authority control between these two institutions and has updated in the latest evaluation.