Improving quality assessment of composite indicators in university rankings: a case study of French and German universities of excellence

Abstract

Composite indicators play an essential role for benchmarking higher education institutions. One of the main sources of uncertainty building composite indicators and, undoubtedly, the most debated problem in building composite indicators is the weighting schemes (assigning weights to the simple indicators or subindicators) together with the aggregation schemes (final composite indicator formula). Except the ideal situation where weights are provided by the theory, there clearly is a need for improving quality assessment of the final rank linked with a fixed vector of weights. We propose to use simulation techniques to generate random perturbations around any initial vector of weights to obtain robust and reliable ranks allowing to rank universities in a range bracket. The proposed methodology is general enough to be applied no matter the weighting scheme used for the composite indicator. The immediate benefit achieved is a reduction of the uncertainty associated with the assessment of a specific rank which is not representative of the real performance of the university, and an improvement of the quality assessment of composite indicators used to rank. To illustrate the proposed methodology we rank the French and the German universities involved in their respective 2008 Excellence Initiatives.