MyGlasgow Research

The policies of the University of Glasgow for the use of quantitative indicators for assessing research comply with and extend the principles outlined in the San Francisco Declaration on Research Assessment (DORA), 2012; The Metric Tide, 2015; and the Leiden Manifesto for Research Metrics, 2015.

Below we list the principles by which the University uses quantitative indicators.

Colleges, Institutes and Schools at the University of Glasgow are invited to develop local, more detailed policies provided that they are consistent with the institutional framework outlined in this document, and to make these widely known to staff.

2.1  Guiding Principles for the Uses of Quantitative Indicators in Research Assessment

The University of Glasgow is committed to applying the following guiding principles where applicable (e.g. in hiring and promotion decisions, grant applications):

1. Quality, influence and impact of research are typically abstract concepts that prohibit direct measurement. There is no simple way to measure research quality and quantitative approaches can only be interpreted as indirect proxies for quality.
2. Different disciplines have different perspectives of what characterises research quality, and different approaches for determining what constitutes a significant research output, for example, the relative importance of book chapters and journal articles, also an increasing recognition of the importance and value of non-traditional outputs which are found across all disciplines. All research outputs must be considered on their own merits, in an appropriate context that reflects the needs and diversity of research disciplines and outcomes.
3. Both quantitative and qualitative forms of research assessment have their benefits and limitations. Depending on the context, the value of different approaches must be considered and balanced. This is particularly important when dealing with a range of disciplines with different publication practices and citation norms. The practice of field-weighting does allow for field normalization when using metrics across different disciplines; however, it should be recognised that many of these databases simply do not offer sufficient coverage to give a comprehensive overview of a number of disciplines. Where quantitative metrics are neither appropriate nor meaningful, University of Glasgow will not impose their use for assessment in that area.
4. When making qualitative assessments, avoid making judgements based on external factors such as the reputation of authors or the journal or publisher of the work; the work itself is more important and must be considered on its own merits.
5. Not all indicators are useful, informative, or will suit all needs; and metrics that are meaningful in some contexts can be misleading or meaningless in others. For example, in some disciplines or subdisciplines, citation counts can estimate elements of usage, but in others they are not useful at all.
6. Avoid using metrics that do not account for differences in career stage or other variations in individual circumstances. Any metric that introduces bias when comparing individuals should be avoided. For example, the h-index should not be used to directly compare individuals, because the number of papers and citations differs dramatically across disciplines and at different points in a career.
7. Ensure that metrics are applied at the correct scale of the subject of investigation and do not apply aggregate level metrics to individual subjects, or vice versa. For example, do not assess the quality of an individual paper based on the impact factor of the journal in which it was published.
8. Quantitative indicators should be clearly defined and easily understood to ensure that the process is transparent and that they are being applied appropriately. Likewise, any quantitative goals or benchmarks must be open to scrutiny, and based on stated criteria that can allow for clarification or reproduction of the metric results obtained.
9. If goals or benchmarks are expressed quantitatively, care should be taken to avoid the metric itself becoming the target of research activity at the expense of research quality.
10. New metrics are continuously being developed to inform the reception, usage and value of all types of research output. All metrics or indicators must be used and interpreted in keeping with the other principles listed here. Always consider the sources and methods behind such metrics and whether they are vulnerable to being gamed, manipulated, or fabricated.
11. Bibliometrics are available from a variety of services, with differing levels of coverage, quality and accuracy, and these aspects should be considered when selecting a source for data or metrics. Where necessary, such as in the evaluation of individual researchers, choose a source that allows records to be verified and curated to ensure records are comprehensive and accurate, or compare publication lists against data from the Enlighten systems.

3.1  Research Income and Postrgaduate Research Student Supervision

The value of research income per R&T staff FTE and the number of postgraduate research students supervised by R&T staff FTE are primary research Key Performance Indicators (KPIs) for the University. These measures are also important indicators of the quality and vibrancy of the research environment as captured in the REF and in many international league tables. When the University applies such metrics at a more granular level, to units and/or individuals, they will always be normalised to account for discipline variations and career stages. Discipline normalisation may be made through HESA cost centres, using data that higher education institutions report annually and that are openly available. Care should be taken however, as not all disciplines or research will fit into predetermined centres and exceptions should be made where the work is a more suitable fit elsewhere.

3.2  Staff Recruitment, Performance Management and Promotion

The use of metrics in any process should be declared in advance of the process commencing, and their use should be considered alongside other metrics and other more qualitative assessments. Any quantitative indicator that is used will be based upon published formulae and will rely on openly available data, such that other experts in the field can reproduce the quantification of the metric.

We encourage practices that combine quantitative with qualitative indicators: the role of the metric is to inform assessment within a broader context and not to dictate. To support the application of this principle, candidates for promotion or academic staff undertaking the Performance and Development Review (PDR) will be asked to provide a narrative that highlights their best outputs, to explain the significance of the output to the advancement of the field and to describe the candidate’s contribution to the output.

This statement is licensed under a Creative Commons Attribution 4.0 International Licence. Please attribute as ‘Developed from the UCL Statement on the Responsible Use of Metrics’.