Researchers from the University of Glasgow’s Adam Smith Business School have published findings from a landmark study that could have significant implications for the future of academic research assessment.

The study challenges the mainstream belief that academic judges are best suited to evaluate research outputs. It seeks to inspire methods for effectively using metrics, considering the diverse communities that require clear evaluation criteria.

In an article published in Research Policy, the research team used data from the UK’s 2021 Research Excellence Framework (REF) to explore the relationship between metrics and expert judgement in assessing research outputs from 108 institutions. This covered 13,973 publications in business and management, one of the largest and most diverse fields in the evaluation.

This is the first conclusive evidence on post-DORA research assessment, offering valuable lessons for research evaluations globally.

The research reveals a strong correlation between journal rankings and expert scores, particularly for top-tier journals, suggesting that the perceived prestige of a journal may influence expert review processes, despite claims of journal ranking neutrality.

Despite widespread endorsement of the Declaration on Research Assessment (DORA), which advocates responsible assessment practices, DORA affiliation appears to make no difference to this correlation.

Many institutions publicly endorse DORA and broader principles of responsible assessment, yet their REF submissions show an implicit reliance on traditional metrics like journal rankings. This "institutional peacocking" suggests a superficial adherence to principles that doesn’t necessarily change the status quo.

Anna Morgan Thomas, Professor of Digital Management and Innovation at the Adam Smith Business School and lead author of the study, said: “This research comes at an extremely challenging time for the higher education sector, with uncertainty over funding and an increasing number of academics experiencing unsustainable workloads and high amounts of stress. Early career researchers are particularly vulnerable, with the current method of academic evaluation relying on knowledge of the system and more experienced voices dominating.

“Concerns about a knowledge gap in responsible research assessment have been raised for some time, and we were eager to contribute to this ongoing debate by providing empirical evidence that questions the practical application of these principles and highlights the need for a genuine commitment to change.

“We hope our findings will spark further debate and lead to more transparent and effective research evaluation practices worldwide.”

Adina Dudau, Professor of Public Management and co-author of the paper, added: “Ultimately, we all have a responsibility as academics to create a research environment where everyone can thrive and conduct impactful research that benefits society and humanity.”

Dr Beth Cloughton, an early career researcher at the Adam Smith Business School, reflected on the study’s findings and current evaluation frameworks, describing the situation as one that “disadvantages early career and marginalised academics, who may be more modest in their self-evaluations of novelty and contributions to a body of work, which neither reflects nor offers a promising career path.”

Read the full study on the University news webpage.

First published: 14 October 2024

<< All resources


More Information