31 May 2025
Jessica Howie, University of Waikato, Aotearoa New Zealand
How might we measure research excellence, and what are the views of researchers around this? The answer, it seems, is surprisingly complex, and intertwined with systemic issues in academia, such as the cultural burden on Indigenous researchers, the politics of contribution, individualism, and ultimately, research culture.
A team of librarians at the University of Waikato Library in Aotearoa New Zealand sought to investigate this at the behest of their Deputy Vice-Chancellor Research. The methodology was based around a “Profiles Check” service, which had been a service offering of the library for some time. The Profiles Check involves a systematic review of a researcher’s online profiles and their outputs. The result is a detailed and personalised report which provides researchers with a range of metrics to help provide evidence of scholarly and non-scholarly impact, and a list of recommendations for optimising their online presence. The intention is to enhance a researcher’s ability to tell the story of their research, whilst providing opportunities for librarians to discuss strategies to increase impact, such as identifying outputs that can be made open through the institutional repository.
In a series of discussions guided by the Profiles Check, participants from a range of disciplines were asked to respond to various metrics associated with their research, while considering their individual research goals. Metrics were presented at both researcher and output level and from multiple sources but ultimately the team presented each measure to the researcher and asked, “what does this mean to you?”. Their responses, which were far more nuanced than expected, painted a picture of an environment where a culture of counting, lack of awareness around metrics and misalignment of incentives impede their ability to develop truly impactful and culturally faithful research.
A lack of awareness around metrics pervades research evaluation at multiple levels. Not only were many metrics not well understood, but there was also a perception on the part of the participants that other players in the research evaluation ecosystem lacked the knowledge to use metrics fairly. Generally, participants were more likely to employ metrics they could find easily and thought others would understand and value, even when they knew those metrics were flawed. The h-index was an example of this.
Participants expressed skepticism around metrics, and there was widespread understanding that they could be gamed. Interestingly, when participants spoke about strategic approaches to both using and generating metrics, many of them did so in terms of contrasting their own practices with the (perceived) practices of others. This kind of externalisation demonstrates that academia and research evaluation is ultimately competitive, and dissonance is created when incentives don’t align with personal or institutional values. The politics of authorship was another theme of this discussion and when asked about specific outputs, their associated metrics and how this contrasted with their personal experience of research excellence and impact, they often reflected on their specific role within a piece of work. Participants tended to value outputs according to the level of contribution they had made, regardless of the level of citation. For example, an output that was sole authored or co-authored with a PhD student might mean more to a researcher than a piece of work which was highly cited, but which they had made only a small contribution to, particularly amongst a large number of listed co-authors. It is clear that there is a tension between excellence as measured through metrics and excellence which is perceived personally by the researcher.
Because this research was undertaken in Aotearoa New Zealand, there was a strong focus on Indigenous research and cultural considerations. The tension between traditional scholarship and Indigenous research, coupled with the ever-present cultural labour imposed by modern (colonial) learning institutions means that the position of Indigenous researchers within our institutions remains fraught. For example, the pressure to publish in a “highly ranked” global journal often resulted in editorial processes that diluted or undermined Indigenous elements of the work to make it more “digestible” to a global audience within the strict confines of journal guidelines. This was contrasted against the desire to publish within Indigenous knowledge paradigms and for local audiences. While the project was limited in scope, the experiences of Indigenous participants captured in this study reflect wider issues in academia, including the reality that producing work that is meaningful to one’s own community will not always align with institutional expectations or standards of excellence. This is highly problematic in a world where Indigenous researchers are already underrepresented and overburdened.
We must build a research culture that allows people to tell their story in their own way and libraries have an important role to play in providing them with the tools to do so. Additionally, we must also shape an environment where researchers can trust that their stories will be received and evaluated in good faith. Our research left no doubt that there is a strong drive to do good research and to have the research do good. But this is at odds with the environment that research is conducted in, which often drives behaviours that undermine true impact, equity and excellence.
Metrics, like data more generally, make good servants and bad masters. We need to move towards a culture of measuring what it is we value, rather than valuing what we can measure, because research culture won’t change until incentives do.
The full report can be read here.