Facing a deluge of funding applications, the UK's primary research and innovation funding body, UKRI, is turning to artificial intelligence to ease the strain on its peer review process. This is a bold move, but is it the right one?
UKRI, which allocates over £8 billion annually to research, is grappling with a significant challenge. While the number of grants awarded has decreased by half in the last seven years, the number of applications has surged by more than 80%. This has created a bottleneck in the peer review system, where experts assess grant proposals.
To address this, a research team led by data scientist Mike Thelwall at the University of Sheffield is exploring the use of generative AI. The team, funded by the UK Metascience Unit, began their work in October. They will analyze data from up to 2,000 grant proposals, both successful and unsuccessful, to see if AI can predict the scores and decisions of human reviewers.
Thelwall's team will feed full-text versions of grant proposals into large language models (LLMs). The LLMs will not be given the original scores or funding decisions. The goal is to determine if the AI can accurately predict the outcomes of the peer review process. According to Thelwall, if the AI can reasonably predict scores, it could help speed up the review system or support reviewers.
Thelwall's previous work explored AI's role in refereeing articles for the UK's Research Excellence Framework. The team found that AI systems generated identical scores to human reviewers 72% of the time. However, Thelwall stated that this figure needs to reach 95% accuracy for the system to be viable.
But here's where it gets controversial... Mohammad Hosseini, a researcher at Northwestern University, raises concerns about AI's ability to generate novel ideas. He argues that if AI cannot create new ideas, it's unlikely to recognize truly creative ones. This is especially relevant in grant proposals, which are forward-looking and based on potential, not just past events.
And this is the part most people miss... Hosseini also points out that transparency is crucial. If funders aren't clear about the criteria fed into the AI, researchers may push back. Conversely, if the process is too transparent, applicants might try to game the system.
So, how might UKRI use generative AI? Thelwall suggests AI could be useful in tiebreaker situations, as an additional reviewer, or for a fast-track desk-reject option. He cites the la Caixa Foundation in Barcelona, which has experimented with AI-assisted grant peer review. In their case, AI helps pre-screen applications, but around 90% still undergo full peer review by human experts. Thelwall notes that even small time savings for reviewers can be significant, freeing up experts from evaluating proposals with low funding chances.
What do you think? Do you believe AI can effectively assist in grant peer review? Are there ethical concerns that outweigh the potential benefits? Share your thoughts in the comments below!
