Having been part of the emerging “impact profession” and followed the agenda closely since 2008, I found Smith et al.’s book an excellent account of the controversies, consequences and challenges that has risen from the impact agenda. I agree with their alternative and broad approach to supporting and incentivising research impact, and hope it gains support institutionally.

Accepting that we live with a result based framework of funding research, I see the assessment side as the main challenge though. I am not sure the alternative approach provides solutions to how we can better assess impact meaningfully with the transparency and rigour required.

For example, I am not sure how you would assess and reward activity if we don’t have evidence of its effectiveness, for example with regards to public engagement. NCCPE did, however, show that nearly half of case studies in REF2014 mentioned public engagement as a route to the claimed impacts, so I think this may be a slightly exaggerated problem. Learning from failure is really important, and from my experience this needs to take place in trusted spaces, so would be hard to demonstrate and reward in REF.

Some panels went too far in expecting a causal link between research outputs and discrete impacts in REF 2014. The current guidance is more specific on the many non-linear ways research can contribute to impact, but internal review panels may find these more risky to pursue. Likewise synthesis of research would met the quality threshold in most panels, but institutions may put additional barriers. I think though that weakening the link between research and impact would be unhelpful in an assessment of research. The impact of universities is of course much wider, but this is already covered through outcome agreements in Scotland, and through the Teaching and Knowledge Exchange Frameworks in England. I think a positive outcome of the research impact agenda is how many academic colleagues have focused their engagement around their research rather than seeing this as separate activities.

Whilst challenges remain with assessment, I think the eight principles, including those mentioned above, provide an excellent basis from which to build a positive impact culture. I share the authors’ worries that academic rigour and autonomy, as well as discovery based research, are challenged in the current environment. This is short sighted, as I’m convinced a research environment that combines the multitude of approaches both to research and to impact is the one that will make the most difference to the world; and also succeed in REF. The academic community, and the “impact profession” I represent, have a common course in advocating for this. Both to national stakeholders, and locally if we see management taking a narrow path to impact focussing solely on developing REF case studies.

–

Anne-Sofie Laegran is the Head of Knowledge Exchange and Impact at the University of Edinburgh Research Office, promoting collaboration and engagement between academia and stakeholders across different sectors.

Earlier this year (against the difficult backdrop of the global COVID-19 pandemic), Policy Press published our book, The Impact Agenda: Controversies, Consequences and Challenges, co-authored with Nasar Meer, Ellen Stewart and Richard Watermeyer. This book brings together earlier work that we had developed in discussion with SKAPE colleagues.

Our aims were:

• To bring together disparate work on the impact agenda to critically reflect on the controversies, consequences and challenges that are arising;
• To reflect on our own role, as academics, within this;
• To collectively propose an alternative approach.

And the last point was the focus of our recent SKAPE seminar, and this linked blog, in which  we set out an alternative approach to research impact – one which we feel is more evidence-informed and sustainable and which, while inevitably still imperfect, addresses the major drawbacks of the current approach.

What might a more evidence-informed approach to supporting, assessing and incentivising research impact involve? 

Since we each brought our own experiences, preferences and convictions to the book, it was not easy for us to identify a way forward that we could all support. However, we all agreed that the following changes would better reflect what we know about both the complex relationship between academic work and real-world change and the consequences of audit regimes and performance assessment.

1. Reward impactful environments, rather than individual achievements: The requirement of impact case studies to demonstrate and document change inevitably narrows the scope of impact activities and outcomes. We should, instead, strengthen a focus on how universities create impactful environments; workplaces that are outward-looking, open and engaged with the world beyond academia.
2. Value a wider range of activities, especially around public engagement: The NCPPE is leading the way in both supporting academics to do public engagement better and encouraging the research audit process to value it fairly. There is more work for to do and, we suggest, it may be desirable to go further, recognising the impacts of the university’s wider role as an ‘anchor institution’.
3. Protect spaces and funding for critical and discovery focused academic scholarship (without obvious impacts): There are many examples of academic scholarship that is valuable for reasons other than impact. This includes critical, theoretical and experimental work (some of which, as several interviewees pointed out, contributed to major impacts at later dates).
4. Reject crude and simplistic classifications of ‘excellence’ (which, for example, denigrate the local): The quality of research in REF promotes the idea that wider geographical relevance equates to higher quality. Although most recent impact case study guidance notes that there is value in ‘having a big impact on a small group of people’, our interview data suggest academics tend to believe that those focusing on local impacts are not viewed as “impact stars”. If we want universities to be active members of their local communities, this could, and should, be changed. It should not be the proximity of external communities that is key to assessing excellence, but relatability to potential research users such as communities of policy and practice.
5. Weaken the link between original research and impact to encourage knowledge synthesis and collaboration: There are very good reasons to support mechanisms that allow for bodies of work to achieve greater influence than single studies. Yet the approach impact of both REF and the UKRI funders appears to do much more to encourage the impact of individual research projects than work to pool and synthesise knowledge for external audiences. We suggest that research funders and REF assessors are encouraged to do more to value academic scholarship that focuses on knowledge synthesis.
6. Develop a conversation about the ethics of impact: The approach to research impact being taken in the UK and elsewhere appears to assume that if research is ‘excellent’ then the impacts will inevitable be positive. Yet, there are plenty of examples in which excellent research has had deleterious societal impacts so we need to develop conversations and tools that allow us to meaningfully consider the ethics of research impact.
7. Defend and promote academic rigour and autonomy: Researchers already exist within governments, NGOs, think tanks, private companies and often produce rapid, responsive research. We worry that some of the impact incentives encourage academics to shift towards this kind of responsive research to such a degree that it risks blurring the role of academics with consultants. What exactly is unique about academic research and scholarship will vary by discipline and field but, if we want to maintain a distinction, we all need to get better at articulating and valuing our USPs.
8. Create spaces in which valiant failures are celebrated and learned from: The current form of impact assessment in REF, and the high financial value of impact case studies, combine to prompt institutions to focus on tried-and-tested pathways to impact. We suggest funders and universities should do more to promote innovation in engagement and knowledge exchange, encouraging contributions that are about learning from challenges and failures (as well as successes).

The above suggestions are not exhaustive but they are intended to be a starting point for discussing how we might improve the current approach to research impact. We, as academics, are involved in constructing, enacting and reviewing impact’s performance indicators and we therefore have opportunities to reshape and improve the current approach. These are opportunities we should take.

–

Kat Smith is Professor of Public Health Policy at Strathclyde University, with a longstanding interest in the relationship between evidence, expertise, policy and practice, especially for issues relating to public health and inequalities.

Justyna Bandola-Gill is a post-doctoral researcher at METRO, where she explores the production and governance of poverty indicators, with a wider interest in how knowledge is organised, governed and mobilised across different settings.

The past couple of months have increased the need for accurate, and transparent, tools that allow policymakers to track and forecast the behaviour of the pandemic we are going through. For instance, different groups of researchers in the UK have used machine learning (ML) algorithms to forecast the type of treatment a person should receive based on the first days of infection [see: 1, 2, 3]. In a broad sense, ML forecasting tools refer to a set of algorithms designed to process data and find patterns. Part of the authority set on quantitative tools (QTs), such as forecasts, is on the idea that as a way of transparency, they can be replicated. However, the notion of replicability understood as obtaining the same results as the original experiment might not be achievable. From a social perspective, a lack of replicability would impact the possibility to challenge a forecast by those been affected. In the case of QTs, at least three elements might not allow their replicability:

• The existence of mundane everyday processes that impact the production of these tools;
• The amount of tacit knowledge among researchers which is not transferred; and,
• Existing infrastructure inequalities where not everyone can have access to the technology required.

My research explores these issues through participant observation during the construction of the Violence Early Warning System (ViEWS). This tool, produced at the Uppsala University Department of Peace and Conflict Research (PCR), aims to forecast the probability of political conflict within the next 36 months; and, of climate-related conflicts within 100 years. Rooted in broader discussions, including SDG 16 and the IPCC, its producers expect to provide an accurate tool for interventions. The expected users of ViEWS include the EU Commission, Western governments and the World Bank. Understanding that their objective could impact people’s lives, ViEWS’ researchers claim to provide a replicable tool as a matter of transparency.

For some at ViEWS, a QT will be understood as replicable when others can intuitively understand the way algorithms deal with data. This implies that it is not enough to provide the instructions, but there needs to be a conscious understanding of the way a tool operates. However, these same researchers acknowledge that the degree of tacit knowledge among individuals is so high, that even within the team, the sudden absence of particular members could avoid starting the project from scratch. For instance, a ViEWS’ researcher told me how, even for them as a team, it can be impossible to replicate the tool:

“I don’t think that anyone would be able to replicate it from scratch. Like in the variables, they would run like copy-paste but trying to figure out what each part does, that might be difficult.”

In the case of the infrastructure required for the production of ViEWS, it is acknowledged that most countries do not possess it. Therefore, most of the countries that have been used as case studies to prove a methodology might not be capable of challenging a measurement. This imbalance between who can measure and who can challenge perpetuates power relations now being reinforced through unreplaceable algorithms. Another ViEWS’ researcher told me that most countries do not possess the supercomputers they need for the construction of ViEWS. Therefore, only rich countries could challenge the results -by inspecting the algorithms behind a policy tool.

In sum, there is an urgency to start the discussion on the social consequences the limitations around replicability of ML and algorithmic tools could have. While those producing QTs claim that any wrong forecast would be discarded through an existing ensemble of other tools used by policymakers, this does not increase the ability of local communities to challenge the algorithms. Acknowledging that these gaps might be impossible to solve in the short-term, QTs’ producers need to involve local communities [4] during the production of their tools. For instance, there is a need to translate the production of algorithms to non-technical terms.

—

Antonio Ballesteros is a 4th Year PhD candidate in Science and  Technology Studies (STS). His research focuses on analysing the role everyday, mundane events play during the construction of environmental policy quantitative tools (rankings and machine learning).

1. Knight, S.R., et al., Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score. BMJ, 2020. 370: p. m3339.
2. Menni, C., et al., Real-time tracking of self-reported symptoms to predict potential COVID-19. Nat Med, 2020. 26(7): p. 1037-1040.
3. The Guardian, Is it possible to predict how sick someone could get from Covid-19? , in Science Weekly, N. Davis, Editor. 2020, The Guardian.
4. Visvanathan, S., Knowledge, justice and democracy, in Science and Citizens, M. Leach, I. Scoones, and B. Wynne, Editors. 2005, Zed Books: London.

Policy revolution

There’s been a revolutionary turn in energy policy and research circles. Organisations such as the Energy Systems Catapult, the Energy Institute, and consultancy firms such as PwC have all suggested that the UK energy system is facing a sweeping energy revolution. As part of its wider industrial strategy, the UK Government now channels much of its energy innovation spending under a ‘Prospering from the Energy Revolution’ programme, including an academic-led Energy Revolution Research Consortium.

While there’s an exciting and important role for research working at the cutting edge of change, it might also be worth questioning the inevitability of a wholesale energy revolution. For many, the defining issue in energy policy, and policy-driven research, is how to rapidly decarbonise while providing secure and affordable energy for all, and reaping maximum social and economic benefit along the way. Whether change is best achieved by revolution or reform is an open question, to be decided on an issue-by-issue basis using the evidence at hand.

Clouded judgement?

Studies of the history of technology suggest that revolutionary change is a common way of thinking about the future. The Danish Science and Technology Studies scholar Kasper Schiølin recently referred to ‘future essentialisms’ that sanction certain modes of change while discouraging others. That sanctioning effect is why some researchers and advisors might be wary of signing-up to a sweeping energy revolution prospectus. Upfront overarching commitments – to either revolution or reform – may cloud independent judgement on the best way forwards.

Is a revolutionary vision common?

To find out whether a revolutionary vision of the UK’s energy future is widely shared among experts, we carried out a large online survey of UK energy academics and stakeholders. We discussed some of our findings in an earlier UKERC post and research briefing. In a newly published paper in Energy Policy(part of an upcoming Special Issue from UKERC researchers) we look more closely at expert responses to the energy revolution proposition.

The results suggest that most academics and stakeholders aren’t committed energy revolutionaries. We found widespread recognition of the growing importance of digitalisation and decentralisation in the energy sector, seen in the rise of ‘smart and local’ energy systems. At the same time, many existing technologies and organisations – adapted to the transition challenge – are also expected to play important roles. Large scale technologies and networks are seen as key contributors to the transition, as are some rather mundane innovations, such as improved buildings fabric.

System change, a mix of disruption and repurposing

Under one-fifth of our survey respondents showed a strong commitment to a wholesale energy revolution. Most saw energy system change as involving a mix of disruption and repurposing, and many highlighted complementarities between established and emerging solutions, such as the way large scale infrastructure can offer resilience for local systems. Interestingly, these views were held by the majority of both academic and non-academic respondents, and across the many different disciplinary groups we surveyed.

Transition over revolution

Despite the revolutionary turn in energy policy and research, our survey suggests that most experts are pragmatic problem solvers. For most, transition rather than revolution is the defining issue in energy policy and research.

The new journal paper is available to download from here. It can be cited as: Winskel, M., Kattirtzi, M., 2020. Transitions, disruptions and revolutions: Expert views on prospects for a smart and local energy revolution in the UK. Energy Policy 147, 111815, https://doi.org/10.1016/j.enpol.2020.111815

In addition to the expression of thanks in the Acknowledgements section of the article, the authors wish to thank two anonymous referees for their detailed comments and suggestions on an earlier version.

This blog was originally published on the UK Energy Research Centre blog and can be found here.