Communication

Analyzing a Canadian Approach to Assessing Impact

January 16, 2018 1056

Consider this fictionalized story that appears in Approaches to Assessing Impacts in the Humanities and Social Sciences, a report published by the Federation for the Humanities and Social Sciences in Canada:

“A senior Canadian psychology professor has spent 15 years researching learning strategies for people facing various cognitive barriers. One day, she receives an unusual invitation: would she consider contributing to a not-for-profit research and social service network focused on skills training for new immigrants? Intrigued, she accepts, and over the next year works with a team of researchers and practitioners to develop teaching strategies designed to help new immigrants succeed in the Canadian labour market. The new strategies are put to work in the skill-training services offered through the network, which then collects data on various learning and employment outcomes – data our protagonist can use the next time she is asked to describe the impacts of her work. Of course, having worked as part of a large team, accounting for the impacts of her individual contribution will be a challenge.”

There are a number of important lessons we can take away from such a story. For instance, a researcher might work 15 years on a problem before a practical application is discovered. A network that connects researchers, practitioners, and users may be far better equipped to track the impacts of research than any one researcher is able to. But even when such a system produces great impact data, it might still not be possible to attribute those impacts back to an individual.

This story is one of four semi-hypothetical case studies we composed for our report to illustrate the many different forms that impact assessment can take in the humanities and social sciences. The characters in the stories are fictional, but the scenarios are all inspired by real activities taking place in Canada’s research community. While each of these stories helps to illustrate various useful lessons for assessing impacts, the message they send as a whole is perhaps the most important: they show just how diverse the world of research can be. This understanding underlies the approaches we recommend to assessing impacts, and leads us to recommend pluralistic and flexible assessment approaches that are able to account for this diversity and complexity.

Impacts in a Canadian context

The federation set out several years ago to help Canadian scholars navigate the thorny issue of impact assessment, recognising that we are not primarily a research organisation. Our goal was therefore not to materially advance the state of knowledge about how to assess scholarly impacts – we’re happy to leave that to the experts – but rather to bring together existing findings from research, help communicate its relevance to our community, and equip our members with knowledge and tools they can use in the challenging conversations about impact assessment that may lie ahead.

LSE-impact-blog-logo

This article by Peter Severinson originally appeared on the LSE Impact of Social Sciences blog as “Approaches to assessing impacts in the humanities and social sciences: recommendations from the Canadian research community and is reposted under the Creative Commons license (CC BY 3.0).

Our approach was based on a few key observations about how the impacts discussion is developing in Canada. First, while conversations about impacts are gaining prominence across the country, they are both diffused and obscured – taking place at different levels across our vast country and frequently out of public view. Canada’s research granting agencies employ and continue to develop various impact-assessment requirements (which, to date, appear to have only a minor effect on research funding decisions). Meanwhile, universities are increasingly including impact assessment in their strategic plans. Both trends are in part a response to increasing pressure from governments for more performance data on publicly funded programmes and institutions.

Our second key observation was that ongoing conversations about impact assessment in Canada are frequently hindered by uncertainty, anxiety, and suspicion (which, we understand, is a state of affairs not unique to Canada!). In particular, scholars in the humanities and social sciences express concerns that an assessment system based on narrow sets of inflexible metrics would skew the research landscape to the detriment of research disciplines whose impacts resist simple quantification. And, to be sure, some concern is warranted. Not knowing what requirements are coming next, it’s easy to imagine the worst, especially when exposed to heated debates taking place in other countries.

We also heard that university administrators and public servants tasked with exploring impact assessment want to develop responsible assessment practices, ones that don’t cause unintended harm. However, they reported being uncertain about how to accomplish this, even after attempts to consult with researchers.

Contributing to a better conversation

Putting these observations together, the federation saw an opportunity to enrich the Canadian conversation about impacts assessment. Our goal has been to summarize some of the key findings from research on scholarly impacts and present them in a way that addresses the hopes and concerns of the research community while also providing guidance to university administrators and public servants.

Our primary observation – as illustrated through our semi-hypothetical case studies – is that scholarly work is highly diverse and that efforts to assess scholarly impacts must account for this diversity. As a result, we recommend flexible and adaptable approaches to assessing impacts. This includes defining the concept of “impact” broadly, employing a diverse mix of impact indicators (both qualitative and quantitative) and ensuring that researchers themselves play a leading role in selecting the indicators that best suit their research.

We also recommend enhancing our ability to assess impacts by improving the ways we work inside and outside of universities. We recommend that research institutions, including universities and government agencies, provide the necessary resources to support the demanding work of impact assessment. We also recommend that assessment approaches recognise the contributions of non-academic partners, which play so vital a role in the pathways between research and impacts.

The approaches we recommend are intentionally broad. Applying these approaches to specific impact-assessment efforts will require work from researchers, scholarly associations, research funders and universities. Our hope is that our report will help these partners work together productively by providing them with some initial common ground based on a shared understanding of the key factors that affect impact assessment.

It’s an exciting time in Canada as these conversations grow and develop – we expect them to produce great things – and the Federation is thrilled to be making a modest contribution.

 

Peter Severinson is policy analyst at the Federation for the Humanities and Social Sciences. He tweets @pseverinson.

View all posts by Peter Severinson

Related Articles

Three Decades of Rural Health Research and a Bumper Crop of Insights from South Africa
Impact
March 27, 2024

Three Decades of Rural Health Research and a Bumper Crop of Insights from South Africa

Read Now
Daniel Kahneman, 1934-2024: The Grandfather of Behavioral Economics
News
March 27, 2024

Daniel Kahneman, 1934-2024: The Grandfather of Behavioral Economics

Read Now
Using Translational Research as a Model for Long-Term Impact
Impact
March 21, 2024

Using Translational Research as a Model for Long-Term Impact

Read Now
Coping with Institutional Complexity and Voids: An Organization Design Perspective for Transnational Interorganizational Projects
Research
March 19, 2024

Coping with Institutional Complexity and Voids: An Organization Design Perspective for Transnational Interorganizational Projects

Read Now
2024 Holberg Prize Goes to Political Theorist Achille Mbembe

2024 Holberg Prize Goes to Political Theorist Achille Mbembe

Political theorist and public intellectual Achille Mbembe, among the most read and cited scholars from the African continent, has been awarded the 2024 Holberg Prize.

Read Now
Norman B. Anderson, 1955-2024: Pioneering Psychologist and First Director of OBSSR

Norman B. Anderson, 1955-2024: Pioneering Psychologist and First Director of OBSSR

Norman B. Anderson, a clinical psychologist whose work as both a researcher and an administrator saw him serve as the inaugural director of the U.S. National Institute of Health’s Office of Behavioral and Social Sciences Research and as chief executive officer of the American Psychological Association, died on March 1.

Read Now
New Feminist Newsletter The Evidence Makes Research on Gender Inequality Widely Accessible

New Feminist Newsletter The Evidence Makes Research on Gender Inequality Widely Accessible

Gloria Media, with support from Sage, has launched The Evidence, a feminist newsletter that covers what you need to know about gender […]

Read Now
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments