UK Strategic Forum - areas for further consideration

The UK Strategic Forum identified five main areas for the HEFCE to consider further in relation to the Research Excellence Framework.

  1. The definition of impact
    The definition of impact needs to be sensitive to the needs of individual disciplines and its potential uses, and should recognise that the types of impact will vary according to each individual discipline. While the Forum welcomes the broad definition given to impact in the consultation – covering social, economic, cultural, public policy and quality-of-life benefits – it considers that the definition should ensure that local, national and global effects are all captured and treated equally, not least because this is the most appropriate way in which the impact of particular disciplines is properly measured. For some disciplines, the nature of the area may mean that local and regional impacts are more appropriate; for others, the global impact is appropriate.
  2. Linking impact to high-quality research
    The Forum welcomes the HEFCE's commitment to ensuring that impact should always be related to research quality. However, it is often not straightforward to identify the impact of high quality research, especially as excellent research sometimes has low impact, and poor-quality research may have high, but not positive, impact.
    • Impact is often mediated through general reputation and expertise rather than being the result of an individual piece of work, which suggests that further consideration needs to be given to the assessment of esteem and its relationship to impact. For example, as a result of a portfolio of world-class contributions in a particular area, a social scientist might be asked to advise on the development of policy broadly, rather than on a particular project. This work may have great impact but could only be seen as indirectly drawing upon any particular piece of research.
    • Consideration should be given as to whether what were previously regarded as indicators of esteem should feature in the assessments of impact or environment as there is scope for ambiguity and possible duplication under the current proposals.
    • As many submitting units within the social sciences (approximately 40-50 per cent based on the RAE 2008 submissions) are relatively small (20 submitted staff or less), the assessment of the impact for a whole unit may be based on the work of only one or two researchers as it is proposed that there should be one case study in each submission for every 5 to 10 full-time equivalent (FTE) staff submitted. This may lead to problems. Issues such as these will need to be considered carefully.
  3. Sub-panel configuration
    The Forum supports the proposal that there should be fewer sub-panels for the social sciences, grouping together a number of social science disciplines in order to reduce the number of social science sub-panels from 12 to 9. As it is proposed that there should be standardised criteria, guidance and weightings for each unit of assessment, care must be taken to ensure that the new configuration of sub-panels maintains the intellectual integrity of the research base and does not have a distorting effect on the research undertaken in certain disciplines.
  4. A holistic approach
    The Forum is pleased that the HEFCE aims to build on – and learn from – the experience of the research councils of assessing impact. The ESRC, in particular, has played a leading role in developing different approaches and methodologies for assessing research impact, and is evidently willing to continue to advise and collaborate. The Forum also encourages the HEFCE to consider further the relationship between outputs, environment and impact, and the ways in which all three elements depend on and contribute to each other and are mutually reinforcing.
  5. Evaluation
    The Forum urges the HEFCE to undertake evaluation of the robustness of its policy initiatives, casting its expectations about the effect of those initiatives in testable form and making data available to interested researchers for analysis. It also urges the HEFCE to ensure that the findings from the impact pilot exercise are carefully evaluated in a transparent manner, possibly subject to independent scrutiny, and that reference should be made to the experiences of foreign funding agencies in this area.