Edited by Christopher Douce, Friday, 30 June 2023, 08:26
On 14 June 2023, I attended a professional development event that was all about the previous REF, the next REF, and the sharing of tips about how to prepare what is called a 4* paper. The event was facilitated by Rachael Luck, who is a lead for a design qualification, and Toni Gladding.
The REF is the UK Research Excellence Framework, which is a controversial system that ranks UK universities in terms of the quality of the research that they carry out. It was interesting to hear that both Rachael and Toni were both on review panels for the previous REF.
The most recent REF happened in 2021, and the next one is due in 2028. The broad aim of this session was to begin to think about preparation for the 2028 REF, to begin to consider what a 4* paper looks like, and to have a discussion with colleagues.
What is a 4* paper looks like is, of course, a highly debated and contested question. A point was: to try to increase quality for next REF, need to be thinking about it, and talking about it early.
The REF score matters. The higher the overall score, the more Quality Related (QR) income (UKRI website) an institution gains from the government. During this session, I learnt there is a multiplier effect, which means that 4* paper attracts 4 times the funding of 3* activity. A small improvement in the number of 4* papers that are published could have a large impact in QR funding. This funding is important since it is used to fund school doctoral studentships and university facilities. This, in turn, can increase the development of the university’s research capacity.
What follows is an edited set of notes I made during the session.
What happened in REF 2021?
University has a code of practice. If someone has a certain amount of research time, has to submit at least one output to REF 2021. There is a long tail of “other colleagues” who were also included. It is the quality of the output from those colleagues who might benefit from additional support. Since the rules for the 2028 REF have not yet been defined (but will be available in January 2024), there is a possibility that a wider group of staff might be included.
In the 2021 REF, 157 universities were assessed. In the OU context (if I’ve noted this down properly), 44% of staff only had 1 output, 10% produced 5 outputs.
What is a 4* output?
Each category of quality assessment has an accompanying “quality descriptor”. A three star paper is considered to be research that “is internationally excellent in terms of originality, significant and rigour but which falls short of the highest standards of excellence”. A 4* bit of research is “beyond internationally excellent”, but I’m not exactly sure what this means.
The university strategy is to maximise QR income, to increase its overall score, to aim to gain higher QR income. A strategy for an individual unit, such as a school, may be to try to gain the highest possible score with a specific assessment area.
An output could be chapters in books, design, exhibitions, journal articles, monographs, performance. This said, the largest and most significant output across universities and all units of assessment (UOA) are journal articles.
Key components of a high quality paper
This section of the session asked the question: what does a 4* paper look like?
One way to answer this question is to consider what the characteristics of a good journal article might be. A paper should have an eye catching title that has a broad appeal, and a clear abstract which highlights clear research questions, summarises methods, and offers a clear summary of the novel contribution made to knowledge.
Turning to the body of the article, a paper should clear clearly state research questions, clearly describe the methods that are applied (and share clear citations that offer more detail about them), explain how the research relates to theory, describe research instruments and ethical approach taken, provide details about the data samples, and conclude with acknowledgements. Not having dealt with ethical issues will limit the journals that you can submit to.
An important tip I noted down was: “know about the journal that you’re writing for”. See whether your research or research article is an argument that fits with the aims of the journal. A phrase I noted down was: “papers are not written in the abstract, they’re written with a journal in mind”.
This section of the presentation reminded me of a blog article that I wrote some years back, which had the title Getting published in Open Learning. Open Learning is, of course, a journal that I co-edit with two other colleagues. The points that were made about the clarity of papers is really important. A suggestion for anyone who is interested in writing papers, is to get involved with the peer review process in some capacity. This way you can learn more about how everything works.
During the session, the notion journal impact factors were mentioned, which are known as bibliometrics which highlight how influential a publication is within an area of research. When it comes to the REF, it isn’t just the impact factor of the journal that matters; the assessment of quality of paper is independent of the quality (or impact factor) of the journal. This is linked to something called DORA, which is an abbreviation of: Declaration On Research Assessment (DORA website), which the OU has signed up to, along with other higher education institutions (HEIs).
What can we change?
Research can be thought of in different ways. Research is, of course, carried out by individual researchers and academics, but since much research (especially within STEM require collaboration, a wider perspective is necessary. Individual academics should develop their own research strategies (which could be shared with others), but also strategies should be defined by units, such as schools, which then should be linked to REF units of assessment. This also links to the importance of developing a research culture.
An interesting question was: what practical things could be done? Some suggestions include working with others to ensure thar articles are thoroughly proofread before submitting to journals, perhaps setting up internal peer review processes within research groups, writing articles with co-authors, and setting up mentoring schemes with colleagues who are new to research.
The discussion of papers led onto a discussion about the challenges that accompany getting articles published in open access journals. Open access publishing means that the readers do not pay for access to research articles. Instead, researchers and writers pay. This means that authors may be asked to pay what can amount to a substantial publishing fee. This can limit the extent to which researchers are able to publish unfunded research, such as personal projects.
A suggestion is to always consider the cost of publishing fees when writing and preparing research bids, to make sure that they are considered during the budgeting. Also, if research is funded by a UK research funding council, the university library has an agreement that they will cover publication fees.
By way of contrast, Open Learning offers a hybrid scheme. The editors of Open Learning chooses to ‘open access’ one article per issue. Individual authors can also secure open access by submitting what amounts to a publication fee, but it is typically less than journals that adopt a completely open access funding model. The advantage of this approach is that it enables institutions that do not have extensive research budgets to disseminate their research. An important point here is: pay attention to the terms and conditions that relate to the journal that you’re hoping to publish through.
One of the final comments I noted was: there are tensions between helping those currently work on 3* papers, to help them to submit 4* paper, and helping those with little research time to start publishing. A big win for the university might be to enable a wider group of staff to be included in the REF. This also links to the tension between supporting the individual researcher, and supporting the institution as a whole to gain an overall higher rating.
Reflections
I don’t like the REF for the simple reason that terms such as quality and excellence are dangerously slippery terms. Also, research is something that takes place over an extended period of time, and certainly not across a neatly defined seven year cycle. I’m also grumpy after learning that 4* papers have a multiplier effect on research income for the simple reason that universities, schools and individuals who are already doing excellent research are likely to gain even more income to do even better research, with more resources. This multiplication factor is not progressive, and it feels far from equitable. Institutions with a lower research ranking will, of course, struggle more to make an impact in comparison to, say, Russell Group universities.
I come to this topic having struggled to build a research strategy and profile. As a staff tutor, half of my time is dedicated to academic leadership and management, and the other half is dedicated to ‘academic stuff’, which means either working on module teams or working on research. If I have module team (teaching) responsibilities, this means that less of my time is available to research, and what really helps with research is building those close collaboration with colleagues, and all this takes time.
Over the last few years, and to this day, a lot of my time has been spent dealing with internal institutional changes (mainly, the new AL contract), of which a lot still remains to be figured out and understood. Whilst this does sound a little grumpy, dealing with the practicalities of my staff tutor day job leaves very little practical time to carry out disciplinary research. Whilst I’m not contractually obliged to make a REF submission, the idea of leading a submission of an article feels current unattainable, never mind the notion of producing an article that is “beyond internationally excellent”.
What are the answers? One of the obvious answers is to work on research culture. This requires the involvement of multiple stakeholders, at multiple levels. It means facilitating conversations, and learning about the interests of others. It means getting involved and encouraging others to get involved. It means joining research groups, and being patient when listening to the perspectives, challenges, and experiences of others. This is also, of course, made harder when everyone is working at a distance.
Two thoughts that I return to regularly are as follows. The first thought come from the words of a former head of school who said: “if you have two reasons to do something, there is a greater likelihood that you’ll get that thing done”. Following this though, the accompanying thought is: “try to align everything together, into a personal strategy”. It is this alignment that I’m trying to do, and this may well mean saying “no” to some things.
Writing a 4* paper for the REF
On 14 June 2023, I attended a professional development event that was all about the previous REF, the next REF, and the sharing of tips about how to prepare what is called a 4* paper. The event was facilitated by Rachael Luck, who is a lead for a design qualification, and Toni Gladding.
The REF is the UK Research Excellence Framework, which is a controversial system that ranks UK universities in terms of the quality of the research that they carry out. It was interesting to hear that both Rachael and Toni were both on review panels for the previous REF.
The most recent REF happened in 2021, and the next one is due in 2028. The broad aim of this session was to begin to think about preparation for the 2028 REF, to begin to consider what a 4* paper looks like, and to have a discussion with colleagues.
What is a 4* paper looks like is, of course, a highly debated and contested question. A point was: to try to increase quality for next REF, need to be thinking about it, and talking about it early.
The REF score matters. The higher the overall score, the more Quality Related (QR) income (UKRI website) an institution gains from the government. During this session, I learnt there is a multiplier effect, which means that 4* paper attracts 4 times the funding of 3* activity. A small improvement in the number of 4* papers that are published could have a large impact in QR funding. This funding is important since it is used to fund school doctoral studentships and university facilities. This, in turn, can increase the development of the university’s research capacity.
What follows is an edited set of notes I made during the session.
What happened in REF 2021?
University has a code of practice. If someone has a certain amount of research time, has to submit at least one output to REF 2021. There is a long tail of “other colleagues” who were also included. It is the quality of the output from those colleagues who might benefit from additional support. Since the rules for the 2028 REF have not yet been defined (but will be available in January 2024), there is a possibility that a wider group of staff might be included.
In the 2021 REF, 157 universities were assessed. In the OU context (if I’ve noted this down properly), 44% of staff only had 1 output, 10% produced 5 outputs.
What is a 4* output?
Each category of quality assessment has an accompanying “quality descriptor”. A three star paper is considered to be research that “is internationally excellent in terms of originality, significant and rigour but which falls short of the highest standards of excellence”. A 4* bit of research is “beyond internationally excellent”, but I’m not exactly sure what this means.
The university strategy is to maximise QR income, to increase its overall score, to aim to gain higher QR income. A strategy for an individual unit, such as a school, may be to try to gain the highest possible score with a specific assessment area.
An output could be chapters in books, design, exhibitions, journal articles, monographs, performance. This said, the largest and most significant output across universities and all units of assessment (UOA) are journal articles.
Key components of a high quality paper
This section of the session asked the question: what does a 4* paper look like?
One way to answer this question is to consider what the characteristics of a good journal article might be. A paper should have an eye catching title that has a broad appeal, and a clear abstract which highlights clear research questions, summarises methods, and offers a clear summary of the novel contribution made to knowledge.
Turning to the body of the article, a paper should clear clearly state research questions, clearly describe the methods that are applied (and share clear citations that offer more detail about them), explain how the research relates to theory, describe research instruments and ethical approach taken, provide details about the data samples, and conclude with acknowledgements. Not having dealt with ethical issues will limit the journals that you can submit to.
An important tip I noted down was: “know about the journal that you’re writing for”. See whether your research or research article is an argument that fits with the aims of the journal. A phrase I noted down was: “papers are not written in the abstract, they’re written with a journal in mind”.
This section of the presentation reminded me of a blog article that I wrote some years back, which had the title Getting published in Open Learning. Open Learning is, of course, a journal that I co-edit with two other colleagues. The points that were made about the clarity of papers is really important. A suggestion for anyone who is interested in writing papers, is to get involved with the peer review process in some capacity. This way you can learn more about how everything works.
During the session, the notion journal impact factors were mentioned, which are known as bibliometrics which highlight how influential a publication is within an area of research. When it comes to the REF, it isn’t just the impact factor of the journal that matters; the assessment of quality of paper is independent of the quality (or impact factor) of the journal. This is linked to something called DORA, which is an abbreviation of: Declaration On Research Assessment (DORA website), which the OU has signed up to, along with other higher education institutions (HEIs).
What can we change?
Research can be thought of in different ways. Research is, of course, carried out by individual researchers and academics, but since much research (especially within STEM require collaboration, a wider perspective is necessary. Individual academics should develop their own research strategies (which could be shared with others), but also strategies should be defined by units, such as schools, which then should be linked to REF units of assessment. This also links to the importance of developing a research culture.
An interesting question was: what practical things could be done? Some suggestions include working with others to ensure thar articles are thoroughly proofread before submitting to journals, perhaps setting up internal peer review processes within research groups, writing articles with co-authors, and setting up mentoring schemes with colleagues who are new to research.
The discussion of papers led onto a discussion about the challenges that accompany getting articles published in open access journals. Open access publishing means that the readers do not pay for access to research articles. Instead, researchers and writers pay. This means that authors may be asked to pay what can amount to a substantial publishing fee. This can limit the extent to which researchers are able to publish unfunded research, such as personal projects.
A suggestion is to always consider the cost of publishing fees when writing and preparing research bids, to make sure that they are considered during the budgeting. Also, if research is funded by a UK research funding council, the university library has an agreement that they will cover publication fees.
By way of contrast, Open Learning offers a hybrid scheme. The editors of Open Learning chooses to ‘open access’ one article per issue. Individual authors can also secure open access by submitting what amounts to a publication fee, but it is typically less than journals that adopt a completely open access funding model. The advantage of this approach is that it enables institutions that do not have extensive research budgets to disseminate their research. An important point here is: pay attention to the terms and conditions that relate to the journal that you’re hoping to publish through.
One of the final comments I noted was: there are tensions between helping those currently work on 3* papers, to help them to submit 4* paper, and helping those with little research time to start publishing. A big win for the university might be to enable a wider group of staff to be included in the REF. This also links to the tension between supporting the individual researcher, and supporting the institution as a whole to gain an overall higher rating.
Reflections
I don’t like the REF for the simple reason that terms such as quality and excellence are dangerously slippery terms. Also, research is something that takes place over an extended period of time, and certainly not across a neatly defined seven year cycle. I’m also grumpy after learning that 4* papers have a multiplier effect on research income for the simple reason that universities, schools and individuals who are already doing excellent research are likely to gain even more income to do even better research, with more resources. This multiplication factor is not progressive, and it feels far from equitable. Institutions with a lower research ranking will, of course, struggle more to make an impact in comparison to, say, Russell Group universities.
I come to this topic having struggled to build a research strategy and profile. As a staff tutor, half of my time is dedicated to academic leadership and management, and the other half is dedicated to ‘academic stuff’, which means either working on module teams or working on research. If I have module team (teaching) responsibilities, this means that less of my time is available to research, and what really helps with research is building those close collaboration with colleagues, and all this takes time.
Over the last few years, and to this day, a lot of my time has been spent dealing with internal institutional changes (mainly, the new AL contract), of which a lot still remains to be figured out and understood. Whilst this does sound a little grumpy, dealing with the practicalities of my staff tutor day job leaves very little practical time to carry out disciplinary research. Whilst I’m not contractually obliged to make a REF submission, the idea of leading a submission of an article feels current unattainable, never mind the notion of producing an article that is “beyond internationally excellent”.
What are the answers? One of the obvious answers is to work on research culture. This requires the involvement of multiple stakeholders, at multiple levels. It means facilitating conversations, and learning about the interests of others. It means getting involved and encouraging others to get involved. It means joining research groups, and being patient when listening to the perspectives, challenges, and experiences of others. This is also, of course, made harder when everyone is working at a distance.
Two thoughts that I return to regularly are as follows. The first thought come from the words of a former head of school who said: “if you have two reasons to do something, there is a greater likelihood that you’ll get that thing done”. Following this though, the accompanying thought is: “try to align everything together, into a personal strategy”. It is this alignment that I’m trying to do, and this may well mean saying “no” to some things.