OU blog

Personal Blogs

Picture of Christopher Douce

Introduction to the REF

Visible to anyone in the world
Edited by Christopher Douce, Wednesday, 24 Oct 2018, 10:42

In November 2018, I had an opportunity to attend something that was called a ‘writing retreat’. The idea behind the event was simple; it was an opportunity to take a bit of time out from day to day activities and focus on writing up various bits of research that colleagues within the school had been working on. There was another reason for running the retreat: there would be a particular emphasis on writing papers that could be submitted to the 2021 REF (the Research Excellence Framework).

What follows is a brief summary of some of the points that were made during the introductory workshop which introduced the REF. Full acknowledgements are extended to Professor Jane Seale who facilitated this workshop. Many of the words here are directly from Jane’s presentation.

Introducing the REF

A key phrase that I’ve heard ever since I’ve been working at a university is: REF submission. A submission relates to particular subjects. Some universities may focus on some submission areas over others to play on their own strengths and weaknesses.

The OU makes a number of submissions, and one of the submission areas that I am connected with is education. This means that ‘education-like’ papers will be grouped together, submitted, and then assessed by an expert panel. The outcome will be a ‘research rating’, and this is directly linked to income that is received by the university. Simply put, the higher the rating, the more research income an institution receives. 

I asked the question: “what does ‘education-like’ actually mean?” These will be papers that more than just describe something, like a system or a tool that has been designed (which is what some computing papers can be). Education research papers need to be firmly linked to an education context. They should also present a critical perspective on the literature, the work that was carried out, or both.

When?

The 2021 REF assesses research that has been published in the public domain that has been carried out between 1 January 2014 and 31 December 2020. Papers that are to be included are typically included within a university repository. (The OU has a repository called ORO).

What?

Every academic whose contract which has a significant research component should submit at least one paper, and there should be a REF average of 2.5 papers per academic (as far as I understand things!)

Publications should be of a 3* or a 4* quality. Three star papers means that a piece of research is of ‘international significance’. The education REF panel will accept different kinds of submissions, including journal papers, conference proceedings and book chapters, but there has been a historic bias towards journal papers for two reasons: they are peer reviewed, and are more readily cited. Books or book chapters should present summaries of research and shouldn’t be student focussed or summarise what is within a field.

In education, some journals contain practice papers or case studies. I noted down that education papers don’t have to be empirical to be considered for the REF. A paper might present a new practice or an innovation or development of an existing practice. A suggestion is to preface it with what you’re doing and why you’re doing something, and offer a thorough criticism of how and why something fits in with existing work. This means that it’s necessary to consider comparisons and contrasts. Descriptions are also necessary to contextualise the research, but the balance needs to be right: practice papers need to be generalisable.

Where?

A question to ask is: where should you publish? It was interesting to hear that the REF panel for Education isn’t particularly concerned with the impact of the journal where research is published; what matters is the research itself. Given that education research tends to be descriptive, a suggestion is to choose journals that have a generous word count. One such journal is: Open Learning, which is thoroughly excellent. 

How?

How are submissions assessed? The submission chair will look at the title, abstract and reviewers and match a set of papers to an expert. Every paper will get read twice, and there will be some kind of process of random sampling. An interesting thought is: ‘don’t make the reviewers work too hard’, which is advice that I also give my students who are writing their end of module assessments or project dissertations. Institutions (including the OU) may having something called a ‘mock REF’, where they try to replicate the official REF submission to get a feel for the direction where the institution is heading.

REF criteria

Papers are judged on three key criteria: significance, originality and rigour; each of these criteria are equally important.

Significance: A paper or piece of research provides a valuable contribution to the field (this relates to the ‘so what?’ question about the purpose of the research). Also, how does the research moves the field forward. The contribution can be theoretical or empirical. A point to note is that a 3* paper contributes ideas that have a lasting influence. A 4* paper has a major influence on a field.

Originality: Is the research engaging with new or complex problems? Perhaps a paper might be using existing methods but in a new way, or challenging accepted wisdom. As a point of reference, a 2* paper contributes a small or incremental development, where as a 4* piece of research is something that is outstandingly novel in the development of concepts, techniques or outcomes.

Rigour: Papers and research offers intellectual precision or robustness of arguments. Some important questions are: is there rigour in the argument, the methods and the analysis? Also, can the readers trust the claims that are made when you ask the question, ‘how have you analysed your data?’ It is also important to include researcher reflexivity (question the role that the researcher had in the analysis of the research), and to critique the work, offering a summary of strengths and weaknesses. A 4* paper is one that is exceptionally rigorous, with the highest standards of intellectual precision.

Reflections

I only have a small amount of time in my staff tutor contract to carry research; I try to do what I can when I can, but I very often get entangled in tutor and student support issues which take me away from exploring really interesting questions and topics. Plus, increasingly, I’ve been playing a role in AL professional development. Carving out time for research is difficult balancing act due to all these competing demands.

When I started the retreat, I planned to write about some of the tutorial observation research that I’ve been doing and reacquaint myself with some of the papers that I had uncovered whilst doing this research. The introduction to the event gave me some really needed context, in terms of what a very good paper actually looks like. It also gave me a bit of direction in terms of what I actually needed to do.

One of the things that I did during the retreat was to consolidate different bits of research into a single document that forms the basis of a paper. I now have a clear and distinct structure. What I need to do now is to do some further close reading of the references to more directly position it within the literature, add a more critical twist to the analysis to broaden its appeal, and to do quite a bit more editing.

Permalink Add your comment
Share post
Picture of Christopher Douce

Staff tutor conference, December 2017

Visible to anyone in the world
Edited by Christopher Douce, Sunday, 14 Jan 2018, 11:58

This post is a personal summary of a two day staff tutor conference that took place at Horwood House, Little Horwood between 5 and 6 December 2017. I’m blogging it for three reasons (1) so I can remember what CPD I’ve done when I have to do my appraisal, (2) it will help me to remember some of the important discussions that took place, and (3) it might be randomly helpful to someone!

Introduction

The conference began with a session by the outgoing associate dean of regions and nations, Steve Walker, a fellow staff tutor from the School of Computing and Communications. Steve presented a sobering description of some of the challenges that had to be addressed over the previous three years: the closure of the university regional centres, the introduction of a new tuition approach called the group tuition strategy (GTP), and the merger of faculties to create a new set of schools. A further change on the horizon is, of course, the potential introduction of a new associate lecturer contract.

Since the last staff tutor conference, I am now a home worker, there is a new student support team (SST) which is based in Manchester, and there is a venue management team that is based in Wales, and a system called the Learning Event Management system (LEM) which is (I understand) less than ideal. Also, AL development has been reconfigured (or, should I say, centralised) to create an entirely new way of working. 

Looking back to when I started in this role back in 2010, geography isn’t as important as it used to be. Support for students is instead organised in terms of curriculum teams as opposed to regional support teams. This means that the connections within schools have been strengthened and that there are more opportunities for activities such as associate lecturer development.

A number of important external drivers were emphasised:  the part time student market is receding due to government policy and fees, the student demographic is changing (they are getting younger), there are new entrants into the HE market and this means increased competition, the government has introduced the degree apprenticeship and the apprenticeship levy. There is also the Research Excellence Framework (REF), the Teaching Excellence Framework (TEF) and even talk about something called the Knowledge Exchange Framework (KEF). There has been a lot going on!

Student’s first transformation

During this period the university has defined a ‘student’s first strategy’ and has realised that it needs to address a potential income deficit due to the fall in part time numbers. In response, the ‘student’s first transformation’ programme has been set up. The programme aims to achieve a number of things: increase student retention, increase student progression and increase student satisfaction scores, whilst at the same time saving 100 million pounds (a figure that has been chosen by senior management).

To make these changes faculties have been asked to review their curriculum and other divisions of the university will be asked to review their working practices. These practices will be defined in something called a new ‘university operating model’ and the university will create a new ‘teaching framework’. Members of the university will be asked to participate in project groups and teams. Change, of course, has its own costs.

One phrase I regularly hear that is connected to the transformation programme is ‘digital by design’. A personal view is that I’m not quite sure exactly what this means; I know that students study in different ways using different technologies. Whilst I’m technologist, I’m a great believer in the usefulness of printed materials. On the point of ‘being digital’, the JISC Building digital capability project was mentioned (JISC).

During this part of the conference another point that I noted down was the term ‘sustainable academic communities’. Again, I’m not quite sure exactly what this refers to, but I did make a note of the principle that perhaps associate lecturers should be more involved and connected to schools. This discussion about associate lecturers takes us to the next part of the conference, which was about debating the purpose of tutorials.

Purpose of tutorials

Three staff tutors were asked to present perspectives about tutorials. The perspectives were: face to face tutorials, online tutorials, the use of pre-recorded tutorials and peer tutorials. I began by talking about face to face tutorials.

Face to face tutorials

Face to face classes, in my opinion, are the best and most effective way to teach. Here are some reasons why:

  1. What really matters in education is, of course, people. Face to face tuition is all about putting student’s first for the simple reason that technology doesn’t get in the way.
  2. Face to face means that learning is personalised to individual students or the group of students who are attending a tutorial.
  3. Tutors can ‘see’ the effect of their actions; they can see the learning that takes place. To assess the effectiveness of their learning tutors can quickly ask questions, and this allows them to correct and develop understandings.
  4. In adult learning, students can themselves become teachers. Students can arrive at class with significant and relevant real-world experience that can be discussed within the class. In some ways, students are their own resources. A skilled tutor can help to make connections between different students and topics.
  5. Face to face gives tutors and teachers opportunities to innovate and to try new things out since they can more directly and readily understand what students feel comfortable with. Tutors can more easily facilitate role play events, facilitate discussions and create tasks that make use of the physical space of a tutorial room.
  6. In a face to face session, a tutor can easily see if something is wrong; they can pick up on blank expressions and uncomfortable body language.
  7. Face to face tutorials benefit all students, irrespective of whether students attend the sessions. The reason for this is simple: a tutor or teacher has to have a good command of their material. 
  8. Following on from the above point, students will more readily question or challenge tutors during face to face sessions, helping themselves to understand how the teaching materials can be understood from different perspectives.
  9. Face to face tutorials are real in a way that other methods are not; tutors can offer direct and personal encouragement to students.

I understand that there are some debates within the university about the cost of certain types of face to face tutorials, since they represent a significant proportion of the tuition budget. During this session I argued that, from an educational perspective, the university can’t afford not to provide face to face tuition.

Online tutorials

The second presentation about tuition events was by Diane Butler from the School of Health, Life and Chemical Sciences. Diane said that online tutorials, like face to face sessions, help students to understand module materials, allow students to interact socially with tutors and other students and to develop skills. Online sessions allow tutors to provide tailored support and, when travelling is difficult, help to alleviate a sense of isolation.

Diane also highlighted some of the challenges: online sessions can very lend themselves to be centred upon PowerPoint presentations, and given the complexity of driving online sessions, tutors can easily rely on the simplest tools. A common issue (and one that I regularly hear of when speaking to tutors) is that students are very reluctant to use microphones. It is a rare session that tutors present a series of activities; it can also be difficult to fully appreciate the learning needs of students.

An important point was that online tutorials don’t easily lend themselves to constructivist learning. During Diane’s presentation, I made the note: ‘we put our tutors in an environment where it makes it difficult’, and ‘[there may not be] much benefit to watching the recordings versus attending live’. A point was that perhaps the module team ought to spend more time creating recordings.

Another point was: ‘we place an unrealistic burden on our tutors to facilitate group work’, and that ‘we need to take online teaching to the next level’.

Pre-recorded tutorials

Hayley Ryder from the School of Mathematics and Statistics shared her experience of creating pre-recorded tutorials for a whole module cohort. An interesting point that Hayley made was ‘students like me to make mistakes’; this can share something about what really happens when people ‘do mathematics’. By exposing the challenges that accompanying studying maths, and demonstrating things that are ‘actually hard’ may encourage students to adopt a growth mindset: if the lecturers make mistakes, then it’s okay that I make them too!

My understanding was that the module team might have prepared a set of recordings once and then rolled them out for different presentations, but this wasn’t happens: new recordings are made every presentation; different people and different cohorts get stuck on different things. 

Hayley’s videos were fun and personable; her presentation got me thinking.

Peer tutorials

The final presentation was by Katherine Leys who is from the School of Life Health and Chemical Sciences. Peer tuition and peer support is mentioned in the new teaching framework. Peer tutorials is all about having students teach each other. Examples of these might be a group of students working together to plan an experiment in an online room or a forum. To facilitate these discussions, students might adopt roles, such as leader, deputy leader and so on, but tutors do need to mediate and some students will not be able to take part, perhaps due to the need to make reasonable adjustments to take account of disabilities or impairments.

Supporting staff tutor practice

Staff tutors offer support to associate lecturers, module teams and students. An interesting question to ask is: what can we do to help to support ourselves and fellow staff tutors? Maggie King, Staff Tutor in the School of Engineering and Innovation began this session by saying that the ‘changes to our jobs have been monumental’. In some respects, our job has changed from being a job that was primarily about people (and our academic discipline), to a job that is increasingly about administration.

We were asked a question: what could we do to make things better? Some suggestions included: the need for further staff tutor staff development, and opportunities for further sharing of practice, perhaps on a monthly basis or at different ‘regions’ or ‘regional areas’ (which could be, potentially, booked and organised through the venue management system). A personal reflection is that constant and effective communication between each other is the only way that we can continue to work well and understand the changes that are emerging through the university.

External Engagement

The title of this session was: ‘it’s good to get out’. It was facilitated by Matt Walkley, Staff Tutor from the School of Computing and Communications. Matt posed three questions that guided group discussions: what external engagement have you done? What external engagement should we be doing or be doing more of? And, what actions can we commit to?

A personal reflection was that I have always struggled to understand what was meant by the term ‘external engagement’; I was always under the impression that it had a more formal definition than it had. I was imagining external engagement as sets of targeted and strategic activities that can be used to help gain insights that could feed into module production, or to make contacts with individuals within organisations that could explicitly benefit from learning can be presented through OU programmes or modules. Although all these things do fall under the topic of external engagement, they are more directly aligned to the more formal concept of ‘knowledge exchange’.

External engagement can, as I understand it, broadly mean ‘getting out there’ to make contact with people to help us learn more about our subject and also to raise the profile of ourselves and the university. In some respects, being an external examiner and visiting the occasional British Computer Society event both represent different types of external engagement.

I learnt that external engagement is an official part of our contract. This means that we can allocate between 3 and 5 days into our work plan for external engagement activities (although there was some debate about exactly how many days we might have available). A point was if we can align our own personal and professional interests to that of our School and Faculty, then so much the better.

Another point I learnt was that each school has its own external engagement officer. I had to ask who the officer for my school was. This, in turn, made me realise that since staff tutors are scattered across the UK, the staff tutor body is at an advantage when it comes to engaging with national initiatives. A thought is that it is up to schools to develop and apply an interesting external engagement strategy; staff tutors are a resource that can be used, and used effectively (when we have the time, of course, and we’re generally pretty busy).

Improving retention and progression

The subtitle for this session that was held on the second day was: what can staff tutors do? Different staff tutors summarised different aspects of their experience. What follows is a very brief summary of each presentation.

U116 Environment: journeys through a changing world

Christine Pearson spoke about U116 and its ethos: the module assumes that students don’t know anything (that they are entirely new to the subject). I made a note that the students are changing, and also have a broad range of digital literacy skills. An interesting point was that some students were invited to take place in a focus group; a key point were that there was a need for induction or some kind of ‘fresher’s week’. Subsequently, a ‘getting started’ video was made, and students were sent a postcard to help them to get online. Another ‘bit’ was a numeracy podcast. An important point was that more and more students are doing concurrent study, which might be a side effect of the loan scheme.

Impact of presentation patterns

Bernie Clarke spoke about the impact of 22-week presentation patterns, where two modules are taught back to back (which is going to be the case for the new Computing modules TM111 and TM112). Using this presentation pattern, students gain credit at an earlier point in their studies. Another change is that in the engineering curriculum the teaching of mathematics is now done within the context of the subject, rather than students being asked to study a maths module that has been written by colleagues in a different school.

M140 early start initiative

Alison Bromley spoke about the effect of enabling students to start earlier on a module (another colleague, Carol Calvert gave a presentation about this same subject at a HEA Conference in April 2017). Over two hundred students accepted the opportunity to start M140 early. Those that started apparently really appreciated the opportunity for early tutor contact. I didn’t note down the detail of the impact, but I did note down that there was a difference between students who were studying for the very first time and students who had gained further experience through study.

Learning analytics and interventions

Nicolette Hapgood, chair of S111, reported that S111 applied an assessment approach that is known as ‘single component assessment’ (which is also going to be applied on TM111 and TM112), which means that the result is based on completing only assignments and not an end of module exam or assessment. Nicolette also described the availability of a data analytics tool that is now available to all module teams; this tool enables module teams to see differences in retention between different module presentations.

An important question to ask is: what is the main role of staff tutors when it comes to improving student retention and progression? An answer I’ve noted down is: our role is to support the associate lecturers who are closest to the students. We also represent an important link between the tutors and the module teams. Some other discussion points related to the knowledge management system that is used by the SST (which is used to offer study advice), the importance of reminding tutors about study support resources (there is an earlier blog about study skills resources), the importance of induction (which remains a mystery to me), and helping module teams to write and develop module materials. 

What will the REF mean to staff tutors?

I’ve forgotten who presented this final section of the conference, which was about the Research Excellence Framework.

There are some differences between the 2014 REF and the 2021 REF. One of the key differences lies in a statement that all staff who have significant time and resources to carry out research are to be submitted into the REF. There is a clear contractual difference between different categories of university staff: central academics are required to do research as a part of their contract, whereas staff tutors are ‘encouraged’. Subsequently, there is an ambiguity as to whether staff tutors will be included into the REF submission.

The reason why staff tutors are only ‘encouraged’ to do research is simple: workload. Time that we could have spent on research is spent supporting and developing associate lecturers and dealing with a whole host of administrative issues. Over the last year, it has more or less been a full time job keeping up with institutional changes, never mind doing institutional research.

My view is that there are two things that could be done to help to tick the research box: rather than doing discipline specific research, one possibility is to do scholarship and research into teaching and learning (since this fits closely with the role of a staff tutor), and secondly, if disciplinary research is important, another approach is to team up with central academic researchers. 

Reflections

This is the second or third staff tutor conference that I’ve attended. Typing everything up helped me to look back and to put a lot into perspective; a lot has changed. As mentioned at the start, the majority of the regional centres in England have closed and the way that tutorials are organised is now very different to how it was before. 

Put another way, I’m now doing a different job to what I was doing six years ago. I’m not going to pretend that homeworking is easy: it isn’t. 

This said, putting difficulties aside, there are some good things about this new way of working that many of us have had to embrace. A final thought is: it was really useful to spend time with so many colleagues; they are a pretty fabulous group of people to work with.

Permalink 3 comments (latest comment by Jonathan Vernon, Friday, 6 Apr 2018, 22:25)
Share post

This blog might contain posts that are only visible to logged-in users, or where only logged-in users can comment. If you have an account on the system, please log in for full access.

Total visits to this blog: 928071