OU blog

Personal Blogs

Christopher Douce

Examining a Doctoral Thesis - the written and unwritten rules

Visible to anyone in the world
Edited by Christopher Douce, Saturday, 28 Oct 2023, 09:55

On Thursday 13th October 22, I attended a workshop facilitated by Sara Spencer, Head of Research Degrees, and Emeritus Prof Marian Petre from the School of Computing and Communication, that was all about examining a doctoral thesis, 

The workshop was described as being intended “for research degree supervisors who are new or fairly new to the process of examining PhD and Professional Doctorate theses” and was open to both new and experienced research supervisors. 

The broad aim of the session was to provide an “introduction to thesis examination at the OU”, to provide a summary of “what is involved in examining a research degree thesis”, explore the roles of different participants in the process, and to say something about expectations in terms of what takes place during the examination process, and the role of the viva and thesis.

In some ways, this event reminds me of an earlier workshop that I attended in June 22, which had the title Supporting EdD/PhD students through the thesis and the viva (OU blog). One of the differences between this session and the earlier session, is that this session provides a bit more information about the different roles.

On the topic of blogs that might be useful, this earlier blog, Doctoral research: a short introduction published in October 2022 might be also helpful for prospective students. 

What follows are a set of (edited) notes that I made during Sara and Marion's workshop.

Session objectives

Participants were invited to contribute to an online document to share what they were looking for from the session. Some of the key points included: to learn from the experience of others, to understand what the overall process is, what to do if there are disagreements, to share general tips about how to approach examining a thesis, and how to provide feedback.

The facilitators shared some of their experiences, and begin to discuss some of the roles within an examination, and what happens. Some points I noted down were that supervisors will try to choose examiners who are appropriate (given the focus of their student’s research) and examiners should always endeavour to set their egos aside: it isn’t about them, it is about the student who has written a viva, and their research.

Roles and responsibilities

Marion made the following important points: the viva is a real examination and conducting it well matters. Also, if a student has a flawed dissertation, a student can strengthen their position through the viva process. Conversely, if a student has a strong thesis, but gives a weak defence, the outcome might not be as hoped.

Panel chair

Every doctoral examination has a panel chair. The job of the chair isn’t to ask questions, but to moderate the session, mediate communication between everyone, check that everything is going okay, ensure that procedures are followed, and ensure that everyone feels comfortable. The chair is someone everyone can appeal to if help or support is needed, and call for breaks, if necessary. Unlike the examiner, the chair may not be a subject specialist, but will be someone independent and experienced who understands the process. In contract to the chair, the doctoral supervisors are outside the examination process. 

Examiners

The examiners assess the quality of the research. There will usually be either two or three examiners, and two of those may be external. The examiners read the dissertation thoroughly in advance, and prepare a re-viva report. Under strict confidence, the chair will then share each report with each of the examiners. Examiners are also expected to be familiar with the university’s regulations and must work with the chair to prepare an examination report. They must also be willing to provide clarifications for the student if required and assess any revisions, and conduct a re-viva if necessary.

Although there is an expectation that the viva examination process a relatively short amount of time, examiners may be employed within the process for considerably longer, especially if a student is required to carry out remedial work to their viva.

Observer

An observer is allowed to come along to the viva, and it is typically the lead supervisor. The role of the observer is very limited, and the observer doesn’t speak unless invited to do so by the chair. An observer may well take notes, to help the candidate understand what happened within the viva, and to help the candidate remember some of the detail of the discussions that took place.

Candidate

The candidate is, arguably, the most important person in the room (although it might be argued that the chair is just as important). The whole event is about the research that the candidate has carried out, and to check to see whether they have a thorough and detailed understanding of what they have done, and their subject. An important point is this: there are very few opportunities in life where we have opportunities to talk to a group of other people, at length, about a subject that we are very interested and passionate about. With this perspective in mind, and it might even be possible to think of the viva as a precious and potentially even enjoyable event. A candidate can request breaks via the chair, and always ask for clarifications to any question that is asked.

The procedures

As mentioned earlier, examiners read the dissertation, and the examiners prepares a report 5 days before the viva, which are then shared with each of the examiners via the panel chair. An examiner may form an opinion which may be expressed within the form, but this need not be fixed: “the report is not a contract; it is an initial assessment”. This assessment can change depending on what happens within the viva.

Pre-viva meeting

During the pre-viva meeting, the chair and examiners meet to discuss their view and opinions about the thesis. The report helps everyone to see if there is a consistent perspective. Using the reports, the examiners will form an approach. They will discuss a plan about how ask questions.

The exact approach will be different, depending upon the examiners, thesis, and subject. On some occasions, examiners might start with some very easy questions and then work towards points that really matter. Other examiners may choose to take turns, and some will go through a chapter at a time. Sometimes the external will lead, and the internal will follow.

Before the viva, the chair will have some idea of what is going to happen, and how the thesis will be assessed. The chair also provides and offers any necessary clarification about regulations. An important note is that every organisation is slightly different.

The Viva

This is the key meeting between the candidate, the panel chair, examiners, and any observer. Typically, the chair introduces the panel and provides an overview of what is going to happen.

A viva lasts as long as it takes. It might typically last between an hour and a half and three hours; online takes a bit longer. There should be no particular end time. A point that was made: there is no correlation between the length of the viva and the outcome. Breaks can be requested by any participant, via the chair.

Post-viva meeting

After the viva, the examiners, and the chair meet. The candidate and observer are asked to leave the room, where they discuss what has happened, and what recommendation is to be made. The duration of the post-viva meeting also takes as long as is necessary. If examiners do not agree (which very is unlikely), and there is a formal procedure to take account of this. It was emphasised during this session that this a very rare occurrence: examiners tend to agree.

Recommendation meeting

Everyone meets up again, and the recommendation is shared with the student. 

During this meeting there is an opportunity for the examiners to provide some feedback. Revisions are discussed (if necessary), and the observer usually takes notes. During this meeting, the candidate may ask questions.

An important part of this process is the completion of an examination report form, which contains an outcome. The outcome is a recommendation to a university authority, and the panel offers a recommendation summarising what revisions are necessary, and why.

Outcomes

Assessment criteria for a thesis is presented on the examination report form. Points include presentation and style of the thesis (whether the candidate is able to contribute to academic debates), evidence of the work being a significant contribution to knowledge, whether the candidate show evidence of being able to carry out research in the future, and whether the thesis contain material worthy of publication. 

On the point of publication, both Marion and Sara emphasised that publication is neither necessary, nor sufficient for a PhD; the thesis is a monograph, not a collection of papers.

In the OU, there are a number of different possible outcomes: the candidate is awarded the degree, the candidate is awarded the degree with minor corrections, or the candidate has to make substantial amendments. Other outcomes include: the candidate must resubmit their thesis for re-examination, a degree of MPhil is awarded subject to dissertation amendments, resubmission of thesis for re-viva for a MPhil award, and finally, a student is not awarded the degree and not permitted to be re-examined.

Outcomes will be based on the quality of the submission, and each category has a specific timeframe, i.e., minor corrections might be required to be completed within 3 months, and major correction may have to be submitted within 6 months.

How to be an effective examiner

Towards the end of the session, there was a discussion, where participants shared tips about how to be an effective examiner. I noted down the following points from a PowerPoint slide that directed the session: “the best examiners bring out the best in the student” and “there is a correlation between examiner experience and moderation/kindness”.

Marion emphasised the point: “look for the value in the work; whether it conveys a sense of confidence and contribution” Another point was: It is about people skills, as much as it about technical skills. Also, create a rapport with the candidate before asking any tough (but necessary) questions, such as: what did you enjoy, how did you come to study this in the first place? Make sure that you listen well to all answers.

Judge the work on its own merits and make sure that you don’t impose your (examiners) framework on the candidate’s work. Break down larger questions to smaller questions, and give sufficient time to allow your questions to be answered. Importantly, reflect on your own tone and way of communicating, and potentially mention this to the candidate to put them at ease. Be very mindful of how the candidate might be experiencing stress during the viva, and encourage breaks.

A really important point I noted down was: what does “good enough” look like in your discipline? In the viva, what matters is a pass. Another comment was: very few dissertations are without flaws. Always look to what is good in a thesis.

Reflections

This session made me think about to my own viva. My viva was a positive experience. At the time, I didn’t have a really thorough understanding of what everyone’s roles were. I remember the internal examiner, and the external examiner, but I can’t remember who the chair was. I do remember the close scrutiny of the work that I submitted, and a feeling of being asked some really difficult questions. 

Interestingly, I also remember that the internal examiner really liked a certain aspect of my thesis, where I drew on materials from outside of my home discipline. In retrospect, I think this may have contributed to the assessment that I was capable of carrying out original research, which is such an important part of the process. The point here is that I remembered the nice bits, just as I remember the tricky bits.

In the next two months, I’m going to be an external examiner. Attending this session has helped me to strengthen my understanding of the process, and really emphasised what my role and responsibilities are going to be.

I remember another bit of advice I was given by a colleague when I was preparing to be an external for the first time. The advice was about how to approach the reading of a thesis: “Look to what happens within the methodology. The methodology is about what has been done. Does the methodology make sense, given the research questions?” Whilst this bit of advice is practical, the most important bit of advice from Sara and Marion’s session was: “make sure you’re approachable”.

Acknowledgements

The structure of this blog directly echoes the session that was designed and facilitated by Marion Petre and Sara Spencer. Many of the words within this blog also reflect points made by both Marion and Sara. I hope I’ve done justice to your excellent session!

Permalink Add your comment
Share post
Christopher Douce

Course for External Examiners

Visible to anyone in the world

On 9 November 2018 I attended a continuing professional development (CPD) course for external examiners run by the Higher Education Academy (or, AdvanceHE, as it is otherwise known). The course was facilitated by two OU colleagues: Professor Mark Brandon from the STEM faculty, and Naomi Watson from the WELLS Faculty. What follows are a set of notes that I made before and during the course. A week after the course, I edited everything together so I would have a rough sketch of what happened during the event. 

The aim of the course was to further understand the role of an external examiner, develop a deeper understanding of the nature of academic standards, and to ‘use evidence-informed approaches’ to inform judgements about ‘academic standards and the enhancement of student learning’.

The course was split up into two parts an online component (part 1) and a face to face component (part 2); participants had to complete both of these parts to complete the CPD. An interesting point was that completion was also recorded by the HEA. 

During the event, delegates were given a nicely printed A4 sized book, which had the catchy title: professional development course for external examiners. During the course we dipped into the book and completed a number of activities, writing down some personal reflections and thoughts. From time to time, I’ll refer to the book, the activities, or both. 

Preparing for the face to face session

To prepare, we all had to login to a virtual learning environment and complete a couple of activities (which I summarise below). The introductory information was useful; we were referred to the Higher Education Academy's A handbook for external examiners (PDF).

I also noted down the words: “external examiners gain oversight of assessment process, provide comment about the assessment process to say whether student learning outcomes are met and to offer informative comment on good practice”.

Activity 1: External examining of student work

We were asked to read an assessment briefing paper that described principles of feedback. The briefing paper contained information about learning outcomes, the level of study, information about the task that students had to complete, and provided a marking scheme. There were three learning outcomes: one about understanding, another about designing and a final one about taking account evidence into. Students were required to write a critical review of their own assessment practice.

We had to review three assignments and judge whether the work has been given the correct marks in coordination with the scheme and write a paragraph to give feedback regarding the academic standards of the module based on the assignments.

Activity 2: The external examiner role

This second activity emphasises that external examiners have a duty to help to maintain academic standards, ensure that institution policies and regulations are followed, that standards are comparable with those in other institutions, and to share good practice, and to be a critical friend. 

We were asked to comment on a short series of scenarios: a situation where a course leader asks for help, a situation where we had to deal with differences of opinion between examiners, and to consider a situation where there was low quality student work despite institutional staff working very hard to maintain standards.

During the face-to-face session

The face-to-face session was split into 7 short components (excluding a concluding section). These components had a mixture of listening, group work, followed by individual reflection activities. What follows is a very short summary of notes from these different sections. 

Session 1: Introduction

The first session was an introduction which referred to the QAA’s UK quality code for Higher Education, Chapter B7: External Examining (pdf) Echoing the introduction to this blog, externals are required to ensure threshold standards (QAA quality code, chapter B7, p. 4), ensure that processes are followed, and also ensure that academic standards are comparable between institutions.

During this first session we reviewed the activity 2 scenarios to further understand the role of the external examiner, and to understand the tensions, and to understand how to balance the different demands of ‘the process checker’, ‘critical friend’, and the maintainer of standards.

As a brief aside, it’s important to mention that appointments to external examining posts are made by universities. Examiners are recruited either through personal recommendations (where people are encouraged to apply), or through a national mailing list, which interested academics can sign up to.

Session 2: Variability in academic standards

In the second session we looked at another scenario and were asked to think about the challenge of how we actively score assignments. During the task I asked myself about the extent to which learning outcomes can or should be linked to assessment criteria. In the context of our scenario, a question that was asked (amongst our table) was: do students have access to the marking criteria?

This session had a second task, where we were asked about what issues might lead to variability in academic standards. I made a note of differences between people, tools and the task (nature of assessment).

Session 3: People as a source of variation

An interesting point that I noted down was an assertion that standards are socially constructed. To understand more about what this meant, and the idea of variation further, we were asked to complete another exercise that began with a question: what shapes our standards? Some points that I noted down were: from our institutions, values and beliefs, but also from specialist and professional knowledge of subjects.

Another question that we were asked was: where did you get your standards from? Some answers included: 

  • Influential people and groups such as colleagues and networks
  • Experience of working with students, carrying out assessments, and working within industry 
  • Known and understood professional values and beliefs, such as the aims of higher education and values from a discipline or subject.

Sessions 4 & 5 : Tools and tasks

This activity complemented one of the online preparation activities.  We were asked to look at module specification documents and descriptions that were from different sectors, professions and institutions. We were asked to think about internal reference points, such as qualification descriptors, learning outcomes and assessment guidance and external reference points, such as subject benchmark statements. 

These discussions led to an activity: we were given a set of cards which related to different scenarios that we might observe as external examiners. We were asked to place the cards on a two dimensional grid. One axis had the title ‘internal/external’ (which related to the types of tool the care related to), and the other had the title: ‘high/low effectiveness’. After a figuring out where the cards went, we were invited to have a look at what other groups had done.

Session 6: Profession practice

This session was all about figuring out what to do in certain situations. In our groups we were given a set of different external examining scenarios, and offered 5 different choice cards. After one of the group read out the scenario, we were invited to vote on what we thought was the best course of action. After the voting, discussed why we had chosen to vote the way we did.

It was a fun exercise; there were friendly differences of opinion about what strategies to adopt. I sensed that there was no right or wrong answer, and the best course of action might depend on a combination of different factors, including the institution, subject, and the colleagues that we’re working with.

Session 7: Social moderation and collaboration of standards

The penultimate session featured a couple of videos. There was one video of exam marking of a musical performance, and another video of a project where academics from Australian universities discussed how they would mark different pieces of work. In some respects, the second video (which featured a marking exercise) very much resembled by own experience of being a project marker on a Computing and IT project module. During the co-ordination meeting different tutors would present their views and justify their marks.

There seemed to be an important point underlying this final session, which I noted down, which is: ‘you need to be talking to other people within your subject to understand what the standards are’.

Reflections

All in all, this CPD was pretty good fun! Having had some experience of being an external examiner, I found that some of the discussions directly resonated with my personal experience. 

One of the key points that I took away from the session was a differences between internal and external documents (or tools) that can help and guide the external examiners. 

Although I had an implicit awareness of the distinction, the way that it was made clear was very helpful. In my own experience, I’ve been reviewing course descriptions and marking guides (internal documents) and also having a look at different qualification outlines (external documents). I have also remembered that I have, on occasions, had a look at module descriptions from other institutions (to help me carry out a comparison of standards).

I enjoyed the interactive element and the opportunity to discuss issues with colleagues from other schools and faculties. I don’t have any suggestions about how to improve the course, since it offers a lot of tools and useful tips. The next step for me is to try my best to connect what I’ve learnt to my current external examining contract.

Permalink
Share post

This blog might contain posts that are only visible to logged-in users, or where only logged-in users can comment. If you have an account on the system, please log in for full access.

Total visits to this blog: 2308388