Personal Blogs

On 2 October I attended another of the OU’s professional development events. This time, it was an event organised by the OU library. Facilitated by Chris Biggs, Research Support Librarian, the session aimed to outline three things: “common bibliometrics, and their use and misuse”, “what are Altmetrics and where they can be found” and DORA, which is an abbreviation for the “Declaration on Research Assessment (DORA)” along with the “responsible use of metrics”.

I was particularly interested in this section since I’m a co-editor of an international journal called Open Learning. Bibliometrics are sometimes discussed during the annual editorial meetings between the editors, the members of the editorial board, and the publisher, who is Taylor and Frances. During these meetings, various numbers are shared and summarised.

When I saw the title of the event, my main thought was: “I should go along; I might pick up on a point or two”. What follows is a set of notes that I made during the session, along some of the useful weblinks that were shared. One thing that I should add is that the structure of these notes come from the facilitator, Chris, and his presentation. Towards the end of these notes, I share a set of reflections.

Citations

I missed the first couple of minutes, joining just at the point when Chris was talking about the ‘the social dimensions’ of citations. Citations are all about giving credit. A useful link to look at is the page Citing Sources: What are citations and why should I use them?

One view is that the more citations an article has, the more popular it is. Subsequently, some might associate popularity to quality. An interesting paper that was referenced had the title Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories. 

Retuning to the notion of the social dimension of citations, one metric I noted down was that self-citations account for 12% of citations. A self-citation is where an author references their own, or earlier work. Whilst this can be used to guide authors to earlier research, it can also be used to increase the visibility of your research.

A concept that I wasn’t familiar with but immediately understood was the notion of a citation circle or cartel. Simply put, this is a group of authors, typically working in a similar field, who regularly reference each other. This may have the effect of increasing the visibility of that group of authors. Chris shared a link to an interesting article about the notion: The Emergence of a Citation Cartel. 

A further notion that I hadn’t officially heard of, but was implicitly familiar with, was the notion of the honorary citation. This is where an author might cite the work of a journal editor to theoretically increase chances of their paper being accepted. As an editor, I have seen that occasionally, but not very often. On a related point, the publisher, Taylor and Francis has published some very clear ethical guidelines that editors are required to adhered to.

Something else that I hadn’t heard of is the Matthew effect. This means that if something has been published, it will continue to be cited, perhaps to the detriment of other articles. Again, we were directed to an interesting article: The Matthew Effect in Science.

It was mentioned there are interesting differences in between academic disciplines. The pace and regularity of citations in the arts and humanities can be much less than, say, a busy area of scientific research. It was also mentioned that there are differences between types of articles. For example, reviews are cited more than original research articles, and methods papers are some of the most cited papers. (It was at this point, I wondered whether there were many articles that carried out reviews of methodologies).

An interesting reflection is that articles that are considered to have societal benefit are not generally picked up by bibliometrics. This immediately reminded me about how funders require researcher to develop what is known as an impact plan. I then remembered that the STEM faculty has a couple of impact managers who are able to provide practical advice on how researchers can demonstrate the impact and the benefits of the research that they carry out.

All these points and suggestions lead to one compelling conclusion, which is that the number of citations cannot be directly considered to be a measure of the quality of an article.

An important element is all this is, of course, the peer review process. Some important points were made: that peer review can be slow, expensive, inconsistent, and prone to bias. As an editor, I recognise each of these points. One of the most frustrating elements of the peer review process is finding experienced and willing reviewers. During this session, I shared an important point: if an author holds views that are incompatible or different to the reviewers, it is okay to have a discussion with an editor. Editors are people, and we’re often happy to chat.

Bibliometrics

There are a few different sources of bibliometrics. There is Scopus, Web of Science, Dimensions, CrossRef and Google Scholar. Scopus and The Web of Science offer limited coverage for social sciences and humanities subject. In contrast, Google Scholar picks up everything, including resources that may not really be academic articles. A link to the following blog, Google Scholar, Web of Science, and Scopus: Which is best for me? was shared.

There are, of course, different types of metrics. Following on from the earlier section where citations were mentioned, there is the notion of normalised citations, percentiles, and field citation ratios. Normalised citations (if I’ve understood this correctly) is the extent of an article being over a time period. Percentiles relate to how popular, or widely cited an article is. There is, of course, a very long tail of publications. Publications that appear within, say, the top 1% or top 5% are, of course, highly popular. Finally, the field citation ratio relates to the extent to which an article is published within a particular field of research.

There is also something called the h-index, which relates to the number of publications made by a researcher. During Chris’ presentation, I made the following notes: the h-index favours people who have a consistent publication record, such established academics. For example, a h index of 19 means 19 papers that have been cited 19 times.

Moving beyond metrics that relate to individual researchers, there is also something called the journal impact factor (JIF). Broadly speaking, the more popular or influential the journal, the higher its impact factor. This has the potential to influence researchers when making decisions about how and where to publish their research findings. It was mentioned that there are two versions of a JIF: a metric that includes self-citations, and another that doesn’t.

Metrics that relate to official academic journals isn’t the whole story. Outside of ‘journal world’ (which you can access through your institutional library) there are an array of ever changing social media platforms. Subsequently, there are a number of alternatives to citation based bibliometrics. Altmetrics, from Digital Science, creates something that is called an attention score, which consolidates different ‘mentions’ of research across different platform. It can only do this if there is a reference to a a persistent digital object identifier, a DOI.

Previews of AltMetric data can be seen through articles that are published on ORO, the university’s research repository. Although I’m risking accusation of self-citation here, an interesting example is the following excellent paper: Mental health in distance learning: a taxonomy of barriers and enablers to student mental wellbeing. Scrolling to the bottom of the page will reveal a summary of tweets and citations; metrics from both Altmetric and Dimensions.

There are a couple of other alternative metrics that were mentioned: PlumX, which is from Elsevier, and Overton, which is about the extent to which research may be potentially influencing policy.

Responsible metrics, the OU and DORA

Towards the end of the event, DORA, Declaration on Open Research Assessment was introduced, of which the OU is a signatory. One of the most salient point from the DORA website is this: “Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions”. There is also a case study that relates to the OU which can be found on the DORA website. 

This final and important part of the session, which had the title ‘The Idea of Responsible Metrics” was covered quite briefly. The topic of DORA is something that I really do need to look at in a bit more detail.

Reflections

I learnt a lot from this session. One thing that really grabbed my attention was the h-index. As soon as the session had finished, I asked myself a question: what is my h-index? It didn’t take too long to find it out.

After finding my h-index figure, I made a mistake; I wondered what the h-index for some of my immediate colleagues were. I found this unnecessarily and seductively interesting. I looked up h scores for professors, some fellow senior lecturers, and some newly recruited members of staff. Through various links, I could see who had collaborated with who. I could also see which of the professors were really high ranking professors.

I then stopped my searching. I asked myself another question, which was: “does any of these numbers really matter?”

It takes different kinds of people to run an academic department and a university. Some colleagues are excellent at research. Some colleagues are excellent at teaching, and some colleagues are even excellent at administration. In my own role as a staff tutor, I do a lot of academic administration and quite a bit of work which could be viewed as teaching. This means that I don’t have a lot of time to do any research. Broadly speaking, central academic staff have a much higher h-index metric than staff tutors, simply because they have more time. What research I do carry out is often applied research. This research can sometimes be labelled as scholarship, which can be considered to be research about the practice of teaching and learning.

It was interesting that one of the important points I took away was that societal impact can’t be directly measured through bibliometrics. I also found it interesting that different types of articles attract a greater number of citations. One of my biggest academic hits (I don’t have very many of them) has been a review paper, where I studied different ways in which a computer could be used to assess the quality of computer programming assessments. The articles that I have published that relate to pure research have certainly attracted less attention.

All this comes back to a broader question: in academia, what is valued? I think the answer is: different types of work is valued. Pure research is valued alongside effective and engaging teaching. The bit that ties the two together is, of course, scholarship. 

Bibliometrics are, in my eyes, are a set of measurements that attempt to quantify academic debate. It only ever tells a part of a bigger and much more complicated story. Metrics are not, in my opinion, surrogates for quality. Also, academic fashions and trends come and go.

Acknowledgements

Many thanks are given to Chris Biggs for running such an engaging and interesting session. Many of the links shared in this article were shared during Chris' presentation.

On 25 January 2022 I attended an event called the School of Computing and Communications Research Fiesta, which took place on the university campus. One of my reasons for attending the fiesta was to try to restart my research activities, having stepped away from research due to taking on a role called ‘lead staff tutor’ for the last three years. 

The last time I attended a school research fiesta was on 10 January 2019 (OU blog) which took place at the nearby Kents Hill conference centre. Following this earlier event, I shared an accompanying post about research funding (OU blog).

This event was advertised as a “… time for us to reconvene and discuss everything research. This event is aimed to help us (re-)connect with one another and understand how we can help and benefit from each other’s research expertise and outputs” and was facilitated by David Bush from Ascolto.

What follows is a summary of the Research Fiesta, in terms of what happened during the meeting, and what I felt the biggest take away points were. This blog may of be interest to anyone who was at the event, couldn’t make it to the event, or broadly interested in the process of research (whether computing research, or research that takes place within other disciplines).

Preparation

Before the event, we were asked to prepare some cards which summarised our research interests. Although I didn’t write the card in advance, I did come to the event with some ideas in mind. Here’s what I wrote down on three cards:

• Understanding and characterising green computing: what it is, what the boundaries and problem are, and how can we embed this theme into our teaching?
• Storytelling, soft skills, and software engineering: what role does storytelling play or could play in software engineering practice, and how might storytelling be used to develop soft skills in the next generation of computing graduates?
• Accessibility of web technologies: how accessible are the current generation of web-based applications, and to what extent are hybrid apps accessible with assistive technology. How useful is WAI-ARIA? It is still useful? Does it have an impact?

Later during the session, I added two more cards:

• Pedagogy of teaching programming at a distance: innovative tutorials; how to develop tutors, and how to help them to be creative, perhaps by embedding and using drama.
• Development of writing skills across the computing curriculum. 

This final idea emerged from discussions with tutors, following some discussions with tutors, and might form the basis of a scholarship project. The university has prepared a lot of materials about writing; the question is whether the computing programme makes effective use of them, given the writing requirements from some courses.

Activity 1: Sharing research ideas

Our first activity could be loosely called “academic speed dating”. 

I’ve done this before (both the academic version, and the non-academic version). 

In this version, we were sent to various tables, where we met up with two other colleagues. Our task was to show our cards (our research ideas) and try to create a new card that combined aspects of all of our cards. When we had done this, we had to pin our cards onto the wall to share our ideas with everyone.

Activity 2: Forming research teams

After a short break, everyone was asked to form a line based on how much research experience everyone has. On one side, there were all the new PhD students, and on the other side, there were the professors and heads of existing research groups.

Approximately 6 PhD students and early researchers were asked to review the cards that had been generated from the speed dating activity, and each had to choose a card they found most interesting.  This card (represented by one of the researchers) would then form the basis of a new team of 3 or 4 researchers.

One at a time, the rest of the researchers were ushered over to speak with the new researchers. If you liked an idea, and there wasn’t already 3 or 4 researchers, you could join a team. The longer the game went on, the harder it becomes for the more experienced researchers. Instead, they would have to make use of all the powers of persuasion to try to join an existing team, or to persuade fellow researchers to create new teams.

After some discussion and reviewing cards, I joined two of my colleagues, Dhouha Kbaier and Yaw Buadu. Two project cards were combined together to create a new project. Paraphrasing our cards, our project intended to:

Develop digital technologies to enhance engagement and participation by integrating more physical computing into the computing curriculum. 

Accompanying research questions were: what are the challenges of using physical computing in a distance learning environment, and how might physical computing devices be connected to and integrated within the Open STEM labs? 

This final question suggests the opportunity to explore costs and trade-offs of a physical computing approach where students use their own equipment, or share equipment with other students through a platform which is accessed remotely.

What might physical computing actually mean? One answer to this is: physical hardware used by students to learn about or to solve computing problems, as opposed to using software simulations. There is a precedent of using (and sharing) physical computing devices at the university. In earlier decades, there was the Hektor computer (computinghistory.org.uk), which was once sent out to computing students (and then later returned to the university).

A more modern and smaller (and much more sophisticated) version is the Raspberry Pi computer (Raspberry Pi website) which can be used with any number of interesting computing projects.

One other aspect that we discussed were about the stakeholders, and who might need to be involved? We identified the following groups: students, tutors, module team members, and administrative university functions.  (The members of module team may include both tutors and curriculum managers, who act as a fundamental link between the academic team and operations of the university bureaucracy). 

Impact: evaluation and presentation

The next bit of the fiesta was a presentation; a double act from two colleagues from the research school, Betul Khalil, an Impact Manager, and Gareth Davies, who is a Research Evidence Impact Manager. 

They began with a question: what is impact, and can we give an example? 

Impact isn’t the same as project outcomes. They are very different things. An outcome might be a report, or some software. An impact can refer to a change that may have led to a positive long term benefit to stakeholders. In terms of the UK Research Excellent Framework (REF Impact case studies), impact could mean a change to society, the economy, and to the natural environment. Also, a measurable change might be on a local, regional or international scale.

The message to us was clear: when working on a project bid, researchers need to proactively consider impact from the outset and define impact objectives, since gathering effective evidence to show how those objectives may have been met takes time. In some respects, impact evidence gathering is a further part of the research process.  To do it well, researchers need an impact plan to accompany a research or project plan. 

We were all given a handout, from which I have noted down some useful questions that researchers need to bear in mind. These are: 

• Who are the stakeholders, and who might be affected by the change your project may facilitate?
• What do the stakeholders (or beneficiaries) gain from your research?
• Why will they engage with your research?
• How will you communicate with beneficiaries?
• What activities might you need to run to effect change?
• How might you evidence change? 
• How will you connect change to your research?

Later, Gareth talked more about what it means to ‘evidence’ impact. An important note I made from Gareth’s presentation was that “upsteam planning is important” and that the analysis of impact should be rigorous. Researchers also need to consider which methods they use to enable them to find a way to observe what is changing. 

Apparently, one of the most common forms of evidence is a written testimonial (in the form of a testimonial letter). Within this assertion lies the reflection that researchers need to make sure they have the time and the means to gather evidence.

Activity 3: How will we do our project?

Our next activity was to sit around a table to figure out how were going to do to answer our research questions.

We began by asking: what might the outputs from our project be? We came up with some rough answers, which were:

• Guidelines about how physical computing could be embedded and used within module teams. If used within a module, tutors could then be offered some accompanying guidance.
• Recommendations about physical kit that could be used (these kits might be bought, or borrowed, or used from a distance); recommendations about the use of software; recommendations about pedagogy and use (which is an idea that can relate to the idea of useful guidelines). 

To produce these, what needs to be done? Our team offered the following suggestions (but the exact order of carrying these out could be easily debated):

1. Examine learning outcomes within various qualifications and accompanying modules.
2. Explore the problem space running focus groups with stakeholders to understand how the terms engagement and participation are understood.
3. Use mixed methods: from the focus group results, carry out a survey to more thoroughly understand how a wider population understands engagement and participation.
4. From these different information sources (and input from the learning outcomes) facilitate a number of curriculum design workshops to understand how physical computing can be brought into the curriculum.
5. Carry out a detailed analysis of all the data that has been captured, writing up all the findings.
6. Implement the findings.

A further reflection was each of these activities needs to be considered in terms of SMART objectives; specific, measurable, achievable, realistic and timebound. 

A new question that we were asked was: what impact will your project have? 

Given that students are key stakeholders, there might be broader impacts in terms of results to the National Students Survey. There might be further impacts both within the university, and to other organisations that provide distance learning. There might also be impacts that could be broadly described as the further development of computing pedagogy. This is all very well, but how might we go about measuring all this? It is this question which the facilitators from the research school may have wanted to encourage us to consider.

What happens next?

 After presenting our plan to all the other groups, we were asked a couple of final questions, which were: how excited are you about the project? Also, how doable (or realistic) is the project?

Given that we all have our own main research interests (which are slightly different to the new project that we have defined), we all had different levels of enthusiasm about going ahead with this project idea. That said, the key concepts of physical computing (in its broadest sense) and student engagement are important topics which other researchers may well be interested in exploring. Even if this particular team may not be in a position to take these ideas forward, the ideas are still worth exploring and studying.

Reflections

I really liked the way that we were asked to focus on trying to get things done. 

When thinking about research (and research projects) impact has always been something that has always been at the back of my mind, but I’ve always tended to consider it as something that is quite intangible and difficult to measure. The presenters from the research school made a really clear point. They emphasised that it is important to plan for impact before your project has started.

A personal reflection is that impact could be thought of as a way to reflect on the success of a project. In some respects, this should be something that researchers should be doing as a matter of course to further develop their professional skills. Of course, the extent and nature of this analysis will depend very much on the nature of the research that is carried out through a project. Given the collaborative nature of research, gathering of impact evidence is likely to be collaborative too. 

It is interesting to compare this Research Fiesta with the one that was held in 2019. One of the differences being that there were a lot fewer people attending this event. This might have been a factor due to the timing (some new module presentations were just about to begin) or a hangover from the 2019-20 pandemic (where so many colleagues switched to homeworking). 

An interesting difference related to the structure: this event was facilitated in a dynamic way, where the research themes emerged from the participants. The earlier event had more emphasis on sharing information about the research groups within the school, and more of the practicalities about how to gain funding for research. There is, of course, no right or wrong way to run a research fiesta. I appreciated the dynamic structure, but equally I’m always up for hearing about new concepts and ideas, and learning about what is happening within and across the school.

Acknowledgements

Many thanks to Amel Bennaceur for organising the event. One of the impacts has been to get to catch up with colleagues, and to learn more about them! It was a pleasure working with my fellow group members, Dhouha Kbaier and Yaw Buadu who kindly reviewed this blog article before it was published.

The Open University runs a centre called eSTEeM which funds research and scholarship to enhance and develop STEM education. For the last few years, the centre has run a conference that serves a number of purposes: to showcase research, to create a space to get people talking (and potentially collaborating) with each other, and to offer an opportunity for academic professional development.

What follows is my own personal summary the two days of the conference. There were a number of parallel sessions to choose from. My approach to choose them was very simple: I chose the sessions that packed a lot in. This meant that I chose the paper sessions rather than the various workshop sessions were on offer. At the end of the blog I offer some very short reflections based on my experience of the session.

Opening keynote

The conference was opened by Diane Butler, who introduced the introductory keynote speaker, Tony Bates who used to work at the OU and also the University of British Columbia. Tony has recently written an Open Text Book called Teaching in a Digital Age. I made a note that Tony opened with the observation that there is ‘a lot of change’ and this has direct implications for teaching and learning at the university. One of the key forces of change is the need for skills, i.e. IT skills that are embedded within a subject area; knowing skills that are specific to a discipline. An accompanying question was: what are employers looking for? Certain skills are really important, such as active listening, speaking and critical thinking. 

Learners need to practice and develop skills and to do this they need regular feedback from experts. I made a note about that technology isn’t perhaps the most appropriate way to develop the soft skills that Tony mentioned earlier. An interesting question was posed: what does an advanced course design look like? There were some suggestions (if my notes serve me well): perhaps there might be student generated multimedia content and assessment by e-portfolios.

Tony also spoke about trends: there are new models of delivery; there is face to face teaching on one side, and fully distance learning on the other (and everything in between). An interesting point was that every university in Canada had fully online courses, with 16% of all course enrolment being to online courses and programmes. Traditional universities are moving into the space where distance learning institutions used to dominate.

An interesting new trend was the notion of hybrid learning: looking at what is best done in the classroom and what is best done online. I made a note that Tony said there was ‘no theory about what is done face to face versus online’, which strikes me as surprising.

A significant trend is, of course, MOOCs, but it was reported that there was no MOOC mania in Canada. Other trends included open educational resources and open text books. The point is that we’re now at a point where university professors offer learning support and not content and this has implications for teaching and learning. 

Tony concluded by leaving some points for the university: that technology is continually changing, that there needs to be flexible accreditation for life-long learning, and perhaps there needs to be an agile approach to course (or module) development. Also, all universities will be or are going to be digital (in some way or another). 

Paper session: Supporting students

Lessons in retention success: using video media to influence students

Jessica Bartlett spoke about her experiences working on S282 Astronomy. There are some immediate and obvious challenges: students numbers are falling and the module contains a lot of maths. An interesting point is that 50% of the students studying this module were not from the STEM faculty (which is where all the maths is studied).

The aim of the project was retain more students and help more students to pass exams. The module uses formative tutor marked assessments (TMAs) which means that the module team can reuse questions but can’t (of course) provide model answers to students. I recognised an interesting comment: ‘students don’t often look at their mark, ignoring their feedback’. The module team made videos about how to deconstruct and approach the TMA questions. I made a note of something called ‘reviewing your TMA’ activities, which encourage students to look back at what they’ve done (which sounds like a great idea). There were also weekly videos, where the filming and editing was done by the module team.

Evidence that bootcamps can help student retention and progression

Tom Wilks also spoke about S282 Astronomy but within the context of a ‘bootcamp’ that was designed to offer addition student support. Tom recorded short tutorial sessions that covered a range of topics: basic maths and physics, general OU study skills, how to use the VLE and how to use the Adobe Connect conferencing tool. 28 Adobe Connect sessions were recorded, each lasting between 2 and 10 minutes in length. These sessions were advertised to all students, who could access a forum and an Adobe connect room. Other resources include something called an ‘are you ready for quiz’ which is also used with some computing. Tom commented that tutors can refer students to his recordings if some students were struggling with certain concepts, and he also found that students did re-engage with materials when they were approaching their TMA. 

Flexible/early start M140

Carol Calvert gave a talk on her work on introducing a Flexible or Early start to M140 Introducing Statistics. I’ve heard Carol speak about this subject before, and she always delivers a great talk. Her research is based on an earlier study where she looked at students who succeed despite the odds that are stacked against them. One of the key findings of this research that one thing can really make a difference, and that is: starting early. 

Carol’s intention was to create an ‘early start’ experience that was close to a student’s experience when it officially begins. This means they have access to materials, can access the VLE site, have access to tutors, and can access to resources such screen casts and software. 400 students were sent a message offering them an invite to start early, and 200 responded saying that they would. Tutors offered sessions on study skills and tutorials on content. Another advantage is that if students do start early, they will know sooner whether they are on the wrong module, which can be really useful, since there are significant fee implications if someone finds themselves on the wrong module. If you’re interested, more information about Carole’s scholarship is available on the eSTEeM website. 

Improving retention amongst marginal students

Anactorial Clarke and Carlton Wood spoke about an access module: Y033 Science Technology and Maths. Access modules are important due to the university’s commitment to widening participation. Y033 is studied by 1K students per year and students who have successfully studied this module (as far as I understand things) can apply for a fee waiver. 25% of students declare a disability and access students are offered 1 to 1 telephone tutorials since previous research has suggested that sympathetic and supportive tutoring is crucial to student success. The study that Anactoria and Carlton introduced use a mixed method. They looked at the completion of S111 Questions in Science and Y033. Students who have taken the access module are more likely to stay with the module; the point being that access level study builds confidence (and emphasises the importance of access).

Paper session: online delivery, tuition and international curriculum

Synchronous online tuition: differences between student and teacher

Lynda Cook and a number of other colleagues asked a really important question: what are online tutorials really like? An accompanying question is: do we meet our student’s expectations? Students on 2nd level modules were surveyed, recorded tutorials were studied, and students and tutors were interviewed. Students reported that very few were using microphones (which isn’t a surprise to anyone who had attended an Adobe Connect session) and an analysis of recorded tutorials suggested that lots of features were not used, with the exception of the chat box.

The interviews with tutors revealed that when the recording button goes on, students are reluctant to talk. One conclusion is that students’ value tutorial recordings but students don’t like to interact. A personal note is that there is a conflict between interactive and recorded lectures and I don’t think the university has quite some way to uncovering the pedagogic opportunities afforded by online tools such as Adobe Connect (and, in some ways, this links back to some of the themes mentioned in Tony’s keynote). 

Understanding tutorial observation practice

It was time for my session. I spoke about a short project that aimed to ask the question: ‘what is the best way to observe tutorials?’ I approached this question by doing three things: carrying out a literature review (with help from a brilliant tutor colleague), and conducting two sets of focus groups: one with tutors, and another with staff tutors (the members of the university who usually carry out tuition observations).

Some of the themes that emerged from the focus groups directly echoed some of the themes in the literature. An important issue is to understand what tuition observations are for: are they for development, or are they for management? (The answer is: they should be used, in the first instance, for development; the observers can learn a lot just by observing). An outcome from the project was to uncover a set of really useful tuition guidelines that have been used and developed by colleagues in Science. The next step in the project is to formally write everything up. 

An international comparative study of tuition models in open and distance learning universities

Ann Walshe, a colleague from the school of Computing and Communications, spoke about her visit to Shanghai Open University (SOU) where she was a part of a group of visiting scholars. Ann reported that SOU emphasises vocational and life-long learning. Whilst it does offer bachelor degrees, it doesn’t offer postgrad qualifications. It was interesting to hear that SOU ‘does its own thing’ and tries not to compete with other local and national universities. It has a particular emphasis on blended learning and face to face teaching, having 41 branch schools for both full time and part time students. Interestingly, students have to attend a mandatory F2F induction.

The visiting scholar group were from a range of different institutions, including Chongqing radio and TV university, University of South Africa, the National Open University of Nigeria, Cavendish University, Zambia, Jose Rizal University, Phillipines, and the Netaji Subhas Open University, India (which apparently has 120 study centres, with more opening). Ann’s talk emphasised the importance of distance learning and its global reach.

Unpacking the STEM students’ experiences and behaviours

Jenna Mittelmeier’s presentation was about the challenges of Online Intercultural group work. I enjoyed Jenna’s talk, since it was a very research focussed talk that asked a very specific question: are students more motivated when they study materials related to their own cultural background? In other words, what are the benefits of matching content and activities to the membership of a multi-cultural group? Jenna described a randomised control trial in the context of a Dutch business school. In an activity, students were asked to look at something called the World Bank statistics dashboard and it was found that students participated more when using content from their own background. A qualitative survey suggested that internationalisation (of a study context) did improve participation but did expose tensions. There was an important point, which is that content needs to be made relevant to student’s lives and experiences.

Paper session: supporting students - STEM practice and engagement

Using a dedicated website in the continuing evolution of a statistics community of learners

Rachel Hilliam and Gaynor Arrowsmith introduced us to something called the Maths and Stats Subject Site. Before the university restructured and closed regional centres, students could attend course choice events where they could look at module materials from the regional centre library and talk to academic support colleagues and speak with other students. In an environment that is increasingly digital, an important question is: can we recreate that in an online environment? I made the note that it is (of course) important that students feel a part of a community.  There is a Maths and Stats advice forum, maths education forum, and information about professional and subject societies. There is also advice about preparing to study, revise and refresh resources, are you ready for quizzes, and early units from some modules.

Implementing additional maths support for Health Science students

Nicola McIntyre, Linda Thomson and Gerry Golding spoke about their experiences on SDK100 Science and health: an evidence based approach. An important aspect of the talk was that a maths tutorial was replaced with 18 short videos covering mathematical concepts, such as decimals, percentages, scientific notation and powers. There were also two workshops which were advertised students by email, and two tutors selected and briefed on format of the workshop. I noted an important point: it’s not enough to only provide videos, the workshops are considered to be an essential component.

Two mathematicians and a ukulele

Hayley Ryder and Toby O’Neil are module team members for M208 Pure Mathematics. The module is run through a single ‘cluster’, which means that there is only one group of tutors who teach on the module, and it has 25 hours of tuition sessions. From what I remember, there was a view that students wanted more contact with module team. 

One way to address this is to record a series of informal online tutorial sessions where Hayley and Toby talk through different mathematical concepts and also discuss what is discussed on the module forum. The idea is to convey a sense of ‘what mathematicians do’ and to build ‘mathematical resilience’, a concept that has a number of aspects: (1) the fostering of a growth mindset, (2) that maths has personal value, (3) knowing how to learn maths, (4) knowing how to find appropriate support. The sessions focussed on the first three of these aspects. 

An important point was that the presenters can easily make mistakes when doing things ‘live’ and this shows that real mathematicians can get stuck, just like everyone else. As for the ukulele, this also connects to the concept of learning; this is an instrument that Toby is learning to play (and I understand that he plays it during sessions!)

A secondary analysis of SEAM responses for programming and non-programming modules by gender

Joseph Osunde from the school of Computing and Communication studies the issue of gender disparity in computing and IT. Joseph offered an important comment during the start of his talk: ‘reasons [for gender disparity] may include learning environment[s] that convey gender stereotypes on interests and anticipated success’. To learn more, Joseph has been looking at university Student Experience on a Module (SEaM) survey results.

As a staff tutor, I regularly get to see SEaM survey results and I have my own views about their usefulness as personal development tools and sources of useful research data. This said, Joseph found that there were no significant differences in achievements between gender for modules that required students to learn about programming and those that didn’t. Joseph (with Anton Dil) looked at M250 Object-oriented Java Programming. It turned out that for modules that contained programming, like M250, men seem to be more satisfied with them. When these were again compared with non-programming modules, the result is a bit more mixed. 

Whilst this is an interesting finding, this does suggest that there is some more research to be done. A related question is: to what extend are different people motivated by modules that contain programming? Also, just as our colleague, Gerry Golding has carried out research (which I mention later on) into ‘mathematics life histories’, I do feel that there might be an opportunity to study something that might be called ‘computing life histories’ to understand the qualitative reasons for differences in satisfaction.

Closing keynote for day 1

The closing keynote by Bart Rientes was entitled a ‘critical discussion of student evaluation scores and academic performance at the OU’. Bart began by telling us that he used to be an economics teacher where his teaching performance was regularly evaluated. Drawing on this experience, he asked a significant question: ‘did my increase in my [evaluation] score mean that I was a better teacher?’  He asked everyone who was attending a similar question: ‘are student evaluations a good proxy for teaching excellence?’ Bart directed us to an article, entitled:  Student satisfaction ‘unrelated’ to academic performance – study  that was featured in the Times Higher.

We were given another reference to some published research that was carried out on behalf of the QAA. Digging into the QAA website later took me to two reports that are both connected to the themes of learning, student satisfaction and quality assurance. The first report, entitled Modelling and Managing Student Satisfaction: Use of Student Feedback to Enhance Learning Experience was by Rienties, Li and Marsh. The second report has the title: The Role of Student Satisfaction Data in Quality Assurance and Enhancement: How Providers Use Data to Improve the Student Experience was by Williams and Mindano.

Onto a personal reflection about this (keynote presentations are, of course, intended to get us thinking!) As mentioned earlier I’m very aware of the OU SEaM surveys. In my experience as a tutor line manager, tutors only tend to receive a couple of responses for a group of twenty students, and the students who do respond often have a particular cause to do so. This observation connects back to Bart’s opening point, which is: what can these measure of performance (or satisfaction) tell us? The fact is that education can be difficult and frustrating, but it can also be transformative. Sometimes we only can truly judge an experience (or feel satisfied with it) when the effects of our experience have become clearer over an extended period of time.

Paper session: Supporting students and technologies for STEM learning 

Using student analytics with ALs to increate retention

Gerry Golding spoke about some of his own research into Maths life histories, an idea that, as far as I remember from Gerry’s talk, originated from a researcher called Cobden. Gerry interviewed people to understand how adults coped when studying advanced maths topics and touched on the importance of high school experiences and maths anxiety. Maths life histories can students help to understand the cause of their anxieties and help them to think about what affected them. In turn, these reflections can be used to build and develop self-efficacy to help them through the hard times and facilitate the development of a growth mindset. In terms of this bit of scholarship, initial contact with students is important. Also, the university virtual learning environment (VLE) is not a big deal, because students are studying using books. I have to confess, that I didn’t pick up on the main outcomes of this bit of research, since I started to think about Gerry’s idea of ‘life histories’.

Analytics for tracking student engagement: TM355 Communications Technology

Allan Jones spoke about TM355 Communications Technology, an important module in the computing and IT undergraduate programme. The module has three 10 point blocks, printed books, 3 TMAs and a final exam. It is also a module that makes extensive use of the VLE. 

Students study what is meant by communication technologies and how they work, such as how you modulate waves and signals, encode data and correct errors. The module also makes use of 30 computer aided learning packages. Data analytics are used to track the use of the online parts and comparisons are made between two presentations and students are interviewed to understand their motivations. 

It’s a bit more complicated than that: STEM OU analyse evaluation

Steve Walker asked a question that was implicitly linked to Allan’s presentation: can learning analytics help students to complete modules? The answer was: no… until something is done with the data. The reason for looking at this subject was both simple and important: retention is important and there is the need to figure out what works, for whom and in what context, and why. 

Steve introduced a term that I had never heard of: realist evaluation, and directed us towards a paper by Pawson and Tilley (PDF) which is (apparently) used in medical education. Points that I noted down that sounded important included: mechanisms, interventions, outcome and context. 7 associate lecturers (ALs) were interviewed by members of a module team using something called ‘intervention interviews’. An observation is that the term ‘analytics’ is used in different ways. I also made a note of a simple model, which has the components: identify, diagnose and intervene.

Java specification checking

Anton Dil spoke about the evaluation of a prototype tool for M250 object-oriented programming tutors. M250 students need to write some object-oriented software code. This includes creating something called ‘classes’. These classes have contain a number of ‘fields’ (or data stores) and are designed to carry out certain actions which are started (or invoked) using something called ‘methods’. Student code can be automatically evaluated in a couple of ways: you could write something called a ‘style checker’ to assess what code a student has written, or you could assess its functionality through using something called unit testing. The module team have written a tool called checkM250 that could be used by tutors.

Eight tutors were surveyed and 6 tutors were interviewed. Tutors didn’t use the tool because they didn’t know about it, didn’t have enough time, or didn’t think they needed it. If they did use it, they were likely to recommend it, but they were unsure whether it could highlight things that were missed. I made note of the quote: ‘if you asked me previously whether I missed things, I would say: of course not’. Tutors did report that it could be useful. My own take on tools for tutors is that any tool may be useful (and I write that in the context of being a tutor!) 

Digital by Design: workshop

The conference workshop was, interestingly, run by a theatre group. The key concept behind the workshop was an observation that the ‘coffee break’ discussions in conferences that can be just as useful as formal presentations. Instead of having further talks, the idea was to create a long session that is, in fact, one long coffee break where participants could move between different discussion groups.

Another important idea is that anyone can propose a topic for discussion. Whenever someone decides on a topic, a delegate chose a post-it note that indicates when the topic is going to be discussed, and where in the large room that discussion is taking place. Participants can see a summary of the topics that are being discussed at any one time and participants are, of course, encouraged to move between different groups, according to their own interests. It was a neat idea.

I proposed a topic: how do we develop and support our associate lecturers to do ‘digital’ in the best possible way?’ Examples of other topics included: how should we be using social media apps to communicate with students and each other, how do we become experts in advising students at how to study onscreen, and how do we decide when digital is appropriate and when is it not?

During our ‘coffee’ conversation, I was joined with two colleagues. I soon began to think about whether there might be something that could be very loosely called the ‘digital associate lecturer capability model’. I sketched out a model that had three levels: university systems and tools (such as Adobe Connect), module tools (such as module specific computer based learning products, like there is in TM355) and common IT systems and products (such as Word, Excel and Powerpoint).

During these discussions, I was reminded of a JISC project called Building digital capability that the OU library was connected to and involved with. This also provided a useful framework that could be used to guide AL development. In later discussions, I discovered that colleagues from the OU library were already using this framework in AL development sessions.

Reflections

Everyone’s experience at a conference is, of course, likely to be different. I had a simple objective when I was attending this eSTEeM conference, which was to attend as many presentations as I could to try to get a feel for the breadth of projects that were happening across the university. In some respects, there was one commonality that jumped out at me, and that was the use of videos or personalised recorded tutors that were customised to the needs of students. Underpinning this is, of course, the use of technology.

During the conference, I heard presentations from module teams and presentation from tutors. I also understand that some students were attending too, but I didn’t get to speak or hear from any of them. This links to an important reflection that it is really important to hear the student voice; we need to hear stories about what has worked and what hasn’t worked. This said, eSTEeM scholars are always asking students questions through surveys and module teams are always looking to figure out what works and what doesn’t.

A final thought is this: I’m still not sure is meant by ‘digital by design’ but I don’t think that really matters. We access materials, write materials, and carry out our scholarship using digital tools. What I think is really important is how we use these digital tools in combination with each other. Digital technologies in their various forms might new and seductive, but ‘digital’ tools cannot be transformative if you can’t see or understand how they might be used. There’s something else that is even more important: what really matters in education is people, not machines. It is people who can show us which digital tools can help our studies.

Going to three events in two days, between 15 and 16 June, was pretty intense but also pretty good fun.  The first event was all about the scholarship of teaching and learning.  A different way of understanding this is: how to go about figuring out how to best do teaching and learning?  It's important to do research into this area not because 'learning' has changed, but the ways in we learn (and the technologies we use) continually changes and evolves: we want to know that we're doing the right thing.

The second event had a similar theme.  It was the Computers And Learning Research Group (or CALRG, for short) conference.  CALRG is a long-running research group in the university's Institute of Educational Technology.

The final event was at the Birmingham regional office, where the MCT learner support staff are based.  During this event, I learnt about a range of different things - but more about this later.  Actually, there was a forth event, an associate lecturer staff development conference, which was organised by the Oxford regional centre (but there isn't the time to write about this!)

Scholarship event

Linda Price opened the event by presenting a definition of scholarship: it is a term that describes research and research action.  Scholarship regarding teaching and learning is activity that uses information relevant to our learning and teaching to inform and enhance our practice.  She also went on to emphasise the importance of evidence.  Linda told us that scholarship was a strategic priority to the university.  She spoke of an internal university project called SHARE, which is intended to help regional staff with the research activities.  An interesting and thought provoking line that Linda gave us was: 'doctors save lives, but we can change them'.

The next part of the event was a series of twelve five minute 'lightening talks' about different research project (I think there were around twelve!)  First up was my colleague Ann from Manchester who talked about 'perceptions, expectations and experience of group tuition'.  Her project was to explore different perceptions and opinions about tutorials.  This research then has the potential to inform a new Group Tuition Policy.   

Next up was a talk that had the title 'What drives scholarship?' I seem to recall that the research was looking at the use of language in assignments, tutor guidance and feedback.  A really important subject is, of course, retention.  I also remember references to Y031 Arts and Languages, Y032 People Work and Society, and Y033 Science Technology and Maths access modules (which help to prepare students for undergraduate level study).

This was followed by: 'A levels based approach to referencing and information management'.  Apparently, some students may drop out, or become demotivated due to the challenge of appropriate referencing.  The national student survey apparently said that different students are given different advice.  The following talk was all about investigating engagement with on-line library services.

An interesting question, from a colleague in the business school was: 'why do we keep failing our Japanese students?'  One of the reasons could be attributed to differences between HE in the UK and Japan.  Understanding and being aware of cultural differences can allow us to gain an understanding of how to support different groups.

Rob Parsons, an associate lecturer colleague from the South East region spoke about peer assessment.  He argued that the student-tutor relationship can be improved.  Themes that Rob's short talk addressed included active learning, learning communities and engagement and retention.

The university is, of course, a big consumer of technology, and there is a perpetual need to figure out how to use new technologies and whether a technology is appropriate for students.  One talk that explicitly explored this issue had the title: Going Live with Google hangouts.

It was then my turn.  I talked about a project that was about the gathering of tutor experiences of tutoring on a second level computing module, TT284 Web Technologies (I horrified to discover that, in front of over sixty people, all of my PowerPoint animations were messed up!)

The talk that went after mine had the title: 'how to get students to do your iCMA and why that is good' (An iCMA is an interactive computer marked assignment).  This was followed by 'investigating one to one tuition: initial findings'.  This project was from the faculty of health and social care.  The research involved interviewing students and carrying out focus groups with associate lecturers.

Sometimes low technology solutions and approaches can be really useful.  An interesting talk was 'an evaluation of the effectiveness of student buddies', which I think focussed on languages and business modules, specifically L185 English for academic purposes and LB160 Professional communication skills for business studies.  The presentation told us about student buddy training through OU Live and a shared 'student buddy forum' on L192 Bon départ: beginners' French.  This reminds me that some of the materials for these modules are available for free through the Open Learn website.

The final 'lightening' presentation of the morning had a slightly different flavour; it was entitled, 'smarter than the average ebook'.  It was given by a colleague from Learning and Teaching Solutions, part of the university that provides some of the technical infrastructure.  We were taken on a journey through different digital formats, ranging from PDFs, epub (Wikipedia) and epub2 files, and then onto OU anywhere.  Some experiments have been performed with ePub3.  These are becoming 'websites wrapped as an off-line experience'.

An interesting point was that 'students are frustrated about being sent away from the text'.  This is a comment that resonates strongly with me.  I found it difficult to study 'off line' even though I wanted to: I printed off lots of on-line materials, but I found myself being directed to various websites and on-line resources.  This is also a comment that arose during my own TT284 research.

One comment I noted was: 'ebook readers are tricky things'.  Different readers do different things.  To close this presentation, we were shown a demonstration entitled 'how to build a methane molecule'.

After a lunch, it was time for the keynote speech.  The keynote was given by our PVC in learning and teaching, Belinda Tynan.  It began with a question: what does scholarship look like across the university.  There are, apparently 15 groups that 'play in the space' that is called scholarship, and that's just the research groups that belong to faculties (not to mention the research that takes place in the library, student services and other departments).  An important question is: 'how is the scholarship impacting the university, and how is it being focussed and directed?'

Moved on to talking about the different methods that can be used in education research. A related question is: how do we learn from each other, how do we share with each other, and how do we cross boundaries and disciplines?  Another really important question, from an institutional perspective is whether we are getting value for the effort that we institutionally put into scholarship.  It was a thought provoking keynote and it really set the scene for the afternoon session (but this was a session I had to duck out of because I had another commitment: to attend one of those fifteen scholarship groups that were mentioned about).

CALRG event

The CALRG conference is a three day conference, which means that I've missed loads of talks.  In case you're interested, the organisers have published a conference programme (CALRG website).

The first talk I attended was entitled: Building understanding of open education: an overview of OER on teaching and learning.  OER is an abbreviation for open educational resources.  The institute of educational technology hosts a research group that is known as an OER hub (website).

The second talk was all about mobility technology and had the title: Conducting a field trial in Milton Keynes: Lessons from the MApp.  I originally thought that MApp was some kind of mapping device, but it seemed to be something rather different.  It seemed to be about helping different people from different cultural backgrounds (or languages) to connect to each other.  I have to confess to being a bit lost at the start of the talk, but then I discovered that the research was using some really interesting methods to gather up some qualitative research.  (This reminded me about a 'diary study project' that I've been mulling over for quite some time, but I haven't managed to get around to doing anything about it yet!)

The third talk was a longer version of the same 'lightening' talk that I gave earlier in the day.  Talking at CALRG allowed me to talk a bit more about the methodology, and some really tentative findings since the analysis is still on-going!  (This, I think, is one the challenges of qualitative research: when do you stop!  One answer is: when you see the similar findings and themes emerging time and time again).

The next talk had the title: Improving language learning and transition into second language learning, through the language learning support dimensions (LLSD).  This talk used an instrument (also known as a survey) to help learners understand more about how they carry out language learning.  Since this wasn't my subject, I struggled to connect with this research, but I appreciated the idea of using a self-directed too to help learners to reflect on how they approach a problem.

The final talk of the CALRG session was 'diverse approaches to using online 'studio' based learning in Open University modules'.  In some respects, the 'studio' can be considered to be an in-house version of the photo sharing website Flickr.  I seem to have a memory that it was used with a digital photography module, to allow students to share examples of their work.  It was interesting to hear that this module was going to be re-launched as a non-credit bearing module (which will have the module code TG189).  Modules such as U101 use a version of this tool called Open Studio.  I learnt that it has now found its way into a total of thirteen different modules, and the Open Studio tool now goes by different names and has a range of different uses. There is also a blog about OpenStudio that is hosted by the university.

The talk led onto an interesting discussion about accessibility.  Whilst an on-line environment might itself be technically accessible, the materials that are transferred to an environment might be fundamentally inaccessible.  One thought about how to remedy this is to try to facilitate collaborations between students.

SST event

The final event I'll mention was held in Birmingham, which is the home of the Computing and IT student support team.  The SST comprises of associate lecturers, learner support staff, advisors and academics.  The purpose of the meeting was to allow the learner support people to meet academic colleagues (and visa-versa) to learn about how we can work more closely with teach other.

There were three rough parts of the day.  The first enabled staff tutors (the academic staff) to learn more about what was going on from the learner support perspective.  The second was a short talk about 'the day in the life' of a staff tutor.  The final section was an update from the faculty Media Fellow.

During this final session I learnt about a project called JIFL, also known as Journeys from Informal to Formal Learning (which remains a bit of a mystery).  There are also a number of FutureLearn MOOCs (Massively Open On-line Courses) that are either currently being delivered, or are in the process of being developed.  These have the titles: 'Introduction of Cybersecurity' and 'Programming one line at a time', which uses the Python programming language (which is used in M269 Algorithms Data Structures and Computability).  Other MOOCs will include one about smart cities and another about renewable energy.

In other news, there is going to be an update to the OpenLearn site, where potential students can gain access to samples of OU materials.  There is also going to be a new programming module, which is intended to help students transition from the first level modules, such as TU100 My Digital Life and TM129 Technologies in practice, to the second level modules, such as M250 Object-oriented Java programming and M269.  This transition module will draw upon materials from an earlier module, M150 Data Computing and Information, and have a focus on problem solving.

In terms of forthcoming media productions, there was a lot of exciting news: there is going to be a programme about Algorithms (and how they relate to our lives), a programme about the life of Ada Lovelace (which contains a bit about gambling), and a documentary called Game Changer, which is about the developers of the Playstation game, Grand Theft Auto.