Personal Blogs

What follows is a set of notes from a professional development event by the library research support team, which took place on 8 Feb 2024 (which now feels like a long time ago).

This even was of interest for a number of reasons: I supervise some EdD research projects, I’ve been successful in gaining a small amount of seed funding, and I thought it might be useful in my role as an undergraduate project supervisor.

What is a data management plan?

A data management plan (DMP) is written at the start of the project, something that funders need to see. It describes what data you’re doing to collect, how data is going to be stored and secured, how will access be granted.

The exact requirements of a DMP may very between different funders. It is especially important if your research data is going to be uploaded into an archive. It is probably a good idea to write one even if one isn’t explicitly needed. Knowing how to do prepare a good DMP is an important research skill. Your plan can, of course, evolve as your project evolves. 

Every project will have different data requirements. Will your project gather data from interviews, focus groups, or surveys? Different types of data may have different types of management requirements. What is the volume of the data, i.e. how much? How will data be collected? Also, where will your data come from? Will it come from other sources, such as databases or third parties? To help with the preparation of a plan, there is a tool called DMP online, provided by the library.

During this session we were led to an activity and asked the question: what do you think are your main data management challenges for your research project?

What follows are some headings which highlight what elements of data collection, storage and management which need to considered.

Storage and backup

Effective backups prevent accidental losses and deletions. Even if your data is backed up, you need to ensure that backups have integrity - in the sense that if you need to access or use them, you can do so without encountering difficulties or data loss.

When thinking storage and backup consider the size and complexity of your data, and where research is carried out. If you work in “the field” (amongst participants), consider the physical security of your data and find ways to backup your data. You also need to consider how to secure your data if you are working with others.

A practical suggestion for collaborative working was to make use of tools such as Teams or Sharepoint. For individual researchers, an alternative suggestion was to make use of OneDrive (which is a part of the university infrastructure). Keep a backup copy on an external device (and make sure it is encrypted).

Ethical and legal issues

Ethics permeate research. If you’re working with human research participants on research project within the boundary of the university, such as an EdD or PhD project, or a funded research project, you need to register your project with the university’s Human Research Ethics Committee (HREC). A part of this you will need to complete a risk checklist.

You must tell your participants what will happen with the data that is collected as a part of the research, not only during the research, but also after the research project. Specifically, you should tell participants where results may be published and whether dataset may be available afterwards.

Data protection is important. If you collect personal information as a part of your research, you must add details about your project to the university Information Asset Register (IAR). 

Selection and preservation

An important question to be asked is: what data needs to be kept, and what data needs to be destroyed? The data protection legislation is important. You should ensure that any personal data that is collected is deleted at the end of a project. Some documents, such as consent forms, need to be maintained for a considerable amount of time after the project has ended (up to 10 years). How are these records going to be kept? The reason for maintaining these records is to ensure protection of both the participant and the researcher.

Data sharing

When a project has been completed, there may be a necessity to share the dataset with others. One of the reasons for wanting to do that is that fellow researchers might want to study your data to not only confirm your findings or to challenge your conclusions, but also to interrogate your dataset in different ways. The data that you collect (and share)  might be useful for the study of different research questions.

Your data should be shared using a trusted repository that is “as open as possible, but as closed as necessary”.  Sometimes, data can be shared through a research funder repository, a discipline specific repository, or an institutional repository. The OU has a repository called ORDO: OU’s Research Data repository.

An interesting resource that was mentioned was a site called re3data, which is short for Registry of Research Data Repositories. It’s also useful to note that repositories often make use of licences. Different repositories will have different licences.

Responsibilities and resources

Finally, there’s the important question of responsibilities. A key question is: who is responsible for the data management plan? In a plan, it would be useful to identify who is responsible for what data. Who, for example, be responsible for managing the upload of data to a repository. 

Reflections

A key point that underpins this post is: do contact the library team for help and support.

I have two reflections to share. The first is that the way that research data is treated has significantly changed from when I started carrying out research, and this is a good thing. Whilst all this can be perceived as an annoying administrative burden, it necessarily helpful to spell out how data is used. In turn, this can open up possibilities in terms of how data can be used by other researchers. A detailed plan can also offer helpful reassurance to participants.

The second reflection relates to my role of a TM470 project tutor. Undergraduate projects sometime requires students to carry out research that involves people – for example, potential users of software systems. Since undergraduate projects sit outside the formal university (HREC) ethics processes, students don’t have to create a data management plan, there is benefit in considering how data is collected, used, and stored. The reason is, of course, these issues speak to the ethical issues that are important to every project. A practical recommendation for TM470 students is: if you collect data, create a short appendix (it could be only one or two pages in length) that summarises your data management plan.

Acknowledgements

Many thanks to the library research support team for running this event. The headings for this blog have been derived directly from their presentation.

In TM470 LSEPI is an abbreviation for Legal Social Ethical and Professional Issues. A good TM470 project report should clearly address these issues to show the examiner that you have thought about how these issues have impacted on your project, and what you have done to take these into account.

LSEP issues are increasingly important in computing due to the increasing impact that computing and IT has within society. When speaking with students I often a recent example: the Volkswagen emissions scandal. In this case, there are clear environmental impacts and legal implications. It is also clear that both the engineers and leaders have to make ethical decisions.

In TM470, LSEP issues are assessed through the following learning outcome: “LO10. Identify and address the legal, social, ethical and professional issues (LSEPIs) and the equality, diversity and inclusion (EDI) concerns that may arise during the development and use of computing and IT systems.” In the marking of the EMA, this learning outcome is assessed with LO2, which is all about the aims and goals of your project.  When just looking at the number of learning outcomes, and the marks available, the LSEPI section could account for 10 marks.

To gain a top score for this learning outcome a student: “has comprehensively identified the relevant LSEPIs and EDI concerns arising during development and use and modified their project work to take these into account and behaved professionally in all aspects of their project work”. EDI being an abbreviation for equality, diversity and inclusion.

Given the importance of both LSEP and EDI, a suggestion is to include it as a top level section in your report, just before the literature review section. The justification for this is that if you identify some issues that need to be explored in greater depth, you can then go onto provide evidence of your reading.

Module materials

At the time of writing, it takes a bit of digging to find two documents that relate to both LSEP and EDI issues. From the module website, click on the Resources heading, and then click on the Study materials section.

The LSEP document contains the following key headings: working with stakeholders, working with human participants, and asking the right questions. Do review the materials that are presented under these headings and review Appendix A Guidelines for conducting research with human participants. Related to these are two template documents: a sample consent form, and a participant information sheet.

Informed consent is the process through which researchers share the aims and purpose of their research with participants, and gain their approval that they are happy to participate in a study. The accompanying information sheet is designed to offer further information under a set of familiar headings.

When working with participants, I always remember two points. The first is that participants are at liberty to leave a study at any point. The second point is related: the participants are always more important than the research that is being carried out.

The equality, diversity and inclusion section addresses “why equality, diversity and inclusion are relevant to computing and IT professionals”, introduces the concept of protected characteristics, and “unconscious bias is and how it might affect your practice as a computing and IT professional” and what mitigations might be adopted (TM470 module materials).

EDI relates to people, and differences between people, irrespective of whether they are perceived or due to physical, cognitive or sensory impairments. Since Computing and IT products are, ultimately, used by people, it is necessary to consider EDI issues. If you design an app or a website, your product should be accessible to the widest possible group of users. The motivations for doing this are twofold: firstly, there is a legal obligation to ensure that products and systems are accessible under the Equality Act, and secondly, all users are potential customers. If a product isn’t accessible or perceived negatively, a consumer might choose another service that has a more accessible, usable, or appealing interface.

Looking at this issue from a slightly different perspective, if your project uses artificial intelligence or machine learning, it is necessary to question the extent to which biases might exist within either data that informs your project, and the extent to which bias might be potentially reinforced, or even magnified.

Questions to ask

As highlighted earlier, the LSEP materials contains a section that has the title: asking the right questions. 

Go through each of these questions in turn. 

When working through these questions, do think about the stakeholders who are involved with your project. A stakeholder can be thought of anyone who is affected by your project, either directly or indirectly. Ask yourself questions about what data might need to be held and collected, and what bits of legislation might play and impact if you were ever to deploy your project. The Equality Act was mentioned above. You might want to also consider data protection and computer misuse legislation.

If your project is a research report it is important to ask: what might be the impact of my report? If something is discovered by the report, what might be the impact of disclosing the results, or not disclosing the results? The point here is that it is important to go further than just the immediate project, but also to consider wider and broader impacts.

Differences between student projects and university projects

Before university staff can carry out research that involved human participants, they must submit project proposals through a formal ethics panel. The aim of this panel is to make sure that researchers have carefully considered everything, and any potential risks to all participants (and to the university) have been mitigated.

Unlike official university projects, undergraduate and postgraduate projects are not required to go through such a rigorous process. Rather than having an ethics panel and a lot of electronic paperwork to complete, students should think of their tutor or project supervisor as a mini ethics panel.

Interacting with your tutor whilst considering your LSEP and EDI issues should be thought of as a useful and necessary part of your project. Your tutor will be able to offer some thoughts about what needs to be considered. Plus, interactions with your tutor or supervisor can be documented in an appendix of your final reports.

Further resources

A lot of good resources about ethics are available, and some of these resources are mentioned in the module materials. Here are a collection of links that might be useful:

• Association of Computing Machinery (ACM) Code of Ethics
• British Computer Society (BCS) Code of Conduct

For those that find this subject really interesting, there is a whole suggested curriculum about Society, Ethics and Professionalism on the ACM website.

Going through the ethics bit of TM470 gives you a taste of what university researchers have to go through when they plan and design studies that involve human participants. More information about what goes on behind the scenes at the OU is presented through Ethics support for projects: Which studies need review, by whom and why? (OU blog)

Reflections

I find ethics a fascinating subject. In computing it comes into play more than you might initially expect since computing touches on so many different areas of human activity. Rather than being a subject that was once on the periphery of the discipline, I now see it as a topic that has moved to the centre. It is an important and necessary part of becoming a computing professional.

It is also interesting to reflect on how ethics has developed since I was a graduate student. There is now a lot more that has to be done, but this isn’t a bad thing. Additional scrutiny along the way helps researchers to carry out better research. For TM470 students, my key bit of advice is: speak with your tutor; they are your own personal ethics panel.

One of the tasks I have to do pretty regularly is to review academic papers for a journal called Open Learning, which I help to co-edit. 

This blog post is intended as a summary of my own thoughts about how I approach reviewing. This post may be useful for reviewers who are new to reviewing papers for journals not too dissimilar to Open Learning.

The blog is split into three parts. The first part is about how I approach the reading (and interrogating) of a journal submission. In this first bit there are some short cuts which I tend to apply to get a feel for a paper.

The second part is about how I approach the offering of feedback to authors. The overall aim is, of course, to try to help the author of a submission to write a better paper. For this part, I should acknowledge some of the ideas of Simon Bell, a former editor of Open Learning, who put in place a really nice framework.

The final part offers an ethical perspective. This is discussed in three different ways; the ethical responsibilities of the reviewer, ethical responsibilities of an author, and the ethical perspective that must be presented through a paper.

The blog post concludes by sharing some additional resources and sharing some further reflections about the role of the reviewer.

Before beginning, an important question to ask is: why should I review? There are a few answers to this. One reason is that is gives you insight into the peer review process. It also enables you to catch sight of the kinds of papers and research that relates to a field or discipline. Also, in some respects, academics serve the discipline that they study and teach; reviewing for journals can be thought of an extension of that service. Another reason is the practice of reviewing and writing reviews develops your critical perspective. Finally, reviewing is a way to gain academic kudos and experience. If you review for a journal, this is something that you can add to your academic CV.

Reading an academic paper

One of the first things I try to do is to get a feel for the paper as a whole. 

Getting a feel for the paper

A key question to have in mind is: what kind of paper is this?

I begin with the title, then the abstract, then the introduction, and then I immediately go to the references section. My justification for this is: if I recognise some of the references, then I may be able to get a quick (and rough) understanding of the type of research that is being presented. If I don’t recognise any of the papers, then I’ll clearly have to work a lot harder than I would if some of the papers were familiar to me.

Looking at references

Whilst I’m in the references section, I look to see whether a paper has referenced any other articles from the journal that it has been submitted to. If it hasn’t referenced any papers from within the journal, this makes me ask myself the question: is this paper appropriate for the journal?

There are two reasons why references from within the paper is important. Firstly, clearly referencing from within the journal shows that the research is placed amongst and next to existing research. This means that it is likely to be following and connected to existing debates and topics. Secondly, referencing popular papers from within the journal you are submitting to is a good strategy; it enables your work to be more easily discovered by researchers. The reason for this is that many journals allow researchers to follow links between different papers. 

Gaining a critical perspective

The next thing I would do is have a quick look through all the different sections. There are always some key headings that I look for: a section that describe methods, a results section, a discussion section and then a conclusion. If any of these are missing, I would certainly be giving the paper a closer look, and asking why the article wasn’t using these headings.

Checking out the detail

When looking through all the different sections, I would also keep an eye out for any figures or graphs. I would typically ask myself a couple of things. One question would be whether there were any figures or images that were presented in colour. The reason for this is simple: printed versions of the journal are still (currently) important. Although it is unlikely that a researcher might handle a physical copy of an issue in a university library, they may well download a PDF and print a copy out. Secondly, if there were graphs, I would check to see if the axes and titles made sense.

When I’m through with looking at these aspects, I might jump from the introduction to the conclusion. Is there a consistent message between the two sections? Doing this should (ideally) give me a good feel for what the paper is all about.  

The next bit is to read through the methods section to find whether there is a clear description of the research questions, before heading onto the methods section. A key question to ask is: “does the approach make any sense?” Another question to ask is: “is there sufficient detail to enable me to get a view about the methods?”

I must confess to being more confident with assessing qualitative papers than I am with quantitative papers. If I feel that I’m not able to make sense a paper, or feel that I don’t have the appropriate expertise to make a judgement or a proper academic assessment of a submission, I tell editor to make them aware of this. This is something that I pick up on later in the ethics section.

Commenting on a paper

When my former colleague Simon Bell started as a co-editor of Open Learning, he requested that all reviewers should be sent some guidelines.

A version of his guidelines have also been published in the System Practice and Action Research Journal (Bell and Flood, 2011). Essentially, they are a set of constructive directives that are intended to create what we called “the spirit of reviewing”. 

For sake of brevity and this blog, I summarise (and paraphrase) the directives (or guidelines) as follows:

• Always be honest but temper honesty with kindness. Ask the question: “How would I feel if I received this review?”
• Be constructive. Articles have been developed over time and should be read with sympathy and honour. 
• Be fair. Always comment on what I liked as well as what parts of a paper I might have had problems with.
• Be humble and say when I do not understand something; do not present myself as a world authority on a subject. 
• Consider myself as a co-worker who is trying to contribute to a wiser and more exciting script. 
• To help both the editor and the author, indicate if I like the text, say whether I would publish it, and highlight what changes could be made to make the text more enjoyable and if I think the author needs to “adapt/change/re-assess the text in some more challenging manner”.

Even before I had been introduced to Simon’s guidelines, I had implicitly devised a way of providing my own feedback to authors. The approach that I take may be familiar with colleagues:

• Highlight what you think is good about the paper and acknowledge the work that has gone into producing it.
• Highlight areas that you might have concerns with. Explain what could be improved, giving a suggestion about how it might be improved, and describe why these improvements are important.
• Ask whether other papers may be useful for the researcher; offer them help and pointers where you think it is appropriate.
• Be practical; if you feel that there is a lot wrong with the paper, highlight only three points. A thought is to say something about the content, say something about the structure, and say something about how it fits in with the discipline or the journal; this will help the editor too.
• End on a positive note. 

An ethical perspective

It is really important that reviewers carry out reviews in an ethical way. 

There are three different perspectives that need to be kept in mind: the ethical practice of the reviewer, the ethical practice of the author of an article, and the ethical practices that are followed within an article. Each of these perspectives are covered in turn.

The reviewer

Reviewers are in a position of power and privilege; their comments can influence whether an article is published. Reviewers must bear in mind the following perspectives:

Impartiality: Reviewers should be impartial. This means that they should be aware of potential biases they may have about any aspect of a submission. A reviewer should not be familiar with the work that is being reviewed.

Expertise: Reviewers should be confident in their assessment of a paper. If they lack sufficient knowledge or expertise to make a judgement, they should either state this in the review, or let an editor know that they do not have sufficient expertise to carry out a review.

Integrity: Through their articles, authors may share new ideas. These ideas are being shared, in confidence, with reviewers. Any interesting and novel research directions that are suggested through an article should remain entirely with the author of an article. Reviewers should not directly draw on or build upon the work of papers that they review.

The author

Authors of papers must not use text, data or images from unattributed sources. Quotations that are used within a paper must be correctly presented; the source of an author should be mentioned, along with accompanying page numbers. If sources are not referenced correctly and fully, authors run the risk of being accused of plagiarism, which is a term that can be applied to not only intentional copying, but also inadvertent copying.

Authors should also be mindful of a concept known as self-plagarism. This is where an author of an article might make use of their own words which might have been used (and published in) other articles. This can occur, for example, if an author writes a paper that describes their doctoral research. The words they use within a doctoral thesis must be substantially different to the words used within an academic article. The exception to this is when authors deliberately quote their earlier work, and earlier publications.

One of the ethical responsibilities of a reviewer is to make a confident judgement that an author’s submission is their own, and to the best of their knowledge, isn’t using the words of other researchers. Reviewers should also let an editor know if they find that a very similar version of a submission has been published in another journal.

In many cases, the journal editor and publisher will also be able to make use of specialist tools to carry out further checks to ensure the originality of submissions.

The research

Since Open Learning publishes education research, many research articles make use of human participants. Whenever human participants are used, reviewers must ensure that there is sufficient evidence within a paper that suggests that research is carried out in an ethical way.

Simply put, participants must be clearly told about the aims of the research they are a part of, and they should be free to leave, or to decline participation in a study at any point. This view should be clearly presented within a section that describes research methods or procedures. Reviewers should feel free to provide comments if they find that a researcher has not provided sufficient description to enable them to decide whether an ethical approach to research has been adopted.

Research ethics is a subject in its own right, and one that has a rich history. Whilst the journal is not expecting reviewers to be an expert in all aspect of research ethics, there is an expectation that reviewers always ask the question: “has this research been carried out in an ethical way?”. More information about research ethics can be found through the British Educational Research Association Ethical Guidelines for Educational Research (BERA website).

Additional resources

The publisher of Open Learning has provided a set of reviewer guidelines (Taylor and Francis website) which may also be useful.

Taylor and Frances have also published some information about their editorial policies and plagiarism (Taylor and Francis website) which may be helpful for both authors and reviewers.

Reflections

Reviewing can be an interesting and rewarding process. Although I spend most of my time reviewing papers for Open Learning, I have also reviewed papers for conferences, workshops and other journals. One of the benefits of reviewing is that is helps to maintain a connection with a discipline. It is rewarding to see how authors respond to comments, and how reviewers can directly (but implicitly) contribute to the continuing professional development of fellow academics and researchers. I would also emphasise that is isn’t necessarily something that is easy; sometimes there are some papers that are difficult to review, and the accompanying comments can be difficult to write.

To conclude, here is a concise summary of what I perceive to be the benefits of being a reviewer:

• It helps to maintain a link with a discipline.
• It provides a way to give academic service to a discipline, which can be highlighted on an academic CV or resume.
• It helps to develop skills to critically assess academic writing.
• The process of providing feedback helps to develop (and maintain) critical writing skills.
• It helps to further understand the peer review process.