OU blog

Personal Blogs

Christopher Douce

STEM Education Research Group - Mixed Methods

Visible to anyone in the world
Edited by Christopher Douce, Thursday, 26 Jan 2023, 15:48

I belong to a couple of research groups within the School of Computing and Communications; I’m trying to find my research home, after not being involved with research for a while. There’s also an informal group called the STEM Education Research Group, which explores topics that are common to some of the groups that I (occasionally) visit.

On 18 January 2023 I attended a research development event that was facilitated by Ann Grand, Senior Lecturer in Astrobiology Education, that was all about mixed methods.

An example

Ann opened with an example, which was also a research question: how are people using their allotments?

You might count how many people are growing different type of crops, or how many hours a week people are ‘using’ their allotments, or you might try to understand ‘why’ they are using their allotment. The nature of the research question might lead to you choosing different methods: you might gather numbers, or you might want to speak with people who grow things on their allotments. I made a note that there’s a difference between multi-methods to answer different research questions, and mixing of methods.

Choosing methods

I noted down a reference to Tashakari and Cresswell, where mixed methods were described as: “research in which the investigator collects and analyzes data, integrates the findings, and draws inferences using both qualitative and quantitative approaches or methods in a single study or a program of inquiry. integrate everything to produce an interpretation” (2007, p.4)

An important question is: how do decide about to use which methods to use? The answer is: It relates to the overall design of what is being studied.

An important point that I noted down was that mixed methods can often take up more time than if a researcher was only using a single method. This leads to the question: under what circumstances should we use them? What is their value?

A reflection that was made during the session is that controlling for variables in education is profoundly difficult, and therefore, it is almost inevitable that we adopt mixed methods to try to understand what the variables might be. They might also be used to mitigate against the impact of extraneous variables. Also being aware of a range of different evidence may enable you to often understand the question, before even carrying out your research.

My colleague Oli Howson made the following point: “quantitative data is lovely for drawling lines around things but humans are messy and colour/context is important”. Understanding the context can, of course which can lead to other (or related) research questions. A research project might not be about asking or understanding a sequence of questions, it may be more of a messy network of questions which exist within a wider research space.

Value of mixed methods research

Mixed methods can be used to investigate and consider bias, and add meaning to data that has been gathered. One useful quote is by Denzin, who writes: “the bias inherent in any particular data source, investigators and particularly method, will be canceled out when used in conjunction with other data sources, investigation, and methods” (p.14, 1978).

Another quote relates to the application of mixed methods, namely that a mixed approach “facilitates generalization to a wider population, especially when the qualitative sample is directly linked to the quantitative sample” (Hesse-Biber & Johnson, 2015).

There is also the importance of being aware of our own biases and being mindful or how we approach any analysis. These points are related to the subject of reflexivity, which relates to how we position ourselves in relation to any research that we do. Sometimes sharing a little bit more about us (and our position) enables us to add validity to any research that we share.

More than methods…

An important reflection is that the choice of methods is one bit of a much broader picture. Our choice of methods reflects what our research paradigm is, and can link to our philosophy of how we view truth and knowledge. In some ways, using two different sets of methods can be an attempt to bridge conceptual differences between interpretivist and positivist world views. In other words, whether truth (or reality) is subjective, or objective.

The ordering of methods is important. A researcher might use a sequential approach, applying one method after another. 

A qualitative method might be used to identify concerns held by a community, which could then be brought into a survey method to quantify, or to test the extent of concerns that might be held by a wider community. In other words, a quantitative approach could be used to validate a qualitative finding.

Looking from the other perspective, a survey (perhaps using a standard instrument) might signify some interesting or curious results. Qualitative methods could then be used to explore why a certain group of participants hold a particular perspective. In other words, a qualitative approach can be used to provide explanations to accompany quantitative findings.

During this part of the session, there was also a short discussion about the use of literature surveys. Systematic reviews can apply mixed methods. For example, a literature survey could begin with a set of themes that have been identified by a researcher. This identification of themes could be thought of as a qualitative approach. During the next step, a researcher might then to go onto identity how many papers explore or study those themes.

A further example…

Towards the end of the session, we had a look at an example of some educational research which asked the question: what impact do science shows have on attitudes to career intentions? 

I understand a science show to be an engaging demonstration or a talk. In other words, is there an effect on the career aspiration of school children who attended those shows? We had a brief look at some data captured from a research student. This included responses from questionnaires, and responses from a focus group. 

Whilst discussing the research methods applied in this study, there was a further discussion point that emerged, which was about the concept of impact. Specifically, how does ‘impact’ relate to your research questions?

Reflections

Whenever research methods are discussed, there are other broader questions which can and should be asked. These questions relate to the philosophy of research, and the nature of truth, and these discussions inform the research paradigm that you adopt. Before even getting into philosophy and paradigms, it is your research questions that should drive everything. When you know the what needs to be found out, you can then think about the how.

It was great to see Creswell mentioned. I first came across his textbook when studying for my MA in Education. Creswell presents a really detailed summary of what mixed methods research is all about and provides a lot of detail about the methods that can be used and applied.

One of the unexpected points that I took away from this session is how systematic literature reviews can both draw on the quantitative and the qualitive. Thinking back to the literature reviews that I did for my MA and other qualifications, it has struck me that I’ve been applying mixed methods research, but in an informal way. Knowing about terminology makes the informal become formal, and also goes a long way to clarifying processes and how this relates to how research is carried out.

References

Denzin, N. (1978) The research act: A theoretical introduction to sociological methods. New York: Praeger.

Hesse-Biber , S. and Johnson, R. (eds) (2015) The Oxford handbook of multimethod and mixed methods research inquiry. Oxford University Press.

Tashakkori, A. and Creswell, J. (2007) Editorial: The New Era of Mixed Methods Journal of Mixed Methods Research 1(1)3 DOI:10.1177/2345678906293042

Acknowledgements

Some of the phrases and quotes shared through this blog have directly come from Ann, who kindly shared her slides following the event. In some of the earlier sections, I’ve added some further points and reflections from other periods of study. Many thanks to Ann for running a useful session.

Permalink Add your comment
Share post
Christopher Douce

Second EdD residential school

Visible to anyone in the world
Edited by Christopher Douce, Thursday, 4 June 2020, 15:19

On 2 November 2019 I attended a couple of seminars from the second EdD residential school programme which took place in Milton Keynes. I’ve made these notes for the student that I am co-supervising, my co-supervisor, and for any other EdD student who might (potentially) find them useful. I have previously blogged about the first EdD residential school I attended.

At this point in their studies EdD students would have completed an initial study and have submitted some reports. A study and a research project should now be starting to take shape. It was suggested, however, that things are going to get tougher, and students will continue to refine and develop their ideas, plans and projects.

To support students who begin the second year, the EdD group offers a number of online seminars for the second year. Topics include: explaining your ethical approach to your research, demonstrating criticality (which sounds really useful), interviews (and interviewing), and being systematic in your literature review. These sessions are to help students clarify their theoretical approach, understand research methods, reflect on ethics, practice academic writing and to reflect on their development as researchers.

What follows is a summary of the two sessions that I attended. During each talk, I tried to make some rough notes, which I’ve transcribed and edited. Since these are notes, there may well be errors and omissions. I also expect slides and handouts to be made available to students after the event.  This blog post complements an earlier post I made about the first EdD residential school that I attended.

Epistemology and Ontology in Educational Research 

The first seminar was by Kieron Sheehy, Professor of Education. Kieron’s presentation was said to draw on slides that were originally created by John Richardson, and a paper that had been written by Peter Twining, a former OU colleague.

Ontology is the study of beliefs about the nature of reality. This connects to the question of whether there can be an objective reality. Epistemology is the study of how we go about finding stuff out. Unpacking this further, this is about what can be known, and how we might come to find out about the world.  The question of ‘what counts for knowledge’ drives the research. 

When discussing these things, a really important point to take on board is that not everyone thinks the same way. To demonstrate this point, we were asked to complete a short exercise where we were asked to respond to a series of statements about how we viewed research and learning.

If we agreed with a statement, we were asked to put an outstretched hand up in the air. If we disagreed with the statement, we put a fist in the air. If we held a position that was in between, we could choose to present a number of fingers, depending on how we felt about something.

In terms of differences, on one hand there was a view where experiments are important (a positivist approach) versus a view where meaning is created through social interaction, and those meaning can be subjective and different. Another point was that your own epistemological beliefs can influence what you do as a researcher. A later point was that examiners also read a dissertation or research report using their own epistemological beliefs. 

A related question was: why is all this stuff important? From a very practical perspective of being a doctoral student, it is important since a common viva question might be “can you explain your position?” along with “why have you chosen the methods that you have chosen, and how do these relate to your beliefs?” Answering these questions enable you to do your research well and having a stance enables you to tackle issues. An epistemological position directly affects your methods. If you choose a qualitative method, different people may have different realities (understand things in different ways), and meaning can be culturally defined.

Kieron presented one of John’s slides that tried to cover all these different issues in one go.  There were a number of different approaches: positivist, interpretive, critical and pragmatic. A critical approach is where you consider injustices and try to enact change. A pragmatic approach is the application of whatever methods and approaches that may be useful to solve a problem or explore a subject. A positivist approach is where you hold the view that there is an objective reality (and you can observe things), whereas an interpretivist approach means that you try to interpret the subject perspectives and understandings of others. 

It was then onto another task. We were asked to work in pairs to answer the question: ‘what is your position?’ and ‘what are you doing in your research?’ During the discussions I noted down the point that a research design, choice of methods, and ethics must all align to each other. A further point was that if we’re carrying out research into a social world, a reflexive approach is important; we need to ask ourselves how our own position has influenced the research. Similarly, it’s important to be critical and ask: are there other explanations or counter examples? Different perspectives can also be used.

During the session, I examined my own epistemology. I’m a computer scientist who sometimes gets involved in social science and education research. My background means that I’m familiar with the idea of creating systems, or employing an engineering method to solve problems. I asked myself the question: how can an engineering approach fit into the view that knowledge can either be objective (explored using a positivist approach) or subjective (explored using an interpretive approach)?

There’s an answer for this. The engineered artefacts can be studied using a positivist (experimental) approach to answer the question of: ‘does it solve the problem it sets out to solve?’ If we’re thinking about computer software (or, for that matter, any engineered or created product), the produce is used by people, and has been created by people, and people start to use, build and interact with products, all starting with different perspectives. Both epistemological perspectives are necessary to understand the full picture (which is a point which connect to the subject of mixed methods).

Keeping your promises? A focus on anonymisation

The second seminar was about ethics and was presented by Alison Fox, Senior Lecturer in Teaching and Learning. Early on during Alison’s presentation she referred to the ethical protocols that were put in place during a study. An accompanying question was: can you always keep your promises (regarding anonymisation)? 

I made a note of the concept of internal versus external anonymisation. Consider the situation of where a study was described in an academic article or report, where the action of participants is described.  That paper could be read by different groups of people: those people who know nothing of the group that was a focus of the study, or it could be read by members of the group that was the focus of the study. In some cases, participants may be able to recognise both themselves and others from descriptions. It must be asked: would this be acceptable?

CERD is an ethical appraisal framework, which is an abbreviation for Consequential, Ecological, Relational and Deontological. 

Consequential relates to a utilitarian perspective, and this involves the importance of gathering of informed consent, and the avoidance of harm. I also noted down some questions, such as: what do you hope to get from a study? Also, what is the impact in practice? Put another way, what are the consequences of a study?

Ecological relates to all the people affected by the study. This requires the researcher to consider the ‘ecology’ of a group of people, and involves considering the power dynamics, sponsors, leaders and audiences.

Relational is about making ethical decisions to maintain harmonious relations with different people. Another note I made was the importance of showing respect to everyone.

Deontological is about not treating people as tools or subject. Another note I made was that this bit of the framework refers to “necessity of obligation”, or “duties and obligations”.  It’s also important to think about how to maximise benefits from a study, who to tell about your study, and how to report your data. This idea can link back to the earlier example where participants may potentially be identified within their own context.

During Alison’s talk, I also made down a note to dig out the British Educational Research Association, Ethical Guidelines for Educational research (website).

For students who are studying on the EdD programme, I also note that Alison has made a recorded of a tutorial entitled ‘A walk through the CERD ethical appraisal framework’ which is available through the EDD-PW online room.

A brief aside (and a further resource)

As well as being an EdD supervisor, I’m current also an MA Education student with the OU.

I decided to study education (specifically higher education management and leadership) for the reason that I thought that it might help me in my day job (as a middle manager). I’m currently towards the end of my studies, having enrolled in EE813 (module description), a dissertation module, where I’ve got to carry a small amount of educational research.

To prepare students to carry out the MA level research, like EdD students, MA students are also introduced to the ideas of ontology, epistemology, methodology and paradigm. During a recent tutorial, my MA tutor mentioned a really useful video that was contained within the module materials:  David James: How to get clear about method, methodology, epistemology and ontology, once and for all (YouTube).

If you’re interested in research methods and these different ‘ologies’ I do recommend the video. James presents everything through a simple iceberg metaphor. The tip of the iceberg are the methods that you use. Just below the surface (setting the foundation of the methods) is the methodology. The ‘ology’ means that there has been a study of the methods, i.e. choices have been made, leading towards giving an approach. Then there’s epistemology: “what is knowable and worth knowing”; the ‘ology’ of knowledge (or, the debate about what knowledge is).

An interesting example was given where different ways of finding things out can lead to different conclusions, since the research methods begin with different underlying assumptions. Whilst listening to the video, the following phrase struck me: “ontology is simply any debate about being; what it is to be; what it is to exist”. A point was made that this question exists, in different forms, within any discipline. 

I really like this video, since it breaks things down really simply and carefully.  Towards the end of the video there is a reference to the idea of “the position from which you speak” (which links to the idea of reflexivity and the activity which Kieron asked us to complete), and the term ‘paradigm’.

Reflections

Through repeated exposure to all these ‘ologies’ I’m developing a more nuanced view of research methods. One thing that really struck me was the commonalities between what was discussed in the higher level EdD programme residential school, and what was taught in in the MA in education programme. The MA programme seems to do a really good job at preparing students for the EdD. (My next task is to write about ontology and epistemology in my next MA assessment)

One thing that is coming up in my MA studies is a section on Ethics, but it’ll take me a couple of months to get there. It’ll be interesting to see whether the approach that Alison mentioned will also feature in the MA materials. I found Alison’s description of the CRED framework useful, since it clearly encourages students to ask some important and searching questions about how to approach ethics when carrying out research.

Permalink Add your comment
Share post

This blog might contain posts that are only visible to logged-in users, or where only logged-in users can comment. If you have an account on the system, please log in for full access.

Total visits to this blog: 2335230