Personal Blogs

STEM Education Research Group - Mixed Methods

Thursday, 26 Jan 2023, 14:12

Visible to anyone in the world

Edited by Christopher Douce, Thursday, 26 Jan 2023, 15:48

I belong to a couple of research groups within the School of Computing and Communications; I’m trying to find my research home, after not being involved with research for a while. There’s also an informal group called the STEM Education Research Group, which explores topics that are common to some of the groups that I (occasionally) visit.

On 18 January 2023 I attended a research development event that was facilitated by Ann Grand, Senior Lecturer in Astrobiology Education, that was all about mixed methods.

An example

Ann opened with an example, which was also a research question: how are people using their allotments?

You might count how many people are growing different type of crops, or how many hours a week people are ‘using’ their allotments, or you might try to understand ‘why’ they are using their allotment. The nature of the research question might lead to you choosing different methods: you might gather numbers, or you might want to speak with people who grow things on their allotments. I made a note that there’s a difference between multi-methods to answer different research questions, and mixing of methods.

Choosing methods

I noted down a reference to Tashakari and Cresswell, where mixed methods were described as: “research in which the investigator collects and analyzes data, integrates the findings, and draws inferences using both qualitative and quantitative approaches or methods in a single study or a program of inquiry. integrate everything to produce an interpretation” (2007, p.4)

An important question is: how do decide about to use which methods to use? The answer is: It relates to the overall design of what is being studied.

An important point that I noted down was that mixed methods can often take up more time than if a researcher was only using a single method. This leads to the question: under what circumstances should we use them? What is their value?

A reflection that was made during the session is that controlling for variables in education is profoundly difficult, and therefore, it is almost inevitable that we adopt mixed methods to try to understand what the variables might be. They might also be used to mitigate against the impact of extraneous variables. Also being aware of a range of different evidence may enable you to often understand the question, before even carrying out your research.

My colleague Oli Howson made the following point: “quantitative data is lovely for drawling lines around things but humans are messy and colour/context is important”. Understanding the context can, of course which can lead to other (or related) research questions. A research project might not be about asking or understanding a sequence of questions, it may be more of a messy network of questions which exist within a wider research space.

Value of mixed methods research

Mixed methods can be used to investigate and consider bias, and add meaning to data that has been gathered. One useful quote is by Denzin, who writes: “the bias inherent in any particular data source, investigators and particularly method, will be canceled out when used in conjunction with other data sources, investigation, and methods” (p.14, 1978).

Another quote relates to the application of mixed methods, namely that a mixed approach “facilitates generalization to a wider population, especially when the qualitative sample is directly linked to the quantitative sample” (Hesse-Biber & Johnson, 2015).

There is also the importance of being aware of our own biases and being mindful or how we approach any analysis. These points are related to the subject of reflexivity, which relates to how we position ourselves in relation to any research that we do. Sometimes sharing a little bit more about us (and our position) enables us to add validity to any research that we share.

More than methods…

An important reflection is that the choice of methods is one bit of a much broader picture. Our choice of methods reflects what our research paradigm is, and can link to our philosophy of how we view truth and knowledge. In some ways, using two different sets of methods can be an attempt to bridge conceptual differences between interpretivist and positivist world views. In other words, whether truth (or reality) is subjective, or objective.

The ordering of methods is important. A researcher might use a sequential approach, applying one method after another.

A qualitative method might be used to identify concerns held by a community, which could then be brought into a survey method to quantify, or to test the extent of concerns that might be held by a wider community. In other words, a quantitative approach could be used to validate a qualitative finding.

Looking from the other perspective, a survey (perhaps using a standard instrument) might signify some interesting or curious results. Qualitative methods could then be used to explore why a certain group of participants hold a particular perspective. In other words, a qualitative approach can be used to provide explanations to accompany quantitative findings.

During this part of the session, there was also a short discussion about the use of literature surveys. Systematic reviews can apply mixed methods. For example, a literature survey could begin with a set of themes that have been identified by a researcher. This identification of themes could be thought of as a qualitative approach. During the next step, a researcher might then to go onto identity how many papers explore or study those themes.

A further example…

Towards the end of the session, we had a look at an example of some educational research which asked the question: what impact do science shows have on attitudes to career intentions?

I understand a science show to be an engaging demonstration or a talk. In other words, is there an effect on the career aspiration of school children who attended those shows? We had a brief look at some data captured from a research student. This included responses from questionnaires, and responses from a focus group.

Whilst discussing the research methods applied in this study, there was a further discussion point that emerged, which was about the concept of impact. Specifically, how does ‘impact’ relate to your research questions?

Reflections

Whenever research methods are discussed, there are other broader questions which can and should be asked. These questions relate to the philosophy of research, and the nature of truth, and these discussions inform the research paradigm that you adopt. Before even getting into philosophy and paradigms, it is your research questions that should drive everything. When you know the what needs to be found out, you can then think about the how.

It was great to see Creswell mentioned. I first came across his textbook when studying for my MA in Education. Creswell presents a really detailed summary of what mixed methods research is all about and provides a lot of detail about the methods that can be used and applied.

One of the unexpected points that I took away from this session is how systematic literature reviews can both draw on the quantitative and the qualitive. Thinking back to the literature reviews that I did for my MA and other qualifications, it has struck me that I’ve been applying mixed methods research, but in an informal way. Knowing about terminology makes the informal become formal, and also goes a long way to clarifying processes and how this relates to how research is carried out.

References

Denzin, N. (1978) The research act: A theoretical introduction to sociological methods. New York: Praeger.

Hesse-Biber , S. and Johnson, R. (eds) (2015) The Oxford handbook of multimethod and mixed methods research inquiry. Oxford University Press.

Tashakkori, A. and Creswell, J. (2007) Editorial: The New Era of Mixed Methods Journal of Mixed Methods Research 1(1)3 DOI:10.1177/2345678906293042

Acknowledgements

Some of the phrases and quotes shared through this blog have directly come from Ann, who kindly shared her slides following the event. In some of the earlier sections, I’ve added some further points and reflections from other periods of study. Many thanks to Ann for running a useful session.

Tags: quantitative research, qualitative research, mixed-methods, cpd, methodology, research, research methods, mixed methods

Permalink Add your comment

Second EdD residential school

Wednesday, 20 Nov 2019, 14:40

Visible to anyone in the world

Edited by Christopher Douce, Thursday, 4 June 2020, 15:19

On 2 November 2019 I attended a couple of seminars from the second EdD residential school programme which took place in Milton Keynes. I’ve made these notes for the student that I am co-supervising, my co-supervisor, and for any other EdD student who might (potentially) find them useful. I have previously blogged about the first EdD residential school I attended.

At this point in their studies EdD students would have completed an initial study and have submitted some reports. A study and a research project should now be starting to take shape. It was suggested, however, that things are going to get tougher, and students will continue to refine and develop their ideas, plans and projects.

To support students who begin the second year, the EdD group offers a number of online seminars for the second year. Topics include: explaining your ethical approach to your research, demonstrating criticality (which sounds really useful), interviews (and interviewing), and being systematic in your literature review. These sessions are to help students clarify their theoretical approach, understand research methods, reflect on ethics, practice academic writing and to reflect on their development as researchers.

What follows is a summary of the two sessions that I attended. During each talk, I tried to make some rough notes, which I’ve transcribed and edited. Since these are notes, there may well be errors and omissions. I also expect slides and handouts to be made available to students after the event. This blog post complements an earlier post I made about the first EdD residential school that I attended.

Epistemology and Ontology in Educational Research

The first seminar was by Kieron Sheehy, Professor of Education. Kieron’s presentation was said to draw on slides that were originally created by John Richardson, and a paper that had been written by Peter Twining, a former OU colleague.

Ontology is the study of beliefs about the nature of reality. This connects to the question of whether there can be an objective reality. Epistemology is the study of how we go about finding stuff out. Unpacking this further, this is about what can be known, and how we might come to find out about the world. The question of ‘what counts for knowledge’ drives the research.

When discussing these things, a really important point to take on board is that not everyone thinks the same way. To demonstrate this point, we were asked to complete a short exercise where we were asked to respond to a series of statements about how we viewed research and learning.

If we agreed with a statement, we were asked to put an outstretched hand up in the air. If we disagreed with the statement, we put a fist in the air. If we held a position that was in between, we could choose to present a number of fingers, depending on how we felt about something.

In terms of differences, on one hand there was a view where experiments are important (a positivist approach) versus a view where meaning is created through social interaction, and those meaning can be subjective and different. Another point was that your own epistemological beliefs can influence what you do as a researcher. A later point was that examiners also read a dissertation or research report using their own epistemological beliefs.

A related question was: why is all this stuff important? From a very practical perspective of being a doctoral student, it is important since a common viva question might be “can you explain your position?” along with “why have you chosen the methods that you have chosen, and how do these relate to your beliefs?” Answering these questions enable you to do your research well and having a stance enables you to tackle issues. An epistemological position directly affects your methods. If you choose a qualitative method, different people may have different realities (understand things in different ways), and meaning can be culturally defined.

Kieron presented one of John’s slides that tried to cover all these different issues in one go. There were a number of different approaches: positivist, interpretive, critical and pragmatic. A critical approach is where you consider injustices and try to enact change. A pragmatic approach is the application of whatever methods and approaches that may be useful to solve a problem or explore a subject. A positivist approach is where you hold the view that there is an objective reality (and you can observe things), whereas an interpretivist approach means that you try to interpret the subject perspectives and understandings of others.

It was then onto another task. We were asked to work in pairs to answer the question: ‘what is your position?’ and ‘what are you doing in your research?’ During the discussions I noted down the point that a research design, choice of methods, and ethics must all align to each other. A further point was that if we’re carrying out research into a social world, a reflexive approach is important; we need to ask ourselves how our own position has influenced the research. Similarly, it’s important to be critical and ask: are there other explanations or counter examples? Different perspectives can also be used.

During the session, I examined my own epistemology. I’m a computer scientist who sometimes gets involved in social science and education research. My background means that I’m familiar with the idea of creating systems, or employing an engineering method to solve problems. I asked myself the question: how can an engineering approach fit into the view that knowledge can either be objective (explored using a positivist approach) or subjective (explored using an interpretive approach)?

There’s an answer for this. The engineered artefacts can be studied using a positivist (experimental) approach to answer the question of: ‘does it solve the problem it sets out to solve?’ If we’re thinking about computer software (or, for that matter, any engineered or created product), the produce is used by people, and has been created by people, and people start to use, build and interact with products, all starting with different perspectives. Both epistemological perspectives are necessary to understand the full picture (which is a point which connect to the subject of mixed methods).

Keeping your promises? A focus on anonymisation

The second seminar was about ethics and was presented by Alison Fox, Senior Lecturer in Teaching and Learning. Early on during Alison’s presentation she referred to the ethical protocols that were put in place during a study. An accompanying question was: can you always keep your promises (regarding anonymisation)?

I made a note of the concept of internal versus external anonymisation. Consider the situation of where a study was described in an academic article or report, where the action of participants is described. That paper could be read by different groups of people: those people who know nothing of the group that was a focus of the study, or it could be read by members of the group that was the focus of the study. In some cases, participants may be able to recognise both themselves and others from descriptions. It must be asked: would this be acceptable?

CERD is an ethical appraisal framework, which is an abbreviation for Consequential, Ecological, Relational and Deontological.

Consequential relates to a utilitarian perspective, and this involves the importance of gathering of informed consent, and the avoidance of harm. I also noted down some questions, such as: what do you hope to get from a study? Also, what is the impact in practice? Put another way, what are the consequences of a study?

Ecological relates to all the people affected by the study. This requires the researcher to consider the ‘ecology’ of a group of people, and involves considering the power dynamics, sponsors, leaders and audiences.

Relational is about making ethical decisions to maintain harmonious relations with different people. Another note I made was the importance of showing respect to everyone.

Deontological is about not treating people as tools or subject. Another note I made was that this bit of the framework refers to “necessity of obligation”, or “duties and obligations”. It’s also important to think about how to maximise benefits from a study, who to tell about your study, and how to report your data. This idea can link back to the earlier example where participants may potentially be identified within their own context.

During Alison’s talk, I also made down a note to dig out the British Educational Research Association, Ethical Guidelines for Educational research (website).

For students who are studying on the EdD programme, I also note that Alison has made a recorded of a tutorial entitled ‘A walk through the CERD ethical appraisal framework’ which is available through the EDD-PW online room.

A brief aside (and a further resource)

As well as being an EdD supervisor, I’m current also an MA Education student with the OU.

I decided to study education (specifically higher education management and leadership) for the reason that I thought that it might help me in my day job (as a middle manager). I’m currently towards the end of my studies, having enrolled in EE813 (module description), a dissertation module, where I’ve got to carry a small amount of educational research.

To prepare students to carry out the MA level research, like EdD students, MA students are also introduced to the ideas of ontology, epistemology, methodology and paradigm. During a recent tutorial, my MA tutor mentioned a really useful video that was contained within the module materials: David James: How to get clear about method, methodology, epistemology and ontology, once and for all (YouTube).

If you’re interested in research methods and these different ‘ologies’ I do recommend the video. James presents everything through a simple iceberg metaphor. The tip of the iceberg are the methods that you use. Just below the surface (setting the foundation of the methods) is the methodology. The ‘ology’ means that there has been a study of the methods, i.e. choices have been made, leading towards giving an approach. Then there’s epistemology: “what is knowable and worth knowing”; the ‘ology’ of knowledge (or, the debate about what knowledge is).

An interesting example was given where different ways of finding things out can lead to different conclusions, since the research methods begin with different underlying assumptions. Whilst listening to the video, the following phrase struck me: “ontology is simply any debate about being; what it is to be; what it is to exist”. A point was made that this question exists, in different forms, within any discipline.

I really like this video, since it breaks things down really simply and carefully. Towards the end of the video there is a reference to the idea of “the position from which you speak” (which links to the idea of reflexivity and the activity which Kieron asked us to complete), and the term ‘paradigm’.

Reflections

Through repeated exposure to all these ‘ologies’ I’m developing a more nuanced view of research methods. One thing that really struck me was the commonalities between what was discussed in the higher level EdD programme residential school, and what was taught in in the MA in education programme. The MA programme seems to do a really good job at preparing students for the EdD. (My next task is to write about ontology and epistemology in my next MA assessment)

One thing that is coming up in my MA studies is a section on Ethics, but it’ll take me a couple of months to get there. It’ll be interesting to see whether the approach that Alison mentioned will also feature in the MA materials. I found Alison’s description of the CRED framework useful, since it clearly encourages students to ask some important and searching questions about how to approach ethics when carrying out research.

Tags: ee813, epistemology, ontology, methodology, research methods, paradigms, edd, education research

Permalink Add your comment

EdD residential weekend, June 2018

Thursday, 6 Sept 2018, 09:09

Visible to anyone in the world

Edited by Christopher Douce, Thursday, 21 Nov 2019, 11:21

Between Friday 8 June and Sunday 10 June 2018 I attended an EdD residential weekend at the university campus in Milton Keynes. The EdD residential weekend was something that was new to me: I was attending in the capacity of a ‘co-supervisor’ (or ‘second supervisor’).

The EdD qualification is a doctorate in education that is at the same level as a PhD, except for one fundamental difference: the research and contribution to knowledge carried out through an EdD is situated in the educational practice or context of the student who is carrying out the research.

One of the things that I learnt from the weekend was that other institutions have their own EdD programmes. Since 1997 more than 370 students have been awarded an EdD through the OU’s EdD programme.

What follows is a transcription and summary of some of the notes that I made during the weekend. There are mostly from my perspective of a supervisor, but they might be of interest and use to EdD students or anyone who is interested in learning more about what EdD research and study entails.

Introduction

The event was introduced by Inma Alvarez, the university’s Centre for Research in Education and Educational Technology EdD programme director. Inma emphasised that the EdD is a professional doctorate that enables students to gain skills in educational research and enquiry and be able to carry out a study that contributes to professional and practice knowledge.

The EdD takes at least 3 and a half years, with a maximum of 6 years. Students are provided with two supervisors; a lead supervisor and a co-supervisor. During the first year, students are required to design and carry out a preliminary study.

Progress on the EdD is not measured through tutor marked assignments (TMAs) but by a series of progress reports. At the end of the first year, students are required to produce a report which is assessed by an academic who is neither of their supervisors. Inma made an important point that a lot of the responsibility is down to the student; an EdD should take notes of their supervision meetings and actively manage their supervisors!

All the students were given a number of useful tips, including: treat the programme guide as your ‘bible’ and subscribe to the student forums, so students can get updates of when people post messages, updates and questions.

The first year is all about becoming an independent researcher, which includes carrying out a literature review, carrying out that initial study, submitting 4 progress reports and the end of year progress report. The final report will contain an introduction to the project, a summary of the research questions, a literature review, a section about the methodology that is adopted, a description of an initial study, outcomes, and a detailed reference section.

In the second and third year students will ‘follow a more independent and individual programme supported by their supervisors’.

Students will have access to resources, which includes access to the OU graduate school network, the EdD programme website and online doctoral training resources. Another important message that was coming through was: ‘be responsible for your own development’, and a connected thought is to start a reflective diary. This diary can be used to keep notes about what is studied and what is learnt, help to develop academic writing and creativity.

Doctoral researchers and supervisors

The aim of this next session was to enable supervisors to meet their students and members of the EdD team. Some notes that I made from this session were about “gaining confidence in plans, getting used to critical feedback, getting some research training, understanding research ethics, talking to some EdD graduates and becoming a research professional”.

I also made a note that there was a group discussion about the question: what is theory in education? I noted that there is the concept of ‘critical theory’, but there are other approaches and theoretical tools that could be used, such as critical pedagogy and activity theory. This said, I was also mindful that the educational research notion of ‘theory’ is slightly different to a scientific understanding of what a theory is.

Day 2: Doctorate in Education Literature Review

The second day began with a presentation by Ursula Stickler in a library seminar room. The aim of the session was to learn more about ‘how to approach a literature review, the criteria, and practical considerations, such as knowing when and where to stop’.

We were given an activity, where we were asked the question: why are you doing a literature review? Answers included: looking at themes, examining ideas and methods, examining debates, learning about academic literacies and making sure you’re not duplicating your research. Other answers also included identifying key authors and researchers and uncovering your own view of the literature and what has been done before. I also noted down some key terms that were used in the REF, the Research Excellence Framework: originality, significance and rigour.

We were then guided to another activity, where we had to answer the question: how to best go about a literature review? Other questions that were asked included: where to start, where to finish, what to include and what to leave out. It was also important to ask the question: what are the key journals and writers? It’s also important to be clear about what the main argument (or arguments) are. Another note I made was: narrow your search, find your gap (within the research) and widen your implications (which I assume relates to the impact that your research can make).

The final activity asked the question: what are the strengths of a good literature review? I didn’t make too many notes during this part of the event, except that the discussions were focussed upon an article by Boote and Beile entitled ‘Scholars before researchers: On the centrality of the dissertation literature review in research preparation’.

Session: Ethics

Ethics are important. The first session on ethics was facilitated by Alison Fox and Kris Stutchbury. I made a note that “your entire project is an ethical task” with an accompanying comment that how students choose their research projects, carry out research tasks and disseminate their research results are all ethical tasks. In this session I was introduced to an new acronym: CURD, which stands for Consequence, Ecological, Relational and Deontological.

The next ethical session was all about case studies. Duncan Banks gave a presentation that had the title: an introduction to research ethics (PDF). We were introduced to the BPS, British Psychological Society, code of ethics of human participants. Some of the key points I noted down were: respect for the autonomy and dignity of persons, that the research must have scientific value, quality and integrity, and that it must maximise benefit and minimise harm. Another dimension of ethics relate to risks, both to research participants and also to the researcher.

Day 3: Designing an initial case study

The third day of the event was organised slightly differently; we were all brought together for a plenary presentation, and then we were able to attend different parallel sessions. In some respects, the weekend turned into a mini conference! What follows is a polished and paraphrased version of the notes that I made during each of these sessions.

Opening plenary session

The opening session was presented by Felicity Fletcher-Campbell.

Literature review

Felicity returned to one of the important topics of the weekend; the literature review. The literature review should take account of a theoretical position, the substantive area and a methodology. A literature review is ongoing, flexible, adaptable, malleable, reactive and proactive. During the literature review, students should add and remove papers, and also reconceptualise their work. It offers a means to inform your empirical work. A key phrase I noted down was “keep it slim and purposeful”, which I thought was great advice.

The initial study

The initial study is important since students need to carry this out to complete an important assessment within the EdD. Students must write a report that must present a clearly structured framework for the whole study. In some ways, this initial study and accompanying report is used to ‘sort out’ any issues regarding theory or theoretical position. I noted that the “report shows your developing knowledge and experience of relevant theoretical traditions and literature”. It is used to critically assess where the different authors and researchers are coming from and their accompanying perspective. The report also allows students to relate the literature to their research questions.

Felicity offered some really useful tips and pointers: “your theoretical position informs your methodology” and “buy yourself a very big box of quotation marks and inverted commas” and “be really boring and put quotes around everything and be obsessed with page numbers”. On the subject of ethics, students were told: “name your supervisors on consent forms, so they get blamed too”.

On the subject of time, “research time is different than normal time; time fills up, everything will take longer than you think they will take”.

There’s also the need to balance everything; to balance the preparation and the doing, the data production and the data collection, and the analysis with the report writing. Also, when it comes to the writing, “the initial study will help you to explain the genesis of your main study”.

The main study

Some key points about main study were: “you need to tell a story” and to ask “what I need to tell the reader? What do they need to know? Why did you tell me this? Also, why didn’t you tell me this earlier?” From a personal perspective, I’ve internalised the point about the need to tell a story, and I’ve passed this message onto the TM470 Computing and IT project students that I help to support. The narrative that is presented to the examiner is really important.

Parallel Session: Working with digital data

The first parallel session that I went to was by Carol Azumah who discussed the usage of digital data and resources, such as blogs and social media. Some resources, such as blogs, can be viewed as public documents. Two terms that I noted down were ‘discourse oriented online ethnography’ and ‘fast ethnography’. An important point is that ethics always need to be considered. We were directed to the Ethics pages of the Association of Internet Researchers (AoIR).

Parallel Session: Concept maps

Diane Harris’s session was about how to use concept maps in research. Concept maps were introduced as tools that can be used to hear the voice of participants and “for them to own what they have said to you”. Diane offered a specific example of how they were used to study music education in a school. Participants could own, add to and create maps. The resulting maps could then be analysed using thematic analysis or critical incident analysis. Regarding this second technique, Diane mentioned two researchers, Harrison and Lee (2011) (Taylor and Francis) who used the approach in medical education.

Closing session: the way ahead

After the parallel sessions, we returned to the plenary room, where we were offered some closing advice from Inma Alvarez. From what I remember and from what I’ve noted down, students were encouraged to work as and think of themselves as independent researchers. They should also think of their supervisors as critical friends. Students were encouraged to identify what skills are needed, reflecting earlier attention that was given to the importance of continuing professional development.

The concluding bits of advice were: “be open to the unexpected; you can modify the title of your study [if you need to]; work to deadlines and use frameworks to guide what you do, and be sure to manage your supervisors”.

Reflections

What really impressed me was how well the EdD weekend was planned. There were ample opportunities to speak with supervisors and fellow students between more formal events and activities. It struck me as being a really nice mix.

There were a couple of highlights. The first one was a presentation by a former EdD student who spoke of some of the challenges of doctoral study. The second was the talk by Felicity Fletcher-Campbell, which was packed filled with useful and practical advice and delivered in a thoroughly engaging way.

I never took place in a formal induction session when I embarked on my own doctoral studies. What really impressed me with the weekend was its emphasis on structure; the importance of the literature review, the importance of the initial study, the main study and how everything connects together. I think the weekend has also positively impacted on my own practice; only by writing this blog have I realised that I have started to pass on some of the tips mentioned during this weekend to some of the undergraduate project students that I support.

Tags: literature review, research methods, residential weekend, doctoral research, research, edd

Permalink Add your comment

This blog might contain posts that are only visible to logged-in users, or where only logged-in users can comment. If you have an account on the system, please log in for full access.

Total visits to this blog: 2339106