Personal Blogs

This paper attempts to consider 'learning analytics' from a variety of academic perspectives rather than concentrating solely on education.

The aim of the authors was to identify trends and also assess the most influential voices within the field of learning analytics. As well as individual voices the authors also noted that multiple disciplines were writing about learning analytics and that the relative contribution to the overall conversation between different disciplines was not equal in quantity or influence. Their method was to analyse citations and map their use in a structured network. The assumption was the papers most regularly cited, and by the widest range of contributors, could be considered as being more significant and more likely to be moving the discipline forward.

The observation was that the discipline of education – with its easy access to vast quantities of data – was not being as innovative in using that data as one might expect. Education was using simple demographic data alongside easy checkpoints such as student retention and outcomes. The suggestion was made that the data being collected could be used to contribute to better learning and teaching but, at the time of writing, it was not being used that way.

Education may seem the obvious discipline which will both discuss and utilise learning analytics the paper makes clear that other disciplines are also taking the discipline forward including psychology, philosophy, sociology, linguistics, learning sciences, statistics, machine    learning/artificial intelligence and computer science.

The authors found that the major disciplines – computer science and education – were diverging and that learning analytics was thus going in more than one direction.

They also found that the most commonly cited papers were not empirical research but more conceptual in nature.

The use of ‘low hanging fruit’ (readily available data) is also discussed with hope that better and more useful learning analytics will develop.

The use of citation networks enables the authors to see where concentrations of papers are being published and how they link to one another. They can assess where ‘crossover’ papers develop which feed into the discussion in more than one academic discipline.

It would be easy to assume that the most regularly cited papers are the most insightful, methodologically consistent and ground-breaking. This would be, I think, an over simplification. Certain journals are more widely read within certain disciplines and the specific place a paper is published will determine, to a great extent, its audience.

I can see the value in this kind of analysis. Where many different researchers from different academic backgrounds are all looking at the same subject – albeit from different angles and with different motives – the potential for a ‘big picture’ (and overarching theory) to emerge is an engaging prospect. I also can see how the varied angles and motives can enable each different discipline to consider new ideas and take their own understanding of, and use of, learning analytics forward.  

I got stuck this weekend.

I grasped the concept that Lockyer et al. were communication. The checkpoint vs process comparison is simple but beautifully so. A major criticism of learning analytics, as we have studied to date, is that it necessarily uses the available data (which has not been collected with pedagogical advances in mind) rather than data being collected specifically with analytics in mind, Checkpoint data is the main data we have and shows us the easy to measure metrics - who clicked on the link? When and where? On what device? How long were they logged in to the VLE? How often did they post in the forum? When did they submit their assignment? How good was their assignment? How does any of this correlate with the socio-demographic data we hold about them?

The process data is more difficult to measure being more nuanced and essentially qualitative. Questions which might generate process data could include:

• What was the quality of the forum postings? Did they demonstrate real understanding and engagement with the materials?
  
• Did the individual make thoughtful comments about other people's work? Did those comments lead to a fruitful discussion about differences in perspective?
  
• Has the learner managed to apply any of their learning into a new context such as their own workplace? What were the results of this?
• Is there evidence of meaningful collaboration, knowledge sharing, discussion and debate between learners?

I know that we use Google Analytics at work but not for Elearning. I have never been involved in the discussions about Google Analytics nor have I seen the data. However - as a data junkie I now carry a strong urge to demand access to all the data sets because I can see how interesting this could be.

With regard to applying Google Analytics to learning I am initially struck by how this model is clearly designed with commercial applications in mind. The services which might be of benefit to a learner or an educator are only useful insomuch as they may enable better, more targeted and more appropriate, learning opportunities to be developed.

Google Analytics is, I believe, a commercial enterprise. As with all such enterprises - the principle aim is profit and other gains are incidental. Gains associated with learning are, therefore, mainly for the aspect of learning which is most associated with profit - the institution.

However - this data *could* be used to help the learner and the educator, The issues with using it that was are not only due to a profit motive of a private company but the way that big data may have to become much more specific to be useful in individual circumstances. I would, for example, be very interested to see how well I am doing in H817 compared to other students on the course, compared to students in previous cohorts. I'd be fascinated to know what expectations the OU may have had for me based on my simple socio-demographic information. I would like to see where I rank in terms of engagement with the material and I would be interested to learn of any aspects of H817 I had simply failed to grasp properly. I would love it if the curriculum I followed in H817 would shift and sway according to my interests, the pace I was most comfortable with and even my personal schedule. If I were to know all of this though it may be at the expense of the privacy of other students, it may be at the expense of the integrity of the course and it may be at the expense of my own work ethic!

Learning analytics is clearly a term which is used to describe a particular phenomena which can happen on a wide range of scales.

Personally I am interested in how my individual socio-demographic circumstances can predict my success (or otherwise) in any aspect of learning and how interventions can be targeted to assist me. I am not entirely selfish - I am also interested in how this can be applied to other people!

Professionally part of my role is to tutor aspiring medics and dentists for the UCAT exam (formerly UKCAT) which is an aptitude test which must be taken as part of an application to most medical / dental schools. Upon registration the UCAT consortium ask each applicant to complete a series of socio-demographic questions - racial group, gender, level of education reached by parents (the vast majority of applicants are 17 years old) and the job title of their parents (presumably used, alongside education, as a proxy for social class). They already have the applicants post code and know if the applicant lives in a household with low enough income to qualify for a bursary. This information is not supplied to universities but is reportedly used to improve the UCAT for future cohorts. Pilot questions are trialed with a view to ensuring they are as equitable as possible. I imagine that groups of statisticians gather around data sets and assess why various questions performed better with people who had parents with higher educational qualifications, or who shared a similar ethnic group. (Incidentally - I teacher friend of mine got very frustrated one year when the disadvantaged children who attended her inner city primary school were faced with SATS questions about going to the theatre, and using a honey dipper - two activities which were entirely unknown to most of the children she taught. A huge failing in learning analytics or a cynical attempt to maintain existing privilege? Who knows!)

The articles we read in this activity looked at early learning analytics on a national scale. I also find this very interesting. The variation between educational styles and ethos between different countries is driven by many factors including culture, politics, and resources and comparisons can be meaningful and meaningless depending on how well the complexities of the issues are understood.

The first paper compares the USA to other countries - noting not only a lack of progress when compared to nations of similar levels of development and wealth - but also an up and coming threat from developing nations with very large populations.

The second paper introduces the idea of using analytics to shape future US educational policy at the national level with a coherent and unified plan. Affordability and value are key values but a need to match education to likely economic requirements is also significant.

The basic premise of these discussions assumes that there is a direct correlation between graduate level education and national prosperity. There is, at least in these papers in this time, little discussion about what is being studied and to what end - merely that high levels of education are necessary for continued improvement.

As is pointed out - both papers were written before 'learning analytics' was a phrase and though they clearly are referring to a similar process the absence of 'big data' on quite the scale as it is available today means that it's not exactly the same thing being discussed. However - the idea of analytics is clearly here with a vision to use data to improve polcy.

As if often the case in Wikipedia, the article on Learning Analytics began as a quick summary and rapidly mushroomed into a far more extensive treatise on the subject. However - the initial definition has had few versions. It changed in the first day, then again a couple of years later, but the sentence written in 2013 is the same as the sentence which opens the article today. The difference is that in today's article this opening sentence is followed by over 4000 words of further information.

I usually like to begin my investigations about an unfamiliar subject with a read of the associated Wikipedia article. I realize that it's not a peer reviewed, 'reliable' source but it is often succinct, accessible (especially to to the non-expert) and well written with good clarity. The learning analytics article is none of these things and it reads as an article written by committee (which is, of course, exactly what it is!).

The impression that the whole article gives me is that the subject of 'Learning Analytics' is as vast, as nebulous, as complex and as multifaceted as the two words would imply. H800 challenged every internal impression and idea I had about the concept of 'learning' so I am keenly aware of how 'simple' ideas can become mosaic when investigated and the word 'analytics' gives us no expectation of a simple and easily defined concept! Put two big concepts together and the creation of a gargantuan concept seems also inevitable!

The simple sentences above describe aspects of learning analytics. My impression is not that those who change the definition claim what is stated is incorrect, but that it's incomplete and inadequate. The extra information, text, ideas and paragraphs don't detract from what has been previously written as much as adding to, augmenting and complementing it. There are a multitude of associated concepts which overlap with Learning Analytics but the edges of each concept is blurry and undefined.

I suspect a concise definition which will satisfy everyone is impossible to develop but by looking at the areas everyone agrees with we can draw some conclusions. Such commonalities include:

• Data - the data is described as 'intelligent' and processes related to collecting, collating and analysing this data are all part of the definitions. Data is an inescapable part of Learning Analytics. It's the key ingredient and without data there can be no analytics.
• Discover, understand - data can enable a great deal of knowledge to be amassed and that knowledge can lead to understanding of crucial patterns within learning and teaching
  
• Prediction, modelling, advising and optimising - four different but overlapping ideas in this context. The way in which the data is used is part of what makes Learning Analytics what it is. The purpose of LA is, at least in part, the improvement of the learning journey for the individual and the cohort.
  

In my professional context the remaining three technologies are inextricably linked - we don't currently have software which enables us to track our learners so we don't have any data to drive assessment or to analyse learning or even to offer prizes! I would love to do this but I suspect that the cost of tracking software (with the associated legal ramifications and headache) would pale into insignificant against the cost of effective analysis of the data it would produce let alone the effective use of the data analysis in order to drive assessment and target future learning. 

If we were to change the software we use to provide good tracking data the other things we would have to do would include:

1. Retrospective 'coding' of existing questions to classify what skill, subject, knowledge and application they are testing

2. Retrospective and future  'coding' of our customers to enable two sided analysis 

3. The development of an analysis methodology and the appropriation of software to support that

All of this could be very worthwhile to provide our learners with targeted resources, personalized analysis of their strengths and areas for improvement and to identify any globally weak / strong areas so we can target future developments BUT pragmatically I don't think our particular market is large enough to justify the costs associated with this. 

Personally - I'm self obsessed enough to want to see analysis on my learning done by some formally recognized software! I am mercenary enough to want as many virtual badges and points as you can shovel in my direction!