A couple of years ago I attended a conference held by the Association of Learning Technology (ALT). I remember a riveting keynote that asked the members of the audience to consider not only the design and the structure of new learning technologies, but also the benefits that they can offer institutions, particularly in terms of costs. I remember being reminded that technology (in terms of educational technology) need not be something that is considered in terms of 'high technology'. A technology could be a bunch of cards tied to a desk chair!
I came away with a message that it is important to try to assess the effectiveness of systems that we construct, and not to get ahead of ourselves in terms of building systems that might not do the job they set out to do, or in fact may even end up solving the wrong problem.
When attending the learning technologies exhibition a couple of months ago the notions of 'return on investment' and 'efficiency' were never too far away. E-learning, it is argued, can help companies to develop training capacities quickly and efficiently and allow company employees to prepare for change.
I might have been mistaken but I think I may have heard the occasional presenter mentioning some figures. The presenters were saying things like, 'if you implement our system in this way, it can help you save a certain number of millions of pounds, dollars or euros per year'. Such proclamations made me wonder, 'how do you go about measuring the return on investment into e-learning systems?' (or any other learning technology system, for that matter).
I do not profess to be an expert in this issue by any means. I am aware that there have been many others (who have greater levels of expertise than myself) have both blogged and written about this issue at some length. I hope by sharing my own meagre notes on the subject might make a small contribution to the debate.
Let's say you build a learning technology or an e-learning tool. You might create an on-line discussion forum or even a hand held classroom response system. You might have created it with a particular problem (or pedagogic practice) in mind. When you have built it, it might be useful to determine whether the learning technology helps learners to learn and acquire new understandings and knowledge.
One way of conducting an evaluation is to ask them about their experience. This can allow you to understand how it worked, whether it was liked, what themes were learnt and what elements of a system, product or process might have inspired learners.
Another way to determine whether a technology is affective is to perform a test. You could take two groups, one that has access to the new technology, the other who do not, and see which group perform better. Of course, such an approach can open up a range of interesting ethical issues that need to be carefully negotiated and conquered.
Dimensions of measurement
When it comes to large e-learning systems the questions that can uncover learners experience can relate to a 'low level' understanding of how learning technologies are used and applied. Attempting to measure the success of a learning technology or e-learning system for a whole department or institution could be described as a 'high level' understanding. It is this 'high level' of understanding that relates to the theme of how much money a system may help to save (or cost) an organisation.
Bearing in mind that organisations are very unlikely to carry out experiments, how is it possible to determine how much return on investment an e-learning system might give you? This is a really tough question to ask since it depends totally on the objectives of a system. The approach taken to measure the return on investment of a training system is likely to be different to one that has been used to instil institutional values or create new ways of working (which may or may not yield employee productivity improvements).
When considering the issue of e-learning systems that aim to train (I'm going to try to steer clear of the debates around what constitutes training and what constitutes education), the questions that you might ask include:
- What are the current skills (or knowledge base) of your personnel?
- What are the costs inherent in developing a training solution?
- What are the costs inherent in rolling this out to those who need access?
Another good question to ask is: what would you have to do to find out the same information if you had not invested in the new technologies? A related question is: would there be any significant travel costs attached to finding out the information?, and, would it be possible to measure the amount of disruption that might take place if you have to ask other people for the information that you require?
These questions relate to actions that can be measured. If you can put a measure on the costs of finding out key pieces of information before and after the implementation of a system, you might be able to edge towards figuring out the value of the system that you have implemented. What, for example, is the cost of running the same face to face training course every year as opposed to creating a digital equivalent that is supported by a forum that is supported by an on-line moderator? You need to factor in issues such as how much time it might take for a learner to take the course. Simply providing an on-line course is not enough. Its provision needs to be supported and endorsed by the organisation that has decided to sponsor the development of e-learning.
The above group of points represents a rather simplistic view. The introduction of a learning technology system may also facilitate the development of new opportunities that were perhaps not previously possible. 'Up skilling' (or whatever it is called) in a limited amount of time may enable employees to respond to a business opportunity that may not have been able to exploited without the application of e-learning.
Learning technologies are not only about the transmission of information (and knowledge) between a training department and their employees. They can also have a role to play in facilitating the development of a common culture and strengthening bonds between work groups.
Instances of success (or failure) can be shared between fellow employees. Details of new initiatives or projects may be disseminated through a learning system. The contents of the learning system, as a result, may gradually change as a result of such discussions.
The wider cultural perspectives that surround the application of learning technologies, in my humble opinion, are a lot harder to quantify. It's hard to put a value on the use a technology to share information (and learning experiences) with your co-workers.
A quick search takes me to the wikipedia definition of ROI and I'm immediately overwhelmed by detail that leaves my head spinning.
Further probing reveals a blog entitled ROI and Metrics in eLearning by Tony Karrer who kindly provides a wealth of links (some of which were updated in April 2008). I have also uncovered a report entitled What Return on Investment does e-Learning Provide? (dated July 2005) (pdf) prepared by SkillSoft found on a site called e-Learning Centre.
The issue of e-learning and learning technology return on investment is one that appears to be, at a cursory glance, one that can be quite difficult to understand thoroughly. Attaching numbers to the costs and benefits of any learning technology is something that is difficult to well or precisely. This can be partly attributed to the nature of software: often, so much of the costs (whether it be in terms of administration or maintenance) or benefits (ability to find things out quicker and collaborate with new people) can be hidden amongst detail that needs to be made clearly explicit to be successfully understood.
When it comes to designing, building and deploying learning technology systems, the idea of return on investment is undoubtedly a useful concept, but those investing in systems should consider issues beyond the immediately discoverable costs and benefits since there are likely to be others lurking in the background just waiting to be discovered.
Acknowledgements: Image licensed under creative commons, liberated via Flickr from Mr Squee. Piano busker in front of building site represents two types of investments: a long term investment (as represented by the hidden building site), and a short term investment (using the decrepit piano to busk). The unheard music represents hidden complexities.