Evaluating online learning
by Clive Shepherd
At times as trainers we behave like ostriches. We'd rather not know how effective our efforts have been, so we bury our heads in the sand. We know that the evaluation of training, like motherhood and apple pie, is inherently a good thing. But, because short term priorities tend to crowd out their longer term competitors, it's typically something we plan to do better next year - after all, we've got away with it so far, so another year won't hurt!
In this article, we examine just why it doesn't pay to be an ostrich. Why, if online learning is going to make a real impact in your organisation, you'd better have better reasons to justify the investment than "well, everyone else is doing it". Evaluation takes a bit of work, but then nothing was every achieved without a little effort. Here's how it's done.
Evaluation - why bother?
What are we measuring?
Measuring reactions
Measuring learning
Measuring application
Measuring business results
Measuring return on investment
Measuring before and after
Evaluation - why bother?
There are many good reasons for measuring the success of your online learning programme. Here's four:
To validate training as a business tool
Training is one of many actions that an organisation can take to improve its performance and profitability. Only if training is properly evaluated can it be compared against these other methods and can you expect it, therefore, to be selected either in preference to or in combination with other methods.
To justify the costs incurred in training
We all know that when money is tight, training budgets are amongst the first to be sacrificed. Only by thorough, quantitative analysis can training departments make the case necessary to resist these cuts.
To help improve the design of training
Training programmes should be continuously improved to provide better value and increased benefits for an organisation. Without formal evaluation, the basis for changes can only be subjective.
To help in selecting training methods
These days there are many alternative approaches available to training departments, including a variety of classroom, on-job and self-study methods. Using comparative evaluation techniques, organisations can make rational decisions about the methods to employ.
What are we measuring?
With evaluation, as in life, there are many ways of measuring success. When you set your life goals - for health, wealth, fame or happiness (and all those unprintable ones that don't quite fit into this list) - you also determine your criteria for success. And so with evaluation. What you measure is a factor of your original goals.
The form of evaluation that we undertake is determined by the criteria that we choose, or are told to use, to measure success. The following four levels of evaluation were defined by Kirkpatrick back in 1975:
Level 1: Reactions
Reactions are what you measure with the happy sheet. Reactions are important because, if learners react negatively to your courses, they are less likely to transfer what they learned to their work and more likely to give bad reports to their peers, leading in turn to lower numbers.
Level 2: Learning
Learning, in terms of new or improved skills, knowledge and attitudes, is the primary aim of a training event. Learning can be measured objectively using a test or exam or some form of assessed exercise. If a learner has to achieve a certain level of learning to obtain a pass mark, then the number of passes may be used as an evaluation measure. Another important aspect of learning is the degree of retention how much of the learning has stuck after the course is over.
Level 3: Application
If a learner has learned something from a course, you hope that this will be reflected in their behaviour on the job. If a learner employs what they have learned appropriately, then their work behaviour will meet desired criteria. Behaviour can be measured through observation or, in some cases, through some automated means. To assess behaviour change requires that the measurements are taken before and after the training.
Level 4: Business results
If, as a result of training, learners are using appropriate behaviours on the job, then you would expect that to have a positive impact on performance. A wide variety of indicators can be employed to measure the impact of training on performance numbers of complaints, sales made, output per hour and so on. It is hard to be sure that it is training that has made the difference without making comparisons to a control group a group of employees who have not been through the training.
Recently, attention has focused on the financial implications of training and, consequently, a fifth level has been defined (Phillips, 1997):
Level 5: Return on investment
Return on investment (ROI) is a measure of the monetary benefits obtained by an organisation over a specified time period in return for a given investment in a training programme. Looking at it another way, ROI is the extent to which the benefits (outputs) of training exceed the costs (inputs).
But leaving aside the five levels of Kirkpatrick and Phillips, in the real world we are measured in other ways as well:
Numbers
One way of measuring the success of training is the good old bums on seats. Although by no means a true measure of the effectiveness of training, learner numbers do reflect the fact that the training is addressing a need and that the design and methodology is meeting expectations.
Direct cost
Direct costs are those costs that are incurred directly as a result of a training programme external design and development, consultancy fees, travel expenses and so on. If the programme did not take place, these costs would not be incurred. Many organisations only ever take direct costs into consideration when measuring training costs.
Indirect cost
Indirect costs are costs that may or may not be directly associated with a training event, but which would have been incurred anyway, whether or not the training took place. Examples are salaries of in-house tutors and learners and the costs of rooms and equipment. Any analysis of the true costs of training will include both direct and indirect costs.
Efficiency
Efficiency is a measure of the amount of learning achieved relative to the amount of effort put in. In practical terms this means the amount of time it takes to complete a piece of training. Efficiency has a direct relation to cost the more efficient a training method is, the less it will cost.
Performance to schedule
Sometimes with a training programme, time is of the essence the training needs to be completed by a given date if a particular business objective is to be achieved. In these situations, the extent to which a training programme performs to schedule is a critical measure of success.
Income received
If you are a training provider operating externally to a client organisation, then income received is a vital measure of your success. Its the financial equivalent of bums on seats the more courses you run or places you fill, the greater the benefit. Some internal training providers may also cross-charge their clients, although, because this correspondingly increases the cost to the organisation, this would not be regarded as a benefit when assessing return on investment.
The extent to which learners mix
A justification often made for training, particularly group events, is that it provides an opportunity for learners who work in different departments or regions to interact with each other, share experiences and make contacts. Because this is a valued outcome of training, it needs to be considered when comparing training methods. Similarly, some training may be regarded as a perk, a benefit of some value, even if this is not directly related to learning.
Measuring reactions
Reactions are the first level of measurement in Kirkpatrick's four-step approach. Although sometimes disregarded, reactions are important because, if students react negatively to your courses, they are less likely to transfer what they learned to their work and more likely to give bad reports to their peers, leading in turn to lower student numbers.
Here are probably the most common methods of measuring reactions:
Questionnaire
The reactions questionnaire, often called a 'happy sheet' because of the way it captures end-of -course euphoria, is almost ubiquitous with classroom courses. Of course, there's no reason why it has to be given out at the end of the course - a more considered opinion may be obtainable some days or even weeks after the course has finished.
Observation
A good trainer or tutor will be able to detect the reactions of learners by observing their behaviour and their comments. This method can generate useful feedback, but not in an objective or structured fashion.
Meetings
Another way to look at reactions is to hold a group meeting with all the learners on the course. This way issues that are brought up can be debated and suggestions found for making improvements in the future.
Interviews
It would also be possible to measure reactions through one-to-one interviews with learners. The best results will be obtained by structured interviews that work methodically through the key issues.
The online environment also provides many opportunities for measuring reactions:
Chat
Chat programs are the equivalent of the group discussion in a classroom. They can be used to gain feedback from a group of learners or for a one-to-one.
Email
Email provides the asynchronous alternative to chat. Learners can use email to submit their comments directly to the tutor or to debate issues in a discussion forum.
Online questionnaires
Web page forms can be used in much the same way as paper happy sheets. In fact, they have quite an advantage, in that the results can be automatically stored in a file on the web server, for analysis by a spreadsheet or database program. With a little programming, you can even have the analysis performed automatically.
Measuring learning
Learning is the primary objective of any training event and, as such, must be measured if you are to know whether the event has been successful. The techniques that you use to measure learning depend to a large extent on the type of learning that the course has been designed to achieve. One way of looking at types of learning is in terms of the three learning domains.
Cognitive
The cognitive domain includes all knowledge and those skills that require thought rather than practical action. There are many ways of testing for cognitive learning:
Affective
The affective domain is concerned with attitudes and emotions. This is a harder area to test for learning, but there are some options:
Psychomotor
The psychomotor domain covers all practical skills. There's usually only one way to test for practical skills and that's to have them performed, although in certain circumstances they can be simulated (many pilots only ever get to use a flight simulator before flying a new plane for real).
Many, but certainly not all types of learning can be assessed online:
Cognitive
Online methods include:
Affective
In this domain, online methods include:
Psychomotor
The psychomotor domain is not well suited to online measurement, unless the subject happens to be use of the mouse or keyboard.
Measuring application
Although the primary purpose of training is to bring about learning, not much money would be made available for training if that was as far as it went. Sponsors of training need to know that what is learned as a result of training will be applied back on the job with some effect, hopefully positive. In fact, as the sponsors, they have a right to know this.
At level 3, we are measuring the extent to which new knowledge, skills and attitudes are translated into new job behaviour - in other words, the extent to which learning is being applied.
Unfortunately, there are many reasons why learning does not get applied:
If any of these are the case, then you need to know about it. You may be able to revise the course or introduce new follow-up procedures that will help cure the problem.
So how can we measure the extent to which learning is applied?
Observation
Tutors, coaches or the learner's supervisor can watch out for the way in which learning is being applied. They can also provide positive reinforcement where the results are successful and constructive feedback and encouragement where they are not.
Questionnaires
A more structured approach is to have the learner's peers, subordinates or supervisor complete a questionnaire, listing the desired behaviours and asking for feedback on the extent to which they are in evidence.
Automatically
In some job positions, where the learner is working with electronic equipment, it may be possible for evidence of behaviour to be gathered automatically.
Self-reporting
Learners will know better than anyone whether they are applying their new learning. You may ask learners to keep counts of what they do, complete checklists or questionnaires.
Measuring business results
The real benefits of training can not be measured in terms of learner reactions, nor the amount of learning that has been achieved; not even the extent to which behaviour may have changed. The real benefits come from improved performance traditionally the hardest training outcome to forecast or measure.
So, what do when the going gets tough? Back away and focus our evaluation efforts on easier measures? No, we do the very best we can, because all other measures fail to reflect the hard reality that training must pay off in measurable business results.
If it is any comfort, trainers are not alone in finding it difficult to calculate the benefits of what they do. Is it any easier to measure the benefits obtained from launching a new product, running an advertising campaign, initiating a research programme or changing the pay and benefits policy?
Let's look at some of the major categories of benefits:
Labour savings
Labour savings occur where, as a result of the training, less effort is needed to achieve current levels of output. We have to assume that savings are realised by a reduction in the amount of labour applied to a particular job, not by utilising the newly available time to achieve further output on the same job.
Labour savings will only be realised if the labour applied to a job can really be reduced, whether this comes as a result of transfers of staff to new positions, re-allocations of work or redundancies. If the time savings simply result in more slack, then there is no saving.
Examples of labour savings include:
Productivity increases
Productivity increases occur where, as a result of training, additional output can be achieved with the same level of effort. This implies that the organisation requires or desires more output in the area in question. If it does not, then it might be better to express the benefit as a cost saving.
Examples of productivity increases include:
Other cost savings
Cost savings can be achieved in a variety of ways, not just through savings in labour. Examples include:
Obtaining measures
Level 4 measures are the key performance indicators of the organisation and, as such, you will probably find that the relevant data is already available. However, you may need to work hard to gain access to it in the form you need.
Measuring return on investment
At level 5 we measure the financial impact of the business results realised at level 4. The most popular way of expressing this impact is as a return on investment (ROI).
ROI is calculated as follows:
% ROI = benefits x 100
costs
ROI relates to a specified period of time, typically a year or two years. First you measure all of the costs associated with the particular training programme over this period:
Then you measure the financial benefits obtained over the same period:
Then you can calculate the ROI.
Let's imagine you have been running an online management training programme and want to calculate the ROI over the first year. You measure the costs as $100,000 and the benefits as $130,000. Your ROI is 130,000 / 100,000 x 100, or 130%.
Measuring before and after
To measure the effect of training, you need to take each indicator in turn and generate before and after data what the level was before the training and what it was afterwards. The difference between the two provides an indication of the effect of the training.
For example, sales in a certain region may be $10m a month leading up to a training programme and $12m a month after. The 20% increase in productivity could be attributed to the training and entered into the ROI analysis.
Control groups
It is not always so clear cut that training has been the cause of a change in a performance indicator. Say that a number of measures were taken simultaneously to cure a problem. How do you know which of the measures was responsible for any positive results? Similarly, at the same time as a training programme is run, there may be a completely unexpected downturn in a market, meaning that performance actually goes down after the training. How can you separate the effect of the training from the effect of the downturn?
In these situations, the interaction between these variables can be significantly reduced by the use of a control group. A control group is a subset of the target population that does not receive the training. By comparing the results of those who have received the training with those of the control group, you can separate the effect of the training on performance.
For example, here are some statistics for a particular performance indicator:
|
Before |
After |
Change |
|
| Group receiving training |
10 |
16 |
+60% |
| Control group |
10 |
12 |
+20% |
| Effect of training |
+40% |
It can be seen from this table that performance has risen by 20% even without training. The increase in performance that can be attributed to the training is therefore 40%.
Where next?
Sometimes it seems that evaluation is becoming a show of machismo. "What level is your company up to? Oh dear, you mean you're still using happy sheets? We've just graduated to level 5 - soon we'll be measuring the effects of our training on the global economy."
It's true to say that evaluation does get trickier the more you move up the levels. It's also true to say that financial impact is an increasingly important issue. Remember that all levels of evaluation are important for different reasons, but that the only ones that count are those where you can produce valid, reliable results. Get evaluating - you may even find that your online learning's been a success.
| © 1999 Fastrak Consulting Ltd |
All rights reserved |