Training Impact Evaluation (ROI)

This session was presented by Robert Brinkerhoff (professor emeritus at W. Michigan). Author of "Telling Training's Story". The main thrust is describing the "Success Case Method" (SCM) for ROI - a methodology based on the simplistic concept: how you would prove that one person has learned something of value from your course.

What criteria matter in learning evaluation? Surey says: credibility, validity, simplicity, getting results that are actionable (ie: what is the ROI of ROI? What are you going to do with the results besides giving yourself a pat on the back).

Training gets predictable results:
15% do not use any of the learning at all in their jobs afterward
70% try to use it, then for various reasons will go back to old ways
15% will use it and achieve concrete results

15% doesn't sound too bad but just think about the unrealized value.
When training does not work, what is the reason? It is seldom because the training or trainer has a problem (only 20% of the time). The reasons for failure are
1. preparation and readiness (40%): lack of senior management alignment, lack of preparation and focus, management alignment and commitment, etc.
2. application environment (40%): didn't get manager support, had no opportunity to try it out, lack of peer support, no incentive to use it, lack of feedback and coaching.

THEREFORE: we do not need to evaluate the training itself but how well is the organization using the training to get results!

Fundamental problem is that we spend good money to teach people skills then their managers don't let them use it. Performance management is really management development. Most managers think of supporting training as allowing their people to attend it. There is no serious thought that the person will come back and be expected to use it. What we really need is to get all managers in a course on how to get value from skills and knowledge. The major content of the course would be the statistics showing the value that can be attained from their employees if courses are successful (apparently people have done this in many companies: amex, pitneybowes, etc and acheived success).

How does SCM work for ROI? Essentially, you look for evidence that someone has used learning for the company's success and meeting goals. Steps:
1. identify the most and least successful trainees (via survey - I suppose surveymonkey would work for this).
2. follow-up in person! In-depth interviews with selected examples. How was training used. Why did it work? Why did it NOT work? Document what they used, what value they got. When it doesn't work, what value was lost to the organization. Understand what factors lead to people using or not using the skills from courses - then we will know how to change these factors!
3. Now we have credible and documented "stories" of sucess - hopefully! ("that could stand up in court") as well as knowledge of factors that drive impact. These stories can be related to senior management and finance people to show the value of the courses. More importantly, it shows the lost value to the organization - case for change in managers' attitudes (as well as possibly more funding for education).

Benefits of the SCM:
-"prove" that training really works (although remember that training alone does not do it)
- market training to additional audiences
- make a business case for performance management system
- educate managers about their role and impact
- improve learning application process
- build success stories into content

We need to build accountability into the process of education to make it effective. There is a cycle: effective manager support -> training that works -> concrete results that add value -> back to the manager support. At the center of it all is evaluation and measurement.
The cycle can work well or negatively because of lack of manager support. An impact booster session for managers is the key to changing and positively impacting the cycle.

My thoughts? Interesting concept. I can definitely see within GDC how managers may not have the right attitude. I think the chemistry culture is that a manager thinks they are supportive if they allow their people to just go to the course or conference but don't really care if they bring something back. There is probably also an ego factor if the trainee comes back and tries to change the way the research group thinks about problems. I like the SCM because I don't think any of us would really believe ROI numbers (eg: 200% ROI) even if we collected them. We should probably also collect real numbers but the SCM seems more credible and real than a bunch of numbers and perhaps we should consider following up and reading his book for more info.