Why measure learning?
The easy answer is to track learner progress. The subtler, more powerful answer is to drive learner behavior.
Dan Ariely, behavioral scientist and best-selling author of Predictably Irrational, discovered firsthand how much a measurement strategy can change how we act:
“At MIT, I was measured on my ability to handle my yearly teaching load, using a complex equation of teaching points. The rating, devised to track performance on a variety of dimensions, quickly became an end in itself. Even though I enjoyed teaching, I found myself spending less time with students because I could earn more points doing other things. I began to scrutinize opportunities according to how many points were at stake.”
Ariely came to regret this arrangement. The point system incentivized him to behave in ways that were not in the best interests of his students or even himself. Instead, he was driven to optimize his score.
If an expert on human behavior is powerless in the face of a point system, what chance do the rest of us stand?
What you measure is what you get
As human beings, we are compelled to adapt our behavior to the metrics we are held against.
When we measure learning by counting up attendance numbers, course completions, and hours of training delivered, what we get is attendance, completions, and training time. What we don’t get is effective learning or business impact.
For those readers who have been struggling to make their way up the Kirkpatrick scale or calculate ROI, let me quote management guru H. Thomas Johnson:
“Perhaps what you measure is what you get. More likely, what you measure is all you’ll get. What you don’t (or can’t) measure is lost.”
The implication is dire. If we don’t measure effective learning and business impact, it’s not just that we fail to report on those things. They are likely not happening at all.
Measurement as motivation
Your measurement strategy is an incentive strategy. If you give people points to learn, they’ll care more about the points than the learning . (This is incidentally a central danger of casually gamifying L&D.) If instead you challenge learners and give them the chance to be recognized for their real achievements, you have the basis of a successful measurement strategy.
Here are four measurement approaches, ordered from most to least formal, that create the right incentives for learning:
1) Behavioral surveys
What are they?
Behavioral surveys get right down to asking about what people actually do on the job, or at least believe they do. In these surveys, learners self-report how frequently they perform a set of behaviors.
With a frequency rating scale, you can avoid a lot of the fuzzy data that comes from asking about learners’ subjective feelings (e.g. “how much do you agree or disagree?” or “how satisfied are you?”). And you can make the frequencies more specific (e.g. “once per day” instead of “sometimes”) so that the distinctions between answer choices are clearer for the respondents.
How behavioral surveys help you measure
By surveying your learners about the frequency of their behaviors, you can set a behavioral baseline prior to a learning experience. Then after the experience, administer the same survey to show the behavioral impact of what people learned.
To show the durability of behavior change, survey learners weeks or months after the initial learning experience.
The key to effective behavioral surveys
Behavioral surveys are subject to various response biases that would lead a learner to either underreport or overreport the frequency of a particular behavior.
A way to un-bias results is to confirm them by surveying the learners’ peers, managers, or other colleagues as well. The knowledge that someone else is rating you makes you more likely to be honest, and any gaps in perception you uncover will be interesting to explore further.
You can also anonymize the data to reduce response bias. You’ll lose data at the individual level, but still collect it at the department or organization level to show change happening across your workforce.
For a more sophisticated, even less-biased survey alternative, see Will Thalheimer’s excellent Performance-Focused Smile Sheets.
What are they?
SMART goals, OKRs, KPIs, MBOs – these are a few of the acronyms we use today to set and measure business goals. When we’re trying to motivate learners, however, this goal-setting jargon often falls flat.
Enter milestones. A milestone is a significant event in someone’s development. Unlike traditional learning objectives, milestones describe achievements that by definition would be meaningful to the learner.
How milestones help you measure
Milestones are a way to track meaningful progress toward a business outcome. When a milestone is achieved, you know that change is happening and that learning is being transferred to the real world.
A good way to measure the progress of individual learners is to count up the number of milestones they are able to achieve in a quarter or in a year. Or, measuring the % of individuals who have achieved a specific milestone can tell you how prevalent a behavior is in an organization.
The key to effective milestones
Select actions that are easily observable to you and the learner. Whether a milestone has been achieved should be a clear yes/no. For example “delivering a powerful presentation” may be an admirable goal, but it is too subjective to be a good milestone. A better milestone would be “presenting to an audience of over 200 people” or “presenting from memory without looking down at notes.”
3) Performance rubrics
What are they?
Performance rubrics make behaviors more specific and visible by spelling out the component parts of a successful performance.
A rubric of productivity behaviors from Francis Wade
Putting rubrics into the hands of managers, or even learners themselves, makes it much easier to observe a behavior, identify gaps, and give consistent feedback.
How performance rubrics help you measure
When giving learners regular feedback is impractical, performance rubrics allow learners to measure their progress against concrete criteria. In my all-time favorite learning book Design for How People Learn, Julie Dirksen offers an example of a self-evaluation flow that allows learners to practice, then evaluate their own performance with a checklist, and finally hear from experts.
The key to effective performance rubrics
Whether you go with a simple checklist or a more sophisticated rubric that spells out multiple levels of quality for each component action, make sure that all criteria are clear to the people using it. Before widely distributing a rubric, observe a manager using it to give feedback to a learner, or observe a learner using the rubric to evaluate their own performance after the fact. A good rubric takes the guesswork out of measurement, so look for any signs of confusion, and tweak the rubric accordingly.
4) Role models
What are they?
Role models are people whose behavior learners can emulate. It needn’t be any more formal than that. Whereas mentorship programs tie a single individual to another and require significant investment from both parties, role modeling is a lot easier to implement and can have a much wider impact.
VP of People leading managers in a workshop on running effective performance reviews
For any learning initiative you want to measure, identify individuals who excel at the behaviors you want people to learn. These role models can tell you what gaps learners are exhibiting and how they are improving, or they can observe learners practicing and performing and give them feedback directly.
How role models help you measure
If you’re wondering what role models have to do with measurement, think of them as human measuring sticks. Not only can role models provide feedback to learners, learners can measure their progress by continually comparing their performance to that of the role model. It’s learning by aspiration.
The measurement here is more informal and less quantitative, but it is deeply meaningful. When offering critical feedback, role models are more likely to point out the correct flaw or offer the right corrective; and when things go well, their praise feels that much better to receive, and is a powerful incentive to continue learning.
The key to effective role models
To identify role models, focus on what people do, not on who they are. If someone in your organization excels at a target behavior, they can help you measure the performance of those learning it.
Get role models to share best practices and counterexamples with the team, and when they deliver these insights, make sure they provide all the specific, nitty gritty details that got them there.
The more examples of expert performance you can feature, the better learners will be able to model the correct behavior and also gauge their level of performance in comparison.
The final tally
So hit pause on the point systems. Squirrel your smile sheets away. Put that ROI calculator down for just a moment.
Behavioral surveys, milestones, performance rubrics, and role models allow you to both measure change and create the right internal incentives for learning. And beyond measurement, these approaches reinforce learning points and give learners the kind of feedback they need to course correct, or celebrate progress. These are all good results. So treat your measurement strategy as part of your learning strategy, not just as a way to evaluate that strategy.
Show me the…metrics
Comment below with an example of your measurement strategy. What have you done that has created the right incentives for learning?
If you haven’t been measuring your efforts, consider taking a microlearning approach. One of the advantages of designing microlearning experiences (i.e. short, focused bursts of learning) is that you can measure faster and more often. To learn how, download Grovo’s ebook on improving employee performance and driving business results with microlearning.