Unhappy Sheets, Part One

“Happy Sheets” have been given a violent battering over the years by otherwise peaceful L&D folk.  Otherwise known as Level 1 reaction sheets on the Kirkpatrick evaluation model, they were recently nominated by learning professionals at #LearningLive to go into “Room 101”.

I understand they were saved, not by a last-minute burst of pity, but by an even greater collective loathing for words such as “millenials”, “passionate” and “blended”. Fair enough.

The Room 101 nominators were probably thinking more of Paul Merton’s lever than George Orwell’s 1984, but let’s start from the thought that handing out evaluation forms is like a living nightmare in which rats are eating our face.

Why do we dislike Level 1 so much?

I suspect there is a logical reason, and a psychological reason.  This post deals with the logical reason.

This is a defence of Happy Sheets, but a very specific and partial defence.

I completely agree with the criticism that Level 1 sheets tell us little or nothing about what people have learnt.  Donald Clark expresses it brilliantly in this general demolition of Kirkpatrick.  “Learners can be happy and stupid.. Learners often learn.. through experiences which, although difficult at the time, prove to be useful later.  Happy sheet data.. is often a skewed sample from those that have pens.”

Agreed.  Move straight to Level 4.  Do Not Pass Go.  Even better, find another model.

But there are also two genuine, real-life benefits of Level 1:

  • the collection of raw data on the customer experience
  • powerful live feedback for trainers and facilitators.

I saw both of these at the National School of Government, where I went from a complete novice to a fairly experienced trainer and facilitator over six years.  The National School, for all its many flaws, had a strong culture of collecting and analysing data to support an operation involving hundreds of trainers and a £30m annual turnover.

The raw data was used in various ways.  It revealed variation in performance among trainers who were delivering the same course, supposedly to the same standard.  You got to spot who the weak trainers were.  (Or maybe they were trainers who were giving their participants a deliberately disruptive experience, enhancing their eventual learning at the expense of short-term popularity – but – no.. – just weak trainers, probably).

Some scores exposed flaws in design.  Falling scores revealed complacency.  Free text comments revealed some trends. Scores even revealed whether the food in the canteen was improving or deteriorating.

The overall figures meant nothing (we always scored over 90% on the key questions), and no learning should have been inferred – but the variations, the sudden dips and the overall trends did have value.  That is the partial, “raw data” defence.

More importantly, though, the forms gave instant feedback to trainers and facilitators.  I improved hugely through reading the evaluation forms.

It was a standing joke that the trainers would stand around nonchalantly at the end of the day, chatting and smiling as participants gathered their bags, put on their coats and left the Happy Sheets on a table – but as soon as the door closed on the last wheeled cabin-bag, we would dash for the feedback.

The next two minutes were quiet, with occasional grunts of acknowledgement, and the odd hiss of “bastard”.  It was an acknowledged “end of course syndrome” that you would travel home brooding on the one negative comment and ignoring the ten good ones.

Some comments were hurtful or unfair, and many were bland and useless, but I found some were gold dust.  Most importantly, I began to see how my own preferences influenced my design, and clouded my judgement.

That is the “Level 1 as Continuing Professional Development for Designers and Deliverers” defence.

So this dislike of Happy Sheets – the splash of blood on the yellow smile – I see where it comes from, on the rational side. We should all be frustrated at organisations which equate mechanistic scoring with actual learning.  That means we should find better evaluation systems, but not, I think, that we should dump immediate customer feedback – it’s too valuable for that.

watchmen

image credit: DC Comics

Advertisements

4 thoughts on “Unhappy Sheets, Part One

  1. This is a brilliant debate! I worked for an organisation not that long ago, who forbid trainers to look at the feedback until they were out of the building and in the car park. Here’s why. Huge importance was placed on the overall course satisfaction rating by this organisation. If a trainer dipped below 98% it was deemed a disaster. Consequently, before the ‘wait till you’re in the car park’ rule, one had trainer pinned a delegate (customer) to the wall and demanded to know why they hadn’t given him more than 95%. The temperamental trainer was later fired (for his temper not his results) and made available to the competition. Having said all that and highlighted how not to use happy sheets, there was a massive amount of valuable stuff to support the customer experience.

  2. It’s interesting to hear you describe the use of data on a large scale as well as the (slavish?) desire from trainers for that near spontaneous feedback. It makes me think of what it would be like if we reviewed Tripadvisor style just as we paid the bill at the restaurant as well as later on when we could be bothered to post a more balanced and reflective view that added to a larger data set. Which would the waiter prefer? Which would the restaurant owner prefer? What would the client like most? Would either truly assure quality going forwards?

    1. Good points! Maybe not slavish, but it’s like having an exam result in an envelope, difficult to resist the temptation to open it. One issue is that memory deteriorates so quickly- hard to get reflective feedback the next week.. – but neither a guarantee of quality anyway

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s