Burn Your Smiley Sheets

Smiley sheets, or reaction sheets, or level 1 assessments are the surveys filled out at the end of a course to gauge learners’ reactions to a course. There is limited evidence to support the use of reaction sheets. Depending on the goal of the course, reaction sheets may or may not be a good idea. 

Evaluating a course can be done in four ways according Donald Kirkpatrick in “Evaluating Training Programs”[1], reaction, learning, behaviour, results. Reaction sheets measure customer satisfaction of the course. Learning evaluation measures the knowledge, skills, or attitudes the learner has at the end of the course. Behaviour evaluation measures the transfer of knowledge from the course to application on the job. Evaluating results measures the effect the course has had on the bottom line of the company. According to Alliger et.al. 90% of courses are evaluated for reaction, 33% for learning, 10% for behaviour and almost nobody evaluates for results.[2] The most important of these evaluations must be behaviour, do learners apply what they learn in a course to the job. Does measuring the reaction to a course help us gauge the ability to apply the knowledge, skills, or attitudes to the job; what circumstances should we be using reaction sheets; do learners reactions matter at all; what do we, as instructional designers do with reaction sheets?

 Typically, the questions on reaction sheets can be broken down into affective questions, utility questions, and combined questions. Affective questions focus on the attitude, or feelings, that the learner has about the course. Examples of affective questions include: The facilities were comfortable, how do you rate the instructor, the material was presented in an interesting way, the breaks were spaced right, the temperature was comfortable. Utility questions ask will the course improve on-the-job performance such as: I was given enough time to practice the skills, the material was relevant to my job, I will be able to apply what I learned today. Combined questions combine both affective and utility questions together.

 Does measuring reaction help us gauge learning or behaviour change? The evidence on this is mixed. For courses that try and impart skill or knowledge tests to measure learning should be used. There is a very poor correlation between reactions and learning or behaviour. Courses that focus on affective change, such as ethics and diversity training, reactions are more important in these courses reaction sheets may be more useful. Reactions are highly influenced by factors such as whether the instructor moves around, and is friendly. The subject matter may also influence whether people like the course. Are the learners being forced to take the course? Are the people having to learn a new task that is going to increase their workload? Are the learners going to face that they have been doing something dangerous and have not realized, and instead of accepting change they may reject the information (the famous, “I’ve been doing this for years and never been hurt”). Because reactions are so highly variable and subjective great care must be taken when interpreting the results.

 What do we, as instructional designers, do with the reaction sheets. All too often, reaction sheets are used to stroke the ego of the instructors and training departments, rarely are the comments reviewed and suggested changes made. In my years as a trainer, the most common comments, about the courses our department ran were, “Doughnuts”and “Give us lunch”. The department didn’t have a budget for meals, nor were the employee’s departments willing to pay for meals. Some of the comments were for changes that were not practical or wouldn’t have met with legislation or due diligence issues. It is not just my courses, I took a Transporting Dangerous Goods Train-the-Trainer course, and I filled out the reaction sheet, with specific advice on how to improve the course. Three years later I took the course again and nothing (not a single word on the PowerPoint, which the instructor read out every one of those words at us) had changed. The strangest experience was a Rigging and Load Calculation course I took last summer. The course had some serious problems in design and execution. I gave my honest feedback. The owner of the company phoned me up two days later, and complained that I had been unfair to the course and the course had never had such a bad review. He then phoned my boss and complained to him. His conversation with me included him swearing at and insulting me. If you are going to ask for reactions, you have an unspoken agreement to respect the person’s opinions, and when possible re-work the courses so they can be improved, of course we should reviewing and improving courses on a regular basis anyway.

 Seriously, consider if a reaction sheet is necessary or even advisable. If time and budget are tight, focus on what counts, the behaviour after the course. If people aren’t changing their behaviour then you need to look at why, and what can be done about it.  If you are going to ask for reactions then respect them and try and learn from them.

[1] Evaluating training programs In-text: (Kirkpatrick, 1998) Bibliography: Kirkpatrick, D. L. 1998. Evaluating training programs. San Francisco, Calif.: Berrett-Koehler

[2] Alliger, G. M., Tannenbaum, S. I., Bennett, W., Traver, H. and Shotl. A meta-analysis of the relations among training criteria In-text: (Alliger and Tannenbaum et al., 1997, pp. 341–358) Bibliography: Alliger, G. M., Tannenbaum, S. I., Bennett, W., Traver, H. and Shotl. 1997. A meta-analysis of the relations among training criteria. Personnel psychology, 50 (2), pp. 341–358.

 

Advertisements

I am a instructional designer and developer specializing in Health and Safety Training, and Technical Training on a variety of subjects.

Posted in Uncategorized

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: