E-Learning Challenge #83 Reaction GIFS

This week the E-Learning Challenge on the Articulate Community is on reaction GIFs. https://community.articulate.com/articles/reaction-gifs-elearning

Here are mine,

1. I can’t get variables to do what I want.

2. I get a complicated variable to work on the first try.

3. When people start insisting that learning styles are valid and must be used.

Posted in Uncategorized

Safety and Sherpas

This post is on a slightly off topic but in some ways does relate to health and safety. On Friday, April 18th, thirteen Sherpas died in an avalanche on Mount Everest. It was the deadliest incident to happen on Everest. They died trying to earn a living. These people died while helping tourists play at climbing. The paying guests are not climbers, you are only climbing when you are on the sharp end of the rope. Sherpas route find and set up the ropes that the guests follow and use as protection on their trek to the summit. Climbing Everest is an accomplishment but it is an accomplishment of endurance and money. No longer is Everest a climbing accomplishment. 


If there was an industrial incident in Canada that killed thirteen workers the uproar would be huge, and rightly so. I don’t know the answer to the issue of climbing Everest with Sherpa support. Sherpas rely on the money from expeditions. Climbing guides earn a good living and the loss of that income would devastate Sherpa communities. But the death of these people also will devastate their communities. One thing I do know is that no job is worth dying for.

Tagged with: ,
Posted in Uncategorized

You Won’t Believe How Many Options You Should Have!

The most common way of testing knowledge, or learning, in an elearning course is the multiple choice question (MCQ). There is debate on how effective this is and the relationship it has to on-the-job behaviour. For now I want to look at how many options a MCQ should have. Typically a MCQ follows the form of a stem and four, sometimes five, options, one of which is right and the others are incorrect and called distractors. Something like this:

What is the most common number of options in a multiple choice question?

  1. 2
  2. 3
  3. 4
  4. 5

Popularity doesn’t mean that this is the correct way to do something (see http://rationalwiki.org/wiki/Argumentum_ad_populum). The paper, “Three Options Are Optimal for Multiple-Choice Items: A Meta-Analysis of 80 years of Research” by Michael C. Rodriguez that  there are good reasons to have three options in MCQs.

The arguments boil down to: 

  • That less time is needed to create questions if less distractors are needed as one of the most difficult parts of creating a question is writing plausible distractors. It easy to write distractors that are not plausible. (Poor quality distractors are chosen less than 5% of the time.)
  • It takes less time to read and answer three-option MCQs as compared to four or five-option MCQs. This means that the learners can answer more questions in a shorter time.
  • By increasing the number of high quality test question that can be completed in a period of time test validity and reliability is increased.

One concern expressed that reducing the number of options will increases the number of correct answers by random guessing. This is really only a concern for the least able students who are stressed for time and just guess for the last questions. The reality is that because of the difficulty in creating high quality distractors it usually easy to quickly eliminate one, or two, distractors and bring the number of possible options down to three. So little difference is seen when the least probable distractors are eliminated from five-option MCQs to four or three-option MCQs.

Be brave, the next time you are writing test questions use only three options. You will save time, you will save your learner’s time, and you will still have a valid test.

 Rodriguez, M. C. 2005. Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24 (2), pp. 3–13

Tagged with: , ,
Posted in Uncategorized

Burn Your Smiley Sheets

Smiley sheets, or reaction sheets, or level 1 assessments are the surveys filled out at the end of a course to gauge learners’ reactions to a course. There is limited evidence to support the use of reaction sheets. Depending on the goal of the course, reaction sheets may or may not be a good idea. 

Evaluating a course can be done in four ways according Donald Kirkpatrick in “Evaluating Training Programs”[1], reaction, learning, behaviour, results. Reaction sheets measure customer satisfaction of the course. Learning evaluation measures the knowledge, skills, or attitudes the learner has at the end of the course. Behaviour evaluation measures the transfer of knowledge from the course to application on the job. Evaluating results measures the effect the course has had on the bottom line of the company. According to Alliger et.al. 90% of courses are evaluated for reaction, 33% for learning, 10% for behaviour and almost nobody evaluates for results.[2] The most important of these evaluations must be behaviour, do learners apply what they learn in a course to the job. Does measuring the reaction to a course help us gauge the ability to apply the knowledge, skills, or attitudes to the job; what circumstances should we be using reaction sheets; do learners reactions matter at all; what do we, as instructional designers do with reaction sheets?

 Typically, the questions on reaction sheets can be broken down into affective questions, utility questions, and combined questions. Affective questions focus on the attitude, or feelings, that the learner has about the course. Examples of affective questions include: The facilities were comfortable, how do you rate the instructor, the material was presented in an interesting way, the breaks were spaced right, the temperature was comfortable. Utility questions ask will the course improve on-the-job performance such as: I was given enough time to practice the skills, the material was relevant to my job, I will be able to apply what I learned today. Combined questions combine both affective and utility questions together.

 Does measuring reaction help us gauge learning or behaviour change? The evidence on this is mixed. For courses that try and impart skill or knowledge tests to measure learning should be used. There is a very poor correlation between reactions and learning or behaviour. Courses that focus on affective change, such as ethics and diversity training, reactions are more important in these courses reaction sheets may be more useful. Reactions are highly influenced by factors such as whether the instructor moves around, and is friendly. The subject matter may also influence whether people like the course. Are the learners being forced to take the course? Are the people having to learn a new task that is going to increase their workload? Are the learners going to face that they have been doing something dangerous and have not realized, and instead of accepting change they may reject the information (the famous, “I’ve been doing this for years and never been hurt”). Because reactions are so highly variable and subjective great care must be taken when interpreting the results.

 What do we, as instructional designers, do with the reaction sheets. All too often, reaction sheets are used to stroke the ego of the instructors and training departments, rarely are the comments reviewed and suggested changes made. In my years as a trainer, the most common comments, about the courses our department ran were, “Doughnuts”and “Give us lunch”. The department didn’t have a budget for meals, nor were the employee’s departments willing to pay for meals. Some of the comments were for changes that were not practical or wouldn’t have met with legislation or due diligence issues. It is not just my courses, I took a Transporting Dangerous Goods Train-the-Trainer course, and I filled out the reaction sheet, with specific advice on how to improve the course. Three years later I took the course again and nothing (not a single word on the PowerPoint, which the instructor read out every one of those words at us) had changed. The strangest experience was a Rigging and Load Calculation course I took last summer. The course had some serious problems in design and execution. I gave my honest feedback. The owner of the company phoned me up two days later, and complained that I had been unfair to the course and the course had never had such a bad review. He then phoned my boss and complained to him. His conversation with me included him swearing at and insulting me. If you are going to ask for reactions, you have an unspoken agreement to respect the person’s opinions, and when possible re-work the courses so they can be improved, of course we should reviewing and improving courses on a regular basis anyway.

 Seriously, consider if a reaction sheet is necessary or even advisable. If time and budget are tight, focus on what counts, the behaviour after the course. If people aren’t changing their behaviour then you need to look at why, and what can be done about it.  If you are going to ask for reactions then respect them and try and learn from them.

[1] Evaluating training programs In-text: (Kirkpatrick, 1998) Bibliography: Kirkpatrick, D. L. 1998. Evaluating training programs. San Francisco, Calif.: Berrett-Koehler

[2] Alliger, G. M., Tannenbaum, S. I., Bennett, W., Traver, H. and Shotl. A meta-analysis of the relations among training criteria In-text: (Alliger and Tannenbaum et al., 1997, pp. 341–358) Bibliography: Alliger, G. M., Tannenbaum, S. I., Bennett, W., Traver, H. and Shotl. 1997. A meta-analysis of the relations among training criteria. Personnel psychology, 50 (2), pp. 341–358.


Posted in Uncategorized

How To Get The Most From A Course

Learning is part of everyone’s job. Whether it is learning how to: operate a new piece of machinery, send an email on your new phone, transport dangerous goods, or keep safe on the job. Sometimes courses are required by legislation such as apprenticeships and fall protection. Sometimes courses are required by companies to ensure safety of employees or increase productivity. Whatever the reasons there are steps we, as learners, can take to make sure that that we can remember and apply what we learn in the course.

Before the course starts, think about what you would like to gain from this session. Meet with your supervisor and find out what they expect you to learn. What skills or knowledge do you want to have when the course is over? Think about specific questions that you can ask the instructor.

During the course, there are many things you can do to maximize your learning. Come into the course with a positive attitude. Take the training seriously, and keep your mind on the course (this is not the time to be answering emails).  Write notes, there is evidence that writing notes strengthens the learning process and that typing may actually impair learning.[1] Link the material from what you know already to how you can apply it on the job.

One effective way to take notes is to use mind maps. Mind maps help you to organize ideas or facts in a visual way.


After the course take a few minutes and reflect on what you have learned and how it connects to your work. Write down specific actions you can take to apply what you learned to your job. Are there any specific triggers that you need to pay attention to; such as taking wind speed or temperature into account? Take note of what obstacles might prevent you from using your training and come up with a plan to get around those obstacles. Meet with your supervisor and put together a plan to apply your training. Finally, use your new knowledge and skills as soon as possible. The single biggest factor that will make your determine the success of your training is using it. Use it or lose it.

[1] The University of Stavanger. (2011, January 24). Better learning through handwriting. ScienceDaily. Retrieved February 25, 2014 from www.sciencedaily.com/releases/2011/01/110119095458.htm

Tagged with: , ,
Posted in Training

Protecting Your Training Investment

If you are a supervisor or manager you’ve probably had a time when you sent someone off to get trained they come back and they do their job but nothing seems to have changed in their performance. You may blame the employee, or the trainer, or the course, but what could you have done to support a change in the employee. After all, one of the responsibilities of a manager is to coach and train their employees.

When an employee is trained we expect a change. The change could be in knowledge, skills, or attitude, but it should be observable. To integrate new knowledge, skills or attitudes to a job the person must have knowledge, skills, or attitude; self-confidence; an opportunity to perform; and a supportive environment. During the training the employee should have learned the new knowledge, skills, or attitude, and the self-confidence to use them. The opportunity to perform and a supportive environment come from the supervisor or manager.

Training employees is expensive, but not training employees is more expensive. So what can be done to protect the investment you have made into your employee. There are actions that supervisors can take before, during, and after training to enhance the transfer of training to the workplace.

Before training you can talk to the trainers about the performance improvement you expect of the employee. In some cases, for instance safety training, it is advisable to attend the training; this way you will be able understand and correct safety deficiencies. Talk to your employees before the training about the goal of the training and what is expected of them when they return.

During training, do not contact your employees about work related matters. Do not call them away to perform job tasks, Make sure somebody is filling in for that employee or reduce the work load for the team. Training must be considered part of the employees job. Employees must understand the importance of training and know that their workload won’t pile up while they are attending training.

Interventions after training are the most important actions a supervisor can implement. Meet with your employees after the training, discuss the learning experience and set goals to improve performance. During the conversation ask if there is anything the you can do to help them improve performance. Look anything preventing employees  from using their new skills; do they need equipment, software, changes in company procedures? Give employees opportunities to practice their training, give assignments that relate to their training. The phrase “use it or lose it” applies to training. Finally, recognize the employee when you see them put the training into practice.

1What Every Manager Should Know About Training, Robert F. Mager, The Center for Effective Performance, Inc., 1992

2 Enhance the Transfer of Training, Dennis E. Coates, ASTD Press

Tagged with: , , , ,
Posted in Training, Uncategorized