AT+D classic: how to start an objective evaluation of your training program

T+D, May, 2004 by Donald L. Kirkpatrick

This excerpt is part of a larger article with the same title that originally appeared in the May-June 1956 issue of the Journal of the American Society of Training Directors, a predecessor to T D. The article heralded Kirkpatrick's now classic four-level evaluation model.

Most training men agree that it is important to evaluate training programs. They also feel that the evaluation should be done by objective means. However, the typical training man uses evaluation sheets or comment sheets as the sole measure of the effectiveness of his programs. He realizes he should do more, but he just doesn't know how to begin an objective evaluation.

According the Raymond Katzell, a well-known authority in this field, the evaluation of a training program falls into a hierarchy of steps that can be briefly stated as follows:

Step One. To determine how the trainees feel about the program.

Step Two. To determine how much the trainees learn in the form of increased knowledge and understanding.

Step Three. To measure the changes in the on-the-job behavior of the trainees.

Step Four. To determine the effects of these behavioral changes on objective criteria such as production, turnover, absenteeism, and waste.

In climbing this ladder of evaluation, most trainers have completed the first step. Typically, the training director asks the trainees to fill out evaluation sheets at the end of the program. Questions that are asked most frequently are

* How do you rate the program?

* What subject did you like best?

* What subject did you like least?

* What did you learn that you can use on the job?

* What subjects would you like to have discussed at future programs?

Usually the trainees are not asked to sign their name for fear they will not give an honest reaction.

This kind of subjective evaluation is important. It gives a good indication of how the trainees reacted to the program. If they react favorably, the trainer can justifiably pat himself on the back and say, "I gave them a program they liked." But he can't rightfully claim that the training program accomplished the objective, unless his objective was to give them a program they liked.

The immediate objective of any training course can be stated in terms of the desired knowledge and understanding that the program is trying to impart to the trainees. It is this stage of evaluation that should be undertaken as the second step. It is much more difficult than step one and, therefore, is not undertaken by many trainers.

Among the possible methods for determining whether increased knowledge and understanding have taken place, the best one seems to be the "before and after" paper and pencil test. If the scores on the posttest are significantly higher than on the pretest, the course can be deemed effective.

In determining the effectiveness of the training, it is important to note that the paper and pencil test or inventory must cover the principles and facts that are discussed in the course. If the trainer can find a test that covers this material, he can use it. If he cannot find a suitable one, he must construct his own inventor. Some of the inventories that are available are: How Supervise? by File and Remmers; Supervisory Inventory by Wesley Osterberg; and the Supervisory Inventory on Human Relations constructed by this writer.

So far, then, it has been stated that a before and after test can be used to deter mine whether or not increased knowledge and understanding have taken place. Also, that the inventory should cover the course content. In order to determine whether or not an available test is suitable, a trainer must examine his course outline and list the principles and facts he is trying to teach. A comparison of test items with these objectives will reveal whether or not the test can be used. Because the construction of a test involves such factors as the choice of items, the wording of the items, the number and type of possible response, and the sequence of items, it is far better to use an available inventory if it covers most of the course content.

Having selected or constructed a test, the trainer should consider some "Do's" for administering it:

* Give the pretest at the start of the first class and the posttest at the close of the last session. This will minimize the influence of factors apart from the training course.

* Have the trainee sign both the pretest and posttest. Then, the increased knowledge and understanding can be computed for each individual.

In instructing the trainees before they take the pretest:

* Tell them it is a before and after procedure.

* Explain the purpose of the test.

* Encourage them to answer truthfully by assuring them that their scores will have no effect on their pay or status in the company.

* Tell them to answer every question even if they have to guess. (This will be taken into account in the statistical analysis of scores.)

* Encourage them to take their time in taking the test. This will help to motivate them to read each item carefully.

In analyzing the test results, there are two kinds of evaluations to be made: