Guidelines for Evaluating Training Programs

An advantage of the retrospective pre-test model is that it can assist in addressing some of the gaps exposed by the lack of range-of-transfer of learning metrics provided by a Level 2 analysis of learners: in effect - to borrow a phrase from Jens Bjornavold – “make learning visible” (2004, p.64). This is essential, because it helps measure learning. In Chapter 5 of Evaluating Training Programs (2006) Kirkpatrick and Kirkpatrick stress the significance of measuring learning, because “no change in behaviour can be expected unless one or more of the learning objectives have been accomplished” (p.42). They set out “helpful” guidelines for the measurement of learning:

Table 1 Guidelines for evaluating learning

Guidelines for evaluating learning

Use a control group if practical

Evaluate knowledge skills and/or attitudes both before and after the program

Use a paper-and-pencil test to measure knowledge and attitudes

Use a performance test to measure skills

Get a 100 per cent response

Use the results of the evaluation to take appropriate action

References:

Bjornavold, J. (2004) Making Learning Visible: Validation of Formal, Non-Formal and Informal Learning: policy and practices in EU Member States1. European Journal of Education [Internet] 39 (1), 69–89. Available from: http://www.acc.eu.org/uploads/
Makinglearningvisible_1.pdf [Accessed 21st February 2007]

Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

--

Assessing learning programs: advantages and disadvantages of the pre-test design

When choosing to integrate the retrospective pre-test design method into your learning program evaluation and measurement methodology, the selection must take place in the after very carefully evaluating arguments both for and against its utility. The method has many advantages for the practitioner-researcher (see Table 1) but one must be cognisant of the threats to validity. However, in the context of the case study method, triangulation of data from other sources (questionnaires, surveys, quantitative analysis of access logs, LMS records, summative assessments and so on) can be used to counteract or balance such threats.

Table 1 Advantages and disadvantages of the retrospective pre-test model

Advantages	Disadvantages
Simple and cost-effective	Possibility that results were due to history in the job or organisation
Reduces costs and time for data collection and analysis	Possible distortions in retrospective reports because of response shift bias
Gathers data as part of the learning intervention
Compares post-intervention data with retrospective pre-data
Avoids attrition from the sample being measured
Decreases likelihood of testing effects

--