Guidelines for Evaluating Training Programs

An advantage of the retrospective pre-test model is that it can assist in addressing some of the gaps exposed by the lack of range-of-transfer of learning metrics provided by a Level 2 analysis of learners: in effect - to borrow a phrase from Jens Bjornavold – “make learning visible” (2004, p.64). This is essential, because it helps measure learning. In Chapter 5 of Evaluating Training Programs (2006) Kirkpatrick and Kirkpatrick stress the significance of measuring learning, because “no change in behaviour can be expected unless one or more of the learning objectives have been accomplished” (p.42). They set out “helpful” guidelines for the measurement of learning:

Table 1 Guidelines for evaluating learning

Guidelines for evaluating learning

Use a control group if practical

Evaluate knowledge skills and/or attitudes both before and after the program

Use a paper-and-pencil test to measure knowledge and attitudes

Use a performance test to measure skills

Get a 100 per cent response

Use the results of the evaluation to take appropriate action

References:

Bjornavold, J. (2004) Making Learning Visible: Validation of Formal, Non-Formal and Informal Learning: policy and practices in EU Member States1. European Journal of Education [Internet] 39 (1), 69–89. Available from: http://www.acc.eu.org/uploads/
Makinglearningvisible_1.pdf [Accessed 21st February 2007]

Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

--

The unbearable lightness of knowledge working

Here's an interesting meme that's been resonating around the blogosphere: according to Jared Sandberg at the WSJ -

In the Information Age, so much is worked on in a day at the office but so little gets done. In the past, people could see the fruits of their labor immediately: a chair made or a ball bearing produced. But it can be hard to find gratification from work that is largely invisible, or from delivering goods that are often metaphorical. You can't even leave your mark on a document in increasingly paperless offices. It can be even harder trying to measure it all.

In Ireland recently, a survey by the recruitment company Executive Connections suggested that most people in the job market are on the move not, as you might imagine for greater remuneration, but because of a perceived lack of career progression and satisfaction with their current role; people really do have an inherent need for responsibility and competency.

As learning practitioners - and particularly in e-learning - we're aware of the importance of motivating and rewarding learners. Many of us have first-hand experience of the numbingly dull exercise of proceeding through linear CBT-style courseware. We try to design, develop and deliver engaging and interactive learning experiences that will reward our learners, or at least evoke a degree of satisfaction when they accomplish a task or successfully complete an activity, and we also know about the statistics of the high drop-out rates for e-learning programmes.

I'm wondering if there's a relationship between these two phenomena? In my view, organisational performance is highly dependent on worker ability and motivation.

Ability depends on educational development, experience, and training; developing this facility is an ongoing process that doesn't reap immediate rewards. However, motivation can be improved quickly. As a guideline, there are broadly seven strategies for motivation.

• Positive reinforcement / high expectations
• Effective discipline and punishment
• Treating people fairly
• Satisfying employees needs
• Setting work related goals
• Restructuring jobs
• Base rewards on job performance

These are the basic strategies, though the mix varies from situation to situation/workplace to workplace. In the under- or unmotivated worker (including those who drop out of e-learning courses for example) we can say that there is a dissonance between an individuals actual state and some desired state: so how do we resolve this dissonance?

Answers on a postcard please.

--

References:

Sandberg, J. (2008) A Modern Conundrum: When Work's Invisible, So Are Its Satisfactions. The Wall Street Journal [Internet]. Available from: http://online.wsj.com/article/
SB120338000214975633.html?mod=psp_editors_picks
[Accessed 28th March 2008]

--

Assessing learning programs: advantages and disadvantages of the pre-test design

When choosing to integrate the retrospective pre-test design method into your learning program evaluation and measurement methodology, the selection must take place in the after very carefully evaluating arguments both for and against its utility. The method has many advantages for the practitioner-researcher (see Table 1) but one must be cognisant of the threats to validity. However, in the context of the case study method, triangulation of data from other sources (questionnaires, surveys, quantitative analysis of access logs, LMS records, summative assessments and so on) can be used to counteract or balance such threats.

Table 1 Advantages and disadvantages of the retrospective pre-test model

Advantages	Disadvantages
Simple and cost-effective	Possibility that results were due to history in the job or organisation
Reduces costs and time for data collection and analysis	Possible distortions in retrospective reports because of response shift bias
Gathers data as part of the learning intervention
Compares post-intervention data with retrospective pre-data
Avoids attrition from the sample being measured
Decreases likelihood of testing effects

--

Evaluating learning initatives: the Retrospective Pretest Design

The Retrospective Pre-test method is an interesting if controversial data collection design. I will examine its advantages and negative aspects here, as well as an argument for including this strategy in in assessing organisational learning initiatives. The retrospective pre-test method (see Table 1 is an extension of the one-shot case study design where data are gathered from participants following the learning intervention (for example, a series of non-formal Information Sessions).

Table 1 Retrospective Pre-test Design

Retrospective Pre-test Design
Groups		Non-equivalent	Intervention	Observation
Locally-based		N	X	O1	O2
Remotely-based		N	X	O1	O2
<- Time ->

However, participants (N) report on their knowledge, understanding or skills (Observation O1) after the intervention (X), and then also reflect and answer (O2) what they believe their understanding or skill was before the intervention. Rockwell & Kohn (1989) applied this method to testing the effectiveness of achieving program outcomes when interventions such as training programs are implemented and concluded that “using a post-then-pre design [retrospective pre-test] to identify self-reported behavioural changes can provide substantial evidence for program impact”. The retrospective pre-test enables researchers to reduce the response shift bias - defined by Klatt & Taylor-Powell in their 2005 paper Synthesis of Literature relative to Retrospective Pretest Design as

the change in the participant’s metric for answering questions from the pre-test to the post-test due to a new understanding of a concept being taught

- because the participants are able to give pre-test responses which are based on a post-intervention frame of reference.

Using the retrospective pre-test, response shift bias can be reduced increasing the likelihood that the observable results are due to intended intervention effects (Pratt, McGuigan & Katzev, 2001). This proposition is complicated and not without its critics – Theodore Lamb (2005), an advocate of the retrospective pre-test method, describes it as “an imperfect but useful tool.” Robson (2002, p.139-141) expresses concerns that used purely as a quasi-experimental method, this strategy could lead to an internal validity threat through regression to the mean.

References:

Klatt, J. and Taylor-Powell, E. (2005) Synthesis of Literature relative to RetrospectivePretest Design. Presentation to the 2005 Joint CES/AEA Conference, Toronto [Internet] Available from: http://www.citra.org/Assets/documents/evaluation%20design.pdf [Accessed 5th August 2007]

Lamb, T. (2005) The Retrospective Pretest: An Imperfect but Useful Tool. Evaluation Exchange. [Internet] 11(2). Available from: http://www.gse.harvard.edu/hfrp/eval/issue30/spotlight.html [Accessed 21st March 2007]

Pratt, C.C., McGuigan, W.M., & Katzev, A.R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3). 341-349.

Robson, C. (2002) Real World Research. 2nd ed. Oxford: Blackwell Publishing.

Rockwell, S. K. & Kohn, H. (1989) Post-Then-Pre Evaluation. Journal of Extension. [Internet] 27(2). Available from: http://www.joe.org/joe/1989summer/a5.html [Accessed 21st March 2007]

--