New York trip

Unfortunately no time to post on e-learning today: I'm off on my holidays for a week - going to New York for a week of rest and relaxation.

More on evaluating NFL using Kirkpatrick's Four-level model

I'm going to begin this post by mentioning Donald Clark's view on Donald Kirkpatrick's model - the post has the tongue-in-cheek title Donald Talks Bollocks, and it gives an alternative view on the four levels; have a read, it's good stuff.

However, as Mr. Clark points out, rejecting Kirkpatrick doesn't provide a answer to how else we should evaluate the effectiveness of learning (particularly non-formal learning) so in lieu of another approach I will carry on discussing how I think Kirkpatrick can be used to assess the effectiveness of NFL initiatives.

Pay attention, here comes the science bit...

I've already discussed why I think Kirkpatrick's guidelines for evaluating learning is incompatible with the assessment of non-formal learning (particularly in the context of NFL), and today I'll discuss how you can begin to counter these deficiencies. Like Steven Kerr, CLO at Goldman Sachs, Kirkpatrick Level 3 is the most interesting to me; in the context of NFL, Level 2 cannot be used as an instrument for evaluation: with no summative assessment at the end of an NFL session, you cannot evaluate knowledge transfer by traditional testing methods.

To understand why NFL cannot be assessed at Level 2, look at what Michael Eraut says in Non-formal learning, implicit learning and tacit knowledge (2000). Eraut makes a clear distinction between his interpretation of the term ‘non-formal learning’ and what others including Scribner (1988), Conner (2002) and Cross (2003) would call ‘informal learning’ – what Eraut terms ‘incidental learning’ or ‘the acquisition of knowledge independently of conscious attempts to learn and the absence of explicit knowledge about what was learned’ (Reber, 1993, quoted by Eraut 2000, p.12)

This is, he argues, because most workplace learning takes place outside formal learning contexts, and informal learning carries with it connotations of

so many other features of a situation, such as dress, discourse, behaviour, diminution of social differences – that its colloquial application as a descriptor of learning contexts may have little to do with learning per se.

(2000, p.12).

Not only does the term carry unwanted and confusing implications, but it is too vague to be of any real utility. For Eraut, an analysis of learning must focus on activities and the outcomes that that contribute to significant changes in capability or understanding. In a sense, Eraut does not define non-formal learning; rather, he defines the characteristics of formal learning (p.12) as:

1. A prescribed learning framework
2. An organised learning event or package
3. The presence of a designated teacher or trainer
4. The award of a qualification or credit
5. The external specification of outcomes.

The implication of this categorisation is that any learning that does not exhibit all of these characteristics should be classed as either informal or non-formal learning. Some reviewers (Colley, Hodkinson & Malcolm, 2002) make the point that Eraut does not make clear what the status is of learning in situations that meet some, but not all, of his ‘formal’ criteria. My interpretation of his characterisation is that the very nature of a formal activity - “following or according with established form, custom, or rule” (Merriam-Webster Online, 2007) validates Eraut’s description.

I am particularly interested in exploring the relationship between Level 1 (Reaction) and Level 3 (Behaviour); as Kirkpatrick & Kirkpatrick (2006, p.22) assert “learning has taken place …when one or more of the following occurs: Attitudes (sic) are changed. Knowledge is increased. Skill is improved. One or more of these changes must take place if a change in behaviour is to occur.” In this context, I believe a mixed method of quantitative data-gathering and a more naturalistic or qualitative information collection approaches can be employed to capture data that will assist in evaluating the success of a non-formal learning initiative.

And it is this approach that I will be discussing tomorrow.

References:

Colley, H. Hodkinson, P. Malcolm, J. (2002) non-formal learning: mapping the conceptual terrain. a consultation report [Internet] Available from: http://www.infed.org/archives/e-texts/colley_informal_learning.htm [Accessed 28th January 2008]

Conner, M. (2002) Informal Learning: more than “What we do between classes”. [Internet] Available from: http://www.productivity-solutions.net/documents/
informallearning-mconner-071202.pdf [Accessed 19th October, 2006]

Cross, J. (2004) An informal history of eLearning. On the Horizon [Internet] 12(3). pp.103-110. Available from: http://www.emeraldinsight.com/Insight/viewPDF.jsp?
Filename=html/Output/Published/EmeraldFullTextArticle/
Pdf/2740120301.pdf [Accessed 20th February, 2007]

Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Scribner, S. (1988) Mental and manual work: An activity theory orientation. In E. Tobah, R. J. Falmagne, M. B. Parlee, L. M. Martin & A. S. Kapelman (Eds.), Mind and Social Practice: Selected Writings of Sylvia Scribner. (pp 367-374).Cambridge: CUP

Whitney, K. (2004) Steve Kerr: Managing the Business of Learning [Internet] Available from: http://www.clomedia.com/content/
templates/clo_cloprofile.asp?articleid=582&zoneid=4 [Accessed 28th February 2008]

--

E-Learning and economic downturns...

I'm getting some interesting feedback from a variety of sources on my blog post Recession and the challenge to e-learning. Steve McKenzie over at Eduspaces has commented that he considers e-learning to be a "recession buster ...on balance costs for institutions and individuals can be saved."

I completely agree with Steve's assessment; the challenge as I see it, is for e-learning practitioners to address traditional/institutional ways of thinking in organizations about how to leverage the benefits of e-learning - break the habits of a lifetime in a sense.

Most decision makers (i.e. C-level executives) in organizations are from a generation that would not have extensively used e-learning during their formative years. In my experience, they become much more risk-averse during economic downturns and push back on what they see as innovation in tough times.

I'm reminded of a conversation I had with a VP about two years ago. I was working on a proposal to introduce an on-demand pre-classroom training module for a ILT course - a bit of blended learning to reduce the workload on instructors to ensure that learners were of an appropriate skill level to actually attend the course in question. His response to the proposal was along the lines of "it's all very well being leading edge, but we don't have to be bleeding edge."

Now, this was hardly an all-singing, all-dancing 3D PLE we were proposing here, just a little 30-minute Flash-based presentation. I think that the assertion this VP made represents that we, as learning professionals, have to be cognisant of more conservative attitudes that exist in the workplace, and we need to develop strategies to counter such lines of argument.

--

Evaluating non-formal learning in the context of the Four-Level Model

Donald L Kirkpatrick first published his ideas on evaluating learning in 1959 in a series of articles in the US Training and Development Journal. The articles were subsequently included in his book Evaluating Training Programs (originally published in 1975; I have the 2006 edition).

In this text he outlined and further developed his theories on evaluating culminating in the Four-Level Model, arguably the most widely used and popular approach for the evaluation of training and learning. Kirkpatrick's Four-level model is now considered an industry standard across the HR and training communities (see Table 1.1).

Table 1.1 Kirkpatrick's Four-level Model

Kirkpatrick’s Model	Learning Effect
Level 1: Reactions.	Evaluate participants' satisfaction with the learning intervention.
Level 2: Learning and Level 3: Behaviour.	What do participants know they didn't know before? How are they using knowledge in their jobs? What is the learning and performance effect of the intervention?
Level 4: Organisation-level benefits.	Has the development of higher levels of domain knowledge improved organisational productivity?

According to the model, evaluation should always begin with Level One, and as time and budget allows, should move sequentially through Levels Two, Three, and Four. Information from each prior level serves as a base for the next level's evaluation. Each successive level represents a more precise measure of the effectiveness of the training program, however, each level also requires a more rigorous and time-consuming analysis. As we shall see, the characteristics of NFL mean that not all of the four levels can be applied to the evaluating of this approach to learning.

In Chapter 5 of Evaluating Training Programs (2006) Kirkpatrick and Kirkpatrick stress the significance of measuring learning, because “no change in behaviour can be expected unless one or more of the learning objectives have been accomplished” (p.42). They set out “helpful” guidelines for the measurement of learning:

Table 1.2 Guidelines for evaluating learning

Guidelines for evaluating learning

Use a control group if practical

Evaluate knowledge skills and/or attitudes both before and after the program

Use a paper-and-pencil test to measure knowledge and attitudes

Use a performance test to measure skills

Get a 100 per cent response

Use the results of the evaluation to take appropriate action

This approach is incompatible with the assessment of non-formal learning (particularly in the context of e-learning) in a number of ways:

1. The singular nature of each non-formal learning event means that no control group can exist to measure a difference between it and the “experimental group” (p.43).
2. Similarly; a significant number of the learners that access NFL do so asynchronously, via a number of different learning channels and over a broad time span in a relatively ad hoc manner. This “just enough, just in time” aspect of non-formal learning is one of its strengths, but it was unrealistic to attempt to measure learning using an experimental method in such an environment.
   
3. The lack of summative assessment in non-formal learning precludes both pencil-and-paper and performance testing to measure learning.
   
4. The distributed nature of access to these NFL interventions, over both time and location makes getting a 100 per cent learner response rate practically impossible.

I'll be discussing approaches to overcoming these challenges in my next posting.


References:

Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

--