Still more on evaluating non-formal learning...

Well now. Back from NY and totally energised by my trip to Manhattan. Where were we...? Oh yes -

The methodology I'll be describing is a generalised account of the type work I carry out when developing and implementing non-formal learning initiatives in organisations - I invite you to argue, discuss, praise etc. as appropriate.

Typically, I consult for small- to mid-sized enterprises in Ireland. The type of learning interventions being discussed here are what I call 'Information Sessions.' Information Sessions are learning events designed to disseminate a broad range of knowledge and information about an organisation and its products. The title 'Information Session' is carefully chosen: these events are not formal training (see this blog entry), nor are they informal learning; they are informational by definition and design, though they are developed according to a strong instructional design process. The Info Session events are hosted as synchronous, SME-led presentations. The content is repurposed post-event for asynchronous distribution on-demand over the Web, as well as being distributed to learners on DVD-ROM. Information Sessions are distributed in the following formats:

Table 1 Information Sessions format

Synchronous Asynchronous Live audience in situ at Information Session event location (i.e. learners' office or other facility) On-demand post-live event delivery over the Web Streamed live over the web DVDs available to learners. Each DVD contains one series of Information Session presentations

In the first phase of data collection, quantitative data are collected on every attendee participating in both the live in situ and the synchronous and asynchronous distribution of the events. An E-learning Readiness Survey in the second phase of research is distributed to internal experts who fall under the aegis of what William Horton (2006, p.107) describes as “those who should know” - managers, learning and knowledge professionals, and key stakeholders in the organisation. A representative sample (proportionate to staff numbers) of employees participate in Phase III of the research design. Informed consent is obtained from each contributor by asking them to complete a sign-off statement form in the questionnaire (we'll get back to the questionnaire) and asking their assent to use their responses before follow-up interviews are conducted.

Note: As the research design follows a case study approach and is based on a learning initiative in a single organisation, I am cognisant of the fact that the potential exists for “individuals’ interests to be affected” (Robson, 2002, p.501) by participating in the research, particularly if their opinion deviates from the corporate policy.

References:

Horton, W. (2006) So how is elearning different? IN: Kirkpatrick, P. & Kirkpatrick, J. Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Robson, C. (2002) Real World Research. 2nd ed. Oxford: Blackwell Publishing.

--

More on evaluating NFL using Kirkpatrick's Four-level model

I'm going to begin this post by mentioning Donald Clark's view on Donald Kirkpatrick's model - the post has the tongue-in-cheek title Donald Talks Bollocks, and it gives an alternative view on the four levels; have a read, it's good stuff.

However, as Mr. Clark points out, rejecting Kirkpatrick doesn't provide a answer to how else we should evaluate the effectiveness of learning (particularly non-formal learning) so in lieu of another approach I will carry on discussing how I think Kirkpatrick can be used to assess the effectiveness of NFL initiatives.

Pay attention, here comes the science bit...

I've already discussed why I think Kirkpatrick's guidelines for evaluating learning is incompatible with the assessment of non-formal learning (particularly in the context of NFL), and today I'll discuss how you can begin to counter these deficiencies. Like Steven Kerr, CLO at Goldman Sachs, Kirkpatrick Level 3 is the most interesting to me; in the context of NFL, Level 2 cannot be used as an instrument for evaluation: with no summative assessment at the end of an NFL session, you cannot evaluate knowledge transfer by traditional testing methods.

To understand why NFL cannot be assessed at Level 2, look at what Michael Eraut says in Non-formal learning, implicit learning and tacit knowledge (2000). Eraut makes a clear distinction between his interpretation of the term ‘non-formal learning’ and what others including Scribner (1988), Conner (2002) and Cross (2003) would call ‘informal learning’ – what Eraut terms ‘incidental learning’ or ‘the acquisition of knowledge independently of conscious attempts to learn and the absence of explicit knowledge about what was learned’ (Reber, 1993, quoted by Eraut 2000, p.12)

This is, he argues, because most workplace learning takes place outside formal learning contexts, and informal learning carries with it connotations of

so many other features of a situation, such as dress, discourse, behaviour, diminution of social differences – that its colloquial application as a descriptor of learning contexts may have little to do with learning per se.

(2000, p.12).

Not only does the term carry unwanted and confusing implications, but it is too vague to be of any real utility. For Eraut, an analysis of learning must focus on activities and the outcomes that that contribute to significant changes in capability or understanding. In a sense, Eraut does not define non-formal learning; rather, he defines the characteristics of formal learning (p.12) as:

1. A prescribed learning framework
2. An organised learning event or package
3. The presence of a designated teacher or trainer
4. The award of a qualification or credit
5. The external specification of outcomes.

The implication of this categorisation is that any learning that does not exhibit all of these characteristics should be classed as either informal or non-formal learning. Some reviewers (Colley, Hodkinson & Malcolm, 2002) make the point that Eraut does not make clear what the status is of learning in situations that meet some, but not all, of his ‘formal’ criteria. My interpretation of his characterisation is that the very nature of a formal activity - “following or according with established form, custom, or rule” (Merriam-Webster Online, 2007) validates Eraut’s description.

I am particularly interested in exploring the relationship between Level 1 (Reaction) and Level 3 (Behaviour); as Kirkpatrick & Kirkpatrick (2006, p.22) assert “learning has taken place …when one or more of the following occurs: Attitudes (sic) are changed. Knowledge is increased. Skill is improved. One or more of these changes must take place if a change in behaviour is to occur.” In this context, I believe a mixed method of quantitative data-gathering and a more naturalistic or qualitative information collection approaches can be employed to capture data that will assist in evaluating the success of a non-formal learning initiative.

And it is this approach that I will be discussing tomorrow.

References:

Colley, H. Hodkinson, P. Malcolm, J. (2002) non-formal learning: mapping the conceptual terrain. a consultation report [Internet] Available from: http://www.infed.org/archives/e-texts/colley_informal_learning.htm [Accessed 28th January 2008]

Conner, M. (2002) Informal Learning: more than “What we do between classes”. [Internet] Available from: http://www.productivity-solutions.net/documents/
informallearning-mconner-071202.pdf [Accessed 19th October, 2006]

Cross, J. (2004) An informal history of eLearning. On the Horizon [Internet] 12(3). pp.103-110. Available from: http://www.emeraldinsight.com/Insight/viewPDF.jsp?
Filename=html/Output/Published/EmeraldFullTextArticle/
Pdf/2740120301.pdf [Accessed 20th February, 2007]

Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Scribner, S. (1988) Mental and manual work: An activity theory orientation. In E. Tobah, R. J. Falmagne, M. B. Parlee, L. M. Martin & A. S. Kapelman (Eds.), Mind and Social Practice: Selected Writings of Sylvia Scribner. (pp 367-374).Cambridge: CUP

Whitney, K. (2004) Steve Kerr: Managing the Business of Learning [Internet] Available from: http://www.clomedia.com/content/
templates/clo_cloprofile.asp?articleid=582&zoneid=4 [Accessed 28th February 2008]

--

Evaluating non-formal learning in the context of the Four-Level Model

Donald L Kirkpatrick first published his ideas on evaluating learning in 1959 in a series of articles in the US Training and Development Journal. The articles were subsequently included in his book Evaluating Training Programs (originally published in 1975; I have the 2006 edition).

In this text he outlined and further developed his theories on evaluating culminating in the Four-Level Model, arguably the most widely used and popular approach for the evaluation of training and learning. Kirkpatrick's Four-level model is now considered an industry standard across the HR and training communities (see Table 1.1).

Table 1.1 Kirkpatrick's Four-level Model

Kirkpatrick’s Model	Learning Effect
Level 1: Reactions.	Evaluate participants' satisfaction with the learning intervention.
Level 2: Learning and Level 3: Behaviour.	What do participants know they didn't know before? How are they using knowledge in their jobs? What is the learning and performance effect of the intervention?
Level 4: Organisation-level benefits.	Has the development of higher levels of domain knowledge improved organisational productivity?

According to the model, evaluation should always begin with Level One, and as time and budget allows, should move sequentially through Levels Two, Three, and Four. Information from each prior level serves as a base for the next level's evaluation. Each successive level represents a more precise measure of the effectiveness of the training program, however, each level also requires a more rigorous and time-consuming analysis. As we shall see, the characteristics of NFL mean that not all of the four levels can be applied to the evaluating of this approach to learning.

In Chapter 5 of Evaluating Training Programs (2006) Kirkpatrick and Kirkpatrick stress the significance of measuring learning, because “no change in behaviour can be expected unless one or more of the learning objectives have been accomplished” (p.42). They set out “helpful” guidelines for the measurement of learning:

Table 1.2 Guidelines for evaluating learning

Guidelines for evaluating learning

Use a control group if practical

Evaluate knowledge skills and/or attitudes both before and after the program

Use a paper-and-pencil test to measure knowledge and attitudes

Use a performance test to measure skills

Get a 100 per cent response

Use the results of the evaluation to take appropriate action

This approach is incompatible with the assessment of non-formal learning (particularly in the context of e-learning) in a number of ways:

1. The singular nature of each non-formal learning event means that no control group can exist to measure a difference between it and the “experimental group” (p.43).
2. Similarly; a significant number of the learners that access NFL do so asynchronously, via a number of different learning channels and over a broad time span in a relatively ad hoc manner. This “just enough, just in time” aspect of non-formal learning is one of its strengths, but it was unrealistic to attempt to measure learning using an experimental method in such an environment.
   
3. The lack of summative assessment in non-formal learning precludes both pencil-and-paper and performance testing to measure learning.
   
4. The distributed nature of access to these NFL interventions, over both time and location makes getting a 100 per cent learner response rate practically impossible.

I'll be discussing approaches to overcoming these challenges in my next posting.


References:

Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

--

Measuring the learning effect of non-formal learning programs

I have suggested in a previous blog entry that non-formal learning (NFL) is located on a continuum between so-called informal and formal learning (see Figure 1)

Figure 1 the Learning Continuum

and that learning is situated in a larger socio-cultural context - a cognitive web - that provides the knowledge-centric framework within which learners operate (see Figure 1.2).


Figure 1.2 Knowledge-centric View of Knowledge (after Marc J. Rosenberg)

Similarly, NFL shares characteristics with both informal and formal learning, but is differentiated enough from both of these approaches to merit it's own category (see Table 1)

Table 1 Definition of learning types

Learning Type	Description
Formal Learning	Learning typically provided by an education or training institution, structured (in terms of learning objectives, learning time or learning support) and leading to certification. Formal learning is intentional from the learner’s perspective [my italics].
Non-formal Learning	Learning that is not provided by an education or training institution and typically does not lead to formalised certification. It is, however, structured (in terms of learning objectives, learning time or learning support). Non-formal learning is intentional from the learner’s perspective [my italics].
Informal Learning	Learning resulting from daily life activities related to work, family or leisure. It is not structured (in terms of learning objectives, learning time or learning support) and typically does not lead to certification. Informal learning may be intentional but in most cases it is non-intentional (or “incidental”/ random) [my italics].

Given the non-formal nature of the learning (and by extension a paucity of the type of assessment metrics available when evaluating the efficacy of formal training programmes) I am going to explore the challenge of assessing non-formal learning in the context of Kirkpatrick's 4-level model, particularly between Level 1 (Reaction) and Level 3 (Behaviour); as Kirkpatrick & Kirkpatrick (2006, p.22) assert “learning has taken place …when one or more of the following occurs: Attitudes (sic) are changed. Knowledge is increased. Skill is improved. One or more of these changes must take place if a change in behaviour is to occur.”

More tomorrow...


References:

Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Rogers, A. (2004) Looking again at non-formal and informal education - towards a new paradigm [Internet] Available from: http://www.infed.org/biblio/non_formal_paradigm.htm [Accessed 30th January 2008]

Rosenberg, M. J. (2001) e-Learning: Strategies for Delivering Knowledge in the Digital Age London: McGraw-Hill.


See Also:

Holford, J. Patulny, R. & Sturgis, P. (2006) Indicators of Non-formal & Informal Educational Contributions to Active Citizenship [Internet] Available from: http://crell.jrc.ec.europa.eu/active_citizenship.htm [Accessed 24th February 2008]

--