Learning Evaluation and Strategy: Using an e-learning readiness survey

During this series of posts on evaluating non-formal learning programs, I have mentioned carrying out an e-learning readiness survey without characterizing or discussing how to implement such a research instrument.

This was deliberate; in my view e-learning readiness surveys represent an alpha and an omega of evaluation: on one level they are the starting point for any evaluation of an organization's learning initiative, on another level they define an organization's ability to implement an effective e-learning strategy. As such, they are a bridge between the theory and the practice of implementing a learning program.

Readiness surveys enable the learning practitioner to understand and measure both the effectiveness of organizational learning and to identify the critical factors for success when developing learning programs.

I have found Marc J. Rosenberg's E-Learning Readiness Survey to be a very effective instrument to evaluate both the effectiveness of an organization's learning strategy, and one of the foundations of any serious evaluation of the effectiveness of learning programs.

The questions are grouped into seven areas of understanding:

1. business readiness
2. the changing nature of learning and e-learning
3. value of instructional and information design
4. change management
5. reinventing the training organization
6. the e-learning industry
7. your personal commitment

The questions provided in this survey represent some of the most important strategic issues organizations face when transitioning to e learning. Certainly there are additional questions and issues that deserve attention; add your own, organization-specific items as required.

Downloads:

Marc J. Rosenberg's E-learning Readiness Survey

References:

Rosenberg, M. J. (2006) Beyond e-Learning. San Francisco, CA: John Wiley & Sons, Inc.
--

Using quantitative data in learning assessments

To support the qualitative data captured using the retrospective pre-test method, I recommend that you gather significant amount of quantitative data associated with the learning initiative or program you're evaluating.


This can seem daunting and difficult to achieve, but there are a range of accessible sources of quantitative data that every learning professional can use.

Now read on...

Always collect Level 1 attendee feedback forms from classroom-based and synchronous online participants at the end of each individual event: over time you accumulate substantial amounts of valuable feedback about learners' reaction to the initiative. Use Web-hosted feedback forms to collect data from asynchronous participants (i.e. for online and DVD-ROM based content).

Other data can be collected automatically from systems including Web servers, learning management systems and learning content management systems; Horton (2006) describes these types of archive data as “meaningful statistics” (p.102): they record detailed information about what participants did while taking the learning event, particularly in the e-learning channels. You may extend this "meaningful statistics" category to include non-automated but standardised data recording processes such as recording learner attendance captured in the synchronous online and classroom-based context, and off-line learner activity such as DVD-ROM requests. By examining logs and reports from these systems and processes useful data can be collected on:

1. Frequency and pattern of accessing the course
2. Number of slides/pages or learning objects accessed
3. Duration of access to learning objects
4. Number of supporting collateral downloads
5. Feedback submitted
6. Participation in discussion and question & answer sessions
7. Rate if individual learner attendance or online access
8. Rate of attendance or online access by role (i.e. developer, support, business analyst etc)
9. Rate of group level (i.e. Sales & Marketing, Support, Manufacturing, Engineering etc) attendance or online access
10. Rate of attendance or online access by length of service in the organisation (i.e. new hire, with the company one year,1-2 years, 2-3 years, 3-5 years, +5 years etc)

More tomorrow.

References:

Horton, W. (2006) So how is elearning different? IN: Kirkpatrick, P. & Kirkpatrick, J. Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

--

More on Kirkpatrick's Guidelines don't work for Non-Formal Learning programs

In my last posting on this topic, we discussed Kirkpatrick's guidelines for evaluating training programs (see Table 1).

Table 1 Guidelines for evaluating learning

Guidelines for evaluating learning

Use a control group if practical

Evaluate knowledge skills and/or attitudes both before and after the program

Use a paper-and-pencil test to measure knowledge and attitudes

Use a performance test to measure skills

Get a 100 per cent response

Use the results of the evaluation to take appropriate action

In this post, I will outline the difficulties encountered in applying these guidelines to non-formal learning initiatives.

Now read on...

Kirkpatick's approach is incompatible with the assessment of non-formal learning in three ways: firstly, the singular nature of each non-formal learning event means that no control group can exist to measure a difference between a control group and the “experimental group” (p.43). Similarly; a significant number of the learners that access this kind of content do so asynchronously, via a number of different learning channels and over a broad time span in a relatively ad hoc manner. This “just enough, just in time” aspect of non-formal learning is one of its strengths, but it is unrealistic to attempt to measure learning using an experimental method in such an environment.

Secondly, the lack of summative assessment in non-formal learning precludes both pencil-and-paper and performance testing to measure learning. Finally, the distributed nature of access to these events, over both time and location makes achieving a 100 per cent learner response rate (expressed by the value R) impossible. As the content (C) is always available (D) the potential (+) always exists for the current number of learners (N₁) to increment upwards (N₂). A complete learner response rate will always be a hostage to N₂ in this context.

R = R+ N₂

where C (is a known value) and N₁ (is a known value) and D (is a known value) + N₂

It follows that you can never achieve a one hundred per cent response rate. Contrast this with formal training courses. They are, by their nature, finite: each course has a specified duration. Content is delivered to learners in an instructor-led environment in a sequence of modules over the duration of the course. Once the course instruction is complete, members of any given class are required to take a comprehensive summative assessment on the learning objectives outlined in the course, usually within a set time after they complete a course to achieve certification.

Thus R = 100

where C = 10 and N₁ = 12 and D = 10

As a result, a 100 per cent response rate to knowledge and skills testing is achieved.

However, I believe that it is possible to capture and use data with the retrospective pre-test method, and I'll be discussing an approach to undertaking this activity next time.

References:

Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

--

Guidelines for Evaluating Training Programs

An advantage of the retrospective pre-test model is that it can assist in addressing some of the gaps exposed by the lack of range-of-transfer of learning metrics provided by a Level 2 analysis of learners: in effect - to borrow a phrase from Jens Bjornavold – “make learning visible” (2004, p.64). This is essential, because it helps measure learning. In Chapter 5 of Evaluating Training Programs (2006) Kirkpatrick and Kirkpatrick stress the significance of measuring learning, because “no change in behaviour can be expected unless one or more of the learning objectives have been accomplished” (p.42). They set out “helpful” guidelines for the measurement of learning:

Table 1 Guidelines for evaluating learning

Guidelines for evaluating learning

Use a control group if practical

Evaluate knowledge skills and/or attitudes both before and after the program

Use a paper-and-pencil test to measure knowledge and attitudes

Use a performance test to measure skills

Get a 100 per cent response

Use the results of the evaluation to take appropriate action

References:

Bjornavold, J. (2004) Making Learning Visible: Validation of Formal, Non-Formal and Informal Learning: policy and practices in EU Member States1. European Journal of Education [Internet] 39 (1), 69–89. Available from: http://www.acc.eu.org/uploads/
Makinglearningvisible_1.pdf [Accessed 21st February 2007]

Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

--

Assessing learning programs: advantages and disadvantages of the pre-test design

When choosing to integrate the retrospective pre-test design method into your learning program evaluation and measurement methodology, the selection must take place in the after very carefully evaluating arguments both for and against its utility. The method has many advantages for the practitioner-researcher (see Table 1) but one must be cognisant of the threats to validity. However, in the context of the case study method, triangulation of data from other sources (questionnaires, surveys, quantitative analysis of access logs, LMS records, summative assessments and so on) can be used to counteract or balance such threats.

Table 1 Advantages and disadvantages of the retrospective pre-test model

Advantages	Disadvantages
Simple and cost-effective	Possibility that results were due to history in the job or organisation
Reduces costs and time for data collection and analysis	Possible distortions in retrospective reports because of response shift bias
Gathers data as part of the learning intervention
Compares post-intervention data with retrospective pre-data
Avoids attrition from the sample being measured
Decreases likelihood of testing effects

--

Evaluating learning initatives: the Retrospective Pretest Design

The Retrospective Pre-test method is an interesting if controversial data collection design. I will examine its advantages and negative aspects here, as well as an argument for including this strategy in in assessing organisational learning initiatives. The retrospective pre-test method (see Table 1 is an extension of the one-shot case study design where data are gathered from participants following the learning intervention (for example, a series of non-formal Information Sessions).

Table 1 Retrospective Pre-test Design

Retrospective Pre-test Design
Groups		Non-equivalent	Intervention	Observation
Locally-based		N	X	O1	O2
Remotely-based		N	X	O1	O2
<- Time ->

However, participants (N) report on their knowledge, understanding or skills (Observation O1) after the intervention (X), and then also reflect and answer (O2) what they believe their understanding or skill was before the intervention. Rockwell & Kohn (1989) applied this method to testing the effectiveness of achieving program outcomes when interventions such as training programs are implemented and concluded that “using a post-then-pre design [retrospective pre-test] to identify self-reported behavioural changes can provide substantial evidence for program impact”. The retrospective pre-test enables researchers to reduce the response shift bias - defined by Klatt & Taylor-Powell in their 2005 paper Synthesis of Literature relative to Retrospective Pretest Design as

the change in the participant’s metric for answering questions from the pre-test to the post-test due to a new understanding of a concept being taught

- because the participants are able to give pre-test responses which are based on a post-intervention frame of reference.

Using the retrospective pre-test, response shift bias can be reduced increasing the likelihood that the observable results are due to intended intervention effects (Pratt, McGuigan & Katzev, 2001). This proposition is complicated and not without its critics – Theodore Lamb (2005), an advocate of the retrospective pre-test method, describes it as “an imperfect but useful tool.” Robson (2002, p.139-141) expresses concerns that used purely as a quasi-experimental method, this strategy could lead to an internal validity threat through regression to the mean.

References:

Klatt, J. and Taylor-Powell, E. (2005) Synthesis of Literature relative to RetrospectivePretest Design. Presentation to the 2005 Joint CES/AEA Conference, Toronto [Internet] Available from: http://www.citra.org/Assets/documents/evaluation%20design.pdf [Accessed 5th August 2007]

Lamb, T. (2005) The Retrospective Pretest: An Imperfect but Useful Tool. Evaluation Exchange. [Internet] 11(2). Available from: http://www.gse.harvard.edu/hfrp/eval/issue30/spotlight.html [Accessed 21st March 2007]

Pratt, C.C., McGuigan, W.M., & Katzev, A.R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3). 341-349.

Robson, C. (2002) Real World Research. 2nd ed. Oxford: Blackwell Publishing.

Rockwell, S. K. & Kohn, H. (1989) Post-Then-Pre Evaluation. Journal of Extension. [Internet] 27(2). Available from: http://www.joe.org/joe/1989summer/a5.html [Accessed 21st March 2007]

--

Evaluating Non-formal e-learning initatives: Case Study Techniques

As I asserted in a previous post, case studies allow naturalistic methods of enquiry such as individual interviews to be carried out, side-by-side with quantitative data gathering from surveys, archival records, server access logs and so on.

Interestingly in discussing the flexibility of the case study design, Yin (1994, p.285) contends that in the future, researchers will focus not on the case study method, but rather on the specifi case study data collection techniques (see Table 1) and we will increasingly see these techniques used in other “non-case study” forms of research.

Table 1 Case Study Techniques

Case Study Techniques

The use of multiple sources of evidence, in a converging manner

The explicit specification and testing of hypotheses and rival hypotheses, especially in lieu of control or comparison groups

The dominance of deductive strategies, whereby research starts with theorizing

Program logic models as a standard way of initiating a program evaluation

Pattern-matching as a common strategy for data analysis

Portfolio analysis, using qualitative criteria to differentially weigh the outcomes from a project or the projects within a program

The use of replication logic, rather than aggregating data, when comparing the results from multiple sites or cases.

This approach of modularising (to coin a term) the methodologies used in a case study may potentially enable future researchers to generalise from results attained through a case study approach to a broader context: this concept is outside the scope of this blog, but is something in my view that would have benefits in future social sciences research and evaluation of learning initiatives.

References:

Yin, R. K. (1994). Discovering the future of the case study method in evaluation research. Evaluation Practice [Internet] 15. Available from: http://aje.sagepub.com/cgi/reprint/15/3/283 [Accessed 15th January 2008 subscription required]

--

Creating a Research Design to evaluate the effectiveness of e-learning projects

In a previous blog entry, I mentioned the importance of data triangulation, particularly if you use a case study approach when evaluating an e-learning initiative. Another strategy I recommend is to collect data over a number of phases, using a number of techniques.

Case studies involve in-depth, descriptive data collection and analysis of a case, or a number of related instances of the same type within the case. In particular, the case study design is useful when answering “how” and “why” questions, and in understanding the particulars, and diversity of the case (Yin, 1994). As Yin asserted in his 1993 paper Applications of Case Study Research

the case study is the method of choice when the phenomenon under study is not readily distinguishable from its context.

(p.3)

The case study design (see Table 1) suited the conditions of this research project very well as it allowed naturalistic methods of enquiry such as individual interviews to be carried out, side-by-side with quantitative data gathering from surveys and archival records.

Table 1 Features of a case study (Russ-Eft & Preskill, 2001, pp.173-174)

Advantages	Disadvantages
Provides descriptive data	Results may not lead to scientific generalizability
Does not require control of participants or setting	Researcher bias may interfere with validity of the findings
Reports include verbatim quotes	May take too long to conduct
Leads to a greater understanding of the context of the evaluand	May produce more data than can be analysed in an effective manner
Gather data using multiple methods
Provides data that are rich in examples
Captures what is important to the participants
Portrays the multiplicity of causes that are associated with various outcomes
Embraces diversity of perspectives and experiences of participants
Allows the researcher to collect information on outcomes not known prior to the learning and performance initiative

As Robson points out, the flexibility of the case study approach allows the design to “emerge” (2002, p.89) during data collection and analysis. Similarly, the case study facilitates the application of other research designs within the case study design framework, enabling the learning professional to capture data during different phases of a project.

References:

Robson, C. (2002) Real World Research. 2nd ed. Oxford: Blackwell Publishing.

Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.

Yin, R. K. (1993). Applications of Case Study Research. Newbury Park, CA. Sage.

Yin, R. K. (1994). Discovering the future of the case study method in evaluation research. Evaluation Practice [Internet] 15. Available from: http://aje.sagepub.com/cgi/reprint/15/3/283 [Accessed 15th January 2008 subscription required]

--

Evaluating e-learning programs: more on the Program Logic Model

In the context of evaluating e-learning programs, the logic model is useful for identifying elements of the programme that are most likely to yield useful evaluation data, and to identify an appropriate sequence for collecting data and measuring learning. In this way, the programme logic model can help you develop a research design, and provide guidance to relevant and appropriate data collection strategies within your case study. Figure 1 illustrates how I apply the Program Logic Model in my case studies.

Figure 1 Programme Logic Model applied to the Information Session non-formal learning initiative

_ _

Evaluating Non-formal e-learning initatives: the Program Logic Model

In undertaking a case study when evaluating any e-learning initiative, it is essential that the data collected in the research is relevant and appropriate to the task of evaluating the impact of the intervention on the audience.

According to Wright & Ross (2001), the Program Logic Model

offers a conceptual and practical framework for this exacting work. At its core it is a simplified picture of a programme, initiative or project. In much the same way that a software developer would employ a use case, the logic model shows the relationships among the resources that are invested, the intervention or the activities that take place, and the outcomes that result from the project. Essentially, it is a tool to guide critical thinking.

(p.153).

Figure 1 The Program Logic Model

This model great utility to the e-learning practitioner, specifically in orienting the research and ensuring their learning evaluation methodology aligns to the investigation being undertaken. As Cooksy, Gill and Kelly discuss in their article The Program Logic Model as an Integrative Framework for a Multimethod Evaluation, “program theory guides an evaluation by identifying key program elements and articulating how these elements are expected to relate to each other” (p.109).

As illustrated in Figure 1 above, there are five components to the Program Logic Model: each component has a range of characteristics associated with it. These are described by Paul McCawley (1997):

1. The situation statement provides an opportunity to communicate the relevance of the project. This establishes a baseline for comparison at the close of a project and provides a way to determine whether change has occurred. Describing who is affected by the initiative allows assessment of who has benefitted.
2. Inputs include those things that we invest in a programme or project such as knowledge, skills, or expertise. Describing the inputs needed for a project provides an opportunity to communicate the quality of the programme.
3. Outputs are “those things that we do” (p.4) - providing deliverables, products - and the people we reach - intended audience. Describing the outputs establishes linkages between the problem (situation) and the impact of the project (intended outcomes).
4. In terms of program or project outcomes, these can be short-term, intermediate-term, or long-term, Outcomes answer the question “What happened as a result of the program?”

• Short-term outcomes of educational programmes may include changes in:
• Awareness – customers recognise the problem or issue
• Knowledge – customers understand the causes and potential solutions
• Skills – customers possess the skills needed to resolve the situation
• Motivation – customers have the desire to effect change
• Intermediate-term outcomes include changes that follow the short-term outcomes, such as changes in:
• Practices used by participants
• Behaviours exhibited by people or organisations
• Long-term outcomes follow intermediate-term outcomes when changed behaviours result in changed conditions or processes.

References:

Cooksy, L. J. Gill, P. & Kelly. P. A. (2001) The Program Logic Model as an Integrative Framework for a Multimethod Evaluation. Evaluation and Program Planning [Internet] 24(2). Available from: http://www.hsrd.houston.med.va.gov/AdamKelly/Logic.html [Accessed 21st October, 2006]

McCawley, P. F. (1997) The Logic Model for Program Planning and Evaluation. University of Idaho Extension [Internet] Available from: http://www.uidaho.edu/extension/LogicModel.pdf [Accessed 18th January 2007]

Wright, L. & Ross, J. W. (2001) The Logic Model: An A-to-Z Guide to Training Development, Implementation, and Evaluation. Association of the American Public Human Services Association (APHSA). [Internet] Available from: http://calswec.berkeley.edu/CalSWEC/2001_08Ross.pdf [Accessed 10th January 2007]

--

Evaluating Non-formal Learning: Establishing validity in your research

As discussed in a previous post, to assist in understanding the “parallel criteria” (1989, p.233) relationship between these test of rigour and the scientific terms for validity, Guba and Lincoln provide the following table (1981, p.104):

Table 1 Relationship between parallel criteria

Aspect	Scientific Term	Naturalistic Term
Truth Value	Internal Validity	Credibility
Applicability	External Validity	Fittingness/transferability
Consistency	Reliability	Dependability/auditability
Neutrality	Objectivity	Confirmability

When considering appropriate research designs to evaluate the effectiveness of learning initiatives, it's important to consider the types of data that you have access to, particularly in the context of the issues evaluating a non-formal learning initiative at Kirkpatrick’s Levels 3 and 4 (Shelton. & Alliger, 1993, pp.43-46, Horton, 2006, p.109), I feel it is essential to ensure validity of the naturalistic information by capturing a range of supporting quantitative data.

To assist evaluators understand and provide a context for the analysis of the information collected in the final, qualitative phase of the study, Russ-Eft and Preskill (2001, pp.155-156) describe a number of useful techniques for establishing the validity of quantitative and qualitative data, among them accuracy checking in data recording and encoding, persistent participant observation and member checking.

As the resultant evaluation will use a mix of qualitative and quantitative methods as well as data collected from multiple sources, use I recommend that you use data triangulation used as a strategy to enhance the rigour and validity of you evaluation.

The term triangulation originated in cartography where two or more reference points are used to locate an exact position.

Knowing a single landmark only locates you somewhere along a line in a direction from the landmark, whereas with two landmarks you can take bearings in two directions and locate yourself at their intersection.

(Patton, 1990, p.187).

Denzin (1988) has identified four types of triangulation – data triangulation, methodological triangulation, investigator triangulation, and theory triangulation; the I use these three triangulation techniques were used to evaluate the effectiveness of non-formal learning initiatives:

Table 2 Denzin's (1988) Definition of triangulation methods

Triangulation method	Definition
Data triangulation	Collecting data from a variety of sources. For example, in evaluating the transfer of learning from a three-hour workshop, the evaluator collects information from the learners, their managers, and their peers. In this case, three different sources have been queried.
Methodological triangulation	Using more than one method to collect data. For example, we may interview 20 per cent of a department’s employees and survey the remaining 80 per cent. By using two methods, the weaknesses of one may be compensated fro by the other.
Theory triangulation	Using different theoretical perspectives to interpret the same data. By applying different theories to make sense of the data, it is possible to see how different assumptions and beliefs influence one’s interpretations. By making these explicit, stakeholders can see how their assumptions might influence various actions taken because of the findings.

References:

Denzin, N. K. (1988) The Research Act: A Theoretical Introduction to Sociological Methods. 3^rd Ed. Englewood Cliffs, NJ: Prentice-Hall.

Guba, E.G. & Lincoln, Y.S. (1981) Effective Evaluation. San Francisco: Jossey-Bass.

Horton, W. (2006) So how is elearning different? IN: Kirkpatrick, P. & Kirkpatrick, J. Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Patton, M. Q. (1990) Qualitative Evaluation and Research Methods. 2^nd Ed. Thousand Oaks, CA. Sage

Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.

Shelton, S. & Alliger, G. (1993) Who's Afraid of Level 4 Evaluation? A Practical Approach. Training & Development. [Internet] 41 (3). Available from: http://www.afc-ispi.org/Repository/
Approaches%20for%20Assessing%20Outcomes%20ROI.doc [Accessed 24th March 2007]

Evaluating Non-formal Learning: Using "tests of rigor" to validate findings

Without rigour, research is worthless, becomes fiction, and loses its utility. Hence, a great deal of attention is applied to reliability and validity in all research methods.

(Morse et al, 2002, p.1)

Morse and his colleagues develop their position by asserting that the challenges to rigour in qualitative inquiry paralleled the growth of statistical packages and the development of computing systems in quantitative research: without “the certainty of hard numbers and p values, qualitative inquiry expressed a crisis of confidence” (Morse et al, 2002, p.2).

A number of leading qualitative researchers (Leininger, 1994, Altheide & Johnson, 1998) argued that the two approaches are not compatible and that they should not be combined, due to the fundamental differences between the two methods in the context of the nature of knowledge, the relationship between researcher and the object of the research, and the means of generating data.

My personal view aligns with the criteria suggested by Guba & Lincoln (1981) and Yin (1994) to determine reliability, and validity, and thus ensuring rigour, in qualitative research. Reliability and validity relate to the degree of confidence we have that the data are representing the participants’ reality or “truth” (Russ-Eft & Preskill, 2001, p.153). How can we define the truth (or even the accuracy) of participants’ responses in the subjective domain of qualitative research? Guba and Lincoln (1981) propose that researchers use so-called “tests of rigour” to establish validity through the naturalistic concept of “trustworthiness,” containing four aspects: credibility, transferability, dependability, and confirmability, that align to parallel scientific terms associated with the quantitative method.

Table 1 Definition of naturalistic terms (Russ-Eft & Preskill, 2001, pp153-155)

Naturalistic Term	Definition
Credibility	The scientific paradigm asserts that there is one reality and that information is valid when all relevant variables can be controlled; a naturalistic paradigm assumes that multiple realities exist in the minds of individuals. Hence when using qualitative methods, the research seeks to establish the credibility of individuals’ responses. The study must be believable by providing a detailed depiction of the multiple perspectives that exist can enhance the data’s credibility. For example, learner satisfaction surveys, along with interviews with training managers and instructors would provide a more holistic picture of the learning experience.
Fittingness/transferability	How transferable the findings are to another setting is called generalisability in the empirical context. The goal of qualitative methods is to provide richly detailed description to help the reader relate certain findings to their own experience. We often think of these as “lessons learned.” For example, as a stakeholder reads an evaluation report, they realise that something very similar has occurred in their organisation and sees where the findings can be used. Although the entire set of findings may not be applicable in their context, some issues identified may have applicability in other contexts.
Dependability/auditability	In the scientific paradigm, the notion of consistency is called reliability where a study’s consistency, predictability or stability is measured. Since reliability is necessary for validity, it is critical that data of any kind be reliable. Instead of considering data unreliable if it is inconsistent, evaluators using qualitative methods look for reasons that cause the data to appear unstable (inconsistent). For example, an interviewee may give an opinion one day, and when asked again the following week might say something slightly different. What would be important to understand and capture are the reasons for this change in perception. Such inconsistencies may stem from respondent error, an increase in available information, or changes in the situation. An audit trail that includes collecting documents and interview notes and a daily journal of how things are proceeding can help to uncover some of the reasons for such inconsistencies.
Confirmability	Objectivity is often viewed as the goal of most evaluation and research studies. Evaluators and researchers who use qualitative methods don’t necessarily believe that true objectivity can ever be fully achieved, rather that it is impossible to completely separate the evaluator from the method. Instead of trying to ensure that the data are free from the evaluator’s biases, the goal is to determine the extent to which the data provide confirming evidence. “This means that data (constructions, assertions, facts and so on) can be tracked to their sources, and that the logic used to assemble the interpretations into structurally coherent and corroborating wholes is both explicit and implicit” (Guba & Lincoln, 1989, p.243). Establishing confirmability, like consistency, often takes the form of auditing.

References:

Altheide, D., & Johnson, J. M. C. (1998). Criteria for assessing interpretive validity in qualitative research. IN: Denzin N. K. & Lincoln Y. S. (Eds.), Collecting and interpreting qualitative materials. Thousand Oaks, CA: Sage.

Guba, E.G. & Lincoln, Y.S. (1981) Effective Evaluation. San Francisco: Jossey-Bass.

Leininger, M. (1994). Evaluation criteria and critique of qualitative research studies. In J. M. Morse (Ed.), Critical Issues in Qualitative Research Methods. Newbury Park, CA: Sage.

Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002). Verification strategies for establishing reliability and validity in qualitative research. [Internet] International Journal of Qualitative Methods 1 (2), Article 2. Available from: http://www.ualberta.ca/~ijqm/ [Accessed 14th March 2008]

Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.

Yin, R. K. (1994). Discovering the future of the case study method in evaluation research. Evaluation Practice [Internet] 15. Available from: http://aje.sagepub.com/cgi/reprint/15/3/283 [Accessed 15th January 2007]

--

Using a Case Study approach to evaluate Non-Formal Learning

I consider a case study methodology to be the most effective strategy to use when evaluating non-formal learning because this approach provides the means to develop a rich description of the non-formal learning (NFL) initiative.

This assists in providing a context, and helps define the NFL initiative's value to the learners and to the organisation. An added benefit of the case study approach is that it enables the learning practitioner to employ a number of different data collection methodologies: participant observation, archive data collection, surveys, and interviews to answer the questions outlined in Table 1. The main purpose of using a number of data collection methodologies was to capture a range of quantitative and qualitative data, and to establish a means of triangulating that data to enhance validity and “trustworthiness” (Russ-Eft & Preskill, 2001, p.153).

Table 1 Questions the case study approach attempts to answer

Quantitative Data	Qualitative Data
Who attends live “in situ” the learning event? (dept., role, skill level, location)	Description and chronology of a the learning event
Who views synchronous live stream of the learning event? (dept., role, skill level, location)	Transfer of new knowledge to the job (Kirkpatrick - Level 3)
Who views asynchronous on-demand version of the learning event? (dept., role, skill level, location)	Does the pedagogical approach used in the learning event fulfil participants’ needs?
Who requests asynchronous DVD version of the learning event? (dept., role, skill level, location)	Do non-formal learning event enhance individuals' and organisation performance? (Kirkpatrick - Level 3)
Participants satisfaction with the learning event? (Kirkpatrick - Level 1)	What is the Business Impact of the the learning event or series of events over time (Kirkpatrick - Level 4)
What do attendees know that they didn’t know before? (Kirkpatrick - Level 2)

References:

Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.

--