Assessing learning programs: advantages and disadvantages of the pre-test design

When choosing to integrate the retrospective pre-test design method into your learning program evaluation and measurement methodology, the selection must take place in the after very carefully evaluating arguments both for and against its utility. The method has many advantages for the practitioner-researcher (see Table 1) but one must be cognisant of the threats to validity. However, in the context of the case study method, triangulation of data from other sources (questionnaires, surveys, quantitative analysis of access logs, LMS records, summative assessments and so on) can be used to counteract or balance such threats.

Table 1 Advantages and disadvantages of the retrospective pre-test model

Advantages	Disadvantages
Simple and cost-effective	Possibility that results were due to history in the job or organisation
Reduces costs and time for data collection and analysis	Possible distortions in retrospective reports because of response shift bias
Gathers data as part of the learning intervention
Compares post-intervention data with retrospective pre-data
Avoids attrition from the sample being measured
Decreases likelihood of testing effects

--

Evaluating learning initatives: the Retrospective Pretest Design

The Retrospective Pre-test method is an interesting if controversial data collection design. I will examine its advantages and negative aspects here, as well as an argument for including this strategy in in assessing organisational learning initiatives. The retrospective pre-test method (see Table 1 is an extension of the one-shot case study design where data are gathered from participants following the learning intervention (for example, a series of non-formal Information Sessions).

Table 1 Retrospective Pre-test Design

Retrospective Pre-test Design
Groups		Non-equivalent	Intervention	Observation
Locally-based		N	X	O1	O2
Remotely-based		N	X	O1	O2
<- Time ->

However, participants (N) report on their knowledge, understanding or skills (Observation O1) after the intervention (X), and then also reflect and answer (O2) what they believe their understanding or skill was before the intervention. Rockwell & Kohn (1989) applied this method to testing the effectiveness of achieving program outcomes when interventions such as training programs are implemented and concluded that “using a post-then-pre design [retrospective pre-test] to identify self-reported behavioural changes can provide substantial evidence for program impact”. The retrospective pre-test enables researchers to reduce the response shift bias - defined by Klatt & Taylor-Powell in their 2005 paper Synthesis of Literature relative to Retrospective Pretest Design as

the change in the participant’s metric for answering questions from the pre-test to the post-test due to a new understanding of a concept being taught

- because the participants are able to give pre-test responses which are based on a post-intervention frame of reference.

Using the retrospective pre-test, response shift bias can be reduced increasing the likelihood that the observable results are due to intended intervention effects (Pratt, McGuigan & Katzev, 2001). This proposition is complicated and not without its critics – Theodore Lamb (2005), an advocate of the retrospective pre-test method, describes it as “an imperfect but useful tool.” Robson (2002, p.139-141) expresses concerns that used purely as a quasi-experimental method, this strategy could lead to an internal validity threat through regression to the mean.

References:

Klatt, J. and Taylor-Powell, E. (2005) Synthesis of Literature relative to RetrospectivePretest Design. Presentation to the 2005 Joint CES/AEA Conference, Toronto [Internet] Available from: http://www.citra.org/Assets/documents/evaluation%20design.pdf [Accessed 5th August 2007]

Lamb, T. (2005) The Retrospective Pretest: An Imperfect but Useful Tool. Evaluation Exchange. [Internet] 11(2). Available from: http://www.gse.harvard.edu/hfrp/eval/issue30/spotlight.html [Accessed 21st March 2007]

Pratt, C.C., McGuigan, W.M., & Katzev, A.R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3). 341-349.

Robson, C. (2002) Real World Research. 2nd ed. Oxford: Blackwell Publishing.

Rockwell, S. K. & Kohn, H. (1989) Post-Then-Pre Evaluation. Journal of Extension. [Internet] 27(2). Available from: http://www.joe.org/joe/1989summer/a5.html [Accessed 21st March 2007]

--

Evaluating Non-formal e-learning initatives: Case Study Techniques

As I asserted in a previous post, case studies allow naturalistic methods of enquiry such as individual interviews to be carried out, side-by-side with quantitative data gathering from surveys, archival records, server access logs and so on.

Interestingly in discussing the flexibility of the case study design, Yin (1994, p.285) contends that in the future, researchers will focus not on the case study method, but rather on the specifi case study data collection techniques (see Table 1) and we will increasingly see these techniques used in other “non-case study” forms of research.

Table 1 Case Study Techniques

Case Study Techniques

The use of multiple sources of evidence, in a converging manner

The explicit specification and testing of hypotheses and rival hypotheses, especially in lieu of control or comparison groups

The dominance of deductive strategies, whereby research starts with theorizing

Program logic models as a standard way of initiating a program evaluation

Pattern-matching as a common strategy for data analysis

Portfolio analysis, using qualitative criteria to differentially weigh the outcomes from a project or the projects within a program

The use of replication logic, rather than aggregating data, when comparing the results from multiple sites or cases.

This approach of modularising (to coin a term) the methodologies used in a case study may potentially enable future researchers to generalise from results attained through a case study approach to a broader context: this concept is outside the scope of this blog, but is something in my view that would have benefits in future social sciences research and evaluation of learning initiatives.

References:

Yin, R. K. (1994). Discovering the future of the case study method in evaluation research. Evaluation Practice [Internet] 15. Available from: http://aje.sagepub.com/cgi/reprint/15/3/283 [Accessed 15th January 2008 subscription required]

--

Creating a Research Design to evaluate the effectiveness of e-learning projects

In a previous blog entry, I mentioned the importance of data triangulation, particularly if you use a case study approach when evaluating an e-learning initiative. Another strategy I recommend is to collect data over a number of phases, using a number of techniques.

Case studies involve in-depth, descriptive data collection and analysis of a case, or a number of related instances of the same type within the case. In particular, the case study design is useful when answering “how” and “why” questions, and in understanding the particulars, and diversity of the case (Yin, 1994). As Yin asserted in his 1993 paper Applications of Case Study Research

the case study is the method of choice when the phenomenon under study is not readily distinguishable from its context.

(p.3)

The case study design (see Table 1) suited the conditions of this research project very well as it allowed naturalistic methods of enquiry such as individual interviews to be carried out, side-by-side with quantitative data gathering from surveys and archival records.

Table 1 Features of a case study (Russ-Eft & Preskill, 2001, pp.173-174)

Advantages	Disadvantages
Provides descriptive data	Results may not lead to scientific generalizability
Does not require control of participants or setting	Researcher bias may interfere with validity of the findings
Reports include verbatim quotes	May take too long to conduct
Leads to a greater understanding of the context of the evaluand	May produce more data than can be analysed in an effective manner
Gather data using multiple methods
Provides data that are rich in examples
Captures what is important to the participants
Portrays the multiplicity of causes that are associated with various outcomes
Embraces diversity of perspectives and experiences of participants
Allows the researcher to collect information on outcomes not known prior to the learning and performance initiative

As Robson points out, the flexibility of the case study approach allows the design to “emerge” (2002, p.89) during data collection and analysis. Similarly, the case study facilitates the application of other research designs within the case study design framework, enabling the learning professional to capture data during different phases of a project.

References:

Robson, C. (2002) Real World Research. 2nd ed. Oxford: Blackwell Publishing.

Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.

Yin, R. K. (1993). Applications of Case Study Research. Newbury Park, CA. Sage.

Yin, R. K. (1994). Discovering the future of the case study method in evaluation research. Evaluation Practice [Internet] 15. Available from: http://aje.sagepub.com/cgi/reprint/15/3/283 [Accessed 15th January 2008 subscription required]

--

Evaluating e-learning programs: more on the Program Logic Model

In the context of evaluating e-learning programs, the logic model is useful for identifying elements of the programme that are most likely to yield useful evaluation data, and to identify an appropriate sequence for collecting data and measuring learning. In this way, the programme logic model can help you develop a research design, and provide guidance to relevant and appropriate data collection strategies within your case study. Figure 1 illustrates how I apply the Program Logic Model in my case studies.

Figure 1 Programme Logic Model applied to the Information Session non-formal learning initiative

_ _

Evaluating Non-formal e-learning initatives: the Program Logic Model

In undertaking a case study when evaluating any e-learning initiative, it is essential that the data collected in the research is relevant and appropriate to the task of evaluating the impact of the intervention on the audience.

According to Wright & Ross (2001), the Program Logic Model

offers a conceptual and practical framework for this exacting work. At its core it is a simplified picture of a programme, initiative or project. In much the same way that a software developer would employ a use case, the logic model shows the relationships among the resources that are invested, the intervention or the activities that take place, and the outcomes that result from the project. Essentially, it is a tool to guide critical thinking.

(p.153).

Figure 1 The Program Logic Model

This model great utility to the e-learning practitioner, specifically in orienting the research and ensuring their learning evaluation methodology aligns to the investigation being undertaken. As Cooksy, Gill and Kelly discuss in their article The Program Logic Model as an Integrative Framework for a Multimethod Evaluation, “program theory guides an evaluation by identifying key program elements and articulating how these elements are expected to relate to each other” (p.109).

As illustrated in Figure 1 above, there are five components to the Program Logic Model: each component has a range of characteristics associated with it. These are described by Paul McCawley (1997):

1. The situation statement provides an opportunity to communicate the relevance of the project. This establishes a baseline for comparison at the close of a project and provides a way to determine whether change has occurred. Describing who is affected by the initiative allows assessment of who has benefitted.
2. Inputs include those things that we invest in a programme or project such as knowledge, skills, or expertise. Describing the inputs needed for a project provides an opportunity to communicate the quality of the programme.
3. Outputs are “those things that we do” (p.4) - providing deliverables, products - and the people we reach - intended audience. Describing the outputs establishes linkages between the problem (situation) and the impact of the project (intended outcomes).
4. In terms of program or project outcomes, these can be short-term, intermediate-term, or long-term, Outcomes answer the question “What happened as a result of the program?”

• Short-term outcomes of educational programmes may include changes in:
• Awareness – customers recognise the problem or issue
• Knowledge – customers understand the causes and potential solutions
• Skills – customers possess the skills needed to resolve the situation
• Motivation – customers have the desire to effect change
• Intermediate-term outcomes include changes that follow the short-term outcomes, such as changes in:
• Practices used by participants
• Behaviours exhibited by people or organisations
• Long-term outcomes follow intermediate-term outcomes when changed behaviours result in changed conditions or processes.

References:

Cooksy, L. J. Gill, P. & Kelly. P. A. (2001) The Program Logic Model as an Integrative Framework for a Multimethod Evaluation. Evaluation and Program Planning [Internet] 24(2). Available from: http://www.hsrd.houston.med.va.gov/AdamKelly/Logic.html [Accessed 21st October, 2006]

McCawley, P. F. (1997) The Logic Model for Program Planning and Evaluation. University of Idaho Extension [Internet] Available from: http://www.uidaho.edu/extension/LogicModel.pdf [Accessed 18th January 2007]

Wright, L. & Ross, J. W. (2001) The Logic Model: An A-to-Z Guide to Training Development, Implementation, and Evaluation. Association of the American Public Human Services Association (APHSA). [Internet] Available from: http://calswec.berkeley.edu/CalSWEC/2001_08Ross.pdf [Accessed 10th January 2007]

--

Evaluating Non-formal Learning: Establishing validity in your research

As discussed in a previous post, to assist in understanding the “parallel criteria” (1989, p.233) relationship between these test of rigour and the scientific terms for validity, Guba and Lincoln provide the following table (1981, p.104):

Table 1 Relationship between parallel criteria

Aspect	Scientific Term	Naturalistic Term
Truth Value	Internal Validity	Credibility
Applicability	External Validity	Fittingness/transferability
Consistency	Reliability	Dependability/auditability
Neutrality	Objectivity	Confirmability

When considering appropriate research designs to evaluate the effectiveness of learning initiatives, it's important to consider the types of data that you have access to, particularly in the context of the issues evaluating a non-formal learning initiative at Kirkpatrick’s Levels 3 and 4 (Shelton. & Alliger, 1993, pp.43-46, Horton, 2006, p.109), I feel it is essential to ensure validity of the naturalistic information by capturing a range of supporting quantitative data.

To assist evaluators understand and provide a context for the analysis of the information collected in the final, qualitative phase of the study, Russ-Eft and Preskill (2001, pp.155-156) describe a number of useful techniques for establishing the validity of quantitative and qualitative data, among them accuracy checking in data recording and encoding, persistent participant observation and member checking.

As the resultant evaluation will use a mix of qualitative and quantitative methods as well as data collected from multiple sources, use I recommend that you use data triangulation used as a strategy to enhance the rigour and validity of you evaluation.

The term triangulation originated in cartography where two or more reference points are used to locate an exact position.

Knowing a single landmark only locates you somewhere along a line in a direction from the landmark, whereas with two landmarks you can take bearings in two directions and locate yourself at their intersection.

(Patton, 1990, p.187).

Denzin (1988) has identified four types of triangulation – data triangulation, methodological triangulation, investigator triangulation, and theory triangulation; the I use these three triangulation techniques were used to evaluate the effectiveness of non-formal learning initiatives:

Table 2 Denzin's (1988) Definition of triangulation methods

Triangulation method	Definition
Data triangulation	Collecting data from a variety of sources. For example, in evaluating the transfer of learning from a three-hour workshop, the evaluator collects information from the learners, their managers, and their peers. In this case, three different sources have been queried.
Methodological triangulation	Using more than one method to collect data. For example, we may interview 20 per cent of a department’s employees and survey the remaining 80 per cent. By using two methods, the weaknesses of one may be compensated fro by the other.
Theory triangulation	Using different theoretical perspectives to interpret the same data. By applying different theories to make sense of the data, it is possible to see how different assumptions and beliefs influence one’s interpretations. By making these explicit, stakeholders can see how their assumptions might influence various actions taken because of the findings.

References:

Denzin, N. K. (1988) The Research Act: A Theoretical Introduction to Sociological Methods. 3^rd Ed. Englewood Cliffs, NJ: Prentice-Hall.

Guba, E.G. & Lincoln, Y.S. (1981) Effective Evaluation. San Francisco: Jossey-Bass.

Horton, W. (2006) So how is elearning different? IN: Kirkpatrick, P. & Kirkpatrick, J. Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Patton, M. Q. (1990) Qualitative Evaluation and Research Methods. 2^nd Ed. Thousand Oaks, CA. Sage

Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.

Shelton, S. & Alliger, G. (1993) Who's Afraid of Level 4 Evaluation? A Practical Approach. Training & Development. [Internet] 41 (3). Available from: http://www.afc-ispi.org/Repository/
Approaches%20for%20Assessing%20Outcomes%20ROI.doc [Accessed 24th March 2007]