Evaluating Non-formal e-learning initatives: the Program Logic Model

In undertaking a case study when evaluating any e-learning initiative, it is essential that the data collected in the research is relevant and appropriate to the task of evaluating the impact of the intervention on the audience.

According to Wright & Ross (2001), the Program Logic Model

offers a conceptual and practical framework for this exacting work. At its core it is a simplified picture of a programme, initiative or project. In much the same way that a software developer would employ a use case, the logic model shows the relationships among the resources that are invested, the intervention or the activities that take place, and the outcomes that result from the project. Essentially, it is a tool to guide critical thinking.

(p.153).

Figure 1 The Program Logic Model

This model great utility to the e-learning practitioner, specifically in orienting the research and ensuring their learning evaluation methodology aligns to the investigation being undertaken. As Cooksy, Gill and Kelly discuss in their article The Program Logic Model as an Integrative Framework for a Multimethod Evaluation, “program theory guides an evaluation by identifying key program elements and articulating how these elements are expected to relate to each other” (p.109).

As illustrated in Figure 1 above, there are five components to the Program Logic Model: each component has a range of characteristics associated with it. These are described by Paul McCawley (1997):

1. The situation statement provides an opportunity to communicate the relevance of the project. This establishes a baseline for comparison at the close of a project and provides a way to determine whether change has occurred. Describing who is affected by the initiative allows assessment of who has benefitted.
2. Inputs include those things that we invest in a programme or project such as knowledge, skills, or expertise. Describing the inputs needed for a project provides an opportunity to communicate the quality of the programme.
3. Outputs are “those things that we do” (p.4) - providing deliverables, products - and the people we reach - intended audience. Describing the outputs establishes linkages between the problem (situation) and the impact of the project (intended outcomes).
4. In terms of program or project outcomes, these can be short-term, intermediate-term, or long-term, Outcomes answer the question “What happened as a result of the program?”

• Short-term outcomes of educational programmes may include changes in:
• Awareness – customers recognise the problem or issue
• Knowledge – customers understand the causes and potential solutions
• Skills – customers possess the skills needed to resolve the situation
• Motivation – customers have the desire to effect change
• Intermediate-term outcomes include changes that follow the short-term outcomes, such as changes in:
• Practices used by participants
• Behaviours exhibited by people or organisations
• Long-term outcomes follow intermediate-term outcomes when changed behaviours result in changed conditions or processes.

References:

Cooksy, L. J. Gill, P. & Kelly. P. A. (2001) The Program Logic Model as an Integrative Framework for a Multimethod Evaluation. Evaluation and Program Planning [Internet] 24(2). Available from: http://www.hsrd.houston.med.va.gov/AdamKelly/Logic.html [Accessed 21st October, 2006]

McCawley, P. F. (1997) The Logic Model for Program Planning and Evaluation. University of Idaho Extension [Internet] Available from: http://www.uidaho.edu/extension/LogicModel.pdf [Accessed 18th January 2007]

Wright, L. & Ross, J. W. (2001) The Logic Model: An A-to-Z Guide to Training Development, Implementation, and Evaluation. Association of the American Public Human Services Association (APHSA). [Internet] Available from: http://calswec.berkeley.edu/CalSWEC/2001_08Ross.pdf [Accessed 10th January 2007]

--