Learning Evaluation and Strategy: Using an e-learning readiness survey

During this series of posts on evaluating non-formal learning programs, I have mentioned carrying out an e-learning readiness survey without characterizing or discussing how to implement such a research instrument.

This was deliberate; in my view e-learning readiness surveys represent an alpha and an omega of evaluation: on one level they are the starting point for any evaluation of an organization's learning initiative, on another level they define an organization's ability to implement an effective e-learning strategy. As such, they are a bridge between the theory and the practice of implementing a learning program.

Readiness surveys enable the learning practitioner to understand and measure both the effectiveness of organizational learning and to identify the critical factors for success when developing learning programs.

I have found Marc J. Rosenberg's E-Learning Readiness Survey to be a very effective instrument to evaluate both the effectiveness of an organization's learning strategy, and one of the foundations of any serious evaluation of the effectiveness of learning programs.

The questions are grouped into seven areas of understanding:

1. business readiness
2. the changing nature of learning and e-learning
3. value of instructional and information design
4. change management
5. reinventing the training organization
6. the e-learning industry
7. your personal commitment

The questions provided in this survey represent some of the most important strategic issues organizations face when transitioning to e learning. Certainly there are additional questions and issues that deserve attention; add your own, organization-specific items as required.

Downloads:

Marc J. Rosenberg's E-learning Readiness Survey

References:

Rosenberg, M. J. (2006) Beyond e-Learning. San Francisco, CA: John Wiley & Sons, Inc.
--

Using quantitative data in learning assessments

To support the qualitative data captured using the retrospective pre-test method, I recommend that you gather significant amount of quantitative data associated with the learning initiative or program you're evaluating.


This can seem daunting and difficult to achieve, but there are a range of accessible sources of quantitative data that every learning professional can use.

Now read on...

Always collect Level 1 attendee feedback forms from classroom-based and synchronous online participants at the end of each individual event: over time you accumulate substantial amounts of valuable feedback about learners' reaction to the initiative. Use Web-hosted feedback forms to collect data from asynchronous participants (i.e. for online and DVD-ROM based content).

Other data can be collected automatically from systems including Web servers, learning management systems and learning content management systems; Horton (2006) describes these types of archive data as “meaningful statistics” (p.102): they record detailed information about what participants did while taking the learning event, particularly in the e-learning channels. You may extend this "meaningful statistics" category to include non-automated but standardised data recording processes such as recording learner attendance captured in the synchronous online and classroom-based context, and off-line learner activity such as DVD-ROM requests. By examining logs and reports from these systems and processes useful data can be collected on:

1. Frequency and pattern of accessing the course
2. Number of slides/pages or learning objects accessed
3. Duration of access to learning objects
4. Number of supporting collateral downloads
5. Feedback submitted
6. Participation in discussion and question & answer sessions
7. Rate if individual learner attendance or online access
8. Rate of attendance or online access by role (i.e. developer, support, business analyst etc)
9. Rate of group level (i.e. Sales & Marketing, Support, Manufacturing, Engineering etc) attendance or online access
10. Rate of attendance or online access by length of service in the organisation (i.e. new hire, with the company one year,1-2 years, 2-3 years, 3-5 years, +5 years etc)

More tomorrow.

References:

Horton, W. (2006) So how is elearning different? IN: Kirkpatrick, P. & Kirkpatrick, J. Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

--

More on Kirkpatrick's Guidelines don't work for Non-Formal Learning programs

In my last posting on this topic, we discussed Kirkpatrick's guidelines for evaluating training programs (see Table 1).

Table 1 Guidelines for evaluating learning

Guidelines for evaluating learning

Use a control group if practical

Evaluate knowledge skills and/or attitudes both before and after the program

Use a paper-and-pencil test to measure knowledge and attitudes

Use a performance test to measure skills

Get a 100 per cent response

Use the results of the evaluation to take appropriate action

In this post, I will outline the difficulties encountered in applying these guidelines to non-formal learning initiatives.

Now read on...

Kirkpatick's approach is incompatible with the assessment of non-formal learning in three ways: firstly, the singular nature of each non-formal learning event means that no control group can exist to measure a difference between a control group and the “experimental group” (p.43). Similarly; a significant number of the learners that access this kind of content do so asynchronously, via a number of different learning channels and over a broad time span in a relatively ad hoc manner. This “just enough, just in time” aspect of non-formal learning is one of its strengths, but it is unrealistic to attempt to measure learning using an experimental method in such an environment.

Secondly, the lack of summative assessment in non-formal learning precludes both pencil-and-paper and performance testing to measure learning. Finally, the distributed nature of access to these events, over both time and location makes achieving a 100 per cent learner response rate (expressed by the value R) impossible. As the content (C) is always available (D) the potential (+) always exists for the current number of learners (N₁) to increment upwards (N₂). A complete learner response rate will always be a hostage to N₂ in this context.

R = R+ N₂

where C (is a known value) and N₁ (is a known value) and D (is a known value) + N₂

It follows that you can never achieve a one hundred per cent response rate. Contrast this with formal training courses. They are, by their nature, finite: each course has a specified duration. Content is delivered to learners in an instructor-led environment in a sequence of modules over the duration of the course. Once the course instruction is complete, members of any given class are required to take a comprehensive summative assessment on the learning objectives outlined in the course, usually within a set time after they complete a course to achieve certification.

Thus R = 100

where C = 10 and N₁ = 12 and D = 10

As a result, a 100 per cent response rate to knowledge and skills testing is achieved.

However, I believe that it is possible to capture and use data with the retrospective pre-test method, and I'll be discussing an approach to undertaking this activity next time.

References:

Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

--