Learning Evaluation and Strategy: Using an e-learning readiness survey

During this series of posts on evaluating non-formal learning programs, I have mentioned carrying out an e-learning readiness survey without characterizing or discussing how to implement such a research instrument.

This was deliberate; in my view e-learning readiness surveys represent an alpha and an omega of evaluation: on one level they are the starting point for any evaluation of an organization's learning initiative, on another level they define an organization's ability to implement an effective e-learning strategy. As such, they are a bridge between the theory and the practice of implementing a learning program.

Readiness surveys enable the learning practitioner to understand and measure both the effectiveness of organizational learning and to identify the critical factors for success when developing learning programs.

I have found Marc J. Rosenberg's E-Learning Readiness Survey to be a very effective instrument to evaluate both the effectiveness of an organization's learning strategy, and one of the foundations of any serious evaluation of the effectiveness of learning programs.

The questions are grouped into seven areas of understanding:

1. business readiness
2. the changing nature of learning and e-learning
3. value of instructional and information design
4. change management
5. reinventing the training organization
6. the e-learning industry
7. your personal commitment

The questions provided in this survey represent some of the most important strategic issues organizations face when transitioning to e learning. Certainly there are additional questions and issues that deserve attention; add your own, organization-specific items as required.

Downloads:

Marc J. Rosenberg's E-learning Readiness Survey

References:

Rosenberg, M. J. (2006) Beyond e-Learning. San Francisco, CA: John Wiley & Sons, Inc.
--

Using quantitative data in learning assessments

To support the qualitative data captured using the retrospective pre-test method, I recommend that you gather significant amount of quantitative data associated with the learning initiative or program you're evaluating.


This can seem daunting and difficult to achieve, but there are a range of accessible sources of quantitative data that every learning professional can use.

Now read on...

Always collect Level 1 attendee feedback forms from classroom-based and synchronous online participants at the end of each individual event: over time you accumulate substantial amounts of valuable feedback about learners' reaction to the initiative. Use Web-hosted feedback forms to collect data from asynchronous participants (i.e. for online and DVD-ROM based content).

Other data can be collected automatically from systems including Web servers, learning management systems and learning content management systems; Horton (2006) describes these types of archive data as “meaningful statistics” (p.102): they record detailed information about what participants did while taking the learning event, particularly in the e-learning channels. You may extend this "meaningful statistics" category to include non-automated but standardised data recording processes such as recording learner attendance captured in the synchronous online and classroom-based context, and off-line learner activity such as DVD-ROM requests. By examining logs and reports from these systems and processes useful data can be collected on:

1. Frequency and pattern of accessing the course
2. Number of slides/pages or learning objects accessed
3. Duration of access to learning objects
4. Number of supporting collateral downloads
5. Feedback submitted
6. Participation in discussion and question & answer sessions
7. Rate if individual learner attendance or online access
8. Rate of attendance or online access by role (i.e. developer, support, business analyst etc)
9. Rate of group level (i.e. Sales & Marketing, Support, Manufacturing, Engineering etc) attendance or online access
10. Rate of attendance or online access by length of service in the organisation (i.e. new hire, with the company one year,1-2 years, 2-3 years, 3-5 years, +5 years etc)

More tomorrow.

References:

Horton, W. (2006) So how is elearning different? IN: Kirkpatrick, P. & Kirkpatrick, J. Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

--