Online Presentation Tools as a Knowledge-Sharing Channel 4: assessment of authorStream and SlideShare

After yesterday's short digression into some of the finer points of heuristic evaluation, today I'm returning to the matter at hand: an assessment of the features and functionality provided by the authorSTREAM and SlideShare hosted presentation distribution services.

Now read on...

authorSTREAM is a web-based PowerPoint presentation sharing platform from authorGEN Technologies. After creating a presentation in Microsoft PowerPoint, content creators can upload presentations to the authorSTREAM website. During the upload process the content is converted Adobe to Flash SWF format displayed via the Flash player. Content creators can use the provided HTML code to embed the presentation in a blog or website, as well to share the presentation via e-mail, and submit the content to YouTube. authorSTREAM also enables users to share their presentations via mobile devices that support MV4 playback such as the Apple iPod or iPhone.

Slideshare supports a range of import file types, including PowerPoint PPT, OpenOffice.org Impress, and PDF format. Broadly speaking, we can say that the SlideShare upload process shares many characteristics with other Web 2.0 tools, and is in fact very similar to the authorSTREAM submission procedure. Generally speaking, once a file is uploaded and converted to SWF, it is publicly available.

The content creators can choose to make the slide show available to be downloaded. Interestingly for a service provided by a for-profit organization Slideshare is linked with Creative Commons, so various attribution licenses are supported (click here for more thoughts on Open Source Software). As with other hosted Web 2.0 services (including YouTube), a presentation can be viewed in the small screen or take the full screen of a monitor.

Table 1 authorSTREAM & SlideShare Feature Comparison

Feature	authorSTREAM	SlideShare
PowerPoint animations	Yes	No
PowerPoint animations with audio	Yes	No
Motion-based content (i.e. Flash animation / movies) in slides	Yes	No
Rehearsed timings	Yes	No
Voice-over narration	Yes	No
Embed code	Yes	Yes
Statistics	No	Yes
RSS Feed support	Yes	Yes
Full-screen playback	Yes	Via SlideShare.net only
Private content	Yes	Yes
PowerPoint file download	Yes	Yes
User Groups	No	Yes

General Observations: Content Authoring

authorGEN provide authorPOINT Lite, an advanced multimedia presentation creation software utility, which installs in PowerPoint and offers a host of features for power users. Presenters can upload presentations directly from within PowerPoint with this add-on software.

authorPOINT is an advanced multimedia presentation creation software, which installs in PowerPoint as an add-in and provides a range of features for power users. Presenters can upload presentations directly from within PowerPoint with this utility. SlideShare does not offer any offline desktop utility to support the website service.

General Observations: Audio
authorSTREAM handles voice-over audio in presentations very well. Audio is retained in narrated PowerPoint presentations when uploaded to authorSTREAM. Authors can also create audio presentations using either live or pre-recorded audio as a media type via authorPOINT, before uploading to authorSTREAM.
SlideShare on the other hand, has a “slidecast” feature where the content author synchronizes a pre-recorded audio file online following upload to the server.

authorSTREAM's offline desktop application is a very flexible feature - potentially very useful for e-learning content, in my view.

General Observations: Content Distribution

One very usable feature of SlideShare is it facility to enable extended distribution of content outside of it's native hosting environment. By clicking on the <embed> button on the UI, both an author and any other user can either share the presentation via any one of a range of social networking services (see the screenshot for some examples) as well as provide the source code to enable the presentation to be embedded in a blog or web page, very similar to YouTube's functionality. A case could definitely be made that this type of portability will support the creation of Internet memes in environments where YouTube is not appropriate (due to bandwidth issues, for example).

Conclusions

So which service is better? Well, I guess that comes down to WIIFM - what's in it for me. The broad sweep of features and functionality are pretty similar on both services: deficiencies in one are accommodated by lack of functionality in the other, so it really comes down to personal choice.

My user experience of authorSTREAM was slightly superior to that of SlideShare: upload and conversion times were lower on the former service, for example. Equally, the enhanced slide animation and transition support on authorSTREAM was a positive advantage, as was the ability to view presentations in full-screen mode via a third party environment (my blog in this case). I also prefer the extended functionality afforded by the desktop-based client client, as well as the better audio integration.

In the end... well, I recommend that you try both out yourself, and you decide which service meets your needs.

--

Guidelines for Evaluating Training Programs

An advantage of the retrospective pre-test model is that it can assist in addressing some of the gaps exposed by the lack of range-of-transfer of learning metrics provided by a Level 2 analysis of learners: in effect - to borrow a phrase from Jens Bjornavold – “make learning visible” (2004, p.64). This is essential, because it helps measure learning. In Chapter 5 of Evaluating Training Programs (2006) Kirkpatrick and Kirkpatrick stress the significance of measuring learning, because “no change in behaviour can be expected unless one or more of the learning objectives have been accomplished” (p.42). They set out “helpful” guidelines for the measurement of learning:

Table 1 Guidelines for evaluating learning

Guidelines for evaluating learning

Use a control group if practical

Evaluate knowledge skills and/or attitudes both before and after the program

Use a paper-and-pencil test to measure knowledge and attitudes

Use a performance test to measure skills

Get a 100 per cent response

Use the results of the evaluation to take appropriate action

References:

Bjornavold, J. (2004) Making Learning Visible: Validation of Formal, Non-Formal and Informal Learning: policy and practices in EU Member States1. European Journal of Education [Internet] 39 (1), 69–89. Available from: http://www.acc.eu.org/uploads/
Makinglearningvisible_1.pdf [Accessed 21st February 2007]

Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

--

Assessing learning programs: advantages and disadvantages of the pre-test design

When choosing to integrate the retrospective pre-test design method into your learning program evaluation and measurement methodology, the selection must take place in the after very carefully evaluating arguments both for and against its utility. The method has many advantages for the practitioner-researcher (see Table 1) but one must be cognisant of the threats to validity. However, in the context of the case study method, triangulation of data from other sources (questionnaires, surveys, quantitative analysis of access logs, LMS records, summative assessments and so on) can be used to counteract or balance such threats.

Table 1 Advantages and disadvantages of the retrospective pre-test model

Advantages	Disadvantages
Simple and cost-effective	Possibility that results were due to history in the job or organisation
Reduces costs and time for data collection and analysis	Possible distortions in retrospective reports because of response shift bias
Gathers data as part of the learning intervention
Compares post-intervention data with retrospective pre-data
Avoids attrition from the sample being measured
Decreases likelihood of testing effects

--

Evaluating learning initatives: the Retrospective Pretest Design

The Retrospective Pre-test method is an interesting if controversial data collection design. I will examine its advantages and negative aspects here, as well as an argument for including this strategy in in assessing organisational learning initiatives. The retrospective pre-test method (see Table 1 is an extension of the one-shot case study design where data are gathered from participants following the learning intervention (for example, a series of non-formal Information Sessions).

Table 1 Retrospective Pre-test Design

Retrospective Pre-test Design
Groups		Non-equivalent	Intervention	Observation
Locally-based		N	X	O1	O2
Remotely-based		N	X	O1	O2
<- Time ->

However, participants (N) report on their knowledge, understanding or skills (Observation O1) after the intervention (X), and then also reflect and answer (O2) what they believe their understanding or skill was before the intervention. Rockwell & Kohn (1989) applied this method to testing the effectiveness of achieving program outcomes when interventions such as training programs are implemented and concluded that “using a post-then-pre design [retrospective pre-test] to identify self-reported behavioural changes can provide substantial evidence for program impact”. The retrospective pre-test enables researchers to reduce the response shift bias - defined by Klatt & Taylor-Powell in their 2005 paper Synthesis of Literature relative to Retrospective Pretest Design as

the change in the participant’s metric for answering questions from the pre-test to the post-test due to a new understanding of a concept being taught

- because the participants are able to give pre-test responses which are based on a post-intervention frame of reference.

Using the retrospective pre-test, response shift bias can be reduced increasing the likelihood that the observable results are due to intended intervention effects (Pratt, McGuigan & Katzev, 2001). This proposition is complicated and not without its critics – Theodore Lamb (2005), an advocate of the retrospective pre-test method, describes it as “an imperfect but useful tool.” Robson (2002, p.139-141) expresses concerns that used purely as a quasi-experimental method, this strategy could lead to an internal validity threat through regression to the mean.

References:

Klatt, J. and Taylor-Powell, E. (2005) Synthesis of Literature relative to RetrospectivePretest Design. Presentation to the 2005 Joint CES/AEA Conference, Toronto [Internet] Available from: http://www.citra.org/Assets/documents/evaluation%20design.pdf [Accessed 5th August 2007]

Lamb, T. (2005) The Retrospective Pretest: An Imperfect but Useful Tool. Evaluation Exchange. [Internet] 11(2). Available from: http://www.gse.harvard.edu/hfrp/eval/issue30/spotlight.html [Accessed 21st March 2007]

Pratt, C.C., McGuigan, W.M., & Katzev, A.R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3). 341-349.

Robson, C. (2002) Real World Research. 2nd ed. Oxford: Blackwell Publishing.

Rockwell, S. K. & Kohn, H. (1989) Post-Then-Pre Evaluation. Journal of Extension. [Internet] 27(2). Available from: http://www.joe.org/joe/1989summer/a5.html [Accessed 21st March 2007]

--

Evaluating Non-formal e-learning initatives: Case Study Techniques

As I asserted in a previous post, case studies allow naturalistic methods of enquiry such as individual interviews to be carried out, side-by-side with quantitative data gathering from surveys, archival records, server access logs and so on.

Interestingly in discussing the flexibility of the case study design, Yin (1994, p.285) contends that in the future, researchers will focus not on the case study method, but rather on the specifi case study data collection techniques (see Table 1) and we will increasingly see these techniques used in other “non-case study” forms of research.

Table 1 Case Study Techniques

Case Study Techniques

The use of multiple sources of evidence, in a converging manner

The explicit specification and testing of hypotheses and rival hypotheses, especially in lieu of control or comparison groups

The dominance of deductive strategies, whereby research starts with theorizing

Program logic models as a standard way of initiating a program evaluation

Pattern-matching as a common strategy for data analysis

Portfolio analysis, using qualitative criteria to differentially weigh the outcomes from a project or the projects within a program

The use of replication logic, rather than aggregating data, when comparing the results from multiple sites or cases.

This approach of modularising (to coin a term) the methodologies used in a case study may potentially enable future researchers to generalise from results attained through a case study approach to a broader context: this concept is outside the scope of this blog, but is something in my view that would have benefits in future social sciences research and evaluation of learning initiatives.

References:

Yin, R. K. (1994). Discovering the future of the case study method in evaluation research. Evaluation Practice [Internet] 15. Available from: http://aje.sagepub.com/cgi/reprint/15/3/283 [Accessed 15th January 2008 subscription required]

--

Creating a Research Design to evaluate the effectiveness of e-learning projects

In a previous blog entry, I mentioned the importance of data triangulation, particularly if you use a case study approach when evaluating an e-learning initiative. Another strategy I recommend is to collect data over a number of phases, using a number of techniques.

Case studies involve in-depth, descriptive data collection and analysis of a case, or a number of related instances of the same type within the case. In particular, the case study design is useful when answering “how” and “why” questions, and in understanding the particulars, and diversity of the case (Yin, 1994). As Yin asserted in his 1993 paper Applications of Case Study Research

the case study is the method of choice when the phenomenon under study is not readily distinguishable from its context.

(p.3)

The case study design (see Table 1) suited the conditions of this research project very well as it allowed naturalistic methods of enquiry such as individual interviews to be carried out, side-by-side with quantitative data gathering from surveys and archival records.

Table 1 Features of a case study (Russ-Eft & Preskill, 2001, pp.173-174)

Advantages	Disadvantages
Provides descriptive data	Results may not lead to scientific generalizability
Does not require control of participants or setting	Researcher bias may interfere with validity of the findings
Reports include verbatim quotes	May take too long to conduct
Leads to a greater understanding of the context of the evaluand	May produce more data than can be analysed in an effective manner
Gather data using multiple methods
Provides data that are rich in examples
Captures what is important to the participants
Portrays the multiplicity of causes that are associated with various outcomes
Embraces diversity of perspectives and experiences of participants
Allows the researcher to collect information on outcomes not known prior to the learning and performance initiative

As Robson points out, the flexibility of the case study approach allows the design to “emerge” (2002, p.89) during data collection and analysis. Similarly, the case study facilitates the application of other research designs within the case study design framework, enabling the learning professional to capture data during different phases of a project.

References:

Robson, C. (2002) Real World Research. 2nd ed. Oxford: Blackwell Publishing.

Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.

Yin, R. K. (1993). Applications of Case Study Research. Newbury Park, CA. Sage.

Yin, R. K. (1994). Discovering the future of the case study method in evaluation research. Evaluation Practice [Internet] 15. Available from: http://aje.sagepub.com/cgi/reprint/15/3/283 [Accessed 15th January 2008 subscription required]

--

Evaluating e-learning programs: more on the Program Logic Model

In the context of evaluating e-learning programs, the logic model is useful for identifying elements of the programme that are most likely to yield useful evaluation data, and to identify an appropriate sequence for collecting data and measuring learning. In this way, the programme logic model can help you develop a research design, and provide guidance to relevant and appropriate data collection strategies within your case study. Figure 1 illustrates how I apply the Program Logic Model in my case studies.

Figure 1 Programme Logic Model applied to the Information Session non-formal learning initiative

_ _

Evaluating Non-formal e-learning initatives: the Program Logic Model

In undertaking a case study when evaluating any e-learning initiative, it is essential that the data collected in the research is relevant and appropriate to the task of evaluating the impact of the intervention on the audience.

According to Wright & Ross (2001), the Program Logic Model

offers a conceptual and practical framework for this exacting work. At its core it is a simplified picture of a programme, initiative or project. In much the same way that a software developer would employ a use case, the logic model shows the relationships among the resources that are invested, the intervention or the activities that take place, and the outcomes that result from the project. Essentially, it is a tool to guide critical thinking.

(p.153).

Figure 1 The Program Logic Model

This model great utility to the e-learning practitioner, specifically in orienting the research and ensuring their learning evaluation methodology aligns to the investigation being undertaken. As Cooksy, Gill and Kelly discuss in their article The Program Logic Model as an Integrative Framework for a Multimethod Evaluation, “program theory guides an evaluation by identifying key program elements and articulating how these elements are expected to relate to each other” (p.109).

As illustrated in Figure 1 above, there are five components to the Program Logic Model: each component has a range of characteristics associated with it. These are described by Paul McCawley (1997):

1. The situation statement provides an opportunity to communicate the relevance of the project. This establishes a baseline for comparison at the close of a project and provides a way to determine whether change has occurred. Describing who is affected by the initiative allows assessment of who has benefitted.
2. Inputs include those things that we invest in a programme or project such as knowledge, skills, or expertise. Describing the inputs needed for a project provides an opportunity to communicate the quality of the programme.
3. Outputs are “those things that we do” (p.4) - providing deliverables, products - and the people we reach - intended audience. Describing the outputs establishes linkages between the problem (situation) and the impact of the project (intended outcomes).
4. In terms of program or project outcomes, these can be short-term, intermediate-term, or long-term, Outcomes answer the question “What happened as a result of the program?”

• Short-term outcomes of educational programmes may include changes in:
• Awareness – customers recognise the problem or issue
• Knowledge – customers understand the causes and potential solutions
• Skills – customers possess the skills needed to resolve the situation
• Motivation – customers have the desire to effect change
• Intermediate-term outcomes include changes that follow the short-term outcomes, such as changes in:
• Practices used by participants
• Behaviours exhibited by people or organisations
• Long-term outcomes follow intermediate-term outcomes when changed behaviours result in changed conditions or processes.

References:

Cooksy, L. J. Gill, P. & Kelly. P. A. (2001) The Program Logic Model as an Integrative Framework for a Multimethod Evaluation. Evaluation and Program Planning [Internet] 24(2). Available from: http://www.hsrd.houston.med.va.gov/AdamKelly/Logic.html [Accessed 21st October, 2006]

McCawley, P. F. (1997) The Logic Model for Program Planning and Evaluation. University of Idaho Extension [Internet] Available from: http://www.uidaho.edu/extension/LogicModel.pdf [Accessed 18th January 2007]

Wright, L. & Ross, J. W. (2001) The Logic Model: An A-to-Z Guide to Training Development, Implementation, and Evaluation. Association of the American Public Human Services Association (APHSA). [Internet] Available from: http://calswec.berkeley.edu/CalSWEC/2001_08Ross.pdf [Accessed 10th January 2007]

--

Evaluating Non-formal Learning: Establishing validity in your research

As discussed in a previous post, to assist in understanding the “parallel criteria” (1989, p.233) relationship between these test of rigour and the scientific terms for validity, Guba and Lincoln provide the following table (1981, p.104):

Table 1 Relationship between parallel criteria

Aspect	Scientific Term	Naturalistic Term
Truth Value	Internal Validity	Credibility
Applicability	External Validity	Fittingness/transferability
Consistency	Reliability	Dependability/auditability
Neutrality	Objectivity	Confirmability

When considering appropriate research designs to evaluate the effectiveness of learning initiatives, it's important to consider the types of data that you have access to, particularly in the context of the issues evaluating a non-formal learning initiative at Kirkpatrick’s Levels 3 and 4 (Shelton. & Alliger, 1993, pp.43-46, Horton, 2006, p.109), I feel it is essential to ensure validity of the naturalistic information by capturing a range of supporting quantitative data.

To assist evaluators understand and provide a context for the analysis of the information collected in the final, qualitative phase of the study, Russ-Eft and Preskill (2001, pp.155-156) describe a number of useful techniques for establishing the validity of quantitative and qualitative data, among them accuracy checking in data recording and encoding, persistent participant observation and member checking.

As the resultant evaluation will use a mix of qualitative and quantitative methods as well as data collected from multiple sources, use I recommend that you use data triangulation used as a strategy to enhance the rigour and validity of you evaluation.

The term triangulation originated in cartography where two or more reference points are used to locate an exact position.

Knowing a single landmark only locates you somewhere along a line in a direction from the landmark, whereas with two landmarks you can take bearings in two directions and locate yourself at their intersection.

(Patton, 1990, p.187).

Denzin (1988) has identified four types of triangulation – data triangulation, methodological triangulation, investigator triangulation, and theory triangulation; the I use these three triangulation techniques were used to evaluate the effectiveness of non-formal learning initiatives:

Table 2 Denzin's (1988) Definition of triangulation methods

Triangulation method	Definition
Data triangulation	Collecting data from a variety of sources. For example, in evaluating the transfer of learning from a three-hour workshop, the evaluator collects information from the learners, their managers, and their peers. In this case, three different sources have been queried.
Methodological triangulation	Using more than one method to collect data. For example, we may interview 20 per cent of a department’s employees and survey the remaining 80 per cent. By using two methods, the weaknesses of one may be compensated fro by the other.
Theory triangulation	Using different theoretical perspectives to interpret the same data. By applying different theories to make sense of the data, it is possible to see how different assumptions and beliefs influence one’s interpretations. By making these explicit, stakeholders can see how their assumptions might influence various actions taken because of the findings.

References:

Denzin, N. K. (1988) The Research Act: A Theoretical Introduction to Sociological Methods. 3^rd Ed. Englewood Cliffs, NJ: Prentice-Hall.

Guba, E.G. & Lincoln, Y.S. (1981) Effective Evaluation. San Francisco: Jossey-Bass.

Horton, W. (2006) So how is elearning different? IN: Kirkpatrick, P. & Kirkpatrick, J. Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Patton, M. Q. (1990) Qualitative Evaluation and Research Methods. 2^nd Ed. Thousand Oaks, CA. Sage

Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.

Shelton, S. & Alliger, G. (1993) Who's Afraid of Level 4 Evaluation? A Practical Approach. Training & Development. [Internet] 41 (3). Available from: http://www.afc-ispi.org/Repository/
Approaches%20for%20Assessing%20Outcomes%20ROI.doc [Accessed 24th March 2007]

Evaluating Non-formal Learning: Using "tests of rigor" to validate findings

Without rigour, research is worthless, becomes fiction, and loses its utility. Hence, a great deal of attention is applied to reliability and validity in all research methods.

(Morse et al, 2002, p.1)

Morse and his colleagues develop their position by asserting that the challenges to rigour in qualitative inquiry paralleled the growth of statistical packages and the development of computing systems in quantitative research: without “the certainty of hard numbers and p values, qualitative inquiry expressed a crisis of confidence” (Morse et al, 2002, p.2).

A number of leading qualitative researchers (Leininger, 1994, Altheide & Johnson, 1998) argued that the two approaches are not compatible and that they should not be combined, due to the fundamental differences between the two methods in the context of the nature of knowledge, the relationship between researcher and the object of the research, and the means of generating data.

My personal view aligns with the criteria suggested by Guba & Lincoln (1981) and Yin (1994) to determine reliability, and validity, and thus ensuring rigour, in qualitative research. Reliability and validity relate to the degree of confidence we have that the data are representing the participants’ reality or “truth” (Russ-Eft & Preskill, 2001, p.153). How can we define the truth (or even the accuracy) of participants’ responses in the subjective domain of qualitative research? Guba and Lincoln (1981) propose that researchers use so-called “tests of rigour” to establish validity through the naturalistic concept of “trustworthiness,” containing four aspects: credibility, transferability, dependability, and confirmability, that align to parallel scientific terms associated with the quantitative method.

Table 1 Definition of naturalistic terms (Russ-Eft & Preskill, 2001, pp153-155)

Naturalistic Term	Definition
Credibility	The scientific paradigm asserts that there is one reality and that information is valid when all relevant variables can be controlled; a naturalistic paradigm assumes that multiple realities exist in the minds of individuals. Hence when using qualitative methods, the research seeks to establish the credibility of individuals’ responses. The study must be believable by providing a detailed depiction of the multiple perspectives that exist can enhance the data’s credibility. For example, learner satisfaction surveys, along with interviews with training managers and instructors would provide a more holistic picture of the learning experience.
Fittingness/transferability	How transferable the findings are to another setting is called generalisability in the empirical context. The goal of qualitative methods is to provide richly detailed description to help the reader relate certain findings to their own experience. We often think of these as “lessons learned.” For example, as a stakeholder reads an evaluation report, they realise that something very similar has occurred in their organisation and sees where the findings can be used. Although the entire set of findings may not be applicable in their context, some issues identified may have applicability in other contexts.
Dependability/auditability	In the scientific paradigm, the notion of consistency is called reliability where a study’s consistency, predictability or stability is measured. Since reliability is necessary for validity, it is critical that data of any kind be reliable. Instead of considering data unreliable if it is inconsistent, evaluators using qualitative methods look for reasons that cause the data to appear unstable (inconsistent). For example, an interviewee may give an opinion one day, and when asked again the following week might say something slightly different. What would be important to understand and capture are the reasons for this change in perception. Such inconsistencies may stem from respondent error, an increase in available information, or changes in the situation. An audit trail that includes collecting documents and interview notes and a daily journal of how things are proceeding can help to uncover some of the reasons for such inconsistencies.
Confirmability	Objectivity is often viewed as the goal of most evaluation and research studies. Evaluators and researchers who use qualitative methods don’t necessarily believe that true objectivity can ever be fully achieved, rather that it is impossible to completely separate the evaluator from the method. Instead of trying to ensure that the data are free from the evaluator’s biases, the goal is to determine the extent to which the data provide confirming evidence. “This means that data (constructions, assertions, facts and so on) can be tracked to their sources, and that the logic used to assemble the interpretations into structurally coherent and corroborating wholes is both explicit and implicit” (Guba & Lincoln, 1989, p.243). Establishing confirmability, like consistency, often takes the form of auditing.

References:

Altheide, D., & Johnson, J. M. C. (1998). Criteria for assessing interpretive validity in qualitative research. IN: Denzin N. K. & Lincoln Y. S. (Eds.), Collecting and interpreting qualitative materials. Thousand Oaks, CA: Sage.

Guba, E.G. & Lincoln, Y.S. (1981) Effective Evaluation. San Francisco: Jossey-Bass.

Leininger, M. (1994). Evaluation criteria and critique of qualitative research studies. In J. M. Morse (Ed.), Critical Issues in Qualitative Research Methods. Newbury Park, CA: Sage.

Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002). Verification strategies for establishing reliability and validity in qualitative research. [Internet] International Journal of Qualitative Methods 1 (2), Article 2. Available from: http://www.ualberta.ca/~ijqm/ [Accessed 14th March 2008]

Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.

Yin, R. K. (1994). Discovering the future of the case study method in evaluation research. Evaluation Practice [Internet] 15. Available from: http://aje.sagepub.com/cgi/reprint/15/3/283 [Accessed 15th January 2007]

--

Still more on evaluating non-formal learning...

Well now. Back from NY and totally energised by my trip to Manhattan. Where were we...? Oh yes -

The methodology I'll be describing is a generalised account of the type work I carry out when developing and implementing non-formal learning initiatives in organisations - I invite you to argue, discuss, praise etc. as appropriate.

Typically, I consult for small- to mid-sized enterprises in Ireland. The type of learning interventions being discussed here are what I call 'Information Sessions.' Information Sessions are learning events designed to disseminate a broad range of knowledge and information about an organisation and its products. The title 'Information Session' is carefully chosen: these events are not formal training (see this blog entry), nor are they informal learning; they are informational by definition and design, though they are developed according to a strong instructional design process. The Info Session events are hosted as synchronous, SME-led presentations. The content is repurposed post-event for asynchronous distribution on-demand over the Web, as well as being distributed to learners on DVD-ROM. Information Sessions are distributed in the following formats:

Table 1 Information Sessions format

Synchronous Asynchronous Live audience in situ at Information Session event location (i.e. learners' office or other facility) On-demand post-live event delivery over the Web Streamed live over the web DVDs available to learners. Each DVD contains one series of Information Session presentations

In the first phase of data collection, quantitative data are collected on every attendee participating in both the live in situ and the synchronous and asynchronous distribution of the events. An E-learning Readiness Survey in the second phase of research is distributed to internal experts who fall under the aegis of what William Horton (2006, p.107) describes as “those who should know” - managers, learning and knowledge professionals, and key stakeholders in the organisation. A representative sample (proportionate to staff numbers) of employees participate in Phase III of the research design. Informed consent is obtained from each contributor by asking them to complete a sign-off statement form in the questionnaire (we'll get back to the questionnaire) and asking their assent to use their responses before follow-up interviews are conducted.

Note: As the research design follows a case study approach and is based on a learning initiative in a single organisation, I am cognisant of the fact that the potential exists for “individuals’ interests to be affected” (Robson, 2002, p.501) by participating in the research, particularly if their opinion deviates from the corporate policy.

References:

Horton, W. (2006) So how is elearning different? IN: Kirkpatrick, P. & Kirkpatrick, J. Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Robson, C. (2002) Real World Research. 2nd ed. Oxford: Blackwell Publishing.

--