Evaluating Non-formal Learning: Using "tests of rigor" to validate findings

Without rigour, research is worthless, becomes fiction, and loses its utility. Hence, a great deal of attention is applied to reliability and validity in all research methods.

(Morse et al, 2002, p.1)

Morse and his colleagues develop their position by asserting that the challenges to rigour in qualitative inquiry paralleled the growth of statistical packages and the development of computing systems in quantitative research: without “the certainty of hard numbers and p values, qualitative inquiry expressed a crisis of confidence” (Morse et al, 2002, p.2).

A number of leading qualitative researchers (Leininger, 1994, Altheide & Johnson, 1998) argued that the two approaches are not compatible and that they should not be combined, due to the fundamental differences between the two methods in the context of the nature of knowledge, the relationship between researcher and the object of the research, and the means of generating data.

My personal view aligns with the criteria suggested by Guba & Lincoln (1981) and Yin (1994) to determine reliability, and validity, and thus ensuring rigour, in qualitative research. Reliability and validity relate to the degree of confidence we have that the data are representing the participants’ reality or “truth” (Russ-Eft & Preskill, 2001, p.153). How can we define the truth (or even the accuracy) of participants’ responses in the subjective domain of qualitative research? Guba and Lincoln (1981) propose that researchers use so-called “tests of rigour” to establish validity through the naturalistic concept of “trustworthiness,” containing four aspects: credibility, transferability, dependability, and confirmability, that align to parallel scientific terms associated with the quantitative method.

Table 1 Definition of naturalistic terms (Russ-Eft & Preskill, 2001, pp153-155)

Naturalistic Term	Definition
Credibility	The scientific paradigm asserts that there is one reality and that information is valid when all relevant variables can be controlled; a naturalistic paradigm assumes that multiple realities exist in the minds of individuals. Hence when using qualitative methods, the research seeks to establish the credibility of individuals’ responses. The study must be believable by providing a detailed depiction of the multiple perspectives that exist can enhance the data’s credibility. For example, learner satisfaction surveys, along with interviews with training managers and instructors would provide a more holistic picture of the learning experience.
Fittingness/transferability	How transferable the findings are to another setting is called generalisability in the empirical context. The goal of qualitative methods is to provide richly detailed description to help the reader relate certain findings to their own experience. We often think of these as “lessons learned.” For example, as a stakeholder reads an evaluation report, they realise that something very similar has occurred in their organisation and sees where the findings can be used. Although the entire set of findings may not be applicable in their context, some issues identified may have applicability in other contexts.
Dependability/auditability	In the scientific paradigm, the notion of consistency is called reliability where a study’s consistency, predictability or stability is measured. Since reliability is necessary for validity, it is critical that data of any kind be reliable. Instead of considering data unreliable if it is inconsistent, evaluators using qualitative methods look for reasons that cause the data to appear unstable (inconsistent). For example, an interviewee may give an opinion one day, and when asked again the following week might say something slightly different. What would be important to understand and capture are the reasons for this change in perception. Such inconsistencies may stem from respondent error, an increase in available information, or changes in the situation. An audit trail that includes collecting documents and interview notes and a daily journal of how things are proceeding can help to uncover some of the reasons for such inconsistencies.
Confirmability	Objectivity is often viewed as the goal of most evaluation and research studies. Evaluators and researchers who use qualitative methods don’t necessarily believe that true objectivity can ever be fully achieved, rather that it is impossible to completely separate the evaluator from the method. Instead of trying to ensure that the data are free from the evaluator’s biases, the goal is to determine the extent to which the data provide confirming evidence. “This means that data (constructions, assertions, facts and so on) can be tracked to their sources, and that the logic used to assemble the interpretations into structurally coherent and corroborating wholes is both explicit and implicit” (Guba & Lincoln, 1989, p.243). Establishing confirmability, like consistency, often takes the form of auditing.

References:

Altheide, D., & Johnson, J. M. C. (1998). Criteria for assessing interpretive validity in qualitative research. IN: Denzin N. K. & Lincoln Y. S. (Eds.), Collecting and interpreting qualitative materials. Thousand Oaks, CA: Sage.

Guba, E.G. & Lincoln, Y.S. (1981) Effective Evaluation. San Francisco: Jossey-Bass.

Leininger, M. (1994). Evaluation criteria and critique of qualitative research studies. In J. M. Morse (Ed.), Critical Issues in Qualitative Research Methods. Newbury Park, CA: Sage.

Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002). Verification strategies for establishing reliability and validity in qualitative research. [Internet] International Journal of Qualitative Methods 1 (2), Article 2. Available from: http://www.ualberta.ca/~ijqm/ [Accessed 14th March 2008]

Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.

Yin, R. K. (1994). Discovering the future of the case study method in evaluation research. Evaluation Practice [Internet] 15. Available from: http://aje.sagepub.com/cgi/reprint/15/3/283 [Accessed 15th January 2007]

--

Using a Case Study approach to evaluate Non-Formal Learning

I consider a case study methodology to be the most effective strategy to use when evaluating non-formal learning because this approach provides the means to develop a rich description of the non-formal learning (NFL) initiative.

This assists in providing a context, and helps define the NFL initiative's value to the learners and to the organisation. An added benefit of the case study approach is that it enables the learning practitioner to employ a number of different data collection methodologies: participant observation, archive data collection, surveys, and interviews to answer the questions outlined in Table 1. The main purpose of using a number of data collection methodologies was to capture a range of quantitative and qualitative data, and to establish a means of triangulating that data to enhance validity and “trustworthiness” (Russ-Eft & Preskill, 2001, p.153).

Table 1 Questions the case study approach attempts to answer

Quantitative Data	Qualitative Data
Who attends live “in situ” the learning event? (dept., role, skill level, location)	Description and chronology of a the learning event
Who views synchronous live stream of the learning event? (dept., role, skill level, location)	Transfer of new knowledge to the job (Kirkpatrick - Level 3)
Who views asynchronous on-demand version of the learning event? (dept., role, skill level, location)	Does the pedagogical approach used in the learning event fulfil participants’ needs?
Who requests asynchronous DVD version of the learning event? (dept., role, skill level, location)	Do non-formal learning event enhance individuals' and organisation performance? (Kirkpatrick - Level 3)
Participants satisfaction with the learning event? (Kirkpatrick - Level 1)	What is the Business Impact of the the learning event or series of events over time (Kirkpatrick - Level 4)
What do attendees know that they didn’t know before? (Kirkpatrick - Level 2)

References:

Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.

--

Martin Dougiamas says: Moodle 1.9 now released! Hurrah!!

Like the man in the title said, Moodle 1.9 has been released.
Headline features include:

• Gradebook - Completely rewritten from scratch for speed and flexibility. The new gradebook consists of plugins for reports, imports and exports. There are a number of standard reports which are useful for graders, students etc. The grader report allows you to treat the gradebook much more like a spreadsheet with manual editing, calculations, aggregations, weighting, locking, hiding, textual notes and so on.
  
  
• Outcomes - You can also now develop a list of expected outcomes (competencies) and connect these to courses and activities. You can even grade against multiple outcomes at once (i.e. Rubrics).
  
  
• Events API - The new Events API provides a way for any code to "hook" into events in a clean, loosely coupled way. A lot of events in Moodle (such as adding a user or a course) now trigger events that developers can hook into.
  
  
• Scalability and performance improvements - A complete overhaul of the Roles implementation for correctness and scalability. Large sites with thousands of courses and users now load quickly and behave well under heavy traffic, thanks to reworked code for Roles. Additional boost for sites using PHP pre-compilers and significant improvements in the database access code for all databases. Many other parts of Moodle have been optimised to cope better with large numbers of courses and students. Overall performance is very noticeably increased.
  
  
• Moodle Network - Moodle 1.9 and Mahara E-portfolio v0.9 now do transparent Single Sign On - one to one, one to many, many to many. Students can maintain their personal E-portfolios in Mahara.
  
  
• Tags - Allows users to describe their own interests in terms of tags, which creates interest pages around those tags, bringing information together from a variety of sources (Blogs, Flickr, Youtube etc)
  
  
• Improved question bank - Allows questions to be shared by the whole site, a course category, a single course, or be kept private to a single module. More control over who can do what to each question. Improved file management for files linked to by questions.
  
  
• Notes - Detailed notes can be kept about individual users (for example teachers might want to keep and share notes about students in their class).
  
  
• Bulk user actions - Administrators can perform bulk user actions, such as the mass deletion of user accounts. Extended features in the bulk user upload script to allow generation of user fields based on templates.
  
  
• Custom corners theme - Beautiful and curvy (in all browsers).

Find out more here [links to Moodle.org]

--

LCBBQ: The Long Tail, the 80:20 Rule and the role of learning professionals

This month's Learning Circuits Blog Big Question is:

"What is the Scope of our Responsibility as Learning Professionals?"

Since Peter Drucker coined the term "knowledge worker", learning professionals have been moving away from the silo of the training department and have become more integral to the broader ongoing development of the primary assets organisations possess: its people and their expertise.

In 1959 Peter Drucker coined the term “knowledge worker” to describe

one who works primarily with information or one who develops and uses knowledge in the workplace. It is performed by subject-matter specialists in all areas of an organisation;

(1973, p.839)

their tools are the knowledge assets they use in an organisation. It is “generally accepted” (Drucker, 2006, p.165) that the knowledge workers’ expertise in their role is the starting point for enhancing productivity, quality of work, and performance. If knowledge workers are to continue contributing to an organisation, their knowledge must remain up-to-date.

With the range of learning technologies and content delivery channels now available, learning professionals are meeting the needs of learners more successfully than ever before. They have done so adapting to the changing nature of organisations, which has meant increasing the breadth and depth of the functions undertaken by learning professionals.

My belief is that the responsibility of learning professionals encompasses these areas:

Knowledge Management

• Information Repository Development (Formal CMS solutions as well as non-formal wikis, blogs & podcasts)
• Content Management / Architecture Leadership
• Communities of Practise
• Knowledge Networks
• Experts & Expertise

Formal Instruction

• E-Learning
• Instructor-led Training
• Online Mentoring
• Certification

Organisational Development

• Workplace Learning & Support
• Performance Support
• Informal Learning Environments
• Non-formal Learning (InfoSession-type events)
• Formal Instruction (Classroom-based / Web-based)

Performance

• Ongoing workplace-related development

...and growing.

Interestingly, these domains of expertise align closely to Kirkpatrick's Four Levels of Evaluation, where (for example) Formal Instruction in the categories I've detailed here equates with Level 2 (Learning) and Level 4 (Results) with Performance.

But we're here to talk about the Long Tail, which I think is an interesting concept to apply to learning & development, but I'm struggling with seeing how to apply it as a rule or axiom in the learning world; I would assert that there is no evidence to support the view that learning professionals / departments specifically have to support Long Tail learning as distinct from all the other modalities of learning within their sphere of influence, except in a certain context.

That context is this: as more and more digital learning content has been developed and distributed using networked channels, online learning has moved from being a net consumer of educational resources to a net producer of learning materials. As such, more types of information / knowledge / learning resources are now available than heretofore. The very existence and availability of these resources to learners ensures their continued, low-level usage over time. Very much like the odd consumer searching for that obscure title on Amazon.com, some (diminishing) number of learners (over time) will continue to access rarely-use or more likely, out-of-date learning materials. I can give you a good example of this; content developed for an earlier version of an application (i.e. Macromedia FlashMX) will still have an audience, but not a very large one; yet the content still remains available for you to find if you so wish to use it.

However, as John Hager points out in Paying Attention what "we know at any point in time has diminishing value." Taking this truism into account I would suggest that a more appropriate model to apply is the Pareto Principle upon which the Long Tail is based.

The Pareto Principle is more commonly known as the "80:20 Rule" and was originally devised as a means of quantifying distribution of income and wealth among a population (20% of a population own 80% of the wealth).

It is reasonable to say that that knowledge is a form of wealth, and knowledge workers, experts, and "More Knowledgeable Others" exemplify the distribution of knowledge in an organisation. To extend the analogy, learning professionals are the "bankers" or economists of knowledge and the role and responsibility of the learning professional should extend to distributing the acquired skills, expertise and knowledge of workers to others within the organisation. The methods and means of this scope are, like all things, reliant on the nature of the organisation the learning professional is in.

References:

Drucker, P. F. (1973) Management: Tasks, Responsibilities, Practices. New York, Harper & Row

Drucker, P. F. (2006) Classic Drucker. Boston, MA. Harvard Business School Publishing Corporation

--