Shiny new technologies used by dusty old professions

I hadn't planned on blogging about informal learning today, but an article in Irish e-zine Silicon Republic interested me, and I thought I'd bring it to you. According to the article Number crunchers find social media a ‘tweet’ surprise, members of the Institute of Certified Public Accountants (CPA) have begun using social media such as Twitter, Facebook and LinkedIn to co-ordinate continuous professional education. The CPA is the educational, representative and regulatory body for over 5000 members and students. The Institute's role is to:

• Regulate CPAs in accordance with the law and the Institute's Code of Ethics in the public interest.
• Ensure that CPAs are constantly up to date in all matters relating to their professional work.
• Maintain the highest levels of educational standards for new entrants to the profession.
• Represent the interests of members where appropriate.

The CPA’s Suzanne Shaw outlined the reason for the emergence of non-formal and informal e-learning technologies in the Institute:

As one of [the bodies] in Ireland that train accountants and regulate them throughout their professional life, our members are predominantly split three ways: practitioners; entrepreneurs; and employees of businesses.

All of them are at the coalface of the current economic climate and many of them use tools like LinkedIn, Twitter and Facebook to give one another practical advice about sustaining businesses and planning for a long-term environment. It’s a great way to get information out to people really quickly.

It seems that the CPA members are using Facebook and Twitter to share articles and information to keep each apprised of developments in their domain. Ms. Shaw again:

The beauty of social networking is it enables two-way communication or, if you want, one-to-many communication. The CPA uses it to gauge feedback on courses and products and adjust them accordingly.

One of the benefits of LinkedIn, Facebook and Twitter is that they are "ready-made." The CPA's experience is that they can concentrate on the business use of the technology without contributors having to worry about the technology per se. Despite being (by it's very nature) a very traditional organization, the obvious business advantages of using these platforms for information-sharing seems to have eased the transition to using social media. There are a number of core uses of social media for learning in the CPA:

• The CPA recently set up a space on Facebook for new students to get and share information.
• In terms of professional use, with closed LinkedIn forums are used, so information can be kept confidential between members.
• CPA accountants are using Twitter as a way of relaying information or lobbying issues.
• Professional members make use of LinkedIn to keep in touch with each other, as well as business associates.

Interestingly, one of the main drivers of the growth in utilization of social media tools is that accountants' clients are "pushing them to be more involved in online communication" according to Ms. Shaw.

It seems that once members are exposed to Web 2.0 technologies, they adapt their own information-sharing practices to include Twitter and Facebook. Ms. Shaw stated that:

Many share war stories and know-how in the forums. With CPE seminars taking place across the country, people not only meet up but can also keep in touch. Because people have hectic working lives and a home life to balance, they can’t get to every course or seminar, so they ...use these tools to share notes and find out where the next course is taking place. Not every one can make it to the centre of Dublin after a day’s work, so we’ve started uploading video lectures. Students ...are recording podcasts of lectures and sharing on places like Facebook. We estimate about 10% of our 5,500 members and students are using social media for continuous professional education. With Facebook, for example, they are truly engaging with one another. Many use it because they are that generation, others have begun dabbling. It can only grow from here.

Brutus, in Shakespeare's Julius Caesar tells Cassius that

There is a tide in the affairs of men.
Which, taken at the flood, leads on to fortune

Act IV, Scene 3.

In a similar vein, I would suggest that there are trends in the uses and the adoption of technology. The current global economic environment as well as the emergence and broad adoption of easy-to-use Read/Write Web tools like Facebook, Twitter, and LinkedIn in society-at-large are profoundly re-shaping the ways people and organizations communicate.

As we know from Rogers' writings on diffusion of innovation, people’s attitude toward a new technology is a key element in its diffusion. Roger’s Innovation Decision Process theory asserts that innovation diffusion is a process that occurs over time through five stages:

1. Awareness
2. Interest
3. Evaluation
4. Trial
5. Adoption

The final phase of the diffusion process is characterized by large-scale continued use of the idea or technology, and by "satisfaction with" (Diffusion of Innovations, 2003, p.2) the idea. This does not mean that the the individual or organization that has accepted the idea will use it constantly, rather, it means that the diffused idea has been integrated into their schema or metal model as a valuable asset or resource.

Figure 1 Diffusion of Innovation curve
[Click to enlarge]

Individuals or organizations will typically go through these processes at varying speeds, depending on factors ranging from the cost, time, and effort required to implement the diffused concept, the return on the investment, how well it aligns with their previous experience with similar concepts, as well as the complexity of the idea or technology under consideration. By endorsing and supporting a range of well-tested, free-to-use solutions, that are currently very positively received in the public consciousness due to their apparent ability to elect presidents (Obama), overthrow despotic regimes (Obama again for Dubya Iran), and circumvent traditional media channels (Michael Jackson's death). Such momentum is hard to ignore, especially when coupled with the economic imperative of clients demanding access to CPA members' skills via social media.

However, a corollary to the curve described in Figure 1 (above) is the Gartner Hype Lifecycle illustrated in Figure 2 (below).

Figure 2 Generic Gartner Hype Cycle
[Click to enlarge]

I would suggest that Twitter, Facebook etc are well on their way to reaching what Gartner describes as the "Peak of Inflated Expectations" associated with this type of innovation. It remains to be seen if the CPA can take this flood in the tide of technology and progress their non-formal learning initiatives, or if they will be "bound in shallows and in miseries" if they are unable to leverage the potential of this phenomenon. Personally, I'd like to see more evidence of planning and strategy behind their informal learning activities. Based on the information in the article, it looks to me that new learning channels are added in an ad hoc fashion. Nothing wrong with this: it's one of the advantages of a non- or informal approach to e-learning, but I would suggest that too much of a 'make it up as we go along' approach can lead to spreading finite resources too thinly for any of them to be truly effective.

___________

References:

Kennedy, J. (2009). Number crunchers find social media a ‘tweet’ surprise. Silicon Republic. [Internet] 29 June. Available from: http://www.siliconrepublic.com/news/article/13271/ [Accessed 29 June 2009]

Rogers, E. M. (2003) Diffusion of Innovations, 5th ed.. Simon & Schuster International.

--

Informal Workplace Learning: KMs and virtual environments

Concluding our overview of influences which are contributing to the emergence of informal and non-formal learning in the workplace. Yesterday, I discussed three of the five change factors influencing how learning activities are being implemented in the workplace:

1. Blended learning
2. Talent management
3. Web / Learning 2.0
4. Knowledge centers
5. Immersive learning simulations (serious gaming).

Today, I'm going to look at the remaining change factors.

Knowledge Centers
A knowledge center provide a central, just-in-time location for information. It is a single-focused interface with information about a specific topic, like IT applications, sales, or project management. A knowledge center is often a portal or part of the company intranet (like Moodle or SharePoint), or it is the knowledge management component of an LMS like BlackBoard. Knowledge Centers are where information workers go for FAQs, the latest tips, tricks and how-to's, links to external resources, relevant information, and other resources about a specific topic that may not be directly related to a formal course (see Figure 1).

Figure 1 Employees Find Help In A Knowledge Center

Sites must be useful, up-to-date, and responsive to employee needs. An organization must decide on the most important knowledge center topics, create a user-friendly site, and populate it with information. A designated person must be responsible for maintaining the knowledge center portal and identifying new information and material for the site. Survey employees about their needs and provide a feedback box on the site.

Immersive Learning Simulations: Serious Games
Simulations are models of the real world. Add pedagogically sound content, learner interaction and involvement, and a gaming element — all focused on achieving a learning goal — and the result is an immersive learning simulation (ILS).

The challenge is getting the right mix of learning and gaming. Today’s most immersive learning simulations are custom creations from a specialized vendors, and they are more obviously expensive than traditional e-learning content development. Achieving the right balance between the gaming and learning components takes expertise from three professionals: a subject matter expert, an instructional designer, and a game developer, all working closely together.

As more immersive 3D social networking environments like Second Life (2L) become available, serious gaming will become a more common modality for non- and informal learning. There are some problems associated with the environment, however. According to Bill Brandon (2007, p.7)

for typical non-students (employees) sitting through a slide-presentation-and-lecture in Second Life is no more effective (and likely less so) than sitting through the same dog and pony show in a physical space.

With this in mind 2L does have potential; e-learning practitioners can use it to create effective constructivist learning environments. For example, if you have a requirement similar to this for sales training, negotiation skills, diversity training, or any other communication- based skill, Second Life might be all you need. Interactive learning activities (like simulations, immersive learning, and gaming) put employees in virtual yet authentic situations, where they can learn the skills and materials while also making decisions and getting immediate feedback on their choices.

The transformation of workplace learning means that organizations must develop informal learning as a resource for employees. As the younger workforce becomes a more significant part of organizations and moves into management positions, these people will demand different kinds of learning opportunities and tools to get to learning resources quickly.

Figure 2 VLE on 2L

They expect to control much of their own learning, so learning professionals must offer an range of useful up-to-date resources inside and outside the organization for information, knowledge, and training. If employees don’t see these learning resources as part of their organization's talent management and development programs, the chances are that they will move to more progressive companies that will acknowledge their learning and career development needs.

__________________

References:

Brandon, B. (2007) ‘Give Your e-Learning Some (Second) Life: Simulation Made Easy.’ eLearning Guild. [Internet] Available from: http://www.elearningguild.com/articles/abstracts/index.cfm?action=viewonly2&id=246 (subscription required). Accessed 19 March 2009

Schooley, C. (2008) Informal Methods Challenge Corporate Learning [Internet] Available from: http://www.forrester.com Accessed 12 March 2009

--

Informal Workplace Learning: paradigm changes - more

I'm looking at some influences which are contributing to the emergence of informal and non-formal learning in the workplace. In my previous post on this topic, I outlined five change factors:

1. Blended learning
2. Talent management
3. Web / Learning 2.0
4. Knowledge centers
5. Immersive learning simulations (serious gaming).

I will discuss the first three of these today.

Multiple-Channel learning or ‘blended’ learning is not a new learning approach, but it is becoming more common within training programs. Multiple-channel learning uses a range of learning modalities to train learners, as appropriate to the content and the context (see Figure 1). For example, a learning module might include some classroom learning, followed by some self-paced e-learning, with periodic online virtual classroom sessions for discussions. When learners reach a level of competence, they could be assigned mentors who work with them during their on-the-job training period.

Figure 1. Multiple “Blended” Learning Channels

However, no formula exists for a successful multiple-channel learning course. The question that a learning and development professional must ask is,

What instructional blend is going to work best, given the kind of students, the nature of the content, and the skills these people need?

Sometimes blended learning includes just two modalities, like self-paced online learning along with virtual classroom for discussion and interaction. Sometimes a face-to-face component is central, with synchronous and asynchronous instruction, mentoring, or community of practice used for review, collaboration, and support.

Increasingly, employers struggle to attract and retain the best talent. Enter Human Capital Management. Organizations need the ability to seamlessly assess worker competencies and to provide employees with learning experiences that will close knowledge gaps and enhance job performance. Learning doesn’t stand by itself any longer — it’s integral to the other components of human capital management (see Figure 2).

Figure 2. The Four Pillars Of Strategic Human Capital Management

But unified learning and talent management systems don’t necessarily integrate. Many organizations have embedded legacy point solutions for learning, performance, and compensation. These solutions do not integrate well, and this inhibits seamless movement of information. For example, once a manager conducting a performance review identifies one or more employee competency gaps, the manager should then have the ability (through integrated applications) to assign learning directly to an employee learning plan that will help close the gap.

Learning 2.0 is a trend in which employees take more control of their learning and knowledge gathering. They decide what they need, where to go, and how to find information from a variety of resources both inside and outside of their corporate intranet (see Figure 3).

Figure 3. Employees Drive Their Own Learning

To facilitate this, workers must have robust tools like expertise location, search, and instant messaging (IM) available to them, and they must know how to make the best use of these tools. A new mind set is required for Learning 2.0 to succeed. Many business leaders (and indeed HR departments) think of learning as classroom courses, even though most are beginning to grudgingly accepted e-learning as appropriate for certain kinds of training. Many people – especially those from a traditionalist or pre-digital era - are still skeptical about the place of informal tools in the work environment. The thinking goes:

Dollars allocated to learning are for formal courses, even though learning after formal college education takes place informally and non-formally.

Worse again, in certain “toxic” workplace environments individual or small group engage in the unsavory activity of empire building - where they attempt to gain control over key projects and initiatives in order to maximize their job security and promotability.

Empire builders hoard credit and prestige for projects, and knowledge-sharing is anathema to them. This approach prevents other people in the organization from contributing in a meaningful manner, and alternative or competing projects to address the project's goals are destroyed regardless of their merit. The outcome of empire building is, inevitably, that the organization suffers as a whole, projects fail, and organizational goals are achieved only partially, inadequately, or not at all. This sort of behavior is very common.

The tender shoots of non-formal learning will inevitably be destroyed by the weeds of empire building. For non- and informal learning initiatives to be implemented successfully, environment of sharing, where employees are expected to share rather than hoard information must be formed.

More...

_________________

References:

Schooley, C. (2008) Informal Methods Challenge Corporate Learning [Internet] Available from: http://www.forrester.com Accessed 12 March 2009

Informal workplace learning – influences and change factors

Many commentators, including Stern and Sommerlad (1999) assert that informal and non-formal learning

…have acquired visibility and saliency [because they] sit at the juncture of new thinking concerning the nature of learning about new forms of knowledge, about the transformation of the nature of work and about the modern enterprise in a globalized economy

(cited in Fuller & Unwin, 2002, p. 95).

As a concept and set of practices, ‘workplace learning’ has entered a period of political, economic and social transformation. Advances in technology, the demise of manufacturing industries and the growth of service sector industries, have led to changes in the meaning of the ‘workplace’ (for example, home-working (Felstead et al, 2000), working on-the-move (Felstead et al, 2005), ‘flexible’ working (Felstead et al, 1999); concomitantly, the shift towards new, post-industrial style workplace structures and practices,

have led to a new set of concept and practices surrounding ‘workplace learning.’ These include:

• different workplace contexts
• different workplace knowledge-sets
• different workers

to those of the past.

Workplace learning used to occur in the classroom or via online ‘e-training courses.’ While this style of learning is still important for regulatory or compliance training, a strong need is emerging for informal learning that is more closely integrated with employee work. Examples include:

• Some categories of business and technical training
• Procedural learning
• Deeper learning that requires concept development and interaction

Here are some reasons why this shift is happening:

Information / cognitive overload is affecting all workers. We can’t store all the facts, details, and data we need to do our jobs today, and more information is created every year.1 Thus, an important new skill is the ability to search effectively to find the information you need when you need it. It might even be information presented in a course that you took online a year ago.

Immediacy of information is critical in today’s workforce. With today’s fast business pace and emphasis on speed to market, employees may need to access a particular 5-minute piece of learning that will get them to the next step fast.2 This means that learning must come in smaller chunks that are only a click away. It also means that information and learning tend to blur in the work environment.

The Internet generation brings a different work style. The computer-savvy, 20-something ‘digital natives’ (Prensky, 2000) are very good at using technology to find what they want. They are impatient and want to access information resources quickly, and they assimilate and connect this information to their work. This workers prefers to drive personal learning, rather than simply receive information from an instructor.

Learning and development professionals need to begin working with lines of business outside of the traditional HR / Training orbit to ensure that the organizations provide a variety of formal and informal learning opportunities for employees. In some cases, the training department within HR organizes both formal and informal learning. In other contexts, HR handles formal learning, and individual lines of business handle the informal learning related to their specific activities (with consultation from learning professionals).

As well as these organizational changes, learning professionals must understand five key emerging trends:

1. Blended learning
2. Talent management
3. Web / Learning 2.0
4. Knowledge centers
5. Immersive learning simulations (serious gaming).

More...

_____________________

References:

Felstead, A. and Jewson, N. (2000) In Work, At Home: Towards an Understanding of Homeworking, London: Routledge.

Felstead, A., Jewson, N. and Walters, S. (2005) Changing Places of Work, Basingstoke: Palgrave Macmillan.

Fuller, A. and Unwin, L. (2003) Learning as apprentices in the contemporary UK workplace: creating and managing expansive and restrictive participation, Journal of Education and Work, 16:4, pp. 407-426.

Lee, T. Fuller, A., Ashton, D., Butler, P., Felstead, A., Unwin, L., & Walters, S. (2004) Learning as Work: Teaching and Learning Processes in the Contemporary Work Organisation,
Workplace Learning: Main Themes & Perspectives Learning as Work Research Paper, No. 2.

Prensky, M. (200) Digital Natives, Digital Immigrants. [Internet] Available from:http://www.marcprensky.com/writing/Prensky%20-%20Digital%20Natives,%20Digital%20Immigrants%20-%20Part1.pdf Accessed 21 October 2007

Stern, A. and Sommerlad, E. (1999) Workplace Learning, Culture and Performance. Institute of personnel and Development, London.

--

Informal and Non-formal Workplace Learning 2

Looking specifically at learning in the workplace, Michael Eraut in Non-formal learning, implicit learning and tacit knowledge (2000) makes a clear distinction between his interpretation of the term ‘non-formal learning’ and what others including Scribner (1988), Conner (2002) and Cross (2003) would call ‘informal learning’ – what Eraut terms ‘incidental learning’ or

the acquisition of knowledge independently of conscious attempts to learn and the absence of explicit knowledge about what was learned’

(Reber, 1993, quoted by Eraut 2000, p.12)

This is, he argues, because most workplace learning takes place outside formal learning contexts, and informal learning carries with it connotations of

so many other features of a situation, such as dress, discourse, behavior, diminution of social differences – that its colloquial application as a descriptor of learning contexts may have little to do with learning per se.

(p.12)

Not only does the term carry unwanted and confusing implications, but it is too vague to be of any real utility. For Eraut, an analysis of learning must focus on activities and the outcomes that that contribute to significant changes in capability or understanding. In a sense, Eraut does not define non-formal learning; rather, he defines the characteristics of formal learning (p.12) as:

1. A prescribed learning framework
2. An organized learning event or package
3. The presence of a designated teacher or trainer
4. The award of a qualification or credit
5. The external specification of outcomes.

The implication of this categorization is that any learning that does not exhibit all of these characteristics should be classed as non-formal. Some reviewers (Colley, Hodkinson & Malcolm, 2002) make the point that Eraut does not make clear what the status is of learning in situations that meet some, but not all, of his ‘formal’ criteria. My interpretation of his characterization is that the very nature of a formal activity -

following or according with established form, custom, or rule

(Merriam-Webster Online, 2007)

validates Eraut’s description.

More…

____________

References:
Colley, Hodkinson, Malcolm (2002) non-formal learning: mapping the conceptual terrain. a consultation report [Internet] Available from: http://www.infed.org/archives/e-texts/colley_informal_learning.htm [Accessed 28th January 2009]
Eraut, M. (2000) Non-formal learning, implicit learning and tacit knowledge, in F. Coffield (Ed) The Necessity of Informal Learning: Policy Press. Bristol

Informal and Non-Formal Workplace Learning

One of the central components of the impact of learning (and specifically the development of information workers’ expertise in organizations) is the context within which the learning takes place. A central pillar of this discussion is the type or format of the learning taking place. In the literature, it is apparent that a dichotomy exists between the paradigms of formal, goal-directed training programs and informal – “learning at the watercooler” (Grebow, 2002) or what Michael Eraut (2000) describes as incidental learning that takes place almost as a side effect of work:

it is difficult to make a clear distinction between formal and informal learning as there is often a crossover between the two

(McGivney, 1999, p.1).

Another complexity in the discussion is where is non-formal learning located in relation to the diametric opposites? For much of the forty years since the term ‘non-formal learning’ was first coined (Coombs, 1968, p.1.) there has been a great deal of debate in the literature as to the nature of formal, informal and non-formal learning; the components of each of the paradigms, their boundaries and their overlaps. The locus of this debate is centered on arguments for “the inherent superiority of one or the other” (Colley, Hodkinson & Malcolm, 2002, p.2).

I support Alan Rogers’ (2004) view that a “new paradigm” for learning exists, in which “most programs [are] partly formal and partly informal” going from formal to informal and from informal to formal in both directions along a continuum (see Figure 1) . “Both forms of education are important elements in the total learning experience” (Looking again at non-formal and informal education - towards a new paradigm, 2004).

Figure 1 the Learning Continuum

Similarly, Hodkinson & Hodkinson argue that focusing on the extent to which learning is planned and intentional may be a way of by-passing the distinction between formal, non-formal and informal altogether.” (Colley, Hodkinson & Malcolm, 2002).

More to follow…

__________

References:
Colley, H., Hodkinson, P., & Malcolm J. (2002) Non-formal learning: mapping the conceptual terrain. a consultation report [Internet] Available from: http://www.infed.org/archives/e-texts/colley_informal_learning.htm [Accessed 28th January 2009]
Coombs, P. (1968) The World Educational Crisis, New York, Oxford University Press.

Eraut, M. (2000) Non-formal learning, implicit learning and tacit knowledge, in F. Coffield (Ed) The Necessity of Informal Learning: Policy Press. Bristol

Grebow, D. (2002) At the Water Cooler of Learning [Internet] Available from: http://agelesslearner.com/articles/watercooler_dgrebow_tc600.html [Accessed 30th February 2009]

McGivney, V. (1999) Informal learning in the community: a trigger for change and development NIACE. Leicester.

Rogers, A. (2004) Looking again at non-formal and informal education - towards a new paradigm [Internet] Available from: http://www.infed.org/biblio/non_formal_paradigm.htm [Accessed 30th January 2008]

--

Characteristics of Informal Learning

Changes in the world economy are forcing corporations to rethink how workers learn and to perform effectively. How do people learn? Why? What accounts for the upswing in interest in less formal learning? Does it work?

In the corporate context, learning is about mastering technical and social skills, and product knowledge. The focus is on attaining the skills. knowledge, and expertise required to meet the promise made to the customer.

In an interview in 2005, the estimable Jay Cross articulated a concept that many (including myself) felt was an emerging trend in corporate learning and development:

Well, I had to redefine all learning …because the world is changing so fast. The concepts we had when knowledge was fixed in place, like something you could put in a library, don’t work anymore. So I look at all learning as adaptation to the communities that matter to you, to your ecosystems, if you will. Informal Learning is simply that, which is not directed by an organization or somebody in a control position.

(Interview with Jay Cross: Informal Learning)

The year 2005 heralded the recovery from the Dot-Com Crash in 2001, a year Kevin Kruse has described as one that

...brought the harsh, steep slope of unfulfilled promises. Several high-profile [e-learning] providers shut their doors while many more announced large-scale layoffs in the face of missed revenue targets and crashing stock prices. E-learning advocates retreated to the more defensible ground of "blended learning. This year [went] down as the Trough of Despair.

As a result of this turn-of-the-century disillusionment, a key document on lifelong learning published by the European Commission in the same year went unnoticed by many training professionals. In their 2001 document Communication on Lifelong Learning, the authors Holford, Patulny & Sturgis defined the terms formal, non-formal and informal learning (p.9):

Table 1 Definition of learning types

Learning Type	Description
Formal Learning	Learning typically provided by an education or training institution, structured (in terms of learning objectives, learning time or learning support) and leading to certification. Formal learning is intentional from the learner’s perspective [my italics].
Non-formal Learning	Learning that is not provided by an education or training institution and typically does not lead to formalized certification. It is, however, structured (in terms of learning objectives, learning time or learning support). Non-formal learning is intentional from the learner’s perspective [my italics].
Informal Learning	Learning resulting from daily life activities related to work, family or leisure. It is not structured (in terms of learning objectives, learning time or learning support) and typically does not lead to certification. Informal learning may be intentional but in most cases it is non-intentional (or “incidental”/ random) [my italics].

More next time...

___________

References:

Cross, J. (2004) An informal history of eLearning. On the Horizon [Internet] 12(3). pp.103-110. Available from: http://www.emeraldinsight.com/Insight/viewPDF.jsp?Filename=html/Output/Published/EmeraldFullTextArticle/Pdf/2740120301.pdf (Subscription required) Accessed 20th February, 2007

Holford, J. Patulny, R. & Sturgis, P. (2005) Indicators of Non-formal & Informal Educational Contributions to Active Citizenship. A Paper Prepared for the European Commission by the University of Surrey. [Internet]. Available from: http://farmweb.jrc.cec.eu.int/CRELL/active_citizenship.htm Accessed 25th October, 2006

Kruse, K. (2002) The State of e-Learning: Looking at History with the Technology Hype Cycle. [Internet] Available from: http://www.e-learningguru.com/articles/hype1_1.htm [Accessed 12th February 2008]

--

Is informal learning this year's L&D Rubik's cube?

According to Josh Bersin (Informal Learning becomes Formal):

Clearly we have reached an inflection point. Where “e-learning” was the big craze in corporate training in the early 2000’s, and “blended learning” was the craze in 2003 and 2004, today, thanks to the slowing economy and the widespread availability of social networking and online wikis and portals, “informal learning” is the next big thing.

He continues:

And best of all, an informal learning strategy saves money. By empowering people to publish their expertise and learn from each other, you can cut spending on content development, external content, and formal training - focusing your energies on the “upper right” training programs in your organization. [his italics]

I've nothing against crazes.

Take The Watchmen. I've been a Watchmen fan since it was first published as a serialized graphic novel back in the mid-Eighties. I think it's fantastic that the recent movie has brought Alan Moore's magnum opus to a whole new audience.

I'm sure I'll be equally pleased when the movie version of The Ballad of Halo Jones and D.R. & Quinch are released, and there’s even more appreciation of the quality of Mr. Moore's work. Oh... you haven't heard of those then?

Here's first the thing about crazes - the objects at their center have usually been around for a very long time before they enter the public consciousness. The Watchmen was first published in 1986. It is a troublesome work in many ways - it inverts the role of mythic archetypes (superheroes with all-to-human flaws), and it espouses a certain non-conformist approach that until recently had a value perceived to be inferior to traditional literary approaches - a "comics for kids."

Yet I would assert that it’s very awkwardness has led to its longevity (if not it’s appreciation in the mainstream culture). When it was published, it was pretty much ignored - and it would probably still be regarded as a piece of interesting cult fiction if Alan Moore hadn't gone on to write Batman: The Killing Joke, the inspiration for last year's award-winning film.

Had The Watchmen less intrinsic value before it became a revenue-generating stream for a conglomeration of media production and distribution outlets like 20th Century Fox and Warner Bros.?

No.

Here's the second thing about crazes: they usually occur in ambiguous socio-economic situations (like a recession) when people are unable to determine the appropriate mode of behavior. Making the assumption that surrounding people possess more knowledge about the situation, it can be said that individuals will deem the behavior of others as better informed. Crazes can lead to conformity of large groups of individuals in either correct or mistaken choices, a phenomenon sometimes referred to as herd behavior. Although informational social influence at least in part reflects a rational motive to take into account the information of others, analysis shows that it can cause people to converge too quickly upon a single choice, so that decisions of even large groups of individuals may reflect very little knowledge.

Here are some stats based on data from research taken from 800+ HR and L&D managers surveyed in 2008 by Bersin & Associates:

• 78% of corporate managers believe that “rapid rate of information change” is one of their top learning challenges.
• 80% of all corporate learning takes place through on-the-job interactions with peers, experts, and managers (estimated data collected from over 1,100 L&D managers late in 2008).
• Over 30% of all corporate training programs (ie. classroom or other formal programs) are not delivering any measurable value (data provided through the same survey).
• Nearly all Millenial employees (under the age of 25) expect to find an on-demand learning portal (similar to Google and YouTube) within their employer’s environment.

Now lets look at some learning strategies and outcomes closely associated with very specific on-the-job learning and professional development needs of employees and line managers in a bank.

• The sharing and exchange of knowledge, experiences, and good practices leading sometimes to the development of refined knowledge and approaches
• Analyzing and developing solutions or major modifications to ideas and practices to increase value for the Bank and for clients
• Integrating efforts across disciplines and developing joint ideas and products
• Evaluating and reflecting on acquired knowledge, developing alternatives to existing knowledge, and generating new knowledge
• Developing common frameworks, language or knowledge sets for mutual trust and joint efforts in development
• Fulfilling a social need to be generative or for self-actualization
• Increasing commitment, passion and honesty in participating in world development

These outcomes in the latter set of bullets align pretty well with the requirements of the former set, don't you agree?

I think that they do.

Would you be surprised to know that the second set of points come from a paper called An Evaluation of Non-Formal Learning in Professional Technical Networks, 2000-2001 by Sukai Prom-Jackson et al, published seven years ago in 2002?

Here's the last thing about crazes. A little knowledge is a dangerous thing.

Now, informal learning seems to have emerged as the shiny new toy. It fits so well with social networking, Web 2.0, and asynchronous media delivery platforms. It's primary value seems to be as a "money-saving strategy" (i.e. cheap), rather than for its effectiveness as a learning modality - and undertaken correctly it is a very effective approach to workplace learning.

But it's is not this year's novelty. Just like The Watchmen, it has been around for much longer than you may suspect. But you would not know it's there if you googled Informal Learning; the domain characterized as "informal learning" by Bersin & Associates (and other organizations) is more correctly called Non-Formal Learning. What's more, there is a solid body of research on the topic going back over forty years. In this context, reviewing the current crop of articles on informal learning is akin to watching people actually trying to reinvent the wheel.

Informal - non-formal - learning is a troublesome concept in many ways: it inverts the role of mythic archetypes (learners transferring knowledge and expertise outside of the context of a formal environment and without instructors), and it espouses a certain non-conformist approach that until recently had a value perceived to be inferior to traditional types of training.

Yet I would assert that it’s very awkwardness has contributed to its longevity (if not it’s appreciation in the mainstream training and development culture).

A little Learning is a dangerous thing;
drink deep, or taste not the Pierian spring:
there shallow draughts intoxicate the brain,
and drinking largely sobers us again.

Pope, A. An Essay on Criticism, 1709

More next time...

______________________

References:

Bersin, J. (2009) Informal Learning becomes Formal. [Internet] Available from: http://joshbersin.com/2009/01/21/informal-learning-becomes-formal/ Accessed 15 March 2009

Pope, A. (1709) An Essay on Criticism.

Prom-Jackson, S., Bina Palmisano, M., Kategile Jackson, W., Novojilov, R., & Tena, M. (2002) An Evaluation of Non-Formal Learning in Professional Technical Networks, 2000-2001. WBI Evaluation Studies No. EG03-61, The World Bank Institute, Washington, DC.

--

M-Learning via the iPhone 4 – some approaches and technologies

Last time, I investigated mobile learning (m-learning) in the context of its parent domain e-learning and I outlined some types and characteristics of devices that enable the delivery of m-learning to users. In this post, I will look at the some of the learning theories that support learning distributed via mobile devices.

Now read on…

In their 2004 paper Literature Review in Mobile Technologies and Learning (Naismith, Lonsdale, Sharples & Vavoula), consider the importance of taking an approach to m-learning that

moves away from the dominant view of mobile learning as an isolated activity to explore mobile learning as a rich, collaborative and conversational experience, whether in classrooms, homes or the streets of a city. …how we might draw on existing theories of learning to help us evaluate the most relevant applications of mobile technologies.

(p.1)

Table 1. Applying Learning Theories to M-Learning (after Naismith et al, 2004)

Learning Theory	Activity Type	Description
Behaviorist	Activities that promote learning as a change in learners’ observable actions	In the behaviorist paradigm, learning is thought to be best facilitated through the reinforcement of an association between a particular stimulus and a response. Applying this to educational technology, computer-aided learning is the presentation of a problem (stimulus) followed by the contribution on the part of the learner of the solution (response). Feedback from the system then provides reinforcement.
Constructivist	Activities in which learners actively construct new ideas or concepts based on both their previous and current knowledge	In the constructivist approach, learning is an active process in which learners construct new ideas or concepts based on both their current and past knowledge. Learners are encouraged to be active constructors of knowledge, with mobile devices now embedding them in a realistic context at the same time as offering access to supporting tools. The most compelling examples of the implementation of constructivist principles with mobile technologies come from a brand of learning experience termed ‘participatory simulations’, where the learners themselves act out key parts in an immersive recreation of a dynamic system.
Situated	Activities that promote learning within an authentic context and culture	Situated learning posits that learning can be enhanced by ensuring that it takes place in an authentic context. Mobile devices are especially well suited to context-aware applications simply because they are available in different contexts, and so can draw on those contexts to enhance the learning activity. The museum and gallery sector has been on the forefront of context-aware mobile computing by providing additional information about exhibits and displays based on the visitor’s location within them.
Collaborative	Activities that promote learning through social interaction	Collaborative learning has sprung out from research on Computer-Supported Collaborative Work and Learning (CSCW/L) and is based on the role of social interactions in the process of learning. Many new approaches to thinking about learning developed in the 1990s, most of which are rooted in Vygotsky’s socio-cultural psychology (Vygotsky 1978), including activity theory. Though not traditionally linked with collaborative learning, another theory that is particularly relevant to our consideration of collaboration using mobile devices is conversation theory (Pask 1976), which describes learning in terms of conversations between different systems of knowledge. Mobile devices can support Mobile Computer Supported Collaborative Learning (MCSCL) by providing another means of coordination without attempting to replace any human-human interactions, as compared to say, online discussion boards which substitute for face-to-face discussions (Zurita et al 2003; Cortez et al 2004; Zurita and Nussbaum 2004).
Informal and non-formal	Activities that support learning outside a dedicated learning environment and formal curriculum	Research on informal and lifelong learning recognizes that learning happens all of the time and is influenced both by our environment and the particular situations we are faced with. Informal learning may be intentional, for example, through intensive, significant and deliberate learning ‘projects’ (Tough 1971), or it may be accidental, by acquiring information through conversations, TV and newspapers, observing the world or even experiencing an accident or embarrassing situation. Such a broad view of learning takes it outside the classroom and, by default, embeds learning in everyday life, thus emphasizing the value of mobile technologies in supporting it.
Learning and teaching support	Activities that assist in the coordination of learners and resources for learning activities	Education as a process relies on a great deal of coordination of learners and resources. Mobile devices can be used by teachers for attendance reporting, reviewing student marks, general access of central school data, and managing their schedules more effectively. In higher education, mobile devices can provide course material to students, including due dates for assignments and information about timetable and room changes.

As yet there is no comprehensive ‘grand theory of mobile learning’ - nor do I expect that there will be any time soon. As I move forward through this series of posts, one of the areas I will consider is integrating pedagogy for the use of mobile devices that in a number of areas. In much the same way as we have many categories of devices, we will discover that there are many ways to integrate learning on mobile devices using a number of instructional designs, developmental approaches and delivery models. I support the view that one of the great strengths of m-learning (and indeed e-learning) is it's facility to transcend traditional learning environments like the classroom or training center, and to combine different elements in ways that are appropriate to the learning activities to be supported.

More…

______________________

References:

Cortez, C., Nussbaum, M., Santelices, R,. Rodríguez, P., Zurita, G., Correa, M. and Cautivo, R. (2004) Teaching science with mobile computer supported collaborative learning (MCSCL). Proceedings of the 2nd International Workshop on Wireless and Mobile Technologies in Education. JungLi, Taiwan: IEEE Computer Society, 67-74

Naismith, L., Lonsdale, P., Vavoula, G. and Sharples, M. (2004) Literature Review in Mobile Technologies and Learning. NESTA Futurelab Series, Report 11. [Internet] Available from: http://www.futurelab.org.uk/research/lit_reviews.htm Accessed 15 February 2009.

Pask, AGS (1976) Conversation Theory: Applications in Education and Epistemology. Amsterdam and New York: Elsevier

Vygotsky, L. S. (1978) Mind in society. Edited by Cole, M. John-Steiner, V. Scribner, Souberman, E. Cambridge, MA, Harvard University Press

Zurita, G, Nussbaum, M and Sharples, M. (2003) Encouraging face-to-face collaborative learning through the use of hand-held computers in the classroom. Proceedings of Mobile HCI 2003, Udine, Italy: Springer-Verlag, 193-208

Zurita, G., Nussbaum, M (2004) Computer supported collaborative learning using wirelessly interconnected hand-held computers. Computers & Education, 42(3): 289-314

--

To-Learn Lists: September 2008 Learning Circuits Blog Big Question

This month's Big Question emerged from a short essay written by James Collins in LEARNING JOURNEYS Top Management Experts Share Hard-Earned Lessons on Becoming Great Mentors and Leaders (2000). The author wrote:

A true learning person also has a “to-learn” list, and the items on that list carry at least as much weight in how one organizes his or her time as the to-do list.

More specifically, The Big Question is about:

• If you have a to-learn list and are willing to share, and willing to share how you work with that list, that would likely be helpful information.
• As Knowledge Workers, work and learning are the same, so how does a to-learn list really differ from a to-do list? How are they different than undirected learning through work, blogging, conferences, etc.?
• Are to-learn lists really important to have? Are they as important as what Jim Collins tells us?
• Should they be captured? If so how?
• How does a to-learn list impact something like a Learning Management System in a Workplace or Educational setting?
• What skills, practices, behaviors do modern knowledge workers need around to-learn lists?

So it's really a bunch of Big Questions this month. I'm not going to answer any of them, as I'm not so conceited as to think that you haven't better things to do with your time, than to think about what I do with my time. However, the topic did get me thinking...

First of all, some context - I undertook some research on the text that the idea of the To-Learn List originated from. Learning Journeys contains 37 two- to three-page essays by

some of the best and most well known of human resources and organizational leaders and pioneers. From Stephen Cover to Dave Ulrich, from Marshall Goldsmith to Robert Eichenger, these individuals have done much to shape current thought in the areas of organizational development and leadership.

Amazon.com reviewer

As such, the text is orientated towards the needs of HR and organizational development people, rather than learning & development professionals. No harm: it's always useful to get a different perspective (or to put it another way - a friend of mine reads a certain national newspaper because he "...likes to know what the enemy is thinking!"). One of the elements of the text that became apparent to me (from reading a part of the text using Amazon.com's Look Inside functionality) was that the learning perspectives discussed would probably work better for people who are not professionally involved with training others: in this capacity, it indicated some potentially useful approaches on how to engage others in continuous learning, and pointed towards some avenues that might assist in achieving this objective.

As discussed in a previous blog entry on the E-Learning Curve, one of the central components of the impact of learning (and specifically the development of knowledge workers’ expertise in organizations) is the context within which the learning takes place. A central element of this context is the type or format of the learning taking place. In the literature, it is apparent that a dichotomy exists between the paradigms of formal, goal-directed training programs, and informal “learning at the watercooler” (Grebow, 2002); what Michael Eraut (2000) describes as incidental learning that takes place almost as a side effect of work: “it is difficult to make a clear distinction between formal and informal learning as there is often a crossover between the two” (McGivney, 1999, p.1). Another complexity in the discussion is where is non-formal learning located in relation to the diametric opposites?

I support Alan Rogers’ (2004) view that a “new paradigm” for learning exists, in which “most programmes will be partly formal and partly informal” going from formal to informal and from informal to formal in both directions along a continuum (see Figure 1) . “Both forms of education are important elements in the total learning experience” (looking again at non-formal and informal education - towards a new paradigm, 2004).

Figure 1 the Learning Continuum

Similarly, Hodkinson and Hodkinson argue that focusing on the extent to which learning is planned and intentional may be a way of by-passing the distinction between formal, non-formal and informal altogether.” (Colley, Hodkinson & Malcolm, 2002).

So, to return to the topic, my To-Learn List has one entry: to learn.

I would categorize myself as a "learnivore" - I continually acquire new knowledge and information through my Web-, book-, podcast-, and presentation reading, blogging (reading and writing), academic study and research, and work-based learning-related tasks. These activities are drivers for the information I try to take on board in my attempts to enhance my skills, abilities, and expertise.

_____________

References:

Colley, Hodkinson, Malcolm (2002) non-formal learning: mapping the conceptual terrain. a consultation report [Internet] Available from: http://www.infed.org/archives/e-texts/colley_informal_learning.htm [Accessed 28th January 2008]
Coombs, P. (1968) The World Educational Crisis, New York, Oxford University Press.

Eraut, M. (2000) Non-formal learning, implicit learning and tacit knowledge, in F. Coffield (Ed) The Necessity of Informal Learning: Policy Press. Bristol

Goldsmith, M. Kaye, B. Shelton, K (eds.) (2000) LEARNING JOURNEYS Top Management Experts Share Hard-Earned Lessons on Becoming Great Mentors and Leaders. Davies-Black Publishing

Grebow, D. (2002) At the Water Cooler of Learning [Internet] Available from: http://agelesslearner.com/articles/watercooler_dgrebow_tc600.html [Accessed 30th January 2008]

McGivney, V. (1999) Informal learning in the community: a trigger for change and development NIACE. Leicester.

Rogers, A. (2004) Looking again at non-formal and informal education - towards a new paradigm [Internet] Available from: http://www.infed.org/biblio/non_formal_paradigm.htm [Accessed 30th January 2008]

--

May 08 Learning Circuits Big Question: Learning design differences for Digital Natives?

This month' LCBQ is:

Learning design differences for Digital Natives?:

1. Do you believe that we have to design, develop and deliver instruction differently for the so-called Digital Natives?
2. Are there differences in learning expectations and styles or can we just design good instruction and know that it meets all generational needs?
3. If you have an audience that includes natives and immigrants, how can you effectively design instruction without breaking the bank?"

In my view, we are on the threshold of a learning revolution, akin to the explosion of knowledge made possible after the general availability of printed texts following the invention of movable type and the printing press in 1440.

Now read on...

One of the first manifestations of this will be a total redefinition (or perhaps definition?) of e-learning. Right now, “E-learning” means different things to different people. When you consider that you can call the discipline “e-learning”, “elearning” or even “eLearning”, it’s no surprise that there is a range of definitions of the subject, as “we prefer to define things according to how we use them” (Morrison, 2004, p.4).

Clark and Mayer (2003) consider the “how, why and what” of e-learning: that the “e” in e-learning refers to the “how” - the course is digitised so that it can be stored. The “learning” refers to the “what” - the course includes content and ways to help people learn it, and the “why” - that the purpose is to help learners achieve educational goals or to help organisations build skills related to improve job performance (pp.13-14). The assumptions in their definition are telling: note the use of the term “course” and references to organisational skill-building – there is an inherent, unspoken claim on e-learning (what ever that is!) to be part of the corporate, human resources development arena.

Now consider Will Richardson’s Blogs, Wikis, Podcasts and Other Powerful Web Tools for Classrooms, published only three years later. In the e-learning context, a quick scan through a text’s table of contents, introduction, or index will usually orient the reader to the author’s perspective on what they think e-learning is: not in this text. The very first point Richardson makes is that “this may look like a book about technology, but it’s really a book about …connections” (p.vii). With a nod to David Jonassen, the he discusses “cool tools”, “collaborations and conversations” and “creatively motivating students to learn more deeply” using a teaching and learning “toolbox” (p.9). He’s obviously talking about e-learning, but does not see the need to define what it is. So, e-learning has become a common if imprecise term used to refer to technology-enhanced learning.

Will digital natives care?

Probably not a jot.

I'm sure they will still refer to the term, much as we digital immigrants will still differentiate between say "book learning" or "instructor-led" for example, but all our "cool tools" will become just part of the range of media channels available to learners. Naturally, their expectations of learning will be different from those prevalent at the moment, just as one-time learning tools such as audio tapes and 16mm informational films seem terribly quaint now. Similarly, even Generation Xers used electronic calculators in the classroom - using tools like these inevitably shape the expectations of the individuals who interact with these devices.

I think we can expect to see

• Enhanced more immersive Second Life-type personal learning environments
• learning on devices like the Wii (good for those kinaesthetic types)
• learning via 'personal electronic things' (PETs)

Instructional Design

I would assert that the fundamentals of instructional design won't change especially; people still learn in the same way, and as much as technology may change, we as a species are still lumbering around with our hunter-gatherer brains. At this point I should say that Marc Prensky suggests that the new technologies actually lead to physiological changes in the way the brain is 'hard-wired' due to the range of stimuli digital natives receive through interacting with new technologies, but there is no evidence to support or refute this, though it is an interesting assessment.

I think that the changes will be more in the area of content authoring and development. If technological progress in learning demonstrates the same vigour as it did during the transition from CBT to e-learning proper, it will be some years before vendors recognise the niche, and there will be a lag before easy-to-use content authoring environments are brought to market - remember how long we had to struggle with Authorware and clunky, inflexible, horrible-to-use proprietary authoring tools, before Captivate and Articulate et al were eventually released?

Having said that, the open-source community are always ahead of the curve in this domain, and there are already some quite user-friendly open source 3-D engines that could form the basis for highly interactive learning environments in a milieu that digital natives are familiar and comfortable with.

...and who knows what else the next decade may bring? If it's as extraordinary as the last ten years, it's going to be one fun ride...

References:

Clark, R. C. & Mayer R. (2003) e-learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning, San Francisco, CA: Pfeiffer

Morrison, D. (2004) E-Learning Strategies: how to get implementation and delivery right first time, Chichester: John Wiley & Sons, Ltd.

Richardson, W. (2006) Blogs, Wikis, Podcasts, and Other Powerful Web Tools for Classrooms. Thousand Oaks, CA: Corwin Press

--

Learning Evaluation and Strategy: Using an e-learning readiness survey

During this series of posts on evaluating non-formal learning programs, I have mentioned carrying out an e-learning readiness survey without characterizing or discussing how to implement such a research instrument.

This was deliberate; in my view e-learning readiness surveys represent an alpha and an omega of evaluation: on one level they are the starting point for any evaluation of an organization's learning initiative, on another level they define an organization's ability to implement an effective e-learning strategy. As such, they are a bridge between the theory and the practice of implementing a learning program.

Readiness surveys enable the learning practitioner to understand and measure both the effectiveness of organizational learning and to identify the critical factors for success when developing learning programs.

I have found Marc J. Rosenberg's E-Learning Readiness Survey to be a very effective instrument to evaluate both the effectiveness of an organization's learning strategy, and one of the foundations of any serious evaluation of the effectiveness of learning programs.

The questions are grouped into seven areas of understanding:

1. business readiness
2. the changing nature of learning and e-learning
3. value of instructional and information design
4. change management
5. reinventing the training organization
6. the e-learning industry
7. your personal commitment

The questions provided in this survey represent some of the most important strategic issues organizations face when transitioning to e learning. Certainly there are additional questions and issues that deserve attention; add your own, organization-specific items as required.

Downloads:

Marc J. Rosenberg's E-learning Readiness Survey

References:

Rosenberg, M. J. (2006) Beyond e-Learning. San Francisco, CA: John Wiley & Sons, Inc.
--

Using quantitative data in learning assessments

To support the qualitative data captured using the retrospective pre-test method, I recommend that you gather significant amount of quantitative data associated with the learning initiative or program you're evaluating.


This can seem daunting and difficult to achieve, but there are a range of accessible sources of quantitative data that every learning professional can use.

Now read on...

Always collect Level 1 attendee feedback forms from classroom-based and synchronous online participants at the end of each individual event: over time you accumulate substantial amounts of valuable feedback about learners' reaction to the initiative. Use Web-hosted feedback forms to collect data from asynchronous participants (i.e. for online and DVD-ROM based content).

Other data can be collected automatically from systems including Web servers, learning management systems and learning content management systems; Horton (2006) describes these types of archive data as “meaningful statistics” (p.102): they record detailed information about what participants did while taking the learning event, particularly in the e-learning channels. You may extend this "meaningful statistics" category to include non-automated but standardised data recording processes such as recording learner attendance captured in the synchronous online and classroom-based context, and off-line learner activity such as DVD-ROM requests. By examining logs and reports from these systems and processes useful data can be collected on:

1. Frequency and pattern of accessing the course
2. Number of slides/pages or learning objects accessed
3. Duration of access to learning objects
4. Number of supporting collateral downloads
5. Feedback submitted
6. Participation in discussion and question & answer sessions
7. Rate if individual learner attendance or online access
8. Rate of attendance or online access by role (i.e. developer, support, business analyst etc)
9. Rate of group level (i.e. Sales & Marketing, Support, Manufacturing, Engineering etc) attendance or online access
10. Rate of attendance or online access by length of service in the organisation (i.e. new hire, with the company one year,1-2 years, 2-3 years, 3-5 years, +5 years etc)

More tomorrow.

References:

Horton, W. (2006) So how is elearning different? IN: Kirkpatrick, P. & Kirkpatrick, J. Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

--

More on Kirkpatrick's Guidelines don't work for Non-Formal Learning programs

In my last posting on this topic, we discussed Kirkpatrick's guidelines for evaluating training programs (see Table 1).

Table 1 Guidelines for evaluating learning

Guidelines for evaluating learning

Use a control group if practical

Evaluate knowledge skills and/or attitudes both before and after the program

Use a paper-and-pencil test to measure knowledge and attitudes

Use a performance test to measure skills

Get a 100 per cent response

Use the results of the evaluation to take appropriate action

In this post, I will outline the difficulties encountered in applying these guidelines to non-formal learning initiatives.

Now read on...

Kirkpatick's approach is incompatible with the assessment of non-formal learning in three ways: firstly, the singular nature of each non-formal learning event means that no control group can exist to measure a difference between a control group and the “experimental group” (p.43). Similarly; a significant number of the learners that access this kind of content do so asynchronously, via a number of different learning channels and over a broad time span in a relatively ad hoc manner. This “just enough, just in time” aspect of non-formal learning is one of its strengths, but it is unrealistic to attempt to measure learning using an experimental method in such an environment.

Secondly, the lack of summative assessment in non-formal learning precludes both pencil-and-paper and performance testing to measure learning. Finally, the distributed nature of access to these events, over both time and location makes achieving a 100 per cent learner response rate (expressed by the value R) impossible. As the content (C) is always available (D) the potential (+) always exists for the current number of learners (N₁) to increment upwards (N₂). A complete learner response rate will always be a hostage to N₂ in this context.

R = R+ N₂

where C (is a known value) and N₁ (is a known value) and D (is a known value) + N₂

It follows that you can never achieve a one hundred per cent response rate. Contrast this with formal training courses. They are, by their nature, finite: each course has a specified duration. Content is delivered to learners in an instructor-led environment in a sequence of modules over the duration of the course. Once the course instruction is complete, members of any given class are required to take a comprehensive summative assessment on the learning objectives outlined in the course, usually within a set time after they complete a course to achieve certification.

Thus R = 100

where C = 10 and N₁ = 12 and D = 10

As a result, a 100 per cent response rate to knowledge and skills testing is achieved.

However, I believe that it is possible to capture and use data with the retrospective pre-test method, and I'll be discussing an approach to undertaking this activity next time.

References:

Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

--

Evaluating learning initatives: the Retrospective Pretest Design

The Retrospective Pre-test method is an interesting if controversial data collection design. I will examine its advantages and negative aspects here, as well as an argument for including this strategy in in assessing organisational learning initiatives. The retrospective pre-test method (see Table 1 is an extension of the one-shot case study design where data are gathered from participants following the learning intervention (for example, a series of non-formal Information Sessions).

Table 1 Retrospective Pre-test Design

Retrospective Pre-test Design
Groups		Non-equivalent	Intervention	Observation
Locally-based		N	X	O1	O2
Remotely-based		N	X	O1	O2
<- Time ->

However, participants (N) report on their knowledge, understanding or skills (Observation O1) after the intervention (X), and then also reflect and answer (O2) what they believe their understanding or skill was before the intervention. Rockwell & Kohn (1989) applied this method to testing the effectiveness of achieving program outcomes when interventions such as training programs are implemented and concluded that “using a post-then-pre design [retrospective pre-test] to identify self-reported behavioural changes can provide substantial evidence for program impact”. The retrospective pre-test enables researchers to reduce the response shift bias - defined by Klatt & Taylor-Powell in their 2005 paper Synthesis of Literature relative to Retrospective Pretest Design as

the change in the participant’s metric for answering questions from the pre-test to the post-test due to a new understanding of a concept being taught

- because the participants are able to give pre-test responses which are based on a post-intervention frame of reference.

Using the retrospective pre-test, response shift bias can be reduced increasing the likelihood that the observable results are due to intended intervention effects (Pratt, McGuigan & Katzev, 2001). This proposition is complicated and not without its critics – Theodore Lamb (2005), an advocate of the retrospective pre-test method, describes it as “an imperfect but useful tool.” Robson (2002, p.139-141) expresses concerns that used purely as a quasi-experimental method, this strategy could lead to an internal validity threat through regression to the mean.

References:

Klatt, J. and Taylor-Powell, E. (2005) Synthesis of Literature relative to RetrospectivePretest Design. Presentation to the 2005 Joint CES/AEA Conference, Toronto [Internet] Available from: http://www.citra.org/Assets/documents/evaluation%20design.pdf [Accessed 5th August 2007]

Lamb, T. (2005) The Retrospective Pretest: An Imperfect but Useful Tool. Evaluation Exchange. [Internet] 11(2). Available from: http://www.gse.harvard.edu/hfrp/eval/issue30/spotlight.html [Accessed 21st March 2007]

Pratt, C.C., McGuigan, W.M., & Katzev, A.R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3). 341-349.

Robson, C. (2002) Real World Research. 2nd ed. Oxford: Blackwell Publishing.

Rockwell, S. K. & Kohn, H. (1989) Post-Then-Pre Evaluation. Journal of Extension. [Internet] 27(2). Available from: http://www.joe.org/joe/1989summer/a5.html [Accessed 21st March 2007]

--