Approaches to evaluating learning

In yesterday's post I put forward some thoughts on terminal second-level examinations and the effect that faring poorly in these could have for young adults' future lives.

I received a comment on the topic from a correspondent who asserted:

...does success in exams equal education? Exams are an outmoded assessment tool - success in them means nothing more than a certificate and maybe entry into higher education - where you sit more exams until you finally move into the real world...where you realise your so called education did you a grave disservice.
Can you think? Can you create? Can you problem solve? Does an exam system fit into 21st century learning?

Now read on...

Do we have practical solutions (as opposed to high aspirations) about the best way to serve students in the 21st century?

Hard to say.

I would suggest that the best way to prepare people for the workplace is to evaluate learners as if they were in the workplace. So let's look briefly at the role of certification in this context.

Certification is the ability to prove through testing if an individual has achieved a mastery of skills, knowledge and attitude. Certification can also prove the ability of an individual to apply those skills and knowledge in specified areas and job functions.

(Certification: Corporate America’s Secret Weapon. Hilbink, P. 2004 p.2).

In 1959, Donald L. Kirkpatrick first published his four-level training evaluation model (see Table 1) in a series of articles for the US Training and Development Journal. “The reason for evaluating is to determine the effectiveness of a training program” (Evaluating Training Programs, 2006, p3). The reason for the four-level model then “was to clarify the elusive term evaluation” (2006, xv). In articulating evaluation through each of the four levels – reaction, learning, behaviour and results – the model aspires to

inspire us to look beyond our traditional classroom content delivery model and opens windows to the many way we can improve the performance of our organisations.

(2006, p.xi)

Table 1 Kirkpatrick’s Four-level Model

In the context of Kirkpatrick’s Four Level Model, second-level certification is typically interpreted at Levels Two. We can say that the most effective approach to understanding how well learners have acquired new knowledge, skills, and expertise is to ask them to demonstrate what they have learned: they sit a test. In the context of second-level education, certification assesses the learner at Level 2: Transfer of Learning. It is important to measure learning transfer because no change in behaviour can be expected unless one or more of these learning objectives have been accomplished. Measuring learning means determining one or more of the following metrics:

• What knowledge was learned?
• What skills were developed or improved?
• What attitudes were changed?
  

The benefits to conducting Level Two tests are that the learner must demonstrate that the learning transfer has occurred, and that the assessment provides verifiable and conclusive evidence that an improvement has occurred in knowledge, skills, or attitudes.

Assessment tests are a powerful tool for organisations, institutions and society-at-large, as they combine a hierarchical observation and a normalising judgement. Testing makes individuals visible (who has attained the qualification? who has not attained certification?) and enables them to be categorised (how well did they do?). Exams also normalize people by assessing them according to the same metric, and subsequently measures them in relation to one common standard.

Having established the value of Level Two assessments to understand how well knowledge has been transferred, what next? Does the education system as it currently exists meet the needs of students as they prepare for life in the Information Age, or is there a lag between national educational policies & strategies, and how students' skill need to be shaped?

Substantial resources (time, well-paid teacher, continuous training for educators etc) are required to undertake these types of evaluations, and the ministries and agencies with responsibility for managing these tasks don't seem to have the influencing powers to ensure these assets are in place.

A cynic might say that it's because such an initiative requires long-term planning. The results of such an approach mightn't be seen for five to ten years, and a politician who advocated such a spend on education might not be in a position, a decade or so later, to reap the rewards of such an innovation in national education policy.

_____________________

References:

Hilbink, P. (2004) Certification: Corporate America’s Secret Weapon [Internet] Available from: <http://www.digital-latitudes.com/docs/Cert_White_Paper.pdf> [Accessed 7 July, 2008]

Kirkpatrick, D. & Kirkpatrick, J. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

--

It's not the end of the world, you know

Today's blog post is not so much about e-learning or learning and development but rather more about education in general. Whether it's the Leaving Certificate (the final course in the Irish secondary school system), SATs in the US, O- and A-Levels in the UK, or le bac in France, I'm sure you are familiar with the day of reckoning just-finished secondary-level students are facing about now as their results are published in advance of matriculation for third-level and university places, as well as those former students who intend to go straight into employment, or take a gap year to figure out what they want to do with their lives.

Now read on...

In Ireland, the examination results were published yesterday, and today they're released in the UK; tuning in to Irish and British radio and TV broadcasts over the last twenty-four hours you would be forgiven for thinking that we're on the edge of a natural catastrophe (well, apart from the monsoon-like rain we've been experiencing for the last week, but that's another story).

You may be familiar with the format:

• Images / sounds of students opening their results envelopes and shouting for joy / groaning with disappointment
• Vox-pop of said students as they discuss their plans now that they've achieved / failed to achieve the marks they needed
• Outro from reporter which goes along the lines of "... you might not have got what you wanted, but it's not the end of the world, you know" before an short excerpt of some successful local business person or politician describing how they left school at 15 with no qualifications and worked their way up from tea-boy to head of a transnational organisation (before giving out the number of the helpline worried parents can call for advice).

While I completely understand the need to sympathize with devastated young adults who see their hopes, dreams, and career options evaporating before their eyes, I believe that to glibly state that "it's not the end of everything" simultaneously devalues the emotional and psychological impact of performing poorly in such an important life event, and provides false hope that somehow it will be all right.

Sadly, the reality is that in the 21st Century knowledge economy, a less-than-average result in these examinations seriously affects most young peoples' ability to move forward with their lives - particularly in these increasingly straitened times. As the world transitions to an Information Age where the primary asset an individual possesses is their expertise, a misstep on this lowest rung of the ladder has the potential to damage an otherwise bright, intelligent individual's potential to both contribute to, and make their way in their society.

For every business leader who "did it the hard way" or "learned from the university of life," there are ten frustrated employees working in the wrong career, and maybe ten times that number pumping gas, or just about making enough to hang in there, not really living, just existing. Unless you're in a position where you have the financial resources, the family or social contacts, or just the pure luck to break into your chosen path, you have to pretty much generate your own career based upon your abilities and talents - hopefully enhanced by what you learned in school.

I never cease to be amazed by (for example) politicians who state that through hard work and perseverance they finally got elected and rose to the position they're in now... while omitting that their father held the seat before them and they're based in a traditional constituency where the electorate has voted for the 'name they know' for generations, or the business-person who was a millionaire by thirty, through their ceaseless efforts and dedication ...and the fact that they come from a wealthy family with the resources to set them up.

I heard one such person on the radio yesterday who asserted that "all you need to be successful in your chosen career is to be focused on your goal and to be passionate about what you want to do." Try this experiment - say for example you want to be a project manager - send your résumé to as many organisations as you like, outlining your passion, enthusiasm and lack of formal qualifications in the field. Then, wait for the employment offers to roll in.

But don't hold your breath.

Passion is great, but in many cases it's the last resort of the incompetent (look at all those candidates on The Apprentice who, when about to be fired protest that they're "passionate about what they do"). Enthusiasm is an admirable quality, but certainly no substitute for expertise, ability, and experience: organisations understand this, though professional competence is boring and makes for poor reality TV.

Which one of these would YOU hire?

One of the primary reasons I take such joy in my career (and it's not even a logical reason) is that it puts me in a position to see people reach their potential. Aristotle (Nicomachean Ethics) believed that education was fundamental to the human condition - the fulfilled person was an educated person.

Perhaps this week, more than at any other time of the year his assertion that

the roots of education are bitter, but the fruit is sweet

is at it's most evocative and apposite.

_____________

References:

Aristotle. Nicomachean Ethics

_____________

Images' source:

Radio Telefís Éireann image library

BBC. The Apprentice. [Internet] Available from: http://www.bbc.co.uk/apprentice/ Accessed 14 August 2008

--

iPhone m-learning content authoring

If you're a regular reader, you'll know that I recently posted on The Mobile Segment of the Learning Marketspace - new report. In that post, I discussed the outcomes of the Elearning Guild Report on the topic.

Well, since then, I have becomes the happy owner of an Apple 3G iPhone (a screenshot of my wallpaper is displayed below) and so I've been searching for an authoring tool to create some content for delivery via iPhone.

My iPhone wallpaper - Ovation six-string acoustic guitar, and 1985 Fender Precision & 1995 Musicman Stingray basses

Now you may know that the Flash Player is not yet available for the iPhone, which leaves content developers like me in a quandary; how to deploy courseware on the device:

• for immediate implementation
• that can be refactored as SWF / FLV formatted files when Adobe and Apple agree to release a version of the Flash Player for iPhone

After quite a bit of research, I've found two authoring tools that that may enable learning professionals to create e-learning content for distribution on the iPhone, yet won't require content authors to develop the content as an extra product stream on top of their ongoing deliverables.

The two tools I am going to investigate are Techsmith Camtasia Studio, and SumTotal ToolBook Instructor 9.5 Beta. I will evaluate each product (and their associate production processes) over the next few days. In the meantime, here (courtesy of SumTotal) are a few handy hints to consider when you're at the pre-production phase of developing learning content for mobile devices:

1. Limit Graphical Content
   With the tight mobile device screen being roughly a twelfth (or smaller) of a desktop screen, large-screen graphics not only increase load times, but also take up valuable space that is needed to display text, hyperlinks, and other important information.
2. Text Considerations
   Text should be limited – brevity is the soul of wit. Layout should be structured to avoid the need for scrolling. This involves breaking up text into smaller pieces/sections and linking to those smaller pieces, instead of putting a lot of text on one screen. Smaller fonts are also recommended. For example, Microsoft Windows Mobile standard is Tahoma with a point size of eight, which is an appropriate size for text viewed in the Apple iPhone’s Safari Web Browser.
3. Limit the use of Data Entry
   Since the keyboard is a limited size, input fields should be kept to a minimum to reduce the amount of typing. Where possible, provide possible text or phrases to avoid needing the user having to type. These text items should have a hyper-link associated with them to serve as an easy way for users to select them, rather than their having to type them on the keyboard.
4. Utilize Screen Space
   Avoid excessive horizontal and vertical spacing between screen elements. Unlike designing a print ad or brochure, you don’t need a lot of “white space” on the mobile device’s screen.
5. Place Non-Essential Links at the Bottom of the Screen
   Hypertext links that are not relevant to the information being displayed should be placed at the bottom of the page to preserve space. This places critical information in the user's view upon entry to the screen or page. For example, a navigation menu that would normally be on the left or top of the screen of a desktop computer should be placed at the bottom of the screen for a mobile device.
6. Media Consideration
   Another matter of importance is the types of media – audio and video – delivered. Generally, this is specific to the mobile device. For example, Windows Mobile devices typically support MPEG and MP3 files, whereas the iPhone currently supports QuickTime and MPEG MP4.

In general, developers should consider the goal of delivery of information via a mobile device. Not all types of information, media, and training materials lend themselves to this channel. With careful planning, content developers should be able to craft and deploy a wide variety of content to help those learners who are on the go.

More...

________________

References:

Wexler, S. Brown, J. Metcalf, D. Rogers, D. and Wagner, E. (2008) MOBILE LEARNING What it is, why it matters, and how to incorporate it into your learning strategy. [Internet] Available from: http://www.elearningguild.com (Subscription required) [Accessed 21 July 2008]

--

More on Web 2.0 technologies - learning professionals' opportunities and challenges

I received an interesting communication from Virginia Yonkers in response to my July 7th post Web 2.0 technologies and learning professionals' opportunities and challenges. In my post I asserted that in theory learning professionals could start with themselves and help to develop it Web / Learning 2.0 technologies throughout organizations, in my experience, building a strategy around personal motivation and initiatives is fraught with difficulty.

Organizations typically view such innovations as the "throw it over the wall and see where it lands" approach. While learning professionals are typically highly-motivated individuals who expend personal time and effort staying "ahead of the curve" in terms of their own skills and competencies, a "viral approach" to learning in this domain can only have success if the learning professional in question is highly influential within an organization (and probably a C-level executive).

For a "footsoldier" to attempt to modify work practices within a large organization would meet high levels of resistance, particularly from managers who have no desire to change production processes that probably work very well, given the potential disruptions entailed in transitioning to a more collaborative environment.

Virginia has obviously deeply considered this topic and was good enough to respond to my post with her own experience in this area. As she wrote to me:

My current dissertation work is finding that the organizational framework and political climate has a lot to do with what and how workers learn and access knowledge. I am finding that "knowledge" is based on what departments define as knowledge. The "foot soldier" as you term it, can try to advocate all they want, but in the end it only causes tension and frustration from all sides. Changes need to be developed organically throughout the organization, aligning with organizational strategies and worker's tasks. Strictly top down and bottom up approaches don't work unless there is a shift in organizational resources for the change (human, time, financial, and support services).

Virginia further described exactly what it feels like when the level of support she had become accustomed to when her organization "changed the infrastructure without changing the support structures needed to implement the change" and how subsequently the "infrastructure within the organization" let her down: she was left facing the ire of "students, department heads, and ITS."

Similarly, she described being left in a situation where an unanticipated extra workload was left on her shoulders, as it was discovered that with the new infrastructure limited present courseware migration to the new environment, forcing teachers to contemplate re-authoring their content to be compatible with the new platform.

In my view, this example of poor communication, planning and execution of an initiative - which I'm sure we're all familiar with to a greater or lesser degree - demonstrates how a change can have unintended effects across an organization. In this informal case study, it seems that the unintended outcome was that a decision was made to implement a new system with little or no consultation for those who actually have to use the solution regularly, yet it is those "on the ground" who have to re-assemble the pieces of a once functioning solution.

Typically this  is what occurs when you  "throw it over the wall" - it breaks.

So today, I'm asking the question - how can learning professionals in organizations and institutions become involved in plans to implement learning solutions in their organizations?

We all know the short answer: get board-level support for the strategy. Which leads me to the next question: how do we go about attaining this level of support, particularly in strictly hierarchical organizations where individual contributors may not have access to decision makers?

I'll be interested to hear your stories - failures, lessons learned, and triumphs.

--