Corporate social networks, long tails, weak ties

Today's post is really a (very) mini case study in the power of integrated corporate social networks, the benefits of long tails in learning, and the strength of weak social ties.

Now read on...

If you're not familiar with the latter two concepts here's a little background: the idea of the strength of weak ties is a theory from sociology; according to its originator Mark Granovetter

the argument asserts that our acquaintances (weak ties) are less likely to be socially involved with one another than are our close friends (strong ties). Thus the set of people made up of any individual and his or her acquaintances comprises a low-density network (one in which many of the possible relational lines are absent) whereas the set consisting of the same individual and his or her close friends will be densely knit (many of the possible lines are present).

It follows, then, that individuals with few weak ties will be deprived of information from distant parts of the social system and will be confined to the provincial news and views of their close friends. This deprivation will [...] insulate them from the latest ideas.

(1983, pp.201-202)

The concept of the long tail is something both Jay Cross and Tony Karrer have recently discussed and is example of how the Web (and particularly Web 2.0 technology) changes the way assets - whether physical artifacts like books, or knowledge and informational assets persist for an extended period beyond their supposed "sell-by" date:

Long tails for the enterprise occur when the power to create and publish is widely held, the content can be distributed at near-zero cost and a market exists that connects knowledge workers with a nearly infinite number of content creators.

(Kilian, D. 2007)

Here's a pertinent example of how these ideas manifest themselves in the workplace: last week, I suffered from a niggly problem with my Outlook e-mail client - it wouldn't poll the Exchange server and update itself every 20 minutes as it was supposed to do. So I logged a snag on the corporate Bugzilla implementation about the issue. The IT person, who I would describe as being a a journeyman level of competence (has passed their certification exams and is no longer a novice, but is not yet an expert) wen though all the things your supposed to do to resolve such issues

• ran ScanPST.exe
• checked my e-mail profile
• consulted MSDN
• looked at forums for similar issues based on the Error ID

... as well as some "beyond the call of duty" activities (a time-consuming MS Office reinstall).

All to no avail.

So I got my laptop back and had resolved myself to living with this seemingly intractable minor inconvenience, when a third contributor (a more knowledgeable IT support person), working from home, happened to encounter the issue when scanning through Bugzilla, entered the discussion with a simple "I know what this is."

So, by accessing my laptop via a PC-sharing application, the issue was resolved in about 20 minutes, after 5 days of dead ends and frustration.

The moral of the story is: by developing a corporate culture that encourages wide-ranging participation, and by providing a corporate knowledge-sharing environment (Bugzilla in this case), you increase the chances that somebody you're associated with, no matter how loosely, will have the appropriate knowledge and expertise to find a solution to an issue. The added learning benefit from the journeyman contributors perspective, is that they have added to their knowledge experientially, by interacting with the More Knowledgeable Other. I would suggest that the knowledge asset acquired by being involved in this problem-solving activity has been aggregated into their personal experience schema, enabling them to grow a little more knowledgeable (or even wiser).

Oh yes... the solution to the Outlook issue?

Delete and recreate your profile in Outlook.

_______________

References:

Cross, J. (2008) Strength of weak knowledge sources. [Internet] Available from: http://internettime.com/2008/04/21/strength-of-weak-knowledge-sources/ Accessed 1 July 2008

Granovetter, M. (1983) The Strength of Weak Ties: A Network Theory Revisited. Sociological Theory, Volume 1, 201-233. State University of New York,
Stonybrook. [Internet] Available from: http://www.si.umich.edu/~rfrost/courses/SI110/readings/In_Out_and_Beyond/Granovetter.pdf Accessed 1 July 2008

Karrer, T. (2008) Corporate Learning Long Tail and Attention Crisis : eLearning Technology. [Internet] Available from: http://elearningtech.blogspot.com/2008/02/corporate-learning-long-tail-and.html Accessed 1 July 2008

Kilian, D. (2007) The Learning Organization Meets the Long Tail (Part 2). [Internet] Available from:
http://www.clomedia.com/guest-editorial/2007/October/1949/index.php Accessed 1 July 2008

Guidelines for Evaluating Training Programs

An advantage of the retrospective pre-test model is that it can assist in addressing some of the gaps exposed by the lack of range-of-transfer of learning metrics provided by a Level 2 analysis of learners: in effect - to borrow a phrase from Jens Bjornavold – “make learning visible” (2004, p.64). This is essential, because it helps measure learning. In Chapter 5 of Evaluating Training Programs (2006) Kirkpatrick and Kirkpatrick stress the significance of measuring learning, because “no change in behaviour can be expected unless one or more of the learning objectives have been accomplished” (p.42). They set out “helpful” guidelines for the measurement of learning:

Table 1 Guidelines for evaluating learning

Guidelines for evaluating learning

Use a control group if practical

Evaluate knowledge skills and/or attitudes both before and after the program

Use a paper-and-pencil test to measure knowledge and attitudes

Use a performance test to measure skills

Get a 100 per cent response

Use the results of the evaluation to take appropriate action

References:

Bjornavold, J. (2004) Making Learning Visible: Validation of Formal, Non-Formal and Informal Learning: policy and practices in EU Member States1. European Journal of Education [Internet] 39 (1), 69–89. Available from: http://www.acc.eu.org/uploads/
Makinglearningvisible_1.pdf [Accessed 21st February 2007]

Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

--

Assessing learning programs: advantages and disadvantages of the pre-test design

When choosing to integrate the retrospective pre-test design method into your learning program evaluation and measurement methodology, the selection must take place in the after very carefully evaluating arguments both for and against its utility. The method has many advantages for the practitioner-researcher (see Table 1) but one must be cognisant of the threats to validity. However, in the context of the case study method, triangulation of data from other sources (questionnaires, surveys, quantitative analysis of access logs, LMS records, summative assessments and so on) can be used to counteract or balance such threats.

Table 1 Advantages and disadvantages of the retrospective pre-test model

Advantages	Disadvantages
Simple and cost-effective	Possibility that results were due to history in the job or organisation
Reduces costs and time for data collection and analysis	Possible distortions in retrospective reports because of response shift bias
Gathers data as part of the learning intervention
Compares post-intervention data with retrospective pre-data
Avoids attrition from the sample being measured
Decreases likelihood of testing effects

--

Evaluating learning initatives: the Retrospective Pretest Design

The Retrospective Pre-test method is an interesting if controversial data collection design. I will examine its advantages and negative aspects here, as well as an argument for including this strategy in in assessing organisational learning initiatives. The retrospective pre-test method (see Table 1 is an extension of the one-shot case study design where data are gathered from participants following the learning intervention (for example, a series of non-formal Information Sessions).

Table 1 Retrospective Pre-test Design

Retrospective Pre-test Design
Groups		Non-equivalent	Intervention	Observation
Locally-based		N	X	O1	O2
Remotely-based		N	X	O1	O2
<- Time ->

However, participants (N) report on their knowledge, understanding or skills (Observation O1) after the intervention (X), and then also reflect and answer (O2) what they believe their understanding or skill was before the intervention. Rockwell & Kohn (1989) applied this method to testing the effectiveness of achieving program outcomes when interventions such as training programs are implemented and concluded that “using a post-then-pre design [retrospective pre-test] to identify self-reported behavioural changes can provide substantial evidence for program impact”. The retrospective pre-test enables researchers to reduce the response shift bias - defined by Klatt & Taylor-Powell in their 2005 paper Synthesis of Literature relative to Retrospective Pretest Design as

the change in the participant’s metric for answering questions from the pre-test to the post-test due to a new understanding of a concept being taught

- because the participants are able to give pre-test responses which are based on a post-intervention frame of reference.

Using the retrospective pre-test, response shift bias can be reduced increasing the likelihood that the observable results are due to intended intervention effects (Pratt, McGuigan & Katzev, 2001). This proposition is complicated and not without its critics – Theodore Lamb (2005), an advocate of the retrospective pre-test method, describes it as “an imperfect but useful tool.” Robson (2002, p.139-141) expresses concerns that used purely as a quasi-experimental method, this strategy could lead to an internal validity threat through regression to the mean.

References:

Klatt, J. and Taylor-Powell, E. (2005) Synthesis of Literature relative to RetrospectivePretest Design. Presentation to the 2005 Joint CES/AEA Conference, Toronto [Internet] Available from: http://www.citra.org/Assets/documents/evaluation%20design.pdf [Accessed 5th August 2007]

Lamb, T. (2005) The Retrospective Pretest: An Imperfect but Useful Tool. Evaluation Exchange. [Internet] 11(2). Available from: http://www.gse.harvard.edu/hfrp/eval/issue30/spotlight.html [Accessed 21st March 2007]

Pratt, C.C., McGuigan, W.M., & Katzev, A.R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3). 341-349.

Robson, C. (2002) Real World Research. 2nd ed. Oxford: Blackwell Publishing.

Rockwell, S. K. & Kohn, H. (1989) Post-Then-Pre Evaluation. Journal of Extension. [Internet] 27(2). Available from: http://www.joe.org/joe/1989summer/a5.html [Accessed 21st March 2007]

--

Evaluating Non-formal e-learning initatives: Case Study Techniques

As I asserted in a previous post, case studies allow naturalistic methods of enquiry such as individual interviews to be carried out, side-by-side with quantitative data gathering from surveys, archival records, server access logs and so on.

Interestingly in discussing the flexibility of the case study design, Yin (1994, p.285) contends that in the future, researchers will focus not on the case study method, but rather on the specifi case study data collection techniques (see Table 1) and we will increasingly see these techniques used in other “non-case study” forms of research.

Table 1 Case Study Techniques

Case Study Techniques

The use of multiple sources of evidence, in a converging manner

The explicit specification and testing of hypotheses and rival hypotheses, especially in lieu of control or comparison groups

The dominance of deductive strategies, whereby research starts with theorizing

Program logic models as a standard way of initiating a program evaluation

Pattern-matching as a common strategy for data analysis

Portfolio analysis, using qualitative criteria to differentially weigh the outcomes from a project or the projects within a program

The use of replication logic, rather than aggregating data, when comparing the results from multiple sites or cases.

This approach of modularising (to coin a term) the methodologies used in a case study may potentially enable future researchers to generalise from results attained through a case study approach to a broader context: this concept is outside the scope of this blog, but is something in my view that would have benefits in future social sciences research and evaluation of learning initiatives.

References:

Yin, R. K. (1994). Discovering the future of the case study method in evaluation research. Evaluation Practice [Internet] 15. Available from: http://aje.sagepub.com/cgi/reprint/15/3/283 [Accessed 15th January 2008 subscription required]

--

Creating a Research Design to evaluate the effectiveness of e-learning projects

In a previous blog entry, I mentioned the importance of data triangulation, particularly if you use a case study approach when evaluating an e-learning initiative. Another strategy I recommend is to collect data over a number of phases, using a number of techniques.

Case studies involve in-depth, descriptive data collection and analysis of a case, or a number of related instances of the same type within the case. In particular, the case study design is useful when answering “how” and “why” questions, and in understanding the particulars, and diversity of the case (Yin, 1994). As Yin asserted in his 1993 paper Applications of Case Study Research

the case study is the method of choice when the phenomenon under study is not readily distinguishable from its context.

(p.3)

The case study design (see Table 1) suited the conditions of this research project very well as it allowed naturalistic methods of enquiry such as individual interviews to be carried out, side-by-side with quantitative data gathering from surveys and archival records.

Table 1 Features of a case study (Russ-Eft & Preskill, 2001, pp.173-174)

Advantages	Disadvantages
Provides descriptive data	Results may not lead to scientific generalizability
Does not require control of participants or setting	Researcher bias may interfere with validity of the findings
Reports include verbatim quotes	May take too long to conduct
Leads to a greater understanding of the context of the evaluand	May produce more data than can be analysed in an effective manner
Gather data using multiple methods
Provides data that are rich in examples
Captures what is important to the participants
Portrays the multiplicity of causes that are associated with various outcomes
Embraces diversity of perspectives and experiences of participants
Allows the researcher to collect information on outcomes not known prior to the learning and performance initiative

As Robson points out, the flexibility of the case study approach allows the design to “emerge” (2002, p.89) during data collection and analysis. Similarly, the case study facilitates the application of other research designs within the case study design framework, enabling the learning professional to capture data during different phases of a project.

References:

Robson, C. (2002) Real World Research. 2nd ed. Oxford: Blackwell Publishing.

Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.

Yin, R. K. (1993). Applications of Case Study Research. Newbury Park, CA. Sage.

Yin, R. K. (1994). Discovering the future of the case study method in evaluation research. Evaluation Practice [Internet] 15. Available from: http://aje.sagepub.com/cgi/reprint/15/3/283 [Accessed 15th January 2008 subscription required]

--

Evaluating Non-formal Learning: Establishing validity in your research

As discussed in a previous post, to assist in understanding the “parallel criteria” (1989, p.233) relationship between these test of rigour and the scientific terms for validity, Guba and Lincoln provide the following table (1981, p.104):

Table 1 Relationship between parallel criteria

Aspect	Scientific Term	Naturalistic Term
Truth Value	Internal Validity	Credibility
Applicability	External Validity	Fittingness/transferability
Consistency	Reliability	Dependability/auditability
Neutrality	Objectivity	Confirmability

When considering appropriate research designs to evaluate the effectiveness of learning initiatives, it's important to consider the types of data that you have access to, particularly in the context of the issues evaluating a non-formal learning initiative at Kirkpatrick’s Levels 3 and 4 (Shelton. & Alliger, 1993, pp.43-46, Horton, 2006, p.109), I feel it is essential to ensure validity of the naturalistic information by capturing a range of supporting quantitative data.

To assist evaluators understand and provide a context for the analysis of the information collected in the final, qualitative phase of the study, Russ-Eft and Preskill (2001, pp.155-156) describe a number of useful techniques for establishing the validity of quantitative and qualitative data, among them accuracy checking in data recording and encoding, persistent participant observation and member checking.

As the resultant evaluation will use a mix of qualitative and quantitative methods as well as data collected from multiple sources, use I recommend that you use data triangulation used as a strategy to enhance the rigour and validity of you evaluation.

The term triangulation originated in cartography where two or more reference points are used to locate an exact position.

Knowing a single landmark only locates you somewhere along a line in a direction from the landmark, whereas with two landmarks you can take bearings in two directions and locate yourself at their intersection.

(Patton, 1990, p.187).

Denzin (1988) has identified four types of triangulation – data triangulation, methodological triangulation, investigator triangulation, and theory triangulation; the I use these three triangulation techniques were used to evaluate the effectiveness of non-formal learning initiatives:

Table 2 Denzin's (1988) Definition of triangulation methods

Triangulation method	Definition
Data triangulation	Collecting data from a variety of sources. For example, in evaluating the transfer of learning from a three-hour workshop, the evaluator collects information from the learners, their managers, and their peers. In this case, three different sources have been queried.
Methodological triangulation	Using more than one method to collect data. For example, we may interview 20 per cent of a department’s employees and survey the remaining 80 per cent. By using two methods, the weaknesses of one may be compensated fro by the other.
Theory triangulation	Using different theoretical perspectives to interpret the same data. By applying different theories to make sense of the data, it is possible to see how different assumptions and beliefs influence one’s interpretations. By making these explicit, stakeholders can see how their assumptions might influence various actions taken because of the findings.

References:

Denzin, N. K. (1988) The Research Act: A Theoretical Introduction to Sociological Methods. 3^rd Ed. Englewood Cliffs, NJ: Prentice-Hall.

Guba, E.G. & Lincoln, Y.S. (1981) Effective Evaluation. San Francisco: Jossey-Bass.

Horton, W. (2006) So how is elearning different? IN: Kirkpatrick, P. & Kirkpatrick, J. Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Patton, M. Q. (1990) Qualitative Evaluation and Research Methods. 2^nd Ed. Thousand Oaks, CA. Sage

Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.

Shelton, S. & Alliger, G. (1993) Who's Afraid of Level 4 Evaluation? A Practical Approach. Training & Development. [Internet] 41 (3). Available from: http://www.afc-ispi.org/Repository/
Approaches%20for%20Assessing%20Outcomes%20ROI.doc [Accessed 24th March 2007]

Evaluating Non-formal Learning: Using "tests of rigor" to validate findings

Without rigour, research is worthless, becomes fiction, and loses its utility. Hence, a great deal of attention is applied to reliability and validity in all research methods.

(Morse et al, 2002, p.1)

Morse and his colleagues develop their position by asserting that the challenges to rigour in qualitative inquiry paralleled the growth of statistical packages and the development of computing systems in quantitative research: without “the certainty of hard numbers and p values, qualitative inquiry expressed a crisis of confidence” (Morse et al, 2002, p.2).

A number of leading qualitative researchers (Leininger, 1994, Altheide & Johnson, 1998) argued that the two approaches are not compatible and that they should not be combined, due to the fundamental differences between the two methods in the context of the nature of knowledge, the relationship between researcher and the object of the research, and the means of generating data.

My personal view aligns with the criteria suggested by Guba & Lincoln (1981) and Yin (1994) to determine reliability, and validity, and thus ensuring rigour, in qualitative research. Reliability and validity relate to the degree of confidence we have that the data are representing the participants’ reality or “truth” (Russ-Eft & Preskill, 2001, p.153). How can we define the truth (or even the accuracy) of participants’ responses in the subjective domain of qualitative research? Guba and Lincoln (1981) propose that researchers use so-called “tests of rigour” to establish validity through the naturalistic concept of “trustworthiness,” containing four aspects: credibility, transferability, dependability, and confirmability, that align to parallel scientific terms associated with the quantitative method.

Table 1 Definition of naturalistic terms (Russ-Eft & Preskill, 2001, pp153-155)

Naturalistic Term	Definition
Credibility	The scientific paradigm asserts that there is one reality and that information is valid when all relevant variables can be controlled; a naturalistic paradigm assumes that multiple realities exist in the minds of individuals. Hence when using qualitative methods, the research seeks to establish the credibility of individuals’ responses. The study must be believable by providing a detailed depiction of the multiple perspectives that exist can enhance the data’s credibility. For example, learner satisfaction surveys, along with interviews with training managers and instructors would provide a more holistic picture of the learning experience.
Fittingness/transferability	How transferable the findings are to another setting is called generalisability in the empirical context. The goal of qualitative methods is to provide richly detailed description to help the reader relate certain findings to their own experience. We often think of these as “lessons learned.” For example, as a stakeholder reads an evaluation report, they realise that something very similar has occurred in their organisation and sees where the findings can be used. Although the entire set of findings may not be applicable in their context, some issues identified may have applicability in other contexts.
Dependability/auditability	In the scientific paradigm, the notion of consistency is called reliability where a study’s consistency, predictability or stability is measured. Since reliability is necessary for validity, it is critical that data of any kind be reliable. Instead of considering data unreliable if it is inconsistent, evaluators using qualitative methods look for reasons that cause the data to appear unstable (inconsistent). For example, an interviewee may give an opinion one day, and when asked again the following week might say something slightly different. What would be important to understand and capture are the reasons for this change in perception. Such inconsistencies may stem from respondent error, an increase in available information, or changes in the situation. An audit trail that includes collecting documents and interview notes and a daily journal of how things are proceeding can help to uncover some of the reasons for such inconsistencies.
Confirmability	Objectivity is often viewed as the goal of most evaluation and research studies. Evaluators and researchers who use qualitative methods don’t necessarily believe that true objectivity can ever be fully achieved, rather that it is impossible to completely separate the evaluator from the method. Instead of trying to ensure that the data are free from the evaluator’s biases, the goal is to determine the extent to which the data provide confirming evidence. “This means that data (constructions, assertions, facts and so on) can be tracked to their sources, and that the logic used to assemble the interpretations into structurally coherent and corroborating wholes is both explicit and implicit” (Guba & Lincoln, 1989, p.243). Establishing confirmability, like consistency, often takes the form of auditing.

References:

Altheide, D., & Johnson, J. M. C. (1998). Criteria for assessing interpretive validity in qualitative research. IN: Denzin N. K. & Lincoln Y. S. (Eds.), Collecting and interpreting qualitative materials. Thousand Oaks, CA: Sage.

Guba, E.G. & Lincoln, Y.S. (1981) Effective Evaluation. San Francisco: Jossey-Bass.

Leininger, M. (1994). Evaluation criteria and critique of qualitative research studies. In J. M. Morse (Ed.), Critical Issues in Qualitative Research Methods. Newbury Park, CA: Sage.

Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002). Verification strategies for establishing reliability and validity in qualitative research. [Internet] International Journal of Qualitative Methods 1 (2), Article 2. Available from: http://www.ualberta.ca/~ijqm/ [Accessed 14th March 2008]

Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.

Yin, R. K. (1994). Discovering the future of the case study method in evaluation research. Evaluation Practice [Internet] 15. Available from: http://aje.sagepub.com/cgi/reprint/15/3/283 [Accessed 15th January 2007]

--

Using a Case Study approach to evaluate Non-Formal Learning

I consider a case study methodology to be the most effective strategy to use when evaluating non-formal learning because this approach provides the means to develop a rich description of the non-formal learning (NFL) initiative.

This assists in providing a context, and helps define the NFL initiative's value to the learners and to the organisation. An added benefit of the case study approach is that it enables the learning practitioner to employ a number of different data collection methodologies: participant observation, archive data collection, surveys, and interviews to answer the questions outlined in Table 1. The main purpose of using a number of data collection methodologies was to capture a range of quantitative and qualitative data, and to establish a means of triangulating that data to enhance validity and “trustworthiness” (Russ-Eft & Preskill, 2001, p.153).

Table 1 Questions the case study approach attempts to answer

Quantitative Data	Qualitative Data
Who attends live “in situ” the learning event? (dept., role, skill level, location)	Description and chronology of a the learning event
Who views synchronous live stream of the learning event? (dept., role, skill level, location)	Transfer of new knowledge to the job (Kirkpatrick - Level 3)
Who views asynchronous on-demand version of the learning event? (dept., role, skill level, location)	Does the pedagogical approach used in the learning event fulfil participants’ needs?
Who requests asynchronous DVD version of the learning event? (dept., role, skill level, location)	Do non-formal learning event enhance individuals' and organisation performance? (Kirkpatrick - Level 3)
Participants satisfaction with the learning event? (Kirkpatrick - Level 1)	What is the Business Impact of the the learning event or series of events over time (Kirkpatrick - Level 4)
What do attendees know that they didn’t know before? (Kirkpatrick - Level 2)

References:

Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.

--