As discussed in a previous post, to assist in understanding the “parallel criteria” (1989, p.233) relationship between these test of rigour and the scientific terms for validity, Guba and Lincoln provide the following table (1981, p.104):
Table 1 Relationship between parallel criteria
| Aspect | Scientific Term | Naturalistic Term |
| Truth Value | Internal Validity | Credibility |
| Applicability | External Validity | Fittingness/transferability |
| Consistency | Reliability | Dependability/auditability |
| Neutrality | Objectivity | Confirmability |
When considering appropriate research designs to evaluate the effectiveness of learning initiatives, it's important to consider the types of data that you have access to, particularly in the context of the issues evaluating a non-formal learning initiative at Kirkpatrick’s Levels 3 and 4 (Shelton. & Alliger, 1993, pp.43-46, Horton, 2006, p.109), I feel it is essential to ensure validity of the naturalistic information by capturing a range of supporting quantitative data.
To assist evaluators understand and provide a context for the analysis of the information collected in the final, qualitative phase of the study, Russ-Eft and Preskill (2001, pp.155-156) describe a number of useful techniques for establishing the validity of quantitative and qualitative data, among them accuracy checking in data recording and encoding, persistent participant observation and member checking.
As the resultant evaluation will use a mix of qualitative and quantitative methods as well as data collected from multiple sources, use I recommend that you use data triangulation used as a strategy to enhance the rigour and validity of you evaluation.
The term triangulation originated in cartography where two or more reference points are used to locate an exact position.
Knowing a single landmark only locates you somewhere along a line in a direction from the landmark, whereas with two landmarks you can take bearings in two directions and locate yourself at their intersection.
(Patton, 1990, p.187).
Denzin (1988) has identified four types of triangulation – data triangulation, methodological triangulation, investigator triangulation, and theory triangulation; the I use these three triangulation techniques were used to evaluate the effectiveness of non-formal learning initiatives:
Table 2 Denzin's (1988) Definition of triangulation methods
| Triangulation method | Definition |
| Data triangulation | Collecting data from a variety of sources. For example, in evaluating the transfer of learning from a three-hour workshop, the evaluator collects information from the learners, their managers, and their peers. In this case, three different sources have been queried. |
| Methodological triangulation | Using more than one method to collect data. For example, we may interview 20 per cent of a department’s employees and survey the remaining 80 per cent. By using two methods, the weaknesses of one may be compensated fro by the other. |
| Theory triangulation | Using different theoretical perspectives to interpret the same data. By applying different theories to make sense of the data, it is possible to see how different assumptions and beliefs influence one’s interpretations. By making these explicit, stakeholders can see how their assumptions might influence various actions taken because of the findings. |
Denzin, N. K. (1988) The Research Act: A Theoretical Introduction to Sociological Methods. 3rd Ed.
Guba, E.G. &
Horton, W. (2006) So how is elearning different? IN: Kirkpatrick, P. & Kirkpatrick, J. Evaluating Training Programs. 3rd ed.
Patton, M. Q. (1990) Qualitative Evaluation and Research Methods. 2nd Ed.
Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change.
Approaches%20for%20Assessing%20Outcomes%20ROI.doc [Accessed 24th March 2007]
0 comments:
Post a Comment