Evaluation of Teaching and Learning StrategiesSybille K Lechner BDS, MDS FRACDS, FPFA, FICD
Introduction The mainstream of tertiary education has seen a massive transformation over the last few decades1 and most educational facilities have now made the quantum leap from trying to be good teachers' to making the learning process more readily available to students. They have also recognized the distinction between deep and surface learning.2 The what, how and why of student learning have become more explicit. What students now need to know is directly related to the information explosion, which is evident in every field of study. The goalposts have changed from teaching facts, to helping students to learn how to find relevant information, how to assess it and how to organize disparate information into a cohesive whole. How students learn is becoming clearer and the move towards student centered learning is finding support in the physiological sciences.3-6 Why students learn has not changed, nor does it seem likely to; students learn primarily to graduate and educational bodies now acknowledge the importance of the link between program aims and assessment, formative assessment as a valuable teaching/learning resource and summative assessment as a motivational force.7 These important insights have triggered a virtual explosion of innovation in teaching/learning strategies and it is becoming increasingly urgent to find ways of evaluating them. However, evaluation of teaching strategies is problematic. The ability of bright people to learn what they need to know despite any curriculum cannot be discounted and high-aptitude students tend to succeed regardless of the instructional strategy used..8,9 This does not mean that educators should consider themselves free to propose changes in an ad hoc manner. There is a need for educational research and ongoing evaluation must be considered to be a fundamental part of educational advance.10 Best Evidence In an attempt to follow the current trend of evidence based medicine, educational bodies are concerned with building a body of best evidence in medical education (BEME).11 A meeting in London was held in 1999 highlighting the need for evidence-based teaching.12 Although this initiative is in the medical field, successful strategies could well be extended into the dental field. Hart and Harden13 identified six steps in the practice of evidence-based teaching: framing the question, developing a search strategy, producing the raw data, evaluating the evidence, implementing change and evaluating it. This group is working to provide guidelines for educational research. However, it is increasingly evident that synthesizing and reviewing evidence is a complex matter.14 Searching databases using the descriptors `education and `evidence-based yields few articles on the concept of best evidence medical education.15 This has led to a suggestion that BEME be renamed BEAME (best evidence available in medical education).16 Even with this proviso, it is difficult to see how the new goals are to be accomplished. The difficulty of evaluating any educational philosophy in a scientific manner is highlighted by the many different methodologies used in attempts to prove the efficacy of problem based learning (PBL) as a lifelong learning resource.17 Subject Pool One of the most frustrating aspects of evaluating education is that academics in the medical and dental field, who have been trained in the rigid scientific methods which are mandatory for evaluating treatment modalities, are tempted to try to transpose such methods to evaluating education. Such attempts, while superficially pleasing in the numerical data they supply, are ultimately unreliable. The rigid quantitative scientific method of controlled experimentation cannot be considered valid in an environment where the variables of the subject pool are almost as great or greater than the pool itself, the student pool in any one year of a dental or medical school being statistically minute. While the subject pool is small, prior knowledge, motivation, opportunity, access to materials, the Hawthorne effect, time constraints, emotional status and even financial pressure can count among the influences affecting learning.Including sequential years and/or different schools can confuse the issue further by adding the variable of different student cultures. Added to this, the value of control groups in educational evaluation is highly questionable. Random allocation of students may address initial systematic differences between the experimental and control subjects. However, in an educational context, the "randomness" soon becomes corrupted since there are rarely placebos in education. Students have opinions and will have an attitude towards whichever group they are in. This, in itself necessarily biases the outcome. Nor is it possible to limit students in the interchange of ideas, as it is to limit types of medication or treatment in clinical trials. Students in one group will find it simple to access friends learning resources if they are interested in doing so.18 The authors have heard of one study testing a computer aided learning (CAL) program where the CAL students did far better in a test than the control, traditionally taught students. However a chance conversation with a student then elicited the fact that the CAL program was so abysmal that students in that group panicked and spent many hours in the library studying that subject. While this story is anecdotal, the possibility of such an occurrence must surely give the serious educational researcher pause when considering the efficacy of control groups as a tool for evaluation in education. Outcomes The other great difficulty is in delineating a clear definition of outcomes. Wilkes and Bligh19 group several types of evaluation into student oriented, program oriented, institution oriented and stakeholder oriented. The indicators cover a wide area, ranging from attendance at class, through patient satisfaction, questionnaires, test results and peer evaluation. Several studies have attempted evaluation using examination,20-24 the number of student inquiries regarding the levels of knowledge required for examinations,25 follow-up surveys26,27 and self-evaluation by the students.28-30 Norman and Schmidt31 are adamant that while outcomes can be measured, they can only be measured if they are small. They contend that big outcomes cannot be measured in any meaningful way as the variables are too great and are so complex and multi-factorial, with so many unseen interacting forces, using outcomes so distant from the learning setting, that any predicted effects would inevitably be diffused by myriad unexplained variables. The simplest measurement of outcome is by examination. Examination results can be shown numerically and sit well with statistical analysis. However, they cannot be relied on to give a true evaluative picture. For example, the fact that 98% students could attain a desired standard of knowledge if a lecture series was replaced by a CAL program.32 cannot be extrapolated to include any other groups of students and cannot be taken to show that the CAL program was a "better" way of teaching. Nor can examination results measure deep learning and lifelong learning, which must now be accepted as ultimate learning goals. Methods for measuring these parameters have not yet been developed.33 For example, a study showing that the students from a school with a problem based learning PBL based curriculum (McMasters) had a better knowledge of hypertension 10 years after graduation than those from a traditional school (Toronto), can be open to several interpretations, further investigation showing that the cardiovascular research was an outstanding accomplishment of McMasters University and that similar results may not have been seen if other fields had been investigated. Even if this were the case, there is no "proof" that this was a direct effect of PBL or an indirect effect mediated by other aspects of the course.34 The possible causes are seemingly endless. Currently, the most pragmatic approach in educational evaluation is to focus on students' perception of their experience with a learning program, and this approach has been used in several studies.35, 20,23,24.36-39 Enjoyment and success engender a winning cycle in the learning environment. If teaching resources can involve students and lead them to be successful in their endeavors, they are more likely enjoy their tasks and want to become even more involved. Exploring students perceptions lends itself to a qualitative research methodology, the focus of which is understanding rather than measuring. Qualitative research can tap into the students' psyche to give a depth of understanding that is the key for ongoing educational development. It is particularly germane to evaluation of the formative aspect of a program and should be part of the development of any educational resource.35,40 Good qualitative research goes beyond reporting what people say to why they are saying it. It makes a distinction between public and spoken attitudes which are socially acceptable, or those which may have no vocabulary or are difficult to verbalize and are therefore suppressed, and private attitudes which may not be socially acceptable and are therefore consciously repressed.41 Qualitative researchers in the commercial field require a high level of expertise to avoid a host of pitfalls and qualitative research methodology has its own strict guidelines. In an educational context it would be imperative that questionnaires dealing with students' perceptions are anonymous and that focus groups be conducted by people known to be far removed from any summative assessment results. Observation of students involved in a program is another invaluable developmental device. However, this is a subjective activity and, with a single observer, is only as valid as the perception of that observer. Quantitative analysis of factors which have been found to be relevant to students via this exploratory research can then proceed, each small positive outcome accumulating to build a broad picture of what is likely to produce success. Triangulation, which is the application and combination of several research methodologies in the study of the same phenomena can be employed in both quantitative(validation) and qualitative(inquiry) studies. It has been proposed that, by combining multiple observers, theories, methods and empirical materials, researchers can hope to overcome the weakness or intrinsic biases of any particular methodology and the problems that come from single method, single-observer, single-theory studies.42 Summary With the growing awareness of the importance of teaching and learning in universities, it behooves the professions to re-examine their educational research methodology. Numbers and statistical significance are very comforting to minds trained in the rigors of laboratory experimentation and clinical trials. However, this methodology cannot be imported wholesale into educational research. There is a need for close scrutiny of the validity of these methods in the very different field of education. The key to good research is flexibility and an understanding of the limitations of any research methodology in any given field. Hopefully an accumulation of the "small outcomes" which Norman and Schmidt envisage will eventually result in a broad spectrum of probability across a multitude of studies. Currently, the most realistic indicator of a program's success is the students' own perception of their learning. The work of the BEME group and other interested health educators will, in time, produce more objective parameters. These parameters will not be unrealistic shadows of those used in clinical and laboratory trials of the health professions. They will be specific to the educational field and recognize the unique aspects of the tertiary student subject pool and the complex nature of the expected outcomes. References
Acknowledgements My thanks to Professor A Sefton, Faculty of Medicine, University of Sydney, for the knowledge about educational evaluation she has shared with me and to KMD Harris. Account Manager, The Leading Edge, Market Research Consultants Pty Ltd Sydney for many interesting insights into research strategies. Correspondence to Sybille K Lechner BDS, MDS FRACDS, FPFA, FICD Phone: 612 9949 5164 E-mail slechner@bigpond.net.au |