Needs Assessment in Postgraduate Medical Education:A ReviewSavithiri Ratnapalan MBBS., and Robert I. Hilliard MD, EdD.
Medical residents learn by several methods such as; self-directed learning, informal bedside teaching and structured teaching activities.1 Some of the structured learning activities are resident lecture series, morning reports, grand rounds, conferences, skill workshops, and special seminars or lectures. The educational activities planned for residents aim to help them gain the necessary knowledge and skills equal to those of their peers and, facilitate improved patient care. There are formative evaluations during the residency program and summative evaluations at the end of residency to become licensed to practice independently. The qualifying examinations are based on well-defined educational objectives determined by the professional body conducting the examination. These examinations attempt to ensure that newly qualified physicians have gained the necessary knowledge and skills equal to those of their peers.2 In the current climate of consumer-driven services, educators face increasing pressure to plan educational activities based on the identified needs of the target audience.3 We feel that assessing learning needs is a fundamental step to ensure the relevance of the educational activity to a target audience. If educators don’t use needs assessments, they may not know what the learners want and what will motivate them to learn. We have attempted in this paper to discuss residents’ learning needs; who determines their learning needs; and methods of assessing the learning needs for various educational activities. Learning Needs Residents are adult learners engaged in post-graduate professional education. Their needs can be classified by two main criteria: who is determining the needs (learners, educators, or others) and what standards are used as the ideal. Normative needs are defined as the measured gap between the set standards and the individual’s or group’s current knowledge.4 In medical education normative needs encompasses knowledge, skills, attitudes and performance of the residents and are determined by the certifying bodies such as the Royal Colleges (United Kingdom, Canada) or the specialty boards such as the American Board of Pediatrics in the USA. These standards are set based on value judgement, expert opinion, or research information. The opinion of those being assessed is not taken into account. Prescribed needs are those areas that educators or program planners determine as inadequate and that need educational intervention.4 Prescribed needs usually take normative needs into account. Felt or perceived needs are what the individuals or the group have identified as what they want to learn.4 Their knowledge, experience, and the environment they work in influence most of these needs. These are the learner’s needs as perceived by the learner. These needs are characterized by the sentence, “I know what I don’t know.” Expressed needs are what an individual or group express as their needs.4 It would be naïve to assume that all perceived needs are expressed because there may be several real or perceived barriers to expression. For example, learners may not want to be identified as lacking knowledge, or they may lack the opportunity to express their needs. Learners may also lack the motivation, communication skills, or assertiveness to express their needs. This lack of expression (or demand) can create an unhealthy situation, because it can be confused with the absence of a need for a particular educational intervention.4 Comparative needs are those learning needs identified by comparing two similar groups or individuals rather than against normative standards.4 For example residents from university X are less skilled in counseling compared to residents from university Y. Unperceived needs are discrepancies not perceived by learners as learning needs, characterized by the sentence, “I do not know what I don’t know.” Teachers, professional bodies, clients or patients, allied health professionals, and national and international organizations are often the bodies that identify these unperceived needs; for example, through epidemiological reviews of health care problems within the institution or community.6
 Needs Assessment A needs assessment is a systematic process to collect and analyze information on what a target group needs to learn. Although the value of conducting a needs assessment is not well established in postgraduate medical education, needs assessment is being used by some medical educators to develop curriculum for the training programs. We feel that needs assessment should be conducted before any adult educational activity so that the available resources can be used to the maximum benefit of the learners. Learning needs assessments are often conducted to identify deficiencies in knowledge, skill, behavior, or attitude in the current teaching practices, or to anticipate deficiencies based on expected changes in health care needs.5 There are several methods of needs assessment described in the literature. We will discuss their applicability to postgraduate medical education. Questionnaire Surveys Questionnaires elicit written responses to specific questions in order to gather information from the responders. They are considered a quantitative form of needs assessment because the frequency of responses can be counted and answers to questions can be weighted with a numerical value. The responses depend on the design and quality of questions asked.6 Questionnaire surveys are a popular form of need assessment because of their low cost and ease of administration. Questionnaires can address a wide array of topics, assess diverse educational needs, and sample a large population. Well-designed questionnaires provide useful information returned to the researcher in a standardized format. They are widely used as a needs assessment tool in continuing medical education. They have been used for broader general needs assessment such as, education in geriatrics for community physicians; and for discipline-specific needs assessments such as, management of arrhythmia by emergency physicians.7,8 Questionnaires can be used alone or in combination with other methods of needs assessment to assess the learning needs of residents to plan seminar topics or workshops objectives. Questionnaires are relatively inexpensive, can sample large numbers and can be returned anonymously. There are several books and articles that can be helpful in designing questionnaires. 6,9 Interviews An interview can be viewed as a conversation with the purpose of gaining in-depth insight into someone’s perspective. Interviews can be conducted in person or over the telephone. The advantages of an interview are that it is personal and gives the researcher an in-depth understanding of the learner’s perspectives. Qualitative interviewing unveils a broader range of learning needs and opportunities for program development than quantitative methods alone 10 Telephone interviews can be less time consuming and less costly than in-person interviews. Questionnaire surveys consisting of set questions with provision for elaboration on certain areas can be used for telephone surveys. Long telephone interviews are difficult for both parties and lack the nonverbal cues that would help the interviewer redirect the questions. On the other hand, interviews are not anonymous, take time and effort, and are not feasible for assessing the needs of large numbers of learners. Analyzing descriptive interview data can be challenging and can take four to five times as long as the time spent in data collection. Francke et. al. observed and interviewed surgical cancer patients and nurses to determine the learning needs associated with pain control.11 The outcome of this needs assessment was used to design a 6-week continuing professional education program. Data collection took 400 to 450 hours, of which 50 to 100 hours were spent on interviews. The analysis of all data required 1,600 to 2,250 working hours. In medical education, interviews as a needs assessment technique are useful to understand learners’ knowledge base and how this shapes their perception of educational needs. For example, most program directors in Canada attempt to have one to one interviews with residents during the academic year for formative evaluation and feedback. These interviews are used to assess individual learning needs and are called ‘ Dr. X ‘s fireside chats’ in our institution. Focus Groups Focused interviews were first developed by a sociologist to help the United States army produce training films to educate new trainees about the latest military hardware and to boost morale. Now focus groups are responsible for everything from what we watch on television, how movies end, how grocery stores display their items, and how politicians speak to the public.12 Focus groups are usually, seven to ten randomly selected participants who meet criteria to be considered consumers or learners for a particular service (or educational activity). A skilled facilitator, who encourages a sense of synergy but explores differences in opinion, conducts the interview. Planning, developing core questions, facilitating or moderating the session, and analysis of data are the essential steps in conducting focus group interviews.12 Focus group interviews differ from individual interviews, because members of the group draw strength from one another to express some opinions that they may otherwise view as unpopular. This is particularly important in eliciting negative views or constructive criticism from learners. The use of focus groups for needs assessments in postgraduate medical education is increasing. In Canada, the Royal College of Physicians and Surgeons conducts focus group interviews of all the residents to assess residency programs for accreditation. Focus groups provide a broad range of qualitative data in a timely, cost-effective manner and help to clarify and elaborate the quantitative data. They are very useful in areas of health care where physician attitudes or behaviour modifications are targeted as behaviour modification can only occur when the individuals or groups concerns are addressed. Data from focus groups may not represent the needs of the target audience unless a sufficient number of groups have been interviewed. There are no clear guidelines as to what constitutes a sufficient sample size for an educational needs assessment. Focus group interviews of all the residents in a program (such as those conducted by the Royal college of Physicians and Surgeons of Canada for residency program evaluations), will be the ideal although not always feasible. As with questionnaires, most often focus groups are used in conjunction with other methods of needs assessment for planning educational activities. Chart Audits Chart audits are reviews and assessments of patients’ medical records that use present criteria and standards. Chart audits are used as a needs assess-ment tool in a variety of situations such as in continuing professional education to determine learning needs prior to education, to assess change after education, and to assess variations in professional practice.13 They are also used to evaluate competence and performance.14 In 1984, McAuley et al used two peer assessors to evaluate 25 randomly selected charts and showed that chart audits had a 93% sensitivity and 88% specificity in identifying unsatisfactory patient care.15 Sengstaken and King used chart audits to identify the need for education in pain management of geriatric patients.16 Residents are the frontline medical staff in most tertiary care facilities. Random or targeted chart audits in these institutions will identify areas of weakness in patient care, and educational interventions to correct these issues can be arranged. Chart-Stimulated Recall Chart-stimulated recall was developed by Maatsch in 1981 to assess clinical competence in emergency medicine.17 In chart stimulated recall, a patient’s chart is used to stimulate the professional’s recall of the particular case and its management. Chart-stimulated recall is a case-based interview that provides the rationale for management decisions and other options that were considered and ruled out. Interviewers must be trained health professionals. Pilot studies assure inter-rater reliability and consistency. Interviews are usually 15 to 20 minutes in length for each chart-stimulated recall. Three to six chart-stimulated recalls are enough to provide reliable and valid assessment.14 Chart-stimulated recall is more expensive than chart audits, and is likely not to be feasible if large numbers of physicians have to be assessed. Chart-stimulated recall gives residents a chance to explain the rationale behind a particular management decision. For example, a chart audit may appear to indicate that patient A received inadequate analgesia on day X suggesting that the resident had poor skills at assessing pain, or that the resident did not know the proper dosages. Chart stimulated recall provides the resident with an opportunity to explain his or her actions that may reveal aspects of situation that justified his or her actions but were not apparent from the chart; or may further clarify the deficiencies in knowledge or skills. Thus the educational objectives identified could be issues such as time management and how to utilize back-up help, rather than pain management alone. Standardized Patients A standardized patient is either a healthy subject who is trained to present a particular medical case accurately or an actual patient who is trained to present his or her illness in a standardized manner. Rosenhan, in his controversial study published in 1973, showed that academic staff in emergency departments of psychiatric hospitals could not differentiate the real patients from the standardized ones who were simulating psychiatric symptoms. 18 Davis et al. showed how standardized patients can be used in needs assessment to plan continuous medical education.19 Standardized patients can be used for assessing needs for medical education and to assess the impact of an educational activity. They can be used to assess both competency in simulated settings and performance in actual practice. It may be impractical or unethical to assess certain skills, attitudes or behaviors in the real setting. Needs assessment in postgraduate education of counseling skills or ethical management issues can be accomplished using standardized patients. Environmental Scans Environmental scanning has been a strategic business management tool since the 1980s.20 In environmental scans, non-threatening and unobtrusive research can be conducted by checking (scanning) existing sources of information. Institutions of higher education adopted these methods to ensure the relevance of their teaching to the rapidly changing society. 20 Environmental scans refer to methods and resources that professional educators use to assess their surrounding to help them identify current and potential learning needs.20 These can be internal (within an institution or hospital) or external (outside the institution). Some resources of internal environmental scanning are previous oral or written recommendations, attendance data, budget information, chart audits, mission statements, and comments from in-house experts and department heads. External sources are guidelines of care, literature searches, published reports from professional organizations or regulatory bodies, and medical core curricula.20 Some of the internal environmental scan resources such as; data on seminar attendance, course evaluations, pharmacy chart audits, incidence reports and departmental morbidity and mortality reports can be used for residents’ needs assessment. Practice guidelines, educational objectives of the American Board of Medicine or Royal College of Physicians and Surgeons of Canada and World Health Organization recommendations are some of the external environmental scan resources useful in residents’ needs assessment. Needs Assessments at Work The published research suggests that educators use more than one needs assessment method to plan educational activities.21 The word triangulation was introduced to describe combining data from different needs assessment methods to identify the learning needs of the target audience.21 This is an important concept in post graduate education as a combination of needs assessments can used to plan educational activities. 22 The objectives of the educational activity guide the method of needs assessment. For example, if the purpose of the education is to prepare the learners for an examination to be held a couple of weeks later, it would make sense to teach the normative needs. However, if the learners (such as the final year residents in a program) are well aware of the normative needs and have identified their learning needs, then the educator should focus on the learners perceived or expressed needs. A study from Dublin describes how they used questionnaires and semi-structured interviews to conduct a national medical educational needs’ assessment of interns to develop an intern education and training program. 23 Another Canadian study describes a successful core curriculum initiative for postgraduate medical education using surveys and environmental scan (objectives from governing bodies ; AAMC Core Curriculum Project from the American Association of Medical Colleges and CanMEDS 2000 Project from the Royal College of Physicians and Surgeons of Canada). 24 Needs assessment by residents and faculty are also used to evaluate and modify training programs. 25 In summary, we think that the type, purpose, and timing of the educational activity will guide the needs assessment methods chosen by educators. We have attempted to describe the types of learning needs, the various methods of needs assessment and, some examples of the situations when these could be used in postgraduate medical education (Table 2 ).
Successful Educational Activities Time, money, and administrative support may be limited in any educational activity. It is always difficult to decide how much of the limited resources should be spent on needs assessment. A careful needs assessment, however, is an important first step in planning education for any health professional. Residents are time- constrained due to the numerous ‘on-call duties’ and the heavy service load in most programs; their time investment should be given serious thought. 26 An educational activity that does not warrant the time investment of the learners would have diminishing attendance and cease to serve any purpose. The teaching has to be relevant, applicable, and bring about a positive change in patient care and physician practices. This has to be achieved in the least possible time to make it not only an effective, but also an efficient process. Needs assessment of postgraduate medical trainees (i.e., the target population) before planning an educational activity would help achieve these goal. Acknowledgement This paper was prepared with the assistance of Editorial Services, The Hospital for Sick Children, Toronto, Ontario, Canada. References
Reference Ratnapalan S, Hilliard RI. Needs assessment in postgraduate medical education: A review Med Educ Online [serial online] 2002;7:8. Available from URL http://www.med-ed-online.org Correspondence | |||
Medical Education Online Editor@Med-Ed-Online.org