EVALUATION OF THE COGNITIVE LEVEL OF ESSAY QUESTIONS OF AN UNDERGRADUATE MEDICAL PROGRAM IN SRI LANKA, USING BLOOM’S TAXONOMY

Background: Medical curricula should encourage the development of advanced cognitive skills . Examination questions are an indicator of the cognitive level expected of students. However, it is believed that many factors, such as difficulty in maintaining objectivity in marking, large numbers of scripts and poor language ability of students lead to compiling of essay questions requiring lower cognitive levels. Objectives: The objective of the study was to categorize essay questions of the undergraduate medical program of a Sri Lankan university according to the cognitive level tested. Methods: Essay questions were classified into cognitive levels based on Bloom‟s taxonomy and subsequently categorized broadly as those requiring lower order thinking skills (LOTS), middle order thinking skills (MOTS) and higher order thinking skills (HOTS). Analysis was done based on discipline and components of study (pre-clinical, paraclinical and clinical). pre-clinical to the clinical part of the course. It is advisable that relevant academic departments reflect upon these findings in view of improving the impact of their examinations on development of higher order cognitive skills among students. Conclusions: Essay questions requiring LOTS predominate in most disciplines and at all levels of the course. An increase in the proportion of questions requiring MOTS and a decline in the proportion of questions requiring HOTS was observed from pre-clinical to the clinical part of the course. Systematic evaluations of examination questions to identify their cognitive levels would generate valid evidence to reflect upon institutional commitments to improve higher order cognitive skills among students.


Introduction
Academic success should be measured not just in terms of what students can remember, but by what students are able to do with their knowledge. Therefore not only do we need to explore teaching and learning techniques that enhance such thinking, but also construct assessments which promote such, as assessment is known to drive learning. 1 Facilitating the development of higher order thinking skills in medical students has never been seen to be more important than today. For example, the technology-mediated ease of access to a vast repository of knowledge which may even be biased and unsubstantiated, necessitates students to be able to evaluate the authenticity of information; and the frequent exposure to complex clinical scenarios requires them to determine ways of applying medical knowledge and synthesize knowledge for the benefit of patients. The current guidelines of the General Medical Council, United Kingdom and the benchmark statement of the University Grants Commission of Sri Lanka, emphasize the development of advanced cognitive skills as an integral component of medical education 2,3 .
In Sri Lanka, medical education is an undergraduate program. Selection of students for this program is by a single highly competitive examination (the advanced level examination). The competitive nature of this examination has led to students and school teachers to resort to educational tactics which do not encourage higher order thinking and are highly teacher dependent. As a result, students who have been exposed to a dependent, knowledge based, and didactic secondary education system are selected for tertiary education. Nevertheless, tertiary education requires that students learn independently, solve problems and synthesize new knowledge. Even though the onus of bridging this gap between secondary and tertiary education is on the teachers of the tertiary education system, it is unclear as to the extent it occurs in actual practice. The way of ensuring higher order thinking among medical students would be to develop curricula, which promote such thinking. 4 It may be reasonable to assume that constructing questions which require higher order thinking would reflect the quality of curricula and the institution"s commitment towards promoting higher order thinking as assessment is known to drive learning. 5 Systematic evaluation of examination questions would provide such institutions with useful evidence to reflect on their educational practices.
Literature reveals the use of Bloom"s taxonomy for assessment of cognitive levels tested by examination questions in medical and non-medical disciplines. 6,7,8,9 The Bloom's taxonomy is a framework for classifying learning objectives which is frequently used to categorize questions and objectives of academic programmes, in order to evaluate their extent of spread across cognitive levels. This method classifies objectives according to the following domains; Remember (Recall data or information), Understand (Understand the meaning, translation, interpolation, and interpretation of instructions and problems), Apply (Use a concept in a new situation), Analyse (Separate material or concepts into component parts so that its organizational structure may be understood, distinguishes between facts and inferences), Evaluate (Make judgments about the value of ideas or materials), Create (Build a structure or pattern from diverse elements, put parts together to form a whole, with emphasis on creating a new meaning or structure).
A single faculty of medicine in Sri Lanka was chosen for the purpose of this study. There are approximately 210 students per single examination in this faculty. The medium of instruction in the Faculty is English (the second language in schools) with a majority of students having followed their school education in their first language (Sinhala or Tamil Paediatrics, Psychiatry, Anaesthesiology, and Radiology. The progression of the academic program is from pre-clinical, para-clinical to clinical components.

Objectives
The general objective of this study was to determine the cognitive level tested by examination essay questions of the selected undergraduate medical program. The essay format was chosen for this analysis as these questions could be used to assess students over the entire spectrum of cognitive abilities and are currently used by all disciplines in the selected medical program. Furthermore, we intended to perform a longitudinal and discipline based analysis of questions, as the literature did not reveal evidence of such analysis in medical curricula. Accordingly, the study was conducted to achieve following specific objectives.  To identify the cognitive level of essay questions of specific disciplines, and  To identify the cognitive level of questions and trends of using such questions along the pre-clinical, para-clinical and clinical components of the curriculum.

Methods
All essay questions from end-semester examinations of the institute from 2008-2013 were categorized into six groups, independently by 3 evaluators, based on Bloom"s taxonomy (Remember, Understand, Apply, Analyze, Evaluate and Create). One evaluator was from the discipline concerned while the other two were independent. Categorization of the questions was based on the wording of each question and the context in which the relevant subject matter was presented. Where there was disagreement between evaluators there was discussion with consensus, subsequently. The questions were then clustered further into 3 broad levels of cognition: questions requiring lower order thinking skills (Remember, Understand: LOTS), middle order thinking Skills (Apply: MOTS) and higher order thinking skills (Analyze, Evaluate, Create: HOTS). Particular attention was paid to the proportion of questions at each level of cognition for each discipline, and at the preclinical, para-clinical and clinical components of the medical program.

Discussion
The stark predominance of LOTS questions in most disciplines was a key finding of this study. Formal and informal discussions with relevant teachers revealed many factors which may be responsible for this situation. Firstly, questions requiring knowledge and comprehension could be evaluated more objectively in comparison to questions requiring HOTS, especially when assessing a large numbers of students. It is believed that inclusion of knowledge-based questions would not only ensure ease of evaluating answer scripts but also provide an objective and fair evaluation with little room for assessor"s bias regarding the student"s view on the subject. Teachers" negative perception of students" English language ability and their higher order thinking skills could also be speculated as a reason for the inclusion of questions requiring LOTS. It is likely that the basis for this view of the teachers is the fact that Sinhala or Tamil has been the medium of instruction for examinations for University entrance, for a majority of medical students; English being the second language. This view is further supported by a study conducted by Babapulle et al., in 2003, which revealed that the English language ability of a majority of students facing the Forensic Medicine examinations in the University of Peradeniya, Sri Lanka, were "barely competent" or less 10 . The fact that the students follow a didactic system of instruction in school too may result in the teachers presuming that students are incapable of responding effectively to questions which require HOTS.
The progression of the medical curriculum is from preclinical where students are familiarized with concept of basic science, to para clinical where they are exposed to applied sciences, and finally to the clinical component where they learn the practice of medicine. This requires the students to progressively develop their higher order thinking skills in order to practice medicine successfully. While a majority of essay questions of all three components of the medical course (pre-clinical, para-clinical and clinical) required mainly LOTS, the highest proportion of questions requiring LOTS are seen in the para-clinical component. Even though the proportion of application type questions (MOTS) increases from the pre-clinical to the clinical part of the course, the proportion of questions requiring HOTS decreases from the preclinical to the clinical part of the course. However, it is important to note at this stage that the results may be skewed due to the nature of questions from single disciplines influencing the results.
Furthermore it is necessary to keep in mind the fact that essay questions are not the only means of evaluation in these disciplines, and that other forms of assessment such as MCQs, OSPE, clinical cases and viva voce examinations may be addressing HOTS of students. Sequeira (2006) has categorized knowledge, comprehension and application as "Lower Order Thinking Skills" and analysis, synthesis and evaluation as "Higher Order Thinking Skills", while others categorized application, analysis and synthesis as requiring higher order thinking. 7 , 8,9 Even though a clear distinction has been made in this study between LOTS and HOTS, the literature reveals that this categorization is somewhat arbitrary, considering the cognitive levels of the Blooms taxonomy as a spectrum and that the basis for categorization is variable. Palmer et al., have adopted a classification with three broad cognitive levels, namely, knowledge (Level I), comprehension and application (Level II), and analysis synthesis and evaluation (level III). 8 Due to the ambiguity in the literature regarding the categorization, the importance of application of knowledge in the practice of medicine and the fact that questions requiring the skill of application assumes an important place in most disciplines of the selected Faculty of Medicine, it was considered appropriate to categorize questions requiring the skills of application as those necessitating "Middle Order Thinking Skills". It is important at this stage to mention that the categorization of the questions was not based purely on the wording of each question, but also the context in which the relevant subject matter was presented to the student. For example, even though the development of a protocol for a particular procedure would amount to a question requiring HOTS, if the same has been discussed in class it would become a question requiring purely recall, and therefore requiring LOTS. This required extensive analysis of the course material in addition to the examination questions and the contribution of evaluators from relevant departments.
Examination questions in this institution are formulated based on the objectives of a particular module and therefore are reflective of what is expected by students at the end of a particular course. Perusal of objectives at the end of each year revealed that the examination questions were well aligned with the objectives, indicating that it may be necessary to revise the learning objectives prior to improving the standard of questions. However, some may argue that since the objective of the MBBS course is to produce a first contact doctor, the objectives may need to be stated to cover more breadth than depth and therefore justifies a lower cognitive level. It is also important to identify that students following the medical program are evaluated in many other ways than purely by essay questions (MCQ, OSPE, OSCE, case presentations/discussions), both during the semester and at the end of each semester. It may be assumed that HOTS are assessed at these examinations. However, it must be acknowledged that it is appropriate to stimulate students to utilize HOTS at all levels of the academic program and assess the same at all types of examinations, including written examinations.
Nonetheless, simply increasing the number of questions requiring middle and higher order thinking level may not be sufficient. It may be necessary to guide and facilitate students to respond to such questions appropriately, in advance, by conveying the expectations explicitly to students and by training students. It may also be important for the teachers to develop a more open minded and receptive attitude towards students" view on subject matter, with encouragement of higher order thinking. Since a question that demands higher level of cognition will most likely be "open ended", and will receive a wide variety of answers, teachers must be prepared to entertain and critically evaluate different viewpoints of students. However, the feasibility of this approach in a setting where a large number of answer scripts need to be marked within a short period of time may force teachers to strike a balance between questions requiring lower and higher order thinking skills.
It is envisaged that this study would create awareness among individual disciplines regarding the cognitive level of their examination questions and would help to reflect on the academic program of respective disciplines and thereby to improve the medical program as a whole. The process of categorizing examination questions according to the cognitive level may be useful to consider in any educational institution, to identify the potential educational impact of questions. It is especially useful in institutions where large numbers of students are evaluated and where the medium of evaluation is not the first language of the students. This is particularly so because questions requiring HOTS need a reasonably high degree of language ability, which the students in such institutions may not possess. Furthermore, it may also be reasonable to assume that questions requiring LOTS may be used by the teachers especially in the context of evaluating large numbers of students. Therefore, it may be recommended that such institutions evaluate the cognitive level of their questions to prevent a drift towards questions requiring LOTS.

Conclusions
Essay questions requiring lower order thinking skills (LOTS) predominate in most disciplines and at all levels of the medical course. The proportion of questions requiring middle order thinking skills (MOTS) increase from pre-clinical to the clinical part of the course while the proportion of questions requiring higher order thinking skills (HOTS) decrease from the pre-clinical to the clinical part of the course.

Recommendations
Evaluating the cognitive level of examination questions of academic programs can be recommended as a good practice for higher education institutions, since such evaluations generate valid evidence to reflect upon relevant curriculum features and institutional commitments that may influence development of higher order thinking among their students.