Analysis of Finnish chemistry Matriculation Examination questions according to Cognitive Complexity

This paper presents an analysis of Finnish chemistry matriculation examination questions according to cognitive complexity. The research data consisted of 257 questions from 28 matriculation examinations between 1996 and 2009. Qualitative approach and theory-driven content analysis method using Bloom’s revised Taxonomy of Cognitive Objectives were employed in the research. The categories of the higher-order (HOCS) and lower-order cognitive skills (LOCS) were formed on the basis of earlier research. This research was guided by the following question: What kinds of cognitive skills and knowledge do Finnish chemistry matriculation examination questions require? The research indicates that the examinations were cognitively demanding. The majority (77%) of the questions required higher-order cognitive skills. The Bloom’s revised Taxonomy of Cognitive Objectives as used in this research gives a useful way for designing or analysing chemistry summative assessment tools. All three higher-order cognitive skills (HOCS) categories should be more evenly presented in chemistry matriculation examinations. Introduction Assessment lies at the heart of the chemistry education. As is well known, teachers teach and students study towards success in tests (e.g. Tamir, 2003). Thus, assessment points to what is considered relevant and ignores what is perceived to be unimportant (Doran, Lawrenz & Helgeson, 1994). Student assessment in chemistry education is often based on summative assessment. It’s a predictive and comparable type of assessment that also gives an overview of students’ previous learning obtained during an instructional unit (e.g. Black, 2004). In general, summative assessment is implemented at the end of an instructional unit to measure and document student achievement in proportion to other students’ performances, or some predetermined instructional standards (e.g. Doran et al., 1994; McMillan, 2008). Various tests and examinations are typical summative assessment tools (McMillan, 2008). However, summative assessment may also include assessment of practical work (e.g. laboratory tasks), or different products (e.g. portfolios, research reports) (e.g. Miller, Linn & Gronlund, 2008). Greta Tikkanen analysed Finnish chemistry matriculation examination questions in her PhD research at the Chemistry Teacher Education Unit at the University of Helsinki. She currently lives in New Zealand. Maija Aksela is a professor and the leader of the Chemistry Teacher Education Unit at the University of Helsinki. She teaches at the Chemistry Teacher Education Unit in Helsinki and also does research in the field of chemistry education. GRETA TIKKANEN The Chemistry Teacher Education Unit, University of Helsinki, Finland MAIJA AKSELA The Chemistry Teacher Education Unit, University of Helsinki, Finland Maija.Aksela@helsinki.fi


Introduction
Assessment lies at the heart of the chemistry education.As is well known, teachers teach and students study towards success in tests (e.g.Tamir, 2003).Thus, assessment points to what is considered relevant and ignores what is perceived to be unimportant (Doran, Lawrenz & Helgeson, 1994).Student assessment in chemistry education is often based on summative assessment.It's a predictive and comparable type of assessment that also gives an overview of students' previous learning obtained during an instructional unit (e.g.Black, 2004).
In general, summative assessment is implemented at the end of an instructional unit to measure and document student achievement in proportion to other students' performances, or some predetermined instructional standards (e.g.Doran et al., 1994;McMillan, 2008).Various tests and examinations are typical summative assessment tools (McMillan, 2008).However, summative assessment may also include assessment of practical work (e.g.laboratory tasks), or different products (e.g.portfolios, research reports) (e.g.Miller, Linn & Gronlund, 2008). [259]

8(3), 2012
According to the current view based on constructivism, assessment is supposed to tap students' higher-order cognitive skills (HOCS) such as their ability to select and apply conceptual knowledge in new situations or solve real-life problems (e.g.Zoller & Pushkin, 2007).It should not merely measure students' lower-order cognitive skills (LOCS) like their ability to remember factual knowledge or solve routine algorithmic exercises.However, the traditional LOCS-type examinations very often predominate in the field of chemistry education.(e.g.Tsaparlis & Zoller, 2003) Anderson and Krathwohl (2001) constructed a revision of Bloom's Taxonomy, which they termed the Taxonomy Table (Table 1).It is a two-dimensional scheme applicable for classifying chemistry learning objectives and examination questions in Cognitive Process and Knowledge dimensions.The noun and verb aspects of the analysed data (e.g.examination questions) are utilised in the classification process.The noun provides the basis for the Knowledge dimension, and the verb forms the basis for the Cognitive Process dimension (Krathwohl, 2002).The Taxonomy Table is a hierarchy in which the cognitive complexity is assumed to increase from left to right in the Cognitive Process dimension.Similarly, the categories of the Knowledge dimension are assumed to lie along a continuum from concrete (Factual) to abstract (Metacognitive).However, the hierarchical structure of the revised taxonomy is not as strict as in the original Taxonomy, and the categories are allowed to overlap one another.For example, some chemistry examination questions classified in the category Understand (e.g.questions in which students have to explain chemical phenomena) may be more cognitively complex than some questions classified in the category Apply (e.g.routine stoichiometric problems).(Krathwohl, 2002).The Knowledge categories applied in the context of chemistry are shown in Table 2.

Category Definition Examples
A.

Factual knowledge
Knowledge of terminology, specific details and elements.

Analysis of Finnish chemistry Matriculation Examination questions
[260]

8(3), 2012
The Cognitive Process dimension of the Taxonomy Table is divided into six main categories: Remember, Understand, Apply, Analyse, Evaluate and Create (Krathwohl, 2002).The Cognitive Process categories applied in the context of chemistry are shown in Table 3.
In recent years, much worldwide research pertaining to cognitive skills and knowledge has been done in the field of chemistry education (e.g.Aksela, 2009).Karamustafaoĝlu, Sevim, Karamustafaoĝlu and Çepni (2003) analysed Turkish high-school chemistry examination questions in terms of the levels of cognitive domain of Bloom's Taxonomy.The research was carried out in three types of high schools: 'Ordinary', 'Vocational' and 'Anatolian' from two Turkish cities with 17 chemistry teachers in 2000-2001.403 examination questions were analysed in the research.The research indicates that 96% of the chemistry examination questions were of the LOCS type, and required only knowledge, comprehension or application.Only 4% of the questions were at the higher level of cognitive domain (analysis, synthesis and evaluation levels).The research also shows that the question types were related to the school type (Karamustafaoĝlu et al., 2003).Krathwohl, 2002, 215).

Greta Tikkanen and Maija Aksela
[261] 8(3), 2012 Zheng, Lawhorn, Lumley and Freeman (2008) classified biology examination questions according to Bloom's Taxonomy in the United States.The examinations analysed in the research came from five sources: AP Biology, introductory biology courses for undergraduate majors from three universities, the biology section of the MCAT (Medical College Admission Test), the biology Graduate Record Examination (GRE), and five first-year medical school courses from an institution with a traditional curriculum.The research shows that the majority of the analysed questions measured lower-order cognitive skills.A significant proportion of the questions was at the comprehension-level.The research data also included several questions requiring application.Analysis was required in all the examinations, but the proportion of those questions was distinctly smaller in the first-year Medical School examinations.Very few questions required students to synthesise, and questions at the evaluation-level were not included in the research data at all (Zheng et al., 2008).Tsaparlis and Zoller (2003) investigated students' performance in chemistry/science examinations, requiring HOCS or LOCS skills, at the high school and university levels in Greece and Israel.The research indicates that the chemistry examination used for entry into higher education in Greece selects the best LOCS-performing students as LOCS-type of questions were emphasised in the examination.
A different pattern of students' performance on examination questions requiring HOCS compared with that on questions requiring LOCS was also found in the research.It was revealed that a high performance on the latter does not necessarily guarantee a high performance on the former.Also, many students didn't perform any better on the supposedly easier LOCS questions, compared with their performance on HOCS questions.This finding was attributed to insufficient pre-examination preparation/study based on the analysis of the research data.In an Israeli study, which was conducted within an introductory freshman general and inorganic chemistry course, it was found that, when the top performing students were given a free choice between HOCS-and LOCS-type questions, they preferred to select and answer the LOCS-type questions.This finding indicates that a short-term HOCSoriented instruction is not sufficient for changing students' 'exam-attitudes/behavior' with respect to 'LOCS vs. HOCS learning'.(Tsaparlis & Zoller (2003) The HOCS and LOCS examination items were defined in the Greek and Israeli studies as follows: HOCS items are quantitative or qualitative, ill-defined/structured, or open-ended questions, mostly unfamiliar to the student, which require for their 'solution' much more than just knowledge and/or application of known algorithms; they may require analysis, synthesis, system thinking, decision making, problem-solving capabilities, but mostly the making of connections, and critical evaluative thinking.This includes the application of known theory or knowledge to unfamiliar situations or situations with an unusual element or dimension.LOCS items are knowledge questions that require simple recall of information or a simple application of known theory or knowledge to familiar situations and context.They can also include the so called 'problems', mostly computational exercises, solvable by the application of taught/recalled/known algorithms, not necessarily understood by the 'solver', which are already familiar to the learner through previous specific directives, or long-term practice, or both (Tsaparlis & Zoller, 2003, p.51).
The matriculation examination is the dominant summative assessment tool in Finnish upper secondary schools.It consists of at least four tests in different subjects that are held at the same time in all Finnish upper secondary schools every spring and autumn.Chemistry test is one of the optional tests.
The test questions and the structure of the tests are designed by the members of the Matriculation Examination Board on the basis of the goals and contents defined in the national Curriculum for the upper secondary schools.The matriculation examination is a very traditional and highly valued institution that has a strong influence on both teaching and learning of chemistry in Finnish upper secondary schools (e.g.Aksela & Juvonen 1999).Therefore, it is very important to find a detailed answer to the following question: What kinds of cognitive skills and knowledge do the chemistry tests of the Finnish Matriculation Examination measure?

Method and Reliability
The research data consisted of 257 chemistry questions from 28 matriculation examinations between 1996 and 2009.Each chemistry matriculation examination contains eight questions in 1996-2005.
Since spring 2006 the examinations contain 12 questions, except for the examination held in spring 2008 that exceptionally contains 13 questions.The 257 questions are all of the questions in these 28 examinations.Hence, none of the chemistry matriculation examination questions between 1996 and 2009 were excluded from the analysis.
Qualitative approach and theory-driven content analysis method were employed in the research (e.g.Cohen, Manion & Morrison, 2007).The chemistry examination questions were classified in the Cognitive Process and Knowledge dimensions of the Taxonomy Table (Table 1).The revised Bloom's Taxonomy was used in this research because it's suitable for analysis of assessment items (e.g.Näsström, 2008) and has been developed on the basis of the current educational research literature (cf. Anderson & Krathwohl, 2001).The division of test questions into LOCS and HOCS questions (cf.Tsaparlis & Zoller, 2003) was also applied in the research.
In this research, it is assumed that the lower-order Cognitive Process categories are included in the higher categories.For example, it is assumed that the examination questions classified into the category Analyse require also cognitive processes Remember, Understand, and Apply.The Cognitive Process categories of the Taxonomy Table do not form a cumulative hierarchy like the categories in the original Bloom's Taxonomy.For example, the most challenging questions that require understanding are cognitively more demanding than the easiest application tasks.However, it is justified to use the aforementioned definition because the "central points" of the categories of Cognitive Process form a hierarchy (cf.Anderson & Krathwohl, 2001).
The questions are always classified into the highest possible category of Cognitive Process.For example, all the examination questions requiring students to plan a laboratory method/demonstration are classified into the Cognitive Process category Create.It is more than likely that these questions require only remembering or using other lower-order cognitive skills from some students because those same laboratory experiments may have been conducted in the chemistry lessons.However, they probably require creating from the students who are not familiar with those experiments, so they are classified into the Create category.
In this research, it is also assumed that the lower-order Knowledge categories are included in the higher categories.It is justified to assume that the examination questions requiring procedural knowledge also require at least some factual and conceptual knowledge.For example, the questions in which students have to plan a laboratory method require Procedural knowledge (knowledge of the method), Conceptual knowledge (e.g. the knowledge of the chemical basis of the method), and Factual knowledge (e.g. the knowledge of the chemical formula of the reagents).
There are two main phases in the research.In the first phase, chemistry matriculation examination questions are classified into the six Cognitive Process categories of the Taxonomy Table (Table 1).The three lowest categories of the Cognitive Process dimension are defined as lower-order cognitive categories (LOCS) in this research.The three highest categories are defined as higher-order categories (HOCS) (Table 3).In the second phase of the research, the test questions are classified into the four Knowledge categories of the Taxonomy Table (Table 2).The classification of the chemistry matriculation examination questions with the Taxonomy Table is challenging and partially interpretative.
Hence, the definitions of the LOCS and HOCS items described in the research literature (cf.Tsaparlis & Zoller, 2003, p.51) are also concurrently used to facilitate the classification process.This supporting framework is especially useful while analysing questions, which lie at the cusp of the categories Apply and Analyse of the Taxonomy Table (Table 3).

8(3), 2012
In this research, the examination questions that may require students to use also higher-order cognitive skills in addition to applying routine algorithms or memorising are classified into one of the three highest Cognitive Process categories (Analyse, Evaluate or Create).The questions that only require simple recall of information or known/well-rehearsed routine applications are classified into one of the three lowest Cognitive Process categories (Remember, Understand or Apply).Typically, it is possible to answer these examination questions (e.g.simple well-rehearsed chemistry stoichiometric problems) fluently without understanding the solving process.This may lead to distortion of the grading system when students who don't understand the reasoning behind the solution score equally high marks in examinations as students who have a deep understanding of how the solution has been reached.
Students' background (e.g. the quality of instruction, number of optional chemistry courses taken in upper secondary school, preparation time for the test) has undeniably a major impact on the cognitive complexity of the examination questions (e.g.Zoller & Tsaparlis, 1997).For example, the students who have thoroughly practiced the solving processes of all the basic quantitative upper secondary school chemistry problems won't probably need to use any higher-order cognitive skills (e.g.analysing) while solving similar basic problems at the examination.On the other hand, the students who haven't prepared that well for the examination cannot solve even the simplest problems routinely, but have to use higher-order cognitive skills during the solving process.Within this research, it is impossible to take students' different backgrounds into consideration.Therefore, it is assumed that all the examination questions are so-called new questions for the students.Thus, the classification of the examination questions is based on the assumption that students have not solved exactly similar tasks in the chemistry lessons nor when preparing for the examinations.Examples of the classification process of the examination questions are shown in the Appendix tables at the end of this paper.
Peer review was used to guarantee the reliability of the research.10% of the examination questions were picked at random and analysed by another research scientist who had specialised in the field of chemistry education and had a deep understanding of the revised Bloom's Taxonomy and its usage as a classification tool.The classification procedure was demonstrated by categorising some example questions together prior to the peer review to achieve good common understanding about the classification process.The value of Cohen's kappa coefficient was calculated based on the results of the peer review.The kappa-values were calculated separately for the classifications in Cognitive Process and Knowledge dimensions.The Cohen's kappa value for the Knowledge classification was 0,924, and for the Cognitive Process classification 0,801.These high kappa values (κ > 0,75) for both classifications indicate an excellent inter-rater agreement between the raters, and thus high reliability for the research (cf.Cohen et al., 2007).

Results
The majority (77%) of the chemistry matriculation examination questions required higher-order cognitive skills in 1996-2009.The distribution of the test questions is shown in Table 4.
The largest percentage (35%) of the chemistry matriculation examination questions required analysing.The examination questions requiring lower-order cognitive skills distributed quite evenly into questions at the Understand (13%) and Apply (11%) level.The researched examinations did not include questions that only required remembering.The majority (79%) of the examination questions required students to have procedural knowledge.A significant proportion of the questions (21%) required students to have conceptual knowledge.The examinations did not include questions that required only factual knowledge.None of the questions required metacognitive knowledge either.

Analysis of Finnish chemistry Matriculation Examination questions
[264] 8(3), 2012 One third of the chemistry matriculation questions (33%) required analysis of procedural knowledge (e.g.question 4 in appendix).They were typically quantitative problems.The tests also included questions pertaining to chemical reactions, and requiring interpretation of graphic representations.
Creating and having procedural knowledge were required in several examination questions (20%) (e.g.question 5 in appendix).They were typically tasks in which students have to plan laboratory experiments, or essay questions pertaining to technology or environmental chemistry.Some essay questions also required interpretation of graphs (e.g.question 6 in appendix).The percentage of the test questions requiring evaluation of procedural knowledge was 16%.They typically included finding the limiting reagent, or making approximations in quantitative problems.Some questions required drawing all possible structural isomers for organic compounds.Evaluation of laboratory methods was required in a few examination questions.
Conceptual knowledge was required in 13% of the examination questions (e.g.question 1 in appendix).They were typically questions in which students have to explain chemical concepts or phenomena.11% of the examination questions required students to apply procedural knowledge (e.g.question 2 in appendix).They were usually questions pertaining to laboratory experiments, or questions in which students have to write structural formulas or perform routine calculations.The research data also included questions pertaining to real-life applications, and to use of the Periodic Table .The percentage of the examination questions requiring creating and conceptual knowledge was 7%.They were essay questions pertaining to chemical concepts and phenomena (e.g.question 8 in appendix).
Only 4% of the research data required students to analyse and have conceptual knowledge.They were questions in which students have to differentiate and compare concepts and chemical compounds.Several examples of different types of LOCS and HOCS questions are presented in the Appendix tables at the end of this paper.

Conclusions and Discussion
The research indicates that chemistry matriculation examinations are cognitively demanding.The proportion of the Finnish chemistry matriculation examination questions requiring higher-order cognitive knowledge and skills is very large compared to what is discussed in the research literature (cf.Karamustafaoĝlu et al., 2003;Zheng et al., 2008).Typically, most of the examination questions used in summative assessment require only lower-order cognitive skills (cf.Tsaparlis & Zoller, 2003).
The research shows that the majority of the examination questions required higher-order cognitive skills.Most of the questions required analysis of procedural knowledge.Least of the questions required evaluation or creation of conceptual knowledge.Thus, all three higher-order cognitive skills (HOCS) categories were not evenly presented in chemistry matriculation examinations, which should be taken into consideration in the process of test designing to increase the variety of the future chemistry examination questions.The examinations didn't include any questions that only required remembering.Also, not any questions that required processing of metacognitive knowledge were found in the research.This can be considered as a reasonable result because it is very challenging to assess the metacognitive levels or their development by using a summative assessment tool like matriculation examination (cf.Krathwohl, 2002).
Chemistry matriculation examination questions must be sufficiently challenging, and suitable for separating the wheat from the chaff (cf.Atjonen, 2007;Kraska, 2008).The Bloom's revised Taxonomy of Cognitive Objectives as used in this research gives a useful way for designing or analysing chemistry summative assessment tools such as matriculation examinations.The research shows that chemistry matriculation examination questions are well suited for the assessment of higher-level cognitive skills and knowledge levels.However, it is important to take into account the major impact of students' different backgrounds on the cognitive complexity of tasks (cf.Krathwohl, 2002;Zoller & Tsaparlis, 1997) when designing the future chemistry matriculation examinations.
Traditional LOCS-type chemistry examination questions can be utilised in the designing process of future HOCS-type chemistry matriculation examination questions.One way to increase the cognitive complexity of routine quantitative problems is to add short verbal sections into questions that require students to evaluate if the answer and/or the solving process makes sense and is meaningful.Also, it's possible to include previously unknown ancillary material (e.g.articles, charts, tables, pictures) in the questions when the examinations will more likely measure students' higher-order cognitive skills than rote memorisation of textbook content, or well-rehearsed routine skills (cf.Zoller & Tsaparlis, 1997).As well, it can be useful to sometimes add unnecessary diversions of input data to the questions when students have to really analyse the task to distinguish the essential content from all given information (cf.Krathwohl, 2002).On the other hand, the cognitive complexity may be increased by not presenting all the necessary information (e.g.solubility or acid and base ionisation constants) in the examination questions when students need to first analyse the situation and then find all the required information in an ancillary data table.
Table is divided into four main categories: Factual knowledge, Conceptual knowledge, Procedural knowledge and Metacognitive knowledge

Table 2 .
The Knowledge dimension of the Taxonomy Table applied in the context of chemistry (e.g.

Table 3 .
The Cognitive Process dimension of the Taxonomy Table applied in the context of chemistry (e.g.

Table 4 .
Distribution of the chemistry matriculation examination questions (n = 257) according to the highestKnowledge and Cognitive Process levels in 1996-2009.