The effectiveness of integrating existing digital learning resources into classroom teaching – an evaluation of the learning achievement

8(2), 2012 Abstract This paper presents the results of an intervention study exploring the effectiveness of an implementation of a selection of digital learning resources (DLR) in lower secondary science classrooms. Eight teachers participated in a quasi-experimental intervention, teaching three different subjects with and without DLRs. The data presented are from pupil tests before and after the teaching combined with observation of lessons and interviews with both pupils and teachers. In two of the three subjects the groups using DLRs did slightly better than the comparison group. The results indicate that successful ICT based learning is just as dependent on teacher competence as the quality and features of the resources.


Introduction
Computers and technology are becoming an ever growing presence in the daily lives of people, even more so for young people in the world of social networks and mobile devices.A recent report shows Nordic youth to be technology savvy with 77% of secondary pupils saying the internet is important in their social lives and a large majority of them spending over two hours a week using a computer at home (Guðmundsdóttir, Sigfússon, Kristjánsson, Pálsdóttir & Sigfúsdóttir, 2010).Schools on the other hand are finding it hard to keep up, information and communication technology (ICT) is still only used to a small extent in compulsory education, and the same report shows over 50 % of pupils in Finland, Iceland and Ålands Islands using them weekly or less but other countries slightly more.In Iceland previous small scale research indicates that science specific requirements have only been met to a limited extent and that ICT is rarely used by students in science teaching in compulsory schools in Iceland (Þórólfsson, Macdonald & Lárusson, 2009).This is confirmed with a large scale survey of ICT use in Iceland in an earlier phase of the study reported here.It showed that ICT use almost entirely consists of generic uses such as searching the internet for information for essay writing and other forms of presentation, watching videos and taking photos.
In science education specifically a growing body of research (Osborne & Hennessy, 2003;Holliman, & Scanlon, 2004) shows that ICT is known to offer many affordances.These affordances come both The effectiveness of integrating existing digital learning resources into classroom teaching -an evaluation of the learning achievement Svava Pétursdóttir is an EdD candidate at the University of Leeds, the topic of her thesis are the uses of ICT in science education.She also has affiliations with the research group on science education at the University of Iceland.Before her studies she taught science and mathematics in compulsory schools in Iceland for 15 years.

SVAVA PÉTURSDÓTTIR
University of Leeds ed07sp@leeds.ac.uk [151] 8(2), 2012 from generic use such as gathering information, analysing and presenting it and also from science specific uses like simulations and multimedia.Simulations and multimedia are believed to have value as visual aids.They have the ability to bring into the classroom phenomena that are too fast, slow, small or dangerous for real life and can help with understanding of abstract concepts (Wellington, 2004).
Furthermore indications are that ICT use leads to higher levels of student involvement and motivation of students (Cox, 2000;Wellington, 2004).Cox (2000) has argued that ICT use may keep the school up-to-date, introducing students to methods of industrial science.Baggott and Nichol (1998) argue that simulations are a form of knowledge representation, with students using high level skills in working with them, and are not an imitation or replacement of lab-based practical work.Simulations have been found to promote conceptual change in science (Hennessy, et al., 1995;Gallop, 1995;Cox, 2000), a better ability to predict and explain (Zacharia & Anderson, 2003) and help articulate a better understanding (Marbach-Ad, Rotbain, & Stavy, 2008).Mostly these studies report a small positive effect but studies with mixed outcomes are also to be found.The studies listed here all contribute to a growing understanding of the effects ICT can have on learning and conceptual understanding.Nevertheless a recent review argues that evidence showing that ICT is beneficial to learning is still scarce (Livingstone, 2011).
The question addressed in this study is whether there are measurable differences, linked to conceptual understanding and knowledge retention, between an intervention group using ICT-based science teaching interventions and a comparison group being taught in their usual way.The study explores the effectiveness of a selection of available DLRs in Icelandic compulsory school settings.

Research design and activities
Interventions with a quasi-experimental design were carried out with eight participating volunteering science teachers in compulsory schools in Iceland.Three topics were offered and each teacher took part in 1 -2 topics, with either one or two classes, as outlined in Table 1.The data presented are from 20 classes, with the number of pupils completing both pre-and post-tests ranging from 13 to 28 giving a total of 395 students aged 13-16.The participating teachers are given female pseudonyms to preserve anonymity even though two of them were male.The effectiveness of integrating existing digital learning resources into classroom teaching [152] 8(2), 2012 Five of the teachers were introduced to and used seven different DLRs readily available online, in three different topics from the science curriculum.The comparison teachers were asked to teach as they would normally do using the resources they are used to.Teaching in these groups consisted of a mixture of lecturing and note taking, discussions, and written questions and exercises from the textbooks.The materials the intervention teachers were provided with consisted of a list and description of possible ways of using the suggested DLRs and worksheets that either came with the websites or were prepared for the intervention.All the teachers in the interventions based their teaching on the same textbooks but the intervention teachers also gave time for use of DLRs.
The topics were: balancing chemical equations, ecosystems and basic genetics; each taught for 4-6 weeks with DLRs used in 2-6 lessons.Before the intervention pre-tests were administered consisting of known tasks from the literature, probing pupil ideas about the topics.The pre-tests were designed by choosing and adapting tasks from research exploring children's ideas about the particular phenomena.
The genetics pre-test explored the ideas pupils have about inheritance, the role of DNA, genes, cells and chromosomes in inheritance, their location and size (based on Solomon, Johnson et al., 1996;Venville, et al., 2005;Lewis & Wood-Robinson, 2000;Saka, et al., 2006).The chemistry pre-test had three tasks; one from Stavridou & Solomonidou (1998) where pupils are asked to categorise 19 instances of everyday phenomena, ten of which represent chemical change and nine physical changes.The second section was a collection of multiple choice questions about conservation of matter (Holding 1985) and the third a task about recognising a diagram of a chemical reaction (Agung and Schwartz, 2007).The ecology pre-test was about forming an ecosystem (from Leach, 1995).Pupils were asked to choose six organisms from a list to form a food web.Three open questions were also included probing pupils' understandings about the workings of ecosystems.
After the intervention pupils took tests compiled for this study consisting of most of the same tasks from the pre-test plus tasks and multiple choice questions from the textbooks.All the tests were administered by the teachers and the post-tests were part of their regular assessment.This was considered important so pupils would take the tests seriously.To make sure the questions were valid and appropriate for the pupils, teachers were asked for comments both about content and the length of the tests.The tests were scored on a scale of 1-10 and results were analysed for statistical differences using the software SPSS.
During the intervention at least two lessons were observed for each class, monitoring both how the teachers made use of the DLRs and how pupils responded and interacted with them.The observed lessons were also videotaped for further reference.After the intervention a group interview was conducted with 4-5 pupils asking them about their thoughts on the different tasks they had done during the teaching.All through the intervention close contact was maintained with all the teachers, with interviews before and after the participation and shorter interviews after the observed lessons.

Choice of topics and DLRs
As Iceland is a small country with a limited supply of teaching materials in Icelandic it was an important question whether any of the vast number of resources available on the web could be used successfully with Icelandic pupils.When choosing the DLRs for the intervention a number of criteria were applied.The resource had to be readily available and free of cost; the resources should have different foci and preferably demonstrate one of the aforementioned affordances from the literature; they should preferably be in Icelandic or involve little use of language as a previous phase of this study had shown that teachers had doubts about using DLRs in English.The resources should target different topics from the National Curriculum and be topics important enough to be included in pupil assessment.A further consideration was that they would target different age groups which would make it possible for a teacher to participate in more than one topic.

Classical genetics
The genetics website http://www.erfdir.is was used by all intervention classes (Erfðavísir, Figure 1).
Baggott and Nichol (1998) criticized many CD-ROMs for being little more than books on screen and this website (Figure 1) has mostly reading material but includes a small number of pictures and diagrams that move and instant responses in quizzes.Exploring whether additional visual features besides reading from a computer screen is sufficient to improve learning is the question that influenced the decision to include this website in the intervention.A worksheet consisting of questions and tasks was written to help pupils navigate the web, give the lesson a focus and keep the pupils on task.

The effectiveness of integrating existing digital learning resources into classroom teaching
Teachers were also introduced to the simulation Bug Lab, an interactive simulation with built-in tasks or experiments for pupils to solve.The simulation is in English and has many features where the user can manipulate the bugs' environment, choose bugs to see their genotype and see phenotype ratio as the simulation runs (see Figure 2). [154] 8(2), 2012

Chemical reactions and balancing chemical equations
Chembalancer (http://funbasedlearning.com/chemistry/chembalancer/default.htm)used in the chemistry intervention was a simple exercise in balancing chemical equations (Figure 3).Accompanying the website is a worksheet that was modified and translated into Icelandic.This topic is usually considered hard by pupils and in previous phases of this study teachers also mentioned it is as difficult to teach.Including it in the intervention explores whether such a simple resource can be helpful in a task considered difficult.

Eco systems
The main website used in the ecosystems intervention is called Sunny Meadows (http://puzzling.caret.cam.ac.uk/game.php?game=foodchain%20).It is a game-like simulation where pupils enter the number of foxes, rabbits and grass and then the website simulates how the species fare over a 50 year period (Figure 4).This resource was chosen because of its game characteristics, visual representation and low language requirements.Furthermore this is an example of an experiment that could never be done in a classroom but here the technology makes is possible to at least get a basic representation of how an ecosystem works.A worksheet was prepared where pupils were expected to record their progress in keeping the species all alive. [155] 8(2), 2012

Results
For all the tests a score was calculated on a scale from 0-10, and for each intervention results for the separate classes are presented.The classes are then combined into treatment conditions to look for general tendencies.An independent sample t-test tests for differences between the intervention group and comparison group.The size of the effect of the independent variable, if any, is determined by Cohen´s d.For the purposes of this analysis, a Cohen´s d value between 0.01 and 0.30 is considered a small effect, 0.31-0.50 a medium effect and all values over and including 0.51 are a large effect.In the next sections a brief description of the observed lessons are given and then the results from tests.

Genetics
Though the DLRs used in the genetics intervention looked promising to start with they did not have measurable effects on pupil achievement.Four classes used Erfðavísir and were all observed while doing so, in all classes very little off task behaviour was noted, except in school 100, but no more than in a classroom lesson previously observed.Most pupils managed to finish the worksheet.Anna (200) took her class twice to the computer room to read through this resource, the second time also using the worksheet.She said there was a notable difference in how pupils worked during the lessons with them, working better with worksheets.Still she had not prepared pupils for what they were supposed to do in the computer rooms and considerable time went into guiding pupils to start the tasks.In school 400 there was a marked difference in pupil activity where they paid little attention to lectures and discussions in the classroom with lots of off task behaviour, but worked very well the whole time on Erfðavísir but were uninterested when back in the classroom where the teacher reviewed the answers.
Olga (400) and Eva (100) chose to spend one lesson manipulating bugs in the simulation BugLab.Anna (200) however decided against it saying it seemed too complicated.In the observed lessons pupils could be heard having fun and using genetic concepts when working together.A good amount of chatter could be heard in Eva´s class but all about what they were doing.Olga in both cases had The effectiveness of integrating existing digital learning resources into classroom teaching [156] 8(2), 2012 prepared pupils for what they were supposed to do in the computer room, showing them both the DLR´s and worksheets on an interactive whiteboard.
From the genetics intervention two sets of scores are analysed, pre-test and post-test (see Table 3).
On the pre-test the mean scores ranged from 3.01 to 4.97 with a total mean of 4.03.Two comparison classes scored under the total mean and one intervention class.An independent t-test between conditions showed that the comparison classes (M= 3.87) did not score significantly lower than the intervention classes (M= 4.16), t(114.07)= 1.10, p = 0.273.
On the post-test the scores ranged from 5.05 to 7.11.Of the four intervention classes only INT210 scored higher than the total mean of 5.80 while two of three classes from the comparison condition scored higher than that.An independent t-test between conditions did not show a statistically significant difference.The intervention group even scored lower (M= 5.80) than the comparison classes (M= 6.02), t(85.38)= 0.78, p = 0.437.For the post-test Cohen's d, the effect size is -0.48 a large effect.
The classes scoring under the total mean on the pre-test are exactly the ones scoring over the total mean on the post-test.These three classes all raised their score well over 3 points or close to doubled their score whereas Comp710 gained 25 % and the other three less.This may have many explanations, among them that teachers made different use of the pre-test.It seems from responses that questions that were both on the pre-and the post-test were discussed or taught specifically between the tests in some classes.An example is a question about ordering genetic components such as a gene and chromosome and cell by size.On the post test only 18% of INT210, 4% of Comp710 and no pupils from classes INT410 and INT110, ordered all components correctly.On the other hand was school 800 where 15% of them ordered correctly on the pre-test and 44% on the post test.
In essence the post-test revealed that the classes using the websites scored lower than the classes not using the websites.This seems to indicate that the websites did not help pupils learn basic genetics.It may even be said that using these websites has had a detrimental effect, with only one of the intervention classes improving their score considerably.The teachers had reservations about the uses of the DLRs used but the majority of pupils talked positively about them but mostly in the sense that they liked working with computers and doing something different but not about learning from them nor finding them useful.The large difference in improvement between the classes especially the two com-

Svava Pétursdóttir
[157] 8(2), 2012 parison teachers indicates that the resources used are not the deciding factor in pupil achievement.
Observations of these teachers point to other things in their practice and preparation.

Chemistry
Chembalancer turned out to give the clearest positive results of all the DLRs used.Five classes used Chembalancer and were observed doing so.For some pupils it took them some time to realise how to use the website, keeping the teachers quite busy going between them and explaining, while some pupils read the instructions and worked independently.All pupils who stayed on task finished balancing the 13 equations in the website, on time, and many with time to spare.The comparison classes learned balancing the equations from teacher demonstration and then balanced the equations in the textbook.Class Comp810 also used a worksheet with similar equations as the intervention group.That lesson was observed and pupils needed help frequently when learning the method.
For the chemical intervention three set of scores are presented; a score from the pre-test, the posttest on chemical change and balancing chemical equations (Table 4).On the post-test there were four items specifically on balancing chemical equations and as the ICT tool Chembalancer works with those items a separate score was calculated for those items.
On the pre-test the mean scores ranged from 1.94 to 3.83 with a total mean of 3.27.All the classes are included as an independent t-test between conditions showed that the comparison classes (M= 3.47) did not score significantly higher than the intervention classes (M= 3.13), t(97.56)= -1.11,p = 0.269.
For balancing chemical equations mean scores ranged from 2.9 -7.18, with three of the intervention classes and one comparison class scoring higher than the total mean of 5.64.When the two conditions are compared using an independent sample t-test the intervention classes scored significantly higher (M= 6.32) than the comparison classes (M= 4.63), t(105.42)= 3.07, p = 0.003.For balancing of chemical equations Cohen's d, the effect size is 0.56.In this instance then the effect from using the website Chembalancer are considered large.The effectiveness of integrating existing digital learning resources into classroom teaching [158] 8(2), 2012 The post-test scores for the whole test are all similar, and a difference can neither be found between the classes nor between conditions.However when only the scores from balancing chemical equations are explored (four items) there is a detectable difference between conditions with the intervention groups who used the website to help practicing balancing equations, scoring on average 0.85 points higher.A further observation is that the intervention classes scored lower on the pre-tests, but ended up overtaking the comparison classes.On top of this both teachers and pupils concurred that the website was really useful.Typical comments from pupils were about the website 'really helping them learn how to balance chemical equations'.As a result it seems clear that the website Chembalancer did indeed help pupils to learn the task and to solve equations on a paper based test.

Ecology
The DLRs in the ecology intervention proved to be useful and resulted in a small significant advantage for the intervention classes.Five classes in three schools participated in this part, two intervention teachers who used all three suggested resources and one comparison teacher.In school 400 both classes were observed both in the computer room working with Sunny Meadows and in classrooms.
In school 500 however it was only possible to observe lessons in the classroom.In school 500 the teacher first demonstrated on an interactive white board before going to the computer room, pupils stayed on task and worked the whole time to keep their ecosystems running.Sunny Meadows has a game-like quality that seemed to encourage pupils to really try their best.These classes were also observed watching the cartoon Mysteries of life.The teacher wrote down the ecology concepts with Icelandic translations and then showed the cartoon, stopping to explain and translate.For the ecology intervention two sets of scores are presented, pre-test scores and post-test scores (see Table 5).
On the pre-test the mean scores ranged from 3.27 to 4.20 with a total mean of 3.83.An independent t-test between conditions did not show a significant difference between the comparison classes (M= 3.61) and the intervention classes (M= 4.06), t(91.87)= 1.52, p = 0.132.The teacher of the class with a missing pre-test gave the information that both the classes were of similar ability and also it was apparent on the pre-test that pupils generally had very little understanding of what roles the different organisms play in the food-chain, and only three of 94 pupils knew that plants are always at the bottom of a food-chain and no one wanted to add decomposers to the organism already chosen.Taking this into account it was considered right to include comparison class 431 in the post-test even though no statistical comparisons can be made on the class score.8(2), 2012 On the post-test the mean scores ranged from 4.80 to 6.30 (see Table 5.).Two of the intervention and one comparison class scored higher than the total mean of 5.63.An independent t-test between conditions did show a statistically significant difference.The intervention group (M= 5.92) did score significantly higher than the comparison group (M= 5.23), t(107.448)= 2.21, p = 0.029.For the posttest the effect size Cohen's d is 0.42 a medium sized effect.
In pupil interviews they were asked to talk about the simulation and what they had been doing.By their responses it was clear that the game-like aspects and a competition that arose in school 400 to get the highest points in the game had not distracted them so much that they were not being helped towards a better understanding of ecosystems.Pupils thought the website was both helpful and fun to use and both teachers agreed.These results indicate that the websites used in this part did help pupils learn the workings of food-chains and ecosystems, even though the difference is not large.

Discussion of results
A large variance exists between the classes in all the interventions, but that is to be expected in a quasi-experiment.Results from comparing the conditions give tentative grounds to assume that the differences found may be due to effects from the treatment.Results from the classes point towards some other variable having effects on the outcomes as the difference between classes within conditions are in some cases rather large.
The results indicate that benefits from using DLRs vary between resources.In balancing chemical equations and in the ecology intervention the intervention classes scored significantly higher than the comparison classes but in basic genetics this was not the case.The difference was slight and not statistically significant.In other studies on the effectiveness of digital learning resources the results have also been mixed and a variety of other factors have been found to be stronger.Ardac and Sezen (2002) found that the guidance the students received was the deciding factor for understanding boiling points and not the simulation use, whereas computer groups showed significant gains in recognizing variables but not the regularly taught group.Other interventions with large samples of university students (Steinberg, 2000;Hsu & Thomas, 2002) have failed to show significant difference from control groups on tests, but still Steinberg concludes that simulations are an excellent tool for conceptual change, and Hsu and Thomas detect a positive impact through interviews.The same can be said for the ecology part of this study where the positive effect was medium but interviews with pupils revealed a stronger conceptual understanding that perhaps traditional knowledge tests did not reflect.Both Ardac and Sezen (2002) and Steinberg (2000) suggest that the failure might be due to the level of interaction with the simulation that to actively manipulate it and infer knowledge is more likely to provide better results.
One interpretation of the findings is that DLRs will produce better learning results the more interactive features they contain.It seemed from the observations that the more interactive the DLR the more pupils engaged with them, and this tendency also seems to be reflected in the test results, in the sense that those that had pupils scoring higher, Sunny Meadows and Chembalancer, are highly interactive and Erfðavísir where intervention pupils scored lower, only interactive to a limited degree.
In the same way pupils had more positive views of those resources with more interactive features, as long as there are not too many complicated features.However the effect of using Chembalancer is larger.This may be due to the nature of the task.In balancing chemical equations the task is exactly the same as in the test, but as Sunny Meadows is meant to increase understanding of a complex phenomenon the test tasks were quite different from the simulation.Another interpretation might then be that that the more alike the DLRs are to the tests the more likely one is to find a difference.The results also seem to support the criticism of DLRs that mostly contain reading material that they have little to offer beyond books.
The effectiveness of integrating existing digital learning resources into classroom teaching [160] 8(2), 2012 Three of the resources used in the intervention were in English.The commentator in the cartoon speaks fast in English and some pupils thought the words had been difficult and that it would have been better in Icelandic.Still Icelandic pupils are exposed to English quite a lot and one pupil knew all the concepts from playing the computer game Spore.Ideally pupils should be able to use resources in their own language but that pupils could use effectively these resources in English suggest that using low language intensive DLRs may be a better option than not using them at all.
When the scores from genetics and chemistry interventions are explored a pattern can be seen.The mean scores of the intervention classes are similar whereas the comparison classes that teacher 800 taught score were consistently high and classes taught by teacher 700 were always lower than the total mean score.Observations showed the practices of these teachers to be quite different.Tina (800) seemed to be a highly efficient and enthusiastic teacher.Her lessons were very well prepared, organised and productive, using both textbook and other materials.Her grasp of the subject seemed excellent and the way she explained phenomena and gave realistic examples made pupils listen and engage with the topic.Comparison teacher Tara (700) however used only the textbook, the lessons were relaxed, sometimes a bit chaotic, lacking in pace and preparation.Her grasp of the subject was good but examples were few and poor and the pupils did not pay much attention to what was going on, with considerable amount of off-task behaviour observed.With these observations in mind, it seems that the expertise and impact of the science teachers may be a more important factor on pupil learning than what learning resources are being used.Hsu and Thomas (2003) in their study seemed to blame lack of explanation by the teachers on the simulation before its use, which agrees with Ardac and Sezen's (2002) study.In the study reported here care was taken to make the websites accessible to pupils and teachers were urged to familiarize themselves carefully with them before the lessons.However to make the intervention conditions the same in all schools the teachers were asked to use the websites in a prescribed manner, that is, with pupils in a computer room.That means the teacher has little chance to apply her expertise and find innovative ways in using the DLRs in teaching.That would be a subject for further study, to explore and implement more varied ways of integrating DLRs into science teaching.
It may be argued that the use of DLRs does not provide a 'dream ticket' to science teaching and learning but that the effective use of DLRs is dependent upon the strength of the teacher's pedagogical content knowledge and can be seen as a potential area of development for the competent science teacher.
This study is yet another example of how difficult it is to provide reliable evidence on the benefits and usefulness of ICT (Livingstone, 2011), which gives rise to the question whether we really need evidence.Evidence of the usefulness of textbooks is not a common subject in educational research; still they are widely accepted as learning resources.Is it not a more useful question for educational research to develop sound pedagogical practices showing how best to utilise technology that is replacing other media more rapidly than we can keep up with?

Figure 1 .
Figure 1.This website was translated into Icelandic from English.The original is called 'DNA from the beginning' (http://www.dnaftb.org/),from Dolan DNA learning Centre.

Figure 2 .
Figure 2. Two intervention teachers chose to use this website but one felt it was too complicated.

Table 1 .
Participants in the interventions.
*This class was an intervention class **same pupils as INT520

Table 2 .
Classroom activities during intervention.

Table 4 .
Chemistry test results.