Formative E-Assessment in Plenary Lectures
- Side: 36-54
- DOI: https://doi.org/10.18261/ISSN1891-943X-2012-01-04
- Publisert på Idunn: 2012-03-22
- Publisert: 2012-03-22
- Creative Commons (CC BY-NC 4.0)
Little is known about how subjective and objective learning outcomes in plenary lectures are related in the Quality Framework of Higher Education and how they are influenced by formative e-assessment. Given the increasing focus on digitalisation and formative assessment in higher education and the increasing diversity among university students, questions relating to these topics should also be explored within plenary lectures. These lectures constitute the most formal, defined and “bounded” educational practice at universities and it is important to study the question of whether the relationship between student diversity, pedagogy and technology can re-define some of the pedagogical underpinnings that are historically associated with lecturer-centred pedagogy. This paper aims to identify: (1) factors that influence the relationship between intended and subjective learning outcomes in plenary lectures; and (2) how formative e-assessment may improve moments of contingency by increasing the consistency between intended and subjective learning outcomes. The results of this study show that audience response systems (ARS) can enhance formative e-assessment in plenary lectures and reduce the discrepancy between the intended learning outcome and the subjective learning outcome in such lectures with several hundred students. The implications of the current paper are twofold: first, a better understanding of similarities and dissimilarities in students’ learning processes in plenary lectures and how these processes may be affected by formative e-assessment has implications for the planning and implementation of teaching and learning in higher education. Second, this has implications for how we can reduce the discrepancy between the intended, subjective and objective learning outcomes in plenary lectures.Keywords: Feedback, formative e-assessment, ICT, subjective learning outcomes, moment of contingency.
This article focuses on the question of whether, and eventually how, audience response systems (ARS) and feedback clickers (TurningPoint®) can be used to overcome some of the challenges experienced by lecturers in large plenary lectures at universities. New international as well as national policies demand that student curricula in Norway must be more specifically formulated around learning outcomes as well as the use of information and communication technology (ICT) as a tool in teaching and student learning processes. However, a recent report by the Ministry of Education and Research concludes that we have access to a great deal of information about intended learning outcomes in the Quality Framework of Higher Education (Ministry of Knowledge (MOK), 2010), but that we have limited knowledge about teaching quality and teaching methods in higher education. Equally importantly, we know very little about students’ subjective and objective learning outcomes. The current paper aims to address the latter question. This implies a need for a greater awareness of the increasing diversity among university students which necessitates the use of innovative ways of engaging students who might previously have experienced educational alienation. These new educational streams and policy regulations create a situation which calls for a re-thinking of some of the most institutionalised and taken-for-granted practices in university education. This re-thinking and revitalisation of pedagogy and digital didactics (Krumsvik & Almås, 2009) can be useful lenses through which to study such processes. In other words, this paper shows the ways in which a particular mindset for conceptualising the relationship between student diversity, pedagogy and technology can create spaces within even the most formal, defined and “bounded” educational practice: the plenary university lecture. This paper explores the ways in which spaces that are historically associated with lecturer-centred pedagogy can be re-defined through the use of technology (ARS) to provide new opportunities for formative assessment with diverse groups of students. Some studies show that feedback clickers can be used to “increase the ease with which teachers can engage all students in frequent formative assessment” (Roschelle, Penuel, & Abrahamson, 2004, p. 21) in lectures. Within this formative assessment, moments of contingency, “in which the direction of the instruction will depend on student responses” (Leahy, Lyon, Thompson, & Wiliam, 2005, p. 80), seem to provide a focal point which is highly relevant in the context of this study. Against this backdrop, the aim of this design-based research (Design Based Research Collective (DBRC), 2003) and mixed methods study (Johnson, Onwuegbuzie, & Turner, 2007) is to examine the relationship between intended and subjective outcomes and how the two are influenced by formative e-assessment in plenary lectures. The research questions are:
Can ARS enhance formative e-assessment in plenary lectures and how do students perceive this relationship?
Can ARS reduce the discrepancy between the intended learning outcome and the subjective learning outcome in plenary lectures and how do students perceive this relationship?
Feedback is a central aspect of formative assessment. In several recognised meta-reviews, feedback has been shown to be the most important factor in students’ learning (Black & Wiliam, 1998; Shute, 2008; Hattie & Timperley, 2007; Hattie, 2009). Below is a definition by Black and Wiliam (2009) that aims to synthesise earlier definitions of formative assessment:
Practice in a classroom is formative to the extent that evidence about student achievement is elicited, interpreted, and used by teachers, learners, or their peers, to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions they would have taken in the absence of the evidence that was elicited (Black & Wiliam, 2009, p. 9, our bolds).
Students often find that the feedback they receive is insufficient with regard to advice on what to do to perform better, and that it is difficult to know how to interpret and make use of the feedback they receive (Price, Handley, Millar, & O’Donovan, 2010).
In order to assist teachers in implementing formative assessment in their own teaching practice, Black and Wiliam (2009) developed a framework. In the light of their own research and that of others in classrooms, they highlight five types of core activities that have been shown to have a positive effect on learning (Figure 1). The activities in the framework are not separate activities, but must be seen in relation to each other. They operate in cycles, and can be short term or long term depending on the nature of the activity, ranging in length from minutes to a day, a week or a year (ibid.). The activities and the roles played by the teacher, the student and his or her peers are illustrated in the figure below.
Increased knowledge about both formative assessment and increased use of technology in teaching and learning requires these areas to be viewed as inseparable and in relation to each other, both in school and in higher education. The use of technology offers opportunities for: (a) rapid feedback; (b) saving work; (c) automatic feedback; (d) communication; and (e) construction and representation as central characteristics of the ways in which ICT can support formative assessment in the classroom (Pachler, Mellar, Daly, Mor, Wiliam, & Laurillard, 2009). Formative e-assessment can therefore be defined as:
… the use of ICT to support the iterative process of gathering and analysing information about student learning by teachers as well as learners and of evaluating it in relation to prior achievement and attainment of intended, as well as unintended learning outcomes, in a way that allows the teacher or student to adjust the learning trajectory (Pachler, Daly, Mor, & Mellar, 2010, p. 7).
Teachers have access to a range of digital tools with which to organise their feedback and assessment in order to make it more efficient, and to make room for formative assessment processes. In addition, they have a range of different technologies that allow students – together or alone – to show their understanding of knowledge, phenomena and ideas by presenting it in various ways using different digital tools (Pachler et al., 2009). One example of this is the possibility of making models, podcasts, videos and multimodal texts into which sound, image and video are integrated.
Kress (2009) argues that the use of multimodal forms of expression can make it easier to understand students’ current positions in their learning process, and concludes from a socio-semiotic standpoint that: “What is not recognised will not and cannot be assessed” (Kress, 2009, p. 38). However, “recognising” each and every student is hard in plenary lectures at which there are several hundred students – even though lecturers have tried different remedies throughout history, such as the use of prompting questions and raising hands in plenary lectures, there are several disadvantages of these kinds of traditional strategy. For instance, many students feel uncomfortable raising their hands in large plenary lectures because they are afraid of giving the wrong answer and the associated “public embarrassement” (Caldwell, 2007). The use of ICT can support and change some of the conditions for recognising students’ understanding and formative assessment in plenary lectures. This provides new possibilities, but we still know too little about the dimensions of the use of ICT and how it can be applied by teachers and students in relation to formative assessment and the different kinds of learning outcomes. Hence, a distinction can be drawn between intended, subjective and objective learning outcomes. Intended learning outcomes refer to the intentions outlined in the Quality Framework of Higher Education (MOK, 2010) – objective learning outcomes that refer to the achievements of the learner, as measured/quantified using an objective performance test (Adam, 2004), whereas subjective learning outcomes refer to students’ subjective impressions/feelings of how much they have learnt, which can be measured using subjective ratings. Thus, this paper aims to focus on the relationship between intended and subjective learning outcomes and how the two are influenced by formative e-assessment.1 Subjective learning outcomes are especially important in formative e-assessment and therefore this paper focuses on this issue. Our next study will be expanded to include objective learning outcomes and summative assessment. Hence, it is important to study the question of whether, and possibly how, the underlying conditions for assessment work change when technology is used to support both teachers and students, as well as how this is played out both in the traditional auditorium and the “virtual” classroom (VLE/LMS’s).
However, research has shown that there is no “quick fix” within this area (Fish & Lumadue, 2010) because of a lack of competence:
Although professors are considered to be content experts, most are not technology experts. This technology gap poses a problem for professors who want to provide quality feedback to their 21st century students who expect their teachers to be using the latest technology (Fish & Lumadue, 2010, p. 2).
It is often found that the patterns of ICT usage among professors and lecturers when teaching are fairly limited and seldom attached to systematic formative assessment (Lumadue & Fish, 2010). Therefore, our paper examines the way in which technology can provide new possibilities for systematic formative e-assessment which are linked to outcome-based teaching and learning (OBTL; Biggs & Tang, 2007). This seems to be particularly important in Norway due to a number of factors. First, there is an increased dropout rate in higher education over the last few years compared to earlier (Statistisk Sentralbyrå, 2011). Second, the average Norwegian student in higher education receives only 13 hours of teaching per week, which is one of the lowest rates in Europe (MOK, 2011). Third, professors’ “one-person businesses” are still the most common method of teaching in higher education (MOK, 2011). Fourth, evaluations show that the quality of higher education needs to be improved: NOKUT’s evaluation of higher education revealed a lack of quality in several areas (NOKUT, 2011) and the NIFU-Step’s evaluation of the Quality Reform (Nifu-Step 2011) revealed the need for improvement in several areas of higher education, and also in relation to quality in general and formative assessment. Fifth, it seems that formative assessment is particularly important for first-year students, for whom feedback contributes to their emotional support and understanding of the standards required at the university (Poulos & Mahoney, 2007).
At the same time, it is important to underline that no technology is in itself formative, but that almost every type of technology can be used in a formative way: “It is the learners and teachers as human actors who ultimately determine the formative effects of engaging with technologies, but technologies can shape the potential for this to happen” (Pachler et al., 2009, p. 21). The formative aspect lies in taking advantage of technological opportunities to make the reflections of the students more visible (ibid.), and to create an arena for collaboration between students, teachers and peers. Moments of contingency seem to be an especially important part of these processes. The concept of moments of contingency is fundamental in Black and Wiliams’ (2009) theory of formative assessment. Moments of contingency are defined as moments "in which the direction of the instruction will depend on student responses" (Leahy et al., 2005, p. 6). In addition, “formative assessment is concerned with the creation of, and capitalization [sic] upon, ‘moments of contingency’ in instruction for the purpose of the regulation of learning processes” (Black & Wiliam, 2009, p. 10). Leahy and colleagues (2005) explain this concept in the following way:
Teachers using assessment for learning continually look for ways in which they can generate evidence about student learning, and they use this evidence to adapt their instruction to better meet their students’ learning needs (...). Teachers design their instruction to yield evidence about student achievement, by carefully crafting hinge-point questions, for example. These create “moments of contingency,” in which the direction of the instruction will depend on student responses. Teachers provide feedback that engages students, make time in class for students to work on improvement, and activate students as instructional resources for one another (Leahy et al., 2005, p. 6).
Pachler et al. (2010) point out that technology itself does not create moments of contingency: "they are dependent on teachers’ and learners’ actions. But for technology to perform formatively, it needs to acknowledge and support these moments" (Pachler et al., 2010, p. 7). Moments of contingency can be both synchronous and asynchronous. They play a fundamental role, as they are central to formative interaction, and each moment “helps to distinguish a theory of formative assessment from an overall theory of teaching and learning” (Black & Wiliam, 2009, p. 10). Our research focuses on “moments of contingency” in which the use of feedback clickers plays an important role in real-time formative assessment.
Relevant research on feedback clickers (ARS)
A literature review by Kay and LeSage (2009) found 14 published studies about formative assessment (out of a total of 67) and claims that there is evidence to suggest that the use of feedback clickers can support and enhance formative assessment: “Experienced teachers can quickly modify explanations or modes of instruction and students can gauge and discuss their understanding of concepts they are being presented [with]” (Kay & LaSage, 2009, p. 823). Feedback to the instructor about students’ understanding of concepts is also pointed out in a review by Lantz (2010): “Clicker questions can give immediate feedback to the instructor and, provided the instructor acts to correct the misconceptions, can lead to better comprehension of material by students” (Lantz, 2010, p. 6). A recent study published by James and Willoughby (2011), which contained analyses of 361 conversations between students in lectures at which feedback clickers were used, found that many of the conversations between the students were less effective than the lecturers had expected. The study also showed that the students often clicked the correct answer but for the wrong reasons. One urgent question is therefore: do clickers provide good enough evidence of students’ understanding of concepts?
Students in higher education have generally positive attitudes towards the use of feedback clickers (Krumsvik, in press). They participate more actively, are motivated to attend classes, read more and are more focused in class when clickers are used (Caldwell, 2007; Kay & LaSage, 2009; Lantz, 2010). Several literature reviews have claimed that students interact more with their peers and promote peer discussions when clickers are used (Cadwell, 2007; Kay & LaSage, 2009). Both Kay and LeSage (2009) and Lantz (2010) point out that as well as an increase in the quantity of discussions, there is an increase in the quality of these discussions, and students hold more effective discussions. However, when Morse, Ruggieri and Whelan-Berry (2010) investigated whether the use of clickers in the classroom had an impact on the quality of the discussion, they reported that students in the clicker class did not participate more than students in the non-clicker class. Both were given the same questions. They found that the use of clicker questions improved class discussion, and that the non-clicker group held more positive views about whether the use of clicker questions in class increased participation in discussions. They point out that: “these results beg the question; Is it the process of asking the clicker question, or the clicker technology that promotes student engagement?” (Morse et al., 2010, p. 105). This finding is also similar to results presented by Anthis (2011).
Studies show mixed results with regard to improved learning outcomes. Some have found that students who use clickers have better learning outcomes in terms of exam scores (Mayer, Stull, DeLeeuw, Almeroth, Bimber, Chun, Bulger, Campbell, Knight & Zhang (2009); Morling, McAuliffe, Cohen, & DiLorenzo, 2008), whereas other studies have reported no significant difference in exam scores between clicker groups and groups that did not use clickers (Anthis, 2011; Filer, 2010; Carnaghan & Webb, 2007). A newly-published study conducted by Anthis (2011) separated clicker use and question use, and found no relationship between the use of the clickers and exam grades: “Previous research asserting positive associations between clicker use and exam scores did not address whether those findings were truly a result of clicker use or question use” (Anthis, 2011, p. 191). In a similar study, Mayer et al. (2009) found the opposite result: “The clicker group outperformed both the control group and the non-clicker group, suggesting that the implementation of the questioning method was less intrusive with clicker technology” (Mayer et al. 2009, p. 56). Filer (2010) did not find that the group that used clickers improved in post-lecture quizzes “compared to the control group, but the students in clicker-enhanced lectures reported significantly higher satisfaction scores. The use of ARS promoted a sense of comfort, encouraged participation, and motivated students to answer questions and interact with the subject matter” (Filer, 2010, p. 247). The use of clickers to create a learning environment with greater activity and cooperation compared to traditional, lecture-based instruction is also reported by Hoekstra (2008). Similarly, Morales (2011) found that “clickers are an effective tool for changing classroom dynamics” and “overall the clicker helped provide an environment where students were more relaxed and less apprehensive” (Morales, 2011, p. 39). Despite the fact that most of the students found that the use of the clickers facilitated a “more positive and active atmosphere” (Morales, 2011, p. 41), the teachers noted that some of the students did not want to participate. Dallaire (2011) found that the “effectiveness of clickers is affected by the way in which an instructor uses them and students’ characteristics” (Dallaire, 2011, p. 203). For example, if the instructor used clickers in “too many or too few ways students’ learning may suffer” (Dallaire, 2011, p. 203).
In conclusion, it is interesting to note that most students like clickers, that they value immediate feedback and that they believe that the use of clickers adds value to their learning. The literature also shows that students interact more with their peers and with the lecturer when clickers are used compared to traditional lectures. It is also claimed that more in-depth processing occurs when clickers are used. However, the published literature on clickers shows mixed results in relation to learning outcomes in summative tests. In this article, we will focus primarily on the relationship between intended and subjective learning outcomes and how this may be influenced by formative e-assessment. In order to answer the research questions, the design and methodology employed are described below.
The design of the study is linked to Brown (1992) and Collins’ (1992) design experiments, which explore, for instance, how a technological innovation affects student learning and educational practice. Design-based research is marked by a participatory design and the fact that “development and research takes place through continuous cycles of design, enactment, analysis, and redesign” (DBRC, 2003, p. 5). This means that this project will include interventions as part of the design. In this way, design-based research positions itself as a participatory research design which often uses mixed methods as well as research partnerships with the practitioners for which the local context forms an important focal point.
A literature review conducted by Kay and LeSage (2009) concluded that reports about clickers are biased towards qualitative research, and they point out that “both qualitative and quantitative research is needed to fully understand the use and the impact of ARS, so triangulation of methods might be a desirable direction for future research efforts” (Kay & LeSage, 2009, p. 825). This is one of the reasons for the application of mixed methods research in our study. In mixed methods research, a researcher or team of researchers combines elements of qualitative and quantitative research approaches "for the broad purposes of breadth and depth of understanding and corroboration" (Johnson et al., 2007, p. 123). Johnson and colleagues (2007), in a meta-analysis of the mixed methods design literature, generated a general definition of the approach:
Mixed methods research is the type of research in which a researcher or team of researchers combine elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purposes of breadth and depth of understanding and corroboration (Johnson et al., 2007, p. 123).
On this basis, this project is grounded on a purely mixed methods design, into which quantitative and qualitative elements are equally integrated. The research question has both qualitative and quantitative components. These components are set out as separate questions, and the purpose is to link them in the analysis, so that they can shed light on one another. A survey, semi-structured interviews and observations of lectures in which clickers were used are the sources of data in this project.
The sample for this survey, chosen using purposeful selection (Maxwell, 2005), consisted of 243 first-year psychology students. This sample is based on the students who attended the plenary lectures for this course and therefore had experienced the use of feedback clickers during the lectures. Thus, the study could have been influenced by a certain “Hawthorne effect” because we could not include the students who did not attend the plenary lectures for different reasons (approx. 150 students). The response rate in the chosen sample was 93%. The survey was analysed using frequency analysis and t-test (Levene's test for equality of variance and the t-test for equality of means), correlation and logistic regression analyses. The data were analysed using scales for motivation, perceived learning outcome and feedback. In addition to the three scales, the material was also analysed using individual questions linked to communication and social interaction. Four observations were carried out by two researchers during the psychology course. One observation used video to film large parts of the lecture. In addition, purposeful selection (Maxwell, 2005) was carried out for the four students who were interviewed. In our final analysis of the data material (both quantitative and qualitative data) in this subproject, we followed Johnson and colleagues’ (2007) guidelines for this kind of mixed methods analysis. The interviews and the observations were analysed using NVivo9 and the survey was analysed in PASW Statistic 18.
The datasets (survey, interview and observation) were collected and analysed separately. The analysis of the survey, the observations and the interviews were linked together, compared and contrasted in NVivo9. The various types of category were linked and used to highlight, supplement, explain, validate and understand the responses from the survey. Below we will present the main findings which relate to the key aspects of formative assessment (presented on p. 3) and the role of the learner, his or her peers and the teacher.
In the first part of this review of the results, we will seek to explain the students’ answers and to look at the relationship between the use of feedback clickers and formative assessment. This review is seen in the light of Black and Wiliam's (2009) theoretical framework for formative assessment and the activities and roles played by the teacher, the student and his or her peers. In the second part, we will examine some interesting differences within the group of students. This section is primarily a review of t-tests and regression analyses that were carried out in order to look at similarities and differences within the group of students.
In the survey, the majority of the students reported that the use of feedback clickers had contributed to their learning. The students emphasised that feedback, motivation and learning are linked together. Feedback from the clickers drives their motivation to pay attention in lectures, to attend, read and adjust their own understanding.
One interesting finding is that the students were most positive in responding to the questions on the feedback scale, compared to the two other scales of motivation and perceived learning outcome. The majority of the students (92%) reported that the feedback clickers provided feedback on their own understanding of the subjects in the curriculum. In addition, the majority of the students reported that they used the feedback they received from using the clickers and the lecturer’s explanations to adjust their own understanding of concepts. The students found that the use of feedback clickers during lectures supported their learning of the subject. In the survey, the majority of the students (92%) reported that the use of feedback clickers contributed to their learning compared to when they were not in use. In the interviews, we attempted to identify what the students thought about feedback during lectures and how they adopted the feedback in their further work on the subject.
Given that students feel positive and motivated to continue working as a result of giving the right answers, do they feel bad and demotivated after giving the wrong answers? All of the students had experienced giving wrong answers. They all had in common that they had been made aware of what they had not understood and what they would study harder or try to understand. In order to make use of feedback at a later stage, a couple of the students mentioned that they “noted” the mistakes they had made, in order to go over them again. There is a difference between becoming aware of the terms or concepts that one does not understand and actually doing something about it in order to achieve a greater level of understanding. All of the interviewed students were keen to do better in the cases in which they had given the wrong answer to a question. Among the strategies they mentioned were “study more,” “study more thoroughly,” “look at it in more detail” and “concentrate harder.” The informants emphasised that they would rather give the wrong answer and receive feedback during the learning process than in an exam. Understanding why, how and to what extent students actually make use of the feedback they are given is crucial to further research into the use of feedback clickers. The students emphasised that feedback, motivation and learning are inextricably linked. Feedback drove their motivation to pay attention in lectures, to attend, read and adjust their own understanding.
Because… when I get immediate feedback I learn more (…) my attention is kept if I get a direct response – it makes me stay focused (Sara).
The survey showed that the use of feedback clickers also motivated the students in the sample to attend the lecture. They also highlighted that the desire to be able to choose the correct answers motivated them to study more in advance: “I think the most valuable aspect is the motivation it creates to study in advance, because it makes you grasp the syllabus better and subsequently learn better” (Lisa). Another student, who was not in the habit of studying in advance, used the questions to guide and focus her studies afterwards: “I think the clickers made me identify the key points more easily. In a way it made it easier to focus on the correct answers and create a kind of framework around them” (Sara). She was also looking for affirmation that what she had understood/learnt by attending the lectures matched the content of the book. One interesting finding is that the use of feedback clickers not only motivated the students to study more; it also affected how they studied. One student who had made a few mistakes said that she had to go back and adjust the way in which she studied in order to better remember the material. The answers to the questions describe how the students learnt to “crack the code” of answering questions during the semester. They may also suggest that the students began to study more or in a different way as the semester progressed. For example, we noted that there were fewer correct answers at the beginning of the semester than after the series of lectures had been completed. This could be explained by the fact that the students had learnt more, but also because they had adopted a different study strategy that allowed them to better remember the concepts. On the other hand, this could also be explained by other variables and thus we need further research in order to obtain a more complete picture regarding this issue.
The majority answered that the use of clickers motivated them to study more, to pay attention during lectures and to attend lectures. This is also reflected in the interviews, in which the informants also pointed out that they were more motivated to concentrate on the content of the lectures.
The fact that feedback clickers can help to improve concentration has also been pointed out by Caldwell (2007), Kay and LeSage (2009) and Lantz (2011). The use of clickers can make it easier to identify holes in the students’ knowledge, and they also become more conscious of specific concepts with which they struggle or are unfamiliar: “I could find concepts very similar (…) reliability and validity, for example. It is easy to get them mixed up unless you really sit down and study them and understand them. And those clickers made me see more easily which (…) then I worked out which terms I was struggling more or less with and that there are some terms that I don’t understand” (Sara). One of the students, on the other hand, highlighted the notion that the focus on factual questions and definitions could come at the expense of the students’ grasp of the breadth of the topic. He pointed to a specific example in which the aim was to understand a model: “We moved back and forth with Maxwell’s model, but I’m not quite sure if it allowed me to fully understand it. He [the teacher] approached it, it became quite staggered, I found it difficult to make the connection, no, it’s not my way of thinking” (Jon). Jon responded positively to lectures in which the lecturer “just stands up there and talks.” He did not like interruptions and found the clickers “fun”, but fairly disruptive in themselves: “We sat there fidgeting with them.” This is interesting and helps to explain some of the findings in the survey, e.g. the fact that those who liked extensive use of presentation tools saw more of a use for the clickers than those who preferred limited or average use of presentation tools.
After controlling for age, sex, grade, motivation and future plans, preference for presentation tools was the most important factor for how students perceived the use of clickers in relation to their learning outcomes. A preference for extensive use of tools predicted the perception of a good learning outcome when using feedback clickers. On the learning outcome scale, we found a correlation (p<0.001) between the preference for the use of presentation tools and perceptions of learning.
An independent samples t-test (Levene’s test for equality of means) was carried out in order to establish whether there were any differences between the students according to their preference for presentation tools with regard to their experience with the use of clickers. Students who liked lectures that used presentation tools extensively experienced a better learning outcome with the use of feedback clickers (M=3.47, S=0.42) than those who liked some use of presentation tools (M=3.29, S=0.43). The difference was significant (p=0.002). This finding was also confirmed in a logistic regression analysis. If a student reported a preference for the extensive use of presentation tools, he/she was 0.3 times more likely to report a better learning outcome than someone who preferred little or average use of presentation tools. The model below illustrates the systematic difference between two student groups for four of the questions on the learning outcome scale.
It is worth noting that in a correlation analysis, we found that older students liked extensive use of presentation tools to a greater extent than younger students (p=0.008). On the other hand, the survey showed that there was a systematic tendency for younger students in general to find the use of clickers to be more useful for their learning outcome than older students, but the difference was not significant.
In addition, we found a correlation (p<0.021) between the group that was highly motivated to do well in the subject and the perception that the clickers increased motivation on the motivation scale. Therfore, an independent samples t-test (Levene’s test for equality of means) was carried out in order to establish whether there were any differences between students with high and average motivation with regard to their experience with the use of clickers. Students who were highly motivated to do well in the subject were more motivated (M=3.04, SD=0.62) by the use of feedback clickers than those with average motivation to excel in the subject (M=2.84, SD=0.67). There was a significant difference on the motivation scale (p=0.040). Girls were generally more motivated to do well in the subject (M=3.14, SD=0.74) than boys (M=2.14, SD=0.94), but the difference was not significant. This may explain the tendency for girls to be more positively against the use of clickers than boys. It should also be noted that there was a systematic tendency for students with grades of 5 or higher to be more positive towards the clickers than students with grades between 4 and 5, but this finding was not significant.
Our results show that students’ characteristics are related to whether they like or dislike feedback clickers and how they value these tools in terms of perceived subjective learning outcomes.
Both in the survey and in the interviews, the students emphasised that there was more interaction in lectures in which feedback clickers were used than in lectures in which they were not used. In total, 91% of the students agreed that feedback clickers (at least to some extent) led to more discussions among students (two and two) than if the clickers were not used, and 98% of the students found that the use of feedback clickers led to greater interactivity during lectures compared to other lectures. The majority of the students also thought that the use of feedback clickers had an impact on overall interactivity compared to other lectures.
The purpose of the interviews was to find out more about the quality of the increased interaction. The students said that the interaction took on a different character when feedback clickers were used than when they were not used, both because it was easier to stay focused and because students attempted to benefit to a greater extent from the knowledge of their peers to find answers. This is particularly interesting, but it can be difficult to establish the exact nature of the difference, and how brainstorming actually plays out compared with when feedback clickers are not used. The desire to give correct answers motivated the students to make good use of the time set aside for brainstorming: “I think it has to do with wanting to perform, and because you actually have to give an answer. I think it's more important for people to get it right” (Lisa). As time is short and because she wanted to make the best use of it, one of the students said that it was easier to “draw quicker conclusions (…) to just rattle off some instant thoughts.” One student said that brainstorming between students therefore became more effective and focused: “You brainstorm in order to reach an answer (…) it’s targeted brainstorming in a way. You’re pressed for time, too, right? (…) You get more focused” (Lisa). Although the intention behind brainstorming groups is to encourage reflection on the subject matter, with or without feedback clickers, it would appear that the use of feedback clickers is more binding and increases motivation to generate concrete answers than if the students are asked to brainstorm without having to provide answers:
If you were to just talk with your peers, without a goal, if you see what I mean, and if you do not have to come up with a specific answer, a reason for why the answer is correct, it will just be, in a way… yes… talk (Lisa).
Our material indicated that there is a difference between discussions held with and without feedback clickers. The students had to provide answers and they experienced a greater interest in actually discussing concrete problems (on topic) rather than “off-topic” activities. One of the students said that she often felt that she was behind with her studies and that she was often uncertain about the answers. She then tried to provide a reason herself and to ask others what they were thinking in order to “work towards a solution.” She tried to listen to what the other person was saying, concluding:
You discuss back and forth and try to establish whether it is a concept. “Is it like this, or should I have something else,” right? Then you sometimes find that you end up with different answers (Sara).
She also found that if someone had identified the correct answer, he or she would not want to reveal it straight away, but “rather get the others to work it out for themselves.” Helping others to think and work out the answer for themselves can give the students a sense of ownership of their own learning.
In other words, the students highlighted three aspects that change the nature of brainstorming. First and most important is the fact that the students had to give an answer, and that they therefore remained focused when discussing the question with their peers. Second, time was a factor. The fact that the students had limited time in which to come up with the answer meant that they were better prepared to join in the discussion even if they were unsure. Third, it appears that they listened to what the others said to a significant extent, and used this actively to adjust their own understanding and to draw their own conclusions.
Our survey also found that many of the informants thought that it was a good thing to be able to see the answers given by others. This has also been identified in other studies. We were interested in establishing why the students found this to be a positive thing. Could it add something to the teaching or to the learning environment? It is essential to do well in exams in order to continue one’s studies, and a couple of the students pointed out that they viewed others as competitors and that this gave them an idea of their own position in the class. Second, they described the positive aspect of seeing that they are not alone in failing to understand the concept embodied in a question, as described by one student: “You do have a tendency to sit and think ‘It’s just me, it’s really embarrassing.’ Sure, people have a tendency to do just that. This applies to almost everything” (Jon). Seeing other people making mistakes creates a safe environment and atmosphere, where, unlike in exams, it is “OK to make mistakes.” Many informants pointed this out as a positive aspect and one that made the students more “equal.”
Everyone can see that you’re not stupid just because you made a mistake. Many people make mistakes. Perhaps no more than 70% get it right. I think that’s good because, OK, it shows that 30% of the auditorium is not stupid. These are clever people who have been admitted, but then you realise it’s still OK to make mistakes. I think this makes it much easier to communicate together about the answer (Sara).
Following a direct question from scientists about whether this safe environment had any effect outside lectures and whether it could encourage communication in fora such as seminar groups and discussion groups, the students responded that this was not something they had considered. How and the extent to which the use of feedback clickers has an effect on the atmosphere among the students outside lectures may be a relevant theme for further research.
Students with high motivation to do well in the subject found that the clickers contributed to a higher level of interactivity in lectures (3.69, SD=0.47) than students with average motivation (3.46, SD= 0.78). The difference was significant (p=0.043).
The lecturer (who led the lectures in this study) created questions based on the intended learning outcome of the psychology course in question. This happened in both a synchronous (in class with ARS) and asynchronous manner (social media and VLE/LMS systems). In addition to the students’ reflections on concepts using the qualitative method, the students' answers gave the lecturers an opportunity to adjust the course and teaching and to clarify concepts that the students did not understand (moments of contingency). Important to the concept of moments of contingency is the question of how and to what extent the lecturer adjusts his or her teaching based on this feedback, both in the lecture hall following a question and in plans for further teaching. The survey showed that most of the students (92%) felt that the lecturer adjusted his lectures according to the result of the poll. We asked the students about the extent to which they found this useful for their perceived learning. The students were motivated by the lecturer’s feedback and adjustments. One student found that he became more motivated if the lecturer adjusted to the answers given by the students. He compared this with a computer game on which the player masters a new level. It is not too easy, but not too difficult either.
The lecturer is happy to make adjustments. What did they understand? I make adjustments, the students become motivated. It goes back and forth, one feeds off the other (David).
Others pointed to the fact that the lecturer was more likely to stop and explain when many incorrect answers were given than when many students gave correct answers. Several of the students stressed the need not only to know whether they were right, but also why they were right:
I think it is very important that feedback is given on why [an answer] was correct, regardless of what the figures show. So, even if everyone got it right, it would be useful in terms of getting an explanation that I remember, at least a retrospective explanation, even if I answered correctly (Tomas).
Several of the students pointed to this idea. Even if only a few of them gave wrong answers, they needed an explanation. Generally speaking, the students found that, in such moments of contingency, the lecturer carried on when many students gave correct answers and stopped more frequently when more people gave incorrect answers. All of the four students interviewed emphasised that he could have taken more time after each question: “I have always found that the easiest way to remember things is when you are given the explanations of why it is like it is, not just that this is like it is. So explanations are very valuable” (Tomas). The students wanted an explanation as to why an answer was correct, and not only that it was either right or wrong. The importance of explaining why an answer is correct has been highlighted by many researchers, including Mayer et al. (2009), and underlines the need for such a moment of contingency to be handled properly and in depth by the lecturer.
Discussion and conclusions
Our research question in this study was:
Can ARS enhance formative e-assessment in plenary lectures and how do students perceive this relationship?
Can ARS reduce the discrepancy between the intended learning outcome and the subjective learning outcome in plenary lectures and how do students perceive this relationship?
As shown in the model presented below, the clickers played a significant role in formative e-assessment, both for the students, their peers and the lecturer.
For individual students, immediate feedback on their own understanding of terms and concepts is highlighted as the most important aspect of the use of feedback clickers. The students reported that they had been made aware of what they had not understood and that they would “study harder,” “read more carefully” or “try to understand better” if they gave incorrect answers. However, there is a difference between becoming aware of the terms or concepts that one does not understand and taking action in order to achieve a greater level of understanding. Understanding why, how and to what extent the students actually make use of the feedback they are given is crucial to further research on the use of feedback clickers. Our study is limited to factual knowledge. This is a weakness, as pointed out by one of the students in the interview. Questions on how the use of feedback clickers can address both factual knowledge and the overall picture (knowledge transfer) are also interesting in relation to further investigations.
Our study also shows that there are differences in student characteristics in terms of the extent to which they feel that the clickers enhance their learning. Preferences regarding presentation tools, overall motivation and age seem to be important factors with regard to students’ perceptions of the clickers in relation to their learning, but we do not know whether and to what extent there is a relation between perceived (subjective) and actual (objective) learning. This will also be an interesting question for further research.
Some of the students found that the use of clickers did not add any value to their learning. This is in line with previous research and this issue should also be examined in future research.
Both in the survey and in the interviews, the students emphasised that there was more interaction in lectures in which feedback clickers were used than in lectures in which they were not used. The students pointed out that as well as an increase in the quantity of discussions, there was an increase in their quality: the students held more effective discussions. This is in line with the findings of Kay and LeSage (2009). The fact that the students found that they held more effective discussions is also described in a similar study by Lantz (2010): “If the instructor allows the students to discuss the questions before responding, the students will be able to consider points made by other students that they themselves may not have considered leading to even deeper processing” (p. 563). The way in which students get involved before, during and after questions have been asked has also been discussed by Mayer (2009):
The act of trying to answer sample questions and then receiving immediate feedback may encourage active cognitive processing in three ways: (a) before answering questions, students may be more attentive to the lecture material, (b) during question answering, students may work harder to organize [sic] and integrate the material, and (c) after receiving feedback, students may develop metacognitive skills for gauging how well they understood the lecture material and for how to answer exam-like questions (Mayer et al. 2009, p. 56).
Seeing other people making mistakes creates a safe environment and atmosphere in which, unlike in exams, it is “OK to make mistakes.” This is also documented in other studies. In what way and the extent to which the use of feedback clickers has an effect on the atmosphere among the students outside lectures may also be a relevant theme for further research.
An analysis of the conversations between the students could be useful for future studies in order to establish what differences exist and what the use of feedback clickers may contribute to the students’ brainstorming. The students found that both the clicker and the question asked by the lecturer played a significant role in their engagement. The fact that students have to choose an answer was more binding and ensured that peer-to-peer conversations were more focused and effective.
The use of feedback clickers allowed the lecturer to engineer effective discussions that elicited evidence of learning, which then allowed him to modify explanations, adjust his focus, methods, progression and teaching strategies. In this way, the clickers allowed both the lecturer and the students to create moments of contingency. These moments were created both synchronously and asynchronously, by the lecturer as well as the students, and played a fundamental role in the formative interaction. In conclusion, we found that feedback clickers can enhance the possibilities of formative e-assessment. Many of the students found that this use of digital artefacts added value to their learning process, and therefore we find that ARS reduce the discrepancy between the intended learning outcome and the subjective learning outcome in plenary lectures. However, at the same time, this study reveals the need for more research within several of the key areas addressed in this article, in order to obtain more consistent results.
Anthis, K. (2011). Is it the clicker, or is it the question? Untangling the effects of student response system use. Teaching of Psychology, 38(3), 189-193.
Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education, 5(1), 7-75.
Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability, 1(21), 5-31.
Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of the Learning Sciences, 2(2), 141-178.
Caldwell, J. (2007). Clickers in the large classroom: Current research and best-practice tips. Life Sciences Education, 6(1), 9-20.
Carnaghan, A., & Webb, C. (2007). Investigating the effects of group response systems on student satisfaction, learning, and engagement in accounting education. Issues in Accounting Education, 22(3), 391–409.
Collins, A. (1992). Toward a design science of education. In E. Scanlon & T. O'Shea (Eds.), New directions in educational technology (pp. 15-22). New York: Springer-Verlag.
Dallaire, D. H. (2011). Effective use of personal response “clicker” systems in psychology courses. Teaching of Psychology, 38(3), 199-204.
Design-Based Research Collective (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8.
Filer, D. (2010). Everyone's answering: Using technology to increase classroom participation. Nursing Education Perspectives, 31(4), 247-250.
Fish, W. W., & Lumadue, R. (2010). A technologically-based approach to providing quality feedback to students: A paradigm shift for the 21st century. Academic Leadership, 1(8), 5-11.
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 1(77), 81-112.
Hattie, J. A. (2009). Visible learning. A synthesis of over 800 meta-analyses related to achievement. New York: Routledge.
Hoekstra, A. (2008). Vibrant student voices: Exploring effects of the use of clickers in large college courses. Learning, Media and Technology, 33(4), 329–341.
James, M. C., & Willoughby, S. (2011). Listening to student conversations during clicker questions: What you have not heard might surprise you! American Journal of Physics, 79(1), 123.
Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods, 2(1), 112-133.
Kay, R. H., & LeSage, A. (2009). Examining the benefits and challenges of using audience response systems: A review of the literature. Computers & Education, 53(3), 819-827.
Kress, G. (2009). Assessment in the perspective of a social semiotic theory of multimodal teaching and learning. In C. Wyatt-Smith & J. Cumming (Eds.), Educational assessment in the 21st century (pp. 19-41). The Netherlands: Springer.
Krumsvik, R. (in press). Feedback clickers in plenary lectures – a new tool for formative assessment? In L. Rowan & P. Bigum (Eds.), Future proofing education: Transformative approaches to new technologies and student diversity in futures oriented classrooms. London: Springer.
Krumsvik, R., & Almås, A. G. (2009). The digital didactic. In R. Krumsvik (Ed.), Learning in the network society and digitized school. New York: Nova Science Publishers.
Lantz, M. E. (2010). The use of ‘clickers’ in the classroom: Teaching innovation or merely an amusing novelty? Computers in Human Behavior, 26(4), 556-561.
Leahy, S., Lyon, C., Thompson, M., & Wiliam, D. (2005). Classroom assessment, minute by minute, day by day. Educational Leadership, 63(3), 19-24.
Maxwell, J. (2005). Qualitative research design: An interactive approach. New York: Sage.
Ministry of Knowledge (2010). Kvalifikasjonsrammeverket for høgere utdanning. Oslo: Statens Forvaltningsteneste. Retrieved 15.08.2010 from http://www.regjeringen.no/nb/dep/kd/tema/hoyere_utdanning/nasjonalt-kvalifikasjonsrammeverk.html?id=564809
Ministry of Knowledge (2011). Tilstandsrapport for høyere utdanningsinstitusjoner 2011. Oslo: Statens Forvaltningsteneste.
Mayer, R.E., Stull, A., DeLeeuw, K., Almeroth, K., Bimber, B, Chun, D., Bulger, M., Campbell, J., Knight, A., & Zhang, H. (2009). Clickers in college classrooms: Fostering learning with questioning methods in large lecture classes. Contemporary Educational Psychology, 34(1), 51-57.
Morales, L. (2011). Can the use of clickers or continuous assessment motivate critical thinking? A case study based on corporate finance students. Higher Learning Research Communications, 1(1), 33-42.
Morling, B., McAuliffe, M., Cohen, L., & DiLorenzo, T. M. (2008). Efficacy of personal response systems (“clickers”) in large, introductory psychology classes. Teaching of Psychology, 35(1), 45-50.
Morse, J., Ruggieri, M., & Whelan-Berry, K. (2010). Clicking our way to class discussion. American Journal of Business Education, 3(3), 99-108.
NIFU-Step (2011). Frafall og gjennomføring i lavere grads studier før og etter Kvalitetsreformen. En sammenlikning mellom begynnerkullene fra 1999, 2003 og 2005. Oslo: NIFU-Step.
NOKUT (2011). NOKUTs programevalueringer - prosesser, resultater og kvalitetseffekter. Oslo: NOKUT.
Pachler, N., Mellar, H., Daly, C., Mor, Y., Wiliam, D., & Laurillard, D. (2009). Scoping a vision for formative e-assessment: A project report for JISC. London: London Knowledge Lab.
Pachler, N., Daly, C., Mor, Y., & Mellar, H. (2010). Formative e-assessment: Practitioner cases. Computers & Education, 54(3), 715-721.
Price, M., Handley, K., Millar, J., & O’Donovan, B. (2010). Feedback: All that effort, but what is the effect? Assessment & Evaluation in Higher Education, 35(3), 277–89.
Roschelle, J., Penuel, W. R., & Abrahamson, L. (2004). Classroom response and communication systems: Research review and theory. Presented at the Annual Meeting of the American Educational Research Association, San Diego, CA.
Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153-189.
Statitsisk Sentralbyrå (2011). Gjennomstrømning i høyere utdanning, 2009/2010. Oslo: SSB.
|1||Subjective learning outcomes are especially important in formative e-assessment and therefore this paper focuses on this issue. Our next study will be expanded to include objective learning outcomes and summative assessment.|