Student reflection and learning through peer reviews
Daniel Boase-Jelinek, Jenni Parker and Jan Herrington
This paper describes lessons learnt whilst using an online peer review system in an undergraduate unit for pre-service teachers. In this unit, students learn to use information technologies as part of their future teaching practice. The unit aims to foster graduates who become lifelong reflective educators by providing opportunities to explore and reflect on how they might use technology in authentic learning situations. Whilst peer review is an appropriate activity for supporting critical thinking and reflective practice in this kind of unit, it requires a number of decisions to be made in relation to student preparation and support, implementation strategy, and technological infrastructure to make it work in specific contexts. Much research has been conducted in recent years to inform educators in making these decisions. However, there are still gaps in the research, particularly in how to improve the quality and consistency of feedback that students give to each other in their feedback. This paper describes the experiences of implementing an online peer review system aiming to improve quality and consistency of feedback. This exploration has revealed that we can learn much about ways to improve our teaching practices by giving students an opportunity to review each other's work and give feedback.
Peer assessment and peer review are processes whereby students grade each other's work for either summative or formative purposes (Bostock, 2006). The term 'peer assessment' is often used to describe the process of giving summative assessment, whereas 'peer review' is generally used for giving and receiving non-summative formative feedback (Wood & Kurzel, 2008). As noted by van den Berg, Admiraal and Pilot (2006), such reviews have benefits not only for the student receiving the feedback, but also for the student giving it. Students generally experience peer review as a non-threatening process that benefits their learning by providing suggestions from their peers about how to improve their work and by helping them understand the criteria that will be used for the summative assessment of their work (Wood & Kurzel, 2008). In this respect, there are well documented benefits from encouraging students to review each other's work (Wessa & De Rycker, 2010). The students whose work is reviewed may benefit from getting external perspectives on ways in which their work may be improved, thus stimulating their critical thinking (Sims, 1989). The students doing the review also benefit as a result of having to process and analyse the work of a peer (Wessa & De Rycker, 2010), and may get ideas for improving their own work (Sims, 1989). The peer review process may extend over a period of time, and may involve students in developing the marking criteria as well as applying those criteria to their own, and others' work. Students are thus encouraged to develop higher level awareness of the task through this extended engagement with the teacher in the assessment process (Wood & Kurzel, 2008).
Addressing the quality and fairness of the marking process involves consideration of the process for implementing the peer reviews. Using a formative (peer review) rather than summative (peer assessment) approach removes most of the unfairness that might arise from peers marking each other's work (Wood & Kurzel, 2008). Making the reviews anonymous can help avoid students feeling conflicted about commenting on the work of friends (Bostock, 2006). Preparing students for their role is also important for maintaining quality of the feedback (Pearce, Mulder & Baik, 2009), especially training them to focus on the aspects of their peer's work that are important.
The second challenge relates to strategies underlying the design of the peer reviews. Ideally, as noted by Lu and Bol (2007), peer assessments for summative purposes need to be completed anonymously so that students are not prejudiced in their marking by personal relationships with their fellow students. As with peer reviews, students are more prepared to make critical and constructive comments when they are anonymous (Howard, Barrett & Frick, 2010). However, there are advantages in peer reviewers being identified because students need to learn to give each other honest and constructive feedback when working as a team (Gulikers, Sluijsmans, Baartman & Bartolo, 2009).
The third challenge when designing peer reviews, is deciding on the number of peer reviews each student performs. The quality of feedback given by students to each other tends to be quite variable (Robinson, 1999), so ideally each student should receive feedback from multiple reviewers. Deciding the number of reviews involves balancing student and teacher workload issues against the benefits of multiple reviews. Furthermore, decisions need to be made on the degree of monitoring reviews to ensure they are not over-critical and provide useful feedback to students (Pearce et al., 2009).
The fourth challenge relates to the practical management of a peer review process (Mostert & Snowball, 2012). Students need to be given access to each other's work and a forum for exchanging assessments. While peer review systems are becoming increasingly sophisticated in providing the means to upload and download documents, at the very least they need to be customised to automate the complex management of files associated with multiple peer reviews that was identified by Knight and Steinback (2011). All assessments need to be monitored and possibly assessed by teachers, which can be a logistical challenge where large numbers of students are involved.
Since student work was online, we developed a web-based system for facilitating the peer reviews. Whilst there are a number of online tools available for facilitating peer reviews (Keppell, Au, Ma & Chan, 2006; Mostert & Snowball, 2012; Pearce et al., 2009), such tools appeared to lack specific features (especially simplicity and flexibility of use) that were necessary in dealing with large numbers of students, so we developed a web-based tool to meet our needs. Students logged into the system using their student ID numbers and were given the name and web address of the item they were to review, along with a web-based rubric specifying the criteria that they were to use. Students were advised that these same criteria would be used for summative marking of their own work later in the semester. When students submitted their review it automatically sent an email to the reviewed student advising them that a review had been performed, and gave them the link to the review. Students could conduct more than one review of that student's work if they wished. For example, if a student modified their work in response to a review they could ask for it to be re-reviewed. At the assignment submission due date, the teaching team used the same rubric to assess each student's work. The teaching team also reviewed the quality of the review each student had performed. Once marking was completed students could view both the peer review and the tutor review.
The technology used in this web-based system was kept basic to avoid technological problems such as incompatible browsers. The review page used simple HTML forms, with no cookies, Flash or Java applets being required to complete the reviews. To make it easy to set up a website to display the marking rubric and manage student peer reviews a Microsoft Excel workbook was created that used Visual Basic macros to generate the web pages required by the peer review system. Through the use of this Excel-based Peer Review System Generator the process for organising and conducting peer reviews involved 10 steps, some of which took only a few minutes:
Figure 1: Rubric used for conducting peer review
The peer review process was purely formative and the marks students awarded each other did not count towards their final grade. Students were given a few days to improve their work in response to the feedback they were given. The quality of reviews given was assessed by tutors (see Figure 2 for the rubric used for marking students' reviews). The focus in assessing students' reviews was that they should give constructive suggestions to their peers on how to improve their work.
Figure 2: Rubric used for marking student peer reviews
Since the purpose of implementing the peer review system was to address misunderstandings by students about the marking criteria and to clarify the purpose of the assignment, we devoted time in one lecture to demonstrating the marking guide and describing how it would be used in assessing student work. This involved discussing what we were looking for in each component of the marking guide and describing what different levels of performance might look like on each criterion.
Virtually all of the 300 enrolled students completed peer reviews as requested. Some experienced problems in conducting a peer review when the student whose work they were reviewing did not submit work in time for a review. The teaching team wanted students to be able to modify their work following the peer review, so the assignment submission deadline was one week after the peer reviews were conducted. Some students did not take advantage of this opportunity, and only submitted their work at the assignment deadline without having it reviewed. However, most students were able to conduct the review without difficulty.
Topping (2009) notes that peer feedback can be 'confirmatory, suggestive, or corrective' (p. 22), and these types were all evident in the feedback students provided to their peers. However, many students were anxious about giving feedback, and they expressed concern about offending a peer with critical (corrective) comments, but noted that giving honest feedback (suggestive) could be of benefit to their peer. A number of students commented that having a marking rubric helped them concentrate on the important aspects to focus on when giving feedback. One student developed a strategy of sandwiching negative comments between positive ones (confirmatory). Another student consulted a teacher on how to give feedback. It seems that students did benefit from the experience, as one pointed out: "it taught me a lot about reviewing, judging, and marking".
Students found that receiving feedback from their peers was a mixture of positive and negative experiences. One student summarised this with the comment
... I began to look through my work and compare it to the peer review, and I was able to see what the student was saying. I repeatedly told myself ... this isn't a personal attack, it is designed to help you get better marks, so stop being upset and improve your work.In general, students found the review process highly beneficial, both in terms of helping them improve their submitted work, and also in terms of learning how to assess their own work. For example, one student commented "... next I want to review my own site and see how I would assess my work against the marking rubric ... this should be interesting!". Another commented: "... by reflecting on the work of someone else it has forced me to reflect on my own work".
Despite using the most basic HTML forms in the web page used for conducting the reviews, one or more students had some technical difficulties "... I got a little frustrated when I went to submit the review because I had it all disappear into cyberspace ...". Fortunately, these events seemed to be rare.
We were pleased to note that by not making the reviews anonymous, and by giving reviewers and reviewees each other's email addresses, we facilitated communication and cooperation between students. One student wrote
... through a few emails I ended up helping her out with a few things ... and over the next few days were both discussing how we were ... (having technical problems) ... I think we supported each other very well.Another observation was that students were generally positive towards each other in their reviews, complimenting their peers on the clarity of their instructions, the attractiveness of their sites, and the quality of their reflective blogs. Their critical feedback tended to focus on minor spelling errors or problems with links. The reviews consequently tended to focus on superficial features rather than critiquing each other's understandings of the role of technology, and their use of technology as an enabling resource. This highlighted a misconception that many novice teachers hold about the role of technology in education. Many students assumed that technology is something that teachers use to facilitate student learning, whereas the philosophy of this unit is that technology is something that students use to facilitate their own learning. As part of this philosophy, each student's use of technology must be set into a learning context. To help students address this misconception, the marking rubric used to assess their work ranged from: 'No context provided of the social setting where the resource or social networking tool would be used' (fail) to 'Very detailed description of context provided, plus clear evidence of experimentation and suggestions of a range of ways to use the resource' (high distinction).
A number of students gave feedback in their review to their peer that there was no context for the resource that the peer had created. However, (and surprisingly), many of those same students also had no context for their own resource. This suggests that at least some students were recognising that there was a problem in their conceptualising of the role of technology in the assignment.
Because the reviews were not anonymous, students expressed some concern about knowing the people whose site they were reviewing. For example, one student wrote:
I was a little nervous in reviewing as I know what it is like to be an absolute amateur at web pages and blogs! I was hoping to receive some-one who I did not know but this was not the case.However, students generally commented positively on the peer review process in their reflective journals during the course of the unit. They valued the opportunity to view each other's work and were especially appreciative of the comments and suggestions they received from the peers on how to improve their own work. One student wrote:
The pleasing thing for me was to receive an email from the student thanking me for so many good pointers and for being so encouraging. I have since taken another look at the site and was really impressed to see that she had made changes from my comments.
In Figure 3, negative values on the X axis indicate that students marked higher than tutors, whilst positive values indicate that tutors marked higher than students. As can be seen from Figure 3, approximately 30% of students gave the same mark as the tutor for the site they reviewed. The range of differences between the student and tutor was 3 out of 25. Students generally marked slightly higher than tutors by a factor of about 1 mark out of 25, especially for the 'context of use' criterion. This is in keeping with Bostock's (2006) conclusion that there is no consistent finding regarding under- or over-marking by students on a range of studies he reviewed. He concluded, however, that overall accuracy is good and can be improved by using methods such as: providing clear criteria, moderation by a tutor, and providing some experience or training in assessment.
Figure 3: Deviations of tutor marks from student marks for two criteria on the rubric
One strategy suggested for bringing about this conceptual shift is for teachers to engage with their students in a dialogue in which the marking rubrics for the peer review and assessment are developed collaboratively (Sluijsmans, 2002). Such dialogue can deepen student understandings about the content and goals of the unit and hence address conceptual misunderstandings. Topping (2009) refers to this as a developmental process that:
leads toward more sophisticated peer assessment, and the delivery of plentiful feedback that can help learners identify their strengths and weaknesses, target areas for remedial action, and develop metacognitive and other personal and professional skills (p. 26).Such negotiations also give students a sense of empowerment and ownership of their learning, and may thus motivate them to participate more fully in the peer review process (Pearce et al., 2009). Using a dialogue to enhance motivation to engage with the process may reduce the need to motivate students by assessing the quality of their reviews, and thus free up the review process to allow students to conduct more reviews, and thus benefit from a greater diversity of feedback (Pearce et al., 2009).
The generally positive comments from students about the peer review process in their reflective blogs suggest that it is a worthwhile activity, and one from which their learning clearly benefited. This is especially the case for students studying the unit online who would otherwise tend to be isolated and lack opportunities to obtain feedback about their work to help them keep on track.
Finally, by giving students a voice, even if that voice is primarily directed to their fellow students, there is much that we can learn about what our students are understanding - and importantly, not understanding - that can help us to improve our own pedagogical practices.
Boud, D., Cohen, R. & Sampson, J. (1999). Peer learning and assessment. Assessment and Evaluation in Higher Education, 24(4), 413-426. http://dx.doi.org/10.1080/0260293990240405
Gulikers, J., Sluijsmans, D., Baartman, L. & Bartolo, P. (2009). The power of assessment in teacher education. In A. Swennen & M. van der Klink (Eds.), Becoming a teacher educator: Theory and practice for teacher educators. Chapter 12. Dordrecht: Springer.
Herrington, J., Reeves, T. C. & Oliver, R. (2010). A guide to authentic e-learning (1st ed.). New York: Routledge. http://www.routledge.com/books/details/9780415998000/
Howard, C. D., Barrett, A. F. & Frick, T. W. (2010). Anonymity to promote peer feedback: Pre-service teachers' comments in asynchronous computer-mediated communication. Journal of Educational Computing Research, 43(1), 89-112. http://dx.doi.org/10.2190/EC.43.1.f
Keppell, M., Au, E., Ma, A. & Chan, C. (2006). Peer learning and learning-oriented assessment in technology-enhanced environments. Assessment & Evaluation in Higher Education, 31(4), 453-464. http://dx.doi.org/10.1080/02602930600679159
Knight, L. V. & Steinback, T. A. (2011). Adapting peer review to an online course: An exploratory case study. Journal of Information Technology Education, 10, 81-100. http://www.jite.org/documents/Vol10/JITEv10p081-100Knight942.pdf
Lu, R. & Bol, L. (2007). A comparison of anonymous versus identifiable e-peer review on college student writing performance and the extent of critical feedback. Journal of Interactive Online Learning, 6(2), 100-115. http://www.ncolr.org/jiol/issues/pdf/6.2.2.pdf
Mostert, M. & Snowball, J. D. (2012). Where angels fear to tread: Online peer-assessment in a large first-year class. Assessment & Evaluation in Higher Education, in press. http://dx.doi.org/10.1080/02602938.2012.683770
Pearce, J., Mulder, R. & Baik, C. (2009). Involving students in peer review: Case studies and practical strategies for university teaching. Melbourne, Australia: Centre for the Study of Higher Education, The University of Melbourne. http://www.cshe.unimelb.edu.au/resources _teach/teaching_in_practice/docs/Student_Peer_Review.pdf
Robinson, J. M. (1999). Anonymous peer review for classroom use: Results of a pilot in a large science unit. In Teaching in the Disciplines/ Learning in Context. Proceedings of the 8th Annual Teaching Learning Forum. http://ctl.curtin.edu.au/professional_development/conferences/tlf/tlf1999/robinson-j.html
Ross, J. A. & Rolheiser, C. (2003). Student self-evaluation: What research says and what practice shows. http://www.cdl.org/resource-library/articles/self_eval.php
Sims, G. K. (1989). Student peer review in the classroom: A teaching and grading tool. Journal of Agronomic Education, 18(2), 105-108. https://www.agronomy.org/files/publications/jnrlse/pdfs/jnr018/018-02-0105.pdf
Sluijsmans, D. (2002). Establishing learning effects with integrated peer assessment tasks. Lancaster, UK: The Higher Education Academy: Palatine. http://184.108.40.206/archive/palatine/files/930.pdf
Topping, K. J. (2009). Peer assessment. Theory Into Practice, 48(1), 20-27. http://dx.doi.org/10.1080/00405840802577569
van den Berg, I., Admiraal, W. & Pilot, A. (2006). Design principles and outcomes of peer assessment in higher education. Studies in Higher Education, 31(3), 341-356. http://dx.doi.org/10.1080/03075070600680836
Wessa, P. & De Rycker, A. (2010). Reviewing peer reviews - A rule-based approach. Proceedings of 5th International Conference on e-Learning. (pp. 408-418). Penang: ICEL. http://www.wessa.net/download/icel10.pdf?
Wood, D. & Kurzel, F. (2008). Engaging students in reflective practice through a process of formative peer review and peer assessment. In ATN Assessment Conference 2008: Engaging students in assessment. http://www.ojs.unisa.edu.au/index.php/atna/article/download/376/252
Znamenskaia, E. (2000). Future teacher misconceptions concerning educational technology. Doctoral dissertation, University of Connecticut. http://digitalcommons.uconn.edu/dissertations/AAI9991599/
|The articles in this Special issue, Western Australia's Teaching and Learning Forum, were invited from the peer-reviewed full papers accepted for the Forum, and were subjected to a further peer review proces conducted by the Editorial Subcommittee for the Special issue. Authors accepted for the Special issue were given options to make minor or major revisions (minor additions in the case of Boase-Jelinek, Parker and Herrington). The reference for the Forum version of their article is:
Boase-Jelinek, D., Parker, J. & Herrington, J. (2013). Peer reviews: What can we learn from our students? In Design, develop, evaluate: The core of the learning environment. Proceedings of the 22nd Annual Teaching Learning Forum, 7-8 February 2013. Perth: Murdoch University. http://ctl.curtin.edu.au/professional_development/conferences/tlf/tlf2013/refereed/boase-jelinek.htmlAuthors: Daniel Boase-Jelinek is a Tutor and Online Development and Communications Officer at the School of Education, Murdoch University. Email: firstname.lastname@example.org
Jenni Parker is a Lecturer at Murdoch University in Perth. She teaches in the educational technology stream in the School of Education, and her main areas of research are authentic e-learning, new technologies and open educational resources (see http://www.elearnopen.info/).
Jan Herrington is a Professor of Education at Murdoch University. She teaches in the educational technology stream in the School of Education, and her main areas of research are authentic e-learning, mobile learning and design-based research (see http://authenticlearning.info/AuthenticLearning/).
Please cite as: Boase-Jelinek, D., Parker, J. & Herrington, J. (2013). Student reflection and learning through peer reviews. In Special issue: Teaching and learning in higher education: Western Australia's TL Forum. Issues in Educational Research, 23(2), 119-131. http://www.iier.org.au/iier23/boase-jelinek.html