|
Learning from evaluations: Probing the reality
Dorothy Spiller and Trudy Harris
University of Waikato
This paper reports on findings from a major New Zealand research project around staff perceptions of student evaluations of teaching. The main focus of this discussion is the insights that the research afforded into staff engagement with and use of student evaluations to inform their teaching practice and to improve student learning. The research data indicates that there is a gap between academics' relatively positive views of evaluation and their actual engagement with the process. A high percentage of academics, particularly at NZ universities, do not report engagement in dialogue or professional development activities around student evaluation. The use of evaluations in teaching tends to be individual, often isolated and unsystematic. This paper recommends some ideas for moving forward and argues for a cultural change that endorses, supports and rewards the systematic integration of student evaluations into teaching practice.
The research project looked at the engagement of academics with student feedback from formal institutional student evaluations. While the research team recognised that there are many other avenues for academics' professional development, the research team was concerned about the possibility of elaborate rituals of data collection that made little difference to the students' learning experiences. Equally they were concerned that students could become cynical and disaffected should they feel that their voices were not being heard. The study probed the perceptions that academics held of student evaluations and the degree to which they engaged with them in order to enhance student learning. The research was a nationally funded collaborative study undertaken by two New Zealand universities and one polytechnic. The findings of the research confirmed that, in keeping with the literature, academics were relatively positively disposed towards student evaluations, accepting them to varied degrees as a feature of contemporary academic life.
The caveats that were raised also replicated the literature and included reservations about students' capacity to judge, reliability of evaluation instruments, fear of manipulation by academics and suspicion of institutional use of evaluations. Despite these qualms, the reported hostility towards student evaluations was not nearly as widespread and intense as the researchers had anticipated. However, this finding was not really a cause for celebration, because acceptance of evaluations did not necessarily translate into engagement with feedback and its use for a process of continuous improvement of student learning.
This paper shares the findings of the New Zealand research and focuses on teachers' engagement with student feedback and the ways in which they incorporate student feedback into the teaching and learning cycle. Overall, our study suggested that there is not a culture of conversation, reflection or action that acknowledges the centrality of student feedback in the continuous improvement of teaching and learning. The evidence from the New Zealand research in these respects will be presented and discussed in this paper. Correspondingly, this paper also begins to articulate the next set of questions engendered by the New Zealand evaluations study. These questions open up the conversation about the kinds of changes that are necessary if student evaluation feedback is to play a pivotal role in enhancing the quality of higher education teaching and learning.
Correspondingly, higher education institutions in NZ have varied expectations around teaching development and student evaluations. In some instances, such as at the University of Waikato, there is no formal probation system and no compulsory professional development, although there is a formal requirement to conduct student evaluations every two years and or for promotion purposes (as the sole indicator of teaching quality). The University of Otago has a probation process, no compulsory professional development, and student evaluations are required within a portfolio for promotion. By contrast, Otago Polytechnic requires all staff members to have teaching credentials and to undertake a student evaluation for all courses.
The literature suggests that one of the barriers to teachers using student evaluation feedback is the difficulties associated with interpreting typical questionnaire data. Related to this concern is the fact that the evaluations process is often an isolated exercise and institutions generally provide very little guidance to staff in relation to the interpretation of student evaluations (Arthur, 2009). Penny and Coe (2004) cited the findings of Cohen's (1981) review which argued that students' ratings on their own were not enough to facilitate teaching improvements and that they needed to be supplemented by a consultation process. Penny and Coe (2004) outlined key elements that they believe should be part of a consultation process. They propose that consultation with teachers around student evaluations and possible teaching improvements work best when other sources of evaluation are incorporated into the discussion. Specifically, they highlight the usefulness of incorporating self-ratings and peer feedback on teaching into the consultations. Another approach was mooted by Smith (2008), who suggested that an interpretative guidance system could help to combat the weak correlation between student evaluations and subsequent improvements to teaching.
The literature reports that along with difficulties in interpreting evaluations feedback, many academics indicated that they do not know how to act on the feedback provided in student evaluations. For example, in trying to account for the disparity between positive attitudes towards evaluations and limited use to inform teaching and learning, Centra (1993) argued that the most significant impediment to teaching improvement is that teachers do not know how to make the appropriate changes to their practice. In order to convert student evaluations into an integral and normal element of professional development, appropriate supporting institutional systems need to be put in place. Smith (2008) argued that for institutional purposes, managers generally focus on aggregate data, but improvements that really enhance the students' learning experience need to happen at the individual level. This is supported by the comments of Penny and Coe (2004) who argued that supporting consultation needs to be context and person specific. Bovill (2011) discussed the importance of thinking about evaluation for learning rather than of learning in her study, and argued for greater and more meaningful involvement of students in evaluation processes. Indeed the New Zealand study argued that:
The models of Ballantyne et al. (2006) and Smith (2008) both provide academic staff members with a degree of personal agency in the interpretation of evaluation results and associated professional development. Ownership is arguably a key component for any system that is designed to integrate student evaluation and professional development more effectively. As Arthur (2009) suggested, academics are "less likely to act on the findings of student feedback if it is collected and analysed centrally (for performativity purposes) because this divorces the findings from the context of teaching and learning" (p. 443). Arthur argued that a "performativity culture" reduces academics' sense of control and influence, and lecturers are more likely to see student evaluations as an imposed ritual. According to the typology he developed, academics in this culture are more likely to respond to negative evaluations with a "blame the students" reaction. This is because, Arthur contends, a performativity culture removes academics from agency for their own teaching. By contrast, according to Arthur, in cultures that emphasise academic professionalism, academics are more likely to modify aspects of their practice ('tame') or re-evaluate it. Discussions about interpretation of, and appropriate support for, subsequent development must therefore be conducted within the broader framework of the institutional use of evaluations data. (Stein et al. 2012, p.17)
One of the questionnaire categories asked specifically about teacher behaviours after receiving student evaluation feedback. At first glance, the questionnaire findings in the research suggest that those academics, who find the student evaluations useful, see them as a way of informing their teaching. Of those who considered evaluations useful (73%), the most commonly cited reason for their usefulness was claimed to be "to inform teacher and course development" (19%), followed by the view that evaluations helped in "identifying students' learning needs" (19%). This apparently strong link between evaluation feedback and teaching and learning is more dubious when academics' actual use of evaluations information is investigated. A high percentage of academics report that they spend time reading the students' comments (95%). However, the percentages decline sharply in response to questions that investigate the practice of continuing conversation with colleagues (47%), with students (16%), or undertaking professional development (12%) in relation to the feedback.
Interestingly, from a learning and development perspective, the majority of teachers at all three institutions appear to deal with their evaluation data in isolation. Even for those teachers who use their evaluations systematically, the responses indicate that there appears to be less discussion with colleagues and/or their teaching teams about evaluation data.
The data also shows that the lowest ranking is given to "seeking assistance with interpreting the results from others" (12%), a finding which suggests that relatively few academics see the evaluations as a springboard for subsequent professional development. Alarmingly, these findings indicate that for many academics the student evaluations are a discreet entity for private perusal and not central either to classroom practice, professional development or departmental cultures.
From the thematic analysis of the qualitative comments there was a recurrent theme concerning the interpretation of evaluation data, and the need for tools and support systems to help with interpretation. This also links to an identified need for professional development in relation to student evaluations. Respondents did make suggestions about how the evaluations system could be used to enhance their usefulness for professional development. These included timing, the use of multiple forms of evaluation, enhanced flexibility and contextual appropriateness of the evaluation instruments. Additionally, suggestions were offered about the need for guidance with interpretation and use of feedback to enhance teaching, and for integrating education around evaluation into the professional development of academics. Examples of comments include the following:
A more collegial/academic mentor and 'professional development' model needs devising, in which student feedback is constructively sought from which teachers would be taught about.The engagement with student evaluation also stalls in terms of communication back to the students. Questionnaire respondents claimed that one of the main factors for not communicating with students about their feedback is that of timing. When most evaluation data becomes available to staff, the cohort of students who provided the feedback has moved on. For example:The one thing I would do is include evaluation education sessions in our whole staff training days. I don't think all staff know how to access the evaluations or that they can modify them to evaluate specific things. I think it would also be good to educate staff about the professional development aspects of evaluations as I think lots of people perceive them as punitive things. I think that any punitive aspects should be played down - not helpful for anyone!
Once students have completed an evaluation for my course, they move on and are not taught in this course again. I do talk with subsequent students about evaluation responses in general terms, but it is not a direct feedback loop to the same students who did the evaluation.In addition to the questionnaire, 20 interviews each approximately one hour's duration were conducted at each institution. These interviews provided an opportunity to examine some of the themes that emerged from the survey in more depth. As with the survey, the interviews suggest that a substantial number of teachers take an interest in the implications of evaluations for the teaching and learning they provide. At both universities, over half of the interviewees reported that they use student evaluations to inform their teaching to some degree. While this is a positive trend, there is a considerable range in the ways in which university interviewees spoke about how they engage with evaluation results and their processes vary from deliberate and systematic usage to a more cursory 'nod' to the student feedback. Only five interviewees at the University of Otago and four at the University of Waikato spoke about deliberate and relatively systematic responses they make to the student evaluation feedback they gathered. Examples of comments included:
I type up a list of the general comments that students make and act on those if I can... I'll often ask them to comment on things I'm trying or testing out, and I want to know if they're working.The comments of the others who responded affirmatively to this question indicated a more piecemeal approach to using student feedback, although in this group there is also considerable variation in approaches. Comments included:I take their comments and use them as objectives for myself of things that I need to change and adjust... I ask myself, how can I transform this into a teaching objective?
Looks at them reflectively and in comparison with past results.... It is just one little thing in amongst the huge amounts of teaching and administration and everything else I'm trying to do.The other respondents at the universities indicated a lack of engagement with evaluations feedback that ranged from apathy to negativity. Comments included:If several students come up with the same idea, then I try to incorporate it or if one student comes up with a brilliant idea, I try to incorporate it.
Needs encouraging to go over them.There was no strong evidence of a pattern of closing the loop around evaluations with very patchy reference to discussion with colleagues and feedback to students. Seven interviewees at the University of Otago and ten at the University of Waikato reported some degree of discussion with colleagues, but this is not a systematic and deliberate process. It would appear that peer conversation around student evaluations is not a normative and expected part of the culture. Comments included:Doesn't use them to improve. Nothing very valuable in them. Vacuous. Written in ten minutes what you have thought about for years.
Informal meeting/discussion with colleagues when looking at the programme as a whole.The university interviewees' comments about feedback were in a similar vein to the questionnaire responses. Six interviewees at the University of Otago and six at the University of Waikato reported some conversation with their students about formal evaluations feedback. This feedback ranged from deliberate purposeful communication to informal chats. Two other points should be noted here. Many of those who report on feeding back to students do so with the next cohort of students. There was also another group who give ongoing feedback to students through regular informal evaluation. This group was particularly noticeable at the University of Waikato. Timing of the formal evaluations is again commonly reported as a drawback. Comments include:Sometimes shares information with other teachers-only informally.
But I tend to say the feedback was better or worse than last year, don't go into specifics that much
Sometimes/informal/in the staffroom but not in meetings.
Important that students believe the process is worthwhile. And that involves them knowing why they are doing it, what might happen as a result of it, and what that might mean to them.There is a clear difference in the extent to which Otago Polytechnic interviewees said they engaged with student feedback as compared with respondents from the two universities. Seventeen reported that they took account of the evaluation feedback provided by students and used it to modify their teaching. While the degree of engagement with student feedback and its incorporation into changes in course design and delivery varied considerably across the interviewees, there was a general recognition of this process as a necessary and constructive routine. Examples of comments included:Does let students know about changes in response to evaluations but not systematically or regularly.
I do discuss mid-semester informal appraisals. I present it to them graphed and categorised and say what we will do about it. I think that's most valuable because of the time it comes.
If there's stuff that's interesting and relevant, I usually discuss with incoming classes what I've learned from previous classes.
I get ongoing feedback just in the normal course of a lecture.. I make it clear to them that we do value feedback, that we're looking at ways to evaluate and improve their learning.
I use feedback to look for themes about learning styles and teaching methods. I use feedback to think about and adjust teaching to engage students in learning.In spite of the high number of interviewees who reported that they found student feedback useful for their teaching, only five of the Otago Polytechnic interviewees reported discussing their responses to student feedback with the students. The relatively small number of academics who saw it as important to complete the feedback loop in this way was similar to the responses from the two universities. Furthermore, even those who did say they discussed their responses with students, did so primarily on the evaluation of courses, as opposed to the evaluation of teaching. Comments included:They are really helpful... useful... I actually will adapt my classes early on to meet as many of these styles that I can.
I see it as quite a continual process... if they are done earlier then there is better feedback and understanding between the students and the teacher.
If I get an evaluation that needs to be actioned, I will tell them this has occurred and this is what we are going to change.As in the case of the universities, the timing of evaluations is cited as a significant reason for not communicating responses to feedback to students.On the whole I don't discuss them. The paper that I co-ordinate is in Semester 2. They have moved on to the next course.
In conclusion, it appears that most teachers engage with the evaluations process, but to varying degrees. Many believe that collecting evaluation data is worthwhile, mainly for ongoing course refinements, and to receive feedback on the students' learning experiences. However, a small number of academic staff members believe that the data that students provide can be biased, and based on poor judgment. There appears to be little feedback of evaluation information to students, mainly because of the timing of the process, but many staff members do indicate that they use other forms of evaluation throughout their teaching.
In terms of professional development, many teachers do not actively seek help with using evaluation data, even though a small group of teachers spoke of problems with interpretation of the data. From the qualitative questionnaire comments, issues were identified with follow-up professional development processes, educating and supporting students around the evaluations process, and also staff engagement being reduced as a consequence of institutional restraints and requirements. It would appear that many teachers deal with their evaluation results in a somewhat isolated and haphazard way.
What is quite clear is the difference in the way that teachers from the polytechnic and the universities engage with the evaluations. The polytechnic's use of evaluations is part of a structured quality assurance process which focuses mainly on the provision of good teaching practice. The universities, meanwhile, have not had such well-defined structures or processes, possibly because their focus is spread more widely across different activities, particularly research.
The findings around engagement with student feedback in the New Zealand study have raised many questions about how the formal evaluations system can be made a more integral part of teaching and student learning, particularly in the universities. Questions have arisen about how instruments, processes and cultural changes could transform evaluations into a core teaching and learning tool. These changes need to occur at all levels.
At the institutional level, there needs to be a well-articulated statement of norms and expectations about the student evaluations and their use for professional development. These norms need to be communicated widely and regularly in all institutional forums and teaching and learning spaces. This research has indicated that without such clarity, individuals tend to do their own thing around evaluations. While there may be some very good individual practices, there are too many uncertainties which individuals tend to deal with in different ways and which are often influenced by personal history or higher education folklore. Complementing this institutional transparency and clarity about and support for the use of evaluations in professional development, institutions should make it obligatory for academics to demonstrate how they use student feedback and how this is communicated to students. Institutions should reward academics through the promotion process, for documenting how to respond to student evaluations for professional development purposes. In keeping with this commitment to and highly public acknowledgement of the professional development purposes of student evaluations, institutions need to recognise and reward multiple forms of obtaining student feedback. Such changes are important not only for enhancement of individual practices, but also can provide the essential framework for the continuous improvement of teaching and learning in the institution.
The need for the institution to clarify and communicate the expectation that evaluations be used for formative purposes has to be complemented by robust evaluation systems and processes that support this goal. These include addressing the timing issue, recognising other modes of evaluation initiated by academics, and reviewing evaluation instruments for their pertinence, helpfulness and usefulness for enhancing the student learning experience. In terms of engagement, it is arguable that the review of the evaluation instruments should include consultation with both staff and students, who will thus have a greater sense of ownership of the evaluations process.
The literature and the research findings suggest that interpretation of evaluations is often a random, ad hoc affair. To make this stage of the teaching and learning cycle as productive for learning as possible, a number of interventions could help. In line with Smith (2008), a set of interpretative guidelines could be developed to aid this stage of the process, and more institutional resources dedicated to providing help with interpretation and the implementation. The comments of university academics suggest that education around the learning possibilities offered by student evaluations is necessary. More generally, it could be argued that compulsory professional development around teaching (that could occur in different ways, such as peer coaching or teaching conversational circles) could help to equip academics with the pedagogical language and tools to make better use of student feedback. These education processes can never be stand alone remedies and the institutional commitment to the use of evaluations to inform teaching and learning needs to be matched at all levels of the organisation. The institution should require discussion and documentation of the use of student evaluation feedback at the department, course and program level. Such a requirement could help to place student evaluations at the centre of reviewing and planning for teaching and learning.
Corresponding to staff education around evaluations, the students, the partners in the dialogue, need to be given ongoing education about ways of providing feedback, and coaching in the process of meaningful evaluation. As critical evaluation and making judgments are widely stated as generic goals of higher education, specific guidance around these attributes should already be embedded into most curricula. Coaching students in the application of these skills and dispositions to the evaluation process should not be difficult. However, the best way of engaging students in thoughtful evaluation is by demonstrating that their views are taken account of in the teaching and learning.
Ballantyne, R., Borthwick, J. & Packer, J. (2000). Beyond student evaluation of teaching: Identifying and addressing academic staff development needs. Assessment and Evaluation in Higher Education, 25(3), 221-236. http://dx.doi.org/10.1080/713611430
Beran, T. N. & Rokosh, J. L. (2009). Instructors' perspectives on the utility of student ratings of instruction. Instructional Science, 37(2), 171-184. http://dx.doi.org/10.1007/s11251-007-9045-2
Bovill, C. (2011). Sharing responsibility for learning through formative evaluation: Moving to evaluation as learning. Practice and Evidence of the Scholarship of Teaching and Learning in Higher Education, 6(2), 96-109. http://www.pestlhe.org.uk/index.php/pestlhe/issue/view/17
Burden, P. (2008). Does the use of end of semester evaluation forms represent teachers' views of teaching in a tertiary education context in Japan? Teaching and Teacher Education, 24(6), 1463-1475. http://dx.doi.org/10.1016/j.tate.2007.11.012
Centra, J. A. (1993). Reflective faculty evaluation. San Francisco: Jossey-Bass.
Ory, J. C. & Ryan, K. (2001). How do student ratings measure up to a new validity framework? New Directions for Institutional Research, 109, 27-44. http://dx.doi.org/10.1002/ir.2
Penny, A. & Coe, R. (2004). Effectiveness of consultation on student ratings feedback: A meta-analysis. Review of Educational Research, 74(2), 215-253. http://dx.doi.org/10.3102/00346543074002215
Smith, C. (2008). Building effectiveness in teaching through targeted evaluation and response: Connecting evaluation to teaching improvement in higher education. Assessment and Evaluation in Higher Education, 33(5), 517-533. http://dx.doi.org/10.1080/02602930701698942
Stein, S. J., Spiller, D., Terry, S., Deaker, L., Harris, T. K. & Kennedy, J. (2012). Unlocking the impact of tertiary teacher's perceptions of student evaluation. Ako Aotearoa, New Zealand. http://akoaotearoa.ac.nz/project/impact-student-evaluations-teaching-behaviour/resources/files/unlocking-impact-tertiary-teachers-perceptions-student-eva
The articles in this Special issue, Teaching and learning in higher education: Western Australia's TL Forum, were invited from the peer-reviewed full papers accepted for the Forum, and were subjected to a further peer review process conducted by the Editorial Subcommittee for the Special issue. Authors accepted for the Special issue were given options to make minor or major revisions (minor additions in the case of Spiller and Harris). The reference for the Forum version of their article is:
Spiller, D. & Harris, T. (2013). Learning from evaluations: Probing the reality. In Design, develop, evaluate: The core of the learning environment. Proceedings of the 22nd Annual Teaching Learning Forum, 7-8 February 2013. Perth: Murdoch University. http://ctl.curtin.edu.au/professional_development/conferences/tlf/tlf2013/refereed/spiller.htmlAuthors: Dorothy Spiller is a Senior Lecturer in the Teaching Development Unit, University of Waikato, New Zealand. She is currently an academic developer and the convenor of the Postgraduate Certificate in Tertiary Teaching. Dorothy has conducted research and published in a number of areas in higher education, including student evaluations. Email: dorothy@waikato.ac.nz Dr Trudy Harris, formerly in the Teaching Development Unit, University of Waikato, is now a Senior Academic Staff Member (Mechanical Engineering) in the School of Engineering, Sciences and Primary Industries at Waikato Institute of Technology (Wintec). Her main research interests include student evaluations of teaching, and their use in the continuous development of the quality of teaching. Please cite as: Spiller, D. & Harris, T. (2013). Learning from evaluations: Probing the reality. In Special issue: Teaching and learning in higher education: Western Australia's TL Forum. Issues in Educational Research, 23(2), 258-268. http://www.iier.org.au/iier23/spiller.html |