Issues in educational research Vol 2, No 1, 1992, 35-44.

The ethics and politics of evaluation

Andrew Thompson
Wanneroo Primary School
Evaluation can be regarded as a distinct branch of social science research and, as such, presents evaluators with specific ethical problems not encountered by other researchers. This paper examines ethical difficulties arising from an evaluation study and puts forward for discussion a code of ethics designed to resolve these.

Introduction

All researchers become aware of ethical issues during their studies. For some, the issues will be addressed a priori, and be part of the research design; while others, particularly those involved in evaluation, will confront them in an unexpected fashion as the research unfolds. More often than not, the latter situation pertains, as Sungalia (1987, p.287) has explained:
I had come to evaluate the work of an educational planning unit. That was my purpose. But what transpired as the real purpose of my study? I learned in time that I ... had come to assist in marketing a product: to bottle up the dreams, hopes frustrations and expectations of the members of a planning unit and label all that neatly - "of Value: Not To Be Disbanded".
The above quotation epitomises the problem I wish to examine in this paper. The hapless and perhaps somewhat naive evaluator, one who moves from educational research into the field of evaluation, can very quickly become embroiled in the politics of evaluation. That is, there often is the urge for clients to view evaluation as a means for marketing their 'product', rather than as an equitable means of assessing its merit.

Clear identification of the audience for an evaluation could serve to alleviate such a problem as the one discussed above. Scriven (1981, p.63) has identified formative evaluation as a process conducted, "during the development or improvement of a program.... It is an evaluation which is conducted for the in-house staff of the program and normally remains in-house". This is somewhat different from summative evaluation which Scriven (1981, p.150) has characterised as being conducted for the benefit of an external audience. This distinction allows for the evaluation audience to be identified prior to commencement of the study and so those involved will know beforehand the nature and possible consequences of the undertaking. In theory, therefore, this distinction between formative and summative evaluation is important because it allows participants to provide information, knowing in advance who their audience will be. In practice, however, political and personal differences amopng program staff can make even formative evaluation a task fraught with ethical difficulties related to the collection and disclosure of data.

In this paper I will examine the idiosyncrasies of evaluation research as compared to other modes of social science research, and argue the case for a code of ethics related specifically to evaluation studies, one that is designed to protect the rights of all those involved in evaluation research: evaluator, evaluated and audience.

The scenario

At the beginning of 1990, as my Master of Education research study, I undertook to evaluate a pastoral care program designed to enhance the self-concepts of students at risk of becoming involved in drug abuse, juvenile crime and other antisocial and self-destructive behaviours. The program required a significant amount of community involvement. Members of the local community were trained to work with small groups of students, as 'community tutors', on self-concept enhancing and team building exercises. In addition to this, an executive committee, also made up of community members, was created to coordinate the management of the program. The functions of this committee were to: raise funds for the program; give approval for and allocate funding to the components of the program as they were implemented; and take over continued implementation in 1991, at which time the development and implementation team were scheduled to withdraw to give the community full ownership of the program. Another significant role of the committee was to provide a forum for the airing of disparate views on, and interests in, the program. This was deemed to be important, because one of the program's characteristics was a flexibility which permitted the incorporation of ideas put forward by members of the local community.

Community involvement in the program meant that part of the evaluation brief was to monitor and evaluate the activities of community members participating in implementation. This was necessary because 'stakeholders' from the community were viewed as having the potential to change the program significantly through their input. Thus, the source, nature and effect of these changes needed to be recorded and evaluated. To this end, the roles, actions and motives of individuals and groups within the community who participated in the program were clearly identified as frames of reference for evaluation. It is in this part of the evaluation that ethical problems were encountered.

Ethics

In designing the evaluation I took into consideration the rights of those from whom I would be collecting data. This was of particular importance because the main mode of data collection was to be interviews and, in the course of being interviewed, subjects were likely to reveal personal and/or private information which they might not want to be made public. For this reason, I was acutely aware of the need to guarantee participants' rights to privacy.

Smith (1980, p.193), in discussing the ethics of fieldwork in ease study research, has stated: "The two most important operating principles in maintaining respect for the persons involved in one's study are informed consent and anonymity of participants." In addition to this, Simons (1984, p.88) has made the following statement in discussing procedures for the conduct of an independent evaluation:

Confidentiality is necessary to protect individuals from inappropriate use of information which is private to them. Rules of access and consultation give individuals opportunities to decide what to share, to reflect on what they have shared, to edit or comment upon their information in context: to control, in other words, the use of their own information.
Thus informed, I committed myself to adhere to a code of ethical conduct based on the principles of informed consent and privacy.

Prior to beginning data collection, I gave and explained to participants a document outlining the purpose, usefulness, methods and expected benefits of the evaluation. Also contained in this document was a detailed description of individuals' rights under the ethical code I had adopted. In summary, these rights were:

  1. Potential participants could decide whether to become involved in the evaluation on the basis of information supplied to them detailing the scope, methods and usefulness of the evaluation.

  2. Participation in the research was not compulsory and individuals had the right to withdraw at any time and for any reason.

  3. Sources of information would remain confidential.

  4. Participants were given the opportunity to view interview transcripts to verify their accuracy.

  5. Information was only used with the permission of participants.

  6. Participants had the right to reply if they did not agree with the evaluator's interpretation of data.

  7. Permission to involve students in the research was negotiated with the school, the students themselves, and students' parents or guardians.

  8. The community, school and individual participants were not be identified in the final evaluation report.
In making this commitment, however, I had neglected to attend to the right of the evaluator to carry out an independent and unbiased evaluation.

Evaluation: Politics and privacy

As previously foreshadowed in the introduction, it is necessary at this point to discuss the role of evaluation as compared to other modes of educational research in order to demonstrate the inadequacy for evaluation of the code of ethics outlined above. In discussing the process of evaluation, House (1980, p.18) has stated:
At its simplest, evaluation leads to a settled opinion that something is the case. It does not necessarily lead to a decision to act in a certain way, though today it is often intended for that purpose.... Evaluation leads to a judgment about the worth of something.
These characteristics of evaluation - that it "leads to a settled opinion that something is the case" and to a judgment about the worth of something - place it squarely into the political arena. In the case of the program I evaluated, the execution of this role required me to judge the actions of individuals in the context of their vested interests and motivations for becoming involved in the program. Thus, as evaluator I became involved in the "politics of personal aspirations".

And, in attempting to identify and judge the effect of these aspirations on the implementation of the program, I had to walk the fine line between what was public and what was private.

Simons (1987) also has discussed the role of evaluation, and has illustrated its distinguishing features when compared to other modes of research, one of which is that: "Immediate utility may not be a requirement for research but it is defining for evaluation" (p. 17). This feature of evaluation emphasises the difficulties the evaluator faces in trying to maintain the balance between protecting the rights of individuals and remaining independent and unbiased. The fact that evaluation focuses on the particular (Simons, 1987, p.9), rather than being disposed toward making generalisations (as is usually the case in other forms of research), means that the evaluator is often required to report intimate, and sometimes controversial, data to an audience that is closely involved in that which is being evaluated. The requirement that evaluation be useful in making value judgments also means that the evaluator must weigh the merit of the actions of individuals and then report to an audience which, if it is not made up of the same individuals, will include people associated with them.

These characteristics of evaluation make it almost impossible for the evaluator to guarantee the anonymity of participants. The usual devices, such as the suppression of names and specific locations, do not guarantee anonymity because the audience's knowledge of the scene would lead, just by inference, to their recognition of people and places no matter how they were disguised. Thus, the evaluator is faced with a number of problems: maintaining his/her independence and ensuring that the evaluation is free from bias; protecting the rights of participants; and reporting in such a way that the private lives of participants are not exposed to public scrutiny and judged as part of the evaluation. This brings us back again to the distinction between what is private and what is public.

On this subject, Pring (1984, pp.15-16) has stated:

The distinction between the private life and the public life of an individual is a difficult and blurred one.... The balance .. between the private and the public must relate to the political significance of the authorities and institutions being evaluated.
To illustrate the problematic nature of attempting to make the distinction between private and public life, I will use an example from the evaluation of the program to which I have been referring.

My first interview with the chairperson of the program's executive committee took place on a Monday morning. The previous weekend there had been a major casts in the implementation of the program. Because of this, I questioned the committee chairperson about the crisis and his actions in dealing with it. The answers to my questions revealed some very interesting data which I considered to be highly relevant to the evaluation for the following reasons:

  1. The way in which this person had resolved the crisis indicated a willingness on his part to act unilaterally and without the approval of other committee members.

  2. This person revealed political enmity between himself and another individual on the committee, and stated the opinion that the person to whom he referred was trying to use the crisis to halt the program's implementation and to tarnish his reputation.
Once I had transcribed the interview, I sent the committee chairperson a copy so that he could verify its accuracy. A few weeks later, the transcript was returned to me with the following note attached:
I have ammended [sic] the transcript to concentrate on items relative to the project and have deleted negative or personal comments on purpose. I would like your final report to be available for public reading and I don't think these comments would allow that.
The amendments to the transcript constituted deletion of data related to his efforts to solve the crisis and his political differences with the other committee member. Through these changes it was made clear to me that, on my behalf, the committee chairperson had decided what was and was not relevant to the evaluation. It also showed that this person was willing to use his rights as a subject of the evaluation to censor data and to ensure that the evaluation was neither independent nor unbiased.

The committee chairperson's action in amending the transcript highlights a dispute over whether the data he gave me should remain private or should be made public. As the evaluator, I have no doubt that the data should be seen as public, because they affected the implementation of the program and thus they dealt with actions carried out in the public arena. This person, however, wanted his actions to remain private, because their revelation in the public arena would have been damaging to him in terms of his standing as a public figure in the community.

In any other form of educational research it is clear that the individual's right to privacy must be respected at all times. Any competent researcher would guarantee these rights to participants explicitly in negotiating to pursue his/her research. Moreover, in the case of disagreements over the relevance of data to the research, such disputes could often be solved by reassuring the individuals that their identities will not be revealed and that the audience for the research report will be at least once removed from the research site. If necessary, the researcher could discontinue the research and pursue the same problem at another site, although this would only be a consideration in extreme circumstances.

It is at this point, however, that the weakness in guaranteeing participants' rights to privacy in an evaluation becomes most apparent. The distinctive characteristics of evaluation bring into question the usual guarantees of privacy and anonymity the evaluator is able to offer participants. Ultimately, it is not possible to shift the focus of an evaluation to another site, because it is particular instances or programs that are of interest. Furthermore, offering participants in an evaluation the right to privacy also has the potential to cause disputes over what is private and what is relevant to the evaluation. This is clearly illustrated in the committee chairperson's censorship of his interview transcript.

Evaluation: A special case

In such cases as the one discussed above, I would argue that the individual's right to privacy should not over-ride the overall goals of an evaluation. If an evaluation is to remain independent and unbiased, and thus useful in its role of permitting judgments of value and worth and informing decision making, those who are to be evaluated must, to some degree, be prepared for their actions and interests to be revealed in the evaluation report. This is reasonable, because the evaluation must be necessary for clients' continued functioning, otherwise they would not have requested it in the first place. With this in mind, I would argue that, particularly in cases where the audience and those whose actions are to be evaluated are one and the same (as is usual in a formative evaluation), clients need to be made aware that they cannot use the right to privacy to obstruct the evaluator in making a fair and reasonable attempt to do the job they have requested.

I do not want to suggest, however, that the subjects of evaluation should place their reputations entirely in the hand of the evaluator. The line of demarcation between what is private and what is public should be drawn through a process of careful negotiation. In carrying out this negotiation, I would like to suggest the following guidelines:

  1. The commissioners of the evaluation should check the credentials of the evaluator(s) to satisfy themselves that they are competent for the job at hand.

  2. The evaluator and client(s) should meet to discuss the terms of reference for the evaluation. This discussion should result in a clear and unambiguous written statement of the terms of reference of the evaluation which is agreed upon by all parties. The terms of reference should include an outline of the evaluator's intended methods of data collection and interpretation, so that the efficacy of these can be decided upon before data collection begins. In addition to this, the evaluator should make the commitment that all possible steps will be taken to protect the identity of individuals. This commitment, however, should not be given or taken as absolute. The evaluator should also warn clients that, because the evaluation will be focused on a particular object, organisation or event, it is possible that those being evaluated may be able to recognise themselves and each other in the final report.

  3. A contingency should be made whereby the terms of reference can be renegotiated in the event of any unanticipated occurrences during the course of the evaluation. Any party involved in the evaluation should be able to implement this contingency plan, and the adjusted terms of reference should be agreed upon by all parties.

  4. Participants in the evaluation should agree to the release of any information relevant to the terms of reference. In this way individuals would commit themselves not to attempt to suppress information during the course of data collection and analysis.

  5. During data collection ownership of data would remain with the evaluator. Individuals would have the right to verify the accuracy of data but, at this stage, decisions on what data are relevant or irrelevant to the terms of reference would be made by the evaluator.

  6. To assist the evaluator in deciding the relevance or irrelevance of data, an external auditor should be employed to check the evaluator's decisions. This person should audit data anonymously and, other than being supplied with a statement of the terms of reference, should not be involved with the evaluator or the evaluation in any way.

  7. Initially, the evaluation report should be released to individual participants to give them the opportunity to respond to the evaluator's interpretations of data. If, at this stage, data in the report are considered to be of a controversial nature, consideration should be given to releasing only sections of the report relevant to each individual together with a summary, which will allow individuals to read the sections they have been given in a meaningful context. If necessary, the evaluator should then rewrite the report to include participants' responses at the appropriate places in the text.

  8. The evaluation report should now be presented to clients in camera. The clients should inspect the data presented in the report to satisfy themselves that it complies with the terms of reference. Any disputes over the relevance of data should be discussed with the author of the report so that both side can argue their case, and agreement can be reached over what, if anything needs to be changed. If it is deemed necessary, the report should be re-write by the evaluator to incorporate the agreed changes.

  9. If information contained in the report is of a controversial nature and/ potentially damaging to individuals, further publication by the evaluator should be restricted to a format and audience that reduces to a minimum possibility of specific events, individuals or locations being identified. To this end, the evaluator should report findings in terms of generalisations, thus removing the emphasis from the particular. The agreed version of the report should be viewed by the persons responsible for devising the evaluation's terms of reference. This will allow all parties to satisfy themselves that the format of information contained therein will not be damaging to them. This guideline does not, however, give clients or participants the power to deny the evaluator the opportunity to publish.
Through the application of these guidelines, it would be possible for the evaluator to complete the job he/she has been commissioned to do as independently and as impartially as possible. The clients of the evaluation would be provided with information which has undergone rigorous checks for accuracy and validity in relation to the evaluation's terms of reference. The evaluator would be free to publish an agreed version of the report useful to a wider audience, but not damaging to those who have been evaluated.

Referring back to the quotation with which I began, this would mean that the "dreams, hopes, frustrations and expectations" of those who were participants in the evaluation would remain in their own hands. It would be the clients who would be involved in the 'marketing' of their product and, in this endeavour, they would be free to use the evaluation report as they saw fit. Moreover, the evaluator would be permitted to evaluate without becoming subject to political manipulation and unencumbered by a code of ethics which is not appropriate for his/her role.

References

Adelman, C., Jenkins, D. & Kemmis, S. (1976). Rethinking case study: Notes from the second Cambridge conference. Cambridge Journal of Education, 6, 139-150.

House, E. R. (1980). Evaluating with validity. Beverley Hills: Sage Publications.

Pring, R. (1984). The ethics of evaluation. In C. Adelman (Ed.), The politics and ethics of evaluation, pp.8-18. London: Croom Helm.

Scriven, M. (1981). Evaluation thesaurus (3rd ed.). Inverness, CA: Edgepress.

Simons, H. (1984). Principles and procedures for the conduct of an independent evaluation. In C. Adelman (Ed.), The politics and ethics of evaluation, pp.87-92. London: Croom Helm.

Simons, H. (1987). Getting to know schools in a democracy: The politics and process of evaluation. London: Falmer Press.

Smith, L. (1980). Some not so random thoughts on doing field work: The interplay of values. In H. Simons (Ed.), Towards a science of the singular: Essays about case study in educational research and evaluation, pp.179- 205. Norwich: Centre for Applied Research in Education, University of East Anglia.

Sungalia, H. M. (1987). On doing an organisational evaluation. In R. J. S. MacPherson (Ed.), Ways and meanings of research in educational administration, pp.287-304. Armidale: University of New England.

Author: Andrew Thompson is currently teaching at Wanneroo Primary School. He has recently completed a Master of Education at Edith Cowan University. His research interests include curriculum development, implementation and evaluation and issues associated with the process of change in education.

Please cite as: Thompson, A. (1992). The ethics and politics of evaluation. Issues In Educational Research, 2(1), 35-44. http://education.curtin.edu.au/iier/iier2/thompson.html

[ IIER Vol 2, 1992 ] [ IIER Home ]

© 1992 Issues in Educational Research
Last revision: 9 Aug 2001. URL: http://education.curtin.edu.au/iier/iier2/thompson.html
Previous URL from 17 June 1997 to 7 Aug 2001: http://cleo.murdoch.edu.edu.au/gen/iier/iier2/92p35.htm
HTML : Clare McBeath, Curtin University [c.mcbeath@curtin.edu.au] and Roger Atkinson [rjatkinson@bigpond.com]