QJER logo 2
[ Contents Vol 2, 1986 ] [ QJER Home ]

In support of an interpretative mode of naturalistic inquiry in educational evaluation practice - responsive, illuminative and democratic evaluation reviewed

Judith Hewton
Research Services Branch
Department of Education, Queensland

INTRODUCTION

The psycho-statistical mode of evaluation has dominated educational enquiry this century, and continues to do so in the face of a growing body of literature which has been accumulating since the early seventies, arguing for another way of looking at educational research.

This paper supports an evaluation style based on an interpretative mode of naturalistic enquiry.

The significant evidentiary issue of negotiation is examined in the light of the failure of the psycho-statistical mode of evaluation to affect educational practice significantly. This failure may be attributed, in part, to the absence of any substantial process of negotiation which could serve to facilitate communications by better informing practice and appealing to professional judgment. In relating naturalistic observation and the issue of negotiation to current educational evaluation practices, it is evident that there is a need for reform which involves starting from where the evaluating body is at, and proceeding incrementally in stages so designed that any established system can realistically tolerate the changes to be effected.

DOMINANT EVALUATION MODE

The traditional evaluation design is based on measuring the effectiveness of a program 'by examining whether or not is has reached required standards on prespecified criteria' (Kemmis & Robottom, 1981:152). This dominant form of evaluation based on measurement of changes in pupil behaviour and formal testing (Tyler, 1949; Popham, 1969) has been criticised for being narrowly conceived. It is considered ill-suited to new evaluation purposes demanded by different, new programs designed to ameliorate what Pagano and Dolan identify as the widespread social crises of the last two decades (1980:367).

BACKGROUND TO THE DEVELOPMENT OF NATURALISTIC ENQUIRY

The expansion of the evaluation of curriculum and educational programs which occurred in the 'sixties was a result of government intervention in education, particularly in the U.S.A. According to Cronbach (1951), curriculum-makers continued to resist evaluation in the post-Sputnik period until pressure from a management-oriented government combined with the following to mark a period of growth:
the policy-analysis community realised that field research could provide a better basis for planning;

criticisms of early large-scale attempts at evaluation led to the promotion of differing methodologies; and

both politics and science were recognised as integral aspects of evaluation. (Cronbach, 1981: 34-35)

As a result of the methodological inappropriateness of a management-science orientation to educational evaluation, new methodologies were sought. Hamilton believes that 'methodologies will be deemed appropriate not simply on the basis of their internal characteristics but also on their ability to render the world more accessible to its inhabitants' (1978:94). Naturalistic enquiry, as typified by Stake, Macdonald and Parlett and Hamilton, represents the first substantial move towards what Parsons calls 'the more flexible, comprehensive, sensitive, "understanding" approach to evaluation' (1976:126).

RESPONSIVE EVALUATION

According to Davis (1980:26), Stake is recognised as possibly the most influential, prolific writer and practising evaluator of this period. From a psychometric background, he came to believe in the importance of the evaluation's responsiveness to both the particular problem and to the needs of those wanting information. In Stake's responsive evaluations, the evaluator negotiates with the client, and responds to a variety of different audiences (House, 1980:40):
An educational evaluation is RESPONSIVE EVALUATION (1) if it orients more directly to programme activities than to program intents, (2) if it responds to audience requirements for information and (3) if the different value-perspectives present are referred to in reporting the success and failure of the program (Stake, 1972: 3.28, not paginated).
Stake relies on natural communication, based on observing and reacting. Having conceived a plan of observations and negotiations, the evaluator periodically has participants react to the accuracy, importance and relevancy of his findings (Stake, 1975:28).

Stake's alternative to pre-ordinate evaluation represents a broader, holistic viewpoint. To portray the processes of the educational situation, Stake builds the structure of the evaluation on issues identified by the participants. Responding to practitioners definitions of 'situations, conceptual structures and language' means involving the evaluator in 'a continuous process of negotiation with those involved' to validate the data collected and to accord confidentiality where deemed by any participant to be important (MacDonald and Walker, 1975:7). Negotiating what knowledge may become public and what is to remain private is essential to maintain 'the integrity of both the record and those under study' (Walker, 1980:55).

Although responsive evaluation avoids rigid, narrow outcomes of little interest to the participants, it is not without its critics. Scriven expressed concern at what could be a lack of structure and a lack of valid proof of claims made in the reporting (1981:138). In attempting to maintain the location of the inquiry mode within the logico-empirical tradition (Pagano and Dolan, 1980:371), Stake requires that the evaluator be scientist, multimedia expert and artist-storyteller. Such extensive personal skills may not easily be found in any one individual.

The issue of negotiation brings into consideration the controversial concept of multiple realities which results from the need to represent different value perspectives. In seeking to fairly represent different points of view by negotiating the accuracy of transcripts, the evaluator - viewed as 'expert' by Stake - has the impossible task of discovering each individual's perception of reality.

ILLUMINATIVE EVALUATION

In Britain, a parallel could be drawn with the work of Parlett and Hamilton, whose illuminative approach resembles Stake's notion of portrayal, and claims a social-anthropological perspective for evaluation methodology. 'Evaluation as illumination' is an attack on the dominant 'agricultural-botany' paradigm, and a proposal for descriptive, interpretative evaluation:
It aims to discover and document what it is like to be participating in the scheme, whether as teacher or pupil; and, in addition, to discern and discuss the innovation's most significant features, recurring concomitants, and critical processes (Parlett and Hamilton, 1972:13).
Scriven described illuminative evaluation as pure process evaluation, multi-perspective description and interpersonal relations (1981:74). He believes that this model has no theoretical basis - rather it encompasses only some procedural recommendations. According to Scriven, both responsive and illuminative modes run the inherent risk of substituting detail for evaluative conclusions (1981: 20). While such studies may result in reasoned interpretation, the weak conceptual framework of illuminative evaluation fails to appeal to the theory of anthropology which it claims as its origin. It is also doubtful that the importation of anthropological tradition into educational research would be of value given the failure of that discipline to deal with continuous change which is an inescapable conditior1 of the educational process (Parsons, 1976:114). In summary, Parsons criticises illuminative evaluation on four points:
inadequate representation of research tradition, insufficient emphasis on field workers' skills,

scant consideration of the role of extant theories an conceptualisations, and

demonstration of a narrow vision of the widening context of evaluation studies. (1976:108-109)

Parsons does acknowledge, however, that illuminative evaluation has come to be seen as a base from which humanistic studies which accommodate 'the intentionality of human action, the importance of subjective opinions and perspectives, and the possibilities and implications of multiple realities' may proceed (1976:106). Although the evaluator aims to 'sharpen discussion, disentangle complexities, isolate the significant from the trivial, and raise the level of sophistication of debate' (Parlett and Hamilton, 1976:99), this cannot be done without recourse to his or her own language. The evaluator cannot feasibly be confined to the language of the different audiences. Concern for the rights of audiences and participants is an important feature of what Kemmis refers to as 'democratizing evaluation' (1976:360) which originated with another British evaluator, MacDonald.

DEMOCRATIC EVALUATION

MacDonald (1974) developed the idea of diverse audiences making their own judgements from investigative reports in his conceptualisation of the evaluator's role. He classifies such roles into three types on the basis of their power relationships, namely: bureaucratic, autocratic and democratic; and stresses the need to recognise the political nature of evaluation. He also believes that the political orientation of the evaluator should be made explicit in the course of the inquiry. Evaluation, for MacDonald, is the process of identifying 'those who will have to make judgements and decisions about the program, and .... (laying) .... before them those facts of the case that are recognised by them as relevant to their concerns' (in Tawney, 1976:127).

Walker (1980:52) recommends democratic evaluation and emphasises that the researcher is placed in the position of 'having to negotiate his interpretations with those involved in the study rather than being free to impose them on the data'.

In promoting the democratic mode with its key justificatory concept 'the right to know', MacDonald (ibid:134) assumes that democracy is capable of its own transmission. This false assumption illustrates the need to address the issues raised by the very 'undemocratic' structures existing within educational systems. All participants do not have equal access to democratic processes. In reality, time and money do not permit the desirable processes of negotiation to continue to the extent that MacDonald would deem necessary. His work is vital, however, as a strong consciousness-raising exercise to warn the evaluator against actions which would only serve to heighten the 'undemocratic' nature of the system or institution.

Scrimshaw (1979:35-43) identifies three major sub-types of illuminative evaluation, of which the 'honest broker' equates to MacDonald's democratic evaluator. Scrimshaw is critical of the assumption that audiences are willing and able to 'tease out possible lines of responses for themselves' (ibid:39). Lakomski (1983:273) is concerned that such evaluation studies could actually serve to affirm the status quo, and legitimate the existing power structure by failing to ask probing questions or judge worth.

Nevertheless, MacDonald's representation of the 'power diffusion thesis in education' (House, 1980:149) is most instructive. Equally important is the moral idea of 'mutual consent', which House believes should be included in the evaluation agreement - not only in the outcomes (ibid:151). MacDonald's three basic concepts of accessibility, confidentiality and negotiation afford the participants with the opportunity to control the editing and release of data (1975:7), although it is acknowledge that the participants may need support in order to learn how to exercise their rights.

CONCLUSION

The work identified in this paper as 'responsive', 'illuminative' and 'democratic' evaluation has led the field in a proliferation of alternative ways of thinking about evaluation, and, despite criticism, have served to strengthen qualitative methodologies.

For such naturalistic observational modes of inquiry, negotiation is a central feature. Once this democratic principle gains a degree of acceptance, progress can be made towards the ideal seen as persuading participants of the need to present their own interpretations of what has happened, or is happening in their schools. Time is always a factor which can work against the realisation of this ideal, and careful planning is necessary to guard against this.

This is 1986 and it is obvious to those in the vanguard of current educational evaluation studies that time has passed by the work of those early writers in the field who first argued for an alternative to pre-ordinate evaluation in the forms of 'illuminative', 'responsive' and 'democratic' evaluation. This paper, however, considers that, springing as they did from the educational evaluation background still considered most appropriate by many educational evaluation bodies today, these arguments for naturalistic observation in evaluation form an ideal starting point for the purposes of reform in difficult situations which could prove intolerant of attempts to effect change. Of course the eventual aim is to extend the existing parameters of educational evaluation beyond naturalistic observation to encompass a wider conceptualisation of evaluation that may serve to enrich the portrayal of educational processes in action today.

REFERENCES

Cronbach, L. et al. Towards Reform of Program Evaluation, Jossey-Bass, San Francisco, 1981.

Davis, E. Teachers as Curriculum Evaluators, George Allen & Unwin, London, 1980.

Hamilton, D. 'On generalisation in the educational sciences', 1978 (conference paper).*

House, E. Evaluating with Validity, Sage Publications, California, 1980.

Kemmis, S. 'Telling it like it is: The problem of making a portrayal of an educational program' in L. Rubin, ed. Handbook of Curriculum, Allyn & Bacon, Boston, 1976:359-373.

Kemmis, S. & Robottom, I. 'Principles of procedure in curriculum evaluation', Journal of Curriculum Studies, 13:2, 1981, 151-155.

MacDonald, B. & Walker, R. 'Case study and the social philosophy of educatior1al research', Cambridge Journal of Education, 5:1, 1975, 2-10.*

MacDonald, B. (1974) 'Evaluation and the control of education' in Tawney, D. [reference incomplete]

Pagano, J. & Dolan, L. 'Foundations for a unified approach to evaluation research', Journal of Curriculum Inquiry, 10:3, 1980, 367-381.

Parlett, M. & Hamilton, D. 'Evaluation as il1umination', in Hamilton, D. et al., Beyond the Numbers Games: A Reader in Educational Evaluation, Macmillan, Basingstoke, 1977, 4-5.*

Parsons, C. 'The new evaluation: a cautionary note', Journal of Curriculum Studies, 8:2, 1976, 125-38.*

Popham, W. et al. Instructional Objectives, A.E.R.A. Monograph Series, Rank McNally, Chicago, 1969.

Reynolds, D. 'The naturalistic method of educational and social research: a Marxist critique', Interchange, 11:4, 1980-81, 77-89.*

Scrimshaw, P. 'Illuminative evaluation: some reflections', Journal of Further and Higher Education, 3:2, 1979, 35-43.

Scriven, M. Evaluation Thesaurus, 3rd ed., Edgepress, California, 1981.

Stake, R. 'Responsive Evaluation', University of Illinois, Illinois, 1972 (mimeo).

Stake, R. 'To evaluate an Arts program', in Stake, R. ed., Evaluating the Arts in Education: a Responsive Approach, Merrill, Colombus, Ohio, 1975, 14-31.*

Tawney, D. ed. Curriculum Evaluation Today: Trends and Implications, MacMillan, London, 1976.

Tyler, R. Basic Principles of Curriculum and Instruction, University of Chicago Press, Chicago, 1949.

Walker, R. 'The conduct of educational case studies: ethics, theory and procedures', in Dockrell, W. & Hamilton, D. eds., Rethinking Educational Research, Hodder & Stoughton, London, 1980, 30-63.*

* These articles were reproduced in, and cited from: Perspectives on case study 1: Book 2 - naturalistic observation, 2nd ed., Deakin University, Victoria, 1983.

Please cite as: Hewton, J. (1986). In support of an interpretative mode of naturalistic inquiry in educational evaluation practice - responsive, illuminative and democratic evaluation reviewed. Queensland Researcher, 2(3), 3-11. http://www.iier.org.au/qjer/qr2/hewton.html

[ Contents Vol 2, 1986 ] [ QJER Home ]
Created 17 Mar 2008. Last revision: 18 Mar 2013.
URL: http://www.iier.org.au/qjer/qr2/hewton.html