QJER logo 2
[ Contents Vol 10, 1994 ] [ QJER Home ]

Undertaking collaborative school reviews: A guide for collaborators

Joy Cumming and Graham Maxwell


As part of the devolution to schools of aspects of school planning and a focus on the match between school programs and community contexts, Queensland Department of Education schools in Queensland have been required to undertake Collaborative School Reviews (CSRs). There are several purposes for these reviews, but in essence they reflect the goals of self-evaluation of educational performance of a school with respect to its own and community aims and procedures for determining future directions for the school. The reviews are about identifying needs, documenting practice, evaluating outcomes and making recommendations. The process also allows for commendations, recognition that areas of performance may be being achieved at a 'commendable' level. This, in fact, is an important component of Collaborative School Reviews, as it is all too easy to identify issues and problems and to forget to take note of the positive outcomes and successes along the way.

The process of a CSR involves the collaboration of teaching and administrative staff and community representatives such as parents. At some levels it may also involve students. The expectation of the Queensland Department of Education is that the topics reviewed through Collaborative School Reviews will relate to School Development Plans. While some assistance with the review process is available from Regional Officers, the ratio of such personnel to the number of schools, in conjunction with their other duties and responsibilities, means that the assistance available is necessarily limited.

The purpose of this paper is to provide some simple guidelines to collaborative teams undertaking reviews. After several years of involvement with teachers undertaking projects of a similar scale for postgraduate studies and work with a small number of schools as a consultant reviewing their CSR activities, we have identified a number of aspects of the review process which teams need to keep in mind to make their reviews worthwhile but not totally stressful experiences.

Although rarely acknowledged as such, Collaborative School Reviews constitute research activities. They may be research activities of a particular type of action research or evaluation, but they still involve the basic research activities of identifying a problem, issue or question, obtaining information relevant to that focus, and interpreting the information. As such, there are a number of issues and quality controls related to research which are relevant to CSRs.

Principle 1: Keep it small

The principle 'keep it small' applies to identification of the topic of any research activity from a CSR to a PhD thesis. What tends to change according to purpose is the level of detail and conceptual analysis at the data collection and interpretation level.

The focus of any CSR needs to be identified as concisely as possible. For example, a general topic might be the literacy program in the school. This is not an issue or focus question, however, and such a question needs to be developed. There are a number of ways by which delimiters can be introduced. Are you interested first of all in the lower school, a particular year level? Are you interested in the links between year levels? Are you interested in the value of a particular reading scheme used by the school? Are you interested in learning outcomes? Are you interested in parent satisfaction with the literacy program? Are you interested in the link between the school literacy program and a vocational application?

It is necessary to identify such a specific focus, for example,

Teacher satisfaction with the reading scheme(s) used in the school literacy program.
In order to undertake a CSR or research project, it is usually best to pose the focus as a question:
How satisfied are teachers with the reading scheme(s) used in the school literacy program?
Notice that the use of the question format and the word 'how' immediately establish some of the parameters of the review. It seems apparent that the best way to answer this question is to ask teachers. It is also apparent that teachers are not just to be asked 'Are you satisfied with ...?', inviting a yes/no response. What is perhaps less obvious is that the focus question allows the CSR team to investigate not only 'how' satisfied teachers are but also 'why/why not', as well as recommendations for future actions. This would provide sufficient definition and detail for one complete and informative Collaborative School Review and future action.

Although in the world of public funding and political policies it is not possible to be sure of the life of various innovations such as devolution to schools, it is important that collaborative teams see that their CSR could be a part of a larger program of investigation. We advocate that a CSR topic for one year be linked to a three year program, with the second year related to the outcomes of the first or a different aspect of the second. Suppose teachers and administrators wanted to find out also what students' reactions were to various reading schemes? We would see that as a second CSR, which could be conducted at the same time as the focus on teacher attitudes, or which could follow on the next year. We do feel in general, however, that addressing both in the one CSR, with other ongoing reviews as well, makes a project which is too big.

Principle 2: Keep it 'do-able'

'Do-able' is an important word for any research, but particularly action research such as CSRs. It has several dimensions including both size and whether or not the study can be done. You may have refined your topic to a specific focus, but it may still not be do-able. The main problem which arises is whether or not you can actually get the information you require. A specific focus could be 'the effectiveness of use of computers in promoting learning in a Year 8 English classroom'. The topic looks to be well-defined and confined to a particular grade level and subject area. However, words such as 'effectiveness' signal dangers. Such a topic requires considerable definition of terms. What exactly is meant by effectiveness? In comparison with what?

Comparison can be with standards of good practice, or with the objectives of the program, or with an alternative ('normal') program. Some points to remember are:

Within this topic, other terms also need to be defined. How will learning be assessed? Are we talking about content acquisition? Skill development? Process or problem-solving? Transfer to other learning? Who will develop or validate the assessment instruments? Or will an available standardised measure be used, regardless of its appropriateness. The measure of performance outcomes is not a simple matter. Development of an appropriate instrument for a CSR such as this or for annual reviews of performance is complicated.

A related issue in terms of 'do-ability' is the getting of data from people such as teachers, students, parents, others in the community, or from documents or records. In getting any such data, it is important to decide how representative such data need to be. In some cases, information from one or two people chosen for a specific reason may be sufficient. In other cases, you may be able to get the information from the whole relevant population, for example, teachers at a staff meeting or all teachers in a subject area, or all Year 2 students on a given day.

However, when information is required from a representative sample, care needs to be taken in ensuring the sample is truly representative. Many schools want to find out parent attitudes to aspects of school performance. The problem is how to survey parents. How can you get the information into homes? Through children? If the information is requested in a written form, what assumptions are being made about the literacy of the parents or the language spoken at home? Which parents are more likely to respond? Would telephoning be better? During the day or at night? What should be done to follow-up those who do not initially respond?

What biases might be introduced through these data? We know that many schools have already tussled with issues such as these. These problems don't mean that you don't go ahead, only that it is important to be aware of such problems and their impact on the outcomes and interpretation of your CSR.

Principle 3: Make it valid

Under the previous principle we have discussed a number of issues that relate to validity. Validity of research is a difficult concept, and one which educational researchers write about all the time. There are two key issues concerning validity which anyone undertaking a research study, no matter what size, needs to consider. First of all, valid research is research undertaken in a manner appropriate to the context. Secondly, valid research is research which is interpreted appropriately.

In the previous principle, we talked about what types of information can be collected. When you plan a CSR, or any research study, you address five basic questions:

Starting with the last question first, the 'why' is what you want to know and the purpose for the CSR. This you have established through the School Development Plan and discussion, and refined through the identification of your specific question.

The next major question then is 'what'- what information do you want and what information can realistically be obtained. In conjunction with 'what', think about 'how' and 'when' you are going to get the data. There are two major considerations you need to be aware of about data collection. Firstly, as you consider the type of data you want, we urge you to be creative. The quality of data relates to the thoroughness with which you pursue it and the care with which you interpret it, not the degree to which it relates to some ideas people might have about 'real' data. Educational researchers now recognise many different ways of collecting information, from standardised tests, to surveys, to personal diaries and interviews. So, examine your CSR focus question very carefully and decide what is the best way to obtain information for the question. Suppose you wanted to study 'teacher stress'. Which do you think is the more valid way to obtain information?

A survey asking teachers 'Do you feel stressed?'
A survey asking teachers 'Do you feel more stressed than you did a year ago?'
A diary account of one or two teachers recording their activities over a few weeks, reflecting on what are usual aspects of their work, new activities introduced by various changes and the effects of these.
This brings us to the second point to consider about quality of data. Just as the way a sample is chosen (or chooses itself) can introduce bias in information collected, so the nature of questions asked can introduce bias. Be very careful as you design questionnaires or other data collection instruments that you do not ask leading questions or questions with socially desirable answers. Remove emotive terms. You are trying to find out information from someone, not convince them of your point of view. Research is about finding out - not proving a point. You can just as easily influence achievement outcome data by setting tasks which are far too easy or far too difficult.

Once you have thought through all these issues concerning the type of information you need to address your CSR question, you will have already determined the 'who'. Just remember that appropriate information from one person can be as good as or better than scatty information from what might superficially appear to be a more representative sample. Educational researchers undertake a lot of case study research, whether the case refers to one child, one teacher, one subject, one classroom or one school.

Careful consideration of all these issues should mean that you will conduct your CSR in a valid way, that is, appropriate to the context. The final point to remember is that appropriate interpretation is also extremely important for validity. CSRs are not just documentation of current practice but are intended to lead to recommendation, change and action. It is important that you base future action on solid grounds. The first simple rule for appropriate interpretation is not to go beyond the data you have. Don't over-interpret. Don't overgeneralise. Remember the possible sources of influence on the quality of your data such as participant response rate and bias, the nature of the data collection instruments, affective or motivational factors such as novelty, fatigue and churlishness. As you think about these things, take care as you analyse the information you have. When you are collecting information from people, you often lose control over the type of data you are getting. But this does not mean that you need lose control over your own interpretation of the study.

At the same time as you are taking care to interpret the data appropriately, you should be aware that you are also the people who can best make sense of the data. When people collect qualitative data such as comments or attitudes, they often wonder how to analyse such data. A simple approach is often most valid. Collate similar responses. Categorise them and give them an overall label. Ask someone else to do the same to see if they come up with the same categories. If they do, you appear to have interpreted the responses in a systematic way. If they don't, discuss your different interpretations so that you arrive at some consensus. Believe in your own capacity as a collaborative team to make judgements. Just beware that you do give equal weighting to the whole range of responses, not just the responses which agree with your own expectations. Simple tallies of different types of responses and categories can assist you to remain unbiased.

If you take the care suggested for the identification of your CSR focus question, the nature of the best information to be collected, the best source, the best way to get the information in a systematic and unbiased manner, and then interpret the information cautiously, you will conduct a good CSR.

Principle 4: Document the outcomes

A CSR involves a great deal of work and commitment by the CSR team. It is a pity for this effort to be wasted by inadequate records of the processes and outcomes. Some recommendations have been made that only one or two page summaries of CSRs should be made. We consider this very inadequate for no other reason than that documentation should enable dissemination of what has been done to assist future staff and parents at the school to understand the basis for actions, and to assist other school communities who may have similar interests. A clear report of a CSR need not be too onerous. One member of the CSR team could have responsibility for writing the report as you go. Alternatively, each member of a CSR could take responsibility for writing a section of the report, with one person (at least) taking responsibility for making sure the whole report reads coherently.

How long need a report be? Not long, but sufficiently long to document what was done and what was determined and recommended. The basic components of a report are:

Make your report look good. Give it a reasonable binding. Make several copies. Place them in the school library. If possible make copies available to anybody expressing interest. Perhaps one day there will be a central holding place where such reports are kept and can be circulated to schools on request. Regardless, your CSR report should be available to both present and future members of the community. Don't let it get lost amongst a pile of other papers.

Each CSR involves substantial commitment from a group of people. Many of these people are undertaking the work in their own time. The saddest comments we have heard cynically identify CSRs as another bureaucratic requirement, not to be made too much of, to be finished and forgotten. Some of this cynicism might be appropriate. However, we do believe that within CSRs members of school communities have the mechanism for examining their own school's practices and improving the quality of education. A CSR does not have to be large; it is better if it is not. However, since it is going to take time and effort, apply the time and effort well and make sure that the outcome is of use to you. Isn't this what you would advise students?

Hence Principle # 5. Make each CSR something you as a school community want to find out about. Make it meaningful from the beginning. Then the effort, although demanding, will be rewarding.

Principle 5: Make it meaningful

Suggested Readings:

Australian Government Publishing Service. (1988 or newer). Style manual for authors, editors and printers. Canberra: AGPS.
This is a useful book which should be in every school library. As you write your report, this will help with heading levels, punctuation, formatting and so on. Two good sections on apostrophes! Available from the AGPS shops if not in your library.
Bell, J. (1992). Doing your research project. A guide for first-time researchers in education and social science. Milton Keynes, UK: Open University Press.
This little book of 145 pages provides a very practical guide to undertaking small research studies, including some good chapters on document analysis, questionnaires and interviews, diaries and observation, and writing reports. This book should be available through university bookshops or able to be ordered through publishing agents.
Gay, L.R. (1987 or on). (Any edition is good). Educational research: Competencies for analysis and application. Columbus, Ohio: Merrill.
This book is a standard research methodology text for introductory university research method studies. It is a very readable book with good sections on planning and data analysis. It has a step by step guide to some basic statistics which all can follow. It's the only book university students have ever been heard to say they read right through. It's available from most university bookshops but check on prices as they do differ for some reason.
Hopkins, D. (1990). A teacher's guide to classroom research. Milton Keynes, UK: Open University.
Again this is a very practical little book with similar focus to the Bell book. It does focus more on obtaining information within classrooms.
Keats, D. (1988). Skilled interviewing. Melbourne: Australian Council for Educational Research.
This book focuses on interviewing. This is only about 100 pages long with small pages. We like the comments on learning to drink a lot of tea. Very useful guide.
Munn, P. & Drever, E. (1991). Using questionnaires in small-scale research. A teachers' guide. Edinburgh, Scotland: Scottish Council for Research in Education.
This is a small but very practical book on questionnaires. It may be more difficult to obtain. Perhaps you could suggest that regional offices get copies for borrowing!
Wadsworth, Y. (1984). Do it yourself social research. The Victorian Council of Social Service, 290 Wellington Street, Collingwood, 3006. (Also available from Allen and Unwin, ISBN 0 04 300099 1)

Wadsworth, Y. (1991). Everyday evaluation on the run. Action Research Issues Association Incorporated, Ross House, 247-251 Flinders Lane, Melbourne 3000. (ISBN 0 7316 7153 8).

There booklets provide lots of good practical advice in a fun format.
Author details: Joy Cumming, Division of Education Griffith University
Graham Maxwell, Graduate School of Education, University of Queensland.

Please cite as: Cumming, J. and Maxwell, G. (1994). Undertaking collaborative school reviews: A guide for collaborators. Queensland Researcher, 10(2), 24-33. http://education.curtin.edu.au/iier/qjer/qr10/cumming.html


[ Contents Vol 10, 1994 ] [ QJER Home ]
Created 11 Sep 2005. Last revision: 11 Sep 2005.
URL: http://education.curtin.edu.au/iier/qjer/qr10/cumming.html