IIER logo 3
Issues In Educational Research, Vol 12, 2002
[ Contents Vol 12 ] [ IIER Home ]

Standards framework: Developing scales of achievement in post-compulsory education: A case study

Graeme Lock & Rees Barrett
Curriculum Council of Western Australia
This paper presents the findings of a research project into the establishment of a standards based scale of achievement using the present Western Australian TEE Geography Year 12 Syllabus. The background to the research, including its relevance to the Curriculum Council's Post-compulsory Review, the use of formative assessment, the issue of standard-referenced assessment and the notion of courses of study, is discussed. A rationale for the choice of Geography as the subject context of the research is outlined and the paper continues by describing the procedure involved in undertaking the research project, including a brief discussion of Rasch analysis. The final section discusses the encouraging results of the investigation and suggests the usefulness of conducting further research on the development of scales of achievement.


In May 1998 the Curriculum Council of Western Australia initiated the Post-compulsory Review. The need for a review of post-compulsory education is emphasised by the Curriculum Council's statutory obligation regarding the implementation of the Curriculum Framework from kindergarten to Year 12 (Curriculum Council Act 1997, Part 3, p. 8). Effectively, the Act has placed on the Council a legal obligation to ensure that the Curriculum Framework's learning outcomes are an integral part of reporting in Years 11 and 12. During the existence of the Interim Curriculum Council, attempts were made to link years 11 and 12 subjects with the Curriculum Framework. This exercise established little of any meaning, mainly due to what the Curriculum Framework represents: viz. a student-centred learning philosophy; a set of principles; open ended learning outcomes; and learning, teaching and assessment practices around which K-12 teaching and learning programs should be established. Most of these concepts are not evident in the current Tertiary Entrance subjects.

Implementing the Curriculum Framework in the post-compulsory years presents the opportunity to address a number of issues in the current system, which has operated since the last major review (McGaw, 1984). Essentially, the current review seeks to refocus post-compulsory education from a subject-centred, inputs-focused approach to one, which is student-centred and outcomes-focused.

From a broader perspective, global and national developments have significant implications for post-compulsory schooling: for example, structural and technological adjustments within industry over the past two decades, rapidly changing information and communication technologies, and patterns of international mobility and migration. Agreements and decisions by Commonwealth and State Ministers, at the national level, on the goals of schooling, benchmarking, standards and targets also have far-reaching consequences for the structure of schooling.

Standards and a new assessment paradigm

The global trend to defining education standards with the goal of improving learning outcomes provides another context for this study. Schmoker and Marzano (1999, p. 17, p. 19) noted the power of standards in "providing a shared language about which skills to concentrate on" ... "clear intelligible standards are a pillar of higher achievement. Aligned with appropriate assessments they can help us achieve the dream of learning for all."

One of the key features of assessment practices, as outlined in the Curriculum Framework, is the emphasis on formative (developmental) assessment. This emphasis is somewhat different from present practices in post-compulsory Tertiary Entrance subjects, which tend to rely upon summative and normative assessment. The educational usefulness of formative assessment is well documented in the research literature. An example of such documentation is that of Black and Wiliam (1998, p. 3) who referred to a meta-study (of students aged from five years through to university graduates, across several school subjects and several counties), which showed that "...innovations which included strengthening the practice of formative assessment produce significant, and often substantial learning gains."

Essentially, Black and Wiliam (1998) argued that assessment regimes, which are built on clearly defined standards that provide a focal point for teacher-student interaction, are the most effective in improving learning. Teachers act as facilitators of learning. They work with students to (1) identify the current level of achievement, (2) target the desired level of achievement and (3) describe the changes that will be required to achieve the target level.

The corollary to the use of formative assessment is the emergence of scales of achievement (which describe clearly stipulated standards), against which students are able to monitor their progress. Teachers can also use scales of achievement in the planning of learning experiences designed to enable students to improve their level of achievement. An integral concept relevant to any scale of achievement is that of standard-referenced assessment. In describing this concept, Willis and Kissane (1997, p. 34) wrote "In essence, sets of standards are based on the notion of a continuum of increasing knowledge, quality or competence with the 'standards' intended to provide stable reference points or frameworks against which a particular student's quality of performance or level of attainment or achievement can be judged directly without reference to other students or to overall scores."

Sadler (1989) added to the evidence about the advantage of formative assessment when commenting on self-assessment by students as being a necessary part of formative assessment. He suggested that through such a process students acquire three types of information: the desired goal, evidence of present progress towards that goal and some understanding of how to narrow the gap between the two positions. Sadler's (1989) considered opinion appears to be supporting the use of a scale of achievement and, by implication, standard-referenced assessment.

Thus, the research project under discussion evolved from a combination of the previously described issues, namely, the Post-compulsory Review and the implementation of the Curriculum Framework, with the associated concepts of formative assessment and a standard-referenced scale of achievement.

The research project


The standards framework research project analysed the present year 12 Geography syllabus to determine if a scale of achievement could be developed. Geography was selected as the subject context for the research for a number of reasons. First, the Geography Syllabus Committee had been working, for some time, on a proposed syllabus change, but this had been interrupted by the Post-compulsory Review. Second, the active citizenship outcome and process skills are readily apparent in the Geography course. Both of these elements are important within the Society and Environment learning area. Subject experts were readily available to participate in the project, with samples of student work being available from an external examination. In addition, one of the exploratory courses of study being developed in the Society and Environment Learning area is based on the discipline of Geography and related learning outcomes.


The research project followed a set of clearly defined steps, as shown in Table 1.

Table 1: Steps in the implementation of the Geography standards project

Analysis of current year 12 geography syllabus
Step 1 Analysis of the current year 12 Geography syllabus to identify subject outcomes and their links with the Curriculum Framework outcomes.
Step 2Mapping of the subject outcomes to identify levels with existing progress maps: e.g. Education Department of Western Australia's Student Outcome Statements.
Step 3Using the levels analysis in step 3 to propose a generic scale of achievement for each subject outcome.
Analysis of student work samples
Step 4Designing a marking key for the 1998 Geography TEE paper according to outcomes and trialing this key with a small number of 1998 TEE Geography scripts.
Step 5Marking a sample of scripts from the 1998 Geography TEE paper according to outcomes.
Step 6Analysing the marks using Rasch item response theory.

Identifying subject outcomes

The current Year 12 Geography syllabus comprises 46 conceptual objectives and 42 process objectives. Using the introductory comments for each section of the syllabus it was possible to identify 10 subject outcomes, as shown in Table 3.

Table 3: Mapping the relationship between Curriculum Framework outcomes and syllabus outcomes

Table 3

This analysis established that the current Geography syllabus is relatively narrow in the range of Curriculum Framework outcomes it covers. Of the ten subject outcomes, eight were linked to the Place and Space outcomes. Furthermore, six of these eight outcomes were related to Features of Places. The only other Curriculum Framework outcome represented was Investigation, Communication and Participation (two subject outcomes). Both of these outcomes were linked to processing and interpreting information. There was no opportunity for students to develop in the areas of Planning Investigations, Conducting Investigations or Evaluating and Applying Findings.

Mapping against existing progress maps

The context and content of the ten subject outcomes were used to map against the Education Department of Western Australia's Student Outcome Statements (SOS), which were derived from the National Profiles for Society and Environment. (In Western Australia each of the education sectors/systems is responsible for the progress maps used in measuring the degree to which a student achieves each Curriculum Framework outcome.) Reference was also made to the Oregon Proficiency Standards and the NSW Geography Course standards.

Through this analysis it was concluded that the current Geography syllabus mapped to levels 6 to 8 for the Place and Space outcome, but at significantly lower levels for the Investigation outcome. In essence the current syllabus lacks breadth and depth.

This kind of analysis, as revealed in Table 2, would be most useful in identifying priorities for redeveloping the current syllabus.

Table 2: Mapping of Geography outcomes against Education Department of
Western Australia's student outcome statements

Curriculum Framework
Learning Area Outcome
Education Department of Western
Australia's student outcome statements
1S & E - Place and SpacePS 6.1 to 8.1 (Features of Places)
2S & E - Place and SpacePS 6.1 to 8.1 (Features of Places)
3S & E - Place and SpacePS 6.1 to 8.1 (Features of Places)
4S & E - Place and SpacePS 6.3 to 8.3 (Care of Places)
5S & E - Place and SpacePS 6.1 to 8.1 (Features of Places)
6S & E - Place and SpacePS 6.1 to 8.1 (Features of Places)
7S & E - Place and SpacePS 6.1 to 8.1 (Features of Places)
8S & E - Place and SpacePS 6.2 to 8.2 (People and Places)
9S & E - Investigation, Communication etc.ICP 3.3 to 5.3 (Processing and Interpreting Information)
10S & E - Investigation, Communication etc.ICP 3.3 to 5.3 (Processing and Interpreting Information)
Note: S&E - Society and Environment; PS - Place and Space; ICP - Investigation, Communication and Participation

Proposing a scale of achievement

Table 4 represents the draft scale of achievement that was based on this analysis. It provided a useful reference point in developing an outcomes-focused marking key in the next step.

Table 4: Generic performance levels

0Not demonstrated.
1Simplistic/elementary: very little knowledge, limited understanding of processes, limited skills.
2Satisfactory: accurate description and some application; limited development of concepts; evidence of developing skills.
3Substantial: limited contextual application; detailed development of concepts; well-developed skills.
4Sophisticated: apply in variety of contexts; hypothesise from conclusions; highly developed skills; recognition of relationships; highly detailed development of concepts.

Development of the marking key

The development of a marking key required reference to a sample of student responses to the 1998 TEE Geography paper. While this paper was not examining an outcomes-focused course, which was acknowledged during the briefing given to markers and those who developed the marking key, the mapping of the relationship between Curriculum Framework outcomes and the Geography syllabus outcomes had demonstrated a strong relationship between the two sets of outcomes. In addition, the instructions given to those who developed the marking key specified that if an item failed to address an outcome then it should not be included in the marking key. Such scrutiny would enable the researchers to generate a general description of the type of response that students at each level would be likely to write. Throughout the design process a clear focus on the appropriate outcome was maintained. In designing the marking key, the participants in the project were given specific guidelines (Peck, 1999(a), p. 23), which included Three experienced Geography teachers were involved in the development of the marking key. The process took two days as it represented a paradigm shift from the traditional marking key. One example summed up the difference between the ranking focussed marking approach and the outcomes-focused approach being developed. In the 1998 TEE paper one of the short-answer questions required students to differentiate between the functions of two settlement types - a farmstead and a metropolis. Students sometimes understand the concepts but have difficulty in applying them to the specific question asked. It took some time for the marking team to reach agreement that the first level of demonstrating the outcome in this question involved understanding the concepts but not correctly applying them. In the traditional marking key these students would have been awarded zero marks.

This process also provided an insight into one useful form of professional development. A marker commented after the two-day exercise "It took us some time to shift our thinking, but now I've been through that I think I'm beginning to understand what an outcomes-focus is about. That is the most effective professional development I've had and I'll use a similar approach with my staff."

Marking a sample of scripts

A small scale marking exercise was undertaken to verify the marking guide and to establish the comparability between markers. Following minor modifications, 319 scripts were marked by a group of experienced Geography teachers. A two-hour training session was provided for this group of six markers.

Analysing the marks

The Rasch model was used to analyse the collected data. This model assumes that a unidimensional scale connects the data. Given that great care was taken to ensure that the marking focused on one specific outcome, namely Place and Space (from the Society and Environment learning area), the selection of the Rasch model appears to be based on a reasonable assumption.

The marks were transferred to a computer file, verified and analysed using RUMM (version 2.7q). This program fits the data to a unidimensional latent trait by calculating the locations, on a scale of achievement, of each item in the examination. As stated by Peck (1999a, p. 1) "The location is a measure of difficulty, or alternatively of progress along the dimension of achievement represented by the Outcome."

Using Rasch analysis it is possible to calculate the location of each threshold - a point on the scale between two consecutive levels of achievement. A student who is located above a threshold is more likely to score the higher of the two marks.


The analysis showed that the data made a good fit to a unidimensional model, and the reliability index (in this case a person separation index was used, which is technically different from the Cronbach alpha statistic that is used to evaluate the TEE) of this set of data was 0.837. This is relatively high (compared with a reliability of 0.76 using the TEE marking guide) and supports the assertion that the data made a good fit to the model.

Preliminary analysis revealed that four items had redundant response categories and were combined, and that another item misfitted badly and was removed. Figure 1 shows each item's thresholds. The reader should note that when interpreting this figure the criterion to consider is that if all of the individual thresholds are aligned on the scale, then generic performance levels have been demonstrated to be applicable in a number of contexts (item to item). In this particular case, there are few exceptions (for example, items 115 and 22a display reverse thresholds) to this criterion of judgement. However, Peck (1999a, p.4) suggests that these exceptions arise

because the sample of data is not large enough to smooth out the statistical fluctuations. These reversals are probably not serious enough to be of concern.
Figure 1

Figure 1: Performance Level Thresholds, Geography TEE Responses, 1998.


The successful calibration of levels of achievement, based on Outcomes, on a scale of achievement, demonstrates that it is possible to reach a reasonable agreement between estimates, based on assessment in a variety of contexts, of levels of achievement of an Outcome. However, the thresholds between levels of achievement were not evenly distributed on the scale. Should it be desirable to make them equally spaced then this could possibly be accomplished by making changes to the marking guide.

In all probability, the alignment of levels of achievement would have been better if the task that students responded to had been written from an outcomes-focused syllabus, rather than in a TEE paper. The availability of responses to TEE questions and the lack of an outcomes-focused curriculum were the main reasons for the selection of the responses' sample. An Outcomes-focused system of pedagogy and assessment would have a set of generic achievement levels similar to those, which appear in Table 4 (Peck, 1999a, p. 3). Subsequently, assessment tasks and specific marking guides would be designed to give students the opportunity of demonstrating their level of achievement of the Outcomes.

In commenting further on the results, Peck (1999a, p. 5) stated, "An incidental product of the Rasch model analysis of this data was a measure of each student's ability. Although it was not the purpose of this analysis to compare these abilities with the results of marking the TEE with the traditional marking guide, it may be of interest to note that the correlation coefficient between the two sets of scores was 0.837. This relatively high correlation suggests that it would make little difference to students' ranking whether a TEE marking guide or an outcomes-focused marking guide was used. (Note: the benefits of an outcomes-focused education can not be demonstrated by a correlational comparison of this type.)"


Overall, the results of this research project have shown that the use of an outcomes-focused assessment process has credibility when applied to measuring levels of student achievement in TEE subjects. Indeed, Barrett (cited in Curriculum Council of Western Australia, 1999, p. 6) commented "This is very empowering for students because it will provide them with a record of what they are able to do, and will certainly help address concerns of any teachers who believe that the overall standard of TEE students has been improving since the early 1990s, but that the marks they are awarded in the TEE do not show this."

The use of an achievement standards framework has the additional advantage that it can rank students for university entrance, while also reporting on what these students have actually learned. Further research into the development of scales of achievement should help contribute to demonstrating the credibility of this type of assessment process.

The results of this research provide a response to those critics of outcomes-focused curriculum who suggest that such an approach will result in the "dumbing-down" of education. Indeed, the higher reliability index of the outcomes-focused marking key, by comparison with the TEE marking guide, and the strong correlation in the ranking of the students when comparing the two marking keys suggests that, in this instance, such a suggestion is not in evidence. Furthermore, throughout the process of developing the marking key, the teachers involved were of the opinion that the required criteria to achieve at the higher levels were substantial.

The research process used in this investigation provides a model for future research in this field. With the development of outcomes-focused courses of study to occur in the near future, valuable information has already been gathered, particularly in respect to development of assessment tasks and associated marking keys. The way, in which the marking key was developed, with active teacher involvement, also provides guidelines for the development of effective professional development programs as curriculum change is implemented in the first decade of the 21st century in Western Australia.

The research demonstrated that it is feasible to set valid achievement standards using curriculum documents and student work samples. The research methodology used in the analysis of student work samples emphasised the role of social construction of valid standards; that is, as Wiliam (1996, p. 287) argued, the standards exist "by virtue of a shared construct in a community of practice". Further research is needed to test the robustness of these standards in 'high-stakes' assessment such as that used for university selection. A starting point for this research is the high degree of correlation noted by Peck (1999a) between the rankings achieved by the 319 students in the study sample on the outcomes-focussed marking scale and the actual rankings achieved in the 1998 TEE.


Barrett, R. (1999). Society and Environment Standards Project. Unpublished paper. Curriculum Council of Western Australia.

Black, P. and Wiliam, D. (1998). Inside the Black Box: Raising Standards Through Classroom Assessment. London: King's College.

Curriculum Council Act 1997. Perth: Government of Western Australia.

Curriculum Council of Western Australia (1998). Curriculum Framework. Osborne Park: Curriculum Council of Western Australia. [verified 2 Jan 2002] http://www.curriculum.wa.edu.au/pages/framework/framework00.htm

Curriculum Council of Western Australia (1999). Curriculum Update 12. Osborne Park: Curriculum Council of Western Australia.

McGaw Report (1984). Assessment in the upper secondary school in Western Australia: Report of the Ministerial Working Party on School Certification and Tertiary Admission Procedures. Perth: Government Printer.

Peck, R. (1999a). A Method for Defining Standards for Outcomes. Unpublished paper. Curriculum Council of Western Australia.

Peck, R. (1999b). Outcomes Standard Project. Unpublished paper. Curriculum Council of Western Australia.

Sadler, R. (1989). Formative Assessment and the Design of Instructional Systems. Instructional Science, 18, 119-144.

Wiliam, D. (1996). Meanings and Consequences in Standard Setting. Assessment in Education, (3)3, 287-308.

Willis, S. & Kissane, B. (1997). Achieving Outcome-Based Education. ACT: Australian Curriculum Studies Association.

Wright, B.D. & Stone, M.H. (1979). Best Test Design. Chicago: Mesa Press.

Authors: Originally trained as a History, Geography and English teacher, Mr Rees Barrett worked for eighteen years with the Education Department in schools, central office (Curriculum Branch), and a district office. Rees was involved in the development of the design brief and the profiles for Society and Environment learning area in the National Collaboration in Curriculum Project. More recently he has led the Curriculum Council's Key Competencies project and Common Assessment Framework project.

Formerly a History Head of Department, Dr Graeme Lock also has extensive experience in lecturing graduate and postgraduate university students in areas including curriculum theory, practice and evaluation; the advanced study of teaching; educational policy studies and educational administration. Previous conference papers and presentations have covered aspects including curriculum evaluation, community participation in school decision-making, teacher occupational stress and small group learning.

Please cite as: Lock, G. and Barrett, R. (2002). Standards framework: Developing scales of achievement in post-compulsory education: A case study. Issues In Educational Research, 12(1), 35-48. http://www.iier.org.au/iier12/lock.html

[ Contents Vol 12 ] [ IIER Home ]
© 2002 Issues In Educational Research. This URL: http://www.iier.org.au/iier12/lock.html
HTML: Clare McBeath [c.mcbeath@bigpond.com] and Roger Atkinson [rjatkinson@bigpond.com]
Created 6 Jan 2003. Last correction: 30 Aug 2013.