In its most basic form competency based assessment is a pass versus not pass or a can do versus cannot do system which provides the minimum differentiation amongst students. The aim of this paper is to explore approaches which would facilitate differentiation amongst students. These include approaches to differentiation which are additional to (and hence outside) the competency based assessment system, as well those which form an integral part of it. Within the discussion we have examined the various options on grounds of cost effectiveness, manageability, rigour, validity and capacity for consistent application.
There are basically, only three ways in which an individual's performance can be described: in relation to other people, in relation to specified criteria, and in relation to his or her own previous performance. In the jargon, we call these norm, criterion and self (or ipsative) referencing. All three have their uses, depending upon the purposes to which the results of assessment are being put, and all three affect the methods used for assessment, the ways in which performance is reported and recorded, and attitudes and practices in relation to teaching and learning.
It is widely understood that competence is a deep structure which cannot be directly observed. We can observe performance, but this may or may not provide an accurate insight into competence, an issue which is discussed in some detail by Wood and Power (1987). They attempt to unravel some of the complexities and levels of competence, and suggest that its development as a coherent structure in a particular area may take a number of years; this view is also taken by Gonczi (1993) and Hager, Gonczi and Athanasou (1994). For this reason alone performances may be inconsistent, even assuming that the tasks which have been chosen to elicit the competence are entirely appropriate.
Gonczi (1994) describes three conceptions of the nature of the competence which is inferred. The first of these conceptions is task-based where competence is seen in terms of the satisfactory completion of a large number of discrete, small-scale tasks but with no exploration of the connections between them. The second concentrates on the general attributes which are required of a practitioner, dealing with some underlying domains such as knowledge or critical-thinking ability. The third links the two previous conceptions by setting the performance of particular tasks into a context of general attributes. We assume that the competency based system as implemented in the Australian vocational education and training (VET) system is concerned with this third conception; this appears to be the only way in which competency based assessment can remain anchored to the completion of vocational or occupation-specific tasks whilst meeting criticisms of the 'atomisation ' of learning and performance.
This is especially important in the context of the particular issues with which we are concerned in this paper. Gonczi (1993) suggests that, in relation to entry into educational institutions, assessment within vocation education and training should relate to the curriculum aims and not solely to the occupation-specific competency standards. Vocational courses, moreover, should incorporate ways of developing and assessing generic competencies, perhaps in occupation-specific contexts.
Such an approach would also interact directly with the mastery of processes which underpin competencies. Although it is frequently the case that the outcomes statements in a competency based scheme cannot be met without due attention to process, it is important that the generalisable nature of the processes is not lost in learning which is directed towards doing specific tasks. If this happens we narrow the focus of eduction to those things most readily observed (McGaw, 1993).
It is also a desirable feature of the third of Gonczi's three conceptions that it readily supports the collection of evidence which supplements that from the performance of specified tasks. If this occurs we can be more certain about inferring underlying competence. The issue has arisen particularly sharply in relation to what is referred to in the British National Vocational Qualifications as underpinning knowledge. This is described in the unit specifications for the award, but it is not necessarily made explicit through normal occupational activities (or would require an unacceptably large number of parallel activities to be undertaken in order to demonstrate all aspects of required knowledge). Supplementary evidence, collected from oral questioning, written tests, written assignments and the like, has therefore become commonplace, and is often complemented by evidence from prior achievements (Black & Wolf, 1990). There is no reason why such supplementary evidence should be confined to matters of knowledge.
Under a competency based system, assessors make judgements, which are based on evidence gathered from performances, about whether or not an individual meets particular criteria. It therefore relates to criterion referencing which, when operated at its simplest level, will lead to outcomes of pass versus not pass or can do versus cannot do; here we would be dealing with a concept of mastery. Under such narrow conditions outcomes may not readily be described in terms of grades or degrees of mastery (Nuttall, 1984; Peddie, 1992a), and as such do not lend themselves to the differentiation needed in a selection process in a competitive environment; this is the dilemma at the heart of the current discussion.
However, if we are actually dealing with more general attributes, and can accept a more continuous view of competence, we should be able to organise assessment so as to observe degrees of performance, and would have much more scope for differentiated outcomes from the assessment. Power (1986), writing about Year 12 students in Australia, comments:
In none of the domains at Year 12 level is competence an all-or-nothing affair. In each case, competence is continuous, acquired gradually rather than by crossing a threshold. There are no naturally occurring discontinuities in knowledge and skill which enable us to draw a pass-not pass line and divide papers into A-B-C-D-E-F.Moreover, we may also be able to determine attributes which characterise performance which goes beyond that required simply to pass, and which would be recognised as meriting greater recognition. In doing this we have moved beyond the elementary concept of mastery.
Whilst there is no doubt that some use of grading within competency based systems would help selectors and others, it would also satisfy the legitimate aspirations of students. In the course of providing a comprehensive discussion of grading within a competency based system, Byrne (1993) (in two separate quotations) makes the following points.
An almost universal quest in education is to maximise each student's performance in relation to his or her abilities.In attempts to address these issues, increased use has been made, by those involved in exploring greater differentiation within competency based assessment systems, of the term standards referenced. Peddie (1992a), in his work for the New Zealand Qualifications Authority, sees standards-based assessment as including a wide variety of assessment types of which the two main ones are competency based and achievement based assessment. He defines standards based assessment as occurring when the measurement or outcome is assessed against some description or level of achievement known as a standard; the nature of the description will vary from something that is very task-specific to some very general grade description, of the type used in some public examinations, where a diversity of performances can lead to the same grade outcome.A good criterion referenced grading system provides incentives for students to master the basics as well as go beyond basics to pursue excellence and creativity.
Withers and Batten (1990) on the other hand, in their continuum of assessment types, use the terms normative, standards referenced, criterion referenced and descriptive, and make a key distinction between the two areas of comparative and non-comparative assessment. In their descriptions, standards referenced assessment clearly falls on the comparative side, so that grading of some sort is easily allowable. Criterion-referencing, in their analysis, can fall within both the comparative and non-comparative sections. In the comparative area, student performance is generalised and converted into grades; in the non-comparative area, it is reported directly without generalised grades through descriptive statements which prohibit direct comparisons.
What is clear is that competency based assessment must be based upon clearly specified and publicly stated outcomes, all of which have to be assessed. Madaus (1992) is clear that assessments must
... be geared into well defined and articulated curricula which need to precede the assessments - not arise out of them ...and Clarke (1993) points out that the challenge is to
... communicate assessment information in ways that adequately reflect the richness and complexity of the performances which our contemporary assessment tasks now require ...Competency based assessment must also involve the selective use of a wide range of evidence, generated through a wide range of learning activities. Moreover, any reporting and assessment arrangements must be clearly related to those learning outcomes and reflect this range of evidence. In so doing they will need also to take account of three issues, all of which are necessary but not sufficient conditions for the supply of information to selectors through competency based assessment, so that the information will be of value to them, and can fairly, reliably and adequately differentiate between individuals. These are
There are some important distinctions to be made within this discussion. Principally, they are between the legitimacy of grading which, for example, recognises a simple differentiation into, say, pass, merit and distinction grades (as in the General National Vocational Qualification in the UK) and the suitability of some system which results in partitioning of candidates into many more categories. While there are some strong arguments in favour of recognising particularly meritorious performance with some suitable grading category, it is more difficult to argue for a multi-grade or mark scale which would enable individuals to be placed on at least an ordinal scale that is, a scale upon which individuals are placed in rank order of merit. To do this we would need to be able to specify many levels of competence and there is evidence (Murphy, 1986; Cresswell, 1987; Gipps, 1992; Wolf, 1993; Wolf, Burgess, Stott & Veasey, 1994) that reliability of assessment is compromised because
In passing, Byrne (1993) provides a useful summary of what she calls the 'pros and cons of grading'. Although she is addressing employers' needs in selection and in the conduct of training the list is a useful summary for our purposes, and points to some of the backwash effects which grading might have within the vocational education and training (VET) system.
The alternative would be an algorithm which would place VET candidates onto a parallel scale, on their own, and for a quota system to operate so that selection was carried out from this and from the other scale(s) on an equitable basis. In order for this to be seen to be just there would need to be points of alignment for the parallel scales so that the cut-off between selection and non-selection was reckoned to be at an equivalent standard on each. However, it is possible that there is little difference in principle between the amalgamation of two scales and their equating in this fashion, since the amalgamation would, by its very nature, need a scale-equating algorithm.
If we are to use either parallel or joint scales we shall need information which generates a rank order which distinguishes candidates sufficiently for selection cut-off points to be decided upon. The selection purpose will not have been served adequately if we have, for example, only three or four scale points, with a large number of candidates grouped at each point. It will be necessary to adopt a strategy which moves assessment outcomes beyond the point where a candidate gains either a pass or not pass outcome on the qualification as a whole. We can regard this as a two-grade scheme in which candidates would either be seen to have satisfied the competency requirements (passed) or not to have satisfied at least one of them (not passed).[1]
In practice the decisions about passing and not passing will usually be made on a unit-by-unit basis. Units can vary in scope; the smallest useful unit in the context of the current discussion is a single learning outcome or performance criterion which can be used as a basis for the judgement of evidence produced by a student. In practice units are usually 'a coherent and explicit set of outcomes ' (Further Education Unit, 1992) organised around a coherent structure for learning (which is sometimes described as a module of learning). When the student achieves the specified outcomes, he or she gains the credits which are explicitly attached to the unit. Credit, which is the currency of the system, may be at various levels. The pass on the whole qualification is an accumulation of credit, sometimes according to rules of combination of acceptable units.
If we are to be able to create a suitable ordinal scale for the whole qualification we may be able to do so from some accumulation of a large number of individual pass-not pass decisions made at the unit level. On complex units, covering large numbers of performance criteria, it may be possible to introduce some grades such as not pass, pass, merit and distinction. This does not violate the principles of the credit model described above (Further Education Unit, 1993) but may limit opportunities for credit transfer into and out of the award (Wilson, 1993). Further, we may wish to treat units in a variety of ways, because they do not all have the same status. Some combination of these approaches is also possible.
The alternative is that we look outside the qualification either completely or partly. It is possible that information could be generated which would stand instead of the pass-not pass information on the units or on the qualification as a whole and which could be used in place of the assessment outcome or in combination with it. This information could come from some further assessment, directly from a portfolio of evidence or from a record of achievement. If such alternative information sources were to be available, they would need to bear some close relationship to the qualification itself. In some cases they could form the basis of an ordinal scale or could contribute to one generated from the competency based assessment; otherwise they would be likely to involve a completely different process of judgment for selection from that used for the ordinal scale. It is also possible that the process would then be unique to TAFE candidates. We must therefore consider not only the suitability of the information but also the consequent complexity and acceptability of the selection processes which would have to be used.
At this stage it is helpful to begin to set down the possible approaches; this is done in table 1. Here it is evident that the available strategies are not entirely distinctive, and that they merge into one another in various ways. We discuss each separately in later sections of this paper, attempting to describe a way of working in each case as well as the advantages and disadvantages of adopting the approach. The emphasis will always be on selection but we wish to include, at each stage, some consideration of the effects of the chosen process on the competency based assessment for the TAFE Diploma.
Type of Approach | Assessment Possibilities |
Using sources of information from outside the learning outcomes | Using a portfolio or record of achievement |
Use of special assessments operated separately from the qualification assessment processes | |
Generating grades from ungraded units | Using combinations of vocational units |
Using core skill units | |
Adding in unit tests or assignments | |
Grading the units | Applying overall criteria to each unit |
Using unit-specific criteria | |
Adding unit tests | |
Grading the whole qualification | Applying overall grading criteria related to the competency base but applied overall |
Grading on consistency or speed of performance or on overall assignment or project |
primary evidence for assessment ... derived from projects and assignments ... to show that [students] have covered all the outcomes from each unit. Besides fulfilling an important role in providing evidence for assessment and grading, this allows people ... to examine the quality of student's work (NCVQ, 1993).A record of achievement may provide a summary of the evidence in a portfolio, in the form of an organised set of descriptions or a profile of the student's achievements. A summary report could also be generated, perhaps as an interpretation of what is in the portfolio, or to stand in its place. This would need to come from the teacher, perhaps in dialogue with the student. There are several attractions in seeking to select students using either the portfolio or a summary of the achievements represented by the evidence in it.
An alternative (which is actually a sort of test) would be an extended assignment or project, to be undertaken for selection purposes. This could cover a wide range of knowledge and skills relevant to the selection purpose, and be assessed on the basis of a viva, written report, or by observation.
The attractions of a special selection device of this kind are the
An alternative could be a set of requirements from selectors which would demand particular units to be included; thus, selection for degree courses in engineering might favour units which dealt with relevant mathematical or scientific competencies. The advantages of this approach are that
These devices, which are part of the structure of GNVQ, offer some possibilities for grading. In their simplest form they can be hurdles: gaining a pass on the test or assignment is a pre-condition to gaining a pass on the unit (which is based on a much wider spectrum of evidence). Used in another way the pass mark on the unit test or assignment may form the basis for a grading scale. Once again the advantages derive from the relative simplicity of the process and its integration within the structure of the qualification. The disadvantages are that
The challenge lies in generating the basis for the grading. There is a limited range of possibilities.
The choice of criteria will be important. They could relate to the 'quality ' of the work which has been undertaken: it is to be of a standard which goes beyond that required for a pass. The qualities could be spelled out in terms of some underlying attributes such as the quality of communication. On the other hand they could relate to some skills which are held in high regard, such as planning, organising information, analysis or evaluation. If this is to be the basis then it may be necessary to provide sets of statements which contextualise the generic skills to the content of each unit.
The considerable advantage of either or both of these approaches is that the generic basis for grading each unit is also the basis for defining the meaning of the grades on the overall qualification. Furthermore,
The advantage would be an increase in the number of scale points available but there would be a more complex structure for unit grading which would be even more difficult for users to understand. It is likely that the marginal increase in apparent discrimination is offset by the problems of making the system transparent to users.
These attributes have to be present across a substantial portion of the evidence in a candidate's portfolio, irrespective of the units from which they came, and the grading judgements can only be made when the portfolio is complete, or nearly so. The units themselves can (and do) vary in content and approach, and each candidate need only satisfy some portions of each grading criterion in each unit. It is therefore not possible to say that a particular piece of work has earned a 'merit ' or 'distinction ' but it is possible to say that each has some specific attributes which meet some specific aspects of one or other set of grading criteria.
Other approaches of a similar type have been proposed. Peddie (1993a) suggests that creativity or originality could be used as a basis for developing criteria, and it is not difficult to imagine a range of constructs which could be used in this way. These could include criteria which are based on the scope of underpinning knowledge and understanding displayed in the portfolio, or criteria which deal with aspects of the 'quality ' of work which has been completed. In all of these cases the criteria are, necessarily, couched in context-free language. As elsewhere in this discussion the criteria will need to reflect competencies which are appropriate to university courses.
This approach has the advantages that the unit content and competency statements are not compromised, and high grades are not awarded simply on the basis of quantity of work or on some unspecified concepts of quality applied across the portfolio. A number of difficulties have, however, emerged in the UK, resulting in suggestions that grading should operate on a unit-by-unit basis, and be more closely related to concepts of 'quality of work '. Amongst these difficulties are that:
It is difficult to see that criteria would be used for consistency which are not, in principle, similar to the application of overall grading criteria described above. The particular difficulty with consistency is that it cannot be judged prior to the completion of the qualification.More likely to be useful for selectors is grading which is separate from the overall statement of pass-not pass, but which is based on the completion of an integrating project or assignment by those who have passed. This project or assignment would reflect competencies gained through the course as a whole and there would need to be a set of criteria for each grade. Byrne (1993) reports that this procedure is being introduced by The Scottish Vocational Education Council (SCOTVEC) and discusses its advantages and disadvantages. On the positive side these are that:There can be no justification for using speed of completion when a qualification is not time-limited. This would be especially true where learners were part-time or were adult-returners.
Obviously, factors of expense, time, staff training and general manageability will loom large in the discussions which precede action, but we would consider it vitally important to ensure that the following two points are borne carefully in mind when coming to decisions:
Byrne, J. (1993). The option of graded assessment in competency-based education and training. Paper presented at the National Assessment Research Forum, Sydney, April 1993.
Clarke, D. J. (1993). The assessment agenda. Paper presented at the 5th European Conference for Research on Learning and Instruction, Aix en Provence, September 1993.
Cresswell, M. J. (1987). Describing examination performance: Grade criteria in public examinations. Education Studies, 13(3), 247-265.
Further Education Unit. (1992). A basis for credit? Developing a post-16 credit accumulation and transfer framework. London: Further Education Unit.
Further Education Unit. (1993). A basis for credit? Developing a post-16 credit accumulation and transfer framework: Feedback and developments. London: Further Education Unit.
Gipps, C. V. (1992). National Curriculum assessment: A research agenda. British Educational Research Journal, 18(3), 277-286.
Gonczi, A. (1993). Integrated approaches to competency based assessment. Paper presented at the National Assessment Research Forum, Sydney, April 1993.
Gonczi, A. (1994). Competency based assessment in the professions in Australia. Assessment in Education, 1(1), 27-44.
Hager, P., Gonczi, A. & Athanasou, J. (1994). General issues about assessment of competence. Assessment and Evaluation in Higher Education, 19(1), 3-16.
Madaus, G. F. (1992). A technological and historical consideration of equity issues associated with proposals to change the nation's testing policy. Paper prepared for the Symposium on Equity and Educational Testing and Assessment. Washington DC, March 1992.
McGaw, B. (1993). Competency-based assessment: Measurement issues. Paper presented at the National Assessment Research Forum, Sydney, April 1993.
Murphy, R. (1986). The emperor has no clothes: Grade-related criteria and the GCSE. In C. V. Gipps (Ed.) The GCSE: An uncommon exam. (Bedford Way Papers No. 29). London: University of London.
NCVQ. (1993). GNVQ information note. London: National Council for Vocational Qualifications.
Nuttall, D. L. (1984). Alternative assessment: Only connect ... (Report for the seminar: New Approaches to Assessment, 13 June 1984). London: Secondary Examinations Council.
Peddie, R. A. (1992a). Beyond the norm? An introduction to standards-based assessment. Wellington: New Zealand Qualifications Authority.
Peddie, R. A. (1992b). Standards of excellence: The award of merit in competency-based assessment. Wellington: New Zealand Qualifications Authority.
Peddie, R. A. (1993a). Standards, levels, grades and 'merit': A critical analysis. Paper presented at the National Assessment Research Forum, Sydney, April 1993.
Peddie, R. A. (1993b). Achieving excellence: A second report on merit in competence-based assessment. Wellington: New Zealand Qualifications Authority.
Pitman, J. (1993). The Queensland Core Skills Test: In profile and in profiles. Paper presented at the 19th Conference of the International Association of Educational Assessment, Mauritius, June 1993.
Power, C. (1986). Criterion based assessment, grading and reporting at Year 12 level. Australian Journal of Education, 30(3), 266-284.
Robertson, D. (1994). Choosing to change (The report of the HEQC CAT Development Project). London: Higher Education Quality Council.
Wilson, P. (1993). Developing a post-16 CAT framework: The technical specifications. In Discussing credit. London: Further Education Unit.
Withers, G. & Batten, M. (1990). Defining types of assessment. In B. Low & G. Withers (Eds), Developments in school and public assessment. Melbourne: Australian Council for Educational Research.
Wood, R. & Power, C. (1987). Aspects of the competence-performance distinction: Educational, psychological and measurement issues. Journal of Curriculum Studies, 19(5), 409-424.
Wolf, A., Burgess, R., Stott, H. & Veasey, J. (1994). GNVQ assessment review project: Final report. London: University of London Institute of Education.
Wolf, A. (1993). Assessment issues and problems in a criterion-based system. London: Further Education Unit.
Author details: John Wilmut is an educational consultant based at The Old Post Office, Bray Shop, Callingdon, Cornwall, England. After moving from engineering to science teaching, he specialised in educational assessment and examinations, first in the School of Education, University of Reading, and then as research officer at the Associated Examining Board. He also worked at the Open University and The University of Queensland. He has worked on the development and evaluation of the National Vocational Qualifications (NVQs) and the General National Vocational Qualifications (GNVQs) and on various assessment and examination projects. His work has included training and advisory programs for India, China, Malawi, Jordan, Namibia and South Africa and he has written widely on assessment issues.
Henry Macintosh is an educational consultant based at Brook Lawn Middleton Road, Camberley, Surrey, England. After teaching History in primary and secondary schools and at the Royal Military Acadmy, Sandhurst, he went into examination administration and became secretary to the Southern Regional Examinations Board in England. Since 1969 he has been the Advisor on Assessment and Accreditation for the Education Division of the Employment Department - recently merged with the Department for Education. He was treasurer and membership secretary of the International Association for Educational Assessment (IAEA) from 1984 to 1995 and has travelled extensively, visiting Australia on twelve occasions. He has written widely on assessment and curriculum. Please cite as: Wilmut, J. and Macintosh, H. G. (1997). Possible options for differentiation in competency based assessment. Queensland Journal of Educational Research, 13(3), 46-70. http://education.curtin.edu.au/iier/qjer/qjer13/wilmut.html |