Wednesday, 26 November 2014


I've talked about NCEA quite a bit and I know a lot of people do. However, a lot of people love to talk (specifically complain) about NCEA with only a vague understanding of how it works. I've decided to add to the wealth of material explaining NCEA with my own understanding of it. Just a quick note, while "the NCEA" is, strictly, more correct than NCEA no-one will take you seriously if you call it that. Stick with NCEA.

NCEA is a three-tiered standard-based qualification. The three tiers are NCEA Levels One, Two and Three; these are usually taken by Year Elevens, Twelves and Thirteens respectively. To pass NCEA Level One, a candidate (i.e. pupil) needs to gain 80 credits at, at least, NCEA Level One. The candidate also requires ten literacy and numeracy credits that are part of a variety of standards across many subjects: not just those in English and maths. The following two levels also require the ten literacy and the ten numeracy credits to have been met. However, Levels Two and Three can carry 20 credits from the preceding level over; thus, a candidate needs only 60 credits at Level Two or higher for Level Two and so on. Oh, and if you get 50 credits above achieved, you get at least a merit endorsement, Now, that's an overview: it needs breaking down further.

What are credits?

They are, simply, a way of counting how much something is worth. A can of baked beans is sold for $1, it's "worth" a dollar. A standard is passed and it's worth, usually, 4 credits. There is some allowance for the amount of effort that is required but this is flawed. For example, the Level Two statistics standard that I sat in 2012 was worth a miserable two credits. It was, in my view, at least three and it would've been better had our other internal, on graph theory, been worth three as well... rather than the 2/4 reality. The Level Three trig and simultaneous equations were both piss easy and more deserved of two credits than the Level Two stats standard that we did. Achievement with excellence in a four credit standard will yield four credits -- as will barely scraping an achieved.

What is a standard?

It's a thing to be assessed. A standard has a number and also an explanation. For example, one of the level three classical studies standards is: 91394 Analyse ideas and values of the classical world. The standard embodies the paper (i.e. the assessment) as well as the criteria that the candidate must meet to achieve (and which additional criteria are needed for merit and excellence). In this sense, standards are the heart of NCEA. They are what candidates do, what teachers mark and they are what performance is measured against. I'm a huge fan of standard based educational systems because they're transparent and the candidate (i.e. the pupil) knows that their work is at a certain level. Standards mean that a candidate's performance is theirs and doesn't reflect the wider intelligence of their cohort. For universities and employers and everyone really, it's totally useless knowing that Johnny is in the top 2% of his cohort. That doesn't actually say whether or not he knows what he's meant to. 99% of the cohort could be thick as bricks, or maybe they're super intelligent. If you see that Johnny achieved 91394 with merit then you know with absolute certainty how much he knows. Relative rankings have their place (for instance, deciding who gets a scholarship) but in the general purpose sense standards are far superior.

You get unit and achievement standards, and it is sufficient to know that unit = N or A and achievement = N, A, M or E.

What are Internals? Externals?

These are the two different ways of marking NCEA standards. Internals are marked throughout the year by the school. They are often written by teachers within the school or modified versions of more widely used standards. Externals are marked externally, i.e. by NCEA markers. They are usually sat in the National exams that happen in November (and run into early December?) but in some cases work done during the year in schools counts (Art and Graphics are good examples of this sort of subject). Externals are, as such, mostly exams with some portfolio work. Internals can be done in tests, written reports, PowerPoints, blogs (I guess), speeches, plays, and really any method of assessment that the teacher considers to be appropriate for the standard. I am given to understand standards are subject to moderation but, for the most part, National Moderation works on a sample basis.

What is Level One? Two? Three?

This is a quick answer... these describe the relative difficulty or level of the work being done. While a candidate may not always feel as if level one work was actually easier than level three work (who knows, they may have just worked harder), the levels represent broad categories of difficulty. This is more or less the experienced reality rather than the theory which I think is clear. There is a definite leap from level one to level two, but I don't recall noticing a jump except in terms of workload (even with a subject less) when going from two to three.

How many credits in a level?

This varies and it varies a lot. The number of credits that a candidate has available is dependent on both their subject choices and their school. It also depends on their teacher. For instance, some schools might offer only one internal for level three calculus and choose to focus on the externals. Other schools might choose to offer as many internals as there are (I'm not sure how many there are, we did two). As a general guideline I found that one could make a functional estimate of six standards per subject and four credits per standard, leaving an estimate of 24 credits a subject. A lot of courses/subjects will have at least one standard worth more than five and it's a rough approximation so you need to check how many credits a candidate has available by counting them. It's worth recalling that a candidate with 124 credits gets the same qualification as a candidate with 81 credits. A lot of people consider this to be an issue with NCEA but it's a minor quibble as NCEA recognises achievement in other ways.

How does NCEA recognise achievement? It's only got four marks...

If you can pass, you get achieved. If you do well, you get merit. If you do very well, you get excellence. This isn't quite how it works (i.e. an excellence means additional criteria as well extended achieved level criteria are met) but it's how NCEA is experienced by candidates. To many people, not being able to tell if Johnny's excellence is better than Mary's is a problem but recall what I said earlier about relative rankings... they don't matter so much. However, NCEA does reward consistent performance and I think it rewards it well. I'll now explain those methods of reward.

Certificate Endorsement

This is probably the main form of reward and it relates to the overall certificate that one achieves. If, for instance, you get 50 excellence credits and achieve NCEA Level One you will be awarded "NCEA Level One with Excellence". In practice this means one needs to get at least a third of their available credits at excellence, which is not that hard to do if one can work at a very high standard for the entire year. Depending on subject choices, it is unreasonable to expect to be able to pass with excellence before externals start but it is entirely possible for some people to do so. Frankly, I agree with those who say that NCEA just lets people slack off before the exams because of this in that 50 is too low, I think it should be raised to 60 credits given that most candidates will always sit at least 120 credits (i.e. 5*24). This will mean basically everyone will not pass with excellence before the external exams. Achieving fifty merit and excellence, or just merit, credits leads to certificate endorsement with merit. Basically, certificate endorsement just means, "This candidate was able to maintain a high standard throughout the year". (That being said, one could bomb all internals and be the bomb in externals and still get a certificate endorsement.)

Course Endorsement

Three credits from each of the internal and external standards, and at least 14 credits altogether obtained in a single year within a course leads to a course endorsement. These aren't nationally comparable because different schools do choose to use different standards and, therefore, offer differing numbers of credits. As such, Johnny's merit course endorsement in Year Eleven English may be less impressive than Mary's because Mary's school's Y11 English offers 20 compared to 28 credits. That's another made up Johnny/Mary example but it shows the point.

What about Scholarship? Isn't that part of NCEA?

Scholarship and NCEA are separate but Scholarship is administered by NZQA and functions pretty much like NCEA does. The candidate turns up to the exam room, opens the plastic wrapping after a life and death struggle and writes stuff to answer the standard. Scholarship is a higher level examination with variable passing scores and all passing does is get the candidate money. In theory, 3% of the NCEA cohort in a subject will achieve Scholarship. This doesn't always happen and low numbers skew the numbers a lot but it's good enough. A lot of CIE and, presumably, IB pupils also have a go at Schol (as is the shorthand) and they can do well. However, it is harder for them as Schol does use the same base material as NCEA. Some subjects will be more affected by this than others... History is one of those that wouldn't be. It's also worth recalling that in Schol the markers look for somewhat different things as well (for instance, writing that has flair). There have recently been some (to my mind grossly unfair/inequitable) changes to how candidates enter Schol and payment is required for all of the standards entered. There are two kinds of pass: Scholarship and Outstanding. There is also a single "Best in Subject" award. Schol is exactly the sort of thing where relative performance is okay.

Is University Entrance a form of Reward?

Sort of. UE is an award that NZQA gives to Level Three achievers who have met some additional criteria. Again, it's not really a part of NCEA but in this case it is exclusively based on NCEA results. To my mind, this is a far superior system to having some additional testing to go to uni. However, UE is a minimum standard. This means that if you've got UE a university doesn't need to accept you but they do know that you've theoretically got the knowledge base to be able to complete a university degree. Prospective university students have to be aware of the additional criteria their desired unis enforce. The current requirements for UE are:

  • NCEA Level Three
  • At least 14 Credits in each of three subjects (i.e. 42 all up)
  • 5 Credits in Reading and 5 Credits in Writing at Level Two or above
  • 10 Numeracy Credits at NCEA Level One or above

What are Literacy and Numeracy Credits?

NCEA recognises that literacy is required to achieve in many standards, not just those in English. Something similar can be said about maths. As a consequence, NCEA publishes lists of standards that meet literacy and numeracy requirements.

What is Grade Score Marking?

As of 2013, this is how all externals are marked. Any question is marked out of 8 with N0, N1, N2, A3, A4, M5, M6, E7 and E8 being the possible results. For standards that have multiple questions (i.e. pretty much all non-essay based standards) NCEA publishes mark ranges that correspond to the NAME marks. A GSM of 32/32 and 29/32 with a GSM of 28-32 achieving excellence are, on paper, worth exactly the same thing. Perfect scores for papers with three or more questions were, in my experience as an NCEA candidate, reasonably rare. The best I ever did was 31/32 for Genetic Variation in Year Eleven... a standard that many of my friends managed 32/32. The message is, have an awareness of how this works and use it to help provide motivation but it doesn't matter too much.

Hopefully that's a clear explanation of both NCEA as a general system and also some specific aspects of NCEA with some commentary provided. I know for a fact, though, that there are clearer explanations out there. NZQA and schools put a lot of effort into making sure pupils and their parents understand the system. In my experience, pupils get it very quickly and understand that NCEA looks at the long term performance, not just a month's worth of cramming... although for most pupils they'll probably have around 50% of their credits available in externals (if they do more conventional subjects).

Thursday, 6 November 2014

Listen to the Axe Grind

Ah, the Listener. It's a pretty good magazine but when it comes to NCEA it has an axe to grind and neither rhyme nor reason is apparently going to fix that. Random comments on some unread blog aren't either but we are here so... I'll follow the standard pattern, i.e. respond paragraphy by paragraph.

The Government has voted confidence in Hekia Parata by retaining her as Education Minister. Let’s hope she can now earn the confidence of parents, students and teachers by taking a more transparent, evidence-based approach to education policies.

Hekia Parata is a failure. Everyone with even the slightest interest in education (in NZ) knows this. John Key knows this, hence her demotion. However, John Key is also aware that given National's ideas (and those of its partners) putting a competent minister into the portfolio will simply tarnish them. It's worth recalling that Parata used to be seen as an up and coming female MP. Now she's literally a joke in school plays. But enough about her, the problem with this paragraph is that it already sets up the farce that is the article. Firstly, notice that I describe Parata as having been demoted. This is a very important point if you're, like the Listener, talking about the Government's confidence in her: my mentioning it isn't a mere reflection of my dislikes of National or her (the extended tirade ridiculing her, though, is; I'm fickle). Secondly, the terms that the Listener uses to describe its vision for the future of education in New Zealand are both out of place. The former because it is hard to imagine a more transparent system than NCEA (we'll elaborate on this) and NZQA's approach to it. The second because of how the editorial goes.

Britain, acknowledging the limits of the all-must-win-prizes philosophy, is scrapping resits, limiting internally assessed work and encouraging students to study tougher, more traditional subjects.

Okay, so how similar are Britain's systems and ours? I don't know this off the top of my head. I guess it's not really very similar to that in Harry Potter (which I do know off the top of my head, hmm) but it's close enough. Like NCEA there are multiple levels. You've GCSEs and you've got A levels. Unlike NCEA's levels which correspond, essentially, with the final years of school there is a gap between them. Looking at this website, pupils doing GCSEs do far more subjects than is the norm in my experience (for instance, at my school we had six subjects... they do eight to ten). A Levels are, then, a two year course following that from which university entrance is determined; as far as I can tell, there is no overall minimum standard for UE as is the case here, everything is determined by the unis themselves (as is the practical reality here). Okay, so what sort of assessment actually happens? To be honest, not having any familiarity with British education it's actually quite hard to answer this question. As it's late I'm going with, there was, previously, too much coursework (i.e. internals) for the conservative Michael Gove (about as well liked as Parata, I believe) so they've started to switch back, for GCSEs, at the least, to a pure exam format. Firstly, that's daft. Secondly, I have no problems sitting exams, in fact I'd go as far to say a lot of them are enjoyable.

Education Secretary Nicky Morgan says the ever-higher grades and pass rates Britain has been trumpeting were not matched by improvement in actual educational standards. On the contrary. As she said in the Times, “England’s performance in international studies stagnated while other countries’ surged ahead, and employers and universities cried out that young people weren’t leaving school with the knowledge or skills they needed. This wasn’t the fault of hard-working teachers, but of a system that prized all the wrong outcomes. And it was the most disadvantaged children who suffered the most, as schools were encouraged to push these young people towards poor-quality qualifications …”

I'm going to assume this is PISA. As a reminder, PISA is fifteen year olds. This means people studying towards GCSEs. At least PISA mostly covers them. To respond to this quotation you need to know more about education in the UK than I do, I'm inclined to say that the Listener is aware of the extreme decentralisation (at least compared to NZ) that complicates assessing a fair view of this (certainly, it's beyond the 12:30am brain).

And here? Our Education Review Office (ERO) has already warned that some schools are shunting Maori and Pasifika students towards easy-to-pass subjects, with low expectations of their abilities. The ERO flagged this as one of four “challenges facing all secondary schools”.
And this is a problem with NCEA, how? Come on, Listener, answer that question. Oh, they can't without explaining that the approaches of such schools, while seriously flawed, aren't issues with NCEA. If we were all doing CIE or IB (the latter of which seems like a foreign version of NCEA with some forced cultural activities thrown in; not that there's anything wrong with the foreign bit) this would still happen. These schools, aware that the likes of North and South and Metro want to create league tables, would still find pathways that flatter themselves. I definitely think that such schools are failing their pupils and violating their duties as schools, but it is not a problem with NCEA or NZQA. The subjects in question are left ill-defined so we cannot even judge what they are. Maybe the ERO didn't do that, so let's look...

However, it is also clear that some schools are seeing vocational programmes mainly as a way to increase qualifications for Maori and Pacific students, particularly for the boys. While many students experience the benefits of these vocational courses, very few schools were developing academic courses specifically to increase the numbers of Maori and Pacific students who are able to enter university. 
While many Maori and Pacific students may succeed in vocational contexts, and thereby achieve NCEA Level 2, the question remains – how many Maori and Pacific students may also have thrived in more academic programmes that responded to their interests, strengths and aspirations? Schools need to raise the expectations for some of these students by ensuring that their curriculum and systems are enabling Maori and Pacific students to achieve to their potential.

I think this is what the Listener's talking about (found here), under the heading, "Vocational courses and ethnicity. The ERO says that schools are shunting pupils off into vocational (which are entirely worthwhile) pathways without dedicating enough time with such pupils in academic pathways (i.e. those that lead to university). This isn't quite what the Listener has to say, but it's close enough (there is a huge difference between easy-to-pass and vocational, for instance). Basically, the ERO is saying that schools need to engage these pupils in school in ways that aren't trade related. That's a difficult issue to solve and I'm not going to pretend I know the answer. However, I'm going to say that forcing people to do the bull that passes for NCEA English probably isn't it. Anyway, back to the Listener.

It’s time to acknowledge that Britain’s missteps are being replicated here. Some evidence, in brief:
• The Government is fixated on raising pass rates and has made pulling off an ambitious rise at Level 2 one of its targets.
• Our performance in international benchmarking tests such as PISA has “stagnated or declined”, while our NCEA results have continued to climb.
On the first point, targets are good... how you achieve them is the issue. On the second point, I must stress that we're going backwards in PISA compared to ourselves. Is this a flaw with NCEA? No. As I have said elsewhere, PISA covers pupils who are either approaching the start of their NCEA journey or are right at the very start of it. PISA is, in other words, indicative of deeper failings. And, in fact, given National's been around for six years now, the backwards performance relative to ourselves probably has a lot more to with late-Clark era Labour and Key-era National not doing enough with early childhood and primary education than anything else. The international side of PISA isn't worth looking at when the big problem is our historical data. If it helps, Finland has similar although less extreme trends in PISA to us.

NCEA-based University Entrance certificates no longer do what they say on the box. NZQA is now making UE harder to get, but the University of Auckland has already substituted its own basic entry requirements. An internal memo said UE was not a sufficient entry threshold because it let in too many students despite “ongoing weaknesses” in literacy. The university’s head of admissions said “we have an obligation to students to make sure that they’re not being set up to fail”.

I have never, not for one second, ever thought that the UE standard that NZQA sets to be anything other than the very minimum level. I'm not sure, to be fair, whether or not this is the point of the UE award. It's worth pointing out that Auckland uses a rank score method which is exactly the same for prospective students whether they do CIE, NCEA or IB... it's just a different formula for each qualification.

Speaking of ongoing weaknesses in literacy, though... NCEA English is a complete joke. Unfamiliar texts and speeches are probably the only standards that are worth their salt. I'd chuck the formal and creative writing ones in there as well but, and this is a big but, too many teachers will let pupils adapt essays written for other purposes to the formal aspect and, simultaneously, allow an absolute minimum of creative writing to count. As a consequence of this, I can recall writing only one thing specifically for the level two standard. This is literally the only area where unis have an advantage over NCEA: not allowing the same work to count for multiple different assessments. Unfamiliar Texts is, in my eyes, hard. It is, also, far superior to the "regurgitate whatever your teacher told you" essays of the external English exams. Find another way of teaching understanding of set texts that avoids that little quibble. NCEA is an assessment system, but the way it is used to assess English is nuts (hell, I reckon less than half my year twelve class actually wrote let alone spoke speeches) and I think a large part of that is because we, as a country, need to seriously re-consider what we want English, the subject, to be about. I know what I want: less regurgitation and more pupil created stuff.

Having surveyed 15 institutions, the Tertiary Education Commission (TEC) had profound and widespread concerns about the way NCEA prepares engineering students. Even among those who attained the requisite standards at school, most weren’t ready for tertiary maths and physics. Maths skills in particular were poor across the board.

Tertiary maths, as in pure maths, is radically different to NCEA maths. It is, also, much harder. However, a PhD student that I know has described NCEA maths as maths for engineers so who knows. I believe this report also included the comments of Dale Carnegie, whose views on NCEA and engineering I have considered earlier. Returning to maths, though, there is room to reconsider here. For one, let's get set notation in schools and also matrices.

 Like Britain, we have mistaken an easy tick-box system for true learning, with Parata appearing almost wilfully blind to the devaluation of the concept of “achievement”. Yet even she can no longer ignore the evidence that although NCEA is generally robust, the huge pressure she has put on schools to achieve pass rates come what may is distorting and devaluing its outcomes, with teachers feeling they must herd the seemingly less able students towards easy subjects rather than challenge them.

NCEA doesn't scale. NCEA doesn't use multiple choice. NCEA makes what it requires of pupils absolutely clear. In fact, NCEA makes its criteria very clear. So clear, in fact, if the criteria seems just as difficult from year to year you'd be able to tell if grade inflation is actually happening. I don't have the resources to do that, the Listener does: time for it to put its money where its mouth is I think.

No one wants students shunted back to the guinea-pig maze of the early NCEA, or to old-school systems like Bursary and School Certificate. A few simple changes could restore the system’s fidelity: limiting the proportion of internally marked credits; requiring students to continue foundation subjects such as maths and English to higher levels; and restoring transparency so parents can again see the illuminating results data from the NZQA, which for the past year have been withheld.

That transparency thing is not quite true. The information is mostly still there: it's just not very accessible. NZQA tries to dodge the point but restoring how the thing was is still perfectly viable: if it would just remove the ability to break school level data down further (i.e. into ethnicity and gender). Its privacy concerns were well founded, but its response has been extreme. Limiting the proportion of internally marked credits is nuts. Sometimes an exam is just plain useless... for instance, they're prone to creating regurgitation and brain dumps, and they also can't test the ability to research. Exams are, inherently, restricted in what they can assess. Assessment should meet the purpose, not the other way around. Maths, from a practical perspective, tends to run out of immediately obvious applications at year eleven. From a university pure maths perspective, it has some way to go (not that I think lectures are a good way of teaching maths). English... has problems.

But most importantly, the Government simply needs to back off, to stop compelling schools to achieve ever-vaulting pass rates, irrespective of the true merit. We need to focus, as the UK has done, on whether the qualifications actually mean anything.
It’s absurd that while Parata’s ministerial colleagues are lauding our science and technology future, the TEC is revealing that schools are steering many students away from the hard-to-pass maths and science studies vital to that future. In the longer term, Parata is setting her own Government up to fail in its key objectives.
Do our qualifications mean things? Well, yes. If you, as an employer, cannot understand what an NCEA student can do, you're an idiot. Every standard is online, type in the number and read its criteria. That's the beauty of a criterion based system. Is grade inflation happening? Who knows? Again, with criteria, you can answer this quite easily.

Is NCEA maths easy to pass? Well, you don't pass the subject but the easiest standards I ever did came from level three calculus. Is NCEA science easy to pass? Well, year eleven isn't (particularly genetic variation). I stopped taking science there (I met the dismal one, and it was more interesting), but a lot of my friends (in fact, nearly all) didn't and they would tell stories of difficult assessments. So, who knows? The Listener definitely doesn't. It has a very clear agenda (NCEA is teh evulz !1!1) and it has shown time and time again that it won't honestly balance what it writes.