Saturday, 24 June 2017

Plagiarism in NCEA Exemplars?

One of the biggest differences that I remember noticing as a first year at uni with the way things are done in NCEA is the absence of exemplars. For externals, these are omnipresent. Well, truthfully, they weren't for my cohort because very few of the standards we sat were old: I was one of the guinea pigs who made the realigned standards safe for everyone else. What we got, as far as I can tell, was either candidates whose answers were adapted from old standards or actual guinea pigs who were told to sit papers they weren't taught. I'm not sure, exactly.

Anyway, one of the really useful things that NZQA does is put up exemplars from the previous year's externals and annotate them so that current candidates better understand what it is they are to do. A philosophical "backronym" for this might be that exams aim to assess learning, but what they often will end up doing is just assessing who is able to put their answers together better. If you put up resources, everyone knows what exactly it is that an answer ought to resemble... it's a bit like being told that your parents have bought you a Camaro because you don't know the colour of the car (answer) but you know the shape of the model beforehand (in theory... I had to do a Google search, I'm not sure why Camaro popped into my head). Exemplars are also useful for understanding what it is that makes an answer as good as it is.

Strictly speaking, merely possessing an exemplar doesn't just give a candidate either of these advantages. After all one still needs to think about what is generic and specific about any given exemplar. At the most basic level: what is present because of the specific question and context and what features transcend these base necessities? Salience matters. One might go as far to argue that the university student is sufficiently experienced at doing this, that they need only the question to be able to figure it out.

On the other hand, it may be the case that possessing a bunch of exemplars is really handy because it turns out that the questions hardly ever change. Therefore, one can just memorise an exemplar and write that down: no learning required. Hell, one knows from English that it doesn't necessarily matter that the question is different: people try to shoe-horn in answers. Many (most?) people aren't like me... in year thirteen classical studies the "religion and ideology" question included "ideology" so I changed tack completely rather than trying to stick with my "religion"-centred preparation.* Teachers know about the shoehorn issue, which is why they always advise preparing for multiple different question typologies (e.g. in English you might prepare for character, symbol and event questions rather than being ready to attack every, i.e. any, question typology... this might be an interesting discussion in itself).

The trouble is that with NCEA history externals, there is only so much scope for changing the questions with the essay standards: they're very narrowly conceived. Observe the level one "essays" and level three essays in their first and most recent years of offering:

91005Describe the causes and consequences of an historical event
What were the causes of an historical event you have studied this year? How did this event affect people, or groups, in society? (2011)

Identify and describe the causes of your chosen historical event. What were the short-term and long-term consequences of the event for people and / or groups? (2016)

91006Describe how a significant historical event affected New Zealand society
Describe what happened in your chosen historical event.

Describe how TWO of the people OR groups in society that you identified on page 3 were affected by the historical event.

Explain why your chosen historical event was of significance to New Zealanders. In your answer, you could discuss aspects such as: • the importance of the event to people alive at the time • how deeply people’s lives were affected at the time • the extent to which the event continues to affect New Zealand society. (2011)

Describe what happened in your chosen historical event.

Describe a specific action taken by a person or group during your chosen historical event.
Describe the reaction / response of another person or group to the action you described in (a), and the reason(s) for the reaction / response

Explain why your chosen historical event was of significance to New Zealand society at the time, and / or how it continues to be significant. Support your answer with relevant evidence. (2016)

91438Analyse the causes and consequences of a significant historical event
Analyse the various causes of a significant historical event, and the consequences of that event on people’s lives. (2013)

Analyse the extent to which particular factors caused a significant historical event, and the different ways this event changed people’s lives over an extended period of time. (2016)

91439Analyse a significant historical trend and the force(s) that influenced it
Analyse the different forces that influenced a significant historical trend, and the extent to which this trend impacted on people's lives. (2013)

Analyse the important forces that impacted on a significant historical trend, and the extent to which change and continuity were reflected in people’s lives. (2016)

Okay, so the non-"causes and consequences" standards seem a bit more different, even giving themselves scope for the religion and ideology issue. That is, NZQA could keep candidates on their toes by mixing it up between questions just about, e.g. "reactions" or "responses" or "reactions and responses" even if what exactly the difference between reaction (physical) and response (mental) are arguable. It's even clearer cut in the trend essay because you could force more narrowly tailored responses on "change" or "continuity" or more generalised treatment of "change and continuity". But I think these examples generally back up the notion that the standards themselves have forced NZQA into a corner... they either don't offer exemplars or greatly facilitate plagiarism because the standards are too narrow for variety. On the other hand, the assessment report for the 2015 version of this last standard did have this to say:
 Some candidates struggled to respond to the question, preferring to write a response to a question from a previous exam. Candidates who approach an examination with a prepared response will always be at a disadvantage.
Now, given why this post exists, that remark is frankly hilarious. You see, I'm talking about this because apparently some candidates have, in fact, noticed plagiarism... the ultimate prepared response. That's right: not NZQA, not markers, not even teachers... pupils. At least, that's what the Herald is saying (NZQA have helpfully removed the exemplars in question). The thing is, this plagiarism happened in 2012... five years ago. I was still at school in 2012. In fact, I would have, probably, looked at both of these exemplars out of interest in the past (I got a merit in 2011 in this standard... and while NZQA did, without asking but I don't care, put two of my merit standards up as exemplars in 2012 they were the only standards I ever had turned into exemplars). The point is: this is a long time for no one to have noticed.

Working from the assumption that no-one is lying (because I cannot check), I imagine that the people who choose the exemplars are drawn from the marking committees. I similarly imagine that these committees undergo some kind of change in personnel from year to year and certainly would see a great many different scripts (and, surely, a great many shortlisted exemplar scripts). I also imagine that there is some kind of checklist like thing which is used to determine what makes a good exemplar (legible handwriting is apparently not one of them... as a friend noted, one of our pseudo-exemplars when we were doing Level Three history was indecipherable chicken scratchings). In this sense, it is quite obvious that something which is exemplar material in 2011 would be pulled up as exemplar material in 2012. Still, there should have been one person, at least, who was there to check that they weren't essentially reposting an old exemplar. And markers really ought to be familiar with the exemplar standards, i.e. the 2012 answer should have been given an N0 right at the start... well before my hypothesised committees got a chance to look at it.

Basically, the minister has every right to want an investigation into this matter. That's one reason why ministers exist: to hold bureaucrats to account, just like any other boss. On the other hand, NZQA five years ago is not the NZQA of today, so to find remarks like the below makes me recall the still widespread scepticism of NCEA (generally by people who know little of it... or those with Freudian obsessions with "employability"):
The authority, which did not spot mistakes in three maths exams last year, has posted almost identical papers on the 1981 Springbok tour as exemplars of scripts that earned "excellence" grades in the 2011 and 2012 history exams.
That's probably more prejudicial than valid contextualisation, in other words. Yet, honestly, it is rather more interesting to look at:
But NZ History Teachers Association treasurer Greg Burnard said memorising previous years' exemplars was "reasonably widespread across the country".
"Memorising an exemplar is not going to be punished, essentially," he said. "It's not seen as cheating, it's just seen as being well prepared."
Compare and contrast what I said in "Listen to the Axe Grind":
Sometimes an exam is just plain useless... for instance, they're prone to creating regurgitation and brain dumps, and they also can't test the ability to research. Exams are, inherently, restricted in what they can assess. Assessment should meet the purpose, not the other way around. 
And Burnard again later on:
"The way forward is to reward analysis rather than just regurgitation," he said.
This, I think, is the real issue raised by this episode. Exemplars are somewhat problematic concepts... even if we disregard the plagiarism potential. Think about the University of Auckland's Comlaw department's critique of model answers: there isn't necessarily one particular way of answering a question, but that is invariably the implication of an exemplar. In terms of the language I used earlier in this post, exemplars make one think that all cars look like Camaros but, of course, we know that thing from Breaking Bad exists and we recognise it as a car. To make the metaphor work a bit harder, the functionality of the Aztek and Camaro is the same: getting from A to B. With every question, there is some function that an answer needs to perform... i.e. actually answering the question. Sometimes this looks like a Camaro... and an answer of this flavour might be red or it might be blue (i.e. is articulated differently)... and sometimes this looks like an Aztek... again of varying different colours.

There isn't too much we can do about the Camaro-Aztek critique. Offering multiple different genera (I should have used a biological metaphor, this ad not withstanding**) doesn't help. If one had several different conceptualisations of a question, the one implies that these represent the set of all conceptualisations. That may be true or it may not be true. Either way, one doesn't provide a structure that pushes the candidate towards open-minded thinking about the question. In this sense, one wants to impart abstract lessons but invariably requires concrete assessments to check this... an inherently flawed task. And possibly the teaching is through a concrete paradigm too. And then one has to remember that content knowledge matters as well.*** I can't think of any other solution that doesn't kick the exemplar out the window entirely. Perhaps annotating the exemplar... but then publishing only the annotations?**** Maybe what I've done with my exam resources works... answers but without questions?

That latter notion is interesting. On one hand, I suggested that exemplars are good things because having everyone know what an answer looks like means everyone is on a level field... and therefore that we provide the best environment for analysis (although God only knows what that looks like in our car metaphor). Just offering an answer gives the candidate this sense of shape. Yet, how meaningful is that shape robbed of context? You and I know what a car is, but show Ugg the Caveman either an Aztek or a Camaro and Ugg would be mystified. Certainly, one would not be able to tell which facets of an answer were specific to the question and which represented examples of what the answer is doing. A bit like how Ugg might think a car's roof is integral to its function, whereas we know it's just a comfort measure. What this would mean in practice isn't clear to me... the ideal case is that one would go, "Oh, so that's the feel of answer of X quality"... assuming the quality of the analysis is what generates the feel (realistically, its rhetorical quality is what probably does that).

Now, it should also be said that regurgitating an answer to an unknown question is ballsy or really, really stupid so perhaps it would help in this respect... but as a lecturer once put it, "[academic dishonesty] usually goes hand in hand with stupidity" (the point being doing silly things, rather than being a dunce). And, again, I find myself unsure of how to interpret this point. My inclination is that it wouldn't really help... even if the idea is useful... and what of the problems in not knowing to expect strangeness? You know, the solution that everyone already talks about because it seems to work: keeping the candidates on their toes with question variety.

I think I've lost the plot a bit here... regurgitation is bad, exemplars definitely encourage it, but we have to balance that against the arbitrary nature of exams and hence the need to ground candidates. In this philosophical context, the plagiarism case should be used to raise awareness of this dilemma primarily rather than being a "bad thing happened, you should know about it" story. That's basically what I've tried to say... with a little deliberation on the difficulties in applying the standard means of balancing. Oh, and let us not forget the grade inflation angle... which you can read about here.

*On the other hand, the only standard whose number I remember (91098) had, as the link shows, radically different questions to what we were used to (and unlike subsequent years we had no clue that they'd try to be funny with the questions). In that case, I am pretty sure I ended up answering a conflict question through conflicts. Sadly, they never turned any of the 2012 scripts into exemplars... I was rather hoping that they'd turn mine into one (I was very pleased with my E).

**I'm also a "call a car a car" kinda guy... I don't give two hoots for cars. I will, however, watch Fast and Furious movies... even the early ones when they were still vaguely about cars.

***A point raised in Not Our Problem... a fictional narrative based in NZ's 1990s "healthcare" reforms (another disastrous reminder that there is a reason why small government societies of the "Western past" left certain aspects of society to government).

****This gives me an idea: I should type up my annotations of some History reading and post it here as a blog post.

No comments:

Post a Comment