Saturday, 7 December 2013

Victoria or Victorian?

The PISA results have sparked a number of articles in the Herald and one of them was this one. Most of the articles and most of the parts within articles have been fine and totally agreeable. However, the views of Professor Dale Carnegie of Victoria University deserve to be ridiculed. The university which he is part of lends its name to the title. In full it should read something like, "Where does he belong?" Hopefully people get the point. I would have put these views in a comment to the article on the Herald but that was not possible. In fact, the entire reason why this blog exists is to explain exactly what I think is wrong with Carnegie's comments. I'd prefer it if these criticisms/comments were likely to reach him but that's the problem with the inability to comment on the Herald article.
The New Zealand Government has set the very laudable goal of increasing the number of engineering graduates that the country produces. International examples clearly illustrate that such an approach does lead to significant future economic growth. In order to achieve these increased engineering graduates, a reality is that we need to increase the number of students leaving secondary school who are well-prepared in mathematics and physical sciences. There is no problem with our very top students, those who are motivated to strive for the merit and excellence grades. But these students cannot fill all the additional places in engineering. Instead we have to up-skill and motivate the next tier of students.
This is his lead paragraph and it's fine. I include this here purely so that his entire view is represented rather than just the snippets I disagree with. Yes, those students who are currently not achieving merits and excellences do need to be bumped up. Obviously not through just handing out the marks but by reaching a level where they can actually achieve with merit and/or excellence. The wonderful thing about NCEA is that it is, theoretically, great for this as the standards are (realignment aside) unchanging and, for the most part, skill relative to the rest of the cohort is irrelevant (if everyone was capable of meeting the standard, everyone could pass).
NCEA has been successful in many areas, especially keeping students in school longer. However it is very clear from our research that for very many students NCEA is not effectively preparing them for engineering study at university.
This is both a broad and specific statement. In terms of university generally NCEA is seen as doing quite well. That hit the news again in relation to the recent uni report critique that the Listener covered. Carnegie, however, talks in the specific case of engineering. A lot of my friends are interested in this area of study and took, as a result, physics and calculus in year thirteen. Because we're in Auckland the main focus for entry, naturally, was Auckland Uni. They specify those subjects (other sciences or statistics simply won't do and would have to be made up) and in terms of overall achievement the rank score is up there at 250. For reference that's 78% of the maximum rank score (for CIE the equivalent is 74% and IB it's 73%)* and the lowest number of excellences is 10 with the remaining 70 credits all at merit. This is far higher a standard than either the outgoing or incoming NCEA university entrance standards. So, judging by what Auckland thinks Engineering requires a high academic standard and, presumably, their entry requirements reflect what they feel is the minimum level to pass. But what, exactly, is the nature of Carnegie's problem. Remember that first paragraph? Surely he cannot be basing his criticisms on those who are at the minimum NCEA passing level? And, hopefully unlike what is described in the Listener, he's not talking about students who did not take the right (i.e. relevant) subjects. But that's the problem, he is.

Our research over a large number of first-year engineering students reveals a systemic problem with the concept of the "achieved" NCEA grade. This grade spans a huge range - it can include excellence-level students who have made some minor errors, through to students who really have not gained competence in the material. Many potentially capable students report that they just "cruised through NCEA" knowing they would be able to obtain the achieved grade with minimal work. A failed assessment is "no big deal"as often they get to re-sit that assessment. When they take that work ethic and expectation into University study, they are in trouble. The result is that there is significant variability in "achieved-level" students' study habits, work ethic and subsequently, actual subject knowledge.
This is all about achieved level students. The excellence students "who have made some minor errors" would not have made those errors elsewhere. In other words, those who are actually mediocre at a lower level of education do not have excellences apparent elsewhere. They may have one or two, but, there would not be evidence of consistent ability to reach the excellence standards. In any case, "some minor errors" typically result in merits and in many subjects a lack of "perception" or possibly even "originality" is enough to leave excellence out of place. Students know this, read their memes on the variety of NCEA Memes Facebook pages (assuming they stay up now the exam period is finished).

Carnegie also attributes the ability of students to "cruise" in NCEA by putting in minimal effort to NCEA itself. This doesn't follow at all. Students who are satisfied with minimal work and just passing will always be like that. They would have been like that prior to NCEA... perfectly happy with Level 5B or whatever and before that at primary with whatever they had to do then. It is not about the system. Any system will have students who can't be bothered and so won't bother and they will do whatever it takes to pass... but no more than that. This differs to his comments about "re-subs" (which are different to re-sits, a re-sit means the entire standard is done again with a new/different assessment, a re-sub parts are added/changed in the original and clarifications are when minor errors/omissions are altered), which are valid enough. Knowing that there is a possibility of trying again does affect attitudes. That can be attributable to NCEA, whereas the earlier comments cannot. It is, perhaps, a black mark against NCEA but most teachers only offer them to students on the margins of a different mark (when a possibility of change exists). Students are also familiar with (and this is particularly relevant as it is always the last thing in any given year) externals where there is no possibility of re-subbing or anything like that.

Compounding this issue, we are aware of secondary schools that actively discourage students who have only gained an achieved grade in mathematics at NCEA Level 2 from enrolling in that subject at Level 3. This seems to be due to a concern that those students might fail, with a consequential adverse effect on that school's league table standings. This removes many potential students from even being able to consider enrolment in engineering.
League Tables don't really exist in New Zealand. Yes, Metro and other publications like North and South (and I'm fairly certain I've seen these in the Herald) do make their own but that's not official. Carnegie is like many parents here when they mistake deciles for some measure of academic performance (although, as PISA notes socio-economic factors are enormous performance predictors in New Zealand). The real reason why schools don't allow such students to carry on with maths (or other subjects) is because they want the students to actually take subjects, which they can pass. Carnegie needs to sit down and think, "If schools think that Level Two Achieveds translates to failure at Level Three, why on earth do I expect Level Three Achieveds to translate to success at university?" All subjects have some sort of requirement like this... typically in specific standards (such as at least achieved in "causes and consequences" plus 12 credits for history). Calculus at my school actually needed a merit at algebra and an achieved in calculus going from level two to three. The reason being that algebra is absolutely huge and the skills need to be better than mediocre to succeed. The solution is not what Carnegie seems to imply (getting rid of the entry requirements) but rather improving maths education from primary all the way up. The standard is there, let's actually get students to that standard rather than getting rid of it because it's "too hard" or "prevents students from considering enrolment".
At the core of the problem is the lack of a real, meaningful grade. I advocate for a return to real percentage scores - these motivate students to improve themselves, to gain the best score they can. It would end the farce of excellent students receiving an achieved grade due to one or two minor mistakes. It would give the universities a real understanding of student ability.
I would contest the reality of one or two minor mistakes. In fact, I would go as far to say that Carnegie has no idea at all about the structure of NCEA tests (particularly those most relevant to him... physics and calculus). They're not a bunch of questions that are all equally hard (i.e. tests don't translate easily to percentage scores) . This is a good thing. An excellence level problem typically requires being able to demonstrate knowledge of a concept within an application -- it's not easily visible. An achieved, in contrast, is textbook 2x + 7 = 0, solve for x type stuff. They're clear demonstrations of the concept and, in fact, are the concept written as simply as it could be while still being a problem. In externals, a question will consist of multiple parts and will be scored out of 8. Students need to meet a certain score to pass, get merit and get achieved across all the questions within a paper/standard. Really, the closest thing to percentages came in two years ago (i.e. 2011) with Grade Score Marking. Don't know what it is? Both NZQA and Wikipedia's NCEA article explain it. Didn't know it existed? Blame the media that only reports about NCEA when exams start or something goes wrong. That's the primary issue with NCEA no-one bothers to actually understand it on the basis that they think they do.
I further advocate for the abolition of secondary school league tables. Let the schools take on marginal students into Level 3 mathematics and physics. Schools should not be penalised (via league table results) if they have taken a risk with some students and it hasn't paid off - rather they should be incentivised to take such risks and make a real difference. If students have never taken Level 3 mathematics or physics, then they are effectively lost to engineering (and many other career options as well).
I'm sorry, did I say implied? He does have a point though, schools should be putting more effort into maths. The issue is that by Year Thirteen it's a bit late.

*This information is erroneous. I believe that I calculated the percentage of 280/360 = 78% when rounded. 250/360 is actually 69% to 2sf. This error casts doubt on the other figures but I may as well check with updated requirements, thus NCEA's really 72%. CIE is 74% (which suggests that this unchanged) and IB 73% (so, again, probably not changed) with all percentages rounded to 2sf. The credit counts are correct. However, in any case, I should probably point out that these percentages must be treated with caution. After all, the CIE one is based on subject units (a single A level is worth 2 subject units) whereas NCEA increases by standard so is far more granular. That is one reason why one has to ask oneself: are these percentages meaningful, even as a rough guide?

No comments:

Post a Comment