The Novopay BIM

TVNZ and Radio New Zealand have done a couple of stories reporting on the leaked Briefing to the Incoming Minister (BIM) which the Ministry of Education supplied to Steven Joyce about the Novopay debacle. They’ve focused on the Ministry’s warning that Novopay could take 1-2 years to get working properly. I’ve also received a copy of this document, and I was more interested in the Ministry’s summary of what actually went wrong. Briefly:

• Novopay is designed so that schools do everything online (presumably through a browser or thin client). When the platform was launched there were ‘significant issues’ with the online user interface (UI); for example, it wasn’t possible to submit time-sheets for part-time teachers.
• The work-around for problems submitting payments via the UI was for the schools to fill in a form and submit it to Talent2′s Novapay service desk via e-mail, and they’d manually enter the payment data into the system.
• But the service desk wasn’t staffed or trained for this – they were supposed to be supporting an online platform in which the schools did almost everything themselves. So this created a huge backlog of manual payments for them to enter, many of which missed the payrolls.
• The service center also generated a vast number of errors in payments because it doesn’t have ‘robust quality assurance’ (I take this to mean there’s no verification when service center staff manually submit data: so if someone is being paid $20/hour and they work for ten hours, the manual system won’t prevent a service center staffer from accidentally paying them $2.00)
• Talent2 aren’t able to produce complete draft payrolls for the schools to QA, so these errors aren’t picked up before the payroll data goes live.
• The briefing is contradictory on the state of the programming defects in Novopay. The background summary insists that the problems leading to the huge submission of manual forms have been fixed, but in a subsequent section on software defects the Minister is told that new defects continue to come to light, and Talent2 have been unable to fix the existing problems. (I get the feeling they’ve been relying on manual work-arounds, rather than debugging the actual code.)

The Ministry’s solution is to put more resources into the service center while Talent2 fixes up software bugs via a release code schedule (I get the impression the extra software engineering cost is being met by the Ministry).

Education Minister translated redux

The Gunning-Fog index is a commonly used algorithm to determine the readability of English writing. (Details on the wiki page here. See also the comments section.) I wrote a perl script that reads in Hansard transcripts from Question Time and looks for sentences that score an 18 on Gunning-Fog, which ranks as incomprehensible, and then replaced that sentence with the word AAARRGGGGGH! Here’s how Hekia Parata’s most recent oral question plays out.

6. CHRIS HIPKINS (Labour—Rimutaka) to the Minister of Education: What specific criteria were used to determine whether a school in Christchurch was identified for restoration, consolidation or rejuvenation?

Hon HEKIA PARATA (Minister of Education) : Tena koe, Mr Speaker. AAARRGGGGGH! However, it is important to note that those categories describe learning community clusters, and not individual schools.

Chris Hipkins: I raise a point of order, Mr Speaker. The Minister’s answer has given me an explanation of what each of the categories are. I have asked for what the criteria were in order to put schools within those categories.

Mr SPEAKER: I think the member has got a legitimate grievance because the member actually asked what specific criteria were used to determine whether a school was identified for restoration, consolidation, or rejuvenation. The Minister, in answering, gave criteria for clusters. If she could clarify for the House whether that applies to individual schools, that would be helpful because that is what the question asked.

Hon HEKIA PARATA: AAARRGGGGGH! They do not relate to individual schools.

Mr SPEAKER: I thank the Minister.

Chris Hipkins: I raise a point of order, Mr Speaker. Schools were specifically listed in her proposal under one of those headings, and I have asked the criteria on which they were listed under those headings. That is a primary question, and it is not an unreasonable question.

Mr SPEAKER: I accept absolutely that it is not an unreasonable question, and that is why I sought clarification from the Minister. What the Minister seems to be pointing out to the House is that those three classifications—restoration, consolidation, or rejuvenation—did not actually apply to individual schools. As to what the Minister has told the House, I have got to take the Minister’s answer at face value. I cannot second-guess that. The Minister has given an answer to that question.

Chris Hipkins: What specific criteria were used to identify whether a specific school was proposed for a merger or a closure?

Hon HEKIA PARATA:  Sorry, could the member repeat the supplementary question?

Mr SPEAKER: I invite the member to repeat his question.

Chris Hipkins: I will try. What specific criteria were used to identify whether a specific school, an individual school, was proposed for a closure or a merger?

Hon HEKIA PARATA:  I raise a point of order, Mr Speaker. AAARRGGGGGH!AAARRGGGGGH!

Mr SPEAKER: Order! If I am going to help members on this matter, they should be a little silent. I think it is not unreasonable—the primary question asked what specific criteria were used to determine whether a school in Christchurch was identified for restoration, consolidation, or rejuvenation. The Minister in answering that question pointed out those three categories applied to clusters of schools, so the member has not unreasonably now dug further into that answer and asked then what criteria were used to identify schools for, I think his language was, merger, which is similar to consolidation, or closure, which is highly relevant to some schools in Christchurch. That is not an unreasonable supplementary question, and I am ruling that it is not an unreasonable supplementary question.

Hon HEKIA PARATA: AAARRGGGGGH!AAARRGGGGGH! That is what—

Hon Members: What are they?

Hon HEKIA PARATA: AAARRGGGGGH!AAARRGGGGGH! Those are some of the criteria.

Chris Hipkins: Were the assessment of earthquake damage and the likely cost of repair for each of the schools proposed for merger or closure based on a physical inspection of each site and building; if so, who conducted that assessment?

Hon HEKIA PARATA: AAARRGGGGGH!AAARRGGGGGH! Some of them involved book assessments. AAARRGGGGGH!

Colin King: Was the change in demographics taken into account when developing the criteria?

Hon HEKIA PARATA:  Yes, it is a people-related issue. AAARRGGGGGH!AAARRGGGGGH!AAARRGGGGGH!AAARRGGGGGH!

Chris Hipkins: Was a physical assessment of the earthquake damage done on each of the schools that she proposed for merger or closure before she proposed that; if not, why not?

Hon HEKIA PARATA: AAARRGGGGGH!

Mr SPEAKER: Tracy Watkins [Interruption]—Tracey Martin. I beg your pardon. My goodness.

Tracey Martin: Kia ora, Mr Speaker.

Mr SPEAKER: I beg your pardon.

Tracey Martin: Kia ora—

Mr SPEAKER: My apologies to the House.

Tracey Martin: Can the Minister assure the House that parental elections for boards of trustees will be held within 6 months of any consolidation or merger of schools in Christchurch as per the requirements of the Education Act 1989?

Hon HEKIA PARATA: AAARRGGGGGH!AAARRGGGGGH!AAARRGGGGGH!

Tracey Martin: I raise a point of order, Mr Speaker. I apologise. Perhaps I missed it. I understand the time line around mergers might not be quite clear at the moment, but my question was whether the elections for boards of trustees would be held within 6 months of any consolidation or merger as per the Education Act 1989.

Mr SPEAKER: I invite the Minister to actually answer that question.

Hon HEKIA PARATA:  Yes.

Chris Hipkins: Did she review all of the information prepared by the Ministry of Education on the likely or estimated cost of repairing schools that she was intending to propose for merger or closure, before she made the decision to propose those schools for merger or closure; if not, why not?

Hon HEKIA PARATA:  Yes.

Education Minister translated

Hekia Parata is closing a couple of special schools in the South Island, explaining:

At the very heart of this difficult decision lies the opportunity to provide services and support for more children with complex needs in their local community. We can link local services with the remaining residential provision to achieve a more personalised and high quality approach for children and their families. “I am satisfied that this combination of services will make sufficient provision for all children with special education needs both locally and nationally.

I’m pretty sure this means:

The Finance Minister is making me close schools to save money because he promised to reduce the deficit before the next election and I’m closing these ones because teaching intellectually disabled students has a high student/teacher cost ratio, which means I can save more money and have fewer parents mad at me.

Also, I’ve found that if you imagine all of Parata’s statements being said in Ralph Wiggum’s voice (‘I’m consulting widely!’) it lowers your expectations of her to a realistic level.

Education Minister fails to learn from extremely recent history

Stuff reports:

Children could be forced into “double-bunked” mega schools under Education Act changes.

One of the country’s biggest schools is already considering its options after changes that allow boards to set new start times – meaning they could offer morning and afternoon programmes from the same school, effectively doubling student numbers.

No prizes for guessing which government department dreamed up this idea. ‘If we have one group of students learning from 5 AM until 1 PM, and a second group learning from 2 PM until 8 PM, we double the education system’s asset utility ratio! Who could object to that?’

An issue dear to my heart

I think Hekia Parata is a poor Minister. Every time I hear her speak I think of the famous Orwell essay Politics and the English Language, and his translation of a passage from Ecclesiastes into modern bureaucratic speak:

I returned and saw under the sun, that the race is not to the swift, nor the battle to the strong, neither yet bread to the wise, nor yet riches to men of understanding, nor yet favour to men of skill; but time and chance happeneth to them all.

Here it is in modern English:

Objective considerations of contemporary phenomena compel the conclusion that success or failure in competitive activities exhibits no tendency to be commensurate with innate capacity, but that a considerable element of the unpredictable must invariably be taken into account.

Hence schools marked for closure are being ‘rejuvenated’, and all of Parata’s debacles are ‘challenges’. Well, one more major ‘challenge’ and Key is going to have to sack his Education Minister for gross incompetence. But I do think the PPTA are being pretty damn precious here:

Education Minister Hekia Parata hit a bum note with secondary teachers today when she said children had told her their teachers weren’t pronouncing their names correctly.

Ms Parata told teachers at the Post Primary Teachers Association (PPTA) conference in Wellington that one of the most common things Maori and Pacific children tell her is that teachers don’t know how to correctly say their names.

“It starts with pronouncing names correctly. It’s one of the most common things I hear [from] focus groups with Maori and Pacifica kids.”

The comment came in the middle of her speech, which had been well-received until that point, but was met with a collective jaw-drop and groan from teachers in the room.

The atmosphere quickly turned icy as the teachers became outraged.

Outraged at the suggestion they pronounce students names correctly? As someone whose name is occasionally mispronounced to rhyme with ‘anal’ (some people get a funny gleam in their eye: ‘Danal! That’s a hell of a nice name!’) I’m with Parata on this one.

Chicken stock update? Or Education Minister’s press release?

After a lengthy consultation and submission process in which I have listened respectfully and lengthily to wide consultations and submitted widely from experts and respectfully consulted the wider community I have bought chicken frames for the stock, because the supermarket didn’t have any necks and I am lengthily confident that this was the correct decision.

So bad education stories in the Herald on Sunday are now a thing

Their story:

Our degrees don’t repay their cost

New Zealand university degrees are the most worthless in the developed world, an international report reveals.

The value of spending years at university has been severely dented by an OECD report that reveals tertiary study adds little to our earning power – less than $1000 a year for women, not much more for men.

New Zealand is at the bottom of the global league tables.

The actual report is here. And the thing that the OECD make really clear is that they distinguish between two categories of tertiary education. Type A – university degrees – and type B: (mostly polytechnics). Taken together New Zealand is at the bottom of the table. But if you look at degrees and advanced tertiary study then New Zealand isn’t doing that badly – and the countries that are doing extremely well on that metric are mostly countries with low rates of type A tertiary education. Their degrees are highly valuable because of their scarcity. Almost every statistic in the Herald story refers to non-university level education, but the entire story is about the alleged worthlessness of degree qualifications!

The fantasy and reality of national standards

Via the Herald:

The majority of people polled think schools should publicly release their national standards performance data.

Results from a Herald-Digipoll survey showed 60.3 per cent of people agreed that schools should be forced to release the information.

However, NZ Principals Federation president Paul Drummond said he had spent years telling people why teachers felt the information would not be credible.

The opposition to National Standards has been based on the principle that lots of similar countries have tried this and it always turns into a disaster, but I think this opposition to the idea of national standards has diluted the criticism of the actual policy as implemented by Anne Tolley and Hekia Parata.

The case for national standards is that parents should be provided with meaningful information about schools so they can make informed choices about their children’s education. But National hasn’t done that – instead they’ve provided meaningless information so that parents can make uninformed, and possibly terrible choices about their children’s education. I’m not sure why they’ve done this, but they have, and the Prime Minister has been pretty relaxed about admitting so.

Here’s a graph showing the results of the individual schools performance weighted for decile (I’ll put the methodology in the comments section). It shows roughly similar outcomes across the deciles and a huge range of variation within the deciles. But the raw data doesn’t show this – so will the government’s policy cause people to take their kids out of a high performing decile one school and move them to a low performing decile five school, say, thus reducing the quality of their education? Probably. I also suspect we’ll see problems based on the narcissism of small differences: people seeing that other local schools have higher – but statistically meaningless – results, and moving their children there because they want their kids to have ‘the best school.’ That’ll happen even if the data is robust, but because of National’s incompetence, many of those parents will be moving their children out of good schools - which are close to where they live and filled with their friends – and into bad schools.

(That decile 2 school at the top – technically the ‘best school in the country’ – is the small, mostly Maori Bluff Community School. Congratulations guys.)

Conspiracy theory of the day, cockup or conspiracy edition?

So the release of the National Standards data makes it easy to do all sorts of incredibly dubious analysis on the incredibly dubious dataset of National Standards results (check out Arahunga School’s results – just one of many ‘what the?’ moments when looking through the raw data).

But looking at the relationship between class size and standards is actually a bit tricky – that information is spread across three different datasets. Standards in one, teacher numbers in another, student numbers in another. And then, when you do measure that data and see the link between class size and results, it’s instantly, really obvious that the special schools are skewing the trend.

So maybe the Herald on Sunday did that analysis, and just didn’t notice the special schools, or did notice and didn’t care. But it strikes me as awesomely convenient that Hekia Parata, the same Education Minister releasing the National Standards data was humiliated and forced to back down on her signature budget policy just a few months ago when she argued that class size didn’t matter, and that larger classes were better.

I wonder if the Herald on Sunday were scammed by the Minister’s office on this one?

Update: HoS editor Jonathan Milne called me to clarify that (a) they ran the analysis with and without the special schools, and acknowledged that the trend is less-pronounced without them, but still there – which is why they angled on it, and (b) there’s commentary and analysis of the data and results in the hard copy that hasn’t made it online yet.

Well below standard in analysis

The HoS trumpets:

Primary schools have disclosed controversial data about pupil achievement, with the surprise revelation that children in bigger classes and bigger schools get better grades.

The Herald on Sunday has conducted a comprehensive survey of schools’ national standards results, before the Ministry of Education publishes them this week.

At schools with fewer pupils for each teacher, around 70 per cent of children are achieving national standards in reading, writing and arithmetic. But at schools with more pupils for each teacher – in effect, bigger classes – the pass rates rise to about 80 per cent.

So too with school rolls: the highest proportions of children achieving or exceeding national standards are at big schools.

I’ve just checked the data – and they’re absolutely correct! There’s a strong linear relationship between class size and National Standards results. Here’s a scatter plot for reading results vs class size.

Bigger classes are better! Just like Treasury and the government said! But wait, what’s that cluster of very small classes with very poor results down there in the lower left quadrant? Well, those, which have well below standard in every single category are:

Blomfield Special School & Resource Ctre
Waitaha Learning Centre
Wilson School
Kea Street Specialist School
Parkside School
Hamilton North School
Sir Keith Park School
Every single one of which is a school for children with disabilities. Take them out of the dataset and see what happens to the trendline:

Most of the other low-quality, low ratio schools I note that they’re predominantly low-decile schools in rural areas. So now we know that very poor, very small schools aren’t that great for the kids.

Update: updated the charts after the statisticians chastened me for improper plotting, and added the regression co-efficient by (actually rather popular) request.