The Dim-Post

January 22, 2013

Why Novopay is still broken and hard to fix

Filed under: Politics — danylmc @ 9:24 am

TVNZ reports:

Schools are bracing themselves for more problems with Novopay just days out from another education pay round.

With many new school staff joining the pay roll and some big issues showing up on pay reports, worries about Novopay’s competence are second only to schools’ concerns about their budgets.

At Mount Eden Normal School in Auckland, the balance available for spending on staff has swung from $10,000 in credit, to tens of thousands in debt.

When things go wrong with big software engineering projects, it’s often hard to explain to non-technical people why the damn thing is so hard to resolve. Why not just throw people and money at it? Hire more staff? Sack the managers and bring in someone competent, etc. The answer to those questions is Brooks’ Law.

Imagine that the Novopay system is a gigantic factory filled with tens of thousands of complex machines, all interlinked and interdependent on each other. Because that’s basically what it is, it’s just built in computer code instead of bricks and steel. If you make a small change to, say, the machine that calculates holiday pay, that change can have a massive impact on the entire factory. It could short out all the power. It could also have a very subtle impact that causes problems over the long term – like an error in calculating long service leave.

So if you have ten software engineers all fixing problems at once, they’re going to spend a lot of time talking to each other, and testing their changes to make sure they don’t cause problems in other parts of the system. And if you hire another ten engineers, all you’ve done is increase the amount of communication and testing that needs to be performed.

Software development companies know all of this stuff. It’s why they conduct extensive testing and often ship products months, or sometimes even years late. Because they’re aware that if they ship a faulty product something like the Novopay fiasco will happen. But Talent2 isn’t a software development company – they’re a recruitment company that decided to branch out into software development.

I guess the lesson here is that they call it software engineering for a reason. If you were building a real bricks n’ mortar factory you wouldn’t hire an HR company who’d never built anything before, and if half of the test subjects told you that a real factory wasn’t ready you wouldn’t ignore them and switch it on anyway. (I acknowledge that it’s possible Craig Foss and Hekia Parata would actually do both of those things.)

About these ads

24 Comments »

  1. “possible”. I think you mean likely :)

    Comment by LucyJH — January 22, 2013 @ 9:36 am

  2. But software is so easy, anyone can build it…

    Look at Novapay, anyone did build it.

    Comment by Moz in Oz — January 22, 2013 @ 10:24 am

  3. So how did they get the contract then?

    Comment by alex — January 22, 2013 @ 10:51 am

  4. Lowest tender price?

    It’s notoriously hard to evaluate software tenders, because the companies that do big projects don’t like to specify who will be doing the management work. So you get a name company putting random staff into the job, supervised by someone who knows what they’re doing. Or so you hope. Then they hire a pile of architects, coders, testers, business analysts and so on, selecting primarily for “ability to work in a team” (ie, suck up the shit poured on them by said big company) rather than for any talent in their nominal field of expertise.

    One extra complication is that often no one person knows what the existing system does, or what the new system is supposed to do. So there’s a lot of time spent trawling around looking for people who do know, then reconciling their ideas. And despite what some people think, the rules for paying teachers (for example) are not simple, and not laid down in legislation or awards. There’s a whole bunch of exceptions, to the point where it can easily seem to be a bunch of arbitrary rules designed to make sure that every teacher gets a unique pay amount.

    Comment by Moz in Oz — January 22, 2013 @ 11:09 am

  5. But Talent2 isn’t a software development company – they’re a recruitment company that decided to branch out into software development.

    I broadly agree with the post, but this sentence is specious. Management at Talent2 didn’t wake up one morning and say “Hey, let’s make Cheryl and Sarah in Human Resources start to program software!” They’ve been in the business a long time, and they’re certainly not the only people in the recruitment industry that offer horizontally integrated services across HR, Payroll and Training.

    Or to put it another way, it would be a bit like looking at General Electric, and saying “Haha, what does a lightbulbmaker know about offering retail financial products, stupid people” and then having to turn around a little while later and acknowledge that GE money makes a butt-fuck ton of money for the group.

    Comment by Phil — January 22, 2013 @ 11:18 am

  6. The competitive tender process is horrible from both sides – from the software company side you’re often doing an incredible amount of work for free up front responding to a tender that has been written with a particular company in mind. Often the tender has been written by a particular company who will be bidding on it. The tender will include a mix of dumb, vague, and completely impossible tasks. There is no way to say well hey, you’re specifically asking for something that is really stupid, we think this is a typo in your RFP – your only option is to either walk away or lie and say sure, we can do this and hope.

    Most of the time you’re just being used as a free comparative price check for the predestined company who will be given the job, in order to tick off the ‘was put out to tender’ part of the process.

    Comment by xy — January 22, 2013 @ 11:22 am

  7. Talent2 did a new payroll system for the University of Otago a while ago. They were widely known as Talentless by the university’s internal HR department, who think that a reasonable product was only finally delivered because of the expertise of the university’s own IT division, which ended up doing much of the work. Evaluating software tenders can be done on the basis of track record, but it’s not necessarily easy to find out the reality behind previous contracts.

    Comment by Dr Foster — January 22, 2013 @ 11:24 am

  8. “…So how did they get the contract then..?”

    They would have rocked up to (who actually dispises the way the education sector works and who willingly drinks the kool-aid of modern mangerial practice of constantly re-weighing the widget) with an out-of-the box SaaS product that they would have claimed could be easily customised – click click click done! – and because it was all in “the cloud” and driven through a browser it is totally OS agnostic and future proof and important managers can see the most important things – reports on reports – directly on their iPhones and iPads from anywhere anytime and it is way, way cheaper than the existing piece of steam age software that needs a soon to be ancient version of Windows to run and – quelle horreur! – requires the support of several staff who have been around for twenty years and understand how it works intimately. Getting rid of those potentially troublesome blackmailing blackard support staff can only be seen as an added bonus to modern HR groupthink. Send it to the cloud! Cheaper, faster, better, less reliant on the enemy… err I mean education payroll IT experts!

    Comment by Sanctuary — January 22, 2013 @ 11:29 am

  9. *…They would have rocked up to a generic CEO clone (who….”

    Comment by Sanctuary — January 22, 2013 @ 11:30 am

  10. If you make a small change to, say, the machine that calculates holiday pay, that change can have a massive impact on the entire factory.
    This was certainly true back in the 90s, but there is no real need to have these kind of dependencies today. Software developers have spent a long time trying to decouple components and there are a stack of patterns and development approaches that remove this risk. IMO (and I’m a professional software/database/BI developer) the best approach is Test-Driven Development. You basically write a test, then you write the code, then you run the test and confirm the code did what was expected.

    You end up with hundreds or thousands of tests, and if you do a modification that breaks the code, then some of the tests will fail and you’ll be able to rectify it.

    Anway, that’s how the pros do it, but by the sounds of it, Novopay has a hacked-together pile of copralitic spaghetti code.

    My guess is that when they shoehorned the complexity of the NZ education employment system into Novopay, they ended up generating multiple entities for each staff member in order to handle the various types and tenures of employment. Then they had to make a kludge that brings them all together presents the payslip as if its for only one entity. That stuff is hard to do, even if you have a properly designed system (which Novopay obviously isn’t), and I reckon that’s why you see all the weird stuff of people getting paid for jobs they didn’t do.

    Comment by mikaerecurtis — January 22, 2013 @ 11:47 am

  11. Your comments seem to be a bit broken today, ironically?

    Comment by richdrich — January 22, 2013 @ 1:29 pm

  12. The IT side of Talent2 comes from the acquisition of a software engineering firm. Now you can argue that group management doesn’t know how to run it etc, but they are actually software engineers in there…

    Comment by garethw — January 22, 2013 @ 1:29 pm

  13. The history behind the switch from Datacom’s payroll solution to Novopay is also interesting.
    Moving from NZs biggest app dev vendor and outsourced payroll supplier to a relative unknown says a lot about the quality of the RFP process.

    Comment by Gregor W — January 22, 2013 @ 1:59 pm

  14. Check Wikipaedia for Software Engineer….a functionary with a dubious claim to “Engineer”. What is missing is a systems analyst with a full understanding of the strategic goal for the project as well as a fine understanding of the minutiae down to the level of program code. Engineers are great at “bottom up” but a good analyst works from top down. Been doing it for 45 years.
    Ian W

    Comment by Ian Way — January 22, 2013 @ 2:36 pm

  15. It’s worth just pausing here to remind ourselves the job was previously handled by a New Zealand company. I don’t know if Datacom handled the task without a single hitch, but if there were problems, they weren’t big enough to stay on the news front pages for long. One aspect of the tender process that’s often overlooked is the risk involved in ANY change – it’s never negligible, but I guess nobody thought to cost that into the procurement process.

    Comment by billbennettnz — January 22, 2013 @ 4:21 pm

  16. I don’t know if Datacom handled the task without a single hitch
    we were paid for a total of about 70 years and i dont recall any hitches, and my pay was complex because of relieving across several schools.

    Comment by DR — January 22, 2013 @ 6:16 pm

  17. One of Howard Fancy’s favorite stories as Secretary of Education was that the new payroll system was released and crashed the week after he started – which is one of the reasons he refused to mess with it once it was fixed. And yes the old system did use to hit the front page of the Dominion from time to time when a new patch broke something like a feed to a bank. I think when you have a long term provider supporting an increasingly clunky dated system (which costs) the grass is always going to look greener on the other side of the fence. And that everyone’s expectations of IT systems has grown exponentially – the scope for a replacement system would include every improvement and iteration of the old 15 (?) year old one plus some extras. It is easy to over simplify what is a very complex layered issue (and yes I do thing Novapay is a disaster and has been mismanaged but given the current government IT environment I’m not convinced that other companies/systems might not have had a similar result – I spent 6 months working for Queensland Health in 2011!)

    Comment by sonjanz — January 22, 2013 @ 7:11 pm

  18. Sanctuary, the minister who approved Novopay’s contract was Chris Carter. National has to fix it.

    Comment by Grant Michael McKenna — January 22, 2013 @ 8:51 pm

  19. Another consideration: Teachers who have had their pay compromised are reliant on a school board “loaning” the difference so teachers can pay their bills like everyone else. Not all school boards are financially rosy, some are in serious debt, usually to the MoE .

    Some schools may find it expedient to refuse financial help to teachers they’d just as soon see the back of – think schools who want to go Charter and would love to be rid of older, experienced teachers because they cost the most. More than one way to skin a cat.

    Note, i am a teacher (day relief) and my main school was quick to loan me the $1k Novopay shortchanged me and are equally happy not to demand back the $500 overpaid to me later. But I have also worked at a school where relationships between senior management (BOT, Principal etc) and the teaching staff was acrimonious and well versed in using such ploys. BTW it took 2 pay cycles (1 month) for novopay to fix the underpayment. They’ve never fixed the other, despite many requests since October.

    Comment by Scintilla — January 23, 2013 @ 11:59 am

  20. “Talent2 isn’t a software development company – they’re a recruitment company that decided to branch out into software development”

    Then how do explain IBM stuffing up QLD Heath payroll system?. It got so bad that the QLD Govt tried to sue them. Stll stuffed btw.

    Comment by K2 — January 23, 2013 @ 6:58 pm

  21. What’s not really mentioned is the pressure put to bear on the Novopay people so that the old Datacom system could be switched off (and the maintenance not paid). I’m not involved, btw, I’ve just seen other projects go bad because the client wants to turn off the old system due to financial pressure and doesn’t want to wait until the new system is properly tested in a real environment.

    mikaerecurtis, “Test driven development” looks like a great way of developers getting more money from the client, and my guess is, it’ll pass. Best test is real test, because developers can test for situations they don’t know about.

    Comment by Lucia Maria — January 23, 2013 @ 8:03 pm

  22. Even big guns like IBM got it wrong with INCIS in the late 1990s. It also didn’t help that the police top brass kept changing their requirements for it.

    Comment by deepred — January 24, 2013 @ 1:20 am

  23. Great, Chris Carter sponsors a complex IT project, what could possibly go wrong.

    Comment by timmy — January 26, 2013 @ 7:43 pm

  24. $182 million dollars spending on a .NET web app that doesn’t even do friendly URLs? Going to http://www.novopay.co.nz gets you http://www.novopay.govt.nz/default.aspx, this seems like a bunch of juniors coming out of school and got hired to do an arse job. If you use Chrome, look at their login page, the layout is broken, you would have thought that they hired some testers right? I smell some nasty deal has been made here.

    Comment by James — June 4, 2013 @ 8:10 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Rubric Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 418 other followers

%d bloggers like this: