I came to write this article, not because I have been abstractly thinking about billion dollar public sector software projects, but because I live in New Zealand and one day I learned that our New Zealand government, and more specifically, the Inland Revenue Department (IRD), was planning to spend one billion dollars or thereabouts on upgrading all its software systems.
Actually it was one billion, but the estimate crept up to 1.5 billion dollars. (For the benefit of my non-New Zealand readers who have no idea how much New Zealand money is worth, I can tell you that one New Zealand dollar is worth about 80 US cents, so 1.5 billion is about 1.2 billion US$.)
Also, New Zealand is only a small country, with only 4 million people in it. So a billion dollars is quite a lot of money to spend on anything.
Now I don't know, off the top of my head, how the IRD could save money on writing its new software cheaper.
One reason I don't know, and can't even begin to find out, is that I don't have access to the relevant details about what the existing software does, or what the full set of requirements is.
Which is part of the problem.
(I have worked in a software consultancy that did government software projects, so I do have some idea of what kind of thing happens in those projects, but I've never actually worked on an IRD project.)
Out in the real world, far away from government departments and their software projects, there live startups, which use modern cutting edge (and sometimes "bleeding edge") software technology to write software which disrupts existing ways of doing things.
"Disruption" means either a whole lot cheaper, or a whole lot better. Sometimes it's a whole lot cheaper and slightly worse, but it's so much cheaper that no one cares that it's slightly worse.
If we can somehow "disrupt" the IRD's software upgrade project, then we should be able to save lots of money. Like hundreds of millions of dollars.
Unfortunately there is no way to make disruption happen. But there are ways we can allow it to happen.
Disruption usually happens when the set of people with access to information about a problem is large enough that a certain subset of creative, intelligent, determined and experienced people actually attempt to solve the problem, and, out of the various attempts made, at least one of those attempts actually succeeds to the degree that we can call it a "disruption".
So the recipe for disrupting a government software project is simple enough:
Unfortunately, the default treatment of government software projects is the exact opposite of this:
Under these conditions, the probability of implementing any "disruptive" solutions falls to zero.
A minor variation is the RFP, or "Request for Proposal", which has a detailed description of your requirements (often over-detailed), and where you ask interested organisations to describe their proposed solutions (and an associated price).
Someone at the government department then reads the responses, and taking into account the quality of the descriptions of the solutions, and the likely reliability of the candidates, and perhaps also the quoted prices, decides which of the proposed solutions will be the solution to be implemented.
There are real costs involved in creating the conditions that make "disruption" possible.
To make sure that the largest number of potential disruptors get to know about your problem, information about the problem has to be made freely available. It costs money just to find this information and present it in a clear and comprehensible form. Most of the people who receive this information will do nothing about it. And there is a risk of unnecessarily releasing "confidential" information.
There is also a wastefulness in making multiple attempts to solve a problem.
But multiple attempts are an essential prerequisite to disruption.
If only one attempt at a problem solution is allowed, then risk management comes into play. If you are only permitted one attempt at a solution, then you will want to be sure that your one permitted solution is not a terrible solution. So to be safe you will pick a method that is very similar to whatever has worked in the past, and that "safe" solution will almost certainly not be disruptive.
The larger the problem, the more urgent the need to be "wasteful" in our attempts to solve it.
For a million dollar project, it would be silly to spend a million dollars looking for better ways to do it.
For a billion dollar project, it would be silly not to spend a million dollars looking for better ways to do it. Or even ten million dollars.
For example, it might be that to write a new tax system more cheaply, you have to invent a new programming language just for that purpose. Or extend a new programming language. Or maybe write a new framework.
At the same time, the one thing you definitely don't want to do is decide that you should invent a new programming language, and then choose one person (or team) to invent that new programming language, and then have them invent it, and then use that new language to write all your code.
Because that plan will almost certainly fail, horribly. (Or it will be abandoned at some point, and the project will fall back onto Plan B, which is to write all the software in Java/Hibernate/Spring, because that is what someone used for the last Government software project that didn't completely fail.)
(Not counting my own efforts, which are difficult for me to evaluate objectively, I've seen about half a dozen attempts to invent new languages or frameworks on government or "enterprise" software projects. Of these, one was better than the open source project it replaced, one was over-enginneered for its actual use-case but better than nothing because nothing else was available at the time, and the rest were not so good.)
If I wake up one morning and read in the newspaper that the New Zealand Government is planning to waste ten million dollars looking for better ways to upgrade the IRD's software systems, then I will sleep easy the following night, knowing that my taxpayer dollars are being spent wisely.
But this hasn't happened.
The need for secrecy, or at least confidentiality, cannot be completely ignored.
I suggested above that information about the problem to be solved needs to be made freely available.
When it comes to a tax department, probably some information about the department's operations needs to be kept confidential (like how they catch tax cheats), and other information doesn't need to be kept confidential (indeed, some information should be made public just for the reason that the tax department is charged with implementing public laws and regulations, and there should be open-ness and transparency in how it goes about its business).
A requirement for confidentiality does not prevent the use of wasteful experimentation as a problem-solving strategy, but it does slightly change the way you allow it to happen.
In particular, if the relevant problem details are confidential, you can't just tell everyone the details and then wait for brilliant creative people to respond to them. Instead you have to actively hire some of those brilliant creative people, and get them to sign the required confidentiality agreements, and then tell them the details, and then tell those brilliant creative people to waste their time experimenting with potentially disruptive ways to better or more cheaply solve those problems. And hopefully something will come of it.
New Zealand is a democracy, with elections and free speech and all that stuff that makes for a proper democracy. And one of the things that democracies always have is effective oppositions.
So, can New Zealand's parliamentary opposition save us from over-spending on the IRD's software upgrade?
I wrote an email to the Labour Party, on this very subject. They "appreciated my input", they assured me they were "following this project very closely", and they gave me a link to this blog article by David Cunliffe.
That article tells us that a billion dollars is a lot of money, and that a billion and a half dollars is even more money, and it generally accuses Peter Dunne (the government minister in charge of the IRD) of not being very good with money.
But Cunliffe doesn't really get to what I consider the nub of the matter, i.e. the need to spend money on experimenting with ways to cut down on such an enormous project cost.
Here I will re-quote his quote from KPMG:
“We do not believe the timeline presented… is achievable. A programme of this complexity, where scoping and articulation of long-list options, a robust options assessment (critical for Treasury support) and the programme’s design (i.e. ordering of tranches and projects) have not yet occurred.”
In other words, KPMG's view, which Cunliffe seems to agree with, is that we need more planning, and more assessment of options in advance. There is no hint that the government should do any kind of exploration or experimentation.
