Click on a line to see the project details. Then click the project title or "Read More" link for even more background.

While it is hard to draw definitive lessons from a handful of programs, it is nearly certain that IT modernization efforts will overrun their cost estimates by significant amounts. The chart also demonstrates the challenges of holding these projects accountable when they do: cost overruns, delays, and reduced functionality are so common that even self-proclaimed success-stories have them.

One solution is to try to make more realistic initial estimates. More data is required (and if you know of a project we're missing, please leave a comment), but trying to combine more than 50 nontrivial legacy government IT systems for less than $400 million to $500 million seems nearly impossible. So we should be skeptical when we see overoptimistic—or outright fraudulent—project estimates that claim to do just that. In fact, we should actually expect future modernization efforts to be even more difficult, as the technical debt of legacy systems and data continues to grow faster than our ability to fix it.

But to make an accurate estimate and to be held accountable for it, you first must have a clear understanding of what you’re trying to accomplish.

How many systems are we replacing anyway?

One of the most fascinating and demoralizing examples of a failed IT modernization project is the U.S. Air Force Expeditionary Combat Support System (ECSS). A U.S. Air Force acquisition audit team that investigated the $1 billion fiasco noted that one of the root causes was the fact that “the number of systems [the] Expeditionary Combat Support System was to replace is unknown.” Such a finding is not surprising, although it is appalling from a project planning, management, and funding perspective. As we noted in a blog post about the 2014 Senate’s ECSS audit report, “The Air Force has, on different occasions, used wildly different estimates on the number of existing legacy programs, ranging from ‘175 legacy systems’ to ‘hundreds of legacy systems’ to ‘over 900 legacy systems.’” Strangely, an article written in 2014 by the USAF general in charge of ECSS at its termination used a figure of some 214 legacy systems, a number never used before. You can follow the varying estimates of the number of systems ECSS was supposedly meant to replace in the chart below.

Click any of the points to see the exact estimate and source. Replay animation

Which, if any, of these estimates are accurate? None of the numerous audit reports actually try to provide a list of systems, so we’ll probably never know. In this case, the shifting measure of complexity was practically screaming out that the problem itself was poorly understood, and success was never likely.