Why is it so hard to estimate software (and control system) projects?

Does this seem familiar? You start out as a grunt in the trenches designing stuff, writing code, and you keep getting a string of projects to work on with insanely low budgets and short timelines, and you think, “I can do better than this.” You decide you want to manage your own projects, and your boss thinks that’s great because (a) he trusts you, and (b) that’s one less thing for him to do. He starts you off with some small project…

The first thing he asks you for is an estimate. You listen to the scope, give it about 10 minutes of thought, write up an estimate and hand it to him:

• Design: 3 days
• Programming: 2 weeks
• Testing: 2 days

So, three weeks. “Let’s just say four weeks,” says your boss. Great!

You start out on your design. It takes about 3 days. Then you start programming, and you get it all working, and you’re on budget. Excellent! Your boss asks you how it’s going, and you say, “I’m on budget! I just have some testing left to do.” Great, let’s have a review with the customer.

After the review with the customer, it’s clear that half of what you did isn’t going to meet their expectations, and the other half is going to get all messed around because of the first half changing. Damn!

“How is this going to affect the budget?” says your boss…

“Well, since we have four weeks, and we’re only two and a half weeks in, we should be ok. I’ll probably have to come in on a weekend.”

So you work your butt off, and the hours are really racking up, but you think you’re going to make it, and all of a sudden, BAM! You have this one annoying problem that you just can’t seem to fix. Maybe it’s a bug in a Windows driver, or a grounding issue, or whatever. It’s something hard, and you burn a whole week trying various solutions trying to solve it. Suddenly you’re overbudget.

After this experience, and a few more like it, you start to get the hint that maybe it’s you. Then you learn there’s an entire industry out there that functions as a support group for people like you: Project Management. You get to go to little two-day project management seminars where they commiserate with you. An instructor gets up and talks about all the common pitfalls of managing a project, and everyone with any experience trying to do it says, “yeah, exactly!”

Then they tell you the solution to this is to impose the tried and true “Project Management Methodology” on it:

• Initiate
• Plan
• Execute
• Monitor
• Complete

This gives you a warm fuzzy feeling. The feeling that you’re back in control of your world, and if you’re into control systems like me, then you’re all about controlling your world.

Now the first step includes the estimation step. You do a work breakdown structure, down to small details, and you assign a best case, probable case, and worst case estimates to each task, and you usually use some pseudo-statistics to come up with a weighted sum of the estimates for all tasks.

This process has worked well, but only in certain industries. If you’re building a house, it works well. If you’re building a skyscraper or a bridge, it works well, particularly if the result is similar to something you’ve done before.

It’s odd that most Project Management books use the “send a man to the moon” example. While that project was completed on time (before the end of the decade), and it achieved the performance goal, it went wayyy over budget. If this was a corporate project, it would be a complete failure. It was only considered a success because it was a government job.

If you’re trying to estimate how long it will take to build a house, the estimate starts with a completed design. Then the Project Manager calls up the framers, roofers, drywallers, electricians, plumbers, etc., and gives them a copy of the blueprints. The plumber has a really good idea, based on the number of fixtures, etc., how much it’s going to cost and how long it will take. He does this all the time.

Software is a different beast. It’s like asking you to estimate the architect’s time when he drew up the blueprints of the house. The construction of the house is analogous to the compiling step in software development. In software, we’ve completely automated that step. We can go from design to prototype in a matter of seconds. What we’re estimating in software development is the design effort, not the construction effort. Unfortunately, before the design is done, we haven’t even figured out what the problems are yet, let alone the solutions.

Computer programming is almost always novel. Repetition is the ultimate sin. If you find repetition, you refactor it out into a re-usable library or module. Each project brings new platforms and technologies, ever higher levels of abstraction, and a novel problem, so we have no baseline to estimate from.

In fact, the easier it is to estimate a project, the less important that project really is. If your job is easy to estimate, it’s probably repetitive enough that it’ll soon be automated away.