Whenever you start a software project, it seems like someone above you wants a schedule.
Now.
Never mind that you haven’t collected requirements, never mind that you haven’t designed the product, much less the program. Never mind that you don’t know who’s going to be on the team or even how large the team is. And they don’t really care about the schedule, anyway. They just want to know that the software will be done by X date.
Dumb. Dumb. Dumb. Dumb. Dumb. Dumb. Did I mention that it’s dumb?
At best, you can only estimate as well as you understand your product and your team. Given that you are at the beginning of a project, you understanding of the product is sketchy, at best. If you don’t have a team yet, your understanding of your team is equally sketchy. The best sort of estimate you are going to come up with at this point is measured in months, quarters, or years, not days or hours. “This is a AAA game title. Figure six to twelve quarters.”
That’s as good as it gets.
Once you have a pretty good understanding of your product, and the design is pretty much fleshed out, and you have team leads assigned and a good guess at how large your team is going to be, you can get that estimating unit down to months. “This is a AAA game title planned as a follow on to last year’s successful product, twelve to eighteen months.”
Eventually, if you’re doing your job right, you hit that magic point where the product is designed, the technology is designed, and the team is hired and all the tasks are assigned to people. Your understanding of the entire development system is now as good as it’s going to get. And if you’re lucky, you can get that estimating unit down to weeks.
Except for one thing. Now that you are actually building your product, reality intrudes. Testing determines that your design isn’t as usable as you wanted. Users discover new features they want and old features that should be different. Technology designs don’t quite mesh the way you thought they would. Hardware is slower than spec. Your understanding of the product and process gets fuzzy again. On the plus side, you now have (an increasing amount of) historical data on this very project, allowing you to scale existing estimates according to what’s really happening.
Then you hit Alpha. All your features are in the product. All that’s left is testing, tuning, internationalization, and debugging. And now, if your boss wants a good hard date, down to the day, you are really screwed.
Because you can’t give him one.
Testing, tuning, and debugging are inherently unschedulable tasks. Any designer who tells you they know how long it will take to tune any particular feature until it’s shippable is lying. Likewise any programmer who tells you how long it will take to fix a bug they haven’t found yet. Any specific instantiation of this problem is unique. Good designers and programmers (and project managers!) can tell you how long it takes on average to tune or debug a project of a given size, but none of them can tell you how long this one will take.
This essential truth is lost on many people.
Fortunately, statistical analysis can help where straightforward human thought may fail. When I was a TD at EA in the early 90s, we had the opportunity to examine schedules for many (10s, maybe 100s) of projects. We found that, for projects of a given size and a consistant definition of Alpha, Beta, and Final, the time from Alpha to Final was constant.
None of us expected that. But the data was clear.
You can’t figure out how long it takes to fix a bug until it’s fixed. You can’t figure out how many bugs you’ll need to fix. But in the aggregate, you can find out how long it takes to final a project of a certain size. And as long as you remember to figure out the standard deviation as well, you can predict (with reasonable confidence, or at least knowing how much confidence you have) how long it will take to have a 75% chance of completing the project, or a 90% chance.
If you don’t have that data, all is not lost. I think everyone must know by now that you can graph daily defect counts and do a regression on the blocking defects. Where that line crosses the horizontal axis is the most likely day you have no blocking bugs left. That’s not the day you call it final, but it is the day you put it in for Release Candidate (RC) testing. If you’re lucky, it comes out the end with no new blocking bugs found, and you call it done.