Insight of Agile Methods

From Zanecorpwiki

Bridge World and Software World

When you build a bridge, it's a well understood process. Most bridges are "crank" projects, meaning, you put in the parameters and turn the crank.^{[notes 1]} Sometimes you have bridges that push the envelope, but even here, the ratio of what's known to what's unknown is very high.

With software, it's entirely different. You can re-use a bridge design (at least the concepts and math) but to replicate a bridge, but you always have to build a new bridge. With software, if something exists that does the job within the required parameters, you just copy it. So, in this sense, the only reason you ever build new software is to do something new. This means that there's always a relatively high degree of unknowns in software construction. I call this the "Replication Implication".^{[notes 2]}

This is important because it's the degree of "known" vs. "unknown" which effects the cost of requirements and specification. In bridge world^{[notes 3]} you start out with a checklist that tells you pretty much everything you're going to need to figure out. Some of the stuff may be costly to figure out, but you know what you need to figure out. It's not bridges are easy, but we at least know how hard they're going to be.

In bridge world, the challenge is figuring out a hard known problem. In software world, the challenge is figuring out what the problem is. I would say that clearly the average solution cost for a defined problem is actually much less in software world than in bridge world. But in software world, there's a much higher relative cost for discovery.^{[notes 4]}

Note that it wasn't always this way. Until relatively recently, bridges that spanned more than a small river were considered wonders of the world. A big reason the Romans took so long to conquer Germania was because they couldn't cross the rivers. This is because before we "had bridges figured out", we had to figure out bridges and in that time, bridge building was like software world. The problem with bridges is that when you discover mid-project a fatal flaw, you probably don't get the chance to try again, the projects just over. You have to wait till the next generation before someone even gets the chance to benefit from new discovery.^{[notes 5]}

The Costs of Requirements

In bridge world, you may have lots of work to do, and it may take you a little time to figure out exactly how much work to do, but the problem is well defined. "Well defined" in the technical sense of being completely and fully defined such that one can work from the "well defined problem description" to the solution without encountering any unknowns.^{[notes 6]}

In software world, the Replication Implication implies that the project is and to some extent, can never be well defined. This means that in bridge world, you spend time cranking through a problem. In software world, you spend time discovering what the problem is (and then dealing with it).

This fundamentally changes the cost of requirements. In bridge world, you know the requirements you have to gather. Those requirements may feed into secondary, tertiary, etc. iterations, but the problem becomes fully parameterized through a relatively clear and predictable path.^{[notes 7]} In other words, up front planning can be done and is itself less far costly than not.

Apply this idea to software world and what happens? Let's say you're going to build an ecommerce platform. You do all your planning up front and say the cost is $80,000 and it'll take 2 guys 4 months to complete. You spend 1 week valued at $5,000 to come up with this estimate. Two weeks into the project you discover a race condition that had been overlooked which could effect data integrity. You know have to go back and refactor a good deal of the code, adding 1 week and $5,000 to the project. Three weeks after that, the customer adds a requirement for "discount codes" which require significant data model changes and a partial rewrite of the application logic and UI. This adds another week, and another $5K. By the end of the project, you end up spending twice the time and either going way over budget, or losing a lot of money on the deal.

Why didn't our ounce of prevention result in a pound of cure? The problem is that each discovery potentially effects everything done up to that point. In other words, while "cranking through a problem" has fundamentally linear cost, "re-engineering" imposes geometric costs.

At first, this seems to be an argument to do more planning up front. The subtle issue (which took software many decades to realize) is that in software world, the distillation of a problem to the point at which it's "well defined" is just as hard as implementing the problem. Software is essentially codified thought, so is no more or less expensive to codify your thought in code than it is in specification. In other words, "specification" is effectively as costly as implementation. To completely deal with a project, you end up essentially implementing it--in specification rather than code--from beginning to end. The specification cost is itself geometric.

While both "cost curves" are geometric, you end up better off "just doing it" because by cranking through the known problems, you get closer to the unknown problems. The closer in you get, the less of a chance the geometric cost has a chance to really kick in. In other words, by building up to rather than anticipating problems, you end up dealing with the unknown on the shallow portion of the cost curve. Your code time is dramatically effected by each change, but since the cost of specification and code are equivalent, the only savings you can hope to have is by dealing with "close" problems.

Notes

1. ↑ Not to say that there isn't a lot of skill and knowledge in bridge design, even crank bridge design. Just that any competent designer could do it, would get it right, and would end up with more or less the same result.
2. ↑ The alliteration is annoying, but the words are pretty spot on.
3. ↑ "Bridge world" refers to most bridges, while "software world" refers to most software. Some bridges have lots of new engineering and some software can be developed from a checklist. From this point on, however, I'm going to drop my caveats and just say bridge world and ask the reader to refrain from using the acknowledged exceptions to attack the rule.
4. ↑ Some skilled in the art may call this "fundamental design", etc. Meaning, bridges have lots of design, but the "fundamental design" has already been done. Software usually requires a fair amount of fundamental design.
5. ↑ As bridges have become bridge like, software too is becoming more bridge like, and doing it much faster than bridges did for the reasons discussed above. The differences I discuss here are not "eternal fundamentals", but are a product of the point in time at which we consider the problem.
6. ↑ This characterization of bridges as being entirely well defined is clearly hyperbole, but is intended to simplify the distinction between bridge and software world.
7. ↑ It would be ludicrous to say that any project can be completely known a priori, but bridge world problems collapse into a relatively solid state where the error bounds are tight through a well understood path.