Let me begin by saying that I have a lot of respect for the Eclipse team. They have maintained tight milestones on a massive project. They have remained responsive to bug reports and feature requests. I could go on for quite a while. Much can be learned from that project.
Unfortunately, they stumbled on the classic tradeoff and that has lead to a classic mistake.
The classic tradeoff is: features vs. quailty. Do you spend more time fixing defects or creating new features? The classic mistake is: releasing a feature with a high defect rate.
Notice that I said defect rate and not defect count. It may be that a feature has zero defects when it is shipped but in the recent past the number of defects found and resolved was high. Understanding the difference goes back to a firm understanding of QA and a little math. The defect rate is the number of defects found per unit time. The unit of time is commonly determined by the frequency and duration of test / fix cycles. A large defect rate means that there were many defects found in a cycle whereas a small defect rate means that there were few defects found. The defect rate for a complex feature typically increases then decreases. This trend follows the common sense rule that a single test cycle will not reveal all defects. In fact, in early test cycles, defects will block other defects from being seen. As fixes are put in place, the number of defects will eventually decrease.
For those more savvy, the change rate of the defect rate (a defect acceleration of sorts) is also useful. A positive acceleration means that there are more defects found in a cycle than previously. A negative acceleration means that less defects were found than previously. And a zero acceleration does not mean that zero defects were found — it means that the same number of defects were found and were found previously.
Releasing a feature while the defect acceleration is positive (or zero with large defect count) or while the defect rate is still large will typically result in a feature with a high defect rate. In effect what you’re doing is replacing a testing cycle with your client. This is the classic mistake.
If there is one thing that clients are exceptionally bad at it would be understanding that development takes a finite amount of time (and typically longer than they expect). After the process of requesting and specifying a feature has finished and while development is occurring, the client is already thinking about the next set of features. By the time that development releases the features, the client already has expectations beyond that of the release. This results in development always trailing or missing expectations. (If you’re a developer reading this, you’ve probably thought Well, duh!. Trust me, clients really don’t understand the basis of these skewed expectations.)
What I’ve just described always occurs. It’s the background noise that developers and project managers have to deal with. Deciding between missing features or having defects while being affected by the background noise is the classic tradeoff.
Eclipse tradeoff and mistake
Late in development of Eclipse 3.0, code folding was introduced. There was a high defect rate (and a positive defect acceleration) associated with the feature. It interacted with a significant number of other features and is highly complex (this adds up to a high quality risk). Lo and behold, code folding is significantly buggy.
Should code folding, a much asked for feature, have been included in the release? Check back for a follow up entry on this subject.