Economic software testing

Economics should determine how and how much you test your software. Your tests must provide more value than they cost, not just to write, but to modify, debug, and run. That means your testing strategy will change over the course of product development – through design, prototyping, construction, iteration, deployment, and maintenance.

For example, much software is written speculatively, before it is known whether it will be valuable. The goal at that stage is to figure out its value, not to make it robust, bug free, or maintainable by people other than the authors. Writing and rewriting automated tests as the software goes through rapid changes might not make economic sense. During maintenance, on the other hand, functionality changes slowly and users expect it to keep working reliably. It then usually makes sense to have a large number of automated tests that provide maintainers confidence that the software works as intended after a change.

This may seem self-evident, but it will only happen if someone has the perspective, the information, ^… In a for-profit business that means a basic understanding of the product economics, the costs and benefits of different tests, and how the two interact. Not a trivial ask.

the incentives, and the authority to make good decisions about how and how much to test. It is the manager’s responsibility to ensure someone does.