In the vexing problems, I dismissed the hard problems of computer science as being incidental to another problem: we can’t say what the value of our work is. That post contained plenty of questions, precisely because the subject is so unknown.
There are plenty of ways in which the “value” of something can be discussed, but let’s stick with economic value for the moment. Imagine you have the opportunity to write some software, but only if I pay you for it. And not even then, but only if you can make me reasonably confident that I will get some return; that having the work done is worth more than the cost of you doing the work.
A corner of the software industry just cowered and stuck its head in the sand. “#NoEstimates!”, they cry. This problem is hard, so why should we solve it?
That’s a really interesting perspective. You should come and talk about #NoEstimates at my conference. It’ll be on sometime, in a place, and there’ll be a flight that can get you there. Maybe. That’s enough information to be going on for it to be clear that you should commit to it, so I’ll see you there.
At the other end are the folks who would literally write whole books about the topic (and given that books are themselves literary, that “literally” is itself meant literally). I pulled a few likely titles off the shelf to see what they would tell me, and ended up with:
- Software Project Management, 5th Edition
- PSP: A Self-Improvement Process for Software Engineers
- Software Engineering Economics
We have to wonder how some people can be certain that estimates are worthlessly inaccurate, while others such as Barry Boehm (the author of Software Engineering Economics and the COnstructive COst MOdel it describes) have built careers out of explaining how to do it well.
Here’s one hypothesis: Boehm is a charlatan, and the COCOMO doesn’t work at all. Let’s see whether there’s evidence for that.
A likely source is COCOMO evaluation and tailoring by Miyazaki and Mori from Fujitsu. The authors of that paper, applying the COCOMO (’81, not the later COCOMO II) model to a corpus of projects undertaken by Fujitsu, conclude:
The original COCOMO overestimates the effort required to develop software in our environment, but its tailoring methodology is applicable. […] The resulting model fits 68% of the projects with less than 20% relative error, after the deletion of two outliers.
That’s maybe short of brilliant – the median project effort in their paper seems to be over 10 person-years, so they’re saying “there’s around two chances in three that we can tell you what this project will cost to within twice a developer’s salary” – but it’s much more precise than “#NoEstimates!” and only quite a lot less accurate.
It might be interesting to track the application of COCOMO through time, and see whether a larger corpus of projects and refinement of the model has improved its accuracy and applicability. Unfortunately, that goal is in conflict with a general industry observation, noted in the introduction of the Fujitsu paper:
We have the impression that software people prefer to start from scratch rather than improving upon the work of others, which seems to be slowing down the progress in this field.
And indeed much of what was written in the sources I’ve been talking about here is no longer applied. All three of the books I mentioned up top talk about Earned Value (EV) Analysis, recording the actual rate of delivering on a project against its Planned Value (PV). For example, Hughes and Cotterell recommend plotting earned value using the “0/100 technique”:
where a task is assigned a value of zero until such time as it is completed when it is given a value of 100% of the budgeted value
Agilistas will recognise the resulting plot: it’s a burn down chart, where a story is not delivered until it has been accepted according to the definition of done, at which point it is completely delivered.
Ultimately somebody in the Agile community must have noticed that they accidentally borrowed a software engineering concept, because the burn down chart is now unfashionable.
If the team starts with a very full, prioritized, and estimated backlog with the expectation they will burn down all tasks to zero — does this not sound like a Waterfall mentality?
Oh noes, the dreaded W word! Run away from this thought, for it is a thought that somebody in the past has already considered!
And that is why I consider these problems to be vexing: when potential solutions are proposed, they are by necessity existing ideas from the old school that must be rejected in favour of…whatever is not those solutions. As Craftsmanship is post-Agile is post-Software Engineering is post-Crisis thinking, surely it will soon be time for post-Craftsmanship thinking. And then post-whatever-comes-next thinking. And we still won’t know how to compare expected value to expected cost.
