In the vexing problems, I dismissed the hard problems of computer science as being incidental to another problem: we can’t say what the value of our work is. That post contained plenty of questions, precisely because the subject is so unknown.
There are plenty of ways in which the “value” of something can be discussed, but let’s stick with economic value for the moment. Imagine you have the opportunity to write some software, but only if I pay you for it. And not even then, but only if you can make me reasonably confident that I will get some return; that having the work done is worth more than the cost of you doing the work.
A corner of the software industry just cowered and stuck its head in the sand. “#NoEstimates!”, they cry. This problem is hard, so why should we solve it?
That’s a really interesting perspective. You should come and talk about #NoEstimates at my conference. It’ll be on sometime, in a place, and there’ll be a flight that can get you there. Maybe. That’s enough information to be going on for it to be clear that you should commit to it, so I’ll see you there.
At the other end are the folks who would literally write whole books about the topic (and given that books are themselves literary, that “literally” is itself meant literally). I pulled a few likely titles off the shelf to see what they would tell me, and ended up with:
- Software Project Management, 5th Edition
- PSP: A Self-Improvement Process for Software Engineers
- Software Engineering Economics
We have to wonder how some people can be certain that estimates are worthlessly inaccurate, while others such as Barry Boehm (the author of Software Engineering Economics and the COnstructive COst MOdel it describes) have built careers out of explaining how to do it well.
Here’s one hypothesis: Boehm is a charlatan, and the COCOMO doesn’t work at all. Let’s see whether there’s evidence for that.
A likely source is COCOMO evaluation and tailoring by Miyazaki and Mori from Fujitsu. The authors of that paper, applying the COCOMO (’81, not the later COCOMO II) model to a corpus of projects undertaken by Fujitsu, conclude:
The original COCOMO overestimates the effort required to develop software in our environment, but its tailoring methodology is applicable. […] The resulting model fits 68% of the projects with less than 20% relative error, after the deletion of two outliers.
That’s maybe short of brilliant – the median project effort in their paper seems to be over 10 person-years, so they’re saying “there’s around two chances in three that we can tell you what this project will cost to within twice a developer’s salary” – but it’s much more precise than “#NoEstimates!” and only quite a lot less accurate.
It might be interesting to track the application of COCOMO through time, and see whether a larger corpus of projects and refinement of the model has improved its accuracy and applicability. Unfortunately, that goal is in conflict with a general industry observation, noted in the introduction of the Fujitsu paper:
We have the impression that software people prefer to start from scratch rather than improving upon the work of others, which seems to be slowing down the progress in this field.
And indeed much of what was written in the sources I’ve been talking about here is no longer applied. All three of the books I mentioned up top talk about Earned Value (EV) Analysis, recording the actual rate of delivering on a project against its Planned Value (PV). For example, Hughes and Cotterell recommend plotting earned value using the “0/100 technique”:
where a task is assigned a value of zero until such time as it is completed when it is given a value of 100% of the budgeted value
Agilistas will recognise the resulting plot: it’s a burn down chart, where a story is not delivered until it has been accepted according to the definition of done, at which point it is completely delivered.
Ultimately somebody in the Agile community must have noticed that they accidentally borrowed a software engineering concept, because the burn down chart is now unfashionable.
If the team starts with a very full, prioritized, and estimated backlog with the expectation they will burn down all tasks to zero — does this not sound like a Waterfall mentality?
Oh noes, the dreaded W word! Run away from this thought, for it is a thought that somebody in the past has already considered!
And that is why I consider these problems to be vexing: when potential solutions are proposed, they are by necessity existing ideas from the old school that must be rejected in favour of…whatever is not those solutions. As Craftsmanship is post-Agile is post-Software Engineering is post-Crisis thinking, surely it will soon be time for post-Craftsmanship thinking. And then post-whatever-comes-next thinking. And we still won’t know how to compare expected value to expected cost.
Boehm has certainly built a career out of COCOMO, but I am not so sure about the “do it well” part.
The original model fitting was flawed in that it ignored outliers and got some of the coefficients wrong: http://shape-of-code.coding-guidelines.com/2016/05/19/cocomo-how-not-to-fit-a-model-to-data/
What do Miyazaki and Mori mean by “…resulting model fits…”? I think you will find it means a power law fits the data collected at the end of the project. What use is a model that fits the data at the end of a project?
I always thought that estimation happened at the start of a project!
I first read a criticism of COCOMO in about 1985 in the ‘Software Engineering Journal’, published by the IEE. It wasn’t conclusive either but said their were grave doubts about whether there was too much uncertainty in estimates of software for the model to be useful and that is only half the problem. That was about estimating to a fixed specification that was assumed to be correct. The real problem is that the customer doesn’t know what they want, so can’t specify it. Design engineers in other disciplines build models to create a feedback loop. We tried prototyping then realised that if our models were build out of software they didn’t have to be thrown away, so modelling and development became increments of the same code and business value could be delivered earlier.
There is more need for engineering within agile processes than waterfall, so those who threw away what they had are fools. We are building a system while changing it, so not only solving a problem but building flexible components that we can re-use or re-engineer.
People who believe they don’t estimate are deluding themselves. They keep chopping a function up until the next few bits will take less than a unit of time, so they estimate but not far into the future. We don’t really need to know how much a project is going to cost, only whether the next thing is worth doing and whether we are going to do it. If it isn’t, don’t. People only demand total cost and plans because it’s the only way they know to feel in control of a software development, but the aren’t.