I’ve recently been talking about software engineering economics, in a very loose way, but so have other people. And now I understand that it’s annoying when people talk about it, and have decided to continue anyway. I’ve decided to continue because what I see is either inaccurate comparisons being made, or valid comparisons that have questionable applicability outside their immediate domain. The world of IT cost comparison is still run by marketing, not by operations.

Recently, I read Don’t Build Private Clouds. Subbu says that the sticker prices (up to $10k for a server that will last four years, vs. up to $1500 per month for a public cloud machine) should not be compared because there are additional costs to self-hosting:

1. Engineering costs
2. Network automation costs
3. Loss of agility
4. Opportunity costs

Fine, but what are those costs? Why do I not need to do any engineering or automation if I use a public cloud provider? If I do, what is that going to cost? What agility and opportunity do I lose by tying my infrastructure to any one cloud vendor, and what will that cost?

Subbu’s blog says that he is a “cloud helper”, and that goes a long way to explaining why we didn’t get a straight answer on the cost comparison. We’re not being told that cloud services are cheaper, instead we are being told of the Fear, Uncertainty and Doubt involved in choosing the less-favoured path.

Similarly, an IBM employee recently said that Macs are cheaper than PCs by up to $543 per user (remember that’s “up to”, not “as much as” – the lower bound given is $273). Let’s ignore the conflict of interest arising from Apple’s global partnership with IBM: IBM wouldn’t say that their partner systems are cheaper so that they could drum up interest in their partnership services, surely. Surely. I mean, this isn’t the IBM of 1984, is it?

What IBM’s VP says is that over a four-year lifetime, among employees who are given the choice of which platform and model computer they want, the Macs are cheaper. That’s of course a figure about which it’s possible to make realistic comparisons: given IBM’s approach to desktop support, IBM’s level of staffing, IBM’s applications, IBM’s approach to working, IBM’s budgeting for IT support operations, it’s cheaper for people who choose Macs to use Macs than for…well, it’s not clear, but it seems to be than for the “everybody else” bucket: not only people who chose PCs to use PCs, but people who weren’t given a choice to use PCs.

So before you make a textexpander macro for that link and insta-reply to anyone who uses the phrase “Apple Tax”, just how similar is your environment to IBM’s?

In the vexing problems, I dismissed the hard problems of computer science as being incidental to another problem: we can’t say what the value of our work is. That post contained plenty of questions, precisely because the subject is so unknown.

There are plenty of ways in which the “value” of something can be discussed, but let’s stick with economic value for the moment. Imagine you have the opportunity to write some software, but only if I pay you for it. And not even then, but only if you can make me reasonably confident that I will get some return; that having the work done is worth more than the cost of you doing the work.

A corner of the software industry just cowered and stuck its head in the sand. “#NoEstimates!”, they cry. This problem is hard, so why should we solve it?

That’s a really interesting perspective. You should come and talk about #NoEstimates at my conference. It’ll be on sometime, in a place, and there’ll be a flight that can get you there. Maybe. That’s enough information to be going on for it to be clear that you should commit to it, so I’ll see you there.

At the other end are the folks who would literally write whole books about the topic (and given that books are themselves literary, that “literally” is itself meant literally). I pulled a few likely titles off the shelf to see what they would tell me, and ended up with:

• Software Project Management, 5th Edition
• PSP: A Self-Improvement Process for Software Engineers
• Software Engineering Economics

We have to wonder how some people can be certain that estimates are worthlessly inaccurate, while others such as Barry Boehm (the author of Software Engineering Economics and the COnstructive COst MOdel it describes) have built careers out of explaining how to do it well.

Here’s one hypothesis: Boehm is a charlatan, and the COCOMO doesn’t work at all. Let’s see whether there’s evidence for that.

A likely source is COCOMO evaluation and tailoring by Miyazaki and Mori from Fujitsu. The authors of that paper, applying the COCOMO (’81, not the later COCOMO II) model to a corpus of projects undertaken by Fujitsu, conclude:

The original COCOMO overestimates the effort required to develop software in our environment, but its tailoring methodology is applicable. […] The resulting model fits 68% of the projects with less than 20% relative error, after the deletion of two outliers.

That’s maybe short of brilliant – the median project effort in their paper seems to be over 10 person-years, so they’re saying “there’s around two chances in three that we can tell you what this project will cost to within twice a developer’s salary” – but it’s much more precise than “#NoEstimates!” and only quite a lot less accurate.

It might be interesting to track the application of COCOMO through time, and see whether a larger corpus of projects and refinement of the model has improved its accuracy and applicability. Unfortunately, that goal is in conflict with a general industry observation, noted in the introduction of the Fujitsu paper:

We have the impression that software people prefer to start from scratch rather than improving upon the work of others, which seems to be slowing down the progress in this field.

And indeed much of what was written in the sources I’ve been talking about here is no longer applied. All three of the books I mentioned up top talk about Earned Value (EV) Analysis, recording the actual rate of delivering on a project against its Planned Value (PV). For example, Hughes and Cotterell recommend plotting earned value using the “0/100 technique”:

where a task is assigned a value of zero until such time as it is completed when it is given a value of 100% of the budgeted value

Agilistas will recognise the resulting plot: it’s a burn down chart, where a story is not delivered until it has been accepted according to the definition of done, at which point it is completely delivered.

Ultimately somebody in the Agile community must have noticed that they accidentally borrowed a software engineering concept, because the burn down chart is now unfashionable.

If the team starts with a very full, prioritized, and estimated backlog with the expectation they will burn down all tasks to zero — does this not sound like a Waterfall mentality?

Oh noes, the dreaded W word! Run away from this thought, for it is a thought that somebody in the past has already considered!

And that is why I consider these problems to be vexing: when potential solutions are proposed, they are by necessity existing ideas from the old school that must be rejected in favour of…whatever is not those solutions. As Craftsmanship is post-Agile is post-Software Engineering is post-Crisis thinking, surely it will soon be time for post-Craftsmanship thinking. And then post-whatever-comes-next thinking. And we still won’t know how to compare expected value to expected cost.