Structure and Interpretation of Computer Programmers

I make it easier and faster for you to write high-quality software.

Saturday, August 17, 2019

On the efficient allocation of scarce resources with alternative uses

Most of what I know about “the economy” is outdated (Adam Smith, Karl Marx, John Maynard Keynes) or incorrect (the news) so I decided to read a textbook. Basic Economics, 5th Edition by Thomas Sowell is clear, modern, and generally an argument against economic regulation, particularly centralised planning, tariffs, and price control. I still have questions.

The premise of market economics is that a free market efficiently uses prices to allocate scarce resources that have alternative uses, resulting in improved standard of living. But when results are compared, they are given in terms of economic metrics, like unemployment, growth, or GDP/GNP. The implication is that more consuming is correlated with a better standard of living. Is that true? Are there non-economic measurements of standard of living, and do they correlate with the economic measurements?

Even if an economy does yield “a better standard of living”, shouldn’t the spread of living standards and the accessibility of high standards across the population be measured, to determine whether the market economy is benefiting all participants or emulating feudalism?

Does Dr. Sowell arrive at his office at 9am and depart at 5pm? The common 40-hour work week is a result of labour unions and legislation, not supply and demand economics. Should we not be free to set our own working hours? Related: is “unemployment” such a bad thing, do we really need everybody to work their forty hours? If it is a bad thing, why not reduce the working week and have the same work done by more people?

Sowell’s argument allows that some expenses, notably defence, are better paid for centrally and collectively than individually. We all get the same benefit from national defence, but even those who are willing to pay would receive less benefit from a decentralised, individually-funded defence. Presumably the same argument can be applied to roads, too, or space races. But where are the boundaries? Why centralised military, say, and not centralised electricity supply, healthcare, mains water, housing, internet service, or food supply? Is there a good “grain size” for such centralising influences (it can’t be “the nation”, because nations vary so much in size and in centralisation/federation) and if so, does it match the “grain size” for a market economy?

The argument against a centralised, planned economy is that there’s too much information required too readily for central planners to make good judgements. Most attempts at a planned economy preceded broad access to the internet and AI, two technologies largely developed through centralised government funding. For example, the attempt to build a planned economy in Chile got as far as constructing a nationwide Telex network before being interrupted by the CIA-funded Pinochet coup. Is this argument still valid?

Companies themselves are centralised, planned economies that allocate scarce resources through a top-down bureaucracy. How big does a company need to get before it is not the market, but the company’s bureaucracy, that is the successful system for allocating resources?

posted by Graham at 23:03  

Sunday, December 11, 2016

No, you can’t ignore politics

I wrote, a couple of years ago, about the fact that you can’t ignore ethics in software engineering. Your software is built for a reason, it’s used for a reason, you need to be aware of those reasons and whether you’re supporting or enabling them.

That goes for politics too. That hacker news declared this week to be politics-free week shows an immaturity and unprofessionalism that makes it a dangerous place to learn about making software from. Making software is the act of some people producing things for other people, it is inherently a political act. Choosing a framework or programming language is political. Attending a meet-up is political. Being paid is oh-so-political. Publishing your side project under an OSI-approved licence? The OSI is a political organisation, the courts that will determine whether the terms of your licence are binding are political, the officials of those courts are selected by a political process.

And, anyway, let’s have a look at the front page of hacker news now, roughly five days into their politics-free week.

What they probably mean is not “political stories are off-topic”, but that political stories that stray from the default politics are off-topic. Anything that doesn’t sound like I agree with it must be subjective, whereas things I agree with are objective. Just as the default narrative in society is often white, male and affluent, so it is those things in the Valley and neolibertarian. Anything else is just politics.

posted by Graham at 16:21  

Saturday, November 26, 2016

On the rhetorical cost of ownership

I’ve recently been talking about software engineering economics, in a very loose way, but so have other people. And now I understand that it’s annoying when people talk about it, and have decided to continue anyway. I’ve decided to continue because what I see is either inaccurate comparisons being made, or valid comparisons that have questionable applicability outside their immediate domain. The world of IT cost comparison is still run by marketing, not by operations.

Recently, I read Don’t Build Private Clouds. Subbu says that the sticker prices (up to $10k for a server that will last four years, vs. up to $1500 per month for a public cloud machine) should not be compared because there are additional costs to self-hosting:

  1. Engineering costs
  2. Network automation costs
  3. Loss of agility
  4. Opportunity costs

Fine, but what are those costs? Why do I not need to do any engineering or automation if I use a public cloud provider? If I do, what is that going to cost? What agility and opportunity do I lose by tying my infrastructure to any one cloud vendor, and what will that cost?

Subbu’s blog says that he is a “cloud helper”, and that goes a long way to explaining why we didn’t get a straight answer on the cost comparison. We’re not being told that cloud services are cheaper, instead we are being told of the Fear, Uncertainty and Doubt involved in choosing the less-favoured path.

Similarly, an IBM employee recently said that Macs are cheaper than PCs by up to $543 per user (remember that’s “up to”, not “as much as” – the lower bound given is $273). Let’s ignore the conflict of interest arising from Apple’s global partnership with IBM: IBM wouldn’t say that their partner systems are cheaper so that they could drum up interest in their partnership services, surely. Surely. I mean, this isn’t the IBM of 1984, is it?

What IBM’s VP says is that over a four-year lifetime, among employees who are given the choice of which platform and model computer they want, the Macs are cheaper. That’s of course a figure about which it’s possible to make realistic comparisons: given IBM’s approach to desktop support, IBM’s level of staffing, IBM’s applications, IBM’s approach to working, IBM’s budgeting for IT support operations, it’s cheaper for people who choose Macs to use Macs than for…well, it’s not clear, but it seems to be than for the “everybody else” bucket: not only people who chose PCs to use PCs, but people who weren’t given a choice to use PCs.

So before you make a textexpander macro for that link and insta-reply to anyone who uses the phrase “Apple Tax”, just how similar is your environment to IBM’s?

posted by Graham at 17:44  

Wednesday, November 16, 2016

On the business case for (or against) software

In the vexing problems, I dismissed the hard problems of computer science as being incidental to another problem: we can’t say what the value of our work is. That post contained plenty of questions, precisely because the subject is so unknown.

There are plenty of ways in which the “value” of something can be discussed, but let’s stick with economic value for the moment. Imagine you have the opportunity to write some software, but only if I pay you for it. And not even then, but only if you can make me reasonably confident that I will get some return; that having the work done is worth more than the cost of you doing the work.

A corner of the software industry just cowered and stuck its head in the sand. “#NoEstimates!”, they cry. This problem is hard, so why should we solve it?

That’s a really interesting perspective. You should come and talk about #NoEstimates at my conference. It’ll be on sometime, in a place, and there’ll be a flight that can get you there. Maybe. That’s enough information to be going on for it to be clear that you should commit to it, so I’ll see you there.

At the other end are the folks who would literally write whole books about the topic (and given that books are themselves literary, that “literally” is itself meant literally). I pulled a few likely titles off the shelf to see what they would tell me, and ended up with:

We have to wonder how some people can be certain that estimates are worthlessly inaccurate, while others such as Barry Boehm (the author of Software Engineering Economics and the COnstructive COst MOdel it describes) have built careers out of explaining how to do it well.

Here’s one hypothesis: Boehm is a charlatan, and the COCOMO doesn’t work at all. Let’s see whether there’s evidence for that.

A likely source is COCOMO evaluation and tailoring by Miyazaki and Mori from Fujitsu. The authors of that paper, applying the COCOMO (’81, not the later COCOMO II) model to a corpus of projects undertaken by Fujitsu, conclude:

The original COCOMO overestimates the effort required to develop software in our environment, but its tailoring methodology is applicable. […] The resulting model fits 68% of the projects with less than 20% relative error, after the deletion of two outliers.

That’s maybe short of brilliant – the median project effort in their paper seems to be over 10 person-years, so they’re saying “there’s around two chances in three that we can tell you what this project will cost to within twice a developer’s salary” – but it’s much more precise than “#NoEstimates!” and only quite a lot less accurate.

It might be interesting to track the application of COCOMO through time, and see whether a larger corpus of projects and refinement of the model has improved its accuracy and applicability. Unfortunately, that goal is in conflict with a general industry observation, noted in the introduction of the Fujitsu paper:

We have the impression that software people prefer to start from scratch rather than improving upon the work of others, which seems to be slowing down the progress in this field.

And indeed much of what was written in the sources I’ve been talking about here is no longer applied. All three of the books I mentioned up top talk about Earned Value (EV) Analysis, recording the actual rate of delivering on a project against its Planned Value (PV). For example, Hughes and Cotterell recommend plotting earned value using the “0/100 technique”:

where a task is assigned a value of zero until such time as it is completed when it is given a value of 100% of the budgeted value

Agilistas will recognise the resulting plot: it’s a burn down chart, where a story is not delivered until it has been accepted according to the definition of done, at which point it is completely delivered.

Ultimately somebody in the Agile community must have noticed that they accidentally borrowed a software engineering concept, because the burn down chart is now unfashionable.

If the team starts with a very full, prioritized, and estimated backlog with the expectation they will burn down all tasks to zero — does this not sound like a Waterfall mentality?

Oh noes, the dreaded W word! Run away from this thought, for it is a thought that somebody in the past has already considered!

And that is why I consider these problems to be vexing: when potential solutions are proposed, they are by necessity existing ideas from the old school that must be rejected in favour of…whatever is not those solutions. As Craftsmanship is post-Agile is post-Software Engineering is post-Crisis thinking, surely it will soon be time for post-Craftsmanship thinking. And then post-whatever-comes-next thinking. And we still won’t know how to compare expected value to expected cost.

posted by Graham at 19:08  

Sunday, November 13, 2016

The Vexing Problems in Programming

I admit it, I’ve been on the internet for quite a while (I could tell you that my ICQ number is 95941970, but I haven’t logged in for years) and my habits haven’t changed. I still regularly get technology news from slashdot, and today was no exception. An interesting article was Here Be Dragons: The 7 most vexing problems in programming. Without wanting to spoil the article for you by giving away the punchline, there are indeed some frustratingly difficult problems mentioned: multithreading, security, encryption are among the list.

All of these problems are sideshows to what I see as one of the largest and most vexing issues in programming: the fundamental rule to business administration is that your income should be greater than your costs, but software makers still, by and large, don’t have a way to compare the expected value of their work to the expected cost of the work.

The problem in space

Different software teams – and individuals – do work in different contexts, and in different ways. The lone wolf micro-ISV is not the same as an individual contract developer. The in-house IT team does not have the same problems to solve as the shrinkwrapped software vendor, and those developing web services for public consumption have yet another context. The team with core hours all working in a single office is different from the distributed team inhabiting multiple time zones.

How much of this variety is essential, and how much is accidental? How much of it is relevant, when considering some intervention, process change, or technique? Consultants speaking at conferences (another context with its own similarities and differences from the others) don’t tend to talk about what researchers in fields such as psychology would recognise as “threats to validity” of their work, but given all of the ways in which software is made, we need to know whether some proposal applies to all of them, or to some of them, or whether it has been applied to some of them and might be applied to others, and what would assist or confound that application.

The problem in time

What are the accepted, tested and validated ways to identify who will be using and otherwise impacted by our software systems? To know whether they can use the system we propose, and whether it is the best system for their intended use? To ensure that our proposed software systems treat those people ethically? To understand the cost (to ourselves and to others) of constructing those systems? To deliver the systems to the people who will interact with them? To choose which people are or aren’t entitled to access? To build a representation of the problem to be solved, to validate that representation, to validate the solution against that representation?

Where an answer exists to those questions, what are the contexts in which it is valid and what are the threats to its validity? How has that answer been compared with other possibilities? How has it been confirmed? How has it been challenged? How can I find out about those confirmations and challenges? How can I find out about any alternatives? What techniques exist to weigh up those alternatives quantitively, rather than relying merely on the persuasiveness of the conference speakers promoting those solutions (and, by the way, the books/screencasts that describe the solutions)?

The lack of a problem

Why should I care? There’s enough money in software at the moment to mean that I don’t need to be any good at knowing what works or doesn’t work, I just need to get out there and sell some software. In the rare situation that I don’t make my money back, that’s just the market forces at work, and I can go and get a high-paying job somewhere while I lick my wounds, and pick another programming language/framework/platform/whatever it is that’s going to make my next attempt definitely succeed.

Clearly, this bottomless pit of money that arises from society’s unwavering faith in software and its ability to cure all ills is never going to run out. There’s no need to worry ever about whether we’re doing it right, because there’ll always be someone out there willing to pay for us to do it wrong. Life as a programmer is like some kind of socialist utopia where whether we’re making a valuable contribution or not, the rest of society is looking out for us.

That’s going to last forever, right?

posted by Graham at 12:49  

Sunday, January 18, 2015

The other pink dollar

How did (a very broad and collective) we go from selling NeXT at $440M to selling Tumblr at $1.1B, in under two decades? Why was Sun Microsystems, one of the most technologically advanced companies in the valley, only worth two Nests?

I don’t think we’re technologists (much) any more. We’ve moved from building value by making interesting, usable and advanced technology to building value by solving problems for people and making interesting, useful and advanced experiences. The good thing about a NeXT workstation was that it was better than other workstations and minicomputers; the bad thing is that you can’t actually do a lot with a Unix workstation. You need applications, functions for turning silicon-rich paperweights into useful tools.

NeXT’s marketing was it’s easier to turn our paperweight into a tool than their paperweight, but today’s tech companies are mostly making things you can already use. The technology is a back-office function, enabling the things you can already use or working around problems the companies discovered in trying to enable those useful things. Making the paperweight with the most potential is no longer interesting to most of the industry, though there will be money in paperweights for a few years yet (even if the paperweights are becoming small enough that they can’t weigh down paper, and even if no-one has a stack of paper to weigh down any more).

Hence the current focus on “disruption”, in the Silicon Valley, not Clayton Christensen, sense. It’s easy to see how an already-solved problem can be solved faster, cheaper or better, by taking out intermediate steps or slow communications.

This is traditional science-fiction advancement. What technology do you need to get the plot moving quickly? Two people need to talk but they’re not on the same planet: you need a mobile phone. Two people need to be in the same room but they’re not on the same planet: you need a teleporter. Two people need to share the specifications of a starship but there isn’t enough paper in the universe: you need a PADD.

It’s harder to identify solutions to unsolved problems, or solutions to unknown problems. This hasn’t changed since the paperweight days, the transition has been from “well, I guess you can find some problem to solve with this workstation” to “we solved this obvious problem for you”. That’s an advance.

It really is good that we’re moving from building things that could potentially solve a problem to things that definitely do solve a problem. That’s more efficient, as fewer people are solving the same problem. Consider the difference between every company buying a Sun workstation and hiring a programmer to write a CRM application, and every company paying someone else to deliver them a CRM system.

It’s also likely the reason for the rise in open-source software, hardware, data centres and related infrastructure. Nobody’s making railroads any more, they’re making haulage companies that use railroads to solve the problem of “you’re in Chicago, Illinois but your crate full of machinery is in a port in Seattle, Washington”. There’s lots of cost in having the best rails, but questionable benefit, so why not share the blueprints for the rails so that anyone can improve them?

Well, what if it turns out that the best way to haul your goods is not on rails? If it’s easy to accept better rails, but the best solution lies in a different direction? Alan Kay would recognise this problem: everyone can see incremental improvements in the pink plane but there are magnitudes of improvements to be made by getting out into the blue plane.

How can a blue plane venture get funded, or adopted? When was the last time it happened? Is there something out there already that just needs adoption to get us onto the blue plane? Or have we set up a system that makes it easy to move quickly on the pink plane, but not to change direction to the blue plane?

posted by Graham at 19:45  

Sunday, August 17, 2014

Intellectual property and software: the nuclear option

There are many problems that arise from thinking about the ownership of software and its design. Organisations like the Free Software Foundation and Open Source Initiative take advantage of the protections of copyright of source code – presumed to be a creative work analogous to a written poem or a painting on canvas – to impose terms on how programs derived from the source code can be used.

Similar controls are not, apparently, appropriate for many proprietary software companies who choose not to publish their source code and control its use through similar licences. Some choose to patent their programs – analogous to a machine or manufactured product – controlling not how they are used but the freedom of competitors to release similar products.

There is a lot of discomfort in the industry with the idea that patents should apply to software. On the other hand, there is also distaste when a competitor duplicates a developer’s software, exactly the thing patents are supposed to protect against.

With neither copyright nor patent systems being sufficient, many proprietary software companies turn to trade secrets. Rather than selling their software, they license it to customers on the understanding that they are not allowed to disassemble or otherwise reverse-engineer its working. They then argue that because they have taken reasonable steps to protect their program’s function from publication, it should be considered a trade secret – analogous to their customer list or the additives in the oil used by KFC.

…and some discussions on software ownership end there. Software is a form of intellectual property, they argue, and we already have three ways to copy with that legally: patents, copyright, and trade secrets. A nice story, except that we can quickly think of some more.

If copyright is how works of art are protected, then we have to acknowledge that not all works of art are considered equal. Some are given special protection as trade marks: exclusive signs of the work or place of operation of a particular organisation. Certain features of a product’s design are considered similarly as the trade dress of that product. Currently the functionality of a product cannot be considered a trademark or trade dress, but what would be the ramifications of moving in that direction?

We also have academic priority. Like the patent system, the academic journal system is supposed to encourage dissemination of new results (like the patent system, arguments abound over whether or not it achieves this aim). Unlike the patent system, first movers are not awarded monopoly, but recognition. What would the software industry look like if companies had to disclose which parts of their products they had thought of themselves, and which they had taken from Xerox, or VisiCorp, or some other earlier creator? Might that discourage Sherlocking and me-too products?

There’s also the way that nuclear propagation is controlled. It’s not too hard to find out how to build nuclear reactors or atomic weapons, particularly as so much of the work was done by the American government and has been released into the public domain. What it is hard to do is to build a nuclear reactor or atomic weapon. While the knowledge is unrestricted, its application is closely controlled. Control is overseen by an international agency, part of the United Nations. This has its parallels with the patent system, where centralisation into a government office is seen as one of the problems.

The point of this post is not to suggest that any one of the above analogues is a great fit for the problem of ownership and competition in the world of software. The point is to suggest that perhaps not all of the available options have been explored, and that accepting the current state of the world “because we’ve exhausted all the possibilities” would be to give up early.

posted by Graham at 22:06  

Thursday, August 7, 2014

The Wealth of Applications

Adam Smith’s Inquiry into the Nature and Causes of the Wealth of Nations opens by discussing the division of labour. How people are able to get more done when they each pick a small part of the work to be done and focus on that, trading their results with others to gain access to the common wealth. He goes on to explain that while what people are really trading is labour, it’s hard to think about that so the more comprehensible stand-in of money is used instead.

Smith’s example is a pin factory, where one person might draw out metal into wire, another cut it to length, a third sharpen one end, a fourth flatten the opposite end. He says that there is a

great increase in the quantity of work, which, in consequence of the division of labour, the same number of people are capable of performing

Great, so dividing labour must be a good thing, right? That’s why a totally post-Smith industry like producing software has such specialisations as:

Oh, this argument isn’t going the way I want. I was kindof hoping to show that software development, as much a product of Western economic systems as one could expect to find, was consistent with Western economic thinking on the division of labour. Instead, it looks like generalists are prized.

[Aside: there really are a few limited areas of specialisation, but then generalists co-opt their words to gain by association a share of whatever cachet is associated with the specialists. For example, did you know that writing UNIX applications makes you an embedded programmer? Another specialisation is kernel programming, but it’s easy to find sentiment that it’s just programming too.]

Maybe the conditions for division of labour just haven’t arisen in programming. Returning to Smith, we learn that such division “is owing to three different circumstances”:

  • the increase of dexterity in every particular [worker]

This seems to be true for programmers. There is an idea that programmers are not fungible resources, and that the particular skills and experiences of any individual programmer can be more or less relevant to the problems your team is trying to solve. If all programmers can truly be generalists, then programmers would basically be interchangeable (though some may be faster, or produce fewer defects, than others).

Indeed Harlan Mills proposed that software teams be built around the idea that different people are suited to different roles. The Chief Programmer Team unsurprisingly included a chief programmer, analogous to a surgeon in a medical team. As needed other specialists including a librarian, technical writers, testers, and “language lawyers” (those with expertise in a particular programming language) could be added to support the chief programmer and perform tasks related to their specialities.

  • the saving of time which is commonly lost in passing from one species of work to another

Programmers believe in this, too. We call it, after DeMarco and Lister, flow.

Which leaves only one place in which to look for a difficulty in applying the division of labour, so let’s try that one.

  • to the invention of a great number of machines which facilitate and abridge labour

So we could probably divide up the labour of computing if only we could invent machines that could do the computing for us.

Hold on. Isn’t inventing machines what we do? Yet they don’t seem to be “facilitating and abridging labour”, at least not for us. Why is that?

At the technical level, there are two things holding back the tools we create:

  • they’re not always very good. Building your software on top of a tool as a way to save labour is perceived as a lottery: the price is the labour involved in working with or around the tool, and the pay-off is the hope that this is less work than not using the tool.

  • they only really turn computers into other computers, and as such aren’t as great a benefit as one might expect. We’re trying to get to “the computer you have can solve the problem you have”, and we only really have tooling for “the computer you have can be treated the same way as any other computer”. We’re still trying to get to “any computer (including the one you have) can solve the problem you have”.

This last point demonstrates that the criterion of facilitating and abridging labour is partially solved. The goal of moving closer to the end state, where our automated machines help with solving actual problems, seems to have come up a few times (mostly around the 1980s) with no lasting impact: in object-oriented programming, artificial intelligence and computer-aided software engineering; specifically Upper CASE.

Why not close that gap, particularly if there is (or at least has been) both academic and research interest in doing so? We have to leave the technology behind now, and look at the economics of software production. Let’s say that computing technology is in the middle of a revolution analogous to the industrial revolution (partly because Brad Cox already went there so I don’t have to overthink the analogy).

The interesting question is: when will we leave the revolution? The industrial revolution was over when mechanisation was no longer the new, exciting, transformative thing. When the change was over, the revolution was over. It was over when people accepted that machines existed, and factored them into the costs of doing their businesses like staff and materials. When you could no longer count on making a machine as a profitable endeavour in itself: it had to do something. It had to pay for itself.

During the industrial revolution, you could get venture capital just for building a cool-looking machine. You could make money from a machine just by showing it off to people. The bear didn’t have to dance well: it was sufficient that it could dance at all.

We’re at this stage now, in the computer revolution. You can get venture capital just for building a cool-looking website. Finding customers who need that thing can come later, if at all. So we haven’t exited the revolution, and won’t while it’s still possible to get money just for building the thing with no customers.

How is that situation supported? In Smith’s world, the transparent relationship between supply and demand drove the efficiency requirements and thus the division of labour and the creation of machines. You know how many pins you need, and you know that your profits will be greater if you can buy cheaper pins. The pin factory owners know how many employees they have and what wages they need, and know that if they can increase the number of pins made per person-time, they can sell pins for less money and make more profit.

How do supply and demand work in software? Supply exists in bucketloads. If you want software written, you can always find some student who wants to improve their skills or even a professional programmer willing to augment their GitHub profile.

[This leads to another deviation from Adam Smith’s world, but one that can be left for later: he writes

In the advanced state of society, therefore, they are all very poor people who follow as a trade, what other people pursue as a pastime.

This is evidently not true in software, where many people will work for free and others are very well-paid.]

There’s plenty of supply, but the demand question is trickier to answer. In the age of supply, not demand Aurel Kleinerman (via Bob Cringely) suggests that supply is driving demand. That people (this seemingly limitless pool of software labour) are building things, then showing them to people and seeing whether they can find some people who want those things. I think this is true, but also over-simplified.

It’s not that there’s no demand, it’s that the demand is confused. People don’t know what could be demanded, and they don’t know what we’ll give them and whether it’ll meet their demand, and they don’t know even if it does whether it’ll be better or not. This comic strip demonstrates this situation, but tries to support the unreasonable position that the customer is at fault over this.

Just as using a library is a gamble for developers, so is paying for software a gamble for customers. You are hoping that paying for someone to think about the software will cost you less over some amount of time than paying someone to think about the problem that the software is supposed to solve.

But how much thinking is enough? You can’t buy software by the bushel or hogshead. You can buy machines by the ton, but they’re not valued by weight; they’re valued by what they do for you. So, let’s think about that. Where is the value of software? How do I prove that thinking about this is cheaper, or more efficient, than thinking about that? What is efficient thinking, anyway?

Could it be that knowledge work just isn’t amenable to the division of labour? Clearly not: I am not a lawyer. A lawyer who is a lawyer may not be a property lawyer. A lawyer who is a property lawyer may only know UK property law. A lawyer who is a UK property lawyer might only know laws applicable to the sale of private dwellings. And so on: knowledge work certainly is divisible.

Could it be that there’s no drive for increased efficiency? Maybe so. In fact, that seems to be how the consumer economy works: invent things that people want so that they need money so that they need to work so that there are jobs in making the things that people want. If it got too easy to do middle-class work, then there’d be less for the middle class to do, which would mean less middle class spending, which would mean less demand for consumer goods, which would… Perhaps knowledge work needs to be inefficient to support its own economy.

If that’s true, will there ever be a drive toward division of labour in the field of software? Maybe, when the revolution is over and the interesting things to create for their own sake lie elsewhere. When computers are no longer novel technology, but are simply a substrate of industry. When the costs and benefits are sufficiently well-understood that business owners can know what they need, when it’s done, and how much it should cost.

That’ll probably be preceded by another software-led recession, as VCs realise that the startups they’re funding are no longer interesting for their own sake and move on to the (initially much smaller) nascent field of the next revolution. Along with that will come the changes in regulation, liability and insurance as businesses accept that software should “just work” and policy adjusts to support that. With stability comes the drive to reduce costs and increase quality and rate of output, and with that comes the division of labour predicted by Adam Smith.

posted by Graham at 13:39  

Monday, May 26, 2014

The lighter side of open source

In a recent post I talked about the apolitical, amoral nature of open source software and how it puts the interests of a small programming class before the interests of the broad collection of people who interact with programmers’ output. The open source movement has been of great benefit to the software industry, and this hasn’t necessarily been a zero-sum game.

Reality is always more nuanced than history, and yet here is a potted guide to open source history. In the beginning, there were military computers. There was no-one else to share your computer programs with, because:

  1. no-one else had a computer.
  2. well, maybe they did, but they weren’t telling you.
  3. you didn’t want to tell anyone else you have a computer.

Then there were academic computers. Now you do want to share your programs with everyone, and they share theirs with you and so everyone is on the cutting edge.

Then there were commercial computer companies (I told you this history would lack nuance), who were happy to share their programs with you because it meant you could get more out of the computers they were selling.

Then there were commercial computer companies who decided that the source code to the programs used to interface with their hardware were their competitive advantage, and decided to stop sharing them. This made an academic (Richard Meriadoc (humour me) Stallman) sad, and so he created the Free Software movement to:

  1. promote sharing of software over not sharing software;
  2. subvert the copyright system usually used to restrict sharing to enable sharing.

Then there were people who wanted to use Free Software in their day jobs but found that the movement was considered too idealogical to be palatable to management, so they rebranded it Open Source Software to re-frame the discussion along business, rather than political, lines.

This is about the point when your protagonist enters, stage right. The dot-com bubble was imploding, leading to changed fortunes for all sorts of people and organisations in the software industry. Everything I would do regarding professional computing depended in some way on the GNU project and the Free Software Foundation:

  1. I learned Unix, thanks to the ability to inexpensively run GNU/Linux on my desktop computer.
  2. The things I learned about Unix, C programming and so on were portable to various platforms beyond GNU/Linux, thanks to the GNU compiler collection, GNU bash, GNU make, GNU debugger and others.
  3. One such platform was Mac OS X, the new hotness from Apple. This was a technology acquired through the purchase of NeXT, who had been able to provide a complete programming environment despite their small size and (comparatively) small budget by wrapping the tools listed above.

Somewhere in all the above I even found it possible to get paid for writing software: a GPLv2-licensed Lisp package for GNU Emacs.

Of course, that’s just my story, but there are plenty like it. Many other programmers work on platforms like iOS, or Android, or Linux, or in environments like Ruby or Objective-C, that either only exist or have only become as successful as they have due to the successes of the Free Software Foundation, and the ability for organisations (commercial or otherwise) to take advantage of Free or Open Source software as building blocks which they can combine or add to.

Since then, the discussion has again been re-framed. Open Source – originally a branding change to make Free Software acceptable to business – has become a principle rather than a tool. A community that owes its financial viability to Free Software now denounces such “viral” licences, as source released under their conditions is harder to profit from than the more permissive, university-style Open Source licences.

Software writers in the 1980s liked to talk about how object technology would be the silver bullet that allowed re-use and composition of software systems, moving programming from a cottage industry where everyone makes everything from scratch to a production-line enterprise where standard parts fit together to provide a base for valuable products. It wasn’t; the sharing-required software licence was.

posted by Graham at 22:48  

Wednesday, April 23, 2014

Code longevity

I recently wrote about the impending centenary of applied computing; a time when we could reflect on the first hundred years to make it easier for people to progress beyond our position into the second hundred years. This necessitates looking at the things we’ve tried, the things that succeeded and the things that failed. It involves recalling and describing the good ideas and the bad ideas.

So, did the bad ideas fail and the good ideas succeed? Can we declare that because something worked, it must have been a success? Is length of service a great proxy for quality of principle?

Let’s start by looking at the lifetime of some of the trappings of applied computing. I’m writing this on the smartphone shown in the picture below. It is, among the many computers I own that claim to be computers and could reasonably be described as modern, one of only two that is not running a recent variant of a minicomputer game–loading system.

Surface RT and Lumia 925

Now is that a fair assessment? Certainly all the Macs, iOSes, Androids (and even routers and television streamy box things) in the house are based on Unix, and Unix is the thing of the 1970s minicomputer. I’ve even used that idea to explain why we still have to deal with PDP-8 problems in iPhones. But is it fair to assume that because the name has lasted, then the idea has been preserved? Did Unix succeed, or has it been replaced by different things with the same name? That happens a lot; is today’s ethernet really the same ethernet that Bob Metcalfe and colleagues at PARC invented? Conversely, just because the name changed is everything new? Does Windows NT really represent a clean break in 1993?

There’s certainly some core, a kernel (f’nar) of the modern Unix that, whether in code or philosophy, can be traced back to the original system (and indeed beyond). But is that there because it’s still a good idea, or because there’s no impetus to remove it? Or even because it’s a bad idea, but removing it would be expensive?

As we’re already talking about Unix, let’s talk about C. In his talk Null References: The Billion-Dollar Mistake, Tony Hoare describes his own mistake as being the introduction of a null reference. He then says that C’s mistake (C follows Algol in having null references, but it also lacks have subscript bounds checking) is an order of magnitude worse. In fact, Hoare also identified a third problem: he says that it’s a good idea to permit a program failure to be diagnosed just from the error message and the high-level program source text. However, runtime failures in C usually end up with a core dump and/or a stack trace through the instructions of the target machine environment.

We can easily wonder just how much (expensive) programmer time has been lost disassembling stack traces, matching up debugger symbols and interpreting core dumps, but without figures for that I’ll generously assume that it’s an order of magnitude smaller than the losses due to buffer overflows. Now that’s only a tens-of-billions-of-dollars value of mistake, and C is the substrate for trillions of dollars of value of industry. So do we say that on balance, C is 99% a Good Thing™? Is it a bad idea that nonetheless enabled plenty of good ones?

[Incidentally, and without wanting to derail the central thesis of this post, I disagree with Hoare’s numbers. Symantec is merely one of the largest companies in the information security sector, with annual revenue in their most recent report of $6.9B. That’s a small part of the total value sunk into that sector, which I’ll guess has an annual magnitude of multiple tens of billions. A large fraction of the problems addressed by infosec can be attributed to C’s lack of bounds checking, so that there’s probably just an annual impact of around ten billion dollars working on fixing the problem. Assuming those businesses have sustainable revenues over multiple years, the integrated cost is well into the hundreds of billions. That only revises the estimated impact on the C software industry from ‘fractions of a per cent’ to ‘a per cent’ though.]

Perhaps it’s fair to say that C was a good idea when it arose, and that it’s since been found to have deficiencies that haven’t yet become expensive enough to warrant decommissioning it. There’s an assumption of rational action in there that I think it’s fair to question, though: am I assuming that C is not worth replacing just because it has not been replaced? Might there actually be other factors involved?

Yes, there might. It’s possible that there are organisations out there for whom C is more expensive than its worth, but where the sunk cost fallacy stops them from moving on. Or organisations who stick with C because their platform vendor gives them a C toolset, even where free or paid alternatives would be cheaper [in fact that would point to a difficulty with any holistic evaluation: that the cost to the people who provide development environments and the cost to the people who consume development environments depends on different factors, and the power in the market is biased towards a few large providers. Welcome to economics]. Or organisations who stick with C because of a perception of a large community of users, which is (perceived to be) more useful than striking out alone with better tools.

It’s also possible that moves in the other direction are based on non-rational factors: organisations that seek novelty rather than improvement, or who move away from C because a vendor convinces them that their alternative is better regardless of objective truth.

It turns out that the simple question we wanted to ask about applied computing: “What works?” leads to such a complex and maybe even chaotic system of forces acting in multiple dimensions that answering it will be very difficult. This doesn’t mean that an answer should not be sought, but that finding the answer will combine expertise from many different fields. Particularly, something that survives for a long time doesn’t necessarily work: it could just be that people are afraid of the alternatives, or haven’t really considered them.

posted by Graham at 16:11  

Powered by WordPress