Software, Science?

Is there any science in software making? Does it make sense to think of software making as scientific? Would it help if we could?

Hold on, just what is science anyway?

Good question. The medieval French philosopher-monk Buridan said that the source of all knowledge is experience, and Richard Feynman paraphrased this as “the test of all knowledge is experiment”.

If we accept that science involves some experimental test of knowledge, then some of our friends who think of themselves as scientists will find themselves excluded. Consider the astrophysicists and cosmologists. It’s all very well having a theory about how stars form, but if you’re not willing to swirl a big cloud of hydrogen, helium, lithium, carbon etc. about and watch what happens to it for a few billion years then you’re not doing the experiment. If you’re not doing the experiment then you’re not a scientist.

Our friends in what are called the biological and medical sciences also have some difficulty now. A lot of what they do is tested by experiment, but some of the experiments are not permitted on ethical grounds. If you’re not allowed to do the experiment, maybe you’re not a real scientist.

Another formulation (OK, I got this from the wikipedia entry on Science) sees science as a sort of systematic storytelling: the making of “testable explanations and predictions about the universe”.

Under this definition, there’s no problem with calling astronomy a science: you think this is how things work, then you sit, and watch, and see whether that happens.

Of course a lot of other work fits into the category now, too. There’s no problem with seeing the “social sciences” as branches of science: if you can explain how people work, and make predictions that can (in principle, even if not in practice) be tested, then you’re doing science. Psychology, sociology, economics: these are all sciences now.

Speaking of the social sciences, we must remember that science itself is a social activity, and that the way it’s performed is really defined as the explicit and implicit rules and boundaries created by all the people who are doing it. As an example, consider falsificationism. This is the idea that a good scientific hypothesis is one that can be rejected, rather than confirmed, by an appropriately-designed experiment.

Sounds pretty good, right? Here’s the interesting thing: it’s also pretty new. It was mostly popularised by Karl Popper in the 20th Century. So if falsificationism is the hallmark of good science, then Einstein didn’t do good science, nor did Marie Curie, nor Galileo, or a whole load of other people who didn’t share the philosophy. Just like Dante’s Virgil was not permitted into heaven because he’d been born before Christ and therefore could not be a Christian, so all of the good souls of classical science are not permitted to be scientists because they did not benefit from Popper’s good message.

So what is science today is not necessarily science tomorrow, and there’s a sort of self-perpetuation of a culture of science that defines what it is. And of course that culture can be critiqued. Why is peer review appropriate? Why do the benefits outweigh the politics, the gazumping, the gender bias? Why should it be that if falsification is important, negative results are less likely to be published?

Let’s talk about Physics

Around a decade ago I was studying particle physics pretty hard. Now there are plenty of interesting aspects to particle physics. Firstly that it’s a statistics-heavy discipline, and that results in statistics are defined by how happy you are with them, not by some binary right/wrong criterion.

It turns out that particle physicists are a pretty conservative bunch. They’ll only accept a particle as “discovered” if the signal indicating its existence is measured as a five-sigma confidence: i.e. if there’s under a one-on-a-million chance that the signal arose randomly in the absence of the particle’s existence. Why five sigma? Why not three (a 99.7% confidence) or six (to keep Motorola happy)? Why not repeat it three times and call it good, like we did in middle school science classes?

Also, it’s quite a specialised discipline, with a clear split between theory and practice and other divisions inside those fields. It’s been a long time since you could be a general particle physicist, and even longer since you could be simply a “physicist”. The split leads to some interesting questions about the definition of science again: if you make a prediction which you know can’t be verified during your lifetime due to the lag between theory and experimental capability, are you still doing science? Does it matter whether you are or not? Is the science in the theory (the verifiable, or falsifiable, prediction) or in the experiment? Or in both?

And how about Psychology, too

Physicists are among the most rational people I’ve worked with, but psychologists up the game by bringing their own feature to the mix: hypercriticality. And I mean that in the technical sense of criticism, not in the programmer “you’re grammar sucks” sense.

You see, psychology is hard, because people are messy. Physics is easy: the apple either fell to earth or it didn’t. Granted, quantum gets a bit weird, but it generally (probably) does its repeatable thing. We saw above that particle physics is based on statistics (as is semiconductor physics, as it happens); but you can still say that you expect some particular outcome or distribution of outcomes with some level of confidence. People aren’t so friendly. I mean, they’re friendly, but not in a scientific sense. You can do a nice repeatable psychology experiment in the lab, but only by removing so many confounding variables that it’s doubtful the results would carry over into the real world. And the experiment only really told you anything about local first year psychology undergraduates, because they’re the only people who:

  1. walked past the sign in the psychology department advertising the need for participants;
  2. need the ten dollars on offer for participation desperately enough to turn up.

In fact, you only really know about how local first year psychology undergraduates who know they’re participating in a psychology experiment behave. The ethics rules require informed consent which is a good thing because otherwise it’s hard to tell the difference between a psychology lab and a Channel 4 game show. But it means you have to say things like “hey this is totally an experiment and there’ll be counselling afterward if you’re disturbed by not really electrocuting the fake person behind the wall” which might affect how people react, except we don’t really know because we’re not allowed to do that experiment.

On the other hand, you can make observations about the real world, and draw conclusions from them, but it’s hard to know what caused what you saw because there are so many things going on. It’s pretty hard to re-run the entire of a society with just one thing changed, like “maybe if we just made Hitler an inch taller then Americans would like him, or perhaps try the exact same thing as prohibition again but in Indonesia” and other ideas that belong in Philip K. Dick novels.

So there’s this tension: repeatable results that might not apply to the real world (a lack of “ecological validity”), and real-world phenomena that might not be possible to explain (a lack of “internal validity”). And then there are all sorts of other problems too, so that psychologists know that for a study to hold water they need to surround what they say with caveats and limitations. Thus is born the “threats to validity” section on any paper, where the authors themselves describe the generality (or otherwise) of their results, knowing that such threats will be a hot topic of discussion.

But all of this—the physics, the psychology, and the other sciences—is basically a systematised story-telling exercise, in which the story is “this is why the universe is as it is” and the system is the collection of (time-and-space-dependent) rules that govern what stories may be told. It’s like religion, but with more maths (unless your religion is one of those ones that assigns numbers to each letter in a really long book then notices that your phone number appears about as many times as a Poisson distribution would suggest).

Wait, I think you were talking about software

Oh yeah, thanks. So, what science, if any, is there in making software? Does there exist a systematic approach to storytelling? First, let’s look at the kinds of stories we need to tell.

The first are the stories about the social system in which the software finds itself: the story of the users, their need (or otherwise) for a software system, their reactions (or otherwise) to the system introduced, how their interactions with each other change as a result of introducing the system, and so on. Call this requirements engineering, or human-computer interaction, or user experience; it’s one collection of stories.

You can see these kinds of stories emerging from the work of Manny Lehman. He identifies three types of software:

  • an S-system is exactly specified.
  • a P-system executes some known procedure.
  • an E-system must evolve to meet the needs of its environment.

It may seem that E-type software is the type in which our stories about society are relevant, but think again: why do we need software to match a specification, or to follow a procedure? Because automating that specification or procedure is of value to someone. Why, or to what end? Why that procedure? What is the impact of automating it? We’re back to telling stories about society. All of these software systems, according to Lehman, arise from discovery of a problem in the universe of discourse, and provide a solution that is of possible interest in the universe of discourse.

The second kind are the stories about how we worked together to build the software we thought was needed in the first stories. The practices we use to design, build and test our software are all tools for facilitating the interaction between the people who work together to make the things that come out. The things we learned about our own society, and that we hope we can repeat (or avoid) in the future, become our design, architecture, development, testing, deployment, maintenance and support practices. We even create our own tools—software for software’s sake—to automate, ease or disrupt our own interactions with each other.

You’ll mostly hear the second kind of story at most developer conferences. I believe that’s because the people who have most time and inclination to speak at most developer conferences are consultants, and they understand the second stories to a greater extent than the first because they don’t spend too long solving any particular problem. It’s also because most developer conferences are generally about making software, not about whatever problem it is that each of the attendees is trying to solve out in the world.

I’m going to borrow a convention that Rob Rix told me in an email, of labelling the first type of story as being about “external quality” and the second type about “internal quality”. I went through a few stages of acceptance of this taxonomy:

  1. Sounds like a great idea! There really are two different things we have to worry about.
  2. Hold on, this doesn’t sounds like such a good thing. Are we really dividing our work into things we do for “us” and things we do for “them”? Labelling the non-technical identity? That sounds like a recipe for outgroup homogeneity effect.
  3. No, wait, I’m thinking about it wrong. The people who make software are not the in-group. They are the mediators: it’s just the computers and the tools on one side of the boundary, and all of society on the other side. We have, in effect, the Janus Thinker: looking on the one hand toward the external stories, on the other toward the internal stories, and providing a portal allowing flow between the two.

JANUS (from Vatican collection) by Flickr user jinnrouge

So, um, science?

What we’re actually looking at is a potential social science: there are internal stories about our interactions with each other and external stories about our interactions with society and of society’s interactions with the things we create, and those stories could potentially be systematised and we’d have a social science of sorts.

Particularly, I want to make the point that we don’t have a clinical science, an analogy drawn by those who investigate evidence-based software engineering (which has included me, in my armchair way, in the past). You can usefully give half of your patients a medicine and half a placebo, then measure survival or recovery rates after that intervention. You cannot meaningfully treat a software practice, like TDD as an example, as a clinical intervention. How do you give half of your participants a placebo TDD? How much training will you give your ‘treatment’ group, and how will you organise placebo training for the ‘control’ group? [Actually I think I’ve been on some placebo training courses.]

In constructing our own scientific stories about the world of making software, we would run into the same problems that social scientists do in finding useful compromises between internal and ecological validity. For example, the oft-cited Exploratory experimental studies comparing online and offline programming performance (by Sackman et al., 1968) is frequently used to support the notion that there are “10x programmers”, that some people who write software just do it ten times faster than others.

However, this study does not have much ecological validity. It measures debugging performance, using either an offline process (submitting jobs to a batch system) or an online debugger called TSS, which probably isn’t a lot like the tools used in debugging today. The problems were well-specified, thus removing many of the real problems programmers face in designing software. Participants were expected to code a complete solution with no compiler errors, then debug it: not all programmers work like that. And where did they get their participants from? Did they have a diverse range of backgrounds, cultures, education, experience? It does not seem that any results from that study could necessarily apply to modern software development situated in a modern environment, nor could the claim of “10x programmers” necessarily generalise as we don’t know who is 10x better than whom, even at this one restricted task.

In fact, I’m also not convinced of its internal validity. There were four conditions (two programming problems and two debugging setups), each of which was assigned to six participants. Variance is so large that most of the variables are independent of each other (the independent variables are the programming problem and the debugging mode, and the dependent variables are the amount of person-time and CPU-time), unless the authors correlate them with “programming skill”. How is this skill defined? How is it measured? Why, when the individual scores are compared, is “programming skill” not again taken into consideration? What confounding variables might also affect the wide variation in scored reported? Is it possible that the fastest programmers had simply seen the problem and solved it before? We don’t know. What we do know is that the reported 28:1 ratio between best and worst performers is across both online and offline conditions (as pointed out in, e.g., The Leprechauns of Software Engineering, so that’s definitely a confounding factor. If we just looked at two programmers using the same environment, what difference would be found?

We had the problem that “programming skill” is not well-defined when examining the Sackman et al. study, and we’ll find that this problem is one we need to overcome more generally before we can make the “testable explanations and predictions” that we seek. Let’s revisit the TDD example from earlier: my hypothesis is that a team that adopts the test-driven development practice will be more productive some time later (we’ll defer a discussion of how long) than the null condition.

OK, so what do we mean by “productive”? Lines of code delivered? Probably not, their amount varies with representation. OK, number of machine instructions delivered? Not all of those would be considered useful. Amount of ‘customer value’? What does the customer consider valuable, and how do we ensure a fair measurement of that across the two conditions? Do bugs count as a reduction in value, or a need to do additional work? Or both? How long do we wait for a bug to not be found before we declare that it doesn’t exist? How is that discovery done? Does the expense related to finding bugs stay the same in both cases, or is that a confounding variable? Is the cost associated with finding bugs counted against the value delivered? And so on.

Software dogma

Because our stories are not currently very testable, many of them arise from a dogmatic belief that some tool, or process, or mode of thought, is superior to the alternatives, and that there can be no critical debate. For example, from the Clean Coder:

The bottom line is that TDD works, and everybody needs to get over it.

No room for alternatives or improvement, just get over it. If you’re having trouble defending it, apply a liberal sprinkle of argumentum ab auctoritate and tell everyone: Robert C. Martin says you should get over it!

You’ll also find any number of applications of the thought-terminating cliché, a rhetorical technique used to stop cognitive dissonance by allowing one side of the issue to go unchallenged. Some examples:

  • “I just use the right tool for the job”—OK, I’m done defending this tool in the face of evidence. It’s just clearly the correct one. You may go about your business. Move along.
  • “My approach is pragmatic”—It may look like I’m doing the opposite of what I told you earlier, but that’s because I always do the best thing to do, so I don’t need to explain the gap.
  • “I’m passionate about [X]”—yeah, I mean your argument might look valid, I suppose, if you’re the kind of person who doesn’t love this stuff as much I do. Real craftsmen would get what I’m saying.
  • and more.

The good news is that out of such religious foundations spring the shoots of scientific thought, as people seek to find a solid justification for their dogma. So just as physics has strong spiritual connections, with Steven Hawking concluding in A Brief History of Time:

However, if we discover a complete theory, it should in time be understandable by everyone, not just by a few scientists. Then we shall all, philosophers, scientists and just ordinary people, be able to take part in the discussion of the question of why it is that we and the universe exist. If we find the answer to that, it would be the ultimate triumph of human reason — for then we should know the mind of God.

and Einstein debating whether quantum physics represented a kind of deific Dungeons and Dragons:

[…] an inner voice tells me that it is not yet the real thing. The theory says a lot, but does not really bring us any closer to the secret of the “old one.” I, at any rate, am convinced that He does not throw dice.

so a (social) science of software could arise as an analogous form of experimental theology. I think the analogy could be drawn too far: the context is not similar enough to the ages of Islamic Science or of the Enlightenment to claim that similar shifts to those would occur. You already need a fairly stable base of rational science (and its application via engineering) to even have a computer at all upon which to run software, so there’s a larger base of scientific practice and philosophy to draw on.

It’s useful, though, when talking to a programmer who presents themselves as hyper-rational, to remember to dig in and to see just where the emotions, fallacious arguments and dogmatic reasoning are presenting themselves, and to wonder about what would have to change to turn any such discussion into a verifiable prediction. And, of course, to think about whether that would necessarily be a beneficial change. Just as we can critique scientific culture, so should we critique software culture.

Principled Lizards

Sixty-five million years ago, there were many huge lizards. Most of them were really happy being lizards, and would spend all of the time they could doing lizardy things. Some wanted to be the biggest lizards, and grew so large and so heavy that it would sound like peals of thunder if you could hear them walking about on their lizardy way. Others wanted to be the most terrible lizards, and they developed big scary teeth and sharp, shiny talons. The most terrible lizards were feared by many of the other lizards, but it was a fear that sprang from awe: they were all happy that each was, in their own way, the most lizardy of the lizards. And they were all happy that each of the other lizards they met was trying to be, in their own way, the most lizardy of lizards.

For the lizards met often. They would have their big get-togethers where the big lizards and the small lizards and the terrible lizards and the scaly lizards would each talk about how they handle being so big, or so small, or so terrible, or so scaly. And the other lizards would listen to these talks, and they would applaud the speakers for being so big, or so small, or so terrible, or so scaly. Having seen these examples of lizardly apotheosis, they would try to emulate them. So it was that the lizard world became bigger, but also smaller, and more terrible, and more scaly.

But it seems that not all of the lizards shared these goals of ever-increasing lizardhood. Some would try different things. A group of lizards found that they could regulate their own blood temperature, they would no longer need to sit in the sun all morning like the other lizards. One group of lizards turned their feathery covering to the task of improved aerodynamics. Another group turned it to a sort of coat, which stopped them getting so cold.

The big meetings of lizardy lizards did not really pay these developments much notice, as they were not very lizard like changes. They knew that they were lizards! They should do the lizardy things, like getting bigger or smaller or more terrible or more scaly! They put over eighty hours a week into it, they were passionate about it. The world was, for them, all about being more lizardly every day.

Some of the things that the decidedly non-lizardlike groups were coming up with did take a sort of root among those who called themselves the “lizard community”, but only to the extent that they could be seen as lizardy things. So ideas from the feather aerodynamics group became diluted, and were called “flight-oriented lizarding”. At the big gatherings of all the lizards, the FOL evangelists would show how they had made things that looked a bit like the feathers used for aerodynamics, but which were more lizardy. They had some benefit to lizards in that they slowed them down slightly as they fell out of trees. And, of course, as this was something that you had to be able to demonstrate expert lizardly competence in, they invented the idea of the master flight-oriented lizard.

All sorts of rules were invented to demonstrate competency and master-lizardliness in the flight-oriented world. This feather and that feather must each have a single responsibility: this for slowing the fall, that for turning. Feathers must be open for falling but closed for impact. Specific types of feathers could be invented, but only where they could be used in place of the more generic feathers. Feathers had to be designed so that they never got into the area around a lizard’s eyes (the in-the-face segregation principle). Despite the fact that flight-oriented lizards only used their feathers for falling out of trees, feathers had to be designed to work when travelling upwards too (the descendency inversion principle).

But to the expert lizards—the biggest, smallest, scaliest and most terrible lizards—something felt uncomfortable. It felt like people were saying that there was something else to do than being an expert lizard, as if lizardness wasn’t enough. So, of course, they arranged another meeting of all the lizards. Expert lizards and novice lizards and improving lizards all came together, that one day sixty-five million years ago, and they met in the town of Chicxulub. And the most expert of the expert lizards got up in front of all the lizards, and said this:

If you want to carry on at lizarding you have to really love it. You’ve got to want to put every waking moment into becoming a better lizard. You’ve got to look up after practising your lizarding, and be shocked at how much time has gone past. If that isn’t you, if you don’t absolutely love everything about lizarding, perhaps it’s time to move on and do something else.

Many of the expert lizards agreed with this idea, and were pleased with themselves. But many that had been trying other things, the fur or the flying or the warm blood, were confused: did they want to be lizards forever, and strive toward the best of lizardliness, or not? Did they perhaps want to explore the opportunities presented by warm blood, or flying, or fur?

And so it was that at Chicxulub, as a rock from outer space danced through the upper atmosphere, pushing and heating and ionising the air in front of it, people chose between the many paths open to them.

A sneaky preview of ClassBrowser

Let me start with a few admissions. Firstly, I have been computering for a good long time now, and I still don’t really understand compilers. Secondly, work on my GNUstep Web side-project has tailed off for a while, because I decided I wanted to try something out to learn about the compiler before carrying on with that work. This post will mostly be about that something.

My final admission: if you saw my presentation on the ObjC runtime you will have seen an app called “ClassBrowser” where I showed that some of the Foundation classes are really in the CoreFoundation library. Well, there were two halves to the ClassBrowser window, and I only showed you the top half that looked like the Smalltalk class browser. I’m sorry.

So what’s the bottom half?

This is what the bottom half gives me. It lets me go from this:

ClassBrowser before

via this:

Who are you calling a doIt?

to this:

ClassBrowser after

What just happened?

You just saw some C source being compiled to LLVM bit code, which is compiled just-in-time to native code and executed, all inside that browser app.

Why?

Well why not? Less facetiously:

  • I’m a fan of Smalltalk. I want to build a thing that’s sort of a Smalltalk, except that rather than being the Smalltalk language on the Objective-C runtime (like F-Script or objective-smalltalk), it’ll be the Objective-C(++) language on the Objective-C runtime. So really, a different way of writing Objective-C.
  • I want to know more about how clang and LLVM work, and this is as good a way as any.
  • I think, when it actually gets off the ground, this will be a faster way of writing test-first Objective-C than anything Xcode can do. I like Xcode 5, I think it’s the better Xcode 4 that I always wanted, but there are gains to be had by going in a completely different direction. I just think that whoever strikes out in such a direction should not release their research project as a new version of Xcode :-).

Where can I get me a ClassBrowser?

You can’t, yet. There’s some necessary housekeeping that needs to be done before a first release, replacing some “research” hacks with good old-fashioned tested code and ensuring GNUstep-GUI compatibility. Once that’s done, a rough-and-nearly-ready first open source release will ensue.

Then there’s more work to be done before it’s anything like useful. Particularly while it’s possible to use it to run Objective-C, it’s far from pleasant. I’ve had some great advice from LLVM IRC on how to address that and will try to get it to happen soon.

The Ignoble Programmer

Two programmers are taking a break from their work, relaxing on a bench in the park across from their office. As they discuss their weekend plans, a group of people jog past, each carrying their laptop in a yoke around their neck and furiously typing as they go.

“Oh, there goes the Smalltalk team,” says the senior of the two programmers on the bench. “They have to do everything at run-time.”

I love jokes. And not just because they’re sometimes funny, though that helps: I certainly find I enjoy a conversation and can relax more when at least two of the people involved are having fun. When only one person is joking, it gets awkward (particularly if everyone else is from HR). But a little levity can go a long way toward disarming an unpleasant truth so that it can be discussed openly. Political leaders through the ages have taken advantage of this by appointing jesters and fools to keep them aware of intrigues in the courts: even the authors of the American bill of rights remembered the satirist before the shooter.

I also like jokes because of the thought that goes into constructing a good (or deliberately bad) one. There’s a certain kind of creativity that goes into identifying an apparently absurd connection, exactly because of the absurdity. Being able to construct a joke, and being practised at constructing jokes, means being able to see new contexts and applications for existing ideas. Welcome to the birthplace of reuse and exploring the bounds of a construct’s application: welcome to the real home of software architecture.

But there’s a problem, or at least an opportunity (or maybe just a few thousand consulting dollars to be made and a book to be written). That problem is this: everyone else puts way more effort into their jokes than programmers do. Take this one, from the scientists:

Neural Correlates of Interspecies Perspective Taking in the Post-Mortem Atlantic Salmon

They didn’t just joke about doing a brain scan of a dead fish, they did a brain scan of a dead fish. And published the (serendipitous and unexpected) results. But they didn’t just angle for a laugh, they had a real point. The subtitle of their paper:

An Argument For Proper Multiple Comparisons Correction

And isn’t it fun that some microbiologists demonstrated that beards are significant vectors for microbial infections?

Both of these examples were lifted from the Annals of Improbable Research’s Ig Nobel Prizes, awarded for “achievements that first make people laugh, and then makes them think”. The Ig Nobels have been awarded every year since 1991, and in that time only one computer science award has been granted. That award was given to the developer behind PawSense, a utility that detects and blocks typing caused by your cat walking across your keyboard.

Jokes that first make you laugh, and then make you think, are absolutely the best jokes you can make about my work. If I conclude “you’re right, that is absurd, but what if…” then you’ve done it right. Jokes that are thought-terminating statements can make us laugh, and maybe make us feel good about what we’re doing, but cannot make us any better at it because they don’t give us the impetus to reflect on our craft. Rather, they make us feel smug about knowing better than the poor sap who’s the butt of the joke. They confirm that we’ve nothing to learn, which is never the correct outlook.

We need more Ig Nobel-quality achievements in computing. Disarming the absurd and the downright broken in programming and presenting them as jokes can first make us laugh, and then make us think.

N.B. My complete connection to the Annals of Improbable Research is that I helped out on the AV desk at a couple of their talks. At their talk in Oxford in 2006 I was inducted into the Luxuriant and Flowing Hair Club for Scientists.

How to answer questions the smart way

You may have read how to ask questions the smart way by Eric S. Raymond. You may have even quoted it when faced with a question you thought was badly-formed. I want you to take a look at a section near the end of the article.

How to answer questions in a helpful way is the part I’m talking about. It’s a useful section. It reminds us that questions are part of a dialogue, which is a two-way process. Sometimes questions seem bad, but then giving bad answers is certainly no way to make up for that. What else should we know about answering questions?

The person who asked the question has had different experiences than you. The fact that you do not understand why the question should be asked does not mean that the question should not be asked. “Why would you even want to do that?” is not an answer.

Answer at a level appropriate to the question. If the question shows a familiarity with the basics, there’s no need to mansplain trivial details in the answer. On the other hand, if the question shows little familiarity with the basics, then an answer that relies on advanced knowledge is just pointless willy-waving.

The shared values that pervade your culture are learned, not innate. Not everyone has learned them yet, and they are not necessarily even good, valuable or correct. This is a point that Raymond misses with quotes like this:

You shouldn’t be offended by this; by hacker standards, your respondent is showing you a rough kind of respect simply by not ignoring you. You should instead be thankful for this grandmotherly kindness.

What this says is: this is how we’ve always treated outsiders, so this is how you should expect to be treated. Fuck that. You’re better than that. Give a respectful, courteous answer, or don’t answer. It’s really that simple. We can make a culture of respect and courtesy normative, by being respectful and courteous. We can make a culture of inclusion by not being exclusive.

I’m not saying that I’m any form of authority on answering questions. I’m far from perfect, and by exploring the flaws I know I perceive in myself and making them explicit I make them conscious, with the aim of detecting and correcting them in the future.

The whole ‘rockstar developer’ thing is backwards

Another day, another clearout of junk from people who want ‘rockstar iPhone developers’ for their Shoreditch startups. I could just say “no”, or I could launch into a detailed discussion of the problems in this picture.

Rockstars are stagnant

No-one, and I mean no-one, wants to listen to your latest album. They want you to play Free Bird, or Jessica, or Smoke on the Water. OK, so they’ll pay more for their tickets than people listening to novel indie acts, you’ll make more money from them (after your promoter has taken their 30%). But you had better use exactly the right amount of sustain in that long note in Parisienne Walkways, just like you did back in ’79, or there’ll be trouble. Your audience doesn’t care whether you’ve incorporated new styles or interesting techniques from other players, or bought new equipment, you’re playing Apache on that pink Stratocaster the way you always have.

That’s exactly the opposite of a good model in software. Solving the same problem over and over, using the same tools and techniques, is ossification. It’s redundant. No-one needs it any more. Your audience are more like New York jazz fans than VH-1 viewers: they want tradition with a twist. Yes, it needs to be recognisable that you’re solving a problem they have – that you’re riffing on a standard. But if you’re not solving new problems, you’re no longer down with the cool cats. As the rock stars might say: who wants yesterday’s papers?

Home taping is killing music

That riff you like to throw out every night, that same problem that needs solving over and over again? Some student just solved the same thing, and they put it on github. The change in code-sharing discourse of the late 1990s – from “Free Software” to “Open Source” – brought with it the ability for other people to take that solution and incorporate it into their own work with few obligations. So now everyone has a solution to that problem, and is allowed to sell it to everyone who has the problem. Tomorrow night, your stadium’s going to have plenty of empty seats.

Programming groupie culture

Programming has a very small number of big names: not many people would be as well-known in the industry as, say, Linus Torvalds, Richard Stallman, DHH. Some people might choose to call these people “polarising”. Others might choose “rude and arrogant”. Either way, they seem to bring their harems of groupies to the internet: cadres of similarly-“polarising” males who want to be seen to act in the same way as their heroes.

A primatologist might make the case that they are imitating the alpha male baboon in order to gain recognition as the highest-status beta.

Now the groupies have moved the goalposts for success from solving new problems to being rude about solutions that weren’t solved by the “in” group. What, you want to patch our software to fix a bug? You’re not from round these parts, are you?

Embrace the boffin

Somehow for the last few years I managed to hang on to the job title “Security Boffin”. Many people ask what a boffin is: the word was World War 2 slang among the British armed forces referring to the scientists working on the war effort. Like “nerd” or “geek”, it meant someone who was clever but perhaps a bit, well, different.

Boffins were also known at the time as “the back room boys”[*] for their tendency to stay out of the way and solve important – and expedient – technical problems. We need these messages decrypting, the boffins in the back room have done it but they keep talking about this “computer” thing they built. Those boffins have come up with a way to spot planes before we can even see them.

The rockstar revels in former glories while their fans insist that nothing made later even comes close to the classics. If you need a problem solving, look for boffins, not Bonos.

[*] Unfortunately in the military establishment of the 1940s it was assumed that the clever problem solvers were all boys. In fact histories of early computing in the States show that the majority-female teams who actually programmed and operated the wartime computers often knew more about the machines’ behaviours than did the back room boys, diagnosing and fixing problems without reporting them. A certain Grace Hopper, PhD, invented the compiler while the back room boys were sure computers couldn’t be used for that.

Know what counts

In Make it Count, Harry Roberts describes blacking out on stage at the end of a busy and sleepless week. Ironically, he was at the start of a talk in which he was to discuss being selective over side projects, choosing only those that you can actually “cash in” on and use to advance your career.

If you’re going to take on side projects and speaking and writing and open source and suchlike then please, make them fucking count. Do not run yourself into the ground working on ‘career moves’ if you’re not going to cash in on them. [emphasis original]

Obviously working until you collapse is not healthy. At that point, choosing which projects to accept is less important than just getting some damned sleep and putting your health back in order. In the 1950s, psychologist Abraham Maslow identified a “hierarchy of needs”, and sleep is at the base of the hierarchy meaning that, along with eating and drinking, you should take care of that before worrying about self-actualisation or esteem in the eyes of your peers.

Maslow's heirarchy, Wikipedia image

Here’s the little secret they don’t tell you in the hiring interview at Silicon Valley start-ups: you’re allowed to do things that aren’t career-centric. This includes, but is not limited to, sleeping, drinking enough water, eating non-pizza foodstuffs, having fun, seeing friends, taking breaks, and indulging in hobbies. It sometimes seems that programmers are locked in an arms race to see who can burn out first^W^W^Wdo more work than the others. That’s a short-term, economist-style view of work. I explained in APPropriate Behaviour that economists take things they don’t want to consider, or can’t work out a dollar value for, and call them “externalities” that lie outside the system.

Your health should not be an externality. Roberts attempted to internalise the “accounting” for all of his side projects by relating them in value to his career position. If you’re unhealthy, your career will suffer. So will the rest of your life. Don’t externalise your health. Worry not whether what you’re doing is good for your position in the developer community, but whether it’s good for you as a healthy individual. If you’ve got the basic things like food, shelter, sleep and safety, then validation in the eyes of yourself and your peers can follow.

What Graham did next

There’s been quite a lot of reaction to this notice on Agant’s website, that Dave is taking the company back to a one-person shop. Indeed that means that I and all of my colleagues (except Dave) are now redundant.

Sad is not the right word. I’m disappointed that this isn’t going to continue, but it’s hard to be sad about getting onto this billing:

Discworld app credits

Thanks for all your concern. I’ve already got my next thing lined up: I’m really excited to be programming and training for the Big Nerd Ranch, starting in August. I’ve known Aaron and a few other big nerds for years and I’m sure this is going to be a great opportunity for me.

In the even shorter term, though, hopefully the experience I’ve built with having more jobs than birthdays since graduation can be helpful to our other developers as we collectively discover what comes next.

Are you an [X] programmer?

On my twitter bio, I describe myself as:

a Lovelacologist for portable transactators

which is, in keeping with the way I’m dressed in the avatar pic, a steampunk way of saying that I’m a programmer of mobile computers. But is that strictly true, or fair? It’s what I’ve spent most of the last couple of years doing, but then I’ve also worked on:

  • web servers
  • SMPP servers
  • one particle accelerator
  • workstation apps
  • desktop apps
  • administration scripts
  • books

and there’s bound to be more things that I haven’t remembered. I don’t think I’m alone in picking quite a narrow definition to expose as “me” (though maybe I should have thought a bit harder before titling this blog). Social scientists refer to this as “doing identity work”, the effort we go to to control the definition of who we are in interactions with others. To confirm this choice of narrow identity work, here’s a not-quite-random look at excerpts from a few other Twitter bios (anonymised for no real reason):

  • iOS, OS X, BMWs, photography, and food.
  • App developer by day – Apple fanboy by night
  • now a Clojure and Ruby programmer
  • iOS Developer

It’s interesting that while we choose these restricted “brands” for ourselves, we actually spend a lot of time solving the same problems. I’ve been working on another web app project lately, and it’s remarkably similar to building a mobile app. Even a lot of the constraints are similar:

  • keep the event loop fast
  • avoid loading lots of large data files
  • maintain separation of concerns between modules
  • try to minimise power consumption

and indeed the solutions turn out to be similar too. The command bus introduced in an earlier post, perfect for (and, according to some readers, informing their own work in) mobile apps, was actually built for this web app project. The problems and the solutions turn out to be interchangeable.

What we need is more of this interchangeability. Rather than waiting for a mobile person to say in a mobile way how to write mobile software, we can take advantage of what people have already said in a software way about how to write software. I have resolved to widen my horizons, and pay broader attention to what my colleagues are up to.

The code you wrote six months ago

We have this trope in programming that you should hate the code you wrote six months ago. This is a figurative way of saying that you should be constantly learning and assimilating new ideas, so that you can look at what you were doing earlier this year and have new ways of doing it.

It would be more accurate, though less visceral, to say “you should be proud that the code you wrote six months ago was the best you could do with the knowledge you then had, and should be able to ways to improve upon it with the learning you’ve accomplished since then”. If you actually hate the code, well, that suggests that you think anyone who doesn’t have the knowledge you have now is an idiot. That kind of mentality is actually deleterious to learning, because you’re not going to listen to anyone for whom you have Set the Bozo Bit, including your younger self.

I wrote a lot about learning and teaching in APPropriate Behaviour, and thinking about that motivates me to scale this question up a bit. Never mind my code, how can we ensure that any programmer working today can look at the code I was writing six months ago and identify points for improvement? How can we ensure that I can look at the code any other programmer was working on six months ago, and identify points for improvement?

My suggestion is that programmers should know (or, given the existence of the internet, know how to use the index of) the problems that have already come before, how we solved them, and why particular solutions were taken. Reflecting back on my own career I find a lot of problems I introduced by not knowing things that had already been solved: it wasn’t until about 2008 that I really understood automated testing, a topic that was already being discussed back in 1968. Object-oriented analysis didn’t really click for me until later, even though Alan Kay and a lot of really other clever people had been working on it for decades. We’ll leave discussion of parallel programming aside for the moment.

So perhaps I’m talking about building, disseminating and updating a shared body of knowledge. The building part already been done, but I’m not sure I’ve ever met anyone who’s read the whole SWEBOK or referred to any part of it in their own writing or presentations so we’ll call the dissemination part a failure.

Actually, as I said we only really need an index, not the whole BOK itself: these do exist for various parts of the programming endeavour. Well, maybe not indices so much as catalogues; summaries of the state of the art occasionally with helpful references back to the primary material. Some of them are even considered “standards”, in that they are the go-to places for the information they catalogue:

  • If you want an algorithm, you probably want The Art of Computer Programming or Numerical Recipes. Difficulties: you probably won’t understand what’s written in there (the latter book in particular assumes a bunch of degree-level maths).
  • If you want idioms for your language, look for a catalogue called “Effective <name of your language>”. Difficulty: some people will disagree with the content here just to be contrary.
  • If you want a pattern, well! Have we got a catalogue for you! In fact, have we got more catalogues than distinct patterns! There’s the Gang of Four book, the PloP series, and more. If you want a catalogue that looks like it’s about patterns but is actually comprised of random internet commentators trying to prove they know more than Alastair Cockburn, you could try out the Portland Pattern Repository. Difficulty: you probably won’t know what you’re looking for until you’ve already read it—and a load of other stuff.

I’ve already discussed how conference talks are a double-edged sword when it comes to knowledge sharing: they reach a small fraction of the practitioners, take information from an even smaller fraction, and typically set up a subculture with its own values distinct from programming in the large. The same goes for company-internal knowledge sharing programs. I know a few companies that run such programs (we do where I work, and Etsy publish the talks from theirs). They’re great for promoting research, learning and sharing within the company, but you’re always aware that you’re not necessarily discovering things from without.

So I consider this one of the great unsolved problems in programming at the moment. In fact, let me express it as two distinct questions:

  1. How do I make sure that I am not reinventing wheels, solving problems that no longer need solving or making mistakes that have already been fixed?
  2. A new (and for sake of this discussion) inexperienced programmer joins my team. How do I help this person understand the problems that have already been solved, the mistakes that have already been made, and the wheels that have already been invented?

Solve this, and there are only two things left to do: fix concurrency, name things, and improve bounds checking.