APPosite Concerns

I’ve started another book project: APPosite Concerns is in the same series as, and is somehow a sequel to, APPropriate Behaviour. So now I just have one question to ask.

What is going to be in the book?

This question is easy to answer in broad terms. My mental conception of who I am and how I make software is undergoing a Narsil-like transformation: it has been broken and is currently being remade.

APPropriate Behaviour was a result of the build-up of stresses that led to it being broken. As I became less and less satisfied with the way in which I made software, I explored higher and higher levels looking for meaning. Finally, I was asking the (pseudo-)profound questions: what is behind software development? How does one philosophise about it? What does it mean?

If APPropriate Behaviour is the ascent, then APPosite Concerns is an exploration of the peak. It’s an exploration of what we find when nothing is worth believing in, of the questions we ask when there is really no understanding of what the answers might be.

It’s clear to me that plenty of the essays in this blog are relevant to this exploration, but of course there’s not much point writing a book that’s just some articles culled from my blog. There needs to be, if you’ll excuse a trip into the world of self-important businessperson vocabulary for a second, some value add.

I’ve written loads recently. As I said right here, I write a lot at the moment. I write to get ideas out of my brain, so that I can ignore them and move on to other ideas. Or so that I can get to sleep. I write on a 1950s typewriter, I write on loose leaf paper, I write in notebooks, I write in Markdown files.

I know that there’ll be plenty in there that can be put to good use, but which pieces are the valuable ones? Is it the fictionalised autobiography, written in the style of a Victorian novel? The submitted-and-rejected science fiction short about the future of the United Nations? The typewritten screed about the difficulties of iOS provisioning? The Platonic dialogue on the ethics of writing software?

One thing that’s evident is that a reorganisation is required. Blogs proceed temporally, but books can take on any other order. The disparate essays from my collection are related: indeed given the same emotional state, any given subject trigger leads me to the same collection of thoughts. I could probably recreate any of the articles in SICPers not from memory, but from the same initial conditions. There’s a consistent, though evidently evolving, worldview expressed in my recent writing. Connecting the various parts conceptually will be useful for both of us.

[By the way, there will eventually be a third part representing the descent: that part has in a very real sense not yet been written.]

Intra-curricular activities

I’m apparently fascinated by the idea of defining curricula for learning programming. I’ve written about how we need to be careful what we try to pay forward from the way we learned in the past, and I’ve talked about how we do need to pay it forward so that the second hundred years see faster progress than the first hundred years.

I’m a fan (with reservations, as seen below) of the book series as a form of curriculum. Take something like Kent Beck’s signature series, which covers a decent subset of both technical and social approaches in software development in breadth and in depth. You could probably imagine developers who would benefit from reading some or all of the books in the series. In fact, you may be one.

Coping with people approaching the curriculum from different skill levels and areas of experience is hard. Not just for the book series, it’s hard in general. Universities take the simplifying approach of assuming that everybody wants to learn the same stuff, and teaching that stuff. And to some extent that’s easy for them, because the backgrounds of prospective students is relatively uniform. Even so, my University course organised incoming students into two groups; those who had studied complex numbers at A-level and those who had not. The difference was simply that the group who had not were given a couple of lectures on complex numbers, then it was assumed that they also knew the topic from the fourth week.

Now consider selling a programming book to the public. Part of the proposal process with all of the publishers I’ve worked with has been describing the target audience. Is this a book for people who have never programmed before? For people who have programmed a little, but never used this particular tool or technique? People who have programmed a lot but never used this tool? Is this thing similar to what they have used before, or very different? For people who are somewhat familiar with the tool? For experts (and how is that defined)? Is it for readers comfortable with maths? For readers with no maths background?

Every “no” in answer to one of those questions is an opportunity to improve the experience for a subset of the potential audience by tailoring it to that subset. It’s also an opportunity to exclude a subset of the audience by making the content less relevant to them.

[I’ll digress here to explain how I worked that out for my books: whether it’s selfishness or a failure of empathy, I wrote books that I wanted to read but that didn’t exist. Therefore the expected experience is something similar to mine, back when I filled in the proposal form.]

Clearly no single publication will cover the whole phase space of potential readers and be any good. The interesting question is how much it’s worth covering with multiple publications; whether the idea of series-as-curriculum pulls in the general direction as much as scope-limiting each book pulls in the specific. Should the curriculum take readers on a straight line from novice to master? Should it “fan in” from multiple introductions? Should it “fan out” in multiple directions of interest and enquiry? Would a non-linear curriculum be inclusive or offputtingly confusing? Should the questions really be answered by substituting the different question “how many people would buy that”?

Preparing for Computing’s Big One-Oh-Oh

However you slice the pie, we’re between two and three decades away from the centenary celebration for applied computing (which is of course significantly after theoretical or hypothetical advances made by the likes of Lovelace, Turing and others). You might count the anniversary of Colossus in 2043, the ENIAC in 2046, or maybe something earlier (and arguably not actually applied) like the Z3 or ABC (both 2041). Whichever one you pick, it’s not far off.

That means that the time to start organising the handover from the first century’s programmers to the second is now, or perhaps a little earlier. You can see the period from the 1940s to around 1980 as a time of discovery, when people invented new ways of building and applying computers because they could, and because there were no old ways yet. The next three and a half decades—a period longer than my life—has been a period of rediscovery, in which a small number of practices have become entrenched and people occasionally find existing, but forgotten, tools and techniques to add to their arsenal, and incrementally advance the entrenched ones.

My suggestion is that the next few decades be a period of uncovery, in which we purposefully seek out those things that have been tried, and tell the stories of how they are:

  • successful because they work;
  • successful because they are well-marketed;
  • successful because they were already deployed before the problems were understood;
  • abandoned because they don’t work;
  • abandoned because they are hard;
  • abandoned because they are misunderstood;
  • abandoned because something else failed while we were trying them.

I imagine a multi-volume book✽, one that is to the art of computer programming as The Art Of Computer Programming is to the mechanics of executing algorithms on a machine. Such a book✽ would be mostly a guide, partly a history, with some, all or more of the following properties:

  • not tied to any platform, technology or other fleeting artefact, though with examples where appropriate (perhaps in a platform invented for the purpose, as MIX, Smalltalk, BBC BASIC and Oberon all were)
  • informed both by academic inquiry and practical experience
  • more accessible than the Software Engineering Body of Knowledge
  • as accepting of multiple dissenting views as Ward’s Wiki
  • at least as honest about our failures as The Mythical Man-Month
  • at least as proud of our successes as The Clean Coder
  • more popular than The Celestial Homecare Omnibus

As TAOCP is a survey of algorithms, so this book✽ would be a survey of techniques, practices and modes of thought. As this century’s programmer can go to TAOCP to compare algorithms and data structures for solving small-scale problems then use selected algorithms and data structures in their own work, so next century’s applier of computing could go to this book✽ to compare techniques and ways of reasoning about problems in computing then use selected techniques and reasons in their own work. Few people would read such a thing from cover to cover. But many would have it to hand, and would be able to get on with the work of invention without having to rewrite all of Doug Engelbart’s work before they could get to the new stuff.

It's dangerous to go alone! Take this.

✽: don’t get hung up on the idea that a book is a collection of quires of some pigmented flat organic matter bound into a codex, though.

The code you wrote six months ago

We have this trope in programming that you should hate the code you wrote six months ago. This is a figurative way of saying that you should be constantly learning and assimilating new ideas, so that you can look at what you were doing earlier this year and have new ways of doing it.

It would be more accurate, though less visceral, to say “you should be proud that the code you wrote six months ago was the best you could do with the knowledge you then had, and should be able to ways to improve upon it with the learning you’ve accomplished since then”. If you actually hate the code, well, that suggests that you think anyone who doesn’t have the knowledge you have now is an idiot. That kind of mentality is actually deleterious to learning, because you’re not going to listen to anyone for whom you have Set the Bozo Bit, including your younger self.

I wrote a lot about learning and teaching in APPropriate Behaviour, and thinking about that motivates me to scale this question up a bit. Never mind my code, how can we ensure that any programmer working today can look at the code I was writing six months ago and identify points for improvement? How can we ensure that I can look at the code any other programmer was working on six months ago, and identify points for improvement?

My suggestion is that programmers should know (or, given the existence of the internet, know how to use the index of) the problems that have already come before, how we solved them, and why particular solutions were taken. Reflecting back on my own career I find a lot of problems I introduced by not knowing things that had already been solved: it wasn’t until about 2008 that I really understood automated testing, a topic that was already being discussed back in 1968. Object-oriented analysis didn’t really click for me until later, even though Alan Kay and a lot of really other clever people had been working on it for decades. We’ll leave discussion of parallel programming aside for the moment.

So perhaps I’m talking about building, disseminating and updating a shared body of knowledge. The building part already been done, but I’m not sure I’ve ever met anyone who’s read the whole SWEBOK or referred to any part of it in their own writing or presentations so we’ll call the dissemination part a failure.

Actually, as I said we only really need an index, not the whole BOK itself: these do exist for various parts of the programming endeavour. Well, maybe not indices so much as catalogues; summaries of the state of the art occasionally with helpful references back to the primary material. Some of them are even considered “standards”, in that they are the go-to places for the information they catalogue:

  • If you want an algorithm, you probably want The Art of Computer Programming or Numerical Recipes. Difficulties: you probably won’t understand what’s written in there (the latter book in particular assumes a bunch of degree-level maths).
  • If you want idioms for your language, look for a catalogue called “Effective <name of your language>”. Difficulty: some people will disagree with the content here just to be contrary.
  • If you want a pattern, well! Have we got a catalogue for you! In fact, have we got more catalogues than distinct patterns! There’s the Gang of Four book, the PloP series, and more. If you want a catalogue that looks like it’s about patterns but is actually comprised of random internet commentators trying to prove they know more than Alastair Cockburn, you could try out the Portland Pattern Repository. Difficulty: you probably won’t know what you’re looking for until you’ve already read it—and a load of other stuff.

I’ve already discussed how conference talks are a double-edged sword when it comes to knowledge sharing: they reach a small fraction of the practitioners, take information from an even smaller fraction, and typically set up a subculture with its own values distinct from programming in the large. The same goes for company-internal knowledge sharing programs. I know a few companies that run such programs (we do where I work, and Etsy publish the talks from theirs). They’re great for promoting research, learning and sharing within the company, but you’re always aware that you’re not necessarily discovering things from without.

So I consider this one of the great unsolved problems in programming at the moment. In fact, let me express it as two distinct questions:

  1. How do I make sure that I am not reinventing wheels, solving problems that no longer need solving or making mistakes that have already been fixed?
  2. A new (and for sake of this discussion) inexperienced programmer joins my team. How do I help this person understand the problems that have already been solved, the mistakes that have already been made, and the wheels that have already been invented?

Solve this, and there are only two things left to do: fix concurrency, name things, and improve bounds checking.

Meta-writing

Barely 4,000 years ago, documents were written on heavy, clay tablets. The Epic of Gilgamesh, one of the earliest known works of fiction, was written on 11 such tablets with a 12th added later. There was only one thing you could do with these tablets: read. Fast forward to the 21-st century and things are very different. The word “tablet” has taken on a new meaning, and documents can be delivered wirelessly, updated as new versions are written. They can also contain rich media and hyperlinked references to other content. And with these new capabilities come new considerations when preparing your documents—or “docs”—for your readers.

The above story seems rambling and pointless, doesn’t it? But change the timescale and the technology, and every single bloody report on mobile technology starts in exactly the same way.

APPropriate Behaviour is complete!

APPropriate Behaviour, the book on things programmers do that aren’t programming, is now complete! The final chapter – a philosophy of software making – has been added, concluding the book.

Just because it’s complete, doesn’t mean it’s finished: as my understanding of what we do develops I’ll probably want to correct things, or add new anecdotes or ideas. Readers of the book automatically get free updates whenever I create them in the future, so I hope that this is a book that grows with us.

As ever, the introduction to the book has instructions on joining the book’s Glassboard to discuss the content or omissions from the content. I look forward to reading what you have to say about the book in the Glassboard.

While the recommended purchase price of APPropriate Behaviour is $20, the minimum price now that it’s complete is just $10. Looking at the prices paid by the 107 readers who bought it while it was still being written, $10 is below the median price (so most people chose to pay more than $10) and the modal price (so the most common price chosen by readers was higher than $10).

A little about writing the book: I had created the outline of the book last Summer, while thinking about the things I believed should’ve been mentioned in Code Complete but were missing. I finally decided that it actually deserved to be written toward the end of the year, and used National Novel Writing Month as an excuse to start on the draft. A sizeable portion of the draft typescript was created in that month; enough to upload to LeanPub and start getting feedback on from early readers. I really appreciate the help and input those early readers, along with other people I’ve talked to the material about, have given both in preparing APPropriate Behaviour and in understanding my career and our industry.

Over the next few months, I tidied up that first draft, added new chapters, and extended the existing material. The end result – the 11th release including that first draft – is 141 pages of reflection over the decade in which I’ve been paid to make software: not a long time, but still nearly 15% of the sector’s total lifespan. I invite you to grab a copy from LeanPub and share in my reflections on that decade, and consider what should happen in the next.

APPropriate Behaviour is almost done

I just pushed another update to APPropriate Behaviour, my work on the things programmers do that aren’t programming. There’s some refinement to the existing material to be done, and a couple of short extra chapters to finish and add. But then it will be complete!

The recommended price of APPropriate Behaviour is $20. While it’s been under development, I’ve allowed readers interested in a sneak peak to buy APPropriate Behaviour at any price above $5. Once the final chapters are in place, the recommended price will remain $20 but the minimum price will be increasing. If you’ve been pondering buying it but haven’t yet, I recommend you do so now to get a bargain. Even if you buy it while I’m still working on it, you’ll get free updates for life as I add new material and make corrections.

As a little taster of things to come, the two remaining chapters are:

  • The ethics of making software
  • The philosophy of making software

Can’t wait to see what that means? Neither can I!

I just updated Appropriate Behaviour

The new release of Appropriate Behaviour—the book about things programmers should do that aren’t programming—is now up. The most obvious, and most awesome, change in this update is a fabulous new cover, designed by Sebastian Hermida of leanpubcovers.com. Should you be in the market for a cover page, I’d strongly recommend him.

Other changes in this release include additions to the (ends of the) chapters on coding practices and learning, and I’ve added part of a new chapter on requirements engineering. As ever, discussion of the book is welcome in its glassboard, details of which are in the introduction.

I’ve found it really interesting in researching this book that I can go back decades and find information that has either been forgotten, or was seemingly ignored even at the time of publication. I think it’s quite clear that there’s a gulf between software as practiced by people who make software, and software as researched by academics; it’s therefore not surprising to see journal articles that apparently never got read by commercial sector developers.

What is more interesting is the extent to which “mainstream” programming books, including ones that apparently made a big splash at the time of their publication, no longer seem relevant. They’ve either been completely dropped from our consciousness (hands up everyone who’s read Peopleware in the last five years), or have been adapted into a one-sentence précis that’s become part of the mythology of programming. A thought experiment by way of an example of this mythologising: quote any sentence from The Mythical Man Month except the one about adding people to a late project. What was the rest of the book about? Is anything else in it relevant to what we do today? Do we know that even that sentence is relevant, or does it just sound plausible?

I’ve been having lots of fun discovering these forgotten entries in our history and bringing some of them into a modern story about programming. But Appropriate Behaviour is not a history book; if anything, it’s a book on social anthropology. The lesson to learn from this post is that it’s not the first anthropological study of programmers; I’d argue that 1971’s The Psychology of Computer Programming is more anthropology than it is psychology. It’s very different from Appropriate Behaviour but they both tread the same ground, analysing the problems faced by a programmer that aren’t directly related to telling a computer what to do.

I imagine the history book would be fun to write, though for the moment I present this, which I hope is also fun to read.

Does the history of making software exist?

A bit of a repeated theme in the construction of APPropriate Behaviour has been that I’ve tried to position certain terms or concepts in their historical context, and found it difficult, or impossible to do so with sufficient rigour. There’s an extent to which I don’t want the book to become historiographical so have avoided going too deep into that angle, but have discovered that either no-one else has or that if they have, I can’t find their work.

What often happens is that I can find a history or even many histories, but that these aren’t reliable. I already wrote in the last post on this blog about the difficulties in interpreting references to the 1968 NATO conference; well today I read another two sources that have another two descriptions of the conference and how it kicked off the software crisis. Articles like that linked in the above post help to defuse some of the myths and partisan histories, but only in very specific domains such as the term “software crisis”.

Occasionally I discover a history that has been completely falsified, such as the great sequence of research papers that “prove” how some programmers are ten (or 25, or 1000) times more productive than others or those that “prove” bugs cost 100x more to fix in maintenance. Again, it’s possible to find specific deconstructions of these memes (mainly by reading Laurent Bossavit), but having discovered the emperor is naked, we have no replacement garments with which to clothe him.

There are a very few subjects where I think the primary and secondary literature necessary to construct a history exist, but that I lack the expertise or, frankly, the patience to pursue it. For example you could write a history of the phrase “software engineering”, and how it was introduced to suggest a professionalism beyond the craft discipline that went before it, only to become a symbol of lumbering lethargy among adherents of the craft discipline that came after it. Such a thing might take a trained historian armed with a good set of library cards a few years to complete (the book The Computer Boys Take Over covers part of this story, though it is written for the lay reader and not the software builder). But what of more technical ideas? Where is the history of “Object-Oriented”? Does that phrase mean the same thing in 2013 as in 1983? Does it even mean the same thing to different people in 2013?

Of course there is no such thing as an objective history. A history is an interpretation of a collection of sources, which are themselves interpretations drawn from biased or otherwise limited fonts of knowledge. The thing about a good history is that it lets you see what’s behind the curtain. The sources used will all be listed, so you can decide whether they lead you to the same conclusions as the author. It concerns me that we either don’t have, or I don’t have access to, resources allowing us to situate what we’re trying to do today in the narrative of everything that has gone before and will go hence. That we operate in a field full of hype and innuendo, and lack the tools to detect Humpty Dumptyism and other revisionist rhetoric.

With all that said, are the histories of the software industry out there? I don’t mean the collectors like the museums, who do an important job but not the one I’m interested in here. I mean the histories that help us understand our own work. Do degrees in computer science, to the extent they consider “real world” software making at all, teach the history of the discipline? Not the “assemblers were invented in 1949 and the first binary tree was coded in 19xy” history, but the rise and fall of various techniques, fads, disciplines, and so on? Or have I just volunteered for another crazy project?

I hope not, I haven’t got a good track record at remembering my library cards. Answers on a tweet, please.

An observation designed to aid the reading of books on software

Wherever a book on writing software describes the 1968 NATO conference in Garmisch on Software Engineering, consider whether the clarity of the argument can be improved by adding the following parenthetical clause:

[…], a straw man version of an otherwise real conference that took place in 1968, […]

Usually it can. The proceedings of the conference, which were written post facto by the editors and typists locking themselves in a hotel room with tapes of the sessions and typewriters in various states of repair, are available at Brian Randell’s website along with reflections on their creation. Does the report actually contain the fact presented in whichever book you’re reading now?

Probably not. The article “Crisis, What Crisis?” Reconsidering the Software Crisis of the 1960s and the Origins of Software Engineering investigates the position of the 1968 report in the rhetoric of the software industry and reliance by secondary authors on its content. The conclusion is that the report was largely ignored for about a decade, when it suddenly became the thing that kickstarted the software crisis and software engineering.

It would only be a little satirical to say “the software crisis was invented circa 1980 by Edsger Dijkstra, who postulated its origins in the NATO conference of 1968, a straw man conference” etc.