Separating user interface from work

Here’s a design I’ve had knocking around my head for a while, and between a discussion we had a few weeks ago at work and Saul Mora’s excellent design patterns talk at QCon I’ve built it.

A quick heads-up: currently the logic is all built into a side project app I’ve been working on so I don’t have a single project download I can point to. The post here should explain all of the relevant code, which is made available under the terms of the MIT Licence. A reusable component is forthcoming.

Motivation

Remove the Massive View Controller from our applications’ architectures. Push Cocoa, Cocoa Touch, or other frameworks to the edges of our codebase, responsible only for working with the UI. Separate the concerns of user interaction, work scheduling and the actual work.

There are maintainability reasons for doing so. We separate unrelated work into different classes, localising the responsibilities and removing coupling between them. The same code can be used in multiple contexts, because the UI frameworks are decoupled from the work that they’re doing. This is not only a benefit for cross-platform work but for re-using the same logic in different places in a single user interface.

We also notice performance optimisations that become possible. With a clear delineation between the user interface code and the work, it’s much easier to understand which parts of the application must be run on the user interface thread and which can be done in other contexts.

Solution

Implement the Message Bus pattern from server applications. In response to a user event, the user interface creates a command object and sends it to a command bus. The command bus picks an appropriate handler, passes it the command and schedules it. The user interface, the work done and the scheduling of that work are therefore all decoupled.

IKBCommand Class Diagram

Workflow

At application launch, the application accesses the shared IKBCommandBus object:

@interface IKBCommandBus : NSObject

+ (instancetype)applicationCommandBus;
- (void)execute: (id <IKBCommand>)command;
- (void)registerCommandHandler: (Class)handlerClass;

@end

and registers command handlers. Command handlers know what commands they can process, and can tell the bus whether they will accept a given command. Handlers can also be loaded later, for example in Mac applications or server processes when a new dynamic bundle is loaded.

Once the application is running, the command bus can be used by user interface controllers. These controllers are typically UIViewControllers in an iOS app, NSViewControllers, NSDocuments or other objects in a Cocoa app, or maybe something else in other contexts. A controller receives an action related to a user interface event, and creates a specific IKBCommand.

@protocol IKBCommand <NSObject, NSCoding>

@property (nonatomic, readonly) NSUUID *identifier;

@end

Commands represent requests to do specific work, so the controller needs to configure the properties of the command it created based on user input such as the current state of text fields and so on. This is done on the user interface thread to ensure that the controller accesses its UI objects correctly.

The controller then tells the application’s command bus to execute the command. This does not need to be done on the user interface thread. The bus looks up the correct handler:

@interface IKBCommandHandler : NSObject

+ (BOOL)canHandleCommand: (id <IKBCommand>)command;
- (void)executeCommand: (id <IKBCommand>)command;

@end

Then the bus schedules the handler’s executeCommand: method.

Implementation and Discussion

The Command protocol includes a unique identifier and conformance to the NSCoding protocol. This supports the Event Sourcing pattern, in which changes to the application can be stored directly as a sequence of events. Rather than storing the current state in a database, the app could just replay all events it has received when it starts up.

This opens up possibilities including journaling (the app can replay messages it received but didn’t get a chance to complete due to some outage) and syncing (the app can retrieve a set of events from a remote source and play those it hasn’t already seen). An extension to the implementation provided here is that the event source acts as a natural undo stack, if commands can express how to revert their work. In fact, even if an event can’t be reversed, you can “undo” it by removing it from the event store and replaying the whole log back into the application from scratch.

When a command is received by the bus, it looks through the handlers that have been registered to find one that can handle the command. Then it schedules that handler on a queue.

@implementation IKBCommandBus
{
  NSOperationQueue *_queue;
  NSSet *_handlers;
}

static IKBCommandBus *_defaultBus;

+ (void)initialize
{
  if (self == [IKBCommandBus class])
    {
      _defaultBus = [self new];
    }
}

+ (instancetype)applicationCommandBus
{
  return _defaultBus;
}

- (id)init
{
  self = [super init];
  if (self)
    {
      _queue = [NSOperationQueue new];
      _handlers = [[NSSet set] retain];
    }
  return self;
}

- (void)registerCommandHandler: (Class)handlerClass
{
  _handlers = [[[_handlers autorelease] setByAddingObject: handlerClass] retain];
}

- (void)execute: (id <IKBCommand>)command
{
  IKBCommandHandler *handler = nil;
  for (Class handlerClass in _handlers)
    {
      if ([handlerClass canHandleCommand: command])
        {
          handler = [handlerClass new];
          break;
        }
    }
  NSAssert(handler != nil, @"No handler defined for command %@", command);
  NSInvocationOperation *executeOperation = [[NSInvocationOperation alloc] initWithTarget: handler selector: @selector(executeCommand:) object: command];
  [_queue addOperation: executeOperation];
  [executeOperation release];
  [handler release];
}

- (void)dealloc
{
  [_queue release];
  [_handlers release];
  [super dealloc];
}

@end

Updating the UI

By the time a command is actually causing code to be run it’s far away from the UI, running a command handler’s method in an operation queue. The application can use the Observer pattern (for example Key Value Observing, or Cocoa Bindings) to update the user interface when command handlers change the data model.

Meta-writing

Barely 4,000 years ago, documents were written on heavy, clay tablets. The Epic of Gilgamesh, one of the earliest known works of fiction, was written on 11 such tablets with a 12th added later. There was only one thing you could do with these tablets: read. Fast forward to the 21-st century and things are very different. The word “tablet” has taken on a new meaning, and documents can be delivered wirelessly, updated as new versions are written. They can also contain rich media and hyperlinked references to other content. And with these new capabilities come new considerations when preparing your documents—or “docs”—for your readers.

The above story seems rambling and pointless, doesn’t it? But change the timescale and the technology, and every single bloody report on mobile technology starts in exactly the same way.

Could effortless lecturers make everything seem too easy?

From the British Psychological Society blog: Engaging lecturers can breed overconfidence.

The students who’d seen the smooth lecturer thought they would do much better than did the students who saw the awkward lecturer, consistent with the idea that a fluent speaker breeds confidence. In fact, both groups of students fared equally well in the test. In the case of the students in the fluent lecturer condition, this wasn’t as good as they’d predicted. Their greater confidence was misplaced.

I’m speaking at a couple of conferences later this year (iOSDev UK in Aberystwyth in September and NSScotland in Edinburgh in October), and will endeavour to be exactly as exciting as the material deserves: a capability to which my track record can attest.

The Liskov Citation Principle

In her keynote speech at QCon London 2013 on The Power of Abstraction, Barbara Liskov referred to several papers contemporary with her work on abstract data types. I’ve collected these references and found links to free copies of the articles where available.

Dijkstra 1968 Go To statement considered harmful

Wirth 1971 Program development by stepwise refinement

Parnas 1971 Information distribution aspects of design methodology

Liskov 1972 A design methodology for reliable software systems

Schuman and Jorrand 1970 Definition mechanisms in extensible programming languages
Not apparently available online for free

Balzer 1967 Dataless Programming

Dahl and Hoare 1972 Hierarchical program structures
Not apparently available online for free

Morris 1973 Protection in programming languages

Liskov and Zilles 1974 Programming with abstract data types

Liskov 1987 Data abstraction and hierarchy

Talking about talking

I recently gave a talk to my colleagues about giving talks. Here is an annotated collection of the notes I made in preparation.

- What do you want the audience to get out of the talk?

As you’re constructing your talk, ensure that you’re actually satisfying your mission. If you want to inspire people, make sure you’re not just promoting your own knowledge, business or ability. If you want people to learn things, make sure your talk is appropriate to the experience level of the audience.

- Find out about the audience
    - likely skill level
    - range of experiences
    - interest in technical, business or other issues
- Don't assume that because you think something's obvious, everyone else does

A big stumbling block for novice speakers I talk to is to assume that because you know something, it’s not worth talking about as there are people out there who know way more. Your own experiences and interpretations are different from everyone else’s, it’s very likely that you’ll have something new to contribute—as long as your talk is personal, and not just a restatement of readily-available documentation.

- Decide what it is you're going to say
    - are you trying to inspire or persuade the audience?
        - decide your conclusion
            - if you're worried about timing, give yourself a couple of different exit points
        - how do you want to start?
            - the conclusion's a good place to start

This is a tough place to start though. The idea is that you’re challenging the audience by telling them something that sounds implausible, so they get mentally engaged. If the leap required is too big then you’ll either turn people off, or they’ll still be thinking about the challenge after you’ve started talking.

            - outline the problem that your solution solves

This is the Jobs approach. Start by saying the current world sucks. Explain what a better world would look like. Make it obvious that your proposal leads to the better world. Tell them the thing you propose is available in the foyer as soon as the talk finishes. It’s based on setting up one or more distinctions between the world as it is (or as you tell the audience they currently perceive it) and as it could be (or as you tell the audience they should want it), then showing that those distinctions have actually been resolved. Nancy Duarte did a good talk on this topic.

        - what's the flow between the problem and the conclusion?
        - notice this isn't "tell them what you're about to tell them…"
    - are you trying to educate the audience?
        - you can't in under an hour; aim for awareness or persuasion
        - put additional relevant content on your blog and refer to it
        - you still need a flow

In this case, the audience’s problem is “we don’t know how to do [x]”, the better world is one where they do know how to do [x], and the solution is your content. Don’t try to cram all the code into your talk because it’s distracting, only relevant in a few cases, and hard to parse while keeping up with the presentation. Instead, give people the key parts of the solution so that when they come across [x], they’ll remember some things from your presentation which will help them piece together the full solution.

We had a discussion about WWDC talks at this point in the “live” version of these notes. WWDC seems to provide a counterexample to this rule about not educating people in a talk, with graphics-poor code-rich presentations. Those sessions have two goals: giving developers who aren’t in the labs something to do, and being available on video afterward. The live presentation frankly is overwhelming and often confusing, but isn’t the main use of the talk. You’re expected to watch it over, to refer to the documentation, to ask people about the content in the labs.

[To be honest I also expect there’s an extent to which a lot of the people presenting at WWDC are both strongly pressured by the additional workload of the conference and are uncomfortable with speaking publicly, and the format they’ve settled on is one that works in that context and doesn’t make too many compromises or create too much additional stress. That’s just speculation on my part though.]

    - entertainment
        - doesn't need to be jokes, a compelling talk is entertaining
        - in fact be careful of jokes unless you know the crowd
        - certainly don't lead the laughter

I put this in as a homage to Thorsten Heins, who ruined an otherwise reasonably competent and well-executed presentation by laughing at and even applauding his own jokes. If you want to try a joke, think twice. I only do it in arenas where I’m comfortable I know the people. If you still want to do it, and it falls flat, move on.

        - if you want a set piece, make it relevant to the talk and practice the bejeezus out of it
- engage the audience
    - "hands up if" exercises are light forms of interaction but make people engage with the talk

As discussed earlier, making people too introspective will distract them from your talk. But these days you have to stop people from diving back into their laptops and working during talks, so you need to provide some form of engagement. You also have to deal with the fact that various sub-sections of your audience may be jet lagged, full of lunch or hungover. If they don’t have a part to play in your talk they’ll sleep through it.

    - make eye contact with every part of the audience
        - you will see people asleep or working; don't worry
        - don't forget the back of the room

It’s easy for introverts particularly to “protect” ourselves from the audience by avoiding looking at them. The problem is it then doesn’t feel like we’re talking to everyone out there. You shouldn’t aim to make eye contact with every individual attendee, that doesn’t scale—you definitely should talk to each “part” of the audience though. Talking to someone at the back of the room when you start your talk helps you pitch your voice correctly.

    - motion, nervous or otherwise, gives people a reason to be concentrating on you rather than the screen/their phones
- slides
    - too much text and you lose people while they read along
    - again, relevant content
    - animation where relevant helps, where irrelevant distracts
    - progressive disclosure and hiding

So much has been said about building good slides that I don’t want to add anything. Make sure your own notes are separate from your slides, and everything on-screen now is germane to what you’re saying now.

- q&a
    - there will be an awkward question. Your goal is to handle it gracefully, not to avoid it coming up
    - continuum fallacy

Most of the “well actually” questions you’ll get in conference Q&As are not actually questions. They’re the phrase “I know more than you” dressed up with some rhetorical sugar to appear more question-like. These are poisonous: the audience gains nothing from them, you gain nothing from them, the person “asking” gains nothing from them. Nonetheless we can’t ban people who ask these questions from conferences, so we just have to cope with them. “That’s an interesting point, we should talk about it later” is an OK response—especially if you need to catch the train as soon as you’re off the stage.

The ones that are questions frequently represent instances of the continuum fallacy: you said X is true. Well actually I’ve found an edge case where X isn’t true, therefore X is never true. No. Politely point out the fallacious reasoning, move on to another question. The biggest mistake I make in handling Q&A is letting people argue the toss over questions like these. Again, the rest of the audience is learning nothing from a pointless to and fro; and there are more of them than the two of you having the discussion.

    - set expectations on questions at the beginning, e.g. "I'll take questions at the end" or "interruptions welcome". "No questions" is hard to get the conference organisers to agree with (though would probably help a lot of nervous speakers)—filibustering sometimes works but is apparently rude :)
    - you're effectively chairing a discussion you're also involved in, so don't be afraid of setting topic boundaries. The q&a has to be valuable to the audience.

On community

This is a post that had been boiling for a while; I talked a little about the topic when I was in Appsterdam earlier this year, and had a few more thoughts which were completely supplanted and rearranged by watching iOSDevUK. I threw away my earlier draft; you’re about to read something different. Where you see “we”, “us” or “our community” you should probably take it to mean Cocoa programmers, though read on to find out why “us” doesn’t always make sense.

Acknowledgements

So many people have contributed to this, by saying things that I agree with, by saying things that I disagree with, by organising conferences, or in other ways. I’ve tried to cite where appropriate but I’ve probably missed someone somewhere. Sorry :-(.

Introduction

This article is more the presentation of a problem and some thoughts about it than an attempt to argue in favour of a particular solution. I’ll investigate what it means to be in “the Cocoa programming community”, beginning with whether or not Apple is in a community of its own devising. I’ll ask whether there’s room for more collaboration in the community, and whether the community of Cocoa programmers encompasses all Cocoa programmers. Finally, I’ll notice that these are questions as yet unanswered, and explore what the solutions and non-solutions might be.

On Apple and the community

This is the bit that I’d done most work on already, as it was the topic of my Appsterdam talk. The summary of that talk is pretty much the same as Dave’s working-with-Apple pro tip in his iOSDevUK talk. As his was more succinct, I’ll use that version:

Apple is people too. Don’t be a dick.

(I’m a fan of people not being dicks.)

The thing is that as Scotty said, the community wins when all of its members win. But he also said that Apple isn’t in the community, so don’t they obviate themselves from this relationship?

Well, no. If we look at the community that most of the people reading this post – and that most of the people at iOSDevUK – consider themselves a member of, it’s the community of iOS app makers. It happens that all of these people depend on the same thing: on iOS. Being nice to Apple and helping them just makes good business sense. If you’re not helping Apple to win, they might decide to help you lose.

On a related subject: for Apple to win, it’s not necessary for anyone else to lose. In fact, I’m not the first person to say this. I’m stealing from a man who was, at the time this quote was coined, freshly CEO after having been a management consultant at Apple:

We have to let go of this notion that for Apple to win, Microsoft has to lose. We have to embrace a notion that for Apple to win, Apple has to do a really good job. And if others are going to help us that’s great, because we need all the help we can get, and if we screw up and we don’t do a good job, it’s not somebody else’s fault, it’s our fault.

So Microsoft, Windows 8 and Windows Phone 8 don’t have to lose. Google and Android don’t have to lose. Enterprise Java programmers don’t have to lose. Your competitors don’t have to lose. The team in Apple that make that thing that just crashed don’t have to lose.

On that last note, Apple is the biggest company in the world and you’re supplying one or a handful of 600,000 or so different replaceable components that helps them make a trivial fraction of their income. So if the choice you give them is “do what I need or I’ll stop working with you”, they’ll pick option 2. “Fix Radar or GTFO”? It’s cheaper and easier for Apple to GTFO.

That’s not to say the best strategy is always to do whatever Apple want. Well, actually it probably is in the short term, but Apple is real people and real people benefit from constructive feedback too.

Just who is “them”, anyway?

Around the time that I started to be a proper software writing person, there was a strong division in Mac development. The side I was in (and I was young, opinionated, easily led, and was definitely in this faction) was the Yellow Box. We knew that the correct way to write software for the Mac was to use the Foundation and AppKit APIs via the Objective-C or Java languages.

We also knew that the other people, the Blue Boxers who were using libraries compatible with Mac OS 8 and the C or C++ languages, were grey-bearded dinosaurs who didn’t get it.

This sounds crazy now, right? Should I also point out that I wrote a Carbon app, just to make it sound a little crazier?

That’s because it is crazy. Somehow those of us who had chosen a different programming language knew that we were better at writing software; much better than those clowns who just made the most successful office suite ever, the most successful picture editing app ever, or the most successful video player ever. Because we’d taken advice on how to write software from a company that was 90 days away from bankruptcy and had proven incapable of executing on software development, we were awesome and the people who were making the shittons of money on the most popular software of all time were clueless idiots.

But what about the people who were writing Mac software with WXWindows (which included myself), or RealBASIC, or the PerlObjCBridge (which also included me)? Where did those fit in this dichotomy? Or the people over on Windows (me again) or Solaris (yup, me here)?

The definition of “us” and “them” is meaningless. It needs to be, in order to remain fluid enough that a new “them” can always be found. Looking through my little corner of history, I can see a few other distinctions that have come and gone over time. CodeWarrior vs Project Builder. Mach-O vs CFM. iPhone vs Android. Windows vs Mac. UNIX vs VMS. BSD vs System V. KDE vs GNOME. Java vs Objective-C. Browser vs native. BitKeeper vs Monotone. Dots vs brackets.

Let’s look in more detail at the Windows vs Mac distinction. If you cast your mind back, you’ll recall that around 2000 it was much easier to make money on Windows. People who were in the Mac camp made hand-waving references to technical superiority, or better user interfaces, or breaking the Microsoft hegemony, or not needing to be super-rich. Many of those Mac developers are now iPhone developers. In the iOS vs Android distinction, iOS developers readily point to the larger amount of money that’s available in making iOS apps…wait.

O(community)

The community contribution fraction

As Scotty said, an important role in a community is that of the reader/consumer/learner, the people who take and use the information that’s shared through the community. Indeed in any community this is likely to be the largest share of the community’s population; the people who produce and share the information are also making use of it too.

The thing is, that means that there are many people who are making use of those great ideas, synthesising them, and making even new and better ideas. And we’re not finding out about them. Essentially there is more knowledge than there is opportunity to share knowledge.

It’d be great to have some way to make it super-easy for everyone who was involved in “the community” to contribute, even if it’s just to add a single thought or idea to the pool. As Scotty said, there’s no way you can force people to contribute, and that’s not even desirable as it’s a great way to put people off talking to you ever again.

So you can’t hold a gun up to people and force them to tell you a fact about Objective-C. You can ensure everyone knows what forms of contribution take place; perhaps they’ll find something that’s easier than they thought or something they’ll enjoy. Perhaps they’ll give it a go, and enjoy it.

Face to face

Conferences are definitely not that simple way for everybody to contribute. Conferences are great, though as I’ve said before there aren’t enough seats for them to have a wide direct impact on the community. Tech conferences will never be a base for broad participation, both due to finite size (even WWDC comprises less than one percent of registered developers on the platforms) and limited scope for contribution – particularly the bias toward contributors with “prior”.

One “fix” to scale up the conference is to run the conference all year long. This allows people who don’t like the idea of being trapped in a convention with the same 200 people for a week the option to dip in and out as they see fit. It gives far more opportunity for contribution – because there are many more occasions on which contribution is needed. On the other hand, part of the point of a conference is that the attendees are all at the same place at the same time, so there’s definitely some trade off to be had.

Conferences and Appsterdams alike lead to face-to-face collaboration; the most awesomest flavour of collaboration there is. In return, they require (like Cocoaheads, NSCoder or whatever you call your pub/café meet) that you have the ability to get to the venue. This can call for anything from a walk down the street via a couple of ten-hour flights to relocating yourself and your family.

Smaller-scale chances for face to face interaction exist: one-on-few training courses and one-on-one mentoring and apprenticeships. These are nearly, but not quite, one-way flows of information and ideas from the trainer or sensei to the students or proteges. There are opportunities to make mentoring a small part of your professional life so it doesn’t seem to require a huge time investment.

Training courses, on the other hand, do. Investment by the trainer, who must develop a course, teach it, respond to feedback, react to technology changes and so on. Investment by the trainees, who must spend an amount of time and money attending the course, then doing any follow-up exercises or exams. They’re great ways to quickly get up to speed with a technology by immersing yourselves in them, but no-one is ever going to answer the question “how can I easily contribute to my community?” with “run a training course”.

Teaching at a distance

A lower barrier to entry is found by decoupling the information from the person presenting the information. For as long as there has been tech there have been tech books; it’s easy (if you have $10-$50) to have a book automatically delivered to your house or reader and start absorbing its facts. For published books, there’s a high probability that the content has been proofread and technically reviewed and therefore says something a bit accurate in a recognisable language.

On the other hand, there are very few “timeless” books about technology. Publisher schedules introduce some delay between finishing a manuscript and having something to sell, further reducing any potential shelf life. If you’re in the world of Apple development and planning to say anything about, for example, Objective-C or Xcode, you’re looking at a book that will last a couple of months before being out of date.

Writing a book, then, takes a long time which already might be a blocker to contribution for a lot of people. There’s also the limitation on who will even be invited to contribute: the finite number of publishers out there will preferentially select for established community members and people who have demonstrated an ability to write. It’s easier to market books that way.

The way to avoid all of that hassle is to write a blog (hello!). You get to write things without having to be selected by some commissioning editor. Conversely, you aren’t slowed down by the hassles of having people help you make the thing you write better, either—unless you choose to seek that help.

You then need to find somebody to read your blog. This is hard.

Stats for this blog: most pages have only ever been read a couple of hundred times.

If someone else already has an audience, you can take advantage of that. Jeff Atwood previously wrote about using stack overflow as a blog, where you’d get great reach because they bring their audience. Of course, another thing you can do on stack overflow is answer questions from other people: so that quick answer you contribute is actually solving someone’s problem.

This is, in my opinion, the hallowed middle ground between books (slow, static, hard to get into, with a wide reach) and blogs (fast, reactive, easy to pick up, hard to get discovered). Self-publishing a book is a lot like spending ages writing a long blog post. On the other hand, contributing to a community resource like a Q and A site or a wiki means only writing the bit of the book that you’re best placed to contribute. It also means sharing the work of ensuring correctness and value among the whole contributor base.

Our community / People with ideas ≪ 1

Whatever your definition of “the community”: the iOS developer community, the object-oriented programming community, the developer community—there are many more people who aren’t in that community. But they still have things to say that could be interesting and help us see what we do in different ways.

I’m not so sure that there are people out there doing what we do who don’t even passively engage with the rest of the community. Maybe there are, maybe there are lots. But I’m sure most people have at least read a book, or done a search that ended up at a mailing list post or blog entry. Very few people will never have used community-supplied resources; although it’s possible that there are programmers out there who’ve learned everything they know from first party documentation.

What I am sure of is that if you’re an Objective-C developer building mobile apps and you only listen to other Objective-C developers building mobile apps, you’re missing out on the information and ideas you could be taking from everyone else. Dave Addey told us to go and visit museums and art galleries to get inspiration, but that’s not all there is to it. Talk to someone doing Objective-C in a different context. Talk to someone doing Java, or Clojure. Talk to business people, or artists, or musicians. Break out of the echo chamber, and find out whether what other people are doing could be applied to what you’re doing.

Conclusions

As promised, there aren’t really any conclusions here. It’s more a collection of my own thoughts dumped out from brain to MarsEdit in order to let me make sense of them, and to stop me having to think about them at bedtime.

What’s clear is that there are a load of different ways for people to contribute to a community. Consumption of other people’s thoughts, advice and ideas is itself a very beneficial service as it’s how new ideas get synthesised, how new practices are formed and how the community collectively improves its output. It would be even better if what those people were doing were also made available and shared with the rest of us, to achieve an exponential growth in experience and advancement across the whole community.

But that’s not guaranteed to happen. The best thing to do is not to try driving people to contribute, but to give them so many opportunities to do so that, at some point, someone in the community will be in the position that sharing something is really easy and they choose to do so.

Other techniques to improve the number of ideas you get from the community are to be less adversarial in your definition of community, and more broad in your inclusion. The “community of people making iOS apps with Objective-C” is small, the “community of people making things” is universal.

How people learn

Don’t you hate those times when you go to a talk or article that says “you should be doing this”, but then doesn’t explain how to do that?

I just wrote one. In “Coding. Standards.” I explained that what software engineers should do is to learn and analyse from all their experiences and interactions. But how do you do that? How does learning work?

The model of learning that I know best is the Kolb learning cycle, which I was taught when I was in management training. Yes – the reason I wear my hair this long is to stop it going pointy. David Kolb published this model in the 1980s, it’s been augmented and adapted since then but is still approximately as written. Having said that, some neuroscientists dispute the validity of this and other models of learning.

Kolb says that there are four distinct processes in learning:

  • Concrete Experience: actually doing a thing.
  • Reflective Observation: analysing how you (or someone else) did a thing.
  • Abstract Conceptualisation: building a model of how things should be done.
  • Active Experimentation: just playing with the plasticine and seeing what comes out.

Not everybody goes through all of the items in the cycle, but most people start out somewhere and progress through at least a couple of the points, probably in the order presented (acknowledging that as a cycle it should be, well, cyclic). The four examples he gave corresponded to four “corners” of the cycle:

  • Diverging: concentrates on concrete experience and reflective observation. Tries something out, and decides how it could be adapted.
  • Assimilating: focuses on observation and conceptualisation. Sees how different examples might fit a common pattern.
  • Converging: conceptualisation and experimentation. Try to build a mental model from theory and then put it into practice.
  • Accommodating: experimentation and experience. Hands-on trying things out and deciding which ones work.

This gives you a starting point for working out how you learn, and what materials best support your style: there are many blog posts and conference talks based on personal experience, which fit the “reflective observation” category. Technical books tend to be more abstract, and fit conceptual learning. Training courses often bridge conceptualisation and experimentation.

It also gives you a way to understand why some people don’t like your good-faith attempts to help them. “RTFM” works for those people who want to build a conceptual model, but not for everybody else. Similarly, those people saying “please give me the codes” may be people who learn by abstracting from examples. You can’t hate people just for learning in a different way.

Thoughts on Tech Conferences

This post is being, um, posted from the venue for GOTO Copenhagen 2012. It’s the end result of a few months of reflection on what I get out of conferences, what I want to get out of conferences, what I put into (and want to put into) conferences and the position of tech conferences in our industry. I’ve also been discussing things a lot with my friends and peers; I’ve tried to attribute specific quotes where I remember who said them but let it be known that many people have contributed to the paragraphs below in many different ways. I’ll make it clear at the outset that I’m talking about my experience at independent commercial and non-profit tech conferences, not scientific conferences (of which I have little experience) or first-party events like WWDC (which are straightforward marketing exercises).

Conference speakers

My favourite quote on this subject is courtesy of Mike; I remember him saying it in his MDevcon keynote this year but I’m also fairly sure he’s said the same thing earlier:

The talks at a conference are only there so that you can claim the ticket cost as an expense.

We’re in a knowledge economy; but knowledge itself is not of any value unless it’s applied. That means it’s not the people who tell other people what’s going on who’re are doing the most important work; that’s being done by the people who take this raw knowledge, synthesise it into a weltanschauung – a model of how the world works – and then make things according to that model. Using an analogy with the economy of physical things, when we think of the sculpture of David in Florence we think of Michelangelo, the sculptor, not of the quarry workers who extracted the marble from the ground. Yes their work was important and the sculpture wouldn’t exist without the rock, but the most important and valuable contribution comes from the sculptor. So it is in the software world. Speakers are the quarry workers; the marble hewers, providing chunks of rough knowledge-stuff to the real artisans – the delegates – who select, combine and discard such knowledge-stuff to create the valuable sculptures: the applications.

Michelangelo's David, source: Wikimedia Commons.

Conference speakers who believe that the value structure is the other way around are deluding themselves. Your talk is put on at the conference to let people count the conference as a work expense, and to inspire further discussion and research among the delegates on the topic you’re talking about. It’s not there so that you can promote your consultancy/book/product, or produce tweet worthy quotes, or show off how clever you are. Those things run the gamut from “fringe benefits” to “deleterious side effects”.

As an aside, the first time I presented at a Voices That Matter conference I was worried due to the name; it sounds like the thing that matters at this conference is the speakers’ voices. In fact I suspect there is some of that as many of the presenters have books published by the conference hosts, but it’s a pretty good conference covering a diverse range of topics, with plenty of opportunities to talk to fellow delegates. And IIRC all attendees got an “I am one of the voices that matter” sticker. Anyway, back to the topic at hand: thus do we discover a problem. Producing a quality conference talk is itself knowledge work, that requires careful preparation, distillation and combination of even more raw knowledge-stuff. It takes me (an experienced speaker who usually gets good, but not rave, reviews) about three days to produce a new one hour talk, a roughly 25:1 ratio of preparation:delivery. That’s about a day of deciding what to say and what to leave out, a day of designing and producing materials like slides, handouts and sample code, and a day of practising and editing. Of course, that’s on top of whatever research it was that led me to believe I could give the talk in the first place. The problem I alluded to at the start of the last paragraph is this: there’s a conflict between acknowledging that the talks are the bricks-and-mortar of the conference rather than the end product, and wanting some return on the time invested. How that conflict’s resolved depends on the personal values of the individual; I won’t try to speak for any of my peers here because I don’t know their minds.

The conference echo chamber

That’s not my phrase; I’ve heard it a lot and can track my most recent recollection to @secwhat’s post Conference Angst. Each industry’s conferences has a kind of accepted worldview that is repeated and reinforced in the conference sessions, and that only permits limited scrutiny or questioning – except for one specific variety which I’m coming onto later. As examples, the groupthink in indie Mac/iOS conferences is “developers only need developer features that have been blessed by Apple”. There’s recently been significant backlash to the RubyMotion framework, as there usually is when a new third-party abstraction for iOS appears. But isn’t abstraction a good thing in software engineering? The truth is, of course, that there are more things in heaven and earth than are dream’t in Apple’s philosophy.

The information security groupthink is that information security is working. Shocking though it may sound, that’s far from obvious, evident or even demonstrable. Show me the blind test where similar projects were run with different levels of info sec engagement and where the outcome was significantly different. Demonstrate how any company’s risk profile has changed since last year. Also, show me an example of security practitioners being ahead of the curve, predicting and preparing for a new development in the field: where were the talks on hacktivism before Anonymous or Wikileaks?

One reason that the same views are repeated over multiple conferences is that the same circuit of speakers travels to all of the conferences. I’m guilty of perpetuating that myself, being (albeit unintentionally in one year) one of the speakers in the iOS circuit. And when I’ve travelled to Seattle or Atlanta or Copenhagen or Aberystwyth, I’ve always recognised at least a few names in the speaker line-up. [While I mentioned Aberystwyth here, both iOSDevUK and NSConf take steps to address the circuit problem. iOSDevUK had a number of first-time speakers and a “bar camp” where people could contribute their own talks. NSConf has the blitz talks which are an accessible way to get a large number of off-circuit speakers, and on one occasion ran a whole day of attendee-contributed sessions called NSConf Mini. When you give people who don’t normally present the opportunity to do so, someone will step up.]

I mentioned before that the echo chamber only permits limited scrutiny, and that comes in the form of the “knowing troll” talk. Indeed at GOTO there’s a track on the final day called “Iconoclasm”, which is populated solely with this form of talk. Where the echo chamber currently resounds to the sound of , it’s permitted to deliver an “ sucks” talk. This will usually present a straw man version of and list its failings or shortcomings. That’s allowed because it actually reinforces – real-world examples are rarely anything like as bad as the straw man version, therefore isn’t really that bad. This form of talk is often a last-session-of-the-day entry and doesn’t really lead people to challenge their beliefs. What happens later is that when everyone moves on to the next big thing, the “ sucks” talks will become the main body of the conference and “<X+1> sucks” will be the new troll talk.

Conferences

Weirdly, while the word conference means a bringing together of people to talk, coming from the same root as “conversation”, many conferences are designed around a one-way flow of words from the speakers to the delegates. Here’s the thing with that. As I said in my keynote talk at MDevcon, we learn from each other by telling and listening to stories. Terry Pratchett, Jack Cohen and Ian Stewart even went as far as to reclassify humans as pan narrans, the storytelling chimpanzee. Now if you’ve got M speakers and 10M<N<100M delegates, then putting a sequence of speakers up and listening to their stories gets you a total of M stories. Letting the N delegates each share their stories, and then letting each of the N-1 other delegates share the stories that the first N stories reminded them of, and so on, would probably lead to a total of N! stories if you had the time to host that. But where that does happen, it’s usually an adjunct to the “big top” show which is the speaker series. [And remember: if you’ve got C conferences, you don’t have C*M speakers, you have M+ε speakers.]

There’s one particular form of wider participation that never works well, and that’s to follow a speaker session with Q&A. Listen carefully to the questions asked at the next Q&A you’re in, and you’ll find that many are not questions, but rhetorical statements crafted to make the “asker” appear knowledgable. Some of those questions that are questions are rhetorical land mines with the intent of putting the speaker on the back foot, again to make the asker seem intellectually talented. Few of these questions will actually be of collective value to the plenus, so there’s not much point in holding the Q&A in front of everyone.

Speaker talks are only one way to run a session, though. Panels, workshops and debates all invite more collaboration than speaker sessions. They’re also much more difficult to moderate and organise, so are rarely seen: many conferences have optional days that are called “workshops” but in reality are short training courses run by an invited speaker. In the iOS development world, lab sessions are escaping the confines of WWDC and being seen at more independent conferences. These are like one-on-one or few-on-few problem solving workshops, which are well focussed and highly collaborative but don’t involve many people (except at Voices that Matter, where they ran the usability workshops on the stage in front of the audience). A related idea being run at GOTO right now, which I need to explore, is a whole track of pair programming sessions. The session host chooses a technology and a problem, and invites delegates onto the stage to work through the challenge with the host in a pair-programming format. That’s a really interesting way to attract wider participation; I’ll wait until I’ve seen it in action before reaching an opinion on whether it works.

There’s another issue, that requires a bit more setup to explain. Here’s a Venn diagram for any industry with a conference scene; the areas are indicative rather than quantitative but they show the relation between:

  • the population of all practitioners;
  • the subsection of that population that attends conferences; and
  • the subsection of that population that speaks at conferences.

Conference Venn diagram

So basically conferences scale really badly. Even once we’ve got past the fact that conferences are geared up to engage the participation of only a handful of their attendees, the next limiting factor is that most people in [whatever industry you’re in] aren’t attending conferences. For the stories told at a conference (in whatever fashion) to have the biggest impact on their industry, they have to break the confines of the conference. This would traditionally, in many conferences, involve either publishing the proceedings (I’ve not heard of this happening in indie tech conferences since the NATO conferences of 1968-9, although Keith Duncan is one of a couple of people to mention to me the more general idea of a peer-reviewed industry journal) or the session videos (which is much more common).

To generate the biggest impact, the stories involved must be inspiring and challenging so that the people who watched them, even those who didn’t attend the conference, feel motivated to reflect on and change the way they work, and to share their experiences (perhaps at the same conference, maybe elsewhere). Before moving on to a summary of everything I’ve said so far, I’ll make one more point about the groups drawn on the Venn diagram. Speakers tend to be specialists (or, as Marcus put it in his NSConf talk, subject matter experts) in one or two fields; that’s not surprising given the amount of research effort that goes into a talk (described above). Additionally, some speakers are asked to conferences because they have published a book on the topic the convenor wishes them to speak on; that’s an even longer project of focussed research. This in itself is a problem, because a lot of the people having difficulty with their work are likely to be neophytes, but apparently we’re not listening to them. We listen to self-selected experts opining on why everyone needs to take security/TDD/whatever seriously and why that involves retaining the experts’ consultancy service: we never listen to the people who can tell us that after a month of trying this Objective-C stuff still doesn’t make sense. These are the people who can give us insight into how to improve our practice, because these are the people reminding the experts (and indeed the journeymen) of the problems they had when they’d been at this for a month. They tell us about the issues everyone has, and give us ideas on how we can fix it for all (future) participants.

Conference goers, then, get the benefit of a small handful of specialists: in other words they have a range of experience to call on (vicariously) that is both broad and deep. Speakers of course have the same opportunity, though don’t always get to take full advantage of the rest of a conference due to preparation, equipment tests, post-talk question sessions and the like. The “non-goers” entry in the diagram represents a vast range of skills and experiences, so it’s hard to find any one thing to say about them. Some will be “distance delegates”, attending every conference by purchasing the videos, transcripts or other materials. Some will absorb information by other means, including meet-ups, books, blogs etc. And some will be lone coders who never interact with anyone in their field. Imagine for a moment that your goal in life is to apply the Boy Scout Rule (which I’m going to attribute again to @ddribin because I can’t remember who he got it from; Uncle Bob probably) to your whole industry. Your impact on $thing_you_do will be to leave the whole field, the whole practice a bit better than it was when you got here. (If that really is your goal, then skip the imagination part for a bit.)

It seems to me that the best people to learn from are the conference delegates (who have seen a wide section of the industry in considerable depth) and the best people to transfer that knowledge to are, well, everybody.

Summary of the current position

Conferences are good. I don’t want people to think I’m hating on conferences. They’re enjoyable events, there are plenty of good ones, there’s an opportunity to learn things, and to see fresh perspectives on many aspects of our industry. They’re also more popular than ever, with new events appearing (and selling out rapidly) every year. However, these perspectives often have an introspective, echo chamber quality. We’re often listening to a small subset of the conference delegates, and if you integrate over multiple conferences you find the subset gets relatively smaller because it’s the same people presenting all the time. Most delegates will not get the benefit of listening to all of the other delegates, which means they’re missing out on engaging with some of the broadest experience in the industry. Most of the practitioners in your corner of the industry probably don’t attend any conferences anyway; there aren’t enough seats for that to work.

The ideal tech conference

OK, I am very clearly lying here: this isn’t the ideal tech conference, it’s my ideal tech conference. In my world, those are the same thing. PerfectConf features a much more diverse portfolio of speakers. In the main this is achieved exactly the way that Appsterdam does it; by offering the chance to speak to anyone who’ll take it, by looking for things that are interesting to hear about rather than accomplished or expert speakers to say it, and by giving novice speakers the chance to train with the experts before they go in front of the stage. Partly this diversity is achieved by allowing people who aren’t comfortable with speaking the opportunity to host a different kind of session, for example a debate or a workshop.

In addition to engaging session hosts who would otherwise be apprehensive about presenting, we get to hear about the successes and tribulations encountered by the whole cohort of delegates. At least one session would be a plenary debate, focussed on a problem that the industry is currently facing. This session has the modest aim of discovering a solution to the problem to move the industry as a whole forward. Another way in which diversity is introduced into the conference is by listening to people outside of our own sector. If infosec is having trouble getting budget for its activities, perhaps they ought to invite more CFOs or comptrollers to its conferences to discuss that. If iPhone app developers find it hard to incorporate concurrency into their application designs, they could do worse than to listen to an Erlang or Occam expert. Above all, the echo chamber would be avoided; session hosts would be asked to challenge the perceived industry status quo.

I’ve long thought that if a talk of mine doesn’t annoy at least one member of the audience then I haven’t said anything useful; a former manager of mine said “if we both think the same way about everything then one of us is redundant”. This way of thinking would be codified into the conference. Essentially, what I’m talking about is the death of the thought leader (or “rock star”). Rather than having one subject matter expert opining on how everyone should think about security, UX, marketing, or whatever, PerfectConf encourages the community to work together like a slime mould, allowing the collective motion of all of the members to explore all opportunities and options and select the best one by communicating freely across the colony.

Slime mold solving a maze (photo: Nature)

Finally, PerfectConf proceedings are published as soon as practical; not just the speaker sessions but the debates too. Where the plenus reaches a consensus, the consensus decision becomes available for all those people who couldn’t make it to the conference of to discover, consider, and potentially adopt or react to. Unfortunately I’m not a conference organiser.

On internal quality

I was asked by attendees at my VTM talk on test-driven development a small collection of questions on a similar theme, which I’ll summarise here.

  • How do I do TDD when my boss doesn’t want me to?
  • What do I do when my boss wants me to ship untested prototype code?
  • Can you give me rhetoric to convince my boss that I should be doing TDD?

I believe that I can adequately present these questions as facets of the following root issue, and then blabber away endlessly about that one instead.

I have externalised the obligation I have to myself to write software that makes me proud by delegating it up to my boss. Can you help me to sleep at night?

The fundamental problem here is one of cognitive dissonance. Belief: I, as a software engineer, believe that process/technique/tool is good for writing software. Introspection: I observe that I am not using said process/technique/tool. Justifications: it must be that my boss doesn’t hold my output to the same quality standard that I do. It must be that my deadlines and ever-shifting requirements don’t give me time to do things properly. It must be that this company doesn’t want great software engineers.

No. I have learned the hard way that such justifications are false. The only person responsible for how you write code is you. If you work on a team, or are handing your source code over to some other stakeholder (which means you’re working on a team), then you may have style guides that limit what source code you write; but ultimately how you get there is solely down to you.

You may be thinking “but I asked my manager for time to implement TDD/CI/buzzword-compliance on my current project and she said no. Also, by the way, you’re an arsehole for being so rude and presuming to know what I think.” Let me address the latter point first: suck it. Now onto the first point.

Your manager was correct to turn down your request for time on the project. Your customer doesn’t want unit tests, or comment documentation, or build server output. Your customer wants working features. The project exists to satisfy the user’s needs. Therefore the time allocated on the project should be dedicated to making working features.

Now it turns out that doing TDD is one way to write code that leads to working features, so what you should have done was to agree to the version of the product plan where you make working features, and then done TDD anyway. Where this all started going wrong was not when your boss turned down your request, but when you asked your boss in the first place. It turns out that you both have the same goal—making a good product—but that you have different views on the process and different motivators. By asking your boss how to write code you gave her permission to micro-manage you, then got frustrated when she did, and decided that the problem was all her fault.

I’ve seen this failure mode quite a few times now. As one example, I worked in a company where there was an ingrained antagonism between the product managers (who clearly just don’t get that software is a craft and a labour of love) and the programmers (who clearly just don’t get that we work in a competitive marketplace and would rewrite the product from scratch every day if they could).

As is common (and often correct) in our industry, the product managers owned the product requirements. As should be welcomed when we care about the products we make, the programmers were invited to criticise the requirements for the projects they were working on. Due to this antagonistic culture, the programmers would typically stuff “engineering requirements” onto the project, which were things like rewriting components from scratch, epic refactorings, or setting up new developer workflows.

Requirements were, as far as I could tell, prioritised based on how much revenue they would attract (i.e. new customers), how much revenue they would protect (retained customers) and how much they would cost. As you can probably already guess, so-called engineering requirements don’t protect any revenue, they don’t generate any revenue, and they do take time and money to implement, so they would inevitably get dropped or indefinitely postponed.

The moral of that little story is this: it is folly to try and express development processes and methodologies as business requirements, because they fail at being business requirements. It’s the equivalent of a taxi driver asking the customer to pay a $100,000 fare for one journey because he wants to switch to automatic transmission for the next journey and that means buying a new vehicle. The customer cares about being driven somewhere, and doesn’t care about how the driver operates the vehicle as long as the journey is bug-free. He certainly doesn’t intend to pay for the driver to select a particular mode of operation: if the driver wants to use automatic transmission, the driver should just get on and do that.

So what I’m saying is this: it’s not up to your boss, your customer or your project manager to choose how you write software. It’s up to you to choose how you write software, and as long as the approach you take leads to working software that delights your customers, the rest of the stakeholders won’t mind what you’re doing “under the hood”. My good friend and long-lost son @bmf once said “don’t let them see you making it.” I would go further: don’t let them know it was made. Let them use exciting, compelling software: the magic is your business. If you want to make the magic in one particular way, that would let you take additional pride in your creation, that’s entirely your call.