Getting better at doing it wrong

For around a month at the end of last year, I kept a long text note called “doing doing it wrong right”. I was trying to understand error handling in programming, look at some common designs and work out a plan for cleaning up some error-handling code I was working with myself (mercifully someone else, with less analysis paralysis, has taken on that task now).

Deliciously the canonical writing in this field is by an author with the completely apt name Goodenough. His Structured Exception Handling and Exception Handling: Issues and a Proposed Notation describe the problem pretty completely and introduce the idea of exceptions that can be thrown on an interesting condition and caught at the appropriate level of abstraction in the caller.

As an aside, his articles show that exception handling can be used for general control flow. Your long-running download task can throw the “I’m 5% complete now” exception, which is caught to update the UI before asking the download to continue. Programming taste moved away from doing that.

In the Cocoa world, exceptions have never been in favour, probably because they’re too succinct. In their place, multi-if statement complex handling code is introduced:

NSError *error = nil;
id thing = [anObject giveMeAThing:&error];
if (!thing) {
  [self handleError:error];
  return;
}
id otherThing = [thing doYourThing:&error];
if (!otherThing) {
  [self handleError:error];
  return;
}
id anotherThing = [otherThing someSortOfThing:&error];

…and so it goes.

Yesterday in his NSMeetup talk on Swift, Saul Mora reminded me of the nil sink pattern in Objective-C. Removing all the error-handling from the above, a fluent (give or take the names) interface would look like this:

id anotherThing = [[[anObject giveMeAThing] doYourThing] someSortOfThing];

The first method in that chain to fail would return nil, which due to the message-sink behaviour means that everything subsequent to it preserves the nil and that’s what you get out. Saul had built an equivalent thing with option types, and a function Maybe a -> (a -> Maybe b) -> Maybe b to hide all of the option-unwrapping conditionals.

Remembering this pattern, I think it’s possible to go back and tidy up my error cases:

NSError *error = nil;
id anotherThing = [[[anObject giveMeAThing:&error]
                               doYourThing:&error]
                           someSortOfThing:&error];
if (!anotherThing) {
  [self handleError:error];
}

Done. Whichever method goes wrong sets the error and returns nil. Everything after that is sunk, which crucially means that it can’t affect the error. As long as the errors generated are specific enough to indicate what went wrong, whether it’s possible to recover (and if so, how) and whether anything needs cleaning up (and if so, what) then this approach is…good enough.

Posted in architecture of sorts, code-level, OOP | 2 Comments

Ch-ch-ch-ch-changes

It’s been almost a year since my first day at Facebook, sitting in an overcrowded meeting room with my bootcamp class because 42 Earlham Street was full and it’d be another week before we moved to 10 Brock Street, with its gargantuan empty spaces (which are no longer empty: nearly half the company has joined since I started).

How has that year been, you ask? Hard. It’s been hard. Fun, worthwhile, educational, exciting, but hard. Facebook is very different from any company I’ve worked at before: it’s bigger, it’s faster, it’s more ambitious. Everything I had learned about making software outside the company was…well no, not wrong, certainly not wrong. But it was perhaps inapplicable. Facebook does things differently, I can’t immediately have impact by doing what I did at my other employers, how can I cope with this?

When you’re used to the life of minor engineering celebrity, where people go to conferences because you’re on the bill, and your pithy statements on programming get retweeted up the wazoo, it’s easy to get hubristic. I did: I know what I’m doing, the engineers here don’t yet know that, and there will be a glorious revelation when I bring the two tablets (an iPad and a Galaxy Note 10, I don’t know) down from the mountain, sell everyone in the company a copy of my book and we all start doing things The Right Way™.

That wasn’t going to work. Not because what I wanted to do couldn’t work, but because I was starting from the wrong place. How many billion-dollar companies had I written software for when I started at Facebook? None. So who was I to say that my way of writing software was “right”? All I could say was that it definitely felt right.

What I had to learn to move past this was that Facebook works on data. If I can show that there’s time being lost, or bugs being introduced, or run times being lengthened, and that what I want to do would save time, or catch bugs, or speed things up, then Facebook will listen.

The opposite of “my way is not incontrovertibly correct” is not “my way is incontrovertibly incorrect”, but it was easy to come to that conclusion too, and to rage-quit the idea of having any effect here. I think I avoided this, but I also think I came pretty close to it. I could easily have decided that while I don’t like Facebook’s way, I need to suck it up and work that way. That’s wrong for the same reason that trying to boil Facebook’s oceans was wrong: just as what I’m used to doing isn’t necessarily correct, what Facebook is doing isn’t necessarily correct. There’s room for change.

Indeed, Facebook engineering is far from change-averse. Everything is set up to let you make changes, from the extreme collective code ownership (“bored changing the code you normally work on? Change the rest of it!”) to incident review (“OK, that change wasn’t the best, let’s see what we can learn from it”).

So, it took me a long time, perhaps longer than it should: I used my first year to realise that I can change Facebook and Facebook can change me, and the way we’ll make this happen is by showing each other how the changes will make us better. Now, as David Bowie said, it’s time to turn and face the strange.

Posted in Uncategorized | Leave a comment

A kata above

The code kata is a method software craftspeople use to practice their craft. The idea is that you take a problem you understand, like FizzBuzz or Conway’s Life, and build an application that implements it. Then build another one. And another. Pretty soon, you should be quite good at building a thing with well-understood requirements.

But that’s the problem, isn’t it? Our professional time can be split into two parts:

  • the part where we don’t think we quite understand what we’re meant to be doing.
  • the part where we are wrong.

So what’s the point of being good at implementing well-understood requirements effortlessly? A partial answer is that being able to do so frees up our minds to think about the real problem, which is understanding the real problem. I’m sure our customers will be happy to hear that the only way we have to practice that is to take their money and pretend that we know what we’re doing.

What are the causes of not knowing what we’re doing? I can think of at least two: not knowing what the requirements actually are (which includes knowing what everyone thinks they are but not knowing how fulfilling those will change them), and having an existing system to adapt to the new requirements. A lot of the time, both of these are unknown.

We already have plenty of “work with legacy code” katas. We call them open source issue trackers. Find a project, find a bug, dive in and fix it. Practice working with code that’s new to you, and already solves most of its problems.

You’ll find that with open source and free software, the two extremes of bugfixing get practiced a lot. At one end, people are willing to fix trivial bugs that mean inserting one if into the existing code. At the other end, people will do complete rewrites, throwing away working code and its bugfixes to introduce “clean”, though not necessarily working, code. It’s the expanse of space between these, where you need to make substantial changes but without throwing out the beneficial existing behaviour, where there’s space for legacy code katas.

But maybe we also need the “requirements kata”. Think of one of your interests that isn’t programming, and find a forum on it. Locate a thread where someone has asked a question that’s led to either no answer or a convoluted process to solve their problem, and think about how you could design a software system to solve that, and what other information you’d need to specify it well. Think about how you’d validate that the derived system actually solves the problem.

Posted in advancement of the self | Leave a comment

In The Design of Design, Fred Brooks makes an interesting point about ESR’s description of the Bazaar model of Linux (and, by extension, “Open Source”) development.

Linux was actually designed in a cathedral. The design was supplied by Unix, where Linux was to be a work-alike replacement for a particular component. There was even a functional specification: the GNU utilities already existed and the kernel had to support them.

Posted on by Graham | Leave a comment

An acceptable tool

It’s easy to forget that adequacy is, well, adequate. It’s easy to go all-in on making some code structure perfect, when good enough would be good enough.

Posted on by Graham | Leave a comment

Beware the IDEs

I recently had the opportunity to talk with a couple of software project managers from IBM. That company is of a kind that I have never worked at, and many of the companies I have worked at are of kinds that these IBMers have not worked at. There was thus plenty of opportunity for us to have different opinions and to explore those.

One such difference, though one we did not investigate in depth, is over IDEs. There was no doubt in the IBM view: a developer with an IDE is a more productive developer. If your programming team wants to license some proprietary IDE to use on your project, you’re probably better off paying for the licensing.

That’s not the world I come from, and indeed I may even have removed IDEs from my consciousness completely. Sure, I use a few, but what’s wrong with emacs, TAGS, and a bit of gud.el and M-x compile? I don’t mind launching an IDE when it’s there, but I don’t miss it when I work with separate tools.

So, are the people who write Rails apps in Sublime Text or vim doing it right, or are they missing out on clear productivity gains? More fundamentally, was it a good idea to build out something like Rails without keeping the IDE support in sync with the features?

There are all sorts of reasons to believe that IDEs are better.

  • IDEs are actually better, and IBM has a mature enough development system to be able to measure that.
  • IBM believes the Rational marketing (Rational’s price ticket was at the high end of software company acquisitions and perhaps IBM’s culture has, ahem, rationalised that).
  • There was a time that IDEs were better than developing without, but now bloatware text editors like emacs and vim have caught up without IBM’s view updating.

There are also reasons to believe that IDEs are not better.

  • IDEs are actually not betSourceKit Service crashed.
  • The cynic in me doesn’t believe the facts presented by the IDE vendors’ marketeers.
  • The bloatware text editors have actually caught up.
  • The productivity problems I have do not stem from my choice of editing environment.

What’s kindof interesting is to compare the views of dyed-in-the-wool IDE evangelists, dyed-in-the-wool text editor/command-line proponents, and fence-sitters and find out why they have different views, whether there are facts that can be teased out of these positions or assertions that can be validated. What’s really fascinating is general situation of which this is a single example: there are whole regions of the software development phase space that I have yet to experience.

Posted in tool-support | 2 Comments

Hiding behind messages

A problem I think about every so often is how to combine the software design practice of hiding implementations behind interfaces with the engineering practice of parallel execution. What are the trade-offs between making parallelism explicit and information hiding? Where are some lines that can be drawn?

Why do we need abstractions for this stuff, when the libraries we have already work? Those libraries make atomic features like threads and locks explicit, or they make change of control flow explicit, and those are things we can manage in one place for the benefit of the rest of our application. Nobody likes to look at the dispatch terrace:

    });
  });
});

Previous solutions have involved object confinement, where every object has its own context and runs its code there, and the command bus, where you ask for work to be done but don’t get to choose where.

Today’s solution is a bit more explicit, but not much. For every synchronous method, create an asynchronous version:

@interface MyObject : Awaitable

- (NSString *)expensiveCalculation;
- async_expensiveCalculation;

@end

@implementation MyObject

- (NSString *)expensiveCalculation
{
  sleep(5);
  return @"result!";
}

@end

int main(int argc, const char * argv[]) {
  @autoreleasepool {
    MyObject *o = [MyObject new];
    id asyncResult = [o async_expensiveCalculation];
    NSLog(@"working...");
    // you could explicitly wait for the calculation...
    NSLog(@"Result initial: %@", [[asyncResult await]   substringToIndex:1]);
    // ...but why bother?
    NSLog(@"Shouty result: %@", [asyncResult uppercaseString]);
  }
  return 0;
}

This is more of a fork-and-join approach than fire and forget. The calling thread carries on running until it actually needs the result of the calculation, at which point it waits (if necessary) for the callee to complete before continuing. It’ll be familiar to programmers on other platforms as async/await.

The implementation is – blah blah awesome power of the runtime – a forwardInvocation: method that looks for messages with the async marker, patches their selector and creates a proxy object to invoke them in the background. That proxy is then written into the original invocation as its return value. Not shown: a pretty straightforward category implementing -[NSInvocation copyWithZone:].

@implementation Awaitable

- (SEL)suppliedSelectorForMissingSelector:(SEL)aSelector
{
  NSString *selectorName = NSStringFromSelector(aSelector);
  NSString *realSelectorName = nil;
  if ([selectorName hasPrefix:@"async_"]) {
    realSelectorName = [selectorName substringFromIndex:6];
  }
  return NSSelectorFromString(realSelectorName);
}

- (NSMethodSignature *)methodSignatureForSelector:(SEL)aSelector
{
  NSMethodSignature *aSignature =   [super methodSignatureForSelector:aSelector];
  if (aSignature == nil) {
    aSignature =    [super methodSignatureForSelector:[self suppliedSelectorForMissingSelector:aSelector]];
  }
  return aSignature;
}

- (void)forwardInvocation:(NSInvocation *)anInvocation
{
  SEL trueSelector =    [self suppliedSelectorForMissingSelector:[anInvocation selector]];
  NSInvocation *cachedInvocation = [anInvocation copy];
  [cachedInvocation setSelector:trueSelector];
  CBox *box = [CBox cBoxWithInvocation:cachedInvocation];
  [anInvocation setReturnValue:&box];
}

@end

Why is the proxy object called CBox? No better reason than that I built this while reading a reflection on Concurrent Smalltalk where that’s the name of this object too.

@interface CBox : NSProxy

+ (instancetype)cBoxWithInvocation:(NSInvocation *)inv;
- await;

@end

@implementation CBox
{
  NSInvocation *invocation;
  NSOperationQueue *queue;
}

+ (instancetype)cBoxWithInvocation:(NSInvocation *)inv
{
  CBox *box = [[self alloc] init];
  box->invocation = [inv retain];
  box->queue = [NSOperationQueue new];
  NSInvocationOperation *op = [[NSInvocationOperation alloc]    initWithInvocation:inv];
  [box->queue addOperation:op];
  return [box autorelease];
}

- init
{
  return self;
}

- await
{
  [queue waitUntilAllOperationsAreFinished];
  id returnValue;
  [invocation getReturnValue:&returnValue];
  return [[returnValue retain] autorelease];
}

- (void)dealloc
{
  [queue release];
  [invocation release];
  [super dealloc];
}

- forwardingTargetForSelector:(SEL)aSelector
{
  return [self await];
}

@end

You don’t always need some huge library to clean things up. Here are about 70 lines of Objective-C that abstract an implementation of concurrent programming and stop my application code from having to interweave the distinct responsibilities of what it’s trying to do and how it’s trying to do it.

Posted in architecture of sorts, code-level, OOP | 4 Comments

It depends? It depends.

Sometimes you ask a question which has a small collection of actionable answers: yes or no. You ask someone who should be able to give that yes or no answer, and they go for the third: it depends.

Maybe they can’t actually answer your question, but want to sound profound.

Maybe they don’t realise that you’re a beginner in this particular area and would benefit from knowing what they’d do in this situation.

Maybe they don’t realise that you’re an expert in this area and are looking for confirmation or refutation of your existing choice.

Maybe they don’t realise that you’re uninterested in this area, and just need a decision made so you can move on.

Maybe they don’t know what the answer is, but are hoping you’ll help them think through the alternatives.

Sometimes, the answer to a question is “it depends”. But is that the answer you should give? It depends.

Posted in learning | Leave a comment

GNU Terry Pratchett

(post-hoc prescript: I admit to being in two minds about sharing this post. Name-dropping can be the ultimate in reflected vanity: I have worth because I knew this worthy person. I title it about them, but we both know it’s about me. I hope this post, containing much as it does about me and my experiences, is actually about what I learned from an Old Fart in a Hat – among others.)

GNU Terry Pratchett

Nearly three years ago, The Compleat Ankh-Morpork was approaching completion, and the publishers were excited about the potential to turn it into an interactive discovery of the Circle Sea’s cleanest cityest city. At the time that I joined Agant, Dave “OMG it’s Dave Addey off the internets” Addey had already been discussing the project with them, and much was ready to go.

Indeed, Dave and I were ready to go. We took a train down to the Discworld Emporium and met Sir Terry Pratchett, his partner in crime Rob Wilkins, their able Aiders and Abettors from the Emporium and some of the most happening among those who Make Things Happen at the publishers.

In the Dolphin Hotel at Wincanton, we had a pub lunch and tried to define exactly what it meant to put Ankh-Morpork into an iPad. I’d already made a start, and here I learned Lesson Two: the creative mind knows it can be done. You don’t need to convince them it can be done. It’s already been done, it’s up there in their mind’s eye. You just need to convince them it can be done well, and in this reality.

Wait, lesson two? Yes, for here was Lesson One: when meeting your heroes, only one of the two of you is likely to try and make a big deal out of it. The other wants their ploughman’s lunch and their G&T and to get back to work. Keeping focussed on the task at hand – a difficult feat as I was dealing with an internal monologue that would not stop going “squeeeeeee” – paid off, and everyone left safe in the knowledge that there was lots to do but that we would do it because it needed to be done and we needed to be the ones to do it and we needed it to be done well.

OK, most of the rest of this story is history. Dave, the developers at Agant, and a dedicated collection of others worked very hard for a number of months and eventually Discworld: the Ankh-Morpork Map for iPad oozed its way out of the alledged waters of the Ankh (there’s no need to disinfect it, the bacteria refuse to touch the stuff) and into the App Store. It remains the project of which I am most proud, both of the contributions I made and of all that was put in by everyone involved.

All good stories, though, come to an end, and Sir Pterry’s did last week. Today I went back over to the Emporium to talk a bit about the past and a lot about the future. It’s from today’s discussions that Lesson Three originates.

Some day, you will be invited by a polite though insistent anthropomorphic personification to take a walk together. Before he arrives, do the things that need to be done. Make sure they’re the things that must be done by you, and that you will be satisfied and pleased to see completed. Otherwise the two of you won’t have a whole lot to talk about.

Posted in whatevs | Leave a comment

Linus’s Bystanders

For some reason, when Eric S. Raymond wanted to make a point about the “bazaar” model of open source software development, he named it after someone else. Thus we have Linus’s Law:

Linus was directly aiming to maximize the number of person-hours thrown at debugging and development, even at the possible cost of instability in the code and user-base burnout if any serious bug proved intractable. Linus was behaving as though he believed something like this:

8. Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.

Or, less formally, “Given enough eyeballs, all bugs are shallow.” I dub this: “Linus’s Law”.

The “proof” of this law, such as it is, is a reductio ad absurdum painting the contradiction of the law as a universe in which the Linux kernel couldn’t be created.

If “Linus’s Law” is false, then any system as complex as the Linux kernel, being hacked over by as many hands as the that kernel was, should at some point have collapsed under the weight of unforseen bad interactions and undiscovered “deep” bugs. If it’s true, on the other hand, it is sufficient to explain Linux’s relative lack of bugginess and its continuous uptimes spanning months or even years.

Let us remember the time that Linux development did collapse under its own weight. Raymond’s original essay was written in 1997; by 2006 it seemed that “we’re adding bugs at a higher rate than we’re fixing them”. In between, more developers had (presumably) contributed, so what happened there? Why weren’t all of the bugs even shallower?

I’d like to examine a result from the social sciences here. From the Wikipedia (I am researcher, hear me roar) page on bystander effect:

The bystander effect, or bystander apathy, is a social psychological phenomenon that refers to cases in which individuals do not offer any means of help to a victim when other people are present. The probability of help is inversely related to the number of bystanders. In other words, the greater the number of bystanders, the less likely it is that any one of them will help.

And then, a few paragraphs later when talking about computer-mediated communication:

Research suggests that the bystander effect may be present in computer-mediated communication situations.

The experiments into bystander apathy are specifically about someone in distress, either asking for help or visibly in trouble. Maybe it applies here, too, when the only “distress” is that some software you didn’t pay for could be better than it is. Perhaps, even before we get into discussions of whether the projects have welcoming communities that accept contributors, our projects are putting people off just because they know that there are plenty of others out there who could fix these bugs before they do.

I’ve met software team leaders who, when asked how their techniques and processes could be made to scale to a team of 1,000, answer that they’d start by giving the project to a team of ten. Maybe they’re onto something.

Posted in social-science | Leave a comment