Skip to content

The Design of the Bazaar

In The Design of Design, Fred Brooks makes an interesting point about ESR’s description of the Bazaar model of Linux (and, by extension, “Open Source”) development.

Linux was actually designed in a cathedral. The design was supplied by Unix, where Linux was to be a work-alike replacement for a particular component. There was even a functional specification: the GNU utilities already existed and the kernel had to support them.

An acceptable tool

An acceptable tool

It’s easy to forget that adequacy is, well, adequate. It’s easy to go all-in on making some code structure perfect, when good enough would be good enough.

Beware the IDEs

I recently had the opportunity to talk with a couple of software project managers from IBM. That company is of a kind that I have never worked at, and many of the companies I have worked at are of kinds that these IBMers have not worked at. There was thus plenty of opportunity for us to have different opinions and to explore those.

One such difference, though one we did not investigate in depth, is over IDEs. There was no doubt in the IBM view: a developer with an IDE is a more productive developer. If your programming team wants to license some proprietary IDE to use on your project, you’re probably better off paying for the licensing.

That’s not the world I come from, and indeed I may even have removed IDEs from my consciousness completely. Sure, I use a few, but what’s wrong with emacs, TAGS, and a bit of gud.el and M-x compile? I don’t mind launching an IDE when it’s there, but I don’t miss it when I work with separate tools.

So, are the people who write Rails apps in Sublime Text or vim doing it right, or are they missing out on clear productivity gains? More fundamentally, was it a good idea to build out something like Rails without keeping the IDE support in sync with the features?

There are all sorts of reasons to believe that IDEs are better.

  • IDEs are actually better, and IBM has a mature enough development system to be able to measure that.
  • IBM believes the Rational marketing (Rational’s price ticket was at the high end of software company acquisitions and perhaps IBM’s culture has, ahem, rationalised that).
  • There was a time that IDEs were better than developing without, but now bloatware text editors like emacs and vim have caught up without IBM’s view updating.

There are also reasons to believe that IDEs are not better.

  • IDEs are actually not betSourceKit Service crashed.
  • The cynic in me doesn’t believe the facts presented by the IDE vendors’ marketeers.
  • The bloatware text editors have actually caught up.
  • The productivity problems I have do not stem from my choice of editing environment.

What’s kindof interesting is to compare the views of dyed-in-the-wool IDE evangelists, dyed-in-the-wool text editor/command-line proponents, and fence-sitters and find out why they have different views, whether there are facts that can be teased out of these positions or assertions that can be validated. What’s really fascinating is general situation of which this is a single example: there are whole regions of the software development phase space that I have yet to experience.

Hiding behind messages

A problem I think about every so often is how to combine the software design practice of hiding implementations behind interfaces with the engineering practice of parallel execution. What are the trade-offs between making parallelism explicit and information hiding? Where are some lines that can be drawn?

Why do we need abstractions for this stuff, when the libraries we have already work? Those libraries make atomic features like threads and locks explicit, or they make change of control flow explicit, and those are things we can manage in one place for the benefit of the rest of our application. Nobody likes to look at the dispatch terrace:

    });
  });
});

Previous solutions have involved object confinement, where every object has its own context and runs its code there, and the command bus, where you ask for work to be done but don’t get to choose where.

Today’s solution is a bit more explicit, but not much. For every synchronous method, create an asynchronous version:

@interface MyObject : Awaitable

- (NSString *)expensiveCalculation;
- async_expensiveCalculation;

@end

@implementation MyObject

- (NSString *)expensiveCalculation
{
  sleep(5);
  return @"result!";
}

@end

int main(int argc, const char * argv[]) {
  @autoreleasepool {
    MyObject *o = [MyObject new];
    id asyncResult = [o async_expensiveCalculation];
    NSLog(@"working...");
    // you could explicitly wait for the calculation...
    NSLog(@"Result initial: %@", [[asyncResult await]   substringToIndex:1]);
    // ...but why bother?
    NSLog(@"Shouty result: %@", [asyncResult uppercaseString]);
  }
  return 0;
}

This is more of a fork-and-join approach than fire and forget. The calling thread carries on running until it actually needs the result of the calculation, at which point it waits (if necessary) for the callee to complete before continuing. It’ll be familiar to programmers on other platforms as async/await.

The implementation is – blah blah awesome power of the runtime – a forwardInvocation: method that looks for messages with the async marker, patches their selector and creates a proxy object to invoke them in the background. That proxy is then written into the original invocation as its return value. Not shown: a pretty straightforward category implementing -[NSInvocation copyWithZone:].

@implementation Awaitable

- (SEL)suppliedSelectorForMissingSelector:(SEL)aSelector
{
  NSString *selectorName = NSStringFromSelector(aSelector);
  NSString *realSelectorName = nil;
  if ([selectorName hasPrefix:@"async_"]) {
    realSelectorName = [selectorName substringFromIndex:6];
  }
  return NSSelectorFromString(realSelectorName);
}

- (NSMethodSignature *)methodSignatureForSelector:(SEL)aSelector
{
  NSMethodSignature *aSignature =   [super methodSignatureForSelector:aSelector];
  if (aSignature == nil) {
    aSignature =    [super methodSignatureForSelector:[self suppliedSelectorForMissingSelector:aSelector]];
  }
  return aSignature;
}

- (void)forwardInvocation:(NSInvocation *)anInvocation
{
  SEL trueSelector =    [self suppliedSelectorForMissingSelector:[anInvocation selector]];
  NSInvocation *cachedInvocation = [anInvocation copy];
  [cachedInvocation setSelector:trueSelector];
  CBox *box = [CBox cBoxWithInvocation:cachedInvocation];
  [anInvocation setReturnValue:&box];
}

@end

Why is the proxy object called CBox? No better reason than that I built this while reading a reflection on Concurrent Smalltalk where that’s the name of this object too.

@interface CBox : NSProxy

+ (instancetype)cBoxWithInvocation:(NSInvocation *)inv;
- await;

@end

@implementation CBox
{
  NSInvocation *invocation;
  NSOperationQueue *queue;
}

+ (instancetype)cBoxWithInvocation:(NSInvocation *)inv
{
  CBox *box = [[self alloc] init];
  box->invocation = [inv retain];
  box->queue = [NSOperationQueue new];
  NSInvocationOperation *op = [[NSInvocationOperation alloc]    initWithInvocation:inv];
  [box->queue addOperation:op];
  return [box autorelease];
}

- init
{
  return self;
}

- await
{
  [queue waitUntilAllOperationsAreFinished];
  id returnValue;
  [invocation getReturnValue:&returnValue];
  return [[returnValue retain] autorelease];
}

- (void)dealloc
{
  [queue release];
  [invocation release];
  [super dealloc];
}

- forwardingTargetForSelector:(SEL)aSelector
{
  return [self await];
}

@end

You don’t always need some huge library to clean things up. Here are about 70 lines of Objective-C that abstract an implementation of concurrent programming and stop my application code from having to interweave the distinct responsibilities of what it’s trying to do and how it’s trying to do it.

It depends? It depends.

Sometimes you ask a question which has a small collection of actionable answers: yes or no. You ask someone who should be able to give that yes or no answer, and they go for the third: it depends.

Maybe they can’t actually answer your question, but want to sound profound.

Maybe they don’t realise that you’re a beginner in this particular area and would benefit from knowing what they’d do in this situation.

Maybe they don’t realise that you’re an expert in this area and are looking for confirmation or refutation of your existing choice.

Maybe they don’t realise that you’re uninterested in this area, and just need a decision made so you can move on.

Maybe they don’t know what the answer is, but are hoping you’ll help them think through the alternatives.

Sometimes, the answer to a question is “it depends”. But is that the answer you should give? It depends.

GNU Terry Pratchett

(post-hoc prescript: I admit to being in two minds about sharing this post. Name-dropping can be the ultimate in reflected vanity: I have worth because I knew this worthy person. I title it about them, but we both know it’s about me. I hope this post, containing much as it does about me and my experiences, is actually about what I learned from an Old Fart in a Hat – among others.)

GNU Terry Pratchett

Nearly three years ago, The Compleat Ankh-Morpork was approaching completion, and the publishers were excited about the potential to turn it into an interactive discovery of the Circle Sea’s cleanest cityest city. At the time that I joined Agant, Dave “OMG it’s Dave Addey off the internets” Addey had already been discussing the project with them, and much was ready to go.

Indeed, Dave and I were ready to go. We took a train down to the Discworld Emporium and met Sir Terry Pratchett, his partner in crime Rob Wilkins, their able Aiders and Abettors from the Emporium and some of the most happening among those who Make Things Happen at the publishers.

In the Dolphin Hotel at Wincanton, we had a pub lunch and tried to define exactly what it meant to put Ankh-Morpork into an iPad. I’d already made a start, and here I learned Lesson Two: the creative mind knows it can be done. You don’t need to convince them it can be done. It’s already been done, it’s up there in their mind’s eye. You just need to convince them it can be done well, and in this reality.

Wait, lesson two? Yes, for here was Lesson One: when meeting your heroes, only one of the two of you is likely to try and make a big deal out of it. The other wants their ploughman’s lunch and their G&T and to get back to work. Keeping focussed on the task at hand – a difficult feat as I was dealing with an internal monologue that would not stop going “squeeeeeee” – paid off, and everyone left safe in the knowledge that there was lots to do but that we would do it because it needed to be done and we needed to be the ones to do it and we needed it to be done well.

OK, most of the rest of this story is history. Dave, the developers at Agant, and a dedicated collection of others worked very hard for a number of months and eventually Discworld: the Ankh-Morpork Map for iPad oozed its way out of the alledged waters of the Ankh (there’s no need to disinfect it, the bacteria refuse to touch the stuff) and into the App Store. It remains the project of which I am most proud, both of the contributions I made and of all that was put in by everyone involved.

All good stories, though, come to an end, and Sir Pterry’s did last week. Today I went back over to the Emporium to talk a bit about the past and a lot about the future. It’s from today’s discussions that Lesson Three originates.

Some day, you will be invited by a polite though insistent anthropomorphic personification to take a walk together. Before he arrives, do the things that need to be done. Make sure they’re the things that must be done by you, and that you will be satisfied and pleased to see completed. Otherwise the two of you won’t have a whole lot to talk about.

Linus’s Bystanders

For some reason, when Eric S. Raymond wanted to make a point about the “bazaar” model of open source software development, he named it after someone else. Thus we have Linus’s Law:

Linus was directly aiming to maximize the number of person-hours thrown at debugging and development, even at the possible cost of instability in the code and user-base burnout if any serious bug proved intractable. Linus was behaving as though he believed something like this:

8. Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.

Or, less formally, “Given enough eyeballs, all bugs are shallow.” I dub this: “Linus’s Law”.

The “proof” of this law, such as it is, is a reductio ad absurdum painting the contradiction of the law as a universe in which the Linux kernel couldn’t be created.

If “Linus’s Law” is false, then any system as complex as the Linux kernel, being hacked over by as many hands as the that kernel was, should at some point have collapsed under the weight of unforseen bad interactions and undiscovered “deep” bugs. If it’s true, on the other hand, it is sufficient to explain Linux’s relative lack of bugginess and its continuous uptimes spanning months or even years.

Let us remember the time that Linux development did collapse under its own weight. Raymond’s original essay was written in 1997; by 2006 it seemed that “we’re adding bugs at a higher rate than we’re fixing them”. In between, more developers had (presumably) contributed, so what happened there? Why weren’t all of the bugs even shallower?

I’d like to examine a result from the social sciences here. From the Wikipedia (I am researcher, hear me roar) page on bystander effect:

The bystander effect, or bystander apathy, is a social psychological phenomenon that refers to cases in which individuals do not offer any means of help to a victim when other people are present. The probability of help is inversely related to the number of bystanders. In other words, the greater the number of bystanders, the less likely it is that any one of them will help.

And then, a few paragraphs later when talking about computer-mediated communication:

Research suggests that the bystander effect may be present in computer-mediated communication situations.

The experiments into bystander apathy are specifically about someone in distress, either asking for help or visibly in trouble. Maybe it applies here, too, when the only “distress” is that some software you didn’t pay for could be better than it is. Perhaps, even before we get into discussions of whether the projects have welcoming communities that accept contributors, our projects are putting people off just because they know that there are plenty of others out there who could fix these bugs before they do.

I’ve met software team leaders who, when asked how their techniques and processes could be made to scale to a team of 1,000, answer that they’d start by giving the project to a team of ten. Maybe they’re onto something.

Inspired by Swift

Gulliver meets the Oopers

Lemuel Gulliver’s world was black. No light, no sound, infinite darkness and solitude. Am I dead?, he asked himself. No, surely not. He opened his eyes.

Still, everything remained black. My God, I am dead! Lemuel Gulliver began to panic. Then, slowly, he realised that panicking itself meant life: My heart is pounding, he thought, therefore it must be beating. Life! Sweet, God-given, beautiful life! But what sort of life is it, to live as a man dead?

Memory failed him but in bursts, though those bursts helped him understand his predicament. Yes, a shipwreck. But he had got into a lifeboat, which had carried him precariously through the swell of the storm.

Of course, the lifeboat! Gulliver raised his hand, and laughed loudly to himself as it hit the planks. He was on land, and it was the upturned lifeboat had given the impression of his premature trip to Hades. He pushed with all of his strength, and shifted the boat over to lie beside him on the beach.

For minutes he lay still. His eyes needed to adjust, his arms to recover, and his mind to regain some composure. Once the blackness of the lifeboat and the whiteness of the sudden return to light had begun to balance, he appraised his condition. It was indeed a beach upon which he lay, marked with dunes and occasional tufts of a tall grassy plant. The water, which apparently had once been turbulent enough to deposit his unconscious self along with its inverted boat some way up the beach, now was smooth and still.

To his right, behind one of the dunes, he could make out rooves denoting that a settlement was not far away. One in particular, a great grey spire, stood out like a flagstaff. Realising that he needed water, food and assistance, Gulliver uneasily got up and started walking toward the town.

As he crested the dune he became aware of another walker, making her way along a path that would bring her closer and heading like him in the direction of the town. He got to the path and waited for her to approach.

“Hello, madam, I don’t mean to burden you but find myself in innumerable difficulties. I’m a traveller who has got himself shipwrecked in this country, and who finds himself in need of water, provisions and other assistance. Can you help me?”

The woman stopped, smiled at him, then executed a perfect curtsey before she replied. “Good day to you, sir, and is not the weather fine? My name is Kaye Allen, delighted to meet you. I’m sorry to hear about your problems and certainly want to help, please come with me to our home, where my husband and I will feed you and lodge you while you make your arrangements.”

Gulliver introduced himself and thanked her heartily for her generosity. Together they walked towards the town, the flagstaff spire always ahead of them. Kaye Allen was engrossed during much of the walk in the task of writing postcards: many of them addressed to different people each containing short messages. Gulliver asked about the spire, and she answered but not before finishing the current postcard, and packing away it and her pen.

“It is best that you not inquire about such things, sir. That building is associated with immorality and debasement, and it would not be becoming for an Ooper to associate with the sort found there.”

“An Ooper, madam? Pardon me but I have never encountered that word before. What is an Ooper?”

“Why, Mr. Gulliver, I am an Ooper, my husband is an Ooper, and indeed our whole community is comprised of Oopers. It is our religion, our philosophy, and our way of life. But let us not talk of that now. There will be time enough later. Let us now make our way onward, the afternoon draws on.”

The rest of their walk to the Allens’ house was mostly without event, though once a man approached them to ask the time. Gulliver thought there was something odd about Mrs. Allen’s reply, though he could not determine what it was as it seemed polite enough. She stopped, smiled at the man, then executed a perfect curtsey before she replied. “Good day to you, sir, and is not the weather fine? My name is Kaye Allen, delighted to meet you. I believe the time to be something after five of the clock, see the sun already begins to set.”

As the twilight drew in they arrived at last at the house. Kaye’s husband was already in, and he greeted Gulliver warmly. He smiled, then executed a perfect bow before saying “Good day to you, sir, and is not the weather fine? My name is Turmo Allen, delighted to meet you. Please, find the wash room upstairs and feel welcome to borrow some clothes, while we prepare dinner.”

Gulliver felt better for a wash and a change of clothes, but not so much improved as he did when they sat down to eat. While the food was simple, it was plentiful and he had no conception of how long it was since he had last eaten. The Allens were very precise about setting the table. Before each dish was served, the fork would be placed on the left of the setting just so. Then the knife to the right, just so. Then the water glass, just so, and filled with water from the pitcher, just so. Finally the dish was brought, placed just so in the centre of the setting, and they would sit, pick up first the fork then the knife, and begin eating.

Gulliver asked about this Oopers religion that Kaye had mentioned, and found Turmo to be most effusive on the topic.

“We Oopers live our lives according to the books of the lady Adele of the golden mountain. The legends say that many centuries ago, before the Oopers, the people of this region were simple backwards folk who believed strongly in superstition. They spent their time in constructing procedures for each other to follow, such procedures becoming more baroque and complicated with time. Even in emergencies and when under attack, if they had begun one of their procedures then needs must it be finished.

“One day, so we are told, a hot air balloon of many colours landed in the market place of this very town. The occupants declared themselves to be emissaries of a wonderful place, a tall tower of ivory on an island far away. They said that they too had previously been enslaved by their procedures, but that they had freed themselves through enlightenment and that they could teach this freedom to our forebears.

“One of their number was the lady Adele of the golden mountain, and it is she who wrote their teachings in the books that have come down through the generations, yea even to us, and taught us the lessons of these people of the balloon. You have seen my wife, Kaye, writing many postcards for her correspondents both near and far?”

“Why, Mr. Gulliver, I have even written one for you.”

“The people of the balloon taught us that complex procedures can be avoided, and our community can function together better, if we instead send each other messages. Our whole philosophy and way of life is based on this message-sending. It has delivered us from the oppression of our procedures, enlightened our society, and advanced us immeasurably.”

Gulliver interrupted at this point. “Thank you, sir, for your most edifying explanation, and thank you madam for your message. May I read it?”

Mrs. Allen laughed. “That would not do at all, Mr. Gulliver! Due process must be observed. See, I have gathered all of the cards that I have written so far today. Tomorrow morning I will take them to the post office, and there they will be despatched to their recipients. Including you! Once you have formally received your card, then you may read your message. Not before!”

“But is that not an example of the sort of procedure you say belonged to the simple times of your forebears? Is that not unnecessarily convoluted and slow, as you could simply tell me the message now or give me the card?”

“No, sir, it is not. We Oopers have done away with the procedure, and replaced it with this simple act of message sending.”

“Then why not give me your message now?”

“That would not do at all, Mr. Gulliver! Due process must be observed.”

Gulliver could sense his own frustration mounting, and thought it prudent to change the subject. Unfortunately in his haste he forgot Mrs. Allen’s reaction to his previous enquiries regarding the spire, and that is the topic to which he next turned.

“It is best for you if you avoid that place, sir,” Mr. Allen quickly responded. “It is a Church, a building associated with the debased Phuncers and their heretical acts. No true Ooper would go there, and it would not do for you to be tainted by their evil ways. Now let us talk no more of them.”

Gulliver recognised that he was not being the most gracious or entertaining of guests, so did not suggest any further metiere for their conversation. In the absence of his conversational selections there was nothing for the Allens to respond to, so the rest of the dinner proceeded in silence until they all departed to bed.

Escaping to the Phuncers

Morning breakfast passed much as evening dinner had. While the Allens were polite enough, Gulliver found it infuriating to deal not only with their strange, repetitive rituals but also with their insistence that they had left such procedure behind. He quizzed them upon this topic.

“You say that the Oopers have freed themselves from the shackles of procedure, yet I observe you both greet me using exactly the same set of rules. Do you not see that this itself is procedure?”

“No, sir, you miss the subtlety that was taught by Adele of the golden mountain. See, while we each say an identical greeting to you, my husband bows before doing it while my action is to curtsey. All people say the greeting, but specifically those people who are men perform a bow while those people who are women perform a curtsey. This ability to make more specific amendments is what liberated the Oopers.”

“But still, all you are doing is executing this procedure!”

“I wish, sir, that I had more time to discuss this with you. But now I must attend to my messages.”

With that, Kaye Allen returned to her postcards and Lemuel Gulliver, with more civility and politeness than he felt, thanked her for the hospitality he had received which in fact had been generous and left their abode hastily. His aim was to discover his location and how he might return home.

Once he was outside, his eyes were once again drawn to that strange spire, the Church by which his hosts had been so repulsed. Perhaps here he would find some useful information, or at least a different insight into this unique land. Resolved to learn more, he set out in the direction of the Church.

As he walked the mile or so to the tall building, he saw plenty of Oopers, recognised by their strange rituals and their stacks of message cards. The closer he came to his goal, however, the thinner their numbers, and when he finally arrived at the huge quadrangle outside the Church there was not one Ooper to be seen.

That is not to say that the square was empty, but clearly the people here were of a different background. They were gathered into groups, usually comprising four to six people, who criss-crossed the square purposefully in files. Perhaps these were the Phuncers he had been told about. Gulliver proceeded toward the Church to learn more.

Outwardly, the Church reminded him in passing of some of the religious institutions he remembered from his home. Indeed where his own house of prayer signed its identity with the Greek letters alpha and omega, this Church even had a similar symbol: the letter lambda. Confident that he was on somewhat familiar territory, he entered.

Gulliver started when, just inside the doorway before his eyes had adjusted to the gloom, he almost bumped into a file of a half dozen people who were on their way out. He mumbled an apology, then introduced himself.

“My name is Lemuel Gulliver, what is your name please?” If the lead person in the file heard him, she did not make it evident. She consulted a clipboard she was holding, made a note on it, then whispered something to the second person in the file. He did the same, and so on down the line. Finally the person at the back checked his clipboard, made a note, and replied “How do you do? My name is Alonso.”

Gulliver was not certain what had just happened. Perhaps the woman to whom he had addressed himself had not heard him clearly. He cleared his throat, and repeated: “My name is Lemuel Gulliver, what is your name please?” Again, she looked at her clipboard. This time, though, she apparently saw something that satisfied her (Gulliver could not be certain, but he thought she was looking at the note she had just made moments earlier) and showed that to the man behind her. He, in his turn, found the corresponding note on his board and showed it to the third. As before, it was the last of them who replied “How do you do? My name is Alonso.”

While this was happening, Gulliver had been able to take a look at the lead woman’s board. It was full of fragments of conversations, the left column apparently representing sentences that had been uttered to her (see! There is the very greeting I introduced myself with, written at the end of the page!) and so he guessed that the second column was the list of responses she had passed on to her colleague. Gulliver surmised that this tally represented an efficiency of sorts: by remembering all of her previous discussions, she could show her friends the answers she had previously given without having to construct them anew.

Conversation was, understandably, engaged at a slow pace. All the while Gulliver addressed himself to the woman at the front of the file; all the while she passed the discussion back through the chain of six; all the while Alonso (he assumed that this name referred to the ultimate member of the group) issued the responses.

“Tell me, is this the place of the Phuncers?”

“Why yes, this Church is our holy building, and you are welcome within it.”

“I hope to learn something of your people. Why is it that you carry on in these files?”

“One of the oldest tenets of the Phuncer creed is that a person has more capability as a member of a group than as an individual. By composing our different abilities we become more efficient.”

“I see, that is an interesting philosophy. When did it arise?”

Here, an interesting modification to the usual process occurred. As before, the question was passed back through the file and the notes were made, but the fourth person made a record, then put their board down, picked it up anew and read back that note. Making another, they repeated the process. And again. (Gulliver could just about see the board, and it seemed that they had written the symbols “1 +” over and over.) Finally satisfied, they passed the whisper on again until Alonso could give a response.

“It was six decades ago that our movement came into being.”

“What was the purpose behind its creation?”

“Back before we were liberated, the people of this part were oppressed by a government that forced people into performing all manner of complicated and perverse rituals and procedures. Our forebears sought to free us from these procedures, from the imperatives coming from the State.”

“And is your number many?” (Again, the repeated setting and resetting of the clipboard.)

“When first we started we were few. We formed a commune that sought to minimise State interference within its boundaries, and to achieve self-sufficience. Recently our number has become bolstered, and even those among the heathen Oopers have joined us and learned from our teachings.”

“And has this surge in membership changed your position?”

“We are now so numerous that our whole conception has changed. Rather than seeking to reduce interaction with the State, we intend to overthrow it completely. Yea, anarchy! But not the chaos that you may associate with it. Compositions of individuals each performing their prescribed function can yield a higher order.”

This last response had taken longer than usual for the group to formulate. Whereas before they had communicated with each other, one after the other, in building their response, they had needed outside help here. One of their file broke rank, and went out from the Church into the square. There, Gulliver saw that he approached a besuited woman, greeted her politely, then a complicated exchange of handshakes, papers and stamps ensued. When this procedure was complete he discussed something (Gulliver assumed the current conversation between himself and Alonso’s group) with her a while. On concluding this business he raised his hat, returned to the file and continued the whisper game.

“Who was that person with whom your colleague discoursed then?”

“She is a civil servant. We occasionally need to engage their help in carrying on our business.”

“You mean she represents the State?”

“Yes, quite so. They can be very helpful in our achievement of our aims.”

“But didn’t you say that you were planning to replace the State?”

“Planning? Yea, it is our very destiny! All trappings of State will be dissolved and a functional order, a higher order, will take its place.”

At this, Gulliver realised that he was not getting much further than in his discussions with the Oopers previously. He bade his interlocutors farewell, and returned to the square.

All around, he saw the Phuncers, and all around, they reminded him very much of the procedures that they, as the Oopers, claimed to have left behind. The way they followed each other around in files. The recording previous discussions in their clipboards. The convoluted discussions with civil servants.

They shared other traits with the Oopers: their claim that they had moved beyond the things that they still actually practiced every day, and the complete refusal to accept nor even see that this was the case. This gave Gulliver a start: maybe there was something both groups had in common, some shared memory that accounted for their commonalities? He had to find out more.

Gulliver unravels

As Gulliver pondered this strange town and its inhabitants, he found himself back in Ooper territory. This was unsurprising as there were many more Oopers than Phuncers that he had seen since his shipwreck. He also realised that he was at the University, with the buildings somewhat coarsely modeled on the ivory tower that the Allens had described to him. This presented perhaps the best opportunity he had to understand this place. He looked for signs and campus maps, and eventually located the library.

Realising that there were political implications to his research, he asked the librarian for the writings of Adele of the golden mountain and associated criticisms and commentaries. She happily met his greeting (a curtsey, then “Good day to you, sir, and is not the weather fine?”), went away and brought back a whole stack of books which he took to his desk. There were the books by Adele herself, and contemporary discussions such as “Oopism: An Evolutionary Approach” (some wag of a student had subtitled this “On the Origin of Classes by Means of Artificial Selection”) and Bertrand de la Tour’s “Oopist Construction”, which seemed to have been written by a contract lawyer. There were also more modern interpretations of the earlier texts, such as Cecil R. Martin’s “Clean Living” and “Clean Liver”.

But this was not enough to satiate Gulliver’s curiosity. Once the librarian had returned to her postcards, he paged through the catalogue to try and find anything on the Phuncers and their foundation. There was nothing explicit, but “Religion: Heresies” and “Politics: Anarchy” seemed appropriate places to look. He soon discovered a couple of likely tomes: “The Structure and Interpretation of Careful Phuncism” and “Concepts in Political Languages”.

And so Gulliver read. There was to much to read in one sitting, but he also didn’t want to risk the Ooper librarian seeing all of his material. He hid books behind shelves, rushed out to the University’s refectory to get a brief meal and slept in a corner of the campus. For three days he read, living like a hybrid of a hermit and a monk.

The more he read, the more he was shocked: he was reading the same thing in two different places! Each collection of material described the problems that had beset the town when the focus had been on procedures, and lay out a manifesto for moving beyond this situation. Each manifesto was broadly the same: break the procedures into focussed skills; find individuals who are good at those skills and give them the single responsibility of working in that skilled area; compose teams that can more efficiently complete large tasks by combining their skill areas.

At some point, then, the two had declared a holy war. Each knew that their own approach was correct and the other was heretical. But what was the source of their disagreement? Gulliver thought back to a previous encounter, in which two peoples considered each other mortal enemies due to their choice of which end of a boiled egg to eat first. At least, he thought to himself, those people disagreed on something.

Fun and games (with rewritten rules) in Objective-C

An object-oriented programming environment is not a set of rules. Programs do not need to be constructed according to the rules supplied by the environment. An object-oriented environment includes tools for constructing new rules, and programs can use these to good effect. Let’s build multiple method inheritance for Objective-C, to see a new set of rules.

The goal

Given three classes, A, B, and C:

@interface A : NSObject

-(NSInteger)a;
-(NSInteger)double:(NSInteger)n;

@end

@implementation A

-(NSInteger)a { return [self double:6]; }

@end

@interface B : NSObject

-(NSInteger)b;

@end

@implementation B

-(NSInteger)b { return 30; }

@end

@interface C : NSObject

-c;

@end

@implementation C

-(NSInteger)double:(NSInteger)a
{
  return a*2;
}

-c { return @([self a] + [self b]); }

@end

We want to find the value of a C instance’s c property. That depends on its values of a and b, but that class doesn’t have those methods. We should add them.

int main(int argc, const char * argv[]) {
  @autoreleasepool {
    C *c = [C new];
    [c addSuperclass:[A class]];
    [c addSuperclass:[B class]];
    NSLog(@"The answer is %@", c.c);
  }
}

We want the following output:

2015-02-17 20:23:36.810 Mixins[59019:418112] The answer is 42

Find a method from any superclass

Clearly there’s some chicanery going on here. I’ve changed the rules: methods are no longer simply being looked up in a single class. My instance of C has three superclasses: A, B and NSObject.

@interface NSObject (Mixable)

- (void)addSuperclass:(Class)aSuperclass;

@end

@implementation NSObject (Mixable)

-superclasses
{
  return objc_getAssociatedObject(self, "superclasses");
}

- (void)addSuperclass:(Class)aSuperclass
{
  id superclasses = [self superclasses]?:@[];
  id newSupers = [superclasses arrayByAddingObject:aSuperclass];
  objc_setAssociatedObject(self, "superclasses", newSupers, OBJC_ASSOCIATION_RETAIN_NONATOMIC);
}

- (Class)superclassForSelector:(SEL)aSelector
{
  __block Class potentialSuperclass = Nil;
  [[self superclasses] enumerateObjectsUsingBlock:^(Class aClass, NSUInteger idx, BOOL *stop) {
    if ([aClass instancesRespondToSelector:aSelector])
    {
      potentialSuperclass = aClass;
      *stop = YES;
    }
  }];
  return potentialSuperclass;
}

- (NSMethodSignature *)original_methodSignatureForSelector:(SEL)aSelector
{
  NSMethodSignature *signature = [self original_methodSignatureForSelector:aSelector];
  if (signature)
  {
    return signature;
  }
  Class potentialSuperclass = [self superclassForSelector:aSelector];
  return [potentialSuperclass instanceMethodSignatureForSelector:aSelector];
}

- (void)forwardInvocation:(NSInvocation *)anInvocation
{
  SEL aSelector = [anInvocation selector];
  Class potentialSuperclass = [self superclassForSelector:aSelector];
  [anInvocation invokeSuperImplementation:potentialSuperclass];
}

+ (void)load
{
  if (self == [NSObject class])
  {
    method_exchangeImplementations(class_getInstanceMethod(self, @selector(original_methodSignatureForSelector:)),
                                   class_getInstanceMethod(self, @selector(methodSignatureForSelector:)));
  }
}

@end

Now invoke that method.

When you write [super foo], the Objective-C runtime needs to send a message to your object but tell the resolution machinery to look at the superclass for the method implementation, not at the current class. It uses a function objc_msgSendSuper to do this. In this case, I don’t have the superclass: I have a superclass, one of potentially many. So what I need to do is more general than what messaging super does.

Luckily for me, objc_msgSendSuper is already sufficiently general. It receives a pointer to self, just like the usual objc_msgSend, but in addition it receives a pointer to the class to be used as the superclass. By controlling that class pointer, I can tell the system which superclass to use.

A category on NSInvocation calls objc_msgSendSuper with the appropriate arguments to get the correct method from the correct class. But how can it call the function correctly? Objective-C messages could receive any number of arguments of any type, and return a value of any type. Constructing a function call when the parameters are discovered at runtime is the job of libffi, which is used here (not shown: a simple, if boring, map of Objective-C value encodings to libffi type descriptions).

@interface NSInvocation (SuperInvoke)

-(void)invokeSuperImplementation:(Class)superclass;

@end

@implementation NSInvocation (SuperInvoke)

- (BOOL)isVoidReturn
{
  return (strcmp([[self methodSignature] methodReturnType], "v") == 0);
}

-(void)invokeSuperImplementation:(Class)superclass
{
  NSMethodSignature *signature = [self methodSignature];
  if (superclass)
  {
    struct objc_super super_class = { .receiver = [self target],
      .super_class = superclass };
    struct objc_super *superPointer = &super_class;
    ffi_cif callInterface;
    NSUInteger argsCount = [signature numberOfArguments];
    ffi_type **args = malloc(sizeof(ffi_type *) * argsCount);
    for (int i = 0; i < argsCount; i++) {
      args[i] = [self ffiTypeForObjCType:[signature getArgumentTypeAtIndex:i]];
    }
    ffi_type *returnType;
    if ([self isVoidReturn]) {
      returnType = &ffi_type_void;
    }
    else {
      returnType = [self ffiTypeForObjCType:[signature methodReturnType]];
    }
    ffi_status status = ffi_prep_cif(&callInterface,
                                     FFI_DEFAULT_ABI,
                                     (unsigned int)[signature numberOfArguments],
                                     returnType,
                                     args);
    if (status != FFI_OK) {
      NSLog(@"I can't make an FFI frame");
      free(args);
      return;
    }

    void *argsBuffer = malloc([signature frameLength]);
    int cursor = 0;
    cursor += args[0]->size;
    void **values = malloc([signature numberOfArguments] * sizeof(void  *));
    values[0] = &superPointer;

    for (int i = 1; i < [signature numberOfArguments]; i++) {
      values[i] = (argsBuffer + cursor);
      [self getArgument:values[i] atIndex:i];
      cursor += args[i]->size;
    }
    if ([self isVoidReturn]) {
      ffi_call(&callInterface, objc_msgSendSuper, NULL, values);
    } else {
      void *result = malloc(returnType->size);
      ffi_call(&callInterface, objc_msgSendSuper, result, values);
      [self setReturnValue:result];
      free(result);
    }
    free(args);
    free(values);
    free(argsBuffer);
  }
}

@end

Conclusion

You’ve seen this conclusion before: blah blah awesome power of the runtime. It doesn’t just let you do expressive things in the Objective-C game, it lets you define a better game.

Objective-Curry

Sadly it’s not called Schoenfinkeling, but that’s the name of the
person who noticed that there’s no reason to ever have a function with
more than one argument. What you think is a function with two
arguments is actually a function with one argument that returns a
function with one argument, this second function closing over the
first argument and acting perhaps on both.

This is, of course, famed for its application (pardon the pun) to Object-Oriented
Programming. We can take a class like this:

@interface NamePrinter : Curryable

- (void)printFirstName:(NSString *)first surname:(NSString *)last;
- (void)printFirstName:(NSString *)first age:(NSInteger)age;
- (void)printFirstName:(NSString *)first middle:(NSString *)middle surname:(NSString *)last;

@end

and use it like this:

int main(int argc, char *argv[]) {
  @autoreleasepool {
    id printer = [[NamePrinter new] autorelease];
    id curried = [printer printFirstName:@"Graham"];
    [curried surname:@"Lee"];
    [curried surname:@"Greene"];
    [curried surname:@"Garden"];
    [curried age:18];

    id alex = [printer printFirstName:@"Alexander"];
    id alexG = [alex middle:@"Graham"];
    [alexG surname:@"Bell"];
  }
}

(your compiler probably complains at this point that it doesn’t know
what you’re up to. Of course, you know better.)

We’ll get results like this:

2015-02-13 00:57:57.421 CurrySauce[41877:134228] Graham Lee
2015-02-13 00:57:57.421 CurrySauce[41877:134228] Graham Greene
2015-02-13 00:57:57.421 CurrySauce[41877:134228] Graham Garden
2015-02-13 00:57:57.422 CurrySauce[41877:134228] Graham is 18 years old
2015-02-13 00:57:57.422 CurrySauce[41877:134228] Alexander Graham Bell

OK, we don’t actually get that for free. There’s some secret sauce I
haven’t shown you: secret curry sauce.

Here be dragons

As with
all the best bits of Objective-C,
you need to turn off the automatic reference counting to do what
follows (you’ll have already seen a call to -autorelease above). ARC
works wonders when you try to write Java in Objective-C, give or take
the strong-weak-strong tango. It isn’t so helpful when you try to
write Objective-C in Objective-C.

Partial application

The NamePrinter class knows that you might call it with a partial
application, so it checks whether the selector you sent in your
message matches the prefix of any method it knows. If it does, and the
return type and argument types in this prefix can be uniquely
determined, then it creates a proxy to listen for the rest of the
application.

The restrictions on types are there due to the way that Objective-C
deals with C types in message sending. It’d be a lot easier to do this
in Ruby or Smalltalk, where everything is an object reference: as
Objective-C needs to deal with C types of different sizes, you must be
able to construct a single invocation from the prefix selector.

int argumentsForSelector(SEL aSelector)
{
  const char *name = sel_getName(aSelector);
  int count = 0;
  while(*name != '\0') {
    if (*name++ == ':') {
      count++;
    }
  }
  return count;
}

@implementation Curryable

- (NSMethodSignature *)methodSignatureForSelector:(SEL)aSelector
{
  NSMethodSignature *superSignature = [super methodSignatureForSelector:aSelector];
  if (superSignature) {
    return superSignature;
  }
  NSString *thisSelectorName = NSStringFromSelector(aSelector);
  NSMutableSet *signatures = [NSMutableSet set];
  int argCount = argumentsForSelector(aSelector);
  Class currentClass = [self class];
  while (currentClass != Nil) {
    unsigned int methodCount = 0;
    Method *methodList = class_copyMethodList(currentClass, &methodCount);
    for (int i = 0; i < methodCount; i++) {
      Method method = methodList[i];
      SEL anotherSelector = method_getName(method);
      NSString *selectorName = NSStringFromSelector(anotherSelector);
      if ([selectorName hasPrefix:thisSelectorName]) {
        NSMethodSignature *fullSignature = [self methodSignatureForSelector:anotherSelector];
        NSMutableString *constructedTypeSignature = [[@"@@:" mutableCopy] autorelease];
        for (int j = 2; j < argCount + 2; j++) {
          [constructedTypeSignature appendString:@([fullSignature getArgumentTypeAtIndex:j])];
        }
        [signatures addObject:[[constructedTypeSignature copy] autorelease]];
      }
    }
    free(methodList);
    currentClass = class_getSuperclass(currentClass);
  }
  if ([signatures count] != 1) {
    NSLog(@"curried selector does not uniquely match the type of any full selector prefix");
    return nil;
  }
  return [NSMethodSignature signatureWithObjCTypes:[[signatures anyObject] UTF8String]];
}

- (void)forwardInvocation:(NSInvocation *)anInvocation
{
  id curry = [CurrySauce curryTarget:self invocation:anInvocation];
  [anInvocation setReturnValue:&curry];
}

@end

All of that, just to return a proxy object.

Finish the application

This proxy also responds to unknown selectors, by trying to tack
them onto the end of the partially-applied selector. If that works,
and it ends up with a selector that the target recognises, then it
constructs the combined invocation, copies the arguments from the two
partial invocations, and invokes it on the target. If the combined
selector is supposed to return something then this proxy unpacks the
return value and puts it into the invocation it received, to ensure
that the caller picks it up.

SEL concatenateSelectors(SEL firstSelector, SEL secondSelector)
{
  NSString *firstPart = NSStringFromSelector(firstSelector);
  NSString *selectorName = [firstPart stringByAppendingString:NSStringFromSelector(secondSelector)];
  return NSSelectorFromString(selectorName);
}

@implementation CurrySauce
{
  id target;
  NSInvocation *invocation;
}

-initWithTarget:object invocation:(NSInvocation *)partialApplication
{
  target = [object retain];
  invocation = [partialApplication retain];
  return self;
}

+curryTarget:object invocation:(NSInvocation *)partialApplication
{
  return [[[self alloc] initWithTarget:object invocation:partialApplication] autorelease];
}

- (NSMethodSignature *)methodSignatureForSelector:(SEL)aSelector
{
  //special case for respondsToSelector
  if (aSelector == @selector(respondsToSelector:)) {
    return [target methodSignatureForSelector:aSelector];
  }
  SEL combinedSelector = concatenateSelectors([invocation selector], aSelector);
  NSMethodSignature *combinedSignature = [target methodSignatureForSelector:combinedSelector];
  if (combinedSignature != nil) {
    NSMutableString *completionType = [NSMutableString stringWithFormat:@"%s@:",[combinedSignature methodReturnType]];
    for (int i = argumentsForSelector([invocation selector]) + 2; i < argumentsForSelector(combinedSelector) + 2; i++) {
      [completionType appendFormat:@"%s",[combinedSignature getArgumentTypeAtIndex:i]];
    }
    return [NSMethodSignature signatureWithObjCTypes:[completionType UTF8String]];
  } else {
    return nil;
  }
}

- (void)forwardInvocation:(NSInvocation *)anInvocation
{
  if ([anInvocation selector] == @selector(respondsToSelector:)) {
    [anInvocation invokeWithTarget:target];
    return;
  }
  SEL realSelector = concatenateSelectors([invocation selector], [anInvocation selector]);
  NSMethodSignature *signature = [target methodSignatureForSelector:realSelector];
  NSInvocation *combined = [NSInvocation invocationWithMethodSignature:signature];
  int argumentToSet = 2;
  void *argBuffer = malloc([[invocation methodSignature] frameLength]);
  for (int i = 2; i < [[invocation methodSignature] numberOfArguments]; i++) {
    [invocation getArgument:argBuffer atIndex:i];
    [combined setArgument:argBuffer atIndex:argumentToSet++];
  }
  free(argBuffer);
  argBuffer = malloc([[anInvocation methodSignature] frameLength]);
  for (int i = 2; i < [[anInvocation methodSignature] numberOfArguments]; i++) {
    [anInvocation getArgument:argBuffer atIndex:i];
    [combined setArgument:argBuffer atIndex:argumentToSet++];
  }
  free(argBuffer);

  [combined setTarget:target];
  [combined setSelector:realSelector];
  [combined invoke];
  if (strcmp([[combined methodSignature] methodReturnType], "v")) {
    void *returnBuffer = malloc([[combined methodSignature] methodReturnLength]);
    [combined getReturnValue:returnBuffer];
    [anInvocation setReturnValue:returnBuffer];
    free(returnBuffer);
  }
}

- (void)dealloc
{
  [target release];
  [invocation release];
  [super dealloc];
}

@end

Conclusion

Blah blah awesome power of the runtime, I suppose. It’s pretty cool
that you can do this sort of thing in 150 lines of ObjC. I doubt many
people would want to use it for reals though.