Structure and Interpretation of Computer Programmers

On October 19th, 1994 NeXT Computer, Inc. (later NeXT Software, Inc.) published a specification for OpenStep, a cross-platform interface for application programming, based on their existing Objective-C frameworks and the Display PostScript graphics system.

A little bit of history

First there came message-passing object oriented programming, in the form of Smalltalk. Well, not first, I mean first there was Simula 67, and there were even things before that but every story has to start somewhere. In 1983 Brad Cox added Smalltalk messaging to the C language to create the Object-Oriented pre-compiler. In his work with Tom Love at Productivity Products International, this eventually became Objective-C.

If PPI (later Stepstone) had any larger customers than NeXT, they had none that would have a bigger impact on the software industry. In 1988 NeXT released the first version of their UNIX platform, NEXTSTEP. Its application programming interface combined the “application kit” of Objective-C objects representing windows, menus, and views with Adobe’s Display PostScript to provide a high-fidelity (I mean, if you like grey, I suppose) WYSIWYG app environment.

N.B.: my reason for picking the Garfinkel and Mahoney book will become clear later. It happens to be the book I learned to make apps from, too.

Certain limitations in the NEXTSTEP APIs became clear. I will not exhaustively list them nor attempt to put them into any sort of priority, suffice it to say that significant changes became necessary. When the Enterprise Objects Framework came along, NeXT also introduced the Foundation Kit, a “small set of base utility classes” designed to promote common conventions, portability and enhanced localisation through Unicode support. Hitherto, applications had used C strings and arrays.

It was time to let app developers make use of the Foundation Kit. For this (and undoubtedly other reasons), the application kit was rereleased as the App Kit, documented in the specification we see above.

The release of OpenStep

OpenStep was not merely an excuse to do application kit properly, it was also part of NeXT’s new strategy to license its software and tools to other platform vendors rather than limiting it to the few tens of thousands of its own customers. Based on the portable Foundation Kit, NeXT made OpenStep for its own platform (now called OPENSTEP) and for Windows NT, under the name OpenStep Enterprise. Sun Microsystems licensed it for SPARC Solaris, too.

What happened, um, NeXT

The first thing to notice about the next release of OpenStep is that book cover designers seem to have discovered acid circa 1997.

Everyone’s probably aware of NeXT’s inverse takeover of Apple at the end of 1996. The first version of OpenStep to be released by Apple was Rhapsody, a developer preview of their next-generation operating system. This was eventually turned into a product: Mac OS X Server 1.0. Apple actually also released another OpenStep product: a y2k-compliant patch to NeXT’s platform in late 1999.

It’s kindof tempting to tell the rest of the story as if the end was clear, but at the time it really wasn’t. With Rhapsody itself it wasn’t clear whether Apple would promote Objective-C for OpenStep (now called “Yellow Box”) applications, or whether they would favour Yellow Box for Java. The “Blue Box” environment for running Mac apps was just a virtual machine with an older version of the Macintosh system installed, there wasn’t a way to port Mac software natively to Rhapsody. It wasn’t even clear whether (or if so, when) the OpenStep software would become a consumer platform, or whether it was destined to be a server for traditional Mac workgroups.

That would come later, with Mac OS X, when the Carbon API was introduced. Between Rhapsody and Mac OS X, Apple introduced this transition framework so that “Classic” software could be ported to the new platform. They also dropped one third of the OpenStep-specified libraries from the system, as Adobe’s Display PostScript was replaced with Quartz and Core Graphics. Again, reasons are many and complicated, though I’m sure someone noticed that if they released Mac OS X with the DPS software then their bill for Adobe licences would increase by a factor of about 1,000. The coloured box naming scheme was dropped as Apple re-used the name of their stagecast creator software: Cocoa.

So it pretty much seemed at the time like Apple were happy to release everything they had: UNIX, Classic Mac, Carbon, Cocoa-ObjC and Cocoa-Java. Throw all of that at the wall and some of it was bound to stick.

Over time, some parts indeed stuck while others slid off to make some sort of yucky mess at the bottom of the wall (you know, it is possible to take an analogy too far). Casualties included Cocoa-Java, the Classic runtime and the Carbon APIs. We end in a situation where the current Mac platform (and, by slight extension, iOS) is a direct, and very close, descendent of the OpenStep platform specified on this day in 1994.

Happy birthday, Cocoa!

I was asked “what books do you consider essential for app making”? Here’s the list. Most of these are not about specific technologies, which are fleeting and teach low-level detail. Those that are tech-specific also contain a good deal of what and why, in addition to the coverage of how.

This post is far from exhaustive.

I would recommend that any engineer who has not yet read it should read Code Complete 2. Then I would ask them the top three things they agreed with and top three they disagreed with, as criticality is the hallmark of a good engineer :-).

Other books I have enjoyed and learned from and I believe others would too:

• Steve Krug, “Don’t make me think!”
• Michael Feathers, “Refactoring” and “Working Effectively with Legacy Code”
• Bruce Tate, “Seven languages in seven weeks”
• Jez Humble and David Farley, “Continuous Delivery”
• Hunt and Thomas, “The Pragmatic Programmer”
• Gerald Weinberg, “The psychology of computer programming”
• David Rice, “Geekonomics”
• Robert M. Pirsig, “Zen and the art of motorcycle maintenance”
• Alan Cooper, “About Face 3”
• Jeff Johnson, “Designing with the mind in mind”
• Fred Brooks, “the design of design”
• Kent Beck, “Test-Driven Development”
• Mike Cohn, “User stories applied”
• Jef Raskin, “The humane interface”

Most app makers are probably doing object-oriented programming. The books that explain the philosophy of this and why it’s important are Meyer’s “Object-oriented software construction” and Cox’s “Object-oriented programming an evolutionary approach”.

Members of comfortable societies such as English towns have expectations of the services they will receive. They want their rubbish disposed of before it builds up too much, for example. They don’t so much care how it’s dealt with, they just want to put the rubbish out there and have it taken away. They want electricity supplied to their houses, it doesn’t so much matter how as long as the electrons flow out of the sockets and into their devices.

Some people do care about the implementation, in that they want it to be far enough away from it not to have to pay it any mind. These people are known as NIMBYs, after the phrase Not In My Back Yard. Think what it will do to traffic/children walking to school/the skyline/property prices etc. to have this thing I intend to use near my house!

A NIMBY wants to have their rubbish taken away, but does not want to be able to see the landfill or recycling centre during their daily business. A NIMBY wants to use electricity, but does not want to see a power station or wind turbine on their landscape.

What does this have to do with software? Modules in applications (which could be—and often are—objects) should be NIMBYs. They should want to make use of other services, but not care where the work is done except that it’s nowhere near them. The specific where I’m talking about is the execution context. The user interface needs information from the data model but doesn’t want the information to be retrieved in its context, by which I mean the UI thread. The UI doesn’t want to wait while the information is fetched from the model: that’s the equivalent of residential traffic being slowed down by the passage of the rubbish truck. Drive the trucks somewhere else, but Not In My Back Yard.

There are two ramifications to this principle of software NIMBYism. Firstly, different work should be done in different places. It doesn’t matter whether that’s on other threads in the same process, scheduled on work queues, done in different processes or even on different machines, just don’t do it anywhere near me. This is for all the usual good reasons we’ve been breaking work into multiple processes for forty years, but a particularly relevant one right now is that it’s easier to make fast-ish processors more numerous than it is to make one processor faster. If you have two unrelated pieces of work to do, you can put them on different cores. Or on different computers on the same network. Or on different computers on different networks. Or maybe on the same core.

The second is that this execution context should never appear in API. Module one doesn’t care where module two’s code is executed, and vice versa. That means you should never have to pass a thread, an operation queue, process ID or any other identifier of a work context between modules. If an object needs its code to run in a particular context, that object should arrange it.

Why do this? Objects are supposed to be a technique for encapsulation, and we can use that technique to encapsulate execution context in addition to code and data. This has benefits because Threading Is Hard. If a particular task in an application is buggy, and that task is the sole responsibility of a single class, then we know where to look to understand the buggy behaviour. On the other hand, if the task is spread across multiple classes, discovering the problem becomes much more difficult.

NIMBY Objects apply the Single Responsibility Principle to concurrent programming. If you want to understand surprising behaviour in some work, you don’t have to ask “where are all the places that schedule work in this context?”, or “what other places in this code have been given a reference to this context?” You look at the one class that puts work on that context.

The encapsulation offered by OOP also makes for simple substitution of a class’s innards, if nothing outside the class cares about how it works. This has benefits because Threading Is Hard. There have been numerous different approaches to multiprocessing over the years, and different libraries to support the existing ones: whatever you’re doing now will be replaced by something else soon.

NIMBY Objects apply the Open-Closed Principle to concurrent programming. You can easily replace your thread with a queue, your IPC with RPC, or your queue with a Disruptor if only one thing is scheduling the work. Replace that one thing. If you pass your multiprocessing innards around your application, then you have multiple things to fix or replace.

There are existing examples of patterns that fit the NIMBY Object description. The Actor model as implemented in Erlang’s processes and many other libraries (and for which a crude approximation was described in this very blog) is perhaps the canonical example.

Android’s AsyncTask lets you describe the work that needs doing while it worries about where it needs to be done. So does IKBCommandBus, which has been described in this very blog. Android also supports a kind of “get off my lawn” cry to enforce NIMBYism: exceptions are raised for doing (slow) network operations in the UI context.

There are plenty of non-NIMBY APIs out there too, which paint you into particular concurrency corners. Consider -[NSNotificationCenter addObserverForName:object:queue:usingBlock:] and ignore any “write ALL THE BLOCKS” euphoria going through your mind (though this is far from the worst offence in block-based API design). Notification Centers are for decoupling the source and sink of work, so you don’t readily know where the notification is coming from. So there’s now some unknown number of external actors defecating all over your back yard by adding operations to your queue. Good news: they’re all being funnelled through one object. Bad news: it’s a global singleton. And you can’t reorganise the lawn because the kids are on it: any attempt to use a different parallelism model is going to have to be adapted to accept work from the operation queue.

By combining a couple of time-honoured principles from OOP and applying them to execution contexts we come up with NIMBY Objects, objects that don’t care where you do your work as long as you don’t bother them with it. In return, they won’t bother you with details of where they do their work.

You can find plenty of dogmatic positions in software development, in blogs, in podcasts, in books, and even in academic articles. “You should (always/never) write tests before writing code.” “Pair programming is a (good/bad) use of time.” “(X/not X) considered harmful.” “The opening brace should go on the (same/next) line.”

Let us ignore, for the moment, that only a maximum of 50% of these commandments can actually be beneficial. Let us skip past the fact that demonstrating which is the correct position to take is fraught with problems. Instead we shall consider this question: dogmatic rules in software engineering are useful to whom?

The Dreyfus model of skill acquisition tells us that novices at any skill, not just programming, understand the skill in only a superficial way. Their recollection of rules is non-situational; in other words they will try to apply any rule they know at any time. Their ability to recognise previously penchant free scenarios is small-scale, not holistic. They make decisions by analysis.

The Dreyfus brothers proposed that the first level up from novice was competence. Competents have moved to a situational recollection of the rules. They know which do or do not apply in their circumstances. Those who are competent can become proficient, when their recognition becomes holistic. In other words, the proficient see the whole problem, rather than a few small disjointed parts of it.

After proficiency comes expertise. The expert is no longer analytical but intuitive, they have internalised their craft and can feel the correct way to approach a problem.

“Rules” of software development mean nothing to the experts or the proficient, who are able to see their problem for what it is and come up with a context-appropriate solution. They can be confusing to novices, who may be willing to accept the rule as a truth of our work but unable to see in which contexts it applies. Only the competent programmers have a need to work according to rules, and the situational awareness to understand when the rules apply.

But competent programmers are also proficient programmers waiting to happen. Rather than being given more arbitrary rules to follow, they can benefit from being shown the big picture, from being led to understand their work more holistically than as a set of distinct parts to which rulaes can be mechanistically – or dogmatically – applied.

Pronouncements like coding standards and methodological commandments can be useful, but not on their own. By themselves they help competent programmers to remain competent. They can be situated and evaluated, to help novices progress to competence. They can be presented as a small part of a bigger picture, to help competents progress to proficiency. As isolated documents they are significantly less valuable.

Dogmatism considered harmful.