Messily Veneered C

A recap: we saw that Model-View-Controller started life as Thing-Model-View-Editor, a way of approaching problems to design Smalltalk user interfaces. As Smalltalk-80 drifted off from its ivory tower, many Smalltalkers were using and talking about MVC, although any kind of consensus on its meaning was limited by the lack of important documentation.

Would this telephone game continue to redefine Model-View-Controller as object-oriented programming grew beyond the Smalltalk-80 community?

Part 3: Objective-MVC

Objective-C is now best known as the programming language Apple promotes for developing iOS and Mac applications. It was first available on the Mac during the 1980s pre-Cambrian explosion of OOP languages: on the Mac alone you could send messages using Smalltalk-80, Object Pascal, Objective-C, ExperCommon Lisp, Allegro Common Lisp, Object Assembler, Object Logo, and there were probably more. Most of the list here was derived from Object-Oriented Programming for the Macintosh, written by Kurt J. Schmucker then of Productivity Products International.

Which brings us back to Objective-C, which is probably the best known of the products produced by Productivity Products. Objective-C did not let programmers use a spatial programming interface, it made them use linear text files.

Objective-C also did not follow the Model-View-Controller paradigm of Smalltalk-80. As described by Brad Cox in this figure adapted from Object-Oriented Programming: an Evolutionary Approach, the PPI programmers were thinking about Models and Views, and then about reusing Views in different UIs.

Objective-C Layer Architecture

So, what happened to the Controllers? Here’s Cox:

This architecture is very similar to the one used in Smalltalk-80, with one exception. […] The outgoing leg of Smalltalk’s user interface is handled by a hierarchy of views much like the ones discussed here. But the incoming leg is implemented by a separate hierarchy of classes, Controllers, that provide the control for the application. […] The need for the separate controller hierarchy is unclear and is the topic of spirited debate even within the Smalltalk-80 community.

Unfortunately the citation for “spirited debate” is personal communication with various Smalltalkers, so we may never know the content.

This gives us three Objective-C modifications to the original Smalltalk-80 concept that persist to this day. One is the move from a user interface paradigm concerning the interactions of individual objects to a layered architecture. The Cox book shows the display screen as a projection of the view layer onto glass, and the models all in their own two-dimensional layer suspended from the views, hanging:

suspended from the presentation level by pointers in much the way that circuit boards are connected to test equipment with a bed-of-nails testing jig.

That still persists in Cocoa MVC, though with a somewhat weak assertion:

The collection of objects of a certain MVC type in an application is sometimes referred to as a layer—for example, model layer.

The second thing is the abstraction of the Presentation layer away from the display primitives.

The other thing that we still don’t have in ObjC is the Controller. Wait, what? Surely Objective-C applications have Controllers. Apple and NeXT both talked about it, and they have objects with the name “Controller” in them. Surely that’s MVC.

For whatever reason, NeXT kept the ObjC-style Views with their handling of both input and output, but also had to reintroduce the idea of a Controller. So they took the same approach followed by others, notably including Ivar Jacobson a few years later, of defining the Controller layer to contains whatever’s left once you’ve worked out what goes into the Model and View layers.

Now there’s only the question of placement. The Controller can’t go where Smalltalk’s Controllers went, because then it would get in the way of the input events which now need to get to the View. It could go between the two:

A controller object acts as an intermediary between one or more of an application’s view objects and one or more of its model objects. Controller objects are thus a conduit through which view objects learn about changes in model objects and vice versa. Controller objects can also perform setup and coordinating tasks for an application and manage the life cycles of other objects.

Here’s a reproduction of the diagram from that document:

Cocoa MVC

Notice that for the Controllers to act as a mediator between Models and Views, the Views actually have to forward action messages on to the Controllers (via delegation, target-action or the responder chain). Had the Controller been left where it was it would already be receiving those events.

Compare that with the diagram derived from Smalltalk MVC:

Smalltalk MVC

It’s clear that these are different concepts that unfortunately share a name.

The Objective-C protocol naming trifecta

Objective-C protocol names throughout history seem to fall into three distinct conventions:

  • some are named after what a conforming object provides. Thus we have DBProperties, DBEntities, DBTypes and the like in Database Kit.
  • others are named after what the object is doing. Thus we have NSCoding, NSLocking, IXCursorPositioning etc.
  • still others are named after what the conforming object is. NSTableViewDataSource, NSDraggingInfo etc.

I think I prefer the second one. It emphasises the dynamic message-sending nature of the language: “because you’re sending messages from this protocol, I’ll act like this”.

Missing Vital Content

After reading Moderately Valuable Cliché, reader Nicholas Levin got in touch to recommend that I look at the back of my Smalltalk-80 books. Here’s the blue one.

Books in the Smalltalk-80 series

The first book mentioned in “Other books in the … Series” is the red book, “Smalltalk-80: The Interactive Programming Environment”. The third is the green book, “Smalltalk-80: Bits of History, Words of Advice”. But the middle one? “Smalltalk-80: Creating a User Interface and Graphical Applications”? No-one ever assigned that one a colour. It was never published.

Discussions of the book in comp.lang.smalltalk in 1986 indicate that it was a manifesto for Model-View-Controller, promoting and describing the pattern for building Smalltalk applications. The thread instead recommends looking at Chapter 9 of a book that has been mentioned on this blog before: Object-Oriented Programming: an evolutionary approach by Brad Cox. More on that later.

So the idea behind MVC was well-known in the Smalltalk user community, but the principle was not documented by the Smalltalk team themselves (as far as I know: it’s possible there are documents I haven’t found) until Adele Goldberg left and founded ParcPlace with others at the end of the 1980s.

Levin also pointed me at a couple of other resources. In this interview for the Computer History Museum, Adele Goldberg confirms that the missing book was “to be an applications level book which was basically the model-view-controller metaphor”.

In an interview for the IEEE Global History Network, Goldberg explains how the whole concept of the Dynabook was to allow people to model real-world problems on their portable computers:

That’s what we thought it was, a PDP-10, which is lightweight compared to what we have now. It was going to be there, like a notepad, but it was going to be a computational-based device that allowed you to build models of the world and test your understanding of the world. What students who succeed really do is, they interact with teachers and parents and other people—and it could be other students—who challenge the models they build of their world; and this is true whether it’s science, math, or social studies: that you’re constantly constructing and deconstructing. And when someone really pushes you to think about your models, make the models explicit, talk about those models, and be able to build those models and then challenge those models—I mean, every well educated person I know remembers in their childhood that that was what was going on all the time.

In other words, Smalltalk is the programming equivalent of “[taking] apart engines and [putting] them back together again, to understand how things work”.

The long time over which the books were written and the fact that the system was being designed in reaction to problems found in documenting it (“In order to get the language book done we kind of organized ourselves so that Dan Ingalls and I, we’d have an aspect of the system we’d need to agree on like, what would be the collection classes? How would you provide collections?”) explains why the blue book’s FinancialHistory example can have no MVC, the red book briefly mentions MVC, and the authors of the ParcPlace paper, looking back, can remember it as a key part of the sample. Perhaps that example was eventually fleshed out for the unprinted book, or was developed further for another reason beyond the publication of the blue book.

We also have a possible reason for the subsequent diversity of interpretations of MVC among applications developers. With no primary reference for the original intention behind MVC, as the Addison-Wesley book was unwritten, people guessed at what Model-View-Controller meant from the bits and pieces that were published or discussed in the community.

Moderately Valuable Cliché

In part 1 of the MVC story, I examined “Thing-Model-View-Editor”, a pattern[*] extracted by Trygve Reenskaug’s work in Smalltalk-76. By the time Smalltalk-80’s hot air balloon set sail from the ivory tower, there was already a structure called Model-View-Controller.

Part Two: MVC and Smalltalk

[*] Bear in mind that I’m retroactively applying the word “pattern” here. Design Patterns as a thing, modelled on the book modelled on the other book, do not belong in this story for another 15 years or so.

This story will be told through the lens of a single overview article: A Description of the Model-View-Controller User Interface Paradigm in the Smalltalk-80 System by Krasner and Pope is the first. The authors give their affiliation as “ParcPlace Systems, Inc.”, a company that built tools on top of Smalltalk. It was founded by ACM president Adele Goldberg, who originally developed Smalltalk along with Alan Kay and others.


Krasner and pope say that MVC is an application of a particular three-way factoring of a software problem into objects:

separating (1) the parts that represent the model of the underlying application domain from (2) the way the model is presented to the user and from (3) the way the user interacts with it.

They go on to say that it was observed that most Smalltalk-76 applications offer the same interaction—images, buttons, text and menus—and that the goal of the MVC factoring in Smalltalk-80 is to handle those interactions in a common way across all applications.

Smalltalk MVC, adapted from Krasner and Pope 1988

The model and the view hold the same positions that they did in TVME: the models represent the thing in a computerey way, and the views represent the model on the bitmap display. “Controllers are used to send messages to the model, and provide the interface between the model with its associated views and the interactive user interface devices (e.g., keyboard, mouse). Each view may be thought of as being closely associated with a controller, each having exactly one model, but a model may have many view/controller pairs.” The controller is to MVC as the editor was to TVME.

So, the controller is the thing that senses and responds to UI events. Because you probably only intend for one thing to happen when you press a key or click a mouse button, only one controller is active at a time. That controller can interpret your command as a requirement to edit or query the model, or to display or change a particular view.

The views can also edit or query the model. So, there’s a need for the model to let the controller know when it was changed by a view, and the views should know when the model was changed by the controller. Enter dependents!

Dependents represent a realisation of what would later be known as the Observer pattern. The model can keep track of a collection of the objects that care about its changes. When it changes, it sends a changed or changed: message to its dependents.

Smalltalk-80 supplies bundled abstract classes for each of Model, View and Controller. The article goes into each of these classes and some of the subclasses that have already been created, showing how inspectors, editors and debuggers in Smalltalk-80 are all built from this factored collection of objects. It also details complete applications, including the FinancialHistory tutorial from “Smalltalk-80: the Language and its Implementation” by Goldberg and Robson.

Looking in the blue book, as it’s known, none of the terms MVC, model, view, nor controller appear in its index. The classes Model, View and Controller are not in the example class index, and the dependents, changed and changed: methods are not in its implementation index. The FinancialHistory object is described throughout the book, but is a subclass of Object and does not participate in any factoring that’s similar to MVC.

It’s probable that this reference is a mistake, and that Krasner and Pope intended to refer to the red book “Smalltalk-80: the Interactive Programming Environment” by Goldberg. This does indeed describe MVC, but only in relationship to the “MVC Inspector”. However, the description of the inspector shows that the pattern was common enough to motivate the construction of bespoke UI:

Many objects in the Smalltalk-80 system are closely related to one another. In particular, the user interface of the system is implemented using subclasses of two classes, class View and class Controller. A View represents ways of presenting information on the screen; a Controller represents ways in which the user interacts with a screen view. Any View is related to a Controller and to another object, the object whose information is accessed in the View. This other object is referred to as a “model.” Whenever you inspect a View, you typically want to inspect its related model or Controller. There is a special inspector in the system for inspecting the two main objects related to a View, as well as the View, whenever a View is sent the message inspect.

Having mentioned the red and blue books, it seems appropriate to introduce the green book, “Smalltalk-80: Bits of History, Words of Advice” edited by Glenn Krasner (the first author of the paper we started at). There is no MVC in the green book, but the three books[*] together are a fascinating snapshot of this time in our industry.

[*] There is a purple book too, but it’s just half of the blue book repackaged.

So Smalltalk-80 MVC is a formalisation of the abstract collaboration between objects documented as Thing-Model-View-Editor. It is an idiom named after the developer tool that was designed to take advantage of the common relationships between these objects. At some point after its initial documentation, authors (Krasner and Pope along with others) decided that this was not merely a factoring, but a “User Interface Paradigm”.

Set the settings set

The worst method naming convention Object-Oriented programming is set{Thing}(). And no, C# doesn’t escape my ire for calling it Set{Thing}(), nor does Smalltalk for calling it {thing}:, though that does handily demonstrate how meaningless set is.

OK, so set isn’t really meaningless. In fact, it’s got the opposite problem: entirely too many meanings. Luckily, when we use set in a programming context, we only mean exactly one of those definitions.

Oh wait, there’s that whole unordered collection thing, too. Two meanings.

And the electrical engineering definition of “making the voltage high”, so there’s three I suppose.

And the arrangement thing, like preferences (“settings”) or big scale configuration (“set up”). Four.

So, let’s all put on the blinkers of bean-fuelled tradition, and assume that when we see set at the beginning of a method we refer to putting something into a specified state. Probably without that digression you would have thought a method starting with the word set would do that anyway.

So far, so Kevlin Henney. “Set” really is a bad word to choose. But now let me ask two questions, which happen to be degenerate in verbal content: why am I setting this state?

Why am I setting this state?

Aren’t data hiding and encapsulation important principles in object-oriented programming? Why should you let me, an unrelated object, diddle with your state? Why not, you know, let me send you commands and queries?

Also, there’s the single responsibility principle. Why do I have information when you’re the one who needs it? Perhaps I should have asked you to work out the new thing rather than doing it myself and telling you my answer. Of course objects often do need to pass one another information: software systems wouldn’t be very interesting if data didn’t flow around them. This leads me on to the same question again:

Why am I setting this state?

As commands go, “set{Thing}()” really doesn’t document its purpose at all. “Here, have this thing”. Err, okay, but why do you want it? What will you use it for? Why should I care what your thing is? What use will I get out of giving you a different thing?

Here are some mutators with descriptive names:

label.setColor(red); //becomes
label.drawTextInThisColor(red);

aStreet name:'Sesame Street'. "becomes"
aStreet wasRenamedTo:'Sesame Street'.

[employee setSalary:@(75000)]; //becomes
[employee futurePayChecksShouldBeBasedOnNewSalary:@(75000)];

Notice that it’s now clearer why I’m setting these. It’s also evident that these changes tell us something about the evolution over time of the objects: the street had an earlier name, and now has this new name. The employee had a different salary for their previous pay check, and will have a new salary for their next pay check. Last month’s pay check should probably still be calculated with last month’s salary. But now the method also makes it clear that there’s a design problem anyway: why should the employee and not the payroll be where pay check information goes?

Soon after posting, the question was asked: isn’t drawTextInThisColor() a bad name because the object might not immediately need to do any drawing? No; sending an object a message and leaving the implementation up to that object is how object-oriented programming works. If we really had to distinguish drawTextLater() and drawTextNow() then we’d not be doing any information hiding. Compare this with things like faults (an ORM claims that the database is in some state, when in fact it’s not yet), or even the way GUI libraries already work (you can tell a view it needs display, and the view can choose not to do anything because it’s off-screen). Object-Oriented Programming involves telling an object what you want from it and leaving the details up to that object. The set…() convention doesn’t tell us what we want from an object.

Bonus question: Why am I setting this state?

Finally, I didn’t tell you what units the salary is in. Good catch. Give yourself a pat on the head, you’re a very special snowflake.

Meaningless Vapid Catchphrase

On the 4th December 2013, I said:

Urge to search the archives for papers on Model-View-Controller and write an essay on its ever-changing meaning in programmer discourse.

Do you have any idea how much work that is? I do, now. So I’m going to cover a very small part of the story here, with a view—your interest and my inclination permitting—to adding to it in subsequent posts.

Part One: you’ve lost your thing

The genesis of MVC comes in 1979 with a pair of memos written by Trygve Reenskaug of the Xerox Learning Research Group. In the first, A Note on Dynabook Requirements, Reenskaug documents an approach to tackling problems in the Smalltalk environment by describing his design for a project management task.

A fun digression at this point is the observation that Reenskaug appears to have described what we now call the user story: a goal-directed statement of what someone’s trying to achieve, as a replacement for a specification of what some theoretical software product should do.

Most problems would start with a rather unclear and often self-contradictory goal. Some examples: I would like to get better control over my finances; I don’t want to be troubled with detailed accounting; I want to settle my account with the butcher; I want to know more about Tarot cards; or I want a small, cheap house with many large, luxurious rooms.

I would expect the user to go more or less subconsciously through a goal-means hierarchy: Certain means are needed to reach a given goal. These means are not immediately available, but constitute a new set of part-goals, each of which needs certain means for their satisfaction, and so on.

An example: To get better control over my finances, I would need to set up a budget; to keep account of all income and expenditure; and to keep a running comparison between budget and accounts. Three new, non-trivial goals that need further consideration.

He goes on to note that given a sufficiently consistent collection of goals, the computer should probably just solve the problem itself. In the absence of evidence that this is possible (Prolog notwithstanding), we’ll need to tell the computer the methods by which the problem is solved.

Anyway, back to MVC. The solution in the context of his project management problem then follows. Along the way he introduces some metaphors that are formalised in a glossary in the second memo, called not Model-View-Controller but Thing-Model-View-Editor: an Example from a planningsystem.

An immediate observation is that where MVC has three parts, TMVE has four: the thing being modelled is explicitly part of the problem. MVC is a way to do computer stuff, TVME is a way to use a computer stuff to solve a problem based in the real world. Of course, you cannot put the thing itself inside the computer, except in the world of Tron which was yet to be released. So instead you find some useful abstraction and represent it in the computer as a model. This model contains both the data and the actions appropriate to the abstraction it represents.

The image below, an adaptation of a figure that appears in both documents, demonstrates that a nebulous thing can be modelled by multiple different abstractions, or that multiple models can represent different parts of the thing.

Models bound a thing for representation in a computer.

Now for any model, there will be one or more views that represent the model in a meaningful way; models do not know how to draw or print themselves. Views can also control the model in ways appropriate to the view’s representation. For example, Reenskaug shows that a view describing a model’s properties could accept changes to those properties, and make the required changes to the model.

Now there’s going to be a complicated object graph in place, with one or more model objects each represented by one or more views, all to solve a problem with one particular thing. The editor is a coordinator for all of these. It acts as a command interface, mediating between the user and this network of objects. As changes are made, the editor coordinates with all of the views.

You will recall that Trygve Reenskaug was on the Learning Research Group at Xerox. A Smalltalk system like the proposed Dynabook is supposed to be a computer that lets people solve their own problems, by being easy to program and to manipulate. Therefore, unlike modern MVC which is a design tool for professional programmers, Thing-Model-View-Editor is a user interface paradigm, describing how people can build their own programs to solve their own problems. The editor may have a doit command to send Smalltalk messages, but sending Smalltalk messages should not be beyond the wit of an interested and engaged user.

This goes some of the way to explaining why “computery stuff” like persistence, networking and the like are not tackled in TVME. They are outside its purview: the computer should be looking after that stuff. Just as the humane interface on a Canon Cat has no save or load buttons, and the computer looks after storage itself, so Dynabooks do not have databases, or filesystems, or memory allocators. These are abstractions of the thing “computer”, but Thing-Model-View-Editor is concerned with the thing that people want to represent in the computer. The computer is not the thing.

ClassBrowser: warts and all

I previously gave a sneak peak of ClassBrowser, a dynamic execution environment for Objective-C. It’s not anything like ready for general use (in fact it can’t really do ObjC very well at all), but it’s at the point where you can kick the tyres and contribute pull requests. Here’s what you need to know:

Have a lot of fun!

ClassBrowser is distributed under the terms of the University of Illinois/NCSA licence (because it is based partially on code distributed with clang, which is itself under that licence).

Standing at the Crossroads

A while back I wrote Conflicts in my Mental Model of Objective-C, in which I listed a few small scale dichotomies or cognitive dissonances that plagued my notion of my work. I just worked out what the overall picture is, the jigsaw into which all of these pieces can be assembled.

And I do mean just. It’s about 1AM on Christmas Eve, but this picture hit me so hard I couldn’t stop thinking about it without writing it down and getting it out of my head. If it doesn’t explain everything, it shows me the shape of the solution at least.

A tale of two Apples

I believe that everything I wrote in the Conflicts post can be understood in terms of two different and (of course) opposed models of Apple. I also believe that the two models are irreconcilable, but that the opposition is also accidental, not essential. That by removing the supposed conflict between them, everything I thought was a problem can be resolved.

It was the best of iPads

The iPad is, I would argue (and accept that this is a subjective argument) the most tasteful application of computing technology to the world of many computer users. I would further argue, and this is perhaps on firmer ground, that the reason the iPad is so tasteful is because Apple spend a lot more time and resources on worrying about questions of taste than many of their competitors and others in the industry.

There are many visions that have combined to produce the iPad, but interestingly the one that I think is clearest is John Scully’s Knowledge Navigator. In the concept videos for Knowledge Navigator, a tablet computer with natural language speech comprehension and a multi-touch screen is able to use the many hyperlinked documents available on the Web to answer a wide range of questions, make information available to its users and even help them to plan their schedules.

This is what we have now. This is the iPad, with Siri and a host of third-party applications. Apple even used to use the slogan “there’s an app for that”. Do you have a problem? You can probably solve it with iOS and a trip to the app store.

It was the worst of iPads

OK, what do you do if your problem isn’t solved on the app store, or the available solutions aren’t satisfactory?

Well first, you’d better get yourself another computer because while the iPad is generally designed for solving problems it isn’t designed for solving general problems. You might be able to find some code editors on the iPad, but you sure aren’t going to use them to write an iPad app without external assistance.

OK, so you’ve got your computer, and you’ve learned how to do the stuff that makes iPad apps. Now you just pay a recurring fee to be allowed to put that stuff onto your iPad. And what if you want to share that with your friends? Only if it meets Apple’s approval.

If the iPad is the Knowledge Navigator, it is not the Dynabook. A Dynabook is a computer that you can use to solve your problems on, but it’s also one on which you can create solutions to your own problems.

The promise of the Dynabook is that if you understand what your problem is, you can model that problem on the Dynabook. You model it with objects-either your own or ones supplied for you. You can change and create these objects until they model the problem you have, at which point you can use them to compute a solution.

The irony is that we have all the parts needed in a Dynabook, all in the iPad. Computer so simple even children can use it? Check. Objects? Check. Repositories of objects created by other people so we don’t have to rewrite our own basic objects all the time? We call that CocoaPods (or RubyGems, or whatever the poison in your area of the world). But we just can’t put all of these things together on that computer itself.

That would be distasteful. That might let people do things that make the iPad look bad. That might mean iPads providing experiences that haven’t been vetted by the mothership.

Does using an iPad ever make you wonder how iPads work? What they can do? What you can make them do? You can answer these questions, but not using an iPad. Your Knowledge Navigator does not know the route to that particular destination.

This is what is truly meant when it is said that the iPad is not upgradeable. Forget swapping out memory chips or radio transmitters. Those are just lumps of sand inside a box made of melted sand and refined rock. The iPad is not upgradeable because you are stuck with the default experience: the out-of-the-box facilities plus those that have been approved from on high. It might be good, but it might not be good enough.

Notice that this is not an “everyone must program” position. That would be a very bad experience. The position is rather “everyone must have the facility, should they be so inclined, to make their computer better for them than the manufacturers did”.

Conclusions?

I think that the Apple described above is not at the intersection of technology and the liberal arts. It is at the border, a self-appointed barrier of things that might flow between the two.

I believe that the two visions can be reconciled, and that a thing can be both the Knowledge Navigator and the Dynabook. I don’t believe you have to disable some experiences to provide others. I believe that Apple the champions of tasteful computing can be applauded at every turn while Apple the high priests of the church of computing can be fought tooth and nail.

Enablers? Yes, please. Arbiters? No, thanks.

A sneaky preview of ClassBrowser

Let me start with a few admissions. Firstly, I have been computering for a good long time now, and I still don’t really understand compilers. Secondly, work on my GNUstep Web side-project has tailed off for a while, because I decided I wanted to try something out to learn about the compiler before carrying on with that work. This post will mostly be about that something.

My final admission: if you saw my presentation on the ObjC runtime you will have seen an app called “ClassBrowser” where I showed that some of the Foundation classes are really in the CoreFoundation library. Well, there were two halves to the ClassBrowser window, and I only showed you the top half that looked like the Smalltalk class browser. I’m sorry.

So what’s the bottom half?

This is what the bottom half gives me. It lets me go from this:

ClassBrowser before

via this:

Who are you calling a doIt?

to this:

ClassBrowser after

What just happened?

You just saw some C source being compiled to LLVM bit code, which is compiled just-in-time to native code and executed, all inside that browser app.

Why?

Well why not? Less facetiously:

  • I’m a fan of Smalltalk. I want to build a thing that’s sort of a Smalltalk, except that rather than being the Smalltalk language on the Objective-C runtime (like F-Script or objective-smalltalk), it’ll be the Objective-C(++) language on the Objective-C runtime. So really, a different way of writing Objective-C.
  • I want to know more about how clang and LLVM work, and this is as good a way as any.
  • I think, when it actually gets off the ground, this will be a faster way of writing test-first Objective-C than anything Xcode can do. I like Xcode 5, I think it’s the better Xcode 4 that I always wanted, but there are gains to be had by going in a completely different direction. I just think that whoever strikes out in such a direction should not release their research project as a new version of Xcode :-).

Where can I get me a ClassBrowser?

You can’t, yet. There’s some necessary housekeeping that needs to be done before a first release, replacing some “research” hacks with good old-fashioned tested code and ensuring GNUstep-GUI compatibility. Once that’s done, a rough-and-nearly-ready first open source release will ensue.

Then there’s more work to be done before it’s anything like useful. Particularly while it’s possible to use it to run Objective-C, it’s far from pleasant. I’ve had some great advice from LLVM IRC on how to address that and will try to get it to happen soon.

By your _cmd

This post is a write-up of a talk I gave at Alt Tech Talks: London on the Objective-C runtime. Seriously though, you should’ve been there.

The Objective-C runtime?

That’s the name of the library of C functions that implement the nuts and bolts of Objective-C. Objects could just be represented as C structures, and methods could just be implemented as C functions. In fact they sort of are, but with some extra capabilities. These structures and functions are wrapped in this collection of runtime functions that allows Objective-C programs to create, inspect and modify classes, objects and methods on the fly.

It’s the Objective-C runtime library works out what methods get executed, too. The [object doSomething] syntax does not directly resolve a method and call it. Instead, a message is sent to the object (which gets called the receiver in this context). The runtime library gives objects the opportunity to look at the message and decide how to respond to it. Alan Kay repeatedly said that message-passing is the important part of Smalltalk (from which Objective-C derives), not objects:

I’m sorry that I long ago coined the term “objects” for this topic because it gets many people to focus on the lesser idea.

The big idea is “messaging” – that is what the kernal[sic] of Smalltalk/Squeak is all about (and it’s something that was never quite completed in our Xerox PARC phase). The Japanese have a small word – ma – for “that which is in between” – perhaps the nearest English equivalent is “interstitial”. The key in making great and growable systems is much more to design how its modules communicate rather than what their internal properties and behaviors should be.

Msg send

Indeed in one article describing the Smalltalk virtual machine the programming technique is called the message-passing or messaging paradigm. “Object-Oriented” is used to describe the memory management system.

Throughout this talk and post I’m talking about the ObjC runtime, but there are many. They all support an object’s introspective and message-receiving capabilities, but they have different features and work in different ways (for example, Apple’s runtime sends messages in one step, but the GNU runtime looks up messages and then invokes the discovered function in two steps). All of the discussion below relates to Apple’s most modern runtime library (the one that is delivered as part of OS X since 10.5 and iOS).

In the talk, I decided to examine a few specific areas of the runtime library’s behaviour. I looked for things that I wanted to understand better, and came up with questions I wanted to answer as part of the talk.

Dynamic class creation

Can I implement Key-Value Observing?

While I was preparing the talk, a post called KVO considered harmful started to get a lot of coverage. That post raises a lot of valid criticisms of the Key-Value Observing API, but rather than throw away the Observer pattern I wanted to explore a new implementation.

The observed (pardon the pun) behaviour of KVO is to privately subclass the observed object’s class, so that it can customise the object’s behaviour to call the KVO callback. That’s done through a function called objc_duplicateClass, unfortunately the documentation tells us that we should not call this function ourselves.

It’s still possible to implement an Observer pattern that uses the same secret-subclass behaviour, by allocating and registering a “class pair”. What’s a class pair? Well each class in Objective-C is really two classes: the class object defines the instance methods, and the “metaclass” defines the class methods. So each class is really a singleton instance of its metaclass.

The ObserverPattern implementation shows how this works. When you add an observer to an object, the receiver first works out whether it’s an instance of the observable class. If it needs to create that class, it does so: adding our own implementations of -dealloc to clean up after ourselves, and -class so that, like KVO observable objects, the generated class name doesn’t appear when you ask an observed object its type.

Having created the class, the code goes on to add a setter for the conventional Key-Value Coding selector name for the property: this setter grabs the old and new values of the property and invokes the callback which was supplied as a block object. Because we can, the block is dispatched asynchronously.

Notice that the -addObserverForKey:withBlock: method uses object_setClass() to replace the receiver’s class with the newly-constructed class. The main effect of this is to change the way messages are resolved onto methods, but you need to be careful that the original and replaced class have the same instance variable layout too. Instance variables are looked up via the runtime too, and changing the class could alter where the runtime thinks the bytes are for any given variable.

We have a little extra hurdle to overcome in storing the collection of observer tokens, because there’s nowhere to put them. Adding an instance variable to the ObserverPattern[…] class would not work, as instances of that class are never actually allocated. The objects involved have the instance variables of their initial class, which won’t include space for the observers.

The Objective-C runtime provides for this situation by giving us associated objects. Any object can have what is, conceptually, a dictionary of other objects maintained by the runtime. Observed objects can store and retrieve their observer tokens via associated references, and no extra instance variables are needed.

A little problem in the ObserverPattern implementation will become clear if you run it enough times. The observation callbacks are sent asynchronously, and can be delivered out of sequence. That means the observer can’t actually tell what the final state of the observed key is, because the “new value” received in the callback might have already been replaced. I left this fun issue in to demonstrate that KVO’s synchronous implementation is a feature, not a bug.

Creating objects

What are those extra bytes for?

When you create an Objective-C object, the runtime lets you allocate some extra storage at the end of the space reserved for its instance variables. What’s the point of that? All you can do is get a pointer to the start of the space (using object_getIndexedIvars)…hmm, indexed ivars. Well, I suppose an array is a pretty obvious use of indexed ivars…

Let’s build NSArray! There are two things to see in SimpleArray: the most obvious is the use of the class cluster pattern. The reason is that the object returned from +alloc—where we’d normally allocate space for the object—cannot know how big it’s going to be. We need to use the arguments to -initWithObjects:count: to know how many objects there are in the array. So +alloc returns a placeholder, which is then able to allocate and return the real array object.

One obvious question to ask is why we’d do this at all. Why not just use calloc() to grab an appropriately-sized buffer in which to store the object pointers? The answer is to do with a low-level performance concern called locality of reference. We know from the design of the array class that pretty much every time the array pointer is used, the buffer pointer will be used too. Putting them next to each other in RAM means we don’t have to look off at some dereferenced pointer just to find another pointer.

Message dispatch

Just how does message forwarding work?

One of the powerful features of Objective-C is that an object doesn’t have to implement a method when it’s compiled to be able to respond to messages with that selector name. It can lazily resolve the methods, or it can forward them to another object, or it can raise an error, or it can do something else. But something about this feature was bugging me: message forwarding (which happens in the runtime) calls -forwardInvocation:, passing it an NSInvocation object. But NSInvocation is defined in Foundation: does the runtime library need to “know” about Foundation to work?

I tracked down what was going on and found that no, it does not need to know about Foundation. The runtime lets applications define the forwarding function, that gets called when objc_msgSend() can’t find the implementation for a selector. On startup, CoreFoundation[+] injects the forwarding function that does -forwardInvocation:. So presumably my application can do its own thing, right?

Let’s build Ruby! OK, not all of Ruby. But Ruby has a #method_missing function that gets called when an object receives a message it doesn’t understand, which is much more similar to Smalltalk’s approach than to Objective-C’s. Using objc_setForwardHandler, it’s possible to implement methodMissing: in our Objective-C classes.

Conclusion

The Objective-C runtime is a powerful way to add a lot of dynamic behaviour to an application for very little work. Some developers don’t use it much beyond swizzling methods for debugging, but it has facilities that make it a powerful tool for real application code too.


[+]CoreFoundation and Foundation are really siblings, and they each expose pieces of the other’s implementation, but one has a C interface and the other an Objective-C interface. Various Objective-C classes are actually part of CoreFoundation, including NSInvocation and the related NSMethodSignature class. NSObject is not in either of these libraries: it’s now defined in the runtime itself, so that the runtime’s memory management functions know about -retain, -release and so on[++]. On the other hand, most of the *behaviour* of NSObject is implemented by categories higher up. And, of course, this is all implementation detail and the locations of these classes could be (and are) moved between versions of the frameworks.

[++]Other languages like Smalltalk and Ruby have a simple base class that does nothing except know how to be an object, called ProtoObject or BaseObject. You could imagine the runtime supplying—and being coupled to—ProtoObject, and (Core)Foundation supplying NSObject and NSProxy as subclasses of ProtoObject.