Or maybe, because we want to

How (and Why) Developers Use the Dynamic Features of Programming Languages: The Case of Smalltalk is an interesting analysis of the reality of dynamic programming in Smalltalk (Squeak and Pharo, really). Taking the 1,000 largest projects on SqueakSource, the authors quantitatively examine the use of dynamic features in projects and qualitatively consider why they were adopted.

The quantitative analysis is interesting: unsurprisingly a small number (under 1.8%) of methods use dynamic features, but they are spread across a large number of projects. Applications make up a large majority of the projects analysed, but only a small majority of the uses of dynamic features. The kinds of dynamic features most commonly used are those that are also supplied in “static” languages like Java (although one of the most common is live compilation).

The qualitative analysis comes from a position of extreme bias: the poor people who use dynamic features of Smalltalk are forced to do so through lack of alternatives, and pity the even poorer toolsmiths and implementors whose static analysis, optimisation and refactoring tools are broken by dynamic program behaviour! Maybe we should forgot that the HotSpot optimisation tools in Java come from the Smalltalk-ish Self environment, or that the very idea of a “refactoring browser” was first explored in Smalltalk.

This quote exemplifies the authors’ distaste for dynamic coding:

Even if Smalltalk is a language where these features are comparitively easier to access than most programming languages, developers should only use them when they have no viable alternatives, as they significantly obfuscate the control flow of the program, and add implicit dependencies between program entities that are hard to track.

One of the features of using Object-Oriented design is that you don’t have to consider the control flow of a program holistically; you have objects that do particular things, and interesting emergent behaviour coming from the network of collaboration and messages passed between the objects. Putting “comprehensible control flow” at the top of the priority list is the concern of the structured programmer, and in that situation it is indeed convenient to avoid dynamic rewriting of the program flow.

I have indeed used dynamic features in software I’ve written, and rather than bewailing the obfuscation of the control flow I’ve welcomed the simplicity of the solution. Looking at a project I currently have open, I have a table data source that uses the column identifier to find or set a property on the model object at a particular row. I have a menu validation method that builds a validation selector from the menu item’s action selector. No, a static analysis tool can’t work out easily where the program counter is going, but I can, and I’m more likely to need to know.

Inheritance still doesn’t make any sense

Some ideas based on feedback to the Why inheritance never made any sense:

Feedback: Subtypes are necessary

The only one of these that is practically workable is behaviour inheritance <=> subtype inheritance: I’m sorry that you were exposed to Java at such an impressionable age. The compilers of languages like Java enable subclass = subtype, by automatically assuming that a subclass is is a valid value for variable binding, for example. However they do nothing to ensure subclass = subtype. This is valid C#, a language very like Java for this discussion:

namespace QuickTestThing
{
    class Class1
    {
        override public string ToString()
        {
            return "Class1";
        }
    }
    class Class2 : Class1
    {
        public override string ToString()
        {
            throw new Exception();
        }
    }
}

Now is Class2 a subtype of Class1? Does the compiler let you pretend that it is?

You don’t even need inheritance

As discussed in the original post, the whole “favour composition over inheritance” movement gets by fine with no inheritance. Composition and delegation (I don’t know about this message, I’ll forward it to someone who does) let you get the same behaviour.

Feedback: build it yourself

Can I demonstrate a language that has all three of subtype inheritance, behaviour inheritance, and categorical inheritance as distinct language features? Yes, but I would need to learn Racket first. I’m on it.

But in the meantime, re-read the “you don’t even need inheritance” paragraph and think about how you would build each of those three ideas out of delegation.

Why inheritance never made any sense

There are three different types of inheritance going on.

  1. Ontological inheritance is about specialisation: this thing is a specific variety of that thing (a football is a sphere and it has this radius)
  2. Abstract data type inheritance is about substitution: this thing behaves in all the ways that thing does and has this behaviour (this is the Liskov substitution principle)
  3. Implementation inheritance is about code sharing: this thing takes some of the properties of that thing and overrides or augments them in this way. The inheritance in my post On Inheritance is this type and only this type of inheritance.

These are three different, and frequently irreconcilable, relationships. Requiring any, or even all, of them, presents no difficulty. However, requiring one mechanism support any two or more of them is asking for trouble.

A common counterexample to OO inheritance is the relationship between a square and a rectangle. Geometrically, a square is a specialisation of a rectangle: every square is a rectangle, not every rectangle is a square. For all s in Squares, s is a Rectangle and width of s is equal to height of s. As a type, this relationship is reversed: you can use a rectangle everywhere you can use a square (by having a rectangle with the same width and height), but you cannot use a square everywhere you can use a rectangle (for example, you can’t give it a different width and height).

Notice that this is incompatibility between the inheritance directions of the geometric properties and the abstract data type properties of squares and rectangles; two dimensions which are completely unrelated to each other and indeed to any form of software implementation. We have so far said nothing about implementation inheritance, so haven’t even considered writing software.

Smalltalk and many later languages use single inheritance for implementation inheritance, because multiple inheritance is incompatible with the goal of implementation inheritance due to the diamond problem (traits provide a reliable way for the incompatibility to manifest, and leave resolution as an exercise to the reader). On the other hand, single inheritance is incompatible with ontological inheritance, as a square is both a rectangle and an equilateral polygon.

The Smalltalk blue book describes inheritance solely in terms of implementation inheritance:

A subclass specifies that its instances will be the same as instances of another class, called its superclass, except for the differences that are explicitly stated.

Notice what is missing: no mention that a subclass instance must be able to replace a superclass instance everywhere in a program; no mention that a subclass instance must satisfy all conceptual tests for an instance of its superclass.

Inheritance was never a problem: trying to use the same tree for three different concepts was the problem.

“Favour composition over inheritance” is basically giving up on implementation inheritance. We can’t work out how to make it work, so we’ll avoid it: get implementation sharing by delegation instead of by subclassing.

Eiffel, and particular disciplined approaches to using languages like Java, tighten up the “inheritance is subtyping” relationship by relaxing the “inheritance is re-use” relationship (if the same method appears twice in unrelated parts of the tree, you have to live with it, in order to retain the property that every subclass is a subtype of its parent). This is fine, as long as you don’t try to also model the problem domain using the inheritance tree, but much of the OO literature recommends that you do by talking about domain-driven design.

Traits approaches tighten up the “inheritance is specialisation” relationship by relaxing the “inheritance is re-use” relationship (if two super categories both provide the same property of an instance of a category, neither is provided and you have to write it yourself). This is fine, as long as you don’t try to also treat subclasses as covariant subtypes of their superclasses, but much of the OO literature recommends that you do by talking about Liskov Substitution Principle and how a type in a method signature means that type or any subclass.

What the literature should do, I believe, is say “here are the three types of inheritance, focus on any one of them at a time”. I also believe that the languages should support that (obviously Smalltalk, Ruby and friends do support that by not having any type constraints).

  • If I’m using inheritance as a code sharing tool, it should not be assumed that my subclasses are also subtypes.
  • If I am using subtypes to tighten up interface contracts, I should be not only allowed to mark a class anywhere in the tree as a subtype of another class anywhere in the tree, but required to do so: once again, it should not be assumed that my subclasses are also subtypes.
  • If I need to indicate conceptual specialisation via classes, this should also not be assumed to follow the inheritance tree. I should be not only allowed to mark a class anywhere in the tree as a subset of another class, but required to do so: once again, it should not be assumed that my subclasses are also specialisations.

Your domain model is not your object model. Your domain model is not your abstract data type model. Your object model is not your abstract data type model.

Now inheritance is easy again.

In defense of `id`

Something you can’t see about my dotSwift talk on OOP in FP in Swift is that to make the conference more interesting while the AV was set up for the next speaker, Daniel Steinberg invited me over to a side table for a question and answer session. He had some great questions and I had some adequate answers; that is now lost to time.

One thing he asked was about how I do things differently when I’m working in Objective-C and in Swift, and I mentioned that I tend not to use the type system in ObjC and just call all of my variables id unless they are C types or the compiler asks otherwise. You can see an example of this in my UIKonf 1995 talk.

I argue (back in 2018, at dotSwift, in the bit that was videoed) that all objects have the same type. Just as I can define the “type” Set through its function signature:

typealias Set<T> = (T) -> Bool

so I can define the “type” object through its function signature. An object – any object – is a function that responds to messages by turning selectors into methods:

typealias Object = (Selector) -> IMP

Now if all objects have the same type, why would I want to use different types for different objects?

Of course, there are reasons to want to refine the definition of an object from “any object” to “an object like this”, but these refinements are inaccessible using Objective-C’s type system (or Java’s, or Swift’s, or most other programming languages). Any object responds to any message, but for the most part they respond by doing whatever the default error-raising behaviour is which is not particularly interesting and doesn’t solve your customer’s problem. So what we want to be able to say is “this object is one that responds to a particular collection of messages in a way that is what I need here”.

We have two tools available to us, and neither gives us an answer to that question. The first is the protocol (or Java interface): we can say “this object is one that has implementations of methods for a particular collection of messages”. That’s not the same as the question we want to answer – it says nothing about whether the object responds in the ways we want. It’s also not generally the correct answer – an object that has the methods we want and behaves in the expected way but that didn’t have the protocol conformance recorded at compile time (even if it conformsToProtocol: at run time) does not satisfy the compiler type check for protocol conformance.

Even less generally useful is using the class name as the type. Classes are ways to say “here is a collection of objects that all have common behaviour and data”; well for starters I don’t care about the data, but also just because those objects have the properties I want, doesn’t mean that others outside that place in the inheritance tree don’t.

I also can’t rely on subtypes behaving as I need, but the compiler type checker will pretend that if I asked for an instance of one class, and got an instance of a subclass, then I got the thing I wanted. Inheritance, in many languages including Objective-C, Java and similar, is a way to borrow behaviour from one class in another class. If we also want refinement from inheritance we have to add a bunch of rules that almost every programming language not named after the designer of the Garabit Viaduct does not support.

So there are three imprecise ways to ask for objects by behaviour in Objective-C, and I choose the one with the least typing. Even once you amortise the cost of this blog post.

On Inheritance

I recently had the chance to give my OOP-in-FP-in-Swift talk again in NSLondon, and was asked how to build inheritance in that object system. It’s a great question, I gave what I hope was a good answer, and it’s worth some more thought and a more coherent response.

Firstly, let’s look at the type signature for an object in this system:

typealias Object = (Selector) -> IMP

A Selector is the name of a method, and an IMP is a function1 implementing that method. But an Object is nothing more or less than a function that maps names of methods to implementations of methods. And that’s incredibly powerful, for two reasons.

Reason one: Inheritance is whatever you want it to be.

You are responsible for writing the code to look up methods, which means you get to choose how it works. If you don’t like inheritance at all, then you’re golden: each object knows its own methods and nothing else.

If you like Javascript or Self or Io, then if your object doesn’t have a method then it can send itself a proto message, and ask that object what method to use.

If you like Smalltalk or ObjC or Ruby, then you can create an object called a Class that creates objects that look up methods by asking the class what methods to use. If the class doesn’t know, then it can ask its superclass.

If you like multiple inheritance, then give an object a list of classes/prototypes instead of a single one.

If you’ve got some other idea, build it! Maybe you always thought classification should be based on higher-order logic; well here’s your chance.

(By the way, if you want to do this, you would be well-off defining a convention where methods all take a parameter that can be bound to the receiver: call it this or self for example. Then when you’re deep in inheritance-land, but you want to send a message to self, you still have a reference to it.)

Reason two: Inheritance is whatever you want it to be at each point.

A failing common to all of the object systems named above (as a reminder, that’s Javascript, Self, Io, Smalltalk, ObjC, Ruby) is that they force you to work with a single object paradigm. If these things naturally follow a singly-inherited classification scheme, but those things can be better described as deviations from a common prototype, well, sorry, but you’ve got to pick one and contort it to fit both situations.

If an object is any arbitrary code that finds a method, then you can build whichever model is most appropriate at the point of use. You can mix and match. The core philosophy at a code level is that objects are just loosely-coupled functions. From a conceptual level that’s incredibly powerful: such loose coupling means that you aren’t forced to make assumptions about how objects are constructed, or glued together. You just use them.


  1. actually a closure, which is even more useful, but that’s not important right now. 

machoo – Object-Oriented Programming in Object-Oriented Programming in the GNU HURD

For the last few weeks, my when-I-get-to-it project has been machoo, which is sort of an object-oriented system on the HURD but mostly an excuse to learn about how Mach messaging works.

I decided to build a Smalltalk-style “give me any old selector, and I’ll do something with it” messaging system on Mach, which is already a messaging system. I don’t yet know whether this is a good idea, but I like the design principle of highly loosely-coupled objects telling each other the names of the things to do.

The substrate for this system is MIG, the Mach Interface Generator. It takes descriptions of routines in an IDL that include the names of the supported routines and the argument names, types and return types. It’d certainly be possible to build each message I want to be able to send as a MIG routine, but then the client would have to import interfaces for every type it wanted to use: no bad thing, and many OO systems work that way, but not what I want.

MIG generates a ‘demuxer’ function that examines a message sent to a port, and dispatches it to a handler to the named routine. As an aside, it is highly likely that I’ve got one to many layers of indirection; that if I wrote my own function to take the place of the demuxer I could handle dispatch there without receiving arguments from a dispatcher to do more dispatch. OK, I could, but I don’t currently know how.

So what I ended up with is a two-level system, something like Smalltalk or ObjC. A “class” is a HURD translator registered on the filesystem, you send the zero-arguments message machoo_create_object to the class and get a send right to the new object. This is two-level in that if you need to initialise the object, you would then send an init message to the created object. Objects are messaged using the machoo_msg_send routine, which takes a string selector like Smalltalk.

There’s currently exactly one implementation of a class, a do-nothing null object class. It does the thing that I would extract out into a pattern for other classes: its response to a machoo_create_object message is to spawn a new thread to back the new object. So it’s like an Erlang-style system where each object has its own execution context, except that task switching is handled by the kernel rather than some virtual machine. All of the objects of a class are in a single task, and they are each on their own thread.

I still need to solve some problems, like how to return a reference to ‘this object’ from a message. I need to build more than one class of object, and extract out the common parts of class-ness so that a new class is only defined by its response to messages. So far this is proving to be an interesting and educational project.

An update on the HURD project

Last time, on Structure and Interpretation of Computer Programmers, I was building an object-oriented programming system on top of the HURD, and had realised that I needed to use its trivfs library for a sender to be able to discover an object to send messages to.

I got it working very quickly, but ended up shaving a yak due to my poor understanding of the HURD translator lifecycle which meant that I didn’t think I had got it working.

My goal was to build a translator that works like the Objective-C nil object: it accepts any message and responds by returning itself. Given that I’m building on the Mach ports abstraction, “itself” is defined as a port over which you can send messages to the same object.

If I returned an integer from the message, everything worked on both sides. However, just because a port name is an integer doesn’t mean that sending it as an int will work, just as opening a file in one process then sending the file descriptor as a number to another process wouldn’t let the receiving process access the file. I tried sending it as a mach_port_t, but got a lifecycle error: the client was told that the server had died.

On doing some reading, I discovered that the port had to be sent as a mach_port_send_t for send rights to be transferred to the client. Making that change, the message now fails with a type error.

An aside, here, on getting help with this problem. There is good documentation: the HURD source is easy to read and with helpful comments, they have good documentation including examples, the OSF documentation is very helpful, there are books from “back in the day” and videos with useful insights.

On the other hand, “help” is hard to come by. I eventually answered my own stack overflow question on the topic, having not received a reply on there, the HURD mailing list or their IRC channel. The videos described above come from FOSDEM and I’m heading out there next week, I’ll try to make some contacts in person and join their community that way.

OK, so back to the main issue, I now have a fix for my problem. Well, sort of, because now that I’m correctly sending a port with a send right I’m back to getting the lifecycle error.

My current plan is not to “fix” that, but to take it as a hint that I’m doing it wrong, and to design my system differently. Using the filesystem as a namespace to look up objects is good, but using the thing I receive as the object to message is a separate responsibility. I’m changing my approach so that the filesystem contains constructors, and they return not their own port but a port to something else that represents the instance they created.

Gently HURDing the side projects

I find it problematic that even at times when I’m avoiding computing outside of work, I still have ideas about things I would like to try out or improve in computing “if I had the time”. I tend to capture these somehow – usually written notes in paper or Evernote, and my personal technology radar.

Why might this be a problem? Isn’t having ideas good, and fun? Well it is, but with each comes guilt that I could be making progress on it but am not. Even when that’s my choice, when I deliberately put more effort into relationships with friends or musical projects or whatever, there’s still that nagging feeling that I’m leaving behind chances to make positive changes to computing.

My approach to addressing that started by building the radar. Now I don’t have a lot of different projects I could be working on but am not; I have a single related web of issues, and progress on any one thing counts as progress toward the whole.

The second change is to note that any progress is progress; sometimes I spend some time reading and make a sentence or two of notes on dealing with a problem. Sometimes I try a solution, find difficulties with it and write down that I discount that solution. If I’ve made some move forward from where I was before I’ve started, I can be satisfied and don’t need to burn the midnight oil to get a complete solution to a complex problem done before putting it down.

All of that goes toward describing the limited progress I’ve made on my current research topic, which is distributed message-passing. I like the idea from Erlang that objects run in separate contexts, completely decoupled except for passing messages between one another. This seems to be the best implementation of an object-oriented runtime environment, except that it is all done on the Erlang VM and in the couple of relatively esoteric languages that target it.

On the other hand, while Objective-C doesn’t make it easy to do that decoupling, it does have a very simple message-passing interface that can be implemented in any language with a C FFI. If you can wrap objc_msgSend or objc_msgLookup and expose it to your language runtime, you too can pass messages.

Why can’t we have both of these things? Why can’t we have the simple-to-integrate message interface that can work anywhere, along with the distributed and decoupled objects?

My theory is that Mach makes this possible so I’ve been investigating it using GNU Mach and the GNU HURD. Much of the documentation of Mach messaging uses name servers that register named ports for clients to find; this is how macOS, NeXTSTEP, OSF/1 and related systems work. HURD does not use a name server, it uses the filesystem: you attach a server to a file system node as a translator and clients find the ports by looking up their paths on the filesystem.

I found examples of filesystem translators in the HURD documentation, but they typically were examples that implemented the filesystem messages: seek, read, write, and so on. One could build message-sending on top of filesystem operations but it would not be pleasant:

  • marshall the message selector and arguments into some stream format
  • write() your message to the node
  • verify that you wrote as many bytes as you expect
  • read() the length of the reply
  • verify that you read as many bytes as you expect
  • read() the reply
  • verify that you read as many bytes as you expect
  • unmarshall the reply into an object of the correct type

Let’s be clear, all of this needs to be done, but it’s all already being done at the Mach message layer and hidden behind the MIG abstraction, so why should our clients and servers do it again in another abstraction built on top? I wanted to find a way to register a port that accepts non-filesystem messages using the HURD’s filesystem-as-name-server approach.

This morning I decided to look at how login was implemented on the HURD and discovered that the password server does exactly what I need. It is configured as a translator on the filesystem, and uses the trivfs library to check in with the bootstrap server and get its ports, but then it handles its own messages for checking passwords rather than the standard filesystem messages. Discovering that gives me enough new information to feel I’ve made progress, and a clear next step (pardon the pun).

An unhelpful distinction

Object-Oriented Programming is quite simple: it’s just choosing what function to run based on the parameters to the function (whether through method sending like Smalltalk, polymorphic lookup like CLOS, or table searching like C++: usually pattern-matching like Haskell would be excluded here).

Object-Oriented Analysis and Design is the thing where we represent our problem domain, and our solution, as a collection of objects, often categorised into classes, where the classes have particular relationships, properties and behaviours. And that is the thing that programmers often struggle with.

Whether it’s hard because OOA/D is hard, or because the problem domains are hard, or because OOA/D is not applicable to the problem domains, is not addressed here.