Skip to content

machoo – Object-Oriented Programming in Object-Oriented Programming in the GNU HURD

For the last few weeks, my when-I-get-to-it project has been machoo, which is sort of an object-oriented system on the HURD but mostly an excuse to learn about how Mach messaging works.

I decided to build a Smalltalk-style “give me any old selector, and I’ll do something with it” messaging system on Mach, which is already a messaging system. I don’t yet know whether this is a good idea, but I like the design principle of highly loosely-coupled objects telling each other the names of the things to do.

The substrate for this system is MIG, the Mach Interface Generator. It takes descriptions of routines in an IDL that include the names of the supported routines and the argument names, types and return types. It’d certainly be possible to build each message I want to be able to send as a MIG routine, but then the client would have to import interfaces for every type it wanted to use: no bad thing, and many OO systems work that way, but not what I want.

MIG generates a ‘demuxer’ function that examines a message sent to a port, and dispatches it to a handler to the named routine. As an aside, it is highly likely that I’ve got one to many layers of indirection; that if I wrote my own function to take the place of the demuxer I could handle dispatch there without receiving arguments from a dispatcher to do more dispatch. OK, I could, but I don’t currently know how.

So what I ended up with is a two-level system, something like Smalltalk or ObjC. A “class” is a HURD translator registered on the filesystem, you send the zero-arguments message machoo_create_object to the class and get a send right to the new object. This is two-level in that if you need to initialise the object, you would then send an init message to the created object. Objects are messaged using the machoo_msg_send routine, which takes a string selector like Smalltalk.

There’s currently exactly one implementation of a class, a do-nothing null object class. It does the thing that I would extract out into a pattern for other classes: its response to a machoo_create_object message is to spawn a new thread to back the new object. So it’s like an Erlang-style system where each object has its own execution context, except that task switching is handled by the kernel rather than some virtual machine. All of the objects of a class are in a single task, and they are each on their own thread.

I still need to solve some problems, like how to return a reference to ‘this object’ from a message. I need to build more than one class of object, and extract out the common parts of class-ness so that a new class is only defined by its response to messages. So far this is proving to be an interesting and educational project.

Discovery

I think the last technical conference I attended was FOSDEM last year, and now I’m sat in the lobby of the Royal Library of Brussels working on a project that I want to take to some folks at this year’s FOSDEM, and checking the mailing lists of some projects I’m interested in to find meetups and relevant talks.

It’s been even longer since I spoke at a conference, I believe it would have been App Builders in 2016. When I left Facebook in 2015 with the view of taking a year out of software, I also ended up taking myself out of community membership and interaction. I felt like I didn’t know who I was, and wasn’t about to find out by continually defining myself in terms of other people.

I would have spoken at dotSwift in 2016, but personal life issues probably also related to not knowing who I was stopped me from doing that. In April, I went to Zurich and gave what would be my last talk as a programmer to anyone except immediate colleagues for over 18 months.

In the meantime, I explored bits of the software world outside of my immediate experience. I worked on a Qt app in high-performance computing, and I’m working on “full-stack” Javascript now.

[Full-stack Javascript is, of course, a lie, but I’d rather it keeps its current meaning as “the application bit of the stack in Javascript” than took a more correct sense.]

During this time, I found that I don’t mind so much what I’m working on: I have positive and negative opinions of all of it. I have lots of strong opinions, and lots of syntheses of other ideas, and need to be a member of a community that can share and critique these opinions, and extend and develop these syntheses.

Which is why, after a bit of social wilderness, I’m reflecting on this week’s first conference and planning my approach to the second.

An update on the HURD project

Last time, on Structure and Interpretation of Computer Programmers, I was building an object-oriented programming system on top of the HURD, and had realised that I needed to use its trivfs library for a sender to be able to discover an object to send messages to.

I got it working very quickly, but ended up shaving a yak due to my poor understanding of the HURD translator lifecycle which meant that I didn’t think I had got it working.

My goal was to build a translator that works like the Objective-C nil object: it accepts any message and responds by returning itself. Given that I’m building on the Mach ports abstraction, “itself” is defined as a port over which you can send messages to the same object.

If I returned an integer from the message, everything worked on both sides. However, just because a port name is an integer doesn’t mean that sending it as an int will work, just as opening a file in one process then sending the file descriptor as a number to another process wouldn’t let the receiving process access the file. I tried sending it as a mach_port_t, but got a lifecycle error: the client was told that the server had died.

On doing some reading, I discovered that the port had to be sent as a mach_port_send_t for send rights to be transferred to the client. Making that change, the message now fails with a type error.

An aside, here, on getting help with this problem. There is good documentation: the HURD source is easy to read and with helpful comments, they have good documentation including examples, the OSF documentation is very helpful, there are books from “back in the day” and videos with useful insights.

On the other hand, “help” is hard to come by. I eventually answered my own stack overflow question on the topic, having not received a reply on there, the HURD mailing list or their IRC channel. The videos described above come from FOSDEM and I’m heading out there next week, I’ll try to make some contacts in person and join their community that way.

OK, so back to the main issue, I now have a fix for my problem. Well, sort of, because now that I’m correctly sending a port with a send right I’m back to getting the lifecycle error.

My current plan is not to “fix” that, but to take it as a hint that I’m doing it wrong, and to design my system differently. Using the filesystem as a namespace to look up objects is good, but using the thing I receive as the object to message is a separate responsibility. I’m changing my approach so that the filesystem contains constructors, and they return not their own port but a port to something else that represents the instance they created.

Gently HURDing the side projects

I find it problematic that even at times when I’m avoiding computing outside of work, I still have ideas about things I would like to try out or improve in computing “if I had the time”. I tend to capture these somehow – usually written notes in paper or Evernote, and my personal technology radar.

Why might this be a problem? Isn’t having ideas good, and fun? Well it is, but with each comes guilt that I could be making progress on it but am not. Even when that’s my choice, when I deliberately put more effort into relationships with friends or musical projects or whatever, there’s still that nagging feeling that I’m leaving behind chances to make positive changes to computing.

My approach to addressing that started by building the radar. Now I don’t have a lot of different projects I could be working on but am not; I have a single related web of issues, and progress on any one thing counts as progress toward the whole.

The second change is to note that any progress is progress; sometimes I spend some time reading and make a sentence or two of notes on dealing with a problem. Sometimes I try a solution, find difficulties with it and write down that I discount that solution. If I’ve made some move forward from where I was before I’ve started, I can be satisfied and don’t need to burn the midnight oil to get a complete solution to a complex problem done before putting it down.

All of that goes toward describing the limited progress I’ve made on my current research topic, which is distributed message-passing. I like the idea from Erlang that objects run in separate contexts, completely decoupled except for passing messages between one another. This seems to be the best implementation of an object-oriented runtime environment, except that it is all done on the Erlang VM and in the couple of relatively esoteric languages that target it.

On the other hand, while Objective-C doesn’t make it easy to do that decoupling, it does have a very simple message-passing interface that can be implemented in any language with a C FFI. If you can wrap objc_msgSend or objc_msgLookup and expose it to your language runtime, you too can pass messages.

Why can’t we have both of these things? Why can’t we have the simple-to-integrate message interface that can work anywhere, along with the distributed and decoupled objects?

My theory is that Mach makes this possible so I’ve been investigating it using GNU Mach and the GNU HURD. Much of the documentation of Mach messaging uses name servers that register named ports for clients to find; this is how macOS, NeXTSTEP, OSF/1 and related systems work. HURD does not use a name server, it uses the filesystem: you attach a server to a file system node as a translator and clients find the ports by looking up their paths on the filesystem.

I found examples of filesystem translators in the HURD documentation, but they typically were examples that implemented the filesystem messages: seek, read, write, and so on. One could build message-sending on top of filesystem operations but it would not be pleasant:

  • marshall the message selector and arguments into some stream format
  • write() your message to the node
  • verify that you wrote as many bytes as you expect
  • read() the length of the reply
  • verify that you read as many bytes as you expect
  • read() the reply
  • verify that you read as many bytes as you expect
  • unmarshall the reply into an object of the correct type

Let’s be clear, all of this needs to be done, but it’s all already being done at the Mach message layer and hidden behind the MIG abstraction, so why should our clients and servers do it again in another abstraction built on top? I wanted to find a way to register a port that accepts non-filesystem messages using the HURD’s filesystem-as-name-server approach.

This morning I decided to look at how login was implemented on the HURD and discovered that the password server does exactly what I need. It is configured as a translator on the filesystem, and uses the trivfs library to check in with the bootstrap server and get its ports, but then it handles its own messages for checking passwords rather than the standard filesystem messages. Discovering that gives me enough new information to feel I’ve made progress, and a clear next step (pardon the pun).

Head of Architecture

My current job title is Head of Architecture, though the word “architecture” means different things to different people in the world of software. So what does it mean to me, what do I do when I’m playing Head of Architecture?

I follow Perry and Wolf in Foundations for the Study of Software Architecture by drawing the analogy between software architecture and built environment architecture, not network or electronics architecture. The work of a built environment architect, particularly one who follows the path laid out by Christopher Alexander, combines elements, form and aesthetics by creating a system that complements and enhances its environment. Software systems are deployed into environments (with existing people, processes, cultural norms) and developed in environments, and should make those environments better for the people who are interacting with them, while also meeting the functionality, performance, security and other goals of the system.

But that doesn’t actually explain what I do, which is more about letting other people do things that are “architecture” than about “doing architecture” for them. Programmers, ops folks, QA people, product owners, and others frequently make decisions that have wide (and hence “architectural”) impact, and I think it’s better to enable that and follow up by asking how it impacts the rest of the system, to refine the choices made, than to stop people from making those decisions for them in the name of “being the architect”.

So playing software architect for me tends to be more about creating a forum in which people can present aspects of the problems they’re trying to solve, needing to solve soon, or the solutions they’re exploring, and getting a view from multiple teams and multiple functions about those problems and solutions. Making sure that ops know what devs are doing, that product team Alpha knows how product team Aleph are solving that issue, and so on.

How about the stereotype that software architects program using Visio or Powerpoint? In my case, I program using JavaScript. I do make documentation, to make sure that decisions made in the forum are captured, that proposed or current approaches can be seen and understood. And yes, much of that documentation is diagrammatic. But ultimately I’m a programmer on a software team too, and that documentation has to reflect working, valuable software. That is, while there is value in comprehensive documentation, we value working software more.

MUI

In my last post I talked about investigating AROS, the modern, open source[*] implementation of the Amiga Operating System. Today I’ve spent some more time on that study, and found some things:

  • my strategy last time was to read the AROS application developer guide in order, up to the part about Intuition (the UI toolkit). This taught me about Windows and Gadgets, and the message-passing event system IDCMP. However, one of the first things I discovered today on digging more into that is that people don’t use these things directly.
  • The AmigaOS Documentation eschews gadtools (the library for working with Intuition gadgets) in favour of BOOPSI, the Basic Object-Oriented Programming System for Intuition. Its documentation uses demos that only work on the latest (proprietary) AmigaOS, and not on earlier AmigaOS or AROS.
  • The AROS documentation eventually led me to discover that even gadgets in BOOPSI are old and busted, and that the Zune[**] widget set for BOOPSI (which is based on the freeware Magic User Interface) is the new hotness.
  • This follow-your-nose-and-end-up-lost documentation issue is a big problem. I’ve learned a lot, but mostly follow “this is how this works” with “but you don’t do this”. Plenty of reference is made to the original Amiga developer documentation (which, luckily, I have, on CD in an Amiga-only format) and the MUI docs (which I don’t have but can probably get). This is not an AROS-specific issue, wherever “open source version of X” is documented, the documentation is likely to say “check out the documentation for X”. This only works where the documentation for X is available (imagine, say, if Commodore were to go bust in 1993 and stop publishing their developer docs), and relevant to the reimplementation (imagine, say, if Apple were to change programming language and GNUstep didn’t catch up).
  • That AROS is both source and (on m68k, or on UAE) binary compatible with programs that were written in the 1980s is amazing to me, and much more valuable than having latest whizzbang features and smartwatch integration from a modern platform. Leaving aside technical aesthetics which I’m likely to disagree with a lot of people on anyway, AROS and open source[*] systems like it (Haiku, ReactOS, FreeDOS, GNUstep, Lesstif) represent a critical piece of heritage infrastructure.

[*] I’m using this as a shorthand. AROS source code is published under the terms of the AROS Public License, which is not OSI-approved. The source code is available to use, study, share and improve, which many would understand to be “open”.

[**] Unrelated to the Microsoft Zune, and it’s unclear to me which came first.

AROS

I spent the weekend teaching myself some Amiga GUI (Intuition) programming using AROS via the Icaros Desktop distribution, their developer guides, and the Amiga developer CD. It’s a nice enough system to program in that works like most other GUI systems; GUI events are sent to your process as messages that you receive, handle, and reply to. GUIs are built out of gadgets attached to windows displayed on screens.

As a system to use, it feels efficient and fast. AROS is, as the Rhapsody developer releases were, a platform displaced in time: someone designed it to run on a 680×0 with a couple of megabytes of RAM and a floppy drive, and here we are with our modern CPU, gigabytes of RAM and solid state storage. It starts near-instantly, responds quickly (even though it’s running in VirtualBox under a Windows 10 host), it’s beautiful. And it’s not overloaded with widgets and gewgaws added by vendors who are keeping up with the Joneses by adding every new feature to every platform they support. No notifications ask me whether I’m interested in a tour of all the latest features that I’d be unable to find myself.

As I write this post (in the Odyssey Web Browser in Icaros Desktop, of course), I wonder whether I need all of the other things I would get if I were doing this directly in Windows, or in macOS, or a modern Linux distribution. Give me a compiler long enough and a place to save and I shall move the Earth.

Computing’s fundamental Principle of No Learning

I haven’t used Taligent’s frameworks or operating systems directly; what I know of it comes from their documentation and the book Inside Taligent Technology. I put some small effort into finding out whether it’s possible to use the Taligent system legitimately on a modern computer, couldn’t find a way, and didn’t get too sad about that.

The Taligent frameworks look – assuming they were ever as complete as the documentation promised – like fairly standard 1990s OOP-in-C++ which almost certainly makes them less fun to use than modern Qt. What I got from considering Taligent was a reflection on trends in modern computing platforms, that discoverability is killing power user features, and the app icon is killing complex workflows.

There is a false dichotomy at work in modern app design: the drive is for apps to be so simple you can use them as soon as you’ve tapped the app icon, but this is taken to mean that there doesn’t need to be anything more to do in the app than what you can see when you have tapped the app icon. As an example, the bookmark manager app on my iPad shows me a list of my bookmarks, and there’s an “add” button. If I click on a bookmark, I can view it. That’s it. That’s “managing” bookmarks, app-style.

But I would suggest it’s not just the apps that are doing it, it’s the platforms too, and that’s how Taligent entered this story. Their central desktop metaphor was called People, Places, and Things, based on the idea that I might want to focus on those things mediated by a computer, rather than on applications that support particular use cases.

A particular example of the People metaphor is that I might want to show a photo of my cat to my sister. Back in the days of the real world, I would do that by showing the photo of my cat to my sister. In the brave new world of the app, I look at my phone to see what messaging apps I have, try to remember which ones I have her as a contact on, which ones she will actually check, whether they send pictures and if so, what that costs or how it futzes with the picture. I launch that, go to the conversation with her (or start one), add an attachment, try to think about whether this needs to be a Camera, Gallery or File attachment, and so on. This is made most painful when using the in-car voice UI on my smartphone, which seems able to transcribe messages well but unable to listen when it asks what app I want it to use to send the message. Like I care, cost and picture-futzing aside (though I’m not usually taking photos with the in-car voice UI).

The Places metaphor is particularly interesting. Each “place” is a virtual gallery, town hall, office lobby, project office, or whatever that contextualises a given group of People and Things relevant to the place. In implementation, it’s a sort of context-sensitive desktop management. In concept, it makes me think of the PARC Ubiquitous Computing principle and how a location-sensitive computing environment could adapt as I sit at my desk, join some colleagues in a meeting room, or collapse on a beanbag at home. It makes me think of the Eudaimonia machine and how a context-sensitive computing interface could support shallow-and-broad to narrow-and-deep work environments. Then I consider how the static home screen on my smartphone…does not do those things.

An important part, I believe, in the downfall of People, Places and Things is what it means for software. Rather than buying “an application” from a vendor that shows me their beautiful, crafted icon until I’m ready to launch it, I buy a capability that extends the ways in which I can interact with the People, Places and Things represented on my computer. My computer does not become Photoshop, rather I gain Photoshopness by adding it to my computer.

It seems that any shift from app-centric to goal-centric computer interfaces – PPT, OpenDoc/OLE, NeXT Services, scripting/automation tools, the spatial desktop – is going to violate the Fundamental Paradox of App Marketing: my app is so intuitive and easy to use that it just fades into the backdrop of your life, however it must never fade so far that you do not see its name or its icon, and are not reminded who we are and how much we help you.

Similarly, they violate the Principle of No Learning discussed above: I must be able to see how to do everything, and I must only be able to do everything that can be seen. The desktop metaphor allows for spatial arrangement of your work, just like strewing papers around a real desktop, but you need to work out what those funny line-and-square icons mean and which bits of the window frame are draggable, so that goes. Drag-and-drop makes for a visual placement metaphor letting me “put” work from one document into another, but you can’t obviously see that it’s a thing (nor indeed what it will actually achieve; Drag and Drop seems intuitively destructive, sometimes is, but sometimes surprisingly isn’t. What’s the metaphorical outcome expected when you try to put your Hackers DVD in the Trash?). Therefore drag-and-drop is replaced with “Share Sheets”. Scripting not only requires learning a new application – the script editor – but a whole new user interface for existing applications. Forget it.

The Paradox of App Marketing is merely misguided self-aggrandisement, but the Principle of No Learning is one of computing’s big, stultifying, patronising, antiethical mistakes. It makes out that computers are simplistic tools that can do the things delivered unto the users by the app makers, and nothing else. It hides the magic behind the curtain by removing the curtain and the magic.

Reasoning about reasoning about software

Functional programmers like to claim that you can’t reason about mutable state programs. Some thoughts:

  • the first half of the book A Discipline of Programming by Edsger W. Dijkstra tells you how to do it. That half of the book is approximately 100 pages (the remainder of the book is worked examples).
  • object-oriented programming breaks a software system up into separate systems running miniature, message-driven programs as if on separate computers. Therefore the consideration of “mutable state” can be split in two: the state internal to the object and the state external to the object which sends messages to the object but is ignorant of its internals. If you can’t split the state that way, you have bad encapsulation.
  • The reasoning done about the external and internal behaviours had better match at the interface. Design by contract probably helps here.
  • Given a state S, an operation O can be defined as O(args \times S) \rightarrow (R \times S'), i.e. it returns a result R and updates the state to S’.
  • However, Bertrand Meyer introduced Command-Query Separation in the 1980s, so you only need to know O(args \times S) \rightarrow (R \times S) and O(args \times S) \rightarrow (\emptyset \times S').
  • Various history “traces” can be considered equivalent and therefore a lot of knowledge about the historical state transitions elided, simplifying the reasoning. For example, given a well-designed stack, it is impossible to distinguish the history of stack.push(3); stack.pop(); stack.push(7) from stack.push(7).
  • Various operations on the state are irrelevant to the behaviour of an operation under consideration. In reasoning about the final operation in a = 3; b = 7; c = 9; stack.push(2) you do not need to consider the assignment operations (and indeed their presence may indicate a cohesion problem in your design).
  • The one remaining source of difficulty is aliasing; I do need to know about the elided operations in the sequence x = 7; *y = &x; ...; z=f(x). This is aliasing, not mutable state.

All the things

It’s been a long time since I had a side project, or one that didn’t get abandoned very early on. I tend to get sidetracked by other thoughts about computing, or think “while I’m doing this, I’m leaving that unsolved” so nothing gets very far.

In an attempt to address that, to clear all of the different thoughts I have about the matter of computing out of my head, organise them, identify conflicts, and prioritise what I work on, I spent this evening jotting down the big points and a brief abstract about each one. I’m hoping this will cut the Gordian knot by letting me see it all in one place and start to make choices.

The format I chose to represent this braindump is this personal Technology Radar, based on the Thoughtworks build-your-own tool. It seemed like a good place to see everything at once, and look for clusters or trends.

You’ll notice that almost everything in this radar is fairly old tech! That’s mostly a matter of taste, as I enjoy learning about things that were tried, what succeeded or failed, and what can be learnt from that to put to use today. I’m not good at novelty for novelty’s sake.

I expect to get some mileage (for my own benefit, you might like it too) out of expanding on some of the entries in this radar over a few more posts, so I’ve created a techradar category in this blog that you can filter on/out.