This episode continues a review of the 1969 conference on software engineering techniques looking at software adaptability, large software projects, and software engineering education. The episode is supported by members of the Chiron Codex Patreon (use this gift link for your first month free), so please do join the community or hit the Ko-Fi button to make a one-off donation. If you enjoy the episode, share it with at least one friend, colleague, or stranger.
Links
- Software Engineering Techniques: Report on a conference sponsored by the NATO Science Committee
- The NATO Software Engineering conferences
- Two Pizza Teams
- Net Negative Producing Programmers
- Software Engineering at Oxford | Subjects (by theme)
Transcript
Hello, and welcome to episode 62 of the Structure and Interpretation of Computer
Programmers podcast. I’m Graham Lee, and this episode is the sixth part in a
mini-series exploring the NATO Software Engineering Conferences. The episode is
sponsored by you, the software engineering community. If you enjoy this podcast,
please share the link with your friends and colleagues.
In episode 61, the conference was dominated by two issues. First, an attempt to
bootstrap a NATO Software Engineering Institute that failed, and it isn’t
covered in the conference report. And second, a gulf between theoreticians and
practitioners, between academia and industry, between computer scientists and
software engineers, and between Edsger Dijkstra and anybody else. That last
ravine appears again at the outset of section 4 on software flexibility.
Alan Perlis starts off trying to make an analogy between a hierarchical design
and an onion. The design has multiple layers, each of which has an inner
interface, the way that it uses things from the lower layers, and an outer
interface, the things that it provides to the upper layers. Dijkstra jumps in to
say that the onion is a bad analogy because it’s three-dimensional, and a
hierarchical design is one-dimensional. He’d prefer if we kept those higher
dimensions free to represent other attributes of the design.
The report editors split flexibility into two categories. Portability, the
ability to run software in multiple systems, and adaptability, the ability to
keep up with changing requirements. Although it’s clear from the report that
some attendees think of migrating software between systems as an adaptability
problem, the two categories aren’t actually disjoint. Perlis was trying to make
a point about portability, which is a valid but incomplete definition; that
portability is concerned with “creating support on new machines sufficient to
accept systems transported from old machines”. In other words, if you can
emulate the facilities or interfaces of the old computer on the new computer,
you can run your old software on your new computer. Think of the way that
Solaris on sun4v hardware, at least until version 10, and I have never used
version 11, still supports system calls defined in BSD, so that Sun 4 programs
that were written for sun4m hardware still work.
Much of the discussion in the section, though, takes different approaches, the
first being a high-level language. As Stephen Warshall of Applied Data Research
puts it, a program is “descriptions by people of computational processes”, and
you can communicate that description without having to describe a computer.
Describe your solution at sufficient abstraction from the hardware capabilities
that it doesn’t rely on specific hardware provision, and then provide the
support facilities, in other words, the language runtime, that allows it to
execute on both the original system and on possible future systems. This fits
Perlis’s Onion model. The program in its high-level language is the outer
interface of the onion layer, and the language runtime support is the inner
interface. The trap is that the more abstract the computational model of the
language, the more capable the runtime needs to be to make the program
executable. Warshall’s solution is to add another layer to the Onion. As Kevlin
Henney says, any problem in computing is solvable with another layer of
indirection, except the problem of having too many layers of indirection. The
layer he adds is the virtual machine—”Consider the use of software as a device
for eliminating representational commitments”. That layer provides a simulated
computer that the high-level program runs on well, that expresses its needs in
terms of the facilities the target hardware provides. You lose the ability to
fine-tune performance by directly expressing the software in terms of the target
system, but you gain a lot in terms of portability.
Two problems with this approach are, one, that you can’t actually predict the
future very well, so an upcoming target system may have facilities you can’t use
from the virtual machine, or a design that makes the VM highly inefficient. As
an example, imagine virtualizing the tape interfaces on some computers, then
receiving hardware that has random access storage. Secondly, that the
abstractions provided by the high-level language become unrealistic. Here, the
example Warshall gives is, you might write all programs as though you had
infinite memory, and then let some automatic device worry entirely about storage
strategy and resource strategy. That’s actually what we do these days. If you
look at the memory allocator in a modern operating system, you’ll usually find
that the malloc function can’t fail, and always returns a non-null pointer to
virtual memory. Or at least with a 64-bit address space, it takes a long time
before it runs out of pointers to give you, which is the only reason that it
actually does fail. It’s only when the program actually tries to use that memory
that the runtime tries to get the kernel to commit some memory to support it.
And it’s only when the kernel runs out of memory to offer that things start to
go sideways.
Of course, by relying on high-level languages, you really just kick the problem
down the road. Yes, your software is defined in a portable way, but until
someone ports the virtual machine or runtime library, it isn’t actually
portable. You can “bootstrap” porting something like a compiler by creating a
compiler that runs on the computer you already support and generates code for
the computer you want to support, and then compiling the compiler for that new
computer using the old computer, and then the compiler for the new computer
using the new computer. Strachey talks about porting Martin Richards’s BCPL
compiler, which is written in BCPL with some assembler language, by rewriting
the code generation part in ALGOL, providing a portable means of compiling the
compiler. Of interest to retro-computing fans is that Martin Richard’s also
wrote the Tripos operating system in BCPL, which became the basis of Commodore’s
AmigaDOS after their own in-house project ran into trouble.
An alternative approach to abstracting the target of your high-level language is
to use macros so that machine-dependent statements in your program have
different values on different target machines. Jerome Feldman and Dijkstra both
explore this idea, which evolved into its ultimate form with the GNU AutoTools—a
very complicated macro facility that generates programs to test what values to
give the macros, and that many people use without understanding it. They both
make the point that what you end up with isn’t a portable program, but a
software product line (though they don’t call it that because it’s a 1990s
phrase). What you end up with is a family of related programs and a process that
selects and creates one instance of that family.
Perlis wishes for tools that stop you doing machine-specific things, and warn
you if you somehow still manage to do it. But John Buxton of Warwick University
comes to the Gordian knot of portability in 1969, which is that the support
tools, the compilers, assemblers, runtime libraries and so on, that people
relied on in the 60s were, for the most part, supplied by the hardware vendors.
These are the people least invested in making it easy to port your software to a
different computer because you’ve already bought their computer if you’re using
their tools.
I just want to add a couple of other points from the discussion on flexibility,
because while they’re small, they’re very important. One is that General
Electric’s Bob Bemer and conference co-chair Professor Friedrich Bauer both
point to the needs to make data portable as well as programs to successfully
port a system, with Bemer lauding the COBOL separation of a program into
divisions as a facility that enables portability. Now, he would say that as he
was one of the creators of COBOL. The algorithms in a COBOL program are defined
in a procedure division. The variables, structures and data interfaces are in a
data division and the target machine and configuration are in an environment
division. You can use the same data and procedures on different machines or in
different configurations on the same machine just by changing the environment
division; the same procedures with different files by switching the data
division, and so on. Bauer’s idea is that you need to record the data schema
somewhere either in the program as COBOL does with its data division or in the
database itself so that the database becomes self-describing.
Another point is that Butler Lampson takes the point on flexible runtimes a
little further by saying that if you have modules with well-defined interfaces
you can wrap those modules in an “envelope” and reuse them in different contexts
without needing to change the module’s program. That idea crops up again in the
Gang of Four book where it’s called the Adapter Pattern.
This is the advert break. It starts now.
This episode is brought to you by me, Graham Lee. But really, by you. Chiron
Codex is a community of people who are learning how to become better software
engineers by adopting AI augmentation in a thoughtful way. We aren’t outsourcing
our understanding to coding assistants like Claude or Codex but becoming
software engineering centaurs by using AI tools to improve our knowledge and the
quality of our work. Join the community over on Patreon to find out about
interaction patterns that improve your work with AI coding tools, running LLMs
for software development locally, discussions of recent research in the field
and more. If you’re a software engineer who’s interested in the promise of AI
tools but sceptical about handing your skills over to the computer, this is the
community for you. Go to patreon.com slash Chiron Codex that’s
C-H-I-R-O-N-C-O-D-E-X now for more information and to join. Use the gift link in
the show notes to get your first month of insider access completely free.
Alternatively, you can show your appreciation by donating at Ko-fi. That’s
ko-fi.com slash Chiron Codex K-O-F-I dot com. Direct support by my audience is
the only revenue I get for my work as a software engineer and communicator. So
your support really means a lot to me and makes it possible for me to produce
this podcast. Thank you so much.
That was the advert break. It’s over now.
Section 5 is on large systems and we’re launched straight in with what has to be
the money quote for this episode from J.I. Schwartz of King Systems. “I must
admit that I have frequently asked myself whether these systems require many
people because they are large or whether they are large because they have many
people.”
Maybe both are true. We saw in the peak eras of Silicon Valley hiring
excess—which I define as the dot com boom that ended in 2001 and the web scale
boom that ended in 2023—that companies would hire software engineers just to try
to stop their competitors from hiring them, and would then need to find projects
for them, and would then grow systems in weird ways with weird personnel needs
that they could then cut when they needed to reduce headcount without materially
affecting their outcome. So maybe this is a kind of spatial variant of
Parkinson’s law, where the work expands to fit the headcount available.
After giving an example of a large project (the NASA Apollo mission which
featured at least 600 people from IBM with 300 programmers), J.D. Aron describes
their experience with “super-programmer” projects. Fred Brooks also writes about
this in 1971’s Mythical Man Month under the name “Chief Programmer Team”. In
both cases the super or chief programmer is Harlan B. Mills a man who with 36
PL/1 manuals open on his desk tried to replicate in six months a 30 person year
“army of ants” project. He failed at that, but still completed the project in a
small number of years. However, IBM were doing what they could to shield Mills
from their customer meaning that his more efficient project turned out to be
worse at doing what the customer actually wanted.
As Brooks presents it, the chief programmer is a highly capable expert in total
charge of the project with a few support staff to delegate some of the work to,
including: a full-time project wrangler; a support tool smith; and a programming
language expert. These teams can indeed get significant work done with fewer
resources than large teams but there are multiple factors at play. Is it the
Brooks’s law issue that there are fewer people so less time spent on
communication? This is the bet behind the two pizza team which doesn’t mention
chief programmers, just small teams. Is it that these chief programmers are
genuine 10x programmers and that the 300 person projects are full of NNPPs: net
negative producing programmers? Or is it that when someone comes along and says
“I bet I could do that in less time than your large team did”, they’ve
pre-selected the project as one that they think they can succeed at?
This is something that happened over and over again with the various Twitter
clones that have come and gone with one of the loudest being App.net. The team
behind App.net proudly announced how quickly they’d got their product off the
ground when it’s taken years for Twitter to build the same features. The two
differences of course were: one, that Twitter had to design and validate Twitter
whereas App.net just had to build an implementation of the existing design; and
two, Twitter still exists.
It’s in section 5.2, “the sources of problems in large systems,” that we find
another of the irreconcilable differences between industry and academia. The
conference has decided, just as its forerunner did, that large projects are the
source of most problems in software and there are a lot of regular software
initiatives that work just fine. So why did the large projects go wrong? On the
one hand Aron from IBM says “virtually all [reasons for failure are] essentially
management problems”. Yes there are technical problems but choosing the correct
techniques is itself a managerial problem. As Alan Perlis says, putting in his
runner-up entry for this episode’s money quote, “the managers wouldn’t know a
good technique if it hit them in the face”.
On the other hand Tony Hoare says that “basically all problems are technical”.
The reason that large projects fail is that they represent problems humanity
doesn’t know how to solve so it was a mistake for the manager to try to solve
them without first doing some basic research.
There’s a bit of presentation from working papers and discussion of the support
systems needed for large projects and the answer is basically an integrated
development and project management environment: version control, programming,
filing system, debugger, linter, job submission, time accounting, dependency
tracking, and more all in one database. It’s interesting to note that even
though IBM developed the environment they describe in this report (called
CLEAR-CASTER) the first commercial IDE was probably the rational R1000
workstation for ADA in 1985. Unless, of course, you consider the online
programming environment in Dartmouth Basic to count as an IDE.
The last section
to cover in today’s episode is section 6 on software engineering education. Alan
Perlis sets up the discussion with three questions:
- Is there a real distinction between software engineering and computer
science? - Given that the answer to question 1 is yes is there a need for education in
software engineering as a separate discipline? - What form should university courses in software engineering take?
It seems like question 3 assumes the answer to question 2 is also yes so in fact
there’s only one question here. I think the subsequent discussion mostly focuses
on the perceived prestige of different degrees in the US and the broken
incentive structure of US academia, with only a little diversion into the
question of how to teach software engineering. It turns out that computer
science at the doctoral level results in a PhD which is considered a, and I’m
using my scare quote fingers here, “proper degree”. You do a lot of teaching and
a bit of original research to get a PhD in computer science. Now, is it
necessarily the case that someone who wants to become qualified in software
engineering needs to advance the state of the art? Can’t they just learn the
practice? That question doesn’t quite go unanswered, but the answers do really
circle around the issue quite a lot.
The problem American academics, at least
the ones at the 1969 conference, have with software engineering is that if you
graduate an engineering program with a doctorate in engineering in the US, these
aren’t, using my quote fingers again, “proper degrees” in the way that a PhD is a
“proper degree”. Feel welcome to email me about this point if you wish, but you
shouldn’t get angry with me, I’m paraphrasing what’s in the report and I don’t
have an opinion on the status of an American doctorate in engineering.
Bob McClure summarises the state thus. Most computer science departments output
lone workers with computer science degrees whose career capability is best
described as “can become faculty at another computer science department and
train PhDs in computer science”. Universities thrive on their reputation and
their reputation is determined by whether academics think they’re any good, so
aspirational universities will try to emulate good universities by outputting CS
PhDs who can become faculty at CS departments, ideally in good universities.
Meanwhile, industry wants a lot of people with bachelors in engineering who can
do quite good work, some people with masters in engineering who can do good
work, some people with doctorates in engineering who can do very good work, or
direct the work of those with the bachelors and masters, and no people with PhDs
in computer science who can extend the state of the art but can’t work with
other people. However, due to the aforementioned degree snobbery, students want
PhDs, which universities are happy to help them with to boost their reputation
for creating PhDs because those are the, fingers again, “proper degrees”.
The four things we learn about software engineering education in this section
are: one, practitioners don’t read any research; two, a practitioner who did
read the research wouldn’t be much of a theoretician, and a theoretician who
wrote a program wouldn’t be much of a practitioner. That division again. Three,
the textbooks aren’t worth anything, assuming that they exist at all. Butler
Lampson only has time for Knuth’s The Art of Computer Programming, and that as
an encyclopedia, not as a textbook to teach from. Four, the body of knowledge
isn’t well defined enough in 1969 for Strachey to believe there’s a whole
degree’s worth of material to teach, which probably also explains why there
aren’t any good textbooks. That last part probably was true. The IEEE Computer
Society didn’t even put out a prototype guide to the software engineering body
of knowledge until 1998.
A question that remained open was whether software engineering in itself was a
valid discipline, or whether people also needed to learn something of hardware
engineering, in which case they would be more like informaticians or
cyberneticians than software engineers. Clearly, the people in favour of the
combined discipline lost. I have an MSc in software engineering. I’ve also
taught on that degree subject. I’ve linked to the syllabus in the show notes. No
hardware in there. Now, the question is, what drove that collective decision
that society made? Did software people not want to bother with the details of
hardware? Did universities not want to add hardware modules to bloated software
courses? Did employers not care whether software people knew about hardware? Did
hardware get too complex to teach in a software course?
I did a little bit of hardware engineering in my undergraduate degree, which was
physics, not computer science or software engineering. We had a little 8-bit
processor made out of TTL chips, that’s transistor-to-transistor logic, and we
had to extend it by adding a subtract instruction. Interesting, yes, but did it
make me a better software engineer? Or, for that matter, did it make me a better
physicist?
All that’s
left in the 1969 conference report, and therefore in this mini-series on the two
NATO software engineering conferences, is the collection of working papers. This
is more than half of the report’s body, but I won’t cover the whole thing in
detail. In the next episode, I’ll share some highlights from those papers. In
the meantime, you can leave your thoughts about this episode at the post on
sicpers.info, or email me, grahamlee at acm.org. Thanks so much for listening.
Please consider supporting the podcast on Patreon or on Ko-fi. Take care, and
we’ll talk soon.
