This episode discusses the final section of the 1969 software engineering techniques conference report…eventually, after discussing how statistics does or doesn’t work. I also reflect on both conferences, and what it means that the experts in 1968-1969 told us how to solve problems we still have in 2026, using solutions we still refuse to try.
The episode is supported by Software Engineering’s Consulting Detective, along with members of the Chiron Codex Patreon (use this gift link for your first month free). Please do join the community or hit the Ko-Fi button to make a one-off donation. If you enjoy the episode, share it with at least one friend, colleague, or stranger.
Links
- Software Engineering Techniques: Report on a conference sponsored by the NATO Science Committee
- The NATO Software Engineering conferences
- Podcast Reading List
- How to Use and Report on p-values
- Big little lies: a compendium and simulation of p-hacking strategies
- Points were a failed experiment
- The NoEstimates Movement
Transcript
Hello, and welcome to episode 63 of the Structure and Interpretation of Computer Programmers podcast. I’m Graham Lee, and this episode is the seventh and final part in a mini-series exploring the NATO Science Committee conferences on software engineering. The episode is sponsored by you, the software engineering community. More on that later, but if you enjoy this podcast, please share the link with your friends and colleagues.
Yes, that’s right, we’re finally getting to the end of the 1969 NATO conference on software engineering techniques, and there hasn’t been another one since then. In this episode,
I’m going to dip in and out of the conference reports collection of working papers, already itself a selected set of the papers submitted to the conference, and pick out some
interesting points to share. Then, I’ll wrap up with an overall summary of my interpretations of the two conferences. After that, I’m going to go away and think about what
material to cover in episode 64, which is a problem I haven’t had in months. Remember, the reading list is available at se100.chironcodex.com (link in show notes) so you can see
what I’m working on and what I’ve already covered in earlier episodes. If there’s something you want to hear an episode on soon and it’s in the list, send me an email. If there’s
something you want to hear an episode on soon and it isn’t in the list, then send me an email. I might get around to adding it and it will be on a much longer lead time.
Before we get into the conference report, an important point about quantitative measurement that frequently gets lost in arguments about how to use quantitative measurements. That point is that quantitative measurements support your ability to make judgments or form opinions, and those are qualitative in nature. It’s easy to evaluate a quantitative measure, for example, by comparing it against a threshold, or by ranking the quantities you measure for two different conditions, and therefore to replace judgment with evaluation. For example, here’s a product that scores 7 on my review scale, and here’s a product that scores 7.1, so the second product must be, and I’m using my podcaster’s scare quote fingers here, “better”. It feels dispassionate or objective to let the numbers guide you, when in fact you are avoiding a lot of opportunity to think about what’s really important in the situation and to consider your numerical measurements in the context of what you need.
Take, as an example, statistical analysis as used by scientific researchers. Science as designed is an attempt to persuade people that a model of the world—or of the universe, or whatever—is useful, supported in part by evidence. Empirical results are a great form of evidence. Use this model of the world to think about things, because we’ve measured how the world works, and the results seem consistent with the model’s predictions. Unfortunately, the real world is nowhere near as neat as the model. I mean, that’s the whole point of building the model in the first place. Sometimes the world can behave in ways that seem pretty inconsistent with our expectations, even if the model produces good predictions in loads of other situations. And sometimes the world can behave in ways that seem pretty consistent with our expectations, even if our expectations are based on a collection of incorrect assumptions.
Statistics offers us a suite of tools to help understand this. Given a collection of predictions from a model, and a collection of observations from the world, a statistician can tell something about how likely it is that the model’s predictions are usually consistent with the real outcomes, even if sometimes the actual results don’t look consistent. Putting the same thing in another way, a statistician can tell something about how likely it is that the model has got nothing to do with the actual outcomes, and just got lucky, even if sometimes the actual results seem to match up with the predictions. That can be useful information, alongside more; the applicability and quality of the model, the intuition you bring as an expert in the problem domain. All of that helps you decide whether to use the model, and in the context of science, thus to reward the creator or experimenter by citing their work in your subsequent work that builds on theirs.
Statistics is also a branch of mathematics, so it’s a quantitative field that gives numerical results. The idea I was just explaining, deciding whether a model has genuine predictive power or just strikes it lucky is summarised by a quantity called the p-value. We can loosely describe the p-value like this. The probability that the hypothesis that the model happened to generate results similar to observations, despite being unrelated to the real conditions, is true. If a statistical test yields a p-value of 1, then that test is saying there’s no way the model is unrelated to reality. The results obtained occurred entirely by chance. If a test yields a p-value of 0, then that test is saying there’s no way the model is unrelated to reality. The results obtained occurred entirely because the model accurately predicts reality.
The p-value is a boolean value, but it isn’t a toggle. It’s a real number. Digging into that sentence would be another aside that I’m deferring for another episode. For the moment, please just accept that the p-value is a number between 0 and 1, potentially including the boundaries. So the p-value might be 0.5, in which case the test is reporting that there’s a 50% probability you got the results you saw because the model’s good, and a 50% probability that you got the results through random chance. So, a statistical test can’t tell you this model is good or this model is bad, or this model is useful, or this model is useless. It gives you a level of confidence. Combine your confidence in the model with the other attributes of the model and come to a qualitative assessment about whether or how to use the model.
But that’s hard, and it doesn’t sound particularly objective. So scientists shortcut a lot of reasoning and just look at the number. They optimise for what they can measure. They declare a particular threshold for the p-value, say 0.05 (a value that’s common in many fields but not universal in science), and that value they call significant. Again, podcast scare quote fingers. So, if there’s a 1 in 20 chance that your model looks good through dumb luck, you still win at science. But notice that there isn’t a great deal of numerical difference between a situation where the p-value is 0.0499 and one where it’s 0.0501. In both of these conditions, there’s about a 1 in 20 chance that your model looks good through dumb luck. That doesn’t matter to the scientists, though. That 1 in 5,000 difference is the difference between science and nonsense.
Because there’s now a yes or no question that represents the difference between success and failure, scientists optimise not for creating good explanations and evaluating them through quality argumentation. They optimise for getting that p-value over the threshold and hitting success. This optimisation is called p-hacking, and there are multiple ways to do it. Different statistical tests make different assumptions about how dumb luck works, so you can get a different p-value by running a different test. So, if the test that’s most relevant to your actual situation says the p-value is 0.07 and you’re trying to sneak under 0.05 to win at science, try running a different test that makes assumptions that aren’t relevant to your actual situation. Get that 0.05 score and then hand-wave past your choice of test in your paper and make sure that your reviewers don’t notice.
Alternatively, don’t report all of the data you gathered. Only report the values where the model and the reality are really well aligned. Then perform your statistical attempts on only those values. You’ll get a much higher p-value because you’ve pre-emptively removed most of the effect of dumb luck from your situation. Alternatively, carry on making more observations and adding more observations, adding more values, until you hit the jackpot, and the number just nudges below 0.05. Alternatively, collect a bunch of data and reverse-engineer a model that has a p of less than 0.05 for those observations, even if that model provides no intellectual value and makes no intuitive sense. These are just some of a whole plethora of different approaches to p-hacking, data dredging, or just straight-up cheating at science, because the career incentives for a scientist aren’t consistent with doing good science.
Why do I tell you this? Well, software engineering has its own version of p-hacking because we have our own numerical quantitative scores, estimation. Estimation itself is a qualitative process that provides a complex and nuanced answer to the question: “does it make sense for us to do this work”? Or it leads us to ask other questions, like “would it make sense to do this work if we had different constraints, say more people, more time, lower expectations, different customers, reduced scope, different work, or professional indemnity insurance”?
While estimation is qualitative, an estimate is quantitative. It’s typically a number in units of time. For example, “this task seems like it will take a couple of days”. Or in person-time, “it would take our team two weeks to build that feature”. Though there was a failed experiment by some methodologists to introduce the dimensionless estimate or story point, and I’m sure someone will post to LinkedIn soon recommending the person-token-time dimension for estimates.
The estimate ought to act as a quantitative assessment that you use with other information to come to a qualitative assessment of your actions. Does the expected value derived from building this justify the cost we estimate it to require? Is there a valuable subset of the scope that we could do instead or a better use of our time? Do we need to hire some more people or train the team? When should we book the customer meeting to get their input? Do we even understand what we’re doing well enough to get started?
But because the estimate’s numeric, people just look at that number and do numbery things with it. They turn estimates into commitments (“you said you would take two weeks, so we’re giving you 10 business days and then doing a press launch, regardless of how many emails we send you and meetings we invite you to during those 10 days”). They set release schedules by adding up all the estimates. They give the tasks of the team or vendor that produces the lowest numbered estimate, and so on.
In the 1969 conference report, J.D. Aron made the point that this is a mistake. An estimate is the output of a model, and as the digression at the beginning of the episode demonstrated, a valid model can give predictions that diverge from reality, and a model can be invalid even if it sometimes gives correct-looking numbers. Use the estimate as input into your qualitative decisions on how to proceed. Don’t treat them as true, and don’t treat them as the only source of data. That said, they are useful input, so don’t avoid their explaining power altogether.
His model, Aron’s model, is the simplest one that makes intuitive sense, a model that still gets used to this day, even if it’s given a load more fancy frills and has been renamed to Velocity. His model’s input is yesterday’s weather, on the basis that the way you work tomorrow is probably similar to how you worked yesterday. So, Aron’s formula, has you work out the average rate at which your team has delivered code in the past, he measures in instructions per person day, then divide your estimate of the size of the future work in instructions by that rate that you worked out to get an estimate of the time required to complete the future work in person days. Yes, that’s right. You use an estimate, your guess at the size of the task, to get your estimate, your guess at the time required. That’s why you use the estimate as an input into the overall judgment of what to do, not as the judgment itself.
This is the advert break.
It starts now.
Your development team is slowed down, your customers are frustrated, and your support team is constantly banging down the door. Nobody can find the information they need, and they struggle to talk to each other to work it out. Relationships on the team have soured, with developers, QA and ops all taking an us-versus-them position. Your backlog grows faster than the team delivers features. Retrospectives involve more finger-pointing than decision-making. This sounds like a case for Software Craftsmanship’s consulting detective. That’s me. Discover your team’s friction points and create realistic action plans for improvement. Restructure your delivery pipeline to bridge the quality gap, automate improvements, and reduce rework. Then, take ownership of the documentation and curate a knowledge hub that keeps information flowing throughout your organization, empowering engineers, executives, partners, and customers alike. Find out more, and book a free initial assessment at sicpers.info.agency. That’s S-I-C-P-E-R-S dot I-N-F-O slash agency. Together with me, Software Craftsmanship’s consulting detective, we’ll crack the case.
That was the advert break.
It’s over now.
I promise that I can get through the rest of the report with, if not fewer, then at least smaller digressions. It’s worth stopping in at least briefly on Dijkstra’s paper, Structured Programming. This contains the admonition that, quote, “Program testing can be used to show the presence of bugs, but never their absence!” This is actually a little different from a quote that I used in episode 61, quote, “Testing shows the presence, not the absence of bugs.” And that makes me think, along with the fact that he used the same aphorism again in his 1972 Turing Award speech, that this is just a favourite phrase of his, he literally did use it both in the working paper and in conversation at the conference. Dijkstra’s main idea in the paper is that correctness, like availability, is a compounding failure. Your confidence in your whole system is at best equal, but probably worse than your confidence in your lowest-quality module.
He also proposes that modules are arranged, quote, “like strings of pearls on a necklace”, a model that implies not only a strict layering, but also no horizontal communication. A module might have another module below it and another module above it, but it doesn’t have any direct access to the other modules at the same or adjacent levels. Given how strict Dijkstra himself was about the onion metaphor in episode 62, I think it’s only fair that we apply the same level of pedantry here. He does describe using the structure of the program to draw conclusions about the behaviour of the program based on the behaviour of the components in the structure. That’s why we call it structured programming, of course. And he says, quote, “sequencing should be controlled by alternative conditional and repetitive clauses and procedure calls rather than by statements transferring control to labelled points”. In other words, Dijkstra here is warming up to the idea that he considers the go-to statement to be harmful.
Falkoff uses APL as an example of his general principle that the appropriate specification language for a system is an executable programming language. English has its place in a specification, particularly in the informal parts and in explanatory documentation, but there’s no substitute for a simple, expressive, formally designed programming language that you can actually run on a computer. This is a different direction from the separate, abstract specification language and concrete implementation language that was argued for in episode 61 and Falkoff is clear on explaining his reasons.
Designing down to the implementation level allows the designer to compare different implementations aesthetically, which can give them clear hints about problems in certain approaches, i.e. if it looks complex when designed one way, that way is probably complex. It also means that the model acts as a walking skeleton that you can explore and critique. Note that he’s talking about a system specification language, not necessarily a software specification language. You might find that you want or need to build some of the modules in hardware, but the executable specification gives you a way to prototype those hardware components in simulation. The language then needs to be abstract enough to let you simulate those modules without committing to designing them like their software that runs on your development machine.
Jules Schwartz’s paper on analysing large-scale system development offers a couple of gems, which, maybe with hindsight, seem to follow the pattern established in the 1968 conference of presaging modern software engineering without any contemporary readers having paid any attention. These include, quote, “measuring user interest and performance on a particular system and how he performs with and benefits from various aspects of the system with which he operates”— and I’m using the pronouns from the original in that quote— in which we see analytics and customer-centric design. The very next paragraph tells us that, quote, “testing should be a consideration throughout the entire development” and that, quote, “overall system evolution is the major problem with testing as one of the major factors”.
Skipping over a few full-page diagrams, the next page of text introduces both the 10x programmer, quote, “the difference between [debugging in interactive and offline environments] was approximately a factor of two while the noticeable difference in programmers who participated in the experiment seemed to range up to more than a factor of 20!” and, seven years before the man himself published it, Brooks’s Law, quote, “as schedules tended to slip or difficulties were recognised, more people were hired. This, of course, required more management and communication. This cycle continued for several years until many hundreds of people were involved in the programming effort. […] When asked what he would do differently if he had to do a system like this again, the manager, after some period of reflection, said he would hire 12 good people to do the whole job.”
Schwartz’s overall conclusion is that large system development is a people problem that needs to be solved by having clear communication and by adapting to new situations as they arise through the whole development, which is something we still need to reiterate to programming managers 57 years later.
But the real diamond in the rough that is this fractious conference is the final working paper that I want to cover here, Warren Teitelman’s Toward a Programming Laboratory. He describes a live programming system called PILOT written in Lisp with features that more than once led me to raise an eyebrow because it’s amazing that they already existed back in 1969. These include automatic error correcting. The system detects mismatched parentheses or misspelt names at runtime, fixes them, and carries on. It includes evaluation of these fixes with a human in the loop. If the system has a low confidence in the correction, it stops and asks the programmer for help. It includes that many programmers automatically accept the system’s suggestion without reviewing it, and I wonder whether they had both the options 1: yes, and 2: yes, and always allow for this session, just like Claude Code.
Unfixable errors break into an interactive debugger where the programmer corrects the program and then carries on running it. The system includes function advising in which the programmer can attach additional codes to or even replace the implementation of any function call. And they describe the beginnings of a “program writing program”, something that accepts English language descriptions of desired software and converts them into Lisp programs that have the requested behavior.
Certain advances in computing, particularly Smalltalk-80, the early 1980s Lisp-based interface builder, and Extreme Programming, make me feel like Charlton Heston when he’s riding along the beach and sees the ruined Statue of Liberty poking out of the sound. If we had that back then, how did we let the world go to ruin and get what we’ve got now? This pilot programming laboratory is another of those moments.
But, its presence in the 1969 Software Engineering Techniques Conference summarizes the whole NATO software engineering effort quite neatly. They do appear to have gathered the Western world’s— and as I covered in episode 58, that means the world’s— foremost experts in software engineering, software project management, and computer science. They appear to have convinced them all to say rather clever things, and somehow subsequently arranged for them to have no influence whatsoever on software engineering practice, except to change some job titles from programmer or systems analyst to software engineer, and to have widened an existing gulf between the mathematical and theoretical computer science discipline and the practical and process-centric programming praxis.
The core themes of the conference were, on the people side, clear and frequent communication, adaptation to feedback, willingness to evolve initial ideas, lessons that still only get partially applied today. On the technical side, they were saying about early and continuous testing and integration, modularity with clean separation of concerns, reliance on validation techniques beyond testing, most of which are barely practiced at all, even in organizations that pay lip service to their disciplined engineering approaches.
It’s not really fair, though, to go around in 2026 and beat people over the head with these 1960s conference reports and say, “look, people knew how to do this 60 years ago.” This is a dissemination or education problem. Both of the conferences make it clear that, at the time, the question of how to train people to write good software was unsolved, and I don’t think we’ve solved it in the meantime. Whether we’re talking about mentorship approaches like software craftsmanship (whether that’s as described by Pete McBreen or as subsequently adopted and recreated by Bob Martin), university courses, boot camps, on-the-job training, or anything, it’s fairer to take an introspective and reflective look at ourselves and ask, “how do we fix this situation so that people even know the things that were state-of-the-art when our grandparents were still in the workforce?”
That bombshell is the end of this mini-series and the last thing I say on the NATO Science Committee conferences on software engineering from 1968 and 1969. The next episode takes us in a new direction. As ever, share your thoughts on this episode in the comments on the post on the SICPERS.info website, that’s S-I-C-P-E-R-S dot info slash podcast, or email Graham Lee at ACM dot org. Recommend the podcast to your friends, find more info at the Patreon page which is linked in the show notes and let’s talk soon.
Seems like a superperfect episode to start something new.