WEBVTT 00:00.000 --> 00:19.080 Hello, and welcome to episode 66 of the Structure and Interpretation of Computer Programmers 00:19.080 --> 00:24.340 podcast. I'm Graham Lee. The episode is sponsored by you, the software engineering community. 00:24.720 --> 00:28.240 If you enjoy this podcast, please share the link with your friends and colleagues. 00:28.240 --> 00:33.520 You can give feedback on this episode by commenting on the post at sicpers.info slash podcast, 00:33.860 --> 00:40.800 that's s-i-c-p-e-r-s dot info slash podcast, or by sending me an email, grahamlee at acm.org. 00:42.080 --> 00:47.660 Of all the monsters who fill the nightmares of our folklore, none terrify more than werewolves, 00:47.960 --> 00:52.680 because they transform unexpectedly from the familiar into horrors. For these, 00:52.680 --> 00:56.420 we seek bullets of silver that can magically lay them to rest. 00:57.480 --> 01:03.240 These are Fred Brooks' opening words in No Silver Bullet: Essence and Accident in Software Engineering. 01:03.880 --> 01:08.420 The essay is frequently read alone, and the International Federation for Information 01:08.420 --> 01:14.620 Processing originally invited Brooks to present the paper at their World Computing Conference in 1986. 01:14.620 --> 01:21.100 Many people who read the essay would have found it in collection, as part of the 1995 20th 01:21.100 --> 01:27.980 anniversary edition of Brooks' book, The Mythical Man Month. Acknowledging that context, this episode 01:27.980 --> 01:36.100 covers both NSB and the subsequent chapter in the 1995 book, No Silver Bullet Refined, which is 01:36.100 --> 01:41.960 incorrectly named No Silver Bullet Refired in many online contexts, apparently on the back of one blog post 01:41.960 --> 01:51.140 that got it wrong. This approach, reading both the article and the subsequent review that Brooks made 01:51.140 --> 01:56.820 himself, hopefully mitigates one of my bugbears, which is that people take aphorisms from software 01:56.820 --> 02:03.240 engineering papers, like No Silver Bullet, Mythical Man Month, Go-To Statement Considered Harmful, 02:03.680 --> 02:09.440 Premature Optimisation Is the Root of All Evil, and more, and quote or discuss them out of context, 02:09.440 --> 02:14.460 changing the meaning of the aphorism along the way. So let's go into this episode with the knowledge that 02:14.460 --> 02:19.460 we're not only reading an important paper from our field's history, but also the commentary that the author 02:19.460 --> 02:23.500 himself produced, reviewing multiple responses to his own work. 02:24.640 --> 02:31.060 By the way, long-time listeners will recall that I discussed NSB in episode 12, No Silver Bullets. 02:31.620 --> 02:35.760 I encourage reviewing that episode and thinking about how my argument has changed in the intervening 02:35.760 --> 02:42.640 six years, along with my 2024 blog post, Still No Silver Bullet. Links to both, of course, in the show notes. 02:44.300 --> 02:50.660 In No Silver Bullet, Brooks takes an Aristotelian view of software complexity. That means he sees two 02:50.660 --> 02:56.080 sources of complexity. The essential complexity that exists because the problem we're trying to solve 02:56.080 --> 03:01.440 using software is hard, and the accidental complexity that exists because we software engineers make 03:01.440 --> 03:08.520 problems for ourselves that we then have to solve. Hitherto, i.e. before 1986, Brooks says that all 03:08.520 --> 03:13.660 advances in software engineering had come about by tackling the accidental complexity, creating 03:13.660 --> 03:19.560 trustworthy compilers so people could write software in domain vocabulary, creating online programming and 03:19.560 --> 03:24.000 debugging environments so people could use computers to work computers, that sort of thing. 03:24.000 --> 03:30.960 His thesis is that more than 90% of that work had been done, so there cannot be any single advance 03:30.960 --> 03:35.560 in the art of software engineering that offers an order of magnitude increase in productivity. 03:36.700 --> 03:41.320 Incremental improvements are available for the taking, but any big wins must necessarily come 03:41.320 --> 03:46.600 from changes to the way that we approach the essential complexity. Specifically, he doesn't see 03:46.600 --> 03:50.840 any order of magnitude improvement coming in the next decade, so before 1996. 03:50.840 --> 03:57.200 I've been in the software business professionally for over 20 years, and I've seen lots of changes 03:57.200 --> 04:01.460 in the art of software engineering, in the way we approach problems and the problems we create 04:01.460 --> 04:07.900 for ourselves as a result over that time. Some of those changes are continuations of trends that 04:07.900 --> 04:12.200 were already in motion when Brooks wrote this paper, and some are approaches or tools that we've 04:12.200 --> 04:17.960 created since then. I'm starting to bristle at the idea that accidental complexity was mostly solved 04:17.960 --> 04:25.080 in 1986, but I'm going to be fair and let Brooks lay out his stall before we compare his expectations 04:25.080 --> 04:33.060 to our reality. He defines the essential complexity as the abstract conceptual tasks of creating a 04:33.060 --> 04:38.720 computable solution to a problem, that is specifying, designing and testing the data structures, 04:38.900 --> 04:43.880 algorithms and connections between them that go to make up software. There's a lot of complexity 04:43.880 --> 04:49.740 in this problem because software is discontinuous, non-linear and non-repetitive. While two statements 04:49.740 --> 04:54.880 in a program might be identical, two subroutines can be entirely unrelated even when they're composed 04:54.880 --> 05:01.000 of the same collection of identical statements. The only real tool we have to manage essential complexity 05:01.000 --> 05:08.140 is abstraction, and that can hide the abstracted concept's essence as easily as it hides its complexity. 05:08.140 --> 05:14.640 Further, this complexity is subject to constant change, and is difficult or impossible to visualise, 05:15.000 --> 05:19.780 and a lot of the complexity is due to arbitrary complexity in the problem domain, which frequently 05:19.780 --> 05:25.940 contains structures and concepts invented by people, not laws of nature, so there's very little 05:25.940 --> 05:33.660 reduction available. From his vantage point in 1986, Brooks describes some technological interventions 05:33.660 --> 05:39.300 that are touted as potential silver bullets, and evaluates them, unsurprisingly mostly negatively. 05:40.060 --> 05:45.300 For new high-level languages like Ada, he points out that we already have high-level languages. 05:45.880 --> 05:50.920 Most of the accidental complexity is removed by the transition from assembler to any high-level 05:50.920 --> 05:57.140 language, so inventing new ones doesn't offer significant benefit. He separates object-oriented 05:57.140 --> 06:02.620 programming into abstract data types and hierarchical types, that is, classes with inheritance. 06:02.620 --> 06:08.680 Both of these remove accidental complexity by closing the gap between the essence of the design and 06:08.680 --> 06:13.880 the syntax of the language. And he seems to allow that OOP may be the end state of such design 06:13.880 --> 06:20.040 techniques, but is sceptical that 90% of the remaining work people do is in fitting badly specified 06:20.040 --> 06:26.080 design paradigms to their problem's essential complexities. In other words, OOP is the best tool for 06:26.080 --> 06:31.140 addressing accidental complexity, and that still isn't worth an order of magnitude improvement. 06:31.140 --> 06:40.040 There are then two entries on AI. The first is the use of computers to "solve problems that previously 06:40.040 --> 06:46.920 could only be solved by applying human intelligence"; Brooks quoting David L. Parnas. He also quotes 06:46.920 --> 06:51.880 Parnas in agreeing that "no facilitation of expression can give more than marginal gains". 06:52.480 --> 06:56.580 This seems like a very big point of contention in today's software engineering landscape, 06:56.580 --> 07:03.140 along with "automatic programming", the generation of a program for solving a problem from a statement 07:03.140 --> 07:07.960 of the problem specification. Because in today's world, both of those are intertwingled. 07:08.760 --> 07:13.900 At the time, Brooks said that he didn't see how program generation could generalize to ordinary 07:13.900 --> 07:20.080 software systems. And indeed, within the decade of Brooks's predictive horizon, they didn't. 07:20.080 --> 07:27.700 The other entry for AI is expert systems, which were the promising AI technique in the 1980s, 07:28.140 --> 07:33.400 where you discover the rules that experts in a domain use to solve problems and teach those rules 07:33.400 --> 07:40.560 to a computer so that the computer can then apply those rules to novel problems. Brooks sees the main 07:40.560 --> 07:46.120 contribution of expert systems being in closing capability gaps. Even the least experienced programmer 07:46.120 --> 07:51.280 can apply the wisdom of the most experienced. That doesn't, of course, address complexity at all. 07:51.560 --> 07:55.640 It just makes it possible for more people to handle the existing complexity. 07:57.520 --> 08:03.020 There was a fad in the 1980s for graphical programming systems. The most likely one that 08:03.020 --> 08:09.180 you will have seen today is called Scratch. Graphical programming is basically a high-level language. 08:09.180 --> 08:15.920 It just happens to be a visual one rather than a textual one. So to Brooks, it's no better or worse 08:15.920 --> 08:21.840 than any other representation of software, which is hard to visualize. So to him, a graphical 08:21.840 --> 08:28.900 programming environment is just a different form of accidental complexity. Similarly, he dismisses 08:28.900 --> 08:34.400 program verification, a technique that tells you whether the software accurately reflects the design, 08:34.400 --> 08:40.160 but not whether the design is any good. It closes off a portion of accidental complexity. 08:40.160 --> 08:46.240 He dismisses development environments and tools, which might be able to yield marginal gains, 08:46.240 --> 08:52.240 and powerful workstations, which might reduce compilation time, but wouldn't reduce thinking 08:52.240 --> 08:57.760 time. Of course, reducing compilation time allows you to make your compiler more complex, 08:57.760 --> 09:04.080 so that it does more work in the same time rather than the same work in less time. So we see languages 09:04.080 --> 09:10.080 like Scala and Rust that have comparatively slow compilers, but that do a lot of safety checks during the 09:10.080 --> 09:16.560 build time. The aforementioned Ada was designed to be a member of that language school too. 09:16.560 --> 09:22.960 So far, 1986 Brooks sounds pretty negative on everything people are trying. Is there anything 09:22.960 --> 09:30.720 he likes the sound of? Yes, there is. OOP is one of them. He's pretty perceptive about some others as 09:30.720 --> 09:39.360 well. All of these are in much wider use today than they were in 1986. Firstly, there's buy versus build. 09:39.360 --> 09:44.560 You can remove complexity from building software by not building software, using an existing 09:44.560 --> 09:50.080 application or component that somebody else built for you. On the one hand, there are way more people 09:50.080 --> 09:55.920 building software today than there were in the 1980s. On the other hand, open source software has 09:55.920 --> 10:01.280 removed the buy from the buy side of the equation, which unlocks even more amounts of reuse. 10:01.280 --> 10:08.160 Additionally, the SaaS market, software as a service, has uncovered additional value in being a component 10:08.160 --> 10:13.360 vendor because you get all of the data about how your customers use the components you sell or 10:13.360 --> 10:20.000 really lease to them. So there's a much stronger component marketplace today, mostly mediated by web 10:20.000 --> 10:26.480 services APIs, in addition to a huge free component reuse scheme, mediated by OSI approved copyright 10:26.480 --> 10:35.520 licenses. His next suggestion, "requirements refinement and rapid prototyping", along with its close cousin, 10:35.520 --> 10:41.600 "incremental development, grow not build software". Both of these saw huge explosions over the last couple 10:41.600 --> 10:46.880 of decades. Both attack essential complexity by comparing your software to the problem it's trying 10:46.880 --> 10:51.680 to solve earlier and more frequently. You get less far before you discover that you're solving the wrong 10:51.680 --> 10:58.400 problem or the wrong snapshot of the right problem. You'd be hard pushed in 2026 to find a software team that 10:58.400 --> 11:06.000 doesn't at least pay lip service to the ideas of iterative and incremental development. Finally, 1986 11:06.000 --> 11:12.480 Brooks sees great promise in training, identifying, rewarding and promoting great designers. He sees this 11:12.480 --> 11:18.160 both resulting in great designs and in a software craftsmanship style mentoring system where upcoming 11:18.160 --> 11:23.200 designers get to learn from the greats and in doing so develop better practices and better taste. 11:23.200 --> 11:30.720 If anything, this is the area that's done least well in the intervening time. Through the first decade 11:30.720 --> 11:35.520 or so of my career, there were some greats that you could expect everyone to know or to have read. 11:36.160 --> 11:41.040 Whether or not you agree with the inclusion of everyone in this list, the work of Steve McConnell, 11:41.040 --> 11:46.480 Kent Beck, Ward Cunningham, Robert Martin, Donald Knuth and Edsger Dijkstra was more or less ubiquitous. 11:47.600 --> 11:52.320 There was a good chance that people knew something about what Barbara Liskov, James Gosling, 11:52.320 --> 11:57.120 Alan Kay, Michael Feathers, Martin Fowler, Bertrand Meyer, Pete McBreen, Richard Stallman, 11:57.120 --> 12:03.040 Eric S. Raymond and indeed Fred Brooks had contributed. In the last 15 years, not so much. 12:03.040 --> 12:07.120 My perspective is that there have been three different factors here, three different shifts. 12:07.680 --> 12:12.880 One is that people tend to be more technology or employer aligned. So people who are working on 12:12.880 --> 12:18.560 back-end stuff or who are working at Google know who Kelsey Hightower is. People who are working on 12:18.560 --> 12:25.120 front-end stuff know Ryan Dahl and Brendan Eich and so on. Another factor is that the algorithmic 12:25.120 --> 12:30.640 nature of social media, along with the sharding along provider lines when we all left Twitter, 12:30.640 --> 12:36.000 means that software engineers don't necessarily see things from the same voices and those voices 12:36.000 --> 12:41.840 need to work harder to build audiences. Thirdly, we've largely drilled into people that they need 12:41.840 --> 12:45.920 to look forward, that the landscape is forever changing and that they might miss out if they don't 12:45.920 --> 12:51.200 stay up to date. So they aren't going to go and read some legacy medium like a book written by 12:51.200 --> 12:57.520 someone who didn't even use Rust. You, my best beloved listeners, are of course the group who know better. 12:58.960 --> 13:03.600 So people aren't really listening to the great designers and indeed the agile pendulum swing 13:03.600 --> 13:08.640 has led many to eschew design completely, believing that any upfront thought about creating software is 13:08.640 --> 13:14.960 the big bad waterfall bogeyman and you should all have Vim open at all times to be a productive software engineer. 13:15.920 --> 13:26.880 This is the advert break. It starts now. Your development team is slowed down, your customers 13:26.880 --> 13:33.440 are frustrated and your support team is constantly banging down the door. Nobody can find the information 13:33.440 --> 13:39.040 they need and they struggle to talk to each other to work it out. Relationships on the team have soured, 13:39.040 --> 13:45.360 with developers, QA and Ops all taking an us versus them position. Your backlog grows faster than the 13:45.360 --> 13:50.320 team delivers features. Retrospectives involve more finger pointing than decision making. 13:51.280 --> 13:57.360 This sounds like a case for Software Craftsmanship's consulting detective. That's me. Discover your 13:57.360 --> 14:03.120 team's friction points and create realistic action plans for improvement. Restructure your delivery pipeline to 14:03.120 --> 14:09.040 bridge the quality gap, automate improvements and reduce rework. Then take ownership of the 14:09.040 --> 14:14.800 documentation and curate a knowledge hub that keeps information flowing throughout your organisation, 14:15.360 --> 14:22.320 empowering engineers, executives, partners and customers alike. Find out more and book a free initial 14:22.320 --> 14:33.440 assessment at sicpers.info slash agency. That's s-i-c-p-e-r-s dot i-n-f-o slash agency. Together with me, 14:33.440 --> 14:37.120 Software Craftsmanship's consulting detective will crack the case. 14:37.120 --> 14:41.200 That was the advert break. It's over now. 14:46.080 --> 14:52.480 Moving on then to 1995 and Brooks wrote a chapter updating NSB and reflecting on its position. 14:52.480 --> 14:57.760 This was nine years into the decade he originally forecast would see no order of magnitude increase 14:57.760 --> 15:02.320 in software productivity. So a good time to check in and see what really had come to pass. 15:02.320 --> 15:09.520 Brooks refers to Brad Cox who wrote two papers; "There is a silver bullet", and "what if there's a 15:09.520 --> 15:15.360 silver bullet and the competition gets it first". Cox's definition of the silver bullet is a particular 15:15.360 --> 15:20.720 view of object-oriented programming that promotes catalogues of reusable components that application 15:20.720 --> 15:27.280 developers select from and integrate to build and use the software. In that way, it marries OOP, 15:27.280 --> 15:33.760 the design paradigm that Brooks thought most likely to yield improvements in 1986, with buy versus 15:33.760 --> 15:39.760 build and reuse, Brooks's own suggestions for promising angles of attack. So really, Cox is saying 15:39.760 --> 15:46.880 that Brooks is right and wording it as if Brooks is wrong. No Silver Bullet refined summarizes a lot of 15:46.880 --> 15:53.280 the other arguments that have been made about the original paper. Steve Lukasik and Lars Sodahl both 15:53.280 --> 15:58.480 point to the facts I raised in the first section of this episode that the essential complexity of 15:58.480 --> 16:04.000 software can itself arise from the accidental complexity of the problem domain, and that an 16:04.000 --> 16:09.680 information theory of organizational structures might help managers simplify their own businesses and 16:09.680 --> 16:16.000 therefore simplify the problems to which they try to apply software. David Harel sees the original 16:16.000 --> 16:21.280 paper as too pessimistic, but at the core of his complaints are the axioms of the paper; the 16:21.280 --> 16:26.000 separation into essential and accidental complexity, and the discussion of whether any one intervention 16:26.000 --> 16:30.400 could lead to a 10x improvement, so it's hard to see his position as anything other than, 16:30.400 --> 16:36.560 "you're wrong". He actually agrees with Brooks that one of the big limitations is software's invisibility, 16:36.560 --> 16:42.400 and the difficulty of accurately describing it, either textually or diagrammatically. But then both 16:42.400 --> 16:47.600 agree that multiple diagrams as thought and design aids are useful, even if the diagrams can never be 16:47.600 --> 16:53.600 complete representations of the software. Users of visual programming environments or executable UML might 16:53.600 --> 17:01.280 disagree there. Capers Jones argued that focusing on productivity is the wrong approach. Focus on 17:01.280 --> 17:07.440 quality and productivity follows. Most of the time a lack of productivity can be ascribed to rework, 17:07.440 --> 17:17.200 so if you get more stuff right, you get more stuff done. In 2025, the DORA report put rework at about 40% 17:17.200 --> 17:24.320 of software engineering time, so we still can't expect an order of magnitude improvement by eradicating rework. 17:25.120 --> 17:30.720 This does remind me though, Capers Jones wrote a technical and social history of software engineering. 17:30.720 --> 17:36.080 I wonder if we can get decade by decade productivity numbers from that. Well, it turns out, kind of. 17:36.640 --> 17:43.200 He gives us the average application size in function points, which is an abstract measurement of amount of 17:43.200 --> 17:49.280 stuff that software does for you, and the average application size in code statements, and reuse percentage. 17:49.280 --> 17:58.960 So, if we make the simplifying and wildly unbelievable assumption that applications have, on average, the same amount of 17:58.960 --> 18:08.800 effort put into them, we can get some idea about productivity. His 1985 figures come out at 1,100 function points per 18:08.800 --> 18:18.960 application with 0-15% reuse, and in 1995, 950 function points per application at 0-25% reuse. 18:18.960 --> 18:26.640 So, it seems, prima facie, that software engineering became less productive during the no-silver-bullet 18:26.640 --> 18:33.760 decade. The average application was smaller and reused about the same amount of software as it did before 18:33.760 --> 18:41.120 in terms of a ratio, so the amount of work being done per application team, which remember we're assuming 18:41.120 --> 18:43.280 is roughly the same, has reduced. 18:45.760 --> 18:52.960 Brooks reflects on the specific predictions in NSB. He gives himself a point for buy-don't-build, not so 18:52.960 --> 18:58.960 much for reusable components, which had yet to see much success by 1995, as for the rise of shrink-wrapped 18:58.960 --> 19:05.440 software. Most people aren't relying on custom applications at all. In 1995, they're buying Microsoft 19:05.440 --> 19:10.640 Works. That's a point that's missing from Capers Jones's analysis. He doesn't measure the 19:10.640 --> 19:15.680 average number of users per application. A nice measurement of software productivity might be 19:16.640 --> 19:20.720 person hours of domain time saved per person hour of development time. 19:22.000 --> 19:29.360 Object-oriented programming, Brooks rates as a brass, not a silver bullet. He says it had the capacity to 19:29.360 --> 19:35.600 be transformative, but wasn't actually applied in transformative ways. This is the idea I explored 19:35.600 --> 19:42.880 in my UIKonf 2015 talk. Nobody knows whether OOP works because nobody's tried it yet. The problem 19:42.880 --> 19:49.280 comes as soon as people decide that they have to be seen to be doing OOP or blockchain or AI or whatever 19:49.280 --> 19:55.680 the current hype train is, so that they aren't seen as being left behind. That's the moment that you 19:55.680 --> 20:02.000 stop doing OOP or blockchain or AI or whatever the current hype train is and just add it onto your 20:02.000 --> 20:07.360 existing process so that everybody can see that you're doing it. So you stop getting completely 20:07.360 --> 20:13.040 new ideas like Smalltalk, and start getting bolt-ons to existing ideas like Java, which is just 20:13.040 --> 20:18.080 Objective-C on a virtual machine, so that people can do whatever top-down structured design techniques 20:18.080 --> 20:25.360 they were previously doing and add a sprinkle of OOP. They add accidental complexity. They don't remove 20:25.360 --> 20:32.560 anything and they certainly don't tackle essential complexity. Brooks concludes that his position in 20:32.560 --> 20:39.520 1995 is unchanged from that in 1986. Of course software has changed a lot since then. Internet 20:39.520 --> 20:45.440 deployed applications, whether through native app stores or web apps; increased levels of virtualization, 20:45.440 --> 20:51.200 containerization, orchestration; distributed team working and computer supported collaborative 20:51.200 --> 20:58.320 working; and of course generative AI. Let me know how you see any of these impacting productivity and 20:58.320 --> 21:04.880 if any might prove to be the folkloric silver bullet. As ever, share your thoughts on this episode in the 21:04.880 --> 21:11.040 comments on the sicpers.info website or email grahamlee at acm.org. Please recommend the podcast to your 21:11.040 --> 21:17.200 friends or review the show at your podcast service. You can find more information at the Patreon page linked in the 21:17.200 --> 21:23.360 show notes and the reading list at se100.chironcodex.com. Let's talk soon.