WEBVTT 00:00.000 --> 00:19.900 Hello, and welcome to episode 67 of the Structure and Interpretation of Computer Programmers 00:19.900 --> 00:25.380 podcast. I'm Graham Lee. The episode is sponsored by you, the software engineering community. 00:25.620 --> 00:29.300 If you enjoy this podcast, please share the link with your friends and colleagues. 00:29.300 --> 00:34.740 You can give feedback on this episode by commenting on the post at sicpers.info/podcast 00:34.740 --> 00:43.460 that's sicpers.info/podcast or by sending me an email grahamlee at acm.org 00:43.460 --> 00:50.340 When I select material for this podcast, I try to pick documents from software engineering's history 00:50.340 --> 00:56.120 that are both interesting and impactful in the field. This usually means older documents for 00:56.120 --> 01:01.040 two reasons. The first is that the field of computing was smaller in the earlier days, 01:01.500 --> 01:07.240 so a higher proportion of people read and built on any material that became famous among professionals. 01:07.880 --> 01:12.340 They then based their work, including their mentorship or training, on that material, 01:12.720 --> 01:18.000 so it became part of the way of working. Nowadays, there are far more people and far less reach for 01:18.000 --> 01:23.700 any one document, so the likelihood that a text achieves industry-wide impact is significantly lower. 01:23.700 --> 01:30.140 The second reason is that during the long period of software's boom, there was a transition away 01:30.140 --> 01:35.280 from reading books about software and from writing books that tried to explain in any deep way 01:35.280 --> 01:41.600 how software worked conceptually. It's time for a bit of a digression, because underlying this move 01:41.600 --> 01:47.160 away from books about software engineering is both a transition to other media and, more importantly, 01:47.160 --> 01:53.400 a radical philosophical shift, a move away from logical positivist epistemology in software engineering 01:53.400 --> 01:59.940 towards an empiricist epistemology. This is a discussion about the metaphysical or philosophical 01:59.940 --> 02:08.200 basis that grounds decisions in the field. Early work in software appears to be informed by the same 02:08.200 --> 02:15.420 kind of logical positivism that underpins a lot of traditional experimental science. Positivists assume that 02:15.420 --> 02:19.980 the universe adheres to rules that govern the evolution of their subject, even if they don't 02:19.980 --> 02:25.080 necessarily know what those rules are. Their scientific goal is to try to uncover the rules 02:25.080 --> 02:30.680 by constructing experiments and defining relationships between the controlled conditions and the observed 02:30.680 --> 02:37.540 outcomes, creating, refining and replacing a model that eventually converges towards a description 02:37.540 --> 02:43.160 or approximation of the rules. So, from positivist software people, you get 02:43.160 --> 02:49.660 denotational semantics and axiomatic semantics, which define programming languages by encoding the rules of 02:49.660 --> 02:55.260 software as they understand them into a language grammar. You get structured programming, which applies the 02:55.260 --> 03:01.180 rules of software to converting problem statements into software. And you get traditional phase-gated project 03:01.180 --> 03:07.340 management, which treats software construction as a series of rule applications and applies each in its turn. 03:07.340 --> 03:14.440 Practitioners of software engineering have adopted different epistemologies with differing degrees of community 03:14.440 --> 03:22.340 uptake. Smalltalk, as developed in Xerox PARC, and Perl, as led by Larry Wall, are both postmodern in approach. 03:22.940 --> 03:28.720 Everyone's reality is different, and the aim of the game is to give people tools they can use to navigate their 03:28.720 --> 03:35.160 realities, not to impose your solutions and your distinct reality on those other people. 03:35.820 --> 03:43.220 Post-Xerox OOP, and as a result UX (that's user experience), and XP (extreme programming), come from a 03:43.220 --> 03:51.400 constructivist epistemology. While there is a shared idea of reality, to constructivists, it's created in a negotiated 03:51.400 --> 03:57.120 space where people come together to try and cope with it. So the important thing to capture isn't the rules of the road. 03:57.120 --> 04:02.040 It's the rules that people make in discussion about the road, and the way their interactions shape 04:02.040 --> 04:08.840 those rules, and the way that the rules they construct constrain those interactions. It's this socially 04:08.840 --> 04:15.180 constructed epistemology that forms the basis of a paper about manufacturing cars and photocopiers that 04:15.180 --> 04:22.820 turns out to be important to software engineering. A paper that isn't the focus of this episode, but forms a good 04:22.820 --> 04:28.880 reminder that I was supposed to be talking about software. Published in the January-February 1986 04:28.880 --> 04:35.100 issue of the Harvard Business Review, the New New Product Development Game describes a constructivist 04:35.100 --> 04:42.220 approach to developing a new product. By the way, you interpret the title as the new game for developing 04:42.220 --> 04:50.640 new products, not the product development game that is doubly new. Takeuchi and Nanaka explain that you can 04:50.640 --> 04:57.080 get better results faster than by using a phased division of labour management approach. Their 04:57.080 --> 05:03.460 alternative to the relay race, which they based on research conducted at NEC, Xerox, Honda, 3M, and 05:03.460 --> 05:09.680 other companies, is a rugby team. A cross-functional group of experts who work collectively on moving the 05:09.680 --> 05:14.340 ball down the field, as it were, given only sponsorship, goals, and moral support by management. 05:14.340 --> 05:21.340 Even though the connections between this paper and the subject of today's episode are clear, 05:21.580 --> 05:25.480 right down to the use of the Scrum metaphor to represent the team coming together to work out 05:25.480 --> 05:30.320 how to move the product forward, the Scrum framework, as described in the Scrum Guide, 05:30.700 --> 05:36.200 comes from a very different place, from a very different philosophical background. I mentioned before 05:36.200 --> 05:42.400 that the software engineering world has moved to an empiricist epistemology. In that worldview, 05:42.400 --> 05:48.460 it doesn't matter overmuch what the rules of the road are, or even if they exist. What works is 05:48.460 --> 05:55.460 whatever you see working. Only experimentation and adaptation to observed results are valid routes 05:55.460 --> 06:02.400 to progress. Clearly, I'm saying that the Scrum framework is empiricist in basis. Well, that's easy. 06:02.500 --> 06:07.600 The Scrum Guide tells us this multiple times. However, I'm going further and saying that almost all 06:07.600 --> 06:13.340 modern methodology in software engineering is informed by empiricism. This comes to Scrum from 06:13.340 --> 06:19.060 Lean Development, and the similarly named Lean Startup takes the same approach, relying on validation 06:19.060 --> 06:26.440 in the marketplace to stand in for product design. So too do design sprints, A/B tests, OKRs, Cynefin, 06:27.080 --> 06:32.540 anywhere that the consultant tells you to follow an OODA (Observe, Orient, Decide, Act) loop, 06:32.540 --> 06:38.860 instead of a PDCA (Plan, Do, Check, Adjust) loop, they're telling you that they don't have a model 06:38.860 --> 06:44.380 for the thing you're trying to do, but they've worked out how to sell FAFO, which stands for 06:44.380 --> 06:50.360 Mess Around and Find Out, as thought leadership. On the surface, this sounds great, because instead 06:50.360 --> 06:54.740 of learning complex models created by scientists or deriving models using ethnographic techniques, 06:55.260 --> 06:58.060 you just mess around and find out with a veneer of professionalism. 06:58.060 --> 07:03.560 Clearly, there are dangers lurking beneath, and they are that the people who do the thing 07:03.560 --> 07:08.240 aren't necessarily the best people to reflect on how the thing was done, and on ways to improve 07:08.240 --> 07:13.440 the thing doing, and also that even if you do improve it, you're only ever going to find a local 07:13.440 --> 07:19.460 maximum, because you're only taking incremental steps away from what you've seen work well in the 07:19.460 --> 07:23.780 past. We'll find more of this as we delve into the Scrum Guide. 07:28.060 --> 07:33.260 This is the advert break. It starts now. 07:33.260 --> 07:38.800 Your development team is slowed down, your customers are frustrated, and your support 07:38.800 --> 07:44.000 team is constantly banging down the door. Nobody can find the information they need, 07:44.440 --> 07:48.500 and they struggle to talk to each other to work it out. Relationships on the team have 07:48.500 --> 07:54.800 soured, with developers, QA, and ops all taking an us-versus-them position. Your backlog grows 07:54.800 --> 08:00.360 faster than the team delivers features. Retrospectives involve more finger-pointing than decision-making. 08:00.360 --> 08:06.180 This sounds like a case for Software Craftsmanship's consulting detective. That's me. 08:07.000 --> 08:10.860 Discover your team's friction points, and create realistic action plans for improvement. 08:11.560 --> 08:16.300 Restructure your delivery pipeline to bridge the quality gap, automate improvements, and 08:16.300 --> 08:22.940 reduce rework. Then, take ownership of the documentation, and curate a knowledge hub that keeps information 08:22.940 --> 08:29.600 flowing throughout your organization, empowering engineers, executives, partners, and customers alike. 08:29.600 --> 08:40.300 Find out more, and book a free initial assessment at sicpers.info slash agency. That's S-I-C-P-E-R-S dot I-N-F-O 08:40.300 --> 08:47.260 slash agency. Together with me, Software Craftsmanship's consulting detective, we'll crack the case. 08:48.220 --> 08:51.240 That was the advert break. It's over now. 08:51.240 --> 09:01.780 The Scrum Guide is far from the first document to describe the Scrum process, but it is the shortest of 09:01.780 --> 09:07.380 those documents. So even given the shift away from reading described above, it's surprising how few 09:07.380 --> 09:13.740 Scrum practitioners have actually read it. There are multiple versions released over the years, which 09:13.740 --> 09:18.760 fundamentally say the same thing, for the most part, but with different ideas in the loud and the 09:18.760 --> 09:26.520 quiet sections. I'm using the first version from 2010, and the most recent version from December 2020. 09:27.620 --> 09:32.820 If you think you're living in the dark Scrum hellscape described by Ron Jeffries, it's worth 09:32.820 --> 09:36.140 checking back with a guide to see if what you're doing is actually Scrum at all. 09:37.660 --> 09:42.720 Both versions of the guide start by explaining the three legs of empirical process control, 09:42.720 --> 09:50.000 transparency, inspection, and adaptation. That is, you have to show what's being done, you have to look 09:50.000 --> 09:55.400 at what's being shown, and you have to adjust what's being done based on what you see. As both 09:55.400 --> 09:59.940 versions of the guide say, the adjustment must be made as quickly as possible to minimise further 09:59.940 --> 10:06.020 deviation. Deferring adaptations, say by filing bug report tickets, or by waiting until the 10:06.020 --> 10:12.260 retrospective, isn't in the spirit of the Scrum framework. The 2020 version then adds five Scrum 10:12.260 --> 10:18.640 values, which underpins successful use of the framework. Commitment, focus, openness, respect, 10:18.960 --> 10:23.660 and courage. It's a bit weird that the values aren't transparency, inspection, and adaptation, 10:24.200 --> 10:29.340 but the ones we're given do make sense. They give off a "criticise the product and the process, 10:29.480 --> 10:34.080 not the people" vibe, and if your culture doesn't promote that, then you can't get the process 10:34.080 --> 10:40.940 improvement aspects of Scrum safely. And remember, Scrum is a process improvement process. It isn't 10:40.940 --> 10:45.800 a product development process. They leave that stuff to whatever your team decides to do in 10:45.800 --> 10:52.060 between the Scrum time boxes. The guide then defines the Scrum team, and here we find some 10:52.060 --> 10:57.440 interesting language that's been removed from the 2010 version, and which was present in earlier 10:57.440 --> 11:03.820 texts about Scrum too. The team are one Product Owner responsible for maximising the product's value, 11:03.820 --> 11:08.620 one Scrum Master responsible for the team's effectiveness and for establishing and maintaining 11:08.620 --> 11:15.720 the Scrum process improvement process, and some number from "7 plus or minus 2" in 2010 to "typically 11:15.720 --> 11:22.400 10 or fewer" in 2020 of developers who are responsible for creating product increments and ensuring their 11:22.400 --> 11:28.160 value. A person can take multiple roles, and there's no expectation that any of the roles is full-time. 11:28.160 --> 11:33.960 Back in the day, these roles were called pigs, while everyone else was called a chicken. 11:35.020 --> 11:39.640 This reference is to the level of commitment in a ham and egg sandwich, and it's supposed to 11:39.640 --> 11:44.600 encourage the team to self-organise and hold each other accountable, and to encourage the people 11:44.600 --> 11:49.880 outside the team to let the team self-organise and hold themselves accountable. In practice, 11:50.000 --> 11:54.580 it was frequently weaponised to discount stakeholder input and avoid transparency, 11:54.580 --> 11:59.860 and it's just a bit offensive. So the pigs and chickens metaphor is no longer used, 12:00.260 --> 12:08.100 in favour of affirming phrases like accountability, respect and service. Besides which, the 2010 guide 12:08.100 --> 12:13.580 is also ambivalent when it comes to considering the Scrum master and product owner as pigs. It seems to do 12:13.580 --> 12:20.640 so on page 4, and to prevaricate over whether they count or not on page 5. Let's quickly go into 12:20.640 --> 12:25.200 the role definitions, because this is another place where alleged Scrum teams differ from the 12:25.200 --> 12:30.040 definition of Scrum. Note that I'm not saying that Scrum is the platonic ideal process improvement in 12:30.040 --> 12:35.620 process, and that deviations are doomed to failure. But I am saying that plenty of teams who think they're 12:35.620 --> 12:40.920 doing Scrum aren't doing Scrum, and also aren't getting the process improvement benefits that Scrum 12:40.920 --> 12:48.460 can potentially yield, and that these two lacks are related, even if not causally. The product owner 12:48.460 --> 12:54.320 maximises the product value by maintaining the product backlog. The product backlog is an ordered 12:54.320 --> 12:59.320 list of things needed to improve the product. Collectively, the items in the product backlog 12:59.320 --> 13:05.440 achieve the product goal, the desired future state of the product. It's weird to suggest that all of 13:05.440 --> 13:11.880 the potentially valuable improvements to a product can be subject to a total ordering, or even that we 13:11.880 --> 13:17.760 can measure what those are without building them all. It wouldn't be weird to notice that a team of up to 13:17.760 --> 13:22.560 10 people has a finite capacity, so they need to know what the most valuable thing they can do next might be. 13:23.240 --> 13:29.180 And therefore, it isn't weird to have most desirable improvements, even if not a totally ordered list of 13:29.180 --> 13:37.700 all potential improvements. But even that short collection of most desirable improvements suffers from one 13:37.700 --> 13:44.300 problem. It's impossible to create. Really, the product owner ought to be responsible for having informed and 13:44.300 --> 13:51.720 defensible opinions on what the team ought to work on next. That phrasing takes us dangerously far from the idea of 13:51.720 --> 13:58.360 empiricism, though. Developers are anybody who contributes to the product increment that the team makes in a 13:58.360 --> 14:05.340 sprint. That's an unfortunate piece of terminology overlap with software roles, where developer usually has a 14:05.340 --> 14:10.920 narrower meaning as a person who makes edits to program source. So if you're trying to apply Scrum to software 14:10.920 --> 14:16.240 development, and remember, it's a process improvement methodology for incrementally building valuable 14:16.240 --> 14:22.200 products, Scrum doesn't actually have anything to say about software. You need to include anybody who 14:22.200 --> 14:28.000 contributes to getting the software into customers' hands as a developer, including, if they contribute to 14:28.000 --> 14:35.560 the product increment, the product owner and the Scrum master. A sprint, by the way, is an iteration of 14:35.560 --> 14:40.800 product development that contains the Scrum events, the planning, review, retrospective, daily Scrums and 14:40.800 --> 14:47.580 the work, and delivers an increment to the product's value. Developers create the sprint backlog for each 14:47.580 --> 14:54.320 sprint, which is the plan for what to do during that iteration. So if the product owner or some external 14:54.320 --> 14:59.860 manager tells the team what the goal is for the sprint or what work to do, that isn't Scrum. There's no reason 14:59.860 --> 15:04.660 that the same task descriptions used in the product backlog need to get carried over into the sprint 15:04.660 --> 15:09.480 backlog, though there is an expectation that the sprint backlog contains work that delivers on the 15:09.480 --> 15:16.060 expectations manifest in the product backlog. Developers also commit to delivering the work in 15:16.060 --> 15:22.120 the sprint backlog to the Definition of Done that they also create, and to adapting the plan every day 15:22.120 --> 15:27.400 toward the sprint goal. That's something that's often missed in the dark Scrum factories. The daily Scrum 15:27.400 --> 15:33.060 is a re-planning session. In Scrum, you change your plan at least once per day. That is, you're allowed 15:33.060 --> 15:38.640 to do it whenever you want. You're expected to do it within the daily Scrum. You don't lock everything 15:38.640 --> 15:42.120 in at the sprint planning session and review whether you achieved it in the sprint review. 15:43.100 --> 15:48.720 If your daily Scrum is what Steven Baker calls a "standus update", where everyone gets a turn at 15:48.720 --> 15:52.820 reporting to their manager what they did yesterday, what they're doing today, and what problems they've 15:52.820 --> 15:58.520 encountered, you're not getting its benefit. At this point, I've shown enough deviations between 15:58.520 --> 16:04.180 Scrum as written and Scrum as practiced that you might think I'm one of those purists who thinks that 16:04.180 --> 16:08.740 if you're not getting the value out of Scrum, it's because you're not doing Scrum hard enough. But no, 16:08.920 --> 16:14.480 I think it's plausible, and maybe even common, to do Scrum exactly as described and still not get the 16:14.480 --> 16:22.040 value out of it. Why? Because Scrum is a framework for empirical process improvement that's in the hands 16:22.040 --> 16:27.260 of the product development experts. That's great for the fuzzy feelings of self-organising teams and 16:27.260 --> 16:32.800 empowering the doers over the managers, sure. But these people didn't get into their status in 16:32.800 --> 16:37.860 their development careers by being good at empirical process improvement. They got there by being good 16:37.860 --> 16:42.400 at demonstrating product development results. On the one hand, that leads to a certain level of 16:42.400 --> 16:47.600 survivor bias. I've got here because my processes are good. Process improvement must mean that everybody 16:47.600 --> 16:53.200 else leaves me alone to do the good processes. On the other hand, it means that both the empirical 16:53.200 --> 16:58.560 and process control parts of empirical process control are skills that the sponsoring organisation 16:58.560 --> 17:03.640 hopes the Scrum team has, but might not have, and doesn't arrange for them to have. 17:04.860 --> 17:09.420 The upshot is that plenty of shops treat the Scrum framework as a development process, 17:09.580 --> 17:14.400 not as a process improvement process. They adopt it once, and then carry on using their baseline 17:14.400 --> 17:19.320 process forever. Not because their scrumming were wrong, but because the bit of Scrum where 17:19.320 --> 17:24.180 your developers are sufficiently metacognitive to run a process improvement process and a 17:24.180 --> 17:30.380 development process in parallel, using the same events for both, is a tacit expectation in the guide. 17:31.480 --> 17:36.900 This brings us on to the last role in the guide, the Scrum Master. That's the person who's supposed to do 17:36.900 --> 17:41.820 the indoctrination, ensuring that the team understands the theory, practice, their responsibilities, 17:41.820 --> 17:46.980 and their commitments within Scrum. But they also act as a kind of ersatz project manager 17:46.980 --> 17:51.420 slash servant leader, enabling customer collaboration, running the events or ensuring 17:51.420 --> 17:56.840 that someone runs them anyway, removing impediments and enabling focus. There can be enough of that 17:56.840 --> 18:02.180 work to do, and enough coaching the Scrum team to follow the process as described, that the Scrum 18:02.180 --> 18:07.600 Master never gets to the other parts. Coaching the team members in self-management and cross-functionality, 18:07.600 --> 18:12.720 helping establish empirical product planning for a complex environment, and leading training and 18:12.760 --> 18:18.880 coaching the organisation in its Scrum adoption. Even where the people on the team understand what 18:18.880 --> 18:24.540 a self-managing team is, that doesn't necessarily mean that the team that they're on is already 18:24.540 --> 18:30.100 self-managing. As Bruce Tuckman says, it takes both time and effort to get a team from forming to 18:30.100 --> 18:36.840 performing, and things get worse before they get better. Finally, there can be organisational factors 18:36.840 --> 18:40.760 that stop Scrum Masters from achieving self-management and cross-functionality, 18:41.120 --> 18:45.760 even when the team's on board. Managers don't like to discover that management isn't required 18:45.760 --> 18:49.960 anymore, for example. And function managers particularly don't like to discover that the 18:49.960 --> 18:53.940 reason management isn't required is that the self-managing team is cross-functional, 18:54.000 --> 18:57.020 so people from outside their fiefdom are trying to muscle in on the work. 18:58.200 --> 19:03.260 I'll sing the verse, and you can join in with the chorus here. If your Scrum team developers 19:03.260 --> 19:07.640 report to the developer manager, your Scrum team QA report to the QA manager, your Scrum 19:07.640 --> 19:11.460 team ops report to the ops manager, your product owner reports to the head of product, 19:11.780 --> 19:16.880 yep, you guessed it, you're not doing Scrum. That's the whole reason DevOps came about. 19:17.280 --> 19:21.280 We were already meant to be doing cross-functional self-organising teams working towards common goals, 19:21.540 --> 19:26.620 but nobody was doing that. Of course, the upshot of DevOps ended up being that everyone hired 19:26.620 --> 19:32.000 into a DevOps functional role, and made those people report to their head of DevOps. Yet another 19:32.000 --> 19:38.840 functional distinction, which was the opposite of the point. The same happened with Scrum. Scrum is 19:38.840 --> 19:45.000 in itself a good framework for empirical process control, but it's sold as a software development 19:45.000 --> 19:50.660 methodology to software development managers, even though there isn't a software development 19:50.660 --> 19:55.000 methodology in Scrum, and it requires an environment without software development managers. 19:55.740 --> 20:00.440 That's why you hear so much about failed adoptions and dark Scrum anti-patterns. 20:00.440 --> 20:05.460 As ever, share your thoughts on this episode in the comments on the SICPERS.info website, 20:05.540 --> 20:12.040 that's S-I-C-P-E-R-S.info, or email grahamlee at acm.org. Please recommend the podcast to your 20:12.040 --> 20:16.760 friends or review the show at your podcast service. You can find more info at the Patreon page linked 20:16.760 --> 20:23.160 in the show notes and the reading list at se100.chironcodex.com. Let's talk soon.