Structure and Interpretation of Computer Programmers

I make it easier and faster for you to write high-quality software.

Tuesday, September 15, 2020

Self-organising teams

In The Manifesto for Anarchic Software Development I noted that one of the agile manifesto principles is for self-organising teams, and that those tend not to exist in software development. What would a self-organising software team look like?

  1. Management hire a gang and set them a goal, and delegate all decisions on how to run the gang and who is in the gang to members of the gang.
  2. The “team lead” is not in charge of decision-making, but a consultant who can advise gang members on their work. The team lead probably works directly on the gang’s product too, unless the gang is too large.
  3. One member of the gang acts as a go-between for management and communicates openly with the rest of the gang.
  4. Any and all members of the gang are permitted to criticise the gang’s work and the management’s direction.
  5. The lead, the management rep, and the union rep are all posts, not people. The gang can recall any of them at any time and elect a replacement.
  6. Management track the outcomes of the gang, not the “productivity” of individuals.
  7. Management pay performance-related benefits like bonuses to the gang for the gang’s collective output, not to individuals.
  8. The gang hold meetings when they need, and organise their work how they want.

posted by Graham at 08:06  

Saturday, September 12, 2020

The manifesto for anarchic software development

Go on, read the manifesto again. You’ll see that it’s a manifesto for anarchism, for people coming together and contributing equally toward solving problems. From each according to their ability, to each according to their need.

The best architectures, requirements, and designs
emerge from self-organizing teams.

While new to software developers in the beginning of this millennium, this would not have been news to architects who noticed the same thing in 1962. A digression: this was more than a decade before architects made their other big contribution to software engineering, the design pattern. The RIBA report noticed two organisations of teams:

One was characterised by a procedure which began by the invention of a building shape and was followed by a moulding of the client’s needs to fit inside this three-dimensional preconception. The other began with an attempt to understand, fully the needs of the people who were to use the building around which, when they were clarified, the building would be fitted.

There were trade-offs between these styles, but the writers of the RIBA report clearly found some reason “to value individuals and interactions over processes and tools”:

The work takes longer and is often unprofitable to the architect, although the client may end up with a much cheaper building put up more quickly than he had expected. Many offices working in this way had found themselves better suited by a dispersed type of work organisation which can promote an informal atmosphere of free-flowing ideas.

Staff retention was higher in the dispersed culture, even though the self-organising nature of the teams meant that sometimes the senior architect was not the project lead, but found themselves reporting to a junior because ideas trumped length of tenure.

This description of self-organising teams in architecture makes me realise that I haven’t knowingly experienced a self-organising team in software, even when working on a team that claimed self-organisation. The idea is prevalent in software of a “platform shop”: a company that builds Rails websites, or Java micro services, or Swift native apps. This is software’s equivalent of beginning “by the invention of a building shape”, only more so: begin by the application of an existing building shape, no invention required.

As the RIBA report notes, this approach “clearly goes with rather autocratic forms of control”. By centralising the technology in the solution design, people can argue that experience with that technology stack (and more specifically, with the way it’s applied in this organisation) is the measure of success, and use that to impose or reinforce a hierarchy.

Clearly, length of tenure becomes a proxy measure for authority in such an organisation. The longer you’ve been in the company, the more experience you have contorting their one chosen solution to attempt to address a client’s problem. Never mind that there are other skills needed in designing a software product (not least of which is actually understanding the problem), and never mind that this “experience” is in repeated application of an unsuitable template: one year of experience ten times over, rather than ten years of experience.

posted by Graham at 19:42  

Tuesday, April 7, 2020

Stay on target…

I introduce the kind of customer who needs the Labrary’s advice with the following description:

Your software team was a sight to behold, when it started out. You very quickly got to an MVP, validated its fit with early successes, iterated on the user experience and added the missing features. You hired a few more developers to cover the demand.

Now, things are starting to feel slower. The team insists they’re still continuing apace, but you haven’t kept that initial excitement. Developers are grumbling about technical debt. The backlog keeps growing. Testers aren’t keeping up – despite automation. The initial customers aren’t getting the benefits they first expected, and new customers aren’t being won at the rate you’d like.

The problem was that the way you hit it out of the park worked well in the early days, when you had a green field project and no existing code or customers to support. Now your customers expect all new features and surprising and delightful interactions: but they also expect nothing to change, and certainly not to break. The desired qualities of your software have changed, and so the quality of your software must change.

Plenty of people, typically CTOs and heads of software development, typically at growth scale, identify with this description, so it’s worth digging deeper into how it comes about.

At the early stage, your company has a small team, a vague idea of what the product is, and no customers. Literally anything your engineering team can do will be valuable: it will either be a product that fits the market, or tell you where the market isn’t. Obviously there’s some hand-waving about being able to market and sell whatever it is that your engineering team build, but by and large anything you come up with is somehow useful. You are either defining a new market, in which case all work is market-leading, or entering an established market, in which case your direction is clear. It’s hard to make a wrong decision at this stage, but very easy to stick to one.

And while we all know the horror stories about shipping your prototype to production, it’s actually not a bad plan at this stage. You don’t know what will or won’t work, so getting something out there quickly is exactly the right thing to do. And your developers probably have a base standard of maturity even for prototype projects, so you’ll have version control, some form of testing infrastructure and CI, external libraries for data storage, it won’t be a complete wild west.

Things go, loosely speaking, in one of two directions here. If you fail to find the right customers and the right product, you’re out of money, thanks for playing. If you find the right product for the right people, then congratulations! You get more money, either through revenue or a funding round, and you grow the company. Maybe the programmer you were paying before becomes the CTO, maybe one or two of the contractors you worked with come on as perms, and you get a couple of new people come on in return for an OK salary and the promise of the stock sometime being worth something. One of those people is even a QA!

Of course, the cash injection (particularly if it came as a lump through funding that can be drawn down as necessary) gives you the headroom to do things properly. Technologies are chosen, an architecture is designed (usually just by connecting the technologies with arrows), and an attempt is made to build the new thing, support the old thing, and continue adding new features and differentiate in the market. A key customer demographic is sold the promise of the new system (it being exciting and more capable, at least that is what the roadmap says), takes delivery of the old system (it being ready), then takes up time asking for the new features. You either divert resources from the new system to the old to add the features there, or invent some unholy hybrid where your existing thing makes calls to the new thing for the new features with a load of data consistency glue binding the two together. We’ll call these customers “saps” for now. Also, whether you’ve caught up to your competitors or your competitors to them, you’re now having to maintain an edge.

Let’s take stock here. You have:

  1. Some saps, giving us actual money, on the old platform.
  2. Some hope that things will be easier once everything’s on the new platform.
  3. Pressure to stay ahead of/catch up to the competition in both places.

That’s more work! But it’s OK, you’ve got more people. But where you add to the old system (which pleases your saps) you take away from the new, so tend to favour unintrusive patches rather than deeper changes there. Which makes it harder to understand, and harder to support, which is more work! OK, so hire more people! But now engineering costs more. OK, so sell the original thing (not the new thing, it isn’t ready yet) to a few more customers! But now there are more customers, demanding more features, and more support. OK, so hire more people!

Run through that cycle a few times and you end up in the place I described in the Labrary blurb. You’ve got a big team, filled with capable engineers, working hard, and delivering…not as much as anyone would like. The problem is that working on the software is pulling them away from working for the company. The other problem is that you’re measuring how much the software gets worked on.

posted by Graham at 15:31  

Tuesday, November 26, 2019

Everyone rejecting everyone else

It’s common in our cooler-than-Agile, post-Agile community to say that Agile teams who “didn’t get it” eschewed good existing practices in their rush to adopt new ways of thinking. We don’t need UML, we’re Agile! Working software over comprehensive documentation!

This short post is a reminder that it ran both ways, and that people used to the earlier ways of thinking also eschewed integrating their tools into the Agile methodology. We don’t need Agile, we’re Model-Driven! Here’s an excerpt from 2004’s UML 2 Toolkit:

Certain object-oriented methods on the market today, such as The Unified Process, might be considered processes. However, other lightweight methods, such as Extreme Programming (XP), are not robust enough to be considered oricesses in the way we use the term here. Although they provide a valuable set of interrelated techniques, they are often referred to as “small m methodologies” rather than as software-engineering processes.

This is in the context of saying that UML supports software-engineering processes in which a sequence of activities produce “documentation…about the system being built” and “a product that solves the initial problems is introduced and delivered”. So XP is not robust enough at doing those things for UML advocates to advocate UML in the context of XP.

So it’s not just that Agilistas abandoned existing practices, but there’s an extent to which existing practitioners abandoned Agilistas too.

posted by Graham at 10:15  

Monday, August 19, 2019

Experts around the table

One of the principles behind the manifesto for Agile software development says:

Business people and developers must work
together daily throughout the project.

I don’t like this language. It sets up the distinction between “engineering” and “the business”, which is the least helpful language I frequently encounter when working in companies that make software. I probably visibly cringe when I hear “the business doesn’t understand” or “the business wants” or similar phrases, which make it clear that there are two competing teams involved in producing the software.

Neither team will win. “We” (usually the developers, and some/most others who report to the technology office) are trying to get through our backlogs, produce working software, and pay down technical debt. However “the business” get in the way with ridiculous requirements like responding to change, satisfying customers, working within budget, or demonstrating features to prospects.

While I’ve long pushed back on software people using the phrase “the business” (usually just by asking “oh, which business do you work for, then?”) I’ve never really had a replacement. Now I try to say “experts around the table”, leaving out the information about what expertise is required. This is more inclusive (we’re all experts, albeit in different fields, working together on our common goal), and more applicable (in research software engineering, there often is no “the business”). Importantly, it’s also more fluid, our self-organising team can identify lack of expertise in some area and bring in another expert.

posted by Graham at 16:55  

Monday, February 18, 2019

The Fragile Manifesto

A lot of what I’ve been reading and thinking about of late is about the agile backlash. More speed, lower velocity reflects on IT teams pursuing “deliver more/newer IT” at the cost of “help the company achieve its mission”. Grooming the Backfog is about one dysfunction that arises as a result: (mis)managing a never-ending road of small changes rather than looking at the big picture and finding a path toward the destination. Our products are not our products attempts to address this problem by recasting teams not as makers of product, but as solvers of problems.

Here’s the latest: UK wasting £37 billion a year on failed agile IT projects. Some people will say that this is a result of not Agiling enough: if you were all Lean and MVP and whatever you’d not get to waste all of that money. I don’t necessarily agree with that: I think there’s actually things to learn by, y’know, reading the article.

The truth is that, despite the hype, Agile development doesn’t always work in practice.

True enough, but not a helpful statement, because “Agile” now means a lot of different things to different people. If we take it to mean the values, principles and practices written by the people who came up with the term, then I can readily believe that it wouldn’t work in practice for people whose context is different from those who came up with the ideas in 2001. Which may well be everyone.

I’m also very confident that it doesn’t mean that. I met a team recently who said they did “Agile”, and discussed their standups and two-week iterations. They also described how they were considering whether to go from an annual to biannual release.

Almost three quarters (73%) of CIOs think Agile IT has now become an industry in its own right while half (50%) say they now think of Agile as “an IT fad”.

The Agile-Industrial Complex is well-documented. You know what isn’t well-documented? Your software.

The report revealed 44% of Agile IT projects that fail, do so because of a failure to produce enough (or any) documentation.

The survey found that 34% of failed Agile projects failed because of a lack of upfront and ongoing planning. Planning is a casualty of today’s interpretation of the Agile Manifesto[…]

68% of CIOs agree that agile teams require more Architects. From defining strategy, to championing technical requirements (such as performance and security) to ensuring development teams stick to the rules of the game, the role of the Architect is sorely missed in the agile space. It must be reintroduced.

A bit near the top of the front page of the manifesto for agile software development is a sentence fragment that says:

Working software over comprehensive documentation

Before we discuss that fragment, I’d just like to quote the end of the sentence. It’s a long way further down the page, so it’s possible that some readers have missed it.

That is, while there is value in the items on the right, we value the items on the left more.

Refactor -> Inline Reference:

That is, while there is value in comprehensive documentation, we value working software more.

Refactor -> Extract Statement:

There is value in comprehensive documentation.

Now I want to apply the same set of transforms to another of the sentence fragments:

There is value in following a plan.

Nobody ever said don’t have a plan. You should have a plan. You should be willing to amend the plan. I was recently asked what I’d do if I found that my understanding of the “requirements” of a system differ from the customer’s understanding. It depends a lot on context but if there truly is a “the customer” and they want something that I’m not expecting to offer them, it’s time for me to either throw away my version or find a different customer.

Similarly, nobody said don’t have comprehensive documentation. I have been on a very “by-the-book” Agile team, where a developer team lead gave feedback that they couldn’t work out where a change would go to enable a particular feature. That’s architecture! What they wanted was an architectural plan of the system. Except that they couldn’t explicitly want that, because software architecture is so, ugh, 1990s and Rational Rose. Wanting an architecture diagram is like wanting to use CORBA, urrr.

Once you get past that bizarre emotional response, give me a call.

posted by Graham at 16:39  

Powered by WordPress