A question of focus

The problem with The Labrary is that I offer to do so many things – because I could do them, and do them well – that it can be hard to find the one thing I could do for you that would be most helpful:

  • Artificial Intelligence
  • Agile Development
  • Continuous Delivery
  • Software Architecture
  • Technical Writing
  • Developer Experience
  • Programmer Mentoring

Each of these supports the mission of “making it faster and easier to make high-quality software that respects privacy and freedom”, but all of them is overwhelming. I have credentials/experience to back up each of them, but probably don’t have the reputation as a general expert that someone like Dan North or Liz Keogh can use to have people ask me anything.

So I want to pick one. One thing, probably from that list, and pivot to focus on that. Or at least get in through the door that way, then have the conversations about the other things once you know how much faster and easier I make it for you to make high-quality software.

And I’d really value your suggestions. Which one thing do you know me for, above all others? Which one thing is the pain that the place you work, or places you’ve worked, most need fixing?

Comment here, have a chat, send an email. Thanks for helping me find out what I want to be when I grow up.

Solving the underlying problem

As a software engineer, it’s easy to get work engineering software. Well, maybe not easy, but relatively so: that is the kind of work that comes along most. The kind of work that people are confident I can do. That they can’t do, so would like me to do for money.

It’s also usually the worst work available.

I don’t want to take your shopping list of features, give you a date and a cost, then make those features. Neither of us will be very happy, even if it goes well.

I want to get an understanding of your problem, and demonstrate how software can help in solving it. Maybe what we need to understand isn’t the problem you presented, but the worse problem that wasn’t on your mind. Or the opportunity that’s worth more than a solution to either problem.

Perhaps we ask a question, in solving your problem, to which the answer is that we don’t know, and now we have another problem.

You might not need me to build all of the features you thought of, just one of them. Perhaps that one works better if we don’t build it, but configure something that already exists. Or make it out of paper.

You understand your problem and its domain very well. I understand software very well. Let’s work together on combining that expertise, and both be happier in the process.

Maybe you are going to need it

In the beginning, there was the green field. The lead developer, who may have been the only developer, agreed with the product owner (or “the other member of the company” as they were known) what they would build for the first two weeks. Then File->New Project… happened, and they smashed it out of the park.

The amorphous and capricious “market” liked what they had to offer, at least enough to win some seed funding. The team grew, and kept the same cadence: see what we need to do for the next ten business days, do it, celebrate that we did it.

As the company, its customers, and its market mature, things start to slow down. It’s imperceptible at first, because velocity stays constant. The CTO can’t help but think that they get a lot less out of a 13-point story than they used to, but that isn’t a discussion they’re allowed to have. If you convert points into time then you’re doing old waterfall thinking, and we’re an agile team.

Initially the dysfunction manifests in other ways. Developers complain that they don’t get time to refactor, because “the business” doesn’t understand the benefits of clean code. Eventually time is carved out to clean things up, whether in “hardening sprints” or in effort allocated to “engineering stories”. We are getting as much done, as long as you ignore that less of it is being done for the customers.

Stories become task-sliced. Yes, it’s just adding a button, but we need to estimate the adding a component task, the binding the action task, the extending the reducer task, the analytics and management intelligence task. Yes we are getting as much done, as long as you ignore that less of it has observable outcomes.

Rework increases too, as the easy way to fit a feature into the code isn’t the way that customers want to use it. Once again, “the business” is at fault for not being clear about what they need. Customers who were previously flagship wins are now talked about as regressive laggards who don’t share the vision. Stories must have clearer acceptance criteria, the definition of done must be more explicit: but obviously we aren’t talking about a specification document because we’re an agile team. Yes we’re getting as much done, as long as you ignore that a lot of what we got done this fortnight was what we said we’d done last fortnight.

Eventually forward progress becomes near zero. It becomes hard to add new features, indeed hard even to keep up with the competitors. It’s only two years ago that we were five years ahead of them. People start demoing new ideas in separate apps, because there’s no point dreaming about adding them to our flagship project. File->New Project… and start all over again.

What happened to this team? Or really, to these teams, as I’ve seen this story repeated over and over. They misread “responding to change over following a plan” as “we don’t need no stinking plan”.

Even if you don’t know exactly where you are going at any time, you have a good idea where you think you’re going. It might be spread around the company, which is why we need the experts around the table. Some examples of where to find this information:

  • The product owner has a backlog of requested features that have yet to be built.
  • The sales team have a CRM indicating which prospects are hottest, and what they need to offer to close those deals.
  • The marketing director has a roadmap slide they’re presenting at a conference next month.
  • The CTO has budget projections for the next financial year, including headcount changes and how they plan to reorganise the team to incorporate these changes.
  • The CEO knows where they want to position the company in the market over the next two years, and knows which competitors, regulatory changes, and customer behaviours threaten that position and what about them makes them a threat.
  • Countless spreadsheets, databases, and “business intelligence” dashboards across multiple people and departments.

No, we don’t know the future, but we do know which futures are likely and of those, which are desirable. Part of embracing change is to make those futures easier to cope with. The failure mode of many teams is to ignore all futures because we aren’t in any of them yet.
We should be ready for the future we expect, and both humble and adaptable enough to get ready for a different future when things change. Our software should represent our current knowledge of our problem and its solution, including knowledge about likely developments (hey, maybe there’s a reason they call us developers!). Don’t add the things you aren’t going to need, but don’t exclude the possibility of adding them out of spite for a future that may well come to pass.

Experts around the table

One of the principles behind the manifesto for Agile software development says:

Business people and developers must work
together daily throughout the project.

I don’t like this language. It sets up the distinction between “engineering” and “the business”, which is the least helpful language I frequently encounter when working in companies that make software. I probably visibly cringe when I hear “the business doesn’t understand” or “the business wants” or similar phrases, which make it clear that there are two competing teams involved in producing the software.

Neither team will win. “We” (usually the developers, and some/most others who report to the technology office) are trying to get through our backlogs, produce working software, and pay down technical debt. However “the business” get in the way with ridiculous requirements like responding to change, satisfying customers, working within budget, or demonstrating features to prospects.

While I’ve long pushed back on software people using the phrase “the business” (usually just by asking “oh, which business do you work for, then?”) I’ve never really had a replacement. Now I try to say “experts around the table”, leaving out the information about what expertise is required. This is more inclusive (we’re all experts, albeit in different fields, working together on our common goal), and more applicable (in research software engineering, there often is no “the business”). Importantly, it’s also more fluid, our self-organising team can identify lack of expertise in some area and bring in another expert.

On the efficient allocation of scarce resources with alternative uses

Most of what I know about “the economy” is outdated (Adam Smith, Karl Marx, John Maynard Keynes) or incorrect (the news) so I decided to read a textbook. Basic Economics, 5th Edition by Thomas Sowell is clear, modern, and generally an argument against economic regulation, particularly centralised planning, tariffs, and price control. I still have questions.

The premise of market economics is that a free market efficiently uses prices to allocate scarce resources that have alternative uses, resulting in improved standard of living. But when results are compared, they are given in terms of economic metrics, like unemployment, growth, or GDP/GNP. The implication is that more consuming is correlated with a better standard of living. Is that true? Are there non-economic measurements of standard of living, and do they correlate with the economic measurements?

Even if an economy does yield “a better standard of living”, shouldn’t the spread of living standards and the accessibility of high standards across the population be measured, to determine whether the market economy is benefiting all participants or emulating feudalism?

Does Dr. Sowell arrive at his office at 9am and depart at 5pm? The common 40-hour work week is a result of labour unions and legislation, not supply and demand economics. Should we not be free to set our own working hours? Related: is “unemployment” such a bad thing, do we really need everybody to work their forty hours? If it is a bad thing, why not reduce the working week and have the same work done by more people?

Sowell’s argument allows that some expenses, notably defence, are better paid for centrally and collectively than individually. We all get the same benefit from national defence, but even those who are willing to pay would receive less benefit from a decentralised, individually-funded defence. Presumably the same argument can be applied to roads, too, or space races. But where are the boundaries? Why centralised military, say, and not centralised electricity supply, healthcare, mains water, housing, internet service, or food supply? Is there a good “grain size” for such centralising influences (it can’t be “the nation”, because nations vary so much in size and in centralisation/federation) and if so, does it match the “grain size” for a market economy?

The argument against a centralised, planned economy is that there’s too much information required too readily for central planners to make good judgements. Most attempts at a planned economy preceded broad access to the internet and AI, two technologies largely developed through centralised government funding. For example, the attempt to build a planned economy in Chile got as far as constructing a nationwide Telex network before being interrupted by the CIA-funded Pinochet coup. Is this argument still valid?

Companies themselves are centralised, planned economies that allocate scarce resources through a top-down bureaucracy. How big does a company need to get before it is not the market, but the company’s bureaucracy, that is the successful system for allocating resources?

My first rails app

I know, right? I first learned how to rails back when Rails 3 was new, but didn’t end up using it (the backend of the project I was working on was indeed written in Rails, but by other people). Then when I worked at Big Nerd Ranch I picked up bits and pieces of knowledge from the former Highgroove folks, but again didn’t use it. The last time I worked on a real web app for real people, it was in node.js (and that was only really vending a React SPA, so it was really in React). The time before that: WebObjects.

The context of this project is that I had a few days to ninja out an end-to-end concept of a web application that’s going to be taken on by other members of my team to flesh out, so it had to be quick to write and easy to understand. My thought was that Rails is stable and trusted enough that however I write the app, with roughly no experience, would not diverge far from however anyone else with roughly no experience would do it, so there wouldn’t be too many surprises. That the testing story for Rails is solid, that websites in Rails are a well-understood problem.

Obviously I could’ve chosen any of a plethora of technologies and made my colleagues live with the choice, but that would potentially have sunk the project. Going overly hipster with BCHS, Seaside or Phoenix would have been enjoyable but left my team-mates with a much bigger challenge than “learn another C-like OOP language and the particular conventions of this three-tier framework”. Similarly, on the front end, I just wrote some raw JS that’s served by Rails’s asset pipeline, with no frameworks (though I did use Rails.ajax for async requests).

With a day and a half left, I’m done, and can land some bonus features to reduce the workload for my colleagues. Ruby is a joy to use, although it is starting to show some of the same warts that JS suffers from: compare the two ways to make a Ruby hash with the two ways to write JS functions. The inconsistency over brackets around message sends is annoying, too, but livable.

Weirdly testing in Rails seems to only be good for testing Ruby, not JS/Coffeescript/whatever you shove down the frontend. I ended up using the teaspoon gem to run Javascript tests using Jasmine, but it felt weird having to set all that up myself when Rails goes out of its way to make tests for you in Ruby-land. Yes, Rails is in Ruby. But Rails is a web framework, and JS is a necessary evil on the web.

Most of my other problems came from the incompatibility of Ruby versions (I quickly gave up on rvm and used Docker, writing a small wrapper script to run the CD pipeline and give other devs commands like ‘build’, ‘test’, ‘run’, ‘stop’, ‘migrate’) and the changes in Rails API between versions 3-5. A lot of content on blogs[*] and stackoverflow don’t specify the version of Rails or Ruby they’re talking about, so the recommendations may not work the same way.

[*] I found a lot of Rails blogs that just reiterate examples and usage of API that’s already present in the rdoc. I don’t know whether this is SEO poisoning, or people not knowing that the official documentation exists, or there being lots of low-quality blogs.

But overall, Railsing was fun and got me quickly to my destination.

My changing relationship with books

My Delicious Library collection just hit 1,000 books. That’s not so big, it’s only a fraction of the books I’ve read in my life. I only started cataloguing my books a few years ago.

What is alarming about that is that most of the books are in my house, and most are in physical form. I read a lot, and the majority of the time I’m reading something I own. The reason it’s worrying is that these books take up a lot of space, and cost a lot of money.

I’ve had an on-again, off-again relationship with ebooks. Of course they take up less space, and are more convenient when travelling. The problems with DRM and ownership mean that I tend to only use ebooks now for books from Project Gutenberg or the internet archive, and PDFs of scholarly papers.

And not even that second one, due to the lack of big enough readers. For a long time I owned and enjoyed a Kindle DX, with a screen big enough that a typical magazine page was legible without zooming in. Zooming in on a columnar page is horrific. It’s like watching a tennis match through a keyhole. But the Kindle DX broke, is no longer a thing, and has no competitors. I don’t enjoy reading on regular computer screens, so the option of using a multipurpose tablet is not a good one.

Ebooks also suffer from being out of sight and out of mind. I actually bought some bundle of UX/HCI/design books over a year ago, and have never read them. When I want to read, I look at my pile of unread books and my shelves. I don’t look in ~/Documents/ebooks.

I do listen to audiobooks when I commute, but only when I commute. It’d be nice to have some kind of multimodal reader, across a “printed” and “spoken” format. The Kindle text-to-speech was not that, when I tried it. Jeremy Northam does a much better job of reading out The Road to Wigan Pier than an automated speech synthesiser does.

The technique I’m trying at the moment involves heavy use of the library. I’m a member of both the local municipal library and a big university library. I subscribe to a literary review magazine, the London Review of Books. When an article in there intrigues me, I add the book to the reading list in the library app. When I get to it, I request the book.

That’s not necessarily earth-shattering news. Both public and subscription libraries have existed for centuries. What’s interesting is that for this dedicated reader and technology professional, the digital revolution has yet to usurp the library and its collection of bound books.

The challenges of teaching software engineering

I’ve just finished teaching a four-day course introducing software engineering for the first time. My plan is to refine the course (I’m teaching it again in October), and it will eventually become the basis for doctoral training programmes in research software engineering at Oxford, and part of a taught Masters. My department already has an M.Sc. in Software Engineering for commercial engineers (in fact I have that degree), and we want to do the same for software engineers in research context.

Of course, I can also teach your team about software engineering!

Some challenges that came up:

  • I’m too comfortable with the command-line to get people past the initial unfamiliar discomfort. From that perspective, command-line tools are all unusably hard. I’ve learnt from various sources to try foo --help, man foo, and other incantations. Others haven’t.

  • git, in particular, is decidedly unfriendly. What I want to do is commit my changes. What I have to do is stage my changes, then commit my staged changes. As a result, teaching git use takes a significant chunk of the available time, and still leaves confusion.

  • you need to either tell people how to set their core.editor, or how to quit vim.

  • similarly, there’s a world of difference between python foo.py and python3 foo.py, and students aren’t going to interpret the sorts of errors you et if you choose the wrong one.

  • Introduce a tangent, and I run the risk of losing people to that tangent. I briefly mentioned UML while discussing diagrams of objects, as a particular syntax for those diagrams. In the subsequent lab, some people put significant time into making sure their diagrams were valid UML.

  • Finding the trade-off between presentation, tutorial, and self-directed exercise is difficult. I’m used to presentations and will happily talk on many topics, but even I get bored of listening to me after the ~50% of the time I’ve spent speaking on this course. It must be worse for the students. And there’s no substitute for practical experience, but that must be supported by guidance.

  • There are so many topics that I didn’t get to cover!

    • only having an hour for OOP is a sin
    • which means I didn’t even mention patterns or principles
    • similarly, other design techniques like functional programming got left off
    • principles like Agile Software Development, Software Craftsmanship, or Devops don’t get a mention
    • continuous integration and continuous delivery got left off. Even if they didn’t, the amount of work involved in going from “I have a Python script” to “I run my tests whenever I change my script, and update my PYpi package whenever they pass” is too damn high.
    • forget databases, web servers, browsers, mobile apps, desktop apps, IoT, or anything that isn’t a command line script or a jupyter notebook
    • and machine learning tools
    • and concurrency, processes and process improvement, risk management, security, team dynamics, user experience, accessibility…

It’s only supposed to be a taster but I have to trade off introducing everything with showing the value present in anything. What this shows, as I found when I wrote APPropriate Behaviour, is that there’s a load that goes into being a programmer that is not programming.

Falsehoods programmers who write “falsehoods programmers believe” articles believe about programmers who read “falsehoods programmers believe” articles

For reasons that will become clear, I can’t structure this article as a “falsehoods programmers believe” article, much as that would add to the effect.

There are plenty of such articles in the world, so turn to your favourite search engine, type in “falsehoods programmers believe”, and orient yourself to this concept. You’ll see plenty of articles that list statements that challenge assumptions about a particular problem domain. Some of them list counterexamples, and a subset of those give suggestions of ways to account for the counterexamples.

As the sort of programmer who writes falsehoods programmers believe articles, my belief is that interesting challenges to my beliefs will trigger some curiosity, and lead me to research the counterexamples and solutions. Or at least, to file away the fact that counterexamples exist until I need it, or am otherwise more motivated to learn about it.

But that motivation is not universal. The fact that I treat it as universal turns it into a falsehood I believe about readers of falsehoods articles. Complaints abound that falsehoods articles do not lead directly to fish on the plate. Some readers want a clear breakdown from “thing you might think is true but isn’t true” to “Javascript you can paste in your project to account for it not being true”. These people are not well-served by falsehoods articles.

Longer, fuller stacks

Thinks to self: OK, this “full-stack” project is going to be fairly complex. I need:

  • a database. I don’t need it yet, I’ll defer that.
  • a thing that runs on the server, listens for HTTP requests from a browser, builds responses, and sends them to the browser.
  • a thing that runs on the browser, built out of bits assembled by the server thing, that the user can interact with.

What I actually got was:

  • a thing that runs on the server.
  • a thing that defines the environment for the server.
  • a thing that defines the environment on development machines so that you can run server-development tasks.
  • a thing that turns code that can’t run in a browser into code that can run in a browser.
  • a thing that turns code that can run in a browser into code that does run in real browsers.
  • a headless browser that lets me test browser code.
    • BTW, it doesn’t work with that server environment.
    • a thing that shows how Linux binaries are loaded, to work out how to fix the environment.
    • also BTW, it doesn’t run headless without setting some environment variable
    • a thing that is used for cross-platform native desktop apps, that I can configure to work headless.
  • a thing that builds the bits assembled by the server thing so that the test thing can see the code being tested.

And somehow, people argue that this is about reducing development costs.