Even more on generalist software engineering

There is a difference between a generalist software engineer, and a polyglot programmer. What is that difference, and why did I smoosh the two together in yesterday’s post?

A polyglot programmer is a programmer who can use, or maybe has used, multiple programming languages. That doesn’t mean that they have relevant experience or skills in any other part of software engineering. They might, and so might a monoglot programmer. But (in a work context) they’re working as a programmer and paid for doing the programming.

A generalist software engineer has knowledge and experience relevant to the rest of the software engineering endeavour. Not just the things that programmers need to know beyond the programming, but the whole practice.

In my experience, a generalist software engineer is usually also a polyglot programmer, for a couple of reasons.

  • to get enough experience at all the bits of software engineering, the generalist’s career has probably survived multiple programming language hype cycles.
  • in getting that experience, they have probably had to work with different languages in different contexts: whatever their software products are made out of; scripts for deployment and operations; plugins for various tools; changes to existing components.

More importantly, the programming part of the process is a small (crucial, admittedly, small nonetheless) intersection with the generalist’s work. Knowing all of the gotchas of any one language is something they can ask the specialists about. Fred Brooks would have had a Chief Programmer who knew what the team was trying to do, and a language lawyer who specialised in the tricks of the programming language.

Probably the closest thing that many organisations know how to pay for to the generalist software engineer in modern times is the “agile consultant”. I don’t think it’s strictly a fair comparison. Strictly speaking, an agile consultant is something like a process expert, systems analyst, or cybernetician. They understand how the various parts of the software engineering endeavour affect each other, and can help a team to improve its overall effectiveness by knowing which part needs to change and how.

And, because they use the word agile in their title, they do this with an air of team empowerment and focus on delivery.

Knowing how the threat model impacts on automated testing enables process improvement, but does not necessarily make a software engineering generalist. An actually-general generalist could perhaps also do the threat modelling and create the automated tests. To avoid drawing distinctions between genuine and inauthentic inhabitants of Scotland let’s say that a generalist can work in ‘several’ parts of the software engineering field.

We come back to the problem described in yesterday’s post, but without yesterday’s programming-centric perspective. A manager knows that they need a security person and a test automation person, but does not necessarily know how to deal with a security-and-test-automation-and-maybe-other-things person.

This topic of moving from a narrow focus on programming to a broader understanding of software engineering is the subject of my newsletter: please consider signing up if this topic interests you.

Posted in software-engineering | 1 Comment

On interviewing and generalist software engineers

After publishing podcast Episode 53: Specialism versus generality, Alan Francis raised a good point:

This could be very timely as I ponder my life as a generalist who has struggled when asked to fit in a neat box career wise.

https://twitter.com/PossiblyAlan/status/1523755064879632384

I had a note about hiring in my outline for the episode, and for some reason didn’t record a segment on it. As it came up, I’ll revisit that point.

It’s much easier to get a job as a specialist software engineer than as a generalist. I don’t think that’s because more people need specialists than need generalists, though I do think that people need more specialists than they need generalists.

For a start, it’s a lot easier to construct an interview for a specialist. Have you got any experience with this specialism? What have you done with it? Do you know the answers to these in-group trick questions?

Generalists won’t do well at that kind of question. Why bother remembering the answer to a trick question about some specific technology, when you know how to research trick answers about many technologies? But the interviewer hears “doesn’t even know the first trick answer” and wonders how do I know you can deliver a single pint of software on our stack if you can’t answer a question set by a junior our-stack-ist?

If you want to hire a generalist software engineer…ah. Yes. I think that maybe some people don’t want to, whether or not they know what the generalist would do. They seem to think it’s a “plural specialist”, and that a generalist would know all the trick questions from multiple specialisms.

This is the same thinking that yields “a senior developer is like a junior developer but faster”; it is born of trying to apply Taylorian management science to knowledge work: a junior Typescript programmer can sling a bushel of Typescript in a day. Therefore a senior Typescript programmer can sling ten gallons, and a generalist programmer can sling one peck of Typescript, two of Swift, and one of Ruby in the same time.

I think that the hiring managers who count contributions by the bushel are the ones who see software engineering as a solitary activity. “The frontend folks aren’t keeping pace with the backend folks, so let’s add another frontend dev and we’ll have four issues in progress instead of three.” I have written about the flawed logic behind one person per task before.

A generalist may be of the “I have solved problems using computers before, and can use computers to solve your problem” kind, in which case I might set aside an hour to pair with them on solving a problem. It would be interesting to learn both how they solve it and how they communicate and collaborate while doing so.

Or they may be of the “I will take everybody on the team out to lunch, then the team will become better” kind. In which case I would invite them to guest-facilitate a team ceremony, be it an architecture discussion, a retrospective, or something else, and see how they uncover problems and enable solutions.

In each case the key is the collaboration. A software engineering generalist might understand and get involved with the whole process using multiple technologies, but that does not mean that they do all of the work in isolation themselves. You don’t replace your whole team, but you do (hopefully) improve the cohesiveness of the whole team’s activity.

Of course, a generalist shouldn’t worry about trying to get hired despite having to sneak past the flawed reasoning of the bushel of trick questions interview. They should recognise that the flawed reasoning means that they won’t work well with that particular manager, and look elsewhere.

Posted in software-engineering | 3 Comments

Episode 53: Specialism versus generality

I look at the difference between being a deep specialist as a software engineer working on a particular “stack” and a generalist who builds software using a wide variety of tools, from the perspective of someone who has done both.

Leave a comment

You say “cave dweller debugging”, I say debug logging

There are still many situations where it’s not feasible to stop a process, attach the debugger, and start futzing with memory. We can argue over whether this is because the industry didn’t learn enough from the Pharo folks later. For now, let’s pretend that’s axiomatic: a certain amount of debugging (and even testing) starts with asking the program to report on its state.

It’s common to do this with print statements, whatever they’re called in your language. The technique is even called “printf debugging” sometimes. Some teams even have lint rules to stop you merging changes that call System.out.println or console.error because they know you’re all doing it. I think you should carry on doing it, and you should be encouraged to commit the changes.

Just don’t call your print function. This isn’t for any performance/timing/buffer flushing reason, though those are sometimes relevant and in the relevant cases sometimes problematic and in the problematic cases sometimes important. It’s more because it’s overwhelming. Instead, call your syslog/OSLog/logger function, with an appropriate severity level (probably DEBUG) and some easily filterable preamble. Now commit those log messages.

One benefit of doing this is that you capture the fact that you need this information to diagnose problems in this module/subsystem/class/whatever. Next time you have the problem, you already know at least some of the information that will help.

Another benefit is that you can enable this logging in the field without having to deploy a new version with different printf statements. Just change the log level or the capture filter and start getting useful information. Change it back when you’re done.

There are caveats here: if the information you need to log is potentially sensitive (personal information, crypto material) you may be better off not having any way to turn it on in production.

The third benefit is that you communicate to everybody else that this part of the code is hard to understand and needed careful inspection. This can be the motivation the team needs to discuss a redesign, or it can help other people find smoking guns when trying to diagnose other failures.

Posted in code-level | 1 Comment

The Requirements Trifecta

It’s hard to argue that any one approach to, well, anything in software is better or worse than any others, because very few people are collecting any data and even fewer are reporting what they’re trying. Worst is understanding how requirements are understood, prioritised, and implemented. Companies can be very opaque when it comes to deciding what they do and when they do it, even if you’re on the inside.

This can be frustrating: if you can see what you think is the path between here and all the money, then it can be enervating to find the rest of the organisation is following a different path. Doubly so if you can’t see, and aren’t shown, any reason to follow that path.

What we do know is that the same things that work in small companies don’t tend to work in large ones: search for any company name X in the phrase “why doesn’t X listen to customer feedback” and you’ll find multiple complaints. Customers get proxied, or aggregated, or weighed against potential customers. We feel like they aren’t listening to us, and it’s because they aren’t. Not to us as individuals anyway. If a hundred thousand of us are experiencing kernel panics or two million of us have stopped engaging with feed ads, something will get done about it.

That said, there are some things I’ve seen at every scale that need to be in balance for an organisation to be pointing in the right direction. Three properties of how they work out what they’re doing, that need to be in synergy or at least balanced against one another. I now ask about these three things when I meet with a new (to me) company, and use these as a sense check for how I think the engagement is going to work out.

Leadership Vision

One third is that the company has some internal driving force telling it which way to go and what challenges to try to solve. Without this, it will make do with quick wins and whatever looks easy, potentially even straying a long way from its original mission as it takes the cheapest step in whatever direction looks profitable. That can lead to dissatisfied employees, who joined the company to change the world and find they aren’t working on what was promised.

On the other hand, too much focus on the vision can lead to not taking material reality into account. A company that goes bust because “our product was ten years ahead of its time, the customers weren’t ready” is deluding itself: it went bust because the thing they wanted to sell was not something enough other people wanted to buy.

Market Feedback

One third is that the company has some external input telling it what customers are trying to do, what problems people have, what people are willing to pay for those problems to go away, and what competitors are doing to address those problems. Without this, the company will make things that people don’t want to buy, lose out on sales opportunities where they don’t describe what people have in a way that makes them want it, or will find themselves outcompeted and losing to alternative vendors.

On the other hand, too much focus on market feedback can lead to a permanently unsatisfying death march. Sales folks will be sure that they can close the next big deal if only they had this one other feature, and always be one other feature from closing that deal. Customers can always find something to be unhappy about if sufficiently prodded, so there will always be more to do.

Technical Reality

The third third is that the company has some internal feedback mechanism telling it what is feasible with the assets it already has, what the costs and risks (including opportunity costs) are of going in any direction, and what work it needs to do now to enable things that are planned in the future. Without this, forward progress collapses under the weight of what is erroneously called technical debt. Nothing can be done, because doing anything takes far too long.

On the other hand, too much focus on technical desiderata can lead to the permanent rewrite. Technical folks always have a great idea for how to clean things up, and even if all of their suggestions are good ones there comes a time when a lumberjack has to accept that the next action is not to replace their axe sharpeners but is to cut down a tree. Features are delayed until the Next Great Replatforming is complete, which never comes because the Subsequent Replatforming After That One gets greenly.

The Trifecta

I don’t think it’s particularly MBA material to say that “a company should have a clear leadership vision moderated by marketing reality and internal capability”. But in software I see the three out of kilter often enough that I think it’s worth writing the need down in the hope that some people read it.

Posted in software-engineering | Leave a comment

On or Between

The new way to model concurrency is with coroutines (1963), i.e. the async/await dance or (building upon) call-with-concurrent-continuation. The new new way is with actors (1973), and the old and busted ways are with threads (1966), and promises (1976).

As implemented in many programming languages, these ideas are on a piece of logic, making the concurrency model an existential part of the software model. Would we say that a payroll budget is a serial queue, or that awarding a pay rise is a coroutine? We probably would not, but our software tools make us do so.

This is not necessary. For example, in Eiffel’s SCOOP (previously discussed on this very blog) the change that is made to the model is that objects indicate when their collaborators can have separate execution contexts. They do not need to: turn off the parallel execution engine and everything runs sequentially. Otherwise, the details of how, and if, objects are running in different contexts are not part of the solution model.

Various other solutions rely on the call as a binding point in software execution, to allow the asynchronous nature to be injected as a connector between parts of the solution model. So in Objective-C or Smalltalk you might inject futures or queue confinement via proxy objects. Then your solution model hasn’t got any noise about concurrent execution at all, but your implementation model still gets to choose where code gets run when one part of the solution sends a message to another part.

What’s the difference? To me it’s one of focus: if I want a clearly correct implementation of my solution model that might be an efficient program, then I would choose the connector approach: putting implementation details between my solution elements. If I want a clearly efficient program that might be a correct implementation of my solution model, then I would choose the annotation approach: putting implementation details on my solution elements. In my experience more of the software I’ve worked on has worked incorrectly than worked poorly, and when it’s worked poorly I haven’t wanted to change its correctness, so I would prefer to separate the two.

None of this is a complaint about any tools, or any libraries: you can build connectors when your tools give you annotations, by putting the annotations on things between your solution models. You can build annotations when your tools give you connectors, by hard-coding the use of connectors in the solution implementations. It’s simply a question of what your design makes obvious.

Posted in architecture of sorts, code-level, design, performance, software-engineering | Leave a comment

WWDC 2022 is a WWDC watch party

Apple have shared initial timings for this year’s WorldWide Developer Conference. In typical in-person years this would be the trigger for various “WWDC attendee tips” posts (don’t forget to drink water! Remember to sleep sometime through the week! Don’t go to the Moscone centre, they’ve moved the conference!) but that has not been the case through the pandemic. Instead WWDC has been fully online, so you just need to get the Developer app and watch the videos.

This year, it’s sort of hybrid, in that it appears the event will be online-first with a watch party of sorts on the first day. This happened at the fully in-person events anyway, at least at the Moscone: the keynote room filled up quickly and attendees were directed to other rooms to watch a stream. Other talks would be streamed to screens around the conference venue: I remember watching Crusty’s guide to protocol-oriented programming at an in-conference sports bar with a couple of good friends.

It’s also a great way to run a hybrid event: it’s much too easy (as those of us who worked remote in the pre-pandemic times will remember) for online attendees to be second-class citizens in the hybrid world. Making it clear that the event is an online event with the ability to engage from an on-site presence removes that distinction.

Some people will stay away, on the basis that travelling all the way to California to watch AppleTV is not a compelling use of resources. Honestly with this pandemic not being over anywhere except the minds of the politicians who need sacrifices to the line, that’s not a bad thing. Except that these people will miss out on networking, which is a bad thing.

Networking is such a big part of WWDC that plenty of people who didn’t have tickets to the for-realsies iterations would go anyway, maybe going to after parties, maybe going to AltConf (another opportunity to watch a stream of WWDC, alongside original talks by community members). But that was for a week of networking, not a day of watching TV.

That’s OK. Hopefully online watch parties, and local watch parties, will spring up, making the networking side of WWDC more accessible. Making WWDC truly world-wide.

Posted in AAPL, WWDC | Leave a comment

Classism in software engineering

I just heard someone using the phrase “first-class citizen” in a programming podcast, and that led me to ponder the use of that phrase. The podcast was Swift Package Manager SuperPowers from Empower Apps. Empower’s a great podcast, this is a great episode, and the idea of first-class citizenship comes up only in passing and is basically immaterial. But not to this post!

Whatever the situation that leads someone to say “first-class citizen”, there’s a value judgement being made: this thing is good, because it is more conformant to the (probably unspoken) rules of the road of the ecosystem, platform, or whatever the thing is supposed to be a citizen of.

Many of us in the privileged software engineering world live in societies that do not have overt “levels” of citizenship. That said, there still are multiple levels: nationals, resident aliens, temporary visitors, and prisoners may all have different rights and responsibilities. And in societies where there are still explicit or tacit class hierarchies, making reference to them is often somewhere between impolite and offensive. So this idea of “first-class citizenship” comes with a big side-wink: it’s a first-class citizen, we both know what I mean.

An obvious way for a technology to be a first-class citizen of something is for it to be made, distributed, or otherwise blessed by the maker of the something it’s a citizen of. That’s the context in this show: Swift Package Manager is a first-class citizen of the Apple software development platform because it’s a first-party component. Now, it’s a first-party component with the job of giving access to third-party components, so there’s a limit to the vendor ownership, but nonetheless it is there.

In this sense, first-class citizenship confers clear benefits. If someone is already playing in the Apple tools sandpit, they already have access to SwiftPM. They may not already have access to CocoaPods, so the one step “fetch all the packages” becomes two steps: fetch the package tool, then fetch all the packages.

That bit is easier, but it evidently isn’t sufficient. Is the other tool better at fetching packages correctly, or better for writing packages, or more secure, or easier to use? When we say “better”, better at what, and for whom?

It’s possible for something that is first-party to not be first-class. Mac OS X came with the Tcl language for a couple of decades but I can’t find evidence online that it was ever referred to as a “first-class citizen” of the Apple ecosystem. In 2022 you wouldn’t call OpenDoc or the Macintosh Runtime for Java first-class citizens either, because the vendor no longer supports them. Actually it’d be an interesting exercise to take an old Apple Developer Connection CD (the earliest I have will be from around 2005), and find out how much of that content is still supported, and of that subset how much you could get broad agreement for the first-class nature of. I’d be willing to bet that even though ObjC is still around, distributed, supported, and developed, a decent chunk of the community would think twice about calling it first-class.

But then, it’s also possible for things that are third-party to be first-class. Apparently, Java is a first-class citizen in a Docker ecosystem now. And Data should be a first-class citizen in the Enterprise (this is, by the way, a spoiler for the Star Trek: The Next Generation episode the measure of a man).

When third-party things are first-class, we’re saying that going the extra step of getting this thing you don’t already have is better than stopping here and using what you already own. Again we have the questions of better at what and for whom. Really any of this stuff lies on a continuum. Consider a database. You use a database because it’s cheaper, faster, and better to turn your stuff into the database’s model and use the database vendor’s structures and algorithms than it is to design your own efficient storage and retrieval format. If you could do that well (and that’s a big if) then your hand-rolled thing would probably be better for your application than the database, but also all maintenance work falls onto a very small community: you. On the other hand you could use whatever comes in the box, which has the exact opposite characteristics. They each have different types of first-classness.

And then there’s a (very old, but very common) definition of a data type being first-class or not depending on whether they can be represented by variables or expressions. So when Javascript developers say “functions are first-class citizens in Javascript”, they mean that JS has a feature that ALGOL did not.

Posted in tool-support | Leave a comment

Software design is refinement, not abstraction

James Koppel tells us that software engineers keep using the word “abstraction” and that he does not think it means what they think it means. I believe that he is correct, and that the confusion over the term abstraction comes from thinking that programming is about abstraction.

Programming is refinement, not abstraction. You take your idea of what the software should be doing and you progressively refine that idea until what you get it so rote, so formulaic, so prescribed, that you can create a plan that a computer will follow reliably and accurately. Back in the day, that process used to be done literally in the order written as a sequence of distinct activities.

  • You take your idea of what the software should be doing: requirements gathering
  • refine that idea: software specification
  • create a plan that a computer will follow: construction
  • reliably and accurately: verification and validation

It doesn’t matter what paradigm you’re using, what tools, or whether you truly go from one step to the next as described here, what you’re doing is specifying the software you need, and getting so specific that you end up with instructions for a computer. Specific is the opposite of abstract: the process is the opposite of abstraction.

Thus Niklaus Wirth talks about Program Development by Stepwise Refinement. He describes a procedure for solving the 8 queens problem, then refines that procedure until it is entirely described in terms of Algol (more or less) instructions. He could have started by describing a function that turns the set of possible chess boards into the set of boards that solve 8 queens, or he could have started by describing the communication between the board and the queens that would cause them to solve 8 queens.

This is not to say that abstraction doesn’t appear in software development. Wirth’s starting point is an abstract procedure for solving a specific problem: you can follow that procedure to solve 8 queens, you just have to do a lot of colouring in yourself which a computer is incapable of. Maybe GPT-3 could follow that first procedure; maybe one of the intermediate ones.

And his end point is an abstract definition of the instructions the computer will do: you can say j<1 and the computer will do something along the lines of loading the word at the same address previously associated with j into an accumulator, subtracting one, checking the flags index, and conditionally modifying the program counter. And “loading the word at the same address” is itself an abstraction: what really happens might involve page faults, loading data from permanent storage, translation lookaside buffers, and other magic.

Abstractions in this view are a “meet in the middle” situation, not a goal: you can refine/specify your solution until you meet the thing that will do the rest of the work of refinement. Or sometimes a “meet off to the side” situation: if you can make your program’s data look like bags of tuples then you can use the relational model to do a lot of the storage and retrieval work, even if nothing in your problem description looks anything like bags of tuples.

Notice that Wirth’s last section is about generalisation, not abstraction: solving the “N queens” problem is not any less specific than solving the 8 queens problem.

Posted in design, software-engineering | Leave a comment

Unit test: you keep using this word.

There’s an idea doing the rounds that the “unit” in “unit test” means the unity of the test, rather than a test of a software unit. Moreover, that it originally meant this, and that anyone who says “unit test” to mean the test of a software unit is misguided.

Here’s the report of the 1968 NATO conference on software engineering. On their page 20 (as marked, not as in the PDF) is a diagram of a waterfall-esque system development process, featuring these three phases among others (emphasis mine):

  • unit design
  • unit development
  • unit test

“Unit” meaning software unit is used throughout.

Posted in test, unittest | Leave a comment