More on UIAutomation tests

Update

The information below is mostly redundant. After filing a bug report with Apple, their engineers determined that the Xcode-detected set of macro actions (find a text field, double click, enter text) weren’t working because the double click action wasn’t editing the text field. It is possible to use UIAutomation Tests, you just have to carefully review the UI actions and determine that they have the effect expected, particularly after letting Xcode record UI macros.

Original Post

Unfortunately my work to organise UIAutomation tests has hit the stumbling block that the UI Automation runner doesn’t use the main thread for main-thread-only APIs.

In Xcode 9 and High Sierra, the authors of that post I just linked found that it was possible to turn off the main thread checker in the Test configuration of the build scheme and get working tests anyway. Unfortunately that doesn’t work for me in Xcode 10 and Mojave: the main thread checker isn’t killing the app: the TSM subsystem is just refusing to do its thing. So my tests can’t do straightforward things like write text into a text field. Unfortunately this is a “it’s not me, it’s you” moment, and I don’t think I can carry on using Xcode’s UI tests for my goals.

However, I still want to be able to write “end-to-end” level tests to drive my development. I have (at least) three ways to proceed:

  • I could find a third party library and discover whether it has the main thread problem. Calabash doesn’t support Mac apps, and the other examples I can find (Cucumberish and TABTestKit) both rely on UI Automation so presumably don’t address the main thread problem.
  • I could write the tests in AppleScript. That would be a good way to build up the AppleScript UI for the app, but it doesn’t represent an end-to-end test of the GUI.
  • I could write the tests using NSApplication.sendEvent(_ event:) to simulate clicks, scrolls and text entry, and use the unit test runner to host them. That could work, but I have my doubts (I would guess that the runner is synchronous and stalls the main thread).

I discovered that it is possible to write the test “at the UI level” but in the unit runner, using a combination of key events and AppKit API like sendAction( to:). The trade-offs of this approach:

  • it takes longer, as the abstractions needed to easily find and use AppKit controls don’t (currently) exist
  • it doesn’t use the Accessibility interface so isn’t an accessibility audit at the same time as a correctness test
  • you don’t hit the same problems as with the UI Automation runner
  • it’s much faster

This may be the best approach for now, though I’d welcome other views.

To become a beginner, first become an expert

We have a whole load of practices in programming that only really work well if you’re already good at whatever the process is supposed to help with.

Scrum is a process improvement framework, but only if you already know how to do process improvement. If you don’t, then Scrum is just the baseline mini-waterfall process with a chance to air your dirty laundry every fortnight.

Agile is good at helping you embrace change, but only if you’re already good enough at managing change to understand which changes should be embraced.

#NoEstimates helps you avoid the overhead of estimates, but only if you’re already good enough at estimates to know that you always write user stories that take 0.5-2 days to implement.

TDD helps you design your APIs, but only if you’re already good enough at API design to understand things like dependency injection and loose coupling.

Microservices help you isolate modules, but only if you’re already good enough at modularity not to get swamped in HTTP calls.

This is all very well for selling consultancy (“if your [agile] isn’t working, then you aren’t [agiling] hard enough, let me [agile] you some more”) but where’s the on-ramp?

In which new developer tools are dull

Over on lobste.rs I said that I don’t hold out much hope for another “blue plane” style event in developer tools. In one of Alan Kay’s presentations, he referred to the ordinary way of things as the pink plane, and incremental advances in the state of affairs being movements in that plane. Like the square in Edwin Abbot’s Flatland that encounters a sphere, a development could take us out of the pink plane into the (orthogonal) blue plane. These blue plane ideas are rare because like the square, it’s hard to even conceive of life outside the pink plane.

In what may just be a surprising coincidence, Apple engineers used Blue and Pink to refer to features in evolutionary and revolutionary developments of their operating system.

Software engineering tooling is, for the majority of developers, in a phase of conservative retreat

Build UIs on the web and you probably won’t use a graphical builder, you’ll type HTML and JavaScript (and maybe JSX) into a text editor.

Build native apps and even where there is a GUI builder, you’ll find people recommending against its use and wanting to do things “programmatically” (by which they mean “through typing”, even though the GUI builder tools are another way to construct a program).

In the last couple of decades, interest in CASE tooling has shrunk to conservative interest in text editors with some syntax highlighting, like vim or Atom. Gone even is the “build and run” button from IDEs, to be replaced with command-line invocations of grunt tasks (a fancy phrase meaning shell scripts), npm scripts (a fancy phrase meaning shell scripts) or rake tasks (you get the idea).

Where previously there were live development environments embedded in the deployment environment (and the Javascript VM is almost perfectly designed for that task), there is now console.log and unit tests. The height of advanced interaction with your programming tools are the REPL (an interactive shell) and the Playground/InstaREPL (an interactive shell that echoes stdin and stdout in different places).

For the most part, and I say that to avoid the inevitable commenter who thinks that a counterexample like LabView or Mathematica or that one person they met who uses Expression Blend renders the whole argument broken, developers have doubled down on the ceremony of programming: the typing of arcane text into an 80×24 character display. Now to be fair, text is an efficient and compact graphical representation of a linear sequence of connected concepts. But it is not the only one, nor the most efficient nor most compact, and neither are many software systems linear.

The rewards in making software to make software are scarce.

You can do like IntelliJ do, and make a better version of the 80×24 text entry thing. You can work for a platform vendor, and make their version of the 80×24 thing. You can go and get an engineering grade 6 or above job in Silicon Valley and tell your manager that whatever it is their business does, you’re going to focus on the 80×24 thing (“at scale”) instead.

What you don’t seem to be able to do is to disrupt the 80×24 thing. It’s free (at least as in beer), it’s ubiquitous, and whether or not it’s as good as it could be it certainly seems to be good enough for the people who not only get paid to make bad software, but get paid again to fix it.

Why I don’t have a favourite programming language

This is my take on Ilya Sher’s similar post, though from a different context. He is mainly interested in systems programming, I have mostly written user apps and backend services, and also some developer tools.

I originally thought that I would write a list of the languages and difficulties I have with them, but I realised that there’s an underlying theme that can be extracted. Programming languages I have used either have too much vendor dependence (I love writing ObjC, but can’t rely on GNUstep when I’m not on Apple), too little interaction with the rest of the software world (I love writing Pharo, but don’t love going through its FFI to use anything else) or, and this is the biggest kicker, I don’t like the development environments.

When I work on JavaScript, my environment is a text editor (something like VSCode or emacs) that has syntax highlighting, maybe has auto-completion…and that’s about it. When I work in something like Java, ObjC or C++, I have a build button, an integrated debugger, and the ability to run tests. And, if I’m lucky, a form designer. When I work in something like Swift or Clojure, I have insta-repls. When I work in Pharo, I have all the live browsers and things you hear about from smug people, but I still have to type code for things you might expect to be ‘live’ in such an environment. I get confused by the version control tools, but that might be because I’m not familiar with image-based development.

It feels like, details of the languages aside, there’s a synthesis of programming language with environment, where the programming language is a tool integrated into the environment just like the compiler and debugger, and the tools are integrated into the programming language, like the Lisp macro system. It feels like environments like Oberon, Lisp machines and Smalltalks all have some of this integration, and that popular programming environments for other languages all have less of it.

I’m not entirely sure what the ideal state is, and whether that’s an ideal just for me or would benefit others. I wrote my MSc thesis on an exploration of this problem, and still have more research to do.

Your build needs to be better

I’ve said it before, build systems are a huge annoyance. If your build is anything other than seemingly instantaneous, it’s costing you severe money.

Your developers are probably off reading HN, or writing blog posts about how slow builds cost them, while the build is going. When they finish doing that, which may be some time after the build completes, they’ll have forgotten some of what they were doing and need to spend some time getting back up to speed.

Your developers are probably suspicious of any build failure, thinking that “the build is flaky” rather than “I made a mistake”. They’ll press the button again and go back to HN. When the same error occurs twice, they might look into it.

Your developers probably know that the build is slow, but not which bit of the build is slow. And they don’t have time to investigate that, where it takes so long to get any work done anyway. So everyone will agree that “there is a problem”, but nothing will get done. Or maybe cargo-cult things will get done, things that speed up “builds” but are not the problem with your build.

The Joel test asks whether you can make a build in one test. Insufficient. If you notice when you’re making a build, you’re slowing your developers down.

…which is not always the worst thing, of course. Sometimes a lengthy translation step from some source language to some optimised form of a machine language program yields better results for your customers, because they get to use a faster program and don’t need to care about the time taken to prepare that program. But let’s be clear: that’s part of the release, and your developers don’t always need to be working from the released product (in fact, they’re usually not). Releases should be asynchronous, and the latency between having something ready to be released and having released it can be fairly high, compared with the latency between having created some source and being able to investigate its utility.

Nonetheless, that should all go off in the background. So really, builds and releases should both be non-events to the developers.

Beware the IDEs

I recently had the opportunity to talk with a couple of software project managers from IBM. That company is of a kind that I have never worked at, and many of the companies I have worked at are of kinds that these IBMers have not worked at. There was thus plenty of opportunity for us to have different opinions and to explore those.

One such difference, though one we did not investigate in depth, is over IDEs. There was no doubt in the IBM view: a developer with an IDE is a more productive developer. If your programming team wants to license some proprietary IDE to use on your project, you’re probably better off paying for the licensing.

That’s not the world I come from, and indeed I may even have removed IDEs from my consciousness completely. Sure, I use a few, but what’s wrong with emacs, TAGS, and a bit of gud.el and M-x compile? I don’t mind launching an IDE when it’s there, but I don’t miss it when I work with separate tools.

So, are the people who write Rails apps in Sublime Text or vim doing it right, or are they missing out on clear productivity gains? More fundamentally, was it a good idea to build out something like Rails without keeping the IDE support in sync with the features?

There are all sorts of reasons to believe that IDEs are better.

  • IDEs are actually better, and IBM has a mature enough development system to be able to measure that.
  • IBM believes the Rational marketing (Rational’s price ticket was at the high end of software company acquisitions and perhaps IBM’s culture has, ahem, rationalised that).
  • There was a time that IDEs were better than developing without, but now bloatware text editors like emacs and vim have caught up without IBM’s view updating.

There are also reasons to believe that IDEs are not better.

  • IDEs are actually not betSourceKit Service crashed.
  • The cynic in me doesn’t believe the facts presented by the IDE vendors’ marketeers.
  • The bloatware text editors have actually caught up.
  • The productivity problems I have do not stem from my choice of editing environment.

What’s kindof interesting is to compare the views of dyed-in-the-wool IDE evangelists, dyed-in-the-wool text editor/command-line proponents, and fence-sitters and find out why they have different views, whether there are facts that can be teased out of these positions or assertions that can be validated. What’s really fascinating is general situation of which this is a single example: there are whole regions of the software development phase space that I have yet to experience.

More Excel-lent Adventures

I previously wrote about Excel as the most successful IDE:

Now what makes a spreadsheet better as a development environment is difficult to say; I’m unaware of anyone having researched it.

That research is indeed extant, and the story is well-told in A Small Matter of Programming. While Professor Nardi’s focus is on end-user programming, reading her book raises questions about gaps in professional programmer tools.

Specifically, in the realm of collaboration. Programming tools support a few limited forms of collaboration:

  • individual work
  • a team of independent individuals working on a shared project
  • pair programmers
  • code review

For everything else, there’s a whiteboard. Why? What’s missing?