Structure and Interpretation of Computer Programmers

As with many of the posts in this blog, this one originally started as a tweet that got too long. With the launch of Path 2, a conversation about Atos ditching email for social media and Yammer posting a video of how their enterprise social network is used at O2, I’ve been thinking about how I’d design the social network for business users.

The TL;DR (or executive summary, if that’s your thing) is “don’t be a lazy arse, read the whole post”.

CSCW

Computer-Supported Collaborative Working. A term that has been around for a few decades, and means pretty much what it says: finding ways to use computers to support people working together. Ideas from CSCW pervade the modern requirements engineering discipline; particularly strong is the notion that the software and the system in which it’s used are related. Change the software and you’ll change the way people work together. Change the people and you’ll change the way they work together too, including how they use the software.

When I finished being an undergraduate and became an actual person, real-world CSCW was synonymous with “groupware” and basically meant an email service with integrated calendars. Wikis existed, and some companies were using them. Trouble ticket systems like req and rt did exist too. All of these tools were completely separate. Most of them, if they needed to tell you something, would send you an email.

Ah, email. The tool that changed “read your memos at 4:30 pm” into a permanent compulsion. How that big red badge on your mail icon taunts you, with its message that there are now THREE REALLY IMPORTANT THINGS you’re ignoring. Only one’s a message from facilities telling you that the wastepaper baskets have been moved, and while the other two are indeed task-related they carry no new information (“I agree we should touch base to talk around this action”). Originally the red badge only mocked me while I was in the office, but then I was issued a smartphone.

Email: the tool that makes conversations in other parts of the company completely undiscoverable. You can no longer get a feel for what’s going on by hanging around the water cooler or the smokers’ shelter, because those discussions are being held over email. Email that you can only read if you’re in on the joke. If you’re supposed to be in on the joke but the author addresses her missive to “j.smith3” instead of “j.smith1”, you’re shit out of luck. Email where you have to look at whether you were CCd or BCCd before weighing in with your tuppen’orth.

We’ve come a long way baby

OK, since those dark days we’ve learned a lot, right? There’s Twitter, and Facebook, and Myspace (yes, it is still a thing), and LinkedIn and Path and Tumblr and Glassboard and…

Lots of these things have good ideas, and lots have not so good ideas. Most of them are designed to be very general things that people in many contexts would use: what would a (by which I mean “my”) business-centric comms tool designed today look like?

The features

Completely replaces internal e-mail for everyone

This is non-negotiable. No more multiple channels of communication (I currently have open: twitter, outlook, skype and IRC. And I’m writing a blog post. And I have a phone. And I get Path push notifications.). No more trying to match up threads that have come in and out of various applications. If the CEO wants to leave a message for the CTO, he does it over the social network, just as when one of the cleaners wants to talk to another.

Pull, don’t push

I will say this nice and loud: I don’t care when your shitty app thinks I need to read my messages. I have work to do, let me get on with it without the guilt of the red badge. Support an OmniFocus-style view where I can see what’s come in, what’s outstanding, and what I think is important or urgent. But let me review that on my own terms. I don’t care whether the personal assistant of the head of some other department thinks this message is double-exclamation-mark-urgent; I’ll decide. Let me set up notifications on things I think important to be notified about.

Everything is a citable object

Twitter gets this correct, facebook does not. Except that Twitter doesn’t really, because I can’t really write long-form posts, styled posts, or put images/video/audio/etc. in Tweets. I can just pretend using URLs. But it does allow you to treat a reply as a first-class object: it’s just a tweet that has a link to a previous tweet. In facebook, comments are special things that are attached to other things but don’t really have an existence of their own.

Facebook also decides that different ways of communicating with someone are, for some reason, different things. Notes appear in the “notes” section, updates on the wall (including updates that are links to notes), photos in photos, music in…you get the idea. I say the important things are who said what to whom, and in what context. If I want to write a long-form post about a comment that was a reply to a photo posted in response to a customer service request, I should be able to do that. That implies that our hypothetical social network would be a platform that can support applications like request tracking, bug reporting, CRM and whatever else it is that people in a big company do. Which brings me on to the next two features.

I need to be able to discover people

People (indeed any primates) naturally organise into smallish close groups, in which they understand the dyads (what each people thinks about the others) quite well. They then know a little bit about what the other groups are, but not necessarily about the relationships inside those groups. Current social network tools don’t really take advantage of that: and no, Google+ circles are not this.

Companies, in fact, do not take this into account. Hierarchical management structures lead to an easy identification of “us” and “them”, so that whenever someone in a different group causes “us” to re-think what we’re doing, we assume that “they” don’t get what “we” are all about. We’re all trying to make the same shareholders rich (or change the world, or gain a pension; whyever it is you do what you do) and we should all be on the same side.

If I’m looking for someone in Legal who knows about open source licenses, the tool should support that. Furthermore, the tool should show me how their circle overlaps with mine, so I can see who to go to (and who to avoid) for an introduction if that’s the best way to proceed. The chance is that in a company with thousands of employees I don’t know who I’m looking for, but someone I know does know. Furthermore, the tool should be able to use sentiment analysis to tell whether the people I’m going to talk to are likely to be on my side or not. This should help combat the “us vs. them” mentality described above, because I can go into my new meeting knowing where I’m similar to my new contact.

[An early draft of the section you’ve just read was titled “It isn’t who you know, it’s who you know who you know knows” but that wasn’t really the name of a feature.]

There are no silos or Chinese walls: everyone can see (and react to) everything.

This has a few benefits. Most importantly, this is a time where valuable advances are being made not by gathering information – which we’ve been doing for decades – but by finding new ways of combining, organising and analysing the data that we’ve already got. Which makes it crazy that most companies stop people from taking a cross-functional view of the things going on in that company. It’s maddening, it costs money, and it needs to stop. If I’m doing some research into, for example, fixing some bug in a product and can find out that Jim on the team downstairs has recently been fixing a related bug on his team’s product, that’s useful. I should be able to see that: we need a platform that supports it.

The second benefit is to remove another barrier that reinforces the “us vs. them” separation of departments. No, sales aren’t hiding anything from you: it’s all available.

Next, if everyone knows that anything they say in “new email” is readable by everyone else in the company, that also helps to increase the collaborative nature of interactions because you can’t hide behind the almost-private nature of email.

Path has an interesting feature in this regard: for each event you can see, you can also see who has seen that event. It’s a useful reminder that what you write is being read, and by whom: once you’ve seen your head of department pop up a few times, you probably remember to make everything work-safe. It’s almost an implementation of the “your mum can read what you write” reminder I wanted to add to facebook.

Now I can hear my security colleagues sharpening their ePencils to write the “waah, insider threat” comment. My question is this: so? Insider threats are an HR problem, so rather than making everybody suffer by carving up the company, consider giving the HR department the tools they need to detect and react to the problem. Such as, I don’t know, a tool that provides a view on what company data people are interacting with along with sentiment analysis…

There are some things that genuinely do need to have restricted access: customer data, employee personal information and the like. These should be exceptions, not the norm, and treated exceptionally. Similarly, contractors and other suppliers who probably should be given access to the platform can have need-to-know access. Again, that doesn’t mean that all your permies do. In fact, show your permies that you trust them with the ability to see what’s going on across the whole company, you’ll probably end up with happier and more motivated permies.

All of this stuff about tearing down the walls doesn’t stop you allowing the employees to organise into groups on the platform. But such groups should be self-organising, transient, and transparent: just because I didn’t join, that doesn’t mean I shouldn’t be able to join in.

Smart searches

Final thing: if you build everything above, you’re plugging everyone in to the company firehose. Make it easy for people to find what they’re after, and to review when new matching results become available.

The MAS section of devforums is, along with a healthy subsection of the rest of the interwebs, aflame with the news that the deadline for sandboxing store-delivered apps is further away than it used to be, but still too close for some people.

What’s the deal?

Bugs have been getting easier to detect over time, as the tools used to create software have got better. Anyone who remembers using a microcomputer BASIC interpreter will be familiar with the syntax error, where you’re allowed to enter nonsensical commands and the program runs anyway, stopping unceremoniously when it reaches the unparseable input. Today, syntax errors can be detected and even automatically corrected in the IDE’s text editor.

A program made of 100% acceptable syntax can still have logic errors, which most people would know as “bugs”. The app is supposed to do one thing, but instead does another. Software testing practices are designed to catch logic errors, and code analysis can detect some of these problems too.

With a little bit of hand-waving, we can describe the next level of complexity as the security error or vulnerability. At this level, an application that both compiles and does what the user expects can be made to do something unexpected under misuse. In other words, logic errors are “will it work”, security errors are “can it be made not to work”. Automatic detection and correction of security errors is, in many situations, less well-advanced than detection of other types of error. Techniques for designing and coding security errors out during software development are still very immature.

A little more hand-waving and we get to the final, and most insidious, level of error I will consider here: the blended threat. This can be defined as “can this application be used in conjunction with other applications to do something unexpected”. An example. Safari for Windows suffered a vulnerability called a “carpet bombing” attack, where the author of a web page could cause the browser to download a file without any user intervention. Not that big of a deal: the browser is supposed to allow downloads, this is a little unexpected but not outside the security model of an application.

However, Safari also has that “automatically open safe files after downloading” feature, and this is where the fun begins. Having downloaded, say, a document file, the browser automatically opens it. Now, what if that document is a PDF, and exploits a vulnerability in Acrobat Reader to execute code on the user’s behalf? A local code execution problem like this might be considered a medium-severity issue by the Reader developers: it requires a user to be coerced or tricked into opening a malicious file, doesn’t it?

Well, no. The combination of the low-severity automatic download bug, the automatic open feature and the medium-severity local code execution bug produce a high-severity remote code execution bug.

Detecting and reacting to this kind of “blended threat” is harder than dealing with any of the aforementioned classes of failure. It relies on knowing the interaction between your own app and the vast number of other apps that exist on the platform, understanding the capabilities and bugs of those other apps and how those interact with the capabilities and bugs in your own. It could, in principle, be done, but currently cannot.

Enter sandboxing.

Fundamentally, the problem is that the expression of different roles in an operating system like Mac OS X is incomplete. I’ll not go into this again: my post on the new lion security things covers this ground. The TL;DR is that Mac OS X, Windows NT and friends all assume that the actors in a computer system are the users, and ignore the fact that the code represents a collection of actors too.

OK, so a user is allowed to download a document, read it, and to delete all of her documents. But does that mean that a collection of apps—each one notionally a separate actor independent from the user—should conspire together to enable this? Probably not; or if so, only over a well-defined inter-process communication system that doesn’t allow this sort of thing to happen implicitly. Creating an Automator action to get some content from a PDF on a web server and delete some documents as a result might be something the OS needs to support, but one click in a browser probably shouldn’t make all of that stuff happen automatically.

App sandboxing introduces two important features: the identification of different apps as different actors with their own privileges on the system is one, and (partially voluntary, on the developers’ parts) restrictions on the ways apps communicate is the other. This restriction is implemented both as a limitation on the IPC mechanisms an app can make use of, and on the files the app may access: the filesystem is an app’s version of sneaker net.

With these restrictions in place, the opportunities for performing a blended-threat attack are severely reduced. If one app is compromised it either doesn’t have permission to contact the apps that the threat relies on, or it tries to contact apps that don’t have permission to talk to it.

My app doesn’t fit the sandbox limitations.

Complaints of that nature on iOS, where the limitations have been in place both for ages and from the beginning of the platform’s support for third-party code, have pretty much died out now. Developers have become used to the idea that if an app cannot fit the limitations of the sandboxed environment, then it cannot be distributed on the platform.

There are two issues that limit the applicability of the “suck it up” solution for Mac apps. The first, and weakest, is that there are existing apps which don’t fit the mould. This is weak because on the Mac there are other distribution mechanisms for getting the app into the customers’ hands; mechanisms that have existed and worked for years, and that don’t require the same controls enforced by the Mac App Store policy. Yes, it would be better if every app were sandboxed, but having any app sandboxed reduces the attack surface of the Mac so a less than 100% hit rate is still a win.

Of course, some apps are only incompatible due to minor technicalities, such as legacy decisions over where to locate user files. In these cases it’s usually somewhat harmless to migrate to a situation where the app is sandbox-compatible, for example using the mediated file read permission to load documents from the legacy storage area.

Other apps are basically never going to work in a sandbox: utility software is a common casualty, especially things like anti-virus apps, disk partition editors, and file managers. IDEs are also likely to fall down here (though one that makes good use of the workspace concept need not). We can conclude that Apple doesn’t want to sell ISV software in those categories on the Mac app store, so using a different store is the solution.

The other, stronger limitation is that there are applications that are already on the store that work currently, because the store doesn’t enforce sandbox restrictions, but won’t work in the future, because they’re incompatible with the sandbox restrictions. Developers of these apps could migrate to a different store but would be unable to bring their existing customers with them, to give those existing customers updated versions, or even to contact those customers to tell them about the change.

Yes, this is a serious problem for the minority of affected apps. Yes, it is a limitation of the App Store’s capability that results from Apple’s business decisions. Yes, it’s going to be a ball ache for affected developers to deal with. No, it’s not a reason to throw the baby out with the bathwater and give up on sandboxing completely. No, I haven’t provided a solution yet. As it’s a business problem your first port of call should be your business contacts at Apple: this is such a big issue that it deserves a separate post. The short version is that if you’re doing business with Apple (or any other company) you need a business relationship with that company.

OK, so technical solutions. It all depends on the design and architecture of your app. Perhaps there are features that can be removed, without destroying the essence of your app while gaining compatibility with the app store. Consider the business impact of removing that one feature versus removing your whole app. Also, remember that unless you’ve sold about 40 million copies, there are more potential customers out there than current customers.

Another possibility is that you separate the product into two components, a service and a controlling app. The service part is incompatible with the app store and distributed separately. The app is available as an additional feature to support easier use of the service. Take a look at the number of apps for controlling MySQL, or Postgres, and you’ll see that having an app store app as a front-end to an external service is something that’s both supported and viable (well, if it isn’t viable, a lot of companies enjoy not making money).

Right from the beginning, you have to accept that this analysis is based on the presentation of Windows 8 shown at the //build/windows conference. I’ve watched the presentation, I’m downloading the developer preview but I’m over an hour away from even discovering whether I have anything I can install that on.

My biggest concern

The thing I was most worried that Windows 8 would bring was yet another chance for Microsoft to flub their developer experience. If you think about it, Microsoft’s future now depends more than it has on any time since the 1980s on their third-party developers. Whereas they used to be able to rely on OEM licensing deals (and strongarming) to ensure that people wanted to use their stuff, that’s starting to wane.

Sure, the main desktop Windows-NT-plus-Active-Directory-plus-Office-plus-Exchange core is still going strong. But look at the web stack: servers are IIS or Apache or one of various other servers that support Rails, PHP and so on. They’re hosted on my Wintel boxen or my Lintel boxen or one of various third-party choices (including Microsoft’s own Azure). The client is a browser (based on anything) or a native mobile app on Windows Phone 7 or one of a handful of other native app platforms, including iOS and Android.

Where Microsoft have the potential to win over all of their competitors is providing a consistent developer experience across the whole of their stack. This isn’t about locking developers in, this is about making it easiest for them to do Microsoft end-to-end.

Consider Apple for a moment. They have a very consistent experience for writing desktop apps and mobile apps, with Cocoa and Cocoa Touch. Do you want to write server software? You can do it on Apple kit, but not with Apple tools. You’ll probably use Rails or another open source UNIX-based stack, at which point you may as well deploy it to someone else’s kit, maybe someone who offers rackmount hardware. You can use iCloud on Apple’s terms for distributed storage, but again if you want generic cloud computing or a different approach to cloud storage, you’re on your own.

On the other hand, Microsoft offers a complete stack. Azure, Windows Server, WPF (desktop client) and SilverLight (browser and phone) all have the same or very similar APIs in the same languages, so it’s easy to migrate between and share code across multiple layers in a complete computing environment.

But, would they do that? The one place where Microsoft itself seems to have failed to recognise its own strength is in the Windows division. They “pulled a Solaris”, and have consistently ignored .NET ever since its inception, in just the same way that the Solaris Operating Environment team in Sun ignored Java. That leads developers to think there must be something wrong with the tech if real developers inside the company who made it don’t want to use it.

So when Microsoft (or rather, Steve Ballmer) announced that the Windows 8 SDK would be JavaScript and HTML5, it made me think that the Windows 8 team might have jumped the shark. That stuff isn’t compatible with C# and XAML, or with the .NET APIs. I thought they’d got themselves into the position where their phone is SilverLight, their desktop/tablet is HTML5 (except legacy stuff which is Win32), their server is .NET… ugh.

So what happened?

Well, in a nutshell, they didn’t do that. The need for a new runtime makes sense because the .NET runtime was only ever a bolt-on on top of Windows, leading to interesting compatibility issues (to run App version x you need .NET version y and Windows version z; but OtherApp version a wants .NET version b). But the fact that the new HTML5 stuff is actually layered on top of the same runtime, and provides access to the same APIs as are available in C#, VB and C++ makes me believe that Microsoft may actually understand that their success depends on developers adopting their entire stack.

I’ll make this point clear now: HTML5 is not the future of app development. Its strength is based on a couple of things: every device needs a high-performance HTML engine anyway in order to provide web browsing; and HTML5 should work across multiple devices. Well, as a third-party developer, the first of those points is not my problem, it’s Microsoft’s problem, or Apple’s, or RIM’s. I don’t care that they get to re-use the same HTML renderer in multiple contexts.

The second of those points will quickly disintegrate when every platform vendor is trying to differentiate itself and provide an HTML5 development environment. Did you see that Expression Blend for HTML demo? That’s a very cool tool, but the demo of using “standard HTML5” relied on a keyword called “ms-grid”. Wait, MS? Is that cross-platform HTML? Also observe that the JS needs to call into the WinRT API, so this really isn’t a cross-platform solution.

No, for me the big story is that you can use C# everywhere: currently with .NET in most places and WinRT on the desktop. Any client application can use XAML for its user interface. That’s big for ecosystem-wide development. There’s one thing I need to learn in order to do mobile, tablet, desktop, server and cloud programming. It’s slightly weird that the namespaces on .NET and WinRT are different, and it would be a good strategic move to support the new namespaces in newer versions of SilverLight (i.e. in Windows Phone 8).

What else happened?

I’m not going to talk much about Metro, it’s a well-designed UI that works well. I’m not sure yet how it works if you’re interacting through a mouse or trackpad. I spoke about it before when discussing Windows Phone 7; where I also expressed my belief that sharing the developer experience across desktop and phone was a good move.

What I will point out is that the Windows team no longer think that they can do what they want and people will buy it. They’re noticing that the competition exists, and that they need to do better than the competition. In this case, “the competition” is Apple, but isn’t Google.

Why do I think that? Sinofsky felt comfortable making a joke about “Chrome-free browser experience”, but the iPad was the elephant in the room. Tablets/slates/whatever were summarised as “other platforms”, although you feel that if Microsoft had a point to score over Android they would have mentioned it specifically.

Conclusions

This means that – perhaps with the exception of Office – Microsoft’s complacency is officially behind it. Sure, Windows 7 and Windows Phone 7 were indications that Microsoft noticed people talking about them falling behind. But now they’ve started to show a strategy that indicates they intend to innovate their way back to the front.

While it’s currently developer preview demoware, Windows 8 looks fast. It also looks different. They’ve chosen to respond to Apple’s touchscreen mobile devices by doing touchscreen everywhere, and to eschew skeuomorphism in favour of a very abstract user interface. Importantly, the Windows APIs are now the same as they’re getting 3rd-party developers to use, and are the same as (or very similar to) the APIs on the rest of their platforms.

Note: as ever, this blog refrains from commenting on speculation regarding undisclosed product innovations from device providers. This post is about the concept of tracking users via a device identifier. You might find the discussion useful in considering future product directions; that’s fine.

Keeping records of what users are up to has its benefits. As a security boffin, I’m fully aware of the benefits of good auditing: discovering what users (and others) have (or haven’t) done to a valuable system. It also lets developers find out how users are getting on with their application: whether they ignore particular features, or have trouble deciding what to do at points in the app’s workflow.

Indeed, sometimes users actively enjoy having their behaviour tracked. Every browser has a history feature; many games let players see what they’ve achieved and compare it with others. Location games would be pretty pointless if they could only tell me where I am now, not tell the story of where I’ve been.

A whole bunch of companies package APIs for tracking user behaviour in smartphone devices. These are the Analytics companies. To paraphrase Scott Adams: analytics is derived from the root word “anal”, and the Greek “lytics” meaning “to pull a business model from”. What they give developers for free is the API to put analytics services in their apps, and the tools to draw useful conclusions from these data.

This is where the fun begins. Imagine that the analytics company uses some material derived from a device identifier (a UDID, IMEI, or some other hardware key) as the database key to associate particular events with users. Now, if the same user uses multiple apps even by different developers on the same device, and they all use that analytics API, then that analytics provider can aggregate the data across all of the apps and build up a bigger picture of that user’s behaviour.

If only one of the apps records the user’s name as part of its analytics, then the analytics company – a company with whom the user has no relationship – gets to associate all of that behaviour with the user’s real name. So, of course, do that company’s customers: remember that the users and developers are provided with their stuff for free, and that businesses have limited tendency toward altruism. The value in an analytics company is their database, so of course they sell that database to those who will buy it: like advertisers, but again, companies with whom the user has no direct relationship.

People tend to be uneasy about invisible or unknown sharing of their information, particularly when the scope or consequences of such sharing are not obvious up front[*]. The level of identifiable[**] information and scope of data represented by a cross-app analysis of a smartphone user’s behaviour – whether aggregated via the model described above or other means – is downright stalker-ish, and will make users uncomfortable.

One can imagine a scenario where smartphone providers try not to make their users uncomfortable: after all, they are the providers’ bread and butter. So they don’t give developers access to such a “primary key” as has been described here. Developers would then be stuck with generating identifiers inside their apps, so tracking a single user inside a single app would work but it would be impossible to aggregate the data across multiple apps, or the same app across multiple devices.

Unless, of course, the developer can coerce the user into associating all of those different identifiers with some shared identifier, such as a network service username. But how do you get users to sign up for network services? By ensuring that the service has value for the user. Look at game networks that do associate user activity across multiple apps, like OpenFeint and Game Center: they work because players like seeing what games their friends are playing, and sharing their achievements with other people.

The conclusion is, in the no-device-identifier world, it’s still possible to aggregate user behaviour, but only if you exchange that ability for something that the user values. Seems like a fair deal.

[*] My “don’t be a dick” guide to data privacy takes into account the fact that people like sharing information via online services such as Facebook, foursquare etc. but that they want to do so on their own terms. It goes like this: you have no right to anything except what the user told you. You have no right to share anything except what the user told you to share; they will tell you who you may share it with, and can change their minds. The user has a right to know what they’re telling you and how you’re sharing it.

[**] Given UK-sized postal zones, your surname and postcode are sufficient to uniquely identify you. Probably your birthday and postcode would also work. It doesn’t take much information to uniquely identify someone, anyway.