The debugger of royalty

We’ve all got little libraries of code or scripts that help us with debugging. Often these are for logging information in a particular way, or wrapping logs/tests such that they’re only invoked in Debug builds but not in production. Or they clean up your IDE’s brainfarts.

Having created these debug libraries, how are you going to get your production code to use them? Are you really going to sprinkle MyCompanyDebugLog(fmt,…) messages throughout your app?

Introducing step one on the road to sanity: the debug proxy. This is useful when you want to find out how a particular class gets used, e.g. when it provides callbacks that will be invoked by a framework. You can intercept all the messages to the object, and inspect them as you see fit. Here’s the code (written with the assumption that ARC is enabled):

FZADebugProxy.h

#import <Foundation/Foundation.h>

@interface FZADebugProxy : NSProxy

- (id)initWithTarget: (NSObject *)aTarget;

@end

FZADebugProxy.m

#import "FZADebugProxy.h"

@implementation FZADebugProxy {
    NSObject *target;
}

- (id)initWithTarget:(NSObject *)aTarget {
    target = aTarget;
    return self;
}

- (NSMethodSignature *)methodSignatureForSelector:(SEL)sel {
    NSMethodSignature *signature = [target methodSignatureForSelector: sel];
    if (signature == nil) {
        signature = [super methodSignatureForSelector: sel];
    }
    return signature;
}

- (BOOL)respondsToSelector:(SEL)aSelector {
    return [target respondsToSelector: aSelector] ? YES : [super respondsToSelector: aSelector];
}

- (void)forwardInvocation:(NSInvocation *)invocation {
    invocation.target = target;
    SEL aSelector = [invocation selector];
    (void)aSelector;
    [invocation invoke];
}

@end

And no, there isn’t a bug in the -initWithTarget: method. The slightly clumsy extraction of the selector in -forwardInvocation: is done to avoid a common problem with using Objective-C inside the debugger where it decides it doesn’t know the return type of objc_msgSend() and refuses to call the method.

You would use it like this. Here, I’ve modified the app delegate from BrowseOverflow to use a proxy object for the object configuration – a sort of domain-specific IoC container.

- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions
{
    BrowseOverflowViewController *firstViewController = [[BrowseOverflowViewController alloc] initWithNibName: nil bundle: nil];
    firstViewController.objectConfiguration = (BrowseOverflowObjectConfiguration *)[[FZADebugProxy alloc] initWithTarget: [[BrowseOverflowObjectConfiguration alloc] init]];
    TopicTableDataSource *dataSource = [[TopicTableDataSource alloc] init];
    [dataSource setTopics: [self topics]];
    firstViewController.dataSource = dataSource;
    self.navigationController.viewControllers = [NSArray arrayWithObject: firstViewController];
    self.window.rootViewController = self.navigationController;
    [self.window makeKeyAndVisible];
    return YES;
}

The bold line is the important change. The cast silences the compiler’s strict type-checking when it comes to property assignment, because it doesn’t believe that NSProxy is of the correct type. Remember this is only debug code that you’re not going to commit: you could switch to a plain old setter, suppress the warning using diagnostic pragmas or do whatever you want here.

At this point, it’s worth running the unit tests and using the app to convince yourself that the behaviour hasn’t changed at all.

So, how do you use it? Shouldn’t there be an NSLog() or something in the proxy class so you can see when the target’s messaged?

No.

Step two on the road to sanity is to avoid printf()-based debugging in all of its forms. What you want to do here is to use Xcode’s debugger actions so that you don’t hard-code your debugging inspection capabilities into your source code.

Set a breakpoint in -[FZADebugProxy forwardInvocation:]. This breakpoint will be met whenever the target object is messaged. Now right-click on the breakpoint marker in the Xcode source editor’s gutter and choose “Edit Breakpoint…” to bring up this popover.

Xcode's breakpoint popover.

In this case, I’ve set the breakpoint to log the selector that was invoked, and crucially to continue after evaluation so that my app doesn’t stop in the debugger every time the target object is messaged. After a bit of a play with the app in the simulator, the debug log looks like this:

GNU gdb 6.3.50-20050815 (Apple version gdb-1752) (Sat Jan 28 03:02:46 UTC 2012)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin".sharedlibrary apply-load-rules all
Attaching to process 3898.
Pending breakpoint 1 - ""FZADebugProxy.m":37" resolved
Current language:  auto; currently objective-c
0x10271:	 "stackOverflowManager"
0x102a0:	 "avatarStore"
0x10271:	 "stackOverflowManager"
0x10271:	 "stackOverflowManager"
0x102a0:	 "avatarStore"

Pretty nifty, yes? You can do a lot with Xcode’s breakpoint actions: running a shell script or an AppleScript are interesting options (you could have Xcode send you an iMessage every time it send your target an Objective-C message). Speaking out the names of selectors is fun for a very short while, but not overly useful.

Xcode’s breakpoint actions give you a much more powerful debugging capability than NSLog(). By using breakpoint actions on an Objective-C proxy object, you can create highly customisable aspect-oriented techniques for debugging your code.

Why we don’t trust -retainCount

I’m pretty sure @bbum must have worn through a few keyboards telling users of StackOverflow not to rely on the value of an Objective-C object’s -retainCount. Why? When we create an object, it has a retain count of 1, right? Retains (and, for immutable objects, copies) bump that up, releases (and, some time later, autoreleases) bring it down, right? If an attempt to release would bring the retain count to 0, that object gets released, right? Right?!?

Well, that’s not true for all objects, but leaving that aside, your code isn’t the only code running in your process. The system libraries along with any third-party code you’ve included in your app are all doing things, including retaining and releasing objects. Let’s take a look at a specific case of that.

The code in this post is from Gnustep-base, the LGPL implementation of OpenStep’s Foundation library. The behaviour shown is identical to behaviour in Apple’s Foundation. Here’s (most of) +[NSNumber initialize].

/*
 * Numbers from -1 to 12 inclusive that are reused.
 */
static NSNumber *ReusedInstances[14];
static NSBoolNumber *boolY;		// Boolean YES (integer 1)
static NSBoolNumber *boolN;		// Boolean NO (integer 0)

+ (void) initialize
{
  int i;

  if ([NSNumber class] != self)
    {
      return;
    }

  // ...

  boolY = NSAllocateObject (NSBoolNumberClass, 0, 0);
  boolY->value = 1;
  boolN = NSAllocateObject (NSBoolNumberClass, 0, 0);
  boolN->value = 0;

  for (i = 0; i < 14; i++)
    {
      NSIntNumber *n = NSAllocateObject (NSIntNumberClass, 0, 0);

      n->value = i - 1;
      ReusedInstances[i] = n;
    }
}

We see that sixteen instances of NSNumber subclasses, representing YES, NO and the integers -1 to 12, have been allocated and stored away in static variables. What’s that about? Well here’s (most of) +numberWithInt:.

/*
 * Macro for checking whether this value is the same as one of the singleton
 * instances.  
 */
#define CHECK_SINGLETON(aValue) \
if (aValue >= -1 && aValue <= 12)\
{\
  return ReusedInstances[aValue+1];\
}

// ...

+ (NSNumber *) numberWithInt: (int)aValue
{
  NSIntNumber *n;

  if (self != NSNumberClass)
    {
      return [[[self alloc] initWithBytes: (const void *)&aValue
        objCType: @encode(int)] autorelease];
    }

  CHECK_SINGLETON (aValue);
  n = NSAllocateObject (NSIntNumberClass, 0, 0);
  n->value = aValue;
  return AUTORELEASE(n);
}

So it looks like the retain count of an object you get back from +numberWithInt: depends on the value you pass, and whether anyone else is trying to use a number with the same value right now.

In other words, while it’s easy to get an object’s retain count using the -retainCount method, the number you expect to see may well be wrong. The real value depends on so many parameters that it’s reasonable to conclude that you don’t know what it’s expected to be right now, so don’t depend on it.

By the way, the NSNumber implementation is a great place to find out more about how Foundation is built. It’s a class cluster (which is why you can see NSIntNumber and NSBoolNumber classes), and you can see how to use marker pointers to encode the value of an NSNumber right into the object pointer.

On explaining stuff to people

An article that recently made the rounds, though it was written back in September, is called Apple’s Idioten Vektor. It’s a discussion of how the CCCrypt() function in Apple’s CommonCrypto library, when used in its default cipher block chaining mode, treats the IV (Initialization Vector) parameter as optional. If you don’t supply an IV, it provides its own IV of 0x0.

Professional Cocoa Application Security also covers CommonCrypto, CBC mode, and the Initialization Vector. Pages 79-88 discuss block encryption. The section includes sample code for both one-shot and staged use of the API. It explains how to set the IV using a random number generator, and why this should be done.[1] Mercifully when the author of the above blog post reviewed the code in my book section, he decided I was doing it correctly.

So both publications cover the same content. There’s a clear difference in presentation technique, though. I realise that the blog post is categorised as a “rant” by the author, and that I’m about to be the pot that calls the kettle black. However, I do not believe that the attitude taken in the post—I won’t describe it, you can read it—is constructive. Calling people out is not cool, helping them get things correct is. Laughing at the “fail” is not something that endears people to us, and let’s face it, security people could definitely be more endearing. We have a difficult challenge: we ask developers to do more work to bring their products to market, to spend more money on engineering (and often consultants), in return for potentially protecting some unquantified future lost revenue and customer hardship.

Yes there is a large technical component in doing that stuff, but solving the above challenge also depends very strongly on relationship management. Security experts need to demonstrate that we’re all on the same side; that we want to work with the rest of the software industry to help make better software. Again, a challenge arises: a lot of the help provided by security engineers comes in the form of pointing out mistakes. But we shouldn’t be self promoting douchebags about it. Perhaps we’re going about it wrong. I always strive to help the developers I work with by identifying and discussing the potential mistakes before they happen. Then there’s less friction: “we’re going to do this right” is a much more palatable story than “you did this wrong”.

On the other hand, the Idioten Vektor approach generated a load of discussion and coverage, while only a couple of thousand people ever read Professional Cocoa Application Security. So there’s clearly something in the sensationalist approach too. Perhaps it’s me that doesn’t get it.

[1]Note that the book was written while iPhone OS 3 was the current version, which is why the file protection options are not discussed. If I were covering the same topic today I would recommend eschewing CCCrypto for all but the most specialised of purposes, and would suggest setting an appropriate file protection level instead. The book also didn’t put encryption into the broader context of cryptographic protocols; a mistake I have since rectified.

On the new Lion security things

This post will take a high-level view of some of Lion’s new security features, and examine how they fit (or don’t) in the general UNIX security model and with that of other platforms.

App sandboxing

The really big news for most developers is that the app sandboxing from iOS is now here. The reason it’s big news is that pretty soon, any app on the Mac app store will need to sign up to sandboxing: apps that don’t will be rejected. But what is it?

Since 10.5, Mac OS X has included a mandatory access control framework called seatbelt, which enforces restrictions governing what processes can access what features, files and devices on the platform. This is completely orthogonal to the traditional user-based permissions system: even if a process is running in a user account that can use an asset, seatbelt can say no and deny that process access to that asset.

[N.B. There’s a daemon called sandboxd which is part of all this: apparently (thanks @radian) it’s just responsible for logging.]

In 10.5 and 10.6, it was hard for non-Apple processes to adopt the sandbox, and the range of available profiles (canned definitions of what a process can and cannot do) was severely limited. I did create a profile that allowed Cocoa apps to function, but it was very fragile and depended on the private details of the internal profile definition language.

The sandbox can be put into a trace mode, where it will report any attempt by a process to violate its current sandbox configuration. This trace mode can be used to profile the app’s expected behaviour: a tool called sandbox-simplify then allows construction of a profile that matches the app’s intentions. This is still all secret internal stuff to do with the implementation though; the new hotness as far as developers are concerned starts below.

With 10.7, Apple has introduced a wider range of profiles based on code signing entitlements, which makes it easier for third party applications to sign up to sandbox enforcement. An application project just needs an entitlements.plist indicating opt-in, and it gets a profile suitable for running a Cocoa app: communicating with the window server, pasteboard server, accessing areas of the file system and so on. Additional flags control access to extra features: the iSight camera, USB devices, users’ media folders and the like.

By default, a sandboxed app on 10.7 gets its own container area on the file system just like an iOS app. This means it has its own Library folder, its own Documents folder, and so on. It can’t see or interfere with the preferences, settings or documents of other apps. Of course, because Mac OS X still plays host to non-sandboxed apps including the Finder and Terminal, you don’t get any assurance that other processes can’t monkey with your files.

What this all means is that apps running as one user are essentially protected from each other by the sandbox: if any one goes rogue or is taken over by an attacker, its effect on the rest of the system is restricted. We’ll come to why this is important shortly in the section “User-based access control is old and busted”, but first: can we save an app from itself?

XPC

Applications often have multiple disparate capabilities from the operating system’s perspective, that all come together to support a user’s workflow. That is, indeed, the point of software, but it comes at a price: when an attacker can compromise one of an application’s entry points, he gets to misuse all of the other features that app can access.

Of course, mitigating that problem is nothing new. I discussed factoring an application into multiple processes in Professional Cocoa Application Security, using Authorization Services.

New in 10.7, XPC is a nearly fully automatic way to create a factored app. It takes care of the process management, and through the same mechanism as app sandboxing restricts what operating system features each helper process has access to. It even takes care of message dispatch and delivery, so all your app needs to do is send a message over to a helper. XPC will start that helper if necessary, wait for a response and deliver that asynchronously back to the app.

So now we have access control within an application. If any part of the app gets compromised—say, the network handling bundle—then it’s harder for the attacker to misuse the rest of the system because he can only send particular messages with specific content out of the XPC bundle, and then only to the host app.

Mac OS X is not the first operating system to provide intra-app access control. .NET allows different assemblies in the same process to have different privileges (for example, a “write files” privilege): code in one assembly can only call out to another if the caller has the privilege it’s trying to use in the callee, or an adapter assembly asserts that the caller is OK to use the callee. The second case could be useful in, for instance, NSUserDefaults: the calling code would need the “change preferences” privilege, which is implemented by writing to a file so an adapter would need to assert that “change preferences” is OK to call “write files”.

OK, so now the good stuff: why is this important?

User-based access control is old and busted

Mac OS X—and for that matter Windows, iOS, and almost all other current operating systems—are based on timesharing operating systems designed for minicomputers (in fact, Digital Equipment Corp’s PDP series computers in almost every case). On those systems, there are multiple users all trying to use the same computer at once, and they must not be able to trip each other up: mess with each others’ files, kill each others’ processes, that sort of thing.

Apart from a few server scenarios, that’s no longer the case. On this iMac, there’s exactly one user: me. However I have to have two user accounts (the one I’m writing this blog post in, and a member of the admin group), even though there’s only one of me. Apple (or more correctly, software deposited by Apple) has more accounts than me: 75 of them.

The fact is that there are multiple actors on the system, but mapping them on to UNIX-style user accounts doesn’t work so well. I am one actor. Apple is another. In fact, the root account is running code from three different vendors, and “I” am running code from 11 (which are themselves talking to a bunch of network servers, all of which are under the control of a different set of people again).

So it really makes sense to treat “provider of twitter.com HTTP responses” as a different actor to “code supplied as part of Accessorizer” as a different actor to “user at the console” as a different actor to “Apple”. By treating these actors as separate entities with distinct rights to parts of my computer, we get to be more clever about privilege separation and assignment of privileges to actors than we can be in a timesharing-based account scheme.

Sandboxing and XPC combine to give us a partial solution to this treatment, by giving different rights to different apps, and to different components within the same app.

The future

This is not necessarily Apple’s future: this is where I see the privilege system described above as taking the direction of the operating system.

XPC (or something better) for XNU

Kernel extensions—KEXTs—are the most dangerous third-party code that exists on the platform. They run in the same privilege space as the kernel, so can grub over any writable memory in the system and make the computer do more or less anything: even actions that are forbidden to user-mode code running as root are open to KEXTs.

For the last eleventy billion years (or since 10.4 anyway), developers of KEXTs for Mac OS X have had to use the Kernel Programming Interfaces to access kernel functionality. Hopefully, well-designed KEXTs aren’t actually grubbing around in kernel memory: they’re providing I/O Kit classes with known APIs and KAUTH veto functions. That means they could be run in their own tasks, with the KPIs proxied into calls to the kernel. If a KEXT dies or tries something naughty, that’s no longer a kernel panic: the KEXT’s task dies and its device becomes unavailable.

Notice that I’m not talking about a full microkernel approach like real Mach or Minix: just a monolithic kernel with separate tasks for third-party KEXTs. Remember that “Apple’s kernel code” can be one actor and, for example, “Symantec’s kernel code” can be another.

Sandboxing and XPC for privileged processes

Currently, operating system services are protected from the outside world and each other by the 75 user accounts identified earlier. Some daemons also have custom sandboxd profiles, written in the internal-use-only Scheme dialect and located at /usr/share/sandbox.

In fact, the sandbox approach is a better match to the operating system’s intention than the multi-user approach is. There’s only one actor involved, but plenty of pieces of code that have different needs. Just as Microsoft has the SYSTEM account for Windows code, it would make sense for Apple to have a user account for operating system code that can do things Administrator users cannot do; and then a load of factored executables that can only do the things they need.

Automated system curation

This one might worry sysadmins, but just as the Chrome browser updates itself as it needs, so could Mac OS X. With the pieces described above in place, every Mac would be able to identify an “Apple” actor whose responsibility is to curate the operating system tasks, code, and default configuration. So it should be able to allow the Apple actor to get on with that where it needs to.

That doesn’t obviate an “Administrator” actor, whose job is to override the system-supplied configuration, enable and configure additional services and provide access to other actors. So sysadmins wouldn’t be completely out of a job.

A Cupertino Yankee in the Court of King Ballmer

This post summarises my opinions of Windows Phone 7 from the Microsoft Tech Day I went to yesterday. There’s a new version of Windows Phone 7 (codenamed Mango) due out in the Autumn, but at the Tech Day the descriptions of the new features were mainly the sorts of things you see in the Microsoft PressPass video below (Silverlight required), the API stuff is going on in a separate event.

I want to provide some context: I first encountered C#, J# and .NET back in around 2002, when I was given a beta of Visual Studio .NET (Rainier) and Windows .NET Server (which later became Windows Server 2003). Since then, of course most of my programming work has been on Objective-C and Java, on a variety of UNIX platforms but mainly Mac OS X and iOS. But I’ve kept the smallest edge of a toe in the .NET world, too.

From the perspective of the phone, however, I really am coming to this as an iOS guy. Almost all of the mobile work I’ve done has been on iOS, with a small amount of Android thrown into the mix. I’ve used a WP7 phone a couple of times, but have no experience programming on WP7 or earlier Windows Mobile platforms.

The UI

The speakers at the Tech Day – a mix of Microsoft developer relations and third-party MVPs – brought as much focus on user experience and visual impact of WP7 apps as you’ll find at any Apple event. Windows Phone uses a very obviously distinctive UI pattern called Metro, which you can see in the demo screencasts, or the Cocktail Flow app.

Metro is almost diametrically opposed to the user experience on iOS. Rather than try to make apps look like physical objects with leather trim and wooden panels, WP7 apps do away with almost all chrome and put the data front and centre (and, if we’re honest, sides and edges too). Many controls are implicit, encouraging the user to interact with their data and using subtle iconography to provide additional guidance. Buttons and tiles are far from photorealistic, they’re mainly understated coloured squares. Users are not interacting with apps, they’re interacting with content so if an app can provide relevant functionality on data from another app, that’s encouraged. A good example is the augmented search results demoed in the above video, where apps can inspect a user’s search terms and provide their own content to the results.

In fact, that part of the video shows one of the most striking examples of the Metro user interface: the panorama view. While technologically this is something akin to a paginated scroll view or a navigation controller, it’s the visual execution that makes it interesting.

Instead of showing a scroll thumb or a page indicator, the panorama just allows the title of the next page to sneak into the page you’re currently looking at, giving the impression that it’s over there, and if you swipe to it you can find it. When the user goes to the next page, a nice parallax scroll moves the data across by a page but the title by only enough to leave the edges of the previous and next titles showing.

The Tools

It’s neither a secret nor a surprise that Microsoft’s developer tools team is much bigger than Apple’s, and that their tools are more feature-rich as a result (give or take some ancient missteps like MSTest and Visual SourceSafe). But the phone is a comparatively new step: WP7 is under a year old, but of course Windows Mobile and Compact Editions are much older. So how have Microsoft coped with that?

Well, just as Apple chose to use their existing Cocoa and Objective-C as the basis of the iOS SDK, Microsoft have gone with .NET Compact Framework, Silverlight and XNA. That means that they get tools that already support the platform well, because they’re the same tools that people are using to write desktop, “rich internet” and Xbox Live applications.

From the view-construction perspective, XAML reminds me a lot more of WebObjects Builder than Interface Builder. Both offer drag-and-drop view positioning and configuration that’s backed by an XML file, but in Visual Studio it’s easy to get precise configuration by editing the XML directly, just as WebObjects developers can edit the HTML. One of the other reasons it reminds me of WebObjects is that Data Bindings (yes, Windows Phone has bindings…) seems to be much more like WebObjects bindings than Cocoa Bindings.

Custom classes work much better in the XAML tools than in IB. IB plugins have always been a complete ‘mare to set up, poorly documented, and don’t even work in the Xcode 4 XIB editor. The XAML approach is similar to IB’s in that it actually instantiates real objects, but it’s very easy to create mock data sources or drivers for UI objects so that designers can see what the app looks like populated with data or on a simulated slow connection.

Speaking of designers, an interesting tool that has no parallel on the iPhone side is Expression Blend, a XAML-editing tool for designers. You can have the designer working on the same files as the developer, importing photoshop files to place graphics directly into the app project.

It’d be really nice to have something similar on iPhone. All too often I have spent loads of time on a project where the UI is specified as a photoshop wireframe or some other graphic provided by a web designer, and I’m supposed to customise all the views to get pixel-perfection with these wireframes. With Blend, the designer can waste his time doing that instead :).

Other tools highlights include:

  • Runtime-configurable debugging output on both phone and emulator, including frame rates, graphics cache miss information, and Quartz Debug-style flashes of updated screen regions
  • The emulator supports virtual accelerometers
  • The emulator supports developer-supplied fake location information and even test drivers generating location updates <3

The biggest missing piece seems to be a holistic debugging app like Apple’s Instruments. Instruments has proved useful for both bug fixing and performance analysis, and it’s pretty much become a necessary part of my iOS and Mac development work.

Update: I’m told by @kellabyte that an Instruments-like tool is coming as part of the Mango SDK, and that this was announced at MIX ’11.

The “ecosystem”

A couple of the demos shown yesterday demonstrated phone apps talking to Azure cloud services, ASP.NET hosted web apps (mainly using the RESTful OData protocol), SOAP services etc. Because there’s .NET on both sides of the fence, it’s very easy to share model code between the phone app and the server app.

That’s something Apple lacks. While Cocoa can indeed be used on Mac OS X Server, if you want to do anything server-side you have to either hope you can find some open-source components or frameworks, or you have to switch to some other technology like Ruby or PHP. While Apple ship that stuff, it’s hard to claim that they’re offering an integrated way to develop apps on iOS that talk to Apple servers in the same way that MS do.

To the extent that WebObjects can still be said to exist, it doesn’t really fill this gap either. Yes, it means that Apple provide a way to do dynamic web applications: but you can’t use Apple’s tools (use Eclipse and WOLips instead), you can’t share code between iOS and WO (iOS doesn’t have Java, and WO hasn’t had ObjC in a long time), and you can just about share data if you want to use property lists as your interchange format.

On the other hand, it’s much easier to distribute the same app on both iPhone and iPad than it would be to do so on WP7 and a Microsoft tablet/slate, because their official line is still that Windows 7 is their supported slate OS. I expect that to change once the Nokia handset thing has shaken out, but making a Silverlight tablet app is more akin to writing a Mac app than porting an iOS app.

The market

This is currently the weakest part, IMO, of the whole Windows Phone 7 deal, though it is clear that MS have put some thought and resources behind trying to address the problems. Given that Windows Phone 7 was such a late response to the iPhone and Android, Microsoft need to convince developers to write on the platform and users to adopt the platform. The problem is, users are driven to use apps, so without any developers there won’t be any users: without any users, there’s no profit on the platform so there are no developers.

Well, not no developers. Apparently the 17,000 apps on the marketplace were written by 7,500 of the 42,000 registered developers (and interestingly the UK has one of the lowest ratio of submitted apps to registered developers). By comparison, there are 500,000 apps on the app store.

Microsoft has clearly analysed the bejeesus out of the way their users interact with the marketplace. They have seen, for instance, that MO billing (essentially having your phone operator add your app purchase costs to your phone bill, rather than having a credit card account on the marketplace itself) increases purchase rates of apps by 5 times, and are working (particularly through Nokia of course who already have these arrangements) to add MO billing in as many marketplace countries as they can.

This makes sense. People already have a payment relationship with their network operators, so it’s easier for them to add a few quid to their phone bill than it is to create a new paying account with the Windows Marketplace and give Microsoft their card details. By analogy, iPhone users already have bought stuff from Apple (an iPhone, usually…and often some music) so throwing some extra coin their way for apps is easy. Incidentally, I think this is why the Android app market isn’t very successful: people see Google as that free stuff company so setting up a Checkout account to buy apps involves an activation energy.

Incidentally, some other stats from the app marketplace: 12 downloads per user per month (which seems high to me), 3.2% of all apps downloaded are paid, and the average price of a bought app is a shave under $3. Assuming around 3 million users worldwide (a very rough number based on MS and analyst announcements), that would mean a total of around $3.5M app sales worldwide per month. That’s nowhere near what’s going on on the iPhone, so to get any appreciable amount of cash out of it you’d better have an app that appeals to all of the platform’s users.

The subject of appeal is a big issue, too. Microsoft aren’t really targeting the WP7 at anyone in particular, just people who want a smartphone. With Mango and Nokia handsets, they’ll be targeting people who want a cheaper smartphone than RIM/Apple/Android offers: I bet that brings down that $3 mean app price. This is, in my opinion, a mistake. Microsoft should play to their strengths, and make a generally-useful device but target it at particular groups who Microsoft can support particularly well.

Who are those groups? Well, I think there’s Xbox 360 gamers, as WP7 has Xbox Live integration; and there’s enterprises with custom app needs, due to the integration with Azure and similarity with ASP.NET. It ought to be cheaper for game shop that’s written an Xbox game to do a WP7 tie-in than an iOS tie-in. It ought to be cheaper for an enterprise with an MS IT department to extend their line-of-business apps onto WP7 than onto Blackberry. Therefore MS should court the crap out of those developers and make the WP7 the go-to device for those people, rather than just saying “buy this instead of an iPhone or Android”.

The reason I’d do it that way is that you bring the users and the developers together on shared themes, so you increase the chance that any one app is useful or relevant to any given customer and therefore increase the likelihood that they pay. Once you’ve got gamers and game devs together, for example, the gamers will want to do other things and so there’ll be a need for developers of other classes of app. I call using Xbox games to sell utilities the Halo Effect.

Conclusion

Windows Phone 7 is a well thought out mobile platform, with some interesting user experience and design. It’s got a good development environment, that’s highly consistent with the rest of the Microsoft development platform. However, no matter how easy and enjoyable you make writing apps, ultimately there needs to be someone to sell them to. Microsoft don’t have a whole lot of users on their platform, and they’re clearly banking on Nokia bringing their huge brand to beef up sales. They should be making the platform better for some people, and then getting those people on to the platform to make WP7 phones aspirational. They aren’t, so we just have to wait and see what happens with Nokia.

Footnote: Nokisoft

Nokia do well everywhere that Microsoft doesn’t, by which I mean Europe and China mainly. Particularly China actually, where the Ovi store is pretty big. Conversely, MS phone isn’t doing too badly in America, where Nokia traditionally are almost unheard of. So on paper, the Nokia deal should be a good thing for both companies.

On platform-specific strategies

I’m writing some library code at the moment that needs to work on both Mac OS X and iOS. The APIs I need to use on each platform are different, so I need different code on each platform. I also happen to think that putting both versions of the code in the same implementation file is icky.

Here’s what I’m doing. I define an abstract class FZAPlatformStrategy, that defines the interface and knows how to choose an appropriate subclass depending on the platform.

FZAPlatformStrategy.h

@interface FZAPlatformStrategy : NSObject {
@private
}
- (void)doThing;
+ (FZAPlatformStrategy *)newPlatformStrategy;

@end

FZAPlatformStrategy.m

@implementation FZAPlatformStrategy

- (void)doThing {
    [[NSException exceptionWithName: @"FZAPlatformStrategyAbstractClassException"
                            reason: @"Use +[FZAPlatformStrategy newPlatformStrategy] to get an appropriate subclass"
                          userInfo: nil] raise];
}

+ (FZAPlatformStrategy *)newPlatformStrategy {
    id strategy = nil;
#if TARGET_OS_IPHONE
    strategy = [[FZAPlatformStrategyiPhone alloc] init];
#else
    strategy = [[FZAPlatformStrategyMac alloc] init];
#endif
    return platformStrategy;
}

@end

OK, so now as you can imagine there is a class FZAPlatformStrategyiPhone that uses the iOS APIs, and another FZAPlatformStrategyMac that uses the OS X APIs. Each of these overrides the -doThing method to provide the appropriate platform-specific implementation. Of course, to get these to compile I conditionally define the whole @implementation contents of each based on the target platform.

The library code that needs to interface with this code just calls [FZAPlatformStrategy newPlatformStrategy]; to get an instance, and then because we’ve encapsulated the platform-specific behaviour it can use whatever it gets in a consistent way. Need to add another platform, say GNUStep/Linux? Define a new FZAPlatformStrategy subclass, and change +newPlatformStrategy to know when to return an instance of that subclass.

The remaining bit of ugly is the preprocessor conditional in +newPlatformStrategy. I decided to leave this as-is: it is at least highly localised ugly. The other approach I considered and discarded was to look for a platform-specific class like UIDevice being non-Nil, but then I realised that someone would probably implement that on the other platform and mess up the test.

Any other solutions for doing the “which platform am I on” test gratefully received.

Storing and testing credentials: Cocoa Touch Edition

This article introduces the concept of key stretching, using code examples to explain the ideas. For code you can use in an app that more closely resembles current practice, see Password checking with CommonCrypto.

There’s been quite the media circus regarding the possibility that Sony was storing authentication credentials for its PlayStation Network credentials in plain text. I was even quoted in a UK national daily paper regarding the subject. But none of this helps you: how should you deal with user passwords?

The best solution is also the easiest: if you can avoid it, don’t store the passwords yourself. On the Mac, you can use the OpenDirectory framework to authenticate both local users and users with accounts on the network (where the Mac is configured to talk to a networked directory service). This is fully covered in Chapter 2 of Professional Cocoa Application Security.

On the iPhone, you’re not so lucky. And maybe on the Mac there’s a reason you can’t use the local account: your app needs to manage its own password. The important point is that you never need to see that password—you need to know that the same password was presented in order to know (or at least have a good idea) that the same user is at the touchscreen, but that’s not the same as seeing the password itself.

That means that we don’t even need to use encryption where we can protect the password and recover it when we must check the password. Instead we can use a cryptographic one-way hash function to store data derived from the password: we can never get the password back, but we can always generate the same hash value when we see the same password.

Shut up Graham. Show me the code.

Here it is. This code is provided under the terms of the WTFPL, and comes without any warranty to the extent permitted by applicable law.

The first thing you’ll need to do is generate a salt. This is a random string of bytes that is combined with the password to hash: the point here is that if two users on the same system have the same password, the fact that the salt is different means that they still have different hashes. So you can’t do any statistical analysis on the hashes to work out what some of the passwords are. Otherwise, you could take your knowledge that, say, 10% of people use “password” as their password, and look for the hash that appears 10% of the time.

It also protects the password against a rainbow tables attack by removing the one-one mapping between a password and its hash value. This mitigation is actually more important in the real world than the one above, which is easier to explain :-).

This function uses Randomization Services, so remember to link Security.framework in your app’s link libraries build phase.

NSString *FZARandomSalt(void) {
    uint8_t bytes[16] = {0};
    int status = SecRandomCopyBytes(kSecRandomDefault, 16, bytes);
    if (status == -1) {
        NSLog(@"Error using randomization services: %s", strerror(errno));
        return nil;
    }
    NSString *salt = [NSString stringWithFormat: @"%2x%2x%2x%2x%2x%2x%2x%2x%2x%2x%2x%2x%2x%2x%2x%2x",
                      bytes[0],  bytes[1],  bytes[2],  bytes[3],
                      bytes[4],  bytes[5],  bytes[6],  bytes[7],
                      bytes[8],  bytes[9],  bytes[10], bytes[11],
                      bytes[12], bytes[13], bytes[14], bytes[15]];
    return salt;
}

Now you pass this string, and the password, to the next function, which actually calculates the hash. In fact, it runs through the hashing function 5,000 times. That slows things down a little—on an A4-equipped iPad it takes nearly 0.088s to compute the hash—but it also slows down brute-force attacks.

NSData *FZAHashPassword(NSString *password, NSString *salt) {
    NSCParameterAssert([salt length] >= 32);
    uint8_t hashBuffer[64] = {0};
    NSString *saltedPassword = [[salt substringToIndex: 32] stringByAppendingString: password];
    const char *passwordBytes = [saltedPassword cStringUsingEncoding: NSUTF8StringEncoding];
    NSUInteger length = [saltedPassword lengthOfBytesUsingEncoding: NSUTF8StringEncoding];
    CC_SHA512(passwordBytes, length, hashBuffer);
    for (NSInteger i = 0; i < 4999; i++) {
        CC_SHA512(hashBuffer, 64, hashBuffer);
    }
    return [NSData dataWithBytes: hashBuffer length: 64];
}

Where do I go now?

You now have two pieces of information: a random salt, like edbfe42b3da2995a159c16c0a7184211, and a hash of the password, like 855fec563d91576db0e66d8745a3a9cb71dbe40d7cb2615a82b1c87958dd2e8e56db02860739422b976f182a7055dd223a3037dd3dcc5e1ca28aaaf0bade8a08. Store both of these on the machine where the password will be tested. In principle there isn’t too much worry about this data being leaked, because it’s super-hard to get the password out of it, but it’s still best practice to restrict access as much as you can so that attackers have to brute-force passwords on your terms.

When you come to verify the user’s password, pass the string presented by the user and the stored salt to FZAHashPassword(). You should get the same hash out that you previously calculated, if the same password was presented.

Anything else?

Yes. The weakest part of this solution is no longer the password storage: it’s the password itself. The salt+hash shown above is actually for the password “password” (try it yourself), and no amount of software is going to change the fact that that’s a questionable choice of password…well, software that finally does away with password authentication will, but that’s a different argument.

If you want to limit a user’s ability to choose a simple password, you have to do this at password registration and change time. Just look at the (plain-text) password the user has given you and decide whether you want to allow its use.

On NSInvocation

I was going to get down to doing some writing, but then I got some new kit I needed to set up, so that isn’t going to happen. Besides which, I was talking to one developer about NSInvocation and writing to another about NSInvocation, then another asked about NSInvocation. So now seems like a good time to talk about NSInvocation.

What is NSInvocation?

Well, we could rely on Apple’s NSInvocation class reference to tell us that

An NSInvocation is an Objective-C message rendered static, that is, it is an action turned into an object.

This means that you can construct an invocation describing sending a particular message to a particular object, without actually sending the message. At some later point you can send the message as rendered, or you can change the target, or any of the parameters. This “store-and-forward” messaging makes implementing some parts of an app very easy, and represents a realisation of a design pattern called Command.

How is that useful?

The Gang of Four describes Command like this:

Encapsulate a request as an object, thereby letting you parameterize clients with different requests, queue or log requests, and support undoable operations.

Well, what is NSInvocation other than a request encapsulated as an object?

You can imagine that this would be useful in a distributed system, such as a remote procedure call (RPC) setup. In such a situation, code in the client process sends a message to its RPC library, which is actually acting as a proxy for the remote service. The library bundles up the invocation and passes it to the remote service, where the RPC implementation works out which object in the server process is being messaged and invokes the message on that target.

Spoiler alert: that really is how Distributed Objects on Mac OS X operates. NSInvocation instances can be serialised over a port connection and sent to remote processes, where they get deserialised and invoked.

An undo manager, similarly, works using the Command pattern and NSInvocation. Registering an undo action creates an invocation, describing what would need to be done to revert some user action. This invocation is placed on a queue, so the undo operations are all recorded in order. When the user hits Cmd-Z, the undo manager sends the most recent undo invocation to its target.

Similarly, an operation queue is just a list of requests that can be invoked later…this also sounds like it could be a job for NSInvocation (though to be sure, blocks are also used, which is another implementation of the same pattern).

The remaining common application of Command is for sending the same method to all of the objects in a collection. You would construct an invocation for the first object, then for each object in the collection change the invocation’s target before invoking it.

Got a Concrete Example?

OK, here’s one. You can use +[NSThread detachNewThreadSelector: toObject: withTarget:] to spawn a new thread. Because every thread in an iOS application needs its own autorelease pool, you need to create an autorelease pool at the beginning of the target selector’s method and release it at the end. Without using the Command pattern, this means one or more of:

  • Having a memory leak, if you can’t edit the method implementation
  • Having boilerplate autorelease pool code on every method that might – sometime – be called on its own thread
  • Having a wrapper method for any method that might – sometime – need to be called with or without a surrounding pool.

Sucks, huh? Let’s see if we can make that any better with NSInvocation and the Command pattern.

- (id)newResultOfAutoreleasedInvocation:(NSInvocation *)inv {
    id returnValue = nil;
    NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
    [inv invoke];
    if ([[inv methodSignature] methodReturnLength] > 0) {
        if (strncmp([[inv methodSignature] methodReturnType],@encode(id), 1)) {
            char *buffer = malloc([[inv methodSignature] methodReturnLength]);
            if (buffer != NULL) {
                [inv getReturnValue: buffer];
                returnValue = [NSValue valueWithBytes: buffer objCType: [[inv methodSignature] methodReturnType]];
                free(buffer);
            }
        }
        else {
            [inv getReturnValue: &returnValue];
        }
        [returnValue retain];
    }
    [pool release];
    return returnValue;
}

Of course, we have to return a retained object, because the NSAutoreleasePool at the top of the stack when the invocation is fired off no longer exists. That’s why the method name is prefixed with “new”: it’s a hint to the analyser that the method will return a retained object.

The other trick here is the mess involving NSValue. That, believe it or not, is a convenience, so that the same method can be used to wrap invocations that have non-object return values. Of course, using NSInvocation means we’re subject to its limitations: we can’t use variadic methods or those that return a union type.

Now, for any method you want to call on a separate thread (or in an operation, or from a dispatch queue, or…) you can use this wrapper method to ensure that it has an autorelease pool in place without having to grub into the method implementation or write a specific wrapper method.

A side note on doing Objective-C properly: this method compares the result of -[NSMethodSignature methodReturnType] with a specific type using the @encode() keyword. Objective-C type encodings are documented to be C strings, and there’s even a page in the documentation listing the current values returned by @encode. It’s best not to rely on those, as Apple might choose to change or extend them in the future.

On Timeless Programming Books

Recently, the Dog Spanner wrote about Programming With Quartz, a book written at the tail end of 2005 but which is still useful to Mac developers everywhere. I have to agree, this book is still on my shelf and gets an airing every now and then when I need to do battle with custom drawing (even on iOS).

Today, I had a related moment of epiphany related to the immortal nature of a book, as I had a problem with memory allocation that saw me reaching to Mac OS X Internals: a Systems Approach by Amit Singh. This book was released just before Apple’s transition to Intel hardware, but is still for me a definitive reference on how Mac OS X works. If I need to know about the innards of HFS+, the memory allocator or anything at a similar level, it is to this book I turn.

There’s never really been anything that investigates the higher level components of Mac OS X in quite the same depth. When I need to refresh my knowledge of the UNIX APIs, I turn first to Advanced UNIX Programming, a book first published in 1985(!); but it is the 2004 edition that is still a canonical description of programming in the UNIX environment.

Notice that each of these books is around five years old, and yet still I find myself referring to them frequently. In the fast-changing world of software development, where books on the iPhone 3 SDK are woefully outdated, that’s a great achievement.

Honourable mention: a book I have on my shelf that deserves discussion here is Object Oriented Programming: An Evolutionary Approach by Brad Cox. Written in 1986, this lays out his vision for component-based software development using object oriented programming languages. In it he describes a little language he created called Objective-C and uses it to explain his vision. This book demonstrates that not all computing principles are long-lived. Today’s programming practices look nothing like the “Software ICs” of OOP, although we’re still stuck using Objective-C.

On the broken(?) Mac App Store

A day after the Mac App Store was launched, people are reporting that it has been cracked. There are two separate stories here, a vapourware circumvention of the FairPlay DRM used to generate the receipts and a report that certain apps aren’t validating the receipts properly. We can ignore the first case for the moment: it’s important, and if it’s true then Apple needs to fix it (and co-ordinate updating the validation code with us third-party developers). But for the moment, it’s more important that developers are implementing the protections that are in place in their applications – it’s those applications that are supposed to be protected.

Let’s skip, for the moment, the question of whether DRM or anti-cracking mechanisms are ethically right, worthwhile, or how much effort you want to put into them. Apple have done most of the legwork, in providing a vendor-signed receipt that’s part of your signed app bundle. What you need to do is:

  • Check whether you have a receipt
  • Check whether Apple signed the receipt you have
  • Check whether the receipt is valid for your product
  • Check whether the receipt is valid for this version of your product
  • Check whether the receipt is valid for this computer

That’s it in a nutshell. Of course, some nutshells surround very big and complex nuts, and that’s true in this case:

  • There’s some good example code for receipt validation at github/roddi/ValidateStoreReceipt, if you’re going to use it then don’t just paste it wholesale. If everyone uses the same code then it’s super-easy for someone to detect and strip that code from each instance.
  • Check at runtime, as well as before startup. If you just check at startup, then all an attacker needs to do is patch main() to jump straight into NSApplicationMain() and your app runs for free.
  • Code obfuscation is not a very effective tool. Having worked in anti-virus, I know it’s much easier to classify code based on what it does than what it is, and it’s quite easy to find the code that opens the receipt file, or calls exit(173). That said, some of the commercial obfuscation companies offer a guaranteed service, so you can still protect your revenue after the app gets cracked.
  • Update I have been advised privately and seen in a blog post that people are recommending hard-coding their app bundle IDs and version numbers into the binary rather than using Info.plist, because that file can be edited. Well, so can the app binary…and in either case you’d need to re-sign the product with a valid certificate to continue, because Apple have used the kill flag:
    heimdall:~ leeg$ codesign -dvvvv /Applications/Twitter.app/
    Executable=/Applications/Twitter.app/Contents/MacOS/Twitter
    Identifier=com.twitter.twitter-mac
    Format=bundle with Mach-O universal (i386 x86_64)
    CodeDirectory v=20100 size=12452 flags=0x200(kill) hashes=616+3 location=embedded
    CDHash=8e0736639d79a108a5a1ebe89f928d1da0d49d94
    Signature size=4169
    Authority=Apple Mac OS Application Signing
    Authority=Apple Worldwide Developer Relations Certification Authority
    Authority=Apple Root CA
    Info.plist entries=21
    Sealed Resources rules=4 files=78
    Internal requirements count=2 size=344
    

    Changing a hard-coded string in a binary file is not difficult. You can of course obfuscate the string, but the motivated cracker still finds the point where the comparison is made (particularly easily if you use NSStrings). Really, how far you want to go depends on how much you’re willing to spend.

Of course, Fuzzy Aliens Ltd has already been implementing receipt validation for customers, so if this is too hard for you or you don’t have the time… ;-)

A last word on publicising receipt-validation vulnerabilities

You and I both make our living by selling software, or by selling services to people who sell software. Crowing on the interwebs about how this application or that application doesn’t validate its receipts properly is not cool, because you are shitting on your own doorstep. There is no public benefit to public disclosure that class of vulnerability, because DRM is not a user security feature. Don’t do that. Send the developer a private message explaining your findings. Give them a chance to put extra effort into protecting their product, if that’s what they want to do.