Structure and Interpretation of Computer Programmers

I make it easier and faster for you to write high-quality software.

Wednesday, April 28, 2010

WWDC dates announced

The entire of Twitter has imploded after noticing that Apple has announced the dates for WWDC, this year June 7-11. That’s too short notice for me to go, and having only recently started working again after a few months concentrating solely on Professional Cocoa Application Security, I can’t scrape together the few thousand pounds needed to reserve flights, hotel and ticket at a month’s notice.

I hope that those of you who are going have a great time. The conference looks decidedly thin on Mac content this year, and while I still class myself as more of a Mac developer than an iP* developer that shouldn’t be too much of a problem. The main value in WWDC is in the social/networking side first, the labs second, and the lecture content third – so as long as you can find an engineer in the labs who remembers how a Mac works, you’ll probably still have a great week and learn a lot.

posted by Graham Lee at 15:52  

Tuesday, March 2, 2010

How to hire Graham Lee

There are few people who can say that when it comes to Cocoa application security, they wrote the book. In fact, I can think of only one: me. I’ve just put the final draft together for Professional Cocoa Application Security and it will hit the shops in June: click the link to purchase through my Amazon affiliate programme.

Now that the book’s more-or-less complete, I can turn my attention to other interesting projects: by which I mean yours! If your application could benefit from a developer with plenty of security experience and knowledge to share in a pragmatic fashion, or a software engineer who led development of a complex Cocoa application from its legacy PowerPlant origins through Snow Leopard readiness, or a programmer who has worked on performance enhancement in networking systems and low-level daemon code on Darwin and other UNIX platforms, then your project will benefit from an infusion of the Graham Lee magic. Even if you have some NeXTSTEP or OPENSTEP code that needs maintaining, I can help you out: I’ve been using Cocoa for about as long as Apple has.

Send an email to iamleeg <at> securemacprogramming <dot> com and let’s talk about your project. The good news is that for the moment I am available, you probably can afford me[], and I really want to help make your product better. Want to find out more about my expertise? Check out my section on the MDN show, and the MDN security column.

[] It came up at NSConference that a number of devs thought I carry a premium due to the conference appearances, podcasts and other material I produce. Because I believe that honesty is the best policy, I want to come out and say that I don’t charge any such premium. My rates are consistent with other contractors with my level of experience, and I even provide a discounted rate for NGOs and academic institutions.

posted by Graham Lee at 13:22  

Wednesday, July 8, 2009

Refactor your code from the command-line

While the refactoring support in Xcode 3 has been something of a headline feature for the development environment, in fact there’s been a tool for doing Objective-C code refactoring in Mac OS X for a long time. Longer than it’s been called Mac OS X.

tops of the form

My knowledge of the early days is very sketchy, but I believe that tops was first introduced around the time of OPENSTEP (so 1994). Certainly its first headline use was in converting code which used the old NextStep APIs into the new, shiny OpenStep APIs. Not that this was as straightforward as replacing NX with NS in the class names. The original APIs hadn’t had much in the way of foundation classes (the Foundation Kit was part of OpenStep, but had been available on NeXTSTEP for use with EOF), so took char * strings rather than NSStrings, id[]s rather than NSArrays and so on. Also much rationalision and learning-from-mistakes was done in the Application Kit, parts of which were also pushed down into the Foundation Kit.

All of this meant that a simple search-and-replace tool was not going to cut the mustard. Instead, tops needed to be syntax aware, so that individual tokens in the source could be replaced without any (well, alright, without too much) worry that any of the surrounding expressions would be broken, without too much inappropriate substitution, and without needing to pre-empt every developer’s layout conventions.

before we continue – a warning

tops performs in-place substitution on your source code. So if you don’t like what it did and want to go back to the original… erm, tough. If you’re using SCM, there’s no problem – you can always revert its changes. If you’re not using SCM, then the first thing you absolutely need to do before attempting to try out tops on your real code is to adopt SCM. Xcode project snapshots also work.

replacing deprecated methods

Let’s imagine that, for some perverted reason, I’ve written the following tool. No, scrub that. Let’s say that I find myself having to maintain the following tool :-).

#import <Foundation/Foundation.h>

int main(int argc, char **argv, char **envp)
{
NSAutoreleasePool *arp = [[NSAutoreleasePool alloc] init];
NSString *firstArg = [NSString stringWithCString: argv[1]];
NSLog(@"Argument was %s", [firstArg cString]);
[arp release];
return 0;
}

Pleasant, non? Actually non. What happens when I compile it?

heimdall:Documents leeg$ cc -o printarg printarg.m -framework Foundation
printarg.m: In function ‘main’:
printarg.m:6: warning: ‘stringWithCString:’ is deprecated (declared at /System/Library/Frameworks/Foundation.framework/Headers/NSString.h:386)
printarg.m:7: warning: ‘cString’ is deprecated (declared at /System/Library/Frameworks/Foundation.framework/Headers/NSString.h:367)

OK so we obviously need to do something about this use of ancient NSString API. For no particular reason, let’s start with -cString:

heimdall:Documents leeg$ tops replacemethod cString with UTF8String printarg.m

So what do we have now?

#import <Foundation/Foundation.h>

int main(int argc, char **argv, char **envp)
{
NSAutoreleasePool *arp = [[NSAutoreleasePool alloc] init];
NSString *firstArg = [NSString stringWithCString: argv[1]];
NSLog@"Argument was %s", [firstArg UTF8String], length);
[arp release];
return 0;
}

Looking good. But we still need to fix the -stringWithCString:. That could be just as easy, replacemethod stringWithCString: with stringWithUTF8String: would do the trick. However let’s be a little
different here. Why don’t we use -stringWithCString:encoding:? If we do that, then we’re going to need to take a guess at the second argument, because we’ve got no idea what the encoding should be (that’s why -stringWithCString: is deprecated, after all. However if we’re happy to assume UTF8 is fine for the output, let’s do that for the input. We’d better let everyone know that’s what happened, though.

So this rule is starting to look quite complex. It says “replace -stringWithCString: with -stringWithCString:encoding:, keeping the C string argument but adding another argument, which should be NSUTF8StringEncoding. While you’re at it, warn the developer that you’ve had to make that assumption”. We also (presumably) want to combine it with the previous rule, so that if we see the original file we’ll catch both of the problems. Luckily tops lets us write scripts, which comprise of one or more rule descriptions. Here’s a script which encapsulates both our cString rules:

replacemethod "cString" with "UTF8String"
replacemethod "stringWithCString:<cString>" with "stringWithCString:<cString>encoding:<encoding>" {
replace "<encoding_arg>" with "NSUTF8StringEncoding"
} warning "Assumed input encoding is UTF8"

So why does the <encoding> token become <encoding_arg> in the sub-rule? Well that means “the thing which is passed as the encoding argument”. This avoids confusion with <encoding_param>, the parameter as declared in the class interface (yes, you can run tops on headers as well as implementations).

Now if we save this script as cStringNoMore.tops, we can run it against our source file:

heimdall:Documents leeg$ tops -scriptfile cStringNoMore.tops printarg.m

Which results in the following source:

#import <Foundation/Foundation.h>

int main(int argc, char **argv, char **envp)
{
NSAutoreleasePool *arp = [[NSAutoreleasePool alloc] init];
#warning Assumed input encoding is UTF8
NSString *firstArg = [NSString stringWithCString:argv[1] encoding:NSUTF8StringEncoding];
NSLog(@"Argument was %s", [firstArg UTF8String]);
[arp release];
return 0;
}

Now, when we compile it, we no longer get told about deprecated API. Cool! But it looks like I need to verify that the use of UTF8 is acceptable:

heimdall:Documents leeg$ cc -o printarg printarg.m -framework Foundation
printarg.m:6:2: warning: #warning Assumed input encoding is UTF8

exercises for the reader, and caveats

There’s plenty more to tops than I’ve managed to cover here. You could (and indeed Apple do) use it to 64-bit-cleanify your sources. Performing security audits is another great use – particularly using constructs such as:

replace strcpy with same error "WTF do you think you're doing?!?"

However, notice that tops is a blunter instrument than the Xcode refactoring capability. Its smallest unit of operation is the source file; refactoring only within particular methods is not quite easily achieved. Also, as I said before, remember to check your source into SCM before running a script! There is a -dont option to make tops output its proposed changes without applying them, too.

Finally tops shouldn’t be used fully automated. Always assume that you need to inspect the output carefully, don’t just Build and Go.

posted by Graham Lee at 04:33  

Saturday, June 13, 2009

WWDC wind-down

As everyone is getting on their respective planes and flying back to their respective homelands, it’s time to look back on what happened and what the conference means.

The event itself was great fun, as ever. Meeting loads of new people (a big thank-you to the #paddyinvasion for my dishonourary membership) as well as plenty of old friends is always enjoyable – especially when everyone’s so excited about what they’re working on, what they’ve discovered and what they’re up to the next day. It’s an infectious enthusiasm.

Interestingly the sessions and labs content has more of a dual impact. On the one hand it’s great to see how new things work, how I could use them, and to realise that I get what they do. The best feeling is taking some new information and being able to make use of it or see how it can be used. That’s another reason why talking to everyone else is great – they all have their own perspectives on what they’ve seen and we can share those views, learning things from each other that we didn’t get from the sessions. If you were wondering what the animated discussions and gesticulations were in the 4th Street Starbucks at 7am every morning, now you know.

On the other hand, it makes me realise that OS X is such a huge platform that there are parts I understand very well, and parts that I don’t really know at all. My own code spreads a wide path over a timeline between January 1, 1970 and September 2009 (not a typo). For instance, it wasn’t until about 2003 that I knew enough NetInfo to be able to write a program to use it (you may wonder why I didn’t just use DirectoryServices – well even in 2003 the program was for NeXTSTEP 3 which didn’t supply that API). I still have a level of knowledge of Mach APIs far below “grok”, and have never known even the smallest thing about HIToolbox.

There are various options for dealing with that. The most time-intensive is to take time to study – I’ve got a huge collection of papers on the Mach design and implementation, and occasionally find time to pop one off the stack. The least is to ignore the problem – as I have done with HIToolbox, because it offers nothing I can’t do with Cocoa. In-between are other strategies such as vicariously channeling the knowledge of Amit Singh or Mark Dalrymple and Aaron Hillegass. I expect that fully understanding Mac OS X is beyond the mental scope of any individual – but it’s certainly fun to try :-).

posted by Graham Lee at 16:01  

Monday, December 22, 2008

Cocoa Memory Management

It becomes evident, thanks to the mass centralisation of the neverending september effect that is stackoverflow, that despite the large number of electrons expended on documenting the retain/release/autorelease reference counting mechanism for managing memory in Cocoa, Cocoa Touch, UIKit, AppKit, Foundation, GNUstep, Cocotron and Objective-C code, very few people are reading that. My purpose in this post is not to re-state anything which has already been said. My purpose is to aggregate information I’ve found on the topic of managing memory in Cocoa, so I can quote this post in answers to questions like these.

In fact, I’ve already answered this question myself, as How does reference counting work? As mentioned in the FAQ, I actually answered the question “how do I manage object lifecycles in (Cocoa|GNUstep|Cocotron)”? It’s actually a very violently distilled discussion, so it’s definitely worth checking out the references (sorry) below.

Apple have a very good, and complete, Memory Management Programming Guide for Cocoa. They also provide a Garbage Collection Programming Guide; remember that Objective-C garbage collection is opt-in on 10.5 and above (and unavailable on iPhone OS or earlier versions of Mac OS X). GNUsteppers reading along should remember that the garbage collector available with the GNU objc runtime is entirely unlike the collector documented in Apple’s guide. GNUstep documentation contains a similar guide to memory management, as well as going into more depth about memory allocation and zones. Apple will also tell you how objects in NIBs are managed.

The article which gave me my personal eureka moment was Hold Me, Use Me, Free Me by Don Yacktman. Stepwise has another article, very simple rules for memory management in Cocoa by mmalc, which is a good introduction though with one caveat. While the table of memory management methods at the top of the article are indeed accurate, they might give you the impression that keeping track of the retain count is what you’re supposed to be doing. It’s not :). What you’re supposed to be doing is balancing your own use of the methods for any given object, as described in rules 1 and 2 of “Retention Count rules” just below that table.

James Duncan Davidson’s book “Learning Cocoa with Objective-C” has not been updated in donkey’s years, but its section on memory management is quite good, especially the diagrams and the “rules of thumb” summary. Luckily, that section on memory management is the free sample on O’Reilly’s website.

If reading the theoretical stuff is all a bit too dry, the Mac Developer Network have a rather comprehensive memory management training video which is $9.99 for non-MDN members and free for paid-up members.

Finally, Chris Hanson has written a good article on interactions between Cocoa memory management and objc-exceptions; if you’re using exceptions this is a good discussion of the caveats you might meet.

posted by Graham Lee at 13:34  

Thursday, October 30, 2008

Solaris iPhone Edition

Apple’s one new feature in Snow Leopard is support for Exchange, which if not squarely an Enterprise lure is certainly bait for medium businesses. But here we hit Apple’s perennial problem; they want to sell more into businesses (because that’s where at least 2/3 of all PC money is to be made) but they want to design their systems for home users. When a system is designed to cover every possible potential use for a computer we end up with Windows, which is the kind of “few things to all people” solution that Apple are – rightly – keen to avoid. But as Tim Cook’s “state of the Mac” segment in the recent laptop event showed, one of Apple’s biggest growth areas is education which is organised along enterprisey lines.

Their solution thus far has been a partial one; we get Mac OS X which is basically a consumer OS, and then we get Mac OS X server which is the same OS with a few configuration changes and extra apps to support being used as a workgroup server. This is less distinct than the changes between Mac OS X and iPhone OS X, but the principle is the same; the same technology is used in different ways, so we get different interfaces to it. Note that these aren’t really very divergent products – a UNIX expert could set up an Open Directory Master on a standard Mac OS X box were they so inclined. We get the Mac Pro and the XServe as nods to the existence of more powerful hardware than the iMac. While Apple do have a network of business development managers, enterprise sales people, sales engineers and so on who can support larger customers, their capabilities and freedom are restricted by working on a consumer product in a consumer organisation.

Assuming that Apple aren’t going to retreat and consolidate all of their effort on the consumer/prosumer, the logical plan seems to be “the same only more so”; carry on the scheme of applying a common technology base to multiple markets, but with the product interfaces and configurations being specific to the role in which they’ll be used. Empower those enterprise sales, support and development teams to make the changes required in both the shared technology base and the domain-specific parts in order to advance their own cause. Allow them to do so in such a way that the consumer focus of the standard products is not diluted. To do all this, what Apple would need is to clearly delineate their Core OS, Consumer OS and Server OS engineering groups, while adding staff, expertise and intellectual property to their Server OS, Server Hardware and Enterprise Support groups.

The bit about “adding staff, expertise and intellectual property to their Server OS, Server Hardware and Enterprise Support groups” can be easily achieved by using the Blue Peter principle. Here’s one I prepared earlier. And no, I’m not going mad. Sun have plenty of experience in supporting larger customers and what marketing people like to call vertical markets, and have some good technology: hardware, operating systems software, enterprise services and applications. Their only problem is that they can’t make any money on it. On the other hand with Apple it seems that the money is there to be made, and the problem is stepping up to that plate without compromising the consumer products. Consolidating Mac OS X [+ Server] and Solaris 10 would not be trivial but is not beyond the realms of fantasy. NeXTSTEP ran on SPARC hardware, and as we know that Mac OS X runs on PPC, two different Intel architectures and ARM it’s likely that the effort to port Mac OS X to SPARC would not be great. But perhaps more useful in the short term is that OpenStep ran on Solaris before, and could do again. Even though Sun have switched Solaris to a SYSV-derived platform, due to Apple’s recent push for standardisation with Leopard the two OS are likely more source-code compatible than NeXTSTEP and SunOS 4 ever were. Getting Cocoa up on Solaris would mean that application portability (for the sorts of apps that server admins will want – including Apple’s own server admin tools, not for OmniDazzle) becomes viable while the combined company (Snapple?) concentrate on integrating the core tech. They could even get Jonathan Schwartz to do the coding.

Another factor in this proposition is that JAVA is cheap. Apple currently have about $20B in cash and Sun’s shares are worth $3.6B, but taking into account that Sun have lost 98% of their dot-com-boom value without slowing their R&D projects, the value for money when you want them for their tech, smarts and goodwill rather than their user base is astounding.

Oh, and speaking of JAVA, what about Java? Java currently represents Sun’s main income due to the licensing scheme, but Apple’s investment in the platform has declined over time from the Rhapsody days of “everything is Java”; currently the available Java on Mac OS X lags behind Sun’s version and isn’t ppc64 compatible. The WebObjects team (and hence the Apple store and iTunes) have a heavy Java investment, while other teams have dropped Java (Cocoa) and still others eschew it completely. The iPhone has a very busy developer ecosystem – and absolutely no Java. Where the hypothetical Snapple would leave Java is entirely open, but the option of packaging up the combined company’s Java assets and re-selling them would seem unnecessary, unless you thought that even $3.6B was too much to pay.

posted by Graham Lee at 00:43  

Friday, June 13, 2008

WWDC day 00000000000000000000000000000101

Technically day five isn’t over yet, but with just one session and a taxi ride remaining I doubt I’m going to get anything prematurely wrong here. In fact, I’m no longer entirely sure that I’ll remain awake through the rest of the day so even if something entirely different does happen (still haven’t heard anything about IXKit this millennium) I expect I’d miss it.

Planning an east-bound flight is always difficult, because an 11-hour flight through eight time zones “takes” 19 hours. So if I sleep on the plane I’ll wake up in the middle of the afternoon UK time, and going back to sleep at newly-local night-time might be complicated. Conversely if I don’t sleep on the plane then I won’t be going to sleep until at least 9am currently-local time, for a total of about 28 hours uptime. I was just talking to someone who recommended fasting, I’m not sure whether it’s worth going through that hardship for an unverified theory (stuff all of that “in the name of science” crap). The other option would be to go into work 3pm-11pm for the beginning of the week, I’m not sure everyone else will appreciate having their meetings moved to the middle of the night and I don’t think the canteen’s open that late either ;-).

Still, of the three WWDCs I’ve been to this is definitely in the top ten list of interesting and exciting WWDCs. There definitely still are fellow NeXT fans in the woodwork (actually, tap tap I think it’s gypsum board) who worm their way out during these events, I guess that most have no motivation not to be working on Cocoa. Actually, I don’t recall having seen Andrew Stone this week, and I know of a couple of other guys who aren’t here, but have definitely managed to gain some traction for the phrase “NeXTSTEP Mobile ;-)

posted by Graham Lee at 20:54  

Friday, June 13, 2008

WWDC day four

Not so many sessions attended today – partly because I’ve reached the limit of what the human physiology can achieve on a diet of coffee and doughnuts. But also due to ducking out of sessions to meet with ex-NeXT guys, ex-Lighthouse guys and ex-colleagues for most of the afternoon. Of course, this was followed by the beer bash, no longer the campus bash which we all know and fondly remember (or at least we all fondly remember queueing for a couple of hours at each end for the coach) but still a good event. Speaking of ex-Lighthouse guys, Wylie and I spent a bit of time putting Sun’s world to rights (essentially, if they were to stop diluting the Java brand, stop pulling themselves in every which way and try to make some money, they might do OK).

So tomorrow is the NeXT meetup at the beginning of lunch, I have a lab appointment at exactly 12 which is a little annoying but I’ll try to make it over. If the G4 cube is the right shape, wrong CPU and Real Men’s Objects use the NX prefix, then come over to the front of the Moscone West after the second session.

posted by Graham Lee at 06:26  

Thursday, June 12, 2008

WWDC day three

Today was not quite a full day, so I managed to spend about an hour or so milling around Yerba Buena park, doing a little gift shopping and generally existing outside of the Mascarpone centre. Also put in another update to the c.l.o-c FAQ (though I didn’t update the date! oops…) to better discuss retain/release memory management. There are still situations (GNUstep, Cocotron, and some particular OS X environments) where garbage collection is either unavailable or unsuitable, and it turns out that a lot of questions on either the group or cocoa-dev recently have been about the memory management system.

In the evening, after dinner at Chevy’s with the u.c.s.m guys (and yet more discussion of the Sophos feature set), checked in to the Apple Design Awards for the first 30 mins or so. All of the winners (and runners-up) I saw were worthy applications, although it was interesting to note that while most of the "productivity" app winners were from small shops, both of the games (runner-up was Command and Conquer 3, winner was Guitar Hero 3) were from large studios, ports of Windows/console games and in existing long-running series. They’re both great games, too, of course. Ian was, of course, rooting for Delicious Library 2, and sadly disappointed ;-).

This was all followed by the AFP548 beer bash over at Thirsty Bear. Yet more "oh, you’re from Sophos? Yeah, let me say this one thing…", which I really enjoy because if something’s either good or bad enough to be the subject of a beer-fuelled rant in a party, I should probably hear about it. Also submitted a couple of entries to Peter’s “Leopard bug Venn diagram” contest; I’m not sure I could do a better job of describing the situation than that but for those who were there, I came up with: Omega = “Closed/Duplicate” and Omega = “NeXTSTEP 7.0”.

posted by Graham Lee at 16:15  

Monday, June 9, 2008

WWDC part 0

well, here it is, the pre-WWDC “I’m jetlagged so you have to put up with my wittering” post. I’m just waiting for a softwareupdate to finish so that I can go out with my camera, taking some early-morning pictures before heading off to stand in line for the Stevenote. I was out for beers with Ian and Neil last night, we’d all heard rumours of a 5 a.m. start to the queue. On the two previous occasions that I’ve been, 9 a.m. has been sufficient; but with the sellout nature of the event it’s likely that the room will fill up rather quickly so we’ve compromised on a 7 a.m. start. Actually, forget the 5 a.m. nonsense, there’s a line of overnight campers – I can’t decide whether they’re deliberately trying to re-enact a Joy of Tech cartoon, or actually have nothing to do with their lives.

posted by Graham Lee at 12:42  
Next Page »

Powered by WordPress