App security consultancy from your favourite boffin

I’m very excited to soon be joining the ranks of Agant Ltd, working on some great apps with an awesome team of people. I’ll be bringing with me my favourite title, Smartphone Security Boffin. Any development team can benefit from a security boffin, but I’m also very excited to be in product development with the people who make some of the best products on the market.

There’s another side to all of this: once again can security boffinry be at your disposal. I’ll be available on a consultancy basis to help out with your application security and privacy issues. If you’re unsure how to do SSL right in your iOS app, are having difficulty getting your Mac software to behave in the App Store sandbox, or need help to identify the security pain points in your application’s design or code, I can lend a hand.

Of course, it’s not just about technology. The best way to help your developers get security right is to help your developers to help themselves. When I’ve done security training for developers before I’ve seen those flashes of enlightenment when delegates realise how the issues relate to their own work; the hasty notes of class or method names to look into back at the desk; the excited discusses in the breaks. Security training for iOS app developers is great for the people, great for the product – and, of course, something I can help out with.

To check availability and book some time (which will be no earlier than July), drop me a line on the twitters or at graham@agant.com.

Class clusters, placeholder objects, value-oriented programming, and all that good stuff.

Have you ever seen this exception in your crash log?

2012-05-29 17:55:37.240 Untitled 2[5084:707] *** Terminating app due to uncaught exception ‘NSInvalidArgumentException’, reason: ‘*** -length only defined for abstract class. Define -[NSPlaceholderString length]!’

What’s that NSPlaceholderString class?

Leaving aside NSMutableString for a moment[*], there’s no way for a developer who’s got an instance of an NSString to modify that string. In this model a string instance represents the value of that string: the word “hello” is always going to be “hello”. You can build a sentence that includes the word “hello” in a sequence – e.g. “hello, world”. You can build a different sentence, e.g. “goodbye, world”. You haven’t changed the value of the word “hello” to “goodbye”, you’ve changed the value of the sentence to include a word with a different value.

OK, let’s take that to an extreme. If any string that a developer gets back from NSString‘s API is immutable, then that should include the string she gets back from +allocWithZone:, right? So any extra data passed in an -initWith… method can’t be used to change the string object we just allocated.

That’s OK, because -init… methods are allowed to return a different object, preserving this “don’t change the value” principle. Imagine the C string initialiser for NSString looking like this (I doubt it does – I think it internally converts the string to UTF-16 – but it’ll do as an example):

-(id)initWithCString: (char *)cString encoding: (NSStringEncoding)encoding
{
  NSCString *otherString = [NSCString allocWithZone: [self zone]];
  [self release];
  otherString->length = strlen(cString);
  otherString->bytes = malloc(otherString->length);
  strlcpy(otherString->bytes, cString, otherString->length);
  otherString->storedEncoding = encoding;
  return otherString;
}

This doesn’t violate the no-modification contract, because it only changes an object that’s being built and that the end developer hasn’t seen yet. Once the developer gets to look at this string – when it’s returned from the initialiser – it’ll be immutable.

So this means that the string which was returned from +allocWithZone: represents a particular value of a string: the string that has yet to be assigned a value. Indeed, it’s a placeholder string. But any string that has yet to be assigned a value can be represented by the same placeholder, because they all mean the same thing. That means we can save some memory by creating a Flyweight instance of the placeholder. Even if multiple call sites in multiple threads all get the same instance of our placeholder string, there’s no danger of them tripping over each other because they’ll all then get different strings as they tell the placeholder what values they need to represent.

In fact, if code in two different threads need to represent the same value, it’s safe to give them both references to the same object. Neither can change that object and spoil things for the other.

This pattern of keeping objects immutable in the eyes of client code, providing transformations that result in new objects rather than modifying existing objects, makes a raft of thread safety problems disappear and reduces the complexity of class APIs. I’ll be using it more often in my object models.

[*]To be honest, I’d like to leave it aside forever. It satisfies the Law of Demeter, but there’s a whole class of concurrency problems that only exist because “a mutable string isa string”.

Is privacy a security feature?

I’ve spoken a lot about privacy recently: mainly because it’s an important problem. Important enough to hit the headlines; important enough for trade associations and independent developers alike to make a priority. Whether it’s talks at conferences, or guiding people on designing or implementing their apps, there’s been a lot of privacy involved. But is it really on-topic for a security boffin?

The “yes” camp: Microsoft

In Michael Howard and David LeBlanc’s book, “Writing Secure Code, Second Edition”, there’s a whole chapter on privacy:

Most privacy threats are information disclosure threats. When performing threat analysis, you should look at all such threats as potential privacy violations.

In this view, a privacy problem is a consequence of a failure of confidentiality being disrupted. You model your application, taking into account what data it protects, what value the customers put on that data, and how important it is to protect the confidentiality. Personally-identifying information is modelled in exactly this way.

Privacy automatically falls out of this modelling technique: if people can get access to confidential data, then you have a privacy violation (that also looks like a security vulnerability because it appears in your threat model).

The “no” camp: Oh, it’s Microsoft again

A different viewpoint is expressed in another book by Michael Howard (with Steve Lipner this time): “The Security Development Lifecycle”.

Many people see privacy and security as different views of the same issue. However, privacy can be seen as a way of complying with policy and security as a way of enforcing policy. […] Privacy’s focus is compliance with regulatory requirements[…], corporate policy, and customer expectations.

So in this model, privacy is a statement of intent, and security is a tool to ensure your software follows through on your intent. It’s the difference between design and implementation: privacy is about ensuring you build the right thing, and security helps you build the thing right. The two have nothing to say about each other, except that if you didn’t get the security right you can’t make any claim about whether the policy expressed in the privacy requirements will successfully be met in deployment.

The “who cares?” camp: me

The argument above seems to be a question of semantics, and trying to apportion responsibility for different aspects of development to different roles. In fact, everyone involved in making a product has the same goal – to make a great product – and such niggling is distracting from that goal.

Most of my professional work fits into one of a few categories:

  • Learning stuff
  • Making stuff
  • Helping other people make better stuff
  • Making other people better at making stuff than I am

So if, in the process of helping someone with their security, I should be able to help with their app’s privacy too, should I really keep quiet until we’ve solved some quibbling point of semantics?

Thoughts on Tech Conferences

This post is being, um, posted from the venue for GOTO Copenhagen 2012. It’s the end result of a few months of reflection on what I get out of conferences, what I want to get out of conferences, what I put into (and want to put into) conferences and the position of tech conferences in our industry. I’ve also been discussing things a lot with my friends and peers; I’ve tried to attribute specific quotes where I remember who said them but let it be known that many people have contributed to the paragraphs below in many different ways. I’ll make it clear at the outset that I’m talking about my experience at independent commercial and non-profit tech conferences, not scientific conferences (of which I have little experience) or first-party events like WWDC (which are straightforward marketing exercises).

Conference speakers

My favourite quote on this subject is courtesy of Mike; I remember him saying it in his MDevcon keynote this year but I’m also fairly sure he’s said the same thing earlier:

The talks at a conference are only there so that you can claim the ticket cost as an expense.

We’re in a knowledge economy; but knowledge itself is not of any value unless it’s applied. That means it’s not the people who tell other people what’s going on who’re are doing the most important work; that’s being done by the people who take this raw knowledge, synthesise it into a weltanschauung – a model of how the world works – and then make things according to that model. Using an analogy with the economy of physical things, when we think of the sculpture of David in Florence we think of Michelangelo, the sculptor, not of the quarry workers who extracted the marble from the ground. Yes their work was important and the sculpture wouldn’t exist without the rock, but the most important and valuable contribution comes from the sculptor. So it is in the software world. Speakers are the quarry workers; the marble hewers, providing chunks of rough knowledge-stuff to the real artisans – the delegates – who select, combine and discard such knowledge-stuff to create the valuable sculptures: the applications.

Michelangelo's David, source: Wikimedia Commons.

Conference speakers who believe that the value structure is the other way around are deluding themselves. Your talk is put on at the conference to let people count the conference as a work expense, and to inspire further discussion and research among the delegates on the topic you’re talking about. It’s not there so that you can promote your consultancy/book/product, or produce tweet worthy quotes, or show off how clever you are. Those things run the gamut from “fringe benefits” to “deleterious side effects”.

As an aside, the first time I presented at a Voices That Matter conference I was worried due to the name; it sounds like the thing that matters at this conference is the speakers’ voices. In fact I suspect there is some of that as many of the presenters have books published by the conference hosts, but it’s a pretty good conference covering a diverse range of topics, with plenty of opportunities to talk to fellow delegates. And IIRC all attendees got an “I am one of the voices that matter” sticker. Anyway, back to the topic at hand: thus do we discover a problem. Producing a quality conference talk is itself knowledge work, that requires careful preparation, distillation and combination of even more raw knowledge-stuff. It takes me (an experienced speaker who usually gets good, but not rave, reviews) about three days to produce a new one hour talk, a roughly 25:1 ratio of preparation:delivery. That’s about a day of deciding what to say and what to leave out, a day of designing and producing materials like slides, handouts and sample code, and a day of practising and editing. Of course, that’s on top of whatever research it was that led me to believe I could give the talk in the first place. The problem I alluded to at the start of the last paragraph is this: there’s a conflict between acknowledging that the talks are the bricks-and-mortar of the conference rather than the end product, and wanting some return on the time invested. How that conflict’s resolved depends on the personal values of the individual; I won’t try to speak for any of my peers here because I don’t know their minds.

The conference echo chamber

That’s not my phrase; I’ve heard it a lot and can track my most recent recollection to @secwhat’s post Conference Angst. Each industry’s conferences has a kind of accepted worldview that is repeated and reinforced in the conference sessions, and that only permits limited scrutiny or questioning – except for one specific variety which I’m coming onto later. As examples, the groupthink in indie Mac/iOS conferences is “developers only need developer features that have been blessed by Apple”. There’s recently been significant backlash to the RubyMotion framework, as there usually is when a new third-party abstraction for iOS appears. But isn’t abstraction a good thing in software engineering? The truth is, of course, that there are more things in heaven and earth than are dream’t in Apple’s philosophy.

The information security groupthink is that information security is working. Shocking though it may sound, that’s far from obvious, evident or even demonstrable. Show me the blind test where similar projects were run with different levels of info sec engagement and where the outcome was significantly different. Demonstrate how any company’s risk profile has changed since last year. Also, show me an example of security practitioners being ahead of the curve, predicting and preparing for a new development in the field: where were the talks on hacktivism before Anonymous or Wikileaks?

One reason that the same views are repeated over multiple conferences is that the same circuit of speakers travels to all of the conferences. I’m guilty of perpetuating that myself, being (albeit unintentionally in one year) one of the speakers in the iOS circuit. And when I’ve travelled to Seattle or Atlanta or Copenhagen or Aberystwyth, I’ve always recognised at least a few names in the speaker line-up. [While I mentioned Aberystwyth here, both iOSDevUK and NSConf take steps to address the circuit problem. iOSDevUK had a number of first-time speakers and a “bar camp” where people could contribute their own talks. NSConf has the blitz talks which are an accessible way to get a large number of off-circuit speakers, and on one occasion ran a whole day of attendee-contributed sessions called NSConf Mini. When you give people who don’t normally present the opportunity to do so, someone will step up.]

I mentioned before that the echo chamber only permits limited scrutiny, and that comes in the form of the “knowing troll” talk. Indeed at GOTO there’s a track on the final day called “Iconoclasm”, which is populated solely with this form of talk. Where the echo chamber currently resounds to the sound of , it’s permitted to deliver an “ sucks” talk. This will usually present a straw man version of and list its failings or shortcomings. That’s allowed because it actually reinforces – real-world examples are rarely anything like as bad as the straw man version, therefore isn’t really that bad. This form of talk is often a last-session-of-the-day entry and doesn’t really lead people to challenge their beliefs. What happens later is that when everyone moves on to the next big thing, the “ sucks” talks will become the main body of the conference and “<X+1> sucks” will be the new troll talk.

Conferences

Weirdly, while the word conference means a bringing together of people to talk, coming from the same root as “conversation”, many conferences are designed around a one-way flow of words from the speakers to the delegates. Here’s the thing with that. As I said in my keynote talk at MDevcon, we learn from each other by telling and listening to stories. Terry Pratchett, Jack Cohen and Ian Stewart even went as far as to reclassify humans as pan narrans, the storytelling chimpanzee. Now if you’ve got M speakers and 10M<N<100M delegates, then putting a sequence of speakers up and listening to their stories gets you a total of M stories. Letting the N delegates each share their stories, and then letting each of the N-1 other delegates share the stories that the first N stories reminded them of, and so on, would probably lead to a total of N! stories if you had the time to host that. But where that does happen, it’s usually an adjunct to the “big top” show which is the speaker series. [And remember: if you’ve got C conferences, you don’t have C*M speakers, you have M+ε speakers.]

There’s one particular form of wider participation that never works well, and that’s to follow a speaker session with Q&A. Listen carefully to the questions asked at the next Q&A you’re in, and you’ll find that many are not questions, but rhetorical statements crafted to make the “asker” appear knowledgable. Some of those questions that are questions are rhetorical land mines with the intent of putting the speaker on the back foot, again to make the asker seem intellectually talented. Few of these questions will actually be of collective value to the plenus, so there’s not much point in holding the Q&A in front of everyone.

Speaker talks are only one way to run a session, though. Panels, workshops and debates all invite more collaboration than speaker sessions. They’re also much more difficult to moderate and organise, so are rarely seen: many conferences have optional days that are called “workshops” but in reality are short training courses run by an invited speaker. In the iOS development world, lab sessions are escaping the confines of WWDC and being seen at more independent conferences. These are like one-on-one or few-on-few problem solving workshops, which are well focussed and highly collaborative but don’t involve many people (except at Voices that Matter, where they ran the usability workshops on the stage in front of the audience). A related idea being run at GOTO right now, which I need to explore, is a whole track of pair programming sessions. The session host chooses a technology and a problem, and invites delegates onto the stage to work through the challenge with the host in a pair-programming format. That’s a really interesting way to attract wider participation; I’ll wait until I’ve seen it in action before reaching an opinion on whether it works.

There’s another issue, that requires a bit more setup to explain. Here’s a Venn diagram for any industry with a conference scene; the areas are indicative rather than quantitative but they show the relation between:

  • the population of all practitioners;
  • the subsection of that population that attends conferences; and
  • the subsection of that population that speaks at conferences.

Conference Venn diagram

So basically conferences scale really badly. Even once we’ve got past the fact that conferences are geared up to engage the participation of only a handful of their attendees, the next limiting factor is that most people in [whatever industry you’re in] aren’t attending conferences. For the stories told at a conference (in whatever fashion) to have the biggest impact on their industry, they have to break the confines of the conference. This would traditionally, in many conferences, involve either publishing the proceedings (I’ve not heard of this happening in indie tech conferences since the NATO conferences of 1968-9, although Keith Duncan is one of a couple of people to mention to me the more general idea of a peer-reviewed industry journal) or the session videos (which is much more common).

To generate the biggest impact, the stories involved must be inspiring and challenging so that the people who watched them, even those who didn’t attend the conference, feel motivated to reflect on and change the way they work, and to share their experiences (perhaps at the same conference, maybe elsewhere). Before moving on to a summary of everything I’ve said so far, I’ll make one more point about the groups drawn on the Venn diagram. Speakers tend to be specialists (or, as Marcus put it in his NSConf talk, subject matter experts) in one or two fields; that’s not surprising given the amount of research effort that goes into a talk (described above). Additionally, some speakers are asked to conferences because they have published a book on the topic the convenor wishes them to speak on; that’s an even longer project of focussed research. This in itself is a problem, because a lot of the people having difficulty with their work are likely to be neophytes, but apparently we’re not listening to them. We listen to self-selected experts opining on why everyone needs to take security/TDD/whatever seriously and why that involves retaining the experts’ consultancy service: we never listen to the people who can tell us that after a month of trying this Objective-C stuff still doesn’t make sense. These are the people who can give us insight into how to improve our practice, because these are the people reminding the experts (and indeed the journeymen) of the problems they had when they’d been at this for a month. They tell us about the issues everyone has, and give us ideas on how we can fix it for all (future) participants.

Conference goers, then, get the benefit of a small handful of specialists: in other words they have a range of experience to call on (vicariously) that is both broad and deep. Speakers of course have the same opportunity, though don’t always get to take full advantage of the rest of a conference due to preparation, equipment tests, post-talk question sessions and the like. The “non-goers” entry in the diagram represents a vast range of skills and experiences, so it’s hard to find any one thing to say about them. Some will be “distance delegates”, attending every conference by purchasing the videos, transcripts or other materials. Some will absorb information by other means, including meet-ups, books, blogs etc. And some will be lone coders who never interact with anyone in their field. Imagine for a moment that your goal in life is to apply the Boy Scout Rule (which I’m going to attribute again to @ddribin because I can’t remember who he got it from; Uncle Bob probably) to your whole industry. Your impact on $thing_you_do will be to leave the whole field, the whole practice a bit better than it was when you got here. (If that really is your goal, then skip the imagination part for a bit.)

It seems to me that the best people to learn from are the conference delegates (who have seen a wide section of the industry in considerable depth) and the best people to transfer that knowledge to are, well, everybody.

Summary of the current position

Conferences are good. I don’t want people to think I’m hating on conferences. They’re enjoyable events, there are plenty of good ones, there’s an opportunity to learn things, and to see fresh perspectives on many aspects of our industry. They’re also more popular than ever, with new events appearing (and selling out rapidly) every year. However, these perspectives often have an introspective, echo chamber quality. We’re often listening to a small subset of the conference delegates, and if you integrate over multiple conferences you find the subset gets relatively smaller because it’s the same people presenting all the time. Most delegates will not get the benefit of listening to all of the other delegates, which means they’re missing out on engaging with some of the broadest experience in the industry. Most of the practitioners in your corner of the industry probably don’t attend any conferences anyway; there aren’t enough seats for that to work.

The ideal tech conference

OK, I am very clearly lying here: this isn’t the ideal tech conference, it’s my ideal tech conference. In my world, those are the same thing. PerfectConf features a much more diverse portfolio of speakers. In the main this is achieved exactly the way that Appsterdam does it; by offering the chance to speak to anyone who’ll take it, by looking for things that are interesting to hear about rather than accomplished or expert speakers to say it, and by giving novice speakers the chance to train with the experts before they go in front of the stage. Partly this diversity is achieved by allowing people who aren’t comfortable with speaking the opportunity to host a different kind of session, for example a debate or a workshop.

In addition to engaging session hosts who would otherwise be apprehensive about presenting, we get to hear about the successes and tribulations encountered by the whole cohort of delegates. At least one session would be a plenary debate, focussed on a problem that the industry is currently facing. This session has the modest aim of discovering a solution to the problem to move the industry as a whole forward. Another way in which diversity is introduced into the conference is by listening to people outside of our own sector. If infosec is having trouble getting budget for its activities, perhaps they ought to invite more CFOs or comptrollers to its conferences to discuss that. If iPhone app developers find it hard to incorporate concurrency into their application designs, they could do worse than to listen to an Erlang or Occam expert. Above all, the echo chamber would be avoided; session hosts would be asked to challenge the perceived industry status quo.

I’ve long thought that if a talk of mine doesn’t annoy at least one member of the audience then I haven’t said anything useful; a former manager of mine said “if we both think the same way about everything then one of us is redundant”. This way of thinking would be codified into the conference. Essentially, what I’m talking about is the death of the thought leader (or “rock star”). Rather than having one subject matter expert opining on how everyone should think about security, UX, marketing, or whatever, PerfectConf encourages the community to work together like a slime mould, allowing the collective motion of all of the members to explore all opportunities and options and select the best one by communicating freely across the colony.

Slime mold solving a maze (photo: Nature)

Finally, PerfectConf proceedings are published as soon as practical; not just the speaker sessions but the debates too. Where the plenus reaches a consensus, the consensus decision becomes available for all those people who couldn’t make it to the conference of to discover, consider, and potentially adopt or react to. Unfortunately I’m not a conference organiser.

BrowseOverflow as a Code Kata

This article was originally posted over at InformIT.

My goal in writing Test-Driven iOS Development was to take readers from not knowing how to write a test for their iOS apps, to understanding the TDD workflow and how it could work for them. That mirrored the journey that I had taken in learning about test-driven development, and that had led me to wanting to write a book to share what I’d learned with my peers.

This has an interesting effect on the structure of the book. Not all of the sample code from the BrowseOverflow is shown (though it’s all available on GitHub). This isn’t an accident we made in the editorial process. It’s a feature: for any test shown in the book, there are numerous different ways you could write application code that would all be just as good. Anything that causes the test to pass, and doesn’t make the other tests fail, is fine.

Just as the many-worlds model in quantum theory says that there are many branches of the universe that are created every time a decision is made, so there’s a “many-BrowseOverflows” model of Test-Driven iOS Development. Every time a test is written, there are many different solutions to passing the test, leading to there being multiple potential BrowseOverflows. The code that you see on GitHub is just one possible BrowseOverflow, but any other code that satisfies the same tests is one of the other possible BrowseOverflows.

This means that you can treat the book like a kata: the Japanese martial art technique of improving a practice by repeating it over and over. The first time you read Test-Driven iOS Development, you can choose to follow the example code very closely. Where the code isn’t given in the book, you might choose to look at the source code to understand how I solved the problems posed by the tests. But the end of the book is not the end of the journey: you can go back, taking the tests but implementing all of the app code yourself. You can do this as many times as you want, trying to find new ways to write code that produces a BrowseOverflow app.

Finally, when you’re more confident with the red-green-refactor way of working, you can write a BrowseOverflow app that’s entirely your own creation. You can define what the app should do, create tests that express those requirements and write the code to implement it however you like. This is a great way to test out new ways of working, for example different test frameworks like GHUnit or CATCH. It also lets you try out different ways of writing the application code: you could write the same app but trying to use more blocks, or try to use the smallest number of properties in any class, or any other challenge you want to set yourself. Because you know what the software should be capable of, you’re free to focus on whatever skill you’re trying to exercise.

Using GNUstep libraries with Xcode

I was recently asked about building projects that use GNUstep from Xcode. The fact is, it’s incredibly easy.

By default, GNUstep on Mac OS X installs its libraries to /usr/local/lib and its frameworks to /Library/Frameworks. Therefore if you want to include GNUstep-base additions, you just hit the + button in your target’s “link binary with libraries” section and find the libgnustep-baseadd.dylib entry under Mac OS X 10.7. If you wanted to use GNUstepWeb, you’d look for WebObjects.framework in the same list.

You can get access to the GNUstep base additions in your code by including <GNUstepBase/GNUStep.h.

Notice that if you manage a GNUstep project using Xcode, you’ll only be able to build it on a Mac (unless you go to the bother of writing a build tool with GNUstep’s unfortunately-named XCode.framework – which ironically doesn’t currently work on Mac OS X.). If you need to target non-Mac platforms, your options are to build Cocotron-style cross-compilers and add them to your Xcode project, or to create an Xcode project with an External Build System target and manage your build via make anyway.

An interesting extension would be to define a new filesystem layout for GNUstep-make that deposited all of the frameworks and libraries into a .sdk folder that could be used in Xcode as an additional SDK.

Building a unit test target with GNUstep make

Just a quick note on how I build my test tools (they run separately, either by manual invocation or via CI) when I’m working in GNUstep.

Firstly, you’ll need Catch. Then given test files that look like this:

test_class.mm
#define CATCH_CONFIG_MAIN
#include "catch.hpp"
#import 

TEST_CASE("Using foundation", "I should be able to use Foundation classes from a test tool")
{
  NSArray *array = [NSArray array];
  REQUIRE([array count] == 0);
}

(only one of your files should define CATCH_CONFIG_MAIN as you must only have one main function).

Then I define a tests target in my Makefile:

GNUmakefile

include $(GNUSTEP_MAKEFILES)/common.make

#… stuff to define my app target

TEST_TOOL_NAME=tests
tests_OBJCC_FILES = test_class.mm other_tests.mm blah.mm
tests_OBJCCFLAGS="-ICatch/include"

-include Makefile.preamble

#… include target definitions for the app target
include $(GNUSTEP_MAKEFILES)/test-tool.make

-include Makefile.postamble

And there it is. Now whenever I make my app I also get obj/tests which’ll run the test suite for me.

So you’re off to WWDC

Jeff has updated his excellent WWDC first-timer’s guide, and I thought I’d augment that with some things I’ve noticed.

  • The easiest and cheapest way to get from SFO to San Francisco is via BART. Once you get to the arrivals hall, the BART station is well signposted. If you follow Jeff’s advice and stay within a few blocks of the Moscone, you’ll probably want Powell Street BART, or maybe Civic Center. It’s only a few dollars on the BART but you might want to buy a $20 or so ticket so you can get around for the rest of the time you’re in town. There are other options: shuttle buses, taxis and limousines, which all cost more.
  • The character of the city changes markedly west of UN Plaza. I’ve stayed in hotels on 7th street, and felt uneasy in that block and further over on my own in the evenings – I’ve never been physically attacked, but in 2005 I was chased by panhandlers who didn’t like the fact that I didn’t give them any money[*]. The hotels are almost an order of magnitude cheaper than those in the financial district, but there’s a reason for it.
  • There are plenty of options for good food. The Moscone isn’t one of them. If you want to make your own packed lunch you’ll find a Whole Foods and a Safeway on 4th Street (get to the Moscone, then head away from Market Street), and I think I’ve seen a farmers market somewhere near Yerba Buena park. The Oasis grill across the road from the Moscone’s main entrance is good, and Mel’s Drive-In a block towards Market Street serves a fine breakfast. Local people can often recommend somewhere further afield, too.
  • Sleep and fresh water are both welcome commodities. Yes there will be parties on every night: don’t feel you have to go to all the events all night, or that you have to drink booze like some kind of alcoholic guppy at each party. Making time to eat, sleep and top up on fluids will keep you healthy and profitable: dead programmers don’t ship.

[*]Some US airports, and IIRC SFO is one of them, have official charity-staffed information desks. The people there have ID and are very helpful, so I usually donate to a homeless shelter there. San Francisco has a very real and visible homeless problem, with all sorts of political, social and geological causes.

Metacognition-driven development

To find out what techniques work for you in a field of practice, you often need to think about how you think. To decide what it is that drives your learning processes, and then adapt your practices to suit that.

For example, I tend to get more done if I’m already getting something done. Success breeds success. If I don’t feel like I’m succeeding, then I’ll break off and do something else (like write a blog post in my lunch break). I don’t think that’s unique to me, though, as common phrases like “on a roll” and “in the zone” seem to imply being in a state where progress begets more progress.

Cognition scientists know this, too. The authors of Yes! describe an experiment in which they tried two types of card in a hotel to get guests to reuse their towel instead of having them laundered every day. One contained the message that if a guest reused her towel, the hotel would make a donation to an environmental charity on her behalf. The other said that the hotel had already made the donation, and all the guest had to do to complete the story was to reuse her towel.

The second card was more effective. It implies that progress has already been made: that the guest has already succeeded (albeit vicariously) in making a donation. Success begets success, so the guests are more likely to make continued progress: i.e. to reuse the towels.

That is, I think, why TDD makes me work faster than when I’m not doing TDD. Because I can see the green lights, because I can see the little nuggets of success that will go together to make the feature work. It turns “I still haven’t posted the message to the server” into “woo! I just configured the URL request”. It turns “still losing” into “yet another win”. It doesn’t trick me into thinking I’m making more progress, but it does let me see that progress is being made.

Messing about with Clang

I’ve been reading the Smalltalk-80 blue book (pdf) recently, and started to wonder what a Smalltalk style object browser for Objective-C would look like. Not just from the perspective of presenting the information that makes up Objective-C classes in novel ways (though this is something I’ve discussed with Saul Mora at great length in the past). What would an object browser in which the compiler is an object, so you can define and manipulate classes in real time, look like?

Well, the first thing you’d need to do is to turn the compiler into an object. I decided to see whether I could see what the compiler sees, using the clang compiler front-end library.

Wait, clang library? Clang’s a command-line tool, isn’t it? Well yes, but it and the entire of LLVM are implemented as a collection of reusable C++ classes. Clang then has a stable C interface wrapping the C++, and this is what I used to produce this browser app. This isn’t the browser I intend to write, this is the one I threw away to learn about the technology.

Objective-Browser

Clang is a stream parser, and there are two ways to deal with source files just like any other stream: event-driven[*], in which you let the parser go and get callbacks from it when it sees interesting things, or document-based[*] where you let the parser build up a document object model (a tree, in this case) which you then visit the nodes of to learn about the data.

[*] Computer scientists probably call these things something else.

Being perverse, I’m going to use the event-driven parser to build a parallel data model in Objective-C. First, I need to adapt the clang library to Objective-C, so that the compiler is an Objective-C object. Here’s my parser interface:

#import <Foundation/Foundation.h>

@protocol FZAClassParserDelegate;

@interface FZAClassParser : NSObject

@property (weak, nonatomic) id <FZAClassParserDelegate>delegate;

- (id)initWithSourceFile: (NSString *)implementation;
- (void)parse;

@end

The -parse method is the one that’s interesting (I presume…) so we’ll dive into that. It actually farms the real work out to an operation queue:

#import <clang-c/Index.h>

//...

- (void)parse {
    __weak id parser = self;
    [queue addOperationWithBlock: ^{ [parser realParse]; }];
}

- (void)realParse {
#pragma warning Pass errors back to the app
    @autoreleasepool {
        CXIndex index = clang_createIndex(1, 1);
        if (!index) {
            NSLog(@"fail: couldn't create index");
            return;
        }
        CXTranslationUnit translationUnit = clang_parseTranslationUnit(index, [sourceFile fileSystemRepresentation], NULL, 0, NULL, 0, CXTranslationUnit_None);
        if (!translationUnit) {
            NSLog(@"fail: couldn't parse translation unit);
            return;
        }
        CXIndexAction action = clang_IndexAction_create(index);

That’s the setup code, which gets clang ready to start reading through the file. Which is done in this function:

        int indexResult = clang_indexTranslationUnit(action,
                                                     (__bridge CXClientData)self,
                                                     &indexerCallbacks,
                                                     sizeof(indexerCallbacks),
                                                     CXIndexOpt_SuppressWarnings,
                                                     translationUnit);

This is the important part. Being a C callback API, clang takes a context pointer which is the second argument: in this case, the parser object. It also takes a collection of callback pointers, which I’ll show next after just showing that the objects created in this method need cleaning up.

        clang_IndexAction_dispose(action);
        clang_disposeTranslationUnit(translationUnit);
        clang_disposeIndex(index);
        (void) indexResult;
    }
}

There’s a structure called IndexCallbacks defined in Index.h, this class’s structure contains functions that call through to methods on the parser’s delegate:

int abortQuery(CXClientData client_data, void *reserved);
void diagnostic(CXClientData client_data,
                CXDiagnosticSet diagnostic_set, void *reserved);
CXIdxClientFile enteredMainFile(CXClientData client_data,
                                CXFile mainFile, void *reserved);
CXIdxClientFile ppIncludedFile(CXClientData client_data,
                               const CXIdxIncludedFileInfo *included_file);
CXIdxClientASTFile importedASTFile(CXClientData client_data,
                                   const CXIdxImportedASTFileInfo *imported_ast);
CXIdxClientContainer startedTranslationUnit(CXClientData client_data,
                                            void *reserved);
void indexDeclaration(CXClientData client_data,
                      const CXIdxDeclInfo *declaration);
void indexEntityReference(CXClientData client_data,
                          const CXIdxEntityRefInfo *entity_reference);

static IndexerCallbacks indexerCallbacks = {
    .abortQuery = abortQuery,
    .diagnostic = diagnostic,
    .enteredMainFile = enteredMainFile,
    .ppIncludedFile = ppIncludedFile,
    .importedASTFile = importedASTFile,
    .startedTranslationUnit = startedTranslationUnit,
    .indexDeclaration = indexDeclaration,
    .indexEntityReference = indexEntityReference
};

int abortQuery(CXClientData client_data, void *reserved) {
    @autoreleasepool {
        FZAClassParser *parser = (__bridge FZAClassParser *)client_data;
        if ([parser.delegate respondsToSelector: @selector(classParserShouldAbort:)]) {
            return [parser.delegate classParserShouldAbort: parser];
        }
        return 0;
    }
}

// …

Internally clang creates its own threads, so the callback functions wrap delegate messages in @autoreleasepool so that the delegate doesn’t have to worry about this.

The delegate still needs to understand clang data structures of course, this is where the real work is done. Here’s the delegate that’s used to build the data model used in the browser app:

#import <Foundation/Foundation.h>
#import "FZAClassParserDelegate.h"

@class FZAClassGroup;

@interface FZAModelBuildingParserDelegate : NSObject <FZAClassParserDelegate>

- (id)initWithClassGroup: (FZAClassGroup *)classGroup;

@end

The FZAClassGroup class is just somewhere to put all the data collected by parsing the file: in a real IDE, this might represent a project, a translation unit, a framework or something else. Anyway, it has a collection of classes. The parser adds classes to that collection, and methods and properties to those classes:

@implementation FZAModelBuildingParserDelegate {
    FZAClassGroup *group;
    FZAClassDefinition *currentClass;
}

- (id)initWithClassGroup:(FZAClassGroup *)classGroup {
    if ((self = [super init])) {
        group = classGroup;
    }
    return self;
}

- (void)classParser:(FZAClassParser *)parser foundDeclaration:(CXIdxDeclInfo const *)declaration {
    const char * const name = declaration->entityInfo->name;
    if (name == NULL) return; //not much we could do anyway.
    NSString *declarationName = [NSString stringWithUTF8String: name];

We’ve now got a named declaration, but a declaration of what?

    switch (declaration->entityInfo->kind) {
        case CXIdxEntity_ObjCProtocol:
        {
            currentClass = nil;
            break;
        }
        case CXIdxEntity_ObjCCategory:
        {
            const CXIdxObjCCategoryDeclInfo *categoryInfo = 
            clang_index_getObjCCategoryDeclInfo(declaration);
            NSString *className = [NSString stringWithUTF8String: categoryInfo->objcClass->name];
            FZAClassDefinition *classDefinition =[group classNamed: className];
            if (!classDefinition) {
                classDefinition = [[FZAClassDefinition alloc] init];
                classDefinition.name = className;
                [group insertObject: classDefinition inClassesAtIndex: [group countOfClasses]];
            }
            currentClass = classDefinition;
            break;
        }
        case CXIdxEntity_ObjCClass:
        {
            FZAClassDefinition *classDefinition =[group classNamed: declarationName];
            if (!classDefinition) {
                classDefinition = [[FZAClassDefinition alloc] init];
                classDefinition.name = declarationName;
                [group insertObject: classDefinition inClassesAtIndex: [group countOfClasses]];
            }
            currentClass = classDefinition;
            break;
        }

I’m ignoring protocols, but recognising that methods declared in a protocol shouldn’t go onto any particular class. Similarly, I’m adding methods found in categories to the class on which that category is defined: real Smalltalk browsers keep the categories, but for this prototype I decided to skip them. I’m using the fact that this is a prototype to justify having left the duplicate code in place, above :-S.

So now we know what class we’re looking at, we can start looking for methods or properties defined on that class:

        case CXIdxEntity_ObjCClassMethod:
        case CXIdxEntity_ObjCInstanceMethod:
        {
            FZAMethodDefinition *method = [[FZAMethodDefinition alloc] init];
            method.selector = declarationName;
            if (declaration->entityInfo->kind == CXIdxEntity_ObjCClassMethod)
                method.type = FZAMethodClass;
            else
                method.type = FZAMethodInstance;
            [currentClass insertObject: method inMethodsAtIndex: [currentClass countOfMethods]];
            break;
        }
        case CXIdxEntity_ObjCProperty:
        {
            FZAPropertyDefinition *property = [[FZAPropertyDefinition alloc] init];
            property.title = declarationName;
            [currentClass insertObject: property inPropertiesAtIndex: [currentClass countOfProperties]];
            break;
        }
        default:
            break;
    }
}

And that’s “it”. The result of collecting all of these callbacks is a tree:

ClassGroup -> Class -> [Method, Property]

I define a tree-ish interface for all of these classes, by adding categories that define the same methods:

@interface FZAMethodDefinition (TreeSupport)

- (NSInteger)countOfChildren;
- (NSString *)name;
- (id)childAtIndex: (NSInteger)index;
- (BOOL)isExpandable;

@end

@implementation FZAMethodDefinition (TreeSupport)

- (NSInteger)countOfChildren {
    return 0;
}

- (BOOL)isExpandable {
    return NO;
}

- (id)childAtIndex:(NSInteger)index {
    return nil;
}

- (NSString *)name {
    switch (self.type) {
        case FZAMethodClass:
            return [@"+" stringByAppendingString: self.selector];
            break;
        case FZAMethodInstance:
            return [@"-" stringByAppendingString: self.selector];
            break;
        default:
            return [@"?" stringByAppendingString: self.selector];
            break;
    }
}

@end

And, well, that’s it. libClang could be the kernel of a thousand visualizers, browsers and editors for C-derived languages, the start of one is outlined above.