Security consultancy from the other side

I used to run an application security consultancy business, back before the kinds of businesses who knew they needed to consider application security had got past assessing creating mobile apps. Whoops!

Something that occasionally, nay, often happened was that clients would get frustrated if I didn’t give them a direct answer to a question they asked, or if the answer to an apparently simple question was a one-day workshop. It certainly seems sketchy that someone who charges a day rate would rather engage in another day’s work than write a quick email, doesn’t it?

Today I was on the receiving end of this interaction. I’m defining the data encryption standards for our applications, so talked to our tame infosecnician about the problem. Of course, rather than specific recommendations about protocols, modes of operation, key lengths, rotation frequencies etc he would only answer my questions with other questions.

And this is totally normal and OK. The reason that I want to encrypt (some of) this data is that its confidentiality and integrity are valuable to me (my employer) and it’s worth investing some of my (our) time in protecting those attributes. How much is it worth? How much risk am I willing to leave on the table? How long should the data be protected for? Who do I trust with the data, and how much do I trust them?

Only I can answer those questions, so it’s not up to someone else to answer them for me, but they can remind me that I need to ask them.

On explaining stuff to people

An article that recently made the rounds, though it was written back in September, is called Apple’s Idioten Vektor. It’s a discussion of how the CCCrypt() function in Apple’s CommonCrypto library, when used in its default cipher block chaining mode, treats the IV (Initialization Vector) parameter as optional. If you don’t supply an IV, it provides its own IV of 0x0.

Professional Cocoa Application Security also covers CommonCrypto, CBC mode, and the Initialization Vector. Pages 79-88 discuss block encryption. The section includes sample code for both one-shot and staged use of the API. It explains how to set the IV using a random number generator, and why this should be done.[1] Mercifully when the author of the above blog post reviewed the code in my book section, he decided I was doing it correctly.

So both publications cover the same content. There’s a clear difference in presentation technique, though. I realise that the blog post is categorised as a “rant” by the author, and that I’m about to be the pot that calls the kettle black. However, I do not believe that the attitude taken in the post—I won’t describe it, you can read it—is constructive. Calling people out is not cool, helping them get things correct is. Laughing at the “fail” is not something that endears people to us, and let’s face it, security people could definitely be more endearing. We have a difficult challenge: we ask developers to do more work to bring their products to market, to spend more money on engineering (and often consultants), in return for potentially protecting some unquantified future lost revenue and customer hardship.

Yes there is a large technical component in doing that stuff, but solving the above challenge also depends very strongly on relationship management. Security experts need to demonstrate that we’re all on the same side; that we want to work with the rest of the software industry to help make better software. Again, a challenge arises: a lot of the help provided by security engineers comes in the form of pointing out mistakes. But we shouldn’t be self promoting douchebags about it. Perhaps we’re going about it wrong. I always strive to help the developers I work with by identifying and discussing the potential mistakes before they happen. Then there’s less friction: “we’re going to do this right” is a much more palatable story than “you did this wrong”.

On the other hand, the Idioten Vektor approach generated a load of discussion and coverage, while only a couple of thousand people ever read Professional Cocoa Application Security. So there’s clearly something in the sensationalist approach too. Perhaps it’s me that doesn’t get it.

[1]Note that the book was written while iPhone OS 3 was the current version, which is why the file protection options are not discussed. If I were covering the same topic today I would recommend eschewing CCCrypto for all but the most specialised of purposes, and would suggest setting an appropriate file protection level instead. The book also didn’t put encryption into the broader context of cryptographic protocols; a mistake I have since rectified.

On the top 5 iOS appsec issues

Nearly 13 months ago, the Intrepidus Group published their top 5 iPhone application development security issues. Two of them are valid issues, the other three they should perhaps have thought longer over.

The good

Sensitive data unprotected at rest

Secure communications to servers

Yes, indeed, if you’re storing data on a losable device then you need to protect the data from being lost, and if you’re retrieving that data from elsewhere then you need to ensure you don’t give it away while you’re transporting it.

Something I see a bit too often is people turning off SSL certificate validation while they’re dealing with their test servers, and forgetting to turn it on in production.

The bad

Buffer overflows and other C programming issues

While you can indeed crash an app this way, I’ve yet to see evidence you can exploit an iOS app through any old buffer overflow due to the stack guards, restrictive sandboxes, address-space layout randomisation and other mitigations. While there are occasional targeted attacks, I would have preferred if they’d been specific about which problems they think exist and what devs can do to address them.

Patching your application

Erm, no. Just get it right. If there are fast-moving parts that need to change frequently, extract them from the app and put them in a hosted component.

The platform itself

To quote Scott Pack in “The DMZ”, If you can’t trust your users to implement your security plan, then your security plan must work without their involvement. In other words, if you have a problem and the answer is to train 110 million people, then you have two problems.

What happens when you jailbreak an iPad

Having played around with an iPad running a jailbreak OS yesterday, I thought it would be useful to explain one possible attack that doesn’t seem to get much coverage.

As I’ve discussed in numerous talks, the data protection feature of iOS (introduced in iOS 4, enabled by setting the NSFileProtectionComplete option on a file or writing data with the NSDataWritingFileProtectionComplete option) only works fully when the user has a passcode lock enabled. The operating system can derive a key to protect the files (indirectly, but that’s another talk) from the passcode, so when the device is locked the files are really inaccessible because the device has no idea what the unlock key is.

This can be seen when you try and access the content via SSH. Of course, the SSH daemon must be installed on a jailbreak operating system, but you don’t need the passcode to jailbreak:

$ ssh -l root@
[key/auth exchange...the default password is still 'alpine']
# cd /User/Applications/C393CDBF-1A82-4D7B-A064-D6DFB8CC20DB/Documents
# cat UnprotectedFile
The flag. You haz it.
# cat ProtectedFile
cat: Error: Operation not permitted
# cat ProtectedFile
The flag. You haz it.

Now of course you do need physical access to jailbreak, but that doesn’t take particularly long. So here’s a situation that should probably appear in your threat models:

  1. Attacker retrieves target’s iPad
  2. Attacker installs jailbreak OS with data-harvesting tools
  3. Attacker returns iPad to the target
  4. Target uses iPad

Of course, an attacker who simply tea leafs the target’s iPad can’t perform this attack, and won’t be able to retrieve the files.

On the broken(?) Mac App Store

A day after the Mac App Store was launched, people are reporting that it has been cracked. There are two separate stories here, a vapourware circumvention of the FairPlay DRM used to generate the receipts and a report that certain apps aren’t validating the receipts properly. We can ignore the first case for the moment: it’s important, and if it’s true then Apple needs to fix it (and co-ordinate updating the validation code with us third-party developers). But for the moment, it’s more important that developers are implementing the protections that are in place in their applications – it’s those applications that are supposed to be protected.

Let’s skip, for the moment, the question of whether DRM or anti-cracking mechanisms are ethically right, worthwhile, or how much effort you want to put into them. Apple have done most of the legwork, in providing a vendor-signed receipt that’s part of your signed app bundle. What you need to do is:

  • Check whether you have a receipt
  • Check whether Apple signed the receipt you have
  • Check whether the receipt is valid for your product
  • Check whether the receipt is valid for this version of your product
  • Check whether the receipt is valid for this computer

That’s it in a nutshell. Of course, some nutshells surround very big and complex nuts, and that’s true in this case:

  • There’s some good example code for receipt validation at github/roddi/ValidateStoreReceipt, if you’re going to use it then don’t just paste it wholesale. If everyone uses the same code then it’s super-easy for someone to detect and strip that code from each instance.
  • Check at runtime, as well as before startup. If you just check at startup, then all an attacker needs to do is patch main() to jump straight into NSApplicationMain() and your app runs for free.
  • Code obfuscation is not a very effective tool. Having worked in anti-virus, I know it’s much easier to classify code based on what it does than what it is, and it’s quite easy to find the code that opens the receipt file, or calls exit(173). That said, some of the commercial obfuscation companies offer a guaranteed service, so you can still protect your revenue after the app gets cracked.
  • Update I have been advised privately and seen in a blog post that people are recommending hard-coding their app bundle IDs and version numbers into the binary rather than using Info.plist, because that file can be edited. Well, so can the app binary…and in either case you’d need to re-sign the product with a valid certificate to continue, because Apple have used the kill flag:
    heimdall:~ leeg$ codesign -dvvvv /Applications/
    Format=bundle with Mach-O universal (i386 x86_64)
    CodeDirectory v=20100 size=12452 flags=0x200(kill) hashes=616+3 location=embedded
    Signature size=4169
    Authority=Apple Mac OS Application Signing
    Authority=Apple Worldwide Developer Relations Certification Authority
    Authority=Apple Root CA
    Info.plist entries=21
    Sealed Resources rules=4 files=78
    Internal requirements count=2 size=344

    Changing a hard-coded string in a binary file is not difficult. You can of course obfuscate the string, but the motivated cracker still finds the point where the comparison is made (particularly easily if you use NSStrings). Really, how far you want to go depends on how much you’re willing to spend.

Of course, Fuzzy Aliens Ltd has already been implementing receipt validation for customers, so if this is too hard for you or you don’t have the time… 😉

A last word on publicising receipt-validation vulnerabilities

You and I both make our living by selling software, or by selling services to people who sell software. Crowing on the interwebs about how this application or that application doesn’t validate its receipts properly is not cool, because you are shitting on your own doorstep. There is no public benefit to public disclosure that class of vulnerability, because DRM is not a user security feature. Don’t do that. Send the developer a private message explaining your findings. Give them a chance to put extra effort into protecting their product, if that’s what they want to do.

Protecting source code

As I mentioned on the missing Live episode, one of the consequences of the Gawker hack was that their source code for their internal software was leaked into the Internet. I doubt any of my readers would want that to happen to their code, so I’m going to share the details of how I protect my clients’ code when I’m working. Maybe some of this will work for you.

In the office, I work at a desktop iMac. This has an external time machine backup disk and a DropBox for off-site storage. However, client code does _not_ go onto the DropBox. Instead I keep a separate, encrypted sparse disk image for each project I’m working on. The password for each is different. As well as protecting against snooping, this helps stop cross-contamination. I rarely have two such images mounted at once. Note that it’s not just source that goes into these images: build products, notes, Instruments traces, and images all go into the encrypted containers.

Obviously that means a lot of passwords, and no I can’t remember them all. I use a keychain. It locks automatically when not in use, and has a passphrase that’s different from my login passphrase.

The devices I test on are all encrypted where available (if a client needs me to test on an iPhone 3G, then I can, but it isn’t encrypted). They are passphrase locked, set to require passphrase immediately. And I NEVER take them away from the desk before deleting any developer builds, unless I need to do something special like a real-world location services test.

I rarely do coding work on the laptop, but when I do I copy the appropriate encrypted image onto it. The laptop additionally has FileVault configured, though I’m evaluating full-disk encryption options. Keychain configuration as above, additionally with a password required on wake from sleep or screensaver, and a firmware password.

For pushing work back to the clients, most clients use github or bitbucket which offer SSL-encrypted connections to the repositories. Personally, I have a self-run repo host available over HTTPS or SSH, but will probably move that to a github-like service because life’s too short. Their security policy seems acceptable to me.

On the Mac App Store

I’ve just come off iDeveloper.TV Live with Scotty and John, where we were talking about the Mac app store. I had some material prepared about the security side of the app store that we didn’t get on to – here’s a quick write up.

There’s a lot of discussion on twitter and the macsb mailing list, and doubtless elsewhere, about the encryption paperwork that Apple are making us fill in. It’s not Apple’s fault, it’s the U.S. Department of Commerce. You see, back in the cold war (and, frankly, ever since) the government have been of the opinion that encryption is a weapon (because it hides data from their agents) and so are powerful computers (because they can do encryption that’s expensive to crack). So the Bureau of Industry and Security developed the Export Administration Regulations to control the flow of such heinous weapons through the commercial sector.

Section 5, part 2 covers computer equipment and software. Specific provision is made for encryption, in the documentation we find that Items may be controlled as encryption items even if the encryption is actually performed by the operating system, an external library, a third-party product or a cryptographic processor. If an item uses encryption functionality, whether or not the code that performs the encryption is included with the item, then BIS evaluates the item based on the encryption functionality it uses.

So there you go. If you’re exporting software from the U.S. (and you are, if you’re selling via Apple’s app store) then you need to fill in the export notification.

Other Mac App Store security things, in “oh God is it that late already” format:

  • Receipt validation. No different really from existing licensing frameworks. All you can do is make it hard to find the tests from the binary. I had an idea about a specific way to do that, but want to test it before I release it. As you’ve no doubt found, anti-cracking measures aren’t easy.
  • Users. The user base for the MAS will be wider, and less tech-savvy, than the users existing micro-ISVs are selling to. Make sure your intent with regard to user data, particularly the consequences of your app’s workflow, are clear.
  • Similarly, be clear about the content of updates. Clearer than Apple are: “contains various fixes and improvements” WTF?
  • As we’ve found with the iOS store, it’s harder to push an update out than when you host the download yourself. Getting security right (or, pragmatically, not too wrong) the first time avoids emergency update submissions.
  • Permissions. Your app needs to run entirely as the current user, who may not be an admin. If you’re a developer, you’re probably running as an admin. Test with a non-admin account. Better, do all of your development in a non-admin account. Add yourself to the _developer group so you can still use gdb/Instruments properly.

On how to get crypto wrong

I’ve said time and time again: don’t write your own encryption algorithm. Once you’ve chosen an existing algorithm, don’t write your own implementation.

Today I had to look at an encryption library that had been developed to store some files in an app. The library used a custom implementation of SHA256-HMAC, and a custom implementation of CBC mode. The implementations certainly looked OK, and seemed to match the descriptions in the textbooks. They also seem to work – you can encrypt a file to get gibberish, and decrypt the gibberish to get the file back.

So the first thing I did was to crack open Xcode and replace these custom functions with CommonCrypto. CommonCrypto’s internals also look a lot like the textbook descriptions of the methods, too. So it would be surprising if these two approaches yielded different results.

These two approaches yielded different results. This was surprising. Specifically, I found that the CBC implementation would sometimes use junk memory, which the CommonCrypto version never does. Of course, the way in which this junk was used was predictable enough that the encryption routine was still reversible – but could it be that the custom implementation was leaking information about the plaintext in the cipher-text by inappropriate re-use of the buffer? Possibly, and that’s good enough for me to throw the custom implementation out. Proving whether or not this implementation is “safe” is something that a specialist cryptographer could probably do in half a day. However, as I was able to use half a day to produce something I had more confidence in, just by using a tested implementation, I decided there was no need to do that work.

Why passwords aren’t always the right answer.

I realised something yesterday. I don’t know my master password.

Users of Mac OS X can use FileVault, a data protection feature that replaces the user’s home folder with an encrypted disk image. Encrypted disk images are protected by AES-128 or AES-256 encryption, but to get at the private key you need to supply one of two pieces of information. The first is the user’s login password, and the second is a private key for a recovery certificate. That private key is stored in a dedicated keychain, which is itself protected by….the master password. More information on the mechanism is available both in Professional Cocoa Application Security and Enterprise Mac.

Anyway, so this password is very useful – any FileVault-enabled home folder can be opened by the holder of the master password. Even if the user has forgotten his login password, has left the company or is being awkward, you can get at the encrypted content. It’s also hardly ever used. In fact, I’ve never used my own master password since I set it – and as a consequence have forgotten it.

There are a few different ways for users to recall passwords – by recital, by muscle memory or by revision. So when you enter the password, you either remember what the characters in the password are, where your hands need to be to type it or you look at the piece of paper/keychain where you wrote it down. Discounting the revision option (the keychain is off the menu, because if you forget your login password you can’t decrypt your login keychain in order to view the recorded password), the only ways to reinforce a password in your memory are to use it. And you never use the FileVault master password.

I submit that as a rarely-used authentication step, the choice of a password to protect FileVault recovery is a particularly bad one. Of course you don’t want attackers able to use the recovery mechanism, but you do want that when you really need to recover your encrypted data, the OS doesn’t keep you out, too.

Integrating SSH with the keychain on Snow Leopard

Not much movement has occurred on projects like or in the last couple of years. The reason is that it’s not necessary to use them these days; you can get all of the convenience of keychain-stored SSH passphrases using the built in software. Here’s a guide to using the Keychain to store your pass phrases.

Create the key pair

We’ll use the default key format, which is RSA for SSH protocol 2.0. We definitely want to enter a passphrase, so that if the private key is leaked it cannot be used.

jormungand:~ leeg$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/leeg/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /Users/leeg/.ssh/id_rsa.
Your public key has been saved in /Users/leeg/.ssh/
The key fingerprint is:
ff:ce:0a:f6:ee:0d:e8:a5:aa:56:a0:f3:0b:81:80:cc leeg@jormungand.local
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|+                |
|oE               |
|..  .            |
|. .. .  S        |
|  o.  .  o       |
|  .o .  + +      |
|   .o  o = =     |
|   .oo..oo=o=    |

If you haven’t seen it before, randomart is not a screenshot from nethack; rather it’s a visual hashing algorithm. Two different public keys are not guaranteed to have different randomart fingerprints, but the chance that they are close enough to pass a quick visual inspection is small.

Deploy the public key to the SSH server

Do this using any available route; I choose to use password-based SSH.

jormungand:~ leeg$ ssh heimdall.local 'cat -  >> .ssh/authorized_keys' < .ssh/

There is no step three

Mac OS X automatically runs ssh-agent, the key-caching service, as a launchd agent. When SSH attempts to negotiate authentication using your key-based identity, you automatically get asked whether you want to store the passphrase in the keychain. Like this:


Now the passphrase is stored in the keychain. Don't believe me? Lookie here:


So your SSH key is protected by the passphrase, and the passphrase is protected by the keychain.

Update: Minor caveat

Of course (and I forgot this :-S), if you use FileVault to protect the home folder on the SSH server, the user's home folder isn't mounted until after you authenticate. This means that the authorized_keys file can't be consulted during negotiation. Once you have logged in once (using your password), subsequent logins will use the keys (as PAM automatically mounts the FileVault volumes the first time, so authorized_keys becomes available).