You heard it here first folks, the killer app for the iPhone 3GS will be augmented reality. Two features in the new release makes the iPhone 3GS the perfect augmented reality platform. First, the inclusion of an onboard compass and second, the opening up of the API to full camera controls.

Almost everyone except developers were unaware of this but the iPhone 3G SDK painfully crippled the camera by forcing you to use the provided Apple API to take pictures (this meant no real time computer vision could run on it except when Apple conveniently didn’t notice you bypassing their allowed APIs). Now that the APIs are open, expect to see a bunch of innovative computer vision apps (face detection, object detection, tracking).

But the second breakthrough is that the compass finally provides a braindead 6 Degree of Freedom (DOF) estimation. Any rigid object in space can be defined by 6 parameters. 3 spatial ones and 3 rotational ones. The iPhone 3G could estimate 5 of those 6 parameters acceptable well with GPS providing rough spatial data, the acceleromter providing fine spatial data as well as the direction of gravity. The compass provides the final, missing degree of freedom that allows for complete pose estimation.

Augmented Reality needs to know two things: Where is the sensor and where is everything else. Once you know that, you can do all sorts of really cool shit. I have no idea what will eventually come out, that’s one of the exciting things about bringing this technology to a mainstream audience, but I can point to some interesting research directions that seem plausible:

  • Instant 3D modelling of everything: Wave your iPhone around an object and it’ll figure out how to create a crude 3D model of it in memory.
  • Interactive furniture arranging: Go to the store, scan a bunch of furniture you want to buy, go home, drag and drop virtual pieces of furniture in your living room to figure out which piece should go where.
  • Interactive tour guide: You’re walking through New York, you see a cool building, take a photo of it and all of a sudden you know it’s the flatiron building and you’re reading a wikipedia article about it.
  • Photosynth the entire damn world: nuff said.
  • AR Quake: nuff said.
  • History view: Point to a space and if there’s a security camera pointed at it, be able to review what happened at that spot at any point in time.
  • Invisible Ink: Leave messages on walls which only your friends can see. Send them on an easter egg hunt.
  • Virtual Ping Pong on the phone: Use the phone screen to see a virtual ping pong table and then swing the phone to make a hit
  • Physical, virtual avatar conferencing: Replicate the real world cocktail party acoustics in a virtual physical space. Cocktail parties are great because they allow the spontaneous formation of ad-hoc small conversation clusters within a larger conversation. Online tools do a poor job of replicating that dynamic but if we could bring physicality back into it, we might be able to bring some of this dynamism back into online conversation.
  • A million goddamn screens: This was a project I personally worked on that unfortunately, never got to far but has a dear place in my heart. Conventional computing is predicated on screens being expensive but if you stick a tracking marker onto a piece of paper, you can turn it into a screen. What would computing be like if you could produce screens out of some cardboard and a laser printer that could be any form factor and would cost 10 cents a pop? How would this help with information overload? Imagine you have a screen in the corner that represents the pile of unread emails. If you want to read an email, drag it onto a new screen. Important emails that you need to reply to each get their own seperate screen which you keep neat and tidy by arranging them in a pile. If you want to send a file to your coworker sitting next to you, you can drag it onto a screen and then physically hand him the screen. You can have a screen for each individual participant in an IM/voice/video conferencing and pulling the screen closer means you want to be alerted but pushing it away means you want to ignore them. What could you do if you had access to a million goddamn screens over the course of your lifetime?

Are some of these examples wildly unrealistic and totally unable to work in real life? Of course, they come from research inspirations. But they demonstrate the enormous power of augmented reality which is about to be unlocked within these next few years as developers grapple with just what’s possible with Augemented Reality.