You heard it here first folks, the killer app for the iPhone 3GS will be augmented reality. Two features in the new release makes the iPhone 3GS the perfect augmented reality platform. First, the inclusion of an onboard compass and second, the opening up of the API to full camera controls.

Almost everyone except developers were unaware of this but the iPhone 3G SDK painfully crippled the camera by forcing you to use the provided Apple API to take pictures (this meant no real time computer vision could run on it except when Apple conveniently didn’t notice you bypassing their allowed APIs). Now that the APIs are open, expect to see a bunch of innovative computer vision apps (face detection, object detection, tracking).

But the second breakthrough is that the compass finally provides a braindead 6 Degree of Freedom (DOF) estimation. Any rigid object in space can be defined by 6 parameters. 3 spatial ones and 3 rotational ones. The iPhone 3G could estimate 5 of those 6 parameters acceptable well with GPS providing rough spatial data, the acceleromter providing fine spatial data as well as the direction of gravity. The compass provides the final, missing degree of freedom that allows for complete pose estimation.

Augmented Reality needs to know two things: Where is the sensor and where is everything else. Once you know that, you can do all sorts of really cool shit. I have no idea what will eventually come out, that’s one of the exciting things about bringing this technology to a mainstream audience, but I can point to some interesting research directions that seem plausible:

  • Instant 3D modelling of everything: Wave your iPhone around an object and it’ll figure out how to create a crude 3D model of it in memory.
  • Interactive furniture arranging: Go to the store, scan a bunch of furniture you want to buy, go home, drag and drop virtual pieces of furniture in your living room to figure out which piece should go where.
  • Interactive tour guide: You’re walking through New York, you see a cool building, take a photo of it and all of a sudden you know it’s the flatiron building and you’re reading a wikipedia article about it.
  • Photosynth the entire damn world: nuff said.
  • AR Quake: nuff said.
  • History view: Point to a space and if there’s a security camera pointed at it, be able to review what happened at that spot at any point in time.
  • Invisible Ink: Leave messages on walls which only your friends can see. Send them on an easter egg hunt.
  • Virtual Ping Pong on the phone: Use the phone screen to see a virtual ping pong table and then swing the phone to make a hit
  • Physical, virtual avatar conferencing: Replicate the real world cocktail party acoustics in a virtual physical space. Cocktail parties are great because they allow the spontaneous formation of ad-hoc small conversation clusters within a larger conversation. Online tools do a poor job of replicating that dynamic but if we could bring physicality back into it, we might be able to bring some of this dynamism back into online conversation.
  • A million goddamn screens: This was a project I personally worked on that unfortunately, never got to far but has a dear place in my heart. Conventional computing is predicated on screens being expensive but if you stick a tracking marker onto a piece of paper, you can turn it into a screen. What would computing be like if you could produce screens out of some cardboard and a laser printer that could be any form factor and would cost 10 cents a pop? How would this help with information overload? Imagine you have a screen in the corner that represents the pile of unread emails. If you want to read an email, drag it onto a new screen. Important emails that you need to reply to each get their own seperate screen which you keep neat and tidy by arranging them in a pile. If you want to send a file to your coworker sitting next to you, you can drag it onto a screen and then physically hand him the screen. You can have a screen for each individual participant in an IM/voice/video conferencing and pulling the screen closer means you want to be alerted but pushing it away means you want to ignore them. What could you do if you had access to a million goddamn screens over the course of your lifetime?

Are some of these examples wildly unrealistic and totally unable to work in real life? Of course, they come from research inspirations. But they demonstrate the enormous power of augmented reality which is about to be unlocked within these next few years as developers grapple with just what’s possible with Augemented Reality.

Responses

  1. Nev says:

    June 10th, 2009 at 5:19 pm (#)

    “Heard it here first”?

    Hardly, this kind of thing has been bandied about for months and months now!

  2. Nev says:

    June 10th, 2009 at 9:19 am (#)

    “Heard it here first”?

    Hardly, this kind of thing has been bandied about for months and months now!

  3. Max Martinez says:

    June 10th, 2009 at 5:51 pm (#)

    “opening up of the API to full camera controls”

    Are you sure this is true? From what I’ve seen they haven’t opened up anything. The only related new API I found was an addition to UIImagePickerController that allows you to capture video, but it works just like capturing images, it only gives you the final result, not a realtime feed.

  4. Max Martinez says:

    June 10th, 2009 at 9:51 am (#)

    “opening up of the API to full camera controls”

    Are you sure this is true? From what I’ve seen they haven’t opened up anything. The only related new API I found was an addition to UIImagePickerController that allows you to capture video, but it works just like capturing images, it only gives you the final result, not a realtime feed.

  5. Steven says:

    June 10th, 2009 at 6:23 pm (#)

    Yeah, the camera API hasn’t been opened, so there is still a huge roadblock to most of these ideas.

  6. Steven says:

    June 10th, 2009 at 10:23 am (#)

    Yeah, the camera API hasn’t been opened, so there is still a huge roadblock to most of these ideas.

  7. Joe Zydeco says:

    June 10th, 2009 at 9:21 pm (#)

    That’s odd, because the Picture Me app is using OpenCV and can do some pretty real image processing on the camera input. Even works on my old 2G phone. It snaps the shutter when it detects a face in the frame.

    http://blog.beetlebugsoftware.com/post/104154581/face-detection-iphone-source

  8. Joe Zydeco says:

    June 10th, 2009 at 1:21 pm (#)

    That’s odd, because the Picture Me app is using OpenCV and can do some pretty real image processing on the camera input. Even works on my old 2G phone. It snaps the shutter when it detects a face in the frame.

    http://blog.beetlebugsoftware.com/post/104154581/face-detection-iphone-source

  9. Joe Zydeco says:

    June 10th, 2009 at 9:23 pm (#)

    And, ironically enough, it’s from the same site as this. Sorry for the confusion! I’m an idiot.

  10. Joe Zydeco says:

    June 10th, 2009 at 1:23 pm (#)

    And, ironically enough, it’s from the same site as this. Sorry for the confusion! I’m an idiot.

  11. links for 2009-06-10 « Blarney Fellow says:

    June 10th, 2009 at 5:08 pm (#)

    […] The killer app for iPhone 3GS « Bumblebee Labs Blog (tags: augmented-reality iphone) […]

  12. Arin Crumley says:

    June 11th, 2009 at 1:20 am (#)

    Hang,

    I love this post. Even if Apple has not yet fully opened up a live video stream for applications to build into their apps, it’s only a matter of time before they do or some other similar device steps up to provide an environment for this innovation.

    I think all of the scenarios you described could occur by way of a variety of devices that I’m sure will be emerging and not all on just the iPhone. The idea of any device being a screen would work best most likely using a portable pocket projector. That way you could point to a surface and then start reaching into the projection and moving things around. And the Quake reality game idea would work best as an eye piece. Maybe you hook an eye piece screen up to an iPhone or similar pocket computer and it recognizes your hand to be a gun or a sword and you can go fight your friends that it recognize to be team players. All of the weapons can be overlaid on top of the video and you could have even added sound effects and voice conversations to be part of the game.

    We’ll see how far the iPhone can take us in terms of scanning the 3d world around us. Maybe we’ll be seeing more laser technology for scanning objects. Maybe that is a direction iPhone will take us at some point. A future model might have a laser pointer that is a fun tool to play with cats but also if your cat sits still allows you to create a very accurate 3D object of the cat as you rotate the device around the cat and the image information that is recorded could create a texture map on top of the cat and then your cat could be the visual representation of the talking voice that tells you what song is playing and who you are calling.

    Who knows what will happen but it’s neat to be along for the ride.

    Thanks for creating this post and to @jgilliam for tweeting it.

    Arin

  13. Arin Crumley says:

    June 10th, 2009 at 5:20 pm (#)

    Hang,

    I love this post. Even if Apple has not yet fully opened up a live video stream for applications to build into their apps, it’s only a matter of time before they do or some other similar device steps up to provide an environment for this innovation.

    I think all of the scenarios you described could occur by way of a variety of devices that I’m sure will be emerging and not all on just the iPhone. The idea of any device being a screen would work best most likely using a portable pocket projector. That way you could point to a surface and then start reaching into the projection and moving things around. And the Quake reality game idea would work best as an eye piece. Maybe you hook an eye piece screen up to an iPhone or similar pocket computer and it recognizes your hand to be a gun or a sword and you can go fight your friends that it recognize to be team players. All of the weapons can be overlaid on top of the video and you could have even added sound effects and voice conversations to be part of the game.

    We’ll see how far the iPhone can take us in terms of scanning the 3d world around us. Maybe we’ll be seeing more laser technology for scanning objects. Maybe that is a direction iPhone will take us at some point. A future model might have a laser pointer that is a fun tool to play with cats but also if your cat sits still allows you to create a very accurate 3D object of the cat as you rotate the device around the cat and the image information that is recorded could create a texture map on top of the cat and then your cat could be the visual representation of the talking voice that tells you what song is playing and who you are calling.

    Who knows what will happen but it’s neat to be along for the ride.

    Thanks for creating this post and to @jgilliam for tweeting it.

    Arin

  14. hans.gerwitz says:

    June 11th, 2009 at 2:19 am (#)

    Amen. I’ve been hoping someone would do this even without the compass… server-offloaded (dare I say “cloud”) video processing with positioning hints should be able to achieve nice accuracy for 3D object capture.

    Hopefully the positioning is close enough to make a SunPad experience good enough.

  15. Hans says:

    June 10th, 2009 at 6:19 pm (#)

    Amen. I’ve been hoping someone would do this even without the compass… server-offloaded (dare I say “cloud”) video processing with positioning hints should be able to achieve nice accuracy for 3D object capture.

    Hopefully the positioning is close enough to make a SunPad experience good enough.

  16. Flow » Blog Archive » Daily Digest for June 11th - The zeitgeist daily says:

    June 10th, 2009 at 7:27 pm (#)

    […] Augmented Reality is the killer app for the iPhone 3GS — 5:53pm via Google […]

  17. New iPhone possibilities « The Daily T says:

    June 11th, 2009 at 1:38 am (#)

    […] BumblebeeLabs fantasizes about what’s coming up on the software front now that the iPhone has a compass and an open video camera API in addition to its accelerometers and GPS.  Exciting stuff. […]

  18. Adeyemi Fowe says:

    June 11th, 2009 at 12:32 pm (#)

    This are my people talking here!
    I love your passion for what is possible in with technology and i love all the comments! :):)

  19. Adeyemi Fowe says:

    June 11th, 2009 at 4:32 am (#)

    This are my people talking here!
    I love your passion for what is possible in with technology and i love all the comments! :):)

  20. Graham says:

    June 11th, 2009 at 1:24 pm (#)

    The accelerometer in the iPhone is far too crude for any of this to work. It can’t be used to measure anything relative to the real world, because the values returned have no units. And it can’t measure orientation independently from acceleration and movement. The only thing it can give you a reliable reading of is the phone’s orientation relative to the ground, and only then, when it’s being held completely still.

    So your ideas require much more sophisticated hardware than Apple currently provides.

  21. Graham says:

    June 11th, 2009 at 5:24 am (#)

    The accelerometer in the iPhone is far too crude for any of this to work. It can’t be used to measure anything relative to the real world, because the values returned have no units. And it can’t measure orientation independently from acceleration and movement. The only thing it can give you a reliable reading of is the phone’s orientation relative to the ground, and only then, when it’s being held completely still.

    So your ideas require much more sophisticated hardware than Apple currently provides.

  22. David A. says:

    June 11th, 2009 at 2:38 pm (#)

    “X-Ray” vision for construction / repair / maintenance. Scan the iPhone over a street and see the buried pipes (documented by plans on archive or “as built” pictures”) See piping in walls, etc. Recognize electrical panels, large mechanical devices, etc. and like existing AR (e.g., used in aircraft maintenance and some other high cost applications), overlay diagrams and other useful information. This brings known value AR into a whole new market.

  23. David A. says:

    June 11th, 2009 at 6:38 am (#)

    “X-Ray” vision for construction / repair / maintenance. Scan the iPhone over a street and see the buried pipes (documented by plans on archive or “as built” pictures”) See piping in walls, etc. Recognize electrical panels, large mechanical devices, etc. and like existing AR (e.g., used in aircraft maintenance and some other high cost applications), overlay diagrams and other useful information. This brings known value AR into a whole new market.

  24. Robert says:

    June 11th, 2009 at 2:44 pm (#)

    The iphone is getting better, but it is far from the ultimate augmented reality device. Sure, expect to see a bunch of basic AR applications coming out that take advantage of the accelerometers and the compass, but will ultimately still be gimmicky. The “really cool stuff” (at least half of the stuff you have listed in this article) will require a few more generations of iphones and a supporting back-end (or cloud) infrastructure. Just because you can do some marker tracking or face tracking (it isn’t really face *recognition* or *identification* just yet) doesn’t mean that AR is ready for prime time. Give it a few years… it is definitely coming.

    Check out my presentation at Mobile Monday Amsterdam if you want to know more about what I think about it… http://www.mobilemonday.nl/talks/robert-rice-augmented-reality/

  25. Robert says:

    June 11th, 2009 at 6:44 am (#)

    The iphone is getting better, but it is far from the ultimate augmented reality device. Sure, expect to see a bunch of basic AR applications coming out that take advantage of the accelerometers and the compass, but will ultimately still be gimmicky. The “really cool stuff” (at least half of the stuff you have listed in this article) will require a few more generations of iphones and a supporting back-end (or cloud) infrastructure. Just because you can do some marker tracking or face tracking (it isn’t really face *recognition* or *identification* just yet) doesn’t mean that AR is ready for prime time. Give it a few years… it is definitely coming.

    Check out my presentation at Mobile Monday Amsterdam if you want to know more about what I think about it… http://www.mobilemonday.nl/talks/robert-rice-augmented-reality/

  26. Hang says:

    June 11th, 2009 at 3:54 pm (#)

    Joe: Any apps that do that are using undocumented APIs and Apple is largely turning a blind eye to it at the moment.

    Graham: Any AR app would probably work using a combination of accelerometer and feature tracking to correct for drift. Andrew Davidson did some great work on this when I was working in the area if you want to check out what was possible back in 2004.

    Robert: It doesn’t have to be good, it just has to be good enough. The iPhone is going to be the first real AR experience for a large number of people and it’s going to cement it as an interaction style of it’s own right.

  27. Hang says:

    June 11th, 2009 at 7:54 am (#)

    Joe: Any apps that do that are using undocumented APIs and Apple is largely turning a blind eye to it at the moment.

    Graham: Any AR app would probably work using a combination of accelerometer and feature tracking to correct for drift. Andrew Davidson did some great work on this when I was working in the area if you want to check out what was possible back in 2004.

    Robert: It doesn’t have to be good, it just has to be good enough. The iPhone is going to be the first real AR experience for a large number of people and it’s going to cement it as an interaction style of it’s own right.

  28. Graham says:

    June 11th, 2009 at 7:33 pm (#)

    It’s not a question of “correcting drift”. That’s like saying the iPhone’s light sensor can be used as a forward-facing camera just by applying a bit of color correction. It’s not the accuracy of the information from the camera that’s the problem but the richness of it.

  29. Graham says:

    June 11th, 2009 at 11:33 am (#)

    It’s not a question of “correcting drift”. That’s like saying the iPhone’s light sensor can be used as a forward-facing camera just by applying a bit of color correction. It’s not the accuracy of the information from the camera that’s the problem but the richness of it.

  30. android user says:

    June 11th, 2009 at 9:45 pm (#)

    Zzz. Heard of wikitude?? Been there, done that

  31. android user says:

    June 11th, 2009 at 1:45 pm (#)

    Zzz. Heard of wikitude?? Been there, done that

  32. uglychart.com: a blog about stocks » Blog Archive » links for June 15th says:

    June 15th, 2009 at 1:02 pm (#)

    […] The killer app for iPhone 3GS « Bumblebee Labs Blog – technology iphone programming toread futures […]

  33. 20fourLabs » Why your next mobile should have a Magnetometer says:

    June 19th, 2009 at 2:14 am (#)

    […] about augmented reality – Stuff.tv IBM launches new smartphone apps for Wimbledon – The Guardian The Killer app for iPhone 3GS – BumbleBee […]

  34. [Petits liens] Autour de la boussole de l’iPhone 3G[S] says:

    June 25th, 2009 at 5:28 am (#)

    […] The Killer App for iPhone 3G[S] chez Bumblebee Labs […]

  35. Flow » Blog Archive » Prediction in June, 2009 - The zeitgeist daily says:

    June 30th, 2009 at 4:49 am (#)

    […] The digital universe is getting closer and closer to you, this new reality will become your natural extension. All these by using obsolete technology […]

  36. Danger359 says:

    July 4th, 2009 at 12:14 pm (#)

    great post. however you have forgotten the potentials of massive interconnectivity via social network. think: distributed application. distributed sensing. global pooling of individually collected data / comments. This is where the true power of the iPhone will appear.

  37. hosting forum says:

    January 16th, 2012 at 7:14 am (#)

    I like this article very much because it is technical and very much useful for me to gather more knowledge. Good to read.

  38. Best web Hosting says:

    February 3rd, 2012 at 1:37 pm (#)

    Good job! This message is a great example of why I go back again to read your quality content that is always updated.

  39. webserv says:

    February 16th, 2012 at 4:14 am (#)

    It’s a nice post!
    Regards,
    Bangalore website design

  40. Sandeep Verma says:

    March 7th, 2013 at 6:48 am (#)

    great post. however you have forgotten the potentials of massive
    interconnectivity via social network. think: distributed application.
    distributed sensing. global pooling of individually collected data /
    comments. This is where the true power of the iPhone will appear.

Leave a Response