I haven’t written much about Augmented Reality since joining Microsoft in 2008. I can’t really say why that is. It’s been a topic of some passion for me for over 20 years. But I think it’s time to start sharing some of my thoughts on the subject. Here’s a first installment.

1. AR is Dead (long live AR!)

Think about this. Virtual Reality was the “it’ll change everything!” buzzword of the early 1990s, well before the dot.com boom made and broke so many millionaires. Today, how many virtual reality experiences do you see a day, a week, a year?

None? Maybe IMVU and Second Life still count. So… two, if you’re so inclined?

Actually, Virtual Reality is a hundred-billion dollar industry, if not more. It’s just not called VR anymore, that’s all.

It’s called: Online Multiplayer Games, Simulation, Virtual Training, Telepresence, and even Natural User Interaction. It made the movie Avatar feasible. Kinect just sold a zillion units in its first six minutes and, yes, when you’re playing Kinectamals on your TV, you’re participating in a virtual world.

Boom. Some marketing person lied to you.

It’s fine. It’s their job to sell stuff by whatever name works best.

(Oculus Rift being the notable exception to this rule — can they revive the original technical term?)

Q: So, given the same trend of a crappy buzzword that will likely die with the downward hype cycle, what will AR actually be called in five years?

A: Shopping, Advertising, Training, Education, Art, NUI, Spatial Games, Location Based Games, Situational Awareness, Geoimmersion, Ambient Computing, SoLoMo, and even Holographic Telepresence, to name a few…

“AR” as a buzzword is near the end of its life. Especially when you see something like this:

Ikea AR App

The app consists of a live camera image with a set of downloaded 3D models of furniture superimposed. There is no AR tracking whatsoever, not even using the compass, as far as I can tell. Moving your iPaid slightly from side to side throws off your careful manual alignment. So it’s basically the same quality as holding up an acetate sheet with a picture on it, although you can manually rotate the object.

Now, I don’t want to bash what’s actually a decently useful app for in-situ furniture visualization (even if it is not actually an IKEA-sponsored app either, as noted in the fine print). I have to seriously commend the three guys who built this in 4 weeks for truly grokking the concept of “minimum viable product.”

But this is “augmented reality” like I’m a supermodel.

A basic definition of AR says there needs to be at least some computational connection to the real world. You must sense the world in order to augment it, right?

2. What the hell is AR anyway?

This is probably a moot point if the term is going away. It’s really whatever people say it is. Who am I to try to make up definitions that stick?

But, it turns out, there is already a kind of basic accepted definition for AR: an experience that captures a user’s view of the world and selectively augments it. But what does “augments it” even mean?

This is where the rubber hits the road. Just overlaying something on a video stream is not AR. That could better be called real-time CGI video effects, video keying, and basic videographics.

Myron Kruger was doing this in stride when I was just becoming aware of real-time anything and he was already somewhat jaded by that point. At the time, I didn’t really understand why. Myron totally got that it wasn’t just the green-screen that did it, but the interaction between the virtual stuff and the real person that mattered.

Consider this: to augment the world is to change the world.

In other words, AR is not AR without impact, even if that impact is only on you or other people who inhabit the world in some limited space. Tagging a wall with a can of spray paint is a better example of augmented reality in a more literal sense. So let’s move away from the simple definition and get down to brass tacks (which, btw, was another form of physical augmented reality, circa 1863).

3. Augmented Perception

Most of the AR that people talk about today makes no change to the world itself, only to the sensory input streams of the user, and indirectly at that. In fact, the world has no idea we even exist, and it’s far from augmented by anything we say or do. We are simply using technology to mediate and improve our perception of the world. So let’s call it augmented perception then. Try it on for size.

Such a service may add a desirable superpower, in the form of seeing information that’s normally hidden from human view. Let’s call this clairvoyance, or perhaps “digital clairvoyance” to distinguish from the paranormal variety. We can see information about real things, names, birthdays, reviews of favorite restaurants, or purely fanciful overlays, like ghosts or zombies chasing you downtown.

The internet itself is an abstract information space, but it has a natural spatial projection on the physical world that we will soon see for ourselves. But as with search engines today, The Google or The Bing, we don’t want to see the internet in its naturally incomprehensible multi-dimensional hyper-space, but rather some arbitrarily filtered  view of what we might find useful to see right now.

Indeed, the goal of Augmented Perception is not, as Will Wright correctly pointed out, to dump more information on our already-overloaded senses. It’s not to take Times Square and square it again with even more sensory/information overload, as in “to everything there is a thought bubble, turn, turn turn.”

Augmented Perception is there to help us understand the world more fully, more readily, and more profoundly.

Augmented Reality can make us more aware of our immediate environment rather than distract us from it…there is value in proximity information…tap into the collective commune memory…situational awareness is what I am dreaming of”

This is increasingly being called Ambient Computing and/or Situational Awareness. Visual representations are just one way to achieve these goals. I’m interested in many others. And indeed one may need to remove information, to dim parts of reality, blur the backgrounds and generally reduce ambient noise in order to actually enhance the signal aka improving people’s perception of that which is important to them right now.

The AR marketers didn’t tell you that either.

4. Augmented Interaction

The conceptual “dual” of Augmented Perception is found in our ability to interact with the real world in enhanced ways. That’s another flavor of mediated reality, focused on intent and action. One’s thoughts may jump straight to perhaps the most commonly wished superpowers — flying, time travel, super strength. For some, like my wife, it’s invisibility, but that’s beyond simple explanations. Those are all nice to want, but there’s something much more profound coming right around the corner, in the form of apps like Google Now, facebook mobile, and more.

Just think of what it means to drive over a pothole, wish it was fixed, and a road crew suddenly shows up to fix it. There’s an app for that, cliche complete.

This is a kind of “loop-closing” narrative that reduces the friction from thought to action, bringing us that much closer to god-like powers of thinking something and seeing it happen in front of us. Well, perhaps potholes and city utilities are not the best example of “instant action,” but every little bit helps.

Just think of what it means to see a product you like, then “like” or “want” it via some pinned photo or phone button, and then magically find an ad or “buy now” button next? Near-instant gratification and wish fulfillment is what that is. That’s certainly high on the marketer’s own wish list, if not yours. The holy grail of marketing requires knowing what you all might want at any given time and offering it to you right when it’s most convenient. That’s not an advertising business model as much as it is a credit card business model — guaranteed revenue from every transaction. This is where Google is heading, btw.

[Though to be totally honest, the subsequent invoice on your impulse buys does kind of ruin the supernatural aspect of this superpower, like buying yourself a birthday surprise. Oh, look! Thanks! I love me…]

What will happen when we have Augmented Perception and Augmented Interaction bound together for a truly symmetric read/write mediated reality? We’ll be able to create Second Life in the Real World, for starters. Why would we (perhaps not) want to? That’s a subject for whole other post.

5. AR is Dead (long live AR)

We’re at the cusp of amazing things like Google Glass, upcoming Vuzix Holographic lenses, Oakley AR Ski Googles (btw, Oakley is owned by Luxottica — along with almost everything else — forming a natural monopoly on most of your vision needs. So you’d better be interested in whatever they do in this space).

And who can say what other companies are lurking on the horizon?

How on earth could we say AR is already dead? It’s clearly and inescapably about to explode into a prismatic rainbow of possibilities.

Yes. This is all true. But as with Virtual Reality, the future of AR is not wallowing in the history of great ideas that might have been. The history of AR will be written by whomever makes the most successful products, the killer hardware and software experiences, picked up by millions, if not billions of people.

It’ll be called whatever they decide to call it. But it won’t be “AR.”

Because they’ll have a huge incentive to distance themselves from all the crappy experiences that came before. “We’re not AR,” they’ll say, “That’s old. That was all about cell phones and jittery graphics, about fiduciary markers and 3D chairs floating in space. What we’ve done is unleash interactive magic on the world, increase human potential, and usher in a world of augmented and yet totally natural human perception and interaction.”

Augmented Perception and Interaction. That’s not half bad. API would be a good acronym, if it wasn’t already taken. But whatever we call it, for better or worse, it will be your computer-mediated interface to the world.