Now, I should caution that Blaise didn’t ultimately want to use this text and Jaron equally had issues with it. The tone is all wrong for an academic journal, plus Jaron disputes some of the dates I recorded from my research (he may well know better). But I felt it might at least be entertaining to RP readers, so I’m posting it for you to enjoy. Still, don’t take any of it as official, just me being a smart-ass.

October 20^th, 2008 marked the 30^th anniversary of the MUD [1]or Multi User Dungeon, widely recognized as the world’s first multi-participant text-based virtual world. Only three years later, a somewhat less interactive work, True Names,[2] by Vernor Vinge, imagined full multi-sensory worlds with millions of participants. The film TRON debuted only a year after that, popularizing, if not actually monetizing, computer-mediated virtual worlds as full-on alternate realities — places with lives onto themselves. But before any of these were even conceived, The Veldt[3], by Ray Bradbury, envisioned “The HappyLife Home,” a fully immersive CAVE-like space, consuming parents and kids alike, way back in 1950.

The history of virtual worlds is a complex mesh of fact and fiction, weaving pioneers, dreamers, authors, and critics in a quest to define a grand vision and to meet an ancient need, dating back to the days of burnt charcoal on cold cave walls. That need is to communicate, to share and persist what is otherwise ephemeral, isolated, and ultimately bounded to the lifespan of memory: our thoughts, our ideas, and our stories about life, real and imagined. It is perhaps fitting that these visionary fictions are themselves conveyed to audiences, new and old, as print- and film-mediated virtual worlds[4], just as the CAVE acronym[5] itself is a recursive allusion to “Plato’s Illusion” playing on those same torch-lit walls.

That grand vision, however, goes beyond mere communication. The desire for ubiquitous virtual worlds is perhaps an understandable manifestation of our collective (if not universal) longing to overcome the rules that bind us, to propel our minds over the limits of matter, space, and even death, of which the “mortality of ideas” is just one example. The end goal, for many, is to construct alternate realities so malleable, so perfectly adapted to our innate desires, that we could fairly call the results magic.[6] In those brave new worlds, there need be no scarcity, no ugliness, nor pain. We can be whatever we imagine or wish ourselves to be, and suffer none of the limits of ordinary, mundane life.

Then again, in reality, as in fiction, the stories never quite seem to work out that way. Reality has a way of always winning in the end. And, as with the previous allegories, this grand quest of ours is so cyclical, so inflated and self-referential, as to often resemble the construction of a Klein bottle. The true history of virtual worlds is one of visionary and often impervious genius, promises made (and made again), and the search for the most elusive human-computer interface ever envisioned – the one that disappears.

One of the early visionary geniuses in the field was the legendary ACM Fellow, Ivan Sutherland. His ground-breaking Sketchpad system plotted the lines along which future computer graphics interfaces would be drawn. He invented the head-mounted display, heavy as it was, to better couple the virtual imagery to our actual head motions, and in so doing, laid the foundation for untold future systems that place one or more virtual interactive cameras in a computer-mediated reality.

But just 10 years earlier and only a few years after Bradbury’s fictional exploration of virtual family values, Walt Disney more literally broke ground on a site in Anaheim that would become, for a significant period of time, the world’s largest physical virtual world. Until then, the best example of a “fantasy land” was the haunted house, a grotesque and distorted reflection of reality, not entirely unlike some early experiments in VR. Disney’s theme parks provided a comfortable level of immersion for many people — not nearly as interactive as, say, Korea or Vietnam, and not nearly as rich as the Renaissance Pleasure Faire and “RenFaires” since, but safe and fun for the whole family.

The 1960s also saw an explosion of experimentation in the media of the time, including Mort Heilig’s Sensorama, which included stereo visuals, smells, sounds and even haptics. Filmic experiments in 3D and surround visuals abounded, with Disney’s Circle*Vision[7] being one of the most widely viewed. The addition of sturdy handrails to keep guests from falling over (in a stationary theater) is a testament to the power of these experiences to move and consume us, for better or worse.

While we’re not sure if Sutherland knew Disney or Heilig in his day, we do know that Evans and Sutherland computers were used to help animate Disney’s TRON in 1981. And ten years after Tron failed to make a dent in any critical or commercial sense, Disney Imagineering’s VR Studio would similarly begin their real-time interactive VR experiments on Evans and Sutherland image generators.[8] The obvious choice of movie content gave way to a failed Rocketeer attraction and later Aladdin’s Magic Carpet Ride, sporting heavy HMDs, counterbalanced much like Sutherland’s Ultimate Display. One of Sutherland’s students, Ed Catmull, also found inspiration in Disney animation. He embarked on 30 year quest to reinvent the art and science of animation, finally coming full circle in the new millennium, as the head of Disney’s animation studios after Disney swallowed Pixar.[9]

The hard road towards better computer-mediated storytelling (in HMDs and on the silver screen) merely proved that obstacles are there to surmount. Randy Pausch, most famous for his “last lecture” at CMU, observed that obstacles "give us a chance to show how badly we want something” — or, perhaps, to make us take the time to understand why. Dr. Pausch worked with the same Disney VR Studio, CMU’s Alice software, and other members of the story, all of whom badly wanted to solve the problems of more easily building and experiencing computer-mediated virtual worlds.

But reinventing the world takes time, often a generation or two. Ivan Sutherland was reportedly influenced by Vannevar Bush’s 1945 atomic-age essay “As We May Think,”[10] which is also widely credited as inspiration for the World Wide Web. That it took 50+ years to produce a Google and an MSN to make the web more tractable is merely a reflection of the difficulty we have in taking grand ideas and rendering them in a form that works for ordinary people – the proper convergence of money and market, timing and technology; but more importantly, a better understanding of just what it is we should be asking for in the first place.

In the early period, well before we had too many cogent glimpses of that sort of revelation, the 1970s saw the exploration, largely using university mainframes, of true multi-user environments. Maze War, in 1974, was the first known multi-user environment, and graphical to boot. But it was very limited in its influence. MUD, in 1978, had much more success. The lack of graphics proved to be no deterrent, and in fact, arguably improved interactivity and depth, simplifying some very hard problems to a more manageable scope.

The 1980s saw a new push into real-time graphical interfaces, with cheaper and more available commodity hardware. On the desktop, Apple and Microsoft pushed 2D windows, icons, and mice to great effect. In movies, CGI went from niche to boutique to a mainstay of visual effects. And VPL Research pushed the envelope in full-sensory virtual reality (and even the term itself), providing visual programming elements and a grab bag of more natural user interfaces. Their Reality-Built-For-Two was the first commercial VR system, shipped in 1989.

Alas, the Achilles heel of commercialization is the requirement of making money. VPL ultimately sold itself and its patents to Sun in 1998. Still, in 1989, Mattel released a PowerGlove that simplified the VPL DataGlove concept into something mass-marketable. It unfortunately failed to spawn the kind of kinetically addictive games that the Nintendo Wii presently enjoys, despite many similar capabilities. And while the Visual Programming Language VPL developed broke new ground, it fell to Lego and Microsoft’s Robotics Studio, years later, to truly push some of these concepts to wider audiences.

The 1980s can be thought of, in a sense, as the early adolescence of virtual worlds, in which the core technological concepts came to be expressed, tested, and propagated to anyone who would listen. Habitat, for example, emerged in 1988 on the popular Commodore 64 platform, and was arguably the first mass-market virtual world, presaging mega hits like Habbo Hotel, Club Penguin, and even Everquest by over a decade, but never quite gaining the market validation they sought.

If the 1980s are the early teens, the 1990s represent the often-raucous teen to adult transition. Unfortunately, as difficult as it is for any child actor to grow up with the hype and spotlight of Hollywood, so too were the overwhelming expectations for VR technology a part of its downfall. Movies like The Lawnmower Man promised an effectively supernatural decoupling of VR’s espoused benefits from any actual truth. Publications liked Wired treated VR visionaries (or indeed, anyone who seemed to have a good futuristic idea) like rock stars. In fact, it’s the expression of difficult computer science concepts in natural human metaphors that is both the greatest strength and the ultimate weakness of VR – anyone describing it to lay audiences can use language that evokes sweeping images and expectations that are currently impossible to meet.

Meanwhile, with all of the attention and potential of VR technologies for computer-mediated virtual worlds, research funding increased dramatically. Among the efforts, the Human Interface Technology Lab at the University of Washington pushed new technologies to solve some of the core interface challenges in VR, leading to devices like the Virtual Retinal Display, a laser-based display that is today most closely related to pocket-sized low-power projectors. UNC pushed the envelope on haptics and core technologies, while Universities like Utah, Ohio State, Brown, and Stanford pushed ahead primarily on 3D rendering. Carolina Cruz-Neira and others pushed the envelope in projected virtual environments, culminating in 1992 with the CAVE at the University of Illinois Electronic Visualization Laboratory. Meanwhile, continued military investment in visual simulation drove investment in graphics supercomputers, ultimately leading to cheap commodity 3D hardware acceleration for PCs.

Virtual Reality went, more or less, like the space program. Massive investments didn’t result in any influx of civilians going into space, no personal robots nor flying cars. It certainly got a lot of public attention, anxiety, and perhaps disappointment after some early climaxes. But the technologists and dreamers marched on. And core technologies produced did, in fact, wind up in everyday consumer devices, from cell phones to gaming consoles.

Indeed, while true Virtual Reality devices failed to take over the world in any meaningful way, the offshoots of the very same work wedged themselves into our daily lives. While Disney Imagineers worked on million dollar SGI hardware to build an immersive Aladdin attraction in 1994, they played Doom and Quake in the office for fun. Giant dinosaurs gave way to nimble mammals, just waiting for their chance. John Carmack’s games simultaneously spawned an entire genre of blood-splattering Demon/Nazi/Zombie shattering realism and helped blast PC-level 3D graphics into the mainstream.

The gaming community can be credited with seeing some of the most lucrative uses of 3D graphics and interactivity, and of turning it all into a sustainable, even thriving business[11]. Gaming has turned into such a serious business that a whole branch of Serious Games has emerged[12].

But while game companies were having fun and making money, the internet boom took full effect, with a much stronger emphasis on the latter over anything else. SGI famously launched Cosmo, an offshoot of VRML 2.0, to take the “Metaverse” by storm. The “dot com” bubble dwarfed by orders of magnitude any investment in games or VR, pushed home broadband adoption, but ultimately left US network infrastructure and networked virtual worlds bobbing in the wake of the April 2000 tsunami.

Phillip Rosedale was one survivor of that bubble. He was one of the inventors of a video codec that made Real Video hum, having sold his company to Progressive Networks in 1996. He founded Linden Labs in 1999 with the proceeds and some help from friends in physics and video games. And while the evolution of massively multiplayer on-line games can be traced from Maze War to XPilots to Meridian 59 and World of Warcraft, Second Life differentiated itself in several important ways: first, there was to be only one shared world, not many shards or instances to split the load; second, the world is malleable and subject to user’s whims; and third, it’s not a game, it’s an alternate existence onto itself, tied into real world currency and the web itself, but separate just the same. Second Life also rejected the standard VRML view of the world, of scenegraphs and polygons, in favor of a completely dynamic soup of what are effectively virtual Legos. The end result is something that is both powerful, and still, eight years later, limited by its own design to a walled-garden and only linearly scalable topology.

Meanwhile, another survivor of the dot-com implosion was busy building a fundamentally different kind of virtual world, right in the bubble’s wake. Keyhole was a spinoff of the defunct game technology company, Intrinsic Graphics, involving many former engineers from SGI, who had worked with VR early adopters from Disney Imagineering to the NIMA and NGA[13]. The key technology could stream an entire planet worth of visual information over a standard network connection and render the relevant window it in real-time on any PC. But the true advantage over contemporary systems, GIS and otherwise, was in how simple and intuitive the interface was. In 2004 Google acquired the company and renamed it Google Earth. And today, it defines what a mirror world is supposed to be – an accurate reflection of the real world, equally navigable from your desktop or mobile phone.

However, there was yet another kind of mirror bubbling to the surface around this time, one much more grass-roots. Bloggers, with unfettered access to the Web, began posting about their daily lives, their interests, and their friends. Social networks began to take root in 2003, in a second wave of web startups, the so-called Web2.0. They developed “what’s new” feeds meant to keep people in the loop. Sites like Twitter emerged later to simplify the process of posting one’s status to the net. And more recently, efforts like Loopt, Whrrl, and others seek to leverage real-time location on mobile devices to build up a view of the mirror world that is both personal and reflective of our individual realities.

A distinct kind of virtual world was just coming into focus around the new millennium as well. The earliest examples of Augmented Reality can be traced to systems like Myron Kruger’s VideoPlace in 1975. These consisted of combining video streams of real people and real or artificial places to simulate a sort of detached immersion. Much as with a meteorologist on the local news, participants could see themselves acting in a virtual environment in the third person only, on monitors for example, vs. the first person points of view obtained with HMDs and CAVEs. Eventually, technology was added to accurately track and emulate camera lenses and to seamlessly register computer imagery into the video. Today, augmented reality typically refers to live captured from a person’s point of view, overlaid with relevant CGI and fed back to some display device, perhaps even a head-mounted-display.

The promise of augmented reality is best epitomized in Vernor Vinge’s novel, Rainbows End, where VR contact lenses provide continuous access to the augmented world both inside and out. Haptic interfaces provide the missing sense of touch to manipulate and make these virtual objects real. But the state of the art is nowhere near this threshold. Problems with tracking and registration of objects still remain, but the principle challenge remains the display device, making it small, unobtrusive, and yet high enough fidelity to enable a person to walk around and actually use it without getting hurt by the real world, which hasn’t gone away.

In the development of Virtual Worlds technology, we’ve seen displays go from a few vectors to thousands, millions, and lately billions of pixels, and from a few dozen to a few billion polygons per second as well. We’ve seen haptic interfaces shrink from room-sized devices, reminiscent of the Inquisition, to gloves and even direct neural stimulation with no mass at all.[14] We’ve seen input devices go from mechanical arms to wired magnetic sensors to wireless and even optical motion capture. And we’ve seen networks go from slow drizzles of bits to full-on torrents, with sophisticated methods of prediction to cover latency and errors.

But yet Virtual Worlds, despite such advancements, and despite the adoption of its technology and methodology in many fields, is still widely seen as a future technology, not relevant to our everyday lives, a walk down the street, a trip to the mall, a day in the office. Web 3D is derided as unnecessary and indeed cumbersome, given the success and simplicity of the current 2D Web. Second Life is seen as niche, despite many businesses trying, for a time, to set up shop in one of the best Metaverses around.

Clearly, something is missing in the equation.

After forty years of research and development, since the ground-breaking days of Ivan Sutherland, we’ve finally beginning to come to a realization. People like escapism and fantasy worlds. Kids love to play in the modestly dimensional clubs of penguins and hotels. And companies are rushing in to tame the wild 3D west and pan their virtual gold. But the vast majority of human beings still spend the vast majority of their time immersed in a much more compelling and less inescapable 3D environment, which we tend to call “real life.”

Computer-mediated virtual worlds are just beginning to catch up with what we do out here in the real world. We buy and sell things. We get and share information. We communicate and “get stuff done.” We can increasingly do all of that in a virtual world too. But we can’t do it nearly as well.

And though we bemoan the limitations of 2D displays and table-top mice who have overstayed their welcome by a decade or more, the reason we don’t typically do these things in a virtual world is not about the interface, or the speed of CPUs or GPUs. It’s because we live out here, in the real world. No one likes to commute across an international — or inter-dimensional border — on their way to work.

A significant push in modern immersive environments research, then, focuses not as much on how to bring us to Neverland, but on how to bring Neverland to us – to make The Virtual a part of our lives in a way that benefits us beyond the hype of paperless offices and social (read: peer pressure) networks.

What the new Virtual is all about is reality but augmented, mirrored, and hyper-realistic, giving benefits beyond VR, AR, and even AI (which has similarly vanished into the lattice of modern life).

But what does it take to make this vision real? How do we mirror the real world in a way that we can make it interactive, turn it into a trellis for overlaying whatever whimsical and/or beneficial fictions we can dream up? Once we have that trellis, how do we interact with these hybrid real/virtual objects? How do we trade them, when their substance costs no more than the electricity they consume?

Only one thing is certain: the story of the next 25 years will be written in the full light of day.

I’ll post more when I get a second, but it’ll take some time to digest.

For what it’s worth, my present take on the Singularity is a cross of Vinge’s and something Stoss said at a WorldCon party (or elsewhere), and Kurzweil, despite some inherent contradictions:

1. The future beyond a singularity is fundamentally unknowable. That’s the whole point. If we can accurately describe what’s past a so-called singularity, then it’s just your basic run-of-the-mill evolution, revolution or "disruptive" sea-change, which happen all the time.

2. People are good at extrapolating linearly, not exponentially. We can predict a few years out, but after that, reality diverges wildly from our naturally limited mental models.

3. We’ve already gone through multiple "singularities" throughout history, though perhaps increasing in frequency. Singularities are never the end of anything, but a new platform on which to complain about our current ways of life and ponder the color of the pasture on the far side of the next singularity.

Before their introduction, could people have predicted how the world would change with Writing? Or Computers? Or Corporations? Could they have even predicted the invention itself? If not, then these may also be singularities, points in history that we can only understand by looking back, not forward, like the approaching event horizon of a black hole.

That is not to say that some visionaries don’t imagine a world past that event horizon or see the event coming. But it’s all speculation, cautionary or wishful fiction at best.

Even the inventor of the mechanical computer, beyond genius for his day, could not have predicted word processors, virtual reality, AI, or even the CAD software that would have unquestionably helped design his mechanical computer.

One could argue that the One True Singularity will occur only when we (our heirs or errs) become smart enough to see through to the future beyond, i.e., the real Singularity is the last Singularity we will ever know.

| View | Upload your own

What he seems to be describing is apparently not too far from what I’ve been writing about for a while. The part I’m still skeptical about is the life-logging, and probably because of my own preference for privacy. You’ll notice I don’t twitter. I have a hard time believing anyone would even care to follow what I do from moment to moment. And I think careful editing is the secret to any compelling narrative. I just don’t want to put gigabytes of sub-standard, often mundane, prose out there into the digital firmament.

But putting that aside, the germ (and/or gem) of what he’s saying, and the part I totally agree with, is this notion of a pervasive synthesis of augmented, mirror, and alternate realities — no need to distinguish between those arbitrary categories. Turns out, there’s an old word for this which I think we can now safely revive to summarize the intent:

magic

Practical Magic (not the movie that has usurped the term) is ultimately what’s driving the vast majority of virtual worlds interest. When I enter a virtual world, I can suddenly do and be things I wouldn’t even try in real life. If that kind of magic invades the real world (and it already has, as I’ll get to in a moment), I get the best of both worlds. I don’t need to "log on" or even "go anywhere," but I can now do things that would have been inconceivable to people just 100 years ago, at least outside of science fiction or fantasy genres.

What would I do? Well, for one thing, in an augmented/mirror world, I can gain a degree of omniscience, knowing anything that has been modeled with meta-data. I can point to objects and see their history, their meaning, their ‘DNA.’ But why limit ourselves to overlaying information onto boring real world objects. When important objects exist only as bits of data, piped to my brain via (pick: desk/wall-mounted, head-mounted, eye-mounted, or cortical interface), I can now manipulate those in any way I choose, as if by magic. I can morph them according to my needs or whims. I can clone them. I can make them do seemingly magical things — fly, dance, interact with real and virtual objects, do work for me, entertain me, and so on.

Is this sort of magic really so new? No. Remember the telephone? It’s a device for putting people, who may be on the other side of the world, right beside your ear. That’s pretty damn magical, considering the ordinary limitations of time, space, and airport security. Cut the wires and walk around while doing so and it’s even more magical. Television would be even more magical still, if only the content wasn’t designed to turn your brain to consumer mush (quality of TV is indeed in the eye of the beholder, and by that I mean the D&D monster that can paralyze you with a glance). All in all, we have been increasingly bypassing limits of the real world through communication technology.

What this all comes down to is not just the future of virtual worlds, but the future of communication itself, of which pure virtual worlds are only the most natural embodiment. Virtual Worlds are not a place, it turns out. They’re not a novel software application either. They are a key component of human-to-human communication, as old as humanity itself. Virtual worlds are what sit in our brains to reflect (i.e., model) the world around us. They are what sit in the bits in a computer’s memory to do much the same, but without as much intelligence or understanding. And they are what "virtually" sit between us when we try to communicate what’s in our heads — they are concepts in context, encapsulated in content.

The real world — physics, biology, even time — has very little to do with it, except as it serves as the funnel through which we must inevitably pass such information in order to communicate (see "limits" above). Virtual worlds widen that funnel to a full-on wind tunnel. They permit all sorts of sensory, conceptual, and sometimes even factual information to pass between us more easily.And what the future holds for that goes way beyond the "metaversal," almost cartoony manifestations we see today. Virtual worlds have the potential to literally open my mind to yours, as directly as possible, to allow the free flow of information back and forth: ideas, and expressions moving between people in a massive increase of both bandwidth and understanding. That’s what we’re dealing with here, and nothing less.

So lets get back to what it will look like. The easiest thing to imagine is the virtual meeting — that’s being built already. Seven people spread across seven cities can meet around a common table. Now, forget the large projection screens and make it a cafe table in Paris, or an office anywhere. The easiest way to make that work is to give everyone their own virtual view overlaid on the real world. It’s not the only way. But it’s certainly the most compatible with our current business models. [Imagine if that cafe has to shell out $30k (even US dollars) for a VR-enabled table+room and it's much less likely to happen.]

So now we walk around our daily fog with displays that let us see things that aren’t really there. What then? Well, we’ll need to interact with them. 3D Cameras are becoming good enough such that we can skip the data gloves of my youth. It doesn’t give much force feedback, but there are better ways of doing both on the horizon — if we can help paralyzed people walk via electrical stimulation, we will eventually be able to simulate sensation and force feedback as neural impulses with no mechanical linkage required. So-called mind-reading rigs may also do a better job deciphering our motions with zero latency and even some predictive abilities to infer intent.

What does it mean, then, if I can not only twitter my friends constantly, or in turn be-twitted, but I can now see my friends around me all day long, as virtual "ghosts" not quite haunting my active life. What happens when I’m sitting at work and my mother (who is in Florida in RL) strolls into my office to talk to me about a recipe for chicken soup? Or better yet, my son plays with his friends in an entirely virtual game of cops and robbers, running through our real neighborhood, shooting at imaginary (to everyone else) targets?

There’s not much left to hold this back except the display and sensory technology, and that’s almost ready. Laser retinal scanners will be mass-marketed in the next 2-3 years, in the way blue-tooth headsets are becoming ubiquitous today. Give it 3-5 years to solve capturing your facial expressions from such a headset, just to be safe.

Rendering, by itself, will be more than good enough in 3 years. For an ever-diminishing price, we will have completely photo-realistic views of objects in real-time, overlaid and indistinguishable from the material ones. And within 5 years, even virtual humans will cross the uncanny valley and come up the other side — especially at least when those virtual humans are driven by real ones at the other end, meaning that teleconferencing can finally do away with video feeds and go 100% virtual in the next 5 years. That alone frees up a few degrees of freedom in terms of interaction.

So that’s most of what Cory was referring to, I imagine. I don’t even think it’ll take as long as he does, except to find the compelling applications. And the part that obviously interests me the most are the issues of communication — inventing new ways to share ideas that transcend the limits of the real world we’ve come to know and overcome.

Now, back to life-logging. In all this, the one element that should be clear is how much more control I can exert over my environment, real and virtual. It’s all about adding magic and power over your world. So I still am not convinced I will ever choose to give up so much power by giving out the stream of information (most of it useless) generated by my daily activities, where I look, who I see, etc.. Using that in a sort of ‘local augmented memory’ is fine. But sending it up to Google, or Lifepress (a fictional Wordpress for lifelogging), etc.. is where I’d draw the line. (note: we already do give a lot of info for free in our daily credit transactions, but slightly less revealing).

In fact, I’d even be concerned about simply outsourcing any visual enhancement of my perceptual space to some seemingly benevolent company. What happens, for example, when that company modifies what I see to subtly persuade me to act a different way? (buy something, vote some way, etc..) 

Well, I can certainly imagine a few useful tools that would at least offer to give me more power (more magic) through recording everything I see or do. But those recordings would inevitably get out or be used by someone to mine me for profit. And so I’m still waiting to see the compelling case for such a potential power loss. It may yet happen, but on this point, I remain to be convinced.

What do you think?

Sections:

1. The Big Picture
2. Towards an Open Future
3. Conclusion

The Big Picture

Just to be clear, I’m going to move the discussion beyond the IBM system above to talk about the "big picture" of what’s coming in the future. Despite the press release, the IBM system is not quite "Google Earth" just yet, because GE is much more than spinning a globe with URLs to click — it’s really a framework for fusing just about any geospatial data together, especially user-generated content, as well as a 3D search engine for the same. But building a true "Google Earth" for the human body turns out to be a challenge many times harder than building the actual Google Earth itself (which was not easy either). Any naive implementation is doomed to be a flash in the pan, a cute but limited toy, if it works at all.

How can I be so bold in my prediction? Because I’ve been researching this beast for a number of years and I have a pretty good idea of what it will take to build it.

For starters, the human body is volumetric, Google Earth isn’t — it’s only mostly 3D — most things live on the surface of our lumpy oblate sphereoid, divided up into essentially 2D zones, with altitude added on top. You can’t (yet) fly inside the Earth and see the layers: crust, mantle, and so on or add any data there. You can’t even see a cross-section, though at least that would be easy enough to add with some graphics engine tricks.

But the biggest problem, beyond dimensionality, scalability, data storage and streaming, and even beyond 3D navigation metaphors (which are all hard enough on their own) is the one most people don’t even think about until they sit down and try to actually build something like this — topology, geography, cartography — coming up with the equivalent of latitude and longitude or even X,Y,Z, for the human body. Oh, that.

Think about it. Cartography in the geographic context has been developing for many centuries, first in a crude approximation ("here, there be monsters…") and growing closer and closer to an accurate representation of the real world ("here, there be McDonalds…").

Cartography has had time to mature, to work out solutions for problems such as: how do we best project our lumpy oblate sphereoid onto a 2D piece of paper to most accurately convey relative size and distance? And, it turns out, there are a dozen or so significant coordinate systems to answer that very question, for various purposes and to varying degrees of success. And now 3D virtual globes bring us back to a more spherical real-time reality and the field evolves  (some would say, revolutionizes) yet again.

But "human body mapping" has a technically much harder problem to solve: there is only one Earth, but there isn’t even one official human body to refer to  – more like seven billion unique ones, and not just superficially, in terms of height, weight, or sex… The location, orientation, shape and size of body parts, organs, even blood vessels, can vary even within a single person over time, as when we move or sit, are injured or sick, pregnant or tense, and especially as we age. The biospatial mapping problem is effectively four dimensional, not three.

The general solution medicine has come up with to deal with that dynamism and individuality is an overly crude and ambiguous mapping system using words like anterior and posterior, medial and lateral with simple counting systems (C1, D6), paired with names that are the stuff of nightmares for first year med students, patients, and software engineers alike. Anatomy textbooks typically resort to artistically rendered, idealized drawings and a few sample photos to help teach what goes where. And then students spend time with actual cadavers getting their gloves dirty to really understand.

Here’s an analogy to help you understand how coarse and confusing the current mapping system can be. It would be like trying to find a specific apartment at the plexus of the posteromedial canal of the Islets of Manna-hata and the posterior broad main artery of New Amsterdam. Sure, you could show up at the corner of Broadway and Canal St. in Manhattan, given that description… and a Latin-to-English dictionary… and perhaps a history book… But without an actual street address and apartment number — or the equivalent of latitude, longitude, and altitude — you’d still have to look around for a while until you found it, and then you’d have to remember its location for next time.

However, that and a magic marker are how many surgeons specify where they’ll place incisions on your body after looking at a few 2D x-rays and doing some hefty mental gymnastics. That’s one reason surgeons get paid so much, and also why there are as many dumb mistakes as there are (which goes back to cost). Just as a map of the coast could save your royal armada from doom, a correct digital dynamic map of the human body would save many lives and who knows how much money.

And the problem is inordinately harder when you talk about the human nervous system. Compared to the brain, most of the body is relatively uniform from person to person, even accounting for size — and simple too. The cortical folds, or convolutions (known as gyri and solci), are as unique as any fingerprint and apparently their topology is functionally significant as well. And if there’s any place where precision really matters, it’s in the brain. We don’t want neurosurgeons stomping around our brains like conquistadors, exploring if you will, to determine if they’re planting the national flag in the correct tissue.

Towards the Open Future

So researchers have been trying to come up with ways to map the brain that can apply to any number of people, despite our many differences and the complexity of the information. It’s an active area of research, with the ultimate goal of solving that important problem for the whole body too, from birth to death and moment to moment. It can and will be done — and hopefully, in an open way, especially if the goal is to unify doctor-patient communication, medical teaching curricula, and even scientific discourse under one framework and with one free-to-use application.

Now, that won’t stop a few eager startups from offering up solutions under dream of being "The Next Google Earth," hoping to be snatched up by Google or go public. That can’t be avoided, I suppose. But when we start talking about mapping your MRIs, CAT scans, x-rays, surgeries, dental work, and so on, managing all of your health information in a way that works from person to person and time to time, we really need to do it right from the get-go. Lives are on the line, and trillions of dollars of vested interest are looking on with intense interest. The problem is just too big, and too complex for any single entity to solve or especially own, even Google, Microsoft and IBM — and they know it, or at least they say they do. Time will tell.

Remember, Keyhole and later Google only patented certain key technical features of how it works, not the fundamental "how to map the Earth" solution. And Google’s goal is now to make KML an open standard, which is the right approach to building such a comprehensive framework to unify all geospatial data. The same kind of approach would need to apply to biospatial data, with even more work on filtering and privacy controls for the personal specifics.

But when the time comes that researchers and hobbyists alike find easy access to that fully-annotatable Google-Earth-like application for the human body, on which they can post HML ("human markup language," which has so far been focused on superficial, external, or expressed traits) or find answers, you’ll really have something special. You’ll see the same kind of emergence and new market potential we see with Google Earth and the like.

So, for example… The software might show you and your doctor your virtual body, with your own medical history of course, but now with all sorts of added information re-mapped from external data sources to match your personal details. Click on your liver, and you’ll bring up expert systems to help diagnose concerns, or reveal the latest experimental treatments, drug suggestions, or links to Alcoholics Anonymous. Medical students would have access to the standard course-work as well as the latest research, contradicting the 30-year old information that’s still being taught (and occasionally killing people) around the world.

I know several of the leading academic researchers who are working on the biospatial problem (outside of Google, of course) — especially in the brain — and it’s a been keen area of interest for me and my wife (who is a neuroscientist and neuroanatomy lecturer, as it turns out). But it will take time and patience and hard work to see the results from the various teams and companies and work through the inevitable phases of secrecy and hype to get to the other side.

Conclusion

While it’s powerful and timely, the IBM Zurich project is not exactly a new idea. Their claim is somewhat of a stretch, as is the comparison to Google Earth just yet, though their progress thus far is still impressive. In reality, some of the same people who were thinking ahead about using the Earth as the most intuitive interface for geospatial data ten, fifteen years ago were well aware of (and even doing research on) how the human body would also make the ideal interface to things like… say, patient medical records…

And that was before the general public started seeing the benefits of a 3D spatial browser like Google Earth. But what’s described from IBM research is only a sliver of the potential win here. I would consider this demo to be the "low hanging fruit" compared to what’s possible and coming in the next 3-5 years…

Keep in mind, when I wrote the old code to draw names on curving roads and near placemarks for the very first version of Keyhole’s software, I could read a few basic textbooks on cartography and apply my engineering skills to make it work in a dynamic, real-time context. When I wrote the early pre-cursor to KML (for adding UI elements and new features mostly), I chose to make up a simple lexical grammar because XML was barely known and not yet standard. In the case of mapping the human body, we’re still in 1492, very little is settled, and it’s still is a whole new world we’re trying to conquer and understand.

_________

_{P.S. apologies to Native-Americans for being on the metaphorical butt-end of my Columbus references.}

I blogged a while back about virtualization and privacy issues for both Street View and Maps/Earth, which, for starters, implies to a [predicted] future version of Street View that erases people and even cars from the imagery you see. So let’s start there.

The main reason for removing people and cars is the same reason you’d want the base Google Earth imagery to lack clouds:

• These things tend to block your view. You can’t really look behind them in a simple (essentially 2D) panoramic image.
• They only represent one snapshot in time vs. a broader/more virtualized essence of the place.
• They make it confusing to add dynamic versions of the same things on top of what’s permanently baked into the imagery. You can already see this with 3D buildings on top of 2D ones. (similarly, adding dynamic lighting and shadows is also hard when there are already shadows in the imagery)
• And in the case of people, we tend to not like being included in commercial imagery without our permission — perhaps cars and clouds feel the same way, but they don’t represent any known marketing demographic, nor do they sue.

Now, that doesn’t mean there shouldn’t ever be people or cars or clouds in Google’s version(s) of the Earth. It just means those should only be included for a much better purpose than "they just happened to get caught in the camera’s lens" and they should be first removed, and then re-added as needed.

For example, dynamic weather overlays make a lot of sense, once you remove the baked-in clouds. Moving 3D avatars also make sense in an opt-in application, e.g., in some 3D augmented-reality social network. And why not represent the real-time traffic layer with some corresponding density of cars?

All of those issues are solvable, if you first remove the items from the imagery and then add them back. The simplest method of removal is oversampling. Imagine taking multiple photos of the same place over a number of days. Most of the time, pixel for pixel, transient objects will show up in only one of the images, so you can basically vote on each pixel — majority wins. But this requires multiple passes over each city, when Google is presently racing to get coverage everywhere, and there are alignment issues, so give it some time (and perhaps some public pressure).

One obvious thing that Google can do right away is better integrate Street View with its generally-strong (though not always right) driving directions — now with impressive "route-dragging." It turns out, there’s a very compelling reason for integrating Street View with driving directions: most people can’t read maps.

Basically, there’s two kinds of people in the world — those who maintain re-orientable 2D/3D mental maps of the world around them, and those who navigate mainly by visual cues, landmarks, i.e., what they can see from their direct 1st-person perspective.

The people who can re-orient their mental maps have what we call high spatial cognition. The vast majority of the human race, however, has just enough spatial cognition to reach for a bag of potato chips and avoid walking into walls. That doesn’t mean those people are dumb — until recently, high spatial cognition was generally only useful for throwing spears. There are many other kinds of intelligence. And those with high spatial intelligence are simply more cut out for jobs in 3D graphics, architecture, industrial design, and perhaps billiards. People who can’t tell the difference between panoramic 2D images and actual 3D graphics might be really good at business or marketing, for example, but still get lost in a cul-de-sac.

The point is that driving directions, for the vast majority of the world, would do much better with a series of 1st person images or video that shows you where to get off the highway with an actual image of the exit sign and off-ramp, saying, "turn right here" in the same way that one might say to a driver, "turn where that blue car just went." That, and as I discovered this weekend, the Home Depot on Route 17 is not quite where Google Maps says it us. Having images of my route would be a nice "sanity check."

Frankly, I’d expect this feature way before true virtualization as described above. But the closest they come to it right now is that you can use Street View when interactively examining your driving directions — drag the little yellow AOL-like guy around to see just one Street View image at a time. Perhaps they’ll soon offer to replace or combine the series of overhead street maps (which they include when you print) with 1st person views of each stage of navigation along the route.

What might come after that? The main thing to do, I’m guessing, is integrate the vast quantities of Street View data with Google Earth in a more 3D fashion, solving these 2D panoramas for depth and turning everything into textures and polygons.

The ultimate goal would be a view of the Earth not only with basic buildings, but with well detailed street-level views so you could zoom down and even [virtually] walk around. Go one step further, and Google could take their $10 per store photography project and recreate virtual interiors of semi-public places, perhaps with links or embedded shopping when you go inside.

That’s when I think we might start to see avatars in Google Earth — when there’s some actual reason for people to have a sense of their own body and of other people in the virtual world, not that a Second Life / GE combination is likely, for reasons I’ve already outlined.

What else might there be? Well, integrating live street cameras would be a neat trick. All it really takes is knowing the position and orientation of the camera and coming up with a flash video player that can project 2D movies on an arbitrary 3D polygon, rather than always in the plane of your screen. It’s even simpler if the video is in a separate window.

Having the density of image capture points (aka "nodes") be high enough to resemble moving video is also on the horizon. The cameras Google is reportedly driving around with are capable of capturing and processing 30 FPS and 360-degrees [edit: see the first comment below for a cool example of panoramic video]. The only real limitation is server-side storage for the higher density of nodes (not a big deal for Google) and bandwidth to your browser. Caveat: virtualizing the scene to a real-time 3D model is still preferable to streaming video because we don’t always want to move in the exact path (and direction) that Google took. Just imagine trying to drive down a street backwards — with Street View Video, time would seem to run backwards.

Clickable store-front information is an obvious must-have feature — show menus for restaurants, and so on. And like Google Earth, we’ll want to turn on information layers beyond the mere street names. Given the vast store of information already in Google Earth, adding this to Street View shouldn’t be too hard — even rendering KML should be doable, given enough time and energy.

The only really hard problem is the one I mentioned earlier — turning essentially 2D panoramic Street View images into 3D models — the start of the art still requires some manual labor, which kind of hampers any planet-scale effort. But new camera technology for recording distance per pixel could soon save the day, as well as an interesting recent acquisition.

At that point, and as browsers are more and more capable of 3D, I expect to see Google Street View, Maps, and Earth become one truly integrated product, offering different perspectives on the same digital earth.

As always, I’m merely an observer. Google doesn’t tell me their plans. But let’s check back in a year or so and see how much of this comes to pass.

After reading an article called "How Google Earth Works" on the great site HowStuffWorks.com, it became apparent that the article was more of a "how cool it is" and "here’s how to use it" than a "how Google Earth [really] works."

So I thought there might be some interest, and despite some valid intellectual property concerns, here we are, explaining how at least part of Google Earth works.

Keep in mind, those IP issues are real. Keyhole (now known as Google Earth) was attacked once already with claims that they copied someone else’s inferior (IMO) technology. The suit was completely dismissed by a judge, but only after many years of pain. Still, it highlights one problem of even talking about this stuff. Anything one says could be fodder for some troll to claim he invented what you did because it "sounds similar." The judge in the Skyline v. Google case understood that "sounding similar" is not enough to prove infringement. Not all judges do.

Anyway, the solution to discussing "How Google Earth [Really] Works" is to stick to information that has already been disclosed in various forms, especially in Google’s own patents, of which there are relatively few. Fewer software patents is better for the world. But in this case, more patents would mean we could talk more openly about the technology, which, btw, was one of the original goals of patents — a trade of limited monopoly rights in exchange for a real public benefit: disclosure. But I digress…

For the more technically inclined, you may want to read these patents directly. Be warned: lawyers and technologists sometimes emulsify to form a sort of linguistic mayonnaise, a soul-deadening substance known as Patent English, or Painglish for short . If you’re brave, or masochistic, here you go:

1. Asynchronous Multilevel Texture Pipeline
2. Server for geospatially organized flat file data

There are also a few more loosely related Google patents. I don’t know why these are shouting, but perhaps because they’re very important to the field. I’ll hopefully get to these in more detail in future articles:

3. DIGITAL MAPPING SYSTEM
4. GENERATING AND SERVING TILES IN A DIGITAL MAPPING SYSTEM
5. DETERMINING ADVERTISEMENTS USING USER INTEREST INFORMATION AND MAP-BASED LOCATION INFORMATION
6. ENTITY DISPLAY PRIORITY IN A DISTRIBUTED GEOGRAPHIC INFORMATION SYSTEM (this one will be huge)

And there is this more informative technical paper from SGI (PDF) on hardware "clipmapping," which we’ll refer to later on. Michael Jones, btw, is one of the driving forces behind Google Earth, and as CTO, is still advancing the technology.

I’m going to stick closely to what’s been disclosed or is otherwise common technical knowledge. But I will hopefully explain it in a way that most humans can understand and maybe even appreciate. At least that’s my goal. You can let me know.

Big Caveat: the Google Earth code base has probably been rewritten several times since I was involved with Keyhole  and perhaps even after these patents were submitted. Suffice it to say, the latest implementations may have changed significantly. And even my explanations are going to be so broad (and potentially out-dated) that no one should use this article as the basis for anything except intellectual curiosity and understanding.

Also note: we’re going to proceed in reverse, strange as it may seem, from the instant the 3D Earth is drawn on your screen, and later trace back to the time the data is served. I believe this will help explain why things are done as they are and why some other approaches don’t work nearly as well.

Part 1, The Result: Drawing a 3D Virtual Globe

There are two principal differences between Google Maps and Earth that inform how things should ideally work under the hood. The first is the difference between fixed-view (often top-down) 2D & free-perspective 3D rendering. The second is between real-time and pre-rendered graphics. These two distinctions are fading away as the products improve and converge. But they highlight important differences, even today.

What both have in common is that they begin with traditional digital photography — lots of it — basically one giant high-resolution (or multi-resolution) picture of the Earth. How they differ is largely in how they render that data.

Consider: The Earth is approximately 40,000 km around the waist. Whoever says it’s a small world is being cute. If you stored only one pixel of color data for every square kilometer of surface, a whole-earth image (flattened out in, say, a mercator projection) would be about 40,000 pixels wide and roughly half as tall. That’s far more than most 3D graphics hardware can handle today. We’re talking about an image of 800 megapixels and 2.4 gigabytes at least. Many PCs today don’t even have 2GB of main memory. And in terms of video RAM, needed to render, a typical PC has maybe 128MB, with a high-end gaming rig having upwards of 512.

And remember, this is just your basic run-of-the-mill one-kilometer-per-pixel whole-earth image. The smallest feature you could resolve with such an image is about 2 kilometers wide (thank you, Mr. Nyquist) — no buildings, rivers, roads, or people would be apparent. But for most major US cities, Google Earth deals in resolutions that can resolve objects as small as half a meter or less, at least four thousand times denser, or sixteen million times more storage than the above example.

We’re talking about images that would (and do) literally take many terabytes to store. There is no way that such a thing could ever be drawn on today’s PCs, especially not in real-time.

And yet it happens every time you run Google Earth.

Consider: In a true 3D virtual globe, you can arbitrarily tilt and rotate your view to pretty much look anywhere (except perhaps underground — and even that’s possible if we had the data). In all 3D globes, there exists some source data, typically, a really high-resolution image of the whole earth’s surface, or at least the parts for which the company bought data. That source data needs to be delivered to your monitor, mapped onto some virtual sphere or ideally onto small 3D surfaces (triangles, etc..) that mimic the real terrain, mountains, rivers and so on.

If you, as a software designer, decide not allow your view of the Earth to ever tilt or rotate, then congrats, you’ve simplified the engineering problem and can take some time off. But then you don’t have Google Earth.

Now, various schemes exist to allow one to "roam" part of this ridiculously large texture. Other mapping applications solve this in their own way, and often with significant limitations or visual artifacts. Most of them simply cut their huge Earth up into small regular tiles, perhaps arranged in a quadtree, and draw a certain number of those tiles on your screen at any given time, either in 2D (like Google Maps) or in 3D, like Microsoft’s Virtual Earth apparently does.

But the way Google Earth solved the problem was truly novel, and worthy of a software patent (and I am generally opposed to software patents). To explain it, we’ll have to build up a few core concepts. A background in digital signal theory and computer graphics never hurts, but I hope this will be easy enough that that won’t be necessary.

I’m not going to explain how 3D rendering works — that’s covered elsewhere. But I am going to focus on texture mapping and texture filtering in particular, because the details are vital to making this work. The progression from basic concepts to the more advanced texture filtering will also help you understand why things work this way, and just how amazing this technology really is. If you have the patience, here’s a very quick lesson in texture filtering.

The Basics

The problem of scaling, rotating and warping basic 2D images was solved a long time ago. The most common solution is called Bilinear Filtering. All that really means is that for each new (rotated, scaled, etc..) pixel you want to compute, you take the four "best" pixels from your source image and blend them together. It’s "bilinear" because it linearly blends two pixels at a time (along one axis), and then linearly blends those two results (along the other axis) for the final answer.

[A "linear blend," in case it's not clear, is butt simple: take 40% of color A, and 60% of color B and add them together. The 40/60 split is variable, depending on how "important" each contributor is, as long as the total adds up to 100%.]

That functionality is built into your 3D graphics hardware such that your computer can nowadays do literally billions of these calculations per second. Don’t ask me why your favorite paint program is so slow.

The problem being addressed can be visualized pretty easily — that’s what I love about computer graphics. It turns out, whenever we map some source pixels onto different (rotated, scaled, tilted, etc…) output pixels, visual information is lost.

The problem is called "aliasing" and it occurs because we digitally sampled the original image one way, at some given frequency (aka resolution), and now we’re re-sampling that digital data in some other way that doesn’t quite match up.

1. A simple low-res (11×11 pixel) image is about to be rotated. (the grid lines are merely to delineate pixels) 3. Close up of one output pixel. Bilinear interpolation averages the "best" four source pixels for each new destination pixel (shown as black border with white dots) based on their relative importance (ideally: fractional area).

2. Each pixel in the destination grid overlaps multiple pixels from the rotating original. 4. After bilinear interpolation, the resulting rotated image has some clear (or rather blurry) issues.

Now, when we talk about output pixels and destinations, it doesn’t much matter if the destination is a bitmap in a paint program or the 3D application window that shows the Earth. Aliasing happens whenever the output pixels do not line up with the sampling interval (frequency, resolution) of the source image. And aliasing makes for poor visual results. Dealing with aliasing is about half of what texture mapping is all about. The rest is mostly memory management. And the constraints of both inform how Google Earth works.

The mission then is to minimize aliasing through cleverness and good design. The best way to do this is to get as close as possible to a 1:1 correspondence between input and output pixels, or at least to generate so many extra pixels that we can safely down-sample the output to minimize aliasing (also known as "anti-aliasing"). We often do both.

Consider: for resizing images, it only gets worse — each pixel in your destination image might correspond to hundreds of pixels of source imagery, or vice-versa. Bilinear interpolation, remember, will only pick the best four source pixels and ignore the rest. So it can therefore skip right over important pixels, like edges, shadows, or highlights. If some such pixel is picked for blending during one frame and skipped over subsequently, you’ll get an ugly "pixel-popping" or scintillation effect. I’m sure you’ve seen it in some video games. Now you know why.

Tilting images (or any 3D transformation) is even more problematic, because now we have elements of scaling and rotation, but also a great variation in pixel density across rendered surfaces. For example, in the "near" part of a scene, your nice high-res ground image might be scaled up such that the pixels look blurry. In the "far" part of the scene, your image might appear scintillated (as above) because simple 2×2 bilinear interpolation is necessarily skipping important visual details from time to time.

Copyright, Microsoft Virtual Earth Here’s an example of where a certain kind of texture filtering causes poor results. The text labels are hardly readable (why they’re painted into the terrain image at all is another issue).

Better Filtering, Revealed

Most consumer 3D hardware already supports what’s called "tri-linear" filtering. With tri-linear and a closely coupled technique called mip-mapping, the hardware computes and stores a series of lower resolution versions of your source image or texture map. Each mip-map is automatically down-sampled by a factor of 2, repeatedly, until we reach a 1×1 pixel image whose color is the average of all source image pixels.

So, for example, if you provided the hardware with a nice 512×512 source image, it would compute and store 8 extra mip-levels for you (256, 128, 64, 32, 16, 8, 4, 2, and 1 pixel square). If you stacked those vertically, you might more easily visualize the "mip-stack" as an upside down pyramid, where each mip-level (each horizontal slice) is always 1/2 the width of the one above.

Drawing of a mip-map pyramid (not to scale). The X depicts trilinear filtering sampling two mip-levels for an in-between pixel to reduce aliasing.

During 3D rendering, mip-mapping and tri-linear filtering take each destination pixel, pick the two most appropriate mip-levels, essentially do a bi-linear blend on both, and then blend those two results again (linearly) for the final tri-linear answer.

So for example, say the next pixel would have no aliasing if only the source image had a resolution of 47.5 pixels across. The system has stored power of two mip maps (16, 32, 64…). So the hardware will cleverly use the 64×64 and 32×32 pixel versions closest to the desired sampling of 47.5, compute a bilinear (4-sample) result for each, and then take those two results and blend them a third time.

That’s tri-linear filtering in a nutshell, and along with mip-mapping, it goes a great distance to minimizing aliasing for many common cases of 3D transformations.

Remember: so far, we’ve been talking about nice, small images, like 512×512 pixels. Our whole-earth image will need to be millions of pixels across. So one might consider making a giant mip-map of our whole-earth image, at say one meter resolution. No problem, right? But you’ll realize fairly soon that would require a mip-map pyramid 26 levels deep, where the highest resolution mip-level is some 66 million pixels across. That simply won’t fit on any 3D video card on the market, at least not in this decade.

I’m guessing Microsoft’s Virtual Earth gets around this limit by cutting their giant earth texture into many smaller distinct tiles of, say, 256 pixels square, where each gets mip-mapped individually. That approach would work to an extent, but it would be relatively slow and give some of the visual artifacts, like the blurring we see above, and a popping in and out of large square areas as you zoom in and out.

There’s one last concept about mip-maps to understand before we move on to the meat of the issue. Imagine for a moment that the pixels in the mip-map pyramid are actually color-coded as I’ve indicated above, with an entire layer colored red, another yellow, etc.. Drawing this on a tilted plane (like the Earth’s ground "plane") would then seem to "slice through" the pyramid at an interesting angle, using only those parts of the pyramid that are needed for this view.

It’s this characteristic of mip-mapping that allows Google Earth to exist, as we’ll see in a minute.

A typical tilted Google Earth image (copyright & courtesy Google).

The same view, using color to show which mip levels inform which pixels

The example on the left shows a normal 3D scene from Google Earth, as well as a rough diagram showing from where in the mip-stack a 3D hardware system might find the best source pixels, if they were so colorized.

The nearer area gets filled from the highest-resolution mip-level (red), dropping off to lower and lower resolutions as we get farther from the virtual point of view. This helps avoid the scintillation and other aliasing problems we talked about earlier, and looks quite nice. We get as close as possible to a 1:1 correspondence between source and destination, pixel for pixel, so aliasing is minimized.

Even better still, tri-linear filtering 3D graphics hardware has been improved with something called anisotropic filtering (a simple preference option in Google Earth) which is essentially the same core idea as the previous examples, but using non-square filters, beyond the basic 2×2. This is very important for visual quality, because even with fancy mip-mapping, if you tilt a textured polygon to a very oblique angle, the hardware must choose a low-resolution mip-level to avoid scintillation on the narrow axis. And that means the whole polygon is sampled at too-low a resolution, when it’s only one direction that needed to dip down to the low-res stuff. Suffice it to say, if your hardware supports anisotropic filtering, turn it on for best results. It’s worth every penny.

Now, to the meat of the issue

We still have to solve the problem of how to mip-map a texture with millions of pixels in either dimension. Universal Texture (in the Google Earth patent) solves the problem while still providing high quality texture filtering. It creates one giant multi-terabyte whole-earth virtual-texture in an extremely clever way. I can say that since I didn’t actually invent it. Chris Tanner figured out a way to do on your PC what had only ever been done on expensive graphics supercomputers with custom circuitry, called Clip Mapping (see SGI’s pdf paper, also by Chris, Michael, et al., for a lot more depth on the original hardware implementation). That technology is essentially what made Google Earth possible. And my very first job on this project was making that work over an internet connection, way back when.

So how does it actually work?

Well, instead of loading and drawing that giant whole-earth texture all at once — which is impossible on most current hardware — and instead of chopping it up into millions of tiles and thereby losing the better filtering and efficiency we want, recall from just above that we typically only ever use a narrow slice or column of our full mip-map pyramid at any given time. The angle and height of this virtual column changes quite a bit depending on our current 3D perspective. And this usage pattern is fairly straightforward for a clever algorithm to compute or infer, knowing where you are and what the application is trying to draw.

A Universal Texture is both a mip-map, plus a software emulated clip-stack, meaning it can mimic a mip-map of many more levels and greater ultimate resolution than can fit in any real hardware. Note: though this diagram doesn’t depict it as precisely as the paper, the clip stack’s "angle" shifts around to best keep the column centered.

So this clever algorithm figures out which sections of the larger virtual texture it needs at any given time and pages only those from system memory to your graphics card’s dedicated texture memory, where it can be drawn very efficiently, even in real-time.

The main modification to basic mip-mapping, from a conceptual point of view, is that the upside down pyramid is no longer just a pyramid, but is now much, much taller, containing a  clipped stack of textures, called, oddly enough, a "clip stack," perhaps 16 to 30+ levels high. Conceptually, it’s as if you had a giant mip-map pyramid that’s 16-30 levels deep and millions to billions of pixels wide, but you clipped off the sides — i.e., the parts you don’t need right now.

Imagine the Washington monument, upside down and you’ll get the idea. In fact, imagine that tower leaning this way or that, like the one in Pisa, and you’ll be even closer. The tower leans in such a way that the pixels inside the tower are what you need for rendering right now. The rest is ignored.

Each clip-level is still twice the resolution of the one "below" it, like all mip-maps, and nice quality filtering still works as before. But since the clip stack is limited to a fixed but roaming footprint, say 512×512 pixels wide (another preference in Google Earth), that means that each clip-level is both twice the effective resolution and half the coverage area of the previous. That’s exactly what we want. We get all the benefits of a giant mip-map, with only the parts relevant to any given view.

Put another way, Google Earth cleverly and progressively loads high-res information for what’s at the focal "center" of your view (the red part above), and resolution drops off by powers of two from there. As you tilt and fly and watch the land run towards the horizon, Universal Texture is optimally sending only the best and most useful levels of detail to the hardware at any given time. What isn’t needed, isn’t even touched. That’s one thing that makes it ultra-efficient.

It’s also very memory-efficient. The total texture memory for an earth-sized texture is now (assuming this 512 wide base mip-map, and say 20 extra clip-levels of data) only about 17 megabytes, not the dozens to hundreds of terabytes we were threatened with before. It’s actually doable, and worked in 1999 on 3D hardware that had only 32 MB or less. Other techniques are only now becoming possible with bigger and bigger 3D cards.

In fact, with only 20 clip-levels (plus 9 mip levels for the base pyramid), we see that 2²⁹ yields a virtual texture capable of up to 536 million pixels in either dimension. Multiply that by 1/2 vertically, gives an virtual image of a few hundred terapixels in area, or enough excess capacity to represent features as small as 0.15 meters (about 5 inches), wherever the data is available. And that’s not the actual limit. I simply picked 20 clip levels as a reasonable number. And you thought the race for more megapixels on digital cameras was challenging. Multiply that by a million and you’re in the planetary ballpack.

Fortunately, for now, Google only really has to store a few dozen terapixels of imagery. The other beauty of the system is that the highest levels of resolution need not exist everywhere for this to work. Wherever the resolution is more limited, wherever there are gaps, missing data, etc.. the system only draws what it has. If there is higher resolution data available, it is fetched and drawn too. If not, the system uses the next lower resolution version of that data (see mip-mapping above) rather than drawing a blank. That’s exactly why you can zoom into some areas and see only a big blur, where other areas are nice and crisp. It’s all about data availability, not any hard limit on the 3D rendering. If the data were available, you could see centimeter resolution in the middle of the ocean.

The key then to making this all work is that, as you roam around the 3D Earth, the system can efficiently page new texture data from your local disk cache and system memory into your graphics texture memory. (We’ll cover some of how stuff gets into your local cache next time). You’ve literally been watching that texture uploading happen without necessarily realizing it. Hopefully, now you will appreciate all the hard work that went into making this all work so smoothly — like feeding an entire planet piecewise through a straw.

Finally, there’s one other item of interest before we move on. The reason this patent emphasizes asynchronous behavior is that these texture bits take some small but cumulative time to upload to your 3D hardware continuously, and that’s time taken away from drawing 3D images in a smooth, jitter-free fashion or handling easy user input — not to mention the hardware is typically busy with its own demanding schedule.

To achieve a steady 60 frames per second on most hardware, the texture uploading is divided into small, thin slices that very quickly update graphics video memory with the source data for whatever area you’re viewing, hopefully just before you need it, but at worst, just after. What’s really clever is that the system needs only upload the smallest parts of these textures that are needed and it does it without making anyone wait. That means rendering can be smooth and the user interface can be as fluid as possible. Without this asynchronicity, forget about those nice parabolic arcs from coast to coast.

Now, other virtual globes can also virtualize the whole-earth texture, perhaps they cut it into tiles, and even use multiple power-of-two resolutions like GE does. But without the Universal Texturing component or something better, they’ll either be limited to 2D top-down rendering, or they’ll do 3D rendering with unsatisfying results, blurring, scintillation, and not nearly as good performance for streaming the data from the cache into texture memory for rendering.

And that’s probably more than you ever wanted to know about how the whole Earth is drawn on your screen each frame.

Now that Linden Labs has open-sourced the Second Life client, if any Google Earth engineers chose to study it, I might no longer be the only person lucky enough to know both the Google Earth and Second Life internals well enough to make a bold statement on a mashup of the two. It would be great if others (besides me) could do so soberly. Because all I’m hearing lately is a lot of “wouldn’t it be great?” and not much “here’s how” and, better yet, “here’s why” practical discussion.

I’ve said before that I don’t think any direct integration between SL and GE is wise at this point, at least not in their present forms and with the present missions of each application. I’ll try to elaborate on that and see if I can convince you that the best thing overall is for each app to evolve along its own path. But I will point out a few areas where each could benefit from at least mimicking the other.

[Edit: See Wade’s comment below. I want to also make it clear that the issues of a SL/GE mashup are purely hypothetical, and are meant more to serve as a framework for understanding some important technical and usability issues in these two different kinds of virtual worlds.]

The issues:

Mirror Worlds vs. Virtual Worlds – Second Life is fictional, whimsical, experimental. Google Earth is a reflection of the real world, and according to its CTO, will remain so in the future. In SL, you can make or do almost anything. GE is meant to be a platform for delivering geo-referenced information that is strictly relevant and useful (as well as fun) for our first lives.

Now, GE bent those rules a little by adding custom layers and 3D import — there’s no clear rule that says your KML file has to represent a real object or building. And KML search will have a hard time distinguishing between real vs. fictional search results (even as they add pagerank-style features). The problem is that fictional results pollute real-world uses, and if they get out of hand, Google will have to somehow segregate those realms. I definitely don’t want to search for directions to the Home Depot and find that my path is blocked by a giant robot or a self-replicating pile of poo.

And for SL, the opposite problem exists — growth of the landscape was designed to be geometric and ad hoc, not mirroring any real-word geography. There’s some exceptions to that as well, one of which I suggested to a SL-oriented colleague a while back, and that is that a single SL sim could serve as a stand-in for the entire earth, in miniature, linking via teleports to any number of other 1:1 scale sims to flesh out the real-earth geography, albeit with giant gaps — meaning you’d have to pop back to the “hub” to get from place to place — so you can see some of the difficulties.

Direct Integration – In this pure hypothetical, we’re talking about the two companies working together (e.g., collaboration, or outright purchase/merger) or GE open sourcing their code so a third party can do the mashup. I don’t think that’ll happen anytime soon. Now, if Google can pay $1.7B for YouTube, I think anything is possible on the merger front. Money seems to be no object, but they’re not being irrational. For the price Linden might demand, they’ll need to show, say, 10-20M regular visitors and a revenue model that Google thinks will integrate with their core business and scale indefinitely.

I’d also guess, purely based on rumors and the engineering personality type, Google would be more likely to try rolling their own social VW before buying an established company like Linden. If Google does that, and they do as well as Google Video did against YouTube, that’s when I think buying Linden would make more sense. I don’t expect it to happen, if at all, for several years. An IBM-style purchase is more likely, IMO. But what do I know?

Mashups – This is a bit less speculative. What Google could do fairly easily in the near term is release a closed-source version of GE that is more like a toolkit or library, closer to the Microsoft model for their Virtual Earth offering, though designed for real-time rendering. A toolkit would allow someone to build a virtual environment that had both the real earth and whatever avatar systems they wanted to throw in. It wouldn’t necessarily be SL, but it would be the first social VW based on the real/mirror Earth. More likely than that, though, is some licensing specific deal for the GE technology if the price is right (I don’t know what that price is).

It would also be technically feasible for Google to do the reverse — embed Google Earth’s OpenGL rendering code into the open-sourced SL client, such that anyone using that new SL client could create an instance of a GE globe inside SL as an in-world application as opposed to the actual terrain you stand on. The only problem would be licensing, if that’s an issue at all. But without the analogous GE source or any license to use GE’s umpteen terabytes of data, neither Linden nor any 3rd parties would be in any position to do this from their end.

Client / Server – The one mashup that pundits seem to call for the most — a horde of real, active, chatting Second Life avatars inside one big shared Google Earth — is unlikely to happen in the near-term for several technical reasons. One is that SL is much more than the PC client you install. Without the requisite number of SL servers arranged into a lat/long map of the earth, or at least the parts we care about, none of your avatars would be able to interact. So while mashing-up the client you’d also need to handle both SL and GE servers at the same time, preferably in some coordinated fashion. For example, if GE holds the terrain and building data, SL servers would need to know about those to perform the physical part of the simulation, collision detection and so on. So there’s at least a three-way dialog going on, not just an integrated SL/GE client. And that makes us engineers very unhappy.

Putting just the bodies in GE is easier, although we’re talking the cheesy, lame, unsupported version with very little interactivity. You can capture your avatar in a static pose from SL using an OpenGL based interceptor, convert that to Collada format and then KML and import it into GE. Animating it is trickier. Forget about dancing or even moving your limbs. The best you can probably hope for without extensive kludge is using KML’s “network update” feature to fetch your avatar’s current location (and that of anyone else near you) from a special server you’d supply. In this case, you also won’t get collision detection with 3D buildings unless you do it yourself. And similarly, you’ll have to handle all interactions among avatars. Essentially, you’ll be recreating a kludgey version of SL’s simulators and using GE + KML pseudo-scripting as the client rendering engine. It’s not something I’d want to spend my time on.

Scale – Given the land area of the earth is reportedly 150,000,000km², and each SL server currently handles 256×256m (16 servers per km²), a quick calculation reveals roughly 2.4 billion servers would be needed for the land alone. If we just did Manhattan (87.46 km²) , it would take about 1400 servers, or roughly 1/4th of their current total just for one densely populated island. And we’re not even talking about the limits on concurrent users yet.

Clearly, if GE and SL were ever to mate, SL would need to move from a rigid grid to something more adaptable, for example, a quad-tree that gets subdivided depending on where the people virtually are, or better yet, based on who is interacting with whom at any given time. I worked with a company briefly (they never got their funding) who was pursuing something like this to handle 1 million simultaneous users. It’s not a trivial problem, but would be necessary to scale SL up to a full planet, if they ever see the need.

The next issue of scale is that of “levels of detail.” GE was designed from the ground up to seamlessly zoom through 20 or more powers of two (2²⁰ : 1 scaling) when zooming from way out in space down to a spot on the ground. SL was designed to let you walk, teleport, or fly relatively close to the ground. It essentially has four levels of detail — near, far, and off, plus we’ll count the nice 2D overhead map. But SL isn’t designed to handle viewing the entire world in so many different scales, smoothly moving through all of them. Without big architectural changes, it would be very difficult to do the kind of flying and zipping around one does in GE.

One final note under the category of “scale.” The Collada file format may have some built-in features to properly convey the scale of objects, but it’s rarely used or even obeyed from what I can tell. All 3D models have built-in assumptions about scale. They’re just a bunch of numbers after all, not smart about their context or nature. If someone exports data in centimeters and another program assumes those are feet or meters, you’re going to have to write yet another program to convert everything to match up, or your avatar may be as tall as the Empire State Building, or too small to see.

Other issues – I haven’t even touched on the issues of converting from procedural or parametric objects to polygons, which is something I have a lot of experience with. That’s a whole other discussion which I could spend hours on. Suffice it to say, it’s a hard problem, but one in which there is an easy solution — if it’s designed in to the system.

But for now, let’s worry more about whether or not people want this and then get around to fixing the “nitty gritties.”

The How’s — What’s really possible?

1. If Google is so inclined, I’d love to see them create a free or licensed DLL that encapsulates the OpenGL rendering code from Google Earth in a form that could be invoked inside of someone else’s OpenGL-based 3D engine. It would at a minimum require functions to set the viewing position, layer state, etc… and probably most things you would already do from the UI. This would at least allow someone to put a Google Earth inside Second Life, as a 3D object one could poke at. It would not be a regular place in SL, but even that could be improved with time. If Google figures out how to monetize GE with ads, such a DLL might require showing those ads as well, or we could see it as a one-off license to some well-funded company.

2. If Linden is so inclined, they could work on a new kind of “grid” that could scale up to a full earth-sized geography, solving the problem of zooming in and out as a method of managing that new expansive scope. FWIW, I think it’s worth their time, because teleporting is a bad way work around the problems of walking and flying. GE’s zooming gets you there just as fast, but without losing your geo-spatial awareness. In other words, the continuous pan across the earth keeps your brain working on the relative positions of the places you visit, as in a big mental map. Teleporting loses the advantages of having one big shared space — you might as well have a bunch of small connected rooms at that point, which some people are working on. If Linden does this, they could be in a position to build an “Earth” app like in SnowCrash — one that they can even zoom into as a method of getting around. This need not be a literal mirror of the earth, but the technology I outlined is still important to make it work, regardless of the source data; satellites or users.

The Why’s — What will people actually want or need?

Here’s the big question, saved for the end. With all the “wouldn’t it be cool” talk, people seem miss the fundamental issue with new technology: people do the dumbest things. It’s not that people are necessarily dumb. It’s more an issue of finding the “lowest-energy” solutions rather than the most elegant ones.

MySpace is a good example of a “low-energy” solution, as compared to many more elegant ones that were presented. And there is no doubt that there will be a host of 3D MySpaces within the next 12 months to further test the theory (I’ve consulted for several but I’m not promoting any). All of them will face a fundamental problem: What is 3D good for? More specifically, why would I want to visit your virtual living room if you’re not there? Those questions have real answers, but perhaps not the most obvious ones.

Now, there are also some great reasons to want avatars in Google Earth. The more GE becomes a destination, rather than a tool to pick your destination, the more likely we are to say “meet me on 5th avenue for shopping,” and we mean the virtual version of each. This requires the avatars to interact. And if we go a little further to an application where I and a paid designer can work on my new house, you can see a need for GE to add in-world building tools to the mix.

As for a virtual economy and social networking tools, I don’t know. As long as Google doesn’t have to host your content themselves or care about bandwidth for user data, there really is no cost to them, and no big reason to quantify value for 3D objects. The “layer” approach makes it much easier to simply turn off content that’s just too slow. But for all of these, I see them being so tightly integrated with Google’s advertising economy and information delivery mission that I can only imagine them doing this in-house, slowly but surely adding more metaverse-like components that are more or less strictly tied to the real world.

Similarly, the path for SL is to make a bigger and bigger world and scale their way towards more and more simultaneous users. The “why” of making an entire earth in Second Life seems obvious — but the old joke “it’s a small world, but I wouldn’t want to paint it” comes to mind. If SL users really had 2.3 billion servers to populate, could 10 – 20 million users even do it in one second lifetime? Not by hand, I think.

GE at least has automated tools for capturing a world that’s already out there. Imagine if people really had to build a second version of the whole world by hand, with the level of detail SL desires?

It’s unlikely. But that’s where procedural modeling comes in. It’s not going to recreate the real world either, but there are known and waiting methods for building out entire planets full of detail. However, that’s clearly a third way — not user generated, and not mirroring the real world. And that discussion is best left for another day.

Sections:

1. Scenegraphs: a brief history and evolution,
2. Scenegraphs Today
3. Scenegraphs Future

In the Beginning…

To help understand where scenegraphs came from, it’s useful to take a quick look at the evolution of graphics languages like OpenGL and DirectX. Early on, real-time graphics existed on special image generation (IG) hardware that contained entire visual databases in closed proprietary form. Modellers created their databases and loaded them onto the hardware IG. Programmers were generally limited to modifying elements of these databases, like the position and rotation of a helicopter or setting the time-of-day.

SGI introduced a more open and programmable option for image generation hardware and along with it, graphical languages that allowed more direct programmability of the image pipeline. OpenGL (from SGI’s original “GL”) consists of a stream of primitive drawing commands (draw polygon, line, point, etc..) state settings (set color, texture, etc..) and matrix manipulations (push/pop to model-view or perspective matrix, etc..). But it contains very little information that allows the system to self-optimize and improve performance.

This was fine for drawing all sorts of scenes. But polygons that are out of view do consume resources – the hardware doesn’t even know they’re out of view until very late in the rendering pipeline. Unnecessary state changes, extra texture loads, and other common graphics procedures are best avoided if they don’t contribute to the final image.

Culling

Culling is the process of removing everything from a scene that will not contribute to the final image, including things that are behind the observer, off-screen, or, in more advanced systems, hidden behind other objects (i.e., occluded). Generalized frustum culling works by comparing each object’s spatial boundaries with a viewing frustum – a truncated pyramid that represents the visible volume of space. OpenGL does this implicitly when you send it polygons – by default, it transforms and clips all polygons to the edges of the viewing volume (most hardware uses a combination of gross clipping and 2D scissoring, but that’s a bit too detailed for this section of the article).

Rather than do the heavy work at the OpenGL and polygon level, scenegraph architects realized they could better perform culling at higher level abstractions for greater efficiency. If we can remove the invisible objects first, then we make the hardware do less work and generally improve performance and the all-important frame-rate.

The way it works is fairly straightforward. Any object that is entirely within the viewing frustum is sent on down to the hardware. For objects that are part in/part out, we usually don’t bother checking individual polygons on the CPU, but we might break a very complex object into several simpler ones so some of them may be culled in or out individually. Of course, any object that is entirely outside the culling volume is rejected early on.

Hierarchy

To efficiently perform this calculation, it’s beneficial to organize the objects into a hierarchy or tree, propagating any shared information towards the root of the tree. There are many kinds of trees we could use. But lets keep it to a simple one-parent N-child hierarchy — a directed acyclic graph or DAG.

Such a basic scenegraph will have a root node, with one or more children. Each child node can in turn contain zero or more children, some of which will be the graphical objects we want to draw. The other nodes are there for structural purposes and can get quite complex, as we’ll see later on.

For example, if a building was composed of rooms, a group node of the scenegraph (call it “Building”) might contain several nodes (called “room-0″ “room-1″ and so on). The bounding box of the “Building” node would be defined such that it contains the bounding boxes of all of the rooms. So if the building node was determined to be invisible, then there would be no need to check the child nodes since they would also be invisible.

Another benefit of hierarchy was in ease of manipulation. Given a car containing doors and wheels, it was much easier to move the “car” node and have the child nodes (doors and wheels) follow automatically. Without a hierarchy, one might probably have to move each of these sub-objects synchronously each time the car moved. Of course, that could be solved with some clever back-pointers among dependent matrices, but that’s exactly what scenegraphs are doing in a more formal fashion.

So for example, consider a tank. It might have the following hierarchical representation:

By splitting the object into “nodes” and representing the connectivity between these nodes, we can better manipulate the final polygons of the tank.

We can animate pieces separately. We can rotate the turret, fire the gun, and open the hatch. We can animate the left and right tread to simulate turning.

Rendering Advantages — State Sorting

Scenegraphs showed clear benefits for improving rendering performance and making more optimal use of the available hardware resources. By keeping a “retained” model of the virtual world, scenegraphs could make additional optimizations, such as parallel processing culling and drawing, and most importantly: state sorting.

State sorting is a concept whereby all of the objects being rendered are sorted by similarities in state (texture map, lighting values, transparency, and so on). Since changing state is often an expensive operation due to hardware implementations, this is usually a big performance win, even on the newest hardware. A good example of this is turning lighting on and off — imagine a generic SIMD hardware architecture, executing the same code over four parallel geometry processors. There may be one version of the code for “lit” objects and one version for “unlit.” Changing from lit to unlit state can cause all four processors to flush and reload. But if we can try to turn lighting on or off only once per frame instead of once per object, we can improve performance.

For an even stronger example, imagine we were drawing 100 cars, each containing some polygons in metal (state 1), rubber (state 2) and glass (state 3), it might be beneficial to draw all of the metal objects first, then the rubber ones, and then the glass. We can have 3 state changes, or we can have 300. And at least some state sorting is already required if we’re depth sorting the windows for correct blending results.

However, early state sorting was hampered by the fact that if two objects had very different transformations (for example two windshields on two cars in different locations), it was costly to sort these objects by state alone because changing the viewing matrices was also a fairly expensive operation. Today, however, it is usually much cheaper to sort by state first, though exactly which state is the most expensive (and therefore the most important sort key) varies from platform to platform. We might even want our engine to be able to vary how it state sorts depending on the hardware. As we’ll see later in the article, this is where scenegraphs can excel.

State Encapsulation

Early scenegraphs employed the concept of state encapsulation to facilitate state sorting. This meant each object in the scenegraph would point to a separate state structure–a set of material colors, texture, lighting, transparency, and so on. The scenegraph could then compare these state objects for similarities or just sort by the pointers. Even still, when switching from one state set to another, the system tried to only change the relevant differences and not blindly apply all state parameters, some of which, like texture loads and binds, could be very expensive time-wise.

In these systems, state sharing was achieved by having two graphical objects point to the same state set. This had other advantages, such as being able to quickly switch from “visible light” states to “infrared” states using simple pointer swaps.

In this example, many of the nodes (rectangles) in the Tank hierarchy are assigned states (ovals). When the tank is drawn, we can sort the objects by state and try to minimize the number of state changes. For example, we can draw the left and right tread at the same time and only set the “rubber” state once. Since depth-first traversal would visit these in that order anyway, we haven’t gained much. But we’d want to draw the base and turret at the same time too; so state encapsulation sorting can provide the needed information to make this possible.

Transform Graphs

Early scenegraphs were primarily transform graphs, representing object hierarchies in terms of inherited parent/child transformation relationships. For example, a car node might have four wheel-nodes that would be specified relative to axle and steering nodes (their center of rotation), which would in turn be specified relative to the car. Or, perhaps, a building might contain walls, floors, windows, and interior rooms, which might contain desks and chairs and so on.

Dynamic Coordinate Systems

Dynamic Coordinate Systems (DCS) were added for things like our tank, where we wanted the tank to be able to move around from frame to frame and the turret to rotate independently. DCS nodes were originally more expensive, mainly because there was extra bookkeeping information that could not be pre-computed, but instead needed to be re-computed when the object moved, or at worst each frame.

What bookkeeping? Take culling, for example. It often uses bounding boxes or spheres to contain all of a node’s children and their bounding boxes, recursively. If the node’s bounding volume is invisible, all of the children are therefore invisible. When a child moves, the bounding box needs to be re-computed. So we might write the logic as: re-compute the bounding box only when a child moves. But what happens when all of the children move? Do we re-compute the bounding box each time or wait till they’re all done moving? In that case, it might be better to re-compute the bounding box once per frame, or better yet, store a flag that says if any of the children changed that frame and then re-compute the box at most once per frame. This sort of tradeoff is the kind of thing scenegraphs excel at, where immediate mode rendering does little to help.

Static Coordinate Systems

In the case of buildings, since they don’t move, we could use static coordinate systems (called SCS in Performer). These were simple matrix transformations without a lot of overhead. The main difference being that SCS nodes could pre-compute important information, like bounding boxes and collision information. More importantly, in a MP (multi-process) system, SCS nodes are guaranteed to remain the same from process to process, whereas DCS nodes need to be buffered so that changes in one process don’t have immediate effects in another.

Aside: for a quick example of the sort of MP problems that arise, consider two cubes that are being manipulated in one process and drawn in another. If the first process modifies both cubes before either is drawn, things are happy. If the first process moves the cubes after they’re drawn, things are okay, but you won’t see the change until the next rendered frame, by which time something else might have happened. But if the first process modifies one cube and then both are drawn before it can modify the other, you can see strange artifacts that make the cubes appear to oscillate with respect to each other. Worse still, in a true MP system, the first process can be in the middle of updating one cube while the other is drawn, causing unpredictable results.

We may not be used to using multi-threading or multi-processing on wintel boxes, but it’s becoming more and more important, even on single CPU machines. With hyper-threading, AGP bottlenecks, and consoles that contain many independent processors, synchronizing a dedicated “draw” process with a main application, possibly running at a different frame-rate is going to be a challenge more and more people will be familiar with.

Adding Groups, LOD, and other useful nodes

In addition to coordinate system nodes and basic graphical objects, scenegraphs added other types of nodes to take advantage of the “retained mode” and frame-to-frame coherence optimizations. Most of these node types derive from the basic group node, which acts as a simple container for any number of children, spatially proximate or not but does not impose any restriction on its children.

Level of Detail nodes use computations about how far an object is from the observer to “dial in” the amount of detail shown or switch between two or more child nodes which represent an object at various fidelities. The basic idea is that a far-away object can be rendered at lower fidelity (fewer polygons, smaller textures, etc.). Many schemes have been invented to deal with object switching or fading between LOD states, and the state of the art lies in various so-called continuous level of detail schemes.

Switch nodes are a form of group node that sets the active child node (zero or one out of N children) based on some key value (e.g., 0 to n-1). Sequence nodes are a form of switch where the key value cycles based on time. Animations can be made with sequence nodes – each frame of animation is stored as a unique child object and the parent sequence node controls the active frame. A DCS-Sequence is useful for motion-captured joint animation, for example, where an array of transformations is applied in the same way a sequence node iterates through the list of children (it used to require having N SCS nodes under a Sequence, which was wasteful). DCSSequences can, for example, be efficiently compressed and stored and take very little CPU time to play back (though their interactivity leaves something to be desired).

Performer

SGI’s Performer was an early example of a scenegraph that was primarily a multi-process transformation graph. Performer had state objects which did not exist in the hierarchy per se, but were referenced by graphical objects. Performer made many advances in the use of MP programming techniques to optimize performance on SGI’s multi processor systems. Performer did a great job of state sorting, though an early design decision limited state sorting to only under individual DCS nodes – in other words, objects could not be grouped for similar-state rendering if they had different DCS nodes above them. Performer also made extensive use of traversal masks and per-node callbacks for special effects.

Adding State Nodes to the Tree

Later scenegraphs added the notion of state as an actual node type. This had some advantages, especially in terms of being able to aggregate common state. For example, if there were 100 brick objects, we could insert a “brick” material node as parent to those 100 objects and the scenegraph render process would implicitly render these together. In fact, one of the principal benefits of state nodes are that explicit state sorting is given to the scenegraph modeler. For skilled modelers, this provides more control and more potential for optimization than automatic state sorting. But in the general case, it probably is not a win.

Why? An illustrative example takes 100 tank objects, each with three states (say tread, metal, and camo). But since we want the tanks to each be independently movable, they would be grouped with each tank having its own parent DCS node, plus some more DCS nodes for the turret and tread wheels if desired. Below that top DCS, we’d see the three state nodes and below those, the individual geometry (shared or instanced). This means, in practical terms, that we’d have 100 tread, metal, and camo nodes and that we’d change state at least 300 times during the rendering of the scene. A better scheme might group the graphical objects by the three common states, but that would require each geometry object to have its own DCS and we’d run the risk of a turret forgetting to drive on when the base of the tank does.

VisKit

Paradigm’s VisKit is a good example of this approach. It also added other useful node types like “cameras” (representing the observer in the scenegraph, rather than as implicitly at 0,0,0 in modelview space. But in other ways, VisKit was very similar to early versions of Performer (not surprisingly, since its designer was the person who had managed the early Performer team at SGI).

Adding Action or Event Nodes

Many scenegraphs had the notion of per-node callbacks that the programmer could specify. In Performer, each node could have multiple callbacks, depending on the context. In Cull processing, any cull callbacks (if present) would be invoked to affect the culling result. In Draw processing, any draw callbacks would similarly be invoked for drawing special effects. Since these processes worked in a hierarchical depth-first traversal fashion, pre- and post- traversal callbacks were often provided to let things be done before and/or after traversal of child nodes. Application-side callbacks were also provided to do computation or automation on a node once each frame (e.g., for conditional logic, for animation, to move a DCS, collect statistics, and so on).

However, the main drawbacks of such automatic actions per node are twofold. First, they are very difficult to schedule efficiently, since the application does not know in advance which nodes will be visible or how much time any given callback might consume. They can take an arbitrary amount of time to execute, and generally block further processing of culling or drawing (blocking on draw can cause “bubbles” or stalls in hardware queues). They are also somewhat scattered in terms of cache coherence and branch prediction—similar operations are almost never performed in repeated series. In Performer apps, for example, callbacks were sometimes found to cause CPU bottlenecks and non-deterministic behaviors.

The second drawback of callbacks is more complicated. Since app-side callbacks need to be invoked before the culling or drawing traversals begin (since the app can change the positions of objects, moving them in and out of view, for example), the app traversal generally visits every object in the scenegraph, even those that are way off screen. This can be very costly and ultimately defeats the advantages that culling gives over a brute-force immediate mode implementation.

A better system might do some culling first and then do per-node processing based on how close an object was to being in view. Far away objects usually need limited processing, usually just to determine when they will enter the view. And the app process may move an object. So there’s still a cyclic dependency between this optimization and culling which needs to be addressed.

Inventor

Inventor existed at SGI at the roughly same time as Performer with a very different approach. The goal there was usability over performance. The result was a very elaborate and highly re-usable set of scenegraph nodes, but at the cost of performance. So much so that Inventor was relegated to academic projects and rapid prototyping but to my knowledge, no serious (i.e., high performance) real-time efforts. Many people tried to mix Performer and Inventor to get the best of both worlds, but this was almost always a dead end.

Adding Event Nodes

Event nodes were a later addition to systems like Inventor and its descendent, VRML. The idea behind a scenegraph event system is fairly clever in theory. If the camera or observer is an object in the scenegraph, we can test to see when this object collides with one or more invisible “trigger” volumes also in the scenegraph. A trigger or sensor object could be linked to an effector or action object that would animate a node, for example. Events could be mouse or keyboard based too, so if you click on a 3D button, something else happens in the virtual world.

In this way, one could write an entire user-interactive program in a scenegraph. Doors could be opened, lights turned on by flicking virtual switches, and so on. All data driven.

VRML

Virtual Reality Modeling/Markup Language was the extension of Inventor, drafted after many competing forces finally came together (lead by SGI at the time). It was very similar to Inventor in form and function and suffered from many of the same performance disabilities. But the main benefit was that it was highly self-contained and simple to transport across network connections. It also added concepts for extensibility and portability that Inventor largely lacked (being SGI-specific) and is now being further revised in something called Web3D or X3D or VRML200x.

Body and Facial Animation

X3D and MPEG-4 add special node types for Body and Facial Animations, since for humans, there are some clever ways to extract differences from a standard (implicit) model for better compression. We can encode phonetic visual expressions (visemes) as well as joint animations for elbows and wrists using many fewer bits than if we were coding these things generically.

GeoSpatial

GeoSpatial problems (like drawing the entire earth) require some special nodes to deal with the inherent hardware precision limitations of graphics hardware, namely single precision floating point. True geospatial information requires more than 23 bits of mantissa to properly represent and scenegraphs are generally done using 32-bit floats, so we add some new GeoNode types to various scenegraph schemes. GeoVrml is one such approach, driven largely by the folks at SRI. Keyhole used its own approach for EarthViewer.

PVS

“Potential Visual Set” is a broad term for a sort of generalized culling technique. In basic culling, we take the entire scene and recursively find which objects fall on or within some bounding volume, usually a frustum (a truncated pyramid, approximating the viewing volume). In generalized culling, we might have pre-computed lists of objects that are spatially grouped (like “group” nodes, only they need not be hierarchically associated) and probably visible at the same time. Other techniques might make use of shadow or blocker objects that rule out certain regions of space.

The “Cell and Portal” approach, for example, usually groups the world into rooms or cells, with each cell having a list of objects in it and a list of portals, doors, or other connections to the adjacent (or even distant but connected) cells. When a portal is deemed visible, the culling routine looks at the portal’s connected cell and checks all of its portals, and so on and so on recursively, each time adding (ORing) the overall set of visible objects and each time, reducing (ANDing) the frustum to the portal (door) we can see through. In simpler implementations, objects within a single cell are considered visible whenever their cell is culled in. Often traditional spatial culling is used to further narrow the visible set.

What’s most interesting about Cells and Portals is that it can also generalize the notion of rendering to framebuffers and destinations and make use of standins or impostors. A doorway can be a portal to another room, or it can be implemented as a textured polygon, pre-rendered from an image of that room from the correct perspective. If it’s done right, there’s no way to tell the difference. Mirrors are implemented in much the same way. A mirror can be rendered by inverting the view matrix and projecting the camera through the mirror, then drawing normally into the framebuffer. Or it can be rendered by projecting the camera through the mirror, rendering the scene to a texture, and applying the texture to the mirror as a painting.

The downside of PVS techniques is that they’re usually added to scenegraphs as an afterthought and not built in from the ground up. NetImmerse is/was a game engine that made extensive use of Cells and Portals.

Inventor Revisited

Inventor is easy to use. It provides a rich set of node types which make it easy to get something up and running quickly. And it adds some nice 3D GUI types too, which make producing a finished application that much quicker.

However, Inventor is a poor performer. It suffers from some critical design flaws, such as virtualizing all interfaces, even to atomic data members, which doesn’t help performance any (even COM objects try not to virtualize member getters and setters). But the biggest flaw is in the execution model, the active nodes in the scenegraph consume CPU time while the scene is being rendered. And since all nodes must be visited, view frustum culling is not common, even at the rendering stage. So richly immersive scenes will be slow unless the programmer makes the effort to optimize it by him or herself.

[note: some people who use Inventor have written in to say many of my criticisms have been addressed more recently.]

VRML Revisited

VRML suffers from many of the same performance limitations as Inventor. It’s nice to be able to specify what are essentially dataflow programs right in the scenegraph by hooking sensors to effectors using routes or linkages and place active clickable objects in the world with a few lines of text. But VRML suffers from a severe namespace problem, where declared objects can be ambiguously or incompletely defined (via dangling external references) and so on.

Just looking at the dataflow problem gives some sense of how buggy a VRML system can be. If a scenegraph finds an effector node first and then finds a sensor node that drives the effector, what is the proper way to process this? Do we normally process the effector node first, then the sensor, thereby potentially computing the effector again this frame (risking an infinite loop or at least a performance hit to fix the problem)? Or do we wait till next frame where it may be too late? Or, perhaps, do we sort the entire scenegraph to make sure all sensors come before their down-wind effectors (if that is even possible given the cyclic possibilities)? This could bring up problems with state and transform dependencies and make objects go haywire.

Given global DEF/USE semantics, can we have two objects using the same global name or is this an error? It could be accidental. If so, did we mean to use the first one or the last one? If we try to use the hierarchy to segment the namespace (as is done in Java, for example), what happens when we subtly reorganize the scenegraph because two objects that had been attached now can move independently (for example, a car riding on a moving flatbed train now drives off at the station). What if we want to reorganize the scenegraph for better state-based performance on different target hardware configurations? We could easily break our nice scenegraph-based program in the process.

Scenegraphs Today

Scenegraphs today are quite sophisticated and quite readily available, even free and open sourced. They’re generally well suited for cross-platform game development. But current scenegraphs do have some important weaknesses. One is an overloading of the tree concept with all sorts of bells and whistles that slow things down. Another is that without structural changes, coordinating changes in distributed systems is difficult. Very few of the current crop of scenegraphs were designed with MMOGs in mind.The heart of the problem is an overloading of what was once a nice, straightforward performance improvement over immediate mode OpenGL. We moved to hierarchies so we could cull and draw more efficiently. Then we added in all this extra stuff, like hanging ornaments on a Christmas tree, except that some of the ornaments are nice juicy steaks and some are whole live cows. They simply don’t belong.Put another way, the original transform-graph concept sought to organize the visual database spatially to take advantage of grouping proximate or linked objects. We propagated shared spatial information up the tree, where we could make earlier traversal decisions and save time in true log-n tree fashion.But we have more than one way of organizing our visual database. Culling and PVS techniques want to have spatially organized databases for optimum performance. If the scenegraph is instead organized largely by state, then we might need to cull each 3-state tank (in the tank example) three times, once for each articulated part, instead of being able to cull out each tank once and only once. But if we want to get the best hardware performance, we really do want to sort the visible set by the most expensive state changes first. Moreover, since states don’t change that often, we don’t want to re-sort the scene every frame. But if we start with a spatial view each time and sort only the visible objects, that seems that’s we’re stuck with (as was the case with Performer, believe it or not). If we re-sort the whole scenegraph for state optimization (only once, hopefully), we lose the nice spatial coherence we count on for fast culling.

Given that we want to hook some nodes up to other nodes to enable event processing, we’d also like a guaranteed consistent way of naming objects that doesn’t change after spatial or state sorting or doesn’t even change if parts of the scene are currently loaded or not (early scenegraphs were entirely memory resident). We want a logical or semantic naming scheme, like in namespaces. We want handles that persist and reflect structures that may not even be local.

By executing actions at each node during a depth-first traversal, we are most likely invoking bits of code in an arbitrary (almost random) order. This runs counter to the advanced scheduling many compilers try to do to take advantage of CPU branch prediction and pipelining, instruction pre-fetch and high-speed local caching, to name a few. Instruction and Data Cache misses can affect performance by up to 10x on many systems. So doesn’t it make sense, that if we have 100 physics nodes and 100 inverse kinematics animation nodes, we try to process those nodes together, just like we tried to do for state (especially for systems with special vectorizing or SIMD capabilities). So this gives yet another competing organizational approach to how to optimize the scenegraph.

Put all of these together and it’s easy to see that the current evolution of scenegraphs has taken a wrong turn somewhere. And it will require a change in approach to move past the roadblock.

Scenegraph’s Tomorrow

Granted, it is probably impossible to find a single perfect organization for a scenegraph that simultaneously optimizes for spatial, state, semantic, and CPU considerations. Some people try to hand-design theirs to straddle the fence and make the best of what they have. But a better idea is to remove one of the fundamental constraints: that there need be a single scenegraph organization for a given visual database.

It is entirely possible that we can have a single set of objects, call it an object soup, but have two, three, four, or more hierarchies linking these objects into independent and complimentary organizations. It’s been on the wish of a number of scenegraph designers for years, though it’s never been a requirement before distributed databases came along.

But how to implement this is another matter. The solution, it seems, lies in the separation of concepts of scenegraph “nodes” from the “objects” they represent. By making shared objects live in a soup, we minimize the amount of waste and miscoordination we might see with four or more simultaneous object hierarchies. This way the “node” part of an object is just a few bytes – just enough to point to the object in the soup and to the parent/child/sibling relationships in this particular view. All of the real “meat” is kept once in the object, which ideally contains back pointers to each node in each graph, limited to a small number like four.

Is this rocket science? Not really. Relational databases have separated indices from data since the dawn of time. And scenegraphs are just one way of indexing into big visual databases. Once scenegraph designers come to grips with that, the rest is downhill.

The second problem is how to correlate among multiple database views (i.e., sets of indices). Since lightweight nodes in two views point back to the same object, it’s easy to see how given a node in one database view, we could find the corresponding node another view — just follow the back pointers. This lets us cull using the optimized spatial view and render using the hardware-optimized state view.

The heart of an efficient distributed database implementation, then, is using the spatial view to limit what happens in the other views (rendering, culling, physics, animation, and so on) and distributing changes in the spatial view among disparate systems. The state, semantic, and application views do not generally change, except for visibility and priority per time interval, so the real meat of the task is in synchronizing the spatial views.

Semantic View

The semantic or logical view of a visual database is just a convenient way of accessing objects in the object soup. Think of it as the google (albeit local, not web-wide) of visual databases. The organization is arbitrary and entirely up to the developer. A developer might use the semantic view as a large dictionary of objects, organized by object type, subtype and so on. Or a game may divide objects up by their role in game play. But the main idea is that the leaves of this tree are the actual objects in the world.

What’s important is that the logical/semantic structure is well known (published) for all concurrent developers to use. It is a rendezvous point, as well a convenience.

But it can be used for more elaborate schemes as well. For example, if the semantic view is organized into “vehicles” and then “cars” under that, we could perform some operation on all of the game universe’s cars at once (perhaps, proximity tracking).

And there is no reason why objects could not be located under more than one branch of the semantic tree. There could be a branch called “physical objects” as well as the “vehicle/car” branch. One could set the physics computation process everything under the “physics objects” branch automatically.

State View

As discussed earlier, the State View is intended to be a platform-specific state sort and state aggregation view. For a platform on which texture fetching is very expensive, we might see textureIDs as the most significant branches in the tree, thereby minimizing the number of textureID changes. On another platform, lighting mode might be more expensive to change. The State View can generally be computed on the client at load-time and does not change much. Which objects are on or off does, but their fundamental draw order does not.

One exception to that rule is depth-sorted objects, like transparent polygons. Here, we might have a branch of the state view that is somewhat dynamic without slowing down the rest of the system.

Shading Languages

One of the latest buzzwords in modern computer graphics is Shading Languages. The main idea is that complex images can be constructed by mathematically combining (adding, subtracting, multiplying, etc..) many simpler images, often through a small assembly language program instead of using actual framebuffer operations. For example, a nice 3D bump mapped brick texture (where the bump mapping provides nice light and shadow cues to make the brick seem more 3D) might be described as a combination of a flat red texture, two or three rendering stages of bump mapping (rendering light and shadows), a light map for global shadows, and perhaps a specular highlight map if the object has little glass or metal bits.

Shaders can be expressed as programs, algorithms, or as a “shading tree,” where the constituent sub-shaders are broken down in hierarchical fashion, like we see for spatial transformations. This shading tree might be explicit, if the underlying scenegraph supports such advanced concepts, or it might be implicit, as an abstract representation of (for purposes of understanding) some pre-compiled code.

It’s important to realize that the shading tree we see could easily vary from hardware platform to hardware platform, depending on the graphics capabilities and other factors. For example, some hardware supports advanced bump mapping in a single operation – so the state tree node in that case would be one node. Other hardware might not support bump mapping at all, but we can still achieve bump mapping effects by making multiple simpler rendering passes (one for the texture, one for light areas and one for dark areas). So the shading tree in that case might have a parent node with three children, representing the three passes.

~vs~ (note: here, boxes are states and circles are rendering objects)

Shading trees will also vary from software API to software API. But since we’ve separated out the notion of our Spatial View (see below) from the State View, this affords us a good place to handle the interface between with underlying graphics APIs we might want to use (such as OpenGL or DirectX).

Application View

The presence of an Application View is not a strict requirement. In fact, it is the least useful view out of the bunch, mainly because compilers are so much better at scheduling code on CPUs. And a big problem with data-driven programs is that they can be very hard to debug and stamp out pseudo race conditions. But, on the other hand, they’re very nice for rapid prototyping and platform-neutral abstraction. They’re also quite useful for giving game players the ability to dynamically change game behaviors (e.g., mod programming or simple tunability).

Spatial View

The Spatial View, on the other hand, is the most important view from a distributed database point of view (and with all of the MMOG pushes out there, who isn’t building a distributed database these days?). By organizing the world into spaces and sub-spaces, we can efficiently decide how to route messages, prioritize computations, and cull the database to minimize rendering time and network traffic.

The subdivision of the world into hierarchical spaces is not arbitrary, but there are a number of valid schemes for doing so. What is important is that the subdivision scheme be fairly well tuned to the culling procedure, that no node has too many or too few children (i.e., too tall or wide a tree). In other words, the same rules that apply to well balanced trees in general.

The choice of whether spaces are static or dynamic is also open. For quad tree schemes, the subdivision is relatively static. If an object moves, it might cause new quad cells to be created or destroyed, but no quad-cells ever move. For spheres or bounding box trees, the bounding volumes will likely move as the objects they contain move. Rules for stretching volumes and forcing children in or out of them are also flexible and fairly easy to impement as iterative solutions (with local, not global optimization). In this scheme, bounding volumes can overlap, but they need not do so.

It’s not even a problem for an object to be contained in multiple bounding volumes as long as it’s not culled in or out more than once per frame. I’ve played with systems with “floating” spaces that group objects for lighting purposes (e.g., all objects that are affected by a light are in one space). Grouping objects in formation is another useful extension. A group of tanks or fighters can be dynamically gathered by their proximity and culled as a group, even if there isn’t a single “parent” node in the traditional scenegraph sense.

Summary

I’ve covered the basics of scenegraphs, where they came from, where they are, and where I think they’re going, at least from one point of view. Much of this work is related to on-going development of a so-called “multi-view” scenegraph. The ultimate goal of this work is to come up with a simple, light-weight system for optimizing rendering across many platforms. Look for future articles on my progress with this work.

This document intentionally doesn’t directly address whether you should or shouldn’t use a scenegraph in your 3D app. I trust that given the full facts you’ll know best what you need. But for those people who dismiss scenegraphs out of hand, I hope this article does at least shed some light on the likelihood that you are using a scenegraph in one way or another, whether you call it “portals,” bones,” “linked matrices,” or anything else. Because when it comes down to it, this is all just common sense and experience put to work.

First, a step back. Regular visitors know that I know a bit about the internal workings of Second Life and Google Earth, though, as I always repeat, I don’t know any of their current plans. What you don’t know is that I’ve also consulted for and even considered roles in some of the newer crop of social MMOs. I think I have a pretty good sense of the field, both good and bad. And I’ve also heard the rumor that Google wants to get into this space.

I can’t speak for anyone at Google, but I know this: they’re certainly capable of it if they invest the bucks. John Hanke was the business/marketing guy behind Meridian59, one of the first 3D online multiplayer games. He’s now in charge of Google Earth. So I’m sure he has a passion for this space and could find great designers and technologists to help him pull it off. But the big question I have is one of fit with Google’s overall mission to organize the world’s information, especially after their "better products, not more" mandate came down.

The thing about GE is that it’s a so-called "mirror world." The whole point was always for GE to accurately and compellingly reflect information about the real world. Opening up 3D content development via SketchUp and COLLADA import allows one to put virtually anything on the planet. That’s extremely useful, even if the information is speculative (like a new home plan or a proposed stadium). But the point is always to relate even the most speculative information back to the overall context: the real world.

So what happens if/when a purely fictional data layer is intentionally introduced? Does GE become a big open sandbox with a nice, but vestigial picture of the Earth on the floor? Is it SecondLife on a sphere? [edit: in case it's not clear, I think mixing fictional and real content is a mistake, unless it gets its own distinct context, like a game. Right now this separation is regulated by what gets promoted to an official layer. In a free-form SL-like world, perhaps not so much.]

So people talk about the technical challenges a lot. But that’s the easy part. Adding avatars certainly wouldn’t be hard. It would require a new server intrastructure. It would require the client to be improved somewhat, mostly to hide communication latency and handle thousands or even millions of active objects (esp. those pesky moving avatars).

Some have said that "resolution" is the limiting factor — this is true for real-world imagery, though this is more a data-availability problem than a technical limitation (1mm pixels are not out of the question). The system could probably support very detailed 3D models for buildings as well. But relying on users to create these may not produce good results in the near term. It takes a lot of work and it’s not yet important to do so. Procedural tools, unlike today’s SketchUp, would be essential. Without paid artists making these things, competition and collaborative rating/filtering of content is also essential.

And so let’s say Google does add whatever is needed and suddenly you can see and even chat with all of the other users of GE in your virtual proximity. That’s cool. But what then? I mean, the key thing for any experience is that it must either be fun or useful (or, ideally, both). So what would make it fun or useful? That’s the hard part. And as history shows, simply having the implicit marketing muscle of Google is only enough to get people in the door.

Here’s a short list of some good (and bad) applications of this strictly potential technology:

Collaborative editing – work together on models in-world. Minimally requires SketchUp functionality to be merged into GE (which is possible, but not at all easy). Initially, it could just be used for guided tours, like for selling real estate. That’s something, but it’s still fairly niche.

Socializing – To talk to people, you first need to find them. Say you fly to NY and see a hot-looking avatar nearby. What do you say? "Hey, I see you’re also searching for French restaurants near 42nd st. Ooh, la la." Second Life (and others) offer the concept of personal spaces, or what I’d call HomeSpaces (like home pages on the web). Where is yours in a social GE? Is it tied to your real home? Do we invite people to come visit our HomeSpace, full of virtual furniture from Ikea and appliances from Sears? Yawn. Beyond the basic 3D MySpace everyone wants to do, the key to socializing is sharing context and doing (hopefully fun) things together.

Networking. It might be cool to discover likable real-world neighbors (assuming knocking on doors is too intimidating). But apart from the obvious privacy issues, I’m not sure you need access to a whole virtual planet to meet the kid next door. The "dating" angle could certainly be made to work, after the privacy issues are solved. Adding a social network like SL has might ultimately allow you to find new friends easily, if Google works on the profiling and discovery tools. But what then? Social Networking by itself is fickle. Again, people need something to do together, or at least a purpose for spending time and forking over their personal information to a big corporation, even Google.

Creative Exploration – Ah, here’s something. Say a group of people get together to turn a bit of empty virtual real estate into a hub of creativity, like Burning Man, where people of similar (or similarly altered) minds know to come. GE then becomes more of a showcase tool. Here, adding scripting to the client would be essential. Just looking at 3D models gets boring. They need to come alive, perhaps even with physical simulation. Go a little further and you have games (some of which are already done as mashups with Google Maps — but these could live inside the system, not outside). This is what SL seeks to do. So could Google do it better?

Alternate Reality – AR is usually about overlaying fictional worlds onto the real one. But why not add fictional places to GE’s map too? So a group of people take the NY skyline and turn it into a fantasy land (middle ages, futuristic, etc..). That might be fun to build and explore. And there could even be a few games built there. But apart from the interesting juxtaposition of an Elven Forest across the Hudson from Jersey City, why does this need to live on a map of the real world? Certainly, GE wouldn’t want people searching for French restaurants to wind up with unreal results. (I can just see the mapping directions now: turn left at the big oak tree, down the rabbit hole, and 1.2mi across the swamp of eternal tears…). They may need some better separation between these two products, without sacrificing fun accidental discoveries. This has always been an issue for the "layers" approach.

Frankly, the most profound thing Google could do with Google Earth right now is like what they did for maps: enable 3D mashups. Any and all of the ideas above would get developed, tried and tested by others. But for 3D applications like GE, this is probably the most difficult technical hurdle of any I mentioned. Had Intrinsic Graphics (the makers of the 3D rendering layer inside GE) survived, I imagine it would be easier to have a nice, free Google Earth Toolkit for building new 3D apps, using GE and its powerful servers under the hood. But that didn’t happen, at least not yet.

On the other hand, two of the founders of Intrinsic Graphics are now at Google. So rather than have Google try to solve all these virtual worldly problems (as they tried with Orkut for social networks), I’d much rather see them open up the system in the way Microsoft has for its VE offering, as a component that others can build on or integrate, for free, but with ad revenue flowing back to Google, of course.

The risk to Google is much lower. They can still make gobs of money. And the potential wins are much greater than going it alone. At least, that’s what I would do.