Is WebGL actually harming your computer in any way? I doubt that’s a serious or credible claim. And, frankly, if Microsoft has taken a formal position against WebGL, no one I know got the memo.

It would be an unfortunate position for Microsoft to take, IMO, because it gives the impression that Microsoft runs away from security issues that require some modest technical mitigation.

After all, what is an operating system but a series of security apparatuses coupled with Hardware Access Layers and useful software development APIs on top?

WebGL has the latter two in spades and most of the former, but clearly needs a bit more assistance on the security angle before everyone is “all warm and fuzzy.”

Operating systems and security mitigation are what Microsoft is known for. It’s our bread and butter. Why would we run away from that challenge with such an alarmist attitude of “shut it off, shut it off, it might hurt me!”

I think we would face these potential threats head on, as we’ve always done.

I mean, the exact same graphics hardware that WebGL uses is available to native applications running DirectX or OpenGL on your PC and/or phone. Are we going to ban downloaded games because they might, in some universe of possibilities, harm our computer or cause us to, God forbid, reboot? No. At most we give a security warning that this .exe might be harmful and let you make the choice.

Are we going to ban Chrome and Firefox as “unsafe” if they continue to unabashedly support WebGL? Are we claiming those guys don’t care about security and aren’t working hard to mitigate any remaining issues as soon as possible?

No. That’s not a tenable position, IMO. WebGL will be running on my PC and yours, one way or another. Microsoft will need to deal with it. And more to the point, we can actually help make it much more robust if we engage instead of apparently running away.

Remember Plugins?

I was mainly disappointed in those posts because I recall vividly that it was Internet Explorer’s pioneering work with plugins (specifically ActiveX controls) that help build the rich interactive web as it exists today. Plugins created capabilities not found in browsers, even to this day. Flash is a native plugin. Silverlight is a native plugin. Google Earth, running in your browser, is a native code plugin. RealVideo, YouTube, and FarmVille would arguably not even exist without plugins (okay, that last one might have been a blessing).

However, ActiveX controls were, at one point, the primary vulnerability for browser-borne attacks on your PC. They are, after all, native code with hardware access that could run malicious operations, perform disk writes, read your personal data and plant viruses. Indeed the MSDN site on ActiveX controls begins with “An ActiveX control can be an extremely insecure way to provide a feature.”

Yes, indeed.

Somehow we survived the existential threat of native code plugins taking over our PCs, or at least we made it through alive. The web prospered in rich user experiences primarily on IE, while the main residual downside of plugins, even today, is that they require user confirmation, code signing, and in some cases circulation of known or suspected threat information among browsers to help block attacks. That’s not ideal, but yet we survived and received the benefits of plugins on the whole.

Well, that’s not to say plugins are all safe. How often does your Flash plugin need to be updated (weekly?) to address vulnerabilities to keep it safe from attacks? If WebGL can help obsolete those security holes, it could actually be in some ways safer than what exists today.

WebGL is not a plugin but rather a “built-in” and it doesn’t ever allow the extreme native access of ActiveX — no disk writes, no main memory access, no CPU code apart from officially signed graphics drivers. A shader can really only affect your graphics hardware and screen output. The most severe vulnerability we know of today is that it might hang your machine. Worst-case solution: reboot.

We can do better. We can require WebGL shaders to be proactively trusted in the same way plugins are trusted and largely avoid the worst threats. We can do even better with code analysis, collaborative filtering, and hardware or OS watchdog  timers (e.g., any shader taking more than a fraction of a second can be reset without anyone complaining). Yes, we can. But if the choice comes down to running WebGL or not, I’d live with a popup asking permission to access my graphics hardware, as we do for GPS, camera, etc..

What’s going on?

From the one discussion I’ve had with leaders from IE, I can reassure folks outside Microsoft that this issue is actually about security and doing the right thing for users. It’s not about “GL” vs. “DX” in the name, as some suggest. It’s not about wanting to disrupt any other browsers, as Microsoft has often been accused. These leaders are genuinely concerned about the possibility that someone on a malicious website could use WebGL to disrupt your experience in a serious way, and incidentally that it would appear to be Microsoft’s fault…

Users are not very discriminating in their blame, after all.

Those leaders may not be fully aware of how big a movement WebGL really is and how it is going to transform the web yet again. But the reality is, if Internet Explorer does not support WebGL and WebGL nevertheless becomes the de facto standard for 3D on the web (which it will, IMO), then IE will be in an uncompetitive position to either help fix any problems and moreover retain or grow market share relative to other browsers. That would be sad, esp. given how long the product cycles are and how long it would take to course-correct. We could miss the boat entirely.

Now, I own Microsoft stock. I want Microsoft to succeed, and that includes IE. If Chrome, Firefox, and Safari support WebGL on Windows and there are new PC-only vulnerabilities found, do you really think people will blame Google, Mozilla, and Apple and praise IE?

Not a chance. They’ll blame Microsoft. They’ll blame the OS. They’ll blame the company. They’ll blame the logo sitting in the corner of the screen that just went blue or blank and say how “this never would happen on Chrome or OS X.” (ignoring market share)

All Microsoft would likely achieve by not supporting and improving WebGL securely is that the people who could really fix the few remaining issues (driver writers, hardware manufacturers, OS makers) will try less hard and take that much longer than they would with IE and DirectX demanding results.

Meanwhile, IE would potentially lose market share due to popular interactive experiences that are not achievable there. And any sort of weaker “safe” shader-less  alternative that IE might conceivably propose in a too-little-too-late DOA standard will make it appear as if IE is trying to disrupt the market, which I don’t believe is their goal. They really want to do the right thing, but it  may not be very clear what that is until the Web clearly and audibly demands it.

There is only one way through this maze. The way forward is to address the security issues head on, get IE the most robust implementation of WebGL on the market, and lead the industry to a new level of user experience, including NUI and rich 3D graphics, hand in hand.

Speaking only for myself, as always, I fully intend to use WebGL as one important tool for applications and platforms I develop for Microsoft. That means “wherever it’s supported.” For other cases, we’ll have to use creative fallbacks, lesser functionality, and/or resort to plugins or augmented browsers for cross-browser capabilities once again. Our charter at Bing requires working cross-platform to reach the greatest number of people possible and I don’t see that changing anytime soon.

The kind of experiences we want to deploy are nothing short of revolutionary – 3D for the masses, tying the real world  to the information space that surrounds us in our everyday lives. This means phones, PCs, and the like will require the kinds of rich, real-time interactive 3D interfaces that right now only WebGL can offer in a cross-platform, stable, browser-based way.

There is clearly only one direction forward for Microsoft and 3D on the web.

WebGL is the way.

With my current project, the situation is much improved blog-wise. We concluded recently that the best way to accomplish our goals was to be open and even solicit cooperation from outside. Quite a concept!

The project’s name is Read/Write World.

If you haven’t heard about it, here’s Blaise’s (my boss’) recent keynote at this year’s Where 2.0 conference on Earth Day:

Below is a link to a live demo we recorded last week of something I’ve spent the prior week or so getting barely polished enough to show.

It’s a real-time editor for the Read/Write World. What you’re seeing is me flying around the map, sifting and editing some data by moving it around in real-time, incorporating (shown with the red lines) real-time match results from a great new service that better connects bits of data together (photos in this case) by visual similarity, aka feature matching.

Behind this visual front-end lies the unseen but much heavier effort of the last six months — real-time read/write Azure services ready to handle ungodly amounts of information. Yes, real-time, as kind of like a MMO, except we’re not tracking players but rather changes in the world itself. Building that, and a new (in some ways old) language called RML, to help describe that information, has taken up most of my time.

Our first goal is really straightforward, but not necessarily easy. Microsoft has lots of data in lots of formats. We also have to cater to every browser, every mobile device and platform known to Man. Multiply those together and you need a thousand talented people to just to keep parity with the rest of the world. What we needed was a way to make one application use one data format do more with less effort. In mathematical terms, 20×10 = 200 but 1×1 = 1.

Good in theory, hard to do in practice. But if we succeeded, the benefits would accrue not just to us, but across the board.

So on the “one application” front, we’ve focused on HTML5+CSS3[D]+WebGL as the most promising way forward (some Silverlight too, just since you wondered). On the data front, we’re building this crazy simple metadata format called RML. Here’s my quick writeup on “What is RML?” if you’re interested.

tl;dr: it’s just basic JSON structures with enough reflection to make it so we can transform to/from other formats/versions very easily, and write butt simple webapps that use it and render it to boot. At the meta level, it’s an Entity-Relationship model woven into a world-scale scenegraph.

Now, one of the debates we had was, does it need to be a standard? Our conclusion, which I’ll elaborate more on in my talk at the ARE2011 conference next month, is “no.” We’re not trying to change anyone else’s standard, nor do we require (or expect) that other people adopt our format as their own. We’re even expecting our own users will grow and change RML organically to suit their needs, without necessarily asking permission. We’re trying to make that easy, supported, and robust. A lot of this is inspired by Open Street Map, btw, which has very compatible goals.

Our intention is to gradually make public some beefy servers that can read and write this RML stuff, mostly visual data for now, such that anyone can easily store and retrieve their geospatial metadata here, for free, and even write apps that do useful things with it, for free.

Caveat: we fully respect copyright and claim no implicit or explicit ownership over other people’s data. So while our service might be free, some of the data we refer to might still reserve rights. For example, the images might be hosted elsewhere and carry a copyright tag. We will respect and pass-on those assertions to any consumer of the data, of course, but still strongly encourage content owners to choose creative commons wherever possible). Beyond that, our big “value-prop” is that we’ll crawl the data we store, trying to find new ways to connect it so it’s more useful than just a list of random cruft sorted by lat/long. That’s one of the hardest and most unique parts of the system.

So, to cut this off before it becomes a novel, between this site and the official ReadWriteWorld.net, I and others on my team will try to keep things pretty current and informative. The main difference is as always that RealityPrime contains no official Microsoft spokesmanship but merely my own personal opinions, which carry no warranty and can change without notice.

Last caveat: I might not be able to answer the most likely questions just yet, especially about availability, betas, and so on. That’s still being worked out. I’ll even be a bit hazy on the specifics of our functional RML format right now, mainly because it’s undergoing rapid evolution as we throw more and different collaborators into the mix. But any questions I can’t answer will go into a queue for the official FAQ.

Change, in this case, is good.

Now, I should caution that Blaise didn’t ultimately want to use this text and Jaron equally had issues with it. The tone is all wrong for an academic journal, plus Jaron disputes some of the dates I recorded from my research (he may well know better). But I felt it might at least be entertaining to RP readers, so I’m posting it for you to enjoy. Still, don’t take any of it as official, just me being a smart-ass.

October 20^th, 2008 marked the 30^th anniversary of the MUD [1]or Multi User Dungeon, widely recognized as the world’s first multi-participant text-based virtual world. Only three years later, a somewhat less interactive work, True Names[2] by Vernor Vinge, imagined full multi-sensory worlds with millions of participants. The film TRON debuted only a year after that, popularizing (if not actually monetizing) computer-mediated virtual worlds as full-on alternate realities — places with lives onto themselves. But before any of these were even conceived, The Veldt[3] by Ray Bradbury, envisioned “The HappyLife Home,” a fully immersive CAVE-like space, consuming parents and kids alike way back in 1950.

The history of virtual worlds is a complex mesh of fact and fiction, weaving pioneers, dreamers, authors and critics in a quest to define a grand vision and to meet an ancient need, dating back to the days of burnt charcoal on cold cave walls. That need is to communicate, to share and persist what is otherwise ephemeral, isolated and ultimately bounded to the lifespan of memory: our thoughts, our ideas and our stories about life, real and imagined. It is perhaps fitting that these visionary fictions are themselves conveyed to audiences new and old as print and film-mediated virtual worlds[4],  just as the CAVE acronym[5] itself is a recursive allusion to “Plato’s Illusion” playing on those same torch-lit walls.

That grand vision goes beyond mere communication. The desire for ubiquitous virtual worlds is an understandable manifestation of our collective (if not universal) longing to overcome the rules that contain us, to propel our minds over the limits of matter, space, and even death of which the “mortality of thought” is just one example. The end goal for many is to construct alternate realities so malleable, so perfectly adapted to our innate desires, that we could fairly call the results magic.[6] In those brave new worlds there need be no scarcity, no ugliness, nor pain. We can be whatever we imagine or wish ourselves to be and suffer none of the limits of ordinary mundane life.

Then again, in reality as in fiction the stories never quite seem to work out that way. Reality has a way of always winning in the end. And as with the previous allegories, this grand quest of ours is so cyclical, so inflated and self-referential as to often resemble the construction of a Klein bottle. The true history of virtual worlds is one of visionary and often impervious genius, promises made (and made again) and the search for the most elusive human-computer interface ever envisioned: the one that disappears.

One of the early visionary geniuses in the field was the legendary ACM Fellow, Ivan Sutherland. His ground-breaking Sketchpad system plotted the lines along which future computer graphics interfaces would be drawn. He invented the head-mounted display, heavy as it was, to better couple the virtual imagery to our actual head motions and in so doing laid the foundation for untold future systems that place one or more virtual interactive cameras in a computer-mediated reality.

But just 10 years earlier and only a few years after Bradbury’s fictional exploration of virtual family values, Walt Disney more literally broke ground on a site in Anaheim that would become, for a significant period of time, the world’s largest physical virtual world.

Until then, the best example of a “fantasy land” was the haunted house — a grotesque and distorted reflection of reality, not entirely unlike some early experiments in VR. Disney’s theme parks provided a comfortable level of immersion for many people — not nearly as inescapably immersive as, say, Korea or Vietnam, and not nearly as interactive as the Renaissance Pleasure Faire and “RenFaires” since, but safe and fun for the whole family.

The 1960s also saw an explosion of experimentation in the media of the time, including Mort Heilig’s Sensorama, which included stereo visuals, smells, sounds and even haptics. Filmic experiments in 3D and surround visuals abounded, with Disney’s Circle*Vision[7] being one of the most widely viewed. The addition of sturdy handrails to keep guests from falling over (in a stationary theater) is a testament to the power of these experiences to move and consume us for better or worse.

While we’re not sure if Sutherland knew Disney or Heilig in his day, we do know that Evans and Sutherland computers were used to help animate Disney’s TRON in 1981. And ten years after Tron failed to make a dent in any critical or commercial sense, Disney Imagineering’s VR Studio would similarly begin their real-time interactive VR experiments on Evans and Sutherland image generators.[8] The obvious choice of movie content gave way to a failed Rocketeer attraction and later Aladdin’s Magic Carpet Ride, sporting heavy HMDs, counterbalanced much like Sutherland’s Ultimate Display.

One of Sutherland’s students, Ed Catmull, also found inspiration in Disney animation. He embarked on 30 year quest to reinvent the art and science of animation, finally coming full circle in the new millennium, as the head of Disney’s animation studios after Disney finally married Pixar.[9]

The hard road towards better computer-mediated storytelling (in HMDs and on the silver screen) merely proved that obstacles are there to surmount. Randy Pausch, the scientist most famous for his profound “last lecture” at CMU, observed that obstacles “give us a chance to show how badly we want something” — or, perhaps, to make us take the time to understand why. Dr. Pausch worked with the same Disney VR Studio, CMU’s Alice software, and other members of the story all of whom badly wanted to solve the problems of more easily building and experiencing computer-mediated virtual worlds.

But reinventing the world takes time — often a generation or two. Ivan Sutherland was reportedly influenced by Vannevar Bush’s 1945 atomic-age essay “As We May Think,”[10] which is also widely credited as inspiration for the World Wide Web. That it took 50+ years to produce a Google and an MSN to make the web more tractable is merely a reflection of the difficulty we have in taking grand ideas and rendering them in a form that works for ordinary people – the proper convergence of money and market, timing and technology; but more importantly a better understanding of just what it is we should be asking for in the first place.

Well before we had many cogent glimpses of that sort of revelation, the 1970s saw the exploration, largely on university mainframes, of true multi-user environments. Maze War in 1974 was the first known multi-user environment and graphical to boot. But it was very limited in its influence. MUD, in 1978, had much more success. The lack of graphics proved to be no deterrent, and in fact arguably improved interactivity and depth, simplifying some very hard problems to a more manageable scope.

The 1980s saw a new push into real-time graphical interfaces, with cheaper and more available commodity hardware. On the desktop, Apple and Microsoft pushed 2D windows, icons and mice to great effect. In movies, CGI went from niche to boutique to a mainstay of visual effects. And VPL Research pushed the envelope in full-sensory virtual reality (and even the term itself), providing visual programming elements and a grab bag of more natural user interfaces. Their Reality-Built-For-Two was the first commercial VR system, shipped in 1989.

The Achilles heel of commercialization is the requirement of making money. VPL ultimately sold itself and its patents to Sun in 1998. Still, in 1989, Mattel released a PowerGlove that simplified the VPL DataGlove concept into something mass-marketable. It unfortunately failed to spawn the kind of kinetically addictive games that the Nintendo Wii (now Kinect) presently enjoys, despite many similar capabilities. And while the Visual Programming Language VPL developed broke new ground, it fell to Lego and Microsoft’s Robotics Studio, years later, to truly push some of these concepts to wider audiences.

The 1980s can be thought of as the early adolescence of virtual worlds in which the core technological concepts came to be expressed, tested and propagated to anyone who would listen. Habitat, for example, emerged in 1988 on the popular Commodore 64 platform, and was arguably the first mass-market virtual world, presaging mega hits like Habbo Hotel, Club Penguin and even Everquest by over a decade, but never quite gaining the market validation they sought.

If the 1980s are the early teens, the 1990s represent the often-raucous teen to adult transition. Unfortunately, as difficult as it is for any child actor to grow up with the hype and spotlight of Hollywood so too were the overwhelming expectations for VR technology a part of its downfall. Movies like The Lawnmower Man promised an effectively supernatural decoupling of VR’s espoused benefits from any actual truth.

Publications liked Wired treated VR visionaries (or indeed, anyone who seemed to have a good futuristic idea) like rock stars. In fact, it’s the expression of difficult computer science concepts in natural human metaphors that is both the greatest strength and the ultimate weakness of VR – anyone describing it to lay audiences can use language that evokes sweeping images and expectations that are currently impossible to meet.

Meanwhile, with all of the attention and potential of VR technologies for computer- mediated virtual worlds, research funding increased dramatically. Among the efforts, the Human Interface Technology Lab at the University of Washington pushed new technologies to solve some of the core interface challenges in VR, leading to devices like the Virtual Retinal Display, a laser-based display that is today most closely related to pocket-sized low-power projectors. UNC pushed the envelope on haptics and core technologies, while Universities like Utah, Ohio State, Brown, and Stanford pushed ahead primarily on 3D rendering. Carolina Cruz-Neira and others pushed the envelope in projected virtual environments, culminating in 1992 with the CAVE at the University of Illinois Electronic Visualization Laboratory. Meanwhile, continued military investment in visual simulation drove investment in graphics supercomputers, ultimately leading to cheap commodity 3D hardware acceleration for PCs.

Virtual Reality went more or less like the space program. Massive investments didn’t result in any influx of civilians going into space, no personal robots nor flying cars. It certainly got a lot of public attention, anxiety and perhaps disappointment after some early climaxes. But the technologists and dreamers marched on. And core technologies produced did in fact wind up in everyday consumer devices, from cell phones to gaming consoles.

While true Virtual Reality devices failed to take over the world in any meaningful way, the offshoots of the very same work wedged themselves into our daily lives. While Disney Imagineers worked on million dollar SGI hardware to build an immersive Aladdin VR attraction in 1994, they played Doom and Quake in the office for fun. Giant dinosaurs gave way to nimble mammals, just waiting for their chance. John Carmack’s games simultaneously spawned an entire genre of blood-splattering Demon/Nazi/Zombie shattering realism and helped blast PC-level 3D graphics into the mainstream.

The gaming community can be credited with seeing some of the most lucrative uses of 3D graphics and interactivity, and of turning it all into a sustainable, even thriving business[11]. Gaming has turned into such a serious business that a whole branch of Serious Games has emerged[12].

But while game companies were having fun and making money the internet boom took full effect with a much stronger emphasis on the latter over anything else. SGI famously launched Cosmo, an offshoot of VRML 2.0, to take the “Metaverse” by storm. The “dot com” bubble dwarfed by orders of magnitude any investment in games or VR pushed home broadband adoption but ultimately left US network infrastructure and networked virtual worlds bobbing in the wake of the April 2000 tsunami.

Phillip Rosedale was one survivor of that bubble. He was one of the inventors of a video codec that made Real Video hum, having sold his company to Progressive Networks in 1996. He founded Linden Labs in 1999 with the proceeds and some help from friends in physics and video games. And while the evolution of massively multiplayer on-line games can be traced from Maze War to XPilots to Meridian 59 and World of Warcraft, Second Life differentiated itself in several important ways: first, there was to be only one shared world, not many shards or instances to split the load; second, the world is malleable and subject to user’s whims; and third, it’s not a game, it’s an alternate existence onto itself, tied into real world currency and the web itself but separate just the same. Second Life also rejected the standard VRML view of the world, of scenegraphs and polygons, in favor of a completely dynamic soup of what are effectively virtual Legos. The end result is something that is both powerful and still, eight years later, limited by its own design to a walled-garden and only linearly scalable topology.

Another survivor of the dot-com implosion was busy building a fundamentally different kind of virtual world, right in the bubble’s wake. Keyhole was a spinoff of the defunct game technology company, Intrinsic Graphics, involving many former engineers from SGI, who had worked with VR early adopters from Disney Imagineering to the NIMA and NGA[13]. The key technology could stream an entire planet worth of visual information over a standard network connection and render the relevant window it in real-time on any PC. But the true advantage over contemporary systems, GIS and otherwise, was in how simple and intuitive the interface was. In 2004 Google acquired the company and renamed it Google Earth. And today, it defines what a mirror world is supposed to be – an accurate reflection of the real world, equally navigable from your desktop or mobile phone.

There was yet another kind of mirror bubbling to the surface around this time, one much more grass-roots. Bloggers, with unfettered access to the Web, began posting about their daily lives, their interests and their friends. Social networks began to take root in 2003, in a second wave of web startups, the so-called Web2.0. They developed “what’s new” feeds meant to keep people in the loop. Sites like Twitter emerged later to simplify the process of posting one’s status to the net. And more recently, efforts like FourSquare, and others seek to leverage real-time location on mobile devices to build up a view of the mirror world that is both personal and reflective of our individual realities.

A distinct kind of virtual world was just coming into focus around the new millennium as well. The earliest examples of Augmented Reality can be traced to systems like Myron Kruger’s VideoPlace in 1975. These consisted of combining video streams of real people and real or artificial places to simulate a sort of detached immersion. Much as with a meteorologist on the local news, participants could see themselves acting in a virtual environment in the third person only, on monitors for example, vs. the first person points of view obtained with HMDs and CAVEs. Eventually, technology was added to accurately track and emulate camera lenses and to seamlessly register computer imagery into the video. Today, augmented reality typically refers to live captured from a person’s point of view, overlaid with relevant CGI and fed back to some display device, perhaps even a head-mounted-display.

The promise of augmented reality is best epitomized in Vernor Vinge’s novel, Rainbows End, where VR contact lenses provide continuous access to the augmented world both inside and out. Haptic interfaces provide the missing sense of touch to manipulate and make these virtual objects real. But the state of the art is nowhere near this threshold. Problems with tracking and registration of objects still remain, but the principle challenge remains the display device, making it small, unobtrusive, and yet high enough fidelity to enable a person to walk around and actually use it without getting hurt by the real world, which hasn’t gone away.

In the development of Virtual Worlds technology, we’ve seen displays go from a few vectors to thousands, millions, and lately billions of pixels, and from a few dozen to a few billion polygons per second as well. We’ve seen haptic interfaces shrink from room-sized devices, reminiscent of the Inquisition, to gloves and even direct neural stimulation with no mass at all.[14] We’ve seen input devices go from mechanical arms to wired magnetic sensors to wireless and even optical motion capture. And we’ve seen networks go from slow drizzles of bits to full-on torrents, with sophisticated methods of prediction to cover latency and errors.

But yet Virtual Worlds, despite such advancements, and despite the adoption of its technology and methodology in many fields, is still widely seen as a future technology, not relevant to our everyday lives, a walk down the street, a trip to the mall, a day in the office. Web 3D is derided as unnecessary and indeed cumbersome, given the success and simplicity of the current 2D Web. Second Life is seen as niche, despite many businesses trying, for a time, to set up shop in one of the best Metaverses around.

Clearly something is missing in the equation.

After forty years of research and development since the ground-breaking days of Ivan Sutherland we’ve finally beginning to come to a realization. People like escapism and fantasy worlds. Kids love to play in the modestly dimensional clubs of penguins and hotels. And companies are rushing in to tame the wild 3D west and pan their virtual gold. But the vast majority of human beings still spend the vast majority of their time immersed in a much more compelling and less inescapable 3D environment, which we tend to call “real life.”

Computer-mediated virtual worlds are just beginning to catch up with what we do out here in the real world. We buy and sell things. We get and share information. We communicate and “get stuff done.” We can increasingly do all of that in a virtual world too. But we can’t do it nearly as well.

Though we bemoan the limitations of 2D displays and table-top mice who have overstayed their welcome by a decade or more, the reason we don’t typically do these things in a virtual world is not about the interface, or the speed of CPUs or GPUs. It’s because we live out here, in the real world. No one likes to commute across an international — or inter-dimensional border — on their way to work.

A significant push in modern immersive environments research, then, focuses not as much on how to bring us to Neverland, but on how to bring Neverland to us – to make The Virtual a part of our lives in a way that benefits us beyond the hype of paperless offices and social (read: peer pressure) networks.

What the new Virtual is all about is reality but augmented, mirrored, and hyper-realistic, giving benefits beyond VR, AR, and even AI (which has similarly vanished into the lattice of modern life).

But what does it take to make this vision real? How do we mirror the real world in a way that we can make it interactive, turn it into a trellis for overlaying whatever whimsical and/or beneficial fictions we can dream up? Once we have that trellis, how do we interact with these hybrid real/virtual objects? How do we trade them, when their substance costs no more than the electricity they consume?

Only one thing is certain: the story of the next 25 years will be written in the full light of day.

I’ll post more when I get a second, but it’ll take some time to digest.

For what it’s worth, my present take on the Singularity is a cross of Vinge’s and something Stross said at a WorldCon party (or elsewhere), and Kurzweil, despite some inherent contradictions:

1. The future beyond a singularity is fundamentally unknowable. That’s the whole point. If we can accurately describe what’s past a so-called singularity, then it’s just your basic run-of-the-mill evolution, revolution or “disruptive” sea-change, which happen all the time.

2. People are good at extrapolating linearly, not exponentially. We can predict a few years out, but after that, reality diverges wildly from our naturally limited mental models.

3. We’ve already gone through multiple “singularities” throughout history, though perhaps increasing in frequency. Singularities are never the end of anything, but a new platform on which to complain about our current ways of life and ponder the color of the pasture on the far side of the next singularity.

Before their introduction, could people have predicted how the world would change with Writing? Or Computers? Or Corporations? Could they have even predicted the invention itself? If not, then these may also be singularities, points in history that we can only understand by looking back, not forward, like the approaching event horizon of a black hole.

That is not to say that some visionaries don’t imagine a world past that event horizon or see the event coming. But it’s all speculation, cautionary or wishful fiction at best.

Even the inventor of the mechanical computer, beyond genius for his day, could not have predicted word processors, virtual reality, AI, or even the CAD software that would have unquestionably helped design his mechanical computer.

One could argue that the One True Singularity will occur only when we (our heirs or errs) become smart enough to see through to the future beyond, i.e., the real Singularity is the last Singularity we will ever know.

| View | Upload your own

What he seems to be describing is apparently not too far from what I’ve been writing about for a while. The part I’m still skeptical about is the life-logging, and probably because of my own preference for privacy. You’ll notice I don’t twitter. I have a hard time believing anyone would even care to follow what I do from moment to moment. And I think careful editing is the secret to any compelling narrative. I just don’t want to put gigabytes of sub-standard, often mundane, prose out there into the digital firmament.

But putting that aside, the germ (and/or gem) of what he’s saying, and the part I totally agree with, is this notion of a pervasive synthesis of augmented, mirror, and alternate realities — no need to distinguish between those arbitrary categories. Turns out, there’s an old word for this which I think we can now safely revive to summarize the intent:

magic

Practical Magic (not the movie that has usurped the term) is ultimately what’s driving the vast majority of virtual worlds interest. When I enter a virtual world, I can suddenly do and be things I wouldn’t even try in real life. If that kind of magic invades the real world (and it already has, as I’ll get to in a moment), I get the best of both worlds. I don’t need to "log on" or even "go anywhere," but I can now do things that would have been inconceivable to people just 100 years ago, at least outside of science fiction or fantasy genres.

What would I do? Well, for one thing, in an augmented/mirror world, I can gain a degree of omniscience, knowing anything that has been modeled with meta-data. I can point to objects and see their history, their meaning, their ‘DNA.’ But why limit ourselves to overlaying information onto boring real world objects. When important objects exist only as bits of data, piped to my brain via (pick: desk/wall-mounted, head-mounted, eye-mounted, or cortical interface), I can now manipulate those in any way I choose, as if by magic. I can morph them according to my needs or whims. I can clone them. I can make them do seemingly magical things — fly, dance, interact with real and virtual objects, do work for me, entertain me, and so on.

Is this sort of magic really so new? No. Remember the telephone? It’s a device for putting people, who may be on the other side of the world, right beside your ear. That’s pretty damn magical, considering the ordinary limitations of time, space, and airport security. Cut the wires and walk around while doing so and it’s even more magical. Television would be even more magical still, if only the content wasn’t designed to turn your brain to consumer mush (quality of TV is indeed in the eye of the beholder, and by that I mean the D&D monster that can paralyze you with a glance). All in all, we have been increasingly bypassing limits of the real world through communication technology.

What this all comes down to is not just the future of virtual worlds, but the future of communication itself, of which pure virtual worlds are only the most natural embodiment. Virtual Worlds are not a place, it turns out. They’re not a novel software application either. They are a key component of human-to-human communication, as old as humanity itself. Virtual worlds are what sit in our brains to reflect (i.e., model) the world around us. They are what sit in the bits in a computer’s memory to do much the same, but without as much intelligence or understanding. And they are what "virtually" sit between us when we try to communicate what’s in our heads — they are concepts in context, encapsulated in content.

The real world — physics, biology, even time — has very little to do with it, except as it serves as the funnel through which we must inevitably pass such information in order to communicate (see "limits" above). Virtual worlds widen that funnel to a full-on wind tunnel. They permit all sorts of sensory, conceptual, and sometimes even factual information to pass between us more easily.And what the future holds for that goes way beyond the "metaversal," almost cartoony manifestations we see today. Virtual worlds have the potential to literally open my mind to yours, as directly as possible, to allow the free flow of information back and forth: ideas, and expressions moving between people in a massive increase of both bandwidth and understanding. That’s what we’re dealing with here, and nothing less.

So lets get back to what it will look like. The easiest thing to imagine is the virtual meeting — that’s being built already. Seven people spread across seven cities can meet around a common table. Now, forget the large projection screens and make it a cafe table in Paris, or an office anywhere. The easiest way to make that work is to give everyone their own virtual view overlaid on the real world. It’s not the only way. But it’s certainly the most compatible with our current business models. [Imagine if that cafe has to shell out $30k (even US dollars) for a VR-enabled table+room and it's much less likely to happen.]

So now we walk around our daily fog with displays that let us see things that aren’t really there. What then? Well, we’ll need to interact with them. 3D Cameras are becoming good enough such that we can skip the data gloves of my youth. It doesn’t give much force feedback, but there are better ways of doing both on the horizon — if we can help paralyzed people walk via electrical stimulation, we will eventually be able to simulate sensation and force feedback as neural impulses with no mechanical linkage required. So-called mind-reading rigs may also do a better job deciphering our motions with zero latency and even some predictive abilities to infer intent.

What does it mean, then, if I can not only twitter my friends constantly, or in turn be-twitted, but I can now see my friends around me all day long, as virtual "ghosts" not quite haunting my active life. What happens when I’m sitting at work and my mother (who is in Florida in RL) strolls into my office to talk to me about a recipe for chicken soup? Or better yet, my son plays with his friends in an entirely virtual game of cops and robbers, running through our real neighborhood, shooting at imaginary (to everyone else) targets?

There’s not much left to hold this back except the display and sensory technology, and that’s almost ready. Laser retinal scanners will be mass-marketed in the next 2-3 years, in the way blue-tooth headsets are becoming ubiquitous today. Give it 3-5 years to solve capturing your facial expressions from such a headset, just to be safe.

Rendering, by itself, will be more than good enough in 3 years. For an ever-diminishing price, we will have completely photo-realistic views of objects in real-time, overlaid and indistinguishable from the material ones. And within 5 years, even virtual humans will cross the uncanny valley and come up the other side — especially at least when those virtual humans are driven by real ones at the other end, meaning that teleconferencing can finally do away with video feeds and go 100% virtual in the next 5 years. That alone frees up a few degrees of freedom in terms of interaction.

So that’s most of what Cory was referring to, I imagine. I don’t even think it’ll take as long as he does, except to find the compelling applications. And the part that obviously interests me the most are the issues of communication — inventing new ways to share ideas that transcend the limits of the real world we’ve come to know and overcome.

Now, back to life-logging. In all this, the one element that should be clear is how much more control I can exert over my environment, real and virtual. It’s all about adding magic and power over your world. So I still am not convinced I will ever choose to give up so much power by giving out the stream of information (most of it useless) generated by my daily activities, where I look, who I see, etc.. Using that in a sort of ‘local augmented memory’ is fine. But sending it up to Google, or Lifepress (a fictional WordPress for lifelogging), etc.. is where I’d draw the line. (note: we already do give a lot of info for free in our daily credit transactions, but slightly less revealing).

In fact, I’d even be concerned about simply outsourcing any visual enhancement of my perceptual space to some seemingly benevolent company. What happens, for example, when that company modifies what I see to subtly persuade me to act a different way? (buy something, vote some way, etc..) 

Well, I can certainly imagine a few useful tools that would at least offer to give me more power (more magic) through recording everything I see or do. But those recordings would inevitably get out or be used by someone to mine me for profit. And so I’m still waiting to see the compelling case for such a potential power loss. It may yet happen, but on this point, I remain to be convinced.

What do you think?

Sections:

1. The Big Picture
2. Towards an Open Future
3. Conclusion

The Big Picture

Just to be clear, I’m going to move the discussion beyond the IBM system above to talk about the "big picture" of what’s coming in the future. Despite the press release, the IBM system is not quite "Google Earth" just yet, because GE is much more than spinning a globe with URLs to click — it’s really a framework for fusing just about any geospatial data together, especially user-generated content, as well as a 3D search engine for the same. But building a true "Google Earth" for the human body turns out to be a challenge many times harder than building the actual Google Earth itself (which was not easy either). Any naive implementation is doomed to be a flash in the pan, a cute but limited toy, if it works at all.

How can I be so bold in my prediction? Because I’ve been researching this beast for a number of years and I have a pretty good idea of what it will take to build it.

For starters, the human body is volumetric, Google Earth isn’t — it’s only mostly 3D — most things live on the surface of our lumpy oblate sphereoid, divided up into essentially 2D zones, with altitude added on top. You can’t (yet) fly inside the Earth and see the layers: crust, mantle, and so on or add any data there. You can’t even see a cross-section, though at least that would be easy enough to add with some graphics engine tricks.

But the biggest problem, beyond dimensionality, scalability, data storage and streaming, and even beyond 3D navigation metaphors (which are all hard enough on their own) is the one most people don’t even think about until they sit down and try to actually build something like this — topology, geography, cartography — coming up with the equivalent of latitude and longitude or even X,Y,Z, for the human body. Oh, that.

Think about it. Cartography in the geographic context has been developing for many centuries, first in a crude approximation ("here, there be monsters…") and growing closer and closer to an accurate representation of the real world ("here, there be McDonalds…").

Cartography has had time to mature, to work out solutions for problems such as: how do we best project our lumpy oblate sphereoid onto a 2D piece of paper to most accurately convey relative size and distance? And, it turns out, there are a dozen or so significant coordinate systems to answer that very question, for various purposes and to varying degrees of success. And now 3D virtual globes bring us back to a more spherical real-time reality and the field evolves  (some would say, revolutionizes) yet again.

But "human body mapping" has a technically much harder problem to solve: there is only one Earth, but there isn’t even one official human body to refer to  – more like seven billion unique ones, and not just superficially, in terms of height, weight, or sex… The location, orientation, shape and size of body parts, organs, even blood vessels, can vary even within a single person over time, as when we move or sit, are injured or sick, pregnant or tense, and especially as we age. The biospatial mapping problem is effectively four dimensional, not three.

The general solution medicine has come up with to deal with that dynamism and individuality is an overly crude and ambiguous mapping system using words like anterior and posterior, medial and lateral with simple counting systems (C1, D6), paired with names that are the stuff of nightmares for first year med students, patients, and software engineers alike. Anatomy textbooks typically resort to artistically rendered, idealized drawings and a few sample photos to help teach what goes where. And then students spend time with actual cadavers getting their gloves dirty to really understand.

Here’s an analogy to help you understand how coarse and confusing the current mapping system can be. It would be like trying to find a specific apartment at the plexus of the posteromedial canal of the Islets of Manna-hata and the posterior broad main artery of New Amsterdam. Sure, you could show up at the corner of Broadway and Canal St. in Manhattan, given that description… and a Latin-to-English dictionary… and perhaps a history book… But without an actual street address and apartment number — or the equivalent of latitude, longitude, and altitude — you’d still have to look around for a while until you found it, and then you’d have to remember its location for next time.

However, that and a magic marker are how many surgeons specify where they’ll place incisions on your body after looking at a few 2D x-rays and doing some hefty mental gymnastics. That’s one reason surgeons get paid so much, and also why there are as many dumb mistakes as there are (which goes back to cost). Just as a map of the coast could save your royal armada from doom, a correct digital dynamic map of the human body would save many lives and who knows how much money.

And the problem is inordinately harder when you talk about the human nervous system. Compared to the brain, most of the body is relatively uniform from person to person, even accounting for size — and simple too. The cortical folds, or convolutions (known as gyri and solci), are as unique as any fingerprint and apparently their topology is functionally significant as well. And if there’s any place where precision really matters, it’s in the brain. We don’t want neurosurgeons stomping around our brains like conquistadors, exploring if you will, to determine if they’re planting the national flag in the correct tissue.

Towards the Open Future

So researchers have been trying to come up with ways to map the brain that can apply to any number of people, despite our many differences and the complexity of the information. It’s an active area of research, with the ultimate goal of solving that important problem for the whole body too, from birth to death and moment to moment. It can and will be done — and hopefully, in an open way, especially if the goal is to unify doctor-patient communication, medical teaching curricula, and even scientific discourse under one framework and with one free-to-use application.

Now, that won’t stop a few eager startups from offering up solutions under dream of being "The Next Google Earth," hoping to be snatched up by Google or go public. That can’t be avoided, I suppose. But when we start talking about mapping your MRIs, CAT scans, x-rays, surgeries, dental work, and so on, managing all of your health information in a way that works from person to person and time to time, we really need to do it right from the get-go. Lives are on the line, and trillions of dollars of vested interest are looking on with intense interest. The problem is just too big, and too complex for any single entity to solve or especially own, even Google, Microsoft and IBM — and they know it, or at least they say they do. Time will tell.

Remember, Keyhole and later Google only patented certain key technical features of how it works, not the fundamental "how to map the Earth" solution. And Google’s goal is now to make KML an open standard, which is the right approach to building such a comprehensive framework to unify all geospatial data. The same kind of approach would need to apply to biospatial data, with even more work on filtering and privacy controls for the personal specifics.

But when the time comes that researchers and hobbyists alike find easy access to that fully-annotatable Google-Earth-like application for the human body, on which they can post HML ("human markup language," which has so far been focused on superficial, external, or expressed traits) or find answers, you’ll really have something special. You’ll see the same kind of emergence and new market potential we see with Google Earth and the like.

So, for example… The software might show you and your doctor your virtual body, with your own medical history of course, but now with all sorts of added information re-mapped from external data sources to match your personal details. Click on your liver, and you’ll bring up expert systems to help diagnose concerns, or reveal the latest experimental treatments, drug suggestions, or links to Alcoholics Anonymous. Medical students would have access to the standard course-work as well as the latest research, contradicting the 30-year old information that’s still being taught (and occasionally killing people) around the world.

I know several of the leading academic researchers who are working on the biospatial problem (outside of Google, of course) — especially in the brain — and it’s a been keen area of interest for me and my wife (who is a neuroscientist and neuroanatomy lecturer, as it turns out). But it will take time and patience and hard work to see the results from the various teams and companies and work through the inevitable phases of secrecy and hype to get to the other side.

Conclusion

While it’s powerful and timely, the IBM Zurich project is not exactly a new idea. Their claim is somewhat of a stretch, as is the comparison to Google Earth just yet, though their progress thus far is still impressive. In reality, some of the same people who were thinking ahead about using the Earth as the most intuitive interface for geospatial data ten, fifteen years ago were well aware of (and even doing research on) how the human body would also make the ideal interface to things like… say, patient medical records…

And that was before the general public started seeing the benefits of a 3D spatial browser like Google Earth. But what’s described from IBM research is only a sliver of the potential win here. I would consider this demo to be the "low hanging fruit" compared to what’s possible and coming in the next 3-5 years…

Keep in mind, when I wrote the old code to draw names on curving roads and near placemarks for the very first version of Keyhole’s software, I could read a few basic textbooks on cartography and apply my engineering skills to make it work in a dynamic, real-time context. When I wrote the early pre-cursor to KML (for adding UI elements and new features mostly), I chose to make up a simple lexical grammar because XML was barely known and not yet standard. In the case of mapping the human body, we’re still in 1492, very little is settled, and it’s still is a whole new world we’re trying to conquer and understand.

_________

_{P.S. apologies to Native-Americans for being on the metaphorical butt-end of my Columbus references.}

I blogged a while back about virtualization and privacy issues for both Street View and Maps/Earth, which, for starters, implies to a [predicted] future version of Street View that erases people and even cars from the imagery you see. So let’s start there.

The main reason for removing people and cars is the same reason you’d want the base Google Earth imagery to lack clouds:

• These things tend to block your view. You can’t really look behind them in a simple (essentially 2D) panoramic image.
• They only represent one snapshot in time vs. a broader/more virtualized essence of the place.
• They make it confusing to add dynamic versions of the same things on top of what’s permanently baked into the imagery. You can already see this with 3D buildings on top of 2D ones. (similarly, adding dynamic lighting and shadows is also hard when there are already shadows in the imagery)
• And in the case of people, we tend to not like being included in commercial imagery without our permission — perhaps cars and clouds feel the same way, but they don’t represent any known marketing demographic, nor do they sue.

Now, that doesn’t mean there shouldn’t ever be people or cars or clouds in Google’s version(s) of the Earth. It just means those should only be included for a much better purpose than "they just happened to get caught in the camera’s lens" and they should be first removed, and then re-added as needed.

For example, dynamic weather overlays make a lot of sense, once you remove the baked-in clouds. Moving 3D avatars also make sense in an opt-in application, e.g., in some 3D augmented-reality social network. And why not represent the real-time traffic layer with some corresponding density of cars?

All of those issues are solvable, if you first remove the items from the imagery and then add them back. The simplest method of removal is oversampling. Imagine taking multiple photos of the same place over a number of days. Most of the time, pixel for pixel, transient objects will show up in only one of the images, so you can basically vote on each pixel — majority wins. But this requires multiple passes over each city, when Google is presently racing to get coverage everywhere, and there are alignment issues, so give it some time (and perhaps some public pressure).

One obvious thing that Google can do right away is better integrate Street View with its generally-strong (though not always right) driving directions — now with impressive "route-dragging." It turns out, there’s a very compelling reason for integrating Street View with driving directions: most people can’t read maps.

Basically, there’s two kinds of people in the world — those who maintain re-orientable 2D/3D mental maps of the world around them, and those who navigate mainly by visual cues, landmarks, i.e., what they can see from their direct 1st-person perspective.

The people who can re-orient their mental maps have what we call high spatial cognition. The vast majority of the human race, however, has just enough spatial cognition to reach for a bag of potato chips and avoid walking into walls. That doesn’t mean those people are dumb — until recently, high spatial cognition was generally only useful for throwing spears. There are many other kinds of intelligence. And those with high spatial intelligence are simply more cut out for jobs in 3D graphics, architecture, industrial design, and perhaps billiards. People who can’t tell the difference between panoramic 2D images and actual 3D graphics might be really good at business or marketing, for example, but still get lost in a cul-de-sac.

The point is that driving directions, for the vast majority of the world, would do much better with a series of 1st person images or video that shows you where to get off the highway with an actual image of the exit sign and off-ramp, saying, "turn right here" in the same way that one might say to a driver, "turn where that blue car just went." That, and as I discovered this weekend, the Home Depot on Route 17 is not quite where Google Maps says it us. Having images of my route would be a nice "sanity check."

Frankly, I’d expect this feature way before true virtualization as described above. But the closest they come to it right now is that you can use Street View when interactively examining your driving directions — drag the little yellow AOL-like guy around to see just one Street View image at a time. Perhaps they’ll soon offer to replace or combine the series of overhead street maps (which they include when you print) with 1st person views of each stage of navigation along the route.

What might come after that? The main thing to do, I’m guessing, is integrate the vast quantities of Street View data with Google Earth in a more 3D fashion, solving these 2D panoramas for depth and turning everything into textures and polygons.

The ultimate goal would be a view of the Earth not only with basic buildings, but with well detailed street-level views so you could zoom down and even [virtually] walk around. Go one step further, and Google could take their $10 per store photography project and recreate virtual interiors of semi-public places, perhaps with links or embedded shopping when you go inside.

That’s when I think we might start to see avatars in Google Earth — when there’s some actual reason for people to have a sense of their own body and of other people in the virtual world, not that a Second Life / GE combination is likely, for reasons I’ve already outlined.

What else might there be? Well, integrating live street cameras would be a neat trick. All it really takes is knowing the position and orientation of the camera and coming up with a flash video player that can project 2D movies on an arbitrary 3D polygon, rather than always in the plane of your screen. It’s even simpler if the video is in a separate window.

Having the density of image capture points (aka "nodes") be high enough to resemble moving video is also on the horizon. The cameras Google is reportedly driving around with are capable of capturing and processing 30 FPS and 360-degrees [edit: see the first comment below for a cool example of panoramic video]. The only real limitation is server-side storage for the higher density of nodes (not a big deal for Google) and bandwidth to your browser. Caveat: virtualizing the scene to a real-time 3D model is still preferable to streaming video because we don’t always want to move in the exact path (and direction) that Google took. Just imagine trying to drive down a street backwards — with Street View Video, time would seem to run backwards.

Clickable store-front information is an obvious must-have feature — show menus for restaurants, and so on. And like Google Earth, we’ll want to turn on information layers beyond the mere street names. Given the vast store of information already in Google Earth, adding this to Street View shouldn’t be too hard — even rendering KML should be doable, given enough time and energy.

The only really hard problem is the one I mentioned earlier — turning essentially 2D panoramic Street View images into 3D models — the start of the art still requires some manual labor, which kind of hampers any planet-scale effort. But new camera technology for recording distance per pixel could soon save the day, as well as an interesting recent acquisition.

At that point, and as browsers are more and more capable of 3D, I expect to see Google Street View, Maps, and Earth become one truly integrated product, offering different perspectives on the same digital earth.

As always, I’m merely an observer. Google doesn’t tell me their plans. But let’s check back in a year or so and see how much of this comes to pass.

After reading an article called "How Google Earth Works" on the great site HowStuffWorks.com, it became apparent that the article was more of a "how cool it is" and "here’s how to use it" than a "how Google Earth [really] works."

So I thought there might be some interest, and despite some valid intellectual property concerns, here we are, explaining how at least part of Google Earth works.

Keep in mind, those IP issues are real. Keyhole (now known as Google Earth) was attacked once already with claims that they copied someone else’s inferior (IMO) technology. The suit was completely dismissed by a judge, but only after many years of pain. Still, it highlights one problem of even talking about this stuff. Anything one says could be fodder for some troll to claim he invented what you did because it "sounds similar." The judge in the Skyline v. Google case understood that "sounding similar" is not enough to prove infringement. Not all judges do.

Anyway, the solution to discussing "How Google Earth [Really] Works" is to stick to information that has already been disclosed in various forms, especially in Google’s own patents, of which there are relatively few. Fewer software patents is better for the world. But in this case, more patents would mean we could talk more openly about the technology, which, btw, was one of the original goals of patents — a trade of limited monopoly rights in exchange for a real public benefit: disclosure. But I digress…

For the more technically inclined, you may want to read these patents directly. Be warned: lawyers and technologists sometimes emulsify to form a sort of linguistic mayonnaise, a soul-deadening substance known as Patent English, or Painglish for short . If you’re brave, or masochistic, here you go:

1. Asynchronous Multilevel Texture Pipeline
2. Server for geospatially organized flat file data

There are also a few more loosely related Google patents. I don’t know why these are shouting, but perhaps because they’re very important to the field. I’ll hopefully get to these in more detail in future articles:

3. DIGITAL MAPPING SYSTEM
4. GENERATING AND SERVING TILES IN A DIGITAL MAPPING SYSTEM
5. DETERMINING ADVERTISEMENTS USING USER INTEREST INFORMATION AND MAP-BASED LOCATION INFORMATION
6. ENTITY DISPLAY PRIORITY IN A DISTRIBUTED GEOGRAPHIC INFORMATION SYSTEM (this one will be huge)

And there is this more informative technical paper from SGI (PDF) on hardware "clipmapping," which we’ll refer to later on. Michael Jones, btw, is one of the driving forces behind Google Earth, and as CTO, is still advancing the technology.

I’m going to stick closely to what’s been disclosed or is otherwise common technical knowledge. But I will hopefully explain it in a way that most humans can understand and maybe even appreciate. At least that’s my goal. You can let me know.

Big Caveat: the Google Earth code base has probably been rewritten several times since I was involved with Keyhole  and perhaps even after these patents were submitted. Suffice it to say, the latest implementations may have changed significantly. And even my explanations are going to be so broad (and potentially out-dated) that no one should use this article as the basis for anything except intellectual curiosity and understanding.

Also note: we’re going to proceed in reverse, strange as it may seem, from the instant the 3D Earth is drawn on your screen, and later trace back to the time the data is served. I believe this will help explain why things are done as they are and why some other approaches don’t work nearly as well.

Part 1, The Result: Drawing a 3D Virtual Globe

There are two principal differences between Google Maps and Earth that inform how things should ideally work under the hood. The first is the difference between fixed-view (often top-down) 2D & free-perspective 3D rendering. The second is between real-time and pre-rendered graphics. These two distinctions are fading away as the products improve and converge. But they highlight important differences, even today.

What both have in common is that they begin with traditional digital photography — lots of it — basically one giant high-resolution (or multi-resolution) picture of the Earth. How they differ is largely in how they render that data.

Consider: The Earth is approximately 40,000 km around the waist. Whoever says it’s a small world is being cute. If you stored only one pixel of color data for every square kilometer of surface, a whole-earth image (flattened out in, say, a mercator projection) would be about 40,000 pixels wide and roughly half as tall. That’s far more than most 3D graphics hardware can handle today. We’re talking about an image of 800 megapixels and 2.4 gigabytes at least. Many PCs today don’t even have 2GB of main memory. And in terms of video RAM, needed to render, a typical PC has maybe 128MB, with a high-end gaming rig having upwards of 512.

And remember, this is just your basic run-of-the-mill one-kilometer-per-pixel whole-earth image. The smallest feature you could resolve with such an image is about 2 kilometers wide (thank you, Mr. Nyquist) — no buildings, rivers, roads, or people would be apparent. But for most major US cities, Google Earth deals in resolutions that can resolve objects as small as half a meter or less, at least four thousand times denser, or sixteen million times more storage than the above example.

We’re talking about images that would (and do) literally take many terabytes to store. There is no way that such a thing could ever be drawn on today’s PCs, especially not in real-time.

And yet it happens every time you run Google Earth.

Consider: In a true 3D virtual globe, you can arbitrarily tilt and rotate your view to pretty much look anywhere (except perhaps underground — and even that’s possible if we had the data). In all 3D globes, there exists some source data, typically, a really high-resolution image of the whole earth’s surface, or at least the parts for which the company bought data. That source data needs to be delivered to your monitor, mapped onto some virtual sphere or ideally onto small 3D surfaces (triangles, etc..) that mimic the real terrain, mountains, rivers and so on.

If you, as a software designer, decide not allow your view of the Earth to ever tilt or rotate, then congrats, you’ve simplified the engineering problem and can take some time off. But then you don’t have Google Earth.

Now, various schemes exist to allow one to "roam" part of this ridiculously large texture. Other mapping applications solve this in their own way, and often with significant limitations or visual artifacts. Most of them simply cut their huge Earth up into small regular tiles, perhaps arranged in a quadtree, and draw a certain number of those tiles on your screen at any given time, either in 2D (like Google Maps) or in 3D, like Microsoft’s Virtual Earth apparently does.

But the way Google Earth solved the problem was truly novel, and worthy of a software patent (and I am generally opposed to software patents). To explain it, we’ll have to build up a few core concepts. A background in digital signal theory and computer graphics never hurts, but I hope this will be easy enough that that won’t be necessary.

I’m not going to explain how 3D rendering works — that’s covered elsewhere. But I am going to focus on texture mapping and texture filtering in particular, because the details are vital to making this work. The progression from basic concepts to the more advanced texture filtering will also help you understand why things work this way, and just how amazing this technology really is. If you have the patience, here’s a very quick lesson in texture filtering.

The Basics

The problem of scaling, rotating and warping basic 2D images was solved a long time ago. The most common solution is called Bilinear Filtering. All that really means is that for each new (rotated, scaled, etc..) pixel you want to compute, you take the four "best" pixels from your source image and blend them together. It’s "bilinear" because it linearly blends two pixels at a time (along one axis), and then linearly blends those two results (along the other axis) for the final answer.

[A "linear blend," in case it's not clear, is butt simple: take 40% of color A, and 60% of color B and add them together. The 40/60 split is variable, depending on how "important" each contributor is, as long as the total adds up to 100%.]

That functionality is built into your 3D graphics hardware such that your computer can nowadays do literally billions of these calculations per second. Don’t ask me why your favorite paint program is so slow.

The problem being addressed can be visualized pretty easily — that’s what I love about computer graphics. It turns out, whenever we map some source pixels onto different (rotated, scaled, tilted, etc…) output pixels, visual information is lost.

The problem is called "aliasing" and it occurs because we digitally sampled the original image one way, at some given frequency (aka resolution), and now we’re re-sampling that digital data in some other way that doesn’t quite match up.

1. A simple low-res (11×11 pixel) image is about to be rotated. (the grid lines are merely to delineate pixels) 3. Close up of one output pixel. Bilinear interpolation averages the "best" four source pixels for each new destination pixel (shown as black border with white dots) based on their relative importance (ideally: fractional area).

2. Each pixel in the destination grid overlaps multiple pixels from the rotating original. 4. After bilinear interpolation, the resulting rotated image has some clear (or rather blurry) issues.

Now, when we talk about output pixels and destinations, it doesn’t much matter if the destination is a bitmap in a paint program or the 3D application window that shows the Earth. Aliasing happens whenever the output pixels do not line up with the sampling interval (frequency, resolution) of the source image. And aliasing makes for poor visual results. Dealing with aliasing is about half of what texture mapping is all about. The rest is mostly memory management. And the constraints of both inform how Google Earth works.

The mission then is to minimize aliasing through cleverness and good design. The best way to do this is to get as close as possible to a 1:1 correspondence between input and output pixels, or at least to generate so many extra pixels that we can safely down-sample the output to minimize aliasing (also known as "anti-aliasing"). We often do both.

Consider: for resizing images, it only gets worse — each pixel in your destination image might correspond to hundreds of pixels of source imagery, or vice-versa. Bilinear interpolation, remember, will only pick the best four source pixels and ignore the rest. So it can therefore skip right over important pixels, like edges, shadows, or highlights. If some such pixel is picked for blending during one frame and skipped over subsequently, you’ll get an ugly "pixel-popping" or scintillation effect. I’m sure you’ve seen it in some video games. Now you know why.

Tilting images (or any 3D transformation) is even more problematic, because now we have elements of scaling and rotation, but also a great variation in pixel density across rendered surfaces. For example, in the "near" part of a scene, your nice high-res ground image might be scaled up such that the pixels look blurry. In the "far" part of the scene, your image might appear scintillated (as above) because simple 2×2 bilinear interpolation is necessarily skipping important visual details from time to time.

Copyright, Microsoft Virtual Earth Here’s an example of where a certain kind of texture filtering causes poor results. The text labels are hardly readable (why they’re painted into the terrain image at all is another issue).

Better Filtering, Revealed

Most consumer 3D hardware already supports what’s called "tri-linear" filtering. With tri-linear and a closely coupled technique called mip-mapping, the hardware computes and stores a series of lower resolution versions of your source image or texture map. Each mip-map is automatically down-sampled by a factor of 2, repeatedly, until we reach a 1×1 pixel image whose color is the average of all source image pixels.

So, for example, if you provided the hardware with a nice 512×512 source image, it would compute and store 8 extra mip-levels for you (256, 128, 64, 32, 16, 8, 4, 2, and 1 pixel square). If you stacked those vertically, you might more easily visualize the "mip-stack" as an upside down pyramid, where each mip-level (each horizontal slice) is always 1/2 the width of the one above.

Drawing of a mip-map pyramid (not to scale). The X depicts trilinear filtering sampling two mip-levels for an in-between pixel to reduce aliasing.

During 3D rendering, mip-mapping and tri-linear filtering take each destination pixel, pick the two most appropriate mip-levels, essentially do a bi-linear blend on both, and then blend those two results again (linearly) for the final tri-linear answer.

So for example, say the next pixel would have no aliasing if only the source image had a resolution of 47.5 pixels across. The system has stored power of two mip maps (16, 32, 64…). So the hardware will cleverly use the 64×64 and 32×32 pixel versions closest to the desired sampling of 47.5, compute a bilinear (4-sample) result for each, and then take those two results and blend them a third time.

That’s tri-linear filtering in a nutshell, and along with mip-mapping, it goes a great distance to minimizing aliasing for many common cases of 3D transformations.

Remember: so far, we’ve been talking about nice, small images, like 512×512 pixels. Our whole-earth image will need to be millions of pixels across. So one might consider making a giant mip-map of our whole-earth image, at say one meter resolution. No problem, right? But you’ll realize fairly soon that would require a mip-map pyramid 26 levels deep, where the highest resolution mip-level is some 66 million pixels across. That simply won’t fit on any 3D video card on the market, at least not in this decade.

I’m guessing Microsoft’s Virtual Earth gets around this limit by cutting their giant earth texture into many smaller distinct tiles of, say, 256 pixels square, where each gets mip-mapped individually. That approach would work to an extent, but it would be relatively slow and give some of the visual artifacts, like the blurring we see above, and a popping in and out of large square areas as you zoom in and out.

There’s one last concept about mip-maps to understand before we move on to the meat of the issue. Imagine for a moment that the pixels in the mip-map pyramid are actually color-coded as I’ve indicated above, with an entire layer colored red, another yellow, etc.. Drawing this on a tilted plane (like the Earth’s ground "plane") would then seem to "slice through" the pyramid at an interesting angle, using only those parts of the pyramid that are needed for this view.

It’s this characteristic of mip-mapping that allows Google Earth to exist, as we’ll see in a minute.

A typical tilted Google Earth image (copyright & courtesy Google).

The same view, using color to show which mip levels inform which pixels

The example on the left shows a normal 3D scene from Google Earth, as well as a rough diagram showing from where in the mip-stack a 3D hardware system might find the best source pixels, if they were so colorized.

The nearer area gets filled from the highest-resolution mip-level (red), dropping off to lower and lower resolutions as we get farther from the virtual point of view. This helps avoid the scintillation and other aliasing problems we talked about earlier, and looks quite nice. We get as close as possible to a 1:1 correspondence between source and destination, pixel for pixel, so aliasing is minimized.

Even better still, tri-linear filtering 3D graphics hardware has been improved with something called anisotropic filtering (a simple preference option in Google Earth) which is essentially the same core idea as the previous examples, but using non-square filters, beyond the basic 2×2. This is very important for visual quality, because even with fancy mip-mapping, if you tilt a textured polygon to a very oblique angle, the hardware must choose a low-resolution mip-level to avoid scintillation on the narrow axis. And that means the whole polygon is sampled at too-low a resolution, when it’s only one direction that needed to dip down to the low-res stuff. Suffice it to say, if your hardware supports anisotropic filtering, turn it on for best results. It’s worth every penny.

Now, to the meat of the issue

We still have to solve the problem of how to mip-map a texture with millions of pixels in either dimension. Universal Texture (in the Google Earth patent) solves the problem while still providing high quality texture filtering. It creates one giant multi-terabyte whole-earth virtual-texture in an extremely clever way. I can say that since I didn’t actually invent it. Chris Tanner figured out a way to do on your PC what had only ever been done on expensive graphics supercomputers with custom circuitry, called Clip Mapping (see SGI’s pdf paper, also by Chris, Michael, et al., for a lot more depth on the original hardware implementation). That technology is essentially what made Google Earth possible. And my very first job on this project was making that work over an internet connection, way back when.

So how does it actually work?

Well, instead of loading and drawing that giant whole-earth texture all at once — which is impossible on most current hardware — and instead of chopping it up into millions of tiles and thereby losing the better filtering and efficiency we want, recall from just above that we typically only ever use a narrow slice or column of our full mip-map pyramid at any given time. The angle and height of this virtual column changes quite a bit depending on our current 3D perspective. And this usage pattern is fairly straightforward for a clever algorithm to compute or infer, knowing where you are and what the application is trying to draw.

A Universal Texture is both a mip-map, plus a software emulated clip-stack, meaning it can mimic a mip-map of many more levels and greater ultimate resolution than can fit in any real hardware. Note: though this diagram doesn’t depict it as precisely as the paper, the clip stack’s "angle" shifts around to best keep the column centered.

So this clever algorithm figures out which sections of the larger virtual texture it needs at any given time and pages only those from system memory to your graphics card’s dedicated texture memory, where it can be drawn very efficiently, even in real-time.

The main modification to basic mip-mapping, from a conceptual point of view, is that the upside down pyramid is no longer just a pyramid, but is now much, much taller, containing a  clipped stack of textures, called, oddly enough, a "clip stack," perhaps 16 to 30+ levels high. Conceptually, it’s as if you had a giant mip-map pyramid that’s 16-30 levels deep and millions to billions of pixels wide, but you clipped off the sides — i.e., the parts you don’t need right now.

Imagine the Washington monument, upside down and you’ll get the idea. In fact, imagine that tower leaning this way or that, like the one in Pisa, and you’ll be even closer. The tower leans in such a way that the pixels inside the tower are what you need for rendering right now. The rest is ignored.

Each clip-level is still twice the resolution of the one "below" it, like all mip-maps, and nice quality filtering still works as before. But since the clip stack is limited to a fixed but roaming footprint, say 512×512 pixels wide (another preference in Google Earth), that means that each clip-level is both twice the effective resolution and half the coverage area of the previous. That’s exactly what we want. We get all the benefits of a giant mip-map, with only the parts relevant to any given view.

Put another way, Google Earth cleverly and progressively loads high-res information for what’s at the focal "center" of your view (the red part above), and resolution drops off by powers of two from there. As you tilt and fly and watch the land run towards the horizon, Universal Texture is optimally sending only the best and most useful levels of detail to the hardware at any given time. What isn’t needed, isn’t even touched. That’s one thing that makes it ultra-efficient.

It’s also very memory-efficient. The total texture memory for an earth-sized texture is now (assuming this 512 wide base mip-map, and say 20 extra clip-levels of data) only about 17 megabytes, not the dozens to hundreds of terabytes we were threatened with before. It’s actually doable, and worked in 1999 on 3D hardware that had only 32 MB or less. Other techniques are only now becoming possible with bigger and bigger 3D cards.

In fact, with only 20 clip-levels (plus 9 mip levels for the base pyramid), we see that 2²⁹ yields a virtual texture capable of up to 536 million pixels in either dimension. Multiply that by 1/2 vertically, gives an virtual image of a few hundred terapixels in area, or enough excess capacity to represent features as small as 0.15 meters (about 5 inches), wherever the data is available. And that’s not the actual limit. I simply picked 20 clip levels as a reasonable number. And you thought the race for more megapixels on digital cameras was challenging. Multiply that by a million and you’re in the planetary ballpack.

Fortunately, for now, Google only really has to store a few dozen terapixels of imagery. The other beauty of the system is that the highest levels of resolution need not exist everywhere for this to work. Wherever the resolution is more limited, wherever there are gaps, missing data, etc.. the system only draws what it has. If there is higher resolution data available, it is fetched and drawn too. If not, the system uses the next lower resolution version of that data (see mip-mapping above) rather than drawing a blank. That’s exactly why you can zoom into some areas and see only a big blur, where other areas are nice and crisp. It’s all about data availability, not any hard limit on the 3D rendering. If the data were available, you could see centimeter resolution in the middle of the ocean.

The key then to making this all work is that, as you roam around the 3D Earth, the system can efficiently page new texture data from your local disk cache and system memory into your graphics texture memory. (We’ll cover some of how stuff gets into your local cache next time). You’ve literally been watching that texture uploading happen without necessarily realizing it. Hopefully, now you will appreciate all the hard work that went into making this all work so smoothly — like feeding an entire planet piecewise through a straw.

Finally, there’s one other item of interest before we move on. The reason this patent emphasizes asynchronous behavior is that these texture bits take some small but cumulative time to upload to your 3D hardware continuously, and that’s time taken away from drawing 3D images in a smooth, jitter-free fashion or handling easy user input — not to mention the hardware is typically busy with its own demanding schedule.

To achieve a steady 60 frames per second on most hardware, the texture uploading is divided into small, thin slices that very quickly update graphics video memory with the source data for whatever area you’re viewing, hopefully just before you need it, but at worst, just after. What’s really clever is that the system needs only upload the smallest parts of these textures that are needed and it does it without making anyone wait. That means rendering can be smooth and the user interface can be as fluid as possible. Without this asynchronicity, forget about those nice parabolic arcs from coast to coast.

Now, other virtual globes can also virtualize the whole-earth texture, perhaps they cut it into tiles, and even use multiple power-of-two resolutions like GE does. But without the Universal Texturing component or something better, they’ll either be limited to 2D top-down rendering, or they’ll do 3D rendering with unsatisfying results, blurring, scintillation, and not nearly as good performance for streaming the data from the cache into texture memory for rendering.

And that’s probably more than you ever wanted to know about how the whole Earth is drawn on your screen each frame.

Now that Linden Labs has open-sourced the Second Life client, if any Google Earth engineers chose to study it, I might no longer be the only person lucky enough to know both the Google Earth and Second Life internals well enough to make a bold statement on a mashup of the two. It would be great if others (besides me) could do so soberly. Because all I’m hearing lately is a lot of “wouldn’t it be great?” and not much “here’s how” and, better yet, “here’s why” practical discussion.

I’ve said before that I don’t think any direct integration between SL and GE is wise at this point, at least not in their present forms and with the present missions of each application. I’ll try to elaborate on that and see if I can convince you that the best thing overall is for each app to evolve along its own path. But I will point out a few areas where each could benefit from at least mimicking the other.

[Edit: See Wade’s comment below. I want to also make it clear that the issues of a SL/GE mashup are purely hypothetical, and are meant more to serve as a framework for understanding some important technical and usability issues in these two different kinds of virtual worlds.]

The issues:

Mirror Worlds vs. Virtual Worlds – Second Life is fictional, whimsical, experimental. Google Earth is a reflection of the real world, and according to its CTO, will remain so in the future. In SL, you can make or do almost anything. GE is meant to be a platform for delivering geo-referenced information that is strictly relevant and useful (as well as fun) for our first lives.

Now, GE bent those rules a little by adding custom layers and 3D import — there’s no clear rule that says your KML file has to represent a real object or building. And KML search will have a hard time distinguishing between real vs. fictional search results (even as they add pagerank-style features). The problem is that fictional results pollute real-world uses, and if they get out of hand, Google will have to somehow segregate those realms. I definitely don’t want to search for directions to the Home Depot and find that my path is blocked by a giant robot or a self-replicating pile of poo.

And for SL, the opposite problem exists — growth of the landscape was designed to be geometric and ad hoc, not mirroring any real-word geography. There’s some exceptions to that as well, one of which I suggested to a SL-oriented colleague a while back, and that is that a single SL sim could serve as a stand-in for the entire earth, in miniature, linking via teleports to any number of other 1:1 scale sims to flesh out the real-earth geography, albeit with giant gaps — meaning you’d have to pop back to the “hub” to get from place to place — so you can see some of the difficulties.

Direct Integration – In this pure hypothetical, we’re talking about the two companies working together (e.g., collaboration, or outright purchase/merger) or GE open sourcing their code so a third party can do the mashup. I don’t think that’ll happen anytime soon. Now, if Google can pay $1.7B for YouTube, I think anything is possible on the merger front. Money seems to be no object, but they’re not being irrational. For the price Linden might demand, they’ll need to show, say, 10-20M regular visitors and a revenue model that Google thinks will integrate with their core business and scale indefinitely.

I’d also guess, purely based on rumors and the engineering personality type, Google would be more likely to try rolling their own social VW before buying an established company like Linden. If Google does that, and they do as well as Google Video did against YouTube, that’s when I think buying Linden would make more sense. I don’t expect it to happen, if at all, for several years. An IBM-style purchase is more likely, IMO. But what do I know?

Mashups – This is a bit less speculative. What Google could do fairly easily in the near term is release a closed-source version of GE that is more like a toolkit or library, closer to the Microsoft model for their Virtual Earth offering, though designed for real-time rendering. A toolkit would allow someone to build a virtual environment that had both the real earth and whatever avatar systems they wanted to throw in. It wouldn’t necessarily be SL, but it would be the first social VW based on the real/mirror Earth. More likely than that, though, is some licensing specific deal for the GE technology if the price is right (I don’t know what that price is).

It would also be technically feasible for Google to do the reverse — embed Google Earth’s OpenGL rendering code into the open-sourced SL client, such that anyone using that new SL client could create an instance of a GE globe inside SL as an in-world application as opposed to the actual terrain you stand on. The only problem would be licensing, if that’s an issue at all. But without the analogous GE source or any license to use GE’s umpteen terabytes of data, neither Linden nor any 3rd parties would be in any position to do this from their end.

Client / Server – The one mashup that pundits seem to call for the most — a horde of real, active, chatting Second Life avatars inside one big shared Google Earth — is unlikely to happen in the near-term for several technical reasons. One is that SL is much more than the PC client you install. Without the requisite number of SL servers arranged into a lat/long map of the earth, or at least the parts we care about, none of your avatars would be able to interact. So while mashing-up the client you’d also need to handle both SL and GE servers at the same time, preferably in some coordinated fashion. For example, if GE holds the terrain and building data, SL servers would need to know about those to perform the physical part of the simulation, collision detection and so on. So there’s at least a three-way dialog going on, not just an integrated SL/GE client. And that makes us engineers very unhappy.

Putting just the bodies in GE is easier, although we’re talking the cheesy, lame, unsupported version with very little interactivity. You can capture your avatar in a static pose from SL using an OpenGL based interceptor, convert that to Collada format and then KML and import it into GE. Animating it is trickier. Forget about dancing or even moving your limbs. The best you can probably hope for without extensive kludge is using KML’s “network update” feature to fetch your avatar’s current location (and that of anyone else near you) from a special server you’d supply. In this case, you also won’t get collision detection with 3D buildings unless you do it yourself. And similarly, you’ll have to handle all interactions among avatars. Essentially, you’ll be recreating a kludgey version of SL’s simulators and using GE + KML pseudo-scripting as the client rendering engine. It’s not something I’d want to spend my time on.

Scale – Given the land area of the earth is reportedly 150,000,000km², and each SL server currently handles 256×256m (16 servers per km²), a quick calculation reveals roughly 2.4 billion servers would be needed for the land alone. If we just did Manhattan (87.46 km²) , it would take about 1400 servers, or roughly 1/4th of their current total just for one densely populated island. And we’re not even talking about the limits on concurrent users yet.

Clearly, if GE and SL were ever to mate, SL would need to move from a rigid grid to something more adaptable, for example, a quad-tree that gets subdivided depending on where the people virtually are, or better yet, based on who is interacting with whom at any given time. I worked with a company briefly (they never got their funding) who was pursuing something like this to handle 1 million simultaneous users. It’s not a trivial problem, but would be necessary to scale SL up to a full planet, if they ever see the need.

The next issue of scale is that of “levels of detail.” GE was designed from the ground up to seamlessly zoom through 20 or more powers of two (2²⁰ : 1 scaling) when zooming from way out in space down to a spot on the ground. SL was designed to let you walk, teleport, or fly relatively close to the ground. It essentially has four levels of detail — near, far, and off, plus we’ll count the nice 2D overhead map. But SL isn’t designed to handle viewing the entire world in so many different scales, smoothly moving through all of them. Without big architectural changes, it would be very difficult to do the kind of flying and zipping around one does in GE.

One final note under the category of “scale.” The Collada file format may have some built-in features to properly convey the scale of objects, but it’s rarely used or even obeyed from what I can tell. All 3D models have built-in assumptions about scale. They’re just a bunch of numbers after all, not smart about their context or nature. If someone exports data in centimeters and another program assumes those are feet or meters, you’re going to have to write yet another program to convert everything to match up, or your avatar may be as tall as the Empire State Building, or too small to see.

Other issues – I haven’t even touched on the issues of converting from procedural or parametric objects to polygons, which is something I have a lot of experience with. That’s a whole other discussion which I could spend hours on. Suffice it to say, it’s a hard problem, but one in which there is an easy solution — if it’s designed in to the system.

But for now, let’s worry more about whether or not people want this and then get around to fixing the “nitty gritties.”

The How’s — What’s really possible?

1. If Google is so inclined, I’d love to see them create a free or licensed DLL that encapsulates the OpenGL rendering code from Google Earth in a form that could be invoked inside of someone else’s OpenGL-based 3D engine. It would at a minimum require functions to set the viewing position, layer state, etc… and probably most things you would already do from the UI. This would at least allow someone to put a Google Earth inside Second Life, as a 3D object one could poke at. It would not be a regular place in SL, but even that could be improved with time. If Google figures out how to monetize GE with ads, such a DLL might require showing those ads as well, or we could see it as a one-off license to some well-funded company.

2. If Linden is so inclined, they could work on a new kind of “grid” that could scale up to a full earth-sized geography, solving the problem of zooming in and out as a method of managing that new expansive scope. FWIW, I think it’s worth their time, because teleporting is a bad way work around the problems of walking and flying. GE’s zooming gets you there just as fast, but without losing your geo-spatial awareness. In other words, the continuous pan across the earth keeps your brain working on the relative positions of the places you visit, as in a big mental map. Teleporting loses the advantages of having one big shared space — you might as well have a bunch of small connected rooms at that point, which some people are working on. If Linden does this, they could be in a position to build an “Earth” app like in SnowCrash — one that they can even zoom into as a method of getting around. This need not be a literal mirror of the earth, but the technology I outlined is still important to make it work, regardless of the source data; satellites or users.

The Why’s — What will people actually want or need?

Here’s the big question, saved for the end. With all the “wouldn’t it be cool” talk, people seem miss the fundamental issue with new technology: people do the dumbest things. It’s not that people are necessarily dumb. It’s more an issue of finding the “lowest-energy” solutions rather than the most elegant ones.

MySpace is a good example of a “low-energy” solution, as compared to many more elegant ones that were presented. And there is no doubt that there will be a host of 3D MySpaces within the next 12 months to further test the theory (I’ve consulted for several but I’m not promoting any). All of them will face a fundamental problem: What is 3D good for? More specifically, why would I want to visit your virtual living room if you’re not there? Those questions have real answers, but perhaps not the most obvious ones.

Now, there are also some great reasons to want avatars in Google Earth. The more GE becomes a destination, rather than a tool to pick your destination, the more likely we are to say “meet me on 5th avenue for shopping,” and we mean the virtual version of each. This requires the avatars to interact. And if we go a little further to an application where I and a paid designer can work on my new house, you can see a need for GE to add in-world building tools to the mix.

As for a virtual economy and social networking tools, I don’t know. As long as Google doesn’t have to host your content themselves or care about bandwidth for user data, there really is no cost to them, and no big reason to quantify value for 3D objects. The “layer” approach makes it much easier to simply turn off content that’s just too slow. But for all of these, I see them being so tightly integrated with Google’s advertising economy and information delivery mission that I can only imagine them doing this in-house, slowly but surely adding more metaverse-like components that are more or less strictly tied to the real world.

Similarly, the path for SL is to make a bigger and bigger world and scale their way towards more and more simultaneous users. The “why” of making an entire earth in Second Life seems obvious — but the old joke “it’s a small world, but I wouldn’t want to paint it” comes to mind. If SL users really had 2.3 billion servers to populate, could 10 – 20 million users even do it in one second lifetime? Not by hand, I think.

GE at least has automated tools for capturing a world that’s already out there. Imagine if people really had to build a second version of the whole world by hand, with the level of detail SL desires?

It’s unlikely. But that’s where procedural modeling comes in. It’s not going to recreate the real world either, but there are known and waiting methods for building out entire planets full of detail. However, that’s clearly a third way — not user generated, and not mirroring the real world. And that discussion is best left for another day.