Archive for category Articles
It’s not often one gets to work on a product with a billion anything. I’m extremely fortunate to have had the opportunity to design/invent/iterate in the earliest days of at least three products in that league, and I’ve learned from others who have had more.
Here, I’m going to focus on Google Earth, which has well over a billion users, thanks to Google’s scale. I’ll pick examples from my experience to help illustrate the lessons.
[as always, my personal technical opinions here]
The LA Times author laments helplessly for most of the linked article. First, he claims Verizon is more evil than Google or Yahoo (who also effectively sell your data) because Verizon charges more up front. Well, I do agree companies that make money off our personal data should give us a cut or a discount, assuming we opt-in at all. It’s our data, right?
But the issue isn’t how badly Verizon is ripping us off relative to other companies. That’s a whole other box of lame.
Here the author gets closer:
Customers may be hard-pressed to understand fully what’s going on with the “enhanced” program. The Verizon Wireless notice is decidedly short on details.
Again, none of the companies listed in this post do a good job of “informed consent” IMO. If what Verizon is doing is not so bad, then people should be able to come to that conclusion given the honest facts, explained in terms they can understand, and then agree.
Were you aware of this? Probably not. Did you agree? Probably not.
But it gets better:
Debra Lewis, a Verizon Wireless spokeswoman, explained to me that when a customer registers on the company’s “My Verizon” website to see a bill or watch TV online, a “cookie,” or tracking software, is downloaded onto the customer’s home computer.
Most cookies are benign, allowing websites to provide better service to frequent visitors.
Verizon Wireless’ cookie allows a data-collection company working on Verizon’s behalf — Lewis declined to name which one — to gather information on which sites you visit after you leave “My Verizon.”
Hint: it could be any of these trackers, found by visiting the offending site with Ghostery installed.
- Adobe Test & Target
- DoubleClick Floodlight
- Omniture (Adobe Analytics)
You may want to install Ghostery and start blocking almost everything like this. These trackers do you no good, and you can put the few exceptions you need in the settings. You will be surprised to see how many tracking turds are quietly slipped into your pocket. The worst part is, if I go to Verizon’s site to find out how to opt out of this bullshit, I apparently get tracked for it…
Here’s the actual opt-out information, btw. FWIW, I collected the above list of trackers after ensuring I was opted out on the site. I guess they promise, on behalf of these random third party trackers, the data won’t be shared…
But it gets even crazier:
That information is “anonymized,” Lewis said, to mask the Verizon customer’s identity and is then shared with marketers, which can use the info to provide ads on the customer’s Verizon Wireless device that match his or her home-computer interests.
What makes this all potentially class-action-worthy IMO (IANAL) is that Verizon claims that by simply replacing your name with a unique ID, the service is anonymous and therefore safe.
“In addition to the customer information that’s currently part of the program, we will soon use an anonymous, unique identifier we create when you register on our websites.”
Does anyone else remember when AOL put their carefully anonymized search logs out on the internet…
It took almost no time for someone to figure out a lot of who was who despite the obfuscation, because the same “anonymous” IDs were used for each search by the same person, over and over, and could therefore be accumulated and cross-referenced. Simple deduction, Watson.
How hard will it be for a 3rd party marketer, given a list of your most frequent locations and your “double secret anonymous” ID, to look up the address of said frequent locations, and a few ownership records of your most frequent location (where you spend roughly half your life) and forever associate your real name and secret ID?
When combined with other websites that you may sign into, how hard would it be to discover your other usernames, some already tied to your real name, and tie them all together into a single linked identity and activity file you never get to see?
Not at all. It’s already common.
There is an actual body of research behind anonymizing people and GPS coordinates that Verizon seems to be [negligently, IMO] ignoring. There are viable techniques to present targeted ads (based only on what Verizon knows about you) without leaking that data to any third party.
Any claims of effective anonymity after sharing this kind of data with third parties are IMO false and misleading. They’re counting on everyone being dumb. Don’t be.
This post by Peter Berkman gets closest to the meat of our collective concerns over the Oculus sale to Facebook. John Carmack even responded in the comments, while Palmer took to reddit. Raph Koster and Blair McIntyre do pretty good analyses in their own rights, putting the pieces together.
Disclaimer: I know many of the people mentioned here and consider them friends. I’m sticking to only what is publicly revealed information. And I am of course happy for these individuals who are getting to work on dream projects and making lots of money as a side benefit.
Here are some collected facts that are out in the open:
1. Industry Legends John Carmack and Michael Abrash have each talked publicly about wanting to build the Metaverse (of Snow Crash fame). I totally get what’s in this deal for them: build it with the right tech, the right people and at Facebook scale. Got to love the opportunity to realize that dream.
2. Cory Ondrejka (kick-ass VP of Mobile at Facebook) was formerly CTO of LindenLab/Second Life. He’s built a slightly less scalable version of the Metaverse already. Notice a “Mark, Chris and Cory” named in Palmer’s blog post. I have no doubt they’re completely sincere about wanting to see VR succeed in its own right.
3. Peter is also right in assuming that Oculus will add real gaze tracking. Palmer hints (skip to just before 19:00) about it here. Oculus is not alone, I expect. While there are many patents (of various quality) on this, there are still many viable ways to determine a user’s gaze, and many reasons to do so. Facebook has even more on its wishlist, I bet, beyond the basics of providing better visuals or more Natural UI.
4. Companies like Tobii make their living today by (effectively) enabling mind-reading via a user’s gaze. Your supermarket shelves were probably arranged with this kind of tech strapped to willing users. Your favorite website’s layout was probably tested using this tech in a lab. Just imagine the power of knowing what everyone is looking at and why, of being able to read subtle emotion in the face, including those micro-expressions that always reveal the truth.
5. Blair is also right that Zuckerberg’s AR long-term vision isn’t a typo. For all the arguments about VR vs. AR vs. camera-based vs. see-through being better for this or that, the bottom line is that VR sucks for mobile. That’s based purely on the old “wall meets face” principle, if not the geek factor of walking around talking to people with a brick over your eyes.
Facebook needs to win on mobile. Yes, it’ll take a few years just to get consumer-friendly seated VR right. But when Oculus eventually gets their glasses down to sunglass size and adds forward-facing cameras that mix real and virtual, then watch out[side]. Here’s the key part of Zuckerberg’s quote again:
But this is just the start. After games, we’re going to make Oculus a platform for many other experiences. Imagine enjoying a courtside seat at a game, studying in a classroom of students and teachers all over the world or consulting with a doctor face-to-face — just by putting on goggles in your home.
This is really a new communication platform. By feeling truly present, you can share unbounded spaces and experiences with the people in your life. Imagine sharing not just moments with your friends online, but entire experiences and adventures.These are just some of the potential uses. By working with developers and partners across the industry, together we can build many more.
One day, we believe this kind of immersive, augmented reality will become a part of daily life for billions of people. [emphasis mine]
6. I’ve heard Oculus folks also talk about telepresence as one of the killer apps for VR too, esp. when one can solve the “eye gaze” problem. You’ve probably experienced that in crappy modern video chat, which is generally how I’d imagine women feel when you stare at their chests. Interestingly, Cory’s old boss from Linden, Philip Rosedale, has announced he’s working on the eye gaze problem in his new startup too, and using some cool software called FaceShift to get a jump on the harder problems there.
[For full disclosure, I did some work on avatar-based telepresence as well, but I wasn’t entirely satisfied with the result. FWIW, I found a better approach, which works with no hardware sitting on your face.]
7. Finally, Facebook makes its money by selling its users. Let’s not be coy. It’s a lot of money and a lot of users. To be fair, is it really any worse than how NBC offers up its users to advertisers (since before I was born)? Well, no and yes. It’s the same “ad, ad” world as before. We learn by age four to distrust the loud man on the TV. But the key difference is that NBC can’t see you, can’t really know your thoughts, except via gross statistics (think Neilsen ratings). Facebook really wants to know you, individually, and forever.
Now, companies like Facebook and Google provide immensely desirable services for free too, more so than NBC IMO. The problem lies in the concept of “informed consent.” Both tech companies still suck at the human stuff IMO, esp. in terms of giving users clear information up front to make that decision wisely, and then giving full control of their data after. It’s still mostly one-sided today, and it has to change.
Ideally, they’d start to emulate companies who’ve adopted an obsessive customer focus. It’s not about what people will tolerate, but what they really need. Earning our trust can’t be an afterthought or a win by attrition.
[for a positive comparison, TV’s “Neilsen families” give truly informed consent and the company promises not to market to them as well.]
So we come back to Peter’s insightful post. The problem with the Oculus sale is not that we all supported Palmer’s fortuitous kickstarter and got no share of the spoils. I got the product I ordered, still sitting in its box, alas. It’s not that he and Brendan Iribe “sold out” the indie gaming dream for big bucks. There are plenty of reasons to seek this level of protection and financial support in the light of Sony (and others) gearing up for a big fight. Gamers should be happy to have more viable options to choose from at the lowest prices.
Hell, I joined the so-called “evil empire” in 2008 to try to use their massive scale to do some good in the world, so who am I to judge? (jury is still out on my contributions, fwiw)
No. The heart of the problem is that VR is the most powerful means ever invented to pipe external ideas into your internal world. It’s pushing remote-controlled information almost directly into your brain via every sense possible. Consider that people believe a lot of crap they can’t even see. You’ll believe this crap on so many levels. That’s why it’s called virtual reality, ok? It’s as close to real as it gets, while still being entirely artificial.
The Facebook purchase highlights the reverse of that flow as well, reminding us that VR may become one of the best ways to pipe your internal world out — via data mining, classification and onto untold dissemination. That’s actually one of the reasons I got into the field, 20 years ago: to more easily tell stories that were rattling around my brain that would have taken millions of dollars and hundreds of people to produce as movies or games. But it has a dark side too, like when the information is sucked out of us without our informed consent or control.
Just think what the headlines would have said last week if Facebook had instead bought a brain-sensing startup like Emotiv, or invested in fMRI brain scanning tech to extract your thoughts. See what I’m talking about? VR doesn’t work quite the same way as those, but it gets to the same place in the end. The protective walls between you and the world come down in favor of more bandwidth in and out.
So closing that loop, even crafting individually designated virtual worlds using all of your private information, Facebook will own the most potent means available, short of mind-control drugs, to read and write to your private inner world, your thoughts, your actions, your dreams. It can free you, or it can enslave you.
What they do with it is entirely up to them (plus certain market and “other” forces). And if that doesn’t scare you, at least a little, then you may already be sold.
Most head mounted VR gear brings me back to my teenage years, where my orthodontist tried to make me wear a night brace to straighten my teeth.
It works while you sleep, he said.
You try sleeping with your head in a vice, I said.
He didn’t care. He got paid $40/month regardless of how long it took. Needless to say, I soon got a new orthodontist. And I’ve kept trying on various VR gear too.
I originally favored the CAVE projection kind of display and built a few variations of my own, including a six-sided one at Disney. The main benefit is zero latency (ignoring stereo parallax changes) — in other words, the image is already there when you turn your head. No blurriness. The main downside, of course, is who has room or money for an 8′ cube in their living room. Not practical until we get digital wallpaper or big flexible roll-up screens.
But even still, I happily bought into the Oculus Rift’s kickstarter, eager to try again. I love seeing people so enthusiastic about this stuff, especially new blood.
Though I’ve personally used the Rift for many minutes at a time, my own purchased dev kit is still sitting in its box, alas, waiting for me to find time to build something useful. The head-tracking latency was actually very good, but the original display felt much like the world was made of LiteBrite. If you’ve never tried it, here’s a good oculus rift simulator to try.
I just pre-ordered the 2nd gen dev kit too, which fixes much of the resolution issue, and I’m sure comes in an even nicer box.
I’m hopeful that Carmack can solve some of the rendering latency issues that fast OLED displays alone can’t. I did some research in this area too, fwiw. There’s a lot that can still be done to wring the delays out of various pipelines.
Some friends and I also got to try out the new Sony Morpheus HMD at GDC this week. We had to get in line the moment the expo doors opened, just to get a ticket to stand in line to wait to try. But it was worth it, I keep telling myself.
The resolution was impressive. The persistence of their LCD displays was not as good as the Rift’s. Now, I can’t be sure what they’re using inside, but I would have thought they’d throw some 4k SXRD panels in there, just like they use in their nicest projectors.
I thought those were akin to DLP in terms of super-fast switching time, but I’m not so sure anymore. Maybe there isn’t room for front-reflection in the optical path. I can say that the LCD-like images we saw seemed to be over-driven and washed out a bit, mostly suffering from slow switching times. Brightness was great, but black blacks were in short supply.
In any event, no one got nauseated, which is a small victory for those of us who can’t watch the Blair Witch Project without dramamine. The Rift also does well on that front. But in both cases, using a simple laptop trackpad or arrow keys to navigate puts me back on the vomit comet.
A nice omni-directional treadmill might do the trick. Someone had one of those on display too, but I didn’t get to try it — seems to need special slippery shoes. But if we’re going through the trouble of a 4′ treadmill, why not just go back to using CAVEs? If it’s just a matter of integrating your furniture, my friends in MSR solved that nicely.
For what it’s worth, I still have my money on real see-through AR displays as the ultimate winner. Let me walk in the real world, augmented with new content. Yes, indeed.
It’s not the $19B price tag of Facebook’s WhatsApp acquisition that’s most amazing to me. That’s just math (lots of math). More users * more engagement * a wealthy/desperate suitor = bigger deals.
It’s not most amazing to me that the deal works out to over $250M per employee, if evenly split. For comparison, Google seems to be worth about $8.5M per employee, last I checked. Apple reportedly pulls in a whopping $2.3M yearly per employee in revenues alone. (average Apple employees, how much of that do you get paid again?).
WhatsApp just did more and faster with fewer people. That’s what makes a great team. I’m sure everyone was amazing. Dysfunctional teams or people barely earn back their salaries, if they survive at all.
The motivation for the deal is not amazing at all, since Facebook desperately needs to connect the very people who don’t need Facebook to connect, i.e., the people you see every day. It seemed almost inevitable, given the trends.
No. What is most amazing to me is the powerful principle behind this simple note at right — one of the core principles that apparently made WhatsApp so popular with hundreds of millions of users. I found this via the Sequoia blog post about the sale, and I honestly had no idea.
Jan keeps a note from Brian taped to his desk that reads “No Ads! No Games! No Gimmicks!” It serves as a daily reminder of their commitment to stay focused on building a pure messaging experience.
It’s most amazing to me, and most inspiring, because there are so many professional CEOs and advisors out there who try to convince their startups that they need to collect and hoard as much user data as possible, then sell it surreptitiously, while also pushing ads and wringing every last monetizable cent with games and gimmics to keep people addicted and virally engaged.
Those same fine folks somehow duck out (or get fired) when the users finally complain, defect and disappear. And they apparently never learn from their mistakes. But they do come back, with the same tired story again and again.
WhatsApp proved them wrong and proved it 19 billion times over.
Build value for users. Give them what they want and need, every day. That’s the recipe for success. This kind of success is something I can truly appreciate and admire.
P.S. I’d like to think that this is the real reason Facebook bought them, given FB’s reliance on those same tired ads, games, etc… Maybe it wasn’t for more European and emerging market penetration. Maybe it wasn’t merely to disarm a growing competitor. Maybe it’s for Facebook to become more like them? Maybe that’s why they put Jan on the board. If so, good for them.
I’ve heard that credit card companies know when you’re cheating or about to get divorced. I’d imagine things like flowers, gifts and hotels would be a tip off, but general expenses probably approach double once a couple is separated.
This data about facebook posts makes perfect sense too. People using FB will post more flirtatiously until the relationship starts. But after, why post so much? Even in real life, it’s rare for couples to make strong public displays of affection once they’re together. If they do, it may come more from insecurity about the relationship, or insensitivity to others’ feelings, than from some unparalleled eternal flame.
This highlights one of Facebook’s core challenges — how to capture sentiments shared between people who spend a lot of time together.
This idea reportedly comes from a competition that Meta (Space Glasses) is holding. The idea is to project a holographic image of your phone in space (using said glasses) and let you virtually interact with it, instead of taking the phone out of your pocket to do exactly the same.
Why is it a brilliant idea?
It’s simple. People get accustomed to their phone’s UIs. Projecting the phone holographically requires not a single new thought and changes nothing about the core experience. Well, it does lose out on touching that sleek and sexy touch-screen, feeling the nicely balanced weight of the phone in your hand, and of course key sensors like accelerometers (to a degree) and cameras (at all) to certain phone experiences.
So I guess that means you couldn’t run old augmented reality apps on your holographic phone for a recursive experience. Oh well. There goes a nice photo op.
Why is this a stupid idea?
Your head mounted device can [eventually] paint pixels anywhere you look. It can detect touch anywhere it can see your hands. Why would we limit ourselves to drawing a 4″ screen when we have an infinitely large screen on our head?
It’s a lot like saying, “Hey, we got used to small CRT TVs so let’s draw a small TV inside our brand new 60″ flat screen TV so people don’t have to learn something new.”
Interfaces for AR will run the gamut from holographic virtual actors who become your daily assistant, to making every physical surface in the world potentially interactive by touch, sight and sound. Why would we limit ourselves to UI mechanisms that were designed around the limits of small screens and touch?
Just for the experience of not having to take our phone out of our pocket? Are we really that lazy? If so, ask yourself how much you’d be willing to pay to use your phone without taking it out of your pocket. I’d pay maybe $1.
This really comes down to a core question about AR. Is it about being the ultimate hands-free device, principally meant to deliver us from holding our phone in our hands or up to our faces? Or is it about re-imagining the analog world with new digital layer(s) of content on top?
I can see an app like this being very popular, at least in the way the fart app is popular. That’s only because people’s imaginations are presently too limited. They just haven’t seen the best ideas yet.
On the other hand, it’s turning out that the most popular interface for your new 60″ flat-screen TV with billions of streaming video options is not some new fancy XBox-like natural UI, but rather just your phone.
So what do I know? People may ultimately find ‘stupid’ brilliant.
The idea that most man-made objects can be represented with sweep surfaces (cylinders, tubes, squares, etc..) isn’t that new. Second Life primitives used exactly the same principle, with some interesting extensions for cuts, twists, tapers and so on.
But selecting photographic imagery based on implicit primitives, in-painting (hallucinating) the background and unseen object views, and (occasional) relighting of the object is all extremely clever and very useful. Combine this and a system that can relight virtual objects based on scene shadows and you have a paint program that can revise reality, at least virtually, but in a way that would fool almost anyone.
The end-goal of all this work is something I used to call “parametric 3D video” — which roughly means we take one or more 2D video streams, split out the objects, backgrounds, people into separate and fully adjustable pieces, send them as 3D content vs. pixels, and then re-synthesize the result from any angle at the receiving end, along with any changes you want to make.
3D (color + depth) video capture makes the problem much easier. Techniques like this paper are still needed to finish the job, but they can be much more automatic in terms of finding and cutting objects.
Here’s a great* Verge article on the AWE 2013 conference that wrapped up last week. I had the honor of speaking, along with a number of my co-workers. Although, as Tish points out, we’re not actually doing AR at Syntertainment, we’re very passionate about the field as one of several key enabling technologies.
* yes, of course mentioning me positively supports my opinion of your article, even if you get my name slightly wrong.
I was checking the Oculus VR site today to see when my dev kit will ship (April, apparently) and I noticed this interesting job posting:
- Design and implement techniques for optical flow and structure from motion and camera sensors
Relevant computer vision research (3d optical flow, structure from motion, feature tracking, SLAM)
What this means to me is that Oculus may be trying to solve AR as well as VR.
Now, it’s possible they’re just trying to use optical flow to create an absolute reference frame for the rendered VR scene. The tiny gyros that track your head rotation tend to drift over time and corresponding magnetometers (basically, compasses) aren’t always reliable enough to help. It’s also desirable to know your absolute lateral translation at any given time, which onboard accelerometers can only guess at. So some amount of computer vision makes sense for VR, implying there will be cameras mounted on the VR glasses, if not already. (I’d certainly encourage them to put cameras facing your eyes too).
However, “structure from motion” and SLAM would be better suited to 3D tracking such that a video camera could overlay virtual 3D objects on video of real world, like many of the AR demos you see on phones and tablets today. Done well, it gives much more precise 3D transformations than VR might need, and SLAM is quite expensive to compute traditionally.
In fact, if you were doing VR glasses with cameras, tracking your arms and legs would be the first problem I’d solve. Getting 3D input right is very important. Oculus seems to be thinking of that as well. (see this other job posting).
So let’s assume AR uses for this SLAM work and see where that leads. The idea is you’d put a camera (or two) on the goggles to capture the real world at a similar field-of-view to the VR display, pipe that video into your 3D renderer with some depth-per-pixel, and you’d theoretically have a better AR solution than Google Glass or similar HUD approaches.
On the plus side, pixels captured from the real world can be correctly occluded by pixels in the virtual scene. That’s very hard to do with see-through AR, which is plagued by the ghostly transparent images we’ve come to call “holograms,” for their ethereal qualities. They’re transparent because there’s currently no solution on the market to stop the natural light that comes from the real world other than to darken the glasses everywhere, so they go from AR to VR anyway…
Of course, latency is the main issue with the camera-based-AR approach. If you turn your head with see-through AR, the real world is visible with zero lag, and the virtual part can be dialed down whenever it would be nauseating. With camera-based AR, you typically have a few frames of latency added to the queue, which can induce nausea and disorientation, especially if the goal is to just walk around with these and not hit or be hit by “things.” Latency has to be extremely low, like 4 ms.
There are tricks to avoid this. Cheaper cameras use what’s called a “rolling shutter” which means they’re effectively only capturing one line of pixels at a time, while that line sweeps across the image at some high rate. This is what causes those funny skewing artifacts when you move your cell phone while taking pictures or video — the picture wasn’t all taken at one point in time like an old film camera or high-end capture device.
The challenge is how to couple a rolling shutter to a “beam-chasing” rendering algorithm, which is the same idea applied in reverse — to the visual output instead input. If you do that right (and I’m sure there is a way), then the distance between the camera’s “scan line” and the currently rendered “raster line” is your actual latency, which would be measured in low milliseconds instead of full frames. Cool stuff.
But that’s not a full solution — the 99% of the scan lines we rendered in the past (this “frame”) would be stuck in their old positions while our heads turn. But at least the newest rendered pixels would be more correct, right?
A full solution includes a very low-latency image capture of the physical scene, 3D pose estimation (SLAM or similar) while a high-fidelity render of the virtual scene is conducted, and then a very high-frame-rate (120-240hz) continuous re-rendering of that scene is done based on the latest head rotation and translation measurements from the gyros and accelerometers.
This is indeed a kind of magic. Could Oculus do it? They’ve seemed to raise enough money. So best of luck to them.