Insider’s Guide to the New Holographic Computing

In my seven happy years at Microsoft before leaving a couple of months ago, I was never happier than when I was involved in a cool “secret project.”

Last year my team and I contributed for many months on a revolutionary secret project – Holographic Computing – which is being revealed today at Microsoft headquarters.  I’ve been blogging for years about a variety of research efforts which additively culminated in today’s announcements: HoloLens, HoloStudio for 3D holographic building, and a series of apps (e.g. HoloSkype, HoloMinecraft) for this new platform on Windows 10.

For my readers in government, or who care about the government they pay for, PAY CLOSE ATTENTION.

It’s real. I’ve worn it, used it, designed 3D models with it, explored the real surface of Mars, played and laughed and marveled with it. This isn’t Einstein’s “spooky action at a distance.” Everything in this video works today:

These new inventions represent a major new step-change in the technology industry. That’s not hyperbole. The approach offers the best benefit of any technology: empowering people simply through complexity, and by extension a way to deliver new & unexpected capabilities to meet government requirements.

Holographic computing, in all the forms it will take, is comparable to the Personal Computing revolution of the 1980s (which democratized computing), the Web revolution of the ’90s (which universalized computing), and the Mobility revolution of the past eight years, which is still uprooting the world from its foundation.

One important point I care deeply about: Government missed each of those three revolutions. By and large, government agencies at all levels were late or slow (or glacial) to recognize and adopt those revolutionary capabilities. That miss was understandable in the developing world and yet indefensible in the United States, particularly at the federal level.

I worked at the Pentagon in the summer of 1985, having left my own state-of-the-art PC at home at Stanford University, but my assigned “analytical tool” was a typewriter. In the early 2000s, I worked at an intelligence agency trying to fight a war against global terror networks when most analysts weren’t allowed to use the World Wide Web at work. Even today, government agencies are lagging well behind in deploying modern smartphones and tablets for their yearning-to-be-mobile workforce.

This laggard behavior must change. Government can’t afford (for the sake of the citizens it serves) to fall behind again, and  understanding how to adapt with the holographic revolution is a great place to start, for local, national, and transnational agencies.

Now some background…

Programmatic Context for HoloLens

An enduring aspect of working on new technologies is stealthiness. It isn’t always the right approach – sometimes open collaboration beyond company borders has superior value to a quiet insular team. I learned the distinction well when I was in government (Intellipedia and A-Space were among our results) and in the startup culture in Silicon Valley before that.

But stealth has an electric appeal, the spark of conspiracy. At Microsoft and some other companies, the terminology is “Tented” – you have no clue about the work if you’re outside the tent, enforced as rigorously as in a SCIF.

Last March my Microsoft Institute team at Microsoft was quietly invited to add our efforts to a startling tented project in Redmond – one which was already gaining steam based on its revolutionary promise and technical wizardry, but which would require extraordinary stealth in development, for a variety of reasons.  I won’t share anything proprietary of course, but will say that our secrecy was Apple-esque, to use a Valley term of high praise.

That project is being announced to the world today, as HoloLens. I couldn’t be prouder of my (erstwhile) colleagues at Microsoft who are launching a revolutionary platform.  The praise is already rolling in.  WIRED‘s story is “Our Exclusive Hands-On With Microsoft’s Unbelievable New Holographic Goggles” while TechCrunch quickly assesses “Augmented reality has had some false starts on mobile, but in this context, it seems more viable, and thus more credible than it ever has before.”)

Next, let’s look at some background on the technical area which HoloLens now stands astride like a colossus among the Oculus Rift and Google Glass lesser-rans. Then below I’ll sketch some initial observations on the relevance for government uses and the world at large.

Technology Context: Ambient Computing

I’ve been writing about virtual reality and augmented reality (the VR/AR split) for a decade, first inside government and over the past seven years on this blog.  The term I prefer for the overall approach is “Ambient Computing” – combining advanced projection, immersion, machine vision, and environmental sensing.

Ambient computing devices are embedded all around in the environment, are addressable or usable via traditional human senses like sight, hearing, and touch/gestures, and can understand people’s intent, and even operate on their behalf.

Previous ShepherdsPi posts on Virtual Reality/Augmented Reality:

2008 War is Virtual Hell on emergent defense thinking about virtual reality

2008 Stretching collaboration with Embodied Social Proxies on robotics and VR

2009 Immersed in Augmented Reality, with the concept of “Instrumenting the World” as an important foundation for what is now called the Internet of Things.

2010 The promise of mobile augmented reality

2010 Air Everything

2011 Kinecting Communities

By 2010 I could look ahead (“Playing with virtual data in spatial reality“) and see clearly where we are heading based on trends:

We’re further along in this area than I thought we’d be five years ago, and I suspect we’ll be similarly surprised by 2015. In particular, there is great interest (both in and out of the government circles I travel in) in the “device-less” or environmental potential of new AR technologies. Not everyone will have a fancy smartphone on them at all times, or want to stare at a wall-monitor while also wearing glasses or holding a cellphone in front of them in order to access other planes of information. The really exciting premise of these new approaches is the fully immersive aspect of  “spatial AR,” and the promise of controlling a live 3D environment of realtime data.

That vision begins to become “virtually real” with today’s HoloLens announcement.

Competitive Context

I’ll leave to analysts, and to the holiday-market later this year and next, to judge where the competing technologies lie on the “hype-curve” of reality and utility.  I can list the efforts I’m playing closest attention to, and why:

Samsung’s Gear VR and Project Beyond: The Gear VR headset hasn’t lit expectations very brightly among analysts or the tech media, but it does now have alongside it the recently announced “Project Beyond,” a 360-degree panopticon camera module which is planned to capture a gigapixel of surrounding 3D footage every second, and stream that footage back to someone wearing a Gear VR headset, “essentially transporting them into that world.” Unlike HoloLens, it’s not a full computer.

Google Glass: The granddaddy of widely available AR experiments. Withdrawn from the public last week, not before inspiring a raft of venture-funded lookalikes which are also now also-rans. Google undoubtedly learned a great deal by dipping its giant toe into the virtual realm so enthusiastically with its Explorers program, but most of my friends who participated developed a “ho-hum” attitude about the device, which now gathers dust on shelves across the world.

Magic Leap: Google’s withdrawal of Glass can be seen in the context of the revelation that the search/advertising giant has now instead plowed in a large amount of cash to this start-up, followed by several A-list Silicon Valley VC funds. Magic Leap has now raised an astonishing $542 million in Series B funding – yes, that’s half a billion, with no product or launch date in sight, but a long list of developer openings on its website. (But don’t worry, the company just hired a novelist as its Chief Futurist.)

Oculus VR and the Rift (or its follow-ons): Oculus Rift has to be considered the leading rival to Microsoft’s HoloLens, so much so that Facebook acquired its parent startup company for an eye-opening $2 billion, ten months ago. Mark Zuckerberg at the time indicated patience and the long-view in his strategy, but industry watchers don’t expect a device release until late 2015 or 2016. And Rift, as of its descriptions to date, isn’t a full computing experience, merely a virtual-reality immersion.  There’s also no see-through aspect to its headset (unlike the visible real-world context of HoloLens), which has led to widely-reported nausea problems among Rift prototype users.

These all feel a bit laggard now, particularly because the companies involved (with the exception of Google) don’t have the experience of Microsoft in launching global computing platforms on which communities of developers can make magic.  Most importantly, none of these efforts are audacious enough to incorporate a full computing device (CPU, GPU, wirelessly connected) into a comfortably wearable device.

Bottom Line for Government…

Ambient computational resources are driving a new revolution, which the private sector is exploiting rapidly. That industrial and consumer revolution is in useful parallel with a virtuous cycle of ubiquitous sensing (Internet of Things) producing zettabytes of Big Data, being manipulated and mined by pioneering Machine Learning techniques for so-called “soft AI” (see IBM’s Irving Wladawsky-Berger in last week’s Wall Street Journal, “Soft Artificial Intelligence is Suddenly Everywhere“).

We humans, we of soft tissue, need all the help we can get to preside over those new and accelerating forces of technological change. The real magic is when our tools give us such powerful command in a simple and fun way. That is the promise of Holographic Computing, and HoloLens.

There are inevitably challenges. There’ll be devious uses of Holographic Computing, of course. Already we see the deceptive capabilities of regular screen-based “virtual reality,” and one can only imagine the perils of viewing these techniques from the wrong hands in full 3D immersion; check out these examples from the Emmy-winning special-video-effects (VFX) team behind HBO’s “Boardwalk Empire”:

We can’t allow government to waddle slowly behind, as real people live their lives increasingly affected by immersive technologies used for good or ill.

Governments exist to answer the needs of their citizens; government agencies and personnel should be using up-to-date tools capable of keeping up with what individual citizens are using, if only to avoid embarrassment and dinosauric irrelevance!

Holographic Computing offers government agencies real benefits:

  • Unique and insanely powerful mission applications; the company has been working on training, modeling & simulation, event forensics, gesture-driven immersive big-data visualization, distance learning, and you can easily imagine uses in widely varied fields like remote logistics management, geospatial analytics, telemedicine… Anything that uses personal computing software.
  • Government workforce and workplace transformed by collaboration transformation already evident in early applications like HoloSkype;
  • Awareness of and contemporaneous familiarity with the technological changes affecting society, through consumer and entertainment channels.

I’ll end with the newly-released overall video on Microsoft’s Holographic Computing; note the NASA/Jet Propulsion Lab scenes studying the surface of Mars first-hand. Note the 3D modeling from HoloStudio and its infinite shelf of parts. Note the HoloSkype example of real-time step-by-step advice on technical repair, from someone remote yet as near as by your side.

Imagine what you could do with HoloLens….

Let me know your ideas.

Virtual recipe stirs in Apple iPad, Microsoft Kinect

Who says Apple and Microsoft can’t work together?  They certainly do, at least when it involves the ingenuity of their users, the more inventive of whom use technologies from both companies (and others).

Here’s a neat example, “a just-for-fun experiment from the guys at Laan Labs” where they whip up a neat Augmented Reality recipe: take one iPad, one Kinect, and stir:

Some technical detail from the Brothers Laan, the engineers who did the work:

We used the String Augmented Reality SDK to display real-time 3d video+audio recorded from the Kinect. Libfreenect from project was used for recording the data coming from the Kinect. A textured mesh was created from the calibrated depth+rgb data for each frame and played back in real-time. A simple depth cutoff allowed us isolate the person in the video from the walls and other objects. Using the String SDK, we projected it back onto a printed image marker in the real world.” - source, Laan Labs blog.

As always, check out for the latest and greatest Kinect hacks – or more accurately now, the latest cool uses of the openly released free Kinect SDK, available here.

There are several quiet projects underway around the DC Beltway to make use of the SDK, testing non-commercial but government-relevant deployments – more detail and examples at the appropriate time. We will eventually release a commercial SDK with even more functionality and higher-level programming controls, which will directly benefit government early adopters.

In the meantime, I may report on some of the new advances being made by our research group on Computational User Experiences, who “apply expertise in machine learning, visualization, mobile computing, sensors and devices, and quantitative and qualitative evaluation techniques to improve the state of the art in physiological computing, healthcare, home technologies, computer-assisted creativity, and entertainment.” That’s a rich agenda, and the group is in the very forefront of defining how Natural User Interaction (NUI) will enhance our personal and professional lives….

Share this post on Twitter

Email this post to a friend

Air Everything

Like many people, I was very impressed by a video over the weekend of the Word Lens real-time translation app for iPhone.  It struck with a viral bang, and within a few days racked up over 2 million YouTube views. What particularly made me smile was digging backwards through the twitter stream of a key Word Lens developer whom I follow, John DeWeese, and finding this pearl of a tweet (right) from several months ago, as he was banging out the app out in my old stomping grounds of the San Francisco Bay Area. That’s a hacker mentality for you :)

But one thought I had in watching the video was, why do I need to be holding the little device in front of me, to get the benefit of its computational resources and display? I’ve seen the studies and predictions that “everything’s going mobile,” but I believe that’s taking too literally the device itself, the form-factor of a little handheld box of magic.

Continue reading

Pre-release hands-on look at WP7

I try to avoid straight-up promotion for my employer on my blog too often (if you didn’t know, it’s that scrappy West-Coast-based startup Microsoft). I like sharing R&D projects, but for consumer products I leave that stuff to others or third-parties.

But with the Apple iPhone 4 controversy going on, and Google’s Droid platform chugging away, frankly I was surprised to see a long story on übergeek tech blog Engadget today about our Windows Phone 7, with a great video of a hands-on demo:

I’m sharing it because I eagerly await the launch for my own use. The review itself is worth a read – it has in-depth looks at the UI and UX, social-networking aspects, music integration (full-on Zune experience), app marketplace, search, maps, innovative photos, wireless cloud syncing, high-def video, elegant online/offline email innovation, Office integration, Xbox Live, and other facets. Some features are good, some missing, some are awesome. Launch coming this fall…

Share this post on Twitter

Email this post to a friend

AddThis Social Bookmark Button

Bing vs Google, the quiet semantic war

On Wednesday night I had dinner at a burger joint with four old friends; two work in the intelligence community today on top-secret programs, and two others are technologists in the private sector who have done IC work for years. The five of us share a particular interest besides good burgers: semantic technology.

Oh, we talked about mobile phones (iPhones were whipped out as was my Windows Phone, and apps debated) and cloud storage (they were stunned that Microsoft gives 25 gigabytes of free cloud storage with free Skydrive accounts, compared to the puny 2 gig they’d been using on DropBox).

But we kept returning to semantic web discussions, semantic approaches, semantic software. One of these guys goes back to the DAML days of DARPA fame, the guys on the government side are using semantic software operationally, and we all are firm believers in Our Glorious Semantic Future.

Continue reading

Slate of the Union Day

Today is “Slate of the Union” day, when the two most charismatic individuals in recent American history go on stage and attempt to reclaim mantles as innovators. I’ll leave aside the fellow with lower poll numbers for now (President Obama). More eyes in the tech world will be watching as Steve Jobs makes his newest product announcement, the Apple tablet/Tabloid/iSlate thing iPad (it’s official).

Back in the late 1980s I worked for the legendary “Mayor of Silicon Valley” Tom McEnery (he was actually the mayor of San Jose), and we did many joint projects with Apple, particularly with CEO John Sculley, a great guy.

Continue reading

Para Bellum Web

Tim O'Reilly, Ray Ozzie

Tim O’Reilly created a bit of a stir last night in the tech world by writing a thoughtful essay entitled “The War for the Web.” He’ll be expanding on his thoughts in his keynote address today at the Web 2.0 Expo in New York. From the essay, here’s the core argument:

“[W]e’ve grown used to a world with one dominant search engine, one dominant online encyclopedia, one dominant online retailer, one dominant auction site, one dominant online classified site, and we’ve been readying ourselves for one dominant social network. But what happens when a company with one of these natural monopolies uses it to gain dominance in other, adjacent areas? I’ve been watching with a mixture of admiration and alarm as Google has taken their dominance in search and used it to take control of other, adjacent data-driven applications.

It could be that everyone will figure out how to play nicely with each other, and we’ll see a continuation of the interoperable web model we’ve enjoyed for the past two decades. But I’m betting that things are going to get ugly. We’re heading into a war for control of the web. And in the end, it’s more than that, it’s a war against the web as an interoperable platform. [emphasis added] Instead, we’re facing the prospect of Facebook as the platform, Apple as the platform, Google as the platform, Amazon as the platform, where big companies slug it out until one is king of the hill.

… P.S. One prediction: Microsoft will emerge as a champion of the open web platform, supporting interoperable web services from many independent players, much as IBM emerged as the leading enterprise backer of Linux.

Continue reading


Get every new post delivered to your Inbox.

Join 6,487 other followers

%d bloggers like this: