Stop being so touchy

Remember Tom Cruise’s virtual 3D in Minority Report? Wouldn’t it be cool if, instead of an iPhone-like touch screen, we could manipulate data and 3D images on screens by simply moving our hands – interacting virtually without touching a keyboard or screen?

Fact: The market for touch-screen devices has been expected to grow at a dramatic rate — one industry group estimated earlier this year that “the touch-panel market is on track to grow to $3.3 billion and 660 million units by 2015,” according to DisplaySearch’s Touch Panel Market Analysis report.  Touch-enabled mobile devices like the iPhone or my Samsung i760 account for only 34 percent of touch-panel shipments.

Analysis: Everybody liked what Tom Cruise could do in “Minority Report,” and for a few years after the movie came out, a lot of people in the national security community wondered, “when will we able to use technology like that?” Pentagon generals asked tech companies to build it; Pentagon critics assumed it was already being used deep within Ft. Meade 🙂 And technologists inside and outside the Beltway just wanted to explore the advantages of new human-computer interfaces. As one blogger wrote in 2006, “There is work going on in this area already, from the U.S. military and others. The rest of us, for now, can only dream.”

Well, Microsoft Office Labs has just released “Touchless,” a webcam-driven multi-touch interface SDK that enables “touch without touching,” producing some of the same kind of magic as seen in the movie. 


The video features a quick demo by Mike Wasserman, a software development engineer at Microsoft, who only graduated from Columbia University a year ago, and began working on the Touchless project as an undergrad and finished it at Microsoft in his “spare time!”

Using the SDK lets developers offer users “a new and cheap way of experiencing multi-touch capabilities, without the need of expensive hardware or software. All the user needs is a camera,” to track the multi-colored objects as defined by the developer.  Just about any webcam will work.

Oh, and surprise: the Touchless SDK is open source.  It is released free and open-source to the world under the Microsoft Public License (Ms-PL) on CodePlex.  (Also, we announced further open-source support today with funding for advanced Silverlight development capabilities with Eclipse.) 

From the Touchless release notes:

What is the Touchless SDK?
The Touchless SDK is an open source SDK for .NET applications. It enables developers to create multi-touch based applications using a webcam for input. Color based markers defined by the user are tracked and their information is published through events to clients of the SDK. In a nutshell, the Touchless SDK enables touch without touching.

What is the Touchless Demo?
The Touchless Demo is an open source application that anyone with a webcam can use to experience multi-touch, no geekiness required. The demo was created using the Touchless SDK and Windows Forms with C#. There are 4 fun demos: Snake – where you control a snake with a marker, Defender – up to 4 player version of a pong-like game, Map – where you can rotate, zoom, and move a map using 2 markers, and Draw – the marker is used to guess what….  draw!


Mike demonstrated Touchless at a recent Office Labs’ Productivity Science Fair where it was voted by attendees as “most interesting project.” Office Labs has a number of interesting projects around “future knowledge work” scenarios, and you can try several others at the Office Labs Projects page.

If you wind up using the SDK, I’d love to hear what use you make of it!

Email this post to a friend

AddThis Social Bookmark Button

4 Responses

  1. There’s been some really fun gesture-driven user interface development around the Nintendo Wii, of course, including multitouch interfaces built out of that infrared sensor + accelerometer system.

    Also worth noting because it’s occupying the ears of some friends of mine is something called RjDj, an iPhone music system that uses the motion sensor on that phone as a controller for various remixes of ambient sounds.

    Of course this reminds me of pinball machines where if you knew just when and how to bump the machine that gave you a new “user interface” to that game.


  2. Ed, you’re right – Johnny Chung Lee (of Carnegie Mellon but now of Microsoft) has done some really interesting Wiimote stuff,

    … and I just saw a Twitter mention of RjDj this morning, didn’t know what it was… There’s a case where a Zero-style “filtering” (vs following) would not have worked in alerting me to something new, I would have had to wait until you told me 🙂 So there’s something to be said still for serendipitous discovery through omnivorous attention.


  3. oh I still have omnivorous attention, just not trying to be omniscient via twitter any more 🙂

    i picked up on rjdj via a reference on techcrunch, which I posted to flickr, which was commented on by a friend there & thus gave me a reason to look at and listen to again. sometimes you have to hear something three times for it to sink in.


  4. […] A perceptive questioner during the Q&A was Tim Gabel, VP of the Survey and Computing Sciences unit at RTI in Research Triangle Park, which is prominent in the world of government and private-sector research. Tim asked whether we were approaching the science-fiction capabilities that Tom Cruise used in the movie Minority Report.   As I’ve done before, I cited individual capabilities that we in Microsoft Research and other groups have done, like Surface multitouch computing, and Photosynth, and I showed a quick video of the “Touchless” fingers-in-the-air computing I wrote about last year. […]


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: