Twitter Search as a Government case study

In addition to periodic think-pieces here at Shepherd’s Pi, I also contribute a monthly online column over at SIGNAL Magazine on topics relating to intelligence. This month I keyed off a recent discussion I had onstage at the 2015 AFCEA Spring Intelligence Symposium with Elon Musk, particularly a colloquy we had on implications of the emerging cleavage (post-Edward Snowden) between Silicon Valley technology companies and their erstwhile innovation partners, U.S. intelligence agencies.

Tearing the Roof off a 2-Terabyte House

I was home last night playing with the new Kinect, integrating it with Twitter, Facebook, and Zune. Particularly because of the last service, I was glad that I got the Xbox 360 model with the 250-gigabyte (gb) hard disk drive. It holds a lot more music, or photos, and of course primarily games and game data.

So we wind up with goofy scenes like my wife zooming along yesterday in Kinect Adventures’ River Rush – not only my photo (right) but in-game photos taken by the Kinect Sensor, sitting there below the TV monitor.

Later as I was waving my hands at the TV screen, swiping magically through the air to sweep through Zune’s albums and songs as if pawing through a shelf of actual LP’s, I absent-mindedly started totting up the data-storage capacity of devices and drives in my household.  Here’s a rough accounting:

• One Zune music-player, 120gb;
• 2 old iPods 30gb + 80gb;
• an iPad 3G at 16gb;
• one HP netbook 160gb;
• an aging iMac G5 with 160gb;
• three Windows laptops of 60gb, 150gb, and 250gb;
• a DirecTV DVR with a 360gb disk;
• a single Seagate 750gb external HDD;
• a few 1gb, 2gb, and a single 32gb SD cards for cameras;
• a handful of 2gb, 4gb, and one 16gb USB flash drives;
• and most recently a 250gb Xbox 360, for Kinect. 

All told, I’d estimate that my household data storage capacity totals 2.5 terabytes. A terabyte, you’ll recall, is 10¹² bytes, or 1,000,000,000,000 (1 trillion) bytes, or alternately a thousand gigabytes.

Bing vs Google, the quiet semantic war

On Wednesday night I had dinner at a burger joint with four old friends; two work in the intelligence community today on top-secret programs, and two others are technologists in the private sector who have done IC work for years. The five of us share a particular interest besides good burgers: semantic technology.

Oh, we talked about mobile phones (iPhones were whipped out as was my Windows Phone, and apps debated) and cloud storage (they were stunned that Microsoft gives 25 gigabytes of free cloud storage with free Skydrive accounts, compared to the puny 2 gig they’d been using on DropBox).

But we kept returning to semantic web discussions, semantic approaches, semantic software. One of these guys goes back to the DAML days of DARPA fame, the guys on the government side are using semantic software operationally, and we all are firm believers in Our Glorious Semantic Future.

A Technical Computing revolution

Last week I enjoyed hosting a visit in Redmond from Chris Kemp, NASA’s new Chief Technology Officer for information technology. Our discussions were with folks from the Windows Azure cloud computing team, the high-performance computing and large-data folks, and our Extreme Computing Group. I smiled when Chris said he was a fan of the book Total Recall: How the E-Memory Revolution Will Change Everything, written by Microsoft’s Gordon Bell and colleague Jim Gemmell. (I wrote about their research projects in an earlier post, Total Recall for Public Servants.)

Using the body in new virtual ways

This is CHI 2010 week, the Association for Computing Machinery’s Conference on Human Factors in Computing Systems in Atlanta. Top researchers in human-computer-interaction (HCI) are together April 10-15 for presentations, panels, exhibits, and discussions. Partly because of our intense interest in using new levels of computational power to develop great new Natural User Interfaces (NUI), Microsoft Research is well represented at CHI 2010 as pointed out in an MSR note on the conference:

This year, 38 technical papers submitted by Microsoft Research were accepted by the conference, representing 10 percent of the papers accepted. Three of the Microsoft Research papers, covering vastly different topics, won Best Paper awards, and seven others received Best Paper nominations.

Follow the USS Carl Vinson to Haiti

As I write on Wednesday afternoon (EST), the scenes of chaos, death, and destruction in Haiti are only now beginning to be visible to the outside world through media. As horrific and heart-rending as those scenes are, they serve a purpose in letting other nations comprehend the magnitude of the crisis and the urgency required in lending direct aid. The U.S. military is uniquely positioned to contribute.

Flight Deck of the USS Carl Vinson

What a difference a day makes: barely 24 hours ago, several hours before the earthquake struck, the Nimitz-class supercarrier USS Carl Vinson (CVN 70) was cranking up its nuclear engines and setting a peaceful course out of Hampton Roads at the base of the Chesapeake Bay, Virginia. At long last after completing a complex overhaul and new sea trials, she was heading south, to make the South America turn and return to homeport in San Diego as part of the Pacific Fleet.

To fix intelligence analysis you have to decide what’s broken

“More and more, Xmas Day failure looks to be wheat v. chaff issue, not info sharing issue.” – Marc Ambinder, politics editor for The Atlantic, on Twitter last night.

Marc Ambinder, a casual friend and solid reporter, has boiled down two likely avenues of intelligence “failure” relevant to the case of Umar Farouk Abdulmutallab and his attempted Christmas Day bombing on Northwest Airlines Flight 253.  In his telling, they’re apparently binary – one is true, not the other, at least for this case.

The two areas were originally signalled by President Obama in his remarks on Tuesday, when he discussed the preliminary findings of “a review of our terrorist watch list system …  so we can find out what went wrong, fix it and prevent future attacks.” 

Let’s examine these two areas of failure briefly – and what can and should be done to address them.

Cruise Missiles and Yemeni Neighborhoods

“The US is planning retaliatory strikes in Yemen against al-Qaida over its attempt to blow up a transatlantic flight on Christmas Day.  American officials […] warn that finding those responsible is unlikely to be swift and say that identifying other ‘high-value’ al-Qaida targets for retaliatory attack would also be a priority.” – The Guardian (U.K.), 12/30/2009

As U.S. officials are quoted mulling cruise-missile strikes on Yemen, it should be noted that the Yemeni government, such as it is, has already been fighting the hard slog – on the ground, rooting out Al Qaeda in Yemeni neighborhoods and villages.

Below is an interesting 7 minutes of film fresh from Yemen,  just posted to youTube today:  a moment-by-moment video documenting the Arhab raid by the Yemeni Counter-Terrorist Unit (CTU) on an Al-Qaeda-in-the-Arabian-Peninsula (AQAP) safe-house, on 17 December, the same day as a coordinated U.S. cruise missile attack on another site.

Total Recall for Public Servants

MyLifeBits is a Microsoft Research project led by the legendary Gordon Bell, designed to put “all of his atom- and electron-based bits in his local Cyberspace….MyLifeBits includes everything he has accumulated, written, photographed, presented, and owns (e.g. CDs).” 

SenseCam - Click to enlarge

Among other technical means, Bell uses the SenseCam, a remarkable prototype from Microsoft Research.  It’s a nifty little wearable device that combines high-capacity memory, a fisheye lens passively capturing 3,000 images a day, along with an infrared sensor, temperature sensor, light sensor, accelerometer, and USB interface. My group has played with SenseCam a bit, and shared it with quite a few interested government parties and partners. More info on SenseCam here, and more on its parent Sensors and Devices Group in MSR.  

43 Gigabytes of Mobile Data per Day

Here’s a nifty infographic, created by Online Education with several striking statistics about “an average day on the Internet” and the volume of data involved in mobile talk and data, Twitter, blogs, wikis, email, news sites and the like. The numbers are staggering!