To fix intelligence analysis you have to decide what’s broken

“More and more, Xmas Day failure looks to be wheat v. chaff issue, not info sharing issue.” – Marc Ambinder, politics editor for The Atlantic, on Twitter last night.

Marc Ambinder, a casual friend and solid reporter, has boiled down two likely avenues of intelligence “failure” relevant to the case of Umar Farouk Abdulmutallab and his attempted Christmas Day bombing on Northwest Airlines Flight 253.  In his telling, they’re apparently binary – one is true, not the other, at least for this case.

The two areas were originally signalled by President Obama in his remarks on Tuesday, when he discussed the preliminary findings of “a review of our terrorist watch list system …  so we can find out what went wrong, fix it and prevent future attacks.” 

Let’s examine these two areas of failure briefly – and what can and should be done to address them.

43 Gigabytes of Mobile Data per Day

Here’s a nifty infographic, created by Online Education with several striking statistics about “an average day on the Internet” and the volume of data involved in mobile talk and data, Twitter, blogs, wikis, email, news sites and the like. The numbers are staggering!

Walled-Garden Wikis and Candlepower

Fact: Last night the U.S. Joint Forces Command (JFCOM) announced it has “moved its C2Pedia Registry to the unclassified network enabling more potential users to access and edit the site, hoping it will ultimately improve the quality of data.”  C2Pedia is a MediaWiki-driven online knowledge base of information about Command and Control (C2), with specific information about more than 200 C2 systems used across the Department of Defense and the armed services.

Analysis: The profusion of wikis in official government circles is an interesting expression of the value of social media for enterprise knowledge management, but for the most part inside agency or network firewalls, denying access to the public at large and therefore incorporating only the wisdom of “the inside crowd.” The State Department’s Diplopedia sits on their intranet (ironically called “OpenNet”), as the New York Times pointed out in a story a few weeks ago (“An Internal Wiki that’s Not Classified“), implying a distinction (without a difference to my mind) between Diplopedia and the IC’s Intellipedia, which has an unclassified version as well – but it also sits on a firewalled network!

Jimmy Wales, founder of Wikipedia, testified to Congress recently about the value of wikis and social media within enterprises, and pointed out the distinction between “within-the-agency” verticalized information sharing, a la Diplopedia, and horizontal sharing across organizations as exemplified by the IC’s Intellipedia, which as I mentioned has a firewalled unclassified version as well as its classified-network versions, all accessible from any of the intelligence community’s sixteen agencies and beyond.

Twitter Says LHC Fires Up, Earth Stubbornly Refuses to End

Fact: This morning’s press release from CERN: “Geneva, 10 September 2008. The first beam in the Large Hadron Collider at CERN was successfully steered around the full 27 kilometres of the world’s most powerful particle accelerator at 10h28 this morning. This historic event marks a key moment in the transition from over two decades of preparation to a new era of scientific discovery.” (LHC background here.)

Analysis: Early this morning, Twitter alerted me that the Earth was still spinning. Below I explain how.

A-SpaceX, Google, and Virtual Tuesday

Yesterday I had a “virtual world vibe” going.  At 5:30 a.m. when my dog Jack woke me up offering to take me for a walk, the first thing I noticed on my mobile was a series of tweets from Chris Rasmussen, NGA’s social software guru, posted the night before.  Twitter is interesting for a lot of reasons, but one is the ability to snatch asynchronous stream-of-consciousness statements, from strangers and friends alike, as they pass by in the microblogosphere conversation.

Chris went on a tear about Second Life, with several hilarious observations and comments within the space of an hour, so here are several from his public Twitter feed:

“The Largest Social Network Ever Analyzed”

FACT: According to ComScore data cited in a story in Monday’s FInancial Times, “Facebook, the fast-growing social network, has taken a significant lead over MySpace in visitor numbers for the first time… Facebook attracted more than 123 million unique visitors in May, an increase of 162 per cent over the same period last year… That compared with 114.6 million unique visitors at MySpace, Facebook’s leading rival, whose traffic grew just 5 per cent during the same period… The findings mark the first time that Facebook, launched in 2004, has taken a significant lead in unique visitors, [and] come at a time of change inside Facebook, as the one-time upstart attempts to transform itself into a leading media company.

ANALYSIS:  This week several members of the Microsoft Institute met in Redmond with a visiting friend from government, and among other talks we had a very interesting discussion with Eric Horvitz, a Microsoft Research principal researcher and manager.  Eric’s well known for his work in artificial intelligence and currently serves as president of the Association for the Advancement of Artificial Intelligence (AAAI).

We talked about one of Eric’s recent projects for quite a while: “Planetary-Scale Views on a Large Instant-Messaging Network,” a project which has been described by his co-author as “the largest social network ever analyzed.” 

Best Joke by a Supreme Court Justice

FACT: Chief Justice John Roberts said in a speech on Friday that he will increase the number of cases heard by the Supreme Court from two a day to three during the coming term beginning in October, according to an AP account of his speech. Roberts says that if the busier fall schedule lightens the caseload by next spring, he may be able to cut back then. 

ANALYSIS: When Roberts became Chief Justice in 2005, some Court observers wondered whether the younger Chief would begin burdening his colleagues with more work, increasing the Court’s caseload by granting more cases.  Now it looks like he may attack the issue in a slightly different way, hoping to cut a swath through the caseload issue with a burst of activity but not necessarily more cases overall.

Friday, the affable Roberts pointed out that the increase to three arguments each court day might put a strain on the Solicitor General’s Office, since it saddles the burden of arguing the federal government’s side in most of the cases.  But Roberts also pointed out that it’ll mean more work for the journalists who cover the Court – a small but tenacious crowd who like to pass judgment on the Court and its performance regularly.  Roberts then joked: “After careful reflection, I decided I didn’t care.”

Social (Network) Science

Fact: The social-networking site LinkedIn claims as users “17,000,000+ Professionals, 500,000+ Senior Executives, Executives from 498 Fortune 500 companies, [and] 65,000 new Professionals every week.”

Analysis: Since I hold the title of “chief technology officer” for my group at Microsoft, I regularly check the widely-read blog CTOvision, written by Bob Gourley, CTO of Crucial Point.

Yesterday he posted a very solid summary of several social networking tools, including my preferred LinkedIn.  If you’re not up to speed on the genre it is a helpful cheatsheet and “buyer’s guide.”

The technology area deserves the attention. There are a dozen or more such sites for each that Gourley covers, and he chooses the ones that have shown growth and potential longevity; why invest any time adding personal data to a site just to watch it disappear? We’ve all had that happen. And yet hockey-stick growth has to be managed – LinkedIn for example has come in for some critical attention for some snafus along the way.

Let’s look at some efforts to understand more about the science behind the software….

Mashup Markoff

John Markoff has a great story (that phrase belongs on a cut-n-paste clipboard) in today’s NY Times about PopFly, one of my favorite new Microsoft programs because it is so darned fun, and easy, and flexible – and potentially revolutionary.

I’m both a friend and fan of Markoff, but read his piece and you’ll see what I mean, as the examples he cites are pretty cool – or jump to the site itself.

PopFly is getting a lot of attention and coverage, landing on PCWorld’s list in December of “25 Most Innovative Products of the Year.”

The Three-Way Race, in Politics and Search

On the eve of the Super Tuesday primaries I’ve noticed a small but curious synchronicity, a sideways rhyming, between the Microsoft – Google – Yahoo elephant dance, and the back-and-forth among the top remaining candidates on the Republican and Democratic sides in the presidential primaries.

In graduate school I once wrote an 85-page study of “The Strategic Triangle: U.S., Soviet, and PRC Realignment during the 1970s.” Ah, the good old days of Henry Kissinger and grand-game geopolitics. But let’s stick to the more prosaic cage match dominating our politics right now.  Last week, John Edwards finally dropped out (or “suspended” his campaign, preserving some shred of pre-convention viability I suppose), and in doing so he refused to endorse either Barack Obama or Hillary Clinton. Both campaigns said great things about Edwards *after* he left the race, of course, the better to woo his supporters.  But before he dropped out, while he was still showing up in debates, both Clinton and Obama (and their surrogates) showed quite a bit of peevish annoyance that the third-place fellow wasn’t giving up and tossing them his endorsement.

Similarly, Mike Huckabee is hanging on by a thread on the Republican side, to the great solace of John McCain and the fuming of Mitt Romney, the latter believing that Huckabee’s conservative supporters would line up with him in a binary choice between Romney and McCain. (Would that count as a Baptist-to-Mormon conversion?) Romney spent the weekend bashing Huckabee even more than his putative rival, McCain.  Triangulation and frustration boil over into a combustible mix, obviously.

The same combination appears to be brewing in the Googleplex, while investors and analysts dump on the once shiny GOOG, which tanked yet again today, dropping below $500.