Debating Big Data for Intelligence

I’m always afraid of engaging in a “battle of wits” only half-armed.  So I usually choose my debate opponents judiciously.

Unfortunately, I recently had a contest thrust upon me with a superior foe: my friend Mark Lowenthal, Ph.D. from Harvard, an intelligence community graybeard (literally!) and former Assistant Director of Central Intelligence (ADCI) for Analysis and Production, Vice Chairman of the National Intelligence Council – and as if that weren’t enough, a past national Jeopardy! “Tournament of Champions” winner.

As we both sit on the AFCEA Intelligence Committee and have also collaborated on a few small projects, Mark and I have had occasion to explore one another’s biases and beliefs about the role of technology in the business of intelligence. We’ve had several voluble but collegial debates about that topic, in long-winded email threads and over grubby lunches. Now, the debate has spilled onto the pages of SIGNAL Magazine, which serves as something of a house journal for the defense and intelligence extended communities.

SIGNAL Editor Bob Ackerman suggested a “Point/Counterpoint” short debate on the topic: “Is Big Data the Way Ahead for Intelligence?” Our pieces are side-by-side in the new October issue, and are available here on the magazine’s site.

Mark did an excellent job of marshalling the skeptic’s view on Big Data, under the not-so-equivocal title, “Another Overhyped Fad.”  Below you will find an early draft of my own piece, an edited version of which is published under the title “A Longtime Tool of the Community”:

Visit the National Cryptologic Museum in Ft. Meade, Maryland, and you’ll see three large-machine displays, labeled HARVEST and TRACTOR, TELLMAN and RISSMAN, and the mighty Cray XMP-24. They’re credited with helping win the Cold War, from the 1950s through the end of the 1980s. In fact, they are pioneering big-data computers.

Here’s a secret: the Intelligence Community has necessarily been a pioneer in “big data” since inception – both our modern IC and the science of big data were conceived during the decade after the Second World War. The IC and big-data science have always intertwined because of their shared goal: producing and refining information describing the world around us, for important and utilitarian purposes

What do modern intelligence agencies run on? They are internal combustion engines burning pipelines of data, and the more fuel they burn the better their mileage. Analysts and decisionmakers are the drivers of these vast engines, but to keep them from hoofing it, we need big data.

Let’s stipulate that today’s big-data mantra is overhyped. Too many technology vendors are busily rebranding storage or analytics as “big data systems” under the gun from their marketing departments. That caricature is, rightly, derided by both IT cognoscenti and non-techie analysts.

I personally get the disdain for machines, as I had the archetypal humanities background and was once a leather-elbow-patched tweed-jacketed Kremlinologist, reading newspapers and HUMINT for my data. I stared into space a lot, pondering the Chernenko-Gorbachev transition. Yet as Silicon Valley’s information revolution transformed modern business, media, and social behavior across the globe, I learned to keep up – and so has the IC. 

Twitter may be new, but the IC is no Johnny-come-lately in big data on foreign targets.  US Government funding of computing research in the 1940s and ‘50s stretched from World War II’s radar/countermeasures battles to the elemental ELINT and SIGINT research at Stanford and MIT, leading to the U-2 and OXCART (ELINT/IMINT platforms) and the Sunnyvale roots of NRO.

In all this effort to analyze massive observational traces and electronic signatures, big data was the goal and the bounty.

War planning and peacetime collection were built on collection of ever-more-massive amounts of foreign data from technical platforms – telling the US what the Soviets could and couldn’t do, and therefore where we should and shouldn’t fly, or aim, or collect. And all along, the development of analog and then digital computers to answer those questions, from Vannevar Bush through George Bush, was fortified by massive government investment in big-data technology for military and intelligence applications.

In today’s parlance big data typically encompasses just three linked computerized tasks: storing collected foreign data (think Amazon’s cloud), finding and retrieving relevant foreign data (Bing or Google), and analyzing connections or patterns among the relevant foreign data (powerful web-analytic tools).

Those three Ft. Meade museum displays demonstrate how NSA and the IC pioneered those “modern” big data tasks.  Storage is represented by TELLMAN/RISSMAN, running from the 1960’s throughout the Cold War using innovation from Intel. Search/retrieval were the hallmark of HARVEST/TRACTOR, built by IBM and StorageTek in the late 1950s. Repetitive what-if analytic runs boomed in 1983 when Cray delivered a supercomputer to a customer site for the first time ever.

The benefit of IC early adoption of big data wasn’t only to cryptology – although decrypting enemy secrets would be impossible without it. More broadly, computational big-data horsepower was in use constantly during the Cold War and after, producing intelligence that guided US defense policy and treaty negotiations or verification. Individual analysts formulated requirements for tasked big-data collection with the same intent as when they tasked HUMINT collection: to fill gaps in our knowledge of hidden or emerging patterns of adversary activities.

That’s the sense-making pattern that leads from data to information, to intelligence and knowledge. Humans are good at it, one by one. Murray Feshbach, a little-known Census Bureau demographic researcher, made astonishing contributions to the IC’s understanding of the crumbling Soviet economy and its sociopolitical implications by studying reams of infant-mortality statistics, and noticing patterns of missing data. Humans can provide that insight, brilliantly, but at the speed of hand-eye coordination.

Machines make a passable rote attempt, but at blistering speed, and they don’t balk at repetitive mindnumbing data volume. Amid the data, patterns emerge. Today’s Feshbachs want an Excel spreadsheet or Hadoop table at hand, so they’re not limited to the data they can reasonably carry in their mind’s eye.

To cite a recent joint research paper from Microsoft Research and MIT, “Big Data is notable not because of its size, but because of its relationality to other data.  Due to efforts to mine and aggregate data, Big Data is fundamentally networked.  Its value comes from the patterns that can be derived by making connections between pieces of data, about an individual, about individuals in relation to others, about groups of people, or simply about the structure of information itself.” That reads like a subset of core requirements for IC analysis, whether social or military, tactical or strategic.

The synergy of human and machine for knowledge work is much like modern agricultural advances – why would a farmer today want to trudge behind an ox-pulled plow? There’s no zero-sum choice to be made between technology and analysts, and the relationship between CIOs and managers of analysts needs to be nurtured, not cleaved apart.

What’s the return for big-data spending? Outside the IC, I challenge humanities researchers to go a day without a search engine. The IC record’s just as clear. ISR, targeting and warning are better because of big data; data-enabled machine translation of foreign sources opens the world; correlation of anomalies amid large-scale financial data pinpoint otherwise unseen hands behind global events. Why, in retrospect, the Iraq WMD conclusion was a result of remarkably-small-data manipulation.

Humans will never lose their edge in analyses requiring creativity, smart hunches, and understanding of unique individuals or groups. If that’s all we need to understand the 21^st century, then put down your smartphone. But as long as humans learn by observation, and by counting or categorizing those observations, I say crank the machines for all their robotic worth.

Make sure to read both sides, and feel free to argue your own perspective in a comment on the SIGNAL site.

Petraeus as Ozymandias

I only met David Petraeus once before he came to CIA, in 2006 at U.S. Central Command while he was winding up his tour as commander of the Multi-National Security Transition Command Iraq (acronymically pronounced “minsticky”), and before he took command of MNF-I or CENTCOM, or the war in Afghanistan for that matter. I briefed him on something topical going on (I was still working at DIA at the time) and we certainly didn’t talk long. In fact I came away with only one impression: not so much about him, but about his already-well-commented-on entourage of “Petraeus guys.” He had a reputation as a fast-moving reformer, but it was an outsized group of admirers, I thought, who showed not respect for him, but devotion – even awe.

They weren’t alone; the man’s been compared as a military leader to “Ulysses S. Grant, John J. Pershing, George Marshall and Dwight D. Eisenhower” – and that was by his own boss! (That’s the comparison made by Chairman of the Joint Chiefs of Staff Adm. Mike Mullen last year when Petraeus retired from the military to join CIA.)

So, yes, news that the Director of the CIA had resigned because of an extramarital affair hit DC like a thunderclap yesterday.  Check out the volume of this twitter search for the prevailing phrase people uttered when they heard the news: “Holy shit.” It was almost comic that the news broke the same day that the new James Bond film opened in DC. Its plot features an intelligence agency director under personal assault and its title mirrors the mood of many in Langley today: “Skyfall.”

I’m not surprised by the fact that a powerful man was having an affair – heck, I did marry a divorce lawyer after all.  The news won’t affect intelligence operations immediately; the professionals at CIA and the intelligence community are still going about their business and tend to look forward to the horizon, not backward. Meanwhile journalists are already delving into the particulars of this peculiar turn of events. Pundits (and the Congressional intelligence oversight committees) will be exploring any linkages or ramifications of this scandal for the Benghazi investigations, and the candidates for Petraeus’s replacement are already making their direct or whisper campaigns known, in emails already bcc’ing around the Beltway. More on that in due time.

I only have two observations now, one larger in scope and one quite small, at human scale. The first is the question of what the scandal says about the intelligence security practices in our modern national security state. Petraeus held the highest security clearances. He earned the confidence of the President, the trust of his silent warrior employees, the endorsement of the U.S. Senate (94-0!) and the faith of a nation that had cheered his battlefield successes in the Iraq surge and in Afghanistan. Yet the CIA’s confidence in its director was undergirded not only by the Petraeus resume, but by our national security infrastructure of clearances, polygraphs, and professional investigators. Forget the question of one man’s integrity – he was living a lie, big-time, and we missed it. Completely. There will be many questions asked about what that means for other high government clearance-holders, but for now there’s a feeling prevalent in DC akin to what happens when a law-enforcement crime lab discovers shoddy mistakes: all previous convictions are under suspicion and, sometimes, verdicts are reversed. Something to ponder about CIA institutional analytic or operational judgment over the past year….

Secondly, I’m struck by the ironies in the personal side of this affair. David Petraeus grew up as a literature-loving son of a New England village librarian. I know this because I read his biography – yes, the hagiographic book All In: The Education of David Petraeus written by the woman at the center of the affair. Now I may be one of the few in DC who actually read the whole book when it came out – as in, I didn’t just flip through the index looking for the “good parts.”

The book has the literature-loving Petraeus actually quoting poetry at a pivotal point in his life. At his change-of-command ceremony, giving up his praetorian position in Afghanistan, Petraeus gave a thoughtful set of remarks and then chose to quote several lines from an obscure poem by young British soldier John Bailey, serving in Afghanistan in 2008. I say “obscure,” because until today the poem itself appears in only one spot on the Internet: a small U.K. site devoted to British war poetry.  Did poetry-lover Petraeus find the poem there himself, or was it simply good staff/speechwriter work? These are the words Petraeus used, in his “emotional” farewell to the wars he had led, and to his chosen career as a military leader:

And what is asked for the service we give?

No high praise or riches if we should live,

Just silence from friends, our name on a wall,

If this time around, it is I that fall.

– from “The Volunteer” by John Bailey

When Petraeus read out that poem, he was standing like Caesar astride a narrow world, a four-star general having “won” two wars in distant ancient lands and commanded USCENTCOM, whose mission area sprawls across Africa, the Middle East and Central Asia.

Perhaps this poetry lover knows Percy Bysshe Shelley well; perhaps like me in school Petraeus read Shelley’s Ozymandias, based on the ironic life of Ramesses II, mighty Egyptian pharaoh. One account writes, “Ramesses could have filled an ancient edition of the Guinness Book of Records all by himself: he built more temples, obelisks and monuments; took more wives (eight, not counting concubines) and claimed to have sired more children (as many as 162, by some accounts) than any other pharaoh in history. And he presided over an empire that stretched from present-day Libya to Iraq in the east, as far north as Turkey and southward into the Sudan.”

Yet Ramesses is mostly forgotten now, and Shelley’s poem about him captures the fall of great men in a short, powerful sonnet. When I first heard the news about Petraeus from my wife, this is the poem I thought of, and I believe its irony pairs with the lines Petraeus quoted quite sadly.

I met a traveller from an antique land

Who said: “Two vast and trunkless legs of stone

Stand in the desert. Near them on the sand,

Half sunk, a shattered visage lies, whose frown

And wrinkled lip and sneer of cold command

Tell that its sculptor well those passions read

Which yet survive, stamped on these lifeless things,

The hand that mocked them and the heart that fed.

And on the pedestal these words appear:

`My name is Ozymandias, King of Kings:

Look on my works, ye mighty, and despair!’

Nothing beside remains. Round the decay

Of that colossal wreck, boundless and bare,

The lone and level sands stretch far away”.

Increasing Jointness and Reducing Duplication in DoD Intelligence

Today I’m publishing an important guest-essay, with a brief introduction.  Last month the Wall Street Journal published a 12-part online series about college graduates and their paths to success, featuring surveys and input from job recruiters. One thing caught my eye, at least when blogged by an acquaintance, Prof. Kristan Wheaton of the Mercyhurst College Institute Of Intelligence Studies. The WSJ’s study included a look at recent graduates’ job satisfaction in their new careers, and as Prof. Wheaton strikingly put it in his own blogpost:

“Intelligence Analysts are Insanely Happy.” 

I’m pretty sure that’s not really true by and large; Prof. Wheaton seems slightly dubious as well. Many readers of this blog are intelligence analysts themselves, so I’d love to hear from you (in comments or email) about your degree of giddyness….

We all know that the intelligence-analysis field as currently practiced in U.S. agencies bears many burdens weighing heavily on job satisfaction, and unfortunately weighing on successful performance.  Our youngest and our most experienced intelligence analysts have been battling those burdens. 

One analyst has now put constructive thoughts on paper, most immediately in response to a call by Defense Secretary Bob Gates asking DoD military and civilian employees to submit their ideas to save money, avoid cost, reduce cycle time and increase the agility of the department (see more about the challenge here).  

Simon Moves On

Jim Simon at the Library of Alexandria, Egypt

One indulgent use of a personal blog is to drop a nod in the direction of a salutary individual, and I’d like to do so for my departing boss, Jim Simon.

Jim has been the founding Director of the Microsoft Institute since 2004, when Bill Gates and Craig Mundie personally decided to establish a small outfit to use the benefits of Microsoft’s advanced research and development activities against intractable problems for the global public sector. They had been talking with Jim for several years, back when he was a senior executive at the Central Intelligence Agency and after, to understand how to improve government’s adoption of modern technologies.

To fix intelligence analysis you have to decide what’s broken

“More and more, Xmas Day failure looks to be wheat v. chaff issue, not info sharing issue.” – Marc Ambinder, politics editor for The Atlantic, on Twitter last night.

Marc Ambinder, a casual friend and solid reporter, has boiled down two likely avenues of intelligence “failure” relevant to the case of Umar Farouk Abdulmutallab and his attempted Christmas Day bombing on Northwest Airlines Flight 253.  In his telling, they’re apparently binary – one is true, not the other, at least for this case.

The two areas were originally signalled by President Obama in his remarks on Tuesday, when he discussed the preliminary findings of “a review of our terrorist watch list system …  so we can find out what went wrong, fix it and prevent future attacks.” 

Let’s examine these two areas of failure briefly – and what can and should be done to address them.

The Purple History of Intelink

When I first began talking with DIA CIO Mike Pflueger and Deputy CIO Mark Greer in the fall of 2003 about the work I’d be doing with them inside government, most of the ideas were big ones: let’s re-architect the DoDIIS enterprise, let’s find and deploy revolutionary new analytical software. One of our thoughts was a little one, but for me personally it turned out to be a most valuable project. They let me pull together a panel for the upcoming 2004 DoDIIS Conference called “Geeks and Geezers,” featuring some of the grand old names of intelligence technology. The panel was a success, and in organizing it, I spent quite a bit of time talking to those giants, or should I say listening to them. I learned an enormous amount about “the early days.” This post describes the important work of one of those fellows. 

A-Space Past and Future

This week marks the second anniversary of the first live internal demo of the intelligence community’s A-Space project, groundbreaking for the IC in its goal of collaborative use of social media across agency lines. Somewhere in Maryland, a remarkable government employee and friend named Mike Wertheimer should pause and quietly celebrate the fruition of his early evangelism for it.

I was still a government employee then, but wrote about the effort at the time here on Shepherd’s Pi (“A-Space: Top-secret social networking“). It makes me chuckle to remember back to those days when it was still mostly unheard-of for IC employees to blog openly on the public web about current technology projects. Now you can’t shut ’em up! 🙂

It made sense, I thought, to set down a few notes at the time for several reasons:

Obama Team Infighting on Intelligence

The easiest prediction in Washington is this one: “bureaucratic turf war.” The Obama Administration isn’t immune. Several months ago when the president-elect announced his names for DNI and CIA director, I put forth this idea:  “Swap Blair and Panetta: A Modest Proposal.”  In it I wondered, between the two of them, “Who gets the top bunk?”

Well, they still haven’t figured that out yet. 

The Spy Chief and Bohemian Rhapsody

Very often, complaints about the stasis of the reform-resistant Intelligence Community recount the same old complaints, about how little the bureaucratic and stovepiped mentality has changed inside its oh-so-thick walls.

Yet I find some encouraging signs, and this week featured another one: the ceremony for this year’s winning entries in the IC’s “Galileo Awards” program, designed to reward innovative ideas and proposals for new thinking and positive change. More than 50 entries came from 14 separate agencies across the community, and this week marked the public crowning (at least, publicly to an intell audience) of the three winning papers.

Just in case you weren't sure - that's DNI Blair on the left, I believe, Freddie Mercury on right

And oh, how far we’ve come from the days of DNI Mike McConnell, not to say gray-faced shadowy DCI’s like Dulles and Colby, when we see a DNI rock the stage as Dennis Blair did in Wednesday’s ceremony.

Rock the stage? Well, I take it as a real sign of progressive change in the IC that we now have a Director of National Intelligence who can easily and with familiarity quote from an iconic rock song of the 1970s to make his point:

Everyone knows something about Galileo – if nothing else, as a lyric in the Queen song, “Bohemian Rhapsody.” You head-bangers in the audience, if you know what I’m talking about, bob your heads up and down.

For the rest of you who are a little less contemporary, Galileo was the Renaissance scientist who proved empirically that the Sun – not the Earth – was the center of the known universe.  It’s too bad he’s still not around in some modern version to prove Washington is not the center of the universe.

Blair did have a more serious point to make: “We have to work to create an integrated global enterprise that can anticipate and respond to rapidly changing threats. Innovation has never been more important. And keeping the Intelligence Community on the cutting edge of innovation is a critical priority for all of us.”

Unfortunately for those of you without security clearances, the submissions – the “innovative ideas” on how to improve intelligence collection, analysis, and operations – are often classified. So you won’t be able to read them in the open.  Nor should you 🙂

But it is at least worth mentioning, as Blair did, that for the first time in the often CIA-dominated awards, a paper submitted by two FBI officers (often considered the red-haired stepchildren of the IC) received honorable mentions. As Freddie Mercury – oh, sorry, as Denny Blair said: ” That’s an important trend toward participation and collaborative thinking, from all corners of our Intelligence Community.”

My congratulations to the winners – and a tip of the hat to all those who submitted.

Email this post to a friend

DNI Flags at Half-Mast

Only the second-ever Director of National Intelligence, Mike McConnell, resigned today effective immediately. As the Associated Press reported this afternoon in the wake of the announcement, “Lt. Gen. Ronald L. Burgess, Jr. is temporarily serving as acting national intelligence director… McConnell’s letter did not explain why he resigned before the Senate’s confirmation of his replacement. President Barack Obama has nominated retired Adm. Dennis Blair to be the next national intelligence director.” 

Analysis:  Given the impending Senate hearings on Denny Blair’s confirmation and the expected smooth sailing, most people I know were mildly surprised that McConnell jumped ship today, rather than waiting for a formal turnover to a confirmed Blair. McConnell has had a solid, successful track record of leading the IC in an era of long-needed reform, while contributing to a track record in his tenure of zero terrorist attacks on American soil.

But then my inbox pinged with another notice from the Office of the DNI: release of “The 500-Day-Plan Update at Day 400” (download the PDF version here).  It contained a graphic depiction of the troubling challenges remaining – actually using graphic “red flags” to mark areas at risk.  More on the flags below.

Those who work in and with the intelligence community have been intimately familiar with the DNI’s 500-Day Plan.  When it was first drafted I was still in government and had my tiny slice of input into its composition through the interagency review process. Its release was hailed by some (“ODNI Earns Kudos for 500-Day Plan,” in Federal Computer Week) and greeted with a yawn in some sectors of the community itself. “Another reform plan? I’ll make room on the shelf.”