Bullshit Detector Prototype Goes Live

I like writing about cool applications of technology that are so pregnant with the promise of the future, that they have to be seen to be believed, and here’s another one that’s almost ready for prime time.

The Washington Post today launched an exciting new technology prototype invoking powerful new technologies for journalism and democratic accountability in politics and government. As you can see from the screenshot (left), it runs an automated fact-checking algorithm against the streaming video of politicians or other talking heads and displays in real time a “True” or “False” label as they’re speaking.

Called “Truth Teller,” the system uses technologies from Microsoft Research and Windows Azure cloud-computing services (I have included some of the technical details below).

But first, a digression on motivation. Back in the late 1970s I was living in Europe and was very taken with punk rock. Among my favorite bands were the UK’s anarcho-punk collective Crass, and in 1980 I bought their compilation LP “Bullshit Detector,” whose title certainly appealed to me because of my equally avid interest in politics 🙂

Today, my driving interests are in the use of novel or increasingly powerful technologies for the public good, by government agencies or in the effort to improve the performance of government functions. Because of my Jeffersonian tendencies (I did after all take a degree in Government at Mr. Jefferson’s University of Virginia), I am even more interested in improving government accountability and popular control over the political process itself, and I’ve written or spoken often about the “Government 2.0” movement.

In an interview with GovFresh several years ago, I was asked: “What’s the killer app that will make Gov 2.0 the norm instead of the exception?”

My answer then looked to systems that might “maintain the representative aspect (the elected official, exercising his or her judgment) while incorporating real-time, structured, unfiltered but managed visualizations of popular opinion and advice… I’m also a big proponent of semantic computing – called Web 3.0 by some – and that should lead the worlds of crowdsourcing, prediction markets, and open government data movements to unfold in dramatic, previously unexpected ways. We’re working on cool stuff like that.”

The Truth Teller prototype is an attempt to construct a rudimentary automated Political Bullshit Detector, and addresses each of those factors I mentioned in GovFresh – recognizing the importance of political leadership and its public communication, incorporating iterative aspects of public opinion and crowd wisdom, all while imbuing automated systems with semantic sense-making technology to operate at the speed of today’s real world.

Real-time politics? Real-time truth detection.  Or at least that’s the goal; this is just a budding prototype, built in three months.

Cory Haik, who is the Post’s Executive Producer for Digital News, says it “aims to fact-check speeches in as close to real time as possible” in speeches, TV ads, or interviews. Here’s how it works:

The Truth Teller prototype was built and runs with a combination of several technologies — some new, some very familiar. We’ve combined video and audio extraction with a speech-to-text technology to search a database of facts and fact checks. We are effectively taking in video, converting the audio to text (the rough transcript below the video), matching that text to our database, and then displaying, in real time, what’s true and what’s false.

We are transcribing videos using Microsoft Audio Video indexing service (MAVIS) technology. MAVIS is a Windows Azure application which uses State of the Art of Deep Neural Net (DNN) based speech recognition technology to convert audio signals into words. Using this service, we are extracting audio from videos and saving the information in our Lucene search index as a transcript. We are then looking for the facts in the transcription. Finding distinct phrases to match is difficult. That’s why we are focusing on patterns instead.

We are using approximate string matching or a fuzzy string searching algorithm. We are implementing a modified version Rabin-Karp using Levenshtein distance algorithm as our first implementation. This will be modified to recognize paraphrasing, negative connotations in the future.

What you see in the prototype is actual live fact checking — each time the video is played the fact checking starts anew.

 – Washington Post, “Debuting Truth Teller“

The prototype was built with funding from a Knight Foundation’s Prototype Fund grant, and you can read more about the motivation and future plans over on the Knight Blog, and you can read TechCrunch discussing some of the political ramifications of the prototype based on the fact-checking movement in recent campaigns.

Even better, you can actually give Truth Teller a try here, in its infancy.

What other uses could be made of semantic “truth detection” or fact-checking, in other aspects of the relationship between the government and the governed?

Could the justice system use something like Truth Teller, or will human judges and  juries always have a preeminent role in determining the veracity of testimony? Will police officers and detectives be able to use cloud-based mobile services like Truth Teller in real time during criminal investigations as they’re evaluating witness accounts? Should the Intelligence Community be running intercepts of foreign terrorist suspects’ communications through a massive look-up system like Truth Teller?

Perhaps, and time will tell how valuable – or error-prone – these systems can be. But in the next couple of years we will be developing (and be able to assess the adoption of) increasingly powerful semantic systems against big-data collections, using faster and faster cloud-based computing architectures.

In the meantime, watch for further refinements and innovation from The Washington Post’s prototyping efforts; after all, we just had a big national U.S.  election but congressional elections in 2014 and the presidential race in 2016 are just around the corner. Like my fellow citizens, I will be grateful for any help in keeping candidates accountable to something resembling “the truth.”

Obama Team Infighting on Intelligence

The easiest prediction in Washington is this one: “bureaucratic turf war.” The Obama Administration isn’t immune. Several months ago when the president-elect announced his names for DNI and CIA director, I put forth this idea:  “Swap Blair and Panetta: A Modest Proposal.”  In it I wondered, between the two of them, “Who gets the top bunk?”

Well, they still haven’t figured that out yet. 

Way Ahead and Far Behind

Today’s Washington Post has a story on its front page: “Staff Finds White House in the Technological Dark Ages.”

Two years after launching the most technologically savvy presidential campaign in history, Obama officials ran smack into the constraints of the federal bureaucracy yesterday, encountering a jumble of disconnected phone lines, old computer software, and security regulations forbidding outside e-mail accounts.”

“What does that mean in 21st-century terms? No Facebook to communicate with supporters. No outside e-mail log-ins. No instant messaging. Hard adjustments for a staff that helped sweep Obama to power through, among other things, relentless online social networking.”  -Washington Post

Some say that whoever has been responsible for information technology in the White House itself should be fired — but then perhaps the change of Administration just took care of that  🙂 

Overall, this situation is familiar to anyone who has worked in what I call “Big-G  IT” or the information technology of a federal government agency. I’ve argued about its challenges and sub-optimality before: see my previous pieces on “Roadmap for Innovation: From the Center to the Edge,” and more specifically “Puncturing Circles of Bureaucracy.”  In that latter piece back in March of 2008, I wrote about the “the defensive perimeters of overwhelming bureaucratic torpor,” and the frustrating reality within much of Big Government: “Federal employees have an entire complex of bizarrely-incented practices and career motivations, which make progress on technology innovation very difficult, not to mention general business-practice transformation as a whole.”

Here’s the truly frustrating, mind-bending part: it isn’t always true!  Other elements of the White House have cutting-edge, world-class technologies operating day in, day out.

Swap Panetta and Blair: A Modest Proposal

First, a quick story from when I was working in government.

Not long after the initial establishment of a “Director of National Intelligence,” the DNI CIO held an inaugural “DNI Information Sharing Conference” in Denver in the summer of 2006. I was asked to sit on a panel about “Innovation across the Intelligence Community,” representing the Defense Intelligence Agency and sharing the stage with two counterparts, from the CIA and NSA.  Our panel chair was Mr. CJ Chapla, then the Chief Technology Officer (CTO) of the old Intelink Management Office, redubbed “Intelligence Community Enterprise Services,” an office now under the Office of the DNI (ODNI). CJ asked the three of us to describe briefly the goals and projects we were each working on, and in seriatim that’s what we did for 90 minutes or so.

When it was time for questions, the very first audience-member asked: “It seems that each of you are independently working on, and paying for, very similar kinds of technology projects. It would make sense to combine or rationalize the work, so why are you continuing to do it independently?”

Elbowing for Obama influence between new CTO, new cyber czar

Today’s Friday – usually a big news day in Washington, whether by design (bury bad news late in a deep weekend news hole) or by human error (bureaucrats tried all week to get something done and slipped it in at the deadline).  There should be Obama cabinet announcements today, and meanwhile tech luminaries across the country are sitting by their phones, drumming their fingers and hoping for a call offering them the position of the nation’s first Chief Technology Officer. Norm Lorentz, who was OMB’s first-ever CTO, told C-SPAN this week that “If I were asked, I would serve in a heartbeat.”

My bold decision to withdraw from consideration as Obama’s CTO

To: President-Elect Barack Obama

From: Lewis Shepherd

RE: My Imminent Selection as Chief Technology Officer for the United States

Mr. President-Elect, I am hereby reluctantly but insistently withdrawing my name from consideration as your appointment to the newly created position of Chief Technology Officer for our nation.

No, no, please don’t try to persuade me otherwise. My decision is final.

Analysis: My earlier post about John Brennan being President-elect Obama’s “imminent” selection as CIA director is now a curio, given Brennan’s decision yesterday to withdraw from consideration. 

Like any good intelligence analyst writing a balanced assessment, I had included the caveat that the only thing standing between Brennan and the appointment was the likelihood of a last-minute political squabble or contretemps.

This being Washington we’re talking about, that is precisely what happened. 

Washington Post Puts Microsoft on Page A1 – For Good Research!

Stop the presses! Microsoft Research is getting national front page coverage!

The work of Eric Horvitz and Jure Leskovec got top coverage in major newspapers and news sites today.  With that fame, Eric will probably never again be willing to just while away a Friday afternoon with our Microsoft Institute folks, brainstorming some outside-the-box ideas for future work, as he did this week with us in Redmond’s Building 99.

Right after that meeting, I bugged out of Redmond for a red-eye to the east coast.  Back home in DC this morning (Saturday), I opened my Washington Post to find on page A1, “Instant-Messagers Really Are About Six Degrees from Kevin Bacon: Big Microsoft Study Supports Small World Theory.”

Tradecraft in the Long Tail

Fact: Chris Anderson, WIRED editor in chief and author of the Internet-era classic book “The Long Tail,” also runs a couple of Ning social networks focusing on what the intelligence community would call IMINT, or imagery intelligence – specifically DIY Drones, “a site for all things about amateur Unmanned Aerial Vehicles (UAVs): How-to’s, videos, discussion and more,” and PictEarth, “a Social Network used to collect, link and geotag RC, UAV/UAS or Kite derived Earth imagery for use in 3D Globe Programs including Google Earth, Virtual Earth, World Wind and ArcGIS Explorer.”

Analysis:  With these sites, Chris Anderson is promoting what he calls “Crowdsourced Aerial Imagery.”  In the mission statement for DIY Drones, he writes that “Reasons to make your own UAV range from a fun technical challenge, student contests, aerial photography and mapping (what we call “GeoCrawling”), and scientific sensing. We are primarily interested in civilian, not military, UAV uses here.” (Emphasis is in the original.)

Let’s presume that individual DoD or intelligence-agency personnel have an interest in such issues, and maybe even in spending their personal time by keeping current and following the crowd’s interest in such topics, by participating in these new social networks.  One can then assume that others from foreign intelligence might have some interest in tracking those very IC personnel, by observing their activities within social networks (and not just Ning ones).  No spectacular logic needed for that.

The CIA has had some challenges in understanding their field presence within the Long Tail. 

Pentagon’s New Program for Innovation, in Context

FACT: According to an article in today’s Washington Post, the Pentagon has announced “the selection of six university professors who will form the first class of the National Security Science and Engineering Faculty Fellows Program. The professors will receive grants of up to $600,000 per year for up to five years to engage in basic research — essentially a bet by the Pentagon that they will make a discovery that proves vital to maintaining the superiority of the U.S. military.”

ANALYSIS: This new program is an innovation from DoD’s Director of Defense Research and Engineering (DDR&E), and since tomorrow I’ll be at Ft. McNair for a two-day conference sponsored by DDR&E on Strategic Communications, I’ll congratulate John Young and his staff for the good idea.

But the Post article falls short in two ways: one immediate (it leaves out key information about next year’s program and the upcoming deadline!) and one longer-term (it ignores the overall context of federal support for R&D).  I’ll fill in the blanks below.

Is It Even Possible to Connect the Dots?

FACT: Among the inspired ideas of polymath Danny Hillis (pioneer of parallel computing) was establishing the Long Now Foundation, whose projects include the millennial Long-Now Clock (“the world’s slowest computer”) and the notion of “Long Bets.”  A Long Bet is an “accountable prediction,” meaning one that has a specified end-date and a testable, wagerable, proposition.  One of the early Long Bets posted wagers $2,000 that “By 2020, no one will have won a Nobel Prize for work on superstring theory, membrane theory, or some other unified theory describing all the forces of nature.”  That particular bet is one of many signs of scientific skepticism about string theory.

ANALYSIS: Even without the ease of hyperlinks, old-fashioned newspapers foster serendipitous connections between articles, particularly if you’re reading a Sunday morning paper with lots of sections. Sunday the Washington Post did me a service by placing in different sections a couple of articles which I connected, about intelligence “failures” and about stock-market prediction, leading me to some web-surfing about the questionable validity of string theory and some related observations about the difficulty of predicting human behavior.

In the Outlook section, the Post has an opinionated and thought-provoking op-ed piece by Mark Lowenthal, one of the most “intelligent” individuals in the recent history of the U.S. intelligence community (after all, he was the 1988 Jeopardy grand champion, as well as a former assistant director of CIA).