Twitter Search as a Government case study

In addition to periodic think-pieces here at Shepherd’s Pi, I also contribute a monthly online column over at SIGNAL Magazine on topics relating to intelligence. This month I keyed off a recent discussion I had onstage at the 2015 AFCEA Spring Intelligence Symposium with Elon Musk, particularly a colloquy we had on implications of the emerging cleavage (post-Edward Snowden) between Silicon Valley technology companies and their erstwhile innovation partners, U.S. intelligence agencies.

That discussion sparked some thinking on the public/private sector divide on tech innovation – and on basic operational performance in building or adopting new technologies. It’s always been a hobbyhorse topic of mine; see previous pieces even from way back in 2007-08 like “Pentagon’s New Program for Innovation in Context,” or “A Roadmap for Innovation – From the Center or the Edge?” or “VC-like Beauty Contests for Government.”

I have an excerpt from my new SIGNAL piece below, but you can read the entire piece here: “The Twitter Hare Versus the Government Turtle.”

Is the public/private divide overstated? Can the government compete? Without going into the classified technology projects and components discussed at the symposium, let’s try a quick proxy comparison, in a different area of government interest: archiving online social media content for public use and research. Specifically, since Twitter data has become so central to many areas of public discourse, it’s important to examine how government and private sector are each addressing that archive/search capability.

First, the government side. More than half a decade ago, the Library of Congress (LoC) announced in April 2010 with fanfare that it was acquiring the “complete digital archives” of Twitter, from its first internal beta tweets. At that time, the LoC noted, the 2006-2010 Twitter archive already consisted of 5 terabytes, so the federal commitment to archiving the data for search and research was significant…

  … Fast forward to today. Unbelievably, after even more years of “work,” there is no progress to report—quite the opposite. A disturbing new report this week in Inside Higher Education entitled “The Archive is Closed” shows LoC at a dead-stop on its Twitter archive search. The publicly funded archive still is not open to scholars or the public, “and won’t be any time soon.”

  … Coincidentally this week, just as the Library of Congress was being castigated for failing in its mission to field a usable archive after five years, Twitter unveiled a new search/analytics platform, Twitter Heron—yes, after just six months [after releasing its previous platform Twitter Storm]. Heron vastly outperforms the original version in semantic throughput and low latency; yet in a dramatic evocation of Moore’s Law, it does so on 3 times less hardware.

Twitter Storm vs Twitter Heron

Oh, and as the link above demonstrates, the company is far more transparent about its project and technology than the Library of Congress has been.

All too often we see government technology projects prove clunky and prone to failure, while industry efforts are better incentivized and managerially optimized for success. There are ways to combat that and proven methods to avoid it. But the Twitter search case is one more cautionary example of the need to reinvigorate public/private partnerships—in this case, directly relevant to big-data practitioners in the intelligence community.

 – Excerpts from SIGNAL Magazine, “The Twitter Hare Versus the Government Turtle.” © 2015 AFCEA International.

Meet the Future-Makers

Question: Why did Elon Musk just change his Twitter profile photo? I notice he’s now seeming to evoke James Bond or Dr. Evil:

twitter photos, Elon v Elon

I’m not certain, but I think I know the answer why. Read on…

________________________________________

“Prediction is very difficult, especially about the future.”

      – Niels Bohr, winner of the 1922 Nobel Prize for Physics

“History will be kind to me for I intend to write it.”

      – Winston Churchill

If you take those two quotations to heart, you might decide to forego the difficulty of predicting the future, instead aiming to bend the future’s story arc yourself. In a nutshell, that’s what R&D is all about: making the future.

Who makes the future for the intelligence community? Who has more influence on the future technologies which intelligence professionals will use: government R&D specialists, or private-sector industry?

On the one hand, commercial industry’s R&D efforts are pulled by billions of invisible consumer hands around the globe, driving rapid innovation and ensuring that bold bets can be rewarded in the marketplace. Recent examples are things like Web search, mobile phones and tablets, and SpaceX launches.

To be fair, though, the US IC and DoD have the ability to focus intently on specific needs, with billions of dollars if necessary, and to drive exotic game-changing R&D for esoteric mission use. During my time in government I saw great recent successes which are of course classified, but they exist.

If you want to explore both sides and you have a Top Secret clearance, you’re in luck, because you can attend what I expect will be an extraordinary gathering of Future-Makers from inside and outside the IC, at next month’s AFCEA Spring Intelligence Symposium.

Spring IntellLast fall, the organizing committee for this annual classified Symposium began our planning on topics and participants. We decided that this year’s overall theme had to be “IC Research & Development” – and we decided to depart from tradition and bring together an unprecedented array of senior leaders from inside and outside, to explore the path forward for IC innovation and change.

The May 20-21 Symposium, held at NGA’s Headquarters, will be a one-of-a-kind event designed to set the tone and agenda for billions of dollars in IC investment.  On the government front, attendees will witness the roll-out of the new (classified) Science & Technology 2015-2019 Roadmap; see this article for some background on that. Attendees will also meet and hear R&D leaders from all major IC agencies, including:

  • Dr. David Honey, Director of Science and Technology, ODNI
  • Dr. Peter Highnam, Director, Intelligence Advanced Research Projects Activity (IARPA)
  • Glenn Gaffney, Deputy Director for Science & Technology, CIA
  • Stephanie O’Sullivan, Principal Deputy Director of National Intelligence
  • Dr. Greg Treverton, Chairman, National Intelligence Council
  • The IC’s functional managers for SIGINT, MASINT, GEOINT, HUMINT, OSINT, and Space

Meanwhile from the private sector, we’ll have:

  • Elon Musk, CEO/CTO of SpaceX, CEO/Chief Product Architect of Tesla Motors, CEO of SolarCity, Co-founder of PayPal
  • Gilman Louie, Partner at Alsop Louie Venture Capital, former CEO of In-Q-Tel
  • Bill Kiczuk, Raytheon VP, CTO, and Senior Principal Engineering Fellow
  • Zach Lemnios, IBM VP for Research Strategy and Worldwide Operations
  • Pres Winter, Oracle VP, National Security Group

When I first proposed that we invite an array of “outside” future-makers to balance the government discussion with a different perspective, I said to my colleagues on the planning committee, “Wouldn’t it be awesome to get someone like Elon Musk…”

Well, we did, and next month I’ll be welcoming him on stage.

These are dark and challenging times in international security, but for scientists, technologists, and engineers, there’s never been a more exciting time – and like them, intelligence professionals should stretch their horizons.

I’m looking forward to the conference… and here’s your link to register to join us.

PS: Just to whet your appetite: new video of this week’s SpaceX revolutionary Falcon9 first-stage landing attempt on a drone barge at sea – nearly made it, very exciting:

MSR gets wired, WIRED gets MSR

MS Research in natural-user-interaction technologies
MSR natural-user-interaction immersive technologies

WIRED Magazine’s online site ran a great long profile of Microsoft Research late yesterday, with interviews and project features: “How Microsoft Researchers Might Invent a Holodeck.”

I have written about or mentioned all of the individual projects or technologies on my blog before, but the writing at WIRED is so much better than my own – and the photographs so cool – that I thought I should post a link to the story. Continue reading

Virtual recipe stirs in Apple iPad, Microsoft Kinect

Who says Apple and Microsoft can’t work together?  They certainly do, at least when it involves the ingenuity of their users, the more inventive of whom use technologies from both companies (and others).

Here’s a neat example, “a just-for-fun experiment from the guys at Laan Labs” where they whip up a neat Augmented Reality recipe: take one iPad, one Kinect, and stir:

Some technical detail from the Brothers Laan, the engineers who did the work:

We used the String Augmented Reality SDK to display real-time 3d video+audio recorded from the Kinect. Libfreenect from http://openkinect.org/ project was used for recording the data coming from the Kinect. A textured mesh was created from the calibrated depth+rgb data for each frame and played back in real-time. A simple depth cutoff allowed us isolate the person in the video from the walls and other objects. Using the String SDK, we projected it back onto a printed image marker in the real world.” – source, Laan Labs blog.

As always, check out http://www.kinecthacks.com/ for the latest and greatest Kinect hacks – or more accurately now, the latest cool uses of the openly released free Kinect SDK, available here.

There are several quiet projects underway around the DC Beltway to make use of the SDK, testing non-commercial but government-relevant deployments – more detail and examples at the appropriate time. We will eventually release a commercial SDK with even more functionality and higher-level programming controls, which will directly benefit government early adopters.

In the meantime, I may report on some of the new advances being made by our research group on Computational User Experiences, who “apply expertise in machine learning, visualization, mobile computing, sensors and devices, and quantitative and qualitative evaluation techniques to improve the state of the art in physiological computing, healthcare, home technologies, computer-assisted creativity, and entertainment.” That’s a rich agenda, and the group is in the very forefront of defining how Natural User Interaction (NUI) will enhance our personal and professional lives….

Share this post on Twitter

Email this post to a friend

Kinecting Communities

On April 16 I will be speaking at the Mobile Citizen Summit in Washington DC (registration still open), which brings together “practitioners across the  government, nonprofit, advocacy, and political spaces—the kinds of  people who develop the strategy and the tools to reach, engage, educate,  and enable citizens across the country and around the world.”

But I’m going to be talking about “mobile” in a different way than others still use the term, i.e. they focus on a handheld device, while I will be focusing on the mobile citizen. As I have said before I don’t believe our future involves experiencing “augmented reality” by always holding up little 3-inch plastic screens in front of our faces. Natural user interfaces and immersive computing offer much more to how we access computational resources – and how technology will help us interact with one another. Here’s an example, in a story from the past week.

Continue reading

Tearing the Roof off a 2-Terabyte House

I was home last night playing with the new Kinect, integrating it with Twitter, Facebook, and Zune. Particularly because of the last service, I was glad that I got the Xbox 360 model with the 250-gigabyte (gb) hard disk drive. It holds a lot more music, or photos, and of course primarily games and game data.

So we wind up with goofy scenes like my wife zooming along yesterday in Kinect Adventures’ River Rush – not only my photo (right) but in-game photos taken by the Kinect Sensor, sitting there below the TV monitor.

Later as I was waving my hands at the TV screen, swiping magically through the air to sweep through Zune’s albums and songs as if pawing through a shelf of actual LP’s, I absent-mindedly started totting up the data-storage capacity of devices and drives in my household.  Here’s a rough accounting:

  • One Zune music-player, 120gb;
  • 2 old iPods 30gb + 80gb;
  • an iPad 3G at 16gb;
  • one HP netbook 160gb;
  • an aging iMac G5 with 160gb;
  • three Windows laptops of 60gb, 150gb, and 250gb;
  • a DirecTV DVR with a 360gb disk;
  • a single Seagate 750gb external HDD;
  • a few 1gb, 2gb, and a single 32gb SD cards for cameras;
  • a handful of 2gb, 4gb, and one 16gb USB flash drives;
  • and most recently a 250gb Xbox 360, for Kinect. 

All told, I’d estimate that my household data storage capacity totals 2.5 terabytes. A terabyte, you’ll recall, is 1012 bytes, or 1,000,000,000,000 (1 trillion) bytes, or alternately a thousand gigabytes.

Continue reading

Four Score and Seven Years Ago

Today, August 5, has a number of interesting anniversaries in the world of technology and government. In 1858 the first transatlantic telegraph cable was completed, allowing President James Buchanan and Queen Victoria to share congratulatory messages the following week. (Unfortunately within a month the cable had broken down for good.)  The first quasar (“quasi-stellar astronomical radio object”) was discovered on Aug. 5, 1962. And exactly one year later the Nuclear Test Ban Treaty was signed on August 5, 1963, between the U.S., U.S.S.R., and Great Britain.

But one important date I’d like to commemorate was a bit different: eighty-seven years ago today, on August 5, 1923, my father was born, in Greensboro, North Carolina. Happy Birthday, Dad!

There’s a shorthand way of telling my father’s life-history which fits with the theme of technological advance: he graduated from college (his beloved N.C. State) as an early recipient of a B.S. degree in Mechanical Engineering; he worked for decades for a growing company interested in adopting new technologies to drive its business; and he capped his career as Corporate Vice President for Research and Development at a Fortune 300 company.

But that misses the fun he had along the way, and the close-up view he had of innovation. He was an early adopter, even before college. (I like to think I get that from him.)  So I thought I’d illustrate a couple of vignettes I’ve heard over the years of his interaction with computers along the way, simply to portray the thrust of radical change that has paced along during the course of one man’s life.

Continue reading

Follow

Get every new post delivered to your Inbox.

Join 6,654 other followers

%d bloggers like this: