So Long, Long Tail?

I’ve been known to disagree with Harvard eggheads before :-) 

 

Chris Anderson's Long TailAnd now, perhaps, another opportunity. A new Harvard Business Review article (“Should You Invest in the Long Tail?” by HBS Professor Anita Elberse) throws water on Chris Anderson’s paradigm, arguing that “hit products” are still more valuable than the conglomerated also-rans in the tail; her research is mostly in retail products. Chris has responded on his blog, sparking many comments and debate, and today the Wall Street Journal covered the back-and-forth debate.

I’m interested in the debate mostly because of the interest in the Long Tail way of thinking in some circles of the intelligence community.  I’ve written about the approach and its relevance to some intelligence issues (see “Tradecraft in the Long Tail” and “IARPA and the Virtual Long Tail“).

I’m just not certain that even a total debunking of the retail-oriented paradigm would undercut its value as applied to intelligence analysis. 

For intelligence analysts, obscure “facts” and patterns hidden snugly within the low-scale noise are all important – whether or not they gain numerative bulk in any accumulative way.  The paradoxical “unknown unknowns” are what’s being sought by dogged collection and analysis, and I’m not sure that’s analogous to Elberse’s acknowledged findings. 

Your thoughts welcome, here or by email back to me.

 
Email this post to a friend

AddThis Social Bookmark Button
 

5 Responses

  1. Lewis,

    Although I have to agree with the HBR article (I think they are writing about human nature, at least in today’s society), I also agree with you when it comes to the intelligence community. Facts, assumptions, conclusions, analysis made today will have a long tail into the future community and can have significant consequences into the future. So we need ways to capture assumptions so they can be revisited and need ways to capture uncertainties and unknowns as well. And that need is not dependent on articles in HBR or Wired.

    Cheers,
    Bob

  2. Frankly, this is fad thinking. The long tail is simply a statistical distribution. Sometimes all the data matters, sometimes it doesn’t. It would be a lot more interesting if you framed the issue in terms of “what are the statistical tools that are relevant to the intelligence community” (probably, all of them) or “what statistical tools can the research community bring to bear that may have been underutilized”.

  3. Hi Fred – the “fad thinking” comment is right, which is why I like the HBR debate. As others have pointed out, since Toffler first started predicting “de-massifying” others have been finding evidence of it, and “Long Tail” is maybe just a memorable hook. On the tools question, that’s been the frustratingly slow path; at DIA in 2005-06 we ran a structured evaluation of 25+ “predictive analysis” tools, some from Wall Street, some statistical, some traditional B.I. tools, etc. – disappointing results for intelligence analysis. I’ll do a longer post about this soon, but basically spotty inconsistent unstructured data produces crappy results. Say, are you the Nimble Books Fred Z? Big fan.

  4. Your point about “spotty inconsistent unstructured data” is right on. I’ve often been amazed by the way that the same people who are willing to pay zillions of dollars for fancy analysis tools aren’t willing to pay often more modest sums for the low-tech methods required for acquiring better data.

  5. I’m glad that Chris set the record straight on the absolute v. relative issue, and would also point out that SoundScan data don’t really capture the market for music *at all*…I think info from a source like Big Champagne would have been a much more interesting data point.

    I mentioned this in an email to Lewis already, but Clay Shirky’s article on Power Laws really gave me some more insight into how to think creatively about the “long tail”:

    http://www.shirky.com/writings/powerlaw_weblog.html

    Thanks,

    Ethan

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: