Mele’s Musings

Real-Time Data: Zeitgeist or Chaos?

I’m all for following the zeitgeist. In a previous posting, I talked about the power of observing in real-time, or relatively real-time, the topics of interest that products like Twitter can expose.  I also warned of the dangers we can run into when we over-focus on this data.

Recently, Yahoo disclosed that it is teaming with OneRiot to run tests on real-time search.  And this makes sense to test.  As a new and powerful data source, services like Twitter can expose the latest information and the latest trends.  It can perhaps provide more timely or more relevant information in this matter.

Maybe.

Link to image

We have to be careful with our head-over-heels romance with real-time sources of information.  While real-time data like search queries gives us the latest information, it doesn’t necessarily mean it is providing the best or most meaningful data.  And we can become actually blinded to the truth by the immediacy of data.

Several years ago when I first became an account manager at good old Avenue A, I was confronted with the problem of having ready access to real-time data first hand.  Digital marketing was still relatively new, and one of the most powerful arguments we made to marketers at the time was that they would have access in real-time to how their campaigns were performing.  We called it data crack.  And it was true.  We had clients who literally watched the data minute by minute, waiting to pounce the second the CPA dipped below goal.  It was nerve-racking.  And idiotic.

I can’t even recall the name of the client at anymore, but I distinctly remember working on my first real campaign, and receiving a frantic client phone call.  The poor man was cursing and spitting on the other end of the phone, insisting that we take his campaign down immediately because it was failing, and we were idiots who were wasting his money.  And this was a mere two hours after the campaign had launched.  Two hours!  His reaction, while severe, was symptomatic of a larger problem - when there is no filter to data or information, we can make incorrect conclusions, but confuse them with truth because, well, there is data.  And data feels like truth.

The truth is that we need filters.  Google is a massive filter.  In fact, it is so popular because it is arguably the best filter.  Google does not represent or deliver links to the whole internet.  It delivers a filtered view of the content of the internet based on your query.  And the filter is not based just on the word you type, but on a subset of content related to the word you typed determined on an algorithm that scores links.  This is why typing the same query into Google, Bing, Yahoo, or Ask can bring up different results.  They put different filters on the data.

The risk we run in seeking out real-time data is that we lose the filter.  Or, maybe it’s more accurate to say that we rely on a very specific and unscientific filter - one based not on a algorithm engineered to deliver on relevance - but rather one based on simply the latest postings of the subset of web users who micro-blog on Twitter, or Facebook, or some other real-time data source.

We should be careful, then, of fooling ourselves that unfiltered is better.  Sometimes it is, but oftentimes filtering makes it possible for us to decipher and decide.  It is why trending over time is a more valuable business tool than what happened in the last hour.

I’m not trying to say that real-time data is not valuable.  It can be extremely powerful.  But it can also be an useless mess.  Can you imagine trying to follow the real-time flow of postings on Twitter and actually gaining any knowledge or insight?  Can’t be done.

Twitter and the like can be powerful tools to get some sense of what people are feeling right now.  As I type this, one of the top Twitter trends is on the DC Sniper who was executed this evening.  Clearly a topic that people have opinions on that they want to voice.  Also in the top trends right now?  #bestfeeling #worstfeeling and #overyou.  Hardly insightful.

So, while we can appreciate the value of tapping into the zeitgeist, we also have to be careful that we don’t overvalue services like Twitter.  Twitter does not represent the world, or even the US, but a specific subset of the population.  We can do ourselves a disservice by putting too much emphasis on this sentiment because it is data we can collect or see.  But, it can be misleading, too.

The same argument can be made about crowdsourcing in general.  More on this later, but I am concerned about the increasing love affair with it.  While crowdsourcing feels good, it rarely is the actual “wisdom of the crowds” that we want it to be - we can only get that view if we get most everyone’s thoughts and input.  Rather, what we usually get is the “wisdom of the overactive or overly-concerned.”  Yes, it is a sort of wisdom of the crowd, but it is the wisdom of a particular small slice of the crowd.

Tweet This Post  Buzz This Post  Delicious This Post  Digg This Post  Reddit This Post  Stumble This Post