Coming to Grips with the Big Data Era

The concept of the “data moat” is not new. But the growth of some companies’ data assets and the speed and dynamism with which certain market leaders are moving to exploit them are stunning. There is a rapid gap opening up in our industry – one that the leaders are keen to exploit but also one that many laggards don’t even know about.

Consider Amazon, which was an early mover with its recommendation engine. Starting over a decade ago, the company began to draw on its enormous database of millions of people and their buying habits to gain a competitive advantage initially to sell more books and DVDs. With the rapidly increasing power of its infrastructure, and the plummeting costs to store and process ever-larger amounts of data, Amazon’s technical and business leaders have transformed that customer database and the analytical engine that runs on top of it into a powerhouse that seamlessly executes processes and drives conversions that few other businesses can even imagine.

To further disrupt industries, Amazon in June announced it would leverage this powerful data moat to enter the advertising business, exploiting from its own platform what it (and only it) knows about its own customers. Working through partners, Amazon will make inventory and data sets available to advertisers interested in reaching consumers with targeted messages, while they have their wallets open. Amazon is setting out to do nothing other than completely disrupt and reinvent point-of-sale advertising, doing things in retargeting and cross-targeting that have been previously unheard of. It would be impossible to attempt this without a massive and continually updating database of 60 million customers, sharing their PII credit card information, addresses, and more. This requires a data infrastructure that captures, manages and analyzes multiple terabytes of data every hour.

Apple, Amazon, Google and Facebook sit at the intersection of the once-clear lines between retailers, media and technology companies. These companies are blowing through the barriers that once separated entire industries, supply chains, and purchase patterns. They do this on the basis of data. And as these companies generate increasing amounts of data, they also encourage other, smaller companies to find new ways to use their own data. Amazon is the best example with its Amazon Web Services cloud business, the backbone for multiple other companies in this space, and Amazon’s next monumentally disruptive billion-dollar business. (In fact, in a recent earnings release, Amazon revealed that for the first time, the infrastructure footprint of its core retailing and streaming operations had been surpassed by the infrastructure footprint of Amazon Web services.)

The explosion of companies offering social analytics tools and services in the last three years is another example of how companies are using the data moat for competitive advantage. So many articles have been written about Facebook’s impressive display advertising business. But the mainstream press is just now waking up to the sleeping giant of Facebook’s data business, which enables it to learn about – and reach — every Facebook user anywhere on the Web. LinkedIn is now poised to completely overturn the multibillion-dollar job-search and recruiting market. Box.net and Dropbox are disrupting SharePoint, Oracle and EMC infrastructure products.

Digital companies of all sizes are generating increasing amounts of information about what we like, share, buy, store, copy, stream and record with every purchase, search, click, upload, file share, status update and check-in. These businesses know our aspirations, our desire for a new job or a new partner; they run anomaly detection over our resume edits and note when we start listening to Tom Waits again.

The business plan is simple: Companies with profound revenue traction will focus on capturing and preserving the data asset from early in their corporate infancy. This data approach is part of the DNA. Think about the value of this data if it can be captured invisibly and deployed to enhance customer experience. Advertising seems like an obvious — and disappointing — path for some of these businesses, a point made by Fred Wilson recently when he identified this class of business as needing to build a native monetization system to exploit the data asset.

What’s next? Fifteen years ago, Jeff Bezos was selling books on static Web pages, and nobody could imagine the Amazon preference engine. Welcome to the Big Data Age and the strategic management of tons of information. If you’re a digital business today but not prepared to manage big data, you are already behind.