Web 2.0 Meets BI 2.0: Learning #2

This the part two of my recent The Data Warehouse Institute (TDWI, August, 2008) keynote presentation on the topic "what happens when an old school data warehousing and Business Intelligence guy falls into the world of web analytics." I'm going to use this and the following posts to cover my 6 "learnings" from this experience. It is my belief that these learnings form a key basis for what I'm calling Decision Support 2.0.

Learning #2: Monetizing Data

Data is the currency of Web 2.0. There is no doubt about that. Heck, data is the currency of Decision Support 2.0. When your product is free, you'd better find other ways to make money. And that way is to leverage the HUGE volumes and fine granularity of the data that can be captured daily on the web. For the average data warehousing guy, these numbers are almost comical:

• 22 terabytes captured daily
• 2 petabyte data warehouse (my spellchecker doesn't even recognize petabyte)
• Hundreds of metrics, some in existence less than a year and more being created each day
  
• An almost endless wealth of dimensional data and dimensional attributes
  

However, the real gold in "them thar hills" is the sub-transaction data. It is the sub-transaction data that provides the rich insights into consumer behaviors. What I mean is that web companies know not only what consumers did on their web sites, but also know what consumers did NOT do! And it is this “did not do” data increases volume – and insights – nearly 100x!

Web companies not only know what ads you clicked on or what actions you took, but they also know what ads you did NOT respond to and what actions you did NOT take. And since most click rates are well below 2%, that means that you are learning from the other 98% of the actions tha the visitors did not take.

Let's think about what this would mean if we could do this in the physical world.

Example: A "brick and morter" retailer gains most of their customer insights from their Point of Sale (POS) data. From this POS data, retailers know what products their customers bought, what products where bought in combination (market basket analysis) and can correlate those purchases either to coupons redeemed, or ads that may have run in the local circulars. However, the retailer does not know what other products the user might have considered before putting their Cap'n Crunch into the shopping cart. They don't know that the customer might have considered the different price discounts posted on the shelves to help make their decision, or that the consumer wrestled with buying something a bit more healthy (like Smart Start). The Web companies do know this information, and that additional insight is a key part of what I'm labeling as Decision Support 2.0.

The real gold in the web world is the ability to learn both when visitors take action and when the do NOT take action. It greatly accelerates the web companies ability to ascertain and leverage for profit the behaviors of their visitors.

My next post will discuss how web companies use these Decision Support 2.0 insights to make money.