Big Data: Competitive Edge

Sorry that I've been away from blogging. I hope that I can stay with this for more than a couple of weeks...

-------------

I was recently in a client meeting (regional bank in Kansas City) and was asked a question by one of the business stakeholders: What do they need to do in order to get competitive advantage from all these new technologies and data sources (thanks Barb)? Here is the answer that I gave the client:

1. You need an agile data platform that doesn't hinder business growth. The amount and variety of data has grown dramatically over the past couple of years, and to be honest, we aren't sure where and when it will end. Consequently you need a data platform that is extensible and scalable for integrating data sources that we may not have even imagined yet. Bottom-line: you data platform should not limit or hinder the types of business questions your business users want to ask and answer.

2. Need a management attitude to treat data as an asset to be cultivated, not as a liability to be minimized. Not only are you looking to retain your existing data, but you are looking to strengthen that data asset through data cleansing and data enrichment activities (e.g., looking at new sources of data, deriving more metrics out of your existing data).

3. Need a management commitment to "monetize" that data asset through more accurate, more timely, more frequent, more refined decision making. Management needs to look to "weave" data and insights-driven decision-making into the very fabric of the organization. That probably has consequences on the incentive side of the operations as much as it has consequences on the technology side.

4. Link your data and analytics efforts to your key business initiatives. Spend the time to understand where and how new data sources and advanced analytics can support, or super-charge, their key business initiatives.

A lot of this is not new guidance. Heck, it's what we've been telling businesses for years now in regards to successful data warehousing and business intelligence. And it's even more appropriate now given the sudden explosion of new data sources and new enabling analytic platforms and capabilities.

Lesson #3: Data Warehouse The Best Foundation

This posting will cover Lesson #3 from my recent The Data Warehouse Institute (TDWI, August, 2008) keynote presentation on the topic "what happens when an old school data warehousing and Business Intelligence guy falls into the world of web analytics." In Lesson #3 we cover the topic of the data warehouse as the foundation for Decision Support 2.0.

Lesson #3: Data Warehouse as The Analytic Foundation

In many of the web companies that I have talked to, I'd astounded by the lack of appreciate for good data warehousing techniques. Many of these folks believe that the infinite amounts of processing power obviates the need for good data warehousing design and fundamentals. Let's review why this is a dangerous assumption, and let's learn how to leverage the 20 years of data warehousing training, techniques, methodology, and tools that we have developed and learned. Let's treat data warehousing with the respect that it deserves!

We have learned over the past twenty years that a well-designed data warehouse is still the best vehicle for aligning disparate data sources. We’ve got years of experience in building agile and extensible data warehouses. There are scores of books (from leading thinkers such as Ralph Kimball, Neil Raden, Claudia Imhoff, and Howard Dresner) and several outstanding education opportunities (at places like The Data Warehouse Institute and Kimball University). And one of the HUGE benefits of a well-designed data warehouse (read that to mean one that uses conformed dimensions and the data warehouse business architecture) is extensibility.


Data Warehouse Benefit: Extensibility

Extensibility is critical in an environment in which most web companies compete. There are new marketing tactics (rich media, smart ads, video, mobile, IPTV) being developed every day. And the analysis of the effectiveness of these new marketing tactics requires an extensible data warehouse foundation. For example, the correction attribution of multiple marketing tactics on a customer event requires the integration of data from multiple operational or serving systems.

Unfortunately, many in the web world don't think data warehouse design is all that important. Why does one need to spend time on data warehouse design when you can store ALL the data at the lowest level of granularity (the benefits of infinite amounts of storage) and have unlimited computational capacity to apply against that data (courtesy of cloud computing)?

Since we are running out of space in this issue of Decision Support 2.0, let me share in the next issue a couple of examples that can not be easily solved by infinite amounts of storage and unlimited computational power.

Until next time, Decision Support 2.0

Lesson #2 Continued: Monetizing Data

Lesson #2 from my recent The Data Warehouse Institute (TDWI, August, 2008) keynote presentation covered "monetizing data." What I want to do on this post is to expand on that thought by providing some examples of how web 2.0 companies monetize data.

Monetizing Data: Targeting


"Delivering the right message to the right person at the right time at the right price" has always been the operating mantra of the world of Marketing. The goal of web or online marketing goes one step further: to deliver a single ad that compels the consumer to take action. And the web can leverage the wealth of sub-transactional and transactional data to create finely-focused behavioral targeting categories that help achieve this monetization objective.

Targeting is probably the most common way that web companies monetize data. Some companies make money via subscriptions and others aggregate and sell their data to businesses, but the focus of this posting is to discuss monetizing data via improved targeting.

Targeting and Analytics are really just different sides of the same coin:

• Targeting tells you what message to deliver to what audience.
  
• Analytics tells you what you can do to improve your targeting effectiveness.
  

The Analytics-to-Targeting relationship is a closed loop process; improved measurability (from analytics) yields improved relevance and audience insights (for targeting) and so on. Unfortunately, many web companies really don't understand or respect this linkage. They offer very compelling targeting products, but don't spend equal effort to offer very compelling analytic products that help to measure and optimize targeting effectiveness. Sort of like having a big block '68 Camaro and forgetting to put a dashboard or steering wheel on that baby!

Until next time! Decision Support 2.0

Why the Term "Decision Support"?

Before I continue with my "Web Meets BI" learnings, let me explain why I prefer the term "Decision Support" to some of the other more fashionable terms like Business Intelligence and Performance Management.

I was introduced to Decision Support in 1984 when I worked for Metaphor Computer Systems. Metaphor developed and marketed a decision support software and hardware package that was used by Fortune 1000 companies to help them make better business decisions. And that last point ("make better business decisions") was made quite clear to me as I worked with companies such as Procter & Gamble, GD Searle, and Coors. These companies, as well as other companies, used the Metaphor system to make better business decisions. In my case, we used Metaphor to help our customers optimize marketing spend across brands and geographies, measure the effectiveness of promotional campaigns, forecast product line sales, identify trans-ship situations, make pricing decisions, and determine the financial viability of different acquisition candidates. Metaphor was truly used to help business users make better business decisions.

The Metaphor system was comprised of many important components, but the two most relevant for this discussion was the underlying data repository (a.k.a. data warehouse) and the end-user reporting and query tools (a.k.a. business intelligence). At the time, we didn't call these things "data warehouse" or "business intelligence." Those were just "capabilities" necessary to support decision making. Let me say that again: the "data warehouse" and the "business intelligence" components where there to support the decision making process. They were just components to a larger mission.

And I think that is still true today, despite what's been marketed over the past two decades. A data warehouse in of itself will not make better business decisions. A suite of business intelligence tools in of itself won't make better business decisions. But combine them together into an environment that is structured around a specific business challenge (e.g., increasing merchandising effectiveness, reducing inventory costs, increasing sales effectiveness) and with humans to help "interpret" the insights, then you have a winning decision support environment that can drive material business benefits to the organization.

My point here isn't to argue semantics, but is to point out that while data (structured in a format that is usable by humans) and tools (that are easy to use) are very important, they of themselves are not sufficient.

And that brings me back to the discussion of Decision Support 2.0.

Web 2.0 Meets BI 2.0: Learning #2

This the part two of my recent The Data Warehouse Institute (TDWI, August, 2008) keynote presentation on the topic "what happens when an old school data warehousing and Business Intelligence guy falls into the world of web analytics." I'm going to use this and the following posts to cover my 6 "learnings" from this experience. It is my belief that these learnings form a key basis for what I'm calling Decision Support 2.0.

Learning #2: Monetizing Data

Data is the currency of Web 2.0. There is no doubt about that. Heck, data is the currency of Decision Support 2.0. When your product is free, you'd better find other ways to make money. And that way is to leverage the HUGE volumes and fine granularity of the data that can be captured daily on the web. For the average data warehousing guy, these numbers are almost comical:

• 22 terabytes captured daily
• 2 petabyte data warehouse (my spellchecker doesn't even recognize petabyte)
• Hundreds of metrics, some in existence less than a year and more being created each day
  
• An almost endless wealth of dimensional data and dimensional attributes
  

However, the real gold in "them thar hills" is the sub-transaction data. It is the sub-transaction data that provides the rich insights into consumer behaviors. What I mean is that web companies know not only what consumers did on their web sites, but also know what consumers did NOT do! And it is this “did not do” data increases volume – and insights – nearly 100x!

Web companies not only know what ads you clicked on or what actions you took, but they also know what ads you did NOT respond to and what actions you did NOT take. And since most click rates are well below 2%, that means that you are learning from the other 98% of the actions tha the visitors did not take.

Let's think about what this would mean if we could do this in the physical world.

Example: A "brick and morter" retailer gains most of their customer insights from their Point of Sale (POS) data. From this POS data, retailers know what products their customers bought, what products where bought in combination (market basket analysis) and can correlate those purchases either to coupons redeemed, or ads that may have run in the local circulars. However, the retailer does not know what other products the user might have considered before putting their Cap'n Crunch into the shopping cart. They don't know that the customer might have considered the different price discounts posted on the shelves to help make their decision, or that the consumer wrestled with buying something a bit more healthy (like Smart Start). The Web companies do know this information, and that additional insight is a key part of what I'm labeling as Decision Support 2.0.

The real gold in the web world is the ability to learn both when visitors take action and when the do NOT take action. It greatly accelerates the web companies ability to ascertain and leverage for profit the behaviors of their visitors.

My next post will discuss how web companies use these Decision Support 2.0 insights to make money.

Web 2.0 Meets BI 2.0: Learning #1

I recently had the opportunity to speak at The Data Warehouse Institute (TDWI, August, 2008) on the topic of what happens when an old school data warehousing and Business Intelligence guy falls into the world of web analytics. I'm going to use this and the following posts to cover my 6 "learnings" from this experience. Plus I think that these learnings form a key basis for what I'm calling Decision Support 2.0.

Learning #1: Business Model Evolution

The old world technology business model was built around a product-based business model. That is, the product is packaged like a normal product - much like buying a laptop or even toothpaste - and sold as a product (with an on-going annual support contract). This model worked well when the industry was starting as it was a business model for which it was easier for Purchasing Departments to buy software / technology.

The problem with this business model was that an increasing percentage of total vendor resources need to be allocated to support and maintenance. This greatly inhibits investments. Also, this business model is that the rate of innovation is totally "you" dependent, which is why the Open Source movement is disrupting this business model.

The new world business model is services-based (e.g., subscriptions, etc.) as demonstrated by companies like SalesForce.com and Red Hat. Cloud Computing has the potential to really accelerate this model, and the result will be new innovation in the way of new products and services, as well as lower switching costs will enable customers to more easily adopt these new offerings. Unfortunately, the burden of business success still falls upon the users.

What we are seeing on the web is potentially a new, more disruptive business model - performance-based business model. With this model, the vendor only makes money when the customers make money, a real win-win scenario. In this model, vendors take on full responsibility for business execution success, and are rewarded accordingly! Massive amounts of data and computer power enabling new web business model.

Final note on this topic: it's not technology that's disruptive, it's the new business models that new technologies enable that are disruptive. It’s not technology that’s disruptive, it’s the new business models enabled by technology.

My next posting will cover the "secret sauce" that fuels what I am calling Decision Support 2.0

Welcome to Decision Support 2.0

Hey folks, this is my first attempt at blogging and hopefully I can do a decent job of sharing my learnings and observations about what life for the discipline of Decision Support looks like in a Web 2.0 world. Yes, I know that Decision Support is a term from the 80's (heck, we could go back another decade and try "Executive Information Systems"!), but there is so much that decision support looked to address that is still relevant today.

So let's have some fun with this and get going!

"Data, data everywhere but not a moment to think"

Watch this space!!