The secret data warehouse tax
There’s a dirty little secret in the traditional transactional data warehouse world: they know how to tax you as effectively as any profligate government.
You invest in your data warehouse. You invest time. You invest money. You dedicate personnel and resources to your data warehouse. You pay the vendor to advise you on how to use their product. You stick to what they advise. You expect a return on your investment in the form of a working, scalable data warehouse. But you may not realize that as quickly as you can get a return, your investment is being clawed right back again.
Like some government department with a line in emergency-fund relief — TARP, FEMA, anyone that can claim to need a lot and need it now, — the data warehouse tax chews through your money, time, and resources, and what’s worse, you’re not even sure where this went. Everything looks pretty much the same as it did before, only this time you have less money in your pocket and fewer results on your desk.
The performance tax is probably the most insidious. Join indexes, materialized views, summary tables, cubes, tuning and re-tuning are all sneaky ways in which vendors of traditional transactional data warehouses try to give their outdated systems a fighting chance in the face of modern demands. But they end up sitting bloated on top of your data warehouse, dramatically increasing your storage costs, and threatening your ability to meet service level agreements and stay competitive, while ultimately still failing to give you the performance you were promised.
The query tax is most noticeable to end users. They simply can’t ask what they want, when they want it, yet they keep spending money. As data volumes grow, more and more of the traditional transactional data warehouse is locked away from users, who are required to make do with canned reports and lowest-common-denominator queries. Does your Chief Information Officer know what your Chief Marketing Officer is going to ask tomorrow? Of course not, and because the system isn’t prepared to deal with any question, it either needs to be time-consumingly re-indexed or re-tuned, or (more and more frequently) the question simply can’t be asked, resulting in the loss of potentially critical business intelligence.
The opportunity tax results from all of the above: because the data warehouse can’t perform and business critical queries can’t be asked as needed, ad hoc and on-demand, regardless of expectation or complexity, vital opportunities are missed, service level agreements are threatened, and risk is incurred.
This is before we start to look at the new data coming into the warehouse: weblogs, application logs, social media and so on. All these bring new taxes.
The data warehouse tax costs you time, it costs you money, and it costs you business.
We no longer just have “big data,” we now have Extreme Data: extreme in terms of variety of data (traditional, social media data, RFID data, etc.), which leads to even more volume of data, and challenges our ability to maintain the velocity of data, moving it quickly from warehouse to end user community.
Extreme data demands extreme action, not extreme taxes.
Imagine your salary slip was tax free. What could you do with all the extra money? Now think about your data warehouse, tax free. Think about everything you can do with all that extra time, money, and business. You deserve it. Your business deserves it. Demand that your vendor provide you with a tax free data warehouse today.