20 Lessons learned in 20 years of Business Intelligence and Data Warehousing (Part I)


Agile has to be implemented right to gain benefits

Agile has been an eye opener. It is not just about User stories, sprints, daily scrums and Kanban or sprint boards. It is really about common sense and continuous improvement. Also, you have to be careful with trying to fit Agile into a traditional hole. There is not enough benefit in taking your existing technology team and distributing the roles to them. The Scrum Master and Product Owner should be roles on their own – a servant leader with no direct authority over the team, and a decision maker on product functionality but not a technology lead. Also, small, self sufficient teams that are empowered to act on their own in the sprint definition and implementation.

The Traditional Data Warehouse architecture still lives

The Traditional Data Warehouse is not dead. It is still of value as a piece of the overall puzzle. An ODS layer with history and detail, topped by a Data Mart with Aggregate metrics, still handles a significant amount of analytics. A relational database is still useful. A commercial platform such as Oracle for RDBMS can still handle significant loads and processing, albeit in a structured manner.

Big Data did not disrupt the Traditional Data Warehouse, it complemented it

When Big Data first became an emerging technology, there was a line of thinking that it would perhaps replace or severly disrupt the TDW. Instead it complemented it, handling specific challenges that the TDW could not – namely ad-hoc and quick analysis of very large, unstructured data. The data warehouse still addresses the need for supporting a set of very repeatable reporting and analytics. The Big Data solution truly stands next to it, mashing data from unstructured sources, quickly.

You don’t necessarily need a ETL tool for large data processing

This has been another eye opener. Several years ago, I argued with a group lead who had been advised that, given a properly optimized database, simply SQL can be used for all transformations. I challenged that notion. Now, I work in an environment where all the heavy lifting is done on the database. A ETL tool is still used as a wrapper, but not entirely. I feel like I someone an apology.

The traditional ad-hoc analyst role will likely die

The role that involved clicking around a report with drill-across and drill-through to find the nugget is likely going to be replaced with a machine learning algorithm.

To be continued…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s