Agile has to be implemented right to gain benefits
Agile has been an eye opener. It is not just about User stories, sprints, daily scrums and Kanban or sprint boards. It is really about common sense and continuous improvement. Also, you have to be careful with trying to fit Agile into a traditional hole. There is not enough benefit in taking your existing technology team and distributing the roles to them. The Scrum Master and Product Owner should be roles on their own – a servant leader with no direct authority over the team, and a decision maker on product functionality but not a technology lead. Also, small, self sufficient teams that are empowered to act on their own in the sprint definition and implementation.
The Traditional Data Warehouse architecture still lives
The Traditional Data Warehouse is not dead. It is still of value as a piece of the overall puzzle. An ODS layer with history and detail, topped by a Data Mart with Aggregate metrics, still handles a significant amount of analytics. A relational database is still useful. A commercial platform such as Oracle for RDBMS can still handle significant loads and processing, albeit in a structured manner.
Big Data did not disrupt the Traditional Data Warehouse, it complemented it
When Big Data first became an emerging technology, there was a line of thinking that it would perhaps replace or severly disrupt the TDW. Instead it complemented it, handling specific challenges that the TDW could not – namely ad-hoc and quick analysis of very large, unstructured data. The data warehouse still addresses the need for supporting a set of very repeatable reporting and analytics. The Big Data solution truly stands next to it, mashing data from unstructured sources, quickly.
You don’t necessarily need a ETL tool for large data processing
This has been another eye opener. Several years ago, I argued with a group lead who had been advised that, given a properly optimized database, simply SQL can be used for all transformations. I challenged that notion. Now, I work in an environment where all the heavy lifting is done on the database. A ETL tool is still used as a wrapper, but not entirely. I feel like I someone an apology.
The traditional ad-hoc analyst role will likely die
The role that involved clicking around a report with drill-across and drill-through to find the nugget is likely going to be replaced with a machine learning algorithm.
To be continued…