Requirements for the next generation of DWBI technology directed towards product vendors


This post was prompted by a TDWI Linked In discussion on whether 20th century architectures are applicable to the 21st century. I have always felt that new technologies tend to be technology driven and not entirely business problem driven. Such new technologies come in waves. The current waves are around columnar databases and in-memory analytics. However, they address niche problems and don’t look at the entire problem holistically. For example, columnar database vendors are entirely focused on the performance aspect of ad-hoc queries. Vendors are so focused on that performance aspect that they have provided scripting interfaces for actually designing and implementing the solution. Thus, there are no interfaces or easy means to design performance management systems. In fact, such vendors are promoting an ELT approach, which sounds great but doesn’t make much sense in practice. So, while providing us a service, they are only doing so in a narrow way, and there is still a lot of work to be done in terms of putting a solution together. Likewise, in-memory analytics are focused on one piece of the problem, which is sub-second analytics for power users. There are many other consumer roles in the enterprise that need to be addressed, and such solutions don’t put much effort into the design aspect either, or focus on how the in-memory solution would integrate with other required technologies.  While both technologies are great advances with respect to what we had in the 20th century, they don’t look at the entire problem we DWBI practitioners face – making DWBI pervasive in the organization.

Piecemeal technologies need a lot of effort in putting them together into an entire performance management solution. This continues to be a barrier to more widespread adoption of DWBI in organizations and an improvement of their reputation. This thought prompted me to write down a set of requirements for the next generation of DWBI technology. My hope is that a vendor will step up and tackle this holistically, and not address it in a piecemeal manner.  Along with each requirement, a list of features for a better product has been requested of product vendors.

Here’s to a better future for DWBI!

Ease of Implementation End to End (DWBI In the Box)

Ease of implementation can make a huge difference to data warehousing in the next generation. DWBI efforts tend to be massive and complicated. One of the reasons is the difficulty and complexity of such engagements. You need a DW Architect, a BI Architect, a team of ETL developers and BI developers. Partly, this is because of the inherent complexity in ETL and BI coding and DW architecture on relational databases. While recent advances in database technologies have simplified the problem somewhat by eliminating the need for summary tables, the lack of focus on the usability and design aspects of performance management solutions as a whole will mean that no relief is offered towards easing the complexity of DWBI solutions. They will continue to be complex, costly, and change-unfriendly.

Functionality Requested from Product Vendors –

  • End to End DWBI in the box – Put everything together in one box such that we don’t have to keep putting together solutions with many different components. A DWBI appliance, if you will. This includes data structures (logical data models), ETL, and BI in all its forms – dashboards, static reports, interactive applications – and OLAP, but with a business focus rather than a strict dimensional focus. See my point on supporting business analytic constructs below.
  • Easier ETL/BI development – Higher level ETL/BI interfaces that are more business rules driven. The ability to take some low level building blocks that capture common transformations and put them together into more complex transformations that can be shared across applications. The standardization of some complex transformations.
  • Focus on Usability in Design – High level design interfaces that allow us to build performance management systems with all their layers – metrics and hierarchies – bottom up or top down. The ability to connect to source system data structures easily.

Support of Objective Oriented (Requirements Driven) Design

We need to be able to demonstrate that a performance management system’s design is closely linked to requirements – objectives, areas of analysis, business questions, or other types of requirements.

Functionality Requested from Product Vendors –

  • Provide the ability to capture requirements as part of the box, and the ability to link the requirements to the design of the performance management system. This will help demonstrate to the business users the clear links between requirements and design.

Adaptability to Evolving Business Needs

Even after an organization has put in the huge initial investment into its DWBI solution, it needs to continue investing in adapting it to changing business needs. Not only is this a huge cost issue, but a huge Time to Deployment issue as well. One major reason is the disintegrated nature of a piecemeal solution. The same metric, dimensional attribute or hierarchy exists in a number of different systems in your entire DWBI environment. Most of the time, the number of points of existence aren’t even tracked or clearly understood. The item exists in the data model, in the ETL layer and in the BI layer. When such a change is proposed, usually it takes the implementation team upfront analysis time to even track the multiple points of change. The effort then needs to be scoped and implemented. A simple change can not only have significant cost, but can be very time consuming as well, because of the ripple effect of the change across the multiple systems it is implemented in. This can be hugely frustrating to the organization, which expects a quick turnaround to evolving analytic needs. Another big reason is the complexity of ETL and BI development. This has already been addressed in the Ease of Implementation End to End point above.

Functionality Requested from Product Vendors –

  • One point of definition of a metric or attribute – By putting End to End DWBI in the box, you are already a big step towards addressing this. Now, ensure that a metric or attribute is stored in one place only. This means that any change will automatically ripple across the entire system. Of course, you need to ensure that development and production systems are handled as separate systems, and adequate notification can be provided for a change. Those are also required.
  • Changes trickle across the End to End solution
  • Versions of the system are tracked such that an older version can be reverted to.

Support of Business Analytic Constructs rather than Dimensional Constructs

Strictly Dimensional  constructs do a poor job of representing business analytic entities – metrics, and the context around them. The item “Products” can represent not only something that is sold and earns revenue  (an item in the level of a hierarchy of a Products & Services dimension, linked to the Revenue metric) but also an Organizational Unit (an item in the Organization hierarchy – the Products division of the Organization, linked to the Expense metric). Dimensions “merge” in business analytics to become such “things” that can go across multiple hierarchies – the Products & Services hierarchy and the Organizational hierarchy, in the above example. Unfortunately, BI products still retain a strictly dimensional flavor that makes it harder to fit it to such “things”. Also, they separate the “metadata” from the “data”. This makes it much harder to create these “things” (e.g., “Products” that go across dimensions named “Products & Services” and “Organization”) in a design – if that makes any sense.

Furthermore, not many BI products allow you to create a system of metrics that typically occur in a performance management system. Metrics are not only Revenue and Cost. They can be “Product Revenue” and “Services Revenue”. They can even be “Q1 Revenue” or “Last Quarter Revenue”. Some of this is supported in some niche environments, but not holistically.

Functionality Requested from Product Vendors –

  • Support for business analytic constructs rather than strictly dimensional constructs. This means supporting the design of systems of metrics that “merge” with the context,  support for the context itself (the “data” as opposed to the “metadata”) in design, and the support for contexts (e.g., “Products”) that can “belong to” multiple hierachies, and is seamlessly shared across.

Performance Performance Performance

Yes, performance, scalability, sub-second response, reliability, and all those good things are still important, but pay attention to the whole problem, not just a niche area.

Bottomline

DWBI continues to be a costly, time-consuming, ungainly beast that is resistant to change. Hearing out these ideas will make a big difference to the community and the profession in the long run. Hope someone is listening.

4 Comments

  1. Couldn’t agree more Shubho. DWBI still has a long way to go for it to become more cohesive as a solution. It’s still pretty much in it’s infancy and is not much more than a bunch of tools used to do some reports in most organisations.

    I think one of the problems is the lack of Enteprise Architecture involvement as well as how the vendors act. I say this because EAs are the ones typically looking at MDM initiatives for example, and these should address the operational master data first of all, but you often get an MDM solution implemented that overlaps horribly with what’s being done in your DWBI system, rather than them being complementary. I also don’t often see the split in an EA team between Data Management and BI. These to me are very different focus areas, and should have their own strategy – DM organises the data ALL data (from Operational domain and Analytical domain), and BI should be a ‘window’ onto this data to make it all available. The difference I’m calling out here is the DW bit of DWBI is only normally addressing the analytic domain.

    In summary, while I agree that vendors need to step up to the mark when thinking about what organisations need, they can only really do this if the market is receptive to it, and there is established demand for fully integrated suites. Many organisations can’t even get their head round the whole data as a corporate asset thing – it’s just too big…. so companies also need to get their act together in terms of how they want to ‘organise’ ALL data in their company.

    1. Phil,
      While I generally agree with you, I still think there is huge value (and a resultant huge market) in at least making the DWBI aspect of the solution easier. EAs tend to focus on the data side of things alone and that too the data in operational systems. DWBI & A (A is for Analytics) has a slightly different focus – performance management systems, the metrics used to measure performance, and the context around it (hierarchies and context attributes). Even if this aspect was solved by making it easier, and MDM was solved separately and parallely, that is adequate. I am speaking from a product and technology perspective, not from a solutions perspective – I think your comments extend to the reality in organizations, which is valid also.
      Thanks for your comments,
      Shubho

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s