FREE DM Review Site Registration!
Sign-up today and access DM Review on the Web!

Your FREE registration entitles you to:
FREE email newsletters

FREE access to all DM Review content

FREE access to web seminars, resource portals, our white paper library and more!

   
BI Review content and features are now in DMReview.com!

One brand, one Web site! DM Review is now the home of all the content you're used to at BIReview.com and much more. If you are registered at BIReview.com, you're already registered at DM Review. If not, take just a moment to sign up for all the free services we have for you at the new DMReview.com.

Shadow IT: A Lesson for BI

What are the most frequently required features of BI tools? In any evaluation, ease of use, performance, security, scalability, report writing and distribution features, OLAP capabilities, graphics and even visualization are usually at the top of the list. But ask the people who will actually use the tools in their daily work and the number one request is, "Can it export data to Excel?" With over 150 million licensed copies of Excel in use, it is the undisputed leader for analytical work. Why does Excel persist even with the explosive proliferation of BI tools in recent years? More importantly, what does it say about BI in general?

The term for the unauthorized development of IT systems is "Shadow IT," but it turns out that the lion's share of Shadow IT development takes place on the desktop, using personal productivity tools like Excel and Access. The IT community and press universally consider Shadow IT a problem, but on closer inspection, it may not be a problem at all - it may be the result of a much larger problem.

The most often cited problems with Shadow IT are that it is impossible to certify the accuracy of the outputs and that it consumes a huge amount of time that could be better spent doing the actual functions assigned. While it is inexpensive at the beginning, maintenance of multi-user spreadsheet applications is an escalating and serious problem.

Current opinion is that Shadow IT results from of a number of factors:

  • Budget cuts for IT projects cause delays and cancellations in systems needed by people to do their work, driving them to find their own solutions
  • Lack of IT/business alignment, an endemic problem that affects more than just analytical systems
  • User Independence, meaning that people instinctively do things on their own, unencumbered by the timing and capabilities of the IT organization
  • Timing, because IT initiatives take so much longer to start and complete

There are elements of truth in all of these reasons, but our research shows there is one overarching reason people rely on their spreadsheets that predates and will outlast all of the other reasons. Spreadsheets simply perform better than BI. This is not a new phenomenon. In 1984, people with PC's turned to Lotus 1,2,3 to perform as much of their reporting and analysis as they could with the limited platforms they had. This begs an important question: Why would someone take on the task of building and maintaining applications in a spreadsheet when someone else was willing to shoulder the responsibility for them?

The answer is pretty obvious: in the long run it is simply easier. But in the long, long run, does it still make sense? Even the people with the spreadsheets ask that question now and the answer has grown more complicated. We have found that the motivations for building applications in spreadsheets (and other personal productivity tools) are not exactly the same as the four reasons mentioned above.

Today, these events are:

  • Output-based, not model-based: Most but not all BI tools are read-only and based on a pre-defined model, whether a semantically mapped schema from a relational database or a multidimensional cube or database. Spreadsheets start with the output. In other words, a developer starts by building a model or models of what he/she wants to see as the result. For people who are not data modelers, this is infinitely easier. (The technical term for this is declarative modeling, a feature missing in almost every BI tool, especially at the business user level.)
  • Expressive: The level of skill needed to develop an application is more easily attained without BI tools. The ability to conceptualize an idea and render it in a spreadsheet is perhaps its greatest advantage. BI works through layers of abstraction that are too difficult to manipulate.
  • Iterative: Spreadsheet development is not formal, and versions of models are developed over time. The results of changes are immediate without the need for version control and escalation through testing and production. This is also a drawback in the long run, but from the developer's perspective, expedience is a benefit
  • Disposable: Because the effort needed to develop a spreadsheet is low, it is possible and economical to develop applications for minimal or one-time use.
  • Easily communicated: Spreadsheets in the mail are a standard part of organizational communication today because the software exists on every desktop and the level of skill needed to at least open and view one is universal.
  • Subversive: Perhaps most important, spreadsheets provide the means to communicate complex ideas and information and bypass the usual lines of authority. It is perfectly reasonable for people to communicate ideas through the "back channel," floating ideas and suggestions with others without formal consent or time-consuming meetings.

Each of these points has its drawbacks, but that doesn't invalidate them. What is missing today, except in a few enlightened vendors, is that BI is only a partial solution for providing information and tools to knowledge workers. Spreadsheets are also a partial solution. Ideally, a marriage between the two could cover the entire landscape. Spreadsheets could technically present the ideal primary interface for analytical work if they were able to retain their ephemeral qualities of expressiveness and collaboration while being backed up with full data quality, integration, security and management of the BI environment. This is easier said than done. Supplying an add-in to a spreadsheet simply transfers all of the BI tool's shortcomings into a smaller window. Providing a spreadsheet interface to the BI solution eliminates most of the positive motivations for using a spreadsheet in the first place. What is needed is a seamless conjoining of the two metaphors that allows people to build iterative, results-based models that can be freely communicated and modified, but spins off the administrative, security and data access responsibilities to a capable application that can insure the integrity of result and maintain the whole process over time.

Shadow IT is, undeniably, a massive problem for organizations today, especially from the perspectives of compliance, transparency and security, but looking for more IT solutions to replace spreadsheets is not likely to succeed. Paraphrasing Dr. Strangelove, it's time to learn to love the spreadsheet.


Neil Raden is the founder of Hired Brains, Inc., http://www.hiredbrains.com. Hired Brains provides consulting, systems integration and implementation services in business intelligence, decision automation and business process integration for clients worldwide. Hired Brains Research provides consulting, market research, product marketing and advisory services to the software ndustry. Based in Santa Barbara, CA, Raden is an active consultant and widely published author and speaker. He welcomes your comments at nraden@hiredbrains.com. 

For more information on related topics, visit the following channels:



Industry Vendors