FREE BI Review Site Registration!
Sign-up today and access BI Review on the Web!

Your FREE registration entitles you to:
FREE email newsletters

FREE access to all BI Review content

FREE access to web seminars, resource portals, our white paper library and more!

   

BI and the Business Experiment

Business intelligence is supposed to be a set of approaches for finding out more about your business. One aspect of business intelligence that is often overlooked is the idea of a business experiment. Just as it sounds, the business experiment applies the scientific method to determine whether a particular business intervention is effective or not.

Companies intervene in their business operations all the time in various ways, whether it’s promoting a product or service to customers, redesigning a Web page, moving employees into new office space, or giving workers a new PC. There is usually an implicit or explicit objective to the intervention—increased sales, productivity, satisfaction, or something else. The problem is that companies rarely have enough discipline to know whether the intervention has worked or not. They don’t measure carefully, they change multiple things at once, and they have no control group. In short, they have no rigor or method to their madness. As a result, they don’t learn or gain anything from their interventions.

What to Test

Increasing numbers of companies are applying the scientific method to business intelligence. They are using “test and learn” approaches to design business experiments. The experiments are conducted with regard to a wide variety of situations, such as the following:

  • Online retailers can test a wide variety of Web page designs to see which is most effective (the design of online experiments is perhaps the easiest domain to apply the techniques); n “Bricks and mortar” retailers can test which aspects of store designs are most effective in getting customers to buy;
  • Marketers can determine what product and promotion attributes are most successful;
  • Human resource managers could test what kinds of managerial attributes yield the best results;
  • A nonprofit can determine which kinds of fundraising letters yield the highest contributions;
  • A bank can determine which balance transfer rate results in the most profitable new customers.

Capital One is, of course, the master of the last type of experiment-and probably several of the other examples too. The bank does about 60,000 experiments a year, testing virtually every aspect of credit card marketing. This company has been fantastically successful with the scientific approach, growing from obscurity to become the third largest issuer of credit cards. Now, however, the company has concluded that in order to achieve its growth objectives it needs to establish more traditional retail bank businesses. It has bought a couple of retail banks, and is now trying to determine how to apply the “test and learn” model to this new business. As one company manager put it, “It’s easier to experiment with direct mail envelopes than with branch bankers.” However, Capital One has so much experience with business experimentation that I’m confident they’ll figure it out.

How to Test

What is involved, exactly, in the capability to experiment on a broad scale? It’s sometimes not that terribly complicated from a technical or statistical standpoint. Some experiments are very simple, comparing only two different ways of intervening (this is usually called an A/B experiment). However, most experiments involve testing more factors, and this can get complex from the standpoint of designing and analyzing the experiment. Some firms employ so-called “Taguchi” methods, which were created to maximize the quality of manufactured goods, but now have been applied more broadly. Techniques such as random assignment to test groups, combining test groups into blocks, and control groups are also features of experimental design.

Experiments generate data, which can be analyzed with software to determine whether the result is statistically significant. A traditional statistical package from a vendor such as SAS or SPSS can be used to analyze the results, or firms can employ software specifically designed for “test and learn” environments. Applied Predictive Technologies, for example, offers hosted software for experiments in retail, financial services, and consumer goods marketing. Offermatica, a firm that was recently acquired by Web analytics firm Omniture, provides the capability to analyze Web and email-based experiments. These tools not only perform the necessary statistical analysis, but can also lead a non-expert through the experimental design process.

If a firm conducts a lot of experimentation, it’s going to need a disciplined approach to capturing and managing the knowledge it gains. For example, if you’re doing 60,000 experiments a year like Capital One, no one human could possibly keep all the experimental results in his or her brain. At that scale it’s even difficult to expect a user to effectively search a database of experimental results. Capital One is experimenting with a workflow system that would incorporate key learnings, and then lead an analyst through the process of designing a credit card offering. It’s difficult to build such systems, but undoubtedly worth the trouble.

Taking Ownership

Software can’t address the most important aspect of an experimental capability. That is the presence of managerial demand for an experimental approach. It’s usually a manager’s idea to intervene in some way in the business, and it must also be the manager’s idea to determine whether the intervention really works. If the demand for experimental rigor is there, the experiment can be designed and the results determined. Gary Loveman, the CEO of Harrah’s Entertainment, is famous for saying that he will fire employees for not using a control group. That kind of cultural pressure will lead to major advances in the use of experimentation. Not every business initiative can be tested, but most can. Experimentation should become the standard way an organization figures out what works in its business, and the result should be to roll out the most effective approaches on a large scale.

The best reason for experimenting is that it works. Firms that have deeply embedded experimental design into their cultures, such as Harrah’s, Capital One, and Toyota tend to be very successful. The scientific method virtually guarantees that their success isn’t accidental.


Tom Davenport is Professor and director of research, Babson Executive Education, Babson College. He can be contacted at tdavenport@babson.edu.

For more information on related topics, visit the following channels:



Industry Vendors