Treat Your Organization as a Prototype: The Essence of Evidence-Based Management

Pfeffer, Jeffrey

When looking for ways to enhance performance, Jeffrey Pfeffer and Robert Sutton counsel gather the facts, look for patterns, and experiment to make things better. Then keep repeating the process. Amplifying their recommendations with engaging anecdotes, they are skeptical of the "latest" breakthroughs and best practices, urging managers, instead, to study the evidence continuously as groundwork for a steady stream of incremental advances.

For the past five years, we have been studying how leaders might learn to practice evidence-based management-how they can find, face, and act on the best facts. We've found that many companies suffer because their leaders continue to do what they've always done-adopt best practices that are actually bad for their companies and then refuse to face the facts about their failures. But we've also found that the most successful leaders think hard about what they do and why they do it, which enables their companies to do a better job of evaluating and applying business knowledge. Table 1 summarizes six key guidelines they follow-from treating old ideas as if they are old ideas, to basing management practices on the best evidence rather than on what is in vogue. These leaders don't follow gurus or seek magical management breakthroughs. They are interested in what is true and not necessarily in what is new. And perhaps most important of all, they ignore conventional wisdom and taken-for-granted assumptions about what works. Instead, they seek out and act on the best evidence.

Bucking conventional wisdom

The gaming, or casino entertainment, industry is rife with conventional wisdom-some of it so commonly believed that it is known outside the industry, as well. One deeply held belief is that the key to success is the ability to attract the high rollers, rich and famous people who drop lots of money at the tables and roulette wheels. Another belief is that casinos must offer discounted hotel rooms and meals, or even give away lodging to entice people into the casino, where they will spend money gambling, at restaurants, and on entertainment. Other beliefs include the notion that building family-friendly places with rides, sort of mini-Disneylands, are useful in getting customers, particularly families, to gaming venues. Or that building lavish-and expensive-facilities that look like Venice, Paris, or the New York skyline is the best way to get people to choose your facility over others; that the "hold" (money the casino retains from slot machines) cannot be changed or people won't play your machines; that media advertising on radio and television is among the best ways to build customer traffic and revenue.

When Gary Loveman was appointed chief operating officer of Harrah's in 1998, taking a leave from his position as an associate professor at Harvard Business School, he knew little about the details of casino operations, interior design, or architecture. He had consulted for Harrah's and had studied the retail store industry. Loveman arrived with a professor's commitment to rigorous analysis and making fact-based decisions. He soon made this such a part of the company's culture that, as he commented when we talked to him, there were three ways to get fired at Harrah's: steal, harass women, or institute a program or policy without first running an experiment. Casinos produce lots of data on things like revenues, occupancy, profitability, and staff turnover. Loveman was determined to use that data and to collect more information by constantly running small experiments to uncover facts that would help the company make more money.

Loveman and his colleagues soon discovered that much of the conventional wisdom in the industry was wrong, and they changed company practice to reflect those discoveries. Rather than relying on extensive media advertising, Harrah's uses direct mail-promotions aimed at targeted customers to tempt them to spend more of their gaming dollars at a Harrah's casino and to get them to return if they haven't visited in a while. Harrah's learned that its most profitable customers were locals, often older retired or semiretired people who visited the casino frequently to play for entertainment. These people weren't as interested in discounted rooms as they were in meals and complimentary chips. In one experiment, Harrah's offered a control group the typical promotional package worth $125 (a free room, two steak dinners, and $30 worth of free chips); customers in the experimental group were offered just $60 worth of free chips. The $60 offer generated more gambling revenue than the $125 offer.

Harrah's figured out that families with small children-a target audience for many competitors-generally have little discretionary time or money, so courting them was unprofitable. The company also discovered that spending money on employee selection and retention, including giving people realistic job previews, enhancing training, and bolstering the quality of front-line supervision, reduced turnover and produced more committed employees. Harrah's was able to reduce staff turnover by almost 50 percent as a result. Loveman and his colleagues also reasoned, on the basis of academic research, that more experienced, committed, and better-led employees would improve customer service, which in turn would bolster guest satisfaction and, ultimately, willingness to return. This attention to employees, plus Harrah's investment in data warehousing and analytics, which permitted the company to track and analyze guest behavior, had a far bigger payoff than just throwing money at facilities.

And, with $50 billion going through Harrah's slot machines each year, Loveman insisted on running experiments to see if holds could be varied-according to the machine's location, for instance-without affecting play. Conventional industry wisdom was that you could not vary the payouts at all, but Loveman didn't accept it. As he asked a group of Stanford students: How can price (which is what the hold really is) be so inelastic when one woman can buy a black dress for $1,000 at a designer store and another woman can buy a similar dress for $100 at Target, and both be equally satisfied with their purchases? Harrah's discovered that you could increase the hold, generating additional money that fell straight to the bottom line.

Know what you don't know

Gary Loveman and other leaders who are able to face and act on the hard facts-Xerox's Ann Mulchay, George Zimmer from The Men's Wearhouse, and Cisco's John Chambers come to mind-all have a way of thinking that is more important than any particular guideline or method. They have realized that true wisdom is reflected in the attitude people have toward what they know, not in how much or how little they know. It's not a new idea; in fact, one of its first manifestations goes all the way back to the philosopher Plato. Describing Socrates' visit to "a man with a high reputation for wisdom," Plato quotes Socrates as concluding:

I formed the impression that although in many people's opinion, and especially his own, he appeared to be wise, in fact, he was not.... I reflected as I walked away, Well, I certainly am wiser than this man. It is only too likely that neither of us has any knowledge to boast of, but he thinks that he knows something which he does not know, whereas I am quite conscious of my ignorance. At any rate it seems that I am wiser than him to this small extent, that I do not think that I know what I do not know.

The power of Plato's ancient insight persists because wisdom (or the lack of it) shapes how people think, feel, and act in so many ways. Psychologists, including John Meacham and Robert Sternberg, have studied the nuances of wisdom, with special attention to the differences between being wise and being smart. One of their most important insights is that, as Plato's quote suggests, wisdom means "knowing what you know and knowing what you don't know." Such an attitude strikes a balance between arrogance (assuming you know more than you do) and insecurity (believing that you know too little to act). This enables people to act on their present knowledge even while doubting what they know. It means they can do things now, as well as keep learning along the way.

Leaders who practice the attitude of wisdom think and act as if their organization is an unfinished prototype, not something that "ain't broke, so let's not fix it." They don't see this prototype as something they might ruin with dangerous new ideas, or that is too much of a mess to fix, or that is impossible to change because there will be too much resistance. You can see this attitude in action at Harrah's, where management constantly looks for patterns in the data and runs experiments to make things better and better.

We also saw the power of the attitude of wisdom in the mid 1990s when Sutton and colleague Andrew Hargadon went to a companywide meeting at IDEO, the renowned innovation company, as part of an intensive 18-month study of IDEO's work practices and culture. Then-CEO (and now chairman) David Kelley announced a major reorganization at this meeting. The company, especially its Palo Alto headquarters, had grown so fast that it was becoming increasingly difficult to manage a culture of innovation in what amounted to a 150-person group of professionals. Sorting out who would staff the 50 or so different design projects IDEO undertook each year was one of the worst headaches. Kelley acknowledged that IDEO had become unwieldy; he then introduced five leaders, each charged with heading a new studio. Each leader made a pitch to his or her colleagues inviting them to "join my studio." It was unlike any reorganization we've ever seen. The people affected were invited to choose the group they wanted to join rather than being assigned. Kelley reminded them of IDEO's guiding philosophy: Enlightened trial and error outperforms the planning of flawless intellects. In other words, he encouraged the IDEO designers to see this new structural arrangement as a changeable prototype, just like the products, services, and experiences IDEO designs for clients. To reinforce this message, Kelley hung hundreds of little Experiment signs throughout the company and-in a move that shocked all of us who knew him-shaved off his trademark Groucho Marx mustache for the meeting, explaining, "The changes we are trying are just like shaving off my mustache-they are temporary and reversible experiments."

Amazon.com also has a management philosophy that emphasizes constant experimentation and learning from results. As CEO Jeff Bezos explained during a speech at the Stanford Business School, people throughout Amazon realize that their job is to "maximize invention per unit of time" and "minimize the cost of experiments with small teams." These small teams have few constraints, but all of them are expected to track their experiments carefully. Amazon uses "the Internet to collect hard facts about what's working and what's not," not only to make better decisions, but because "that prevents in-fighting and also eradicates hierarchical rank."

Or take home shopping network QVC, which is a bit smaller than Amazon but twice as profitable, ranking third among US broadcasters in revenue. QVC products are hawked live on TV and are available for immediate purchase. They include virtually everything, from recipe books to jewelry to clothes and Dell computers. QVC had more than 7 million customers in 2004, with 93 percent of the company's revenue coming from repeat buyers. The company's products are selected for sale through a process of constant experimentation, punctuated by evidencebased reflection about which things work and which don't. While a product is being sold on the air, producers are making minute-by-minute adjustments in camera angles, lighting, and the host's dialogue, persisting with changes that seem to increase sales and dropping changes that seem to dampen sales. More reflection and learning occurs at daily postmortems, where QVC employees watch videos of advertisements to try and figure out why some ads succeeded and others failed the day before. For example, at one meeting described in Business 2.0, they discussed the prior day's least successful product and whether it failed because of a poor pitch by the TV host or because the product itself was flawed. In this case, they decided the product was the problem, and so it was liquidated on the QVC website (which has, by the way, grown to become the sixth-largest US Internet retailer).

Work in progress

QVC's decisions are data-driven, but the company acts on the best data producers can get, up to the very moment of sale as they experiment with camera angles and dialogue. One key implication of treating an organization as an unfinished prototype is that on many occasions, trying something half-baked based on the best data you can get and doing so quickly is better than trying something polished, with every tiny detail considered and analyzed at the cost of much delay. After all, by the time you have completed an in-depth study, the issue may have passed, all the executives may have been fired for inaction, and the company may be dead. An organizational change agent once complained to us that too many academics forget that it is often better to get a little evidence now than a massive amount much later. He teased us, "You professors seem to forget that, when you go to the doctor for a blood test, they don't drain your whole body and look at every cell. A little sample is enough."

He had a good point. A manager's job is to act on the best available evidence, and keep updating. You will never have all the evidence. In any field, whether airline safety, medicine, the military, or a private company, conditions and circumstances change and new knowledge is acquired all the time. Leaders deal with uncertainty-that's what they have to do. Doctors prescribe treatments and medicines not based on perfect information or perfect knowledge but on the best evidence available. That evidence-and thus the best treatment-changes over time. In much the same way, evidence-based management means acting on what you know at a moment in time, based on the best available data you have, even as you try to create the conditions for learning more. This means seeing the truth as a moving target-in other words, seeing both your organization and your knowledge about how to manage it as unfinished but useful prototypes.

Reprint #6173SUT10

Jeffrey Pfeffer, Professor, Stanford University

Robert Sutton, Professor, Stanford Engineering School

by Jeffrey Pfeffer and Robert I. Sutton, Stanford University1

1. This essay is based on Jeffrey Pfeffer and Robert Button's new book Hard Facts, Dangerous Half-Truths, and Total Nonsense: Profiting from Evidence-Based Management (Harvard Business School Press, 2006).

Jeffrey Pfeffer is the Thomas D. Dee II Professor of Organizational Behavior at the Graduate School of Business, Stanford University, where he has taught since 1979. He is the author or co-author of eleven books, including The Human Equation: Building Profits by Putting People First, Managing with Power: Politics and Influence in Organizations, The Knowing-Doing Gap: How Smart Companies Turn Knowledge into Action and Hidden Value: How Great Companies Achieve Extraordinary Results with Ordinary People, as well as more than 110 articles and book chapters. His most recent book, Hard Facts, Dangerous Half-Truths, and Total Nonsense: Profiting from Evidence-Based Management, co-authored with Robert Sutton, was published by Harvard Business School Press in spring 2006. Pfeffer received his BS and MS degrees from Carnegie-Mellon University and his PhD from Stanford. He began his career at the business school at the University of Illinois and then taught at the University of California, Berkeley, and he has been a visiting professor at the Harvard Business School.

Pfeffer currently serves on the board of directors of Audible Magic, SonoSite (SONO), and Unicru and writes a monthly column on management issues, "The Human Factor" for the 650,000 circulation business magazine, Business 2.0. He has presented seminars in 28 countries, as well as consulting and providing executive education for numerous companies, associations, and universities in the United States.

Robert Sutton is professor of management science and engineering in the Stanford Engineering School, where he is co-director of the Center for Work, Technology, and Organization, an active member of the Stanford Technology Ventures Program, and a cofounder of the new Hasso Planter Institute of Design. He is also an IDEO Fellow. Sutton studies the links between managerial knowledge and organizational action, innovation, and performance, and he has published more than 100 articles in academic and applied publications. He is the author of Weird Ideas that Work: 11? Practices for Promoting, Managing, and Sustaining Innovation (Free Press, 2002); the coauthor (with Jeffrey Pfeffer) of The Knowing-Doing Gap: How Smart Firms Turn Knowledge into Action (Harvard Business School Press, 2000), and the author of Hard Facts, Dangerous Half-Truths, and Total Nonsense: Profiting from Evidence-Based Management (Harvard Business School Press, 2006). Sutton's next book, The No Asshole Rule: Building a Civilized Workplace and Surviving One that Isn't, will be published by Warner in early 2007. His email is Robert.sutton@stanford.edu; his blog is www.bobsutton.net.

Copyright Design Management Institute Summer 2006
Provided by ProQuest Information and Learning Company. All rights Reserved