January 01, 2007

Meet the Metrics: The New Math of Measuring Effectiveness

by Patrick Marren

One of the more interesting things I have found in my life and business experience is that numerical measures, or “metrics,” as we consultants are forced by law to call them, start to lose their usefulness on the day they are dreamt up.

In baseball, for many decades, batting average was seen as the ultimate measure for batters, and earned run average (ERA) served the same purpose for pitchers. For decades, no one complained, and awards were given to the top performers in both categories. The best pitcher had the lowest ERA; the best hitter had the highest batting average.

Then a statistical revolution began in the 1980s, and it was found that there were other measures that might better describe a player’s usefulness to his team. Such stats as on-base percentage (OBP) and “runs created” attempted to penetrate to the core of what made a batter valuable to a team; and pitchers were now evaluated by such arcana as “Equivalent ERA” (ballpark-neutral ERA).

Some real progress has resulted. The Boston Red Sox hired the Babe Ruth of baseball-statistics nerds, Bill James, in 2003 to help them analyze things, and presto! The Red Sox won the World Series the next year for the first time in 86 years. Although some general managers eschew the stats in favor of their “feel” for the game, many more have been quietly convinced that putting together a lineup without reference to on-base percentage will not only guarantee failure, but also lots of angry fans.

But each year, it seems, another new statistic arises that purports to better express the value of ballplayers’ contributions. In recent years, “win shares” for batters and “support-neutral won-loss record” for pitchers have claimed to shed more light on the subject.

Searching for the Perfect “Metric”

A similar process has taken place in business over the past few decades. Net income has been supplanted by income per share, then by net present value, and then by Economic Value Added (EVA). Other statisticians focused on market share, margins, customer satisfaction, quality, cost-cutting, return on equity, on capital, on net assets employed, etc., etc.

As stated above, however, no particular statistical measure of the effectiveness of a business is perfect, and its relevance tends to wane as time passes. There are many possible reasons for this, but almost all of them arise from one or both of two sources: (1) the imperfect correlation of any number with the health of the business in question, and (2) the inevitability of change in the business environment.

To begin with, no number perfectly represents the actual value, prospects, or health of any business – not even its share value. Markets can be systematically under- or over-valued, as many an investor can ruefully attest. But a company can profit for some time by guiding its operations by the light of a numerical measure. Monsanto, back in the 1980s and 1990s, under the leadership of CEO Dick Mahoney, set a goal of 20% return on equity, and this simple goal served to organize the efforts of Monsanto executives for his term.

Now, Return on Equity did not perfectly represent all that was good, true, and beautiful in Monsanto’s world. In fact, pursuit of that lofty numerical goal caused some dislocations and imbalances within the company – as resolute pursuit of any particular goal, in baseball, in business, or even in private life will often do – and the goal had to be revised after Mahoney left the company.

But that does not mean that the choice of the goal was a failure. Choosing to pursue any goal in business strategy is always a tradeoff, a forsaking of other goals that might offer different benefits. The point is to choose the goal that is optimal, for the company’s strengths and weaknesses, its competitive milieu, and for the time period in question. A clear, crisp goal is also a way of efficiently refocusing the management of a company toward certain objectives. When all managers know that their efforts will be judged against the ROE goal, not a lot more needs to be communicated – and this is not a trivial thing. In addition, when other market actors – customers, suppliers, and shareholders – understand your goals, a great deal of uncertainty can be eliminated, and this also is far from a trivial advantage.

But as time goes on, the statistic by which you are guiding your company is inevitably going to lose relevance. Financial imbalances may be encouraged by the numerical nature of the goal; in the extreme case, an Enron-like catastrophe can follow. Competitors will also attempt to take advantage of your clear focus, perhaps by maneuvering you away from real market opportunities by lowering their short-term attractiveness, or, conversely, maneuvering you into less attractive situations. Shareholders may be wooed away from your pet number and toward a new measure of “business win shares” that revises downward the price they are willing to pay for a 20% ROE, for example.

What is true of numerical goals also holds true for a company’s strategy. Winning companies not only exhibit strategic consistency, such as a commitment to a 20% ROE; they also exhibit the strategic flexibility to review and change that strategy more often than is comfortable. This is because newly instituted corporate strategies also begin immediately to diverge from the optimal, based on numerous uncontrollable variables: the reactions of current competitors; changes in the supply chain; changes in customer desires; new substitute products or services; unexpected entrants; and other changes in market conditions.

Balancing Consistency and Adaptability

The best corporate strategy must therefore be the one that is most resilient to change, that anticipates in detail the vicissitudes of the marketplace, that looks out far enough in time so that what can be anticipated is anticipated. Developing such a strategy requires that management invests in developing a mental model of the true scale of potential change; that it have the creativity to imagine market conditions that are radically different from the present; and conversely, and perhaps most important, that it be prepared to abandon the current comfortable modes of operation – and the current comforting measures of success – when market conditions suddenly change.

Perhaps the biggest statistical change in baseball over the past two decades has been the increase in runs scored, and a concomitant increase in pitchers’ earned run averages. A Rip Van Winkle manager from 1980 would regard the league-leading team ERA of 2006 as disastrous, and a league-lagging team home run total as quite healthy. He might disdain the 2004 Red Sox’ reliance on the home run as fundamentally unsound, and their pitching as awful. He would be wrong.

Times change, and so do measures of effectiveness.

(But they still ought to get rid of the designated hitter.)

*       *      *

Full disclosure: Patrick Marren was sucked into the awful quagmire of Chicago Cub fandom as an innocent child growing up a mile from Wrigley Field. He cannot understand why his team has not seemed to use statistics (or logic of any kind, for that matter) to assemble its lineup for the past 99 years. Mr. Marren is nevertheless happy to note that pitchers and catchers are scheduled to report to spring training in less than 60 days.

Thoughts?