3 Reasons to Be a Metrics Skeptic

Are metrics imperfect? Can they do as much harm as good? Are they easily influenced by ideology and dogma?

The answer to all of these questions, of course, is “of course.” A recent New York Times piece by Anand Giridharadas does a nice job of pointing this out. There is an inherent danger in handing over our understanding of the world to data. At the same time, metrics would not be experiencing their current heyday if not for the clear benefits.

This post covers three inherent challenges of metrics and reasons to proceed with caution. Tomorrow’s post will discuss how to make them work better for your social enterprise.

1. Metrics don’t just supply numbers – they tell a compelling story. That story creates new conclusions and assumptions that impact future decisions. The problem is that anything not measured – perhaps 99% of “stuff” – often fails to become part of the narrative. It is this very “selective memory” of metrics that makes them so potentially misleading . Depending on what’s measured and how, as well as what’s not measured, the stories that metrics tell can be as much fiction as fact.

For example, the story of global economic success relies largely on measures of GDP. Less integral to the narrative have been measures of income distribution and almost non-existent have been measures of environmental impact. On the other end of the continuum, the former King of Bhutan decided in the 1970s to measure the country’s progress in terms of Gross National Happiness. This measure inevitably weaves a different narrative, but is similarly driven by a particular world view and prone to manipulation.

Another great example of the misleading nature of metrics and how they are susceptible to ideology plays out in the book “Moneyball” by Michael Lewis. Based on the story of Billy Beane and the Oakland A’s, Moneyball tackles the MLB’s religious belief in its age-old success measures like Runs Batted In (RBIs) and its resistance to recruiting players based on new, arguably better metrics like On-Base Percentage.

2. Metrics – even bad metrics – become “sticky.” Metrics are more meaningful when they become benchmarks. A number is just a number until it can be compared to something – generally how someone else did against the same metric, or your performance on that metric in previous years. In this way, much like computer software, metrics are both the victims and beneficiaries of the equivalent of network effects. The longer they are around and more widespread they become, the more invaluable they seem.  This means that, just like the program/overhead ratio, old ubiquitous metrics die hard, even those that have clearly lost their relevance.

3. Metrics tend to force arbitrary categorization and thresholds. Have you ever received a B in a class where you had an 89%, while your neighbor with a 90% received an A? What impact did that have on your GPA? Did you ever wonder who set that threshold?

The recent reports and recommendations related to cervical and breast cancer prevention also illustrate this principle brilliantly. Evidence-based medicine holds the potential to improve care while also reducing health care costs. However, no medical procedures are 100% bad or good, effective or ineffective. Rather, medicine operates in the realm of greyness. Procedures work very well for some and less well for others. Treatments may extend life (what is that worth?) but at the risk of decreased quality of life (which carries what cost?). If doctors begin recommending and insurance companies begin covering only treatments that are “effective” in terms of outcomes or “worthwhile” from a cost-benefit standpoint, someone will need to create the values and thresholds that allow us to make these somewhat arbitrary distinctions.