How Do We Manage Our Metrics? — Measures of Success
Below is an excerpt from Chapter 1 of the book Measures of Success: React Less, Lead Better, Improve More, by Mark Graban.
Google Analytics sends me an email each month that shows a comparison of two data points related to my blog, www.LeanBlog.org. It presents the number of users with a comparison to the previous month, as shown below:
I can log in to Google Analytics to find a month that shows a nice percentage increase in a metric, like the number of page views. I can illustrate this below in the type of table we often see in management reports or slide decks (these reports often place the newest numbers to the left, which seems confusing):
I might say, “Hooray, an increase of almost 40% from last month! Up 50% since last year!” I’m rounding up, of course, to paint an even better picture of progress.
How do I know the number hadn’t decreased by a similar amount the month before? Comparing any two data points doesn’t provide enough information for me to understand my system and to properly evaluate trends in its performance.
KEY POINT #2: Two data points are not a trend.
Thankfully, I don’t have investors or an executive whom I’m trying to convince that 40% improvement is going to happen every month. I’d only be fooling myself to think so, and doing that doesn’t help improve my business. Ries always warns against creating “success theater.”
One form of “success theater” is using or displaying metrics in a way that makes performance look better than it is. As Ries wrote, “Energy invested in success theater is energy that could have been used to build a sustainable business.” Another form of success theater would be unethical tactics that artificially boost the number of page loads (an example of the distortions that Joiner warned about).
Comparing two data points means we are missing a lot of context, including the other months’ data and the trends we might see by looking at more data.
As Wheeler wrote in Understanding Variation:
KEY POINT #3: “No data have meaning apart from their context.”
If comparing two numbers isn’t very helpful, organizations often try to provide additional context by displaying tables with many numbers on bulletin boards or electronic dashboards.
It’s very difficult for people to see trends in a table of numbers, the way some of my blog data is shown below. This style of presenting a metric over time is sometimes called a “scorecard.” Or, it’s often referred to as a “Bowling Chart” because it’s like the grid you’d use to keep score when going bowling, with numbers going from left to right.
Is my blog traffic increasing or decreasing? It’s hard to tell. The best we can usually do is to compare a point to the previous year or to the same month in the previous year. We might look at the first few data points and the last few — but there’s a risk that we might draw the wrong conclusion from this visualization of the data.
A graph or a chart is a much more effective way of making sense of data and metrics. A chart is pictorial, and our human brain processes images much better than lists of numbers. Watch a cable business channel and you’ll see they usually show a graph of a company’s stock performance over time instead of a table of numbers showing the stock price on different days. Unfortunately, their scroll at the bottom of the screen shows a constant stream of two-data-point comparisons of a stock’s price compared to yesterday.
There’s no technological excuse for bombarding people with tables of numbers instead of presenting a chart. Charts are easy to produce with modern technology. We can even use very old “technology” to draw them by hand.
When given two data points, such as “blog traffic is higher this month,” drawing a Run Chart to visualize that limited data set isn’t very helpful, either, as we see below:
We’re missing important context that tells us if the difference between those numbers is routine or exceptional. How much do the numbers normally change from month to month? All we know from this treatment of the data is that the April 2017 number is higher than it was in March 2017.
Even saying that April is 39% higher than March doesn’t provide much context, since we don’t know if it normally fluctuates that much month to month. It’s possible that a large percentage change is not statistically significant; it’s also conceivable that a small percentage change would be significant in a different metric and situation.
KEY POINT #4: A chart will always tell us more than a list of numbers.
A Run Chart tells us much more than two numbers or a table of many numbers, as seen below:
As you learn and practice this methodology, I hope you’ll challenge others when they present two data points or a simple before-and-after comparison. You can ask them to “plot the dots,” as some professionals from the National Health Service in England say, using the Twitter hashtag #plotthedots.
What do you see? What does the chart tell you?
The Run Chart tells the honest story that my blog traffic fluctuates from month to month. The Bowling Chart tries to tell us the same thing, but it’s much easier to hear what Deming and Wheeler call “the voice of the process” when we allow a chart to speak to us.
Alternatively, the “voice of the customer” tells us what is required, such as a specification or target. We hope the voice of the process tells us that our system is capable of meeting those needs all of the time. If not, we need to improve.
The same system, with the same people doing the same work in the same situation, will not always produce the exact same results each day, each week, or each month. This is a fact of life — there’s always variation in a metric. As Deming said, “Life is variation.”
KEY POINT #5: The job of management is not just to look backward but also to look forward and predict, if possible, what is likely to occur.
Metrics are most often used to look backward and to make evaluations about past performance. But, we also need to prepare for the future of our organization and its performance.
As the late MIT professor Myron Tribus said:
“Managing a company by means of the monthly report is like trying to drive a car by watching the yellow line in the rear-view mirror.”
Glancing back at the Run Chart for my blog traffic data, it looks like the April data point is indeed a significant increase from the previous months. Before April, the metric seemed to be fluctuating around an average. Is this April increase something exceptional? Does it deserve our attention or an investigation? Do we have to guess? How would we know? As you will see in Chapter 2, the Process Behavior Chart points the way.
When we react less and lead better, we can improve more. Read more in the sample PDF chapter of Measures of Success, the book preview on Amazon… or buy the book! :-) The book teaches you how to distinguish signal from noise in your metrics… a very helpful skill to have in this age of “big data!”
