The Wrong Metric – When The Best Looks Like the Worst

Written by Howard Fenton
Senior Technology Consultant
NAPL
There are three general categories of in-plants: universities, government and commercial. Then there are smaller categories such as local school districts. It is rare to get to work with someone as innovative as Charlie Holden at the Houston Independent School District and learn more about her insourcing efforts. Therefore, I was looking forward to learning more about a few school districts in Washington State.
In this assignment, I was asked to evaluate someone else’s consulting report for several nearby school districts. The issue was not a lack of measurements, but instead a dependence on one measurement, and a misleading metric. The report focused on page volume and staffing as the metric and ratio to benchmark performance.
In 1996, a new business strategy was described in the book The Balanced Scorecard: Translating Strategy into Action by Robert S. Kaplin and David P. Norton. The central metaphor in the book is the notion of a passenger visiting the cockpit of an airliner horrified at seeing that it contains only a single instrument such as airspeed or a fuel gauge. The authors develop this metaphor, noting that in the Industrial Age using only one gauge to manage business, such as the quarterly report, is just as crazy.
The metric the other consulting firm chose was page volume and the ratio chosen was pages per person. Ratios are considered good comparative data tools. But any benchmarks require a deeper understanding of what the data represents.
An underlying assumption in benchmarking performance is that you are comparing the same things. According to Best Practices, LLC, “benchmarking is a performance measurement tool used in conjunction with improvement initiatives; it measures comparative operating performance of companies and identifies the best practices.”
In other words, benchmarking performance requires an apples-to-apples comparison. An article entitled “Manufacturing Benchmarking: An Apples-to-Apples Comparison” from the University of Tennessee Center for Industrial Studies states, “Before you begin benchmarking your operations, you must consider several aspects. First, make sure the group your company is being compared to is similar to your operation in size, market, types of equipment, and other metrics that assure a reasonable apples-to-apples comparison.”
Therefore, I visited the in-plant printing sites the other consulting firm deemed the “best” and the “worst” based on page volume and staffing. First I visited one of the best school districts to learn more about their operation. They were a well run organization with well maintained, old, offset printing equipment which allowed them to produce 16 million impressions a year with 3 staff running on one shift. The volume based on staff was high and it was clear why they were considered the best.
Next I visited the school deemed the “worst” by the other consulting firm, which had digital printing equipment, large format capabilities and scanners in all the schools. According to the analysis, they had 9 staff and produced 20 million impressions. Two issues became immediately clear. The analysis miscalculated the page count by 10 million because they looked at paper instead of calculating double for 2-sided impressions.
Second, the kind of work they did was different because it was highly customized. If they wanted pages 2-4 and 17-21 out of a course pack, they could get that. They provided different services including design services and large format poster work.
It was not an apples-to-apples comparison. While the “best” had the highest ratio of pages/staff, the “worst” provided a better service, and many more jobs per year. If you changed the metric from number of pages to jobs per year, the worst would be the best and the best would be the worst. And some could argue that more customized printing is a more effective teaching tool.
Two important take home messages. First, the wrong metric can lead to the wrong conclusion. Second, offering shorter run lengths and more customization is not “worst,” in fact it can be the  “best.”
Interested in similar topics? You may want to check out:

• Performance Based Metrics
• The Pros and Cons of the Sales Per Employee Ratio
• How do you measure the productivity of your digital press? (Part 1)
• How do you measure the productivity of your digital press? (Part 2)
• How do you measure the productivity of your digital press? (Part 3)

­­__
Howard Fenton is a Senior Technology Consultant at NAPL. Howie advises commercial printers, in-plants, and manufacturers on workflow management, operations, digital services, and customer research. He is a paid contributor to this blog.