Playing To Win
Reliability versus Validity in Strategy
Without Balance Your Organization Won’t Persist
In the development of strategy, there is a hidden tension between two largely implicit forces that must be balanced productively to produce strategy that helps your organization persist. To provide guidance on that delicate balance, I decided to make the 24th Playing to Win/Practitioner Insights piece on Reliability versus Validity in Strategy. (Links for the rest of the PTW/PI series can be found here.)
A Fundamental Tension
When people work together to make decisions and solve problems, there is a hidden tension between two opposing forces that influence from the background. One force is reliability, which is the desire for a dependable, consistent, and replicable outcome. The Stanford-Binet IQ (SBIQ) test exemplifies reliability. If you take it numerous times from childhood to adulthood, you will produce a nearly identical score each time, a feature that has made it the dominant IQ test in the world. The other force is validity, which is the desire for an outcome that meets the desired objective.
Most people would not view the desired objective of an IQ test to be only the generation of a consistent number that we can assign as your intelligence. Instead, they would want the test to tell you something about your future prospects owing to the measure of your intelligence. It turns out that SBIQ does a pretty dreadful job of that. Only 30% of anything you accomplish in the rest of your life can be correlated with your SBIQ score. Hence while it is highly reliable, it has poor validity. Daniel Goleman famously provided an alternative, emotional intelligence, or EQ, which he argues correlates much better with future outcomes. However, EQ measurement hasn’t displaced IQ measurement, in large part because EQ measurement is not as reliable as IQ measurement.
Reliability and validity are produced by very different ways of thinking.
Reliability is generated by only considering variables that are objectively quantifiable without judgment — so that there is no bias involved. That is why SBIQ defines intelligence as one’s ability to solve little logical puzzles for which there is one objectively correct answer, which helps generate its impressive test-retest reliability. However, success in life requires more than solving little logical puzzles. And that is why Goleman includes one’s ability to interact successfully with others and demonstrate empathy in doing so as part of his EQ evaluation procedure. He broadens the number of variables he utilizes to include those that require judgment to assess and with judgment comes the potential for bias. In this way, the features that drive more reliability drive less validity and vice versa, hence the degree to which the tension is fundamental.
These contrasting thinking approaches play out in business on a daily basis. Think of the classic approaches of General Electric versus Apple. At GE, the numbers ruled. Businesses were cost-reduced year after year to eke out more profits. Six Sigma black belts roamed the company optimizing the here and now. For decades, GE produced consistent, replicable, reliable outcomes, hitting its quarterly guidance to the penny. Meanwhile, at Apple Steve Jobs focused on outcomes that he desperately wanted to generate: an exquisite user experience, devotion from his users, to dent the universe. The former obeyed the principles of reliability; the latter those of validity.
The Tension at Play in Strategy
This tension plays out in the development of strategy. When those involved push for strategy based on the objective analysis of past quantitative data, whether they know it or not, they are pushing for a reliable outcome, as did successive CEOs at GE. When others may push to imagine future possibilities and consider attributes of the decision that can’t be quantified and that require judgment, they are pushing for a valid outcome, as did Jobs at Apple.
Within a given corporation, these camps tend to argue back and forth without knowing the true roots of their disagreement. The reliability-focused strategists think their validity-focused colleagues are being dangerously cavalier. The validity-focused strategists think that their reliability-focused colleagues are narrowminded and miss the bigger picture.
The Limits to Reliability
The tricky thing about reliability is that it is reliable until it isn’t. The pursuit of consistent replicable outcomes does not guarantee the production thereof. If you base your decisions on the objective crunching of past data, you will produce reliable results only as long as the context stays fixed and doesn’t change in any meaningful way. For example, if for the past six quarters, we have been able to increase earnings by cutting product costs each quarter, our reliability-driven analysis may suggest that another round of product cost reduction will produce another quarterly earnings increase. But that may be the straw that breaks the camel’s back: the reduction in product quality that creates a broadscale customer revolt or spurs competitive entry under the price umbrella we have set. If either is the case, a reliability-driven approach will fail to produce a reliable result.
At GE, the reliability driven management approach eventually hit the wall. It took a long time for it to do so, but as is often the case, when it crashed, it crashed precipitously, and GE may never recover.
The Limits of Validity
That having been said, the opposite is not a panacea. An entirely validity-driven approach will embrace the extremes of uncertainty and not take advantage of the data that can be accessed and used. It is arguable that Jobs was unceremoniously fired from his own company because in his first stint as CEO he was unbalanced in favor of validity. He believed that he would be substantiated by future events and arguably paid little attention to the warnings of those concerned about at least a base level of reliability — financial planning, supply chain management, etc. The extreme attention to validity without reliability balance took Apple to the brink of insolvency.
I would argue that when Jobs was rehired, he came back to Apple as a leader who demonstrated more balance between validity and reliability. He still showed the same dedication to creating a future that did not then exist. But he paid more attention to robust supply chains, operational thoroughness, and financial planning than in his first stint as CEO. And it was in this more balanced era that he created the vast majority of Apple’s value.
The Implications for Strategy
The key is to drive for balance between reliability and validity in the development of strategy. It is the only way that an organization can ensure persistence. Too much reliability and it grinds to a halt. Too much validity and it goes off the rails. Balance means crunching the data that you can gather on the variables that lend themselves to objective quantification. But it also means considering variables that can’t be quantified, objectively or otherwise, and applying your judgment to them. It means avoiding bias to the extent possible but recognizing that the possibility of bias cannot be allowed to paralyze you.
While danger lurks on either side of the divide, I find that more often reliability dominates validity in strategy development in the modern large corporation. Ironically, the initial strategies of entrepreneurial enterprises tend to be highly validity oriented. An entrepreneur believes in a future that will be different — and better — than the past, often based on judgment, not objective measures, and then sets out to make that vision come true. If it does come true, the enterprise prospers, often goes public, and acquires a board of directors. As it gets bigger and more formally managed, there tends to be more calls for reliability — for meeting quarterly earnings targets reliably, for formal approval of projects based on financial projections, etc. Validity oriented thinking, which got the business started, gets increasingly discouraged or suppressed entirely.
Because the world has a nasty habit of shifting, changing, and adjusting, a heavy bias toward reliability in development of strategy will leave the corporation vulnerable. The changing context will make the reliability-driven strategy a bad match for the future. But the confidence that comes from all that objective analysis will typically cause the corporation to be slow to recognize that its strategy is flawed and make productive changes. Meanwhile, competitors will have an open field for creating the future.
Practitioner Insights
Reliability and validity are the hidden forces that shape the way we think and decide. Everyone has their own comfort zone, whether more reliability-oriented or more validity-oriented. Recognize your own predilection. If you are reliability-oriented, be extra careful not to dismiss your validity-oriented colleagues. They aren’t dangerous; they can help you strive for better outcomes. If you are validity-oriented, don’t dismiss your reliability-oriented colleagues. They aren’t dull and boring; they can protect your flank.
Throughout the process of creating strategy, keep asking yourself whether you are considering both reliability and validity in your thinking. Pull out the chart above and check whether your thinking draws from both columns or just one. If it is just one, find a colleague who can help stretch your thinking to incorporate the other column. It will help you generate a more robust strategy and enable your organization to persist.
