Goodhart’s Law

Written by

Sergio Caredda

Posted on

September 2, 2024

Edited on

September 2, 2024

The Four Forms of Goodhart’s Law.

Regressive

When selecting for a proxy measure, you select not only for the true goal, but also for the difference between the proxy and the goal. — Scott Garrabrant, Goodhart Taxonomy, (2017).

The first form happens because the measurement you are using as a proxy for a goal is imperfectly correlated with that goal. Let’s imagine you are building a hiring process and you discover that IQ is correlated to job performance at around 0.6. You start administering IQ tests, and initially your company improves its recruiting results. What happens, however, is that then you tend to optimize your entire recruiting process for IQ, and IQ alone. Over time you start realizing, however, that people with highest IQ tend to perfom worse than people that have an IQ just above average. What happened?

0.6 is a good correlation to social science standards, but it also means that there are other factors to consider. Which are, however, cut off from the process you have designed. This effect is called the tails comes apart, and is ofter also described as a tendency towards mediocrity, because of the focus on a sole or a reduced number of measures, which means you are loosing outliers.

This form of Goodhart’s law “is impossible to avoid because nearly every measurement you can think of is an imperfect reflection of the true thing you want to measure” (Chin, 2020). The only option being very intentional in selecting the types of measures you need, an maybe pairing them logically.

Extremal

Worlds in which the proxy takes an extreme value may be very different from the ordinary worlds in which the correlation between the proxy and the goal was observed. — Scott Garrabrant, Goodhart Taxonomy, (2017).

This forms happens when you pick a measure that is correlated to your goal in normal situations. Then you optimize for this measure, but at the extreme ends of that measure, the correlation with your goals breaks down. Manheim and Garrabrant give an example on the relationship that humans have with sugars (Manheim and Garrabrant, 2018). We did evolve to like sugars, because these were correlated with calories in our ancestrale environment. Today however this form of optimization leads us to drink sodas and eat unhealthy snacks full of sugars… often leading to health conditions, diabetes and obesity.

Another way of seeing this is that we can’t easily measure what we want directly and need to use a proxy. But for extreme values, the proxy might not be related to the target anymore. Let’s assume I want to find good basketball players. We establish that basketball skills are related to height of the player. However, finding the tallest person on earth is a terrible strategy to find the best basketball player.

Causal

When there is a non-causal correlation between the proxy and the goal, intervening on the proxy may fail to intervene on the goal. — Scott Garrabrant, Goodhart Taxonomy, (2017).

This form happens based on a very common bias, when we mix up correlation with causation. To follow up on the example of basketball, if I learn that good basketball players are tall, and I want to become tall, starting to play basketball is not the right strategy to become taller.

The idea is that we think that a measure produces an outcome, whereas in reality they are simply correlated, but it’s another factor that causes the correlation of the two. If you optimize the measure, you don’t get the result you want.

Adversarial

When you optimize for a proxy, you provide an incentive for adversaries to correlate their goal with your proxy, thus destroying the correlation with your goal. — Scott Garrabrant, Goodhart Taxonomy, (2017).

This form happens when an agent may optimize for a metric in a way that defeats the metric’s goal (the so called Cobra effect), or the agent may choose to optimize for a measure in a way that reduces that measure’s predictive effect.

This specific form focuses on the behaviour of agents trying to influence the results to their own interest, and can be, as you can imagine, very dangerous when linked to incentives in an organization.

Key Implications of Goodhart’s Law:

Whatever form Goodhart’s Law takes, there are multiple implications that should be considered.

1. Distortion of Behavior:

When individuals or organizations are incentivized to meet specific targets, they may focus narrowly on achieving those targets, potentially through gaming the system or prioritizing short-term gains over long-term success. This can lead to unintended consequences, where the quality or value of the overall outcome is compromised.

Example: In a sales organization, if the target is purely based on the number of units sold, salespeople might push products that are easier to sell in high quantities, rather than focusing on customer needs or selling more profitable or sustainable products.

2. Erosion of Original Goals:

Goodhart’s Law highlights the risk that the original purpose of a metric can be lost when it becomes a target. The focus shifts from achieving the underlying objective (e.g., improving customer satisfaction) to simply hitting the number associated with it, which can lead to actions that undermine the original goal.

Example: In healthcare, if the target is to reduce patient wait times, hospitals might achieve this by rushing consultations rather than ensuring quality care, thus compromising patient outcomes.

3. Over-Optimization:

Organizations may over-optimize for specific metrics at the expense of other important factors. This can create imbalances and lead to a situation where the organization is “doing well on paper” but failing in critical areas that are not being measured.

Example: A company that focuses exclusively on cutting costs as a target might achieve that goal but at the cost of employee morale, product quality, or customer satisfaction.

4. Metric Manipulation:

Individuals might manipulate or artificially inflate the numbers to meet the targets, rather than genuinely improving performance. This can involve unethical practices, data manipulation, or simply focusing on “easy wins” that don’t contribute to real progress.

Example: In academic settings, if the target is to increase the number of published papers, researchers might focus on producing a higher volume of low-quality papers rather than contributing meaningful research.

5. Complex Systems and Simplification:

In complex systems, simplifying performance measurement to a single or a few metrics can be problematic. It often ignores the complexity and interdependencies within the system, leading to misleading conclusions or suboptimal decisions.

Example: In software development, if the sole focus is on lines of code written (as a measure of productivity), developers might write more code than necessary, leading to bloated and inefficient software.

Application and Countermeasures

Goodhart’s Law serves as a caution against the over-reliance on quantitative metrics as the sole indicators of success. In practice, it suggests the need for a more nuanced approach to measurement and target setting, where metrics are used as guides rather than strict goals. It also highlights the importance of maintaining a broader perspective on organizational objectives, ensuring that targets align with long-term goals and do not incentivize counterproductive behavior.

I would like to stress, however, that in many ways this law acts at the level of individual bias of managers and team members. People might not be aware (or intentionally act on it).

This is why, even if there are multiple strategies and countermesures that can help balancing the quality of measuring systems, none of these alone is guaranteeing the avoidance of issues when measures are at stake. Here a few examples of some potential countermeasures.

Balanced Scorecards: Use a balanced set of metrics that reflect multiple aspects of performance rather than relying on a single measure.
Qualitative Assessment: Combine quantitative metrics with qualitative evaluations to get a fuller picture of performance.
Regular Review and Adjustment: Periodically review and adjust targets to ensure they remain relevant and aligned with organizational goals.
Focus on Outcomes, Not Just Outputs: Emphasize the importance of achieving meaningful outcomes rather than just meeting numerical targets.

One Final Warning: Compliance Measures

There is a final aspect that we have to be aware: there are many examples of Goodhart’s law linked to the application of measures linked to compliance metrics. An example of this is the structure of P&L established based on accounting principles, which often leads to suboptimal choices (we optimize the P&L instead of the performance of the organisation). Which is why companies should carefully establish management accounting practices.

Another example is the usage of “benchmarks” in incentives or targets. This can actually also lead to wrong behaviors (think about share buy back programs to susitain share price in presence of stock-based incentive schemes).

A final example is ESG targets. Many of the related measures don’t necessarily apply to all organizations equally. Adopting them may actually interfere with the real performance measurement of an organization.

The “Laws of Organisation Design” Collection. This article belongs to a series focused on the different principles and laws that govern organisational behaviours, decision-making and group work, and that should be taken into account in organisation design projects.

The Laws of Organisation Design

Law of Requisite Variety
Law of Alignment

Conclusion

Goodhart’s Law is a powerful reminder that while metrics are essential for guiding and assessing performance, they should be used with caution. When metrics become targets, they can distort behaviour, erode the quality of outcomes, and lead to unintended consequences. Therefore, it’s crucial for organisations to set targets thoughtfully, considering the broader goals and the potential impact of over-focusing on specific measures.