Goodhart’s Law explains why a metric can become less useful when people start optimizing it as a target. A proxy metric is an indirect measurement used to estimate an outcome that is difficult to measure directly. In data analytics, the solution is to validate proxy metrics against real outcomes, monitor multiple indicators, and use a scorecard with clear weights, thresholds, and safeguards against manipulation.
Businesses rely on metrics to make decisions about marketing campaigns, product development, customer retention, employee performance, and artificial intelligence. However, a number that initially indicates success can become misleading when teams begin optimizing that number instead of the result it was intended to represent.
This problem connects two important concepts: Goodhart’s Law and proxy metrics. Understanding their relationship helps analysts distinguish genuine progress from performance that only looks good in a dashboard.
This guide explains the concepts from beginner to advanced level, with practical examples, comparison tables, formulas, and a step-by-step method for building a proxy metric scorecard.
1. What Is Goodhart’s Law?
Goodhart’s Law is commonly summarized as: “When a measure becomes a target, it ceases to be a good measure.” The idea is associated with economist Charles Goodhart and his work on economic policy and statistical relationships.
The principle is straightforward: a metric may accurately reflect an outcome under normal conditions, but deliberately optimizing that metric can weaken its relationship with the outcome.
Imagine a customer support department that measures service quality using average call duration. Initially, shorter calls may indicate efficient service. But when employees are rewarded exclusively for reducing call duration, they may end conversations before fully solving customers’ problems.
The dashboard improves, yet customer satisfaction declines. The organization has optimized the measurement rather than the intended outcome.
2. What Are Proxy Metrics in Data Analytics?
A proxy metric is an observable measurement used as an indirect indicator of a goal that is difficult, expensive, delayed, or impossible to measure directly.
For example, a marketing team may use qualified leads as a proxy for future sales because actual revenue may take weeks or months to materialize. A product team may use successful onboarding as an early indicator of long-term customer retention.
A proxy is not automatically a bad metric. It is useful when evidence shows that it has a meaningful, sufficiently stable relationship with the outcome.
| Business goal | Possible proxy metric | Actual outcome to verify |
|---|---|---|
| Increase revenue | Qualified leads | Sales revenue and profit |
| Improve customer loyalty | Repeat purchases | Long-term retention and customer value |
| Improve website experience | Task completion rate | Customer success and satisfaction |
| Improve SEO performance | Relevant organic clicks | Qualified traffic, conversions, and business value |
| Improve AI classification | F1 score | Performance under real-world error costs |
For machine learning applications, see Kovendo’s guide to F1 score calculation and interpretation. It explains why a model’s score must be interpreted in the context of precision, recall, and the errors that matter to the application.
3. How Goodhart’s Law Affects Proxy Metrics
The connection becomes clear when an organization turns an indirect indicator into a performance target. Employees, algorithms, or automated systems begin finding ways to increase the measured value, sometimes without improving the underlying goal.
Example 1: Website traffic versus business growth
A website receives 20,000 organic visits per month and generates 400 qualified leads. The marketing team decides that increasing traffic is the primary goal and launches content designed to attract broad, low-intent searches.
After three months, traffic rises to 35,000 visits, but qualified leads remain at 400. The additional visits have not produced additional qualified demand.
| Metric | Before | After | Interpretation |
|---|---|---|---|
| Organic visits | 20,000 | 35,000 | 75% increase |
| Qualified leads | 400 | 400 | No improvement |
| Lead rate | 2% | 1.14% | Conversion efficiency declined |
Traffic was a proxy for audience reach, not proof of business growth. A better dashboard would track relevant traffic, qualified leads, conversion rate, and revenue together.
For related strategy, read Kovendo’s AI-powered digital marketing growth strategy, which discusses connecting digital marketing activities with meaningful business results.
Example 2: Customer support speed
Suppose a support team resolves 90% of tickets within four hours. Management sets a strict target to raise this figure to 98%.
Agents may begin closing tickets quickly, even when the underlying issue is not resolved. Reopened tickets, repeat contacts, and complaints increase.
The better approach is to measure resolution time alongside first-contact resolution, reopened-ticket rate, customer satisfaction, and repeat-contact rate. Speed matters, but only when service quality remains acceptable.
Example 3: Machine learning model performance
A classification model is evaluated using accuracy. On a dataset where 99% of transactions are legitimate, a model that predicts every transaction as legitimate can achieve 99% accuracy while detecting no fraud.
Accuracy is therefore an inadequate proxy for fraud-detection quality on this dataset. The team should inspect precision, recall, the confusion matrix, and the operational cost of missed fraud and false alarms.
Explore Kovendo’s guides to accuracy versus F1 score and precision versus recall for practical examples of choosing metrics that better represent model performance.
4. Proxy Metrics vs. Actual Outcome Metrics
The most important distinction is that a proxy provides indirect evidence, while an outcome metric measures the result the organization ultimately wants to achieve.
| Dimension | Proxy metric | Outcome metric |
|---|---|---|
| Purpose | Signals likely progress | Measures the intended result |
| Availability | Often available quickly | May require more time or cost |
| Example in SEO | Relevant organic clicks | Qualified leads or sales |
| Example in product analytics | Feature adoption | Retention or customer value |
| Main risk | Weak or changing relationship with the goal | Delayed feedback or incomplete attribution |
| Best practice | Validate against independent outcomes | Use to confirm actual performance |
Outcome metrics are not perfect either. Revenue may increase because of seasonality rather than a campaign, and retention can be influenced by pricing or market conditions. Use both proxy and outcome metrics, and investigate what causes them to change.
5. How to Choose a Reliable Proxy Metric
Before putting a proxy on a dashboard, test whether it represents the goal well enough to support the intended decision.
- Define the real objective. State the desired result precisely, such as increasing profitable customer retention rather than simply increasing app sessions.
- Select a plausible proxy. Choose a measurable behavior or signal that could reasonably precede or reflect the outcome.
- Test historical relationships. Check whether changes in the proxy have consistently corresponded with changes in the outcome across relevant periods and customer groups.
- Validate on new data. Test the relationship on a separate period or sample rather than relying only on the data used to select the metric.
- Check for unintended effects. Determine whether optimizing the proxy encourages spam, low-quality activity, risky decisions, or exclusion of important user groups.
- Review it regularly. Reassess the relationship after changes to products, markets, incentives, or customer behavior.
6. How to Build a Proxy Metric Scorecard
A proxy metric scorecard combines multiple indicators into a structured view of performance. It can help teams avoid making decisions from a single number, provided the scorecard is validated and its limitations are clear.
Use the following process to build a practical scorecard.
Step 1: Define the objective and outcome
Imagine an online subscription business wants to predict whether new customers will remain subscribed for at least 90 days.
The outcome metric is 90-day retention. Because this outcome takes time to observe, the business selects three early indicators:
- Successful onboarding completion.
- Meaningful product usage during the first two weeks.
- Completion of a key product task.
These are candidate proxies, not automatically proven predictors. The business must test whether they help predict 90-day retention.
Step 2: Normalize the metrics
Metrics often use different scales. One may be a percentage, another a count, and another a duration. Normalization converts them to a comparable scale.
For a metric where higher values are better, a simple min-max normalization formula is:
For a metric where lower values are better, reverse the direction:
For example, suppose meaningful usage has an observed value of 60%, with a defined minimum of 20% and maximum of 80%.
Score = 66.67
The normalized score is 66.67 out of 100. The minimum and maximum should be defined using meaningful benchmarks, historical data, or business requirements—not arbitrary values selected to make performance look better.
If the minimum and maximum are equal, the formula cannot be applied directly. Define a fallback rule for that case. For metrics with severe outliers, percentile-based scaling or other robust methods may be more appropriate.
Step 3: Assign weights
Not every proxy contributes equally to the objective. Assign weights based on evidence, business relevance, and validation results. The weights must add up to 1, or 100%.
| Proxy metric | Normalized score | Weight | Weighted contribution |
|---|---|---|---|
| Onboarding completion | 80 | 0.40 | 32.00 |
| Meaningful usage | 70 | 0.35 | 24.50 |
| Key task completion | 60 | 0.25 | 15.00 |
| Total | — | 1.00 | 71.50 |
These illustrative weights assume onboarding has the strongest relationship with retention. In a real deployment, the team should estimate or validate that relationship rather than treating the example weights as universal recommendations.
Step 4: Calculate the composite score
The weighted score is the sum of each normalized metric multiplied by its assigned weight.
Using the example above:
Score = 32 + 24.5 + 15
Composite Score = 71.5 / 100
The scorecard produces a result of 71.5. It summarizes the selected indicators, but it does not mean the customer has a 71.5% probability of being retained. A probability interpretation would require a separately validated predictive model and calibration.
Step 5: Set thresholds and decision rules
Thresholds translate a score into an action. They should be based on historical outcome rates, operational capacity, risk tolerance, and the cost of mistakes.
| Illustrative score | Status | Possible action |
|---|---|---|
| 80–100 | Strong early engagement | Continue monitoring; identify successful onboarding patterns. |
| 60–79.99 | Moderate engagement | Investigate the weakest component and offer relevant guidance. |
| Below 60 | Low early engagement | Review barriers and consider appropriate customer support. |
These thresholds are examples for explaining the method, not validated retention cutoffs. Before using them in production, examine how each band relates to actual 90-day retention and whether the resulting actions improve outcomes.
Step 6: Add guardrails to prevent Goodhart’s Law
A scorecard can also be gamed. If employees are rewarded for onboarding completion alone, they may push customers through onboarding without helping them understand the product.
Use safeguards alongside the composite score:
- Track the real outcome, such as 90-day retention.
- Monitor complaints, refunds, and early cancellations.
- Check whether results differ across customer segments.
- Audit unusual jumps in component metrics.
- Review the scorecard after major product or incentive changes.
7. Advanced Considerations: Weights, Correlation, and Gaming
Should weights be equal or data-driven?
Equal weights are easy to explain and can be a reasonable starting point when evidence is limited. Data-driven weights may improve prediction, but they can overfit historical patterns and become unreliable when customer behavior changes.
Compare alternative weighting methods on data that was not used to build the scorecard. Check calibration if the score is intended to represent a probability, and test whether performance remains stable across time and important user groups.
Why correlated proxies can distort a score
Suppose a scorecard gives separate weights to page views, session duration, and number of pages visited. These metrics may capture overlapping aspects of engagement. Counting all three independently can make engagement dominate the score more than intended.
Check correlations and the conceptual meaning of each metric. Remove redundant indicators or adjust the design when they measure essentially the same behavior.
How to detect proxy gaming
Look for a rising proxy alongside a flat or declining outcome. For example, if campaign clicks increase while qualified leads decline, investigate traffic quality, targeting, and landing-page relevance. If model accuracy rises while minority-class recall falls, review class-level performance and the evaluation setup.
Also monitor abrupt behavioral changes after a target or incentive is introduced. Such changes do not prove manipulation, but they are a reason to inspect the underlying data and process.
For further machine-learning context, Kovendo’s article on Mixture of Experts models in AI provides a useful starting point for exploring how complex model architectures should be evaluated against their actual objectives, rather than architecture or benchmark scores alone.
8. Proxy Metrics in SEO, Digital Marketing, and AI
The same principle applies across different analytical disciplines, although the appropriate outcome varies by business model.
| Area | Potential proxy | Outcome or guardrail |
|---|---|---|
| SEO | Impressions and organic clicks | Relevant traffic, conversions, and business value |
| Email marketing | Open rate | Qualified clicks, conversions, and deliverability |
| Product analytics | Feature usage | Retention, task success, and customer value |
| AI classification | Accuracy or F1 score | Class-level errors and application-specific costs |
| AI assistants | Benchmark or preference score | Task success, factual reliability, and real-world safety |
| Customer support | Average resolution time | First-contact resolution and customer satisfaction |
For SEO, impressions and clicks can help diagnose visibility and interest, but neither guarantees qualified visitors or revenue. For AI systems, benchmark performance can be useful, but it does not establish that a system will behave reliably in every real-world situation.
Read Kovendo’s practical guide to deploying AI-powered digital marketing for additional context on connecting marketing workflows with measurable business objectives.
9. Common Mistakes When Using Proxy Metrics
- Choosing a proxy because it is easy to measure: Convenience does not establish relevance to the goal.
- Assuming correlation proves causation: Validate whether changes in the proxy actually contribute to the desired outcome.
- Using too many overlapping metrics: Redundant indicators can distort composite scores.
- Changing thresholds without evidence: Thresholds should reflect observed outcomes and operational requirements.
- Ignoring unintended consequences: A metric can improve while customer experience, fairness, quality, or profitability worsens.
- Never revisiting the scorecard: Relationships can change as markets, products, and incentives evolve.
10. Frequently Asked Questions
Goodhart’s Law describes how a metric can lose its usefulness when it becomes a target. In data analytics, people or systems may optimize the measured number without improving the underlying business outcome. Using independent outcome measures and monitoring unintended consequences helps reduce this risk.
Qualified leads can be a proxy for future sales, product activation can be a proxy for retention, and organic clicks can be a proxy for search visibility and interest. Each proxy must be validated against the actual outcome because its relationship with the goal may vary.
Choose relevant indicators, normalize them to a comparable scale, assign weights that sum to 1, and calculate the weighted sum. Then establish evidence-based thresholds and validate the score against actual outcomes. Do not interpret a composite score as a probability unless it has been built and calibrated for that purpose.
Use a balanced set of metrics, retain visibility into the true outcome, audit incentives, check for gaming, and regularly validate proxy relationships. Avoid rewarding a single indicator without considering quality, customer impact, and long-term results.
Conclusion
Goodhart’s Law highlights a fundamental challenge in data analytics: a measurement that once reflected success may become misleading when people optimize it directly. Proxy metrics remain valuable when the real outcome is difficult or slow to observe, but they must be treated as evidence rather than unquestionable proof.
A reliable approach starts with a clearly defined objective, selects plausible proxies, validates their relationship with real outcomes, and combines them through a transparent scorecard. Normalization, justified weights, evidence-based thresholds, and guardrails make that scorecard more useful.
Most importantly, continue checking whether the proxy still represents the result you care about. The purpose of analytics is not to maximize a dashboard number; it is to make better decisions and produce better outcomes.
Editorial note: The numerical examples, weights, and scorecard thresholds in this article are illustrative. Validate them against your own data before making operational decisions.
Comments
Post a Comment