Vanity Metrics in Engineering and Product Delivery

Definition

Vanity metrics are data points that look impressive on paper and show high activity, but do not correlate with business outcomes, actual productivity, or actionable improvement.

In engineering leadership, vanity metrics offer a false sense of success. They make a team or department look busy, productive, or successful without providing any insights on how to improve performance or deliver real customer value. In contrast, actionable metrics help leaders make informed decisions, diagnose bottlenecks, and align engineering efforts with business growth.

Vanity Metrics vs. Actionable Metrics

The easiest way to identify a vanity metric is to ask: “If this number goes up or down, does it tell us what we should do next?” If the answer is no, it is likely a vanity metric.

Examples in Software Engineering & Product Delivery

1. Lines of Code (LoC) or Commit Counts

• The Vanity Metric: Measuring developer productivity by the volume of code written or the number of commits pushed.
• Why it is a Vanity Metric: More code does not mean more value; in fact, it often means more complexity, technical debt, and maintenance cost. A developer who deletes 100 lines of redundant code to achieve the same result is often delivering more value than one who writes 1,000 lines.
• Actionable Alternatives: Lead Time for Changes (how fast we deliver changes) or Flow Efficiency (how much time work spends waiting vs. active development).

2. Story Points Completed (Velocity)

• The Vanity Metric: Using Agile velocity as a measure of team productivity or comparing velocity between different teams.
• Why it is a Vanity Metric: Velocity is a team-specific capacity planning tool, not a performance metric. When treated as a target, teams naturally inflate point estimates to show growth, yielding a classic example of Goodhart’s Law.
• Actionable Alternatives: Features shipped to production, Flow Velocity (the count of completed flow items in a given period), or customer-facing cycle times.

3. Raw Deployment Count

• The Vanity Metric: Boasting about deploying 100 times a day without context.
• Why it is a Vanity Metric: Frequent deployments of tiny, trivial changes or broken releases do not help the business.
• Actionable Alternatives: Deployment Frequency coupled with Change Failure Rate to ensure speed does not compromise quality.

4. 100% Code Coverage

• The Vanity Metric: Requiring 100% test coverage as a quality gateway.
• Why it is a Vanity Metric: Code coverage only measures which lines of code were executed during a test run, not whether the tests assert correct behavior or test edge cases. Pursuing 100% coverage often leads to fragile, mock-heavy tests that slow down development without reducing real defects.
• Actionable Alternatives: Defect Escape Rate (bugs found in production vs. testing) or Change Failure Rate.

5. Registered Users / Raw Page Views

• The Vanity Metric: Focusing on sign-up volume or landing page traffic.
• Why it is a Vanity Metric: A million sign-ups are useless if they churn immediately. Page views do not indicate engagement, satisfaction, or revenue.
• Actionable Alternatives: Monthly Active Users (MAU) retention cohorts, Customer Acquisition Cost (CAC) to Lifetime Value (LTV) ratio, or feature adoption rates.

6. Token Usage and "Tokenmaxxing" (AI/LLM Applications)

• The Vanity Metric: Tracking success or capability in AI engineering by the number of raw LLM tokens processed (input + output tokens) or the count of LLM API calls.
• Why it is a Vanity Metric: Unnecessarily inflating prompt lengths or context usage—sometimes referred to as "tokenmaxxing"—frequently indicates inefficient prompting, excessive verbosity, or lazy RAG chunking strategies. Rather than improving output, it leads to higher API costs, increased latency, and a higher risk of model confusion.
• Actionable Alternatives: User task completion rates, cost per successful outcome, latency (e.g., Time to First Token), and model output quality evaluations (such as faithfulness and relevance).

Why Tech Leaders Must Avoid Vanity Metrics

For a CTO or VP of Engineering, relying on vanity metrics is not just harmless tracking—it introduces significant strategic risk:

1. They Create a False Sense of Progress

When dashboards show green and upward-trending lines of activity (e.g., high velocity, high commit rates), leaders can easily fall into the trap of assuming delivery is healthy. This blinds the organization to systemic bottlenecks or slipping product quality until it is too late.

2. They Incentivise Dysfunctional Behaviour

According to Goodhart’s Law, once a metric becomes a target, it ceases to be a good metric. Vanity metrics are highly gameable. If you reward commit count, developers will split single commits into dozens. If you reward velocity, estimates will inflate. This shifts focus from solving customer problems to manipulating metrics.

3. They Waste Engineering Capacity

Tracking, reporting, and attempting to optimize vanity metrics consumes valuable engineering hours. Time spent chasing arbitrary code coverage targets or writing fluff commits is time taken away from high-leverage product improvements.

4. They Erode Trust with the Business

When a CTO tells the Board or CEO that "engineering throughput is up 40%" but the product roadmap is slipping and customer retention is dropping, non-technical peers lose trust. Actionable, business-aligned metrics (such as lead time, feature delivery rates, and reliability) bridge this gap.

How to Shift to Actionable Metrics

1. Start with the Value: Define the core value your product delivers (your North Star Metric). Every engineering metric should have a clear path of influence toward this star.
2. Focus on Outcomes over Outputs: Shift focus from how much code we wrote (output) to how the system performed and how quickly we delivered value (outcomes).
3. Use Balanced Frameworks: Implement research-backed frameworks like DORA Metrics (which balance speed and quality) or SPACE Metrics (which balance productivity, efficiency, and developer well-being).
4. Emphasise Qualitative Context: Never look at numbers in isolation. Pair metrics with regular retrospectives, team feedback, and user research.

References

• Wikipedia: Vanity MetricWikipedia
• The Lean Startup by Eric Ries (Crown Business, 2011) — Popularised the distinction between vanity and actionable metrics in product development.
• Accelerate: The Science of Lean Software and DevOps by Nicole Forsgren, Jez Humble, and Gene Kim (IT Revolution Press, 2018) — Provides the research backing for DORA metrics as actionable indicators.
• Goodhart's Law — The strategic danger of turning metrics into targets.
• North Star Metric — Structuring engineering alignment around a single customer value metric.