You're here when: A number is going up and the team is celebrating. But you have a nagging feeling: does this metric actually connect to business outcomes, or does it just feel good to watch?
The Heuristic
A vanity metric is any number that goes up without telling you whether you're winning. The test is three questions:
- Does it predict revenue or retention? If the metric has no demonstrated relationship to money coming in or users sticking around, it's vanity. Total page views, total signups, total downloads, these numbers always go up. That's not growth. That's accumulation.
- Can you act on it? If the metric moves and you don't know what to change in response, it's vanity. "We had 50,000 page views this month." Okay, what do you do differently? A metric you can act on points to a specific lever: "Our signup-to-activation rate dropped from 35% to 28%." That tells you exactly where to look.
- Does changing it change user behavior? If you can make the number go up without actually improving the product, it's vanity. You can boost total signups with a bigger ad budget. That doesn't mean users are getting more value. You can't fake activation rate or Day 30 retention.
Decision Tree
Real vs. Vanity Pairs
The difference between a vanity metric and a real one is often just a shift in framing. Same data, different denominator.
Quick Example
At Station, we built a browser-type product and celebrated hitting 150,000 signups after launch. The number felt incredible. Weekly active users looked strong too, around 45,000 WAU. We put those numbers in every update, every pitch. Then we started tracking daily active users. The picture shifted: 12,000 DAU. That's a fraction of what the weekly number suggested. But the real gut punch came when we asked the question that actually mattered for a browser: how many people use this at least 6 hours a day? For a product that's supposed to replace your browser, that's the bar. The answer was roughly 3,000. Out of 150,000 signups, 3,000 people were using Station the way it needed to be used. The vanity metrics, total signups and WAU, were masking a deep engagement problem. Once we switched to tracking heavy daily usage, every conversation changed. Instead of "how do we get more signups?" we started asking "why aren't daily users sticking around for a full workday?" That's the question that actually drives product improvement.
The Anti-Pattern
The Dashboard of Green Numbers. Building a dashboard where every metric is carefully chosen to only go up. Total users: up. Total revenue: up. Total sessions: up. Total features shipped: up. Everything looks great because you're only tracking cumulative counters and outputs. The pattern hides decay: churning users, dropping conversion rates, shrinking cohort retention, declining unit economics. If every number on your dashboard is green, you're not measuring the right things. A healthy dashboard has at least one number that makes you uncomfortable.
Written with ❤️ by a human (still)