When developing a product, it’s important to follow the product vision and focus on it first, rather than relying solely on metrics.

In the context of this article, by "vision," I mean how the product should work, look, and bring value in the long run. This is not about business strategy but about understanding how that strategy translates into the product itself.

Building a product without a vision is like constructing a house without knowing what it should look like in the end or endlessly rearranging furniture in an apartment without a clear idea of the overall interior design.

Metrics are numbers that reflect how users behave within the product. They are important, but they have limitations.

First, user behavior is not always easy to explain and is often subject to misinterpretation. Teams tend to come up with rational explanations for any change, even when the real reasons might be completely different. Here are two completely opposite conclusions that could be drawn from the same fact:

• We made the video larger → users got stuck watching it, which caused button clicks to drop.
• We made the video larger → users understood the content better, which led to an increase in button clicks.

Which one is correct? Maybe both? Maybe neither? That’s the problem.

Second, metrics are not the most reliable tool in the short term (up to six months). In most products, experiment results are evaluated around 14 days after launch. Unfortunately, such short time frames often lead to situations where a follow-up experiment in 3–6 months shows that the initial effect has either disappeared or even reversed.

Shopify has found that 30% to 40% of experiments that show positive short-term results have no long-term impact. Initial lifts can be misleading, and some of your “losers” might actually yield unexpected long-term value. Lenny’s Podcast, Archie Abrams (VP Product, Head of Growth at Shopify)

And here’s the problem. A vision is rarely implemented all at once; it usually happens step by step. In this case, if you only focus on numbers, the first step might look like a "failure" without the second. The team gets stuck in endless discussions and analysis, and in the end, the business decides that the idea doesn’t work, causing the entire concept to collapse.

An obsession with metrics leads companies to fear temporary dips. This is especially true if your time-to-market is long, but you’re still operating under the pressure of quarterly reports to investors, where any drop in numbers is seen as a disaster.

In response, product teams come up with hacks. One of the funniest things I’ve seen is when a complex experiment that initially showed a statistically significant negative result is split into three smaller steps, each of which is "gray"—meaning not statistically significant. Since none of them alone show a clear negative impact, the experiment is considered acceptable. Voilà! The team still implements the change they wanted, but without an "official" failure—although they’ve now spent three times as much time on it.

Vision is more than just metrics. Implementing a vision should create new opportunities and growth points, and that’s a good thing — even if metrics temporarily decline. Metrics are there to ensure you’re not making a catastrophic mistake, but if they fluctuate by ±2–5%, that’s normal. Newness, habit changes, and user adaptation — there could be many reasons for small shifts.

Ideally, a vision should be launched as a complete update rather than in small pieces. That’s how companies like Airbnb, Figma, or Sketch do it — releasing updates once a quarter or every six months. This helps avoid pointless analysis of intermediate metrics and saves teams from stressing over every number.

At the end of the day, metrics are a tool, not a goal. A product doesn’t thrive on numbers — it thrives on how it improves users’ lives.

Additional links

Lenny’s Podcast. Archie Abrams, VP Product, Head of Growth at Shopify

Companies should step off the quarterly report treadmill. Nicolai Tangen

Airbnb 2024 Winter Release