The Basics of Product Measurement

Connor Wang

Jun 7, 2021 • 6 min read

In a seminar where Mixpanel Co-founder Suhail Doshi shared his view on how to formulate a successful start-up. At a glance, it's nothing special or even a bit cliché, but is the very essence of making the right product.

At first, he asked 3 questions:

Is my product easy to understand?
Is it easy to get started with my product?
Are people coming back to my product?

Then he gave the formula like this:

GROWTH =

Visits: People hitting your landing page or mobile app screen
Sign-ups: The number of people who then sign up
People who found value: The number of people who do the thing the product was made for
Retention: The number of people who then subsequently come back & do that valuable thing again
Spread the product: The number of people who want other people to know about your product (sharing, word-of-mouth, sales, etc.)

The above bold texts are the basics - and the basics are fucking important. Those are the levers for growth.

Simplify

Usually companies that get it wrong overcomplicate things they need to measure and track. It's really easy to make 25 metrics and hundreds of sub-metrics. The problem is that once you have that many metrics, the company immediately goes into paralysis and have decision making problems. One thing we need to understand is that causation is really hard to discover, so simplify and have 3 to 5 things to monitor and a Northstar will do more good for us.

Pick one north star & monitor 3-5 metrics. Less is more. A frequent mistake is having too many metrics that you monitor.

So pick a number you're willing to bet the company on. The Northstar metric doesn't stay forever but you need to choose it for 6 months, and if you're wrong, you can change it and commit again for another 6 months. Where you can get really complex and sophisticated is discovering why the number is going down or up (generate learning).

There are two types of metrics

Metrics that guide the team
Metrics that help assess and monitor if anything is going wrong

Let's focus on the metrics that guide the team.

As a contrarian, I'd like to look at things from the opposite aisle - instead of looking at what makes a product grow, let's look at why the product is not growing.

Basically you're trying to solve the 3 questions on the left for each and every step down the funnel.

Is my product easy to understand?
Is it easy to get started with my product?
Are people coming back to my product?

Asses what's wrong and we'll have our levers and should be able to take actions and fix it.

Is My Product Easy to Understand?

Paul Graham famously said the following on bad landing pages

The enemy is the back button.

This is especially true when people are dealing with unfamiliar products or features. When they're in a strange place and feeling psychologically unsafe, they'll hit the back button immediately - and that's how most of the company lose their customers.

So how do we measure if our product is easy to understand?

Do people bother to sign up?
Ratio of single page visits to multi-page engagement?
B2B: Do they visit your pricing page?
Consumer: Performing A/B tests of your copywriting

Is It Easy to Get Started with My Product?

You only need 1 click to get started on Google.

Then this was the Airbnb website before COVID-19.

When you look at it, you'll immediately think - it's so easy! I can do it with Bootstrap in probably 20 minutes. That's absolutely true, but figuring out the right image, copywriting and layout took them over 50 iterations. The key question to ask is: "What are we optimizing for?" before coming up with choices / hypothesis.

So how do we measure if our product is easy to get started?

People who do one valuable thing (e.g. search, publish a post)
Measure a funnel of your initial user experience
Speed of doing that valuable thing

Tips and tricks:

Email/text confirmation will cause significant drop-off - early stage products won't have massive fraud and spam, so it's unnecessary to worry about those
Constantly iterate on your initial user experience - this will be a never ending process, you will do this forever. Especially when the product changes, so having one responsible person thinking about that all the time is very important
Let users experience the product before asking them to sign up

Are People Coming Back to Our Product?

Poor retention is often the most ignored metric by early founders. It's often the reason even product with million of users die.

- Suhail Doshi, Co-Founder and Chairman of Mixpanel

Suhail has seen at least 15 companies grew virally and died. He calls this "The Shark Fin Effect".

It usually goes like this:

You find a way to go viral, you feel like you struck gold and acquired tons of users. Life is awesome and you're going to be a billionaire
Eventually the rate you're losing users becomes high enough that you can't acquire the new ones

It's one of the common problems for startups, if that varaility happens too early and you didn't quite think about retention, it will become really problematic later on. Even very smart founders experience this problem - the viral moment is so enticing and they end up keep optimizing for growth and ignore everything else. That ends up not being very valuable because if you're just losing all the users, it's extremely hard to re-activate them.

Re-activation campaign generally don't go very well. Usually you just lose those users until they decide to give the product a second chance

- Suhail Doshi, Co-Founder and Chairman of Mixpanel

So it's critical to think through the retention.

So how do we measure if people are coming back?

New users who came back D7 or D30 to do a valuable thing again
Number of people who use the product every day (DAU) - Mixpanel found out that DAU churn strong correlated with dollar churn, 1.4% DAU reduction results in 10% dollar churn!
B2B: Revenue churn - monthly revenue lost in a given month. Imaging if your monthly revenue churn is 7% - extrapolate that into a year, that's 56%. Starting a new year losing more than half of previous year's revenue while trying to grow on top of that. That'll be really tough. Another thing is that a really high basis revenue will become very difficult to keep up and turn into a long shark fin effect. This gave Suhail the most trouble at Mixpanel. So it's very helpful to think in terms of graph - what would our rate of growth be if churn stayed the same.

Another really important thing is talking to customers.

You're going to get way more information and depth from talking to customers than you ever will looking at a data point, ever.

- Suhail Doshi, Co-Founder and Chairman of Mixpanel

Q: When you have less than 50 users/customers?

A: Talk to your users. Write everything down.

TL; DR

Get the basics right.

SIMPLIFY - A Northstar + 3-5 monitoring metrics. Less is more.

Really think about retentions.

Talk to customers. Talk to customer. Talk to customers.

Good Optimizations require a good mixture of art and science. What matters is that you run the experiment (science) to develop the hypothesis (art) and then measure it (science) to find out whether you are right or not.