Mastering Quantitative Research is Product Management

As the discipline of product management has advanced in recent years, so too have the strategies, tactics, and tools for conducting product research. Product managers have the potential to access more data to make decisions than ever before.

One critical tool that modern product managers need to have in their tool kit is quantitative research. In other words, using in-product analytics to analyze user behavior and find opportunities to improve the experience.

Since Suhail Doshi is the CEO Mixpanel, one of the most advanced in-product analytics solutions on the market, he was the perfect guest to ask about the latest best practices.

He began by sharing his perspective on how to balance data with intuition when making decisions. Intuition helps you come up with hypotheses about how something could work, and data helps you measure whether the experiments you’re running are working, Suhail says. He warns listeners of the downsides failing to find balance between the two. If you over-optimize for hitting KPIs, you might create a poor user experience. If you over-rely on intuition, your success is dependent on luck.

I think it's kind of like a science experiment. It's kind of like when we were in sixth grade and we learn the science methodology. The first component is just having a hypothesis. The second component is testing it. The third component is measuring to see what the results are.

Suhail then provides several strategies for making better decisions using in-product analytics. I particularly liked his strategy of using a paired metric to measure the success of experiments. Measuring success based on two metrics, instead of just one, hinders you from hitting one metric at the expense of another. This may not be apparent in the short term, but that's because "Not all of your tests, not all of your changes are things that have short term ramifications, either you have to keep monitoring them, even six months down the road."

Suhail also acknowledges that data doesn’t always speak perfect truth:

The reality is that if you only use data, you'll sort of get stuck in this local maxima, this place where you're kind of optimizing and optimizing, but you're only eking out incremental benefits. There's probably a point where spending as much time and energy on making the UI easier to use or better looking has diminishing returns. There might be a better place to spend your time. So I think data can help you figure out when that's happening, but it can't really invent the creative things, or it can't tell you the creative things that you need to do.

When it comes to actually conducting research, Suhail encourages listeners to simply ship features and analyze the results when there isn’t much downside risk:

I think with most decisions, if they're not critical, [and] there's some critical decisions where getting to 90% accuracy is very important, [but] for situations that aren't, usually it's better to just try and see what happens, to move quickly.

He also stresses the importance of setting goals and targets and of making a bet in terms of your own sense of what you think will happen. Suhail recommends betting with your teams: It can just be your pride, it doesn't have to be money or anything, but he says it's good to have a clear bet before you go into it.

You’ll learn a lot from this episode about analytics, experimentation, and data-driven decision-making.

Here are the highlights:

• Suhail describes the role of quantitative research in modern product management (5:00)
• Best practices for conducting quantitative research (9:25)
• The most common mistakes product teams make when conducting quantitative research (17:25)
• Suhail’s approach to building teams (19:50)