High-Impact Experimentation is Product Management

Holly Hester-Reilly, Product Consultant and former Product Owner at Shutterstock, shares how she runs experiments to test critical assumptions and make better decisions throughout the product lifecycle.

Experimentation brings a scientific approach to understanding users’ needs and building products that they will find valuable. It gives product managers the ability to make decisions based on data as opposed to relying exclusively on intuition.

However, there are many common misconceptions about how to run an experiment effectively. When an experiment isn’t run properly, it can lead to poor decisions and lackluster products.

I was excited to interview Holly to learn exactly how to run experiments that lead to better product decisions -- an approach she refers to as “high-impact experimentation.”

High-impact experimentation prioritizes and answers the most important questions teams have, in order to reduce the risk of failure, and build a better product.

Holly shares the exact steps she takes to run high-impact experiments, and shares real examples from her experience working with Shutterstock, Toys “R” Us and Weight Watchers.

She begins by providing a framework for determining what assumptions to test. Holly encourages product managers to ask themselves “what would the cost be if you got this wrong?” before leaving an assumption untested. “We don’t need to test everything. We need to test the things that are risky,” she says.

Holly then shares a number of ways to test these critical assumptions -- including analyzing existing user data to running surveys. She goes on to discuss how to analyze the results of an experiment -- including how to know when you need to dig deeper and when you’re ready to make a decisions, and how to share the results with stakeholders to get alignment on key decisions.

Here are the highlights:

• Holly describes what makes an experiment “high-impact” (6:45)
• Holly breaks down how to determine what assumptions need to be tested (9:45)
• How Holly decides what kind of experiment to run given the question she’s trying to answer (17:30)
• Holly’s best practices for analyzing the results after a product experiment (19:35)
• How Holly shares the results of experiments with stakeholders to get alignment on key product decisions (23:15)