Good Experiment, Bad Experiment

Good experiments are built around product themes. Every good experiment furthers the product team’s understanding of the customer and her needs. They build on each other, creating insight and depth of understanding in an area. Bad experiments are scattered, one-off tests that stand alone. They come from unfocused brainstorms that result in not being connected to the product vision or a growth strategy.

Good experiments drive impact by solving real user problems. They have strong, well-reasoned hypotheses grounded in data analysis, customer insights, and market research. Good experiments are about understanding true customer behavior around the things that matter. Bad experiments move metrics by confusing or tricking your users. They make things harder for your users, rather than solving underlying problems.

Good experiments are conceived as bets. You know they have a chance to fail, but based on the info you have available, it is a good investment to make. They help you learn about the things that matter, enabling you to take bigger bets over time. Bad experiments are endless optimizations. They adjust things around the edges in an attempt to improve performance in a marginal way. They steal time, energy, and resources from validating more meaningful bets. Businesses aren’t built on optimizations.

Good experiments ask “What do we do next with what we learned?” They dig in on questions like why did it work or not work? Does this prove or disprove our hypothesis? Was it a good test of the hypothesis? If it failed, should we invest more to make it successful? If it succeeded, should we do more to make it even more successful? Bad experiments ask “Should we ship or turn off?” and are quickly forgotten.

Good experiments are used as a tool of humility. They are used to test something we believe in, but we are humble enough to know that we aren’t sure if it will work. Bad experiments are used to defer decision making, settle disagreements, or let the data tell us what the right thing to build is.

Good experiments define success up-front. They properly instrument the metrics associated with the definition of success. Bad experiments add many metrics to the dashboard, then cherry pick the ones that improve. They segment results in endless ways in a search for a positive signal used to justify the experiment idea rather than prove the hypothesis. Bad experiments get restarted because the right metrics weren’t instrumented from the beginning.

Good experiments consider tradeoffs. They understand that the levers of the business work together as a system. Everything is connected. If you move sign-ups, you might decrease activation. If you move monetization, you might decrease retention. Bad experiments focus on their own OKRs and goals. They treat every improvement as a win, even if that win came at a cost elsewhere in the system.

Good experiments use tight exposure groups. Everyone in the test group experienced the change, with an equally sized true control group.. Bad experiments use random assignments to wide audiences. Users in the test group weren’t exposed to the change while those in the control wouldn’t have experienced the change if in the test group. Good experiments are well instrumented at launch. Bad experiments get restarted because of missing data. Garbage in, garbage out.

Good experiments are communicated. They build a narrative around learning. They are written up, shared broadly, discussed in groups, and documented for the future. Bad experiments run a test but never close the loop.

Good experiments are integrated into the core product process. They incorporate cultural principles and think about long term team success. Bad experiments act as renegades. They create endless friction with other teams and lead to distrust and lack of collaboration.

Good experiments have a plan for success. They invest in making a successful experiment a real part of the product and think about full go-to-market integration. Bad experiments leave MVPs in the product forever.