After being introduced by a mutual friend, I had a mind-opening conversation with Ed Baker, the former head of growth at Facebook and Uber. During our insightful call, Ed imparted a piece of wisdom that resonated deeply. At Uber, one experiment stood out for its impact, surpassing the cumulative effect of all other experiments in a year. This interaction solidified my belief in the transformative power of strategic experimentation.

What is a Lean Experiment?

At its core, a lean experiment is a method used to test and validate hypotheses in the most efficient and low-risk manner possible. Derived from the principles of Lean Startup methodology, this approach emphasizes speed, agility, and cost-effectiveness. The goal is to learn more about what works and what doesn’t with the least effort and resources.

Characteristics of Lean Experiments:

1. Hypothesis-Driven: Every lean experiment starts with a clear hypothesis. For instance, “If we add a user tutorial, we will see an increase in user engagement.”
2. Simple: Lean experiments focus on creating an idea's most minor or simplest version to test its validity.
3. Measured and Iterative: The effectiveness of lean experiments hinges on their ability to produce measurable results. Iteration is vital to the process, as each experiment leads to new insights and refinements.
4. Customer-Centric: These experiments are designed around the customer’s needs and behaviors. Understanding the target audience and how they respond to changes is fundamental to lean experimentation.
5. Rapid Execution: Lean experiments are conducted quickly to avoid unnecessary expenditures of time and resources.

Why Experimentation is Critical

As the CEO of CodeCheck, a shopping app with a large audience in the German-speaking markets, I started to embrace the topic of experiments. While we only ran a few yearly, we’ve seen some good results. A new onboarding process, introduced after a design sprint (an experiment on its own), showed a massive increase in retention against the old version. That helped us to make the (data-driven) decision to improve onboarding further.

The graphic above captures a stark reality: regular product validation is essential to mitigate the risk of startup failure. With constant feedback, a startup’s trajectory can stay on its intended path. Lean experiments act as checkpoints, realigning the startup’s course with the market’s pulse.

Let’s say you’re a chef experimenting with a new recipe to see what your customers like best. You tweak the ingredients slightly each time and keep making the original recipe. The original recipe acts as a “control group” — it’s the standard you compare the new versions against. In tech and startup experiments, this control group is essential to understand the actual effect of your changes.

TikTok’s Algorithm Tuning
TikTok’s approach to refining its content recommendation algorithm is quite similar. Imagine TikTok’s algorithm as a recipe they’re trying to perfect. Their goal is to make their app so engaging that users spend more time on it and enjoy the experience. To achieve this, they didn’t change the entire recipe at once. Instead, they made small changes — like a chef altering one ingredient at a time.

For instance, TikTok might tweak its algorithm to show users more videos that people typically watch all the way through, thinking this might keep users engaged longer. But how do they know if this change is working? Here’s where the control group comes in. While they test this new algorithm version on some users, they keep the original version for others. These others — the control group — help TikTok see what user behavior would have been like without the changes.
By comparing the behavior of users experiencing the new algorithm with those using the original (the control group), TikTok can see if their new ‘recipe’ is more effective.

They look at how long users stay on the app, how much they interact with the content, and whether they keep returning. If the new algorithm shows better results than the control group, it’s a win. If not, it’s back to the drawing board, like our chef returning to the kitchen to try another ingredient tweak.
Making small, measured changes and comparing them to a control group is at the heart of lean experimentation in the tech and startup world. It’s about learning what works best by trying out new ideas on a small scale before deciding if they’re good enough to implement fully.

Airbnb’s Photographic Transformation
Airbnb is a notable example of a startup that used experimentation to improve its product and scale its business. In their early days, Airbnb was struggling to increase bookings. The team hypothesized that the quality of the property images was impacting users’ booking decisions. To test this hypothesis, they experimented: they went to New York, rented a professional camera, and replaced the amateur photographs of listings with high-quality photos.
The results were immediate and significant. The professional photos doubled the weekly revenue to $400 per week in New York. This experiment was so successful that Airbnb rolled out professional photography as a standard service feature.

Running Experiments at Large-Scale

This lean experiment allowed Airbnb to identify a simple yet critical lever for their platform’s success and scale it efficiently. The company continued to use a data-driven approach to decision-making, constantly running experiments to optimize every aspect of its service, from search algorithms to user interface design.

“Feedback is the breakfast of champions.” — Ken Blanchard

The Airbnb and the TikTok cases are often cited in discussions on lean startup methodologies and growth hacking, showing the impact that intelligent, well-executed experiments can have on a company’s trajectory.
This underscores a crucial point: experimentation isn’t just beneficial; it’s a competitive edge. For startups, especially in the early stages, time is the most precious currency. Wasting it isn’t an option. Establishing a pipeline of experiments is imperative to ensure continuous learning and improvement. In my work with founders, I emphasize the critical role of structured experimentation — it’s often the deciding factor in outpacing the competition and achieving success.

Here are two examples of potential experiments:

Experiment 1: Landing Page Conversions

• KPI to Influence: Landing Page Conversion Rates
• Hypothesis: I believe that adding a testimonial section to our landing page will enhance users’ trust and increase conversion rates.
• Experiment Design: Introducing a testimonial section on the landing page will improve conversion rates. Create two versions of the landing page — Version A without testimonials and Version B with a section showcasing customer testimonials.
• Timing: Run for two weeks to allow users to engage with the new element.
• Measurement: Measure and compare conversion rates between the two landing page versions.

Experiment 2: User Retention

• KPI to Influence: User Retention Rates
• Hypothesis: I believe that offering an in-app tutorial will elucidate the app’s value, leading to higher user retention within the first week of use.
• Experiment Design: Implementing an in-app tutorial will increase user retention within the first seven days.. Group A receives no tutorial, while Group B users are introduced to a step-by-step in-app tutorial.
• Timing: Monitor user behavior over a month to assess retention over different milestones (Day 1, Day 7, Day 30).
• Measurement: Compare the retention rates of both groups at each milestone.

Watch Out For Correlation vs. Causation

Understanding correlation and causation is vital if you plan to conduct lean experiments. Correlation suggests a relationship between two variables but doesn’t prove that one causes the other. Causation, on the other hand, means that one event results from the occurrence of the other event; there is a cause-and-effect relationship.

For instance, a startup may see that as their social media engagement goes up, so do their sales. They might conclude that increasing social media efforts leads directly to sales (causation). However, sales and social media engagement are increasing due to a third factor, such as a seasonal trend or a market-wide uptick in demand (correlation).

Misinterpreting these two concepts can lead startups astray. Suppose a startup ramps up social media spending, assuming it will increase sales. In that case, they might be investing resources into an area with no direct effect on revenue, potentially neglecting other areas that could substantially impact their growth. This is why designing experiments that can isolate variables and test causative relationships, not just correlations, is crucial.

Sample Size

The sample size in any experiment is crucial to make data-driven decisions. A sample size that’s too small can lead to misleading results, where the success or failure of an experiment is merely due to chance rather than a true reflection of your market’s response.

For instance, if a startup testing a new feature only solicits feedback from a handful of users, they might wrongly assume the feature is a hit or miss based on this limited input. This could lead them to fully roll out a feature that hasn’t been adequately vetted or to discard a potentially valuable one prematurely.

Here are some actionable insights on how to calculate the sample size for your startup’s experiments:

1. Set the Confidence Level: Typically, a confidence level of 95% is used, which means you can be 95% certain that the results fall within a specific range.
2. Determine the Margin of Error: Decide on the maximum margin of error you are willing to accept. A smaller margin of error requires a larger sample size.
3. Estimate the Variability: If you have historical data, use it to estimate your metric's variability (standard deviation). If not, you might assume a 50% variability as a conservative estimate.
4. Use a Sample Size Calculator: Utilize an online sample size calculator where you can input the above parameters (confidence level, margin of error, and variability) to estimate the sample size needed.
5. Adjust for Population Size: If your customer base is small, you must adjust your sample size to reflect the population size.
6. Plan for Response Rate: Consider the response rate if you conduct surveys. Not everyone will respond, so you may need to send out more invitations than the number in your sample size.

What About Early Stage Startups?

At CodeCheck, we achieved an early product channel fit with millions of users. This advantage could have been leveraged for running experiments to refine our approach even before we nailed down the product-market fit. Looking back, that knowledge would have been invaluable. Yet, it’s essential to acknowledge that such circumstances are outliers, not the standard experience for most startups.
Traditional experimental options like A/B testing might be constrained in most cases due to a small user base. Anyway, startups in that phase are one big experiment with a singular focus: attaining product-market fit.

Demand testing via landing pages is a practical experiment for B2C startups or B2B ventures with larger markets. An MVP test, too, can take on an experimental nature by inviting distinct cohorts for closed trials, ensuring there’s always a benchmark group for comparison.

However, it’s essential to recognize the challenge of running frequent experiments without a substantial user pool. The key isn’t just to experiment but to do so smartly, ensuring that every test yields valuable insights.
Introducing the Experiment Canvas

To streamline the experimentation process, I’ve crafted the Experiment Canvas, a strategic tool designed to bring order to the experimentation process. This canvas is a roadmap for startups to navigate through the process of testing and learning.

The Structure of the Experiment Canvas:

• Name of the Experiment: Assign a unique identifier to each experiment for easy reference.
• Underlying Hypothesis: Clearly state the assumption driving your experiment.
• Main Metric and Support Metric: Identify the primary data point you’re testing and a secondary metric for additional insight.
• Experiment Description and Criteria for Success: Detail what the experiment entails and define what success looks like.
• Timing and Goal: Set the duration and objectives of your experiment.
• Result: Record the outcomes, analyzing the data gathered.
• Conclusion and Next Steps: Interpret the results and decide on the subsequent actions.

Why the Experiment Canvas is a Game-Changer

The Experiment Canvas is more than a worksheet; it instills a philosophy that data-driven decisions outweigh gut feelings. Offering a structured approach to experimentation empowers startups to dissect their assumptions, measure their progress accurately, and pivot with purpose.

“Success is not built on success. It’s built on failure. It’s built on frustration. Sometimes it’s built on catastrophe.” — Sumner Redstone.

Implementing the Experiment Canvas into your startup’s routine could mean the difference between navigating in obscurity and moving towards the light of validated actionable insights.

Final Thoughts

ean experimentation, backed by the structure provided by Experiment Canvas, is the cornerstone of a robust startup strategy. It allows you to traverse the complex startup landscape with evidence and adapt based on honest user feedback. So keep experimenting, learning, and growing your startup with intention and precision.