It is a commonly held belief that startups can measure their way to success. And while there are always exceptions, early-stage companies often can’t leverage data easily, at least not in the way that later stage companies can. It’s imperative that startups recognize this early on — it makes all the difference.
In this piece, I draw on my experiences using data to take Framer from seed round to Series B. More concretely, I’ll describe what to (not) focus on, and then, how to get real results.
There are good and bad ways for startups to use data. In my opinion, the bad way unfortunately is often preached on saas blogs, a/b test tool marketing pages, and especially growth hacker conferences: that by simply measuring and looking at data you’ll find simple things to do that will drive explosive growth. Silver bullets, if you will.
The good way is comparable to first principles thinking. Below the surface of your day to day results, your startup can be described by a set of numbers. It takes some work to discover these numbers, but once you have them you can use them to make predictions and spot underlying trends. If everyone in your company knows these numbers by heart, they will inevitably make better decisions.
But most importantly, using data the right way will help answer the single most important – but complex – question at any moment for a startup: how are we really doing?
Let’s start with looking at what not to do as a startup.
Table of Contents
Common pitfalls
Don’t measure too much
Technically, it’s easy to measure everything, so most startups start out that way. But when you measure everything, you learn nothing. Just the sheer noise makes it hard to discover anything useful and it can be demotivating to look at piles of numbers in general.
My advice is to carefully plan what you want to measure upfront, then implement and conclude. You should only expand your set of measurements once you’ve made the most important ones actionable. Later in this article, I provide a clear set of ways to plan what you measure.
A/B tests are anti-startup
To make decisions based on data you need volume. Without volume, the data itself is not statistically significant and is basically just noise. To detect a 3% difference with 95% confidence you would need a sample size of 12,000 visitors, signups, or sales. That sample size is generally too high for most early-stage startups and forces your product development into long cycles.
While on the subject of shipping fast and iterating later, let’s talk about A/B testing. To get reliable measurements, you should only be changing one variable at a time. During the early stages of Framer, we changed our homepage in the middle of a checkout A/B test, which skewed our results. But as a startup, it was the right decision to adjust the way we marketed our product. What you’ll find is that those two factors are often incompatible. In general, constant improvements should trump tests that block quick reactionary changes.