Many people know growth teams experiment with their product to drive growth. But how should growth teams set goals? At Pinterest, we’ve experimented with how we set goals too. I’ll walk you through where we started, some learning along the way, and the way we try to set goals now.

Mistake #1: Seasonality
Flash back to early 2014. I started product managing the SEO team at Pinterest. The goal we set was a 30% improvement in SEO traffic. Two weeks in we hit the goal. Time to celebrate, right? No. The team didn’t do anything. We saw a huge seasonal lift that raised the traffic without team interference. Teams should take credit for what they do, not for what happens naturally. What happens when seasonality drops traffic 20%? Does the team get blamed for that? (We did, the first time.) So, we built an SEO experiment framework to actually track our contribution.

Mistake #2: Only Using Experiment Data
So, we then started setting goals that were entirely based on experiment results. Our key result would look something like “Increase traffic 20% in Germany Q/Q non-seasonally as measured by experiments” with a raw number representing what a 20% was. Let’s say it was 100K. Around this time though, we also started investing a lot in infrastructure as a growth team. For example, we created local landing pages from scratch. We had our local teams fix a lot of linking issues. You can’t create an experiment on new pages. The control is zero. So, at the end of the quarter, we looked at our experiments, and only saw a lift of around 60K. When we look at our German site though, traffic was up 300%. The landing pages started accruing a lot of local traffic not accounted for in our experiment data.

Mistake #3: Not Factoring In Mix Shift
In that same quarter, we beat our traffic and conversion goals, but came up short on company goal for signups. Why did that happen? Well, one of the major factors was mix shift. What does mix shift mean? Well, if you grow traffic to a lower converting country (Germany) away from a higher converting country (U.S.), you will hit traffic goals, but not signup goals. Also, if you end up switching between page types you drive traffic to/convert (Pin pages convert worse than boards, for example), or if you switch between platform (start driving more mobile traffic, which converts lower, but has higher activation), you will miss goals.

Mistake #4: Setting Percent Change Goals
On our Activation team, we set goals that looked like “10% improvement in activation rate.” That sounds like a lofty goal, right? Well, let’s do the math. Let’s say you have an activation rate of 20%. What most people read when they see a 10% improvement is “oh, you’re going to move it to 30%.” But that’s not what that means. It’s a relative percent change, meaning the goal is a 2% improvement. With this tactic, you can set goals that look impressive that don’t actually move the business forward.

Mistake #5: Goaling on Rates
Speaking again about that activation goal, there’s actually two issues. The last paragraph talked about the first part, the percent change. The rate is also an issue. An activation rate is two numbers: activated users / total users. There are two ways to move that metric in either direction: change activated users or change total users. What happens when you goal on rates is you have an activation team that wants less users so they hit their rate goals. So, if the traffic or conversion teams identify ways to bring in more users at slightly lower activation rates, the activation team misses their goal.

Best Approach: Set Absolute Goals
What you really care about for a business like Pinterest is increasing the total number of activated users. At real scale, you also care about decreasing churned users because for many business re-acquiring churned users is harder than acquiring someone for the first time. So, those should be the goals: active users and decrease in churned users. Absolute numbers are what matter in growth. What we do now at Pinterest is set absolute goals, and we make sure we account for seasonality, mix shift, experiment data as well as infrastructure work to hit those goals.

Currently listening to RY30 Trax by µ-ziq.