Tag Archives: activation

What Is Good Retention: An Exhaustive Benchmark Study with Lenny Rachitsky

At the end of 2019, I presented Eventbrite’s product plans to the board for 2020. These plans included a lot of the goals you likely have in your company: improvements in acquisition, activation, and retention. One of our board members asked: “I understand these goals for the year. But long term, how high could we push this retention number? What would great retention be for Eventbrite?”

I actually didn’t have a great answer. Soon after, I was chatting with Lenny Rachitsky, and we decided to embark on a holistic study across the industry to ask “what is great retention?” across business models, customer types, etc. Lenny surveyed a lot of the top practitioners in the industry across a variety of companies, and we’re happy to share the results here. You can see the raw data below, but I recommend reading Lenny’s analysis here. Done? Good.

Why is retention so damn important?
Why are Lenny and I spending so much time researching retention? Because it is the single most important factor in product success. Retention is not only the primary measure of product value and product/market fit for most businesses; it is also the biggest driver of monetization and acquisition as well.

We typically think of monetization as the lifetime value formula, which is how long a user is active along with revenue per active user. Retention has the most impact on how many users are active and lengthens the amount of time they are active. For acquisition, retention is the enabler of the best acquisition strategies. For virality or word of mouth, for example, one of the key factors in any virality formula is how many people can talk about or share your product. The more retained users, the more potential sharers. For content, the more retained users, the more content, the more that content be shared or discovered to attract more users. For paid acquisition or sales, the more retained users, the higher lifetime value, the more you can spend on paid acquisition or sales and still have a comfortable payback period. Retention really is growth’s triple word score.

What are effective ways to increase retention?
Okay, so you understand retention is important and want to improve it. What do you do? Well, at a high level, there are three types of efforts you can pursue to increase retention:

  1. Make the product more valuable: Every product is a bundle of features, and your product may be missing features that get more marginal users to retain better. This is a journey for feature/product fit.
  2. Connect users better to the value of the product that already exists: This is the purpose of a growth team leveraging tactics like onboarding, emails and notifications, and reducing friction in the product where it’s too complex and adding friction when it’s required to connect people to the value.
  3. Create a new product: Struggling to retain users at all? You likely don’t have product/market fit and may need to pivot to a new product.

We discuss these strategies in a lot more depth in the upcoming Product Strategy program coming soon from Reforge, and if you really want a deep dive on retention, we build the Retention & Engagement deep dive.

Why does retention differ so much across categories?
One question you might be asking yourself is why does retention differ so much by different categories? This was the impetus for the initial research, and why I couldn’t give a great answer to our board. Every company has a bunch of different factors that impact retention:

  • Customer type: For example, small businesses fail at a much higher rate than enterprise businesses, so businesses that target small businesses will almost always have lower retention.* This does not make them inferior businesses! They also have many more customers they can acquire.
  • Customer variability: Products that have many different types of customers will typically have lower retention than products that hone in on one type of customer very well.
  • Revenue model: How much money you ask from customers and how can play a big role in retention. For example, a customer may be more likely to retain for a product they marginally like if it costs $30 vs. $300,000. A product that expands revenue per user over time can have lower retention than ones that have a fixed price.
  • Natural frequency: Many products have different natural frequencies. For example, you may only look for a place to live once every few years (like my time at Apartments.com), but you look for something to eat multiple times of day (like my time at Grubhub).
  • Acquisition strategy: The way a company acquires users affects its retention. A wide spread approach to new users may retain worse than carefully targeting users to bring to your product.
  • Network effects: Network effects may drive retention rates up more over time vs. businesses that do not have these effects. For example, all of your friends on Facebook or all of your co-workers on Slack makes it hard to churn from either product whereas churning from Calm or Grammarly is entirely up to you.

* In those businesses, the business failing and churning as a result is called “involuntary churn”, though that can also mean a payment method not working for someone who wants to retain in other models.

BONUS: Why are Casey’s benchmarks for consumer transactional businesses lower than others?

For the demand side of transactional businesses, where the retention graph flattens is more important to me than the six month retention rate. And unlike other models, these businesses can take longer than six months to have their graphs flatten. Also, for marketplaces, one of the two common models along with ecommerce in this category, a healthy demand side retention rate is very dependent on what supply side retention looks like and acquisition costs. For example, since Uber and Lyft have to spend so much time and money acquiring drivers due to a low retention rate, in order for their model to work, demand side retention either has to be high or demand side acquisition has to be low cost. For a business where supply side retention is high and acquisition costs are low, demand side retention can be lower, and the company can still be very successful. Etsy and Wag I imagine fit more into this model.

Currently listening to We All Have An Impact by Boreal Massif.

What Are Growth Teams For, and What Do They Work On?

This blog post was adapted from a presentation I did recently. Hence, slides. Don’t say I didn’t warn you.

I receive a lot of questions about growth teams. Naturally, there is a lot of confusion. Is this marketing being re-branded? Who does this team report to? What is the goal of it? What do they actually work on? When do I start a growth team for my business?

The purpose of growth is to scale the usage of a product that has product-market fit. You do this by building a playbook on how to scale the usage of a product. A playbook can also be called a growth model or a loop.

The first question you should ask before asking about growth is if you have product-market fit?

The traditional definition above is qualitative, and if you’re like me, you like to have data to answer questions. The best way to get that data for most businesses is to measure retention.

The best way to identify the key action is to find a metric that means the user must have received value from your product. The best way to understand the frequency on which you should measure that metric is how often people solved this problem before your product existed. Let’s look at some examples from my career.

For Pinterest, a Pinner receives value if we showed them something cool related to their interests. The best way to determine if the Pinner thought something we showed them was cool is that they saved it.

For Grubhub, this was even easier to determine. People only receive value if they order food, and when we surveyed people, they ordered food once or twice a month (except for New York).

Once you have a key metric and a designated frequency, you can graph a retention curve or a cohort curve. If it flattens out, that means some people are finding continual value in the product. But that is not enough.

Brian Balfour has a great post on this, which he calls product-channel fit.

If you’ve been around startups for a while, you might remember this tweet from Paul Graham. It talked about the fastest growing startup Y Combinator has ever funded. It is a graph of revenue growth from $0 to $350,000 per month in just 12 months.

The startup was Homejoy, an on demand cleaning service. Investors liked this graph, so they gave the company $38 million to expand.

20 months later Homejoy shut down. From a post-mortem of the company, I highlight the following quote.

If you discounted to get to product-market fit, you didn’t get to product-market fit. Product-market fit is not revenue growth, it’s not growth in users, it’s not being #1 in the App Store. Product-market fit is retention that allows for sustained growth.

So, I though product teams were in charge of creating a product people loved, and marketing teams were in charge of getting people to try the product. What changed?

What changed is an acknowledgement of what actually drives startup growth. There are three main levers. Phase I is simultaneously the most important and the least understood. In Phase I, you change the product to increase its growth rate. Some changes include improving onboarding, helping the product acquire more customers through activities like virality or SEO, incresing the conversion rate, et al.

These initiatives are “free” in that they don’t require an advertising budget. Their cost is the opportunity cost of a product team’s time. They are measurable in that you can create an experiment and understand the exact impact of the change. They are also scalable in that if you make a change that, say, improves your conversion rate, and it has a certain amount of impact, it likely will have that same impact tomorrow, weeks from now, and potentially even years from now.

The other two phases are what we traditionally think of as marketing. Performance marketing initiatives, like buying ads on Facebook or Google, are also measurable and scalable, but scale with an advertising budget. Brand marketing usually requires an even larger ad budget, and is harder to measure or scale. The time frame over which brand marketing works takes years, and can be hard to confirm. If you do create a PR campaign or a TV ad that seems to work on a more immediate time frame, it can be hard to scale. that is because brand marketing always requires new stories to keep people’s attention.

This is why marketing can’t be in charge of all growth initiatives. They don’t have access or capability to the most important ones. They might know they need to improve the site’s conversion rate or get more traffic from referrals, but they don’t have access to the product roadmap to get them prioritized appropriately, and if they do get engineering and design help, they don’t have the expertise in working with them to build the best solutions.


Perhaps what’s more important to understand in the difference between marketing and growth is how the traditional marketing funnel changes with startups. Above is the traditional marketing. This model is based on the old school model of product development before the internet in spending a lot of money to make people want things.

Startups by definition should be making things people already want. When you do that, you can invert the funnel and focus on people that already want the product or people that are already using it. This is more effective on a small (or no) budget.

When you translate that into tactics, you see how product-driven growth initiatives dominate the top of the list of priorities. It does not mean you won’t work on performance marketing or brand marketing, but that they usually become important later on in a product lifecycle as an accelerant to an already sustainably growing company.

So I spent a lot of time explaining why growth is different from marketing. How is different from product?

Growth teams don’t create value. They make sure people experience the value that’s already been created.

The most common examples to start a growth team to address are:

  • improving the logged out experience (for conversion or SEO)
  • sending better emails and/or notifications
  • increasing referrals or virality
  • improving onboarding

SEO and onboarding are harder places to start because their iteration cycles are much longer than the other areas.

Growth teams don’t start by finding mythical VP’s of Growth to come in and solve all of your problems. They are usually started by existing employees at a company (or founders) that really understand the company and what’s preventing it from growing faster. They report to their dedicated functions, but sit together to focus on problem solving.

To find out which area you work on after you have the team, you have to analyze the data. For example, at Pinterest, they originally wanted me and my team to work on SEO. What we saw was that while there was a lot of opportunity to get more traffic via SEO, a bigger issue was the conversion rate from that traffic. So we decided to work on conversion instead.

Then we had to figure out what to work on. Jean, an engineer on the team, had recently run an experiment that gave us a key insight. So, we said, we could use this same modal when people clicked on Pins. Clicking on a Pin could show enough interest in Pinterest for you to want to sign up.

The other thing people did when they liked what they saw scroll to see more. So, we decided to try stopping them where we stopped the Google crawler, and asking them to sign up then.

It took Jean two days of work to launch this experiment, and it resulted in a much bigger impact than expected.

So that’s an example of finding a conversion issue in you data, and putting together a really scrappy experiment to try to improve it. What else can growth teams work on? Here are some examples from my time at Pinterest, and some best practices we’ve learned.

Usually, the biggest area a growth team focuses on improving is retention. That’s right; growth teams are not just about acquisition. Retention comes from a maniacal focus on improving the core product, which I define as core product, not growth, work. Where growth comes in is reducing friction to experience that core product. Simplifying how the current product works usually has much more impact than adding new features. New features complicate the product, making it harder for new people to understand.

So how do you simplify the core product? Well, you have to have data to understand what people do, and pair it with qualitative research to understand why they do it. We spent countless hours at Pinterest putting laptops in front of non-users watching them sign up for the product to figure out why people didn’t activate.

At Grubhub, data pointed out that Grubhub was an S curve when it came to both conversion and retention. This graph is a (now very old) graph of conversion rate in Boston based on how many restaurants Grubhub returned when you searched your address. After 55 results, conversion rate essentially doubled for new and returning users.

Qualitative research gave us different insights. When we asked users why they didn’t use Grubhub more often, they would say, “it’s expensive.” We thought that was weird, because Grubhub wasn’t charging them anything. What they meant is that delivery was expensive due to minimums and delivery fees. So, we went back to our restaurants, convinced a few to try lowering their minimums and fees to see if increased volume could make up for lower margin. When it did, we creates case studies to help convince other restaurants.

At Pinterest, we simplified the signup and onboarding flow. What used to be a flow that required five steps was now three with one of them pre-filled and the other two optional. What we did do was introduce friction that we knew made it more likely a Pinner would find content they care about. This was asking them which topics interested them before showing them a feed of content.

We also realized that the more content people see, the more likely they will find something they like, which will lead to retention. So, we removed content around Pins that was non-critical, like who Pinned it to what board and how they described it. All of these increased activation rates.

We also contextually educated people on what to do next when they were onboarding. There is a common saying that if you need to add education to your design, it’s a bad design. It’s pithy and sounds smart, but it’s actually dangerous. My response is that a design with education is better than a design that doesn’t educate.

Search engine optimization has been a really great lever for organic growth for every company I’ve worked on in my career. It’s not for every business though. People need to already be searching for what you do. That alone isn’t enough though. You need to be an authority on the subject, which Google determines by relevant external links to your domain and your content. You also need to be relevant to what was just searched.

We worked on improving both of these at Grubhub. When Grubhub launched new market, by definition we weren’t locally relevant yet. So we would go to local blogs and press outlets and tell them we were launching there, and that we wanted to give their readers $10 off their first order. All they had to do was link to a page where the discount would auto-apply. After a while, that page would have enough local links so that even though the promotional discount was over, it would still rank #1 for the local delivery terms e.g. “san francisco food delivery”.

For relevance, Grubhub knew which restaurants delivered where, their menu data, and reviews from real people. So we aggregated them into landing pages for every locale + cuisine combination e.g. ‘nob hill chinese delivery”.

We applied the same landing page strategy at Pinterest. While Pinners had created boards on their favorite topics, it was one person’s opinion on what was relevant for a topic. Pinterest has repin data globally for every topic, so we knew what the best Pins were across the Pinterest community. So we created topic pages with the best Pins, and they performed better than individual boards on search engines and with search engine users.

We also worked a lot on emails and notifications on the Pinterest growth team. Emails are a key driver of retention. They won’t solve your retention problem, but they will certainly help if you do them right. At every company I have been at, people hated email and didn’t want to send them to their customers. When they finally did, they saw lifts. You are not your customer. You get more email than they do. Emails help them if they’re connected to the core value of why they use your product. Emails are not helpful if they’re pushing a marketing message.

At Pinterest, I made this mistake. I set up campaigns with emails that explained all of the things Pinterest could do. People don’t care about what Pinterest can do. They care about seeing cool content related to their interests. We needed to stop sending email like a marketer, and start sending email like a personal assistant. So we replaced those emails with popular content in topics of interest for each Pinner, and our retention increased.

Then, we built a system around it. Each Pinner likes different content, at different times, and different amount of it. So we learned for each Pinner what content they liked, when they liked to receive emails and notifications (based on when they opened them), and how much they liked to receive based on testing different volumes and seeing open rate impacts.

If you’re testing emails and notifications, you can test manually first, then automate and personalize. What I have learned at Pinterest and Grubhub is what seems to be worth testing. At Pinterest, one engineer tested 4,500 different subject lines, resulting in hundreds of thousands of additional weekly active users. Around the same time, we spent three months redesigning all of our emails, and it had no impact on usage.

A common issue I see with growth and marketing teams is they think that emails and notifications can only have positive impact. This is not true. You have to measure the lift in usage vs. the unsubscribes (and the impact of an unsubscribe) and app deletions. Those will impact usage, and you need to know how.

—

Growth teams have a clear purpose, and that purpose makes sense only if you have first found product-market fit. Once you have that, you will find traditional product and marketing lacking in their ability to help scale usage of your product. That’s where growth teams come in. Growth teams use data and qualitative research to help understand the frictions that prevent more people from finding the value in your product. That can mean acquisition, but it can also mean reducing friction in the core product, working on conversion or onboarding, or finding ways to remind existing users about the value you’re creating. If you have questions about growth teams, don’t hesitate to reach out to cwinters@greylock.com.

This presentation was made in conjunction with @omarseyal, who is awesome.

Currently listening to Everybody Works by Jay Som.

Why Onboarding is the Most Crucial Part of Your Growth Strategy

When people talk about growth, they usually assume the discussion is about getting more people to your product. When we really dig into growth problems, we often see that enough people are actually coming to the products. The real growth problems start when people land… and leave. They don’t stick. This is an onboarding problem, and it’s often the biggest weakness for startups. It can also take the longest to make meaningful improvements when compared to other parts of the growth funnel.

In my role as Growth Advisor-in-Residence at Greylock, I talk to startups in the portfolio about getting new users to stick around. Through many failed experiments and long conversations poring over data and research, I have learned some fundamental truths about onboarding. I hope this can function as a guide for anyone tackling this problem at their company.

What is Successful Onboarding?
Before you can fix your onboarding efforts, you need to define what successful onboarding is to you. What does it mean to have someone habitually using your product? Only then can you measure how successful you are at onboarding them. To do so, you need to answer two questions:

  • What is your frequency target? (How often should we expect the user to receive value?)
  • What is your key action? (The action signifies the user is receiving enough value to remain engaged)

To benchmark frequency, look at offline analogs. At Grubhub, we determined how often people ordered delivery by calling restaurants. The answer was once or twice a month, so we used a “once a month” as a benchmark for normal frequency for Grubhub. At Pinterest, the analog was a little harder to determine, but using Pinterest was most like browsing a magazine, which people read weekly or monthly. So we started with monthly, and now they look at weekly metrics.

Identifying the key action can be easy or hard — it depends on your business. At Grubhub, it was pretty easy to determine. You only received value if you ordered food, so we looked at if you placed a second order. At Pinterest, this was a little harder to determine. People derive value from Pinterest in different ways, from browsing lots of images to saving images to clicking through to the source of content. Eventually, we settled on saving (pinning an image to your board), because, while people can get value from browsing or clicking through on something, we weren’t sure if it was satisfying. You only save things if you like them.

Once you know your key action and your frequency target, you have to track that target over time. You should be able to draw a line of all users who sign up during a specific period, and measure if they do the key action within the frequency target after signup. For products with product/market fit, the line flattens as a percentage of the users complete the key action every period:

If the line flattens rather quickly, your successful activation metric is people who are still doing [key action] at [set interval] at [this period after signup]. So, for Pinterest, that was weekly savers four weeks after signup. If your cohort takes a longer time to flatten, you measure a leading indicator. At Grubhub, the leading indicator was a second order within thirty days of first order.

How should you research onboarding?
You can break down cohort curve above into two sections. The part above where the curve flattens are people who “churn”, — or did not receive enough value to make the product a habit. The people below where the curve flattens have been successfully onboarded.

To research onboarding, talk to both groups of people to get their thoughts. I like to do a mix of surveys, phone calls, and qualitative research using the product. I usually start with phone calls to see what I can learn from churners and activators. Our partner Josh Elman talks about best practices to speaking with churners, or bouncebacks. If I am able to glean themes from those conversations, I can survey the broader group of churners and activators to quantify the reasons for success and failure to see which are most common. (Sidenote: You’ll need to incentivize both groups to share their thoughts with you. For those that didn’t successfully activate, give them something of value for their time, like an Amazon gift card or money. For those that did, you may be able to give them something free in your product.)

But it is not enough to just talk to people who already have activated or churned. You also want to watch the process as it’s happening to understand it deeper. In this case, at Pinterest, we brought in users and watched them sign up for the product and go through the initial experience. When we needed to learn about this internationally, we flew out to Brazil, France, Germany and other countries to watch people try to sign up for the product there. This was the most illuminating part of the research, because you see the struggle or success in real time and can probe it with questions. Seeing the friction of international users first hand allowed us to understand it deeper and focus our product efforts on removing that friction.

The principles of successful onboarding
#1: Get to product value as fast as possible — but not faster
A lot of companies have a “cold start problem” — that is, they start the user in an empty state where the product doesn’t work until the user does something. This frequently leaves users confused as to what to do. If we know a successful onboarding experience leads to the key action adopted at the target frequency, we can focus on best practices to maximize the number of people who reach that point.

The first principle we learned at Pinterest is that we should get people to the core product as fast as possible — but not faster. What that means is that you should only ask the user for the minimum amount of information you need to get them to the valuable experience. Grubhub needs to know your address. Pinterest needs to know what topics you care about so they can show you a full feed of ideas.

You should also reinforce this value outside the product. When we first started sending emails to new users at Pinterest, we sent them education on the features of Pinterest. When Trevor Pels took a deeper look at this area, he changed the emails to deliver on the value we promised in the first experience, instead of telling users what we thought was important about the product. This shift increased activation rates. And once the core value is reinforced, you can actually introduce more friction to deepen the value created. When web signups clicked on this content on their mobile devices, we asked them to get the app, and because they were now confident in the value, they did get the app. Conversely, sending an email asking users to get the app alone led to more unsubscribes than app downloads.
Many people will use this principle as a way to refute any attempts to add extra steps into the signup or onboarding process. This can be a mistake. If you make it clear to the user why you are asking them for a piece of information and why it will be valuable to them, you can actually increase activation rate because it increases confidence in the value to be delivered, and more actual value is delivered later on.

Principle #2: Remove all friction that distracts the user from experiencing product value
Retention is driven by a maniacal focus on the core product experience. That is more likely to mean reducing friction in the product than adding features to it. New users are not like existing users. They are trying to understand the basics of how to use a product and what to do next. You have built features for existing users that already understand the basics and now want more value. New users not only don’t need those yet; including them makes it harder to understand the basics. So, a key element of successful onboarding is removing everything but the basics of the product until those basics are understood. At Pinterest, this meant removing descriptions underneath Pins as well as who Pinned the item, because the core product value had to do with finding images you liked, and removing descriptions and social attribution allowed news users to see more images in the feed.

Principle #3: Don’t be afraid to educate contextually
There’s a quote popular in Silicon Valley that says if your design requires education, it’s a bad design. It sounds smart, but its actually dangerous. Product education frequently helps users understand how to get value out of a product and create long term engagement. While you should always be striving for a design that doesn’t need explanation, you should not be afraid to educate if it helps in this way.

There are right and wrong ways to educate users. The wrong way: show five or six screens when users open the app to explain how to do everything — or even worse, show a video. This is generally not very effective. The right way: contextually explain to the user what they could do next on the current screen. At Pinterest, when people landed on the home feed for the first time, we told them they could scroll to see more content. When they stopped, we told them they could click on content for a closer look. When they clicked on a piece of content, we told them they could save it or click through to the source of the content. All of it was only surfaced when it was contextually relevant.

Onboarding is both the most difficult and ultimately most rewarding part of the funnel to improve to increase a company’s growth. And it’s where most companies fall short. By focusing on your onboarding, you can delight users more often and be more confident exposing your product to more people. For more advice on onboarding, please read Scott Belsky’s excellent article on the first mile of product.

Currently listening to Easy Pieces by Latedeuster.