Tag Archives: user experience

Feature/Product Fit

Through various methods, Silicon Valley has drilled into the minds of entrepreneurs the concept of product/market fit. Marc Andreessen says it’s the only thing that matters, and Brian Balfour has an amazing series of posts that talk about how to find it. But what happens after you find product/market fit? Do you stop working on product? I think most people would argue definitely not. Post-product/market fit, companies have to balance work creating new product value, improving on the current product value, and growing the number of people who experience the current value of the product. While I have written a lot about how to do that third step, and even wrote a post about thinking about new product value, I haven’t written much about how do that second part.

What is Feature/Product Fit?
Every product team tries to make their core product better over time. But sadly, at most companies, the process for this is launching new features and hoping or assuming they are useful to your existing customers. Companies don’t have a great rubric for understanding if that feature is actually valuable for the existing product. This process should be similar to finding product/market fit, but there are some differences. I call this process feature/product fit, and I’ll explain how to find it.

In product/market fit, there are three major components you are searching for. I have written about my process for product/market fit, and Brian Balfour, Shaun Clowes, and I have built an entire course about the retention component. To give a quick recap from my post though, you need:

  • Retention: A portion of your users building a predictable habit around usage of your product
  • Monetization: The ability at some point in the future to monetize that usage
  • Acquisition: The combination of the product’s retention and monetization should create a scalable and profitable acquisition strategy

Feature/Product Fit has a similar process. We’ll call this the Feature/Product Fit Checklist:

  • The feature has to retain users for that specific feature
  • The feature has to have a scalable way to drive its own adoption

Feature/Product Fit has a third requirement that is a bit different: the feature has to improve retention, engagement, and/or monetization for the core product.

This last part can be a bit confusing for product teams to understand. Not only do the products they are building need to be used regularly and attract their own usage to be successful, they also need to make the whole of the product experience better. This is very difficult, which is why most feature launches inside companies are failures. What happens when a feature has retention and adoption, but does not increase retention, engagement, or monetization for the company? This means it is cannibalizing another part of the product. This might be okay. As long as those three components do not decrease, shipping the feature might be the right decision. The most famous example of this is Netflix introducing streaming so early in that technology’s lifecycle, which cannibalized the DVD by mail business, but was more strategic for them long term.

What is a Feature Team’s Job?
You would be surprised how many core product features are shipped when the new feature decreases one of those three areas. How does this happen? It’s very simple. The team working on the feature is motivated by feature usage instead of product usage, so they force everyone to try it. This makes the product experience more complicated and distracts from some of the core product areas that have feature/product fit.

If you own a feature (and I’m not saying it’s the right structure for teams to own features), your job is not to get people to use that feature. Your job is to find out if that feature has feature/product fit. You do this by checking the three components listed above related to feature retention, feature adoption, and core product retention, engagement and monetization. During this process, you also need to determine for which users the feature has feature/product fit (reminder: it’s almost never new users). Some features should only target a small percentage of users e.g. businesses on Facebook or content creators on Pinterest. Then and only then does your job shift to owning usage of that feature. And in many teams, it’s still not your job. The feature then becomes a tool that can be leveraged by the growth team to increase overall product retention, engagement, and monetization.

Mistakes Feature Teams make searching for feature/product fit
Feature teams commonly make mistakes that dissatisfy the third component of feature/product fit at the very beginning of their testing.

  • Mistake #1: Email your entire user base about your new feature
    Your users do not care about your features. They care about the value that you provide to them. You have not proven you provide value with the feature when you email them early on. Feature emails in my career always perform worse than core product emails. This inferior performance affects the value of the email channel for the entire product, which can decrease overall product retention.
  • Mistake #2: Put a banner at the top of the product for all users introducing the new feature
    New features usually target specific types of users, and are therefore not relevant to all users. They are especially irrelevant to new users who are trying to learn the basics about the core product. These banners distract from them, decreasing activation rates. It’s like asking your users if they’re interested in the Boston Marathon when they don’t know how to crawl yet.
  • Mistake #3: Launch with a lot of press about the new feature
    PR for your feature feels great, but it won’t help you find feature/product fit. PR can be a great tool to reach users after you have tested feature/product fit though. It should not happen before you have done experiments that prove feature/product fit. And it will not fix a feature that doesn’t have feature/product fit.

Many features won’t find feature/product fit
Many of the features product teams work on will not find feature/product fit. When this happens, the features need to be deleted. Also, some older features will fall out of feature/product fit. If they cannot be redeemed, they also need to be deleted. If you didn’t measure feature/product fit for older features, go back and do so. If they don’t have it, delete them. Some of our most valuable work at Pinterest was deleting features and code. A couple examples:

  • The Like button (RIP 2016): People did not know how to use this vs. the Save button, leading to confusion and clutter in the product
  • Place Pins (RIP 2015): Pinterest tried to create a special Pin type and board for Pins that were real places. As we iterated on this feature, the UI drifted further and further away from core Pinterest Pins and boards, and never delivered Pinner value
  • Pinner/Board Attribution in the Grid (RIP 2016): Attributing Pins to users and their boards made less sense as the product pivoted from a social network to an interest network, and cluttered the UI and prevented us from showing more content on the screen at the same time

How do I help my feature find Feature/Product Fit?
All features should be launched as experiments that can test for feature/product fit. During this experiment, you want to expose the new feature to just enough people to determine if it can start passing your feature/product fit checklist. For smaller companies, this may mean testing with your entire audience. For a company like Pinterest, this might start with only 1% of users. The audience for these experiments is usually your current user base, but can be done through paid acquisition if you are testing features for a different type of user.

I’ll give you a few tactics that have helped the companies I’ve worked on find feature/product fit over the years. Most good product development starts with a combination of data analysis and user research. User research should be involved at the right times to add the most impact, prevent confirmation bias, and determine what components users are struggling to see the value of. For example, when we launched the Grubhub mobile app, we saw in the data when people used the current location feature, their conversation rate was lower than people who typed in their address. This turned out to be an accuracy problem, so we turned that component until off until we were able to improve its accuracy.

In research, we saw people were having trouble figuring out which restaurants to click on in search results. On the web, they might open up multiple tabs to solve this problem, but this was not possible in the app. So, we determined what information would help them decide which restaurants were right for them, and started adding that information into the search results page. That included cuisine, estimated delivery time, minimum, star rating, number of ratings, On the surface, the page now feels cluttered, but this improved conversion rates and retention.

Since Grubhub is a transactional product, we were able to leverage incentives as a strategy to help find feature/product fit. Our early data showed that people who started to use the mobile app had double the lifetime value of web users. So, we offered $10 off everyone’s first mobile order. This transitioned web users to mobile for the first time and acquired many people on mobile first. The strategy was very successful, and the lifetime value improvements remained the same despite the incentive.

Grubhub also uses people to help find feature/product fit. Since mobile apps were new at the time (and we were the first food delivery app), we monitored any issues on social media, and had our customer service team intervene immediately.


Not all complaints are this entertaining.

At Pinterest, we launched Related Pins in 2013. For Pinterest, we did not have a revenue model at the time, so tactics around incentives and people do not make as much sense. One thing we did instead was use notifications to drive feature/product fit. Once this algorithm was developed, after you Pinned something, we could now email you more Pins you might like that are related to that Pin. These emails were very successful.

Pinterest also used the product to drive feature/product fit. We launched the algorithm results underneath the Pin page to start, and interaction when people scrolled was great. But many people didn’t scroll below the Pin. So, we tried moving them to the right of the Pin, which increased engagement, and we started inserting related Pins of items you saved into the core home feed as well, which increased engagement.

The Feature/Product Fit Checklist
When helping a product find feature/product fit, you should run through this checklist to help your feature succeed:

  • What is the data telling me about usage of the feature?
  • What are users telling me about the feature?
  • How can I use the core product to help drive adoption of the feature?
  • How can I use notifications to help drive adoption of the feature?
  • How can I use incentives to help drive adoption of the feature?
  • How can I use people to help drive adoption of the feature?

When confirming feature/product fit, you need to ask:

  • Is the feature showing retention?
  • What type of user(s) is retaining usage of the feature?
  • How do I limit exposure to only those users?
  • What is the scalable adoption strategy for the feature for those users?
  • How is this feature driving retention, engagement, or monetization for the overall product?

Every company wants to improve its product over time. You need to start measuring if the features you’re building actually do that. You also need to measure if existing features are adding value, and if not, start deleting them. Asking these questions when you build new features and measure old features will make sure you are on the path to having features that find feature/product fit and add value to users and your business.

Thanks to Omar Seyal and Brian Balfour for reading early drafts of this.

Currently listening to Breaking by Evy Jane.

The Four Types of Traffic to Your Home Page

Every founder’s favorite project is redesigning their home page. It’s the introduction of your company, brand, mission, and hopes and dreams to all of your customers. At least, you think it is. When you analyze your actual business, you might find the majority of new customers don’t actually start on your home page. If you’re doing paid acquisition, they might land on specific landing pages designed for those ads. If you’re growing via social traffic or SEO, most people are landing on specific pieces of content. The typical growth team’s response to this urge is that the home page is not a high priority, and that they should work on landing pages or content pages This is correct. But the home page for some businesses is still a major source of traffic. You have to learn what type of traffic that is in order for a redesign of it not be a complete waste of time. I’ll talk about what those types of traffic are, how to identify them, and what types of designs you might pursue based on your mix of traffic.

Traffic Type #1: People who want to login
On sites with a lot of engagement, a lot of the traffic to a home page is from people who are already customers and want to login. In this case, your goal is to just make logging iin as easily as possible or auto-log them in via something like Google Smart Lock. Better yet, if you can make sure you never logout your customers, then they never see this page and get right back into the product. Facebook, Pinterest, and Tumblr are good examples of this.

While Facebook has sign up front and center, it has a top navigation bar for login, and that is where the cursor begins.

Pinterest has a unified sign up and login field with a friendly call to action that just says continue. It also has a dedicated Login button for those searching for it.

Tumblr has a login button front and center equal to the signup button.

The metric when you are optimizing for logins is login conversion rate. This may be harder to calculate than you think. Let’s do an example. Let’s say you have 10,000 people visit your home page. 2,000 login. Let’s say another 500 sign up on this page. Your login conversion rate is 2,000 / (10,000 – 500) = ~21%. You don’t actually know if the remaining 7,500 who did not login or sign up were there to sign up or login. If you have cookie data, you can check to see if they had ever logged in, and that may give a clue on how to segment them. If you don’t have that data, you assume they were there to login. If at all possible, you should never log people out. The best way to help them login is to keep them logged in. Tools like Google SmartLock are also effective.

Traffic Type #2: People who want to sign up
Someone coming to your home page directly is not doing so because they randomly type it into a browser. They already have some context. A friend told them about it, they read an article about, or something similar. Many of those people are already convinced and want to sign up, and the job of the home page is to get out of their way and make that as easy as possible. Generally, sites do this by putting a sign up form front and center, and making that form really easy to fill out. You can see how Facebook and Pinterest do this really well. If you look at the images above, Facebook’s signup formis right aligned, and Pinterest’s is front and center. There isn’t much to distract you from signing up.

The metric when you are optimizing for signups is signup conversion rate. It is similar to login conversion rate, just with signup as the numerator instead of logins, and logins are subtracted by the numerator. Given the same number from above, your signup conversion rate is 500 / (10,000 – 2,000) = ~6%. You still don’t actually know if the remaining 7,500 who did not login or sign up were there to sign up or login. So, to be conservative, you assume they were there to sign up.

Traffic Type #3: People who want to learn more
There can be a wide discrepancy between the people who come to your home page directly, and are not existing users. Some may want to sign up quickly like above, but others just want to learn more. Preferably, they would not like to have to give you their information before they understand if the site is for them. The ideal scenario for these these users is to see a free preview. If personal information is required to show a convincing product, then the home page sells the value instead of shows it, or asks for filtering criteria without asking for personal information. Tumblr, Pinterest, and Facebook all address this type of user in different ways. Facebook, as seen in the example above, left aligns the explanation of Facebook on the page. Pinterest and Tumblr create a separate call to action to learn more that triggers a dedicated “learn more” experience. Pinterst has a red call out with a How It Works button, and Tumblr has a clickable green bar at the bottom of the page asking “What is Tumblr?”. Both clicks result in scrolling explanations of what you can use these products for.
Tumblr:


Pinterest:

Traffic Type #4: People who are skeptical
There is a fourth type of new user that visits your home page. This is the person who heard about it, but is very skeptical. The best way to engage this user is to let them have free usage of the product for as long as possible. You usually encounter these users with very well penetrated products that are reaching the very late majority or laggards. Google is a good example of a site that lets you experience the product value without signing up.

You can see from the above examples that companies try to address multiples of these audiences at the same time on the page. In some cases, based on activity of the page, they will be able to bucket you into one of these groups definitively. Gibson Biddle has a great post on how Netflix evolved their design with regards to these different types of users and these users’ understanding of their brand over time.


As you’re working on your home page, you should first make sure it is the most important page to work. For many sites, their landing pages are where a lot more people get introduced to the product. When you do work on your home page, think about these four audiences, which one you really need to optimize for, and if you easily segment and address the other groups that are not your primary focus. Also, be sure to revisit thes decisions over time as audiences and brand awareness changes.

Currently listening to Mind Bokeh by Bibio.

Why Onboarding is the Most Crucial Part of Your Growth Strategy

When people talk about growth, they usually assume the discussion is about getting more people to your product. When we really dig into growth problems, we often see that enough people are actually coming to the products. The real growth problems start when people land… and leave. They don’t stick. This is an onboarding problem, and it’s often the biggest weakness for startups. It can also take the longest to make meaningful improvements when compared to other parts of the growth funnel.

In my role as Growth Advisor-in-Residence at Greylock, I talk to startups in the portfolio about getting new users to stick around. Through many failed experiments and long conversations poring over data and research, I have learned some fundamental truths about onboarding. I hope this can function as a guide for anyone tackling this problem at their company.

What is Successful Onboarding?
Before you can fix your onboarding efforts, you need to define what successful onboarding is to you. What does it mean to have someone habitually using your product? Only then can you measure how successful you are at onboarding them. To do so, you need to answer two questions:

  • What is your frequency target? (How often should we expect the user to receive value?)
  • What is your key action? (The action signifies the user is receiving enough value to remain engaged)

To benchmark frequency, look at offline analogs. At Grubhub, we determined how often people ordered delivery by calling restaurants. The answer was once or twice a month, so we used a “once a month” as a benchmark for normal frequency for Grubhub. At Pinterest, the analog was a little harder to determine, but using Pinterest was most like browsing a magazine, which people read weekly or monthly. So we started with monthly, and now they look at weekly metrics.

Identifying the key action can be easy or hard — it depends on your business. At Grubhub, it was pretty easy to determine. You only received value if you ordered food, so we looked at if you placed a second order. At Pinterest, this was a little harder to determine. People derive value from Pinterest in different ways, from browsing lots of images to saving images to clicking through to the source of content. Eventually, we settled on saving (pinning an image to your board), because, while people can get value from browsing or clicking through on something, we weren’t sure if it was satisfying. You only save things if you like them.

Once you know your key action and your frequency target, you have to track that target over time. You should be able to draw a line of all users who sign up during a specific period, and measure if they do the key action within the frequency target after signup. For products with product/market fit, the line flattens as a percentage of the users complete the key action every period:

If the line flattens rather quickly, your successful activation metric is people who are still doing [key action] at [set interval] at [this period after signup]. So, for Pinterest, that was weekly savers four weeks after signup. If your cohort takes a longer time to flatten, you measure a leading indicator. At Grubhub, the leading indicator was a second order within thirty days of first order.

How should you research onboarding?
You can break down cohort curve above into two sections. The part above where the curve flattens are people who “churn”, — or did not receive enough value to make the product a habit. The people below where the curve flattens have been successfully onboarded.

To research onboarding, talk to both groups of people to get their thoughts. I like to do a mix of surveys, phone calls, and qualitative research using the product. I usually start with phone calls to see what I can learn from churners and activators. Our partner Josh Elman talks about best practices to speaking with churners, or bouncebacks. If I am able to glean themes from those conversations, I can survey the broader group of churners and activators to quantify the reasons for success and failure to see which are most common. (Sidenote: You’ll need to incentivize both groups to share their thoughts with you. For those that didn’t successfully activate, give them something of value for their time, like an Amazon gift card or money. For those that did, you may be able to give them something free in your product.)

But it is not enough to just talk to people who already have activated or churned. You also want to watch the process as it’s happening to understand it deeper. In this case, at Pinterest, we brought in users and watched them sign up for the product and go through the initial experience. When we needed to learn about this internationally, we flew out to Brazil, France, Germany and other countries to watch people try to sign up for the product there. This was the most illuminating part of the research, because you see the struggle or success in real time and can probe it with questions. Seeing the friction of international users first hand allowed us to understand it deeper and focus our product efforts on removing that friction.

The principles of successful onboarding
#1: Get to product value as fast as possible — but not faster
A lot of companies have a “cold start problem” — that is, they start the user in an empty state where the product doesn’t work until the user does something. This frequently leaves users confused as to what to do. If we know a successful onboarding experience leads to the key action adopted at the target frequency, we can focus on best practices to maximize the number of people who reach that point.

The first principle we learned at Pinterest is that we should get people to the core product as fast as possible — but not faster. What that means is that you should only ask the user for the minimum amount of information you need to get them to the valuable experience. Grubhub needs to know your address. Pinterest needs to know what topics you care about so they can show you a full feed of ideas.

You should also reinforce this value outside the product. When we first started sending emails to new users at Pinterest, we sent them education on the features of Pinterest. When Trevor Pels took a deeper look at this area, he changed the emails to deliver on the value we promised in the first experience, instead of telling users what we thought was important about the product. This shift increased activation rates. And once the core value is reinforced, you can actually introduce more friction to deepen the value created. When web signups clicked on this content on their mobile devices, we asked them to get the app, and because they were now confident in the value, they did get the app. Conversely, sending an email asking users to get the app alone led to more unsubscribes than app downloads.
Many people will use this principle as a way to refute any attempts to add extra steps into the signup or onboarding process. This can be a mistake. If you make it clear to the user why you are asking them for a piece of information and why it will be valuable to them, you can actually increase activation rate because it increases confidence in the value to be delivered, and more actual value is delivered later on.

Principle #2: Remove all friction that distracts the user from experiencing product value
Retention is driven by a maniacal focus on the core product experience. That is more likely to mean reducing friction in the product than adding features to it. New users are not like existing users. They are trying to understand the basics of how to use a product and what to do next. You have built features for existing users that already understand the basics and now want more value. New users not only don’t need those yet; including them makes it harder to understand the basics. So, a key element of successful onboarding is removing everything but the basics of the product until those basics are understood. At Pinterest, this meant removing descriptions underneath Pins as well as who Pinned the item, because the core product value had to do with finding images you liked, and removing descriptions and social attribution allowed news users to see more images in the feed.

Principle #3: Don’t be afraid to educate contextually
There’s a quote popular in Silicon Valley that says if your design requires education, it’s a bad design. It sounds smart, but its actually dangerous. Product education frequently helps users understand how to get value out of a product and create long term engagement. While you should always be striving for a design that doesn’t need explanation, you should not be afraid to educate if it helps in this way.

There are right and wrong ways to educate users. The wrong way: show five or six screens when users open the app to explain how to do everything — or even worse, show a video. This is generally not very effective. The right way: contextually explain to the user what they could do next on the current screen. At Pinterest, when people landed on the home feed for the first time, we told them they could scroll to see more content. When they stopped, we told them they could click on content for a closer look. When they clicked on a piece of content, we told them they could save it or click through to the source of the content. All of it was only surfaced when it was contextually relevant.

Onboarding is both the most difficult and ultimately most rewarding part of the funnel to improve to increase a company’s growth. And it’s where most companies fall short. By focusing on your onboarding, you can delight users more often and be more confident exposing your product to more people. For more advice on onboarding, please read Scott Belsky’s excellent article on the first mile of product.

Currently listening to Easy Pieces by Latedeuster.

The Right Way to Involve a Qualitative Research Team

Most teams significantly under-invest in qualitative research. Growth teams especially are all about data, but they think that data can only come from experiments. This can make teams overly reliant on what they can learn from experiments and the quality of the data they have, and under-invest from what they can learn from talking to users. This problem is usually exacerbated by the fact that existing researchers at startups aren’t usually assigned directly to teams or work independently. I’ll talk about some of the problems I’ve seen, and the right way to invest in qualitative research for your growth team.

Learning and Applying from Research
Using the right type and method for your question is key. Of course, qualitative research is one component of the research stack along with quantitative research and market research. There is also different types of qualitative research depending on what you are trying to learn.

I remember when I was at Apartments.com and went to my first focus group, a common type of qualitative research. It was a mess for multiple reasons. The first reason was structure. Finding an apartment is not a large social behavior, so why were we talking with a group of ten strangers at once? As what I later learned usually happened, one or two participants volunteered the majority of the feedback, so while we paid for ten people’s opinions, we really only received two people’s opinions. So, I now only do research with multiple people in the room if it’s a social product, and it’s a group that would use it togethers e.g. friends or co-workers.

The second issue was delivering the feedback to people who weren’t there. I wrote up a long perspective on what the issues were with Apartments.com vs. our competitors. It primarily included product feedback on why we were getting crushed by Craigslist in major cities. I sent it to my VP and received a one sentence reply, “Don’t get ahead of yourself.” What a waste of time, I thought. We do all this research, generate real insights, and no one’s interested.

I’ve now learned that research teams inside companies feel this every day. At Pinterest, we had an amazing research team, but they were originally a functional team, which meant they had to determine their own roadmap of what to research. Depending on the stakeholders you listen to, this can be broad strategic projects like “What is the deal with men?” to specific projects like “Help us test this new search flow already built.” Research can add value at both stages, so the team worked on both.

What I think research found when they worked on the broader strategic issues was similar to my response at Apartments.com. “Cool, but not my roadmap!” say the product managers. Research then gets filed away never to be looked at again. Researchers get very frustrated. To be clear, this is a failure of leadership — not the product teams — if these areas aren’t prioritized. But it is common. On the flipside of working on something already built, success was more variable based on how well the product team defined what they wanted to learn. Frequently, what the product team wanted to learn was that they could ship it, so they selectively listened to feedback to things that indicated they were on the right path.

What I have learned suggests that qualitative research cannot be effective unless 1) its people are dedicated a cross-functional product team and 2) research is involved throughout the entire product development process, from initial research on market to determining a strategy to testing concepts to testing nearly finished products. The value of research accrues the more it is a part of each step in the process.

This approach solves for two main problems. One is that product teams will only pay attention to feedback that is directly related to their current product and on their own timeline. Without being part of the cross-functional team that includes product, engineering, and design, it is hard for research to to be on the same timeline. The second problem this solves is it helps research prevent the rest of the team from locking on assumptions that they may be wrong, so they are focused on the right solution to the problem with research, instead of confirmation bias at the end of a project. The Pinterest team has moved to this model, and for my teams, it made both sides much more successful.

When to Research and When to Experiment
For teams that rely too much on experiments and not enough on research, I tell them two things:

  • Experiments are great for understanding what people do and don’t do. Research helps you understand why they do or do not do those things
  • Experiments don’t help you understand the under-represented groups that might be the most important to learn from e.g. non-users or smaller segments of users

A great way to get started with research as a team is to answer why your experiment didn’t work. Sometimes, the answer is there in the experiment data, but frequently it is not. You have to talk to users to understand why they are doing what they are doing. The best way to do that is to ask them the context of them doing or not doing it.

There is also the middle ground of quantitative research that can be helpful (usually surveys). What I usually like to do is use qualitative research to understand the universe of reasons for something, and use quantitative research if I need to quantify the importance/commonality of those reasons.

Research also helps you isolate users you may not be able to isolate with your usage data. For example, at Grubhub, we were trying to understand how many people used Grubhub regularly for delivery, but not for every order. So, we asked. Then, we called those users to understand why they sometimes don’t use Grubhub, then sent another survey with those reasons to quantify which ones were most important to address. I outline that process more here.

But I Don’t Even Have a Research Team
At Grubhub, we didn’t have a research team for the first couple of years (or even a product team for that matter). So, when we needed to learn things, me, someone on my team, or our sole designer (hello Jack!) would do one of three things: 1) throw flows up on usertesting.com, 2) survey users on our email list, or 3) call users on the phone, and provide them with free food for their time.

You don’t need to be a professional researcher to do this, though they are better at it. You just need to determine what you’re trying to learn and who from. You want to watch people go through that situation if you can. If you can’t, ask them about the last time it happened and what they did and why. You will get better at it the more times you do it. Startups are starting to hire researchers earlier in their development because of the importance of understanding users beyond the data. So, you may be able to justify a full time role here earlier than you thought.

Thanks to Gabe Trionfi for reading early drafts of this and providing his feedback. HeHAH!

Currently listening to Beyond Serious by Bibio.

Three Mistakes Integrating Growth and Design Teams and How to Address Them

While I encourage growth teams to start with a dedicated designer, that doesn’t usually happen. Usually, growth teams scale with just engineering and product, and as they scale to a certain size start to eventually “earn” a dedicated designer from the organization. This news makes growth teams extremely happy for a short period of time, but pretty quickly starts to create problems and culture clash. I’ll talk about what happens, and some ways I’ve found to solve these issues. Note: You can replace growth with marketing, and just about the same thing happens in these scenarios.


When you hear you’re getting a dedicated designer, you’re happier than Jim Carrey was back when he had a career.

Problem #1: Team Control AKA The Two Scenarios
One of two scenarios usually emerges when designers join a growth team. In the first scenario, the engineers or PM or marketing person starts with, “I’m so glad you’re here. I need this done and this done and this done for an experiment…tomorrow. No, we don’t need research to understand the problem. Don’t worry. I’ll only ship it if the metrics increase.” The designer realizes the growth team didn’t want a designer. They wanted a pixel monkey to just do what they say, not ever use their brain.


I got 99 problems, but a designer using their brain ain’t one.

The other scenario is just as bad. In this scenario, I call it the designers “moving in”. I liken this to the scenario when a significant other moves into your place and brings way more things than you own to move in. “Don’t worry, I got better furniture and books, and, oh, we’re going to have to get rid of those curtains.”

The design version of this is something like “I’m going to completely rethink all our strategies. I need to spend three months just doing research and then at least double that for design concepts. That home page really needs to change though.” Engineering thinks “this isn’t what we need. These people don’t even know what they’re talking about.” Engineering gets kicked out of the process of figuring out what to work on, and since the design process has no deadlines on it, engineering begins to have nothing to work on.


I tried to find a picture of people dressed in all black moving in, but the best I could do was chambray.

In both scenarios, instead of designers, engineers, product managers, and marketers trying to unify to form one team that leverages all of their strengths, one team tries to dominate the direction. What we did at Pinterest to try to solve this problem was design a process where the teams, specifically design and engineering, are jointly responsible for problem definition and solutions. Here’s what it looks like:

Project Kickoff:
Attendees: PM (leader), engineering lead, design lead, engineer and designer that will be working on the project

Goals:

  • Define the problem we’re trying to solve
  • Any existing ideas on how we might solve it
  • Any previous attempts to solve the problem
  • What metrics we need to inform potential solutions
  • How will we tell if we’re successful

Output:

  • Notes emailed to broader sub-team
  • Slack channel created just for the project, which includes notes and all future communication for the project

Project Brainstorming:
Attendees: engineer and designer on the project, PM (optional so as to prevent their calendar from being a bottleneck)

Goals:

  • Produce multiple potential solutions to the defined problem
  • Prepare those for feedback from attendees of kickoff
  • Cover additional measurement needed for directions that have been chosen e.g. how many people click top right login button

Brainstorm Review:
Attendees: PM, engineering lead, design lead, engineer and designer working on the project (leaders)

Goals:

  • Feedback on concepts from brainstorm
  • Choose one or two directions for experiment

Experiment Launch:
Attendees: engineer and designer working on the project (leaders), PM

Output:

  • Experiment doc that team agrees reflects what we’re testing and why
  • QA and approval over Slack from every team member that experiment is ready to launch to 1%
  • Note: Ramp ups of experiment communicated in Slack channel

Experiment Review:
Attendees: PM (facilitator), engineering lead, design lead, engineer and designer that worked on the project (leaders)

Goals:

  • Determine if you gathered data on key questions you needed to answer for the experiment
  • If yes, what does the data say?
  • If not, how do we get the data?

Output:

  • Ship/kill/iterate decision
  • Emailed notes to the entire team of what happened and why
  • Updated experiment doc on what happened
  • Updated Slack channel
  • Designer brings ship experiment candidates to design manager/lead for any feedback before shipping
    • Problem #2: Design Best Practices
      Designers bring with them a history of best practices from working on other design teams and schooling/training. It can be a culture shock for a designer joining a growth team and seeing little of these being followed. Designers will usually respond to this environment by trying to educate on many things the growth team is managing are designed “wrong” and how they need to be changed. These recommendations based on best practices are usually rejected by engineers on the team as it contradicts what they’ve seen in the data.


      A growth engineer after he hears about a marketing or design best practice.

      Design best practices were created with good purpose, but they quickly become inferior to AB testing in an environment where a direction can be put in front of users and its value quantitatively determined in days or weeks. This doesn’t mean you should AB test every design decision, but it does mean a designer can’t put their foot down by saying something is a best practice. In growth, the users being tested on determine what’s best.

      Another issue with design best practices being applied to growth is that design best practices are usually suited entirely for building user value only. The growth team in many cases needs to trade off user value and business value, or short term user value vs. long term user value. For example, someone can come to my site, and I can offer a great experience. The user then leaves and never comes back. Growth might decide to let someone get a preview of that experience, then ask the user to sign up so they can personalize and deliver more value. The result is that some people will not sign up (creating a worse user experience) and some will sign up, allowing the site to create a better user experience that day and over the long term. This accrues value for the business as well.

      Designers usually feel threatened by this environment initially. They see testing as a threat to their expertise. What you have to do is teach them to use it as a tool to get closer to user feedback at scale and be more efficient with their time. Most design work is wasted because it is spent on a direction that later proves to be flawed. With AB testing, a design can get quick feedback on an idea, validate the direction, then spend the time making it amazing once they know it’s worth that effort.

      That said, while this user feedback at scale is good at showing what users do, it’s not great at explaining why. So, along with spending time getting designers comfortable with testing, growth teams need to start doing more regular qualitative research as well. Designers will usually volunteer to do it themselves if there isn’t the right person to staff for it. Some PM’s are comfortable doing it as well. Engineers and engineering managers can be resistant to spending time watching the sessions, so the first couple you schedule have to be on critical areas you feel there is some understanding the team is missing by only looking at the metrics.

      Once designers get a little more seasoned on growth, you should also work with them to create the new best practices for growth to make the design and engineering process move faster. They will look very different than what designers recommended at the beginning, and be backed by data. Along with this process, I think it’s important to start creating user experience goals for your growth team at this time. These will be components that may not affect the metrics, but ensure a quality and consistent user experience. At Pinterest, we made compliance to design spec a requirement to ship after an experiment is validated, and said our top five flows had to be completely audited for quality user experience. This is a worthy compromise with design to show you actually care about the users and not just the business, a common complaint.

      Problem #3: Growth Onboarding
      Sometimes, you want to make sure you just nail the basics. Most growth designers are joining a new team in a discipline they’ve never worked on before, yet they don’t receive any onboarding as to what growth is and why it’s different. This is also an issue with other new growth team members. It’s like being dropped into a foreign country, not knowing the language, having no one help you, and being expected to contribute to the culture. Like France (okay, I’ve actually been to France, and the people were surprisingly welcoming even though they’re not known for being so). What happens is designers get frustrated and churn from the team.

      It’s critical that every new team member, especially designers, go through a growth team onboarding process. The first thing you do is state the purpose for your growth team, why it’s different from other product teams, and why it’s important. At Pinterest, I would say that while other product teams create value or improve the value of the product, growth teams focus on connecting people to the current value of the product and reducing the friction that prevents that connection. This is important because it’s not enough to build a good product. If no one knows about it, it will die. If it’s too hard to uncover its value, it will also die.

      What we did at Pinterest was create a history of major growth projects that were successful, walked through them, and explained why they were successful. That led into some of our team principles. We also spent a lot of time educating new people on the metrics we use and why. You can’t be expected to design winning experiences if you don’t understand the major criterion for success we’ll be using.

      After onboarding, instead of starting designers on large, complicated projects, it’s important to start them on a well scoped, smaller project, preferably with an experienced PM and engineer, hopefully patient ones. These should still be projects that are worth doing, but not large projects. We had a designer start on growth at Pinterest, and we immediately put her on one of the most strategic, long term investments for the team. While she did great work there, after a few months, she did a smaller side project redesigning our mobile web home page. The conversion rate increased, and we shipped it. She said, “Guys, this is the first time my design increased the metrics!” She was beaming. You want to get new growth designers to that moment as soon as possible.

      Lastly, once onboarded, you want designers contributing growth ideas as soon as possible. I like the idea of forcing people to bring one new idea to the team per week. I believe this is a muscle that needs to be developed via practice. New entrants to the team (design or otherwise) will typically propose bad ideas for a while. That’s okay. The trick is to provide a framework for generating ideas that makes new team members think about elements that typically make for good growth ideas, give feedback on the ideas submitted, and have the ideas be submitted as metrics wins or user experience wins. The Pinterest template focused on :

      • How many people would see this experience if it were built? This is usually by far the most important criterion for a successful growth idea. Any experience has to be seen by a lot of people to have a big possible impact on the business.
      • Has the company tried something in this area before? If so what were the results? This helps us make sure we use any previous learnings to avoid making the same mistakes. Just because we tried something before doesn’t mean it’s a bad idea though as growth environments change frequently.
      • How much effort is required for this idea? On growth, we also try to do the least amount of work to validate an idea is worth it.

      There is no reason designers can’t be key contributors to a growth team, but expecting it to happen automatically is usually a recipe for failure. I hope some of these tips can help you create a thriving cross-functional growth team with all the right disciplines involved. If you’ve have any other issues integrating design into your team, I’d love to hear about them in the comments.

      Currently listening to Sorry I Make You Lush by Wagon Christ.

The Perils and Benefits of AB Testing

Bob, our head of product design at Pinterest, asked me to write up a post on the perils and benefits of AB testing after reading my post on building cross-functional teams. This is me obliging.

One thing it is never difficult to do is to convince an engineer to do an experiment. In general, this is a good thing. Famous engineer W. Edwards Deming said, “In God we trust, all others bring data.” AB experiments generate data, and data settles arguments. AB experiments have helped us move from product decisions made by HIPPO (highest paid person’s opinion) to those made by effectiveness. We build better products as a result that delight more people.

An AB experiment culture can also have a dark side though. Once people figure out that AB experiments can settle disputes where multiple viewpoints are valid, that fact can lead people to not make any decisions at all and test everything. I liken this type of approach to being in a dark room and feeling around for a door. If you blindly experiment, you might find the way out. But turning the light on would make it way easier. Rabid AB testing can be an excuse for not searching for those light bulbs. It can be thought of as easier to experiment than to actually talk to your users or develop a strategy or vision for a product. Here are some perils and benefits of AB testing to think about as you experiment in your organization.

Benefits
1) Quantifying Impact
This one is pretty obvious. AB experiments tell you exactly what the lift or decrease a treatment causes versus a control. So, you can always answer the question of what an investment in Project X did for the company.

2) Limiting Effort on Bad Ideas
Another great benefit of AB testing is that you can receive feedback on an idea before committing a ton of effort into it. Not sure if an investment in a new project is worth it from a design and engineering perspective? Whip up a quick prototype and put it in front of people. If they don’t like it, then you didn’t waste a lot of time on it.

3) Limiting Negative Impact of Projects
Most additions to a product hurt performance. AB testing allows you to test on only a segment of an audience and receive feedback without it affecting all users. Obviously, the larger the company, the smaller the percentage you can trigger an experiment on to get a solid read.

4) Learning What Makes Something Work
In AB experiments, you isolate one variable at a time, so you know exactly what causes a change in metrics. You don’t have to debate about whether it was a headline or background color or the logo size anymore.

Perils
1) Not Building a Strategy or Vision
Many places convince themselves that testing is a strategy in and of itself. While AB testing can inform a strategy or vision, it is not one in and of itself. What happens in these cases is that companies do tons of random experiments in totally different directions, and their failure rate is very high because they have no unifying vision on what they are trying to do.

2) Wasting Time
AB testing can slow companies down when they convince themselves that every single thing needs to be tested thoroughly. Everyone knows of the anecdote from Google where they tested 41 shades of blue.

3) Optimizing for the Wrong Metric
AB experiments are designed to measure the impact of a change on a metric. If you don’t pick the right metric or do not evaluate all of the important ones, you can be making tradeoffs in your product you do not realize. For example, revenue over user engagement.

4) Hitting A Local Maxima
AB experiments do a very good job at helping optimize a design. They do not do as well as helping to identify bold new designs that may one day beat current designs. This leads many companies to stay stuck in a design rut where their current experience is so well optimized, they can’t change anything. You need to be able to challenge assumptions and invest in a new designs that may need quite a bit of time to beat control. This is why most travels sites look like they were last re-designed in 2003.

So, I’d prefer to optimize Deming’s quote to “When the goal is quantified and the ROI is worth it, run an AB experiment. All others bring vision.” It doesn’t have quite the same ring to it.

Currently listening to Forsyth Gardens by Prefuse 73.

Failover Fails


One of my favorite quotes is from Mitch Hedberg:

An escalator can never break — it can only become stairs. You would never see an “Escalator Temporarily Out Of Order” sign, just “Escalator Temporarily Stairs. Sorry for the convenience”. We apologize for the fact that you can still get up there.

I’m reminded of this quote whenever I travel because so many apps on my phone don’t fail over this gracefully. My favorite historically have been foursquare’s. Whenever it couldn’t tell where I was, it would say “A surprising new error has occured.” or “must provide il or bounds”. Let’s forget for a second that these are the most unhelpful error messages ever and get to the bigger problem. foursquare (and most other apps) are designed only for a fully connected, localized experience. This is in a world where, even in cities, 4G connections can be intermittent, and GPS quality is sketchy at best, especially inside. Apps break all the time because of these issues, and it’s not excusable.

As we move towards always in the cloud, total internet and GPS reliant services, software makers need to think about these alternate states and design better experiences for them. It will be a long time before a wireless connection is reliable anywhere in the US, let alone internationally. It will be a while before GPS tracking works well in many places. Connections may be slow. Batteries may be low. Available storage my be low. I like to think of it like testing for different browsers and devices. If you aren’t testing for different connection states or with/without certain pieces of key information, you’re not testing right.

The Contradictory Nature of Mobile Unbundling and the Emergence of Niche Marketplaces

Two specific, but highly related, points of view are gaining widespread acceptance among venture capitalists in the technology industry. The first is succinctly explained by venture capitalist Albert Wenger in a post called Facebook’s Real Mobile Problem: Unbundling. The gist of the post can be summed up by this comment: “Mobile devices are doing to web services what web services did to print media: they unbundle.” Fellow venture capitalist Andrew Weissman expanded on this idea in a post called The Great Fragmentation. In it, Andrew goes further, arguing that unbundling might be a core feature of the internet.

A second, but related point, is the emergence of the niche marketplace. Venture capitalist Andrew Parker has a post called The Spawn of Craigslist in which he shows how the behemoth marketplace Craigslist is getting slowly disrupted in a vertical-specific way. Venture capitalist Chris Dixon expands on this idea, saying that the only way to be successful as an online marketplace now is to take a vertical-specific approach.

Together, these venture capitalists describe a future in which there is a specific app or specific marketplace for every need a user might have. Instead of going to Craigslist to find an apartment, movers, a maid, a freelance web designer for your home business, a date, and last minute tickets, a mobile user would instead have an app for Padmapper, TaskRabbit, Homejoy, ODesk, HowAboutWe, and WillCall. The key to being a successful venture capitalist then shifts from finding businesses that tackle very large markets e.g. Craigslist to finding businesses that target markets that could be much bigger with unbundling e.g. Airbnb.

All of these VC’s are clearly smarter than me, but I take a somewhat contrarian view here. I hope the above example points out the main problem with this theory. In the above picture, in order for this mobile user to accomplish his/her goals, instead of needing to just know about and have an app for Craigslist, s/he now needs to know about and have apps for six separate businesses. One other thing venture capitalists agree on is that mobile app discovery is hard, and that the amount of apps mobile users will download and use is limited by both device memory as well as human memory. This same problem faces the sellers of services on marketplaces. With no aggregate marketplace, it may be harder for a seller of multiple services to know which ones exist for which product/service they are selling. Marketplaces thrive on a multitude of buyers and sellers. Unbundling of marketplaces makes building that two-sided network harder.

Something has to give here. You can’t have a future where everything is accomplished online via a mobile device, consumer’s preference on mobile is for apps, there will be hundreds of specific services for anything a user needs that are more powerful than aggregate services, app discovery is difficult, and people will only have 41 apps per phone. I think there is some sort of equilibrium here. Even if app discovery is solved (and that’s a hard problem), the rate of successful unbundling certainly seems like it has to be limited by 1) the amount of space on someone’s phone, and 2) user’s inability to be aware of hundreds of niche services they may need at any time. If you think a recommendation engine could solve this with big data, I recommend you read this article about how successful that’s been for other services.

If I had to guess, I would surmise that user unbundling will not be a trend in and of itself, even if it is a trend in technology startups building new businesses. Unbundling will continue when either 1) the frequency of the activity that is being unbundled is high (my standard would be weekly), or 2) the advantage of the unbundling is exponentially more valuable than the bundled version of the same activity. For criterion 2, that advantage will also be a moving target where the advantage has to become greater and greater to justify phone/brain space as more apps improve their utility. Number of apps per phone will continue to grow, but a decreasing rate, and with that growth, there will be a decreasing state of awareness for both apps that are on a user’s phone and ones that are not. If you doubt this, just think of how many websites you visit regularly. Think hard. It isn’t that high, is it? Now think about apps? Even less? Me too.

So, what does this all mean? Well, my take is that high frequency services like chat or picture taking continue to become unbundled from any aggregate services consumers use for them because of the ability of mobile to create superior user experiences for succinct actions. But, marketplaces that aggregate niche activities that users need only occasionally can continue to thrive e.g. eBay and Craigslist. One should expect only a handful of the dozens of services hoping to disrupt Craigslist or eBay or Amazon to survive, because of fantastic user experience or a high frequency of use. Finally, one should not be so quick to anoint the niche marketplace model as the emergence of mobile presents as many limitations to their success as it does opportunities for growth.

On Lagniappe

I grew up in New Orleans, and there’s a word in use there that isn’t known anywhere else. Mark Twain said it was “a word worth travelling to New Orleans to get”. It’s a French word based on a Spanish phrase, the type of co-mingling of language only possible in such a place as New Orleans. So, what does it mean, and why is it so powerful?

Everyone in New Orleans will define lagniappe the same way, as “a little something extra”. The real meaning’s a bit more complicated than that. For example, the other day I ordered from Leona’s on GrubHub. Every time you order, no matter what you order, they add in at least one of these mini-cupcakes they make for free. First off, they are delicious. Secondly, there’s no message about it, and you didn’t have to pay anything for it. It’s just there for your enjoyment. That’s lagniappe, that little something given to you for free that you weren’t expecting but is such a treat to find. This type of practice is common in New Orleans, but you rarely see it elsewhere outside of the extremely common baker’s dozen or the fortune cookie in Chinese restaurants, which, because they’re so common, are expected and therefore no longer really lagniappe.

This type of practice creates such an overwhelmingly positive impact on the experience, I don’t understand why the practice in business seems to stay limited to small shops in New Orleans and bakeries. A small gesture can go a long way in creating a memorable experience with your customer, and whether you deliver food, make doughnuts, or do management consulting, there is always a way to give lagniappe. So, what little something extra are you providing your customers? How can you give lagniappe?

Design and Business Inspirations: A Reflection on the Remarkable

With the end of the year approaching, I thought I’d take some time to reflect on some of the things that have impressed me this year in design and business.

New Businesses:

Everlane Logo

Everlane is a new startup that sells designer quality clothing for under $100. They premiere a new collection monthly of different types of items. The first month was shirts, the second month ties and bows, etc. They have also featured scarves and backpacks so far. Especially as a man, the ability to find quality clothing at an affordable price can be quite the challenge. Everlane solves the problem by consistently producing quality “basics” every person needs. Everlane has been in a limited launch where it requires you to get others to sign up to get access. This tactic is not without its hiccups as some people have been waiting forever that are early adopters, but it spreads the message and anticipation and continues to motivate even after you are granted access because more friends = more perks. See below.

Everlane Loyalty Program

Postmates Logo

Postmates is an on demand service for couriers. As someone who has the iPhone app, you can request a pickup/delivery, and Postmates sends the orders to couriers that have signed up to receive more business who then can accept or deny the order. This is the classic approach of a two-sided model where a particular industry has idle time and can use more business. At the same time, this service makes it easier for people to find courier services so they can get packages across town without using the post office. Postmates will take a fee from each courier transaction.

OneReceipt Logo

OneReceipt is an online home for all of your receipts. Just sync up your email and have all of your email receipts organized in one place. Snap pictures of offline receipts to add them as well. OneReceipt will itemize and categorize all of your spending and provide reports. Long-term, they will attempt to use your spending to match you to targeted offers. The business model end will be a work in progress as it’s unclear how they will get past deal fatigue or even if what people want from a service that’s telling them how much they spend where to spend more. I had the pleasure of talking with the founders before their beta launch, and they are sharp guys, so they will figure out the right way to monetize. That said, the sync technology is impressive, and something competitor Lemon doesn’t have.

Fab Logo

Fab is a flash sales site for design. Fab has curators that handpick vendors to provide flash sales on the site and their mobile apps. The sales are categorized in such a way that you can immediately tell if it’s something you are interested in or not. You can purchase easily through the site or the app, and the merchandise seems to be of consistently high quality. My experience with Fab has been so nice that it made me think I’ve just been missing out on the whole flash sale scene. So I signed up for some other flash sale sites, and no, Fab is just special. The site and app are so enjoyable to use that I don’t need a daily email to remind me to come back to the site. It is fun to just browse the app if you have a few minutes to waste on the bus ride home. That said, the Fab website has a bit of a confused identity, as part of it is flash sale site and part of it is Pinterest competitor. I also question their ability to scale here as they definitely do not have drop ship relationships with all of their vendors, so inventory and warehousing control is a very important issue for them I imagine. It also needs personalization long-term, and no one’s doing that right yet.

Fab Mobile App

Existing Businesses:

Evernote Logo

It’s hard not be impressed with what Evernote has done this year. The company broke 20 million users and raised a $50 million Series D round that values them at somewhere north of $500 million. That’s only scratching the surface though. They bought their first company Skitch, and then helped it pass 3 million downloads on Android. They also started pursuing a multi-app strategy using the Evernote brand. In addition to Evernote and Skitch, they released Evernote Food and Evernote Hello. By separating these new functions as new apps, they get the benefit of using their brand to promote usage while not having what these apps do become hidden as just one of the many features of the main Evernote app.

Path Logo

Path launched in 2010 as a photo sharing service. Its differentiator was that it was a private network for sharing special moments with those close to you. Well, that value proposition really limited their virality, and photo sharing became a crowded space with an increasingly clear winner with Instagram. Path had already rejected a $100 million acquisition offer from Google, they needed to regroup. Path 2, as it’s been called is more of life a journal than photo album. The most important thing to know about it is that it is beautiful. The mobile app is at the forefront of interaction design, and their home page is even innovative (go check it out right now, seriously). The most impressive parts to me are the + button that is so easy to use the and going to sleep/waking up experience (though it does remind me of a joke from Kicking and Screaming). This update still makes me wonder if Path as a company is just a bunch of talented designers without a real problem to solve, but ever since the relaunch, my Path has been way more active with fellow users, so they are gaining traction.