Tag Archives: strategy

Why Focus Is Critical to Growing Your Startup, Until It Isn’t

When I was a teenager, I told my dad about a friend and his dad and how they had seven businesses. He immediately replied, “And none of them make money.” I thought it was an extremely arrogant thing to say at the time, but later, I realized it might be the smartest piece of advice he ever gave me.

When I joined Grubhub, I quickly noticed the founders were incredibly good at staying focused. They said we were building a product for online ordering for food delivery — and only delivery — not pickup, not delivery of other items, not catering, and that’s all we would do for a long time. I remember thinking, “but there’s so much we could do in [XYZ]!” I was wrong. By staying focused on one thing, we were able to execute technically and operationally extremely well and grow the business both very successfully and efficiently. When we added pickup functionality four years later, it proved not to be a very valuable addition, and hurt our conversion rate on delivery.

If you have product/market fit in a large market, you should be disincentivized to work on anything outside of securing that market for a very long time. There is so much value in securing the market that any work on building new value propositions and new markets is destructive to securing the market you have already validated.

There is an interesting switch in the mindset of a startup that needs to occur when a startup hits product/market fit. This group of people that found product/market fit by creating something new now have to realize they should not work on any new value propositions for years. They now need to work on honing the current product value or getting more people to experience that value. Founders can easily hide from the issues of a startup by working on what they’re good at, and by definition, they’re usually good at creating new products. So that tends to be a founder’s solution to all problems. But it’s frequently destructive.

If a product team can work on innovation, iteration, or growth, they need to quickly shift on which of those they prioritize based on key milestones and value to the business. In this scenario, it’s important to define what innovation, iteration, and growth mean. In this context:

  • Innovation is defined as creating new value for customers or opening up value to new customers. This is Google creating Gmail.
  • Iteration is improving on the value proposition you already provide. This can range from small things like better filters for search results at Grubhub to large initiatives like UberPool. In both cases, they improve on the value proposition the company is already working on (making it easier to find food in the case of Grubhub, and being the most reliable and cheapest way to get from A to B in the case of Uber).
  • Growth is defined as anything that attempts to connect more people to the existing value of the service, like increasing a product’s virality or reducing its friction points.

I have graphed the rollercoaster of what that looks like below around the key milestone of product/market fit.

Market Saturation
The time to think about expanding into creating new value propositions or new markets is when you feel the pressure of market saturation. Depending on the size of the market, this may happen quickly or slowly over time. For Grubhub, expansion into new markets made sense after the company went public and had signed up most of the restaurants that performed delivery in the U.S. The only way the company could continue to grow was to expand more into cities that did not have a lot of delivery restaurants by doing the delivery themselves.

All markets are eventually saturated, and that means all growth will slow unless you create new products or open up new markets. But most entrepreneurs move to doing this too early because it’s how they created the initial value in the company. Timing when to work on iteration and growth and when to work on innovation are very important decisions for founders, and getting it right is the key difference to maximizing value and massively under-performing.

Don’t Become a Victim of One Key Metric

The One Key Metric, North Star Metric, or One Metric That Matters has become standard operating procedure in startups as a way to manage a growing business. Pick a metric that correlates the most to success, and make sure it is an activity metric, not a vanity metric. In principle, this solves a lot of problems. It has people chasing problems that affect user engagement instead of top line metrics that look nice for the business. I have seen it abused multiple times though, and I’ll point to a few examples of how it can go wrong.

Let’s start with Pinterest. Pinterest is a complicated ecosystem. It involves content creators (the people who make the content we link to), content curators (the people who bring the content into Pinterest), and content consumers (the people who view and save that content). Similar to a marketplace, all of these have to work in concert to create a strong product. If no new content comes in, there are less new things to save or consume, leading to a less engaging experience. Pinterest has tried various times over the years to optimize this complex ecosystem using one key metric. At first, it was MAUs. Then it became clear that the company could optimize usage on the margin, instead of very engaged users. So, the company then thought about what metric really showed a person got value out of what Pinterest showed them. This led to the creation of a WARC, a weekly active repinner or clicker. A repin is a save of content already on Pinterest. A click is a clickthrough to the source of the content from Pinterest. Both indicate Pinterest showed you something interesting. A weekly event made it impossible to optimize for marginal activity.

There are two issues at play here. The first is the combination of two actions: a repin and a click. This creates what our head of product calls false rigor. You can do an experiment that increases WARCs that might actually trade off repins for clicks or vice versa and not even realize it because the combined metric increased. Take that to the extreme, and the algorithm optimizes clickbait images instead of really interesting content, and the metrics make it appear that engagement is increasing. It might be, but it is an empty calorie form that will affect engagement in a very negative way over the long term.

The second issue is how it ignores the supply side of the network entirely. No team wants to spend time on increasing unique content or surfacing new content more often when there is tried and true content that we know drives clicks and repins. This will cause content recycling and stale content for a service that wants to provide new ideas. Obviously, Pinterest doesn’t use WARCs anymore as its one key metric, but the search for one key metric at all for a complex ecosystem like Pinterest over-simplifies how the ecosystem works and prevents anyone from focusing on understanding the different elements of that ecosystem. You want the opposite to be true. You want everyone focused on understanding how different elements work together in this ecosystem. The one key metric can make you think that is not important.

Another example is Grubhub vs. Seamless. These were very similar businesses with different key metrics. Grubhub never subscribed entirely to the one key metric philosophy, so we always looked at quite a few metrics to analyze the health of the business. But if we were forced to boil it down to one, it would be revenue. Seamless used gross merchandise volume. On the surface, these two appear to be the same. If you break the metrics down though, you’ll notice one difference, and it had a profound impact on how the businesses ran.

One way to think of it is that revenue is a subset of GMV, therefore GMV is a better metric to focus on. Another way to think of it is the reverse. Revenue equals GMV multiplied by a commission rate for the marketplace. So, what did they do differently because of this change? Well, Seamless optimized for orders and order size, as that increased GMV. Grubhub optimized for orders, order size, and average commission rate. So, while Seamless would show restaurants in alphabetical order in their search results, Grubhub sorted restaurants by the average commission we made from their orders. Later on, Grubhub had the opportunity to test average commission of a restaurant along with its conversion rate, to maximize that an order would happen and maximize its commission for the business. When GrubHub and Seamless became one company, this was one of the first changes that was made to the Seamless model as it would drastically increase revenue for the business even though it didn’t affect GMV.

This is not to say that revenue is a great one key metric. It may be better than GMV, but it’s not a good one. Homejoy, a service for cleaners, optimized for revenue. Their team found it was easier to optimize for revenue by driving first time use instead of repeat engagement. As a result, their retention rates were terrible, and they eventually shut down.

Startups are complicated businesses. Fooling anyone at the company that only one metric matters oversimplifies what is important to work on, and can create tradeoffs that companies don’t realize they are making. Figure out the portfolio of metrics that matter for a business and track them all religiously. You will always have to make tradeoffs between metrics in business, but they should be done explicitly and not hide opportunities.

Currently listening to A Mineral Love by Bibio.

If It Ain’t Fixed, Don’t Break It

Frequently, products achieve popularity out of nowhere. People don’t realize why or how a product got so popular, but it did. Now, much of the time, this is from years of hard work no one ever saw. As our co-founder at GrubHub put it, “we were an overnight success seven years in the making.” But sometimes, it really just does happen without people, inside or outside the company, knowing why. Especially with social products, sometimes things just take off. When you’re in one of these situations, you can do a couple of things to your product: not change it until you understand why it’s successful now, or try to harness what you understand into something better that fits your vision. This second approach can be a killer for startups, and I’ve seen it happen multiple times.

Let’s take two examples in the same space: Reddit and Digg. Both launched within six months of each other with missions to curate the best stories across the internet. Both became popular in sensational, but somewhat different ways, but Digg was clearly in breakout mode.

What happened after the end of that graph is a pretty interesting AB test. Digg kept changing things up, launching redesigns and changing policies. Some of these might have been experiments that showed positive metric increases even. Reddit kept the same design and the same features, allowing new “features” to come from the community via subreddits, like AMA. By the launch of Digg’s major redesign in August of 2010 (intended to take on elements from Twitter), Reddit exploded ahead of Digg.

This is what the long term result of these two strategies look like. Digg is a footnote of the internet, and Reddit is now a major force.

Now, neither of these companies are ideal scenarios. The best option in the situations these companies found themselves in is to deeply understand the value their product provides and to which customers, and to completely devote your team to increasing and expanding that value over time. But, if you can’t figure out exactly why something is working, it is better to do nothing then to start messing with your product in a way that may adversely affect the user experience. This has become one of my unintuitive laws of startups: if ain’t fixed, don’t break it. If you don’t know why something is working (meaning it’s fixed and not a variable), do nothing else but explore why the ecosystem works, and don’t change it until you do. If you can’t figure it out, it’s better to change nothing like Reddit and Craigslist than to take a shot in the dark like Digg.

Currently listening to Sisters by Odd Nosdam.

Branding Gives Your Company the Benefit of the Doubt

People tend to assume the worst, especially about companies. So, when companies screw up, and they inevitably will, consumers (and partially as a result, the press) are ready to pounce on you and your vile type of corporate evil. Every company has this moment, and some companies are more prepared for it than others. Yes, being prepared does mean having a crisis PR strategy and all that tactical jazz, but what’s more important is to have a brand people know. I’ll explain a bit why.

The brand of a company tells the consumer what it stands for, what it promises, and what it can deliver. Most companies invest handsomely in their brand and for good reason. Brand building can increase loyalty and command higher prices. But a crucial piece of brand building is that since consumers know what you stand for, and many of them have already identified with that, they give you more leeway in how you do business and when you make mistakes. Another word for this is trust. In really great brand building examples, a consumer will say, “That can’t possibly be right. I want to hear what they have to say about it.” In absence of this work, a brand is just identified as the product experience, which means when the product has issues, the brand has issues. Companies should try to elevate their brands to mean something beyond the product experience, and bad things happen when they don’t.

Brand building also crystallizes what you stand for inside a company, making your company less likely to make a strategic mistake against what you stand for. In absence of a strong brand, different departments optimize for different things, typically creating both a Frankenstein experience for the consumer, but also distrust among departments. When a core engineering team sees a new signup flow that seems particularly aggressive, they might be inclined to curse the growth team instead of saying, “I know what that team is about. Let me go talk talk to them to see why things seem amiss here.”

You can absolutely be successful without building a strong brand outside of the core product experience, but it is harder, and you’ll have more bumps along the road. I’ll give one example that comes to mind. The first is Netflix. Netflix is undoubtedly one of the most well known brands in the U.S. It is also a brand that has grown entirely through its product experience and direct response advertising. All of its marketing is tied to signing up for Netflix. Its TV ads, display ads, pop unders, etc. eschew brand building to attract direct signups. This worked very well to grow Netflix into a powerhouse, but when they inevitably made some major and minor mistakes, consumers, the press, and the public markets went after them. In 2011, Netflix announced a price increase and then after that a split of their DVD and streaming business, including a new name. Netflix is an amazingly valuable service at an incredibly affordable prices, especially compared to cable. But, because they lacked a strong brand, consumers associated a lot of their brand with the price. Furthermore, separating the two businesses was clearly a case of not having a strong understanding of their brand internally. The result: 800,000 subscribers lost in one quarter and a 77% drop in stock price.

Now, Netflix recovered from this, but it took years and a pretty radical change in strategy toward original content. While this provides Netflix more of a brand than “cheap access to tons of movies and TV shows” and pushes that branding more so to “quality content that I sometimes can’t get anywhere else”, it still associates the brand entirely with the product experience. If they go a few seasons without a hit show, or need to raise prices again, they may be in the same situation in the future.

Currently listening to Panda Bear Vs. The Grim Reaper by Panda Bear.

First to Product-Market Scale

I like to think of this blog as balancing between business school theory and startup execution. While there are many places they don’t add up, usually the combination of the two provides an insightful truth that is hard to see without the theory plus the experience of trying to implement it. One area where I struggled for a while between my experience and the theory was the first mover disadvantage as it relates to barriers to entry.

The first mover disadvantage states that, while being the first firm in a market to do something has its advantages in terms of brand recognition and speed to market, the firm bares an even greater cost of R&D, education, etc. that second movers do not. These second movers can fast follow without all of these additional costs the first mover had to deal with and quickly compete. See HBR for details. In my Chicago Booth studies, both Eric Lefkofsky (CEO of Groupon who taught Building Internet Startups) and up and coming economist Matthew Gentzkow (who taught Competitive Strategy) argued about how potent the first mover disadvantage would be for Groupon, and that now that everyone knew how profitable the Groupon model was, it would be copied as there was no competitive advantage.

In a case study about Groupon in Gentzkow’s class, I did a one man filibuster against this argument. I looked at the data. During the time of the class, Facebook and OpenTable were winding down their Groupon clones, Yelp called theirs “not a priority” six months after shifting almost their entire team to work on it. Living Social started having financial issues. Groupon was winning despite the first mover disadvantage. The question was not would Groupon win, it what the prize was going to be for being first. Why was that the case when economics would argue against it?

I saw this same phenomenon in my own work at GrubHub. Online ordering was not a hard technology to clone, and once we had educated restaurants on the value of online ordering and shown them the additional business we could bring them, a competitor would have a much easier time with their pitch. Yet, we were still winning in every market except New York and college towns, where competitors had entered well before us. After acquiring those competitors, we talked candidly about competing with each other. The folks at Seamless (the New York competitor) talked repeatedly about feeling boxed out due to GrubHub’s first mover advantage in the rest of the country, even though we weren’t first in many of those areas.

Having taken two classes emphasizing first mover disadvantage before hearing this, I knew something wasn’t right, but couldn’t quite nail the hidden truth. Last year, I read Andy Rachleff’s post on first to product market fit. Andy argued it’s not about first mover advantage, it’s about first to product-market fit. It felt warmer, but not quite right either. GrubHub was not first to product-market fit in many of the markets it entered and later dominated.

If we tweak Andy’s definition slightly from fit to scale, the model fits better. One thing about GrubHub is that everything we thought about we thought about at scale and with velocity. We would systematically try to grow every market we entered with the same focus and the same process. If we achieved this, we would overtake successful players that were already in the market. It also didn’t matter who entered the market and tried the same after that. We had already won. Product-market fit implies a product that works with a small product, and the next step in the company’s evolution should be scale. So, the target for startups or large firms entering new markets in order to be successful should not just be product-market fit, but product-market scale. If you achieve that, you dominate markets and cannot seem to be usurped no matter how few barriers to entry you have.

Value Trade Offs in Online Food Delivery

If you’ve been following the online food delivery space, now is a pretty exciting time. Multiple services are starting up, competing on different value propositions, and many corporations are theoretically launching businesses here as well. There is one clear giant, and it is unclear if any of the upstarts will challenge them. But what is so interesting is how large companies entering the space and new startups alike are confronting the different value trade offs in online food delivery. I’ll first describe the different types of services, their different components, and then their trade offs.

Types of Services

Marketplaces
Services: GrubHub, Seamless, Eat24
Marketplaces aggregates delivery restaurants and allow diners to search for restaurants that deliver to them. The restaurants do their own delivery.

Delivery Services
Services: Postmates, DoorDash, Caviar, Uber Eats
Delivery services offer delivery from restaurants that don’t do their own delivery and deliver the food themselves.

Delivery Only Restaurants
Services: Sprig, Spoonrocket, Maple
Delivery only restaurants have no storefront. They just make food that is available for delivery and deliver the food themselves.

Delivery Only Restaurants that Require Prep
Services: Munchery, Gobble
These restaurant services require some prep work ranging from microwave to stove or oven, but usually it’s only a few minutes of prep required.

Delivery of Ingredients/Recipe Only
Services: Blue Apron, Plated
These services deliver the ingredients and the recipe required to make a meal, but the diner has to cook it themselves.

Delivery of Groceries
Services: Instacart, Fresh Direct
These services deliver whatever items you want from a grocery store.

I won’t go into corporate focused services in this post.

Value Propositions

Variety
People rarely agree on what food they like, let alone on which food they want to eat at a specific time. While GrubHub is currently unmatched in its variety nationally with over 35,000 restaurants, different companies are tackling variety on both sides of the spectrum. Postmates will theoretically offer the most variety as it will pick up food from any establishment. Online food companies like Sprig, Munchery, and Spoonrocket limit options considerably each day. Doordash, Uber Eats and Caviar have the most confusing approach here, as their ability to use their own delivery network does not restrict them to restaurants who already offer delivery, but they curate the list to provide supposedly only great options. GrubHub works with every restaurant that does delivery already, and has expanded the market by convincing many restaurants to start delivery because they see how well other restaurants do by offering that option with GrubHub.

Prep
Convenience has two components: how much work you have to do to eat (prep), and how quickly the food arrives (time). Marketplaces, delivery only restaurants and delivery services deliver ready-to-eat food. Then, there are some that require a little prep, some that require full cooking, and some that require figuring out what to cook and cooking it.

Time
The other convenience layer is time. Delivery only restaurants target 10 minute delivery times by pre-pepping meals and loading them into the cars of their drivers, whereas GrubHub and Eat24 are closer to 45 minutes to an hour depending on the restaurant’s location and type of food. Delivery services tend to take over an hour as they require extra coordination with restaurants. I believe Uber Eats is attempting a hybrid of the delivery service model and the delivery only restaurant model, but I can’t confirm. None of the other services deliver food ready to eat, but they range on how much work is required. The some prep restaurants are more like 10 minutes to heat, and ingredient/recipe services require typically cook time of over 30 minutes to an hour.

Price
Price varies for all of these services. Delivery only restaurants target less than $15 everything included. While that is possible in some cities with marketplaces, it is not in others. Ingredient/recipe delivery services have plans that are under $10 per person. Delivery services tend to charge a fee for delivery or mark up restaurant prices, so they are typically more at $20 and above per person. This incentivizes group order to spread the delivery cost around to multiple people. This is why most delivery services end up focusing on corporate catering instead of consumers over time. Prep delivery only restaurants have different plans to entice regular ordering.

Quality
In marketplaces, the quality options are set by the market, and the diner chooses how good they want their food to be. Delivery services have the same option with perhaps a higher end than marketplaces as the very best restaurants tend not to deliver. The delivery only restaurants tend to be cheap and low quality so far. Whether you had a hand in making it yourself can also be considered a quality parameter, as some people to tend to prefer things they cook themselves.

Planning
With food delivery, one typically does not need to plan in advance to use it, but with new grocery delivery and ingredient/recipe prep services, diners need to plan ahead of time to use the service.

Trade Offs

As you start playing with these value propositions, you recognize some additional constraints. I don’t need to lecture you in price vs. quality. That’s pretty obvious. But what may not be obvious is the trade off between time and quality. Even if you are delivering food from an amazing restaurant, if it takes a long time to get to a diner, it’s typically not very amazing by the time it gets there due to the food being cold. The other interesting trade off is quality vs. variety. At GrubHub, our stance was akin to the saying “quantity is a quality all its own.” In that, if you organized all of the supply, even if you had many amazing restaurants and many not so good ones, the good ones quickly emerged to the top due to ratings and reviews and overall quality of the service improved. So, all GrubHub worried about was variety and convenience, with convenience mostly limited to the ordering and customer service experience. Price and quality were set by the market, but presumably, variety solved quality, with a cap on the high end.

What these new services are doing is taking constants in the marketplace equation and making them variables: prep, time, price, and quality. It is way too early to tell if changing the equation is valuable to the broader market as GrubHub does way more orders in a day than the rest of these services combined. But it will be interesting to watch.

Why Everyone Link Builds, or Why You Sometimes Do Things When You Don’t Know If They Work

I was talking to an analyst about how SEO works, and we inevitably got to the authority side of SEO. I started talking to him about how many companies spend a lot of effort trying to get external links to their site to build authority. It’s not something we have to worry about at Pinterest as our authority is super high naturally, but most companies do not have that luxury. The analyst, being a good analyst, asked how you track effectiveness of link building as a program. My answer surprised him: most of the time, you don’t.

Sure, sometimes you can see a correlation between link building and average weighted rank improvement, and maybe you didn’t make any improvements on the relevance side during that time. But, while you can experiment with SEO in relevance changes, it is pretty much impossible to experiment with link building as it works at both a domain and page level, its impact is felt over such a long period of time, and there are almost always so many other factors one can’t control, namely from competitors for those same search results.

So, he then asked, how do we know it works? The answer is: we usually don’t. So, the next question of course is, if it’s a pain to do and we don’t know it works, why do we do it? This question can be generalized to almost any competitive question via game theory. To really hammer the point home though, I actually used the climate change example.

In climate change, their are four scenarios and two dimensions. The first dimension is whether or not it is actually happening (or whether it’s man made), and the second dimension is whether we do something about it.

Next, you examine the outcome of each scenario to determine the outcome if that’s the box you pick.

As you can see here, in every box you are fine, except for the upper left. Sure, you might have wasted quite a bit of money and slowed the growth of some businesses, but none of that compares to possible catastrophe. Even if you think it’s a waste of time, the risk is so great if you’re wrong, and the answer reveals itself over such a long period of time, the obvious answer becomes to assume it is true and invest in fixing it.

Now, let’s apply this to a much less risky scenario of link building.

While we aren’t saving the world when we work on SEO, from a business perspective, the risk is just as great. In one belief, you lose, and in all other scenarios you don’t. If your competitors are smart, they all do this exact analysis and come to the same conclusion: to invest in link building. This cannot be a prisoner’s dilemma either, as one company always outranks another, and links occur organically that presumably change the rankings.

The Three Stages of Online Marketplaces

Prior to Pinterest, I worked on two sided network businesses my entire career, for apartment rentals (Apartments.com), real estate (Homefinder.com), and food delivery (GrubHub). As a result, I’ve admittedly become somewhat of a marketplace geek. And today is a very exciting time for online marketplaces. Marketplaces are evolving online. It’s hard to keep up with the innovation, but I’ll describe the three phases I am seeing, and why certain ones may prevail in different industries.

Phase 1: Connect buyers and sellers
This is the basic requirement of a marketplace. Early marketplace businesses like Ebay allowed you find people looking for your service if you were a seller, or find people selling what you were interested in if you were a buyer. To make this work, companies need to get past the chicken and egg scenario and build trust through their network. Things like ratings and reviews and guarantees make buyers trust they would get what they paid for, and sellers knew they would get paid if they delivered the service. Marketplaces in this scenario also had to find a way to get paid, using taking a lead generation or transaction fee for increasing the seller’s volume of sales. This phase is still in use with successful marketplaces like Airbnb, GrubHub, OpenTable, and others, but almost all are desperately trying to migrate into phase two or three right now, as you’ll see in the following paragraphs.

Phase 2: Own the delivery network
More recent marketplaces, not content to just facilitate a transaction, are actually working to implement the transaction by owning the element of bringing the service to the buyer. Marketplaces know that if they don’t control more of the experience, a great experience can be ruined by things outside of their control, supply side fault or not. Startups like Instacart don’t just allow you buy groceries online, but their workers deliver the groceries to you. Postmates and Doordash do the same for delivery food, picking up food from restaurants that don’t deliver and deliver it using their own workers. While this model is not new (restaurant delivery services have been around since before the internet), companies are now trying to build delivery networks at scale.

This is risky, as delivery networks all rely on the same pool of drivers. So, on the delivery side, marketplaces in different industries compete for delivery drivers. In a zero sum game there, it’s most likely the marketplace with the most demand wins (at this point, that’s undoubtedly Uber).

GrubHub, for example, bought two delivery services in Q4. OpenTable is moving into payments at restaurants. Airbnb is working on concierge services to improve the stay of guests. The companies starting in phase 1 see this as owning more of service blueprint, injecting their brand into the blueprint wherever possible.

Phase 3: Own supply
An even newer trend than owning the delivery network for an online marketplace is to vertically integrate the supply side of the business. Now, you may ask, what makes this a marketplace? In reality, it’s not, but from every other element, the business is designed or is mimicking an existing marketplace. Sprig and Spoonrocket do this with food delivery. They are delivery only restaurants that make their own food and have their own delivery drivers. MakeSpace and Boxbee, instead of just building a marketplace to help you find storage space, built their own storage spaces and will pick up your items and deliver them to storage and deliver them back for retrieval if needed. Margins are very different for their businesses.

The question for me becomes how far up the pyramid can you build a successful business. In many cases, owning supply will be victorious, but in many others, owning the delivery network is the best option. In other, a traditional marketplace is the best option. It will be interesting to watch almost every vertical determine the best model for customer satisfaction, scale, and profitability over the next decade.

The Perils and Benefits of AB Testing

Bob, our head of product design at Pinterest, asked me to write up a post on the perils and benefits of AB testing after reading my post on building cross-functional teams. This is me obliging.

One thing it is never difficult to do is to convince an engineer to do an experiment. In general, this is a good thing. Famous engineer W. Edwards Deming said, “In God we trust, all others bring data.” AB experiments generate data, and data settles arguments. AB experiments have helped us move from product decisions made by HIPPO (highest paid person’s opinion) to those made by effectiveness. We build better products as a result that delight more people.

An AB experiment culture can also have a dark side though. Once people figure out that AB experiments can settle disputes where multiple viewpoints are valid, that fact can lead people to not make any decisions at all and test everything. I liken this type of approach to being in a dark room and feeling around for a door. If you blindly experiment, you might find the way out. But turning the light on would make it way easier. Rabid AB testing can be an excuse for not searching for those light bulbs. It can be thought of as easier to experiment than to actually talk to your users or develop a strategy or vision for a product. Here are some perils and benefits of AB testing to think about as you experiment in your organization.

Benefits
1) Quantifying Impact
This one is pretty obvious. AB experiments tell you exactly what the lift or decrease a treatment causes versus a control. So, you can always answer the question of what an investment in Project X did for the company.

2) Limiting Effort on Bad Ideas
Another great benefit of AB testing is that you can receive feedback on an idea before committing a ton of effort into it. Not sure if an investment in a new project is worth it from a design and engineering perspective? Whip up a quick prototype and put it in front of people. If they don’t like it, then you didn’t waste a lot of time on it.

3) Limiting Negative Impact of Projects
Most additions to a product hurt performance. AB testing allows you to test on only a segment of an audience and receive feedback without it affecting all users. Obviously, the larger the company, the smaller the percentage you can trigger an experiment on to get a solid read.

4) Learning What Makes Something Work
In AB experiments, you isolate one variable at a time, so you know exactly what causes a change in metrics. You don’t have to debate about whether it was a headline or background color or the logo size anymore.

Perils
1) Not Building a Strategy or Vision
Many places convince themselves that testing is a strategy in and of itself. While AB testing can inform a strategy or vision, it is not one in and of itself. What happens in these cases is that companies do tons of random experiments in totally different directions, and their failure rate is very high because they have no unifying vision on what they are trying to do.

2) Wasting Time
AB testing can slow companies down when they convince themselves that every single thing needs to be tested thoroughly. Everyone knows of the anecdote from Google where they tested 41 shades of blue.

3) Optimizing for the Wrong Metric
AB experiments are designed to measure the impact of a change on a metric. If you don’t pick the right metric or do not evaluate all of the important ones, you can be making tradeoffs in your product you do not realize. For example, revenue over user engagement.

4) Hitting A Local Maxima
AB experiments do a very good job at helping optimize a design. They do not do as well as helping to identify bold new designs that may one day beat current designs. This leads many companies to stay stuck in a design rut where their current experience is so well optimized, they can’t change anything. You need to be able to challenge assumptions and invest in a new designs that may need quite a bit of time to beat control. This is why most travels sites look like they were last re-designed in 2003.

So, I’d prefer to optimize Deming’s quote to “When the goal is quantified and the ROI is worth it, run an AB experiment. All others bring vision.” It doesn’t have quite the same ring to it.

Currently listening to Forsyth Gardens by Prefuse 73.

The Startup Marketing Funnel

Quite a few startups have asked me how to approach their marketing plan. They hear that it’s important to do specific things, but that list eventually grows long, and they don’t have a plan of attack or a prioritization. While my post on three phases of startup marketing helps, it doesn’t go into enough detail on the framework behind that prioritization, and what to do with a new idea not represented there. Well, the good news is there is a framework you can apply to evaluate a list of ideas and prioritize them, and it’s not too different from the traditional marketing world. It just may be a bit inverted.

You may have seen a marketing funnel like this before (everyone calls the stages different things, but it’s generally something like this):

Startup marketing is a bit different. Instead of products being driven top down as in the above diagram, startups have to work bottom up due to budgets and what will be effective. Also, startups need to focus more on the inverted funnel post-use not seen in the above. So, your startup marketing funnel looks like this:

Now, instead of working top down here, startups need to work inside out. You work on the on site experience to make sure the few users who come convert and have a great experience. Then, you get them to come back and have another great experience. Once they are hooked, you ask them to invite friends. At this time, you also target those who came and didn’t convert. Then, you target those with a need for your product that haven’t tried. Then, you can define your core audience well from those using the product and do core audience targeting to find more like them. After saturating all of those methods, you finally work on general awareness.

Now, how does that translate to tactics. Well, let’s have a look:

Making sure people convert is all about conversion rate optimization. Email and push can help trials turn into repeat purchases, but the big winner there is an engaging product experience or a community. To get those who checked the product out but didn’t convert, you use use retargeting. To find others in need of the product, you focus on search (paid or organic). To find more people like your current audience, you can use Facebook lookalike targeting or interest targeting (thousands of other options here as well, of course). to pursue general awareness, that’s typically when you work on larger spend initiatives like TV, radio, outdoor, and sometimes PR.

Follow this funnel, focusing on each step until it saturates, and you can be sure you’re always working on the most effective and impactful projects to grow your business. Conversion, product, and community never tend to saturate, so you’ll almost always have dedicated people working on that even as you move further up the funnel.

Currently listening to Sonnet by Benoît Pioulard.