Author Archives: Casey Winters

The Analyst Career Path

I’ve written before about analytics teams as a crucial function in today’s technology companies. Technology companies are rapidly hiring analyst roles to pair with their product teams. And while my previous post discussed how to hire analysts and structure their teams within organizations, I haven’t written about how analysts should approach their careers.

Many technology roles, at startups in particular, have an issue with career progression. While established industries have defined career ladders, the path of career advancement is much less clear in many technology roles. Engineering, being the largest and oldest function in technology companies, now has a well defined individual contributor and manager career path all the way up to VP Engineering and CTO. Product Managers know they can progress to manager roles at their companies all the way to VP Product, and if they want to remain and individual contributor, they can still grow by working on more and more complex and strategic products over time. As I’ve talked to many analysts and analytics teams, this progression is not as well defined. I will outline how I think about this progression as someone who has been an analyst and managed analysts.

Option 1: Graduate into Data Science

If someone wants to remain an individual contributor and not manage, at some point the only way to become a better analyst is to graduate into a data science role. Now, there is some confusion with where the line is between analyst and data scientist, and many companies just call all of their analysts data science as a form of title inflation. I define the role of an analyst as someone who uses data to help identify and communicate business opportunities, and drive decisions for teams. This includes targeted analysis driven by others as well as free form analysis driven the analyst. From a process perspective, this includes everything from making recommendations, helping with experimentation, and creating dashboards to help others make decisions. From a tooling perspective, this means everything from writing SQL queries, identifying logging opportunities for product engineering and database design opportunities with data engineering, creating new dashboards and visualizations. An analyst retrieves, analyzes, and recommends, and is judged by not only how good those recommendations are, but how often they are followed.

So, how does data science differ? A data scientist writes code beyond SQL to manipulate data for analysis and potentially for product experience. A data scientist can write an algorithm that powers a personalized experience in the product, or just do more complicated analyses requiring more sophisticated querying using Python, R, etc. Data scientists jump in when analyses are too complicated to be handled by analysts, and also frequently partner or embed with product and engineering teams to change the product. This is more than just a higher-power analyst role though. Data scientists have deep expertise in certain areas, like machine learning, statistical inference, and focus on solving specific, hard problems over longer time horizons.

Option 2: Become an Analytics Manager
If you wish to get on a management track, becoming an analytics manager is the natural path. Since analysts are being hired so frequently, they need managers who can mentor and coordinate learnings between teams. While analysts are best embedded, analytics management bears the important responsibility of solving company-wide analytics issues related to tooling, process, etc.

Option 3: Graduate into Product Management
The third path that analysts can choose to grow their career is migrate into product management. Technically, product managers and analysts are peers in cross-functional teams, but product management has better career pathing that doesn’t require as much technical investment as data science, and product managers tend to have a bit more power in organizations today.

The migration of analysts to product manager is increasingly common as more and more product teams rely on data as the foundation for most decision-making. This has certainly been most true on growth teams and teams that utilize personalization, but I believe all future product teams are data savvy. A significant percentage of product managers at Pinterest started as analysts at the company. This same migration is also true for marketing analysts. They tend to become quantitative marketers over time, or switch to product analytics.


Being successful as an analyst is peculiar is that it almost requires a switch in roles over the time in ways that are not true for design, engineering, and many other roles in technology companies. Fortunately, the analyst has a lot of choices on how to progress within an organization. Hopefully, managers of analysts get better at outlining these different opportunities and help analysts position themselves toward the best ones for them over time.

Thanks to George Xing for reading early drafts of this.

Currently listening to Compro by Skee Mask.

The Autonomy Spectrum: How Much Autonomy Should You Give Your Teams?

Many people are familiar with Dan Pink’s work in Drive that to be the most productive, happy at work, etc., you need to have autonomy, mastery, and purpose. While a lot has been written about how to create purpose inside companies and crafting compelling missions, I would like to spend some time to dig into the details on autonomy. As founders and senior leaders of companies, it can be tough to understand how to give autonomy to employees and also drive toward a singular vision and commit to things that need to happen for other components of the business to work. I have found creating a spectrum of autonomy is helpful, and discussing with senior leadership and your team where you are on that spectrum.

How do you create an autonomy spectrum for your team or company? It’s helpful to start at the extremes. On one end of the spectrum, you would tell your team exactly what to do and how to do it. This is a complete lack of autonomy. On another end of the spectrum, employees have free reign to work on whatever they would like to work on. These projects may have nothing to do with the business, or could be completely aligned. That represents 100% autonomy.

Usually, people agree that neither extreme is particularly helpful. So how do you decide where in between your company or team should be? I like to work backward from 0% autonomy, and see where leaders start to get less comfortable. Let’s take an example recently from my time working with one of the companies I advise. The CEO asked me to come in and help the leadership team think through an expansion of their product. So that is step 1 of the autonomy spectrum, handing a business opportunity to the team from the CEO.

CEO: We have this opportunity for expansion (Business Opportunity)

Working down from that requirement, one executive volunteered to lead this initiative and did a bunch of research on the expansion opportunity. She talked to potential customers of the expansion, validated the pain points of this segment, and discovered a need the company could solve. So, she started settling around a vision of a new product experience leveraging the company’s existing strengths and identifying what new strengths needed to be built.

Executive: The way to attack this opportunity is to solve this specific problem for this segment (Persona and Product Vision)

The team she managed built an MVP attempting to solve this problem, and measured a lot of early usage and did qualitative research on what they had built. At this point, I stepped in to temporarily lead the product team and saw specific issues from the data and the research around the breadth of the solution, the quality of the options we provided, and the value of our solution vs. existing competitors in the market.

VP Product: Solve these four problems with the event discovery app (Problem Scope)

I also saw a team whose operating cadence matched a mature product with clear requirements rather than a fast moving team that uses research and experiments to find product-market fit. So, I created a new process that would structure them better for the problems they were facing and give them more autonomy.

At this point, we worked as a team to organize all of the product teams to work on these problems. As a leadership team, we chose not to identify solutions to these problems, but to put the product teams in charge of determining their own potential solutions. We created a weekly time where our persona would be in the building to provide feedback to our work. We also created a weekly meeting to review research feedback and experiment results.

VP Product: Test ideas with our target persona and run experiments targeting these problems (Process)

This is where we chose to sit on the autonomy spectrum. Prioritize business opportunity, vision, scope, and some process. Create autonomy for the team to define solutions and deeper process on how they want to get there. There are of course other options here. We could have chosen to prioritize ideas the teams invent to solve these problems, even come up with the solutions or how they would validate with research and experiments. There is no clear cut place to draw the line where you start thinking this way.

What is more important is to try to move the line to the left over time. When I advise leadership teams, I tell them that their goal should be to push decision-making as far down the org chart as possible. For leaders that have historically erred on the side of low autonomy, they need to understand how far they can start to push decision-making down without creating chaos. After all, Pink says that mastery is about providing stretch assignments, but not assignments employees cannot possibly be successful in. This process usually starts by moving to one additional layer of autonomy than normal and measuring results. If that is successful, leaders can be confident moving to another additional layer of autonomy. If this is not successful, then leadership needs to think about what type of training it needs to build so that this is possible in the future. I foresee a future at this company where the teams are in charge of prioritizing the problems as well as the solutions because they will be the experts from talking to our persona every week.

This is how Pinterest operated. Product teams created their own missions and problems they wanted to solve, and they were approved by the leadership team. Every six months, teams updated their OKRs to define the new problems they wanted to target and how they would measure success. No company starts there. Especially with startups, founders are used to setting the vision as well as coming up with product ideas. But as companies scale, leaders have to defer decision-making more so they can continue to move fast. The founders cannot be in every meeting nor can they be the experts on every topic.


Autonomy is a difficult topic to grasp inside a company. No CEO or senior leader wants to hand over the reigns to every decision to a team of varying levels of experience and confidence, but trying to make every decision stifles creativity and limits execution pace. Use the autonomy spectrum to build a honest understanding of where the company or teams you manage are and where you would like it to be over time. When you’re confident you are in the right place, communicate this spectrum and why you have made the decisions you’ve made on the level of autonomy you’re comfortable with. Employees will have a much clearer understanding of their role and can build better mastery and purpose as a result.

Google’s New Strategy and How It Affects Aggregators

As someone who has had a lot of success using SEO as a tactic to grow companies (for Apartments.com, Grubhub, and Pinterest it was the dominant channel for new users), I get asked a lot of questions about SEO as a strategy today. Andrew Chen’s Law of Shitty Clickthroughs states that all acquisition channels have a shelf life and decay over time. SEO has had by far the longest shelf life of any major internet channel. It has been a stable platform (unlike Facebook), it consistently grew itself, and it was supported by a very strong business model that could drive revenue growth for Google for well over a decade, so Google didn’t need to monetize all of the free traffic they distributed to other companies. Perhaps a more elegant way of explaining this is that since organic search exists to serve user needs, not advertiser needs, it was a more sustainable acquisition channel, precisely because it was not built to be a channel. People never wanted ads, more email, etc. like other acquisition channels. On Google organic search, people do want answers.

It’s this last statement remaining true amidst a platform shift to mobile that will mark the inevitable decline of SEO as a channel for user acquisition. Ben Thompson declared Peak Google a few years ago as a company. Why he was wrong then is why I am right today by declaring we are now past Peak Google as an acquisition channel. To understand this, you have to understand Google’s strategy. Google’s search engine is driven by optimizations that help its users. Ben Thompson does a good job explaining how this plays out with publishers. If you, like Google, have been analyzing its users for the last few years, you may have learned a few things. The first is that the majority of them are on mobile, where their time is more limited, their connections are (still) slower, and there is the threat of an app replacing frequent queries.

What Google is seeing is that their users no longer want to click ten blue links. They don’t have the time or the bandwidth, and there are now a plethora of competitors in the form of apps for many of those queries. The form factor is dictating the optimal user experience, and forcing Google to evolve. Users want an answer, and they want it immediately. So, that is what Google is doing. If you type a question into Google with a clear answer, there’s a good chance Google will just answer the question instead of recommending a site for it. We’ve all seen that. What’s more interesting is what Google is doing when there isn’t an answer, and the solution is to provide options, or what they would likely call a discovery experience. Recipes is a great example. Where you used to be treated to a bunch of “21 best recipes for X” pages, you now just see the recipes as results.

One would think these queries play perfectly into Google’s existing strategy of ten blue links. But Google knows consumers don’t want to click back and forth onto multiple sites to see each site’s recommendations. They want Google to surface up the options directly. And that is what Google is now doing. Take any top query category, and you will see Google replacing links with either answers or options showing up directly in search.

Whereas the top strategy historically for these “options” queries was to build an aggregator and rank at the top by having the most options, Google is now stating that it should be the only aggregator. In the same way Ben Thompson described the squeeze between Google’s demand for fast loading pages and banning of obtrusive ads on Chrome, Google’s search policies are doing the same for aggregators who do well on organic search.

How is Google doing this algorithmically? Google has started to seriously enforce two new policies over the last couple of years in their algorithm: internal search and doorway pages. For internal search, Google says:

Don’t let your internal search result pages be crawled by Google. Users dislike clicking a search engine result only to land on another search result page on your site.
Source

For doorway pages, Google says:
We have a long-standing view that doorway pages that are created solely for search engines can harm the quality of the user’s search experience.
Source

On the surface, these rooms seem sensible. If you continue to read to the end of the doorway pages update, you may start to see a problem:

  • Is the purpose to optimize for search engines and funnel visitors into the actual usable or relevant portion of your site, or are they an integral part of your site’s user experience?
  • Do the pages duplicate useful aggregations of items (locations, products, etc.) that already exist on the site for the purpose of capturing more search traffic?
  • Do these pages exist as an “island?” Are they difficult or impossible to navigate to from other parts of your site? Are links to such pages from other pages within the site or network of sites created just for search engines?

Reading these two together, what Google is saying is that creating pages indexing your search result pages is a bad experience, and creating new pages that replicate your search experience in a different way for search engine visitors is a bad experience. These two rules effectively penalize any presentation of your inventory of content. How do they tell if these pages are created solely for search engines? It’s similar to how they are detecting bad ads: they use Chrome browser data. So, the guideline for an aggregator who would like to show their content to Google is simple: give us your content, and we’ll aggregate it, or play in the tiny space that is a non-internal search page and a non-doorway page. That effectively means creating a page with unique inventory that does not look like search, yet receives traffic to the page from other sources besides Google. There is a window there, but it is a small one.

While updating the algorithms for these rule changes, Google is figuring out how to be the aggregator in many categories now, and over the next ten years will go down the list of every top searched category and figure out exactly how to do that. To do this, Google will either build, buy, or partner with existing players. Let’s take a look at some of these examples.

The Build Case: Google Local
Google tried to acquire Yelp to build local listings and relationships with local businesses. When Yelp refused, Google built out Google Local on top of Google Maps and Google Search, and now has direct relationships with thousands of businesses managing their information directly with Google. This was not very difficult when you have the dominant search product and the dominant maps product to build on top of. When you search a local query, you see no ads, just Google Local above all other search results.

The Buy Case: Google Flights
In July of 2010, Google acquired ITA Software, forever depressing the market caps of many travel-related internet businesses. ITA powered flight search and pricing information for many top online travel agencies. As you might have guessed, that data now appears on Google directly for free. Google is monetizing that space, and looks to moving from a pay per click model to a more transactional model over time.

The Partner Case: Google Events
Late last year, Google launched a dedicated section when people search for events that aggregates events from third party sources, including Eventbrite. Third parties give their inventory to Google, and Google ranks the events on its own. You cannot rely on your business to be in this category. Google will likely partner if:

  • There is no dominant player they can buy
  • Supply is fragmented and data unstructured
  • There are multiple companies willing to implement specific markup to appear in these discovery experiences
  • The area is not one of the leading categories for monetization for Google today

What do you do if you’re affected?
If you are an aggregator, and Google is moving into your space, it changes your SEO strategy entirely. Whereas you used to create and optimize pages that aggregate inventory for popular queries e.g. “san francisco food delivery” for Grubhub, those pages will now be de-valued as Google replaces those listings with its own aggregator. The best solution to this problem is to shift your strategy to distribution of your individual listings, so that you can outrank competition inside Google’s new discovery experiences. These pages usually need to updated to AMP formats, as that it what is powering all of these new discovery experiences inside Google search.

Many companies will attempt to opt out of this strategy, thinking it will help them preserve their current rankings if they delay or hurt Google’s ability to build a compelling, competitive aggregator to their own. This is unlikely to work. If there are any competitors to your aggregator, game theory will incentivize one of them to partner with Google to steal share. If you are a monopoly and opt out, it incentivizes Google to build a competitor that will threaten your monopoly, you will receive a lot less traffic that also threatens your monopoly in the interim, and Google can dedicate a lot of resources to a competitive product over many years. This appears to be working with Google Local vs. Yelp. Certain companies have been able to thrive despite these types of platform changes in the past by building loyal audiences with high switching costs, like Amazon with Amazon Prime when Google launched Google Shopping.


Google’s strategy has changed, though it will take years to propagate throughout every popular category of search queries. You can’t fault them for this change, as it is the right response to cater to their users. Right now is the right time to understand their strategy and to best position your company for its inevitable rollout. Gone are the days where you can rely on Google for a steady stream of free customers without putting in that much effort. You need to think strategically about where the company is going, if there are still opportunities where your content can attract Google visitors, and how you maximize the now declining opportunity.

Thanks to Randy Befumo for helping me through an early draft of this.

Currently listening to Challenge Me Foolish by μ-ziq.

Feature/Product Fit

Through various methods, Silicon Valley has drilled into the minds of entrepreneurs the concept of product/market fit. Marc Andreessen says it’s the only thing that matters, and Brian Balfour has an amazing series of posts that talk about how to find it. But what happens after you find product/market fit? Do you stop working on product? I think most people would argue definitely not. Post-product/market fit, companies have to balance work creating new product value, improving on the current product value, and growing the number of people who experience the current value of the product. While I have written a lot about how to do that third step, and even wrote a post about thinking about new product value, I haven’t written much about how do that second part.

What is Feature/Product Fit?
Every product team tries to make their core product better over time. But sadly, at most companies, the process for this is launching new features and hoping or assuming they are useful to your existing customers. Companies don’t have a great rubric for understanding if that feature is actually valuable for the existing product. This process should be similar to finding product/market fit, but there are some differences. I call this process feature/product fit, and I’ll explain how to find it.

In product/market fit, there are three major components you are searching for. I have written about my process for product/market fit, and Brian Balfour, Shaun Clowes, and I have built an entire course about the retention component. To give a quick recap from my post though, you need:

  • Retention: A portion of your users building a predictable habit around usage of your product
  • Monetization: The ability at some point in the future to monetize that usage
  • Acquisition: The combination of the product’s retention and monetization should create a scalable and profitable acquisition strategy

Feature/Product Fit has a similar process. We’ll call this the Feature/Product Fit Checklist:

  • The feature has to retain users for that specific feature
  • The feature has to have a scalable way to drive its own adoption

Feature/Product Fit has a third requirement that is a bit different: the feature has to improve retention, engagement, and/or monetization for the core product.

This last part can be a bit confusing for product teams to understand. Not only do the products they are building need to be used regularly and attract their own usage to be successful, they also need to make the whole of the product experience better. This is very difficult, which is why most feature launches inside companies are failures. What happens when a feature has retention and adoption, but does not increase retention, engagement, or monetization for the company? This means it is cannibalizing another part of the product. This might be okay. As long as those three components do not decrease, shipping the feature might be the right decision. The most famous example of this is Netflix introducing streaming so early in that technology’s lifecycle, which cannibalized the DVD by mail business, but was more strategic for them long term.

What is a Feature Team’s Job?
You would be surprised how many core product features are shipped when the new feature decreases one of those three areas. How does this happen? It’s very simple. The team working on the feature is motivated by feature usage instead of product usage, so they force everyone to try it. This makes the product experience more complicated and distracts from some of the core product areas that have feature/product fit.

If you own a feature (and I’m not saying it’s the right structure for teams to own features), your job is not to get people to use that feature. Your job is to find out if that feature has feature/product fit. You do this by checking the three components listed above related to feature retention, feature adoption, and core product retention, engagement and monetization. During this process, you also need to determine for which users the feature has feature/product fit (reminder: it’s almost never new users). Some features should only target a small percentage of users e.g. businesses on Facebook or content creators on Pinterest. Then and only then does your job shift to owning usage of that feature. And in many teams, it’s still not your job. The feature then becomes a tool that can be leveraged by the growth team to increase overall product retention, engagement, and monetization.

Mistakes Feature Teams make searching for feature/product fit
Feature teams commonly make mistakes that dissatisfy the third component of feature/product fit at the very beginning of their testing.

  • Mistake #1: Email your entire user base about your new feature
    Your users do not care about your features. They care about the value that you provide to them. You have not proven you provide value with the feature when you email them early on. Feature emails in my career always perform worse than core product emails. This inferior performance affects the value of the email channel for the entire product, which can decrease overall product retention.
  • Mistake #2: Put a banner at the top of the product for all users introducing the new feature
    New features usually target specific types of users, and are therefore not relevant to all users. They are especially irrelevant to new users who are trying to learn the basics about the core product. These banners distract from them, decreasing activation rates. It’s like asking your users if they’re interested in the Boston Marathon when they don’t know how to crawl yet.
  • Mistake #3: Launch with a lot of press about the new feature
    PR for your feature feels great, but it won’t help you find feature/product fit. PR can be a great tool to reach users after you have tested feature/product fit though. It should not happen before you have done experiments that prove feature/product fit. And it will not fix a feature that doesn’t have feature/product fit.

Many features won’t find feature/product fit
Many of the features product teams work on will not find feature/product fit. When this happens, the features need to be deleted. Also, some older features will fall out of feature/product fit. If they cannot be redeemed, they also need to be deleted. If you didn’t measure feature/product fit for older features, go back and do so. If they don’t have it, delete them. Some of our most valuable work at Pinterest was deleting features and code. A couple examples:

  • The Like button (RIP 2016): People did not know how to use this vs. the Save button, leading to confusion and clutter in the product
  • Place Pins (RIP 2015): Pinterest tried to create a special Pin type and board for Pins that were real places. As we iterated on this feature, the UI drifted further and further away from core Pinterest Pins and boards, and never delivered Pinner value
  • Pinner/Board Attribution in the Grid (RIP 2016): Attributing Pins to users and their boards made less sense as the product pivoted from a social network to an interest network, and cluttered the UI and prevented us from showing more content on the screen at the same time

How do I help my feature find Feature/Product Fit?
All features should be launched as experiments that can test for feature/product fit. During this experiment, you want to expose the new feature to just enough people to determine if it can start passing your feature/product fit checklist. For smaller companies, this may mean testing with your entire audience. For a company like Pinterest, this might start with only 1% of users. The audience for these experiments is usually your current user base, but can be done through paid acquisition if you are testing features for a different type of user.

I’ll give you a few tactics that have helped the companies I’ve worked on find feature/product fit over the years. Most good product development starts with a combination of data analysis and user research. User research should be involved at the right times to add the most impact, prevent confirmation bias, and determine what components users are struggling to see the value of. For example, when we launched the Grubhub mobile app, we saw in the data when people used the current location feature, their conversation rate was lower than people who typed in their address. This turned out to be an accuracy problem, so we turned that component until off until we were able to improve its accuracy.

In research, we saw people were having trouble figuring out which restaurants to click on in search results. On the web, they might open up multiple tabs to solve this problem, but this was not possible in the app. So, we determined what information would help them decide which restaurants were right for them, and started adding that information into the search results page. That included cuisine, estimated delivery time, minimum, star rating, number of ratings, On the surface, the page now feels cluttered, but this improved conversion rates and retention.

Since Grubhub is a transactional product, we were able to leverage incentives as a strategy to help find feature/product fit. Our early data showed that people who started to use the mobile app had double the lifetime value of web users. So, we offered $10 off everyone’s first mobile order. This transitioned web users to mobile for the first time and acquired many people on mobile first. The strategy was very successful, and the lifetime value improvements remained the same despite the incentive.

Grubhub also uses people to help find feature/product fit. Since mobile apps were new at the time (and we were the first food delivery app), we monitored any issues on social media, and had our customer service team intervene immediately.


Not all complaints are this entertaining.

At Pinterest, we launched Related Pins in 2013. For Pinterest, we did not have a revenue model at the time, so tactics around incentives and people do not make as much sense. One thing we did instead was use notifications to drive feature/product fit. Once this algorithm was developed, after you Pinned something, we could now email you more Pins you might like that are related to that Pin. These emails were very successful.

Pinterest also used the product to drive feature/product fit. We launched the algorithm results underneath the Pin page to start, and interaction when people scrolled was great. But many people didn’t scroll below the Pin. So, we tried moving them to the right of the Pin, which increased engagement, and we started inserting related Pins of items you saved into the core home feed as well, which increased engagement.

The Feature/Product Fit Checklist
When helping a product find feature/product fit, you should run through this checklist to help your feature succeed:

  • What is the data telling me about usage of the feature?
  • What are users telling me about the feature?
  • How can I use the core product to help drive adoption of the feature?
  • How can I use notifications to help drive adoption of the feature?
  • How can I use incentives to help drive adoption of the feature?
  • How can I use people to help drive adoption of the feature?

When confirming feature/product fit, you need to ask:

  • Is the feature showing retention?
  • What type of user(s) is retaining usage of the feature?
  • How do I limit exposure to only those users?
  • What is the scalable adoption strategy for the feature for those users?
  • How is this feature driving retention, engagement, or monetization for the overall product?

Every company wants to improve its product over time. You need to start measuring if the features you’re building actually do that. You also need to measure if existing features are adding value, and if not, start deleting them. Asking these questions when you build new features and measure old features will make sure you are on the path to having features that find feature/product fit and add value to users and your business.

Thanks to Omar Seyal and Brian Balfour for reading early drafts of this.

Currently listening to Breaking by Evy Jane.

The Email Marketing and Notifications Evolution Inside Companies

“Stop sending emails like a marketer. Start sending email like a personal assistant.”

I’ve communicated this line in a lot of my presentations on growth, but I haven’t talked about in depth the evolution on how to get there. There is a clear evolution that companies follow in terms of evolving their emails and notifications, from not sending them at all, to sending one-off blasts to their entire audience, to creating a lifecycle, to having a holistic personalized messaging platform. Having worked on these systems at my last two companies, I thought it would be beneficial to outline these transitions for people earlier along in their process.

Phase 0: We Hate Email

“We hate email, so we don’t send it to our customers.”

Almost every company I have worked with has communicated this at some point in its early stages, and it’s always wrong. Except for very specific groups, people don’t dislike email; they dislike bad email. The path to figuring what email your customers would like to receive is largely to ask what kind of value do they see in my product, and can I deliver that value in email form.

Phase 1: Mass Promotional Email + Personalized Order Notifications
If you are a transactional service, you have to send personalized order notifications, so that is where most companies start. At some point later, companies start sending mass emails to their broader audience about certain things like new features, discounts, etc. This effort will show improvements in key metrics, but it is very unsophisticated.

Promotions train users to wait for promotions to order, decreasing profitability. Also, with this approach, marketing is assuming a cadence for the customer instead of adapting to the customer’s cadence (and ultimately improving customer cadence to increase lifetime value).

If you are engineering constrained, there are some simple optimizations to this approach that will improve your performance:

  • Develop additional emails intended to drive habit formation (instead of just timely purchase). Examples include trending items, item sales, new merchants added, recommended items
  • If emails are successful, test them as push notifications as well
  • Take existing confirmation emails, and add marketing messages to them (other things to buy, set up a re-order, most popular items, etc.)
  • Send every non-transactional email and tweak to confirmation emails as an experiment with an enabled and control group to prove impact on lifetime value vs. unsubscribes/push permissions/app deletions

Phase II: Moving to Lifecycle Messaging
In Phase II, these additional emails and notifications that have been successful in Phase I start to form an automated program that consistently drives additional engagement from customers. In order to address messaging fatigue, these email and notification templates are managed to a frequency per month based on the team’s expected value of a good customer. This frequency is not based on data, but if everyone used the product properly, what would ideal frequency look like. The goal is to use messaging to remind people of the service and reinforce the habit. They are paired with personalized discount emails intended to drive new use cases and increase frequency.

These emails and notification templates are also managed against each other, so messaging does not get stale. For example, if you have three templates outside of confirmation emails, and you sent template 1 last week, you would attempt to send templates 2 and 3 before sending template 1 again. Also, each week, these emails have new subject lines to present them from looking like the same email as the previous weeks.

Phase III: Holistic, Personalized Messaging
As the Phase II approach flattens in terms of the additional impact it can drive, companies shift toward a more holistic, personalized system. This is a considerable investment, which we made at Pinterest. Essentially, product and engineering determine for each customer:

  • The right content
  • The right time to send it (day of week and time or day)
  • The right amount (how many emails and pushes to send)
  • The right channel (email, push, or both)

This requires a team to develop a log of every email/push sent to a subscriber, when it was sent, when it was opened, when it was clicked, what downstream engagement occurred from clicks, and which template it was. All emails and notifications are run through the same experiment dashboard as product changes to understand the impact on all key metrics. From this, it needs to determine:

  • The best day(s) of week and time(s) to send messages to each user
  • A prioritization of the templates to send based on historical click through rates and/or purchase rates
  • How many messages per a generic time frame maximizes lifetime value of each user

This usually starts via a rules based approach, and eventually becomes powered by machine learning. If you lack enough historical data on a user to do this, for example new users, you group people who used to look like those users as a segment i.e. previous new users and look at the best performing approach for them. Email can no longer be considered marketing at this point. It is considered an extension of the core product.

The team also starts optimizing deliverability through choosing better message transfer agent partners and segmenting IP addresses for different templates to isolate issues. The team may also start investing in more advanced security measures like DMARC.

This is a considerable investment, which is why most companies only start building this once there are sending millions of emails a day with a lot of history from operating in the first two phases originally. At this point, companies know the value of email, and can justify the investment.

In my opinion, every company should end up at phase III at some point. The question is how long it takes to get there. This varies based on engineering constraints, scale, and how long it takes emails and notifications to flatten off in terms of additional engagement by the previous phases. Outsourcing this to a marketing technology company is also very problematic as it requires access to all of your user data, and any migration of data from system to system slows down performance. At a certain scale (like Pinterest), it is not even possible.

If you’re not at Pinterest’s level of sophistication, don’t dismay. Very few companies are. Just start to think about the long term evolution, and when is the right time to push for a step change in email and notification performance vs. continued optimization. It’s a big investment to shift from phase to phase, but the returns are usually worth it, and the impact of these emails and notifications in the current phase, and the struggle to improve their performance, should be what drives that decision to make additional investment to get to the next phase.

Currently listening to XTLP by μ-Ziq.

The Four Types of Traffic to Your Home Page

Every founder’s favorite project is redesigning their home page. It’s the introduction of your company, brand, mission, and hopes and dreams to all of your customers. At least, you think it is. When you analyze your actual business, you might find the majority of new customers don’t actually start on your home page. If you’re doing paid acquisition, they might land on specific landing pages designed for those ads. If you’re growing via social traffic or SEO, most people are landing on specific pieces of content. The typical growth team’s response to this urge is that the home page is not a high priority, and that they should work on landing pages or content pages This is correct. But the home page for some businesses is still a major source of traffic. You have to learn what type of traffic that is in order for a redesign of it not be a complete waste of time. I’ll talk about what those types of traffic are, how to identify them, and what types of designs you might pursue based on your mix of traffic.

Traffic Type #1: People who want to login
On sites with a lot of engagement, a lot of the traffic to a home page is from people who are already customers and want to login. In this case, your goal is to just make logging iin as easily as possible or auto-log them in via something like Google Smart Lock. Better yet, if you can make sure you never logout your customers, then they never see this page and get right back into the product. Facebook, Pinterest, and Tumblr are good examples of this.

While Facebook has sign up front and center, it has a top navigation bar for login, and that is where the cursor begins.

Pinterest has a unified sign up and login field with a friendly call to action that just says continue. It also has a dedicated Login button for those searching for it.

Tumblr has a login button front and center equal to the signup button.

The metric when you are optimizing for logins is login conversion rate. This may be harder to calculate than you think. Let’s do an example. Let’s say you have 10,000 people visit your home page. 2,000 login. Let’s say another 500 sign up on this page. Your login conversion rate is 2,000 / (10,000 – 500) = ~21%. You don’t actually know if the remaining 7,500 who did not login or sign up were there to sign up or login. If you have cookie data, you can check to see if they had ever logged in, and that may give a clue on how to segment them. If you don’t have that data, you assume they were there to login. If at all possible, you should never log people out. The best way to help them login is to keep them logged in. Tools like Google SmartLock are also effective.

Traffic Type #2: People who want to sign up
Someone coming to your home page directly is not doing so because they randomly type it into a browser. They already have some context. A friend told them about it, they read an article about, or something similar. Many of those people are already convinced and want to sign up, and the job of the home page is to get out of their way and make that as easy as possible. Generally, sites do this by putting a sign up form front and center, and making that form really easy to fill out. You can see how Facebook and Pinterest do this really well. If you look at the images above, Facebook’s signup formis right aligned, and Pinterest’s is front and center. There isn’t much to distract you from signing up.

The metric when you are optimizing for signups is signup conversion rate. It is similar to login conversion rate, just with signup as the numerator instead of logins, and logins are subtracted by the numerator. Given the same number from above, your signup conversion rate is 500 / (10,000 – 2,000) = ~6%. You still don’t actually know if the remaining 7,500 who did not login or sign up were there to sign up or login. So, to be conservative, you assume they were there to sign up.

Traffic Type #3: People who want to learn more
There can be a wide discrepancy between the people who come to your home page directly, and are not existing users. Some may want to sign up quickly like above, but others just want to learn more. Preferably, they would not like to have to give you their information before they understand if the site is for them. The ideal scenario for these these users is to see a free preview. If personal information is required to show a convincing product, then the home page sells the value instead of shows it, or asks for filtering criteria without asking for personal information. Tumblr, Pinterest, and Facebook all address this type of user in different ways. Facebook, as seen in the example above, left aligns the explanation of Facebook on the page. Pinterest and Tumblr create a separate call to action to learn more that triggers a dedicated “learn more” experience. Pinterst has a red call out with a How It Works button, and Tumblr has a clickable green bar at the bottom of the page asking “What is Tumblr?”. Both clicks result in scrolling explanations of what you can use these products for.
Tumblr:


Pinterest:

Traffic Type #4: People who are skeptical
There is a fourth type of new user that visits your home page. This is the person who heard about it, but is very skeptical. The best way to engage this user is to let them have free usage of the product for as long as possible. You usually encounter these users with very well penetrated products that are reaching the very late majority or laggards. Google is a good example of a site that lets you experience the product value without signing up.

You can see from the above examples that companies try to address multiples of these audiences at the same time on the page. In some cases, based on activity of the page, they will be able to bucket you into one of these groups definitively. Gibson Biddle has a great post on how Netflix evolved their design with regards to these different types of users and these users’ understanding of their brand over time.


As you’re working on your home page, you should first make sure it is the most important page to work. For many sites, their landing pages are where a lot more people get introduced to the product. When you do work on your home page, think about these four audiences, which one you really need to optimize for, and if you easily segment and address the other groups that are not your primary focus. Also, be sure to revisit thes decisions over time as audiences and brand awareness changes.

Currently listening to Mind Bokeh by Bibio.

Addressing Common Misconceptions about Food Delivery Marketplaces

I spent five and a half years working at Grubhub, from series A to right before IPO. This allowed me to learn about many of the intricacies of the restaurant market and food delivery in general. More people have started to take notice of the market because of a slew of market entrants and Grubhub ($9.1B), Just Eat ($4.8B), and Delivery Hero ($7.2B) being successful on the public markets. With that, more articles in the press. Articles… that are wrong. I am reminded of the Murray Gell-Mann Amnesia Effect, invented by Michael Crichton, when I read these articles. It says:

Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray’s case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the “wet streets cause rain” stories. Paper’s full of them.

In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.

As someone who does know, I want to explain some of these misconceptions, so people don’t think wet streets cause rain (even though rain does cause delivery orders).

Misconception #1: Restaurant Margins
One major gripe journalists cite about food delivery marketplaces (more so UberEats than Grubhub due to its higher fees) is that restaurants operate on slim margins. Therefore, if food delivery marketplaces are charging 15-30% for delivery orders, restaurants are not making any money. The issue is understanding the difference between restaurant margins and delivery margins, which are very different.

Most successful marketplaces are built on top of an under-utilized fixed asset. For food delivery marketplaces, this under-utilized fixed asset is not the restaurant, but the kitchen. Restaurants have a fixed capacity they can seat at a restaurant. The kitchen, however, is usually capable of producing much more food on a daily basis than is needed by the patrons that dine in the restaurants. Restaurants are paying for that kitchen capacity regardless of how much they use because one of the highest costs for most restaurants in cities is the cost of rent. That’s why I thought it was so silly when all of these delivery service startups started making their own food. You’re spending a lot of money to build what the incumbent gets for free: excess kitchen capacity to make food. This is why restaurants love catering orders so much. They get big orders that can better leverage their kitchen capacity. After catering, their next favorite is delivery.

Why delivery? It allows them to serve many more customers at a time with their fixed asset, spreading their fixed costs across many more customers. Catering and delivery are pretty much pure margin for restaurants because their only extra cost is a delivery driver (or not, in many cases) who is subsidized by the people ordering the food via tips.

Misconception #2: Paying for Repeat Orders
The second major gripe I hear about food delivery marketplaces is that they charge the same amount for a customer’s first order to a restaurant and repeat orders. Now, a lot of this is drummed up by a competitor who does not drive demand, so it is biased, but I’ll endear it. The misconception here is that all a restaurant has to do is pay an advertising fee to induce trial, and if the food is good, the customer will order again based on that experience. That is now how food delivery works. Food delivery is very fragmented, and while there is differentiation by way of restaurant quality, there are usually quite a few worthy substitutes. Also, the way delivery orderers usually make decisions is method first, restaurant last. The way the average person decides to use Grubhub operates something like this (flowchart):

Restaurant loyalty is one of the least important and last steps in the process of the person ordering food. This means people ordering food do not have loyalty, and you need to compete for every order as if it’s the first. Google Adwords does not charge Apartments.com less if someone who clicked their ad a year ago when searching “apartments” does so again the following year. The reason they don’t is because if Apartments.com wasn’t showing up there, that user would have gladly clicked the ad for ApartmentGuide. Restaurants should absolutely be working to build loyalty, and some do. But expecting that acquisition was the hard work, and restaurants should only pay a SAAS-level fee for retention does not align to the value these marketplaces actually create for restaurants.

Changes That Do Make Sense
Now, there are some elements I would change in regard to repeat orders if I worked at these marketplaces. While Grubhub already charges a much cheaper rate if the order originates from a restaurant’s own website, there are other forms of orders that the restaurant seemingly drives itself without the marketplace’s marketing engine or aggregation. One is if the person ordering food directly types the restaurant’s name into the marketplace. Charging a lower rate for that makes sense as the restaurant is clearly driving the business. I know from the data that this is a small percentage of orders, but this would show good faith to restaurants that marketplaces want to align their revenue model to the value they create.

Another scenario is if the person lands directly on the restaurant’s page on the marketplace from somewhere else e.g. Google. If this happens organically (because the marketplace ranks high or because the restaurant does not have its own website), the marketplace should charge a lower rate. The reality is Yelp and Google Local take 90% of this traffic, but again, it would send the right message.

The other example of this is a little harder to parse. These marketplaces also bid on restaurant names on Google. If that drives an online order, what should be the charge? The restaurant drove the demand, but the marketplace spent the advertising dollars to close the order. In this case, I think these marketplaces should evolve to asking if restaurants want this type of marketing, and if so, charge for the advertising as a service. This was not possible for Grubhub when I worked there due to game theory issues with all the competitors, but with a shrinking field of credible players, it may be possible.

The Three Personas: How Marketing, Product, and Analytics Attempt to Define The Customer

In my career, I’ve worked in marketing, product, and analytics. While the American Marketing Association defines all of that as marketing, the reality is those are rarely under the scope of one functional team, and the people in those groups see things very differently much of the time. One of the key ways this manifests that creates confusion for the organization is in the creation of personas. All three groups have their own ways to define personas that don’t tell the same story. And in many cases, these are called marketing personas even though they are very different. I’ll walk through each of them to try to define them separately, and talk about how to use each of them best and avoid common pitfalls.

The Analytics Persona
The analytics persona is the most straightforward of the group. The analytics persona is created by looking at clusters of users based on their usage and defining them based on that usage. At Pinterest, these were defined as core, casual, marginal, and dormant users. Core people came every day, casual people came every week, marginal people came every month, and dormant users had stopped coming to Pinterest altogether. This segmentation can be useful to see if your product is becoming more or less engaging over time. Key projects can include migrating people from, say, marginal to casual, by understanding the statistical differences between the two groups. Then, a product team might take something the casual group does that the marginal group does not and try to make the marginal group try to do that thing as well. Some of these differences are just correlation (or represent the people more so than the action itself being important), but some may be causal, and experiments like incentives or education to marginal users may help them become casual users.

The Product Persona
The product persona, just like the analytics persona, focuses on understanding existing users. In contrast to the the analytics persona, it relies on qualitative research to define who these people are, not just what actions they take. Someone that uses the product monthly can be the same personas as someone who uses it every day. When we built our personas at Grubhub, we had to use a mix of qualitative and quantitative research to define them. After a back of forth of customer calls and surveys, we were able to define four personas that used Grubhub based on two specific criteria: whether they ordered spontaneously or planned ahead of time, and whether they ordered for themselves or with others. When we mapped these personas back to our data, we were able to find that one segment was detrimental to serve because of high customer service costs, and that one segment was high potential, but low value currently because we had not built the right product for them yet. This helped inform our product strategy for the future.

The Marketing Persona
The marketing persona, unlike the first two personas, is usually a forward-looking persona: about who the company is going to try to reach. These are customers you want rather than customers you have. Marketing personas are common for product launches and market expansions since there are no existing users or data to build analytics or product personas.

The marketing persona exists to define a target market to go after. Marketers rarely try to target all people. They attempt to define a niche with specific needs (physical and emotional) and make their product attractive to that group. Marketers have many tools to define this persona. They pore over demographic and psychographic data and map competitive landscapes to spot opportunities for new markets. Since this persona is about targeting people outside the product, one common tool created during this process is a mapping of the target customer’s typical day. What they listen to on the radio on the trip to work, where they get their morning coffee, what they watch on TV during dinner, etc. From that, marketers identify opportunities to advertise to the target during one of those times to introduce them to the product.

Issues with Different Personas
All of these personas have pros and cons, and I recommend using them in tandem rather than a one size fits all approach. The danger with the analytics persona is it looks at people solely based on their activity and not also their motivations, their personalities, etc. There are key insights you will never find out from data that you can learn within ten minutes by talking to a customer. The product and the analytics personas also assume the current customers are the most important customers to understand. This is not always the case. Many times, you need to expand your market. Other times, focusing on existing users makes the product worse for incoming users.

Marketing personas also have their flaws. Since they are based on secondary data, marketers can sometimes invent personas that don’t exist or are too small to be important. This is why so many marketing teams create personas that are just cooler versions of themselves. It’s why whenever a fashion designer is asked who they design for, they say something like “a gallery owner in New York City.” They are maybe 500 of those people in the world. Now, I should be clear in that last scenario there is a blurred line between the marketing persona and the aspirations of the true persona, but you get the idea.

Marketing personas also can be unhelpful to some organic growth strategies or for products that have infinitely large markets since they are based on niches and targeting. When we tried to build marketing personas to target at Pinterest at around 50 million users, I asked, “What exactly are we going to do with this? We don’t spend money on advertising. Our targeting is defined by whoever searches for something on the internet that Pinterest is relevant for.” Similarly, does Google search have a target market? Sure, if you can call every single person with an internet connection a target. I bet you the Google Home team had a very specific target in mind for its launch though.

Personas of all types are also mistakenly used in lieu of personalization for many internet products. Email is a common example. Many companies still spend a lot of time creating dedicated emails for specific personas, segmenting lists, writing custom copy, and adding custom content. This frequently doesn’t makes sense in a world of personalization and one to one messaging. Pinterest has automated, personalized one to one messaging across email and push to over 200 million users. It doesn’t need to know what analytics, product, or marketing persona you are to be effective. This is not proprietary tech either. Many third party companies can help build this same type of system for smaller companies with less data.


Personas are very useful tools to help you identify opportunities to grow your business and better serve your customers. You should be using them in almost all phases of a company. Understanding the different types of personas, how you could use them, and how to prevent making mistakes with them is key to making sure they are worth the effort to define them.

Currently listening to Big Loada by Squarepusher.

Product Visionary vs. Product Leader

Many people want to work in product management. One of the most common questions I receive is how to break into product management. It’s a hard question for me to answer, because 1) there is no default path (the same is true for trying to land a business development role), and 2) most of these people really don’t know what they’re asking for. My most common response is, “Are you sure? Product management can kind of suck.” The reason for the dichotomy of people who haven’t done product management finding it so alluring, and people who have done it cautioning people trying to get in is the difference between what I call a product visionary and the product leader.

Product visionaries are who we all hear about in the press. They are the people who come up with brilliant products that go viral or solve real needs in the market that no one else thought of. They appear to be masters of finding product/market fit. I’ve been lucky enough to work with a few of them in my previous jobs and a few in the portfolio at Greylock. These tend to be founder/CEOs, and they generate brilliant insights that create product opportunities others don’t see. Ev Williams is on his third breakout product in Medium after Blogger and Twitter. Ben Rubin created two products that hit product/market fit in two years with Meerkat and now Houseparty.

Anyone working in technology hears these stories, and they think the shortest path to that sort of glory is becoming a product manager. They are excited to get to a role where they can drive the vision of a product, even if it’s only one part of the company. This excitement is exacerbated by the commonly propagated myth that the product manager is the mini-CEO of their product. The reality is that in 90+% of cases, product management is not about being a visionary. It’s about being a leader.

What does a product leader do at a tech company? It’s actually very little of creating a vision and a strategy from scratch. It’s about helping everyone understand what the vision and strategy is. It’s about communicating to the entire team why the company is doing what it is doing. It’s about building a process that helps a team execute on that vision. It’s about when there are competing visions, aligning and motivating the team to focus on one, and getting people to disagree and commit (including sometimes yourself). It’s about looking at data to measure if product changes are having a positive impact on the customer and the company’s growth. It’s about talking to users to understand why they’re doing what they’re doing, and the problems they still face even though your product exists. It’s about mentoring more junior people on your team, across product as well as engineering, design, and analytics. And they don’t have to listen to you, so you have to use influence rather than authority to be successful.

In the scaling phase of a startup, it’s product leadership that drives performance, not vision. Vision is needed early to find product/market fit and plot a course to scale, and then the less that vision wavers over time the better. This vision is usually done by the founder/CEO. The reason founders hire product managers and VPs of Product is not to set vision, but to help execute the vision. Don’t get me wrong; that will sometimes mean coming up with solutions to problems your customers face. If a company is scaling by having the founders solve all the customers’ problems instead of product teams, it will struggle. But much of the time, it will be wrangling the ideas of the individual engineers, designers, and analysts on your team and matching that to an overall vision set by the founder(s). It’s very rarely your ideas you’re executing on as a product leader, and it shouldn’t be.

I also don’t want to make it sounds like I am devaluing visionaries. They are, of course, critical in finding the initial idea(s) that create a growing company and maintaining a vision for that growing company. Having visionaries also becomes more important as you saturate your core market and need to tackle new value propositions to drive new growth opportunities. That is the ideal time for non-founder visionaries to enter a growing company. These are not going to be typical product managers or VP’s though. They are usually ex-founders. The best outcomes for these people entering an organization that is scaling is giving them a team and space to experiment with ideas until they find their own product/market fit, where a business unit is built out around that vision with product leaders to help them scale.

As you’re thinking about your business, think about whether you need a product visionary or a product leader. Most founder/CEOs are already visionaries, so they need a product leader to help execute. Some businesses minded founders need the opposite to be successful. If you’re thinking about product management, think about whether what you really want to be a is product visionary instead of a product leader. In that case, it might be a better idea to start a company than take a role expecting to execute on your vision and instead managing other people’s visions.

Currently listening to Ambivert Tools Vol. 3 by Lone.

How to Set Up and Hire an Analytics Team


Analytics has become a critical role at tech companies. A common question I receive is how to hire analysts and where they fit into an organizational structure. Below I share some tribal knowledge around common team structures, options I think work best and hiring tips that leverage this talent.

Functional vs. Embedded Teams
One of the first questions organizations face when hiring analysts is how they should structure the team. There are two common choices organizations pursue: a functional and an embedded model. The functional model is an analytics team reporting to a Head of Analytics. In the embedded model, every department (sales, marketing, product, customer service et al.) is in charge of solving for its analytics needs, hiring analysts for their teams when needed, and determining to whom they report.

Benefits of the functional model are having a senior seat at the table in major discussions for the company. The advantages this presents is getting analytics its own budget for tools and infrastructure and solving other analyst specific needs that may not be a top priority on any other specific department. The downside of a functional team is how analysts’ time are allocated. In a functional team, an analysts’ time is usually allocated on a project by project level, meaning they usually enter a project once that project is already well defined (whereas analytics could have helped by define the project, had it been involved earlier). And the analyst has not developed any specific expertise for that area. In my experience, analysts get frustrated in this model because they can’t go deep into any one area, and the other departments get frustrated because analysts provide a more cursory benefit than expected.

The embedded model solves a lot of these pain points, but introduces its own. With an embedded model, analysts are hired into one specific team, and therefore can develop expertise for that team very quickly. Teams are happy because they always have a teammate ready to help who understands their problems. While analysts seem to be happier in this model, the downsides are the reverse of the functional model. When there are cross-departmental analytics needs, they usually fall by the wayside. Investment in infrastructure and tooling is usually massively under-invested in, and it’s unclear where budget comes from to solve these needs.

At Apartments.com and at Grubhub, we implemented the embedded model. Marketing took control of analytics infrastructure initially, but we had trouble applying it cross-functionally. The analysts across all teams started meeting regularly to share learnings, but also limitations. When we added the Seamless team into Grubhub, they were used to the functional model. Analyzing the two together created awareness for me of a new approach.

The Hybrid Model
At Grubhub, the value of a dedicated analytics team for infrastructure and tools became clear, but also the value of the embedded analyst. Once I started at Pinterest and dealt with the functional model, we began to work toward a hybrid approach. This is a dedicated analytics team with a Head of Analytics, but with the analysts dedicated to specific areas full-time. So the person reports to a Head of Analytics, but sits with the department they support (in my case, the growth team). As the growth team grew, we created a Growth Analytics Lead who reported to the Head of Analytics and managed other growth analysts dedicated to specific areas, like conversion optimization or on-boarding. This allowed Pinterest to have the analytics seat at the table for budgets and resourcing, but the expertise at the team level to make the most impact. It’s now what I recommend to all teams that are scaling.

Hiring Analysts
If you are scaling a company and need more analytics help, it can be hard to understand who to hire that will actually help your teams. Hiring from analysts at other companies, especially larger ones, proved not to be a great strategy for me. I found during the interview process that most analysts were actually what I call “reporters”, in that they ran well defined reports for people who needed them but didn’t actually analyze anything. If you read analyst job descriptions, they inadvertently screen for these types of people by saying the candidate needs experience with all of these special tools. I can’t tell you how many job requirements that list Omniture (or whatever it’s called this week), Google Analytics, SPSS, Tableau, etc.

Experience with tools is not actually what you care about (though SQL and Excel are a big help). The more tools someone has worked with, the less likely they are to analyze the output of those tools. What you actually want are people who are analytically curious. Our first successful analyst at Grubhub was a new graduate whose cover letter talked about how he tracked his sleep patterns and his diet to find ways to improve his health. He crushed dozens of analysts with multiple years’ experience in our interviews because he was using his brain to analyze results instead of just report. So I now screen for roles where analysis, not reporting, is the unit of value. Many analyst teams at other companies are structured that way, but the majority are not.

You also have to test analytical ability in these interviews. At Grubhub, I gave people a laptop with a bunch of data in Excel and some vague questions to answer from it. The question was based on a real question we gave an analyst intern, who returned it to me saying there was no trend in the data. I ran the analysis myself and found one of the most important correlations for our business (the impact of restaurants per search on the likelihood to order). So I said, you have to be better than our intern to get an offer. It turned out to be an incredible screener. Most people never got far with the data, or their answers were spectacularly wrong. The few good analysts cut right through to a direct way to solve the problem and could explain it easily.

I like this approach because it actually shows the analyst what the job is (and if they’ll like it), and I can walk the candidate through how I would solve the problem so if they did get it wrong, they could learn something from it.

 — 
Analytics team are one of the hardest teams to scale. One of the keys is building a model of a team that will scale with the needs of a company, and the hybrid model is the best model I have found to maximize the important levers of effectiveness (and happiness!). Structure is not all that is important though. Hiring the right candidate is critical, and the market is doing a poor job of preparing people for what emerging companies actually need in an analytics organization. If you can hire correctly and structure correctly though, you will have a competitive advantage over those who do not.

Currently listening to Rezzett by Rezzett.