Tag Archives: team building

Buffing, Nerfing, and OP: What Video Games Can Teach Us About Talent Management

One of the more interesting, but less talked about, dynamics in the video games industry in the last few decades is that the product they initially ship to customers is no longer the final product. Because of online connections via mobile phones, PCs, and consoles alike, video game creators can push constant updates to their games to make them better over time in response to real-time player feedback and behavior. This makes their approach to development much more like other software in that it can be more agile and less like producing a film or a piece of hardware.

Like any change in the video game community, gamers notice, and a whole lexicon has been created based on these new abilities of game developers. One of the more common genres of video games that receive these changes over time are fighting games and shooters. If you grew up pre-internet like me, then examples include Street Fighter and Mortal Kombat and GoldenEye. If you’re, the youth, as they say, then you might be more familiar with Overwatch, Fortnite, and Call of Duty.

What these franchises do now is ship their initial products, watch the community play, and make adjustments over time. As the community plays, players and developers notice that certain characters or items are incredibly powerful, perhaps more so than the developers intended, or they change the game in some unexpected way. The community deems these OP for overpowered or OD for overdone/overdosed.

Before game developers could update their games over the air through the internet, that would mean certain characters would stay overpowered and make games lopsided if you didn’t use those characters. Players would create rules to adjust for these issues, like not allowing anyone to play as Oddjob in GoldenEye, or just blaming losses on your friend using a “cheap” character.


Oddjob was considered OP in GoldenEye because he was smaller than other characters, making him harder to hit.

What developers now do is notice these patterns and update the games over time to re-balance the game. When this occurs through over the air updates, serious gamers read through the patch notes to figure out who has been buffed and nerfed. Buffing is when a character or item is underpowered, and the developers do something to make it stronger. Nerfing is when a character or item is overpowered, and the developers do something to make it weaker. When these patches occur, the power dynamics change overnight in the game, leading to a lot of players trying new characters to see who they can gain an advantage with. This also can extend a game’s lifetime because it forces people to try new things in the game.


Cassie Cage was nerfed in the first major update to Mortal Kombat 11. Holy graphic improvements, Batman!

I find the concepts of buffing and nerfing fascinating and incredibly relevant to organizations, where employees are the players. All organizations are going to, intentionally or unintentionally, overpower people or areas of the company over time. Organizations need more tactful ways to adjust when these mistakes occur, and the buffing and nerfing of the video game industry is a concept I think can be applied successfully to organizations in this case.

So, how do founders and executives get good signal as to the current dynamics of their “players”? Video game developers watch play data and observe the community. Founders and other executives don’t usually have the same analytics video game developers have, so they need to rely on qualitative signals. Many organizations use business results and culture surveys as a signal into organizational success, and use performance reviews, 360’s, 1:1’s, calibrations, skip levels, Q&A’s, and all hands to build even more signal. If you’re not doing any of these today in your organization, start. Organizations usually know how to use these tools to identify who is doing well and needs to be buffed, and this can result in more scope, more resources, or official promotions. But our tools for nerfing are incredibly crude today, so I’m going to talk more about how to effectively nerf in an organization.

How To Execute a Nerf — Conditions for Play

One thing I want to call out about being overpowered. This is often the failure of an organization, not an individual or a team. We tend to treat OP situations in companies as a failure of the employee or team because it’s a lot easier, and we have mechanisms for those situations, such as demotions and firings. And this is typically the way organizations fix OP problems. To do a nerf effectively requires an organization to understand that it failed in structure or talent management, creating the OP situation. Talent management doesn’t just include performance management, but mentorship opportunities and career pathing. Only then, can it think about using these nerfing tactics effectively.

Nerf Tactic #1: Change the process

One way an employee or team becomes OP is because they manage a key process in some way that gives them undue influence over others. The easiest change then is to change the process to re-balance some of that power. In order to understand how to do this, you need to a) understand the process and b) understand the failings of it according to others. When members of your team that are OP manage a process, they seldom see the flaws in it because of how much control it gives them. So you have to seek out different people’s opinions in the organization as to what is going wrong. A signal that revisiting a process may be a good solution is when other senior team members complain, or if they have enough power, start opting out of certain other processes.

Nerf Tactic #2: Change the structure

Another way an employee or team becomes OP is the structure itself. This tends to happen when functions are grouped. For example, if a GM also has control of sales and marketing, they may become OP. Or if your CMO owns customer service, they may become OP. Neither of these are bad if they occur, but they may end up giving that executive too much power, so they can bully other teams, or they may not be as effective at managing either the scope or the function they’re less familiar with. In this case, the easiest path is to separate these functions or create more dotted line ownership to re-balance the teams.

Nerfing Mistakes to Avoid

#1: Don’t Wait Too Long to Nerf

The longer a power balance is allowed, the higher the chance that the organization will blame the strife on the person/team that is OP rather than the structure itself. Even when that person or team is nerfed, animosity will remain that may make the organization not want to work with the person or team, and continue to impair their performance. When you wait too long to nerf, you’re effectively destroying people’s careers in your organization, which may lead to them leaving. The worst situation is when you have to let go of someone who could have been great with a less powerful scope, but you waited too long to do it. In this case, firing or radically moving the person to something else is the only option.

#2: Don’t Change Titles When You Nerf, Especially Downward

As I said earlier, the most common way organizations nerf today is through demotions, which implies the employee failed instead of the organizational structure. If you absolutely have to change titles because the scope change is that significant, I prefer keeping the level the same, just changing the functional name of the role. Say, for example, you have a Director of Strategy, and they became OP, but you want to retain them. Changing their title to Director of Competitive Research is better than changing their title to Manager, Strategy. Why is this? The demotion indicates failure not only to them, but the rest of the organization. These people usually exit their organizations shortly after the demotion.

Organizational Mistakes That Prevent Effective Nerfs

#1: Inflate Titles

I was talking with a startup recently who hired someone to be their first product manager. They had made a mistake that happens commonly in startups — to make the role attractive or to reward performance once in the role, startups inflate the role title (to be more senior than what that role means on the market). One common example is calling your initial Product Manager a VP Product, or worse, a Chief Product Officer. When the company grows 3x and you’re now asking that product manager to manage people for the first time, these people frequently aren’t ready. Then, in order to fix your organizational mistake, you either have to change their title, which will look like a demotion to them and the organization to hire someone more senior, or try to seek out on the market effective mentorship for them to scale up their career progression faster than is normal.


In the words of Keith Rabois: Wrong.

It’s enticing to inflate titles to make people happy or to attract talent, but it removes your ability to nerf without demoting. In practice, I like to reserve C level titles for public companies and VP titles for managers of managers, though I will be the first to admit that impact is not defined by the number of direct reports. This is not always possible with companies that need their employees to do a lot of external to company work. “Head of” is a happy medium startups are starting to use, but it can still feel like somewhat of a demotion when that title is converted to, say, a Director, when a VP is hired.

#2: Confuse Being OP and Poor Performance

I want to be perfectly clear: do not nerf for poor performance! You nerf when someone was performing well, and a process or scope changed to unintentionally give them too much scope. If you have a good CMO, and you ask them to take Customer Service, and they don’t manage it well, you still have a good CMO. Change the scope, not your performance management strategy for the CMO.

What if you promoted someone to a new level, and they are not meeting it? Well, that is not following performance management best practices. You should be promoting someone when they are already performing at the next level for some time, which eliminates the risk of overpowering them. This is why you want to have career guides with clear levels and expectations for them, so you promote only when it is appropriate. Hired someone at too high a level? Do not nerf; demote or fire. Yes, they will most likely leave, which is why you want to be careful about the titles you give just based on interviews and not actual job performance. What if you have a big hole to fill in the organization and have to do it internally with someone who hasn’t demonstrated all the skills necessary? It happens, but recognize this should be more of a last resort situation.

What about people who move into new roles where they are not effective? This is also a situation I prefer not to have happen. I much prefer creating apprenticeship programs within organizations where individuals try the role for a while before they are permanently placed in that role. Apprenticeship is therefore the evaluation of whether the person can meet the expectations of the role. I recognize this is not always possible, but it’s a sound strategy nonetheless. We do this in product management at Eventbrite today. Those in other parts of the organization in good standing with their manager can work with a product manager on a project to see if they like it, and for the organization to see if they would be good at it.

Nerfing Best Practices

#1 Take the Blame

If you are nerfing an individual, it’s important to make sure they don’t feel like it’s a demotion. When announcing the nerf to your employee, make sure they know the reason for the nerf to occur, and that it is your fault, not theirs. You gave them too demanding a scope, and you’re fixing it. If you’re nerfing a team, this is usually less of an issue. Also, make sure that others in the organization that were affected by the OP individual or team know that you believe this to be your fault, and they should not hold animosity toward the individual going forward.

#2 Thank the Individual You’re Nerfing

Whether the change that made the individual or team OP was intentional or not, thank them for their effort to help the company, and say that the nerf is a way to make sure they can create a positive impact for the company with the right scope. People should not be penalized for trying to step up for the organization.


I think it’s incredibly important to manage performance of people and organizations. The gaming community has created a vocabulary that provides a solid analogy with corporations, allowing us to separate performance issues from structural mistakes. The more companies can separate those issues from each other and use different tactics to manage them, the stronger those organizations will be.

Currently listening to my 2019 playlist.

Thinking Outside the Job Title Box: How to Thrive in Undefined Roles

As a leader of a large team, the members of my team tend to have pretty well-defined roles, like designer, or product manager, or researcher. They also tend to interface with other employees of the company with pretty well-defined roles like engineers, analysts, data scientists, etc. Now, most of the time, these well-defined roles operate in cross-functional teams. But, what if you don’t operate within one of those roles, or don’t want to fit the mold of these well-defined roles? How do you work with teams? To answer that, it may first be helpful to understand how these teams form.

“In the beginning, there was an engineer.”

Most startups begin with an engineer building something from scratch. As the company scales, usually a designer is added next. Then, as keeping track of projects between the designer(s) and engineer(s) becomes onerous, a product manager is added. Then, as the team scales and problems become harder, the product manager and engineer(s) don’t have enough time to fulfill the analytical needs of the team, so they hire an analyst. Then, keeping up with users becomes too time consuming for the designer or product manager, so they add a researcher, etc. This is an overly simple example, but all cross-functional teams grow larger as the company scales, with more specialized roles over time. One issue you may have if you don’t fit into this model is that you want to perform a more specialized role than the company has scaled into needing yet.

The opposite can also be true. You may want to do bits and pieces of a well-defined role, but not all of it, or may want to combine some elements of a few different roles into your job. Either of these situations can be totally fine. But each require a more tactical, subtle approach to working with teams than fitting within well-defined roles. When people outside of the cross-functional teams want to work with these cross-functional teams, they frequently perceive friction. They interpret this as political, when in fact, it is structural.

By definition, when you don’t have a clearly defined role to others, they do not know how to work with you. The onus is on you to prove value so that they want to work with you, because they don’t have to. Usually, my advice when people come to me with these problems is to switch to a more defined role. There may be a valid reason to leave your role less defined though, and in this case, I propose a framework for finding a valuable fit within a cross-functional team.

While all cross-functional teams have well-defined roles on paper, in reality, for all the needs of a team, people within the cross-functional team trade off responsibilities based on skill set and interest. You may have a PM who’s better at execution, so the designer takes on strategic duties, for example. What you’re trying to find with a less defined role is a team that has a need for a skill set, and wants someone to fulfill that need.

What happens in each of these boxes? Well, if your role is not needed or wanted, then you don’t get an opportunity to help. If your role is wanted, but not needed, you tend to be superfluous and lowly leveraged. If your role is needed, but not wanted, you experience rejection. If your role is wanted and needed, that’s where the magic happens. You integrate into the cross-functional team well, help them achieve their goals, and likely are happy in what you’re doing.

Two Tips to Make These Roles Work

#1 Get a senior leader to sponsor your effort
It’s almost impossible to make this type of role if your manager and ideally someone very senior in the organization support it. If senior leaders are very strict about team formation, it just might not be the company where you can be successful in a non-uniform role. Also, if your manager doesn’t support the direction you’re targeting, there will be a big mismatch come review time that will stifle your career.

One way to be more successful with managers and senior leaders is to be very clear why you want to work this way and what value it adds to the company. I would not recommend going to managers and senior leaders suggesting you not fit a typical role in the organization, and then ask how you can be effective. You are not giving them two problems: 1) someone who won’t fit into the traditional organizational structure and 2) someone who doesn’t know how to help the company. For many managers, this would trigger them to ask why you’re at the company in the first place if you don’t know how to help.

#2 Approach teams with humility
Your approach to talking to managers and senior leaders about your role should be very different from how you approach teams. While with managers and senior leaders, you want to make a clear case of the value you can add and what you want to do, that can not work so well for approaching teams. A better approach gets across a few key points:

  • You’re interested in the problem they’re working on
  • You think they are doing interesting work
  • You’re just here to help in whatever way you can
  • These are the skill sets you have that may be valuable
  • Finally, the ask: What can I help with?

Whether you have a traditional or non-traditional role in a team, the first step is building trust, and that is usually earned by doing smaller tasks the team is not getting to and would like help on. From there, you earn the right to work on more critical tasks. It may take some time to get to the role you’re most interested in playing on the team, and that is normal.


At Eventbrite, we have multiple people who sit outside the traditional paradigm of well-defined roles who are thriving. They all found a way to add value to a team that was wanted and not competitive, and the team operates better for it. But this all happens on an opt-in basis. The teams chose to accept them. If you want to play outside the lines, you have to understand that other teams playing with you is entirely opt-in on their part. This is the risk of not participating in the structure the company operates in; you may find opportunities to help that the team isn’t welcoming to, and there’s nothing you can do about it.

An Alternative Approach to Re-Orgs At Your Company

Re-orgs are an essential part of scaling a team at a company. The organizational structure of the company six months ago may no longer align to the needs of the company or its customers today. While most people would agree with the statement above that insists re-orgs are necessary, everyone hates them. They almost always make some people unhappy, cause employee departures, and stifle productivity both before and after they are executed.

I’ll start with a story of how this works in practice. Grubhub had a fairly stable structure for most of the time I worked there. While it was stable, it certainly wasn’t traditional. While we had crafted large sales, marketing, and customer service teams, we had a very small engineering team for our size and no official product and design teams. The latter two we de facto managed by marketing and a combination of executive leadership. While most companies at the time had a clear product manager role, Grubhub did not. We had product strategy led by marketing and the co-founders, and project managers within engineering that worked with those stakeholders to build effective software. I hired a member of my team to build a loyalty program. This meant that they would do user research, build models that project impact and costs, and work with engineering to launch experiments that would increase frequency of ordering on Grubhub. The person we hired was, in short, awesome. She did a bunch of great research partnering with our UX researcher, built detailed financial models that projected impact, brainstormed many ideas with our designers, and built good rapport with our project managers and engineers when it came time to finally build something.

Around this time, we started discussing as a leadership team if it made sense to start building a product management function for Grubhub. Being privy to those conversations, as I was having my quarterly review with this member of my team, I suggested product management would be a good avenue for her if we create that function, but she would probably need to leave my team to do it. We talked through the details of why I felt that made sense given what she was doing, and she was very open to it.

Fast forward three months, and as I’m on my way to work, I receive an email from my manager the VP Marketing asking to meet when I get in (she always got in super early). When I did, she announced that we were creating a product management function, and that as they thought about what the members of that team should look like, they felt like this member of my team was a perfect model of what a product manager should be at Grubhub. So, effective immediately, they were moving her off my team to be the first consumer product manager. The co-founder of the company was meeting with her when she got in to explain the move, and email would go out right after, and my 1:1 was with her later in the day.

That 1:1 was awkward. While this was ultimately what I wanted for her, and she was nervously happy about it (it’s nice having the co-founder of the company say your behavior is a model of behavior they want at the company), things still felt off. What is supposed to happen to her current projects? What new projects is she picking up? She at one point during the meeting said “oh, you seem sad”. And I wasn’t, just more caught off-guard, and thinking even though we’re making the right decision, are we making it in the right way?

This I find is usually the best case scenario for re-orgs. VPs and C level execs are attuned enough to make the right calls, but execute it top down without director, middle management or IC involvement or feedback on their ideas, leading to a change that is good on paper and may be good in practice too, but seems to strip the team of control. And far more common is the flip side of this scenario: when VPs and C level execs think they know what is good for the people and the team, don’t seek out necessary feedback, and make the wrong call for both the organization and people’s careers who are affected by the re-orgs.

For one of my the companies I advise, going into 2019, for one our business units we knew we likely had to change our organizational structure. Trying not to repeat re-org mistakes, we started working on a structure that would make the re-org act like a feedback-fueled progress driven by the teams instead of by people above them. The first thing we worked on as a leadership team was the objectives for 2019. What did we need to achieve next year to be successful? We then went to the product managers, designers, and engineering managers and explained the objectives. We then tasked them to propose the organizational structure that would help them with these objectives.

They worked directly with their teams to make sure everyone understood the objectives, everyone’s interests and career aspirations, and then they proposed the structure to the leadership team. After this presentation, we worked more to understand the constraints that led to this recommendation, worked through some of those constraints so the team didn’t need to make as many compromises on what they wanted, and then solidified the structure. The product managers, engineering managers, and designers talked through the changes with the rest of the teams, and organically the teams started planning with the new structure in mind. They then set their own team objectives to the align to the business units, as well as their roadmap and key results.

By involving the team members that would be effected from the beginning and making it their decision, we avoided a lot of the awkwardness or bad calls of many re-orgs I have participated in. The teams are happy and working on the new objectives seamlessly. Now, there will certainly be re-orgs that can’t be this inclusive, such as those that involve the transitioning out of an executive, or with teams that would not be capable of tying the objectives to an effective team structure. The former will never be seamless when people’s managers leave. The latter indicates a separate problem of lack of team responsibility that needs to be addressed first. But if you are not facing one of these scenarios, here are some things I have learned you may want to incorporate into your next re-org.

  1. Start with making sure the objectives of the company/team/business unit, etc. are clear, and that the executive team is aligned on them.
  2. Inform the teams affected that the new objectives create an opportunity for them to re-organize to be more effective at achieving these objectives.
  3. Empower the teams to propose a new structure that would better allow them to achieve the objectives
  4. When these teams present their proposals, make sure they focus on talking through the constraints that led to their proposal. These are frequently resourcing e.g. not enough Android engineers, cross-department collaboration or lack thereof e.g. no SRE support next quarter, technical or design debt, et al., They can be relationship based or based on location for distributed teams.
  5. Resist the urge to edit the choices directly as a leadership team. Instead, focusing on editing their constraints.

The Analyst Career Path

I’ve written before about analytics teams as a crucial function in today’s technology companies. Technology companies are rapidly hiring analyst roles to pair with their product teams. And while my previous post discussed how to hire analysts and structure their teams within organizations, I haven’t written about how analysts should approach their careers.

Many technology roles, at startups in particular, have an issue with career progression. While established industries have defined career ladders, the path of career advancement is much less clear in many technology roles. Engineering, being the largest and oldest function in technology companies, now has a well defined individual contributor and manager career path all the way up to VP Engineering and CTO. Product Managers know they can progress to manager roles at their companies all the way to VP Product, and if they want to remain and individual contributor, they can still grow by working on more and more complex and strategic products over time. As I’ve talked to many analysts and analytics teams, this progression is not as well defined. I will outline how I think about this progression as someone who has been an analyst and managed analysts.

Option 1: Graduate into Data Science

If someone wants to remain an individual contributor and not manage, at some point the only way to become a better analyst is to graduate into a data science role. Now, there is some confusion with where the line is between analyst and data scientist, and many companies just call all of their analysts data science as a form of title inflation. I define the role of an analyst as someone who uses data to help identify and communicate business opportunities, and drive decisions for teams. This includes targeted analysis driven by others as well as free form analysis driven the analyst. From a process perspective, this includes everything from making recommendations, helping with experimentation, and creating dashboards to help others make decisions. From a tooling perspective, this means everything from writing SQL queries, identifying logging opportunities for product engineering and database design opportunities with data engineering, creating new dashboards and visualizations. An analyst retrieves, analyzes, and recommends, and is judged by not only how good those recommendations are, but how often they are followed.

So, how does data science differ? A data scientist writes code beyond SQL to manipulate data for analysis and potentially for product experience. A data scientist can write an algorithm that powers a personalized experience in the product, or just do more complicated analyses requiring more sophisticated querying using Python, R, etc. Data scientists jump in when analyses are too complicated to be handled by analysts, and also frequently partner or embed with product and engineering teams to change the product. This is more than just a higher-power analyst role though. Data scientists have deep expertise in certain areas, like machine learning, statistical inference, and focus on solving specific, hard problems over longer time horizons.

Option 2: Become an Analytics Manager
If you wish to get on a management track, becoming an analytics manager is the natural path. Since analysts are being hired so frequently, they need managers who can mentor and coordinate learnings between teams. While analysts are best embedded, analytics management bears the important responsibility of solving company-wide analytics issues related to tooling, process, etc.

Option 3: Graduate into Product Management
The third path that analysts can choose to grow their career is migrate into product management. Technically, product managers and analysts are peers in cross-functional teams, but product management has better career pathing that doesn’t require as much technical investment as data science, and product managers tend to have a bit more power in organizations today.

The migration of analysts to product manager is increasingly common as more and more product teams rely on data as the foundation for most decision-making. This has certainly been most true on growth teams and teams that utilize personalization, but I believe all future product teams are data savvy. A significant percentage of product managers at Pinterest started as analysts at the company. This same migration is also true for marketing analysts. They tend to become quantitative marketers over time, or switch to product analytics.


Being successful as an analyst is peculiar is that it almost requires a switch in roles over the time in ways that are not true for design, engineering, and many other roles in technology companies. Fortunately, the analyst has a lot of choices on how to progress within an organization. Hopefully, managers of analysts get better at outlining these different opportunities and help analysts position themselves toward the best ones for them over time.

Thanks to George Xing for reading early drafts of this.

Currently listening to Compro by Skee Mask.

The Autonomy Spectrum: How Much Autonomy Should You Give Your Teams?

Many people are familiar with Dan Pink’s work in Drive that to be the most productive, happy at work, etc., you need to have autonomy, mastery, and purpose. While a lot has been written about how to create purpose inside companies and crafting compelling missions, I would like to spend some time to dig into the details on autonomy. As founders and senior leaders of companies, it can be tough to understand how to give autonomy to employees and also drive toward a singular vision and commit to things that need to happen for other components of the business to work. I have found creating a spectrum of autonomy is helpful, and discussing with senior leadership and your team where you are on that spectrum.

How do you create an autonomy spectrum for your team or company? It’s helpful to start at the extremes. On one end of the spectrum, you would tell your team exactly what to do and how to do it. This is a complete lack of autonomy. On another end of the spectrum, employees have free reign to work on whatever they would like to work on. These projects may have nothing to do with the business, or could be completely aligned. That represents 100% autonomy.

Usually, people agree that neither extreme is particularly helpful. So how do you decide where in between your company or team should be? I like to work backward from 0% autonomy, and see where leaders start to get less comfortable. Let’s take an example recently from my time working with one of the companies I advise. The CEO asked me to come in and help the leadership team think through an expansion of their product. So that is step 1 of the autonomy spectrum, handing a business opportunity to the team from the CEO.

CEO: We have this opportunity for expansion (Business Opportunity)

Working down from that requirement, one executive volunteered to lead this initiative and did a bunch of research on the expansion opportunity. She talked to potential customers of the expansion, validated the pain points of this segment, and discovered a need the company could solve. So, she started settling around a vision of a new product experience leveraging the company’s existing strengths and identifying what new strengths needed to be built.

Executive: The way to attack this opportunity is to solve this specific problem for this segment (Persona and Product Vision)

The team she managed built an MVP attempting to solve this problem, and measured a lot of early usage and did qualitative research on what they had built. At this point, I stepped in to temporarily lead the product team and saw specific issues from the data and the research around the breadth of the solution, the quality of the options we provided, and the value of our solution vs. existing competitors in the market.

VP Product: Solve these four problems with the event discovery app (Problem Scope)

I also saw a team whose operating cadence matched a mature product with clear requirements rather than a fast moving team that uses research and experiments to find product-market fit. So, I created a new process that would structure them better for the problems they were facing and give them more autonomy.

At this point, we worked as a team to organize all of the product teams to work on these problems. As a leadership team, we chose not to identify solutions to these problems, but to put the product teams in charge of determining their own potential solutions. We created a weekly time where our persona would be in the building to provide feedback to our work. We also created a weekly meeting to review research feedback and experiment results.

VP Product: Test ideas with our target persona and run experiments targeting these problems (Process)

This is where we chose to sit on the autonomy spectrum. Prioritize business opportunity, vision, scope, and some process. Create autonomy for the team to define solutions and deeper process on how they want to get there. There are of course other options here. We could have chosen to prioritize ideas the teams invent to solve these problems, even come up with the solutions or how they would validate with research and experiments. There is no clear cut place to draw the line where you start thinking this way.

What is more important is to try to move the line to the left over time. When I advise leadership teams, I tell them that their goal should be to push decision-making as far down the org chart as possible. For leaders that have historically erred on the side of low autonomy, they need to understand how far they can start to push decision-making down without creating chaos. After all, Pink says that mastery is about providing stretch assignments, but not assignments employees cannot possibly be successful in. This process usually starts by moving to one additional layer of autonomy than normal and measuring results. If that is successful, leaders can be confident moving to another additional layer of autonomy. If this is not successful, then leadership needs to think about what type of training it needs to build so that this is possible in the future. I foresee a future at this company where the teams are in charge of prioritizing the problems as well as the solutions because they will be the experts from talking to our persona every week.

This is how Pinterest operated. Product teams created their own missions and problems they wanted to solve, and they were approved by the leadership team. Every six months, teams updated their OKRs to define the new problems they wanted to target and how they would measure success. No company starts there. Especially with startups, founders are used to setting the vision as well as coming up with product ideas. But as companies scale, leaders have to defer decision-making more so they can continue to move fast. The founders cannot be in every meeting nor can they be the experts on every topic.


Autonomy is a difficult topic to grasp inside a company. No CEO or senior leader wants to hand over the reigns to every decision to a team of varying levels of experience and confidence, but trying to make every decision stifles creativity and limits execution pace. Use the autonomy spectrum to build a honest understanding of where the company or teams you manage are and where you would like it to be over time. When you’re confident you are in the right place, communicate this spectrum and why you have made the decisions you’ve made on the level of autonomy you’re comfortable with. Employees will have a much clearer understanding of their role and can build better mastery and purpose as a result.

Product Visionary vs. Product Leader

Many people want to work in product management. One of the most common questions I receive is how to break into product management. It’s a hard question for me to answer, because 1) there is no default path (the same is true for trying to land a business development role), and 2) most of these people really don’t know what they’re asking for. My most common response is, “Are you sure? Product management can kind of suck.” The reason for the dichotomy of people who haven’t done product management finding it so alluring, and people who have done it cautioning people trying to get in is the difference between what I call a product visionary and the product leader.

Product visionaries are who we all hear about in the press. They are the people who come up with brilliant products that go viral or solve real needs in the market that no one else thought of. They appear to be masters of finding product/market fit. I’ve been lucky enough to work with a few of them in my previous jobs and a few in the portfolio at Greylock. These tend to be founder/CEOs, and they generate brilliant insights that create product opportunities others don’t see. Ev Williams is on his third breakout product in Medium after Blogger and Twitter. Ben Rubin created two products that hit product/market fit in two years with Meerkat and now Houseparty.

Anyone working in technology hears these stories, and they think the shortest path to that sort of glory is becoming a product manager. They are excited to get to a role where they can drive the vision of a product, even if it’s only one part of the company. This excitement is exacerbated by the commonly propagated myth that the product manager is the mini-CEO of their product. The reality is that in 90+% of cases, product management is not about being a visionary. It’s about being a leader.

What does a product leader do at a tech company? It’s actually very little of creating a vision and a strategy from scratch. It’s about helping everyone understand what the vision and strategy is. It’s about communicating to the entire team why the company is doing what it is doing. It’s about building a process that helps a team execute on that vision. It’s about when there are competing visions, aligning and motivating the team to focus on one, and getting people to disagree and commit (including sometimes yourself). It’s about looking at data to measure if product changes are having a positive impact on the customer and the company’s growth. It’s about talking to users to understand why they’re doing what they’re doing, and the problems they still face even though your product exists. It’s about mentoring more junior people on your team, across product as well as engineering, design, and analytics. And they don’t have to listen to you, so you have to use influence rather than authority to be successful.

In the scaling phase of a startup, it’s product leadership that drives performance, not vision. Vision is needed early to find product/market fit and plot a course to scale, and then the less that vision wavers over time the better. This vision is usually done by the founder/CEO. The reason founders hire product managers and VPs of Product is not to set vision, but to help execute the vision. Don’t get me wrong; that will sometimes mean coming up with solutions to problems your customers face. If a company is scaling by having the founders solve all the customers’ problems instead of product teams, it will struggle. But much of the time, it will be wrangling the ideas of the individual engineers, designers, and analysts on your team and matching that to an overall vision set by the founder(s). It’s very rarely your ideas you’re executing on as a product leader, and it shouldn’t be.

I also don’t want to make it sounds like I am devaluing visionaries. They are, of course, critical in finding the initial idea(s) that create a growing company and maintaining a vision for that growing company. Having visionaries also becomes more important as you saturate your core market and need to tackle new value propositions to drive new growth opportunities. That is the ideal time for non-founder visionaries to enter a growing company. These are not going to be typical product managers or VP’s though. They are usually ex-founders. The best outcomes for these people entering an organization that is scaling is giving them a team and space to experiment with ideas until they find their own product/market fit, where a business unit is built out around that vision with product leaders to help them scale.

As you’re thinking about your business, think about whether you need a product visionary or a product leader. Most founder/CEOs are already visionaries, so they need a product leader to help execute. Some businesses minded founders need the opposite to be successful. If you’re thinking about product management, think about whether what you really want to be a is product visionary instead of a product leader. In that case, it might be a better idea to start a company than take a role expecting to execute on your vision and instead managing other people’s visions.

Currently listening to Ambivert Tools Vol. 3 by Lone.

How to Set Up and Hire an Analytics Team


Analytics has become a critical role at tech companies. A common question I receive is how to hire analysts and where they fit into an organizational structure. Below I share some tribal knowledge around common team structures, options I think work best and hiring tips that leverage this talent.

Functional vs. Embedded Teams
One of the first questions organizations face when hiring analysts is how they should structure the team. There are two common choices organizations pursue: a functional and an embedded model. The functional model is an analytics team reporting to a Head of Analytics. In the embedded model, every department (sales, marketing, product, customer service et al.) is in charge of solving for its analytics needs, hiring analysts for their teams when needed, and determining to whom they report.

Benefits of the functional model are having a senior seat at the table in major discussions for the company. The advantages this presents is getting analytics its own budget for tools and infrastructure and solving other analyst specific needs that may not be a top priority on any other specific department. The downside of a functional team is how analysts’ time are allocated. In a functional team, an analysts’ time is usually allocated on a project by project level, meaning they usually enter a project once that project is already well defined (whereas analytics could have helped by define the project, had it been involved earlier). And the analyst has not developed any specific expertise for that area. In my experience, analysts get frustrated in this model because they can’t go deep into any one area, and the other departments get frustrated because analysts provide a more cursory benefit than expected.

The embedded model solves a lot of these pain points, but introduces its own. With an embedded model, analysts are hired into one specific team, and therefore can develop expertise for that team very quickly. Teams are happy because they always have a teammate ready to help who understands their problems. While analysts seem to be happier in this model, the downsides are the reverse of the functional model. When there are cross-departmental analytics needs, they usually fall by the wayside. Investment in infrastructure and tooling is usually massively under-invested in, and it’s unclear where budget comes from to solve these needs.

At Apartments.com and at Grubhub, we implemented the embedded model. Marketing took control of analytics infrastructure initially, but we had trouble applying it cross-functionally. The analysts across all teams started meeting regularly to share learnings, but also limitations. When we added the Seamless team into Grubhub, they were used to the functional model. Analyzing the two together created awareness for me of a new approach.

The Hybrid Model
At Grubhub, the value of a dedicated analytics team for infrastructure and tools became clear, but also the value of the embedded analyst. Once I started at Pinterest and dealt with the functional model, we began to work toward a hybrid approach. This is a dedicated analytics team with a Head of Analytics, but with the analysts dedicated to specific areas full-time. So the person reports to a Head of Analytics, but sits with the department they support (in my case, the growth team). As the growth team grew, we created a Growth Analytics Lead who reported to the Head of Analytics and managed other growth analysts dedicated to specific areas, like conversion optimization or on-boarding. This allowed Pinterest to have the analytics seat at the table for budgets and resourcing, but the expertise at the team level to make the most impact. It’s now what I recommend to all teams that are scaling.

Hiring Analysts
If you are scaling a company and need more analytics help, it can be hard to understand who to hire that will actually help your teams. Hiring from analysts at other companies, especially larger ones, proved not to be a great strategy for me. I found during the interview process that most analysts were actually what I call “reporters”, in that they ran well defined reports for people who needed them but didn’t actually analyze anything. If you read analyst job descriptions, they inadvertently screen for these types of people by saying the candidate needs experience with all of these special tools. I can’t tell you how many job requirements that list Omniture (or whatever it’s called this week), Google Analytics, SPSS, Tableau, etc.

Experience with tools is not actually what you care about (though SQL and Excel are a big help). The more tools someone has worked with, the less likely they are to analyze the output of those tools. What you actually want are people who are analytically curious. Our first successful analyst at Grubhub was a new graduate whose cover letter talked about how he tracked his sleep patterns and his diet to find ways to improve his health. He crushed dozens of analysts with multiple years’ experience in our interviews because he was using his brain to analyze results instead of just report. So I now screen for roles where analysis, not reporting, is the unit of value. Many analyst teams at other companies are structured that way, but the majority are not.

You also have to test analytical ability in these interviews. At Grubhub, I gave people a laptop with a bunch of data in Excel and some vague questions to answer from it. The question was based on a real question we gave an analyst intern, who returned it to me saying there was no trend in the data. I ran the analysis myself and found one of the most important correlations for our business (the impact of restaurants per search on the likelihood to order). So I said, you have to be better than our intern to get an offer. It turned out to be an incredible screener. Most people never got far with the data, or their answers were spectacularly wrong. The few good analysts cut right through to a direct way to solve the problem and could explain it easily.

I like this approach because it actually shows the analyst what the job is (and if they’ll like it), and I can walk the candidate through how I would solve the problem so if they did get it wrong, they could learn something from it.

 — 
Analytics team are one of the hardest teams to scale. One of the keys is building a model of a team that will scale with the needs of a company, and the hybrid model is the best model I have found to maximize the important levers of effectiveness (and happiness!). Structure is not all that is important though. Hiring the right candidate is critical, and the market is doing a poor job of preparing people for what emerging companies actually need in an analytics organization. If you can hire correctly and structure correctly though, you will have a competitive advantage over those who do not.

Currently listening to Rezzett by Rezzett.

The Right Way to Involve a Qualitative Research Team

Most teams significantly under-invest in qualitative research. Growth teams especially are all about data, but they think that data can only come from experiments. This can make teams overly reliant on what they can learn from experiments and the quality of the data they have, and under-invest from what they can learn from talking to users. This problem is usually exacerbated by the fact that existing researchers at startups aren’t usually assigned directly to teams or work independently. I’ll talk about some of the problems I’ve seen, and the right way to invest in qualitative research for your growth team.

Learning and Applying from Research
Using the right type and method for your question is key. Of course, qualitative research is one component of the research stack along with quantitative research and market research. There is also different types of qualitative research depending on what you are trying to learn.

I remember when I was at Apartments.com and went to my first focus group, a common type of qualitative research. It was a mess for multiple reasons. The first reason was structure. Finding an apartment is not a large social behavior, so why were we talking with a group of ten strangers at once? As what I later learned usually happened, one or two participants volunteered the majority of the feedback, so while we paid for ten people’s opinions, we really only received two people’s opinions. So, I now only do research with multiple people in the room if it’s a social product, and it’s a group that would use it togethers e.g. friends or co-workers.

The second issue was delivering the feedback to people who weren’t there. I wrote up a long perspective on what the issues were with Apartments.com vs. our competitors. It primarily included product feedback on why we were getting crushed by Craigslist in major cities. I sent it to my VP and received a one sentence reply, “Don’t get ahead of yourself.” What a waste of time, I thought. We do all this research, generate real insights, and no one’s interested.

I’ve now learned that research teams inside companies feel this every day. At Pinterest, we had an amazing research team, but they were originally a functional team, which meant they had to determine their own roadmap of what to research. Depending on the stakeholders you listen to, this can be broad strategic projects like “What is the deal with men?” to specific projects like “Help us test this new search flow already built.” Research can add value at both stages, so the team worked on both.

What I think research found when they worked on the broader strategic issues was similar to my response at Apartments.com. “Cool, but not my roadmap!” say the product managers. Research then gets filed away never to be looked at again. Researchers get very frustrated. To be clear, this is a failure of leadership — not the product teams — if these areas aren’t prioritized. But it is common. On the flipside of working on something already built, success was more variable based on how well the product team defined what they wanted to learn. Frequently, what the product team wanted to learn was that they could ship it, so they selectively listened to feedback to things that indicated they were on the right path.

What I have learned suggests that qualitative research cannot be effective unless 1) its people are dedicated a cross-functional product team and 2) research is involved throughout the entire product development process, from initial research on market to determining a strategy to testing concepts to testing nearly finished products. The value of research accrues the more it is a part of each step in the process.

This approach solves for two main problems. One is that product teams will only pay attention to feedback that is directly related to their current product and on their own timeline. Without being part of the cross-functional team that includes product, engineering, and design, it is hard for research to to be on the same timeline. The second problem this solves is it helps research prevent the rest of the team from locking on assumptions that they may be wrong, so they are focused on the right solution to the problem with research, instead of confirmation bias at the end of a project. The Pinterest team has moved to this model, and for my teams, it made both sides much more successful.

When to Research and When to Experiment
For teams that rely too much on experiments and not enough on research, I tell them two things:

  • Experiments are great for understanding what people do and don’t do. Research helps you understand why they do or do not do those things
  • Experiments don’t help you understand the under-represented groups that might be the most important to learn from e.g. non-users or smaller segments of users

A great way to get started with research as a team is to answer why your experiment didn’t work. Sometimes, the answer is there in the experiment data, but frequently it is not. You have to talk to users to understand why they are doing what they are doing. The best way to do that is to ask them the context of them doing or not doing it.

There is also the middle ground of quantitative research that can be helpful (usually surveys). What I usually like to do is use qualitative research to understand the universe of reasons for something, and use quantitative research if I need to quantify the importance/commonality of those reasons.

Research also helps you isolate users you may not be able to isolate with your usage data. For example, at Grubhub, we were trying to understand how many people used Grubhub regularly for delivery, but not for every order. So, we asked. Then, we called those users to understand why they sometimes don’t use Grubhub, then sent another survey with those reasons to quantify which ones were most important to address. I outline that process more here.

But I Don’t Even Have a Research Team
At Grubhub, we didn’t have a research team for the first couple of years (or even a product team for that matter). So, when we needed to learn things, me, someone on my team, or our sole designer (hello Jack!) would do one of three things: 1) throw flows up on usertesting.com, 2) survey users on our email list, or 3) call users on the phone, and provide them with free food for their time.

You don’t need to be a professional researcher to do this, though they are better at it. You just need to determine what you’re trying to learn and who from. You want to watch people go through that situation if you can. If you can’t, ask them about the last time it happened and what they did and why. You will get better at it the more times you do it. Startups are starting to hire researchers earlier in their development because of the importance of understanding users beyond the data. So, you may be able to justify a full time role here earlier than you thought.

Thanks to Gabe Trionfi for reading early drafts of this and providing his feedback. HeHAH!

Currently listening to Beyond Serious by Bibio.

Starting and Scaling a Growth Team

I get a lot of questions about how to start and scale growth teams. Growth teams need to spend as much time thinking about how to scale themselves as they do on scaling the product. In this post, I’ll outline how I’ve seen growth teams start and evolve, define the ideal end state, and describe some of what you work on at each stage along the way.

The first question to ask is why you need to start a growth team. Why not just have traditional product and marketing teams? It’s a good question, and not well understood. Traditionally, product teams make the product, and marketing is in charge of getting people to try and continue using the product. Marketing can include traditional efforts like events, advertising, and PR, and perhaps some branding efforts and some online marketing. Of note, online marketers and traditional marketers have historically been very different, so depending on the manager, all of the budget and team allocation typically went to either online or traditional — not evenly split.

However, the best startups grow super fast not because of traditional marketing or online marketing, but because they tune the product to drive growth. This is why growth teams matter. Marketing teams typically don’t have access to the product roadmap to make changes to things that will impact SEO, conversion, optimization, or virality, nor the expertise to work with engineers and designers on making these changes. And product managers often prioritize new product features over engineering that will drive more people to the existing value. Growth teams are the connective tissue between product, marketing and engineering.

Starting a Growth Team: Focus On One (Easier) Problem
Goal: Improve one area important to the growth of the business
Team: PM, Designer, Engineer(s), Analyst/Data Scientist (sometime PM or engineer assumes this role as well early on)

It can be overwhelming to consider all the problems you need to solve. When you start a growth team, do not to try to solve all those problems at once. A growth team should start by going deep on one carefully chosen problem:

  • Pick a problem the company is not currently working on (to prevent turf wars)
  • Pick one of the easier — not harder — problems to improve (to build up credibility and drive toward early wins)

Since growth teams are typically treated with a healthy dose of skepticism by other teams when they start, carefully choosing your first area of focus will help maximize the growth team’s chances for success. Let’s say a growth team starts with a hard problem: Trying to increase activation of new users. It takes a month to measure increases in activation rates, and it has one of the smaller sample sizes in the traditional growth area. You run an experiment. It doesn’t work. You run a second, and another month goes by, and it still may or may not work. Now two months have gone by without a growth team win. People on the growth team are questioning if this is the right team to be on, and other teams in the company begin to question your purpose. (Hat tip to Andy Johns from whom I am blatantly stealing this example).

There are common growth problems where you can likely show quick progress. One is conversion optimization. Another is returning users via email and notifications. Still another is improving referrals or virality. These areas allow you to run multiple experiments quickly because the metrics will move immediately (they don’t take time to measure like activation), and because they have high sample sizes (all new users or all existing users opted in to communication).

Last note: When starting a growth team, it’s important that they sit together and not with their functional team members. The collaboration between the different skill sets and short feedback loops is how the magic happens.

Growing a Growth Team: Own the Growth Funnel (And Some Real Estate)
Goal: Improve metrics across the growth funnel
Team: PMs, Designers, Engineers, Data Scentists/Analysts

As a growth team grows, the team can start looking at multiple problems. These problems are typically across the AARRR framework by Dave McClure, (however revenue is frequently a separate team in an ad supported business.) The growth team separates into sub-teams who have their own meetings, and entire growth team meetings become less frequent. PMs, designers, and analysts may work on multiple teams. At this point, the growth team assumes ownership of certain areas of the product, including (but not limited to):

  • The logged out experience, which can involve both SEO and conversion
  • All emails and notifications (sometimes coordinating with marketing)
  • The onboarding flow, sharing flows, et al.

Owning these areas creates easy swim lanes between teams and prevents turf wars with the core product team. Growth leaders have to determine how to allocate people to work on the most impactful problems across the growth stack, and if a sub-team does not have enough support to make an impact, the growth leader should consider whether to support that problem at all. Growth teams also have to own goal setting, which means understanding historical performance and setting absolute goals.

At this point, teams start to work on both improvements in infrastructure and experiments. For example, at Pinterest, I was managing both the acquisition and retention teams. The retention team was struggling to grow because of an unwieldy email infrastructure. The infrastructure was built to support triggered, social notifications, but our strategy had evolved to more personalized, content-based recommendations. These jobs were taking days to run and would send automatically when complete, even though someone might have received a social notification five minutes before. So we spent nine months rebuilding it to allow us to scale more effectively. The retention team saw a step change in its performance and actually hit its Q1 goal in the middle of January after it was complete.

Evolving a Growth Team: A Special Forces Team Operating Across Borders
Goal: Reduce friction across the product that prevents people from connecting to product’s core value
Team: PMs, Designers, Engineers, Data Scentists/Analysts, Researchers, Marketers, Operations

As the growth team continues to grow, it involves more stakeholders and expands its scope. Instead of having clear areas of focus separate from the core product, the team shifts to analyzing the entire product and trying to figure out the biggest obstacles that are preventing the company from growing faster. At this point, the growth team has built up enough credibility that it doesn’t fight turf wars — it is laser focused on finding and solving the biggest problems that prevent people from connecting to the current value of the product.

Usually, the problems at this stage are deep inside product features and go beyond improving SEO, signup, viral and on-boarding flows, and sending the right emails and notifications (though growth teams still work on those too). The growth team is now identifying the friction that prevents people connecting to the core value elsewhere in the product. A lot of this work is simplifying product flows or building country-specific optimizations for slower growth countries. Growth teams can also become service organizations to marketing initiatives around this time, and I’ll talk about that in another post.

At Pinterest, we migrated to this phase rather recently. As we started to look at friction in the entire product, we saw in qualitative research that new users were getting bombarded with so many concepts it confused them: what a Pin is, where they come from, how to add your own Pins, what a board is, group boards, profiles, etc. The growth team made a list of the core things new people needed to know to get excited and start to build an understanding of the service. Then, we started experiments to remove everything else from the new user experience and only introduce it back once the core concepts were clearly learned. This helped improve activation rates.

What I described above is an amalgam of what I’ve seen in the market that works best. Of course, you should always tweak your approach based on the talent of your team and your company’s culture. I would love to hear what you think of this approach, and any additional insights you think would be useful to share with other growth leaders.

This posted originally appeared on the Greylock blog.

Currently listening to 28 by Aoki Takamasa + Tujiko Noriko.

Three Mistakes Integrating Growth and Design Teams and How to Address Them

While I encourage growth teams to start with a dedicated designer, that doesn’t usually happen. Usually, growth teams scale with just engineering and product, and as they scale to a certain size start to eventually “earn” a dedicated designer from the organization. This news makes growth teams extremely happy for a short period of time, but pretty quickly starts to create problems and culture clash. I’ll talk about what happens, and some ways I’ve found to solve these issues. Note: You can replace growth with marketing, and just about the same thing happens in these scenarios.


When you hear you’re getting a dedicated designer, you’re happier than Jim Carrey was back when he had a career.

Problem #1: Team Control AKA The Two Scenarios
One of two scenarios usually emerges when designers join a growth team. In the first scenario, the engineers or PM or marketing person starts with, “I’m so glad you’re here. I need this done and this done and this done for an experiment…tomorrow. No, we don’t need research to understand the problem. Don’t worry. I’ll only ship it if the metrics increase.” The designer realizes the growth team didn’t want a designer. They wanted a pixel monkey to just do what they say, not ever use their brain.


I got 99 problems, but a designer using their brain ain’t one.

The other scenario is just as bad. In this scenario, I call it the designers “moving in”. I liken this to the scenario when a significant other moves into your place and brings way more things than you own to move in. “Don’t worry, I got better furniture and books, and, oh, we’re going to have to get rid of those curtains.”

The design version of this is something like “I’m going to completely rethink all our strategies. I need to spend three months just doing research and then at least double that for design concepts. That home page really needs to change though.” Engineering thinks “this isn’t what we need. These people don’t even know what they’re talking about.” Engineering gets kicked out of the process of figuring out what to work on, and since the design process has no deadlines on it, engineering begins to have nothing to work on.


I tried to find a picture of people dressed in all black moving in, but the best I could do was chambray.

In both scenarios, instead of designers, engineers, product managers, and marketers trying to unify to form one team that leverages all of their strengths, one team tries to dominate the direction. What we did at Pinterest to try to solve this problem was design a process where the teams, specifically design and engineering, are jointly responsible for problem definition and solutions. Here’s what it looks like:

Project Kickoff:
Attendees: PM (leader), engineering lead, design lead, engineer and designer that will be working on the project

Goals:

  • Define the problem we’re trying to solve
  • Any existing ideas on how we might solve it
  • Any previous attempts to solve the problem
  • What metrics we need to inform potential solutions
  • How will we tell if we’re successful

Output:

  • Notes emailed to broader sub-team
  • Slack channel created just for the project, which includes notes and all future communication for the project

Project Brainstorming:
Attendees: engineer and designer on the project, PM (optional so as to prevent their calendar from being a bottleneck)

Goals:

  • Produce multiple potential solutions to the defined problem
  • Prepare those for feedback from attendees of kickoff
  • Cover additional measurement needed for directions that have been chosen e.g. how many people click top right login button

Brainstorm Review:
Attendees: PM, engineering lead, design lead, engineer and designer working on the project (leaders)

Goals:

  • Feedback on concepts from brainstorm
  • Choose one or two directions for experiment

Experiment Launch:
Attendees: engineer and designer working on the project (leaders), PM

Output:

  • Experiment doc that team agrees reflects what we’re testing and why
  • QA and approval over Slack from every team member that experiment is ready to launch to 1%
  • Note: Ramp ups of experiment communicated in Slack channel

Experiment Review:
Attendees: PM (facilitator), engineering lead, design lead, engineer and designer that worked on the project (leaders)

Goals:

  • Determine if you gathered data on key questions you needed to answer for the experiment
  • If yes, what does the data say?
  • If not, how do we get the data?

Output:

  • Ship/kill/iterate decision
  • Emailed notes to the entire team of what happened and why
  • Updated experiment doc on what happened
  • Updated Slack channel
  • Designer brings ship experiment candidates to design manager/lead for any feedback before shipping
    • Problem #2: Design Best Practices
      Designers bring with them a history of best practices from working on other design teams and schooling/training. It can be a culture shock for a designer joining a growth team and seeing little of these being followed. Designers will usually respond to this environment by trying to educate on many things the growth team is managing are designed “wrong” and how they need to be changed. These recommendations based on best practices are usually rejected by engineers on the team as it contradicts what they’ve seen in the data.


      A growth engineer after he hears about a marketing or design best practice.

      Design best practices were created with good purpose, but they quickly become inferior to AB testing in an environment where a direction can be put in front of users and its value quantitatively determined in days or weeks. This doesn’t mean you should AB test every design decision, but it does mean a designer can’t put their foot down by saying something is a best practice. In growth, the users being tested on determine what’s best.

      Another issue with design best practices being applied to growth is that design best practices are usually suited entirely for building user value only. The growth team in many cases needs to trade off user value and business value, or short term user value vs. long term user value. For example, someone can come to my site, and I can offer a great experience. The user then leaves and never comes back. Growth might decide to let someone get a preview of that experience, then ask the user to sign up so they can personalize and deliver more value. The result is that some people will not sign up (creating a worse user experience) and some will sign up, allowing the site to create a better user experience that day and over the long term. This accrues value for the business as well.

      Designers usually feel threatened by this environment initially. They see testing as a threat to their expertise. What you have to do is teach them to use it as a tool to get closer to user feedback at scale and be more efficient with their time. Most design work is wasted because it is spent on a direction that later proves to be flawed. With AB testing, a design can get quick feedback on an idea, validate the direction, then spend the time making it amazing once they know it’s worth that effort.

      That said, while this user feedback at scale is good at showing what users do, it’s not great at explaining why. So, along with spending time getting designers comfortable with testing, growth teams need to start doing more regular qualitative research as well. Designers will usually volunteer to do it themselves if there isn’t the right person to staff for it. Some PM’s are comfortable doing it as well. Engineers and engineering managers can be resistant to spending time watching the sessions, so the first couple you schedule have to be on critical areas you feel there is some understanding the team is missing by only looking at the metrics.

      Once designers get a little more seasoned on growth, you should also work with them to create the new best practices for growth to make the design and engineering process move faster. They will look very different than what designers recommended at the beginning, and be backed by data. Along with this process, I think it’s important to start creating user experience goals for your growth team at this time. These will be components that may not affect the metrics, but ensure a quality and consistent user experience. At Pinterest, we made compliance to design spec a requirement to ship after an experiment is validated, and said our top five flows had to be completely audited for quality user experience. This is a worthy compromise with design to show you actually care about the users and not just the business, a common complaint.

      Problem #3: Growth Onboarding
      Sometimes, you want to make sure you just nail the basics. Most growth designers are joining a new team in a discipline they’ve never worked on before, yet they don’t receive any onboarding as to what growth is and why it’s different. This is also an issue with other new growth team members. It’s like being dropped into a foreign country, not knowing the language, having no one help you, and being expected to contribute to the culture. Like France (okay, I’ve actually been to France, and the people were surprisingly welcoming even though they’re not known for being so). What happens is designers get frustrated and churn from the team.

      It’s critical that every new team member, especially designers, go through a growth team onboarding process. The first thing you do is state the purpose for your growth team, why it’s different from other product teams, and why it’s important. At Pinterest, I would say that while other product teams create value or improve the value of the product, growth teams focus on connecting people to the current value of the product and reducing the friction that prevents that connection. This is important because it’s not enough to build a good product. If no one knows about it, it will die. If it’s too hard to uncover its value, it will also die.

      What we did at Pinterest was create a history of major growth projects that were successful, walked through them, and explained why they were successful. That led into some of our team principles. We also spent a lot of time educating new people on the metrics we use and why. You can’t be expected to design winning experiences if you don’t understand the major criterion for success we’ll be using.

      After onboarding, instead of starting designers on large, complicated projects, it’s important to start them on a well scoped, smaller project, preferably with an experienced PM and engineer, hopefully patient ones. These should still be projects that are worth doing, but not large projects. We had a designer start on growth at Pinterest, and we immediately put her on one of the most strategic, long term investments for the team. While she did great work there, after a few months, she did a smaller side project redesigning our mobile web home page. The conversion rate increased, and we shipped it. She said, “Guys, this is the first time my design increased the metrics!” She was beaming. You want to get new growth designers to that moment as soon as possible.

      Lastly, once onboarded, you want designers contributing growth ideas as soon as possible. I like the idea of forcing people to bring one new idea to the team per week. I believe this is a muscle that needs to be developed via practice. New entrants to the team (design or otherwise) will typically propose bad ideas for a while. That’s okay. The trick is to provide a framework for generating ideas that makes new team members think about elements that typically make for good growth ideas, give feedback on the ideas submitted, and have the ideas be submitted as metrics wins or user experience wins. The Pinterest template focused on :

      • How many people would see this experience if it were built? This is usually by far the most important criterion for a successful growth idea. Any experience has to be seen by a lot of people to have a big possible impact on the business.
      • Has the company tried something in this area before? If so what were the results? This helps us make sure we use any previous learnings to avoid making the same mistakes. Just because we tried something before doesn’t mean it’s a bad idea though as growth environments change frequently.
      • How much effort is required for this idea? On growth, we also try to do the least amount of work to validate an idea is worth it.

      There is no reason designers can’t be key contributors to a growth team, but expecting it to happen automatically is usually a recipe for failure. I hope some of these tips can help you create a thriving cross-functional growth team with all the right disciplines involved. If you’ve have any other issues integrating design into your team, I’d love to hear about them in the comments.

      Currently listening to Sorry I Make You Lush by Wagon Christ.