Product Reviews

I’ve been fortunate to work with exceptional product leaders at Google and Facebook. This distills some of the best practices I’ve learned around product reviews. 

Purpose: These thoughts are targeted at product managers (PMs), product leaders at larger companies, and CEOs of startups. This document outlines 1/the goals of a product review, 2/common mistakes, and 3/the logistics before, during, and after a review.

I use CEO or founder as a proxy for “decision maker.” For product-oriented companies under 500 people they are likely the same. At larger sizes, responsibilities often recurse down to a VP as the Decision Maker.

0. Why do product reviews?
Product reviews are a process through which a company can scale product development, and are useful for both teams and leaders. Product reviews allow teams to have ownership over product decisions and a clear, predictable way to move forward through strategic and operational roadblocks. They are a useful tool when a CEO can no longer be involved in parallel work streams in real-time. In my experience, this will happen around 20 employees — when it’s hard for one person to be both the full time PM for the product and the CEO at once — and are a staple process as a company grows from 20 to 20,000+ employees.

1. The Goal of a Product Review

There are three flavors of reviews:

• I. Input/Discussion — The goal is to get input early in the ideation/development of a product. The questions on which you want input and the type of input you want should be clear. It is better to get the CEO/founder to weigh in rather than spend 3 months on something and then realize it is totally out of whack with her vision, that some other team is already doing it, that the company tried something similar before and the founder has context, etc. It’s good to get in front of her as soon as you have a good idea of what the team is trying to accomplish and get her input.
• II. Unblocking a decision – The goal is to make a strategic decision. If multiple paths forward are plausible and reasonable people disagree about the right path, go to the CEO to make a call. In most companies, the CEO has more context than anyone else about what’s going on, more data intuition than anyone else at the company because of historically accumulated knowledge, and is phenomenal at asking the right questions. The CEO is the ultimate arbiter between teams and can unblock while optimizing for the company. Over time teams can build ways to formalize these decisions, but early on someone needs to make a call.
• III. CEO priority – The goal is to build and ship a product with the CEO deeply involved in the process. This is a rare third form of review which is a bit more like a team scrum, with the CEO co-PMing with the product manager. This happens at every company. These are projects that are pre-launch, new products, or initiatives that are critical but struggling. The CEO will get involved because she has strong intuitions about what a team needs to build in a strategically critical area. She wants to understand what’s happening the same way any PM member would and help guide the team to build the right product. It’s not uncommon to go in every week to the CEO as the product gets built. After launch, the team should be on its own trajectory, and the team falls into the normal iteration cycle + product review cadence with review types I-II.

2. Common Mistakes

• Reviews as updates – The most common mistake when asking for review time is to just offer an update. This is often conflated with a discussion style review (type I). Type I reviews have clear questions where the team needs input and will require the CEO to actively engage. Updates are information flowing from the team to the CEO and do not require active engagement. Teams should rarely burn a CEO’s time to do an update. Ten people in a room for thirty minutes = five hours wasted. Send updates over email and if the CEO has questions, get in a room to talk through it. This serves the dual purpose of forcing documentation that can be shared broadly.
• Reviews as exposure/reward – Do not use reviews as an opportunity to give people exposure or as a reward. Teams should be 100% focused on winning and a review has a clear goal to unblock the team on its path to winning. For example, this may mean that a more senior product manager actually presents for the room or the most senior designer represents the entire design team’s work. Find other ways to give exposure and reward people.
• Surprises during/after the review – everyone who will be impacted by the decision made should have context that this review is happening and has had a chance to weigh in with their perspective. This requires understanding not just the decision to be made but the implications/consequences of the decision. Not everyone has to agree with how to make the trade-off, but it is a big failure on the product manager if someone in the room is surprised with the content of the review or if someone who should have been involved in the decision is surprised when they hear the outcome.

3. Review Logistics
A. Before the review

• An efficient approach is to have every team in the company have a standing block of time to go to the CEO during the week. This is efficient because a team knows they can go in every week if needed. If not, they give up the time a few days in advance so other teams who need extra time can use it.
• Create a document or a deck with mocks, and send it the morning of the day before the review. So if you’re going in on Thursday afternoon, the CEO gets the document to read on Wednesday morning. The questions you want to discuss are called out clearly. The day before the review or the morning of the review, the CEO may respond with questions she’d like to talk through in the review. This gives the team time to get data or converge as a team on an opinion before the meeting.
• Try to create standard formats/templates that are shared across teams. Uniformity across conversations makes it much easier for people to come up to speed on the content.
• Anyone who has depth of information/context should be in the room – product managers, designers, analysts, marketing, user research, engineering, sales, marketing, legal, etc. The attendees will change depending on the decision to be made and should be determined and communicated in advance of the review so everyone can prepare for a successful review. Don’t have people who want to “stay in the loop” or “hear about it firsthand” in the room. If you’re not actively contributing to the discussion, you shouldn’t be in the room.
• If you anticipate the meeting will take 30 min, block 45 min and shoot to be done at the 30 min mark. This also gives some buffer and prevents a domino effect since a CEO’s day is often back-to-back.

B. During the review

• Everyone should have read the document beforehand and the meeting should start with a discussion on the open questions. Most of the time in a good review should be on productive discussion.
• If you must present a deck, shoot for six slides per thirty minutes. Do not schedule content to fill the time and move through the presentation 3x faster than normal. If the CEO has a question or needs clarification, she will ask. The most productive use of time is to discuss and decide.
• A great CEO focuses almost entirely on high level questions for most things that are brought to her. She wants to understand the goal and if the team is solving the right problem. This means she will look for a clear articulation of the problem you’re solving, data to back up that this is a real problem, a clear articulation of how to measure success, etc. Historical trust with her helps.
• If there are known truisms inside the company, she will call them out if you violate them. For example, in many consumer apps, more information density means every metric will go up. So in areas where we want users to take some action and she sees a bunch of whitespace, that’s going to get called out. Another example is that certain design patterns may have become core to the product and often it’s better for the entire system to maintain these. So if you add too much complexity to the system by creating multiple ways to do something, a good CEO will ask why you are introducing new ways to solve this problem instead of using existing design patterns.
• A good CEO avoids micromanaging (other than CEO-pri things). Likely if you’re getting micromanaged, it’s because the team is broken and the CEO will work through her directs to fix the team and perhaps move people between teams. After seeing this a couple of times, you realize that good CEOs aren’t interested in design specifics on most things because it’s not scalable. Hiring great people, having good processes, and giving people autonomy scales far better. A good CEO moves people and risks thrash to get the best people on the most important problems. She doesn’t care about who owns what, because it’s all one company, and all that matters is the company’s success. This is at odds with how many people think about their jobs and careers, and takes some adjustment.
• She isn’t afraid to call out organizational issues. For example, if a design solution seems suboptimal, she may probe and then ask if we’re doing this because it’s too painful to work with some other team or if this is really the best way to do it. Good executives will back-channel information to their CEO as well so she has additional context and can make moves as necessary. It’s awesome to know a great CEO won’t hold back on organizational/political issues and wants the company to globally do the right thing.
• She shares context across groups by calling out why she thinks something is the case, not just what she thinks is true. This is useful because she sees information across teams in a way no one else does.
• Good CEOs are extremely logical and operates from first principles. This means that if you go in again in a week, she will give you the same answers as last week because the answers are derived. If there is no logical reason for something (she just has a strong intuition), she will make that explicit. This is unlike a lot of leaders who conflate what they believe emotionally with what they know rationally. It makes great CEOs very predictable, which allows the company to scale.
• Other times, she will have very strong intuition about key pieces of product strategy. This is informed by historical information/experience that something will or won’t work. These are almost always strategic calls – for example, what the default privacy model of a social product should be – not something at the pixel level. In these situations it’s best to take the feedback and decide if you agree/disagree. Ultimately code and data win. If the team disagrees with her intuition, they should find a way to test the CEO’s intuition and the team’s intuition and validate (maybe a user study, maybe a small test, maybe confirming the historical data she has in mind).
• If the CEO asks for something, you are on the hook to deliver a follow up, even if all it does is prove her wrong.
• Great leaders are comfortable with silence. They will sit silently to think through something before articulating what they believe and why. This can feel like an eternity and be very uncomfortable for new people. A good CEO will take this time because she’s trying to think through why she believes something so she can articulate the principles, not just the intuition.
• CEOs have to be good at demonstrating positivity when things are going well. Many CEOs’ and founders’ natural inclination is to be critical and point out what can be done better (e.g. the countless stories about Elon Musk or Steve Jobs). This leaves people feeling like they are constantly failing because they receive little positive reinforcement and makes it challenging to calibrate when things are truly not going well. If the team is not doing the right things, great CEOs do not hold back. She makes it clear what she thinks this is a mistake and why. She does not do this in a personal way and focuses on the outcome.
• Curveballs happen (this should happen in < 10% of reviews) when the team is solving the wrong problem. This could be because of information asymmetry or it could be because of a communication breakdown. For example, the team thought the CEO wanted us to solve X but she was simply using X as an example and wanted us to think about the generalized version of X.

C. After the review:

• CEOs rely on senior people to interpret for the team. After every review, the team should debrief without the CEO, and the senior people in the room should interpret what happened. For example, the PM and designer on the team incorrectly thought they were being asked to change the design of X (interpretation #1). The senior person in the debrief clarified that the decision maker was really saying that the team had introduced a new way to perform X in the app, when there was already another way to do X. While the new approach maybe good, two different ways to do the same action will erode the system over time, and the decision maker was calling out this principle. So the team should go think hard about whether a new way to do X is good, or if we should just do X the same way it’s done in the other part of the app (interpretation #2). Interpretation #1 and #2 are very different and would send the team in very different directions.
• CEOs rely on their executives to make people shifts. For example, maybe an engineer is too junior for the problem they’re being asked to solve. The CEO shouldn’t throw this person under the bus in a meeting, but will work through the leadership for the group who is in the review, get the engineer additional support and if it doesn’t get the team where it needs to be, work through her management chain to get someone more experienced on the team. All of this will happen behind the scenes. A good CEO helps people save face in public because good junior people who trust their management chain eventually become great, loyal senior people.
• CEOs rely most often on PMs to collect and synthesize notes from everyone and circulate them to everyone impacted by the discussion — the team, engineers, partner teams, marketing, sales, etc. Writing things down removes ambiguity, keeps everyone informed, and allows people who weren’t in the room to ask for clarification. These notes should be sent out the day of the review so the context is fresh in everyone’s heads and any ambiguity is resolved quickly.
• CEOs rely on product managers to schedule more time for subsequent follow ups. CEOs tend to have great memories because they live and breathe their product. No PM wants to get an email four weeks later asking what happened with the open questions from the review last month. That reflects extremely poorly on the PM (and the team).

Many of these principles will scale to other, non-product types of reviews, but some of these observations may not carry to all cultures or types of companies.

Thanks to Dan Siroker, Yin Yin Wu, Andrew Bosworth, Zohar Yardeni, Alex Himel, Pratiti Raychoudry, and Hema Budaraju for their feedback.