Amazon Product Management Interview Question: How Would You Design an A/B Test for Amazon's Voice Shopping Feature?

Step 1: Ask Clarifying Questions

Before jumping into the experiment design, I want to make sure I understand the scope and the specific business question.

Q: What does “voice shopping” cover here? Is this a reorder-only flow, or full discovery and checkout via voice?

Let us assume the new feature spans the full flow: voice search, conversational product discovery, voice reorder, voice checkout, and personalized purchase reminders. This is broader than the legacy Alexa shopping flow that mostly handled reorders.

Q: Which surface are we testing on? Alexa-enabled smart speakers, Echo Show, the mobile app’s voice button, or all of them?

Let us focus on Alexa-enabled devices (Echo, Echo Dot, Echo Show), since that is where voice is the primary input modality and where habit formation matters most. Mobile voice can be a follow-up phase.

Q: Are we measuring purchase frequency only, or do we care about revenue per user as well?

The question specifies purchase frequency, so let us treat that as the primary outcome. Revenue per user is a secondary metric. We want to test the habit hypothesis specifically: does voice make people shop more often?

Q: Which user segment should we target?

Let us focus on existing Amazon customers who own Alexa-enabled devices and have made at least two purchases in the last 90 days. This gives us a clean baseline of habitual shoppers to measure incremental lift against.

Q: Do we have prior data on voice shopping adoption inside Amazon?

Yes. Voice shopping at Amazon has historically had low penetration despite many years of Alexa availability. The new feature is meant to change that. So we are testing whether a redesigned flow can convert a previously dormant capability into a real shopping habit.