He pulled the test down but he couldn’t ignore the signal.

Mulling it over, the data showed that once a user started a trial, they stayed, so the subscription product itself wasn’t the issue — it was exceptional. The friction must be psychological - what could he do?

Mike leads product at Abide, a Christian meditation app with 6M+ downloads. I worked with him for five years on Citibank’s Online Banking team, so I’ve seen this pattern: when something isn’t working, he doesn’t abandon it, he reframes it, looking for the deeper insight that can unlock real user value.

👋 Hey, I’m Alex. I write Shipping on Fridays to explore the craft of how great products get built and what we can learn from the people behind them. I publish 1–2x a month on topics ranging from design sprints to how my sailing team a/b tests. This week I caught up with Mike to share the story of how he and his team tested their way to a breakthrough.

Subscribe now

Setting the Scene: Playing on Hard Mode

Alex: First off, thanks for joining me, Mike. I’d love to hear a bit about Abide and the challenges you’re facing.

Mike: Yeah, so Abide is part of a nonprofit called Guidepost, and we’re a Christian meditation app. We’ve got about 8 billion elapsed minutes per year and a really passionate, loyal user base. But our big focus right now is new user growth—and honestly, we’re playing on hard mode.

Hard mode?

Since we launched, we’ve been the number one Christian meditation app. But since then, there’s been about 3 to 5 big players—all VC-backed with hundreds of millions of dollars. They’re outspending us on ads, and with AI making it easier than ever to build apps, the barrier to entry is gone. As we’re a nonprofit, we have to understand the game being played and how we can win at that game. So instead of buying growth, we have to earn it, by extracting more value from every install.

That sounds like a massive challenge. Where did you start?

We knew our Premium experience was already exceptional, top percentile in trial-to-paid conversion and retention. Once people experienced it, 7 out of 10 stayed.

So the question isn’t how do we improve Premium? It is: how do we get that value into users’ hands, without friction and at the right moment?

The Problem: A Hesitant Audience

A lot of our users are hesitant to put down a credit card. Half of our user base is over 60 years old. Some don’t even understand what a free trial is. Others have this aversion to paying for a faith-based service in the first place.

So we had this hypothesis: what if we removed the credit card requirement entirely? Give them 7 days of premium for free, no payment required upfront. On day 7, they’d have to subscribe to continue.

Paymentless free trials. That makes sense given your audience.

Exactly. And I was confident it would work. I mean, if our free-to-paid is that strong, even if half as many people convert, that’s still solid, right? Especially at onboarding—that’s where the majority of our revenue comes from.

So we ran the test in onboarding. A 50-50 test.

Bold move.

I felt so good about it. I thought we were about to juice our numbers. And then… it was basically flat. Like, 0.01% higher.

Oh man.

Test #1: The Onboarding Flop

Yeah, I was humbled. At first I’m thinking, “Wait, maybe my product’s not that great?” But then we realized the issue: we were asking people to commit before they had felt the value. People hadn’t experienced the product yet. They didn’t know what they were getting into.

So you took it down?

We took the test down and went back to control. But I didn’t give up on the concept. I still believed in it. The data told me that once people get into the product, it’s sticky. They just needed to feel that first.

After the onboarding test, we realized we had introduced unintended complexity by bypassing our standard account creation flow, which likely hurt performance. That insight led us to re-architect the experience so we could deploy it intentionally at any moment in the user journey.

er

Test #2: Playlists as the Gateway

So what did you do next?

We had a new feature we wanted to build: playlists. This is an audio app—how do you not have playlists, right? Our users wanted to fall asleep to curated content that would play all night without interruption. It was a highly requested feature.

And we realized: this was something users clearly wanted…and it was meaningful enough to make Premium feel distinct.

Wait, so you made playlists a premium feature?

Sort of. For free users, when they hit the playlist feature, instead of showing them a traditional paywall, we let them unlock it by accepting a paymentless free trial.

So the second test was basically putting paymentless trials in front of a high-intent feature?

Exactly. These were users who had already experienced the app and were actively trying to do something specific. They had felt the value.

Now the trial wasn’t abstract — it was access to something they genuinely wanted. And you know what? It worked. Not a massive revenue driver because it’s a smaller cohort, but for those users, the conversion rate was strong. About 3 out of every 7 users who saw it converted.

That’s solid! But I’m guessing playlists alone wasn’t enough to move the business?

Right. It was impactful, but it was only a sliver of our acquisition and upgrades. We still needed subscriptions. Not everyone was making it to playlists, and we needed something with higher volume.

Test #3: Content Gating + The 4-Hour Window

So that brings us to the third test?

Yeah, this past September we made a major shift. We realized we were giving away so much that users couldn’t clearly see what made Premium special. So we shifted from having 90% of content free to making only about 10% free for new users.

Whoa. That’s a huge philosophical change for the business, especially for a nonprofit.

Very much. But we looked at apps like Calm and Headspace. Try finding 2-3 pieces of free content in your first 10 minutes. Everything is locked for a reason. We were in a season where we couldn’t just pump downloads to the App Store. We had to make the Premium experience unmistakable.

How hard was it to get buy-in from the nonprofit leaders?

The key was showing them the risk was minimal. We were getting 80-85% of our revenue in onboarding before this gated experience even kicked in. So we were only impacting the next 10% of users who converted in the first week, and the 5% long tail that happened over 90 days.

We started at 10% of new users, saw good traction, rolled it out to 25%, then 50%. Subscriptions went up 8-10% versus control. Not because we restricted access, but because we clarified the value. Free listening dropped ~30%, primarily among low-intent users — while revenue per new user increased materially.

So you had to decide: what’s a free user worth to us?

Exactly. And my CFO made it easy. He said, “How much are they worth to us, Mike?” When we did the math, it was clear. Turn it up.

So now all iOS users who skip upgrading on onboarding clearly see what’s included in Premium and what’s free.

And that’s when you brought back paymentless trials?

Two weeks ago, we shipped the third iteration. Now, 4 hours after someone’s last play, they get a push notification or in-app message saying: “Congrats, you’ve unlocked premium. No credit card required.”

It’s an unsolicited surprise and delight moment. But they’ve experienced the app, they’ve seen what’s Premium, and now when we offer frictionless access, it feels meaningful.

And how’s it performing?

Two weeks in, we’re up 6% in subscriptions versus the control at 25% of users. Early days, but it’s showing real signal.

What Made the Difference

So looking back, what made the third test work when the first two didn’t?

The offer didn’t change. The experience before the offer did. The first test failed because we asked too early—before people knew what they were getting. The second test worked because we asked at a high-intent moment, but the volume was too small. The third test combined the best of both: high volume and the right context.

By the time people get that notification, they’ve already:

• Skipped onboarding and chosen not to pay

• Created an account (which is now required)

• Experienced the app

• Hit the content limits and felt the friction

Now when we offer them a paymentless premium, it’s not abstract. They know what they’re unlocking.

So it’s not just about the feature—it’s about the timing and the user’s state of mind?

Exactly. It’s about respecting where they are in their journey. In onboarding, they don’t know us yet. At playlists, they’re invested but it’s a narrow use case. But 4 hours after they’ve been using the app and bumping into gates? That’s the sweet spot.

Lessons Learned: Don’t Give Up After One Test

Looking back on Mike’s journey, a few things stand out:

1. One flat test doesn’t mean the idea is wrong. The first test was flat not because paymentless trials were bad, but because the timing was wrong. Paymentless trials aren’t pricing mechanics, they’re trust mechanics. After value, they feel like a reward. Mike didn’t abandon it, he iterated to make it feel like a reward rather than an offer.

2. Volume matters, but so does context. The playlist test worked with the right users, but at a small scale. The third test proved the pattern:

   1. Right user + wrong moment = flat test.

   2. Right moment + wrong scale = niche win.

   3. Right moment + scale = business impact.

3. Build on what you learn. Each test taught Mike something new. Test 1: too early, trust wasn’t there. Test 2: value first, high intent works. Test 3: combine volume with context and remove friction.

4. Sometimes you need to change the environment . Content gating created the conditions for paymentless trials to succeed. Without that friction and clarifying what was truly Premium, made frictionless access more meaningful and appealing

Too often in product, we run a test, see it’s flat, and move on. Mike’s story is a reminder that iteration isn’t just about tweaking variables—it’s about fundamentally rethinking where and when you’re asking the question.

Read more Shipping on Fridays

It was a short chapter at Personio, but in that time I learned a lot about myself, product craft and culture. Here are 3 short lessons I’m carrying forward.

👋 Hey, I’m Alex. I write Shipping on Fridays to explore the craft of how great products get built and what we can learn from the people behind them. I publish 1–2x a month, and every post is meant to be fun, useful, and a little unexpected; from design sprints to sailing races to holiday chaos. If you’re into learning, product or design, this is for you.

Subscribe now

Lesson 1: Ask the Uncomfortable Questions

When I started on the Activation team, morale was low. The Engineering Manager had just left with no backfill planned. An engineer was stuck on a large project that had been labeled “engineering-led,” which in practice meant all the product and design decisions got punted.

In our 1:1s, his frustration poured out. He was demotivated, spinning cycles and unclear on what success even looked like.

What I wanted to do was to avoid conflict, wave a magic product wand of buzzwords and frameworks to make the symptoms go away. I had enough to keep myself busy.

But I’d been here before and my instincts were screaming at me to not make the same mistake. At a previous startup, low morale undermined the roadmap and led to attrition (including my own). I knew that avoiding the hard conversations wouldn’t make the problems disappear, it would probably make them worse.

So I leaned into what felt uncomfortable: for every frustration I heard, I tried to dig deeper and understand the root cause.

I also knew I shouldn’t try to be a solo-hero. Together with another team member, we ran team retrospectives to capture pain points and brainstorm possible solutions. We documented all of this on a Trello board and assigned people to next steps. As a team we also agreed to treat our ways of working as experiments. Try a new team process, measure if it felt better, keep it or scrap it and try something new. Above all, be open to change, new ideas from teammates and feedback.

The changes that emerged weren’t revolutionary, but they were concrete:

• Our staff engineer carved out time each week for pairing sessions and project check-ins

• We broke large projects into iterative milestones tied to clear success criteria and customer problems

• All project discussions (even 1:1 questions) moved to public Slack channels to democratize information

Over time, team morale shifted as we experimented with these (and other changes). Trust grew and we started gaining a reputation as a team that ships (although never on a Friday 😎).

For me, the biggest shift was personal. I faced a conflict I once would’ve dodged, leaned into discomfort and opened myself up to feedback and change.

Lesson: When something feels off, dive in. Treat your team processes just like your product - a living entity up for constant iteration, experimentation and growth.

Lesson 2: Empowerment with Accountability Wins

At Personio, I was empowered to prioritize anything I felt would help us hit our goals. No approval checkpoints or micromanagement. This also isn’t revolutionary, many companies empower individual product teams this way.

The difference was that my managers at Personio built a strict system of accountability to go with this empowerment.

Every couple of weeks we had a product review where my team met with cross-functional leadership (product director, design director, eng director). Every month we had SteerCo meetings where we presented to our C-suite sponsor. For Activation, that was the COO.

This was a structure for self-driven accountability. The agenda for these meetings and what I shared was largely up to me (again, emphasizing independence and empowerment), but each meeting always started with the metric I was responsible for driving. My mentor created a Product Review Agenda format that I shamelessly copied (you can too!). This doc included sections for metrics, team updates, project updates, and discussion topics. I sent out the document ahead of time so that all the update sections could be covered async. Then the meeting itself focused on discussion topics and questions people had raised in their comments.

What made this work was the cadence. Two weeks isn’t long enough to hide mediocre prioritization. You can’t spend a month building something fluffy and hope no one notices. These frequent check-ins create accountability without stripping away autonomy.

The structure at Personio pushed me to get ruthless about prioritization. My managers would consistently ask:

• Would you still prioritize this if you knew it was the last thing your team could build? Ha!

• Is this solving the problem that keeps customers awake at night?

• Are we addressing a deeply held concern or building a nice-to-have?

I’ve written before about these questions (see posts about the re-org question and solving the deepest customer problem). What I hadn’t fully appreciated was how the structure of accountability reinforced them. The questions were embedded in our process.

Many orgs set OKRs, let teams run and it is empowering. But without regular check-ins, it’s easy to drift as nice-to-haves and pet projects creep in. This, in turn, can lead to teams who miss OKRs despite looking productive all quarter.

Lesson: Build in process (like Product Reviews) to pair empowerment with accountability so that your work truly drives your metric and improves life for your customer.

Lesson 3: Culture Doesn’t Need Stability to Thrive

Even as layoff rumors ghosted about, our NYC office stayed warm and collaborative.

The office also had a culture committee that organized lunches, demos, Mario Kart tournaments, and the occasional party. Most days we’d eat lunch together at a long bench table, jumping between work problems and which Broadway show to catch next (the consensus was Oh Mary!). The culture committee was also open if people came up with their own events to build community.

One of our senior engineers started a weekly coffee walk. Anyone free would grab a cup at Black Fox (yummy, but expensive!) and stroll the High Line or chat in Hudson Yards. The best part was how organically it started. One Thursday he posted in our office slack channel asking if anyone wanted to grab coffee. He repeated this a few times then eventually upped the formality with a standing calendar invite.

Another afternoon, a teammate hosted a volunteer event for Heart of Dinner, an org that delivers hand-decorated care packages to elderly Asian Americans in NYC. Our job? Decorate the bags. It was a small, tactile break from laptops and a big reminder of what community looks like in action.

When the office closed, the culture didn’t disappear. NYC ex-Personios still meet for happy hours and hikes. One engineering leader kicked off a weekly interview prep call and created a Slack community. When someone lands a new role, we all celebrate.

Lesson: You’re empowered to change culture. Don’t be afraid to start small rituals like coffee walks or in-office volunteering - it all builds community.

Fin

It’s tempting to measure a job by its duration. But Personio proved that some of the best lessons come from the shortest chapters.

I leaned into hard conversations I once would’ve avoided. I learned what empowerment paired with accountability actually looks like in practice. And I saw what a thoughtful culture can create, even during uncertainty.

I even got to visit Munich twice (although never for Oktoberfest - which, I learned, actually takes place in September). By the second trip, I could confidently order coffee in German.

Auf wiedersehen, Personio. Onward!

Read more Shipping on Fridays

I’m breaking code freeze 🥶 (ha!) to publish this special edition of Shipping on Fridays because this weather miss was more than just a funny moment. It’s a case study of a daily usage product getting a core use case wrong and how misses like this erode customer trust and increase churn.

👋 Hey, I’m Alex. I write Shipping on Fridays to explore the craft of how great products get built and what we can learn from the people behind them. I publish 1–2x a month, and every post is meant to be fun, useful, and a little unexpected; from design sprints to sailing races to holiday chaos. If you’re into learning, product or design, this is for you.

Subscribe now

Prelude to the Storm

I did not want it to storm. On Friday morning, I was taking my family into Brooklyn to see a Monet exhibit at the Brooklyn Museum (highly worth it!). Over breakfast my dad asked if I was concerned about the weather. I wasn’t. Apple had said that there was a “wintery mix” coming in later that day.

Then my dad showed me NOAA’s forecast:

Still dubious, I reached out to the dad group chat to get their consensus:

The Storm Comes for Apple

Apple has built its brand on reliability and trust. Back in 2015 when I joined Hipmunk I remember someone asking one of the co-founders why everyone only used Macs and the answer quickly came back “because we need everyone’s computer to just work”.

My partner and friends will shame me on this if I don’t come clean - I was a late Apple convert. In fact I explored both Windows Phones (yes, I was part of that 1%) and Android before coming over to the Apple ecosystem in 2019 (so late, I know!). I haven’t looked back. Why? Because Apple makes everything easy and “it works”.

In this era when the Federal government is shrinking and shirking some of its traditional responsibilities, it seems easier to trust Apple before NOAA.

Except when you can’t. Except when the app shows a government-issued winter storm warning predicting 5–8 inches, on the same screen as “wintery mix.” Why does the app not defer to NOAA in cases of a major discrepancy? Perhaps this use case hasn’t been considered. In this case, the simplest explanation may be the most likely: Apple missed this storm and doesn’t have any logic to validate its own forecast against NOAA’s warnings.

Another curious aspect about this storm forecast was that Apple was correct on the temperature forecast - the high would barely get above freezing. Intuitively if the temperature is below freezing if there’s precipitation it will be snow, yet Apple still insisted on a combination of “rain” and “wintery mix”.

For a weather team, forecasting accuracy should be a primary success metric. It’s easy to imagine a growth engagement loop revolving around accuracy: I try Apple Weather, it gets the forecast right, I dress/plan my day appropriately, I use Apple Weather again and recommend it.

Conversely, in this situation, I’m learning the lesson that I shouldn’t use Apple Weather in the snow. This might lead me to experiment with other weather apps, reducing my usage of Apple Weather from daily to occasional. We’ve seen this before. Does anyone remember the early days of Apple Maps?

How might Apple Weather Improve? What might we learn?

Clearly this Snow storm was an edge case (I’ve found Apple Weather highly reliable, although as Pat pointed out on the group chat Apple tends to get snow forecasts consistently wrong). If we suspend reality and pretend to be on the Apple Weather team, what might we do?

First, as discussed, forecast accuracy should be one of our core success metrics. Why? Because the more often our forecasts are accurate the more our customers will turn to directly use Apple Weather. Forecast accuracy will propel growth when high and can spur churn when low.

Given this, it would be important to understand when we are wrong and to uncover patterns in our errors. I’m curious if the Apple Weather team has recognized that they got the NYC / CT snowstorm of Dec 26 wrong. Is there a pattern of misses with snow? As the pretend Weather PM I’d want to get an alert when our forecast deviated so strongly from reality.

Coming back to reality, are there proverbial snowstorms in our own products? What are core aspects of our products that we don’t do as well as we should and could be degrading customer trust?

The Aftermath

I paid attention to NOAA and we left Brooklyn before the snow really started. Had I trusted Apple blindly I would have stayed in Brooklyn later (our original plan was to have dinner there) and we would probably have gotten stuck somewhere on the road between Brooklyn and Connecticut.

In a heartfelt effort to help Apple learn and grow I’ve sent this post to a few people I found on LinkedIn who work at Apple Weather.

Read more Shipping on Fridays

In an effort to help you be a hero, here’s a guide to PM’ing the holidays.

👋 Hey, I’m Alex. I write Shipping on Fridays to explore the craft of how great products get built and what we can learn from the people behind them. I publish 1–2x a month, and every post is meant to be fun, useful, and a little unexpected; from design sprints to sailing races to holiday chaos. If you’re into learning, product or design, this is for you.

Subscribe now

Know your Holidate

Who are you “holidaying” for? Are your in-laws coming to town? Are you going away with the kids? Is it you and your partner? Just you?

Competing segments? Prioritize, like any rational PM. Your holiday KPI? Holiday Cheer, a roll-up metric of happiness and the viral spread of jolliness.

Above, I mapped Holiday Cheer vs my confidence that I could actually boost the metric for each group. I thought about the trade-offs. For instance, focusing on my kids would result in medium cheer: they’re thrilled, but it drains the rest of us. I also rated them low confidence because they are so fickle - one forgotten candy cane and the day could go off the rails. 😕

On the other end of the spectrum, focusing on friends only and ignoring your family won’t generate too much holiday cheer (imagine the repercussions from your partner?!), but I’m highly confident I can do it really well.

The sweet spot for me is my wife (green dot above). Increasing her holiday cheer makes her a better person for my kids, parents, her parents and friends - everyone! It’s not a guaranteed win, but the upside is massive - which is why I’ve prioritized her.

Identify your Grinch

So now that we know our holidate, we must holiday them (ie increase their Holiday Cheer). How, you might ask? Is it donning a Santa Outfit? Pulling a cheesy stunt a la Love Actually?

No! Of course not - we’re jumping straight to solutions. First rule of PM’ing the holidays? Start with customer problems. What issues does our segment face over the holidays? Who’s the Grinch stealing our holidate’s Holiday Cheer?

For my chosen segment - my wife - there’s dozens of potential issues, but her Grinch is all the little stressors that stack into one sleep-stealing monster. Kid gifts, kid’s school parties, cards for teachers, mailing holiday cards, planning meals, etc. Taking these off her plate is a gift that will reduce her stress, allowing her to have more fun (a key driver of our metric “Holiday Cheer”).

Increase Holiday Cheer

From here, you know what to do. Once you’ve got your Grinch now it’s all about identifying solutions to defeat that Grinch. What will these be? More candy canes? Ensuring the latkes are served warm and with apple sauce? Maybe.

Here’s how I approached my Grinch. I went to the old PM standby: Effort vs Impact, err Holiday Cheer. On one end of the spectrum I could attempt to automate away all these small holiday stresses, which might increase my wife’s holiday cheer, but it’s not even clear I could reasonably automate everything. I settled on just silently taking on some of these tasks each day. I reasoned that disappearing the tasks without fanfare would surprise, delight and lead to larger increases in our North Star metric - Holiday Cheer.

Defeat your own Grinch

Now that you’ve got the framework, go out identify your own Grinch and ship some Holiday Cheer (just not on a Friday).

Read more Shipping on Fridays

We needed some rocket fuel. Nathan Bilbao, the activation product designer, and I had been planning an offsite in Munich. We asked: what if we used that time to chase moonshots? How might we do that in a productive way? Would it be a design sprint? A hackathon?

It turned out to be a little of both. This is the story of how we challenged a group of 15 people from across Europe and the US to create concepts that would empower customers to activate in 5 days. What followed was an AI-first hybrid design sprint where four bold prototypes emerged; each reimagining how we onboard customers. This post breaks down how we ran it, what worked, what we’d change, and how you can use the format to tackle big, thorny problems in your own org.

👋 Hey, I’m Alex. I write Shipping on Fridays to explore the craft of how great products get built and what we can learn from the people behind them. I publish 1–2x a month, and every post is meant to be fun, useful, and a little unexpected; from design sprints to sailing races to holiday chaos. If you’re into learning, product or design, this is for you.

Subscribe now

How we created a supercharged design sprint

How might we get moonshot concepts out of 15 people traveling across two continents to come together in Munich for just 3 days?

We built a hybrid: the structure of a design sprint, the speed of a hackathon. We divided the group into 4 x-functional teams that included PMs, Designers, Engineers and Customer Experience members. The teams were intentionally shuffled so that people were working with other people they usually didn’t work with. Mixing teams killed old assumptions, forced fresh thinking, and ensured cross-pollination.

In the six weeks leading up to the design sprint we did a lot of prep work in addition to the normal planning and logistics of running a design sprint. First, we mapped the full onboarding journey across teams and channels to expose problems that were slowing activation. We wanted to arrive at the design sprint with a journey map to align everyone on the current process, customer pain points, and internal friction. To incorporate customer quotes into the journey map, we leveraged reams of raw data: Gong calls, customer surveys, as well as Salesforce data detailing win and loss reasons. Nathan leveraged NotebookLLM, a research specific LLM, to help us sift through this data. Nathan used the LLM to create summary reports of top customer problems identified within each source.

Knowing that we would be encouraging teams to take an AI-centric approach to prototyping, Nathan also prebuilt code assets so that each team would have a library to build from - giving our design sprint teams a jump start.

At the design sprint itself, we created a culture that balanced group collaboration with individual team identity. We wanted to spark some competition and for the teams to work independently and take full ownership over the concept they develop, while also ensuring that the group learned from each other.

We accomplished this by creating daily agendas that balanced points of convergence - where we heard from each other or subject matter experts with long periods of divergence and individual teamwork. See our sample agenda (that you can copy!) here. Our three days looked like this:

1. Day 1 — Understand + Brainstorm
    AM: Customer pain points (empathy exercise, expert talks and journey map)
    PM: Team breakouts + brainstorming / early storyboarding

2. Day 2 — Storyboard + Prototype
    Storyboarding → Prototype building

3. Day 3 — Refine + Demo
    Finish prototyping → Rapid-fire testing → Iterate → Demos!

Building Empathy Early

To kick things off on Day 1, we gave each team a fake company account, the onboarding project plan given to customers, and 45 minutes to get as far as they could into onboarding. The goal? Feel the pain our customers face every day.

We took a few steps to make this scenario real. First, we worked with Implementation Managers (the folks who help our customers onboard) to build company dossiers based off of actual customers complete with messy data and typical human errors. Second, we included “mystery envelopes” with special requirements. Implementation Managers had stressed to us that edge cases show up for every customer. It was crucial for our design sprint participants to experience these curve balls - again based on actual customer scenarios. Lastly, after the 45 minutes ended we came back as a group (one of our convergence times!) and several Implementation Managers led the whole group in a discussion to reflect on the customer problems.

We ended with a group of frustrated people. Sticky notes filled with “why can’t I…?” and “how am I supposed to…?” moments. One team member nailed it in our reflection:

“This sprint isn’t just about optimizing form fields — we’re going to have to rethink onboarding.”

Exactly. That moment flipped the switch - our empathy exercise had not only helped team members understand the end user pain, but also the severity of the business problem. It also acted as a fast-forward button for team cohesion. Working under pressure on a shared problem quickly broke the ice.

Empathy exercises are common in design sprints. We took it further by designing one we knew was impossible. No team was going to successfully onboard a fake company in 45 minutes — and that was the point. They’d hit the same roadblocks our customers face.

AI Centric Approach

We replaced typical low-fidelity paper prototypes with an AI centric approach. Instead of teams expressing their ideas on flip charts we urged teams to utilize the code assets Nathan created coupled with Figma Make and Cursor. This had three main benefits.

First, it allowed more members of the team to actually contribute to the prototype creation. Instead of team members operating within their single sphere of expertise, designers, PMs and engineers were all able to build and create together. Non-designers were able to express ideas visually. Designers were able to make interactive flows. Utilizing AI allowed teams to take ideas from sticky notes to interactive prototypes in minutes and hours. This sped everything up and let teams test ideas earlier. Instead of debating solutions, using AI allowed teams to quickly put those ideas in front of Implementation Managers for feedback.

Lastly, AI gave us durability and momentum to move forward at speed with next steps after the design sprint ended. By utilizing AI all the teams created designs and prototypes in a company repo, which we were then able to access and begin iterating and remixing in the weeks after the design sprint.

In Context Testing

When Nathan and I developed the agenda we built in six touchpoints for teams to test their concepts. I’m a strong believer in testing early and often (read more on testing strategies here) and being in Munich gave us a unique opportunity.

Munich is home base for our Implementation Managers, whom, given their daily work with customers, we could proxy for actual customer testing. We locked in 2-3 30-minute sessions with them for Day 2, giving each sprint team direct feedback on storyboards and early concepts. This differed from a classic design sprint where testing is normally in a last validation phase. By starting testing early we ground the culture of the workshop in constant iteration (like a hackathon). The check-ins themselves accomplished three things: they ensured ideas were based on real-world experience, they created time pressure by splitting the day into short, focused build sprints (a la hackathon), and it built cross-functional engagement with the Implementation Managers for future adoption.

Day 3 was more intense. Each team ran three back-to-back user tests with different Implementation Managers — no breaks, no time to iterate. Test, capture, reset. Only after the final test did teams regroup, synthesize the feedback, and adjust their prototypes ahead of the final demo.

This rhythm worked. The Implementation Managers didn’t just review concepts, they pressure-tested them, shared customer stories, made ideas sharper and allowed sprint teams to kill concepts that wouldn’t work. With over five points of feedback, teams were able to spot patterns. The result? Prototypes that were more grounded, more useful, and way more likely to stick.

The Results

The sprint wrapped with a full-room demo: our COO, his leadership team, and the entire Implementation crew. Basically, everyone in the company who cared about Activation.

Each team took a bold swing at rethinking onboarding. Some leaned into AI to cut manual steps, letting customers start with whatever documentation they could provide: tax statement, employee handbooks, even email threads. Another team designed ways for customers to browse configuration examples and copy settings into their own account.

All the teams made clear which customer problems they were solving, why those problems mattered and their hypothesis to solve them. Our sprint had both deepened our understanding of the problems customers face and offered concrete concepts (which with some iteration could be shipped). We could now chart a plan for getting customers to go live in a fraction of the time - maybe even 5 days! 😉

By working in a hybrid design sprint hackathon format we let teams move fast, stay focused and accountable. The final demos didn’t just showcase the prototypes, they also revealed how much can be built when rapid iteration is balanced with ownership and structure.

Reflections & Learning

1. Plan obsessively. Start early to align calendars, secure travel, book rooms, and prep materials. Details like sticky notes and lunch matter more than you’d think.

2. Use empathy as your icebreaker. We skipped a traditional icebreaker in favor of a hands-on onboarding simulation. It got teams talking fast and framed the challenge perfectly.

3. Protect your agenda. Time is tight. Every session needs to earn its spot. To sanity check, ask yourself the re-org question — would I fight to keep this if I knew this was the last time I was working on this problem? (More on that in this post.)

4. Prepare for success / Lighten your participants load Give your teams the best chance to take big swings by pre-seeding assets you think they might need to build their prototypes.

5. Test early and often. The biggest unlock came from repeated, real-time feedback. Don’t save testing for the end.

6. Own the tension between structure and speed. We gave teams autonomy but held a tight daily rhythm. That mix pushed teams to move fast and stay focused.

7. Design for moments, not just outputs. The onboarding simulation, feedback loops, and demo hour all created energy and alignment. Think beyond the sprint wall.

Steal our Playbook

A design sprint gives you structure. A hackathon gives you speed. Together, they gave us something better: momentum to improve customer onboarding.

We didn’t leave Munich with a perfect prototype or something we could immediately ship. We left with a deep understanding of the customer problem, lots of ideas on how to work better cross functionally, and four prototypes that contained the kernels of a solution to supercharge customer onboarding.

If you’re stuck in slow progress, give it a shot. Use our agenda. Try the empathy setup. Adapt the format to your world — and let us know how it goes.

Read more Shipping on Fridays

At first it shocked me - he’s my manager - what org changes are executives conjuring up? But this wasn’t a threat, it was more of a mantra: “Prioritize your features like you’re about to get re-org’d”.

My manager was introducing a mindset of building with urgency and clarity. Scarcity of time, of resources and of certainty sharpens focus. It forces you to choose what really matters. And re-orgs aren’t the only source of scarcity. Budget freezes, layoffs and roadmap resets all create moments where priorities get rewritten overnight.

👋 Hey, I’m Alex. I write Shipping on Fridays to explore the craft of how great products get built and what we can learn from the people behind them. I publish 1–2x a month, and every post is meant to be fun, useful, and a little unexpected; from design sprints to sailing races to holiday chaos. If you’re into learning, product or design, this is for you.

Subscribe now

A new edge to my prioritization framework

As a PM I already knew I should be consistently prioritizing the work that hits the trifecta of:

1. Solving the deepest customer problem

2. Generating the greatest business impact

3. Costing the least effort

But this re-org framing added a sharp edge: what if you only get to ship one more thing that impacts your OKR? What would it be?

Wait, isn’t this just a scare tactic?

At first, it felt like one: my manager threatening a re-org to force prioritization.

But the truth is, every team has finite resources. And in tech, priorities can shift fast - and sometimes without any warning. To make the most progress against our OKRs, we need to stay ruthlessly focused on the highest impact work and stress-test our roadmaps to keep ourselves honest. Realistically, as PMs, we rarely know when our teams will be pulled in a new direction.

Tell me about a time…

This mantra of scarcity prioritization sunk in a few months ago. My manager and I were reviewing a large new feature that I had broken down into small iterations.

The first iteration was an analytics widget to give customer admins an easy way to understand their platform utilization. My hypothesis: by making it easy for admins to see their utilization, Admins will invite employees at a faster rate.

I excitedly explained that the team would build this fast, delivering incremental, but immediate value to customers even as the next phase of the project was in development.

My manager listened, leaned back and asked me that simple question: “does it pass the re-org test?”

We grabbed a whiteboard and ran a quick RICE analysis: Reach, Impact, Confidence, Effort. This framework is great when you need to strip away emotional attachment and compare ideas apples-to-apples.

We compared the analytics widget to phase two of the project: Access Review, a feature designed to give admins confidence in who had access to what. This was a much bigger blocker to activation and something customers told us kept them up at night. It also was higher effort.

As we talked through the project phases and scored them, it became obvious that the core difference was their potential impact. While it would be fast, my proposed first phase wasn’t going to be very impactful. In contrast, the next phase of the project, Access Review, was directly aimed at increasing customer admin confidence to invite their employees to our platform. Access Review directly hit at the core customer problem - confidence, while the Utilization Widget didn’t actually solve anything for customers.

If we were going to be re-org’d Access Review was the project we should work on. Imagine if we launched the Utilization Widget and then stopped working on Activation? We wouldn’t have actually solved any customer problems.

Lessons Learned:

1. Push yourself and your team to adopt a scarcity mindset. What would you ship if you could only ship one more thing to impact your goals?

2. Give those “quick win” projects extra scrutiny. Are they actually wins that will move your core success metric? Or just easy vanity work?

3. Use frameworks like RICE to strip away bias and make confident calls.

Post Script:

Glad we asked the hard questions. Turns out, it really was our last project focusing on Activation 😉

Read more Shipping on Fridays

That was the recurring theme when I caught up with Nathaniel Halsey, my former manager at Citi who led personalization and more recently created the go-to-market strategy for an AI decision engine at Curinos. Altogether Nathaniel has spent 20 years in financial service spanning product marketing, digital transformation and consulting including stints at Stripe and Visa.

As we swapped stories, a clear theme emerged: great personalization provokes. It surprises. It delights. And it drives meaningful action.

This week’s Shipping on Friday is that conversation — part nostalgia, part strategy session, and a glimpse at where personalization could go next.

👋 Hey, I’m Alex. I write Shipping on Fridays to explore the craft of how great products get built and what we can learn from the people behind them. I publish 1–2x a month, and every post is meant to be fun, useful, and a little unexpected; from design sprints to sailing races to holiday chaos. If you’re into learning, product or design, this is for you.

Subscribe now

Provocation #1: Good Personalization Surprises and Delights

Alex: First of all, thanks for joining me! When we caught up about our contextual messaging days at Citi, it felt like there was a deeper story — and that this is something you’re still thinking about.

Nathaniel: For sure it is! Since Citi I’ve continued to work in and think about personalization. Most recently, I attended a roundtable discussion hosted by Contentful with marketing leaders from Logitech, Wells Fargo, and Visa. The conversation was all about the future of digital experiences: Personalization at Scale.

Right up your alley.

Indeed! One point that really resonated with me was how great personalization provokes. Personalize and provoke. Provoke a response. Provoke a reaction. And that could be surprise and delight.

That phrase — ‘surprise and delight’ — takes me back to Citi.

Yes, and I think of our time at Citi as being at the forefront of personalization at scale. It comes down to two things: 1) the plumbing to deliver the message; 2) the value of what you’re actually saying. Is the offer contextual? Is it compelling? You can have the best data and targeting capabilities, but if your message isn’t relevant, helpful, or well-timed — why would I care?

Provocation #2: Relevance Comes From Timing and Context

Right. I remember we always strived to be relevant.

Do you remember Mobile Check Deposit? That was a great early example. If someone had just deposited a check at a branch or ATM, we’d follow up with a message showing how they could save time by using the mobile app instead. It was one of the first use cases we tested — and it worked.

It goes back to our principles at Citi: demonstrate that we know the customer, and demonstrate that we care. One key to success was timing. We wanted the message to hit while the pain of waiting in line at the bank branch was still fresh. That’s when the relevance was highest.

And it worked. Our contextual messages had click-through rates up to 51% higher than generic marketing offers.

That’s because relevance drives results. And so does content that provokes.

Remember Mike Rotella’s work during Hurricane Sandy? That was provocative. He had the idea to show customers in New York a link to donate to the Red Cross. He was excellent at driving relevance, but it was always so much manual work - impossible to scale. Imagine if Mike had AI back then?

It would have really super charged his ability to match customers to relevant content. He could have been matching spend data with real-time context. Today, LLMs can surface relevant content instantly, pulling from transactions and user behavior to match the moment.

We actually tried to do some of that manually. Like analyzing large airline purchases and sending proactive reminders: no foreign transaction fees, or prompts to set travel notifications. Our roadmap even included reminders post-landing.

Yes, that’s right — I’d forgotten about that. With AI and LLMs, you can now automate those kinds of journeys. But when it comes to delivering a truly differentiated merchant offer, automation isn’t enough. Someone still has to source the deal.

Provocation #3: AI Can Learn You — But It Can’t Woo You

Has anyone cracked that well at scale?

Not really. CVS is a brand that impresses me; they tailor offers based on what I actually buy and reinforce the value of Extra Rewards. I recently went in for Tide detergent — what would’ve been $19 was $9 with my rewards. That’s a true surprise and delight moment with a compelling offer. Too often, offers are generic or irrelevant. 5% off something I wasn’t going to buy anyway isn’t motivating. When brands get it right, it feels personal. When they don’t, it feels transactional.

Right, when a brand does that it doesn’t feel personal at all. What it looks like is that they’ve struck a deal to steer you to spend.

You have all this data about me, and this is the best you can do? Now in fairness to these teams, it’s not easy to source deals that are relevant to people. AI can spot patterns. But turning those into compelling offers? That still takes hustle — sourcing deals, shaping context, crafting timing.

The brands that will win are the ones who can leverage the full spectrum of available customer data — securely, compliantly, and with care — to create experiences that are not only personalized, but meaningful. The challenge is ensuring that data fuels content that is compelling and timely, not just accurate.

And financial services comes with added complexity. Personal data is sensitive; regulations are strict; and even well-intentioned personalization can feel intrusive if not handled carefully. That’s why AI must be deployed thoughtfully — not as a blunt tool, but as a precision instrument. It shouldn’t just automate; it anticipates.

The Future: Personalization That Feels Like a Superpower

So where do you think personalization and offers from banks are going?

I think AI will drive a curated experience like a virtual concierge. Let me illustrate. I’m flying to Boston, then renting a car at Avis and heading to Provincetown. I booked it all on my Citi card. Citi knows my Avis pickup is off-airport at the Seaport.

Imagine they send me a discount offer for an Uber to Avis — if I use my Citi card. Then layer in a limited-time offer at Flour Bakery, because they know I always stop there. That’s not just personalization — that’s anticipation.

That’s powerful.

Today, Citi knows they aren’t my default card on Uber or Lyft. This kind of orchestration would help them win that spend. They could offer a discount as soon as I book the flight. The Flour Bakery offer would be the delighter. It says: we see you. We know you. We’re helping you.

Thanks to recent advances, this level of orchestration is no longer hypothetical. AI and large language models can now analyze signals, craft messaging, and deliver hyper-relevant content at scale — something we couldn’t have done even a few years ago.

If personalization is going to escape the realm of discounts and awkward emails, it needs to provoke. Not just know who I am, but make me feel known.

AI is the most powerful tool we’ve ever had — but it’s human creativity, ethics, and care that turn it into something truly valuable. The brands that win the next decade won’t just have the best data — they’ll use it to create moments that matter.

Shipping Takeaways: How to Personalize Without Being Boring

• Be provocative — and specific. When we used travel data to trigger proactive messages like “no foreign transaction fees,” engagement spiked. Personalization that provokes is personalization that performs.

• Use what you know — or lose trust. Brands have the data. When they ignore it, customers feel unseen. CVS got it right when they offered $10 off detergent I actually buy.

• Timing is everything. Our Mobile Check Deposit campaign worked because we hit users right after they waited in line at a branch. The pain was fresh — and the message was magic.

Have you seen a moment of personalization that actually worked? Share it in the comments — I’d love to hear what stood out to you.

Read more Shipping on Fridays

‘There are things that I see / That I never could spell if I stopped with the Z.’

And just like that, I was reminded — great product thinking doesn’t stop at Z.

As I read, and the narrator took me on a magical journey, I thought about a few companies that thought differently about the world. In this post I want to celebrate 3 features that approached problems with solutions that - at the time - set them apart from their peers.

Subscribe now

Hipmunk’s Gaant chart flight search

Hipmunk was a travel search startup where I worked for two years until it was acquired by SAP. One of Hipmunk’s unique qualities was the way in which flight search results were shown and organized. Hipmunk sorted flights by a brutal but brilliant metric: agony. Not just price — but layovers, red-eyes, and midnight connections. Then it visualized that pain with a Gantt chart. Suddenly, the worst parts of flying were clear at a glance.

Every other site defaulted to a list — rows of flights, departure times, and prices. Hipmunk flipped the script. Instead of dumping a list of flights, they visualized pain — with a Gantt chart. It was a radical shift toward empathy in design.

Waze’s real-time routing updates

Waze was a map app that gained notoriety for being able to get you to your destination faster than Google Maps. Waze had such a reputation for this, in fact, that they were ultimately acquired by Google. How did Waze do this? Beat a superpower at their own game? They crowdsourced real-time, hyperlocal knowledge — turning every driver into a data point. Suddenly, Waze wasn’t just a map. It was a movement.

How’d they do this? Waze handed the map to its users, asking them to report crashes, speed traps, and construction in real-time. That turned early users into heroes. Their tips powered smarter routes, and every ping of data made the map feel alive. That stream of user data fed Waze’s algorithm, enabling real-time route updates.

Foursquare’s mayorships

Go back to ~2012 and Foursquare was an app that allowed people to check-in at any location, leave reviews, photos and tips. Sounds a lot like Yelp or Google Maps, right? The problem they faced was how to gain traction in such a crowded space?

They didn’t try to out market Yelp or Google, they innovated and made check-ins feel like a game. It was location-based dopamine — status and bragging rights with every check-in. Rack up enough check-ins and you became ‘mayor.’ The bragging rights made it fun, addictive, and inspired users to leave amazing tips. The game generated talk and viral growth.

Let these products inspire you

These products may be gone, but their ideas live on — each one a masterclass in thinking beyond the obvious. The teams that created Hipmunk’s Gantt chart, Waze’s responsive algorithm and Foursquare’s mayorships all went beyond Z to solve problems that transformed their companies and influenced industries.

So next time you’re stuck in a brainstorm or mired in the obvious, remember: the magic doesn’t stop at Z.

I tried hard to tell

Young Conrad Cornelius O’Donald O’Dell

A few brand-new wonderful words he might spell

I led him around and I tried hard to show

There are things beyond Z that most people don’t know

I took him past Zebra. As far as I could

And I thing, perhaps, maybe I did him some good…

Last fall my sailing team entered the final stretch of a race in last place - and then a/b tested our way past 3 boats!

👋 Hey, I’m Alex. I write Shipping on Fridays to explore the craft of how great products get built. In honor of the summer, this week’s Shipping on Fridays highlights the similarities I’ve noticed between a high functioning sailing team and a cross-functional product team. If you’re into learning, product or design, this is for you.

Subscribe now

The tiniest bit of background on sailing

I race on my friend’s 33-ft sailboat on weekends. It has a top speed of about 9-10 knots (~12mph) surfing in 15 knots (17mph) of wind.

Like a product team, each crew member has a specialized role — the skipper steers, the main trimmer controls the big sail, the jib trimmer fine-tunes the front sail, and the bow crew (me) handles everything up front to keep the boat moving fast.

It’s a perfect cross-functional setup: different skills, constant communication, and one shared goal — make the boat go faster.

Our a/b testing win

Last fall we found ourselves rounding the 2nd to last mark in last place. It was the sailing equivalent of being down 4 runs, starting the bottom of the ninth inning in baseball. Heartwrenching.

Even worse, since everyone we race against sails the same type of boat and we all use the same wind - there’s no way we could just “push harder / faster” to catch up. Morale was low. It’s one thing to lose, it’s another to see no clear way to win. We had to think differently.

The skipper called out “how might we pass these boats ahead of us?”. Without thinking about it, he had begun a cross-functional brainstorm. We already had a strong culture of open communication and learning on our boat - so it felt natural for him to ask for ideas.

Someone suggested shifting our weight in the boat. There was agreement that we should test that, so we did. We moved where we were sitting in the boat to give the boat a little more heel and watched our speedometer to see if there had been any change. After a few minutes it was clear - our speed had consistently ticked up a little bit. We had our first winning test!

Throughout the remainder of that race we “launched” 3-5 more “tests”. For instance, the jib trimmer suggested we use the outboard leads to trim the jib. We tried it, waited for a speed reading and got another tenth of a knot.

Some of our experiments worked, a bunch more failed, but the winners ended up netting us enough incremental speed to put us back into the competition and pass two or three boats before the end of that race. A remarkable achievement!

Lessons Learned

That race taught me an important lesson that I take back to work: speed (winning!) comes from learning faster than your competition.

We didn’t have a faster boat. We didn’t have more resources. We had a team with the resilience to turn around a bad situation. The culture of open communication and continuous learning we had built all summer allowed us to have that impromptu brainstorm. Each test was lightweight and fast — and together, they stacked into a win.

In product teams, it’s the same. We’re often under-resourced with ambitious goals. We must test and learn to discover the solutions that really move the needle. And those tests don’t need to be high fidelity or production code to learn (see related post).

Lesson: Build an open culture where anyone can suggest a test. Run small, iterative experiments. Stack the wins, and you’ll make big comebacks possible.

Read more Shipping on Fridays

We’d just wrapped a month-long design sprint aimed at solving a thorny customer onboarding problem: how to give customer admins the confidence that they had correctly configured access rights. We’d done all the right things: x-functional brainstorms, customer interviews, stakeholder meetings, surveys, sketches, prototypes, user testing. We were ready for our product review.

In this edition of Shipping on Fridays (boldly shipped on a Saturday!) I’m going to dive into an aha moment that emerged from a Product Review, which saved my team from building the wrong solution. Enjoy!

👋 Hey, I’m Alex. I write Shipping on Fridays to explore the craft of how great products get built and what we can learn from the people behind them. I publish 1–2x a month, and every post is meant to be fun, useful, and a little unexpected; from design sprints to sailing races to holiday chaos. If you’re into learning, product or design, this is for you.

Subscribe now

Our perfect solution

The stakes were high. If you’re setting up a platform that handles sensitive things like payroll, attendance, and personal employee data (think Workday or Rippling), one mistaken permission can have serious consequences. Imagine a junior employee stumbling across everyone’s salaries. 👻

Our solution let customers preview the platform from any user’s perspective. What made us so confident we had it nailed was that customers loved it. In seven customer interviews it consistently ranked as the top concept. Our internal stakeholders also gave it glowing reviews - the Customer Success team couldn’t wait for the launch.

Why’d they love it so much? It turned a customer support hack into a polished, in-product feature. Our solution took something familiar and already helpful and made it even better.

The aha moment that saved us

With this confidence we went to a product review with other PMs, designers and the VP of Product. Everyone was impressed with our positive customer feedback. It seemed like we had a slam dunk.

Then the VP of Product leaned forward and asked. “How does this actually solve the customer problem more elegantly than today?”

The room went quiet.

We’d just spent weeks refining this idea. Customers loved it. Our own team loved it. But now, the question hung in the air—and we didn’t have a crisp answer.

Would this really help customers find misconfigured permissions? Would it save them time? Would they honestly click through dozens of profiles just to verify access? Or were we simply giving them a shinier version of what already existed—a solution that didn’t move our north star metric?

That’s when it hit us: We had a false positive.

We’d created something customers asked for, but not what they truly needed.

Back to the whiteboard we went. This time, we asked ourselves: what would genuinely build confidence? Several weeks and multiple sketches and prototypes later the answer emerged: a clear, digestible summary of who has access to what—across the whole org. Something you could scan in seconds and act on immediately.

Customers and internal teams lit up. We hadn’t just improved the UX—we’d removed a blocker.

💡 Lessons Learned

1. Be a steel man. Challenge your own solution and try to approach your work from the opposite perspective and poke holes in it.

2. Get peer feedback early. A fresh perspective can expose hidden flaws, especially when you're too close to the work. Encourage your org to host product reviews with other PMs and designers - you’ll raise the quality of each other’s work.

3. Lean into hard questions. When you provide peer reviews don’t shy away from the hard questions. The tough questions are often the ones that unlock the real solution.

✅ Final Thoughts

You can do everything “right” in product development—talk to customers, run design sprints, test with users—and still almost ship the wrong thing. That’s why it’s so important to stay humble, stay curious, and stay open to course correction.

Read more Shipping on Fridays

Have you ever ignored a suggestion for fixing a small customer problem? Maybe it didn’t align with the roadmap, seemed too minor, or was hard to quantify. Whatever the excuse it’s easy to put off such work.

Yet, fixing these little things show customers you truly understand. In a world where winning and retaining customers is all about delivering a superior experience, when employees are empowered to come forward with solutions to customer pain points, the business will thrive.

This is the story of one such instance, drawn from my own experience on spring break.

Subscribe now

On vacation the little things can sometimes present big dilemmas. It was a beautiful Tuesday in St Croix and we were heading out on a snorkel trip the next day. The tour was BYO lunch—but we were staying at a hotel and didn’t have a car, a deli, or any groceries.

I went to the front desk to ask for their advice. I was expecting an unapetizing recommendation like “buy cliff bars from our store” or “order room service tonight and save it for tomorrow”. For me and my picky-eater kids, this was a mini moment of truth.

To my surprise and delight the woman behind the front desk reached into a drawer and produced a “Boxed Lunch Order Form”. It had a short list of simple options—PB&J, ham and cheese, fruit—and instructions to fill it out the night before. The lunches would be ready at the front desk by morning.

I was so amazed that I asked more about it. She told me this came from staff who kept hearing the same guest question. So, they created a fix. No systems overhaul. No new ops process. Just a paper form and some sandwiches.

Why I Loved It

This little boxed lunch menu reminded me of how great product teams should work:

1. The best features often come from the frontlines.
The front desk team saw the same guest issue come up over and over. They didn’t wait for a leadership mandate to brainstorm solutions—they just built one.

2. You don’t need to “scale” everything.
This wasn’t automated, personalized, or AI-powered. But it worked. Sometimes “scrappy and reliable” is exactly what’s needed.

3. Delight comes from solving real problems.
There was no surprise-and-delight budget. No free champagne. Just a clear signal that someone had thought about this problem before—and built something to make it easier.

What’s Your Boxed Lunch Menu?

Now I’m thinking about how I might carry this same ethos back to my team building software for our customers. Here are three prompts I’m thinking about:

1. What customer problems do frontline teams hear over and over?

2. Are users hacking their way around gaps in our product?

3. Do those same frontline employees have ideas on how to close these gaps?

As for the snorkel trip?

It was gorgeous. I can’t recommend St Croix enough. And that boxed lunch powered the Shipping on Fridays family through an adventurous day.

Read more Shipping on Fridays

As my time wound down, three core elements of PagerDuty’s culture stood out: collaboration, learning, and an almost zealous obsession with customer trust.

In this week’s Shipping on Friday, I’ll share three vignettes that stuck with me.

Subscribe now

Collaboration: The time we Sent 5 Emails in One Day—and Fixed It

Early on at PagerDuty, I started digging into our free trial experience (for the full backstory, Drew McKinney and I wrote it up here). One thing jumped out immediately: new customers were getting bombarded with emails. On Day 1 alone, they received five messages from three different teams. It was a classic case of “shipping the org chart”—and the result was confusion and clutter for the very people we were trying to convert.

Fixing this wasn’t as easy as saying “let’s just send fewer emails.” Each message had an internal owner, a stakeholder, a purpose. To combat this, we formed an unlikely team: a sales leader, a UX content writer, a sales data analyst, a product marketing manager, and me, a PM. Most of us had never worked together before, and now in this first mission to fix the Day 1 experience, we were being asked to each give up emails for a shared outcome.

That’s where culture came in. What could’ve been a turf war turned into a design sprint. We leaned into the mess with crazy 8s, dot voting, and a healthy dose of curiosity over control. Everyone approached the problem with empathy—for each other and for the customer. We went from five disconnected emails to one clear, cohesive message.

And it worked: the new email drove more than 300% increase in 7-day conversion for those who opened it.

We didn’t just improve the onboarding experience. We proved that collaboration at speed is possible—when the culture allows it.

Learning: The Time we Stretched our Minds Across Time Zones

PagerDuty has a tradition called PagerCon—a two-day, internal conference where anyone in product/dev can give a talk. One really special aspect: PagerCon is the only time people gather in co-working spaces across the world on the same day to connect with each other. And the talks can be about... pretty much anything.

A few topics that stuck with me:

• How to overcome writer’s block

• Being attached to outcomes, not specific ideas

• Using Navy SEAL breathing to stay calm in stressful situations

• Finding $8M after examining how month-to-month pricing actually worked

• Gold mining in Tahoe

What stood out to me was that the product development team (cheered on by the CTO) gave up two days of development time to stretch everyone’s mind. And it worked! Those two days were thought provoking and the spontaneous conversations that I had with the team in NYC always ended up sparking new ideas and energy that I brought back to the Growth team.

Customer Trust: The time the Co-founder Disagreed—and Was Right

Some time after launching the Operational Maturity Model (OMM - a feature that helps customers understand the health of their account’s configuration on PagerDuty), I was presenting an update on success metrics in our monthly product metrics meeting. It was one of the few standing meetings I both dreaded and loved. The CPO and PMs alike could chime in, ask questions, and challenge assumptions (looking at you, Sean Scott).

​​Midway through my slide on the OMM, Alex Solomon, our co-founder, stopped me.

His question? Does the model make it too easy to score well? Could customers check the boxes without actually improving their operational maturity? Alex feared negative outcomes for customers and the business. PagerDuty risked inflating our % of well configured customers which might lead us to underestimate our churn risk. If customers, on the other hand, didn’t actually gain operational efficiencies, but saw themselves as more mature they might gain a false sense of security. In essence he was concerned there was a potential for a breach of trust.

Ultimately these questions led to a complete examination of how the OMM score was calculated, ensuring that customers had to complete foundational steps before they could level up and be considered well configured. It ensured the system aligned to real outcomes.

Alex’s question sparked change in the name of improving customer trust.

Fin.

It’s impossible to sum up years in a few snapshots, but these moments—of collaborating, learning, and standing up for customers—capture why PagerDuty was such a special ride.

I’m off to the next adventure, but I’m taking these cultural lessons with me.

Read more Shipping on Fridays

In this edition of Shipping on Fridays I’m going to critique two different approaches to promoting a feature using Slack and Apple Fitness as case studies.

Subscribe now

Interrupting the User: Apple Fitness

Recently when opening Apple Fitness I was hit with this interstitial promoting new features. This is not an uncommon pattern. An app is updated, presumably with features that solve real customer problems and the team announces the new features.

The interstitial appears once for anyone opening Apple Fitness after an update—likely targeting all users regardless of relevance. I tested it myself: after closing and reopening the app, it was gone. The team behind it was probably aiming to boost daily active users of the new features, or tracking the percentage of users who see the interstitial and engage with the feature afterward.

Let’s take one of the features, “Flexible Goals” as an example. Presumably Apple got customer feedback that fitness users wanted their goals to be dynamic and adjustable to changing circumstances. That feedback probably came in qualitative form (app reviews) and quantitative (perhaps they saw a sizeable population of people adjusting their goals up and down with frequency).

Despite the feature being built for a targeted user segment, it is promoted broadly, interrupting everyone’s journey. Ooof! It’s also out of context. Imagine there was a contextual prompt that displayed inline with my goals?

In-context: Slack

In contrast, Slack uses a discreet contextual promotion to let me know about a new feature. In this journey, I was already on the DM tab of the app and had pressed the “plus” icon. This modal popped with options of actions I could take, with the largest CTA to draft a message.

In this case, the new feature, a shortcut to DM, is directly relevant to the action I’ve just taken (hitting the “plus” button). I’m in the target audience for this message. I’m on the DM tab, I’m about to DM a colleague and Slack is trying to be helpful and show me a faster way to accomplish my job.

The UX of the contextual prompt is also well thought out. The prompt itself doesn’t try to disguise itself as part of the content, it’s shaded and is a clear promotion. It also respects me as a user - it doesn’t interrupt my journey (you could imagine an alternate reality where Slack blocked my ability to DM until I acknowledged their new feature - like in Apple Fitness).

Conclusion:

Next time you're launching a new feature or trying to hit those tough growth OKRs, keep a few things in mind:

1. Who did you build it for? Be specific. A great feature promoted to the wrong audience is a miss.

2. Where and when will they care? Consider the user's context. Are they in the right mindset to learn something new? To try something new?

3. Does the prompt guide or block? Helpful nudges win. Roadblocks rarely do.

4. How might success metrics shape the experience? If you're optimizing for short-term adoption, are you sacrificing long-term trust?

Read more Shipping on Fridays

I chose Uber Eats, an app I use nearly every Friday and started fresh. Here’s a tear down of the experience and some “if I had a magic wand, I’d change” thoughts:

Subscribe now

Login & Initial Sign-Up: Too Much Security Too Soon?

After re-downloading the app, I began the sign-up process by entering my phone number. I patiently completed the 2 factor authentication and then, to my utter shock, I had to do it again!

The app presented me with two separate verification steps, including one screen that asked if I had changed my number. I wondered if this was because I was using my mom’s phone number and Uber smartly recognized the number as associated with someone else’s account?

Regardless, the security felt excessive—especially for a food delivery app. I immediately wondered: is this extra friction really necessary at this stage?

🪄Magic Wand Optimization:

Imagine if I could browse restaurants and stores to order from without being logged in? My hunch is that the majority of brand new downloads are from people that are hungry, or need something from a store immediately. Why not let them browse, add food to their cart and get them to checkout before requiring an account?

Bolder option: allow guest checkout!

Location & Push Notifications: Is Uber Asking at the Right Time?

Once I completed verification, the app prompted me to share my location. I thought, “ok” this is a sensible pre-requisite. In order to show me legitimate restaurant and store options Uber needs to know my location. I accepted.

The immediate next screen asked me to allow push notifications. As a customer I wondered why I need push notifications at this stage of my journey? As a Growth PM I have an inkling that this is so they can send me a notification if I abandon my cart. As a hungry guy looking to get lunch, however, I thought “just show me the food already!”.

🪄Magic Wand Optimization:

• Delay the push notification request until after the first order is placed.

• Hypothesis: Users who are actively tracking their food will be more willing to enable notifications, leading to higher opt-in rates without increasing drop-off during onboarding.

Uber One Membership: Growth Hack or a Trust Killer?

Next up was their subscription play. I was shown the Uber One membership option screen. I get this, membership at this point might be enticing for a first time customer - the savings are real. From Uber’s perspective it is probably one of the highest conversion placements. A few aspects of the screen itself surprised me, however:

• CTA below the fold (where do I click?!) How do I sign up or skip?!

• While Uber One offers 1 month free, this was not prominently displayed. In fact it was almost downplayed, like someone whispering “pssst, ‘1 month free’”.

• The copy also seems sterile. For example: “See benefit details”. This CTA almost seems written to deter clicks. It sounds like it’s going to take me to a screen of legalese. If there are truly more benefits or details that would be beneficial to know, perhaps “Details about everything you get” or even just “See all Membership benefits” would be more enticing. And if it actually is just legalese, why not move it below the primary sign-up CTA?

I clicked “skip” ready to get into the product.

At first I thought that “skip” hadn’t worked because the very next screen was another Uber One Membership screen. My second thought was that I had done something wrong and perhaps had clicked “subscribe” by accident. Then I thought perhaps I’ve landed on an enrollment screen for a new program. Too many thoughts!! By now I’m thoroughly confused and forgot to get a screenshot (sorry!).

I read the subheader on the new screen “As a welcome to Uber Eats try out our Uber One membership free for 1 week”. Wait, this is the thing I just said “no thank you” to, right? Or is Uber trying to tell me that even though I clearly said “no” to the free trial, they enrolled me anyway? 🤔

The only CTA is “got it” so I have to accept the offer / non offer and in a confused, frustrated state I proceed.

🪄Magic Wand Optimization:

Don't deceive your customers. There’s no faster way to kill trust than to trick people into programs. And when trust dies, retention falls and other metrics that Uber Eats cares about also decline (like orders!).

The last emotion you want a customer to feel after onboarding is confusion and frustration. A good onboarding flow should leave a customer feeling excited to get going with your app. The feeling you should go for is that childhood euphoria of unwrapping a long desired present on your birthday.

First-Time Home Screen: A Missed Opportunity?

After completing onboarding, I landed on the home screen, which looked identical to the standard Uber Eats experience. There was no introduction, no walkthrough, and no personalized suggestions—just categories of food and restaurant options. Having used Uber Eats before, the only noticeable difference was the absence of an “Order Again” section.

🪄Magic Wand Optimization:

Imagine how this might be different if Uber Eats asked a few questions (either in onboarding or on this screen) to better understand the goal of the user. For instance, perhaps I downloaded Uber Eats because I am hungry for lunch and want to treat myself. A few questions with multiple choice quick replies to get at a customer’s reason for using Uber Eats and their preferences could go a long way. Knowing this information could allow Uber Eats to present a few highly tailored options on the home screen rather than metaphorically dropping the customer off at the front door of a shopping mall and letting them wander in.

Final Thoughts: Actionable Lessons to Improve Onboarding

Uber Eats’ onboarding experience is functional but leaves room for improvement. The current flow appears designed primarily around business priorities—securing user permissions, enabling marketing channels, and driving Uber One subscriptions. While these elements are important, they risk overshadowing the user’s needs and expectations.

Uber has an opportunity to transform onboarding into a seamless, user-first experience—one that not only facilitates transactions but also personalizes the journey. By better understanding customer goals, preferences, and intent, Uber Eats could act as more than just a delivery service; it could become a true food concierge, guiding users effortlessly from app download to a great meal.

A seamless onboarding experience should feel intuitive, build trust, and guide users toward their first successful action. Here’s how Uber Eats—and any product team—can improve onboarding for new users:

1. Reduce friction at login. Limit security barriers to what's essential. If a phone number is flagged as previously used, provide clear messaging rather than redundant authentication steps.

2. Ask for permissions at the right time. Location makes sense upfront, but push notifications would be better received after a first order—when users are actively tracking their food.

3. Respect user intent. If a user declines an Uber One trial, don’t reintroduce the same offer in a way that creates confusion. Clarity builds trust, and trust builds long-term retention.

4. Guide users toward action. Instead of a generic home screen, use onboarding to personalize the experience. Ask users why they’re here—whether for a quick meal, groceries, or a group order—and tailor recommendations accordingly.

What do you think? Have you seen similar onboarding friction points in other apps?

Root cause analysis, as this work is called, is a key part of being a product manager. I believe these analyses help PMs better understand their products. In this post I’m going to share a story from 2 and a half years ago of a root cause analysis I worked on when I first joined PagerDuty (all numbers either fake or obscured) and provide a guide for you to use for your own root cause analysis.

I joined the PagerDuty Growth team in September 2022 and our north star metric (Note: see last week’s post on Finding Your North Star) was 30 day conversion rate (the % of customers who started a free trial 30 days ago that converted to paid accounts). Soon after I joined my manager presented me with a problem - conversion rates had been declining for the last 3 months and we needed to figure out why.

Subscribe now

Step 1: Gather Context

The first step in a root cause analysis is to understand more about the problem, asking questions to ensure you have all the background information you need to develop hypotheses.

For me, being brand new to the company and team here were a few questions I asked my manager:

• Can you confirm the free trial flows? (I was still so new!)

• Can you put me in touch with people from Marketing and Data Analysis to talk with?

Step 2: Exploring Hypotheses

Here’s the fun part, where you put on your detective hat and begin exploring hypotheses. When I begin this stage I like to create a document to record everything I’ve learned so that I can eliminate possibilities and build on others.

Here’s a table for how I structure the questions I explore:

In the case of the conversion decline I began working through the questions, starting with the Environment category. Here I discovered that the free trial was predominantly used on web and was not available in PagerDuty’s mobile app and infrequently accessed via mobile web. In terms of Geography, I discovered that the decline is global and was not isolated to a specific geography. Environmental category eliminated.

Next, I began examining possible Internal reasons and uncovered a few clues. While I was able to eliminate bugs, data quality and a change in marketing budgets or traffic patterns, there were product and operational changes that had occurred. A price increase was implemented in April, just before the decline started. Additionally, a product change occurred in August that made a key action - triggering a test incident - became harder for customers. Digging into the impact of test incidents on conversion rates I found that trials that do not trigger an incident convert significantly worse. Another clue.

While I already had two clues, I also took a look at External factors. One major change in 2022 was a decline in the macroeconomic condition. Recall, this was when inflation fears began, interest rates started rising, funding dried up for startups and tech layoffs began. A few Google searches led me to realize that all of these events began in the spring/summer (call it June 2022), shortly after our conversion decline started. After interviewing different members of the marketing and product teams, I realized that most of the customers who use our free trial are startups and small businesses, the same group most likely to be significantly impacted by these macro events.

Here’s an updated table with our findings:

Step 3: Putting it all together

After gathering clues and exploring hypotheses (a process that usually takes days/weeks - not the few short minutes it took you to read those paragraphs) it’s time to put it all together. The process I recommend is returning to your investigation document and writing an executive summary. This can then be copied into email / slack and distributed along with a link to your doc. I also try to help people visualize the analysis.

In the case of our conversion rate decline, I had 3 clues I was able to graph on our conversion rate chart:

1. April price increase

2. June macroeconomic decline

3. August onboarding flow product change that reduced the % of customers who triggered an incident

Conclusion

Once you’ve gathered the data, explored hypotheses, and connected the dots, it’s time to act. In the case of PagerDuty’s conversion rate issue, we identified three key factors: a price increase, a macroeconomic downturn, and a product change that made a critical user action harder.

After identifying the root causes, the next steps were clear. My team focused on improving the onboarding flow and test incident trigger, while the marketing team adjusted targeting strategies to account for the macroeconomic impact on our startup customer base. Cross-functionally we also began reviewing our pricing strategies.

Root cause analysis isn’t just about identifying problems—it’s about creating action plans that can be implemented across teams. Through thorough investigation, collaboration, and experimentation, you can not only identify the cause of a metric decline but also drive targeted improvements to address it.

Steps to Conducting a Root Cause Analysis:

1. Gather Context: Understand the problem by asking key questions and gathering relevant data.

2. Explore Hypotheses: Brainstorm possible causes, categorizing them into internal, external, and technical factors.

3. Dig Into the Data: Analyze data to validate or eliminate hypotheses, looking for patterns or anomalies.

4. Validate External Factors: Consider market or environmental factors that could influence the metric.

5. Put It All Together: Summarize findings, visualize key data points, and present actionable recommendations.

Read more Shipping on Fridays

Last fall, our Incident Analysis team revisited our North Star metric. Here’s how we systematically defined it—and how you can apply this framework to your own team.

Subscribe now

Step 1: Start with Your Mission

A North Star Metric should directly connect to your mission. Our team’s mission is to “help every customer learn from incidents and improve operational efficiency.” This ladders into PagerDuty’s overall mission to “revolutionize operations and build customer trust by anticipating the unexpected in an unpredictable world”.

Step 2: Map the Customer Journey

Next, we turn to our customers—examining their experience, the key moments in their Incident Analysis journey, and the metrics that shape (and indicate!) their success. (Note: for a complete look at how to go about journey mapping see “Escaping the Product Onboarding Trap”).

The Incident Analysis journey starts when incidents are resolved. That’s the moment a customer will choose to start a post incident review. Post incident reviews may also be opened automatically if the customer has created a workflow or other trigger automation. Once a post incident review has been opened customers start working on it, usually by importing data and creating a summary. The next step customers take is to host an incident review meeting with the team to reflect, record take-aways and create action items. All of this is in service of identifying operational gaps and process improvements to ensure the issue (or similar issues) don’t happen again. Finally the post incident review is completed.

Step 3: Identify Metrics

Here are a few metrics that emerge from this journey:

• Monthly active users of post incident reviews

• % of resolved incidents with a post incident review

• Number of post incident reviews opened

  • % opened manually

  • % opened via automation

• Number of post incident reviews started

• % of post incident reviews with a summary or data imported

• % of post incident reviews with action items

• % of post incident reviews completed

Thinking deeper about our customers we also know that the result of a successful post incident review process leads to metrics that include:

• Increased business resiliency

• Reduced risk (especially less likely for a similar incident to occur in the future)

• Cost savings (gained from closing operational gaps exposed by the incident)

Step 4: Choosing the Right North Star Metric

Now it’s time to evaluate our metrics and determine a North Star. A strong North Star metric should:

✅ Reflect customer success

✅ Align with business impact

✅ Be measurable & actionable

We evaluated our brainstormed metrics to find the one that best encapsulates our mission and business goals, bucketing them:

We immediately ruled out User Experience and Hard-to-Measure metrics as our North Star. While valuable for assessing product quality, they don’t directly capture customer impact. Metrics like cost savings or reduced risk, for example, are important outcomes but can only be fully measured by our customers. Our North Star needed to reflect an actionable, leading indicator of success.

After evaluating our options, we chose % of post-incident reviews completed as our North Star Metric. A completed review includes a summary, key learnings, and action items, all of which drive operational improvement. While % of resolved incidents with a post-incident review might indicate adoption, it doesn’t ensure follow-through. Customers who start but don’t complete reviews miss the opportunity to strengthen their operational resilience—a core part of our mission.

Key Takeaways: How to Find Your North Star Metric

1️⃣ Align with your mission – Your NSM should directly reflect customer and business success.
2️⃣ Map the user journey – Identify key touchpoints and metrics at each stage.
3️⃣ Evaluate trade-offs – Choose a metric that’s measurable, actionable, and drives meaningful change.

By applying this framework, any team can define and refine their North Star Metric to stay focused on what matters most.

Read more Shipping on Fridays

In April 2016 Alec Winograd hit the hackweek jackpot at Quizlet, a global learning platform now used by over 60 million students every month for flashcards, quizzes and AI-powered learning. Alec designed and built an in-classroom game during a company hackathon and then led a x-functional team to scale that game, Quizlet Live, to millions of students.

This is the story of how Alec, now VP of Engineering at Oui Therapeutics, brought Quizlet Live into the world. I overlapped with Alec briefly at Quizlet where I worked from early 2017 thru to the end of 2019 leading Monetization. By the time I arrived at Quizlet, Alec’s successful hack project was already a Silicon Valley (or at least Quizlet) legend. I never had the courage to ask Alec about Quizlet Live when we worked together, but we caught up recently via zoom.

Subscribe now

Alex: So, I have to admit, from the time I first heard that Quizlet Live was born in a hackweek I’ve wanted to have this conversation. I’m pumped!

Alec: Me too, this brings back good memories.

So let’s set the scene - what was Quizlet like back in 2015, 2016?

Well it was a lot smaller. I joined straight out of college in 2015. There were about 20 of us and we had about 10 million monthly active users (MAUs). So we were each roughly responsible for a half million users.

Talk about impact!

Seriously. That was a big reason I joined.

So how did Quizlet Live come about?

Every quarter or so Quizlet held a hackathon. It was a full day where you could work on new ideas, no strings attached. Around that time, in-class educational games were picking up steam. Kahoot was getting popular, and we all knew it was something worth exploring.

As part of a senior capstone project, I discovered this game called Space Team.

Space Team?

Yeah, you should give it a try. It's a game meant to be played by a group in the same room. The concept is you're a spaceship crew going through a meteor storm, and you need to perform certain operations to keep the spaceship from crashing and burning.

So no pressure.

None! And to make it even crazier, instructions for one of your teammates pop up on your screen and vice versa. So Space Team is really about how you pull together, communicate and complete the instructions in time before you crash and burn.

Ok, I think I can see how you might apply that to a classroom.

Right! Space Team gave us the kernel of an idea. I loved the idea of applying that collaborative energy to learning. So that’s what we built in one frantic day - a working prototype of a collaborative learning game where students work together in small teams to answer questions from their Quizlet study sets. We focused on just the bare bones of a working game we could demo and play with the company at the end of the hack day. We presented it at the end of the hackathon, and people loved it–it was good, chaotic fun.

So did the leadership team latch onto it right then and there?

Not exactly. The team was working on a bunch of other stuff around that time so there wasn’t really any bandwidth to turn it into a real product. It was a few months later that Quizlet decided to invest in a classroom game and a team was created. But turning a prototype into a real, scalable product was a whole new challenge.

Love hearing about challenges. What were some of the biggest hurdles?

Oh, there were plenty! First, we needed to refine the actual game mechanics. Our PM, Sophia had a lot of experience going into classrooms. She really understood the environment and was attuned to what teachers would and wouldn’t like (since ultimately they would control whether students got to play the game).

Change incoming!

Exactly. She was like “I like this chaotic thing where everyone's shouting random words, but teachers are going to hate it.” So the first challenge was rethinking our game mechanics.

The way we approached this was constant rounds of testing and iteration (Editor’s note, for more on testing see Shipping on Friday’s “A/B Testing in Action”).

Sophia and I developed ideas, tested them with teammates, and refined them based on feedback. Eventually we got to a point where we felt confident enough to go into a classroom and test with real students. This was invaluable for learning. We were able to observe how our target audience interacted with the game. After each classroom visit we synthesized learnings and iterated on the game play, ultimately arriving at the MVP we launched.

I love that approach to building. Learn and iterate. You discussed user experience challenges, did you face any technical challenges?

Yes! The first major challenge was the real time nature of the game. Most web applications follow a simple request-response model—you click a button, and something loads. But with Quizlet Live, we needed data constantly flowing between 20+ student devices, all within the same classroom. With every game move - every new answer right or wrong - data was trying to sync.

All that synching added up and we noticed it would congest classroom wifi networks causing unreliability and slowness. We had to optimize how often devices communicated and how much data we sent at a time.

How did you approach that?

We started thinking through which players needed what information and we realized that in our original architecture we sent all players all the game data including some that they didn’t need and wasn’t even displayed (think info related to other teams). We refined who we sent what, which cut the data flow significantly and solved the majority of our issues.

So once you solved the real-time networking, how did you think about scaling? After all, Quizlet is used by millions.

That was another tricky part. At first, we ran Quizlet Live on a single server. But as more classrooms started using it, we needed multiple servers. Normally, in a large-scale web service, you load-balance across many servers and can scale up and down dynamically. But with Quizlet Live, every student in a classroom had to be on the same server for the game to work.

That sounds like a nightmare.

Yeah, if we didn’t handle it right, students wouldn’t be able to connect with their classmates. Imagine we’re scaling up to an additional server just as your teacher is starting a new game. It would be a disaster. Our solution? Well, when a teacher launched a game, their device would generate a game PIN. We implemented a custom load balancer strategy that used the PIN to consistently route all students in that session to the same game server.

Clever!

Thanks! This solution allowed us to scale and within a year, Quizlet Live had grown to account for about 10% of Quizlet’s total traffic. Scaling that quickly was intense, but super rewarding.

Any fun stories from development?

One of my favorite moments was when we were testing team names. We knew we wanted students to be in teams, but we weren’t sure how to name them. Giving each team a number was just boring.

Someone suggested colors, another person suggested cities, and then someone proposed animals. We liked the animals idea, but were a little hesitant because we weren’t sure how our high school users would respond. Were animal names too childish?

What did you decide?

We tested it, of course! We found that in classrooms the moment the team names appeared—Lions, Tigers, Bears—the students went wild. They absolutely loved it. Any concerns we had about it feeling too juvenile vanished instantly. Students would chant their team names - they were excited to play!

There is a fun Easter egg related to team names. Every 1,000th game, a team gets named “Unicorns.”

That’s amazing.

Yeah, little things like that make a big difference.

That's such a great moment when you realize, you've built something that truly resonates.

Definitely. Though, we also had a humbling moment slash learning experience.

Oh?

A few months after launching, we received feedback from a teacher at a predominantly black school that some students got offended when they were assigned to the “Monkey” team. We realized that our own unconscious bias had missed the historical context. We thanked the teacher for the feedback, removed “Monkey” as a team name, and audited the other animals for similar issues.

That’s a great lesson about putting users first and striving to think from their perspective even when it comes to the most seemingly trivial decisions like choosing animal team names.

Looking back, what’s your biggest takeaway from building Quizlet Live?

I think the biggest lesson is the power of rapid iteration. The hackathon gave us the space to explore a wild idea, but refining it into a great product required constant user feedback and testing. Without Sophia driving that effort and iteration, Quizlet Live would have fallen flat–too unfocused and chaotic.

Also, making learning fun isn’t just a gimmick—it genuinely helps students engage better. Watching Quizlet Live go from a small side project to a major part of the platform was an incredible journey.

And now it’s played in classrooms all over the world. Pretty amazing.

It really is.

And that’s the story of how Quizlet Live was born—from a simple hackathon idea to a global classroom favorite. It’s a great reminder that the best products often come from experimentation, teamwork, and a willingness to learn from real users.

Do you know a Hack project that hit the jackpot? Let me know in the comments! 🚀

Read more Shipping on Fridays

In this edition of Shipping on Fridays I’m going to dive into a/b testing best practices and share 4 common pitfalls using a recent test I ran as a guide (all data obscured or fake).

A quick refresher: what is a/b testing:

A/B testing is a controlled experiment where the team develops a hypothesis based on a customer insight to change the customer experience in some meaningful way and then test that hypothesis by showing a % of the population the original (“A” or control) experience and showing a % of the population the test (“B” or experimental) experience. The team measures whether or not the B experience fulfills the hypothesis (thus becoming a “winner”).

Subscribe now

Our real example: Introducing an Onboarding Wizard

Starting with an insight:

The genesis of any a/b test should be a customer insight. These emerge from a combination of quantitative research (looking at data which shows how customers use your product) and qualitative research (talking to those customers about their experience).

Our insight:

For my team, customer interviews revealed a painful truth: setting up a monitoring tool was extremely difficult for first time users. Customers said:

• “There’s no step-by-step process on how this should work.”

• “This took me hours”

• “[I want] Something that takes your hand and guides you through the process of setting up”

Yet without a monitoring tool set up our product doesn’t work very well. Unsurprisingly, free trial customers without a monitoring tool converted more than 50% lower. Thus, if we could make it easier for customers to complete this set up we could alleviate a major pain point and increase customer conversion.

Developing a hypothesis:

Once you have that insight and have dug in to tease out the deep rooted customer problem, you can begin to develop a hypothesis and possible solutions. Every hypothesis should have 4 components:

1. Who you will impact

2. What you will build or change

3. The specific metric this change will impact (this is your success metric)

4. How much the specific metric will be moved

It’s crucial for your hypothesis to be specific in order to avoid common pitfalls.

⚠️Pitfall 1: The Texas Sharpshooter’s Fallacy

A common pitfall of a/b testing is what’s called the Texas Sharpshooter’s fallacy or moving the goal to fit the story of your test results. If you’ve ever heard someone say “the test wasn’t statistically significant, but it does improve X metric by Y%, so we rolled it out” - they’ve fallen prey to Texas Sharpshooters. When you do this you are shipping an experience you and your team prefer, but not necessarily an experience preferred by customers (see my post recounting a time I fell for this).

Our hypothesis:

Based on the insight we found about the difficulty of integrating monitoring tools we developed this hypothesis:

1. Who: trialists

2. What: make it easy to connect two of our top integrations

3. Metric: % of trialists who actively use integrations within 7 days (called % 7 day Integration Incident)

4. How much: Increase by 20%

Testing the hypothesis:

Often when people think of testing they think the only way is to test is to create a design, turn that design into a software feature (via code) and ship that into production to get a reaction from customers. That’s another pitfall.

⚠️Pitfall 2: You have to test in prod

Testing in prod is expensive! Often it requires the team to prioritize non trivial code changes into a sprint, write that code, test that code, ship that code. Not to mention the PM, Design, Data, PMM and other cycles (such as communicating to leadership) that may be incurred. As a result, I’m a big proponent of starting off by testing the lowest fidelity concept as early as possible (once you have your insight and hypothesis). For instance, have customers walk through a design prototype of the control and test experiences and provide their reaction. This can be done in person or systemically using UserTesting.com (which can get you more scientific results). Testing early also helps ensure that the feature you prioritize for an in-product test (if you prioritize - be ruthless!) is actually a v3 or v4 of the original concept.

There’s another pitfall baked in here:

⚠️Pitfall 3: Everything must be tested

No! Not everything needs to end up as a test in production. As mentioned above, concepts should be tested in front of users, but not all of them need to become a/b tests in production. It’s tempting to build smaller changes (button colors, copy, etc) as simple a/b tests, but all of these tests have hidden costs. For instance, smaller changes might take a very long time to reach statistical significance, tying up your team (or at least a portion of their mental cycles) on changes that may have little to no impact.

Our test:

To improve the 7 day Integration Incident rate by 20% our team developed a new onboarding wizard to allow free trial customers to complete the integration set up directly in the onboarding flow. On the scale of a/b tests (from bug fixes to major features) this was a major feature.

Our testing process started internally, testing different iterations of the wizard with other product teams (especially the teams that managed those integrations and solutions consultants who helped customers set them up). Once we felt our design had passed the internal quality bar we then began to test designs with customers.

We made a habit of starting these conversations with open ended questions about how customers set up integrations and then moved to show them the new prototype. We recorded these meetings and iterated on the design along the way.

After passing these bars, we were ready for the team to begin development. Engineering had been involved from the beginning (in fact our engineers helped concept the design) and so they were already very familiar with the test and it’s two variants:

Setting the bar for Success:

Before implementing the test, it’s important to set the bar for success. Our original hypothesis provides our benchmark (must see a 20% improvement in 7 day Integration Incident rate). Now, it’s crucial to understand traffic and forecast when you should take test reads so that you don’t fall into Pitfall 4.

⚠️Pitfall 4: Calling a test too early

One of the most thrilling aspects of a/b testing is cheering on your experiment variant. It’s like a race. At Priceline I remember we used to watch hotel bookings come in real time to the different test variants - it was addictive. The thrill of being able to call a winner and posting your #winning announcement is so much fun. But to be a true winner, the experiment must reach statistical significance with a certain volume of traffic.

Our success bar:

For our test, I worked with our data scientist to define “exit” criteria or definitions based on our test volume for when we could call a winner:

Reading Results:

Lastly, it’s time to read the results and record the learnings. Ultimately a/b testing is all about learning and building upon what you learn. Whether or not the test is a “winner” will depend on the test’s performance against your success metric, but regardless you will learn (and often you can learn more from tests that fail).

When looking at the results of a test push yourself to go beyond your primary success metric. Work to uncover why the metric moved one way or another (or didn’t move at all). A few questions to consider:

• What surprised you about the result?

• What metrics moved that you didn’t expect to move?

• How did the test impact your product engagement metrics?

• How might the test impact other teams at the company or inform other teams about customer behavior?

Our results:

For our team, the test variant proved the hypothesis true, increasing the % of trialists who triggered incidents via an integration from 12% to 22%, a 90% increase! There were a few other key learnings we had:

1. Customers will complete seemingly difficult tasks in an onboarding flow if they see a possible benefit (and the tasks are discrete). The experiment added a significant number of steps to Onboarding (over 5!), yet over 75% of customers ended up completing the flow.

2. Customer who are introduced to integrations in Onboarding use them. Usage of the integrations did not wane after onboarding - it continued. This indicated our impact wasn’t localized, it lasted with customers.

3. Engaging onboarding reduces tourism. The experiment reduced the percent of tourists (trial accounts that take no action) by more than 20%.

Tips for your own A/B Testing

1. Relentlessly gather insights. Talk with customers regularly. Be up on your data and engagement metrics. Review competitive experiences. Talk with stakeholders about their observations.

2. Form hypotheses and hold yourself accountable to your success metric. Don’t become a Texas Sharpshooter.

3. Ruthlessly prioritize to ensure your test structure (effort) is aligned to your expected output (benefit). Remember there are lots of different ways to test from low effort (UserTesting.com prototypes) to high effort (in production a/b test).

4. Set parameters for success. Don’t call a test too early and be caught with a false positive. Hold yourself and your team accountable to stat sig results.

5. Analyze your results. A/B testing is about learning. A/B testing is about learning. Dig into what happened, write about it, share the writing.

Read more Shipping on Fridays

Drew and I caught up to reflect on how we solved that challenge and provide a framework (and Figma template!) for other product teams facing the same problem.

Subscribe now

The Problem: Misunderstanding the new customer journey

Looking back a few years, PagerDuty had a significant blind spot: we didn’t fully understand the challenges new customers faced or how they experienced our free trial. Sure we watched the data closely, reported on conversion rates and understood which features new customers used, but there were a lot of unknowns.

For instance, as leaders of the Growth Team, Drew and I knew that during the trial customers received marketing messages and sales outreach, but how many messages? At what cadence? How did customers react to these messages?

At the time we didn’t have a holistic understanding of the new customer experience. Likewise, our sales and marketing partners didn’t have a great view into what we were building in product. We were siloed. As we dug deeper and talked with more customers, we realized that without cross-channel visibility we could be mis-prioritizing our improvements. Worse, some initiatives teams launched might inadvertently worsen the user experience (and hurt conversion!).

What we needed was a way to view the trial experience through our customers’ eyes. We decided to approach this problem as storytellers—building a cohesive narrative about our customers, their challenges, and their path to conversion.

The Approach: Storytelling Through a Journey Map

Every story has key components:

• Protagonist: Our customers. Their voice had to drive the narrative. We interviewed over 20 customers and incorporated direct quotes into the storyboard.

• Conflict: The challenges customers faced while navigating the trial. When an interview revealed a challenge we dug in, asking open ended questions to tease out the core problems.

• Resolution: The paths they could take to successfully convert.

Journey Map FigJam Template

Drew built a FigJam template for our storyboard. This is a living document which we still use to this day. Using FigJam has several advantages, namely the ability to collaborate with other stakeholders, which became very helpful later in the process.

This template is designed to be multi-channel: incorporating our web product, mobile app, marketing communications, and outside sources of information (more on that later).

You can download the template here

Creating this template was essential and allowed us to quickly put together a full picture of our customer journey. After an initial pass, we invited members of our marketing and customer success teams to fill in gaps that we did not have visibility into. It’s important to note that this was designed to deliberately be “ugly” in order to encourage participation from those who might be wary of disrupting a finished-looking deliverable.

The Data We Gathered

To bring this story to life, we collected a mix of qualitative and quantitative data:

• Quantitative behavior data: What were trialists clicking? When were they active? Since our storyboard broke the experience down by day, we took a look at timestamps actions were taken.

• Communication Channels: What emails or messages were customers receiving from whom internally? We tracked down screenshots from internal teams and customers. We found that in some cases customers received as many as 5 emails from different PagerDuty orgs (ex: Marketing vs Sales) in a single day.

• Screenshots: Visuals of what users encountered

• Interviews: We conducted in-depth sessions with over 20 trial users. Working in pairs, we took notes and added them as sticky notes in Figma. We also interviewed our Sales, Marketing, Data and other x-functional partners to understand their perspectives.

• Outside sources of information: Likely sources of onboarding information from other places (e.g., YouTube)

We added all of our data to Figma. We met frequently to debrief, come together and tease out themes. For instance, a theme emerged around day 1 backed by both quantitative data (engagement dropped exponentially after day 1) and qualitative data (stories of toil and frustration). We especially looked for themes around challenges. The more we learned the more we were able to tie the themes together into a story.

The Results: Immediate and Long-Term Wins

The finished storyboard became a powerful tool to help stakeholders across PagerDuty understand the experiences of new customers. By creating a fully cross-channel view, we facilitated cross-functional design thinking sessions, walking teams through the storyboard to identify recurring problems and brainstorm targeted solutions.

In several cases, the journey map prompted immediate action. For instance, we discovered that customers received up to five emails from different PagerDuty teams in a single day. Collaborating with a cross-functional team, we consolidated these into a single, clear email, which increased the % of customers who converted within 7 days of opening the email by a whopping 450%.

Beyond these quick wins, the storyboard uncovered key strategic insights. One major finding was the importance of setting up integrations on Day 1. This insight informed subsequent A/B tests, which focused on optimizing the onboarding flow to prioritize integrations. Over time, these changes significantly improved trial completion rates and overall customer satisfaction.

Focusing solely on in-product onboarding is a common pitfall. However, by resisting this temptation and broadening our perspective, we uncovered a richer, more actionable story of the customer journey. Sometimes, the key to driving growth isn’t adding more features—it’s understanding and telling the right story.

Tips for Building Your Own Journey Map

1. Think Like a Storyteller: Treat your customer’s experience as a story with a beginning, middle, and end.

2. Collaborate: Pair up for interviews and group findings collaboratively.

3. Visualize: Use tools like Figma to make the data tangible and shareable.

4. Act Quickly: Address low-hanging fruit immediately, but don’t lose sight of longer-term opportunities.

Read more Shipping on Fridays

With the holidays over, many of us are now playing with new toys. For my daughter (age 7) one of those new toys she can’t put down is the Yoto Player. The Yoto Player (see Wirecutter review) is a screenless device that plays audio books, a kids radio station and podcast. My daughter loves it - it gives her a sense of independence and empowerment.

Who was the Yoto Player built for?

The Yoto Player is aimed at kids aged 3 - 12. While they are the primary users, parents will also be involved. After all, kids at that age are not independent and parents will be pulled into set up, troubleshooting and anything in between.

What problems does it attempt to solve?

1. Self entertainment outside of screen time. It’s easy to park your kids in front of a screen and let YouTube / PBS Kids / Disney function as a babysitter. However all sorts of research (ex UCSF, Mayo Clinic) points to the negative impacts of excessive screen time. Yet many children (our daughter included) struggle to self entertain and (as any parent knows) it is exhausting for a parent to facilitate activities for the entire day.

2. The feeling of a lack of control. For young children, much of life feels like it’s dictated by others. Parents set the schedule, choose meals, decide on clothing, and generally run the show. This lack of agency can lead to the infamous stubbornness of toddlers and young kids, as they seek ways to assert their independence—even in seemingly trivial decisions, like rejecting their favorite grilled cheese simply because it’s been offered. Empowering kids with age-appropriate choices and responsibilities can significantly boost their confidence and reduce these power struggles, but finding tools that encourage this autonomy in a safe and productive way can be a challenge.

How is the Yoto great at solving these problems?

The Yoto Player offers an elegant solution to both problems by providing kids with a device that is entirely their own. Its screen-free design focuses on age-appropriate content, including kids’ radio, podcasts, and audio cards, which eliminates the temptation of adult interference. This makes it a “kid-first” device in every sense. Importantly, the simplicity of the Yoto fosters a sense of ownership and control for the child. They can select which card to play, choose between different audio options, and operate the device independently. This empowerment encourages children to engage with the Yoto proactively, making it a reliable tool for independent entertainment. In my experience, the Yoto has become a favorite because it gives my daughter the autonomy she craves while ensuring her time is spent meaningfully.

There have already been several moments when my daughter’s use of the Yoto Player has both surprised and delighted me. On a recent road trip, she eagerly brought it along and entertained the whole family in the car by playing Roald Dahl’s The Witches through the speaker, holding the device confidently in the backseat. It completely shifted the dynamic of the drive. Instead of being just a passenger, passively listening to the music or entertainment my wife and I chose from the front seat, she became an active participant—an empowered family member taking charge of how to entertain the group. It was a small but meaningful moment that showcased her growing independence and creativity.

Where does the Yoto fall short? How might it be improved?

One notable limitation of the Yoto Player is its relatively closed ecosystem for internet connectivity. While it is Wi-Fi enabled, its functionality is largely limited to downloading a daily kids’ podcast or accessing audio content purchased directly from Yoto. This lack of flexibility can be frustrating for families who already use other platforms for audio content (although the walled garden creates a monetization growth loop for Yoto). For instance, my daughter has a carefully curated playlist on our family Spotify account, filled with her favorite songs. Unfortunately, there’s no way to stream or sync that playlist to the Yoto. The only workaround is to download MP3 files (which isn’t possible with Spotify) and manually transfer them onto a custom Yoto card—a cumbersome and time-intensive process. Similarly, many libraries offer free audiobook collections via apps, but the Yoto doesn’t support these services, limiting its potential as an all-in-one audio solution.

Expanding compatibility to include platforms like Spotify, Audible, or library apps would significantly enhance the Yoto’s versatility while preserving its kid-friendly design. For example, a partnership with Spotify could allow parents to designate specific playlists in the app for Yoto access. Kids could then use the Yoto’s intuitive knobs or future voice commands to play their playlists, empowering them to make independent choices while keeping the process safe and simple.

Another area for growth is the Yoto’s ability to evolve with children as they grow. While the closed ecosystem is perfect for younger kids, as children mature and gain independence, they may require more flexible functionality. For example, Amazon Alexa Kids allows parents to gradually scale up access and capabilities based on a child’s readiness. A similar feature for Yoto would enable it to remain relevant for older kids while maintaining parental control, ensuring it continues to grow with families over time.

These improvements would not only make the Yoto more functional but also expand its total addressable market (TAM). By integrating more diverse content options and providing scalable features, Yoto could become an even more valuable tool for families. It would better support parents looking to enrich their children’s listening experiences while preserving the screen-free, empowering design that makes it unique.

Wrap up

The Yoto Player has quickly become a beloved part of our household, offering a refreshing alternative to screen-based entertainment while empowering my daughter with a sense of independence. It’s not without its limitations—such as the lack of integration with external content sources—but its ability to engage kids meaningfully makes it a worthwhile investment for parents of younger children seeking balance in their children's media consumption. As my daughter continues to explore her library of Yoto cards, I find myself appreciating it - and of course brainstorming how the product might grow and improve :)

Are there any new toys that have you captivated?

Read more Shipping on Fridays