Behind the scenes: MVP-style testing of Priceline’s Pricebreakers with Dan Sternfeld

How an empowered product team ran a scrappy test that changed Priceline

Priceline.com, one of the original dot-com boom companies, is famous among travel deal hunters for innovations to help travelers save money. For many years, its marquee feature was its ability to offer deep savings through the ability to bid on hotel rooms where the hotel itself isn’t revealed until after you’ve paid (that was called Name Your Own Price - see article describing how it worked). A few years ago, Name Your Own Price was dethroned by Pricebreakers in response to changing customer and business needs.

This is the story of how Pricebreakers came to be. I worked at Priceline from 2013-2015 with Dan Sternfeld, an engineering leader and Priceline veteran who joined in 2005. We caught up recently and he shared the tale of the original Pricebreakers MVP including its technical and design challenges.

Alex: Set the scene for us, Dan. What problems were you trying to solve?

Dan: First, and as you well know, while Name Your Own Price was fun -

Yes, I remember the fun stories of travelers getting thrills when their bid was accepted.

Yes, while it generated excitement, only around 15-20% of people were willing to gamble and bid on a hotel. This limited the company's reach and constrained our potential for growth. And this number was only decreasing - traveler preferences were changing. More people wanted assurances about where they’d be staying. Additionally, Name Your Own Price was built on a legacy code stack and was getting very hard to maintain.

So there was a business reason, a customer reason and a technical reason to all move away from Name Your Own Price.

Ok, so the stage is set for change. How’d you make it happen?

We formed a tiger team - PM, Design and front end only engineers. We wanted this to be a true MVP - shipping just enough of a product to tell us if we had signal to invest more. We purposefully kept the team small and worked hard so that we didn’t have to change anything on the backend.

Oh wow. So that sounds fairly limiting. What do you mean by build MVP-style?

Well this had to be an a/b test - it was a true experiment - could we create a product that had the conversion and raw hotel booking numbers (our KPIs) to displace Name Your Own Price - a product that had been around since 1998?

A challenge.

So we came up with this concept of showing 3 hotels - including the names and locations and the savings (screenshot of the product today below). And we’d show this right on the hotel listings (search results). But this introduced a bunch of technical challenges.

Pricebreakers as seen 12/12/24 - which hotel do you hope it is?

Did you have a way to integrate this sort of multi-hotel card into the listings screen?

Well, as the saying goes, we had 99 problems, but that wasn’t one. Luckily we’d introduced different types of hotel cards into listings before. There were other problems:

Ok.

So the first problem was around backend response time. This single card pulled information from different sources and required multiple API calls. This meant that our Pricebreaker card would be slower to load than the rest of the page. And because it was slower and the design of the card was slightly taller, we had a trade off decision to make.

Oh, so since the card was a different size and it loaded last the page might jump slightly?

Bingo! I know we all want to keep Cumulative Layout Shift (CLS) at 0 [Alex’s note: CLS is a measurement of how much a webpage’s layout shifts while loading. Google uses it to help rank search results, favoring lower scores]. So we had some choices. Right? We could put a placeholder card that would prevent any shift. We could move Pricebreakers further down the listing, so you don't see it till you scroll, but not that many people actually scroll. Or we could delay the full paint [ie the loading] of the listing page until this card was ready.

Really fascinating trade-offs here. What’d you decide?

So we first eliminated some choices. Delaying the full paint was a non-starter with the business - time to Largest Contentful Paint (LCP) is money, especially with travelers shopping on multiple browser tabs [Alex’s note: LCP is another metric Google uses to help rank search results. LCP measures how long it takes a website to display the largest content on the screen ready for interaction]. Since the whole point of the MVP was to get validation on this concept we had to promote this new product, we didn’t want to stick it at the bottom. The test had to elicit a customer reaction one way or another and to get that reaction - and a statistical test read - we needed Pricebreakers to be visible. So we said, all right, we can deal with a little bit of layout shift right? The layout shift only happened about a fraction of a second after the page was loaded and it is small - not a big jump in content.

Since there were no backend folks on your team was there any problem actually getting the data you needed to create Pricebreakers?

Yes, we couldn't really mess with the inventory in too many ways, and we had to be careful not to disclose anything that would give away which hotel was actually the chosen hotel, to maintain the “opacity” of the product. So we couldn't just call the API that tells you a bunch of stuff from the front end. In the end we utilized GraphQL to orchestrate data and carefully manage the flow of information.

You mentioned design challenges also?

Right, how do you get customers to realize that they’re looking at a set of three candidates and they’re getting one of these. And maybe it's not the one that you're looking at right now. We used all the evidence from years of hotel a/b testing to help us. We knew we needed to give the customer reassurance and positive brand association - all without giving away the game and revealing which of the three hotels they’d get or misleading them into thinking that by clicking on 1 hotel they were getting that hotel.

So we opted for a carousel layout where the customer could look over each hotel, and if they wanted to they could go buy retail, but we emphasized in the header, on the image, even in the CTA that they’d get 1 of these 3 hotels for the steep discount.

Very neat. What a cool innovation to bring to life.

Yes, ultimately PriceBreakers proved to be a conversion lift and we eventually sunset Name Your Own Price.

I love how this story illustrates how a small, empowered team with limited resources tested a brand new concept and introduced a new business to Priceline. Thank you for sharing this story, Dan!