The reality is a bit more complicated. Amazon is not abandoning Just Walk Out technology. In fact, the company says it’s expanding the number of places where it’s available—it’s just focusing on different kinds of venues like airports and sports stadiums. And in grocery stores, Amazon is replacing Just Walk Out technology with fleets of tech-enabled “Dash” shopping carts that include item scanners, touch screens that display nearby items, and automatic billing that also allows for skipping the checkout line.
Still, the fact that Amazon is rolling back Just Walk Out technology in its U.S. grocery stores (U.K. grocery stores will continue to use the technology as of now) is clearly a black eye for Amazon and a setback to its ambitions. Just Walk Out technology was at the heart of the company’s plan to expand into the physical shopping realm by becoming a transformational leader in the business, not just another player.
To better understand what the grocery store pullback means for the future of Just Walk Out technology, and how it affects Amazon’s overall physical store strategy, Fortune sat down with Dilip Kumar, an Amazon VP who was one of the inventors of Amazon’s cashierless technology. Kumar, a former “shadow”—or chief of staff—to Jeff Bezos, described the company’s current strategy, and also set the record straight about how much humans are really involved behind the scenes to make Just Walk Out technology work.
And so what we’re seeing in third-party, small-format stores is that anywhere where there are throughput constraints, wherever people are time-constrained—like stadiums, airports, convention centers, universities, in hospitals—it really is resonating. [Amazon has sold the technology to around 140 third-party convenience stores and concession stands worldwide.]
We’re going to be doubling the number of stores this year. I feel like when you tell [businesses] about Just Walk Out it feels like a leap of faith for people to hear, “Rip out your point-of-sale system and [use] this completely different customer experience, and trust us, it’s going to work.” And so people start with one store or two, and then they come back and they see that it’s doubling or tripling their sales.
Like at [Seattle’s] Lumen Field. They started with two stores, and now they have nine. In [Chicago’s] United Center, they started with two stores, and now they have seven. Hudson [airport convenience stores] started with a store, and now they have more than a dozen.
And so I think one of the things that they’re seeing is, it is the single best thing that they can do that doubles, triples their throughput and the number of transactions. But they also find that it is not cannibalistic even if you put multiple locations in the same [building].
Why wasn’t the technology a fit for the larger Amazon Fresh stores?
When you think about your grocery trip, it’s not just time that is important. What people also want is the notion of how much are they spending? Where are the items located? Can I get this real-time receipt or tally of knowing exactly what’s going into my cart? Are my coupons being applied? These are a bunch of other things that also become just as important.
And by the way, a lot of these things increase in importance as the size of your trip increases. So as the total amount of dollars that you’re spending increases, as the total amount of time that you’re spending increases, all of these other elements also become important. And for those reasons, we felt like the Dash Cart is a much better platform for experimentation and continued invention in large-format grocery stores rather than Just Walk Out.
Is that your decision? Or is that [Amazon grocery store SVP] Tony Hoggett’s decision?
It was Tony and I. [His team] is in the process of changing how the Amazon Fresh stores are getting designed. They’re sort of moving things around. They’re sort of saying, “We need certain things in a certain way.” And when you try to do two things [simultaneously] that are constantly evolving, it actually slows a bunch of things down.
The cart still allows you to be able to skip the line, but it also has a bunch of these other benefits that are important while you are shopping that customers were telling us about.
It also allows the store to be able to make changes with the level that they needed to. [An Amazon spokesman later clarified that the changes to Amazon Fresh stores include adjustments to store layouts and product selection that would require significant retraining of the Just Walk Out technology, but less so for Dash Cart.]
So for a variety of reasons, it was the right decision for large-format stores.
I was still a little surprised by this. I understand all the things you’re telling me that customers have told you, but how much of a factor was the cost differential between the carts and the Just Walk Out system?
The carts are always going to be lower cost than any Just Walk Out system … because one scales with the number of shoppers and the other scales based on the square footage that it has to cover.
But … we felt like there’s a lot of information that happens during the trip, which is relevant, that the carts can provide. With the carts, there’s a little bit more of a human interaction [necessary], but one of the things that we see is that the cart will eventually get to the place … to being no different than just picking up a traditional cart. Plus there is the augmentation of a bunch of things that are relevant while the person is in the store shopping [compared] to the current Just Walk out experience, where it is completely devoid of any feedback, or any helpful information, until you get [the receipt].
I’m sure you’ve seen the test run that GeekWire reporters went through and some of the complaints about the Dash Cart experience, or just things that didn’t go as smoothly as planned. Do you think that’s sort of normal with where you’re at with the cart experience?
I think their experiences are not what we see from the vast majority of the customer satisfaction scores that we get. We have a 98% customer satisfaction score for our Dash Carts and 80% of Dash Cart shoppers are repeat shoppers.
Having said that, it’s always useful to acknowledge what folks went through in their experience, things that we could do better. There’s a bunch of things in the works which will continue to make that experience more and more intuitive for shoppers. Today we require people to use an app to be able to use the cart. But, you know, when Amazon Go started, the only way to enter the store was using an app and a QR code in the app. But since then we said you can use Amazon One [palm scanning], you can use a credit card to enter, you can even use your badge.
A lot of those kinds of things will happen to the cart as well. A lot of the things like, how intuitive is it, how easy is it to add or remove products. There’ll be plenty of improvements made in that dimension, which I’m already seeing in the next generation. So I’m reasonably confident that we will get closer and closer to the experience where people find these to be invaluable. We’re already seeing that for the subset of customers who use it. But it will become more ubiquitous. Amazon Fresh is going to [continue] rolling these out; we’re going to be launching these with several third-party grocers as well. So we’re getting good feedback from folks, but also things that we know that we have to get better at and make intuitive for the first-time user.
Obviously a big storyline that got distorted over the past couple of weeks was about the team behind the scenes helping to ensure accuracy and also to help train the Just Walk Out technology through labeling and other means. Years ago, I reported before your first Amazon Go store opened to the public that there were some humans working behind the scenes. Can you just clarify what is and isn’t happening with this staff now?
Humans built this technology. Humans also annotate and label some of these [videos of customers shopping] in order for our algorithms to be able to learn and get trained. We are also generating a lot of synthetic data to train our algorithms. So the combination of this has always been consistent.
There were several articles about “Oh, people are actually watching these videos in real time.” It was just sort of beyond ludicrous. But it’s also a good point to be able to clarify for folks who are not as involved in this technology and don’t realize how some of these things work.
What humans actually are doing behind the scenes is very useful. All of these algorithms need a little bit of training data in order to be able to generalize. And so when we’re thinking about this, staff are annotating videos, or different video frames, or figuring out who took what, and that requires a series of annotations. And so the humans are actually helping with that aspect of it.
So that’s one thing they are doing.
But one of the other things is that the system that is generating the output can’t audit itself. So humans actually sample a small sample of videos, in order to ascertain the accuracy [before Amazon provides a Just Walk Out customer with a receipt].
Machine learning algorithms work on confidence boundaries, like, “How confident are you that something happened?” And so as the confidence gets lower and lower to a certain threshold, if it’s below, maybe somebody should verify it. But even the things that are really close to that threshold, those are the ones that we would subsample. And we would say, “What made us get so close? What would make it more confident? What would make us way more confident that X took Y product or this happened?”
Those particular sessions are the kinds of things that humans are annotating. It is a very small percentage of these videos that humans actually review as part of the receipt [process]. The vast majority [of receipts] go in on an automated basis.
So are you willing to share what percentage of receipts need to be reviewed?
We don’t share those numbers. It’s a small portion of the overall number of transactions that get reviewed by humans as a percentage of receipts.
There was a report that in 2022 humans were involved in 700 of every 1,000 Just Walk Out sessions. Is that true?
In the earlier days when we were creating the large format stores, we tagged and annotated a significant portion of those initial transactions, because it was super important for us to be able to build the algorithms necessary for that. So yes, we did it.
But it wasn’t in real time; it wasn’t while the receipt was being generated. What we wanted to do is to be able to tag as many of those sessions as possible. We would tag a lot of sessions, we would then run our algorithms, and then know where there were gaps … refine it to the next set of things that we didn’t have enough information on … and lather, rinse, repeat. So yes, we tagged different types of transactions in different quantities, in order to be able to make sure that our algorithms could generalize.
But like 700 out of 1,000 for receipts? No. It was probably a combination. I don’t know if it was 700, that sounds high, but we tagged a sufficient amount.
This story was originally featured on Fortune.com