The Back Story for the $300 Million Button

By far, The $300 Million Button is the most popular article on UIE.com. Here’s the back story for how we discovered the problem and the role that analytics played:

We had been working on a client project, helping their team redesign their checkout process with some new user research and design techniques.

As we were watching seasoned shoppers buy products in the lab, we noticed that people were getting stuck at a screen right before the checkout process, where they had to authenticate their account. Repeat customers couldn’t remember their user ID and password combination. The site used an email address as the user ID, but many of our repeat customers had set up their accounts years before and couldn’t remember which of their email addresses was the ID.

In the lab, the customers who couldn’t authenticate would either give up or request a password reset. However, the password reset required they remember which email address was their ID, which many couldn’t do. We witnessed a remarkable number of abandonments on the password reset screen.

When the password reset was successful, the customer had to go to their email client, find the reset message (often lost in a spam folder), and click on a link in the email. In the lab, we observed that this was a complex process.

All of this led us to ask if this was only happening in the artificial environment of the lab, because we were watching users in our space, not using their own machine. We set out to look at the site’s analytics to see if there were clues to this behavior happening in the real world.

The first thing we asked the analytics team was what percentage of visitors to the authentication page ended up on the reset password request screen. Turns out, they had never instrumented either page. We had to wait three weeks while they instrumented it and we collected a reasonable sample size.

We learned a substantial percentage of customers were requesting password reset, approximately 40%. Two out of every five users was getting stuck and needing their password to be reset.

We then wondered what percentage of those people actually came back to finish the transaction after the reset. Again, we discovered the analytics team hadn’t instrumented the return from the reset. That was another three week delay.

We learned that fewer than 25% of the resets were executed — the user clicked on the reset link and returned to the site. Of those who did execute it, fewer than 20% finished their purchases.

A little math and we could calculate out the amount of revenue being abandoned in the carts by all the people who couldn’t authenticate. That’s where the $300,000,000/year number came from.

Once the team implemented a guest purchase capability (which didn’t require authentication to start the checkout sequence), they saw an immediate jump in sales increase of about $6,000,000 in the first week, which remained constant. Password reset requests dropped by about 80% in that first week and remained constant too.

Authentication pages are usually owned by a different group in the company. In this case, they were owned by IT. IT didn’t have the foresight to instrument these pages.

Until we did this research and asked these questions, nobody knew how many people were dropping off at authentication. Because authentication was between the shopping-cart and the Enter-your-shipping-info pages, everyone thought they were getting a much lower percentage of users clicking the Checkout button than they really were. The site had a huge abandonment on authentication that heretofore had gone undetected.

Analytics only work when we know that they are measuring everything correctly. Working with clients, we regularly discover they aren’t capturing the entire picture, leaving out critical information.