Beware of Poorly Formed Inferences

MarketingSherpa, a great resource for online marketing case studies, recently published a nifty article showing the thinking behind the recent redesign of CareerBuilder.com’s Job Posting home page. (Note: The MarketingSherpa article is only available for free viewing until September 4. After that, you’ll have to pay to see it.)

According to the article, Michael DeHaven, who is in charge of the site’s ecommerce marketing efforts, used A/B testing to try out two different versions of the new design.

Version “B” looked like this:

Version “C” looked like this:

What it came down to was this (from Version “B”):

versus this (from Version “C”):

The article reports how everybody on Mr. DeHaven’s staff who saw version “C” said it was too crowded and people would feel overwhelmed with links, so they expected version “B” to win the A/B Test. As we would’ve predicted based on our work with link-rich home pages, version “C” should do better.

And we were right. Version “C” won more customer accounts than “the cleaner, more graphical design”.

But what really interested me about the article was the reason that Mr. DeHaven gave for the surprise success of version “C”:

It seems that business executives prefer to look at fairly plain textual content online rather than cheerful graphical interfaces. Plus, they prefer vertical to horizontal groupings of options and longer, wordier textual click links.

“After I thought about it awhile, it made total sense,” explains DeHaven. “Users are trained to allow their eye to scan down something that looks like search results. And what we have here now is something that looks an awful lot like search results. It’s a good quick scan. Graphics and images are not what the eye’s trained for online.

“It was a HUGE finding. Really exciting.”

Wow! What an inference! Mr. DeHaven got the observation right — people bought more when presented with version “C” — but I don’t agree with his inference at all.

Jumping to poorly formed inferences can get us into trouble. Now that he believes that (a) people prefer things that look like search results and (b) users’ eyes’ are not trained for graphics or images, he’s likely to make design decisions based on those beliefs. And, while many of the resulting designs are likely to work, many probably won’t.

Part of the problem comes from exclusively using tools like A/B tests. A/B tests tell you that one version performs better than another, but doesn’t tell you why. So, the designer has to guess at why. And, in my experience, they often guess wrong.

It’s not just A/B tests that do this. Inspecting your web logs will produce the same results. You’ll see a bunch of folks are ending up on a certain page. But you can’t tell if they are there because they want to be or because they are lost. You have to guess as to why they end up there. And you’re likely to guess wrong.

Had Mr. DeHaven conducted usability tests or interviewed users, in addition to the A/B tests, he probably would’ve come to a different conclusion about why users prefer the page with the extra links. I’m guessing, based on our research, he would’ve found out the users found the pages to have more trigger words in the right places.

Whatever he would find out, it is likely he’d be more informed about the decisions he was making going forward. And, in the long run, he’d have more information to work with when making future decisions about his designs.