Determining Usability from Analytics
Hats off to Ziya for pointing me to MapSurface this morning. It’s an interesting analytics tool, competing for the same mindshare as a plethora of other tools, such as Google Analytics and even MeasureMap.
MapSurface’s interesting take is they report the results by layering it over the existing page using some very clever AJAX code. Nicely done. Seeing the link report mapped out on top of the page adds clarity that isn’t afforded by just a tabular report of the same information.
However, like all of today’s available analytics tools, it really can’t be too helpful in measuring whether the page or site is usable or not.
As I’m sure you’re aware, analytics can only measure what users do. They can’t measure what users are trying to do.
To assess the usability of a page or site, you need to know two things: (1) What the user wants to accomplish and (2) whether they accomplished it or not. Today’s analytics tools can’t tell you either of these things, making it impossible to use them to assess usability.
For example, in 1996, our observational studies determined about 1 out of every 7 people who tried to find a hotel in the Walt Disney World theme park at Disney.com accidentally looked in they Disneyland theme park without realizing it. It’s still pretty much that way today. Would MapSurface (or any other analytics tool) point out that 14% of users were lost or confused? While it’s obvious in our studies, it’s impossible to determine from inspecting the analytic reports.
Tools like this can help support observational data. For example, if we see certain behaviors in our studies, we can turn to the analytics to see if confirming patterns exist. However, that’s tricky since we can’t control the sampling process. In the Disney example, there’s no way to only look at the click patterns of people intending to stay in a WDW hotel versus everyone else who comes to the site.
Someday, I believe we’ll see tools that can help with these problems. We’re just not there today.
[If this is a topic that interests you, I highly recommend you check out what’s happening at the upcoming E-Metrics Summits. Jim Sterne’s put together a great lineup this year. I wish I could go.]