Usability test: Does iPhone match the hype?

We all know that in the technology world, the hype about new products often doesn't match reality. So it's fair to ask: Is the iPhone as good as its hype? In particular, does iPhone's much-discussed touch-screen interface really make using the device simpler and more intuitive?

Everybody will have an opinion, but what's need is something more objective and definitive. So an expert in the field -- Perceptive Sciences, an Austin, Texas-based usability consulting firm -- was asked to examine and compare the iPhone and two competitors.

The results of its tests were unequivocal: While the iPhone is not the most feature-rich device, this group of experts found that when it comes to usability, iPhone does, indeed, live up to its hype.


The phones

Besides iPhone, the two other products in this usability comparison test were selected for two reasons: They were available, and they had competitive feature sets. In particular, the testers needed a touch-screen phone to compare to the iPhone and a more traditional button-based phone with strong multimedia capabilities.

For now, there are few touch-screen devices available. One that has received a fair amount of publicity is the LG Prada, which is not yet available from a U.S. cellular carrier. LG declined to participate in these tests.

Timothy Ballew, seated, and Tom Thornton of Perceptive Science observe as tester Mindy Cambel tries out a smart phone.Another is the HTC Touch, a Windows Mobile device that has both standard button-based navigation and touch-screen capabilities. Like the Prada, it isn't offered yet by U.S. carriers, but HTC still agreed to participate in this test. This device is based on the Windows Mobile 6.0 platform, has a 2.8-in., 240-by-320 resolution display and a 2-megapixel camera. It supports both Wi-Fi and Bluetooth.

While the HTC Touch currently isn't available from a U.S. cellular carrier, the company has indicated that it will be before the end of the year. An unlocked GSM version of the Touch is currently available from numerous resellers for between $450 and $600.

Nokia's N95, based on the Series 60 variant of the Symbian platform, provides only the more traditional type of button-based navigation, but it is a multimedia powerhouse. It boasts a 5-megapixel camera and can create VGA-quality, 30-frames-per-second videos. It also supports many types of media playback and has a long list of other features including built-in Wi-Fi, Bluetooth and GPS, not to mention a bar-code reader that came with the test device. It has a 2.6-in., 320-by-240 resolution display.

Unlocked versions of the Nokia N95 are available for between $600 and $700.

Obviously, iPhone is the best known of the three devices. It has received much attention for its 3.5-in., 480-by-320 resolution display screen and its touch-screen interface, in which you use finger gestures for virtually all tasks. Based on the Mac OS X, it comes with a built-in 2-megapixel camera.

The tests

Perceptive Sciences designed this test to be as objective as possible, according to senior research scientist Tom Thornton and research scientist Tim Ballew. That's particularly important, they said, because of the high level of attention iPhone has received; it would be easy for that hype to influence the results of more subjective tests.

The company brought in 10 testers who had never used any of the three devices. It then asked the testers to perform a series of tasks on each device with quantifiable results, such as the time needed to find and use the on/off switch. Other tasks included setting the phone to vibrate, making a call, saving a phone number to the contact list, sending a brief e-mail, taking a photo and finding a Web site using the device's built-in browser.

Based on the test results and on Thornton's and Ballew's observations, each phone was given a score of between one and five (five being the highest) in each of five categories. In addition, each phone was given an overall score.

It's important to remember that these are usability tests, not tests of functionality. Perceptive Sciences took a broad look at the features on each phone, but largely as they related to usability. For instance, the Nokia N95 is justly famous for its strong feature set. But did that feature set contribute to overall usability, or detract from it?

It's also important to remember that the tests focused on how easy it was to pick up the device and use it right out of the box.

"People can eventually learn to use any device," Ballew said. "But that's not true usability. We wanted to see how long it took to figure out how to use the phones. That's the difference between learnability and usability."

The results

Let's cut to the bottom line: In terms of usability, iPhone blew away its two competitors. Its overall score in the usability tests was 4.6 out of 5. The HTC Touch was a distant second at 3.4, and the Nokia N95 scored 3.2.

"Testers were [typically] about twice as fast doing specific tasks on the iPhone, which is pretty remarkable," Thornton said.

Here is a breakdown of how each device scored in the five sometimes-overlapping categories, along with comments from Ballew and Thornton.

Global navigation

Apple iPhone 5

HTC Touch 4

Nokia N95 2.5

Global navigation refers to how simple it is to navigate through the operating system and how easy it is to find and select specific applications. The clear winner in this category was the iPhone. One reason for iPhone's high marks is its simplicity, Ballew noted.

It has one top-level menu, and it has only one button, which lessens the learning curve," he said.

The HTC Touch did reasonably well, in part because, as a Windows Mobile device, it looks and acts somewhat like desktop Windows, which many people are familiar with. People can choose between standard Windows Mobile navigation and using its touch-screen capabilities.

The Nokia N95, however, suffered from too much complexity, even for basic tasks.

"People had a hard time turning it on," Thornton said. "Only 30% of our sample group could turn it on right away."

Once turned on, navigation was confusing to some users.

"You can access different menus from different locations, which can be disorienting," Ballew said. "And common features are hard to find -- turning the sound down, putting it on vibrate, things like that."

Usability/Information Architecture (IA)

Apple iPhone 5

HTC Touch 3

Nokia N95 2

Usability/IA is similar to global navigation, but it more specifically refers to how easy and fun the interface is. This category also includes such issues as how clearly icons are labeled and how easy it is to find applications and specific files. Once again, the iPhone received perfect marks.

"iPhone's [touch-screen] feature makes it fun to use," Ballew said. "Plus, the screen layout is simple and intuitive, and most of the labels are clear. And the file structure is transparent -- you don't see a list of files unless you go into iPod functionality, and even then, a lot of people are familiar with iPod functionality."

The HTC Touch, by contrast, was confusing to some users.

"It has a bunch of icons that are small, hard to see and not intuitive," Thornton said. "And it was hard for people to see the keyboard and to type. Also, the screen isn't brilliant. We had one [tester] who had had her eyes dilated a few hours earlier -- we kept her in the test because she represented a person with imperfect vision. She had trouble with the HTC Touch but not with the iPhone."

Some testers even had trouble making a phone call with the HTC Touch, Ballew said. "That's pretty basic functionality, but we had people who couldn't complete a call at all."

While the Touch had some rocky aspects in this category, the Nokia N95 was almost uniformly difficult to use for beginners.

"Some of the button functionality and labels weren't intuitive," Ballew said. "And some external stuff like the power button being hard to find and use, and the camera not functioning as expected, really hurt it."

Ergonomics
Apple iPhone 4.5

HTC Touch 3.5

Nokia N95 3

Ergonomics refers to such issues as size, weight and shape, as well as the placement of switches and buttons and the quality of the display. Once again, iPhone led the way in this category but not as decisively as in the previous two categories.

"The iPhone is nice and thin, although it's a bit large," Ballew said.

A minor issue is that there's no tactile feedback from the iPhone screen, he added. "People are used to keyboard click, and it wasn't there," Ballew said.

Overall, the HTC Touch scored well in this category, but with some caveats. "There are a limited number of buttons, which is a good thing," Ballew said. "The screen is pretty small, though, and the display isn't that good."

The Nokia N95 has a solid, often satisfying feel, the testers said.

"It's fat, but it's pretty light, and it does have a full-fledged camera," Ballew said. "The display is nice -- better than the HTC, although not as good as the iPhone."

Look and feel

Apple iPhone 5

HTC Touch 3

Nokia N95 4.5

Look and feel refers to subjective issues such as how the device looks, how pleasing various graphical design elements are and its color and style. Once again, the iPhone received perfect marks.

"It's clean and plain, but it has that large screen, which increases the appeal," Ballew said. "It has a minimalist design, which, interestingly, made people feel it was a more high-end phone."

The Nokia N95 didn't trail far behind the iPhone in terms of look and feel.

"The outside look of the Nokia is very nice," Ballew said. "And incorporating a full-fledged camera makes it seem high end. On the inside, it has nice animated icons. So the look and feel on both the outside and inside is pretty nice."

The HTC Touch suffered a bit because its screens were sometimes confusing.

"The home page is different than the start menu, which is confusing," Thornton said. This problem is shared by virtually all Windows Mobile devices, he added.

Functionality
Apple iPhone 3.5

HTC Touch 3.5

Nokia N95 4.5

Functionality refers to what some consider the meat and potatoes of the device -- its applications and how complex and customizable they are. It also refers to the quality of subsystems within the device, such as the camera. In this area, the Nokia was the clear leader.

"It has a really nice feature set," Ballew said. On the other hand, he stressed, its strong feature set contributed to its relatively poor usability scores in previous categories.

"It's right on the verge of feature bloat," he said. "I mean, I'm not sure when I'd ever use the bar-code scanner. And some of the features are hard to set up." In particular, Ballew said it took four hours to set up Wi-Fi on the N95, which was a fast, simple task on both the HTC Touch and the iPhone.

The HTC Touch did well in this category, as do most Windows Mobile devices.

"You can easily add new applications, new widgets," Ballew said. "The screen quality probably decreased the ability to use some of the functionality, though."

By contrast, this is one area in which the iPhone did not excel.

"It has really basic functionality," Ballew said. "For example, the camera functionality is pretty basic. We're starting to see more third-party apps, but they're Web-based, and some aren't very good."

However, Thornton noted that some of the functionality the iPhone did have was extremely well implemented. That was particularly true with the ability to use the Safari browser to see a whole Web page on-screen, then to zoom in on what you specifically wanted to see.

"People were faster and more successful in getting to a Web page with iPhone," Thornton said.

The bottom line in this category is that there often are trade-offs between the feature-richness and usability, Thornton said.

source