So… Does the iPhone live up to its hype?

Summary
At last. The usability test of the iPhone! Without doubt, the hype around the iPhone has managed to exceed most consumer devices in modern time. But… will it live up to the hype?

We let some of our best usability experts run an inUse comparative usability test with four phones and five users. It is easy to fall in love with something so beautiful and sexy, but if the device is not easy to use the initial euphoria will quickly turn into despair and frustration.

The results? Stunning. The iPhone has introduced a new interaction paradigm to the world, in an uncompromising way that proves that “less is more” when it comes to true user experience.
Darja Isaksson, CEO inUse

Without doubt, the hype around the iPhone has managed to exceed most consumer devices in modern time. Rumors, information leaks and patent applications along with total silence from Apple led to a fanatic guessing-game that spread like wildfire on the internet. There is a lot to be said on how Apple managed to pull it off, how much was intentional and how much was accidental, but from a user-perspective we are mostly interested in why people care in the first place.

What is the hype?
A colleague was taking pictures in La Sagrada Familia in Barcelona using his iPhone when he was approached by a stranger who recognized the device, asking to have a look. After playing around with it for a moment he concluded that ”We will not get it here in Spain for a long time. You are very lucky!”.

This is just one of many similar stories of people coming up to us and asking to see and try out the phone and so the question was born: What is it that people expect from the iPhone? What is it they think it will do for them, and how do they expect it to make them happier? Apple attempts to position themselves as a company with a strong focus on user experience. Apple personal computers may or may not be better or more usable than the competitors, they are essentially used in the same way and the differences are only skin deep. The area where Apple really stands out is with the iTunes store and the iPod. The iPod changed the way we access and enjoy music, iTunes changed how we purchase it and in the end the whole music industry. The iPod is a phenomenon. Apple made a difference in our daily lives. We suspect that the fascination for the iPhone that we have seen in people’s faces carries the expectation that the iPhone will simplify and enhance their daily “phoning” life.

That people expect the iPhone to change how we communicate in the same way the iPod has already changed our relation to music. If this is the case, how does the iPhone live up to such incredible expectations? In short, does the iPhone live up to the hype?

So… Does the iPhone live up to its hype?

The test
We decided to do a usability test1 to determine if the iPhone will meet people’s expectations and live up to its hype. Simply having people trying out the iPhone, and then asking them if they liked it or not, would not make much of a test. In order to see how the iPhone really worked in action, and how well that corresponds to people’s expectations of it, we had five people carry out a set of tasks using the iPhone as well as with three other devices from leading competing brands.

We selected what we consider to be simple everyday tasks that are crucial to the overall user experience of each device. Use of mobile phones is typically characterized by carrying out the same or similar sequences of actions over and over; calling someone from the contacts list and reading new text messages may be repeated every hour of every day. The specific tasks were chosen because we consider them to be key use cases that determine the ease-of-use of a mobile phone, as well as showing the different interaction methods used in the iPhone. Five people with different usage patterns when it comes to mobile phones were invited to perform the test. Each person was given five minutes to play around with each device before the actual test started. The tasks that the users were asked to perform were: make a call by dialing a number manually and then by calling a person from the address book, change volume during a call add a new contact to the address book, create a new calendar event, set the alarm, lock and unlock the device, compose and send a text message, compose and send a text message, put the phone in silent mode, and take a picture and send it to a person in the address book.

A test leader and an observer were present throughout the test session and one person performed the tasks using each phone. We have developed a refined technique to support the user and, at the same time, carry out an interview that reveals the user´s understanding of the product and it´s features. It is important to differentiate between how well people actually succeed with what they are doing, and what they think of the product or system. If the device is not easy to use the initial euphoria will quickly turn into frustration. The test results should thus not be interpreted as a measure of how cool the iPhone is, rather how well people are able to carry out common everyday actions and how well it does this compared to other phones. Furthermore, a short test with five subjects cannot give an absolute answer to the posed question. However, when observing users carrying out specified real life tasks, usage patterns do emerge quite quickly.
The devices

We went through a bit of trouble trying to find a suitable range of devices to compare with. Our initial ambition was to test touch screen devices only but we simply couldn’t find many enough that were available on the market. We then figured that comparing to other smartphones would provide the most appropriate devices. However, the iPhone doesn’t really qualify since it does not allow third party applications to be installed. In the end, we decided to pick a selection of new high end devices in the same price range but of different brands that provide roughly the same functionality.

We used our standard setting for usability testing, inspired by Wright & Monk´s cooperative evaluation. The user performs pre-defined tasks that should be representative for her normal usage, whilst a test leader communicates with her in a format that sounds like a normal conversation, but really is a refined method to find out what the user conceive about the product. An observer is present, with the one assignment to note everything of what is said and done that is important for understanding how the product succeeds in meeting the user´s expectations.
” If the device is not easy to use the initial euphoria will quickly turn into frustration.”

HTC TyTN
Our HTC TyTN uses the Windows mobile 6 operating system, a platform that is feature rich from the start and supports lots of third party software to be installed. It has a touch screen as well as a foldable qwerty-keyboard which means it uses the same basic layout as most computers.

There is also a set of dedicated keys that open the web browser, the e-mail application and different menus as well as a couple of contextual, or “soft”, keys that change action depending on what the user is doing or what state the device is in. It has a music player, a calendar, a little camera, expandable memory and it can connect to wireless internet over WiFi and 3,5G. This and many other things make it a powerful and generic smartphone.

Sony Ericsson W910i
The Sony Ericsson W910i is marketed as ”Your portable sound machine” and carries the Walkman brand, targeting it as a music phone. It has a slider form factor, meaning that the if you have to slide the top part to use the keys. The keypad uses a traditional 0-9 layout. It has a few dedicated keys such as start the Walkman application and change the volume.

While it certainly is possible to install third party applications, most people would disagree that this is a smart phone as there are not that many to choose from. Also, it does not read formats like .pdf or .doc, nor does it allow secure access to company e-mail.

All in all, the Sony Ericsson W910i is a high end multimedia device but if you have to put a label on it, smartphone may not be accurate.

Nokia N95
The Nokia N95 uses the Symbian S60 operative system, and is also packed with features out of the box. As with the Windows mobile platform, there is plenty of third party software to be installed by the user. Like the Sony Ericsson device it has a traditional 0-9 keypad, a couple of dedicated keys to control different menus, change volume and start the camera as well as quite a few contextual keys. The Nokia phone has the same slider form factor as the Sony Ericsson with the interesting addition that if you slide it the other way, a set of media controller keys such as play, pause and fast forward show up. It has a rather nice camera, a YouTube interface, FM radio, calendar and most of the stuff you expect from a smartphone today. It connects to the internet using WiFi or 3.5G and also has a built-in GPS tracker. Despite its phone-like appearance, the Nokia N95 does without any doubt qualify as a smartphone.

Apple iPhone
The iPhone stands out since it uses touch screen as the primary medium for all user input. There are a few dedicated keys; a home key that always returns to the main menu, a key for silent mode, keys for changing volume and one on the top to lock the phone. Besides being a phone, it also hosts an iPod interface, a camera, Google maps, calendar, a YouTube interface, and a few more applications. The iPhone does not support third party applications, effectively ruling it out as a smartphone. On the technical side, is supports GRPS and EDGE (the old standard for wireless GSM data traffic) as well as WiFi. It is the only device in the test that does not support 3 or 3,5G, meaning that users will be stuck with a lot slower data transfers while outside of WiFi coverage.

Results
After the users had completed all tasks, they were asked which phone they would prefer to use, should they choose today. Four persons picked the iPhone and one the Sony Ericsson W910i. In terms of subjective preference, this means that the iPhone won by wide margins. The Sony Ericsson came in second place and the Nokia third.

Most of the subjects did not mention the HTC at all when speaking of which device they would prefer. Looking at actual performance, four users were able to carry out all tasks without any help from the test leader while using the iPhone. The fifth person needed help to find the back-key in the upper left corner of the screen to step back in the interface. He also had problems making a call using the address book, trying to find a green phone icon to push. When using the Sony Ericsson W910i, two people did not manage to set the phone in silent mode, and one failed to set the alarm correctly. There were even more issues with the Nokia N95 as two users were unable to find the silent mode, two failed to take a picture, two failed to send the picture, and one had severe difficulties sending a text message. The numbers of failed tasks also reflect how much time users spent trying to figure out how to carry out the tasks they managed to figure out themselves. For example, all of the users found what they were looking for pretty much straight away using the iPhone, except realizing how to back out trough the interface as mentioned above. With the Sony Ericsson W910i, one person found it difficult to change volume during a call, two had trouble entering silent mode, another had problems creating a contact and one person even ran into difficulties entering a number.

Using the Nokia N95, issues were even more frequent; almost all users had difficulties taking a picture, either the cover was closed or they did not press the key long enough for the camera to focus and actually take the picture. One user had problems finding a contact as well as adding a new one, and several users had difficulties sending a taken picture to a contact.

When looking at the HTC TyTN, things did indeed take a turn for the worse. Only one of the users was able to carry out all tasks without the assistance of the test leader and all five users had severe difficulties with at least three of the tasks.

Two were unable to lock the device (one person gave up when realized he had opened the battery hatch), one could not send the picture or set the alarm and one could not add a contact. One user was unable to compose a text message, and one failed to make a call. Needless to say, the actions that some were not able to carry out at all still caused problems for the others. In terms of actual performance, the iPhone clearly stands out! Issues were certainly observed but nowhere near as many as with the other devices.

What makes the iPhone different?
What is it then that makes the iPhone different? Most importantly, it has removed one level of abstraction by allowing the user to act on objects using the finger directly on the phone’s surface. The difference between this and having to press keys on a keyboard and watch the screen to see what happens is striking. Instead of having to press one key to focus on the list item representing your contact and then clicking another key to make the call, the iPhone allows you to actually click the contact right on the screen. To scroll, you pull the list itself instead of clicking a down-key, and to flip between pictures in the album, you drag them from one side to another.

”Apple has removed one level of abstraction – the difference is striking!”

”An entertainment machine”
While testing, one of the users concluded that “this is an entertainment machine that you can make calls from”. What makes this comment particularly interesting is that the iPhone is actually the device in the test with the least number of features. Every other device in the test can do all the things the iPhone does already! In fact, out of all the devices in the test, the iPhone is the device that least of all qualifies as “an entertainment machine”, at least if you see to the number of entertainment related functions it can offer. Another user said about the different devices that “Apple is right on target! They make entertainment. These are just the same crap as usual [about the other phones].” What is it that makes the iPhone so enjoyable? Obviously there are many answers to this, but the fact is that all devices provide the same basic functionality, but most of it passes unnoticed on the other devices. The iPhone is different in at least three aspects: Transparency. What you see when you start the iPhone is pretty much what you get. There are no hidden applications. It’s a matter of seconds until the user has understood the whole range of services and features.

Accessibility. There is no deep menu system, one touch is often enough to get to where the user wants. There is no digging down in lists or shifting focus across different icons. Seduction. Simply browsing the user interface is quite enjoyable. The bright, high resolution screen, the smooth animations, all the subtle but informative sounds and the attention put into the graphic details all contribute to the positive impression of the device.

Users realized they were seduced by the iPhone and it did not seem to bother them. One of them said that “the Nokia N95 is more completed and the smartest of them but I would still choose the iPhone as I am temporarily seduced by it”.

So, does the iPhone live up to its hype?
The iPhone aims for the stars, and the expectations are incredibly high. However, from what we have seen in this initial test, it manages very well. Crucial everyday actions are intuitive and effective to carry out, at least better implemented than with competing mobile device conventions. The iPhone also seduces with its glossy interface and bright screen. So – yes. The iPhone does live up to the hype!
7
Contact Stockholm info@inuse.se inUse AB www.inuse.se Drottninggatan 83 111 60 Stockholm Phone: +46 8 - 58 88 77 82 Mail: stockholm@inuse.se Malmö Göteborg inUse AB inUse AB Södra Förstadsgatan 23A Östra Hamngatan 45 211 43 Malmö 411 10öteborg Phone: +46 40 - 600 94 00 Phone: +46 31 - 72 21 120 Mail: malmo@inuse.se Mail: goteborg@inuse.se

www.inuseful.se