Conducting a Usability Study with Mobile Devices: Preparation Tips

Usability studies have become part of any comprehensive design project of a web site or application. Conducting a study with mobile devices - observing how a person interacts with their device and any web sites or applications on it - is similar in many ways to a usability study with a computer. However, there are some key issues when working in a mobile environment that should be taken into account while preparing, moderating and reporting on a study:

· Participant recruiting has more variables to nail down, making it a longer than the usual process.

Obviously you need people. In a mobile study, you need people who have a mobile phone, plus other variables. These include:

- Other features on the phone that are used in the study, i.e., camera, GPS, mp3 player, etc.

- Service provider compatibility with your application.

- Service plan details, i.e., can the participant perform the tasks you are studying?

- Verification that the user can access the application you are working on.

This may mean you need to conduct a quick test after the participant has been selected and before they arrive for the study. You should have a couple of extra phones on hand during the study that can be used in case issues come up with the participant’s phone. Just be aware that the participant may not be familiar with the phone design, making complex tasks take longer and impacting completion rate metrics.

· Gathering data will take more work.

Unfortunately, you can’t easily sit behind a person and watch them use their phone and get the same visual information that you can from watching a person using a full-sized computer screen. Technology has yet to develop simple software that allows you to watch in real time on a larger screen what a user is doing on their mobile device. To gain the most from a mobile usability session you may need to:

- Have at least two video cameras; one on the user, and the other on the phone.

- Adjust the camera focusing on the phone on the fly if the participant moves their phone out of the lens range while they are performing a task.

- Be extra vigilant about asking the user to think aloud so you can always be aware of where they are in the process or on a particular screen.

- Slow a task down or ask the user to repeat an action if it was not immediately visually available.

· Adapting to new situations during the session is key.

Due to variances in phones, services and applications, there are more variables out of your control during mobile usability sessions. Potential areas to watch out for are:

- Devices may show information differently.

- Service providers may cause issues, such as a slow response time or cutting out.

- The participant may get calls during the session.

- User’s proficiency with devices varies widely, more so than general keyboard and mouse skills, influencing task completion.

· Being flexible and having a back up plan(s) can reign in the potential chaos.

Usability studies are mostly designed to have a predictable flow of tasks or events. Since device or connection issues may interfere with such a flow, it is important to have alternative flows or plans at the ready. Briefing your team at the beginning of each session about the order in which you will try different plans will slow the time needed to re-adjust a new plan flow. It may also help to “chunk” your tasks or investigations in a modular fashion so they can be moved around in the study if users are having issues completing a particular task. Recording errors will also let you (and your client) see if there are patterns, such as phone type and/or service plan having issues with a particular download, that you can try avoid in future sessions.

In sum, mobile usability studies, like the mobile device environment, is more open and variable than traditional usability studies. Be prepared to spend more time than usual finding the appropriate participants that have the correct device/carrier combination you need. Be extra vigilant during the study to make sure you are capturing – visually or audibly – the important data, which can be hard to see. And most of all, be flexible and prepared to adapt to a variety of unforeseen circumstances. The extra preparation and effort will pay off in the end, though you may always have the need to improvise. Don’t forget you have more opportunities to learn - off task investigation can lead to important discoveries about your user, their device use, and their expectations of mobile experiences.