How to Run an Unmoderated Usability Test

What is a Usability Test?

Usability testing is a very popular user research methodology. In a usability test, researchers create a set of tasks for participants and evaluate how easily these tasks can be completed. The end goal for usability research is to identify friction, eliminate confusion, and design intuitive products.

Any product that needs a manual to work is broken. - Elon Musk

There are many different flavours of UX test:

Products can be evaluated at different stages in development (concept vs. production)
The study can be moderated or unmoderated (no researcher present)
The test can be in-person or conducted remotely

In this guide, we’ll be covering how to run an unmoderated remote usability study. When crafted properly, unmoderated studies can yield significant advantages over moderated ones in terms of scale, cost, and reducing bias.

A visual representation of the usability testing landscape.

Step 1: Goals

This is where all research starts. As a user researcher, these objectives are often provided to us from product teams or executive leadership; however, lots of research teams also explore which questions are most important. These questions might include:

Why are users suddenly churning from our platform?
Why are we seeing lacklustre metrics on our latest product launch?
How do users interact with our competitors’ product versus our own?
What do users think of a new concept we have?

Once you’re confident about the problem you’re trying to solve, it’s recommended that you break the objective down into a very specific set of questions. If your main objective is to test a new user interface, some example questions might be:

What do users miss about our old interface?
Have existing friction points been alleviated?
How intuitive is our new interface?
Are users able to locate their old workflows?
How would users describe our look and feel versus before?

Step 2: Audience

Before you build your study you’ll need to think about who you’re going to collect data from, and on what medium you’ll deliver your study to them. If you plan to use Phonic, we recommend Phonic Recruit for compatibility; however, you’re free to use whichever platform you prefer. Respondent.io and userinterviews.com are both great options for recruiting.

Note: Be careful recruiting from quant panels for usability studies. It’s easy to get carried away with cost savings with unmoderated research; however, you risk invalidating your work with faulty or fraudulent panel.

Step 3: Design

Unmoderated usability tests generally include three sections:

Introduction/screening
A set of tasks or “mission”
Post-mortem/conclusion questions

The introduction is designed to collect background info on the participant- who is this person? Where do they work? Do they currently use products like mind? The introduction is also the place to provide any background on the software or tool being tested and the platform the study is being run on.

Next is of course the task completion. Here you’ll begin recording the screen, webcam, and microphone of your respondents as they complete a set of instructions. It’s important that these instructions are concise and easy to follow. Respondents will likely provide valuable feedback on the experience regardless of if they successfully complete the mission or not.

Step 4: Programming

Once you have a study design, the next step is to program things in user testing software. There are many tools out there which allow you to implement unmoderated remote usability tests. In Phonic we support a wide variety of qualitative and quantitative question types which map very well onto most study designs. Other tools you can check out include UserZoom, UserTesting, and UserBrain.

Step 5: Field

Your study is now designed and you’re ready to go into field. If you’re using Phonic recruit, we’ll handle this step for you! If you’re using a professional panel, make sure your screen-out links are wired and that the panel has access to a testing link for their internal pre-launch procedures. If you’re testing with your own set of users, share the study link with them over email.

Step 6: Analysis

As data rolls in you’ll have access to full, qualitative sessions from each participant. Most unmoderated research platforms will provide transcriptions in many languages, as well as some other out-of-the-box analytics. You can also navigate your data either by respondent or by question. As you go through sessions, we definitely recommend tagging and highlighting your data for future use. It’ll make your life far easier when it comes time to create deliverables such as reports or showreels.

Step 7: Reporting

The final step is to wrap your study up into a set of concise insights to share with your team or client. As mentioned above, reports and showreels are fantastic ways of showing off your work.

That’s all! Please feel free to reach out to the team at contact@phonic.ai if you want to talk more about unmoderated research.