Approaches to Usability Testing

You have decided your materials or services are ready for usability testing but can't decide which approach might work best to meet your needs. Well, let's take a look at four different approaches to determine which one may work best for your usability team.

The approach you choose will depend on your goals, budget, timeframe, and what resources you have.

Exploratory Usability Testing

What It Is: Exploratory usability testing, also known as concept testing, focuses on assessing what is and what is not working in the design of your materials. This approach is often task-based or scenario-based to establish empathy towards future users of the materials.

Best For: Early stages of development for a quick review of your material.

Example: The Secret Service (USSS) may conduct exploratory usability testing on an early prototype for an internal facing web-based application to obtain feedback from their colleagues on the applications contexts and performance. The USSS usability testing team would be looking for initial thoughts and feedback on the early prototype to further develop the web-based application to be more aligned with the future users' needs.

Comparative Usability Testing

What It Is:  Comparative usability testing focuses on comparing and contrasting materials or services to see which one is more effective and efficient for the participant to use. During testing you want to obtain both quantitative and qualitative data points to compare services.

Best For: Comparing two or more drafts of materials or services to see which one works best for the participants.

Example: The Federal Law Enforcement Training Center (FLETC) may conduct comparative usability testing to determine which of two training courses is more effective to those using it. The FLETC usability testing team would be looking to test efficiency, effectiveness, and satisfaction by using both comments and metric data about each training course to determine which course to include in required law enforcement training.

Assessment Usability Testing

What it Is: Assessment usability testing focuses on observing participants' interaction with the service through task-based scenarios to obtain qualitative data based on the participants ability to complete tasks while using the service.  

Best For: Testing at any stage of a service's development to gain future user feedback. This is the most common approach to usability testing.

Example: Customs and Border Protection (CBP) may conduct assessment usability testing on a prototype midway through development to evaluate a new regulatory document for user comprehension. The CBP usability testing team would be looking to gain feedback and recommendations on the language used in the document and the clarity of the information to ensure the content was correctly understood.

Validation Usability Testing

What It Is: Validation usability testing focuses on obtaining quantitative data, such as completion rates and time taken to complete a task, to establish benchmarks to help set a standard for future designs and iterations.

Best For: Testing the final version or updated version of your service.

Example: The Coast Guard (USCG) may conduct validation usability testing on a prototype close to completion for an online supply request form. The USCG usability testing team would be looking to gain feedback on completion rate, time of completion, and an ease-of-use rating scale for the online supply request form to ensure the form can be completed with ease and in a timely manner.

In-Person... Remote... Which Testing Method Should I Use?

Your usability team's goals, time, funds, and resources will impact your decision to conduct usability testing in-person or remotely. It is important to remember that there isn’t a best method to conduct usability testing, just a more beneficial one depending on your needs.  

Benefits of In-person Usability Testing  

• Better insights into a participant's non-verbal communication.
• Easier to recognize when to probe and ask questions based on participant’s body language.
• Able to recruit participants in lower socioeconomic areas that may not have access to technology.
• Better controlled environment to limit distractions during the testing session.
• Testing can be conducted at earlier stages in development without running into prototype dead ends.

Benefits of Remote Usability Testing

• Participants may feel more comfortable testing in their own space and using their own technology.
• You can test participants outside of your geographic location.
• Easier to recruit participants based on their availability.
• More flexible with time and budget friendly.

Additional Testing Considerations

Quantitative and Qualitative Data: Information gathered during usability testing sessions will take the form of quantitative and/or qualitative data.

Quantitative data focuses on collecting insights, findings, and anecdotes about how a participant uses the service. For example, discovering a participant's pain-points by observing and listening to their frustrations or confusions while they test a service.

Qualitative data focuses on collecting and using insights, findings, and anecdotes shared by the participant to describe their experience. For example, documenting a participant's pain-points by observing and listening to their verbal frustrations while they test a service.

Number of User Groups: Depending on the material or service you are testing; you may choose to have multiple user groups to collect enough user feedback to make iterations.

In Summary

No matter which approach you choose, conducting a usability test brings your DHS Component one step closer to creating a more customer-focused service that will lead to a better customer experience.

Usability testing gives you important information about how real users interact with your materials and services, helping you to make informed decisions about how to create and update your services to ensure your customers' voices are heard.