How to test for accessibility with users at every design stage

View the original post

An illustration with a coral-red background showing an email envelope icon with a roll of paper unfolding out of it, with icons on the paper, including keyboard cursor, audio and sight and icons.
Illustration by Alisha Giroux.

A research template to test solutions for people with disabilities

User research has always been a core part of the design process at Shopify. But, until recently, I’d only ever participated in sessions as a notetaker, or secondary facilitator.

This year, that all changed. Through a partnership with Fable, an accessibility testing platform, I developed and executed four research sessions for my project. As a result, the team was able to successfully improve the Shopify Email product experience to make it much more inclusive.

Going from 0 to 60 in user research testing as a product designer was a daunting task — especially in a realm I was still just learning about (designing for accessibility). In this post, I’ll share how I leveraged user testing with people using assistive technologies — which let me improve Shopify Email by gaining first-hand feedback at every stage of the design process — and suggest a template you can use to plan your own user tests.

Taking the time to integrate inclusive user research, beyond just applying “best practices,” is invaluable to ensure products can be used by a broader audience. The hardest part is just getting started.

The problem

Shopify Email was initially built in a way that was inaccessible to some users of assistive technology. In particular, we noticed that the ability to add a new section to an email was only possible using a mouse. This meant some people using alternative navigation, most screen reader users, and people who rely on the keyboard alone, were limited in how much they could customize an email.

That was a huge problem: around 15% of our merchants self-identify as having a vision disability.

In the initial experience, adding an email section was only visible while hovering between sections.

Additionally, the tab sequence navigation and labeling for screen readers needed refinement to allow those users to complete core task flows within the app more easily. To make our product more usable for more merchants, we sought to advance Shopify Email to better meet the Web Content Accessibility Guidelines (WCAG).

Some of the key considerations I established with my team were:

  • Introducing a keyboard shortcut wouldn’t be enough to make the product accessible; the option to add a section needed to be a permanently visible action.
  • A new “add section” solution should consider future mobile feasibility.
  • The tab sequence should follow best practices and provide a logical way to move between sections and their corresponding settings.

What we designed

To enable blind and low-vision users to add a new section to their email template, we added a permanently visible button in the Editor. We also maintained the hover-over behavior for those already familiar with that way of doing things.

In parallel with these changes, we updated the labelling and tab sequencing to ensure alternative navigation users were able to move around the app more easily.

To ensure the usability of all of these improvements for people using assistive technologies, we ran four research sessions with Fable testers.

About Fable

Fable is a user testing platform that facilitates recruitment of professional testers to help explore and vet solutions at all stages of fidelity.

All testers are people who use assistive technologies and some even have a technical background to offer implementation recommendations. (And, yes, all are under NDA.)

We piloted using Fable at Shopify in 2019 and again at the end of 2020. It’s been used by the Retail team for interviews, and by other teams for checking products after they’ve been built. The Email team is the first team at Shopify to use Fable more broadly, with multiple requests across different design stages.

Testing accessibility with Fable

Shortly after kicking off the project, we engaged the Fable community over the course of two months. Each round of research (four in total) took about 3–4 days to complete, accounting for recruitment time. (Because the Fable platform operates almost like a marketplace, finding testers was always the easy part once I submitted a research request.) While I ran point on three research sessions, the lead engineer on the project also stepped in to lead his own session.

To familiarize myself with the Fable platform, and help the team learn common preferences of folks using assistive technologies, I ran several informal, educational interviews. After I had a loose concept of a solution in mind, I conducted co-design sessions with Fable testers to validate the approach before polishing high-fidelity prototypes. When we were refining the tab sequencing solution, our lead engineer took the lead on a more casual implementation discussion with a Fable tester who had technical expertise. And finally, once the solution was live behind a beta flag, we conducted a few unmoderated sessions with testers running through a simple task flow.

After every research round, I would aggregate insights into a short, high-level deck to share with the project team for visibility. Although the research sessions did not surface any major red flags, they did help us build confidence that our solutions were actually accessible from a human perspective — as well as a technical one.

The impact: button vs. hover

When we initially rolled out the new button feature, we ran a 2-week experiment to track the user behavior of using the new button compared to the original hover behavior for adding a section. We saw 20% of users only used the button during their session of editing an email, which indicates that there are potentially folks who either didn’t realize they could add, didn’t know how to add, or were not able to add sections before this button existed.

Although the long-tail impact of this button since launch doesn’t indicate that merchants are adding more sections overall, the team is confident the improved experience has enabled more merchants to use the app successfully based on our initial experiment after launch.

This project was my first experience running this type of usability testing. While the project helped us successfully make our Email product more inclusive and usable, we’re still trying to work out a usability testing workflow that scales for all our projects.

Here are my tips and takeaways from the experience:

  • Familiarizing myself with common challenges and user behaviors for folks using assistive technologies — before imagining a solution — helped build empathy and changed the lens I used when initially envisioning a solution.
  • Vetting our solution from both a usability and technical perspective ensured an experience that was accessible from both a human-centric and compliance-centric perspective.
  • Testing is easy — getting started is the hard part. Just think of your user testing like a conversation.

To help you get started, below is a useful starting point for planning accessibility testing at every design stage.

An illustration of a green envelope with a piece of paper showing an icon of an eye on a piece of paper inside.

A template for accessibility user testing

This template outlines a series of four research sessions, conducted at each design stage, to help successfully validate your products and solutions with people with disabilities.

Session 1: Informational interviews

What: Casual 1:1 conversations to understand common challenges for people using assistive technologies (AT).
When: Before exploring specific solutions to the problem at hand
Who: 3 users: 1 screen reader, 1 screen magnification, 1 alternative navigation (voice control)

  • To get your feet wet using Fable and get comfortable conducting UX research
  • To understand the general preferences and challenges of folks using different AT
  • To get a quick “first impression” review of the existing product experience from folks using AT

How: Casual Q&A followed by a very basic task flow within the product overall.

  • Start with a quick company overview, and context about the specific product or app for the testers
  • General AT preference questions, to educate yourself and your team
  • High-level, guided task flow

Research questions that helped inform a detailed script:

  1. What kinds of assistive technologies are folks using to interact with web applications?
  2. What are their preferences in terms of using those technologies?
  3. What are the frequent pain points they have? What do they consider characteristics of great experiences?
  4. What does the product or app look like/how does it behave when using AT?

Session 2: Design review/concept preview

What: Moderated user interview to validate a proposed concept or solution
When: With initial screen mockups, before build begins
Who: 3 users: 1 screen reader, 1 screen magnification, 1 alternative navigation (voice control)

  • To identify any potential pitfalls in a proposed design, before moving into high-fidelity prototype and design
  • To vet initial impressions and user sentiment about solution

How: Quick Q&A followed by a guided task flow with the proposed solution

  • Start with a quick company overview, and context about the product for user
  • General AT preference questions
  • Share Figma link (or screen share your mockups) to get initial impression of solution

General format of questions while showing prototype:

  • Based on this mockup, how would you go about [doing some task]?
  • What would you expect to happen when you do that?
  • Can you tell me your impression of this mockup at this point?
  • What do you think about the experience around [doing that task] based on this mockup?

Research questions:

  1. What is lacking or misaligned with their expectations of how the proposed flow should work?
  2. Is the potential solution accessible from a human (versus technical) perspective?

Session 3: Implementation validation

What: Collaborative discussion between engineering and Fable tester to discuss implementation.
When: After build begins, before launch
Who: 1 screen reader user
Why: To quickly test and discuss the technical implementation of our solution (in particular, labels and behavior for the tabbing sequencing)
How: Informal discussion led by engineering

  • Demo solution for user
  • Discuss any pitfalls and plan ways to address them

Session 4: Compatibility testing

What: Unmoderated testing of the solution to ensure users can complete the task at hand.
When: Just before launch, when solution is usable in beta
Who: 5 random users of different AT (i.e. screen reader, screen magnification, alternative nav)
Why: To validate that our implementation technically and functionally supported the core task flow for most users
How: Unmoderated user test in demo store

  • Provide login credentials and specific 10-step task flow for users to test
  • Include company and product context brief with login instructions
  • Create any foundational drafts or individual test environments to provide as a base for each user flow

Research questions:

  1. Can users accomplish the core task using their respective assistive technology?
  2. Does the experience meet expectations of users of assistive technologies?

How to test for accessibility with users at every design stage was originally published in Shopify UX on Medium, where people are continuing the conversation by highlighting and responding to this story.