GRIT Toronto Test #6: Vehicle Recalls Canada

GRIT Toronto is a civic usability testing service that connects technology creators — including governments, startups and civic tech groups — with diverse and underrepresented communities to help test their products. Our mission is to make usability testing more accessible, inclusive and effective.

One of the ways we do that is by working in the open. These blog posts open up our testing process, introducing readers to some of our clients as well as key findings from our tests.

With smart speakers becoming increasingly common, Transport Canada has been exploring how voice assistants like Alexa could make government services and information more accessible to residents.

Transport Canada has started by prototyping an Alexa Skill that provides vehicle recall information from their database. Each year, vehicle manufacturers issue an average of 650 recalls due to safety defects, affecting over five million vehicles. However, about 25 percent of vehicle owners never take their vehicles to the dealer to have the defects repaired.

With the Vehicle Recalls Canada Skill, Alexa simply asks a few questions about the vehicle make and model, and then provides all the recall information for that vehicle.

You can see an early version of the Skill in action in the video below:

Here

While the team had previously done some testing of the Alexa Skill internally, they knew they had to get outside of government for additional insights. So, they reached out to GRIT Toronto to help connect them with people with different disabilities. They wanted to learn more about their needs and how these users are — and could be — using voice technology.

The sessions also helped Transport Canada test the assumption that custom apps for voice assistants, such as Alexa Skills, do in fact make government services more accessible.

Read more about the development and initial testing of the Alexa Skill here.

One-on-one user research interviews were conducted with four participants over two evenings at the Centre for Social Innovation. Three of the participants were blind (one of the three also had auditory processing challenges), while one had upper body mobility impairments.

Each interview was an hour long and conducted by two design researchers from Transport Canada (one present in the room, with a second person calling in remotely). Marisa Bernstein from GRIT Toronto, was also present to observe the sessions.

Participants interacted with the Skill using the Alexa app on an iPhone. They were first asked about their experience with accessing government services online and using voice assistants. They then followed a script to talk to Alexa — answering questions about the make, model and year of a car — and provided feedback on their interactions with Alexa.

Results

Screen readers can interfere with Alexa on a smartphone

Some of the participants were already comfortable using screen readers and other assistive technologies. However, the iPhone screen reader, VoiceOver, was reading out the Alexa app screen while participants were trying to speak with Alexa. Allowing Alexa to temporarily override other voices, also known as audio ducking, could help to solve this issue.

Additionally, when designing the visual interface of a voice assistant app, it’s important to consider accessible design concerns like labeling buttons.

Alexa’s voice has potential, but still has limitations

Participants had a mixed response to Alexa’s voice. A key learning was that intonation and inflection, which make the voice sound more human, can help people with auditory processing challenges better understand what it’s saying. Alexa also sometimes had trouble properly recognizing and responding to commands.

One participant suggested that new, commercial voice products like Alexa are not as accessible as technologies and devices designed explicitly for that purpose. For government departments like Transport Canada, it’s important that voice technology services continue to improve. Being able to customize the voice output, such as by adjusting the talking speed or altering the intonation, would make these services much more accessible.

However, soon after this test, Amazon did in fact update Alexa to allow users to change how fast it speaks.

The Skill could provide more information and be less rigid

Some participants made mistakes while answering Alexa’s questions, such as confusing vehicle make and model. When they did so, it was difficult to course correct, especially if they didn’t have the ability to read a script. Instead, they often had to start over from the beginning.

Transport Canada is looking into making it possible to ask Alexa for clarification and making the process of answering Alexa’s questions more flexible.

Additionally, after providing the recall info for a vehicle, Alexa provides the phone number of Transport Canada’s customer support. Participants thought it should also offer to dial the number itself, rather than just reading it out.

What we learned

This test was different in many ways from previous GRIT Toronto tests. For example, our role primarily consisted of recruiting participants for the test. The sessions themselves were also overseen by Transport Canada staff, rather than GRIT Toronto proctors.

However, by being able to observe the sessions, we learned a lot about the context and capabilities of voice assistants, and came away with insights that will inform our practices around how we accommodate participants with disabilities.

Testing voice assistants using different environments and devices is valuable

While user testing typically takes place in a quiet room, we found that it is also valuable to conduct voice assistant testing in different conditions. For example, an environment with background noises (e.g., other people, pets, TV) might be more similar to how the device would be used at home. Testing with a home speaker device, such as an Amazon Echo, would also be valuable.

There are ways we could better accommodate parents and people with disabilities in future tests

This was the first test where we focused on recruiting participants with different abilities. While this is an important part of inclusive usability testing — we’re firm believers that testing at the margins improves the experience for everyone else — there are still many things we could improve when it comes to recruiting and accommodating people with disabilities.

Documents and forms that need to be read before the test should be sent as Word or accessible PDF documents, so that they can be used with screen readers. We also need to have a better understanding of how Wheel-Trans— Toronto’s accessible transit service — operates, so that we can make the drop-off and pick-up process more seamless for participants.

This was also the first test where a participant brought along their child. Fortunately, there was someone who could look after them while the participant was in the test session. Child care is important for ensuring parents can participate in our tests, and we’re looking into how we can best ensure child care services are available at all our tests.

Check it out

The Vehicle Recalls Canada Alexa Skill is now available through Amazon and can be enabled on any Alexa device. You can also check out the entire, open source project on Github.

The service is slated to run as a six-month pilot test. While it’s still a work in progress, we’re happy to see that Transport Canada has released an early version to get feedback from the public!

Interested in working with GRIT Toronto? Join our waiting list to become a participant or even run a test with us. If you have any questions, reach out to Marisa Bernstein at marisa@codefor.ca.