GRIT Toronto Test #2: ChalmersBot

GRIT Toronto is a civic usability testing service that connects technology creators — including governments, startups and civic tech groups — with diverse and underrepresented communities to help test their products. Our mission is to make usability testing more accessible, inclusive and effective.

One of the ways we do that is by working in the open. These regular blog posts open up the black box of technology testing, introducing readers to some of our clients as well as key findings from our tests.

Built by civic tech startup Ample Labs, ChalmersBot is a beta web app that helps people in Toronto who are experiencing homelessness find much-needed social services, including free meals, drop-ins, overnight shelters, and clothing banks. It uses a chatbot function to allow users to type in questions or needs, and quickly get information about relevant services near them.

For this test, we gathered six testers, between the ages of 18–44, at the Centre for Social Innovation in Toronto. All six had either current or past experience with homelessness, precarious housing, or couch-surfing. The sessions consisted not only of a usability test, but also a participant journey where testers recalled their own experience accessing (or needing) the services ChalmersBot aims to help users access.

Before we jump into the test findings, we want to express our gratitude to testers for contributing their time and sharing their experiences with us. Their honesty and candour was inspiring, and made us feel privileged to be doing this work!

Results

Overall, testers said they would recommend ChalmersBot to others, especially during winter or if users are new to Canada. They found it to be a simpler way of finding information on the services they would be looking for, compared to doing so through a patchwork of other sources.

Look and feel

Testers generally found the web app’s interface to be clean and easy to use. They also liked that the chatbot used plain language and a friendly, or “human,” tone.

Navigation

Many testers felt the app’s navigation could be improved. In particular, they found returning to the start after searching for a resource to be confusing. Many simply reloaded the app, causing them to lose the information from their previous search. This was off-putting for many users, as once they found a service that worked for them, such as a specific food bank or shelter, they wanted to be able to easily see that information in their history instead of having to replay the same conversation with the chatbot.

We recommend creating an easier method, or prompt, to return to the homepage without reloading. Testers also suggested being able to save searches and select multiple resources in one search (e.g., searching for a meal and shelter at the same time).

Search options

Testers liked the option of finding a free meal based on the age group and gender it was targeted for. However, this option was only asked at the end of the search flow, which was not intuitive for some users. Therefore, we recommend asking about this option earlier on in the search process.

Similarly, testers also appreciated being able to search for a meal by time and day. However, they were less confident in completing this task and felt unsure about the accuracy of the results.

Resource information

The information on meals and shelters is quite comprehensive, but testers also wanted more details for mental health resources (e.g., counselling) and the services offered by drop-ins (e.g., laundry, showers). Testers were also frustrated when search terms like “counselling” or “depression” did not display any results for mental health services.

Therefore, we recommend adding more information and search options for both drop-ins and mental health services.

Challenges faced by newcomers

Half of the testers were newcomers to Canada when they were experiencing homelessness or housing precarity. As a result, they were able to provide lots of feedback on challenges newcomers might face when using ChalmersBot. This included barriers like limited English proficiency, unfamiliarity with local geography, or not having a working phone plan in order to call shelters and inquire about the availability of beds.

ChalmersBot may not be able to address all of these challenges, but testers noted that providing more buttons and pre-made prompts (or translations into other languages) could improve the experience of users who are less familiar with English.

What we learned

This is the first GRIT test to incorporate more formal user research and ask testers to expand on their personal journeys. It was really valuable, and generated findings that may have been missed if the test focused only on usability.

However, deciding the order of the usability test and participant journey survey was difficult. We decided to do the usability test first to get testers comfortable with the product and the proctors before discussing sensitive topics. Testers’ openness and willingness to discuss their journeys with proctors after the test appears to have validated our choice!

Test it yourself

You can try out the beta version of ChalmersBot at www.chalmersbot.app. Ample Labs will also soon be launching a crowdfunding campaign for the app, so stay tuned!

And if you have any feedback, thoughts or suggestions for ChalmersBot, you can reach the team at general@amplelabs.co.

Interested in working with GRIT Toronto? Become a tester or even run a test with us. If you have any questions, reach out to Marisa Bernstein at marisa@codefor.ca.