Grafisk illustrasjon av mennesker, verktøy og teknologi.

The opportunities and limitations of automated accessibility test tools


Automated test tools can be extremely useful in a lot of projects and bring a great value proposition to projects. However, as most experienced developers and testers know there are some caveats to these tools. Automated accessibility testing tools are no exception. In this article we will dive deeper into the pros and cons of the most popular tools and how one can overcome the cons of these tools.

General limitations to automated accessibility testing tools


As research shows, there is a vast amount of concerns regarding automated test tools (Baazeem & Al-Khalifa, 2015). Among some are the high rate of false positives, and the low rate of test coverage (Vigo, Brown, & Conway, 2013). In fact, a research paper that looked at the six most popular tools for automated accessibility testing showed that half of the success criteria would be missed, and 6 of 10 violations would not be caught. This coupled with the tendencies testers has to rely on the first results that are presented, (Tversky & Kahneman, 1974) (Harder & Begnum, 2016) kan automatiserte testrverktøy føre til store konsekvenser. Personlig har jeg sett dette på ulike måter, og i sin tur kan hele prosjektet bli satt i en sårbar posisjon, da det kan stilles spørsmål rundt prosjektmedlemmenes kunnskap og integritet.

Summarized:

  • 6 out of 10 violations will not be caught.
  • Half of the success criteria’s (WCAG 2.0) is missed
  • Testers have a tendency to rely heavily on the first results given.

Some examples of tools


Siteimprove is a for-profit organization that sells automated testing tools for accessibility and GDPR. The tool itself is built up like most other automated test tools but offers a user-portal that will save test results. The most commonly used accessibility test tools are Axe and Google Chrome lighthouse audit tool. As Googles audit tool uses Axe’s engine and success criteria («Google Selects Deque’s axe for Chrome DevTools,» 2017), under the copyleftist license Mozilla Public License version 2.0, as a baseline for their tool (Accessibility engine for automated Web UI testing. 2015/2018) the only difference between Axe and Googles services, is the presentation of its findings. Siteimporve has been shown to produce the same test scores as Axe (Duran, 2017) and due to the nature of the test-criteria, there is a natural limit of how much such an automated tool can do. Another aspect to keep in mind is the testing teams experience with the tools that are being used. As an example the automated testing tool called Axe also includes WCAG: Best practices as a part of the test scope (axe-core/doc/rule-descriptions.md, GitHub).

Comic with one employee saying
Siteimprove tilbyr mange kakediagremmer… Være dog oppmerksom på at testskriteriene ikke dekker hele WCAG 2.0.

How to use automated accessibility testing tools


Automated accessibility testing tool must be used in combination with at other testing methods. In the accessibility sphere it is commonly known that a combination of automated test tools, expert testing and user testing is the “Triforce” of accessibility testing (Vigo et Al., 2015). Automatiserte testverktøy gir et begrenset førsteinntrykk til nåværende tilgjengelighetstilstand for et prosjekt. Automatiserte testverktøy for tilgjengelighet virker underverk med problemer relatert til kontrast og metadata, men faller kort med hensyn til testdekning og pålitelighet. Brukes derimot denne metoden i kombinasjon med eksperttesting og brukertesting, kompletterer de tre testmetodene hverandre. Andre studier fremhever også WCAGs lave testbarhet av eksperter, som du kan lese mer om i denne artikkelen av Andreas Jacobsen. Den eneste anbefalte tilnærmingen som kan gis, er å bruke en kombinasjon av automatisert, ekspert og brukertesting for å oppnå pålitelige resultater .

Share on facebook
Facebook
Share on google
Google+
Share on twitter
Twitter
Share on linkedin
LinkedIn

Share this post:

Share on facebook
Facebook
Share on linkedin
LinkedIn
Share on email
Email