Do we need to test accessibility with real screen readers?

Yes, once what NVDA or VoiceOver announces on each step matters, a person runs the main flows with both before each major release, because no automated tool reports announcements. One pass with NVDA on Windows and one with VoiceOver on macOS covers both main desktop platforms.

Can automated tools replace manual accessibility testing?

No, automated tools cannot replace manual accessibility testing, because tools miss barriers that need judgment. In a UK Government Digital Service test, 10 tools together missed 29% of 143 barriers. Automate the axe scan and the keyboard test; add a manual screen reader pass once announcements matter.

Should accessibility tests run in CI?

Yes, run axe scans of the main pages in CI and fail the build on a violation, so a new barrier blocks the change that adds it. Run the keyboard test of each main flow in the same CI job.

Should I test accessibility with automated tools?

Verdict

Yes

Yes, test accessibility with automated tools, minimally: run one axe scan per main page in CI and one keyboard test per main flow, so a button without a name or a control that Tab cannot reach fails the build.

Why

Blast radiususers
Change frequencyregularly
Detectabilityeventually
Reversibilitytrivial
Test costmoderate

Yes, test accessibility with automated tools, minimally: one axe scan per main page and one keyboard test per main flow. Blast radius is users, because a button without an accessible name stops a screen reader user from finishing signup. Change frequency is regularly, since main pages change about once a month. Detectability is eventually: sighted developers see nothing wrong, and a blocked user more often leaves than writes to support. Reversibility is trivial, because a markup fix removes the barrier and no data stays behind. Test cost is moderate: adding axe to a Playwright suite takes about an hour, and the first run reports a backlog someone must triage. Rule R12 gives Test minimally.

When the decision changes
When	Decision	Why
An accessibility law or contract covers the site, such as the European Accessibility Act for EU e-commerce and banking	Test mandatory: axe scans on every page template in CI, keyboard tests for every flow, and a manual WCAG audit before each major release	Blast radius rises to safety-or-legal, because a barrier breaks a law or a contract
The flow is checkout, where a customer who cannot reach the Pay button buys elsewhere	Test mandatory: an axe scan and a keyboard test on every checkout step	Blast radius rises to money and Reversibility to costly, because a lost order does not come back
Editors publish pages and images through a CMS in most weeks	Test: scan every sitemap URL with axe each night and make alt text a required CMS field	Change frequency rises to constantly, because each new page can add an image without alt text
The same screens ship in a native iOS or Android app	Test: run Espresso AccessibilityChecks on Android and performAccessibilityAudit in XCUITest on iOS	Reversibility rises to costly, because a fix waits for an app store release
The check you want is what NVDA or VoiceOver announces on each step	Test it differently: a person runs the main flows with NVDA and VoiceOver before each major release, and a feedback link reports what users meet	Test cost rises to heavy, because driving a screen reader from a test needs a real machine per reader, and announcements change between versions
An internal admin tool that five colleagues use changes a few times a year	Do not test: build it from native HTML controls and fix a barrier when a colleague reports it	Blast radius falls to internal and Change frequency to rarely

What breaks if you don't test

A redesign replaces the Save button text with an unlabelled icon. A screen reader now announces "button" and nothing else, while the developer and the reviewer see a working icon. The first signal is a support email weeks later, if the user writes at all.

What you lose if you over-test

A green scan does not mean an accessible page. The UK Government Digital Service built a page with 143 barriers and ran 10 automated tools on it: none of them found 29% of the barriers, and the best single tool flagged 41%. An axe assertion in every component test repeats what the page scan checks. jest-axe runs in jsdom, which renders nothing, so its colour contrast rule is off. Keyboard traps, focus order and confusing announcements need a person.

How to test

Test at the end-to-end level, where a real browser computes colours and focus:

Run axe in Playwright on each main page with the WCAG 2.1 A and AA tags, and fail the build on any violation.
Write one keyboard test per main flow: reach each control with Tab and submit with Enter.

The W3C guide to evaluation tools says tools can only assist a judgment of accessibility.

Procedure and references

When the answer changes

A law, a public-sector contract or a customer's procurement form names WCAG.
The flow takes money, as checkout does.
Content changes weekly through a CMS, outside code review.

Real incident + Code example

The arrow that axe could not see

On a scheduling product I worked on, a redesign replaced the Next button of the signup form with an arrow icon inside a div with a click handler. Our axe scan ran on the home page only, and it would have passed the new step anyway: axe does not see click handlers, so a div without a role raises no violation. Twenty-six days later a user wrote to support that the Tab key skipped the arrow and her screen reader stopped at step one. We made the div a button named Next, which the axe rule for button names now covers, and added this test:

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

for (const path of ['/', '/signup', '/signup/team']) {
  test(`no axe violations on ${path}`, async ({ page }) => {
    await page.goto(path);
    const results = await new AxeBuilder({ page })
      .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'])
      .analyze();
    expect(results.violations).toEqual([]);
  });
}

test('signup continues from the keyboard', async ({ page }) => {
  await page.goto('/signup');
  await page.getByLabel('Work email').fill('ana@example.com');
  await page.keyboard.press('Tab');
  await expect(page.getByRole('button', { name: 'Next' })).toBeFocused();
  await page.keyboard.press('Enter');
  await expect(page.getByRole('heading', { name: 'Your team' })).toBeVisible();
});

The keyboard test fails on the old div, because Tab never focuses it.

Sources

FAQ

Do we need to test accessibility with real screen readers?: Yes, once what NVDA or VoiceOver announces on each step matters, a person runs the main flows with both before each major release, because no automated tool reports announcements. One pass with NVDA on Windows and one with VoiceOver on macOS covers both main desktop platforms.
Can automated tools replace manual accessibility testing?: No, automated tools cannot replace manual accessibility testing, because tools miss barriers that need judgment. In a UK Government Digital Service test, 10 tools together missed 29% of 143 barriers. Automate the axe scan and the keyboard test; add a manual screen reader pass once announcements matter.
Should accessibility tests run in CI?: Yes, run axe scans of the main pages in CI and fail the build on a violation, so a new barrier blocks the change that adds it. Run the keyboard test of each main flow in the same CI job.