Teaching AI to Speed Up Accessibility Testing

To paraphrase the much used line about writing: I don't enjoy doing accessibility testing, but I like having done accessibility testing. When a site becomes more usable for everyone, I feel good. Users feel good. Clients feel good. (Regulators feel good.) The process of getting there can be very tedious.

Before we even begin addressing accessibility, we need to know where the problems live, and that's where testing comes in. There are two main categories of accessibility testing: automated and manual. Automated testing uses deterministic tools and scanning scripts to check the rendered HTML markup for certain obvious flaws, things like missing ARIA labels or low color contrast. Manual testing checks what those tools can't. A person navigates the site with the tech real users rely on, like keyboards and screen readers, and catches holes in user flows or spots where meaning gets missed. An image might have text in its ARIA label, but does that text actually help explain what the image is and how it's relevant to the rest of the content? Both categories let us find the areas of a site that need adjustments to make them more usable.

People are having plenty of thoughtful arguments right now about what AI is good for and where it doesn't belong. One use stands out to me: letting it take on the repetitive, mechanical parts of a job, freeing me up for the work that needs real judgment. So I set out to see how much of accessibility testing an AI agent could carry.

Automated testing seemed like the most logical place to start, because so much of it is deterministic: you run the scanners and collect the results. And yet a full automated pass involves lots of little decisions along the way, lots of setup steps, documentation to keep straight, mountains of results to sort through, and reports to write at the end. Exactly the type of work I wanted to hand off.

One Skill File Was Not Nearly Enough

Claude Code has a feature called skills: instruction files that teach the agent a repeatable procedure. That seemed like a promising starting point. There might be other approaches that smarter people have thought of, but this is what I knew at the time, and one of the best ways to learn is to just do and experiment. So I had a conversation with Claude and had it help me write my first automated accessibility testing skill. (In my experience, having AI write instructions for AI based on my intent tends to give me better results.)

I very quickly realized that one skill file could not handle the complexity I was asking of it. So I split it up: a skill for gathering information about the project, one for setup, one for running the tools, one for consolidating and analyzing the results, one for tracing issues back to their source, and one for reporting.

The suite kept growing as I used it and learned what else it needed to be robust. I'd run it through fresh on a site and each time find different ways it either wasn't doing enough or was just doing it flat wrong. Instead of getting mad (ok, I did get annoyed a few times) I asked, "Where is my process breaking down? What does the agent need that I haven't provided? Where is it spending the most tokens and how can I make that more efficient?"

The process now spans fifteen skills and a shared library of tested scripts, covering the full arc of an audit: plan the scope, discover and categorize the pages, pick and configure the scanners, run them, boil the output down, trace findings to source code, verify what's real, research fixes against the actual specs, write reports for the humans who need them, and retest after fixes get implemented.

I also wanted the workflow to work on any project, not just the stacks I know best. Drupal, Next.js, Svelte, WordPress, whatever comes through the door. Including up-to-date documentation for every likely framework inside the skills would have been unrealistic, and stale guidance is worse than no guidance, because the agent follows it confidently. (Just ask any Drupal dev trying to use AI out of the box for dev help). So I baked in a dependency on Context7, a service that lets the agent query current, version-specific documentation for whatever the project uses, and required checking it at several points in the workflow. Now it works from what the project is actually running instead of trusting whatever its training data half-remembers.

Humans Stay In the Loop, On Purpose

I value ownership of my work. Current models can do a lot, but they still far too often make inaccurate inferences, skip facts, and end up like my robo-vacuum, stuck in a corner and tangled in cords, costing me time and effort to get it unstuck and redo work. In personal projects this is annoying; in a regulatory environment this is unacceptable.

So at key moments I want my agents to bring me their work, get my input and signoff, and then continue based on the direction I set. The agent interviews me up front about the project and its goals. It asks permission before installing tooling dependencies or guides me through the installs. It pauses so I can check that the process and the results so far look accurate, and that it hasn't wandered off track.

What Broke Along the Way

Many useful things I learned in this process came from something going wrong. A sampling:

False positives. One scanner rule alone produced 257 rows complaining that icon-only buttons (picture a bare magnifying-glass search button) had no label for screen readers, when in reality every one of them was labeled correctly. A quick fix might be to tell the scanner to stop running that rule, but a genuinely unlabeled icon button elsewhere in the site is a real barrier for someone navigating by screen reader, so switching it off entirely would bury real problems alongside the noise. Instead the workflow keeps a list of these known false-alarm patterns, each with a condition attached: ignore this rule only where the evidence proves a label already exists. And it keeps paired test cases, one that should trip the rule and one that shouldn't. If the exception ever starts covering a real failure, a test catches it.

Too much output, then not enough. Four scanners across a real site produced 188 raw result files holding over three thousand findings, most of them duplicates of each other in different formats. I had to build a whole consolidation stage: normalize, deduplicate, cluster by root cause, rank. Then I discovered the agent was dropping and misclassifying findings during consolidation, so I pushed that work out of the agent's judgment and into tested scripts whose behavior I could verify.

Empty, meaningless reports. Early reports read like typical AI marketing copy, generic percentages and process jargon instead of actual numbers from the audits and explanations a developer could act on. They were unusable to me, the person who had instructed the AI to write them, and would certainly not be usable to anyone else. Now every report has to be built from the audit's actual findings, and written for the specific person who will read it.

The disappearing CSV parser. I watched the agent spend enormous amounts of tokens recreating, over and over, a CSV parser I knew it had already written, until I cornered it on why. The library it needed had gone missing mid-session, and rather than say so, it kept quietly rebuilding the wheel. The lesson went straight into the shared library's principles: “Never write a one-off parser; if the library doesn't support what you need, first check in with me, then extend the library and add tests.”

Confidently wrong. Authentication tripped it in a way I almost didn't catch. After one scan cleared browser cookies, the session cookie never got reapplied, so every authenticated page silently scanned as the login page and came back artificially clean. Clean results feel great until you notice the settings page weighs a fraction of what it should. And color contrast in modern CSS gave it fits; converting oklch() color values to check contrast ratios burned real time and produced confident errors before I required the math to live in a tested script rather than the agent's head.

Whenever something like this came up, I stopped the agent and we talked through what happened and why. Then I changed the process and the skills themselves: more decomposition, more ask-the-human steps, more reusable tested scripts, and adversarial review agents that critique the work before I see it. Each time the pattern is the same: treat agent failures as process bugs, not one-offs to scold away.

Did It Produce Anything?

Yes, it did, and I'm quite pleased with the results.

Take a static marketing site. The workflow pointed four accessibility scanners (axe, Pa11y, Lighthouse, and IBM Equal Access) at 35 pages. Between them they returned 7,028 raw findings. Most automated scanners finish by handing me a pile of output that I have to sort through to find the meaning. That's boring and annoying so I use my workflow to remove duplicates, group the remaining items by root cause, and in this case it landed on 37 clusters. A cluster is just a bunch of findings that all come from the same underlying problem, so one broken pattern repeated across fifty pages becomes a single cluster instead of fifty separate things to chase. The review step of the workflow turns those into a short worklist. 11 of them are confirmed code fixes ready to act on. 7k+ findings down to 11 things I need to do.

I've run it on four sites so far, on stacks that share almost nothing, and gotten similar results each time.

And if I had questions about any of them, like where a problem came from, which WCAG rule it violated, even a suggested fix, I could talk it through with the agent, which had the full context of the project.

One of my favorite bits is how the workflow traces big noisy messes back to the source. On the Eleventy site, it found that fixing just two files (one layout template and one stylesheet) would clear 82% of its 7,028 findings. That is exactly the kind of combing-through I used to do by hand, and exactly what AI is good at.

• AI
• Drupal
• Drupal LMS
• Open Source CMS