GitHub Pilots Accessibility Agent for Copilot, Key Lessons
GitHub relies on agent-based approaches for code creation and editing in various projects. The company now pilots an experimental general-purpose accessibility agent to support its accessibility goals. This agent works within the GitHub Copilot CLI and the Copilot VS Code extension.
The agent pursues two primary objectives. First, it delivers dependable, immediate responses to accessibility queries for engineers. Second, it detects and fixes straightforward, clear-cut accessibility problems before code reaches production. For the second goal, the agent scans pull requests that alter front-end code.
So far, the agent has assessed 3,535 pull requests and achieved a 68% resolution rate. The most frequent issues, in sequence, involve making structure and relationships clear to assistive technologies, providing clear and concise names for interactive controls, ensuring users know about important announcements, offering text alternatives for non-text content, and managing keyboard focus through pages and views in a logical order. These fixes eliminate automatic barriers that could block GitHub use for people depending on assistive technology.
Mindset for Agent Development
The social model of disability shows that access barriers, and thus impairments, arise from how environments get built. This idea extends to digital products. GitHub's accessibility agent aims to support colleagues' work, not fix accessibility alone. It helps remove barriers created during user interface construction.
The agent cannot handle every possible case as a complete fix. Teams recognize this limit, define its role clearly, and communicate it. This clarity allowed quicker rollout and greater team support.
Past Efforts and Data Foundations
The European Accessibility Act applies now. Title II of the Americans with Disabilities Act will define WCAG 2.1 AA as the standard starting April 2027. LLM agents can access and act on the accessibility tree.
Companies face risks without prior manual accessibility fixes. GitHub maintains a solid system for tracking issues and confirming repairs. It features a structured reporting template, reproduction steps, metadata on severity, service area, and WCAG criterion, links to fixing pull requests, and acceptance criteria. All issues sit in one repository.
This setup, created before LLMs surged, suits agent training due to its consistency. The agent studies these issues to pull related code and language examples. LLMs' fuzzy matching helps here as a strength.
Detailed instructions alone fail for specialized tasks like accessibility. LLMs often generate poor practices from training on old bad code. Manual issue catalogs and fixes provide strong, context-specific examples matching GitHub's style. This data forms the agent's best resource.
Efficient Token Use with Sub-Agents
Accessibility spans code, design, writing, and more. It requires full context for advice. Agents thus use many tokens, leading to errors, delays, and costs.
The agent began as one unit but shifted to sub-agents for better control. Rather than many parallel sub-agents, it uses two: a passive reviewer and researcher, and an active implementer. They operate in sandboxes without direct contact. They produce structured, templated outputs for the main agent to process, check, and direct.
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
This setup offers escalation points for human needs like severe WCAG failures or tough patterns. It handles complexity by switching to guidance mode for hard code. The reviewer lists all findings; the main agent picks relevant ones. Direct sub-agent talk would waste resources and lose traceability for audits.
Agents follow fixed instruction sequences. Phases have ordered steps with tools and skills. This order matches manual audit, fix, and report processes.
Template schemas ensure focus: reviewer schema for audits and info, implementer for fixes. Without them, agents wander, waste tokens, hallucinate, or alter code wrongly.
Limitations and Ongoing Improvements
Agents have clear weaknesses. Users, especially novices, might not question bad outputs.
A simple shell script scores code complexity with heuristics. High scores trigger no-fixes mode; users consult the accessibility team.
The agent skips high-risk patterns like drag and drop, toasts, rich text editors, tree views, and data grids. These need deep detail beyond current LLM skills.
Instructions curb LLMs' urge to generate code always. Anti-gaming rules block workarounds.
Only 35 of 55 WCAG A and AA criteria work with automated checks, leaving about 36% manual. Agents help but imperfectly. Early design pairing with accessibility experts prevents issues.
Teams manually check agent outputs and reviewer feedback for tweaks.
Key Takeaways from the Experiment
The agent augments, not replaces, accessibility work. It excels with organization-specific issue data. Sub-agents cut token use. Linear steps boost accuracy. Templates maintain consistency. It knows limits and directs to humans. Reviews refine it.
Work continues to make GitHub inclusive. GitHub plans to open source the agent to aid open source accessibility.