Tuning Accessibility Alert Thresholds: Signal Without the Noise
Alert fatigue kills monitoring programs. Here is how to set severity thresholds so the alerts you get are the ones worth acting on.
The fastest way to kill an accessibility monitoring program is to flood the team with alerts nobody acts on. Within a month, the channel gets muted and the violations pile up silently. Good alerting is not about catching everything. It is about making sure every alert that fires is worth someone's attention. Here is how to tune thresholds so the signal survives.
Separate new from known
The single highest-value alert is a new violation on a page that was clean yesterday. That means a release introduced a regression, and the fix is usually small because the change is fresh. Everything else is triage material.
Route new-violation alerts to the team channel immediately. Route known-issue recurrences to a weekly digest. The moment a developer learns that an alert means 'something you just shipped broke,' response times collapse from weeks to hours. Make the alert message itself actionable: include the page, the rule, and a direct link to the offending element so the fix starts from the notification, not from a search.
Set severity by user impact, not by rule count
A missing form label on a checkout field blocks a task. A duplicate id in the footer does not. Yet many tools score them similarly because they count rule hits. Override the defaults: map each rule to the task it can break, and set thresholds on task-blocking rules only for real-time alerts.
A practical starting point is three tiers. Tier one, page immediately: keyboard traps, missing labels on transactional forms, broken skip links. Tier two, same sprint: contrast failures on primary actions, heading order breaks. Tier three, backlog: redundant links, minor landmark issues.
Throttle by page, not just by rule
Ten violations of the same rule on one template are one bug, not ten alerts. Group alerts by template and rule so a single bad deploy produces one ticket instead of fifty notifications. The deduplication key should be rule plus template, not rule plus URL.
Also set quiet hours for non-tier-one alerts. Nobody fixes a landmark issue at 2 AM, and a 2 AM page trains people to ignore the system. Tier one pages during business hours, digest for the rest.
Review the thresholds quarterly
Alert configs rot. Rules get fixed upstream, templates change, and thresholds set six months ago no longer match reality. Once a quarter, pull the alert log and ask two questions: which alerts led to a fix, and which were ignored. Delete or demote the ignored ones.
Track the fix rate per alert type. Anything below about a third is noise by definition. Either the threshold is wrong, the routing is wrong, or the rule does not matter to your users. Adjust until the alerts you send are the alerts that get fixed.
Closing the loop: measure whether alerts work
An alerting system needs its own metrics. Track two numbers monthly: the percentage of alerts that lead to a fix, and the median time from alert to fix. If the fix rate is high and the time is short, the system is working. If either degrades, something changed and the thresholds need attention.
Share these numbers with the team. When developers see that alerts lead to fixes quickly, trust in the system grows and response times improve further. It is a virtuous cycle, but only if someone is watching the metrics. Assign that job explicitly, because unowned metrics rot.
Frequently asked questions
Should we alert on every WCAG failure the scanner finds?
No. Scanners report everything they can detect, which includes many issues with no user impact. Alert on regressions and task-blocking rules. Put the full scan output in a dashboard for periodic review instead.
How do we handle third-party widgets that trigger alerts?
Tag violations by source, first-party versus third-party. Alert on third-party regressions so you have a record to take to the vendor, but route them to a separate queue so they do not drown out your own bugs.
What is a healthy alert volume?
Low enough that every alert gets read. For most teams that means a handful per week of actionable alerts, not dozens per day. If the team mutes the channel, the volume is wrong regardless of what the tool says.
Frequently asked questions
What is alert threshold tuning in accessibility monitoring?
It is setting how many new findings, or what severity, must appear before an alert fires and pages someone. Thresholds that are too loose flood the team until alerts get ignored. Thresholds that are too strict let a real blocker sit quietly for weeks.
How do you know a threshold is too loose?
When the team stops reading alerts, or when most alerts close with no action. That pattern means the signal to noise ratio is wrong: routine minor findings should roll into a digest while only blockers and regressions alert immediately.
How often should thresholds be reviewed?
After every major site change and at least quarterly. A threshold tuned for a quiet period will misfire during a sale or a redesign, and one tuned during a noisy launch will hide problems once things calm down.