How much accessibility testing do I really need to do?

Checklist Problems!

Human Error

• Possible misinterpretations of Success Criteria
• Insufficient Success Criteria summary
• Insufficient check categorization
• Determining which Success Criteria covers which check
• Omitting necessary checks
• Inconsistency in how checks are summarized
• Inconsistent use/identification of must (violations) and should (best practices) checks

Methodology

• Biased for desktop
• Biased for HTML

Automated Tool Related

• Constant possibility of false positives
• Mismatches between how I classified checks and how automated tool authors classified checks
• Not thorough analysis of automated tool checks
• AI Inspector mixes violations and possible violations (i.e. Manual Checks)
• In some cases, not to same level of granularity as automated tool checks