You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When pa11y scans a URL and returns a scanning error, the CI job fails. But when pa11y fails to scan any urls and returns a generic Failed to run error, the CI job still succeeds.
I propose failing the entire CI job in this scenario, in order to prevent silent failures.
User story
As a developer, I want to know if my build is failing so I can prevent a11y bugs from entering production.
Other considerations and design implementation suggestions
Note that existing behavior for pa11y job in CI (before and after the recent pa11y bug) is to SIGHUP regardless if the URL scans succeed. I'm not sure if this behavior was intended or correct, but a SIGHUP may be causing the job to succeed in all scenarios.
The text was updated successfully, but these errors were encountered:
Feature suggestion
When pa11y scans a URL and returns a scanning error, the CI job fails. But when pa11y fails to scan any urls and returns a generic
Failed to run
error, the CI job still succeeds.I propose failing the entire CI job in this scenario, in order to prevent silent failures.
User story
As a developer, I want to know if my build is failing so I can prevent a11y bugs from entering production.
Other considerations and design implementation suggestions
Note that existing behavior for pa11y job in CI (before and after the recent pa11y bug) is to SIGHUP regardless if the URL scans succeed. I'm not sure if this behavior was intended or correct, but a SIGHUP may be causing the job to succeed in all scenarios.
The text was updated successfully, but these errors were encountered: