Skip to content

Sort axe report violations by severity, then standard#14676

Open
cwickham wants to merge 3 commits into
axe/vendor-axe-corefrom
axe-report-sort-order
Open

Sort axe report violations by severity, then standard#14676
cwickham wants to merge 3 commits into
axe/vendor-axe-corefrom
axe-report-sort-order

Conversation

@cwickham

@cwickham cwickham commented Jul 13, 2026

Copy link
Copy Markdown
Member

Important

Stacked on #14677 — do not merge before it. The base is axe/vendor-axe-core; when #14677 merges, GitHub retargets this PR to main and it merges cleanly (the changelog conflict between the two is already resolved here).

Description

The axe document reporter currently lists violations in axe-core's own order. This PR sorts them most-important-first, matching axe DevTools' impact-first convention:

  1. Impact: critical → serious → moderate → minor → null/unknown
  2. Conformance standard: WCAG A → AA → AAA → Best Practice → obsolete criteria → untagged
  3. Rule id alphabetically, for deterministic output

The sort happens in createReportElement, so all three report containers (fixed overlay, revealjs report slide, dashboard offcanvas) and the dashboard rescan path inherit it. The reporter sorts a copy of the violations array, so output: json keeps emitting axe's raw result untouched, and console output is unchanged.

The rank helpers (impactRank, standardRank, compareViolations) are exported for unit testing, following the axeConformanceLevel pattern, and share a new parseConformanceTags helper with axeConformanceLevel rather than duplicating the tag regexes. The exported helpers are also intended for reuse by the upcoming project-level accessibility scanner.

Part of the axe accessibility QoL series.

Checklist

I have (if applicable):

  • referenced the GitHub issue this PR closes
  • updated the appropriate changelog in the PR
  • ensured the present test suite passes
  • added new tests
  • created a separate documentation PR in Quarto's website repo and linked it to this PR

Notes: no GitHub issue exists for this change (QoL improvement agreed in the axe PR series plan); no user-facing option is added, so no docs PR is needed. New unit tests in tests/unit/axe-sort.test.ts (8 tests) pass, as do the existing tests/unit/axe-conformance.test.ts tests (9) covering the shared-parser refactor. The existing Playwright axe assertions use toContainText and are order-insensitive, so they pass unchanged.

AI-assisted PR
  • AI tool used: Claude Code
  • Codebase grounding: local clone
  • Human review: I have reviewed, tested, and verified the AI-generated content before submitting.

cwickham added a commit that referenced this pull request Jul 13, 2026
@posit-snyk-bot

posit-snyk-bot commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@cwickham

Copy link
Copy Markdown
Member Author

Added a Playwright regression test (e8c9b3a): a new fixture html/axe-sort-order.qmd triggers image-alt (critical), color-contrast (serious), and heading-order (moderate) — axe-core emits these roughly alphabetically by rule id, which is not severity order, so the test fails if the report stops sorting. Verified passing locally on chromium. Standard tie-breaks within one impact level stay unit-tested only, since reliably triggering two same-impact rules with different standards is fragile against axe-core upgrades.

cwickham added 3 commits July 13, 2026 11:11
The axe document reporter previously listed violations in axe-core's own
order. Sort them most-important-first: by impact (critical, serious,
moderate, minor, then null), then by conformance standard (WCAG A, AA,
AAA, Best Practice, obsolete, untagged), then by rule id for
deterministic output.

Sorting happens in createReportElement, so the fixed overlay, revealjs
report slide, dashboard offcanvas, and dashboard rescans all inherit it.
The reporter sorts a copy, so the json reporter keeps emitting axe's raw
result untouched.

The rank helpers are exported for unit tests (and for reuse by the
upcoming project scanner), alongside a parseConformanceTags helper now
shared with axeConformanceLevel.
The fixture triggers image-alt (critical), color-contrast (serious),
and heading-order (moderate); axe-core emits these roughly
alphabetically by rule id, which is not severity order, so the test
fails if the report stops sorting. Standard tie-breaks within one
impact level are covered by the unit tests instead — triggering two
same-impact rules with different standards is fragile against axe-core
upgrades.
@cwickham cwickham force-pushed the axe-report-sort-order branch from e8c9b3a to c6c63bd Compare July 13, 2026 18:12
@cwickham cwickham changed the base branch from main to axe/vendor-axe-core July 13, 2026 18:12
@cwickham cwickham marked this pull request as ready for review July 13, 2026 18:43
@cwickham cwickham requested a review from cderv July 13, 2026 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants