Skip to content

chore(ci): e2e report improve#2355

Open
universal-itengineer wants to merge 10 commits into
mainfrom
chore/ci/e2e-report-improve
Open

chore(ci): e2e report improve#2355
universal-itengineer wants to merge 10 commits into
mainfrom
chore/ci/e2e-report-improve

Conversation

@universal-itengineer
Copy link
Copy Markdown
Member

@universal-itengineer universal-itengineer commented May 15, 2026

Description

Improve the E2E messenger report generated by CI:

  • add a Reason column to the failed-tests thread table;
  • rename the failed-tests table column from Test group to Tests;
  • link cluster names in failed-tests thread replies to the corresponding workflow/job URL;
  • hide the Errors column in the main test-results table when there are no Ginkgo errors;
  • calculate Success Rate from executed specs only: passed / (passed + failed + errors);
  • add report table header emojis;
  • use the full Ginkgo Failure.Message as the failed-test reason;
  • avoid wrapping SSH command execution errors with %w so Ginkgo does not dump *exec.ExitError internals into the report;
  • capture Ginkgo stdout/stderr as a fallback report source and surface suite-level failures such as SynchronizedBeforeSuite when the JSON report is missing or contains only setup failure data.

Why do we need it, and what problem does it solve?

The current E2E notification thread only lists failed test groups and does not show why a test failed. This forces engineers to open CI artifacts/logs even for simple failures.

The report also displayed skipped specs as part of the success-rate denominator, which made Ginkgo success rate look worse than the actual executed-spec result. In addition, SSH command failures could include noisy Go internals in Failure.Message because *exec.ExitError was wrapped with %w.

When E2E fails in SynchronizedBeforeSuite, Ginkgo may skip all specs and the JSON report may be absent or contain no regular test failures. Previously the messenger report could only show a generic storage failure, for example nfs with E2E TEST FAILED, without the actual setup failure reason.

This change makes E2E notifications more actionable directly in the messenger thread and keeps the main summary easier to read.

What is the expected result?

  1. Run the E2E report generation with a Ginkgo JSON report containing failed specs.
  2. Check the main messenger report:
    • the Errors column is hidden when all error counts are zero;
    • Success Rate is calculated without skipped specs;
    • table headers include emojis.
  3. Check the failed-tests thread:
    • cluster names are links;
    • the table has Tests and Reason columns;
    • Reason contains the Ginkgo Failure.Message text.
  4. SSH command failures are rendered as useful text instead of dumping *exec.ExitError internals.
  5. If Ginkgo fails in SynchronizedBeforeSuite and no JSON report is created, the report is still generated from the captured Ginkgo output and the failed-tests thread shows the suite-level failure reason. Suite-level failures do not change spec metrics such as passed, failed, errors, or total.

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

Changelog entries

section: ci
type: fix
summary: Improve E2E messenger report failure reasons, suite setup failure reporting, and success-rate calculation.
impact_level: low

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
@universal-itengineer universal-itengineer added this to the v1.9.0 milestone May 15, 2026
@universal-itengineer universal-itengineer marked this pull request as ready for review May 15, 2026 15:08
Capture Ginkgo output as a fallback report source so BeforeSuite failures are surfaced even when the JSON report is missing.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Comment thread .github/scripts/js/e2e/report/messenger/markdown.js Outdated
Comment thread .github/scripts/js/e2e/report/messenger/markdown.js Outdated
- Merge Ginkgo JSON and output parsers behind a single parseGinkgoFile
  helper with declarative source descriptors.
- Build a single suiteNodePattern and extract small predicates for the
  Ginkgo stdout reason block.
- Show the suite node name (e.g. SynchronizedAfterSuite) as the failed
  group in the messenger thread instead of "Unknown".

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
The previous suite node regex was too permissive: it matched a plain
"[SynchronizedBeforeSuite]" section header that appears in Ginkgo stdout
before the actual "[FAILED]" line. As a result, the log-only fallback
could not extract suite setup failures from real CI output.

Tighten the pattern to require either "[X] [FAILED]" (body) or
"[FAIL] [X]" (summary) and add a regression test based on a real
BeforeSuite timeout from CI.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
- Describe the test-results table as a single columns array with header,
  alignment, and value functions so the "Errors" column is just one
  entry that is appended conditionally, instead of duplicating the row
  template for the with- and without-errors variants.
- Move getFailedTestEntries inside summarizeFailedTestGroups so the
  caller passes a single report argument instead of pre-extracting
  entries on every call site.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Document the new helpers introduced by recent refactors:

- cluster-report.js: findGinkgoOutput, emptyParsedReport, parseGinkgoFile
  plus a GinkgoSourceDescriptor typedef used by ginkgoJsonSource and
  ginkgoOutputSource.
- shared/ginkgo-report-utils.js: findFailedSuiteNode,
  extractFailureReasonFromOutput, parseGinkgoOutput.
- messenger/markdown.js: buildTestResultsColumns, buildMarkdownRow,
  summarizeFailedTestGroups plus a TestResultsColumn typedef.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants