test_runner: improve coverage failure diagnostics by trivikr · Pull Request #64050 · nodejs/node

trivikr · 2026-06-21T21:54:39Z

Improve diagnostics for flaky parallel/test-runner-coverage failures.

This changes coverage file parsing to include the specific V8 coverage file path
when JSON parsing fails, including a clearer message for empty coverage files.

It also updates test/parallel/test-runner-coverage.js so missing coverage
report assertions include the spawned child process status, signal, stdout, and
stderr. This should make future CI failures actionable instead of only showing
that stdout.includes(report) was false.

Refs: https://github.com/nodejs/reliability/issues?q=sort%3Aupdated-desc%20is%3Aissue%20state%3Aopen%20%22parallel%2Ftest-runner-coverage%22

Example

not ok 3920 parallel/test-runner-coverage
  ---
  duration_ms: 2735.17800
  severity: fail
  exitcode: 1
  stack: |-
    Test failure: 'coverage is reported and dumped to NODE_V8_COVERAGE if present'
    Location: test/parallel/test-runner-coverage.js:102:11
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(result.stdout.toString().includes(report))
    
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:114:5)
        at Test.runInAsyncScope (node:async_hooks:226:14)
        at Test.run (node:internal/test_runner/test:1382:25)
        at Test.start (node:internal/test_runner/test:1242:17)
        at TestContext.test (node:internal/test_runner/test:394:20)
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:102:11)
        at Test.runInAsyncScope (node:async_hooks:226:14)
        at Test.run (node:internal/test_runner/test:1382:25)
        at Test.processPendingSubtests (node:internal/test_runner/test:960:18)
        at Test.postRun (node:internal/test_runner/test:1522:19) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '==',
      diff: 'simple'
    }
    
    Test failure: 'coverage is reported without NODE_V8_COVERAGE present'
    Location: test/parallel/test-runner-coverage.js:120:11
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(result.stdout.toString().includes(report))
    
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:132:5)
        at Test.runInAsyncScope (node:async_hooks:226:14)
        at Test.run (node:internal/test_runner/test:1382:25)
        at Test.start (node:internal/test_runner/test:1242:17)
        at TestContext.test (node:internal/test_runner/test:394:20)
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:120:11)
        at async Test.run (node:internal/test_runner/test:1389:7)
        at async Test.processPendingSubtests (node:internal/test_runner/test:960:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '==',
      diff: 'simple'
    }
    
    Test failure: 'coverage is reported and dumped to NODE_V8_COVERAGE if present'
    Location: test/parallel/test-runner-coverage.js:140:11
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(result.stdout.toString().includes(report))
    
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:152:5)
        at Test.runInAsyncScope (node:async_hooks:226:14)
        at Test.run (node:internal/test_runner/test:1382:25)
        at Test.start (node:internal/test_runner/test:1242:17)
        at TestContext.test (node:internal/test_runner/test:394:20)
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:140:11)
        at Test.runInAsyncScope (node:async_hooks:226:14)
        at Test.run (node:internal/test_runner/test:1382:25)
        at Test.processPendingSubtests (node:internal/test_runner/test:960:18)
        at Test.postRun (node:internal/test_runner/test:1522:19) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '==',
      diff: 'simple'
    }
    
    Test failure: 'coverage is reported without NODE_V8_COVERAGE present'
    Location: test/parallel/test-runner-coverage.js:158:11
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(result.stdout.toString().includes(report))
    
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:169:5)
        at Test.runInAsyncScope (node:async_hooks:226:14)
        at Test.run (node:internal/test_runner/test:1382:25)
        at Test.start (node:internal/test_runner/test:1242:17)
        at TestContext.test (node:internal/test_runner/test:394:20)
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:158:11)
        at async Test.run (node:internal/test_runner/test:1389:7)
        at async Test.processPendingSubtests (node:internal/test_runner/test:960:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '==',
      diff: 'simple'
    }
    
    Test failure: 'single process coverage is the same with --test'
    Location: test/parallel/test-runner-coverage.js:176:1
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(result.stdout.toString().includes(report))
    
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:190:3)
        at Test.runInAsyncScope (node:async_hooks:226:14)
        at Test.run (node:internal/test_runner/test:1382:25)
        at Test.processPendingSubtests (node:internal/test_runner/test:960:18)
        at Test.postRun (node:internal/test_runner/test:1522:19)
        at Test.run (node:internal/test_runner/test:1447:12)
        at async Test.processPendingSubtests (node:internal/test_runner/test:960:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '==',
      diff: 'simple'
    }
    
    Test failure: 'coverage is combined for multiple processes'
    Location: test/parallel/test-runner-coverage.js:195:1
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(result.stdout.toString().includes(report))
    
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:229:3)
        at Test.runInAsyncScope (node:async_hooks:226:14)
        at Test.run (node:internal/test_runner/test:1382:25)
        at Test.processPendingSubtests (node:internal/test_runner/test:960:18)
        at Test.postRun (node:internal/test_runner/test:1522:19)
        at Test.run (node:internal/test_runner/test:1447:12)
        at async Test.processPendingSubtests (node:internal/test_runner/test:960:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '==',
      diff: 'simple'
    }
    
    Test failure: 'coverage reports on lines, functions, and branches'
    Location: test/parallel/test-runner-coverage.js:275:1
    SyntaxError: Unexpected end of JSON input
        at JSON.parse (<anonymous>)
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:288:25)
        at Test.runInAsyncScope (node:async_hooks:226:14)
        at Test.run (node:internal/test_runner/test:1382:25)
        at Test.processPendingSubtests (node:internal/test_runner/test:960:18)
        at Test.postRun (node:internal/test_runner/test:1522:19)
        at Test.run (node:internal/test_runner/test:1447:12)
        at async Test.processPendingSubtests (node:internal/test_runner/test:960:7)
    
    Test failure: 'coverage with ESM hook - source irrelevant'
    Location: test/parallel/test-runner-coverage.js:336:1
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(result.stdout.toString().includes(report))
    
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:369:3)
        at Test.runInAsyncScope (node:async_hooks:226:14)
        at Test.run (node:internal/test_runner/test:1382:25)
        at Test.processPendingSubtests (node:internal/test_runner/test:960:18)
        at Test.postRun (node:internal/test_runner/test:1522:19)
        at Test.run (node:internal/test_runner/test:1447:12)
        at async Test.processPendingSubtests (node:internal/test_runner/test:960:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '==',
      diff: 'simple'
    }
    
    Test failure: 'coverage with ESM hook - source transpiled'
    Location: test/parallel/test-runner-coverage.js:373:1
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(result.stdout.toString().includes(report))
    
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:404:3)
        at Test.runInAsyncScope (node:async_hooks:226:14)
        at Test.run (node:internal/test_runner/test:1382:25)
        at Test.processPendingSubtests (node:internal/test_runner/test:960:18)
        at Test.postRun (node:internal/test_runner/test:1522:19)
        at Test.run (node:internal/test_runner/test:1447:12)
        at async Test.processPendingSubtests (node:internal/test_runner/test:960:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '==',
      diff: 'simple'
    }
    
    Test failure: 'coverage with excluded files'
    Location: test/parallel/test-runner-coverage.js:408:1
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(result.stdout.toString().includes(report))
    
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:438:3)
        at Test.runInAsyncScope (node:async_hooks:226:14)
        at Test.run (node:internal/test_runner/test:1382:25)
        at Test.processPendingSubtests (node:internal/test_runner/test:960:18)
        at Test.postRun (node:internal/test_runner/test:1522:19)
        at Test.run (node:internal/test_runner/test:1447:12)
        at async Test.processPendingSubtests (node:internal/test_runner/test:960:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '==',
      diff: 'simple'
    }
    
    Test failure: 'coverage with included files'
    Location: test/parallel/test-runner-coverage.js:443:1
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(result.stdout.toString().includes(report))
    
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:475:3)
        at Test.runInAsyncScope (node:async_hooks:226:14)
        at Test.run (node:internal/test_runner/test:1382:25)
        at Test.processPendingSubtests (node:internal/test_runner/test:960:18)
        at Test.postRun (node:internal/test_runner/test:1522:19)
        at Test.run (node:internal/test_runner/test:1447:12)
        at async Test.processPendingSubtests (node:internal/test_runner/test:960:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '==',
      diff: 'simple'
    }
    
    Test failure: 'coverage with included and excluded files'
    Location: test/parallel/test-runner-coverage.js:480:1
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(result.stdout.toString().includes(report))
    
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:509:3)
        at Test.runInAsyncScope (node:async_hooks:226:14)
        at Test.run (node:internal/test_runner/test:1382:25)
        at Test.processPendingSubtests (node:internal/test_runner/test:960:18)
        at Test.postRun (node:internal/test_runner/test:1522:19)
        at Test.run (node:internal/test_runner/test:1447:12)
        at async Test.processPendingSubtests (node:internal/test_runner/test:960:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '==',
      diff: 'simple'
    }
    
    Test failure: 'correctly prints the coverage report of files contained in parent directories'
    Location: test/parallel/test-runner-coverage.js:514:1
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(result.stdout.toString().includes(report))
    
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:550:3)
        at Test.runInAsyncScope (node:async_hooks:226:14)
        at Test.run (node:internal/test_runner/test:1382:25)
        at Test.processPendingSubtests (node:internal/test_runner/test:960:18)
        at Test.postRun (node:internal/test_runner/test:1522:19)
        at Test.run (node:internal/test_runner/test:1447:12)
        at async Test.processPendingSubtests (node:internal/test_runner/test:960:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '==',
      diff: 'simple'
    }
    
    Test failure: 'coverage with directory and file named "file"'
    Location: test/parallel/test-runner-coverage.js:555:1
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    1 !== 0
    
        at TestContext.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-runner-coverage.js:566:10)
        at Test.runInAsyncScope (node:async_hooks:226:14)
        at Test.run (node:internal/test_runner/test:1382:25)
        at Test.processPendingSubtests (node:internal/test_runner/test:960:18)
        at Test.postRun (node:internal/test_runner/test:1522:19)
        at Test.run (node:internal/test_runner/test:1447:12)
        at async Test.processPendingSubtests (node:internal/test_runner/test:960:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: 1,
      expected: 0,
      operator: 'strictEqual',
      diff: 'simple'
    }

Assisted-by: openai:gpt-5.5

nodejs-github-bot · 2026-06-21T21:54:44Z

Review requested:

@nodejs/test_runner

Include the coverage filename when parsing V8 coverage data fails, and make test-runner coverage assertions report spawned child process output when the expected coverage report is missing. Signed-off-by: Kamat, Trivikram <16024985+trivikr@users.noreply.github.com> Assisted-by: openai:gpt-5.5

nodejs-github-bot · 2026-06-22T14:17:18Z

CI: https://ci.nodejs.org/job/node-test-pull-request/74359/

nodejs-github-bot added needs-ci PRs that need a full CI run. test_runner Issues and PRs related to the test runner subsystem. labels Jun 21, 2026

trivikr added flaky-test Issues and PRs related to the tests with unstable failures on the CI. request-ci Add this label to start a Jenkins CI on a PR. labels Jun 21, 2026

trivikr force-pushed the test-runner-coverage branch from 4b8ec85 to aeff006 Compare June 21, 2026 22:15

MoLow approved these changes Jun 22, 2026

View reviewed changes

trivikr added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Jun 22, 2026

atlowChemi approved these changes Jun 22, 2026

View reviewed changes

github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jun 22, 2026

This comment was marked as outdated.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test_runner: improve coverage failure diagnostics#64050

test_runner: improve coverage failure diagnostics#64050
trivikr wants to merge 1 commit into
nodejs:mainfrom
trivikr:test-runner-coverage

trivikr commented Jun 21, 2026

Uh oh!

nodejs-github-bot commented Jun 21, 2026

Uh oh!

This comment was marked as outdated.

nodejs-github-bot commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

trivikr commented Jun 21, 2026

Uh oh!

nodejs-github-bot commented Jun 21, 2026

Uh oh!

This comment was marked as outdated.

nodejs-github-bot commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants