Parser test coverage: validate against real xcodebuild output and fill coverage gaps

[cameroncooke]

Problem

An audit of the xcodebuild/Swift Testing parsers revealed that synthetic test data was written based on assumed output formats rather than captured real output. Several parser patterns were incorrect and have been fixed in the rendering pipeline PR stack (#320), but significant test coverage gaps remain.

Bugs found and fixed

• Verbose (aka 'func()') suffix broke all Swift Testing result/issue parsers
• Skipped test format was completely wrong (wrong symbol, no quotes, different syntax)
• Parameterized test results and issues silently dropped
• .xcodeproj: error: path-based build errors silently dropped
• Lowercase Test suite from Xcode 26 not matched by stage detection

Remaining coverage gaps

Event parser integration tests (HIGH priority)

• Pending failure duration reverse-ordering (duration before diagnostic)
• flush() orphaned diagnostics that never got a matching duration
• Mixed XCTest + Swift Testing output in one test run
• Swift Testing continuation line integration through event parser

Fixture-based regression tests

• No tests feed real multi-line captured xcodebuild output through the parsers
• SPM example project needs failing, skipped, and parameterized test cases
• Need capture script to generate fixtures from real builds

Line parser edge cases

• Fatal error and linker error paths
• All compile/link/package-resolution pattern variants
• Run state pass-through events, finalize options

Plan

1. Capture real output fixtures from example projects

Add missing test scenarios to SPM example project

example_projects/spm/Tests/TestLibTests/ currently only has passing tests. Add:

• A failing test (@Test func deliberateFailure() with #expect(false)) to produce ✘ result and issue lines
• A disabled/skipped test (@Test(.disabled("reason")) func disabledTest()) to produce ➜ skipped lines
• A parameterized test (@Test(arguments: [1, 0, -1]) func parameterized(value: Int) where one value fails) to produce with N test cases and with N argument value formats
• A named @Suite to produce suite-level result lines
• A test with multiple #expect failures to produce multiple issue lines for a single test

Capture output from real builds

Create a script (extend scripts/capture-xcodebuild-wrapper.ts or add scripts/capture-parser-fixtures.ts) that:

1. Runs swift test --package-path example_projects/spm and captures stdout/stderr
2. Runs swift test --package-path example_projects/spm --verbose for verbose format
3. Runs xcodebuild test against example_projects/iOS_Calculator for XCTest format
4. Runs xcodebuild test against example_projects/macOS for xcodebuild Swift Testing format
5. Runs xcodebuild build with an intentionally broken scheme to capture build error output
6. Saves captured output to src/utils/__fixtures__/parser-audit/

These fixtures serve as the ground truth for parser tests.

2. Event parser integration tests

Priority: HIGH

Pending failure duration reverse-ordering: Feed a Test Case '...' failed (0.5 seconds) line BEFORE the corresponding stderr diagnostic. Verify the failure event gets the correct duration, then verify flush() emits orphaned diagnostics that never got a matching duration.

Flush orphaned diagnostics: Feed a failure diagnostic with no subsequent failed test case line. Call flush(). Verify the test-failure event is emitted without duration.

Mixed XCTest + Swift Testing output: Feed a realistic interleaved sequence through the event parser:

• XCTest Test Case pass/fail lines
• Swift Testing ✔/✘ result lines
• XCTest Executed N tests totals
• Swift Testing ✔ Test run with summary

Verify correct cumulative counts with no double-counting.

Swift Testing continuation line integration: Feed ✘ Test "X" recorded an issue at file:line:col: msg followed by ↳ additional context. Verify the emitted test-failure event has the full concatenated message.

Priority: MEDIUM

xcresultPath detection: Feed a line containing a .xcresult path, verify parser.xcresultPath is populated.

finalize() options: Test emitSummary: false suppresses the summary event. Test tailEvents appends events after summary.

Fatal error path: Feed fatal error: ... with and without file location through the event parser.

Linker error path: Feed common linker error formats (Undefined symbols for architecture arm64, ld: error: undefined symbol) and verify they produce compiler-error events.

Skipped XCTest case: Feed Test Case '...' skipped through the event parser and verify skippedCount increments.

3. Fixture-based regression tests

Create parameterized tests that feed captured fixture files through the full parser pipeline (event parser + run state) and assert the complete event sequence:

describe.each(fixtureFiles)('parser regression: %s', (fixtureName) => {
  it('produces expected events', async () => {
    const { stdout, stderr } = loadFixture(fixtureName);
    const events = parseFullOutput(stdout, stderr);
    expect(events).toMatchSnapshot();
  });
});

Fixtures to create:

• spm-test-pass.txt -- all tests pass (swift test)
• spm-test-mixed.txt -- mix of pass/fail/skip (swift test)
• spm-test-verbose.txt -- verbose mode output
• spm-test-parameterized.txt -- parameterized test output
• xcodebuild-build-success.txt -- successful build (xcodebuild)
• xcodebuild-build-error.txt -- build with errors (xcodebuild)
• xcodebuild-test-xctest.txt -- XCTest results (xcodebuild)
• xcodebuild-test-swift-testing.txt -- Swift Testing via xcodebuild
• xcodebuild-test-mixed.txt -- mixed XCTest + Swift Testing

4. Additional line parser test coverage

xcodebuild-line-parsers.ts

• parseBuildErrorDiagnostic: Test the new path-based error format (/path/to/Project.xcodeproj: error: Missing package product)
• parseBuildErrorDiagnostic: Test fatal error: with and without location
• compilePatterns: Test SwiftCompile (modern Xcode) in addition to CompileSwift
• compilePatterns: Test ProcessInfoPlistFile, CodeSign, CompileAssetCatalog, ProcessProductPackaging
• packageResolutionPatterns: Test Fetching from, Checking out, Creating working copy, Updating https://
• Stage detection: Test Test suite (lowercase, Xcode 26) alongside Test Suite (uppercase)

xcodebuild-run-state.ts

• Pass-through events: Verify status-line, section, detail-tree, table, file-ref, test-discovery are passed through (currently only header and next-steps are tested)
• finalize with emitSummary: false: Verify no summary event
• finalize with tailEvents: Verify tail events appended after summary

5. Parser versioning prep (optional, future)

If/when Apple changes output formats in a new Xcode version:

1. Extract current regex patterns into a versioned profile object (xcode16Profile)
2. Event parser accepts profile as constructor parameter
3. Version detection at startup selects correct profile
4. Fixture tests run captured output against corresponding profile
5. Latest profile serves as fallback for unknown versions

Execution order

1. Add test cases to SPM example project (prerequisite for fixture capture)
2. Create/extend capture script
3. Run captures, save fixtures
4. Write fixture-based regression tests
5. Write missing unit tests for parser edge cases
6. Write event parser integration tests