Skip to content

[Enhancement] Format junit reports for clarity #659

@kumvprat

Description

@kumvprat

Is your feature request related to a problem? Please describe.

When using cfn-guard validate with --output-format junit, the generated JUnit XML reports contain non-compliant logs but lack line number and column information in the proper sections of the JUnit report structure. This makes it difficult for CI/CD tools and IDEs to provide precise navigation to the exact location of compliance violations in CloudFormation templates.

Currently, users need to implement custom post-processing logic to parse the failure messages and extract location information. For example, in the AWS CDK Security Guardian tool, we have to manually parse and reformat the XML output to make it more useful for developers (see: https://github.com/aws/aws-cdk/blob/main/tools/%40aws-cdk/security-guardian/src/cfn-guard-runner.ts).

Describe the solution you'd like

Enhance the JUnit XML output format to include line number and column information in standard JUnit XML attributes. Specifically:

  1. Add line and column attributes to <testcase> elements to indicate where the violation occurred
  2. Include file, line, and column attributes in <failure> elements for precise error location
  3. Ensure the location information is extracted from the Guard evaluation context and properly formatted

Example desired output:

<testsuite name="template.yaml" file="template.yaml">
  <testcase name="S3BucketEncryption" classname="AWS::S3::Bucket" file="template.yaml" line="42" column="5">
    <failure message="Check was not compliant as property [ServerSideEncryptionConfiguration] is missing" 
             type="FAIL" 
             file="template.yaml" 
             line="42" 
             column="5">
      Detailed failure message with context...
    </failure>
  </testcase>
</testsuite>

This would allow CI/CD systems, IDEs, and other tools to automatically navigate to the exact location of compliance violations without requiring custom parsing logic.

Describe alternatives you've considered

  1. Post-processing the JUnit XML (current approach): Parse the XML output and extract location information from failure messages, then inject it into the proper XML attributes. This works but requires every user to implement their own parsing logic and is fragile if message formats change.

  2. Using structured output format: The --structured flag provides JSON output with more detailed information, but this requires completely different tooling and doesn't integrate with standard JUnit report consumers.

  3. Parsing console output: Extract location information from the human-readable console output, but this is even more fragile and doesn't work well with CI/CD pipelines.

Additional context

  • The Guard evaluation engine already has access to line and column information during rule evaluation
  • JUnit XML format supports custom attributes, so this enhancement would be backward compatible
  • This would benefit all Guard users, not just those using it with AWS CDK
  • Many static analysis tools (ESLint, cfn-lint, etc.) already provide location information in their JUnit outputs
  • Reference implementation showing the need for post-processing: https://github.com/aws/aws-cdk/blob/main/tools/%40aws-cdk/security-guardian/src/cfn-guard-runner.ts

This enhancement would make CloudFormation Guard's JUnit output more useful out-of-the-box and align it with industry standards for static analysis tool reporting.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions