Skip to content

fix: add cycle detection to TOML renderer (manifestTomlEx)#963

Open
He-Pin wants to merge 1 commit into
databricks:masterfrom
He-Pin:fix/toml-cyclic-detection
Open

fix: add cycle detection to TOML renderer (manifestTomlEx)#963
He-Pin wants to merge 1 commit into
databricks:masterfrom
He-Pin:fix/toml-cyclic-detection

Conversation

@He-Pin

@He-Pin He-Pin commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Motivation

std.manifestTomlEx has its own recursive traversal in renderTableInternal that bypassed the Materializer's IdentityHashMap-based cycle detection. Cyclic objects like {a: $} caused OutOfMemoryError instead of the clean "Stackoverflow while materializing" error that JSON and YAML renderers produce.

Modification

  • Add depth: Int and visited: IdentityHashMap[Val.Obj, Boolean] parameters to renderTableInternal, matching the pattern in Materializer.MaterializeContext
  • Check depth against maxMaterializeDepth and detect revisited objects at method entry; remove from visited set in finally block (backtracking)
  • Wrap evalRhs in try-catch for StackOverflowError/OutOfMemoryError
  • Add 3 cyclic reference tests: manifestJsonEx, manifestTomlEx, manifestYamlDoc — all using {a: $} pattern

Result

Cyclic objects now produce "Stackoverflow while materializing" error instead of OOM, consistent with JSON and YAML renderers. All existing TOML tests (array, null, standard) continue to pass.

Cross-implementation comparison

Test case: std.manifestTomlEx({a: $}, " ") (cyclic self-reference)

Implementation Result Detection mechanism
go-jsonnet v0.22.0 RUNTIME ERROR: max manifest depth exceeded, possible infinite recursion Depth limit in TOML renderer
jrsonnet v0.5.0-pre99 stack overflow, try to reduce recursion, or set --max-stack to bigger value Stack overflow detection
C++ jsonnet HEAD-2ef32ab HANGS (process killed after 5s timeout) No cycle detection — same bug as sjsonnet pre-fix
sjsonnet (before) java.lang.OutOfMemoryError: Java heap space No cycle detection in TOML renderer
sjsonnet (after) sjsonnet.Error: Stackoverflow while materializing, possibly due to recursive value IdentityHashMap + depth limit

Key observations

  • go-jsonnet is the only implementation with proper TOML-specific cycle detection (depth limit)
  • C++ jsonnet has the same bug as sjsonnet had: TOML renderer lacks cycle detection, causing hang/OOM
  • jrsonnet catches stack overflow generically, but still detects the cycle
  • sjsonnet (after) now matches go-jsonnet's robustness with IdentityHashMap-based detection

Test plan

  • std.manifestTomlEx({a: $}, " ") produces clean error instead of OOM
  • std.manifestJsonEx({a: $}, ...) continues to work (already had detection)
  • std.manifestYamlDoc({a: $}) continues to work (already had detection)
  • All existing TOML tests pass (array, null, standard cases)
  • Error message matches JSON/YAML renderer format

Motivation:
std.manifestTomlEx has its own recursive traversal in renderTableInternal
that bypassed the Materializer's IdentityHashMap-based cycle detection.
Cyclic objects like `{a: $}` caused OOM instead of a clean error.

Modification:
- Add depth counter and IdentityHashMap parameters to renderTableInternal
- Check depth against maxMaterializeDepth and detect revisited objects
- Remove from visited set in finally block (backtracking)
- Wrap evalRhs in try-catch for StackOverflowError/OutOfMemoryError
- Add cyclic reference tests for manifestJsonEx, manifestTomlEx, and
  manifestYamlDoc using `{a: $}` pattern

Result:
Cyclic objects now produce "Stackoverflow while materializing" error
instead of OOM, consistent with JSON and YAML renderers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant