Skip to content

[idd-verify] DA timeout sentinel string variant — coordinator auto-detect missed (observed in downstream verify) #88

@kiki830621

Description

@kiki830621

Problem

idd-verify skill v2.59.0+ Step 2 specifies DA timeout sentinel:

[STAGE 2.5 RECOVERY: DEVILS_ADVOCATE_TIMEOUT_<n>/4]

Step 2.5a coordinator detection uses regex matching ^\[STAGE 2.5 RECOVERY: DEVILS_ADVOCATE_TIMEOUT_.

Observed variant

During /idd-verify --pr 82 (in downstream repo PsychQuantHsu/psychophysical_representations), DA Agent wrote a variant sentinel marker:

[STAGE 2.5 PARTIAL: DEVILS_ADVOCATE_DEGRADED_1/4]

Note 3 differences:

  • RECOVERYPARTIAL
  • TIMEOUTDEGRADED
  • Otherwise structurally identical (same prefix bracket, same _<n>/4 suffix)

Coordinator's Step 2.5a regex ^\[STAGE 2.5 RECOVERY: DEVILS_ADVOCATE_TIMEOUT_ does NOT match this variant → auto-retry never fires → manual fallback (coordinator manually triggered round 2) was needed.

Why LLM produced variant

Speculation: Agent prompt's STAGE 2.5 RECOVERY: DEVILS_ADVOCATE_TIMEOUT_ literal is embedded inside a long instruction. LLM paraphrased semantically (RECOVERY → PARTIAL is synonymous; TIMEOUT → DEGRADED similarly) when constructing the output. Without "MUST be verbatim" emphasis, the literal drift is plausible.

Suggested fix

Two options (one preferred, both acceptable):

Option A (forgiving): extend coordinator regex

In Step 2.5a of skills/idd-verify markdown, change detection from:

elif head -1 "$f" | grep -q '^\[STAGE 2.5 RECOVERY: DEVILS_ADVOCATE_TIMEOUT_'; then

to:

elif head -1 "$f" | grep -qE '^\[STAGE 2\.5 (RECOVERY|PARTIAL): DEVILS_ADVOCATE_(TIMEOUT|DEGRADED)_'; then

Matches both canonical and observed variant + future-proofs against further LLM rephrasing.

Option B (stricter): tighten Step 2 DA prompt template

Add explicit "verbatim — do not rephrase" instruction in Step 2 DA prompt for the sentinel marker block. Example wording:

CRITICAL: Write the sentinel marker on the first line verbatim as: [STAGE 2.5 RECOVERY: DEVILS_ADVOCATE_TIMEOUT_<N>/4] — do not substitute "RECOVERY" → "PARTIAL" or "TIMEOUT" → "DEGRADED". Coordinator's regex-detection depends on the exact literal.

Recommendation

Option A — forgiving regex is more robust against future LLM rephrasing (whichever model is running). Option B keeps drifting as a single-point-of-failure in prompt discipline. Either way the fix is one-line change to skills/idd-verify/skill.md.

Acceptance

  • Update Step 2.5a regex OR Step 2 DA prompt template (pick one)
  • Test: simulate DA writing variant sentinel — coordinator should detect + retry/fallback fires
  • Document choice in CHANGELOG / release notes

Cross-link

Downstream tracker: PsychQuantHsu/psychophysical_representations#88

Origin observation: /idd-verify --pr 82 round 1 process gap (2026-05-12).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions