feat: generate skill tool definitions at index time and serve them via file-meta

[jurgenwerk]

This is a development step where we want to have functioning tools coming from .md skill files that are pulled on demand by the model. Once the pulled skill is read in the ai bot, the bot will fetch the indexed tool definition and send it to the model so that it can call the tools defined in pulled skills.

SkillFrontmatterField.fromFrontmatter now generates each declared tool's LLM definition during file extraction and stamps it onto the skill's file-meta, so consumers get ready-to-use tool definitions straight from the index — no module loader needed on the consumer side.

skills/my-skill/SKILL.md
  boxel.tools: [{ codeRef, requiresApproval }]
        │
        ▼  file-extract visit (prerender page, full host runtime)
  SkillFrontmatterField.fromFrontmatter:
  resolve codeRef absolute → load tool class → getInputJsonSchema
        │
        ├─ ok ───────► file-meta resource
        │                frontmatter.tools[n] = { codeRef (absolute),
        │                                         functionName,
        │                                         requiresApproval,
        │                                         definition (LLM tool) }
        │
        └─ failure ──► diagnostics.toolSchemaErrors
                         row still indexes; remaining tools still enrich;
                         surfaced by /_indexing-errors and
                         boxel-cli realm indexing-errors

  search_doc ──► keeps tools as authored; definitions never land there
  deps ────────► tool module URLs, so editing a realm-hosted tool
                 module reindexes the referencing skill

• Only the indexing path enriches: the file-extract render route hands the extractor an owner-carrying toolContext, and without one fromFrontmatter skips generation — interactive extract paths are unchanged.

• The enriched value and the failure diagnostics travel out-of-band on symbol channels (FRONTMATTER_FILE_META_VALUE_SYMBOL, FRONTMATTER_DIAGNOSTICS_SYMBOL), keeping both off the flat search_doc.

• ToolField gains an optional definition JsonField so enriched resources rehydrate host-side.

• Stamping the absolute codeRef also fixes the known relative-module functionName mismatch, since rehydrated tools now recompute from it.

• Deploy side effects to be aware of: legacy Skill cards' commands entries now serialize with definition: null, so each enabled legacy skill card re-uploads its tool definitions once (the content-hash cache repopulates), and committed skill fixtures gain definition: null diffs on next save. Skill rows indexed before this deploys stay un-enriched until their file changes or the realm fully reindexes — plan a reindex when the pull-model consumer starts reading stamped definitions.

Verified locally against the dev stack: after touching a skill markdown file that declares boxel.tools (reindex is per-file incremental), its file-meta carries the stamped definitions —

curl -sk -H "Accept: application/vnd.card.file-meta+json" \
  https://localhost:4201/skills/skills/catalog-listing/SKILL.md \
  | jq '.data.attributes.frontmatter.tools[0]'

returns the authored tool entry (just a codeRef in the YAML) plus the index-generated additions — resolved codeRef, functionName, explicit requiresApproval, and the ready-to-use definition:

{
  "codeRef": {
    "name": "default",
    "module": "@cardstack/catalog/commands/listing-create"
  },
  "definition": {
    "type": "function",
    "function": {
      "name": "listing-create_5cac",
      "parameters": {
        "type": "object",
        "required": [
          "attributes",
          "description"
        ],
        "properties": {
          "attributes": {
            "type": "object",
            "required": [
              "codeRef",
              "targetRealm"
            ],
            "properties": {
              "codeRef": {
                "type": "object",
                "properties": {
                  "name": {
                    "type": "string"
                  },
                  "module": {
                    "type": "string"
                  }
                }
              },
              "openCardIds": {
                "type": "array",
                "items": {
                  "type": "string"
                }
              },
              "targetRealm": {
                "type": "string"
              }
            }
          },
          "description": {
            "type": "string"
          }
        }
      },
      "description": "Create a catalog listing for a card, field, skill, theme, or component"
    }
  },
  "functionName": "listing-create_5cac",
  "requiresApproval": true
}

🤖 Generated with Claude Code

[github-actions]

Preview deployments

• Host staging preview

• Host production preview

Host Test Results

    1 files      1 suites   2h 53m 20s ⏱️
3 472 tests 3 457 ✅ 15 💤 0 ❌
3 491 runs  3 476 ✅ 15 💤 0 ❌

Results for commit 0a6b9fd.

Realm Server Test Results

    1 files  ±0      1 suites  ±0   11m 26s ⏱️ -42s
1 817 tests ±0  1 817 ✅ ±0  0 💤 ±0  0 ❌ ±0 
1 896 runs  ±0  1 896 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 0a6b9fd. ± Comparison against earlier commit bdeffd7.

[Copilot]

Pull request overview

This PR moves skill tool schema generation into the indexing pipeline: during file extraction (indexing-only), the host resolves each skill frontmatter tool’s codeRef, loads the tool class, generates its input JSON schema, and stamps a ready-to-use LLM tool definition onto the skill’s file-meta resource. Failures are recorded as diagnostics.toolSchemaErrors so indexing still succeeds while surfacing actionable errors via /_indexing-errors and the boxel-cli command.

Changes:

• Add ToolSchemaError diagnostics plumbing end-to-end (extract response → indexer merge → /_indexing-errors endpoint → boxel-cli formatter).
• Implement indexing-only skill tool schema stamping in the host file extractor and enable it in the prerender file-extract route.
• Extend ToolField with an optional tool JsonField to support rehydrating enriched tool definitions from indexed file-meta.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
packages/runtime-common/realm.ts	Surface toolSchemaErrors in /_indexing-errors and emit tool-schema-error resources.
packages/runtime-common/index.ts	Define ToolSchemaError and add it to extract response + diagnostics types.
packages/runtime-common/index-runner/file-indexer.ts	Merge extract-time toolSchemaErrors onto persisted diagnostics without failing indexing.
packages/runtime-common/commands.ts	Clarify functionName generation consistency assumptions across producers.
packages/realm-server/tests/realm-endpoints/indexing-errors-test.ts	Add coverage ensuring tool-schema-error surfaces even when has_error = FALSE.
packages/host/tests/acceptance/prerender-file-extract-test.gts	Add acceptance coverage for stamping schemas on skill file-meta and error reporting for broken tools.
packages/host/app/utils/file-def-attributes-extractor.ts	Implement stampSkillToolSchemas() and return toolSchemaErrors from extract.
packages/host/app/routes/render/file-extract.ts	Enable tool schema generation on the indexing (file-extract) render route.
packages/boxel-cli/src/commands/realm/indexing-errors.ts	Add tool-schema-error handling and compact formatting for tool schema failures.
packages/base/tool-field.gts	Add optional tool JsonField to store stamped LLM tool definitions.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0a6b9fd1df

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Strip authored schemas from failed tool entries

When a skill frontmatter entry already contains generated-looking keys such as tool or functionName and its codeRef fails to load or generate a schema, this failure path returns the original entry, which is then persisted into the file-meta tools array. In that scenario, downstream pull-model consumers cannot distinguish an author-supplied/stale schema from an index-generated one and may expose a tool definition that was never validated against the tool class; sanitize failed entries by removing generated fields before storing them.

Useful? React with 👍 / 👎.

[lukemelia]

Would it be possible to do the tool schema work in fromFrontmatter within SkillFrontmatterField?

[lukemelia]

Since this is ToolField it is a little odd to have a field named tool. Maybe it should be schema or definitionJson or something like that?

[lukemelia]

[Claude Code 🤖] Renamed to definition in #5482 — it holds the whole LLM tool definition ({ type: "function", function: { … } }), not just the parameters schema, and the Json suffix felt redundant given the field is a JsonField. Happy to switch to one of your suggested spellings if you prefer.

[lukemelia]

Why is this needed?

[lukemelia]

[Claude Code 🤖] Schema generation loads each declared tool's module and instantiates the tool class, so it should only run on the indexing path — the interactive extract paths (room file attach, the store's direct file-meta fallback) shouldn't pay that module-loading cost, and their consumers already generate definitions on demand (uploadToolDefinitions). This render route is the indexing path, so it opts in. In #5482 the boolean is gone: the route instead passes the owner-carrying ToolContext that tool construction needs, and the presence of that context is what enables the work.

[lukemelia]

[Claude Code 🤖] Yes — it works nicely there. #5482 (based on this branch) moves the schema work into SkillFrontmatterField.fromFrontmatter: the hook is now async and receives { fileURL, toolContext? }, the skill subclass generates the definitions itself, and the generic extractor no longer inspects skill shapes — it just lifts the enriched frontmatter and any ToolSchemaErrors off the same out-of-band symbol channels the frontmatter parse error already uses (keeping the multi-KB definitions out of search_doc). That PR also renames the ToolField field to definition per your other comment.

[jurgenwerk]

@copilot resolve the merge conflicts in this pull request

[Copilot]

@copilot resolve the merge conflicts in this pull request

Done — merged origin/main and resolved the two conflicts:

• packages/runtime-common/index.ts (Diagnostics interface): included both toolSchemaErrors (this branch) and indexVisitClientMs (main) since they are independent additions.
• packages/runtime-common/index-runner/file-indexer.ts (dependency-error branch): used withBookkeeping(fileDiagnostics) to combine main's bookkeeping timing with this branch's frontmatter/tool-schema diagnostics.