diff --git a/docs/README.skills.md b/docs/README.skills.md index b66b3bd76..5b8983bf7 100644 --- a/docs/README.skills.md +++ b/docs/README.skills.md @@ -45,15 +45,15 @@ See [CONTRIBUTING.md](../CONTRIBUTING.md#adding-skills) for guidelines on how to | [arch-linux-triage](../skills/arch-linux-triage/SKILL.md)
`gh skills install github/awesome-copilot arch-linux-triage` | Triage and resolve Arch Linux issues with pacman, systemd, and rolling-release best practices. | None | | [architecture-blueprint-generator](../skills/architecture-blueprint-generator/SKILL.md)
`gh skills install github/awesome-copilot architecture-blueprint-generator` | Comprehensive project architecture blueprint generator that analyzes codebases to create detailed architectural documentation. Automatically detects technology stacks and architectural patterns, generates visual diagrams, documents implementation patterns, and provides extensible blueprints for maintaining architectural consistency and guiding new development. | None | | [arduino-azure-iot-edge-integration](../skills/arduino-azure-iot-edge-integration/SKILL.md)
`gh skills install github/awesome-copilot arduino-azure-iot-edge-integration` | Design and implement Arduino integration with Azure IoT Hub and IoT Edge, including secure provisioning, resilient telemetry, command handling, and production guardrails. | `references/arduino-iot-checklist.md`
`references/arduino-official-best-practices.md` | -| [arize-ai-provider-integration](../skills/arize-ai-provider-integration/SKILL.md)
`gh skills install github/awesome-copilot arize-ai-provider-integration` | INVOKE THIS SKILL when creating, reading, updating, or deleting Arize AI integrations. Covers listing integrations, creating integrations for any supported LLM provider (OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Vertex AI, Gemini, NVIDIA NIM, custom), updating credentials or metadata, and deleting integrations using the ax CLI. | `references/ax-profiles.md`
`references/ax-setup.md` | -| [arize-annotation](../skills/arize-annotation/SKILL.md)
`gh skills install github/awesome-copilot arize-annotation` | INVOKE THIS SKILL when creating, managing, or using annotation configs or annotation queues on Arize (categorical, continuous, freeform), or applying human annotations to project spans via the Python SDK. Configs are the label schema for human feedback; queues are review workflows that route records to annotators. Triggers: annotation config, annotation queue, label schema, human feedback schema, bulk annotate spans, update_annotations, labeling queue, annotate record. | `references/ax-profiles.md`
`references/ax-setup.md` | -| [arize-dataset](../skills/arize-dataset/SKILL.md)
`gh skills install github/awesome-copilot arize-dataset` | INVOKE THIS SKILL when creating, managing, or querying Arize datasets and examples. Also use when the user needs test data or evaluation examples for their model. Covers dataset CRUD, appending examples, exporting data, and file-based dataset creation using the ax CLI. | `references/ax-profiles.md`
`references/ax-setup.md` | -| [arize-evaluator](../skills/arize-evaluator/SKILL.md)
`gh skills install github/awesome-copilot arize-evaluator` | INVOKE THIS SKILL for LLM-as-judge evaluation workflows on Arize: creating/updating evaluators, running evaluations on spans or experiments, tasks, trigger-run, column mapping, and continuous monitoring. Use when the user says: create an evaluator, LLM judge, hallucination/faithfulness/correctness/relevance, run eval, score my spans or experiment, ax tasks, trigger-run, trigger eval, column mapping, continuous monitoring, query filter for evals, evaluator version, or improve an evaluator prompt. | `references/ax-profiles.md`
`references/ax-setup.md` | -| [arize-experiment](../skills/arize-experiment/SKILL.md)
`gh skills install github/awesome-copilot arize-experiment` | INVOKE THIS SKILL when creating, running, or analyzing Arize experiments. Also use when the user wants to evaluate or measure model performance, compare models (including GPT-4, Claude, or others), or assess how well their AI is doing. Covers experiment CRUD, exporting runs, comparing results, and evaluation workflows using the ax CLI. | `references/ax-profiles.md`
`references/ax-setup.md` | -| [arize-instrumentation](../skills/arize-instrumentation/SKILL.md)
`gh skills install github/awesome-copilot arize-instrumentation` | INVOKE THIS SKILL when adding Arize AX tracing or observability to an app for the first time, or when the user wants to instrument their LLM app or get started with LLM observability. Follow the Agent-Assisted Tracing two-phase flow: analyze the codebase (read-only), then implement after user confirmation. When the app uses LLM tool/function calling, add manual CHAIN + TOOL spans. Leverages https://arize.com/docs/ax/alyx/tracing-assistant and https://arize.com/docs/PROMPT.md. | `references/ax-profiles.md` | -| [arize-link](../skills/arize-link/SKILL.md)
`gh skills install github/awesome-copilot arize-link` | Generate deep links to the Arize UI. Use when the user wants a clickable URL to open or share a specific trace, span, session, dataset, labeling queue, evaluator, or annotation config, or when sharing Arize resources with team members. | `references/EXAMPLES.md` | -| [arize-prompt-optimization](../skills/arize-prompt-optimization/SKILL.md)
`gh skills install github/awesome-copilot arize-prompt-optimization` | INVOKE THIS SKILL when optimizing, improving, or debugging LLM prompts using production trace data, evaluations, and annotations. Also use when the user wants to make their AI respond better or improve AI output quality. Covers extracting prompts from spans, gathering performance signal, and running a data-driven optimization loop using the ax CLI. | `references/ax-profiles.md`
`references/ax-setup.md` | -| [arize-trace](../skills/arize-trace/SKILL.md)
`gh skills install github/awesome-copilot arize-trace` | INVOKE THIS SKILL when downloading, exporting, or inspecting Arize traces and spans, or when a user wants to look at what their LLM app is doing using existing trace data, or when an already-instrumented app has a bug or error to investigate. Use for debugging unknown runtime issues, failures, and behavior regressions. Covers exporting traces by ID, spans by ID, sessions by ID, and root-cause investigation with the ax CLI. | `references/ax-profiles.md`
`references/ax-setup.md` | +| [arize-ai-provider-integration](../skills/arize-ai-provider-integration/SKILL.md)
`gh skills install github/awesome-copilot arize-ai-provider-integration` | Creates, reads, updates, and deletes Arize AI integrations that store LLM provider credentials used by evaluators and other Arize features. Supports any LLM provider (e.g. OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Vertex AI, Gemini, NVIDIA NIM). Use when the user mentions AI integration, LLM provider credentials, create integration, list integrations, update credentials, delete integration, or connecting an LLM provider to Arize. | `references/ax-profiles.md`
`references/ax-setup.md` | +| [arize-annotation](../skills/arize-annotation/SKILL.md)
`gh skills install github/awesome-copilot arize-annotation` | Creates and manages annotation configs (categorical, continuous, freeform label schemas) and annotation queues (human review workflows) on Arize. Applies human annotations to project spans via the Python SDK. Use when the user mentions annotation config, annotation queue, label schema, human feedback, bulk annotate spans, update_annotations, labeling queue, annotate record, or human review. | `references/ax-profiles.md`
`references/ax-setup.md` | +| [arize-dataset](../skills/arize-dataset/SKILL.md)
`gh skills install github/awesome-copilot arize-dataset` | Creates, manages, and queries Arize datasets and examples. Covers dataset CRUD, appending examples, exporting data, and file-based dataset creation using the ax CLI. Use when the user needs test data, evaluation examples, or mentions create dataset, list datasets, export dataset, append examples, dataset version, golden dataset, or test set. | `references/ax-profiles.md`
`references/ax-setup.md` | +| [arize-evaluator](../skills/arize-evaluator/SKILL.md)
`gh skills install github/awesome-copilot arize-evaluator` | Handles LLM-as-judge evaluation workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, hallucination, faithfulness, correctness, relevance, run eval, score spans, score experiment, trigger-run, column mapping, continuous monitoring, or improve evaluator prompt. | `references/ax-profiles.md`
`references/ax-setup.md` | +| [arize-experiment](../skills/arize-experiment/SKILL.md)
`gh skills install github/awesome-copilot arize-experiment` | Creates, runs, and analyzes Arize experiments for evaluating and comparing model performance. Covers experiment CRUD, exporting runs, comparing results, and evaluation workflows using the ax CLI. Use when the user mentions create experiment, run experiment, compare models, model performance, evaluate AI, experiment results, benchmark, A/B test models, or measure accuracy. | `references/ax-profiles.md`
`references/ax-setup.md` | +| [arize-instrumentation](../skills/arize-instrumentation/SKILL.md)
`gh skills install github/awesome-copilot arize-instrumentation` | Adds Arize AX tracing to an LLM application for the first time. Follows a two-phase agent-assisted flow to analyze the codebase then implement instrumentation after user confirmation. Use when the user wants to instrument their app, add tracing from scratch, set up LLM observability, integrate OpenTelemetry or openinference, or get started with Arize tracing. | `references/ax-profiles.md` | +| [arize-link](../skills/arize-link/SKILL.md)
`gh skills install github/awesome-copilot arize-link` | Generates deep links to the Arize UI for traces, spans, sessions, datasets, labeling queues, evaluators, and annotation configs. Produces clickable URLs for sharing Arize resources with team members. Use when the user wants to link to or open a trace, span, session, dataset, evaluator, or annotation config in the Arize UI. | `references/EXAMPLES.md` | +| [arize-prompt-optimization](../skills/arize-prompt-optimization/SKILL.md)
`gh skills install github/awesome-copilot arize-prompt-optimization` | Optimizes, improves, and debugs LLM prompts using production trace data, evaluations, and annotations. Extracts prompts from spans, gathers performance signal, and runs a data-driven optimization loop using the ax CLI. Use when the user mentions optimize prompt, improve prompt, make AI respond better, improve output quality, prompt engineering, prompt tuning, or system prompt improvement. | `references/ax-profiles.md`
`references/ax-setup.md` | +| [arize-trace](../skills/arize-trace/SKILL.md)
`gh skills install github/awesome-copilot arize-trace` | Downloads, exports, and inspects existing Arize traces and spans to understand what an LLM app is doing or debug runtime issues. Covers exporting traces by ID, spans by ID, sessions by ID, and root-cause investigation using the ax CLI. Use when the user wants to look at existing trace data, see what their LLM app is doing, export traces, download spans, investigate errors, or analyze behavior regressions. | `references/ax-profiles.md`
`references/ax-setup.md` | | [aspire](../skills/aspire/SKILL.md)
`gh skills install github/awesome-copilot aspire` | Aspire skill covering the Aspire CLI, AppHost orchestration, service discovery, integrations, MCP server, VS Code extension, Dev Containers, GitHub Codespaces, templates, dashboard, and deployment. Use when the user asks to create, run, debug, configure, deploy, or troubleshoot an Aspire distributed application. | `references/architecture.md`
`references/cli-reference.md`
`references/dashboard.md`
`references/deployment.md`
`references/integrations-catalog.md`
`references/mcp-server.md`
`references/polyglot-apis.md`
`references/testing.md`
`references/troubleshooting.md` | | [aspnet-minimal-api-openapi](../skills/aspnet-minimal-api-openapi/SKILL.md)
`gh skills install github/awesome-copilot aspnet-minimal-api-openapi` | Create ASP.NET Minimal API endpoints with proper OpenAPI documentation | None | | [audit-integrity](../skills/audit-integrity/SKILL.md)
`gh skills install github/awesome-copilot audit-integrity` | Shared audit integrity framework for all AppSec agents — enforces output quality, intellectual honesty, and continuous improvement through anti-rationalization guards, self-critique loops, retry protocols, non-negotiable behaviors, self-reflection quality gates (1-10 scoring, ≥8 threshold), and a self-learning system with lesson/memory governance for security analysis agents. | `references/anti-rationalization-guard.md`
`references/clarification-protocol.md`
`references/non-negotiable-behaviors.md`
`references/retry-protocol.md`
`references/self-critique-loop.md`
`references/self-learning-system.md`
`references/self-reflection-quality-gate.md` | diff --git a/skills/arize-ai-provider-integration/SKILL.md b/skills/arize-ai-provider-integration/SKILL.md index dbf2fc169..806be8e59 100644 --- a/skills/arize-ai-provider-integration/SKILL.md +++ b/skills/arize-ai-provider-integration/SKILL.md @@ -1,6 +1,10 @@ --- name: arize-ai-provider-integration -description: "INVOKE THIS SKILL when creating, reading, updating, or deleting Arize AI integrations. Covers listing integrations, creating integrations for any supported LLM provider (OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Vertex AI, Gemini, NVIDIA NIM, custom), updating credentials or metadata, and deleting integrations using the ax CLI." +description: Creates, reads, updates, and deletes Arize AI integrations that store LLM provider credentials used by evaluators and other Arize features. Supports any LLM provider (e.g. OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Vertex AI, Gemini, NVIDIA NIM). Use when the user mentions AI integration, LLM provider credentials, create integration, list integrations, update credentials, delete integration, or connecting an LLM provider to Arize. +metadata: + author: arize + version: "1.0" +compatibility: Requires the ax CLI and a configured Arize profile. --- # Arize AI Integration Skill diff --git a/skills/arize-annotation/SKILL.md b/skills/arize-annotation/SKILL.md index 0e66ee462..3f69f32bc 100644 --- a/skills/arize-annotation/SKILL.md +++ b/skills/arize-annotation/SKILL.md @@ -1,6 +1,10 @@ --- name: arize-annotation -description: "INVOKE THIS SKILL when creating, managing, or using annotation configs or annotation queues on Arize (categorical, continuous, freeform), or applying human annotations to project spans via the Python SDK. Configs are the label schema for human feedback; queues are review workflows that route records to annotators. Triggers: annotation config, annotation queue, label schema, human feedback schema, bulk annotate spans, update_annotations, labeling queue, annotate record." +description: Creates and manages annotation configs (categorical, continuous, freeform label schemas) and annotation queues (human review workflows) on Arize. Applies human annotations to project spans via the Python SDK. Use when the user mentions annotation config, annotation queue, label schema, human feedback, bulk annotate spans, update_annotations, labeling queue, annotate record, or human review. +metadata: + author: arize + version: "1.0" +compatibility: Requires the ax CLI and a configured Arize profile. --- # Arize Annotation Skill diff --git a/skills/arize-dataset/SKILL.md b/skills/arize-dataset/SKILL.md index 76258eecd..4046570a8 100644 --- a/skills/arize-dataset/SKILL.md +++ b/skills/arize-dataset/SKILL.md @@ -1,6 +1,10 @@ --- name: arize-dataset -description: "INVOKE THIS SKILL when creating, managing, or querying Arize datasets and examples. Also use when the user needs test data or evaluation examples for their model. Covers dataset CRUD, appending examples, exporting data, and file-based dataset creation using the ax CLI." +description: Creates, manages, and queries Arize datasets and examples. Covers dataset CRUD, appending examples, exporting data, and file-based dataset creation using the ax CLI. Use when the user needs test data, evaluation examples, or mentions create dataset, list datasets, export dataset, append examples, dataset version, golden dataset, or test set. +metadata: + author: arize + version: "1.0" +compatibility: Requires the ax CLI and a configured Arize profile. --- # Arize Dataset Skill diff --git a/skills/arize-evaluator/SKILL.md b/skills/arize-evaluator/SKILL.md index 660e9bd62..1336030d8 100644 --- a/skills/arize-evaluator/SKILL.md +++ b/skills/arize-evaluator/SKILL.md @@ -1,6 +1,10 @@ --- name: arize-evaluator -description: "INVOKE THIS SKILL for LLM-as-judge evaluation workflows on Arize: creating/updating evaluators, running evaluations on spans or experiments, tasks, trigger-run, column mapping, and continuous monitoring. Use when the user says: create an evaluator, LLM judge, hallucination/faithfulness/correctness/relevance, run eval, score my spans or experiment, ax tasks, trigger-run, trigger eval, column mapping, continuous monitoring, query filter for evals, evaluator version, or improve an evaluator prompt." +description: Handles LLM-as-judge evaluation workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, hallucination, faithfulness, correctness, relevance, run eval, score spans, score experiment, trigger-run, column mapping, continuous monitoring, or improve evaluator prompt. +metadata: + author: arize + version: "1.0" +compatibility: Requires the ax CLI and a configured Arize profile with an AI integration. --- # Arize Evaluator Skill diff --git a/skills/arize-experiment/SKILL.md b/skills/arize-experiment/SKILL.md index 0d9c3320a..45759467a 100644 --- a/skills/arize-experiment/SKILL.md +++ b/skills/arize-experiment/SKILL.md @@ -1,6 +1,10 @@ --- name: arize-experiment -description: "INVOKE THIS SKILL when creating, running, or analyzing Arize experiments. Also use when the user wants to evaluate or measure model performance, compare models (including GPT-4, Claude, or others), or assess how well their AI is doing. Covers experiment CRUD, exporting runs, comparing results, and evaluation workflows using the ax CLI." +description: Creates, runs, and analyzes Arize experiments for evaluating and comparing model performance. Covers experiment CRUD, exporting runs, comparing results, and evaluation workflows using the ax CLI. Use when the user mentions create experiment, run experiment, compare models, model performance, evaluate AI, experiment results, benchmark, A/B test models, or measure accuracy. +metadata: + author: arize + version: "1.0" +compatibility: Requires the ax CLI and a configured Arize profile. --- # Arize Experiment Skill diff --git a/skills/arize-instrumentation/SKILL.md b/skills/arize-instrumentation/SKILL.md index 6da715d99..f1a16a54b 100644 --- a/skills/arize-instrumentation/SKILL.md +++ b/skills/arize-instrumentation/SKILL.md @@ -1,6 +1,10 @@ --- name: arize-instrumentation -description: "INVOKE THIS SKILL when adding Arize AX tracing or observability to an app for the first time, or when the user wants to instrument their LLM app or get started with LLM observability. Follow the Agent-Assisted Tracing two-phase flow: analyze the codebase (read-only), then implement after user confirmation. When the app uses LLM tool/function calling, add manual CHAIN + TOOL spans. Leverages https://arize.com/docs/ax/alyx/tracing-assistant and https://arize.com/docs/PROMPT.md." +description: Adds Arize AX tracing to an LLM application for the first time. Follows a two-phase agent-assisted flow to analyze the codebase then implement instrumentation after user confirmation. Use when the user wants to instrument their app, add tracing from scratch, set up LLM observability, integrate OpenTelemetry or openinference, or get started with Arize tracing. +metadata: + author: arize + version: "1.0" +compatibility: Python and TypeScript/JavaScript apps use openinference-instrumentation packages for auto-instrumentation. Java and Go apps use the OpenTelemetry SDK with manual OpenInference spans. See https://arize.com/docs/PROMPT.md for setup details. --- # Arize Instrumentation Skill @@ -46,6 +50,7 @@ Before changing code: - Python: `pyproject.toml`, `requirements.txt`, `setup.py`, `Pipfile` - TypeScript/JavaScript: `package.json` - Java: `pom.xml`, `build.gradle`, `build.gradle.kts` + - Go: `go.mod` 2. **Scan import statements** in source files to confirm what is actually used. @@ -57,8 +62,8 @@ Before changing code: | Item | Examples | |------|----------| -| Language | Python, TypeScript/JavaScript, Java | -| Package manager | pip/poetry/uv, npm/pnpm/yarn, maven/gradle | +| Language | Python, TypeScript/JavaScript, Java, Go | +| Package manager | pip/poetry/uv, npm/pnpm/yarn, maven/gradle, go modules | | LLM providers | OpenAI, Anthropic, LiteLLM, Bedrock, etc. | | Frameworks | LangChain, LangGraph, LlamaIndex, Vercel AI SDK, Mastra, etc. | | Existing tracing | Any OTel or vendor setup | @@ -86,8 +91,9 @@ The **canonical list** of supported integrations and doc URLs is in the [Agent S - **Python frameworks:** [LangChain](https://arize.com/docs/ax/integrations/python-agent-frameworks/langchain), [LangGraph](https://arize.com/docs/ax/integrations/python-agent-frameworks/langgraph), [LlamaIndex](https://arize.com/docs/ax/integrations/python-agent-frameworks/llamaindex), [CrewAI](https://arize.com/docs/ax/integrations/python-agent-frameworks/crewai), [DSPy](https://arize.com/docs/ax/integrations/python-agent-frameworks/dspy), [AutoGen](https://arize.com/docs/ax/integrations/python-agent-frameworks/autogen), [Semantic Kernel](https://arize.com/docs/ax/integrations/python-agent-frameworks/semantic-kernel), [Pydantic AI](https://arize.com/docs/ax/integrations/python-agent-frameworks/pydantic), [Haystack](https://arize.com/docs/ax/integrations/python-agent-frameworks/haystack), [Guardrails AI](https://arize.com/docs/ax/integrations/python-agent-frameworks/guardrails-ai), [Hugging Face Smolagents](https://arize.com/docs/ax/integrations/python-agent-frameworks/hugging-face-smolagents), [Instructor](https://arize.com/docs/ax/integrations/python-agent-frameworks/instructor), [Agno](https://arize.com/docs/ax/integrations/python-agent-frameworks/agno), [Google ADK](https://arize.com/docs/ax/integrations/python-agent-frameworks/google-adk), [MCP](https://arize.com/docs/ax/integrations/python-agent-frameworks/model-context-protocol), [Portkey](https://arize.com/docs/ax/integrations/python-agent-frameworks/portkey), [Together AI](https://arize.com/docs/ax/integrations/python-agent-frameworks/together-ai), [BeeAI](https://arize.com/docs/ax/integrations/python-agent-frameworks/beeai), [AWS Bedrock Agents](https://arize.com/docs/ax/integrations/python-agent-frameworks/aws). - **TypeScript/JavaScript:** [LangChain JS](https://arize.com/docs/ax/integrations/ts-js-agent-frameworks/langchain), [Mastra](https://arize.com/docs/ax/integrations/ts-js-agent-frameworks/mastra), [Vercel AI SDK](https://arize.com/docs/ax/integrations/ts-js-agent-frameworks/vercel), [BeeAI JS](https://arize.com/docs/ax/integrations/ts-js-agent-frameworks/beeai). - **Java:** [LangChain4j](https://arize.com/docs/ax/integrations/java/langchain4j), [Spring AI](https://arize.com/docs/ax/integrations/java/spring-ai), [Arconia](https://arize.com/docs/ax/integrations/java/arconia). +- **Go:** No first-party auto-instrumentation packages today — use the OpenTelemetry Go SDK with manual [OpenInference](https://github.com/Arize-ai/openinference) attributes per [Manual instrumentation](https://arize.com/docs/ax/instrument/manual-instrumentation). - **Platforms (UI-based):** [LangFlow](https://arize.com/docs/ax/integrations/platforms/langflow), [Flowise](https://arize.com/docs/ax/integrations/platforms/flowise), [Dify](https://arize.com/docs/ax/integrations/platforms/dify), [Prompt flow](https://arize.com/docs/ax/integrations/platforms/prompt-flow). -- **Fallback:** [Manual instrumentation](https://arize.com/docs/ax/observe/tracing/setup/manual-instrumentation), [All integrations](https://arize.com/docs/ax/integrations). +- **Fallback:** [Manual instrumentation](https://arize.com/docs/ax/instrument/manual-instrumentation), [All integrations](https://arize.com/docs/ax/integrations). **Fetch the matched doc pages** from the [full routing table in PROMPT.md](https://arize.com/docs/PROMPT.md) for exact installation and code snippets. Use [llms.txt](https://arize.com/docs/llms.txt) as a fallback for doc discovery if needed. @@ -104,13 +110,14 @@ Proceed **only after the user confirms** the Phase 1 analysis. - Python: `pip install arize-otel` plus `openinference-instrumentation-{name}` (hyphens in package name; underscores in import, e.g. `openinference.instrumentation.llama_index`). - TypeScript/JavaScript: `@opentelemetry/sdk-trace-node` plus the relevant `@arizeai/openinference-*` package. - Java: OpenTelemetry SDK plus `openinference-instrumentation-*` in pom.xml or build.gradle. + - Go: `go get go.opentelemetry.io/otel go.opentelemetry.io/otel/sdk go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp` — no auto-instrumentors yet, so the agent sets OpenInference attributes manually on spans. **Wire the exporter** with `otlptracehttp.WithEndpoint("otlp.arize.com")` (US) or `otlptracehttp.WithEndpoint("otlp.eu-west-1a.arize.com")` (EU) — pass the bare hostname, no `https://` scheme — and `otlptracehttp.WithHeaders(map[string]string{"space_id": ..., "api_key": ...})`. Recent OTel Go modules require Go ≥ 1.23 — `go mod tidy` may bump the toolchain. 3. **Credentials** — User needs an **Arize API Key** and **Space ID**. Check existing `ax` profiles for `ARIZE_API_KEY` and `ARIZE_SPACE` — never read `.env` files: - Run `ax profiles show` to check for an existing profile. - If no profile exists, guide the user to run `ax profiles create` which provides an **interactive wizard** that walks through API key and space setup. See [CLI profiles docs](https://arize.com/docs/api-clients/cli/profiles) for details. - If the user needs to find their API key manually, direct them to **https://app.arize.com** and to navigate to the settings page (do not use organization-specific URLs with placeholder IDs — they won't resolve for new users). - - If credentials are not set, instruct the user to set them as environment variables — never embed raw values in generated code. All generated instrumentation code must reference `os.environ["ARIZE_API_KEY"]` (Python) or `process.env.ARIZE_API_KEY` (TypeScript/JavaScript). + - If credentials are not set, instruct the user to set them as environment variables — never embed raw values in generated code. All generated instrumentation code must reference `os.environ["ARIZE_API_KEY"]` (Python), `process.env.ARIZE_API_KEY` (TypeScript/JavaScript), or `os.Getenv("ARIZE_API_KEY")` (Go). - See references/ax-profiles.md for full profile setup and troubleshooting. -4. **Centralized instrumentation** — Create a single module (e.g. `instrumentation.py`, `instrumentation.ts`) and initialize tracing **before** any LLM client is created. +4. **Centralized instrumentation** — Create a single module (e.g. `instrumentation.py`, `instrumentation.ts`, `instrumentation.go`) and initialize tracing **before** any LLM client is created. 5. **Existing OTel** — If there is already a TracerProvider, add Arize as an **additional** exporter (e.g. BatchSpanProcessor with Arize OTLP). Do not replace existing setup unless the user asks. ### Implementation rules @@ -119,8 +126,11 @@ Proceed **only after the user confirms** the Phase 1 analysis. - Prefer the repo's native integration surface before adding generic OpenTelemetry plumbing. If the framework ships an exporter or observability package, use that first unless there is a documented gap. - **Fail gracefully** if env vars are missing (warn, do not crash). - **Import order:** register tracer → attach instrumentors → then create LLM clients. -- **Project name attribute (required):** Arize rejects spans with HTTP 500 if the project name is missing — `service.name` alone is not accepted. Set it as a **resource attribute** on the TracerProvider (recommended — one place, applies to all spans): Python: `register(project_name="my-app")` handles it automatically (sets `"openinference.project.name"` on the resource); TypeScript: Arize accepts both `"model_id"` (shown in the official TS quickstart) and `"openinference.project.name"` via `SEMRESATTRS_PROJECT_NAME` from `@arizeai/openinference-semantic-conventions` (shown in the manual instrumentation docs) — both work. For routing spans to different projects in Python, use `set_routing_context(space_id=..., project_name=...)` from `arize.otel`. -- **CLI/script apps — flush before exit:** `provider.shutdown()` (TS) / `provider.force_flush()` then `provider.shutdown()` (Python) must be called before the process exits, otherwise async OTLP exports are dropped and no traces appear. +- **Project name attribute (required):** Arize rejects spans with HTTP 500 if the project name is missing — `service.name` alone is not accepted. Set it as a **resource attribute** on the TracerProvider (recommended — one place, applies to all spans): + - **Python:** `register(project_name="my-app")` handles it automatically (sets `"openinference.project.name"` on the resource). For routing spans to different projects, use `set_routing_context(space_id=..., project_name=...)` from `arize.otel`. + - **TypeScript:** Arize accepts both `"model_id"` (shown in the official TS quickstart) and `"openinference.project.name"` via `SEMRESATTRS_PROJECT_NAME` from `@arizeai/openinference-semantic-conventions` (shown in the manual instrumentation docs) — both work. + - **Go:** Pass `attribute.String("openinference.project.name", "my-app")` to `resource.New(...)` and apply via `sdktrace.WithResource(res)`. The Go SDK has no helper for this, so it must be set manually on every TracerProvider. +- **CLI/script apps — flush before exit:** `provider.shutdown()` (TS) / `provider.force_flush()` then `provider.shutdown()` (Python) / `tp.Shutdown(ctx)` (Go) must be called before the process exits, otherwise async OTLP exports are dropped and no traces appear. - **When the app has tool/function execution:** add manual CHAIN + TOOL spans (see **Enriching traces** below) so the trace tree shows each tool call and its result — otherwise traces will look sparse (only LLM API spans, no tool input/output). ## Enriching traces: manual spans for tool use and agent loops @@ -151,10 +161,26 @@ To avoid sparse traces where tool inputs/outputs are missing: | Attribute | Use | |-----------|-----| -| `openinference.span.kind` | `"CHAIN"` or `"TOOL"` | +| `openinference.span.kind` | Pick the right value: `"LLM"` for raw provider API calls (OpenAI, Anthropic, etc.); `"CHAIN"` for orchestration / agent-loop boundaries; `"TOOL"` for tool/function execution; `"RETRIEVER"` for vector-store / search lookups; `"EMBEDDING"` for embedding API calls; `"AGENT"` for an autonomous sub-agent run nested inside a larger chain; `"RERANKER"` for rerank API calls; `"GUARDRAIL"` for guardrail/policy checks; `"EVALUATOR"` for online eval calls. | | `input.value` | string (e.g. user message or JSON of tool args) | | `output.value` | string (e.g. final reply or JSON of tool result) | +**LLM-span attributes (set these in addition to the three above when the span is an actual LLM call):** + +| Attribute | Use | +|-----------|-----| +| `llm.model_name` | model identifier (e.g. `"gpt-4o-mini"`) | +| `llm.provider` / `llm.system` | provider name (e.g. `"openai"`, `"anthropic"`) | +| `llm.input_messages.{i}.message.role` | `"system"` / `"user"` / `"assistant"` / `"tool"` for the i-th input message | +| `llm.input_messages.{i}.message.content` | text content of the i-th input message | +| `llm.output_messages.{i}.message.role` | role of the i-th output message | +| `llm.output_messages.{i}.message.content` | text content of the i-th output message | +| `llm.token_count.prompt` | int — prompt/input tokens | +| `llm.token_count.completion` | int — completion/output tokens | +| `llm.token_count.total` | int — total tokens | + +In Python and TypeScript these names are exposed via `openinference-semantic-conventions` packages; in Go they must be hand-typed as the strings above. + **Python pattern:** Get the global tracer (same provider as Arize), then use context managers so tool spans are children of the CHAIN span and appear in the same trace as the LLM spans: ```python @@ -177,7 +203,51 @@ with tracer.start_as_current_span("run_agent") as chain_span: chain_span.set_attribute("output.value", final_reply) ``` -See [Manual instrumentation](https://arize.com/docs/ax/observe/tracing/setup/manual-instrumentation) for more span kinds and attributes. +**Go pattern:** Get a tracer from the global TracerProvider (registered via `otel.SetTracerProvider`), then nest spans with `tracer.Start` so tool spans become children of the CHAIN span. + +> **Critical for short-lived processes:** never call `log.Fatalf` / `os.Exit` after a span has started — they skip the deferred `tp.Shutdown(ctx)` and the in-flight CHAIN/LLM spans never flush. Use `log.Printf` + `return` from `main` instead, and keep `tp.Shutdown(ctx)` deferred at the top of `main`. + +```go +import ( + "context" + "encoding/json" + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" +) + +var tracer = otel.Tracer("my-app") + +func runAgent(ctx context.Context, userMessage string) string { + ctx, chainSpan := tracer.Start(ctx, "run_agent") + defer chainSpan.End() + chainSpan.SetAttributes( + attribute.String("openinference.span.kind", "CHAIN"), + attribute.String("input.value", userMessage), + ) + + // ... LLM call ... + for _, toolUse := range toolUses { + ctx, toolSpan := tracer.Start(ctx, toolUse.Name) + argsJSON, err := json.Marshal(toolUse.Input) + if err != nil { + toolSpan.RecordError(err) + } + toolSpan.SetAttributes( + attribute.String("openinference.span.kind", "TOOL"), + attribute.String("input.value", string(argsJSON)), + ) + result := runTool(toolUse.Name, toolUse.Input) + toolSpan.SetAttributes(attribute.String("output.value", result)) + toolSpan.End() + // ... append tool result to messages, call LLM again ... + } + + chainSpan.SetAttributes(attribute.String("output.value", finalReply)) + return finalReply +} +``` + +See [Manual instrumentation](https://arize.com/docs/ax/instrument/manual-instrumentation) for more span kinds and attributes. ## Verification @@ -192,7 +262,7 @@ After implementation: 1. Run the application and trigger at least one LLM call. 2. **Use the `arize-trace` skill** to confirm traces arrived. If empty, retry shortly. Verify spans have expected `openinference.span.kind`, `input.value`/`output.value`, and parent-child relationships. -3. If no traces: verify `ARIZE_SPACE` and `ARIZE_API_KEY`, ensure tracer is initialized before instrumentors and clients, check connectivity to `otlp.arize.com:443`, and inspect app/runtime exporter logs so you can tell whether spans are being emitted locally but rejected remotely. For debug set `GRPC_VERBOSITY=debug` or pass `log_to_console=True` to `register()`. Common gotchas: (a) missing project name resource attribute causes HTTP 500 rejections — `service.name` alone is not enough; Python: pass `project_name` to `register()`; TypeScript: set `"model_id"` or `SEMRESATTRS_PROJECT_NAME` on the resource; (b) CLI/script processes exit before OTLP exports flush — call `provider.force_flush()` then `provider.shutdown()` before exit; (c) CLI-visible spaces/projects can disagree with a collector-targeted space ID — report the mismatch instead of silently rewriting credentials. +3. If no traces: verify `ARIZE_SPACE` and `ARIZE_API_KEY`, ensure tracer is initialized before instrumentors and clients, check connectivity to `otlp.arize.com:443`, and inspect app/runtime exporter logs so you can tell whether spans are being emitted locally but rejected remotely. For debug set `GRPC_VERBOSITY=debug` or pass `log_to_console=True` to `register()`. Common gotchas: (a) missing project name resource attribute causes HTTP 500 rejections — `service.name` alone is not enough; Python: pass `project_name` to `register()`; TypeScript: set `"model_id"` or `SEMRESATTRS_PROJECT_NAME` on the resource; Go: add `attribute.String("openinference.project.name", "my-app")` to `resource.New(...)`; (b) CLI/script processes exit before OTLP exports flush — call `provider.force_flush()` then `provider.shutdown()` (Python/TS) or `tp.Shutdown(ctx)` (Go) before exit; (c) CLI-visible spaces/projects can disagree with a collector-targeted space ID — report the mismatch instead of silently rewriting credentials. 4. If the app uses tools: confirm CHAIN and TOOL spans appear with `input.value` / `output.value` so tool calls and results are visible. When verification is blocked by CLI or account issues, end with a concrete status: diff --git a/skills/arize-link/SKILL.md b/skills/arize-link/SKILL.md index fbb3a339d..44d9f470a 100644 --- a/skills/arize-link/SKILL.md +++ b/skills/arize-link/SKILL.md @@ -1,6 +1,9 @@ --- name: arize-link -description: Generate deep links to the Arize UI. Use when the user wants a clickable URL to open or share a specific trace, span, session, dataset, labeling queue, evaluator, or annotation config, or when sharing Arize resources with team members. +description: Generates deep links to the Arize UI for traces, spans, sessions, datasets, labeling queues, evaluators, and annotation configs. Produces clickable URLs for sharing Arize resources with team members. Use when the user wants to link to or open a trace, span, session, dataset, evaluator, or annotation config in the Arize UI. +metadata: + author: arize + version: "1.0" --- # Arize Link diff --git a/skills/arize-prompt-optimization/SKILL.md b/skills/arize-prompt-optimization/SKILL.md index 968255da1..12b381467 100644 --- a/skills/arize-prompt-optimization/SKILL.md +++ b/skills/arize-prompt-optimization/SKILL.md @@ -1,6 +1,10 @@ --- name: arize-prompt-optimization -description: "INVOKE THIS SKILL when optimizing, improving, or debugging LLM prompts using production trace data, evaluations, and annotations. Also use when the user wants to make their AI respond better or improve AI output quality. Covers extracting prompts from spans, gathering performance signal, and running a data-driven optimization loop using the ax CLI." +description: Optimizes, improves, and debugs LLM prompts using production trace data, evaluations, and annotations. Extracts prompts from spans, gathers performance signal, and runs a data-driven optimization loop using the ax CLI. Use when the user mentions optimize prompt, improve prompt, make AI respond better, improve output quality, prompt engineering, prompt tuning, or system prompt improvement. +metadata: + author: arize + version: "1.0" +compatibility: Requires the ax CLI and a configured Arize profile. --- # Arize Prompt Optimization Skill diff --git a/skills/arize-trace/SKILL.md b/skills/arize-trace/SKILL.md index 28420f2f3..5b76c5821 100644 --- a/skills/arize-trace/SKILL.md +++ b/skills/arize-trace/SKILL.md @@ -1,6 +1,10 @@ --- name: arize-trace -description: "INVOKE THIS SKILL when downloading, exporting, or inspecting Arize traces and spans, or when a user wants to look at what their LLM app is doing using existing trace data, or when an already-instrumented app has a bug or error to investigate. Use for debugging unknown runtime issues, failures, and behavior regressions. Covers exporting traces by ID, spans by ID, sessions by ID, and root-cause investigation with the ax CLI." +description: Downloads, exports, and inspects existing Arize traces and spans to understand what an LLM app is doing or debug runtime issues. Covers exporting traces by ID, spans by ID, sessions by ID, and root-cause investigation using the ax CLI. Use when the user wants to look at existing trace data, see what their LLM app is doing, export traces, download spans, investigate errors, or analyze behavior regressions. +metadata: + author: arize + version: "1.0" +compatibility: Requires the ax CLI and a configured Arize profile. --- # Arize Trace Skill diff --git a/skills/phoenix-cli/SKILL.md b/skills/phoenix-cli/SKILL.md index 3134e9175..bb7d71d9e 100644 --- a/skills/phoenix-cli/SKILL.md +++ b/skills/phoenix-cli/SKILL.md @@ -24,18 +24,28 @@ px trace list px trace get px trace annotate px trace add-note +px trace-annotations delete px span list px span annotate px span add-note +px span-annotations delete px session list px session get px session annotate px session add-note +px session-annotations delete px dataset list px dataset get px project list +px project get px annotation-config list px auth status +px profile list +px profile show [name] +px profile create +px profile use +px profile edit +px profile delete ``` ## Setup @@ -52,9 +62,13 @@ Always use `--format raw --no-progress` when piping to `jq`. | Task | Files | | ---- | ----- | -| Look at sampled traces and write specific notes about what went wrong (no taxonomy yet) | [references/open-coding](references/open-coding.md) | +| Look at sampled traces, spans, or sessions and write specific notes about what went wrong (no taxonomy yet) | [references/open-coding](references/open-coding.md) | | Group those notes into a structured failure taxonomy and quantify what matters | [references/axial-coding](references/axial-coding.md) | +Both stages tag every artifact with one shared **coding annotation identifier** (descriptive shape, e.g. `coding-run:chatbot-context-loss-2026-05-06`) so the run is queryable, reversible, and viewable as a unit. Pass `--identifier ` explicitly on every `px` call — shell inheritance is unreliable across agent harnesses. Open coding writes notes via `px ... add-note` and records a small local JSONL sidecar at `.px/coding/.jsonl`; axial coding reads that sidecar as the deterministic handoff and records labels in `.px/coding/-axial.jsonl`. Pick the identifier once per run (see [references/open-coding.md](references/open-coding.md#coding-annotation-identifier-pick-this-first)), then share the Phoenix UI link from the wrap-up section. Revert is opt-in and runs three identifier-bound DELETEs only after explicit user confirmation. + +> **Workflow term vs. server annotation name.** The skill prose calls this value the **coding annotation identifier** (shell-variable hint: `CODING_ANNOTATION_IDENTIFIER`). The server-side annotation NAME used for the UI filter is unchanged — `coding_session_id` — for data compatibility with rows already written by previous runs. Don't try to rename the server-side annotation; treat the asymmetry as load-bearing. + ## Workflows **"What do I do after instrumenting?" / "Where do I focus?" / "What's going wrong?"** @@ -64,7 +78,7 @@ Always use `--format raw --no-progress` when piping to `jq`. | Prefix | Description | | ------ | ----------- | -| `references/open-coding` | Free-form notes against sampled traces — reach for it whenever the user wants to make sense of traces but has no failure categories yet | +| `references/open-coding` | Free-form notes against sampled traces, spans, or sessions — reach for it whenever the user wants to make sense of LLM traffic but has no failure categories yet. Includes a unit-of-analysis diagnostic so the workflow runs at the level the failure modes actually live at (trace for stateless single-shot calls, session for multi-turn agents, span for mechanical/in-isolation failures). | | `references/axial-coding` | Inductive grouping of notes into a MECE taxonomy with counts — reach for it whenever the user has observations and needs categories or eval targets | ## Auth @@ -72,15 +86,46 @@ Always use `--format raw --no-progress` when piping to `jq`. ```bash px auth status # check connection and authentication px auth status --endpoint http://other:6006 # check a specific endpoint +px auth status --profile staging # check a named profile's connection +``` + +## Profiles + +Named profiles let you switch between multiple Phoenix instances (local, staging, cloud) without juggling environment variables. Profiles are stored in `~/.px/settings.json` (or `$XDG_CONFIG_HOME/px/settings.json`). + +Configuration priority (highest to lowest): CLI flags > env vars > active profile > built-in defaults. + +```bash +px profile list # list all profiles (shows active profile) +px profile show # show the active profile's settings +px profile show staging # show a named profile's settings +px profile create prod --endpoint https://app.phoenix.arize.com --api-key --activate +px profile create local --endpoint http://localhost:6006 --project my-app +px profile use prod # switch the active profile +px profile edit prod # open profile JSON in $EDITOR (validates on save) +px profile delete prod --yes # delete a profile (--yes skips confirmation) ``` +Use `--profile ` on any command to target a specific profile without changing the active one: + +```bash +px trace list --profile staging --limit 10 --format raw --no-progress | jq . +px auth status --profile prod +``` + +`px profile create` options: `--endpoint `, `--project `, `--api-key `, `--header ` (repeatable), `--activate`. + ## Projects ```bash px project list # list all projects (table view) px project list --format raw --no-progress | jq '.[].name' # project names as JSON +px project get my-project --format raw --no-progress # single record by exact name +px project get my-project --format raw --no-progress | jq -r '.id' # extract project id ``` +`project get` exits with `ExitCode.FAILURE` (1) on a name miss and writes a `StructuredError` `{error, code: "FAILURE", hint}` to stderr in `--format json|raw`. + ## Traces ```bash @@ -94,9 +139,14 @@ px trace get --format raw | jq '.spans[] | select(.status_code != "OK px trace get --include-notes --format raw | jq '.notes' px trace annotate --name reviewer --label pass px trace annotate --name reviewer --score 0.9 --format raw --no-progress +px trace annotate --name reviewer --label pass --identifier "" # tag with a coding annotation identifier px trace add-note --text "needs follow-up" +px trace add-note --text "needs follow-up" --identifier "" # tag + upsert on identifier +px trace-annotations delete --identifier "" --all -y # nuke every annotation tied to this coding annotation identifier ``` +`px -annotations delete` requires `--all` or both `--start-time` and `--end-time` and emits `{deleted: true, target, filter}` on success. + ### Trace JSON shape ``` @@ -147,7 +197,10 @@ px span list output.json --limit 100 # save to JSON file px span list --format raw --no-progress | jq '.[] | select(.status_code == "ERROR")' px span annotate --name reviewer --label pass px span annotate --name checker --score 1 --annotator-kind CODE +px span annotate --name reviewer --label pass --identifier "" # tag with a coding annotation identifier px span add-note --text "verified by agent" +px span add-note --text "verified by agent" --identifier "" # tag + upsert on identifier +px span-annotations delete --identifier "" --all -y # nuke every annotation tied to this coding annotation identifier ``` ### Span JSON shape @@ -183,7 +236,10 @@ px session get --include-annotations --format raw | jq '.session.an px session get --include-notes --format raw | jq '.session.notes' px session annotate --name reviewer --label pass px session annotate --name reviewer --score 0.9 --format raw --no-progress +px session annotate --name reviewer --label pass --identifier "" # tag with a coding annotation identifier px session add-note --text "verified by agent" +px session add-note --text "verified by agent" --identifier "" # tag + upsert on identifier +px session-annotations delete --identifier "" --all -y # nuke every annotation tied to this coding annotation identifier ``` ### Session JSON shape @@ -192,6 +248,7 @@ px session add-note --text "verified by agent" SessionData id, session_id, project_id start_time, end_time + token_count_prompt, token_count_completion, token_count_total — cumulative across all LLM spans in the session (int, default 0) annotations[] (with --include-annotations, excludes note) name, result { score, label, explanation } notes[] (with --include-notes) diff --git a/skills/phoenix-cli/references/axial-coding.md b/skills/phoenix-cli/references/axial-coding.md index b0b961964..b37804877 100644 --- a/skills/phoenix-cli/references/axial-coding.md +++ b/skills/phoenix-cli/references/axial-coding.md @@ -4,17 +4,40 @@ Group open-ended observations into structured failure taxonomies. Axial coding t **Reach for this whenever** the user has observations and needs structure — e.g., "what categories of failures do we have", "what should I build evals for", "how do I prioritize fixes", "group these notes", "MECE breakdown", or any framing that asks for categories or counts grounded in real traces rather than invented top-down. +## Coding annotation identifier (reuse the open-coding value) + +Reuse the **coding annotation identifier** chosen in open coding — every `annotate` call below passes `--identifier "$CODING_ANNOTATION_IDENTIFIER"` explicitly. In a fresh shell or fresh agent invocation, set `CODING_ANNOTATION_IDENTIFIER` to the same value (recoverable from the wrap-up UI URL or by listing `.px/coding/*.jsonl`); don't mint a new id. See [open-coding.md#coding-annotation-identifier-pick-this-first](open-coding.md#coding-annotation-identifier-pick-this-first) for the rationale and the sanitization rule. + +> **Workflow term vs. server annotation name.** The skill calls this value the **coding annotation identifier**; the server annotation NAME used for the UI filter stays `coding_session_id` for data compatibility. Don't try to rename the server-side key. + +```bash +CODING_ANNOTATION_IDENTIFIER="coding-run:chatbot-context-loss-2026-05-06" +SLUG=$(echo -n "$CODING_ANNOTATION_IDENTIFIER" | sed 's/[^a-zA-Z0-9_-]/-/g') +NOTES_SIDECAR=".px/coding/${SLUG}.jsonl" +AXIAL_SIDECAR=".px/coding/${SLUG}-axial.jsonl" +``` + ## Choosing the unit -Open-coding notes are usually **trace-level** (see [open-coding.md#choosing-the-unit](open-coding.md#choosing-the-unit)) — examples below lead with `px trace` and fall back to `px span` for span-level notes. **An axial label can live at a different level than the note that informed it** — that's a feature: a trace-level note "answered shipping when asked returns" can produce a span-level annotation on the retrieval span once a pattern reveals retrieval as the consistent culprit. Re-attribution at axial coding time is what axial coding *is*. Session-level rollups go through REST `/v1/projects/{id}/session_annotations` (no CLI write path). +Open coding's diagnostic in [open-coding.md#choosing-the-unit-of-analysis](open-coding.md#choosing-the-unit-of-analysis) commits to a unit (trace, span, or session). Axial coding inherits that unit by default — if open coding ran at the session level, axial labels will too; same for trace and span. + +**An axial label can live at a different level than the note that informed it** — that's a feature, and it works in every direction: + +- *Trace → span*: a trace-level note "answered shipping when asked about returns" can produce a span-level annotation on the retrieval span once a pattern reveals retrieval as the consistent culprit. +- *Trace → session*: a batch of trace-level notes describing single-turn confusion can produce a session-level annotation once you see the pattern is "the agent doesn't track the user's stated context across turns." +- *Session → trace*: a session-level note about cross-turn drift may, on closer reading, attribute to one specific turn where the agent dropped the thread; a trace-level annotation can name that turn. + +Whichever level you write the axial label on, write the matching `coding_session_id` UI-filter annotation on the same entity (see [UI-filter annotation](#ui-filter-annotation) below) so the UI link picks it up. ## Process -1. **Gather** — Collect open-coding notes from the entities you reviewed (trace-level by default) -2. **Pattern** — Group notes with common themes -3. **Name** — Create actionable category names -4. **Attribute** — Decide what level each category lives at; an axial label can move from the note's level to the component the pattern implicates -5. **Quantify** — Count failures per category +1. **Set the coding annotation identifier** — set `CODING_ANNOTATION_IDENTIFIER` to the value used in open coding and re-derive `SLUG`, `NOTES_SIDECAR`, `AXIAL_SIDECAR` (see [Coding annotation identifier](#coding-annotation-identifier-reuse-the-open-coding-value)) +2. **Gather** — read open-coding notes from `$NOTES_SIDECAR` (at the unit committed in open coding); no server round-trip +3. **Pattern** — group notes with common themes +4. **Name** — create actionable category names +5. **Attribute** — decide what level each category lives at; an axial label can move up (trace → session) or down (trace → span) from the source note's level to the level the pattern actually implicates +6. **Record** — `px {trace,span,session} annotate ... --name axial_coding_category --label --identifier "$CODING_ANNOTATION_IDENTIFIER"`, add/update one JSONL sidecar row for the label, then write the matching `coding_session_id` UI-filter annotation +7. **Quantify** — count failures per category from `$AXIAL_SIDECAR` ## Example Taxonomy @@ -39,97 +62,110 @@ failure_taxonomy: ## Reading -### 1. Gather — extract open-coding notes +### 1. Gather — read this run's open-coding notes from the sidecar -Open-coding notes are stored as annotations with `name="note"` and are only returned when `--include-notes` is passed. Use `--include-annotations` instead and you will get structured annotations but **not** notes — the server excludes notes from the annotations array. +Open-coding wrote one JSONL line per note to `$NOTES_SIDECAR` (`.px/coding/${SLUG}.jsonl`). Read it directly — no server round-trip is needed. Each line has `entity_kind`, `entity_id`, `note`, `identifier`, and `ts`. If the same `(entity_kind, entity_id)` appears more than once, use the newest `ts` as the current note. -```bash -# Trace-level notes (default for open coding) -px trace list --include-notes --format raw --no-progress | jq ' - [ .[] | select((.notes // []) | length > 0) ] - | map({ trace_id: .traceId, notes: [ .notes[].result.explanation ] }) -' +**Missing-file behavior.** An absent `$NOTES_SIDECAR` means open coding hasn't run for this coding annotation identifier in this CWD — stop and run open coding first, do not silently treat it as zero notes. -# Span-level notes (when open coding dropped to span for mechanical failures) -px span list --include-notes --format raw --no-progress | jq ' - [ .[] | select((.notes // []) | length > 0) ] - | map({ span_id: .context.span_id, notes: [ .notes[].result.explanation ] }) -' -``` +**Malformed lines.** Each line is independently parseable JSON. If `jq` reports a parse error, fix or drop that line manually; do not edit other lines. + +**Notes outside this run.** The sidecar only carries notes this CWD wrote. To pull notes another reviewer or earlier run wrote, fetch them via `px {trace,span,session} list --include-notes` (embeds notes into row output) — the workflow's sidecar is intentionally per-CWD-per-coding-identifier. ### 2. Group — synthesize categories Review the note text collected above. Manually identify recurring themes and draft candidate category names. Aim for MECE coverage: each note should fit exactly one category. -### 3. Record — write axial-coding annotations +### 3. Record — write axial-coding labels -Write one annotation per entity using `px trace annotate` or `px span annotate`. The level can differ from where the source note lives — see the **Recording** section below. +Write one annotation per entity using `px {trace,span,session} annotate`, passing `--identifier "$CODING_ANNOTATION_IDENTIFIER"` explicitly on every call, and record one JSONL row in `$AXIAL_SIDECAR` so [Quantify](#4-quantify--count-per-category-from-the-axial-sidecar) below can count without a server round-trip. The level can differ from where the source note lives — see [Recording](#recording) below. -### 4. Quantify — count per category +### 4. Quantify — count per category from the axial sidecar -After recording, use `--include-annotations` to count how many entities carry each label. Examples below show span-level counts; for trace-level annotations, swap `px span list` for `px trace list` (the `.annotations[]` shape is the same). +Counts come from `$AXIAL_SIDECAR` (populated by [Record](#3-record--write-axial-coding-labels)). No server query, no project-wide history mixed in — the sidecar holds exactly the labels this run wrote. Count the current rows by `axial_label`; if an entity appears more than once, use the newest `ts`. -```bash -px span list --include-annotations --format raw --no-progress | jq ' - [ .[] | .annotations[]? | select(.name == "failure_category" and .result.label != null) ] - | group_by(.result.label) - | map({ label: .[0].result.label, count: length }) - | sort_by(-.count) -' -``` +Same missing-file and malformed-line rules as `$NOTES_SIDECAR`: a missing axial sidecar means no labels have been written yet (run [Record](#3-record--write-axial-coding-labels)); malformed lines are line-local — fix or drop, don't edit neighbors. -Filter to a specific annotation name to check coverage: +## Recording -```bash -px span list --include-annotations --format raw --no-progress | jq ' - [ .[] | select((.annotations // []) | any(.name == "failure_category")) ] - | length -' +Use the matching annotate command for the level the **label** belongs at — which may differ from where the source note lives (see [Choosing the unit](#choosing-the-unit)). Every call carries `--identifier "$CODING_ANNOTATION_IDENTIFIER"` and `--format raw --no-progress`, and is paired with a JSONL row in `$AXIAL_SIDECAR`. + +**Axial sidecar JSONL line shape (one per `annotate`):** + +```json +{"entity_kind":"trace","entity_id":"","annotation_name":"axial_coding_category","axial_label":"