Sync .github directory with azure-sdk-tools for PR 15210#48875
Open
Sync .github directory with azure-sdk-tools for PR 15210#48875
Conversation
azure-sdk
added
EngSys
Contributor
There was a problem hiding this comment.
Pull request overview
This PR syncs the repository’s .github/skills/ content with azure-sdk-tools PR 15210 by adding/refreshing shared “utility skills” and their associated task/eval configurations used to validate triggering and behavior.
Changes:
- Add new shared skills for SDK release, release-plan preparation, pipeline troubleshooting, local SDK generation, and APIView feedback resolution.
- Add task suites (
tasks/*.yaml) and trigger tests (evals/trigger_tests.yaml) for each skill. - Add evaluation configs (
eval.yamlandevals/eval.yaml) plus fixtures/references supporting the skills.
Reviewed changes
Copilot reviewed 64 out of 64 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/skills/azsdk-common-sdk-release/tasks/release-trigger-001.yaml | Adds a task validating release pipeline triggering behavior. |
| .github/skills/azsdk-common-sdk-release/tasks/release-readiness-001.yaml | Adds a task validating readiness check behavior. |
| .github/skills/azsdk-common-sdk-release/tasks/release-negative-001.yaml | Adds a negative task ensuring unrelated prompts don’t trigger. |
| .github/skills/azsdk-common-sdk-release/tasks/release-basic-001.yaml | Adds a “full flow” release task. |
| .github/skills/azsdk-common-sdk-release/evals/trigger_tests.yaml | Adds trigger/anti-trigger prompt set for activation testing. |
| .github/skills/azsdk-common-sdk-release/evals/tasks/basic-trigger.yaml | Adds eval task asserting the skill activates on release prompts. |
| .github/skills/azsdk-common-sdk-release/evals/tasks/anti-trigger.yaml | Adds eval task asserting the skill does not activate on unrelated prompts. |
| .github/skills/azsdk-common-sdk-release/evals/eval.yaml | Adds eval suite config for the skill (evals folder). |
| .github/skills/azsdk-common-sdk-release/eval.yaml | Adds eval suite config for the skill (root). |
| .github/skills/azsdk-common-sdk-release/SKILL.md | Adds shared skill documentation and MCP tool usage steps. |
| .github/skills/azsdk-common-prepare-release-plan/tasks/should-not-trigger.yaml | Adds negative task to prevent activation for package-release requests. |
| .github/skills/azsdk-common-prepare-release-plan/tasks/link-sdk-prs.yaml | Adds task for linking SDK PRs to an existing release plan. |
| .github/skills/azsdk-common-prepare-release-plan/tasks/edge-case.yaml | Adds edge-case task for invalid package name format handling. |
| .github/skills/azsdk-common-prepare-release-plan/tasks/basic-usage.yaml | Adds basic task for creating a release plan from a spec PR. |
| .github/skills/azsdk-common-prepare-release-plan/references/release-plan-details.md | Adds detailed guidance/reference content for release plan workflow. |
| .github/skills/azsdk-common-prepare-release-plan/fixtures/tspconfig.yaml | Adds fixture TypeSpec config used by tasks/evals. |
| .github/skills/azsdk-common-prepare-release-plan/evals/trigger_tests.yaml | Adds trigger/anti-trigger prompt set for activation testing. |
| .github/skills/azsdk-common-prepare-release-plan/evals/tasks/basic-trigger.yaml | Adds eval task asserting activation for release-plan prompts. |
| .github/skills/azsdk-common-prepare-release-plan/evals/tasks/anti-trigger.yaml | Adds eval task asserting non-activation for unrelated prompts. |
| .github/skills/azsdk-common-prepare-release-plan/evals/eval.yaml | Adds eval suite config for the skill (evals folder). |
| .github/skills/azsdk-common-prepare-release-plan/eval.yaml | Adds eval suite config for the skill (root). |
| .github/skills/azsdk-common-prepare-release-plan/SKILL.md | Adds shared skill documentation and MCP tool usage steps. |
| .github/skills/azsdk-common-pipeline-troubleshooting/tasks/should-not-trigger.yaml | Adds negative task to prevent activation for TypeSpec authoring prompts. |
| .github/skills/azsdk-common-pipeline-troubleshooting/tasks/local-reproduction.yaml | Adds task for guiding local reproduction of pipeline failures. |
| .github/skills/azsdk-common-pipeline-troubleshooting/tasks/edge-case.yaml | Adds edge-case task covering multi-language pipeline failures. |
| .github/skills/azsdk-common-pipeline-troubleshooting/tasks/basic-usage.yaml | Adds basic task for analyzing a pipeline build failure. |
| .github/skills/azsdk-common-pipeline-troubleshooting/references/failure-patterns.md | Adds reference content with common failure patterns/resolutions. |
| .github/skills/azsdk-common-pipeline-troubleshooting/evals/trigger_tests.yaml | Adds trigger/anti-trigger prompt set for activation testing. |
| .github/skills/azsdk-common-pipeline-troubleshooting/evals/tasks/basic-trigger.yaml | Adds eval task asserting activation for pipeline-troubleshooting prompts. |
| .github/skills/azsdk-common-pipeline-troubleshooting/evals/tasks/anti-trigger.yaml | Adds eval task asserting non-activation for unrelated prompts. |
| .github/skills/azsdk-common-pipeline-troubleshooting/evals/eval.yaml | Adds eval suite config for the skill (evals folder). |
| .github/skills/azsdk-common-pipeline-troubleshooting/eval.yaml | Adds eval suite config for the skill (root). |
| .github/skills/azsdk-common-pipeline-troubleshooting/SKILL.md | Adds shared skill documentation and MCP tool usage steps. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/update-version.yaml | Adds task for package version update flow. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/update-metadata.yaml | Adds task for metadata/ci.yml update flow. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/update-changelog.yaml | Adds task for changelog update flow. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/rename-client.yaml | Adds task for TypeSpec customization rename scenario. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/hide-operation.yaml | Adds task for hiding internal operations via customization. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/full-workflow.yaml | Adds task covering end-to-end local generation workflow. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/edge-case.yaml | Adds task for build-failure recovery via customization. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/customization-workflow.yaml | Adds task for duplicate-field conflicts post-regeneration. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/breaking-changes.yaml | Adds task for resolving breaking changes after spec updates. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/basic-usage.yaml | Adds basic task for local Python SDK generation. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/anti-trigger.yaml | Adds negative task for pipeline-based generation prompts. |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/analyzer-errors.yaml | Adds task for .NET analyzer errors handled via customization. |
| .github/skills/azsdk-common-generate-sdk-locally/references/sdk-repos.md | Adds reference mapping languages to SDK repos and tooling. |
| .github/skills/azsdk-common-generate-sdk-locally/references/detailed-workflow.md | Adds detailed workflow reference content. |
| .github/skills/azsdk-common-generate-sdk-locally/references/customization-workflow.md | Adds detailed customization workflow reference content. |
| .github/skills/azsdk-common-generate-sdk-locally/fixtures/tspconfig.yaml | Adds fixture TypeSpec config used by tasks/evals. |
| .github/skills/azsdk-common-generate-sdk-locally/eval.yaml | Adds eval suite config for the skill. |
| .github/skills/azsdk-common-generate-sdk-locally/SKILL.md | Adds shared skill documentation and MCP tool usage steps. |
| .github/skills/azsdk-common-apiview-feedback-resolution/tasks/should-not-trigger.yaml | Adds negative task to prevent activation for TypeSpec project creation prompts. |
| .github/skills/azsdk-common-apiview-feedback-resolution/tasks/no-feedback.yaml | Adds task covering the “no APIView feedback” scenario. |
| .github/skills/azsdk-common-apiview-feedback-resolution/tasks/edge-case.yaml | Adds edge-case task for feedback requiring TypeSpec changes. |
| .github/skills/azsdk-common-apiview-feedback-resolution/tasks/basic-usage.yaml | Adds basic task for retrieving and reviewing APIView comments. |
| .github/skills/azsdk-common-apiview-feedback-resolution/references/feedback-resolution-steps.md | Adds reference guidance for resolving APIView feedback. |
| .github/skills/azsdk-common-apiview-feedback-resolution/fixtures/apiview-comment.json | Adds fixture data representing APIView comments. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/trigger_tests.yaml | Adds trigger/anti-trigger prompt set for activation testing. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/tasks/url-trigger.yaml | Adds eval task asserting activation on APIView URL prompts. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/tasks/basic-trigger.yaml | Adds eval task asserting activation on APIView feedback prompts. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/tasks/anti-trigger.yaml | Adds eval task asserting non-activation on unrelated prompts. |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/eval.yaml | Adds eval suite config for the skill (evals folder). |
| .github/skills/azsdk-common-apiview-feedback-resolution/eval.yaml | Adds eval suite config for the skill (root). |
| .github/skills/azsdk-common-apiview-feedback-resolution/SKILL.md | Adds shared skill documentation and MCP tool usage steps. |
Comment on lines
+7
to
+8
| description: "Analyze and resolve APIView review feedback on Azure SDK PRs. **UTILITY SKILL**. USE FOR: APIView comments, API review feedback, SDK API surface changes. DO NOT USE FOR: general code review, non-APIView feedback. INVOKES: azure-sdk-mcp:azsdk_apiview_get_comments, azure-sdk-mcp:azsdk_typespec_customized_code_update." | ||
| compatibility: |
There was a problem hiding this comment.
azsdk_typespec_customized_code_update is referenced as an invoked MCP tool, but the rest of the repo (and the other shared skills in this PR) reference azsdk_customized_code_update for applying TypeSpec + code customizations. If the MCP tool name here is wrong, the skill will direct agents to call a tool that doesn’t exist; align this to the actual MCP tool name.
Comment on lines
+25
to
+26
| - (?i)error occurred | ||
| - (?i)failed |
There was a problem hiding this comment.
The regex_not_match grader forbids very common words like “failed” (and “error occurred”), but several tasks in this suite are explicitly about build failures. This will likely cause false eval failures even when the assistant responds correctly; consider narrowing these patterns to true fatal/unhandled conditions (similar to the other skill evals) instead of generic failure wording.
Suggested change
| - (?i)error occurred | |
| - (?i)failed | |
| - (?i)fatal error | |
| - (?i)unhandled exception |
Comment on lines
+16
to
+23
| | Tool | Purpose | | ||
| |------|---------| | ||
| | `azsdk_verify_setup` | Verify environment | | ||
| | `azsdk_package_generate_code` | Generate SDK | | ||
| | `azsdk_package_build_code` | Build package | | ||
| | `azsdk_package_run_check` | Validate package | | ||
| | `azsdk_package_run_tests` | Run tests | | ||
| | `azsdk_customized_code_update` | Apply customizations (includes regeneration and build) | |
There was a problem hiding this comment.
Tool names are inconsistently qualified in this skill (tool table lists azsdk_* without the azure-sdk-mcp: prefix, but the steps later instruct calling azure-sdk-mcp:azsdk_*). Standardizing the notation in one place helps avoid calling the wrong tool name in automation/evals.
Comment on lines
+20
to
+27
| graders: | ||
| - type: text | ||
| name: no_fatal_errors | ||
| config: | ||
| regex_not_match: | ||
| - (?i)fatal error | ||
| - (?i)unhandled exception | ||
| - '(?i)panic:' |
There was a problem hiding this comment.
graders list item is not indented under graders: (and regex_not_match items aren’t indented under the key), which makes this YAML invalid / changes the structure. Indent the list under graders: and the patterns under regex_not_match: so the file parses as intended.
Comment on lines
+20
to
+27
| graders: | ||
| - type: text | ||
| name: no_fatal_errors | ||
| config: | ||
| regex_not_match: | ||
| - (?i)fatal error | ||
| - (?i)unhandled exception | ||
| - '(?i)panic:' |
There was a problem hiding this comment.
graders list item is not indented under graders: (and regex_not_match items aren’t indented under the key), which makes this YAML invalid / changes the structure. Indent the list under graders: and the patterns under regex_not_match: so the file parses as intended.
Comment on lines
+21
to
+27
| - type: text | ||
| name: no_fatal_errors | ||
| config: | ||
| regex_not_match: | ||
| - (?i)fatal error | ||
| - (?i)unhandled exception | ||
| - '(?i)panic:' |
There was a problem hiding this comment.
graders list item is not indented under graders: (and regex_not_match items aren’t indented under the key), which makes this YAML invalid / changes the structure. Indent the list under graders: and the patterns under regex_not_match: so the file parses as intended.
Suggested change
| - type: text | |
| name: no_fatal_errors | |
| config: | |
| regex_not_match: | |
| - (?i)fatal error | |
| - (?i)unhandled exception | |
| - '(?i)panic:' | |
| - type: text | |
| name: no_fatal_errors | |
| config: | |
| regex_not_match: | |
| - (?i)fatal error | |
| - (?i)unhandled exception | |
| - '(?i)panic:' |
Member
There was a problem hiding this comment.
@ronniegeraghty do we want to track fixing the yaml's as a separate PR?
Comment on lines
+20
to
+27
| graders: | ||
| - type: text | ||
| name: no_fatal_errors | ||
| config: | ||
| regex_not_match: | ||
| - (?i)fatal error | ||
| - (?i)unhandled exception | ||
| - '(?i)panic:' |
There was a problem hiding this comment.
graders list item is not indented under graders: (and regex_not_match items aren’t indented under the key), which makes this YAML invalid / changes the structure. Indent the list under graders: and the patterns under regex_not_match: so the file parses as intended.
samvaity
reviewed
Apr 22, 2026
Comment on lines
+21
to
+27
| - type: text | ||
| name: no_fatal_errors | ||
| config: | ||
| regex_not_match: | ||
| - (?i)fatal error | ||
| - (?i)unhandled exception | ||
| - '(?i)panic:' |
Member
There was a problem hiding this comment.
@ronniegeraghty do we want to track fixing the yaml's as a separate PR?
| metadata: | ||
| version: "1.0.0" | ||
| distribution: shared | ||
| description: "Analyze and resolve APIView review feedback on Azure SDK PRs. **UTILITY SKILL**. USE FOR: APIView comments, API review feedback, SDK API surface changes. DO NOT USE FOR: general code review, non-APIView feedback. INVOKES: azure-sdk-mcp:azsdk_apiview_get_comments, azure-sdk-mcp:azsdk_typespec_customized_code_update." |
Member
There was a problem hiding this comment.
azsdk_typespec_customized_code_update is the wrong MCP tool name, it should be azsdk_customized_code_update
| | Tool | Purpose | | ||
| |------|---------| | ||
| | `azure-sdk-mcp:azsdk_apiview_get_comments` | Get APIView comments | | ||
| | `azure-sdk-mcp:azsdk_typespec_customized_code_update` | Apply TypeSpec changes locally | |
Member
There was a problem hiding this comment.
Suggested change
| | `azure-sdk-mcp:azsdk_typespec_customized_code_update` | Apply TypeSpec changes locally | | |
| | `azure-sdk-mcp:azsdk_customized_code_update` | Apply TypeSpec & code customization changes locally | |
|
|
||
| 1. **Retrieve** — Get APIView URL from SDK PR, run `azsdk_apiview_get_comments`. | ||
| 2. **Categorize** — Group as Critical/Suggestions/Informational per [feedback steps](references/feedback-resolution-steps.md). | ||
| 3. **Resolve** — Use `azsdk_typespec_customized_code_update` for TypeSpec changes; delegate via `azsdk_typespec_delegate_apiview_feedback` for complex cases. |
Member
There was a problem hiding this comment.
Suggested change
| 3. **Resolve** — Use `azsdk_typespec_customized_code_update` for TypeSpec changes; delegate via `azsdk_typespec_delegate_apiview_feedback` for complex cases. | |
| 3. **Resolve** — Use `azsdk_customized_code_update` for TypeSpec changes; delegate via `azsdk_typespec_delegate_apiview_feedback` for complex cases. |
azure-sdk
force-pushed
the
sync-.github-fix/azsdk-common-skill-rename-sync-15210
branch
from
bb2edc5 to
5548a4a
Compare
azure-sdk
pushed a commit
that referenced
this pull request
Apr 22, 2026
- Fix wrong MCP tool name: azsdk_typespec_customized_code_update -> azsdk_customized_code_update (apiview-feedback-resolution SKILL.md) - Add azure-sdk-mcp: prefix to tool table in generate-sdk-locally SKILL.md for consistency with step references - Narrow overly broad regex_not_match patterns in generate-sdk-locally eval.yaml (remove 'failed'/'error occurred' which cause false failures for build troubleshooting tasks) - Fix YAML indentation in 4 evals/eval.yaml files (graders section not properly nested under graders: key) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
azure-sdk
force-pushed
the
sync-.github-fix/azsdk-common-skill-rename-sync-15210
branch
from
5548a4a to
b00b112
Compare
Re-creates PR #15062 on a fresh branch off current upstream/main to avoid sync-pipeline CI issues on the original branch. Original branch squad/15049-sync-shared-skills and PR #15062 are preserved unchanged for comparison. Makes skills the single source of truth by consolidating instruction file content into the corresponding skill SKILL.md files, then deleting the redundant instruction files. Also renames 5 shared skill directories with the 'azsdk-common-' prefix required by the sync pipeline. 7 instruction files deleted (content merged into skills): - local-sdk-workflow.instructions.md -> azsdk-common-generate-sdk-locally - create-release-plan.instructions.md -> azsdk-common-prepare-release-plan - sdk-details-in-release-plan.instructions.md -> azsdk-common-prepare-release-plan - verify-namespace-approval.instructions.md -> azsdk-common-prepare-release-plan - check-package-readiness.instructions.md -> azsdk-common-sdk-release - check-package-validation.instructions.md -> azsdk-common-generate-sdk-locally - copilot/sdk-release.instructions.md -> azsdk-common-sdk-release 3 skills enriched with details from instruction files: - generate-sdk-locally: commit checkpoints, language selection, config file identification, SDK project path, metadata update tools, validation params - prepare-release-plan: critical LLM instructions (NextSteps field), Release Planner Tool link, per-language package name extraction rules, valid/invalid examples table, JSON structure, namespace approval flow - sdk-release: readiness check details, interaction flow, do-not-ask-for-PR guard, case-sensitive language note 5 shared skill directories renamed with 'azsdk-common-' prefix: apiview-feedback-resolution, generate-sdk-locally, pipeline-troubleshooting, prepare-release-plan, sdk-release typespec-to-sdk orchestrator updated to reference skills directly. Removed all related_skill frontmatter and blockquotes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lease-plan; only change is name prefix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…efix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix wrong MCP tool name: azsdk_typespec_customized_code_update -> azsdk_customized_code_update (apiview-feedback-resolution SKILL.md) - Add azure-sdk-mcp: prefix to tool table in generate-sdk-locally SKILL.md for consistency with step references - Narrow overly broad regex_not_match patterns in generate-sdk-locally eval.yaml (remove 'failed'/'error occurred' which cause false failures for build troubleshooting tasks) - Fix YAML indentation in 4 evals/eval.yaml files (graders section not properly nested under graders: key) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
azure-sdk
force-pushed
the
sync-.github-fix/azsdk-common-skill-rename-sync-15210
branch
from
b00b112 to
cdb8203
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sync .github directory with azure-sdk-tools for PR Azure/azure-sdk-tools#15210 See eng/common workflow