Sync .github directory with azure-sdk-tools for PR 15257#48917
Open
Sync .github directory with azure-sdk-tools for PR 15257#48917
Conversation
azure-sdk
added
EngSys
Contributor
There was a problem hiding this comment.
Pull request overview
Syncs the .github/skills/azsdk-common-live-and-recorded-tests shared skill content from azure-sdk-tools into this repo, adding an evaluation suite plus task/reference assets to validate the skill behavior around live/record/playback testing workflows.
Changes:
- Added a new shared skill definition for deploying test resources and running tests in live/record/playback modes.
- Added an eval suite (
eval.yaml) and several task YAMLs (happy-path + anti-trigger) to exercise the skill. - Added reference markdown files documenting test modes and test resource deployment/cleanup scripts.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/skills/azsdk-common-live-and-recorded-tests/SKILL.md | Defines the shared skill, its triggers, and the step-by-step workflow. |
| .github/skills/azsdk-common-live-and-recorded-tests/eval.yaml | Declares the evaluation suite configuration and graders. |
| .github/skills/azsdk-common-live-and-recorded-tests/tasks/deploy-and-run-live.yaml | Happy-path eval task for deploying resources and running live tests. |
| .github/skills/azsdk-common-live-and-recorded-tests/tasks/record-with-env.yaml | Happy-path eval task for recording using an existing environment file. |
| .github/skills/azsdk-common-live-and-recorded-tests/tasks/cleanup-resources.yaml | Happy-path eval task for cleaning up test resources after a run. |
| .github/skills/azsdk-common-live-and-recorded-tests/tasks/anti-trigger.yaml | Negative eval task to ensure playback-only prompts don’t trigger deployment guidance. |
| .github/skills/azsdk-common-live-and-recorded-tests/tasks/anti-trigger-write-tests.yaml | Negative eval task to ensure “write tests” requests don’t trigger this skill. |
| .github/skills/azsdk-common-live-and-recorded-tests/references/test-resource-deployment.md | Reference doc for New-TestResources.ps1 usage and parameters. |
| .github/skills/azsdk-common-live-and-recorded-tests/references/test-resource-cleanup.md | Reference doc for Remove-TestResources.ps1 usage and parameters. |
| .github/skills/azsdk-common-live-and-recorded-tests/references/test-modes.md | Reference doc describing playback/record/live modes. |
| metadata: | ||
| version: "0.1.0" | ||
| distribution: shared | ||
| description: "Deploy test resources and run Azure SDK tests in live, record, or playback mode. WHEN: \"run live tests\", \"run recorded tests\", \"deploy test resources\", \"record tests\", \"run tests in record mode\", \"clean up test resources\", \"run tests against live resources\". DO NOT USE FOR: writing new tests, authoring Bicep templates, playback-only test runs without resource deployment. INVOKES: azure-sdk-mcp:azsdk_package_run_tests." |
There was a problem hiding this comment.
The SKILL.md frontmatter description uses comma-separated trigger phrases with embedded quotes. In this repo, skill descriptions are expected to use semicolons to separate trigger phrases (YAML-safe) for vally lint consistency; please rewrite the WHEN/DO NOT USE triggers using semicolons instead of quoted comma lists.
Suggested change
| description: "Deploy test resources and run Azure SDK tests in live, record, or playback mode. WHEN: \"run live tests\", \"run recorded tests\", \"deploy test resources\", \"record tests\", \"run tests in record mode\", \"clean up test resources\", \"run tests against live resources\". DO NOT USE FOR: writing new tests, authoring Bicep templates, playback-only test runs without resource deployment. INVOKES: azure-sdk-mcp:azsdk_package_run_tests." | |
| description: "Deploy test resources and run Azure SDK tests in live, record, or playback mode. WHEN: run live tests; run recorded tests; deploy test resources; record tests; run tests in record mode; clean up test resources; run tests against live resources. DO NOT USE FOR: writing new tests; authoring Bicep templates; playback-only test runs without resource deployment. INVOKES: azure-sdk-mcp:azsdk_package_run_tests." |
| 4. **Confirm deployment** — Even if no existing `.env` file is found, confirm with the user before proceeding to deploy test resources. Deployment creates Azure resources that may incur costs. | ||
| 5. **Deploy test resources** — Run `eng/common/TestResources/New-TestResources.ps1` with the service directory and any user-provided parameters. See [deployment parameters](references/test-resource-deployment.md) for details. The script outputs environment variables needed for live/record test runs. | ||
| 6. **Save environment** — Capture the environment variables output by the deployment script. If the script writes a `.env` file, note its path. Otherwise, collect the environment variables from the script output. | ||
| 7. **Run tests** — Run `azure-sdk-mcp:azsdk_package_run_tests` with the appropriate test mode (`record`, `live`, or `playback`) and the path to the `.env` file containing test environment variables. When tests run in record mode and all tests pass, the tool automatically pushes recorded test assets to the assets repo. |
There was a problem hiding this comment.
references/test-modes.md exists under this skill but isn't linked anywhere from SKILL.md. vally lint's orphan-files check expects every file in references/ to be referenced; either add a link to test-modes.md from SKILL.md or remove the unused reference file.
Suggested change
| 7. **Run tests** — Run `azure-sdk-mcp:azsdk_package_run_tests` with the appropriate test mode (`record`, `live`, or `playback`) and the path to the `.env` file containing test environment variables. When tests run in record mode and all tests pass, the tool automatically pushes recorded test assets to the assets repo. | |
| 7. **Run tests** — Run `azure-sdk-mcp:azsdk_package_run_tests` with the appropriate test mode (`record`, `live`, or `playback`) and the path to the `.env` file containing test environment variables. See [test modes](references/test-modes.md) for guidance. When tests run in record mode and all tests pass, the tool automatically pushes recorded test assets to the assets repo. |
Comment on lines
+30
to
+34
| 2. **Check for existing deployment** — Look for a `.env` file at the service directory level (next to `test-resources.bicep`). If one exists, inform the user that a previous deployment appears to be available and ask whether to **reuse the existing deployment** or **redeploy test resources**. If reusing, skip to step 7. | ||
| 3. **Verify Azure context** — Run `Get-AzContext` to check for an active Azure PowerShell session. If no context exists, guide the user through `Connect-AzAccount`. Confirm the correct subscription is selected. | ||
| 4. **Confirm deployment** — Even if no existing `.env` file is found, confirm with the user before proceeding to deploy test resources. Deployment creates Azure resources that may incur costs. | ||
| 5. **Deploy test resources** — Run `eng/common/TestResources/New-TestResources.ps1` with the service directory and any user-provided parameters. See [deployment parameters](references/test-resource-deployment.md) for details. The script outputs environment variables needed for live/record test runs. | ||
| 6. **Save environment** — Capture the environment variables output by the deployment script. If the script writes a `.env` file, note its path. Otherwise, collect the environment variables from the script output. |
There was a problem hiding this comment.
The steps assume a .env file exists at the service directory level next to test-resources.bicep. In azure-sdk-for-java many services use test-resources/test-resources.json, and $SupportsTestResourcesDotenv isn't enabled, so the common output is test-resources.json.env (or test-resources.bicep.env) next to the template. Update steps 2/6/7 to refer to the actual test-resources.*.env output file(s) rather than a generic .env next to test-resources.bicep.
| - happy-path | ||
| - live-mode | ||
| inputs: | ||
| prompt: "Deploy test resources and run live tests for the keyvault package at ~/repos/azure-sdk-for-python/sdk/keyvault/azure-keyvault-keys" |
There was a problem hiding this comment.
This eval task prompt points to an azure-sdk-for-python repo path and a package name (azure-keyvault-keys) that doesn't exist in this repo. To keep the eval representative for azure-sdk-for-java (and avoid cross-language assumptions), update the prompt to reference a Java package path (e.g., sdk/keyvault/azure-security-keyvault-keys).
Suggested change
| prompt: "Deploy test resources and run live tests for the keyvault package at ~/repos/azure-sdk-for-python/sdk/keyvault/azure-keyvault-keys" | |
| prompt: "Deploy test resources and run live tests for the keyvault package at ~/repos/azure-sdk-for-java/sdk/keyvault/azure-security-keyvault-keys" |
| - happy-path | ||
| - record-mode | ||
| inputs: | ||
| prompt: "Run the tests in record mode using the live test deployment specified in sdk/keyvault/azure-keyvault-keys/.env" |
There was a problem hiding this comment.
This task references sdk/keyvault/azure-keyvault-keys/.env, but in this repo the Key Vault package is sdk/keyvault/azure-security-keyvault-keys, and test resources typically produce test-resources.*.env files (e.g., test-resources.json.env) rather than a top-level .env. Update the prompt to use a path/file name that matches azure-sdk-for-java's test resources layout.
Suggested change
| prompt: "Run the tests in record mode using the live test deployment specified in sdk/keyvault/azure-keyvault-keys/.env" | |
| prompt: "Run the tests in record mode using the live test deployment specified in sdk/keyvault/azure-security-keyvault-keys/test-resources.json.env" |
azure-sdk
force-pushed
the
sync-.github-skills/live-testing-skill-15257
branch
from
000dbae to
6499121
Compare
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <copilot@github.com>
azure-sdk
force-pushed
the
sync-.github-skills/live-testing-skill-15257
branch
from
6499121 to
5c8128f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sync .github directory with azure-sdk-tools for PR Azure/azure-sdk-tools#15257 See eng/common workflow