Skip to content

[Cosmos Benchmark]AddBenchmarkAgentAndSkills#48165

Draft
xinlian12 wants to merge 4 commits intoAzure:mainfrom
xinlian12:cosmos-benchmark-agent
Draft

[Cosmos Benchmark]AddBenchmarkAgentAndSkills#48165
xinlian12 wants to merge 4 commits intoAzure:mainfrom
xinlian12:cosmos-benchmark-agent

Conversation

@xinlian12
Copy link
Member

@xinlian12 xinlian12 commented Feb 27, 2026

Summary

Add a Copilot agent and 5 workflow-aligned skills for running Cosmos DB benchmark and DR drill workflows.

Benchmark Infrastructure

  • 8 shell/Python scripts for VM provisioning, setup, execution, monitoring, diagnostics, dashboards
  • Java changes to BenchmarkConfig, BenchmarkOrchestrator, TenantWorkloadConfig for multi-tenant orchestration
  • .gitignore updates and test scaffolding (no real credentials)

Agent + 5 Skills

Skill Purpose
provision Create/reuse Cosmos DB accounts, App Insights, Azure VMs
setup Install tools, clone repo (branch/PR/commit/tag), build in tmux
run CHURN preset, multi-VM parallel, auto App Insights config
analyze CSV analysis, run comparison, heap/thread dumps, Kusto export
status Resource health, run overview, App Insights verification
  • Skill-creator utility for authoring new skills
  • Reference docs: VM sizing, 20 operation types, presets, thresholds, Kusto schema

Design Decisions

  • Single vs multi-tenant is config only (CLI flags vs tenants.json)
  • CHURN only preset for now (extensible later)
  • Always tmux for remote operations
  • App Insights KQL queries are placeholder

37 files changed | No secrets committed

Annie Liang and others added 2 commits February 27, 2026 11:40
Add benchmark shell scripts for VM provisioning, setup, execution,
monitoring, diagnostics capture, and dashboard generation.

Update BenchmarkConfig, BenchmarkOrchestrator, and TenantWorkloadConfig
to support multi-tenant benchmark orchestration with per-tenant
configuration overrides.

Add .gitignore entries for benchmark artifacts and Copilot skills.
Add test-setup and test-results directory scaffolding with READMEs
and a sample tenants.json template (no real credentials).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Agent routing file dispatches to 5 skills covering the full
benchmark/DR drill lifecycle:

- provision: Cosmos DB accounts, App Insights, Azure VMs
- setup: JDK/Maven install, repo clone, config generation, build
- run: CHURN preset execution, multi-VM parallel, App Insights config
- analyze: CSV metrics, run comparison, heap/thread dumps, Kusto export
- status: resource health, run overview, App Insights verification

Also includes skill-creator utility for authoring new skills.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Annie Liang and others added 2 commits February 27, 2026 14:07
…ate runtime config

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@xinlian12 xinlian12 force-pushed the cosmos-benchmark-agent branch from bf46a2f to c43f5b6 Compare February 27, 2026 22:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant