Skip to content

Python: Foundry Evals integration for Python#4750

Draft
alliscode wants to merge 19 commits intomicrosoft:mainfrom
alliscode:af-foundry-evals-python
Draft

Python: Foundry Evals integration for Python#4750
alliscode wants to merge 19 commits intomicrosoft:mainfrom
alliscode:af-foundry-evals-python

Conversation

@alliscode
Copy link
Member

@alliscode alliscode commented Mar 17, 2026

Add evaluation framework with local and Foundry-hosted evaluator support:

  • EvalItem/EvalResult core types with conversation splitting strategies
  • @evaluator decorator for defining custom evaluation functions
  • LocalEvaluator for running evaluations locally
  • FoundryEvals provider for Azure AI Foundry hosted evaluations
  • evaluate_agent() orchestration with expected values support
  • evaluate_workflow() for multi-agent workflow evaluation
  • Comprehensive test suite and evaluation samples

Contribution Checklist

  • The code builds clean without any errors or warnings
  • The PR follows the Contribution Guidelines
  • All unit tests pass, and I have added new tests where possible
  • Is this a breaking change? If yes, add "[BREAKING]" prefix to the title of the PR.

@markwallace-microsoft markwallace-microsoft added documentation Improvements or additions to documentation python labels Mar 17, 2026
@github-actions github-actions bot changed the title Foundry Evals integration for Python Python: Foundry Evals integration for Python Mar 17, 2026
@alliscode alliscode force-pushed the af-foundry-evals-python branch from a0edd5f to fe9e621 Compare March 17, 2026 21:21
eavanvalkenburg
eavanvalkenburg reviewed Mar 18, 2026
View reviewed changes
@alliscode alliscode force-pushed the af-foundry-evals-python branch 6 times, most recently from 15d8640 to aad92ac Compare March 19, 2026 20:41
@markwallace-microsoft
Copy link
Member

markwallace-microsoft commented Mar 19, 2026 •
edited
Loading

Python Test Coverage

Python Test Coverage Report •
FileStmtsMissCoverMissing
packages/azure-ai/agent_framework_azure_ai
   _foundry_evals.py251697%271, 322–323, 443, 628, 667
packages/core/agent_framework
   _evaluation.py6417388%220, 235, 450, 452, 556, 559, 638–640, 645, 682–685, 740–741, 744, 750–752, 756, 789–791, 843, 876, 888–890, 895, 919–924, 1011, 1089–1090, 1092–1096, 1102, 1140, 1469, 1471, 1479, 1489, 1493, 1538, 1556–1557, 1661, 1667, 1682, 1686–1688, 1718, 1724–1728, 1762, 1810–1811, 1813, 1838–1839, 1844
TOTAL28191331588% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
5560 21 💤 0 ❌ 0 🔥 1m 26s ⏱️

@alliscode alliscode force-pushed the af-foundry-evals-python branch 2 times, most recently from af0ccf6 to 45527ee Compare March 20, 2026 21:24
alliscode and others added 3 commits March 23, 2026 08:48
Foundry Evals integration for Python
db05e3a 
Merged and refactored eval module per Eduard's PR review:

- Merge _eval.py + _local_eval.py into single _evaluation.py
- Convert EvalItem from dataclass to regular class
- Rename to_dict() to to_eval_data()
- Convert _AgentEvalData to TypedDict
- Simplify check system: unified async pattern with isawaitable
- Parallelize checks and evaluators with asyncio.gather
- Add all/any mode to tool_called_check
- Fix bool(passed) truthy bug in _coerce_result
- Remove deprecated function_evaluator/async_function_evaluator aliases
- Remove _MinimalAgent, tighten evaluate_agent signature
- Set self.name in __init__ (LocalEvaluator, FoundryEvals)
- Limit FoundryEvals to AsyncOpenAI only
- Type project_client as AIProjectClient
- Remove NotImplementedError continuous eval code
- Add evaluation samples in 02-agents/ and 03-workflows/
- Update all imports and tests (167 passing)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
fix: resolve mypy redundant-cast errors while keeping pyright happy
39415c2 
Use cast(list[Any], x) with type: ignore[redundant-cast] comments to
satisfy both mypy (which considers casting Any redundant) and pyright
strict mode (which needs explicit casts to narrow Unknown types).

Also fix evaluator decorator check_name type annotation to be
explicitly str, resolving mypy str|Any|None mismatch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
fix: CI failures — pyupgrade, evaluator overloads, sample API, reset …
5dccdc2 
…attr

- Apply pyupgrade: Sequence from collections.abc, remove forward-ref quotes
- Add @overload signatures to evaluator() for proper @evaluator usage
- Fix evaluate_workflow sample to use WorkflowBuilder(start_executor=) API
- Fix _workflow.py executor.reset() to use getattr pattern for pyright
- Remove unused EvalResults forward-ref string in default_factory lambda

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@alliscode alliscode force-pushed the af-foundry-evals-python branch from 5c6ab9b to 5dccdc2 Compare March 23, 2026 15:48
alliscode and others added 10 commits March 23, 2026 09:21
fix: skip gRPC-dependent observability test
c4c6632 
The test_configure_otel_providers_with_env_file_and_vs_code_port test
triggers gRPC OTLP exporter creation, but the grpc dependency is
optional and not installed by default. Add skipif decorator matching
the pattern used by all other gRPC exporter tests in the same file.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
fix: add nosec B101 for bandit assert check
b4f23a0 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
style: align eval samples with repo conventions
959e51d 
- Move module docstrings before imports (after copyright header)
- Add -> None return type to all main() and helper functions
- Fix line-too-long in multiturn sample conversation data
- Add Workflow import for typed return in all_patterns_sample

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address PR review feedback: async fixes, sample bugs, deprecation war…
e740f43 
…nings

- Simplify _ensure_async_result to direct await (async-only clients)
- Replace get_event_loop() with get_running_loop()
- Narrow _fetch_output_items exception handling to specific types
- Add warning log when _filter_tool_evaluators falls back to defaults
- Add DeprecationWarning to options alias in Agent.__init__
- Add DeprecationWarning to evaluate_response()
- Rename raw key to _raw_arguments in convert_message fallback
- Fix evaluate_agent_sample.py: replace evals.select() with FoundryEvals()
- Fix evaluate_multiturn_sample.py: use Message/Content/FunctionTool types
- Fix evaluate_workflow_sample.py: replace evals.select() with FoundryEvals()
- Update test mocks to use AsyncMock for awaited API calls

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add test coverage for review feedback items
564b8a2 
- Add num_repetitions=2 positive test verifying 2×items and 4 agent calls
- Add _poll_eval_run tests: timeout, failed, and canceled paths
- Add evaluate_traces tests: validation error, response_ids path, trace_ids path
- Add evaluate_foundry_target happy-path test with target/query verification

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix ruff ISC004 lint error and apply formatter
aca368f 
- Wrap implicit string concatenation in parens in evaluate_multiturn_sample.py
- Apply ruff formatter to 6 other files with minor formatting drift

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove core type changes (extracted to fix/workflow-stale-session bra…
375dcba 
…nch)

Reverts changes to _agents.py, _agent_executor.py, and _workflow.py
back to upstream/main. These fixes are now in a separate PR.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address PR review round 2: bugs, tests, and architecture
39e7dfb 
Code fixes:
- Fix _normalize_queries inverted condition (single query now replicates
  to match expected_count)
- Fix substring match bug: 'end' in 'backend' matched; use exact set
  lookup for executor ID filtering
- Fix used_available_tools sample: tool_definitions→tools param, use
  FunctionTool attribute access instead of dict .get()
- Add None-check in _resolve_openai_client for misconfigured project
- Add Returns section to evaluate_workflow docstring
- Cache inspect.signature in @evaluator wrapper (avoid per-item reflection)

Architecture:
- Extract _evaluate_via_responses as module-level helper; evaluate_traces
  now calls it directly instead of creating a FoundryEvals instance
- Move Foundry-specific typed-content conversion out of core to_eval_data;
  core now returns plain role/content dicts, FoundryEvals applies
  AgentEvalConverter in _evaluate_via_dataset

Tests:
- evaluate_response() deprecation warning emission and delegation
- num_repetitions > 1 with expected_output and expected_tool_calls
- Mock output_items.list in test_evaluate_calls_evals_api
- Update to_eval_data assertions for plain-dict format
- Unknown param error now raised at @evaluator decoration time

Skipped (separate PR): executor reset loop, xfail removal, options alias

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix CI: revert test_full_conversation, fix pyright errors
2e002f0 
- Revert test_full_conversation.py to upstream/main (the session
  preservation test was incorrectly changed to assert clearing)
- Fix pyright reportUnnecessaryComparison on get_openai_client() None
  check by adding ignore comment
- Fix pyright reportPrivateUsage: add public EvalItem.split_messages()
  method and use it in FoundryEvals._evaluate_via_dataset instead of
  accessing private _split_conversation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address PR review round 3: reliability, test gaps, cleanup
753eee2 
- Add try/except guard for non-numeric score in _coerce_result
- Add poll_interval minimum bound (0.1s) to prevent tight loops
- Add runtime async client check in _resolve_openai_client
- Remove _ensure_async_result wrapper (10 call sites → direct await)
- Better error message when queries provided without agent
- Import-time asserts for evaluator set consistency
- Remove 28 redundant @pytest.mark.asyncio decorators
- Add doc note about _raw_arguments sensitive data
- Tests: tool_called_check mode=any, _normalize_queries branches,
  _extract_result_counts paths, _extract_per_evaluator, bare check
  via evaluate_agent, output_items assertion, modulo wrapping,
  async client check, queries-without-agent error

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@alliscode alliscode force-pushed the af-foundry-evals-python branch from 8c46ebe to 753eee2 Compare March 24, 2026 17:34
Fix CI: ruff S101 assert, pyright and mypy arg-type errors
1bf5f7d 
- Replace module-level assert with if/raise for evaluator set
  consistency checks (ruff S101 disallows bare assert)
- Add type: ignore[arg-type] and pyright: ignore[reportArgumentType]
  on OpenAI SDK evals API calls that pass dicts where typed params
  are expected (SDK accepts dicts at runtime)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address PR review round 4: bugs, reliability, test fixes
d52c85e 
- Fix all_passed ignoring parent result_counts when sub_results present
- Fix _extract_tool_calls: parse string arguments via json.loads before
  falling back to None (real LLM responses use string arguments)
- Sanitize _raw_arguments to '[unparseable]' to avoid leaking sensitive
  tool-call data to external evaluation services
- Add NOTE comment on to_eval_data message serialization dropping
  non-text content (tool calls, results)
- Eliminate double conversation split in _evaluate_via_dataset: build
  JSONL dicts directly from split_messages + AgentEvalConverter
- Raise poll_interval floor from 0.1s to 1.0s to prevent rate-limit
  exhaustion
- Fix MagicMock(name=...) bug in test: sets display name not .name attr
- Fix mock_output_item.sample: use MagicMock object instead of dict so
  _fetch_output_items exercises error/usage/input/output extraction

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@alliscode alliscode force-pushed the af-foundry-evals-python branch from 901ea59 to d52c85e Compare March 24, 2026 20:03
alliscode and others added 4 commits March 24, 2026 14:15
Address PR review round 5: reliability, docs, test coverage
12b48a2 
Code fixes:
- Move import-time RuntimeError checks to unit tests (avoids breaking
  imports for all users on developer set-drift mistake)
- _filter_tool_evaluators now raises ValueError when all evaluators
  require tools but no items have tools (was silently substituting)
- Add poll_interval upper bound (60s) to prevent single-iteration sleep
- Log exc_info=True in _fetch_output_items for debugging API changes
- Fix evaluate() docstring: remove claim about Responses API optimization
- Validate target dict has 'type' key in evaluate_foundry_target
- Document to_eval_data() limitation: non-text content is omitted

Tests:
- TestEvaluatorSetConsistency: verify _AGENT/_TOOL subsets of _BUILTIN
- TestEvaluateTracesAgentId: agent_id-only path with lookback_hours
- TestFilterToolEvaluatorsRaises: ValueError on all-tool no-items
- TestEvaluateFoundryTargetValidation: target without 'type' key
- Assert items==[] on failed/canceled poll results
- Mock output_items.list in response_ids test for full flow
- TestAllPassedSubResults: result_counts=None + sub_results delegation
  and parent failures override sub_results
- TestBuildOverallItemEmpty: empty workflow outputs returns None

Skipped r5-07 (_raw_arguments length hint): marginal debugging value,
could leak content size information.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix error message: evaluate_responses() → evaluate_traces(response_id…
5097a6b 
…s=...)

The referenced function doesn't exist; the correct API is
evaluate_traces(response_ids=...) from the azure-ai package.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove dead to_eval_data() method, fix docstring claims
3dd465f 
- Remove to_eval_data() from EvalItem (dead code after r4-05 JSONL refactor)
- Migrate 15 tests from to_eval_data() to split_messages()
- Update sample to use split_messages() + Message properties
- Remove unimplemented Responses API optimization docstring claim
- Update split_messages() docstring to not reference removed method

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reduce default eval timeout from 600s to 180s (3 minutes)
79e1957 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eavanvalkenburg eavanvalkenburg eavanvalkenburg left review comments

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@alliscode @markwallace-microsoft @eavanvalkenburg