feat: improve app insights telemetry

[Shreyas-Microsoft]

Purpose

This pull request introduces robust Application Insights and OpenTelemetry integration for the backend API, providing production-grade observability while minimizing noise and ingestion costs. It adds environment-driven configuration for telemetry, custom span processors to filter noisy spans (such as per-chunk ASGI responses and Cosmos DB dependencies), and a lightweight helper for emitting structured business events. Logging verbosity for known noisy packages is also clamped to keep logs actionable. The infrastructure Bicep templates are updated to avoid duplicate log ingestion.

Observability and Telemetry Integration:

• Added Application Insights and OpenTelemetry support, configurable via the APPLICATIONINSIGHTS_CONNECTION_STRING environment variable. This includes wiring up Azure Monitor and FastAPI instrumentation, with exclusion of health/startup probe routes from telemetry to avoid flooding. [1] [2] [3] [4] [5]
• Introduced libs/logging/event_utils.py with the track_event_if_configured helper for emitting custom App Insights events, which is safe to call even when telemetry is not configured. [1] [2] [3]

Noise Suppression and Filtering:

• Added custom OpenTelemetry SpanProcessor implementations (DropASGIResponseBodySpanProcessor, DropCosmosDependencySpanProcessor) in libs/logging/span_filters.py to filter out high-volume, low-value spans before export, reducing cost and improving signal-to-noise. [1] [2] [3]
• Hard-clamped logging levels to WARNING for known noisy packages (e.g., Azure SDK internals, Cosmos diagnostics), regardless of operator configuration, to prevent log flooding in Application Insights. [1] [2]

Infrastructure Improvements:

• Updated Bicep modules (infra/main.bicep, infra/main_custom.bicep) to document and prevent duplicate log ingestion by removing redundant diagnosticSettings when Application Insights is already wired to Log Analytics via workspaceResourceId. [1] [2]

These changes collectively ensure that the backend API emits actionable telemetry and business events to Application Insights with minimal configuration, while keeping the signal clean and ingestion costs predictable.

Does this introduce a breaking change?

• Yes
• No

Golden Path Validation

• I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

• I have validated the deployment process successfully and all services are running as expected with this change.

What to Check

Verify that the following are valid

• ...

Other Information

[github-actions]

Coverage Report •

File	Stmts	Miss	Cover	Missing
__init__.py	4	4	0%	1, 3–5
application.py	60	60	0%	1–3, 5–8, 12–16, 21–22, 25–26, 28, 34, 37, 43–44, 46–47, 49, 57, 63, 65, 78, 80–82, 87, 89, 101, 104–105, 109, 111–112, 114, 124, 127–128, 133, 140, 144–146, 148, 151, 155–156, 161, 170, 177–178, 180, 187, 235–236
libs/base
application_base.py	49	13	73%	37, 41, 62, 71, 74, 77–78, 83, 92, 100–103
libs/logging
__init__.py	0	0	100%
event_utils.py	26	0	100%
span_filters.py	22	22	0%	30, 32–33, 35–36, 38, 41, 54, 86, 99–101, 108–117
routers
router_debug.py	25	25	0%	1–6, 8, 15–16, 18–19, 21, 23, 39–42, 48–52, 55–57
router_files.py	88	88	0%	1–2, 4, 11, 14–26, 29, 31–40, 43, 45–52, 55, 62–63, 65, 75–76, 81–82, 84–86, 88–89, 91–93, 95–96, 98–100, 102, 104–107, 109–110, 116, 120, 127–128, 130, 134–136, 138–139, 143, 150, 160, 166–168, 177–181, 189–190
router_process.py	427	427	0%	1–5, 7–20, 28, 33, 39–48, 51, 53–60, 63, 70–79, 82–86, 88–90, 92–93, 95, 97–99, 101–102, 105–108, 116–120, 124–125, 128–129, 131, 133–134, 137, 140, 142–144, 147–149, 157–158, 161–162, 164, 166–167, 170, 173, 175–177, 180–182, 190–191, 194–195, 206–207, 209–210, 213–214, 217–218, 221–222, 224–225, 228, 230–232, 235–236, 242, 244, 247, 249, 253, 258, 260, 265, 280–281, 283, 290, 298–301, 309–310, 313–314, 325–326, 328–329, 332–333, 336–337, 340–341, 343–344, 347, 350, 352, 355, 357, 362, 377–378, 380, 387, 395, 397–398, 406–407, 411–413, 422–423, 426–427, 437–438, 440–441, 446–447, 450–451, 453–454, 457, 460, 462, 467, 475–476, 478, 481, 485, 487–489, 497–498, 501–502, 511–512, 514–515, 518–519, 522–523, 525–526, 529, 535, 543, 545, 548–549, 551–552, 555, 561–563, 571–572, 577–578, 586–587, 589–590, 595–596, 598–599, 602, 605, 607–608, 613–615, 617, 619, 621, 625, 631, 637, 645–647, 656–660, 668–669, 674–675, 683–684, 686–687, 692–693, 695–696, 699, 702, 705, 714, 718, 721, 726, 728–730, 739–743, 751–752, 757–758, 767–768, 770–771, 776–777, 779–780, 783, 786, 790, 794–795, 800, 802–803, 811–812, 816–817, 825–826, 830–832, 842–846, 855–856, 861–862, 872–873, 875–877, 880–881, 883–884, 887–889, 892–894, 897–898, 902–903, 908, 910–912, 917–918, 921, 926, 928, 932, 940, 948–950, 957–958, 962–964, 971–972, 976–977, 986–990, 998–999, 1004–1005, 1013–1014, 1016–1018, 1021–1022, 1024–1025, 1028–1030, 1033–1035, 1038–1039, 1043–1044, 1048, 1050–1052, 1057–1058, 1061, 1066, 1068, 1071, 1073–1075, 1079–1080, 1084–1086, 1090–1091, 1095–1096, 1105–1109, 1117–1118
TOTAL	2964	2675	9%

Tests	Skipped	Failures	Errors	Time
51	0 💤	0 ❌	0 🔥	12.956s ⏱️