Observability operations guide
Operating ownership
Every deployed package instance requires an accountable platform owner, a workload owner, a FinOps contact, and a notification action group. The Owner tag identifies the accountable party; alert definitions additionally identify the response runbook. An alert without an owner, severity, suppression plan, and runbook must remain disabled.
Daily and weekly operating view
| Cadence | Review | Evidence | Required action |
|---|---|---|---|
| Daily | Budget threshold, forecast, anomaly, and daily-cap signal | Cost Management and alert history | Investigate variance; do not raise a cap before identifying the source |
| Daily | Failed deployments, deletes, Service Health, and Resource Health | Activity Log and active alerts | Triage owner, scope, and customer impact |
| Weekly | Inventory, missing tags, temporary resources, and expiry | Resource Graph query library | Assign owner, correct tags, extend lifecycle, or schedule removal |
| Weekly | Alert noise and failed notifications | Alert history and action-group checks | Tune scope or suppression; never silence an alert without a replacement control |
| Monthly | Cost allocation, retention, diagnostics volume, and collection value | Cost analysis and workspace usage | Keep, reduce, or remove paid collection based on demonstrated use |
Severity and response policy
| Severity | Use | Response expectation |
|---|---|---|
| Sev 0 | Subscription access loss, widespread Azure service impact, or imminent budget-risk decision | Immediate owner engagement and incident record |
| Sev 1 | Workload unavailable, failed critical deployment, or sustained service error | Triage immediately; update stakeholders and invoke workload runbook |
| Sev 2 | Degraded availability, throttling, resource health change, or forecast risk | Investigate during active support period; create follow-up work item |
| Sev 3 | Tag drift, lifecycle expiry approaching, noncritical anomaly, or advisory | Resolve through weekly operations backlog |
Cost controls
- A budget is an alerting control, not a spending stop. Use actual and, where approved, forecast notifications with values supplied in a private overlay.
- Treat Log Analytics daily caps as a safety circuit. A cap can interrupt data collection and does not guarantee a final bill. Investigate top sources before changing it.
- Start with the shortest retention that supports the operating question. Increase only after a documented retention need and cost review.
- Enable diagnostic categories one at a time. Do not use catch-all log categories.
- Require a data-volume estimate for Application Insights, tracing, availability tests, managed Prometheus, scheduled query alerts, and exports.
- On a shared or credit-backed subscription, keep lower budget notifications in the private overlay. The public package does not prescribe a currency amount.
Tag and lifecycle governance
The query library identifies missing Owner, Project, Environment, CostCenter, ManagedBy, and Lifecycle values. For temporary or sandbox resources, it also identifies missing, invalid, or expired ExpiresOn values. Adopt policy in two steps:
- Assign policies in audit mode, with explicit scope and exclusions, and resolve existing drift.
- Switch to enforce mode only after the owner has accepted the impact and a remediation process exists.
Alert maintenance
Before enabling an alert, verify its scope, metric namespace, dimensions, threshold, evaluation frequency, window, severity, action group, and runbook link. For planned maintenance, use a parameterized alert processing rule or disable the individual definition through the private overlay, then record when it will be restored.
Diagnostic and tracing approval
The owner must record the operational question, category or signal, classification, whether prompts or outputs could be present, retention, sampling, workspace access, estimated daily volume, monthly cost owner, and removal date. If any item is unknown, do not enable the module. This is particularly important for Foundry diagnostics and Application Insights tracing.
Change and rollback
Changes follow the same Plan, Provision, Configure, Validate lifecycle. Configuration rolls back by reverting the parameter or definition file and deploying the reviewed change. Resource deletion requires a separate approved change, because it can remove alerts, historical telemetry, or a dashboard reference. Before deletion, export the dashboard and record the expected loss of retained data.