Unified Alerting Architecture

Implement Grafana's Unified Alerting (UA) as the single source of truth for all alert rules. Migrate Prometheus alert rules using a migration tool that converts PrometheusRuleSet YAML to Grafana alert rule definitions, stored in Grafana's database. Implement a three-tier routing strategy: (1) Alert evaluation—UA evaluates rules across all datasources (Prometheus, Loki, CloudWatch), generating alert instances with consistent metadata (team, severity, service). (2) Alert deduplication—assign fingerprints to alerts based on labels; group identical alerts firing across multiple instances into single incidents using group labels. (3) Contact point routing—define routing trees that match alert labels to contact points (PagerDuty for P1, Slack for P2, email for P3). Use Grafana's notification policies to apply conditional routing based on team ownership, service tier, and severity. For high-volume alerts, implement aggregation: group related alerts (same service, same error type) into single notifications at 5-minute intervals. Store routing rules in Git, provisioned via API at startup. Implement alert silencing API for temporary suppression during deployments or known issues, with audit trails.

Implement multi-tier query optimization: (1) Query caching—store frequently evaluated metric queries in Redis with 30-second TTL, avoiding repeated Prometheus roundtrips. (2) Query splitting—break complex queries into simpler subqueries executed in parallel, then combine results. (3) Data pre-aggregation—push expensive aggregations (percentiles, histograms) into Prometheus recording rules executed at ingest time, reducing alert query latency. (4) Datasource load balancing—route alert queries across multiple Prometheus instances or use remote read replicas to distribute load. Implement query timeout enforcement—set a 10-second hard limit on alert rule queries; if exceeded, trigger a fallback alert ("Query timeout on alert X"). Use Grafana's alert rule profiling to identify slow queries, then optimize those specific rules. For SLA compliance, implement a dual-query strategy: run fast heuristic queries (e.g., rate() without histograms) for quick alerting, then run detailed queries for context. Set up monitoring of alert evaluation latency using Prometheus metrics, with alerts on latencies exceeding SLA targets.

Implement intelligent alert grouping and root cause analysis: (1) Group by tag hierarchy—define topology labels (cluster, region, service, instance) and group alerts sharing parent labels (e.g., all alerts from us-east-1 cluster). (2) Temporal correlation—alerts firing within 5 seconds of each other are likely related; group them into a single incident. (3) Causal inference—use service dependency maps (from Tempo/Jaeger) to infer that a database error likely caused downstream service alerts; suppress symptom alerts, surfacing only root cause. (4) Alert suppression rules—define that if alert X fires, suppress alerts Y and Z (dependencies). Use Grafana's alert instances to compute a dependency DAG at evaluation time, propagating suppression. Implement a "deduplicate and fold" algorithm: when multiple related alerts fire, consolidate into a single PagerDuty incident with details linking to all constituent alerts. For high-cardinality services, implement bucketing: group alerts by (service, error_type, region) instead of individual instances, reducing fan-out. Display a dependency graph in Grafana/PagerDuty showing root cause at the top, cascading dependencies below. Use machine learning (if available) to learn common alert correlation patterns from historical incidents.

Implement tiered alert sensitivity based on severity and context: (1) Severity-based for_duration—P0 alerts (customer-facing outages) use 0-30 second for_duration, P1 use 1-2 minutes, P2 use 5+ minutes. This catches catastrophic failures immediately while reducing noise for gradual anomalies. (2) Dynamic thresholds—use adaptive baselines (e.g., traffic 2x normal) to adjust alert sensitivity in real-time. During a deployment window, increase for_duration temporarily to reduce false positives. (3) Context-aware suppression—disable for_duration entirely if previous deployments in the same region just finished (high-risk window), or if the deployment is flagged as high-risk. (4) Canary alerts—before triggering production alert, fire a "canary" alert to a subset of on-call (Slack), letting them acknowledge if it's expected; if 3+ canaries fire, auto-escalate to PagerDuty. Implement a "fast-track" alert rule that bypasses for_duration for clearly anomalous conditions (error rate > 50% instantly triggers). Store alert rule configs in Git with version history and CI/CD testing—run simulations of past incidents against rule changes to validate they would have triggered appropriately.

Export all Grafana alert rules to Git-tracked YAML/JSON files using Grafana's HTTP API (GET /api/v1/rules). Set up a CI/CD pipeline that provisions alert rules from Git files on startup using Grafana's provisioning system. Treat alert rules like infrastructure-as-code: all changes go through Git PRs, requiring peer review before merging. Implement a pre-commit hook that validates alert rule syntax (valid PromQL, valid datasources, sensible thresholds). Set up a periodic sync job (every 5 minutes) that compares Grafana's actual rules against Git source; if a rule is deleted in Grafana but exists in Git, auto-restore it and alert admins of the drift. Implement soft deletes: instead of deleting, mark rules as "deprecated: true" in Git, preventing accidental removal. Archive all deleted rules in a Git history branch for auditability. For UI safety, add confirmation dialogs and require RBAC permissions to delete alert rules; log all deletions with user context. Implement a "rule restore" API endpoint that can quickly restore any historical version of an alert rule from Git.

Implement a multi-layer audit system: (1) Git-based audit—all alert rules stored in Git; every change is a commit with author, timestamp, and change description. Use Git blame to track rule evolution. (2) Grafana API logging—hook Grafana's alert provisioning API to log all changes to an external audit log (Loki, CloudWatch). Capture: user, timestamp, rule ID, old config, new config, change reason. (3) Change approval workflow—integrate alert rule Git PRs with change management system (Jira, ServiceNow), requiring explicit approval before merge. (4) Incident correlation—when an incident is created in PagerDuty/ServiceNow, query your audit log to identify which alert rules fired and who authored them. (5) Regression analysis—track which alert rules led to incident resolutions; build analytics on alert rule effectiveness. Implement a dashboard showing: total alert rule changes by team/week, approval SLA compliance, mean time from alert creation to first incident detection, and false positive rates per team. For compliance exports, generate monthly audit reports with snapshots of all alert rules, approval traces, and incident correlations. Store audit logs immutably (Append-Only Logs in S3) with long retention (7+ years) for compliance.

Implement a multi-stage alert rule testing pipeline: (1) Syntax validation—use PromQL linter (promtool) to validate queries are syntactically correct and reference valid metrics. (2) Integration testing—run alert rules against historical metrics (replay) from staging environment, verifying they fired at expected times and didn't fire during normal operation. (3) Threshold tuning—use statistics on past incidents to calibrate thresholds: extract metric values during known-bad events, set thresholds to trigger during those events but not during normal operation. (4) Dry-run deployment—deploy rule to staging Grafana with same datasources as prod; let it evaluate for 24-48 hours, collecting fire events without sending notifications. Analyze dry-run results for false positives. (5) Canary firing—deploy to prod with PagerDuty integration but route to a canary contact point (Slack channel) for 1 week before routing to primary on-call. This lets teams tune without impacting on-call SLA. Implement a rule scorecard showing: incidents the rule should have caught but didn't (false negatives), false positive rate, mean time to notification. Store rule configs with baseline thresholds from testing; document how thresholds were derived. Version alert rules in Git with change logs explaining threshold rationale.

Implement an abstraction layer for alert metrics: define logical metric names (e.g., "http_request_errors_rate") that map to physical datasource-specific queries (Prometheus: rate(http_requests_total{status=~"5.."}[5m]), CloudWatch: m1 / m2 where m1 is 5xx errors, m2 is total). Store the mapping in a configuration file (YAML) versioned in Git. Alert rules reference logical metric names; the evaluation engine translates to datasource-specific queries at runtime. Implement datasource health checks: if primary datasource (Prometheus) is unavailable, auto-fail-over to secondary (CloudWatch) using cached translation mappings. For migration, set up a parallel evaluation period: run alert rules against both old and new datasources simultaneously, alerting if results diverge. Once you've validated equivalence, switch to new datasource. Implement versioning for metric definitions—if query translation changes, old alert rule versions continue using old translations until explicitly updated. Document metric translation logic in Git with comments explaining why queries differ between datasources. Create a datasource migration runbook: checklist of alert rules to validate, testing procedures, and rollback steps.