Your team's Alertmanager is forwarding every alert notification to PagerDuty, but during a database incident, engineers got paged 1,200 times for the same underlying issue. Alerts fired from 50 different services all relating to the same root cause. How do you restructure routing and grouping to prevent alert storms?
Configure route grouping by common labels that represent the failure domain. Use group_by to cluster related alerts: group_by: ['alertname', 'cluster', 'service']. Set group_wait (initial aggregation time, ~10s) and group_interval (regroup period, ~5m). Create a catch-all route with repeat_interval: 4h and a firing route with repeat_interval: 12h for critical incidents. Implement inhibition rules to suppress lower-priority alerts when upstream services are down. For example, suppress http_request_errors for all services if database_connection_failures is firing. Use route matching on severity labels to filter what goes to PagerDuty vs Slack.
Follow-up: How would you implement deduplication if you have multiple Prometheus instances scraping the same targets (federated setup), and how do the dedup_interval in scrape config and Alertmanager grouping interact?
You've set up a complex routing tree with 8 nested routes, each with its own receiver (Slack, PagerDuty, OpsGenie, email). During an incident, a critical alert matched multiple routes and notifications went to all receivers simultaneously. How do you debug and fix routing precedence?
Routes are evaluated top-down and use exact label matching combined with regex. The first matching route wins. Enable -log.level=debug on Alertmanager to see route matching logs. Check that route definitions are ordered correctly—more specific routes (with exact label matchers) should come before generic ones. Use continue: true to allow an alert to match multiple routes. Without continue, only the first matching route fires. Verify grouping labels don't interfere: if a route groups by 'service' but the alert doesn't have that label, all instances of that alert collapse together. Test routing with amtool: amtool alert --alertmanager.url=http://localhost:9093 query 'alertname=~".*"'
Follow-up: What's the difference between route matchers (match/match_re) and inhibition matchers, and how would you use both to eliminate duplicate notifications to on-call?
Your production cluster experiences a rolling node drain during maintenance. Nodes go down and come back up sequentially over 30 minutes, firing and then resolving thousands of alerts repeatedly. Each time a node recovers, all its services re-register and previous alerts fire again. How do you handle this in Alertmanager without changing Prometheus or alert rules?
Use Alertmanager's mute_time_intervals (or Silences API) to temporarily mute known maintenance windows. Create a silence with matchers covering all affected services and a duration covering the maintenance window. Or use route-level muting: define a time interval for maintenance windows and apply it to the root route or specific sub-routes. Set group_wait to a higher value (30-60s) during known maintenance to batch transient alerts. Use repeat_interval to control how often repeated notifications are sent—set to 24h for firing alerts during maintenance so pager fatigue is minimized. Consider using an external tool like Grafana's OnCall or PagerDuty's event routing to deduplicate on PagerDuty's side as well.
Follow-up: How would you automate Silences for scheduled maintenance windows, and what race conditions could occur between Prometheus alerts resolving and Silences expiring?
You're implementing an SLA-aware routing system where high-severity incidents must reach on-call within 5 minutes, medium within 1 hour, and low can be batched into a daily report. Your current Alertmanager config has all alerts going to the same Slack channel. How do you redesign routing to meet SLA expectations?
Structure routes hierarchically by severity label. Root route has continue: false. Create top-level sub-routes for severity=critical (group_by: [alertname, service], group_wait: 0s, group_interval: 5m, repeat_interval: 15m, receiver: pagerduty), severity=warning (group_wait: 5m, repeat_interval: 1h, receiver: slack), and severity=info (group_wait: 1h, repeat_interval: 24h, receiver: daily-digest). Use inhibition to suppress lower severities when critical fires. For critical route, set repeat_interval to 15m to ensure re-notification even if the alert still fires. On the receiver side, use webhook to create PagerDuty incidents for critical, Slack messages for warning, and store info alerts in a database for daily batching.
Follow-up: If a critical alert should bypass all grouping and fire immediately (no group_wait), how does this affect alert fanout and Alertmanager resource usage at scale (10k+ alerts/min)?
Your team wants to implement "on-call rotation" awareness in Alertmanager: if Alice is on call for database incidents and Bob for API incidents, route database alerts to Alice and API alerts to Bob via different email addresses. However, on-call rotations change hourly. How do you keep Alertmanager routing in sync without restarting it?
Alertmanager routing is static and read from config at startup (or reload via SIGHUP). For dynamic routing, use an external webhook receiver that looks up on-call information in real-time. Add a webhook receiver in Alertmanager and call an API that queries PagerDuty's GetOnCall endpoint or your internal CMDB. The webhook can then route to the current on-call engineer's email/Slack/PagerDuty ID. Alternatively, use a sidecar that watches your on-call schedule (Opsgenie, Grafana OnCall) and dynamically rebuilds the Alertmanager config, then triggers a reload. For true dynamic routing without external APIs, use label rewriting in Prometheus before alerts reach Alertmanager—add a owner=team_name label based on scrape metadata, then route in Alertmanager by owner.
Follow-up: What are the failure modes if your webhook receiver is slow or down? How does Alertmanager handle receiver timeouts (default 10s) and how would you implement exponential backoff retry logic?
You're operating a multi-tenant SaaS platform where customer A's alerts should never route to customer B's receivers, and vice versa. You have a single Alertmanager instance. How do you ensure tenant isolation in routing and deduplication?
Use a mandatory customer_id label on all alerts (enforced in Prometheus recording rules or alert rules). Build routes that match specific customer_id values exactly. Each customer gets a route with match: {customer_id: "CUST_123"}. If a route doesn't match any customer_id, that alert is silenced or routed to an error handler. For deduplication, Alertmanager groups by all labels including customer_id, so alerts from different customers never merge. However, a single compromised Prometheus instance could send alerts with wrong customer_id labels. Mitigate by validating customer_id in a webhook receiver and rejecting mismatched pairs. Use external_labels in scrape config to inject customer_id automatically from scrape targets. For strong isolation, consider separate Alertmanager instances per customer (Alertmanager is stateless and lightweight).
Follow-up: What happens to grouping and deduplication if a label value contains special characters or is extremely long? How would you implement label value sanitization without breaking alert matching?
Your organization runs Alertmanager behind a load balancer with 3 replicas. A new rule fires 50k alerts simultaneously. One replica queues them but a second replica hasn't received them yet (state is eventually consistent). How does Alertmanager's design handle this distributed deduplication, and can you get duplicate notifications?
Alertmanager replicas are independent and keep separate memory state (alerts, silences, groups). Prometheus sends all alerts to all replicas via a round-robin or broadcast mechanism. Each replica independently deduplicates and notifies, which means duplicate notifications can occur. The canonical solution is to place a single Alertmanager instance or use Alertmanager's own HA via gossip clustering (--cluster.* flags). With clustering enabled, replicas gossip state and synchronize silences, but not alert state. Each replica still independently notifies. To prevent duplicate notifications to downstream receivers (PagerDuty, Slack), implement deduplication on the receiver side using alert fingerprint or use a single Alertmanager. If load balancing is required for HA, put a stateless proxy in front (e.g., Nginx) that replicates to multiple instances, then deduplicate at the receiver/webhook layer.
Follow-up: How does the Alertmanager --cluster.* gossip protocol work? What happens during a network partition where some replicas can't reach others—how is state reconciled post-partition?
Your alerting SLA requires that each alert notification includes context (current metric value, graph URL, runbook link). You're using Alertmanager's template system to format notifications. However, template rendering is stateless—Alertmanager doesn't have access to Prometheus to fetch current values. How do you inject live context into notifications?
Prometheus embeds query results in alert annotations via the annotations section in alert rules. For example, in your Prometheus alert rule: annotations: { summary: "High latency detected", value: "{{ $value }}", dashboard: "https://grafana.com/d/abc" }. The $value variable is evaluated at alert evaluation time and includes the exact metric value. Pass these annotations to Alertmanager; they're included in the alert object. In Alertmanager templates, reference them: {{ .GroupLabels.alertname }} - {{ .Alerts[0].Annotations.value }}. For dynamic links, add templating: runbook: "https://runbooks.internal/{{ .GroupLabels.service }}/high-latency". To add live context at notification time (not at alert-fire time), use a webhook receiver that calls Prometheus's query API to fetch latest values and format enriched notifications. This adds latency but ensures real-time context.
Follow-up: What's the maximum size of annotations before Alertmanager rejects an alert? How do you handle cases where template rendering fails (e.g., missing labels)?
You've configured inhibition rules to suppress low-severity alerts when high-severity alerts fire (e.g., suppress pod_cpu_high if node_cpu_high fires). But your team reports that sometimes they don't see expected alerts, and silences are being applied inconsistently. How do you debug inhibition and silencing logic?
Inhibition and silences are independent. Silences are explicit manual suppression (via Alertmanager API or UI). Inhibition is rule-based automatic suppression when target_matchers match existing alerts. Enable debug logging: --log.level=debug. Check Alertmanager's web UI (/api/v1/alerts) to see which alerts are active, silenced, or inhibited. The response includes 'status': { 'state': 'suppressed', 'silencedBy': [...], 'inhibitedBy': [...] }. Verify inhibition rule logic: a source alert must match source_matchers, and a target alert must match target_matchers AND have labels that equal the source alert's labels for the equal keys. For example, inhibition { source_match: {severity: critical}, target_match_re: {severity: 'warning|info'}, equal: ['instance', 'alertname'] } suppresses warning/info alerts only if a critical alert exists for the exact same instance and alertname. Use amtool to test: amtool silence query 'severity=~"warning"' or amtool inhibition query.
Follow-up: If an inhibition rule has zero equal keys, what happens? Can this cause unexpected alert suppression at scale?