Handlers and Notification Chains

Handlers execute only once per play by default—they deduplicate notifications automatically. The issue is likely handler dependency ordering or not waiting for handlers. Add `meta: flush_handlers` between task groups to control handler timing. Ensure all config modifications notify the handler: use `notify: 'restart nginx'` consistently. Set `listen` directive in handler to group related handlers: `listen: 'restart webserver'` and notify this group. Use explicit handler ordering with `flush_handlers` at critical points. For distributed deployments, implement serial strategy with handler execution: `serial: 50` executes handlers after each batch. If handlers still restart multiple times, check for nested playbooks or role handlers that might not be deduplicating correctly. Use `--start-at-task` debug mode to verify handler execution flow. Implement monitoring to alert on unexpected restarts.

Use `force_handlers: true` in ansible.cfg or playbook to execute handlers even on task failures or interruptions. This is critical for cleanup operations: logging, metrics collection, cleanup on failure. However, handlers after the failing task won't execute—place handlers early or use a specific handler ordering strategy. Implement rescue blocks with handlers: `tasks/rescue/always` ensures handlers can be called explicitly in rescue blocks. For critical cleanup, don't rely on handlers—use explicit cleanup tasks in `always` blocks: `tasks: [...], rescue: [cleanup], always: [final_cleanup]`. Create a dedicated cleanup handler and call it explicitly with `meta: flush_handlers` before allowing failures. Test handler execution under failure scenarios: use `block/rescue/always` to verify cleanup runs. Implement monitoring that alerts if cleanup handlers fail. For critical systems, implement dual approach: handlers for normal execution + explicit cleanup tasks for robustness.

Handlers are play-scoped, so cross-play notification requires explicit orchestration. Implement three patterns: 1) Use `meta: flush_handlers` at play end, then subsequent plays check changed status via register variables. 2) Create a shared fact in handlers to signal dependent plays: `set_fact: needs_restart=true`, then dependent play checks this fact in `when` condition. 3) Use callback plugins to track notifications across plays and trigger playbook execution via Tower API. For role chains, use role dependencies instead: structure roles so dependent role explicitly calls tasks from upstream role. Implement a state machine using task conditionals: `when: hostvars[inventory_hostname]['needs_service_restart']` to trigger dependent tasks in next play. Use explicit play ordering with `pre_tasks/tasks/post_tasks` to ensure proper sequencing within single play. For complex orchestration, implement Ansible Tower workflows that chain multiple playbooks and handle inter-playbook signaling.

Handlers fail silently by default—failures don't stop playbook execution. Implement explicit error handling by registering handler results and checking them: `register: handler_result` in handler, then use post_tasks to validate handler success. Set `any_errors_fatal: true` in playbook to stop on handler failures. Implement notification of handler failures using callback plugins or explicit post-task checks. Create a post-play task: `fail: msg='Handler sync failed'` if handler result indicates failure. For critical handlers, implement retry logic: wrap handler in a retry block with exponential backoff. Monitor handler execution with custom callback plugins that log handler start/end and status. Implement application-level monitoring: if the handler syncs to central repo, verify data arrived with timestamp checking. Use callback plugin to aggregate handler statistics and alert on failures. Implement replay capability: maintain handler execution logs that can be replayed for failed handlers.

Handlers don't natively support async, but implement asynchronous behavior through separate mechanisms: 1) Create an external service that monitors for handler signals and executes async, 2) Use callback plugins to trigger async jobs instead of blocking handler execution, 3) Implement handler as a simple task that queues a job asynchronously. Pattern: Instead of handler performing drain, have handler submit job to separate queue/service, then playbook continues. Implementation: Use callback plugin to queue to Redis/SQS, background service processes the queue. For simpler cases, use `async: 900` and `poll: 0` within handler to fire-and-forget, but this requires careful orchestration. Better approach: Don't use handlers for long-running operations—use explicit tasks with async and let playbook continue. Example: Create pre-restart task with `async: 900, poll: 0` to drain connections without blocking. Implement callback plugin to monitor async job completion and alert on failure.

Implement batched handler execution. Use handler that runs on control node (not on each managed host) to trigger batch sync operations. Use playbook-level `serial` setting separately from handlers: serial applies to tasks, handlers execute on changed hosts independently. Create handler that invokes bulk API call instead of per-host operations: instead of restarting service on each server sequentially, invoke orchestration API to restart batch of 50 servers. Use handler that queues async jobs for bulk processing. Implement handler pattern: `run_once: true` combined with `serial` for batch operations. Alternative: Use callback plugin to batch notifications and execute handler for groups of hosts. Use connection plugin to implement parallel remote execution within handlers. Monitor handler latency and implement alerting if handler execution exceeds expected duration. Test parallelization in staging to verify services can handle concurrent updates.

Implement retry logic in handlers with exponential backoff. Add task-level retries in handler: `retries: 3` with `delay: 10` for initial network-transient failures. For API calls, implement circuit breaker: if API fails consistently, stop attempting and queue for later retry. Use handler that logs failures to a persistent queue (Redis, database) instead of failing immediately. Implement external service that processes queued handler notifications with full retry/circuit-breaker logic. For critical handlers, implement alternative notification methods: if primary API fails, fallback to secondary method (webhook, message queue). Implement monitoring of handler API failures and alert on thresholds. Add `changed_when: false` strategically to prevent re-notification loops. Implement idempotent handlers that safely retry without side effects. Use callback plugins to log all handler notifications and their outcomes for audit/troubleshooting.

Implement callback plugin to intercept handler notifications and log before/after execution. Callback hooks: `v2_runner_on_start` for task start, `v2_runner_on_ok` for success, etc. Log to centralized system (ELK, Splunk) with full context: playbook, play, task name, handler name, status, timestamp, host, user. Create wrapper handler that logs details, then calls actual handler. Use `pre_tasks` to set handler context facts that get logged. Implement explicit handler logging with `debug` module before handler execution. Use callback plugin to aggregate all handler events and send to monitoring system. Implement handler metadata: associate each handler with change ticket, approval, reason. Store in database for historical audit. Implement alerting on failed handlers. Create dashboard showing handler execution trends, failure rates, execution times. Use `ansible_callback_handler_details` hook to log handler specifics. Implement role-based audit visibility where audit reviewers can query handler execution history.