Missing or flaky alerts are one of the most dangerous problems in any monitoring stack: they erode trust, delay incident response, and increase mean time to resolution. Whether an alert never fires, fires intermittently, or reaches the wrong person, the result is the same — critical issues go unaddressed. This post gives a practical, technical checklist for diagnosing and fixing missing or flaky alerts, with concrete steps you can run now and preventative measures to avoid repeat incidents. We'll also explain how our service can simplify alert reliability at scale.
Quick triage: Are alerts truly missing or just delayed?
Missing vs. flaky: define the behavior
Missing alerts are notifications that should have been generated and delivered but were not observed at all. Flaky alerts are notifications that arrive intermittently, arrive late, or appear duplicated. The first step in troubleshooting is to classify the symptom accurately.
Initial checks
- Check the incident timeline: correlate the event time in logs/metrics with any alert evaluation window.
- Confirm alert configuration: verify which alerts should have fired for the observed metric value.
- Inspect delivery channels: was the notification queued, retried, or rejected by the downstream provider?
- Ask: is the problem isolated to one alert, one team, or system-wide?
"If you can't reproduce the failure reliably, you can't fix it reliably." — a pragmatic troubleshooting mantra.
Common root causes and targeted verification steps
1. Data collection gaps (metrics, traces, logs)
Alerts rely on accurate data. Missing metrics, broken exporters, or logging outages will cause alerts to not evaluate correctly.
- Verify metric ingestion: check last metric timestamp for the relevant series.
- Look for agent errors: inspect exporter/agent logs for crashes, OOMs, or permission errors.
- Confirm retention and rollups: long-term downsampling might hide short spikes that should trigger alerts.
2. Alert rule logic and evaluation windows
Incorrect thresholds, labels, or evaluation intervals are common culprits.
- Confirm query correctness: run the alerting query manually against the time window that had the incident.
- Check evaluation interval and for clause: does the rule require a sustained condition that was too short-lived?
- Validate label selectors: mismatched labels can make an alert rule miss the target metric.
3. Notification delivery pipeline
Even a correctly firing alert can be lost in delivery.
- Check queuing and retries: are messages being retried or dropped after a throttle limit?
- Inspect third-party provider logs (email, SMS, chatops): are they rejecting or delaying messages?
- Test webhook/push endpoints: verify the endpoint returns 200 and within acceptable latency.
4. Infrastructure and network issues
Agent-to-collector connectivity, DNS problems, or time sync issues can cause nondeterministic behavior.
- Verify NTP/time skew: timestamp mismatches can make alerts appear out of order or not match evaluation windows.
- Check host/agent connectivity: packet loss can cause missing samples.
- Inspect resource saturation: CPU, memory, or I/O pressure in the monitoring pipeline can lead to dropped evaluations.
5. Deduplication, silencing, and suppression rules
Silences, maintenance windows, and dedupe logic are often set up to reduce noise but can hide important alerts.
- Audit active silences and maintenance windows during the incident timeframe.
- Check deduplication grouping: you may be dropping unique alerts by grouping too broadly.
- Review escalation policies: suppressed or routed incorrectly due to misconfigured policies.
Step-by-step troubleshooting checklist
-
Reproduce the condition
Simulate the threshold or use a synthetic test to trigger the alert deterministically.
-
Verify data presence
Run queries for the relevant metric/span/log over the exact incident window. Example (Prometheus):
promql:
rate(http_requests_total{job="api"}[5m]) -
Check rule evaluation
Review the alert manager or rule engine evaluation history and timestamps.
-
Test delivery
Send a test notification through the same channel: use curl for webhooks, sendmail for SMTP, or your provider's test tool. Observe response codes and latencies.
-
Inspect logs
Collect logs for the monitoring stack components (collector, rule engine, alert dispatcher) around the incident time.
-
Check external providers
Look for known incidents with your notification providers that could explain delays or drops.
-
Validate suppression rules
Confirm there were no active silences, deduplication collisions, or global throttles.
-
Document and automate
Capture the root cause and automate synthetic tests and postmortem checks to prevent recurrence.
Practical commands and tests
- Check latest metric timestamp: query your metrics store for max(timestamp) for the series.
- Test webhook endpoint:
curl -X POST -d '{"test":"payload"}' -H 'Content-Type: application/json' https://your-webhook - Verify SMTP delivery: use swaks or sendmail to test SMTP connectivity and authentication.
- Inspect agent logs: tail logs for exporters and collectors, grep for "error", "timeout", "OOM".
Best practices to prevent missing or flaky alerts
- Monitor the monitors: create heartbeat alerts for your monitoring agents and pipelines so you know when they stop reporting.
- Synthetic and blackbox tests: use probes that exercise endpoints and alerting paths end-to-end.
- Alert reliability metrics: measure delivery success, latency, retry rates, and false positive/negative counts.
- Redundancy and fallback channels: configure multiple notification channels with escalation paths.
- Runbooks and automation: have documented, automatable remediation steps for common failures (restart agent, requeue notifications, clear silences).
- Careful suppression: use scoped silences, not global, and ensure silences auto-expire.
- Periodic review: audit alert rules for correctness, relevance, and noise reduction.
How our service helps
Our service is built to make alerting reliable and simple, so you can focus on resolving incidents instead of chasing notifications. Here’s how we address the common failure modes above:
- End-to-end observability: we track each alert from evaluation through delivery, including timestamps, retries, and final status so you can pinpoint where a failure occurred.
- Robust delivery pipeline: built-in retries, exponential backoff, multi-channel failover, and provider-specific optimizations reduce dropped notifications and delays.
- Heartbeat and synthetic tests: automated probes ensure your monitoring agents and notification channels are healthy, alerting you before real incidents occur.
- Actionable analytics: alert reliability metrics and historical trends help you reduce flakiness and alert fatigue over time.
- Easy integrations and runbooks: connect to popular monitoring backends and attach remediation steps to alerts so responders can act quickly and consistently.
Conclusion
Missing or flaky alerts are fixable with a methodical approach: confirm the symptom, verify data and rule evaluation, inspect the delivery pipeline, and prevent recurrence with monitoring of your monitoring. Use synthetics, heartbeats, and analytics to detect problems early and reduce incident risk. Our service streamlines these best practices with end-to-end visibility, robust delivery, and built-in testing so you can trust your alerts when it matters most.
Start improving alert reliability today: Sign up for free today and set up heartbeat checks and synthetic tests in minutes.