How to reduce alert fatigue across Datadog, New Relic and CloudWatch

Alert fatigue is what happens when most pages need nobody to do anything: engineers learn to ignore them, and the one that matters gets missed. The fix is a routine, not a single setting: measure how many pages needed a human, fix the noisiest monitors, route alerts to the team that owns them, group related alerts from every monitoring tool into one incident before anyone is paged, and keep planned noise out of the pager.

Why teams with several monitoring tools get it worst

One failure rarely raises one alert. A bad deploy to checkout can fire a Datadog APM error-rate monitor, an Amazon CloudWatch alarm on the load balancer's 5xx count, and a New Relic synthetic check, each repeating every few minutes. Without grouping, that is dozens of pages, sometimes to three different people, for one problem. Each monitoring tool can only deduplicate its own alerts; none of them sees the other two.

Step 1: Measure the noise

You can't reduce what you don't count. For the last 30 days, track:

The noisiest few monitors usually account for most of the pages, so that list is the to-do list.

Step 2: Fix or delete the noisiest monitors

For each monitor at the top of the list, decide one of four things:

• Fix the threshold so it fires on real impact, not on a brief spike.
• Demote it to a notification or a ticket if it needs attention but not at 3 a.m.
• Merge it with another monitor that watches the same thing.
• Delete it if nobody has acted on it in a month.

This is work in the monitoring tools themselves, and no incident platform replaces it.

Step 3: Route every alert to the team that owns it

An alert that pages the wrong team is noise twice: once for the person woken up, and again for the delay before the right one is found. Map each alert to a service with an owner and an escalation policy, and route on fields the alert already carries, such as the source, the tenant or the component.

Step 4: Group related alerts into one incident before paging

This is the step that ends alert storms, and it has to happen in the one place that receives alerts from every tool. Common ways to group:

• Deduplicate repeats of the same alert into the open incident.
• Group by service and time window, so alerts for the same service within a set window join one incident.
• Group by fields in the alert, such as the component or the customer.
• Group by similarity of the alert text, for alerts that describe the same failure in different words.

Whatever the method, the tool should show why each alert was grouped, so engineers trust it, and let you test a rule against past alerts before it goes live.

Step 5: Keep planned noise out of the pager

• Maintenance windows for planned work, so alerts during a migration or a deploy are recorded but nobody is paged.
• Short mutes for a known, noisy problem that is already being fixed.
• Severity-based paging, so low-severity alerts wait for working hours instead of waking someone.

Step 6: Review it every week

Put the noisiest monitors and the actionable-page share on the agenda of a weekly ops review. Alert fatigue comes back as systems and teams change; a weekly look keeps it from building up.

How to do this in Oppex

Oppex receives alerts from Datadog, New Relic, Amazon CloudWatch, Grafana, Google Cloud Monitoring, Splunk, OpenSearch, Site24x7, Cloudflare, GroundCover and any tool that can send a webhook, so it is the one place that sees them all.

• Grouping strategies, per service. You describe which alerts belong together, and Oppex attaches each matching alert to the open incident instead of opening a new one. A group stays open for a window you set, from 5 minutes to 24 hours. (Grouping strategy)
• Test before you rely on it. The Simulate tab replays a strategy against the incidents a service has already received, so you see how much traffic it would have grouped.
• Rules on alert fields. Route, group or notify by an alert's source, tenant, component, type or external ID. (Use incident fields in rules)
• Maintenance windows for one service or many, for every alert or only those matching a filter: the incident is recorded and resolved, and nobody is paged. (Maintenance windows)
• Mute a recurring incident for up to 7 days while the fix is in progress. (Mute)
• Severity-based routing, so each severity follows its own escalation policy. (Severity-based routing)

And when a real incident does page, Oppex Nova, the Oppex AI SRE agent, starts investigating the cause as the alert arrives. Alert grouping is included in every Oppex plan, including the free Starter plan for up to 3 users.