CloudWatch Alarms Fundamentals

Two alarms, same night, same account. The first never fired while the payment queue backed up for 40 minutes. The second fired 14 times between 02:00 and 03:00 while nothing was actually wrong. Both were built by engineers who knew exactly what CPUUtilization and ApproximateAgeOfOldestMessage mean. Neither knew how CloudWatch decides when an alarm changes state.

That decision is where alarms are won and lost. A metric is just a list of numbers, and you already know how CloudWatch stores them. An alarm is a small state machine sitting on top of that list, and almost every alarm complaint in production traces back to one of four settings inside it.

What an alarm is made of

A metric alarm watches one metric (or one metric math expression) and holds one of three states. To decide which, it needs answers to six questions:

A seventh setting, the missing-data treatment, only matters when data goes silent, and it gets its own section below because it causes more surprises than the other six combined.

Here is one alarm with real values: watch CPUUtilization for instance i-0a1b2c3d4e5f67890, take the Average over each 60-second period, call a period bad when the average is above 80, and go to ALARM when 3 of the last 5 periods were bad.

aws cloudwatch put-metric-alarm \
  --alarm-name "checkout-api-cpu-high" \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0a1b2c3d4e5f67890 \
  --statistic Average \
  --period 60 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 5 \
  --datapoints-to-alarm 3 \
  --treat-missing-data missing \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:ops-critical

Three states, and the rule almost everyone gets wrong

An alarm is always in exactly one of these:

• OK: the metric is within the threshold.
• ALARM: the metric is outside the threshold.
• INSUFFICIENT_DATA: the alarm just started, the metric is not available, or there is not enough data to decide.

INSUFFICIENT_DATA is not an error. A brand new alarm starts there by design, and an EBS volume that is available but not attached to any instance stops publishing metrics, which is a perfectly healthy reason for its alarm to sit in that state.

Now the rule: an alarm invokes its actions only when the alarm changes state. Not every period. Not every minute. Once, on the transition.

It is tempting to assume that an alarm stuck in ALARM keeps paging you, and that a silent phone means the problem cleared. Neither is true. The alarm sent one notification when it crossed into ALARM and has been quietly sitting there ever since. If your process needs repeated reminders while an incident is open, that repetition comes from your paging tool, not from CloudWatch.

There is exactly one exception, and the exam likes it: for Auto Scaling actions, the alarm continues to invoke the action once per minute for as long as it stays in the new state. That is deliberate, because a scaling policy that fired once and then went quiet would never finish scaling out a badly overloaded group.

Period, Evaluation Periods, and Datapoints to Alarm

These three settings are where the "never fired" and "fired 14 times" alarms come from, so work through them with numbers rather than definitions.

Period is how much time each data point covers. Valid values are 10, 20, 30, or any multiple of 60 seconds.

Evaluation Periods (N) is how many of the most recent data points the alarm looks at.

Datapoints to Alarm (M) is how many of those N must be breaching for the alarm to go to ALARM. The breaching points do not have to be consecutive. They only have to fall inside the window of the last N.

When M equals N, you have a "consecutive periods" alarm: every point in the window must be breaching. When M is smaller than N, you have an M out of N alarm, and it tolerates a healthy blip in the middle of a bad stretch.

The evaluation interval is simply N multiplied by the period. Four out of five data points at a 1-minute period is a 5-minute interval. Three out of three at a 10-minute period is a 30-minute interval.

For any period of one minute or longer, the alarm is evaluated every minute, and the window slides. With a 5-minute period and 1 evaluation period, the end of minute 5 evaluates minutes 1 through 5, and the end of minute 6 evaluates minutes 2 through 6. If the period is 10, 20, or 30 seconds, the alarm is evaluated every 10 seconds instead.

Two quotas bound how far back an alarm can look. Period multiplied by Evaluation Periods can be at most 604,800 seconds (seven days) for alarms with a period of at least one hour, and at most 86,400 seconds (one day) for anything shorter. And an alarm whose window exceeds one day becomes a multi-day alarm, which is evaluated only once per hour and takes into account only metrics up to the current hour at the :00 minute. A job that fails at 10:02 will not move such an alarm at 10:03; the alarm reacts at 11:03.

Missing data: what silence means

Every data point in the window is one of three things: not breaching, breaching, or missing. The first two are obvious. The third is a judgment call that only you can make, because CloudWatch has no way to know whether a metric that went quiet means "everything is fine" or "the thing that publishes this metric is dead".

So you tell it, with one of four settings:

The default is missing. Two exceptions are worth memorizing. Alarms on metrics in the AWS/DynamoDB namespace always default to ignore. And AWS specifically recommends missing for alarms that stop, terminate, reboot, or recover EC2 instances, because EC2 metric reporting can be interrupted briefly on a perfectly healthy instance, and you do not want a gap in reporting to terminate a production server.

Concrete picks: DynamoDB ThrottledRequests publishes a data point only when a request is throttled, so notBreaching is right. An alarm that triggers a deployment rollback watches a metric that reports continuously, so a gap probably means the app stopped answering, and breaching is right.

The evaluation range, and why your missing-data setting is often ignored

Here is the part that makes alarms behave in ways the settings do not obviously predict.

Whenever an alarm evaluates, CloudWatch retrieves more data points than Evaluation Periods. The time frame of those extra points is called the evaluation range. For an alarm with 3 evaluation periods, the evaluation range is 5 data points.

Then it applies three rules in order:

1. If no points in the evaluation range are missing, it evaluates the most recent N and ignores the extras.
2. If some are missing but the total number of real points retrieved is at least N, it evaluates the most recent N real points, reaching back into the extras. Your missing-data setting is not used at all.
3. Only if real points are still fewer than N does CloudWatch fill the gaps using your missing-data setting, and even then it uses as few substituted points as possible.

Think of it as a teacher grading your last 3 homework scores. If two of the last 3 were never handed in, the teacher reaches back to the 4th and 5th assignment rather than marking the missing ones as zero. The analogy breaks in one place: the teacher would keep reaching back indefinitely, while CloudWatch only reaches as far as the evaluation range and then falls back to your setting.

This is why two engineers can set breaching and notBreaching on the same intermittent metric and watch both alarms behave identically for weeks.

There is one more piece of logic in there, called premature alarm state avoidance. With Datapoints to Alarm set to 3, data of - - - - X (four missing, then one breaching) does not immediately go to ALARM, because the next point might be healthy. But data of - - X - - does go to ALARM even with missing treated as missing, because the oldest available breaching point is at least as old as the value of M and everything newer is breaching or missing. Do not expect the missing-data table alone to predict these edges.

High-resolution alarms

Set the period to 10, 20, or 30 seconds and you have created a high-resolution alarm, evaluated every 10 seconds and billed at a higher rate than a normal alarm.

Use those periods only for metrics that were published with a storage resolution of 1, meaning your own high-resolution custom metrics. Point a 10-second alarm at a standard-resolution metric and CloudWatch still tries to gather data every 10 seconds, finds nothing for five out of six attempts, and drops the alarm into INSUFFICIENT_DATA regularly. You get an unreliable alarm and the premium price at the same time.

Beyond a static number

Not every question has a fixed threshold, and CloudWatch gives you two ways past that.

Metric math alarms watch the output of an expression rather than a raw metric. The classic case is an error rate, because "500 errors" means something completely different at 600 requests than at 6 million:

{
  "Metrics": [
    { "Id": "errors",   "MetricStat": { "Metric": { "Namespace": "MyService", "MetricName": "ConnectionsFailed" }, "Period": 60, "Stat": "Sum" }, "ReturnData": false },
    { "Id": "attempts", "MetricStat": { "Metric": { "Namespace": "MyService", "MetricName": "ConnectionAttempts" }, "Period": 60, "Stat": "Sum" }, "ReturnData": false },
    { "Id": "error_rate", "Expression": "(errors/attempts)*100", "ReturnData": true, "Label": "Connection error rate" }
  ],
  "Threshold": 40,
  "ComparisonOperator": "GreaterThanThreshold",
  "EvaluationPeriods": 3
}

Exactly one item in the Metrics array sets ReturnData to true, and that is the expression the alarm watches.

Anomaly detection alarms have no static threshold at all. CloudWatch trains a machine learning model on up to two weeks of the metric's past data, learning its hourly, daily, and weekly patterns plus its longer trend, and produces a band of expected values. The alarm then fires when the metric goes above the band, below it, or outside it in either direction.

{
  "Metrics": [
    { "Id": "m1", "ReturnData": true, "MetricStat": { "Metric": { "Namespace": "AWS/EC2", "MetricName": "CPUUtilization" }, "Stat": "Average", "Period": 60 } },
    { "Id": "t1", "Expression": "ANOMALY_DETECTION_BAND(m1, 3)" }
  ],
  "ThresholdMetricId": "t1",
  "ComparisonOperator": "LessThanLowerOrGreaterThanUpperThreshold",
  "EvaluationPeriods": 2
}

The 3 is the anomaly detection threshold, and a higher number produces a thicker band and fewer alerts. Four details decide whether this is the right tool:

• The model is specific to one metric and one statistic. A model trained on Average tells you nothing about Maximum.
• You can exclude time periods from training, which is how you stop last month's load test from teaching the model that a 10x spike is normal.
• Alarms based on an anomaly detection model cannot have Auto Scaling actions.
• Each anomaly detection alarm bills as three standard-resolution alarm metrics (the metric plus the upper and lower bounds), so roughly $0.30 per month against $0.10 for a plain metric alarm.

Anomaly detection earns its place on metrics with a strong daily shape, such as request count on a consumer app. It is the wrong tool for a metric where any nonzero value is bad.

Diagnosing an alarm that will not leave INSUFFICIENT_DATA

This is the single most common alarm ticket, and it has a short list of causes:

• The period is shorter than the metric's resolution. Basic EC2 monitoring publishes every 5 minutes, so a 60-second period leaves four out of five periods empty.
• The dimension set was never published. An alarm can be created before its custom metric exists, and it will happily sit in INSUFFICIENT_DATA forever if the dimensions do not match exactly what you publish.
• A Unit was specified that the metric never uses. AWS recommends omitting Unit entirely, because a mismatch produces an alarm that is stuck rather than one that errors.
• The resource is genuinely idle. Detached EBS volumes, Lambda functions with no invocations, and Auto Scaling groups at zero instances all stop publishing.

Alarm history is kept for 30 days, and describe-alarm-history shows each state transition with its timestamp. That is your first stop when someone asks whether an alarm ever fired.

aws cloudwatch describe-alarm-history \
  --alarm-name "checkout-api-cpu-high" \
  --history-item-type StateUpdate \
  --max-records 10

Exam tips

• "The alarm stayed in ALARM but we only got one email" is never a bug. Actions fire on state change. The only action re-invoked while the state holds is an Auto Scaling action, once per minute.
• Read the wording of thresholds carefully. "3 consecutive periods" means M equals N; "3 out of 5" means M is 3 and N is 5, and the breaching points may be scattered.
• Map the missing-data options to the metric's nature: a metric that only publishes on failure points at notBreaching; a continuously reported metric where silence is suspicious points at breaching; EC2 stop/terminate/reboot/recover alarms point at missing, which is also the global default.
• A 10, 20, or 30 second period means a high-resolution alarm, and it only works on metrics stored at 1-second resolution. Question stems mentioning both a sub-minute period and an AWS-published metric are describing a broken alarm.
• If a scenario says "no fixed threshold works, the normal level changes by time of day", the answer is anomaly detection, and remember it cannot drive Auto Scaling.
• An alarm window longer than a day is evaluated hourly against data up to the top of the hour, and the total window caps at seven days.

The rule to carry forward: an alarm's state is decided by a sliding window of the last N data points, and everything that feels unpredictable about alarms comes from what CloudWatch does when that window has holes in it. Next you will follow the notification out of the alarm and into Amazon SNS, where a different set of quiet failures waits for you.