CloudWatch Metrics Fundamentals

An alert fires at 09:14: the checkout API is timing out. You open the EC2 console, look at the instance, and see CPU at 22 percent. Nothing looks wrong. The reason nothing looks wrong is that the instance ran out of memory, and CloudWatch has no memory metric for EC2 unless you put one there yourself.

Every lesson in this course eventually reads a number out of CloudWatch. Before you can trust those numbers you need to know how CloudWatch stores them: what makes one metric different from another, how often values arrive, how long they survive, and which statistic answers which question. Get this layer wrong and everything built on top of it, alarms, scaling policies, dashboards, gives you confident answers to the wrong question.

The address of a metric

A metric is a time-ordered set of data points. Each one has a value, a timestamp, and optionally a unit. On its own that is just a list of numbers, so CloudWatch needs a way to tell one list from another.

Three things make up the address:

• Namespace: the container. AWS services use the pattern AWS/service, so EC2 metrics live in AWS/EC2, Lambda metrics in AWS/Lambda. There is no default namespace, and you must name one for every data point you publish.
• Metric name: CPUUtilization, Invocations, FreeStorageSpace.
• Dimensions: zero or more name/value pairs, such as InstanceId=i-0a1b2c3d. A metric can carry up to 30 of them.

Namespaces isolate. Metrics in different namespaces never get aggregated together, which is why a custom application metric called Errors will not contaminate Lambda's Errors. Metrics also live only in the Region where they were created. A dashboard in eu-west-1 will not find a metric published in us-east-1 unless you build a cross-Region view deliberately.

Dimensions are part of the identity, not a filter

This is the part that trips people up, so work through it with real values.

Suppose you publish four data points to a namespace called Checkout, all with the metric name OrdersProcessed:

Dimensions: Service=api,    Stage=prod  Value: 105
Dimensions: Service=api,    Stage=beta  Value: 115
Dimensions: Service=worker, Stage=prod  Value: 95
Dimensions: Service=worker, Stage=beta  Value: 97

You can retrieve statistics for exactly those four combinations. You cannot retrieve Service=api on its own, and you cannot retrieve Stage=prod on its own, because you never published a data point with only those dimensions. The query returns nothing at all, which reads on a dashboard like "the metric is broken" when it actually means "that metric was never created".

The misconception worth naming out loud: dimensions feel like filters on one big table, so people expect Stage=prod to slice the data. They are not filters. CloudWatch treats every unique combination of dimensions as its own separate metric. Adding one new dimension value creates a new variation of the metric rather than a new row inside the old one.

There is one exception that makes the confusion worse. For metrics from certain AWS services, EC2 among them, CloudWatch does aggregate across dimensions: search AWS/EC2 for CPUUtilization with no dimensions and you get an average across your instances. CloudWatch never does this for custom metrics. So the behavior you learn from clicking around EC2 graphs is exactly the behavior that will not happen to the metrics your own code publishes.

If you want a total as well as a per-instance breakdown for a custom metric, publish both: one data point with the full dimension set, and one with the dimensions you want the rollup on.

Resolution: how often a number arrives

Every metric is one of two resolutions.

Standard resolution has one-minute granularity. Everything AWS services publish is standard resolution.

High resolution stores data at 1-second granularity and is only available for custom metrics you publish yourself. You can then read it back at a period of 1, 5, 10, or 30 seconds, or any multiple of 60.

A period is the length of time a statistic covers. Valid values are 1, 5, 10, 30, or any multiple of 60 seconds, and the default is 60. Aggregated data is stamped with the start of its period, so data covering 19:00 to 20:00 is stamped 19:00.

High resolution costs more in two places. Every PutMetricData call is billed, so publishing every second multiplies that charge, and a high-resolution alarm with a 10-second or 30-second period carries a higher alarm charge than a normal one. Reach for it when you genuinely need sub-minute reaction time, not by default.

Basic and detailed monitoring on EC2

EC2 publishes a fixed set of metrics for free at 5-minute intervals: CPUUtilization, NetworkIn, NetworkOut, DiskReadOps, DiskWriteOps, DiskReadBytes, DiskWriteBytes, and a few others. That is basic monitoring, and it is on by default.

Detailed monitoring is a paid per-instance option that drops the interval to 1 minute. Same metrics, five times the resolution.

aws ec2 monitor-instances --instance-ids i-0a1b2c3d4e5f67890

Neither setting adds memory utilization or file system usage. Those numbers live inside the guest operating system, and the hypervisor that publishes EC2 metrics cannot see in. Collecting them needs the CloudWatch agent, which is the subject of a later lesson in this topic.

The exam-relevant consequence: an alarm's period must be at least as long as the metric's resolution. Put a 60-second period on a basic-monitoring metric and four out of five periods contain no data, so the alarm parks itself in INSUFFICIENT_DATA and never tells you anything. Either use a 300-second period or turn on detailed monitoring.

Retention: your data ages into coarser buckets

CloudWatch does not keep your 1-minute data forever, and it does not throw it away either. It aggregates it.

Follow one data point published today at 1-minute resolution. For 15 days you can graph it minute by minute. On day 16 the minute detail is gone, but the data still exists as 5-minute aggregates, and stays that way until day 63. After that only the hourly rollup survives, for a total of 455 days.

Two consequences that show up in real incidents. First, a post-mortem written three weeks after the event cannot show you the exact minute a spike started, only the 5-minute bucket it fell into. If minute-level detail matters for compliance or analysis, export it or stream it somewhere durable while it still exists. Second, a metric that stops receiving data disappears from the console search after two weeks even though get-metric-data can still return it. A metric nobody can find in the console is not the same as a metric that was deleted.

Metrics cannot be deleted at all. They expire on their own after 15 months without new data. That makes a badly chosen dimension, an instance ID baked into a metric name, or a stray namespace something you live with rather than clean up.

Statistics: the same data, different stories

A statistic is an aggregation over a period. The choice changes the answer.

• Average smooths. Good for capacity trends, bad for finding the one bad instance, because a fleet average of 40 percent hides the instance at 99 percent.
• Maximum is the opposite. One anomalous spike sets the whole graph's ceiling and makes a healthy system look sick.
• Sum is right for counters. Errors, invocations, request counts: the question is "how many", not "how big".
• SampleCount tells you how many data points went into the period, which is how you notice that a metric quietly stopped reporting.
• Percentiles sit between Average and Maximum. p95 is the value below which 95 percent of your data falls, so it shows sustained heavy load while ignoring the occasional outlier. You can specify up to ten decimal places, as in p95.0123456789.

Percentiles come with conditions. CloudWatch needs the raw, unsummarized data points to calculate them, so if you published a pre-aggregated statistic set, percentiles are usually unavailable. They also do not work on metrics with negative values. Services that support percentile statistics include API Gateway, Application Load Balancer, EC2, ELB, Kinesis, Lambda, and RDS.

A concrete pick: for "is my latency acceptable for most users", use p95 on Duration. For "did anything break", use Sum on Errors. For "am I about to hit a concurrency limit", use Maximum on ConcurrentExecutions.

Publishing your own metrics

Anything CloudWatch does not measure for you, you can publish yourself with PutMetricData.

aws cloudwatch put-metric-data \
  --namespace Checkout \
  --metric-name QueueDepth \
  --dimensions Service=worker,Stage=prod \
  --value 47 \
  --unit Count

Three details decide whether this works in production.

Timestamps. A data point can be stamped up to two weeks in the past and up to two hours into the future. Omit the timestamp and CloudWatch stamps it on arrival. Alarms evaluate against the current UTC time, so a metric arriving with a skewed clock lands in periods the alarm has already evaluated. The symptom is an alarm stuck in INSUFFICIENT_DATA or firing long after the event, not an API error, which makes clock skew a genuinely hard bug to spot.

Volume. Every PutMetricData call is billed. If you need many samples per minute, publish a statistic set instead: give CloudWatch the Min, Max, Sum, and SampleCount for a batch of observations in one call. The tradeoff is that you lose percentile statistics, since the raw points are gone.

Same identity, many sources. CloudWatch does not care which host sent a data point. Twenty web servers publishing to the same namespace, metric name, and dimensions produce one metric describing all of them. That is often exactly what you want for a fleet-level latency number.

For applications that already write structured logs, the embedded metric format (EMF) is worth knowing: you write a specially structured JSON log event to CloudWatch Logs, and CloudWatch extracts metrics from it automatically. Container Insights and Lambda Insights both work this way, which you will see later in this topic.

Exam tips

• "Memory utilization" and "disk space used on the file system" always point at the CloudWatch agent, never at basic or detailed monitoring. Detailed monitoring changes frequency, not coverage.
• Basic monitoring is 5 minutes, detailed is 1 minute. An alarm in INSUFFICIENT_DATA with a 60-second period on an EC2 metric is the classic basic-monitoring trap.
• A query that returns no data for a partial dimension set is not a bug. Check whether that exact combination was ever published, and remember that custom metrics never aggregate across dimensions.
• High resolution means 1 second, and it applies only to custom metrics. High-resolution alarms can use 10 or 30 second periods and cost more.
• Memorize the retention ladder as 3 hours, 15 days, 63 days, 455 days. Questions phrase it as "can we still see minute-level data from three months ago", and the answer is no, only the hourly rollup.
• If a question contrasts hiding outliers with being dominated by them, the intended answer is a percentile.

The rule to carry forward: a metric is defined by namespace plus name plus its exact dimension set, and everything else, resolution, retention, statistics, follows from how it was published. Next you will move from numbers to text, where the raw evidence lives: CloudWatch Logs, and the query language that turns a million log lines into an answer.