EC2 Auto Scaling Groups

A media company runs 12 EC2 instances behind an Application Load Balancer. Peak traffic arrives for about 3 hours on weekday evenings. The rest of the week, 3 instances would carry the load with room to spare, but the fleet stays at 12 because nobody wants to be the person who shrank it the week before a launch. Then on a Saturday one instance fills its disk, stops answering, and sits there dead until Monday morning because no health check was wired to anything that could act.

Those are 2 different failures, and an Auto Scaling group fixes both with the same mechanism. It is a group of EC2 instances that AWS keeps at a size you declare, launching and terminating instances to hold that size, and replacing any instance that stops looking healthy. The scaling policies that make the size move come in the next lesson. This one covers the machinery underneath them, because almost every scaling question that goes wrong in production goes wrong here, not in the policy.

Three numbers, and a policy only moves one of them

An Auto Scaling group is defined by 3 capacity values.

• Minimum capacity is a floor. The group never runs fewer instances than this.
• Maximum capacity is a ceiling. The group never runs more.
• Desired capacity is the number the group is actually trying to run right now.

Say you set minimum 2, desired 6, maximum 12. The group launches 6 instances. If a scaling policy later calculates that 15 instances are needed to hold CPU at target, the group goes to 12 and stops, because 15 is above the ceiling. If a scale-in calculation says 1 instance is enough, the group goes to 2 and stops.

The sentence worth memorizing: a scaling policy changes desired capacity, and only desired capacity. Minimum and maximum are clamps that constrain the result. This is where a common misreading lives. Setting maximum capacity to 12 does not mean the group runs 12 instances. It means the group will never run more than 12. A group sitting at its maximum because desired capacity was raised there is a completely different situation from a group sitting at 6 with headroom to 12, and the exam likes stems where the group has already hit its ceiling and the scaling policy appears broken.

There is also value in a group with no scaling policy at all. With desired capacity fixed at 6, the group still watches those 6 instances and replaces any that fail a health check. Maintaining capacity and scaling capacity are separate features, and the first one is the reason a group is worth creating even for a fleet whose size never changes.

The launch template is the blueprint

The group needs to know what to launch. That comes from a launch template: AMI ID, instance type, key pair, security groups, IAM instance profile, block device mappings, user data, and the rest of the EC2 launch surface.

Launch templates are versioned. You create version 3 with a new AMI, point the group at it, and every instance launched from that moment uses version 3 while the existing instances keep running whatever they launched with. That version history is what makes the default termination policy and instance refresh work, and both show up later.

The older launch configuration still exists in old material and old accounts. It is not versioned, it cannot be edited (you replace it), and AWS does not extend it to new EC2 features. Treat "migrate the group from a launch configuration to a launch template" as the expected answer whenever a scenario mentions Spot and On-Demand in one group, multiple instance types, or an instance refresh, because none of those work with a launch configuration.

Health checks decide what gets replaced

An instance in an Auto Scaling group starts as Healthy, and stays that way until something tells the group otherwise. The group listens to several sources:

That table hides the single most common misconfiguration in this topic, so say it out loud. By default an Auto Scaling group only uses EC2 status checks. An instance whose application is returning HTTP 500 to every request still passes EC2 status checks, because the hypervisor is fine and the operating system is running. The ALB will notice, mark the target unhealthy, and stop sending it traffic. The Auto Scaling group will not replace it, and you end up with a group reporting 6 healthy instances behind a target group reporting 5 healthy targets, indefinitely. Turning on the Elastic Load Balancing health check for the group is what closes that gap.

aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name web-asg \
  --health-check-type ELB \
  --health-check-grace-period 300

The health check grace period in that command is the second thing to get right. It is the minimum time a newly in-service instance is left alone before an unhealthy verdict can terminate it. Elastic Load Balancing health checks start as soon as the instance is registered, so an application that needs 4 minutes to warm caches would fail its first several checks and be killed for it. The grace period buys that time.

The defaults are not the same everywhere, and this is a real trap:

• Console: 300 seconds
• AWS CLI or SDK: 0 seconds, which turns the grace period off entirely

A group created by a CloudFormation template or a CLI script with no explicit grace period will therefore judge instances from the first second, and a slow-booting fleet can land in a loop where every replacement is killed before it finishes booting. One exception applies during the grace period: if the instance leaves the EC2 running state, for example because someone stopped it, the group marks it Unhealthy and replaces it immediately without waiting.

Do not solve the problem by pushing the grace period to an hour. A high grace period is dead time in which a genuinely broken instance stays in service. The better fix, covered in the lifecycle lesson, is a launch lifecycle hook that keeps the instance out of service until its bootstrap finishes, which lets you set the grace period low.

Two more behaviors worth holding: health check replacement does not wait for a cooldown period, and an instance the group has already marked unhealthy skips termination policy evaluation entirely. Unhealthy instances are not chosen, they are simply removed.

Zonal balance comes before everything else

You give the group subnets, and each subnet lives in exactly one Availability Zone. When the group launches an instance, it picks the enabled zone with the fewest instances, and inside that zone the subnet with the most free IP addresses. When it terminates, it looks first at the zone with the most instances.

That ordering matters more than it looks. Zonal balance is evaluated before the termination policy, always. So a group with 5 instances in us-east-1a and 3 in us-east-1b will terminate from us-east-1a even if the single oldest instance in the group is sitting in us-east-1b. When a scenario says "the newest instance was terminated and we do not understand why", zonal imbalance is usually the answer.

Groups drift out of balance for ordinary reasons: a zone ran out of capacity for a while and recovered, you changed the enabled zones, you put instances in standby, or a Spot price dropped back under your maximum in a zone that had been priced out. When that happens the group runs an Availability Zone rebalancing activity. It launches the new instance first and terminates the old one afterward, so rebalancing never dips your capacity.

Launching before terminating creates an obvious problem when the group is already at maximum capacity, and AWS solved it with a documented exception: during a rebalancing activity the group may temporarily exceed maximum capacity by 10 percent or one instance, whichever is greater. The margin lasts only as long as the rebalance, usually a few minutes. If you cannot tolerate even that overshoot, an instance maintenance policy lets you set the healthy percentage range instead.

Which instance dies on scale in

Once the zone is chosen, the default termination policy works through unprotected instances in this order:

1. Outdated configurations first. For a group on launch templates, that means instances launched from a launch configuration, then instances launched from a different launch template, then instances on the oldest version of the current launch template.
2. Closest to the next billing hour. If several candidates remain, the group picks the one nearest its next billing hour, and breaks a tie at random. This step matters far less than it used to, since most EC2 usage bills per second.

The design goal is to retire stale configurations naturally as the group scales in, which is why a group that scales up and down through the day slowly converges on the current launch template version without anyone doing anything.

Mixed instances groups add one step in front. The group first decides whether a Spot or an On-Demand Instance should go, so the fleet trends back toward the ratio you configured, then it checks whether terminating a particular instance improves alignment with your allocation strategy, and only then falls through to outdated configurations and billing hour.

You can override the policy when the default is wrong for your case:

Whatever you choose, zonal balance still wins first. A termination policy only decides which instance inside the selected zone goes.

Protecting an instance from scale in, and what that does not buy you

Some instances should not be chosen. A container host mid-job, a build agent 40 minutes into a compile, a node holding a long-running session. Instance scale-in protection marks an instance as ineligible for termination by a scale-in event. You can set it on the group so every new instance inherits it, then clear it per instance when the work finishes, which is the pattern container schedulers use.

The important half of this feature is what it does not do. Scale-in protection does not prevent:

• Replacement after a failed health check
• A Spot Instance interruption
• The end of a Capacity Block reservation
• Manual termination through terminate-instance-in-auto-scaling-group
• Manual termination from the EC2 console, CLI, or API

That last one surprises people. To stop a human from terminating the instance in the EC2 console you need EC2 termination protection, a separate setting on the instance itself. The two names sound like the same feature and are not.

One edge case shows up in real incidents: if every instance in the group is protected and a scale-in event fires, the group decrements desired capacity but cannot terminate anything. The activity history records Could not scale to desired capacity because all remaining instances are protected from scale in, and the group quietly runs above its own desired capacity until protection is cleared somewhere.

Turning parts of the group off while you work

When you need the group to stop acting for a while, suspend individual processes rather than deleting policies:

Picking the right one is a diagnosis question. Suspending Launch stops the churn but also blocks a scheduled scale-out you may still want; suspending ReplaceUnhealthy stops only the replacement loop. And if a group has been failing to launch instances for more than about 24 hours, AWS applies an administrative suspension on its own. When someone reports that a group "just stopped working", check for suspended processes before checking the policy.

Exam tips

• A scenario where the group refuses to grow and the policy looks correct is a maximum capacity question. Read the 3 numbers before reading the policy.
• "The load balancer shows the target unhealthy but the instance is never replaced" is always the health check type. The group defaults to EC2 status checks only.
• "Instances are terminated and relaunched in a loop right after launch" is the health check grace period, and the give-away is a group created by CLI or CloudFormation, where the default is 0 rather than the console's 300.
• Do not mix up the 2 timers. Health check grace period is how long before health checks can kill a new instance. Default cooldown, 300 seconds, is a pause between simple scaling activities and belongs to the next lesson.
• "A newer instance was terminated before an older one" means the zones were unbalanced. Zonal balance outranks the termination policy.
• Anything about Spot plus On-Demand in one group, multiple instance types, or instance refresh rules out launch configurations. The answer needs a launch template.
• Scale-in protection blocks scale-in only. If the stem says health check, Spot interruption, or a person clicking Terminate in the console, protection is the wrong answer.

The model to carry into the next lesson is narrow and load-bearing: the group has one dial it moves, desired capacity, and a set of rules for how instances get created, judged, and chosen for removal around it. Everything that follows, target tracking, step scaling, predictive scaling, is a different way of deciding what that one number should be.