CloudFormation StackSets

A security team needs an IAM read-only role, a Config recorder, and a log-shipping rule in 40 accounts across 4 Regions. That is 160 stacks. The scripted version is a loop that assumes a role in each account and calls create-stack, which works until account 23 hits a service quota, the loop keeps going, and nobody notices for three weeks that four accounts have no Config recorder.

The problem is not the loop. It is that the loop has no memory. Nothing records that those 160 stacks are supposed to be the same thing, so nothing can tell you which ones are current, which ones failed, and which ones someone edited by hand.

StackSets is that memory. You define the template once, name the accounts and Regions, and CloudFormation tracks every resulting stack as part of one managed set.

Stack set, stack instance, stack

Three words that sound interchangeable and are not.

A stack set is the container: one template, one set of parameters, and the deployment configuration. It lives in the administrator account, and it is a Regional resource. Create a stack set in eu-west-1 and you will not see it when the console is switched to us-east-1, which surprises people who assume a multi-Region feature must itself be global.

A stack instance is a reference to a stack in one target account within one Region. Three accounts across two Regions gives you 6 stack instances, and each one carries its own status.

A stack is the ordinary CloudFormation stack that actually holds resources, sitting in the target account.

The split between instance and stack looks like bookkeeping pedantry until the first failure. A stack instance can exist without a stack: if the create failed, there is no stack, but the instance survives and holds the reason. That is what makes a failed rollout diagnosable instead of invisible.

aws cloudformation list-stack-instances --stack-set-name org-baseline \
  --filters Name=DETAILED_STATUS,Values=FAILED

Two permission models, one real choice

Deploying into another account means assuming a role there. StackSets gives you two ways to arrange that, and the exam tests which one a scenario forces.

Self-managed permissions means you build the trust chain yourself, with two roles whose names matter:

• AWSCloudFormationStackSetAdministrationRole in the administrator account. Its trust policy allows cloudformation.amazonaws.com to assume it, and its permissions policy allows sts:AssumeRole on the execution role in the targets.
• AWSCloudFormationStackSetExecutionRole in every target account, trusting the administrator account. This role needs full CloudFormation permissions plus permission for everything the template creates.

Use those exact names and StackSets picks them up automatically. Use custom names and every operation has to name them explicitly. The sample templates AWS publishes for these roles grant "Action": "*" on the execution role, which is fine for a first test and wrong for anything permanent. Scope the execution role down to the resource types your template actually creates, because that role is the ceiling on what any stack set operation can do in that account.

Service-managed permissions hands the role problem to CloudFormation. You activate trusted access between CloudFormation and AWS Organizations, and from then on CloudFormation creates and maintains the roles in member accounts for you. You target OUs instead of listing account IDs, and you get automatic deployment.

Two constraints on the service-managed model are worth memorizing because they contradict what people assume.

The management account never receives a stack. Target the whole organization and CloudFormation still skips it. If the management account needs the same baseline, it gets its own stack or its own self-managed stack set.

Delegated administrators cannot be scoped. The management account can register up to 5 member accounts as delegated administrators, so a central platform team can run organization-wide stack sets without holding management account credentials. But a delegated administrator has full reach across every account in the organization. There is no "this team may only deploy to the Sandbox OU" setting.

aws organizations register-delegated-administrator \
  --service-principal member.org.stacksets.cloudformation.amazonaws.com \
  --account-id 444455556666

From the delegated administrator account, every command carries --call-as DELEGATED_ADMIN. Leave it off and the CLI looks for self-managed stack sets in the member account itself, finds nothing, and returns an empty list that looks like a permissions problem.

Choosing what gets a stack

With service-managed permissions, deployment targets are the organization root or a list of OU IDs. Targeting a parent OU automatically includes its children, which is the behavior you want for a security baseline and the behavior that surprises you when a nested sandbox OU picks up a production policy.

By default, every account in a targeted OU receives a stack. Account filters narrow that:

aws cloudformation create-stack-instances --stack-set-name org-baseline \
  --deployment-targets OrganizationalUnitIds=ou-rcuk-1x5j1lwo,Accounts=111122223333,AccountFilterType=DIFFERENCE \
  --regions eu-west-1 us-east-1

DIFFERENCE is the one that earns its place in practice. A baseline applies to the whole Workloads OU except the one legacy account that would break, and you express that exception in the deployment rather than by pulling the account out of its OU.

Concurrency and failure tolerance

Pushing a template to 160 places at once is a good way to break 160 places at once. Four settings control the blast radius, and they interact.

Maximum concurrent accounts caps how many target accounts an operation touches at the same time, as a count or a percentage. Percentages round down: 25 percent of 10 accounts is 2, not 3.

Failure tolerance is the number or percentage of failures allowed per Region before CloudFormation stops. It also rounds down.

Region concurrency is SEQUENTIAL (the default, one Region at a time in your specified deployment order) or PARALLEL (all Regions at once).

Concurrency mode decides what happens to concurrency when failures start. STRICT_FAILURE_TOLERANCE keeps maximum concurrent accounts at no more than failure tolerance plus 1, and slows down as failures accumulate. SOFT_FAILURE_TOLERANCE holds your concurrency level regardless.

Walk one operation through. You deploy to 10 accounts in eu-west-1, us-east-1, and ap-southeast-2, in that deployment order, with failure tolerance 20 percent, maximum concurrent accounts 50 percent, and sequential Regions.

1. 20 percent of 10 rounds down to 2 allowed failures per Region. 50 percent of 10 is 5 accounts at a time.
2. eu-west-1 runs 5 accounts, then the other 5. Two fail. That is exactly the tolerance, so the Region finishes and the operation moves on.
3. us-east-1 runs and a third account fails in that Region. Tolerance is exceeded there, so the operation status becomes FAILED and ap-southeast-2 is cancelled entirely.

Failure tolerance resets per Region. That is what makes sequential deployment a useful safety mechanism: a template that is broken everywhere fails in the first Region and never reaches the rest.

The default settings are deliberately timid, MaxConcurrentCount=1 and FailureToleranceCount=0 in the CLI examples, which means one account at a time and a stop at the first failure. For a first rollout of a new template, that is the right choice. Raise it once the template has proven itself.

Statuses, and the one that lies

Stack set operations report RUNNING, SUCCEEDED, FAILED, QUEUED, STOPPING, or STOPPED. QUEUED shows up with automatic deployment: move an account between OUs and StackSets runs a delete for the old OU's stack and queues a create for the new one.

Stack instances carry their own statuses, and two of them mean work is waiting for you:

• OUTDATED means the stack is not current with the stack set, almost always because a create or update failed there, or because the operation stopped before reaching it.
• INOPERABLE means a delete-instances operation failed and left the stack in an unstable state. Instances in this state are excluded from further stack set updates, so they silently stop receiving your changes. Recovering means deleting the instance with RetainStacks set to true, then cleaning up the stack by hand.

Now the misconception that costs people real coverage. A SUCCEEDED operation does not mean every stack succeeded. It means the failure tolerance was never exceeded. Set failure tolerance to 10 across 10 accounts and an operation in which every single stack fails still returns SUCCEEDED, because you told CloudFormation that many failures were acceptable.

The number you actually want is in the status details:

aws cloudformation describe-stack-set-operation \
  --stack-set-name org-baseline --operation-id 5550e62f-c822-4331-88fa-21c1d7bafc60

{
  "StackSetOperation": {
    "Status": "SUCCEEDED",
    "StatusDetails": { "FailedStackInstancesCount": 3 }
  }
}

SUCCEEDED with a non-zero FailedStackInstancesCount is the signature of a partially deployed baseline. Treat that field, not the operation status, as the definition of done.

Drift across a stack set

Drift detection at the stack set level runs stack-level drift detection on every stack instance and rolls the answers up. One drifted resource makes its stack drifted, one drifted stack makes its instance drifted, and one drifted instance makes the entire stack set DRIFTED.

aws cloudformation detect-stack-set-drift --stack-set-name org-baseline

The operation returns an ID because it is long running, and describe-stack-set-operation reports the counts as it progresses: DriftedStackInstancesCount, InSyncStackInstancesCount, InProgressStackInstancesCount, FailedStackInstancesCount, out of TotalStackInstancesCount. Only one drift detection operation can run on a given stack set at a time, and you can stop it with stop-stack-set-operation.

Two boundaries decide how much a clean result is worth:

• Changes made through CloudFormation are never drift. Update one target account's stack directly to a different template and drift detection still reports IN_SYNC, because the stack matches its own expected configuration. That stack is now inconsistent with its siblings, which is a genuine problem, and drift detection is not the tool that finds it.
• Stack-level detection does not roll up. Run detection on an individual stack in a target account and those results never appear on the StackSets console page. Start detection at the stack set level or the stack set's drift status stays stale.

Parameter overrides are handled correctly: because detection runs per stack, an instance with overridden parameter values is compared against its own overridden expectations, not the stack set default.

When operations fail

The failures cluster into a short list, and the status reason usually names the cause directly.

Retrying is not a special API. You fix the underlying cause, then run an update on the stack set with the same or a corrected template, and the OUTDATED instances catch up.

The quotas that matter at scale: 1,000 stack sets per administrator account, 100,000 stack instances per stack set, and 10,000 stack instance operations running at once per Region per administrator account across all stack sets. Hitting the last one is what makes a large rollout mysteriously slow rather than failed.

Exam tips

• "Deploy the same template to many accounts and Regions from one place" is StackSets. An answer describing a script that loops over accounts is wrong even when it would work.
• "Accounts managed by AWS Organizations" plus "new accounts should get it automatically" is service-managed permissions with automatic deployment. "Accounts outside the organization" or "no organization" forces self-managed.
• AWSCloudFormationStackSetAdministrationRole lives in the administrator account, AWSCloudFormationStackSetExecutionRole in every target. A status reason naming the execution role is a self-managed trust problem, never a trusted access problem.
• The management account never gets a service-managed stack instance. Watch for scenarios where the missing account is the management account.
• Failure tolerance is per Region and rounds down. Exceeding it in one Region cancels the remaining Regions.
• An operation status of SUCCEEDED with missing resources means failure tolerance absorbed the failures. Look at FailedStackInstancesCount.
• Retain stacks keeps the resources and drops the stack set association. It does not delete anything.
• OUTDATED means a failed or skipped operation. INOPERABLE means the instance is excluded from future updates until you delete it.
• Service-managed stack sets do not support nested stacks, macros, or transforms, including AWS::Serverless.
• Drift detection on a stack set never treats a CloudFormation-made change as drift.

Carry one rule out of this lesson: with StackSets, "deployed" is a per-instance fact, not a per-operation one. Before you call a rollout complete, look at the instance statuses and the failed instance count, because the operation status is designed to tell you something narrower than what you want to know.

StackSets pushes identical resources into accounts. The next lesson covers the opposite move: keeping one resource in one account and letting other accounts use it, with AWS RAM.