Systems Manager Fleet Management

You have 400 EC2 instances and a question that takes 10 seconds to answer on any one of them: which version of the agent is installed? The SSH answer costs you a bastion host, a key distribution problem, an inbound port on every security group, and a shell script wrapped around 400 hostnames with no record of what ran where. Systems Manager exists so that question costs one API call and produces an audit trail.

The previous topic ended with shipping a new version of your software. This topic is about the software already running: reaching it, keeping it consistent, and automating the parts of your week that are the same every week.

The three conditions that make a node manageable

Every Systems Manager tool in this domain (Session Manager, Run Command, Patch Manager, State Manager) works on managed nodes, and nothing works until an instance becomes one. Three conditions must all hold.

SSM Agent must be installed and running on a supported operating system. Most AWS-provided AMIs ship with it preinstalled, which is why this condition usually passes without you doing anything and why it is easy to forget when someone brings a custom image.

The node must have credentials that let it call the Systems Manager API. On EC2 that means an IAM instance profile or the account-level alternative covered in the next section. On an on-premises server or VM it means an IAM service role and a hybrid activation.

The agent must be able to reach a Systems Manager endpoint on port 443 to register itself. After registration, the service checks the node's health with a signal every 5 minutes.

Hold onto the order, because it is also the diagnostic order later in this lesson. Note the direction of the connection too: SSM Agent initiates every connection outbound. You never open an inbound port for Systems Manager, which is the single fact that makes this whole toolset a security improvement over SSH rather than a convenience layer on top of it.

Two ways to give an instance credentials

The classic approach is an IAM instance profile carrying the AmazonSSMManagedInstanceCore managed policy. It is per instance, it is explicit, and it is what a question about a single misconfigured node almost always turns on.

Default Host Management Configuration is the account-level alternative, and AWS recommends it where the use case allows. Turn it on and every instance in that account and Region that runs Instance Metadata Service Version 2 with SSM Agent 3.2.582.0 or later becomes a managed instance automatically, with no instance profile at all. It uses a service role named AWSSystemsManagerDefaultEC2InstanceManagementRole carrying the AmazonSSMManagedEC2InstanceDefaultPolicy policy.

Three details decide questions here.

It is per Region. Turning it on in eu-west-1 does nothing for us-east-1. Teams discover this when half their fleet is managed and half is not.

An instance profile wins. SSM Agent tries instance profile permissions before Default Host Management Configuration permissions, so an old instance profile that allows ssm:UpdateInstanceInformation keeps that instance on the old path and the account-level role never gets used. Before you turn the feature on, remove that permission from existing instance profiles.

Propagation is not instant. After you turn it on, instances can take up to 30 minutes to pick up the new role's credentials.

Session Manager: a shell with no inbound port

You need a shell on a production instance at 2 in the morning. Session Manager gives you one from the console or the CLI, with no open inbound port, no SSH key, and no bastion host.

The connection is a bidirectional channel between your client and SSM Agent. Traffic is encrypted with TLS 1.2, requests to open the channel are signed with Sigv4, and you can layer a KMS key on top to encrypt the session data beyond the default TLS encryption. Access is granted entirely through IAM policy, which is what makes "give the on-call engineer production access for the length of their rotation" a policy change rather than a key rotation exercise.

Two capabilities beyond an interactive shell are worth naming:

Port forwarding redirects a port inside the node to a local port on your machine. A database listening on 5432 inside a private subnet becomes localhost:9999 on your laptop, with no VPN and no public IP.

Configurable shell profiles let you set the shell, environment variables, working directory, and startup commands for every session, which is how you make sessions land somewhere predictable.

For auditing, sessions can stream to a CloudWatch Logs log group or an S3 bucket, with or without your own KMS key, and CloudTrail records the API calls that started them. An EventBridge rule on session start and end can push a notification to SNS.

Here is the misconception that survives most first readings of this feature. Session logging does not cover port forwarding or SSH sessions. In those modes Session Manager is only a tunnel; SSH encrypts everything inside the TLS connection, so the service has nothing to record. If a scenario requires a transcript of commands, the answer involves interactive shell sessions, not port forwarding.

Run Command: one action across the fleet

Session Manager is for one node and a human. Run Command is for many nodes and one document, and it costs nothing extra.

You pick a Command-type SSM document (AWS-RunShellScript, AWS-RunPowerShellScript, AWS-RunPatchBaseline, and so on), pick targets, and set the rate controls.

Targets come in four forms:

# by instance ID
--targets Key=instanceids,Values=i-02573cafcfEXAMPLE,i-0471e04240EXAMPLE

# by tag
--targets Key=tag:Environment,Values=Production

# by resource group name (maximum one per command)
--targets Key=resource-groups:Name,Values=web-tier

# by resource type inside resource groups (maximum five types)
--targets Key=resource-groups:ResourceTypeFilters,Values=AWS::EC2::Instance

Two rules about tag targeting decide questions. Multiple Key criteria are combined with AND, so Key=tag:Department,Values=Finance Key=tag:ServerRole,Values=Database hits only nodes carrying both. And an array of targets holds a maximum of 5 keys with 5 values each.

Now the rate controls, which are the same pair you met on Automation runbooks:

• --max-concurrency is how many nodes run the command at once, as a number or a percentage. The default is 50. Delivery ramps up: the command goes to one node, waits for acknowledgement, then two more, then grows exponentially until it reaches the limit.
• --max-errors is how many failures are tolerated before Systems Manager stops sending to more nodes. The default is 0, which means the first failure stops further dispatch.

Walk the arithmetic AWS gives for this. Send a command to 50 nodes with --max-errors 10%: the threshold is 5, so the system stops sending when the sixth error arrives. Invocations already in flight are allowed to finish, and some of them may fail too. If a scenario demands that no more than N nodes ever fail, --max-errors N alone is not enough; you also need --max-concurrency 1 so invocations proceed one at a time.

aws ssm send-command \
  --document-name "AWS-RunShellScript" \
  --targets Key=tag:Environment,Values=Production \
  --parameters 'commands=["systemctl restart nginx"]' \
  --max-concurrency 10 \
  --max-errors 1 \
  --output-s3-bucket-name ops-command-output \
  --service-role-arn arn:aws:iam::111122223333:role/SSMRunCommandNotifications \
  --notification-config NotificationArn=arn:aws:sns:eu-west-1:111122223333:ops-alerts,NotificationEvents=Failed,NotificationType=Command

Two operational notes on output. Command history is available for up to 30 days, so anything you need to keep longer goes to S3 or CloudWatch Logs, which is also how you get past the console's truncated view of long output. And never pass a secret in a command's plaintext parameters: all Systems Manager API activity is logged, so anyone with access to those logs can read it. Use a SecureString parameter instead, which is the subject of a later lesson in this topic.

Fleet Manager and Inventory: seeing what you have

Fleet Manager is the console over all of this. It shows which managed nodes are running or stopped, and it lets you do systems administration work without opening a session at all: browse the file system and read file contents, manage the Windows registry, manage operating system user accounts and groups, view running processes, view log files on the node, connect to a Windows instance over RDP, and manage the EBS volumes attached to an instance. Every one of those actions is gated by IAM, so you can grant a support team log reading without granting them a shell.

Inventory answers the fleet-wide questions. It collects metadata on a schedule: applications and versions, AWS components, files, network configuration, Windows updates, instance details, services, tags, Windows registry keys, Windows roles, and any custom inventory you drop on the node as a JSON file. The shortest collection interval is every 30 minutes, so treat it as a periodic snapshot rather than a live feed.

Inventory only collects metadata. It does not read your data.

The piece that turns Inventory into a real reporting tool is a resource data sync: point every account and Region at one S3 bucket, then query the aggregated data with Athena. That is the difference between "which nodes in this Region run OpenSSL 1.0" and "which nodes anywhere in the organization run OpenSSL 1.0."

When a node does not appear in the list

You confirmed the instance is running. It is not in the managed node list. Work the three conditions in order, because that order finds the cause with the least work.

Start with the fast check from the node itself. SSM Agent 3.1.501.0 and later ships a standalone tool:

ssm-cli get-diagnostics --output table

On Windows Server, run ssm-cli.exe get-diagnostics --output table from C:\Program Files\Amazon\SSM. It returns one row per check with a Success, Failed, or Skipped status, and the failing row names the condition.

Map the rows to causes:

From the API side, aws ssm describe-instance-associations-status --instance-id i-02573cafcfEXAMPLE returns an empty InstanceAssociationStatusInfos array until registration succeeds, so an empty result after 5 minutes is itself the signal.

For the connectivity condition, an instance either reaches the public endpoints over an outbound HTTPS route or reaches interface VPC endpoints instead:

Two more causes worth carrying, because they look like bugs. A node whose status has been Connection Lost for at least 30 days may drop off the Fleet Manager list entirely until the underlying problem is fixed. And every managed node needs a TLS certificate from Amazon Trust Services in its trust store; a custom or on-premises image without one fails with an SSL error where you would expect an UnknownOperationException from curl -L https://ssm.region.amazonaws.com.

Exam tips

• Any "the instance is running but does not appear in Systems Manager" question is testing the three conditions. Read the stem for which one it removed: no agent, no instance profile, or no route to the endpoints.
• A private subnet with no NAT gateway plus a Session Manager requirement means interface VPC endpoints, and ssmmessages is the one that carries the session.
• Default Host Management Configuration is per account and per Region, needs IMDSv2, and loses to an instance profile that allows ssm:UpdateInstanceInformation.
• "No inbound ports, no bastion host, no SSH keys, and an audit trail" is the phrase that points at Session Manager. If the same question also demands a command transcript, remember that port forwarding and SSH sessions are not logged.
• Run Command defaults: concurrency 50, error threshold 0. A question where a single failure stopped a fleet-wide command has not set --max-errors.
• To bound total failures rather than just stop dispatch, pair --max-errors with --max-concurrency 1.
• Command history lasts 30 days. Any requirement to retain output longer means S3 or CloudWatch Logs.
• Inventory is metadata on a schedule with a 30-minute floor, and a resource data sync into S3 is what makes it queryable across accounts and Regions with Athena.

The rule to carry out of this lesson: on AWS, fleet access is an IAM problem, not a network problem. Once the three conditions hold, granting or revoking access to a server is a policy edit rather than a key rotation, and every action leaves a record. The next lesson uses that same managed node foundation to keep those servers patched and their configuration from drifting.