KBY Technologies Logo
runbooks/aws-multi-cloud/aws-drs-replication-agent-crash.md
HIGH SEVERITYAWS & Multi-CloudAmazon Web Services

AWS DRS Replication Agent Crash or Agent Not Seen

Severity
HIGH
Target Time
30 minutes to restore agent communication or escalate safely
DomainAWS & Multi-Cloud
Verified
Overview

Diagnose an AWS Elastic Disaster Recovery replication agent crash or AGENT_NOT_SEEN state by checking the source service, agent logs, regional endpoint access and TCP 1500 path before any restart or reinstall that could trigger a rescan and affect RPO.

Share

01 // Diagnose

Symptom

Incident signalWhat responders observe

An AWS DRS source server shows Disconnected, Stalled, Agent not seen or a replication communication error. The local AWS Replication Agent service may be stopped, repeatedly failing, unable to authenticate to the regional DRS endpoint on TCP 443, or unable to reach the staging replication server on TCP 1500.

Detection Signature

Detection evidenceMetrics, logs, and confirmation commands

  1. Record sourceServerID, account, target region, dataReplicationState, dataReplicationError and lag before touching the source host.

  2. Check the AWS Replication Agent service and agent.log.0 on the affected operating system.

  3. Correlate ERROR, FATAL, connection, certificate and timeout entries with host reboots, kernel updates, security changes and network events.

  4. Verify TCP 443 from the source to the regional DRS service and TCP 1500 from the source to the staging replication server.

  5. Distinguish a stopped agent from network, IAM, TLS, time-skew, kernel-driver or staging-area failures; reinstall is not the first diagnostic step.

Root Cause Analysis

Causal chainWhy the incident occurred

  1. The aws-replication-agent or AwsReplicationService process stopped or failed after a host, kernel or security change.

  2. Egress TCP 443, DNS, proxy trust or system-time drift prevented service authentication.

  3. Security groups, network ACLs or routing blocked TCP 1500 to the staging replication server.

  4. The driver or supported-operating-system prerequisites no longer matched after an update.

  5. A changed MAC address or agent lifecycle action caused DRS to lose source identity and require a new replication cycle.

  6. The staging replication software or regional service path failed while the source agent remained healthy.

02 // Contain & Prevent

Blast Radius

  • Replication lag grows and recent recovery points can fall outside the intended RPO while the agent is disconnected.

  • A rescan can temporarily prevent point-in-time recovery newer than the pre-rescan recovery point.

  • Uninstall, disconnect or source-identity changes can remove replication resources or require replication from the beginning.

Prevention Measures

Prevent recurrenceControls and architectural guardrails

  • Monitor DRS dataReplicationState, lag and agent-seen status alongside the local service.

  • Validate DRS endpoints, TCP 443 and TCP 1500 after firewall, proxy, subnet or security-group changes.

  • Check supported kernels and operating systems before patching protected sources.

  • Keep the AWS diagnostic tool and agent log collection process ready for support.

  • Treat reinstall, hard reboot, disk or MAC-address changes as RPO-affecting changes and link the AWS EC2 health-check workflow in the recovery record.

03 // Fix & Intervention

Pre-Flight Checks

Change gateChecks required before intervention

  1. Confirm this is the source server, not a recovery instance or failback client, and record its sourceServerID.

  2. Capture the DRS state, lag, replication error, service state, logs and agent version.

  3. Confirm no drill, recovery or failback is active for the server.

  4. Obtain the DR owner change approval before starting a stopped service or rebooting.

  5. Do not uninstall, disconnect, delete or reinstall the agent unless AWS guidance and the RPO impact are accepted.

Execution CommandsCOMMANDS

# AWS control-plane state (read-only).
aws drs describe-source-servers --filters "sourceServerIDs=$SOURCE_SERVER_ID" --region "$AWS_REGION" --query "items[0].dataReplicationInfo"

# Linux inspection (read-only).
systemctl status aws-replication-agent
tail -100 /var/lib/aws-replication-agent/agent.log.0
grep -i "ERROR\|FATAL" /var/lib/aws-replication-agent/agent.log.0
cat /var/lib/aws-replication-agent/agent.version

# Windows PowerShell inspection (read-only).
Get-Service -Name AwsReplicationService
Get-Content "C:\Program Files (x86)\AWS Replication Agent\agent.log.0" -Tail 100
Select-String -Pattern "ERROR|FATAL" -Path "C:\Program Files (x86)\AWS Replication Agent\agent.log.0"

04 // Verify & Recover

Verification Steps

Recovery proofEvidence required before closure

  1. Confirm the agent service is running and remains stable for at least 15 consecutive minutes.

  2. Verify the DRS console or DescribeSourceServers shows the agent communicating and dataReplicationState progressing toward a healthy state.

  3. Confirm replication lag decreases to the environment baseline and no new ERROR or FATAL log entries appear.

  4. Verify the latest recovery point advances after communication resumes.

  5. Run a scheduled recovery drill later; do not convert an active incident into an unplanned failover test.

Rollback Protocol

Safe reversal path

  1. If an approved service start immediately destabilises the source, capture the new logs and return the service to its recorded pre-change state while preserving the existing DRS registration.

  2. Revert only the network or host configuration changed during this incident.

  3. Do not use uninstall, disconnect or delete as rollback

  4. escalate before any action that can force a rescan or full re-replication.

Escalation

Conditions requiring additional ownership

  • The service repeatedly crashes, the driver fails to load, or the operating system/kernel is unsupported.

  • Replication remains disconnected after local service and network paths are healthy.

  • A restart, reboot, reinstall, MAC-address correction or source disconnect is proposed.

  • Lag threatens the agreed RPO or no current recovery point is available.

  • Escalate to the DR owner and AWS Support with sourceServerID, region, timestamps, agent version, logs and diagnostic-tool output.

Authoritative Sources

Authoritative · Amazon Web Services

Replication errors: agent communication

Defines AGENT_NOT_SEEN and the TCP 443/TCP 1500 communication checks and CLI state query.

Authoritative · Amazon Web Services

Agent logs and diagnostics

Verifies Linux and Windows service names, log paths and diagnostic commands.

Authoritative · Amazon Web Services

AWS DRS agent related FAQ

Documents agent log locations and the RPO impact of rescans.

Authoritative · Amazon Web Services

Installation requirements for AWS Replication Agent

Documents conditions such as MAC-address changes that can stop replication and require reinstallation from the beginning.

Related Runbooks

HIGH35 minutes to identify the timed-out state and recovery path

AWS Step Functions Execution Timeout: Diagnose States.Timeout

Diagnose an AWS Step Functions States.Timeout by separating workflow, task, heartbeat, HTTP Task and downstream-service limits, then change retry, catch or timeout policy only after duplicate side effects and execution type are understood.