CloudTrail's IsLogging Flag Stays True While S3 Delivery Silently Fails

Symptom

A CloudTrail trail reports itself as healthy while a defined window of API activity never reaches the delivered log files. The trail exists, the console shows "Logging: On", and aws cloudtrail describe-trails lists it without error. Only when someone goes looking for a specific event — during an audit, a permissions review, or an incident timeline reconstruction — does the gap surface: the expected object is not in the destination S3 bucket for that period, or an entire day’s worth of logs is missing.

• The trail is enabled and appears in describe-trails.
• The console toggle and the CLI both show logging as on.
• No alert, error banner or failed deployment pointed at the problem.

False Assumption

The team treats a logging-enabled state as proof that events are being delivered. In practice, the flag that shows a trail is "on" describes CloudTrail’s own configuration state, not whether each batch of log files successfully reached the destination bucket. Delivery is a separate downstream step, gated by the bucket policy and, when the bucket uses server-side encryption with a customer-managed KMS key, by the key policy as well. A trail can be fully configured and still fail to write a single object if either of those policies does not authorise the CloudTrail service principal.

Root Cause

Delivery fails silently because the destination bucket policy, or the KMS key policy protecting an SSE-KMS-encrypted bucket, no longer contains the specific statement that lets the CloudTrail service principal write log objects, and CloudTrail has no built-in mechanism that forces a trail into a visibly failed state when that happens. In practice this gap tends to appear through one of a few routes: a bucket policy trimmed during a security hardening pass that removed a service-principal statement someone believed was unused; a bucket reused across trails or accounts without updating the account ID or trail ARN condition referenced in the policy; a KMS key rotated or replaced without carrying its permission grant for CloudTrail over to the new key; or a change to bucket ownership settings that quietly invalidates the ACL condition CloudTrail’s writes depend on.

Impact

The organisation operates with a fabricated sense of audit completeness, so evidence relied on for compliance attestations, incident timelines and access reviews can have undetected holes. Consequences that follow from this include compliance claims of continuous logging that do not hold for the affected window, incident responders who cannot reconstruct exactly what a compromised credential did during the gap, and detection tooling such as GuardDuty or a SIEM correlation rule that depends on CloudTrail receiving no events to analyse — and therefore staying silent without raising its own failure signal.

Diagnosis

Start from the trail’s own delivery status rather than its on/off flag.

aws cloudtrail get-trail-status --name <trail-name>

Review the response fields that report delivery outcome and last delivery time alongside the logging flag itself. Field names and exact behaviour have been stable across recent CloudTrail API versions, but confirm the current names against the live AWS CloudTrail API Reference before scripting monitoring against them, since this detail was not independently re-verified for this article. Cross-check by listing the most recent objects under the trail’s S3 prefix and comparing the newest timestamp against the current time and CloudTrail’s expected delivery interval; a stale newest object is a strong independent signal of a stalled pipeline. Then inspect the bucket policy, and the KMS key policy if SSE-KMS is in use, for the CloudTrail service-principal statement.

Correction

Restore the exact bucket policy statement, and matching KMS key policy statement if applicable, that authorises the CloudTrail service principal to deliver logs, then confirm delivery resumes before treating the trap as closed. Capture the existing policy first, apply a narrowly scoped correction, and re-run the diagnosis commands afterwards.

{
  "Sid": "AWSCloudTrailWrite",
  "Effect": "Allow",
  "Principal": {"Service": "cloudtrail.amazonaws.com"},
  "Action": "s3:PutObject",
  "Resource": "arn:aws:s3:::<trail-bucket>/AWSLogs/<account-id>/*",
  "Condition": {"StringEquals": {"s3:x-amz-acl": "bucket-owner-full-control"}}
}

Scope the statement to the specific account ID and trail ARN rather than a broad principal, and, where SSE-KMS is used, add the corresponding grant for the CloudTrail service principal on the key policy rather than widening key access generally.

Validation

Treat the fix as unverified until a full delivery cycle actually lands in the bucket after the change, not merely until the error field goes quiet. Re-run get-trail-status after one delivery interval and confirm no outstanding delivery error and an advancing delivery timestamp. List the trail’s S3 prefix again and confirm at least one new object with a timestamp after the change. Finally, issue a low-risk marker call such as aws sts get-caller-identity immediately after the fix and confirm that event appears in the next delivered log file, correlated by event time.

Rollback

If delivery still does not resume, or the corrected policy breaks an unrelated access pattern, restore the exact pre-change bucket policy and KMS key policy from the backups captured before editing, then re-open diagnosis rather than layering further changes on an unverified state.

• Restore the bucket policy: aws s3api put-bucket-policy --bucket <trail-bucket> --policy file://backup-bucket-policy.json
• Restore the key policy, if it was touched: aws kms put-key-policy --key-id <key-id> --policy-name default --policy file://backup-key-policy.json
• Re-run get-trail-status to confirm the trail’s reported state matches the pre-change baseline.

Do not disable the trail, delete the trail, or delete the destination bucket at any point during diagnosis, correction or rollback; any of those actions destroys the audit evidence the investigation depends on and cannot be undone.

Prevention

The next safe decision is to stop relying on the logging flag as a health signal at all.

• Add a scheduled, read-only check that reads the trail’s delivery status fields on a fixed interval and alerts on a stale delivery timestamp, rather than relying on manual review of the on/off flag.
• Treat any edit to a CloudTrail destination bucket policy or its KMS key policy as a change that requires an explicit CloudTrail delivery validation step, not a generic S3 access review.
• Encode the required service-principal statement in infrastructure as code so a manual policy edit or hardening pass cannot silently drop it without a diff being visible.
• Whenever a destination bucket’s KMS key is rotated or replaced, make a CloudTrail permission check part of the rotation’s own completion criteria, not an afterthought discovered later.