Secure Cross-Account CloudTrail Logging

1. Situation

An attacker obtains an administrator role in a production account, changes a bucket policy, creates access keys, and tries to hide the activity.

The security team needs evidence answering who called which AWS API, from where, when, against what resource, and with what result. That is CloudTrail's core job.

The harder question is who owns the evidence. If the compromised account owns and can delete its only logs, the investigator is asking the suspect to guard the recording.

2. Naive Design

flowchart LR
  Dev["Dev account"] --> DevLogs["Dev-owned bucket"]
  Prod["Prod account"] --> ProdLogs["Prod-owned bucket"]
  New["New account"] -.-> Missing["No trail configured"]

Coverage, event selectors, encryption, and retention drift. A production administrator may be able to alter the production evidence bucket. Another mistake is assuming the CloudTrail Event history view is the organization's durable, centrally retained audit archive.

3. What Breaks

1. A new account is created but no local trail is added.
2. S3 object access is investigated, but only management events were recorded.
3. Central delivery is configured, but the bucket policy rejects the organization prefix.
4. SSE-KMS is enabled, but the key policy does not allow CloudTrail delivery.
5. Logs arrive, but workload admins can delete them and nobody monitors trail status.

“CloudTrail enabled” is therefore incomplete. The design needs coverage, event scope, delivery permission, protected storage, and usable investigation access.

4. AWS Architecture

Step 1: create organization-wide coverage

flowchart LR
  Org["AWS Organizations"] --> Trail["Multi-Region organization trail"]
  A["Account A events"] --> Trail
  B["Account B events"] --> Trail
  New["New member account"] --> Trail

An organization trail logs events for the management account and member accounts. The management account or a CloudTrail delegated administrator can manage it with the required permissions. A multi-Region trail protects against forgetting active Regions.

Step 2: choose event scope deliberately

Management events are the audit baseline. Data events are not universally logged by default and can be high volume, so select high-value resources and use cases intentionally.

Step 3: separate the storage boundary

flowchart LR
  Trail["Organization trail"] -->|"CloudTrail service delivery"| Bucket["S3 bucket<br/>log archive account"]
  Workload["Workload administrators"] -.->|"No delete control"| Bucket
  Reader["Security investigation role"] -->|"Scoped read"| Bucket

The bucket policy is a resource policy. It must allow the CloudTrail service to check the bucket and write to the expected organization/account prefixes while avoiding broad workload access. S3 Block Public Access and TLS-only access remain appropriate.

Step 4: add encryption without breaking delivery

S3 encrypts objects at rest. When requirements call for a customer-managed KMS key, coordinate three permission paths:

1. CloudTrail can use the key for delivery.
2. Authorized investigators can decrypt.
3. Workload principals cannot gain broad decrypt access.

An IAM allow cannot overcome a key policy that does not permit the necessary use.

Step 5: distinguish integrity from retention

flowchart LR
  Logs["CloudTrail log files"] --> Digest["Signed digest files"]
  Digest --> Validate["Validation detects<br/>changed or deleted files"]
  Logs --> Lock["S3 Object Lock<br/>prevents deletion when configured"]

CloudTrail log file validation delivers signed digest files. Validation helps detect modification or deletion after delivery; enabling it does not automatically run validation and does not prevent deletion.

S3 Versioning retains versions. S3 Object Lock supplies stronger retention behavior when configured appropriately. Lifecycle transitions control storage cost but must preserve required retrieval and retention.

Step 6: monitor and expose a controlled investigation path

Alert on trail stop/delete attempts, delivery errors, bucket-policy changes, KMS-key changes, Object Lock changes, and missing expected log delivery. Give analysts read-only or query roles rather than S3 administration.

CloudWatch Logs can support near-real-time metric filters and alarms; S3 remains the durable central archive in this design. Athena or a security analytics system can query archived logs.

Completed architecture

flowchart LR
  Accounts["All organization accounts<br/>and Regions"] --> Trail["Organization trail<br/>selected event types"]
  Trail --> KMS["Customer-managed KMS key<br/>when required"]
  Trail --> S3["Central S3 log archive"]
  S3 --> Protect["Validation + versioning / Object Lock<br/>+ lifecycle"]
  S3 --> Investigate["Scoped Athena / SIEM / analyst access"]
  Monitor["Delivery and configuration alarms"] -.-> Trail
  Monitor -.-> S3

5. Request Or Data Flow

1. A principal calls an AWS API in a member account.
2. CloudTrail evaluates whether the event matches trail selectors.
3. CloudTrail batches and delivers log files to the central S3 prefix.
4. If SSE-KMS is configured, the service uses the permitted key path.
5. Digest files support later integrity validation when enabled.
6. Lifecycle rules retain or transition the objects.
7. During investigation, a scoped security role queries the evidence and correlates it with findings and workload logs.

CloudTrail delivery is not an inline blocker for the API call and is not a zero-delay SIEM feed.

6. Security Controls

• separate log archive administration from workload administration;
• restrict bucket writes to expected CloudTrail delivery conditions and prefixes;
• restrict reads and decrypt permission to approved investigation roles;
• deny insecure transport and keep Block Public Access enabled;
• use validation and immutable retention according to evidence requirements; and
• audit changes to the trail, bucket, key, retention, and Organizations relationship.

7. Resilience Controls

Use an organization, multi-Region trail for broad coverage. Monitor actual delivery rather than assuming configuration equals evidence. Test KMS and bucket-policy changes safely.

If requirements demand regional disaster recovery for audit evidence, evaluate S3 replication and destination key/policy behavior. Maintain break-glass investigation access without granting delete authority.

8. Performance Controls

Collection is managed; analysis becomes the scaling concern. Partition-aware queries should filter account, Region, service, and date. Avoid repeatedly scanning years of logs.

Send only actionable subsets to expensive real-time analytics while retaining the authoritative archive. Select high-volume data events from investigation value, not fear of missing everything.

9. Cost Controls

Cost comes from trail copies and optional features, data-event volume, S3 retention, KMS requests, CloudWatch Logs ingestion, cross-Region copies, and downstream queries or SIEM ingestion.

Apply lifecycle by evidence policy. Narrow Athena columns and partitions. Never reduce cost by allowing critical audit coverage to disappear silently.

10. Exam Variants

11. Common Traps

• Do not store each account's only evidence inside that account.
• Do not assume Event history replaces a persistent organization trail.
• Do not assume every data event is logged automatically.
• Do not confuse validation with prevention or automatic validation execution.
• Do not confuse CloudTrail API evidence with CloudWatch performance metrics.
• Do not forget bucket and KMS resource policies.
• Do not grant investigators delete or policy-administration access merely to query logs.

Final Mental Model: One-Minute Review

flowchart LR
  Activity["Organization API activity"] --> Trail["Organization trail"]
  Trail --> Archive["Separate S3 archive"]
  Archive --> Protect["Encrypt + validate + retain"]
  Protect --> Investigate["Controlled investigation"]

CloudTrail records the activity; the organization trail supplies coverage; the separate archive preserves trustworthy evidence.

12. Related Topics

Review AWS CloudTrail, AWS Organizations, Amazon S3, and Identity Policies vs Resource Policies.

Official references: Creating an organization trail and CloudTrail log file integrity validation.