Secure Partner File Ingest On S3

1. Situation

A logistics company receives a daily shipment file from several external partners. Each partner already runs an automated SFTP job and cannot immediately rewrite it to use the S3 API.

The company wants the files in S3 because downstream analytics and reconciliation systems already use AWS-native storage. The files may contain customer names, addresses, and commercial data.

The business requirements are:

1. partners keep their existing SFTP clients;
2. one partner cannot see another partner's files;
3. files are encrypted and never public;
4. downstream systems do not trust a file merely because it arrived;
5. processing survives bursts and temporary failures;
6. repeated events do not process the same object twice; and
7. operators can tell which expected files arrived, failed, or remain delayed.

The central lesson is:

received bytes != trusted business data

This architecture is a trust transition, not simply a file-copy diagram.

2. Naive Design

Naive option 1: operate an SFTP server on EC2

flowchart LR
  Partner["Partner SFTP client"] --> Server["EC2 SFTP server<br/>users, patches, disks, keys"]
  Server --> Storage["Local or mounted storage"]

This can work, but the team owns host patching, SFTP configuration, high availability, user lifecycle, host keys, logging, scaling, and storage integration.

The partner requires SFTP—not necessarily a server the company operates itself.

Naive option 2: give partners IAM users and broad S3 access

This forces partners to change tooling and creates long-lived AWS credentials. A broad policy can allow one partner to list, read, overwrite, or delete another partner's objects.

Naive option 3: upload directly into the trusted analytics location

partner -> trusted/orders/2026/08/06/orders.csv -> analytics starts

The SFTP transfer completed, but the file may have the wrong name, wrong partner, wrong schema, unexpected row count, corrupted business contents, or prohibited sensitive data. Transfer success is not validation success.

Naive option 4: invoke processing directly without a failure plan

An S3 event invokes one Lambda function for every arrival. Traffic bursts create high concurrency, large files exceed the processing model, duplicate notifications repeat side effects, and failed files have no durable work queue or replay process.

3. What Breaks

Follow five failure stories.

Failure 1: a partner sees the wrong directory

A logical home directory looks isolated in the SFTP client, but the underlying IAM role still allows broad bucket listing. Interface appearance is not an authorization boundary.

The architecture needs both user-to-directory mapping and storage policies scoped to the partner's permitted prefixes.

Failure 2: the same object event arrives twice

S3 Event Notifications are designed for at-least-once delivery and do not guarantee event order. The processor copies the same file twice and triggers duplicate downstream work.

The architecture needs an idempotency record based on stable object identity and processing version.

Failure 3: encryption succeeds at upload but processing cannot decrypt

The bucket uses a customer-managed KMS key. Transfer Family can write the object, but the processor's IAM policy or the KMS key policy does not allow the required decrypt operation.

The architecture needs coordinated IAM, bucket, and key policies for every required actor.

Failure 4: a large archive reaches a small Lambda processor

The file is valid but too large or computationally expensive for the chosen Lambda timeout, memory, or temporary-storage design.

The architecture needs to choose processing compute from file size and work duration, not from the word "event."

Failure 5: Macie finds sensitive data after the file was already trusted

The team treated Macie as an inline malware scanner. Macie later reports personally identifiable information, but the file has already entered broad analytics access.

Macie discovers and classifies sensitive data in S3. Malware detection and synchronous admission control require a separate scanning design.

4. AWS Architecture

Build the ingest path one trust boundary at a time.

Step 1: preserve SFTP with a managed Transfer Family endpoint

Create an AWS Transfer Family server for SFTP backed by S3.

flowchart LR
  Partner["Partner's existing<br/>SFTP client"] -->|"SFTP session"| Transfer["AWS Transfer Family<br/>managed protocol endpoint"]
  Transfer --> S3["Amazon S3<br/>storage backend"]

Transfer Family handles the protocol endpoint so the company does not operate an EC2 SFTP fleet. Choose a public or VPC-hosted endpoint from the partner connectivity and network-control requirement.

Transfer Family can use service-managed users or supported external identity patterns. For each user, map the authenticated identity to:

• an IAM role that controls access to the S3 backend; and
• a home or logical directory that limits what the SFTP client sees.

The directory mapping improves the user experience. The IAM and bucket policies enforce storage authorization.

Step 2: isolate partners in an encrypted incoming zone

Map each partner to a dedicated incoming prefix:

s3://partner-ingest/incoming/partner-a/...
s3://partner-ingest/incoming/partner-b/...

flowchart LR
  PartnerA["Partner A"] --> Transfer["Transfer Family"]
  PartnerB["Partner B"] --> Transfer
  Transfer --> PrefixA["S3 incoming/partner-a/"]
  Transfer --> PrefixB["S3 incoming/partner-b/"]
  RoleA["Partner A IAM role"] -.->|"Only partner-a prefix"| PrefixA
  RoleB["Partner B IAM role"] -.->|"Only partner-b prefix"| PrefixB

Keep S3 Block Public Access enabled. Use least-privilege identity and bucket policies so a partner can perform only the required operations in its mapped prefix.

A prefix is a naming organization, not a security boundary on its own. Stronger isolation requirements may justify separate buckets or accounts, but the lesson's central boundary is enforced permissions.

Use default S3 encryption. Choose SSE-KMS with a customer-managed key when the business requires customer-controlled key policy, separation of duties, audit control, or specific cross-account permissions.

Step 3: turn an object arrival into buffered work

Configure an S3 object-created notification for the incoming prefix and send it to SQS.

flowchart LR
  Incoming["S3 incoming prefix"] -->|"Object-created notification"| Queue["SQS ingest queue<br/>durable work buffer"]
  Queue --> Processor["File processor"]
  Queue -->|"Repeated processing failure"| DLQ["Ingest DLQ"]

The prefix filter matters. If the processor writes to processed/, that output must not match the same incoming notification and create a processing loop.

SQS absorbs bursts and preserves work while the processor is unavailable. The message is a notification about the object, not the file contents; the processor reads the actual object from S3.

Because S3 notifications can be duplicated or arrive out of order, the queue and processor must not assume one perfectly ordered event per object.

Step 4: make file processing idempotent

The processor first claims a stable unit of work, such as the bucket, object key, object version, and processing-rule version. A DynamoDB item or another durable record can enforce that only one completed processing result is recorded for that identity.

bucket + object key + version ID + processor version

Then validate what the business actually cares about:

• expected partner and path;
• filename and allowed extension;
• object size and required metadata;
• checksum or integrity evidence supplied by the workflow;
• schema, columns, and data types;
• row counts and business rules; and
• encryption and classification requirements.

An S3 object-created event means the object write completed from S3's perspective. It does not prove that the partner sent the complete business dataset or correct records.

For short bounded validation, Lambda may fit. For large archives, long transformations, antivirus engines, or heavy data work, use suitable compute such as ECS/Fargate, AWS Batch, Glue, or another processing service. Lambda can remain the coordinator without reading and transforming every byte itself.

Step 5: promote or quarantine explicitly

Use separate trust zones:

flowchart LR
  Incoming["incoming/<br/>untrusted but retained"] --> Validate["Validate and scan"]
  Validate -->|"Pass"| Processed["processed/<br/>trusted for downstream use"]
  Validate -->|"Fail"| Quarantine["quarantine/<br/>isolated for review"]
  Validate -->|"Processor error"| Failed["failed or DLQ<br/>operational retry"]

These outcomes mean different things:

• Quarantine means the file was evaluated and should not be trusted.
• Failed processing means the system could not complete its decision.
• Processed means the defined validation policy passed.

Do not delete the only raw copy until validation, reconciliation, and the retention policy allow it. In S3, a logical "move" is commonly implemented as a copy to the new key followed by deletion of the old key, so make the operation retry-safe.

Downstream analytics should consume the processed zone, not watch the raw incoming area directly.

Step 6: add sensitive-data discovery without confusing its job

Use Amazon Macie when the requirement is to discover sensitive data or monitor S3 data-security risk.

flowchart LR
  S3["S3 objects"] --> Macie["Amazon Macie<br/>sensitive data discovery"]
  Macie --> Finding["Sensitive data or policy finding"]
  Finding --> Review["Security review or EventBridge workflow"]

Macie can identify categories such as personally identifiable or financial data in supported S3 objects. It is not an antivirus engine and should not be described as the component that proves an uploaded executable or archive is malware-free.

If sensitive data must block promotion, design an explicit classification workflow whose timing, supported formats, KMS access, and failure behavior match that requirement. Do not assume automated discovery is a synchronous upload gate.

Step 7: make freshness and audit visible

Monitor the business feed as well as the AWS resources:

• partner authentication and transfer failures;
• expected file arrival by deadline;
• SQS queue depth and age;
• processor errors, duration, and concurrency;
• messages in the ingest DLQ;
• files in quarantine or failed zones;
• KMS and access-denied errors; and
• time from upload to trusted promotion.

CloudTrail records relevant AWS API activity. Transfer Family and processing logs provide operational evidence. S3 Versioning, Object Lock, or dedicated archival controls may enter when retention and immutability requirements justify them.

Completed architecture

flowchart LR
  Partner["Partner SFTP client"] --> Transfer["Transfer Family<br/>managed SFTP endpoint"]
  Transfer --> Incoming["S3 incoming zone<br/>encrypted and partner-scoped"]
  Incoming -->|"Filtered object-created event"| Queue["SQS ingest queue"]
  Queue --> Processor["Idempotent validator<br/>Lambda or suitable compute"]
  Processor -->|"Pass"| Processed["S3 processed zone"]
  Processor -->|"Policy failure"| Quarantine["S3 quarantine zone"]
  Queue -->|"Repeated processor failure"| DLQ["Ingest DLQ"]
  Macie["Macie<br/>sensitive data discovery"] -.->|"Analyzes in-scope S3 data"| Incoming
  KMS["KMS key policy"] -.->|"Authorizes encryption use"| Incoming

5. Request Or Data Flow

Learn three lifecycles: transfer, validation, and failure recovery.

Lifecycle 1: partner upload

1. The partner connects to the Transfer Family SFTP endpoint.
2. Transfer Family authenticates the user through the configured identity source.
3. The user mapping supplies the permitted logical directory and S3 access role.
4. Transfer Family writes the object into that partner's incoming prefix.
5. S3 encrypts the object according to bucket configuration.
6. After object creation completes, S3 sends a notification to the ingest queue.

Lifecycle 2: successful validation

1. The processor receives an SQS message and claims the object identity idempotently.
2. It reads the object using its own IAM and KMS permissions.
3. It validates integrity evidence, naming, schema, data rules, and required security checks.
4. It writes the normalized or approved result into the processed zone.
5. It records successful completion and deletes the SQS message.
6. Downstream systems discover only the trusted output.

Lifecycle 3: operational failure versus quarantine

If the file violates a policy, record the reason and place it in quarantine. That is a completed business decision.

If the processor times out or a dependency is unavailable, do not call the file invalid. Let SQS retry. After the configured receive attempts, isolate the work in the DLQ and alarm an owner.

This distinction prevents temporary infrastructure failures from being mistaken for bad partner data.

6. Security Controls

Isolate partner identities

Use a separate user identity and scoped storage role or session policy per partner pattern. Limit bucket listing and object actions to the intended prefixes. Logical directories should complement, not replace, IAM and bucket policy enforcement.

Keep the bucket private

Enable S3 Block Public Access. Partners enter through the approved Transfer Family endpoint or another explicitly designed path—not a public bucket URL.

Coordinate KMS permissions

With SSE-KMS, consider every actor:

• the Transfer Family user role that writes incoming objects;
• the processing role that decrypts incoming data and encrypts output;
• Macie or scanning services that require access to in-scope objects; and
• administrators or recovery workflows that need controlled access.

An IAM allow alone may be insufficient when the KMS key policy does not allow the principal or delegated permission.

Separate transfer encryption from storage encryption

SFTP encrypts data in transit during the partner session. S3 server-side encryption protects stored objects. They cover different parts of the lifecycle.

Limit sensitive events and logs

Do not place file contents or unnecessary personal data in SQS messages or logs. Store object references, processing identifiers, and safe diagnostic metadata.

7. Resilience Controls

Transfer Family removes the need to operate the protocol server fleet, and S3 provides durable object storage. The ingestion workflow still needs application-level recovery.

Use SQS to retain arrival work during processor outages and to control concurrency against scanners, databases, or partner-specific dependencies.

Make processing idempotent because S3 notifications and SQS delivery can repeat. If the same object key can be overwritten, enable versioning or otherwise include a version-aware identity so a new object is not confused with an old processing record.

Retain raw input until the business retention and replay window closes. A corrected processor should be able to replay a known object without asking the partner to resend it unnecessarily.

Alarm on oldest-message age, DLQ depth, quarantine growth, expected-file freshness, and promotion latency. A queue with no visible errors can still represent a missed daily feed.

Use separate buckets or accounts when blast-radius, legal, or organizational isolation requires a stronger boundary than prefixes.

8. Performance Controls

Transfer speed depends on partner bandwidth, latency, protocol implementation, file size, and endpoint connectivity. A managed endpoint cannot make the partner's network faster.

Many small files create more transfer sessions, S3 requests, notifications, queue messages, and processor invocations than a smaller number of appropriately sized files.

Set processor concurrency from sustainable downstream capacity. SQS can absorb a burst, but excessive Lambda concurrency can overwhelm a malware scanner, database, or external validation API.

Select compute from the file workload:

Organize processed keys for downstream discovery and partition pruning. Do not make analytics jobs scan the entire bucket to find one day's approved files.

9. Cost Controls

Important cost drivers include:

• Transfer Family endpoint uptime and transferred data;
• S3 storage, requests, versioning, and lifecycle;
• KMS API usage;
• SQS requests and Lambda or container processing;
• malware or data-validation tooling;
• Macie sensitive data analysis;
• logs, metrics, and retained DLQ or quarantine objects.

Endpoint cost matters even when no partner is transferring a file. Compare the managed service with the full cost of operating and securing a highly available SFTP fleet, not only an EC2 hourly price.

Use lifecycle policies separately for incoming, processed, quarantine, failed, and audit data. Their retention requirements are not automatically the same.

Filter events to avoid recursive processing loops and unnecessary invocations. Use SQS batching where it improves efficiency without making individual failures hard to isolate.

Scope Macie discovery to the security requirement. Scanning every object continuously may cost more than risk-based automated discovery and targeted jobs.

10. Exam Variants

11. Common Traps

Final Mental Model: One-Minute Review

flowchart LR
  Partner["Partner SFTP client"] --> Transfer["Transfer Family<br/>managed loading dock"]
  Transfer --> Incoming["S3 incoming<br/>untrusted inspection bay"]
  Incoming -->|"Object-created event"| Queue["SQS<br/>inspection tickets"]
  Queue --> Processor["Idempotent validator"]
  Processor -->|"Pass"| Processed["S3 processed<br/>trusted data"]
  Processor -->|"Policy failure"| Quarantine["S3 quarantine"]
  Queue -->|"Operational failure"| DLQ["DLQ"]
  Identity["Partner identity + IAM role"] -.-> Transfer
  KMS["KMS key permissions"] -.-> Incoming
  Macie["Macie<br/>sensitive-data discovery"] -.-> Incoming

The trust path is authenticate → land → buffer → validate → promote or quarantine. Transfer Family proves who delivered the bytes, S3 stores them, SQS preserves processing work, the validator decides trust, and Macie supplies sensitive-data visibility rather than malware protection.

12. Related Topics

Use AWS Transfer Family for protocols, endpoints, identity providers, logical directories, and storage backends. Review Amazon S3 for object storage and bucket controls, AWS Key Management Service for key-policy boundaries, Amazon SQS for buffered processing, and Amazon Macie for sensitive data discovery.

Continue to Analytics Data Lake On S3 after the ingest system can reliably produce trusted, organized data.

Official AWS references:

• What is AWS Transfer Family?
• Managing users for Transfer Family server endpoints
• Amazon S3 Event Notifications
• S3 event notification types and destinations
• Using server-side encryption with AWS KMS keys in Amazon S3
• Monitoring data security and privacy with Amazon Macie