Arrival Is Not Trust
A logistics company receives nightly shipment files from business partners. The partners already use SFTP, so forcing them to adopt S3 APIs would create unnecessary integration work. Running an SFTP fleet on EC2 would preserve the protocol, but the company would own patching, availability, user keys, scaling, and storage integration.
AWS Transfer Family provides the managed SFTP endpoint and stores each upload in Amazon S3. That solves protocol compatibility and receipt of the bytes. It does not prove that a file has the expected name, schema, record count, business contents, or security classification.
The architecture is therefore a controlled trust transition:
authenticate -> land as untrusted -> buffer -> validate once -> promote or quarantine
The Secure Ingest Path
- Transfer Family authenticates the partner. An identity provider verifies the password or SSH key. A logical directory can make Partner A see only its folder-like view, but it is not the security boundary. Partner A's IAM role and the S3 bucket policy must authorize only keys under its own prefix.
- S3 stores the upload privately in an incoming zone. S3 Block Public Access remains enabled. The incoming object is durable but untrusted. SFTP protects the network transfer; S3 encryption protects the stored object. With customer-managed SSE-KMS, every writer, processor, scanner, and recovery role also needs the appropriate KMS key permission—not merely S3 access.
- S3 announces the arrival to SQS. A filtered
ObjectCreatednotification forincoming/sends a small work ticket containing the bucket, object key, version, and safe metadata. The file bytes remain in S3. Filtering prevents processed output from triggering the ingest pipeline again. - SQS absorbs bursts and processor outages. It holds and retries work until compute is available. Lambda may fit short, bounded validation; ECS on Fargate, AWS Batch, or Glue may fit large archives or longer processing. The event source does not dictate the compute service.
- The processor claims the exact work idempotently. S3 notifications and SQS delivery can repeat. A durable conditional record—often in DynamoDB—lets one worker claim
bucket + key + object version + processor version. A duplicate delivery observes the existing claim and cannot publish a second result; after completion, it can reuse the recorded outcome. A new S3 version or intentionally updated validator remains new work. - The processor performs the trust checks. It reads the object with its own S3 and KMS permissions, then checks the filename, expected partner, integrity evidence, schema, row data, and required security rules.
- The result receives an explicit state. Passing data is written to a processed zone that downstream analytics may trust. A completed inspection that finds invalid data sends the file to quarantine. If infrastructure prevents inspection from finishing, SQS retries; repeated operational failure moves the message to a dead-letter queue (DLQ), while the source file remains untrusted.
Boundaries That Prevent Wrong Designs
- Logical directory vs authorization: the directory mapping controls what the SFTP client sees; IAM and bucket policies control what S3 operations are actually permitted.
- SQS vs idempotency: SQS preserves and retries work. It does not stop repeated business effects; the processor's atomic claim does.
- Quarantine vs DLQ: quarantine means inspection completed and rejected the data. A DLQ means processing could not complete after retries. One is a data-trust result; the other is operational failure isolation.
- Macie vs malware scanning: Amazon Macie discovers sensitive data and S3 security risk. It is not antivirus and is not automatically a synchronous promotion gate. A malware or mandatory inline classification requirement needs an explicit control with defined failure behavior.
- Transfer encryption vs storage encryption: SFTP protects data in transit. S3 server-side encryption protects data at rest. Customer-managed KMS keys add a separate authorization boundary for key use.
SAA Recognition Signals And Traps
- “Partners keep using SFTP while files land in S3” points to AWS Transfer Family because it supplies the managed protocol endpoint. DataSync instead moves datasets through managed transfer tasks; Storage Gateway exposes supported storage protocols to on-premises environments.
- “Prevent partners from seeing one another's uploads” requires per-partner identity mapping plus scoped IAM and bucket policies. A folder-like view alone is not authorization.
- “Process S3 arrivals reliably through bursts” points to S3 notification → SQS → processor because the queue buffers work and supports retry isolation.
- “The same arrival event may be delivered twice” requires an idempotent consumer because at-least-once delivery can repeat. A queue alone does not make the result duplicate-safe.
- “Customer-controlled encryption key” requires SSE-KMS plus coordinated IAM and key-policy permissions for every actor that must use the object.
- “Discover PII in S3” points to Macie; “scan an uploaded file for malware” requires a separate malware-scanning workflow.
- Never let downstream consumers read directly from incoming, assume Lambda fits every file size, or treat a successful upload as a successful business validation.
One-Minute Review
partner SFTP client
-> Transfer Family authenticates and maps a scoped S3 role
-> private encrypted S3 incoming zone holds untrusted bytes
-> filtered object event places a reference in SQS
-> processor claims the object-processing version and validates it
-> pass: processed zone; policy failure: quarantine
-> unfinished operational failure: retry, then DLQ
If you remember only one thing: a successful transfer proves that bytes arrived; the pipeline must still establish authorization, duplicate safety, and data trust before downstream use.