File Upload & Storage Expert
Design secure, scalable file upload and storage systems with presigned URLs, multipart uploads, CDN distribution, and lifecycle management.
Activation Triggers
Activate on: "file upload", "S3", "presigned URL", "multipart upload", "CDN", "blob storage", "image upload", "R2", "media storage", "upload progress"
NOT for: Data lake storage → lakehouse-architect | Database BLOB columns → dimensional-modeler | Video processing pipelines → relevant media skill
Quick Start
- Choose storage backend — S3, R2 (zero egress), GCS, or Supabase Storage
- Generate presigned URLs server-side — never expose credentials to clients
- Implement multipart upload for files >10MB with resume capability
- Process asynchronously — thumbnails, transcoding, virus scanning via queue
- Serve via CDN — CloudFront, Cloudflare, or Vercel Edge for global delivery
Core Capabilities
| Domain | Technologies |
|---|
| Object Storage | AWS S3, Cloudflare R2, GCS, MinIO |
| Upload Libraries | @aws-sdk/client-s3, tus-js-client, Uppy 4.x |
| CDN | CloudFront, Cloudflare CDN, Fastly |
| Processing | Sharp (images), FFmpeg (video), ClamAV (scanning) |
| Managed | Supabase Storage, Vercel Blob, Uploadthing |
Architecture Patterns
Presigned URL Upload Flow
Client Server S3/R2
│ │ │
├─ POST /upload/init ────→│ │
│ { filename, type } │ │
│ ├─ createPresignedPost() ─→│
│ │ │
│←── { url, fields } ────┤ │
│ │ │
├─ PUT (direct to S3) ──────────────────────────────→│
│ │ │
│ │←── S3 Event Notification│
│ ├─ Process (thumbnail, │
│ │ scan, metadata) │
│←── webhook: ready ─────┤ │
Multipart Upload with Resume
import { S3Client, CreateMultipartUploadCommand,
UploadPartCommand, CompleteMultipartUploadCommand } from '@aws-sdk/client-s3';
const PART_SIZE = 10 * 1024 * 1024; // 10MB parts
async function multipartUpload(file: Buffer, key: string) {
const { UploadId } = await s3.send(
new CreateMultipartUploadCommand({ Bucket: BUCKET, Key: key })
);
const parts: { ETag: string; PartNumber: number }[] = [];
for (let i = 0; i < file.length; i += PART_SIZE) {
const partNumber = Math.floor(i / PART_SIZE) + 1;
const { ETag } = await s3.send(new UploadPartCommand({
Bucket: BUCKET, Key: key, UploadId,
PartNumber: partNumber,
Body: file.subarray(i, i + PART_SIZE),
}));
parts.push({ ETag: ETag!, PartNumber: partNumber });
}
await s3.send(new CompleteMultipartUploadCommand({
Bucket: BUCKET, Key: key, UploadId,
MultipartUpload: { Parts: parts },
}));
}
CDN-First Serving Architecture
User Request → CDN Edge (cache hit?) ──yes──→ Serve cached
│ no
↓
Origin (S3/R2)
↓
Response + Cache-Control: public, max-age=31536000, immutable
(use content-hash in filename for cache busting)
Anti-Patterns
- Proxying uploads through your server — use presigned URLs so clients upload directly to object storage; your server should never be a relay
- Synchronous processing — never resize/transcode during the upload request; push to a queue and process async
- Mutable filenames — use content-addressable keys (
sha256-{hash}.ext) for CDN cacheability
- No upload size limits — enforce max file size server-side in presigned URL policy AND client-side
- Storing credentials client-side — presigned URLs expire; never send AWS keys to the browser
Quality Checklist