AWS Data Platform Review
When to Use
- Designing or reviewing an AWS-native data platform
- Auditing IAM permissions for data pipelines
- Choosing between AWS managed services vs self-managed
- Optimizing S3 data lake costs
- Setting up Lake Formation for fine-grained data access control
S3 Data Lake Architecture
# Terraform: 3-zone data lake (Bronze/Silver/Gold)
resource "aws_s3_bucket" "zones" {
for_each = toset(["bronze", "silver", "gold"])
bucket = "${local.prefix}-data-lake-${each.key}"
tags = merge(local.common_tags, { Zone = each.key })
}
# Encryption with separate KMS keys per zone
resource "aws_s3_bucket_server_side_encryption_configuration" "zones" {
for_each = aws_s3_bucket.zones
bucket = each.value.id
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "aws:kms"
kms_master_key_id = aws_kms_key.zones[each.key].arn
}
bucket_key_enabled = true # reduces KMS API calls by 99%
}
}
# Lifecycle: tiered storage by zone age
resource "aws_s3_bucket_lifecycle_configuration" "bronze" {
bucket = aws_s3_bucket.zones["bronze"].id
rule {
id = "raw-data-tiering"
status = "Enabled"
transition {
days = 30
storage_class = "STANDARD_IA"
}
transition {
days = 90
storage_class = "GLACIER_IR"
}
expiration { days = 365 }
}
}
# Block all public access
resource "aws_s3_bucket_public_access_block" "all" {
for_each = aws_s3_bucket.zones
bucket = each.value.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
IAM — IRSA (IAM Roles for Service Accounts)
# Airflow on EKS: worker pods assume an IAM role without static keys
module "irsa_airflow" {
source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
version = "~> 5.0"
role_name = "${local.prefix}-airflow-worker"
oidc_providers = {
main = {
provider_arn = module.eks.oidc_provider_arn
namespace_service_accounts = ["airflow:airflow-worker"]
}
}
role_policy_arns = {
s3_access = aws_iam_policy.airflow_s3.arn
glue_access = aws_iam_policy.airflow_glue.arn
}
}
# Scoped S3 policy
resource "aws_iam_policy" "airflow_s3" {
name = "${local.prefix}-airflow-s3"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"]
Resource = ["${aws_s3_bucket.zones["bronze"].arn}/*",
"${aws_s3_bucket.zones["silver"].arn}/*"]
},
{
Effect = "Allow"
Action = ["s3:ListBucket"]
Resource = [aws_s3_bucket.zones["bronze"].arn,
aws_s3_bucket.zones["silver"].arn]
}
]
})
}
MSK Kafka Configuration
resource "aws_msk_cluster" "main" {
cluster_name = "${local.prefix}-kafka"
kafka_version = "3.5.1"
number_of_broker_nodes = 3
broker_node_group_info {
instance_type = "kafka.m5.xlarge"
client_subnets = module.vpc.private_subnets
storage_info {
ebs_storage_info {
volume_size = 1000 # GB per broker
provisioned_throughput {
enabled = true
volume_throughput = 250 # MiB/s
}
}
}
security_groups = [aws_security_group.msk.id]
}
encryption_info {
encryption_in_transit {
client_broker = "TLS"
in_cluster = true
}
at_rest_encryption_info {
encryption_at_rest_kms_key_arn = aws_kms_key.msk.arn
}
}
client_authentication {
sasl {
iam = true # SASL/IAM auth — no credentials to manage
}
}
configuration_info {
arn = aws_msk_configuration.main.arn
revision = aws_msk_configuration.main.latest_revision
}
}
resource "aws_msk_configuration" "main" {
kafka_versions = ["3.5.1"]
name = "${local.prefix}-kafka-config"
server_properties = <<-EOF
auto.create.topics.enable=false
log.retention.hours=168
num.partitions=3
default.replication.factor=3
min.insync.replicas=2
compression.type=lz4
EOF
}
Lake Formation — Fine-Grained Access Control
# Register S3 location with Lake Formation
resource "aws_lakeformation_resource" "data_lake" {
arn = aws_s3_bucket.zones["gold"].arn
role_arn = aws_iam_role.lakeformation.arn
}
# Grant table-level access to analytics team
resource "aws_lakeformation_permissions" "analytics_read" {
principal = aws_iam_role.analytics_team.arn
permissions = ["SELECT"]
permissions_with_grant_option = []
table {
database_name = "gold"
name = "fact_orders"
}
}
# Column-level security: hide PII columns from non-privileged role
resource "aws_lakeformation_permissions" "pii_columns" {
principal = aws_iam_role.data_scientist.arn
permissions = ["SELECT"]
table_with_columns {
database_name = "gold"
name = "customers"
column_names = ["customer_id", "order_count", "total_spend"]
# excluded: email, phone, address
}
}
MWAA vs Self-Managed Airflow on EKS
| Aspect | MWAA | EKS Helm Chart |
|---|
| Setup effort | Low (minutes) | High (days) |
| Cost (2 schedulers) | ~$300/mo + workers | ~$150/mo + workers |
| Airflow version | Limited (2-3 supported) | Any version |
| Plugin support | Limited | Unlimited |
| KubernetesExecutor | No | Yes |
| Custom Python deps | S3 requirements.txt | Dockerfile |
| HA | Managed | Manual (2+ schedulers) |
| Scaling | Limited | Full autoscaling |
| Best for | Small teams, quick start | Large teams, custom needs |
Glue vs EMR vs Spark-on-EKS
| Aspect | AWS Glue | Amazon EMR | Spark on EKS |
|---|
| Management | Serverless | Semi-managed | Self-managed |
| Startup time | 1-2 min | 5-10 min | 30-60s (reuse) |
| Cost model | Per DPU-hour | Per EC2-hour | Per pod resource |
| Spot support | Limited | Full | Full |
| Custom jars | Limited | Full | Full |
| Iceberg support | Via Glue catalog | Full | Full |
| Best for | Simple ETL, serverless | Large Spark jobs | Kubernetes platform |
Cost Optimization
# S3 Intelligent-Tiering: auto-tier objects not accessed in 30 days
resource "aws_s3_bucket_intelligent_tiering_configuration" "bronze" {
bucket = aws_s3_bucket.zones["bronze"].id
name = "auto-tiering"
status = "Enabled"
tiering {
access_tier = "DEEP_ARCHIVE_ACCESS"
days = 180
}
}
# EMR spot configuration
resource "aws_emr_instance_fleet" "task" {
cluster_id = aws_emr_cluster.main.id
name = "task-fleet"
target_spot_capacity = 20 # 20 units of spot (80% cheaper)
target_on_demand_capacity = 2
instance_type_configs {
instance_type = "m5.xlarge"
bid_price_as_percentage_of_on_demand_price = 100
ebs_config {
size = 100
type = "gp3"
}
}
}
VPC Networking for Data Platform
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "~> 5.0"
name = "${local.prefix}-data-platform"
cidr = "10.0.0.0/16"
azs = ["us-east-1a", "us-east-1b", "us-east-1c"]
private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
public_subnets = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
enable_nat_gateway = true
single_nat_gateway = false # one per AZ for HA
enable_vpn_gateway = false
# S3 VPC endpoint (free data transfer within VPC)
enable_s3_endpoint = true
# Glue VPC endpoint
enable_glue_endpoint = true
}
Review Checklist
[ ] S3 buckets: encryption (KMS), versioning, public access blocked, lifecycle
[ ] IAM: IRSA for EKS workloads (no static keys), least-privilege policies
[ ] MSK: TLS + IAM auth, min.insync.replicas=2, encryption at rest
[ ] Lake Formation: column-level security for PII tables
[ ] VPC: private subnets for all data services, S3/Glue VPC endpoints
[ ] EKS: node groups in private subnets, IRSA enabled
[ ] CloudTrail: enabled for all regions, S3 bucket with lifecycle
[ ] Cost: S3 Intelligent-Tiering, Spot for EMR/EKS batch, Savings Plans
[ ] Monitoring: CloudWatch → Prometheus metrics via CW exporter
[ ] Backups: automated RDS snapshots (7+ day retention), S3 versioning
Anti-Patterns
- Static AWS credentials in Airflow connections — rotate every 90 days or face locked pipelines; use IRSA on EKS or instance profiles on EC2.
- S3 public access not blocked — one misconfigured bucket policy exposes entire data lake; always set
restrict_public_buckets = true.
- No S3 lifecycle rules on bronze zone — raw data accumulates indefinitely; raw zone grows 10x faster than gold zone.
- Lake Formation disabled with broad S3 bucket policies — no row/column-level control; enable Lake Formation for all gold/silver tables.
- No VPC endpoints for S3/Glue — data transfers via internet gateway incur NAT Gateway data processing costs; S3 VPC endpoint is free.
References
- AWS Lake Formation:
docs.aws.amazon.com/lake-formation/latest/dg/
- MSK best practices:
docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html
- IRSA:
docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
- EMR on EKS:
docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/
- Related skills:
[[terraform-data]], [[infra-kubernetes-security-audit]], [[de-cost-optimization]]