Multi-Cloud Governance
When to Use
- Operating data infrastructure across 2+ cloud providers
- Preventing vendor lock-in for a critical data platform
- Setting up unified cost visibility across AWS, GCP, and Azure
- Implementing centralized secrets management across clouds
- Designing cross-cloud disaster recovery for data pipelines
Multi-Cloud Architecture Principles
Core principle: Use open formats and standards at every layer
Storage: Parquet / Iceberg (not Snowflake/BigQuery native)
Compute: Spark / Trino (not cloud-specific SQL engines)
Streaming: Kafka protocol (MSK/Event Hubs/Confluent)
Secrets: HashiCorp Vault (not AWS SSM / GCP Secret Manager)
IaC: Terraform (not CloudFormation/ARM)
Containers: OCI/Kubernetes (not Lambda/Cloud Functions)
Observability: OpenTelemetry (not CloudWatch/Cloud Logging)
HashiCorp Vault — Centralized Secrets
# Vault deployment on Kubernetes (multi-cloud compatible)
helm upgrade --install vault hashicorp/vault \
-n vault \
--set server.ha.enabled=true \
--set server.ha.replicas=3 \
--set server.ha.raft.enabled=true \
--set server.ha.raft.setNodeId=true \
--set server.dataStorage.storageClass=gp3
# Configure Vault auth backends per cloud
resource "vault_auth_backend" "aws" {
type = "aws"
}
resource "vault_aws_auth_backend_config" "main" {
backend = vault_auth_backend.aws.path
access_key = var.vault_aws_access_key
secret_key = var.vault_aws_secret_key
}
resource "vault_auth_backend" "gcp" {
type = "gcp"
}
resource "vault_auth_backend" "azure" {
type = "azure"
}
resource "vault_auth_backend" "kubernetes" {
type = "kubernetes" # for all K8s clusters across clouds
}
# Retrieve secrets from Vault (cloud-agnostic)
import hvac
import os
def get_vault_secret(path: str, key: str) -> str:
client = hvac.Client(
url=os.environ["VAULT_ADDR"],
token=os.environ.get("VAULT_TOKEN") or _get_k8s_token()
)
secret = client.secrets.kv.v2.read_secret_version(path=path)
return secret["data"]["data"][key]
db_password = get_vault_secret("data-platform/production/trino", "password")
Cross-Cloud Data Replication
S3 → GCS (Batch Sync)
# Use Storage Transfer Service (managed) or rclone
# Terraform: GCS Transfer Job from S3
resource "google_storage_transfer_job" "s3_to_gcs" {
description = "Daily sync S3 bronze → GCS bronze"
project = var.gcp_project
transfer_spec {
aws_s3_data_source {
bucket_name = "my-company-bronze-prod"
aws_access_key {
access_key_id = var.aws_transfer_key
secret_access_key = var.aws_transfer_secret
}
}
gcs_data_sink {
bucket_name = "my-company-bronze-gcp"
path = "replicated/"
}
transfer_options {
overwrite_objects_already_existing_in_sink = false
delete_objects_from_source_after_transfer = false
}
}
schedule {
schedule_start_date { year = 2024; month = 1; day = 1 }
start_time_of_day { hours = 3; minutes = 0 }
}
}
# rclone cross-cloud sync (works with any S3-compatible + GCS + Azure)
rclone sync \
s3:my-company-bronze-prod/orders/ \
gcs:my-company-bronze-gcp/orders/ \
--transfers 32 \
--checkers 16 \
--s3-region us-east-1 \
--gcs-project-number ${GCP_PROJECT_NUMBER} \
--progress
Open Table Format (Iceberg) — Cloud-Agnostic Tables
-- Iceberg table works with Trino on AWS, GCP, or Azure
-- Just change the catalog's warehouse location
-- AWS: s3://my-bucket/warehouse/
-- GCP: gs://my-bucket/warehouse/
-- Azure: abfss://container@account.dfs.core.windows.net/warehouse/
CREATE TABLE orders.fact_orders (
order_id VARCHAR,
customer_id VARCHAR,
order_date DATE,
amount DECIMAL(18,2)
)
WITH (
format = 'PARQUET',
partitioning = ARRAY['month(order_date)'],
sorted_by = ARRAY['customer_id']
);
Unified Cost Management (FinOps)
# Aggregate costs from multiple cloud providers
import boto3 # AWS Cost Explorer
from google.cloud import billing_v1
from azure.mgmt.costmanagement import CostManagementClient
def get_aws_costs(start: str, end: str) -> dict:
ce = boto3.client("ce", region_name="us-east-1")
response = ce.get_cost_and_usage(
TimePeriod={"Start": start, "End": end},
Granularity="DAILY",
Metrics=["UnblendedCost"],
GroupBy=[{"Type": "DIMENSION", "Key": "SERVICE"}],
)
return {r["Keys"][0]: float(r["Total"]["UnblendedCost"]["Amount"])
for r in response["ResultsByTime"][0]["Groups"]}
# Tag-based cost attribution (consistent tags across clouds)
REQUIRED_TAGS = {
"AWS": {"team": "tag:team", "project": "tag:project"},
"GCP": {"team": "labels.team", "project": "labels.project"},
"Azure": {"team": "tags/team", "project": "tags/project"},
}
Federated Identity (OIDC Cross-Cloud)
# Allow GCP service account to assume AWS role (cross-cloud OIDC)
resource "aws_iam_role" "gcp_to_aws" {
name = "gcp-data-pipeline-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = {
Federated = "accounts.google.com"
}
Action = "sts:AssumeRoleWithWebIdentity"
Condition = {
StringEquals = {
"accounts.google.com:sub" = var.gcp_service_account_id
}
}
}]
})
}
# GCP → AWS credential chain (no static keys)
# GCP service account obtains a token via GCP metadata server
# exchanges it for AWS STS credentials via AssumeRoleWithWebIdentity
OPA Gatekeeper — Unified Policy Across Clusters
# Policy: all containers must have resource limits (applies to any K8s cluster)
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: k8srequiredlimits
spec:
crd:
spec:
names:
kind: K8sRequiredLimits
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8srequiredlimits
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not container.resources.limits.cpu
msg := sprintf("Container '%s' must have CPU limits", [container.name])
}
---
# Deploy same policy to AWS EKS, GCP GKE, and Azure AKS clusters
# via ArgoCD app-of-apps pattern with a shared policy repo
Disaster Recovery Cross-Cloud
RTO (Recovery Time Objective): < 4 hours
RPO (Recovery Point Objective): < 1 hour
DR Strategy:
├── Primary: AWS (US-East-1)
├── Secondary: GCP (US-Central-1) — warm standby
└── Failover trigger: Route 53 health check → DNS failover
Data sync:
- Iceberg tables: replicated hourly to GCS via Storage Transfer
- Kafka: Kafka MirrorMaker 2 (topic replication lag < 5 min)
- Metadata DB: PostgreSQL logical replication to CloudSQL
Compute:
- GKE cluster pre-deployed with same Helm charts
- ArgoCD synced to same GitOps repo
- "Scale to 0" — minimal running cost until failover
Vendor Lock-In Risk Matrix
| Technology | Lock-In Risk | Mitigation |
|---|
| Snowflake native tables | HIGH | Use Iceberg external tables + Parquet |
| BigQuery native storage | HIGH | Export to GCS Parquet + Iceberg |
| AWS Glue crawler | MEDIUM | Use Apache Atlas or OpenLineage instead |
| MSK (Kafka) | LOW | Standard Kafka protocol, migratable |
| Lambda/Cloud Functions | HIGH | Use Kubernetes + containered functions |
| CloudFormation | HIGH | Terraform for all IaC |
| Databricks (Delta) | MEDIUM | Open Delta format, portable |
Anti-Patterns
- Cloud-native ETL tools for core pipelines — AWS Glue/ADF tightly couples pipeline logic to a cloud; use Spark on K8s for portability.
- No tagging standard across clouds — AWS uses
tag:team, GCP uses labels.team, Azure uses tags/team; define a universal tagging spec and enforce via OPA.
- Different secrets backends per cloud — managing AWS SSM + GCP Secret Manager + Azure Key Vault separately triples ops burden; centralize on Vault.
- Assuming multi-cloud = double the cost — idle standby cluster costs < 5% of active; warm standby is affordable; DR costs more than running 24/7.
- No cross-cloud cost visibility — each cloud has its own billing console; without a unified view, FinOps optimization is impossible.
References
- HashiCorp Vault:
vaultproject.io/docs
- FinOps Foundation:
finops.org/framework/
- rclone cross-cloud:
rclone.org/docs/
- OPA Gatekeeper:
open-policy-agent.github.io/gatekeeper/
- Related skills:
[[infra-aws-data-platform-review]], [[infra-gcp-data-platform-review]], [[infra-azure-data-platform-review]], [[infra-terraform-review]]