本文へ移動
cccskills
無料GitHub で公開

infra-azure-data-platform-review

Azure data platform review — ADLS Gen2 (hierarchical namespace/RBAC/lifecycle), Azure Synapse vs Databricks vs HDInsight trade-offs, Event Hubs (Kafka protocol compatible), ADF vs Azure Databricks for ETL, Azure Managed Airflow, AKS with Workload Identity (AAD Pod Identity), Key Vault for secrets, Azure Monitor/Log Analytics, Purview for data catalog/lineage, Private Endpoints for data services, cost management (reserved instances/spot VMs)

日本語の概要は準備中です。原文の説明を表示しています。

ivanshamaev/de-agent-skillsスター 22(GitHub 22 + cccskills 0)
連携ツールAzureKubernetesTerraform

実行スコア 未検証

インストール方法を見る

含まれるファイル（1）

  • SKILL.md11.1 KB

SKILL.md（原文）

インストールする前に、エージェントに与えられる指示の中身を確認できます。

Azure Data Platform Review

When to Use

  • Designing or reviewing an Azure-native data platform
  • Auditing ADLS Gen2 access controls and RBAC
  • Choosing between Azure Synapse, Databricks, and HDInsight
  • Setting up Managed Identity for AKS workloads
  • Implementing Azure Purview for data governance

ADLS Gen2 Data Lake

# Terraform: ADLS Gen2 with hierarchical namespace
resource "azurerm_storage_account" "data_lake" {
  name                     = "${replace(local.prefix, "-", "")}datalake"
  resource_group_name      = azurerm_resource_group.main.name
  location                 = var.location
  account_tier             = "Standard"
  account_replication_type = "ZRS"           # zone-redundant storage
  account_kind             = "StorageV2"
  is_hns_enabled           = true            # hierarchical namespace = ADLS Gen2

  blob_properties {
    versioning_enabled  = true
    delete_retention_policy { days = 30 }
  }

  identity {
    type = "SystemAssigned"
  }

  # Encrypt with customer-managed key
  customer_managed_key {
    key_vault_key_id          = azurerm_key_vault_key.storage.id
    user_assigned_identity_id = azurerm_user_assigned_identity.storage.id
  }

  tags = local.common_tags
}

# Hierarchical namespace containers (zones)
resource "azurerm_storage_data_lake_gen2_filesystem" "zones" {
  for_each           = toset(["bronze", "silver", "gold"])
  name               = each.key
  storage_account_id = azurerm_storage_account.data_lake.id
}

# RBAC: Airflow service principal gets Storage Blob Data Contributor on bronze/silver
resource "azurerm_role_assignment" "airflow_bronze" {
  scope                = azurerm_storage_data_lake_gen2_filesystem.zones["bronze"].id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_user_assigned_identity.airflow.principal_id
}

# Lifecycle policy
resource "azurerm_storage_management_policy" "data_lake" {
  storage_account_id = azurerm_storage_account.data_lake.id
  rule {
    name    = "bronze-tiering"
    enabled = true
    filters {
      prefix_match = ["bronze/"]
      blob_types   = ["blockBlob"]
    }
    actions {
      base_blob {
        tier_to_cool_after_days_since_modification_greater_than    = 30
        tier_to_archive_after_days_since_modification_greater_than = 90
        delete_after_days_since_modification_greater_than          = 365
      }
    }
  }
}

AKS Workload Identity

# AKS cluster with OIDC issuer (required for Workload Identity)
resource "azurerm_kubernetes_cluster" "main" {
  name                = "${local.prefix}-aks"
  resource_group_name = azurerm_resource_group.main.name
  location            = var.location
  oidc_issuer_enabled = true
  workload_identity_enabled = true

  identity {
    type = "SystemAssigned"
  }

  default_node_pool {
    name       = "system"
    vm_size    = "Standard_D4s_v3"
    node_count = 3
    vnet_subnet_id = azurerm_subnet.aks.id
  }
}

# Managed Identity for Airflow
resource "azurerm_user_assigned_identity" "airflow" {
  name                = "${local.prefix}-airflow"
  resource_group_name = azurerm_resource_group.main.name
  location            = var.location
}

# Federated credential: link K8s SA to Managed Identity
resource "azurerm_federated_identity_credential" "airflow_worker" {
  name                = "airflow-worker"
  resource_group_name = azurerm_resource_group.main.name
  parent_id           = azurerm_user_assigned_identity.airflow.id
  audience            = ["api://AzureADTokenExchange"]
  issuer              = azurerm_kubernetes_cluster.main.oidc_issuer_url
  subject             = "system:serviceaccount:airflow:airflow-worker"
}

# K8s ServiceAccount annotated with Managed Identity
apiVersion: v1
kind: ServiceAccount
metadata:
  name: airflow-worker
  namespace: airflow
  annotations:
    azure.workload.identity/client-id: "<managed-identity-client-id>"
  labels:
    azure.workload.identity/use: "true"

Azure Event Hubs (Kafka-Compatible)

# Event Hubs works with standard Kafka clients
from confluent_kafka import Producer, Consumer

# Connection string auth (or Managed Identity via OAUTHBEARER)
producer_conf = {
    "bootstrap.servers": "my-namespace.servicebus.windows.net:9093",
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "PLAIN",
    "sasl.username": "$ConnectionString",
    "sasl.password": "Endpoint=sb://my-namespace.servicebus.windows.net/;SharedAccessKeyName=...;SharedAccessKey=...",
}

# Event Hubs = Kafka topic; Namespace = Kafka cluster
# Partition = Event Hub partition (fixed at creation time)
producer = Producer(producer_conf)
producer.produce("orders-raw", key="order-1", value='{"order_id": "1"}')
producer.flush()

resource "azurerm_eventhub_namespace" "main" {
  name                = "${local.prefix}-events"
  resource_group_name = azurerm_resource_group.main.name
  location            = var.location
  sku                 = "Standard"
  capacity            = 4           # throughput units (1 TU = 1 MB/s ingress)

  kafka_enabled            = true   # enable Kafka protocol
  auto_inflate_enabled     = true
  maximum_throughput_units = 20
}

resource "azurerm_eventhub" "orders" {
  name                = "orders-raw"
  namespace_name      = azurerm_eventhub_namespace.main.name
  resource_group_name = azurerm_resource_group.main.name
  partition_count     = 32
  message_retention   = 7           # days
}

Azure Synapse vs Databricks

AspectAzure SynapseAzure Databricks
SQL analyticsDedicated/serverless SQL poolDatabricks SQL Warehouse
SparkSynapse Spark poolsFull Databricks Runtime
Delta LakeVia Spark poolNative (Delta Engine)
OrchestrationSynapse PipelinesDatabricks Workflows
MLflowLimitedBuilt-in
Unity CatalogNoYes (data governance)
CostSQL pool: per DWU-hourPer DBU-hour
Best forSQL warehouse workloadsLakehouse + ML platform

Key Vault for Secrets

resource "azurerm_key_vault" "data_platform" {
  name                       = "${local.prefix}-kv"
  resource_group_name        = azurerm_resource_group.main.name
  location                   = var.location
  tenant_id                  = data.azurerm_client_config.current.tenant_id
  sku_name                   = "standard"
  soft_delete_retention_days = 90
  purge_protection_enabled   = true

  network_acls {
    default_action = "Deny"
    bypass         = "AzureServices"
    ip_rules       = [var.corp_vpn_cidr]
    virtual_network_subnet_ids = [azurerm_subnet.aks.id]
  }
}

resource "azurerm_key_vault_access_policy" "airflow" {
  key_vault_id = azurerm_key_vault.data_platform.id
  tenant_id    = data.azurerm_client_config.current.tenant_id
  object_id    = azurerm_user_assigned_identity.airflow.principal_id

  secret_permissions = ["Get", "List"]
}

# Access Key Vault from AKS pod with Workload Identity
from azure.keyvault.secrets import SecretClient
from azure.identity import WorkloadIdentityCredential

credential = WorkloadIdentityCredential()
client = SecretClient(
    vault_url="https://my-kv.vault.azure.net/",
    credential=credential,
)

db_password = client.get_secret("airflow-db-password").value

Azure Monitor and Log Analytics

resource "azurerm_log_analytics_workspace" "main" {
  name                = "${local.prefix}-logs"
  resource_group_name = azurerm_resource_group.main.name
  location            = var.location
  sku                 = "PerGB2018"
  retention_in_days   = 90
}

# AKS diagnostic settings → Log Analytics
resource "azurerm_monitor_diagnostic_setting" "aks" {
  name               = "aks-diagnostics"
  target_resource_id = azurerm_kubernetes_cluster.main.id
  log_analytics_workspace_id = azurerm_log_analytics_workspace.main.id

  enabled_log {
    category = "kube-apiserver"
  }
  enabled_log {
    category = "kube-controller-manager"
  }
  metric {
    category = "AllMetrics"
  }
}

// KQL: Airflow task failures in last 24h
ContainerLog
| where TimeGenerated > ago(24h)
| where ContainerName == "airflow-scheduler"
| where LogEntry contains "ERROR" or LogEntry contains "FAILED"
| parse LogEntry with * "dag_id=" dag_id "," * "task_id=" task_id ","*
| summarize count() by dag_id, task_id
| order by count_ desc

Private Endpoints for Data Services

# Private endpoint: ADLS Gen2 accessible only within VNet
resource "azurerm_private_endpoint" "storage" {
  name                = "${local.prefix}-storage-pe"
  resource_group_name = azurerm_resource_group.main.name
  location            = var.location
  subnet_id           = azurerm_subnet.private_endpoints.id

  private_service_connection {
    name                           = "storage"
    private_connection_resource_id = azurerm_storage_account.data_lake.id
    subresource_names              = ["dfs"]   # ADLS Gen2 endpoint
    is_manual_connection           = false
  }
}

Review Checklist

[ ] ADLS Gen2: hierarchical namespace, RBAC (not SAS tokens), lifecycle rules
[ ] No SAS tokens with long expiry — use Managed Identity instead
[ ] AKS: Workload Identity enabled (not AAD Pod Identity v1)
[ ] Key Vault: network deny default, private endpoint or VNet rules
[ ] Event Hubs: RBAC auth (not connection strings in code), min TLS 1.2
[ ] Private endpoints for all data services (no public internet access)
[ ] Azure Purview registered and scanning key datasets
[ ] Diagnostic settings: all services → Log Analytics (retention 90d)
[ ] ADLS Gen2: versioning enabled, soft delete configured
[ ] Cost: Reserved capacity for Synapse DWU or Databricks DBUs

Anti-Patterns

  1. SAS tokens with long expiry or no expiry — leaked token gives permanent access; use Managed Identity and RBAC for service-to-service auth.
  2. Storage account key rotation neglected — keys are valid indefinitely unless rotated; use RBAC and disable shared key access.
  3. No hierarchical namespace on ADLS — flat storage doesn't support directory-level RBAC or efficient rename operations; enable HNS from creation.
  4. Public endpoint on Key Vault — accessible from any IP; add network deny default + VNet rules.
  5. Event Hubs with basic SKU — no consumer groups beyond $Default, no Kafka protocol; use Standard SKU minimum for data pipelines.

References

  • ADLS Gen2: docs.microsoft.com/azure/storage/blobs/data-lake-storage-introduction
  • AKS Workload Identity: docs.microsoft.com/azure/aks/workload-identity-overview
  • Azure Databricks: docs.microsoft.com/azure/databricks/
  • Azure Purview: docs.microsoft.com/azure/purview/
  • Related skills: [[terraform-data]], [[infra-kubernetes-security-audit]], [[de-cost-optimization]]

レビュー

まだレビューはありません。使ってみた感想をお寄せください。

コメント

ログインなしで書き込めます。使ってみた感想、質問、動かなかった環境などを気軽にどうぞ。

まだコメントはありません。最初のコメントを書いてみませんか。

送信するとすぐに公開されます。メールアドレスや電話番号などの個人情報は書かないでください。URL は 2 つまでです。

0 / 1000 文字

このフォームは reCAPTCHA で保護されており、Google のプライバシーポリシーと利用規約が適用されます。

同じリポジトリのスキル

aiops-autonomous-incident-response

無料

AIOps autonomous incident response agent — LLM-driven diagnosis loop (Claude tool-use agent with kubectl/SQL/Prometheus tools), alert-to-action pipeline (PagerDuty webhook → agent trigger), automated RCA generation (failure taxonomy + log correlation), self-healing action executor with approval gate, incident severity classification, runbook automation (structured YAML runbooks executed by agent), escalation logic, incident timeline auto-generation, Slack integration for human-in-the-loop approvals

日本語の概要は準備中です。原文の説明を表示しています。

ivanshamaev/de-agent-skills222026年5月20日 更新

aiops-capacity-planning-agent

無料

AIOps capacity planning agent — predictive resource forecasting (Prophet/linear regression on Prometheus metrics), Kubernetes HPA/VPA right-sizing recommendations, node capacity headroom calculator, Kafka partition and broker capacity model, Spark executor sizing from job history, data lake storage growth forecast (S3/GCS), cluster autoscaler simulation, automatic HPA threshold tuning, cost-per-job attribution, capacity planning report generator

日本語の概要は準備中です。原文の説明を表示しています。

ivanshamaev/de-agent-skills222026年5月20日 更新

aiops-infrastructure-anomaly-detection

無料

AIOps infrastructure anomaly detection — statistical baseline (z-score/IQR/seasonal decomposition), Prometheus anomaly rules (predict_linear/stddev_over_time/MAD), ML-based detection (Isolation Forest/Prophet/LSTM autoencoder on metrics), metric correlation clustering, automated alert threshold tuning from historical data, Kubernetes resource anomaly detection, data pipeline anomaly signals (row count z-score/freshness drift/throughput drop), Python sklearn/Prophet integration, Grafana anomaly panel

日本語の概要は準備中です。原文の説明を表示しています。

ivanshamaev/de-agent-skills222026年5月20日 更新

aiops-observability-copilot

無料

AIOps observability copilot — natural language to PromQL/LogQL translation, alert explanation in plain English (what fired/why/impact), Grafana dashboard auto-generation from service topology, anomaly narrative generation from metric patterns, on-call context enrichment (related alerts/recent deploys/runbook links), log pattern clustering for noise reduction, SLO status explanation, automated weekly observability health digest

日本語の概要は準備中です。原文の説明を表示しています。

ivanshamaev/de-agent-skills222026年5月20日 更新

aiops-platform-optimization-agent

無料

AIOps continuous platform optimization agent — autonomous optimization loop (observe→analyze→recommend→apply), Kubernetes resource rightsizing (VPA recommendation reader + apply), Kafka consumer lag auto-scale, Spark job configuration tuning from history, idle resource detection (unused deployments/PVCs/topics), compaction and VACUUM scheduling, auto-ANALYZE stale statistics, cost savings attribution, optimization audit trail, human approval gate for high-risk changes

日本語の概要は準備中です。原文の説明を表示しています。

ivanshamaev/de-agent-skills222026年5月20日 更新

aiops-query-cost-analyzer

無料

AIOps autonomous query cost analyzer — Trino system.runtime.query_history cost SQL (CPU time/peak memory/bytes scanned), ClickHouse system.query_log top expensive queries, Spark UI cost attribution by user/job/DAG, BigQuery INFORMATION_SCHEMA.JOBS cost analysis, LLM-driven query rewrite recommendations (missing partition filters/broadcast hints/aggregation pushdown), cost anomaly detection (user over daily budget), automated query tagging, chargeback report by team/project

日本語の概要は準備中です。原文の説明を表示しています。

ivanshamaev/de-agent-skills222026年5月20日 更新

ivanshamaev のスキルをすべて見る

このスキルの問題を報告する