Ivan Shamaev

Data platform SLA monitoring — freshness SLAs (data available by X:XX), completeness SLAs (row count variance < N%), latency SLAs (pipeline completes within N minutes), SLA breach detection (Prometheus/Soda/SQL), SLA burn rate alerting, SLA reporting dashboards, SLA miss root cause linking, cascading SLA dependencies (upstream delay propagation), SLA definition process, error budget tracking, consumer notification on SLA breach

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Alert fatigue reduction — SLO-based alerting (burn rate vs threshold), multi-window burn rate alerts (short/long window), AlertManager inhibition and silencing, alert deduplication (group_by), routing by severity (PagerDuty for critical / Slack for warning), alert ownership labels, runbook links in annotations, alert review process (weekly noise audit), flapping detection (for/pending period tuning), absent() vs rate() alerting patterns, ticket-based escalation for warning-level noise

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

AWS data platform review — S3 data lake (lifecycle/replication/encryption/access points), EMR vs Glue ETL trade-offs, MSK Kafka configuration, RDS/Aurora for metadata, Redshift vs Athena for analytics, MWAA (managed Airflow), EKS for containerized pipelines, IAM roles (IRSA for EKS/EMREC2 instance profiles), Lake Formation row/column security, AWS Glue catalog, VPC data platform networking, cost optimization (S3 Intelligent-Tiering/Spot instances)

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Multi-cloud governance — cloud-agnostic data platform patterns, federated identity (OIDC/SAML between AWS/GCP/Azure), Terraform multi-cloud modules, cross-cloud data replication (S3↔GCS/Azure), unified cost management (FinOps Foundation framework), cloud-agnostic observability (OpenTelemetry), policy enforcement (OPA Gatekeeper across clouds), disaster recovery cross-cloud, vendor lock-in avoidance (open formats Iceberg/Parquet), centralized secrets management (HashiCorp Vault)

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Network security review for data platforms — Kubernetes NetworkPolicy (default-deny + allow patterns), VPC/subnet design (private subnets for all data services), security group rules audit (0.0.0.0/0 detection), TLS everywhere (Kafka mTLS/Trino HTTPS/DB SSL), service mesh (Istio mTLS), DNS security (private hosted zones), egress filtering (no unrestricted outbound), VPC peering vs PrivateLink, network flow logs analysis, firewall rules review

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Observability stack review — three pillars (metrics/logs/traces), Prometheus+Grafana+Loki+Tempo (LGTM stack), OpenTelemetry Collector as universal agent, cardinality management, log aggregation patterns (Fluent Bit→Loki), distributed tracing (Jaeger/Tempo), alerting pipeline (Alertmanager routing), SLO definition and error budget tracking, data platform observability (pipeline freshness/throughput/error rate), observability as code (Grafana provisioning)

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

RBAC audit for data platforms — Kubernetes RBAC (ClusterRole/Role/Binding audit), AWS IAM permission boundary and policy review (AWS IAM Access Analyzer), GCP IAM audit (principle of least privilege/owner role detection), database RBAC (PostgreSQL/Trino/ClickHouse role hierarchies), Airflow RBAC (role-based menu/DAG access), dbt permissions (warehouse roles), service account proliferation detection, wildcard permission detection, privilege escalation paths

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Terraform cost estimation — Infracost breakdown and diff commands (monthly cost per resource), PR cost comments (before/after change delta), budget thresholds (block PRs over cost limit), usage files for variable quantities (S3 GB/requests), AWS pricing data sources, cost tagging strategy, OpenTofu/Terraform cost attribution, FinOps tagging policy enforcement, multi-environment cost comparison (dev vs prod)

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Terraform code review — module structure (main/variables/outputs/modules layout), variable validation blocks, type constraints, sensitive variables, locals vs variables, resource naming conventions, provider version pinning, data sources vs resources, module composition patterns, state management (remote backend S3/GCS), workspace strategy, dependency management (depends_on vs implicit), terraform fmt/validate/tflint checks, DRY with Terragrunt

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Data platform self-service API — FastAPI REST API for platform operations (Kafka topic CRUD/Airflow DAG trigger/dbt job run/Trino query execution), OAuth2+JWT authentication with team-based RBAC, async job tracking with status polling, rate limiting per team, OpenAPI spec generation, Terraform provider for API resources, versioned API (v1/v2), audit logging of all platform operations, SDK generation (Python/TypeScript), Kubernetes deployment with HPA

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Internal Developer Platform (IDP) for data engineering — Backstage Software Catalog (catalog-info.yaml for pipelines/datasets/services), Software Templates (scaffold new DAG/dbt project/Kafka topic), TechDocs integration, golden path enforcement, Port.io alternative, self-service data pipeline provisioning, platform scorecard (production readiness checks), team ownership model, API gateway for platform services, paved road vs escape hatch pattern

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Airflow + StarRocks backfill patterns — historical partition reprocessing, date range loops with INSERT OVERWRITE, Broker Load replay for S3 partitions, safe backfill with max_active_runs=1, partial backfill resume from checkpoint, backfill idempotency via label strategy, clearing downstream tasks before rerun, parallel partition backfill with concurrency limits

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Airflow + StarRocks data quality gates — post-load row count validation, freshness checks (MAX(updated_at) vs expected), null explosion detection, duplicate key audit on Primary Key tables, volume anomaly detection (z-score vs 7-day avg), SODA-style SQL quality checks embedded in DAG, quarantine pattern for bad partitions

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

DE pipeline root cause analysis — failure taxonomy, lineage tracing, upstream/downstream impact, log analysis patterns, Airflow/Spark/dbt failure diagnosis, data quality anomaly RCA, 5-Whys template, incident timeline reconstruction, runbook-style investigation checklists

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Delta Lake — table DDL (CREATE/PARTITIONED BY/LOCATION/TBLPROPERTIES), DML (INSERT/UPDATE/DELETE/MERGE), MERGE patterns (upsert/SCD2/CDC), schema evolution (mergeSchema/overwriteSchema/ALTER TABLE), OPTIMIZE compaction, Z-ORDER BY data skipping, VACUUM, Time Travel (VERSION AS OF/TIMESTAMP AS OF), RESTORE, DESCRIBE HISTORY, shallow/deep clone, Change Data Feed, streaming read/write, deletion vectors, table properties

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Docker for data engineering environments — multi-stage Dockerfiles for dbt/Spark/Airflow images, BuildKit layer caching (--mount=type=cache), private registries (ghcr.io/Harbor), docker buildx multi-platform builds, Docker Compose local data stacks (Spark+Airflow+Kafka+MinIO+Postgres), BuildKit secrets for private PyPI, security hardening (non-root user, slim base images, read-only FS), CI/CD with GitHub Actions build-push-action

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Feast feature store — feature repo layout, Entity/FeatureView/FeatureService definitions, offline store (BigQuery/Redshift/Snowflake/file), online store (Redis/DynamoDB/SQLite), point-in-time correct joins, feast materialize, feast apply, Python SDK get_online_features/get_historical_features, push sources, streaming feature views, Airflow integration

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Use when building or reviewing GitHub Actions CI/CD pipelines for data engineering — covering dbt slim CI (state:modified+, --defer, manifest.json), SQLFluff SQL linting with PR annotations, Airflow DAG integrity tests (pytest/DagBag), Great Expectations and Soda data quality gates, multi-stage Docker image builds for Spark/dbt/Airflow pushed to ghcr.io, secrets management with OIDC for AWS/GCP (no static keys), environment-scoped secrets, reusable workflows (workflow_call), matrix builds, composite actions, and full end-to-end dbt project CI/CD workflows.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Use when designing or implementing dimensional data models following Kimball methodology — fact tables (transaction/snapshot/accumulating), dimension tables, SCD types (1/2/3/4/6), star/snowflake schemas, surrogate keys, conformed dimensions, role-playing dimensions, junk dimensions, bridge tables, date dimensions, degenerate dimensions, DDL patterns, and DML update strategies for loading and maintaining dimensional tables.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Data platform on Kubernetes — Spark-on-K8s (spark-submit, pod templates, dynamic allocation, RBAC), Airflow on K8s (Helm chart, KubernetesExecutor, KubernetesPodOperator, git-sync, DAG storage), resource management, namespaces, autoscaling

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

MCP (Model Context Protocol) server development — FastMCP/Python SDK, tools/resources/prompts primitives, STDIO and Streamable HTTP transports, Claude Desktop/Claude Code client config, MCP Inspector testing, security best practices (input validation/OAuth2/confused deputy prevention), production Docker/Kubernetes deployment, agentic data platform integration

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Use when designing or implementing Medallion (Bronze/Silver/Gold) data lakehouse architecture — layer definitions, DDL for each layer, DML load and update patterns, incremental pipeline design, deduplication strategies (row_number/merge/hash/CDC/watermark), schema evolution, data quality gates, partitioning per layer, late-arriving data, orchestration patterns, and best practices for Iceberg-based lakehouses.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Ray Data distributed data processing — Dataset API, read_parquet/read_csv/read_json/read_delta_sharing, map/filter/flat_map/map_batches, groupby/aggregations, Actors for stateful transforms, Ray remote functions, write_parquet/write_iceberg, streaming execution, GPU batch inference, integration with Spark and Airflow

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

SQLFluff SQL linter — .sqlfluff config, dialect selection (ansi/bigquery/clickhouse/duckdb/hive/postgres/snowflake/sparksql/trino), rule sets, templating (Jinja/dbt), fix command, VS Code integration, pre-commit hook, CI/CD GitHub Actions, custom rules, noqa inline suppression

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.