Ivan Shamaev

Terraform for data infrastructure — S3/MinIO data lake buckets (versioning, lifecycle, SSE-KMS), IAM roles for Spark/Airflow (least-privilege, IRSA on EKS), MSK/Kafka clusters (aws_msk_cluster, encryption, custom broker config), Kubernetes data platform (helm_release Airflow + Spark, namespace resource quotas), module layout (modules/ + envs/), typed variables with validation, S3 remote state + DynamoDB locking, Terragrunt DRY configs, GitHub Actions CI/CD pipeline with plan/apply

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Use when writing, reviewing, debugging, or optimizing production Spark SQL for Hive/lakehouse/HDFS tables, including CTE-heavy queries, joins, windows, partition pruning, Hive Metastore operations, insert/overwrite safety, query hints, statistics, EXPLAIN plans, AQE, skew, materialization, and SQL performance diagnostics.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Apache Airflow DAG authoring — DAG definition, TaskFlow API (@task/@dag decorators), operators (Bash/Python/SQL/HTTP), sensors (poke/reschedule modes), TaskGroups, dynamic task mapping (expand/partial), branching, trigger rules, XComs, Pools, callbacks, cross-DAG pipelines, best practices

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Apache Flink — streaming architecture (JobManager/TaskManager/slots/parallelism), PyFlink Table API and SQL (Kafka/filesystem DDL, windowing TUMBLE/HOP/SESSION, aggregations), DataStream API (KeyedStream, stateful operations ValueState/MapState, process functions), event-time vs processing-time, watermark strategies, checkpointing (exactly-once, RocksDB state backend), Kafka source/sink (delivery guarantees), backpressure, savepoints, deployment (standalone/YARN/Kubernetes)

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Soda Core data quality — SodaCL checks (row_count, missing, invalid, duplicate, freshness, schema, reference, custom SQL), configuration.yml for PostgreSQL/Spark/ClickHouse/BigQuery, soda scan CLI, Airflow integration, dbt integration, alerting

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Redpanda Kafka-compatible streaming — cluster setup, topic config, rpk CLI, producer/consumer tuning, tiered storage, Schema Registry, Kafka Connect compatibility, monitoring, Docker/Kubernetes deployment

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Use when designing, building, or debugging a RAG (Retrieval-Augmented Generation) data pipeline — document ingestion, chunking strategies (fixed/recursive/semantic/structure-aware), embedding models (OpenAI/Cohere/sentence-transformers), vector stores (pgvector/Chroma/Qdrant/Weaviate), incremental refresh, hybrid retrieval (dense + BM25 + RRF), re-ranking with cross-encoders, metadata filtering, and production monitoring.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

PySpark Structured Streaming — streaming sources (Kafka/file/rate), output modes (append/complete/update), triggers (ProcessingTime/AvailableNow/Once/Continuous), watermarks, event-time windows (tumbling/sliding/session), stateful aggregations, deduplication, foreachBatch, stream-stream joins, checkpointing, fault tolerance, RocksDB state store, Kafka source/sink options, production patterns

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Airbyte ELT — source/destination connectors, sync modes (Full Refresh Overwrite/Append, Incremental Append/Deduped), cursor fields, primary keys, catalog/streams, deployment (abctl, Kubernetes Helm, Airbyte Cloud), Connector Builder, Python CDK (HttpStream, IncrementalMixin), normalization (dbt-based, _airbyte_raw_ tables), Airbyte API, Terraform provider, schema evolution, monitoring, Airflow AirbyteTriggerSyncOperator integration

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Use when building, reviewing, debugging, or scaling Apache Airflow DAGs declaratively with dag-factory YAML configs — including project setup, loader configuration, defaults hierarchy, custom/provider operators, dynamic task mapping, datasets/assets, callbacks, TaskFlow decorators, Jinja2 templating, environment variables, YAML DRY patterns, large-scale multi-DAG generation, CI/CD linting, and migration from pre-1.0.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Apache Kafka — topics, partitions, consumer groups, producer/consumer configuration (acks, idempotence, compression, commit strategies), exactly-once semantics, Kafka Connect (source/sink connectors, SMTs, DLQ), Schema Registry, consumer lag monitoring, CLI operations, Python confluent-kafka examples, partition strategy, replication, retention

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Dagster Software-Defined Assets (SDA) — @asset/@multi_asset decorators, asset dependencies, partitions (daily/static/dynamic/multi), sensors, IO managers, declarative automation (AutomationCondition), jobs, schedules, resource definitions, deployment

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

DataHub data catalog — metadata ingestion recipes (PostgreSQL/Hive/Spark/dbt/Airflow/Kafka/S3), Python SDK (DatahubRestEmitter, MetadataChangeProposalWrapper), column-level lineage (FineGrainedLineage), GMS/MCE/MCP architecture, Kubernetes Helm deployment, CLI operations, search/discovery REST API and GraphQL

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

dbt Core (adapter-agnostic) — project structure, models (SQL/Python), sources and refs, materializations (table/view/incremental/ephemeral), incremental strategies (append/merge/delete+insert), is_incremental() macro, snapshots (SCD Type 2), seeds, tests (generic/singular/severity), Jinja macros, dbt-utils package, node selection syntax (graph operators), hooks, exposures, metrics (MetricFlow), CI/CD (slim CI with state:modified+), dbt_project.yml, profiles.yml, multi-adapter support (Spark/PostgreSQL/ClickHouse/BigQuery)

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Data engineering production readiness — idempotency patterns (INSERT ON CONFLICT, MERGE, partition overwrite), retry strategies (exponential backoff, Airflow retry config), SLA monitoring (Airflow sla_miss_callback, Prometheus PromQL), data freshness checks (Soda freshness, max event_time), structured logging (JSON logs, OpenTelemetry tracing), data reconciliation (row count, checksum, duplicate detection), graceful degradation (circuit breaker, fallback, skip failed partitions), change management (blue/green table swap, schema migration rollback)

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.