Ivan Shamaev

SQLMesh data transformation framework — model kinds (FULL/INCREMENTAL_BY_TIME_RANGE/INCREMENTAL_BY_UNIQUE_KEY/SCD_TYPE_2/VIEW/SEED), plan/apply workflow, virtual environments, state-aware deploys, audits, unit tests, CI/CD, dbt migration

The English summary is being prepared. Showing the original description.

Use when writing, optimizing, or maintaining Apache Iceberg tables with Trino — covering table DDL, all ALTER TABLE operations, partition transforms, bucketing, sorted tables, DML (INSERT/UPDATE/DELETE/MERGE), EXPLAIN plan reading, join optimization, statistics with ANALYZE, table maintenance (optimize/expire_snapshots/remove_orphan_files), schema evolution, time travel, and metadata table diagnostics.

The English summary is being prepared. Showing the original description.

Prefect 3.x workflows — flows, tasks, deployments, work pools, event-driven triggers, caching, retries, state hooks, Prefect Cloud/server, Python SDK, Docker/K8s infrastructure

The English summary is being prepared. Showing the original description.

Data Contracts — datacontract.com YAML specification (schema, quality, SLA, ownership, servers, changelog), Data Contract CLI (init/test/lint/diff/breaking/publish/export), SodaCL-style quality checks embedded in contract, breaking change detection (column removal/type change/constraint tightening), CI/CD GitHub Actions PR enforcement, Kafka/S3/database server definitions, contract-driven development (producer-first), semver versioning and deprecation, DataHub and OpenMetadata catalog publishing

The English summary is being prepared. Showing the original description.

Use when designing or implementing Data Vault 2.0 data models — Hubs, Links, Satellites, Reference Tables, Same-As Links, Multi-Active Satellites, Effectivity Satellites, Point-in-Time tables, Bridge tables, Business Vault patterns, hash key generation, DDL for all entity types, insert-only DML load patterns, pipeline architecture, and constructing Information Marts from the vault.

The English summary is being prepared. Showing the original description.

Use when writing, configuring, or optimizing dbt projects targeting Trino or Starburst — covering profiles.yml setup, all authentication methods, materializations (table/view/incremental/materialized_view/ephemeral), incremental strategies (append/merge/delete+insert), table properties (format/partitioning/sorted_by), on_schema_change, seeds, snapshots, grants, session properties, data modeling patterns (Kimball/staging/intermediate/mart), dbt project structure, tests, and CI/CD.

The English summary is being prepared. Showing the original description.

DE blameless postmortem — incident timeline reconstruction, impact quantification, root cause chain (5 Whys), corrective actions (CAPA), postmortem document template, severity classification, SLA breach reporting, communication templates, lessons-learned facilitation, postmortem review checklist

The English summary is being prepared. Showing the original description.

DuckDB in-process OLAP analytics — reading Parquet/CSV/JSON/Iceberg/Delta from S3/local, SQL (window functions, PIVOT, ASOF join, QUALIFY), Python API (duckdb.connect, fetchdf, register, UDFs), extensions (httpfs/iceberg/delta/postgres), performance tuning, COPY TO, persistent DB

The English summary is being prepared. Showing the original description.

Great Expectations (GX) — DataContext setup, Data Sources (Pandas/Spark/SQL/file), Expectation Suites, built-in expectations (null/uniqueness/range/set/regex/table-level/statistical), Validation Definitions, Checkpoints, Data Docs, Airflow integration, custom expectations, dbt integration, CI/CD usage, severity levels (warning/critical)

The English summary is being prepared. Showing the original description.

Mage AI data pipelines — block types (data loader/transformer/exporter/sensor/custom), pipeline YAML, hybrid SQL+Python blocks, triggers (schedule/event/API), streaming pipelines, dbt integration, Spark integration, Docker/Kubernetes deployment, io_config.yaml, pipeline variables, callbacks, backfills

The English summary is being prepared. Showing the original description.

OpenLineage data lineage tracking — RunEvent/Job/Dataset/facet spec, Marquez backend setup, Airflow/Spark/dbt integrations, column-level lineage, custom emitters, lineage-based impact analysis

The English summary is being prepared. Showing the original description.

PostgreSQL for data engineering — declarative partitioning (RANGE/LIST/HASH/pg_partman), index types (B-Tree/BRIN/GIN/GIST/partial/covering), COPY bulk load, EXPLAIN ANALYZE plan reading, autovacuum tuning, window functions, JSONB, CTEs, LATERAL joins, and bulk-load patterns (UNLOGGED/pg_bulkload)

The English summary is being prepared. Showing the original description.

Use when designing, implementing, reviewing, or optimizing production PySpark ETL/DataFrame pipelines at GB-TB+ scale, including schemas, joins, partitioning, window functions, writes, UDF avoidance, and Spark performance diagnostics.

The English summary is being prepared. Showing the original description.

MLflow for data engineering — tracking server setup (PostgreSQL backend + S3 artifacts), experiment tracking (params/metrics/artifacts/autolog), ETL job metadata logging (row counts, DQ metrics, lineage tags), Model Registry (register/alias/promote), model serving (pyfunc/REST API/batch scoring), MLproject files, Airflow integration (MLflowClientHook/XCom/model promotion), Spark integration (mlflow.spark.autolog/log_model/Delta metadata)

The English summary is being prepared. Showing the original description.

DE cost optimization — query cost analysis (Trino/Spark/ClickHouse/BigQuery), compute right-sizing, storage tier recommendations, spot/preemptible instances, partition pruning, Z-order/clustering, materialized view economics, data lifecycle policies, FinOps tagging, cost attribution, budget alerts

The English summary is being prepared. Showing the original description.

DE architecture decision records (ADR) — ADR template, trade-off analysis framework, technology evaluation criteria, common DE architecture decisions (storage format, orchestrator, streaming vs batch, lakehouse vs warehouse), decision reversibility, stakeholder communication

The English summary is being prepared. Showing the original description.

AIOps infrastructure anomaly detection — statistical baseline (z-score/IQR/seasonal decomposition), Prometheus anomaly rules (predict_linear/stddev_over_time/MAD), ML-based detection (Isolation Forest/Prophet/LSTM autoencoder on metrics), metric correlation clustering, automated alert threshold tuning from historical data, Kubernetes resource anomaly detection, data pipeline anomaly signals (row count z-score/freshness drift/throughput drop), Python sklearn/Prophet integration, Grafana anomaly panel

The English summary is being prepared. Showing the original description.

AIOps capacity planning agent — predictive resource forecasting (Prophet/linear regression on Prometheus metrics), Kubernetes HPA/VPA right-sizing recommendations, node capacity headroom calculator, Kafka partition and broker capacity model, Spark executor sizing from job history, data lake storage growth forecast (S3/GCS), cluster autoscaler simulation, automatic HPA threshold tuning, cost-per-job attribution, capacity planning report generator

The English summary is being prepared. Showing the original description.

AIOps autonomous incident response agent — LLM-driven diagnosis loop (Claude tool-use agent with kubectl/SQL/Prometheus tools), alert-to-action pipeline (PagerDuty webhook → agent trigger), automated RCA generation (failure taxonomy + log correlation), self-healing action executor with approval gate, incident severity classification, runbook automation (structured YAML runbooks executed by agent), escalation logic, incident timeline auto-generation, Slack integration for human-in-the-loop approvals

The English summary is being prepared. Showing the original description.

Use when writing, reviewing, debugging, or optimizing SQL for Vertica — covering DDL (CREATE/ALTER/DROP TABLE, columns, projections, segmentation, partitions), DML (INSERT, UPDATE, DELETE, MERGE, TRUNCATE, COPY), CRUD patterns, and Vertica-specific performance guidance including encoding, segmentation keys, partition pruning, and query optimization.

The English summary is being prepared. Showing the original description.

dbt Jinja macros — macro authoring fundamentals, Jinja syntax (blocks/filters/tests), context variables (this/target/adapter/execute), run_query, adapter.dispatch for cross-database macros, generate_schema_name, hooks, custom materializations, dbt-utils patterns

The English summary is being prepared. Showing the original description.

Use when optimizing, diagnosing, or reviewing Vertica 11.x SQL query performance — covering EXPLAIN plan reading, projection design for predicates/joins/GROUP BY/ORDER BY/analytic functions, segmentation strategies, column encoding, RLE, sort elimination, Top-K, INSERT-SELECT tuning, DELETE/UPDATE internals, and Data Collector diagnostics.

The English summary is being prepared. Showing the original description.

ClickHouse OLAP — MergeTree family engines (MergeTree/ReplacingMergeTree/AggregatingMergeTree/SummingMergeTree/CollapsingMergeTree), ORDER BY/PARTITION BY/PRIMARY KEY design, data skipping indexes (minmax/set/bloom_filter), TTL tiered storage, materialized views, projections, LowCardinality/Nullable/AggregateFunction types, INSERT batching, FINAL keyword, query optimization (PREWHERE/parallel_replicas), dictionaries, Kafka engine, Python clickhouse-connect, best practices

The English summary is being prepared. Showing the original description.

Change Data Capture with Debezium — PostgreSQL/MySQL/Oracle CDC connectors, change event structure (before/after/op/source), snapshot modes, Kafka Connect deployment, SMT transformations, outbox pattern, Iceberg/Delta sink integration, idempotency guarantees

The English summary is being prepared. Showing the original description.