Ivan Shamaev

Production Apache Iceberg best practices with Trino — hidden partitioning (day/month/year/hour/bucket/truncate transforms), partition evolution, sorted tables, file compaction (optimize/expire_snapshots/remove_orphan_files), snapshot management, metadata tables ($snapshots/$files/$partitions/$history), schema evolution, time travel queries, MERGE/UPDATE/DELETE DML, ANALYZE for CBO, Iceberg catalog types (HMS/Glue/REST/Nessie), small file prevention, Bloom filters

The English summary is being prepared. Showing the original description.

Airflow orchestration for Trino pipelines — TrinoOperator and TrinoHook usage, trino_conn_id connection setup, idempotent DAG patterns (INSERT INTO / MERGE with deterministic run_ids), partition-aware scheduling with Airflow logical_date, sensor patterns (TrinoCheckOperator), SLA monitoring, retry configuration for Trino queries, incremental load orchestration, dbt + Trino Airflow integration (BashOperator/DbtRunOperator), metadata-driven DAG generation

The English summary is being prepared. Showing the original description.

StarRocks table types — Duplicate Key (append-only log/event), Aggregate Key (pre-aggregation metrics), Unique Key (upsert dedup), Primary Key (full DML upsert/delete, MVCC), CREATE TABLE DDL, key column selection, aggregate functions (SUM/MAX/MIN/REPLACE/HLL_UNION/BITMAP_UNION), when to use each type, storage engine differences

The English summary is being prepared. Showing the original description.

StarRocks memory tuning — BE memory architecture (mem_limit, query pool, load pool, compaction pool, metadata cache), query memory limits (query_mem_limit session var), memory spill to disk (spill_mode), OOM analysis (be.out log), mem_tracker hierarchy, FE heap tuning (JAVA_OPTS), Primary Key table persistent index memory, jemalloc tuning

The English summary is being prepared. Showing the original description.

StarRocks EXPLAIN plan reading — EXPLAIN/EXPLAIN VERBOSE/EXPLAIN COSTS output structure, operator nodes (OlapScanNode/HashJoinNode/AggregationNode/ExchangeNode/SortNode/ProjectNode), reading fragment instances, identifying bottlenecks (shuffle exchange cost, large scan rows, memory spill), runtime filter nodes, partition pruning in scan, query profile (SHOW PROFILELIST/EXPLAIN ANALYZE)

The English summary is being prepared. Showing the original description.

StarRocks Routine Load for Kafka — CREATE ROUTINE LOAD DDL (all PROPERTIES/KAFKA clause parameters), JSON/CSV/Avro format config, desired_concurrent_number tuning, SHOW ROUTINE LOAD columns, PAUSE/RESUME/ALTER/STOP, consumer lag monitoring, error log analysis, exactly-once semantics, Schema Registry for Avro, Kafka SASL/SSL config, idempotent upsert to Primary Key table

The English summary is being prepared. Showing the original description.

dbt + StarRocks models — dbt-starrocks adapter setup (profiles.yml), all materializations (table/view/incremental/ephemeral), incremental strategies (append/insert_overwrite/unique_key merge), StarRocks-specific model config (engine/keys/partition_by/distributed_by/properties), Duplicate/Aggregate/Primary Key table DDL from dbt, partition_by with date_trunc, on_schema_change behavior

The English summary is being prepared. Showing the original description.

StarRocks self-healing automation — auto-restart failed Routine Load jobs (detect PAUSED/CANCELLED + RESUME), rebalance tablet distribution after BE add/remove (REBALANCE command), trigger manual compaction for high-score tablets, auto-ANALYZE stale statistics, partition management (auto-drop old partitions), dead load label cleanup, BE disk space alerts + auto-tiering, scheduled health check + remediation agent

The English summary is being prepared. Showing the original description.

StarRocks schema evolution — ALTER TABLE ADD/DROP/MODIFY/REORDER COLUMN, column type widening rules, adding columns to Aggregate/Unique/Primary Key tables, schema change types (direct vs linked vs sorted), SHOW ALTER TABLE COLUMN progress, rollback safety, backward-compatible patterns, dbt model evolution, zero-downtime migration checklist

The English summary is being prepared. Showing the original description.

StarRocks Stream Load — HTTP PUT API (curl/Python requests), CSV/JSON format parameters (column_separator/jsonpaths/strip_outer_array), columns mapping, partial_update for Primary Key tables, label idempotency, max_filter_ratio, timeout tuning, merge commit mode (3.4+), response JSON parsing, Python bulk loader pattern, error diagnosis

The English summary is being prepared. Showing the original description.

Airflow + StarRocks pipeline orchestration — MySqlHook for DDL/DML, Broker Load trigger + poll pattern, Stream Load via HTTP operator, Routine Load job lifecycle (pause/resume/stop), sensor for load completion, ANALYZE TABLE post-load, partition-aware DAG templating, Airflow variables for StarRocks connection config, XCom label passing

The English summary is being prepared. Showing the original description.

StarRocks backup and restore — CREATE REPOSITORY (S3/HDFS/NFS), BACKUP DATABASE/TABLE, SHOW BACKUP/RESTORE status, RESTORE with partition granularity, cross-cluster restore, incremental backup strategy, snapshot lifecycle, disaster recovery RTO/RPO targets

The English summary is being prepared. Showing the original description.

dbt + StarRocks production readiness — slim CI with state:modified+, manifest.json artifact storage and --defer for dev, dbt Cloud vs dbt Core deployment, RBAC for dbt users in StarRocks, secrets management for profiles.yml, dbt run/test exit codes in CI, breaking change detection, scheduled dbt runs via Airflow/dbt Cloud, dbt docs generation and hosting, model versioning

The English summary is being prepared. Showing the original description.

Trino warehouse cost optimization — query scan cost analysis (system.runtime.queries/system.runtime.tasks), identifying expensive queries by CPU time and data scanned, scan reduction via partition pruning and file compaction, worker autoscaling patterns (scale-to-zero for batch), spot instance strategies for workers, S3 object storage cost (storage vs request costs), Iceberg compaction economics (fewer splits = fewer S3 GET requests), cost attribution by team/user, materialized view break-even analysis, result caching

The English summary is being prepared. Showing the original description.

StarRocks security and RBAC — CREATE USER/ROLE, GRANT/REVOKE privileges (catalog/database/table/view/MV/function/resource group level), built-in roles (cluster_admin/db_admin/user_admin/public), LDAP/LDAP group auth, row-level security policies, column masking, SSL/TLS, audit log, privilege inheritance, security best practices

The English summary is being prepared. Showing the original description.

StarRocks storage balancing — tablet distribution across BEs, disk skew detection (SHOW BACKENDS disk usage), cluster_balance analysis (SHOW PROC '/cluster_balance'), balance_load_score_threshold tuning, tablet migration monitoring, BE decommission, adding new BE nodes, storage capacity forecasting, imbalance alerts

The English summary is being prepared. Showing the original description.

StarRocks AI query autotuner — autonomous SQL optimization agent workflow (EXPLAIN COSTS → analyze → recommend), materialized view recommendation from slow query log, index recommendation (bitmap/bloom filter), partition pruning diagnosis, join order hints generation, statistics staleness detection and auto-ANALYZE trigger, slow query pattern classification, query rewrite suggestions

The English summary is being prepared. Showing the original description.

dbt + StarRocks testing — generic tests (not_null/unique/accepted_values/relationships), singular tests (custom SQL assertions), source freshness tests (loaded_at_field), StarRocks-specific volume and freshness tests, dbt-expectations integration, test severity (warn vs error), store_failures for debugging failed tests, ANALYZE before test runs, partition-scoped test SQL

The English summary is being prepared. Showing the original description.

StarRocks cluster health monitoring — FE/BE/CN node status, tablet health, replica status, compaction backlog, storage imbalance, SHOW commands, system tables (information_schema/sys), alert thresholds, FE quorum checks, BE disk usage, dead replica detection, Prometheus metrics

The English summary is being prepared. Showing the original description.

StarRocks aggregation optimization — one-phase vs two-phase aggregation, pre-aggregation pushdown to scan, Aggregate Key table automatic merge, GROUP BY on distribution key (local aggregation), COUNT DISTINCT optimization (multi-phase), HLL/BITMAP approximate aggregation, aggregation spill tuning, EXPLAIN aggregation nodes

The English summary is being prepared. Showing the original description.

StarRocks real-time data modeling — Primary Key table streaming upserts (Flink/Kafka Routine Load), mutable dimension tables (partial column UPDATE), real-time pre-aggregation with Aggregate Key + streaming inserts, Routine Load for Kafka CDC, real-time materialized view refresh, low-latency BI patterns, latency vs consistency trade-offs

The English summary is being prepared. Showing the original description.

StarRocks cost-based optimizer — ANALYZE TABLE (full/sample/histogram/predicate columns), AUTO ANALYZE configuration, statistics storage (_statistics_ database), SHOW ANALYZE STATUS, stale statistics detection, cardinality estimation failures, CBO join hints (LEADING/broadcast/shuffle), EXPLAIN COSTS reading, multi-column statistics (3.5+)

The English summary is being prepared. Showing the original description.

Trino cluster health monitoring and administration — coordinator/worker health checks, REST API status endpoints (/v1/info /v1/node /v1/query /v1/cluster), JMX MBean metrics (running queries/failed queries/OOM kills/memory pool), Prometheus JMX exporter config, Grafana dashboards, query queue depth monitoring, worker node stability, memory pressure detection, BLOCKED query diagnosis, Web UI interpretation, log analysis, graceful shutdown

The English summary is being prepared. Showing the original description.

dbt + StarRocks performance tuning — model DAG optimization (fan-in/fan-out), incremental merge window sizing, partition-aware incremental filters, avoiding full table rebuilds, query plan hints in dbt SQL (LEADING/JOIN hints), materialized view as dbt model target, concurrent model execution with threads, pre/post hooks for ANALYZE TABLE, dbt run selectors to minimize rebuilt models

The English summary is being prepared. Showing the original description.