Ivan Shamaev

Production Apache Iceberg best practices with Trino — hidden partitioning (day/month/year/hour/bucket/truncate transforms), partition evolution, sorted tables, file compaction (optimize/expire_snapshots/remove_orphan_files), snapshot management, metadata tables ($snapshots/$files/$partitions/$history), schema evolution, time travel queries, MERGE/UPDATE/DELETE DML, ANALYZE for CBO, Iceberg catalog types (HMS/Glue/REST/Nessie), small file prevention, Bloom filters

中文简介正在准备中。当前显示原文说明。

Airflow orchestration for Trino pipelines — TrinoOperator and TrinoHook usage, trino_conn_id connection setup, idempotent DAG patterns (INSERT INTO / MERGE with deterministic run_ids), partition-aware scheduling with Airflow logical_date, sensor patterns (TrinoCheckOperator), SLA monitoring, retry configuration for Trino queries, incremental load orchestration, dbt + Trino Airflow integration (BashOperator/DbtRunOperator), metadata-driven DAG generation

中文简介正在准备中。当前显示原文说明。

StarRocks table types — Duplicate Key (append-only log/event), Aggregate Key (pre-aggregation metrics), Unique Key (upsert dedup), Primary Key (full DML upsert/delete, MVCC), CREATE TABLE DDL, key column selection, aggregate functions (SUM/MAX/MIN/REPLACE/HLL_UNION/BITMAP_UNION), when to use each type, storage engine differences

中文简介正在准备中。当前显示原文说明。

StarRocks memory tuning — BE memory architecture (mem_limit, query pool, load pool, compaction pool, metadata cache), query memory limits (query_mem_limit session var), memory spill to disk (spill_mode), OOM analysis (be.out log), mem_tracker hierarchy, FE heap tuning (JAVA_OPTS), Primary Key table persistent index memory, jemalloc tuning

中文简介正在准备中。当前显示原文说明。

StarRocks EXPLAIN plan reading — EXPLAIN/EXPLAIN VERBOSE/EXPLAIN COSTS output structure, operator nodes (OlapScanNode/HashJoinNode/AggregationNode/ExchangeNode/SortNode/ProjectNode), reading fragment instances, identifying bottlenecks (shuffle exchange cost, large scan rows, memory spill), runtime filter nodes, partition pruning in scan, query profile (SHOW PROFILELIST/EXPLAIN ANALYZE)

中文简介正在准备中。当前显示原文说明。

StarRocks Routine Load for Kafka — CREATE ROUTINE LOAD DDL (all PROPERTIES/KAFKA clause parameters), JSON/CSV/Avro format config, desired_concurrent_number tuning, SHOW ROUTINE LOAD columns, PAUSE/RESUME/ALTER/STOP, consumer lag monitoring, error log analysis, exactly-once semantics, Schema Registry for Avro, Kafka SASL/SSL config, idempotent upsert to Primary Key table

中文简介正在准备中。当前显示原文说明。

dbt + StarRocks models — dbt-starrocks adapter setup (profiles.yml), all materializations (table/view/incremental/ephemeral), incremental strategies (append/insert_overwrite/unique_key merge), StarRocks-specific model config (engine/keys/partition_by/distributed_by/properties), Duplicate/Aggregate/Primary Key table DDL from dbt, partition_by with date_trunc, on_schema_change behavior

中文简介正在准备中。当前显示原文说明。

StarRocks self-healing automation — auto-restart failed Routine Load jobs (detect PAUSED/CANCELLED + RESUME), rebalance tablet distribution after BE add/remove (REBALANCE command), trigger manual compaction for high-score tablets, auto-ANALYZE stale statistics, partition management (auto-drop old partitions), dead load label cleanup, BE disk space alerts + auto-tiering, scheduled health check + remediation agent

中文简介正在准备中。当前显示原文说明。

StarRocks schema evolution — ALTER TABLE ADD/DROP/MODIFY/REORDER COLUMN, column type widening rules, adding columns to Aggregate/Unique/Primary Key tables, schema change types (direct vs linked vs sorted), SHOW ALTER TABLE COLUMN progress, rollback safety, backward-compatible patterns, dbt model evolution, zero-downtime migration checklist

中文简介正在准备中。当前显示原文说明。

StarRocks Stream Load — HTTP PUT API (curl/Python requests), CSV/JSON format parameters (column_separator/jsonpaths/strip_outer_array), columns mapping, partial_update for Primary Key tables, label idempotency, max_filter_ratio, timeout tuning, merge commit mode (3.4+), response JSON parsing, Python bulk loader pattern, error diagnosis

中文简介正在准备中。当前显示原文说明。

Airflow + StarRocks pipeline orchestration — MySqlHook for DDL/DML, Broker Load trigger + poll pattern, Stream Load via HTTP operator, Routine Load job lifecycle (pause/resume/stop), sensor for load completion, ANALYZE TABLE post-load, partition-aware DAG templating, Airflow variables for StarRocks connection config, XCom label passing

中文简介正在准备中。当前显示原文说明。

StarRocks backup and restore — CREATE REPOSITORY (S3/HDFS/NFS), BACKUP DATABASE/TABLE, SHOW BACKUP/RESTORE status, RESTORE with partition granularity, cross-cluster restore, incremental backup strategy, snapshot lifecycle, disaster recovery RTO/RPO targets

中文简介正在准备中。当前显示原文说明。

dbt + StarRocks production readiness — slim CI with state:modified+, manifest.json artifact storage and --defer for dev, dbt Cloud vs dbt Core deployment, RBAC for dbt users in StarRocks, secrets management for profiles.yml, dbt run/test exit codes in CI, breaking change detection, scheduled dbt runs via Airflow/dbt Cloud, dbt docs generation and hosting, model versioning

中文简介正在准备中。当前显示原文说明。

Trino warehouse cost optimization — query scan cost analysis (system.runtime.queries/system.runtime.tasks), identifying expensive queries by CPU time and data scanned, scan reduction via partition pruning and file compaction, worker autoscaling patterns (scale-to-zero for batch), spot instance strategies for workers, S3 object storage cost (storage vs request costs), Iceberg compaction economics (fewer splits = fewer S3 GET requests), cost attribution by team/user, materialized view break-even analysis, result caching

中文简介正在准备中。当前显示原文说明。

StarRocks security and RBAC — CREATE USER/ROLE, GRANT/REVOKE privileges (catalog/database/table/view/MV/function/resource group level), built-in roles (cluster_admin/db_admin/user_admin/public), LDAP/LDAP group auth, row-level security policies, column masking, SSL/TLS, audit log, privilege inheritance, security best practices

中文简介正在准备中。当前显示原文说明。

StarRocks storage balancing — tablet distribution across BEs, disk skew detection (SHOW BACKENDS disk usage), cluster_balance analysis (SHOW PROC '/cluster_balance'), balance_load_score_threshold tuning, tablet migration monitoring, BE decommission, adding new BE nodes, storage capacity forecasting, imbalance alerts

中文简介正在准备中。当前显示原文说明。

StarRocks AI query autotuner — autonomous SQL optimization agent workflow (EXPLAIN COSTS → analyze → recommend), materialized view recommendation from slow query log, index recommendation (bitmap/bloom filter), partition pruning diagnosis, join order hints generation, statistics staleness detection and auto-ANALYZE trigger, slow query pattern classification, query rewrite suggestions

中文简介正在准备中。当前显示原文说明。

dbt + StarRocks testing — generic tests (not_null/unique/accepted_values/relationships), singular tests (custom SQL assertions), source freshness tests (loaded_at_field), StarRocks-specific volume and freshness tests, dbt-expectations integration, test severity (warn vs error), store_failures for debugging failed tests, ANALYZE before test runs, partition-scoped test SQL

中文简介正在准备中。当前显示原文说明。

StarRocks cluster health monitoring — FE/BE/CN node status, tablet health, replica status, compaction backlog, storage imbalance, SHOW commands, system tables (information_schema/sys), alert thresholds, FE quorum checks, BE disk usage, dead replica detection, Prometheus metrics

中文简介正在准备中。当前显示原文说明。

StarRocks aggregation optimization — one-phase vs two-phase aggregation, pre-aggregation pushdown to scan, Aggregate Key table automatic merge, GROUP BY on distribution key (local aggregation), COUNT DISTINCT optimization (multi-phase), HLL/BITMAP approximate aggregation, aggregation spill tuning, EXPLAIN aggregation nodes

中文简介正在准备中。当前显示原文说明。

StarRocks real-time data modeling — Primary Key table streaming upserts (Flink/Kafka Routine Load), mutable dimension tables (partial column UPDATE), real-time pre-aggregation with Aggregate Key + streaming inserts, Routine Load for Kafka CDC, real-time materialized view refresh, low-latency BI patterns, latency vs consistency trade-offs

中文简介正在准备中。当前显示原文说明。

StarRocks cost-based optimizer — ANALYZE TABLE (full/sample/histogram/predicate columns), AUTO ANALYZE configuration, statistics storage (_statistics_ database), SHOW ANALYZE STATUS, stale statistics detection, cardinality estimation failures, CBO join hints (LEADING/broadcast/shuffle), EXPLAIN COSTS reading, multi-column statistics (3.5+)

中文简介正在准备中。当前显示原文说明。

Trino cluster health monitoring and administration — coordinator/worker health checks, REST API status endpoints (/v1/info /v1/node /v1/query /v1/cluster), JMX MBean metrics (running queries/failed queries/OOM kills/memory pool), Prometheus JMX exporter config, Grafana dashboards, query queue depth monitoring, worker node stability, memory pressure detection, BLOCKED query diagnosis, Web UI interpretation, log analysis, graceful shutdown

中文简介正在准备中。当前显示原文说明。

dbt + StarRocks performance tuning — model DAG optimization (fan-in/fan-out), incremental merge window sizing, partition-aware incremental filters, avoiding full table rebuilds, query plan hints in dbt SQL (LEADING/JOIN hints), materialized view as dbt model target, concurrent model execution with threads, pre/post hooks for ANALYZE TABLE, dbt run selectors to minimize rebuilt models

中文简介正在准备中。当前显示原文说明。