Ivan Shamaev

Trino-based Modern Data Platform architecture design — decoupled storage/compute, Iceberg as open table format, Hive Metastore/Glue catalog, MinIO/S3 object storage, Kafka ingestion layer, dbt transformation, Airflow orchestration, Superset BI — coordinator/worker topology, catalog design, metadata-driven architecture, multi-layer lakehouse (Bronze/Silver/Gold), federated query across heterogeneous sources

The English summary is being prepared. Showing the original description.

Trino observability and monitoring platform — JMX Prometheus exporter configuration (running queries/failed queries/OOM kills/execution latency P50/P90/P99/memory pool metrics), Grafana dashboard panels, OpenTelemetry trace propagation, query-level event listener for structured logging, Prometheus alert rules (worker loss/queue depth/OOM/failure rate/p99 latency), log aggregation patterns, query history analysis via REST API, slow query detection SQL

The English summary is being prepared. Showing the original description.

Trino production readiness checklist and review — coordinator HA (active/passive with load balancer), worker autoscaling, TLS/HTTPS configuration, authentication setup, resource groups for multi-tenancy, JVM sizing, spill configuration, fault-tolerant execution for batch workloads, monitoring (Prometheus/alerting), graceful shutdown, backup strategy for Hive Metastore, catalog security, query history retention, log rotation, Kubernetes deployment checklist

The English summary is being prepared. Showing the original description.

Trino memory management and spill-to-disk tuning — query.max-memory/query.max-total-memory/query.max-memory-per-node properties, memory.heap-headroom-per-node, JVM heap sizing (80% of RAM), spill-enabled/spiller-spill-path/spill-compression-codec configuration, exchange buffer tuning, OOM kill diagnosis, memory pool sizing, fault-tolerant execution exchange manager (S3/filesystem), session-level memory overrides, memory-intensive operator patterns (HashJoin/Sort/Window/GroupBy)

The English summary is being prepared. Showing the original description.

Trino resource group workload governance — resource-groups.properties configuration (maxQueued/hardConcurrencyLimit/softMemoryLimit/hardCpuLimit/schedulingPolicy/schedulingWeight), hierarchical group trees, selector rules (user/source/queryType/clientTags/regex), multi-tenant tenant isolation patterns, per-user dynamic sub-groups (${USER} template), scheduling policies (fair/weighted_fair/weighted/query_priority), CPU quota periods, database-backed configuration (MySQL/PostgreSQL), JMX monitoring of group utilization

The English summary is being prepared. Showing the original description.

Trino security hardening and data governance — TLS/HTTPS setup, internal-communication shared secret, authentication (LDAP/OAuth2/JWT), file-based access control (catalog/schema/table/column rules, column masking, row-level filtering, query kill permissions), OPA integration, Ranger for dynamic row-filter and column-mask, catalog isolation by domain, user group mapping, impersonation rules, audit logging via event listener, password file authentication for development

The English summary is being prepared. Showing the original description.

Autonomous self-healing Trino platform — Python watchdog agents for automatic worker restart detection, hung query killer, OOM-killed query resubmission, Iceberg compaction auto-trigger on small-file detection, stale statistics auto-ANALYZE, cluster memory pressure relief (kill low-priority queries), anomaly detection on query latency (Z-score), Claude-based RCA generation for incidents, Prometheus AlertManager webhook integration, Airflow self-healing sensor

The English summary is being prepared. Showing the original description.

Trino distributed SQL query optimization — predicate/projection/aggregation pushdown, join reordering (AUTOMATIC/ELIMINATE_CROSS_JOINS), broadcast vs partitioned joins, dynamic filtering, CBO with ANALYZE, filter-early patterns, partition pruning, avoiding SELECT *, reducing shuffle, cross-catalog query cost, session property tuning, query hints, anti-patterns for slow Trino queries

The English summary is being prepared. Showing the original description.

Trino Modern Data Stack end-to-end reference architecture — full docker-compose (Kafka + MinIO + Hive Metastore + Trino + Airflow + dbt + Superset + Prometheus + Grafana), medallion lake DDL and pipeline DAGs, Kafka→Iceberg streaming ingest, dbt project layout with Trino profiles, Superset dataset config, Prometheus alert rules, production Kubernetes Helm values, operational runbook for daily maintenance

The English summary is being prepared. Showing the original description.

StarRocks memory tuning — BE memory architecture (mem_limit, query pool, load pool, compaction pool, metadata cache), query memory limits (query_mem_limit session var), memory spill to disk (spill_mode), OOM analysis (be.out log), mem_tracker hierarchy, FE heap tuning (JAVA_OPTS), Primary Key table persistent index memory, jemalloc tuning

The English summary is being prepared. Showing the original description.

StarRocks data modeling — star schema vs wide table design, fact/dimension DDL patterns, denormalization strategy for OLAP, bitmap indexes, bloom filter indexes, sort key optimization, schema review checklist, SCD type 2 with Primary Key table, aggregation table design, BI acceleration patterns, avoiding over-normalization

The English summary is being prepared. Showing the original description.

StarRocks bucketing (distribution) — DISTRIBUTED BY HASH vs RANDOM, bucket count selection formula, distribution key selection (high cardinality/join key), skew detection and prevention, colocate join setup (colocate_with property), bucket count modification (ALTER TABLE), auto bucket sizing (StarRocks 3.2+), tablet sizing target (1-10 GB)

The English summary is being prepared. Showing the original description.

StarRocks table types — Duplicate Key (append-only log/event), Aggregate Key (pre-aggregation metrics), Unique Key (upsert dedup), Primary Key (full DML upsert/delete, MVCC), CREATE TABLE DDL, key column selection, aggregate functions (SUM/MAX/MIN/REPLACE/HLL_UNION/BITMAP_UNION), when to use each type, storage engine differences

The English summary is being prepared. Showing the original description.

StarRocks EXPLAIN plan reading — EXPLAIN/EXPLAIN VERBOSE/EXPLAIN COSTS output structure, operator nodes (OlapScanNode/HashJoinNode/AggregationNode/ExchangeNode/SortNode/ProjectNode), reading fragment instances, identifying bottlenecks (shuffle exchange cost, large scan rows, memory spill), runtime filter nodes, partition pruning in scan, query profile (SHOW PROFILELIST/EXPLAIN ANALYZE)

The English summary is being prepared. Showing the original description.

StarRocks data quality — freshness queries (MAX(updated_at) vs expected SLA), duplicate detection on Primary Key tables, null anomaly SQL (column-level null rate vs baseline), volume drift detection (day-over-day ratio), cross-table referential integrity checks, data completeness for partitions, StarRocks-specific quality checks (tablet health, compaction lag, replication factor), Python DQ scan class

The English summary is being prepared. Showing the original description.

StarRocks real-time analytics — Kafka → Routine Load → Primary Key table for sub-second freshness, low-latency BI query patterns, real-time dashboard design (materialized view for pre-aggregation), metric store patterns, window-based freshness for streaming dashboards, colocate join for real-time multi-table queries, resource group isolation for OLAP vs ingestion

The English summary is being prepared. Showing the original description.

StarRocks aggregation optimization — one-phase vs two-phase aggregation, pre-aggregation pushdown to scan, Aggregate Key table automatic merge, GROUP BY on distribution key (local aggregation), COUNT DISTINCT optimization (multi-phase), HLL/BITMAP approximate aggregation, aggregation spill tuning, EXPLAIN aggregation nodes

The English summary is being prepared. Showing the original description.

StarRocks security and RBAC — CREATE USER/ROLE, GRANT/REVOKE privileges (catalog/database/table/view/MV/function/resource group level), built-in roles (cluster_admin/db_admin/user_admin/public), LDAP/LDAP group auth, row-level security policies, column masking, SSL/TLS, audit log, privilege inheritance, security best practices

The English summary is being prepared. Showing the original description.

StarRocks storage balancing — tablet distribution across BEs, disk skew detection (SHOW BACKENDS disk usage), cluster_balance analysis (SHOW PROC '/cluster_balance'), balance_load_score_threshold tuning, tablet migration monitoring, BE decommission, adding new BE nodes, storage capacity forecasting, imbalance alerts

The English summary is being prepared. Showing the original description.

StarRocks AI incident RCA — autonomous root cause analysis agent for FE/BE failures, memory spike diagnosis (BE heap/process memory), Routine Load pause RCA (ErrorLogUrls + error patterns), Kafka consumer lag spike analysis, compaction backlog detection, tablet replication failures, OOM kill patterns, query timeout clusters, FE GC pressure detection, structured incident report generation

The English summary is being prepared. Showing the original description.

dbt + StarRocks production readiness — slim CI with state:modified+, manifest.json artifact storage and --defer for dev, dbt Cloud vs dbt Core deployment, RBAC for dbt users in StarRocks, secrets management for profiles.yml, dbt run/test exit codes in CI, breaking change detection, scheduled dbt runs via Airflow/dbt Cloud, dbt docs generation and hosting, model versioning

The English summary is being prepared. Showing the original description.

Airflow + StarRocks pipeline orchestration — MySqlHook for DDL/DML, Broker Load trigger + poll pattern, Stream Load via HTTP operator, Routine Load job lifecycle (pause/resume/stop), sensor for load completion, ANALYZE TABLE post-load, partition-aware DAG templating, Airflow variables for StarRocks connection config, XCom label passing

The English summary is being prepared. Showing the original description.

StarRocks backup and restore — CREATE REPOSITORY (S3/HDFS/NFS), BACKUP DATABASE/TABLE, SHOW BACKUP/RESTORE status, RESTORE with partition granularity, cross-cluster restore, incremental backup strategy, snapshot lifecycle, disaster recovery RTO/RPO targets

The English summary is being prepared. Showing the original description.

StarRocks cluster health monitoring — FE/BE/CN node status, tablet health, replica status, compaction backlog, storage imbalance, SHOW commands, system tables (information_schema/sys), alert thresholds, FE quorum checks, BE disk usage, dead replica detection, Prometheus metrics

The English summary is being prepared. Showing the original description.