Ivan Shamaev

Airflow orchestration for Trino pipelines — TrinoOperator and TrinoHook usage, trino_conn_id connection setup, idempotent DAG patterns (INSERT INTO / MERGE with deterministic run_ids), partition-aware scheduling with Airflow logical_date, sensor patterns (TrinoCheckOperator), SLA monitoring, retry configuration for Trino queries, incremental load orchestration, dbt + Trino Airflow integration (BashOperator/DbtRunOperator), metadata-driven DAG generation

The English summary is being prepared. Showing the original description.

StarRocks self-healing automation — auto-restart failed Routine Load jobs (detect PAUSED/CANCELLED + RESUME), rebalance tablet distribution after BE add/remove (REBALANCE command), trigger manual compaction for high-score tablets, auto-ANALYZE stale statistics, partition management (auto-drop old partitions), dead load label cleanup, BE disk space alerts + auto-tiering, scheduled health check + remediation agent

The English summary is being prepared. Showing the original description.

StarRocks compaction tuning — base compaction vs cumulative compaction, compaction score analysis, write amplification reduction, BE config parameters (compaction_threads/min_cumulative_compaction_num_singleton_deltas/max_compaction_candidate_num), SHOW PROC '/compactions', manual compaction trigger, Primary Key table compaction, rowset management

The English summary is being prepared. Showing the original description.

dbt + StarRocks production readiness — slim CI with state:modified+, manifest.json artifact storage and --defer for dev, dbt Cloud vs dbt Core deployment, RBAC for dbt users in StarRocks, secrets management for profiles.yml, dbt run/test exit codes in CI, breaking change detection, scheduled dbt runs via Airflow/dbt Cloud, dbt docs generation and hosting, model versioning

The English summary is being prepared. Showing the original description.

Full local Trino Lakehouse Docker Compose stack — trinodb/trino coordinator+worker, Hive Metastore 3.x with PostgreSQL backend, MinIO object storage (S3-compatible), Apache Airflow, dbt-trino, Apache Superset BI, Prometheus+Grafana monitoring, healthchecks, startup ordering, persistent volumes, isolated networks, catalog property files, Iceberg catalog config, environment variable injection, Trino CLI access

The English summary is being prepared. Showing the original description.

StarRocks cluster health monitoring — FE/BE/CN node status, tablet health, replica status, compaction backlog, storage imbalance, SHOW commands, system tables (information_schema/sys), alert thresholds, FE quorum checks, BE disk usage, dead replica detection, Prometheus metrics

The English summary is being prepared. Showing the original description.

StarRocks Stream Load — HTTP PUT API (curl/Python requests), CSV/JSON format parameters (column_separator/jsonpaths/strip_outer_array), columns mapping, partial_update for Primary Key tables, label idempotency, max_filter_ratio, timeout tuning, merge commit mode (3.4+), response JSON parsing, Python bulk loader pattern, error diagnosis

The English summary is being prepared. Showing the original description.

StarRocks storage balancing — tablet distribution across BEs, disk skew detection (SHOW BACKENDS disk usage), cluster_balance analysis (SHOW PROC '/cluster_balance'), balance_load_score_threshold tuning, tablet migration monitoring, BE decommission, adding new BE nodes, storage capacity forecasting, imbalance alerts

The English summary is being prepared. Showing the original description.

StarRocks aggregation optimization — one-phase vs two-phase aggregation, pre-aggregation pushdown to scan, Aggregate Key table automatic merge, GROUP BY on distribution key (local aggregation), COUNT DISTINCT optimization (multi-phase), HLL/BITMAP approximate aggregation, aggregation spill tuning, EXPLAIN aggregation nodes

The English summary is being prepared. Showing the original description.

StarRocks AI incident RCA — autonomous root cause analysis agent for FE/BE failures, memory spike diagnosis (BE heap/process memory), Routine Load pause RCA (ErrorLogUrls + error patterns), Kafka consumer lag spike analysis, compaction backlog detection, tablet replication failures, OOM kill patterns, query timeout clusters, FE GC pressure detection, structured incident report generation

The English summary is being prepared. Showing the original description.

StarRocks AI query autotuner — autonomous SQL optimization agent workflow (EXPLAIN COSTS → analyze → recommend), materialized view recommendation from slow query log, index recommendation (bitmap/bloom filter), partition pruning diagnosis, join order hints generation, statistics staleness detection and auto-ANALYZE trigger, slow query pattern classification, query rewrite suggestions

The English summary is being prepared. Showing the original description.

Airflow production readiness — idempotent tasks (UPSERT/partition overwrite), no top-level code at DAG parse time (Variable.get inside tasks), KubernetesExecutor configuration, connection/variable management (no hardcoded secrets), SLA callbacks, retry with exponential backoff, pool management, max_active_runs, dag_concurrency, celery vs kubernetes executor trade-offs, metadata DB maintenance (airflow db clean), health check endpoints, structured logging

The English summary is being prepared. Showing the original description.

Airflow observability — StatsD/OpenTelemetry metrics (scheduler_heartbeat/task_duration/pool_open_slots/dag_processing_total_parse_time), Prometheus scraping, Grafana dashboards (DAG success rate/task duration/slot utilization), structured task logging (JSON formatter), OpenTelemetry traces for task spans, DAG SLA miss alerts, anomaly detection on task duration, Airflow audit log, DagBag parse error monitoring, dead letter queue for failed tasks

The English summary is being prepared. Showing the original description.

Airflow high availability review — multi-scheduler setup (HA schedulers with PostgreSQL row-level locking), CeleryExecutor worker pool sizing, KubernetesExecutor pod limits, metadata DB HA (PostgreSQL HA with PgBouncer connection pooling), DAG sync strategy (git-sync sidecar/S3/GitDagBundle), log storage (S3/GCS), webserver multi-replica, triggerer HA, rolling upgrades without downtime, Celery Flower monitoring, heartbeat alerts

The English summary is being prepared. Showing the original description.

AIOps continuous platform optimization agent — autonomous optimization loop (observe→analyze→recommend→apply), Kubernetes resource rightsizing (VPA recommendation reader + apply), Kafka consumer lag auto-scale, Spark job configuration tuning from history, idle resource detection (unused deployments/PVCs/topics), compaction and VACUUM scheduling, auto-ANALYZE stale statistics, cost savings attribution, optimization audit trail, human approval gate for high-risk changes

The English summary is being prepared. Showing the original description.

DataOps blameless postmortem generation — severity matrix (SEV1-4), postmortem Markdown template with timeline/impact/root cause/action items, 3 filled-in examples (Kafka rebalance/dbt incremental data loss/ClickHouse schema mutation), CAPA framework (Corrective/Action/Preventive/Action), impact quantification SQL, stakeholder communication templates, 60-min facilitation guide, postmortem review checklist, Git-based postmortem repository management

The English summary is being prepared. Showing the original description.

Agentic control plane for data platform — MCP (Model Context Protocol) server exposing platform tools to AI agents (FastMCP/Python SDK), natural language platform operations (trigger DAG/create topic/run dbt/query Trino via LLM), Claude-based platform assistant with tool use, multi-agent platform governance (audit agent/cost agent/reliability agent), MCP server deployment (Docker/Kubernetes), tool authorization and audit logging, agentic workflow patterns (plan-execute-verify), platform chat interface

The English summary is being prepared. Showing the original description.

Terraform security scanning — tfsec static analysis (AWS/GCP/Azure misconfigurations), Checkov IaC policy checks (750+ rules, CIS benchmarks), tflint security rules, KICS scanner, S3 encryption/public access/versioning checks, IAM least-privilege patterns, security group rule review (0.0.0.0/0), KMS key rotation, VPC flow logs, CloudTrail enabled, pre-commit hooks, SARIF output for GitHub Security tab, policy-as-code with OPA/Sentinel

The English summary is being prepared. Showing the original description.

Streaming pipeline reliability — exactly-once semantics (idempotent producer/transactional API/read_committed/Kafka Streams EOS), at-least-once patterns with idempotent consumers, dead letter queue (DLQ) design with Kafka Connect SMT/custom error handler, poison message handling, reprocessing strategy, watermarks and late data handling, checkpoint/savepoint for Flink/Spark Streaming, consumer group rebalance storm prevention, end-to-end latency SLO monitoring, streaming reliability checklist

The English summary is being prepared. Showing the original description.

Secrets management review — HashiCorp Vault (KV v2/dynamic credentials/lease renewal/audit log), External Secrets Operator (Vault/AWS SSM/GCP Secret Manager → K8s secrets), secret rotation strategies (database dynamic credentials/TLS cert-manager), detecting secrets in code (gitleaks/truffleHog/Semgrep), Airflow connections from Vault backend, dbt profiles without hardcoded credentials, CI/CD secrets (GitHub OIDC/GitLab CI variables), no plaintext secrets in logs/configs

The English summary is being prepared. Showing the original description.

Workflow orchestration comparison and review — Airflow vs Prefect vs Dagster vs Temporal (architecture, execution model, dependency management, observability), when to choose each, migration patterns from Airflow to Prefect/Dagster, orchestrator evaluation scorecard, event-driven vs schedule-based triggering, Software-Defined Assets (Dagster) vs task-based (Airflow), hybrid orchestration patterns

The English summary is being prepared. Showing the original description.

Prometheus optimization — recording rules (level:metric:operations naming), cardinality explosion diagnosis (tsdb status API), scrape interval tuning, remote write configuration (batch size/queue capacity), Thanos/Mimir for long-term retention, federation vs remote write, AlertManager routing trees (inhibition/silencing), absent() alerts for missing series, relabeling to drop expensive labels, TSDB compaction, ServiceMonitor vs PodMonitor patterns

The English summary is being prepared. Showing the original description.

OpenTelemetry instrumentation for data platforms — Python auto-instrumentation (opentelemetry-bootstrap), manual span creation for ETL tasks, trace context propagation across Airflow→Spark→dbt, OTLP exporter configuration, W3C TraceContext headers in Kafka messages, resource attributes (service.name/environment/version), Baggage for partition metadata propagation, Collector pipeline (tail sampling), correlation between traces/logs/metrics, Grafana Tempo integration

The English summary is being prepared. Showing the original description.

Kubernetes storage review — PVC lifecycle (provisioning/binding/reclaim), StorageClass selection (SSD/HDD/NVMe), dynamic vs static provisioning, StatefulSet volume management, volume performance tuning (ReadWriteMany vs ReadWriteOnce), CSI drivers (AWS EBS/GCP PD/Ceph/Longhorn), PVC resize, snapshot and backup, storage class access modes, orphaned PVs detection, shared storage patterns (NFS/CephFS)

The English summary is being prepared. Showing the original description.