88 个 · 按相关度排序
Guides the design of data pipelines, continuous stream processing, and data quality checks. It compares tools such as Airflow, Dagster, and Flink, and provides implementation patterns and checklists for scheduling, retries, validation, and monitoring.
中文简介正在准备中。当前显示英语简介。
Apache Airflow DAG Patterns workflow skill. Use this skill when the user needs Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.
中文简介正在准备中。当前显示原文说明。
Design and implement systems that collect, transform, validate, and store data for analysis. Covers scheduled and real-time processing, data warehouses, quality checks, monitoring, and cost tuning using tools such as Apache Spark, dbt, and Airflow.
中文简介正在准备中。当前显示英语简介。
Design systems that collect, transform, and store data. Choose between scheduled and continuous processing, generate workflow configurations, validate data quality, and analyze performance using guidance for tools including Python, SQL, Spark, Airflow, dbt, and Kafka.
中文简介正在准备中。当前显示英语简介。
Provides guidance for troubleshooting Apache Airflow DAGs (failed DAG runs and task instances) in Managed Service for Apache Airflow (MSAA; formerly Cloud Composer). Use when figuring out reasons for DAG run or task instance failures. Don't use when looking for overall recommendations for Managed Airflow environment performance.
中文简介正在准备中。当前显示原文说明。
Apache Airflow DAGs, operators, SLA monitoring, and workflow orchestration. Activate on: Airflow, DAG, operator, sensor, scheduler, task dependency, SLA, backfill, XCom. NOT for: dbt transformations (use dbt-analytics-engineer), streaming pipelines (use streaming-pipeline-architect).
中文简介正在准备中。当前显示原文说明。
Use when working with Airflow — apache Airflow workflow orchestration management. Covers DAG management, task instance monitoring, executor health, pool status, variable management, connection audit, and scheduler diagnostics. Use when checking DAG run status, investigating task failures, managing Airflow resources, or auditing configurations.
中文简介正在准备中。当前显示原文说明。
Use when working with Apache Airflow Deep — apache Airflow deep management covering DAG inventory, task instance monitoring, scheduler health, executor status, pool and variable management, connection auditing, and SLA miss tracking. Use when investigating DAG failures, analyzing task duration trends, monitoring scheduler performance, or auditing Airflow configurations.
中文简介正在准备中。当前显示原文说明。
Apache Airflow DAGs, operators, SLA monitoring, and workflow orchestration. Activate on: Airflow, DAG, operator, sensor, scheduler, task dependency, SLA, backfill, XCom. NOT for: dbt transformations (use dbt-analytics-engineer), streaming pipelines (use streaming-pipeline-architect).
中文简介正在准备中。当前显示原文说明。
Design and automate machine learning workflows from data preparation through training, validation, deployment, and monitoring. Provides guidance and templates for reproducible experiments, quality checks, gradual model rollouts, and restoring a previous model when problems occur.
中文简介正在准备中。当前显示英语简介。
Create videos of anomalous events, such as falls or fires, from seed images using Cosmos3. Run a Kubernetes workflow that generates clips, automatically labels them, and builds an anomaly dataset, with progress monitoring and result retrieval.
中文简介正在准备中。当前显示英语简介。
Apply airflow dag patterns in reproducible local data workflows with version-aware APIs, explicit assumptions, and validation. Use when the user chooses airflow dag patterns or its strengths fit the task.
中文简介正在准备中。当前显示原文说明。
Apache Airflow is a platform for programmatically authoring, scheduling, and monitoring workflows. Learn to write DAGs, use operators, set up connections, configure scheduling, and deploy with Docker Compose.
中文简介正在准备中。当前显示原文说明。
Design machine learning workflows from data preparation through training, validation, deployment, and monitoring. Use templates and guidance to connect stages, track versions and experiments, handle failures, and plan gradual releases with rollback options.
中文简介正在准备中。当前显示英语简介。
Run a Kubernetes workflow that changes clothing in cropped person images, searches person attributes, and builds augmented datasets. The skill handles input preparation, run monitoring, and result retrieval for person re-identification data work.
中文简介正在准备中。当前显示英语简介。
Authors and deploys MWAA workflow artifacts: Python Airflow DAGs for provisioned environments or YAML workflow files for Serverless. Covers operator selection, timeout design, retry strategy, scheduling, failure notifications, idempotency, and MWAA Serverless schema compliance. Deploys the artifact (S3 DAG upload or Serverless CreateWorkflow/UpdateWorkflow), creates an environment inline when approved, and redeploys fixes, then optionally hands off to testing-mwaa-workflow. Triggers on: create a DAG, write a pipeline, build a workflow, orchestrate tasks, Airflow DAG, data pipeline, schedule a job, deploy a DAG, deploy a workflow, YAML workflow. Not applicable to converting or migrating existing DAGs between provisioned and serverless (conversion is out of scope), running or smoke-testing a deployed workflow (handled by testing-mwaa-workflow) or diagnosing a failed run (handled by debugging-mwaa-workflow).
中文简介正在准备中。当前显示原文说明。
Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.
中文简介正在准备中。当前显示原文说明。
Data engineering for batch and streaming pipelines with Airflow, dbt, Spark, and Kafka. Use when designing data architectures, building pipelines, adding data-quality checks, optimizing ETL/ELT, or troubleshooting pipeline failures.
中文简介正在准备中。当前显示原文说明。
Create Apache Airflow DAGs for construction data pipelines. Orchestrate ETL, validation, and reporting workflows.
中文简介正在准备中。当前显示原文说明。
Route Apache Airflow repo tasks across Dag authoring, operations, providers, deployment, and contribution workflows.
中文简介正在准备中。当前显示原文说明。
Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.
中文简介正在准备中。当前显示原文说明。
数据工程 — 数据平台从业者的认知操作系统, 覆盖把数据从源系统搬运成可靠 / 可查询 / 可信赖形态供分析 / ML / 数据产品消费的全生命周期 (生成 → 摄取 → 存储 → 转换 → 服务 + 安全/数据管理/DataOps/数据架构/编排/软件工程 六条暗流, Reis & Housley 框架): 摄取与集成 (批 + CDC 变更数据捕获 Debezium + EL 工具 Fivetran/Airbyte/Meltano/dlt + Kafka Connect + schema drift) / 存储与文件表格式 (对象存储数据湖 + 列存 Parquet/ORC/Arrow/Avro + 开放表格式 Apache Iceberg/Delta Lake/Apache Hudi + lakehouse + 分区/compaction) / 转换与建模 (ELT dbt/SQLMesh + Spark + 维度建模 Kimball + Inmon + Data Vault + 大宽表 OBT + 渐变维 SCD + 增量模型 + 语义/指标层) / 编排与工作流 (Apache Airflow/Dagster/Prefect/Mage/Kestra/Apache DolphinScheduler + DAG + 幂等 + 回填 backfill + 数据资产调度) / 批流与实时 (Apache Kafka/Apache Flink/Spark Structured Streaming/Kinesis/Pulsar/Redpanda + Lambda vs Kappa + watermark/窗口/exactly-once + 流式 SQL Materialize/RisingWave + 实时 OLAP ClickHouse/Apache Druid/Apache Pinot/StarRocks/Apache Doris) / 数仓与查询引擎 (Snowflake/BigQuery/Redshift/Databricks SQL/Trino/Presto/DuckDB/Polars + 存算分离 + MPP) / 数据质量测试与可观测性 (dbt tests/Great Expectations/Soda + 数据契约 + Monte Carlo data downtime + 新鲜度/量/schema 异常检测) / 数据治理编目与血缘 (DataHub/Amundsen/OpenMetadata/Unity Catalog + 列级血缘 + PII 分类 + 访问控制 + GDPR) / DataOps 与可靠性 (数据 CI/CD + 转换版本控制 + 环境隔离 + 幂等重处理 + 数据 SLA/SLO + 计算存储 FinOps) / 数据架构范式 (现代数据栈 + lakehouse + data mesh + data fabric + 去中心化 vs 中心化所有权) / 分析工程角色 (dbt 时代连接数据工程与分析的桥) — 不含 数据科学/ML 建模本身 (是下游消费者) / BI 仪表盘制作 (serving 下游) / 数据分析报表为终点 / 'data engineer = 跑 Hadoop 的' 过时窄化 / 通用后端应用开发 (平行学科) (Data Engineering — the cognitive operating system of practitioners who design, build, and operate the data platform: moving data from source systems into reliable, queryable, trustworthy form for analytics / ML / products, covering (a) the data engineering lifecycle (generation → ingestion → storage → transformation → serving, with the undercurrents security / data management / DataOps / data architecture / orchestration / software engineering — Reis & Housley framing), (b) ingestion & integration (batch + CDC change-data-capture with Debezium, EL tools Fivetran / Airbyte / Meltano / dlt, Kafka Connect, API + file + database sources, schema drift handling), (c) storage & file/table formats (object storage data lakes, columnar formats Parquet / ORC / Arrow / Avro, open table formats Apache Iceberg / Delta Lake / Apache Hudi, lakehouse architecture, partitioning / compaction / Z-ordering), (d) transformation & modeling (ELT with dbt / SQLMesh, Spark, dimensional modeling Kimball, Inmon CIF, Data Vault, One Big Table / wide tables, normalization vs denormalization, slowly changing dimensions, incremental models, the semantic / metrics layer), (e) orchestration & workflow (Apache Airflow, Dagster, Prefect, Mage, Kestra, Apache DolphinScheduler, DAGs, idempotency, backfills, data-aware / asset-based scheduling), (f) batch vs streaming & real-time (Apache Kafka, Apache Flink, Spark Structured Streaming, Kinesis / Pulsar / Redpanda, the Lambda vs Kappa debate, watermarks / windowing / exactly-once, streaming SQL Materialize / RisingWave, real-time OLAP ClickHouse / Apache Druid / Apache Pinot / StarRocks / Apache Doris), (g) warehouses & query engines (Snowflake, BigQuery, Redshift, Databricks SQL, Trino / Presto, DuckDB, Polars, decoupled storage & compute, MPP), (h) data quality, testing & observability (dbt tests, Great Expectations, Soda, data contracts, Monte Carlo / data downtime, freshness / volume / schema anomaly detection, unit / integration testing of pipelines), (i) data governance, catalog & lineage (DataHub, Amundsen, OpenMetadata, Unity Catalog, column-level lineage, PII / data classification, access control, GDPR / data privacy), (j) DataOps & reliability (CI/CD for data, version control of transformations, environments, idempotent reprocessing, SLAs / SLOs for data, cost / FinOps for compute & storage), (k) data architecture paradigms (modern data stack, data lakehouse, data mesh, data fabric, decentralized vs centralized ownership), (l) the analytics engineering role (the dbt-era bridge between data engineering and analysis); N
中文简介正在准备中。当前显示原文说明。
Expert guide for Data Pipelines, ETL/ELT, and Analytics Engineering. Covers dbt, Apache Airflow, Dagster, BigQuery, ClickHouse, and DuckDB / Panduan ahli untuk Data Pipelines, ETL/ELT. Mencakup dbt, Airflow, Dagster, BigQuery, ClickHouse, dan DuckDB.
中文简介正在准备中。当前显示原文说明。
Thermal engineering for infrastructure: heat transfer (conduction, convection, radiation), cooling load analysis for data centers, heat exchanger sizing via LMTD and e-NTU, PUE/TUE/WUE efficiency metrics, and airflow management patterns. Activates for thermal analysis, data center cooling design, heat exchanger sizing, efficiency calculations, and airflow management.
中文简介正在准备中。当前显示原文说明。