Results for “data flow”

数据工程 — 数据平台从业者的认知操作系统, 覆盖把数据从源系统搬运成可靠 / 可查询 / 可信赖形态供分析 / ML / 数据产品消费的全生命周期 (生成 → 摄取 → 存储 → 转换 → 服务 + 安全/数据管理/DataOps/数据架构/编排/软件工程 六条暗流, Reis & Housley 框架): 摄取与集成 (批 + CDC 变更数据捕获 Debezium + EL 工具 Fivetran/Airbyte/Meltano/dlt + Kafka Connect + schema drift) / 存储与文件表格式 (对象存储数据湖 + 列存 Parquet/ORC/Arrow/Avro + 开放表格式 Apache Iceberg/Delta Lake/Apache Hudi + lakehouse + 分区/compaction) / 转换与建模 (ELT dbt/SQLMesh + Spark + 维度建模 Kimball + Inmon + Data Vault + 大宽表 OBT + 渐变维 SCD + 增量模型 + 语义/指标层) / 编排与工作流 (Apache Airflow/Dagster/Prefect/Mage/Kestra/Apache DolphinScheduler + DAG + 幂等 + 回填 backfill + 数据资产调度) / 批流与实时 (Apache Kafka/Apache Flink/Spark Structured Streaming/Kinesis/Pulsar/Redpanda + Lambda vs Kappa + watermark/窗口/exactly-once + 流式 SQL Materialize/RisingWave + 实时 OLAP ClickHouse/Apache Druid/Apache Pinot/StarRocks/Apache Doris) / 数仓与查询引擎 (Snowflake/BigQuery/Redshift/Databricks SQL/Trino/Presto/DuckDB/Polars + 存算分离 + MPP) / 数据质量测试与可观测性 (dbt tests/Great Expectations/Soda + 数据契约 + Monte Carlo data downtime + 新鲜度/量/schema 异常检测) / 数据治理编目与血缘 (DataHub/Amundsen/OpenMetadata/Unity Catalog + 列级血缘 + PII 分类 + 访问控制 + GDPR) / DataOps 与可靠性 (数据 CI/CD + 转换版本控制 + 环境隔离 + 幂等重处理 + 数据 SLA/SLO + 计算存储 FinOps) / 数据架构范式 (现代数据栈 + lakehouse + data mesh + data fabric + 去中心化 vs 中心化所有权) / 分析工程角色 (dbt 时代连接数据工程与分析的桥) — 不含 数据科学/ML 建模本身 (是下游消费者) / BI 仪表盘制作 (serving 下游) / 数据分析报表为终点 / 'data engineer = 跑 Hadoop 的' 过时窄化 / 通用后端应用开发 (平行学科) (Data Engineering — the cognitive operating system of practitioners who design, build, and operate the data platform: moving data from source systems into reliable, queryable, trustworthy form for analytics / ML / products, covering (a) the data engineering lifecycle (generation → ingestion → storage → transformation → serving, with the undercurrents security / data management / DataOps / data architecture / orchestration / software engineering — Reis & Housley framing), (b) ingestion & integration (batch + CDC change-data-capture with Debezium, EL tools Fivetran / Airbyte / Meltano / dlt, Kafka Connect, API + file + database sources, schema drift handling), (c) storage & file/table formats (object storage data lakes, columnar formats Parquet / ORC / Arrow / Avro, open table formats Apache Iceberg / Delta Lake / Apache Hudi, lakehouse architecture, partitioning / compaction / Z-ordering), (d) transformation & modeling (ELT with dbt / SQLMesh, Spark, dimensional modeling Kimball, Inmon CIF, Data Vault, One Big Table / wide tables, normalization vs denormalization, slowly changing dimensions, incremental models, the semantic / metrics layer), (e) orchestration & workflow (Apache Airflow, Dagster, Prefect, Mage, Kestra, Apache DolphinScheduler, DAGs, idempotency, backfills, data-aware / asset-based scheduling), (f) batch vs streaming & real-time (Apache Kafka, Apache Flink, Spark Structured Streaming, Kinesis / Pulsar / Redpanda, the Lambda vs Kappa debate, watermarks / windowing / exactly-once, streaming SQL Materialize / RisingWave, real-time OLAP ClickHouse / Apache Druid / Apache Pinot / StarRocks / Apache Doris), (g) warehouses & query engines (Snowflake, BigQuery, Redshift, Databricks SQL, Trino / Presto, DuckDB, Polars, decoupled storage & compute, MPP), (h) data quality, testing & observability (dbt tests, Great Expectations, Soda, data contracts, Monte Carlo / data downtime, freshness / volume / schema anomaly detection, unit / integration testing of pipelines), (i) data governance, catalog & lineage (DataHub, Amundsen, OpenMetadata, Unity Catalog, column-level lineage, PII / data classification, access control, GDPR / data privacy), (j) DataOps & reliability (CI/CD for data, version control of transformations, environments, idempotent reprocessing, SLAs / SLOs for data, cost / FinOps for compute & storage), (k) data architecture paradigms (modern data stack, data lakehouse, data mesh, data fabric, decentralized vs centralized ownership), (l) the analytics engineering role (the dbt-era bridge between data engineering and analysis); N

The English summary is being prepared. Showing the original description.

Build machine learning workflows that connect data checks, training, evaluation, and deployment. The skill implements experiment tracking, model version management, and automated retraining, with reproducible environments and explicit validation thresholds before models enter production.

• Automate model training and retraining workflows
• Record and compare machine learning experiments
• Build a feature store for model input data

Implement asynchronous work and changing data streams in Android and Kotlin Multiplatform apps using Kotlin Coroutines and Flow. Apply patterns for parallel loading, UI state updates, cancellation, error handling, and tests that check emitted values.

• Loading several data sources in parallel
• Managing screen state with StateFlow
• Handling search input delays and network retries

Design systems that collect, transform, and store data. Choose between scheduled and continuous processing, generate workflow configurations, validate data quality, and analyze performance using guidance for tools including Python, SQL, Spark, Airflow, dbt, and Kafka.

• Designing data ingestion and transformation pipelines
• Choosing scheduled or real-time data processing
• Checking data freshness, completeness, and uniqueness

Analyze flow cytometry data in Python by correcting fluorescence spillover and applying gates that select cell populations. Produce population counts, percentages, and fluorescence summaries, or reproduce supported FlowJo 10 workspace analyses with recorded analysis settings.

• Applying the same gating strategy to multiple FCS samples
• Reporting cell-population counts and percentages with denominators
• Reproducing supported FlowJo 10 analyses in Python

Generate Python code for Dataverse SDK workflows, including retries, bulk data changes, efficient queries, table metadata, and large file uploads. The skill also covers pandas integration and asks for type hints, docstrings, and official API references.

• Implement retries for temporary Dataverse errors
• Create bulk data operations and efficient queries
• Manage custom tables and metadata caches

Configure and troubleshoot Apache Beam runners, the engines that execute data processing pipelines. Covers local testing, Dataflow, Flink, and Spark, with Java and Python examples, runner validation tests, and guidance on logs and execution settings.

• Test a Beam pipeline locally
• Configure pipeline execution on Dataflow
• Review Flink or Spark execution settings

Use when creating or managing CRM Analytics datasets: configuring dataflows, scheduling refreshes, selecting fields, handling date types, managing row-count growth, or troubleshooting stale or broken datasets. Triggers: 'dataset creation', 'dataflow schedule', 'dataflow quota', 'date field not filterable', 'dataset row limit', 'CRM Analytics data refresh', 'field type mismatch in dataset'. NOT for making an existing dataset smaller or faster — use data/analytics-dataset-optimization. NOT for choosing between Reports, CRM Analytics and Tableau in the first place — use admin/einstein-analytics-basics.

The English summary is being prepared. Showing the original description.

Monitor Power Automate across an organization's tenant using FlowStudio MCP's cached data. Review failure rates, run-health trends, flow and app inventories, and inactive owners to prepare health and compliance reports. Requires FlowStudio for Teams or MCP Pro+.

• Identify flows with high failure rates across the organization
• Review run-health trends over a selected period
• Inventory environments, flows, apps, and connections

Analyze NetFlow v9 and IPFIX records, which summarize network connections, to spot traffic spikes, scanning, unusual outbound transfers, and periodic connections. Compare traffic against normal patterns and produce a prioritized findings report.

• Investigating sudden increases in network traffic
• Checking unusual outbound transfers for possible data theft
• Finding periodic connections to external control servers

Design n8n automations for service integrations, scheduled tasks, databases, and AI agents. Choose from six workflow patterns, plan data flow and error handling, check connections before activation, and improve workflows that process large numbers of items.

• Send form submissions to Slack
• Email a scheduled analytics report
• Synchronize records between databases

Organize Power Automate flows by business impact and responsible team using FlowStudio’s cached records. Find resources without active owners, audit connections to other services, review compliance metadata, and configure notifications for failures or missing runs.

• Review flows for missing governance information
• Find resources owned by departing employees
• Audit connectors and premium license needs

Assess potential data breach impact by reviewing sensitive data and its movement through a codebase. Produce an inventory, exposure rankings, financial estimates, and prioritized safeguards, distinguishing legal source figures from planning estimates.

• Inventory sensitive data before a security review
• Trace where customer data could leak
• Estimate breach impact and costs for planning

Build data processing pipelines for machine learning, from preprocessing to batch predictions. Uses Ray Data to handle datasets larger than memory, distribute work across CPUs and GPUs, and feed data into training frameworks.

• Preprocessing datasets larger than memory
• Running batch predictions across multiple GPUs
• Preparing image and tabular data for training

Build data processing pipelines for machine learning, from preprocessing to batch predictions. Uses Ray Data to handle datasets larger than memory, distribute work across CPUs and GPUs, and feed data into training frameworks.

• Preprocessing datasets larger than memory
• Running batch predictions across multiple GPUs
• Preparing image and tabular data for training

Use this skill when configuring Data Manager in CRM Analytics: enabling objects for sync, scheduling data sync runs, setting up remote connections to external databases, monitoring sync status and error logs, or troubleshooting connected object issues. Trigger keywords: data sync, connected objects, Data Manager, sync schedule, remote connection, Snowflake connector, sync error, incremental sync, CRM Analytics data pipeline. NOT for pushing external data in programmatically via the External Data API — use data/analytics-external-data. NOT for authoring dataflow nodes — use admin/analytics-dataflow-development.

The English summary is being prepared. Showing the original description.

Build and operate bioinformatics workflows on Latch using Python, Nextflow, or Snakemake. Configure computing resources and input forms, move data, connect Registry records, and register, debug, launch, and monitor analysis runs.

• Create and register Python analysis workflows
• Package Nextflow or Snakemake pipelines for Latch
• Connect Latch Data and Registry records

Helps organize biological datasets and models in LaminDB. Covers data registration, table validation, annotation with standardized biological terms, metadata searches, and tracking analysis inputs and outputs so researchers can preserve the context needed to repeat their work.

• Register and organize biological datasets
• Validate DataFrame or AnnData metadata
• Standardize annotations with biological vocabularies

Build Apache Airflow data pipelines with clear task dependencies and schedules. This skill guides workflow design, retries, monitoring, local testing, and production deployment, with patterns for investigating failed runs.

• Designing data pipelines and task dependencies
• Scheduling recurring batch jobs
• Testing Airflow workflows locally

Investigate failed Power Automate cloud flows through FlowStudio MCP. Inspect each action’s inputs and outputs, trace incorrect data upstream, fix expression or data issues, and replay a previous run to check the repair.

• Investigate why a cloud flow failed
• Find null values or incorrect field paths in expressions
• Inspect HTTP error response details

Connect an AI agent to Power Automate through FlowStudio MCP. This skill covers authentication, discovering available tools, reusable Python or Node.js helpers, and extracting useful data from responses too large to display directly.

• Set up authentication for a Power Automate connection
• Discover available tools for a task
• Build a reusable Python or Node.js helper

Trace where BigQuery and Cloud Storage data comes from and where it goes. The skill summarizes table or column lineage, meaning data origins and movement, in a readable Markdown report to support data quality investigations.

• Investigating data origins during a quality issue
• Summarizing data sources and destinations
• Tracing the flow of a specific column

Plan data ingestion and transformation workflows with Apache Airflow, Spark, and dbt. Get guidance on processing changed records, validating data quality, and rerunning failed or historical jobs without creating duplicate results.

• Designing data ingestion and transformation workflows
• Processing only new or changed records
• Adding data quality checks between processing stages

Run supported Datagma operations through Rube MCP, a connection between agents and external services. The skill covers tool discovery, connection checks, required input formats, and workflows for executing multiple operations.

• Find supported tools for a Datagma task
• Verify the Datagma connection is active
• Run multiple Datagma operations together