“data pipeline”的搜索结果

数据工程 — 数据平台从业者的认知操作系统, 覆盖把数据从源系统搬运成可靠 / 可查询 / 可信赖形态供分析 / ML / 数据产品消费的全生命周期 (生成 → 摄取 → 存储 → 转换 → 服务 + 安全/数据管理/DataOps/数据架构/编排/软件工程 六条暗流, Reis & Housley 框架): 摄取与集成 (批 + CDC 变更数据捕获 Debezium + EL 工具 Fivetran/Airbyte/Meltano/dlt + Kafka Connect + schema drift) / 存储与文件表格式 (对象存储数据湖 + 列存 Parquet/ORC/Arrow/Avro + 开放表格式 Apache Iceberg/Delta Lake/Apache Hudi + lakehouse + 分区/compaction) / 转换与建模 (ELT dbt/SQLMesh + Spark + 维度建模 Kimball + Inmon + Data Vault + 大宽表 OBT + 渐变维 SCD + 增量模型 + 语义/指标层) / 编排与工作流 (Apache Airflow/Dagster/Prefect/Mage/Kestra/Apache DolphinScheduler + DAG + 幂等 + 回填 backfill + 数据资产调度) / 批流与实时 (Apache Kafka/Apache Flink/Spark Structured Streaming/Kinesis/Pulsar/Redpanda + Lambda vs Kappa + watermark/窗口/exactly-once + 流式 SQL Materialize/RisingWave + 实时 OLAP ClickHouse/Apache Druid/Apache Pinot/StarRocks/Apache Doris) / 数仓与查询引擎 (Snowflake/BigQuery/Redshift/Databricks SQL/Trino/Presto/DuckDB/Polars + 存算分离 + MPP) / 数据质量测试与可观测性 (dbt tests/Great Expectations/Soda + 数据契约 + Monte Carlo data downtime + 新鲜度/量/schema 异常检测) / 数据治理编目与血缘 (DataHub/Amundsen/OpenMetadata/Unity Catalog + 列级血缘 + PII 分类 + 访问控制 + GDPR) / DataOps 与可靠性 (数据 CI/CD + 转换版本控制 + 环境隔离 + 幂等重处理 + 数据 SLA/SLO + 计算存储 FinOps) / 数据架构范式 (现代数据栈 + lakehouse + data mesh + data fabric + 去中心化 vs 中心化所有权) / 分析工程角色 (dbt 时代连接数据工程与分析的桥) — 不含 数据科学/ML 建模本身 (是下游消费者) / BI 仪表盘制作 (serving 下游) / 数据分析报表为终点 / 'data engineer = 跑 Hadoop 的' 过时窄化 / 通用后端应用开发 (平行学科) (Data Engineering — the cognitive operating system of practitioners who design, build, and operate the data platform: moving data from source systems into reliable, queryable, trustworthy form for analytics / ML / products, covering (a) the data engineering lifecycle (generation → ingestion → storage → transformation → serving, with the undercurrents security / data management / DataOps / data architecture / orchestration / software engineering — Reis & Housley framing), (b) ingestion & integration (batch + CDC change-data-capture with Debezium, EL tools Fivetran / Airbyte / Meltano / dlt, Kafka Connect, API + file + database sources, schema drift handling), (c) storage & file/table formats (object storage data lakes, columnar formats Parquet / ORC / Arrow / Avro, open table formats Apache Iceberg / Delta Lake / Apache Hudi, lakehouse architecture, partitioning / compaction / Z-ordering), (d) transformation & modeling (ELT with dbt / SQLMesh, Spark, dimensional modeling Kimball, Inmon CIF, Data Vault, One Big Table / wide tables, normalization vs denormalization, slowly changing dimensions, incremental models, the semantic / metrics layer), (e) orchestration & workflow (Apache Airflow, Dagster, Prefect, Mage, Kestra, Apache DolphinScheduler, DAGs, idempotency, backfills, data-aware / asset-based scheduling), (f) batch vs streaming & real-time (Apache Kafka, Apache Flink, Spark Structured Streaming, Kinesis / Pulsar / Redpanda, the Lambda vs Kappa debate, watermarks / windowing / exactly-once, streaming SQL Materialize / RisingWave, real-time OLAP ClickHouse / Apache Druid / Apache Pinot / StarRocks / Apache Doris), (g) warehouses & query engines (Snowflake, BigQuery, Redshift, Databricks SQL, Trino / Presto, DuckDB, Polars, decoupled storage & compute, MPP), (h) data quality, testing & observability (dbt tests, Great Expectations, Soda, data contracts, Monte Carlo / data downtime, freshness / volume / schema anomaly detection, unit / integration testing of pipelines), (i) data governance, catalog & lineage (DataHub, Amundsen, OpenMetadata, Unity Catalog, column-level lineage, PII / data classification, access control, GDPR / data privacy), (j) DataOps & reliability (CI/CD for data, version control of transformations, environments, idempotent reprocessing, SLAs / SLOs for data, cost / FinOps for compute & storage), (k) data architecture paradigms (modern data stack, data lakehouse, data mesh, data fabric, decentralized vs centralized ownership), (l) the analytics engineering role (the dbt-era bridge between data engineering and analysis); N

中文简介正在准备中。当前显示原文说明。

Design systems that collect, transform, and store data. Choose between scheduled and continuous processing, generate workflow configurations, validate data quality, and analyze performance using guidance for tools including Python, SQL, Spark, Airflow, dbt, and Kafka.

中文简介正在准备中。当前显示英语简介。

• Designing data ingestion and transformation pipelines
• Choosing scheduled or real-time data processing
• Checking data freshness, completeness, and uniqueness

Build custom cognee processing by chaining Python tasks, defining data types, and saving results to graph and vector databases. Use it to enrich existing data, inspect run status, and troubleshoot how tasks pass data.

中文简介正在准备中。当前显示英语简介。

• Building custom extraction and processing steps
• Saving custom data types to graph and vector databases
• Adding information to an existing graph

Build data processing pipelines for machine learning, from preprocessing to batch predictions. Uses Ray Data to handle datasets larger than memory, distribute work across CPUs and GPUs, and feed data into training frameworks.

中文简介正在准备中。当前显示英语简介。

• Preprocessing datasets larger than memory
• Running batch predictions across multiple GPUs
• Preparing image and tabular data for training

Build data processing pipelines for machine learning, from preprocessing to batch predictions. Uses Ray Data to handle datasets larger than memory, distribute work across CPUs and GPUs, and feed data into training frameworks.

中文简介正在准备中。当前显示英语简介。

• Preprocessing datasets larger than memory
• Running batch predictions across multiple GPUs
• Preparing image and tabular data for training

Use this skill when configuring Data Manager in CRM Analytics: enabling objects for sync, scheduling data sync runs, setting up remote connections to external databases, monitoring sync status and error logs, or troubleshooting connected object issues. Trigger keywords: data sync, connected objects, Data Manager, sync schedule, remote connection, Snowflake connector, sync error, incremental sync, CRM Analytics data pipeline. NOT for pushing external data in programmatically via the External Data API — use data/analytics-external-data. NOT for authoring dataflow nodes — use admin/analytics-dataflow-development.

中文简介正在准备中。当前显示原文说明。

World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, real-time streaming, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, Flink, Kinesis, and modern data stack. Includes data modeling, pipeline orchestration, data quality, streaming quality monitoring, and DataOps. Use when designing data architectures, building batch or streaming data pipelines, optimizing data workflows, or implementing data governance.

中文简介正在准备中。当前显示原文说明。

Configure and troubleshoot Apache Beam runners, the engines that execute data processing pipelines. Covers local testing, Dataflow, Flink, and Spark, with Java and Python examples, runner validation tests, and guidance on logs and execution settings.

中文简介正在准备中。当前显示英语简介。

• Test a Beam pipeline locally
• Configure pipeline execution on Dataflow
• Review Flink or Spark execution settings

Import data into the AWS data lake from S3 files, local uploads, JDBC databases (Oracle, SQL Server, PostgreSQL, MySQL, RDS, Aurora), Amazon Redshift, Snowflake, BigQuery, DynamoDB, or existing Glue catalog tables (migration). Default target is S3 Tables; standard Iceberg on a general purpose bucket is supported where S3 Tables is not adopted. Handles one-time loads, recurring pipelines, migrations. Triggers on: import data, load data, ingest, sync database, migrate table, move data to AWS, set up pipeline, ETL, pull from Snowflake, query BigQuery into S3, export DynamoDB, CTAS, convert to Iceberg. Do NOT use for setting up or troubleshooting Glue connections (use connecting-to-data-source), creating empty tables (use creating-data-lake-table), running queries (use querying-data-lake), finding tables by fuzzy name (use finding-data-lake-assets), catalog audit (use exploring-data-catalog), or SaaS platforms like Salesforce, ServiceNow, SAP, MongoDB, Kafka.

中文简介正在准备中。当前显示原文说明。

Data pipeline specialist for ETL design, data quality, CDC patterns, and batch/stream processingUse when "data pipeline, etl, cdc, data quality, batch processing, stream processing, data transformation, data warehouse, data lake, data validation, data-engineering, etl, cdc, batch, streaming, data-quality, dbt, airflow, dagster, data-pipeline, ml-memory" mentioned.

中文简介正在准备中。当前显示原文说明。

Build data quality checks with Great Expectations, dbt tests, and data contracts, which define shared expectations between teams. Check missing values, duplicates, and freshness, then integrate validation into data pipelines and CI/CD.

中文简介正在准备中。当前显示英语简介。

• Check data for missing values and duplicates
• Build a dbt test suite
• Define shared data contracts between teams

Import SaaS data into DuckDB using dlt, a data loading tool, then generate a Wren project with table definitions and relationships. Verify the setup with real SQL queries, or start from an existing DuckDB file.

中文简介正在准备中。当前显示英语简介。

• Importing HubSpot or Stripe data for SQL analysis
• Creating a Wren project from an existing dlt DuckDB file
• Generating table definitions and parent-child relationships

Build Apache Spark pipelines for large datasets, including transformations, aggregations, and streaming analysis. Get PySpark or Scala code, partitioning strategies for dividing data across workers, and configuration advice to investigate slow jobs and resource usage.

中文简介正在准备中。当前显示英语简介。

• Build a pipeline to transform and aggregate large datasets
• Investigate slow Spark jobs and uneven data distribution
• Tune memory settings and data partitions

Validate data quality in CSV, JSON, and database exports by checking for missing values, type mismatches, duplicates, outliers, and schema violations. Use when building ETL pipelines, auditing data imports, checking data freshness, or ensuring data contracts between teams. Trigger words: data quality, validation, null values, duplicates, schema check, data contract, ETL, pipeline, data drift.

中文简介正在准备中。当前显示原文说明。

Plans and scripts data migrations between SQL databases: schema mapping, type conversion, batched and resumable loading, validation and cutover. Use when the user mentions "data migration," "database migration," "migrate from MySQL to PostgreSQL," "schema migration," "ETL pipeline," "data transfer," "database consolidation," "legacy migration," or "move data between databases." Covers schema analysis, mapping, transformation, batch processing, validation, and cutover planning. For query optimization during migration, see sql-optimizer.

中文简介正在准备中。当前显示原文说明。

Marketing analytics and data-driven optimization for B2B SaaS campaigns. Use this skill when analyzing marketing performance, optimizing conversion rates, running A/B tests, conducting customer research, creating marketing dashboards, building attribution models, analyzing funnel metrics, or storytelling with data. Also triggers on: marketing ops, performance analytics, CRO, A/B testing, data storytelling, customer insights, tracking plan, event taxonomy, UTM taxonomy, GA4 audit, consent mode, instrumentation contract, sample size calculation, minimum detectable effect, test duration, not enough traffic to A/B test, statistical significance, Bayesian vs frequentist, sample ratio mismatch, experiment feasibility, martech stack, marketing tech stack audit, tool consolidation, build vs buy, vendor evaluation, shadow IT tools, ESP/CRM/platform migration, switching cost, cutover plan, suppression list migration, data portability, customer interviews, survey design, sampling frame, non-response bias, how many people should I interview, thematic saturation, voice of customer, jobs-to-be-done, buyer persona research, is this finding real, data enrichment, which enrichment provider should we buy, waterfall enrichment, contact data quality, match rate, our list data is bad, email verification, catch-all domain, accept-all, bounce rate from bad data, B2B data decay, how often should we re-verify contacts, ZoomInfo vs Apollo vs Clearbit evaluation, cost per usable record, where did this list come from, GDPR Article 14 notice, purchased list compliance, suppression screening, TAM list building, demand planning, marketing plan, how many leads do we need, what pipeline target should marketing carry, funnel model, top-down vs bottom-up plan, plan of record, annual marketing planning, quarterly planning, pipeline coverage we have to create, sales cycle lag, our leads cannot close this quarter, carry-in pipeline, scenario planning, conservative moderate aggressive forecast, stretch target, reforecast, re-forecast triggers, assumption register, capacity ceiling, addressable audience ceiling, marginal cost per lead, blended CPL is wrong, we missed plan and nobody knows why, AI policy for the marketing team, AI governance, AI acceptable use policy, which AI tools are we allowed to use, can I paste customer data into ChatGPT, shadow AI, AI use register, human review of AI output, when can we stop reviewing every AI draft, can the agent send on its own, the AI made a mistake now what, pull back AI autonomy, AI evals, our AI vendor changed the model, AI output went out wrong, prompt injection, can our AI read prospect replies safely, lethal trifecta, the AI leaked internal notes, turn off auto-generated ad assets, auto-applied recommendations, AI literacy training, EU AI Act Article 4, duplicate records in the CRM, dedupe before import, which value wins in a merge, CRM cleanup.

中文简介正在准备中。当前显示原文说明。

Use when working with Aws Data Pipeline — aWS Data Pipeline workflow management and execution analysis. Covers pipeline inventory, pipeline definitions, execution status, object status, task runner health, and pipeline scheduling. Use when inspecting data pipeline health, debugging failed executions, reviewing pipeline definitions, or auditing pipeline schedules.

中文简介正在准备中。当前显示原文说明。

Build training datasets for LLM specialization from production data, frontier model distillation, and synthetic bootstrapping. Use when formatting production logs into SFT data, distilling from frontier APIs, or preparing data for fine-tuning. Covers JSONL formatting, data quality validation, deduplication, and train/eval splitting.

中文简介正在准备中。当前显示原文说明。

Convert laboratory instrument files into Allotrope Simple Model (ASM) JSON, a standardized data format, or tabular CSV. Detect instrument types, validate converted data, and export Python parsing code for analysis workflows or handoff to data engineers.

中文简介正在准备中。当前显示英语简介。

• Standardizing instrument data for a laboratory management system
• Converting ELISA results into an importable CSV
• Exporting Python code to parse NanoDrop files

Set up KubeSphere’s WizTelemetry Data Pipeline to collect, transform, and route logs, audit data, events, and notifications. Configure delivery to OpenSearch, deploy agents across selected clusters, and check their operating status.

中文简介正在准备中。当前显示英语简介。

• Setting up a shared collection pipeline for observability data
• Configuring delivery to OpenSearch
• Deploying agents across multiple clusters

[omh] Data pipeline work -- an ETL or streaming job, a backfill or replay, duplicate events, a schema change downstream, a lineage question, a data-quality regression: make every rerun idempotent, bound every replay, and gate each load on observed checks. Use when the user says: data-pipelines, data pipeline, data pipelines, etl, elt, etl pipeline, etl job, etl backfill.

中文简介正在准备中。当前显示原文说明。

Use when reviewing, designing, or modifying Java enterprise systems that expose, exchange, store, process, export, or port data across users, businesses, connected products, cloud providers, APIs, event streams, AI data pipelines, data spaces, or SaaS platforms and need EU Data Act engineering controls. This should trigger for requests such as Review a Java platform for EU Data Act controls; Design data access and portability evidence; Add data-sharing request workflows, export formats, interoperability, metadata, audit logs, cloud-switching support, non-personal data safeguards, or trade-secret handoffs; Assess Data Act engineering readiness before production release. Part of Plinth Toolkit

中文简介正在准备中。当前显示原文说明。

Write a data pipeline design document. Use when the user says "pipeline design doc", "document this pipeline", "pipeline architecture doc", "data flow document", "how does this pipeline work", "design doc for ETL", "pipeline spec", or needs to capture the architecture, data flow, and operational details of a data pipeline - even if they don't explicitly say "design doc".

中文简介正在准备中。当前显示原文说明。

Expert data engineer for ETL/ELT pipelines, streaming, data warehousing. Activate on: data pipeline, ETL, ELT, data warehouse, Spark, Kafka, Airflow, dbt, data modeling, star schema, streaming data, batch processing, data quality. NOT for: API design (use api-architect), ML training (use ML skills), dashboards (use design skills).

中文简介正在准备中。当前显示原文说明。