「data pipeline」の検索結果

数据工程 — 数据平台从业者的认知操作系统, 覆盖把数据从源系统搬运成可靠 / 可查询 / 可信赖形态供分析 / ML / 数据产品消费的全生命周期 (生成 → 摄取 → 存储 → 转换 → 服务 + 安全/数据管理/DataOps/数据架构/编排/软件工程 六条暗流, Reis & Housley 框架): 摄取与集成 (批 + CDC 变更数据捕获 Debezium + EL 工具 Fivetran/Airbyte/Meltano/dlt + Kafka Connect + schema drift) / 存储与文件表格式 (对象存储数据湖 + 列存 Parquet/ORC/Arrow/Avro + 开放表格式 Apache Iceberg/Delta Lake/Apache Hudi + lakehouse + 分区/compaction) / 转换与建模 (ELT dbt/SQLMesh + Spark + 维度建模 Kimball + Inmon + Data Vault + 大宽表 OBT + 渐变维 SCD + 增量模型 + 语义/指标层) / 编排与工作流 (Apache Airflow/Dagster/Prefect/Mage/Kestra/Apache DolphinScheduler + DAG + 幂等 + 回填 backfill + 数据资产调度) / 批流与实时 (Apache Kafka/Apache Flink/Spark Structured Streaming/Kinesis/Pulsar/Redpanda + Lambda vs Kappa + watermark/窗口/exactly-once + 流式 SQL Materialize/RisingWave + 实时 OLAP ClickHouse/Apache Druid/Apache Pinot/StarRocks/Apache Doris) / 数仓与查询引擎 (Snowflake/BigQuery/Redshift/Databricks SQL/Trino/Presto/DuckDB/Polars + 存算分离 + MPP) / 数据质量测试与可观测性 (dbt tests/Great Expectations/Soda + 数据契约 + Monte Carlo data downtime + 新鲜度/量/schema 异常检测) / 数据治理编目与血缘 (DataHub/Amundsen/OpenMetadata/Unity Catalog + 列级血缘 + PII 分类 + 访问控制 + GDPR) / DataOps 与可靠性 (数据 CI/CD + 转换版本控制 + 环境隔离 + 幂等重处理 + 数据 SLA/SLO + 计算存储 FinOps) / 数据架构范式 (现代数据栈 + lakehouse + data mesh + data fabric + 去中心化 vs 中心化所有权) / 分析工程角色 (dbt 时代连接数据工程与分析的桥) — 不含 数据科学/ML 建模本身 (是下游消费者) / BI 仪表盘制作 (serving 下游) / 数据分析报表为终点 / 'data engineer = 跑 Hadoop 的' 过时窄化 / 通用后端应用开发 (平行学科) (Data Engineering — the cognitive operating system of practitioners who design, build, and operate the data platform: moving data from source systems into reliable, queryable, trustworthy form for analytics / ML / products, covering (a) the data engineering lifecycle (generation → ingestion → storage → transformation → serving, with the undercurrents security / data management / DataOps / data architecture / orchestration / software engineering — Reis & Housley framing), (b) ingestion & integration (batch + CDC change-data-capture with Debezium, EL tools Fivetran / Airbyte / Meltano / dlt, Kafka Connect, API + file + database sources, schema drift handling), (c) storage & file/table formats (object storage data lakes, columnar formats Parquet / ORC / Arrow / Avro, open table formats Apache Iceberg / Delta Lake / Apache Hudi, lakehouse architecture, partitioning / compaction / Z-ordering), (d) transformation & modeling (ELT with dbt / SQLMesh, Spark, dimensional modeling Kimball, Inmon CIF, Data Vault, One Big Table / wide tables, normalization vs denormalization, slowly changing dimensions, incremental models, the semantic / metrics layer), (e) orchestration & workflow (Apache Airflow, Dagster, Prefect, Mage, Kestra, Apache DolphinScheduler, DAGs, idempotency, backfills, data-aware / asset-based scheduling), (f) batch vs streaming & real-time (Apache Kafka, Apache Flink, Spark Structured Streaming, Kinesis / Pulsar / Redpanda, the Lambda vs Kappa debate, watermarks / windowing / exactly-once, streaming SQL Materialize / RisingWave, real-time OLAP ClickHouse / Apache Druid / Apache Pinot / StarRocks / Apache Doris), (g) warehouses & query engines (Snowflake, BigQuery, Redshift, Databricks SQL, Trino / Presto, DuckDB, Polars, decoupled storage & compute, MPP), (h) data quality, testing & observability (dbt tests, Great Expectations, Soda, data contracts, Monte Carlo / data downtime, freshness / volume / schema anomaly detection, unit / integration testing of pipelines), (i) data governance, catalog & lineage (DataHub, Amundsen, OpenMetadata, Unity Catalog, column-level lineage, PII / data classification, access control, GDPR / data privacy), (j) DataOps & reliability (CI/CD for data, version control of transformations, environments, idempotent reprocessing, SLAs / SLOs for data, cost / FinOps for compute & storage), (k) data architecture paradigms (modern data stack, data lakehouse, data mesh, data fabric, decentralized vs centralized ownership), (l) the analytics engineering role (the dbt-era bridge between data engineering and analysis); N

日本語の概要は準備中です。原文の説明を表示しています。

Use this skill when configuring Data Manager in CRM Analytics: enabling objects for sync, scheduling data sync runs, setting up remote connections to external databases, monitoring sync status and error logs, or troubleshooting connected object issues. Trigger keywords: data sync, connected objects, Data Manager, sync schedule, remote connection, Snowflake connector, sync error, incremental sync, CRM Analytics data pipeline. NOT for pushing external data in programmatically via the External Data API — use data/analytics-external-data. NOT for authoring dataflow nodes — use admin/analytics-dataflow-development.

日本語の概要は準備中です。原文の説明を表示しています。

Data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when designing data architectures, building data pipelines, optimizing data workflows, implementing data governance, or troubleshooting data issues.

日本語の概要は準備中です。原文の説明を表示しています。

World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, real-time streaming, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, Flink, Kinesis, and modern data stack. Includes data modeling, pipeline orchestration, data quality, streaming quality monitoring, and DataOps. Use when designing data architectures, building batch or streaming data pipelines, optimizing data workflows, or implementing data governance.

日本語の概要は準備中です。原文の説明を表示しています。

Import data into the AWS data lake from S3 files, local uploads, JDBC databases (Oracle, SQL Server, PostgreSQL, MySQL, RDS, Aurora), Amazon Redshift, Snowflake, BigQuery, DynamoDB, or existing Glue catalog tables (migration). Default target is S3 Tables; standard Iceberg on a general purpose bucket is supported where S3 Tables is not adopted. Handles one-time loads, recurring pipelines, migrations. Triggers on: import data, load data, ingest, sync database, migrate table, move data to AWS, set up pipeline, ETL, pull from Snowflake, query BigQuery into S3, export DynamoDB, CTAS, convert to Iceberg. Do NOT use for setting up or troubleshooting Glue connections (use connecting-to-data-source), creating empty tables (use creating-data-lake-table), running queries (use querying-data-lake), finding tables by fuzzy name (use finding-data-lake-assets), catalog audit (use exploring-data-catalog), or SaaS platforms like Salesforce, ServiceNow, SAP, MongoDB, Kafka.

日本語の概要は準備中です。原文の説明を表示しています。

Data pipeline specialist for ETL design, data quality, CDC patterns, and batch/stream processingUse when "data pipeline, etl, cdc, data quality, batch processing, stream processing, data transformation, data warehouse, data lake, data validation, data-engineering, etl, cdc, batch, streaming, data-quality, dbt, airflow, dagster, data-pipeline, ml-memory" mentioned.

日本語の概要は準備中です。原文の説明を表示しています。

Validate data quality in CSV, JSON, and database exports by checking for missing values, type mismatches, duplicates, outliers, and schema violations. Use when building ETL pipelines, auditing data imports, checking data freshness, or ensuring data contracts between teams. Trigger words: data quality, validation, null values, duplicates, schema check, data contract, ETL, pipeline, data drift.

日本語の概要は準備中です。原文の説明を表示しています。

When the user needs to migrate data between databases, transform schemas, or consolidate data sources. Use when the user mentions "data migration," "database migration," "migrate from MySQL to PostgreSQL," "schema migration," "ETL pipeline," "data transfer," "database consolidation," "legacy migration," or "move data between databases." Covers schema analysis, mapping, transformation, batch processing, validation, and cutover planning. For query optimization during migration, see sql-optimizer.

日本語の概要は準備中です。原文の説明を表示しています。

Marketing analytics and data-driven optimization for B2B SaaS campaigns. Use this skill when analyzing marketing performance, optimizing conversion rates, running A/B tests, conducting customer research, creating marketing dashboards, building attribution models, analyzing funnel metrics, or storytelling with data. Also triggers on: marketing ops, performance analytics, CRO, A/B testing, data storytelling, customer insights, tracking plan, event taxonomy, UTM taxonomy, GA4 audit, consent mode, instrumentation contract, sample size calculation, minimum detectable effect, test duration, not enough traffic to A/B test, statistical significance, Bayesian vs frequentist, sample ratio mismatch, experiment feasibility, martech stack, marketing tech stack audit, tool consolidation, build vs buy, vendor evaluation, shadow IT tools, ESP/CRM/platform migration, switching cost, cutover plan, suppression list migration, data portability, customer interviews, survey design, sampling frame, non-response bias, how many people should I interview, thematic saturation, voice of customer, jobs-to-be-done, buyer persona research, is this finding real, data enrichment, which enrichment provider should we buy, waterfall enrichment, contact data quality, match rate, our list data is bad, email verification, catch-all domain, accept-all, bounce rate from bad data, B2B data decay, how often should we re-verify contacts, ZoomInfo vs Apollo vs Clearbit evaluation, cost per usable record, where did this list come from, GDPR Article 14 notice, purchased list compliance, suppression screening, TAM list building, demand planning, marketing plan, how many leads do we need, what pipeline target should marketing carry, funnel model, top-down vs bottom-up plan, plan of record, annual marketing planning, quarterly planning, pipeline coverage we have to create, sales cycle lag, our leads cannot close this quarter, carry-in pipeline, scenario planning, conservative moderate aggressive forecast, stretch target, reforecast, re-forecast triggers, assumption register, capacity ceiling, addressable audience ceiling, marginal cost per lead, blended CPL is wrong, we missed plan and nobody knows why, AI policy for the marketing team, AI governance, AI acceptable use policy, which AI tools are we allowed to use, can I paste customer data into ChatGPT, shadow AI, AI use register, human review of AI output, when can we stop reviewing every AI draft, can the agent send on its own, the AI made a mistake now what, pull back AI autonomy, AI evals, our AI vendor changed the model, AI output went out wrong, prompt injection, can our AI read prospect replies safely, lethal trifecta, the AI leaked internal notes, turn off auto-generated ad assets, auto-applied recommendations, AI literacy training, EU AI Act Article 4, duplicate records in the CRM, dedupe before import, which value wins in a merge, CRM cleanup.

日本語の概要は準備中です。原文の説明を表示しています。

Use when working with Aws Data Pipeline — aWS Data Pipeline workflow management and execution analysis. Covers pipeline inventory, pipeline definitions, execution status, object status, task runner health, and pipeline scheduling. Use when inspecting data pipeline health, debugging failed executions, reviewing pipeline definitions, or auditing pipeline schedules.

日本語の概要は準備中です。原文の説明を表示しています。

Build training datasets for LLM specialization from production data, frontier model distillation, and synthetic bootstrapping. Use when formatting production logs into SFT data, distilling from frontier APIs, or preparing data for fine-tuning. Covers JSONL formatting, data quality validation, deduplication, and train/eval splitting.

日本語の概要は準備中です。原文の説明を表示しています。

[omh] Data pipeline work -- an ETL or streaming job, a backfill or replay, duplicate events, a schema change downstream, a lineage question, a data-quality regression: make every rerun idempotent, bound every replay, and gate each load on observed checks. Use when the user says: data-pipelines, data pipeline, data pipelines, etl, elt, etl pipeline, etl job, etl backfill.

日本語の概要は準備中です。原文の説明を表示しています。

Use when reviewing, designing, or modifying Java enterprise systems that expose, exchange, store, process, export, or port data across users, businesses, connected products, cloud providers, APIs, event streams, AI data pipelines, data spaces, or SaaS platforms and need EU Data Act engineering controls. This should trigger for requests such as Review a Java platform for EU Data Act controls; Design data access and portability evidence; Add data-sharing request workflows, export formats, interoperability, metadata, audit logs, cloud-switching support, non-personal data safeguards, or trade-secret handoffs; Assess Data Act engineering readiness before production release. Part of Plinth Toolkit

日本語の概要は準備中です。原文の説明を表示しています。

Write a data pipeline design document. Use when the user says "pipeline design doc", "document this pipeline", "pipeline architecture doc", "data flow document", "how does this pipeline work", "design doc for ETL", "pipeline spec", or needs to capture the architecture, data flow, and operational details of a data pipeline - even if they don't explicitly say "design doc".

日本語の概要は準備中です。原文の説明を表示しています。

Expert data engineer for ETL/ELT pipelines, streaming, data warehousing. Activate on: data pipeline, ETL, ELT, data warehouse, Spark, Kafka, Airflow, dbt, data modeling, star schema, streaming data, batch processing, data quality. NOT for: API design (use api-architect), ML training (use ML skills), dashboards (use design skills).

日本語の概要は準備中です。原文の説明を表示しています。

Marketing analytics and data-driven optimization for B2B SaaS campaigns. Use this skill when analyzing marketing performance, optimizing conversion rates, running A/B tests, conducting customer research, creating marketing dashboards, building attribution models, analyzing funnel metrics, or storytelling with data. Also triggers on: marketing ops, performance analytics, CRO, A/B testing, data storytelling, customer insights, tracking plan, event taxonomy, UTM taxonomy, GA4 audit, consent mode, instrumentation contract, sample size calculation, minimum detectable effect, test duration, not enough traffic to A/B test, statistical significance, Bayesian vs frequentist, sample ratio mismatch, experiment feasibility, martech stack, marketing tech stack audit, tool consolidation, build vs buy, vendor evaluation, shadow IT tools, ESP/CRM/platform migration, switching cost, cutover plan, suppression list migration, data portability, customer interviews, survey design, sampling frame, non-response bias, how many people should I interview, thematic saturation, voice of customer, jobs-to-be-done, buyer persona research, is this finding real, data enrichment, which enrichment provider should we buy, waterfall enrichment, contact data quality, match rate, our list data is bad, email verification, catch-all domain, accept-all, bounce rate from bad data, B2B data decay, how often should we re-verify contacts, ZoomInfo vs Apollo vs Clearbit evaluation, cost per usable record, where did this list come from, GDPR Article 14 notice, purchased list compliance, suppression screening, TAM list building, demand planning, marketing plan, how many leads do we need, what pipeline target should marketing carry, funnel model, top-down vs bottom-up plan, plan of record, annual marketing planning, quarterly planning, pipeline coverage we have to create, sales cycle lag, our leads cannot close this quarter, carry-in pipeline, scenario planning, conservative moderate aggressive forecast, stretch target, reforecast, re-forecast triggers, assumption register, capacity ceiling, addressable audience ceiling, marginal cost per lead, blended CPL is wrong, we missed plan and nobody knows why.

日本語の概要は準備中です。原文の説明を表示しています。

Expert data engineer for ETL/ELT pipelines, streaming, data warehousing. Activate on: data pipeline, ETL, ELT, data warehouse, Spark, Kafka, Airflow, dbt, data modeling, star schema, streaming data, batch processing, data quality. NOT for: API design (use api-architect), ML training (use ML skills), dashboards (use design skills).

日本語の概要は準備中です。原文の説明を表示しています。

Use when building your own cognee processing — writing custom tasks, chaining them into a pipeline with run_custom_pipeline or the lightweight run_pipeline (from cognee.pipelines import run_pipeline), storing custom DataPoints with add_data_points, running custom extraction/enrichment over the existing graph with memify, checking pipeline run status, or debugging how data flows between tasks (batch_size, data_per_batch, ctx, Drop, enriches).

日本語の概要は準備中です。原文の説明を表示しています。

Use whenever datasets, cloud storage buckets, or data pipelines are mentioned — creating, saving, querying, listing, exploring, deleting, or processing data in S3, GCS, Azure Blob, or local storage. Also use when running any script that may create datasets as a side effect. Maintains a knowledge base at dc-knowledge/ (JSON + markdown). ALWAYS use this skill when the user creates a dataset, saves pipeline output, runs a data script, or references any storage bucket.

日本語の概要は準備中です。原文の説明を表示しています。

Ensures the tenant has a usable Power Platform Pipelines host environment before any pipeline operation runs. Detects host state via the same resolution order as the Power Apps UI (org-db setting → BAP env metadata → default-custom-host setting); if any existing host (Platform or Custom) is found, uses it. If no host is bound to the source env, provisions a new **Platform Host** (recommended, idempotent) or a **Custom Host** via the BAP env-create API with the `D365_ProjectHost` template, or guides the user through PPAC install / `New custom host` (manual fallbacks). Polls lifecycle operations, verifies the host responds to Pipelines API calls, writes a host-check artifact other ALM skills consume. Use when asked to: "set up pipelines host", "ensure pipelines host", "no pipelines host", "install pipelines", "create pipelines host", "provision platform host", "provision custom host". Also invoked transparently by /power-pages:setup-pipeline when its host discovery step finds nothing.

日本語の概要は準備中です。原文の説明を表示しています。

Access ENCODE uniform analysis pipelines, generate user-specific Nextflow/WDL pipelines, manage compute resources, and integrate with cloud platforms. Use when the user wants to understand ENCODE pipelines, run pipelines on their own data, generate custom Nextflow workflows from ENCODE pipeline code, check compute requirements (CPU/GPU/memory), run pipelines in background, or integrate with Google Cloud, AWS, or other cloud platforms. Also use when the user asks about ENCODE pipeline outputs, processing standards, software versions, or wants to replicate ENCODE processing. Covers local execution, HPC, and cloud deployment with resource-aware scheduling. Use this skill for ANY pipeline execution, workflow generation, or compute resource management task involving ENCODE data.

日本語の概要は準備中です。原文の説明を表示しています。

iOS Core Data architect for persistent storage, CloudKit sync, schema migrations, and SwiftData migration. Activate on: Core Data, NSManagedObject, CloudKit sync, Core Data migration, NSFetchedResultsController, NSPersistentContainer, SwiftData migration path. NOT for: SwiftUI state management (use swiftui-data-flow-expert), server databases (use data-pipeline-engineer), SQLite direct (use mobile-offline-sync-architect).

日本語の概要は準備中です。原文の説明を表示しています。

Expert in turning data visualizations into compelling video and animation content for social media and marketing. Activate on: data commercial, animated chart, data storytelling video, chart animation, Remotion data video, Motion Canvas, Flourish animation, D3 animation, data viz social media, dashboard recording, data narrative video. NOT for: static data visualization (use data-viz-2025), general video editing (use video-processing-editing), full video production pipeline (use ai-video-production-master).

日本語の概要は準備中です。原文の説明を表示しています。

Adaption AI SDK for synthetic data augmentation and dataset adaptation. Use when building data pipelines with the Adaption Python SDK, uploading datasets (local files, Hugging Face, Kaggle), running augmentation/adaptation jobs, configuring brand controls (hallucination mitigation, safety categories, length), recipe specifications (reasoning traces, deduplication, preference pairs, prompt rephrase), evaluating dataset quality, downloading results, or any workflow involving `pip install adaption`, `from adaption import Adaption`, Adaptive Data, or the adaptionlabs.ai API. Also trigger when the user mentions synthetic data generation for fine-tuning, dataset augmentation pipelines, DPO preference pair generation, or grounding-based hallucination reduction on training data.

日本語の概要は準備中です。原文の説明を表示しています。