「data pipeline」の検索結果

Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.

日本語の概要は準備中です。原文の説明を表示しています。

Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.

日本語の概要は準備中です。原文の説明を表示しています。

Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.

日本語の概要は準備中です。原文の説明を表示しています。

Comprehensive data visualization toolkit for creating beautiful, mathematically elegant visualizations with D3.js, Chart.js, and custom SVG. Use when (1) building interactive data visualizations, (2) designing color palettes for charts, (3) choosing scales and visual encodings, (4) creating data pipelines from Census/SEC/Wikipedia APIs, (5) crafting narrative-driven data stories, (6) making perceptually accurate charts, or (7) implementing force-directed networks, timelines, or geographic maps.

日本語の概要は準備中です。原文の説明を表示しています。

Fetch weather/climate data via Earth2Studio data sources for specific variables and times. Do NOT use for inference pipelines, model discovery, or installation.

日本語の概要は準備中です。原文の説明を表示しています。

Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.

日本語の概要は準備中です。原文の説明を表示しています。

Migrates Synapse Data Factory pipelines to Fabric Data Factory, turning linked services into Fabric connections, inlining datasets into activities, converting global parameters to Variable Libraries, and mapping SynapseNotebook to TridentNotebook. Use for pipeline JSON. For Synapse Spark notebook code use synapse-migration.

日本語の概要は準備中です。原文の説明を表示しています。

Audit data quality across pipelines, warehouses, and stores. Use when designing a DQ program, defining DQ dimensions, building rule-based checks, detecting schema drift, monitoring freshness SLAs, or responding to a DQ incident.

日本語の概要は準備中です。原文の説明を表示しています。

Develop and deploy Lakeflow Jobs on Databricks via DABs, Python SDK, or the CLI. Use when creating data engineering jobs with notebooks, Python wheels, SQL, dbt, or pipelines. Invoke BEFORE starting implementation.

日本語の概要は準備中です。原文の説明を表示しています。

Create, configure, validate, deploy, run, and manage Declarative Automation Bundles (DABs, formerly Databricks Asset Bundles). Use when working with Databricks resources via DABs including dashboards, jobs, pipelines, alerts, volumes, and apps.

日本語の概要は準備中です。原文の説明を表示しています。

Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).

日本語の概要は準備中です。原文の説明を表示しています。

Applies pipes-and-filters for sequential data transformations. Use when data flows through discrete stages like ETL, streaming analytics, or CI/CD pipelines.

日本語の概要は準備中です。原文の説明を表示しています。

Think and work like an expert Clinical Data Manager. Use when a task calls for Clinical Data Manager judgment. Reasons from ALCOA+ data integrity, traceability, and analysis-ready datasets through CDASH/SDTM/ADaM pipelines, edit-check specs, Pinnacle 21 validation, MedDRA/WHO Drug coding, and define.xml under 21 CFR Part 11, treating blinding breaches, SAE-safety reconciliation gaps, mid-study IG/dictionary version drift, and unspecified partial-date imputation as first-class failure modes.

日本語の概要は準備中です。原文の説明を表示しています。

ClearML is an open-source MLOps platform that records machine-learning experiments, versions datasets, chains tasks into pipelines and runs them on remote machines through agents and queues. Use when a user asks to "track experiments with ClearML", "log metrics, artifacts and models", "version a dataset", "run training on a remote GPU with clearml-agent", "build a ClearML pipeline", "run hyperparameter optimization", or "self-host ClearML Server". Covers the clearml 2.x Python SDK, clearml-agent 3.x and ClearML Server 2.x.

日本語の概要は準備中です。原文の説明を表示しています。

Use when reading from or writing to Neo4j with Apache Spark or Databricks using the Neo4j Connector for Apache Spark 6.0 (org.neo4j.connectors:spark) or 5.x (org.neo4j:neo4j-connector-apache-spark). Covers SparkSession setup, DataFrame reads via labels/Cypher/relationship scan, DataFrame writes with SaveMode, node.keys for MERGE, relationship write mapping, partition and batch tuning, PySpark and Scala examples, Databricks cluster config, Databricks secrets for credentials, Delta Lake to Neo4j pipelines. Does NOT handle Cypher authoring — use neo4j-cypher-skill. Does NOT handle the Python bolt driver — use neo4j-driver-python-skill. Does NOT handle GDS algorithms — use neo4j-gds-skill.

日本語の概要は準備中です。原文の説明を表示しています。

Design a data cleaning and transformation pipeline — missing values, outliers, and deduplication. Use when asked to "clean this dataset", "handle missing values", or "deduplicate this data".

日本語の概要は準備中です。原文の説明を表示しています。

Design a data validation pipeline — schema checks, range validation, and quality metrics. Use when asked to "validate incoming data", "add schema checks", or "define data quality metrics".

日本語の概要は準備中です。原文の説明を表示しています。

Expert guide for Data Pipelines, ETL/ELT, and Analytics Engineering. Covers dbt, Apache Airflow, Dagster, BigQuery, ClickHouse, and DuckDB / Panduan ahli untuk Data Pipelines, ETL/ELT. Mencakup dbt, Airflow, Dagster, BigQuery, ClickHouse, dan DuckDB.

日本語の概要は準備中です。原文の説明を表示しています。

Review a data analysis for quality, correctness, and reproducibility. Covers data quality assessment, assumption checking, model validation, data leakage detection, and reproducibility verification. Use when reviewing a colleague's analysis before publication, validating an ML pipeline before production deployment, auditing a report for regulatory or business decision-making, or performing a second-analyst review in a regulated environment.

日本語の概要は準備中です。原文の説明を表示しています。

Set up systematic data labeling workflows using Label Studio or similar tools. Implement quality controls, measure inter-annotator agreement, manage labeler teams, and integrate labeled data into ML training pipelines. Use when starting a supervised ML project that requires labeled training data, when model performance is limited by insufficient labeled examples, when labeling text, images, audio, or video, or when implementing active learning to prioritize the most valuable examples.

日本語の概要は準備中です。原文の説明を表示しています。

对 DiT（扩散/多模态生成）模型执行 W8A8 动态（data-free）或 W8A8 MXFP8（data-free）量化。覆盖 HunyuanVideo、Wan2.2-T2V/I2V/TI2V、FLUX.1-dev（已迁移）、SD3、Sana、HunyuanDiT、CogViewX 等。YAML apiversion 固定 multimodal_sd_modelslim_v1，统一 MultimodalPipelineInterface 重构路径（不再保留 LegacyMultimodalPipelineInterface）。模板与字段以 msmodelslim/model/hunyuan_video/model_adapter.py（唯一已迁移到重构路径且 lab_practice YAML 已 validated 的 DiT）为参考；其它 DiT 需按各自推理仓的 `parse_args` 调整字段集合。data-free 指不引入外部激活校准数据，spec.dataset（校准集短名，短名如 wan2_2_t2v；enable_dump: false 时不参与 dump）仍必填；精度调优通过下游经验库 L2 §7 + 直接调 `quantization-expert-experience-tuning-rules/scripts/apply_rollback.py`（整层回退）扩展 exclude 列表，闭环由 `quantization-accuracy-tuning-orchestrator` 调度。

日本語の概要は準備中です。原文の説明を表示しています。

Build data pipelines for AI model training. TRIGGERS - Use when user needs help with ai-data-pipeline related tasks.

日本語の概要は準備中です。原文の説明を表示しています。

Use when a task needs the judgment of a Data Engineer — designing an idempotent pipeline operation so retries and backfills don't corrupt data, choosing event-time over processing-time partitioning for time-series aggregation, enforcing schema compatibility rules before an upstream change propagates downstream, adding automated data quality checks to a pipeline, or deciding between batch and streaming architecture for a specific freshness requirement.

日本語の概要は準備中です。原文の説明を表示しています。

Execute ENCODE Whole Genome Bisulfite Sequencing (WGBS) pipeline from FASTQ to methylation calls. Child of pipeline-guide. Provides Nextflow execution with Docker and cloud deployment. Use when processing WGBS/bisulfite-seq data, calling methylation levels, generating bedMethyl files. Trigger on: WGBS pipeline, bisulfite sequencing, methylation calling, DNA methylation pipeline, bismark, bwa-meth, bedMethyl.

日本語の概要は準備中です。原文の説明を表示しています。