'data quality' 검색 결과

数据工程 — 数据平台从业者的认知操作系统, 覆盖把数据从源系统搬运成可靠 / 可查询 / 可信赖形态供分析 / ML / 数据产品消费的全生命周期 (生成 → 摄取 → 存储 → 转换 → 服务 + 安全/数据管理/DataOps/数据架构/编排/软件工程 六条暗流, Reis & Housley 框架): 摄取与集成 (批 + CDC 变更数据捕获 Debezium + EL 工具 Fivetran/Airbyte/Meltano/dlt + Kafka Connect + schema drift) / 存储与文件表格式 (对象存储数据湖 + 列存 Parquet/ORC/Arrow/Avro + 开放表格式 Apache Iceberg/Delta Lake/Apache Hudi + lakehouse + 分区/compaction) / 转换与建模 (ELT dbt/SQLMesh + Spark + 维度建模 Kimball + Inmon + Data Vault + 大宽表 OBT + 渐变维 SCD + 增量模型 + 语义/指标层) / 编排与工作流 (Apache Airflow/Dagster/Prefect/Mage/Kestra/Apache DolphinScheduler + DAG + 幂等 + 回填 backfill + 数据资产调度) / 批流与实时 (Apache Kafka/Apache Flink/Spark Structured Streaming/Kinesis/Pulsar/Redpanda + Lambda vs Kappa + watermark/窗口/exactly-once + 流式 SQL Materialize/RisingWave + 实时 OLAP ClickHouse/Apache Druid/Apache Pinot/StarRocks/Apache Doris) / 数仓与查询引擎 (Snowflake/BigQuery/Redshift/Databricks SQL/Trino/Presto/DuckDB/Polars + 存算分离 + MPP) / 数据质量测试与可观测性 (dbt tests/Great Expectations/Soda + 数据契约 + Monte Carlo data downtime + 新鲜度/量/schema 异常检测) / 数据治理编目与血缘 (DataHub/Amundsen/OpenMetadata/Unity Catalog + 列级血缘 + PII 分类 + 访问控制 + GDPR) / DataOps 与可靠性 (数据 CI/CD + 转换版本控制 + 环境隔离 + 幂等重处理 + 数据 SLA/SLO + 计算存储 FinOps) / 数据架构范式 (现代数据栈 + lakehouse + data mesh + data fabric + 去中心化 vs 中心化所有权) / 分析工程角色 (dbt 时代连接数据工程与分析的桥) — 不含 数据科学/ML 建模本身 (是下游消费者) / BI 仪表盘制作 (serving 下游) / 数据分析报表为终点 / 'data engineer = 跑 Hadoop 的' 过时窄化 / 通用后端应用开发 (平行学科) (Data Engineering — the cognitive operating system of practitioners who design, build, and operate the data platform: moving data from source systems into reliable, queryable, trustworthy form for analytics / ML / products, covering (a) the data engineering lifecycle (generation → ingestion → storage → transformation → serving, with the undercurrents security / data management / DataOps / data architecture / orchestration / software engineering — Reis & Housley framing), (b) ingestion & integration (batch + CDC change-data-capture with Debezium, EL tools Fivetran / Airbyte / Meltano / dlt, Kafka Connect, API + file + database sources, schema drift handling), (c) storage & file/table formats (object storage data lakes, columnar formats Parquet / ORC / Arrow / Avro, open table formats Apache Iceberg / Delta Lake / Apache Hudi, lakehouse architecture, partitioning / compaction / Z-ordering), (d) transformation & modeling (ELT with dbt / SQLMesh, Spark, dimensional modeling Kimball, Inmon CIF, Data Vault, One Big Table / wide tables, normalization vs denormalization, slowly changing dimensions, incremental models, the semantic / metrics layer), (e) orchestration & workflow (Apache Airflow, Dagster, Prefect, Mage, Kestra, Apache DolphinScheduler, DAGs, idempotency, backfills, data-aware / asset-based scheduling), (f) batch vs streaming & real-time (Apache Kafka, Apache Flink, Spark Structured Streaming, Kinesis / Pulsar / Redpanda, the Lambda vs Kappa debate, watermarks / windowing / exactly-once, streaming SQL Materialize / RisingWave, real-time OLAP ClickHouse / Apache Druid / Apache Pinot / StarRocks / Apache Doris), (g) warehouses & query engines (Snowflake, BigQuery, Redshift, Databricks SQL, Trino / Presto, DuckDB, Polars, decoupled storage & compute, MPP), (h) data quality, testing & observability (dbt tests, Great Expectations, Soda, data contracts, Monte Carlo / data downtime, freshness / volume / schema anomaly detection, unit / integration testing of pipelines), (i) data governance, catalog & lineage (DataHub, Amundsen, OpenMetadata, Unity Catalog, column-level lineage, PII / data classification, access control, GDPR / data privacy), (j) DataOps & reliability (CI/CD for data, version control of transformations, environments, idempotent reprocessing, SLAs / SLOs for data, cost / FinOps for compute & storage), (k) data architecture paradigms (modern data stack, data lakehouse, data mesh, data fabric, decentralized vs centralized ownership), (l) the analytics engineering role (the dbt-era bridge between data engineering and analysis); N

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Design systems that collect, transform, and store data. Choose between scheduled and continuous processing, generate workflow configurations, validate data quality, and analyze performance using guidance for tools including Python, SQL, Spark, Airflow, dbt, and Kafka.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Designing data ingestion and transformation pipelines
• Choosing scheduled or real-time data processing
• Checking data freshness, completeness, and uniqueness

Check single-cell RNA-seq data quality, filter low-quality cells using median absolute deviation, a measure of spread, and compare plots before and after filtering. Outputs include filtered data and the original dataset with quality annotations.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Assess single-cell RNA-seq data quality
• Filter low-quality cells and rarely detected genes
• Compare quality plots before and after filtering

Trace where BigQuery and Cloud Storage data comes from and where it goes. The skill summarizes table or column lineage, meaning data origins and movement, in a readable Markdown report to support data quality investigations.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Investigating data origins during a quality issue
• Summarizing data sources and destinations
• Tracing the flow of a specific column

Build data quality checks with Great Expectations, dbt tests, and data contracts, which define shared expectations between teams. Check missing values, duplicates, and freshness, then integrate validation into data pipelines and CI/CD.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Check data for missing values and duplicates
• Build a dbt test suite
• Define shared data contracts between teams

Build automated data checks with Great Expectations and dbt to detect missing values, duplicates, invalid ranges, and stale records. Define data contracts, shared agreements on data structure and quality, and summarize validation results across tables.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Add missing-value and duplicate checks to order data
• Build dbt tests for relationships between tables
• Define shared data requirements between teams

World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, real-time streaming, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, Flink, Kinesis, and modern data stack. Includes data modeling, pipeline orchestration, data quality, streaming quality monitoring, and DataOps. Use when designing data architectures, building batch or streaming data pipelines, optimizing data workflows, or implementing data governance.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Data pipeline specialist for ETL design, data quality, CDC patterns, and batch/stream processingUse when "data pipeline, etl, cdc, data quality, batch processing, stream processing, data transformation, data warehouse, data lake, data validation, data-engineering, etl, cdc, batch, streaming, data-quality, dbt, airflow, dagster, data-pipeline, ml-memory" mentioned.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Validate data quality in CSV, JSON, and database exports by checking for missing values, type mismatches, duplicates, outliers, and schema violations. Use when building ETL pipelines, auditing data imports, checking data freshness, or ensuring data contracts between teams. Trigger words: data quality, validation, null values, duplicates, schema check, data contract, ETL, pipeline, data drift.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

OpenLineage, DataHub, Marquez for data lineage tracking and impact analysis. Activate on: data lineage, OpenLineage, DataHub, Marquez, impact analysis, data catalog, column lineage, data discovery. NOT for: data quality validation (use data-quality-guardian), dbt documentation (use dbt-analytics-engineer).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

OpenLineage, DataHub, Marquez for data lineage tracking and impact analysis. Activate on: data lineage, OpenLineage, DataHub, Marquez, impact analysis, data catalog, column lineage, data discovery. NOT for: data quality validation (use data-quality-guardian), dbt documentation (use dbt-analytics-engineer).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Prepare training datasets for large language models with GPU-assisted cleaning. Covers duplicate removal, quality filtering, and personal information redaction, with workflows for text, images, video, and audio using NeMo Curator.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Clean web data for model training
• Remove exact and near duplicates from large text collections
• Redact personal information from training text

Prepare training datasets for large language models with GPU-assisted cleaning. Covers duplicate removal, quality filtering, and personal information redaction, with workflows for text, images, video, and audio using NeMo Curator.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Clean web data for model training
• Remove exact and near duplicates from large text collections
• Redact personal information from training text

Explore scientific data across more than 200 file formats to understand its structure, contents, and quality. Produce Markdown reports with statistical summaries, format-specific findings, and recommendations for preprocessing, visualization, and further analysis.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Inspecting unfamiliar scientific data files
• Checking missing values and data quality
• Reviewing sequence or microscopy data before analysis

Prepare high-quality datasets for LLM fine-tuning with filtering, deduplication, augmentation, and RLHF data formatting. Activate on: fine-tuning data, training data curation, RLHF dataset, data quality filtering, SFT dataset. NOT for: model training infrastructure (ai-engineer), prompt engineering without fine-tuning (prompt-engineer).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Build training datasets for LLM specialization from production data, frontier model distillation, and synthetic bootstrapping. Use when formatting production logs into SFT data, distilling from frontier APIs, or preparing data for fine-tuning. Covers JSONL formatting, data quality validation, deduplication, and train/eval splitting.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Prepare high-quality datasets for LLM fine-tuning with filtering, deduplication, augmentation, and RLHF data formatting. Activate on: fine-tuning data, training data curation, RLHF dataset, data quality filtering, SFT dataset. NOT for: model training infrastructure (ai-engineer), prompt engineering without fine-tuning (prompt-engineer).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Design and implement systems that collect, transform, validate, and store data for analysis. Covers scheduled and real-time processing, data warehouses, quality checks, monitoring, and cost tuning using tools such as Apache Spark, dbt, and Airflow.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Building scheduled data collection and transformation workflows
• Designing a data warehouse for analysis
• Synchronizing database changes in real time

Inspect scientific data files to understand their format, structure, and quality. Covering over 200 formats, this skill produces Markdown reports with statistical summaries, potential issues, and recommendations for preprocessing, visualization, and further analysis.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Inspecting an unfamiliar scientific data file
• Checking missing values and duplicates in experimental tables
• Reviewing sequence lengths and quality scores in FASTQ files

Check CSV, Excel, TSV, and JSON tables for missing values, duplicates, unusual values, and inconsistent formats. Receive scores across 12 quality dimensions, an overall grade, and practical suggestions for fixing the detected issues.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Checking missing values and duplicate rows before analysis
• Reviewing date formats and mixed data types
• Verifying unique IDs and relationships between columns

Run exploratory data analysis on a dataset and produce a structured report. Use when the user says "explore this dataset", "EDA on X", "analyze this data", "what's in this dataset", "summarize this data", "first look at the data", "understand this dataset before modeling", "data quality check", "describe this dataframe", or wants to understand a new dataset before building models or dashboards.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Router for the 101 SfSkills `data` skill packages. Data model, data movement and data quality: migrations, bulk loads, query optimisation, deduplicating at volume, archival. Ordinary-volume duplicate cleanup and prevention use salesforce-admin; come here for hundreds-of-thousands+ dedup or third-party tools. LDV architecture uses salesforce-architect. Use when the request mentions data model, data migration, data load, Data Loader, Bulk API, external id, deduplication at volume, archival, SOSL, cross-object search, sandbox seed data, SandboxPostCopy, native Data Seeding. Finds and opens the exact skill package to read; it does not contain the guidance itself.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Great Expectations, dbt tests, anomaly detection, and data contracts for data quality. Activate on: data quality, data validation, Great Expectations, data contract, anomaly detection, SLA, freshness check, schema validation. NOT for: dbt model structure (use dbt-analytics-engineer), schema evolution (use schema-evolution-manager).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Great Expectations, dbt tests, anomaly detection, and data contracts for data quality. Activate on: data quality, data validation, Great Expectations, data contract, anomaly detection, SLA, freshness check, schema validation. NOT for: dbt model structure (use dbt-analytics-engineer), schema evolution (use schema-evolution-manager).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.