Results for “data processing”

数据工程 — 数据平台从业者的认知操作系统, 覆盖把数据从源系统搬运成可靠 / 可查询 / 可信赖形态供分析 / ML / 数据产品消费的全生命周期 (生成 → 摄取 → 存储 → 转换 → 服务 + 安全/数据管理/DataOps/数据架构/编排/软件工程 六条暗流, Reis & Housley 框架): 摄取与集成 (批 + CDC 变更数据捕获 Debezium + EL 工具 Fivetran/Airbyte/Meltano/dlt + Kafka Connect + schema drift) / 存储与文件表格式 (对象存储数据湖 + 列存 Parquet/ORC/Arrow/Avro + 开放表格式 Apache Iceberg/Delta Lake/Apache Hudi + lakehouse + 分区/compaction) / 转换与建模 (ELT dbt/SQLMesh + Spark + 维度建模 Kimball + Inmon + Data Vault + 大宽表 OBT + 渐变维 SCD + 增量模型 + 语义/指标层) / 编排与工作流 (Apache Airflow/Dagster/Prefect/Mage/Kestra/Apache DolphinScheduler + DAG + 幂等 + 回填 backfill + 数据资产调度) / 批流与实时 (Apache Kafka/Apache Flink/Spark Structured Streaming/Kinesis/Pulsar/Redpanda + Lambda vs Kappa + watermark/窗口/exactly-once + 流式 SQL Materialize/RisingWave + 实时 OLAP ClickHouse/Apache Druid/Apache Pinot/StarRocks/Apache Doris) / 数仓与查询引擎 (Snowflake/BigQuery/Redshift/Databricks SQL/Trino/Presto/DuckDB/Polars + 存算分离 + MPP) / 数据质量测试与可观测性 (dbt tests/Great Expectations/Soda + 数据契约 + Monte Carlo data downtime + 新鲜度/量/schema 异常检测) / 数据治理编目与血缘 (DataHub/Amundsen/OpenMetadata/Unity Catalog + 列级血缘 + PII 分类 + 访问控制 + GDPR) / DataOps 与可靠性 (数据 CI/CD + 转换版本控制 + 环境隔离 + 幂等重处理 + 数据 SLA/SLO + 计算存储 FinOps) / 数据架构范式 (现代数据栈 + lakehouse + data mesh + data fabric + 去中心化 vs 中心化所有权) / 分析工程角色 (dbt 时代连接数据工程与分析的桥) — 不含 数据科学/ML 建模本身 (是下游消费者) / BI 仪表盘制作 (serving 下游) / 数据分析报表为终点 / 'data engineer = 跑 Hadoop 的' 过时窄化 / 通用后端应用开发 (平行学科) (Data Engineering — the cognitive operating system of practitioners who design, build, and operate the data platform: moving data from source systems into reliable, queryable, trustworthy form for analytics / ML / products, covering (a) the data engineering lifecycle (generation → ingestion → storage → transformation → serving, with the undercurrents security / data management / DataOps / data architecture / orchestration / software engineering — Reis & Housley framing), (b) ingestion & integration (batch + CDC change-data-capture with Debezium, EL tools Fivetran / Airbyte / Meltano / dlt, Kafka Connect, API + file + database sources, schema drift handling), (c) storage & file/table formats (object storage data lakes, columnar formats Parquet / ORC / Arrow / Avro, open table formats Apache Iceberg / Delta Lake / Apache Hudi, lakehouse architecture, partitioning / compaction / Z-ordering), (d) transformation & modeling (ELT with dbt / SQLMesh, Spark, dimensional modeling Kimball, Inmon CIF, Data Vault, One Big Table / wide tables, normalization vs denormalization, slowly changing dimensions, incremental models, the semantic / metrics layer), (e) orchestration & workflow (Apache Airflow, Dagster, Prefect, Mage, Kestra, Apache DolphinScheduler, DAGs, idempotency, backfills, data-aware / asset-based scheduling), (f) batch vs streaming & real-time (Apache Kafka, Apache Flink, Spark Structured Streaming, Kinesis / Pulsar / Redpanda, the Lambda vs Kappa debate, watermarks / windowing / exactly-once, streaming SQL Materialize / RisingWave, real-time OLAP ClickHouse / Apache Druid / Apache Pinot / StarRocks / Apache Doris), (g) warehouses & query engines (Snowflake, BigQuery, Redshift, Databricks SQL, Trino / Presto, DuckDB, Polars, decoupled storage & compute, MPP), (h) data quality, testing & observability (dbt tests, Great Expectations, Soda, data contracts, Monte Carlo / data downtime, freshness / volume / schema anomaly detection, unit / integration testing of pipelines), (i) data governance, catalog & lineage (DataHub, Amundsen, OpenMetadata, Unity Catalog, column-level lineage, PII / data classification, access control, GDPR / data privacy), (j) DataOps & reliability (CI/CD for data, version control of transformations, environments, idempotent reprocessing, SLAs / SLOs for data, cost / FinOps for compute & storage), (k) data architecture paradigms (modern data stack, data lakehouse, data mesh, data fabric, decentralized vs centralized ownership), (l) the analytics engineering role (the dbt-era bridge between data engineering and analysis); N

The English summary is being prepared. Showing the original description.

Build data processing pipelines for machine learning, from preprocessing to batch predictions. Uses Ray Data to handle datasets larger than memory, distribute work across CPUs and GPUs, and feed data into training frameworks.

• Preprocessing datasets larger than memory
• Running batch predictions across multiple GPUs
• Preparing image and tabular data for training

Build data processing pipelines for machine learning, from preprocessing to batch predictions. Uses Ray Data to handle datasets larger than memory, distribute work across CPUs and GPUs, and feed data into training frameworks.

• Preprocessing datasets larger than memory
• Running batch predictions across multiple GPUs
• Preparing image and tabular data for training

Design systems that collect, transform, and store data. Choose between scheduled and continuous processing, generate workflow configurations, validate data quality, and analyze performance using guidance for tools including Python, SQL, Spark, Airflow, dbt, and Kafka.

• Designing data ingestion and transformation pipelines
• Choosing scheduled or real-time data processing
• Checking data freshness, completeness, and uniqueness

Review personal data handling against GDPR requirements, covering processing records, legal grounds, individual rights, breach procedures, and international transfers. Produce findings by article and a prioritized improvement roadmap with responsibilities, estimated costs, and timelines.

• Preparing evidence for a regulatory audit
• Reviewing privacy requirements for a new system
• Assessing GDPR gaps during acquisition due diligence

Build custom cognee processing by chaining Python tasks, defining data types, and saving results to graph and vector databases. Use it to enrich existing data, inspect run status, and troubleshoot how tasks pass data.

• Building custom extraction and processing steps
• Saving custom data types to graph and vector databases
• Adding information to an existing graph

Design AI agent analytics workflows across distributed tables and documents. Combine Knowledge Catalog for metadata, meaning information about data and its usage rules, with managed Spark processing to produce architecture diagrams, deployment guidance, and validation plans.

• Design analytics across data stored in multiple clouds
• Define metadata and governance with Knowledge Catalog
• Plan Data Agent Kit use in VS Code or Antigravity

Clean, combine, summarize, and reshape tabular data with pandas DataFrames, Python's table-like data structures. Produces transformation code and validation checks, including time series processing and memory-conscious approaches for large datasets.

• Fill missing values and correct data types
• Combine tables using multiple matching columns
• Create grouped summaries and pivot tables

Data pipeline specialist for ETL design, data quality, CDC patterns, and batch/stream processingUse when "data pipeline, etl, cdc, data quality, batch processing, stream processing, data transformation, data warehouse, data lake, data validation, data-engineering, etl, cdc, batch, streaming, data-quality, dbt, airflow, dagster, data-pipeline, ml-memory" mentioned.

The English summary is being prepared. Showing the original description.

Retrieve focused genome, assembly, and taxonomy metadata from NCBI Datasets v2, a biological data service. Get concise Markdown summaries by default, with raw JSON or text available when requested.

• Checking genome metadata for a specific organism
• Looking up an assembly by its accession identifier
• Checking taxonomy information by taxon ID

Implement and audit privacy controls in product and infrastructure — GDPR, CCPA / CPRA, LGPD, PIPEDA. Covers data minimization, lawful basis, consent management, data subject access requests (DSARs — access, deletion, portability), data processing agreements, DPIA / TIA, breach notification timing, data classification, and the technical implementation of 'right to be forgotten' across backups, caches, analytics, and third parties. Use when the user mentions 'GDPR,' 'CCPA,' 'CPRA,' 'data privacy,' 'privacy engineering,' 'data subject access request,' 'DSAR,' 'right to deletion,' 'right to be forgotten,' 'data portability,' 'consent management,' 'cookie consent,' 'data minimization,' 'DPIA,' 'data protection impact assessment,' 'breach notification,' 'BAA,' 'DPA,' 'data processing agreement,' 'sub-processor,' 'cross-border data transfer,' 'SCCs,' or needs to implement or review privacy controls.

The English summary is being prepared. Showing the original description.

Builds comprehensive data inventory per GDPR Art. 30 Records of Processing Activities. Covers system-by-system discovery, data flow diagramming, third-party identification, and legal basis per category. Keywords: data inventory, data mapping, Art 30, RoPA, data flow, processing activities.

The English summary is being prepared. Showing the original description.

Helps developers send conversion and event data to Google Ads, Google Analytics, and Floodlight through the Data Manager API. It covers destination setup, data formatting, validation, migration from other APIs, and checking processing results.

• Implement offline conversion uploads to Google Ads
• Send recommended or custom GA4 events
• Migrate event ingestion from another Google API

Generate Python code to clean, transform, analyze, visualize, and model BigQuery data with BigQuery DataFrames. The skill keeps processing in the cloud and uses familiar table operations, with guidance for previews, schema checks, and machine learning.

• Clean and transform BigQuery data with Python
• Preview data and check column types
• Visualize aggregated or sampled data

Plan data ingestion and transformation workflows with Apache Airflow, Spark, and dbt. Get guidance on processing changed records, validating data quality, and rerunning failed or historical jobs without creating duplicate results.

• Designing data ingestion and transformation workflows
• Processing only new or changed records
• Adding data quality checks between processing stages

Configure and troubleshoot Apache Beam runners, the engines that execute data processing pipelines. Covers local testing, Dataflow, Flink, and Spark, with Java and Python examples, runner validation tests, and guidance on logs and execution settings.

• Test a Beam pipeline locally
• Configure pipeline execution on Dataflow
• Review Flink or Spark execution settings

Conducts Privacy Impact Assessment for health data processing under GDPR Article 9, HIPAA, and sector-specific health privacy regulations. Covers special category data safeguards, clinical research data, patient portals, health wearables, genetic data, and cross-border health data transfers. Keywords: health data PIA, DPIA, Article 9, HIPAA, special category data, clinical research, patient privacy, genetic data.

The English summary is being prepared. Showing the original description.

Plans and scripts data migrations between SQL databases: schema mapping, type conversion, batched and resumable loading, validation and cutover. Use when the user mentions "data migration," "database migration," "migrate from MySQL to PostgreSQL," "schema migration," "ETL pipeline," "data transfer," "database consolidation," "legacy migration," or "move data between databases." Covers schema analysis, mapping, transformation, batch processing, validation, and cutover planning. For query optimization during migration, see sql-optimizer.

The English summary is being prepared. Showing the original description.

Design and implement systems that collect, transform, validate, and store data for analysis. Covers scheduled and real-time processing, data warehouses, quality checks, monitoring, and cost tuning using tools such as Apache Spark, dbt, and Airflow.

• Building scheduled data collection and transformation workflows
• Designing a data warehouse for analysis
• Synchronizing database changes in real time

Design n8n automations for incoming events, API connections, database synchronization, AI agents, scheduled tasks, and batch processing. Choose a suitable structure, plan error handling, check connections, test results, and review performance when processing large numbers of items.

• Sending form submissions to Slack
• Generating and emailing scheduled reports
• Synchronizing records between databases

Write JavaScript for n8n Code nodes to aggregate, filter, and transform workflow data. Guidance covers data access, output formats, execution modes, common errors, and performance choices when processing large numbers of records.

• Aggregating and deduplicating multiple records
• Transforming complex API responses
• Fixing data access and return format errors

Build Apache Spark pipelines for large datasets, including transformations, aggregations, and streaming analysis. Get PySpark or Scala code, partitioning strategies for dividing data across workers, and configuration advice to investigate slow jobs and resource usage.

• Build a pipeline to transform and aggregate large datasets
• Investigate slow Spark jobs and uneven data distribution
• Tune memory settings and data partitions

Design or extend DataFusion benchmarks using SQL where practical. Select representative workload variations, keep unrelated processing lightweight, and integrate runs with bench.sh so measurements reflect useful execution paths and remain practical to run.

• Add SQL benchmarks for DataFusion operators
• Select representative input sizes and filter conditions
• Reduce unrelated processing overhead in measurements

Expert GDPR compliance assistant covering all four core workflows: (1) auditing code and systems for GDPR violations, (2) drafting GDPR-compliant documents such as privacy policies, Data Processing Agreements (DPAs), and consent notices, (3) answering GDPR compliance questions with authoritative article citations, and (4) reviewing data flows and PII handling practices. Use this skill whenever the user mentions GDPR, data protection, privacy compliance, lawful basis, data subject rights, DPA, privacy notices, consent management, data breaches, DPIAs, controller/ processor relationships, cross-border data transfers, or any EU/UK data privacy topic. Also trigger for questions like "is this GDPR compliant?", "how do I handle personal data?", "what does a privacy policy need?", or any request involving PII, personal data, or data retention in a regulatory context.

The English summary is being prepared. Showing the original description.