“data”的搜索结果

数据工程 — 数据平台从业者的认知操作系统, 覆盖把数据从源系统搬运成可靠 / 可查询 / 可信赖形态供分析 / ML / 数据产品消费的全生命周期 (生成 → 摄取 → 存储 → 转换 → 服务 + 安全/数据管理/DataOps/数据架构/编排/软件工程 六条暗流, Reis & Housley 框架): 摄取与集成 (批 + CDC 变更数据捕获 Debezium + EL 工具 Fivetran/Airbyte/Meltano/dlt + Kafka Connect + schema drift) / 存储与文件表格式 (对象存储数据湖 + 列存 Parquet/ORC/Arrow/Avro + 开放表格式 Apache Iceberg/Delta Lake/Apache Hudi + lakehouse + 分区/compaction) / 转换与建模 (ELT dbt/SQLMesh + Spark + 维度建模 Kimball + Inmon + Data Vault + 大宽表 OBT + 渐变维 SCD + 增量模型 + 语义/指标层) / 编排与工作流 (Apache Airflow/Dagster/Prefect/Mage/Kestra/Apache DolphinScheduler + DAG + 幂等 + 回填 backfill + 数据资产调度) / 批流与实时 (Apache Kafka/Apache Flink/Spark Structured Streaming/Kinesis/Pulsar/Redpanda + Lambda vs Kappa + watermark/窗口/exactly-once + 流式 SQL Materialize/RisingWave + 实时 OLAP ClickHouse/Apache Druid/Apache Pinot/StarRocks/Apache Doris) / 数仓与查询引擎 (Snowflake/BigQuery/Redshift/Databricks SQL/Trino/Presto/DuckDB/Polars + 存算分离 + MPP) / 数据质量测试与可观测性 (dbt tests/Great Expectations/Soda + 数据契约 + Monte Carlo data downtime + 新鲜度/量/schema 异常检测) / 数据治理编目与血缘 (DataHub/Amundsen/OpenMetadata/Unity Catalog + 列级血缘 + PII 分类 + 访问控制 + GDPR) / DataOps 与可靠性 (数据 CI/CD + 转换版本控制 + 环境隔离 + 幂等重处理 + 数据 SLA/SLO + 计算存储 FinOps) / 数据架构范式 (现代数据栈 + lakehouse + data mesh + data fabric + 去中心化 vs 中心化所有权) / 分析工程角色 (dbt 时代连接数据工程与分析的桥) — 不含 数据科学/ML 建模本身 (是下游消费者) / BI 仪表盘制作 (serving 下游) / 数据分析报表为终点 / 'data engineer = 跑 Hadoop 的' 过时窄化 / 通用后端应用开发 (平行学科) (Data Engineering — the cognitive operating system of practitioners who design, build, and operate the data platform: moving data from source systems into reliable, queryable, trustworthy form for analytics / ML / products, covering (a) the data engineering lifecycle (generation → ingestion → storage → transformation → serving, with the undercurrents security / data management / DataOps / data architecture / orchestration / software engineering — Reis & Housley framing), (b) ingestion & integration (batch + CDC change-data-capture with Debezium, EL tools Fivetran / Airbyte / Meltano / dlt, Kafka Connect, API + file + database sources, schema drift handling), (c) storage & file/table formats (object storage data lakes, columnar formats Parquet / ORC / Arrow / Avro, open table formats Apache Iceberg / Delta Lake / Apache Hudi, lakehouse architecture, partitioning / compaction / Z-ordering), (d) transformation & modeling (ELT with dbt / SQLMesh, Spark, dimensional modeling Kimball, Inmon CIF, Data Vault, One Big Table / wide tables, normalization vs denormalization, slowly changing dimensions, incremental models, the semantic / metrics layer), (e) orchestration & workflow (Apache Airflow, Dagster, Prefect, Mage, Kestra, Apache DolphinScheduler, DAGs, idempotency, backfills, data-aware / asset-based scheduling), (f) batch vs streaming & real-time (Apache Kafka, Apache Flink, Spark Structured Streaming, Kinesis / Pulsar / Redpanda, the Lambda vs Kappa debate, watermarks / windowing / exactly-once, streaming SQL Materialize / RisingWave, real-time OLAP ClickHouse / Apache Druid / Apache Pinot / StarRocks / Apache Doris), (g) warehouses & query engines (Snowflake, BigQuery, Redshift, Databricks SQL, Trino / Presto, DuckDB, Polars, decoupled storage & compute, MPP), (h) data quality, testing & observability (dbt tests, Great Expectations, Soda, data contracts, Monte Carlo / data downtime, freshness / volume / schema anomaly detection, unit / integration testing of pipelines), (i) data governance, catalog & lineage (DataHub, Amundsen, OpenMetadata, Unity Catalog, column-level lineage, PII / data classification, access control, GDPR / data privacy), (j) DataOps & reliability (CI/CD for data, version control of transformations, environments, idempotent reprocessing, SLAs / SLOs for data, cost / FinOps for compute & storage), (k) data architecture paradigms (modern data stack, data lakehouse, data mesh, data fabric, decentralized vs centralized ownership), (l) the analytics engineering role (the dbt-era bridge between data engineering and analysis); N

中文简介正在准备中。当前显示原文说明。

Helps startup leaders make strategic data decisions: assess permission to train AI on customer data, choose data architecture, evaluate customer data as a business asset, and sequence hiring around the decisions the business needs to make.

中文简介正在准备中。当前显示英语简介。

• Assessing customer data for AI training
• Choosing data architecture and what to build or buy
• Evaluating data assets before fundraising or M&A

Manage scientific datasets with DataLad and git-annex, which tracks large files separately from Git history. Retrieve selected data, record analysis commands and their inputs and outputs, rerun recorded work, and publish history alongside file content.

中文简介正在准备中。当前显示英语简介。

• Retrieve only the research data you need
• Resolve files that contain pointers instead of data
• Record and rerun analysis steps

Generate Python code for Dataverse SDK workflows, including retries, bulk data changes, efficient queries, table metadata, and large file uploads. The skill also covers pandas integration and asks for type hints, docstrings, and official API references.

中文简介正在准备中。当前显示英语简介。

• Implement retries for temporary Dataverse errors
• Create bulk data operations and efficient queries
• Manage custom tables and metadata caches

Design systems that collect, transform, and store data. Choose between scheduled and continuous processing, generate workflow configurations, validate data quality, and analyze performance using guidance for tools including Python, SQL, Spark, Airflow, dbt, and Kafka.

中文简介正在准备中。当前显示英语简介。

• Designing data ingestion and transformation pipelines
• Choosing scheduled or real-time data processing
• Checking data freshness, completeness, and uniqueness

Helps organize biological datasets and models in LaminDB. Covers data registration, table validation, annotation with standardized biological terms, metadata searches, and tracking analysis inputs and outputs so researchers can preserve the context needed to repeat their work.

中文简介正在准备中。当前显示英语简介。

• Register and organize biological datasets
• Validate DataFrame or AnnData metadata
• Standardize annotations with biological vocabularies

Assess potential data breach impact by reviewing sensitive data and its movement through a codebase. Produce an inventory, exposure rankings, financial estimates, and prioritized safeguards, distinguishing legal source figures from planning estimates.

中文简介正在准备中。当前显示英语简介。

• Inventory sensitive data before a security review
• Trace where customer data could leak
• Estimate breach impact and costs for planning

Create and manage Arize datasets for evaluation and experiments using the ax CLI. Import records from files, append examples, inspect dataset versions, and export a chosen version for offline analysis, with checks for field consistency and export completeness.

中文简介正在准备中。当前显示英语简介。

• Creating evaluation datasets from files
• Adding examples to an existing test set
• Inspecting dataset metadata and versions

Use when enabling Data Cloud (Data 360) in an org, creating data spaces, assigning permission sets, registering ingestion API sources, or configuring activation targets. Triggers: 'enable data cloud', 'data spaces setup', 'data cloud permission sets', 'ingestion API connected app', 'activation target setup', 'data cloud licensing', 'dedicated home org vs existing org'. NOT for data streams or DLO-to-DMO mapping — use data/data-cloud-data-streams. NOT for identity resolution rulesets — use admin/data-cloud-identity-resolution.

中文简介正在准备中。当前显示原文说明。

Use when creating or managing CRM Analytics datasets: configuring dataflows, scheduling refreshes, selecting fields, handling date types, managing row-count growth, or troubleshooting stale or broken datasets. Triggers: 'dataset creation', 'dataflow schedule', 'dataflow quota', 'date field not filterable', 'dataset row limit', 'CRM Analytics data refresh', 'field type mismatch in dataset'. NOT for making an existing dataset smaller or faster — use data/analytics-dataset-optimization. NOT for choosing between Reports, CRM Analytics and Tableau in the first place — use admin/einstein-analytics-basics.

中文简介正在准备中。当前显示原文说明。

Use this skill when configuring Data Manager in CRM Analytics: enabling objects for sync, scheduling data sync runs, setting up remote connections to external databases, monitoring sync status and error logs, or troubleshooting connected object issues. Trigger keywords: data sync, connected objects, Data Manager, sync schedule, remote connection, Snowflake connector, sync error, incremental sync, CRM Analytics data pipeline. NOT for pushing external data in programmatically via the External Data API — use data/analytics-external-data. NOT for authoring dataflow nodes — use admin/analytics-dataflow-development.

中文简介正在准备中。当前显示原文说明。

Run supported Datagma operations through Rube MCP, a connection between agents and external services. The skill covers tool discovery, connection checks, required input formats, and workflows for executing multiple operations.

中文简介正在准备中。当前显示英语简介。

• Find supported tools for a Datagma task
• Verify the Datagma connection is active
• Run multiple Datagma operations together

Draft or review manuscript statements explaining where data and code can be accessed. Organize repository choices, access restrictions, identifiers, and dataset citations, and check metadata that helps others find and reuse the research data.

中文简介正在准备中。当前显示英语简介。

• Draft data and code availability statements
• Choose repositories and access routes for research data
• Review dataset citations and metadata

Build data processing pipelines for machine learning, from preprocessing to batch predictions. Uses Ray Data to handle datasets larger than memory, distribute work across CPUs and GPUs, and feed data into training frameworks.

中文简介正在准备中。当前显示英语简介。

• Preprocessing datasets larger than memory
• Running batch predictions across multiple GPUs
• Preparing image and tabular data for training

Review data strategy plans through six focused questions covering training data permissions, infrastructure choices, data asset value, and hiring. Produce a structured verdict with unresolved issues and three concrete next steps.

中文简介正在准备中。当前显示英语简介。

• Review customer data permissions before AI training
• Compare data architecture options
• Assess data assets before productization or M&A

Run supported Datagma operations through Rube MCP, a connection between agents and external services. The skill covers tool discovery, connection checks, required input formats, and workflows for executing multiple operations.

中文简介正在准备中。当前显示英语简介。

• Find supported tools for a Datagma task
• Verify the Datagma connection is active
• Run multiple Datagma operations together

Build data processing pipelines for machine learning, from preprocessing to batch predictions. Uses Ray Data to handle datasets larger than memory, distribute work across CPUs and GPUs, and feed data into training frameworks.

中文简介正在准备中。当前显示英语简介。

• Preprocessing datasets larger than memory
• Running batch predictions across multiple GPUs
• Preparing image and tabular data for training

Find and assess datasets for a research question. Dispatches Explorer agents to search across data source categories, then Explorer-Critic to stress-test each candidate. Produces a ranked list with feasibility grades. Make sure to use this skill whenever the user wants to identify or evaluate data sources — not to search for papers or run analysis. Triggers include: "find data", "what data should I use", "find a dataset for this", "where can I get data on X", "assess datasets", "what datasets exist for", "help me find data", "is there data on this", "what are my data options", "I need data for this project", or any request to locate empirical data sources for a research question.

中文简介正在准备中。当前显示原文说明。

Connect MATLAB to Databricks via Spark (Databricks Connect) or JDBC (Database Toolbox). Use when setting up the MATLAB Interface for Databricks, configuring authentication (OauthU2M, OauthM2M, PAT), creating Spark sessions with getDatabricksSession(), reading Unity Catalog tables with server-side filtering, creating JDBC connections with databricks.JDBCConnection or StandaloneJDBCConnection, connecting to SQL Warehouses, selecting JDBC drivers (Simba/OSS), or writing data back to Databricks. Triggers on: Databricks Connect, Spark from MATLAB, getDatabricksSession, .databrickscfg, databricks.JDBCConnection, StandaloneJDBCConnection, SQLWarehouse, Databricks JDBC, Databricks cluster, large table server-side filtering.

中文简介正在准备中。当前显示原文说明。

Reads from, writes to, and manages relational databases using MATLAB Database Toolbox. Use when connecting to databases, reading data with sqlread or fetch, filtering with rowfilter, writing with sqlwrite, updating with sqlupdate, executing SQL statements, managing transactions with commit and rollback, mapping MATLAB classes to tables with ORM (Mappable, ormread, ormwrite, ormupdate), or performing any database operation from MATLAB. Triggers on: database, SQL, sqlread, sqlwrite, sqlupdate, fetch, execute, rowfilter, RowFilter, ORM, Mappable, ormread, ormwrite, ormupdate, orm2sql, transaction, commit, rollback, Database Toolbox, PostgreSQL, MySQL, SQLite, SQL Server, Oracle, database connection, database table, query database, insert data, update rows, delete rows, stored procedure, prepared statement, odbc, databaseConnectionOptions, datasource, data source, DSN, connection string, multithreaded, parallel.

中文简介正在准备中。当前显示原文说明。

Generate Python code for Dataverse integrations with error handling, retries, logging, and shared connection management. Include type hints, usage examples, and queries that retrieve only the records and columns needed for the task.

中文简介正在准备中。当前显示英语简介。

• Build Python integrations with Dataverse
• Add retries and logging to API operations
• Limit retrieved records and columns

Design AI agent analytics workflows across distributed tables and documents. Combine Knowledge Catalog for metadata, meaning information about data and its usage rules, with managed Spark processing to produce architecture diagrams, deployment guidance, and validation plans.

中文简介正在准备中。当前显示英语简介。

• Design analytics across data stored in multiple clouds
• Define metadata and governance with Knowledge Catalog
• Plan Data Agent Kit use in VS Code or Antigravity

Trace where BigQuery and Cloud Storage data comes from and where it goes. The skill summarizes table or column lineage, meaning data origins and movement, in a readable Markdown report to support data quality investigations.

中文简介正在准备中。当前显示英语简介。

• Investigating data origins during a quality issue
• Summarizing data sources and destinations
• Tracing the flow of a specific column

Choose a Google Cloud database by clarifying data needs, workload, and scale. The skill explains its recommendation and prepares validated Terraform configurations or provisioning scripts for review before infrastructure changes are made.

中文简介正在准备中。当前显示英语简介。

• Choosing a database for a new Google Cloud application
• Clarifying data models, workloads, and scale requirements
• Preparing and validating database provisioning code for review