Results for “big data”

数据工程 — 数据平台从业者的认知操作系统, 覆盖把数据从源系统搬运成可靠 / 可查询 / 可信赖形态供分析 / ML / 数据产品消费的全生命周期 (生成 → 摄取 → 存储 → 转换 → 服务 + 安全/数据管理/DataOps/数据架构/编排/软件工程 六条暗流, Reis & Housley 框架): 摄取与集成 (批 + CDC 变更数据捕获 Debezium + EL 工具 Fivetran/Airbyte/Meltano/dlt + Kafka Connect + schema drift) / 存储与文件表格式 (对象存储数据湖 + 列存 Parquet/ORC/Arrow/Avro + 开放表格式 Apache Iceberg/Delta Lake/Apache Hudi + lakehouse + 分区/compaction) / 转换与建模 (ELT dbt/SQLMesh + Spark + 维度建模 Kimball + Inmon + Data Vault + 大宽表 OBT + 渐变维 SCD + 增量模型 + 语义/指标层) / 编排与工作流 (Apache Airflow/Dagster/Prefect/Mage/Kestra/Apache DolphinScheduler + DAG + 幂等 + 回填 backfill + 数据资产调度) / 批流与实时 (Apache Kafka/Apache Flink/Spark Structured Streaming/Kinesis/Pulsar/Redpanda + Lambda vs Kappa + watermark/窗口/exactly-once + 流式 SQL Materialize/RisingWave + 实时 OLAP ClickHouse/Apache Druid/Apache Pinot/StarRocks/Apache Doris) / 数仓与查询引擎 (Snowflake/BigQuery/Redshift/Databricks SQL/Trino/Presto/DuckDB/Polars + 存算分离 + MPP) / 数据质量测试与可观测性 (dbt tests/Great Expectations/Soda + 数据契约 + Monte Carlo data downtime + 新鲜度/量/schema 异常检测) / 数据治理编目与血缘 (DataHub/Amundsen/OpenMetadata/Unity Catalog + 列级血缘 + PII 分类 + 访问控制 + GDPR) / DataOps 与可靠性 (数据 CI/CD + 转换版本控制 + 环境隔离 + 幂等重处理 + 数据 SLA/SLO + 计算存储 FinOps) / 数据架构范式 (现代数据栈 + lakehouse + data mesh + data fabric + 去中心化 vs 中心化所有权) / 分析工程角色 (dbt 时代连接数据工程与分析的桥) — 不含 数据科学/ML 建模本身 (是下游消费者) / BI 仪表盘制作 (serving 下游) / 数据分析报表为终点 / 'data engineer = 跑 Hadoop 的' 过时窄化 / 通用后端应用开发 (平行学科) (Data Engineering — the cognitive operating system of practitioners who design, build, and operate the data platform: moving data from source systems into reliable, queryable, trustworthy form for analytics / ML / products, covering (a) the data engineering lifecycle (generation → ingestion → storage → transformation → serving, with the undercurrents security / data management / DataOps / data architecture / orchestration / software engineering — Reis & Housley framing), (b) ingestion & integration (batch + CDC change-data-capture with Debezium, EL tools Fivetran / Airbyte / Meltano / dlt, Kafka Connect, API + file + database sources, schema drift handling), (c) storage & file/table formats (object storage data lakes, columnar formats Parquet / ORC / Arrow / Avro, open table formats Apache Iceberg / Delta Lake / Apache Hudi, lakehouse architecture, partitioning / compaction / Z-ordering), (d) transformation & modeling (ELT with dbt / SQLMesh, Spark, dimensional modeling Kimball, Inmon CIF, Data Vault, One Big Table / wide tables, normalization vs denormalization, slowly changing dimensions, incremental models, the semantic / metrics layer), (e) orchestration & workflow (Apache Airflow, Dagster, Prefect, Mage, Kestra, Apache DolphinScheduler, DAGs, idempotency, backfills, data-aware / asset-based scheduling), (f) batch vs streaming & real-time (Apache Kafka, Apache Flink, Spark Structured Streaming, Kinesis / Pulsar / Redpanda, the Lambda vs Kappa debate, watermarks / windowing / exactly-once, streaming SQL Materialize / RisingWave, real-time OLAP ClickHouse / Apache Druid / Apache Pinot / StarRocks / Apache Doris), (g) warehouses & query engines (Snowflake, BigQuery, Redshift, Databricks SQL, Trino / Presto, DuckDB, Polars, decoupled storage & compute, MPP), (h) data quality, testing & observability (dbt tests, Great Expectations, Soda, data contracts, Monte Carlo / data downtime, freshness / volume / schema anomaly detection, unit / integration testing of pipelines), (i) data governance, catalog & lineage (DataHub, Amundsen, OpenMetadata, Unity Catalog, column-level lineage, PII / data classification, access control, GDPR / data privacy), (j) DataOps & reliability (CI/CD for data, version control of transformations, environments, idempotent reprocessing, SLAs / SLOs for data, cost / FinOps for compute & storage), (k) data architecture paradigms (modern data stack, data lakehouse, data mesh, data fabric, decentralized vs centralized ownership), (l) the analytics engineering role (the dbt-era bridge between data engineering and analysis); N

The English summary is being prepared. Showing the original description.

Generate Python code to clean, transform, analyze, visualize, and model BigQuery data with BigQuery DataFrames. The skill keeps processing in the cloud and uses familiar table operations, with guidance for previews, schema checks, and machine learning.

• Clean and transform BigQuery data with Python
• Preview data and check column types
• Visualize aggregated or sampled data

Work with BigQuery to create datasets and tables, run SQL queries, ingest data, and manage processing jobs. The skill also covers connections to BigQuery ML and Gemini for machine learning and AI analysis.

• Creating datasets and tables
• Analyzing data with SQL queries
• Ingesting data and managing processing jobs

Manage BigQuery datasets, tables, views, and jobs, and run SQL queries for basic data analysis. Provides setup examples and references for data ingestion, command-line operations, client libraries, access permissions, and infrastructure configuration.

• Create BigQuery datasets and tables
• Search and aggregate data with SQL
• Load data and manage processing jobs

Trace where BigQuery and Cloud Storage data comes from and where it goes. The skill summarizes table or column lineage, meaning data origins and movement, in a readable Markdown report to support data quality investigations.

• Investigating data origins during a quality issue
• Summarizing data sources and destinations
• Tracing the flow of a specific column

Assess how changes, failures, or paused updates to a BigQuery table or view affect downstream assets. Trace data lineage, the relationships showing where data flows, and summarize dependent tables, dashboards, and processes with direct and indirect impacts.

• Assess downstream effects of a table failure
• Check dependencies before changing or deleting a view
• Evaluate the impact of pausing data updates

Automate Big Data Cloud operations through Rube MCP, a connection for agent tools. The skill finds available actions, checks their required inputs and connection status, and executes them using Composio's toolkit.

• Finding available Big Data Cloud tools
• Checking current tool input requirements
• Automating operations after verifying the connection

Automate Big Data Cloud operations through Rube MCP, a connection for agent tools. The skill finds available actions, checks their required inputs and connection status, and executes them using Composio's toolkit.

• Finding available Big Data Cloud tools
• Checking current tool input requirements
• Automating operations after verifying the connection

Design a shared data platform across Google Cloud, AWS, and Azure for analytics and AI. The skill turns requirements into architecture diagrams, infrastructure code, deployment instructions, and validation checks using Apache Iceberg and BigQuery data agents.

• Designing a data platform across multiple clouds
• Planning data ingestion and AI serving components
• Preparing infrastructure code and deployment instructions

Find and download public cancer imaging data from NCI Imaging Data Commons. Select CT, MR, PET, or pathology datasets using metadata, view images in a browser, and check licenses and citations before using the data.

• Find cancer images by disease, modality, or anatomical site
• Download selected DICOM image series
• View radiology and pathology images in a browser

Create and work with Google Bigtable instances and tables, design row keys and column groups, and query data with SQL or client libraries. Investigate slow queries and concentrated access patterns using Key Visualizer and command-line diagnostics.

• Designing row keys and column families
• Creating Bigtable instances and tables
• Writing SQL queries or data access code

Import data into the AWS data lake from S3 files, local uploads, JDBC databases (Oracle, SQL Server, PostgreSQL, MySQL, RDS, Aurora), Amazon Redshift, Snowflake, BigQuery, DynamoDB, or existing Glue catalog tables (migration). Default target is S3 Tables; standard Iceberg on a general purpose bucket is supported where S3 Tables is not adopted. Handles one-time loads, recurring pipelines, migrations. Triggers on: import data, load data, ingest, sync database, migrate table, move data to AWS, set up pipeline, ETL, pull from Snowflake, query BigQuery into S3, export DynamoDB, CTAS, convert to Iceberg. Do NOT use for setting up or troubleshooting Glue connections (use connecting-to-data-source), creating empty tables (use creating-data-lake-table), running queries (use querying-data-lake), finding tables by fuzzy name (use finding-data-lake-assets), catalog audit (use exploring-data-catalog), or SaaS platforms like Salesforce, ServiceNow, SAP, MongoDB, Kafka.

The English summary is being prepared. Showing the original description.

Create or improve a company-specific data analysis skill by gathering analysts’ knowledge and examining database structures. Document business terms, metric formulas, table relationships, and common query patterns in reusable reference files.

• Create a data analysis skill for a company
• Document metric formulas and standard exclusions
• Add knowledge about a new business domain

Manage which n8n instance an MCP session targets when working across production, staging, teams, or clients. Guides instance switching, checks before credential changes, and recovery from unexpected missing data or ambiguous credential writes.

• Switching between production and staging
• Checking the target before credential changes
• Investigating unexpected missing data

Write SQL that uses BigQuery’s built-in machine learning and generative AI for forecasting, anomaly detection, identifying drivers of change, searching by meaning, and classifying or summarizing text stored in your data.

• Forecasting future values from time-series data
• Detecting anomalies and investigating drivers of change
• Searching documents by meaning

Find the Wren CLI workflow for answering data questions, connecting databases or SaaS sources, generating model definitions, adding business metrics, or building a shareable analytics app. Guides are retrieved from the installed CLI version.

• Set up Wren and connect a database
• Load HubSpot or Stripe data through dlt
• Generate model definitions and add business metrics

Manages datasets, tables, and jobs in BigQuery, and integrates with BigQuery ML and Gemini for advanced data analytics and AI-driven insights. Use when you need to interact with BigQuery, run SQL queries, manage BigQuery resources, or leverage BigQuery's built-in ML capabilities. Also use when performing data analysis, ingesting data into BigQuery, or developing AI applications on BigQuery.

The English summary is being prepared. Showing the original description.

Build Apache Spark pipelines for large datasets, including transformations, aggregations, and streaming analysis. Get PySpark or Scala code, partitioning strategies for dividing data across workers, and configuration advice to investigate slow jobs and resource usage.

• Build a pipeline to transform and aggregate large datasets
• Investigate slow Spark jobs and uneven data distribution
• Tune memory settings and data partitions

Create and troubleshoot AWS Glue connections to JDBC databases (Oracle, SQL Server, PostgreSQL, MySQL, RDS), Redshift, Snowflake, and BigQuery. Gathers connection hints from user, discovers existing connections and RDS/Redshift candidates, registers credentials in Secrets Manager or IAM DB auth, configures VPC, and tests. Triggers on: connect to database, set up Glue connection, register data source, connect to Snowflake/BigQuery/RDS, connection timeout, test connection, troubleshoot connection. Do NOT use for moving data (use ingesting-into-data-lake), creating tables (use creating-data-lake-table), queries (use querying-data-lake), catalog exploration (use exploring-data-catalog), or SaaS (Salesforce, ServiceNow, SAP, MongoDB, Kafka).

The English summary is being prepared. Showing the original description.

Pulls Bigdata.com (RavenPack) financial and news data via the official bigdata-client SDK and /v1/* REST endpoints — structured financials, prices, analyst estimates, entity-sentiment series, annotated chunk search, screener. Use when the Bigdata MCP's tearsheets/search feel thin and you need the machine-readable substrate: mentions of Bigdata.com, RavenPack, a bd_v2_ key, rp_entity_id, or chunk/query_unit cost.

The English summary is being prepared. Showing the original description.

Reads, queries, and writes bigWig indexed binary signal tracks (coverage, fold-change, conservation, methylation-rate) with pyBigWig (Python) and the UCSC Kent tools (bedGraphToBigWig, bigWigToBedGraph, bigWigInfo, bigWigSummary, bigWigAverageOverBed) and deepTools (multiBigwigSummary, computeMatrix, bigwigCompare). Covers the central trap that a wide query returns a precomputed zoom-level summary (by default the mean, which annihilates narrow peaks) not per-base data, when exact=True/values() is mandatory, the NaN-not-zero gap-handling fork, choosing mean vs max vs sum vs coverage by biological question, and the sorted-bedGraph plus chrom.sizes build requirement. Use when extracting signal at regions, computing mean signal per gene/peak, building a browser track from bedGraph, comparing tracks, or building TSS/gene-body metaprofiles.

The English summary is being prepared. Showing the original description.

Guide users or agents to the correct MATLAB tool for processing large tabular data in file-based formats (CSV, Parquet, delimited text, spreadsheets, MDF) that may not fit in memory. Use when a user or agent mentions large files, big data, out-of-memory errors, OOM, scaling up, tall arrays, datastores, or needs to process multiple tabular files. Covers the decision between datastore + tall, datastore + transform, and parallel execution. Also use when a user or agent has working in-memory code (readtable, parquetread) that runs out of memory and needs a migration path. Also covers building custom datastore classes for proprietary or non-standard formats — use when the task requires subclassing matlab.io.Datastore, implementing a custom reader, building an extensible datastore, or integrating a new file format with tall arrays or parallel computing. Do NOT use for MAT files (use matfile instead).

The English summary is being prepared. Showing the original description.

Analyzes the downstream impact (blast radius) when a BigQuery table or view is broken, stale, or modified. Identifies all downstream tables, dashboards, and processes that will be affected. Use when: - Performing a blast radius or impact analysis for a BigQuery table or view. - Assessing the consequences of modifying, deleting, or pausing updates to a BigQuery asset. - Identifying downstream dependencies (tables, dashboards, processes) of a BigQuery asset. Don't use for: - General BigQuery querying or data analysis (use BigQuery-related tools instead). - Non-BigQuery assets (e.g., Cloud Storage files) unless they are part of the BigQuery lineage. - Creating or modifying lineage links directly.

The English summary is being prepared. Showing the original description.

Analyzes Google Cloud BigQuery slot consumption, query costs, and execution bottlenecks using INFORMATION_SCHEMA. Use when diagnosing slow BigQuery queries, slot starvation, high on-demand query costs, unpartitioned table scans, or join performance issues. Don't use for generic BigQuery administration (use bigquery-basics), BigQuery ML (use bigquery-ai-ml), or DataFrame operations (use bigquery-bigframes).

The English summary is being prepared. Showing the original description.