“parquet”的搜索结果

Use when running analytical SQL over Parquet/CSV/JSON without a warehouse, replacing pandas for data wrangling, joining S3 data in-place, building local data marts, or embedding OLAP into an app. Triggers: read_parquet/read_csv setup, partitioned dataset queries, hive partitioning, glob patterns for S3, COPY TO export, attach Postgres/MySQL, UDFs in Python/R, MotherDuck cloud sync, columnar performance vs row stores. NOT for OLTP workloads (concurrent writes), distributed analytics at petabyte scale (use Spark/Trino), or vector search (use pgvector/Lance).

中文简介正在准备中。当前显示原文说明。

Answers plain-word questions from a repo's CSV, JSON, Parquet and YAML files, or a .duckdb file, with DuckDB, and shows the SQL it ran. Read-only. Trigger to query, count, join, filter or summarise data files, including analytics CSVs exported from a video platform; to build a Parquet or DuckDB cache; to check no .duckdb, .wal or .parquet cache is committed or stale; when a query hits a locked database file; or say duckrunner (ask, cache, check, refresh). Needs the user-installed duckdb Python package; never a database server. DuckDB docs, S3 and MotherDuck are the duckdb-skills plugin's; outside research researchscribe's.

中文简介正在准备中。当前显示原文说明。

Work with geographic points, lines, and areas in Python using GeoPandas. Guidance and local audit tools help check coordinate systems, geometry validity, spatial join matches, and export settings before trusting or sharing analysis results.

中文简介正在准备中。当前显示英语简介。

• Checking geographic data before analysis
• Reviewing coordinate systems and transformations
• Auditing matches between points and areas

Create and manage Arize datasets for evaluation and experiments using the ax CLI. Import records from files, append examples, inspect dataset versions, and export a chosen version for offline analysis, with checks for field consistency and export completeness.

中文简介正在准备中。当前显示英语简介。

• Creating evaluation datasets from files
• Adding examples to an existing test set
• Inspecting dataset metadata and versions

Clean a target Excel column, convert its values to numbers, and compare the calculated total with a summary sheet. Export the totals, match status, and difference in Excel and CSV files.

中文简介正在准备中。当前显示英语简介。

• Standardize numerical formats before calculating a column total
• Check detail totals against a summary sheet
• Export reconciliation results to Excel and CSV

Scan multiple Excel sheets for rows with missing or invalid entries in a target column. Combine matching records with their source sheet names, then export an Excel file with every extracted data row filled red for review.

中文简介正在准备中。当前显示英语简介。

• Find missing entries across multiple Excel sheets
• Create a review list with source sheet names
• Export matching rows with red highlighting

Analyze one Excel worksheet by cleaning column names, extracting numbers, and handling gaps from merged cells. Create category counts, percentages, cross-tabulations comparing groups, high-resolution charts, and CSV exports, with Parquet storage for the data.

中文简介正在准备中。当前显示英语简介。

• Clean worksheets containing merged cells
• Summarize category counts and percentages
• Compare groups with cross-tabulations

Analyze Excel workbooks across multiple sheets, locate matching fields, clean values, and compare groups. Produce counts, cross-tabulations that compare categories, charts, and an Excel report with a download link.

中文简介正在准备中。当前显示英语简介。

• Count matching fields across multiple sheets
• Clean numeric values and merged-cell groups
• Compare counts and shares by category

Clean category data in Excel, group entries, and calculate counts, totals, and shares. Create a summary spreadsheet with a total row and highlighted maximum values, alongside a bar chart for comparing groups.

中文简介正在准备中。当前显示英语简介。

• Cleaning category labels before analysis
• Combining labels into defined groups
• Calculating group counts, totals, and shares

Extract key metrics from spreadsheet data, check whether recorded values match calculations after unit conversion, and rank selected metrics from highest to lowest. Export the ranking or validation results to an Excel file for download.

中文简介正在准备中。当前显示英语简介。

• Check calculations involving different units
• Rank business metrics from highest to lowest
• Export validation results to Excel

Migrate a whole ArcGIS Hub site into a PortalJS Arc portal end-to-end. Harvests the Hub /data.json (DCAT-US) inventory, exports every FeatureService layer through the ArcGIS REST query API with resultOffset paging, converts each to the serverless dual tier (PMTiles render + GeoParquet query) with tabular items to Parquet, pushes everything to Cloudflare R2 via Git LFS, appends dual-tier datasets.json entries, and writes a source-vs-derived parity report. Use to move a City or sector ArcGIS Hub open-data portal onto PortalJS with no server-side compute.

中文简介正在准备中。当前显示原文说明。

Auto-ingest a geospatial file (GeoJSON, Shapefile, GeoPackage, KML/KMZ, FlatGeobuf, CSV-with-geometry) into a PortalJS portal on the user's own machine, with no server. Normalizes CRS to EPSG:4326, derives a PMTiles render tier and a GeoParquet query tier, pushes all three artifacts to Cloudflare R2 via Git LFS, and appends one dual-tier datasets.json entry the showcase auto-renders. Use when the source is a vector geo format that needs a map or spatial-query view.

中文简介正在准备中。当前显示原文说明。

Guide users or agents to the correct MATLAB tool for processing large tabular data in file-based formats (CSV, Parquet, delimited text, spreadsheets, MDF) that may not fit in memory. Use when a user or agent mentions large files, big data, out-of-memory errors, OOM, scaling up, tall arrays, datastores, or needs to process multiple tabular files. Covers the decision between datastore + tall, datastore + transform, and parallel execution. Also use when a user or agent has working in-memory code (readtable, parquetread) that runs out of memory and needs a migration path. Also covers building custom datastore classes for proprietary or non-standard formats — use when the task requires subclassing matlab.io.Datastore, implementing a custom reader, building an extensible datastore, or integrating a new file format with tall arrays or parallel computing. Do NOT use for MAT files (use matfile instead).

中文简介正在准备中。当前显示原文说明。

Convert construction data to/from Parquet format. Optimize storage, enable fast queries, and integrate with data lakehouses.

中文简介正在准备中。当前显示原文说明。

Use for writing, reviewing, debugging, testing, or optimizing Python code that uses PyArrow arrays, schemas, Tables, RecordBatches, RecordBatchReaders, compute kernels, Dataset scans, Parquet, or Arrow IPC and C data interfaces. Trigger on schema drift, null/type semantics, chunking, batch streaming, predicate projection pushdown, conversion, and memory ownership. Do not use for pandas-only transformations, Polars-only expressions, generic SQL, Arrow Flight service design, C++/Rust Arrow, or Parquet work that does not use PyArrow.

中文简介正在准备中。当前显示原文说明。

DuckDB is an in-process analytical database that runs embedded inside your application with zero external dependencies. It can query CSV, Parquet, and JSON files directly without loading them into tables first, making it ideal for local data exploration, ETL pipelines, and analytical workloads where spinning up a server is overkill.

中文简介正在准备中。当前显示原文说明。

数据工程 — 数据平台从业者的认知操作系统, 覆盖把数据从源系统搬运成可靠 / 可查询 / 可信赖形态供分析 / ML / 数据产品消费的全生命周期 (生成 → 摄取 → 存储 → 转换 → 服务 + 安全/数据管理/DataOps/数据架构/编排/软件工程 六条暗流, Reis & Housley 框架): 摄取与集成 (批 + CDC 变更数据捕获 Debezium + EL 工具 Fivetran/Airbyte/Meltano/dlt + Kafka Connect + schema drift) / 存储与文件表格式 (对象存储数据湖 + 列存 Parquet/ORC/Arrow/Avro + 开放表格式 Apache Iceberg/Delta Lake/Apache Hudi + lakehouse + 分区/compaction) / 转换与建模 (ELT dbt/SQLMesh + Spark + 维度建模 Kimball + Inmon + Data Vault + 大宽表 OBT + 渐变维 SCD + 增量模型 + 语义/指标层) / 编排与工作流 (Apache Airflow/Dagster/Prefect/Mage/Kestra/Apache DolphinScheduler + DAG + 幂等 + 回填 backfill + 数据资产调度) / 批流与实时 (Apache Kafka/Apache Flink/Spark Structured Streaming/Kinesis/Pulsar/Redpanda + Lambda vs Kappa + watermark/窗口/exactly-once + 流式 SQL Materialize/RisingWave + 实时 OLAP ClickHouse/Apache Druid/Apache Pinot/StarRocks/Apache Doris) / 数仓与查询引擎 (Snowflake/BigQuery/Redshift/Databricks SQL/Trino/Presto/DuckDB/Polars + 存算分离 + MPP) / 数据质量测试与可观测性 (dbt tests/Great Expectations/Soda + 数据契约 + Monte Carlo data downtime + 新鲜度/量/schema 异常检测) / 数据治理编目与血缘 (DataHub/Amundsen/OpenMetadata/Unity Catalog + 列级血缘 + PII 分类 + 访问控制 + GDPR) / DataOps 与可靠性 (数据 CI/CD + 转换版本控制 + 环境隔离 + 幂等重处理 + 数据 SLA/SLO + 计算存储 FinOps) / 数据架构范式 (现代数据栈 + lakehouse + data mesh + data fabric + 去中心化 vs 中心化所有权) / 分析工程角色 (dbt 时代连接数据工程与分析的桥) — 不含 数据科学/ML 建模本身 (是下游消费者) / BI 仪表盘制作 (serving 下游) / 数据分析报表为终点 / 'data engineer = 跑 Hadoop 的' 过时窄化 / 通用后端应用开发 (平行学科) (Data Engineering — the cognitive operating system of practitioners who design, build, and operate the data platform: moving data from source systems into reliable, queryable, trustworthy form for analytics / ML / products, covering (a) the data engineering lifecycle (generation → ingestion → storage → transformation → serving, with the undercurrents security / data management / DataOps / data architecture / orchestration / software engineering — Reis & Housley framing), (b) ingestion & integration (batch + CDC change-data-capture with Debezium, EL tools Fivetran / Airbyte / Meltano / dlt, Kafka Connect, API + file + database sources, schema drift handling), (c) storage & file/table formats (object storage data lakes, columnar formats Parquet / ORC / Arrow / Avro, open table formats Apache Iceberg / Delta Lake / Apache Hudi, lakehouse architecture, partitioning / compaction / Z-ordering), (d) transformation & modeling (ELT with dbt / SQLMesh, Spark, dimensional modeling Kimball, Inmon CIF, Data Vault, One Big Table / wide tables, normalization vs denormalization, slowly changing dimensions, incremental models, the semantic / metrics layer), (e) orchestration & workflow (Apache Airflow, Dagster, Prefect, Mage, Kestra, Apache DolphinScheduler, DAGs, idempotency, backfills, data-aware / asset-based scheduling), (f) batch vs streaming & real-time (Apache Kafka, Apache Flink, Spark Structured Streaming, Kinesis / Pulsar / Redpanda, the Lambda vs Kappa debate, watermarks / windowing / exactly-once, streaming SQL Materialize / RisingWave, real-time OLAP ClickHouse / Apache Druid / Apache Pinot / StarRocks / Apache Doris), (g) warehouses & query engines (Snowflake, BigQuery, Redshift, Databricks SQL, Trino / Presto, DuckDB, Polars, decoupled storage & compute, MPP), (h) data quality, testing & observability (dbt tests, Great Expectations, Soda, data contracts, Monte Carlo / data downtime, freshness / volume / schema anomaly detection, unit / integration testing of pipelines), (i) data governance, catalog & lineage (DataHub, Amundsen, OpenMetadata, Unity Catalog, column-level lineage, PII / data classification, access control, GDPR / data privacy), (j) DataOps & reliability (CI/CD for data, version control of transformations, environments, idempotent reprocessing, SLAs / SLOs for data, cost / FinOps for compute & storage), (k) data architecture paradigms (modern data stack, data lakehouse, data mesh, data fabric, decentralized vs centralized ownership), (l) the analytics engineering role (the dbt-era bridge between data engineering and analysis); N

中文简介正在准备中。当前显示原文说明。

Use when the user wants to run SQL - especially analytical SQL - on local files (parquet/csv/json), URLs, S3 paths, or remote databases (Postgres, MySQL, MongoDB, ClickHouse Cloud, Iceberg, Delta Lake) without setting up a server. Provides chDB - embedded ClickHouse SQL in Python with 1000+ functions, Session for stateful multi-step pipelines, parametrized queries, and cross-source joins via `s3()`, `mysql()`, `postgresql()`, `iceberg()`, `deltaLake()`, `remoteSecure()` table functions. TRIGGER when: user wants SQL on parquet/csv/files or across remote analytical sources; uses ClickHouse SQL features (window functions, windowFunnel, geoToH3, JSON path ops, Session, parametrized queries); imports `chdb` or calls `chdb.query()`. SKIP this skill for pandas-style DataFrame method-chaining (use chdb-datastore instead) or ClickHouse server administration.

中文简介正在准备中。当前显示原文说明。

Use when the user has tabular data (pandas DataFrame, parquet, csv, Arrow, json) and wants to filter, group, aggregate, join, or speed up slow pandas. Provides chDB DataStore - same pandas API, ClickHouse engine underneath. Also handles reading from S3, MySQL, PostgreSQL, MongoDB, ClickHouse Cloud, Iceberg, Delta Lake as DataFrames and joining across sources. TRIGGER when: user mentions DataFrame, parquet, csv, "fast pandas", "speed up pandas", or cross-source DataFrame joins; user imports `chdb.datastore` or `from datastore import DataStore`. SKIP this skill for raw SQL syntax (use chdb-sql instead), ClickHouse server administration, or non-Python DataStore API work.

中文简介正在准备中。当前显示原文说明。

DuckDB CLI specialist for SQL analysis, data processing and file conversion. Use for SQL queries, CSV/Parquet/JSON analysis, database queries, or data conversion. Triggers on "duckdb", "sql", "query", "data analysis", "parquet", "convert data".

中文简介正在准备中。当前显示原文说明。

Prepare raw credit data for pre-loan modeling by checking data quality and screening variables. Review missing values, predictive usefulness, stability, noise, and correlation, then export an Excel report with organization-level statistics and cleaning results.

中文简介正在准备中。当前显示英语简介。

• Prepare credit data before building a model
• Compare missing values and bad-sample rates across organizations
• Screen variables for usefulness, stability, and redundancy

Prepare training datasets for large language models with GPU-assisted cleaning. Covers duplicate removal, quality filtering, and personal information redaction, with workflows for text, images, video, and audio using NeMo Curator.

中文简介正在准备中。当前显示英语简介。

• Clean web data for model training
• Remove exact and near duplicates from large text collections
• Redact personal information from training text

Explore a table or data file before starting analysis. Summarize its structure, missing values, duplicates, and distributions, flag potential quality issues, and suggest useful measures, grouping columns, and follow-up analyses.

中文简介正在准备中。当前显示英语简介。

• Understanding an unfamiliar table or data file
• Checking missing values, duplicates, and suspicious entries
• Choosing measures and grouping columns for analysis

Diagnoses and fixes Java style violations in the FE module using Checkstyle, a tool that checks coding rules. It interprets errors, applies the project's formatting and import restrictions, and reruns checks to verify fixes.

中文简介正在准备中。当前显示英语简介。

• Checking FE Java changes before committing
• Fixing build failures caused by Checkstyle violations
• Understanding project-specific style and import rules