28건 · 관련도순
Build Python workflows for filtering, grouping, joining, and reshaping table data with Polars. Get guidance on migrating pandas code and using lazy evaluation, which optimizes a processing plan before execution, for large datasets.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Work with genomic regions in Polars tables to find overlaps, nearby regions, covered bases, and gaps. Read and write bioinformatics files, query data with SQL, and check coordinate conventions when joining datasets or migrating from bioframe.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Use for writing, reviewing, debugging, or testing Pandera schemas and runtime validation for Polars DataFrame or LazyFrame pipelines installed with pandera[polars]. Trigger on pandera.polars DataFrameSchema, DataFrameModel, Column, Field, Check, PolarsData, decorators, coercion, strictness, lazy error collection, and validation-depth decisions. Do not use for pandas-backed Pandera, Pydantic object models, Polars transformations without Pandera, static dataframe typing alone, or generic data-quality platforms.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Analyze CSV, Parquet, and JSON files to answer questions through grouping, joins, cleaning, time series, and charts. Chooses DuckDB or Polars and a suitable processing location based on the task and available memory.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Apply polars in reproducible local data workflows with version-aware APIs, explicit assumptions, and validation. Use when the user chooses polars or its strengths fit the task.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Spark, pandas, polars, DuckDB optimization for batch data processing. Activate on: batch processing, Spark optimization, polars, DuckDB, pandas performance, data frame, shuffle, partition, memory optimization. NOT for: streaming pipelines (use streaming-pipeline-architect), warehouse queries (use data-warehouse-optimizer).
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Spark, pandas, polars, DuckDB optimization for batch data processing. Activate on: batch processing, Spark optimization, polars, DuckDB, pandas performance, data frame, shuffle, partition, memory optimization. NOT for: streaming pipelines (use streaming-pipeline-architect), warehouse queries (use data-warehouse-optimizer).
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Bioinformatics-specific data operations using Polars DataFrames
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Best practices for polars data processing with dataframely. Covers definitions of Schema and Collection, usage of .validate() and .filter(), type hints, and testing. Use when writing or modifying code involving dataframely or polars data frames.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Transform raw data into analytical assets using ETL/ELT patterns, SQL (dbt), Python (pandas/polars/PySpark), and orchestration (Airflow). Use when building data pipelines, implementing incremental models, migrating from pandas to polars, or orchestrating multi-step transformations with testing and quality checks.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
数据工程 — 数据平台从业者的认知操作系统, 覆盖把数据从源系统搬运成可靠 / 可查询 / 可信赖形态供分析 / ML / 数据产品消费的全生命周期 (生成 → 摄取 → 存储 → 转换 → 服务 + 安全/数据管理/DataOps/数据架构/编排/软件工程 六条暗流, Reis & Housley 框架): 摄取与集成 (批 + CDC 变更数据捕获 Debezium + EL 工具 Fivetran/Airbyte/Meltano/dlt + Kafka Connect + schema drift) / 存储与文件表格式 (对象存储数据湖 + 列存 Parquet/ORC/Arrow/Avro + 开放表格式 Apache Iceberg/Delta Lake/Apache Hudi + lakehouse + 分区/compaction) / 转换与建模 (ELT dbt/SQLMesh + Spark + 维度建模 Kimball + Inmon + Data Vault + 大宽表 OBT + 渐变维 SCD + 增量模型 + 语义/指标层) / 编排与工作流 (Apache Airflow/Dagster/Prefect/Mage/Kestra/Apache DolphinScheduler + DAG + 幂等 + 回填 backfill + 数据资产调度) / 批流与实时 (Apache Kafka/Apache Flink/Spark Structured Streaming/Kinesis/Pulsar/Redpanda + Lambda vs Kappa + watermark/窗口/exactly-once + 流式 SQL Materialize/RisingWave + 实时 OLAP ClickHouse/Apache Druid/Apache Pinot/StarRocks/Apache Doris) / 数仓与查询引擎 (Snowflake/BigQuery/Redshift/Databricks SQL/Trino/Presto/DuckDB/Polars + 存算分离 + MPP) / 数据质量测试与可观测性 (dbt tests/Great Expectations/Soda + 数据契约 + Monte Carlo data downtime + 新鲜度/量/schema 异常检测) / 数据治理编目与血缘 (DataHub/Amundsen/OpenMetadata/Unity Catalog + 列级血缘 + PII 分类 + 访问控制 + GDPR) / DataOps 与可靠性 (数据 CI/CD + 转换版本控制 + 环境隔离 + 幂等重处理 + 数据 SLA/SLO + 计算存储 FinOps) / 数据架构范式 (现代数据栈 + lakehouse + data mesh + data fabric + 去中心化 vs 中心化所有权) / 分析工程角色 (dbt 时代连接数据工程与分析的桥) — 不含 数据科学/ML 建模本身 (是下游消费者) / BI 仪表盘制作 (serving 下游) / 数据分析报表为终点 / 'data engineer = 跑 Hadoop 的' 过时窄化 / 通用后端应用开发 (平行学科) (Data Engineering — the cognitive operating system of practitioners who design, build, and operate the data platform: moving data from source systems into reliable, queryable, trustworthy form for analytics / ML / products, covering (a) the data engineering lifecycle (generation → ingestion → storage → transformation → serving, with the undercurrents security / data management / DataOps / data architecture / orchestration / software engineering — Reis & Housley framing), (b) ingestion & integration (batch + CDC change-data-capture with Debezium, EL tools Fivetran / Airbyte / Meltano / dlt, Kafka Connect, API + file + database sources, schema drift handling), (c) storage & file/table formats (object storage data lakes, columnar formats Parquet / ORC / Arrow / Avro, open table formats Apache Iceberg / Delta Lake / Apache Hudi, lakehouse architecture, partitioning / compaction / Z-ordering), (d) transformation & modeling (ELT with dbt / SQLMesh, Spark, dimensional modeling Kimball, Inmon CIF, Data Vault, One Big Table / wide tables, normalization vs denormalization, slowly changing dimensions, incremental models, the semantic / metrics layer), (e) orchestration & workflow (Apache Airflow, Dagster, Prefect, Mage, Kestra, Apache DolphinScheduler, DAGs, idempotency, backfills, data-aware / asset-based scheduling), (f) batch vs streaming & real-time (Apache Kafka, Apache Flink, Spark Structured Streaming, Kinesis / Pulsar / Redpanda, the Lambda vs Kappa debate, watermarks / windowing / exactly-once, streaming SQL Materialize / RisingWave, real-time OLAP ClickHouse / Apache Druid / Apache Pinot / StarRocks / Apache Doris), (g) warehouses & query engines (Snowflake, BigQuery, Redshift, Databricks SQL, Trino / Presto, DuckDB, Polars, decoupled storage & compute, MPP), (h) data quality, testing & observability (dbt tests, Great Expectations, Soda, data contracts, Monte Carlo / data downtime, freshness / volume / schema anomaly detection, unit / integration testing of pipelines), (i) data governance, catalog & lineage (DataHub, Amundsen, OpenMetadata, Unity Catalog, column-level lineage, PII / data classification, access control, GDPR / data privacy), (j) DataOps & reliability (CI/CD for data, version control of transformations, environments, idempotent reprocessing, SLAs / SLOs for data, cost / FinOps for compute & storage), (k) data architecture paradigms (modern data stack, data lakehouse, data mesh, data fabric, decentralized vs centralized ownership), (l) the analytics engineering role (the dbt-era bridge between data engineering and analysis); N
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Use this when performing DataFrame operations — including loading, filtering, joining, aggregating, transforming, or reshaping tabular data with polars or pandas.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Specialist skill for Python data engineering — pandas, polars, DuckDB, numpy, ETL pipelines, tabular data ingestion, and notebook-to-module extraction. Use when working with dataframes, data validation at ingress boundaries, merge/join operations, typed column contracts, or choosing between pandas vs polars vs DuckDB for a data task.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Processes and analyzes data with resident-kernel engines (DuckDB, Polars) and one-shot tools. Use for CSV/parquet/JSON analysis, group-by/join/aggregation, time series, distributions, cleaning, or plotting a dataset.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Fast in-memory DataFrame library for datasets that fit in RAM. Use when pandas is too slow but data still fits in memory. Lazy evaluation, parallel execution, Apache Arrow backend. Best for 1-100GB datasets, ETL pipelines, faster pandas replacement. For larger-than-RAM data use dask or vaex.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
High-performance DataFrame operations for large-scale data analysis
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Out-of-core DataFrame for billion-row data via lazy evaluation and memory-mapped files. Use when data exceeds RAM (10 GB–TB) for fast aggregation, filtering, virtual columns, and visualization without loading. Supports HDF5, Arrow, Parquet, CSV with cloud (S3, GCS, Azure). Built-in ML transformers (scaling, PCA, K-means). In-memory: polars; distributed: dask.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Fast in-memory DataFrame with lazy evaluation, parallel execution, Arrow backend. Use for tabular data in RAM (1–100 GB) when pandas is too slow. Expression API: select, filter, group_by, joins, pivots, window. Lazy mode enables predicate/projection pushdown. Reads CSV, Parquet, JSON, Excel, DBs, cloud. Larger-than-RAM: Dask; GPU: cuDF.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Core Python library for astronomy/astrophysics: units with dimensional analysis, celestial coordinate transforms (ICRS/Galactic/AltAz/FK5), FITS I/O, tables (FITS/HDF5/VOTable/CSV), cosmology (Planck18, distance/age), precise time (UTC/TAI/TT/TDB, Julian, barycentric), WCS pixel-world mapping, model fitting. For general tables use pandas/polars; for radio interferometry use CASA.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Parallel/distributed computing for larger-than-RAM data. Components: DataFrames (parallel pandas), Arrays (parallel NumPy), Bags, Futures, Schedulers. Scales laptop to HPC cluster. For single-machine speed use polars; for out-of-core without cluster use vaex.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Use for writing, reviewing, debugging, testing, or optimizing Python code that uses PyArrow arrays, schemas, Tables, RecordBatches, RecordBatchReaders, compute kernels, Dataset scans, Parquet, or Arrow IPC and C data interfaces. Trigger on schema drift, null/type semantics, chunking, batch streaming, predicate projection pushdown, conversion, and memory ownership. Do not use for pandas-only transformations, Polars-only expressions, generic SQL, Arrow Flight service design, C++/Rust Arrow, or Parquet work that does not use PyArrow.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Think and work like an expert Aeronautical Engineer. Use when a task calls for Aeronautical Engineer judgment. Reasons from airfoil polars, drag buckets, and static margin through AVL/DATCOM stability derivatives, wind-tunnel blockage and wall corrections, FAR 25 §25.101–25.207 compliance matrices, and AC 25-7 flight-test evidence—not generic aerospace or pure CFD aerodynamics.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Think and work like an expert Aerodynamicist. Use when a task calls for Aerodynamicist judgment. Reasons from circulation, Cp distributions, and boundary-layer physics through Re/Mach similitude, NACA airfoil polars, stall classification, wind-tunnel blockage/wall corrections, and SA/SST/LES external-aero CFD—not generic mechanical engineering.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Use when dataframe code is slow or runs out of memory, when data outgrows pandas, or when writing pandas 3, Polars or DuckDB code — vectorisation, dtypes, Copy-on-Write, lazy and out-of-core execution, and measuring before and after
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.