209 skills · sorted by relevance
Audit one datafusion-spark function against Apache Spark 4.2.0. Compare source code, accepted types, and edge cases, verify expected results with PySpark, then fix differences and record remaining gaps as issues and disabled tests.
**未導入プロジェクト向け**の sparkle-design 初期導入・セットアップを支援するスキル。 npm インストール、sparkle-design-cli による初回 CSS 生成、sparkle.config.json の 新規作成、AI ガード（Sparkle Design Guard）の導入、コンポーネント選択ガイドまでをカバー。 **導入済みプロジェクトでテーマ（primary / font / radius）を変えたい場合は `change-sparkle-config` スキルを使うこと** — この setup スキルは初期導入専用。 「Sparkle Design を導入」「sparkle-design をインストール」「デザインシステムをセットアップ」 「コンポーネントライブラリを追加」「どのコンポーネントを使えばいい」で発動。 English: "install sparkle-design", "add sparkle design", "set up sparkle design", "which component should I use"
The English summary is being prepared. Showing the original description.
Build Apache Spark pipelines for large datasets, including transformations, aggregations, and streaming analysis. Get PySpark or Scala code, partitioning strategies for dividing data across workers, and configuration advice to investigate slow jobs and resource usage.
sparkle-design（公開 npm パッケージ）の新バージョンをリリースするための手順スキル。 package.json の version bump、CHANGELOG.md の更新、リリース PR 作成、PR マージ後の npm への stage、メンテナーによる 2FA 承認（npm stage approve）、承認後の git tag・ GitHub Release 作成までを一連の手順で実行する。 CHANGELOG 更新漏れと GitHub Release 作成漏れを防ぐためのチェックリストを含む。 「sparkle-design をリリース」「sparkle-design の新バージョンを切る」「vX.Y.Z をリリース」 「sparkle-design の CHANGELOG を更新」で発動。 English: "release sparkle-design", "cut a new sparkle-design version", "publish sparkle-design", "bump sparkle-design version".
The English summary is being prepared. Showing the original description.
Sparkle Design の UI コンポーネントを shadcn registry 経由でプロジェクトに追加するスキル。 パッケージマネージャの自動検出、設定バリデーション、コンポーネントインストール、 Storybook 統合、CSS セットアップ、トラブルシューティングをカバーする。 「Sparkle コンポーネントを追加」「sparkle-design のコンポーネントをインストール」 「@sparkle-design のセットアップ」「コンポーネントをどう追加する」 「registry の設定」「components.json の設定」への言及で発動する。 English: "add a sparkle button", "install sparkle-design card", "add @sparkle-design/input", "set up components from sparkle registry", "how do I install components", "registry setup"
The English summary is being prepared. Showing the original description.
Check ten recurring problems before or during machine learning runs on NVIDIA DGX Spark. Diagnose startup failures, out-of-memory errors, and falling throughput, with guidance on compatible environments, resource use, numeric formats, and training across two devices.
Improve slow Apache Spark jobs by reviewing data partitions, caching, memory settings, and shuffles, which move data between workers. Use the guidance to investigate bottlenecks and tune pipelines for larger datasets.
**導入済みの sparkle-design プロジェクト**で、ユーザーが目指したい「雰囲気」を 自然言語で伝えたら、`sparkle.config.json` の primary / font-pro / font-mono / radius を 書き換えて `sparkle-design-cli generate` まで実行するスキル。選択肢は Theme Settings Figma プラグインが扱える範囲（primary 7 色 / radius 8 段階 / fonts 11 種）に揃えて おり、Figma と CLI の見た目がずれません。**未導入プロジェクトの初期セットアップは `setup-sparkle-design`（internal 環境では `install-sparkle-design`）を使うこと。** 「雰囲気を変えたい」「もっとポップに」「ビジネスライクに」「高級感を出したい」 「primary を変えたい」「角丸をもっと丸く」「フォントを変えたい」「テーマを提案して」 「テナントごとに配色を変えたい」「役割ごとに色を出し分けたい」（後者2つは複数 バリアント要望として範囲外に誘導するために発動） で発動。English: "change the vibe", "make it more playful", "make it more business-like", "adjust the theme", "change primary color".
The English summary is being prepared. Showing the original description.
Set up machine learning training and inference on NVIDIA DGX Spark. Get guidance on choosing containers and packages, resolving CUDA compatibility errors, and checking that PyTorch can access the GPU before starting a workload.
Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say "read from X in Spark" or "write DataFrame to Y" and there's no native connector, this skill applies.
The English summary is being prepared. Showing the original description.
Plan memory use and monitor long training jobs on NVIDIA DGX Spark. It guides troubleshooting for out-of-memory failures, tracks temperature and power alongside training logs, and helps assess competition between training and inference workloads.
Plan data ingestion and transformation workflows with Apache Airflow, Spark, and dbt. Get guidance on processing changed records, validating data quality, and rerunning failed or historical jobs without creating duplicate results.
Use when reading from or writing to Neo4j with Apache Spark or Databricks using the Neo4j Connector for Apache Spark 6.0 (org.neo4j.connectors:spark) or 5.x (org.neo4j:neo4j-connector-apache-spark). Covers SparkSession setup, DataFrame reads via labels/Cypher/relationship scan, DataFrame writes with SaveMode, node.keys for MERGE, relationship write mapping, partition and batch tuning, PySpark and Scala examples, Databricks cluster config, Databricks secrets for credentials, Delta Lake to Neo4j pipelines. Does NOT handle Cypher authoring — use neo4j-cypher-skill. Does NOT handle the Python bolt driver — use neo4j-driver-python-skill. Does NOT handle GDS algorithms — use neo4j-gds-skill.
The English summary is being prepared. Showing the original description.
Design systems that collect, transform, and store data. Choose between scheduled and continuous processing, generate workflow configurations, validate data quality, and analyze performance using guidance for tools including Python, SQL, Spark, Airflow, dbt, and Kafka.
Design AI agent analytics workflows across distributed tables and documents. Combine Knowledge Catalog for metadata, meaning information about data and its usage rules, with managed Spark processing to produce architecture diagrams, deployment guidance, and validation plans.
Build, publish, and manage apps on 妙搭（Spark/Miaoda） through local development or cloud AI sessions. Create page designs, manage data and permissions, inspect logs and usage, and configure automated triggers.
Configure and troubleshoot Apache Beam runners, the engines that execute data processing pipelines. Covers local testing, Dataflow, Flink, and Spark, with Java and Python examples, runner validation tests, and guidance on logs and execution settings.
Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access. Automatically invoke when the user asks to "run PySpark in Fabric", "create a Livy session", "execute Python on Fabric compute", "run Spark without a notebook", "submit code to Fabric", "ephemeral Spark execution", "run ETL in Fabric".
The English summary is being prepared. Showing the original description.
Apache Spark is a distributed engine for batch, SQL and streaming data processing, used from Python through PySpark. Use when a user asks to process big data, run distributed computations, build ETL pipelines, analyse data at scale, read Parquet from S3, or stream from Kafka with PySpark.
The English summary is being prepared. Showing the original description.
NVIDIA DGX Spark User Guide リファレンス。GB10 Grace Blackwell Superchip 仕様、 first-boot / UEFI、ConnectX-7 Networking (旧 Spark Stacking)、複数 Spark 接続、 NVIDIA Sync Cluster Assistant、DGX OS / NGC / container runtime、 Release Notes (DGX OS 7.5.0)、known issues、PXE / cloud-init、fleet 運用、 apt / fwupdmgr、recovery、PSIRT、 NVIDIA Developer Forums 由来 troubleshooting、 build.nvidia.com/spark playbook (vLLM / vLLM cluster / TensorRT-LLM / Ollama / ComfyUI / NIM / NeMo / VLM fine-tuning / NVIDIA PAIR / NCCL)、Tailscale、VS Code。
The English summary is being prepared. Showing the original description.
Design and implement systems that collect, transform, validate, and store data for analysis. Covers scheduled and real-time processing, data warehouses, quality checks, monitoring, and cost tuning using tools such as Apache Spark, dbt, and Airflow.
Research a contact in Common Room using an email, social handle, or name. Bring together available background, activity, website visits, and company context, then suggest conversation starters supported by actual engagement signals.
Manages Fabric Spark work, including notebook cell code with %%configure, %%sql, PySpark and notebookutils, named notebook runs, Livy sessions, triage of failed or OOM notebook and pipeline Spark runs, and the Materialized Lake View (MLV) lifecycle. Load it before writing MLV SQL, since CREATE MATERIALIZED LAKE VIEW and its CONSTRAINT clause are Fabric-only. KQL materialized views belong to eventhouse-cli.
The English summary is being prepared. Showing the original description.
数据工程 — 数据平台从业者的认知操作系统, 覆盖把数据从源系统搬运成可靠 / 可查询 / 可信赖形态供分析 / ML / 数据产品消费的全生命周期 (生成 → 摄取 → 存储 → 转换 → 服务 + 安全/数据管理/DataOps/数据架构/编排/软件工程 六条暗流, Reis & Housley 框架): 摄取与集成 (批 + CDC 变更数据捕获 Debezium + EL 工具 Fivetran/Airbyte/Meltano/dlt + Kafka Connect + schema drift) / 存储与文件表格式 (对象存储数据湖 + 列存 Parquet/ORC/Arrow/Avro + 开放表格式 Apache Iceberg/Delta Lake/Apache Hudi + lakehouse + 分区/compaction) / 转换与建模 (ELT dbt/SQLMesh + Spark + 维度建模 Kimball + Inmon + Data Vault + 大宽表 OBT + 渐变维 SCD + 增量模型 + 语义/指标层) / 编排与工作流 (Apache Airflow/Dagster/Prefect/Mage/Kestra/Apache DolphinScheduler + DAG + 幂等 + 回填 backfill + 数据资产调度) / 批流与实时 (Apache Kafka/Apache Flink/Spark Structured Streaming/Kinesis/Pulsar/Redpanda + Lambda vs Kappa + watermark/窗口/exactly-once + 流式 SQL Materialize/RisingWave + 实时 OLAP ClickHouse/Apache Druid/Apache Pinot/StarRocks/Apache Doris) / 数仓与查询引擎 (Snowflake/BigQuery/Redshift/Databricks SQL/Trino/Presto/DuckDB/Polars + 存算分离 + MPP) / 数据质量测试与可观测性 (dbt tests/Great Expectations/Soda + 数据契约 + Monte Carlo data downtime + 新鲜度/量/schema 异常检测) / 数据治理编目与血缘 (DataHub/Amundsen/OpenMetadata/Unity Catalog + 列级血缘 + PII 分类 + 访问控制 + GDPR) / DataOps 与可靠性 (数据 CI/CD + 转换版本控制 + 环境隔离 + 幂等重处理 + 数据 SLA/SLO + 计算存储 FinOps) / 数据架构范式 (现代数据栈 + lakehouse + data mesh + data fabric + 去中心化 vs 中心化所有权) / 分析工程角色 (dbt 时代连接数据工程与分析的桥) — 不含 数据科学/ML 建模本身 (是下游消费者) / BI 仪表盘制作 (serving 下游) / 数据分析报表为终点 / 'data engineer = 跑 Hadoop 的' 过时窄化 / 通用后端应用开发 (平行学科) (Data Engineering — the cognitive operating system of practitioners who design, build, and operate the data platform: moving data from source systems into reliable, queryable, trustworthy form for analytics / ML / products, covering (a) the data engineering lifecycle (generation → ingestion → storage → transformation → serving, with the undercurrents security / data management / DataOps / data architecture / orchestration / software engineering — Reis & Housley framing), (b) ingestion & integration (batch + CDC change-data-capture with Debezium, EL tools Fivetran / Airbyte / Meltano / dlt, Kafka Connect, API + file + database sources, schema drift handling), (c) storage & file/table formats (object storage data lakes, columnar formats Parquet / ORC / Arrow / Avro, open table formats Apache Iceberg / Delta Lake / Apache Hudi, lakehouse architecture, partitioning / compaction / Z-ordering), (d) transformation & modeling (ELT with dbt / SQLMesh, Spark, dimensional modeling Kimball, Inmon CIF, Data Vault, One Big Table / wide tables, normalization vs denormalization, slowly changing dimensions, incremental models, the semantic / metrics layer), (e) orchestration & workflow (Apache Airflow, Dagster, Prefect, Mage, Kestra, Apache DolphinScheduler, DAGs, idempotency, backfills, data-aware / asset-based scheduling), (f) batch vs streaming & real-time (Apache Kafka, Apache Flink, Spark Structured Streaming, Kinesis / Pulsar / Redpanda, the Lambda vs Kappa debate, watermarks / windowing / exactly-once, streaming SQL Materialize / RisingWave, real-time OLAP ClickHouse / Apache Druid / Apache Pinot / StarRocks / Apache Doris), (g) warehouses & query engines (Snowflake, BigQuery, Redshift, Databricks SQL, Trino / Presto, DuckDB, Polars, decoupled storage & compute, MPP), (h) data quality, testing & observability (dbt tests, Great Expectations, Soda, data contracts, Monte Carlo / data downtime, freshness / volume / schema anomaly detection, unit / integration testing of pipelines), (i) data governance, catalog & lineage (DataHub, Amundsen, OpenMetadata, Unity Catalog, column-level lineage, PII / data classification, access control, GDPR / data privacy), (j) DataOps & reliability (CI/CD for data, version control of transformations, environments, idempotent reprocessing, SLAs / SLOs for data, cost / FinOps for compute & storage), (k) data architecture paradigms (modern data stack, data lakehouse, data mesh, data fabric, decentralized vs centralized ownership), (l) the analytics engineering role (the dbt-era bridge between data engineering and analysis); N
The English summary is being prepared. Showing the original description.