14 个 · 按相关度排序
Audit one datafusion-spark function against Apache Spark 4.2.0. Compare source code, accepted types, and edge cases, verify expected results with PySpark, then fix differences and record remaining gaps as issues and disabled tests.
中文简介正在准备中。当前显示英语简介。
Build Apache Spark pipelines for large datasets, including transformations, aggregations, and streaming analysis. Get PySpark or Scala code, partitioning strategies for dividing data across workers, and configuration advice to investigate slow jobs and resource usage.
中文简介正在准备中。当前显示英语简介。
Apache Spark is a distributed engine for batch, SQL and streaming data processing, used from Python through PySpark. Use when a user asks to process big data, run distributed computations, build ETL pipelines, analyse data at scale, read Parquet from S3, or stream from Kafka with PySpark.
中文简介正在准备中。当前显示原文说明。
Improve slow Apache Spark jobs by reviewing data partitions, caching, memory settings, and shuffles, which move data between workers. Use the guidance to investigate bottlenecks and tune pipelines for larger datasets.
中文简介正在准备中。当前显示英语简介。
Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access. Automatically invoke when the user asks to "run PySpark in Fabric", "create a Livy session", "execute Python on Fabric compute", "run Spark without a notebook", "submit code to Fabric", "ephemeral Spark execution", "run ETL in Fabric".
中文简介正在准备中。当前显示原文说明。
Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say "read from X in Spark" or "write DataFrame to Y" and there's no native connector, this skill applies.
中文简介正在准备中。当前显示原文说明。
Plan and explain Lakehouse solutions in Microsoft Fabric, a place to manage tables and files together. Understand data organization, links to data without copying, access controls, and table maintenance options for designing and improving a solution.
中文简介正在准备中。当前显示英语简介。
Manages Fabric Spark work, including notebook cell code with %%configure, %%sql, PySpark and notebookutils, named notebook runs, Livy sessions, triage of failed or OOM notebook and pipeline Spark runs, and the Materialized Lake View (MLV) lifecycle. Load it before writing MLV SQL, since CREATE MATERIALIZED LAKE VIEW and its CONSTRAINT clause are Fabric-only. KQL materialized views belong to eventhouse-cli.
中文简介正在准备中。当前显示原文说明。
Plans and builds end-to-end Fabric data platforms on the Bronze, Silver, and Gold medallion pattern: ingestion to analytics, PySpark and Delta layers, pipeline orchestration, per-layer Spark tuning, Materialized Lake View versus notebook choices for Silver and Gold, and Direct Lake handoff. Use for layered architecture design, planning and recommendations. For questions over an existing Power BI report use fabriciq.
中文简介正在准备中。当前显示原文说明。
Transform raw data into analytical assets using ETL/ELT patterns, SQL (dbt), Python (pandas/polars/PySpark), and orchestration (Airflow). Use when building data pipelines, implementing incremental models, migrating from pandas to polars, or orchestrating multi-step transformations with testing and quality checks.
中文简介正在准备中。当前显示原文说明。
byted-emr-skills提供管理火山引擎EMR（火山引擎 E-MapReduce（简称“EMR”）是开源Hadoop生态的企业级大数据分析系统，完全兼容开源）的技能，包括管理EMR on ECS集群、EMR on VKE集群、EMR serverless队列、计算组、作业模板/实例、日志、监控并提供 EMR Agent 智能诊断与知识问答能力。当用户提及“EMR on ECS集群”、“EMR on VKE集群”、“Serverless 队列”、“Serverless 作业”、“SparkSQL/PrestoSQL/Ray/PySpark/SparkJar 作业”、“作业日志”、“作业监控”、“作业诊断”等需求时，应优先使用此技能。
中文简介正在准备中。当前显示原文说明。
Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).
中文简介正在准备中。当前显示原文说明。
Use when reading from or writing to Neo4j with Apache Spark or Databricks using the Neo4j Connector for Apache Spark 6.0 (org.neo4j.connectors:spark) or 5.x (org.neo4j:neo4j-connector-apache-spark). Covers SparkSession setup, DataFrame reads via labels/Cypher/relationship scan, DataFrame writes with SaveMode, node.keys for MERGE, relationship write mapping, partition and batch tuning, PySpark and Scala examples, Databricks cluster config, Databricks secrets for credentials, Delta Lake to Neo4j pipelines. Does NOT handle Cypher authoring — use neo4j-cypher-skill. Does NOT handle the Python bolt driver — use neo4j-driver-python-skill. Does NOT handle GDS algorithms — use neo4j-gds-skill.
中文简介正在准备中。当前显示原文说明。
Apache Spark 4.0.2 best practices for PySpark and Scala distributed data processing
中文简介正在准备中。当前显示原文说明。