Improve slow Apache Spark jobs by reviewing data partitions, caching, memory settings, and shuffles, which move data between workers. Use the guidance to investigate bottlenecks and tune pipelines for larger datasets.
中文简介正在准备中。当前显示英语简介。
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
安装前可以查看交给 AI 代理的指令内容。
Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
Driver Program
↓
Job (triggered by action)
↓
Stages (separated by shuffles)
↓
Tasks (one per partition)
| Factor | Impact | Solution |
|---|---|---|
| Shuffle | Network I/O, disk I/O | Minimize wide transformations |
| Data Skew | Uneven task duration | Salting, broadcast joins |
| Serialization | CPU overhead | Use Kryo, columnar formats |
| Memory | GC pressure, spills | Tune executor memory |
| Partitions | Parallelism | Right-size partitions |
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
# Create optimized Spark session
spark = (SparkSession.builder
.appName("OptimizedJob")
.config("spark.sql.adaptive.enabled", "true")
.config("spark.sql.adaptive.coalescePartitions.enabled", "true")
.config("spark.sql.adaptive.skewJoin.enabled", "true")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config("spark.sql.shuffle.partitions", "200")
.getOrCreate())
# Read with optimized settings
df = (spark.read
.format("parquet")
.option("mergeSchema", "false")
.load("s3://bucket/data/"))
# Efficient transformations
result = (df
.filter(F.col("date") >= "2024-01-01")
.select("id", "amount", "category")
.groupBy("category")
.agg(F.sum("amount").alias("total")))
result.write.mode("overwrite").parquet("s3://bucket/output/")
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
.count() for existence - Use .take(1) or .isEmpty()还没有评价。欢迎分享你的使用感受。
无需登录即可发表。欢迎分享使用感受、提问，或说明在哪些环境下无法运行。
还没有评论。来写第一条吧。
Build and review interfaces against WCAG 2.2 accessibility guidelines. Covers keyboard navigation, screen readers that read interface content aloud, accessible forms, mobile support, and display preferences, helping identify and fix barriers to using a website or app.
中文简介正在准备中。当前显示英语简介。
Review AI-generated code for problems hidden behind successful runs. Check failure handling, resource cleanup, unusual inputs, nonexistent dependencies, and departures from project conventions to identify missing safeguards before merging or accepting the work as complete.
中文简介正在准备中。当前显示英语简介。
Build Apache Airflow workflows for data pipelines and scheduled jobs. The skill guides task dependencies, retries, waiting for conditions, testing, and deployment, with patterns for tasks that can be safely rerun.
中文简介正在准备中。当前显示英语简介。
Plan and carry out an AngularJS to Angular migration, from choosing a rewrite strategy to converting components, services, and routes. Includes a hybrid approach that runs both frameworks together while individual features are migrated.
中文简介正在准备中。当前显示英语简介。
Identify software protections that hinder inspection, such as debugger detection and obfuscation that makes code harder to understand. Support authorized binary analysis and CTF security challenges with protection findings, bypass strategies, reports, and analysis scripts.
中文简介正在准备中。当前显示英语简介。
Design and review REST and GraphQL APIs with consistent naming, data structures, error responses, pagination for large result sets, and versioning strategies. Helps teams define shared standards and plan changes that existing clients can adopt gradually.
中文简介正在准备中。当前显示英语简介。