Improve slow Apache Spark jobs by reviewing data partitions, caching, memory settings, and shuffles, which move data between workers. Use the guidance to investigate bottlenecks and tune pipelines for larger datasets.
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
Review the instructions the agent will receive before you install.
Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
Driver Program
↓
Job (triggered by action)
↓
Stages (separated by shuffles)
↓
Tasks (one per partition)
| Factor | Impact | Solution |
|---|---|---|
| Shuffle | Network I/O, disk I/O | Minimize wide transformations |
| Data Skew | Uneven task duration | Salting, broadcast joins |
| Serialization | CPU overhead | Use Kryo, columnar formats |
| Memory | GC pressure, spills | Tune executor memory |
| Partitions | Parallelism | Right-size partitions |
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
# Create optimized Spark session
spark = (SparkSession.builder
.appName("OptimizedJob")
.config("spark.sql.adaptive.enabled", "true")
.config("spark.sql.adaptive.coalescePartitions.enabled", "true")
.config("spark.sql.adaptive.skewJoin.enabled", "true")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config("spark.sql.shuffle.partitions", "200")
.getOrCreate())
# Read with optimized settings
df = (spark.read
.format("parquet")
.option("mergeSchema", "false")
.load("s3://bucket/data/"))
# Efficient transformations
result = (df
.filter(F.col("date") >= "2024-01-01")
.select("id", "amount", "category")
.groupBy("category")
.agg(F.sum("amount").alias("total")))
result.write.mode("overwrite").parquet("s3://bucket/output/")
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
.count() for existence - Use .take(1) or .isEmpty()No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Build and review interfaces against WCAG 2.2 accessibility guidelines. Covers keyboard navigation, screen readers that read interface content aloud, accessible forms, mobile support, and display preferences, helping identify and fix barriers to using a website or app.
Review AI-generated code for problems hidden behind successful runs. Check failure handling, resource cleanup, unusual inputs, nonexistent dependencies, and departures from project conventions to identify missing safeguards before merging or accepting the work as complete.
Build Apache Airflow workflows for data pipelines and scheduled jobs. The skill guides task dependencies, retries, waiting for conditions, testing, and deployment, with patterns for tasks that can be safely rerun.
Plan and carry out an AngularJS to Angular migration, from choosing a rewrite strategy to converting components, services, and routes. Includes a hybrid approach that runs both frameworks together while individual features are migrated.
Identify software protections that hinder inspection, such as debugger detection and obfuscation that makes code harder to understand. Support authorized binary analysis and CTF security challenges with protection findings, bypass strategies, reports, and analysis scripts.
Design and review REST and GraphQL APIs with consistent naming, data structures, error responses, pagination for large result sets, and versioning strategies. Helps teams define shared standards and plan changes that existing clients can adopt gradually.