Improve slow Apache Spark jobs by reviewing data partitions, caching, memory settings, and shuffles, which move data between workers. Use the guidance to investigate bottlenecks and tune pipelines for larger datasets.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
설치하기 전에 에이전트에게 전달되는 지시 내용을 확인할 수 있습니다.
Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
Driver Program
↓
Job (triggered by action)
↓
Stages (separated by shuffles)
↓
Tasks (one per partition)
| Factor | Impact | Solution |
|---|---|---|
| Shuffle | Network I/O, disk I/O | Minimize wide transformations |
| Data Skew | Uneven task duration | Salting, broadcast joins |
| Serialization | CPU overhead | Use Kryo, columnar formats |
| Memory | GC pressure, spills | Tune executor memory |
| Partitions | Parallelism | Right-size partitions |
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
# Create optimized Spark session
spark = (SparkSession.builder
.appName("OptimizedJob")
.config("spark.sql.adaptive.enabled", "true")
.config("spark.sql.adaptive.coalescePartitions.enabled", "true")
.config("spark.sql.adaptive.skewJoin.enabled", "true")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config("spark.sql.shuffle.partitions", "200")
.getOrCreate())
# Read with optimized settings
df = (spark.read
.format("parquet")
.option("mergeSchema", "false")
.load("s3://bucket/data/"))
# Efficient transformations
result = (df
.filter(F.col("date") >= "2024-01-01")
.select("id", "amount", "category")
.groupBy("category")
.agg(F.sum("amount").alias("total")))
result.write.mode("overwrite").parquet("s3://bucket/output/")
Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.
.count() for existence - Use .take(1) or .isEmpty()아직 리뷰가 없습니다. 사용해 본 소감을 남겨 주세요.
로그인 없이 쓸 수 있습니다. 사용 후기, 질문, 작동하지 않은 환경 등을 편하게 남겨 주세요.
아직 댓글이 없습니다. 첫 댓글을 남겨 보세요.
Build and review interfaces against WCAG 2.2 accessibility guidelines. Covers keyboard navigation, screen readers that read interface content aloud, accessible forms, mobile support, and display preferences, helping identify and fix barriers to using a website or app.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Review AI-generated code for problems hidden behind successful runs. Check failure handling, resource cleanup, unusual inputs, nonexistent dependencies, and departures from project conventions to identify missing safeguards before merging or accepting the work as complete.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Build Apache Airflow workflows for data pipelines and scheduled jobs. The skill guides task dependencies, retries, waiting for conditions, testing, and deployment, with patterns for tasks that can be safely rerun.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Plan and carry out an AngularJS to Angular migration, from choosing a rewrite strategy to converting components, services, and routes. Includes a hybrid approach that runs both frameworks together while individual features are migrated.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Identify software protections that hinder inspection, such as debugger detection and obfuscation that makes code harder to understand. Support authorized binary analysis and CTF security challenges with protection findings, bypass strategies, reports, and analysis scripts.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Design and review REST and GraphQL APIs with consistent naming, data structures, error responses, pagination for large result sets, and versioning strategies. Helps teams define shared standards and plan changes that existing clients can adopt gradually.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.