Specific optimization strategies for Python scripts working with larger-than-memory datasets via Dask.
The English summary is being prepared. Showing the original description.
See how to installReview the instructions the agent will receive before you install.
dask.compute() only once at the end of a pipeline — multiple intermediate compute() calls break the lazy evaluation graph and eliminate Dask's ability to fuse and parallelize operations.df.apply(lambda ...) with Dask DataFrames for element-wise operations — Pandas-style apply forces row-by-row Python execution that bypasses Dask's vectorized C extensions and is slower than single-threaded Pandas.blocksize= for CSV, chunksize= for Parquet) — auto-detected partition sizes frequently produce thousands of tiny partitions (slow scheduler overhead) or a single giant partition (no parallelism).len(df) or df.shape on a Dask DataFrame without wrapping in compute() — these trigger immediate full dataset computation and negate lazy evaluation.dask.distributed.Client for multi-machine or CPU-bound workloads — the default threaded scheduler serializes Python-heavy operations due to the GIL; the distributed scheduler bypasses this.| Anti-Pattern | Why It Fails | Correct Approach |
|---|---|---|
Multiple compute() calls in pipeline | Breaks lazy graph; forces data to materialize and re-partition at each call | Build complete computation graph first; call compute() once at the end |
df.apply(lambda ...) on large DataFrames | Row-by-row Python; GIL contention; slower than equivalent Pandas on single core | Use vectorized Dask operations (map_partitions, assign, arithmetic operators) |
| Default blocksize on large CSV files | 128MB default creates thousands of partitions for 100GB files; scheduler overhead dominates | Set blocksize="256MB" or blocksize="1GB" for large files; profile optimal size |
len(df) without compute() | Triggers full dataset read and count; defeats lazy evaluation | Use df.shape[0].compute() explicitly; only compute when size is truly needed |
| Threaded scheduler for CPU-bound work | Python GIL serializes CPU computation across threads; no true parallelism | Use dask.distributed.LocalCluster() or process-based scheduler for CPU tasks |
Before starting:
cat .claude/context/memory/learnings.md
After completing: Record any new patterns or exceptions discovered.
ASSUME INTERRUPTION: Your context may reset. If it's not in memory, it didn't happen.
No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Ensure accessibility in UI components including semantic HTML, ARIA attributes, keyboard navigation, and WCAG 2.2 AA compliance.
The English summary is being prepared. Showing the original description.
Use when you want to improve response quality through meta-cognitive reasoning. Applies 15+ reasoning methods to reconsider and refine initial outputs.
The English summary is being prepared. Showing the original description.
N-round opposing-stance debates for trade-off analysis. Assigns pro/con roles to agents, runs structured debate rounds with quality scoring, and produces a moderator synthesis with confidence-rated recommendation. Generalizable to architecture, technology, security, and design decisions.
The English summary is being prepared. Showing the original description.
Force adversarial code review stance that eliminates confirmation bias — reviewer must find issues or re-analyze
The English summary is being prepared. Showing the original description.
Creates specialized AI agents on-demand when no existing agent matches a request. Use when the Router cannot find a suitable agent for a task. Enables self-evolution by generating persistent agents.
The English summary is being prepared. Showing the original description.
LLM-as-judge evaluation framework with 5-dimension rubric (accuracy, groundedness, coherence, completeness, helpfulness) for scoring AI-generated content quality with weighted composite scores and evidence citations
The English summary is being prepared. Showing the original description.