Automatic Stateful Prompt Improver
DECISION POINTS
PROMPT ASSESSMENT:
├── Simple question/command (what, when, how)
│ └── Skip optimization → Answer directly
├── Complex task (multi-step, reasoning, technical)
│ ├── Token budget < 1000
│ │ └── APE: 3-5 iterations
│ ├── Token budget 1000-5000
│ │ └── OPRO: 5-10 iterations
│ └── Token budget > 5000
│ └── DSPy compilation: 10-20 iterations
└── Reusable template/system prompt
└── Full optimization with historical retrieval
OPTIMIZATION TECHNIQUE SELECTION:
├── Instruction rewriting needed
│ └── Use APE (Automatic Prompt Engineer)
├── Parameter tuning with constraints
│ └── Use OPRO (Optimization by PROmpting)
├── Complex pipeline with multiple modules
│ └── Use DSPy compilation patterns
└── Unknown/exploratory domain
└── Hybrid APE→OPRO→DSPy cascade
ITERATION CONTROL:
├── Improvement < 1% for 3 rounds → STOP
├── Quality score > 0.95 → STOP
├── Max iterations reached → STOP
├── User satisfaction confirmed → STOP
└── Continue → Next iteration
FEEDBACK INTEGRATION:
├── Task successful (user confirms/metrics good)
│ └── Record positive feedback + embed for retrieval
├── Task failed/poor quality
│ └── Record negative feedback + analyze failure mode
└── Unclear outcome
└── Ask user for explicit feedback before recording
FAILURE MODES
Over-Optimization Spiral
- SYMPTOM: Prompt grows to 500+ tokens with many nested constraints
- DIAGNOSIS: Chasing diminishing returns instead of stopping at "good enough"
- FIX: Apply 80/20 rule - if improvement drops below 5% per iteration, stop
Template Obsession
- SYMPTOM: Spending iterations on formatting/structure vs. task clarity
- DIAGNOSIS: Confusing presentation with performance
- FIX: Measure actual task success, not template conformity
Historical Overfitting
- SYMPTOM: Optimized prompt works for past examples but fails on new inputs
- DIAGNOSIS: Training on too narrow a dataset from retrieval
- FIX: Include diverse examples in optimization, test on held-out cases
Capability Misjudgment
- SYMPTOM: Adding extensive scaffolding for tasks model handles natively
- DIAGNOSIS: Assuming model limitations without testing
- FIX: Test baseline capability before heavy prompting
Measurement Blindness
- SYMPTOM: Multiple iterations without clear success metrics
- DIAGNOSIS: Optimizing without knowing what "better" means
- FIX: Define measurable success criteria in first step
WORKED EXAMPLES
Example 1: Code Optimization Request
Original: "Make this code better"
def process_data(data):
results = []
for item in data:
if item > 0:
results.append(item * 2)
return results
Decision Point Navigation:
- Assessment: Complex task (requires code analysis) → Trigger optimization
- Technique Selection: Code review + improvement → APE with 5 iterations
- Retrieved Context: Similar code optimization prompts from history
- Optimized Prompt: "Analyze this Python function for performance, readability, and Pythonic patterns. Identify specific improvements: algorithmic complexity, memory usage, edge cases, and style. Provide refactored code with explanations."
What Novice Misses: Vague "make better" doesn't specify criteria
What Expert Catches: Need explicit dimensions (performance, style, edge cases)
Result: Clear analysis of list comprehension opportunity, edge case handling, type hints
Example 2: Reasoning Task Template
Original: "Help me think through this decision"
Decision Point Navigation:
- Assessment: Reusable template + reasoning task → Full optimization
- Historical Retrieval: Found decision framework prompts (0.87 similarity)
- Technique Selection: DSPy compilation (structured reasoning pipeline)
- Iteration Strategy: 10 rounds, measuring decision quality
Optimized Template:
Decision Analysis Framework:
1. SITUATION: State the decision clearly with constraints
2. STAKEHOLDERS: List affected parties and their interests
3. OPTIONS: Generate 3-5 distinct alternatives
4. CRITERIA: Define success metrics and weighting
5. TRADE-OFFS: Analyze each option against criteria
6. RECOMMENDATION: Select best option with confidence level
Quality Gates Applied: Template completeness, reusability score, user satisfaction
Example 3: Ambiguous Technical Request
Original: "Set up monitoring"
Decision Point Navigation:
- Assessment: Underspecified + technical → Trigger optimization
- Clarification Strategy: OPRO with constraint elicitation
- Domain Context: Retrieved monitoring setup patterns
Optimized Prompt:
"Design monitoring setup by specifying: (1) Infrastructure scope (servers, containers, applications), (2) Key metrics (performance, availability, business), (3) Alert thresholds and escalation, (4) Technology stack constraints, (5) Budget/complexity limits. Provide implementation roadmap with priorities."
Before/After Trade-offs:
- Before: Endless back-and-forth clarification
- After: Structured requirements gathering in single exchange
- Cost: Longer initial prompt
- Benefit: Complete specification in one round
QUALITY GATES
Pre-execution checklist before calling optimize_prompt:
Post-optimization validation:
Quality scoring rubric (0-100):
- Clarity: Can naive user understand requirements? (25 pts)
- Specificity: Concrete vs. abstract instructions? (25 pts)
- Completeness: Covers edge cases and constraints? (25 pts)
- Efficiency: Achieves goals without bloat? (25 pts)
NOT-FOR BOUNDARIES
Do NOT use this skill for:
- Simple factual questions → Answer directly with knowledge
- File operations without reasoning → Use file-management skill
- Direct command execution → Execute immediately
- Conversational responses → Respond naturally
- Already optimized prompts → Check history first to avoid re-optimization
- User explicitly says "don't optimize" → Respect user preference
Delegate instead:
- For mathematical problems → Use calculation-focused skills
- For creative writing → Use creative-writing skill (unless template creation)
- For data analysis → Use data-analysis skill (unless complex reasoning required)
- For debugging → Use debugging skill (unless systematic improvement needed)
Gray areas requiring judgment:
- Medium complexity tasks (score 2-4) → Test baseline performance first
- Domain expertise requests → Optimize only if reusable template potential
- Follow-up questions → Optimize if expanding scope significantly