Incident Response Patterns

Severity Classification

Level	Tanım	Response Time	Escalation
SEV-1	Tam kesinti, data loss	5 dk	Tüm ekip
SEV-2	Major feature bozuk	30 dk	On-call + lead
SEV-3	Minor feature bozuk	4 saat	On-call
SEV-4	Kozmetik, low impact	Sonraki iş günü	Ticket

Incident Response Protocol

1. DETECT  → Alert tetiklendi / kullanıcı bildirdi
2. TRIAGE  → Severity belirle, incident channel aç
3. ASSIGN  → Incident Commander + Responder ata
4. CONTAIN → Bleeding'i durdur (rollback, feature flag off)
5. FIX     → Root cause'u düzelt
6. VERIFY  → Fix'i doğrula (monitoring, smoke test)
7. RESOLVE → Incident kapat, stakeholder bilgilendir
8. REVIEW  → Post-incident review (48 saat içinde)

Runbook Template

# Runbook: [Service Name] - [Issue Type]

## Symptoms
- Alert: [alert name]
- Dashboard: [link]
- Expected: [normal behavior]
- Actual: [observed behavior]

## Diagnosis Steps
1. Check [metric/log/dashboard]
2. Run: `kubectl get pods -n production`
3. Check: `SELECT count(*) FROM errors WHERE ...`

## Mitigation
1. Quick fix: [rollback/restart/scale]
2. Feature flag: [disable feature X]
3. Redirect: [failover to backup]

## Escalation
- L1: [on-call engineer]
- L2: [team lead]
- L3: [CTO/VP Eng]

Root Cause Analysis

5 Whys

Problem: API 500 hatası
Why 1: Database connection timeout
Why 2: Connection pool tükendi
Why 3: Slow query connection'ları tutuyor
Why 4: Missing index on frequently queried column
Why 5: Schema change review process eksik
→ Action: Schema change PR'da EXPLAIN ANALYZE zorunlu

Post-Incident Review Template

## Incident Summary
- Duration: [start - end]
- Impact: [affected users/revenue]
- Severity: SEV-[X]

## Timeline
- HH:MM - Alert triggered
- HH:MM - Incident declared
- HH:MM - Root cause identified
- HH:MM - Fix deployed
- HH:MM - Incident resolved

## Root Cause
[Detailed technical explanation]

## Action Items
- [ ] [Fix] Add missing index (owner: @dev, due: date)
- [ ] [Prevent] Add schema review step (owner: @lead)
- [ ] [Detect] Add connection pool alert (owner: @sre)

Checklist

• Severity matrix tanımlı
• On-call rotation aktif
• Runbook'lar güncel
• Incident channel template hazır
• Post-incident review 48 saat içinde
• Action items tracked (Jira/Linear)
• Blameless culture enforced
• Alerting threshold'lar doğru

Anti-Patterns

• Blame culture (kişi değil, sistem düzelt)
• Runbook'suz servis
• Post-incident review atlamak
• Alert fatigue (çok fazla false positive)
• Hotfix without rollback plan