postmortem-writing

Day 0: Incident occurs
Day 1-2: Draft postmortem document
Day 3-5: Postmortem meeting
Day 5-7: Finalize document, create tickets
Week 2+: Action item completion
Quarterly: Review patterns across incidents

 
### Template 2: 5 Whys Analysis
 
```markdown
# 5 Whys Analysis: [Incident]
 
## Problem Statement
 
Payment service experienced 47-minute outage due to database connection exhaustion.
 
## Analysis
 
### Why #1: Why did the service fail?
 
**Answer**: Database connections were exhausted, causing all new requests to fail.
 
**Evidence**: Metrics showed connection count at 100/100 (max), with 500+ pending requests.
 
---
 
### Why #2: Why were database connections exhausted?
 
**Answer**: Each incoming request opened a new database connection instead of using the connection pool.
 
**Evidence**: Code diff shows direct `DriverManager.getConnection()` instead of pooled `DataSource`.
 
---
 
### Why #3: Why did the code bypass the connection pool?
 
**Answer**: A developer refactored the repository class and inadvertently changed the connection acquisition method.
 
**Evidence**: PR #1234 shows the change, made while fixing a different bug.
 
---
 
### Why #4: Why wasn't this caught in code review?
 
**Answer**: The reviewer focused on the functional change (the bug fix) and didn't notice the infrastructure change.
 
**Evidence**: Review comments only discuss business logic.
 
---
 
### Why #5: Why isn't there a safety net for this type of change?
 
**Answer**: We lack automated tests that verify connection pool behavior and lack documentation about our connection patterns.
 
**Evidence**: Test suite has no tests for connection handling; wiki has no article on database connections.
 
## Root Causes Identified
 
1. **Primary**: Missing automated tests for infrastructure behavior
2. **Secondary**: Insufficient documentation of architectural patterns
3. **Tertiary**: Code review checklist doesn't include infrastructure considerations
 
## Systemic Improvements
 
| Root Cause    | Improvement                       | Type       |
| ------------- | --------------------------------- | ---------- |
| Missing tests | Add infrastructure behavior tests | Prevention |
| Missing docs  | Document connection patterns      | Prevention |
| Review gaps   | Update review checklist           | Detection  |
| No canary     | Implement canary deployments      | Mitigation |

# Quick Postmortem: [Brief Title]
 
**Date**: 2024-01-15 | **Duration**: 12 min | **Severity**: SEV3
 
## What Happened
 
API latency spiked to 5s due to cache miss storm after cache flush.
 
## Timeline
 
- 10:00 - Cache flush initiated for config update
- 10:02 - Latency alerts fire
- 10:05 - Identified as cache miss storm
- 10:08 - Enabled cache warming
- 10:12 - Latency normalized
 
## Root Cause
 
Full cache flush for minor config update caused thundering herd.
 
## Fix
 
- Immediate: Enabled cache warming
- Long-term: Implement partial cache invalidation (ENG-999)
 
## Lessons
 
Don't full-flush cache in production; use targeted invalidation.

## Meeting Structure (60 minutes)
 
### 1. Opening (5 min)
 
- Remind everyone of blameless culture
- "We're here to learn, not to blame"
- Review meeting norms
 
### 2. Timeline Review (15 min)
 
- Walk through events chronologically
- Ask clarifying questions
- Identify gaps in timeline
 
### 3. Analysis Discussion (20 min)
 
- What failed?
- Why did it fail?
- What conditions allowed this?
- What would have prevented it?
 
### 4. Action Items (15 min)
 
- Brainstorm improvements
- Prioritize by impact and effort
- Assign owners and due dates
 
### 5. Closing (5 min)
 
- Summarize key learnings
- Confirm action item owners
- Schedule follow-up if needed
 
## Facilitation Tips
 
- Keep discussion on track
- Redirect blame to systems
- Encourage quiet participants
- Document dissenting views
- Time-box tangents