skillstore-cli 1.0.0

package/data/shared/framework/skills/databases/references/troubleshooting.md

@@ -0,0 +1,214 @@
+# Database Troubleshooting Guide
+
+## Slow Query Diagnosis Workflow
+
+**Step 1: Identify slow queries**
+```sql
+-- PostgreSQL: find queries exceeding threshold
+SELECT pid, now() - query_start AS duration, query, state
+FROM pg_stat_activity
+WHERE state = 'active' AND now() - query_start > interval '1 second'
+ORDER BY duration DESC;
+
+-- Enable slow query logging
+ALTER SYSTEM SET log_min_duration_statement = '500';  -- log queries > 500ms
+SELECT pg_reload_conf();
+```
+
+**Step 2: Analyze with EXPLAIN**
+```sql
+EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT) SELECT ...;
+-- Look for: Seq Scan (missing index), Nested Loop on large tables, high "actual rows" vs "planned rows"
+```
+
+**Step 3: Fix by priority**
+1. **Add missing index** — If Seq Scan on a filtered/joined column: `CREATE INDEX CONCURRENTLY idx_name ON table(column);`
+2. **Rewrite query** — Replace correlated subqueries with JOINs, use CTEs for readability but be aware of optimization fences (pre-PG12).
+3. **Update statistics** — If planner estimates are wildly off: `ANALYZE table_name;`
+4. **Partial index** — If only a subset of rows is queried: `CREATE INDEX idx ON orders(status) WHERE status = 'pending';`
+
+## Lock Contention Diagnosis
+
+**Identify blocking locks:**
+```sql
+-- Active locks with blocking info
+SELECT
+  blocked.pid AS blocked_pid,
+  blocked.query AS blocked_query,
+  blocking.pid AS blocking_pid,
+  blocking.query AS blocking_query,
+  blocked.wait_event_type,
+  now() - blocked.query_start AS wait_duration
+FROM pg_stat_activity blocked
+JOIN pg_locks bl ON blocked.pid = bl.pid AND NOT bl.granted
+JOIN pg_locks kl ON bl.locktype = kl.locktype
+  AND bl.database IS NOT DISTINCT FROM kl.database
+  AND bl.relation IS NOT DISTINCT FROM kl.relation
+  AND bl.pid != kl.pid AND kl.granted
+JOIN pg_stat_activity blocking ON kl.pid = blocking.pid;
+
+-- Kill a blocking query if needed
+SELECT pg_terminate_backend(<blocking_pid>);
+```
+
+**Common lock contention scenarios:**
+1. **Long-running `ALTER TABLE`** — Acquires `ACCESS EXCLUSIVE` lock, blocks all reads/writes.
+   - Fix: Use `CREATE INDEX CONCURRENTLY`, `ALTER TABLE ... ADD COLUMN` (non-blocking for nullable columns without defaults in PG11+).
+2. **Bulk UPDATE without batching** — Locks thousands of rows in one transaction.
+   - Fix: Batch updates with `LIMIT` and loop.
+3. **`LOCK TABLE` left in application code** — Forgotten explicit lock from debugging.
+
+## Replication Lag Monitoring and Mitigation
+
+**Monitor lag:**
+```sql
+-- On primary: check replication slots and lag
+SELECT slot_name, active,
+       pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn) AS lag_bytes
+FROM pg_replication_slots;
+
+-- On replica: check lag in seconds
+SELECT now() - pg_last_xact_replay_timestamp() AS replication_lag;
+```
+
+**Common causes and fixes:**
+1. **Replica under-provisioned** — Replica has fewer resources than primary.
+   - Fix: Match replica resources (CPU, IOPS, memory) to primary.
+2. **Long-running queries on replica** — Conflict with WAL replay.
+   - Fix: Set `hot_standby_feedback = on` or increase `max_standby_streaming_delay`.
+3. **Large transactions** — Bulk inserts/updates generate massive WAL.
+   - Fix: Break into smaller batches.
+4. **Network bottleneck** — Insufficient bandwidth between primary and replica.
+   - Fix: Check network throughput, consider `wal_compression = on`.
+
+**Application-level mitigation:**
+- Read-after-write consistency: route reads to primary for N seconds after a write.
+- Use synchronous replication for critical reads (trade-off: write latency).
+
+## Connection Limit Reached
+
+**Symptoms:** `FATAL: too many connections for role "appuser"`, new connections refused.
+
+**Diagnosis:**
+```sql
+-- Current connection count by source
+SELECT usename, client_addr, state, count(*)
+FROM pg_stat_activity
+GROUP BY usename, client_addr, state
+ORDER BY count DESC;
+
+-- Check limits
+SHOW max_connections;
+SELECT rolname, rolconnlimit FROM pg_roles WHERE rolconnlimit > 0;
+```
+
+**Fix patterns:**
+1. **Connection pooler missing** — Each app instance opens its own pool.
+   - Fix: Deploy PgBouncer or Pgpool-II in front of PostgreSQL.
+2. **Idle connections from crashed workers** — Connections not closed on process exit.
+   - Fix: Set `idle_in_transaction_session_timeout = '30s'` in PostgreSQL config.
+3. **Too many microservices** — Each service has a pool of 10, 20 services = 200 connections.
+   - Fix: Centralize through PgBouncer in transaction mode.
+4. **`max_connections` too low** — Default is 100, may be insufficient.
+   - Fix: Increase, but prefer pooling first (each connection costs ~10MB RAM).
+
+## Migration Failure Rollback Procedures
+
+**Before running migrations (safety net):**
+```bash
+# Backup current schema
+pg_dump --schema-only -f schema_backup_$(date +%Y%m%d).sql dbname
+
+# For critical migrations, backup data too
+pg_dump -Fc -f full_backup_$(date +%Y%m%d).dump dbname
+```
+
+**When migration fails mid-way:**
+1. Check migration tool state — most tools track applied migrations in a table (e.g., `schema_migrations`, `_prisma_migrations`).
+2. If the tool supports rollback: `npx prisma migrate reset` (dev) or apply the down migration.
+3. If stuck in dirty state:
+   ```sql
+   -- Knex: fix migration lock
+   DELETE FROM knex_migrations_lock;
+   -- Prisma: mark failed migration as rolled back
+   UPDATE _prisma_migrations SET rolled_back_at = NOW() WHERE migration_name = 'failed_migration';
+   ```
+4. Manual rollback: apply the reverse DDL statements from the failed migration.
+5. Restore from backup as last resort: `pg_restore -d dbname full_backup.dump`.
+
+**Prevention:** Always test migrations against a copy of production data before applying.
+
+## Data Corruption Detection and Recovery
+
+**Detection signals:**
+- Queries return unexpected results or errors like `could not read block`.
+- `pg_catalog.pg_stat_database` shows `checksum_failures > 0` (if data checksums enabled).
+
+**Verification:**
+```bash
+# Check for corruption (requires downtime or read replica)
+pg_amcheck --all --heapallindexed dbname
+
+# Verify specific table
+SELECT ctid, * FROM table_name WHERE condition LIMIT 100;
+-- If errors occur on specific pages, note the ctid range
+```
+
+**Recovery steps:**
+1. **From replica** — If replica is clean, promote it or copy affected tables.
+2. **From backup** — Restore specific table: `pg_restore -d dbname -t table_name backup.dump`.
+3. **REINDEX** — If only indexes are corrupt: `REINDEX TABLE table_name;`
+4. **Enable checksums** for future detection: `pg_checksums --enable -D /data` (requires restart).
+
+## Bloated Tables and Indexes
+
+**Detection:**
+```sql
+-- Estimate table bloat
+SELECT schemaname, tablename,
+       pg_size_pretty(pg_total_relation_size(schemaname || '.' || tablename)) AS total_size,
+       n_dead_tup, n_live_tup,
+       round(n_dead_tup::numeric / NULLIF(n_live_tup, 0) * 100, 1) AS dead_pct
+FROM pg_stat_user_tables
+ORDER BY n_dead_tup DESC LIMIT 20;
+
+-- Check if autovacuum is running
+SELECT relname, last_vacuum, last_autovacuum, last_analyze
+FROM pg_stat_user_tables
+WHERE schemaname = 'public'
+ORDER BY n_dead_tup DESC;
+```
+
+**Fix patterns:**
+1. **Manual VACUUM** — `VACUUM (VERBOSE) table_name;` (reclaims space for reuse, no lock).
+2. **VACUUM FULL** — `VACUUM FULL table_name;` (reclaims disk space, but takes ACCESS EXCLUSIVE lock).
+3. **pg_repack** (zero-downtime alternative to VACUUM FULL):
+   ```bash
+   pg_repack -d dbname -t table_name --no-superuser-check
+   ```
+4. **Tune autovacuum** — If bloat recurs, autovacuum is too conservative:
+   ```sql
+   ALTER TABLE hot_table SET (autovacuum_vacuum_scale_factor = 0.01, autovacuum_vacuum_threshold = 1000);
+   ```
+
+## Deadlock Analysis from Logs
+
+**PostgreSQL log format for deadlocks:**
+```
+ERROR: deadlock detected
+DETAIL: Process 1234 waits for ShareLock on transaction 5678; blocked by process 9012.
+        Process 9012 waits for ShareLock on transaction 1234; blocked by process 1234.
+HINT: See server log for query details.
+```
+
+**Analysis workflow:**
+1. Enable detailed deadlock logging: `SET deadlock_timeout = '1s';` and `SET log_lock_waits = on;`
+2. Parse logs for `deadlock detected` entries.
+3. Map the processes to queries — the log shows the exact SQL of each participant.
+4. Identify the conflicting access pattern — usually two transactions updating the same rows in opposite order.
+
+**Resolution:**
+- Enforce consistent ordering: both transactions should update table A then table B (not A→B and B→A).
+- Use `SELECT ... FOR UPDATE` with explicit ordering: `ORDER BY id` to lock rows in deterministic order.
+- Reduce transaction scope — fewer rows locked means fewer collision opportunities.
+- Add application-level retry with exponential backoff for deadlock error code `40P01`.

package/data/shared/framework/skills/debugging-investigation/SKILL.md

@@ -0,0 +1,84 @@
+---
+name: debugging-investigation
+description: Systematic debugging and root cause investigation — log analysis, performance profiling, production incident diagnosis across all tech stacks
+---
+
+# Debugging & Investigation
+
+## Triggers
+
+Activate this skill when:
+- Investigating bug reports or unexpected behavior
+- Diagnosing test failures (unit, integration, e2e)
+- Responding to production incidents or outages
+- Analyzing performance degradation (slow responses, high latency)
+- Tracking down intermittent or flaky errors
+- Investigating memory leaks or resource exhaustion
+- Diagnosing data inconsistencies or corruption
+- Debugging race conditions or concurrency issues
+
+## Process
+
+### 1. Symptom Collection
+- Gather exact error messages, stack traces, and log output
+- Identify when the issue started (deploy, config change, traffic spike)
+- Determine scope: one user, one endpoint, one service, or system-wide
+- Check environment: local, staging, or production
+- Collect reproduction steps if available
+
+### 2. Hypothesis Formation
+Load: `references/systematic-debugging.md`
+
+- List 2-4 plausible root causes ranked by likelihood
+- For each hypothesis, define what evidence would confirm or refute it
+- Apply differential debugging: what changed recently? (git log, dependency updates, config changes)
+- Check for known bug patterns that match symptoms
+
+### 3. Evidence Gathering
+Load: `references/tool-specific-guides.md`
+
+- Use appropriate debugging tools for the stack (DevTools, profilers, log aggregators)
+- Collect quantitative data: metrics, timings, memory snapshots, query plans
+- Reproduce the issue in the most controlled environment possible
+- Narrow the problem space using binary search debugging (bisect code, data, config)
+- Document each finding with timestamps and exact values
+
+### 4. Root Cause Confirmation
+Load: `references/troubleshooting-patterns.md`
+
+- Confirm the root cause explains ALL observed symptoms, not just some
+- Verify by predicting a consequence of the root cause and testing for it
+- Rule out alternative hypotheses with evidence
+- Identify contributing factors (the root cause vs. the trigger)
+
+### 5. Fix & Verify
+- Implement the minimal fix that addresses the root cause
+- Write a regression test that reproduces the original symptom
+- Verify the fix in the same environment where the bug was observed
+- Check for side effects: run full test suite, monitor metrics post-deploy
+- Document the incident: timeline, root cause, fix, and prevention measures
+
+## Quick Reference — Symptom to Technique
+
+| Symptom | First Technique | Tool |
+|---|---|---|
+| Slow API response | EXPLAIN ANALYZE + flame chart | Database profiler, 0x/clinic.js |
+| Memory climbing over time | Heap snapshot diffing | Chrome DevTools Memory, clinic.js |
+| Intermittent test failure | Timing/ordering analysis | Retry with seed, race detector |
+| UI not rendering correctly | Component state inspection | React/Flutter DevTools |
+| 500 errors after deploy | Differential debugging | git bisect, deploy diff |
+| Network timeout | Waterfall analysis | Chrome Network panel, tcpdump |
+| Auth failures | Token/session inspection | jwt.io, browser storage, logs |
+| Data mismatch | Query + transformation trace | SQL logs, data pipeline logs |
+| High CPU usage | CPU profiling | perf, clinic flame, Activity Monitor |
+| Infinite loop / re-render | Execution tracing | Profiler, why-did-you-render |
+
+## References
+
+- [Systematic Debugging](references/systematic-debugging.md) — binary search, rubber duck, differential debugging, scientific method, cognitive biases
+- [Tool-Specific Guides](references/tool-specific-guides.md) — Chrome DevTools, Node.js, React DevTools, Flutter DevTools, database, VS Code debugger
+- [Troubleshooting Patterns](references/troubleshooting-patterns.md) — race conditions, memory leaks, deadlocks, N+1 queries, timezone bugs, encoding, floating point
+
+## Assets
+
+- [Sample Output](assets/sample-output.md) — complete debug report for a production Node.js memory leak investigation

package/data/shared/framework/skills/debugging-investigation/assets/sample-output.md

@@ -0,0 +1,314 @@
+# Debug Report: Production Memory Leak in Node.js Express API
+
+## Incident Summary
+
+| Field | Value |
+|---|---|
+| Service | order-service (Node.js 18, Express 4.18) |
+| Environment | Production (3 pods, Kubernetes) |
+| First Detected | 2026-03-17 14:30 UTC (Datadog alert) |
+| Resolved | 2026-03-18 09:15 UTC |
+| Duration | ~19 hours |
+| Impact | API p99 latency increased from 120ms to 2400ms; 3 pod restarts due to OOM |
+
+---
+
+## 1. Symptom Collection
+
+### Observed Symptoms
+- **Memory growth**: RSS climbing from ~200MB at pod start to 1.2GB over 48 hours before OOM kill
+- **Latency degradation**: p50 stable at 45ms, but p99 increased from 120ms to 2400ms (20x)
+- **GC pressure**: V8 garbage collection pauses increased from 5ms avg to 180ms avg
+- **Restart pattern**: Pods restarting every 36-48 hours (hitting the 1.5GB memory limit)
+
+### Timeline
+```
+Mar 15 10:00  Deploy v2.14.0 (includes WebSocket notification feature)
+Mar 15 10:30  All metrics normal
+Mar 16 08:00  First Datadog memory alert (pod-2 at 800MB)
+Mar 17 02:15  pod-2 OOM killed and restarted
+Mar 17 14:30  pod-1 and pod-3 both above 1GB, p99 latency alert fires
+Mar 17 15:00  Investigation begins
+```
+
+### Key Observation
+The leak started after deploying v2.14.0, which added a WebSocket notification feature for order status updates. Memory growth is steady (~20MB/hour) and correlates with request volume, not WebSocket connection count.
+
+---
+
+## 2. Hypothesis Formation
+
+### Hypothesis 1: Database Connection Pool Leak
+**Reasoning**: New feature adds additional database queries for notification preferences. Connections may not be returning to the pool.
+**If true**: `pg_stat_activity` would show growing active connections. Pool metrics would show exhaustion.
+**Likelihood**: Medium
+
+### Hypothesis 2: EventEmitter Listener Accumulation
+**Reasoning**: The new WebSocket feature registers event listeners to broadcast order status changes. If listeners are added per-request and never removed, they accumulate.
+**If true**: `process._getActiveHandles().length` would grow over time. Node.js would eventually emit a MaxListenersExceeded warning.
+**Likelihood**: High (most consistent with steady-per-request growth pattern)
+
+### Hypothesis 3: In-Memory Cache Without Eviction
+**Reasoning**: The notification feature caches user notification preferences to avoid repeated DB lookups. If the cache has no TTL or max size, it grows indefinitely.
+**If true**: Inspecting the cache object would show size growing proportionally to unique users served.
+**Likelihood**: Medium
+
+---
+
+## 3. Evidence Gathering
+
+### Testing Hypothesis 1: Database Connection Pool
+
+**Method**: Query `pg_stat_activity` and check pool metrics.
+
+```sql
+SELECT count(*), state FROM pg_stat_activity
+WHERE datname = 'orders_db'
+GROUP BY state;
+```
+
+**Result**:
+```
+ count | state
+-------+--------
+     8 | active
+     2 | idle
+```
+
+Pool size is configured as 10 connections. Active connections are stable at 8-10, cycling normally. No connection leak.
+
+**Verdict**: REFUTED. Connection pool is healthy.
+
+### Testing Hypothesis 2: EventEmitter Listener Accumulation
+
+**Method**: Add diagnostic endpoint to production (behind internal-only middleware):
+
+```javascript
+app.get('/_debug/listeners', internalOnly, (req, res) => {
+  const orderEmitter = require('./events/orderEvents').emitter;
+  res.json({
+    listenerCounts: {
+      statusChange: orderEmitter.listenerCount('statusChange'),
+      created: orderEmitter.listenerCount('created'),
+      cancelled: orderEmitter.listenerCount('cancelled'),
+    },
+    activeHandles: process._getActiveHandles().length,
+    activeRequests: process._getActiveRequests().length,
+    memoryUsage: process.memoryUsage(),
+  });
+});
+```
+
+**Result after 2 hours of running** (pod-3, restarted at 15:00):
+```json
+{
+  "listenerCounts": {
+    "statusChange": 14847,
+    "created": 1,
+    "cancelled": 1
+  },
+  "activeHandles": 14862,
+  "activeRequests": 3,
+  "memoryUsage": {
+    "rss": 487653376,
+    "heapUsed": 401283072,
+    "heapTotal": 460324864,
+    "external": 12419072
+  }
+}
+```
+
+14,847 listeners on `statusChange` after 2 hours. This matches approximately 2 hours of request volume (~120 requests/minute = 14,400 requests). Each request is adding a listener and never removing it.
+
+**Verdict**: CONFIRMED. Listener count grows 1:1 with requests.
+
+### Testing Hypothesis 3: Cache Without Eviction
+
+**Method**: Add cache size to diagnostic endpoint.
+
+```javascript
+const prefCache = require('./cache/notificationPreferences');
+// Added to /_debug/listeners response:
+cacheSize: prefCache.size,
+```
+
+**Result**:
+```json
+{
+  "cacheSize": 342
+}
+```
+
+Cache size is 342 entries (unique users). This is small and bounded by unique user count. Not the cause.
+
+**Verdict**: REFUTED. Cache is small and not growing significantly.
+
+### Heap Snapshot Comparison
+
+**Method**: Took two heap snapshots 30 minutes apart using `v8.writeHeapSnapshot()`.
+
+**Comparison results** (filtered to top allocations):
+```
+Constructor           Count Delta    Size Delta
+----------------------------------------------
+(closure)             +3,621         +14.2 MB
+EventEmitter          +3,621         +2.1 MB
+Object                +7,242         +4.8 MB
+Array                 +3,621         +1.4 MB
+```
+
+Closures and EventEmitter objects growing at the same rate — confirms listener accumulation.
+
+---
+
+## 4. Root Cause Confirmation
+
+### Root Cause
+In `/src/features/notifications/orderStatusBroadcast.js`, a listener is registered on `orderEmitter` inside the Express request handler:
+
+```javascript
+// BUG: This runs on EVERY request
+router.post('/orders/:id/status', async (req, res) => {
+  const { id } = req.params;
+  const { status } = req.body;
+
+  await updateOrderStatus(id, status);
+
+  // This adds a NEW listener every time a request comes in
+  orderEmitter.on('statusChange', (data) => {
+    if (data.orderId === id) {
+      broadcastToSubscribers(data);
+    }
+  });
+
+  orderEmitter.emit('statusChange', { orderId: id, status });
+  res.json({ success: true });
+});
+```
+
+Each request adds a new listener that captures `id` in its closure. The listener is never removed. After thousands of requests, the emitter has thousands of listeners — each holding a reference to its closure scope, preventing garbage collection.
+
+### Why All Symptoms Are Explained
+- **Memory growth ~20MB/hour**: Each listener + closure + captured scope ~1.5KB. At ~120 req/min = 7,200 req/hour = ~10.8MB/hour in listener objects alone, plus associated GC overhead and fragmentation.
+- **p99 latency increase**: When `statusChange` fires, ALL accumulated listeners execute. After 48h of uptime, one emit triggers 300,000+ listener callbacks.
+- **p50 stable**: Most requests complete before the emit, or the synchronous emit overhead is small at lower accumulation levels.
+- **Correlates with request volume**: One listener added per request, independent of WebSocket connections.
+
+---
+
+## 5. Fix
+
+### Code Change
+
+```javascript
+// FIXED: Register the broadcast listener ONCE at module level
+const orderEmitter = require('./events/orderEvents').emitter;
+const { broadcastToSubscribers } = require('./websocket/broadcast');
+
+// Single listener registered at startup
+orderEmitter.on('statusChange', (data) => {
+  broadcastToSubscribers(data);
+});
+
+router.post('/orders/:id/status', async (req, res) => {
+  const { id } = req.params;
+  const { status } = req.body;
+
+  await updateOrderStatus(id, status);
+  orderEmitter.emit('statusChange', { orderId: id, status });
+
+  res.json({ success: true });
+});
+```
+
+### Changes Made
+1. Moved `orderEmitter.on('statusChange', ...)` out of the request handler to module scope (runs once at startup)
+2. Removed per-request `orderId` filtering from the listener — `broadcastToSubscribers` already handles routing to the correct WebSocket clients based on their subscriptions
+3. Added a regression test that verifies listener count stays constant after N requests
+
+### Regression Test
+```javascript
+describe('orderStatusBroadcast', () => {
+  it('should not accumulate listeners across requests', async () => {
+    const orderEmitter = require('./events/orderEvents').emitter;
+    const initialCount = orderEmitter.listenerCount('statusChange');
+
+    // Simulate 100 requests
+    for (let i = 0; i < 100; i++) {
+      await request(app)
+        .post(`/orders/order-${i}/status`)
+        .send({ status: 'shipped' })
+        .expect(200);
+    }
+
+    const finalCount = orderEmitter.listenerCount('statusChange');
+    expect(finalCount).toBe(initialCount);
+  });
+});
+```
+
+---
+
+## 6. Verification
+
+### Staging Verification
+- Deployed fix to staging environment
+- Ran 4-hour load test with production-equivalent traffic (120 req/min)
+- Memory stable at 230-250MB throughout (no growth trend)
+- Listener count stable at 3 (startup listeners only)
+- p99 latency stable at 95-130ms throughout
+
+### Production Verification (72-hour observation)
+```
+Time Since Deploy    RSS Memory    statusChange Listeners    p99 Latency
+---------------------------------------------------------------------------
+0h (deploy)          198 MB        3                         105 ms
+12h                  237 MB        3                         112 ms
+24h                  241 MB        3                         108 ms
+48h                  239 MB        3                         115 ms
+72h                  244 MB        3                         110 ms
+```
+
+Memory stable at ~240MB. No growth trend. p99 latency normal. Zero OOM kills. Zero pod restarts.
+
+### Full Test Suite
+All 847 tests passing. No regressions introduced.
+
+---
+
+## 7. Prevention Measures
+
+### Immediate
+- Added `orderEmitter.setMaxListeners(10)` to catch future accumulation early (Node.js warns when exceeded)
+- Added memory and listener count to the service's health check endpoint
+- Added Datadog monitor: alert if any EventEmitter has more than 50 listeners
+
+### Process
+- Added to code review checklist: "EventEmitter listeners must not be registered inside request handlers or loops"
+- Added ESLint rule to flag `.on()` or `.addListener()` calls inside Express route handlers
+- Updated onboarding documentation with this incident as a case study
+
+---
+
+## Appendix: Commands Used
+
+```bash
+# Heap snapshot from running process
+kill -USR2 <pid>
+
+# Check listener counts via diagnostic endpoint
+curl -s http://pod-3:3000/_debug/listeners | jq .
+
+# Compare heap snapshots
+# Used Chrome DevTools → Memory → Load snapshots → Comparison view
+
+# Git bisect to confirm which commit introduced the bug
+git bisect start
+git bisect bad v2.14.0
+git bisect good v2.13.0
+# Result: commit a3f7c2e "Add WebSocket order status notifications"
+
+# Load test the fix in staging
+npx autocannon -c 50 -d 14400 -p 2 http://staging:3000/orders/test-1/status \
+  -m POST -H "Content-Type: application/json" -b '{"status":"shipped"}'
+```