wicked-garden 12.27.0

package/skills/platform/errors/SKILL.md

@@ -0,0 +1,183 @@
+---
+name: errors
+description: |
+  Use when investigating a production error spike or pattern across services — aggregates errors from
+  discovered tracking sources, correlates with deployments, and assesses user impact.
+  NOT for general observability (use platform/observability) or tracing latency (use the platform:traces command).
+phase_relevance: ["build", "review", "operate"]
+archetype_relevance: ["*"]
+---
+
+# Error Analysis Skill
+
+Aggregate and analyze errors from discovered error tracking sources with pattern detection.
+
+## When to Use
+
+- Error spike investigation
+- Pattern detection across services
+- User impact assessment
+- Correlation with deployments
+- Error trend analysis
+- User asks "errors", "exceptions", "why failing"
+
+## Error Analysis Approach
+
+### 1. Discover Error Tracking Sources
+
+Use capability-based discovery:
+
+```bash
+# Discover available integrations via capability detection
+
+# Scan for error tracking capabilities by analyzing server descriptions:
+# - error-tracking capability: Dedicated exception/error tracking
+# - apm capability: APM tools that include error tracking
+# - logging capability: Log platforms with error search/aggregation
+```
+
+### 2. Aggregate Error Data
+
+For each discovered source:
+- Current error rate vs baseline
+- Top errors by frequency
+- New vs recurring errors
+- User sessions affected
+- Error distribution by service/region
+
+### 3. Detect Patterns
+
+Look for:
+- **Error spikes**: Sudden increase in error rate
+- **New errors**: First seen in recent time window
+- **Error clusters**: Related errors happening together
+- **User impact**: Same users hitting multiple errors
+- **Service correlation**: Errors across dependent services
+
+### 4. Correlate with Changes
+
+Check for correlation with:
+- Recent deployments
+- Code changes via wicked-garden:search
+- Infrastructure changes
+- Traffic patterns
+- External dependency changes
+
+### 5. Provide Investigation Path
+
+Based on patterns:
+- Root cause hypothesis
+- Investigation steps
+- Recommended actions (rollback, hotfix, etc.)
+- Integration with engineering for debugging
+
+## Integration Discovery
+
+| Capability | What to Look For | Provides |
+|------------|------------------|----------|
+| **error-tracking** | Exception tracking, crash reporting, error grouping | Stack traces, user context, grouping |
+| **apm** | Performance monitoring with error tracking features | Errors with performance context |
+| **logging** | Log platforms with error filtering and search | Error logs, patterns, search |
+
+**Fallback**: Search code for error patterns via wicked-garden:search (catch blocks, error handling, throw statements).
+
+## Output Format
+
+```markdown
+## Error Analysis Report
+
+**Analysis Time**: {timestamp}
+**Time Range**: {period analyzed}
+**Data Sources**: {list of integrations}
+
+### Error Summary
+
+**Current Error Rate**: {rate} ({change} from baseline)
+**Total Errors**: {count} in last {period}
+**Unique Errors**: {count} distinct error types
+**Affected Users**: {count or percentage}
+
+### Top Errors
+
+| Error | Count | Users | First Seen | Trend |
+|-------|-------|-------|------------|-------|
+| {message} | {count} | {users} | {time} | {↑↓→} |
+
+### Pattern Detection
+
+**Pattern: {Pattern Name}**
+- Type: [ERROR_SPIKE | NEW_ERROR | CASCADING | USER_CLUSTER]
+- Description: {what the pattern indicates}
+- Affected: {services, users, regions}
+- Started: {timestamp}
+- Correlation: {deployment, code change, etc.}
+
+### Investigation Path
+
+**Hypothesis**: {most likely root cause}
+
+**Evidence**:
+1. {supporting evidence point}
+2. {supporting evidence point}
+
+**Next Steps**:
+1. {specific action to take}
+2. {specific action to take}
+
+**Engage**: the engineering skill's debugging workflow (`skills/engineering/debugging/`) for code-level root-cause analysis
+```
+
+## Common Error Patterns
+
+Four main patterns to detect. See refs/patterns.md for detailed analysis guides.
+
+### ERROR_SPIKE
+Sudden increase in error rate (>2x baseline). Often deployment-related.
+
+### NEW_ERROR
+Error never seen before. Indicates new code path or edge case.
+
+### CASCADING_FAILURE
+One error causing downstream errors. Check service dependencies.
+
+### USER_CLUSTER
+Same users experiencing multiple errors. User-specific data issue.
+
+## Integration with wicked-crew
+
+When crew completes build phase:
+1. Compare error rates pre/post deployment
+2. Alert on new errors
+3. Detect error spikes
+4. Recommend rollback if critical
+
+Emit events:
+- `observe:error:spike:warning`
+- `observe:error:pattern:warning`
+- `observe:correlation:found:success`
+
+## Integration with engineering
+
+When errors detected, run the engineering debugging workflow with context:
+- Error stack traces from discovered sources
+- Log context from logging integrations
+- Recent code changes from git
+- Deployment timeline
+
+## Severity Classification
+
+See refs/severity.md for detailed classification.
+
+**CRITICAL**: Error rate >10x baseline, critical path errors, data corruption
+**HIGH**: Error rate >3x baseline, affecting >10% users, payment/security errors
+**MEDIUM**: Error rate >1.5x baseline, affecting <10% users, non-critical features
+**LOW**: Error rate <1.5x baseline, cosmetic issues, logging errors
+
+## Notes
+
+- Focus on error rate changes, not absolute numbers
+- New errors often more critical than recurring ones
+- Always correlate with deployments and code changes
+- Distinguish between symptoms and root causes
+- Use traces to understand error propagation
+- Consider user impact, not just error count

package/skills/platform/errors/refs/patterns.md

@@ -0,0 +1,271 @@
+---
+phase_relevance: ["build", "review", "operate"]
+archetype_relevance: ["*"]
+---
+# Error Pattern Analysis Guide
+
+Detailed investigation guides for common error patterns.
+
+## ERROR_SPIKE
+
+Sudden increase in error rate (>2x baseline in short time).
+
+### Typical Causes
+
+1. **Recent deployment**
+   - New bugs introduced
+   - Breaking changes
+   - Configuration errors
+   - Migration issues
+
+2. **Traffic spike**
+   - Unexpected load
+   - Resource exhaustion
+   - Rate limiting triggered
+   - Cache invalidation
+
+3. **Dependency failure**
+   - External API down
+   - Database issues
+   - Third-party service degradation
+   - Network problems
+
+4. **Configuration change**
+   - Environment variable changes
+   - Feature flag toggles
+   - Infrastructure updates
+   - Security policy changes
+
+### Investigation Steps
+
+1. **Check deployment timeline**
+   - When was last deployment?
+   - What changed in that deployment?
+   - Were there any rollbacks recently?
+
+2. **Compare error messages pre/post spike**
+   - Are these new error messages?
+   - Are existing errors occurring more frequently?
+   - What services are affected?
+
+3. **Check dependency health**
+   - Query APM for external service health
+   - Check database performance metrics
+   - Review third-party service status pages
+
+4. **Review recent configuration changes**
+   - Environment variable changes
+   - Feature flag modifications
+   - Infrastructure or scaling changes
+
+### Recommended Actions
+
+If error rate >5x baseline:
+- Consider immediate rollback
+- Engage incident response
+- Notify stakeholders
+
+If error rate 2-5x baseline:
+- Investigate root cause
+- Prepare rollback plan
+- Monitor for escalation
+
+## NEW_ERROR
+
+Error message never seen before (or not in last 30 days).
+
+### Typical Causes
+
+1. **New code path**
+   - Recently deployed features
+   - New integrations
+   - Refactored code
+   - New dependencies
+
+2. **New edge case**
+   - Unexpected input data
+   - Rare race conditions
+   - Boundary conditions
+   - Data type mismatches
+
+3. **Data format change**
+   - API contract changes
+   - Database schema migration
+   - External data source changes
+   - Serialization issues
+
+4. **Integration change**
+   - Third-party API updates
+   - New external dependencies
+   - Authentication changes
+   - Network routing changes
+
+### Investigation Steps
+
+1. **Identify when first seen**
+   - Exact timestamp of first occurrence
+   - How many times since first seen?
+   - Is frequency increasing?
+
+2. **Find related code changes**
+   - Git log around first occurrence time
+   - Deployment records
+   - PR/commit analysis via wicked-garden:search
+
+3. **Review stack trace for new code**
+   - Files/functions in the trace
+   - Recently modified code
+   - New dependencies or imports
+
+4. **Check for new integrations or data sources**
+   - External API calls
+   - Database queries
+   - File system operations
+   - Network requests
+
+### Recommended Actions
+
+For critical path errors:
+- Immediate investigation
+- Disable feature if possible
+- Prepare hotfix or rollback
+
+For non-critical errors:
+- Log detailed context
+- Create bug ticket
+- Schedule fix in next sprint
+
+## CASCADING_FAILURE
+
+One error causing downstream errors in dependent services.
+
+### Typical Causes
+
+1. **Service timeout propagation**
+   - Upstream service slow/failing
+   - Timeout values too long
+   - No circuit breaker
+   - Synchronous calls blocking
+
+2. **Missing circuit breaker**
+   - Failures propagate to callers
+   - Resource exhaustion
+   - Thread pool saturation
+   - Connection pool exhaustion
+
+3. **Retry storms**
+   - Aggressive retry logic
+   - No exponential backoff
+   - No jitter in retries
+   - Amplifies load on failing service
+
+4. **Resource exhaustion**
+   - Memory leaks
+   - Connection leaks
+   - Thread exhaustion
+   - File descriptor limits
+
+### Investigation Steps
+
+1. **Use traces to map service dependencies**
+   - Identify call chain
+   - Find root failing service
+   - Analyze error propagation path
+
+2. **Identify root failing service**
+   - Which service failed first?
+   - What triggered the failure?
+   - Check service health metrics
+
+3. **Check timeout configurations**
+   - Are timeouts set appropriately?
+   - Are they too long (blocking)?
+   - Are they too short (false failures)?
+
+4. **Look for retry logic amplification**
+   - How many retries per request?
+   - Is there exponential backoff?
+   - Is jitter applied?
+   - What's the total load multiplier?
+
+### Recommended Actions
+
+Immediate:
+- Implement circuit breaker if missing
+- Reduce retry aggressiveness
+- Add timeout if missing
+- Scale failing service if possible
+
+Long-term:
+- Review resilience patterns
+- Implement bulkhead isolation
+- Add health check endpoints
+- Monitor circuit breaker state
+
+## USER_CLUSTER
+
+Same users experiencing multiple different errors.
+
+### Typical Causes
+
+1. **User-specific data issue**
+   - Corrupted user data
+   - Invalid data format
+   - Missing required fields
+   - Data migration errors
+
+2. **Session corruption**
+   - Serialization issues
+   - Session storage problems
+   - Token expiration
+   - Cache inconsistency
+
+3. **Authentication problem**
+   - Invalid credentials
+   - Expired tokens
+   - Permission issues
+   - OAuth flow errors
+
+4. **Client version issue**
+   - Old mobile app version
+   - Browser compatibility
+   - API version mismatch
+   - Feature flag inconsistency
+
+### Investigation Steps
+
+1. **Analyze user characteristics**
+   - Geographic region
+   - Client version/platform
+   - Account age/type
+   - Feature flags enabled
+
+2. **Check user data integrity**
+   - Query user record
+   - Validate data format
+   - Check for missing fields
+   - Review recent data changes
+
+3. **Review session management**
+   - Session creation/validation
+   - Token handling
+   - Cache consistency
+   - Storage backend health
+
+4. **Check for client-side issues**
+   - Client version distribution
+   - Browser/OS compatibility
+   - Network conditions
+   - Local storage issues
+
+### Recommended Actions
+
+If affecting many users:
+- Identify common characteristics
+- Disable feature for affected segment
+- Fix data issue or rollback
+
+If affecting few users:
+- Support ticket resolution
+- Manual data correction
+- Client update recommendation

package/skills/platform/errors/refs/severity.md

@@ -0,0 +1,264 @@
+---
+phase_relevance: ["build", "review", "operate"]
+archetype_relevance: ["*"]
+---
+# Error Severity Classification
+
+Guidelines for classifying error severity and determining appropriate response.
+
+## Severity Levels
+
+### CRITICAL
+
+**Criteria**:
+- Error rate >10x baseline
+- New error in critical path (auth, payment, checkout, etc.)
+- Complete service failure or outage
+- Data corruption or data loss errors
+- Security breach indicators
+
+**Response Time**: Immediate (page on-call if after hours)
+
+**Actions**:
+- Declare incident
+- Engage incident responder
+- Consider immediate rollback
+- Notify stakeholders every 30 minutes
+- All hands on deck if needed
+
+**Examples**:
+```
+- Authentication service returning 500 for all requests
+- Payment processing failing for 100% of transactions
+- Database writes failing (data loss risk)
+- SQL injection attack detected in logs
+- Memory leak causing OOM crashes every 5 minutes
+```
+
+### HIGH
+
+**Criteria**:
+- Error rate >3x baseline
+- New error affecting >10% of users
+- Payment or transaction-related errors
+- Security-related errors (non-critical)
+- Major feature completely broken
+
+**Response Time**: Within 15 minutes during business hours
+
+**Actions**:
+- Investigate immediately
+- Prepare rollback plan
+- Notify stakeholders hourly
+- Consider feature disable if possible
+- Hotfix or rollback within 2 hours
+
+**Examples**:
+```
+- Checkout flow failing for 15% of users
+- User registration errors spiking
+- Password reset emails not sending
+- Critical API endpoints returning errors
+- Data export feature completely broken
+```
+
+### MEDIUM
+
+**Criteria**:
+- Error rate >1.5x baseline
+- New error affecting <10% of users
+- Non-critical feature broken
+- Recoverable errors (retries succeeding)
+- Performance degradation errors
+
+**Response Time**: Within business hours (same day)
+
+**Actions**:
+- Investigate and root cause
+- Schedule fix in current sprint
+- Update status page if user-facing
+- Daily stakeholder updates
+- Implement monitoring/alerting
+
+**Examples**:
+```
+- Search returning errors for 5% of queries
+- Image upload failing occasionally
+- Non-critical API rate limiting errors
+- Cache misses causing slow responses
+- Background job failures with retry
+```
+
+### LOW
+
+**Criteria**:
+- Error rate <1.5x baseline
+- Cosmetic or UI issues
+- Logging or instrumentation errors
+- Low-traffic feature errors
+- Development/staging environment errors
+
+**Response Time**: Next sprint or backlog
+
+**Actions**:
+- Create bug ticket
+- Include in weekly update
+- Fix when capacity available
+- Consider if fix is worth effort
+- May close as "won't fix"
+
+**Examples**:
+```
+- Typo in error message
+- Analytics event tracking error
+- Debug logging errors
+- Admin panel minor issues
+- Deprecated API warnings
+```
+
+## Decision Matrix
+
+Use this matrix to classify errors:
+
+| Factor | CRITICAL | HIGH | MEDIUM | LOW |
+|--------|----------|------|--------|-----|
+| **Error Rate** | >10x baseline | >3x baseline | >1.5x baseline | <1.5x baseline |
+| **User Impact** | All/most users | >10% users | <10% users | Minimal |
+| **Function** | Critical path | Major feature | Minor feature | Cosmetic |
+| **Data Risk** | Loss/corruption | Inconsistency | None | None |
+| **Revenue Impact** | Direct loss | Potential loss | Minimal | None |
+| **Security** | Active breach | Vulnerability | Weakness | Info disclosure |
+
+## Special Considerations
+
+### Escalate Severity If:
+
+1. **Error rate is increasing rapidly**
+   - MEDIUM → HIGH if doubling every 15 minutes
+   - HIGH → CRITICAL if affecting critical path
+
+2. **Multiple services affected**
+   - Indicates cascading failure
+   - Potential systemic issue
+
+3. **During high-traffic period**
+   - Black Friday, holiday season
+   - Product launch day
+   - Marketing campaign
+
+4. **Media/PR sensitivity**
+   - Public-facing issue
+   - Social media mentions
+   - Competitor awareness
+
+### Downgrade Severity If:
+
+1. **Isolated to test/staging environment**
+   - Production not affected
+   - No user impact
+
+2. **Auto-recovery happening**
+   - Retries succeeding
+   - Circuit breaker protecting
+   - Self-healing in progress
+
+3. **Feature flag isolation**
+   - Can disable feature
+   - Limited rollout (1% users)
+   - Canary deployment
+
+4. **Workaround available**
+   - Users can accomplish task differently
+   - Support can manually process
+   - Alternative feature exists
+
+## Response Actions by Severity
+
+### CRITICAL Response
+
+```markdown
+1. [ ] Declare incident (PagerDuty, Slack, etc.)
+2. [ ] Page on-call engineer immediately
+3. [ ] Engage incident responder
+4. [ ] Start incident timeline
+5. [ ] Assess rollback vs hotfix
+6. [ ] Notify stakeholders (30min cadence)
+7. [ ] Execute rollback if safe
+8. [ ] Monitor for resolution
+9. [ ] Verify user impact reduced
+10. [ ] Schedule postmortem
+```
+
+### HIGH Response
+
+```markdown
+1. [ ] Create incident ticket
+2. [ ] Notify team in Slack
+3. [ ] Investigate root cause (15min)
+4. [ ] Assess rollback vs hotfix
+5. [ ] Prepare rollback plan
+6. [ ] Notify stakeholders (hourly)
+7. [ ] Implement fix within 2 hours
+8. [ ] Deploy and verify
+9. [ ] Monitor for 1 hour
+10. [ ] Document in incident log
+```
+
+### MEDIUM Response
+
+```markdown
+1. [ ] Create bug ticket
+2. [ ] Assign to team member
+3. [ ] Root cause analysis
+4. [ ] Schedule fix this sprint
+5. [ ] Update status page
+6. [ ] Implement fix
+7. [ ] Deploy in normal release
+8. [ ] Verify resolution
+9. [ ] Close ticket
+```
+
+### LOW Response
+
+```markdown
+1. [ ] Create backlog item
+2. [ ] Prioritize in grooming
+3. [ ] Fix when capacity available
+4. [ ] Or close as won't fix
+```
+
+## Communication Templates
+
+### CRITICAL
+```
+[CRITICAL] {Service} - {Brief Description}
+Impact: {percentage}% of users / ${revenue} at risk
+Started: {timestamp}
+Status: Investigating / Mitigating / Resolved
+Team: {responders}
+Next Update: 30 minutes
+```
+
+### HIGH
+```
+[HIGH] {Service} - {Brief Description}
+Impact: {user count} users affected
+Started: {timestamp}
+ETA: {best estimate}
+Next Update: 1 hour
+```
+
+### MEDIUM
+```
+[MEDIUM] {Service} - {Brief Description}
+Impact: Limited to {feature}
+Fix Scheduled: {sprint/date}
+Status Page: {link}
+```
+
+### LOW
+```
+[LOW] {Service} - {Brief Description}
+Tracked: {ticket link}
+Minimal impact, will fix when capacity available
+```