mindforge-cc 10.0.2 → 10.7.0

package/.mindforge/skills/observability-stack/SKILL.md

@@ -0,0 +1,136 @@
+---
+name: observability-stack
+version: 1.0.0
+min_mindforge_version: 10.0.7
+status: stable
+triggers: observability stack, structured logging, distributed tracing, OpenTelemetry setup, RED method metrics, USE method metrics, alerting threshold design, dashboard design, correlation ID, metrics collection strategy, log aggregation, trace context propagation
+---
+
+# Observability Stack
+
+## When this skill activates
+
+This skill activates when designing, implementing, or reviewing observability infrastructure for production systems. It covers the three pillars (logs, traces, metrics), alerting strategy, dashboard design, and correlation techniques. Use this skill whenever you need to ensure a system can be understood, debugged, and monitored in production.
+
+## Mandatory actions when this skill is active
+
+### Before
+
+1. **Identify observability goals** — What questions must you answer? "Why is this request slow?" (traces), "What's the error rate?" (metrics), "What happened at 3:42am?" (logs).
+2. **Assess current state** — What instrumentation exists? What gaps cause blind spots? Never add observability without understanding what's already there.
+3. **Define SLIs/SLOs** — Service Level Indicators (what to measure) and Service Level Objectives (acceptable thresholds) must exist before alerting makes sense.
+4. **Choose the stack** — Select instrumentation (OpenTelemetry), storage (Prometheus/Loki/Tempo, Datadog, Grafana Cloud), and visualization (Grafana, Datadog) based on team expertise and budget.
+
+### During
+
+#### Pillar 1: Structured Logging
+
+**Format** — Always structured JSON with fields: timestamp, level, message, service, trace_id, span_id, correlation_id, error_code, duration_ms, metadata.
+
+**Severity levels:**
+- `DEBUG` — Development-only detail. Never enable in production by default.
+- `INFO` — Normal operations: request received, task completed, state transitions.
+- `WARN` — Degraded but functional: retry succeeded, fallback activated, approaching limit.
+- `ERROR` — Operation failed: request could not be fulfilled, exception caught.
+- `FATAL` — System cannot continue: startup failure, unrecoverable state, crash.
+
+**Correlation IDs:**
+- Generate at the edge (API gateway). Propagate via `X-Correlation-ID` header through all services.
+- Include in every log line. Distinct from trace_id but can be linked.
+
+**Logging rules:**
+- Log at service boundaries (incoming requests, outgoing calls).
+- Log state transitions (order created, payment confirmed, email sent).
+- Log errors with full context (what was attempted, what failed, what the input was).
+- NEVER log secrets, tokens, passwords, or full credit card numbers.
+- NEVER log PII without a documented legal basis and redaction strategy.
+
+#### Pillar 2: Distributed Tracing
+
+**Span design:**
+- Create spans for: HTTP requests (in/out), database queries, cache operations, external API calls, queue publish/consume, significant business logic.
+- DO NOT span: trivial operations, tight loops, in-memory calculations.
+- Name spans with: `verb.noun` format: `process.payment`, `query.users`, `send.email`.
+
+**Context propagation:**
+- Use W3C Trace Context standard (`traceparent` header) for cross-service propagation.
+- Ensure all HTTP clients and message queues propagate context automatically.
+- Test propagation: a trace should span from the edge to the deepest downstream call.
+
+**Sampling strategy:**
+- 100% sampling in development and staging.
+- Production: tail-based sampling (keep all errors + slow requests + 1-10% of normal traffic).
+- Never sample at 100% in high-traffic production. Storage costs scale linearly.
+- Always keep traces for errors regardless of sampling rate.
+
+#### Pillar 3: Metrics
+
+**RED Method (for services/endpoints):**
+- **Rate** — Requests per second. Indicates load.
+- **Errors** — Failed requests per second (and error rate as %). Indicates reliability.
+- **Duration** — Latency distribution (p50, p95, p99). Indicates performance.
+
+**USE Method (for resources: CPU, memory, disk, network):**
+- **Utilization** — Percentage of resource capacity in use.
+- **Saturation** — Work queued waiting for the resource.
+- **Errors** — Resource-level errors (disk I/O errors, network drops).
+
+**Metric types:**
+- **Counter** — Monotonically increasing value. Use for: total requests, errors, bytes transferred.
+- **Gauge** — Point-in-time value that can go up/down. Use for: active connections, queue depth, temperature.
+- **Histogram** — Distribution of values. Use for: request latency, payload sizes.
+
+**Naming convention:** `<service>_<component>_<metric>_<unit>` (e.g., `payment_api_request_duration_seconds`).
+
+**Cardinality discipline:**
+- Never use unbounded values as label/tag keys (user IDs, request IDs, timestamps).
+- Maximum 10-20 unique values per label. High cardinality kills metric storage.
+- Safe labels: HTTP method, status code, endpoint path (grouped), environment, region.
+
+#### Alerting Design
+
+**Principles:**
+- Alert on symptoms (user impact), not causes (CPU spike). Users don't care about CPU; they care about latency.
+- Every alert must be actionable. If no one can do anything about it at 3am, it should not page.
+- Use SLO-based alerting: alert when the error budget burn rate exceeds safe levels.
+
+**Threshold design:**
+- Static thresholds: set at baseline + 2 standard deviations (for stable metrics).
+- Anomaly detection: use ML-based for metrics with seasonal patterns (traffic).
+- Burn rate: alert when 1-hour burn rate > 14x (consumes 24hr budget in 1hr).
+
+**Alert severity:**
+- **Critical (page)** — User-facing impact right now. Revenue loss. Data loss.
+- **Warning (ticket)** — Degraded performance. Will become critical if unaddressed. Handle next business day.
+- **Info (log)** — Notable but not impactful. For awareness only.
+
+**Anti-patterns:**
+- Alert fatigue: >5 alerts/day/team = too many. Tune or eliminate.
+- Flapping alerts: add hysteresis (alert fires at threshold, clears at threshold - 10%).
+- Duplicate alerts: one incident should fire one alert, not one per symptom.
+
+#### Dashboard Design
+
+**Layout:** Row 1: Traffic, Error rate, Latency (p50/p95/p99). Row 2: Saturation (CPU, memory, connections), Active alerts. Row 3: Deployment markers, SLO burn-down.
+
+**Principles:** Top = current health (red/yellow/green). Middle = golden signal time-series. Bottom = drill-down investigation panels. NO vanity metrics. Consistent time ranges, default 1 hour.
+
+### After
+
+1. **Verify correlation** — Generate a test request and verify it can be traced from edge to database across all services via correlation ID and trace ID.
+2. **Validate alerting** — Trigger each alert condition deliberately. Verify it fires, routes correctly, and auto-resolves.
+3. **Load test observability** — Ensure the observability stack itself handles production load without becoming a bottleneck.
+4. **Document runbooks** — Every alert needs a runbook: what it means, how to diagnose, how to mitigate.
+
+## Self-check before task completion
+
+- [ ] All logs are structured JSON with correlation IDs, timestamps, and severity levels
+- [ ] Distributed traces span from edge to deepest downstream call with proper context propagation
+- [ ] RED metrics (Rate, Errors, Duration) are collected for every service endpoint
+- [ ] USE metrics (Utilization, Saturation, Errors) are collected for critical resources
+- [ ] Metric label cardinality is bounded (no unbounded labels like user IDs)
+- [ ] Alerts are symptom-based, actionable, and severity-appropriate
+- [ ] Dashboard shows golden signals with drill-down capability
+- [ ] No secrets or unbounded PII appear in logs
+- [ ] Sampling strategy is defined for production tracing
+- [ ] Runbooks exist for every critical alert

package/.mindforge/skills/offline-first-design/SKILL.md

@@ -0,0 +1,43 @@
+---
+name: offline-first-design
+version: 1.0.0
+min_mindforge_version: 10.4.0
+status: stable
+triggers: offline-first architecture, conflict resolution sync, sync protocol design, local-first data, CRDT implementation, offline data persistence, background sync, offline queue pattern, network reconnection handling, eventual consistency mobile, offline cache strategy, optimistic offline update
+---
+
+# Skill — Offline-First Design & Sync Architecture
+
+## When this skill activates
+This skill activates when designing offline-capable mobile or web applications, including conflict resolution strategies, sync protocols, local-first data patterns, or CRDT-based collaborative features.
+
+## Mandatory actions when this skill is active
+
+### Before writing any code
+1. Define conflict resolution strategy (last-write-wins, CRDT, operational transformation, or custom merge logic)
+2. Choose local storage solution appropriate for data volume and query patterns (SQLite, Realm, WatermelonDB, IndexedDB)
+3. Design sync protocol with idempotency, versioning, and resumable transfers for unreliable networks
+4. Establish data model with conflict-free properties where possible (append-only logs, tombstone deletions)
+
+### During implementation
+- Implement optimistic UI updates with rollback capability — show immediate feedback, reconcile on sync
+- Use operation queues with exponential backoff for failed network requests, persist queue to survive app restarts
+- Store vector clocks, lamport timestamps, or hybrid logical clocks to track causality across devices
+- Implement delta sync to minimize bandwidth — send only changed fields, not entire documents
+- Handle edge cases: simultaneous edits on multiple devices, partial sync failures, stale read after write
+- Use CRDTs (Automerge, Yjs, or custom implementations) for collaborative editing with automatic conflict resolution
+- Implement sync status indicators and manual sync triggers for user transparency
+
+### After implementation
+- Test offline scenarios exhaustively: airplane mode, flaky networks, background app termination during sync
+- Simulate conflict scenarios with multiple devices editing same data simultaneously
+- Verify data integrity after sync — no duplicate records, no lost updates, correct conflict resolution
+- Measure sync performance: time to sync after reconnection, battery impact, bandwidth usage
+
+## Self-check before task completion
+- [ ] App remains functional offline with clear UI indication of sync status
+- [ ] Conflicts are resolved correctly according to business rules, with audit trail if needed
+- [ ] Sync protocol is idempotent — replaying operations produces same result
+- [ ] Local data persists correctly across app restarts and OS updates
+- [ ] Background sync works within platform constraints (iOS background fetch, Android WorkManager)
+- [ ] Edge cases handled: clock skew, partial sync, corrupted local data, schema migrations during offline period

package/.mindforge/skills/on-call-design/SKILL.md

@@ -0,0 +1,111 @@
+---
+name: on-call-design
+version: 1.0.0
+min_mindforge_version: 10.1.0
+status: stable
+triggers: on-call design, rotation fairness, escalation SLA, runbook coverage, toil budget, on-call compensation, burnout prevention, page frequency, on-call rotation schedule, alert fatigue, on-call handbook, incident response time
+compose: incident-management
+---
+
+# On-Call Design
+
+## When this skill activates
+
+This skill activates when designing, evaluating, or improving on-call rotations,
+escalation policies, alerting strategies, and the operational sustainability of
+incident response. It addresses rotation fairness, alert fatigue prevention,
+runbook coverage, and burnout prevention to create a humane and effective on-call
+system.
+
+## Mandatory actions when this skill is active
+
+### Before
+
+1. **Audit current state** — How many pages per week per person? What percentage have
+   runbooks? What is the false-positive rate? How is on-call currently compensated?
+2. **Identify pain points** — Survey on-call participants: What is most frustrating?
+   What causes the most stress? What pages feel unnecessary?
+3. **Define service level objectives** — What response times does the business actually
+   need? Not everything requires a 2 AM page.
+
+### During
+
+4. **Rotation design principles:**
+   - Minimum 2 people in rotation at all times (primary + secondary)
+   - Maximum 1 week per rotation (longer causes burnout)
+   - Follow-the-sun for global teams (no one gets paged at 3 AM regularly)
+   - Handoff includes: active incidents, known issues, upcoming risky changes
+   - New team members shadow for 2 rotations before going primary
+   - Easy swap mechanism for personal conflicts
+
+5. **Fairness and compensation:**
+   - Equal distribution of on-call burden across all eligible engineers
+   - Track after-hours pages per person — redistribute if imbalanced
+   - Compensate on-call time (stipend for carrying the pager, additional compensation
+     for after-hours pages)
+   - Swap-friendly policy — no guilt for requesting swaps
+   - On-call load counts toward sprint capacity (reduce story points by 20-30%)
+   - Never mandate on-call for people who did not agree to it at hiring
+
+6. **Escalation SLAs:**
+   - **Acknowledge** — 5 minutes (confirm human is aware)
+   - **Triage** — 15 minutes (assess severity, determine if action needed now)
+   - **Resolve or escalate** — 60 minutes (either fix it or pull in help)
+   - Clear escalation path: primary → secondary → team lead → engineering manager
+   - Automatic escalation if acknowledgment SLA is missed
+   - No shame in escalating — better to involve help early than struggle alone
+
+7. **Runbook coverage requirements:**
+   - Every alert MUST have a corresponding runbook
+   - Runbook contains: what the alert means, impact if ignored, diagnostic steps,
+     resolution steps, escalation criteria
+   - If no runbook exists for an alert, the alert should not page — it is toil
+   - Runbooks must be tested quarterly (can a new engineer follow them successfully?)
+   - Runbook updates are mandatory after any incident where the runbook was insufficient
+
+8. **Toil budget management:**
+   - Define toil: repetitive, automatable, reactive work with no lasting value
+   - If >30% of on-call time is toil, invest in automation
+   - Track toil hours separately from incident response hours
+   - Dedicate engineering time each sprint to reducing top toil sources
+   - Toil that persists for >3 months without investment is a leadership failure
+
+9. **Alert fatigue prevention:**
+   - If an alert does not require human action within 5 minutes, delete it
+   - Target: fewer than 5 pages per week per person
+   - Review all alerts quarterly: keep, tune, or delete
+   - Silence non-actionable alerts aggressively (log them, don't page for them)
+   - Group related alerts to prevent page storms (one page for a cascading failure,
+     not 50)
+   - Every page that did not need human intervention is a bug in your alerting
+
+10. **Burnout prevention:**
+    - Monitor page frequency trends — rising trends require immediate action
+    - After a high-severity incident with extended resolution, give the responder
+      compensatory time off
+    - Regular check-ins on on-call stress (not just "how many pages" but "how do you
+      feel about on-call?")
+    - Option to temporarily step out of rotation for personal reasons without stigma
+    - Celebrate good on-call: quiet weeks are victories, not luck
+
+### After
+
+11. **Publish on-call handbook** — Document all policies, rotations, escalation paths,
+    compensation, and expectations in one discoverable location.
+12. **Review quarterly** — Analyze page frequency, response times, toil percentage,
+    fairness distribution, and team satisfaction.
+13. **Iterate on alerts** — After every on-call rotation, the outgoing person reviews
+    alerts received and recommends: keep as-is, tune threshold, add runbook, or delete.
+
+## Self-check before task completion
+
+- [ ] Rotation has minimum 2 people with maximum 1-week shifts
+- [ ] Compensation model defined for on-call time and after-hours pages
+- [ ] Escalation SLAs documented (acknowledge: 5min, triage: 15min, resolve: 60min)
+- [ ] Every alert has a corresponding runbook (or is deleted/silenced)
+- [ ] Page frequency target set (<5/week/person) with current baseline measured
+- [ ] Toil budget defined (<30%) with automation investment plan for top sources
+- [ ] Alert review cadence established (quarterly minimum)
+- [ ] Burnout prevention measures in place (comp time, swap policy, check-ins)
+- [ ] On-call handbook published and discoverable
+- [ ] Fairness tracking mechanism active (pages per person distribution)

package/.mindforge/skills/pagination-patterns/SKILL.md

@@ -0,0 +1,230 @@
+---
+name: pagination-patterns
+version: 1.0.0
+min_mindforge_version: 0.3.0
+status: stable
+triggers: pagination pattern, cursor pagination, offset pagination, keyset pagination, total count strategy, infinite scroll, deep pagination, page token, relay connection, pagination performance, next page token, pagination API design
+compose: api-design
+---
+
+# Skill — Pagination Patterns
+
+## When this skill activates
+Any task involving list endpoint pagination, cursor vs offset strategies, infinite
+scroll implementation, total count optimization, or pagination API design.
+
+## Mandatory actions when this skill is active
+
+### Before implementing pagination
+1. Determine the access pattern (jump to page N vs sequential browsing).
+2. Assess data volume and write frequency (affects consistency).
+3. Choose the strategy based on trade-offs, not convention.
+
+### Offset pagination
+
+**Implementation:**
+```sql
+SELECT * FROM orders
+WHERE tenant_id = 'abc'
+ORDER BY created_at DESC
+LIMIT 20 OFFSET 40;  -- page 3
+```
+
+**API design:**
+```json
+GET /orders?page=3&per_page=20
+
+{
+  "data": [...],
+  "meta": {
+    "page": 3,
+    "per_page": 20,
+    "total": 1542,
+    "total_pages": 78
+  }
+}
+```
+
+**Pros:**
+- Simple to implement and understand.
+- Allows jumping to any page directly.
+- Total count and page numbers are intuitive for users.
+
+**Cons:**
+- Performance degrades on deep pages (OFFSET 10000 still scans 10000 rows).
+- Inconsistent during writes (rows shift between pages as new data is inserted).
+- Total count is expensive on large tables (separate COUNT query).
+
+**Best for:** admin panels, back-office UIs, small datasets (< 100K rows).
+
+### Cursor/Keyset pagination
+
+**Implementation:**
+```sql
+-- First page
+SELECT * FROM orders
+WHERE tenant_id = 'abc'
+ORDER BY created_at DESC, id DESC
+LIMIT 20;
+
+-- Next page (cursor = last seen created_at + id)
+SELECT * FROM orders
+WHERE tenant_id = 'abc'
+  AND (created_at, id) < ('2025-01-15T10:00:00Z', 'order-xyz')
+ORDER BY created_at DESC, id DESC
+LIMIT 20;
+```
+
+**API design:**
+```json
+GET /orders?limit=20&after=eyJjcmVhdGVkX2F0IjoiMjAyNS0wMS0xNVQxMDowMDowMFoiLCJpZCI6Im9yZGVyLXh5eiJ9
+
+{
+  "data": [...],
+  "meta": {
+    "has_next": true,
+    "next_cursor": "eyJjcmVhdGVkX...",
+    "has_previous": true,
+    "previous_cursor": "eyJjcmVhdGVk..."
+  }
+}
+```
+
+**Pros:**
+- Constant performance regardless of depth (no OFFSET scan).
+- Consistent results during writes (no row shifting).
+- Works well at any scale.
+
+**Cons:**
+- Cannot jump to arbitrary page N.
+- Cursor is opaque — clients cannot construct cursors manually.
+- Requires a unique, sequential ordering key (composite if not unique).
+
+**Best for:** feeds, timelines, infinite scroll, large datasets, real-time data.
+
+### Cursor encoding
+
+**Rules:**
+- Cursor is opaque to the client (base64-encode the actual values).
+- Include all ORDER BY columns in the cursor (for deterministic positioning).
+- Add a tiebreaker (id) when the sort column is not unique.
+- Validate cursors server-side (prevent injection via crafted cursors).
+
+**Example encoding:**
+```javascript
+// Encode
+const cursor = Buffer.from(JSON.stringify({
+  created_at: lastItem.createdAt,
+  id: lastItem.id
+})).toString('base64url');
+
+// Decode
+const { created_at, id } = JSON.parse(Buffer.from(cursor, 'base64url').toString());
+```
+
+### Relay Connection specification
+
+```graphql
+type OrderConnection {
+  edges: [OrderEdge!]!
+  pageInfo: PageInfo!
+  totalCount: Int  # nullable — expensive to compute
+}
+
+type OrderEdge {
+  node: Order!
+  cursor: String!
+}
+
+type PageInfo {
+  hasNextPage: Boolean!
+  hasPreviousPage: Boolean!
+  startCursor: String
+  endCursor: String
+}
+```
+
+**Usage:**
+- `first: 20, after: "cursor"` — forward pagination.
+- `last: 20, before: "cursor"` — backward pagination.
+- Standardized across the entire GraphQL schema.
+
+### Total count strategies
+
+**Problem:** `SELECT COUNT(*) FROM large_table` is slow (full table scan in PostgreSQL).
+
+**Solutions:**
+
+| Strategy | Accuracy | Performance | Use case |
+|----------|----------|-------------|----------|
+| Exact count (separate query) | 100% | Slow (large tables) | Admin UIs, small datasets |
+| Estimated count (`pg_stat_user_tables.reltuples`) | ~95% | Instant | "About X results" display |
+| Count with cap (`COUNT(*) ... LIMIT 1001`) | Exact up to cap | Fast | "1000+ results" |
+| Cached count (materialized/Redis) | Stale by TTL | Instant | Dashboards, frequently queried |
+| No count (cursor only) | N/A | N/A | Infinite scroll, feeds |
+
+**Recommendation:** Make totalCount optional in the API. Let clients request it only when needed.
+
+### Infinite scroll implementation
+
+**Frontend pattern:**
+```javascript
+function useInfiniteList(fetchPage) {
+  const [pages, setPages] = useState([]);
+  const [cursor, setCursor] = useState(null);
+  const [hasMore, setHasMore] = useState(true);
+  const [isLoading, setIsLoading] = useState(false);
+
+  const loadMore = async () => {
+    if (isLoading || !hasMore) return;
+    setIsLoading(true);
+    const result = await fetchPage(cursor);
+    setPages(prev => [...prev, result.data]);
+    setCursor(result.nextCursor);
+    setHasMore(result.hasNext);
+    setIsLoading(false);
+  };
+
+  // Trigger loadMore on scroll near bottom
+  // Virtualize rendered list for performance
+}
+```
+
+**Rules:**
+- Use cursor-based pagination (offset causes duplicates on insert).
+- Prefetch next page when user is near the bottom (IntersectionObserver).
+- Virtualize the rendered list (react-window, tanstack-virtual) for DOM performance.
+- Show loading skeleton, not spinner, for smoother experience.
+- Handle "back to top" — maintain scroll position on return navigation.
+
+### Deep pagination protection
+
+- For offset: cap maximum page (e.g., max 500 pages) and show "refine your search."
+- For cursor: no cap needed (performance is constant).
+- Rate-limit rapid sequential page requests (scraping protection).
+- Log deep pagination usage — it often indicates a missing search/filter feature.
+
+### API design best practices
+
+**REST:**
+- Use Link headers for next/prev URLs (HATEOAS).
+- Include pagination metadata in response body.
+- Consistent parameter names across all endpoints.
+- Document maximum page size (prevent `per_page=999999`).
+
+**GraphQL:**
+- Use Connection spec (edges + pageInfo) universally.
+- Make totalCount nullable (computed only when requested).
+- Support both first/after and last/before.
+
+**General:**
+- Default page size: 20-50 (not too small, not too large).
+- Maximum page size: 100 (enforce server-side).
+- Sort order must be deterministic (include tiebreaker column).
+- Return empty array (not null) when no results.
+
+## Self-check before task completion
+- [ ] Did I follow the mandatory actions for this skill?
+- [ ] Did I apply the patterns appropriate to the context?
+- [ ] Did I verify the implementation meets the criteria above?
+- [ ] Did I document decisions and trade-offs made?