kodelyth-ecc 2.18.0 → 2.19.0

package/.kimi/skills/observability/SKILL.md

@@ -0,0 +1,390 @@
+---
+name: observability
+description: Production observability patterns — structured logging, distributed tracing, metrics, health checks, alerting, and error budgets. Build systems you can understand when they break. Powered by Kodelyth.
+origin: Kodelyth
+---
+
+# Observability — See Everything in Production
+
+The three pillars of observability — Logs, Metrics, and Traces — plus health checks, alerting, and error budgets. Powered by Kodelyth.
+
+## When to Use
+
+- Setting up logging for a new service
+- Adding tracing to an API or background job
+- Designing a metrics + alerting strategy
+- Debugging a production issue you can't reproduce locally
+- Setting up health check endpoints
+- Building SLOs and error budgets
+
+---
+
+## The Three Pillars
+
+```
+Logs    → What happened (events, errors, audit trail)
+Metrics → How much / how often (counts, rates, durations)
+Traces  → Why it was slow (request flow across services)
+```
+
+You need all three. Metrics tell you *something is wrong*. Traces tell you *where*. Logs tell you *why*.
+
+---
+
+## Structured Logging
+
+### Never use console.log in production
+
+```typescript
+// BAD: Unstructured — impossible to query, filter, or alert on
+console.log("User logged in: " + userId)
+console.log("Error: " + error.message)
+
+// GOOD: Structured — every field is queryable
+logger.info("user.login", {
+  userId,
+  email: user.email,
+  ip: request.ip,
+  userAgent: request.headers["user-agent"],
+  durationMs: Date.now() - startTime,
+})
+
+logger.error("payment.charge.failed", {
+  userId,
+  orderId,
+  amount,
+  currency,
+  errorCode: error.code,
+  errorMessage: error.message,
+  stripeErrorType: error.type,
+})
+```
+
+### Log Levels — Use the Right Level
+
+```typescript
+logger.debug("cache.hit", { key, ttlRemaining })   // Dev only, verbose
+logger.info("order.created", { orderId, amount })  // Normal business events
+logger.warn("rate.limit.approaching", { userId, count, limit }) // Potential problem
+logger.error("db.query.failed", { query, error })  // Needs attention
+logger.fatal("service.crashed", { reason })        // Immediate action required
+```
+
+**Rule:** INFO is for things you'd want to search in production. DEBUG is noise you turn on temporarily.
+
+### TypeScript Logging Setup (Pino — fastest Node.js logger)
+
+```typescript
+// lib/logger.ts
+import pino from 'pino'
+
+export const logger = pino({
+  level: process.env.LOG_LEVEL ?? 'info',
+  base: {
+    service: 'api',
+    version: process.env.npm_package_version,
+    env: process.env.NODE_ENV,
+  },
+  // Pretty print in dev, JSON in production
+  transport: process.env.NODE_ENV === 'development'
+    ? { target: 'pino-pretty', options: { colorize: true } }
+    : undefined,
+})
+
+// Child logger with request context
+export function requestLogger(requestId: string, userId?: string) {
+  return logger.child({ requestId, userId })
+}
+```
+
+### Python Logging Setup (structlog)
+
+```python
+# lib/logging.py
+import structlog
+import logging
+
+structlog.configure(
+    processors=[
+        structlog.stdlib.filter_by_level,
+        structlog.stdlib.add_logger_name,
+        structlog.stdlib.add_log_level,
+        structlog.processors.TimeStamper(fmt="iso"),
+        structlog.processors.StackInfoRenderer(),
+        structlog.processors.format_exc_info,
+        structlog.processors.JSONRenderer(),
+    ],
+    wrapper_class=structlog.stdlib.BoundLogger,
+    context_class=dict,
+    logger_factory=structlog.stdlib.LoggerFactory(),
+)
+
+log = structlog.get_logger()
+
+# Usage:
+log.info("order.created", order_id=order_id, amount=amount, user_id=user_id)
+log.error("payment.failed", error=str(e), order_id=order_id)
+```
+
+---
+
+## Metrics
+
+### Naming Convention
+
+```
+# Format: namespace_subsystem_name_unit
+http_requests_total              # counter
+http_request_duration_seconds    # histogram
+db_connections_active            # gauge
+cache_hits_total                 # counter
+cache_misses_total               # counter
+queue_depth_messages             # gauge
+payment_amount_dollars           # histogram
+```
+
+### Three Metric Types
+
+```typescript
+// Counter — only goes up (requests, errors, events)
+const requestsTotal = new Counter({
+  name: 'http_requests_total',
+  help: 'Total HTTP requests',
+  labelNames: ['method', 'route', 'status_code'],
+})
+requestsTotal.inc({ method: 'POST', route: '/api/orders', status_code: '201' })
+
+// Gauge — goes up and down (active connections, queue depth, memory)
+const activeConnections = new Gauge({
+  name: 'db_connections_active',
+  help: 'Active database connections',
+})
+activeConnections.set(pool.totalCount)
+
+// Histogram — distribution of values (latency, request size)
+const requestDuration = new Histogram({
+  name: 'http_request_duration_seconds',
+  help: 'HTTP request duration',
+  labelNames: ['method', 'route'],
+  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
+})
+const end = requestDuration.startTimer({ method, route })
+// ... handle request ...
+end()  // records duration automatically
+```
+
+### The Four Golden Signals (Google SRE)
+
+```
+1. Latency    → How long do requests take? (p50, p95, p99)
+2. Traffic    → How many requests/sec?
+3. Errors     → What % of requests fail?
+4. Saturation → How full is the system? (CPU, memory, queue depth)
+```
+
+Alert on these four before anything else.
+
+---
+
+## Distributed Tracing (OpenTelemetry)
+
+```typescript
+// setup/tracing.ts
+import { NodeSDK } from '@opentelemetry/sdk-node'
+import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
+import { Resource } from '@opentelemetry/resources'
+import { SEMRESATTRS_SERVICE_NAME } from '@opentelemetry/semantic-conventions'
+
+const sdk = new NodeSDK({
+  resource: new Resource({
+    [SEMRESATTRS_SERVICE_NAME]: 'payment-service',
+  }),
+  traceExporter: new OTLPTraceExporter({
+    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
+  }),
+})
+
+sdk.start()  // call before importing anything else
+
+// Usage — spans are created automatically for HTTP requests
+// Add custom spans for important operations:
+import { trace } from '@opentelemetry/api'
+
+const tracer = trace.getTracer('payment-service')
+
+async function processPayment(orderId: string) {
+  const span = tracer.startSpan('payment.process')
+  span.setAttributes({ 'order.id': orderId })
+
+  try {
+    const result = await chargeCard(orderId)
+    span.setStatus({ code: SpanStatusCode.OK })
+    return result
+  } catch (error) {
+    span.setStatus({ code: SpanStatusCode.ERROR, message: error.message })
+    span.recordException(error)
+    throw error
+  } finally {
+    span.end()
+  }
+}
+```
+
+---
+
+## Health Check Endpoints
+
+### Basic Structure
+
+```typescript
+// Every service must expose /health and /ready
+// /health  → is the process alive? (used by load balancer)
+// /ready   → is the process ready to serve traffic? (used by k8s)
+
+app.get('/health', (req, res) => {
+  res.json({ status: 'ok', uptime: process.uptime() })
+})
+
+app.get('/ready', async (req, res) => {
+  const checks = await Promise.allSettled([
+    checkDatabase(),
+    checkRedis(),
+    checkExternalApi(),
+  ])
+
+  const results = {
+    database: checks[0].status === 'fulfilled' ? 'ok' : 'error',
+    redis:    checks[1].status === 'fulfilled' ? 'ok' : 'error',
+    external: checks[2].status === 'fulfilled' ? 'ok' : 'error',
+  }
+
+  const allHealthy = Object.values(results).every(v => v === 'ok')
+
+  res.status(allHealthy ? 200 : 503).json({
+    status: allHealthy ? 'ready' : 'degraded',
+    checks: results,
+    timestamp: new Date().toISOString(),
+  })
+})
+```
+
+---
+
+## Alerting — What to Alert On
+
+### Alert Design Rules
+
+```
+1. Alert on SYMPTOMS, not causes
+   BAD:  Alert when CPU > 80%
+   GOOD: Alert when p99 latency > 2s (the symptom users feel)
+
+2. Every alert must be actionable
+   BAD:  "High error rate" with no runbook
+   GOOD: "Error rate > 1% — check Datadog dashboard, likely DB issue"
+
+3. Alert on SLOs, not arbitrary thresholds
+   BAD:  Alert when error rate > 5% (where does 5% come from?)
+   GOOD: Alert when error budget burn rate > 2x (based on SLO math)
+
+4. Avoid alert fatigue
+   BAD:  50 alerts, most noise
+   GOOD: 5 alerts, all critical, all actionable
+```
+
+### The Four Alerts Every Service Needs
+
+```yaml
+# 1. High error rate
+alert: HighErrorRate
+expr: rate(http_requests_total{status_code=~"5.."}[5m]) /
+      rate(http_requests_total[5m]) > 0.01
+for: 2m
+annotations:
+  summary: "Error rate above 1% for 2 minutes"
+  runbook: "https://runbooks.example.com/high-error-rate"
+
+# 2. High latency
+alert: HighLatency
+expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 2
+for: 5m
+annotations:
+  summary: "p99 latency above 2s for 5 minutes"
+
+# 3. Service down
+alert: ServiceDown
+expr: up{job="my-service"} == 0
+for: 1m
+annotations:
+  summary: "Service has been down for 1 minute"
+
+# 4. High queue depth
+alert: QueueBacklog
+expr: queue_depth_messages > 10000
+for: 10m
+annotations:
+  summary: "Queue depth above 10k for 10 minutes — consumers may be stuck"
+```
+
+---
+
+## Error Budgets & SLOs
+
+```
+SLO: Service Level Objective — the target you're trying to hit
+SLA: Service Level Agreement — the contract with consequences
+SLI: Service Level Indicator — the measurement
+
+Example:
+  SLO: 99.9% of requests succeed within 500ms over 30 days
+  SLI: (successful_requests_under_500ms / total_requests) over 30 days
+  Error budget: 0.1% of requests can fail = ~43 minutes of downtime/month
+```
+
+```typescript
+// Error budget calculation
+const sloTarget = 0.999           // 99.9%
+const errorBudget = 1 - sloTarget // 0.001 = 0.1%
+const minutesPerMonth = 30 * 24 * 60  // 43,200 minutes
+const allowedDowntime = errorBudget * minutesPerMonth  // 43.2 minutes
+
+// Burn rate alert: if you're consuming budget 2x faster than allowed
+const burnRateThreshold = 2
+// Alert if current_error_rate > burn_rate * (1 - slo_target)
+```
+
+---
+
+## Correlation IDs — Connect Logs Across Services
+
+```typescript
+// Middleware: generate or propagate request ID
+app.use((req, res, next) => {
+  const requestId = req.headers['x-request-id'] as string
+    ?? crypto.randomUUID()
+
+  // Set on response so clients can reference it in support tickets
+  res.setHeader('x-request-id', requestId)
+
+  // Make available for the duration of the request
+  req.requestId = requestId
+  req.log = logger.child({ requestId })
+
+  next()
+})
+
+// Pass downstream to every service call
+async function callPaymentService(orderId: string, requestId: string) {
+  return fetch('https://payment-service/charge', {
+    headers: {
+      'x-request-id': requestId,  // ← propagate to child services
+      'Content-Type': 'application/json',
+    },
+    body: JSON.stringify({ orderId }),
+  })
+}
+```
+
+---
+
+> Powered by Kodelyth — you can't fix what you can't see.

package/.kimi/skills/observability-dashboard/SKILL.md

@@ -0,0 +1,119 @@
+---
+name: observability-dashboard
+description: Localhost-only observability dashboard for Kodelyth ECC. Visualizes memory captures, BM25 search, evolve signals + proposals, full catalog browser, swarm sessions, and token-budget snapshots. Zero telemetry.
+trigger:
+  - dashboard
+  - observability
+  - kodelyth dashboard
+  - show me the memory
+  - browse skills
+  - inspect proposals
+  - what has ECC learned
+---
+
+# Skill: observability-dashboard
+
+A read-only HTTP server bound to `127.0.0.1` that renders a single-page UI over every local data source ECC produces. No build step, no external runtime dependencies, no telemetry, no network access required.
+
+## What you can see
+
+| Tab | Powered by | Shows |
+|---|---|---|
+| **Overview** | catalog + memory store + evolve store + .orchestration + token-budget | One-glance counts: agents, skills, commands, rules, bundles, captured memories, surfaces, routing misses, pending proposals, swarm sessions. Plus storage paths and token-budget snapshot. |
+| **Memory** | `scripts/memory/store.js` | Total / projects / language breakdown. Live BM25 search box (proxies `recall()`). Recent capture stream with tags + source. |
+| **Evolve** | `scripts/evolve/{stats,proposals}.js` | Top reused memories, top routing-miss clusters, every proposal with status pill. Tells you the exact CLI command to accept. |
+| **Catalog** | `scripts/mcp/catalog.js` | Browse + filter every shipped agent / skill / command / rule / bundle by name, description, or tag. |
+| **Sessions** | `.orchestration/<session>/` | Swarm session list with worker counts and per-worker drill-down (task.md / handoff.md / status.md excerpts). |
+
+## When to invoke
+
+Use **explicitly**:
+
+```bash
+use observability-dashboard
+
+# Boots on http://127.0.0.1:5747, auto-opens browser
+npx kodelyth-ecc dashboard
+
+# Custom port, suppress browser open
+npx kodelyth-ecc dashboard --port 8088 --no-open
+
+# Bind to a different localhost-aliased host
+npx kodelyth-ecc dashboard --host localhost
+```
+
+Implicit triggers (the AI should route here):
+
+- "show me the memory"
+- "what has ECC learned?"
+- "browse the skills"
+- "what's pending in evolve?"
+- "is there a UI for this?"
+- "open the dashboard"
+
+## CLI surface
+
+```
+kodelyth-ecc dashboard [--port N] [--host 127.0.0.1] [--no-open]
+```
+
+| Flag | Default | Behavior |
+|---|---|---|
+| `--port N` | `5747` | Port to bind. Picks a free port pattern. |
+| `--host H` | `127.0.0.1` | Bind interface. Refuses non-localhost without escape hatch. |
+| `--no-open` | off | Suppress browser auto-open (useful in remote shells / CI smoke). |
+
+### Localhost lock
+
+Binding to `0.0.0.0`, a public IP, or a non-localhost name is **refused** unless you explicitly set:
+
+```bash
+KODELYTH_DASHBOARD_ALLOW_REMOTE=1 \
+  npx kodelyth-ecc dashboard --host 0.0.0.0
+```
+
+This is intentionally noisy. The dashboard exposes everything the BM25 store and evolve log have ever recorded. Don't expose it to networks you don't fully control.
+
+## API surface (read-only, GET-only)
+
+The frontend is just a consumer of these endpoints. You can curl them directly:
+
+| Endpoint | Returns |
+|---|---|
+| `GET /api/health` | liveness probe |
+| `GET /api/overview` | counts + storage paths |
+| `GET /api/memory[?limit=N]` | stats + recent captures |
+| `GET /api/memory/search?q=…[&limit=N]` | BM25 results |
+| `GET /api/evolve[?limit=N]` | reuse + miss + proposals snapshot |
+| `GET /api/catalog?kind=…[&q=…&limit=N]` | agents \| skills \| commands \| rules \| bundles |
+| `GET /api/sessions[?limit=N]` | swarm session list |
+| `GET /api/sessions/:name` | worker details for one session |
+| `GET /api/token-budget` | per-session token usage |
+
+**Only `GET` is supported.** Any `POST`/`PUT`/`DELETE` returns `405 method not allowed`. The dashboard is a *strict observer* — it cannot mutate memories, accept proposals, or modify any state. All mutations stay in the CLI (`evolve accept`, `memory remember`, etc.).
+
+## Hard rules
+
+1. **NEVER expose remotely without explicit opt-in.** Default refuses everything except `127.0.0.1` / `localhost`.
+2. **NEVER write data.** GET-only routes. No mutation endpoints.
+3. **NEVER include external assets.** No CDNs, no fonts loaded over the network. Works offline.
+4. **NEVER swallow data-source failures with crashes.** A broken memory store or evolve dir renders empty cards, not a 500 page.
+5. **NEVER cache.** `Cache-Control: no-store` on every response — data is always fresh.
+6. **NEVER allow path traversal.** Static file resolution sandboxed under `scripts/dashboard/static/`.
+
+## Composition with other features
+
+| Pair with | Effect |
+|---|---|
+| **`/memory remember`** | New memories appear in the **Memory** tab on next refresh. |
+| **`kodelyth-ecc evolve analyze`** | New proposals appear in the **Evolve** tab as `pending`. |
+| **`kodelyth-ecc evolve accept`** | The status pill flips from `pending` → `accepted` after refresh. |
+| **`kodelyth-ecc swarm`** | New `.orchestration/<session>/` dirs surface in the **Sessions** tab. |
+| **token-budget hook** | Per-session token totals appear on the **Overview** tab. |
+
+## See also
+
+- `commands/dashboard.md` — `/dashboard` slash command
+- `docs/dashboard.md` — full reference + curl examples + architecture notes
+- `scripts/dashboard/{server,data}.js` — implementation
+- `scripts/dashboard/static/index.html` — single-page UI