@evoclock/pi-agentic-driver 0.9.2 → 0.9.4

package/janus/README.md

@@ -0,0 +1,196 @@
+# janus — Jev evaluation wrapper service
+
+janus is the local wrapper service for the Jev decision layer (design of
+record: `.private-planning/JEV_DECISION_LAYER_DESIGN.md` §5–§7, Revision 2.1).
+It wraps the official `@typesafe-ai/sdk` (TypeSafe Direct — the sole
+provider; the Vercel AI Gateway and merge gateway were dropped by owner
+decision) and exposes one evaluation endpoint plus health/readiness/version
+probes on loopback.
+
+## What janus does
+
+- `POST /evaluate` — evaluates typed questions (boolean / choice / score)
+  against one shared state via TypeSafe Direct (`typesafe-ai`, model `jev-latest`).
+- `GET /healthz` — liveness.
+- `GET /readyz` — readiness (gateway key resolvable from the Keychain).
+- `GET /version` — janus version, AI SDK version, schema versions, allowlist.
+
+Guarantees:
+
+- **Model allowlist** (server-side config; request-level model overrides are
+  rejected with `model_unauthorized`).
+- **Loopback only** — binds `127.0.0.1` and nothing else. No auth beyond
+  loopback; it is a local tool. There is deliberately no bind-address option.
+- **Single-flight** — identical requests (same evalInputDigest) coalesce;
+  distinct in-flight evaluations are capped (default 4) and rejected with
+  `busy`.
+- **30k-token request budget** — oversized requests are rejected with
+  `state_too_large`, never truncated silently.
+- **Redaction** — a general secrets/PII pass runs over the state before any
+  external call.
+- **Audit log** — append-only JSONL of request id, timestamps, question
+  names, verdict probabilities, and token usage. **State contents never
+  reach the audit log.** 90-day default retention, size-capped with
+  oldest-first rotation.
+- **Daily cost breaker** — token/cost ceilings in config; on exceed,
+  `/evaluate` returns `busy` with `budget_exhausted` detail until the UTC
+  day window resets.
+- **Timeouts** — 30s service-side cap; callers may pass a shorter
+  per-request `timeoutMs` (capped at 30s).
+
+## Provisioning the Keychain item
+
+The TypeSafe Direct key is read from the macOS Keychain at runtime — janus
+does not rely on inherited environment variables. The `typesafe-ai` /
+`Typesafe AI` generic-password item does not exist by default; create it
+once (interactive provisioning):
+
+```sh
+security add-generic-password -U -a typesafe-ai -s "Typesafe AI" -w
+# prompts for the secret; nothing is echoed
+```
+
+janus reads it with `security find-generic-password -w` under guarded
+capture: stdout is piped directly into memory, never echoed, never logged,
+never passed to spawned subprocesses. If the item is missing, janus returns
+`model_unauthorized` with `detail.key_unavailable: true` and `GET /readyz`
+reports 503 with the provisioning hint.
+
+## Running
+
+Dev (from `janus/`):
+
+```sh
+npx tsx janus/server.ts          # listens on 127.0.0.1:8787
+```
+
+Config is read from `~/.config/typesafe-ai/jev/janus-config.json`
+(override with `JANUS_CONFIG_PATH`). Defaults work with no file:
+
+```json
+{
+  "port": 8787,
+  "modelAllowlist": ["typesafe-ai/jev"],
+  "defaultModel": "typesafe-ai/jev",
+  "maxConcurrent": 4,
+  "dailyTokenCeiling": 2000000,
+  "dailyCostCeilingUsd": 10,
+  "audit": { "path": "~/.local/share/janus/audit", "maxBytes": 10485760, "retentionDays": 90 }
+}
+```
+
+launchd — template at `janus/user.janus.plist`. The template is a starting
+point: every user-specific value (the label, the checkout path, log
+locations) is marked ADJUST in the file. Copy it out of the repository,
+adjust it, then load it:
+
+```sh
+cp janus/user.janus.plist ~/Library/LaunchAgents/
+```
+
+```sh
+launchctl load ~/Library/LaunchAgents/user.janus.plist
+```
+
+Tests (no live gateway calls — the SDK is mocked):
+
+```sh
+cd janus && npm test
+```
+
+## Endpoint contract
+
+### POST /evaluate
+
+Request:
+
+```json
+{
+  "state": "string | object | array (message histories pass directly)",
+  "questions": [
+    { "type": "boolean", "question": "Was a refund issued?", "threshold": 0.9 },
+    { "type": "choice", "question": "Route this ticket", "choices": ["billing", "tech", "other"] },
+    { "type": "score", "question": "Rate this PR" }
+  ],
+  "model": "optional; must be on the server allowlist, overrides rejected",
+  "timeoutMs": 10000
+}
+```
+
+Question schema v1 (§6), enforced atomically — any malformed, absent, or
+contradictory element fails the whole request with `validation_error`:
+
+- **boolean** — `threshold` optional, default `0.9`, comparison
+  `probability >= threshold`. Optional `criteria: {true?, false?}` labels.
+- **choice** — `choices` array REQUIRED; the verdict's `choice` is guaranteed
+  to be a member.
+- **score** — numeric 0..5, one decimal. Ordinal vocabularies must be
+  **choice** questions, not scores.
+
+§6 → SDK mapping (verified against ai@7.0.106 at launch step 1): the SDK's
+`experimental_evaluate` has **no top-level `criteria` argument**; the
+candidate set lives inside each question as a REQUIRED `criteria` field
+(choice: `Record<option, description | null>`; score: ordered level array;
+boolean: optional `{true?, false?}`). janus accepts §6's `choices` array at
+the transport layer and maps it to `criteria: {a: null, b: null, ...}`. The
+score vocabulary is passed as the ordered level array `["0".."5"]`.
+
+Success response (200):
+
+```json
+{
+  "request_id": "uuid",
+  "verdict": {
+    "Was a refund issued?": { "type": "boolean", "probability": 0.97, "passed": true },
+    "Route this ticket": { "type": "choice", "choice": "billing", "probabilities": {} },
+    "Rate this PR": { "type": "score", "score": 3.0 }
+  },
+  "usage": { "inputTokens": 0, "outputTokens": 0, "totalTokens": 0 },
+  "model": "typesafe-ai/jev",
+  "digest": "sha256 evalInputDigest"
+}
+```
+
+"Model returned false" is a **200** with `passed: false` / low probability —
+not an error.
+
+### Error taxonomy
+
+All errors: `{ "error": code, "message": ..., "request_id": ..., "detail"?: {...} }`
+
+| Code | HTTP | Meaning |
+|---|---|---|
+| `validation_error` | 400 | bad request (schema, JSON, unknown route) |
+| `state_too_large` | 413 | over the 30k-token budget; never truncated |
+| `unknown_question` | 400 | reserved for router/schema-version mismatches |
+| `model_unauthorized` | 403/502/503 | not on allowlist, credential rejected, or key unavailable (`detail.key_unavailable`) |
+| `gateway_unavailable` | 503 | gateway unreachable / model missing / rate limit |
+| `timeout` | 504 | evaluation exceeded the budget |
+| `busy` | 429 | concurrency cap or `detail.budget_exhausted` |
+| `degraded` | 503 | internal error |
+
+Callers distinguish **"model returned false"** (200, low probability) from
+**"no verdict"** (`timeout` / `gateway_unavailable`) from **"bad request"**
+(`validation_error`).
+
+## Layout
+
+```
+janus/
+  janus/
+    config.ts          config load/validate (allowlist, ceilings, retention)
+    schema.ts          §6 question schema v1 enforcement (atomic fail-closed)
+    verdict.ts         verdict enforcement (membership, ranges, thresholds)
+    budget.ts          token estimation, 30k budget, evalInputDigest
+                       redaction pass: shared library typesafe-secure
+                       lib/redaction (MIT), wired via a file: dependency
+    keychain.ts        guarded `security find-generic-password -w` capture
+    single_flight.ts   digest coalescing + concurrency cap
+    budget_breaker.ts  daily token/cost breaker
+    audit.ts           append-only JSONL audit log, rotation + retention
+    sdk.ts             THE ONLY AI SDK import site (pinned versions)
+    evaluate.ts        /evaluate pipeline + SDK error classification
+    server.ts          HTTP surface, loopback bind, entrypoint
+  tests/               node:test suites (mocked SDK, no live calls)
+  user.janus.plist  launchd template
+```

package/janus/janus/audit.ts

@@ -0,0 +1,213 @@
+// SPDX-FileCopyrightText: 2026 Julen Gamboa <j.a.r.gamboa@gmail.com>
+// SPDX-License-Identifier: AGPL-3.0-only
+
+// Audit log (§5, M9 + round-2 retention rule): append-only local log of
+// request id, timestamps, question NAMES (non-sensitive identifiers, never
+// question text or state contents), verdict probabilities, and token usage.
+// 90-day default retention (config-adjustable), size-capped with
+// oldest-first rotation.
+//
+// Retention (review fix #7): enforced for the ACTIVE file as well as rotated
+// files. On start, after each rotation, and on a periodic sweep, records
+// older than the retention window are purged from the active file by
+// rewriting it without them, and rotated files older than the window are
+// deleted. Records older than retention never survive indefinitely.
+
+import {
+  closeSync,
+  existsSync,
+  fstatSync,
+  mkdirSync,
+  openSync,
+  readSync,
+  renameSync,
+  statSync,
+  unlinkSync,
+  writeSync,
+} from "node:fs";
+import { dirname } from "node:path";
+
+export interface AuditEntry {
+  requestId: string;
+  timestamp: string;
+  event: "evaluate" | "evaluate_result" | "evaluate_error";
+  /** Non-sensitive question identifiers — never question text. */
+  questionNames: string[];
+  verdictProbabilities?: Record<string, number>;
+  tokenUsage?: { inputTokens?: number | undefined; outputTokens?: number | undefined; totalTokens?: number | undefined };
+  errorCode?: string;
+  durationMs?: number;
+  coalesced?: boolean;
+  /** How many provider attempts this request consumed (1 = no failover). */
+  providerAttempts?: number;
+  /** Provider id that produced the result (or last attempted). */
+  provider?: string;
+}
+
+const MAX_ROTATED_FILES = 5;
+/** How often the active-file retention sweep runs while the log is open. */
+const RETENTION_SWEEP_INTERVAL_MS = 60 * 60 * 1000;
+
+function entryTimestampMs(line: string): number | undefined {
+  try {
+    const parsed = JSON.parse(line) as { timestamp?: unknown };
+    if (typeof parsed.timestamp !== "string") return undefined;
+    const ms = Date.parse(parsed.timestamp);
+    return Number.isFinite(ms) ? ms : undefined;
+  } catch {
+    return undefined;
+  }
+}
+
+export class AuditLog {
+  private fd: number | null = null;
+  private readonly activePath: string;
+  private lastSweepAt = 0;
+
+  constructor(
+    private readonly basePath: string,
+    private readonly maxBytes: number,
+    private readonly retentionDays: number,
+    private readonly now: () => Date = () => new Date(),
+  ) {
+    this.activePath = basePath === "stdout" ? "stdout" : `${basePath}.jsonl`;
+  }
+
+  /** Open (or reopen) the log and apply retention. Call once at startup. */
+  start(): void {
+    if (this.activePath === "stdout") return;
+    mkdirSync(dirname(this.activePath), { recursive: true });
+    this.applyRetention();
+    this.openFd();
+  }
+
+  stop(): void {
+    if (this.fd !== null) {
+      closeSync(this.fd);
+      this.fd = null;
+    }
+  }
+
+  private openFd(): void {
+    this.fd = openSync(this.activePath, "a");
+  }
+
+  /**
+   * Purge records older than the retention window from the ACTIVE file by
+   * rewriting it without them. Rotated-file deletion is handled separately
+   * by mtime in applyRetention(). This guarantees active-file records do not
+   * survive past retention (review fix #7).
+   */
+  private sweepActiveFile(): void {
+    if (this.activePath === "stdout") return;
+    const cutoff = this.now().getTime() - this.retentionDays * 24 * 60 * 60 * 1000;
+
+    let size = 0;
+    try {
+      size = statSync(this.activePath).size;
+    } catch {
+      return;
+    }
+    if (size === 0) return;
+
+    // Read the whole active file (bounded by maxBytes + slack), filter, and
+    // rewrite in place if anything expired.
+    let raw: string;
+    const fd = openSync(this.activePath, "r");
+    try {
+      const buffer = Buffer.alloc(size);
+      const read = readSync(fd, buffer, 0, size, 0);
+      raw = buffer.toString("utf8", 0, read);
+    } finally {
+      closeSync(fd);
+    }
+
+    const lines = raw.split("\n");
+    const kept: string[] = [];
+    for (const line of lines) {
+      if (line.length === 0) continue;
+      const ts = entryTimestampMs(line);
+      if (ts === undefined || ts >= cutoff) kept.push(line);
+    }
+    if (kept.length === lines.filter((l) => l.length > 0).length) return; // nothing expired
+
+    const replacement = `${kept.join("\n")}\n`;
+    // Rewrite via a temp file + rename for atomicity, then reopen for append.
+    const tempPath = `${this.activePath}.sweep`;
+    const tempFd = openSync(tempPath, "w");
+    try {
+      writeSync(tempFd, replacement);
+    } finally {
+      closeSync(tempFd);
+    }
+    if (this.fd !== null) {
+      closeSync(this.fd);
+      this.fd = null;
+    }
+    renameSync(tempPath, this.activePath);
+    this.openFd();
+  }
+
+  private applyRetention(): void {
+    const cutoff = this.now().getTime() - this.retentionDays * 24 * 60 * 60 * 1000;
+    for (let i = 1; i <= MAX_ROTATED_FILES; i += 1) {
+      const rotated = `${this.activePath}.${i}`;
+      if (!existsSync(rotated)) continue;
+      try {
+        if (statSync(rotated).mtimeMs < cutoff) unlinkSync(rotated);
+      } catch {
+        // A missing file mid-scan is fine; retention is best-effort.
+      }
+    }
+    this.sweepActiveFile();
+    this.lastSweepAt = this.now().getTime();
+  }
+
+  private rotateIfNeeded(): void {
+    if (this.activePath === "stdout" || this.fd === null) return;
+    let size = 0;
+    try {
+      size = fstatSync(this.fd).size;
+    } catch {
+      return;
+    }
+    if (size < this.maxBytes) return;
+
+    // Oldest-first rotation: .4→.5, .3→.4, … .1→.2, active→.1.
+    for (let i = MAX_ROTATED_FILES - 1; i >= 1; i -= 1) {
+      const from = `${this.activePath}.${i}`;
+      const to = `${this.activePath}.${i + 1}`;
+      if (existsSync(from)) {
+        try {
+          unlinkSync(to);
+        } catch {
+          /* first shift may have no target */
+        }
+        renameSync(from, to);
+      }
+    }
+    closeSync(this.fd);
+    this.fd = null;
+    renameSync(this.activePath, `${this.activePath}.1`);
+    this.applyRetention();
+    this.openFd();
+  }
+
+  append(entry: AuditEntry): void {
+    const line = `${JSON.stringify(entry)}\n`;
+    if (this.activePath === "stdout") {
+      process.stdout.write(line);
+      return;
+    }
+    if (this.fd === null) this.openFd();
+    // eslint-disable-next-line @typescript-eslint/no-non-null-assertion
+    writeSync(this.fd!, line);
+    this.rotateIfNeeded();
+    // Periodic active-file retention sweep (review fix #7).
+    const nowMs = this.now().getTime();
+    if (nowMs - this.lastSweepAt >= RETENTION_SWEEP_INTERVAL_MS) {
+      this.sweepActiveFile();
+      this.lastSweepAt = nowMs;
+    }
+  }
+}

package/janus/janus/budget.ts

@@ -0,0 +1,101 @@
+// SPDX-FileCopyrightText: 2026 Julen Gamboa <j.a.r.gamboa@gmail.com>
+// SPDX-License-Identifier: AGPL-3.0-only
+
+// Budget accounting (§2, §5): the authoritative request budget is 30k tokens
+// for state + questions + framing. Anything exceeding the budget after
+// shaping is rejected with state_too_large — never truncated silently.
+//
+// Conservative token bound (review fix #3): the estimate is byte-based with
+// an explicit worst-case floor of 1 token per 2 UTF-8 bytes. This dominates
+// chars/4 for every encoding shape:
+//   - ASCII prose:        1 byte/char  → 2 bytes/token  (≥ chars/4)
+//   - CJK / emoji:        3–4 bytes/char → 1.5–2 bytes/token (chars/4 would
+//     undercount a 4-byte emoji as 0.25 tokens; this bound charges ≥ 2)
+//   - dense/base64 JSON:  1 byte/char → 2 bytes/token (chars/4 undercounts
+//     by 2×; dense JSON tokenizes at ~1–2 bytes/token in practice)
+// The bound can over-reject (e.g. very compressive scripts) but never
+// materially undercounts. It runs BEFORE any external call so oversized
+// requests never reach a provider.
+
+import { createHash } from "node:crypto";
+import { stateTooLarge } from "./errors.js";
+import type { EnforcedQuestion } from "./schema.js";
+
+/** Conservative worst-case bytes per token (finding #3): 1 token per 2 UTF-8 bytes. */
+export const BYTES_PER_TOKEN_CONSERVATIVE = 2;
+
+/** Canonical JSON with object keys sorted (stable digests). */
+export function canonicalJson(value: unknown): string {
+  return JSON.stringify(canonicalize(value));
+}
+
+function canonicalize(value: unknown): unknown {
+  if (Array.isArray(value)) return value.map(canonicalize);
+  if (value !== null && typeof value === "object") {
+    const sorted: Record<string, unknown> = {};
+    for (const k of Object.keys(value).sort()) {
+      sorted[k] = canonicalize((value as Record<string, unknown>)[k]);
+    }
+    return sorted;
+  }
+  return value;
+}
+
+/**
+ * Conservative token estimate for a string: UTF-8 bytes / 2, rounded up.
+ * Dominates chars/4 for CJK, emoji, and dense encoded content; can
+ * over-reject but never materially undercounts.
+ */
+export function estimateStringTokens(text: string): number {
+  return Math.ceil(Buffer.byteLength(text, "utf8") / BYTES_PER_TOKEN_CONSERVATIVE);
+}
+
+export interface RequestSize {
+  estimatedTokens: number;
+  stateBytes: number;
+}
+
+export function estimateRequestTokens(
+  state: unknown,
+  questions: readonly EnforcedQuestion[],
+  budgetTokens: number,
+  maxStateBytes: number,
+): RequestSize {
+  const stateJson = canonicalJson(state);
+  const stateBytes = Buffer.byteLength(stateJson, "utf8");
+  if (stateBytes > maxStateBytes) {
+    throw stateTooLarge(
+      `state is ${stateBytes} bytes; the hard byte ceiling is ${maxStateBytes} — shape to digests/summaries and retry`,
+      { stateBytes, maxStateBytes },
+    );
+  }
+
+  let tokens = estimateStringTokens(stateJson);
+  for (const question of questions) {
+    tokens += estimateStringTokens(question.question);
+    tokens += estimateStringTokens(question.instructions);
+    if (question.kind === "choice") {
+      tokens += question.choices.reduce((sum, c) => sum + estimateStringTokens(c), 0);
+    }
+  }
+
+  if (tokens > budgetTokens) {
+    throw stateTooLarge(
+      `estimated request size ${tokens} tokens exceeds the ${budgetTokens}-token budget ` +
+        `(state + questions + framing); never truncated silently — shape the state down`,
+      { estimatedTokens: tokens, budgetTokens },
+    );
+  }
+
+  return { estimatedTokens: tokens, stateBytes };
+}
+
+/** evalInputDigest (M9): stable digest of the exact evaluation input. */
+export function evalInputDigest(
+  state: unknown,
+  questions: readonly EnforcedQuestion[],
+): string {
+  const hash = createHash("sha256");
+  hash.update(canonicalJson({ state, questions }));
+  return hash.digest("hex");
+}

package/janus/janus/budget_breaker.ts

@@ -0,0 +1,105 @@
+// SPDX-FileCopyrightText: 2026 Julen Gamboa <j.a.r.gamboa@gmail.com>
+// SPDX-License-Identifier: AGPL-3.0-only
+
+// Daily cost breaker (§5, M10): a daily token/cost ceiling in config; on
+// exceed, /evaluate returns busy with budget_exhausted detail until the
+// window resets. Windows are UTC calendar days (the audit log uses the same
+// day key). Usage arrives from the SDK result after each evaluation; the
+// breaker is updated by the evaluate path.
+
+import { busyError } from "./errors.js";
+
+export interface UsageRecord {
+  inputTokens?: number | undefined;
+  outputTokens?: number | undefined;
+  totalTokens?: number | undefined;
+  costUsd?: number | undefined;
+}
+
+export interface BudgetStatus {
+  dayKey: string;
+  tokensUsed: number;
+  costUsdUsed: number;
+  tokenCeiling: number;
+  costCeilingUsd: number;
+  exhausted: boolean;
+}
+
+export class DailyBudget {
+  private dayKey = "";
+  private tokensUsed = 0;
+  private costUsdUsed = 0;
+
+  constructor(
+    private readonly tokenCeiling: number,
+    private readonly costCeilingUsd: number,
+    private readonly costPerMtokenUsd: number,
+    private readonly now: () => Date = () => new Date(),
+  ) {}
+
+  static dayKeyFor(date: Date): string {
+    return date.toISOString().slice(0, 10);
+  }
+
+  private rollIfNeeded(): void {
+    const key = DailyBudget.dayKeyFor(this.now());
+    if (key !== this.dayKey) {
+      this.dayKey = key;
+      this.tokensUsed = 0;
+      this.costUsdUsed = 0;
+    }
+  }
+
+  /**
+   * Record usage after an evaluation completes (success or failure after the
+   * gateway accepted the call — retries consume budget too).
+   */
+  record(usage: UsageRecord): void {
+    this.rollIfNeeded();
+    const total =
+      usage.totalTokens ??
+      (usage.inputTokens !== undefined || usage.outputTokens !== undefined
+        ? (usage.inputTokens ?? 0) + (usage.outputTokens ?? 0)
+        : 0);
+    this.tokensUsed += total;
+    this.costUsdUsed +=
+      usage.costUsd ?? (total / 1_000_000) * this.costPerMtokenUsd;
+  }
+
+  /** Check before dispatch. Throws busy/budget_exhausted when over either ceiling. */
+  assertWithinBudget(): void {
+    this.rollIfNeeded();
+    if (this.tokenCeiling > 0 && this.tokensUsed >= this.tokenCeiling) {
+      throw busyError(
+        `daily token ceiling of ${this.tokenCeiling} reached; resets at 00:00 UTC`,
+        { budget_exhausted: { kind: "tokens", dayKey: this.dayKey, used: this.tokensUsed } },
+      );
+    }
+    if (this.costCeilingUsd > 0 && this.costUsdUsed >= this.costCeilingUsd) {
+      throw busyError(
+        `daily cost ceiling of $${this.costCeilingUsd} reached; resets at 00:00 UTC`,
+        {
+          budget_exhausted: {
+            kind: "cost",
+            dayKey: this.dayKey,
+            usedUsd: Number(this.costUsdUsed.toFixed(4)),
+          },
+        },
+      );
+    }
+  }
+
+  status(): BudgetStatus {
+    this.rollIfNeeded();
+    return {
+      dayKey: this.dayKey,
+      tokensUsed: this.tokensUsed,
+      costUsdUsed: Number(this.costUsdUsed.toFixed(4)),
+      tokenCeiling: this.tokenCeiling,
+      costCeilingUsd: this.costCeilingUsd,
+      exhausted:
+        (this.tokenCeiling > 0 && this.tokensUsed >= this.tokenCeiling) ||
+        (this.costCeilingUsd > 0 && this.costUsdUsed >= this.costCeilingUsd),
+    };
+  }
+}