@evoclock/pi-agentic-driver 0.9.2 → 0.9.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +78 -54
- package/janus/README.draft.md +210 -0
- package/janus/README.md +196 -0
- package/janus/janus/audit.ts +213 -0
- package/janus/janus/budget.ts +101 -0
- package/janus/janus/budget_breaker.ts +105 -0
- package/janus/janus/config.ts +332 -0
- package/janus/janus/errors.ts +90 -0
- package/janus/janus/evaluate.ts +230 -0
- package/janus/janus/keychain.ts +119 -0
- package/janus/janus/providers.ts +544 -0
- package/janus/janus/schema.ts +344 -0
- package/janus/janus/server.ts +267 -0
- package/janus/janus/single_flight.ts +99 -0
- package/janus/janus/verdict.ts +192 -0
- package/janus/package-lock.json +616 -0
- package/janus/package.json +29 -0
- package/janus/tests/accounting.test.ts +229 -0
- package/janus/tests/allowlist.test.ts +97 -0
- package/janus/tests/config.test.ts +151 -0
- package/janus/tests/errors.test.ts +147 -0
- package/janus/tests/evaluate_pipeline.test.ts +358 -0
- package/janus/tests/helpers.ts +217 -0
- package/janus/tests/providers.test.ts +292 -0
- package/janus/tests/redaction_budget.test.ts +227 -0
- package/janus/tests/schema.test.ts +356 -0
- package/janus/tests/server.test.ts +288 -0
- package/janus/tests/single_flight.test.ts +84 -0
- package/janus/tsconfig.json +19 -0
- package/janus/user.janus.plist +51 -0
- package/package.json +3 -2
- package/scripts/enforcement/router_schemas_pi.js +63 -13
package/README.md
CHANGED
|
@@ -53,6 +53,8 @@ proofs for agentic workflows.
|
|
|
53
53
|
| `agentic_kanban_board_update` | Moves, closes, flags, edits, or removes cards on the board, always recording who authorized the change. | shipped |
|
|
54
54
|
| `agentic_kanban_board_dispatch` | Claims an eligible board card for automated contained work and creates its assignment envelope. | shipped |
|
|
55
55
|
| `agentic_kanban_pulse` | Checks for ready work and available capacity, then starts assignments under the board policy. | shipped |
|
|
56
|
+
| Router | Capacity/policy routing for Board Pulse: seats with scope statements, tunable policy values, and a binding dispatch gate. | shipped |
|
|
57
|
+
| Janus (bundled service) | Local Jev evaluation service on loopback: semantic ranking, the dispatch-gate judgment, redaction before every external call. See `janus/README.md`. | shipped |
|
|
56
58
|
|
|
57
59
|
**Status: active development and testing.** Each extension ships only after
|
|
58
60
|
it passes fixture-based acceptance, native tests, live-session checks, and
|
|
@@ -102,6 +104,82 @@ only, not a runtime dependency).
|
|
|
102
104
|
- **inventory refresh.** Git-aware codebase inventory regeneration with
|
|
103
105
|
verification receipts, so prior-art matching stays honest.
|
|
104
106
|
|
|
107
|
+
<details>
|
|
108
|
+
<summary><strong>router, capacity/policy routing with a binding dispatch gate</strong> <em>(released, 0.9.2)</em></summary>
|
|
109
|
+
|
|
110
|
+
Board Pulse selects a route seat through the capacity/policy router. The
|
|
111
|
+
router reads a closed configuration file and a local-first SQLite file for
|
|
112
|
+
operational state (reservations, observations, route-decision audit).
|
|
113
|
+
|
|
114
|
+
Two router features call **Janus** — a separate local service, bundled under
|
|
115
|
+
`janus/`, that is the only component allowed to talk to the TypeSafe Jev
|
|
116
|
+
models:
|
|
117
|
+
|
|
118
|
+
- **Semantic ranking** (optional): Janus ranks up to eight eligible seats;
|
|
119
|
+
on any Janus failure the router falls back to preference order.
|
|
120
|
+
- **The binding dispatch gate**: before a task is claimed, Janus judges
|
|
121
|
+
whether the task fits the selected seat. No verdict, no dispatch — the
|
|
122
|
+
gate fails closed when Janus is unreachable.
|
|
123
|
+
|
|
124
|
+
Janus binds loopback only (`127.0.0.1:8787`), reads its API key from the
|
|
125
|
+
macOS Keychain at runtime, redacts state before every external call, and
|
|
126
|
+
coalesces identical requests. See `janus/README.md` for the endpoint
|
|
127
|
+
contract, Keychain provisioning, and configuration.
|
|
128
|
+
|
|
129
|
+
- **Repository defaults** ship at `.agentic-driver/router.defaults.json`. The
|
|
130
|
+
template contains no accounts and one disabled placeholder seat, so it loads
|
|
131
|
+
out of the box and schedules nothing until you enable a real seat.
|
|
132
|
+
- **User profile (precedence over defaults):**
|
|
133
|
+
`~/.config/agentic-driver/router/profile.json`, or the path in the
|
|
134
|
+
`AGENTIC_DRIVER_ROUTER_PROFILE` environment variable. A profile section
|
|
135
|
+
replaces the matching defaults section wholesale; the merged config must
|
|
136
|
+
validate or the router refuses to produce decisions (fails closed).
|
|
137
|
+
- **Operational state:** `~/.local/share/agentic-driver/router/state.db`, or
|
|
138
|
+
`AGENTIC_DRIVER_ROUTER_DB`. This store is operational state only. The
|
|
139
|
+
authenticated claims file and the board writer remain the only dispatch
|
|
140
|
+
authority; the system repairs SQLite from them, never the reverse.
|
|
141
|
+
- **Runtime requirement:** the operational store uses the built-in
|
|
142
|
+
`node:sqlite` module. You need Node.js 22.5 or newer.
|
|
143
|
+
|
|
144
|
+
<details>
|
|
145
|
+
<summary><strong>Tunable configuration values</strong> <em>(all keys required, defaults are conservative starting values)</em></summary>
|
|
146
|
+
|
|
147
|
+
Every numeric policy in the router is a configuration key — never a
|
|
148
|
+
constant in code. All keys are **required**. The repository defaults below
|
|
149
|
+
are conservative starting values. Your profile overrides them. A missing key
|
|
150
|
+
is a configuration validation failure: the router fails closed rather than
|
|
151
|
+
guess. Changing a value participates in the configuration digest, so cached
|
|
152
|
+
routing decisions computed under the old values no longer apply.
|
|
153
|
+
|
|
154
|
+
| Section | Key | Default | What it controls |
|
|
155
|
+
|---|---|---|---|
|
|
156
|
+
| `eligibility.reserve` | `floorPercent` | 40 | Share of every quota window kept in reserve (per account, per window). Headroom for an unexpected burst and your next interactive session, without idling the account. |
|
|
157
|
+
| `eligibility.reserve` | `scope` | `account-window` | How widely the reserve floor applies (per account per window at launch). |
|
|
158
|
+
| `eligibility.reserve` | `coldStartFraction` | 0.25 | Consumption assumed for a seat with too little history: deliberately generous, so a mis-estimate fails eligibility instead of draining the account. |
|
|
159
|
+
| `eligibility.reserve` | `estimateSamples` | 20 | How many recent dispatches the consumption estimate averages: enough to smooth outliers, quick enough to track a model change. |
|
|
160
|
+
| `eligibility.reserve` | `estimateMinSamples` | 5 | Below this many observations, the estimator treats the average as noise and uses the cold-start value instead. |
|
|
161
|
+
| `eligibility.reserve` | `estimateOutlierSigma` | 3 | Outlier cut for the estimate: one unusually slow dispatch cannot inflate the average, real tail latency is kept. |
|
|
162
|
+
| `eligibility.reserve` | `ownerInteractiveOverride` | false | Whether your own interactive dispatches bypass the reserve rule (they do not, at launch). |
|
|
163
|
+
| `ranking` | `janusUrl` | `http://127.0.0.1:8787` | The local Janus evaluation service endpoint used for semantic ranking. |
|
|
164
|
+
| `ranking` | `maxCandidates` | 8 | The router ranks at most eight eligible seats per request; preference order trims the list first. |
|
|
165
|
+
| `ranking` | `fallback` | `preference-order` | What happens when ranking is unavailable: dispatch falls back to your preference order and never blocks. |
|
|
166
|
+
| `gate` | `threshold` | 0.9 | The dispatch gate's confidence threshold. A task is only dispatched when the evaluation clears it. Lower it only with evidence from your own audit log. |
|
|
167
|
+
| `gate` | `timeoutMs` | 8000 | How long the gate waits for an evaluation. On timeout there is no verdict, and the gate blocks dispatch (fail closed). |
|
|
168
|
+
| `gate` | `requestBudgetTokens` | 30000 | Maximum size of a gate evaluation request. Normal evaluations use under a thousand tokens. |
|
|
169
|
+
| seat record | `maxConcurrency` | 4 | How many workers one seat may run at once. The default suits a typical single machine or hosted seat. Tune it per seat: a cluster takes more, a constrained subscription endpoint may take less. |
|
|
170
|
+
| automation policy | `maxConcurrent` | 4 | How many workers may run at once across all seats. The default is a conservative start. Tune it to the capacity you actually have, and raise it when you add seats or hardware. |
|
|
171
|
+
|
|
172
|
+
Seats are named for infrastructure, not models: `dgx-spark`,
|
|
173
|
+
`dgx-spark-cluster`, `mac-studio`, `mac-studio-cluster`, `strix-halo`,
|
|
174
|
+
`strix-halo-cluster`, `merge-gateway-worker`. The model is a mutable field on
|
|
175
|
+
the seat, so changing models never renumbers your configuration. Each seat
|
|
176
|
+
also carries `scopeStatements`: plain-language descriptions of what the seat
|
|
177
|
+
is for. The dispatch gate reads them when judging whether a task fits.
|
|
178
|
+
|
|
179
|
+
</details>
|
|
180
|
+
|
|
181
|
+
</details>
|
|
182
|
+
|
|
105
183
|
## Multi-agent communication
|
|
106
184
|
|
|
107
185
|
*Extensions for controlled coordination between agents.*
|
|
@@ -580,60 +658,6 @@ The package includes the `herdr-communication.ts`, `herdr-lifecycle.ts`, and
|
|
|
580
658
|
disable individual resources from an installed package. You are not required
|
|
581
659
|
to use every extension.
|
|
582
660
|
|
|
583
|
-
## Router configuration
|
|
584
|
-
|
|
585
|
-
Board Pulse selects a route seat through the capacity/policy router. The
|
|
586
|
-
router reads a closed configuration file and a local-first SQLite file for
|
|
587
|
-
operational state (reservations, observations, route-decision audit).
|
|
588
|
-
|
|
589
|
-
- **Repository defaults** ship at `.agentic-driver/router.defaults.json`. The
|
|
590
|
-
template contains no accounts and one disabled placeholder seat, so it loads
|
|
591
|
-
out of the box and schedules nothing until you enable a real seat.
|
|
592
|
-
- **User profile (precedence over defaults):**
|
|
593
|
-
`~/.config/agentic-driver/router/profile.json`, or the path in the
|
|
594
|
-
`AGENTIC_DRIVER_ROUTER_PROFILE` environment variable. A profile section
|
|
595
|
-
replaces the matching defaults section wholesale; the merged config must
|
|
596
|
-
validate or the router refuses to produce decisions (fails closed).
|
|
597
|
-
- **Operational state:** `~/.local/share/agentic-driver/router/state.db`, or
|
|
598
|
-
`AGENTIC_DRIVER_ROUTER_DB`. This store is operational state only. The
|
|
599
|
-
authenticated claims file and the board writer remain the only dispatch
|
|
600
|
-
authority, and SQLite is repaired from them, never the reverse.
|
|
601
|
-
- **Runtime requirement:** the operational store uses the built-in
|
|
602
|
-
`node:sqlite` module. Node.js 22.5 or newer is required.
|
|
603
|
-
|
|
604
|
-
### Tunable configuration values
|
|
605
|
-
|
|
606
|
-
Every numeric policy in the router is a configuration key — never a
|
|
607
|
-
constant in code. All keys are **required**. The repository defaults below
|
|
608
|
-
are conservative starting values. Your profile overrides them. A missing key
|
|
609
|
-
is a configuration validation failure: the router fails closed rather than
|
|
610
|
-
guess. Changing a value participates in the configuration digest, so cached
|
|
611
|
-
routing decisions computed under the old values no longer apply.
|
|
612
|
-
|
|
613
|
-
| Section | Key | Default | What it controls |
|
|
614
|
-
|---|---|---|---|
|
|
615
|
-
| `eligibility.reserve` | `floorPercent` | 40 | Share of every quota window kept in reserve (per account, per window). Headroom for an unexpected burst and your next interactive session, without idling the account. |
|
|
616
|
-
| `eligibility.reserve` | `scope` | `account-window` | How widely the reserve floor applies (per account per window at launch). |
|
|
617
|
-
| `eligibility.reserve` | `coldStartFraction` | 0.25 | Consumption assumed for a seat with too little history: deliberately generous, so a mis-estimate fails eligibility instead of draining the account. |
|
|
618
|
-
| `eligibility.reserve` | `estimateSamples` | 20 | How many recent dispatches the consumption estimate averages: enough to smooth outliers, quick enough to track a model change. |
|
|
619
|
-
| `eligibility.reserve` | `estimateMinSamples` | 5 | Below this many observations the average is treated as noise and the cold-start value is used instead. |
|
|
620
|
-
| `eligibility.reserve` | `estimateOutlierSigma` | 3 | Outlier cut for the estimate: one unusually slow dispatch cannot inflate the average, real tail latency is kept. |
|
|
621
|
-
| `eligibility.reserve` | `ownerInteractiveOverride` | false | Whether your own interactive dispatches bypass the reserve rule (they do not, at launch). |
|
|
622
|
-
| `ranking` | `janusUrl` | `http://127.0.0.1:8787` | The local janus evaluation service endpoint used for semantic ranking. |
|
|
623
|
-
| `ranking` | `maxCandidates` | 8 | At most eight eligible seats are ranked per request; preference order trims first. |
|
|
624
|
-
| `ranking` | `fallback` | `preference-order` | What happens when ranking is unavailable: dispatch falls back to your preference order and never blocks. |
|
|
625
|
-
| `gate` | `threshold` | 0.9 | The dispatch gate's confidence threshold. A task is only dispatched when the evaluation clears it. Lower it only with evidence from your own audit log. |
|
|
626
|
-
| `gate` | `timeoutMs` | 8000 | How long the gate waits for an evaluation. On timeout there is no verdict and dispatch is blocked (fail closed). |
|
|
627
|
-
| `gate` | `requestBudgetTokens` | 30000 | Maximum size of a gate evaluation request. Normal evaluations use under a thousand tokens. |
|
|
628
|
-
| seat record | `maxConcurrency` | 4 | How many workers one seat may run at once. The default suits a typical single machine or hosted seat. Tune it per seat: a cluster takes more, a constrained subscription endpoint may take less. |
|
|
629
|
-
| automation policy | `maxConcurrent` | 4 | How many workers may run at once across all seats. The default is a conservative start. Tune it to the capacity you actually have, and raise it when you add seats or hardware. |
|
|
630
|
-
|
|
631
|
-
Seats are named for infrastructure, not models: `dgx-spark-cluster`,
|
|
632
|
-
`mac-studio`, `merge-gateway-worker`. The model is a mutable field on the
|
|
633
|
-
seat, so changing models never renumbers your configuration. Each seat also
|
|
634
|
-
carries `scopeStatements`: plain-language descriptions of what the seat is
|
|
635
|
-
for. The dispatch gate reads them when judging whether a task fits.
|
|
636
|
-
|
|
637
661
|
## License
|
|
638
662
|
|
|
639
663
|
AGPL-3.0-only with author-attribution additional terms (Section 7(b));
|
|
@@ -0,0 +1,210 @@
|
|
|
1
|
+
# janus — Jev evaluation wrapper service
|
|
2
|
+
|
|
3
|
+
janus is the local wrapper service for the Jev decision layer (design of
|
|
4
|
+
record: `.private-planning/JEV_DECISION_LAYER_DESIGN.md` §5–§7, Revision 2.1).
|
|
5
|
+
It wraps the official `@typesafe-ai/sdk` (TypeSafe Direct — the sole
|
|
6
|
+
provider; the Vercel AI Gateway and merge gateway were dropped by owner
|
|
7
|
+
decision) and exposes one evaluation endpoint plus health/readiness/version
|
|
8
|
+
probes on loopback.
|
|
9
|
+
|
|
10
|
+
## What janus does
|
|
11
|
+
|
|
12
|
+
- `POST /evaluate` — evaluates typed questions (boolean / choice / score)
|
|
13
|
+
against one shared state via TypeSafe Direct (`typesafe-ai`, model `jev-latest`).
|
|
14
|
+
- `GET /healthz` — liveness.
|
|
15
|
+
- `GET /readyz` — readiness (gateway key resolvable from the Keychain).
|
|
16
|
+
- `GET /version` — janus version, AI SDK version, schema versions, allowlist.
|
|
17
|
+
|
|
18
|
+
Guarantees:
|
|
19
|
+
|
|
20
|
+
- **Model allowlist** (server-side config; request-level model overrides are
|
|
21
|
+
rejected with `model_unauthorized`).
|
|
22
|
+
- **Loopback only** — binds `127.0.0.1` and nothing else. No auth beyond
|
|
23
|
+
loopback; it is a local tool. There is deliberately no bind-address option.
|
|
24
|
+
- **Single-flight** — identical requests (same evalInputDigest) coalesce;
|
|
25
|
+
distinct in-flight evaluations are capped (default 4) and rejected with
|
|
26
|
+
`busy`.
|
|
27
|
+
- **30k-token request budget** — oversized requests are rejected with
|
|
28
|
+
`state_too_large`, never truncated silently.
|
|
29
|
+
- **Redaction** — a general secrets/PII pass runs over the state before any
|
|
30
|
+
external call.
|
|
31
|
+
- **Audit log** — append-only JSONL of request id, timestamps, question
|
|
32
|
+
names, verdict probabilities, and token usage. **State contents never
|
|
33
|
+
reach the audit log.** 90-day default retention, size-capped with
|
|
34
|
+
oldest-first rotation.
|
|
35
|
+
- **Daily cost breaker** — token/cost ceilings in config; on exceed,
|
|
36
|
+
`/evaluate` returns `busy` with `budget_exhausted` detail until the UTC
|
|
37
|
+
day window resets.
|
|
38
|
+
- **Timeouts** — 30s service-side cap; callers may pass a shorter
|
|
39
|
+
per-request `timeoutMs` (capped at 30s).
|
|
40
|
+
|
|
41
|
+
## Provisioning the Keychain item
|
|
42
|
+
|
|
43
|
+
The TypeSafe Direct key is read from the macOS Keychain at runtime — janus
|
|
44
|
+
does not rely on inherited environment variables. The `typesafe-ai` /
|
|
45
|
+
`Typesafe AI` generic-password item does not exist by default; create it
|
|
46
|
+
once (interactive provisioning):
|
|
47
|
+
|
|
48
|
+
```sh
|
|
49
|
+
security add-generic-password -U -a typesafe-ai -s "Typesafe AI" -w
|
|
50
|
+
# prompts for the secret; nothing is echoed
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
janus reads it with `security find-generic-password -w` under guarded
|
|
54
|
+
capture: stdout is piped directly into memory, never echoed, never logged,
|
|
55
|
+
never passed to spawned subprocesses. If the item is missing, janus returns
|
|
56
|
+
`model_unauthorized` with `detail.key_unavailable: true` and `GET /readyz`
|
|
57
|
+
reports 503 with the provisioning hint.
|
|
58
|
+
|
|
59
|
+
Verify the item is present (no secret material is printed):
|
|
60
|
+
|
|
61
|
+
```sh
|
|
62
|
+
/usr/bin/security find-generic-password -s "Typesafe AI" -a typesafe-ai >/dev/null 2>&1 && echo present
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
To rotate the secret, re-run the `security add-generic-password -U` command
|
|
66
|
+
above and enter the new value at the prompt (`-U` updates the existing item).
|
|
67
|
+
To remove the item entirely:
|
|
68
|
+
|
|
69
|
+
```sh
|
|
70
|
+
security delete-generic-password -a typesafe-ai -s "Typesafe AI"
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
## Running
|
|
74
|
+
|
|
75
|
+
Dev (from `janus/`):
|
|
76
|
+
|
|
77
|
+
```sh
|
|
78
|
+
npx tsx janus/server.ts # listens on 127.0.0.1:8787
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Config is read from `~/.config/typesafe-ai/jev/janus-config.json`
|
|
82
|
+
(override with `JANUS_CONFIG_PATH`). Defaults work with no file:
|
|
83
|
+
|
|
84
|
+
```json
|
|
85
|
+
{
|
|
86
|
+
"port": 8787,
|
|
87
|
+
"modelAllowlist": ["typesafe-ai/jev"],
|
|
88
|
+
"defaultModel": "typesafe-ai/jev",
|
|
89
|
+
"maxConcurrent": 4,
|
|
90
|
+
"dailyTokenCeiling": 2000000,
|
|
91
|
+
"dailyCostCeilingUsd": 10,
|
|
92
|
+
"audit": { "path": "~/.local/share/janus/audit", "maxBytes": 10485760, "retentionDays": 90 }
|
|
93
|
+
}
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
launchd — template at `janus/user.janus.plist`. The template is a starting
|
|
97
|
+
point: every user-specific value (the label, the checkout path, log
|
|
98
|
+
locations) is marked ADJUST in the file. Copy it out of the repository,
|
|
99
|
+
adjust it, then load it:
|
|
100
|
+
|
|
101
|
+
```sh
|
|
102
|
+
cp janus/user.janus.plist ~/Library/LaunchAgents/
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
```sh
|
|
106
|
+
launchctl load ~/Library/LaunchAgents/user.janus.plist
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
Tests (no live gateway calls — the SDK is mocked):
|
|
110
|
+
|
|
111
|
+
```sh
|
|
112
|
+
cd janus && npm test
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
## Endpoint contract
|
|
116
|
+
|
|
117
|
+
### POST /evaluate
|
|
118
|
+
|
|
119
|
+
Request:
|
|
120
|
+
|
|
121
|
+
```json
|
|
122
|
+
{
|
|
123
|
+
"state": "string | object | array (message histories pass directly)",
|
|
124
|
+
"questions": [
|
|
125
|
+
{ "type": "boolean", "question": "Was a refund issued?", "threshold": 0.9 },
|
|
126
|
+
{ "type": "choice", "question": "Route this ticket", "choices": ["billing", "tech", "other"] },
|
|
127
|
+
{ "type": "score", "question": "Rate this PR" }
|
|
128
|
+
],
|
|
129
|
+
"model": "optional; must be on the server allowlist, overrides rejected",
|
|
130
|
+
"timeoutMs": 10000
|
|
131
|
+
}
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Question schema v1 (§6), enforced atomically — any malformed, absent, or
|
|
135
|
+
contradictory element fails the whole request with `validation_error`:
|
|
136
|
+
|
|
137
|
+
- **boolean** — `threshold` optional, default `0.9`, comparison
|
|
138
|
+
`probability >= threshold`. Optional `criteria: {true?, false?}` labels.
|
|
139
|
+
- **choice** — `choices` array REQUIRED; the verdict's `choice` is guaranteed
|
|
140
|
+
to be a member.
|
|
141
|
+
- **score** — numeric 0..5, one decimal. Ordinal vocabularies must be
|
|
142
|
+
**choice** questions, not scores.
|
|
143
|
+
|
|
144
|
+
§6 → SDK mapping (verified against ai@7.0.106 at launch step 1): the SDK's
|
|
145
|
+
`experimental_evaluate` has **no top-level `criteria` argument**; the
|
|
146
|
+
candidate set lives inside each question as a REQUIRED `criteria` field
|
|
147
|
+
(choice: `Record<option, description | null>`; score: ordered level array;
|
|
148
|
+
boolean: optional `{true?, false?}`). janus accepts §6's `choices` array at
|
|
149
|
+
the transport layer and maps it to `criteria: {a: null, b: null, ...}`. The
|
|
150
|
+
score vocabulary is passed as the ordered level array `["0".."5"]`.
|
|
151
|
+
|
|
152
|
+
Success response (200):
|
|
153
|
+
|
|
154
|
+
```json
|
|
155
|
+
{
|
|
156
|
+
"request_id": "uuid",
|
|
157
|
+
"verdict": {
|
|
158
|
+
"Was a refund issued?": { "type": "boolean", "probability": 0.97, "passed": true },
|
|
159
|
+
"Route this ticket": { "type": "choice", "choice": "billing", "probabilities": {} },
|
|
160
|
+
"Rate this PR": { "type": "score", "score": 3.0 }
|
|
161
|
+
},
|
|
162
|
+
"usage": { "inputTokens": 0, "outputTokens": 0, "totalTokens": 0 },
|
|
163
|
+
"model": "typesafe-ai/jev",
|
|
164
|
+
"digest": "sha256 evalInputDigest"
|
|
165
|
+
}
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
"Model returned false" is a **200** with `passed: false` / low probability —
|
|
169
|
+
not an error.
|
|
170
|
+
|
|
171
|
+
### Error taxonomy
|
|
172
|
+
|
|
173
|
+
All errors: `{ "error": code, "message": ..., "request_id": ..., "detail"?: {...} }`
|
|
174
|
+
|
|
175
|
+
| Code | HTTP | Meaning |
|
|
176
|
+
|---|---|---|
|
|
177
|
+
| `validation_error` | 400 | bad request (schema, JSON, unknown route) |
|
|
178
|
+
| `state_too_large` | 413 | over the 30k-token budget; never truncated |
|
|
179
|
+
| `unknown_question` | 400 | reserved for router/schema-version mismatches |
|
|
180
|
+
| `model_unauthorized` | 403/502/503 | not on allowlist, credential rejected, or key unavailable (`detail.key_unavailable`) |
|
|
181
|
+
| `gateway_unavailable` | 503 | gateway unreachable / model missing / rate limit |
|
|
182
|
+
| `timeout` | 504 | evaluation exceeded the budget |
|
|
183
|
+
| `busy` | 429 | concurrency cap or `detail.budget_exhausted` |
|
|
184
|
+
| `degraded` | 503 | internal error |
|
|
185
|
+
|
|
186
|
+
Callers distinguish **"model returned false"** (200, low probability) from
|
|
187
|
+
**"no verdict"** (`timeout` / `gateway_unavailable`) from **"bad request"**
|
|
188
|
+
(`validation_error`).
|
|
189
|
+
|
|
190
|
+
## Layout
|
|
191
|
+
|
|
192
|
+
```
|
|
193
|
+
janus/
|
|
194
|
+
janus/
|
|
195
|
+
config.ts config load/validate (allowlist, ceilings, retention)
|
|
196
|
+
schema.ts §6 question schema v1 enforcement (atomic fail-closed)
|
|
197
|
+
verdict.ts verdict enforcement (membership, ranges, thresholds)
|
|
198
|
+
budget.ts token estimation, 30k budget, evalInputDigest
|
|
199
|
+
redaction pass: shared library typesafe-secure
|
|
200
|
+
lib/redaction (MIT), wired via a file: dependency
|
|
201
|
+
keychain.ts guarded `security find-generic-password -w` capture
|
|
202
|
+
single_flight.ts digest coalescing + concurrency cap
|
|
203
|
+
budget_breaker.ts daily token/cost breaker
|
|
204
|
+
audit.ts append-only JSONL audit log, rotation + retention
|
|
205
|
+
sdk.ts THE ONLY AI SDK import site (pinned versions)
|
|
206
|
+
evaluate.ts /evaluate pipeline + SDK error classification
|
|
207
|
+
server.ts HTTP surface, loopback bind, entrypoint
|
|
208
|
+
tests/ node:test suites (mocked SDK, no live calls)
|
|
209
|
+
user.janus.plist launchd template
|
|
210
|
+
```
|
package/janus/README.md
ADDED
|
@@ -0,0 +1,196 @@
|
|
|
1
|
+
# janus — Jev evaluation wrapper service
|
|
2
|
+
|
|
3
|
+
janus is the local wrapper service for the Jev decision layer (design of
|
|
4
|
+
record: `.private-planning/JEV_DECISION_LAYER_DESIGN.md` §5–§7, Revision 2.1).
|
|
5
|
+
It wraps the official `@typesafe-ai/sdk` (TypeSafe Direct — the sole
|
|
6
|
+
provider; the Vercel AI Gateway and merge gateway were dropped by owner
|
|
7
|
+
decision) and exposes one evaluation endpoint plus health/readiness/version
|
|
8
|
+
probes on loopback.
|
|
9
|
+
|
|
10
|
+
## What janus does
|
|
11
|
+
|
|
12
|
+
- `POST /evaluate` — evaluates typed questions (boolean / choice / score)
|
|
13
|
+
against one shared state via TypeSafe Direct (`typesafe-ai`, model `jev-latest`).
|
|
14
|
+
- `GET /healthz` — liveness.
|
|
15
|
+
- `GET /readyz` — readiness (gateway key resolvable from the Keychain).
|
|
16
|
+
- `GET /version` — janus version, AI SDK version, schema versions, allowlist.
|
|
17
|
+
|
|
18
|
+
Guarantees:
|
|
19
|
+
|
|
20
|
+
- **Model allowlist** (server-side config; request-level model overrides are
|
|
21
|
+
rejected with `model_unauthorized`).
|
|
22
|
+
- **Loopback only** — binds `127.0.0.1` and nothing else. No auth beyond
|
|
23
|
+
loopback; it is a local tool. There is deliberately no bind-address option.
|
|
24
|
+
- **Single-flight** — identical requests (same evalInputDigest) coalesce;
|
|
25
|
+
distinct in-flight evaluations are capped (default 4) and rejected with
|
|
26
|
+
`busy`.
|
|
27
|
+
- **30k-token request budget** — oversized requests are rejected with
|
|
28
|
+
`state_too_large`, never truncated silently.
|
|
29
|
+
- **Redaction** — a general secrets/PII pass runs over the state before any
|
|
30
|
+
external call.
|
|
31
|
+
- **Audit log** — append-only JSONL of request id, timestamps, question
|
|
32
|
+
names, verdict probabilities, and token usage. **State contents never
|
|
33
|
+
reach the audit log.** 90-day default retention, size-capped with
|
|
34
|
+
oldest-first rotation.
|
|
35
|
+
- **Daily cost breaker** — token/cost ceilings in config; on exceed,
|
|
36
|
+
`/evaluate` returns `busy` with `budget_exhausted` detail until the UTC
|
|
37
|
+
day window resets.
|
|
38
|
+
- **Timeouts** — 30s service-side cap; callers may pass a shorter
|
|
39
|
+
per-request `timeoutMs` (capped at 30s).
|
|
40
|
+
|
|
41
|
+
## Provisioning the Keychain item
|
|
42
|
+
|
|
43
|
+
The TypeSafe Direct key is read from the macOS Keychain at runtime — janus
|
|
44
|
+
does not rely on inherited environment variables. The `typesafe-ai` /
|
|
45
|
+
`Typesafe AI` generic-password item does not exist by default; create it
|
|
46
|
+
once (interactive provisioning):
|
|
47
|
+
|
|
48
|
+
```sh
|
|
49
|
+
security add-generic-password -U -a typesafe-ai -s "Typesafe AI" -w
|
|
50
|
+
# prompts for the secret; nothing is echoed
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
janus reads it with `security find-generic-password -w` under guarded
|
|
54
|
+
capture: stdout is piped directly into memory, never echoed, never logged,
|
|
55
|
+
never passed to spawned subprocesses. If the item is missing, janus returns
|
|
56
|
+
`model_unauthorized` with `detail.key_unavailable: true` and `GET /readyz`
|
|
57
|
+
reports 503 with the provisioning hint.
|
|
58
|
+
|
|
59
|
+
## Running
|
|
60
|
+
|
|
61
|
+
Dev (from `janus/`):
|
|
62
|
+
|
|
63
|
+
```sh
|
|
64
|
+
npx tsx janus/server.ts # listens on 127.0.0.1:8787
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
Config is read from `~/.config/typesafe-ai/jev/janus-config.json`
|
|
68
|
+
(override with `JANUS_CONFIG_PATH`). Defaults work with no file:
|
|
69
|
+
|
|
70
|
+
```json
|
|
71
|
+
{
|
|
72
|
+
"port": 8787,
|
|
73
|
+
"modelAllowlist": ["typesafe-ai/jev"],
|
|
74
|
+
"defaultModel": "typesafe-ai/jev",
|
|
75
|
+
"maxConcurrent": 4,
|
|
76
|
+
"dailyTokenCeiling": 2000000,
|
|
77
|
+
"dailyCostCeilingUsd": 10,
|
|
78
|
+
"audit": { "path": "~/.local/share/janus/audit", "maxBytes": 10485760, "retentionDays": 90 }
|
|
79
|
+
}
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
launchd — template at `janus/user.janus.plist`. The template is a starting
|
|
83
|
+
point: every user-specific value (the label, the checkout path, log
|
|
84
|
+
locations) is marked ADJUST in the file. Copy it out of the repository,
|
|
85
|
+
adjust it, then load it:
|
|
86
|
+
|
|
87
|
+
```sh
|
|
88
|
+
cp janus/user.janus.plist ~/Library/LaunchAgents/
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
```sh
|
|
92
|
+
launchctl load ~/Library/LaunchAgents/user.janus.plist
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
Tests (no live gateway calls — the SDK is mocked):
|
|
96
|
+
|
|
97
|
+
```sh
|
|
98
|
+
cd janus && npm test
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
## Endpoint contract
|
|
102
|
+
|
|
103
|
+
### POST /evaluate
|
|
104
|
+
|
|
105
|
+
Request:
|
|
106
|
+
|
|
107
|
+
```json
|
|
108
|
+
{
|
|
109
|
+
"state": "string | object | array (message histories pass directly)",
|
|
110
|
+
"questions": [
|
|
111
|
+
{ "type": "boolean", "question": "Was a refund issued?", "threshold": 0.9 },
|
|
112
|
+
{ "type": "choice", "question": "Route this ticket", "choices": ["billing", "tech", "other"] },
|
|
113
|
+
{ "type": "score", "question": "Rate this PR" }
|
|
114
|
+
],
|
|
115
|
+
"model": "optional; must be on the server allowlist, overrides rejected",
|
|
116
|
+
"timeoutMs": 10000
|
|
117
|
+
}
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
Question schema v1 (§6), enforced atomically — any malformed, absent, or
|
|
121
|
+
contradictory element fails the whole request with `validation_error`:
|
|
122
|
+
|
|
123
|
+
- **boolean** — `threshold` optional, default `0.9`, comparison
|
|
124
|
+
`probability >= threshold`. Optional `criteria: {true?, false?}` labels.
|
|
125
|
+
- **choice** — `choices` array REQUIRED; the verdict's `choice` is guaranteed
|
|
126
|
+
to be a member.
|
|
127
|
+
- **score** — numeric 0..5, one decimal. Ordinal vocabularies must be
|
|
128
|
+
**choice** questions, not scores.
|
|
129
|
+
|
|
130
|
+
§6 → SDK mapping (verified against ai@7.0.106 at launch step 1): the SDK's
|
|
131
|
+
`experimental_evaluate` has **no top-level `criteria` argument**; the
|
|
132
|
+
candidate set lives inside each question as a REQUIRED `criteria` field
|
|
133
|
+
(choice: `Record<option, description | null>`; score: ordered level array;
|
|
134
|
+
boolean: optional `{true?, false?}`). janus accepts §6's `choices` array at
|
|
135
|
+
the transport layer and maps it to `criteria: {a: null, b: null, ...}`. The
|
|
136
|
+
score vocabulary is passed as the ordered level array `["0".."5"]`.
|
|
137
|
+
|
|
138
|
+
Success response (200):
|
|
139
|
+
|
|
140
|
+
```json
|
|
141
|
+
{
|
|
142
|
+
"request_id": "uuid",
|
|
143
|
+
"verdict": {
|
|
144
|
+
"Was a refund issued?": { "type": "boolean", "probability": 0.97, "passed": true },
|
|
145
|
+
"Route this ticket": { "type": "choice", "choice": "billing", "probabilities": {} },
|
|
146
|
+
"Rate this PR": { "type": "score", "score": 3.0 }
|
|
147
|
+
},
|
|
148
|
+
"usage": { "inputTokens": 0, "outputTokens": 0, "totalTokens": 0 },
|
|
149
|
+
"model": "typesafe-ai/jev",
|
|
150
|
+
"digest": "sha256 evalInputDigest"
|
|
151
|
+
}
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
"Model returned false" is a **200** with `passed: false` / low probability —
|
|
155
|
+
not an error.
|
|
156
|
+
|
|
157
|
+
### Error taxonomy
|
|
158
|
+
|
|
159
|
+
All errors: `{ "error": code, "message": ..., "request_id": ..., "detail"?: {...} }`
|
|
160
|
+
|
|
161
|
+
| Code | HTTP | Meaning |
|
|
162
|
+
|---|---|---|
|
|
163
|
+
| `validation_error` | 400 | bad request (schema, JSON, unknown route) |
|
|
164
|
+
| `state_too_large` | 413 | over the 30k-token budget; never truncated |
|
|
165
|
+
| `unknown_question` | 400 | reserved for router/schema-version mismatches |
|
|
166
|
+
| `model_unauthorized` | 403/502/503 | not on allowlist, credential rejected, or key unavailable (`detail.key_unavailable`) |
|
|
167
|
+
| `gateway_unavailable` | 503 | gateway unreachable / model missing / rate limit |
|
|
168
|
+
| `timeout` | 504 | evaluation exceeded the budget |
|
|
169
|
+
| `busy` | 429 | concurrency cap or `detail.budget_exhausted` |
|
|
170
|
+
| `degraded` | 503 | internal error |
|
|
171
|
+
|
|
172
|
+
Callers distinguish **"model returned false"** (200, low probability) from
|
|
173
|
+
**"no verdict"** (`timeout` / `gateway_unavailable`) from **"bad request"**
|
|
174
|
+
(`validation_error`).
|
|
175
|
+
|
|
176
|
+
## Layout
|
|
177
|
+
|
|
178
|
+
```
|
|
179
|
+
janus/
|
|
180
|
+
janus/
|
|
181
|
+
config.ts config load/validate (allowlist, ceilings, retention)
|
|
182
|
+
schema.ts §6 question schema v1 enforcement (atomic fail-closed)
|
|
183
|
+
verdict.ts verdict enforcement (membership, ranges, thresholds)
|
|
184
|
+
budget.ts token estimation, 30k budget, evalInputDigest
|
|
185
|
+
redaction pass: shared library typesafe-secure
|
|
186
|
+
lib/redaction (MIT), wired via a file: dependency
|
|
187
|
+
keychain.ts guarded `security find-generic-password -w` capture
|
|
188
|
+
single_flight.ts digest coalescing + concurrency cap
|
|
189
|
+
budget_breaker.ts daily token/cost breaker
|
|
190
|
+
audit.ts append-only JSONL audit log, rotation + retention
|
|
191
|
+
sdk.ts THE ONLY AI SDK import site (pinned versions)
|
|
192
|
+
evaluate.ts /evaluate pipeline + SDK error classification
|
|
193
|
+
server.ts HTTP surface, loopback bind, entrypoint
|
|
194
|
+
tests/ node:test suites (mocked SDK, no live calls)
|
|
195
|
+
user.janus.plist launchd template
|
|
196
|
+
```
|