@sleepy_panda_srl/portulan 0.3.0 → 0.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +18 -6
- package/cli/README.md +56 -4
- package/cli/advisory.mjs +167 -53
- package/cli/cache-risk.mjs +69 -0
- package/cli/comments.mjs +8 -5
- package/cli/compile.mjs +492 -146
- package/cli/context.mjs +204 -43
- package/cli/doctor.mjs +211 -33
- package/cli/drills.mjs +32 -7
- package/cli/finish.md +34 -0
- package/cli/finish.mjs +9 -67
- package/cli/form.mjs +35 -19
- package/cli/frontmatter.mjs +39 -0
- package/cli/gate.mjs +14 -12
- package/cli/guidance-files.mjs +335 -0
- package/cli/handoff-policy.mjs +76 -0
- package/cli/html-entities.mjs +2130 -0
- package/cli/import-files.mjs +15 -0
- package/cli/index.mjs +77 -24
- package/cli/init-preview.mjs +553 -0
- package/cli/init.mjs +261 -46
- package/cli/instructions-journal.mjs +1185 -0
- package/cli/instructions.mjs +332 -57
- package/cli/ledger.mjs +123 -37
- package/cli/librarian.mjs +103 -19
- package/cli/persona-routing.mjs +209 -0
- package/cli/plugin-lint.mjs +77 -56
- package/cli/portulan.mjs +7 -2
- package/cli/recipe-output.mjs +91 -0
- package/cli/recipe-runner.md +68 -0
- package/cli/recipe-runner.mjs +131 -0
- package/cli/recipe-set.mjs +12 -3
- package/cli/ritual-contexts.mjs +103 -0
- package/cli/sessions.mjs +2 -1
- package/cli/spend-analysis.mjs +111 -0
- package/cli/spend-offer.md +53 -0
- package/cli/spend-offer.mjs +93 -0
- package/cli/stop-diagnostics.mjs +114 -0
- package/cli/stop-gate.mjs +133 -90
- package/cli/stop-policy.mjs +14 -0
- package/cli/symbols.mjs +55 -8
- package/cli/telemetry.mjs +19 -12
- package/cli/upgrade.mjs +62 -44
- package/cli/vendor.mjs +210 -126
- package/core/operating/context.md +368 -60
- package/core/operating/evolution.md +3 -2
- package/core/operating/loop.md +9 -4
- package/core/operating/sessions.md +6 -15
- package/core/personas/README.md +52 -10
- package/core/personas/implementer.md +3 -0
- package/core/personas/librarian.md +3 -0
- package/core/personas/reviewer.md +3 -0
- package/core/templates/handoff.md +6 -2
- package/evals/ab/personas/README.md +111 -0
- package/evals/ab/personas/acquisition.json +330 -0
- package/evals/ab/personas/bindings/implementer.txt +43 -0
- package/evals/ab/personas/bindings/librarian.txt +40 -0
- package/evals/ab/personas/bindings/reviewer.txt +41 -0
- package/evals/ab/personas/corpus.json +69 -0
- package/evals/ab/personas/grader.mjs +171 -0
- package/evals/ab/personas/implementer.json +774 -0
- package/evals/ab/personas/implementer.md +22 -0
- package/evals/ab/personas/librarian.json +758 -0
- package/evals/ab/personas/librarian.md +22 -0
- package/evals/ab/personas/protocol/continuation.mjs.txt +284 -0
- package/evals/ab/personas/protocol/first-outcome.mjs.txt +227 -0
- package/evals/ab/personas/protocol/first-probe.mjs.txt +172 -0
- package/evals/ab/personas/protocol/publisher.mjs.txt +126 -0
- package/evals/ab/personas/protocol/second-probe.mjs.txt +193 -0
- package/evals/ab/personas/repairs/first-control-answer.txt +23 -0
- package/evals/ab/personas/repairs/grader-before-decode-repair.mjs.txt +143 -0
- package/evals/ab/personas/reviewer.json +758 -0
- package/evals/ab/personas/reviewer.md +22 -0
- package/evals/releases/0.4.0.json +154 -0
- package/evals/releases/0.4.0.md +69 -0
- package/evals/releases/0.4.1.json +154 -0
- package/evals/releases/0.4.1.md +69 -0
- package/package.json +6 -2
- package/packs/rituals/checkpoints/README.md +35 -0
- package/packs/rituals/checkpoints/pack.json +3 -2
- package/packs/tools/local-llm/README.md +46 -6
- package/packs/tools/local-llm/runner/local-llm.mjs +127 -13
- package/packs/tools/local-llm/skills/delegate-local/SKILL.md +26 -2
- package/spec/README.md +37 -4
- package/spec/migrations/0009-instruction-sections-on-read.mjs +60 -11
- package/spec/migrations/0011-persona-bindings-compiled.mjs +23 -0
- package/spec/migrations/README.md +6 -0
- package/spec/pack.schema.json +12 -3
- package/spec/slots.md +262 -117
- package/spec/workspace.schema.json +67 -11
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# reviewer — acquired routing comparison
|
|
2
|
+
|
|
3
|
+
Host: 2.1.321 (Claude Code). Concrete model: claude-sonnet-5-5.
|
|
4
|
+
|
|
5
|
+
A uses the shipped inherited model and undeclared child effort (observed high). B requests medium through the native child tool. Each case has one attempt per arm.
|
|
6
|
+
|
|
7
|
+
| Arm | Semantic passes | Observed cost index | Cold-normalized cost index |
|
|
8
|
+
|---|---:|---:|---:|
|
|
9
|
+
| A | 3/3 | 150520.60 | 426477.60 |
|
|
10
|
+
| B | 3/3 | 161948.05 | 426186.55 |
|
|
11
|
+
|
|
12
|
+
Medium wins this observed corpus: **yes**. The shipped policy remains inherited model, undeclared effort, instruction floor retained and five-minute cache lifetime.
|
|
13
|
+
|
|
14
|
+
These are three observations, not a reliability estimate. A result on this concrete model does not establish medium effort for other inherited models. Cost indexes use the frozen declared vector, include parent and child requests, and are not subscription bills. First-request input is normalized separately in each fresh parent and child context.
|
|
15
|
+
|
|
16
|
+
| Case | A | B |
|
|
17
|
+
|---|---|---|
|
|
18
|
+
| reviewer-verdict | PASS | PASS |
|
|
19
|
+
| reviewer-clean | PASS | PASS |
|
|
20
|
+
| reviewer-boundaries | PASS | PASS |
|
|
21
|
+
|
|
22
|
+
The common acquisition record accounts for both mechanism probes and discloses the one retained-answer decoder repair after the first outcome. The original failed decode score and the repaired score are both retained; that child was never rerun.
|
|
@@ -0,0 +1,154 @@
|
|
|
1
|
+
{
|
|
2
|
+
"portulan": {
|
|
3
|
+
"releaseEval": "1"
|
|
4
|
+
},
|
|
5
|
+
"version": "0.4.0",
|
|
6
|
+
"captured": "2026-11-11",
|
|
7
|
+
"source": {
|
|
8
|
+
"commit": "660cbbfd7c9a4cf234e5a85d82d2ae21d248d3b1",
|
|
9
|
+
"clean": true
|
|
10
|
+
},
|
|
11
|
+
"host": {
|
|
12
|
+
"node": "v26.13.0",
|
|
13
|
+
"platform": "darwin"
|
|
14
|
+
},
|
|
15
|
+
"recipes": [
|
|
16
|
+
{
|
|
17
|
+
"id": "cache-stability",
|
|
18
|
+
"exit": 0
|
|
19
|
+
},
|
|
20
|
+
{
|
|
21
|
+
"id": "docs",
|
|
22
|
+
"exit": 0
|
|
23
|
+
},
|
|
24
|
+
{
|
|
25
|
+
"id": "json",
|
|
26
|
+
"exit": 0
|
|
27
|
+
},
|
|
28
|
+
{
|
|
29
|
+
"id": "doctor",
|
|
30
|
+
"exit": 0
|
|
31
|
+
},
|
|
32
|
+
{
|
|
33
|
+
"id": "tests",
|
|
34
|
+
"exit": 0
|
|
35
|
+
},
|
|
36
|
+
{
|
|
37
|
+
"id": "plugin",
|
|
38
|
+
"exit": 0
|
|
39
|
+
},
|
|
40
|
+
{
|
|
41
|
+
"id": "compile",
|
|
42
|
+
"exit": 0
|
|
43
|
+
},
|
|
44
|
+
{
|
|
45
|
+
"id": "workflow-filters",
|
|
46
|
+
"exit": 0
|
|
47
|
+
},
|
|
48
|
+
{
|
|
49
|
+
"id": "index",
|
|
50
|
+
"exit": 0
|
|
51
|
+
},
|
|
52
|
+
{
|
|
53
|
+
"id": "context",
|
|
54
|
+
"exit": 0
|
|
55
|
+
},
|
|
56
|
+
{
|
|
57
|
+
"id": "ledger",
|
|
58
|
+
"exit": 0
|
|
59
|
+
},
|
|
60
|
+
{
|
|
61
|
+
"id": "control-chars",
|
|
62
|
+
"exit": 0
|
|
63
|
+
},
|
|
64
|
+
{
|
|
65
|
+
"id": "comments",
|
|
66
|
+
"exit": 0
|
|
67
|
+
},
|
|
68
|
+
{
|
|
69
|
+
"id": "rule-carriers",
|
|
70
|
+
"exit": 0
|
|
71
|
+
},
|
|
72
|
+
{
|
|
73
|
+
"id": "pack-version",
|
|
74
|
+
"exit": 0
|
|
75
|
+
},
|
|
76
|
+
{
|
|
77
|
+
"id": "pack-identity",
|
|
78
|
+
"exit": 0
|
|
79
|
+
},
|
|
80
|
+
{
|
|
81
|
+
"id": "payload",
|
|
82
|
+
"exit": 0
|
|
83
|
+
},
|
|
84
|
+
{
|
|
85
|
+
"id": "eval-bundle",
|
|
86
|
+
"exit": 0
|
|
87
|
+
},
|
|
88
|
+
{
|
|
89
|
+
"id": "goldens",
|
|
90
|
+
"exit": 0
|
|
91
|
+
},
|
|
92
|
+
{
|
|
93
|
+
"id": "mutants",
|
|
94
|
+
"exit": 0
|
|
95
|
+
},
|
|
96
|
+
{
|
|
97
|
+
"id": "fuzz-shell",
|
|
98
|
+
"exit": 0
|
|
99
|
+
},
|
|
100
|
+
{
|
|
101
|
+
"id": "version-carriers",
|
|
102
|
+
"exit": 0
|
|
103
|
+
},
|
|
104
|
+
{
|
|
105
|
+
"id": "skill-goldens",
|
|
106
|
+
"exit": 0
|
|
107
|
+
},
|
|
108
|
+
{
|
|
109
|
+
"id": "review-loop",
|
|
110
|
+
"exit": 0
|
|
111
|
+
},
|
|
112
|
+
{
|
|
113
|
+
"id": "telemetry",
|
|
114
|
+
"exit": 0
|
|
115
|
+
},
|
|
116
|
+
{
|
|
117
|
+
"id": "drills",
|
|
118
|
+
"exit": 0
|
|
119
|
+
},
|
|
120
|
+
{
|
|
121
|
+
"id": "ab",
|
|
122
|
+
"exit": 0
|
|
123
|
+
},
|
|
124
|
+
{
|
|
125
|
+
"id": "ab-grade",
|
|
126
|
+
"exit": 0
|
|
127
|
+
},
|
|
128
|
+
{
|
|
129
|
+
"id": "ab-run",
|
|
130
|
+
"exit": 0
|
|
131
|
+
},
|
|
132
|
+
{
|
|
133
|
+
"id": "tools/github:actions-pinned",
|
|
134
|
+
"exit": 0
|
|
135
|
+
},
|
|
136
|
+
{
|
|
137
|
+
"id": "tools/local-llm:runner-rails",
|
|
138
|
+
"exit": 0
|
|
139
|
+
}
|
|
140
|
+
],
|
|
141
|
+
"excluded": [
|
|
142
|
+
{
|
|
143
|
+
"id": "release-eval",
|
|
144
|
+
"why": "a capture cannot be accurate about the record it is inside — captured before the record is written it is red for the record's absence, and captured after it is stale the moment the register is rendered. Its verdict for this release is the rail's own run on the pull request carrying this file."
|
|
145
|
+
}
|
|
146
|
+
],
|
|
147
|
+
"abBaseline": {
|
|
148
|
+
"snapshot": "evals/ab/baseline.json",
|
|
149
|
+
"register": "evals/ab/baseline.md",
|
|
150
|
+
"captured": "2026-08-31T09:05:32.699Z",
|
|
151
|
+
"commit": "a642d551eec6c146c67ef0acfe3f8ba535c5ce6e",
|
|
152
|
+
"clean": false
|
|
153
|
+
}
|
|
154
|
+
}
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# Eval result — Portulan 0.4.0
|
|
2
|
+
|
|
3
|
+
> **Generated. Do not edit.** Rendered from [`evals/releases/0.4.0.json`](0.4.0.json) by [`cli/release-eval.mjs`](../../cli/release-eval.mjs) and byte-compared by the `release-eval` verify recipe. This is the eval result milestone 8 requires a release to carry — `docs/plan.md`, Protocol → Versioning.
|
|
4
|
+
|
|
5
|
+
## What this release was measured at
|
|
6
|
+
|
|
7
|
+
| | |
|
|
8
|
+
|---|---|
|
|
9
|
+
| Version | `0.4.0` |
|
|
10
|
+
| Captured | 2026-11-11 |
|
|
11
|
+
| Commit | `660cbbfd7c9a4cf234e5a85d82d2ae21d248d3b1` |
|
|
12
|
+
| Tree at capture | clean |
|
|
13
|
+
| Node | `v26.13.0` |
|
|
14
|
+
| Platform | `darwin` |
|
|
15
|
+
|
|
16
|
+
## The rails, and what each returned
|
|
17
|
+
|
|
18
|
+
31 of the 32 recipes this workspace yielded at capture. The set is derived from [`cli/recipe-set.mjs`](../../cli/recipe-set.mjs) and never listed by hand.
|
|
19
|
+
|
|
20
|
+
| Recipe | Verdict |
|
|
21
|
+
|---|---|
|
|
22
|
+
| `cache-stability` | green |
|
|
23
|
+
| `docs` | green |
|
|
24
|
+
| `json` | green |
|
|
25
|
+
| `doctor` | green |
|
|
26
|
+
| `tests` | green |
|
|
27
|
+
| `plugin` | green |
|
|
28
|
+
| `compile` | green |
|
|
29
|
+
| `workflow-filters` | green |
|
|
30
|
+
| `index` | green |
|
|
31
|
+
| `context` | green |
|
|
32
|
+
| `ledger` | green |
|
|
33
|
+
| `control-chars` | green |
|
|
34
|
+
| `comments` | green |
|
|
35
|
+
| `rule-carriers` | green |
|
|
36
|
+
| `pack-version` | green |
|
|
37
|
+
| `pack-identity` | green |
|
|
38
|
+
| `payload` | green |
|
|
39
|
+
| `eval-bundle` | green |
|
|
40
|
+
| `goldens` | green |
|
|
41
|
+
| `mutants` | green |
|
|
42
|
+
| `fuzz-shell` | green |
|
|
43
|
+
| `version-carriers` | green |
|
|
44
|
+
| `skill-goldens` | green |
|
|
45
|
+
| `review-loop` | green |
|
|
46
|
+
| `telemetry` | green |
|
|
47
|
+
| `drills` | green |
|
|
48
|
+
| `ab` | green |
|
|
49
|
+
| `ab-grade` | green |
|
|
50
|
+
| `ab-run` | green |
|
|
51
|
+
| `tools/github:actions-pinned` | green |
|
|
52
|
+
| `tools/local-llm:runner-rails` | green |
|
|
53
|
+
|
|
54
|
+
**`release-eval` is excluded**, and the reason is printed rather than the row silently dropped: a capture cannot be accurate about the record it is inside — captured before the record is written it is red for the record's absence, and captured after it is stale the moment the register is rendered. Its verdict for this release is the rail's own run on the pull request carrying this file.
|
|
55
|
+
|
|
56
|
+
## The A/B baseline this release ships against
|
|
57
|
+
|
|
58
|
+
[`evals/ab/baseline.md`](https://github.com/sleepy-panda-srl/portulan/blob/v0.4.0/evals/ab/baseline.md), rendered from `evals/ab/baseline.json` — captured 2026-08-31T09:05:32.699Z at `a642d551eec6c146c67ef0acfe3f8ba535c5ce6e`, over a tree that was **not clean**.
|
|
59
|
+
|
|
60
|
+
**Its figures are not repeated here.** They have one carrier and this is not it.
|
|
61
|
+
|
|
62
|
+
## What this record does not establish
|
|
63
|
+
|
|
64
|
+
- **A green says this record agrees with its own capture — never that the release is good.** Every line below is what the recipes returned at one commit, and a recipe's own green establishes only what that recipe's documentation says it establishes.
|
|
65
|
+
- **The recipes were not run at the tag.** They ran at `660cbbfd7c9a4cf234e5a85d82d2ae21d248d3b1`, and this record is committed *in* the cut change — so the tagged tree is this commit plus whatever that change still adds after the capture, the record itself at minimum. A record cannot be captured at a commit that does not exist yet, and printing the one it was captured at is the only honest form.
|
|
66
|
+
- **`release-eval` is excluded from the rows above**, because a capture cannot be accurate about the record it is inside. Its verdict for this release is the rail's own run on the pull request that carries this file.
|
|
67
|
+
- **This record measures the Portulan repository's own build**, at the commit named above — never the workspace, project or package this release is installed into. Nothing here has looked at your tree.
|
|
68
|
+
- **The tree was clean at capture**, so the commit named above is what was measured.
|
|
69
|
+
- **The A/B baseline's figures are NOT restated here.** This record names which baseline the release ships against; the figures and everything that may not be concluded from them live in that register alone.
|
|
@@ -0,0 +1,154 @@
|
|
|
1
|
+
{
|
|
2
|
+
"portulan": {
|
|
3
|
+
"releaseEval": "1"
|
|
4
|
+
},
|
|
5
|
+
"version": "0.4.1",
|
|
6
|
+
"captured": "2026-11-11",
|
|
7
|
+
"source": {
|
|
8
|
+
"commit": "a83966a56af78bd76a63c48afdce457aa3c5b0d5",
|
|
9
|
+
"clean": true
|
|
10
|
+
},
|
|
11
|
+
"host": {
|
|
12
|
+
"node": "v26.13.0",
|
|
13
|
+
"platform": "darwin"
|
|
14
|
+
},
|
|
15
|
+
"recipes": [
|
|
16
|
+
{
|
|
17
|
+
"id": "cache-stability",
|
|
18
|
+
"exit": 0
|
|
19
|
+
},
|
|
20
|
+
{
|
|
21
|
+
"id": "docs",
|
|
22
|
+
"exit": 0
|
|
23
|
+
},
|
|
24
|
+
{
|
|
25
|
+
"id": "json",
|
|
26
|
+
"exit": 0
|
|
27
|
+
},
|
|
28
|
+
{
|
|
29
|
+
"id": "doctor",
|
|
30
|
+
"exit": 0
|
|
31
|
+
},
|
|
32
|
+
{
|
|
33
|
+
"id": "tests",
|
|
34
|
+
"exit": 0
|
|
35
|
+
},
|
|
36
|
+
{
|
|
37
|
+
"id": "plugin",
|
|
38
|
+
"exit": 0
|
|
39
|
+
},
|
|
40
|
+
{
|
|
41
|
+
"id": "compile",
|
|
42
|
+
"exit": 0
|
|
43
|
+
},
|
|
44
|
+
{
|
|
45
|
+
"id": "workflow-filters",
|
|
46
|
+
"exit": 0
|
|
47
|
+
},
|
|
48
|
+
{
|
|
49
|
+
"id": "index",
|
|
50
|
+
"exit": 0
|
|
51
|
+
},
|
|
52
|
+
{
|
|
53
|
+
"id": "context",
|
|
54
|
+
"exit": 0
|
|
55
|
+
},
|
|
56
|
+
{
|
|
57
|
+
"id": "ledger",
|
|
58
|
+
"exit": 0
|
|
59
|
+
},
|
|
60
|
+
{
|
|
61
|
+
"id": "control-chars",
|
|
62
|
+
"exit": 0
|
|
63
|
+
},
|
|
64
|
+
{
|
|
65
|
+
"id": "comments",
|
|
66
|
+
"exit": 0
|
|
67
|
+
},
|
|
68
|
+
{
|
|
69
|
+
"id": "rule-carriers",
|
|
70
|
+
"exit": 0
|
|
71
|
+
},
|
|
72
|
+
{
|
|
73
|
+
"id": "pack-version",
|
|
74
|
+
"exit": 0
|
|
75
|
+
},
|
|
76
|
+
{
|
|
77
|
+
"id": "pack-identity",
|
|
78
|
+
"exit": 0
|
|
79
|
+
},
|
|
80
|
+
{
|
|
81
|
+
"id": "payload",
|
|
82
|
+
"exit": 0
|
|
83
|
+
},
|
|
84
|
+
{
|
|
85
|
+
"id": "eval-bundle",
|
|
86
|
+
"exit": 0
|
|
87
|
+
},
|
|
88
|
+
{
|
|
89
|
+
"id": "goldens",
|
|
90
|
+
"exit": 0
|
|
91
|
+
},
|
|
92
|
+
{
|
|
93
|
+
"id": "mutants",
|
|
94
|
+
"exit": 0
|
|
95
|
+
},
|
|
96
|
+
{
|
|
97
|
+
"id": "fuzz-shell",
|
|
98
|
+
"exit": 0
|
|
99
|
+
},
|
|
100
|
+
{
|
|
101
|
+
"id": "version-carriers",
|
|
102
|
+
"exit": 0
|
|
103
|
+
},
|
|
104
|
+
{
|
|
105
|
+
"id": "skill-goldens",
|
|
106
|
+
"exit": 0
|
|
107
|
+
},
|
|
108
|
+
{
|
|
109
|
+
"id": "review-loop",
|
|
110
|
+
"exit": 0
|
|
111
|
+
},
|
|
112
|
+
{
|
|
113
|
+
"id": "telemetry",
|
|
114
|
+
"exit": 0
|
|
115
|
+
},
|
|
116
|
+
{
|
|
117
|
+
"id": "drills",
|
|
118
|
+
"exit": 0
|
|
119
|
+
},
|
|
120
|
+
{
|
|
121
|
+
"id": "ab",
|
|
122
|
+
"exit": 0
|
|
123
|
+
},
|
|
124
|
+
{
|
|
125
|
+
"id": "ab-grade",
|
|
126
|
+
"exit": 0
|
|
127
|
+
},
|
|
128
|
+
{
|
|
129
|
+
"id": "ab-run",
|
|
130
|
+
"exit": 0
|
|
131
|
+
},
|
|
132
|
+
{
|
|
133
|
+
"id": "tools/github:actions-pinned",
|
|
134
|
+
"exit": 0
|
|
135
|
+
},
|
|
136
|
+
{
|
|
137
|
+
"id": "tools/local-llm:runner-rails",
|
|
138
|
+
"exit": 0
|
|
139
|
+
}
|
|
140
|
+
],
|
|
141
|
+
"excluded": [
|
|
142
|
+
{
|
|
143
|
+
"id": "release-eval",
|
|
144
|
+
"why": "a capture cannot be accurate about the record it is inside — captured before the record is written it is red for the record's absence, and captured after it is stale the moment the register is rendered. Its verdict for this release is the rail's own run on the pull request carrying this file."
|
|
145
|
+
}
|
|
146
|
+
],
|
|
147
|
+
"abBaseline": {
|
|
148
|
+
"snapshot": "evals/ab/baseline.json",
|
|
149
|
+
"register": "evals/ab/baseline.md",
|
|
150
|
+
"captured": "2026-08-31T09:05:32.699Z",
|
|
151
|
+
"commit": "a642d551eec6c146c67ef0acfe3f8ba535c5ce6e",
|
|
152
|
+
"clean": false
|
|
153
|
+
}
|
|
154
|
+
}
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# Eval result — Portulan 0.4.1
|
|
2
|
+
|
|
3
|
+
> **Generated. Do not edit.** Rendered from [`evals/releases/0.4.1.json`](0.4.1.json) by [`cli/release-eval.mjs`](../../cli/release-eval.mjs) and byte-compared by the `release-eval` verify recipe. This is the eval result milestone 8 requires a release to carry — `docs/plan.md`, Protocol → Versioning.
|
|
4
|
+
|
|
5
|
+
## What this release was measured at
|
|
6
|
+
|
|
7
|
+
| | |
|
|
8
|
+
|---|---|
|
|
9
|
+
| Version | `0.4.1` |
|
|
10
|
+
| Captured | 2026-11-11 |
|
|
11
|
+
| Commit | `a83966a56af78bd76a63c48afdce457aa3c5b0d5` |
|
|
12
|
+
| Tree at capture | clean |
|
|
13
|
+
| Node | `v26.13.0` |
|
|
14
|
+
| Platform | `darwin` |
|
|
15
|
+
|
|
16
|
+
## The rails, and what each returned
|
|
17
|
+
|
|
18
|
+
31 of the 32 recipes this workspace yielded at capture. The set is derived from [`cli/recipe-set.mjs`](../../cli/recipe-set.mjs) and never listed by hand.
|
|
19
|
+
|
|
20
|
+
| Recipe | Verdict |
|
|
21
|
+
|---|---|
|
|
22
|
+
| `cache-stability` | green |
|
|
23
|
+
| `docs` | green |
|
|
24
|
+
| `json` | green |
|
|
25
|
+
| `doctor` | green |
|
|
26
|
+
| `tests` | green |
|
|
27
|
+
| `plugin` | green |
|
|
28
|
+
| `compile` | green |
|
|
29
|
+
| `workflow-filters` | green |
|
|
30
|
+
| `index` | green |
|
|
31
|
+
| `context` | green |
|
|
32
|
+
| `ledger` | green |
|
|
33
|
+
| `control-chars` | green |
|
|
34
|
+
| `comments` | green |
|
|
35
|
+
| `rule-carriers` | green |
|
|
36
|
+
| `pack-version` | green |
|
|
37
|
+
| `pack-identity` | green |
|
|
38
|
+
| `payload` | green |
|
|
39
|
+
| `eval-bundle` | green |
|
|
40
|
+
| `goldens` | green |
|
|
41
|
+
| `mutants` | green |
|
|
42
|
+
| `fuzz-shell` | green |
|
|
43
|
+
| `version-carriers` | green |
|
|
44
|
+
| `skill-goldens` | green |
|
|
45
|
+
| `review-loop` | green |
|
|
46
|
+
| `telemetry` | green |
|
|
47
|
+
| `drills` | green |
|
|
48
|
+
| `ab` | green |
|
|
49
|
+
| `ab-grade` | green |
|
|
50
|
+
| `ab-run` | green |
|
|
51
|
+
| `tools/github:actions-pinned` | green |
|
|
52
|
+
| `tools/local-llm:runner-rails` | green |
|
|
53
|
+
|
|
54
|
+
**`release-eval` is excluded**, and the reason is printed rather than the row silently dropped: a capture cannot be accurate about the record it is inside — captured before the record is written it is red for the record's absence, and captured after it is stale the moment the register is rendered. Its verdict for this release is the rail's own run on the pull request carrying this file.
|
|
55
|
+
|
|
56
|
+
## The A/B baseline this release ships against
|
|
57
|
+
|
|
58
|
+
[`evals/ab/baseline.md`](https://github.com/sleepy-panda-srl/portulan/blob/v0.4.1/evals/ab/baseline.md), rendered from `evals/ab/baseline.json` — captured 2026-08-31T09:05:32.699Z at `a642d551eec6c146c67ef0acfe3f8ba535c5ce6e`, over a tree that was **not clean**.
|
|
59
|
+
|
|
60
|
+
**Its figures are not repeated here.** They have one carrier and this is not it.
|
|
61
|
+
|
|
62
|
+
## What this record does not establish
|
|
63
|
+
|
|
64
|
+
- **A green says this record agrees with its own capture — never that the release is good.** Every line below is what the recipes returned at one commit, and a recipe's own green establishes only what that recipe's documentation says it establishes.
|
|
65
|
+
- **The recipes were not run at the tag.** They ran at `a83966a56af78bd76a63c48afdce457aa3c5b0d5`, and this record is committed *in* the cut change — so the tagged tree is this commit plus whatever that change still adds after the capture, the record itself at minimum. A record cannot be captured at a commit that does not exist yet, and printing the one it was captured at is the only honest form.
|
|
66
|
+
- **`release-eval` is excluded from the rows above**, because a capture cannot be accurate about the record it is inside. Its verdict for this release is the rail's own run on the pull request that carries this file.
|
|
67
|
+
- **This record measures the Portulan repository's own build**, at the commit named above — never the workspace, project or package this release is installed into. Nothing here has looked at your tree.
|
|
68
|
+
- **The tree was clean at capture**, so the commit named above is what was measured.
|
|
69
|
+
- **The A/B baseline's figures are NOT restated here.** This record names which baseline the release ships against; the figures and everything that may not be concluded from them live in that register alone.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@sleepy_panda_srl/portulan",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.4.1",
|
|
4
4
|
"description": "Portulan — an agentic-engineering framework whose product is its files. The CLI wraps the file-based mechanics for hosts beyond Claude Code.",
|
|
5
5
|
"license": "Apache-2.0",
|
|
6
6
|
"type": "module",
|
|
@@ -16,6 +16,7 @@
|
|
|
16
16
|
"spec/",
|
|
17
17
|
"packs/",
|
|
18
18
|
"evals/releases/",
|
|
19
|
+
"evals/ab/personas/",
|
|
19
20
|
"README.md",
|
|
20
21
|
"LICENSE",
|
|
21
22
|
"NOTICE",
|
|
@@ -27,9 +28,12 @@
|
|
|
27
28
|
"!cli/ab-run.mjs",
|
|
28
29
|
"!cli/ab-grade.mjs",
|
|
29
30
|
"!cli/local-routing.mjs",
|
|
31
|
+
"!cli/acquire-personas.mjs",
|
|
32
|
+
"!cli/normalize-personas.mjs",
|
|
30
33
|
"!cli/warm.mjs",
|
|
31
34
|
"!cli/payload.mjs",
|
|
32
|
-
"!cli/roster.mjs"
|
|
35
|
+
"!cli/roster.mjs",
|
|
36
|
+
"!cli/cache-stability.mjs"
|
|
33
37
|
],
|
|
34
38
|
"repository": {
|
|
35
39
|
"type": "git",
|
|
@@ -18,8 +18,43 @@ sentence reads as a citation now and did not before.
|
|
|
18
18
|
|---|---|
|
|
19
19
|
| Skills | [`session-open`](skills/session-open/SKILL.md) · [`pre-commit`](skills/pre-commit/SKILL.md) · [`milestone-close`](skills/milestone-close/SKILL.md) |
|
|
20
20
|
| Persona | [`supervisor`](personas/supervisor.md) — grades, never implements or merges |
|
|
21
|
+
| Context admission | One ordinary fresh native Agent at a time per Claude session; see the scope below. |
|
|
21
22
|
| Gate fragments | `commit-without-the-hooks` (gated) · `self-certify-a-checkpoint` (prohibited) |
|
|
22
23
|
|
|
24
|
+
## Context admission
|
|
25
|
+
|
|
26
|
+
Pack Definition 1.1 declares `contributes.contexts.max_concurrent_subagents: 1`.
|
|
27
|
+
One supervisor at a time fits this sequential ritual; the three checkpoints can
|
|
28
|
+
still open three or more contexts over the work's lifetime. The ledger reports
|
|
29
|
+
actual contexts separately. Composed ritual counts tighten by minimum, independent
|
|
30
|
+
of pack order. An unresolved pack beside a known cap refuses compilation because
|
|
31
|
+
its contribution is unknown. No workspace `rituals` slot is introduced.
|
|
32
|
+
|
|
33
|
+
Claude compilation writes project `env.CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS`, even
|
|
34
|
+
without a gate policy. It applies across rituals within one session and excludes
|
|
35
|
+
the parent. It governs **ordinary fresh native Agent admission**, not all child
|
|
36
|
+
activity: completed-agent resumes and `/subtask` forks can exceed it; workflow
|
|
37
|
+
agents and team teammates have independent limits. Higher-precedence settings,
|
|
38
|
+
experimental flags and ultracode paths can bypass the guard. It bounds neither
|
|
39
|
+
lifetime children nor requests, time or tokens. Other adapters report the count
|
|
40
|
+
as not compiled; that is a limit of these adapters, not a claim about their hosts.
|
|
41
|
+
|
|
42
|
+
The [native trace](../../../cli/fixtures/ritual-cap/README.md) tests Claude Code
|
|
43
|
+
2.1.321 with synthetic loopback responses. The [current host documentation](https://code.claude.com/docs/en/sub-agents#concurrent-subagent-limit)
|
|
44
|
+
was checked on 2026-11-11. Neither establishes a universal hard ceiling. The
|
|
45
|
+
compiler composes caps with default restart advice in authored governing workspaces,
|
|
46
|
+
independently of a gate policy. It owns this dedicated native settings artifact
|
|
47
|
+
only while its complete payload, internal marker and digest remain intact; this
|
|
48
|
+
is accidental-edit detection, not authentication. Foreign content, additions,
|
|
49
|
+
links and old policy settings refuse before any output write. Formatting and key
|
|
50
|
+
order may normalize. Exact legacy cap-only output can migrate after validating
|
|
51
|
+
its shape and source/count consistency; it has no historical digest to authenticate
|
|
52
|
+
plausible edits. Removing a cap leaves advice settings in place. Missing manifests
|
|
53
|
+
and pointers do not adopt or remove those settings. The legacy internal cap-only
|
|
54
|
+
path removes its own file when its cap disappears. Normal policy compilation
|
|
55
|
+
retains its existing ownership. Removing both policy and cap while normal settings
|
|
56
|
+
still carry a cap refuses in every mode and preserves the file for explicit resolution.
|
|
57
|
+
|
|
23
58
|
## The three checkpoints
|
|
24
59
|
|
|
25
60
|
| Checkpoint | When | What it grades |
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"portulan": {
|
|
3
|
-
"pack": "1.
|
|
4
|
-
"version": "0.3.
|
|
3
|
+
"pack": "1.1",
|
|
4
|
+
"version": "0.3.1"
|
|
5
5
|
},
|
|
6
6
|
|
|
7
7
|
"name": "checkpoints",
|
|
@@ -10,6 +10,7 @@
|
|
|
10
10
|
"doc": "README.md",
|
|
11
11
|
|
|
12
12
|
"contributes": {
|
|
13
|
+
"contexts": { "max_concurrent_subagents": 1 },
|
|
13
14
|
"skills": ["skills/"],
|
|
14
15
|
|
|
15
16
|
"personas": ["personas/supervisor.md"],
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# `tools/local-llm` — Hybrid local-model execution under Portulan's rails
|
|
2
2
|
|
|
3
|
-
A tool pack for **Hybrid AI Orchestration**, argued in Proposal [`0039`](../../../.portulan/proposals/0039-a-local-model-runs-without-the-host-floor-and-under-a-thinking-cap.md). It lets a frontier orchestrator session offload bounded code transformations, verify-gated red→green retries and a screen of a curated slot for duplicate files to a local OpenAI-compatible runtime (such as **LM Studio** on `http://127.0.0.1:1234/v1` or **Ollama** on `http://127.0.0.1:11434/v1`)
|
|
3
|
+
A tool pack for **Hybrid AI Orchestration**, argued in Proposal [`0039`](../../../.portulan/proposals/0039-a-local-model-runs-without-the-host-floor-and-under-a-thinking-cap.md). It lets a frontier orchestrator session offload bounded code transformations, verify-gated red→green retries and a screen of a curated slot for duplicate files to a local OpenAI-compatible runtime (such as **LM Studio** on `http://127.0.0.1:1234/v1` or **Ollama** on `http://127.0.0.1:11434/v1`) with **no API tokens for the local model's own work** and a **zero-floor prompt prefix**. Each call to it is still a parent request, which re-reads the parent's context like any other (fact 1 below).
|
|
4
4
|
|
|
5
5
|
Composed through a workspace's `packs` array as `tools/local-llm`.
|
|
6
6
|
|
|
@@ -19,24 +19,58 @@ this one.
|
|
|
19
19
|
|
|
20
20
|
Proposal [`0036`](../../../.portulan/proposals/0036-what-a-host-loads-into-every-context-is-budgeted.md) budgets what a host loads into every fresh context, and Proposal [`0038`](../../../.portulan/proposals/0038-what-a-change-spends-is-measured.md) prices what a change spends across turns (`m_r × (F·T + g·T²/2)`). Together they expose two structural facts about hybrid local+cloud development:
|
|
21
21
|
|
|
22
|
-
1. **Parent-context turns are quadratic
|
|
22
|
+
1. **Parent-context turns are quadratic, and a local call is one of them.** Every tool call in a frontier session re-reads the entire session history, the one that runs the runner included, so a local call costs nothing only in the local model's own tokens. A one-span `transform` that replaces one parent Edit saves the few hundred output tokens between writing the instruction and writing the code, and a miss, a call that keeps nothing or a kept edit the parent then touches up, adds a whole request. One run measured in [#583](https://github.com/sleepy-panda-srl/portulan/issues/583) missed 20% of its calls, against a break-even of 5–12% at a 236k-token context and the price ratios the skill's [*When it pays*](skills/delegate-local/SKILL.md#when-it-pays) states, and spent more parent tokens than it saved. A 10-file comment cleanup sent as ten `transform` calls, each its own tool call, is ten parent requests; a loop over the ten inside one Bash call is one, and that batching is not measured.
|
|
23
23
|
2. **Local models choke on the host floor (`F`), not on the task.** A cloud subagent inherits a 44k–85k token prefix (`F`) subsidized by provider prompt caching. On local hardware, pre-filling 85k tokens into a 30B model's KV cache takes tens of seconds before the first token appears, whereas a **surgical prompt of a few hundred tokens** (built using [`cli/symbols.mjs`](../../../cli/symbols.mjs) spans) pre-filled in 0.85–1.9 seconds in proposal 0039's measurement.
|
|
24
24
|
|
|
25
25
|
## What the runner enforces in code
|
|
26
26
|
|
|
27
|
-
[`runner/local-llm.mjs`](runner/local-llm.mjs) has **zero npm dependencies** (Node.js `>=20` built-in `fetch` and `node:child_process`) and enforces
|
|
27
|
+
[`runner/local-llm.mjs`](runner/local-llm.mjs) has **zero npm dependencies** (Node.js `>=20` built-in HTTP, `fetch` and `node:child_process`) and enforces these rails:
|
|
28
28
|
|
|
29
29
|
1. **Loopback-only by default.** Only `127.0.0.1`, `localhost`, and `[::1]` are permitted unless `--allow-remote` is explicitly passed, and a redirect is refused rather than followed off the machine. A proxy listening on loopback that forwards elsewhere is not visible to it, which is what the `local-llm-off-loopback` gate fragment records.
|
|
30
30
|
2. **Zero-floor input budget (`--max-input-bytes`, default `32768`).** Refuses oversized prompts, and for `dedup` any file over the ceiling, before hitting the local server, directing the caller to scope with [`cli/symbols.mjs`](../../../cli/symbols.mjs) and `--lines` first. A value that is not a whole number is refused rather than read as no limit.
|
|
31
|
-
3. **Explicit thinking
|
|
31
|
+
3. **Explicit thinking request (`--effort low`).** Passes `reasoning_effort: "low"` and `enable_thinking: false` by default. Backend/template support determines their effect; sending these fields is not a hard reasoning-token cap or a guarantee against output-budget exhaustion.
|
|
32
32
|
4. **Verify-gated transforms (`transform --file <path> --verify <cmd>`, `--verify` required).** Applies the local model's edit, runs the verify command, retries with the end of its output on exit `1`, and **restores the original file byte-for-byte** unless the command reaches exit `0`. Any exit other than `0` or `1` — `2`, a missing command, a signal — is no verdict: the file is restored at once and the runner exits `2`. SIGINT, SIGTERM and SIGHUP, whether they arrive during the model call or while the verify command runs, stop the verify, restore the file and exit `2`. **The limits:** this is a rollback, not an atomic write — any other signal that ends the runner (SIGKILL and SIGQUIT among them) or a power loss between the edit and its verdict leaves the edit in place, and since the verify runs in a process group of its own, such a signal leaves the verify running too, even one sent to the runner's whole group. Signal handling is POSIX's: on Windows no SIGTERM arrives, and stopping the verify stops only its shell.
|
|
33
33
|
5. **Role screening (`eval`).** Grades an installed local model against Portulan's own [`cli/comments.mjs`](../../../cli/comments.mjs), [`cli/doctor.mjs`](../../../cli/doctor.mjs), and [`cli/skill-goldens.mjs`](../../../cli/skill-goldens.mjs) predicates across `code`, `codify`, and `handoff` tasks. It takes **one sample per task and has no baseline arm**, so it is a screen, not the A/B run `0038` requires before a role is routed: a model that fails it is not routed, and passing it earns nothing. It needs `cli/` beside `packs/`, as the repository and the npm package carry it, exiting `2` where only `packs/` was installed. The A/B run is [*The A/B record*](#the-ab-record) below.
|
|
34
34
|
|
|
35
|
-
|
|
35
|
+
Default loopback `/chat/completions` calls use a finite 15-minute deadline over the
|
|
36
|
+
connection, headers and complete response body, with a 256-KiB response limit.
|
|
37
|
+
This avoids Node fetch's five-minute header wait for non-streaming thinking models.
|
|
38
|
+
`localhost` is pinned to 127.0.0.1 without DNS; `[::1]` remains available. Redirects
|
|
39
|
+
are refused. Explicit injected fetches, authorized remote calls and embeddings
|
|
40
|
+
retain their existing transport behavior and response capacity. A caller injecting
|
|
41
|
+
its own transport can explicitly use the exported `localModelFetch`; the default
|
|
42
|
+
repair cannot change a transport hidden inside an injection.
|
|
43
|
+
|
|
44
|
+
A response ending with `finish_reason: length` is incomplete and is rejected before
|
|
45
|
+
a replacement or verifier can run. Workspaces may enforce stronger completion or
|
|
46
|
+
slice limits without changing earned routing or model profiles. The code extractor
|
|
47
|
+
accepts valid whitespace/language fences, preserves inline backticks, and rejects
|
|
48
|
+
an unfinished fence. Provenance: owner Marius, November 5, 2026; local thinking
|
|
49
|
+
generations were interrupted by a client deadline and valid generated fences were
|
|
50
|
+
misparsed. These repairs change transport/completion handling, not model quality.
|
|
51
|
+
|
|
52
|
+
A receipt prints `reasoning=` as a subset of `out=` and `cached-in=` as a subset
|
|
53
|
+
of `in=`; neither is added to its parent. Counts must be safe nonnegative integers
|
|
54
|
+
and subsets cannot exceed their reported parent. Missing, invalid or unverifiable
|
|
55
|
+
counts print `?`; reported zero stays zero. Transform attempts sum each measurement
|
|
56
|
+
once, with an unknown attempt making that sum unknown. `time=` is summed model
|
|
57
|
+
request time: the runner does not stream, so it cannot see a first token. The ledger
|
|
58
|
+
accepts both historical and extended receipts and continues counting parent totals.
|
|
59
|
+
Update runner and ledger together: older strict receipt readers do not understand
|
|
60
|
+
the added fields.
|
|
61
|
+
|
|
62
|
+
On `length`, the diagnostic distinguishes a measured all-reasoning response from
|
|
63
|
+
other combined-budget exhaustion. A short input does not cap thinking. Inspect the
|
|
64
|
+
backend's actual thinking controls and the task contract before another request;
|
|
65
|
+
do not silently lower a pinned profile, retry identical work, or accept truncated
|
|
66
|
+
output. An advertised reasoning-budget parameter needs a binding runtime test,
|
|
67
|
+
not just a short successful response. Provenance: the November 6 Q06 investigation
|
|
68
|
+
on [#578](https://github.com/sleepy-panda-srl/portulan/issues/578#issuecomment-6695966000),
|
|
69
|
+
with accounting on [#571](https://github.com/sleepy-panda-srl/portulan/issues/571).
|
|
36
70
|
|
|
37
71
|
## Screening a slot for duplicates
|
|
38
72
|
|
|
39
|
-
`dedup --dir <slot-dir> [--threshold 0.80] [--model <id>] [--max-input-bytes <per file>]` embeds every file directly in the directory whose name ends in `.md`, in any letter case, sorted by name, in one `POST /v1/embeddings` request to the loopback endpoint, with `text-embedding-nomic-embed-text-v1.5` (`PORTULAN_LOCAL_EMBED_MODEL`) unless `--model` names another; `--role` picks a chat model and is refused here. It prints each pair whose cosine similarity is at or above the threshold, highest first and ties in file-name order, one line each, `0.829 0036-….md 0038-….md`, and two lines on stderr: `dedup files=8 pairs=1 threshold=0.8`, followed by the shared receipt `OK mode=dedup model=… attempts=1 in=…t out=? time=…ms rate=? spend=$0.00`. The ledger reads the receipt as embedding work at $0.00; output tokens and rate are unreported, never zero.
|
|
73
|
+
`dedup --dir <slot-dir> [--threshold 0.80] [--model <id>] [--max-input-bytes <per file>]` embeds every file directly in the directory whose name ends in `.md`, in any letter case, sorted by name, in one `POST /v1/embeddings` request to the loopback endpoint, with `text-embedding-nomic-embed-text-v1.5` (`PORTULAN_LOCAL_EMBED_MODEL`) unless `--model` names another; `--role` picks a chat model and is refused here. It prints each pair whose cosine similarity is at or above the threshold, highest first and ties in file-name order, one line each, `0.829 0036-….md 0038-….md`, and two lines on stderr: `dedup files=8 pairs=1 threshold=0.8`, followed by the shared receipt `OK mode=dedup model=… attempts=1 in=…t out=? time=…ms rate=? reasoning=? cached-in=? spend=$0.00`. The ledger reads the receipt as embedding work at $0.00; output tokens and rate are unreported, never zero.
|
|
40
74
|
|
|
41
75
|
- **A slot's `README.md` is left out**, in any letter case: it describes the slot rather than being one of its entries, and would rank against every entry it summarizes. Subdirectories, symbolic links and other files are left out too.
|
|
42
76
|
- **The ceiling is per file**, since the model embeds each input on its own, and it measures the file's text: its size on disk before it is read, and its UTF-8 bytes after. A file over it refuses the run with exit `1`, naming every such file with its size, and nothing is truncated. The request as a whole is not capped: a slot of `n` files sends at most `n` times the ceiling of file text, which JSON escaping can enlarge in the request body, to about six times for control characters. Raise `--max-input-bytes` only as far as the embedding model's context holds.
|
|
@@ -53,6 +87,12 @@ supported role by task shape, gate every edit with `--verify`, and read only the
|
|
|
53
87
|
The acquired [record](../../../evals/ab/local-worker.md) supports `code` at low effort; shared `prose`
|
|
54
88
|
is disabled because codify failed all five local turns. `summarize` defaults to that disabled role and
|
|
55
89
|
refuses. An explicit `--model` without `--role` is an operator-selected candidate, not automatic routing.
|
|
90
|
+
Delegate one complete function or a small test group per call, specify one observable change, prefer
|
|
91
|
+
serial calls, and keep output to the selected replacement. On truncated output inspect the diagnostic and task contract; on transport
|
|
92
|
+
failure inspect the transport; on verifier failure inspect code. For reasoning-only
|
|
93
|
+
exhaustion, investigate backend/profile/task behavior before selecting another span.
|
|
94
|
+
Do not lower an operator-pinned
|
|
95
|
+
model/effort or bypass verification. This advice reduces failures, does not guarantee completion.
|
|
56
96
|
|
|
57
97
|
## The A/B record
|
|
58
98
|
|