@turri-dev/cli 0.1.8-next.bcca62a → 0.1.8-next.bfcb24e

package/docs/config.md

@@ -47,6 +47,7 @@ Each change says when it takes effect. None needs turri started again - what tur
 | the next test run - the running pulse's too | the testing scripts, `testing-scripts-run-on` |
 | the next pulse that lands - the running one too | `release-after-pulse` |
 | the next regeneration that names no `.plain` file | `module` |
+| the next time turri starts here | `ledger-keep` |
 
 A coding model chosen this session - `--coding-model`, or `engine` - gives way to one set here: the last word wins. It says, too, what wins over the change: `models.chat` over `harness-model` (and so on), and a stack's `config.turri.<stack>.yaml`, which is not changed here, while that stack is chosen. In a project still read from `config.yaml`, the first change copies it to `config.turri.yaml` first - see [the old config.yaml](#the-old-configyaml).
 
@@ -91,6 +92,7 @@ models:                              # each agent role's model, thinking and tur
 | `conformance-tests-script` | the conformance testing script, by path from the project root | looked for - see below |
 | `prepare-environment-script` | optional: run as `<script> <build>` before every run of the conformance testing script, which runs only once it passes - see [the testing scripts](regenerating.md#the-testing-scripts). turri never writes one | the one `prepare_environment*.sh` at the root or in `test_scripts/`, else none |
 | `compact-at` | how many tokens of work - past its instructions - an agent kept through a job holds before the harness compacts it between turns; at least 1000; the project's alone | 100,000 - see [agent sessions](regenerating.md#agent-sessions) |
+| `ledger-keep` | how many megabytes of the ledger - what turri did, as facts, in `.turri/ledger` - the project keeps: past it, the oldest runs' are pruned when a run first writes; `0` keeps none, and writes none; the project's alone. See [the ledger](using.md#the-ledger) | 100 |
 | `max-specs-per-pulse` | the most functional specs the planner groups into one pulse - what builds one feature goes together, up to this many; a stack's file wins | 3 |
 | `max-lines-per-pulse` | the most lines of spec in one pulse - its functional specs with the definitions they bring; a group past it is split again, a functional spec a pulse; a stack's file wins | 120 |
 | `parallel-jobs` | how many jobs the chat runs at once - modules that share nothing build side by side up to it; read when the chat starts, from the folder it starts in | half this machine's cores, between one and four - see [how many jobs at once](#how-many-jobs-at-once) |
@@ -229,7 +231,7 @@ How a stack's config is made:
 - **Laid over `config.turri.yaml`, key by key.** What the stack's file gives wins; what it leaves out is the project's. A model it names takes what it leaves out from the project's same model - above, with the stack frontend chosen, everything renders on the project's coding model, thinking `high`.
 - **Pinned to a module when it is asked for.** A module is built with the stack it was asked for with, from its first pulse to its last: the pulses planned after it, its test runs, the scripts found, its plans and its conformance tests, whatever stack is chosen meanwhile. A pulse waiting for a free slot keeps the stack it was queued with. A change to its spec keeps its stack - unless it was built with none and has a stack of its own now; asked for again, its stack is worked out again. What `regenerate` says back names each module's stack, and so does a pulse's report.
 - **Refused with none, when that cannot build.** In a project of stacks, a module asked for with no stack - none named, none named after it, none chosen - is refused before anything is planned when `config.turri.yaml` alone names no unit testing script: every pulse would stop for want of one. The refusal names the stacks and the script each names. A spec change `write_spec_diff` writes is never refused: it is written, and said not to be built yet - passed the stack, everything written is built. `stack: none` builds with `config.turri.yaml` alone all the same.
-- **The project's alone.** `module:`, `harness-model:`, `parallel-jobs:`, `rules-off:` and `compact-at:` are read only from `config.turri.yaml` - there is one conversation, one module built when none is named, one machine. `agent-env:` and `toolchains:` in a stack's file add to the project's.
+- **The project's alone.** `module:`, `harness-model:`, `parallel-jobs:`, `rules-off:`, `compact-at:` and `ledger-keep:` are read only from `config.turri.yaml` - there is one conversation, one module built when none is named, one machine. `agent-env:` and `toolchains:` in a stack's file add to the project's.
 
 The chat is told each stack's config and the one chosen. The skills that write testing scripts put each stack's in its own file once a project has two stacks, each stack named after the module it builds.
 

package/docs/development.md

@@ -57,6 +57,7 @@ The workflow signs in to npm with npm's trusted publishing (OIDC), not a token:
 | `src/screen-padding.ts` | the blank row above and below the whole screen and the two columns either side, in both of pi's ways of drawing it |
 | `src/regenerate/spec-map.ts`, `src/regenerate/spec-map-view.ts` | regen: each item's mark, from the records, the `.plain` files and the work loop's plan (`mapPlan`), and the map drawn |
 | `src/regenerate/debug-log.ts` | each render agent's whole conversation, always streamed to `.turri/debug/` |
+| `src/regenerate/ledger.ts`, `src/regenerate/ledger-view.ts`, `src/ledger-tools.ts`, `src/ledger-cli.ts` | the ledger: what turri did, as typed facts, in `.turri/ledger/` - `record()`, read back for the `ledger` tool, `pulses`, `diagnose` and `turri ledger` - see [the ledger](using.md#the-ledger) |
 | `genome/` | what turri knows and can do: `genome/skills/`, the skill library, and `genome/rules/`, the \*\*\*plain authoring rules |
 
 ## Known edges

package/docs/hosting.md

@@ -16,6 +16,7 @@
 | `highlightLines`, `roleOf` | a spec's lines coloured by turri's own \*\*\*plain grammar. |
 | `functionalityCompletion`, `createWorkLoop`, `createJobRegistry`, `promptOf`, `describeModules` | how far each functional spec is built, and - on a loop of the host's own with `dryRun` - the work a spec change would plan, as `turri glimpse` does. |
 | `onModelEvent`, `describeTurnIncident` | a provider failing and waited out, a turn turri ended itself. |
+| `onLedgerEvent`, `readLedger`, `ledgerDir`, `runViews`, `pulseView`, `describePulseView`, `describeRuns`, `ledgerProjects`, `earlierPulses`, `asJob` | what turri did, as facts - the [ledger](using.md#the-ledger): each event heard as it is recorded, and every run and pulse read back, earlier sessions' too - see [the ledger in a host](#the-ledger-in-a-host). |
 | `sweepOrphanedGroups` | stop what a turri killed outright left running, before anything starts. |
 
 It is turri's own code, as the terminal runs it: a 0.x release may change any of it.
@@ -64,6 +65,33 @@ the chat, and the toolbar's buttons send them too.
 - The Run button releases what is built and opens it.`;
 ```
 
+## The ledger in a host
+
+The conversation's extension starts the ledger itself, in the folder the host made the session in - so a host that runs it there, the project's, needs nothing to have one: its pulses are recorded as `host` runs, and the conversation's `pulses` and `diagnose` already tell the model of earlier sessions' pulses. What the host can do with it is show it.
+
+```js
+import { onLedgerEvent, pulseView, readLedger, runViews } from "@turri-dev/cli/host";
+
+// Each event as it is recorded - written or not, so a project with ledger-keep: 0 is heard too.
+const stop = onLedgerEvent((entry, project) => {
+  if (entry.kind === "healer.failed") notify(`the ${entry.healer} failed: ${entry.error}`);
+  if (entry.kind === "provider.error" && !entry.transient) notify(`the model provider: ${entry.error}`);
+});
+
+// The project's runs, the most recent first, each with its pulses - how each ended, how long, what it cost.
+const runs = runViews(projectDir);
+
+// One pulse, from this session or an earlier one: its rounds, test runs, verdicts, healers, refusals, history.
+const view = pulseView("646d", readLedger(projectDir, { job: "646d" }));
+
+// Earlier sessions' pulses as jobs, to list beside this session's - none this session's registry knows.
+const { jobs, more } = earlierPulses(ledgerProjects(projectDir), new Set(registry.list().map((job) => job.id)));
+```
+
+- **Live.** `onLedgerEvent(listener)` is told every event as turri records it, with the project it is about, until what it returns is called; a listener that throws misses that event and nothing else. It is the pulse's own detail - its stages, test runs, verdicts, refusals - where `onJobChange` is the job as a whole.
+- **Read back.** `readLedger(project, { job, run, kinds, since })` is the project's events, oldest first, the gzipped runs too; `runViews` and `pulseView` gather them into runs and pulses, and `describePulseView` and `describeRuns` say them in lines, as `turri ledger` prints them. `ledgerProjects(dir)` finds the ledgers of a folder and the projects under it. `earlierPulses(projects, known, runs = 3)` gives the last runs before this one as `Job`s - the shape a host already draws for this session's - each marked `earlier: { run, runMode, lines }`, `lines` what the ledger says of it; `more` is how many runs are beyond them. They are read only: never started, halted or waited for. turri's own side panel lists them so, as **earlier sessions**.
+- **What it is for.** Looking up, never deciding: what is built and what is left come from the records and the work loop, as before. Its events and their fields are `LedgerEvent`'s - a kind is added, never changed, and every event carries `v: 1`.
+
 ## Running it
 
 The host entry is compiled JavaScript, so any Node from 22.19 runs it - Electron's own included; nothing strips types. It finds turri's own folders - `genome/`, `stdlib/`, `.pi/` - beside `dist/` in the package, wherever npm installed it.

package/docs/installation.md

@@ -93,3 +93,4 @@ For Gemini, `GEMINI_API_KEY` is what makes pi *offer* the google provider, but w
 | --- | --- |
 | `PLAIN_TEMPLATE_DIRS` | colon-separated template directories, e.g. the standard library |
 | `TURRI_COMMAND_TIMEOUT` | seconds a command may run before it is stopped - an agent's own, its tests (default `120`) |
+| `TURRI_LEDGER` | `off`: write no [ledger](using.md#the-ledger) at all, whatever `ledger-keep:` says - the test suite sets it |

package/docs/regenerating.md

@@ -462,7 +462,7 @@ Two records per module, each committed with what it describes in the module's `.
 | `code/.turri.json` | the code record: the spec the code is built from - its whole requires chain's: the definitions and implementation reqs of the modules below, each tagged with its module, their functional specs among the ones it inherits, and its own - without the test reqs and the acceptance tests, which are for conformance only |
 | `tests/.turri.json` | the tests record: the functional specs the conformance tests are written for - of the code record's, with their acceptance tests - the definitions and implementation reqs they were written against, and the test reqs they follow |
 
-Nothing about how a build went: whether tests passed and which files a job touched are in the history. And nothing else is stored: what is left to build is worked out from the records whenever a job ends or the spec changes - the code work from the code record to the spec now (`src/regenerate/drift.ts`), the conformance work from the tests record to what the tests are to be written for. Only the running job is kept; what the panel shows as planned is worked out again. The tests record never goes ahead of the code's: a conformance test exercises functionality, and only what is in the code can be tested.
+Nothing about how a build went: whether tests passed and which files a job touched are in the history, and how each pulse went is in the [ledger](using.md#the-ledger), which nothing reads to decide what to build. And nothing else is stored: what is left to build is worked out from the records whenever a job ends or the spec changes - the code work from the code record to the spec now (`src/regenerate/drift.ts`), the conformance work from the tests record to what the tests are to be written for. Only the running job is kept; what the panel shows as planned is worked out again. The tests record never goes ahead of the code's: a conformance test exercises functionality, and only what is in the code can be tested.
 
 What goes into them:
 

package/docs/using.md

@@ -36,7 +36,8 @@ While a skill runs, a line above the input shows its phases, every one by name -
 | `engine` | show or choose the model the code is regenerated on - see [below](#choosing-a-model) |
 | `pulses` | the pulses - running, planned and finished - or one of them |
 | `halt` | halt a pulse, a module's planned pulses, or what `activate` started |
-| `diagnose` | why a pulse did not land, from its transcript: how it ended, the stages that went wrong, the last failing test run, the last review, the tool calls that failed |
+| `diagnose` | why a pulse did not land, from its transcript: how it ended, the stages that went wrong, the last failing test run, the last review, the tool calls that failed - and what the [ledger](#the-ledger) recorded of it |
+| `ledger` | what turri did, from the [ledger](#the-ledger), this session's and every earlier one's: a pulse as facts, by its id - its rounds, test runs, verdicts, healers, refusals and commits; events of some kinds counted and listed, for a module or since a time - `healer.failed`, or `healer` for the group; or the last runs, a line a pulse |
 | `lineage` | a module's lineage: what each regeneration and conformance run changed in it, and what it was regenerated from |
 | `cultivate` | run a gated skill, its steps and review gates enforced |
 | `release`, `activate` | try what is regenerated: release it to `dist/<module>` and activate it on this machine, installing what it needs - see [deploy](deploy.md) |
@@ -80,7 +81,7 @@ A render takes minutes, so in the TUI every render tool starts the work as a pul
 
   One module is shown at a time, with a tab for each module of the chain on the line under the top line, in the chain's order - the module it is built on to the left: `d365_version_api  [d365_http_client]  d365_utils 12 ›`. A tab regenerating spins, one stuck says `!`; what does not fit is counted at either end. `←` `→` go from one tab to the next. The module shown is the one in focus - the one regenerating now, else the one asked for, else the first - until you pick another tab; it goes back to following once another module is in focus, or the one in focus starts regenerating. Under the module's own text is what it imports, under **imported**, a folded block a file, `▸ from stdlib/backend/python-fastapi · 7 reqs · all built`, closed until opened. The modules it requires are tabs of their own: a chain of many modules is never drawn nested, one in the next. A map longer than the screen has a minimap down its right edge, a cell a slice in the colour of its most pressing mark, the slice shown solid. The map is read from the files - the code record, the `.plain` files, the tests record - so it is right the moment turri opens a project, before anything runs; the work loop's plan adds the pulses on top. It maps the files asked to build this conversation, else `module:` in `config.turri.yaml`, else the `.plain` files of the folder that no other one there imports or requires - the folder's files looked at again only once one of them, or its config, changes. `↑` `↓` move row by row, `enter` or `space` open and close an item or an import block - a functional spec opens to its sub-bullets and its `***acceptance tests***`, nested under it as the spec has them - `←` `→` go to the next module's tab, `o` and `c` open or close everything shown, `pgup` `pgdn` and the wheel scroll, `g`/`G` go to the top and the bottom.
 
-  `p` goes to the **pulses** - and back - onto the pulse that regenerates the item under the cursor, and back onto the first item the pulse under the cursor regenerates. The pulses are every pulse as three sections - running, queued, finished - each opened and closed on its own (running and queued start open). A pulse is one line: its module, kind and number (`hubspot · regenerate pulse 1`, `hubspot · conformance pulse 2`) - so it is named wherever a pulse is named in a line (the job panel, the `pulses` tool, `/halt`, the command line), never by what it regenerates, which is in it, opened - its time (none while it is queued: it has not started) and its cost - a finished one's mark green when it passed, red when it failed or did not pass, amber when it was stopped. Opened, a running one shows what it is doing; a queued one, what it waits for; a finished one, its agents and what each spent, and whether it did not pass. Under it is a row for each file its spec diff changes - the file of the module being rendered first and open, the rest by name and closed - each opening onto that file's lines. A file with nothing added or taken out is not listed. `↑` `↓` move, `enter` or `space` open and close what is under the cursor, `→` `←` open and close (`←` on a closed pulse goes to its section, on a closed file to its pulse), `o` and `c` open or close every pulse of the section - or, on a file, every file of its pulse - `pgup` `pgdn` and the wheel scroll, `g`/`G` go to the top and the bottom. The cursor stays on its row when the pulse under it moves on; a pulse opened closes when it moves to another section. What is open is kept while the panel is closed. What the agents said and did is not shown - it is in each pulse's log (see [Agent logs](#agent-logs)).
+  `p` goes to the **pulses** - and back - onto the pulse that regenerates the item under the cursor, and back onto the first item the pulse under the cursor regenerates. The pulses are every pulse as three sections - running, queued, finished - each opened and closed on its own (running and queued start open) - and a fourth, **earlier sessions**, closed, when the [ledger](#the-ledger) has pulses of sessions before this one: its last three runs', each pulse read only - opened, what the ledger says of it (how it ended, its rounds and test runs, the healers, what turri refused and committed, its last stages and its log) - and a line for how many more runs it has. A pulse is one line: its module, kind and number (`hubspot · regenerate pulse 1`, `hubspot · conformance pulse 2`) - so it is named wherever a pulse is named in a line (the job panel, the `pulses` tool, `/halt`, the command line), never by what it regenerates, which is in it, opened - its time (none while it is queued: it has not started) and its cost - a finished one's mark green when it passed, red when it failed or did not pass, amber when it was stopped. Opened, a running one shows what it is doing; a queued one, what it waits for; a finished one, its agents and what each spent, and whether it did not pass. Under it is a row for each file its spec diff changes - the file of the module being rendered first and open, the rest by name and closed - each opening onto that file's lines. A file with nothing added or taken out is not listed. `↑` `↓` move, `enter` or `space` open and close what is under the cursor, `→` `←` open and close (`←` on a closed pulse goes to its section, on a closed file to its pulse), `o` and `c` open or close every pulse of the section - or, on a file, every file of its pulse - `pgup` `pgdn` and the wheel scroll, `g`/`G` go to the top and the bottom. The cursor stays on its row when the pulse under it moves on; a pulse opened closes when it moves to another section. What is open is kept while the panel is closed. What the agents said and did is not shown - it is in each pulse's log (see [Agent logs](#agent-logs)).
 
   `ctrl+h` again, or `esc`, closes it and goes back to the chat. Beside the chat, one side has the keys and the other is drawn faint: `ctrl+h` switches sides - from the chat to the panel, from the panel to the chat, the panel staying and following the pulses - `esc` gives the chat the keys too, and a click gives them to the side clicked. `ctrl+c` closes the panel; so do `s`, over the whole screen, then `esc`. (pi's regular screen, `--tui-mode regular`, reads no mouse: there it is the keys alone.) While the panel has the keys, every key is the panel's, whatever else happens: a question the chat model asks in the background (`ask_user`) waits in the chat, unanswered - an `enter` in the panel never answers it - and the panel's top line says "the chat is waiting for your answer". Back in the chat, the question is there, with the keys.
 - `turri` runs pi in fullscreen mode, so the conversation stays where you scroll it while the pulses redraw (PgUp/PgDn, the wheel); `--tui-mode regular` goes back to the terminal's own scrollback.
@@ -170,6 +171,31 @@ Beside them, each pulse has a log of its own in `.turri/jobs/` - `<time>-<pulse
 
 No agent can read the logs: they are outside `plain_modules` - or, for a `--build-dir` outside it, beside that folder rather than in it. Nothing prunes them either - delete the folder when you are done with it. A log that cannot be written is given up on without failing the render.
 
+## The ledger
+
+The logs keep everything a pulse's agents said, for reading one pulse closely. The ledger keeps what turri did, as facts, across pulses, sessions and runs - to answer *why did last night's run fail*, *how often does the debugger fail*, *what did this module cost*, long after the session that ran it is gone (`src/regenerate/ledger.ts`).
+
+It is in `.turri/ledger/` in the project, a JSON line an event, one file a run - a conversation, a headless run, a host's session: `<time>-<run>-p<pid>.jsonl`. Each line has the time with its date, its order in the run, the run, and - where they apply - the pulse, the module and the agent. What it records:
+
+| | |
+| --- | --- |
+| runs | `run.started` (how it was started, turri's version and commit, the machine), `run.ended` |
+| pulses | `job.queued`, `job.started`, every `job.stage`, `job.retried`, `job.finished` - its status, whether it landed, how long it ran, what it cost, how it ended |
+| the work loop | `loop.halted`, `loop.stopped` (a pulse the spec moved on from), `planner.failed`, `planner.fellBack` |
+| the model | `model.call` (time and tokens), `model.turn`, `model.compaction`, `provider.retry`, `provider.stall`, `provider.error`, `turn.stopped` (a turn turri ended itself, and why) |
+| tools and tests | `tool.call` (the tool, how long, whether it failed - never its arguments or result), `script.run` (the testing script, its exit, how long, whether the time limit stopped it, the failure's fingerprint, and `mutated` for a `can_it_fail` run on code broken on purpose) |
+| the gates | `gate.round`, `review.verdict`, `arbiter.ruling`, `judge.verdict` |
+| what turri refused | `guard.refused` (a repeated call, a call past its budget, a tool the turn does not have), `command.refused`, `audit.putBack` |
+| the healers | `healer.started`, `healer.ended` with what it found, `healer.failed` and why - the debugger, the diagnosis of a failed pulse |
+| the history | `history.committed`, `history.failed`, `history.reset`, `history.rolledBack` |
+| the environment | `environment.error` |
+
+Never in it: prompts, replies, thinking, tool arguments and results, file contents. What an event does carry - a stage, an error's first line - is masked as the logs are.
+
+`turri ledger` prints the last runs, a line a pulse; `turri ledger <pulse>` one pulse - its rounds, its test runs and how many failed the same way, the verdicts, the healers, what turri refused, what its history recorded, its log; `--json` the events themselves, for `jq`. In the chat, `ledger` is the way in - a pulse of this session or an earlier one, the events of some kinds counted (how often the debugger failed this week: `kinds: [healer.failed], since: 7d`), the last runs - and `pulses` lists the last three earlier runs under this session's pulses, `pulses` and `diagnose` tell one of them by its id, and `diagnose` adds the ledger's facts to a pulse of this session; the side panel's pulses list has them as **earlier sessions** (see [the side panel](#in-a-conversation)).
+
+It is for looking up, never for deciding: what to build comes from the records alone (see [what the code is built from](regenerating.md#what-the-code-is-built-from)), so deleting `.turri/ledger/` loses nothing but what can be looked up. When a run first writes to the project, the files of runs that have ended are gzipped and the oldest pruned past `ledger-keep:` megabytes - 100 unset, `0` for none at all; a run still going is never touched. `.turri` ignores itself, so the ledger is never committed, and nothing of it is sent anywhere. A ledger that cannot be written is said once and given up on, never failing the work.
+
 ## From the command line
 
 The same work, without a conversation. Run it from the directory that holds the spec, so its linked resources resolve the way the parser expects. For a long run - a whole spec overnight - see [running unattended](headless.md).
@@ -185,6 +211,7 @@ turri regenerate <file.plain> --from-scratch-chain   # the same for every module
 turri spec-diff <file.plain> <diff>    # apply a spec diff, and regenerate it
 turri conform <file.plain> 1           # black-box tests for functionality 1
 turri lineage <module>                 # what each regeneration changed in a module, and when
+turri ledger [pulse]                   # what turri did here: the last runs and their pulses, or one pulse
 turri cultivate brood                  # run a skill directly - by name, the project's too, or by its folder
 turri release <file.plain>             # release what is regenerated to dist/<module>, and activate it here
 ```

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@turri-dev/cli",
-  "version": "0.1.8-next.bcca62a",
+  "version": "0.1.8-next.bfcb24e",
   "license": "MIT",
   "type": "module",
   "repository": {