provost 2.2.0__tar.gz → 2.3.0__tar.gz

{provost-2.2.0 → provost-2.3.0}/CHANGELOG.md

@@ -1,5 +1,407 @@
 # provost — Changelog
 
+## 2026-11-04 — release v2.3.0 (PV-003)
+
+`VERSION` 2.2.0 → 2.3.0, and the `man/provost.1` header. Everything below has
+landed since 2.2.0 (published to PyPI from the 2026-10-30 release entry): the
+entries above that one, from 2026-10-29 to 2026-11-04, hold the detail. Nothing
+in an existing store or an existing command line changes; 2.3.0 only adds.
+
+**Why now.** `pip install provost[lm]` only works from PyPI once a release
+carries the `lm` extra, and quarterdeck's `qd rm RUN` refuses on a provost
+without `rm --capture` (PyPI's 2.2.0 and older).
+
+**What's new since 2.2.0**
+- **`provost rm --capture ID[,ID...]`** deletes captures out of datasets that
+  other captures also filed into: the capture's directory and tags, its rows
+  and their provenance, in one commit or not at all (2026-11-02).
+- **`pip install provost[lm]`:** `--model` labels with the 18-layer tiny LM,
+  run on numpy alone, no torch. The weights come from the `provost-model`
+  package (0.1.0 on PyPI). `PROVOST_LAYOUT_LM=0` keeps the trees;
+  `PROVOST_LAYOUT_LM=DIR` uses a model of your own; `PROVOST_PYTHON` names the
+  interpreter that runs the layout model (2026-10-29, 2026-10-31, 2026-11-02).
+- **The wheel ships the layout model's LM runtime** (`lm_runtime.py`,
+  `lm_numpy.py`, `lm_encode.py`, `bpe.py`), so a pip-installed provost with no
+  model or no numpy labels with the trees, as a checkout does.
+- **`batch` is faster on a fleet's answers:** the first input of each name
+  sets a detection hint the others reuse when it gives the same table
+  (`PROVOST_BATCH_REUSE=0` turns it off), and each name's first input is
+  parsed side by side, `-j` at a time (2026-10-30). quarterdeck filing 200
+  hosts went from 70 s to 22 s.
+- **Fixed:** the smoke check on help lines wider than 79 columns could never
+  fail; it now counts them, and `provost next`'s line is shorter.
+- **ml/ (dev-only, not shipped):** `train_lm.py`, `export_lm.py`,
+  `fold_eval.py --learner lm`, `model_round.sh`, and a validation split that
+  no longer puts every validation family in fold 1.
+- **Work is ticketed** (PV-001, ST-004, ST-010): `CLAUDE.md`'s "Every task is
+  a ticket" section. Docs only.
+
+**Publishing.** Run the "Release to PyPI" workflow (`release.yml`) on `main`,
+or publish a GitHub release tagged `v2.3.0`, which runs the same workflow —
+one or the other: PyPI refuses a second upload of 2.3.0. Then, in a fresh
+venv, `pip install 'provost[lm]'` should give provost 2.3.0 and
+provost-model, and `provost rm --help` should list `--capture`.
+
+## 2026-11-04 — Step 3 of the ticket section is one command line (ST-010)
+
+Step 3 of `CLAUDE.md`'s "Every task is a ticket" section now puts its
+`ticket.py new` command on one line, in a code block of its own, as
+shared_tools' copy does since ST-008. It used to wrap across two lines inside
+one code span, so pasting it into a shell ran two commands and failed.
+Workflow docs only, no code and no version change.
+
+- **`CLAUDE.md`:** step 3 only; nothing else in the file changes.
+- **`HANDOFF.md`:** unchanged; its step 0s is still accurate.
+
+## 2026-11-04 — The ticket section of CLAUDE.md is the current text (ST-004)
+
+`CLAUDE.md`'s "Every task is a ticket" section is replaced with the current
+text from shared_tools' `CLAUDE.md`, again verbatim with this repository's
+name filled in. Workflow docs only, no code and no version change.
+
+- Step 1 now says to pull an existing clone of `shared_tools` rather than only
+  to clone it: a stale clone read an old `CLAUDE.md` and ran an old
+  `ticket.py`.
+- Step 3 names the `shared_tools` clone's script (`<shared_tools
+  clone>/scripts/ticket.py`, not a path in this repository) and passes
+  `--request`.
+- The columns read Backlog → In Progress → Review → Done: Adversarial Test is
+  now only on the owner's request (ST-005).
+- **`CLAUDE.md`:** that section only; nothing else in the file changes.
+- **`HANDOFF.md`:** next step 0s mentions the pull and the columns.
+
+## 2026-11-03 — Every task is a ticket on the shared board (PV-001)
+
+provost's work is now tracked as tickets in
+[shared_tools](https://github.com/MartinGallagher-code/shared_tools)
+(`tickets/provost/`, prefix PV), on the shared kanban board. Workflow docs
+only, no code and no version change.
+
+- **`CLAUDE.md`:** the "Every task is a ticket" section, verbatim from
+  shared_tools' `CLAUDE.md` with this repository's name filled in, directly
+  after the opening paragraph.
+- **`HANDOFF.md`:** next step 0s.
+
+## 2026-11-02 — `rm --capture`: delete captures, and the rows they filed
+
+`provost rm NAME` deletes whole datasets. Nothing could delete one capture
+out of a dataset that others also filed into, and quarterdeck needs exactly
+that: its `qd rm RUN` deletes one run, whose answers sit in datasets beside
+other runs' answers.
+
+- **`provost rm [-f] --capture ID[,ID...]`** (repeatable; an ID as
+  elsewhere: `c0007`, `7`, `capture-c0007`) deletes those captures: each
+  `captures/<id>/` and its tags (`capture-<id>`, and `files-<id>` where it
+  kept a controlled directory's files), the rows it filed from every
+  dataset it fed,
+  and their provenance rows. A dataset left with no rows goes too. It all
+  goes in one commit, like any `rm`.
+- It asks first, like `rm`, unless `-f`. It refuses the whole operation if
+  any id is not a capture, and refuses dataset names given beside
+  `--capture`.
+- A row is a capture's when provenance says so. provost's capture column
+  (column 2, after `#`; `_capture` beside a data column named `capture`) is
+  read only for a row provenance does not know, because a reconciled
+  dataset can hold a data value in that position. A data column named
+  `capture` or `#` is never matched.
+- It commits the whole deletion or leaves the store as it was. What it
+  touches must already be committed, untracked files included, and no
+  other git may hold the store's index (it refuses otherwise, and says
+  so).
+  Every dataset and `provenance.tsv` is rewritten in a scratch directory
+  first, and any failure after that, up to and including the commit, puts
+  those paths back from HEAD, with signals ignored and tried again, after
+  a pause, if git is interrupted or locked out all the same. A commit that
+  landed although git then failed counts as done, and the index is brought
+  into line with it. It stages
+  and commits those paths alone: anything else uncommitted in the store,
+  staged or not, is left as it was. INT, TERM and HUP wait for the outcome
+  and then set the exit status, from the first change to the last message
+  (tags and the report included). Everything that changes the store runs
+  in a child in a process group of its own, so a Ctrl-C at the terminal
+  reaches only the waiting shell and never a git part way through. Under a
+  held-down Ctrl-C (INT to the group every 5 ms, from anywhere in the run)
+  150 runs ended committed or unchanged and none in between, where the
+  same code without its own process group left 14 of 75 half-applied.
+  The child holds the store's lock as its own while it works (taking the
+  owner record over only while it is still its shell's, and handing it
+  back only while it is still its own, so another provost's lock is never
+  overwritten; both under the lock-break mutex, waited for briefly (3 s at
+  most) and cleared once when over a minute old as `_lock_break` does, so a stale-lock
+  break cannot fall between the compare and the write; if the lock was taken
+  from it before it changed anything, it stops there, nothing deleted),
+  so if the shell that took it is killed the lock stays live
+  until the child is done. A child killed on its own is waited for (its
+  git may still be running), its record taken back, and the outcome read
+  from HEAD. What made it fail is shown. Tags go only once the
+  commit has landed, and if they cannot go it says so. The
+  scratch directory goes on any exit. The datasets and rows are found again
+  under the lock, so one deleted while it waited at the prompt is skipped,
+  and the counts it prints are the rows that actually go.
+- The ids go through a file, not the environment or the command line, and
+  the tags go in one `git update-ref --stdin`. 750 captures out of 1,501
+  take 3 s.
+- Files: `provost` (`cmd_rm`, `_rm_captures`, `_rm_cap_scan`,
+  `_rm_cap_apply`, `_rm_cap_commit`, `rm --help`, `help advanced`;
+  `_release_lock_dir` reads the lock's owner with `read`, not `$(cat)`, so
+  a Ctrl-C cannot leave a lock it was releasing), `docs/usage.md`.
+- Tests (`tests/smoke.sh`, +84): refusals (missing, malformed, mixed with
+  names, uncommitted changes), the prompt and its count, which rows and
+  datasets go and which stay (data columns named `capture` and `#`;
+  provenance against a reconciled capture column; no provenance; no final
+  newline), failure at each stage and INT/TERM/HUP (the store as it was or
+  the deletion committed, the exit status, tags kept on failure and said
+  so, the lock released, no scratch dir, a Ctrl-C to the group during the
+  put-back, a put-back retried, a commit that landed, a commit killed
+  after landing, another git holding the index, a Ctrl-C while the tags
+  go, the shell killed outright mid-commit, the child killed after
+  landing, another provost's lock left alone, a lock taken from it
+  before it began, the cause of a failure
+  shown),
+  its `files-<id>` tag, untracked and
+  hidden-untracked files refused, other uncommitted and staged files left
+  alone, a dataset that
+  went while it waited, provenance, dirs, tags, one clean commit, ids never
+  reused.
+
+## 2026-11-02 — The tiny LM without torch: `provost[lm]` and the `provost-model` package
+
+`pip install provost[lm]` gives `--model` the 18-layer tiny LM, run with
+numpy alone. The weights live in their own package and repository
+([provost_model](https://github.com/MartinGallagher-code/provost_model),
+85 MB, Apache-2.0), not in this repository. `provost-model` 0.1.0 is on PyPI
+(2026-11-02). In a fresh venv, `pip install provost[lm]` from this
+release's wheel pulls it from there, and `--model` labels with it.
+
+- **`extract/model/bpe.py`:** SmolLM2's byte-level BPE tokenizer in plain
+  Python (the GPT-2 pattern over `unicodedata` classes). It gives ids
+  identical to transformers on every piece of the corpus and on 20,000 random
+  strings.
+- **`extract/model/lm_numpy.py`:** the Llama forward pass and the two heads
+  on numpy.
+  - It matches transformers to 1e-4 on the same weights; an exported float
+    model makes every decision torch does.
+  - Weights stay at 8 bits in memory and are widened one matrix at a time.
+    Peak memory is 282 MB, the load takes 0.5 s, and torch is never
+    imported.
+- **`extract/model/lm_encode.py`:** the echo encoding, moved out of
+  `lm_runtime.py` (which re-exports it) so the numpy path is stdlib plus
+  numpy.
+- **`ml/export_lm.py`:** merges the LoRA adapter and keeps the trained
+  layers. Linear layers and the vocabulary table are stored at 8 bits with
+  one scale per row, in `.npz` shards under 45 MB. `--check N` compares
+  with torch; `--embed 4` gives a 4-bit table.
+- **How faithful the 8-bit model is:**
+  - **Held out on fold 0,** the 30-layer model exported this way scores
+    0.863 and 39/48 gold, the same as torch (0.863, 39/48). A 4-bit
+    vocabulary table scored 0.865 (41/48).
+  - **On round 4,** the 18-layer model's outputs are byte-identical to
+    torch's on 84 of 107 captures; the 4-bit table manages 65 of 107.
+  - **Rated blind,** the 23 that differ rate the same as torch's, so the
+    total is the same: 5 / 30 / 72, and 0 / 2 / 16 on the gate-held 18.
+- **Finding the model:**
+  - with `PROVOST_LAYOUT_LM` unset, `layout_model.load_lm` uses an installed
+    `provost-model`;
+  - `PROVOST_LAYOUT_LM=0` keeps the trees;
+  - a directory `export_lm.py` wrote runs on numpy, one `train_lm.py` saved
+    on torch.
+- **`PROVOST_PYTHON`:** the interpreter that runs the layout model
+  (`provost`, `layout-model.sh`). The pip launcher sets it to its own, so a
+  `provost-model` installed beside provost is found.
+- **`pyproject.toml`:**
+  - the `lm` extra (`numpy`, `provost-model`);
+  - the wheel ships `bpe.py`, `lm_encode.py` and `lm_numpy.py`.
+- **Checked from wheels in a fresh venv:**
+  - `pip install provost[lm]` resolves both packages;
+  - `provost --model` labels with the package's model;
+  - `PROVOST_LAYOUT_LM=0` gets the trees back.
+- **Tests:**
+  - `ml/tests/test_lm_numpy.py` (10: the tokenizer, quantizers, export
+    layout, causality, against transformers' Llama, no torch import);
+  - smoke +2 (`PROVOST_LAYOUT_LM=0`, a missing `PROVOST_PYTHON`), +2 for the
+    new setting in help and the man page: 495;
+  - launcher +1;
+  - CI's python-package job installs numpy and runs the numpy tests.
+- **Documented:** README (`provost[lm]`, the two settings), `help advanced`,
+  the man page, `ml/README.md`, `docs/lm-2026-10-29-tiny-labeller.md`.
+
+## 2026-10-31 — The 18-layer tiny LM on round 4: the same as 30 layers
+
+`lm-all-l18` (the user's final model: 18 of 30 layers, every fold, a 14.5 MB
+adapter) was run model-first on round 4 and rated blind. The 42 outputs
+byte-identical to an earlier model's keep its rating; the other 65 were rated
+afresh.
+- **The total:** 5 / 30 / 72, against 5 / 31 / 71 for the 30-layer `lm-all`
+  and 12 / 51 / 44 for the default engine.
+- **Head to head with `lm-all`:** 94 the same, 6 better, 7 worse.
+- **The 18 gate-held captures:** 0 / 2 / 16, the same as `lm-all`.
+- **Conclusion:** 18 layers costs nothing on whole captures either, so it is
+  the model to use with `PROVOST_LAYOUT_LM`. Its blank result on round 3's
+  `cmake-cache-advanced-help` is a one-off.
+
+Files: `docs/lm-2026-10-29-tiny-labeller.md`, `HANDOFF.md`.
+
+## 2026-10-29 — A tiny pretrained LM as the layout labeller, trained on a CPU
+
+An experiment, not yet in provost's capture path: can an ultra-small
+pretrained language model, fine-tuned on provost's own labels, get past the
+~83% plateau of the from-scratch students (trees, GridNet)?
+
+- **`ml/train_lm.py`:**
+  - fine-tunes SmolLM2-135M (LoRA rank 16 on every linear layer, plus two
+    heads; 5.55M trained of 140M) to give the trees' two answers, so
+    `decode.py` and the scoring are reused;
+  - echoes the capture, so a causal model sees the rows below a header;
+  - cuts each line at its field-start candidates before tokenizing, so
+    offsets map to tokens exactly.
+  - `--batch-tok` (2,048 default; 6,000 ran out of 15 GB), `--checkpointing`,
+    `--full`, `--base`, checkpoints in `--save DIR/{last,best}`.
+  - `--start-step N` with `--init DIR/last` resumes a run a worker restart
+    killed, at its step in the schedule and the data; checkpoints record
+    their step (test +1).
+  - `--device auto|cpu|cuda|mps`: auto trains on an NVIDIA GPU, else an
+    Apple one (an M-series Mac's Metal GPU), else the CPU; `lm_runtime.run`
+    labels on whatever device the model is on (test +1).
+  - `--layers N` keeps only the base's first N transformer layers, so a
+    smaller model can be measured (6 of 30: 51M weights instead of 140M).
+    The count is saved in `config.json`, and `lm_runtime.load_model`
+    rebuilds the same cut; models without it load whole (tests +2).
+  - **The layer experiment** (the user's iMac, 2026-10-30/31):
+    - **18 of 30 layers matches the full model on all five folds:** 0.862
+      both (12,476 against 12,477 of 14,468 lines), gold 246 against 242
+      of 287, at 68% of the parameters (95.6M) and 63% of the training
+      time.
+    - **12 layers is not safe:** on fold 4 it scored 0.796, below the
+      engine.
+    - **On fold 0, 6 and 3 layers** scored 0.821 and 0.796.
+    - **The final 18-layer model, `lm-all-l18`:**
+      - every fold, 14 min, a 14.5 MB adapter, validation loss 0.540
+        against 0.563 at 30 layers;
+      - trained without `--no-val`, so on 3,271 captures;
+      - it loads and labels through `lm_runtime`, but on round 3's
+        `cmake-cache-advanced-help` its labels file nothing where the
+        trees and the 30-layer fold-0 model file a table. Its
+        whole-capture behaviour is unmeasured.
+- **`ml/fold_eval.py --learner lm [--lm-load DIR]`** scores it on the usual
+  fold. `--limit N` scores the first N held-out docs for a quick look.
+- **`ml/tests/test_train_lm.py`:** 7 tests on a tiny random model (no
+  download); they skip without torch, transformers and peft, so CI's
+  stdlib-only job is unaffected.
+- **Measured** (`docs/lm-2026-10-29-tiny-labeller.md`):
+  - about 220 training tokens a second on 4 cores, about an hour per pass over
+    fold 0's 2,493 training captures, 9 GB RAM;
+  - fold 0 (627 held-out captures, 3,163 lines): **tiny LM 0.863, 39/48
+    gold**, against trees 0.829 and engine 0.814 (both 32/48), and the
+    documented trees + GridNet average of 0.835: the first learner past the
+    plateau;
+  - averaged with the trees: 0.865, 38/48;
+  - two passes, about 2 h on 4 cores;
+  - continued on generated layouts (`ml/gen`): 0.854, 37/48, so the corpus
+    labels stay the training set;
+  - **round 4, whole captures, model-first** (no engine, no gate), rated
+    blind: tiny LM 7 / 30 / 70, trees 2 / 32 / 73, default engine
+    12 / 51 / 44. Neither model replaces the engine; on the 18 captures the
+    gate holds, the only ones `--model` touches, the LM rates 0 / 5 / 13
+    against the engine's 0 / 1 / 17;
+  - **round 4 again with the user's all-data model `lm-all`:** 5 / 31 / 71
+    against fold 0's 7 / 30 / 70. The 39 captures with byte-identical output
+    carry their ratings; the other 68 were rated blind. Head to head: 93
+    the same, 6 better, 8 worse; on the 18 gate-held captures, 0 / 2 / 16
+    against 0 / 5 / 13. More training data does not move whole captures;
+  - **folds 1–4, trained on the user's iMac** (Apple M4, `--device` picked
+    its GPU, about 4.5 times the container's speed, 15.5–19 min a fold).
+    The tiny LM wins every fold, 0.842–0.893. Over all five folds (14,468
+    lines): LM 0.862, trees 0.819, engine 0.814; gold 240 / 212 / 189 of 287.
+    Averaging with the trees does not pay. Worker restarts had killed fold 1
+    four times in the container.
+  - **Found by those runs:**
+    - every validation family fell in fold 1 (`is_val`'s `% 10 == 1` implies
+      `fold`'s `% 5 == 1`); fixed below;
+    - `fold_eval.py` crashes on macOS with two `libomp` copies loaded; not
+      fixed, but work around it with `OMP_NUM_THREADS=1` and `--threads 1`
+      (noted in `ml/README.md`).
+- **The validation split** (`ml/dataset.py` `is_val`):
+  - **What it was:** `crc32(family) % 10 == 1` put all 403 validation
+    captures in cross-validation fold 1. So fold 1 trained with no validation
+    set, and every other fold picked its epoch on the same families.
+  - **What it is:** a salted hash, still never a test family. 346 captures of
+    125 families (of 1,261), spread 91 / 101 / 69 / 85 over folds 1–4, so
+    every fold now validates on 245–346 captures.
+  - **What it changes:** LM and GridNet results recorded before this used
+    the old split and may move slightly if retrained. The trees do not use
+    it, so the shipped model is unchanged.
+  - `train_lm.py --no-val` holds nothing back: the final model, on every
+    capture, for the 2 epochs cross-validation chose (`split`).
+  - Tests +2.
+- **The PyPI wheel ships `extract/model/lm_runtime.py`** (`pyproject.toml`).
+  - Before, a pip-installed provost with `PROVOST_LAYOUT_LM` set failed the
+    import, and `--model` declined.
+  - Now it labels with the trees when there is no model or no torch, as a
+    checkout does.
+  - CI's python-package job checks this from the installed wheel.
+- **`tests/benchmark/model_records.py` + `model_round.sh --roundN OUT [--lm
+  DIR]`:** the layout model alone on a benchmark round, filed as `--model`
+  files it (one table or key/value record, else the model's own sections).
+- `ml/README.md`: files table and reproduce steps.
+- **Capture time, opt-in:**
+  - **`extract/model/lm_runtime.py`:** the encoding and the model now live
+    beside `layout.py` and `decode.py`, shared by training and capture time.
+    `train_lm.py` imports them. `lm_runtime.load` returns None when torch,
+    transformers or peft or the model is missing.
+  - **`PROVOST_LAYOUT_LM=DIR`** (`layout_model.py`: `load_lm`, `label_lm`):
+    `--model` labels with the tiny LM saved in DIR instead of the trees,
+    without the arbiter (it was fitted to the trees' probabilities). With no
+    model or no torch, the trees label as before (smoke +1).
+  - Documented in `help advanced` and the man page.
+- **Fixed:** the smoke check "no help line is wider than 79 columns" could
+  never fail: an empty expected substring matches anything. It now counts
+  the wide lines. It found one, `provost next`, now shortened.
+
+## 2026-10-30 — `batch` parses each name's first input side by side
+
+The detection reuse just above parsed the first input of every name one
+after another, before the parallel parse of the rest began. A batch of one
+name barely notices; a batch of many does. quarterdeck's `qd probe facts
+inventory` files the answers to 7 probes and its own `qd.hosts` and
+`qd.runs` in one batch: 9 names, so 9 full detections in turn while the
+other CPUs waited. The first parses now run `-j` at a time, as the rest do,
+and are waited for in the order they started (`cmd_batch`,
+`_batch_first_done`). On six hosts that request went from 26 s to 16 s. The
+captures, their order, and each name's hint are as before.
+
+- Tests (`tests/smoke.sh`, +3): two names in one batch file the same rows
+  with `-j 1` and `-j 8`, and the inputs after each name's first record
+  the reused detection.
+
+## 2026-10-30 — `batch` reuses a name's detection: a fleet's answers, three times faster
+
+A fleet's answers to one command are one format, and detection -- trying
+every strategy -- is most of the engine's time on a small input (0.44 s of
+it for a three-line `key: value` answer, against 0.06 s with the strategy
+named). quarterdeck filing 200 hosts' answers took 70 s; it takes 22 s.
+
+- **Detection reuse** (`cmd_batch`, `_ingest_parse`, `_batch_hint`). The
+  first input of each name that has others after it is parsed on its own.
+  When it is read cleanly -- one table, nothing held, no auto-multi, model
+  or template, coverage measured -- its strategy, coverage and header line
+  are a hint the others try first (`-s STRATEGY`), each kept only if the
+  engine gives the same strategy, the same header and coverage no lower;
+  anything else is detected from scratch, as before. A reused capture's
+  `meta.tsv` says `detection  reused (STRATEGY, from the batch's first input
+  of this name)`. Off with `PROVOST_BATCH_REUSE=0`, and under `--multi`,
+  `--model` and `--llm`.
+- **Why it is safe.** On 400 corpus inputs, the winner pinned gave a table
+  byte for byte the same as full detection on 376; on the other 24 (a hint
+  row's label, which is not a strategy file, or lower coverage when pinned)
+  the acceptance rule refuses the pinned result, and the input is detected
+  from scratch. A 41-input batch (40 alike and an error message) files the
+  same datasets either way, in 5.1 s against 13.1 s.
+- `_read_engine_meta` reads the engine's meta line for both paths.
+- Tests (`tests/smoke.sh`, +7): the same rows with and without reuse; the
+  meta line on the inputs after the first; none on the first, on an input
+  the hint does not fit, or with `PROVOST_BATCH_REUSE=0`; the setting in
+  the help and the man page. 485 passed.
+
 ## 2026-10-30 — release v2.2.0
 
 `VERSION` 2.1.0 → 2.2.0, and the `man/provost.1` header. Everything below has

{provost-2.2.0 → provost-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: provost
-Version: 2.2.0
+Version: 2.3.0
 Summary: Turn command output into data: capture, auto-detect the format, present as a tidy git-backed dataset
 Project-URL: Homepage, https://github.com/MartinGallagher-code/provost
 Project-URL: Documentation, https://provost.readthedocs.io
@@ -22,6 +22,9 @@ Classifier: Programming Language :: Unix Shell
 Classifier: Topic :: System :: Systems Administration
 Classifier: Topic :: Utilities
 Requires-Python: >=3.8
+Provides-Extra: lm
+Requires-Dist: numpy; extra == 'lm'
+Requires-Dist: provost-model; extra == 'lm'
 Description-Content-Type: text/markdown
 
 # provost
@@ -50,6 +53,16 @@ The PyPI package (`provost`) bundles the
 complete tool — the `provost` entry point plus its extract/analyze/transform
 engines — and a thin Python launcher execs it in place. Requires bash and git.
 
+```bash
+pip install provost[lm]
+```
+
+This adds a better reader for `--model`: a tiny pretrained language model
+fine-tuned on provost's own labels. It comes as the
+[`provost-model`](https://github.com/MartinGallagher-code/provost_model)
+package (85 MB) and runs on numpy, with no torch or GPU. Without it,
+`--model` uses the small decision trees provost ships.
+
 ## Quick start
 
 ```bash
@@ -190,6 +203,8 @@ index — including stores on remote (SFTP/SSH) hosts.
 | `PROVOST_CONTEXT_COLUMNS` | `0` | `1` = each `key=value` word of a `#%` prologue becomes a column on the capture's rows (same as `--context-columns`) |
 | `PROVOST_AUTO_MULTI` | `1` | segment a parse the quality gate would hold, when the parts are clearly better (`0` = hold it; same as `--no-auto-multi`) |
 | `PROVOST_MODEL` | `0` | `1` = when the gate would still hold a parse, try the learned layout model and file its table if it passes the same gate (needs `python3`; same as `--model`) |
+| `PROVOST_LAYOUT_LM` | unset | the tiny LM `--model` labels with: a model directory, or unset for the installed `provost-model` package (`pip install provost[lm]`); `0` = the trees only |
+| `PROVOST_PYTHON` | `python3` | the interpreter that runs the layout model (the pip launcher sets it to its own) |
 | `PROVOST_LLM` | `0` | `1` = parse with a template Claude wrote for that tool (regexes, or a small boxed Python parser for records that span lines; stored in the store's `llm-templates/`, or shipped), before a generic parse is trusted or a capture held; with no fitting template and an API key, ask Claude to write one. Writing one sends the capture to the Anthropic API, once per tool; lines a template sets aside that still carry a word or a number (titles, version banners, summary counts, errors) are filed beside its datasets as `NAME.notes`, never dropped; later captures of the tool are parsed offline, except one that shows a kind of line the template has not met, which (with a key) is sent once with the template's earlier captures so Claude can extend it (repair; kept only if it still reads them) (same as `--llm`) |
 | `PROVOST_LLM_MODEL` | `claude-opus-5-5` | the model that writes templates |
 | `PROVOST_LLM_API_KEY` | — | Anthropic API key for `--llm` (else `ANTHROPIC_API_KEY`; prefer this one where `ANTHROPIC_API_KEY` is taken, e.g. in a Claude Code environment, which keeps that name for its own sign-in) |

{provost-2.2.0 → provost-2.3.0}/README.md

@@ -24,6 +24,16 @@ The PyPI package (`provost`) bundles the
 complete tool — the `provost` entry point plus its extract/analyze/transform
 engines — and a thin Python launcher execs it in place. Requires bash and git.
 
+```bash
+pip install provost[lm]
+```
+
+This adds a better reader for `--model`: a tiny pretrained language model
+fine-tuned on provost's own labels. It comes as the
+[`provost-model`](https://github.com/MartinGallagher-code/provost_model)
+package (85 MB) and runs on numpy, with no torch or GPU. Without it,
+`--model` uses the small decision trees provost ships.
+
 ## Quick start
 
 ```bash
@@ -164,6 +174,8 @@ index — including stores on remote (SFTP/SSH) hosts.
 | `PROVOST_CONTEXT_COLUMNS` | `0` | `1` = each `key=value` word of a `#%` prologue becomes a column on the capture's rows (same as `--context-columns`) |
 | `PROVOST_AUTO_MULTI` | `1` | segment a parse the quality gate would hold, when the parts are clearly better (`0` = hold it; same as `--no-auto-multi`) |
 | `PROVOST_MODEL` | `0` | `1` = when the gate would still hold a parse, try the learned layout model and file its table if it passes the same gate (needs `python3`; same as `--model`) |
+| `PROVOST_LAYOUT_LM` | unset | the tiny LM `--model` labels with: a model directory, or unset for the installed `provost-model` package (`pip install provost[lm]`); `0` = the trees only |
+| `PROVOST_PYTHON` | `python3` | the interpreter that runs the layout model (the pip launcher sets it to its own) |
 | `PROVOST_LLM` | `0` | `1` = parse with a template Claude wrote for that tool (regexes, or a small boxed Python parser for records that span lines; stored in the store's `llm-templates/`, or shipped), before a generic parse is trusted or a capture held; with no fitting template and an API key, ask Claude to write one. Writing one sends the capture to the Anthropic API, once per tool; lines a template sets aside that still carry a word or a number (titles, version banners, summary counts, errors) are filed beside its datasets as `NAME.notes`, never dropped; later captures of the tool are parsed offline, except one that shows a kind of line the template has not met, which (with a key) is sent once with the template's earlier captures so Claude can extend it (repair; kept only if it still reads them) (same as `--llm`) |
 | `PROVOST_LLM_MODEL` | `claude-opus-5-5` | the model that writes templates |
 | `PROVOST_LLM_API_KEY` | — | Anthropic API key for `--llm` (else `ANTHROPIC_API_KEY`; prefer this one where `ANTHROPIC_API_KEY` is taken, e.g. in a Claude Code environment, which keeps that name for its own sign-in) |

provost-2.3.0/VERSION

@@ -0,0 +1 @@
+2.3.0

provost-2.3.0/extract/model/bpe.py

@@ -0,0 +1,159 @@
+# SPDX-License-Identifier: GPL-3.0-or-later
+# SPDX-FileCopyrightText: 2026 Martin J. Gallagher
+"""A byte-level BPE tokenizer read from a Hugging Face tokenizer.json, in
+plain Python: what the tiny LM needs at capture time without transformers.
+
+    tok = Tokenizer.load("tokenizer.json")
+    tok.encode("Filesystem  Size")    # -> [token ids], no special tokens added
+
+It covers the tokenizer the tiny LM was trained with (transformers loads
+SmolLM2's as a GPT2Tokenizer): no normalizer; added tokens split out first;
+then ByteLevel's GPT-2 pattern; then BPE merges over the byte-to-unicode
+alphabet. The model directory carries that tokenizer as transformers had it
+(ml/export_lm.py writes it), not the Hub's tokenizer.json, whose Digits step
+transformers does not apply. The GPT-2 pattern needs Unicode letter and
+number classes, which the re module lacks, so it is a small scanner over
+unicodedata categories. ml/export_lm.py --check compares it with
+transformers on every piece of the corpus.
+"""
+import json
+import unicodedata
+
+# Unicode White_Space, which the pattern's \s means
+_SPACE = set("\t\n\x0b\x0c\r \x85\xa0 

  　") | {
+    chr(c) for c in range(0x2000, 0x200B)}
+_CONTRACTIONS = ("s", "t", "re", "ve", "m", "ll", "d")
+
+
+def _cat(ch):
+    """'L' letter, 'N' number, 'S' whitespace, 'O' anything else."""
+    if ch in _SPACE:
+        return "S"
+    c = unicodedata.category(ch)[0]
+    return c if c in "LN" else "O"
+
+
+def _byte_alphabet():
+    """GPT-2's reversible map from bytes to printable characters."""
+    keep = list(range(ord("!"), ord("~") + 1)) + list(range(ord("¡"), ord("¬") + 1)) + \
+        list(range(ord("®"), ord("ÿ") + 1))
+    out, n = {}, 0
+    for b in range(256):
+        if b in keep:
+            out[b] = chr(b)
+        else:
+            out[b] = chr(256 + n)
+            n += 1
+    return out
+
+
+def split_gpt2(text):
+    """GPT-2's pattern, alternatives tried in order at each position:
+    's|'t|'re|'ve|'m|'ll|'d| ?\\p{L}+| ?\\p{N}+| ?[^\\s\\p{L}\\p{N}]+|\\s+(?!\\S)|\\s+"""
+    out, i, n = [], 0, len(text)
+    cats = [_cat(ch) for ch in text]
+    while i < n:
+        ch = text[i]
+        if ch == "'":
+            hit = next((c for c in _CONTRACTIONS if text.startswith(c, i + 1)), None)
+            if hit:
+                out.append(text[i:i + 1 + len(hit)])
+                i += 1 + len(hit)
+                continue
+        j = i + 1 if ch == " " and i + 1 < n and cats[i + 1] != "S" else i
+        k = cats[j] if j < n else None
+        if k in ("L", "N", "O") and not (j == i and k == "S"):
+            e = j
+            while e < n and cats[e] == k:
+                e += 1
+            out.append(text[i:e])
+            i = e
+            continue
+        # whitespace: the run, minus its last character when a non-space follows
+        e = i
+        while e < n and cats[e] == "S":
+            e += 1
+        if e < n and e - i > 1:
+            e -= 1
+        out.append(text[i:e])
+        i = e
+    return out
+
+
+class Tokenizer:
+    def __init__(self, vocab, merges, added=()):
+        self.vocab = vocab
+        self.ranks = {tuple(m.split(" ", 1)) if isinstance(m, str) else tuple(m): r
+                      for r, m in enumerate(merges)}
+        self.added = sorted(added, key=lambda a: -len(a[0]))   # (content, id), longest first
+        self.bytes = _byte_alphabet()
+        self.cache = {}
+
+    @classmethod
+    def load(cls, path):
+        with open(path, encoding="utf-8") as fh:
+            t = json.load(fh)
+        m, pre = t["model"], t.get("pre_tokenizer") or {}
+        if (m.get("type") != "BPE" or t.get("normalizer") or m.get("byte_fallback")
+                or pre.get("type") != "ByteLevel" or not pre.get("use_regex", True)
+                or pre.get("add_prefix_space")):
+            raise ValueError(f"{path}: not a plain byte-level BPE tokenizer")
+        return cls(m["vocab"], m["merges"], [(a["content"], a["id"]) for a in t.get("added_tokens", [])])
+
+    def _bpe(self, word):
+        """Ids of one pre-token (already mapped to the byte alphabet)."""
+        hit = self.cache.get(word)
+        if hit is not None:
+            return hit
+        parts = list(word)
+        while len(parts) > 1:
+            best, at = None, -1
+            for k in range(len(parts) - 1):
+                r = self.ranks.get((parts[k], parts[k + 1]))
+                if r is not None and (best is None or r < best):
+                    best, at = r, k
+            if best is None:
+                break
+            pair = (parts[at], parts[at + 1])
+            merged, k = [], 0
+            while k < len(parts):
+                if k < len(parts) - 1 and (parts[k], parts[k + 1]) == pair:
+                    merged.append(parts[k] + parts[k + 1])
+                    k += 2
+                else:
+                    merged.append(parts[k])
+                    k += 1
+            parts = merged
+        ids = [self.vocab[p] for p in parts]
+        self.cache[word] = ids
+        return ids
+
+    def _plain(self, text):
+        ids = []
+        for word in split_gpt2(text):
+            ids.extend(self._bpe("".join(self.bytes[b] for b in word.encode("utf-8"))))
+        return ids
+
+    def encode(self, text):
+        """Token ids of text, no special tokens added (added tokens that occur
+        in the text are still matched, as transformers does)."""
+        if not self.added or not any(a in text for a, _ in self.added):
+            return self._plain(text)
+        ids, i, start = [], 0, 0
+        while i < len(text):
+            hit = next(((a, t) for a, t in self.added if text.startswith(a, i)), None)
+            if hit:
+                if start < i:
+                    ids.extend(self._plain(text[start:i]))
+                ids.append(hit[1])
+                i += len(hit[0])
+                start = i
+            else:
+                i += 1
+        if start < len(text):
+            ids.extend(self._plain(text[start:]))
+        return ids
+
+    def __call__(self, texts):
+        """The Encoder's tokenize: a list of strings -> a list of id lists."""
+        return [self.encode(t) for t in texts]