jevify 0.1.0__tar.gz → 0.2.0__tar.gz

jevify-0.2.0/Makefile

@@ -0,0 +1,50 @@
+# Shortcuts for the two everyday jevify commands: `make ask` asks typed questions
+# about one state and prints the answer as JSON; `make serve` serves Jev's HTTP API
+# for one recipe. Values reach jevify through the environment, so a state may hold
+# spaces, quotes and apostrophes. `make help` lists the variables and the recipes.
+
+JEVIFY ?= uv run jevify
+HOST   ?= 127.0.0.1
+PORT   ?= 8600
+
+export RECIPE STATE CHOICE SCORE NOUL IMAGE RUNG HOST PORT
+
+.DEFAULT_GOAL := help
+.PHONY: help ask serve
+
+help:
+	@echo "make ask   RECIPE=<file> STATE=<text|@file> [CHOICE=...] [SCORE=...] [NOUL=...] [IMAGE=<png|jpg|webp>] [RUNG=<rung>]"
+	@echo "make serve RECIPE=<file> [HOST=$(HOST)] [PORT=$(PORT)]"
+	@echo ""
+	@echo "  CHOICE  id:question=option1,option2,...   one option from a menu"
+	@echo "  SCORE   id:question=level0,level1,...     a position on ordered levels, low to high"
+	@echo "  NOUL    id:statement                      the probability that the statement is true"
+	@echo "  Give at least one of CHOICE, SCORE, NOUL. For several of one type, call jevify ask directly."
+	@echo "  make reads \$$ in a value: write \$$\$$49.99 for \$$49.99, or put the text in a file and pass STATE=@path."
+	@echo "  HOST other than 127.0.0.1 needs JEVIFY_API_KEY set: clients must then send it as a bearer key."
+	@echo ""
+	@echo "Example:"
+	@echo "  make ask RECIPE=recipes/gemma-4-e4b-it.llamacpp.yaml \\"
+	@echo "    STATE=\"Customer: I was charged twice. Fix it today or I cancel.\" \\"
+	@echo "    CHOICE=\"dept:Which team should handle this?=billing,support,sales\" \\"
+	@echo "    SCORE=\"urgency:How urgent is this?=low,medium,high\" \\"
+	@echo "    NOUL=\"cancel:The customer threatens to cancel.\""
+	@echo ""
+	@echo "Recipes:"
+	@ls recipes/*.yaml | sed 's/^/  /'
+
+ask:
+	@test -n "$$RECIPE" || { echo "make ask: set RECIPE=recipes/<name>.yaml (make help lists them)" >&2; exit 2; }
+	@test -n "$$STATE" || { echo "make ask: set STATE=\"...\" (the text, or @path to a file)" >&2; exit 2; }
+	@set -- --state "$$STATE"; \
+	[ -z "$$CHOICE" ] || set -- "$$@" --choice "$$CHOICE"; \
+	[ -z "$$SCORE" ] || set -- "$$@" --score "$$SCORE"; \
+	[ -z "$$NOUL" ] || set -- "$$@" --noul "$$NOUL"; \
+	[ -z "$$IMAGE" ] || set -- "$$@" --image "$$IMAGE"; \
+	[ -z "$$RUNG" ] || set -- "$$@" --rung "$$RUNG"; \
+	$(JEVIFY) ask "$$RECIPE" "$$@"
+
+serve:
+	@test -n "$$RECIPE" || { echo "make serve: set RECIPE=recipes/<name>.yaml (make help lists them)" >&2; exit 2; }
+	@case "$$HOST" in 127.0.0.1|localhost|::1) ;; *) test -n "$$JEVIFY_API_KEY" || { echo "make serve: HOST=$$HOST opens the API to the network; export JEVIFY_API_KEY first" >&2; exit 2; } ;; esac
+	$(JEVIFY) serve "$$RECIPE" --host "$$HOST" --port "$$PORT"

{jevify-0.1.0 → jevify-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: jevify
-Version: 0.1.0
+Version: 0.2.0
 Summary: Make a pretrained model you already have answer typed questions in one pass, and measure how well it does.
 Project-URL: Homepage, https://github.com/fidecastro/jevify
 Author: Felipe Infante de Castro
@@ -39,7 +39,8 @@ Jev does, and measure how well it does.**
 [![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/)
 
 You bring a model behind an OpenAI-compatible endpoint (vLLM, llama.cpp,
-anything with logprobs) or a small model that runs in-process. jevify sends it
+TensorFold, anything that returns logprobs for a reply token or for the
+prompt) or a small model that runs in-process. jevify sends it
 a **state** once, then asks **typed questions** about it: a choice among
 options, a score on ordered levels, a yes/no with a probability. Each answer
 is read straight off the model's next-token distribution, no generation, no
@@ -81,10 +82,34 @@ jevify ask recipes/gemma-4-e4b-it.llamacpp.yaml \
 jevify serve recipes/gemma-4-e4b-it.llamacpp.yaml --port 8600
 ```
 
+Any model behind an OpenAI-compatible server works, not only the ones under
+`recipes/`: copy the nearest recipe, set `endpoint.base_url` and
+`model.name`, and run the probe, which re-verifies the answer token ids for
+that model and writes what it found into the file. vLLM and llama.cpp get
+their dialect extras; any other server is driven through the plain OpenAI
+surface. A server that refuses logprobs on the reply token but returns
+vLLM's `prompt_logprobs`, such as TensorFold's native server with
+GLM-5.3-Flash, is read through the `prompt_logprobs` rung, which needs a
+raw-mode recipe (the template written out with the model's own control
+tokens, served on `/v1/completions`).
+
+`serve` binds to `127.0.0.1`. To reach it from other machines pass
+`--host 0.0.0.0`, and set `JEVIFY_API_KEY` first so that every request must
+carry that bearer key; to leave it running, detach it and keep its log:
+
+```sh
+jevify serve recipes/<name>.yaml --host 0.0.0.0 --port 8600 > ~/jevify.log 2>&1 &
+curl -s http://<this-host>:8600/health
+```
+
 Every answer carries Jev's fields (`choice`, `confidence`, `probabilities`,
 `score`, `legend`, `noul`) plus an `x_jevify` block naming the readout method,
 the cached tokens, the latency and the recipe hash that produced it.
 
+From a clone, `make ask` and `make serve` wrap the two commands;
+[`docs/playbook.md`](https://github.com/fidecastro/jevify/blob/main/docs/playbook.md)
+walks through them and explains every answer field.
+
 ## How it works
 
 1. **Recipe.** A hashed YAML file: the endpoint, the prompt template (chat
@@ -99,7 +124,9 @@ the cached tokens, the latency and the recipe hash that produced it.
    that reuses the server's prefix cache. A 32k-token state costs one warm and
    then a few hundred milliseconds per question.
 4. **Readout ladder.** Named-token logprobs, grammar with post-sampling
-   probabilities, top-k, equal bias, and a floored top-k as the last resort.
+   probabilities, top-k, prompt logprobs (each answer token appended to the
+   prompt and its logprob read back, for servers that give none on the reply
+   token), equal bias, and a floored top-k as the last resort.
    Auto walks down the proven rungs; the answer names the rung that read it
    and whether it was degraded.
 5. **Evidence.** `jevify eval` runs a frozen suite with controls (shuffled
@@ -110,7 +137,7 @@ the cached tokens, the latency and the recipe hash that produced it.
 
 | Kind | Talks to | Proven live with |
 |---|---|---|
-| `endpoint` | Any OpenAI-compatible server; dialect extras for vLLM and llama.cpp | DeepSeek-V4-Flash on vLLM; Ternary Bonsai 2 27B, Gemma 4 E4B and Qwen3-VL-Reranker 8B/2B on llama.cpp; text and images |
+| `endpoint` | Any OpenAI-compatible server that returns reply-token or prompt logprobs; dialect extras for vLLM and llama.cpp | DeepSeek-V4-Flash on vLLM; Ternary Bonsai 2 27B, Gemma 4 E4B and Qwen3-VL-Reranker 8B/2B on llama.cpp; text and images |
 | `embedding` | A `/v1/embeddings` route, or a model in-process | Qwen3-Embedding-0.6B in-process |
 | `encoder` | An NLI sequence-classification head in-process | facebook/bart-large-mnli in-process |
 | `rerank` | A `/v1/rerank` route | Fakes only |
@@ -123,6 +150,28 @@ Four frozen suites, every recipe on each, one run each, latencies per
 question on one machine on one day. Every row links to the scorecard that
 records the command, the recipe hash, the suite hash and the raw file digest.
 
+### Latency per question
+
+The median of each suite, combined across the suites a model ran; one run
+each. A question is one request that reads the answer off the next-token
+probabilities, after the state was sent once and cached.
+
+| Model | How it runs | Hardware | Latency per question |
+|---|---|---|---|
+| facebook/bart-large-mnli | in-process NLI classifier | RTX 4090 | 7–9 ms, avg 8 |
+| Qwen3-Embedding-0.6B | in-process embedding, cosine | RTX 4090 | 8–10 ms, avg 9 |
+| Gemma 4 E4B (Q8_0) | llama.cpp, chat | RTX 4090 | 28–37 ms, avg 32 |
+| Qwen3-VL-Reranker-2B (Q8_0) | llama.cpp, yes/no readout | RTX 4090 | 47–64 ms, avg 53 |
+| Qwen3-VL-Reranker-8B (Q8_0) | llama.cpp, yes/no readout | RTX 4090 | 68 ms (one suite) |
+| Ternary Bonsai 2 27B (PQ2_0) | llama.cpp (PrismML fork), raw prompt | RTX 4090 | 97–113 ms, avg 102 |
+| DeepSeek-V4-Flash | vLLM, chat | 2× DGX Spark, over LAN | 359–672 ms, avg 483 |
+
+Image questions cost the llama.cpp models nothing extra, since the frame is
+paid once in the warm; DeepSeek pays about half again on images. The two
+in-process models are the fastest and the least accurate.
+
+### Accuracy by suite
+
 **policy-29**, the diagnostic suite carried over from the author's earlier experiments (choice only):
 
 | Recipe | Kind | Accuracy | Median latency | Scorecard |
@@ -221,6 +270,8 @@ jevify is written to be operated through a coding agent. Point yours at
 | [`tools/`](https://github.com/fidecastro/jevify/blob/main/tools) | Suite authoring, the Doom recorder and player |
 | [`AGENTS.md`](https://github.com/fidecastro/jevify/blob/main/AGENTS.md) | The engineering principles that bind every change |
 | [`SKILL.md`](https://github.com/fidecastro/jevify/blob/main/SKILL.md) | How an agent operates jevify |
+| [`docs/playbook.md`](https://github.com/fidecastro/jevify/blob/main/docs/playbook.md) | How a person operates jevify: ask, read an answer, serve, add a model, fix common problems |
+| [`Makefile`](https://github.com/fidecastro/jevify/blob/main/Makefile) | `make ask` and `make serve` shortcuts; `make help` lists the variables and recipes |
 
 To work on jevify: `uv sync --group dev` (add `--extra encoder --extra
 games` as needed) and `uv run pytest`. Live and in-process tests skip unless
@@ -229,14 +280,21 @@ you opt in.
 ## Not yet
 
 Named so that nobody has to discover it: a direct-call control, so no
-"faster than generation" claim exists yet; a Doom player that survives (both
-models die within seconds of `deadly_corridor`); the rerank kind and the
+"faster than generation" claim exists yet; clearing Doom's `deadly_corridor`
+(in `defend_the_center` Bonsai's perception plus a fixed rule scores 10 kills
+in 16 seconds, but in the corridor even the rule fed the engine's own labels
+dies before the armour, so the corridor needs a policy with memory, not
+better perception); the rerank kind and the
 embedding route against a live server; a marker-slot encoder layout; the
 grammar rung on the PrismML llama.cpp fork, which accepts a grammar without
-constraining the reported probabilities; Windows. Choice menus on the
-endpoint kind are bounded by the single-token identifier alphabet the probe
-verifies (26 on the DeepSeek recipe); Jev's 255-option ceiling is not
-reachable there.
+constraining the reported probabilities; the `prompt_logprobs` rung against a
+live server (written for TensorFold's native server with GLM-5.3-Flash, run
+only against the fake server in `tests/` so far; a server that skips its
+prefix cache for such requests, as TensorFold's source shows it does, makes
+every question read the whole state again, which the probe measures and
+notes); Windows. Choice menus on the endpoint kind are bounded by the
+single-token identifier alphabet the probe verifies (26 on the DeepSeek
+recipe); Jev's 255-option ceiling is not reachable there.
 
 ## Licence
 

{jevify-0.1.0 → jevify-0.2.0}/README.md

@@ -9,7 +9,8 @@ Jev does, and measure how well it does.**
 [![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/)
 
 You bring a model behind an OpenAI-compatible endpoint (vLLM, llama.cpp,
-anything with logprobs) or a small model that runs in-process. jevify sends it
+TensorFold, anything that returns logprobs for a reply token or for the
+prompt) or a small model that runs in-process. jevify sends it
 a **state** once, then asks **typed questions** about it: a choice among
 options, a score on ordered levels, a yes/no with a probability. Each answer
 is read straight off the model's next-token distribution, no generation, no
@@ -51,10 +52,34 @@ jevify ask recipes/gemma-4-e4b-it.llamacpp.yaml \
 jevify serve recipes/gemma-4-e4b-it.llamacpp.yaml --port 8600
 ```
 
+Any model behind an OpenAI-compatible server works, not only the ones under
+`recipes/`: copy the nearest recipe, set `endpoint.base_url` and
+`model.name`, and run the probe, which re-verifies the answer token ids for
+that model and writes what it found into the file. vLLM and llama.cpp get
+their dialect extras; any other server is driven through the plain OpenAI
+surface. A server that refuses logprobs on the reply token but returns
+vLLM's `prompt_logprobs`, such as TensorFold's native server with
+GLM-5.3-Flash, is read through the `prompt_logprobs` rung, which needs a
+raw-mode recipe (the template written out with the model's own control
+tokens, served on `/v1/completions`).
+
+`serve` binds to `127.0.0.1`. To reach it from other machines pass
+`--host 0.0.0.0`, and set `JEVIFY_API_KEY` first so that every request must
+carry that bearer key; to leave it running, detach it and keep its log:
+
+```sh
+jevify serve recipes/<name>.yaml --host 0.0.0.0 --port 8600 > ~/jevify.log 2>&1 &
+curl -s http://<this-host>:8600/health
+```
+
 Every answer carries Jev's fields (`choice`, `confidence`, `probabilities`,
 `score`, `legend`, `noul`) plus an `x_jevify` block naming the readout method,
 the cached tokens, the latency and the recipe hash that produced it.
 
+From a clone, `make ask` and `make serve` wrap the two commands;
+[`docs/playbook.md`](https://github.com/fidecastro/jevify/blob/main/docs/playbook.md)
+walks through them and explains every answer field.
+
 ## How it works
 
 1. **Recipe.** A hashed YAML file: the endpoint, the prompt template (chat
@@ -69,7 +94,9 @@ the cached tokens, the latency and the recipe hash that produced it.
    that reuses the server's prefix cache. A 32k-token state costs one warm and
    then a few hundred milliseconds per question.
 4. **Readout ladder.** Named-token logprobs, grammar with post-sampling
-   probabilities, top-k, equal bias, and a floored top-k as the last resort.
+   probabilities, top-k, prompt logprobs (each answer token appended to the
+   prompt and its logprob read back, for servers that give none on the reply
+   token), equal bias, and a floored top-k as the last resort.
    Auto walks down the proven rungs; the answer names the rung that read it
    and whether it was degraded.
 5. **Evidence.** `jevify eval` runs a frozen suite with controls (shuffled
@@ -80,7 +107,7 @@ the cached tokens, the latency and the recipe hash that produced it.
 
 | Kind | Talks to | Proven live with |
 |---|---|---|
-| `endpoint` | Any OpenAI-compatible server; dialect extras for vLLM and llama.cpp | DeepSeek-V4-Flash on vLLM; Ternary Bonsai 2 27B, Gemma 4 E4B and Qwen3-VL-Reranker 8B/2B on llama.cpp; text and images |
+| `endpoint` | Any OpenAI-compatible server that returns reply-token or prompt logprobs; dialect extras for vLLM and llama.cpp | DeepSeek-V4-Flash on vLLM; Ternary Bonsai 2 27B, Gemma 4 E4B and Qwen3-VL-Reranker 8B/2B on llama.cpp; text and images |
 | `embedding` | A `/v1/embeddings` route, or a model in-process | Qwen3-Embedding-0.6B in-process |
 | `encoder` | An NLI sequence-classification head in-process | facebook/bart-large-mnli in-process |
 | `rerank` | A `/v1/rerank` route | Fakes only |
@@ -93,6 +120,28 @@ Four frozen suites, every recipe on each, one run each, latencies per
 question on one machine on one day. Every row links to the scorecard that
 records the command, the recipe hash, the suite hash and the raw file digest.
 
+### Latency per question
+
+The median of each suite, combined across the suites a model ran; one run
+each. A question is one request that reads the answer off the next-token
+probabilities, after the state was sent once and cached.
+
+| Model | How it runs | Hardware | Latency per question |
+|---|---|---|---|
+| facebook/bart-large-mnli | in-process NLI classifier | RTX 4090 | 7–9 ms, avg 8 |
+| Qwen3-Embedding-0.6B | in-process embedding, cosine | RTX 4090 | 8–10 ms, avg 9 |
+| Gemma 4 E4B (Q8_0) | llama.cpp, chat | RTX 4090 | 28–37 ms, avg 32 |
+| Qwen3-VL-Reranker-2B (Q8_0) | llama.cpp, yes/no readout | RTX 4090 | 47–64 ms, avg 53 |
+| Qwen3-VL-Reranker-8B (Q8_0) | llama.cpp, yes/no readout | RTX 4090 | 68 ms (one suite) |
+| Ternary Bonsai 2 27B (PQ2_0) | llama.cpp (PrismML fork), raw prompt | RTX 4090 | 97–113 ms, avg 102 |
+| DeepSeek-V4-Flash | vLLM, chat | 2× DGX Spark, over LAN | 359–672 ms, avg 483 |
+
+Image questions cost the llama.cpp models nothing extra, since the frame is
+paid once in the warm; DeepSeek pays about half again on images. The two
+in-process models are the fastest and the least accurate.
+
+### Accuracy by suite
+
 **policy-29**, the diagnostic suite carried over from the author's earlier experiments (choice only):
 
 | Recipe | Kind | Accuracy | Median latency | Scorecard |
@@ -191,6 +240,8 @@ jevify is written to be operated through a coding agent. Point yours at
 | [`tools/`](https://github.com/fidecastro/jevify/blob/main/tools) | Suite authoring, the Doom recorder and player |
 | [`AGENTS.md`](https://github.com/fidecastro/jevify/blob/main/AGENTS.md) | The engineering principles that bind every change |
 | [`SKILL.md`](https://github.com/fidecastro/jevify/blob/main/SKILL.md) | How an agent operates jevify |
+| [`docs/playbook.md`](https://github.com/fidecastro/jevify/blob/main/docs/playbook.md) | How a person operates jevify: ask, read an answer, serve, add a model, fix common problems |
+| [`Makefile`](https://github.com/fidecastro/jevify/blob/main/Makefile) | `make ask` and `make serve` shortcuts; `make help` lists the variables and recipes |
 
 To work on jevify: `uv sync --group dev` (add `--extra encoder --extra
 games` as needed) and `uv run pytest`. Live and in-process tests skip unless
@@ -199,14 +250,21 @@ you opt in.
 ## Not yet
 
 Named so that nobody has to discover it: a direct-call control, so no
-"faster than generation" claim exists yet; a Doom player that survives (both
-models die within seconds of `deadly_corridor`); the rerank kind and the
+"faster than generation" claim exists yet; clearing Doom's `deadly_corridor`
+(in `defend_the_center` Bonsai's perception plus a fixed rule scores 10 kills
+in 16 seconds, but in the corridor even the rule fed the engine's own labels
+dies before the armour, so the corridor needs a policy with memory, not
+better perception); the rerank kind and the
 embedding route against a live server; a marker-slot encoder layout; the
 grammar rung on the PrismML llama.cpp fork, which accepts a grammar without
-constraining the reported probabilities; Windows. Choice menus on the
-endpoint kind are bounded by the single-token identifier alphabet the probe
-verifies (26 on the DeepSeek recipe); Jev's 255-option ceiling is not
-reachable there.
+constraining the reported probabilities; the `prompt_logprobs` rung against a
+live server (written for TensorFold's native server with GLM-5.3-Flash, run
+only against the fake server in `tests/` so far; a server that skips its
+prefix cache for such requests, as TensorFold's source shows it does, makes
+every question read the whole state again, which the probe measures and
+notes); Windows. Choice menus on the endpoint kind are bounded by the
+single-token identifier alphabet the probe verifies (26 on the DeepSeek
+recipe); Jev's 255-option ceiling is not reachable there.
 
 ## Licence
 

{jevify-0.1.0 → jevify-0.2.0}/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: jevify
-description: Operate jevify, the bring-your-own-model harness that makes a pretrained model answer Jev-shaped typed questions (choice, score, noul) over a shared state and measures how well it does. Use when asked to connect a model behind an OpenAI-compatible endpoint (vLLM, llama.cpp, others) or an in-process encoder or embedding model, write or probe a recipe, ask questions, run a suite, read a scorecard, calibrate, or serve Jev's API.
+description: Operate jevify, the bring-your-own-model harness that makes a pretrained model answer Jev-shaped typed questions (choice, score, noul) over a shared state and measures how well it does. Use when asked to connect a model behind an OpenAI-compatible endpoint (vLLM, llama.cpp, TensorFold, others) or an in-process encoder or embedding model, write or probe a recipe, ask questions, run a suite, read a scorecard, calibrate, or serve Jev's API.
 ---
 
 # Operating jevify
@@ -31,7 +31,9 @@ result.
    rows carry `images`, regenerate the frames with
    `uv run python tools/record_doom_suite.py` after `uv sync --extra games`),
    `long-state-24` (2k to 32k tokens). `tools/play_doom.py <recipe> --scenario
-   defend_the_center --policy perception` records a video of a recipe playing.
+   defend_the_center --policy perception` records a video of a recipe playing;
+   `--policy expert` replays the same rule from the engine's labels, the
+   ceiling any perception can reach with that rule.
    Raw decisions land in `runs/` (git-ignored); the summary in `docs/evidence/`
    (committed). Read the Markdown summary, then the `wrong` list in the JSON.
 5. **Serve**: `jevify serve recipes/<name>.yaml --port 8600`. Jev's SDK works
@@ -51,15 +53,35 @@ Every recipe has `schema_version: 1`, `model: {name, kind, revision}` and
 the rest.
 
 **`endpoint`** (any OpenAI-compatible server): `endpoint: {base_url,
-api_key_env, dialect: auto|vllm|llamacpp|generic, timeout_s}`, `template`
-and `answers`.
+api_key_env, dialect: auto|vllm|llamacpp|generic, timeout_s, extra_body,
+headers, headers_env}`, `template` and `answers`.
 - `template.mode: messages` sends chat messages; put thinking off in
   `template.kwargs` (`{thinking: false}` for DeepSeek, `{enable_thinking:
   false}` for Qwen3-class models; names are model-specific, the probe checks
   that no `<think>` token opens the answer).
+- `endpoint.extra_body` is merged into the top level of every request, for
+  servers that take a switch there rather than as a template argument. The
+  `generic` dialect sends no template arguments, so on Ollama's `/v1` route
+  a thinking model needs `extra_body: {reasoning_effort: none}` (`think:
+  false` is ignored there). A key jevify itself sets (`max_tokens`,
+  `logprobs`, `logit_bias`, ...) is refused when the backend is built, before
+  any request, so probe, ask, eval and serve all fail the same way. The
+  fields also reach the tokenize and template routes the probe uses.
+- `endpoint.headers` adds static HTTP headers to every request, for a router
+  or gateway that requires one (`{x-client-id: agenticApp}`). The values are
+  written in the recipe and hashed with it, so never put a secret there: a
+  bearer key goes in the env var that `api_key_env` names, and any other
+  secret header in `endpoint.headers_env` (header name to env var name, read
+  when the backend is built; an unset variable fails then). A static header
+  whose name looks like a credential (`key`, `token`, `secret`, `auth`,
+  `cookie`, ...) is refused when the recipe loads; `Content-Type` and
+  `Authorization` are refused when the backend is built. Every
+  endpoint-backed kind sends them (endpoint, rerank, embedding).
 - `template.mode: raw` sends a completion prompt you write with the model's
   own control tokens; use it when the chat template would not put the model
-  in its trained regime (the Qwen3-VL reranker recipe is the example).
+  in its trained regime (the Qwen3-VL reranker recipe is the example), and
+  for a server that refuses reply-token logprobs (TensorFold's native server
+  with GLM-5.3-Flash): only a raw prompt can carry the `prompt_logprobs` rung.
 - `template.questions.choice_strategy`: `identifiers` (one request, options
   labelled A, B, C) or `per_option` (one yes/no request per option; needed
   for a model that never emits letters, such as a reranker).
@@ -67,8 +89,11 @@ and `answers`.
   the probe fills their ids.
 - `readout.rung: auto` walks the rungs the probe proved, strongest first:
   `named_token_logprobs` (vLLM with `logprob_token_ids`), `grammar`
-  (llama.cpp), `top_k`, `equal_bias`, `top_k_floor` (degraded, missing
-  labels floored and listed). A rung that cannot see every label hands the
+  (llama.cpp), `top_k`, `prompt_logprobs` (each answer spelling appended to
+  a raw prompt and read back from vLLM-style `prompt_logprobs` rows, one
+  request more for every spelling the rows did not show; exact, off-menu mass
+  visible), `equal_bias`, `top_k_floor` (degraded, missing labels floored and
+  listed). A rung that cannot see every label hands the
   question to the next; the answer names the rung that read it. A pinned rung
   never walks: it fails loudly instead.
 
@@ -88,7 +113,10 @@ sequence-classification head; all three question types; semantics `readout`.
 ## Reading an answer
 
 Jev's fields come first (`choice`, `confidence`, `probabilities`; `score`,
-`legend`; `noul`). Everything jevify adds is under `x_jevify`:
+`legend`; `noul`). `confidence` is Jev's statistic, `(n × peak − 1) / (n − 1)`
+over the options or levels: 0 when uniform, 1 when certain, and the number
+TypeSafe's published thresholds refer to. Everything jevify adds is under
+`x_jevify`:
 
 | Field | Meaning |
 |---|---|
@@ -138,6 +166,16 @@ Jev's fields come first (`choice`, `confidence`, `probabilities`; `score`,
   checkpoint at the end of an earlier prompt, and a chat template closes the
   user turn right after the state. Use `template.mode: raw` so the warm prompt
   ends at the state boundary; the Bonsai recipe is the example.
+- **Probe notes "reply-token logprobs refused"**: the server gives no
+  logprobs on a generated token (TensorFold with GLM-5.3-Flash). Write a raw
+  recipe so the probe can prove `prompt_logprobs`; with a chat recipe the
+  probe ends in "no readout is possible".
+- **"did not come back as its own prompt token"**: the appended answer fuses
+  with the end of the raw template; end the template where the answer starts
+  a token of its own (after a newline, or before a leading-space spelling).
+- **Probe notes "prompt_logprobs requests reuse no cached prefix"**: the
+  server reads the whole state again for every question on that rung; keep
+  states short there, or serve the model where reply-token logprobs exist.
 - **Missing encoder extra**: the error names the install command.
 
 ## House rules that bind you here

{jevify-0.1.0 → jevify-0.2.0}/docs/00-invariants.md

@@ -126,7 +126,7 @@ API response may claim more than it permits.
 
 Rules that follow:
 
-- **Confidence** is a statistic of the returned distribution: one minus its normalized entropy for choice and score, distance from one half for noul. It is not an independent estimate that the decision is right, and copy may not describe it as one.
+- **Confidence** is Jev's statistic of the returned distribution: `(n × peak − 1) / (n − 1)` over the n options or levels for choice and score, so a uniform distribution scores 0 and a certain one scores 1 (TypeSafe's published definition). Jev's noul answers carry no confidence; jevify reports the distance of P(true) from one half, scaled to [0, 1], under `x_jevify` only. It is not an independent estimate that the decision is right, and copy may not describe it as one.
 - **"Faster"** may be claimed only against a measured direct call on the same backend, same model, same state, with the measurement in a scorecard.
 - **"Accuracy"** names its suite and the suite's hash. A suite whose expected answers came from a teacher model measures agreement with that teacher, and copy says so.
 - **A fine-tuned result is never compared with a zero-shot result in one table** without the words fine-tuned and zero-shot in the row.
@@ -150,6 +150,7 @@ number and the previous wording. The edit and the ADR land in the same commit.
 | Date | Invariant | ADR | Change |
 |---|---|---|---|
 | 2026-09-19 | all | ADR-0001 | Adopted. |
+| 2026-09-23 | §5, confidence rule | ADR-0003 D8 | Confidence is Jev's peak statistic, `(n × peak − 1) / (n − 1)`. Previous wording: "a statistic of the returned distribution: one minus its normalized entropy for choice and score, distance from one half for noul". The entropy statistic disagreed with Jev's published definition, so the SDK-facing `confidence` field did not carry Jev's meaning. |
 
 ## 8. Document map
 

{jevify-0.1.0 → jevify-0.2.0}/docs/adr/0002-backend-port-and-adapter-kinds.md

@@ -66,7 +66,7 @@ on an adapter module; a structure guard enforces the import ban.
 
 | Kind | Talks to | Readout | Semantics |
 |---|---|---|---|
-| `endpoint` | Any OpenAI-compatible HTTP server | One generated token; distribution from logprobs at the answer position | `readout` |
+| `endpoint` | Any OpenAI-compatible HTTP server | One generated token, or an answer token appended to the prompt; distribution from logprobs at the answer position | `readout` |
 | `encoder` | An in-process encoder classifier loaded through Transformers | The head's scores at option marker slots (Laya's layout) or label logits (GLiClass, NLI) | `readout` |
 | `rerank` | A rerank or score endpoint, or a reranker model in-process | One independent relevance score per option | `relevance` |
 | `embedding` | An embeddings endpoint or model | Cosine between instruction-conditioned state vector and cached option vectors | `similarity` |
@@ -91,7 +91,7 @@ requests and records:
 - whether each answer token under the recipe's template is a single token
   (via the tokenizer or the server's tokenize endpoint);
 - whether all answer tokens come back in the logprobs unbiased, biased, or
-  only through named-token logprobs or a grammar;
+  only through named-token logprobs, a grammar or prompt logprobs;
 - whether assistant prefill and template arguments (thinking off) are honored;
 - whether a second request over the same state is measurably faster than the
   first, which is the only proof that the prefix is cached;
@@ -115,8 +115,11 @@ allows, in this order:
    (exact over the answer set, off-menu mass hidden);
 3. an unbiased top-k request when every answer token is present (exact, and
    off-menu mass visible);
-4. equal bias on all answer tokens (exact ratios, off-menu mass hidden);
-5. top-k with a floor for missing tokens, marked degraded in the answer.
+4. prompt logprobs at an answer token appended to a raw prompt, one request
+   more for each answer spelling the rows did not show (exact, off-menu mass
+   visible; added 2026-11-04, below);
+5. equal bias on all answer tokens (exact ratios, off-menu mass hidden);
+6. top-k with a floor for missing tokens, marked degraded in the answer.
 
 The method used is recorded on every answer. A recipe may pin a rung; it may
 not skip the ladder's recording.
@@ -131,6 +134,30 @@ also proves two things it used to assume: that an image adds prompt tokens
 question request reuses (a recurrent model resumes only from a checkpoint at
 the end of an earlier prompt, so a chat-template warm can buy nothing).
 
+Amended 2026-11-04, for a server that refuses reply-token logprobs:
+TensorFold's native server (pin `f350cd3e`) serves logprobs on a generated
+token only for a family that gives the first token's row, and GLM-5.3-Flash
+does not, so every rung above that reads a generated token is refused with a
+400. The same server, like vLLM, returns `prompt_logprobs` on
+`/v1/completions`: null, then one `{id: {logprob, rank, decoded_token}}` row
+for each prompt token, the token the prompt holds listed first. The
+`prompt_logprobs` rung appends an answer spelling to the raw prompt and reads
+the last row: the appended token's exact logprob and the top-k beside it, all
+from the one distribution at the answer slot. A spelling the rows did not
+show is appended in a request of its own, so every label is exact and the
+off-menu mass visible. The rung refuses a row whose prompt token is not the
+appended one: an answer that fuses with the template's tail would be read at
+the wrong position. It reads raw prompts only (TensorFold serves
+`prompt_logprobs` on the completions route alone). It sits below `top_k`,
+which needs one request, and above `equal_bias`, which hides the off-menu
+mass. The probe no longer assumes reply-token logprobs: a 400 on its first
+readout is recorded, the rungs that need them stay unproven, its own usage
+and timing requests stop asking for them, and the probe proves
+`prompt_logprobs` by appending an answer and checking the row. It also sends
+that request twice and records when the repeat reuses no cached prefix,
+since TensorFold skips its prefix cache for any request with
+`prompt_logprobs` and each question then reads the whole state again.
+
 ### D5 — Choice questions on decoders
 
 Options are labeled with single-token identifiers and the distribution is

{jevify-0.1.0 → jevify-0.2.0}/docs/adr/0003-jev-compatible-api.md

@@ -1,6 +1,7 @@
 # ADR-0003 — Jev's evaluate call is the API; extensions are namespaced; the SDK is the acceptance test
 
-- **Status:** accepted (2026-09-19).
+- **Status:** accepted (2026-09-19); D8 added 2026-09-23: `confidence` is
+  Jev's peak statistic, amending `docs/00-invariants.md` §5.
 - **Extends:** ADR-0001 D2. Implements INV-8; constrained by INV-4, INV-9.
 
 ## Context
@@ -77,6 +78,31 @@ implementation, and a structure guard forbids it from touching the port.
 The library is the product; the server is one caller of it, and a program
 that imports jevify gets identical behaviour without HTTP.
 
+### D8 — `confidence` is Jev's statistic, computed Jev's way
+
+Jev's `confidence` on a `choice` or `score` answer is
+`(n × peak − 1) / (n − 1)`, where `n` is the number of options or levels and
+`peak` is the largest probability: 0 for a uniform distribution, 1 for a
+certain one. This is TypeSafe's published definition
+(<https://docs.typesafe.ai/confidence>, worked example
+`[0.90, 0.06, 0.04] → 0.85`), and it is what the thresholds TypeSafe
+recommends to clients (act above 0.9, review between 0.5 and 0.9, route to
+a human below 0.5) are calibrated against. Jev's `noul` answers carry no
+confidence.
+
+jevify computes exactly this in the one statistics chokepoint
+(`jevify/domain/distribution.py`) and nowhere else. Until 2026-09-23 it
+served one minus the normalized entropy under Jev's field name, which is
+systematically lower (`[0.90, 0.06, 0.04]` gave 0.64) and would have moved
+a client's threshold decisions; a reader of the NVIDIA forum thread caught
+it. For `noul`, jevify keeps its own statistic (distance of P(true) from one
+half, scaled to [0, 1]) under `x_jevify.confidence`, never on Jev's field.
+
+Scorecards under `docs/evidence/` whose `run_at` precedes 2026-09-23 record
+the entropy statistic in their wrong-case column; the reliability table bins
+on the top probability and is unaffected. They are not rewritten, because a
+number in a document comes from a run.
+
 ## Consequences
 
 - Compatibility is tested, not asserted. A client written for Jev runs
@@ -84,8 +110,9 @@ that imports jevify gets identical behaviour without HTTP.
 - The `x_jevify` namespace grows without ever touching Jev's fields.
 - The explicit ingest introduces server-side state and its expiry; the
   stateless route remains the default and the ingest is opt-in.
-- Jev's `confidence` is served with Jev's meaning, a statistic of the
-  distribution, and the claims contract forbids describing it otherwise.
+- Jev's `confidence` is served with Jev's meaning and Jev's formula (D8), a
+  statistic of the distribution, and the claims contract forbids describing
+  it otherwise.
 
 ## Rejected alternatives
 

{jevify-0.1.0 → jevify-0.2.0}/docs/adr/0004-recipes.md

@@ -32,6 +32,7 @@ hash is the recipe's identity in every answer and every scorecard.
 | Section | Holds |
 |---|---|
 | `model` | Identifier as the backend knows it, revision when known, the adapter kind, and for endpoints the dialect the probe detected. |
+| `endpoint` | For endpoint-backed kinds: base URL, the environment variable holding an API key, the dialect, the timeout, and `extra_body`, server-specific fields merged into the top level of every request (a field jevify itself sets is refused when the backend is built, from one registry in the endpoint client; omitted from the hash when empty, so older recipes keep theirs), `headers`, static HTTP headers sent on every request (`Content-Type` and `Authorization` are refused when the backend is built; never a secret, since the recipe is hashed and shared, so a name that looks like a credential is refused when the recipe loads; omitted from the hash when empty), and `headers_env`, header names mapped to environment variables whose values are read when the backend is built and never written to the recipe (an unset variable fails then; omitted from the hash when empty). |
 | `template` | For endpoints: how state, instructions and options are rendered into messages or a raw prompt; where the question goes; the answer-slot suffix; template arguments such as thinking off; whether assistant prefill is used. For encoders: the sequence layout and marker scheme. |
 | `answers` | The answer tokens for `noul`, the identifier scheme for `choice`, the level tokens for `score`, each with the token ids the probe verified. |
 | `readout` | The rung of the ladder to use or `auto`; the permutation policy and call count for `choice`. |

{jevify-0.1.0 → jevify-0.2.0}/docs/adr/README.md

@@ -14,7 +14,7 @@ later documents can cite "ADR-0002 D3".
 |---|---|---|
 | [0001](0001-founding-choices.md) | Founding choices: name, scope, licence, toolchain, conventions | Accepted (2026-09-19); D3 amended 2026-09-20: GPL-3.0 to MIT |
 | [0002](0002-backend-port-and-adapter-kinds.md) | One backend port, four adapter kinds, a capability probe, a readout ladder | Accepted (2026-09-19) |
-| [0003](0003-jev-compatible-api.md) | Jev's evaluate call is the API; extensions are namespaced; the SDK is the acceptance test | Accepted (2026-09-19) |
+| [0003](0003-jev-compatible-api.md) | Jev's evaluate call is the API; extensions are namespaced; the SDK is the acceptance test | Accepted (2026-09-19); D8 added 2026-09-23: `confidence` is Jev's peak statistic, amends invariants §5 |
 | [0004](0004-recipes.md) | Recipes: the hashed file that is the only way a model is asked | Accepted (2026-09-19) |
 | [0005](0005-evaluation-evidence-and-calibration.md) | Suites, scorecards, committed evidence, post-hoc calibration | Accepted (2026-09-19) |
 | [0006](0006-toolchain-dependency-budget-and-portability.md) | Toolchain, dependency budget and portability: five light core packages, extras, argparse, uv/pipx, Python 3.12–3.14, no lock-in | Accepted (2026-09-19); makes ADR-0001 D4 precise |