logit-classifier 0.4.0__tar.gz → 0.5.0__tar.gz

{logit_classifier-0.4.0 → logit_classifier-0.5.0}/FINDINGS.md

@@ -56,9 +56,10 @@ options, but that is the split path defect below, not a gain. `ab_temperature.py
 ## The prior corrects letters, not fixed positions
 
 The running prior subtracts the model's standing preference for one letter over another.
-Two positions carry a fixed meaning instead. The escape label always means none of these,
-and a score's letters always name the same rungs. Their mean mass follows the traffic, so
-correcting them moves one request's answer with unrelated requests.
+Some positions carry a fixed meaning instead. The escape label always means none of these,
+a score's letters always name the same rungs, and a noul's `A` is always yes. Their mean mass
+follows the traffic, so correcting them moves one request's answer with unrelated requests.
+The prior therefore covers the lettered options of choice and member branches only.
 
 Banking77, 462 held out rows, Qwen3-VL-4B-Instruct at its fitted temperature.
 
@@ -72,6 +73,41 @@ The fix does not regress calibration. The accuracy gain is four rows in 462, nea
 of this sample. The prior still costs calibration against no prior, because the shipped
 temperature was fitted with the prior off. `banking77.py`, with and without `--no-prior`.
 
+A noul under the prior shows the same coupling. Banking77 statements ("This message is about
+<intent>", true for the row's own intent) ran as request streams at four true rates, the first
+32 requests of each stream unscored.
+
+| Stream | Prior | AUC | Recall at 0.5 | False positives at 0.5 | Recall at 0.9999 |
+|---|---|---|---|---|---|
+| 50% true | off | 0.9538 | 0.891 | 0.065 | 0.050 |
+| 50% true | on | 0.9531 | 0.891 | 0.065 | 0.059 |
+| 80% true | off | 0.9419 | 0.872 | 0.091 | 0.055 |
+| 80% true | on | 0.9410 | 0.848 | 0.068 | 0.006 |
+| 95% true | off | 0.9428 | 0.878 | 0.083 | 0.056 |
+| 95% true | on | 0.9409 | 0.821 | 0.000 | 0.005 |
+| 20% true | off | 0.9439 | 0.878 | 0.078 | 0.000 |
+| 20% true | on | 0.9435 | 0.878 | 0.102 | 0.122 |
+
+The AUC does not move, since the correction is one shift of the yes/no gap. The operating point
+moves with the traffic instead. A request took 126.6 ms with the prior off and 126.4 ms with it
+on, so the prior has no speed cost or gain. `ab_prior_traffic.py`.
+
+ContextAnchoredTileRefine's tags bench kept 8.2 correct tags per tile with the prior off and
+5.4 with it on, over 45 judged tiles on 15 pictures. The bench ran on 00518c6, before 3ac0294
+took the prior off nouls, so its prior also corrected the verify and locate nouls. The loss grew
+over the later tiles, which the earlier tiles' mostly true verify answers had trained.
+
+| Tile in run order | Correct tags, prior off | Correct tags, prior on |
+|---|---|---|
+| first | 114 | 111 |
+| second | 110 | 80 |
+| third | 146 | 52 |
+
+The first tile kept the same count on 13 of 15 pictures. It lost 2 tags on dragon4k and 1 on
+market4k, both under the long prompt. `ab_tags_bench.py` in
+ContextAnchoredTileRefine's `tests-AB/`, arms `prior-off` and `prior-on`, added in its commit
+bf6c7ba.
+
 ## Option order helps above 52 options and nowhere else
 
 | Options | Letterings | Accuracy | ECE |
@@ -160,7 +196,7 @@ the CUDA attention path, only one of them changes a logit on either shipped mode
 |---|---|---|
 | `cudnn.benchmark` | 0.0 | 0.0 |
 | `cudnn.deterministic` | 0.0 | 0.0 |
-| `allow_bf16_reduced_precision_reduction` | **0.5** | **0.75** |
+| `allow_bf16_reduced_precision_reduction` | **0.375** | **1.125** |
 | float32 matmul precision | 0.0 | 0.0 |
 
 Max absolute logit shift against the fully pinned run, window removed so the process
@@ -175,9 +211,10 @@ logits, and a run under a hostile host state matches a run under a pinned one, b
 gap of 0.0 on both models.
 
 `allow_fp16_bf16_reduction_math_sdp` is held for a different reason and the sweep above
-could not reach it. `sdpa_kernel` turns the math backend **off** while it holds the CUDA
-backends, so this setting is live only when `_usable_attention_backends` returns nothing and
-no `sdpa_kernel` block is entered. ComfyUI turns it **on** at import, at
+could not reach it. The math backend is **off** while `sdpa_kernel` holds the backends the
+probe found. When `_usable_attention_backends` returns nothing, the `FALLBACK_ATTENTION` block
+puts math last. So this setting is live only on a pass that flash, cuDNN and memory efficient
+attention all reject. ComfyUI turns it **on** at import, at
 `comfy/model_management.py:569`, so inside ComfyUI that path would run with reduced
 precision reductions.
 
@@ -214,7 +251,7 @@ Every row in a chunk is left-padded to that chunk's longest suffix, and each pad
 token costs a full forward plus attention over the whole prefix. Taking branches in the
 order the caller sent them drags short branches to the longest width.
 
-| Request shape | ms | peak GB | suffix tokens | chunks | |
+| Request shape | ms | peak GiB | suffix tokens | chunks | |
 |---|---|---|---|---|---|
 | mixed, four choices plus 12 nouls plus 4 scores | 2055.6 | 13.66 | 14600 | 1 | request order |
 | | **623.9** | **10.66** | **2614** | 3 | packed |
@@ -226,7 +263,7 @@ order the caller sent them drags short branches to the longest width.
 | 30 three option choices | 396 | 8.99 | 2160 | 1 | both, bitwise identical |
 | 12 twenty option choices | 586 | 9.24 | 3960 | 1 | both, bitwise identical |
 
-3.29x and 5.06x on the two shapes that mix widths, each giving back about 3 GB. Nothing
+3.29x and 5.06x on the two shapes that mix widths, each giving back about 3 GiB. Nothing
 at all on the five that do not, which is the point. The sort is stable, so equal lengths
 keep the caller's order and the grouping stays a pure function of the request.
 
@@ -241,12 +278,12 @@ width, which is every choice question, and moved its logits for no gain.
 
 | ceiling | mixed shape, 800 token prefix | mixed widths, long prefix |
 |---|---|---|
-| 512 | 610.9 ms, 9.91 GB, 5 chunks | 2015.7 ms, 5 chunks |
-| 1024 | 608.3 ms, 10.47 GB, 3 chunks | 2014.9 ms, 5 chunks |
-| **2048** | **626.0 ms, 10.66 GB, 3 chunks** | **2015.8 ms, 5 chunks** |
-| 4096 | 793.5 ms, 11.03 GB, 2 chunks | 2286.5 ms, 4 chunks |
-| 8192 | 1119.4 ms, 11.84 GB, 2 chunks | 2289.5 ms, 4 chunks |
-| none | 2062.2 ms, 13.66 GB, 1 chunk | 2299.1 ms, 4 chunks |
+| 512 | 610.9 ms, 9.91 GiB, 5 chunks | 2015.7 ms, 5 chunks |
+| 1024 | 608.3 ms, 10.47 GiB, 3 chunks | 2014.9 ms, 5 chunks |
+| **2048** | **626.0 ms, 10.66 GiB, 3 chunks** | **2015.8 ms, 5 chunks** |
+| 4096 | 793.5 ms, 11.03 GiB, 2 chunks | 2286.5 ms, 4 chunks |
+| 8192 | 1119.4 ms, 11.84 GiB, 2 chunks | 2289.5 ms, 4 chunks |
+| none | 2062.2 ms, 13.66 GiB, 1 chunk | 2299.1 ms, 4 chunks |
 
 Anything from 512 to 2048 measured the same on both, and 4096 upward is clearly worse.
 The value is not a knife edge. 2048 sits in the middle of the flat band and leaves the
@@ -256,44 +293,91 @@ most room before two near-equal wide branches are split apart.
 
 ## Throughput and where it falls off
 
-Warm model, RTX 3090 Ti. Adding questions to a record is close to free up to about 8,
-because the fixed cost of two forward passes dominates a short record.
+Warm model, RTX 3090 Ti. Each row repeats one request, so its sequence lengths are already
+seen. Adding questions to a record is close to free up to about 8, because the fixed cost of
+two forward passes dominates a short record.
 
 | Questions per record | Total ms | ms per tag | Records per hour |
 |---|---|---|---|
-| 1 | 138 | 138 | 26,100 |
-| 4 | 136 | 34 | 26,500 |
-| 8 | 140 | 17.5 | 25,800 |
-| 32 | 278 | 8.7 | 13,000 |
+| 1 | 135 | 135 | 26,700 |
+| 4 | 132 | 33 | 27,300 |
+| 8 | 135 | 16.9 | 26,600 |
+| 32 | 295 | 9.2 | 12,200 |
 
 The state is prefilled once per record, so a longer record costs more.
 
-| State tokens | Total ms, 8 questions | ms per tag | Records per hour |
-|---|---|---|---|
-| 172 | 140 | 17.5 | 25,700 |
-| 556 | 185 | 23 | 19,500 |
-| 2,092 | 451 | 56 | 8,000 |
-| 8,236 | 1,575 | 197 | 2,300 |
-| 32,812 | 41,515 | 5,189 | 87 |
-
-Prefill runs at 46 to 48 TFLOP/s from about 2,000 tokens up, near this card's practical
-BF16 ceiling. Below that the time is almost all kernel launch rather than arithmetic, so a
-428 token request spends 0.14 ms on the math. The last row falls off the ceiling for the
-reason under Known weaknesses. `benchmark.py`.
-
-## The attention backend has to be pinned on Windows
-
-Torch 2.11 ships no FlashAttention kernel for Windows. Left to choose for itself the
-dispatcher reaches the math backend, which builds the full attention matrix. Measured on an
-8,192 token prefill.
-
-| Backend selection | Time | Peak memory |
+| State tokens | Total ms, 8 questions | ms per tag | Records per hour | Peak GiB |
+|---|---|---|---|---|
+| 172 | 144 | 18.0 | 24,900 | 8.36 |
+| 556 | 199 | 24.9 | 18,100 | 8.48 |
+| 2,092 | 494 | 62 | 7,300 | 8.92 |
+| 8,236 | 1,805 | 226 | 2,000 | 10.23 |
+| 32,812 | 11,453 | 1,432 | 314 | 15.23 |
+
+The TFLOP/s count covers the weights' FLOPs only. Prefill runs at about 42 TFLOP/s from 2,348
+to 8,492 prefilled tokens. A 428 token prefill runs at 26.3 TFLOP/s and a 33,068 token prefill
+at 25.6. Attention work grows with the square of the state length, so the count leaves out the
+most work on the 33,068 token row. The peak is `torch.cuda.max_memory_allocated` in GiB, weights
+included. `benchmark.py`.
+
+## The memory efficient kernel, not cuDNN
+
+Torch 2.11 ships no FlashAttention kernel for Windows, so `HFBackend` probes the backends at
+startup, at the model's own dtype, and pins one. cuDNN builds an execution plan the first time it sees a sequence length, and served requests
+rarely repeat a length. The memory efficient kernel has no such cost, but it rejects grouped
+key and value heads. transformers passes grouped heads whenever a pass has no mask, which is
+the prefix pass. So the model loads under its own attention name, which expands the heads
+before the kernel sees them.
+
+Both arms ran on one loaded model.
+
+| | cuDNN, grouped heads | memory efficient, expanded heads |
 |---|---|---|
-| left to the dispatcher | 147,221 ms | 27.6 GB |
-| pinned to cudnn and efficient | 1,291 ms | 9.2 GB |
-
-`HFBackend` probes the backends at startup, at the model's own dtype, and pins the working
-ones. `benchmark.py`.
+| 40 requests of unseen length, 96 to 2,865 tokens, p50 | 1,501 ms | 335 ms |
+| the same 40 requests repeated, p50 | 335 ms | 327 ms |
+| image request of unseen length, 132 image tokens | 2,257 ms | 214 ms |
+| 8,192 token state and 8 questions, unseen | 3,664 ms | 1,927 ms |
+| 8,192 token state and 8 questions, repeated | 1,672 ms | 1,742 ms |
+| peak memory at 8,192 tokens | 15.76 GiB | 15.76 GiB |
+| Banking77, 462 held out rows, accuracy | 0.6255 | 0.6212 |
+| Banking77, 462 held out rows, ECE at T=1.25 | 0.0882 | 0.0834 |
+
+The two kernels round differently in bfloat16. Logits moved by a median of 0.25 and at most
+1.5, and the top letter changed on 36 of 924 rows. Accuracy moved by 2 of 462 held out rows,
+which is inside the noise of that sample, so the fitted temperature stands. The same request
+still returns bitwise equal logits on 924 of 924 rows. `ab_attention_kernel.py`.
+
+## One frozen prefix cache, not a copy per chunk
+
+Each chunk used to deep copy the prefix cache and broadcast it across its rows, which stored
+rows times the prefix in every layer for the whole forward. The prefix cache is now frozen
+after the prefix pass, and each layer joins the broadcast prefix to the chunk's own keys and
+values only for that layer's attention.
+
+Both arms ran on one idle GPU, the commit before the change against the commit after.
+
+| Request | peak before | peak after | logits |
+|---|---|---|---|
+| mixed, four choices plus nouls plus scores | 10.67 GiB | 8.75 GiB | bitwise identical |
+| skewed, one wide choice beside twenty nouls | 10.95 GiB | 8.76 GiB | bitwise identical |
+| uniform, 256 nouls at the Jev cap | 8.74 GiB | 8.39 GiB | bitwise identical |
+| split, one 77 option choice | 8.61 GiB | 8.40 GiB | bitwise identical |
+| tile fragments, 24 nouls | 8.63 GiB | 8.36 GiB | bitwise identical |
+| uniform, 30 three option choices | 8.99 GiB | 8.47 GiB | bitwise identical |
+| uniform, 12 twenty option choices | 9.24 GiB | 8.60 GiB | bitwise identical |
+| 8,192 token state and 8 questions | 15.76 GiB | 10.24 GiB | bitwise identical |
+| image state, top-L.png | 8.37 GiB | 8.34 GiB | bitwise identical |
+
+The peaks in the packing and kernel sections above were measured with the copy.
+`ab_frozen_prefix_cache.py`.
+
+A request of 33,068 prefilled tokens took 155,903 ms at 1.9 TFLOP/s on the memory efficient
+kernel and 52,813 ms on cuDNN while each chunk copied the prefix cache. The frozen cache brings
+it to 11,453 ms. `benchmark.py`.
+
+That request's prefix cache holds about 4.5 GiB. The figure is derived from the shipped config
+rather than measured, as 147,456 bytes per token (36 layers × 8 key value heads × 128 head_dim
+× 2 tensors × 2 bytes) times the 32,812 prefix tokens.
 
 ## One presence wording, two threshold pairs
 
@@ -326,12 +410,14 @@ wording.
 |---|---|---|
 | score used the choice confidence formula | wrong confidence on every score answer | ported both official formulas |
 | attention probe ran in bfloat16 whatever the model dtype | any other dtype crashed on the first forward | probe at the model's dtype |
-| attention probe used one call shape | approved a backend that then failed | probe causal and masked, with grouped query attention |
+| attention probe used one call shape | approved a backend that then failed | probe the causal and masked forms the forward sends |
+| cuDNN attention built an execution plan for every new sequence length | about a second on almost every served request | memory efficient kernel, with grouped heads expanded before it |
 | abstain combined groups by product | shrinks with group count, never fires at scale | weakest decline |
 | image source detected by string length | an 8x8 PNG encodes shorter than the threshold and was read as a path | check whether the file exists |
 | M-RoPE offset dropped on branch rows | Qwen3-VL raises rather than misplacing tokens | read `rope_deltas` after the prefix pass |
 | prior learned and corrected the escape label | traffic that often declines taught the prior to undo a decline | prior over the lettered options only |
-| a joint score shared a prior bucket with a choice of the same width | a score's top rung learned the choice traffic's escape mass | score has its own bucket |
+| a joint score shared a prior bucket with a choice of the same width | a score's top rung learned the choice traffic's escape mass | no prior on a score |
+| the yes/no and score layouts learned the traffic's answer rate | mostly true traffic cut recall at 0.9999 from 0.055 to 0.006 | prior over choice and member branches only |
 
 ## Known weaknesses
 
@@ -345,13 +431,6 @@ now saturate near 0 and 1, so the best threshold sits at 0.9999 rather than near
 
 **Noul regressed 8 points** moving to the vision model, and image tagging leans on noul.
 
-**Very long states fall off the prefill ceiling.** At 33,068 prefilled tokens the card runs
-at 7.1 TFLOP/s against 47.9 at 8,492. The prefix key value cache is about 4.9 GB at that
-length and every chunk copies it, which leaves a 24 GB card no room. Expanding the cache
-once and cropping between chunks was measured and is worse, at 12,732 ms against 1,710 ms
-and 23.66 GB against 15.12 GB, because a cropped cache is a non-contiguous view. Unsolved.
-`benchmark.py`.
-
 ## Layout
 
 `tests/` holds the pytest suite, the unit tests and 14 tests that need the GPU. `tests-AB/` holds

{logit_classifier-0.4.0 → logit_classifier-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: logit-classifier
-Version: 0.4.0
+Version: 0.5.0
 Summary: Local zero-shot classifier for text and images. Declares options and reads a calibrated probability for each from the logits. Accepts the TypeSafe System One request format, and ships a toolkit of image tag tools and ComfyUI node tools.
 Project-URL: Repository, https://github.com/Blakeem/logit-classifier
 Project-URL: Bug Tracker, https://github.com/Blakeem/logit-classifier/issues
@@ -52,7 +52,8 @@ A `choice` holds up to 255 options. A `score` holds 2 to 10 levels. One request
 to 256 questions, and every question is answered against the same state in one pass.
 
 A branch is one prompt whose last token position carries a distribution. A question needs
-one branch, or several when it holds more than 52 options or scores each level alone.
+one branch, or several when it holds more than 51 options (52 with `abstain` off) or scores
+each level alone.
 
 `abstain` is how strongly the model wants none of your options. Every choice question is
 offered an extra label named `none of these`, and that label's mass comes back as
@@ -68,6 +69,15 @@ over its options, so one winner takes nearly all the mass.
 
 The package needs Python 3.12 or newer.
 
+The `hf` extra needs a CUDA build of torch. The torch wheels on PyPI for Windows have no CUDA,
+so on Windows install torch from the PyTorch index first.
+
+```
+pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
+```
+
+Then install the package.
+
 ```
 pip install "logit-classifier[hf,service]"
 ```
@@ -78,8 +88,10 @@ pip install "logit-classifier[hf,service]"
 | `hf` | torch, torchvision, transformers, accelerate, safetensors, huggingface-hub, pillow | loading the weights in this process |
 | `service` | fastapi, uvicorn, pillow | the HTTP endpoint and the browser page |
 
-The `hf` extra needs an NVIDIA GPU with at least 10 GB of free VRAM. The weights are
-about 8 GB per model and download on first use.
+The `hf` extra needs an NVIDIA GPU. A short request peaks under 9 GiB of VRAM, and a state
+near 8,000 tokens peaks at about 10.2 GiB. Every VRAM figure in this README is an allocated
+peak, and the process needs more for the CUDA context and the allocator's cache. The weights
+take 8 to 9 GB of disk per model and download on first use.
 
 They go to the Hugging Face cache by default. Point `models_dir` at a folder to keep them
 beside your project instead, which is what the examples do.
@@ -178,7 +190,7 @@ the model.
 
 ```json
 {
-  "model": "logit-classifier-0.4.0",
+  "model": "logit-classifier-0.5.0",
   "answers": {
     "department": {
       "type": "choice",
@@ -254,7 +266,7 @@ reads none of them. `logit-classifier config` prints what they produce.
 | `LOGIT_SCORE_METHOD` | `joint` | set to `independent` to judge each level alone |
 | `LOGIT_PERMUTATIONS` | `1` | letterings averaged per question, each one adds branches |
 | `LOGIT_ABSTAIN` | `1` | set to `0` to drop the `none of these` label below 52 options |
-| `LOGIT_CALIBRATION_PATH` | `calibration.json` | where the service stores the running prior |
+| `LOGIT_CALIBRATION_PATH` | `calibration.json` | where `Config.from_env()` stores the running prior |
 
 ## Calibration
 
@@ -262,16 +274,17 @@ Two corrections are applied before the softmax.
 
 The label prior is the model's standing preference for the token `A` over `B`. It is
 learned as a running mean over served requests, so it needs no labelled data or extra
-forward passes. It stays unused until a bucket has 32 observations, so the first requests
+forward passes. It corrects choice questions only, since the letters of a noul and a score
+always name the same answers. It stays unused until a bucket has 32 observations, so the first requests
 after a cold start can answer a little differently from later ones.
 
 The temperature is fitted per model and divides the logits. Both shipped values were
 fitted on Banking77 rather than on your data, so refit yours when the absolute numbers
 matter.
 
-The library holds the prior in memory and writes no file. The service stores it in
-`calibration.json`. Pass `Config(calibration_path=...)` to keep it across runs from the
-library. A stored prior carries a fingerprint of the model and the prompt shape, and
+`Config()` holds the prior in memory and writes no file. `Config.from_env()` stores it at
+`LOGIT_CALIBRATION_PATH` after every request, so the service keeps it across restarts. Pass
+`Config(calibration_path=...)` to pick the file yourself. A stored prior carries a fingerprint of the model and the prompt shape, and
 changing either one starts the prior over.
 
 ## Determinism
@@ -292,17 +305,18 @@ own settings everywhere outside a pass.
 
 ## Options Above 52
 
-Options are labeled `A` to `Z` then `a` to `z`, which gives 52 single token labels. A
-question with more options is split into groups, and each group carries an escape label so
+Options are labeled `A` to `Z` then `a` to `z`, which gives 52 single token labels. While
+`abstain` is on, the `none of these` label takes one of them. A question with more options
+than the free labels is split into groups, and each group carries an escape label so
 the groups can be weighed against each other. Accuracy is lower on that path and `abstain`
 is much weaker, so keep a question under 52 options where you can.
 
 ## Speed
 
-On a warm RTX 3090 Ti a short record takes about 140 ms whatever it asks, up to about 8
+On a warm RTX 3090 Ti a short record takes about 135 ms whatever it asks, up to about 8
 questions, because two forward passes dominate. So send every question for a record in one
-request. Keep a record under about 8,000 tokens, since a longer state falls off the
-prefill ceiling.
+request. A state near 8,000 tokens takes about 1.8 seconds. A state near 33,000 tokens takes
+about 11.5 seconds and peaks at about 15.2 GiB of VRAM.
 
 ## ComfyUI Node Packs
 

{logit_classifier-0.4.0 → logit_classifier-0.5.0}/README.md

@@ -19,7 +19,8 @@ A `choice` holds up to 255 options. A `score` holds 2 to 10 levels. One request
 to 256 questions, and every question is answered against the same state in one pass.
 
 A branch is one prompt whose last token position carries a distribution. A question needs
-one branch, or several when it holds more than 52 options or scores each level alone.
+one branch, or several when it holds more than 51 options (52 with `abstain` off) or scores
+each level alone.
 
 `abstain` is how strongly the model wants none of your options. Every choice question is
 offered an extra label named `none of these`, and that label's mass comes back as
@@ -35,6 +36,15 @@ over its options, so one winner takes nearly all the mass.
 
 The package needs Python 3.12 or newer.
 
+The `hf` extra needs a CUDA build of torch. The torch wheels on PyPI for Windows have no CUDA,
+so on Windows install torch from the PyTorch index first.
+
+```
+pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
+```
+
+Then install the package.
+
 ```
 pip install "logit-classifier[hf,service]"
 ```
@@ -45,8 +55,10 @@ pip install "logit-classifier[hf,service]"
 | `hf` | torch, torchvision, transformers, accelerate, safetensors, huggingface-hub, pillow | loading the weights in this process |
 | `service` | fastapi, uvicorn, pillow | the HTTP endpoint and the browser page |
 
-The `hf` extra needs an NVIDIA GPU with at least 10 GB of free VRAM. The weights are
-about 8 GB per model and download on first use.
+The `hf` extra needs an NVIDIA GPU. A short request peaks under 9 GiB of VRAM, and a state
+near 8,000 tokens peaks at about 10.2 GiB. Every VRAM figure in this README is an allocated
+peak, and the process needs more for the CUDA context and the allocator's cache. The weights
+take 8 to 9 GB of disk per model and download on first use.
 
 They go to the Hugging Face cache by default. Point `models_dir` at a folder to keep them
 beside your project instead, which is what the examples do.
@@ -145,7 +157,7 @@ the model.
 
 ```json
 {
-  "model": "logit-classifier-0.4.0",
+  "model": "logit-classifier-0.5.0",
   "answers": {
     "department": {
       "type": "choice",
@@ -221,7 +233,7 @@ reads none of them. `logit-classifier config` prints what they produce.
 | `LOGIT_SCORE_METHOD` | `joint` | set to `independent` to judge each level alone |
 | `LOGIT_PERMUTATIONS` | `1` | letterings averaged per question, each one adds branches |
 | `LOGIT_ABSTAIN` | `1` | set to `0` to drop the `none of these` label below 52 options |
-| `LOGIT_CALIBRATION_PATH` | `calibration.json` | where the service stores the running prior |
+| `LOGIT_CALIBRATION_PATH` | `calibration.json` | where `Config.from_env()` stores the running prior |
 
 ## Calibration
 
@@ -229,16 +241,17 @@ Two corrections are applied before the softmax.
 
 The label prior is the model's standing preference for the token `A` over `B`. It is
 learned as a running mean over served requests, so it needs no labelled data or extra
-forward passes. It stays unused until a bucket has 32 observations, so the first requests
+forward passes. It corrects choice questions only, since the letters of a noul and a score
+always name the same answers. It stays unused until a bucket has 32 observations, so the first requests
 after a cold start can answer a little differently from later ones.
 
 The temperature is fitted per model and divides the logits. Both shipped values were
 fitted on Banking77 rather than on your data, so refit yours when the absolute numbers
 matter.
 
-The library holds the prior in memory and writes no file. The service stores it in
-`calibration.json`. Pass `Config(calibration_path=...)` to keep it across runs from the
-library. A stored prior carries a fingerprint of the model and the prompt shape, and
+`Config()` holds the prior in memory and writes no file. `Config.from_env()` stores it at
+`LOGIT_CALIBRATION_PATH` after every request, so the service keeps it across restarts. Pass
+`Config(calibration_path=...)` to pick the file yourself. A stored prior carries a fingerprint of the model and the prompt shape, and
 changing either one starts the prior over.
 
 ## Determinism
@@ -259,17 +272,18 @@ own settings everywhere outside a pass.
 
 ## Options Above 52
 
-Options are labeled `A` to `Z` then `a` to `z`, which gives 52 single token labels. A
-question with more options is split into groups, and each group carries an escape label so
+Options are labeled `A` to `Z` then `a` to `z`, which gives 52 single token labels. While
+`abstain` is on, the `none of these` label takes one of them. A question with more options
+than the free labels is split into groups, and each group carries an escape label so
 the groups can be weighed against each other. Accuracy is lower on that path and `abstain`
 is much weaker, so keep a question under 52 options where you can.
 
 ## Speed
 
-On a warm RTX 3090 Ti a short record takes about 140 ms whatever it asks, up to about 8
+On a warm RTX 3090 Ti a short record takes about 135 ms whatever it asks, up to about 8
 questions, because two forward passes dominate. So send every question for a record in one
-request. Keep a record under about 8,000 tokens, since a longer state falls off the
-prefill ceiling.
+request. A state near 8,000 tokens takes about 1.8 seconds. A state near 33,000 tokens takes
+about 11.5 seconds and peaks at about 15.2 GiB of VRAM.
 
 ## ComfyUI Node Packs
 

{logit_classifier-0.4.0 → logit_classifier-0.5.0}/examples/quickstart.py

@@ -65,7 +65,7 @@ def report(answers: dict) -> None:
 
 def main() -> None:
     parser = argparse.ArgumentParser(description="Ask one state a few questions.")
-    parser.add_argument("--text", default=SAMPLE_TEXT, help="the text to classify")
+    parser.add_argument("--text", default=None, help="the text to classify, or a caption beside --image")
     parser.add_argument("--image", default=None, help="a picture to classify instead of the text")
     parser.add_argument("--model", default=None, help="a model id, else the configured default")
     args = parser.parse_args()
@@ -73,7 +73,11 @@ def main() -> None:
     config = Config(models_dir=MODELS_DIR)
     backend = load_model(args.model, config)
     classifier = Classifier(config, backend=backend)
-    state = {"image": args.image, "caption": args.text} if args.image else args.text
+    state: object = args.text if args.text is not None else SAMPLE_TEXT
+    if args.image:
+        state = {"image": args.image}
+        if args.text is not None:
+            state = {"image": args.image, "caption": args.text}
     questions = IMAGE_QUESTIONS if args.image else QUESTIONS
 
     print(f"model {backend.model_id}, reads images: {backend.sees_images}\n")

{logit_classifier-0.4.0 → logit_classifier-0.5.0}/pyproject.toml

@@ -143,11 +143,15 @@ ignore_missing_imports = true
 [[tool.mypy.overrides]]
 # transformers annotates from_pretrained and eval loosely, which strict mode reads
 # as untyped calls. torch resolves to Any where the hf extra is absent, which makes
-# @torch.inference_mode() an untyped decorator. Every other strict check still
-# applies to this module.
+# @torch.inference_mode() an untyped decorator. CI installs no hf extra, so DynamicCache
+# resolves to Any there, which makes the prefix cache a subclass of Any and its update
+# return Any. An inline ignore would go unused locally, where transformers is typed.
+# Every other strict check still applies to this module.
 module = ["logit_classifier.backends.hf"]
 disallow_untyped_calls = false
 disallow_untyped_decorators = false
+disallow_subclassing_any = false
+warn_return_any = false
 
 [[tool.mypy.overrides]]
 # torch leaves fp16_bf16_reduction_math_sdp_allowed unannotated, and a line ignore

{logit_classifier-0.4.0 → logit_classifier-0.5.0}/src/logit_classifier/_version.py

@@ -1,3 +1,3 @@
 """The package version, apart from __init__ so config.py can read it without a cycle."""
 
-__version__ = "0.4.0"
+__version__ = "0.5.0"

{logit_classifier-0.4.0 → logit_classifier-0.5.0}/src/logit_classifier/backends/_torch_window.py

@@ -51,8 +51,8 @@ def _determinism() -> Iterator[None]:
         saved_matmul = None if has_slots else torch.get_float32_matmul_precision()
         saved_cuda = torch.backends.cuda.matmul.fp32_precision if has_slots else None
         saved_mkldnn = mkldnn.fp32_precision if has_slots else None
-        # ComfyUI turns this on at import, at comfy/model_management.py:569. It governs
-        # the math attention backend, which runs only when attention_backends is empty.
+        # ComfyUI turns this on at import, at comfy/model_management.py:569. It governs the math
+        # attention backend, last in HFBackend's FALLBACK_ATTENTION and in core's own SDPA priority.
         has_sdp = hasattr(torch.backends.cuda, "allow_fp16_bf16_reduction_math_sdp") and hasattr(
             torch.backends.cuda, "fp16_bf16_reduction_math_sdp_allowed"
         )

{logit_classifier-0.4.0 → logit_classifier-0.5.0}/src/logit_classifier/backends/base.py

@@ -1,7 +1,8 @@
 """The readout port, the one boundary between the arithmetic and a host's forward pass.
 
 Everything above this port is stdlib and numpy. Everything below it is one host's
-way of turning token ids into a logit row: transformers here, a ComfyUI CLIP next.
+way of turning token ids into a logit row: transformers in hf.py, a ComfyUI CLIP in
+comfy_clip.py.
 """
 
 from __future__ import annotations