logit-classifier 0.1.0__py3-none-any.whl

logit_classifier-0.1.0.dist-info/METADATA

@@ -0,0 +1,354 @@
+Metadata-Version: 2.5
+Name: logit-classifier
+Version: 0.1.0
+Summary: Local zero-shot classifier for text and images. Declares options, returns a calibrated probability for each, generates no text. Accepts the TypeSafe System One request format.
+Project-URL: Repository, https://github.com/Blakeem/logit-classifier
+Project-URL: Bug Tracker, https://github.com/Blakeem/logit-classifier/issues
+Author: Blake
+License-Expression: GPL-3.0-or-later
+License-File: LICENSE
+Keywords: calibration,classification,comfyui,local-llm,logits,vision-language-model,zero-shot
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: numpy>=2.1.0
+Provides-Extra: hf
+Requires-Dist: accelerate>=1.0.0; extra == 'hf'
+Requires-Dist: huggingface-hub[hf-xet]>=0.34.0; extra == 'hf'
+Requires-Dist: pillow>=12.3.0; extra == 'hf'
+Requires-Dist: safetensors>=0.4.5; extra == 'hf'
+Requires-Dist: torch>=2.11.0; extra == 'hf'
+Requires-Dist: torchvision>=0.26.0; extra == 'hf'
+Requires-Dist: transformers>=4.57.0; extra == 'hf'
+Provides-Extra: service
+Requires-Dist: fastapi>=0.115.0; extra == 'service'
+Requires-Dist: pillow>=12.3.0; extra == 'service'
+Requires-Dist: uvicorn[standard]>=0.32.0; extra == 'service'
+Description-Content-Type: text/markdown
+
+# Logit Classifier
+
+A local zero-shot classifier. One state and a set of questions go in, and a calibrated
+probability per option comes out. No text is generated, since every answer is read from
+the logits at a single token position.
+
+The request format is TypeSafe's System One, so a Jev request body works unchanged. The
+default model reads images as well as text.
+
+## Question Types
+
+| Type | You declare | You get back |
+|---|---|---|
+| `choice` | a map of option name to description | the winning name, a probability per option, a confidence, and an abstain |
+| `score` | an ordered list of level descriptions | a weighted level, a probability per level, the legend, and a confidence |
+| `noul` | one statement, with optional true and false descriptions | one probability that the statement is true |
+
+A `choice` holds up to 255 options. A `score` holds 2 to 10 levels. One request holds up
+to 256 questions, and every question is answered against the same state in one pass.
+
+A branch is one prompt whose last token position carries a distribution. A question needs
+one branch, or several when it holds more than 52 options or scores each level alone.
+
+`abstain` is how strongly the model wants none of your options. Every choice question is
+offered an extra label named `none of these`, and that label's mass comes back as
+`abstain` instead of in `probabilities`. So `probabilities` covers your own options and
+sums to 1.
+
+`confidence` measures how peaked the answer is, using TypeSafe's published formulas.
+
+Ask one `noul` per fact when several answers can be true at once. A `choice` normalizes
+over its options, so one winner takes nearly all the mass.
+
+## Install
+
+The package needs Python 3.12 or newer.
+
+```
+pip install "logit-classifier[hf,service]"
+```
+
+| Extra | Adds | For |
+|---|---|---|
+| none | numpy | passing a model you already loaded |
+| `hf` | torch, torchvision, transformers, accelerate, safetensors, huggingface-hub, pillow | loading the weights in this process |
+| `service` | fastapi, uvicorn, pillow | the HTTP endpoint and the browser page |
+
+The `hf` extra needs an NVIDIA GPU with at least 10 GB of free VRAM. The weights are
+about 8 GB per model and download on first use.
+
+They go to the Hugging Face cache by default. Point `models_dir` at a folder to keep them
+beside your project instead, which is what the examples do.
+
+```python
+Config(models_dir=Path("models"))
+```
+
+`LOGIT_MODELS_DIR` sets the same folder for every process, and `HF_HOME` moves the
+Hugging Face cache itself.
+
+## Models
+
+| | Qwen/Qwen3-VL-4B-Instruct | Qwen/Qwen3-4B-Instruct-2507 |
+|---|---|---|
+| reads images | yes | no |
+| better at | choice and score | noul |
+
+`Qwen3-VL-4B-Instruct` is the default. Set `LOGIT_MODEL_ID` to use the other one. Any Qwen
+chat model loads, and a model with no fitted temperature gets 2.5.
+
+## Python API
+
+```python
+from logit_classifier import Classifier, Config, load_model, parse_request
+
+config = Config()
+backend = load_model("Qwen/Qwen3-VL-4B-Instruct", config)
+classifier = Classifier(config, backend=backend)
+
+request = parse_request({
+    "state": "the payment failed again",
+    "questions": {
+        "urgent": {"type": "noul", "instructions": "The message is urgent"},
+    },
+})
+response, diagnostics = classifier.classify(request)
+print(response.to_dict())
+```
+
+`load_model` loads the weights through the `hf` extra and returns a `Backend`.
+`Classifier(Config())` calls it for you when you pass no backend.
+
+Loading each model yourself is what lets one script compare several. The fitted
+temperature follows the backend, so every model is scored with its own value.
+
+```python
+for model_id in ("Qwen/Qwen3-VL-4B-Instruct", "Qwen/Qwen3-4B-Instruct-2507"):
+    classifier = Classifier(config, backend=load_model(model_id, config))
+```
+
+## HTTP API
+
+```
+logit-classifier serve
+```
+
+Run it as `python -m logit_classifier serve` when the scripts directory is not on PATH.
+
+`POST /v1/systemone` takes one `state` and a map of questions.
+
+```json
+{
+  "state": "I have been trying to connect my Stripe account for 3 days and it keeps failing.",
+  "questions": {
+    "department": {
+      "type": "choice",
+      "instructions": "Which team should handle this",
+      "criteria": {
+        "billing": "Payment or subscription issues",
+        "technical": "Bugs or integration problems",
+        "sales": null
+      }
+    },
+    "frustration": {
+      "type": "score",
+      "instructions": "How frustrated the customer appears",
+      "criteria": ["Calm", "Frustrated but civil", "Very angry"]
+    },
+    "is_urgent": { "type": "noul", "instructions": "The message conveys urgency" }
+  }
+}
+```
+
+Each answer comes back under the key you gave its question. The keys are never sent to
+the model.
+
+```json
+{
+  "model": "logit-classifier-0.1.0",
+  "answers": {
+    "department": {
+      "type": "choice",
+      "choice": "technical",
+      "confidence": 0.9518,
+      "probabilities": { "billing": 0.0237, "technical": 0.9679, "sales": 0.0084 },
+      "abstain": 0.0068
+    },
+    "frustration": {
+      "type": "score",
+      "score": 1.0199,
+      "confidence": 0.9583,
+      "legend": { "0": "Calm", "1": "Frustrated but civil", "2": "Very angry" },
+      "probabilities": { "0": 0.004, "1": 0.9722, "2": 0.0238 }
+    },
+    "is_urgent": { "type": "noul", "noul": 0.9924 }
+  },
+  "usage": { "input_tokens": 210, "output_tokens": 3 }
+}
+```
+
+Add `?diagnostics=1` to also return the label mass, the branch count, the latency and the
+calibration settings.
+
+`GET /v1/models` lists the served model. `GET /healthz` reports whether the weights are
+loaded. Open `http://127.0.0.1:8077/` for a browser form that builds a request and draws
+each probability as a bar.
+
+## Images
+
+A `state` that is a mapping may carry one image under `image` or `screenshot`, as a file
+path, a data URL or bare base64. Everything else in that mapping is rendered as text
+beside the picture. The HTTP service accepts a data URL or base64, and a file path works
+only in process.
+
+```json
+{ "state": { "image": "tile-3.png", "caption": "tile 3 of 6" },
+  "questions": { "sky": { "type": "noul", "instructions": "This crop shows the night sky" } } }
+```
+
+The image is encoded once for the entire request, so asking several questions about one
+picture costs little more than asking one.
+
+A model with no vision tower rejects the request rather than ignoring the picture.
+
+## Examples
+
+Each script runs on its own.
+
+| Script | Shows |
+|---|---|
+| `examples/quickstart.py` | one request with all three question types |
+| `examples/question_types.py` | every field each answer type returns |
+| `examples/images.py` | a picture, and a picture with text beside it |
+| `examples/compare_models.py` | two models answering the same questions in one command |
+| `examples/http_client.py` | calling the service over HTTP |
+| `examples/own_backend.py` | implementing the backend port, with no weights or GPU |
+
+## Configuration
+
+Every setting reads from the environment at startup. `logit-classifier config` prints what
+they produce.
+
+| Variable | Default | Effect |
+|---|---|---|
+| `LOGIT_MODEL_ID` | `Qwen/Qwen3-VL-4B-Instruct` | any Qwen chat model, with or without vision |
+| `LOGIT_MODELS_DIR` | the Hugging Face cache | where downloaded weights are kept |
+| `LOGIT_DEVICE` | `cuda` | torch device |
+| `LOGIT_TEMPERATURE` | the model's fitted value | divides the logits before the softmax |
+| `LOGIT_PRIOR_DEBIAS` | `1` | set to `0` to skip the label prior |
+| `LOGIT_BATCH_BRANCHES` | `1` | set to `0` for one forward pass per branch |
+| `LOGIT_SCORE_METHOD` | `joint` | set to `independent` to judge each level alone |
+| `LOGIT_ABSTAIN` | `1` | set to `0` to drop the `none of these` label below 52 options |
+| `LOGIT_CALIBRATION_PATH` | `calibration.json` | where the service stores the running prior |
+
+## Calibration
+
+Two corrections are applied before the softmax.
+
+The label prior is the model's standing preference for the token `A` over `B`. It is
+learned as a running mean over served requests, so it needs no labelled data or extra
+forward passes. It stays unused until a bucket has 32 observations, so the first requests
+after a cold start can answer a little differently from later ones.
+
+The temperature is fitted per model and divides the logits. Both shipped values were
+fitted on Banking77 rather than on your data, so refit yours when the absolute numbers
+matter.
+
+The library holds the prior in memory and writes no file. The service stores it in
+`calibration.json`. Pass `Config(calibration_path=...)` to keep it across runs from the
+library. A stored prior carries a fingerprint of the model and the prompt shape, and
+changing either one starts the prior over.
+
+## Determinism
+
+Nothing is sampled, so the same request returns bitwise identical logits.
+
+Batch composition and padding length both change the low bits of a bfloat16 forward pass,
+and both are pure functions of the request. So the same question asked inside two
+different requests can differ slightly. Set `LOGIT_BATCH_BRANCHES=0` to score each branch
+alone, which makes a question independent of the questions sent with it and costs one
+forward pass per branch.
+
+A forward pass needs several process-global torch settings held at known values. Torch
+exposes none of them as a call argument, so the backend sets them around each pass and
+puts the host's values back after. This way a host that shares the interpreter keeps its
+own settings everywhere outside a pass.
+
+## Options Above 52
+
+Options are labeled `A` to `Z` then `a` to `z`, which gives 52 single token labels. A
+question with more options is split into groups, and each group carries an escape label so
+the groups can be weighed against each other. Accuracy is lower on that path and `abstain`
+is much weaker, so keep a question under 52 options where you can.
+
+## Speed
+
+On a warm RTX 3090 Ti a short record takes about 140 ms whatever it asks, up to about 8
+questions, because two forward passes dominate. So send every question for a record in one
+request. Keep a record under about 8,000 tokens, since a longer state falls off the
+prefill ceiling.
+
+## ComfyUI Node Packs
+
+A node pack installs the bare package without extras, so pip is never asked to resolve
+torch beside ComfyUI's own CUDA build. The pack then passes its own backend over a model
+ComfyUI already loaded, and this library never imports torch or transformers.
+
+A node pack uses its own backend in place of `load_model`. Everything after that is the
+same call.
+
+```python
+from logit_classifier import Backend, Classifier, Config, verify_backend
+
+assert isinstance(my_backend, Backend)
+label_ids = verify_backend(my_backend)
+classifier = Classifier(Config(), backend=my_backend)
+```
+
+`Backend` declares `model_id`, `label_ids` and `sees_images`, plus `render`, `encode`,
+`encode_prefix` and `score`. `model_id` is the string the fitted temperature is looked up
+by, so a backend over the text model must report the text model. `Backend` is runtime
+checkable, so a host can check its own object before passing it. `verify_backend` proves
+the render and label contracts and returns one token id per label.
+
+`COMFY_SOCKET_TYPE` holds the ComfyUI socket name for a loaded classifier.
+`examples/own_backend.py` implements the entire port.
+`logit_classifier.backends.hf.HFBackend` is the transformers implementation to read
+against.
+
+## Differences From Jev
+
+This project matches the System One request and response format and the documented limits.
+It is not affiliated with TypeSafe. Its numbers do not match Jev's.
+
+Jev is trained for calibrated probabilities. This project reads them from a general model,
+so the choices agree more often than the confidences do.
+
+Jev judges a score level without its number or its neighbours. This project judges all
+levels together by default. Set `LOGIT_SCORE_METHOD=independent` for the documented
+behavior.
+
+Jev publishes status codes but no error body. The error shape here is our own.
+
+## Development
+
+```
+uv sync --all-extras
+uv run pytest tests/test_unit.py
+uv run pytest tests/test_model.py -m model
+uv run ruff check .
+uv run mypy
+```
+
+The unit tests need no weights. The model tests load the GPU and take about 15 seconds.
+
+`FINDINGS.md` is the measurement record. Every number there names the script in
+`tests-AB/` that produced it.
+
+## License
+
+GPL-3.0. See `LICENSE`.

logit_classifier-0.1.0.dist-info/RECORD

@@ -0,0 +1,24 @@
+logit_classifier/__init__.py,sha256=XwjWECGU5V8j6HdV_Eqd9XDeAFjXsHr2Fd4kax4NPeA,2124
+logit_classifier/__main__.py,sha256=GHrjXTqm3z2G291fBuqXVvSQeUWgPqRgBY7He7f-mx4,240
+logit_classifier/calibrate.py,sha256=shz-qAKNMDn5APZEpKiOsV4M9Dizk8UBW4i3_OrGNXU,5865
+logit_classifier/classifier.py,sha256=LKs8mCOd6cyLz1AkBPgQpKxg89ogKuBEJAP3nKHHGug,14439
+logit_classifier/cli.py,sha256=t0z_BPtuA1Nyy50w7Abb9HDdrJvTgovHxI6MElRXTo0,1795
+logit_classifier/config.py,sha256=k18QD7egAG0JsgAi6EnlodbuZRmW1mdOMKQici1LYrc,7749
+logit_classifier/deps.py,sha256=nofR5WjToaHXoWgaDPkq2Txm9B5FhwWvmCVeO3poauE,996
+logit_classifier/errors.py,sha256=s0InrI-vx8MvjOxrJ-XriMT0h2EETqZR_Gpss_aejbY,573
+logit_classifier/labels.py,sha256=av2tlxgY6glCuBdhSJX3gwEdoF1kPVEFws6-knvomXU,3418
+logit_classifier/prompt.py,sha256=W0AE0rjTX721kuljnBIPHUPZl1jKVQJe757utL5KN-g,6379
+logit_classifier/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+logit_classifier/schema.py,sha256=jTVRyO7B-pTkydUYccCHvVoG8K6YjoHDpj6YUp2bDCY,9258
+logit_classifier/scoring.py,sha256=MRCpUZbQBPwSXdOCFZxao8mC_NpZTrUqZc1ZNh0oqO8,4075
+logit_classifier/service.py,sha256=b34FBAVUCzXWJhIV6RL9VxhFEG7aqCaZWdpFOwp7K80,5668
+logit_classifier/vision.py,sha256=Z64ACRm9G1EWigSkQQpaAx7CA6kUOjHwJoKh7Gqv80Q,3549
+logit_classifier/backends/__init__.py,sha256=wzfkDHPWgK3x1-ilAFih1ryJAqZ3M7VknGO_lEBJ9B0,224
+logit_classifier/backends/base.py,sha256=dedOkmALap3QM1-Lr2asUnoib5kr0ZdNM9oVDZhHKls,5273
+logit_classifier/backends/hf.py,sha256=OV3blHX2OZB-Fa1RsefwO57tqPfmv2hT6n89RTNq6js,19127
+logit_classifier/web/index.html,sha256=wx8XQaEdaiL1RDcsGO5JQhYsJre5gsHOyjH2wLGAez4,17705
+logit_classifier-0.1.0.dist-info/METADATA,sha256=GzAVNQ-eHHBAiwYUQD66MoQSxv7NGDZL_2AaRTDX0oQ,13877
+logit_classifier-0.1.0.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
+logit_classifier-0.1.0.dist-info/entry_points.txt,sha256=5umLAfkRklfpJgFgNhcNzz3xu0aHhtrJSpuxStIbwYI,63
+logit_classifier-0.1.0.dist-info/licenses/LICENSE,sha256=OXLcl0T2SZ8Pmy2_dmlvKuetivmyPd5m1q-Gyd-zaYY,35149
+logit_classifier-0.1.0.dist-info/RECORD,,

logit_classifier-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.4
+Root-Is-Purelib: true
+Tag: py3-none-any

logit_classifier-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+logit-classifier = logit_classifier.cli:main