decision-gem 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- decision_gem-0.1.0/LICENSE +21 -0
- decision_gem-0.1.0/PKG-INFO +366 -0
- decision_gem-0.1.0/README.en.md +341 -0
- decision_gem-0.1.0/README.md +340 -0
- decision_gem-0.1.0/decision_gem/__init__.py +4 -0
- decision_gem-0.1.0/decision_gem/__main__.py +34 -0
- decision_gem-0.1.0/decision_gem/core.py +474 -0
- decision_gem-0.1.0/decision_gem.egg-info/PKG-INFO +366 -0
- decision_gem-0.1.0/decision_gem.egg-info/SOURCES.txt +15 -0
- decision_gem-0.1.0/decision_gem.egg-info/dependency_links.txt +1 -0
- decision_gem-0.1.0/decision_gem.egg-info/entry_points.txt +2 -0
- decision_gem-0.1.0/decision_gem.egg-info/requires.txt +6 -0
- decision_gem-0.1.0/decision_gem.egg-info/top_level.txt +1 -0
- decision_gem-0.1.0/pyproject.toml +41 -0
- decision_gem-0.1.0/setup.cfg +4 -0
- decision_gem-0.1.0/tests/test_core.py +383 -0
- decision_gem-0.1.0/tests/test_open_model.py +73 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 tumi
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,366 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: decision-gem
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Jev-style decision model (noul / choice / score) on a local Gemma, MLX; prompts too
|
|
5
|
+
Author: tumi
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/tumikosha/decision-gem
|
|
8
|
+
Project-URL: Changelog, https://github.com/tumikosha/decision-gem/blob/main/CHANGELOG.md
|
|
9
|
+
Keywords: llm,mlx,gemma,classification,decision-model,jev,apple-silicon
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Intended Audience :: Science/Research
|
|
13
|
+
Classifier: Operating System :: MacOS
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
16
|
+
Requires-Python: >=3.10
|
|
17
|
+
Description-Content-Type: text/markdown
|
|
18
|
+
License-File: LICENSE
|
|
19
|
+
Requires-Dist: mlx>=0.32
|
|
20
|
+
Requires-Dist: mlx-lm>=0.32
|
|
21
|
+
Requires-Dist: huggingface_hub>=0.30
|
|
22
|
+
Provides-Extra: test
|
|
23
|
+
Requires-Dist: pytest; extra == "test"
|
|
24
|
+
Dynamic: license-file
|
|
25
|
+
|
|
26
|
+
# decision-gem
|
|
27
|
+
|
|
28
|
+
A decision model in the style of [TypeSafe Jev](https://developers.cloudflare.com/ai/models/typesafe/jev/) on a local
|
|
29
|
+
Gemma (MLX). It answers three kinds of questions about a text (the "state"): yes / no, a choice among options, a score
|
|
30
|
+
on a scale. No text is generated: the answer is read from the probabilities of the options' letters. Runs offline.
|
|
31
|
+
The same model also answers ordinary prompts by generating (`generate`, see "Prompts").
|
|
32
|
+
|
|
33
|
+
Russian version: [README.md](https://github.com/tumikosha/decision-gem/blob/main/README.md).
|
|
34
|
+
|
|
35
|
+
## What it does
|
|
36
|
+
|
|
37
|
+
Request: `{"state": text or JSON object, "questions": {id: question}}`.
|
|
38
|
+
|
|
39
|
+
| Type | Question | Answer |
|
|
40
|
+
|---|---|---|
|
|
41
|
+
| `noul` | `instructions`, `criteria` `{"true": …, "false": …}` (optional) | `noul` = P(true) |
|
|
42
|
+
| `choice` | `instructions`, `criteria` `{option: description}` or `[option, …]`, 2..26 | `choice`, `confidence`, `probabilities` |
|
|
43
|
+
| `score` | `instructions`, `criteria` `{value: description}` or `[description, …]` (levels 1..N), 2..10 | `score` = Σ p·value, `confidence`, `legend`, `probabilities` |
|
|
44
|
+
|
|
45
|
+
Beyond Jev, every answer has `orders_agree`: whether the choice is the same in every option order (below).
|
|
46
|
+
`false` means an unsure answer, worth checking. The whole response: `{"model", "answers": {id: answer},
|
|
47
|
+
"usage": {"input_tokens", "output_tokens": 0}}`.
|
|
48
|
+
|
|
49
|
+
How it works:
|
|
50
|
+
1. The state is read once into a KV cache.
|
|
51
|
+
2. Each question is appended to a copy of the cache, its options under the letters A, B, C …
|
|
52
|
+
3. The logits of those letters at the first answer position are read.
|
|
53
|
+
4. Each question is read in several option orders. Each option's log-probabilities are averaged, so a
|
|
54
|
+
preference for a letter or a position cancels out.
|
|
55
|
+
|
|
56
|
+
### Option orders
|
|
57
|
+
|
|
58
|
+
`DecisionGem(orders=...)`, per request `decide(..., orders=...)`, on the CLI `--orders`:
|
|
59
|
+
|
|
60
|
+
| `orders` | Passes per question | What cancels |
|
|
61
|
+
|---|---|---|
|
|
62
|
+
| `"reverse"` (default) | 2: in order and reversed | with 2 options, everything; with 3+ the middle stays in the middle (of 3 options, B is B both times), so a pull to or from the middle letters stays |
|
|
63
|
+
| `"cyclic"` | n: every rotation | each option once under every letter: position cancels fully |
|
|
64
|
+
|
|
65
|
+
For noul (2 options) both are the same two passes.
|
|
66
|
+
|
|
67
|
+
Under `cyclic`, `orders_agree` means all n passes agree, stricter than two; its `false` rate is not comparable
|
|
68
|
+
across modes. Calibration `T` was fitted on `reverse`; fit it again for `cyclic`.
|
|
69
|
+
|
|
70
|
+
### Calibration
|
|
71
|
+
|
|
72
|
+
Raw LLM probabilities are overconfident: `noul` 1.0 does not mean "yes" 100% of the time. Calibration remaps them so
|
|
73
|
+
that 0.8 means "yes" in about 80% of such answers. Each question has its own bias, so the coefficients are kept per
|
|
74
|
+
question id: `[a, b]` for noul (`p = sigmoid(a·logit + b)`, `logit` being the log-probability of true minus that of
|
|
75
|
+
false), `T` for choice / score (`softmax(logprob / T)`, `T > 1` softens).
|
|
76
|
+
|
|
77
|
+
```json
|
|
78
|
+
{"spam": [0.35, -0.8], "kind": 2.1}
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Pass it as `DecisionGem(calibration=...)` or `--calibration cal.json`. To fit it:
|
|
82
|
+
|
|
83
|
+
1. Label 100–300 of your own texts by hand: the answer to every question.
|
|
84
|
+
2. Get answers with the raw log-probabilities: `decide(..., raw=True)`, `{"raw": true}` in a request or `--raw` in the
|
|
85
|
+
CLI. Every answer gets `logprobs` {option: log-probability before calibration}.
|
|
86
|
+
3. `fit_calibration(answers, labels)` returns the calibration.
|
|
87
|
+
|
|
88
|
+
```python
|
|
89
|
+
from decision_gem import DecisionGem, fit_calibration
|
|
90
|
+
|
|
91
|
+
with DecisionGem() as gem:
|
|
92
|
+
answers = [gem.decide(text, qs, raw=True)["answers"] for text in texts]
|
|
93
|
+
cal = fit_calibration(answers, labels) # labels: [{"spam": True, "kind": "phishing"}, ...] in the same order
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
A label is `True` / `False` for noul, the option for choice, the level for score; a question with no label in a text is
|
|
97
|
+
skipped. noul is fitted by Platt scaling (logistic regression on `logit`), `T` by the least log-loss. Both noul labels
|
|
98
|
+
and some model mistakes must be in the set: where the model is always right, `T` goes to 0.01 and `a` grows.
|
|
99
|
+
|
|
100
|
+
|
|
101
|
+
## Differences from Jev
|
|
102
|
+
|
|
103
|
+
The request and the response are Jev's (noul / choice / score). Everything under the contract differs. Facts about Jev
|
|
104
|
+
come from the Cloudflare and TypeSafe docs (`docs.typesafe.ai/models.md`), November 2026; they do not disclose how Jev
|
|
105
|
+
works inside.
|
|
106
|
+
|
|
107
|
+
| | Jev (TypeSafe) | decision-gem |
|
|
108
|
+
|---|---|---|
|
|
109
|
+
| Where it runs | a cloud API (Cloudflare Workers AI, TypeSafe `POST /v1/systemone`), $0.042 per 1M input tokens | locally on Apple silicon (MLX), offline, data stays on the machine |
|
|
110
|
+
| Model | its own, trained to return calibrated decisions (RLCD), architecture not disclosed | any instruct LLM in mlx-lm format, see "Other models" |
|
|
111
|
+
| How it computes | not disclosed | the logits of the option letters on a copy of the state's KV cache, averaged over option orders |
|
|
112
|
+
| Calibration | built in by training | fitted on your own labels: `fit_calibration`, see "Calibration" |
|
|
113
|
+
| Question types | noul, choice, score, Box (from Jev 1.14) | noul, choice, score |
|
|
114
|
+
| Images (`assets`) | yes (beta) | no: text or JSON |
|
|
115
|
+
| Limits | 64k tokens per request, 32k per question (262k with a header) | the model's context and memory; choice 2..26 options, score 2..10 levels |
|
|
116
|
+
|
|
117
|
+
Only in decision-gem:
|
|
118
|
+
|
|
119
|
+
- `orders_agree` in every answer and `orders="cyclic"` (see "Option orders");
|
|
120
|
+
- `raw=True`: the raw option `logprobs`, and `fit_calibration` on them;
|
|
121
|
+
- keyed states: `prefill`, `decide(key=...)`, `forget`, `save` / `load`;
|
|
122
|
+
- open states and `extend`: text is read into the cache as it grows;
|
|
123
|
+
- `generate`: the same model answers ordinary prompts;
|
|
124
|
+
- score `criteria` also as `{value: description}`, choice `criteria` also as a list.
|
|
125
|
+
|
|
126
|
+
## Keyed states
|
|
127
|
+
|
|
128
|
+
`decide(state, questions)` reads the state, answers and drops its cache. To ask one email many questions at different
|
|
129
|
+
times, or to switch between emails, read the state once and keep it under a key:
|
|
130
|
+
|
|
131
|
+
| Method | What it does |
|
|
132
|
+
|---|---|
|
|
133
|
+
| `prefill(key, state)` | reads the state into a KV cache and keeps it under `key`; returns the number of tokens |
|
|
134
|
+
| `decide(key=key, questions=qs)` | questions on a kept state; exactly one of `state` / `key`, else `ValueError` |
|
|
135
|
+
| `gem({"key": key, "questions": qs})` | the same as a Jev request |
|
|
136
|
+
| `forget(key)` | frees the state's memory |
|
|
137
|
+
| `save(key, path)` / `load(key, path)` | the state to a `.safetensors` file and back; `load` takes only a file saved with the same model |
|
|
138
|
+
|
|
139
|
+
Each question runs on its own copy of the cache, so the kept state does not change: questions are independent and the
|
|
140
|
+
model does not remember earlier questions (it is not a chat). An answer by key is byte-identical to the answer of
|
|
141
|
+
`decide(state, ...)` — checked on 3 calls in reverse order and after `save` + `load`. Copying the cache is almost
|
|
142
|
+
free: MLX arrays are immutable, the data is shared until a question is appended to the copy.
|
|
143
|
+
|
|
144
|
+
A state file is the email's text in another form: keep it only where the text itself may be kept.
|
|
145
|
+
|
|
146
|
+
### Open states
|
|
147
|
+
|
|
148
|
+
`prefill(key, state, open=True)` stops the cache right after the state text; the closing `</state>` is not read yet.
|
|
149
|
+
`extend(key, text)` reads more text into that cache for good (it returns the tokens added; `ValueError` on a closed
|
|
150
|
+
state), so a growing thread can be kept open and new emails read into it. `decide` on an open state puts `</state>` and the question after the
|
|
151
|
+
cache on every copy and never writes the kept cache. `save` / `load` keep the open flag. `split_open(prompt)` is the
|
|
152
|
+
pure helper that cuts a rendered prompt before the closing tag.
|
|
153
|
+
|
|
154
|
+
Seam rule: `extend` equals one read only when the new text starts at a token boundary. Put the leading space or newline
|
|
155
|
+
in the new text, never split a word, and never end a chunk in whitespace (`extend` raises `ValueError`): `"hello. "` +
|
|
156
|
+
`"Operator"` tokenizes differently from `"hello. Operator"`. `prefill` strips the state, so it never ends in whitespace.
|
|
157
|
+
|
|
158
|
+
Precision: the gem computes in float32 by default (`float32=True`), see "The bf16 bug" below. `extend` equals one
|
|
159
|
+
read only in float32.
|
|
160
|
+
|
|
161
|
+
## Prompts
|
|
162
|
+
|
|
163
|
+
`generate(prompt, state=None, *, key=None, max_tokens=512, temperature=0.0)` runs the model as an ordinary chat model:
|
|
164
|
+
the answer to the prompt is generated (`mlx_lm.stream_generate`) by the model already loaded.
|
|
165
|
+
|
|
166
|
+
| Call | What the model sees |
|
|
167
|
+
|---|---|
|
|
168
|
+
| `generate(prompt)` | the prompt only |
|
|
169
|
+
| `generate(prompt, state)` | the state, then the prompt where a question would go |
|
|
170
|
+
| `generate(prompt, key=key)` | the kept state (`prefill` / `load`), not read again; generated on a copy of its cache |
|
|
171
|
+
| `gem({"prompt": …, "state" or "key", "max_tokens", "temperature"})` | the same as a request; the CLI takes this JSON too |
|
|
172
|
+
|
|
173
|
+
Response: `{"model", "text", "finish_reason": "stop" | "length", "usage": {"input_tokens", "output_tokens"}}`.
|
|
174
|
+
`temperature` 0 is greedy: the same prompt gives the same text. Generating leaves a kept state as it was:
|
|
175
|
+
`decide(key=…)` before and after `generate(key=…)` gives byte-identical answers (the `-m models` test); the answer on a
|
|
176
|
+
key equals the answer on the same `state`.
|
|
177
|
+
|
|
178
|
+
Limits: the answer is not constrained to a JSON schema (mlx-lm has no structured output) — check JSON and ask again
|
|
179
|
+
on an error; do not run beside Ollama with the same Gemma (memory).
|
|
180
|
+
|
|
181
|
+
## The bf16 bug: noisy answers on long states (found 2026-10-19)
|
|
182
|
+
|
|
183
|
+
**Symptom.** The same text gave a different answer depending on how it was read: whole, in chunks with `extend`,
|
|
184
|
+
or with `</state>` in another place. Up to ~460 tokens the answers agreed, beyond that they drifted:
|
|
185
|
+
|
|
186
|
+
| State tokens | Whole | In chunks (`extend`) |
|
|
187
|
+
|---|---|---|
|
|
188
|
+
| ≤ 462 | same | same |
|
|
189
|
+
| 692 | 0.75 | 0.37 |
|
|
190
|
+
| ~1400 | 0.45 | 0.22 |
|
|
191
|
+
|
|
192
|
+
Open vs closed state on one long text: 0.45 vs 0.61. One 1590-token text read in different ways gave `noul` from
|
|
193
|
+
0.14 to 0.65: at a 0.5 threshold the yes / no answer depended on the way of reading, not on the text.
|
|
194
|
+
|
|
195
|
+
**Cause.** bf16 compute flips the expert choice of the MoE router. bf16 rounding depends on how many tokens one
|
|
196
|
+
read holds. Each layer's router makes a hard top-8 expert pick (`argpartition`). When the 8th and 9th expert scores
|
|
197
|
+
are nearly equal (at layer 8 the gap was 2.8e-5), rounding flips the pick, and the difference grows through the
|
|
198
|
+
later layers. The norms and the quantization scales and biases were bf16; the weights themselves are 4-bit.
|
|
199
|
+
|
|
200
|
+
**Fix.** `DecisionGem(..., float32=True)` by default. After loading, every bf16 parameter
|
|
201
|
+
(quantization scales and biases, norms, embeddings, router) is cast to float32; the packed 4/8-bit weights stay as
|
|
202
|
+
they are. In float32 the spread is 0.568..0.574 and `extend` matches one read to within 0.006 (not bit-exact).
|
|
203
|
+
`float32=False` keeps the old behaviour.
|
|
204
|
+
|
|
205
|
+
Cost: +1.5 GiB for the model (13.2 → 14.7 GiB), twice the KV cache (≈0.43 MiB per token), reads about 1.3x slower,
|
|
206
|
+
a short +1.5 GiB at load. The "Resources" numbers were measured in bf16.
|
|
207
|
+
|
|
208
|
+
**Saved states.** `save` records the compute dtype (`dtype`), `load` raises `ValueError` when it
|
|
209
|
+
differs from the gem's. A file without the field counts as bf16 (saved before the fix). Without the check a bf16
|
|
210
|
+
state loaded silently into a float32 gem and gave a mixed cache with the same noise.
|
|
211
|
+
|
|
212
|
+
**What it touched.**
|
|
213
|
+
- Everything computed before 2026-10-19 (before version 0.1.0) ran in bf16.
|
|
214
|
+
- Operator / client roles on 145 calls (state = first 20 lines): in float32 no call changed its operator, accuracy
|
|
215
|
+
is the same; 163 s vs 127 s. On short states the noise does not show.
|
|
216
|
+
- Answers on long calls (the checklist, the whole call in the state) may have depended on the noise. Calibrations
|
|
217
|
+
fitted on bf16 answers need refitting.
|
|
218
|
+
|
|
219
|
+
**TF32 on M5 (found 2026-10-19).** Cause: on GPUs with neural accelerators (M5) MLX multiplies float32 matrices in
|
|
220
|
+
TF32 (relative error ~5e-4 instead of ~1e-6) from 128 rows up (dense: from 2), with another kernel below that. So the
|
|
221
|
+
same tokens get different rounding depending on how many one read holds, and the router's near-ties flip as in bf16.
|
|
222
|
+
Symptom on an M5 Max: 0.347 in chunks vs 0.546 whole. `DecisionGem(float32=True)` sets `MLX_ENABLE_TF32=0` and
|
|
223
|
+
checks a float32 matmul against the CPU (`float32_matmuls`); MLX reads the variable once, at the process's first
|
|
224
|
+
matmul, so if one already ran with TF32 on the gem raises `RuntimeError`. With it: 0.5739 in chunks vs 0.5741 whole.
|
|
225
|
+
Cost: reads 2x slower than in TF32 (1594 tokens 1.6 s vs 0.8 s, a question 0.30 s vs 0.18 s).
|
|
226
|
+
|
|
227
|
+
**Check.** `tests/test_open_model.py` (with the model): a long state in chunks equals one read, open equals closed,
|
|
228
|
+
tolerance 0.02. In bf16 these two tests fail (0.2306 and 0.1544). Model-free tests in `tests/test_core.py` cover the
|
|
229
|
+
cast and `dtype` in `save` / `load`.
|
|
230
|
+
|
|
231
|
+
## Where it was used
|
|
232
|
+
|
|
233
|
+
Speech analytics of call-centre calls, model `mlx-community/gemma-4-26b-a4b-it-4bit`.
|
|
234
|
+
|
|
235
|
+
- Operator / client roles: the right operator in 99 of the 100 calls of the roles reference.
|
|
236
|
+
- Checklist item statuses, 199 calls, 2250 items: 7.1 s per call, 9 of 9 manual rulings matched, the option orders
|
|
237
|
+
agree on 2094 of 2250.
|
|
238
|
+
- The full checklist with no analysis prompt: scenario, outcome, call flags, then the items; 199 calls, 15.5 s per
|
|
239
|
+
call, 9 of 9 manual rulings.
|
|
240
|
+
|
|
241
|
+
## Resources
|
|
242
|
+
|
|
243
|
+
Measured on the longest call (3051 tokens of state) and on 199 calls, 2026-10-13, an Apple silicon Mac with 36 GB:
|
|
244
|
+
|
|
245
|
+
| What | Time / memory |
|
|
246
|
+
|---|---|
|
|
247
|
+
| model load | about 6 s, 13.2 GiB |
|
|
248
|
+
| reading a state (`prefill` or the start of `decide`) | 4.2 s for 3051 tokens (0.8 s for 882) |
|
|
249
|
+
| a state's KV cache | ≈0.2 MiB per token: 656 MiB for 3051 tokens; 3 calls of 882–2430 tokens — 1.11 GiB |
|
|
250
|
+
| cache copy per question | 0.1 ms |
|
|
251
|
+
| one question (two passes, options in order and reversed) | 221 ms |
|
|
252
|
+
| generation (`generate`) | 86–97 tokens/s; a call's summary (653 state tokens, 71 answer tokens) in 0.7 s (16.10) |
|
|
253
|
+
| `save` / `load` of a state | 0.34 s / 0.02 s, file 656 MiB |
|
|
254
|
+
| peak memory on long calls | about 17 GiB (27.8 GiB with the MLX buffer cache) |
|
|
255
|
+
| a call in the runs above | 2–30 s, depends on the number of questions |
|
|
256
|
+
|
|
257
|
+
On a 36 GB Mac do not run it next to another large model. The GPU gets about 27 GiB: besides the model, about 15
|
|
258
|
+
kept long calls fit in memory; more through `save` / `load`.
|
|
259
|
+
|
|
260
|
+
## Memory
|
|
261
|
+
|
|
262
|
+
- `cache_limit` (default 0) — bytes of freed buffers MLX keeps for reuse (`mx.set_cache_limit`); `None` — leave the
|
|
263
|
+
MLX limit as it is. With the buffers the peak was 27.8 GiB instead of 17. The limit is process-wide: `unload()`
|
|
264
|
+
puts back the previous one (the MLX default is 34.2 GiB on this Mac).
|
|
265
|
+
- `gem.unload()` frees the model and all kept states (13.2 GiB → 0) without ending the process; `decide()` after it
|
|
266
|
+
raises `RuntimeError`. Or `with DecisionGem() as gem: ...` — unloaded on exit.
|
|
267
|
+
- `forget(key)` — one state's memory (3 calls: 1.11 GiB, after `forget` of one 0.59 GiB).
|
|
268
|
+
|
|
269
|
+
## Other models
|
|
270
|
+
|
|
271
|
+
The model is set by `DecisionGem(repo=...)` or `--model`: an HF repo id or a local folder in mlx-lm format.
|
|
272
|
+
|
|
273
|
+
```python
|
|
274
|
+
gem = DecisionGem("mlx-community/Qwen3-14B-4bit")
|
|
275
|
+
```
|
|
276
|
+
|
|
277
|
+
```bash
|
|
278
|
+
decision-gem request.json --model mlx-community/Qwen3-14B-4bit
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
What a model needs:
|
|
282
|
+
|
|
283
|
+
- mlx-lm format (`mlx_lm.load` loads it), Apple silicon;
|
|
284
|
+
- instruction-tuned, with a chat template;
|
|
285
|
+
- every letter A..Z after the answer template is exactly one token. Checked at load (`letter_ids`): otherwise
|
|
286
|
+
`ValueError` before the first question;
|
|
287
|
+
- the first answer token is a letter, not the start of reasoning. The template gets `enable_thinking=False`. A model
|
|
288
|
+
that always reasons is read wrong, and the check does not catch it.
|
|
289
|
+
|
|
290
|
+
**Small models.** Models up to ~8B parameters, with no fine-tuning and no calibration, answer poorly. In JevBench
|
|
291
|
+
v1.6.0 (Benchmark Heaven, 2026-10-05) raw Qwen3 with a logit readout scored a Capability of 34.6 (4B Instruct),
|
|
292
|
+
30.1 (8B), 8.3 (1.7B) and 8.2 (0.6B); frozen Gemma-4-12B-it and Gemma-4-31B-it scored 70.9 and 77.6, Jev 1.13.0 76.5.
|
|
293
|
+
This covers `gemma-4-e2b` and `gemma-4-e4b` in "Model comparison" too: the tests on them check the package, not the
|
|
294
|
+
quality of the answers. Take a small model only with a calibration fitted on your own labels and its accuracy checked
|
|
295
|
+
on them.
|
|
296
|
+
|
|
297
|
+
Calibration (`[a, b]`, `T`) is per model. The numbers in this README were measured on `gemma-4-26b-a4b-it-4bit`.
|
|
298
|
+
|
|
299
|
+
### Model comparison
|
|
300
|
+
|
|
301
|
+
Measured on 2026-11-13, an Apple silicon Mac with 36 GB, `float32=True`, `orders="reverse"`. Set: 3 short calls,
|
|
302
|
+
6 questions each — 3 noul, 2 choice, 1 score, 18 in all. Tests: `DECISION_GEM_TEST_MODEL=<repo> pytest -m models`.
|
|
303
|
+
|
|
304
|
+
| Model | `-m models` tests | Load | 18 questions |
|
|
305
|
+
|---|---|---|---|
|
|
306
|
+
| `mlx-community/gemma-4-e2b-it-4bit` | 6 of 6 | 1.4 s | 1.7 s |
|
|
307
|
+
| `mlx-community/gemma-4-e4b-it-4bit` | 6 of 6 | 1.8 s | 3.1 s |
|
|
308
|
+
| `mlx-community/Qwen3-14B-4bit` | 6 of 6 | 3.1 s | 9.2 s |
|
|
309
|
+
| `mlx-community/gemma-4-26b-a4b-it-4bit` (default) | 6 of 6 | 4.6 s | 6.7 s |
|
|
310
|
+
|
|
311
|
+
Gemma 26B is an MoE (~4B active parameters), so it answers faster than the dense Qwen3-14B.
|
|
312
|
+
|
|
313
|
+
## Install and run
|
|
314
|
+
|
|
315
|
+
Apple silicon only (MLX). The model must already be in the HF cache, or run with `--online` / `offline=False`.
|
|
316
|
+
|
|
317
|
+
```bash
|
|
318
|
+
pip install decision-gem # or from source: pip install -e ".[test]"
|
|
319
|
+
python -m decision_gem request.json # or: decision-gem request.json; - reads stdin
|
|
320
|
+
python -m decision_gem request.json --calibration cal.json --model <repo or folder>
|
|
321
|
+
pytest # tests without the model; pytest -m models — on the model
|
|
322
|
+
```
|
|
323
|
+
|
|
324
|
+
```python
|
|
325
|
+
from decision_gem import DecisionGem
|
|
326
|
+
|
|
327
|
+
email = {"from": '"Secure Bank" <security@secure-bank-verify.top>', # the state: text or a JSON object
|
|
328
|
+
"subject": "Your account has been limited!",
|
|
329
|
+
"body": "Dear customer, we noticed unusual activity. Verify your identity within 24 hours: "
|
|
330
|
+
"http://secure-bank-verify.top/login or your account will be permanently suspended."}
|
|
331
|
+
qs = {"spam": {"type": "noul", "instructions": "Is this email spam or phishing?"},
|
|
332
|
+
"kind": {"type": "choice", "instructions": "What kind of email is it?",
|
|
333
|
+
"criteria": {"phishing": "tries to steal logins or payment data", "ads": "unsolicited advertising",
|
|
334
|
+
"scam": "promises money or prizes", "normal": "an ordinary email"}},
|
|
335
|
+
"urgency": {"type": "score", "instructions": "How hard does the email push the reader to act right away?",
|
|
336
|
+
"criteria": ["no pressure", "some pressure", "strong pressure"]}}
|
|
337
|
+
|
|
338
|
+
gem = DecisionGem() # mlx-community/gemma-4-26b-a4b-it-4bit from the local HF cache
|
|
339
|
+
gem({"state": email, "questions": qs})
|
|
340
|
+
```
|
|
341
|
+
|
|
342
|
+
The answer (Gemma 26B, 474 input tokens; `legend` and `probabilities` cut):
|
|
343
|
+
|
|
344
|
+
```json
|
|
345
|
+
{"spam": {"type": "noul", "noul": 1.0, "orders_agree": true},
|
|
346
|
+
"kind": {"type": "choice", "choice": "phishing", "confidence": 1.0, "orders_agree": true},
|
|
347
|
+
"urgency": {"type": "score", "score": 3.0, "confidence": 1.0, "orders_agree": true}}
|
|
348
|
+
```
|
|
349
|
+
|
|
350
|
+
On an ordinary email from a colleague about meeting notes the same questions give `spam` 0.0, `kind` `normal`,
|
|
351
|
+
`urgency` 1.01.
|
|
352
|
+
|
|
353
|
+
Several states at once (see "Keyed states"):
|
|
354
|
+
|
|
355
|
+
```python
|
|
356
|
+
with DecisionGem() as gem:
|
|
357
|
+
gem.prefill("mail-1", email1) # number of tokens; ≈0.2 MiB of memory per token until forget / unload
|
|
358
|
+
gem.prefill("mail-2", email2)
|
|
359
|
+
gem.decide(key="mail-2", questions=qs) # the same answer as decide(email2, qs), without reading again
|
|
360
|
+
gem({"key": "mail-1", "questions": qs}) # a Jev request with key instead of state
|
|
361
|
+
gem.save("mail-1", "mail-1.safetensors") # to disk; load() reads it back in ~0.02 s
|
|
362
|
+
gem.forget("mail-1")
|
|
363
|
+
gem.load("mail-1", "mail-1.safetensors") # only a file saved with the same model
|
|
364
|
+
```
|
|
365
|
+
|
|
366
|
+
The full contract is also in the docstring of `decision_gem/core.py`.
|