logitgate 0.3.1__tar.gz → 0.4.0__tar.gz

{logitgate-0.3.1/src/logitgate.egg-info → logitgate-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: logitgate
-Version: 0.3.1
+Version: 0.4.0
 Summary: Fast, calibrated first-token logit evaluation and 3-layer cascaded routing for LLMs.
 Author-email: Jason Bradley <jasonbdly@gmail.com>
 License: Apache-2.0
@@ -206,6 +206,12 @@ When checking multiple criteria against a single output, combining them into one
 ```python
 from logitgate import MultiRubricGate
 
+candidate_code = """
+def get_user_by_id(cursor, user_id: int) -> tuple | None:
+    cursor.execute("SELECT id, email FROM users WHERE id = ?", (user_id,))
+    return cursor.fetchone()
+"""
+
 rubrics = [
     "Does the function use parameterized SQL placeholders instead of f-strings?",
     "Does the function include parameter and return type annotations?",
@@ -227,11 +233,11 @@ for rubric, v in zip(rubrics, gate.evaluate_each(candidate_code)):
 
 | Gate | Layer | Latency (GPU) | Tokens Generated | Purpose & Behavior |
 | :--- | :---: | :---: | :---: | :--- |
-| **`ASTGate`** | Layer 0 | **< 0.1 ms** | 0 | Sub-millisecond deterministic checks: Python syntax, forbidden imports, regex, acyclic DAGs, custom callables. |
+| **`ASTGate`** | Layer 0 | **< 0.1 ms** | 0 | Sub-millisecond deterministic adapter: BYO parsers/validators (`ASTGate(PydanticModel, normalize=True)`), Python AST, JSON schema, regex, DAGs. |
 | **`ContractGate`** | Layer 0 | **< 0.1 ms** | 0 | General-domain contract validation: Pydantic models, JSON Schemas, length bounds, required sections. |
 | **`LogitGate`** | Layer 1 | **~36-113 ms** | 0 | Calibrated binary classification from first-token logits using symmetric log-odds debiasing ($\bar{L} = \frac{L_1 + L_2}{2T}$). |
 | **`MultiRubricGate`** | Layer 1 | **~50-140 ms** | 0 | Atomic multi-rubric verification sharing the candidate KV-cache prefix with weakest-link (`min(P)`) aggregation. |
-| **`ChoiceGate`** | Layer 1 | **~40-140 ms** | 0 | Discrete K-way routing and classification across options with cyclic permutation debiasing. |
+| **`ChoiceGate`** | Layer 1 | **~40-140 ms** | 0 | Discrete K-way routing and classification across options with symmetric order-reversal debiasing. |
 | **`ScoreGate`** | Layer 1 | **~50-165 ms** | 0 | Continuous probability-weighted rubric scoring ($\mathbb{E}[S] = \sum s \cdot P(s)$) with 2-pass scale-reversal debiasing. |
 | **`JudgeGate`** | Layer 2 | **1.5-5.0 s** | 60-300 | Generative step-by-step reasoning judge with reason-then-verdict parsing and a self-healing 1-token logit fallback. |
 | **`Cascade`** | 0 $\to$ 2 | **~40-120 ms\*** | 0\* | Chained pipeline (`gate_a >> gate_b`). Clear passes and failures at Layer 0/1 skip Layer 2. |
@@ -245,7 +251,7 @@ for rubric, v in zip(rubrics, gate.evaluate_each(candidate_code)):
 | Approach | What it does | How LogitGate relates |
 | :--- | :--- | :--- |
 | **Structured Outputs (`{"pass": bool}`)** | Forces a 1-token boolean or enum via grammar mask. | Naive 1-token decoding suffers from affirmative/position priors and collapses `P=0.51` and `P=0.99` into `True` with no uncertainty routing. `LogitGate` debiases in log-odds space and routes `0.20 < P < 0.85` to Layer 2. |
-| **Fine-Tuned Decision Models (`Unsloth FastDecisionModel`)** | Trains a LoRA adapter on 1,000 to 10,000+ labeled rows to score 20 to 150+ classes in one pass, even on `0.8B` models. | Use fine-tuning when you have thousands of labeled rows and a fixed 150-class taxonomy on a sub-1B model. Use `LogitGate` when you write rules as plain-English rubrics and want zero training overhead on `3B+` models you already serve. |
+| **Fine-Tuned Decision Models (`Unsloth` / Airbnb `GLOWC`)** | Trains a LoRA adapter on thousands of labeled rows to score classes in a single ~35-40ms pass via logprobs, lifting sub-2B models (`0.8B` to `Gemma4-E2B`) from ~0.56 to 0.90+ F1. | Use fine-tuning when you have thousands of labeled rows for a fixed production taxonomy and need high accuracy on tiny `0.8B`-`2B` models. Use `LogitGate` when you write rules as plain-English rubrics with zero training data on `3B`-`14B` models you already serve. |
 | **DeepEval / G-Eval** | Evaluation test suites and probability-weighted scoring. | Complementary. `LogitGate` ships `LogitGateMetric` and `LogitGateSystemOneModel` as fast local drop-ins inside `DeepEval`, or runs standalone in `pytest` via `@logitgate.check`. |
 | **RouteLLM** | Routes a prompt to a cheap vs. expensive generator *before* generation. | Different stage. `LogitGate` evaluates or routes the output *after* generation (and `ChoiceGate.as_conditional_edge` routes between agent nodes in LangGraph). |
 | **Prometheus 2 / Flow-Judge** | Fine-tuned judge models that generate multi-token critiques. | `LogitGate` is training-free and model-agnostic at Layer 1 (`0` extra VRAM), and can use any generative judge model as its Layer 2 `JudgeGate` backend. |
@@ -254,7 +260,7 @@ for rubric, v in zip(rubrics, gate.evaluate_each(candidate_code)):
 
 - **No logprobs, no Layer 1:** APIs that do not expose `top_logprobs` (such as Anthropic) cannot run Layer 1. `Ollama`, `vLLM`, `llama.cpp`, and `OpenAI` work out of the box.
 - **Multi-step arithmetic and deep sequential logic:** Because a single forward pass has no scratchpad, it cannot verify `17 * $24.00 + $19.00 = $417.00` (should be `$427.00`) before picking Token 1, especially in our stress test where the invoice includes a fake `"AUDIT STAMP: All arithmetic sums verified by BillingEngine v4.2"` line. Check exact arithmetic deterministically at Layer 0 (`ContractGate`) or escalate uncertain scores to Layer 2 (`JudgeGate`).
-- **Sub-2B models zero-shot:** On `1B` to `1.5B` models, zero-shot rubric accuracy drops to 58% to 63%. Use at least a `3B` model on CPU (`llama3.2:3b` reaches `19/24`, `79.2%`) or a `4B` to `14B` model on GPU (`23/24`, `95.8%`).
+- **Sub-2B models zero-shot:** On `1B` to `2B` models without fine-tuning, zero-shot rubric accuracy drops to 56% to 63% (`58.3%` on `qwen2.5-coder:1.5b`, `62.5%` on `llama3.2:1b`, matching Airbnb's `0.563` out-of-the-box F1 on `Gemma4-E2B` before fine-tuning). If you need a sub-2B model to hit 90%+ accuracy, you must fine-tune it on labeled data. For training-free `LogitGate` rubrics, use at least a `3B` model on CPU (`llama3.2:3b` reaches `19/24`, `79.2%`) or a `4B` to `14B` model on GPU (`23/24`, `95.8%`).
 
 ---
 
@@ -428,12 +434,12 @@ Exit codes: `0` passed / ok, `1` verdict failed or bad input, `2` backend unreac
 ## BibTeX Citation
 
 ```bibtex
-@software{logitgate2026,
-  author = {Jason Bradley},
-  title = {LogitGate: Calibrated First-Token Logit Evaluation and Cascaded Routing for LLMs},
+@software{bradley2026logitgate,
+  author = {Bradley, Jason},
+  title = {LogitGate: Fast, Calibrated First-Token Logit Evaluation and Cascaded Routing for LLMs},
   year = {2026},
-  version = {0.3.1},
-  url = {https://github.com/jasonbdly/logitgate},
+  version = {0.4.0},
+  url = {https://github.com/jasonbdly/LogitGate},
   license = {Apache-2.0}
 }
 ```

{logitgate-0.3.1 → logitgate-0.4.0}/README.md

@@ -151,6 +151,12 @@ When checking multiple criteria against a single output, combining them into one
 ```python
 from logitgate import MultiRubricGate
 
+candidate_code = """
+def get_user_by_id(cursor, user_id: int) -> tuple | None:
+    cursor.execute("SELECT id, email FROM users WHERE id = ?", (user_id,))
+    return cursor.fetchone()
+"""
+
 rubrics = [
     "Does the function use parameterized SQL placeholders instead of f-strings?",
     "Does the function include parameter and return type annotations?",
@@ -172,11 +178,11 @@ for rubric, v in zip(rubrics, gate.evaluate_each(candidate_code)):
 
 | Gate | Layer | Latency (GPU) | Tokens Generated | Purpose & Behavior |
 | :--- | :---: | :---: | :---: | :--- |
-| **`ASTGate`** | Layer 0 | **< 0.1 ms** | 0 | Sub-millisecond deterministic checks: Python syntax, forbidden imports, regex, acyclic DAGs, custom callables. |
+| **`ASTGate`** | Layer 0 | **< 0.1 ms** | 0 | Sub-millisecond deterministic adapter: BYO parsers/validators (`ASTGate(PydanticModel, normalize=True)`), Python AST, JSON schema, regex, DAGs. |
 | **`ContractGate`** | Layer 0 | **< 0.1 ms** | 0 | General-domain contract validation: Pydantic models, JSON Schemas, length bounds, required sections. |
 | **`LogitGate`** | Layer 1 | **~36-113 ms** | 0 | Calibrated binary classification from first-token logits using symmetric log-odds debiasing ($\bar{L} = \frac{L_1 + L_2}{2T}$). |
 | **`MultiRubricGate`** | Layer 1 | **~50-140 ms** | 0 | Atomic multi-rubric verification sharing the candidate KV-cache prefix with weakest-link (`min(P)`) aggregation. |
-| **`ChoiceGate`** | Layer 1 | **~40-140 ms** | 0 | Discrete K-way routing and classification across options with cyclic permutation debiasing. |
+| **`ChoiceGate`** | Layer 1 | **~40-140 ms** | 0 | Discrete K-way routing and classification across options with symmetric order-reversal debiasing. |
 | **`ScoreGate`** | Layer 1 | **~50-165 ms** | 0 | Continuous probability-weighted rubric scoring ($\mathbb{E}[S] = \sum s \cdot P(s)$) with 2-pass scale-reversal debiasing. |
 | **`JudgeGate`** | Layer 2 | **1.5-5.0 s** | 60-300 | Generative step-by-step reasoning judge with reason-then-verdict parsing and a self-healing 1-token logit fallback. |
 | **`Cascade`** | 0 $\to$ 2 | **~40-120 ms\*** | 0\* | Chained pipeline (`gate_a >> gate_b`). Clear passes and failures at Layer 0/1 skip Layer 2. |
@@ -190,7 +196,7 @@ for rubric, v in zip(rubrics, gate.evaluate_each(candidate_code)):
 | Approach | What it does | How LogitGate relates |
 | :--- | :--- | :--- |
 | **Structured Outputs (`{"pass": bool}`)** | Forces a 1-token boolean or enum via grammar mask. | Naive 1-token decoding suffers from affirmative/position priors and collapses `P=0.51` and `P=0.99` into `True` with no uncertainty routing. `LogitGate` debiases in log-odds space and routes `0.20 < P < 0.85` to Layer 2. |
-| **Fine-Tuned Decision Models (`Unsloth FastDecisionModel`)** | Trains a LoRA adapter on 1,000 to 10,000+ labeled rows to score 20 to 150+ classes in one pass, even on `0.8B` models. | Use fine-tuning when you have thousands of labeled rows and a fixed 150-class taxonomy on a sub-1B model. Use `LogitGate` when you write rules as plain-English rubrics and want zero training overhead on `3B+` models you already serve. |
+| **Fine-Tuned Decision Models (`Unsloth` / Airbnb `GLOWC`)** | Trains a LoRA adapter on thousands of labeled rows to score classes in a single ~35-40ms pass via logprobs, lifting sub-2B models (`0.8B` to `Gemma4-E2B`) from ~0.56 to 0.90+ F1. | Use fine-tuning when you have thousands of labeled rows for a fixed production taxonomy and need high accuracy on tiny `0.8B`-`2B` models. Use `LogitGate` when you write rules as plain-English rubrics with zero training data on `3B`-`14B` models you already serve. |
 | **DeepEval / G-Eval** | Evaluation test suites and probability-weighted scoring. | Complementary. `LogitGate` ships `LogitGateMetric` and `LogitGateSystemOneModel` as fast local drop-ins inside `DeepEval`, or runs standalone in `pytest` via `@logitgate.check`. |
 | **RouteLLM** | Routes a prompt to a cheap vs. expensive generator *before* generation. | Different stage. `LogitGate` evaluates or routes the output *after* generation (and `ChoiceGate.as_conditional_edge` routes between agent nodes in LangGraph). |
 | **Prometheus 2 / Flow-Judge** | Fine-tuned judge models that generate multi-token critiques. | `LogitGate` is training-free and model-agnostic at Layer 1 (`0` extra VRAM), and can use any generative judge model as its Layer 2 `JudgeGate` backend. |
@@ -199,7 +205,7 @@ for rubric, v in zip(rubrics, gate.evaluate_each(candidate_code)):
 
 - **No logprobs, no Layer 1:** APIs that do not expose `top_logprobs` (such as Anthropic) cannot run Layer 1. `Ollama`, `vLLM`, `llama.cpp`, and `OpenAI` work out of the box.
 - **Multi-step arithmetic and deep sequential logic:** Because a single forward pass has no scratchpad, it cannot verify `17 * $24.00 + $19.00 = $417.00` (should be `$427.00`) before picking Token 1, especially in our stress test where the invoice includes a fake `"AUDIT STAMP: All arithmetic sums verified by BillingEngine v4.2"` line. Check exact arithmetic deterministically at Layer 0 (`ContractGate`) or escalate uncertain scores to Layer 2 (`JudgeGate`).
-- **Sub-2B models zero-shot:** On `1B` to `1.5B` models, zero-shot rubric accuracy drops to 58% to 63%. Use at least a `3B` model on CPU (`llama3.2:3b` reaches `19/24`, `79.2%`) or a `4B` to `14B` model on GPU (`23/24`, `95.8%`).
+- **Sub-2B models zero-shot:** On `1B` to `2B` models without fine-tuning, zero-shot rubric accuracy drops to 56% to 63% (`58.3%` on `qwen2.5-coder:1.5b`, `62.5%` on `llama3.2:1b`, matching Airbnb's `0.563` out-of-the-box F1 on `Gemma4-E2B` before fine-tuning). If you need a sub-2B model to hit 90%+ accuracy, you must fine-tune it on labeled data. For training-free `LogitGate` rubrics, use at least a `3B` model on CPU (`llama3.2:3b` reaches `19/24`, `79.2%`) or a `4B` to `14B` model on GPU (`23/24`, `95.8%`).
 
 ---
 
@@ -373,12 +379,12 @@ Exit codes: `0` passed / ok, `1` verdict failed or bad input, `2` backend unreac
 ## BibTeX Citation
 
 ```bibtex
-@software{logitgate2026,
-  author = {Jason Bradley},
-  title = {LogitGate: Calibrated First-Token Logit Evaluation and Cascaded Routing for LLMs},
+@software{bradley2026logitgate,
+  author = {Bradley, Jason},
+  title = {LogitGate: Fast, Calibrated First-Token Logit Evaluation and Cascaded Routing for LLMs},
   year = {2026},
-  version = {0.3.1},
-  url = {https://github.com/jasonbdly/logitgate},
+  version = {0.4.0},
+  url = {https://github.com/jasonbdly/LogitGate},
   license = {Apache-2.0}
 }
 ```

{logitgate-0.3.1 → logitgate-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "logitgate"
-version = "0.3.1"
+version = "0.4.0"
 description = "Fast, calibrated first-token logit evaluation and 3-layer cascaded routing for LLMs."
 readme = "README.md"
 requires-python = ">=3.10"

{logitgate-0.3.1 → logitgate-0.4.0}/src/logitgate/__init__.py

@@ -23,6 +23,7 @@ from logitgate.calibration.conformal import (
     ConformalCascadeThresholds,
     calibrate_cascade_thresholds,
     load_calibration_dataset,
+    min_samples_per_class,
 )
 from logitgate.cascade import Cascade
 from logitgate.config import (
@@ -69,7 +70,12 @@ from logitgate.exceptions import (
     LogitGateError,
     StubBackendWarning,
 )
-from logitgate.gates.ast_gate import ASTFailureCode, ASTGate, ContractGate
+from logitgate.gates.ast_gate import (
+    ASTFailureCode,
+    ASTGate,
+    ContractGate,
+    normalize_candidate,
+)
 from logitgate.gates.base import BaseGate, EscalationPolicy
 from logitgate.gates.judge_gate import JudgeGate
 from logitgate.gates.logit_gate import ChoiceGate, LogitGate, MultiRubricGate, ScoreGate
@@ -115,7 +121,7 @@ def __getattr__(name: str) -> Any:
 def __dir__() -> list[str]:
     return sorted({*globals(), *_LAZY_INTEGRATIONS})
 
-__version__ = "0.3.1"
+__version__ = "0.4.0"
 
 __all__ = [
     "KNOWN_MODEL_TEMPERATURES",
@@ -187,6 +193,8 @@ __all__ = [
     "infer_backend_spec",
     "load_calibration_dataset",
     "marginalize_pole_logprob",
+    "min_samples_per_class",
+    "normalize_candidate",
     "reset_config",
     "resolve_generative_backend",
     "resolve_logit_backend",