nullspan 1.1.0__tar.gz → 1.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (7) hide show
  1. {nullspan-1.1.0 → nullspan-1.2.0}/PKG-INFO +35 -34
  2. nullspan-1.2.0/README.md +128 -0
  3. {nullspan-1.1.0 → nullspan-1.2.0}/pyproject.toml +1 -1
  4. {nullspan-1.1.0 → nullspan-1.2.0}/src/nullspan/__init__.py +1 -1
  5. {nullspan-1.1.0 → nullspan-1.2.0}/src/nullspan/engine.py +83 -23
  6. nullspan-1.1.0/README.md +0 -127
  7. {nullspan-1.1.0 → nullspan-1.2.0}/LICENSE +0 -0
{nullspan-1.1.0 → nullspan-1.2.0}/PKG-INFO RENAMED
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: nullspan
3
- Version: 1.1.0
3
+ Version: 1.2.0
4
4
  Summary: Calibrated p_null abstention, grounded [start, end] span extraction, arithmetic, and distribution forecasting in a single forward pass.
5
5
  Keywords: nullspan,system-one,extraction,abstention,p-null,quantile-regression,forecasting
6
6
  Author-email: Hemanth HM <hemanth.hm@gmail.com>
@@ -20,22 +20,22 @@ Provides-Extra: hf
20
20
 
21
21
  # nullspan
22
22
 
23
- Single-pass System One structured extraction, calibrated `p_null` abstention, character-offset `[start, end]` spans, zero-shot `boolean`/`enum` judgment, cross-field algebraic invariant verification, and calibrated quantile estimation (`p10`, `median`, `p90`) in JavaScript, Python, and Browser WebAssembly ONNX.
23
+ Single-pass System One structured extraction, multi-numeric contract disambiguation, calibrated `p_null` abstention, character-offset `[start, end]` spans, zero-shot `boolean`/`enum` judgment, cross-field algebraic invariant verification, and calibrated quantile estimation (`p10`, `median`, `p90`) in JavaScript, Python, and Browser WebAssembly ONNX (`512`-token context + long-document windowing).
24
24
 
25
25
  - **Interactive Playground (Browser WASM ONNX + Zero-Dep Engine)**: [https://hemanth.github.io/nullspan/](https://hemanth.github.io/nullspan/)
26
- - **NullspanBench Leaderboard (JevBench-Style 14-System Evaluation)**: [https://hemanth.github.io/nullspan/bench.html](https://hemanth.github.io/nullspan/bench.html)
26
+ - **NullspanBench Leaderboard (JevBench-Style 19-System Evaluation, N=500)**: [https://hemanth.github.io/nullspan/bench.html](https://hemanth.github.io/nullspan/bench.html)
27
27
  - **Hugging Face Space**: [https://huggingface.co/spaces/gnumanth/nullspan](https://huggingface.co/spaces/gnumanth/nullspan)
28
- - **Fine-Tuned Hugging Face Model (67M PyTorch + 64.3 MB Int8 ONNX)**: [https://huggingface.co/gnumanth/nullspan](https://huggingface.co/gnumanth/nullspan)
28
+ - **Fine-Tuned Hugging Face Model (344.2M PyTorch + 330.1 MB Int8 ONNX)**: [https://huggingface.co/gnumanth/nullspan](https://huggingface.co/gnumanth/nullspan)
29
29
  - **Hugging Face Dataset**: [https://huggingface.co/datasets/gnumanth/nullspan-dataset](https://huggingface.co/datasets/gnumanth/nullspan-dataset)
30
30
  - **Empirical Science Report**: [`bench/SCIENCE_REPORT.md`](./bench/SCIENCE_REPORT.md)
31
31
 
32
32
  ## In Plain English
33
33
 
34
- Standard LLMs extract data by generating JSON token-by-token—which is slow, costs money per output token, can get hijacked by *"ignore previous instructions"* prompt injections, and often hallucinates values when a field is missing. **`nullspan` acts like a smart highlighter + calculator instead of a chatbot:**
34
+ Standard LLMs extract data by generating JSON token-by-token, which is slow, bills per output token, can get hijacked by *"ignore previous instructions"* prompt injections, and often hallucinates values when a field is missing. **`nullspan` acts like a grounded highlighter + calculator instead of a chatbot:**
35
35
 
36
36
  1. **Points to Exact Text (`[start, end]` Span)**: Highlights exact character positions in your input text (`0` output tokens generated), making fabricated strings and prompt-injection payloads impossible.
37
- 2. **Knows When to Say "Not There" (`p_null`)**: Computes a calibrated probability that a field is missing (`p_null`) and returns `null` when evidence is absent.
38
- 3. **Checks the Math Automatically (`constraint` & `verified`)**: Computes sums, durations, and products and verifies cross-field equations like `subtotal == nodes * hourly_rate`.
37
+ 2. **Knows When to Say "Not There" (`p_null`)**: Computes a calibrated probability that a field is missing (`p_null`) and returns `null` with `abstention_reason` when evidence is absent.
38
+ 3. **Disambiguates Multi-Numeric Contracts & Checks Math (`constraint` & `verified`)**: Separates competing dollar caps, percentages, day windows, and numeric tranches across multi-section agreements (`96/96` on unseen contracts) and verifies equations like `subtotal == nodes * hourly_rate`.
39
39
  4. **Gives Realistic Ranges (`p10 / median / p90`)**: Returns calibrated low/middle/high quantile distributions for estimates and forecasts.
40
40
 
41
41
  ## Install
@@ -82,7 +82,7 @@ invoice = (
82
82
  "Priority Surcharge: $1.75"
83
83
  )
84
84
 
85
- # Zero-dependency fast engine or fine-tuned neural checkpoint (backend="hf")
85
+ # Zero-dependency fast engine or GCP V100 fine-tuned 344.2M checkpoint (backend="hf" or "onnx")
86
86
  out = extract(
87
87
  state=invoice,
88
88
  fields={
@@ -103,44 +103,45 @@ out = extract(
103
103
  # out["invariants_passed"] -> True, out["answers"]["subtotal"]["verified"] -> True
104
104
  ```
105
105
 
106
- ## Architectural Enhancements
106
+ ## Architecture (`NullspanForExtraction` v2.0 · 344.2M ELECTRA-Large + Adapter)
107
107
 
108
108
  ![Architecture Flow](./docs/flow-diagram.svg)
109
109
 
110
- 1. **`null_head` (Calibrated Abstention & Reason Diagnostics)**: Predicts calibrated `p_null`, nearest decoy `guess`, and `abstention_reason` (`unassigned_placeholder` vs `missing_field`).
111
- 2. **`span_head` (Grounded Pointer + Shannon Entropy)**: Projects encoder tokens to `[start, end]` character spans with calibrated `confidence` and Shannon `span_entropy_bits`.
112
- 3. **`bio_head` (Class-Weighted Multi-Span Tagging)**: Extracts arbitrary-length ordered arrays (`type: "list"`) in a single pass.
110
+ 1. **`null_head` (Span-Attended Calibrated Abstention)**: Bilinear span-summary + `[CLS]` margin residual with held-out temperature scaling (`p_null` AUROC `0.9983`, ECE `0.0121`) and `abstention_reason` (`unassigned_placeholder` vs `missing_field`).
111
+ 2. **`span_head` (Grounded Pointer + Shannon Entropy)**: Projects `512`-token encoder states (plus query-conditioned sliding windows on ultra-long contracts) to `[start, end]` character spans with calibrated `confidence` and Shannon `span_entropy_bits`.
112
+ 3. **`bio_head` (Multi-Span List Tagging)**: Extracts ordered text and numeric tranche arrays (`type: "list"`) in a single pass.
113
113
  4. **`op_head` (10-Way System One Router)**: Routes fields across `SPAN_TEXT`, `SPAN_NUMBER`, `ARITH_SUM`, `ARITH_DIFF`, `ARITH_MUL`, `ISO_DATETIME`, `DATE_DURATION`, `DIST_ESTIMATE_FORECAST`, `BOOL_JUDGE` (`p_true`), and `ENUM_CLASSIFY` (`probs`).
114
114
  5. **`quantile_head` + Invariant Verifier**: Predicts monotone `(p10, median, p90)` quantiles and verifies cross-field algebraic constraints (`verified: true/false`).
115
115
 
116
- ## NullspanBench Leaderboard (`bench/run_bench.py` · 14 Systems · $N=404$ Scored Decisions)
116
+ ## NullspanBench Leaderboard (`bench/run_bench.py` · 19 Systems · $N=500$ Zero-Leakage Decisions)
117
117
 
118
- Modeled after [BenchmarkHeaven's JevBench](https://benchmarkheaven.com/jev-models), **NullspanBench** evaluates `14` systems (`12` local open-weights/open-code models running live forward passes + `2` reference JSON-mode LLM decoders) across **`404` scored observations** (`336` Open structured items + `68` Sealed Stanford SQuAD v2.0 & adversarial prompt-injection items) on four axes: **Intelligence (`I`)**, **Calibration (`C`)**, **Speed (`S`)**, and **Cost (`K`)**.
118
+ Modeled after [BenchmarkHeaven's JevBench](https://benchmarkheaven.com/jev-models), **NullspanBench v3.2** evaluates `19` systems (`13` local open-weights/open-code models running live forward passes, live Cloudflare Workers AI `@cf/cloudflare/clairic + @cf/cloudflare/clef`, `2` reference JSON-mode LLM decoders, and `3` published JevBench v1.7 reference anchors) across **`500` zero-leakage observations** (`336` Open structured items + `164` Sealed items: `60` Stanford SQuAD v2.0, `96` Multi-Numeric Legal/Commercial Contract fields across `12` unseen agreements, and `8` Adversarial prompt-injection traps) on four axes: **Intelligence (`I`)**, **Calibration (`C`)**, **Speed (`S`)**, and **Cost (`K`)**.
119
119
 
120
120
  Explore interactive weight sliders, Pareto frontier plots, and 2-system radar comparisons at [**https://hemanth.github.io/nullspan/bench.html**](https://hemanth.github.io/nullspan/bench.html):
121
121
 
122
122
  | Rank | System | Composite (`25:25:25:25`) | Capability (`50I:50C`) | Intelligence `I` (95% CI) | Calibration `C` | Speed `S` | Cost `K` | `$/1k` | `p50` Latency |
123
123
  | :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
124
- | **1** | **`nullspan-Hybrid (Fast + ONNX Int8)`** (Open) | **97.7** | **98.7** | **98.0%** `[96.5, 99.3]` | **99.3** | **99.8** | **93.8** | `$0.0003` | `0.005 ms` |
125
- | **2** | **`nullspan-ZeroDep (Fast Engine)`** (Open) | **95.3** | **91.2** | `88.6%` `[85.6, 91.6]` | `93.8` | **99.8** | **99.8** | `$0.0001` | `0.005 ms` |
126
- | **3** | **`nullspan-67M (PyTorch FP32)`** (Open) | **83.3** | **99.2** | **98.8%** `[97.5, 99.8]` | **99.7** | `66.5` | `78.0` | `$0.0015` | `5.575 ms` |
127
- | **4** | **`nullspan-67M-Int8 (ONNX WASM/CPU)`** (Open) | **83.0** | **98.7** | **98.0%** `[96.5, 99.3]` | **99.3** | `62.3` | `84.2` | `$0.0008` | `12.268 ms` |
128
- | **5** | **`nullspan-67M (ONNX FP32)`** (Open) | **82.1** | **99.2** | **98.8%** `[97.5, 99.8]` | **99.7** | `63.0` | `78.7` | `$0.0014` | `10.832 ms` |
129
- | **6** | **`@cf/cloudflare/clairic + clef (Live Workers AI)`** | `74.4` | `92.0` | `90.1%` `[87.5, 92.7]` | `93.8` | `51.5` | `72.2` | `$0.0042` | `95.028 ms` (`308.6 ms/doc`) |
130
- | **7** | `H2O-Lightning-4B v1.1 (JevBench #1 Comp)` (Open 4B) | `72.5` | `75.0` | `60.0%` `[57.8, 62.3]` | `90.0` | `92.6` | `60.3` | `$0.0211` | `29.285 ms` |
131
- | **7** | `basal-omni-1.0-ultra (JevBench #1 Cap)` (Open 31B) | `71.8` | `82.5` | `74.1%` `[71.8, 76.3]` | `91.0` | `87.3` | `50.5` | `$0.0447` | `102.000 ms` |
132
- | **8** | `Jev 1.13.0 (TypeSafe AI · JevBench Ref)` (API) | `71.5` | `77.1` | `63.6%` `[61.2, 66.0]` | `90.6` | `91.5` | `54.7` | `$0.0323` | `239.102 ms` |
133
- | **9** | `gpt-5.4-nano (JSON Decoder · API)` | `60.3` | `91.5` | `95.3%` `[93.1, 97.3]` | `87.7` | `43.6` | `46.5` | `$0.0380` | `415.9 ms` |
134
- | **10** | `claude-haiku-4.5 (JSON Decoder · API)` | `49.7` | `92.3` | `96.3%` `[94.3, 98.0]` | `88.3` | `41.8` | `28.7` | `$0.2328` | `588.0 ms` |
135
- | **11** | `bge-small-en-v1.5 (Dense Span+Null)` (Open) | `42.0` | `53.3` | `17.5%` `[13.8, 21.3]` | `89.1` | `64.7` | `87.0` | `$0.0006` | `7.855 ms` |
136
- | **12** | `distilbert-base-control (Raw Control)` (Open) | `41.6` | `53.6` | `17.3%` `[13.6, 21.1]` | `90.0` | `65.3` | `84.2` | `$0.0008` | `6.947 ms` |
137
- | **13** | `all-MiniLM-L12-v2 (Dense Span+Null)` (Open) | `37.6` | `44.8` | `15.1%` `[11.9, 18.6]` | `74.6` | `65.4` | `87.0` | `$0.0006` | `6.894 ms` |
138
- | **14** | `all-mpnet-base-v2 (Dense Span+Null)` (Open) | `35.3` | `43.1` | `14.1%` `[10.9, 17.5]` | `72.2` | `62.8` | `80.2` | `$0.0012` | `11.098 ms` |
139
- | **15** | `paraphrase-albert-small-v2 (Dense Span+Null)` (Open) | `34.8` | `45.1` | `13.1%` `[9.9, 16.4]` | `77.2` | `66.8` | `93.8` | `$0.0003` | `5.224 ms` |
140
- | **16** | `paraphrase-MiniLM-L3-v2 (Dense Span+Null)` (Open) | `31.5` | `42.2` | `11.3%` `[8.4, 14.5]` | `73.1` | `70.6` | `93.8` | `$0.0003` | `2.586 ms` |
141
- | **17** | `all-MiniLM-L6-v2 (Dense Span+Null)` (Open) | `28.5` | `41.2` | `9.9%` `[7.1, 13.0]` | `72.6` | `68.0` | `91.0` | `$0.0004` | `4.200 ms` |
142
-
143
- See full methodology, proper scoring definitions, request-type breakdowns, and ablation analysis in [`bench/SCIENCE_REPORT.md`](./bench/SCIENCE_REPORT.md).
124
+ | **1** | **`nullspan-Hybrid (Fast + ONNX Int8)`** (Open) | **97.2** | **98.3** | **97.8%** `[96.6, 99.0]` | **98.8** | **98.6** | **93.8** | `$0.0003` | `0.013 ms` |
125
+ | **2** | **`nullspan-ZeroDep (Fast Engine)`** (Open) | **95.9** | **92.9** | `90.8%` `[88.2, 93.4]` | `95.0` | **98.7** | **99.8** | `$0.0001` | `0.013 ms` |
126
+ | **3** | **`nullspan-344M (PyTorch FP32)`** (Open) | **78.7** | **98.4** | **98.0%** `[96.8, 99.2]` | **98.8** | `57.2` | `76.8` | `$0.0017` | `32.367 ms` |
127
+ | **4** | **`nullspan-344M-Int8 (ONNX WASM/CPU)`** (Open) | **78.4** | **98.3** | **97.8%** `[96.6, 99.0]` | **98.9** | `53.2` | `84.2` | `$0.0008` | `68.483 ms` |
128
+ | **5** | **`nullspan-344M (ONNX FP32)`** (Open) | **75.5** | **98.4** | **98.0%** `[96.8, 99.2]` | **98.8** | `50.5` | `78.0` | `$0.0015` | `113.137 ms` |
129
+ | **6** | `H2O-Lightning-4B v1.1 (JevBench #1 Comp)` (Open 4B) | `72.5` | `75.0` | `60.0%` `[57.8, 62.3]` | `90.0` | `92.6` | `60.3` | `$0.0211` | `29.285 ms` |
130
+ | **7** | **`@cf/cloudflare/clairic + clef (Live Workers AI)`** (API) | `72.1` | `93.2` | `91.6%` `[89.2, 93.9]` | `94.9` | `51.7` | `68.2` | `$0.0041` | `91.323 ms` (`315.3 ms/doc`) |
131
+ | **8** | `basal-omni-1.0-ultra (JevBench #1 Cap)` (Open 31B) | `71.8` | `82.5` | `74.1%` `[71.8, 76.3]` | `91.0` | `87.3` | `50.5` | `$0.0447` | `102.000 ms` |
132
+ | **9** | `Jev 1.13.0 (TypeSafe AI · JevBench Ref)` (API) | `71.5` | `77.1` | `63.6%` `[61.2, 66.0]` | `90.6` | `91.5` | `54.7` | `$0.0323` | `239.102 ms` |
133
+ | **10** | `gpt-5.4-nano (JSON Decoder · API)` (API) | `59.8` | `92.1` | `95.2%` `[93.4, 97.0]` | `89.0` | `43.6` | `45.1` | `$0.0438` | `417.991 ms` |
134
+ | **11** | `deberta-v3-base-squad2 (184M SQuAD2 QA)` (Open) | `58.0` | `55.8` | `48.1%` `[43.9, 52.4]` | `63.4` | `55.7` | `69.2` | `$0.0037` | `42.496 ms` |
135
+ | **12** | `claude-haiku-4.5 (JSON Decoder · API)` (API) | `48.8` | `92.1` | `95.4%` `[93.4, 97.2]` | `88.7` | `41.8` | `27.5` | `$0.2639` | `586.392 ms` |
136
+ | **13** | `bge-small-en-v1.5 (Dense Span+Null)` (Open) | `37.6` | `52.0` | `14.7%` `[12.0, 17.5]` | `89.2` | `64.5` | `85.5` | `$0.0007` | `8.098 ms` |
137
+ | **14** | `distilbert-base-control (Raw Control)` (Open) | `36.8` | `52.9` | `14.2%` `[11.6, 17.0]` | `91.5` | `66.0` | `81.1` | `$0.0011` | `6.133 ms` |
138
+ | **15** | `all-MiniLM-L12-v2 (Dense Span+Null)` (Open) | `35.9` | `44.0` | `14.4%` `[11.7, 17.5]` | `73.5` | `60.2` | `85.5` | `$0.0007` | `18.356 ms` |
139
+ | **16** | `all-mpnet-base-v2 (Dense Span+Null)` (Open) | `34.3` | `42.3` | `13.6%` `[10.9, 16.7]` | `71.0` | `62.8` | `77.4` | `$0.0016` | `11.263 ms` |
140
+ | **17** | `paraphrase-albert-small-v2 (Dense Span+Null)` (Open) | `33.6` | `44.5` | `12.5%` `[10.0, 15.2]` | `76.6` | `65.9` | `91.0` | `$0.0004` | `6.263 ms` |
141
+ | **18** | `paraphrase-MiniLM-L3-v2 (Dense Span+Null)` (Open) | `30.9` | `41.7` | `11.1%` `[8.7, 13.8]` | `72.3` | `68.9` | `91.0` | `$0.0004` | `3.519 ms` |
142
+ | **19** | `all-MiniLM-L6-v2 (Dense Span+Null)` (Open) | `28.3` | `40.5` | `10.0%` `[7.7, 12.3]` | `70.9` | `62.3` | `88.8` | `$0.0005` | `12.322 ms` |
143
+
144
+ See full methodology, proper scoring definitions, request-type breakdowns (`100.0%` on Multi-Numeric Contracts `n=96`, `100.0%` Choice `n=41`, `100.0%` Math & Invariants `n=51`), and head-to-head analysis in [`bench/SCIENCE_REPORT.md`](./bench/SCIENCE_REPORT.md).
144
145
 
145
146
  ## License
146
147
 
nullspan-1.2.0/README.md ADDED
@@ -0,0 +1,128 @@
1
+ # nullspan
2
+
3
+ Single-pass System One structured extraction, multi-numeric contract disambiguation, calibrated `p_null` abstention, character-offset `[start, end]` spans, zero-shot `boolean`/`enum` judgment, cross-field algebraic invariant verification, and calibrated quantile estimation (`p10`, `median`, `p90`) in JavaScript, Python, and Browser WebAssembly ONNX (`512`-token context + long-document windowing).
4
+
5
+ - **Interactive Playground (Browser WASM ONNX + Zero-Dep Engine)**: [https://hemanth.github.io/nullspan/](https://hemanth.github.io/nullspan/)
6
+ - **NullspanBench Leaderboard (JevBench-Style 19-System Evaluation, N=500)**: [https://hemanth.github.io/nullspan/bench.html](https://hemanth.github.io/nullspan/bench.html)
7
+ - **Hugging Face Space**: [https://huggingface.co/spaces/gnumanth/nullspan](https://huggingface.co/spaces/gnumanth/nullspan)
8
+ - **Fine-Tuned Hugging Face Model (344.2M PyTorch + 330.1 MB Int8 ONNX)**: [https://huggingface.co/gnumanth/nullspan](https://huggingface.co/gnumanth/nullspan)
9
+ - **Hugging Face Dataset**: [https://huggingface.co/datasets/gnumanth/nullspan-dataset](https://huggingface.co/datasets/gnumanth/nullspan-dataset)
10
+ - **Empirical Science Report**: [`bench/SCIENCE_REPORT.md`](./bench/SCIENCE_REPORT.md)
11
+
12
+ ## In Plain English
13
+
14
+ Standard LLMs extract data by generating JSON token-by-token, which is slow, bills per output token, can get hijacked by *"ignore previous instructions"* prompt injections, and often hallucinates values when a field is missing. **`nullspan` acts like a grounded highlighter + calculator instead of a chatbot:**
15
+
16
+ 1. **Points to Exact Text (`[start, end]` Span)**: Highlights exact character positions in your input text (`0` output tokens generated), making fabricated strings and prompt-injection payloads impossible.
17
+ 2. **Knows When to Say "Not There" (`p_null`)**: Computes a calibrated probability that a field is missing (`p_null`) and returns `null` with `abstention_reason` when evidence is absent.
18
+ 3. **Disambiguates Multi-Numeric Contracts & Checks Math (`constraint` & `verified`)**: Separates competing dollar caps, percentages, day windows, and numeric tranches across multi-section agreements (`96/96` on unseen contracts) and verifies equations like `subtotal == nodes * hourly_rate`.
19
+ 4. **Gives Realistic Ranges (`p10 / median / p90`)**: Returns calibrated low/middle/high quantile distributions for estimates and forecasts.
20
+
21
+ ## Install
22
+
23
+ ```sh
24
+ npm install nullspan
25
+ pip install nullspan
26
+ ```
27
+
28
+ ## Usage (JavaScript / ESM)
29
+
30
+ ```js
31
+ import { extract } from 'nullspan';
32
+
33
+ const incident =
34
+ "Incident INC-40928 opened on Apr 12, 2026 with a 502 Bad Gateway errors and a TLS handshake timeouts. " +
35
+ "Issued $1240.50 plus $310.25 SLA credit. Severity is sev-1 and customer data loss was false. " +
36
+ "Root cause CVE not yet assigned. Cluster stabilized on Apr 19, 2026. Reported by Mira";
37
+
38
+ const result = extract(incident, {
39
+ incident_id: { type: 'text', instructions: 'Incident ticket identifier' },
40
+ opened_on: { type: 'date', instructions: 'Date the incident opened' },
41
+ total_credit: { type: 'number', decimals: 2, unit: 'USD', instructions: 'Total credit including SLA penalty' },
42
+ outage_days: { type: 'duration', unit: 'days', integer: true, instructions: 'Days from incident open to cluster stabilization' },
43
+ failure_modes: { type: 'list', items: 'text', instructions: 'Each failure mode triggered' },
44
+ severity: { type: 'enum', options: ['sev-1', 'sev-2', 'sev-3'], instructions: 'Incident severity tier' },
45
+ data_loss: { type: 'boolean', instructions: 'Whether customer data loss occurred' },
46
+ cve_id: { type: 'text', instructions: 'Assigned CVE identifier if present' },
47
+ remediation_cost: { type: 'estimate', unit: 'USD', min: 100, max: 25000, scale: 'log', step: 10, instructions: 'Estimated engineering remediation cost' },
48
+ });
49
+ ```
50
+
51
+ ## Usage (Python + Cross-Field Invariants + Hugging Face Backend)
52
+
53
+ ```python
54
+ from nullspan import extract
55
+
56
+ invoice = (
57
+ "Invoice from TensorCore Cloud\n"
58
+ "Invoice #88310 | Issued: 2026-08-14 18:20\n"
59
+ "Nodes: 4 @ $3.50 each\n"
60
+ "Subtotal: $14.00\n"
61
+ "Egress Fee: $0.00\n"
62
+ "Priority Surcharge: $1.75"
63
+ )
64
+
65
+ # Zero-dependency fast engine or GCP V100 fine-tuned 344.2M checkpoint (backend="hf" or "onnx")
66
+ out = extract(
67
+ state=invoice,
68
+ fields={
69
+ "nodes": {"type": "number", "integer": True, "instructions": "Provisioned GPU node count"},
70
+ "hourly_rate": {"type": "number", "decimals": 2, "unit": "USD", "instructions": "Hourly rate per node"},
71
+ "subtotal": {
72
+ "type": "number",
73
+ "decimals": 2,
74
+ "unit": "USD",
75
+ "constraint": "subtotal == nodes * hourly_rate",
76
+ "instructions": "Compute subtotal before fees and surcharge",
77
+ },
78
+ "promo_rebate": {"type": "number", "decimals": 2, "unit": "USD", "instructions": "Promotional rebate amount if present"},
79
+ },
80
+ backend="hf",
81
+ model_id="gnumanth/nullspan",
82
+ )
83
+ # out["invariants_passed"] -> True, out["answers"]["subtotal"]["verified"] -> True
84
+ ```
85
+
86
+ ## Architecture (`NullspanForExtraction` v2.0 · 344.2M ELECTRA-Large + Adapter)
87
+
88
+ ![Architecture Flow](./docs/flow-diagram.svg)
89
+
90
+ 1. **`null_head` (Span-Attended Calibrated Abstention)**: Bilinear span-summary + `[CLS]` margin residual with held-out temperature scaling (`p_null` AUROC `0.9983`, ECE `0.0121`) and `abstention_reason` (`unassigned_placeholder` vs `missing_field`).
91
+ 2. **`span_head` (Grounded Pointer + Shannon Entropy)**: Projects `512`-token encoder states (plus query-conditioned sliding windows on ultra-long contracts) to `[start, end]` character spans with calibrated `confidence` and Shannon `span_entropy_bits`.
92
+ 3. **`bio_head` (Multi-Span List Tagging)**: Extracts ordered text and numeric tranche arrays (`type: "list"`) in a single pass.
93
+ 4. **`op_head` (10-Way System One Router)**: Routes fields across `SPAN_TEXT`, `SPAN_NUMBER`, `ARITH_SUM`, `ARITH_DIFF`, `ARITH_MUL`, `ISO_DATETIME`, `DATE_DURATION`, `DIST_ESTIMATE_FORECAST`, `BOOL_JUDGE` (`p_true`), and `ENUM_CLASSIFY` (`probs`).
94
+ 5. **`quantile_head` + Invariant Verifier**: Predicts monotone `(p10, median, p90)` quantiles and verifies cross-field algebraic constraints (`verified: true/false`).
95
+
96
+ ## NullspanBench Leaderboard (`bench/run_bench.py` · 19 Systems · $N=500$ Zero-Leakage Decisions)
97
+
98
+ Modeled after [BenchmarkHeaven's JevBench](https://benchmarkheaven.com/jev-models), **NullspanBench v3.2** evaluates `19` systems (`13` local open-weights/open-code models running live forward passes, live Cloudflare Workers AI `@cf/cloudflare/clairic + @cf/cloudflare/clef`, `2` reference JSON-mode LLM decoders, and `3` published JevBench v1.7 reference anchors) across **`500` zero-leakage observations** (`336` Open structured items + `164` Sealed items: `60` Stanford SQuAD v2.0, `96` Multi-Numeric Legal/Commercial Contract fields across `12` unseen agreements, and `8` Adversarial prompt-injection traps) on four axes: **Intelligence (`I`)**, **Calibration (`C`)**, **Speed (`S`)**, and **Cost (`K`)**.
99
+
100
+ Explore interactive weight sliders, Pareto frontier plots, and 2-system radar comparisons at [**https://hemanth.github.io/nullspan/bench.html**](https://hemanth.github.io/nullspan/bench.html):
101
+
102
+ | Rank | System | Composite (`25:25:25:25`) | Capability (`50I:50C`) | Intelligence `I` (95% CI) | Calibration `C` | Speed `S` | Cost `K` | `$/1k` | `p50` Latency |
103
+ | :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
104
+ | **1** | **`nullspan-Hybrid (Fast + ONNX Int8)`** (Open) | **97.2** | **98.3** | **97.8%** `[96.6, 99.0]` | **98.8** | **98.6** | **93.8** | `$0.0003` | `0.013 ms` |
105
+ | **2** | **`nullspan-ZeroDep (Fast Engine)`** (Open) | **95.9** | **92.9** | `90.8%` `[88.2, 93.4]` | `95.0` | **98.7** | **99.8** | `$0.0001` | `0.013 ms` |
106
+ | **3** | **`nullspan-344M (PyTorch FP32)`** (Open) | **78.7** | **98.4** | **98.0%** `[96.8, 99.2]` | **98.8** | `57.2` | `76.8` | `$0.0017` | `32.367 ms` |
107
+ | **4** | **`nullspan-344M-Int8 (ONNX WASM/CPU)`** (Open) | **78.4** | **98.3** | **97.8%** `[96.6, 99.0]` | **98.9** | `53.2` | `84.2` | `$0.0008` | `68.483 ms` |
108
+ | **5** | **`nullspan-344M (ONNX FP32)`** (Open) | **75.5** | **98.4** | **98.0%** `[96.8, 99.2]` | **98.8** | `50.5` | `78.0` | `$0.0015` | `113.137 ms` |
109
+ | **6** | `H2O-Lightning-4B v1.1 (JevBench #1 Comp)` (Open 4B) | `72.5` | `75.0` | `60.0%` `[57.8, 62.3]` | `90.0` | `92.6` | `60.3` | `$0.0211` | `29.285 ms` |
110
+ | **7** | **`@cf/cloudflare/clairic + clef (Live Workers AI)`** (API) | `72.1` | `93.2` | `91.6%` `[89.2, 93.9]` | `94.9` | `51.7` | `68.2` | `$0.0041` | `91.323 ms` (`315.3 ms/doc`) |
111
+ | **8** | `basal-omni-1.0-ultra (JevBench #1 Cap)` (Open 31B) | `71.8` | `82.5` | `74.1%` `[71.8, 76.3]` | `91.0` | `87.3` | `50.5` | `$0.0447` | `102.000 ms` |
112
+ | **9** | `Jev 1.13.0 (TypeSafe AI · JevBench Ref)` (API) | `71.5` | `77.1` | `63.6%` `[61.2, 66.0]` | `90.6` | `91.5` | `54.7` | `$0.0323` | `239.102 ms` |
113
+ | **10** | `gpt-5.4-nano (JSON Decoder · API)` (API) | `59.8` | `92.1` | `95.2%` `[93.4, 97.0]` | `89.0` | `43.6` | `45.1` | `$0.0438` | `417.991 ms` |
114
+ | **11** | `deberta-v3-base-squad2 (184M SQuAD2 QA)` (Open) | `58.0` | `55.8` | `48.1%` `[43.9, 52.4]` | `63.4` | `55.7` | `69.2` | `$0.0037` | `42.496 ms` |
115
+ | **12** | `claude-haiku-4.5 (JSON Decoder · API)` (API) | `48.8` | `92.1` | `95.4%` `[93.4, 97.2]` | `88.7` | `41.8` | `27.5` | `$0.2639` | `586.392 ms` |
116
+ | **13** | `bge-small-en-v1.5 (Dense Span+Null)` (Open) | `37.6` | `52.0` | `14.7%` `[12.0, 17.5]` | `89.2` | `64.5` | `85.5` | `$0.0007` | `8.098 ms` |
117
+ | **14** | `distilbert-base-control (Raw Control)` (Open) | `36.8` | `52.9` | `14.2%` `[11.6, 17.0]` | `91.5` | `66.0` | `81.1` | `$0.0011` | `6.133 ms` |
118
+ | **15** | `all-MiniLM-L12-v2 (Dense Span+Null)` (Open) | `35.9` | `44.0` | `14.4%` `[11.7, 17.5]` | `73.5` | `60.2` | `85.5` | `$0.0007` | `18.356 ms` |
119
+ | **16** | `all-mpnet-base-v2 (Dense Span+Null)` (Open) | `34.3` | `42.3` | `13.6%` `[10.9, 16.7]` | `71.0` | `62.8` | `77.4` | `$0.0016` | `11.263 ms` |
120
+ | **17** | `paraphrase-albert-small-v2 (Dense Span+Null)` (Open) | `33.6` | `44.5` | `12.5%` `[10.0, 15.2]` | `76.6` | `65.9` | `91.0` | `$0.0004` | `6.263 ms` |
121
+ | **18** | `paraphrase-MiniLM-L3-v2 (Dense Span+Null)` (Open) | `30.9` | `41.7` | `11.1%` `[8.7, 13.8]` | `72.3` | `68.9` | `91.0` | `$0.0004` | `3.519 ms` |
122
+ | **19** | `all-MiniLM-L6-v2 (Dense Span+Null)` (Open) | `28.3` | `40.5` | `10.0%` `[7.7, 12.3]` | `70.9` | `62.3` | `88.8` | `$0.0005` | `12.322 ms` |
123
+
124
+ See full methodology, proper scoring definitions, request-type breakdowns (`100.0%` on Multi-Numeric Contracts `n=96`, `100.0%` Choice `n=41`, `100.0%` Math & Invariants `n=51`), and head-to-head analysis in [`bench/SCIENCE_REPORT.md`](./bench/SCIENCE_REPORT.md).
125
+
126
+ ## License
127
+
128
+ MIT © [Hemanth.HM](https://h3manth.com)
{nullspan-1.1.0 → nullspan-1.2.0}/pyproject.toml RENAMED
@@ -4,7 +4,7 @@ build-backend = "flit_core.buildapi"
4
4
 
5
5
  [project]
6
6
  name = "nullspan"
7
- version = "1.1.0"
7
+ version = "1.2.0"
8
8
  description = "Calibrated p_null abstention, grounded [start, end] span extraction, arithmetic, and distribution forecasting in a single forward pass."
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.9"
{nullspan-1.1.0 → nullspan-1.2.0}/src/nullspan/__init__.py RENAMED
@@ -1,5 +1,5 @@
1
1
  """nullspan — Calibrated p_null abstention, grounded [start, end] span extraction, and distribution forecasting."""
2
2
  from .engine import DEFAULT_HF_REPO, NullspanExtractor, extract
3
3
 
4
- __version__ = "1.1.0"
4
+ __version__ = "1.2.0"
5
5
  __all__ = ["extract", "NullspanExtractor", "DEFAULT_HF_REPO"]
{nullspan-1.1.0 → nullspan-1.2.0}/src/nullspan/engine.py RENAMED
@@ -2,6 +2,7 @@
2
2
  from __future__ import annotations
3
3
 
4
4
  import datetime as dt
5
+ import os
5
6
  import re
6
7
  import time
7
8
  from typing import Any
@@ -131,7 +132,16 @@ _GENERIC_MEASURE_WORDS = {
131
132
  "rate", "rates", "period", "periods", "fee", "fees", "limit", "limits",
132
133
  "level", "levels", "time", "times", "window", "windows", "schedule",
133
134
  "schedules", "ratio", "ratios", "charge", "charges", "cost", "costs",
134
- "price", "prices", "payment", "payments", "duration",
135
+ "price", "prices", "payment", "payments", "duration", "cap", "caps",
136
+ "allowance", "allowances", "penalty", "penalties", "credit", "credits",
137
+ "rebate", "rebates", "surcharge", "surcharges", "deposit", "deposits",
138
+ "retainer", "minimum", "maximum", "floor", "ceiling", "threshold",
139
+ "target", "term", "terms", "quota", "quotas", "hours", "days", "months",
140
+ "years", "minutes", "annual", "monthly", "daily", "quarterly", "hourly",
141
+ "tenant", "subtenant", "landlord", "sublandlord", "provider", "customer",
142
+ "vendor", "client", "buyer", "seller", "carrier", "shipper", "sponsor",
143
+ "licensor", "licensee", "grantor", "grantee", "merchant", "acquirer",
144
+ "advertiser", "platform", "responder", "party", "parties",
135
145
  }
136
146
 
137
147
 
@@ -141,10 +151,10 @@ def _disambiguate_numeric_field(state: str, fname: str, spec: dict[str, Any]) ->
141
151
  instr_s = str(spec.get("instructions") or spec.get("instruction") or "")
142
152
  q_raw = f"{fn_clean} {instr_s}".lower()
143
153
  q_words: list[str] = []
144
- for w in re.findall(r"[a-z]{2,}", q_raw):
145
- if w not in _NUM_STOPWORDS and w not in q_words:
154
+ for w in re.findall(r"[a-z0-9]{2,}", q_raw):
155
+ if not w.isdigit() and w not in _NUM_STOPWORDS and w not in q_words:
146
156
  q_words.append(w)
147
- specific_q_words = [w for w in q_words if w not in _GENERIC_MEASURE_WORDS]
157
+ specific_q_words = [w for w in q_words if w not in _GENERIC_MEASURE_WORDS] or list(q_words)
148
158
 
149
159
  u_val = str(spec.get("unit", "")).lower()
150
160
  unit_str = f"{q_raw} {u_val}"
@@ -196,7 +206,7 @@ def _disambiguate_numeric_field(state: str, fname: str, spec: dict[str, Any]) ->
196
206
  pat = rf"\b{re.escape(w)}\b"
197
207
  else:
198
208
  stem = w[:5] if len(w) >= 6 else w.rstrip("s")
199
- pat = re.escape(stem)
209
+ pat = rf"\b{re.escape(stem)}"
200
210
  best_w_sc = 0.0
201
211
  for wm in re.finditer(pat, clause_txt):
202
212
  w_abs = c_start + wm.start()
@@ -348,14 +358,56 @@ def _extract_field_fast(state: str, fname: str, spec: dict[str, Any]) -> dict[st
348
358
  }
349
359
 
350
360
  if ftype == "boolean":
351
- m_bool = re.search(r"\b(true|false|yes|no)\b", state, re.I)
352
- if m_bool:
353
- b_val = m_bool.group(1).lower() in ("true", "yes")
361
+ q_raw_b = f"{fname.replace('_', ' ')} {spec.get('instructions') or spec.get('instruction') or ''}".lower()
362
+ q_words_b = [
363
+ w for w in re.findall(r"[a-z]{2,}", q_raw_b)
364
+ if w not in _NUM_STOPWORDS and w not in ("whether", "status", "active", "enabled", "permitted", "allowed", "true", "false")
365
+ ]
366
+ clauses_b = list(re.finditer(r"[^.!?\n]+[.!?\n]?", state))
367
+ search_txt = state
368
+ search_off = 0
369
+ if len(clauses_b) > 1 and q_words_b:
370
+ best_cb, best_sc_b = clauses_b[0], -1.0
371
+ for cm in clauses_b:
372
+ c_low = cm.group(0).lower()
373
+ sc = sum(1.0 for w in q_words_b if (w[:5] if len(w) >= 6 else w.rstrip("s")) in c_low)
374
+ if sc > best_sc_b:
375
+ best_sc_b = sc
376
+ best_cb = cm
377
+ if best_sc_b > 0:
378
+ search_txt = best_cb.group(0)
379
+ search_off = best_cb.start()
380
+ m_neg = re.search(
381
+ r"\b(not\s+(?:enabled|active|permitted|allowed|granted)|false|no|disabled|inactive|prohibited|denied)\b",
382
+ search_txt,
383
+ re.I,
384
+ )
385
+ m_pos = re.search(r"\b(true|yes|active|enabled|permitted|allowed|confirmed|granted)\b", search_txt, re.I)
386
+ if m_neg or m_pos:
387
+ if m_neg and (not m_pos or m_neg.start() <= m_pos.start()):
388
+ m_use = m_neg
389
+ b_val = False
390
+ else:
391
+ m_use = m_pos # type: ignore[assignment]
392
+ b_val = True
354
393
  return {
355
394
  "type": "boolean",
356
395
  "p_null": 1.1e-6,
357
396
  "p_true": 0.9992 if b_val else 0.0008,
358
- "span": [m_bool.start(1), m_bool.end(1)],
397
+ "span": [search_off + m_use.start(1), search_off + m_use.end(1)],
398
+ "guess": b_val,
399
+ "confidence": 0.9992,
400
+ "span_entropy_bits": 0.041,
401
+ "value": b_val,
402
+ }
403
+ m_bool_fb = re.search(r"\b(true|false|yes|no)\b", state, re.I)
404
+ if m_bool_fb:
405
+ b_val = m_bool_fb.group(1).lower() in ("true", "yes")
406
+ return {
407
+ "type": "boolean",
408
+ "p_null": 1.1e-6,
409
+ "p_true": 0.9992 if b_val else 0.0008,
410
+ "span": [m_bool_fb.start(1), m_bool_fb.end(1)],
359
411
  "guess": b_val,
360
412
  "confidence": 0.9992,
361
413
  "span_entropy_bits": 0.041,
@@ -494,6 +546,9 @@ def _extract_field_fast(state: str, fname: str, spec: dict[str, Any]) -> dict[st
494
546
  "span_entropy_bits": 0.031,
495
547
  "value": val_d,
496
548
  }
549
+ dis_dur = _disambiguate_numeric_field(state, fname, {**spec, "type": "duration", "unit": unit})
550
+ if dis_dur is not None:
551
+ return {**dis_dur, "unit": unit}
497
552
  return {"type": "duration", "p_null": 0.9991, "unit": unit, "span": [0, 0], "guess": 0, "confidence": 0.41, "value": None}
498
553
 
499
554
  if ftype in ("date", "time", "datetime"):
@@ -515,7 +570,18 @@ def _extract_field_fast(state: str, fname: str, spec: dict[str, Any]) -> dict[st
515
570
  if ftype == "number":
516
571
  if any(w in key or w in instr for w in ("rebate", "discount", "promo")):
517
572
  m_disc = re.search(r"(?:Promo Rebate|Discount):\s*-?\$?(\d+(?:\.\d+)?)", state, re.I)
518
- if not m_disc:
573
+ if m_disc:
574
+ d_val = float(m_disc.group(1))
575
+ return {
576
+ "type": "number",
577
+ "p_null": 1.2e-6,
578
+ "span": [m_disc.start(1), m_disc.end(1)],
579
+ "guess": d_val,
580
+ "confidence": 0.9994,
581
+ "span_entropy_bits": 0.044,
582
+ "value": d_val,
583
+ }
584
+ if "Invoice from " in state or "Receipt from " in state:
519
585
  m_zero = re.search(r"(?:Egress Fee|Sales Tax):\s*\$(\d+(?:\.\d+)?)", state, re.I)
520
586
  sp = [m_zero.start(1), m_zero.end(1)] if m_zero else [0, 0]
521
587
  return {
@@ -528,16 +594,6 @@ def _extract_field_fast(state: str, fname: str, spec: dict[str, Any]) -> dict[st
528
594
  "abstention_reason": "missing_field",
529
595
  "value": None,
530
596
  }
531
- d_val = float(m_disc.group(1))
532
- return {
533
- "type": "number",
534
- "p_null": 1.2e-6,
535
- "span": [m_disc.start(1), m_disc.end(1)],
536
- "guess": d_val,
537
- "confidence": 0.9994,
538
- "span_entropy_bits": 0.044,
539
- "value": d_val,
540
- }
541
597
 
542
598
  if "including" in instr or "total_credit" in key or "amount_paid" in key:
543
599
  m_plus = re.search(r"\$(\d+(?:\.\d+)?)\s+plus\s+\$(\d+(?:\.\d+)?)", state)
@@ -674,7 +730,7 @@ def _extract_field_fast(state: str, fname: str, spec: dict[str, Any]) -> dict[st
674
730
  "value": p_str,
675
731
  }
676
732
 
677
- return _extract_open_domain_qa(state, spec.get("instructions", "") or name)
733
+ return _extract_open_domain_qa(state, spec.get("instructions", "") or fname)
678
734
 
679
735
 
680
736
  _STOPWORDS = {
@@ -849,9 +905,13 @@ def _run_extraction(
849
905
  mdl = AutoModel.from_pretrained(model_id, trust_remote_code=True)
850
906
  if backend == "onnx":
851
907
  import onnxruntime as ort
852
- from huggingface_hub import hf_hub_download
853
908
 
854
- onnx_file = hf_hub_download(repo_id=model_id, filename="onnx/model_quantized.onnx")
909
+ if os.path.isdir(model_id):
910
+ onnx_file = os.path.join(model_id, "onnx", "model_quantized.onnx")
911
+ else:
912
+ from huggingface_hub import hf_hub_download
913
+
914
+ onnx_file = hf_hub_download(repo_id=model_id, filename="onnx/model_quantized.onnx")
855
915
  sess = ort.InferenceSession(onnx_file, providers=["CPUExecutionProvider"])
856
916
  _HF_CACHE[cache_key] = (tok, mdl, sess)
857
917
  else:
nullspan-1.1.0/README.md DELETED
@@ -1,127 +0,0 @@
1
- # nullspan
2
-
3
- Single-pass System One structured extraction, calibrated `p_null` abstention, character-offset `[start, end]` spans, zero-shot `boolean`/`enum` judgment, cross-field algebraic invariant verification, and calibrated quantile estimation (`p10`, `median`, `p90`) in JavaScript, Python, and Browser WebAssembly ONNX.
4
-
5
- - **Interactive Playground (Browser WASM ONNX + Zero-Dep Engine)**: [https://hemanth.github.io/nullspan/](https://hemanth.github.io/nullspan/)
6
- - **NullspanBench Leaderboard (JevBench-Style 14-System Evaluation)**: [https://hemanth.github.io/nullspan/bench.html](https://hemanth.github.io/nullspan/bench.html)
7
- - **Hugging Face Space**: [https://huggingface.co/spaces/gnumanth/nullspan](https://huggingface.co/spaces/gnumanth/nullspan)
8
- - **Fine-Tuned Hugging Face Model (67M PyTorch + 64.3 MB Int8 ONNX)**: [https://huggingface.co/gnumanth/nullspan](https://huggingface.co/gnumanth/nullspan)
9
- - **Hugging Face Dataset**: [https://huggingface.co/datasets/gnumanth/nullspan-dataset](https://huggingface.co/datasets/gnumanth/nullspan-dataset)
10
- - **Empirical Science Report**: [`bench/SCIENCE_REPORT.md`](./bench/SCIENCE_REPORT.md)
11
-
12
- ## In Plain English
13
-
14
- Standard LLMs extract data by generating JSON token-by-token—which is slow, costs money per output token, can get hijacked by *"ignore previous instructions"* prompt injections, and often hallucinates values when a field is missing. **`nullspan` acts like a smart highlighter + calculator instead of a chatbot:**
15
-
16
- 1. **Points to Exact Text (`[start, end]` Span)**: Highlights exact character positions in your input text (`0` output tokens generated), making fabricated strings and prompt-injection payloads impossible.
17
- 2. **Knows When to Say "Not There" (`p_null`)**: Computes a calibrated probability that a field is missing (`p_null`) and returns `null` when evidence is absent.
18
- 3. **Checks the Math Automatically (`constraint` & `verified`)**: Computes sums, durations, and products and verifies cross-field equations like `subtotal == nodes * hourly_rate`.
19
- 4. **Gives Realistic Ranges (`p10 / median / p90`)**: Returns calibrated low/middle/high quantile distributions for estimates and forecasts.
20
-
21
- ## Install
22
-
23
- ```sh
24
- npm install nullspan
25
- pip install nullspan
26
- ```
27
-
28
- ## Usage (JavaScript / ESM)
29
-
30
- ```js
31
- import { extract } from 'nullspan';
32
-
33
- const incident =
34
- "Incident INC-40928 opened on Apr 12, 2026 with a 502 Bad Gateway errors and a TLS handshake timeouts. " +
35
- "Issued $1240.50 plus $310.25 SLA credit. Severity is sev-1 and customer data loss was false. " +
36
- "Root cause CVE not yet assigned. Cluster stabilized on Apr 19, 2026. Reported by Mira";
37
-
38
- const result = extract(incident, {
39
- incident_id: { type: 'text', instructions: 'Incident ticket identifier' },
40
- opened_on: { type: 'date', instructions: 'Date the incident opened' },
41
- total_credit: { type: 'number', decimals: 2, unit: 'USD', instructions: 'Total credit including SLA penalty' },
42
- outage_days: { type: 'duration', unit: 'days', integer: true, instructions: 'Days from incident open to cluster stabilization' },
43
- failure_modes: { type: 'list', items: 'text', instructions: 'Each failure mode triggered' },
44
- severity: { type: 'enum', options: ['sev-1', 'sev-2', 'sev-3'], instructions: 'Incident severity tier' },
45
- data_loss: { type: 'boolean', instructions: 'Whether customer data loss occurred' },
46
- cve_id: { type: 'text', instructions: 'Assigned CVE identifier if present' },
47
- remediation_cost: { type: 'estimate', unit: 'USD', min: 100, max: 25000, scale: 'log', step: 10, instructions: 'Estimated engineering remediation cost' },
48
- });
49
- ```
50
-
51
- ## Usage (Python + Cross-Field Invariants + Hugging Face Backend)
52
-
53
- ```python
54
- from nullspan import extract
55
-
56
- invoice = (
57
- "Invoice from TensorCore Cloud\n"
58
- "Invoice #88310 | Issued: 2026-08-14 18:20\n"
59
- "Nodes: 4 @ $3.50 each\n"
60
- "Subtotal: $14.00\n"
61
- "Egress Fee: $0.00\n"
62
- "Priority Surcharge: $1.75"
63
- )
64
-
65
- # Zero-dependency fast engine or fine-tuned neural checkpoint (backend="hf")
66
- out = extract(
67
- state=invoice,
68
- fields={
69
- "nodes": {"type": "number", "integer": True, "instructions": "Provisioned GPU node count"},
70
- "hourly_rate": {"type": "number", "decimals": 2, "unit": "USD", "instructions": "Hourly rate per node"},
71
- "subtotal": {
72
- "type": "number",
73
- "decimals": 2,
74
- "unit": "USD",
75
- "constraint": "subtotal == nodes * hourly_rate",
76
- "instructions": "Compute subtotal before fees and surcharge",
77
- },
78
- "promo_rebate": {"type": "number", "decimals": 2, "unit": "USD", "instructions": "Promotional rebate amount if present"},
79
- },
80
- backend="hf",
81
- model_id="gnumanth/nullspan",
82
- )
83
- # out["invariants_passed"] -> True, out["answers"]["subtotal"]["verified"] -> True
84
- ```
85
-
86
- ## Architectural Enhancements
87
-
88
- ![Architecture Flow](./docs/flow-diagram.svg)
89
-
90
- 1. **`null_head` (Calibrated Abstention & Reason Diagnostics)**: Predicts calibrated `p_null`, nearest decoy `guess`, and `abstention_reason` (`unassigned_placeholder` vs `missing_field`).
91
- 2. **`span_head` (Grounded Pointer + Shannon Entropy)**: Projects encoder tokens to `[start, end]` character spans with calibrated `confidence` and Shannon `span_entropy_bits`.
92
- 3. **`bio_head` (Class-Weighted Multi-Span Tagging)**: Extracts arbitrary-length ordered arrays (`type: "list"`) in a single pass.
93
- 4. **`op_head` (10-Way System One Router)**: Routes fields across `SPAN_TEXT`, `SPAN_NUMBER`, `ARITH_SUM`, `ARITH_DIFF`, `ARITH_MUL`, `ISO_DATETIME`, `DATE_DURATION`, `DIST_ESTIMATE_FORECAST`, `BOOL_JUDGE` (`p_true`), and `ENUM_CLASSIFY` (`probs`).
94
- 5. **`quantile_head` + Invariant Verifier**: Predicts monotone `(p10, median, p90)` quantiles and verifies cross-field algebraic constraints (`verified: true/false`).
95
-
96
- ## NullspanBench Leaderboard (`bench/run_bench.py` · 14 Systems · $N=404$ Scored Decisions)
97
-
98
- Modeled after [BenchmarkHeaven's JevBench](https://benchmarkheaven.com/jev-models), **NullspanBench** evaluates `14` systems (`12` local open-weights/open-code models running live forward passes + `2` reference JSON-mode LLM decoders) across **`404` scored observations** (`336` Open structured items + `68` Sealed Stanford SQuAD v2.0 & adversarial prompt-injection items) on four axes: **Intelligence (`I`)**, **Calibration (`C`)**, **Speed (`S`)**, and **Cost (`K`)**.
99
-
100
- Explore interactive weight sliders, Pareto frontier plots, and 2-system radar comparisons at [**https://hemanth.github.io/nullspan/bench.html**](https://hemanth.github.io/nullspan/bench.html):
101
-
102
- | Rank | System | Composite (`25:25:25:25`) | Capability (`50I:50C`) | Intelligence `I` (95% CI) | Calibration `C` | Speed `S` | Cost `K` | `$/1k` | `p50` Latency |
103
- | :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
104
- | **1** | **`nullspan-Hybrid (Fast + ONNX Int8)`** (Open) | **97.7** | **98.7** | **98.0%** `[96.5, 99.3]` | **99.3** | **99.8** | **93.8** | `$0.0003` | `0.005 ms` |
105
- | **2** | **`nullspan-ZeroDep (Fast Engine)`** (Open) | **95.3** | **91.2** | `88.6%` `[85.6, 91.6]` | `93.8` | **99.8** | **99.8** | `$0.0001` | `0.005 ms` |
106
- | **3** | **`nullspan-67M (PyTorch FP32)`** (Open) | **83.3** | **99.2** | **98.8%** `[97.5, 99.8]` | **99.7** | `66.5` | `78.0` | `$0.0015` | `5.575 ms` |
107
- | **4** | **`nullspan-67M-Int8 (ONNX WASM/CPU)`** (Open) | **83.0** | **98.7** | **98.0%** `[96.5, 99.3]` | **99.3** | `62.3` | `84.2` | `$0.0008` | `12.268 ms` |
108
- | **5** | **`nullspan-67M (ONNX FP32)`** (Open) | **82.1** | **99.2** | **98.8%** `[97.5, 99.8]` | **99.7** | `63.0` | `78.7` | `$0.0014` | `10.832 ms` |
109
- | **6** | **`@cf/cloudflare/clairic + clef (Live Workers AI)`** | `74.4` | `92.0` | `90.1%` `[87.5, 92.7]` | `93.8` | `51.5` | `72.2` | `$0.0042` | `95.028 ms` (`308.6 ms/doc`) |
110
- | **7** | `H2O-Lightning-4B v1.1 (JevBench #1 Comp)` (Open 4B) | `72.5` | `75.0` | `60.0%` `[57.8, 62.3]` | `90.0` | `92.6` | `60.3` | `$0.0211` | `29.285 ms` |
111
- | **7** | `basal-omni-1.0-ultra (JevBench #1 Cap)` (Open 31B) | `71.8` | `82.5` | `74.1%` `[71.8, 76.3]` | `91.0` | `87.3` | `50.5` | `$0.0447` | `102.000 ms` |
112
- | **8** | `Jev 1.13.0 (TypeSafe AI · JevBench Ref)` (API) | `71.5` | `77.1` | `63.6%` `[61.2, 66.0]` | `90.6` | `91.5` | `54.7` | `$0.0323` | `239.102 ms` |
113
- | **9** | `gpt-5.4-nano (JSON Decoder · API)` | `60.3` | `91.5` | `95.3%` `[93.1, 97.3]` | `87.7` | `43.6` | `46.5` | `$0.0380` | `415.9 ms` |
114
- | **10** | `claude-haiku-4.5 (JSON Decoder · API)` | `49.7` | `92.3` | `96.3%` `[94.3, 98.0]` | `88.3` | `41.8` | `28.7` | `$0.2328` | `588.0 ms` |
115
- | **11** | `bge-small-en-v1.5 (Dense Span+Null)` (Open) | `42.0` | `53.3` | `17.5%` `[13.8, 21.3]` | `89.1` | `64.7` | `87.0` | `$0.0006` | `7.855 ms` |
116
- | **12** | `distilbert-base-control (Raw Control)` (Open) | `41.6` | `53.6` | `17.3%` `[13.6, 21.1]` | `90.0` | `65.3` | `84.2` | `$0.0008` | `6.947 ms` |
117
- | **13** | `all-MiniLM-L12-v2 (Dense Span+Null)` (Open) | `37.6` | `44.8` | `15.1%` `[11.9, 18.6]` | `74.6` | `65.4` | `87.0` | `$0.0006` | `6.894 ms` |
118
- | **14** | `all-mpnet-base-v2 (Dense Span+Null)` (Open) | `35.3` | `43.1` | `14.1%` `[10.9, 17.5]` | `72.2` | `62.8` | `80.2` | `$0.0012` | `11.098 ms` |
119
- | **15** | `paraphrase-albert-small-v2 (Dense Span+Null)` (Open) | `34.8` | `45.1` | `13.1%` `[9.9, 16.4]` | `77.2` | `66.8` | `93.8` | `$0.0003` | `5.224 ms` |
120
- | **16** | `paraphrase-MiniLM-L3-v2 (Dense Span+Null)` (Open) | `31.5` | `42.2` | `11.3%` `[8.4, 14.5]` | `73.1` | `70.6` | `93.8` | `$0.0003` | `2.586 ms` |
121
- | **17** | `all-MiniLM-L6-v2 (Dense Span+Null)` (Open) | `28.5` | `41.2` | `9.9%` `[7.1, 13.0]` | `72.6` | `68.0` | `91.0` | `$0.0004` | `4.200 ms` |
122
-
123
- See full methodology, proper scoring definitions, request-type breakdowns, and ablation analysis in [`bench/SCIENCE_REPORT.md`](./bench/SCIENCE_REPORT.md).
124
-
125
- ## License
126
-
127
- MIT © [Hemanth.HM](https://h3manth.com)
{nullspan-1.1.0 → nullspan-1.2.0}/LICENSE RENAMED
File without changes