crossfoot 0.1.0__tar.gz

crossfoot-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ram Vikram Singh
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

crossfoot-0.1.0/PKG-INFO

@@ -0,0 +1,244 @@
+Metadata-Version: 2.4
+Name: crossfoot
+Version: 0.1.0
+Summary: Fail-closed numeric grounding checks for LLM output: verifies stated and derived numbers, direction words and citations.
+Author: Ram Vikram Singh
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/ramvikrams/crossfoot
+Project-URL: Source, https://github.com/ramvikrams/crossfoot
+Project-URL: Issues, https://github.com/ramvikrams/crossfoot/issues
+Project-URL: Benchmark history, https://github.com/ramvikrams/crossfoot/blob/main/benchmarks/results/HISTORY.md
+Keywords: llm,evaluation,hallucination,grounding,rag,guardrails
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: hypothesis>=6; extra == "dev"
+Provides-Extra: deepeval
+Requires-Dist: deepeval; extra == "deepeval"
+Provides-Extra: bench
+Requires-Dist: pyarrow>=12; extra == "bench"
+Dynamic: license-file
+
+# crossfoot
+
+**Fail-closed numeric grounding checks for LLM output.**
+
+*To crossfoot*, in accounting, is to check that the figures in a ledger agree when they are added up both down the columns and across the rows. crossfoot does the same for an LLM's answer: every number must agree with the source, and every calculation must add up.
+
+LLMs that summarise reports, filings or dashboards often get the numbers almost right: a growth rate with the wrong sign, a figure taken from the wrong document, or more precision than the source supports. crossfoot checks every number in a model's answer against the sources it was given. The check is deterministic, needs no model calls, and has no dependencies.
+
+Output from `crossfoot eval examples/cases.jsonl -v` (excerpt):
+
+```text
+ok   derived-growth               expected=pass got=pass
+       [derived] 12% - % change revenue_2023 (2,870,000,000) -> revenue_2024 (3,214,000,000)
+       [supported_rounded] $3.21bn - [R1] revenue_2024 = 3,214,000,000
+ok   wrong-direction              expected=fail got=fail
+       [contradicted] 9.5% - magnitude matches % change net_income_2023 (455,000,000) -> net_income_2024 (412,000,000), but the wording states the opposite direction
+       [supported_exact] $412m - [R1] net_income_2024 = 412,000,000
+ok   wrong-record                 expected=fail got=fail
+       [wrong_record] 1,850 - value is in [R2] employees_2024, but the sentence cites [R1]
+ok   false-precision              expected=fail got=fail
+       [unsupported] $3,214 million - no source value or derivation matches at the stated precision
+
+15/15 cases as expected (accuracy 100.00%, threshold 100.00%)
+```
+
+## What it checks
+
+| Status | Meaning | Passes? |
+|---|---|---|
+| `supported_exact` | Number equals a source value | yes |
+| `supported_rounded` | A rounding of a source value at the precision the output states (`$3.2bn` for 3,214,000,000) | yes |
+| `derived` | Equals a % change, share, difference, sum or percentage-point change of source values, and the direction words ("rose", "fell") agree | yes |
+| `contradicted` | Magnitude matches a derivation, but the wording says the opposite direction | **no** |
+| `wrong_record` | The value exists, but in a record the sentence does not cite | **no** |
+| `wrong_row` | The value is in a table, but in a different row from the one the sentence names. For example, "net sales were $473m" when 473 is in the operating profit row | **no** |
+| `arithmetic_error` | The output writes out a calculation whose stated result is wrong at its stated precision. For example, "$7630.0m − $6710.4m = $918.2m", when the answer is $919.6m | **no** |
+| `unsupported` | Nothing matches at the stated precision, including false precision (`$3,214 million` when the source only says `$3.2bn`) | **no** |
+| `uncited` | Grounded, but `require_citations=True` and the sentence cites nothing | **no** |
+| `unknown_citation` | The sentence cites only records that do not exist, such as `[R9]`. crossfoot does not fall back to searching other sources. | **no** |
+
+Every unknown citation is also reported as an issue, which fails the report.
+
+**Fail-closed by design:** if crossfoot cannot prove a number, the number fails. Expect some false alarms. That trade-off fits the places where numbers matter: finance, compliance, analytics, and regulated reporting.
+
+## Install
+
+```bash
+pip install crossfoot                 # from PyPI
+pip install "crossfoot[deepeval]"     # with the DeepEval metric
+pip install -e ".[dev]"               # from a clone, for development
+```
+
+## Python API
+
+```python
+from crossfoot import check
+
+sources = [
+    {"id": "R1", "fields": {"revenue_2023": 2_870_000_000, "revenue_2024": 3_214_000_000}},
+    "In FY2024 Acme shipped 12,400 units, up from 10,000 in FY2023.",   # becomes S2
+]
+report = check("Revenue grew 12% to $3.2bn [R1]. Shipments rose 24% [S2].", sources)
+
+report.passed      # True
+report.score       # 1.0
+print(report.summary())
+```
+
+Sources can be plain strings (ids `S1`, `S2`, ... by position, citable as `[1]` or `[S1]`) or records `{"id", "text", "fields"}`. Numeric fields are exact. Field names ending in `_pct` or `_percent` are percentages.
+
+Options: `require_citations`, `allow_rounding` (set it to `False` for strict mode), `allow_derived`.
+
+To check many outputs against the same sources, build the sources once:
+
+```python
+from crossfoot import Checker
+checker = Checker(sources, require_citations=True)
+for answer in answers:
+    print(checker.check(answer).passed)
+```
+
+## CLI and CI gate
+
+```bash
+crossfoot check --output answer.txt --sources sources.json [--require-citations] [--strict] [--json]
+crossfoot eval examples/cases.jsonl --min-accuracy 1.0
+```
+
+`eval` runs a labelled JSONL case set: one line per case with `output`, `sources` or `sources_file`, an optional `expect` (`pass`/`fail`, default `pass`) and `options`. The command exits non-zero below the threshold, so you can use it to gate pull requests that change prompts, models or retrieval.
+
+### As a GitHub Action
+
+```yaml
+- uses: ramvikrams/crossfoot@v0.1.0
+  with:
+    cases: evals/numeric_cases.jsonl
+```
+
+## Integrations
+
+**DeepEval:**
+
+```python
+from crossfoot.integrations.deepeval_metric import NumericGroundingMetric
+assert_test(LLMTestCase(input=q, actual_output=answer, retrieval_context=chunks),
+            [NumericGroundingMetric(require_citations=True)])
+```
+
+**promptfoo** (Python assertion):
+
+```yaml
+assert:
+  - type: python
+    value: file://path/to/crossfoot/integrations/promptfoo_assert.py
+```
+
+The assertion reads `vars.sources` and the optional `vars.crossfoot_options`.
+
+## How it decides
+
+1. Split the output into sentences and resolve citations such as `[R1]`, `[R1, R2]` or `[1]`.
+2. Extract each number with its kind (money, number, percent, pp, bps, year), currency, scale (k / m / bn / lakh / crore) and **stated precision**.
+3. Look for a direct match in the cited records, or in all records if the sentence cites nothing. A match must be exact, or within half a unit of the output's last stated digit.
+4. Otherwise, look for a derivation. Cited sentences can combine values from any of their cited records. Uncited sentences can only combine values within a single record, which limits chance matches. Two more limits apply:
+   - Derived amounts must be stated to about 1% precision.
+   - Growth wording ("grew", "up") can only be matched by a change, never by a share or a sum.
+5. If a cited sentence still has no match, check the other records and report `wrong_record` when the value is found there.
+6. **Step-by-step arithmetic:** a number that is still unsupported can be derived from numbers the output has *already grounded*, as long as they appear in the same sentence or the two before it. For example, "$1.11 − $1.08" followed by "= $0.03". Wrong arithmetic is still caught. "A decrease of $6 million ($5829m − $5735m)" fails, because the real difference is $94m.
+
+7. **Written calculations:** arithmetic the output shows, such as `(7167.1 − 6383.0) / 6383.0 × 100 = 12.28%` or `(76 + 73 + 52) / 3 = 67`, is parsed with a small safe parser (never `eval`) and recomputed. If the result is right and every operand is grounded, the result passes. If the stated result disagrees at its stated precision, it is an `arithmetic_error`. Constants such as 100, or the 3 in `/ 3`, need no grounding.
+8. **Table rows:** a value found in a table cell is checked against the row label. If the sentence names a different row in full and none of the matching cells sit in a row the sentence mentions, the claim is `wrong_row`. Set `check_table_rows=False` to switch this off.
+
+Some cases get special handling:
+- **Restatements:** a later sentence may repeat a number the output already grounded, for example "= 67" followed by "the average is $67 million".
+- **Dropped units:** a number written without its unit inside a calculation ("2.2" for "$2.2 billion") matches directly, but is never used as a derivation operand.
+- **Tables:** JSON-style tables in a source, such as `[["", "2019", "2018"], ["sales", "5", "4"]]`, are read cell by cell. Each value is labelled with its row and column, and gets a year when its column header or row label has one.
+- **Direction needs a known order:** a difference such as "decreased by $29,253" has the same size in both orders. crossfoot therefore takes the order from table years (earlier → later), or from how the answer writes the calculation: "$5,836,000 − $4,657,000", "from A to B" or "subtracting A from B". If the wording contradicts that order, the claim is `contradicted`. With no known order, only the magnitude is checked.
+- **Averages:** the average of two values is accepted, for example "($81 + $120) / 2 = $100.5 million".
+- **Table units:** in a source whose text contains a unit header such as "(in millions)", "( millions )" or "in thousands", unscaled numbers also match their scaled value. For example, "7170" under "(in millions)" supports "$7170 million", and the evidence names the header.
+- **Fractions:** a plain number such as `0.1233` can match a percentage change or share written as a fraction.
+- **Not claims:** these are skipped:
+  - formula constants: the `100` in `* 100` or "multiplying by 100", and small divisors such as the `2` in `/ 2` or "divide by 2"
+  - the day in dates (`December 31`) and numeric dates (`11/30/04`, `2019-06-30`)
+  - version and IP strings (`8.4.1.1`)
+
+The derivation search runs backwards from the stated number. For each operand, crossfoot computes the partner value that would produce the stated figure and finds it by binary search. A record with n values therefore costs O(n log n) per claim, with no cap on the number of operands.
+
+## Benchmark
+
+[benchmarks/ragbench.py](https://github.com/ramvikrams/crossfoot/blob/main/benchmarks/ragbench.py) measures crossfoot against the sentence-level support labels in [RAGBench](https://huggingface.co/datasets/rungalileo/ragbench) (CC BY 4.0). It uses the numeric-heavy financial QA subsets `finqa` and `tatqa` by default.
+
+```bash
+pip install -e ".[bench]"
+python benchmarks/ragbench.py --subsets finqa tatqa --limit 300
+```
+
+Latest results (run 7: full `test` splits, about 20,000 numeric sentences; full log in [HISTORY.md](https://github.com/ramvikrams/crossfoot/blob/main/benchmarks/results/HISTORY.md)):
+
+| | finqa (2,294 rows) | tatqa (3,338 rows) |
+|---|---:|---:|
+| Injected value errors caught (±10–30%) | 89.7% (5841/6509) | 92.6% (5303/5726) |
+| Injected scale errors caught (million ↔ billion) | 96.3% (3120/3240) | 94.3% (1662/1762) |
+| Injected direction errors caught (increase ↔ decrease) | 69.9% (214/306) | 70.0% (243/347) |
+| Swapped values caught (another number from the same source) | 13.4% (815/6097) | 9.8% (539/5525) |
+| False alarms on sentences labelled supported | 20.1% | 19.4% |
+
+The first 300 rows of four non-financial subsets give these false-alarm rates: covidqa 2.5%, techqa 16.8%, emanual 26.7% (only 32 numeric sentences) and hagrid 19.0%. On the same subsets, injected value errors are caught 58–91% of the time.
+
+The detection rates come from a controlled test. It starts from sentences that are labelled supported and that crossfoot passes, and injects one known error into each. The false-alarm rate is high because crossfoot is fail-closed by design: anything it cannot prove fails, including correct multi-step arithmetic that it does not yet model. Some of these "false alarms" are also genuine numeric errors that the LLM annotator accepted.
+
+**Hand-checking.** Run `--review-out benchmarks/results/review-sample.csv` to export a seeded random sample of flagged sentences. Each row includes the documents. Fill in the `verdict` column with `real error`, `false alarm` or `unsure`, then run `python benchmarks/review_summary.py benchmarks/results/review-sample.csv` to get a hand-checked precision that does not depend on LLM labels.
+
+The script writes results to [benchmarks/results/](https://github.com/ramvikrams/crossfoot/tree/main/benchmarks/results) and saves example errors under `.cache/ragbench/` for manual review. Keep two caveats in mind when reading the results:
+- RAGBench labels come from an LLM annotator, not from human review.
+- Sentences can be unsupported for non-numeric reasons, so the recall figure is a lower bound.
+
+## Known limitations
+
+- **Coincidental derivations.** With many values per record, a wrong figure can match some unrelated derivation by chance. Above `max_derivation_operands` values per scope (default 30), crossfoot stops deriving from the sources alone. The output must then state the operands it used, which step-by-step arithmetic accepts, or cite a narrower record. Below the limit, coincidences are still possible, so the evidence string shows the formula for a reviewer to audit.
+- **Two-operand derivations only.** No sums of three or more values, averages of more than two values, or CAGR yet.
+- **Ratios** written as "a:b" are not understood.
+- **Direction words** are read from a 60-character window before the number, in English only. Negations ("did not rise") are not understood. When the order of two values is unknown (no table years and no written order), changes below about 8% cannot be told apart by direction at whole-number precision. For example, 100 → 105 is +5%, while 105 → 100 is −4.8%, which rounds to 5. The property-based tests document this.
+- **Row matching** uses word overlap with row labels. It only fires when the sentence names another row in full, so a paraphrased row name is not caught.
+- **Swapped values.** A number replaced with another value from the same source is caught only 9-13% of the time on FinQA and TAT-QA. crossfoot checks values, not which row or column a value belongs to.
+- Small integers ("2 threads") are weakly verifiable in long documents.
+- Numbers written as words ("twelve"), fractions, ranges as a single claim ("10-12%"), dates other than four-digit years, and negative numbers in parentheses ("(1,234)") are not parsed.
+- Any bracketed token is treated as a citation, so `[sic]` will be reported as an unknown citation.
+- `$` is treated as compatible with any dollar currency (USD, HKD). Explicit codes are compared strictly.
+- The DeepEval wrapper targets the `BaseMetric` interface. Confirm it against your DeepEval version.
+
+## Production-gap backlog
+
+Not done yet, and needed before relying on crossfoot in a production pipeline:
+
+- [x] Benchmark harness for a public faithfulness dataset (RAGBench)
+- [ ] Publish full-split benchmark results, plus a hand-reviewed sample of errors
+- [ ] Multi-operand derivations (totals, averages, CAGR) with a bounded search
+- [ ] Unit-aware quantities beyond currency (%, units, km, tonnes) and date parsing
+- [ ] Negation handling and multilingual direction words
+- [ ] Configurable ignore rules (section numbers, ordinals)
+- [ ] Packaged releases on PyPI and the GitHub Marketplace, plus signed tags
+- [x] Sub-quadratic derivation search (no operand cap)
+
+## Related work
+
+- [hardfacts](https://github.com/seekerPrice/hardfacts) checks stated numbers, dates and IDs against sources. According to its README, it treats doubtful matches as supported and flags derived figures without confirming them. crossfoot takes the opposite stance: it is fail-closed, and it verifies derived figures and their direction.
+- [DeepEval](https://deepeval.com), [promptfoo](https://promptfoo.dev) and [Ragas](https://docs.ragas.io) are general LLM evaluation frameworks. crossfoot plugs into the first two as a deterministic metric.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest -q
+crossfoot eval examples/cases.jsonl -v
+```
+
+All example data is fictional.
+
+MIT licensed.

crossfoot-0.1.0/README.md

@@ -0,0 +1,219 @@
+# crossfoot
+
+**Fail-closed numeric grounding checks for LLM output.**
+
+*To crossfoot*, in accounting, is to check that the figures in a ledger agree when they are added up both down the columns and across the rows. crossfoot does the same for an LLM's answer: every number must agree with the source, and every calculation must add up.
+
+LLMs that summarise reports, filings or dashboards often get the numbers almost right: a growth rate with the wrong sign, a figure taken from the wrong document, or more precision than the source supports. crossfoot checks every number in a model's answer against the sources it was given. The check is deterministic, needs no model calls, and has no dependencies.
+
+Output from `crossfoot eval examples/cases.jsonl -v` (excerpt):
+
+```text
+ok   derived-growth               expected=pass got=pass
+       [derived] 12% - % change revenue_2023 (2,870,000,000) -> revenue_2024 (3,214,000,000)
+       [supported_rounded] $3.21bn - [R1] revenue_2024 = 3,214,000,000
+ok   wrong-direction              expected=fail got=fail
+       [contradicted] 9.5% - magnitude matches % change net_income_2023 (455,000,000) -> net_income_2024 (412,000,000), but the wording states the opposite direction
+       [supported_exact] $412m - [R1] net_income_2024 = 412,000,000
+ok   wrong-record                 expected=fail got=fail
+       [wrong_record] 1,850 - value is in [R2] employees_2024, but the sentence cites [R1]
+ok   false-precision              expected=fail got=fail
+       [unsupported] $3,214 million - no source value or derivation matches at the stated precision
+
+15/15 cases as expected (accuracy 100.00%, threshold 100.00%)
+```
+
+## What it checks
+
+| Status | Meaning | Passes? |
+|---|---|---|
+| `supported_exact` | Number equals a source value | yes |
+| `supported_rounded` | A rounding of a source value at the precision the output states (`$3.2bn` for 3,214,000,000) | yes |
+| `derived` | Equals a % change, share, difference, sum or percentage-point change of source values, and the direction words ("rose", "fell") agree | yes |
+| `contradicted` | Magnitude matches a derivation, but the wording says the opposite direction | **no** |
+| `wrong_record` | The value exists, but in a record the sentence does not cite | **no** |
+| `wrong_row` | The value is in a table, but in a different row from the one the sentence names. For example, "net sales were $473m" when 473 is in the operating profit row | **no** |
+| `arithmetic_error` | The output writes out a calculation whose stated result is wrong at its stated precision. For example, "$7630.0m − $6710.4m = $918.2m", when the answer is $919.6m | **no** |
+| `unsupported` | Nothing matches at the stated precision, including false precision (`$3,214 million` when the source only says `$3.2bn`) | **no** |
+| `uncited` | Grounded, but `require_citations=True` and the sentence cites nothing | **no** |
+| `unknown_citation` | The sentence cites only records that do not exist, such as `[R9]`. crossfoot does not fall back to searching other sources. | **no** |
+
+Every unknown citation is also reported as an issue, which fails the report.
+
+**Fail-closed by design:** if crossfoot cannot prove a number, the number fails. Expect some false alarms. That trade-off fits the places where numbers matter: finance, compliance, analytics, and regulated reporting.
+
+## Install
+
+```bash
+pip install crossfoot                 # from PyPI
+pip install "crossfoot[deepeval]"     # with the DeepEval metric
+pip install -e ".[dev]"               # from a clone, for development
+```
+
+## Python API
+
+```python
+from crossfoot import check
+
+sources = [
+    {"id": "R1", "fields": {"revenue_2023": 2_870_000_000, "revenue_2024": 3_214_000_000}},
+    "In FY2024 Acme shipped 12,400 units, up from 10,000 in FY2023.",   # becomes S2
+]
+report = check("Revenue grew 12% to $3.2bn [R1]. Shipments rose 24% [S2].", sources)
+
+report.passed      # True
+report.score       # 1.0
+print(report.summary())
+```
+
+Sources can be plain strings (ids `S1`, `S2`, ... by position, citable as `[1]` or `[S1]`) or records `{"id", "text", "fields"}`. Numeric fields are exact. Field names ending in `_pct` or `_percent` are percentages.
+
+Options: `require_citations`, `allow_rounding` (set it to `False` for strict mode), `allow_derived`.
+
+To check many outputs against the same sources, build the sources once:
+
+```python
+from crossfoot import Checker
+checker = Checker(sources, require_citations=True)
+for answer in answers:
+    print(checker.check(answer).passed)
+```
+
+## CLI and CI gate
+
+```bash
+crossfoot check --output answer.txt --sources sources.json [--require-citations] [--strict] [--json]
+crossfoot eval examples/cases.jsonl --min-accuracy 1.0
+```
+
+`eval` runs a labelled JSONL case set: one line per case with `output`, `sources` or `sources_file`, an optional `expect` (`pass`/`fail`, default `pass`) and `options`. The command exits non-zero below the threshold, so you can use it to gate pull requests that change prompts, models or retrieval.
+
+### As a GitHub Action
+
+```yaml
+- uses: ramvikrams/crossfoot@v0.1.0
+  with:
+    cases: evals/numeric_cases.jsonl
+```
+
+## Integrations
+
+**DeepEval:**
+
+```python
+from crossfoot.integrations.deepeval_metric import NumericGroundingMetric
+assert_test(LLMTestCase(input=q, actual_output=answer, retrieval_context=chunks),
+            [NumericGroundingMetric(require_citations=True)])
+```
+
+**promptfoo** (Python assertion):
+
+```yaml
+assert:
+  - type: python
+    value: file://path/to/crossfoot/integrations/promptfoo_assert.py
+```
+
+The assertion reads `vars.sources` and the optional `vars.crossfoot_options`.
+
+## How it decides
+
+1. Split the output into sentences and resolve citations such as `[R1]`, `[R1, R2]` or `[1]`.
+2. Extract each number with its kind (money, number, percent, pp, bps, year), currency, scale (k / m / bn / lakh / crore) and **stated precision**.
+3. Look for a direct match in the cited records, or in all records if the sentence cites nothing. A match must be exact, or within half a unit of the output's last stated digit.
+4. Otherwise, look for a derivation. Cited sentences can combine values from any of their cited records. Uncited sentences can only combine values within a single record, which limits chance matches. Two more limits apply:
+   - Derived amounts must be stated to about 1% precision.
+   - Growth wording ("grew", "up") can only be matched by a change, never by a share or a sum.
+5. If a cited sentence still has no match, check the other records and report `wrong_record` when the value is found there.
+6. **Step-by-step arithmetic:** a number that is still unsupported can be derived from numbers the output has *already grounded*, as long as they appear in the same sentence or the two before it. For example, "$1.11 − $1.08" followed by "= $0.03". Wrong arithmetic is still caught. "A decrease of $6 million ($5829m − $5735m)" fails, because the real difference is $94m.
+
+7. **Written calculations:** arithmetic the output shows, such as `(7167.1 − 6383.0) / 6383.0 × 100 = 12.28%` or `(76 + 73 + 52) / 3 = 67`, is parsed with a small safe parser (never `eval`) and recomputed. If the result is right and every operand is grounded, the result passes. If the stated result disagrees at its stated precision, it is an `arithmetic_error`. Constants such as 100, or the 3 in `/ 3`, need no grounding.
+8. **Table rows:** a value found in a table cell is checked against the row label. If the sentence names a different row in full and none of the matching cells sit in a row the sentence mentions, the claim is `wrong_row`. Set `check_table_rows=False` to switch this off.
+
+Some cases get special handling:
+- **Restatements:** a later sentence may repeat a number the output already grounded, for example "= 67" followed by "the average is $67 million".
+- **Dropped units:** a number written without its unit inside a calculation ("2.2" for "$2.2 billion") matches directly, but is never used as a derivation operand.
+- **Tables:** JSON-style tables in a source, such as `[["", "2019", "2018"], ["sales", "5", "4"]]`, are read cell by cell. Each value is labelled with its row and column, and gets a year when its column header or row label has one.
+- **Direction needs a known order:** a difference such as "decreased by $29,253" has the same size in both orders. crossfoot therefore takes the order from table years (earlier → later), or from how the answer writes the calculation: "$5,836,000 − $4,657,000", "from A to B" or "subtracting A from B". If the wording contradicts that order, the claim is `contradicted`. With no known order, only the magnitude is checked.
+- **Averages:** the average of two values is accepted, for example "($81 + $120) / 2 = $100.5 million".
+- **Table units:** in a source whose text contains a unit header such as "(in millions)", "( millions )" or "in thousands", unscaled numbers also match their scaled value. For example, "7170" under "(in millions)" supports "$7170 million", and the evidence names the header.
+- **Fractions:** a plain number such as `0.1233` can match a percentage change or share written as a fraction.
+- **Not claims:** these are skipped:
+  - formula constants: the `100` in `* 100` or "multiplying by 100", and small divisors such as the `2` in `/ 2` or "divide by 2"
+  - the day in dates (`December 31`) and numeric dates (`11/30/04`, `2019-06-30`)
+  - version and IP strings (`8.4.1.1`)
+
+The derivation search runs backwards from the stated number. For each operand, crossfoot computes the partner value that would produce the stated figure and finds it by binary search. A record with n values therefore costs O(n log n) per claim, with no cap on the number of operands.
+
+## Benchmark
+
+[benchmarks/ragbench.py](https://github.com/ramvikrams/crossfoot/blob/main/benchmarks/ragbench.py) measures crossfoot against the sentence-level support labels in [RAGBench](https://huggingface.co/datasets/rungalileo/ragbench) (CC BY 4.0). It uses the numeric-heavy financial QA subsets `finqa` and `tatqa` by default.
+
+```bash
+pip install -e ".[bench]"
+python benchmarks/ragbench.py --subsets finqa tatqa --limit 300
+```
+
+Latest results (run 7: full `test` splits, about 20,000 numeric sentences; full log in [HISTORY.md](https://github.com/ramvikrams/crossfoot/blob/main/benchmarks/results/HISTORY.md)):
+
+| | finqa (2,294 rows) | tatqa (3,338 rows) |
+|---|---:|---:|
+| Injected value errors caught (±10–30%) | 89.7% (5841/6509) | 92.6% (5303/5726) |
+| Injected scale errors caught (million ↔ billion) | 96.3% (3120/3240) | 94.3% (1662/1762) |
+| Injected direction errors caught (increase ↔ decrease) | 69.9% (214/306) | 70.0% (243/347) |
+| Swapped values caught (another number from the same source) | 13.4% (815/6097) | 9.8% (539/5525) |
+| False alarms on sentences labelled supported | 20.1% | 19.4% |
+
+The first 300 rows of four non-financial subsets give these false-alarm rates: covidqa 2.5%, techqa 16.8%, emanual 26.7% (only 32 numeric sentences) and hagrid 19.0%. On the same subsets, injected value errors are caught 58–91% of the time.
+
+The detection rates come from a controlled test. It starts from sentences that are labelled supported and that crossfoot passes, and injects one known error into each. The false-alarm rate is high because crossfoot is fail-closed by design: anything it cannot prove fails, including correct multi-step arithmetic that it does not yet model. Some of these "false alarms" are also genuine numeric errors that the LLM annotator accepted.
+
+**Hand-checking.** Run `--review-out benchmarks/results/review-sample.csv` to export a seeded random sample of flagged sentences. Each row includes the documents. Fill in the `verdict` column with `real error`, `false alarm` or `unsure`, then run `python benchmarks/review_summary.py benchmarks/results/review-sample.csv` to get a hand-checked precision that does not depend on LLM labels.
+
+The script writes results to [benchmarks/results/](https://github.com/ramvikrams/crossfoot/tree/main/benchmarks/results) and saves example errors under `.cache/ragbench/` for manual review. Keep two caveats in mind when reading the results:
+- RAGBench labels come from an LLM annotator, not from human review.
+- Sentences can be unsupported for non-numeric reasons, so the recall figure is a lower bound.
+
+## Known limitations
+
+- **Coincidental derivations.** With many values per record, a wrong figure can match some unrelated derivation by chance. Above `max_derivation_operands` values per scope (default 30), crossfoot stops deriving from the sources alone. The output must then state the operands it used, which step-by-step arithmetic accepts, or cite a narrower record. Below the limit, coincidences are still possible, so the evidence string shows the formula for a reviewer to audit.
+- **Two-operand derivations only.** No sums of three or more values, averages of more than two values, or CAGR yet.
+- **Ratios** written as "a:b" are not understood.
+- **Direction words** are read from a 60-character window before the number, in English only. Negations ("did not rise") are not understood. When the order of two values is unknown (no table years and no written order), changes below about 8% cannot be told apart by direction at whole-number precision. For example, 100 → 105 is +5%, while 105 → 100 is −4.8%, which rounds to 5. The property-based tests document this.
+- **Row matching** uses word overlap with row labels. It only fires when the sentence names another row in full, so a paraphrased row name is not caught.
+- **Swapped values.** A number replaced with another value from the same source is caught only 9-13% of the time on FinQA and TAT-QA. crossfoot checks values, not which row or column a value belongs to.
+- Small integers ("2 threads") are weakly verifiable in long documents.
+- Numbers written as words ("twelve"), fractions, ranges as a single claim ("10-12%"), dates other than four-digit years, and negative numbers in parentheses ("(1,234)") are not parsed.
+- Any bracketed token is treated as a citation, so `[sic]` will be reported as an unknown citation.
+- `$` is treated as compatible with any dollar currency (USD, HKD). Explicit codes are compared strictly.
+- The DeepEval wrapper targets the `BaseMetric` interface. Confirm it against your DeepEval version.
+
+## Production-gap backlog
+
+Not done yet, and needed before relying on crossfoot in a production pipeline:
+
+- [x] Benchmark harness for a public faithfulness dataset (RAGBench)
+- [ ] Publish full-split benchmark results, plus a hand-reviewed sample of errors
+- [ ] Multi-operand derivations (totals, averages, CAGR) with a bounded search
+- [ ] Unit-aware quantities beyond currency (%, units, km, tonnes) and date parsing
+- [ ] Negation handling and multilingual direction words
+- [ ] Configurable ignore rules (section numbers, ordinals)
+- [ ] Packaged releases on PyPI and the GitHub Marketplace, plus signed tags
+- [x] Sub-quadratic derivation search (no operand cap)
+
+## Related work
+
+- [hardfacts](https://github.com/seekerPrice/hardfacts) checks stated numbers, dates and IDs against sources. According to its README, it treats doubtful matches as supported and flags derived figures without confirming them. crossfoot takes the opposite stance: it is fail-closed, and it verifies derived figures and their direction.
+- [DeepEval](https://deepeval.com), [promptfoo](https://promptfoo.dev) and [Ragas](https://docs.ragas.io) are general LLM evaluation frameworks. crossfoot plugs into the first two as a deterministic metric.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest -q
+crossfoot eval examples/cases.jsonl -v
+```
+
+All example data is fictional.
+
+MIT licensed.

crossfoot-0.1.0/pyproject.toml

@@ -0,0 +1,39 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "crossfoot"
+version = "0.1.0"
+description = "Fail-closed numeric grounding checks for LLM output: verifies stated and derived numbers, direction words and citations."
+readme = "README.md"
+requires-python = ">=3.9"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Ram Vikram Singh" }]
+keywords = ["llm", "evaluation", "hallucination", "grounding", "rag", "guardrails"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Topic :: Software Development :: Testing",
+]
+dependencies = []
+
+[project.urls]
+Homepage = "https://github.com/ramvikrams/crossfoot"
+Source = "https://github.com/ramvikrams/crossfoot"
+Issues = "https://github.com/ramvikrams/crossfoot/issues"
+"Benchmark history" = "https://github.com/ramvikrams/crossfoot/blob/main/benchmarks/results/HISTORY.md"
+
+[project.optional-dependencies]
+dev = ["pytest>=7", "hypothesis>=6"]
+deepeval = ["deepeval"]
+bench = ["pyarrow>=12"]
+
+[project.scripts]
+crossfoot = "crossfoot.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

crossfoot-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

crossfoot-0.1.0/src/crossfoot/__init__.py

@@ -0,0 +1,29 @@
+"""crossfoot: fail-closed numeric grounding checks for LLM output."""
+from .extract import Mention, extract
+from .expressions import find_equations
+from .verify import (
+    ARITHMETIC_ERROR,
+    WRONG_ROW,
+    CONTRADICTED,
+    DERIVED,
+    EXACT,
+    PASSING,
+    ROUNDED,
+    UNCITED,
+    UNKNOWN_CITATION,
+    UNSUPPORTED,
+    WRONG_RECORD,
+    Checker,
+    Claim,
+    Report,
+    check,
+    load_sources,
+)
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "check", "Checker", "extract", "load_sources", "Claim", "Report", "Mention", "PASSING",
+    "EXACT", "ROUNDED", "DERIVED", "WRONG_RECORD", "CONTRADICTED", "UNSUPPORTED", "UNCITED",
+    "UNKNOWN_CITATION", "ARITHMETIC_ERROR", "WRONG_ROW", "find_equations",
+]