proofpath 0.1.0__tar.gz → 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- proofpath-0.2.0/.gitattributes +3 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/CHANGELOG.md +122 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/PKG-INFO +89 -38
- {proofpath-0.1.0 → proofpath-0.2.0}/README.md +86 -37
- {proofpath-0.1.0 → proofpath-0.2.0}/docs/eval/2026-09-11-pairing.md +65 -4
- proofpath-0.2.0/docs/eval/2026-09-12-ghosts.md +222 -0
- proofpath-0.2.0/docs/eval/2026-09-12-pairing-author-year.md +238 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/docs/eval/2026-09-12-v0.1-live.md +48 -0
- proofpath-0.2.0/docs/eval/2026-09-15-v0.2-live.md +264 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/docs/superpowers/OPEN-ITEMS.md +71 -12
- {proofpath-0.1.0 → proofpath-0.2.0}/docs/superpowers/plans/2026-09-10-proofpath-implementation-plan.md +20 -0
- proofpath-0.2.0/docs/superpowers/plans/2026-09-12-phases-9-10-plan.md +414 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/docs/superpowers/specs/2026-09-10-proofpath-design.md +25 -5
- {proofpath-0.1.0 → proofpath-0.2.0}/pyproject.toml +6 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/scripts/eval_pairing.py +85 -21
- {proofpath-0.1.0 → proofpath-0.2.0}/scripts/zero_network_check.py +10 -6
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/__init__.py +1 -1
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/browser.py +78 -3
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/cache.py +256 -15
- proofpath-0.2.0/src/proofpath/claims.py +652 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/cli.py +122 -135
- proofpath-0.2.0/src/proofpath/commands.py +337 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/ingest.py +64 -4
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/polite.py +59 -10
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/report.py +29 -2
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/resolve.py +157 -22
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/retrieval.py +15 -1
- proofpath-0.2.0/src/proofpath/sarif.py +240 -0
- proofpath-0.2.0/src/proofpath/tui/__init__.py +7 -0
- proofpath-0.2.0/src/proofpath/tui/app.py +1813 -0
- proofpath-0.2.0/src/proofpath/tui/banner.py +89 -0
- proofpath-0.2.0/src/proofpath/tui/commands.py +119 -0
- proofpath-0.2.0/src/proofpath/tui/runs.py +459 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/ui.py +45 -4
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/verify.py +80 -9
- proofpath-0.2.0/tests/conftest.py +23 -0
- proofpath-0.2.0/tests/data/draft-author-year.md +25 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/data/ghost_set.jsonl +16 -0
- proofpath-0.2.0/tests/data/pairing_author_year.jsonl +55 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/data/pairing_set.jsonl +6 -6
- proofpath-0.2.0/tests/data/sarif-schema-2.1.0.json +2882 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_browser.py +119 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_cache.py +143 -8
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_cache_cli.py +27 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_check_cli.py +133 -7
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_claims.py +299 -6
- proofpath-0.2.0/tests/test_cli.py +87 -0
- proofpath-0.2.0/tests/test_commands.py +411 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_eval_pairing.py +78 -15
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_fetch_cli.py +1 -1
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_ingest.py +123 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_polite.py +70 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_report.py +73 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_resolve.py +291 -8
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_resolve_cli.py +55 -1
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_retrieval.py +34 -0
- proofpath-0.2.0/tests/test_sarif.py +380 -0
- proofpath-0.2.0/tests/test_tui_app.py +1464 -0
- proofpath-0.2.0/tests/test_tui_banner.py +117 -0
- proofpath-0.2.0/tests/test_tui_commands.py +141 -0
- proofpath-0.2.0/tests/test_tui_runs.py +522 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_ui.py +42 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_verify.py +196 -9
- {proofpath-0.1.0 → proofpath-0.2.0}/uv.lock +334 -0
- proofpath-0.1.0/docs/eval/2026-09-12-ghosts.md +0 -121
- proofpath-0.1.0/src/proofpath/claims.py +0 -327
- proofpath-0.1.0/tests/test_cli.py +0 -20
- {proofpath-0.1.0 → proofpath-0.2.0}/.editorconfig +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/.env.example +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/.github/ISSUE_TEMPLATE/bug_report.yml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/.github/ISSUE_TEMPLATE/config.yml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/.github/ISSUE_TEMPLATE/feature_request.yml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/.github/ISSUE_TEMPLATE/wrong_verdict.yml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/.github/PULL_REQUEST_TEMPLATE.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/.github/workflows/ci.yml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/.github/workflows/release.yml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/.gitignore +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/CLAUDE.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/CONTRIBUTING.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/LICENSE +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/SECURITY.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/docs/eval/2026-09-11-coverage.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/docs/eval/2026-09-11-ghosts.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/docs/eval/2026-09-11-scifact-dev.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/docs/eval/2026-09-12-scifact-dev.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/docs/eval/2026-09-12-tiers.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/docs/research/2026-09-11-free-llm-api-tiers.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/scripts/eval_coverage.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/scripts/eval_ghosts.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/scripts/eval_scifact.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/config.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/device.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/document.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/entailment.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/eval/__init__.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/eval/metrics.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/eval/scifact.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/events.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/fetch.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/judge.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/models.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/numerics.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/oa.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/paths.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/pipeline.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/src/proofpath/py.typed +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/__init__.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/data/draft-live.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fakes.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/oa/crossref_work.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/oa/crossref_work_no_abstract.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/oa/europepmc_fulltext.xml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/oa/europepmc_search.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/oa/europepmc_search_empty.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/oa/europepmc_search_not_in_epmc.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/oa/openalex_work.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/oa/s2_paper.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/oa/s2_paper_no_abstract.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/oa/unpaywall.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/oa/unpaywall_closed.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/oa/unpaywall_landing_only.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/arxiv_id_roberta.xml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/arxiv_title_fabricated.xml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/arxiv_title_roberta.xml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/crossref_alphafold.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/crossref_fabricated.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/crossref_numpy.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/crossref_rag.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/crossref_retracted_wakefield.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/crossref_roberta.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/crossref_work_alphafold.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/crossref_work_numpy.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/openalex_alphafold.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/openalex_fabricated.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/openalex_rag.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/openalex_retracted_wakefield.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/openalex_roberta.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/openlibrary_fabricated.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/openlibrary_using_language.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/s2_alphafold.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/s2_fabricated.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/s2_rag.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/fixtures/resolve/s2_roberta.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_config.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_config_cli.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_device.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_document.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_entailment.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_eval_coverage.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_eval_scifact.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_fetch.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_judge.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_metrics.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_models.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_numerics.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_oa.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_pipeline.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_scifact.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.0}/tests/test_version.py +0 -0
|
@@ -6,6 +6,128 @@ All notable changes to this project are documented here. The format follows
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.2.0] - 2026-09-15
|
|
10
|
+
|
|
11
|
+
The interactive front-end, author-year citations, SARIF output and a cache for the
|
|
12
|
+
two lookups that made a warm re-run slow. Bare `proofpath` now opens the TUI; the
|
|
13
|
+
one-shot verbs are unchanged for CI. Live checks in `docs/eval/2026-09-15-v0.2-live.md`.
|
|
14
|
+
|
|
15
|
+
### Added
|
|
16
|
+
- **TUI** (spec §13.1): bare `proofpath` opens a `textual` session — a four-line banner
|
|
17
|
+
with the pet, one prompt bar, and a scrolling log of run blocks. Every CLI verb is a
|
|
18
|
+
slash command (`/check`, `/resolve`, `/fetch`, `/config`, `/cache`; plus `/allow`,
|
|
19
|
+
`/cancel #n`, `/help`, `/quit`), a bare verb waits for its argument (`Esc` leaves
|
|
20
|
+
it), and a line that is not a command is something to check. Runs are scheduled
|
|
21
|
+
concurrently through one `Scheduler` (network stages in parallel under a shared
|
|
22
|
+
politeness limiter, the NLI model one run at a time), each run is **cancellable**
|
|
23
|
+
mid-flight and keeps what it had decided, the coverage footer never scrolls away,
|
|
24
|
+
and the §7.1 permission prompt is drawn **inline under the stage that hit the wall**
|
|
25
|
+
with `[allow once] [always] [no] [never]` buttons (or `/allow …`). Every log line
|
|
26
|
+
works by mouse and by keyboard: run headers fold, stage rows hide their summary,
|
|
27
|
+
findings open the full quoted passage, `⧉`/`c` copies it, a finding's reference is
|
|
28
|
+
an OSC 8 link to its source. The pet's eyes blink, watch a run and react to its
|
|
29
|
+
result — only with colour, never under `--no-color` or `-q`.
|
|
30
|
+
- **Author-year citations** (spec §9, §17): `(Smith et al., 2020)`, `Smith (2020)`,
|
|
31
|
+
`(Smith, 2020; Jones, 2019)`, `2020a`/`2020b` collisions, `ibid.` and `op. cit.`
|
|
32
|
+
back-references, and mixed `(Smith, 2020; [12])` all pair with their bibliography
|
|
33
|
+
entry. Hand-built set of 55 passages, 83 expectations, rate **0.940**
|
|
34
|
+
(`docs/eval/2026-09-12-pairing-author-year.md`); the numeric set is unchanged at
|
|
35
|
+
108/109. A marker no entry matches is reported as `UNRESOLVED MARKER`, never guessed.
|
|
36
|
+
- **`check --format sarif`** (spec §13.2): the run as a SARIF 2.1.0 log for VS Code or
|
|
37
|
+
any SARIF 2.1.0 viewer — one result per finding on its line, the quoted passage in
|
|
38
|
+
the message of every asserting result, the exact honesty state in `properties`, and
|
|
39
|
+
the run's coverage in `runs[0].properties`. `--out FILE` writes the same document.
|
|
40
|
+
Validated against the schemastore schema, vendored in
|
|
41
|
+
`tests/data/sarif-schema-2.1.0.json`; not yet exercised against GitHub code scanning.
|
|
42
|
+
- **Resolution and retraction cache** (cache schema v3): `resolutions` (30-day TTL,
|
|
43
|
+
keyed by the marker-free folded entry) and `retractions` (30 days for a notice, 7
|
|
44
|
+
for its absence). Neither an `UNVERIFIED (provider unavailable)` nor a retraction
|
|
45
|
+
check every provider failed is ever stored. A warm re-run of the seven-reference
|
|
46
|
+
draft: **1.35 s** wall clock, 0 network calls (v0.1: 15.8 s). `proofpath cache`
|
|
47
|
+
counts both tables.
|
|
48
|
+
- The §7.1 aggregate line `skipped N source(s) because the browser was not permitted`
|
|
49
|
+
is printed by `check` too (terminal and markdown), never dropped by `-q`.
|
|
50
|
+
- `browser_binary_present()`: the consent gate checks for a chromium build under
|
|
51
|
+
`PLAYWRIGHT_BROWSERS_PATH` or the platform default, so a half-installed environment
|
|
52
|
+
runs the idempotent installer instead of failing inside the fetch; the consent log
|
|
53
|
+
gains `browser binary: found|missing`.
|
|
54
|
+
- `commands.py`: the shared wiring behind every mirrored verb (`resolve_reference`,
|
|
55
|
+
`fetch_target`, `config_*`, `cache_*`) returns result objects and never prints, so
|
|
56
|
+
`cli.py` and the TUI can only differ in how a result is drawn.
|
|
57
|
+
- Bibliography fallback for PDFs without a `References` heading (paged formats):
|
|
58
|
+
the last contiguous run of numbered paragraphs is read as the list. AlphaFold:
|
|
59
|
+
0 → 17 references found, 97 markers now reported as unresolved instead of invisible.
|
|
60
|
+
- Resolver rescues: `First Last and First Last` author lists resolve instead of landing
|
|
61
|
+
in `AMBIGUOUS`; an arXiv id whose record agrees on author and year is accepted as
|
|
62
|
+
`RESOLVED (low confidence)` the way a DOI already was. Ghost set 274 rows: false-ghost
|
|
63
|
+
**0.0 %**, ghost recall 99.1 % (`docs/eval/2026-09-12-ghosts.md`).
|
|
64
|
+
- `tests/data/draft-author-year.md`, a committed author-year draft with an offline
|
|
65
|
+
smoke test; live: 6 citations paired, 0 unresolved.
|
|
66
|
+
|
|
67
|
+
### Changed
|
|
68
|
+
- Cache schema **v3** (migrated in place from v1/v2; versions compared as integers).
|
|
69
|
+
- `Claims.unsupported` now means "a style this version cannot pair" and is empty by
|
|
70
|
+
construction; `UNSUPPORTED CITATION STYLE` is left for footnote-only and
|
|
71
|
+
superscript-letter styles. Markers that pair with nothing are `unresolved`.
|
|
72
|
+
- `resolve.looks_unindexed` reads only the initials-path author patterns: a full-name
|
|
73
|
+
list (`First Last and First Last`) no longer counts as "this entry printed an
|
|
74
|
+
author list", so a title-first book keeps its title (see Known issues for the price).
|
|
75
|
+
- `--format sarif` on `resolve` or `fetch` now says `applies to check only` instead of
|
|
76
|
+
naming a future version.
|
|
77
|
+
- Stage row `Claims` reads `N citations, M unresolved` (was `M unsupported`).
|
|
78
|
+
|
|
79
|
+
### Fixed
|
|
80
|
+
- A provider answering with a non-JSON body (a bot wall or maintenance page under a
|
|
81
|
+
200) escaped `resolve.py` as a raw `JSONDecodeError` traceback. Every provider body
|
|
82
|
+
is decoded in one place and that failure is a `ProviderError`, so `resolve`, `fetch`
|
|
83
|
+
and `check` report `UNVERIFIED (provider unavailable)` and never read it as evidence
|
|
84
|
+
the work does not exist.
|
|
85
|
+
- The §7.1 prompt said `blocked this request (HTTP 200)` for the empty-body bot wall;
|
|
86
|
+
it now says `answered without readable text (HTTP 200)`.
|
|
87
|
+
- `resolve` on the CLI leaked one HTTP client per invocation; `commands.resolve_reference`
|
|
88
|
+
owns and closes it.
|
|
89
|
+
- A retraction check every provider failed was recorded as "not retracted". It now
|
|
90
|
+
raises, is reported as `retraction check unavailable` and counted in the stage
|
|
91
|
+
summary (`1 unavailable`), and nothing is cached, so the next run asks again
|
|
92
|
+
(product rule 2). Found alongside it: the fetching stage replaced a source's notes
|
|
93
|
+
instead of appending, which would have dropped that very note.
|
|
94
|
+
- Versions in the cache file compared as strings (`"10" < "9"`); compared as integers now.
|
|
95
|
+
|
|
96
|
+
### Known issues
|
|
97
|
+
- `api_calls` in the footer counts LLM calls only (none yet), so a cold run that spent
|
|
98
|
+
30 s on Crossref and Semantic Scholar still prints `0 API calls`; the provider lookups
|
|
99
|
+
are shown on the stage lines instead.
|
|
100
|
+
- A cached resolution reprints the notes it was stored with (for example an `openalex
|
|
101
|
+
unavailable (HTTP 429)` from the day it was resolved) as if they were current.
|
|
102
|
+
- `proofpath check -` names its SARIF artifact `-`; give the draft a file name when the
|
|
103
|
+
log is meant for a viewer.
|
|
104
|
+
- Author-year pairing matches the first author and the exact year; `(Lindqvist, 2019)`
|
|
105
|
+
against "Okafor, C. and Lindqvist, S." and a year off by one are reported unresolved,
|
|
106
|
+
and two surnames sharing a last word (`Berg` / `van der Berg`) are reported ambiguous
|
|
107
|
+
rather than guessed. `Smith 2020` with no comma is not a marker; `(WHO, 2020)` does
|
|
108
|
+
not pair with "World Health Organization".
|
|
109
|
+
- An unnumbered two-column bibliography is cut at line breaks (RoBERTa: 103 entries
|
|
110
|
+
for ~50), so author-year pairing over such a list resolves few items. The largest
|
|
111
|
+
open item for author-year pairing.
|
|
112
|
+
- The headless bibliography fallback takes only the *last* run of numbered paragraphs
|
|
113
|
+
(AlphaFold: 17 of 84 entries) and drops the prose printed before a block's first entry.
|
|
114
|
+
- A fabricated `First Last and First Last. Title. Venue.` with no year between the
|
|
115
|
+
names and the title reaches `UNVERIFIED (not in bibliographic indexes)`, not `GHOST`
|
|
116
|
+
(the price of the rescue above; rule 3 outranks recall). A proceedings-*volume*
|
|
117
|
+
record can accept a fabricated paper cited into that volume as `RESOLVED (low
|
|
118
|
+
confidence)` — the ghost set's one fabricated acceptance (0.9 %).
|
|
119
|
+
- A cold run of a long bibliography is still serial through the providers (129
|
|
120
|
+
references: minutes); only the TUI's concurrent scheduler runs them in parallel.
|
|
121
|
+
- In the TUI the `loading models …` note is drawn after the `Verifying` row it precedes
|
|
122
|
+
(it is emitted inside that stage).
|
|
123
|
+
- A `#n` run reference in the TUI is a click target that folds its block, not a
|
|
124
|
+
hyperlink (spec §13.1); a run number has no address to open.
|
|
125
|
+
- A provider body that is valid JSON but not an object (a bare list or string) still
|
|
126
|
+
escapes `resolve` as an error, exit 2, rather than being reported as `UNVERIFIED
|
|
127
|
+
(provider unavailable)`.
|
|
128
|
+
- `/quit` in the TUI waits for an in-flight mirrored `/fetch` to finish before the
|
|
129
|
+
app exits; a `/check` run is cancelled, a `/fetch` is not.
|
|
130
|
+
|
|
9
131
|
## [0.1.0] - 2026-09-12
|
|
10
132
|
|
|
11
133
|
First working release: `proofpath check` verifies a document's citations end to end
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: proofpath
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.2.0
|
|
4
4
|
Summary: Check whether the sources behind a claim actually say what the claim says.
|
|
5
5
|
Project-URL: Homepage, https://github.com/Yigtwxx/proofpath
|
|
6
6
|
Project-URL: Repository, https://github.com/Yigtwxx/proofpath
|
|
@@ -42,7 +42,9 @@ Requires-Dist: typer>=0.15
|
|
|
42
42
|
Provides-Extra: browser
|
|
43
43
|
Requires-Dist: scrapling[fetchers]>=0.4.15; extra == 'browser'
|
|
44
44
|
Provides-Extra: dev
|
|
45
|
+
Requires-Dist: jsonschema>=4.23; extra == 'dev'
|
|
45
46
|
Requires-Dist: mypy>=1.14; extra == 'dev'
|
|
47
|
+
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
|
|
46
48
|
Requires-Dist: pytest-cov>=6.0; extra == 'dev'
|
|
47
49
|
Requires-Dist: pytest>=8.3; extra == 'dev'
|
|
48
50
|
Requires-Dist: respx>=0.22; extra == 'dev'
|
|
@@ -71,12 +73,42 @@ Runs locally and free: no API key, no Docker, no server. Windows, Linux, macOS.
|
|
|
71
73
|
|
|
72
74
|
```bash
|
|
73
75
|
uv tool install proofpath
|
|
76
|
+
proofpath
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
```
|
|
80
|
+
,_,
|
|
81
|
+
(o.o)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~[PROOF]
|
|
82
|
+
" " proofpath v0.2.0 academic . online . coreml
|
|
83
|
+
paste a file path, a URL, or a claim. /help /config /quit
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Bare `proofpath` opens the terminal UI ([a recorded session](docs/eval/2026-09-15-v0.2-live.md)).
|
|
87
|
+
Paste a path and it runs; every one-shot verb is a slash command (`/check`, `/resolve`,
|
|
88
|
+
`/fetch`, `/config`, `/cache`), runs can be started while others are in flight and
|
|
89
|
+
stopped with `/cancel #n` — a stopped run keeps what it had decided — and the
|
|
90
|
+
coverage footer never scrolls away. Click (or press `enter` on) a finding to read the
|
|
91
|
+
whole quoted passage; `⧉` copies it; a finding's reference is a link to its source.
|
|
92
|
+
When a publisher blocks the plain fetch, the permission question is asked **inline,
|
|
93
|
+
under the stage that hit the wall**, with `[allow once] [always] [no] [never]`.
|
|
94
|
+
|
|
95
|
+
The same engine behind a pipe or in CI:
|
|
74
96
|
|
|
75
|
-
|
|
97
|
+
```bash
|
|
98
|
+
proofpath check paper.pdf # report → report.md, exit 0/1/2
|
|
76
99
|
proofpath check draft.md --format json | jq '.coverage'
|
|
77
|
-
proofpath
|
|
100
|
+
proofpath check draft.md --format sarif --out draft.sarif # any SARIF 2.1.0 viewer (VS Code's SARIF Viewer, …)
|
|
101
|
+
proofpath -q check - < draft.md # stdin; findings and coverage only
|
|
78
102
|
```
|
|
79
103
|
|
|
104
|
+
`--format sarif` writes a SARIF 2.1.0 log: one result per finding on its line, the
|
|
105
|
+
quoted passage in the message, the exact honesty state in `properties`, and the run's
|
|
106
|
+
coverage in the run's properties, so a log opened without the terminal still says how
|
|
107
|
+
much was read. Exit codes are the interface: `0` clean, `1` findings (every
|
|
108
|
+
`UNVERIFIED` and `LOW CONFIDENCE` counts), `2` the run itself failed — no text
|
|
109
|
+
parsing needed to gate a job. Piped or in CI there is **no prompt**: an `ask`
|
|
110
|
+
permission is treated as `deny` and reported.
|
|
111
|
+
|
|
80
112
|
The install carries **no browser engine**. When a publisher blocks the plain fetch,
|
|
81
113
|
proofpath asks once before downloading one — about 280 MB, into its own environment
|
|
82
114
|
and the shared browser cache, never system-wide — and remembers the answer.
|
|
@@ -99,7 +131,9 @@ unverified 14%
|
|
|
99
131
|
```
|
|
100
132
|
|
|
101
133
|
When a quarter or more of the sources could not be read, a further line says so:
|
|
102
|
-
`coverage is weak: unread sources may hold more, so this is a lower bound`.
|
|
134
|
+
`coverage is weak: unread sources may hold more, so this is a lower bound`. When the
|
|
135
|
+
browser step was not permitted, a line counts the sources it cost:
|
|
136
|
+
`skipped N source(s) because the browser was not permitted`.
|
|
103
137
|
|
|
104
138
|
**No verdict without its passage.** `SUPPORTED` and `REFUTED` cannot exist without
|
|
105
139
|
the quoted sentence they rest on — the cache schema itself refuses to store one.
|
|
@@ -113,6 +147,19 @@ was near-certain here", not as a guarantee of 85 % precision. And `medium` lands
|
|
|
113
147
|
almost exactly on `decide`, so `low` is practically empty among asserted verdicts —
|
|
114
148
|
the display is effectively **two tiers**: near-certain, and asserted at all.
|
|
115
149
|
|
|
150
|
+
## Citations it reads
|
|
151
|
+
|
|
152
|
+
Numeric markers — `[12]`, `[12,15]`, `[12-15]`, Nature-style superscripts — and, since
|
|
153
|
+
v0.2, **author-year**: `(Smith et al., 2020)`, `Smith (2020)`, `(Smith, 2020; Jones,
|
|
154
|
+
2019)`, `2020a`/`2020b` collisions, `ibid.` and `op. cit.`, and a mixed
|
|
155
|
+
`(Smith, 2020; [12])`. Pairing rate on the hand-built author-year set: **0.940**
|
|
156
|
+
(83 expectations over 55 passages, [details](docs/eval/2026-09-12-pairing-author-year.md));
|
|
157
|
+
the numeric set is at 108 of 109. Still unpaired, and reported rather than guessed:
|
|
158
|
+
a surname that is not the entry's *first* author, a year off by one, two surnames
|
|
159
|
+
sharing a last word (`Berg` / `van der Berg`), `Smith 2020` with no comma, and an
|
|
160
|
+
initialism such as `(WHO, 2020)` against "World Health Organization". A marker no
|
|
161
|
+
entry matches is listed as an unresolved marker.
|
|
162
|
+
|
|
116
163
|
## What the states mean
|
|
117
164
|
|
|
118
165
|
Absence of evidence is never reported as evidence of absence. Each of these is a
|
|
@@ -127,65 +174,68 @@ distinct, printed state (spec §15), never collapsed into a verdict:
|
|
|
127
174
|
| `UNVERIFIED (unreachable)` | dead link, Wayback miss |
|
|
128
175
|
| `UNVERIFIED (reached, no text extracted)` | 200 answered, nothing readable came back |
|
|
129
176
|
| `UNVERIFIED (network not permitted)` | `permissions.network = deny` |
|
|
130
|
-
| `UNVERIFIED (provider unavailable)` | API down
|
|
177
|
+
| `UNVERIFIED (provider unavailable)` | API down, rate limited after backoff, or answering with a page instead of a record |
|
|
131
178
|
| `UNVERIFIED (not in bibliographic indexes)` | web page, blog, report — indexes do not cover it, so absence proves nothing |
|
|
132
179
|
| `AMBIGUOUS` | several plausible records, all listed |
|
|
133
180
|
| `NEI` | the source was read and neither supports nor contradicts |
|
|
134
181
|
| `PARAGRAPH-SCOPED` | the citation covers a paragraph; each sentence is judged separately |
|
|
135
|
-
| `UNSUPPORTED CITATION STYLE` |
|
|
182
|
+
| `UNSUPPORTED CITATION STYLE` | reserved; no detected style produces it in v0.2 |
|
|
136
183
|
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
teardown); fixed in this release — 20 of 20 piped runs exit `1` ([live runs](docs/eval/2026-09-12-v0.1-live.md)).
|
|
184
|
+
An earlier build sometimes aborted with `134` after printing a complete report (ONNX
|
|
185
|
+
runtime teardown); fixed in v0.1 — 20 of 20 piped runs exit `1`
|
|
186
|
+
([live runs](docs/eval/2026-09-12-v0.1-live.md)).
|
|
141
187
|
|
|
142
|
-
##
|
|
188
|
+
## Known limitations
|
|
143
189
|
|
|
144
|
-
- **Numeric citation markers only** — `[12]`, `[12,15]`, `[12-15]`; an author–year
|
|
145
|
-
citation is listed as `UNSUPPORTED CITATION STYLE`, not judged.
|
|
146
190
|
- **Superscript citations only when the PDF marks them as superscript**; a PDF that
|
|
147
191
|
draws them as ordinary digits loses them, and `km²` can be read as `[2]`.
|
|
148
192
|
- **Abstract fallback**: when only an abstract is reachable the verdict is labelled
|
|
149
193
|
`LOW CONFIDENCE (abstract only)`. Three sentences is not a source.
|
|
150
|
-
- **
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
- **
|
|
155
|
-
|
|
156
|
-
|
|
194
|
+
- **An unnumbered two-column bibliography is cut at line breaks** (RoBERTa's ACL
|
|
195
|
+
list: 103 entries for ~50, half of them with no author), so author-year pairing over
|
|
196
|
+
such a list resolves few items — 15 claims where a rejoined list would give 65. The
|
|
197
|
+
largest open item for author-year pairing.
|
|
198
|
+
- **A reference list without a `References` heading** is read by its shape in paged
|
|
199
|
+
formats — the last contiguous run of numbered paragraphs — which recovers only that
|
|
200
|
+
last block (AlphaFold: 17 of 84 entries) and drops the prose printed before a
|
|
201
|
+
block's first entry.
|
|
202
|
+
- **The 0 % false-ghost rate is measured on 274 hand-built references**, and rule 3
|
|
203
|
+
is paid for in recall: a fabricated `First Last and First Last. Title. Venue.` with
|
|
204
|
+
no year between the names and the title is reported as `not in bibliographic
|
|
205
|
+
indexes`, not as a ghost, and a proceedings-*volume* record can accept a fabricated
|
|
206
|
+
paper cited into that volume as `RESOLVED (low confidence)` — the set's one
|
|
207
|
+
fabricated acceptance (0.9 %) ([details](docs/eval/2026-09-12-ghosts.md)).
|
|
208
|
+
- **A cold run of a long bibliography is still serial** through the providers (129
|
|
209
|
+
references: minutes). Only the TUI runs several documents' network stages at once.
|
|
157
210
|
- **A refused or failed browser install is reported, not hidden**: the source is
|
|
158
211
|
`UNVERIFIED (blocked, browser not permitted)` with the install log, never silently
|
|
159
212
|
counted as unreachable.
|
|
160
213
|
- **Coverage is not perfectly reproducible**: two runs minutes apart can read a
|
|
161
214
|
different number of sources, depending on which providers answered.
|
|
162
|
-
- The
|
|
163
|
-
in **v0.3**.
|
|
215
|
+
- The LLM judge and `--summarize` arrive in **v0.3**.
|
|
164
216
|
|
|
165
217
|
## Speed
|
|
166
218
|
|
|
167
|
-
Apple Silicon Mac, models already downloaded
|
|
219
|
+
Apple Silicon Mac, models already downloaded:
|
|
168
220
|
|
|
169
221
|
| document | first run | cached re-run |
|
|
170
222
|
|---|---|---|
|
|
171
|
-
| 1-page markdown draft, 7 references | 79.3 s | 15.8 s
|
|
172
|
-
| 19-page arXiv PDF, 68 references | 12 m 57 s | 3 m 42 s |
|
|
173
|
-
|
|
174
|
-
* the draft's cached re-run followed the earlier of the two recorded cold runs; the live doc keeps both.
|
|
223
|
+
| 1-page markdown draft, 7 references | 79.3 s (v0.1 cold) | **1.35 s** ([v0.2](docs/eval/2026-09-15-v0.2-live.md)) — 15.8 s in v0.1 |
|
|
224
|
+
| 19-page arXiv PDF, 68 references | 12 m 57 s | 3 m 42 s ([v0.1](docs/eval/2026-09-12-v0.1-live.md)) |
|
|
175
225
|
|
|
176
|
-
The **first ever** run also downloads about 250 MB of ONNX models.
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
read is fetched again — the PDF re-run above
|
|
181
|
-
|
|
226
|
+
The **first ever** run also downloads about 250 MB of ONNX models. Since v0.2 a
|
|
227
|
+
cached re-run asks the network for nothing: reference resolution and the retraction
|
|
228
|
+
check are cached (resolutions 30 days; a retraction hit 30 days, a miss 7 days), the
|
|
229
|
+
fetched text for 7 days, and chunks and verdicts for as long as the text is unchanged. The models are still loaded, and any source whose text
|
|
230
|
+
has expired or was never read is fetched again — the PDF re-run above (v0.1) still
|
|
231
|
+
spent 22 s fetching and re-scored 3 of 102 claims.
|
|
182
232
|
|
|
183
233
|
## Looking inside the cache
|
|
184
234
|
|
|
185
235
|
Everything proofpath fetches, embeds and decides lands in one plain SQLite file:
|
|
186
236
|
|
|
187
237
|
```bash
|
|
188
|
-
proofpath cache # where it is
|
|
238
|
+
proofpath cache # where it is, what it holds, how many lookups it remembers
|
|
189
239
|
proofpath cache ls # sources, chunk/verdict counts, text expiry
|
|
190
240
|
proofpath cache show <id> # one source's chunks and verdicts
|
|
191
241
|
proofpath cache clear --expired
|
|
@@ -193,8 +243,8 @@ proofpath cache clear --expired
|
|
|
193
243
|
|
|
194
244
|
Open `proofpath cache path` in [DB Browser for SQLite](https://sqlitebrowser.org/),
|
|
195
245
|
TablePlus or DBeaver — plain tables, no extension. Raw publisher text expires after
|
|
196
|
-
7 days; verdicts keep the passage they quote
|
|
197
|
-
fetch URL|DOI` run either half on its own.
|
|
246
|
+
7 days; verdicts keep the passage they quote; a provider outage is never stored.
|
|
247
|
+
`proofpath resolve REF` and `proofpath fetch URL|DOI` run either half on its own.
|
|
198
248
|
|
|
199
249
|
## Optional LLM judge — arrives in v0.3
|
|
200
250
|
|
|
@@ -209,9 +259,10 @@ comes from the environment or a `.env` file, never from config, and is never pri
|
|
|
209
259
|
| What | Set | Result |
|
|
210
260
|
|---|---|---|
|
|
211
261
|
| Retrieval + entailment | SciFact dev, 340 pairs | 0.609 accuracy, 0.597 macro-F1, against a 0.406 trivial baseline ([details](docs/eval/2026-09-12-scifact-dev.md)) |
|
|
212
|
-
| Reference resolution | hand-built ghost set,
|
|
262
|
+
| Reference resolution | hand-built ghost set, 274 references | 0.0 % false-ghost, 99.1 % ghost recall ([details](docs/eval/2026-09-12-ghosts.md)) |
|
|
213
263
|
| Source access | 50 DOIs | 72 % full text, 18 % abstract only, 10 % nothing ([details](docs/eval/2026-09-11-coverage.md)) — a real biomedical paper in the live runs reached 33 % full text |
|
|
214
|
-
| Citation pairing | 61 hand-built passages | 0.
|
|
264
|
+
| Citation pairing, numeric | 61 hand-built passages | 0.99 ([details](docs/eval/2026-09-11-pairing.md)) |
|
|
265
|
+
| Citation pairing, author-year | 55 hand-built passages, 83 expectations | 0.940 ([details](docs/eval/2026-09-12-pairing-author-year.md)) |
|
|
215
266
|
|
|
216
267
|
Published SciFact results sit around 70–75 F1, not 95. Nothing is tuned on a test
|
|
217
268
|
split, and no number is quoted without the run that produced it.
|
|
@@ -17,12 +17,42 @@ Runs locally and free: no API key, no Docker, no server. Windows, Linux, macOS.
|
|
|
17
17
|
|
|
18
18
|
```bash
|
|
19
19
|
uv tool install proofpath
|
|
20
|
+
proofpath
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
```
|
|
24
|
+
,_,
|
|
25
|
+
(o.o)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~[PROOF]
|
|
26
|
+
" " proofpath v0.2.0 academic . online . coreml
|
|
27
|
+
paste a file path, a URL, or a claim. /help /config /quit
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
Bare `proofpath` opens the terminal UI ([a recorded session](docs/eval/2026-09-15-v0.2-live.md)).
|
|
31
|
+
Paste a path and it runs; every one-shot verb is a slash command (`/check`, `/resolve`,
|
|
32
|
+
`/fetch`, `/config`, `/cache`), runs can be started while others are in flight and
|
|
33
|
+
stopped with `/cancel #n` — a stopped run keeps what it had decided — and the
|
|
34
|
+
coverage footer never scrolls away. Click (or press `enter` on) a finding to read the
|
|
35
|
+
whole quoted passage; `⧉` copies it; a finding's reference is a link to its source.
|
|
36
|
+
When a publisher blocks the plain fetch, the permission question is asked **inline,
|
|
37
|
+
under the stage that hit the wall**, with `[allow once] [always] [no] [never]`.
|
|
38
|
+
|
|
39
|
+
The same engine behind a pipe or in CI:
|
|
20
40
|
|
|
21
|
-
|
|
41
|
+
```bash
|
|
42
|
+
proofpath check paper.pdf # report → report.md, exit 0/1/2
|
|
22
43
|
proofpath check draft.md --format json | jq '.coverage'
|
|
23
|
-
proofpath
|
|
44
|
+
proofpath check draft.md --format sarif --out draft.sarif # any SARIF 2.1.0 viewer (VS Code's SARIF Viewer, …)
|
|
45
|
+
proofpath -q check - < draft.md # stdin; findings and coverage only
|
|
24
46
|
```
|
|
25
47
|
|
|
48
|
+
`--format sarif` writes a SARIF 2.1.0 log: one result per finding on its line, the
|
|
49
|
+
quoted passage in the message, the exact honesty state in `properties`, and the run's
|
|
50
|
+
coverage in the run's properties, so a log opened without the terminal still says how
|
|
51
|
+
much was read. Exit codes are the interface: `0` clean, `1` findings (every
|
|
52
|
+
`UNVERIFIED` and `LOW CONFIDENCE` counts), `2` the run itself failed — no text
|
|
53
|
+
parsing needed to gate a job. Piped or in CI there is **no prompt**: an `ask`
|
|
54
|
+
permission is treated as `deny` and reported.
|
|
55
|
+
|
|
26
56
|
The install carries **no browser engine**. When a publisher blocks the plain fetch,
|
|
27
57
|
proofpath asks once before downloading one — about 280 MB, into its own environment
|
|
28
58
|
and the shared browser cache, never system-wide — and remembers the answer.
|
|
@@ -45,7 +75,9 @@ unverified 14%
|
|
|
45
75
|
```
|
|
46
76
|
|
|
47
77
|
When a quarter or more of the sources could not be read, a further line says so:
|
|
48
|
-
`coverage is weak: unread sources may hold more, so this is a lower bound`.
|
|
78
|
+
`coverage is weak: unread sources may hold more, so this is a lower bound`. When the
|
|
79
|
+
browser step was not permitted, a line counts the sources it cost:
|
|
80
|
+
`skipped N source(s) because the browser was not permitted`.
|
|
49
81
|
|
|
50
82
|
**No verdict without its passage.** `SUPPORTED` and `REFUTED` cannot exist without
|
|
51
83
|
the quoted sentence they rest on — the cache schema itself refuses to store one.
|
|
@@ -59,6 +91,19 @@ was near-certain here", not as a guarantee of 85 % precision. And `medium` lands
|
|
|
59
91
|
almost exactly on `decide`, so `low` is practically empty among asserted verdicts —
|
|
60
92
|
the display is effectively **two tiers**: near-certain, and asserted at all.
|
|
61
93
|
|
|
94
|
+
## Citations it reads
|
|
95
|
+
|
|
96
|
+
Numeric markers — `[12]`, `[12,15]`, `[12-15]`, Nature-style superscripts — and, since
|
|
97
|
+
v0.2, **author-year**: `(Smith et al., 2020)`, `Smith (2020)`, `(Smith, 2020; Jones,
|
|
98
|
+
2019)`, `2020a`/`2020b` collisions, `ibid.` and `op. cit.`, and a mixed
|
|
99
|
+
`(Smith, 2020; [12])`. Pairing rate on the hand-built author-year set: **0.940**
|
|
100
|
+
(83 expectations over 55 passages, [details](docs/eval/2026-09-12-pairing-author-year.md));
|
|
101
|
+
the numeric set is at 108 of 109. Still unpaired, and reported rather than guessed:
|
|
102
|
+
a surname that is not the entry's *first* author, a year off by one, two surnames
|
|
103
|
+
sharing a last word (`Berg` / `van der Berg`), `Smith 2020` with no comma, and an
|
|
104
|
+
initialism such as `(WHO, 2020)` against "World Health Organization". A marker no
|
|
105
|
+
entry matches is listed as an unresolved marker.
|
|
106
|
+
|
|
62
107
|
## What the states mean
|
|
63
108
|
|
|
64
109
|
Absence of evidence is never reported as evidence of absence. Each of these is a
|
|
@@ -73,65 +118,68 @@ distinct, printed state (spec §15), never collapsed into a verdict:
|
|
|
73
118
|
| `UNVERIFIED (unreachable)` | dead link, Wayback miss |
|
|
74
119
|
| `UNVERIFIED (reached, no text extracted)` | 200 answered, nothing readable came back |
|
|
75
120
|
| `UNVERIFIED (network not permitted)` | `permissions.network = deny` |
|
|
76
|
-
| `UNVERIFIED (provider unavailable)` | API down
|
|
121
|
+
| `UNVERIFIED (provider unavailable)` | API down, rate limited after backoff, or answering with a page instead of a record |
|
|
77
122
|
| `UNVERIFIED (not in bibliographic indexes)` | web page, blog, report — indexes do not cover it, so absence proves nothing |
|
|
78
123
|
| `AMBIGUOUS` | several plausible records, all listed |
|
|
79
124
|
| `NEI` | the source was read and neither supports nor contradicts |
|
|
80
125
|
| `PARAGRAPH-SCOPED` | the citation covers a paragraph; each sentence is judged separately |
|
|
81
|
-
| `UNSUPPORTED CITATION STYLE` |
|
|
126
|
+
| `UNSUPPORTED CITATION STYLE` | reserved; no detected style produces it in v0.2 |
|
|
82
127
|
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
teardown); fixed in this release — 20 of 20 piped runs exit `1` ([live runs](docs/eval/2026-09-12-v0.1-live.md)).
|
|
128
|
+
An earlier build sometimes aborted with `134` after printing a complete report (ONNX
|
|
129
|
+
runtime teardown); fixed in v0.1 — 20 of 20 piped runs exit `1`
|
|
130
|
+
([live runs](docs/eval/2026-09-12-v0.1-live.md)).
|
|
87
131
|
|
|
88
|
-
##
|
|
132
|
+
## Known limitations
|
|
89
133
|
|
|
90
|
-
- **Numeric citation markers only** — `[12]`, `[12,15]`, `[12-15]`; an author–year
|
|
91
|
-
citation is listed as `UNSUPPORTED CITATION STYLE`, not judged.
|
|
92
134
|
- **Superscript citations only when the PDF marks them as superscript**; a PDF that
|
|
93
135
|
draws them as ordinary digits loses them, and `km²` can be read as `[2]`.
|
|
94
136
|
- **Abstract fallback**: when only an abstract is reachable the verdict is labelled
|
|
95
137
|
`LOW CONFIDENCE (abstract only)`. Three sentences is not a source.
|
|
96
|
-
- **
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
- **
|
|
101
|
-
|
|
102
|
-
|
|
138
|
+
- **An unnumbered two-column bibliography is cut at line breaks** (RoBERTa's ACL
|
|
139
|
+
list: 103 entries for ~50, half of them with no author), so author-year pairing over
|
|
140
|
+
such a list resolves few items — 15 claims where a rejoined list would give 65. The
|
|
141
|
+
largest open item for author-year pairing.
|
|
142
|
+
- **A reference list without a `References` heading** is read by its shape in paged
|
|
143
|
+
formats — the last contiguous run of numbered paragraphs — which recovers only that
|
|
144
|
+
last block (AlphaFold: 17 of 84 entries) and drops the prose printed before a
|
|
145
|
+
block's first entry.
|
|
146
|
+
- **The 0 % false-ghost rate is measured on 274 hand-built references**, and rule 3
|
|
147
|
+
is paid for in recall: a fabricated `First Last and First Last. Title. Venue.` with
|
|
148
|
+
no year between the names and the title is reported as `not in bibliographic
|
|
149
|
+
indexes`, not as a ghost, and a proceedings-*volume* record can accept a fabricated
|
|
150
|
+
paper cited into that volume as `RESOLVED (low confidence)` — the set's one
|
|
151
|
+
fabricated acceptance (0.9 %) ([details](docs/eval/2026-09-12-ghosts.md)).
|
|
152
|
+
- **A cold run of a long bibliography is still serial** through the providers (129
|
|
153
|
+
references: minutes). Only the TUI runs several documents' network stages at once.
|
|
103
154
|
- **A refused or failed browser install is reported, not hidden**: the source is
|
|
104
155
|
`UNVERIFIED (blocked, browser not permitted)` with the install log, never silently
|
|
105
156
|
counted as unreachable.
|
|
106
157
|
- **Coverage is not perfectly reproducible**: two runs minutes apart can read a
|
|
107
158
|
different number of sources, depending on which providers answered.
|
|
108
|
-
- The
|
|
109
|
-
in **v0.3**.
|
|
159
|
+
- The LLM judge and `--summarize` arrive in **v0.3**.
|
|
110
160
|
|
|
111
161
|
## Speed
|
|
112
162
|
|
|
113
|
-
Apple Silicon Mac, models already downloaded
|
|
163
|
+
Apple Silicon Mac, models already downloaded:
|
|
114
164
|
|
|
115
165
|
| document | first run | cached re-run |
|
|
116
166
|
|---|---|---|
|
|
117
|
-
| 1-page markdown draft, 7 references | 79.3 s | 15.8 s
|
|
118
|
-
| 19-page arXiv PDF, 68 references | 12 m 57 s | 3 m 42 s |
|
|
119
|
-
|
|
120
|
-
* the draft's cached re-run followed the earlier of the two recorded cold runs; the live doc keeps both.
|
|
167
|
+
| 1-page markdown draft, 7 references | 79.3 s (v0.1 cold) | **1.35 s** ([v0.2](docs/eval/2026-09-15-v0.2-live.md)) — 15.8 s in v0.1 |
|
|
168
|
+
| 19-page arXiv PDF, 68 references | 12 m 57 s | 3 m 42 s ([v0.1](docs/eval/2026-09-12-v0.1-live.md)) |
|
|
121
169
|
|
|
122
|
-
The **first ever** run also downloads about 250 MB of ONNX models.
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
read is fetched again — the PDF re-run above
|
|
127
|
-
|
|
170
|
+
The **first ever** run also downloads about 250 MB of ONNX models. Since v0.2 a
|
|
171
|
+
cached re-run asks the network for nothing: reference resolution and the retraction
|
|
172
|
+
check are cached (resolutions 30 days; a retraction hit 30 days, a miss 7 days), the
|
|
173
|
+
fetched text for 7 days, and chunks and verdicts for as long as the text is unchanged. The models are still loaded, and any source whose text
|
|
174
|
+
has expired or was never read is fetched again — the PDF re-run above (v0.1) still
|
|
175
|
+
spent 22 s fetching and re-scored 3 of 102 claims.
|
|
128
176
|
|
|
129
177
|
## Looking inside the cache
|
|
130
178
|
|
|
131
179
|
Everything proofpath fetches, embeds and decides lands in one plain SQLite file:
|
|
132
180
|
|
|
133
181
|
```bash
|
|
134
|
-
proofpath cache # where it is
|
|
182
|
+
proofpath cache # where it is, what it holds, how many lookups it remembers
|
|
135
183
|
proofpath cache ls # sources, chunk/verdict counts, text expiry
|
|
136
184
|
proofpath cache show <id> # one source's chunks and verdicts
|
|
137
185
|
proofpath cache clear --expired
|
|
@@ -139,8 +187,8 @@ proofpath cache clear --expired
|
|
|
139
187
|
|
|
140
188
|
Open `proofpath cache path` in [DB Browser for SQLite](https://sqlitebrowser.org/),
|
|
141
189
|
TablePlus or DBeaver — plain tables, no extension. Raw publisher text expires after
|
|
142
|
-
7 days; verdicts keep the passage they quote
|
|
143
|
-
fetch URL|DOI` run either half on its own.
|
|
190
|
+
7 days; verdicts keep the passage they quote; a provider outage is never stored.
|
|
191
|
+
`proofpath resolve REF` and `proofpath fetch URL|DOI` run either half on its own.
|
|
144
192
|
|
|
145
193
|
## Optional LLM judge — arrives in v0.3
|
|
146
194
|
|
|
@@ -155,9 +203,10 @@ comes from the environment or a `.env` file, never from config, and is never pri
|
|
|
155
203
|
| What | Set | Result |
|
|
156
204
|
|---|---|---|
|
|
157
205
|
| Retrieval + entailment | SciFact dev, 340 pairs | 0.609 accuracy, 0.597 macro-F1, against a 0.406 trivial baseline ([details](docs/eval/2026-09-12-scifact-dev.md)) |
|
|
158
|
-
| Reference resolution | hand-built ghost set,
|
|
206
|
+
| Reference resolution | hand-built ghost set, 274 references | 0.0 % false-ghost, 99.1 % ghost recall ([details](docs/eval/2026-09-12-ghosts.md)) |
|
|
159
207
|
| Source access | 50 DOIs | 72 % full text, 18 % abstract only, 10 % nothing ([details](docs/eval/2026-09-11-coverage.md)) — a real biomedical paper in the live runs reached 33 % full text |
|
|
160
|
-
| Citation pairing | 61 hand-built passages | 0.
|
|
208
|
+
| Citation pairing, numeric | 61 hand-built passages | 0.99 ([details](docs/eval/2026-09-11-pairing.md)) |
|
|
209
|
+
| Citation pairing, author-year | 55 hand-built passages, 83 expectations | 0.940 ([details](docs/eval/2026-09-12-pairing-author-year.md)) |
|
|
161
210
|
|
|
162
211
|
Published SciFact results sit around 70–75 F1, not 95. Nothing is tuned on a test
|
|
163
212
|
split, and no number is quoted without the run that produced it.
|