proofpath 0.1.0__tar.gz → 0.2.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- proofpath-0.2.1/.gitattributes +3 -0
- proofpath-0.2.1/CHANGELOG.md +325 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/PKG-INFO +116 -38
- proofpath-0.2.1/README.md +246 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/docs/eval/2026-09-11-pairing.md +65 -4
- proofpath-0.2.1/docs/eval/2026-09-12-ghosts.md +222 -0
- proofpath-0.2.1/docs/eval/2026-09-12-pairing-author-year.md +238 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/docs/eval/2026-09-12-v0.1-live.md +48 -0
- proofpath-0.2.1/docs/eval/2026-09-15-tui-v2-live.md +267 -0
- proofpath-0.2.1/docs/eval/2026-09-15-v0.2-live.md +264 -0
- proofpath-0.2.1/docs/eval/tui-v2-plain.svg +279 -0
- proofpath-0.2.1/docs/eval/tui-v2-rich.svg +280 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/docs/superpowers/OPEN-ITEMS.md +96 -12
- {proofpath-0.1.0 → proofpath-0.2.1}/docs/superpowers/plans/2026-09-10-proofpath-implementation-plan.md +20 -0
- proofpath-0.2.1/docs/superpowers/plans/2026-09-12-phases-9-10-plan.md +414 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/docs/superpowers/specs/2026-09-10-proofpath-design.md +44 -5
- proofpath-0.2.1/docs/superpowers/specs/2026-09-15-tui-v2-design.md +142 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/pyproject.toml +6 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/scripts/eval_pairing.py +85 -21
- {proofpath-0.1.0 → proofpath-0.2.1}/scripts/zero_network_check.py +10 -6
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/__init__.py +1 -1
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/browser.py +78 -3
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/cache.py +256 -15
- proofpath-0.2.1/src/proofpath/claims.py +652 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/cli.py +122 -135
- proofpath-0.2.1/src/proofpath/commands.py +337 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/ingest.py +64 -4
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/polite.py +59 -10
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/report.py +29 -2
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/resolve.py +157 -22
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/retrieval.py +15 -1
- proofpath-0.2.1/src/proofpath/sarif.py +240 -0
- proofpath-0.2.1/src/proofpath/tui/__init__.py +7 -0
- proofpath-0.2.1/src/proofpath/tui/app.py +681 -0
- proofpath-0.2.1/src/proofpath/tui/banner.py +89 -0
- proofpath-0.2.1/src/proofpath/tui/commands.py +119 -0
- proofpath-0.2.1/src/proofpath/tui/pet.py +309 -0
- proofpath-0.2.1/src/proofpath/tui/runs.py +459 -0
- proofpath-0.2.1/src/proofpath/tui/theme.py +259 -0
- proofpath-0.2.1/src/proofpath/tui/verbs.py +328 -0
- proofpath-0.2.1/src/proofpath/tui/widgets/__init__.py +47 -0
- proofpath-0.2.1/src/proofpath/tui/widgets/_shared.py +96 -0
- proofpath-0.2.1/src/proofpath/tui/widgets/banner.py +194 -0
- proofpath-0.2.1/src/proofpath/tui/widgets/finding.py +341 -0
- proofpath-0.2.1/src/proofpath/tui/widgets/footer.py +162 -0
- proofpath-0.2.1/src/proofpath/tui/widgets/prompt.py +109 -0
- proofpath-0.2.1/src/proofpath/tui/widgets/run_block.py +565 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/ui.py +58 -6
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/verify.py +80 -9
- proofpath-0.2.1/tests/conftest.py +23 -0
- proofpath-0.2.1/tests/data/draft-author-year.md +25 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/data/ghost_set.jsonl +16 -0
- proofpath-0.2.1/tests/data/pairing_author_year.jsonl +55 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/data/pairing_set.jsonl +6 -6
- proofpath-0.2.1/tests/data/sarif-schema-2.1.0.json +2882 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_browser.py +119 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_cache.py +143 -8
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_cache_cli.py +27 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_check_cli.py +133 -7
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_claims.py +299 -6
- proofpath-0.2.1/tests/test_cli.py +87 -0
- proofpath-0.2.1/tests/test_commands.py +411 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_eval_pairing.py +78 -15
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_fetch_cli.py +1 -1
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_ingest.py +123 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_polite.py +70 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_report.py +73 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_resolve.py +291 -8
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_resolve_cli.py +55 -1
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_retrieval.py +34 -0
- proofpath-0.2.1/tests/test_sarif.py +380 -0
- proofpath-0.2.1/tests/test_tui_app.py +1480 -0
- proofpath-0.2.1/tests/test_tui_banner.py +117 -0
- proofpath-0.2.1/tests/test_tui_commands.py +141 -0
- proofpath-0.2.1/tests/test_tui_pet.py +483 -0
- proofpath-0.2.1/tests/test_tui_rich.py +679 -0
- proofpath-0.2.1/tests/test_tui_runs.py +522 -0
- proofpath-0.2.1/tests/test_tui_theme.py +319 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_ui.py +42 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_verify.py +196 -9
- {proofpath-0.1.0 → proofpath-0.2.1}/uv.lock +334 -0
- proofpath-0.1.0/CHANGELOG.md +0 -162
- proofpath-0.1.0/README.md +0 -170
- proofpath-0.1.0/docs/eval/2026-09-12-ghosts.md +0 -121
- proofpath-0.1.0/src/proofpath/claims.py +0 -327
- proofpath-0.1.0/tests/test_cli.py +0 -20
- {proofpath-0.1.0 → proofpath-0.2.1}/.editorconfig +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/.env.example +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/.github/ISSUE_TEMPLATE/bug_report.yml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/.github/ISSUE_TEMPLATE/config.yml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/.github/ISSUE_TEMPLATE/feature_request.yml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/.github/ISSUE_TEMPLATE/wrong_verdict.yml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/.github/PULL_REQUEST_TEMPLATE.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/.github/workflows/ci.yml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/.github/workflows/release.yml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/.gitignore +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/CLAUDE.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/CONTRIBUTING.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/LICENSE +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/SECURITY.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/docs/eval/2026-09-11-coverage.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/docs/eval/2026-09-11-ghosts.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/docs/eval/2026-09-11-scifact-dev.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/docs/eval/2026-09-12-scifact-dev.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/docs/eval/2026-09-12-tiers.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/docs/research/2026-09-11-free-llm-api-tiers.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/scripts/eval_coverage.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/scripts/eval_ghosts.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/scripts/eval_scifact.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/config.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/device.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/document.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/entailment.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/eval/__init__.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/eval/metrics.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/eval/scifact.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/events.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/fetch.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/judge.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/models.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/numerics.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/oa.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/paths.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/pipeline.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/src/proofpath/py.typed +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/__init__.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/data/draft-live.md +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fakes.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/oa/crossref_work.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/oa/crossref_work_no_abstract.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/oa/europepmc_fulltext.xml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/oa/europepmc_search.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/oa/europepmc_search_empty.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/oa/europepmc_search_not_in_epmc.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/oa/openalex_work.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/oa/s2_paper.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/oa/s2_paper_no_abstract.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/oa/unpaywall.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/oa/unpaywall_closed.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/oa/unpaywall_landing_only.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/arxiv_id_roberta.xml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/arxiv_title_fabricated.xml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/arxiv_title_roberta.xml +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/crossref_alphafold.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/crossref_fabricated.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/crossref_numpy.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/crossref_rag.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/crossref_retracted_wakefield.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/crossref_roberta.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/crossref_work_alphafold.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/crossref_work_numpy.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/openalex_alphafold.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/openalex_fabricated.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/openalex_rag.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/openalex_retracted_wakefield.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/openalex_roberta.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/openlibrary_fabricated.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/openlibrary_using_language.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/s2_alphafold.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/s2_fabricated.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/s2_rag.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/fixtures/resolve/s2_roberta.json +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_config.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_config_cli.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_device.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_document.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_entailment.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_eval_coverage.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_eval_scifact.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_fetch.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_judge.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_metrics.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_models.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_numerics.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_oa.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_pipeline.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_scifact.py +0 -0
- {proofpath-0.1.0 → proofpath-0.2.1}/tests/test_version.py +0 -0
|
@@ -0,0 +1,325 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented here. The format follows
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and this project adheres to
|
|
5
|
+
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
|
+
|
|
7
|
+
## [Unreleased]
|
|
8
|
+
|
|
9
|
+
## [0.2.1] - 2026-09-15
|
|
10
|
+
|
|
11
|
+
The TUI's second look. No behaviour change: every state word, every honesty sentence,
|
|
12
|
+
the exit codes, the scheduler and the one-shot CLI are exactly v0.2.0's. Design in
|
|
13
|
+
`docs/superpowers/specs/2026-09-15-tui-v2-design.md`; the by-hand session in both
|
|
14
|
+
themes, with SVG screenshots, in `docs/eval/2026-09-15-tui-v2-live.md`.
|
|
15
|
+
|
|
16
|
+
### Changed
|
|
17
|
+
- **Two themes, one truth** (`tui/theme.py`). `rich` draws box borders, Unicode glyphs
|
|
18
|
+
and truecolor tones when the terminal gives evidence of them (`COLORTERM`, Windows
|
|
19
|
+
Terminal, iTerm2, kitty, WezTerm, Ghostty, VS Code, Terminal.app); `plain` is a pure-ASCII, ANSI-16 look
|
|
20
|
+
(v0.2.0's `⏺ ✗ ⚠ › ⧉` become `* x ! > [copy]`, and findings now print their `= note:`
|
|
21
|
+
lines as the CLI does), chosen under `NO_COLOR`, `--no-color`, `-q`,
|
|
22
|
+
`TERM=dumb`, legacy conhost, a CJK locale, or any session without truecolor evidence (an SSH
|
|
23
|
+
or tmux session that strips `COLORTERM` gets `plain`). `PROOFPATH_THEME=rich|plain` overrides
|
|
24
|
+
detection. The meaning colours stay `ui.py`'s tables; a theme changes how a meaning
|
|
25
|
+
looks, never what a word means. Spec §13.1's "pure ASCII" rule for the pet now binds
|
|
26
|
+
`plain` only.
|
|
27
|
+
- **The pet** (`tui/pet.py`): `rich` draws a seven-line ferret with a real tail running
|
|
28
|
+
to the `[PROOF]` stamp; `plain` keeps the three-line one unchanged. Same eyes, same
|
|
29
|
+
blink, same reactions; the tail wags in `rich` while a run works.
|
|
30
|
+
- **Run panels** (`tui/widgets/run_block.py`): in `rich` each run is a rounded panel in
|
|
31
|
+
its accent, the command and the state word on the top border, the coverage on the
|
|
32
|
+
bottom one; the stages are a fixed-column table (symbol, name, summary with `·`
|
|
33
|
+
separators, attribution, elapsed) and the active stage carries a real `▰▱` progress
|
|
34
|
+
bar with `done/total`. A finished stage that left something unverified keeps the
|
|
35
|
+
`⏺` mark instead of a tick. Below 60 columns the borders go and the flat rows
|
|
36
|
+
return; a run that crossed the floor either way says its coverage exactly once.
|
|
37
|
+
- **Findings** (`tui/widgets/finding.py`): the state word is a badge, the location a
|
|
38
|
+
fixed cell, the tier right-aligned; the finding's notes and the claim (`you`) and
|
|
39
|
+
the passage (`source`) it was checked against are printed under it. A tier-less
|
|
40
|
+
badge keeps its row at 60–79 columns. `plain` prints the notes as the CLI's own
|
|
41
|
+
`= note:` rows and keeps `[copy]`.
|
|
42
|
+
- **Footer** (`tui/widgets/footer.py`): `rich` draws the coverage as a proportional
|
|
43
|
+
`█▓░` bar in the three meaning colours over the counts line; `plain` keeps the
|
|
44
|
+
`kv` lines. The prompt wears a one-line rounded border in the run's accent.
|
|
45
|
+
- **Structure** (OPEN-ITEMS 12.4): `tui/app.py` is split — `theme.py`, `pet.py`,
|
|
46
|
+
`verbs.py` and `tui/widgets/` (`run_block`, `finding`, `footer`, `prompt`,
|
|
47
|
+
`banner`, `_shared`) — and is now 681 lines of composition, scheduler wiring and
|
|
48
|
+
slash commands, no rendering.
|
|
49
|
+
|
|
50
|
+
## [0.2.0] - 2026-09-15
|
|
51
|
+
|
|
52
|
+
The interactive front-end, author-year citations, SARIF output and a cache for the
|
|
53
|
+
two lookups that made a warm re-run slow. Bare `proofpath` now opens the TUI; the
|
|
54
|
+
one-shot verbs are unchanged for CI. Live checks in `docs/eval/2026-09-15-v0.2-live.md`.
|
|
55
|
+
|
|
56
|
+
### Added
|
|
57
|
+
- **TUI** (spec §13.1): bare `proofpath` opens a `textual` session — a four-line banner
|
|
58
|
+
with the pet, one prompt bar, and a scrolling log of run blocks. Every CLI verb is a
|
|
59
|
+
slash command (`/check`, `/resolve`, `/fetch`, `/config`, `/cache`; plus `/allow`,
|
|
60
|
+
`/cancel #n`, `/help`, `/quit`), a bare verb waits for its argument (`Esc` leaves
|
|
61
|
+
it), and a line that is not a command is something to check. Runs are scheduled
|
|
62
|
+
concurrently through one `Scheduler` (network stages in parallel under a shared
|
|
63
|
+
politeness limiter, the NLI model one run at a time), each run is **cancellable**
|
|
64
|
+
mid-flight and keeps what it had decided, the coverage footer never scrolls away,
|
|
65
|
+
and the §7.1 permission prompt is drawn **inline under the stage that hit the wall**
|
|
66
|
+
with `[allow once] [always] [no] [never]` buttons (or `/allow …`). Every log line
|
|
67
|
+
works by mouse and by keyboard: run headers fold, stage rows hide their summary,
|
|
68
|
+
findings open the full quoted passage, `⧉`/`c` copies it, a finding's reference is
|
|
69
|
+
an OSC 8 link to its source. The pet's eyes blink, watch a run and react to its
|
|
70
|
+
result — only with colour, never under `--no-color` or `-q`.
|
|
71
|
+
- **Author-year citations** (spec §9, §17): `(Smith et al., 2020)`, `Smith (2020)`,
|
|
72
|
+
`(Smith, 2020; Jones, 2019)`, `2020a`/`2020b` collisions, `ibid.` and `op. cit.`
|
|
73
|
+
back-references, and mixed `(Smith, 2020; [12])` all pair with their bibliography
|
|
74
|
+
entry. Hand-built set of 55 passages, 83 expectations, rate **0.940**
|
|
75
|
+
(`docs/eval/2026-09-12-pairing-author-year.md`); the numeric set is unchanged at
|
|
76
|
+
108/109. A marker no entry matches is reported as `UNRESOLVED MARKER`, never guessed.
|
|
77
|
+
- **`check --format sarif`** (spec §13.2): the run as a SARIF 2.1.0 log for VS Code or
|
|
78
|
+
any SARIF 2.1.0 viewer — one result per finding on its line, the quoted passage in
|
|
79
|
+
the message of every asserting result, the exact honesty state in `properties`, and
|
|
80
|
+
the run's coverage in `runs[0].properties`. `--out FILE` writes the same document.
|
|
81
|
+
Validated against the schemastore schema, vendored in
|
|
82
|
+
`tests/data/sarif-schema-2.1.0.json`; not yet exercised against GitHub code scanning.
|
|
83
|
+
- **Resolution and retraction cache** (cache schema v3): `resolutions` (30-day TTL,
|
|
84
|
+
keyed by the marker-free folded entry) and `retractions` (30 days for a notice, 7
|
|
85
|
+
for its absence). Neither an `UNVERIFIED (provider unavailable)` nor a retraction
|
|
86
|
+
check every provider failed is ever stored. A warm re-run of the seven-reference
|
|
87
|
+
draft: **1.35 s** wall clock, 0 network calls (v0.1: 15.8 s). `proofpath cache`
|
|
88
|
+
counts both tables.
|
|
89
|
+
- The §7.1 aggregate line `skipped N source(s) because the browser was not permitted`
|
|
90
|
+
is printed by `check` too (terminal and markdown), never dropped by `-q`.
|
|
91
|
+
- `browser_binary_present()`: the consent gate checks for a chromium build under
|
|
92
|
+
`PLAYWRIGHT_BROWSERS_PATH` or the platform default, so a half-installed environment
|
|
93
|
+
runs the idempotent installer instead of failing inside the fetch; the consent log
|
|
94
|
+
gains `browser binary: found|missing`.
|
|
95
|
+
- `commands.py`: the shared wiring behind every mirrored verb (`resolve_reference`,
|
|
96
|
+
`fetch_target`, `config_*`, `cache_*`) returns result objects and never prints, so
|
|
97
|
+
`cli.py` and the TUI can only differ in how a result is drawn.
|
|
98
|
+
- Bibliography fallback for PDFs without a `References` heading (paged formats):
|
|
99
|
+
the last contiguous run of numbered paragraphs is read as the list. AlphaFold:
|
|
100
|
+
0 → 17 references found, 97 markers now reported as unresolved instead of invisible.
|
|
101
|
+
- Resolver rescues: `First Last and First Last` author lists resolve instead of landing
|
|
102
|
+
in `AMBIGUOUS`; an arXiv id whose record agrees on author and year is accepted as
|
|
103
|
+
`RESOLVED (low confidence)` the way a DOI already was. Ghost set 274 rows: false-ghost
|
|
104
|
+
**0.0 %**, ghost recall 99.1 % (`docs/eval/2026-09-12-ghosts.md`).
|
|
105
|
+
- `tests/data/draft-author-year.md`, a committed author-year draft with an offline
|
|
106
|
+
smoke test; live: 6 citations paired, 0 unresolved.
|
|
107
|
+
|
|
108
|
+
### Changed
|
|
109
|
+
- Cache schema **v3** (migrated in place from v1/v2; versions compared as integers).
|
|
110
|
+
- `Claims.unsupported` now means "a style this version cannot pair" and is empty by
|
|
111
|
+
construction; `UNSUPPORTED CITATION STYLE` is left for footnote-only and
|
|
112
|
+
superscript-letter styles. Markers that pair with nothing are `unresolved`.
|
|
113
|
+
- `resolve.looks_unindexed` reads only the initials-path author patterns: a full-name
|
|
114
|
+
list (`First Last and First Last`) no longer counts as "this entry printed an
|
|
115
|
+
author list", so a title-first book keeps its title (see Known issues for the price).
|
|
116
|
+
- `--format sarif` on `resolve` or `fetch` now says `applies to check only` instead of
|
|
117
|
+
naming a future version.
|
|
118
|
+
- Stage row `Claims` reads `N citations, M unresolved` (was `M unsupported`).
|
|
119
|
+
|
|
120
|
+
### Fixed
|
|
121
|
+
- A provider answering with a non-JSON body (a bot wall or maintenance page under a
|
|
122
|
+
200) escaped `resolve.py` as a raw `JSONDecodeError` traceback. Every provider body
|
|
123
|
+
is decoded in one place and that failure is a `ProviderError`, so `resolve`, `fetch`
|
|
124
|
+
and `check` report `UNVERIFIED (provider unavailable)` and never read it as evidence
|
|
125
|
+
the work does not exist.
|
|
126
|
+
- The §7.1 prompt said `blocked this request (HTTP 200)` for the empty-body bot wall;
|
|
127
|
+
it now says `answered without readable text (HTTP 200)`.
|
|
128
|
+
- `resolve` on the CLI leaked one HTTP client per invocation; `commands.resolve_reference`
|
|
129
|
+
owns and closes it.
|
|
130
|
+
- A retraction check every provider failed was recorded as "not retracted". It now
|
|
131
|
+
raises, is reported as `retraction check unavailable` and counted in the stage
|
|
132
|
+
summary (`1 unavailable`), and nothing is cached, so the next run asks again
|
|
133
|
+
(product rule 2). Found alongside it: the fetching stage replaced a source's notes
|
|
134
|
+
instead of appending, which would have dropped that very note.
|
|
135
|
+
- Versions in the cache file compared as strings (`"10" < "9"`); compared as integers now.
|
|
136
|
+
|
|
137
|
+
### Known issues
|
|
138
|
+
- `api_calls` in the footer counts LLM calls only (none yet), so a cold run that spent
|
|
139
|
+
30 s on Crossref and Semantic Scholar still prints `0 API calls`; the provider lookups
|
|
140
|
+
are shown on the stage lines instead.
|
|
141
|
+
- A cached resolution reprints the notes it was stored with (for example an `openalex
|
|
142
|
+
unavailable (HTTP 429)` from the day it was resolved) as if they were current.
|
|
143
|
+
- `proofpath check -` names its SARIF artifact `-`; give the draft a file name when the
|
|
144
|
+
log is meant for a viewer.
|
|
145
|
+
- Author-year pairing matches the first author and the exact year; `(Lindqvist, 2019)`
|
|
146
|
+
against "Okafor, C. and Lindqvist, S." and a year off by one are reported unresolved,
|
|
147
|
+
and two surnames sharing a last word (`Berg` / `van der Berg`) are reported ambiguous
|
|
148
|
+
rather than guessed. `Smith 2020` with no comma is not a marker; `(WHO, 2020)` does
|
|
149
|
+
not pair with "World Health Organization".
|
|
150
|
+
- An unnumbered two-column bibliography is cut at line breaks (RoBERTa: 103 entries
|
|
151
|
+
for ~50), so author-year pairing over such a list resolves few items. The largest
|
|
152
|
+
open item for author-year pairing.
|
|
153
|
+
- The headless bibliography fallback takes only the *last* run of numbered paragraphs
|
|
154
|
+
(AlphaFold: 17 of 84 entries) and drops the prose printed before a block's first entry.
|
|
155
|
+
- A fabricated `First Last and First Last. Title. Venue.` with no year between the
|
|
156
|
+
names and the title reaches `UNVERIFIED (not in bibliographic indexes)`, not `GHOST`
|
|
157
|
+
(the price of the rescue above; rule 3 outranks recall). A proceedings-*volume*
|
|
158
|
+
record can accept a fabricated paper cited into that volume as `RESOLVED (low
|
|
159
|
+
confidence)` — the ghost set's one fabricated acceptance (0.9 %).
|
|
160
|
+
- A cold run of a long bibliography is still serial through the providers (129
|
|
161
|
+
references: minutes); only the TUI's concurrent scheduler runs them in parallel.
|
|
162
|
+
- In the TUI the `loading models …` note is drawn after the `Verifying` row it precedes
|
|
163
|
+
(it is emitted inside that stage).
|
|
164
|
+
- A `#n` run reference in the TUI is a click target that folds its block, not a
|
|
165
|
+
hyperlink (spec §13.1); a run number has no address to open.
|
|
166
|
+
- A provider body that is valid JSON but not an object (a bare list or string) still
|
|
167
|
+
escapes `resolve` as an error, exit 2, rather than being reported as `UNVERIFIED
|
|
168
|
+
(provider unavailable)`.
|
|
169
|
+
- `/quit` in the TUI waits for an in-flight mirrored `/fetch` to finish before the
|
|
170
|
+
app exits; a `/check` run is cancelled, a `/fetch` is not.
|
|
171
|
+
|
|
172
|
+
## [0.1.0] - 2026-09-12
|
|
173
|
+
|
|
174
|
+
First working release: `proofpath check` verifies a document's citations end to end
|
|
175
|
+
and writes a report that states its own coverage.
|
|
176
|
+
|
|
177
|
+
### Added
|
|
178
|
+
- `proofpath check TARGET` as the v0.1 surface: compiler-style diagnostics, a
|
|
179
|
+
markdown `report.md`, `--format json`, `-q`, and exit codes `0` / `1` / `2`.
|
|
180
|
+
- Confidence tiers calibrated on SciFact dev rather than chosen by hand
|
|
181
|
+
(`decide=0.45`, `medium=0.457948`, `high=0.99933`;
|
|
182
|
+
`docs/eval/2026-09-12-tiers.md`), shipped as `pipeline.DEFAULT_THRESHOLDS`.
|
|
183
|
+
- Nine live user-like runs recorded in `docs/eval/2026-09-12-v0.1-live.md`, and
|
|
184
|
+
`scripts/zero_network_check.py`, which guards both HTTP clients and re-verifies a
|
|
185
|
+
document: with the network gone the fetch ladder makes no attempt at all, because
|
|
186
|
+
resolution produces no identifiers to fetch with.
|
|
187
|
+
- Config and permissions module: `config.toml` under the platform config dir,
|
|
188
|
+
`proofpath config` / `proofpath config set permissions.<key>`, and the rule that an
|
|
189
|
+
`ask` permission without a TTY resolves to `deny` and is reported (spec §7.1).
|
|
190
|
+
- Core types (`Verdict` cannot be `SUPPORTED`/`REFUTED` without a passage), device
|
|
191
|
+
selection (CUDA → CoreML → CPU), sentence retrieval over `fastembed` embeddings
|
|
192
|
+
scanned with numpy, ONNX NLI entailment on `cross-encoder/nli-deberta-v3-base`, and the aggregation
|
|
193
|
+
pipeline.
|
|
194
|
+
- Reference resolution (spec §8): Crossref + Semantic Scholar first, then arXiv,
|
|
195
|
+
Open Library and OpenAlex before any ghost call; identity decided only by
|
|
196
|
+
field agreement against the raw string; DOI / arXiv id resolved directly;
|
|
197
|
+
retraction check via Crossref's Retraction Watch data and OpenAlex; new state
|
|
198
|
+
`UNVERIFIED (not in bibliographic indexes)`; per-host throttling, `Retry-After`,
|
|
199
|
+
OpenAlex daily-budget handling. `proofpath resolve "<reference>"` on the CLI.
|
|
200
|
+
Hand-built ghost set (106 real, 100 fabricated, 20 mutated) and
|
|
201
|
+
`scripts/eval_ghosts.py`: false-ghost rate 0 %.
|
|
202
|
+
- Persistent cache: one plain SQLite file (`sources`, `raw_text` with 7-day TTL,
|
|
203
|
+
`chunks` with float32 embeddings, `verdicts`), a schema `CHECK` that refuses an
|
|
204
|
+
asserted verdict without a passage, and `proofpath cache` / `cache path` / `ls` /
|
|
205
|
+
`show` / `clear [--expired]`.
|
|
206
|
+
- Numeric claim layer (spec §10): percentages, factors and unit counts with
|
|
207
|
+
direction are compared before NLI; an unambiguous contradiction is refuted by
|
|
208
|
+
rule with both figures named (`Verdict.reason`). Conservative by design: one
|
|
209
|
+
comparable figure on each side, change never against level.
|
|
210
|
+
- Judge settings (`[judge]` in config, Groq default) with `proofpath config check`
|
|
211
|
+
and `proofpath config set judge.<key>`; API key resolved from the environment or
|
|
212
|
+
`.env`, never stored or printed. `.env.example` added.
|
|
213
|
+
- SciFact loader pinned to the AI2 tarball by sha256, evaluation metrics, and
|
|
214
|
+
`scripts/eval_scifact.py`. First measured result: dev accuracy 0.606 vs 0.406
|
|
215
|
+
trivial baseline (`docs/eval/2026-09-11-scifact-dev.md`).
|
|
216
|
+
|
|
217
|
+
- Fetch ladder (spec §7): `httpx` → `curl_cffi` TLS impersonation → browser engine
|
|
218
|
+
behind the §7.1 consent prompt → Wayback Machine; `robots.txt` via `protego`;
|
|
219
|
+
content type from headers; per-host throttling and backoff shared in `polite.py`;
|
|
220
|
+
fetched text cached with the 7-day TTL. Distinct honesty states for blocked,
|
|
221
|
+
blocked-by-robots, browser-not-permitted, unreachable, provider-unavailable and
|
|
222
|
+
network-denied — never collapsed.
|
|
223
|
+
- Consent gate for the ~280 MB browser engine: asks at most once per run, never
|
|
224
|
+
without a TTY, `always`/`never` persist to config, installs with `pip` (or `uv`)
|
|
225
|
+
and `scrapling install`, and reports how many sources were skipped.
|
|
226
|
+
- Open-access chain: Semantic Scholar → Crossref TDM links → Unpaywall (only with a
|
|
227
|
+
contact address) → Europe PMC → arXiv → landing page → abstract (OpenAlex last);
|
|
228
|
+
abstract-only results labelled `LOW CONFIDENCE (abstract only)`. DataCite arXiv
|
|
229
|
+
DOIs resolve straight to the arXiv PDF. `proofpath fetch <url|doi|arXiv id>`.
|
|
230
|
+
- `scripts/eval_coverage.py`: measured 72 % full text / 18 % abstract / 10 % none
|
|
231
|
+
on 50 DOIs (`docs/eval/2026-09-11-coverage.md`).
|
|
232
|
+
|
|
233
|
+
- Document ingest and claim extraction (spec §9): `document.py` value types with
|
|
234
|
+
page/line locators; `ingest.py` for PDF (pymupdf blocks, per-page line numbers,
|
|
235
|
+
running header/footer removal, superscript citations), docx (paragraphs and tables),
|
|
236
|
+
markdown and plain text, with the bibliography kept as raw strings, every
|
|
237
|
+
unparseable page reported as a `PageError` and a page holding an image and no text
|
|
238
|
+
reported as a scan rather than passed on as an empty page; `claims.py` pairs numeric
|
|
239
|
+
markers (`[12]`, `[12,15]`, `[12-15]`) with their sentence, applies the
|
|
240
|
+
`PARAGRAPH-SCOPED` rule, reports author-year markers as `UNSUPPORTED CITATION STYLE`
|
|
241
|
+
and any number the bibliography does not print as unresolved. Hand-built pairing set
|
|
242
|
+
(61 passages, rate 0.98) and `scripts/eval_pairing.py`; four real documents (three
|
|
243
|
+
PDFs and one extracted text) measured in `docs/eval/2026-09-11-pairing.md`.
|
|
244
|
+
|
|
245
|
+
- `proofpath check` (spec §9, §13.2, §15): one `verify()` entry point built as
|
|
246
|
+
`prepare()` (parse, claims, resolve, retractions, fetch) and `decide_all()` (retrieval,
|
|
247
|
+
numeric rule, NLI, cached per source and claim); compiler-style diagnostics with the
|
|
248
|
+
quoted passage, the confidence tier and the exact honesty state; a coverage block in
|
|
249
|
+
every run and a "coverage is weak" line when a quarter or more of the sources could
|
|
250
|
+
not be read; `report.md` written by default, `--format json`, `-q`, `--out`, `check -`
|
|
251
|
+
for stdin, a real Ctrl-C that keeps what was decided (exit 2). A second run of the
|
|
252
|
+
same document re-decides nothing: chunks and verdicts come from the cache and neither
|
|
253
|
+
model scores again, although the models are still loaded and reference resolution and
|
|
254
|
+
the retraction check still go to the network. Cache schema v2: chunks carry the text
|
|
255
|
+
digest they were cut from and a source's verdicts are dropped when its text changes.
|
|
256
|
+
`permissions.network = deny` now also skips reference resolution and the retraction
|
|
257
|
+
check, each reported as not attempted.
|
|
258
|
+
|
|
259
|
+
### Changed
|
|
260
|
+
- CLI surface (spec §13.3): `permissions` and `judge` groups replaced by `config`
|
|
261
|
+
(`config` / `show` / `path` / `set SECTION.KEY VALUE` / `check`); global
|
|
262
|
+
`--no-color` and `-q`; one `ui.py` layer over `rich` owns every colour and the
|
|
263
|
+
10-column key/value layout; `resolve --format json`; "provider unavailable"
|
|
264
|
+
exits `1` (a finding), no longer `2`. A resolved but retracted reference and a
|
|
265
|
+
URL that was reached but yielded no text (`reached but no text extracted`) are
|
|
266
|
+
findings too (`1`). `skipped N source(s)` counts sources, not the URLs tried for
|
|
267
|
+
them; `--format json` adds `browser.skipped_urls`. `permissions.network` binds
|
|
268
|
+
the open-access providers as well as the ladder, and `ask` without a TTY is
|
|
269
|
+
`deny`, reported.
|
|
270
|
+
- Retrieval no longer depends on `sqlite-vec`: a numpy cosine scan is faster at
|
|
271
|
+
every measured scale and the plain SQLite file opens in any GUI.
|
|
272
|
+
- Spec: `PARAGRAPH-SCOPED` and `UNSUPPORTED CITATION STYLE` states, three-tier
|
|
273
|
+
confidence display, 7-day raw-text cache TTL, v0.1 limited to numeric citation
|
|
274
|
+
markers.
|
|
275
|
+
|
|
276
|
+
### Fixed (found by the release's own live runs, `docs/eval/2026-09-12-v0.1-live.md`)
|
|
277
|
+
- A real reference whose author list carries a surname particle (`van der Walt`) was
|
|
278
|
+
called a `GHOST REFERENCE` — product rule 3. Reference resolution now understands
|
|
279
|
+
particles (including glued `al-`/`el-` forms), scores the title against every
|
|
280
|
+
title-like segment, and accepts a DOI whose record agrees on first author and year
|
|
281
|
+
as `RESOLVED (low confidence)` instead of a ghost. The ghost set grew to 258 rows
|
|
282
|
+
(`docs/eval/2026-09-12-ghosts.md`): false-ghost rate 0.0 %, fabricated recall 100 %.
|
|
283
|
+
- A fabricated reference in a numbered bibliography was reported as
|
|
284
|
+
`UNVERIFIED (not in bibliographic indexes)` instead of a ghost, because the printed
|
|
285
|
+
marker (`[7] `) blinded the "is this even a paper" check. The marker is stripped once
|
|
286
|
+
at the resolver's entry, guarded so it can never remove a year, an identifier or a
|
|
287
|
+
title that begins with a number.
|
|
288
|
+
- A markdown draft piped through `proofpath -q check -` lost its `## References`.
|
|
289
|
+
- The consent-gated browser step (fetch-ladder step 3) could not install itself
|
|
290
|
+
(`scrapling` has no `__main__`); it now runs the package's own CLI after `pip`/`uv`,
|
|
291
|
+
and a paywalled Cell landing page was read through it live (3,632 words).
|
|
292
|
+
- A piped run occasionally aborted with exit `134` from ONNX runtime teardown after
|
|
293
|
+
printing a complete report; the engine now releases both models explicitly and the
|
|
294
|
+
CLI flushes its streams before exiting (20 of 20 consecutive runs exit `1`).
|
|
295
|
+
- A document with citation markers but no detected bibliography printed a clean-looking
|
|
296
|
+
`0 %` coverage block; it now says `no bibliography was found; N citation markers could
|
|
297
|
+
not be checked`.
|
|
298
|
+
|
|
299
|
+
### Known issues
|
|
300
|
+
- Reference resolution and the retraction check are not cached, so a warm re-run is
|
|
301
|
+
still a network run: a 68-reference PDF takes about 13 minutes cold and 3.7 minutes
|
|
302
|
+
cached, a 129-reference one 24 minutes cold. Concurrent resolution and a resolution
|
|
303
|
+
cache are v0.2 work.
|
|
304
|
+
- Two-author lists written `First Last and First Last` resolve to `AMBIGUOUS`, not
|
|
305
|
+
`RESOLVED`; an arXiv id whose record agrees on author and year is not yet rescued the
|
|
306
|
+
way a DOI is.
|
|
307
|
+
- `browser.is_installed()` checks that the packages import, not that a browser binary
|
|
308
|
+
exists; a half-installed environment skips the installer and fails inside the browser
|
|
309
|
+
fetch (reported as an unverified source, never a crash).
|
|
310
|
+
|
|
311
|
+
## [0.0.1] - 2026-09-10
|
|
312
|
+
|
|
313
|
+
First release. The verification pipeline is not implemented; this reserves the name
|
|
314
|
+
and establishes the interface, packaging and CI that later phases build on.
|
|
315
|
+
|
|
316
|
+
### Added
|
|
317
|
+
- Design specification with measured source-access data (spec §6), the fetch ladder
|
|
318
|
+
and its permission model (§7), and corrected reference resolution (§8).
|
|
319
|
+
- Phased implementation plan, ordered by risk retired rather than user-visible
|
|
320
|
+
progress.
|
|
321
|
+
- `proofpath` command. A bare invocation is a first-class entry point rather than a
|
|
322
|
+
help screen, which is where the TUI will attach. Exit codes are fixed: `0` clean,
|
|
323
|
+
`1` findings, `2` the run itself failed.
|
|
324
|
+
- Cross-platform CI on Linux, macOS and Windows, and PyPI publishing through trusted
|
|
325
|
+
publishing rather than a stored API token.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: proofpath
|
|
3
|
-
Version: 0.1
|
|
3
|
+
Version: 0.2.1
|
|
4
4
|
Summary: Check whether the sources behind a claim actually say what the claim says.
|
|
5
5
|
Project-URL: Homepage, https://github.com/Yigtwxx/proofpath
|
|
6
6
|
Project-URL: Repository, https://github.com/Yigtwxx/proofpath
|
|
@@ -42,7 +42,9 @@ Requires-Dist: typer>=0.15
|
|
|
42
42
|
Provides-Extra: browser
|
|
43
43
|
Requires-Dist: scrapling[fetchers]>=0.4.15; extra == 'browser'
|
|
44
44
|
Provides-Extra: dev
|
|
45
|
+
Requires-Dist: jsonschema>=4.23; extra == 'dev'
|
|
45
46
|
Requires-Dist: mypy>=1.14; extra == 'dev'
|
|
47
|
+
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
|
|
46
48
|
Requires-Dist: pytest-cov>=6.0; extra == 'dev'
|
|
47
49
|
Requires-Dist: pytest>=8.3; extra == 'dev'
|
|
48
50
|
Requires-Dist: respx>=0.22; extra == 'dev'
|
|
@@ -71,12 +73,69 @@ Runs locally and free: no API key, no Docker, no server. Windows, Linux, macOS.
|
|
|
71
73
|
|
|
72
74
|
```bash
|
|
73
75
|
uv tool install proofpath
|
|
76
|
+
proofpath
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
```
|
|
80
|
+
╭╮ ╭╮
|
|
81
|
+
╭╯╰─╯╰────────────────────────────────────────────────────────────────────╮
|
|
82
|
+
╸┤ o o ╰~~~~~~~~~~~~~[PROOF]
|
|
83
|
+
╰─┬─┬────────────────────────────────────────────────────────────────┬─┬──╯
|
|
84
|
+
˘ ˘ ˘ ˘
|
|
85
|
+
proofpath v0.2.1 academic . online . coreml
|
|
86
|
+
paste a file path, a URL, or a claim. /help /config /quit
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
Bare `proofpath` opens the terminal UI ([a recorded session in both themes](docs/eval/2026-09-15-tui-v2-live.md),
|
|
90
|
+
with [SVG screenshots](docs/eval/tui-v2-rich.svg)). Each run is one panel in its own
|
|
91
|
+
accent: the command on the top border and the run's state at its right, a fixed-column
|
|
92
|
+
stage table underneath (`✓` finished, `⏺` still running or finished with something
|
|
93
|
+
unverified, a real progress bar on the active stage, the provider that produced each
|
|
94
|
+
number at the right), a rule, then the findings — location, reference, the state word
|
|
95
|
+
as a badge, the tier — with the finding's notes and the claim (`you`) and the passage
|
|
96
|
+
(`source`) it was checked against under it. The bottom border carries the run's
|
|
97
|
+
coverage; the docked footer draws it as a bar and never scrolls away.
|
|
98
|
+
|
|
99
|
+
Paste a path and it runs; every one-shot verb is a slash command (`/check`, `/resolve`,
|
|
100
|
+
`/fetch`, `/config`, `/cache`), runs can be started while others are in flight and
|
|
101
|
+
stopped with `/cancel #n` — a stopped run keeps what it had decided. Click (or press
|
|
102
|
+
`enter` on) a finding to read the whole quoted passage; `⧉` copies it; a finding's
|
|
103
|
+
reference is a link to its source. When a publisher blocks the plain fetch, the
|
|
104
|
+
permission question is asked **inline, under the stage that hit the wall**, with
|
|
105
|
+
`[allow once] [always] [no] [never]`.
|
|
106
|
+
|
|
107
|
+
**Windows / `NO_COLOR`.** The look above is the `rich` theme, chosen when the terminal
|
|
108
|
+
gives evidence of truecolor (`COLORTERM`, Windows Terminal, iTerm2, kitty, WezTerm,
|
|
109
|
+
Ghostty, VS Code, Terminal.app). Under `NO_COLOR`, `--no-color`, `-q`, `TERM=dumb`,
|
|
110
|
+
legacy conhost or a CJK locale the `plain` theme draws the same runs as flat ASCII rows
|
|
111
|
+
in the terminal's own 16 colours — the state words, the coverage and every honesty
|
|
112
|
+
sentence are identical, only the drawing changes. `PROOFPATH_THEME=rich|plain` forces
|
|
113
|
+
either, for screenshots and bug reports.
|
|
114
|
+
|
|
115
|
+
```
|
|
116
|
+
,_,
|
|
117
|
+
(o.o)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~[PROOF]
|
|
118
|
+
" " proofpath v0.2.1 academic . online . coreml
|
|
119
|
+
paste a file path, a URL, or a claim. /help /config /quit
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
The same engine behind a pipe or in CI:
|
|
74
123
|
|
|
75
|
-
|
|
124
|
+
```bash
|
|
125
|
+
proofpath check paper.pdf # report → report.md, exit 0/1/2
|
|
76
126
|
proofpath check draft.md --format json | jq '.coverage'
|
|
77
|
-
proofpath
|
|
127
|
+
proofpath check draft.md --format sarif --out draft.sarif # any SARIF 2.1.0 viewer (VS Code's SARIF Viewer, …)
|
|
128
|
+
proofpath -q check - < draft.md # stdin; findings and coverage only
|
|
78
129
|
```
|
|
79
130
|
|
|
131
|
+
`--format sarif` writes a SARIF 2.1.0 log: one result per finding on its line, the
|
|
132
|
+
quoted passage in the message, the exact honesty state in `properties`, and the run's
|
|
133
|
+
coverage in the run's properties, so a log opened without the terminal still says how
|
|
134
|
+
much was read. Exit codes are the interface: `0` clean, `1` findings (every
|
|
135
|
+
`UNVERIFIED` and `LOW CONFIDENCE` counts), `2` the run itself failed — no text
|
|
136
|
+
parsing needed to gate a job. Piped or in CI there is **no prompt**: an `ask`
|
|
137
|
+
permission is treated as `deny` and reported.
|
|
138
|
+
|
|
80
139
|
The install carries **no browser engine**. When a publisher blocks the plain fetch,
|
|
81
140
|
proofpath asks once before downloading one — about 280 MB, into its own environment
|
|
82
141
|
and the shared browser cache, never system-wide — and remembers the answer.
|
|
@@ -99,7 +158,9 @@ unverified 14%
|
|
|
99
158
|
```
|
|
100
159
|
|
|
101
160
|
When a quarter or more of the sources could not be read, a further line says so:
|
|
102
|
-
`coverage is weak: unread sources may hold more, so this is a lower bound`.
|
|
161
|
+
`coverage is weak: unread sources may hold more, so this is a lower bound`. When the
|
|
162
|
+
browser step was not permitted, a line counts the sources it cost:
|
|
163
|
+
`skipped N source(s) because the browser was not permitted`.
|
|
103
164
|
|
|
104
165
|
**No verdict without its passage.** `SUPPORTED` and `REFUTED` cannot exist without
|
|
105
166
|
the quoted sentence they rest on — the cache schema itself refuses to store one.
|
|
@@ -113,6 +174,19 @@ was near-certain here", not as a guarantee of 85 % precision. And `medium` lands
|
|
|
113
174
|
almost exactly on `decide`, so `low` is practically empty among asserted verdicts —
|
|
114
175
|
the display is effectively **two tiers**: near-certain, and asserted at all.
|
|
115
176
|
|
|
177
|
+
## Citations it reads
|
|
178
|
+
|
|
179
|
+
Numeric markers — `[12]`, `[12,15]`, `[12-15]`, Nature-style superscripts — and, since
|
|
180
|
+
v0.2, **author-year**: `(Smith et al., 2020)`, `Smith (2020)`, `(Smith, 2020; Jones,
|
|
181
|
+
2019)`, `2020a`/`2020b` collisions, `ibid.` and `op. cit.`, and a mixed
|
|
182
|
+
`(Smith, 2020; [12])`. Pairing rate on the hand-built author-year set: **0.940**
|
|
183
|
+
(83 expectations over 55 passages, [details](docs/eval/2026-09-12-pairing-author-year.md));
|
|
184
|
+
the numeric set is at 108 of 109. Still unpaired, and reported rather than guessed:
|
|
185
|
+
a surname that is not the entry's *first* author, a year off by one, two surnames
|
|
186
|
+
sharing a last word (`Berg` / `van der Berg`), `Smith 2020` with no comma, and an
|
|
187
|
+
initialism such as `(WHO, 2020)` against "World Health Organization". A marker no
|
|
188
|
+
entry matches is listed as an unresolved marker.
|
|
189
|
+
|
|
116
190
|
## What the states mean
|
|
117
191
|
|
|
118
192
|
Absence of evidence is never reported as evidence of absence. Each of these is a
|
|
@@ -127,65 +201,68 @@ distinct, printed state (spec §15), never collapsed into a verdict:
|
|
|
127
201
|
| `UNVERIFIED (unreachable)` | dead link, Wayback miss |
|
|
128
202
|
| `UNVERIFIED (reached, no text extracted)` | 200 answered, nothing readable came back |
|
|
129
203
|
| `UNVERIFIED (network not permitted)` | `permissions.network = deny` |
|
|
130
|
-
| `UNVERIFIED (provider unavailable)` | API down
|
|
204
|
+
| `UNVERIFIED (provider unavailable)` | API down, rate limited after backoff, or answering with a page instead of a record |
|
|
131
205
|
| `UNVERIFIED (not in bibliographic indexes)` | web page, blog, report — indexes do not cover it, so absence proves nothing |
|
|
132
206
|
| `AMBIGUOUS` | several plausible records, all listed |
|
|
133
207
|
| `NEI` | the source was read and neither supports nor contradicts |
|
|
134
208
|
| `PARAGRAPH-SCOPED` | the citation covers a paragraph; each sentence is judged separately |
|
|
135
|
-
| `UNSUPPORTED CITATION STYLE` |
|
|
209
|
+
| `UNSUPPORTED CITATION STYLE` | reserved; no detected style produces it in v0.2 |
|
|
136
210
|
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
teardown); fixed in this release — 20 of 20 piped runs exit `1` ([live runs](docs/eval/2026-09-12-v0.1-live.md)).
|
|
211
|
+
An earlier build sometimes aborted with `134` after printing a complete report (ONNX
|
|
212
|
+
runtime teardown); fixed in v0.1 — 20 of 20 piped runs exit `1`
|
|
213
|
+
([live runs](docs/eval/2026-09-12-v0.1-live.md)).
|
|
141
214
|
|
|
142
|
-
##
|
|
215
|
+
## Known limitations
|
|
143
216
|
|
|
144
|
-
- **Numeric citation markers only** — `[12]`, `[12,15]`, `[12-15]`; an author–year
|
|
145
|
-
citation is listed as `UNSUPPORTED CITATION STYLE`, not judged.
|
|
146
217
|
- **Superscript citations only when the PDF marks them as superscript**; a PDF that
|
|
147
218
|
draws them as ordinary digits loses them, and `km²` can be read as `[2]`.
|
|
148
219
|
- **Abstract fallback**: when only an abstract is reachable the verdict is labelled
|
|
149
220
|
`LOW CONFIDENCE (abstract only)`. Three sentences is not a source.
|
|
150
|
-
- **
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
- **
|
|
155
|
-
|
|
156
|
-
|
|
221
|
+
- **An unnumbered two-column bibliography is cut at line breaks** (RoBERTa's ACL
|
|
222
|
+
list: 103 entries for ~50, half of them with no author), so author-year pairing over
|
|
223
|
+
such a list resolves few items — 15 claims where a rejoined list would give 65. The
|
|
224
|
+
largest open item for author-year pairing.
|
|
225
|
+
- **A reference list without a `References` heading** is read by its shape in paged
|
|
226
|
+
formats — the last contiguous run of numbered paragraphs — which recovers only that
|
|
227
|
+
last block (AlphaFold: 17 of 84 entries) and drops the prose printed before a
|
|
228
|
+
block's first entry.
|
|
229
|
+
- **The 0 % false-ghost rate is measured on 274 hand-built references**, and rule 3
|
|
230
|
+
is paid for in recall: a fabricated `First Last and First Last. Title. Venue.` with
|
|
231
|
+
no year between the names and the title is reported as `not in bibliographic
|
|
232
|
+
indexes`, not as a ghost, and a proceedings-*volume* record can accept a fabricated
|
|
233
|
+
paper cited into that volume as `RESOLVED (low confidence)` — the set's one
|
|
234
|
+
fabricated acceptance (0.9 %) ([details](docs/eval/2026-09-12-ghosts.md)).
|
|
235
|
+
- **A cold run of a long bibliography is still serial** through the providers (129
|
|
236
|
+
references: minutes). Only the TUI runs several documents' network stages at once.
|
|
157
237
|
- **A refused or failed browser install is reported, not hidden**: the source is
|
|
158
238
|
`UNVERIFIED (blocked, browser not permitted)` with the install log, never silently
|
|
159
239
|
counted as unreachable.
|
|
160
240
|
- **Coverage is not perfectly reproducible**: two runs minutes apart can read a
|
|
161
241
|
different number of sources, depending on which providers answered.
|
|
162
|
-
- The
|
|
163
|
-
in **v0.3**.
|
|
242
|
+
- The LLM judge and `--summarize` arrive in **v0.3**.
|
|
164
243
|
|
|
165
244
|
## Speed
|
|
166
245
|
|
|
167
|
-
Apple Silicon Mac, models already downloaded
|
|
246
|
+
Apple Silicon Mac, models already downloaded:
|
|
168
247
|
|
|
169
248
|
| document | first run | cached re-run |
|
|
170
249
|
|---|---|---|
|
|
171
|
-
| 1-page markdown draft, 7 references | 79.3 s | 15.8 s
|
|
172
|
-
| 19-page arXiv PDF, 68 references | 12 m 57 s | 3 m 42 s |
|
|
173
|
-
|
|
174
|
-
* the draft's cached re-run followed the earlier of the two recorded cold runs; the live doc keeps both.
|
|
250
|
+
| 1-page markdown draft, 7 references | 79.3 s (v0.1 cold) | **1.35 s** ([v0.2](docs/eval/2026-09-15-v0.2-live.md)) — 15.8 s in v0.1 |
|
|
251
|
+
| 19-page arXiv PDF, 68 references | 12 m 57 s | 3 m 42 s ([v0.1](docs/eval/2026-09-12-v0.1-live.md)) |
|
|
175
252
|
|
|
176
|
-
The **first ever** run also downloads about 250 MB of ONNX models.
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
read is fetched again — the PDF re-run above
|
|
181
|
-
|
|
253
|
+
The **first ever** run also downloads about 250 MB of ONNX models. Since v0.2 a
|
|
254
|
+
cached re-run asks the network for nothing: reference resolution and the retraction
|
|
255
|
+
check are cached (resolutions 30 days; a retraction hit 30 days, a miss 7 days), the
|
|
256
|
+
fetched text for 7 days, and chunks and verdicts for as long as the text is unchanged. The models are still loaded, and any source whose text
|
|
257
|
+
has expired or was never read is fetched again — the PDF re-run above (v0.1) still
|
|
258
|
+
spent 22 s fetching and re-scored 3 of 102 claims.
|
|
182
259
|
|
|
183
260
|
## Looking inside the cache
|
|
184
261
|
|
|
185
262
|
Everything proofpath fetches, embeds and decides lands in one plain SQLite file:
|
|
186
263
|
|
|
187
264
|
```bash
|
|
188
|
-
proofpath cache # where it is
|
|
265
|
+
proofpath cache # where it is, what it holds, how many lookups it remembers
|
|
189
266
|
proofpath cache ls # sources, chunk/verdict counts, text expiry
|
|
190
267
|
proofpath cache show <id> # one source's chunks and verdicts
|
|
191
268
|
proofpath cache clear --expired
|
|
@@ -193,8 +270,8 @@ proofpath cache clear --expired
|
|
|
193
270
|
|
|
194
271
|
Open `proofpath cache path` in [DB Browser for SQLite](https://sqlitebrowser.org/),
|
|
195
272
|
TablePlus or DBeaver — plain tables, no extension. Raw publisher text expires after
|
|
196
|
-
7 days; verdicts keep the passage they quote
|
|
197
|
-
fetch URL|DOI` run either half on its own.
|
|
273
|
+
7 days; verdicts keep the passage they quote; a provider outage is never stored.
|
|
274
|
+
`proofpath resolve REF` and `proofpath fetch URL|DOI` run either half on its own.
|
|
198
275
|
|
|
199
276
|
## Optional LLM judge — arrives in v0.3
|
|
200
277
|
|
|
@@ -209,9 +286,10 @@ comes from the environment or a `.env` file, never from config, and is never pri
|
|
|
209
286
|
| What | Set | Result |
|
|
210
287
|
|---|---|---|
|
|
211
288
|
| Retrieval + entailment | SciFact dev, 340 pairs | 0.609 accuracy, 0.597 macro-F1, against a 0.406 trivial baseline ([details](docs/eval/2026-09-12-scifact-dev.md)) |
|
|
212
|
-
| Reference resolution | hand-built ghost set,
|
|
289
|
+
| Reference resolution | hand-built ghost set, 274 references | 0.0 % false-ghost, 99.1 % ghost recall ([details](docs/eval/2026-09-12-ghosts.md)) |
|
|
213
290
|
| Source access | 50 DOIs | 72 % full text, 18 % abstract only, 10 % nothing ([details](docs/eval/2026-09-11-coverage.md)) — a real biomedical paper in the live runs reached 33 % full text |
|
|
214
|
-
| Citation pairing | 61 hand-built passages | 0.
|
|
291
|
+
| Citation pairing, numeric | 61 hand-built passages | 0.99 ([details](docs/eval/2026-09-11-pairing.md)) |
|
|
292
|
+
| Citation pairing, author-year | 55 hand-built passages, 83 expectations | 0.940 ([details](docs/eval/2026-09-12-pairing-author-year.md)) |
|
|
215
293
|
|
|
216
294
|
Published SciFact results sit around 70–75 F1, not 95. Nothing is tuned on a test
|
|
217
295
|
split, and no number is quoted without the run that produced it.
|