sciwrite-lint 0.2.1__tar.gz → 0.3.0__tar.gz

{sciwrite_lint-0.2.1/sciwrite_lint.egg-info → sciwrite_lint-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sciwrite-lint
-Version: 0.2.1
+Version: 0.3.0
 Summary: A linter for scientific manuscripts — reference verification, consistency checking, and structural validation.
 Author: Sergey Samsonau
 Maintainer-email: Authentic Research Partners LLC <info@arpconnect.com>
@@ -9,6 +9,7 @@ Project-URL: Homepage, https://arpconnect.com
 Project-URL: Repository, https://github.com/authentic-research-partners/sciwrite-lint
 Project-URL: Documentation, https://github.com/authentic-research-partners/sciwrite-lint#readme
 Project-URL: Issues, https://github.com/authentic-research-partners/sciwrite-lint/issues
+Project-URL: Paper, https://arxiv.org/abs/2604.08501
 Keywords: linter,latex,bibtex,pdf,scientific-writing,academic-writing,citation-checker,reference-verification,claim-verification,manuscript,peer-review,arxiv,vllm,qwen
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
@@ -40,17 +41,22 @@ Requires-Dist: numpy>=2.0
 Requires-Dist: sqlite-vec>=0.1.6
 Requires-Dist: torch>=2.5
 Requires-Dist: transformers>=4.50
-Requires-Dist: Pillow>=12.0
-Requires-Dist: pypdf>=6.0
+Requires-Dist: Pillow>=12.2
+Requires-Dist: pypdf>=6.10.2
 Requires-Dist: pypdfium2>=5.0
 Requires-Dist: pdf2image>=1.17
+Requires-Dist: lxml>=6.1
+Requires-Dist: pypandoc-binary>=1.15
+Requires-Dist: rfc3986>=2.0
 Provides-Extra: dev
-Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest>=9.0.3; extra == "dev"
 Requires-Dist: pytest-xdist>=3.0; extra == "dev"
 Dynamic: license-file
 
 # sciwrite-lint
 
+[![arXiv](https://img.shields.io/badge/arXiv-2604.08501-b31b1b.svg)](https://arxiv.org/abs/2604.08501)
+
 A linter for scientific manuscripts. Checks that your references exist, your metadata is accurate, your citations support the claims you make, and your cited papers' own bibliographies are real. Works on LaTeX and PDF. Runs entirely on your machine. Produces a **SciLint Score**.
 
 The only open-source tool that combines reference verification, claim checking, manuscript consistency, figure analysis, and bibliography verification in one pipeline — powered entirely by a local LLM on your own GPU.
@@ -63,7 +69,7 @@ AI writing tools produce text that *looks* like good science — fluent prose, c
 
 ## Features
 
-**22 automated checks:**
+**23 automated checks:**
 
 ### Reference verification
 - **Do your references exist?** — checked against CrossRef, OpenAlex, Semantic Scholar, Open Library, and Library of Congress
@@ -72,7 +78,7 @@ AI writing tools produce text that *looks* like good science — fluent prose, c
 - **Robust matching** — when references lack DOIs, a multi-signal matching engine scores candidates across title, author, year, and venue (handles the metadata errors that LLMs routinely introduce)
 
 ### Claim verification (local LLM)
-- **Do cited papers support your claims?** — downloads full text from 8 open-access sources (arXiv, Semantic Scholar, OpenAlex, PubMed Central, Europe PMC, Unpaywall, bioRxiv/medRxiv, CORE), parses via GROBID, embeds sections, and verifies each claim against the actual source text
+- **Do cited papers support your claims?** — downloads full text from 14 open-access sources (arXiv, Semantic Scholar, OpenAlex, PubMed Central, Europe PMC, Unpaywall, bioRxiv/medRxiv, NBER, RePEc/IDEAS, HAL, ERIC, NASA ADS, OSF Preprints, CORE), parses via GROBID, embeds sections, and verifies each claim against the actual source text
 - **What role does each citation play?** — classifies citation purpose (evidence, contrast, method, attribution, context…) with graduated weights: an unsupported *evidence* citation is serious; an unsupported *context* citation barely matters
 - **Are your references' own bibliographies real?** — batch-checks cited papers' reference lists for existence, metadata accuracy, and retraction. Papers built on fabricated evidence are flagged
 
@@ -85,6 +91,7 @@ AI writing tools produce text that *looks* like good science — fluent prose, c
 - **Abstract–body alignment** — abstract makes factual claims the body contradicts
 - **Statistical reporting** — p-values vs. their verbal interpretation
 - **Structure promises** — contributions promised in the introduction but never delivered
+- **Prose quality** — syntactic grammar errors and semantic word-choice mistakes (e.g. "object" where "purpose" is intended). Per-sentence review with paragraph context; severity driven by the model's confidence
 
 ### Figure checks (vision model + LLM)
 - **Caption vs. content** — does the caption match what the figure actually shows?
@@ -125,7 +132,7 @@ sciwrite-lint contributions paper.pdf --format json
 ### Two audiences
 
 - **Humans** — colored terminal output with severity levels, locations, and explanations. Decide in seconds whether each finding is real
-- **AI writing agents** — `--format json` output with structured fields (`level`, `rule_id`, `message`, `context`). Run sciwrite-lint in a write → check → fix → recheck loop. Configurable exit codes for CI integration
+- **AI writing agents** — `--format json` output with structured fields (`level`, `rule_id`, `message`, `context`). Run sciwrite-lint in a write → check → fix → recheck loop. Standard `ruff`/`mypy`-style exit codes: `0` clean, `1` findings at error level, `2` tool error. Python-API consumers can `except sciwrite_lint.LLMConnectionError` (typed exception hierarchy under `SciWriteLintError`) instead of matching on error strings
 
 ### Optimizations
 
@@ -137,7 +144,7 @@ Three models — a vision model (Qwen3-VL-2B default, or 8B FP8 via `--vision-ba
 - **Adaptive embedding batches** — token-aware batch sizing gives ~50x speedup over CPU while staying within VRAM limits
 - **Batch-staged multi-paper pipeline** — when checking 2+ papers, GPU models load once per batch (vision/embedding/cited-vision) and vLLM/network stages run concurrently, giving a meaningful speedup over sequential per-paper runs. Tune via `--concurrency` (default 2, validated up to 4 on a single consumer GPU)
 - **Phased API resolution** — citations flow through OpenAlex → Semantic Scholar → CrossRef → Open Library/LoC, each phase only processing what previous phases didn't resolve
-- **8-source full-text cascade** — early exit on first successful download across arXiv, Semantic Scholar, OpenAlex, PubMed Central, Europe PMC, Unpaywall, bioRxiv, CORE
+- **14-source full-text cascade** — early exit on first successful download across arXiv, Semantic Scholar, OpenAlex, PubMed Central, Europe PMC, Unpaywall, bioRxiv, NBER, RePEc/IDEAS, HAL, ERIC, NASA ADS, OSF Preprints, CORE
 - **Live monitoring** *(advanced)* — `sciwrite-lint containers monitor` shows service health, VRAM usage, and KV cache utilization in a terminal dashboard
 
 Full pipeline on a 50-reference paper: ~30 minutes initial (dominated by downloads and claim verification), minutes on cached reruns. On native Linux, the pipeline automatically swaps vLLM containers to free GPU for embedding and vision stages (~50x faster than CPU). *(Native Linux GPU swap is preliminary — tested on WSL2 only; expected to work, may need minor fixes.)*
@@ -154,10 +161,36 @@ uv tool install sciwrite-lint --python 3.13
 
 uv downloads Python 3.13 automatically (does not affect any Python you may already have) and installs sciwrite-lint as a globally available command.
 
+### Reproducible environment (optional)
+
+`pyproject.toml` declares lower bounds for dependencies, so `uv tool install` / `pip install` may resolve newer versions as upstream releases ship. If you want the exact versions the maintainer develops and tests against — for reproducing benchmarks, debugging version-drift issues, or CI — install from [`requirements-pinned.txt`](requirements-pinned.txt) instead.
+
+**What's in it:** top-level packages declared in `pyproject.toml` pinned to `==` versions. Transitive dependencies are not pinned — pip resolves them within each top-level package's own constraints. This is a deliberate middle ground: it locks the packages sciwrite-lint actually imports and tests against, without over-constraining the dep tree.
+
+```bash
+# Clone the public repo (required — uv tool install can't read a local file)
+git clone https://github.com/authentic-research-partners/sciwrite-lint.git
+cd sciwrite-lint
+
+# With uv (recommended — fast, isolated, manages Python itself)
+uv venv --python 3.13 .venv
+source .venv/bin/activate
+uv pip install -r requirements-pinned.txt
+uv pip install -e . --no-deps    # install sciwrite-lint source; deps already pinned
+
+# Or with plain pip
+python3.13 -m venv .venv
+source .venv/bin/activate
+pip install -r requirements-pinned.txt
+pip install -e . --no-deps
+```
+
+Activate the venv in each new terminal with `source .venv/bin/activate`. For most users, the plain `uv tool install sciwrite-lint` above is simpler — use the pinned file only when you need exact version reproduction.
+
 ## Setup
 
 ```bash
-sciwrite-lint init                     # scaffold .sciwrite-lint.toml + references/ + local_pdfs/
+sciwrite-lint init                     # scaffold .sciwrite-lint.toml + references/ + local_pdfs/ + local_web/
 sciwrite-lint config set-email you@example.com  # required for Unpaywall + Retraction Watch
 sciwrite-lint containers start         # start GROBID + vLLM (needs GPU for vLLM)
 sciwrite-lint containers monitor       # live dashboard: service health, VRAM, KV cache
@@ -167,9 +200,45 @@ sciwrite-lint containers monitor       # live dashboard: service health, VRAM, K
 
 `init` detects `.tex` files and their `.bib` references and generates a `.sciwrite-lint.toml` config. Review to confirm the right files were detected.
 
-**Paywalled references:** drop PDFs into `local_pdfs/` with filenames matching the reference title. The tool fuzzy-matches filenames against your `.bib` titles and uses local copies instead of downloading.
+**Providing references manually — two drop folders.** sciwrite-lint reads two local directories before going to the open-access waterfall, kept separate because the folder itself signals how much to trust the source:
+
+- `local_pdfs/` — **academic sources**. Accepts `.pdf` (primary) and `.md` summaries you've written yourself. Use this for paywalled papers, for OA papers whose publisher requires a browser (Cloudflare/JS walls, Cell, Springer, some PMC pages), and for anything you'd cite as peer-reviewed evidence.
+- `local_web/` — **web captures**. Accepts `.md` (hand-written or pre-extracted) and `.mhtml` / `.mht` saved from a browser via **File → Save As → Webpage (Single File)**. Use this for JavaScript-heavy pages where a headless HTTP fetch would only see the pre-hydration shell; MHTML captures the rendered DOM post-JS. sciwrite-lint converts MHTML to markdown at ingest using the same trafilatura extractor it uses for live web fetches, so the output format is consistent either way. Anything landing here is classified `local_type=web_page` — softer evidence than a peer-reviewed PDF.
+
+Name files by the reference title, or prefix them with the citekey:
+
+```
+local_pdfs/                                     local_web/
+├── States of Curiosity … Learning.pdf          ├── dewey1910_How_We_Think.mhtml
+├── Mind in Society.pdf                         ├── earthwatch2025_About.md
+├── vanfraassen1980_The_Scientific_Image.pdf    └── bca2026_Research_Teachers.md
+└── dewey2016_summary.md   # reader-written
+```
+
+Matching runs in two passes for both folders. **(1) Citekey prefix:** if the filename begins with a known bib citekey (case-insensitive, e.g. `vanfraassen1980_…`), the file is matched directly to that reference — no fuzzy scoring. This is the authoritative path and works reliably for archives that follow a `{citekey}_{Title}.<ext>` convention, including short titles where length differences would otherwise penalize the score. **(2) Fuzzy title:** filenames without a recognized citekey prefix fall through to fuzzy matching against `.bib` titles (threshold 0.80). Academic matches win over web matches when the same reference is present in both folders — PDF trumps web capture. Matched files are copied into the workspace; on the next run the reference upgrades from T2 to T1 and goes through GROBID parsing (for PDFs) or direct read (for `.md`) during claim verification.
 
-**Optional API keys** increase rate limits for Semantic Scholar, NCBI, and CORE:
+**Per-paper curated archives.** Each paper can point both drop folders at its own directories in its `[[papers]]` block:
+
+```toml
+[[papers]]
+name = "my-paper"
+file_path = "papers/my-paper/my-paper.tex"
+bib = "papers/my-paper/my-paper.bib"
+local_pdfs_dir = "papers/my-paper/Sources/full_text"      # academic archive
+local_web_dir  = "papers/my-paper/Sources/full_text_web"  # web captures
+```
+
+If you don't set them explicitly, sciwrite-lint auto-detects sibling `Sources/full_text/` and `Sources/full_text_web/` directories next to `file_path`. Resolution order per paper: explicit override → auto-detected `Sources/full_text{,_web}/` → project-wide default.
+
+**Refreshing a source file in place.** sciwrite-lint records the SHA-256 of each drop-folder file when it first ingests it. On every subsequent run the hash is re-checked: if you overwrite `dewey1910.pdf` with a corrected scan (or re-capture an MHTML, or edit a markdown summary) under the same filename, the hash changes and the tool automatically re-copies the file into the workspace — triggering a fresh GROBID parse and embedding for PDFs, a fresh MHTML-to-markdown conversion for web captures. You don't need `--fresh`, and you don't need to rename files. Unchanged files cost a single hash read per run and are otherwise skipped.
+
+**Footnote URLs (`\footnote{\url{…}}`) get verified too.** Papers often cite informational web pages inline as a footnote URL rather than through a formal `.bib` entry — organisation pages, program descriptions, press releases, FAQs. sciwrite-lint matches every such URL to a `.md` file in `local_web_dir` by reading a `Source: https://…` (or `Source URL: …`) header from the first 20 lines of each file. The matched capture becomes a T1 source for claim verification — same as a cited reference. Browser-saved `.mhtml` gets this header written automatically at ingest; for hand-authored `.md` you add one line at the top. See [docs/local-sources.md](https://github.com/authentic-research-partners/sciwrite-lint/blob/main/docs/local-sources.md) for details and examples.
+
+**Finding what to download:** `sciwrite-lint verify` lists T2 references that are confirmed open access with direct URLs — open each in your browser and save to `local_pdfs/` (or `local_web/` if the "paper" is actually a blog post or documentation page that the publisher serves as HTML). For paywalled papers, use your institution's library access.
+
+**Fetched-PDF match validation.** Every downloaded PDF goes through a multi-signal match gate (title similarity, author surname, DOI, year, plus hard rejects for common bibliographic-record landing pages) before being cached. Wrong-paper downloads from API mismatches and template landing pages are rejected and deleted; the tool moves on to the next open-access source. You can still drop the correct PDF into `local_pdfs_dir` to skip the waterfall entirely.
+
+**API keys** increase rate limits for Semantic Scholar, NCBI, and CORE, and enable NASA ADS (required for that source):
 
 ```bash
 sciwrite-lint config show              # see what's configured
@@ -187,6 +256,7 @@ sciwrite-lint check                             # all papers (batch-staged when
 sciwrite-lint check --concurrency 4             # batch parallelism (default 2, validated up to 4)
 sciwrite-lint check paper.tex                   # text + LLM rules on a .tex file
 sciwrite-lint check paper.pdf                   # check a PDF (GROBID required)
+sciwrite-lint check --checks prose-quality paper.tex   # run only the listed checks (comma-separated)
 sciwrite-lint contributions --paper my_paper    # add contribution axes to SciLint Score
 sciwrite-lint contributions paper.pdf           # standalone file scoring
 ```
@@ -195,6 +265,8 @@ sciwrite-lint contributions paper.pdf           # standalone file scoring
 
 Use `--fresh` to start from scratch (backs up the existing workspace before recreating it).
 
+Use `--checks ID[,ID...]` to run only the listed checks (comma-separated), disabling the rest for this invocation. Useful for fast iteration during editing (`--checks prose-quality` runs just the prose review), for debugging a single check, or for scripting a two-pass workflow. Unknown check IDs fail loudly — run `sciwrite-lint checks` to list available IDs.
+
 ### Contribution axes (`sciwrite-lint contributions`)
 
 `contributions` computes 5 contribution axes from philosophy of science (Popper, Lakatos, Kitcher, Laudan, Mayo) and updates the SciLint Score. Requires vLLM.
@@ -263,12 +335,28 @@ Output formats: terminal (default) or `--format json`.
 - `sciwrite-lint checks` — list all checks
 - `sciwrite-lint <command> --help` — detailed usage for any command
 - [docs/services.md](https://github.com/authentic-research-partners/sciwrite-lint/blob/main/docs/services.md) — GROBID, vLLM, external APIs, configuration
+- [docs/local-sources.md](https://github.com/authentic-research-partners/sciwrite-lint/blob/main/docs/local-sources.md) — drop directories, filename conventions, `Source:` header for footnote URLs
 
 For contributors and advanced users:
 
 - [docs/evals.md](https://github.com/authentic-research-partners/sciwrite-lint/blob/main/docs/evals.md) — detection evaluation framework, adding test cases
 - [docs/calibration.md](https://github.com/authentic-research-partners/sciwrite-lint/blob/main/docs/calibration.md) — SciLint Score calibration against ground-truth papers
 
+## Citation
+
+If you use sciwrite-lint in your research, please cite the accompanying paper ([arXiv:2604.08501](https://arxiv.org/abs/2604.08501)):
+
+```bibtex
+@article{samsonau2026sciwritelint,
+  title   = {sciwrite-lint: Verification Infrastructure for the Age of Science Vibe-Writing},
+  author  = {Samsonau, Sergey V.},
+  journal = {arXiv preprint arXiv:2604.08501},
+  year    = {2026},
+}
+```
+
+GitHub's "Cite this repository" button (powered by [`CITATION.cff`](CITATION.cff)) will also generate this for you.
+
 ## License
 
 MIT

{sciwrite_lint-0.2.1 → sciwrite_lint-0.3.0}/README.md

@@ -1,5 +1,7 @@
 # sciwrite-lint
 
+[![arXiv](https://img.shields.io/badge/arXiv-2604.08501-b31b1b.svg)](https://arxiv.org/abs/2604.08501)
+
 A linter for scientific manuscripts. Checks that your references exist, your metadata is accurate, your citations support the claims you make, and your cited papers' own bibliographies are real. Works on LaTeX and PDF. Runs entirely on your machine. Produces a **SciLint Score**.
 
 The only open-source tool that combines reference verification, claim checking, manuscript consistency, figure analysis, and bibliography verification in one pipeline — powered entirely by a local LLM on your own GPU.
@@ -12,7 +14,7 @@ AI writing tools produce text that *looks* like good science — fluent prose, c
 
 ## Features
 
-**22 automated checks:**
+**23 automated checks:**
 
 ### Reference verification
 - **Do your references exist?** — checked against CrossRef, OpenAlex, Semantic Scholar, Open Library, and Library of Congress
@@ -21,7 +23,7 @@ AI writing tools produce text that *looks* like good science — fluent prose, c
 - **Robust matching** — when references lack DOIs, a multi-signal matching engine scores candidates across title, author, year, and venue (handles the metadata errors that LLMs routinely introduce)
 
 ### Claim verification (local LLM)
-- **Do cited papers support your claims?** — downloads full text from 8 open-access sources (arXiv, Semantic Scholar, OpenAlex, PubMed Central, Europe PMC, Unpaywall, bioRxiv/medRxiv, CORE), parses via GROBID, embeds sections, and verifies each claim against the actual source text
+- **Do cited papers support your claims?** — downloads full text from 14 open-access sources (arXiv, Semantic Scholar, OpenAlex, PubMed Central, Europe PMC, Unpaywall, bioRxiv/medRxiv, NBER, RePEc/IDEAS, HAL, ERIC, NASA ADS, OSF Preprints, CORE), parses via GROBID, embeds sections, and verifies each claim against the actual source text
 - **What role does each citation play?** — classifies citation purpose (evidence, contrast, method, attribution, context…) with graduated weights: an unsupported *evidence* citation is serious; an unsupported *context* citation barely matters
 - **Are your references' own bibliographies real?** — batch-checks cited papers' reference lists for existence, metadata accuracy, and retraction. Papers built on fabricated evidence are flagged
 
@@ -34,6 +36,7 @@ AI writing tools produce text that *looks* like good science — fluent prose, c
 - **Abstract–body alignment** — abstract makes factual claims the body contradicts
 - **Statistical reporting** — p-values vs. their verbal interpretation
 - **Structure promises** — contributions promised in the introduction but never delivered
+- **Prose quality** — syntactic grammar errors and semantic word-choice mistakes (e.g. "object" where "purpose" is intended). Per-sentence review with paragraph context; severity driven by the model's confidence
 
 ### Figure checks (vision model + LLM)
 - **Caption vs. content** — does the caption match what the figure actually shows?
@@ -74,7 +77,7 @@ sciwrite-lint contributions paper.pdf --format json
 ### Two audiences
 
 - **Humans** — colored terminal output with severity levels, locations, and explanations. Decide in seconds whether each finding is real
-- **AI writing agents** — `--format json` output with structured fields (`level`, `rule_id`, `message`, `context`). Run sciwrite-lint in a write → check → fix → recheck loop. Configurable exit codes for CI integration
+- **AI writing agents** — `--format json` output with structured fields (`level`, `rule_id`, `message`, `context`). Run sciwrite-lint in a write → check → fix → recheck loop. Standard `ruff`/`mypy`-style exit codes: `0` clean, `1` findings at error level, `2` tool error. Python-API consumers can `except sciwrite_lint.LLMConnectionError` (typed exception hierarchy under `SciWriteLintError`) instead of matching on error strings
 
 ### Optimizations
 
@@ -86,7 +89,7 @@ Three models — a vision model (Qwen3-VL-2B default, or 8B FP8 via `--vision-ba
 - **Adaptive embedding batches** — token-aware batch sizing gives ~50x speedup over CPU while staying within VRAM limits
 - **Batch-staged multi-paper pipeline** — when checking 2+ papers, GPU models load once per batch (vision/embedding/cited-vision) and vLLM/network stages run concurrently, giving a meaningful speedup over sequential per-paper runs. Tune via `--concurrency` (default 2, validated up to 4 on a single consumer GPU)
 - **Phased API resolution** — citations flow through OpenAlex → Semantic Scholar → CrossRef → Open Library/LoC, each phase only processing what previous phases didn't resolve
-- **8-source full-text cascade** — early exit on first successful download across arXiv, Semantic Scholar, OpenAlex, PubMed Central, Europe PMC, Unpaywall, bioRxiv, CORE
+- **14-source full-text cascade** — early exit on first successful download across arXiv, Semantic Scholar, OpenAlex, PubMed Central, Europe PMC, Unpaywall, bioRxiv, NBER, RePEc/IDEAS, HAL, ERIC, NASA ADS, OSF Preprints, CORE
 - **Live monitoring** *(advanced)* — `sciwrite-lint containers monitor` shows service health, VRAM usage, and KV cache utilization in a terminal dashboard
 
 Full pipeline on a 50-reference paper: ~30 minutes initial (dominated by downloads and claim verification), minutes on cached reruns. On native Linux, the pipeline automatically swaps vLLM containers to free GPU for embedding and vision stages (~50x faster than CPU). *(Native Linux GPU swap is preliminary — tested on WSL2 only; expected to work, may need minor fixes.)*
@@ -103,10 +106,36 @@ uv tool install sciwrite-lint --python 3.13
 
 uv downloads Python 3.13 automatically (does not affect any Python you may already have) and installs sciwrite-lint as a globally available command.
 
+### Reproducible environment (optional)
+
+`pyproject.toml` declares lower bounds for dependencies, so `uv tool install` / `pip install` may resolve newer versions as upstream releases ship. If you want the exact versions the maintainer develops and tests against — for reproducing benchmarks, debugging version-drift issues, or CI — install from [`requirements-pinned.txt`](requirements-pinned.txt) instead.
+
+**What's in it:** top-level packages declared in `pyproject.toml` pinned to `==` versions. Transitive dependencies are not pinned — pip resolves them within each top-level package's own constraints. This is a deliberate middle ground: it locks the packages sciwrite-lint actually imports and tests against, without over-constraining the dep tree.
+
+```bash
+# Clone the public repo (required — uv tool install can't read a local file)
+git clone https://github.com/authentic-research-partners/sciwrite-lint.git
+cd sciwrite-lint
+
+# With uv (recommended — fast, isolated, manages Python itself)
+uv venv --python 3.13 .venv
+source .venv/bin/activate
+uv pip install -r requirements-pinned.txt
+uv pip install -e . --no-deps    # install sciwrite-lint source; deps already pinned
+
+# Or with plain pip
+python3.13 -m venv .venv
+source .venv/bin/activate
+pip install -r requirements-pinned.txt
+pip install -e . --no-deps
+```
+
+Activate the venv in each new terminal with `source .venv/bin/activate`. For most users, the plain `uv tool install sciwrite-lint` above is simpler — use the pinned file only when you need exact version reproduction.
+
 ## Setup
 
 ```bash
-sciwrite-lint init                     # scaffold .sciwrite-lint.toml + references/ + local_pdfs/
+sciwrite-lint init                     # scaffold .sciwrite-lint.toml + references/ + local_pdfs/ + local_web/
 sciwrite-lint config set-email you@example.com  # required for Unpaywall + Retraction Watch
 sciwrite-lint containers start         # start GROBID + vLLM (needs GPU for vLLM)
 sciwrite-lint containers monitor       # live dashboard: service health, VRAM, KV cache
@@ -116,9 +145,45 @@ sciwrite-lint containers monitor       # live dashboard: service health, VRAM, K
 
 `init` detects `.tex` files and their `.bib` references and generates a `.sciwrite-lint.toml` config. Review to confirm the right files were detected.
 
-**Paywalled references:** drop PDFs into `local_pdfs/` with filenames matching the reference title. The tool fuzzy-matches filenames against your `.bib` titles and uses local copies instead of downloading.
+**Providing references manually — two drop folders.** sciwrite-lint reads two local directories before going to the open-access waterfall, kept separate because the folder itself signals how much to trust the source:
+
+- `local_pdfs/` — **academic sources**. Accepts `.pdf` (primary) and `.md` summaries you've written yourself. Use this for paywalled papers, for OA papers whose publisher requires a browser (Cloudflare/JS walls, Cell, Springer, some PMC pages), and for anything you'd cite as peer-reviewed evidence.
+- `local_web/` — **web captures**. Accepts `.md` (hand-written or pre-extracted) and `.mhtml` / `.mht` saved from a browser via **File → Save As → Webpage (Single File)**. Use this for JavaScript-heavy pages where a headless HTTP fetch would only see the pre-hydration shell; MHTML captures the rendered DOM post-JS. sciwrite-lint converts MHTML to markdown at ingest using the same trafilatura extractor it uses for live web fetches, so the output format is consistent either way. Anything landing here is classified `local_type=web_page` — softer evidence than a peer-reviewed PDF.
+
+Name files by the reference title, or prefix them with the citekey:
+
+```
+local_pdfs/                                     local_web/
+├── States of Curiosity … Learning.pdf          ├── dewey1910_How_We_Think.mhtml
+├── Mind in Society.pdf                         ├── earthwatch2025_About.md
+├── vanfraassen1980_The_Scientific_Image.pdf    └── bca2026_Research_Teachers.md
+└── dewey2016_summary.md   # reader-written
+```
+
+Matching runs in two passes for both folders. **(1) Citekey prefix:** if the filename begins with a known bib citekey (case-insensitive, e.g. `vanfraassen1980_…`), the file is matched directly to that reference — no fuzzy scoring. This is the authoritative path and works reliably for archives that follow a `{citekey}_{Title}.<ext>` convention, including short titles where length differences would otherwise penalize the score. **(2) Fuzzy title:** filenames without a recognized citekey prefix fall through to fuzzy matching against `.bib` titles (threshold 0.80). Academic matches win over web matches when the same reference is present in both folders — PDF trumps web capture. Matched files are copied into the workspace; on the next run the reference upgrades from T2 to T1 and goes through GROBID parsing (for PDFs) or direct read (for `.md`) during claim verification.
 
-**Optional API keys** increase rate limits for Semantic Scholar, NCBI, and CORE:
+**Per-paper curated archives.** Each paper can point both drop folders at its own directories in its `[[papers]]` block:
+
+```toml
+[[papers]]
+name = "my-paper"
+file_path = "papers/my-paper/my-paper.tex"
+bib = "papers/my-paper/my-paper.bib"
+local_pdfs_dir = "papers/my-paper/Sources/full_text"      # academic archive
+local_web_dir  = "papers/my-paper/Sources/full_text_web"  # web captures
+```
+
+If you don't set them explicitly, sciwrite-lint auto-detects sibling `Sources/full_text/` and `Sources/full_text_web/` directories next to `file_path`. Resolution order per paper: explicit override → auto-detected `Sources/full_text{,_web}/` → project-wide default.
+
+**Refreshing a source file in place.** sciwrite-lint records the SHA-256 of each drop-folder file when it first ingests it. On every subsequent run the hash is re-checked: if you overwrite `dewey1910.pdf` with a corrected scan (or re-capture an MHTML, or edit a markdown summary) under the same filename, the hash changes and the tool automatically re-copies the file into the workspace — triggering a fresh GROBID parse and embedding for PDFs, a fresh MHTML-to-markdown conversion for web captures. You don't need `--fresh`, and you don't need to rename files. Unchanged files cost a single hash read per run and are otherwise skipped.
+
+**Footnote URLs (`\footnote{\url{…}}`) get verified too.** Papers often cite informational web pages inline as a footnote URL rather than through a formal `.bib` entry — organisation pages, program descriptions, press releases, FAQs. sciwrite-lint matches every such URL to a `.md` file in `local_web_dir` by reading a `Source: https://…` (or `Source URL: …`) header from the first 20 lines of each file. The matched capture becomes a T1 source for claim verification — same as a cited reference. Browser-saved `.mhtml` gets this header written automatically at ingest; for hand-authored `.md` you add one line at the top. See [docs/local-sources.md](https://github.com/authentic-research-partners/sciwrite-lint/blob/main/docs/local-sources.md) for details and examples.
+
+**Finding what to download:** `sciwrite-lint verify` lists T2 references that are confirmed open access with direct URLs — open each in your browser and save to `local_pdfs/` (or `local_web/` if the "paper" is actually a blog post or documentation page that the publisher serves as HTML). For paywalled papers, use your institution's library access.
+
+**Fetched-PDF match validation.** Every downloaded PDF goes through a multi-signal match gate (title similarity, author surname, DOI, year, plus hard rejects for common bibliographic-record landing pages) before being cached. Wrong-paper downloads from API mismatches and template landing pages are rejected and deleted; the tool moves on to the next open-access source. You can still drop the correct PDF into `local_pdfs_dir` to skip the waterfall entirely.
+
+**API keys** increase rate limits for Semantic Scholar, NCBI, and CORE, and enable NASA ADS (required for that source):
 
 ```bash
 sciwrite-lint config show              # see what's configured
@@ -136,6 +201,7 @@ sciwrite-lint check                             # all papers (batch-staged when
 sciwrite-lint check --concurrency 4             # batch parallelism (default 2, validated up to 4)
 sciwrite-lint check paper.tex                   # text + LLM rules on a .tex file
 sciwrite-lint check paper.pdf                   # check a PDF (GROBID required)
+sciwrite-lint check --checks prose-quality paper.tex   # run only the listed checks (comma-separated)
 sciwrite-lint contributions --paper my_paper    # add contribution axes to SciLint Score
 sciwrite-lint contributions paper.pdf           # standalone file scoring
 ```
@@ -144,6 +210,8 @@ sciwrite-lint contributions paper.pdf           # standalone file scoring
 
 Use `--fresh` to start from scratch (backs up the existing workspace before recreating it).
 
+Use `--checks ID[,ID...]` to run only the listed checks (comma-separated), disabling the rest for this invocation. Useful for fast iteration during editing (`--checks prose-quality` runs just the prose review), for debugging a single check, or for scripting a two-pass workflow. Unknown check IDs fail loudly — run `sciwrite-lint checks` to list available IDs.
+
 ### Contribution axes (`sciwrite-lint contributions`)
 
 `contributions` computes 5 contribution axes from philosophy of science (Popper, Lakatos, Kitcher, Laudan, Mayo) and updates the SciLint Score. Requires vLLM.
@@ -212,12 +280,28 @@ Output formats: terminal (default) or `--format json`.
 - `sciwrite-lint checks` — list all checks
 - `sciwrite-lint <command> --help` — detailed usage for any command
 - [docs/services.md](https://github.com/authentic-research-partners/sciwrite-lint/blob/main/docs/services.md) — GROBID, vLLM, external APIs, configuration
+- [docs/local-sources.md](https://github.com/authentic-research-partners/sciwrite-lint/blob/main/docs/local-sources.md) — drop directories, filename conventions, `Source:` header for footnote URLs
 
 For contributors and advanced users:
 
 - [docs/evals.md](https://github.com/authentic-research-partners/sciwrite-lint/blob/main/docs/evals.md) — detection evaluation framework, adding test cases
 - [docs/calibration.md](https://github.com/authentic-research-partners/sciwrite-lint/blob/main/docs/calibration.md) — SciLint Score calibration against ground-truth papers
 
+## Citation
+
+If you use sciwrite-lint in your research, please cite the accompanying paper ([arXiv:2604.08501](https://arxiv.org/abs/2604.08501)):
+
+```bibtex
+@article{samsonau2026sciwritelint,
+  title   = {sciwrite-lint: Verification Infrastructure for the Age of Science Vibe-Writing},
+  author  = {Samsonau, Sergey V.},
+  journal = {arXiv preprint arXiv:2604.08501},
+  year    = {2026},
+}
+```
+
+GitHub's "Cite this repository" button (powered by [`CITATION.cff`](CITATION.cff)) will also generate this for you.
+
 ## License
 
 MIT

{sciwrite_lint-0.2.1 → sciwrite_lint-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "sciwrite-lint"
-version = "0.2.1"
+version = "0.3.0"
 description = "A linter for scientific manuscripts — reference verification, consistency checking, and structural validation."
 requires-python = ">=3.13,<3.14"
 license = "MIT"
@@ -42,10 +42,13 @@ dependencies = [
     "sqlite-vec>=0.1.6",
     "torch>=2.5",
     "transformers>=4.50",
-    "Pillow>=12.0",
-    "pypdf>=6.0",
+    "Pillow>=12.2",
+    "pypdf>=6.10.2",
     "pypdfium2>=5.0",
     "pdf2image>=1.17",
+    "lxml>=6.1",
+    "pypandoc-binary>=1.15",
+    "rfc3986>=2.0",
 ]
 
 [project.urls]
@@ -53,10 +56,11 @@ Homepage = "https://arpconnect.com"
 Repository = "https://github.com/authentic-research-partners/sciwrite-lint"
 Documentation = "https://github.com/authentic-research-partners/sciwrite-lint#readme"
 Issues = "https://github.com/authentic-research-partners/sciwrite-lint/issues"
+Paper = "https://arxiv.org/abs/2604.08501"
 
 [project.optional-dependencies]
 dev = [
-    "pytest>=8.0",
+    "pytest>=9.0.3",
     "pytest-xdist>=3.0",
 ]
 
@@ -105,5 +109,5 @@ DEP003 = ["sciwrite_lint", "evals", "eval_real_world"]
 plugins = ["pydantic.mypy"]
 [[tool.mypy.overrides]]
 # No py.typed marker or typeshed stubs available
-module = ["bibtexparser", "sqlite_vec", "pypdf", "pypdfium2", "pypdfium2.raw", "transformers", "pdf2image"]
+module = ["bibtexparser", "sqlite_vec", "pypdf", "pypdfium2", "pypdfium2.raw", "transformers", "pdf2image", "lxml", "lxml.html", "lxml.html.etree", "rfc3986", "pypandoc"]
 ignore_missing_imports = true

sciwrite_lint-0.3.0/sciwrite_lint/__init__.py

@@ -0,0 +1,28 @@
+"""sciwrite-lint: a linter for scientific manuscripts."""
+
+from sciwrite_lint.exceptions import LLMConnectionError, SciWriteLintError
+
+__version__ = "0.3.0"
+
+from sciwrite_lint.oa import (
+    DownloadResult,
+    FetchConfig,
+    SearchHit,
+    WebResult,
+    download_pdf,
+    fetch_web,
+    search_by_title,
+)
+
+__all__ = [
+    "DownloadResult",
+    "FetchConfig",
+    "LLMConnectionError",
+    "SciWriteLintError",
+    "SearchHit",
+    "WebResult",
+    "__version__",
+    "download_pdf",
+    "fetch_web",
+    "search_by_title",
+]

{sciwrite_lint-0.2.1 → sciwrite_lint-0.3.0}/sciwrite_lint/__main__.py

@@ -189,6 +189,17 @@ def main(argv: list[str] | None = None) -> int:
         default=None,
         help="Vision backend: transformers (2B, default) or vllm (8B FP8 on port 5002)",
     )
+    p_check.add_argument(
+        "--checks",
+        default=None,
+        metavar="ID[,ID...]",
+        help=(
+            "Run only the listed check IDs (comma-separated). Other checks "
+            "are disabled for this invocation. Example: "
+            "--checks prose-quality,dangling-cite. Run `sciwrite-lint checks` "
+            "to list available IDs."
+        ),
+    )
     p_check.set_defaults(func=run_check)
 
     # --- checks ---

{sciwrite_lint-0.2.1 → sciwrite_lint-0.3.0}/sciwrite_lint/_network.py

@@ -147,8 +147,15 @@ async def stream_with_limit(
     size we actually read.  Always enforces *max_bytes* on the
     decompressed data stream via ``aiter_bytes()``.
 
-    Returns a synthetic ``httpx.Response`` with the full body populated.
-    Raises ``ResponseTooLarge`` if the limit is exceeded.
+    Returns a synthetic ``httpx.Response`` with the full decompressed body
+    populated.  Raises ``ResponseTooLarge`` if the limit is exceeded.
+
+    Content-Encoding (and the now-stale Content-Length) is stripped from the
+    synthetic response's headers. ``aiter_bytes`` already decompressed the
+    body; leaving the header intact would cause httpx to try to decompress a
+    second time when the caller reads ``.content``, which fails with
+    ``DecodingError: incorrect header check`` on servers that gzip real
+    binary content (observed on ``mpra.ub.uni-muenchen.de`` serving PDFs).
     """
     async with client.stream("GET", url, **kwargs) as resp:  # type: ignore[arg-type]
         # Non-success responses are small — read eagerly for diagnostics
@@ -176,9 +183,16 @@ async def stream_with_limit(
             chunks.append(chunk)
 
     body = b"".join(chunks)
+    synthetic_headers = httpx.Headers(
+        [
+            (name, value)
+            for name, value in resp.headers.raw
+            if name.lower() not in (b"content-encoding", b"content-length")
+        ]
+    )
     return httpx.Response(
         status_code=resp.status_code,
-        headers=resp.headers,
+        headers=synthetic_headers,
         content=body,
         request=resp.request,
     )

{sciwrite_lint-0.2.1 → sciwrite_lint-0.3.0}/sciwrite_lint/api.py

@@ -1436,15 +1436,33 @@ async def _verify_web_resource(
             c.issues.append(
                 f"URL alive but content extraction failed: {web_result.error or 'unknown'}"
             )
+    elif web_result.blocked:
+        # Unverifiable: 4xx refusal, 5xx error, TLS/timeout/connection/
+        # decoding/protocol/oversized. The URL may still be valid — we
+        # just could not confirm. Downstream emits a WARN telling the
+        # user to verify manually; distinct from web_dead (ERROR).
+        c.api_match = "web_blocked"
+        c.api_source = "web"
+        c.api_data = {
+            "source": "web",
+            "url": resolved_url,
+            "status_code": web_result.status_code,
+        }
+        reason = web_result.error or "unknown reason"
+        c.issues.append(
+            f"Blocked by {reason}: {resolved_url} — "
+            f"unable to verify automatically, please check manually"
+        )
     else:
+        # Genuinely dead: server explicitly returned 404 or 410 (or the
+        # URL was structurally invalid). Safe to flag as ERROR — cited
+        # resource cannot be retrieved at this URL and likely cannot be
+        # retrieved anywhere at this URL.
         c.api_match = "web_dead"
         c.api_source = "web"
         c.api_data = {"source": "web", "url": resolved_url}
-        if web_result.error and web_result.status_code == 0:
-            # Connection/DNS/timeout error — include the reason, not just "0"
-            c.issues.append(f"Dead URL ({web_result.error}): {resolved_url}")
-        else:
-            c.issues.append(f"Dead URL (HTTP {web_result.status_code}): {resolved_url}")
+        reason = web_result.error or f"HTTP {web_result.status_code}"
+        c.issues.append(f"Dead URL ({reason}): {resolved_url}")
 
     return c.api_data
 

{sciwrite_lint-0.2.1 → sciwrite_lint-0.3.0}/sciwrite_lint/checks/full_paper_consistency.py

@@ -96,11 +96,15 @@ Return {{"issues": []}} if no genuine issues found.\
 _OVERHEAD_TOKENS = 2500
 
 # Output budget reserved for the JSON response. Matches the vLLM model
-# default (VLLM_MODELS["qwen3"]["max_tokens"]). FullPaperIssueList caps
-# the issues list at 5, which fits comfortably in this budget.
+# default (VLLM_MODELS["qwen3"]["max_tokens"]) — this is the response
+# portion only. ``llm_query`` adds the active thinking budget on top
+# when dispatching to vLLM. FullPaperIssueList caps the issues list at
+# 5, which fits comfortably in this budget.
 _OUTPUT_RESERVE_TOKENS = 2048
 
 # Worst-case thinking budget (medium preset, see THINKING_PRESETS).
+# Reserved in the context-window accounting alongside the output budget;
+# the sum matches what ``llm_query`` sends as ``max_tokens``.
 _THINKING_RESERVE_TOKENS = 1024
 
 # Rough chars-to-tokens ratio (conservative: overestimates tokens).