arxivdigest 0.4.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- arxivdigest-0.4.1/.dockerignore +6 -0
- arxivdigest-0.4.1/.github/workflows/ci.yml +31 -0
- arxivdigest-0.4.1/.github/workflows/release.yml +37 -0
- arxivdigest-0.4.1/.gitignore +6 -0
- arxivdigest-0.4.1/CHANGELOG.md +132 -0
- arxivdigest-0.4.1/Dockerfile +11 -0
- arxivdigest-0.4.1/LICENSE +21 -0
- arxivdigest-0.4.1/PKG-INFO +283 -0
- arxivdigest-0.4.1/README.md +235 -0
- arxivdigest-0.4.1/TODOS.md +51 -0
- arxivdigest-0.4.1/config.example.toml +21 -0
- arxivdigest-0.4.1/docs/audit-untouched-ideas-2026-07-13.md +304 -0
- arxivdigest-0.4.1/docs/images/concept-note.png +0 -0
- arxivdigest-0.4.1/docs/paperdigest-story.md +25 -0
- arxivdigest-0.4.1/docs/smoke-test-crosslinks.md +206 -0
- arxivdigest-0.4.1/docs/smoke-test-scaffold.md +105 -0
- arxivdigest-0.4.1/docs/superpowers/plans/2026-07-10-paperdigest.md +1714 -0
- arxivdigest-0.4.1/docs/superpowers/plans/2026-07-12-scaffold-subcommand.md +1241 -0
- arxivdigest-0.4.1/docs/superpowers/plans/2026-07-17-agents-brief-crosslinks.md +591 -0
- arxivdigest-0.4.1/docs/superpowers/smoke-test.md +65 -0
- arxivdigest-0.4.1/docs/superpowers/specs/2026-07-10-paperdigest-design.md +85 -0
- arxivdigest-0.4.1/docs/superpowers/specs/2026-07-11-figure-explanations.md +160 -0
- arxivdigest-0.4.1/docs/superpowers/specs/2026-07-12-scaffold-subcommand-design.md +146 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/README.md +57 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/logs/digest-final.log +18 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/logs/model-identity.txt +8 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/logs/scaffold-final.log +13 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/.gitignore +10 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/AGENTS.md +43 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/EXPERIMENTS.md +21 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/README.md +18 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/configs/base.yaml +12 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/configs/smoke.yaml +12 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/data/processed/.gitkeep +0 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/data/raw/.gitkeep +0 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/evaluate.py +43 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/experiments/exp001_smoke/README.md +19 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/experiments/runs.jsonl +0 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/logs/.gitkeep +0 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/notebooks/.gitkeep +0 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/pyproject.toml +19 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/reports/figures/.gitkeep +0 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/src/attention_is_all_you_need/__init__.py +0 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/src/attention_is_all_you_need/_config.py +34 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/src/attention_is_all_you_need/attention.py +74 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/src/attention_is_all_you_need/layers.py +73 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/src/attention_is_all_you_need/position.py +33 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/src/attention_is_all_you_need/tracking.py +41 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/src/attention_is_all_you_need/transformer.py +43 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/tests/test_smoke.py +45 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/train.py +43 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/adam optimizer.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/attention mask.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/attention mechanism.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/attention.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/auto-regressive.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/bleu score.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/byte-pair encoding.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/constituency parsing.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/convolutional neural network.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/decoder.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/dot product.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/dropout.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/embedding layer.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/embedding.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/encoder-decoder.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/encoder.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/feed-forward network.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/key.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/label smoothing.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/layer normalization.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/machine translation.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/matrix multiplication.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/multi-head attention.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/parallelization.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/perplexity.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/positional encoding.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/query key value.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/query.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/recurrent neural network.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/residual connection.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/scaled dot-product attention.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/scaled dot-product.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/self-attention.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/sequence transduction.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/softmax.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/transformer.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/value.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/weight sharing.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/word-piece.md +11 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/00 Overview.md +67 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/01 Scaled Dot-Product Attention.md +60 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/02 Multi-Head Attention.md +63 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/03 Positional Encoding.md +60 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/04 Transformer Architecture.md +104 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/05 Why Self-Attention.md +74 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/06 Self-Attention.md +60 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/fig1.png +0 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/fig2.png +0 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/fig3.png +0 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/fig4.png +0 -0
- arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/fig5.png +0 -0
- arxivdigest-0.4.1/examples/README.md +12 -0
- arxivdigest-0.4.1/paperdigest/__init__.py +14 -0
- arxivdigest-0.4.1/paperdigest/cli.py +203 -0
- arxivdigest-0.4.1/paperdigest/config.py +61 -0
- arxivdigest-0.4.1/paperdigest/digest.py +347 -0
- arxivdigest-0.4.1/paperdigest/extract.py +110 -0
- arxivdigest-0.4.1/paperdigest/fetch.py +143 -0
- arxivdigest-0.4.1/paperdigest/llm.py +278 -0
- arxivdigest-0.4.1/paperdigest/mermaid.py +101 -0
- arxivdigest-0.4.1/paperdigest/render.py +176 -0
- arxivdigest-0.4.1/paperdigest/scaffold.py +633 -0
- arxivdigest-0.4.1/paperdigest/templates.py +119 -0
- arxivdigest-0.4.1/pyproject.toml +55 -0
- arxivdigest-0.4.1/tests/conftest.py +60 -0
- arxivdigest-0.4.1/tests/fixtures/figure1.png +0 -0
- arxivdigest-0.4.1/tests/fixtures/paper.html +37 -0
- arxivdigest-0.4.1/tests/test_cli.py +198 -0
- arxivdigest-0.4.1/tests/test_config.py +115 -0
- arxivdigest-0.4.1/tests/test_digest.py +379 -0
- arxivdigest-0.4.1/tests/test_extract.py +42 -0
- arxivdigest-0.4.1/tests/test_fetch.py +187 -0
- arxivdigest-0.4.1/tests/test_llm.py +481 -0
- arxivdigest-0.4.1/tests/test_mermaid.py +97 -0
- arxivdigest-0.4.1/tests/test_render.py +338 -0
- arxivdigest-0.4.1/tests/test_scaffold.py +15 -0
- arxivdigest-0.4.1/tests/test_scaffold_cli.py +111 -0
- arxivdigest-0.4.1/tests/test_scaffold_stages.py +528 -0
- arxivdigest-0.4.1/tests/test_scaffold_write.py +103 -0
- arxivdigest-0.4.1/tests/test_templates.py +75 -0
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
name: CI
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main]
|
|
6
|
+
pull_request:
|
|
7
|
+
|
|
8
|
+
jobs:
|
|
9
|
+
test:
|
|
10
|
+
runs-on: ubuntu-latest
|
|
11
|
+
strategy:
|
|
12
|
+
matrix:
|
|
13
|
+
python-version: ["3.11", "3.12", "3.13"]
|
|
14
|
+
steps:
|
|
15
|
+
- uses: actions/checkout@v4
|
|
16
|
+
- uses: actions/setup-python@v5
|
|
17
|
+
with:
|
|
18
|
+
python-version: ${{ matrix.python-version }}
|
|
19
|
+
- run: pip install -e '.[dev]'
|
|
20
|
+
- run: python -m ruff check .
|
|
21
|
+
- run: python -m pytest -q --cov=paperdigest --cov-report=term-missing --cov-fail-under=93
|
|
22
|
+
|
|
23
|
+
build:
|
|
24
|
+
runs-on: ubuntu-latest
|
|
25
|
+
steps:
|
|
26
|
+
- uses: actions/checkout@v4
|
|
27
|
+
- uses: actions/setup-python@v5
|
|
28
|
+
with:
|
|
29
|
+
python-version: "3.12"
|
|
30
|
+
- run: pip install build
|
|
31
|
+
- run: python -m build
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
name: Release
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
tags: ["v*"]
|
|
6
|
+
|
|
7
|
+
jobs:
|
|
8
|
+
release:
|
|
9
|
+
runs-on: ubuntu-latest
|
|
10
|
+
permissions:
|
|
11
|
+
contents: write
|
|
12
|
+
id-token: write # required for PyPI trusted publishing
|
|
13
|
+
steps:
|
|
14
|
+
- uses: actions/checkout@v4
|
|
15
|
+
- uses: actions/setup-python@v5
|
|
16
|
+
with:
|
|
17
|
+
python-version: "3.12"
|
|
18
|
+
- run: pip install -e '.[dev]'
|
|
19
|
+
- run: python -m ruff check .
|
|
20
|
+
- run: python -m pytest -q --cov=paperdigest --cov-report=term-missing --cov-fail-under=93
|
|
21
|
+
- name: Check tag matches pyproject.toml version
|
|
22
|
+
run: |
|
|
23
|
+
TAG_VERSION="${GITHUB_REF_NAME#v}"
|
|
24
|
+
PROJECT_VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
|
|
25
|
+
if [ "$TAG_VERSION" != "$PROJECT_VERSION" ]; then
|
|
26
|
+
echo "Tag version ($TAG_VERSION) does not match pyproject.toml version ($PROJECT_VERSION)" >&2
|
|
27
|
+
exit 1
|
|
28
|
+
fi
|
|
29
|
+
- run: pip install build
|
|
30
|
+
- run: python -m build
|
|
31
|
+
- uses: softprops/action-gh-release@v2
|
|
32
|
+
with:
|
|
33
|
+
files: |
|
|
34
|
+
dist/*.whl
|
|
35
|
+
dist/*.tar.gz
|
|
36
|
+
- name: Publish to PyPI (trusted publishing — no stored credentials)
|
|
37
|
+
uses: pypa/gh-action-pypi-publish@release/v1
|
|
@@ -0,0 +1,132 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.4.1 — 2026-10-13
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
- Published to PyPI as **`arxivdigest`**: `pipx install arxivdigest` (or
|
|
7
|
+
`pip install arxivdigest`) installs the CLI without cloning the repo.
|
|
8
|
+
The `paperdigest` distribution name was unavailable — PyPI's name-similarity
|
|
9
|
+
guard reserves it for an unrelated, dormant `paper-digest` project.
|
|
10
|
+
The command remains `paperdigest`.
|
|
11
|
+
- The release workflow publishes to PyPI via GitHub **trusted publishing**
|
|
12
|
+
(OIDC) — no API tokens or passwords are stored on any machine.
|
|
13
|
+
|
|
14
|
+
### Changed
|
|
15
|
+
- README quick start leads with the packaged install; source-install
|
|
16
|
+
instructions moved below it.
|
|
17
|
+
|
|
18
|
+
### Fixed
|
|
19
|
+
- `__version__` now resolves package metadata under the `arxivdigest`
|
|
20
|
+
distribution name (with `paperdigest` as fallback for stale dev environments);
|
|
21
|
+
previously it looked only for the old name and reported `0.0.0` on clean
|
|
22
|
+
installs, which failed the release test gate.
|
|
23
|
+
|
|
24
|
+
## 0.4.0 — 2026-07-25
|
|
25
|
+
|
|
26
|
+
### Added
|
|
27
|
+
- `digest --project-dir` (or the `project_dir` config key) links notes to the
|
|
28
|
+
scaffolded project for the same paper: the overview gains a `## Build it`
|
|
29
|
+
section, and concept notes link the stub module(s) whose cited `§x.y`
|
|
30
|
+
paper sections match the concept, resolved by scanning the project on disk.
|
|
31
|
+
- Scaffold projects now include a generated `AGENTS.md` coding-agent brief
|
|
32
|
+
(dependency-ordered implementation plan, TODO counts, definition of done,
|
|
33
|
+
ground rules), assembled deterministically with no extra LLM call;
|
|
34
|
+
`scaffold --vault` links it back to the paper's Obsidian notes.
|
|
35
|
+
- Figures: paper figures are downloaded and explained by a vision-capable
|
|
36
|
+
model, with the image and explanation embedded in the matching concept
|
|
37
|
+
note. `--figures/--no-figures` and the `max_figures` config key (default
|
|
38
|
+
8) control it; backends without vision skip figures after one warning.
|
|
39
|
+
- `--version` flag; the scaffold failure debug dump now includes the version.
|
|
40
|
+
- `--quiet` flag on both commands to suppress progress output.
|
|
41
|
+
- `--max-input-chars`, `--max-tokens`, and `--cache-dir` CLI overrides for
|
|
42
|
+
their config keys.
|
|
43
|
+
- Concept explanations and module stubs are generated with up to 4 parallel
|
|
44
|
+
requests on cloud backends (local servers stay serial).
|
|
45
|
+
- CI and the release workflow enforce a 93% test-coverage floor.
|
|
46
|
+
- `max_tokens` config option: the LLM output-token cap is now set explicitly
|
|
47
|
+
on both backends instead of hard-coded (Anthropic) or left to the server
|
|
48
|
+
default (OpenAI-compatible). See Changed for the current default.
|
|
49
|
+
- Scaffold stages now get the same one-shot JSON repair round-trip as digest:
|
|
50
|
+
a malformed response is sent back to the model once for correction before
|
|
51
|
+
the run aborts.
|
|
52
|
+
- Tag-triggered release workflow: pushing a `v*` tag lints, tests, builds,
|
|
53
|
+
verifies the tag matches `pyproject.toml`, and attaches the wheel and sdist
|
|
54
|
+
to a GitHub Release.
|
|
55
|
+
- `examples/9b-attention-is-all-you-need/`: real, unedited output from a live
|
|
56
|
+
end-to-end run of both pipelines against arXiv:1706.03762 using a local 9B
|
|
57
|
+
model (Qwen3.5-9B-Instruct, NVFP4), with the run logs, as a worked example of
|
|
58
|
+
what paperdigest produces on self-hostable hardware.
|
|
59
|
+
|
|
60
|
+
### Changed
|
|
61
|
+
- Default `max_tokens` raised from 8192 to 16384 so mid-size local models
|
|
62
|
+
(e.g. a 9B) don't truncate long generated code files; still overridable
|
|
63
|
+
per run with `--max-tokens` or the config key.
|
|
64
|
+
|
|
65
|
+
### Fixed
|
|
66
|
+
- `__version__` is now derived from package metadata instead of a hard-coded
|
|
67
|
+
string that had drifted behind `pyproject.toml`.
|
|
68
|
+
- `~` in `vault`/`cache_dir` config values is now expanded.
|
|
69
|
+
- Concept-to-section matching is tiered (number-stripped exact, exact, then
|
|
70
|
+
longest-overlap containment), so short or numeric section titles can no
|
|
71
|
+
longer attach the wrong section's text to a concept.
|
|
72
|
+
- Generated code that imported sibling/infra modules by bare name (e.g.
|
|
73
|
+
`from attention import ...`) is now deterministically rewritten to the full
|
|
74
|
+
package path, so a scaffolded project's own smoke test imports correctly once
|
|
75
|
+
the project is installed.
|
|
76
|
+
- Generated `EXPERIMENTS.md` no longer fabricates benchmark result numbers;
|
|
77
|
+
targets are cited to the paper's own tables instead.
|
|
78
|
+
- The scaffold dependency scanner no longer lists the project's own modules as
|
|
79
|
+
pip dependencies in the generated `pyproject.toml`.
|
|
80
|
+
- The config loader generated into scaffolded projects now accepts
|
|
81
|
+
`key = value` as well as `key: value`, matching either separator the model
|
|
82
|
+
may emit. (This is the *generated project's* loader; paperdigest's own
|
|
83
|
+
`config.toml` remains strict TOML.)
|
|
84
|
+
|
|
85
|
+
## 0.3.0 — 2026-07-15
|
|
86
|
+
|
|
87
|
+
Robustness and safety hardening.
|
|
88
|
+
|
|
89
|
+
### Added
|
|
90
|
+
- Safety scan of all LLM-generated Python in `scaffold`: imports of
|
|
91
|
+
`subprocess`/`socket`/`ctypes`/`urllib`/`requests` and friends, calls to
|
|
92
|
+
`eval`/`exec`/`compile`/`__import__`, and dangerous attribute calls
|
|
93
|
+
(`os.system`, `pickle.load`, `shutil.rmtree`, ...) abort the pipeline
|
|
94
|
+
loudly with filename and line numbers.
|
|
95
|
+
- `--refresh` flag on `digest` and `scaffold` to re-fetch a paper, bypassing
|
|
96
|
+
the HTML cache.
|
|
97
|
+
- CI now runs ruff lint, reports test coverage, and verifies the package
|
|
98
|
+
builds (sdist + wheel).
|
|
99
|
+
|
|
100
|
+
### Changed
|
|
101
|
+
- LLM calls have a 300 s request timeout (previously unbounded).
|
|
102
|
+
- Responses truncated at the token limit now raise a clear error instead of
|
|
103
|
+
being silently written as complete notes.
|
|
104
|
+
- Only transient errors (connection, timeout, rate limit, 5xx) are retried;
|
|
105
|
+
permanent errors fail immediately with the original cause preserved.
|
|
106
|
+
- Module planning rejects duplicate filenames instead of silently
|
|
107
|
+
overwriting one generated module with another.
|
|
108
|
+
- Scaffold debug dumps are written to the cache directory (keyed by paper id
|
|
109
|
+
and stage) instead of the current working directory.
|
|
110
|
+
|
|
111
|
+
### Fixed
|
|
112
|
+
- HTML cache writes are atomic; an interrupted fetch can no longer leave a
|
|
113
|
+
corrupt cache file that is served forever.
|
|
114
|
+
- A failure partway through rendering removes the half-written paper folder
|
|
115
|
+
(existing glossary notes are never touched).
|
|
116
|
+
- Mermaid fence sanitization and validation now handle CRLF line endings and
|
|
117
|
+
trailing whitespace instead of silently skipping those blocks.
|
|
118
|
+
- Concurrent runs against the same paper report a clean error instead of a
|
|
119
|
+
traceback.
|
|
120
|
+
|
|
121
|
+
## 0.2.0 — 2026-07-12
|
|
122
|
+
|
|
123
|
+
- Mermaid diagrams by default with sanitizer and optional node-based
|
|
124
|
+
validation; `--diagram` option.
|
|
125
|
+
- JSON-mode structured output with trailing-junk-tolerant parsing.
|
|
126
|
+
- Prompt-echo rejection in generated scaffold code.
|
|
127
|
+
|
|
128
|
+
## 0.1.0 — 2026-07-11
|
|
129
|
+
|
|
130
|
+
- Initial release: `digest` (arXiv paper to plain-English Obsidian notes)
|
|
131
|
+
and `scaffold` (paper to git-initialized research project skeleton),
|
|
132
|
+
local-first with optional Anthropic/OpenAI backends.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Thad Reber
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,283 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: arxivdigest
|
|
3
|
+
Version: 0.4.1
|
|
4
|
+
Summary: Turn AI/ML papers into plain-English Obsidian notes and reproduction-ready project scaffolds — local-first, runs on a 9B model
|
|
5
|
+
Project-URL: Repository, https://github.com/thadreber-web/paperdigest
|
|
6
|
+
Author: Thad Reber
|
|
7
|
+
License: MIT License
|
|
8
|
+
|
|
9
|
+
Copyright (c) 2026 Thad Reber
|
|
10
|
+
|
|
11
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
12
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
13
|
+
in the Software without restriction, including without limitation the rights
|
|
14
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
15
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
16
|
+
furnished to do so, subject to the following conditions:
|
|
17
|
+
|
|
18
|
+
The above copyright notice and this permission notice shall be included in all
|
|
19
|
+
copies or substantial portions of the Software.
|
|
20
|
+
|
|
21
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
22
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
23
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
24
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
25
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
26
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
27
|
+
SOFTWARE.
|
|
28
|
+
License-File: LICENSE
|
|
29
|
+
Keywords: arxiv,llama-cpp,llm,local-llm,obsidian,papers,research
|
|
30
|
+
Classifier: Development Status :: 4 - Beta
|
|
31
|
+
Classifier: Intended Audience :: Science/Research
|
|
32
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
33
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
34
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
35
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
36
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
37
|
+
Requires-Python: >=3.11
|
|
38
|
+
Requires-Dist: anthropic>=0.40
|
|
39
|
+
Requires-Dist: beautifulsoup4>=4.12
|
|
40
|
+
Requires-Dist: httpx>=0.27
|
|
41
|
+
Requires-Dist: openai>=1.50
|
|
42
|
+
Requires-Dist: typer>=0.12
|
|
43
|
+
Provides-Extra: dev
|
|
44
|
+
Requires-Dist: pytest-cov; extra == 'dev'
|
|
45
|
+
Requires-Dist: pytest>=8; extra == 'dev'
|
|
46
|
+
Requires-Dist: ruff; extra == 'dev'
|
|
47
|
+
Description-Content-Type: text/markdown
|
|
48
|
+
|
|
49
|
+
# paperdigest
|
|
50
|
+
|
|
51
|
+
Read any AI/ML arXiv paper without having every piece of jargon memorized.
|
|
52
|
+
paperdigest turns a paper into a folder of plain-English, wikilinked Obsidian
|
|
53
|
+
notes — concept-by-concept explainers, Mermaid diagrams, equations walked
|
|
54
|
+
through line by line, and a vault-wide glossary that defines the terms as you
|
|
55
|
+
hit them.
|
|
56
|
+
|
|
57
|
+
<p align="center">
|
|
58
|
+
<img src="docs/images/concept-note.png" alt="A generated concept note open in Obsidian, showing a plain-English analogy, an auto-generated Mermaid flowchart, and the paper's equation" width="620">
|
|
59
|
+
</p>
|
|
60
|
+
<p align="center">
|
|
61
|
+
<em>One generated note in Obsidian: a plain-English analogy for the concept, an
|
|
62
|
+
auto-generated Mermaid diagram of the computation, and the paper's equation
|
|
63
|
+
walked through line by line. Everything below is produced from the arXiv source
|
|
64
|
+
— no hand-editing.</em>
|
|
65
|
+
</p>
|
|
66
|
+
|
|
67
|
+
Built for the reader who follows AI/ML advances but isn't a specialist in every
|
|
68
|
+
subfield: instead of bouncing between the PDF and ten browser tabs, you get the
|
|
69
|
+
paper's ideas explained in your own vault, each unfamiliar term a click away.
|
|
70
|
+
|
|
71
|
+
## See a full example
|
|
72
|
+
|
|
73
|
+
[`examples/9b-attention-is-all-you-need/`](examples/9b-attention-is-all-you-need)
|
|
74
|
+
is a complete run on *Attention Is All You Need*, checked into the repo — the
|
|
75
|
+
generated Obsidian [vault](examples/9b-attention-is-all-you-need/vault) (concept
|
|
76
|
+
notes + glossary), a
|
|
77
|
+
[scaffolded project](examples/9b-attention-is-all-you-need/scaffold-project), and
|
|
78
|
+
the [run logs](examples/9b-attention-is-all-you-need/logs). It was produced
|
|
79
|
+
entirely by a local 9B model — no cloud, no API key.
|
|
80
|
+
|
|
81
|
+
**Local-first.** By default paperdigest talks to a local OpenAI-compatible
|
|
82
|
+
server (llama.cpp, Ollama, vLLM) on your own machine — no API key, no cost.
|
|
83
|
+
Cloud backends (Anthropic, OpenAI) are opt-in for those who want them.
|
|
84
|
+
|
|
85
|
+
## Quick start (local, free)
|
|
86
|
+
|
|
87
|
+
Install the CLI (Python 3.11+; `pipx` keeps it in an isolated environment):
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
pipx install arxivdigest # or: pip install arxivdigest
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
(The PyPI package is named `arxivdigest`; the command you run is
|
|
94
|
+
`paperdigest`, unchanged.)
|
|
95
|
+
|
|
96
|
+
Have a llama.cpp server running (`llama-server -m model.gguf` serves
|
|
97
|
+
`http://localhost:8080/v1` by default), then:
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
paperdigest https://arxiv.org/abs/1706.03762 --vault ~/ObsidianVault
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Installing from a clone instead? `python3 -m venv .venv && .venv/bin/pip
|
|
104
|
+
install -e .`, then use `.venv/bin/paperdigest …` for every command below.
|
|
105
|
+
|
|
106
|
+
Non-default port or engine? Point at it:
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
# llama.cpp on another port
|
|
110
|
+
.venv/bin/paperdigest 1706.03762 --vault ~/ObsidianVault --base-url http://localhost:8001/v1
|
|
111
|
+
# Ollama (needs the real model name)
|
|
112
|
+
.venv/bin/paperdigest 1706.03762 --vault ~/ObsidianVault \
|
|
113
|
+
--base-url http://localhost:11434/v1 --model llama3.1
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
Output lands in `<vault>/Papers/<year>-<title-slug>/` plus new term notes in
|
|
117
|
+
`<vault>/Glossary/`. Re-running the same paper refuses to overwrite unless you
|
|
118
|
+
pass `--force`. Glossary term notes are never overwritten. Pass `--quiet` to
|
|
119
|
+
suppress progress output, or `--max-input-chars`/`--max-tokens`/`--cache-dir`
|
|
120
|
+
to override those config keys per run (both `digest` and `scaffold` accept
|
|
121
|
+
these). `paperdigest --version` prints the installed version — it is a
|
|
122
|
+
top-level flag, not available on the subcommands.
|
|
123
|
+
|
|
124
|
+
### Choosing a model & context window
|
|
125
|
+
|
|
126
|
+
Use a capable instruct model (~7B+); the pipeline needs structured JSON
|
|
127
|
+
output, and very small models fail cleanly with an `error:` message. Set
|
|
128
|
+
`max_input_chars` in `config.toml` to fit your model's context window (e.g.
|
|
129
|
+
24000 for ~8k-token contexts) — long papers are trimmed with a warning; set
|
|
130
|
+
`max_tokens` alongside it to cap output length for the same small context.
|
|
131
|
+
Big *thinking* models need the opposite: reasoning tokens count against the
|
|
132
|
+
output cap, so if a run aborts with a `truncated at max_tokens` error, raise
|
|
133
|
+
it (e.g. `--max-tokens 32768` — see `docs/smoke-test-crosslinks.md` for a
|
|
134
|
+
live example on a 122B).
|
|
135
|
+
|
|
136
|
+
## How it works
|
|
137
|
+
|
|
138
|
+
```mermaid
|
|
139
|
+
flowchart TD
|
|
140
|
+
A["arXiv paper<br>(HTML, fetched & cached)"] --> B["parse into<br>sections + figures"]
|
|
141
|
+
B --> C["outline (LLM):<br>concepts, jargon, TL;DR"]
|
|
142
|
+
C --> D["per-concept note (LLM):<br>plain-English + equation walk-through<br>+ embedded Mermaid / ASCII diagram"]
|
|
143
|
+
B --> E["figures explained<br>by a vision model"]
|
|
144
|
+
C --> F["glossary (LLM):<br>define jargon not already in your vault"]
|
|
145
|
+
D --> G["Obsidian vault:<br>Papers/ + Glossary/"]
|
|
146
|
+
E --> G
|
|
147
|
+
F --> G
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
The outline and glossary stages ask the server for JSON (`response_format:
|
|
151
|
+
json_object`) when it's supported — that's what makes small local models
|
|
152
|
+
reliable here. Each concept's diagram is generated as part of its note, then
|
|
153
|
+
sanitized deterministically before it's written; figures are embedded next to
|
|
154
|
+
the concept they illustrate.
|
|
155
|
+
|
|
156
|
+
### Diagrams: Mermaid or ASCII
|
|
157
|
+
|
|
158
|
+
Structural concepts get a diagram. The default is **Mermaid** (renders
|
|
159
|
+
natively in Obsidian); pass `--diagram ascii` (or set `diagram = "ascii"` in
|
|
160
|
+
`config.toml`) for plain-text diagrams that can never have syntax errors.
|
|
161
|
+
Small local models sometimes emit Mermaid with unquoted special characters in
|
|
162
|
+
node labels, so every diagram goes through a deterministic sanitizer that
|
|
163
|
+
auto-quotes them. Optionally, if `node` is on your PATH and the `mermaid` npm
|
|
164
|
+
package is importable from the directory you run paperdigest in
|
|
165
|
+
(`npm install mermaid jsdom global-jsdom`), each diagram is also parse-checked
|
|
166
|
+
and a warning names any note whose diagram still fails. No node? Validation is
|
|
167
|
+
skipped silently — the worst case is one diagram rendering as an error box in
|
|
168
|
+
Obsidian instead of a picture.
|
|
169
|
+
|
|
170
|
+
### Figures
|
|
171
|
+
|
|
172
|
+
By default, figures are downloaded from the paper and explained by a
|
|
173
|
+
vision-capable model, with the image and its explanation embedded together
|
|
174
|
+
in the matching concept note (or the overview, if unmatched). This needs a
|
|
175
|
+
vision-capable backend — for local llama.cpp, serve with an `--mmproj`
|
|
176
|
+
projector file. Backends without vision skip figures automatically after a
|
|
177
|
+
warning on the first figure, and the rest of the digest completes normally.
|
|
178
|
+
Pass `--no-figures` (or set `figures = false` in `config.toml`) to turn this
|
|
179
|
+
off; it's also the budget option on cloud backends, where images are
|
|
180
|
+
token-expensive. `max_figures` (default 8) caps how many figures per paper
|
|
181
|
+
are explained, by document order.
|
|
182
|
+
|
|
183
|
+
### Linking to a scaffolded project
|
|
184
|
+
|
|
185
|
+
Pass `--project-dir` (or set `project_dir` in `config.toml`) and notes gain a
|
|
186
|
+
`## Build it` section pointing at `<project-dir>/<year>-<title-slug>` — the
|
|
187
|
+
project `scaffold` would create for the same paper. If that project exists on
|
|
188
|
+
disk, concept notes also get a `*Build it: …*` line linking the stub module(s)
|
|
189
|
+
whose cited `§x.y` paper sections match the concept, so you can jump straight
|
|
190
|
+
from the explanation to the code that implements it.
|
|
191
|
+
|
|
192
|
+
## Scaffold a research project
|
|
193
|
+
|
|
194
|
+
Turn a paper into a standalone, git-initialized project skeleton (Cookiecutter-DS-style
|
|
195
|
+
layout, JSONL experiment tracking, stub modules mapped to paper sections):
|
|
196
|
+
|
|
197
|
+
```bash
|
|
198
|
+
.venv/bin/paperdigest scaffold https://arxiv.org/abs/1706.03762 --dest ~/projects
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
Output lands in `<dest>/<year>-<title-slug>/` with its own git history. The code is a
|
|
202
|
+
*skeleton*: module stubs with `TODO(paper §x.y)` markers plus a complete train/evaluate
|
|
203
|
+
harness and a toy smoke experiment — not a working reimplementation. Any generation
|
|
204
|
+
failure aborts loudly (nothing half-written); the raw model output is saved to
|
|
205
|
+
`scaffold-debug-<stage>.txt` for inspection. Re-running refuses to overwrite an
|
|
206
|
+
existing project unless you pass `--force`.
|
|
207
|
+
|
|
208
|
+
You don't need a big model: scaffolding is tested live end-to-end on a 9B via
|
|
209
|
+
llama.cpp (see `docs/smoke-test-scaffold.md`; the cross-linking features are
|
|
210
|
+
live-tested on a 122B via vLLM in `docs/smoke-test-crosslinks.md`). JSON
|
|
211
|
+
stages request structured
|
|
212
|
+
output (`response_format: json_object`) from the server when supported, which
|
|
213
|
+
is what makes small local models reliable here; generated code is validated
|
|
214
|
+
(syntax + prompt-echo rejection) before anything is written. Besides
|
|
215
|
+
`log_run()` for JSONL experiment tracking, the scaffolded `tracking.py` has
|
|
216
|
+
`get_logger()` for per-run log files under `logs/`.
|
|
217
|
+
|
|
218
|
+
**A note on trust:** every generated `.py` file is still LLM output. Before writing it,
|
|
219
|
+
paperdigest checks it parses, rejects leftover prompt-echo lines, and scans it for
|
|
220
|
+
dangerous calls (`subprocess`, `socket`, `eval`/`exec`, `os.system`, `pickle.load`,
|
|
221
|
+
`shutil.rmtree`, and similar) plus networking/FFI imports (`urllib`, `requests`,
|
|
222
|
+
`ctypes`, and similar) — aborting loudly if it finds any. That scan catches obviously
|
|
223
|
+
hazardous patterns, not everything — read the scaffolded code before you run it.
|
|
224
|
+
|
|
225
|
+
Every project also gets a generated `AGENTS.md` — a coding-agent brief with a
|
|
226
|
+
dependency-ordered implementation plan, a TODO count and cited paper sections per
|
|
227
|
+
module, a definition of done, and ground rules for the harness. It's assembled
|
|
228
|
+
deterministically from the same stage data, no extra LLM call. Pass `--vault` (or set
|
|
229
|
+
`vault` in your config file) and it links back to the paper's Obsidian notes so an
|
|
230
|
+
agent can read the plain-English explanation before touching code.
|
|
231
|
+
|
|
232
|
+
## Cloud backends (optional, cost money)
|
|
233
|
+
|
|
234
|
+
| Backend | Flags | Key (env var only) |
|
|
235
|
+
|---|---|---|
|
|
236
|
+
| Local (default) | `--base-url http://localhost:8080/v1 --model local` | none |
|
|
237
|
+
| OpenAI | `--backend openai --model gpt-5` | `OPENAI_API_KEY` |
|
|
238
|
+
| Anthropic | `--backend anthropic --model claude-sonnet-5` | `ANTHROPIC_API_KEY` |
|
|
239
|
+
|
|
240
|
+
Every LLM call is retried up to 2 more times with exponential backoff on
|
|
241
|
+
transient errors (connection drops, timeouts, rate limits, 5xx); malformed
|
|
242
|
+
JSON gets one repair round-trip; other errors fail immediately. Requests
|
|
243
|
+
time out after 300s. On cloud backends, concept and module-stub generation
|
|
244
|
+
runs a few requests in parallel to cut wall-clock time; against a local
|
|
245
|
+
server it stays serial, since a single llama.cpp instance generally can't
|
|
246
|
+
service concurrent requests.
|
|
247
|
+
|
|
248
|
+
## Docker
|
|
249
|
+
|
|
250
|
+
```bash
|
|
251
|
+
docker build -t paperdigest .
|
|
252
|
+
docker run --rm -v ~/ObsidianVault:/vault \
|
|
253
|
+
--add-host=host.docker.internal:host-gateway \
|
|
254
|
+
paperdigest https://arxiv.org/abs/1706.03762 \
|
|
255
|
+
--base-url http://host.docker.internal:8080/v1
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
(`--add-host` is needed on Linux to reach the host-side local server; for a
|
|
259
|
+
cloud backend pass `-e ANTHROPIC_API_KEY` or `-e OPENAI_API_KEY` instead.)
|
|
260
|
+
|
|
261
|
+
## Config
|
|
262
|
+
|
|
263
|
+
Copy `config.example.toml` to `config.toml` (or pass `--config`) so you don't
|
|
264
|
+
retype flags. CLI flags override config values. API keys are env-only and
|
|
265
|
+
never read from config files.
|
|
266
|
+
|
|
267
|
+
## Inputs
|
|
268
|
+
|
|
269
|
+
v1 accepts arXiv URLs (`abs`/`pdf`/`html`) or bare IDs (`1706.03762`). It uses
|
|
270
|
+
the paper's arXiv HTML rendering (ar5iv as fallback); papers with no HTML
|
|
271
|
+
rendering fail with a clear message. Fetched papers are cached in
|
|
272
|
+
`~/.cache/paperdigest/` (configurable via `cache_dir` in `config.toml`); pass
|
|
273
|
+
`--refresh` to re-fetch and overwrite the cached HTML, e.g. after arXiv
|
|
274
|
+
publishes a revision.
|
|
275
|
+
|
|
276
|
+
## Development
|
|
277
|
+
|
|
278
|
+
```bash
|
|
279
|
+
.venv/bin/pip install -e ".[dev]"
|
|
280
|
+
.venv/bin/pytest # fully offline: fixtures + fake LLM backend
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
Known minor cleanups (all reviewed, none load-bearing) are tracked in `TODOS.md`.
|