arxivdigest 0.4.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (131) hide show
  1. arxivdigest-0.4.1/.dockerignore +6 -0
  2. arxivdigest-0.4.1/.github/workflows/ci.yml +31 -0
  3. arxivdigest-0.4.1/.github/workflows/release.yml +37 -0
  4. arxivdigest-0.4.1/.gitignore +6 -0
  5. arxivdigest-0.4.1/CHANGELOG.md +132 -0
  6. arxivdigest-0.4.1/Dockerfile +11 -0
  7. arxivdigest-0.4.1/LICENSE +21 -0
  8. arxivdigest-0.4.1/PKG-INFO +283 -0
  9. arxivdigest-0.4.1/README.md +235 -0
  10. arxivdigest-0.4.1/TODOS.md +51 -0
  11. arxivdigest-0.4.1/config.example.toml +21 -0
  12. arxivdigest-0.4.1/docs/audit-untouched-ideas-2026-07-13.md +304 -0
  13. arxivdigest-0.4.1/docs/images/concept-note.png +0 -0
  14. arxivdigest-0.4.1/docs/paperdigest-story.md +25 -0
  15. arxivdigest-0.4.1/docs/smoke-test-crosslinks.md +206 -0
  16. arxivdigest-0.4.1/docs/smoke-test-scaffold.md +105 -0
  17. arxivdigest-0.4.1/docs/superpowers/plans/2026-07-10-paperdigest.md +1714 -0
  18. arxivdigest-0.4.1/docs/superpowers/plans/2026-07-12-scaffold-subcommand.md +1241 -0
  19. arxivdigest-0.4.1/docs/superpowers/plans/2026-07-17-agents-brief-crosslinks.md +591 -0
  20. arxivdigest-0.4.1/docs/superpowers/smoke-test.md +65 -0
  21. arxivdigest-0.4.1/docs/superpowers/specs/2026-07-10-paperdigest-design.md +85 -0
  22. arxivdigest-0.4.1/docs/superpowers/specs/2026-07-11-figure-explanations.md +160 -0
  23. arxivdigest-0.4.1/docs/superpowers/specs/2026-07-12-scaffold-subcommand-design.md +146 -0
  24. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/README.md +57 -0
  25. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/logs/digest-final.log +18 -0
  26. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/logs/model-identity.txt +8 -0
  27. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/logs/scaffold-final.log +13 -0
  28. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/.gitignore +10 -0
  29. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/AGENTS.md +43 -0
  30. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/EXPERIMENTS.md +21 -0
  31. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/README.md +18 -0
  32. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/configs/base.yaml +12 -0
  33. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/configs/smoke.yaml +12 -0
  34. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/data/processed/.gitkeep +0 -0
  35. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/data/raw/.gitkeep +0 -0
  36. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/evaluate.py +43 -0
  37. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/experiments/exp001_smoke/README.md +19 -0
  38. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/experiments/runs.jsonl +0 -0
  39. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/logs/.gitkeep +0 -0
  40. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/notebooks/.gitkeep +0 -0
  41. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/pyproject.toml +19 -0
  42. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/reports/figures/.gitkeep +0 -0
  43. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/src/attention_is_all_you_need/__init__.py +0 -0
  44. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/src/attention_is_all_you_need/_config.py +34 -0
  45. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/src/attention_is_all_you_need/attention.py +74 -0
  46. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/src/attention_is_all_you_need/layers.py +73 -0
  47. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/src/attention_is_all_you_need/position.py +33 -0
  48. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/src/attention_is_all_you_need/tracking.py +41 -0
  49. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/src/attention_is_all_you_need/transformer.py +43 -0
  50. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/tests/test_smoke.py +45 -0
  51. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/scaffold-project/train.py +43 -0
  52. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/adam optimizer.md +11 -0
  53. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/attention mask.md +11 -0
  54. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/attention mechanism.md +11 -0
  55. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/attention.md +11 -0
  56. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/auto-regressive.md +11 -0
  57. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/bleu score.md +11 -0
  58. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/byte-pair encoding.md +11 -0
  59. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/constituency parsing.md +11 -0
  60. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/convolutional neural network.md +11 -0
  61. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/decoder.md +11 -0
  62. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/dot product.md +11 -0
  63. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/dropout.md +11 -0
  64. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/embedding layer.md +11 -0
  65. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/embedding.md +11 -0
  66. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/encoder-decoder.md +11 -0
  67. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/encoder.md +11 -0
  68. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/feed-forward network.md +11 -0
  69. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/key.md +11 -0
  70. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/label smoothing.md +11 -0
  71. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/layer normalization.md +11 -0
  72. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/machine translation.md +11 -0
  73. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/matrix multiplication.md +11 -0
  74. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/multi-head attention.md +11 -0
  75. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/parallelization.md +11 -0
  76. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/perplexity.md +11 -0
  77. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/positional encoding.md +11 -0
  78. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/query key value.md +11 -0
  79. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/query.md +11 -0
  80. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/recurrent neural network.md +11 -0
  81. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/residual connection.md +11 -0
  82. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/scaled dot-product attention.md +11 -0
  83. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/scaled dot-product.md +11 -0
  84. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/self-attention.md +11 -0
  85. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/sequence transduction.md +11 -0
  86. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/softmax.md +11 -0
  87. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/transformer.md +11 -0
  88. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/value.md +11 -0
  89. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/weight sharing.md +11 -0
  90. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Glossary/word-piece.md +11 -0
  91. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/00 Overview.md +67 -0
  92. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/01 Scaled Dot-Product Attention.md +60 -0
  93. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/02 Multi-Head Attention.md +63 -0
  94. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/03 Positional Encoding.md +60 -0
  95. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/04 Transformer Architecture.md +104 -0
  96. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/05 Why Self-Attention.md +74 -0
  97. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/06 Self-Attention.md +60 -0
  98. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/fig1.png +0 -0
  99. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/fig2.png +0 -0
  100. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/fig3.png +0 -0
  101. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/fig4.png +0 -0
  102. arxivdigest-0.4.1/examples/9b-attention-is-all-you-need/vault/Papers/2017-attention-is-all-you-need/fig5.png +0 -0
  103. arxivdigest-0.4.1/examples/README.md +12 -0
  104. arxivdigest-0.4.1/paperdigest/__init__.py +14 -0
  105. arxivdigest-0.4.1/paperdigest/cli.py +203 -0
  106. arxivdigest-0.4.1/paperdigest/config.py +61 -0
  107. arxivdigest-0.4.1/paperdigest/digest.py +347 -0
  108. arxivdigest-0.4.1/paperdigest/extract.py +110 -0
  109. arxivdigest-0.4.1/paperdigest/fetch.py +143 -0
  110. arxivdigest-0.4.1/paperdigest/llm.py +278 -0
  111. arxivdigest-0.4.1/paperdigest/mermaid.py +101 -0
  112. arxivdigest-0.4.1/paperdigest/render.py +176 -0
  113. arxivdigest-0.4.1/paperdigest/scaffold.py +633 -0
  114. arxivdigest-0.4.1/paperdigest/templates.py +119 -0
  115. arxivdigest-0.4.1/pyproject.toml +55 -0
  116. arxivdigest-0.4.1/tests/conftest.py +60 -0
  117. arxivdigest-0.4.1/tests/fixtures/figure1.png +0 -0
  118. arxivdigest-0.4.1/tests/fixtures/paper.html +37 -0
  119. arxivdigest-0.4.1/tests/test_cli.py +198 -0
  120. arxivdigest-0.4.1/tests/test_config.py +115 -0
  121. arxivdigest-0.4.1/tests/test_digest.py +379 -0
  122. arxivdigest-0.4.1/tests/test_extract.py +42 -0
  123. arxivdigest-0.4.1/tests/test_fetch.py +187 -0
  124. arxivdigest-0.4.1/tests/test_llm.py +481 -0
  125. arxivdigest-0.4.1/tests/test_mermaid.py +97 -0
  126. arxivdigest-0.4.1/tests/test_render.py +338 -0
  127. arxivdigest-0.4.1/tests/test_scaffold.py +15 -0
  128. arxivdigest-0.4.1/tests/test_scaffold_cli.py +111 -0
  129. arxivdigest-0.4.1/tests/test_scaffold_stages.py +528 -0
  130. arxivdigest-0.4.1/tests/test_scaffold_write.py +103 -0
  131. arxivdigest-0.4.1/tests/test_templates.py +75 -0
@@ -0,0 +1,6 @@
1
+ .venv
2
+ .git
3
+ tests
4
+ docs
5
+ __pycache__
6
+ *.egg-info
@@ -0,0 +1,31 @@
1
+ name: CI
2
+
3
+ on:
4
+ push:
5
+ branches: [main]
6
+ pull_request:
7
+
8
+ jobs:
9
+ test:
10
+ runs-on: ubuntu-latest
11
+ strategy:
12
+ matrix:
13
+ python-version: ["3.11", "3.12", "3.13"]
14
+ steps:
15
+ - uses: actions/checkout@v4
16
+ - uses: actions/setup-python@v5
17
+ with:
18
+ python-version: ${{ matrix.python-version }}
19
+ - run: pip install -e '.[dev]'
20
+ - run: python -m ruff check .
21
+ - run: python -m pytest -q --cov=paperdigest --cov-report=term-missing --cov-fail-under=93
22
+
23
+ build:
24
+ runs-on: ubuntu-latest
25
+ steps:
26
+ - uses: actions/checkout@v4
27
+ - uses: actions/setup-python@v5
28
+ with:
29
+ python-version: "3.12"
30
+ - run: pip install build
31
+ - run: python -m build
@@ -0,0 +1,37 @@
1
+ name: Release
2
+
3
+ on:
4
+ push:
5
+ tags: ["v*"]
6
+
7
+ jobs:
8
+ release:
9
+ runs-on: ubuntu-latest
10
+ permissions:
11
+ contents: write
12
+ id-token: write # required for PyPI trusted publishing
13
+ steps:
14
+ - uses: actions/checkout@v4
15
+ - uses: actions/setup-python@v5
16
+ with:
17
+ python-version: "3.12"
18
+ - run: pip install -e '.[dev]'
19
+ - run: python -m ruff check .
20
+ - run: python -m pytest -q --cov=paperdigest --cov-report=term-missing --cov-fail-under=93
21
+ - name: Check tag matches pyproject.toml version
22
+ run: |
23
+ TAG_VERSION="${GITHUB_REF_NAME#v}"
24
+ PROJECT_VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
25
+ if [ "$TAG_VERSION" != "$PROJECT_VERSION" ]; then
26
+ echo "Tag version ($TAG_VERSION) does not match pyproject.toml version ($PROJECT_VERSION)" >&2
27
+ exit 1
28
+ fi
29
+ - run: pip install build
30
+ - run: python -m build
31
+ - uses: softprops/action-gh-release@v2
32
+ with:
33
+ files: |
34
+ dist/*.whl
35
+ dist/*.tar.gz
36
+ - name: Publish to PyPI (trusted publishing — no stored credentials)
37
+ uses: pypa/gh-action-pypi-publish@release/v1
@@ -0,0 +1,6 @@
1
+ .venv/
2
+ __pycache__/
3
+ *.egg-info/
4
+ .pytest_cache/
5
+ config.toml
6
+ .coverage
@@ -0,0 +1,132 @@
1
+ # Changelog
2
+
3
+ ## 0.4.1 — 2026-10-13
4
+
5
+ ### Added
6
+ - Published to PyPI as **`arxivdigest`**: `pipx install arxivdigest` (or
7
+ `pip install arxivdigest`) installs the CLI without cloning the repo.
8
+ The `paperdigest` distribution name was unavailable — PyPI's name-similarity
9
+ guard reserves it for an unrelated, dormant `paper-digest` project.
10
+ The command remains `paperdigest`.
11
+ - The release workflow publishes to PyPI via GitHub **trusted publishing**
12
+ (OIDC) — no API tokens or passwords are stored on any machine.
13
+
14
+ ### Changed
15
+ - README quick start leads with the packaged install; source-install
16
+ instructions moved below it.
17
+
18
+ ### Fixed
19
+ - `__version__` now resolves package metadata under the `arxivdigest`
20
+ distribution name (with `paperdigest` as fallback for stale dev environments);
21
+ previously it looked only for the old name and reported `0.0.0` on clean
22
+ installs, which failed the release test gate.
23
+
24
+ ## 0.4.0 — 2026-07-25
25
+
26
+ ### Added
27
+ - `digest --project-dir` (or the `project_dir` config key) links notes to the
28
+ scaffolded project for the same paper: the overview gains a `## Build it`
29
+ section, and concept notes link the stub module(s) whose cited `§x.y`
30
+ paper sections match the concept, resolved by scanning the project on disk.
31
+ - Scaffold projects now include a generated `AGENTS.md` coding-agent brief
32
+ (dependency-ordered implementation plan, TODO counts, definition of done,
33
+ ground rules), assembled deterministically with no extra LLM call;
34
+ `scaffold --vault` links it back to the paper's Obsidian notes.
35
+ - Figures: paper figures are downloaded and explained by a vision-capable
36
+ model, with the image and explanation embedded in the matching concept
37
+ note. `--figures/--no-figures` and the `max_figures` config key (default
38
+ 8) control it; backends without vision skip figures after one warning.
39
+ - `--version` flag; the scaffold failure debug dump now includes the version.
40
+ - `--quiet` flag on both commands to suppress progress output.
41
+ - `--max-input-chars`, `--max-tokens`, and `--cache-dir` CLI overrides for
42
+ their config keys.
43
+ - Concept explanations and module stubs are generated with up to 4 parallel
44
+ requests on cloud backends (local servers stay serial).
45
+ - CI and the release workflow enforce a 93% test-coverage floor.
46
+ - `max_tokens` config option: the LLM output-token cap is now set explicitly
47
+ on both backends instead of hard-coded (Anthropic) or left to the server
48
+ default (OpenAI-compatible). See Changed for the current default.
49
+ - Scaffold stages now get the same one-shot JSON repair round-trip as digest:
50
+ a malformed response is sent back to the model once for correction before
51
+ the run aborts.
52
+ - Tag-triggered release workflow: pushing a `v*` tag lints, tests, builds,
53
+ verifies the tag matches `pyproject.toml`, and attaches the wheel and sdist
54
+ to a GitHub Release.
55
+ - `examples/9b-attention-is-all-you-need/`: real, unedited output from a live
56
+ end-to-end run of both pipelines against arXiv:1706.03762 using a local 9B
57
+ model (Qwen3.5-9B-Instruct, NVFP4), with the run logs, as a worked example of
58
+ what paperdigest produces on self-hostable hardware.
59
+
60
+ ### Changed
61
+ - Default `max_tokens` raised from 8192 to 16384 so mid-size local models
62
+ (e.g. a 9B) don't truncate long generated code files; still overridable
63
+ per run with `--max-tokens` or the config key.
64
+
65
+ ### Fixed
66
+ - `__version__` is now derived from package metadata instead of a hard-coded
67
+ string that had drifted behind `pyproject.toml`.
68
+ - `~` in `vault`/`cache_dir` config values is now expanded.
69
+ - Concept-to-section matching is tiered (number-stripped exact, exact, then
70
+ longest-overlap containment), so short or numeric section titles can no
71
+ longer attach the wrong section's text to a concept.
72
+ - Generated code that imported sibling/infra modules by bare name (e.g.
73
+ `from attention import ...`) is now deterministically rewritten to the full
74
+ package path, so a scaffolded project's own smoke test imports correctly once
75
+ the project is installed.
76
+ - Generated `EXPERIMENTS.md` no longer fabricates benchmark result numbers;
77
+ targets are cited to the paper's own tables instead.
78
+ - The scaffold dependency scanner no longer lists the project's own modules as
79
+ pip dependencies in the generated `pyproject.toml`.
80
+ - The config loader generated into scaffolded projects now accepts
81
+ `key = value` as well as `key: value`, matching either separator the model
82
+ may emit. (This is the *generated project's* loader; paperdigest's own
83
+ `config.toml` remains strict TOML.)
84
+
85
+ ## 0.3.0 — 2026-07-15
86
+
87
+ Robustness and safety hardening.
88
+
89
+ ### Added
90
+ - Safety scan of all LLM-generated Python in `scaffold`: imports of
91
+ `subprocess`/`socket`/`ctypes`/`urllib`/`requests` and friends, calls to
92
+ `eval`/`exec`/`compile`/`__import__`, and dangerous attribute calls
93
+ (`os.system`, `pickle.load`, `shutil.rmtree`, ...) abort the pipeline
94
+ loudly with filename and line numbers.
95
+ - `--refresh` flag on `digest` and `scaffold` to re-fetch a paper, bypassing
96
+ the HTML cache.
97
+ - CI now runs ruff lint, reports test coverage, and verifies the package
98
+ builds (sdist + wheel).
99
+
100
+ ### Changed
101
+ - LLM calls have a 300 s request timeout (previously unbounded).
102
+ - Responses truncated at the token limit now raise a clear error instead of
103
+ being silently written as complete notes.
104
+ - Only transient errors (connection, timeout, rate limit, 5xx) are retried;
105
+ permanent errors fail immediately with the original cause preserved.
106
+ - Module planning rejects duplicate filenames instead of silently
107
+ overwriting one generated module with another.
108
+ - Scaffold debug dumps are written to the cache directory (keyed by paper id
109
+ and stage) instead of the current working directory.
110
+
111
+ ### Fixed
112
+ - HTML cache writes are atomic; an interrupted fetch can no longer leave a
113
+ corrupt cache file that is served forever.
114
+ - A failure partway through rendering removes the half-written paper folder
115
+ (existing glossary notes are never touched).
116
+ - Mermaid fence sanitization and validation now handle CRLF line endings and
117
+ trailing whitespace instead of silently skipping those blocks.
118
+ - Concurrent runs against the same paper report a clean error instead of a
119
+ traceback.
120
+
121
+ ## 0.2.0 — 2026-07-12
122
+
123
+ - Mermaid diagrams by default with sanitizer and optional node-based
124
+ validation; `--diagram` option.
125
+ - JSON-mode structured output with trailing-junk-tolerant parsing.
126
+ - Prompt-echo rejection in generated scaffold code.
127
+
128
+ ## 0.1.0 — 2026-07-11
129
+
130
+ - Initial release: `digest` (arXiv paper to plain-English Obsidian notes)
131
+ and `scaffold` (paper to git-initialized research project skeleton),
132
+ local-first with optional Anthropic/OpenAI backends.
@@ -0,0 +1,11 @@
1
+ FROM python:3.12-slim
2
+
3
+ WORKDIR /app
4
+ COPY pyproject.toml README.md ./
5
+ COPY paperdigest ./paperdigest
6
+ RUN pip install --no-cache-dir .
7
+
8
+ VOLUME /vault
9
+
10
+ ENTRYPOINT ["paperdigest"]
11
+ CMD ["--help"]
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Thad Reber
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,283 @@
1
+ Metadata-Version: 2.5
2
+ Name: arxivdigest
3
+ Version: 0.4.1
4
+ Summary: Turn AI/ML papers into plain-English Obsidian notes and reproduction-ready project scaffolds — local-first, runs on a 9B model
5
+ Project-URL: Repository, https://github.com/thadreber-web/paperdigest
6
+ Author: Thad Reber
7
+ License: MIT License
8
+
9
+ Copyright (c) 2026 Thad Reber
10
+
11
+ Permission is hereby granted, free of charge, to any person obtaining a copy
12
+ of this software and associated documentation files (the "Software"), to deal
13
+ in the Software without restriction, including without limitation the rights
14
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
15
+ copies of the Software, and to permit persons to whom the Software is
16
+ furnished to do so, subject to the following conditions:
17
+
18
+ The above copyright notice and this permission notice shall be included in all
19
+ copies or substantial portions of the Software.
20
+
21
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
22
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
23
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
24
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
25
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
26
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
27
+ SOFTWARE.
28
+ License-File: LICENSE
29
+ Keywords: arxiv,llama-cpp,llm,local-llm,obsidian,papers,research
30
+ Classifier: Development Status :: 4 - Beta
31
+ Classifier: Intended Audience :: Science/Research
32
+ Classifier: License :: OSI Approved :: MIT License
33
+ Classifier: Programming Language :: Python :: 3.11
34
+ Classifier: Programming Language :: Python :: 3.12
35
+ Classifier: Programming Language :: Python :: 3.13
36
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
37
+ Requires-Python: >=3.11
38
+ Requires-Dist: anthropic>=0.40
39
+ Requires-Dist: beautifulsoup4>=4.12
40
+ Requires-Dist: httpx>=0.27
41
+ Requires-Dist: openai>=1.50
42
+ Requires-Dist: typer>=0.12
43
+ Provides-Extra: dev
44
+ Requires-Dist: pytest-cov; extra == 'dev'
45
+ Requires-Dist: pytest>=8; extra == 'dev'
46
+ Requires-Dist: ruff; extra == 'dev'
47
+ Description-Content-Type: text/markdown
48
+
49
+ # paperdigest
50
+
51
+ Read any AI/ML arXiv paper without having every piece of jargon memorized.
52
+ paperdigest turns a paper into a folder of plain-English, wikilinked Obsidian
53
+ notes — concept-by-concept explainers, Mermaid diagrams, equations walked
54
+ through line by line, and a vault-wide glossary that defines the terms as you
55
+ hit them.
56
+
57
+ <p align="center">
58
+ <img src="docs/images/concept-note.png" alt="A generated concept note open in Obsidian, showing a plain-English analogy, an auto-generated Mermaid flowchart, and the paper's equation" width="620">
59
+ </p>
60
+ <p align="center">
61
+ <em>One generated note in Obsidian: a plain-English analogy for the concept, an
62
+ auto-generated Mermaid diagram of the computation, and the paper's equation
63
+ walked through line by line. Everything below is produced from the arXiv source
64
+ — no hand-editing.</em>
65
+ </p>
66
+
67
+ Built for the reader who follows AI/ML advances but isn't a specialist in every
68
+ subfield: instead of bouncing between the PDF and ten browser tabs, you get the
69
+ paper's ideas explained in your own vault, each unfamiliar term a click away.
70
+
71
+ ## See a full example
72
+
73
+ [`examples/9b-attention-is-all-you-need/`](examples/9b-attention-is-all-you-need)
74
+ is a complete run on *Attention Is All You Need*, checked into the repo — the
75
+ generated Obsidian [vault](examples/9b-attention-is-all-you-need/vault) (concept
76
+ notes + glossary), a
77
+ [scaffolded project](examples/9b-attention-is-all-you-need/scaffold-project), and
78
+ the [run logs](examples/9b-attention-is-all-you-need/logs). It was produced
79
+ entirely by a local 9B model — no cloud, no API key.
80
+
81
+ **Local-first.** By default paperdigest talks to a local OpenAI-compatible
82
+ server (llama.cpp, Ollama, vLLM) on your own machine — no API key, no cost.
83
+ Cloud backends (Anthropic, OpenAI) are opt-in for those who want them.
84
+
85
+ ## Quick start (local, free)
86
+
87
+ Install the CLI (Python 3.11+; `pipx` keeps it in an isolated environment):
88
+
89
+ ```bash
90
+ pipx install arxivdigest # or: pip install arxivdigest
91
+ ```
92
+
93
+ (The PyPI package is named `arxivdigest`; the command you run is
94
+ `paperdigest`, unchanged.)
95
+
96
+ Have a llama.cpp server running (`llama-server -m model.gguf` serves
97
+ `http://localhost:8080/v1` by default), then:
98
+
99
+ ```bash
100
+ paperdigest https://arxiv.org/abs/1706.03762 --vault ~/ObsidianVault
101
+ ```
102
+
103
+ Installing from a clone instead? `python3 -m venv .venv && .venv/bin/pip
104
+ install -e .`, then use `.venv/bin/paperdigest …` for every command below.
105
+
106
+ Non-default port or engine? Point at it:
107
+
108
+ ```bash
109
+ # llama.cpp on another port
110
+ .venv/bin/paperdigest 1706.03762 --vault ~/ObsidianVault --base-url http://localhost:8001/v1
111
+ # Ollama (needs the real model name)
112
+ .venv/bin/paperdigest 1706.03762 --vault ~/ObsidianVault \
113
+ --base-url http://localhost:11434/v1 --model llama3.1
114
+ ```
115
+
116
+ Output lands in `<vault>/Papers/<year>-<title-slug>/` plus new term notes in
117
+ `<vault>/Glossary/`. Re-running the same paper refuses to overwrite unless you
118
+ pass `--force`. Glossary term notes are never overwritten. Pass `--quiet` to
119
+ suppress progress output, or `--max-input-chars`/`--max-tokens`/`--cache-dir`
120
+ to override those config keys per run (both `digest` and `scaffold` accept
121
+ these). `paperdigest --version` prints the installed version — it is a
122
+ top-level flag, not available on the subcommands.
123
+
124
+ ### Choosing a model & context window
125
+
126
+ Use a capable instruct model (~7B+); the pipeline needs structured JSON
127
+ output, and very small models fail cleanly with an `error:` message. Set
128
+ `max_input_chars` in `config.toml` to fit your model's context window (e.g.
129
+ 24000 for ~8k-token contexts) — long papers are trimmed with a warning; set
130
+ `max_tokens` alongside it to cap output length for the same small context.
131
+ Big *thinking* models need the opposite: reasoning tokens count against the
132
+ output cap, so if a run aborts with a `truncated at max_tokens` error, raise
133
+ it (e.g. `--max-tokens 32768` — see `docs/smoke-test-crosslinks.md` for a
134
+ live example on a 122B).
135
+
136
+ ## How it works
137
+
138
+ ```mermaid
139
+ flowchart TD
140
+ A["arXiv paper<br>(HTML, fetched &amp; cached)"] --> B["parse into<br>sections + figures"]
141
+ B --> C["outline (LLM):<br>concepts, jargon, TL;DR"]
142
+ C --> D["per-concept note (LLM):<br>plain-English + equation walk-through<br>+ embedded Mermaid / ASCII diagram"]
143
+ B --> E["figures explained<br>by a vision model"]
144
+ C --> F["glossary (LLM):<br>define jargon not already in your vault"]
145
+ D --> G["Obsidian vault:<br>Papers/ + Glossary/"]
146
+ E --> G
147
+ F --> G
148
+ ```
149
+
150
+ The outline and glossary stages ask the server for JSON (`response_format:
151
+ json_object`) when it's supported — that's what makes small local models
152
+ reliable here. Each concept's diagram is generated as part of its note, then
153
+ sanitized deterministically before it's written; figures are embedded next to
154
+ the concept they illustrate.
155
+
156
+ ### Diagrams: Mermaid or ASCII
157
+
158
+ Structural concepts get a diagram. The default is **Mermaid** (renders
159
+ natively in Obsidian); pass `--diagram ascii` (or set `diagram = "ascii"` in
160
+ `config.toml`) for plain-text diagrams that can never have syntax errors.
161
+ Small local models sometimes emit Mermaid with unquoted special characters in
162
+ node labels, so every diagram goes through a deterministic sanitizer that
163
+ auto-quotes them. Optionally, if `node` is on your PATH and the `mermaid` npm
164
+ package is importable from the directory you run paperdigest in
165
+ (`npm install mermaid jsdom global-jsdom`), each diagram is also parse-checked
166
+ and a warning names any note whose diagram still fails. No node? Validation is
167
+ skipped silently — the worst case is one diagram rendering as an error box in
168
+ Obsidian instead of a picture.
169
+
170
+ ### Figures
171
+
172
+ By default, figures are downloaded from the paper and explained by a
173
+ vision-capable model, with the image and its explanation embedded together
174
+ in the matching concept note (or the overview, if unmatched). This needs a
175
+ vision-capable backend — for local llama.cpp, serve with an `--mmproj`
176
+ projector file. Backends without vision skip figures automatically after a
177
+ warning on the first figure, and the rest of the digest completes normally.
178
+ Pass `--no-figures` (or set `figures = false` in `config.toml`) to turn this
179
+ off; it's also the budget option on cloud backends, where images are
180
+ token-expensive. `max_figures` (default 8) caps how many figures per paper
181
+ are explained, by document order.
182
+
183
+ ### Linking to a scaffolded project
184
+
185
+ Pass `--project-dir` (or set `project_dir` in `config.toml`) and notes gain a
186
+ `## Build it` section pointing at `<project-dir>/<year>-<title-slug>` — the
187
+ project `scaffold` would create for the same paper. If that project exists on
188
+ disk, concept notes also get a `*Build it: …*` line linking the stub module(s)
189
+ whose cited `§x.y` paper sections match the concept, so you can jump straight
190
+ from the explanation to the code that implements it.
191
+
192
+ ## Scaffold a research project
193
+
194
+ Turn a paper into a standalone, git-initialized project skeleton (Cookiecutter-DS-style
195
+ layout, JSONL experiment tracking, stub modules mapped to paper sections):
196
+
197
+ ```bash
198
+ .venv/bin/paperdigest scaffold https://arxiv.org/abs/1706.03762 --dest ~/projects
199
+ ```
200
+
201
+ Output lands in `<dest>/<year>-<title-slug>/` with its own git history. The code is a
202
+ *skeleton*: module stubs with `TODO(paper §x.y)` markers plus a complete train/evaluate
203
+ harness and a toy smoke experiment — not a working reimplementation. Any generation
204
+ failure aborts loudly (nothing half-written); the raw model output is saved to
205
+ `scaffold-debug-<stage>.txt` for inspection. Re-running refuses to overwrite an
206
+ existing project unless you pass `--force`.
207
+
208
+ You don't need a big model: scaffolding is tested live end-to-end on a 9B via
209
+ llama.cpp (see `docs/smoke-test-scaffold.md`; the cross-linking features are
210
+ live-tested on a 122B via vLLM in `docs/smoke-test-crosslinks.md`). JSON
211
+ stages request structured
212
+ output (`response_format: json_object`) from the server when supported, which
213
+ is what makes small local models reliable here; generated code is validated
214
+ (syntax + prompt-echo rejection) before anything is written. Besides
215
+ `log_run()` for JSONL experiment tracking, the scaffolded `tracking.py` has
216
+ `get_logger()` for per-run log files under `logs/`.
217
+
218
+ **A note on trust:** every generated `.py` file is still LLM output. Before writing it,
219
+ paperdigest checks it parses, rejects leftover prompt-echo lines, and scans it for
220
+ dangerous calls (`subprocess`, `socket`, `eval`/`exec`, `os.system`, `pickle.load`,
221
+ `shutil.rmtree`, and similar) plus networking/FFI imports (`urllib`, `requests`,
222
+ `ctypes`, and similar) — aborting loudly if it finds any. That scan catches obviously
223
+ hazardous patterns, not everything — read the scaffolded code before you run it.
224
+
225
+ Every project also gets a generated `AGENTS.md` — a coding-agent brief with a
226
+ dependency-ordered implementation plan, a TODO count and cited paper sections per
227
+ module, a definition of done, and ground rules for the harness. It's assembled
228
+ deterministically from the same stage data, no extra LLM call. Pass `--vault` (or set
229
+ `vault` in your config file) and it links back to the paper's Obsidian notes so an
230
+ agent can read the plain-English explanation before touching code.
231
+
232
+ ## Cloud backends (optional, cost money)
233
+
234
+ | Backend | Flags | Key (env var only) |
235
+ |---|---|---|
236
+ | Local (default) | `--base-url http://localhost:8080/v1 --model local` | none |
237
+ | OpenAI | `--backend openai --model gpt-5` | `OPENAI_API_KEY` |
238
+ | Anthropic | `--backend anthropic --model claude-sonnet-5` | `ANTHROPIC_API_KEY` |
239
+
240
+ Every LLM call is retried up to 2 more times with exponential backoff on
241
+ transient errors (connection drops, timeouts, rate limits, 5xx); malformed
242
+ JSON gets one repair round-trip; other errors fail immediately. Requests
243
+ time out after 300s. On cloud backends, concept and module-stub generation
244
+ runs a few requests in parallel to cut wall-clock time; against a local
245
+ server it stays serial, since a single llama.cpp instance generally can't
246
+ service concurrent requests.
247
+
248
+ ## Docker
249
+
250
+ ```bash
251
+ docker build -t paperdigest .
252
+ docker run --rm -v ~/ObsidianVault:/vault \
253
+ --add-host=host.docker.internal:host-gateway \
254
+ paperdigest https://arxiv.org/abs/1706.03762 \
255
+ --base-url http://host.docker.internal:8080/v1
256
+ ```
257
+
258
+ (`--add-host` is needed on Linux to reach the host-side local server; for a
259
+ cloud backend pass `-e ANTHROPIC_API_KEY` or `-e OPENAI_API_KEY` instead.)
260
+
261
+ ## Config
262
+
263
+ Copy `config.example.toml` to `config.toml` (or pass `--config`) so you don't
264
+ retype flags. CLI flags override config values. API keys are env-only and
265
+ never read from config files.
266
+
267
+ ## Inputs
268
+
269
+ v1 accepts arXiv URLs (`abs`/`pdf`/`html`) or bare IDs (`1706.03762`). It uses
270
+ the paper's arXiv HTML rendering (ar5iv as fallback); papers with no HTML
271
+ rendering fail with a clear message. Fetched papers are cached in
272
+ `~/.cache/paperdigest/` (configurable via `cache_dir` in `config.toml`); pass
273
+ `--refresh` to re-fetch and overwrite the cached HTML, e.g. after arXiv
274
+ publishes a revision.
275
+
276
+ ## Development
277
+
278
+ ```bash
279
+ .venv/bin/pip install -e ".[dev]"
280
+ .venv/bin/pytest # fully offline: fixtures + fake LLM backend
281
+ ```
282
+
283
+ Known minor cleanups (all reviewed, none load-bearing) are tracked in `TODOS.md`.