recursive-llm 0.5.0__tar.gz

recursive_llm-0.5.0/.github/assets/benchmark-100k.svg

@@ -0,0 +1,39 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="1200" height="630" viewBox="0 0 1200 630" role="img" aria-labelledby="title description">
+  <title id="title">Direct versus RLM on a 100k-character exact aggregation task</title>
+  <desc id="description">GPT-5 mini direct used 37,928 tokens and passed zero of three runs, while RLM used 8,224 tokens and passed three of three. DeepSeek V4 Flash direct used 39,364 tokens and passed zero of three, while RLM used 15,209 tokens and passed two of three.</desc>
+  <rect width="1200" height="630" rx="28" fill="#0b1020"/>
+  <text x="72" y="82" fill="#f8fafc" font-family="Inter, Arial, sans-serif" font-size="38" font-weight="700">100k-character exact aggregation</text>
+  <text x="72" y="122" fill="#94a3b8" font-family="Inter, Arial, sans-serif" font-size="20">Mean model tokens, with exact pass count from 3 live runs</text>
+
+  <line x1="300" y1="178" x2="1080" y2="178" stroke="#334155" stroke-width="2"/>
+  <line x1="300" y1="178" x2="300" y2="524" stroke="#334155" stroke-width="2"/>
+  <line x1="690" y1="178" x2="690" y2="524" stroke="#1e293b" stroke-width="1"/>
+  <line x1="1080" y1="178" x2="1080" y2="524" stroke="#1e293b" stroke-width="1"/>
+  <text x="300" y="160" text-anchor="middle" fill="#64748b" font-family="Inter, Arial, sans-serif" font-size="15">0</text>
+  <text x="690" y="160" text-anchor="middle" fill="#64748b" font-family="Inter, Arial, sans-serif" font-size="15">20k</text>
+  <text x="1080" y="160" text-anchor="middle" fill="#64748b" font-family="Inter, Arial, sans-serif" font-size="15">40k tokens</text>
+
+  <text x="72" y="232" fill="#e2e8f0" font-family="Inter, Arial, sans-serif" font-size="20">GPT-5 mini direct</text>
+  <rect x="300" y="202" width="740" height="42" rx="8" fill="#475569"/>
+  <text x="1024" y="230" text-anchor="end" fill="#ffffff" font-family="Inter, Arial, sans-serif" font-size="17" font-weight="700">37,928</text>
+  <text x="1094" y="230" fill="#f87171" font-family="Inter, Arial, sans-serif" font-size="17" font-weight="700">0/3</text>
+
+  <text x="72" y="310" fill="#e2e8f0" font-family="Inter, Arial, sans-serif" font-size="20">GPT-5 mini RLM</text>
+  <rect x="300" y="280" width="160" height="42" rx="8" fill="#22c55e"/>
+  <text x="474" y="308" fill="#bbf7d0" font-family="Inter, Arial, sans-serif" font-size="17" font-weight="700">8,224</text>
+  <text x="1094" y="308" fill="#4ade80" font-family="Inter, Arial, sans-serif" font-size="17" font-weight="700">3/3</text>
+  <text x="474" y="332" fill="#86efac" font-family="Inter, Arial, sans-serif" font-size="15">78% fewer tokens</text>
+
+  <text x="72" y="410" fill="#e2e8f0" font-family="Inter, Arial, sans-serif" font-size="20">DeepSeek direct</text>
+  <rect x="300" y="380" width="768" height="42" rx="8" fill="#475569"/>
+  <text x="1052" y="408" text-anchor="end" fill="#ffffff" font-family="Inter, Arial, sans-serif" font-size="17" font-weight="700">39,364</text>
+  <text x="1094" y="408" fill="#f87171" font-family="Inter, Arial, sans-serif" font-size="17" font-weight="700">0/3</text>
+
+  <text x="72" y="488" fill="#e2e8f0" font-family="Inter, Arial, sans-serif" font-size="20">DeepSeek RLM</text>
+  <rect x="300" y="458" width="297" height="42" rx="8" fill="#38bdf8"/>
+  <text x="611" y="486" fill="#bae6fd" font-family="Inter, Arial, sans-serif" font-size="17" font-weight="700">15,209</text>
+  <text x="1094" y="486" fill="#4ade80" font-family="Inter, Arial, sans-serif" font-size="17" font-weight="700">2/3</text>
+  <text x="611" y="510" fill="#7dd3fc" font-family="Inter, Arial, sans-serif" font-size="15">61% fewer tokens</text>
+
+  <text x="72" y="578" fill="#64748b" font-family="Inter, Arial, sans-serif" font-size="16">One generated corpus seed. Exact grader. July 15, 2026. Not a paper reproduction.</text>
+</svg>

recursive_llm-0.5.0/BENCHMARK_RESULTS.md

@@ -0,0 +1,268 @@
+# Benchmark Results
+
+These live measurements were collected on July 15, 2026 with Python 3.12.13. They are a small
+engineering check, not a paper reproduction. Provider behavior is stochastic, only one generated
+corpus seed was used, and prices are LiteLLM estimates.
+
+**Grading-version note (October 2, 2026):** the historical generated-transaction,
+literary, and multi-document scores below used legacy graders. Those graders could accept duplicate
+contradictory labels or expected words assigned to the wrong facts. Current graders reject
+these cases, and the literary tasks now request explicit fact fields. The original scores
+remain historical measurements; they have not been regraded without the original raw
+answers. The four available September transaction responses also pass the new grader; this
+does not revalidate the older July/August runs. New runs record their grader version and must
+be reported separately.
+
+## TXT, PDF, CSV, and HTML evaluation
+
+An August 31, 2026 evaluation added a SHA-pinned four-format corpus, exact graders, a direct-model
+comparison, staged regression runs, and independent Codex CLI verification. GPT-5 mini passed all
+four baseline tasks in both modes. RLM used 78.2% fewer model tokens and 38.2% less estimated cost,
+while direct completion had 5.24x lower median latency. Two isolated implementation changes were
+retained: tree-wide deadline enforcement and a bounded REPL parent snapshot. Each primary live
+post-change run had one model-dependent failure, and each targeted retry passed; these single runs
+are therefore compatibility checks, not evidence of a quality improvement.
+
+See [DOCUMENT_EVALUATION.md](DOCUMENT_EVALUATION.md) for sources, hashes, commands, stage-by-stage
+results, rejected interpretations, and the implementation roadmap.
+
+## Unreleased Reliability Validation
+
+The provider-retry, JSONL-export, and final-answer-validation changes were checked on July 16, 2026
+against commit `71a5e43` and the updated working tree.
+
+Deterministic fault injection changed the intended failure paths without changing defaults:
+
+| Scenario | `71a5e43` | Updated behavior |
+| --- | --- | --- |
+| HTTP 429 followed by success | Run aborted on the 429 | Recovered with two counted calls and one retry |
+| Invalid final answer followed by a correction | Invalid answer was accepted | Validator feedback produced the corrected answer in two calls |
+| Structured run persistence | No result exporter | One versioned, redacted JSONL record per completed run |
+
+With retries and validation left at their defaults, a mocked healthy-provider microbenchmark took
+111.499 microseconds per completion on `71a5e43` and 115.053 microseconds after the changes. The
+3.2% relative increase is 3.554 microseconds per completion and is negligible beside provider
+latency.
+
+One live exact-grader smoke run per previously tested model also passed without retries:
+
+| Model | Passed | Latency | Calls | Retries | Tokens | Estimated cost |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| GPT-5 mini | 1/1 | 29.712 s | 4 | 0 | 3,472 | $0.00362075 |
+| DeepSeek V4 Flash | 1/1 | 17.883 s | 7 | 0 | 4,237 | $0.000466816 |
+
+These smoke runs verify provider integration after retry accounting changed; they are not a new
+quality comparison because model trajectories are stochastic and each model ran only once.
+
+## Multi-document candidate experiments
+
+Three proposed changes were evaluated on July 17, 2026 with DeepSeek V4 Flash. The two model-facing
+candidates used the same three tasks, model, `max_depth=2`, 20-iteration limit, 24-call tree limit,
+300-second limit, and three repetitions per task. The baseline and each model-facing candidate
+therefore contain nine exact-graded runs. This sample is useful for rejecting clear regressions; it
+is not large enough to claim a universal model-quality improvement. The non-raising result API was
+evaluated separately because it does not change model behavior.
+
+The corpora were independently sourced and distributed the required evidence across each document:
+
+| Corpus | Prepared size | DeepSeek tokens | Task |
+| --- | ---: | ---: | --- |
+| Project Gutenberg *War and Peace* | 3,227,519 characters | 777,114 | Five facts from Petya's final-night sequence |
+| Official 9/11 Commission Report | 1,986,811 characters | 472,370 | Six facts split between the preface and intelligence reform section |
+| Official Python 3.14 text documentation | 14,626,944 characters | 3,479,078 | Four API facts from separate documentation files |
+
+| Configuration | Passed | p50 latency | Mean calls | Mean tokens | Mean cost | Verdict |
+| --- | ---: | ---: | ---: | ---: | ---: | --- |
+| Existing baseline | 6/9 | 62.686 s | 16.11 | 59,033 | $0.0024388 | Retained |
+| Bounded Unicode-aware search helper | 4/9 | 53.602 s | 16.33 | 70,741 | $0.0022753 | Reverted |
+| Repetition and near-budget progress guard | 4/9 | 65.269 s | 16.22 | 60,898 | $0.0024707 | Reverted |
+
+The search helper cut median latency but reduced the exact pass count from six to four and increased
+mean model-token use by 19.8%. A content-bearing diagnostic trace showed that the model first guessed
+the helper's result keys incorrectly, then replaced compact REPL searches with many small helper
+calls. The API was computationally fast after indexing: repeat searches took 0.4 to 5.8
+milliseconds, but the model interaction was worse, so the feature was not retained.
+
+The progress guard occasionally shortened successful runs to five or six calls, but it reduced the
+overall pass count from six to four, increased median latency by 4.1%, and increased mean token use
+by 3.2%. It was also reverted. These results argue for improving task planning or training data
+before adding generic model-facing controls.
+
+The third proposal, structured failed-run results, does not alter prompts, tools, or model control
+flow. It was retained because it turns the same failure into inspectable data without changing the
+quality path. Twenty paired samples of 5,000 mocked successful runs measured a median difference of
+-0.149 microseconds per run between `atry_complete_result` and `acomplete_result`, inside a measured
+noise range of -3.543 to +6.509 microseconds. Three live, deliberately one-iteration runs, one per
+corpus, each returned a typed `MaxIterationsError` record with exactly one counted provider call and
+continued to the next task.
+
+### Multi-document reproduction
+
+The inputs were fetched on July 17, 2026. The benchmark refuses artifacts whose hashes do not match
+the pinned values in `benchmarks/multi_document.py`.
+
+```bash
+mkdir -p /tmp/rlm-multi-document
+
+curl -L https://www.gutenberg.org/files/2600/2600-0.txt \
+  -o /tmp/rlm-multi-document/war-and-peace.txt
+
+curl -L https://www.govinfo.gov/content/pkg/GPO-911REPORT/pdf/GPO-911REPORT.pdf \
+  -o /tmp/rlm-multi-document/911-commission-report.pdf
+pdftotext -layout /tmp/rlm-multi-document/911-commission-report.pdf \
+  /tmp/rlm-multi-document/911-commission-report.txt
+
+curl -L https://docs.python.org/3.14/archives/python-3.14-docs-text.zip \
+  -o /tmp/rlm-multi-document/python-3.14-docs-text.zip
+
+python benchmarks/multi_document.py deepseek/deepseek-v4-flash \
+  /tmp/rlm-multi-document --runs 3 --label baseline --jsonl results.jsonl
+```
+
+Pinned prepared-artifact hashes are:
+
+```text
+war-and-peace.txt              e4bcf9042609b62c7de72a6f1b311f54c412943a9d641b7efcf79a464b5f31c8
+911-commission-report.pdf      657d41475eb3a9a5e3e87a6c7c51ac1dfbe1af7566d1abff7bf7286e7e1c0e1b
+911-commission-report.txt      33e5f373e542c58a872dde753caaf80e3c60c2b98c29c18898ae4590c9f4cfbe
+python-3.14-docs-text.zip      c8ee0347f282f97e5a57f0b010cecd441464db9fe679862f51aeda0dad12ab47
+```
+
+The PDF has 585 pages. Pages 1, 15, 429, and 432 were rendered and visually compared with the
+extracted text used by the grader. Poppler versions can produce different whitespace; if the text
+hash differs, inspect the extraction before deliberately updating the pin.
+
+## Reproduction
+
+```bash
+# Small deterministic tasks, three repetitions each
+python benchmarks/compare_same_model.py MODEL --full --runs 3 --mode direct
+python benchmarks/compare_same_model.py MODEL --full --runs 3 --mode rlm --max-depth 2
+
+# One 100k-character deterministic task, three repetitions
+python benchmarks/compare_same_model.py MODEL --generated-chars 100000 --seed 2026 --runs 3 --mode direct
+python benchmarks/compare_same_model.py MODEL --generated-chars 100000 --seed 2026 --runs 3 --mode rlm --max-depth 2
+
+# A larger RLM-only scale check
+python benchmarks/compare_same_model.py MODEL --generated-chars 1000000 --seed 2026 --runs 3 --mode rlm --max-depth 2
+
+# A SHA-pinned public-domain real document
+curl -L https://www.gutenberg.org/files/2600/2600-0.txt -o /tmp/war-and-peace-2600-0.txt
+python benchmarks/war_and_peace.py MODEL /tmp/war-and-peace-2600-0.txt --max-depth 2
+```
+
+The generated context has 100,098 characters and SHA-256
+`1ee43f3b42f8db55369c337d2e37f1e7f61224abe3d538581a716865df8a6fcc`. Its exact answer key is:
+
+```text
+count=57 total_amount_cents=27404392 max_transaction_id=TX-0000883 max_amount_cents=993620
+```
+
+## Small tasks
+
+The direct baseline is the correct choice for these short contexts. Both models passed every direct
+run with one call. DeepSeek RLM also passed, but recursion added substantial latency and usage.
+
+| Model and mode | Passed | p50 latency | Mean calls | Mean tokens | Mean cost |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| GPT-5 mini, direct | 6/6 | 4.221 s | 1.00 | 598 | $0.0007996 |
+| DeepSeek V4 Flash, direct | 6/6 | 2.635 s | 1.00 | 502 | $0.0000929 |
+| DeepSeek V4 Flash, RLM depth 2 | 6/6 | 28.204 s | 7.17 | 6,875 | $0.0008071 |
+| GPT-5 mini, RLM depth 2, incident task only | 1/3 | 167.341 s | 18.33 | 25,830 | $0.0262083 |
+
+The GPT-5 mini RLM incident runs passed once and reached the configured 24-call cap twice. The full
+small-task RLM suite was not repeated after that result because additional calls would not answer a
+useful engineering question. The cap prevented a 25th provider request in both failed runs.
+
+## Generated 100k-character task
+
+RLM materially improved long-context correctness. GPT-5 mini also used about 78% fewer tokens and
+46% lower estimated cost per run than its direct baseline. DeepSeek used about 61% fewer tokens and
+17% lower estimated cost, but one RLM answer omitted the `TX-` prefix and failed exact grading.
+
+| Model and mode | Passed | p50 latency | Mean calls | Mean tokens | Mean cost |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| GPT-5 mini, direct | 0/3 | 35.278 s | 1.00 | 37,928 | $0.0088788 |
+| GPT-5 mini, RLM depth 2 | 3/3 | 26.716 s | 5.00 | 8,224 | $0.0048132 |
+| DeepSeek V4 Flash, direct | 0/3 | 20.738 s | 1.00 | 39,364 | $0.0012240 |
+| DeepSeek V4 Flash, RLM depth 2 | 2/3 | 20.624 s | 7.33 | 15,209 | $0.0010156 |
+
+The generated suite originally inherited a six-iteration benchmark cutoff. Raising only this suite
+to ten iterations improved the combined RLM pass rate from 4/6 to 5/6: GPT-5 mini improved from 2/3
+to 3/3, while DeepSeek remained at 2/3. This setting is now the generated-suite default; the global
+24-call and 300-second caps remain unchanged.
+
+## Generated 1M-character scale check
+
+The larger corpus was generated on July 17, 2026 with the same seed. It has 1,000,066 characters,
+approximately 337,800 tokens according to LiteLLM's model token counters, and SHA-256
+`076e6d74f9917a3ed7f0b17c0f3be49ee49ff98187d7e9918c716aa0a8074ae0`. Its exact answer key is:
+
+```text
+count=563 total_amount_cents=275857555 max_transaction_id=TX-0004073 max_amount_cents=999063
+```
+
+All six RLM runs passed exact grading without provider retries. A direct baseline was deliberately
+not run: this check measures whether externalized context continues to work at the larger scale,
+not whether sending the entire corpus to a model is economical.
+
+| Model and mode | Passed | p50 latency | p95 latency | Mean calls | Mean tokens | Mean cost |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| GPT-5 mini, RLM depth 2 | 3/3 | 42.744 s | 97.755 s | 7.00 | 17,613 | $0.0107284 |
+| DeepSeek V4 Flash, RLM depth 2 | 3/3 | 32.203 s | 52.680 s | 9.33 | 17,538 | $0.0012495 |
+
+Increasing the source context by roughly 10x increased mean model-token usage by 2.14x for GPT-5
+mini and 1.15x for DeepSeek relative to the 100k-character RLM runs. Median latency increased by
+1.60x and 1.56x, respectively. The individual GPT-5 trajectories remained high variance: 4 to 10
+calls and 19.448 to 97.755 seconds. Every successful run used only the root REPL, with
+`max_depth_reached=0` in all results. This experiment therefore validates externalized context and
+local computation rather than recursive semantic decomposition.
+
+## Real 3.2M-character document
+
+The real-document check used the public-domain English translation of *War and Peace*, Project
+Gutenberg eBook #2600. The downloaded file has SHA-256
+`e4bcf9042609b62c7de72a6f1b311f54c412943a9d641b7efcf79a464b5f31c8`; after UTF-8 decoding and
+line-ending normalization it contains 3,227,519 characters. LiteLLM's local counters estimate
+769,910 GPT-5 mini tokens and 777,114 DeepSeek V4 Flash tokens in the source document.
+
+Three tasks exercise different long-document behaviors:
+
+- exact counting of 365 narrative-body chapter headings;
+- retrieval of three facts located near the beginning and end of the novel;
+- synthesis of five pieces of narrative evidence from Petya's final-night sequence.
+
+The results below use a 20-iteration cap. A direct baseline was not sent because it would require
+placing roughly 770,000 source tokens in one provider request.
+
+| Model and mode | Passed | p50 latency | p95 latency | Mean calls | Mean tokens | Mean cost |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| GPT-5 mini, RLM depth 2 | 2/3 | 90.757 s | 117.601 s | 15.00 | 68,633 | $0.0234759 |
+| DeepSeek V4 Flash, RLM depth 2 | 2/3 | 51.055 s | 100.354 s | 12.67 | 58,816 | $0.0027428 |
+
+Both models counted the chapters exactly and retrieved the distant facts exactly. GPT-5 mini
+needed 19 iterations for distant retrieval; DeepSeek needed 12. Neither model completed the
+narrative synthesis task within 20 iterations, so these were execution failures rather than
+incorrect final answers.
+
+A content-bearing failure trace showed the main weakness clearly: the model found the death scene
+but spent later iterations on broad `Cossack` searches, a literal `Petya` search that missed
+`Pétya`, large match dumps, and a character-by-character scan. A proposed generic search-efficiency
+prompt was tested at the original 12-iteration cap, did not improve the pass rate, and increased
+token use on the narrative task, so it was reverted. The retained improvement is observability:
+failed traced runs now preserve their partial trajectory instead of returning an empty event list.
+
+All measured calls stayed at the root REPL with `max_depth_reached=0`. The benchmark therefore
+identifies a concrete next problem: efficient semantic navigation and deliberate child-model use
+over real prose, rather than raw context capacity alone.
+
+## Interpretation
+
+- Use direct completion for short contexts that fit comfortably in the model window.
+- RLM is useful here when exact computation over a large context matters; the context stays outside
+  model prompts and Python performs the aggregation.
+- `max_depth=2` did not force recursion. The successful generated runs used the root REPL only, which
+  is an important reminder that RLM's value includes externalized context and local computation, not
+  only child-model calls.
+- Three repetitions and one seed are enough to catch regressions, but not enough for broad model
+  quality claims. Use more seeds and runs for release decisions.

recursive_llm-0.5.0/CHANGELOG.md

@@ -0,0 +1,146 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+## [Unreleased]
+
+## [0.5.0] - 2026-10-02
+
+### Added
+
+- A side-by-side web visualizer and downloadable film of real RLM and direct GPT-5.6 Luna runs,
+  with synchronized replay, call trees, recorded Python steps, import, and JSON export.
+- An optional async completion handler, propagated across child RLMs, and a bounded Codex CLI
+  transport for ChatGPT-authenticated subscription experiments without API keys.
+- A reproducible paired capture script, 10k and 100k generated-source recordings, and an explicit
+  methodology distinguishing total CLI tokens from subscription billing.
+
+### Changed
+
+- Made the paired Codex capture model, reasoning effort, and recursion depth configurable while
+  retaining the historical Luna/medium/depth-1 defaults and recording the actual settings.
+- Versioned stricter generated-transaction and document fact graders; historical benchmark scores
+  remain labeled as results of their original grading rules.
+- Added frontend type, test, and build checks and Python PDF coverage to CI.
+- Updated the visualizer's React, Vite, Vinext, and Cloudflare dependencies to patched,
+  peer-compatible releases, with a scoped Satori decompressor override.
+
+### Fixed
+
+- Stopped admitting calls after a shared usage budget is exhausted and drained cancelled callbacks
+  before publishing the final run statistics and trajectory.
+- Removed deleted REPL variables from recovery snapshots and prevented final answers from using
+  an obsolete parent snapshot after worker loss.
+- Enabled ordinary, nested, and starred assignment unpacking in the restricted Python worker.
+- Separated process-readiness allowances from cancellation assertions in coverage-instrumented tests.
+- Deferred the public `RLM` import so standalone REPL workers start without importing LiteLLM.
+- Kept the visualizer's required `lib` sources visible to Git.
+
+## [0.4.0] - 2026-08-31
+
+### Added
+
+- A SHA-pinned exact-graded document benchmark covering a Project Gutenberg TXT book, the NIST AI
+  RMF Playbook in PDF and CSV forms, and the official Python sqlite3 HTML documentation.
+- A deterministic REPL snapshot IPC benchmark and structured Codex CLI verification schema.
+
+### Changed
+
+- Capped parent-side REPL state snapshots at 1 MB by default while preserving complete variables
+  in the persistent worker and explicit variable retrieval.
+
+### Fixed
+
+- Applied the shared elapsed-time budget to REPL exchanges and prevented answers completed by a
+  late final-answer validator from being accepted after the tree deadline.
+- Capped LiteLLM below 1.98.0 because that release imports Python 3.11-only typing symbols during
+  startup and breaks the supported Python 3.9 and 3.10 environments.
+
+## [0.3.1] - 2026-07-28
+
+### Fixed
+
+- Pinned Ruff 0.16.0 and made the stable baseline rules explicit for reproducible local and CI
+  checks.
+
+## [0.3.0] - 2026-07-25
+
+### Added
+
+- Opt-in, budget-aware retries for transient provider failures with exponential backoff,
+  `Retry-After` support, exact retry statistics, and trajectory events.
+- Versioned JSONL export for structured completion results with secret-free run configuration.
+- Final-answer validation with deterministic model feedback and support for directives, REPL
+  variables, and mutable answer publication.
+- A reproducible 1M-character scale check with exact grading and live GPT-5 mini and DeepSeek V4
+  Flash results.
+- A SHA-pinned real-document benchmark over the public-domain English translation of *War and
+  Peace*, with exact graders for structure, distant retrieval, and narrative evidence synthesis.
+- A read-only `RLM.trajectory` snapshot that preserves partial events from failed runs.
+- Non-raising `try_complete_result` and `atry_complete_result` APIs with typed, versioned failure
+  records and exact per-run diagnostics.
+- A SHA-pinned three-document benchmark covering a 3.2M-character novel, a 2.0M-character official
+  report, and a 14.6M-character documentation corpus.
+
+### Changed
+
+- Normalized null provider text content into the existing empty-response repair path and added
+  explicit errors for malformed provider response structures.
+- Disabled hidden LiteLLM retries so every real retry is governed by tree-wide budgets.
+- Preserved content-bearing benchmark trajectories when a traced run raises an exception.
+- Updated the benchmark runner to consume success and failure results through one structured API.
+
+## [0.2.0] - 2026-07-15
+
+### Migration notes
+
+- Replace `RLM.completion(...)` with `RLM.complete(...)` and `RLM.acompletion(...)` with
+  `RLM.acomplete(...)`.
+- The default `max_depth` is now `1` instead of `5`. Depth follows the paper's capability-based
+  convention: `0` enables only the root REPL, `1` permits plain-LM subcalls, and `2` permits one
+  child RLM level with a plain-LM boundary fallback.
+
+### Added
+
+- Tree-wide call, token, cost, and elapsed-time budgets with partial statistics on budget errors.
+- Structured completion results with exact per-run statistics and root/child/leaf trajectories.
+- Safe concurrent use of the same `RLM` instance through isolated per-invocation state.
+- Optional POSIX memory, CPU-time, and open-file limits for REPL workers.
+- Deterministic long-context benchmark generation, exact task graders, repeated runs, direct-model
+  baselines, JSONL output, and checked-in live benchmark results.
+- GitHub Actions checks across Python 3.9-3.12 on Linux and Python 3.12 on macOS and Windows.
+- Security guidance describing the REPL isolation model and its trust boundary.
+
+### Changed
+
+- Aligned recursion-depth behavior and documentation with the RLM paper's depth convention.
+- Hardened REPL execution with spawned workers, persistent state, hard local-step timeouts, bounded
+  output, restricted imports, and ordered bounded-concurrency subcalls.
+- Aggregated usage and best-effort cost statistics across the complete recursion tree.
+- Updated repository, installation, citation, release, and issue links to `grishahq/recursive-llm`.
+- Pinned the formatter to a Python 3.9-compatible version and updated GitHub Actions to Node 24-based
+  releases for reproducible, warning-free CI runs.
+- Expanded the test suite from 43 initial-release tests to 135 tests with enforced branch coverage.
+
+### Fixed
+
+- Required final-answer directives to be standalone executable statements instead of accepting
+  occurrences embedded in arbitrary text.
+- Prevented models from guessing context contents before inspecting the REPL context.
+- Corrected parameter handling for GPT-5-family models.
+- Made persistent REPL variables visible inside comprehension bodies on Python 3.9-3.11 while
+  keeping restricted runtime helpers out of parent snapshots.
+- Prevented REPL worker pipe errors from leaking tracebacks during budget-triggered shutdown.
+- Corrected offline-demo aggregation and strengthened benchmark numeric-boundary grading.
+
+## [0.1.0] - 2025-10-17
+
+- Initial public release.
+
+[Unreleased]: https://github.com/grishahq/recursive-llm/compare/v0.5.0...HEAD
+[0.5.0]: https://github.com/grishahq/recursive-llm/compare/v0.4.0...v0.5.0
+[0.4.0]: https://github.com/grishahq/recursive-llm/compare/v0.3.1...v0.4.0
+[0.3.1]: https://github.com/grishahq/recursive-llm/compare/v0.3.0...v0.3.1
+[0.3.0]: https://github.com/grishahq/recursive-llm/compare/v0.2.0...v0.3.0
+[0.2.0]: https://github.com/grishahq/recursive-llm/compare/v0.1.0...v0.2.0
+[0.1.0]: https://github.com/grishahq/recursive-llm/releases/tag/v0.1.0

recursive_llm-0.5.0/CODEX_COMPARISON.md

@@ -0,0 +1,89 @@
+# Codex subscription comparison
+
+Recorded on September 8, 2026 to accompany the 600-star visualizer. Both modes use `gpt-5.6-luna` with medium reasoning through Codex CLI 0.149.0, authenticated with ChatGPT. No API key was used.
+
+Each row is **one paired experiment**, with direct mode first. These observations do not establish a general quality, latency, or subscription-efficiency advantage.
+
+| Generated source | Mode | Exact fields | Total CLI model tokens | Cached input | Model calls | Elapsed |
+| --- | --- | ---: | ---: | ---: | ---: | ---: |
+| 10,104 characters | Direct | 4/4 | 6,007 | 0 | 1 | 9.172 s |
+| 10,104 characters | RLM | 4/4 | 19,430 | 0 | 5 | 48.488 s |
+| 100,098 characters | Direct | 4/4 | 37,521 | 0 | 1 | 29.040 s |
+| 100,098 characters | RLM | 4/4 | 11,656 | 0 | 3 | 30.707 s |
+
+On the larger task, RLM used **68.9% fewer total CLI model tokens**, while taking approximately 1.7 seconds longer. The smaller task favored direct completion in both token use and latency. All four answers matched the deterministic answer key. Model tokens are not a measurement of how many subscription credits or percentage points were consumed.
+
+## What is held constant
+
+Both sides receive the same query and deterministic source (seed 2026), use the same exact model and reasoning effort, and share the same Codex text-completion wrapper. Each model call starts a fresh ephemeral Codex session. User configuration is ignored; existing ChatGPT authentication is retained. Neither side receives answer-key feedback.
+
+Direct mode receives the full context in one model request. RLM runs the library's existing control loop and restricted Python REPL. The model receives RLM instructions and observations; the full source remains in the REPL. Therefore the mode-specific prompts differ by design. RLM call history and CLI instruction overhead count toward reported usage. Both displayed runs used the root REPL, without child RLM calls.
+
+Codex host capabilities are disabled where supported. Luna requires its Code Mode host to initialize; its presence is not counted as a tool call. Any actual tool-use item in the CLI event stream invalidates the run. Both completed pairs contain only `agent_message` items, with no Codex tool calls. Python execution occurs in the library's REPL.
+
+## Metric definitions and replay
+
+- **Total CLI model tokens:** reported input plus output over every completed model call. Cached input is included in total input, not subtracted. Incomplete usage is reported as unknown rather than as a complete total.
+- **Elapsed:** wall-clock time for each mode, including CLI startup, network time, and RLM execution. Runs were sequential; the page aligns both timelines by elapsed time for comparison.
+- **Exact fields:** count, sum of amount in cents, largest transaction ID, and its amount. No grader feedback is supplied during generation.
+- **Subscription billing:** no per-run dollar charge or inferred subscription-percentage saving is reported.
+- **Video:** real event timestamps are replayed with the same acceleration factor on both sides. Introductory diagrams explain the execution path; recorded Python and final answers come from the captured runs.
+
+The 10k run encountered REPL helper errors and recovered within the original library loop. Those events remain in the recording; the REPL was not changed to improve the experiment. An initial CLI startup failure on both sides is preserved separately as `startup-diagnostic*`; it is infrastructure evidence, not a failed model answer or a performance trial. The finalized runner marks execution failures as ungraded.
+
+## Artifacts and reproduction
+
+[Captured experiments](experiments/codex-luna-20260908/) contain the comparison JSON, raw provider events and messages, and a transport manifest with the CLI version and shared wrapper. The corpus is synthetic; no private document was used. The hashes are:
+
+- 10k: `753ce91f5a0d19351081775d4529a379358b0ff132bfb350c8f2050627f15236`
+- 100k: `1ee43f3b42f8db55369c337d2e37f1e7f61224abe3d538581a716865df8a6fcc`
+
+```bash
+pip install -e .
+codex login
+python examples/capture_codex_comparison.py --chars 10000 --output comparison-10k.json
+python examples/capture_codex_comparison.py --chars 100000 --output comparison-100k.json
+```
+
+The historical settings remain the defaults. For a new comparison with Sol 6.1,
+select the shared model and reasoning effort explicitly:
+
+```bash
+python examples/capture_codex_comparison.py --model gpt-6.1-sol \
+  --reasoning-effort low --max-depth 0 --chars 10000 --output sol-root-only.json
+```
+
+`--max-depth 0` runs the root REPL without subcalls; depth 1 permits plain model
+leaves and depth 2 permits child RLMs. Both modes receive the selected model and
+reasoning effort, and raw and replay recordings retain those settings. The web
+companion continues to use Luna, medium reasoning, and depth 1.
+
+These commands consume the signed-in account's Codex subscription allowance. [The visualizer](visualizer/README.md) can open the resulting comparison JSON locally in the browser. The companion `.raw.json` retains messages and raw events and should be reviewed before sharing a run on private source material.
+
+For a custom source, supply `--context-file` and `--query` together. Custom answers require manual review unless you provide a reference map with `--expected-fields answers.json`:
+
+```bash
+python examples/capture_codex_comparison.py \
+  --context-file document.txt --query "What are the main findings?" \
+  --output my-comparison.json
+```
+
+## Run your own document from the web page
+
+Install the optional document parser and start the companion in your own terminal:
+
+```bash
+pip install -e '.[visualizer]'
+codex login
+python examples/serve_codex_comparison.py
+```
+
+The companion listens at `http://127.0.0.1:8766` and prints a temporary pairing code. Paste that code into the page's connection form. Keep the terminal open. The code changes whenever the companion restarts; it is never saved in a recording or logged with HTTP requests. `--port` changes the local port. `--allow-origin https://your-page.example` permits an additional exact page origin.
+
+Choose a preset or upload TXT, Markdown, CSV, JSON, RST, PDF, or DOCX, then enter the question and run the comparison. Each explicit run uses the signed-in account's Codex allowance. Selecting a preset alone does not invoke a model. The companion downloads the selected preset over HTTPS on demand and verifies its SHA256 before attaching the corresponding reference answer.
+
+Files are limited to 5 MiB and extracted text to 300,000 characters. DOCX parsing reads body text and tables without extracting archive files; expanded content is limited to 20 MiB. PDF parsing extracts existing text with no OCR, so scans need OCR first and complicated tables may lose structure. The extracted preview is shown before the model results. Uploaded source text stays in the local companion and is sent through Codex when you run the comparison. The hosted page receives the preview, progress, model outputs, and evaluation fields.
+
+Preset questions have explicit reference fields. Changing a preset question disables its reference key unless you supply a new one. Custom questions without a key show **manual review**, with no invented quality score. An optional reference map is a JSON object such as `{"project_name":"Orion","total":12,"approved":true}`. The model receives only the required field names and JSON output format; expected values are withheld. Numbers are compared exactly, booleans retain their JSON type, and ordinary text ignores case, whitespace, and quote/dash typography. Whitespace-only strings are compared exactly. Missing, duplicate, or incorrectly typed fields fail their checks; transport failures remain ungraded.
+
+The page polls actual progress once per second. Only one comparison can run at a time, and Cancel interrupts the active run. Raw traces are saved in the private temporary directory printed by the companion, or a directory selected with `--output-dir`. Review private source material before sharing any recording. The companion binds only the loopback interface, requires bearer pairing on every data endpoint, and accepts browser requests only from its explicit origin allowlist.