miniverl 0.2.0__tar.gz

miniverl-0.2.0/.gitignore

@@ -0,0 +1,52 @@
+# Every directory rule below is anchored with a leading slash so it can only
+# match at the repository root. An unanchored `models/` once matched
+# `src/miniverl/models/` and silently excluded that package from both git and
+# the built wheel; the clean-environment install in .github/workflows/build.yml
+# is what caught it.
+
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+/build/
+/dist/
+/.venv/
+/venv/
+.python-version
+
+# Tooling caches
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.hypothesis/
+.coverage
+.coverage.*
+coverage.xml
+htmlcov/
+
+# Run artifacts. Never commit weights, teacher caches or checkpoints.
+/runs/
+/artifacts/
+*.safetensors
+*.bin
+*.pt
+*.ckpt
+*.sqlite
+*.db
+
+# Local model and dataset caches (root level only)
+/.hf-cache/
+/hf_cache/
+/models/
+/checkpoints/
+/teacher-cache/
+
+# Editors / OS
+.vscode/
+.idea/
+.DS_Store
+Thumbs.db
+
+# Committed benchmark results are small, schema-validated JSON and are wanted.
+!benchmarks/results/*.json

miniverl-0.2.0/CHANGELOG.md

@@ -0,0 +1,188 @@
+# Changelog
+
+All notable changes to miniVERL are recorded here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses
+[semantic versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.2.0] - 2026-07-28
+
+Protocol-aligned teacher support and scientifically explicit benchmark
+accounting.
+
+### Added
+
+- Standard frozen PEFT teacher-adapter validation, loading and
+  `miniverl export-adapter`, with base/tokenizer compatibility checks,
+  checksums, run/checkpoint provenance and an optional tool-policy competence
+  gate.
+- An executable Qwen3-1.7B QLoRA protocol-teacher recipe using deterministic
+  oracle traces in the same tool protocol as the student.
+- Benchmark schema/config v2 with explicit common versus cold-start overrides,
+  pre-allocation structured config diffs, resolved-config/checkpoint digests,
+  mode-aware objectives, cumulative accounting and separate train/eval/wall
+  timings. Existing schema-v1 measurements remain readable and unchanged.
+- Policy competence metrics: strict success, diagnostic lenient success, valid
+  tool-call rate/count, final-answer format validity and average turns.
+- A deterministic temperature-gradient sweep across forward KL, reverse KL and
+  JSD in near-uniform and sharply peaked regimes.
+- CI compatibility rows for Transformers 4.51.x and 5.x, plus a disabled OIDC
+  PyPI publishing job.
+- A measured five-arm, two-seed RTX 4080 comparison and data-bound SVG: the
+  protocol-trained teacher prevents the 0% raw/privileged-teacher collapse and
+  reaches 100% on both seeds, tying rather than beating SFT.
+- A public, checksum-validated protocol-teacher adapter on the Hugging Face Hub,
+  pinned by the benchmark to an immutable revision with a local/offline config.
+- Destructive trainer lifecycle tests, including weak-reference coverage and a
+  measured sequential CUDA-allocation regression.
+- A strict `--offline` contract shared by train, benchmark, standalone
+  evaluation and adapter export, including cached pinned Hub adapters and
+  socket-denial regression tests.
+- A tag-only release supply chain that builds wheel and sdist once, publishes
+  those exact artifacts with OIDC attestations, verifies public PyPI metadata,
+  hashes, provenance and a clean install, then creates the GitHub Release.
+
+### Changed
+
+- Strict OPD freshness is the default and permits exactly one optimizer update
+  per newly sampled rollout batch. Explicit replay is labeled
+  `online_distillation_with_replay` and is never reported as genuine OPD.
+- `loss.sampled_token_nll_weight` replaces ambiguous distillation uses of
+  `loss.ce_weight`; its labels are explicitly the student's sampled tokens.
+- `k == V` now bypasses epsilon tail smoothing and reduces to the exact
+  full-vocabulary objective.
+- Lower-bound and `T^2` documentation now distinguishes mathematical theorems
+  from the epsilon-smoothed implementation and reverse-KL/JSD heuristics.
+- Qwen3's verified minimum dependency is `transformers>=4.51,<6`.
+- GitHub Actions are pinned to full v7 commit SHAs, and release validation runs
+  the complete CPU scientific test suite.
+- Source installation is the primary README path until a real PyPI publication
+  exists.
+- Published schema-v2 provenance replaces machine-local absolute paths before
+  hashing or rendering artifacts.
+- `OPDTrainer.close()` now destructively and idempotently releases target
+  providers, scorer, runner, optimizer, teacher, student, environment and CUDA
+  allocator state; public operations fail with `LifecycleError` after close.
+- Cold starts and benchmark arms now run inside isolated function-level trainer
+  contexts, with garbage collection and CUDA cache release between arms.
+- New schema-v2 output separates declared scientific differences,
+  runtime-resolution decisions and harness-only bookkeeping while retaining
+  compatibility fields for existing readers.
+- Hub teacher validation now returns the exact resolved local snapshot and
+  PEFT loads only that directory, preventing a second independent Hub
+  resolution after checksum validation.
+
+### Corrected
+
+- The legacy RTX 4080 result now has an explicit erratum: its shared cold start
+  used `medium`, continuations/evaluation used `hard`, the old `controlled`
+  block came from the base recipe, selected-token fields held only the final
+  cycle and SFT's teacher-query ratio was not meaningful.
+- Periodic evaluation now honors `eval.enabled: false`; the incomplete GPU
+  attempt that exposed the defect was preserved and the full benchmark rerun.
+- Hub teacher adapters now download and validate the miniVERL manifest and
+  checksums at the pinned revision instead of losing the competence record.
+- Legacy schema-v1 commands now point to their immutable source commit, and
+  historical `peak_reserved_bytes` are explicitly caveated rather than
+  silently rewritten.
+
+## [0.1.0] - 2026-07-27
+
+First public release. Multi-turn, tool-aware on-policy distillation that runs on
+one consumer GPU.
+
+### Added
+
+**Objective**
+
+- Exact full-vocabulary forward KL, reverse KL and beta-weighted Jensen-Shannon
+  divergence, each checked against a brute-force Python reference.
+- Compressed `top-k + tail` variants, named `bucketed_forward_kl`,
+  `bucketed_reverse_kl` and `bucketed_jsd` so they cannot be mistaken for the
+  exact objective. Tests assert the data-processing-inequality lower bound and
+  convergence to the exact loss at `k == V`.
+- Optional temperature with the documented `T^2` gradient correction, per-token
+  weights, weight-sum normalization, and a convex cross-entropy mixing term.
+- A chunked selected-position objective whose two-stage backward reproduces the
+  unchunked gradient exactly while never materializing more than
+  `[chunk_size, vocab]`.
+
+**Provenance**
+
+- Span-partitioned trajectories with `system`, `user`, `assistant_text`,
+  `assistant_tool_call`, `tool_result` and `assistant_final` types. Masks are
+  stored and re-derived on every read; a mismatch is rejected.
+- Explicit target-position versus prediction-position conversion, with position
+  `0` permanently excluded.
+- Privileged-context teacher mode with a per-segment alignment map that verifies
+  target-token identity on both sides.
+
+**Backends**
+
+- A reversible ~190-entry toy tokenizer and a RoPE/RMSNorm/SwiGLU toy
+  transformer, so the whole pipeline runs on a CPU with no network.
+- A Hugging Face causal-LM backend that calls the decoder backbone directly and
+  projects only the selected positions, with QLoRA (NF4), gradient
+  checkpointing, SDPA attention and deterministic generation.
+- An architecture adapter that resolves the backbone and LM head through PEFT
+  wrappers; tested against `Qwen3ForCausalLM`.
+
+**Training**
+
+- SFT, offline KD and genuine OPD behind one trainer, with the on-policy
+  distinction enforced by policy-version checks rather than documentation.
+- `resident`, `swap` and `auto` memory strategies; bounded, mathematically
+  neutral OOM retries that only shrink the projection chunk.
+- Pickle-free checkpoints (safetensors plus JSON) and exact resume, asserted
+  parameter-for-parameter against an uninterrupted run.
+
+**Environments**
+
+- Calculator (AST-only evaluation, no `eval`), JSON navigation, and SQLite with
+  an authorizer-enforced read-only connection and an instruction budget. All
+  three are seeded, have disjoint splits, exact verifiers and deterministic
+  oracles.
+
+**Tooling**
+
+- `miniverl doctor / validate / demo / train / eval / benchmark / inspect /
+  report / cache / export-benchmark / schema`, all with JSON output where
+  useful, and a `--dry-run` path that downloads nothing.
+- A versioned, checksummed teacher-target cache with compression statistics and
+  corruption detection, readable without torch.
+- Self-contained offline HTML reports with a token-level teacher/student
+  divergence view, plus Markdown and JSON summaries.
+- A matched-budget benchmark harness that starts every arm from one shared
+  cold-start checkpoint and records what it held constant.
+- `scripts/attribute_failures.py`, which re-scores collected trajectories with a
+  lenient answer parser so a zero success rate can be split into "could not do
+  the task" and "did the task and formatted it wrong".
+
+### Measured
+
+- RTX 4080 (16 GB), `recipes/qwen_consumer_gpu_calc.yaml`: 16 optimizer steps in
+  481.1 s; peak 4.251 GiB allocated / 4.762 GiB reserved; held-out greedy task
+  success 0.0% to 100.0% on 12 tasks. The supervised cold start does most of that
+  work; see `docs/rtx4080-baselines.md`.
+- Decode throughput on the same machine is kernel-launch bound: 11.19 tok/s with
+  NF4 and 12.84 tok/s with bf16 LoRA, and a 14-token prefill (37.0 ms) costs
+  about the same as a cached single-token step (30.9 ms).
+- **The matched-budget comparison on the non-saturating `hard` split came out
+  negative for on-policy distillation.** From one shared cold start at 62.5%,
+  with 12 identical optimizer steps: supervised continuation 100.0%, OPD against
+  the raw instruct teacher 0.0%, OPD against a privileged-context teacher 0.0%.
+  The failure is diagnosed from decoded transcripts in
+  `docs/rtx4080-baselines.md` and is most likely caused by the teacher, which was
+  never trained on the tool protocol; the experiment that would confirm it is
+  specified there but was not run. This is a single seed on one task family and
+  is not a general claim about the method -- but it is what this repository
+  measured about its own headline feature, so it is reported here rather than
+  only in the docs.
+
+### Known limitations
+
+Same-tokenizer only; one trajectory per forward pass; `swap` unavailable for
+quantized models; only Qwen3 and Qwen2 architectures tested; single-seed GPU
+results. The full list is in `docs/limitations.md`.
+
+[0.2.0]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.1.0...v0.2.0
+[0.1.0]: https://github.com/DaoyuanLi2816/mini-verl/releases/tag/v0.1.0

miniverl-0.2.0/CITATION.cff

@@ -0,0 +1,99 @@
+cff-version: 1.2.0
+title: "miniVERL: On-policy distillation for tool-using agents on one GPU"
+message: "If you use miniVERL in your work, please cite it as below."
+type: software
+version: 0.2.0
+date-released: 2026-07-28
+license: Apache-2.0
+repository-code: "https://github.com/DaoyuanLi2816/mini-verl"
+url: "https://github.com/DaoyuanLi2816/mini-verl"
+abstract: >-
+  miniVERL is a single-GPU laboratory for multi-turn, tool-aware on-policy
+  distillation. A student language model samples its own tool-using
+  trajectories against deterministic local environments; a teacher scores
+  exactly the states the student visited; and the student is updated with
+  token-level distributional supervision on its own generated tokens only.
+  It implements exact full-vocabulary forward KL, reverse KL and
+  Jensen-Shannon divergences as well as a compressed top-k plus tail
+  coarse-graining, enforces per-token provenance so that tool output can never
+  become a training label, and stores teacher targets in a versioned,
+  checksummed, pickle-free cache with policy-version enforcement. It is
+  designed and measured for a single 16 GB consumer GPU and requires neither
+  Ray nor a cluster.
+authors:
+  - family-names: Li
+    given-names: Daoyuan
+    email: lidaoyuan2816@gmail.com
+keywords:
+  - on-policy distillation
+  - knowledge distillation
+  - large language models
+  - tool use
+  - agents
+  - QLoRA
+  - consumer GPU
+  - post-training
+  - reproducibility
+references:
+  - type: article
+    title: "Distilling the Knowledge in a Neural Network"
+    authors:
+      - family-names: Hinton
+        given-names: Geoffrey
+      - family-names: Vinyals
+        given-names: Oriol
+      - family-names: Dean
+        given-names: Jeff
+    year: 2015
+    url: "https://arxiv.org/abs/1503.02531"
+    notes: "Source of the high-temperature forward-KL correction; reverse-KL and JSD use it only as an explicit heuristic."
+  - type: article
+    title: >-
+      On-Policy Distillation of Language Models: Learning from Self-Generated
+      Mistakes
+    authors:
+      - family-names: Agarwal
+        given-names: Rishabh
+    year: 2024
+    url: "https://arxiv.org/abs/2306.13649"
+    notes: "Generalized JSD and training on student-sampled sequences."
+  - type: article
+    title: "On-Policy Context Distillation for Language Models"
+    authors:
+      - family-names: Ye
+        given-names: Tianzhu
+      - family-names: Dong
+        given-names: Li
+      - family-names: Wu
+        given-names: Xun
+      - family-names: Huang
+        given-names: Shaohan
+      - family-names: Wei
+        given-names: Furu
+    year: 2026
+    url: "https://arxiv.org/abs/2602.12275"
+    notes: "The privileged-context teacher mode follows this formulation."
+  - type: article
+    title: "Entropy-Aware On-Policy Distillation of Language Models"
+    authors:
+      - family-names: Jin
+        given-names: Woogyeol
+      - family-names: Min
+        given-names: Taywon
+      - family-names: Yang
+        given-names: Yongjin
+      - family-names: Wei
+        given-names: Dennis
+      - family-names: Zhou
+        given-names: Yi
+      - family-names: Kadhe
+        given-names: Swanand Ravindra
+      - family-names: Baracaldo
+        given-names: Nathalie
+      - family-names: Lee
+        given-names: Kimin
+    year: 2026
+    url: "https://arxiv.org/abs/2603.07079"
+    notes: >-
+      Motivates recording per-token teacher entropy. Entropy-aware divergence
+      mixing is a roadmap item and is not implemented in miniVERL v0.2.0.

miniverl-0.2.0/LICENSE

@@ -0,0 +1,216 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+   Copyright 2026 Daoyuan Li
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.