inductive-mlxrl 0.2.4__tar.gz → 0.3.0__tar.gz

{inductive_mlxrl-0.2.4 → inductive_mlxrl-0.3.0}/CHANGELOG.md

@@ -5,6 +5,25 @@ All notable changes to this project are documented here.
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project uses semantic versioning once releases are cut.
 
+## [0.3.0] - 2026-10-30
+
+Score centering is experimental and disabled by default. Existing configurations
+continue to use GRPO. Enable it explicitly with `[algorithm] name = "score-centering"`.
+The local benchmarks do not establish a reliable quality or long-run stability gain.
+
+### Added
+
+- Experimental, opt-in single-turn score centering with actual post-filter,
+  post-temperature sampler probabilities, detached top-k/tail correction,
+  fp32 trainer probabilities, and micro-batch support.
+- Correctness gates for exact gradients, drift cancellation, sampling laws,
+  seeded capture parity across rollout engines, and padded optimizer updates.
+- A paired multi-seed 4-bit QLoRA benchmark with raw training diagnostics.
+
+### Changed
+
+- Update locked filelock to 4.2.1 and Ruff to 0.17.1, and pin setup-uv 10.3.2.
+
 ## [0.2.4] - 2026-10-22
 
 ### Security

{inductive_mlxrl-0.2.4 → inductive_mlxrl-0.3.0}/DESIGN.md

@@ -37,6 +37,13 @@ than widening the single-turn `Algorithm` contract. GiGPO is the first
 implementation: the rollout engine records env-provided state ids, while the
 algorithm owns anchor-state grouping and credit assignment.
 
+The experimental single-turn `SamplerAwareAlgorithm` sibling extends the loss
+input with a recorded sampler head and corresponding full-forward trainer
+logprobs. The engine dispatches through that protocol when a prepared batch has
+sampler records; it still imports no algorithm implementations. The first user
+is score centering, a token-mean REINFORCE objective with centered rewards.
+Existing algorithms continue through the original protocol and probability path.
+
 ## Agentic Environment Seam
 
 Agentic rollout is built around `EnvFactory(task, seed, group_index) ->
@@ -86,6 +93,17 @@ diagnostic data, not training data.
 
 ## Memory As A First-Class Constraint
 
+Experimental score centering captures the actual categorical sampling law after
+filtering and temperature, rather than treating raw model logprobs as sampler
+probabilities. Records contain the selected-token probability and a top-k head;
+the head is not renormalized. Persistent capture storage scales with completion
+tokens times k, rather than vocabulary size. All gradient-bearing candidate
+logprobs come from the trainer's full-forward path with fp32 normalization.
+The detached tail correction follows equations 8--10 of arXiv:2609.20807, using
+an exact zero sampler tail when its head exhausts all mass and a 1e-6 floor on
+the trainer tail for numerical stability. This path is opt-in, single-turn only,
+and requires separate measured memory and stability gates.
+
 Apple Silicon's unified memory is the deployment target, not an afterthought.
 The code supports per-layer MLX-LM gradient checkpointing because DeltaNet and
 linear-attention models otherwise keep O(sequence length) recurrent state live

{inductive_mlxrl-0.2.4 → inductive_mlxrl-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: inductive-mlxrl
-Version: 0.2.4
+Version: 0.3.0
 Summary: Small single-process RL post-training for LLMs on Apple Silicon with MLX.
 Project-URL: Homepage, https://github.com/inductiveML/mlxrl
 Project-URL: Repository, https://github.com/inductiveML/mlxrl
@@ -355,6 +355,55 @@ import concrete algorithm implementations.
 | GSPO | sequence-level, length-normalized importance ratio and clipping |
 | RLOO | leave-one-out group baseline, no critic, no std-normalized advantage |
 | GiGPO | multi-turn episode advantage plus anchor-state step advantage |
+| Score centering (experimental) | centered rewards, token-mean REINFORCE with sampler-score correction |
+
+### Experimental score centering
+
+Score centering is an explicit single-turn opt-in based on
+[Marek and Ryabinin, 2026](https://arxiv.org/abs/2609.20807). Existing algorithm
+defaults and old-policy recompute semantics stay unchanged.
+
+```bash
+UV_CACHE_DIR=.uv-cache uv run mlxrl train \
+  --config examples/qwen3_0_6b_score_centering.toml \
+  --available-memory-gb 48
+```
+
+The config selects the objective and the number of sampler candidates to retain:
+
+```toml
+[algorithm]
+name = "score-centering"
+score_centering_top_k = 128
+```
+
+This objective uses group-centered rewards without standard-deviation scaling.
+The sampler records probabilities **after top-p, min-p, top-k, and temperature**,
+including the selected token's probability even when it falls outside the
+recorded head. Greedy sampling is recorded as a point mass. Candidate
+probabilities retain their original mass; the trainer approximates the remaining
+tail as described in the paper. The correction coefficients are detached, and
+trainer logprobs are normalized in fp32. `kl_beta` remains the independent
+adapter-disabled reference-model penalty.
+
+For the import API, construct `ScoreCenteringAlgorithm(sampler_top_k=128)` from
+`mlxrl.algo` and use `SamplingConfig(record_sampler_top_k=128)` for generation.
+Pass the resulting completions and that algorithm to `batch_from_rollouts` and
+`optimizer_step`. Missing sampler records are rejected. `Completion.sampler_trace`
+contains the selected-token and candidate probabilities; config-driven runs also
+save `sampler_logprobs`, `sampler_topk_ids`, and `sampler_topk_logprobs` in the
+output NPZ. Ordinary `old_policy_logprobs` remain separate.
+
+Capture works with naive, prefix-cached, batched, and compiled single-turn
+rollouts and preserves seeded token draws. The objective supports token-weighted
+micro-batching. Multi-turn/ECHO training is outside this initial experiment.
+The memory preflight estimates have not been calibrated for this objective.
+Its benefit and overhead on a particular model must be measured; the paper's
+large-model results do not establish a benefit for MLX QLoRA.
+
+The reproducible paired benchmark is `benchmarks/run_score_centering.py`; its
+protocol and measured results are in
+[the 4-bit benchmark report](benchmarks/results/score_centering_4bit_2026-10-30.md).
 
 ## Memory Preflight
 

{inductive_mlxrl-0.2.4 → inductive_mlxrl-0.3.0}/README.md

@@ -330,6 +330,55 @@ import concrete algorithm implementations.
 | GSPO | sequence-level, length-normalized importance ratio and clipping |
 | RLOO | leave-one-out group baseline, no critic, no std-normalized advantage |
 | GiGPO | multi-turn episode advantage plus anchor-state step advantage |
+| Score centering (experimental) | centered rewards, token-mean REINFORCE with sampler-score correction |
+
+### Experimental score centering
+
+Score centering is an explicit single-turn opt-in based on
+[Marek and Ryabinin, 2026](https://arxiv.org/abs/2609.20807). Existing algorithm
+defaults and old-policy recompute semantics stay unchanged.
+
+```bash
+UV_CACHE_DIR=.uv-cache uv run mlxrl train \
+  --config examples/qwen3_0_6b_score_centering.toml \
+  --available-memory-gb 48
+```
+
+The config selects the objective and the number of sampler candidates to retain:
+
+```toml
+[algorithm]
+name = "score-centering"
+score_centering_top_k = 128
+```
+
+This objective uses group-centered rewards without standard-deviation scaling.
+The sampler records probabilities **after top-p, min-p, top-k, and temperature**,
+including the selected token's probability even when it falls outside the
+recorded head. Greedy sampling is recorded as a point mass. Candidate
+probabilities retain their original mass; the trainer approximates the remaining
+tail as described in the paper. The correction coefficients are detached, and
+trainer logprobs are normalized in fp32. `kl_beta` remains the independent
+adapter-disabled reference-model penalty.
+
+For the import API, construct `ScoreCenteringAlgorithm(sampler_top_k=128)` from
+`mlxrl.algo` and use `SamplingConfig(record_sampler_top_k=128)` for generation.
+Pass the resulting completions and that algorithm to `batch_from_rollouts` and
+`optimizer_step`. Missing sampler records are rejected. `Completion.sampler_trace`
+contains the selected-token and candidate probabilities; config-driven runs also
+save `sampler_logprobs`, `sampler_topk_ids`, and `sampler_topk_logprobs` in the
+output NPZ. Ordinary `old_policy_logprobs` remain separate.
+
+Capture works with naive, prefix-cached, batched, and compiled single-turn
+rollouts and preserves seeded token draws. The objective supports token-weighted
+micro-batching. Multi-turn/ECHO training is outside this initial experiment.
+The memory preflight estimates have not been calibrated for this objective.
+Its benefit and overhead on a particular model must be measured; the paper's
+large-model results do not establish a benefit for MLX QLoRA.
+
+The reproducible paired benchmark is `benchmarks/run_score_centering.py`; its
+protocol and measured results are in
+[the 4-bit benchmark report](benchmarks/results/score_centering_4bit_2026-10-30.md).
 
 ## Memory Preflight
 

{inductive_mlxrl-0.2.4 → inductive_mlxrl-0.3.0}/THIRD_PARTY_LICENSES.md

@@ -44,3 +44,12 @@ CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 - License: MIT per installed package metadata.
 - Used for: benchmark configuration inspection and external baseline adapter
   patterns only. No `mlx-lm-lora` implementation code is vendored in `mlxrl`.
+
+## Score centering research
+
+- Martin Marek and Max Ryabinin, *Score Centering Stabilizes Off-policy
+  Reinforcement Learning* (2026): https://arxiv.org/abs/2609.20807.
+- The experimental MLX objective implements the mathematics in equations 8--10
+  with independently written MLX code and explicit gradient-equivalence tests.
+  The paper is a research reference, not an installed dependency. This objective
+  is distinct from Tinker's StableReinforce service implementation.