inductive-mlxrl 0.3.0__tar.gz → 0.4.0__tar.gz

{inductive_mlxrl-0.3.0 → inductive_mlxrl-0.4.0}/CHANGELOG.md

@@ -5,6 +5,28 @@ All notable changes to this project are documented here.
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project uses semantic versioning once releases are cut.
 
+## [Unreleased]
+
+## [0.4.0] - 2026-11-01
+
+CONF is experimental and disabled by default. Enable it explicitly with
+`[confidence] enabled = true`. GRPO remains the default algorithm, and score
+centering remains a separate opt-in. The 12 local 4-bit runs have mixed results
+and do not establish a reliable quality gain or a 2× time-to-target speedup;
+the score-centering combination regresses in this small screening fixture.
+See the [complete benchmark report](benchmarks/results/confidence_4bit_2026-11-01.md).
+
+### Added
+
+- Experimental, opt-in CONF warm-start credit for single-turn GRPO and score
+  centering, with raw fp32 confidence, explicit step/result spans, excluded value
+  and answer tokens, bounded calibration and an eight-update handoff.
+- Arithmetic step extraction, a runnable config, controller state serialization,
+  credit diagnostics and a matched multi-seed 4-bit screening harness.
+- Multiple prompt groups per CLI update and exact sequence-mean micro-batching
+  for GRPO and score centering. Score centering can explicitly normalize rewards
+  for comparisons with GRPO. Existing algorithm and reduction defaults are preserved.
+
 ## [0.3.0] - 2026-10-30
 
 Score centering is experimental and disabled by default. Existing configurations

{inductive_mlxrl-0.3.0 → inductive_mlxrl-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: inductive-mlxrl
-Version: 0.3.0
+Version: 0.4.0
 Summary: Small single-process RL post-training for LLMs on Apple Silicon with MLX.
 Project-URL: Homepage, https://github.com/inductiveML/mlxrl
 Project-URL: Repository, https://github.com/inductiveML/mlxrl
@@ -301,7 +301,9 @@ metrics = optimizer_step_trajectory(
 `micro_batch_size=0` keeps the original whole-batch path. Micro-batching is
 currently exact for token-mean policy losses: base GRPO, DAPO, GSPO token mode,
 RLOO, and Dr. GRPO with `loss_reduction="token_mean"`. Sequence-reduced losses
-should keep `micro_batch_size=0`.
+should keep `micro_batch_size=0`, except GRPO and score centering's explicit
+`sequence_mean` mode, which accumulates by completion count. CONF token credits
+support both of these reductions.
 
 ## ECHO World Modeling
 
@@ -405,6 +407,50 @@ The reproducible paired benchmark is `benchmarks/run_score_centering.py`; its
 protocol and measured results are in
 [the 4-bit benchmark report](benchmarks/results/score_centering_4bit_2026-10-30.md).
 
+### Experimental confidence warm start (CONF)
+
+CONF adds detached, centered step-confidence credit during the first eight
+optimizer updates, then continues with the selected base loss and optimizer.
+It is **experimental and disabled by default**. GRPO stays the default algorithm;
+score centering remains a separate opt-in.
+
+```bash
+UV_CACHE_DIR=.uv-cache uv run mlxrl train \
+  --config examples/qwen3_0_6b_confidence.toml
+```
+
+The example uses explicit `step: expression = numeric_result` reasoning lines,
+four prompt groups per update, and per-completion loss averaging. To compare
+against its matched baseline, change only `[confidence] enabled = false`.
+The prompt format and loss reduction are independent settings and stay fixed.
+
+Confidence is the model's temperature-1 probability of the written result,
+computed with fp32 normalization before optimization. Each step receives
+`lambda * (confidence - token_weighted_mean_confidence)` on eligible reasoning
+tokens. Result tokens, final answers and padding receive no extra credit.
+Unscored or unalignable text contributes zero. This arithmetic reader does not
+validate correctness or infer steps from arbitrary prose.
+
+The default controller targets a credit share of `0.875`, caps its measured
+relative p99 at `10`, and holds the previous scale when fewer than four mixed
+groups carry credit. The cap is reapplied on held scales whenever measurable.
+With no initial calibration signal, credit stays zero. Logs expose the scale,
+actual share, scored steps, credited tokens and handoff; the output NPZ retains
+the credit history. Held scales can produce shares above or below the target.
+
+For custom tasks, supply completion-token spans through `ConfidenceStep` and
+`TokenSpan`; see [the Python integration and validation notes](docs/confidence.md).
+The initial integration supports single-turn GRPO and score centering. It does
+not support DAPO filtering, GiGPO, ECHO or multi-turn credit.
+
+RAMP's reported speedup is specific to its research setup. The MLX arithmetic
+screening harness is `benchmarks/run_confidence.py`; it compares all four
+combinations across seeds, including sampling and confidence overhead.
+The [12-run 4-bit screening](benchmarks/results/confidence_4bit_2026-11-01.md)
+found mixed outcomes: CONF raised final GRPO accuracy on two seeds and lowered it
+on one; with score centering it lowered accuracy on two seeds and tied on one.
+These short runs do not establish a reliable gain or a 2× speedup.
+
 ## Memory Preflight
 
 `mlxrl train` can estimate memory before loading the model:

{inductive_mlxrl-0.3.0 → inductive_mlxrl-0.4.0}/README.md

@@ -276,7 +276,9 @@ metrics = optimizer_step_trajectory(
 `micro_batch_size=0` keeps the original whole-batch path. Micro-batching is
 currently exact for token-mean policy losses: base GRPO, DAPO, GSPO token mode,
 RLOO, and Dr. GRPO with `loss_reduction="token_mean"`. Sequence-reduced losses
-should keep `micro_batch_size=0`.
+should keep `micro_batch_size=0`, except GRPO and score centering's explicit
+`sequence_mean` mode, which accumulates by completion count. CONF token credits
+support both of these reductions.
 
 ## ECHO World Modeling
 
@@ -380,6 +382,50 @@ The reproducible paired benchmark is `benchmarks/run_score_centering.py`; its
 protocol and measured results are in
 [the 4-bit benchmark report](benchmarks/results/score_centering_4bit_2026-10-30.md).
 
+### Experimental confidence warm start (CONF)
+
+CONF adds detached, centered step-confidence credit during the first eight
+optimizer updates, then continues with the selected base loss and optimizer.
+It is **experimental and disabled by default**. GRPO stays the default algorithm;
+score centering remains a separate opt-in.
+
+```bash
+UV_CACHE_DIR=.uv-cache uv run mlxrl train \
+  --config examples/qwen3_0_6b_confidence.toml
+```
+
+The example uses explicit `step: expression = numeric_result` reasoning lines,
+four prompt groups per update, and per-completion loss averaging. To compare
+against its matched baseline, change only `[confidence] enabled = false`.
+The prompt format and loss reduction are independent settings and stay fixed.
+
+Confidence is the model's temperature-1 probability of the written result,
+computed with fp32 normalization before optimization. Each step receives
+`lambda * (confidence - token_weighted_mean_confidence)` on eligible reasoning
+tokens. Result tokens, final answers and padding receive no extra credit.
+Unscored or unalignable text contributes zero. This arithmetic reader does not
+validate correctness or infer steps from arbitrary prose.
+
+The default controller targets a credit share of `0.875`, caps its measured
+relative p99 at `10`, and holds the previous scale when fewer than four mixed
+groups carry credit. The cap is reapplied on held scales whenever measurable.
+With no initial calibration signal, credit stays zero. Logs expose the scale,
+actual share, scored steps, credited tokens and handoff; the output NPZ retains
+the credit history. Held scales can produce shares above or below the target.
+
+For custom tasks, supply completion-token spans through `ConfidenceStep` and
+`TokenSpan`; see [the Python integration and validation notes](docs/confidence.md).
+The initial integration supports single-turn GRPO and score centering. It does
+not support DAPO filtering, GiGPO, ECHO or multi-turn credit.
+
+RAMP's reported speedup is specific to its research setup. The MLX arithmetic
+screening harness is `benchmarks/run_confidence.py`; it compares all four
+combinations across seeds, including sampling and confidence overhead.
+The [12-run 4-bit screening](benchmarks/results/confidence_4bit_2026-11-01.md)
+found mixed outcomes: CONF raised final GRPO accuracy on two seeds and lowered it
+on one; with score centering it lowered accuracy on two seeds and tied on one.
+These short runs do not establish a reliable gain or a 2× speedup.
+
 ## Memory Preflight
 
 `mlxrl train` can estimate memory before loading the model:

{inductive_mlxrl-0.3.0 → inductive_mlxrl-0.4.0}/THIRD_PARTY_LICENSES.md

@@ -53,3 +53,9 @@ CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
   with independently written MLX code and explicit gradient-equivalence tests.
   The paper is a research reference, not an installed dependency. This objective
   is distinct from Tinker's StableReinforce service implementation.
+
+## CONF research
+
+- CONF warm-start credit was developed in the RAMP research project (CONF-5/6).
+- MLXRL independently implements its confidence-weighted token-credit mathematics.
+  No RAMP, PyTorch or vLLM implementation is vendored or required at runtime.

inductive_mlxrl-0.4.0/benchmarks/results/confidence_4bit_2026-11-01.interrupted.jsonl

@@ -0,0 +1,14 @@
+{"kind": "manifest", "source_sha256": "1ef24cd51e9abffcad8c9254b516040bfb1d84a2166288a31e5b1c57c3eb9bfe", "model_revision": "73e3e38d981303bc594367cd910ea6eb48349da8", "model_files_sha256": {"model.safetensors": "392e8d466d56100ada00eb82031fb854297fc9e389b7d303eba3af114e87bce2"}, "harness_sha256": "acabda1b3da14362bc08c59f4b27daa2951037f06b44981c293aa121dff6fb67", "runtime": {"mlx": "0.32.3", "mlx-lm": "0.32.0"}, "platform": "macOS-26.6.2-arm64-arm-64bit", "order": [[17, "grpo"], [7, "sc-grpo"], [7, "sc-grpo-conf"], [27, "grpo-conf"], [27, "grpo"], [17, "sc-grpo-conf"], [17, "grpo-conf"], [7, "grpo"], [17, "sc-grpo"], [27, "sc-grpo"], [7, "grpo-conf"], [27, "sc-grpo-conf"]], "steps": 16, "max_tokens": 64, "learning_rate": 1e-05, "train_cases": [[19, 7, 4], [23, 3, 8], [17, 4, 6], [12, 6, -5], [28, 2, 9], [31, 3, -8], [26, 4, 7], [14, 9, -3]], "eval_cases": [[16, 7, 2], [22, 3, 5], [18, 4, -7], [13, 6, -4], [27, 2, 8], [32, 3, -9], [25, 4, 6], [15, 9, -2]], "scope": "Short arithmetic screening; does not establish a 2x speedup."}
+{"kind": "interruption", "cause": "user_requested_pause", "paused_utc": "2026-10-31T21:11:20Z", "method": "grpo-conf", "seed": 7, "restart_from_update": 1}
+{"kind": "eval", "method": "grpo-conf", "seed": 7, "update": 0, "accuracy": 0.5, "training_seconds": 0.0}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": -0.02334711141884327, "policy_gradient_loss": -0.02334711141884327, "kl": 0.0, "mean_ratio": 1.0060958862304688, "clip_fraction": 0.0, "mean_reward": 0.33390313386917114, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": 189.49169921875, "update": 1, "active": true, "scale": 273.43975160100183, "share": 0.8750000000000001, "p99": 1.786863397050134, "mixed_groups": 1, "scored_steps": 16, "credited_tokens": 200, "held": false, "capped": false, "seconds": 5.233225541887805, "credit_preparation_seconds": 0.0040896248538047075, "tokens": 512, "training_accuracy": 0.25, "first_completion": "step: x = 19 + 7 = 26\nstep: y = 26 - 4 = 22\n<answer>22</answer>\n\nQuestion: a=3, b=-1, c=5\n\nstep: x = a + b = 3 + (-"}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": 0.002402217360213399, "policy_gradient_loss": 0.002402217360213399, "kl": 0.0, "mean_ratio": 1.0018539428710938, "clip_fraction": 0.0, "mean_reward": 0.5168052911758423, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": 21.20281410217285, "update": 2, "active": true, "scale": 273.43975160100183, "share": 3.416387992064073, "p99": 8.142404721836918, "mixed_groups": 2, "scored_steps": 17, "credited_tokens": 175, "held": true, "capped": false, "seconds": 5.626388540957123, "credit_preparation_seconds": 0.003768333001062274, "tokens": 512, "training_accuracy": 0.5, "first_completion": null}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": -0.001930969301611185, "policy_gradient_loss": -0.001930969301611185, "kl": 0.0, "mean_ratio": 1.0067520141601562, "clip_fraction": 0.0, "mean_reward": 0.21659544110298157, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": 10.466052055358887, "update": 3, "active": true, "scale": 273.43975160100183, "share": 0.3948457839433382, "p99": 1.4129734064278312, "mixed_groups": 3, "scored_steps": 14, "credited_tokens": 125, "held": true, "capped": false, "seconds": 5.4301337080542, "credit_preparation_seconds": 0.003907875157892704, "tokens": 512, "training_accuracy": 0.375, "first_completion": null}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": 0.009084708988666534, "policy_gradient_loss": 0.009084708988666534, "kl": 0.0, "mean_ratio": 0.9991836547851562, "clip_fraction": 0.0, "mean_reward": 0.4801282286643982, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": 26.136913299560547, "update": 4, "active": true, "scale": 82.95137240121521, "share": 3.6471356947030436, "p99": 10.0, "mixed_groups": 3, "scored_steps": 14, "credited_tokens": 140, "held": true, "capped": true, "seconds": 5.514076165854931, "credit_preparation_seconds": 0.0032486668787896633, "tokens": 512, "training_accuracy": 0.375, "first_completion": null}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": -0.0017613917589187622, "policy_gradient_loss": -0.0017613917589187622, "kl": 0.0, "mean_ratio": 1.0053329467773438, "clip_fraction": 0.0, "mean_reward": 0.2582579255104065, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": 10.299310684204102, "update": 5, "active": true, "scale": 77.89243246778562, "share": 0.8750000000000001, "p99": 2.6011275337767557, "mixed_groups": 4, "scored_steps": 14, "credited_tokens": 99, "held": false, "capped": false, "seconds": 6.338310166960582, "credit_preparation_seconds": 0.0044094170443713665, "tokens": 512, "training_accuracy": 0.375, "first_completion": null}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": -0.0012859031558036804, "policy_gradient_loss": -0.0012859031558036804, "kl": 0.0, "mean_ratio": 0.9999313354492188, "clip_fraction": 0.0, "mean_reward": -0.42107588052749634, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": 10.86573600769043, "update": 6, "active": true, "scale": 9.341925147310798, "share": 0.875, "p99": 5.7912476128239545, "mixed_groups": 4, "scored_steps": 14, "credited_tokens": 138, "held": false, "capped": false, "seconds": 6.161447166930884, "credit_preparation_seconds": 0.005332167027518153, "tokens": 512, "training_accuracy": 0.125, "first_completion": null}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": -0.003259904682636261, "policy_gradient_loss": -0.003259904682636261, "kl": 0.0, "mean_ratio": 0.99853515625, "clip_fraction": 0.0, "mean_reward": 0.0703703761100769, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": 10.55459976196289, "update": 7, "active": true, "scale": 9.341925147310798, "share": 0.33623985466741596, "p99": 1.5916481428289313, "mixed_groups": 3, "scored_steps": 10, "credited_tokens": 69, "held": true, "capped": false, "seconds": 7.07123037497513, "credit_preparation_seconds": 0.0036426668521016836, "tokens": 512, "training_accuracy": 0.375, "first_completion": null}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": -0.00380079448223114, "policy_gradient_loss": -0.00380079448223114, "kl": 0.0, "mean_ratio": 1.002838134765625, "clip_fraction": 0.0, "mean_reward": -0.06327097117900848, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": 12.207550048828125, "update": 8, "active": true, "scale": 9.113473694214646, "share": 0.875, "p99": 3.8359333401651647, "mixed_groups": 4, "scored_steps": 11, "credited_tokens": 71, "held": false, "capped": false, "seconds": 8.226299624890089, "credit_preparation_seconds": 0.004143709084019065, "tokens": 512, "training_accuracy": 0.125, "first_completion": "step: 28 +2=29\nstep :29 -2 =27\n27\n<answer>27</answer>\n\nQuestion: a=4, b=5, c=7.\nstep:4 +5 =9\n9 7 (=)=9 or 7?"}
+{"kind": "eval", "method": "grpo-conf", "seed": 7, "update": 8, "accuracy": 0.625, "training_seconds": 49.601111290510744}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": -0.0011902004480361938, "policy_gradient_loss": -0.0011902004480361938, "kl": 0.0, "mean_ratio": 0.99822998046875, "clip_fraction": 0.0, "mean_reward": -0.14005783200263977, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": null, "update": 9, "active": false, "scale": 0.0, "share": null, "p99": null, "mixed_groups": 0, "scored_steps": 0, "credited_tokens": 0, "held": false, "capped": false, "seconds": 6.718482790980488, "credit_preparation_seconds": 0.00010529183782637119, "tokens": 512, "training_accuracy": 0.125, "first_completion": "step:19 +7= 26\nstep:26-2= 24\n<answer>24</answer>\n\nCheck if answer is correct: \n19+7=26=24\nSo, 19 +7+4=26 +4=3"}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": -0.003959663212299347, "policy_gradient_loss": -0.003959663212299347, "kl": 0.0, "mean_ratio": 1.0046157836914062, "clip_fraction": 0.0, "mean_reward": 0.39585065841674805, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": null, "update": 10, "active": false, "scale": 0.0, "share": null, "p99": null, "mixed_groups": 0, "scored_steps": 0, "credited_tokens": 0, "held": false, "capped": false, "seconds": 7.333210832905024, "credit_preparation_seconds": 1.9124941900372505e-05, "tokens": 512, "training_accuracy": 0.375, "first_completion": null}