inductive-mlxrl 0.2.4__tar.gz → 0.4.0__tar.gz

{inductive_mlxrl-0.2.4 → inductive_mlxrl-0.4.0}/CHANGELOG.md

@@ -5,6 +5,47 @@ All notable changes to this project are documented here.
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project uses semantic versioning once releases are cut.
 
+## [Unreleased]
+
+## [0.4.0] - 2026-11-01
+
+CONF is experimental and disabled by default. Enable it explicitly with
+`[confidence] enabled = true`. GRPO remains the default algorithm, and score
+centering remains a separate opt-in. The 12 local 4-bit runs have mixed results
+and do not establish a reliable quality gain or a 2× time-to-target speedup;
+the score-centering combination regresses in this small screening fixture.
+See the [complete benchmark report](benchmarks/results/confidence_4bit_2026-11-01.md).
+
+### Added
+
+- Experimental, opt-in CONF warm-start credit for single-turn GRPO and score
+  centering, with raw fp32 confidence, explicit step/result spans, excluded value
+  and answer tokens, bounded calibration and an eight-update handoff.
+- Arithmetic step extraction, a runnable config, controller state serialization,
+  credit diagnostics and a matched multi-seed 4-bit screening harness.
+- Multiple prompt groups per CLI update and exact sequence-mean micro-batching
+  for GRPO and score centering. Score centering can explicitly normalize rewards
+  for comparisons with GRPO. Existing algorithm and reduction defaults are preserved.
+
+## [0.3.0] - 2026-10-30
+
+Score centering is experimental and disabled by default. Existing configurations
+continue to use GRPO. Enable it explicitly with `[algorithm] name = "score-centering"`.
+The local benchmarks do not establish a reliable quality or long-run stability gain.
+
+### Added
+
+- Experimental, opt-in single-turn score centering with actual post-filter,
+  post-temperature sampler probabilities, detached top-k/tail correction,
+  fp32 trainer probabilities, and micro-batch support.
+- Correctness gates for exact gradients, drift cancellation, sampling laws,
+  seeded capture parity across rollout engines, and padded optimizer updates.
+- A paired multi-seed 4-bit QLoRA benchmark with raw training diagnostics.
+
+### Changed
+
+- Update locked filelock to 4.2.1 and Ruff to 0.17.1, and pin setup-uv 10.3.2.
+
 ## [0.2.4] - 2026-10-22
 
 ### Security

{inductive_mlxrl-0.2.4 → inductive_mlxrl-0.4.0}/DESIGN.md

@@ -37,6 +37,13 @@ than widening the single-turn `Algorithm` contract. GiGPO is the first
 implementation: the rollout engine records env-provided state ids, while the
 algorithm owns anchor-state grouping and credit assignment.
 
+The experimental single-turn `SamplerAwareAlgorithm` sibling extends the loss
+input with a recorded sampler head and corresponding full-forward trainer
+logprobs. The engine dispatches through that protocol when a prepared batch has
+sampler records; it still imports no algorithm implementations. The first user
+is score centering, a token-mean REINFORCE objective with centered rewards.
+Existing algorithms continue through the original protocol and probability path.
+
 ## Agentic Environment Seam
 
 Agentic rollout is built around `EnvFactory(task, seed, group_index) ->
@@ -86,6 +93,17 @@ diagnostic data, not training data.
 
 ## Memory As A First-Class Constraint
 
+Experimental score centering captures the actual categorical sampling law after
+filtering and temperature, rather than treating raw model logprobs as sampler
+probabilities. Records contain the selected-token probability and a top-k head;
+the head is not renormalized. Persistent capture storage scales with completion
+tokens times k, rather than vocabulary size. All gradient-bearing candidate
+logprobs come from the trainer's full-forward path with fp32 normalization.
+The detached tail correction follows equations 8--10 of arXiv:2609.20807, using
+an exact zero sampler tail when its head exhausts all mass and a 1e-6 floor on
+the trainer tail for numerical stability. This path is opt-in, single-turn only,
+and requires separate measured memory and stability gates.
+
 Apple Silicon's unified memory is the deployment target, not an afterthought.
 The code supports per-layer MLX-LM gradient checkpointing because DeltaNet and
 linear-attention models otherwise keep O(sequence length) recurrent state live

{inductive_mlxrl-0.2.4 → inductive_mlxrl-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: inductive-mlxrl
-Version: 0.2.4
+Version: 0.4.0
 Summary: Small single-process RL post-training for LLMs on Apple Silicon with MLX.
 Project-URL: Homepage, https://github.com/inductiveML/mlxrl
 Project-URL: Repository, https://github.com/inductiveML/mlxrl
@@ -301,7 +301,9 @@ metrics = optimizer_step_trajectory(
 `micro_batch_size=0` keeps the original whole-batch path. Micro-batching is
 currently exact for token-mean policy losses: base GRPO, DAPO, GSPO token mode,
 RLOO, and Dr. GRPO with `loss_reduction="token_mean"`. Sequence-reduced losses
-should keep `micro_batch_size=0`.
+should keep `micro_batch_size=0`, except GRPO and score centering's explicit
+`sequence_mean` mode, which accumulates by completion count. CONF token credits
+support both of these reductions.
 
 ## ECHO World Modeling
 
@@ -355,6 +357,99 @@ import concrete algorithm implementations.
 | GSPO | sequence-level, length-normalized importance ratio and clipping |
 | RLOO | leave-one-out group baseline, no critic, no std-normalized advantage |
 | GiGPO | multi-turn episode advantage plus anchor-state step advantage |
+| Score centering (experimental) | centered rewards, token-mean REINFORCE with sampler-score correction |
+
+### Experimental score centering
+
+Score centering is an explicit single-turn opt-in based on
+[Marek and Ryabinin, 2026](https://arxiv.org/abs/2609.20807). Existing algorithm
+defaults and old-policy recompute semantics stay unchanged.
+
+```bash
+UV_CACHE_DIR=.uv-cache uv run mlxrl train \
+  --config examples/qwen3_0_6b_score_centering.toml \
+  --available-memory-gb 48
+```
+
+The config selects the objective and the number of sampler candidates to retain:
+
+```toml
+[algorithm]
+name = "score-centering"
+score_centering_top_k = 128
+```
+
+This objective uses group-centered rewards without standard-deviation scaling.
+The sampler records probabilities **after top-p, min-p, top-k, and temperature**,
+including the selected token's probability even when it falls outside the
+recorded head. Greedy sampling is recorded as a point mass. Candidate
+probabilities retain their original mass; the trainer approximates the remaining
+tail as described in the paper. The correction coefficients are detached, and
+trainer logprobs are normalized in fp32. `kl_beta` remains the independent
+adapter-disabled reference-model penalty.
+
+For the import API, construct `ScoreCenteringAlgorithm(sampler_top_k=128)` from
+`mlxrl.algo` and use `SamplingConfig(record_sampler_top_k=128)` for generation.
+Pass the resulting completions and that algorithm to `batch_from_rollouts` and
+`optimizer_step`. Missing sampler records are rejected. `Completion.sampler_trace`
+contains the selected-token and candidate probabilities; config-driven runs also
+save `sampler_logprobs`, `sampler_topk_ids`, and `sampler_topk_logprobs` in the
+output NPZ. Ordinary `old_policy_logprobs` remain separate.
+
+Capture works with naive, prefix-cached, batched, and compiled single-turn
+rollouts and preserves seeded token draws. The objective supports token-weighted
+micro-batching. Multi-turn/ECHO training is outside this initial experiment.
+The memory preflight estimates have not been calibrated for this objective.
+Its benefit and overhead on a particular model must be measured; the paper's
+large-model results do not establish a benefit for MLX QLoRA.
+
+The reproducible paired benchmark is `benchmarks/run_score_centering.py`; its
+protocol and measured results are in
+[the 4-bit benchmark report](benchmarks/results/score_centering_4bit_2026-10-30.md).
+
+### Experimental confidence warm start (CONF)
+
+CONF adds detached, centered step-confidence credit during the first eight
+optimizer updates, then continues with the selected base loss and optimizer.
+It is **experimental and disabled by default**. GRPO stays the default algorithm;
+score centering remains a separate opt-in.
+
+```bash
+UV_CACHE_DIR=.uv-cache uv run mlxrl train \
+  --config examples/qwen3_0_6b_confidence.toml
+```
+
+The example uses explicit `step: expression = numeric_result` reasoning lines,
+four prompt groups per update, and per-completion loss averaging. To compare
+against its matched baseline, change only `[confidence] enabled = false`.
+The prompt format and loss reduction are independent settings and stay fixed.
+
+Confidence is the model's temperature-1 probability of the written result,
+computed with fp32 normalization before optimization. Each step receives
+`lambda * (confidence - token_weighted_mean_confidence)` on eligible reasoning
+tokens. Result tokens, final answers and padding receive no extra credit.
+Unscored or unalignable text contributes zero. This arithmetic reader does not
+validate correctness or infer steps from arbitrary prose.
+
+The default controller targets a credit share of `0.875`, caps its measured
+relative p99 at `10`, and holds the previous scale when fewer than four mixed
+groups carry credit. The cap is reapplied on held scales whenever measurable.
+With no initial calibration signal, credit stays zero. Logs expose the scale,
+actual share, scored steps, credited tokens and handoff; the output NPZ retains
+the credit history. Held scales can produce shares above or below the target.
+
+For custom tasks, supply completion-token spans through `ConfidenceStep` and
+`TokenSpan`; see [the Python integration and validation notes](docs/confidence.md).
+The initial integration supports single-turn GRPO and score centering. It does
+not support DAPO filtering, GiGPO, ECHO or multi-turn credit.
+
+RAMP's reported speedup is specific to its research setup. The MLX arithmetic
+screening harness is `benchmarks/run_confidence.py`; it compares all four
+combinations across seeds, including sampling and confidence overhead.
+The [12-run 4-bit screening](benchmarks/results/confidence_4bit_2026-11-01.md)
+found mixed outcomes: CONF raised final GRPO accuracy on two seeds and lowered it
+on one; with score centering it lowered accuracy on two seeds and tied on one.
+These short runs do not establish a reliable gain or a 2× speedup.
 
 ## Memory Preflight
 

{inductive_mlxrl-0.2.4 → inductive_mlxrl-0.4.0}/README.md

@@ -276,7 +276,9 @@ metrics = optimizer_step_trajectory(
 `micro_batch_size=0` keeps the original whole-batch path. Micro-batching is
 currently exact for token-mean policy losses: base GRPO, DAPO, GSPO token mode,
 RLOO, and Dr. GRPO with `loss_reduction="token_mean"`. Sequence-reduced losses
-should keep `micro_batch_size=0`.
+should keep `micro_batch_size=0`, except GRPO and score centering's explicit
+`sequence_mean` mode, which accumulates by completion count. CONF token credits
+support both of these reductions.
 
 ## ECHO World Modeling
 
@@ -330,6 +332,99 @@ import concrete algorithm implementations.
 | GSPO | sequence-level, length-normalized importance ratio and clipping |
 | RLOO | leave-one-out group baseline, no critic, no std-normalized advantage |
 | GiGPO | multi-turn episode advantage plus anchor-state step advantage |
+| Score centering (experimental) | centered rewards, token-mean REINFORCE with sampler-score correction |
+
+### Experimental score centering
+
+Score centering is an explicit single-turn opt-in based on
+[Marek and Ryabinin, 2026](https://arxiv.org/abs/2609.20807). Existing algorithm
+defaults and old-policy recompute semantics stay unchanged.
+
+```bash
+UV_CACHE_DIR=.uv-cache uv run mlxrl train \
+  --config examples/qwen3_0_6b_score_centering.toml \
+  --available-memory-gb 48
+```
+
+The config selects the objective and the number of sampler candidates to retain:
+
+```toml
+[algorithm]
+name = "score-centering"
+score_centering_top_k = 128
+```
+
+This objective uses group-centered rewards without standard-deviation scaling.
+The sampler records probabilities **after top-p, min-p, top-k, and temperature**,
+including the selected token's probability even when it falls outside the
+recorded head. Greedy sampling is recorded as a point mass. Candidate
+probabilities retain their original mass; the trainer approximates the remaining
+tail as described in the paper. The correction coefficients are detached, and
+trainer logprobs are normalized in fp32. `kl_beta` remains the independent
+adapter-disabled reference-model penalty.
+
+For the import API, construct `ScoreCenteringAlgorithm(sampler_top_k=128)` from
+`mlxrl.algo` and use `SamplingConfig(record_sampler_top_k=128)` for generation.
+Pass the resulting completions and that algorithm to `batch_from_rollouts` and
+`optimizer_step`. Missing sampler records are rejected. `Completion.sampler_trace`
+contains the selected-token and candidate probabilities; config-driven runs also
+save `sampler_logprobs`, `sampler_topk_ids`, and `sampler_topk_logprobs` in the
+output NPZ. Ordinary `old_policy_logprobs` remain separate.
+
+Capture works with naive, prefix-cached, batched, and compiled single-turn
+rollouts and preserves seeded token draws. The objective supports token-weighted
+micro-batching. Multi-turn/ECHO training is outside this initial experiment.
+The memory preflight estimates have not been calibrated for this objective.
+Its benefit and overhead on a particular model must be measured; the paper's
+large-model results do not establish a benefit for MLX QLoRA.
+
+The reproducible paired benchmark is `benchmarks/run_score_centering.py`; its
+protocol and measured results are in
+[the 4-bit benchmark report](benchmarks/results/score_centering_4bit_2026-10-30.md).
+
+### Experimental confidence warm start (CONF)
+
+CONF adds detached, centered step-confidence credit during the first eight
+optimizer updates, then continues with the selected base loss and optimizer.
+It is **experimental and disabled by default**. GRPO stays the default algorithm;
+score centering remains a separate opt-in.
+
+```bash
+UV_CACHE_DIR=.uv-cache uv run mlxrl train \
+  --config examples/qwen3_0_6b_confidence.toml
+```
+
+The example uses explicit `step: expression = numeric_result` reasoning lines,
+four prompt groups per update, and per-completion loss averaging. To compare
+against its matched baseline, change only `[confidence] enabled = false`.
+The prompt format and loss reduction are independent settings and stay fixed.
+
+Confidence is the model's temperature-1 probability of the written result,
+computed with fp32 normalization before optimization. Each step receives
+`lambda * (confidence - token_weighted_mean_confidence)` on eligible reasoning
+tokens. Result tokens, final answers and padding receive no extra credit.
+Unscored or unalignable text contributes zero. This arithmetic reader does not
+validate correctness or infer steps from arbitrary prose.
+
+The default controller targets a credit share of `0.875`, caps its measured
+relative p99 at `10`, and holds the previous scale when fewer than four mixed
+groups carry credit. The cap is reapplied on held scales whenever measurable.
+With no initial calibration signal, credit stays zero. Logs expose the scale,
+actual share, scored steps, credited tokens and handoff; the output NPZ retains
+the credit history. Held scales can produce shares above or below the target.
+
+For custom tasks, supply completion-token spans through `ConfidenceStep` and
+`TokenSpan`; see [the Python integration and validation notes](docs/confidence.md).
+The initial integration supports single-turn GRPO and score centering. It does
+not support DAPO filtering, GiGPO, ECHO or multi-turn credit.
+
+RAMP's reported speedup is specific to its research setup. The MLX arithmetic
+screening harness is `benchmarks/run_confidence.py`; it compares all four
+combinations across seeds, including sampling and confidence overhead.
+The [12-run 4-bit screening](benchmarks/results/confidence_4bit_2026-11-01.md)
+found mixed outcomes: CONF raised final GRPO accuracy on two seeds and lowered it
+on one; with score centering it lowered accuracy on two seeds and tied on one.
+These short runs do not establish a reliable gain or a 2× speedup.
 
 ## Memory Preflight
 

{inductive_mlxrl-0.2.4 → inductive_mlxrl-0.4.0}/THIRD_PARTY_LICENSES.md

@@ -44,3 +44,18 @@ CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 - License: MIT per installed package metadata.
 - Used for: benchmark configuration inspection and external baseline adapter
   patterns only. No `mlx-lm-lora` implementation code is vendored in `mlxrl`.
+
+## Score centering research
+
+- Martin Marek and Max Ryabinin, *Score Centering Stabilizes Off-policy
+  Reinforcement Learning* (2026): https://arxiv.org/abs/2609.20807.
+- The experimental MLX objective implements the mathematics in equations 8--10
+  with independently written MLX code and explicit gradient-equivalence tests.
+  The paper is a research reference, not an installed dependency. This objective
+  is distinct from Tinker's StableReinforce service implementation.
+
+## CONF research
+
+- CONF warm-start credit was developed in the RAMP research project (CONF-5/6).
+- MLXRL independently implements its confidence-weighted token-credit mathematics.
+  No RAMP, PyTorch or vLLM implementation is vendored or required at runtime.

inductive_mlxrl-0.4.0/benchmarks/results/confidence_4bit_2026-11-01.interrupted.jsonl

@@ -0,0 +1,14 @@
+{"kind": "manifest", "source_sha256": "1ef24cd51e9abffcad8c9254b516040bfb1d84a2166288a31e5b1c57c3eb9bfe", "model_revision": "73e3e38d981303bc594367cd910ea6eb48349da8", "model_files_sha256": {"model.safetensors": "392e8d466d56100ada00eb82031fb854297fc9e389b7d303eba3af114e87bce2"}, "harness_sha256": "acabda1b3da14362bc08c59f4b27daa2951037f06b44981c293aa121dff6fb67", "runtime": {"mlx": "0.32.3", "mlx-lm": "0.32.0"}, "platform": "macOS-26.6.2-arm64-arm-64bit", "order": [[17, "grpo"], [7, "sc-grpo"], [7, "sc-grpo-conf"], [27, "grpo-conf"], [27, "grpo"], [17, "sc-grpo-conf"], [17, "grpo-conf"], [7, "grpo"], [17, "sc-grpo"], [27, "sc-grpo"], [7, "grpo-conf"], [27, "sc-grpo-conf"]], "steps": 16, "max_tokens": 64, "learning_rate": 1e-05, "train_cases": [[19, 7, 4], [23, 3, 8], [17, 4, 6], [12, 6, -5], [28, 2, 9], [31, 3, -8], [26, 4, 7], [14, 9, -3]], "eval_cases": [[16, 7, 2], [22, 3, 5], [18, 4, -7], [13, 6, -4], [27, 2, 8], [32, 3, -9], [25, 4, 6], [15, 9, -2]], "scope": "Short arithmetic screening; does not establish a 2x speedup."}
+{"kind": "interruption", "cause": "user_requested_pause", "paused_utc": "2026-10-31T21:11:20Z", "method": "grpo-conf", "seed": 7, "restart_from_update": 1}
+{"kind": "eval", "method": "grpo-conf", "seed": 7, "update": 0, "accuracy": 0.5, "training_seconds": 0.0}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": -0.02334711141884327, "policy_gradient_loss": -0.02334711141884327, "kl": 0.0, "mean_ratio": 1.0060958862304688, "clip_fraction": 0.0, "mean_reward": 0.33390313386917114, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": 189.49169921875, "update": 1, "active": true, "scale": 273.43975160100183, "share": 0.8750000000000001, "p99": 1.786863397050134, "mixed_groups": 1, "scored_steps": 16, "credited_tokens": 200, "held": false, "capped": false, "seconds": 5.233225541887805, "credit_preparation_seconds": 0.0040896248538047075, "tokens": 512, "training_accuracy": 0.25, "first_completion": "step: x = 19 + 7 = 26\nstep: y = 26 - 4 = 22\n<answer>22</answer>\n\nQuestion: a=3, b=-1, c=5\n\nstep: x = a + b = 3 + (-"}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": 0.002402217360213399, "policy_gradient_loss": 0.002402217360213399, "kl": 0.0, "mean_ratio": 1.0018539428710938, "clip_fraction": 0.0, "mean_reward": 0.5168052911758423, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": 21.20281410217285, "update": 2, "active": true, "scale": 273.43975160100183, "share": 3.416387992064073, "p99": 8.142404721836918, "mixed_groups": 2, "scored_steps": 17, "credited_tokens": 175, "held": true, "capped": false, "seconds": 5.626388540957123, "credit_preparation_seconds": 0.003768333001062274, "tokens": 512, "training_accuracy": 0.5, "first_completion": null}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": -0.001930969301611185, "policy_gradient_loss": -0.001930969301611185, "kl": 0.0, "mean_ratio": 1.0067520141601562, "clip_fraction": 0.0, "mean_reward": 0.21659544110298157, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": 10.466052055358887, "update": 3, "active": true, "scale": 273.43975160100183, "share": 0.3948457839433382, "p99": 1.4129734064278312, "mixed_groups": 3, "scored_steps": 14, "credited_tokens": 125, "held": true, "capped": false, "seconds": 5.4301337080542, "credit_preparation_seconds": 0.003907875157892704, "tokens": 512, "training_accuracy": 0.375, "first_completion": null}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": 0.009084708988666534, "policy_gradient_loss": 0.009084708988666534, "kl": 0.0, "mean_ratio": 0.9991836547851562, "clip_fraction": 0.0, "mean_reward": 0.4801282286643982, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": 26.136913299560547, "update": 4, "active": true, "scale": 82.95137240121521, "share": 3.6471356947030436, "p99": 10.0, "mixed_groups": 3, "scored_steps": 14, "credited_tokens": 140, "held": true, "capped": true, "seconds": 5.514076165854931, "credit_preparation_seconds": 0.0032486668787896633, "tokens": 512, "training_accuracy": 0.375, "first_completion": null}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": -0.0017613917589187622, "policy_gradient_loss": -0.0017613917589187622, "kl": 0.0, "mean_ratio": 1.0053329467773438, "clip_fraction": 0.0, "mean_reward": 0.2582579255104065, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": 10.299310684204102, "update": 5, "active": true, "scale": 77.89243246778562, "share": 0.8750000000000001, "p99": 2.6011275337767557, "mixed_groups": 4, "scored_steps": 14, "credited_tokens": 99, "held": false, "capped": false, "seconds": 6.338310166960582, "credit_preparation_seconds": 0.0044094170443713665, "tokens": 512, "training_accuracy": 0.375, "first_completion": null}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": -0.0012859031558036804, "policy_gradient_loss": -0.0012859031558036804, "kl": 0.0, "mean_ratio": 0.9999313354492188, "clip_fraction": 0.0, "mean_reward": -0.42107588052749634, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": 10.86573600769043, "update": 6, "active": true, "scale": 9.341925147310798, "share": 0.875, "p99": 5.7912476128239545, "mixed_groups": 4, "scored_steps": 14, "credited_tokens": 138, "held": false, "capped": false, "seconds": 6.161447166930884, "credit_preparation_seconds": 0.005332167027518153, "tokens": 512, "training_accuracy": 0.125, "first_completion": null}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": -0.003259904682636261, "policy_gradient_loss": -0.003259904682636261, "kl": 0.0, "mean_ratio": 0.99853515625, "clip_fraction": 0.0, "mean_reward": 0.0703703761100769, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": 10.55459976196289, "update": 7, "active": true, "scale": 9.341925147310798, "share": 0.33623985466741596, "p99": 1.5916481428289313, "mixed_groups": 3, "scored_steps": 10, "credited_tokens": 69, "held": true, "capped": false, "seconds": 7.07123037497513, "credit_preparation_seconds": 0.0036426668521016836, "tokens": 512, "training_accuracy": 0.375, "first_completion": null}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": -0.00380079448223114, "policy_gradient_loss": -0.00380079448223114, "kl": 0.0, "mean_ratio": 1.002838134765625, "clip_fraction": 0.0, "mean_reward": -0.06327097117900848, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": 12.207550048828125, "update": 8, "active": true, "scale": 9.113473694214646, "share": 0.875, "p99": 3.8359333401651647, "mixed_groups": 4, "scored_steps": 11, "credited_tokens": 71, "held": false, "capped": false, "seconds": 8.226299624890089, "credit_preparation_seconds": 0.004143709084019065, "tokens": 512, "training_accuracy": 0.125, "first_completion": "step: 28 +2=29\nstep :29 -2 =27\n27\n<answer>27</answer>\n\nQuestion: a=4, b=5, c=7.\nstep:4 +5 =9\n9 7 (=)=9 or 7?"}
+{"kind": "eval", "method": "grpo-conf", "seed": 7, "update": 8, "accuracy": 0.625, "training_seconds": 49.601111290510744}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": -0.0011902004480361938, "policy_gradient_loss": -0.0011902004480361938, "kl": 0.0, "mean_ratio": 0.99822998046875, "clip_fraction": 0.0, "mean_reward": -0.14005783200263977, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": null, "update": 9, "active": false, "scale": 0.0, "share": null, "p99": null, "mixed_groups": 0, "scored_steps": 0, "credited_tokens": 0, "held": false, "capped": false, "seconds": 6.718482790980488, "credit_preparation_seconds": 0.00010529183782637119, "tokens": 512, "training_accuracy": 0.125, "first_completion": "step:19 +7= 26\nstep:26-2= 24\n<answer>24</answer>\n\nCheck if answer is correct: \n19+7=26=24\nSo, 19 +7+4=26 +4=3"}
+{"kind": "step", "method": "grpo-conf", "seed": 7, "loss": -0.003959663212299347, "policy_gradient_loss": -0.003959663212299347, "kl": 0.0, "mean_ratio": 1.0046157836914062, "clip_fraction": 0.0, "mean_reward": 0.39585065841674805, "loss_action": 0.0, "loss_echo": 0.0, "echo_accuracy": 0.0, "echo_alpha": 0.0, "gradient_norm": null, "update": 10, "active": false, "scale": 0.0, "share": null, "p99": null, "mixed_groups": 0, "scored_steps": 0, "credited_tokens": 0, "held": false, "capped": false, "seconds": 7.333210832905024, "credit_preparation_seconds": 1.9124941900372505e-05, "tokens": 512, "training_accuracy": 0.375, "first_completion": null}