openpond 0.0.34 → 0.0.35

package/dist/web/courses/post-training/full-course.vtt

@@ -0,0 +1,623 @@
+WEBVTT
+
+1
+00:00:00.700 --> 00:00:10.887
+Post-training changes what a model is likely to do. An input creates a distribution over possible actions, evaluation supplies evidence, and an optimizer changes that distribution.
+
+2
+00:00:12.598 --> 00:00:24.693
+Here a cancellation test is failing. The policy assigns chances to return None, raise the cancellation error, or retry. Those probabilities describe the available behavior; sampling produces one actual patch.
+
+3
+00:00:26.404 --> 00:00:34.313
+Loss is a number saying how bad behavior looks under the training rule. Useful training lowers both training loss and held-out error.
+
+4
+00:00:36.024 --> 00:00:42.025
+If training loss falls while held-out error rises, the model is learning the wrong thing.
+
+5
+00:00:43.736 --> 00:00:56.875
+The sampled patch passes. Training makes the exception-raising action slightly easier to choose in similar states. Ordinary backpropagation implements the change; reinforcement learning determines which sampled behavior receives credit.
+
+6
+00:00:58.587 --> 00:01:04.267
+A usable training record must also identify who generated the attempt and what evidence accompanied it.
+
+
+7
+00:01:05.667 --> 00:01:17.209
+The expression pi theta of a given state is a compact description of model behavior. Pi names the policy: a probability distribution, not one completed answer. Theta means all adjustable model parameters.
+
+8
+00:01:18.341 --> 00:01:25.205
+The state is the context available now, and an action can be one token, one tool call, or another sampled decision.
+
+9
+00:01:26.336 --> 00:01:34.837
+Before sampling, the network produces raw scores called logits. A logit is not a percentage and logits do not need to add to one. They can be positive or negative.
+
+10
+00:01:35.968 --> 00:01:43.968
+Only their relative differences matter: a larger logit means the model prefers that action compared with the alternatives available at that position.
+
+11
+00:01:45.099 --> 00:01:56.460
+Softmax converts logits into probabilities. It exponentiates each logit, then divides by the sum of all exponentiated logits. This makes every result positive and makes the distribution sum to one.
+
+12
+00:01:57.592 --> 00:02:11.864
+In the example, logits three, one, and point two become probabilities of roughly eighty-four, eleven, and five percent. Temperature divides the logits before softmax. Lower temperature sharpens the distribution; higher temperature flattens it.
+
+13
+00:02:12.995 --> 00:02:22.590
+A rollout, also called a trajectory and written tau, is the interaction record used for learning. The behavior policy mu generated each sampled action.
+
+14
+00:02:23.721 --> 00:02:37.452
+Its log-probability is stored beside the action so training can later compare the old behavior distribution with the current policy. Environment observations affect later states, but only the model's sampled actions receive policy-gradient terms.
+
+15
+00:02:38.583 --> 00:02:50.899
+Reward is the scalar number emitted by an evaluator. A math checker might return one for an exact answer and zero otherwise. A code environment might return one only when every hidden test passes.
+
+16
+00:02:52.030 --> 00:03:05.710
+A tool task might inspect whether the final database state matches a target. The rule can be binary or graded, but the optimizer sees the number, not the evaluator's full explanation. That richer explanation is feedback.
+
+17
+00:03:06.841 --> 00:03:19.067
+Reward, return, and advantage are related but not interchangeable. Reward is the scalar emitted at a step or at the end. Return, G at time t, combines future rewards and can discount distant outcomes with gamma.
+
+18
+00:03:20.198 --> 00:03:34.149
+Advantage subtracts a baseline from that return. Positive advantage means the action did better than expected; negative advantage means it did worse. PPO learns a value baseline, while GRPO estimates one from sibling responses.
+
+19
+00:03:35.280 --> 00:03:50.506
+PPO means Proximal Policy Optimization. It is a policy-gradient method that trains a second network, called a critic or value function, to estimate expected return. The observed return minus that expectation becomes advantage.
+
+20
+00:03:51.638 --> 00:03:58.954
+PPO also clips the probability-ratio incentive so one sampled action cannot drive an arbitrarily large update.
+
+21
+00:04:00.085 --> 00:04:12.591
+GRPO means Group Relative Policy Optimization. It removes the learned critic and samples several sibling responses for the same prompt. Each response is compared with the group's reward mean and standard deviation.
+
+22
+00:04:13.723 --> 00:04:23.177
+Better-than-group responses receive positive advantage; worse responses receive negative advantage. GRPO keeps policy ratios and clipping.
+
+23
+00:04:24.308 --> 00:04:38.540
+Both methods need a baseline because raw reward lacks context. PPO's baseline comes from the critic, and Generalized Advantage Estimation combines value predictions across time. GRPO's baseline comes from sibling rewards.
+
+24
+00:04:39.671 --> 00:04:46.676
+If every sibling receives the same reward, the group standard deviation is zero and there is no relative advantage to learn from.
+
+25
+00:04:47.807 --> 00:04:59.762
+An objective, often written L of theta, turns the training goal into one number. The gradient is the local slope of that objective with respect to every adjustable parameter. Backpropagation computes those slopes.
+
+26
+00:05:00.893 --> 00:05:13.801
+The optimizer applies a learning rate alpha to make a small update from theta to theta prime. In policy-gradient training, the RL method determines the credit weights inside the loss; ordinary differentiation performs the parameter update.
+
+27
+00:05:14.932 --> 00:05:26.434
+PPO and GRPO use a probability ratio. The numerator is the current policy's probability for the stored action. The denominator is the behavior probability recorded during rollout.
+
+28
+00:05:27.565 --> 00:05:35.745
+A ratio above one means that action became more likely. Clipping with epsilon caps the incentive once the ratio moves too far in one update.
+
+29
+00:05:36.876 --> 00:05:41.965
+It limits this sampled term, but does not prove that the entire model stayed close.
+
+30
+00:05:43.096 --> 00:05:55.512
+Entropy measures how spread out one policy distribution is. KL divergence compares two distributions and is commonly used to measure movement from a reference model or mismatch between teacher and student. Its direction matters.
+
+31
+00:05:56.643 --> 00:06:07.784
+Cross-entropy uses target probabilities q to weight the log of student probabilities p. Distillation usually minimizes this quantity so the student assigns probability where the teacher does.
+
+32
+00:06:08.915 --> 00:06:24.965
+RL means reinforcement learning. RFT means reinforcement fine-tuning: applying an RL-style objective to an already pretrained model. RLVR adds verifiable rewards, such as executable tests. PPO and GRPO are policy update rules.
+
+33
+00:06:26.096 --> 00:06:45.096
+PPO is Proximal Policy Optimization. GRPO is Group Relative Policy Optimization. GAE is Generalized Advantage Estimation. Pass-at-k asks whether at least one of k sampled attempts succeeds. On-policy and off-policy describe who generated the data.
+
+34
+00:06:46.227 --> 00:07:01.453
+The teacher-guided methods use dense distributions instead of only scalar outcomes. OPSD is On-Policy Self-Distillation and conditions the teacher on a trusted solution. SDFT is Self-Distillation Fine-Tuning and supplies a demonstration.
+
+35
+00:07:02.584 --> 00:07:18.634
+SDPO is Self-Distillation Policy Optimization and supplies failure feedback. In each case the student creates the prefix and the frozen teacher receives extra evidence. SRPO, Self-Rewarding Policy Optimization, routes successful and unsuccessful samples through different signals.
+
+
+36
+00:07:20.034 --> 00:07:26.808
+On-policy data comes from the learner being updated. Policy version twelve generates an attempt that will update version twelve.
+
+37
+00:07:28.399 --> 00:07:35.716
+Off-policy data comes from a teacher, human, or older checkpoint. These labels describe source, not quality.
+
+38
+00:07:37.307 --> 00:07:45.898
+An RL rollout stores the problem, sampled actions, generating checkpoint, behavior probabilities, environment observations, and reward.
+
+39
+00:07:47.489 --> 00:07:54.173
+Text alone cannot connect evaluated behavior to an update. The missing record matters as much as the final response.
+
+40
+00:07:55.764 --> 00:08:04.135
+Other sources carry different signals: teacher targets, chosen and rejected responses, or old actions with original probabilities and rewards.
+
+41
+00:08:05.726 --> 00:08:16.867
+PPO and GRPO usually score learner attempts. Offline distillation uses teacher examples. On-policy distillation labels the student's current prefix.
+
+42
+00:08:18.458 --> 00:08:25.001
+Inside any RL rollout, the final patch is only one part of a sequence of actions and observations.
+
+
+43
+00:08:26.401 --> 00:08:35.354
+An RL rollout is a sequence, not only a final response. Credit assignment asks which decisions inside that sequence should become more or less likely.
+
+44
+00:08:37.082 --> 00:08:49.990
+The issue and repository form state zero. The behavior policy samples “inspect files.” File contents are an environment observation, not another sampled action. The next state contains those contents.
+
+45
+00:08:51.717 --> 00:08:57.579
+The policy then writes raise Cancelled Error, and test output closes the trajectory.
+
+46
+00:08:59.306 --> 00:09:12.124
+The tuple makes those roles explicit. Tau contains states, sampled actions, behavior log-probabilities, observations, and terminal reward. Gradients apply to action positions controlled by the policy.
+
+47
+00:09:13.851 --> 00:09:18.538
+Observations condition later actions but are masked out of the policy-gradient loss.
+
+48
+00:09:20.266 --> 00:09:31.316
+Reward and feedback carry different information. A patch can receive reward zero while the environment still knows that eight tests passed, cancellation failed, and the expected exception was Cancelled Error.
+
+49
+00:09:33.044 --> 00:09:38.634
+Reward is the scalar consumed by an objective. Feedback is the richer evidence the environment produced.
+
+50
+00:09:40.362 --> 00:09:51.633
+A terminal score also leaves the cause ambiguous. Was the wrong file inspected? Was the timeout branch incorrect? Was “None” the decisive token? Response-level credit applies one outcome broadly.
+
+51
+00:09:53.360 --> 00:09:58.640
+Step- or token-level signals can be denser, but density alone does not prove causal accuracy.
+
+52
+00:10:00.367 --> 00:10:13.867
+For multi-step problems, return carries future rewards backward. G at time t is the discounted sum of later rewards. With gamma point nine and success two steps later, the earlier action receives return point eight one.
+
+53
+00:10:15.595 --> 00:10:21.185
+PPO critics and Generalized Advantage Estimation, or GAE, build on this idea.
+
+54
+00:10:22.912 --> 00:10:34.686
+Advantage then asks: better than what was expected? If observed return is point eight and the baseline is point five, advantage is positive point three. A return of point two gives negative point three.
+
+55
+00:10:36.413 --> 00:10:49.010
+Positive advantage raises the sampled behavior’s probability; negative advantage lowers it. PPO learns a critic for the baseline. GRPO replaces that critic with statistics from sibling responses.
+
+56
+00:10:50.738 --> 00:11:01.878
+Learning also changes exploration. Pass-at-one can rise while entropy and pass-at-k fall if the model collapses onto one strategy. Reliability and breadth therefore need separate metrics.
+
+57
+00:11:03.606 --> 00:11:18.149
+Finally, a frozen reference policy anchors behavior outside the optimized task. Kullback–Leibler divergence, shortened to KL, measures distribution shift. Too little constraint permits drift and reward exploits; too much prevents useful learning.
+
+58
+00:11:19.877 --> 00:11:24.835
+A test-based reward is useful only when the test measures the intended outcome reliably.
+
+
+59
+00:11:26.235 --> 00:11:32.828
+Every reward-based update depends on whether its evaluator measures the right thing. This is the verifier problem.
+
+60
+00:11:33.989 --> 00:11:44.487
+RLVR means Reinforcement Learning with Verifiable Rewards. For our repair, the verifier applies the patch, runs the cancellation tests, and returns a reproducible outcome.
+
+61
+00:11:45.647 --> 00:11:50.696
+The same idea applies to checked math answers, tool states, and agent environments.
+
+62
+00:11:51.856 --> 00:11:58.399
+A preference such as “patch A is cleaner” depends on a judge. A verifiable reward applies an explicit rule.
+
+63
+00:11:59.560 --> 00:12:07.468
+Verifiable does not mean infallible; it means the same patch should receive the same result under the same evaluator state.
+
+64
+00:12:08.629 --> 00:12:20.081
+Write the rule as r equals V of x, y, and e. X is the public task. Y is the sampled attempt. E is evaluator-only state such as hidden tests.
+
+65
+00:12:21.241 --> 00:12:25.426
+For a binary verifier, reward is zero or one.
+
+66
+00:12:26.586 --> 00:12:39.364
+This equation separates the setting from the optimizer. RLVR supplies reward. PPO, GRPO, or another policy optimizer supplies the update. DeepSeek-R1 is an important example of reasoning training built from this combination.
+
+67
+00:12:40.524 --> 00:12:51.796
+The full loop is prompt, policy attempt, execution, verification, reward, and policy update. Automatic checks scale because a person does not need to score every trajectory.
+
+68
+00:12:52.956 --> 00:12:57.733
+The verifier also becomes both the task specification and an attack surface.
+
+69
+00:12:58.894 --> 00:13:12.213
+RLVR applies beyond short math answers. Code can compile and run tests. Tool agents can be checked against a target database state. Search can verify retrieved evidence. Scientific tasks can use simulators and constraints.
+
+70
+00:13:13.374 --> 00:13:20.509
+It is a poor fit when quality is inherently subjective, outcomes cannot be reproduced, or the checker is easy to game.
+
+71
+00:13:21.670 --> 00:13:33.262
+Verifier errors have two directions. A false negative rejects a valid solution. A false positive is more dangerous under optimization because a wrong solution receives positive gradient and can become an exploit.
+
+72
+00:13:34.423 --> 00:13:43.788
+Suppose visible tests cover only three examples. Hard-coding those outputs and implementing the general rule both receive reward one. The optimizer cannot distinguish them.
+
+73
+00:13:44.948 --> 00:13:52.405
+Hidden tests, randomized cases, invariants, and shadow verifiers make the measured objective closer to the intended task.
+
+74
+00:13:53.565 --> 00:14:07.567
+That requires an information boundary. The policy sees the public task and allowed observations. The evaluator alone sees hidden checks and anti-cheat state. Training-only teacher evidence belongs in a third protected channel.
+
+75
+00:14:08.727 --> 00:14:17.369
+So “RLVR with GRPO” is precise: the first term names how outcomes are labeled, and the second names how those labels become an update.
+
+
+76
+00:14:18.769 --> 00:14:27.321
+A trustworthy reward still has to become a learning signal. PPO and GRPO differ mainly in how they decide whether an outcome was better or worse than expected.
+
+77
+00:14:28.868 --> 00:14:43.281
+PPO means Proximal Policy Optimization. For the successful inspect, patch, and test trajectory, observed return is point eight. A learned critic expected point five. Their difference, positive point three, is the advantage.
+
+78
+00:14:44.829 --> 00:14:49.877
+PPO combines that advantage with a clipped probability ratio and updates the policy.
+
+79
+00:14:51.425 --> 00:15:06.249
+GRPO means Group Relative Policy Optimization. It keeps rollouts, rewards, probability ratios, and clipping, but removes the critic. Instead, it samples several patches for the same cancellation bug and compares their verifier scores.
+
+80
+00:15:07.797 --> 00:15:16.710
+Our six sibling patches produce rewards one, zero, zero, one, one, zero. The mean is one half and the standard deviation is one half.
+
+81
+00:15:18.257 --> 00:15:24.119
+Normalizing gives every passing patch advantage positive one and every failing patch negative one.
+
+82
+00:15:25.666 --> 00:15:35.031
+That comparison is useful but coarse. Raise Timeout Error and return None both failed, so binary GRPO treats them alike even if one was closer.
+
+83
+00:15:36.579 --> 00:15:40.804
+The group says which siblings did better; it does not explain why.
+
+84
+00:15:42.351 --> 00:15:52.538
+The clipping graph controls how much one sampled action can influence a single update. If its probability moves from point one zero to point one three, the ratio is one point three.
+
+85
+00:15:54.086 --> 00:16:03.902
+With a cap at one point two, additional positive credit stops growing. Clipping limits a local incentive; it does not guarantee that the whole policy stayed close.
+
+86
+00:16:05.450 --> 00:16:08.681
+A group teaches only when it contains contrast.
+
+87
+00:16:10.228 --> 00:16:22.684
+If one patch succeeds with probability p, a group of size G is mixed with probability one minus p to the G minus one minus p, all to the G. That subtracts all-success and all-failure groups.
+
+88
+00:16:24.232 --> 00:16:38.735
+Near zero success, almost every group fails. Near one, almost every group passes. Either case produces little relative signal. Larger groups widen the useful middle but require more patch generation and test execution.
+
+89
+00:16:40.283 --> 00:16:50.690
+This is why prompts need to sit near the learning frontier: difficult enough to fail, possible enough to solve. If every cancellation patch receives the same reward, normalized group advantage is zero.
+
+90
+00:16:52.238 --> 00:17:04.283
+Both PPO and GRPO need trajectories, rewards, behavior log-probabilities, clipped ratios, and gradients. PPO learns expected value with a critic. GRPO estimates a baseline from sibling patches.
+
+91
+00:17:05.830 --> 00:17:10.969
+The cancellation environment stays the same; only the baseline estimator changes.
+
+
+92
+00:17:12.369 --> 00:17:22.275
+Outcome rewards compress an attempt to a scalar. Distillation supplies richer guidance by specifying a probability distribution over plausible next tokens.
+
+93
+00:17:25.634 --> 00:17:40.639
+A one-hot target says “Cancelled Error” and assigns every alternative zero. A teacher distribution can say that “Cancelled Error” is most likely, “Timeout Error” is also plausible, “return” is weak, and “retry” is unlikely.
+
+94
+00:17:43.997 --> 00:17:47.770
+That structure contains more information than the one sampled token.
+
+95
+00:17:51.128 --> 00:17:59.499
+Teacher probability q weights the log of student probability p over the vocabulary. Minimizing that cross-entropy moves the student toward the teacher.
+
+96
+00:18:02.857 --> 00:18:11.268
+In our example, the student initially favors “return,” while the teacher favors “Cancelled Error.” After an update, the two distributions move closer.
+
+97
+00:18:14.626 --> 00:18:25.898
+Teacher temperature controls how much of that structure is visible. Low temperature makes the target almost one-hot. Higher temperature reveals alternatives in the tail; too much can magnify noise.
+
+98
+00:18:29.256 --> 00:18:34.164
+Teacher-target temperature is separate from the rollout temperature used to sample behavior.
+
+99
+00:18:37.522 --> 00:18:51.523
+KL direction changes the lesson too. Forward KL penalizes the student for missing teacher-supported alternatives. Reverse KL strongly penalizes student probability where the teacher assigns little mass, often favoring a narrower mode.
+
+100
+00:18:54.882 --> 00:18:58.655
+“We used KL” is incomplete without direction and temperature.
+
+101
+00:19:02.013 --> 00:19:15.693
+Prefix provenance determines which states receive teacher targets. Offline distillation uses fixed teacher trajectories. On-policy distillation lets the student reach its own strange cancellation state, then asks the teacher what should come next there.
+
+102
+00:19:19.052 --> 00:19:31.417
+The teacher does not need larger weights. The same frozen model can see the student's failed prefix plus extra training-only evidence: a verified repair, an expert demonstration, or a test explanation.
+
+103
+00:19:34.776 --> 00:19:38.227
+That evidence changes the teacher distribution.
+
+104
+00:19:41.586 --> 00:19:52.636
+Training transfers the useful part of that privileged view into student weights. Deployment removes the evidence. A verified solution, demonstration, or failure explanation produces a different teacher target.
+
+
+105
+00:19:54.036 --> 00:20:04.996
+OPSD, SDFT, and SDPO share one mechanism. A frozen teacher scores the student's prefix while receiving additional evidence. The evidence source defines the method.
+
+106
+00:20:06.426 --> 00:20:16.333
+The student distribution is p theta given the issue and its prefix. Teacher target q scores the same next token at the same prefix, but also conditions on evidence e.
+
+107
+00:20:17.763 --> 00:20:22.761
+Training matches q into p; deployment removes e.
+
+108
+00:20:24.191 --> 00:20:33.516
+On-Policy Self-Distillation, or OPSD, gives the teacher a trusted solution. Here that is the verified repair: preserve cleanup and raise Cancelled Error.
+
+109
+00:20:34.946 --> 00:20:41.630
+The teacher uses it to guide the student's own prefix without placing the privileged patch in the deployed student's prompt.
+
+110
+00:20:43.060 --> 00:20:51.240
+The OPSD paper evaluated this mechanism on reasoning tasks; the animation maps the information flow onto our code case.
+
+111
+00:20:52.671 --> 00:21:04.354
+Self-Distillation Fine-Tuning, or SDFT, gives the teacher an expert demonstration. A related repair shows the sequence inspect the flag, run cleanup, then raise the exception.
+
+112
+00:21:05.784 --> 00:21:13.964
+Unlike offline imitation, SDFT scores the current student's return-None prefix, so the demonstration guides the state the student actually reached.
+
+113
+00:21:15.394 --> 00:21:29.807
+Self-Distillation Policy Optimization, or SDPO, gives the teacher feedback about the current failure. The scalar reward is zero, but the test trace says test-cancel failed, expected Cancelled Error, and return skipped cancellation.
+
+114
+00:21:31.237 --> 00:21:36.787
+The teacher converts that explanation into dense token probabilities along the failed prefix.
+
+115
+00:21:38.217 --> 00:21:49.719
+The choice follows the evidence. A trusted solution supports OPSD. A demonstration supports SDFT. An explanatory failure trace supports SDPO.
+
+116
+00:21:51.150 --> 00:22:00.153
+If all you have is a binary outcome, GRPO may be the honest baseline; inventing a generic “feedback” field would hide these different trust boundaries.
+
+117
+00:22:01.583 --> 00:22:11.400
+None of the dense methods is automatically safe. A privileged patch can leak. A demonstration can be irrelevant. A teacher can misread feedback and densify an error.
+
+118
+00:22:12.830 --> 00:22:21.291
+New capability can overwrite old behavior, entropy can collapse, and teacher forward passes can erase claimed efficiency savings.
+
+119
+00:22:22.721 --> 00:22:35.770
+So the durable comparison is not acronym against acronym. Hold the student prefix fixed, vary the evidence, record the cost, and measure both the cancellation repair and unrelated retained capability.
+
+
+120
+00:22:37.170 --> 00:22:44.627
+A research result requires many versioned tasks, protected information boundaries, matched baselines, and held-out evaluation.
+
+121
+00:22:46.016 --> 00:22:58.020
+Policy-visible input includes the issue, repository revision, allowed tools, and public tests. Training-only evidence may contain an expert repair, privileged patch, or public diagnostic.
+
+122
+00:22:59.409 --> 00:23:06.003
+Evaluator-only hidden tests, anti-exploit checks, and held-out repository clusters must remain outside both.
+
+123
+00:23:07.391 --> 00:23:20.982
+The Taskset preserves those boundaries together with source revision, license, split, verifier version, and content hash. Related issues from the same repository should be clustered before splitting; otherwise memorization can look like generalization.
+
+124
+00:23:22.371 --> 00:23:33.240
+Hugging Face support is worthwhile when it behaves as a reproducible importer. Pin the repository revision. Inspect the dataset card and license. Choose configuration and split explicitly.
+
+125
+00:23:34.629 --> 00:23:45.900
+Map columns through a reviewed transform, log rejected rows, preserve original identifiers, and materialize an immutable snapshot. A training run should never silently follow a changing main branch.
+
+126
+00:23:47.289 --> 00:24:00.107
+Every proposed method needs a simple credible baseline. Begin with the base model and the most direct token-imitation objective supported by the data. Add offline distillation or preference learning when those signals are relevant.
+
+127
+00:24:01.496 --> 00:24:10.048
+Compare with GRPO or another RLVR baseline before claiming value from OPSD, SDFT, or SDPO.
+
+128
+00:24:11.436 --> 00:24:22.256
+Equal optimizer tokens and equal total compute answer different questions. A distillation method may use fewer rollouts while spending additional teacher forward passes. Report both controls.
+
+129
+00:24:23.645 --> 00:24:41.010
+Evaluation should span four dimensions. Capability measures task success. Diversity measures pass-at-k, entropy, and distinct strategies. Retention tests old domains and distribution drift. Integrity audits reward exploits, leakage, and disagreement with a shadow verifier.
+
+130
+00:24:42.399 --> 00:24:45.670
+A single average can hide serious regressions.
+
+131
+00:24:47.059 --> 00:24:57.838
+Compute accounting includes student rollout tokens, teacher-scored tokens, backward tokens, verifier time, external calls, failed jobs, memory, storage, and wall clock.
+
+132
+00:24:59.227 --> 00:25:15.638
+The primary study compares GRPO, SDPO, and a routed success-and-failure method on code repair while separating public diagnostics from hidden tests. Verified math can replicate the GRPO and OPSD claims with exact graders.
+
+133
+00:25:17.027 --> 00:25:28.800
+A tool protocol can replicate token imitation and SDFT while measuring retention. The extra domains test whether a conclusion transfers; they do not replace the main story halfway through it.
+
+134
+00:25:30.189 --> 00:25:40.868
+The paper should bind every claim to a dataset revision, metric, run set, seed, confidence interval, and ablation. That structure keeps a local result from turning into a universal claim.
+
+135
+00:25:42.257 --> 00:25:56.309
+The final map is simple. Demonstrations produce imitation targets. Preferences produce relative targets. Verifiable outcomes produce rewards. Privileged solutions and failure explanations produce context-conditioned teacher distributions.
+
+136
+00:25:57.698 --> 00:26:07.704
+Evidence determines the target. The target determines the update. The update changes future behavior. Evaluation measures what improved—and what changed elsewhere.
+
+
+137
+00:26:09.104 --> 00:26:15.969
+The first appendix isolates two GRPO evaluation choices that matter after the main mechanism is understood.
+
+138
+00:26:20.463 --> 00:26:30.741
+Length normalization changes gradient weight. A fourteen-hundred-token correct answer and a one-hundred-eighty-token correct answer may receive the same reward but contribute very different numbers of token terms.
+
+139
+00:26:35.235 --> 00:26:43.235
+Sequence averaging, token averaging, overlong penalties, and group normalization therefore change the effective objective.
+
+140
+00:26:47.730 --> 00:26:56.733
+Pass-at-one measures reliability from one sample. Pass-at-k asks whether any of k samples succeeds and exposes remaining search diversity.
+
+141
+00:27:01.228 --> 00:27:08.504
+A model can improve pass-at-one while collapsing onto one strategy, so report both metrics with the sampling configuration.
+
+142
+00:27:09.904 --> 00:27:17.994
+The systems problem is storage. A large vocabulary creates one teacher logit for every token at every trajectory position.
+
+143
+00:27:21.415 --> 00:27:30.459
+Top-k compression stores the most likely teacher logits and approximates the remaining tail. This reduces memory and bandwidth but changes the target distribution.
+
+144
+00:27:33.880 --> 00:27:40.604
+Measure divergence from full logits on a validation sample, and count teacher-forward cost when comparing efficiency.
+
+145
+00:27:42.004 --> 00:27:48.547
+The reported graphs belong in an appendix because they come from different models, datasets, and experimental budgets.
+
+146
+00:27:50.405 --> 00:27:56.226
+The OPSD study reports its largest displayed aggregate gain at the smallest Qwen3 model.
+
+147
+00:27:58.084 --> 00:28:10.541
+At one point seven billion parameters, the displayed base score is thirty-seven point one, GRPO is thirty-seven point seven, and OPSD is forty-three point four. The gaps are smaller at four and eight billion parameters.
+
+148
+00:28:12.398 --> 00:28:17.537
+That is a hypothesis for replication, not a universal model-size law.
+
+149
+00:28:19.395 --> 00:28:30.124
+The SDFT study emphasizes knowledge acquisition and retention. Its displayed strict aggregate moves from eighty to eighty-nine, while its out-of-distribution aggregate moves from eighty to ninety-eight.
+
+150
+00:28:31.982 --> 00:28:44.117
+The SDPO study reports forty-one point two for GRPO and forty-eight point eight for SDPO on its code benchmark. Their bars answer different questions and should not be read as a controlled head-to-head comparison.
+
+151
+00:28:45.975 --> 00:28:58.612
+SRPO means Sample-Routed Policy Optimization. It sends verified successes toward a GRPO-style outcome update and failures with explanations toward an SDPO-style corrective target.
+
+152
+00:29:00.470 --> 00:29:12.243
+The useful principle is signal routing. A successful rollout already shows what worked and can support sparse selection. A failed rollout with diagnostics may reveal how to recover and can support dense correction.
+
+153
+00:29:14.101 --> 00:29:19.471
+The router preserves that asymmetry instead of forcing every sample through one uniform loss.
+

package/dist/web/courses/post-training/scripts/script_01.md

@@ -0,0 +1,31 @@
+# How post-training works
+
+Post-training from first principles · Lesson 1 of 10 · 1:05
+
+## Learning objective
+
+The choose, judge, and update loop behind every method in this series.
+
+## Using this with an LLM
+
+Use this script as lesson source material. Preserve its distinctions and caveats when summarizing, making flash cards, proposing experiments, or answering questions. The narration is the source of truth; ask for missing experimental details instead of inventing them.
+
+## Visual context
+
+OpenPond lesson intro, full chapter timestamp map, one cancellation-policy choice, useful versus misleading loss curves, and a plain-language choose-test-update loop.
+
+## Narration transcript
+
+Post-training changes what a model is likely to do. An input creates a distribution over possible actions, evaluation supplies evidence, and an optimizer changes that distribution.
+
+Here a cancellation test is failing. The policy assigns chances to return None, raise the cancellation error, or retry. Those probabilities describe the available behavior; sampling produces one actual patch.
+
+Loss is a number saying how bad behavior looks under the training rule. Useful training lowers both training loss and held-out error. If training loss falls while held-out error rises, the model is learning the wrong thing.
+
+The sampled patch passes. Training makes the exception-raising action slightly easier to choose in similar states. Ordinary backpropagation implements the change; reinforcement learning determines which sampled behavior receives credit.
+
+A usable training record must also identify who generated the attempt and what evidence accompanied it.
+
+## Provenance
+
+This is the production narration for the OpenPond learning series, generated from the canonical course script. Equations, diagrams, and cited paper results remain in the accompanying video and research document.