openpond 0.0.34 → 0.0.36

package/dist/web/courses/post-training/07-distillation.vtt

@@ -0,0 +1,53 @@
+WEBVTT
+
+1
+00:00:00.700 --> 00:00:10.606
+Outcome rewards compress an attempt to a scalar. Distillation supplies richer guidance by specifying a probability distribution over plausible next tokens.
+
+2
+00:00:13.965 --> 00:00:28.970
+A one-hot target says “Cancelled Error” and assigns every alternative zero. A teacher distribution can say that “Cancelled Error” is most likely, “Timeout Error” is also plausible, “return” is weak, and “retry” is unlikely.
+
+3
+00:00:32.328 --> 00:00:36.101
+That structure contains more information than the one sampled token.
+
+4
+00:00:39.459 --> 00:00:47.830
+Teacher probability q weights the log of student probability p over the vocabulary. Minimizing that cross-entropy moves the student toward the teacher.
+
+5
+00:00:51.188 --> 00:00:59.599
+In our example, the student initially favors “return,” while the teacher favors “Cancelled Error.” After an update, the two distributions move closer.
+
+6
+00:01:02.957 --> 00:01:14.229
+Teacher temperature controls how much of that structure is visible. Low temperature makes the target almost one-hot. Higher temperature reveals alternatives in the tail; too much can magnify noise.
+
+7
+00:01:17.587 --> 00:01:22.495
+Teacher-target temperature is separate from the rollout temperature used to sample behavior.
+
+8
+00:01:25.853 --> 00:01:39.854
+KL direction changes the lesson too. Forward KL penalizes the student for missing teacher-supported alternatives. Reverse KL strongly penalizes student probability where the teacher assigns little mass, often favoring a narrower mode.
+
+9
+00:01:43.213 --> 00:01:46.986
+“We used KL” is incomplete without direction and temperature.
+
+10
+00:01:50.344 --> 00:02:04.024
+Prefix provenance determines which states receive teacher targets. Offline distillation uses fixed teacher trajectories. On-policy distillation lets the student reach its own strange cancellation state, then asks the teacher what should come next there.
+
+11
+00:02:07.383 --> 00:02:19.748
+The teacher does not need larger weights. The same frozen model can see the student's failed prefix plus extra training-only evidence: a verified repair, an expert demonstration, or a test explanation.
+
+12
+00:02:23.107 --> 00:02:26.558
+That evidence changes the teacher distribution.
+
+13
+00:02:29.917 --> 00:02:40.967
+Training transfers the useful part of that privileged view into student weights. Deployment removes the evidence. A verified solution, demonstration, or failure explanation produces a different teacher target.

package/dist/web/courses/post-training/08-opsd-sdft-sdpo.vtt

@@ -0,0 +1,61 @@
+WEBVTT
+
+1
+00:00:00.700 --> 00:00:11.660
+OPSD, SDFT, and SDPO share one mechanism. A frozen teacher scores the student's prefix while receiving additional evidence. The evidence source defines the method.
+
+2
+00:00:13.090 --> 00:00:22.997
+The student distribution is p theta given the issue and its prefix. Teacher target q scores the same next token at the same prefix, but also conditions on evidence e.
+
+3
+00:00:24.427 --> 00:00:29.425
+Training matches q into p; deployment removes e.
+
+4
+00:00:30.855 --> 00:00:40.180
+On-Policy Self-Distillation, or OPSD, gives the teacher a trusted solution. Here that is the verified repair: preserve cleanup and raise Cancelled Error.
+
+5
+00:00:41.610 --> 00:00:48.294
+The teacher uses it to guide the student's own prefix without placing the privileged patch in the deployed student's prompt.
+
+6
+00:00:49.724 --> 00:00:57.904
+The OPSD paper evaluated this mechanism on reasoning tasks; the animation maps the information flow onto our code case.
+
+7
+00:00:59.335 --> 00:01:11.018
+Self-Distillation Fine-Tuning, or SDFT, gives the teacher an expert demonstration. A related repair shows the sequence inspect the flag, run cleanup, then raise the exception.
+
+8
+00:01:12.448 --> 00:01:20.628
+Unlike offline imitation, SDFT scores the current student's return-None prefix, so the demonstration guides the state the student actually reached.
+
+9
+00:01:22.058 --> 00:01:36.471
+Self-Distillation Policy Optimization, or SDPO, gives the teacher feedback about the current failure. The scalar reward is zero, but the test trace says test-cancel failed, expected Cancelled Error, and return skipped cancellation.
+
+10
+00:01:37.901 --> 00:01:43.451
+The teacher converts that explanation into dense token probabilities along the failed prefix.
+
+11
+00:01:44.881 --> 00:01:56.383
+The choice follows the evidence. A trusted solution supports OPSD. A demonstration supports SDFT. An explanatory failure trace supports SDPO.
+
+12
+00:01:57.814 --> 00:02:06.817
+If all you have is a binary outcome, GRPO may be the honest baseline; inventing a generic “feedback” field would hide these different trust boundaries.
+
+13
+00:02:08.247 --> 00:02:18.064
+None of the dense methods is automatically safe. A privileged patch can leak. A demonstration can be irrelevant. A teacher can misread feedback and densify an error.
+
+14
+00:02:19.494 --> 00:02:27.955
+New capability can overwrite old behavior, entropy can collapse, and teacher forward passes can erase claimed efficiency savings.
+
+15
+00:02:29.385 --> 00:02:42.434
+So the durable comparison is not acronym against acronym. Hold the student prefix fixed, vary the evidence, record the cost, and measure both the cancellation repair and unrelated retained capability.

package/dist/web/courses/post-training/09-credible-experiments.vtt

@@ -0,0 +1,69 @@
+WEBVTT
+
+1
+00:00:00.700 --> 00:00:08.157
+A research result requires many versioned tasks, protected information boundaries, matched baselines, and held-out evaluation.
+
+2
+00:00:09.546 --> 00:00:21.550
+Policy-visible input includes the issue, repository revision, allowed tools, and public tests. Training-only evidence may contain an expert repair, privileged patch, or public diagnostic.
+
+3
+00:00:22.939 --> 00:00:29.533
+Evaluator-only hidden tests, anti-exploit checks, and held-out repository clusters must remain outside both.
+
+4
+00:00:30.921 --> 00:00:44.512
+The Taskset preserves those boundaries together with source revision, license, split, verifier version, and content hash. Related issues from the same repository should be clustered before splitting; otherwise memorization can look like generalization.
+
+5
+00:00:45.901 --> 00:00:56.770
+Hugging Face support is worthwhile when it behaves as a reproducible importer. Pin the repository revision. Inspect the dataset card and license. Choose configuration and split explicitly.
+
+6
+00:00:58.159 --> 00:01:09.430
+Map columns through a reviewed transform, log rejected rows, preserve original identifiers, and materialize an immutable snapshot. A training run should never silently follow a changing main branch.
+
+7
+00:01:10.819 --> 00:01:23.637
+Every proposed method needs a simple credible baseline. Begin with the base model and the most direct token-imitation objective supported by the data. Add offline distillation or preference learning when those signals are relevant.
+
+8
+00:01:25.026 --> 00:01:33.578
+Compare with GRPO or another RLVR baseline before claiming value from OPSD, SDFT, or SDPO.
+
+9
+00:01:34.966 --> 00:01:45.786
+Equal optimizer tokens and equal total compute answer different questions. A distillation method may use fewer rollouts while spending additional teacher forward passes. Report both controls.
+
+10
+00:01:47.175 --> 00:02:04.540
+Evaluation should span four dimensions. Capability measures task success. Diversity measures pass-at-k, entropy, and distinct strategies. Retention tests old domains and distribution drift. Integrity audits reward exploits, leakage, and disagreement with a shadow verifier.
+
+11
+00:02:05.929 --> 00:02:09.200
+A single average can hide serious regressions.
+
+12
+00:02:10.589 --> 00:02:21.368
+Compute accounting includes student rollout tokens, teacher-scored tokens, backward tokens, verifier time, external calls, failed jobs, memory, storage, and wall clock.
+
+13
+00:02:22.757 --> 00:02:39.168
+The primary study compares GRPO, SDPO, and a routed success-and-failure method on code repair while separating public diagnostics from hidden tests. Verified math can replicate the GRPO and OPSD claims with exact graders.
+
+14
+00:02:40.557 --> 00:02:52.330
+A tool protocol can replicate token imitation and SDFT while measuring retention. The extra domains test whether a conclusion transfers; they do not replace the main story halfway through it.
+
+15
+00:02:53.719 --> 00:03:04.398
+The paper should bind every claim to a dataset revision, metric, run set, seed, confidence interval, and ablation. That structure keeps a local result from turning into a universal claim.
+
+16
+00:03:05.787 --> 00:03:19.839
+The final map is simple. Demonstrations produce imitation targets. Preferences produce relative targets. Verifiable outcomes produce rewards. Privileged solutions and failure explanations produce context-conditioned teacher distributions.
+
+17
+00:03:21.228 --> 00:03:31.234
+Evidence determines the target. The target determines the update. The update changes future behavior. Evaluation measures what improved—and what changed elsewhere.

package/dist/web/courses/post-training/10-technical-appendix.vtt

@@ -0,0 +1,69 @@
+WEBVTT
+
+1
+00:00:00.700 --> 00:00:07.565
+The first appendix isolates two GRPO evaluation choices that matter after the main mechanism is understood.
+
+2
+00:00:12.059 --> 00:00:22.337
+Length normalization changes gradient weight. A fourteen-hundred-token correct answer and a one-hundred-eighty-token correct answer may receive the same reward but contribute very different numbers of token terms.
+
+3
+00:00:26.831 --> 00:00:34.831
+Sequence averaging, token averaging, overlong penalties, and group normalization therefore change the effective objective.
+
+4
+00:00:39.326 --> 00:00:48.329
+Pass-at-one measures reliability from one sample. Pass-at-k asks whether any of k samples succeeds and exposes remaining search diversity.
+
+5
+00:00:52.824 --> 00:01:00.100
+A model can improve pass-at-one while collapsing onto one strategy, so report both metrics with the sampling configuration.
+
+6
+00:01:01.500 --> 00:01:09.590
+The systems problem is storage. A large vocabulary creates one teacher logit for every token at every trajectory position.
+
+7
+00:01:13.011 --> 00:01:22.055
+Top-k compression stores the most likely teacher logits and approximates the remaining tail. This reduces memory and bandwidth but changes the target distribution.
+
+8
+00:01:25.476 --> 00:01:32.200
+Measure divergence from full logits on a validation sample, and count teacher-forward cost when comparing efficiency.
+
+9
+00:01:33.600 --> 00:01:40.143
+The reported graphs belong in an appendix because they come from different models, datasets, and experimental budgets.
+
+10
+00:01:42.001 --> 00:01:47.822
+The OPSD study reports its largest displayed aggregate gain at the smallest Qwen3 model.
+
+11
+00:01:49.680 --> 00:02:02.137
+At one point seven billion parameters, the displayed base score is thirty-seven point one, GRPO is thirty-seven point seven, and OPSD is forty-three point four. The gaps are smaller at four and eight billion parameters.
+
+12
+00:02:03.994 --> 00:02:09.133
+That is a hypothesis for replication, not a universal model-size law.
+
+13
+00:02:10.991 --> 00:02:21.720
+The SDFT study emphasizes knowledge acquisition and retention. Its displayed strict aggregate moves from eighty to eighty-nine, while its out-of-distribution aggregate moves from eighty to ninety-eight.
+
+14
+00:02:23.578 --> 00:02:35.713
+The SDPO study reports forty-one point two for GRPO and forty-eight point eight for SDPO on its code benchmark. Their bars answer different questions and should not be read as a controlled head-to-head comparison.
+
+15
+00:02:37.571 --> 00:02:50.208
+SRPO means Sample-Routed Policy Optimization. It sends verified successes toward a GRPO-style outcome update and failures with explanations toward an SDPO-style corrective target.
+
+16
+00:02:52.066 --> 00:03:03.839
+The useful principle is signal routing. A successful rollout already shows what worked and can support sparse selection. A failed rollout with diagnostics may reveal how to recover and can support dense correction.
+
+17
+00:03:05.697 --> 00:03:11.067
+The router preserves that asymmetry instead of forcing every sample through one uniform loss.