spawnfile 0.1.25 → 0.1.27

package/dist/training/dspy/README.md

@@ -0,0 +1,206 @@
+# DSPy native-trial bridge
+
+This is the real released `dspy.GEPA` optimizer behind a versioned JSONL process.
+TypeScript supplies subscription reflection and isolated native subject trials;
+Python contains no subject tool loop, rubric implementation, dataset loader or
+paid API route. See [the protocol](protocol.md) for messages and ownership.
+
+From the repository root of a clean checkout, with `uv` installed:
+
+```sh
+cd src/training/dspy
+env -u TYPESAFE_API_KEY uv venv --python 3.14 .venv
+env -u TYPESAFE_API_KEY uv pip install --python .venv/bin/python -r requirements.lock
+env -u TYPESAFE_API_KEY uv pip install --python .venv/bin/python --no-deps -e .
+env -u TYPESAFE_API_KEY .venv/bin/python -m pytest
+env -u TYPESAFE_API_KEY .venv/bin/python -c 'import json; data=json.load(open("coverage.json")); summaries=[data["totals"],*[file["summary"] for file in data["files"].values()]]; assert all(s["covered_lines"] >= .9*s["num_statements"] and s["covered_branches"] >= .9*s["num_branches"] for s in summaries), "Every module and the total must reach 90% lines and branches"; print("COVERAGE: every module and total passed 90% lines and branches")'
+```
+
+The installed executable is `.venv/bin/spawnfile-training-dspy`. It waits for the initial
+optimization message on stdin, issues host callbacks on stdout, then emits an
+explicit terminal result/error and exits. The TypeScript supervisor must answer
+callbacks, enforce native isolation/candidate policy, and own cancellation of both
+the process and its outstanding host work.
+
+Readers accept historical DSPy schema IDs through the shared legacy-wire alias
+table; all outgoing envelopes use the current schema. Pre-rename optimizer
+checkpoints bind different bridge and request identities and fail with an explicit
+re-prepare instruction before any callback.
+
+The released integration uses normal named `dspy.Predict` components and the public
+custom-adapter/proposer seams. One native trial consumes the entire candidate; each
+Predict records the same authentic output/capture as a component trace. All files
+are optimized together. A single-line JSON envelope prevents DSPy's instruction
+normalization from stripping Markdown whitespace before native evaluation.
+The proposer requests native JSON-schema output for exactly its selected string
+resources. The TypeScript host validates the completed response independently of
+the provider; unsupported schemas fail before cognition. Plain-text legacy callbacks
+remain readable, but new real schema-constrained generations require bare JSON.
+The bridge's compatible model decoder accepts bare JSON or exactly one whole lowercase `json`
+code fence with LF delimiters; surrounding prose, multiple fences and arbitrary
+JSON extraction are rejected. Duplicate keys, nonfinite constants and changes to
+the selected resource set remain invalid. Replacement text is never trimmed.
+This model-response normalization does not change the strict host JSONL transport
+or trigger another model call.
+
+The conformance test uses the actual installed optimizer with a **simulated host**:
+two proposals update two files, seven trial callbacks compare three candidates,
+and validation scores progress from 0 to 0.5 to 1. It checks final candidate lineage,
+native callback bytes, held-out validation text exclusion, and real JSONL process
+completion. Auth/quota errors, invalid frames/replies, unsupported generation
+controls, changed component sets and budget overrun are negative tests.
+
+These tests establish the bridge seam. Real subscription generations and native
+agent acceptance are separate TypeScript integration evidence; simulated scores
+are not measurements of a real agent. Python 3.14.7 was used for the checked run.
+
+The checkpoint conformance test interrupts actual GEPA mid-iteration, restores its
+paired state and sampler, then verifies seven unique native captures and two total
+generations. Repeated restoration returns the same selected components without new
+host work. Corrupt/changed identities fail; cumulative failed-attempt offsets cannot
+reset the proposal or evaluation budgets. All checkpoint directories are private
+local evaluator state, never candidate inputs.
+
+Explicit host-authorized checkpoint forks preserve that paired state under a new
+experiment/execution identity. Exact parent bytes and unchanged optimizer inputs
+are verified before callbacks; the child durably records lineage and never writes
+the parent. The real GEPA fork test resumes a checkpoint two completed callbacks
+behind the host journal, replays those exact calls, then continues pending call 17
+without duplicate native or generation work. This is a trusted-state continuation,
+not permission to change criteria, budgets or historical successful measurements.
+
+## Proposal memory and authored rewrites
+
+The optimizer reflects on earlier **training** prompts and results, including
+criterion score/status vectors, native token usage, rejected proposals, hypotheses
+and expected impacts. Validation remains scalar selection evidence; neither its
+text nor sealed test data enters reflection. GEPA still owns search and selection.
+
+```yaml
+training:
+  optimizer:
+    model: reflection
+    proposal:
+      strategy: rewrite
+      min_change: 0.15
+      max_attempts: 2
+      history_entries: 20
+      history_bytes: 65536
+```
+
+Defaults are `refine`, `0`, `2`, `20`, and `65536`. Omitted `proposal` retains the
+legacy exact resource-map response; authored policy requests a structured
+`components`, `hypothesis`, `expectedImpact` envelope:
+
+```json
+{
+  "components": {"AGENTS.md": "Complete replacement instructions."},
+  "hypothesis": "Explicit verification addresses unsupported attribution.",
+  "expectedImpact": [
+    {"criterion": "grounding", "effect": "Fewer unsupported attributions."}
+  ]
+}
+```
+
+`expectedImpact` contains 1–20 objects with exactly the required `criterion` and
+`effect` fields. Criterion IDs must be distinct nonblank strings; effects must be
+nonblank strings of at most 1000 characters. The bridge enforces these bounds
+after parsing and converts the array to the internal criterion-to-effect map
+used by proposal history. This closed item schema supports strict structured-output providers.
+Expected impacts are the optimizer's predictions, never measured grades. Rewrite
+permits reorganizing only already-editable text; all locked resource policies
+still apply.
+
+Whitespace-equivalent or previously proposed bundles never run a native trial.
+`min_change` measures token-trigram Jaccard distance from the parent, a lexical
+novelty guard rather than semantic quality. Each repair generation consumes the
+existing total proposal budget. Once `max_attempts` is exhausted, GEPA skips the
+proposal and stops cleanly with its measured best candidate. Real provider and
+malformed-protocol failures retain their failure classification.
+
+Full prompt bodies, raw train outputs and callback results remain in protected
+checkpoints and original trial reports. Reflection receives recent chronological
+entries, with per-criterion scores/status and native usage before optional prose.
+Public atomic failures are reserved in `measurements.atomic` in current feedback
+and history before allocating prose space. This includes the host's oversized
+root envelope and nested/live check projections. Each retained row gets at most
+half its remaining allowance for atomic JSON, capped at 8 KiB and 40 checks;
+templates/units are prefix excerpts of at most 256/1000 characters, shortened
+further when escaping exceeds the wire budget. Probabilities (including null)
+remain exact; failure/omission counts include upstream and bridge omissions.
+Large prompt/reason/output fields carry explicit excerpts, original byte counts
+and SHA-256 references; historical outputs reference the original callback rather
+than repeat articles. The current training batch is separately bounded by the
+same byte allowance and deduplicates GEPA's per-component repeated feedback.
+`history_entries` and `history_bytes` bound the historical projection, not durable
+storage. If mandatory summaries alone cannot fit, oldest rows are omitted with an
+explicit count. These limits describe partial context; they never claim all text
+was shown. Final upstream state is paired with proposal bookkeeping on completion;
+exact resume restores rejected proposals and does not repeat completed work.
+
+## Authored population search
+
+With `optimizer.variants` present, Training runs the explicit population algorithm
+beside GEPA. Declare `keep` and `budget.rounds` too (`1 <= keep <= variants`):
+
+```yaml
+training:
+  optimizer:
+    method: gepa
+    model: reflection
+    variants: 4
+    keep: 2
+    budget: { rounds: 3 }
+    selection: pareto
+    boundaries: []
+    proposal:
+      rewrite_share: 0.5
+      min_change: 0.08
+      hypothesis_similarity: 0.6
+      max_attempts: 2
+```
+
+Defaults: method `gepa`, selection `pareto`, rewrite share `0.5`, behavioral floor
+`0.08`, hypothesis similarity threshold `0.6`, patience equal to rounds (off),
+boundaries `[]`, realization attempts `2`. Existing history limits remain 20
+entries and 65536 bytes. Omitted total proposal cap derives as
+`rounds * (keep + variants) * max_attempts`; it must fit the existing 1000 ceiling,
+and an explicit cap must be at least `keep + variants`. Planning calls count.
+The unchanged legacy proposal defaults apply when variants is absent.
+
+Each round plans distinct hypotheses using train-only feedback, realizes complete
+component replacements, rejects duplicates/insufficient behavioral change before
+trials, evaluates complete validation, and retains a bounded Pareto or top-k pool.
+Provenance includes both change measures, rejected attempts, per-case scores,
+rounds, stop reason and effective whole-file/heading scope. See
+[the protocol](protocol.md#optional-population-search) for exact wire shapes,
+behavioral scoring and the ordered Evaluator seam. Population training feedback
+and validation use at most `concurrency` active callbacks (default 1), returning
+results in request order even when callbacks complete out of order.
+
+Deterministic simulated tests cover search and actual-process host verification.
+Settled-callback checkpoint resume replays with no extra spend or Python pickle.
+Population recovery with uncertain/failed attempts and checkpoint forks fails
+closed; start a new run for those cases. These tests make no claim of live model
+quality or native-agent improvement.
+
+Population change evidence is explicitly `bridge-reported-v2`: behavioral and
+lexical heuristics are not host-certified. Logical Markdown wrapping/heading
+labels are excluded and repeated units count once per component; independent rules split alike at periods, commas
+and semicolons. Adjacent-token features retain directive/action attachment within
+units, keeping conditional and workflow clauses together. History rejects
+zero-distance copies with the same metric, including when the floor is zero.
+Repairs exclude their own
+slot's hypotheses from deduplication. The host binds accepted components to exact
+recorded realization responses and checks seed training, strategy/attempt records
+and stop feasibility; see protocol.md for the unrecorded-shuffle limitation.
+Integration `trainingOptimizer` may supply `boundaries` under the same 50-string,
+1..500-character limits; dual declarations must match verbatim and in order.
+Declared text constraints are enforced by Python admission and independently by the
+TypeScript host: `preserve ["component"]`, `require-text ["component", "literal"]`,
+`forbid-text ["component", "literal"]`. Arguments are JSON strings. Unknown components,
+unsupported prose and invalid seeds fail before callbacks; violations never reach trials.
+These are exact text guarantees, not runtime semantic guarantees. See
+[population search](../../docs/population-search.md) for examples and limits.
+The change heuristic excludes HTML comments, leading closed YAML front matter and
+single-line reference definitions, while retaining fenced literal content.

package/dist/training/dspy/protocol.md

@@ -0,0 +1,311 @@
+# DSPy host protocol v2
+
+Run `spawnfile-training-dspy` from the bridge virtual environment. One process handles one
+optimization. Send one finite UTF-8 JSON object per line, terminated by LF, maximum
+4 MiB per frame. Duplicate keys, unknown fields and invalid IDs fail closed.
+Stdout contains protocol messages only; DSPy diagnostics use stderr.
+
+The host owns input resolution, grouped splits, sealed tests, candidate policy,
+native isolated trials, scoring, subscription routes, capture, cancellation and
+the total experiment deadline. The bridge receives no datasets, credentials or
+sealed-test cases. It never opens a resource path or invokes a native tool loop.
+Only allowlisted editable text components belong in `components`; locked text stays
+with the host. The host must check every callback candidate before running it.
+
+## Start: host → bridge
+
+```json
+{"schema":"spawnfile.training.dspy.v2","kind":"optimize","id":"experiment-1","components":{"AGENTS.md":"Agent instructions\n","skills/report/SKILL.md":"# Report\n"},"trainCaseIds":["train-1"],"validationCaseIds":["validation-1"],"budget":{"maxMetricCalls":10,"maxProposals":2},"seed":7}
+```
+
+`concurrency` is an optional integer from 1 through 8 (wire default 1). Training
+resolves the run setting to 4 with provisioned slots, or legacy 1 without slots.
+GEPA uses this value as `num_threads`. The host bounds concurrent delivery and
+preserves ordered observation handling within each trial. Writes are atomic JSONL
+lines; errors/disconnect fail every waiter. Cancellation drains cooperative work;
+unsettled work is reported incomplete and its resources are retained.
+
+Callback IDs emitted by the bridge are SHA-256 over canonical `[method, params]`
+plus a per-request occurrence ordinal. The paired checkpoint saves those ordinals,
+so replay does not depend on completion order. This replaces v1 sequential IDs;
+v1 optimizer checkpoints require their original bridge.
+
+Components are a nonempty map of resource IDs to complete text (maximum 64).
+Resource IDs are opaque to Python; TypeScript owns native path mapping. Train and
+validation IDs must be nonempty, unique, disjoint lists. No test role exists.
+The optional seed (default 0) controls GEPA sampling only, never a subscription
+model's generation seed. Both budget fields are required positive integers.
+
+`maxMetricCalls` caps actual native evaluation callback attempts, including failed
+attempts; asking again for feedback from a captured result spends no new attempt.
+GEPA may reevaluate the same candidate/case; each callback requests a fresh trial.
+`maxProposals` caps actual reflection generation callback attempts. One proposal
+can replace all editable components together. No internal API retry is permitted.
+
+The complete baseline validation must fit the budget. Before each later iteration,
+the bridge reserves one training baseline, one proposed-candidate training trial,
+and one complete validation pass. It can finish below the cap. A result with zero
+proposals means baseline-only selection, not demonstrated optimization.
+
+## Trial: bridge → host
+
+```json
+{"schema":"spawnfile.training.dspy.v2","kind":"request","id":"call-1","method":"evaluate","params":{"caseId":"train-1","split":"train","components":{"AGENTS.md":"Agent instructions\n","skills/report/SKILL.md":"# Report\n"}}}
+```
+
+```json
+{"schema":"spawnfile.training.dspy.v2","kind":"response","id":"call-1","result":{"score":0.5,"feedback":"The filing lacked evidence.","output":"Observed native output","captureId":"capture-1"}}
+```
+
+Evaluation params may include `round`, a nonnegative integer. The current bridge
+always records seed round zero, then GEPA's upstream iteration-start ID or the
+population's declared round. The host persists attribution with the native report;
+round is part of callback identity. Legacy callbacks may omit it and remain
+unattributed. Interrupted checkpoints made before round attribution require their
+original bridge; adding params changes content-based callback IDs.
+
+All four result fields are required. `score` is a finite number in [0,1]; the host
+owns how gates, unknown/error states and coverage affect that scalar. Infrastructure
+failure must use an error response, never a fabricated subject score. `captureId`
+identifies an authentic host capture. Missing evidence is not successful evidence.
+
+For `split:"validation"`, both `feedback` and `output` **must be empty strings**.
+Validation supplies a selection scalar and capture reference only. Training outputs
+and feedback reach reflection. Repeated predictor traces refer to one whole-bundle
+native trial; they do not establish per-file causal attribution or observed loading.
+
+Current and historical proposer views reserve public atomic failures under
+`measurements.atomic` before optional prose. They read root `atomic` envelopes,
+evaluation checks/children and live results only. Per-row atomic JSON is capped at
+8 KiB, 40 checks and half the available row allowance, with template/unit prefix
+excerpts capped at 256/1000 characters and further shortened for wire size.
+Nullable probabilities remain exact; `failed_check_count` and
+`omitted_failed_checks` account for upstream and bridge omissions. Numerical
+summaries and minimum diagnostic/detail space determine row retention; under
+small budgets oldest rows may be omitted with explicit `omittedEntries`.
+
+## Reflection: bridge → host
+
+```json
+{"schema":"spawnfile.training.dspy.v2","kind":"request","id":"call-2","method":"generate","params":{"purpose":"reflection","messages":[{"role":"system","content":"..."},{"role":"user","content":"..."}],"options":{}}}
+```
+
+```json
+{"schema":"spawnfile.training.dspy.v2","kind":"response","id":"call-2","result":{"text":"{\"AGENTS.md\":\"Revised instructions\\n\",\"skills/report/SKILL.md\":\"# Revised skill\\n\"}","model":"observed-subscription-model","usage":null}}
+```
+
+The host uses its explicitly selected subscription inference adapter and the exact
+messages. No paid API fallback, native tools, hidden judge, teacher or DSPy default
+model route exists. `options` may contain `responseSchema`, an object JSON Schema.
+The public component proposer derives required string properties from exactly the
+selected resource paths and forbids extra properties. The host forwards that schema
+to its selected provider and validates the completed JSON itself; unsupported
+schemas fail before generation. Sampling controls, nontext parts and tool use remain
+unsupported. An omitted schema retains plain-text compatibility.
+
+Return the generation's reported model and reported usage (or `null` when unavailable).
+If the inference provider supplied only a requested model fallback, include
+`modelBasis: "requested-assumption"`. `"provider-reported"` is the explicit observed
+alternative; omission retains the existing provider-reported contract. Preserve
+this field in final generation provenance. Requested fallbacks are never evidence
+for actual-model pricing, even when their string matches the rate catalog.
+The bridge retains these without inventing token totals or price estimates. Success
+means completed text generation; auth, quota, cancellation and unsupported controls
+must be errors. The reflection prompt asks for a JSON map replacing exactly the
+selected resource IDs. A malformed response aborts; it is not silently retried.
+
+## Errors and process completion
+
+```json
+{"schema":"spawnfile.training.dspy.v2","kind":"response","id":"call-2","error":{"code":"quota_exhausted","message":"Subscription quota exhausted"}}
+```
+
+Exactly one `result` or `error` is allowed, with the pending callback ID. One
+reader dispatches replies by ID; up to `concurrency` callbacks may be outstanding. EOF, malformed replies and host errors are latched;
+DSPy's internal score/proposal error handling cannot convert them into success.
+
+The bridge terminates with exit 0 and a `kind:"result"` envelope containing the
+original request `id`, selected `components`, and `provenance`:
+
+- `optimizer:"dspy.GEPA"`, `dspyVersion:"3.3.1"`, `gepaVersion:"0.1.4"`, `seed`.
+- `evaluations`, `proposals`, `selectedCandidateIndex`.
+- `candidates`: full decoded component maps, parent indices, `validationScore`,
+  and GEPA's `discoveryEvaluations` accounting (distinct from host attempt count).
+- `captures`: `{caseId,split,captureId}` for completed host evaluations.
+- `generations`: `{model,usage,modelBasis?}` for completed host reflection generations.
+
+Failure emits `kind:"error"`, `error:{code,message}`, and exits 1. A rejected start
+has `id:"unknown"`. Neither a process start nor exit 0 alone is acceptance: the host
+must validate the nonempty terminal result, complete component set, bounds and
+provenance. The host applies final candidate policy, freezes the selected candidate,
+and independently owns any sealed-test comparison or local candidate export.
+
+The host owns timeout/cancellation: terminate this process and cancel its outstanding
+callback work together. Killing Python alone does not stop a native trial owned by
+the TypeScript host. An optional `checkpoint:{directory,identity}` selects caller-protected durable
+state. The caller owns exclusive access. The host journal records exact callback
+reservations/results; the Python bridge pairs released GEPA state with session and
+sampler state. On resume, host-derived `recovery:{evaluations,proposals}` accounts for
+prior failed or uncertain attempts without inventing completed captures. These
+offsets reduce remaining budget; they are not candidate feedback. No arbitrary
+agent-produced checkpoint or pickle is accepted.
+
+Seed validation is reused on restore, because released GEPA calls it before loading
+state. Completed callbacks after the last checkpoint are replayed by exact original
+callback identity. Model/criteria/environment drift rejects resume before native work.
+The TypeScript training owner separately persists frozen selection and sealed-test
+reports; sealed test data never enters this bridge.
+
+## Explicit checkpoint fork
+
+An authorized host may start a **new** experiment from a trusted paired checkpoint:
+
+```json
+{"checkpoint":{"directory":"/child/checkpoint/dspy","identity":"new-execution","fork":{"directory":"/parent/checkpoint/dspy","identity":"original-execution","id":"original-experiment","digest":"sha256:<64 lowercase hex characters>"}}}
+```
+
+`fork.digest` binds the exact parent `optimizer.json` bytes. Its envelope digest and
+old optimizer identity must also match. The old identity is recomputed from the
+current request with only experiment ID and execution identity reverted: components,
+case splits, budgets, seed, proposal policy and pinned bridge versions stay exact.
+Host-derived recovery offsets remain separately accountable as on ordinary resume.
+
+The bridge copies committed GEPA bytes, session outcomes/history/counters and
+sampler/RNG state into a disjoint child directory, then atomically commits the new
+identity and parent lineage **before any host callback**. It never writes the
+parent. Subsequent child resume requires the same fork descriptor and uses its own
+committed state; it does not require the parent directory to remain online. Further
+forks retain previous lineage. Forks are explicit authorization to use trusted local
+pickle state, not a format for importing untrusted agent artifacts.
+
+Callbacks retain their content-derived IDs and persisted occurrence ordinals. A paired Python checkpoint can
+lag the host journal: the host must replay matching completed calls byte-for-byte
+and fulfill only the pending work. Historical successful outcomes are not rescored
+by this import. The host owns any separately authorized measurement repair and must
+preserve cumulative failed-attempt accounting; copying state is not a new budget.
+
+## Released integration seam
+
+The actual `dspy.GEPA.compile` receives ordinary named `dspy.Predict` components,
+nonempty separate train/validation examples, whole-program scalar/text feedback,
+and the public `instruction_proposer` hook. GEPA owns proposal iteration, acceptance,
+validation selection and lineage. Python's hook only packages reflection text.
+
+DSPy 3.3.1 normalizes `Signature.instructions` with `inspect.cleandoc`; each native
+text is therefore stored as a single-line JSON envelope and decoded before trials.
+This preserves untouched Markdown indentation and final newlines exactly. Normal
+Predict execution records authentic host results through a custom `dspy.Adapter`.
+The separate reflection model uses `BaseLM.forward_contract = "typed_lm"`.
+
+Checked against the installed releases and their sources:
+[DSPy 3.3.1 GEPA](https://github.com/stanfordnlp/dspy/blob/3.3.1/dspy/teleprompt/gepa/gepa.py),
+[custom proposal hook](https://dspy.ai/api/optimizers/GEPA/GEPA_Advanced/#custom-instruction-proposers),
+[typed BaseLM](https://github.com/stanfordnlp/dspy/blob/3.3.1/dspy/clients/base_lm.py).
+
+## Optional population search
+
+`search` switches only the optimizer search owner; an omitted block retains the
+released GEPA path, prompts and checkpoint format. The strict block is:
+
+```json
+{"variants":4,"keep":2,"rounds":3,"selection":"pareto","rewriteShare":0.5,"minChange":0.08,"hypothesisSimilarity":0.6,"patience":3,"boundaries":[]}
+```
+
+`variants` and `keep` are required integers in 1..1000; `rounds` is a required
+positive safe integer, with
+`keep <= variants`. Defaults for the remaining fields are `pareto`, `0.5`, `0.08`,
+`0.6`, `rounds` (no early patience stop), and `[]`. Boundaries contain at most 50
+strings of 1..500 characters. Each string is `preserve ["component"]`,
+`require-text ["component", "literal"]`, or `forbid-text ["component", "literal"]`.
+Arguments are JSON strings; names must identify editable components. Preserve means
+exact equality with the seed; required/forbidden text means case-sensitive substring
+presence/absence. All constraints must pass. Invalid declarations/prose fail with
+`unsupported_boundaries`; invalid seeds fail with `boundary_violation`, before callbacks.
+Violating proposals receive `hard-boundary` rejection before evaluation; bounded repairs
+are allowed. The host independently enforces these constraints before native trials
+and on final candidates. These gates assert text properties, not semantic compliance.
+See docs/population-search.md for the authoring contract. Proposal `max_attempts` still defaults to 2 and
+caps realization attempts per slot; each planning and realization call consumes
+one `maxProposals` unit. The seed's complete validation **and training** must fit
+`maxMetricCalls`; `maxProposals` must be at least `keep + variants`.
+
+Round 0 evaluates the seed on complete validation and training. Each later round
+assigns N slots across a deterministically shuffled kept population. Each parent
+receives one planning generation listing its slot strategies, then one realization
+per hypothesis (plus bounded rejection repairs). The global rewrite slot count is
+`ceil(variants * rewriteShare)`. Train feedback is cached by candidate/case; only
+training text enters generations. Validation always evaluates every declared case.
+
+Pre-trial gates reject normalized duplicates (including earlier rejected attempts
+and zero-distance formatting/reordering copies under the instruction-change metric),
+behavioral change below `minChange`, same-round accepted siblings below that floor,
+and hypothesis content-token Jaccard similarity >= `hypothesisSimilarity` with an
+planning or realization hypothesis from another slot or earlier round.
+Repairs exclude their own slot history from hypothesis deduplication. Replacement keys must equal selected editable keys.
+Realizations use the existing `components`, `hypothesis`, `expectedImpact` envelope;
+the `criterion` IDs in its `expectedImpact` array must equal the plan's
+`targetedCriteria`. Each item contains exactly `criterion` and `effect`; the bridge
+converts the array to a criterion-to-effect map for internal history. Planning returns
+`hypotheses: [{strategy,hypothesis,targetedCriteria,plan}]` in requested slot order.
+Both prompts treat component text, outputs and feedback as data.
+
+Instruction change v2 joins soft-wrapped paragraph/list continuations and treats
+Setext and ATX headings as unscored labels. It splits lists, rows, sentences and
+fenced blocks, strips inline emphasis/link syntax and applies NFKC/case-folding.
+Independent rules split alike at periods, semicolons and commas. Leading
+conditional/workflow clauses (if/unless/when/before/after/until/once) retain their
+comma-delimited actions; otherwise/else/then alternatives attach to the preceding
+rule. Identical whole
+units match independent of position; repetitions count once per component, in
+both the numerator and denominator. Remaining units pair by highest weighted
+adjacent-token multiset Jaccard similarity, including start/end boundaries. Each
+pair weighs 4 if either token is a directive or numeric, otherwise 1. This detects
+directive/action reversals even within a clause. History uses zero distance under
+this same metric regardless of the configured floor. Unpaired units cost 1;
+paired units cost 1 minus similarity. Divide by
+the maximum total unit count and clamp to [0,1]. This is a deterministic heuristic,
+not proof of semantic change. Lexical trigram distance is reported separately.
+
+Pareto retention uses complete per-case score vectors. Rank the nondominated front
+by mean, breaking ties by stable candidate index; fill remaining capacity from the
+rest by the same ordering. `top-k` ranks the whole pool by mean. Retain at most K;
+the seed may drop. The result selects the best mean among all measured candidates,
+with stable index ties. It reports `optimizer:"training.population"`, the existing
+counters/captures/generations, candidate `perCaseScores`, and `rounds` including
+round 0. Each round records parents, all variant slots and kept indexes; every slot
+records strategy, hypothesis, targeted criteria, behavioral/lexical change, status,
+rejection reason, candidate index (or null), and individual realization attempts.
+`stopReason` is `rounds`, `budget` or `patience`. Unfunded slots remain empty.
+Population generations include the exact response `text` (legacy generations
+remain unchanged); the host checks these bytes against its recorded responses.
+`changeEvidence: "bridge-reported-v2"` labels behavioral and lexical values as
+bridge-reported heuristics. The host does not recompute or certify novelty scores,
+threshold compliance, or hypothesis similarity.
+
+Optional `editScope` maps exactly the component keys to `"whole-file"` or a nonempty
+array of declared headings. Population provenance echoes its effective value,
+defaulting every resource to whole-file. The host retains actual edit-policy
+validation and independently verifies full validation captures, per-case scores,
+mean scores, lineage, retention, seed training, strategy allocation, attempt/status
+consistency and accepted components/hypotheses against ordered realization
+responses before accepting a selected result. It checks stop reasons against round
+count, observed stagnation and remaining budget. For a budget stop before a round
+starts, the unrecorded shuffled parent is unknown to the host: at least one retained
+parent must be unfundable including uncached training. This is feasibility checking,
+not independent replay of Python RNG or certification of novelty gates.
+
+`Evaluator.evaluate_many(requests)` returns results in **request order**. The
+session-backed thread pool bounds active training and validation callbacks by
+`concurrency` (default 1). Completion timing does not change case-score binding
+or candidate selection; the host owns leased native trial slots.
+
+Population state is reconstructed from the request, seed and ordered callback
+responses. The protected TypeScript journal replays settled callbacks without new
+spend; Python creates no checkpoint or pickle. A settled callback interrupted before
+receipt is safe to replay. A pending/failed callback produces recovery offsets that
+can change earlier budget decisions: population rejects such recovery before any
+callback with `unsupported_checkpoint`. Population checkpoint forks also fail closed.
+Legacy paired GEPA checkpoint behavior and absent-field identity bytes are retained.
+
+Instruction-change parsing excludes HTML comments, closed leading YAML front matter
+and single-line reference definitions before segmentation; fenced content remains scored.

package/dist/training/dspy/pyproject.toml

@@ -0,0 +1,36 @@
+[build-system]
+requires = ["setuptools==80.9.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "spawnfile-training-dspy-bridge"
+version = "0.1.0"
+description = "Host-mediated native trials through the released DSPy GEPA optimizer"
+requires-python = ">=3.11,<3.15"
+dependencies = ["dspy==3.3.1", "gepa==0.1.4"]
+
+[project.optional-dependencies]
+test = ["pytest==8.4.2", "pytest-cov==7.0.0", "coverage==7.16.0"]
+
+[project.scripts]
+spawnfile-training-dspy = "spawnfile_training_dspy.main:cli"
+
+[tool.setuptools.packages.find]
+include = ["spawnfile_training_dspy*"]
+
+[tool.setuptools.package-data]
+spawnfile_training_dspy = ["legacy_wire_aliases.json"]
+
+[tool.pytest.ini_options]
+testpaths = ["spawnfile_training_dspy"]
+python_files = ["*_test.py"]
+addopts = "--strict-config --strict-markers --cov=spawnfile_training_dspy --cov-branch --cov-report=term-missing --cov-report=json --cov-fail-under=90"
+
+[tool.coverage.run]
+branch = true
+source = ["spawnfile_training_dspy"]
+omit = ["*/*_test.py"]
+
+[tool.coverage.report]
+show_missing = true
+fail_under = 90

package/dist/training/dspy/requirements.lock

@@ -0,0 +1,69 @@
+aiohappyeyeballs==2.7.1
+aiohttp==3.14.3
+aiosignal==1.4.0
+annotated-types==0.8.0
+anyio==4.15.1
+attrs==26.1.0
+boto3==1.43.93
+botocore==1.43.93
+cachetools==7.1.8
+certifi==2026.7.22
+charset-normalizer==3.5.1
+click==8.5.0
+cloudpickle==3.1.2
+coverage==7.16.0
+diskcache==5.6.3
+distro==1.9.0
+dspy==3.3.1
+fastuuid==0.14.0
+filelock==3.32.6
+frozenlist==1.8.0
+fsspec==2026.7.0
+gepa==0.1.4
+h11==0.16.0
+hf-xet==1.6.0
+httpcore==1.0.9
+httpx==0.28.1
+huggingface-hub==1.31.0
+idna==3.19
+importlib-metadata==8.9.0
+iniconfig==2.3.0
+jinja2==3.1.6
+jiter==0.17.0
+jmespath==1.1.0
+json-repair==0.63.4
+jsonschema==4.26.0
+jsonschema-specifications==2025.9.1
+litellm==1.100.1
+markupsafe==3.0.3
+multidict==6.8.0
+openai==2.54.0
+orjson==3.12.0
+packaging==26.3
+pluggy==1.6.0
+propcache==0.5.2
+pydantic==2.13.5
+pydantic-core==2.46.5
+pydantic-settings==2.15.0
+pygments==2.21.0
+pytest==8.4.2
+pytest-cov==7.0.0
+python-dateutil==2.9.0.post0
+python-dotenv==1.2.3
+pyyaml==6.0.3
+referencing==0.37.0
+regex==2026.9.10
+requests==2.34.2
+rpds-py==2026.6.3
+s3transfer==0.19.2
+six==1.17.0
+sniffio==1.3.1
+tenacity==9.1.4
+tiktoken==0.14.0
+tokenizers==0.23.2
+tqdm==4.70.1
+typing-extensions==4.16.0
+typing-inspection==0.4.4
+urllib3==2.7.0
+yarl==1.24.5
+zipp==4.1.0