spawnfile 0.1.25 → 0.1.27

package/dist/training/dspy/spawnfile_training_dspy/AGENTS.md

@@ -0,0 +1,63 @@
+# DSPy process implementation
+
+Owns the versioned JSONL endpoint, exact-text component representation and the
+released DSPy GEPA integration. Inherit the bridge guide.
+
+- `protocol.py` validates messages and canonicalizes legacy schema IDs through the
+  shared legacy-wire alias table linked as `legacy_wire_aliases.json`; `transport.py` serializes host callbacks.
+- `session.py` owns bridge-side attempt counts and latched host failures.
+- `native.py` exposes component predictors and authentic host trial results.
+- `reflection.py` uses a typed DSPy BaseLM backed only by the host callback. Its
+  model response decoder may unwrap one whole JSON fence; never loosen JSONL
+  transport parsing or alter replacement text to recover a model envelope.
+- `optimizer.py` invokes DSPy GEPA; `main.py` provides one-request process I/O.
+- No native cognition, dataset splitting, rubric logic, credentials or paid API
+  fallback here. Native component bytes must survive DSPy's text normalization.
+- Tests sit beside source. Run the bridge's pinned pytest/coverage command.
+- Simulated hosts establish integration conformance, not real native acceptance.
+
+- `checkpoint.py` pairs upstream GEPA state with this session's exact callback
+  sequence and upstream epoch sampler/RNG state. Never substitute optimizer replay
+  from scratch or accept an unpaired upstream pickle. Checkpoint bytes are trusted
+  evaluator state; they are not an import format for agent-produced artifacts.
+- Resume reuses the original seed validation, because upstream invokes it before
+  loading its state. Host callbacks following the saved boundary replay only if their
+  exact identities match. Recovery attempt offsets are host-owned accounting, never
+  optimizer feedback or newly generated trials.
+
+- `proposal_history.py` mechanically projects train-only callback history and
+  proposal metadata into bounded reflection context. It never scores outputs or
+  summarizes through another model. Preserve full source records and explicit
+  omission/excerpt metadata; numerical criterion results take priority over prose.
+- Reserve public atomic diagnostics as `measurements.atomic` before prose in both
+  current feedback and history. Read only projected `atomic` envelopes (root,
+  evaluation children and live results); never inspect raw grader receipts.
+  Bound diagnostics by encoded bytes, 40 checks and unit excerpts; retain failure
+  and omission counts. Test the TypeScript-to-Python proposer path with large text.
+  Current-output compaction reserves each declared artifact before optional chat;
+  generic prefix truncation can discard file-based articles a second time.
+  Refuse unrepresentable artifact identifiers rather than silently dropping them.
+- Benign duplicate/novelty exhaustion uses GEPA's skip seam and stops the session;
+  it must not latch a provider failure or launch duplicate native trials. Pair the
+  upstream final state save too: GEPA omits the final state-saved callback.
+
+- An explicit checkpoint fork verifies the exact parent optimizer bytes and old
+  request/execution identity, imports paired state under the new identity, and
+  commits parent lineage before callbacks. Parent directories remain read-only.
+  Child resume requires the same lineage; importing never resets callback IDs,
+  completed outcomes, sampler state or attempt budgets.
+
+Protocol v2 uses one locked whole-line writer and one response reader dispatching
+by callback ID. Session locks protect reservations/bookkeeping, never host calls.
+Replay IDs use request content plus occurrence counts, saved explicitly with the
+paired checkpoint, rather than deriving a sequence from completed-result count.
+- `search/` owns opt-in population rounds, behavioral novelty and bounded retention.
+  Omitted search dispatches to the unchanged released GEPA invocation. The search
+  package reconstructs state by settled host-journal replay, without pickle; failed
+  or uncertain recovery offsets and population forks fail closed before callbacks.
+  `Evaluator.evaluate_many` preserves request order and bounds active evaluations
+  by request.concurrency for both training feedback and validation.
+
+Session evaluation callbacks carry round zero for the seed and GEPA's actual
+on_iteration_start iteration thereafter. Register the session alongside checkpoint
+callbacks; resumed upstream iteration IDs remain authoritative.

package/dist/training/dspy/spawnfile_training_dspy/__init__.py

@@ -0,0 +1,6 @@
+"""A narrow, host-mediated bridge to the actual DSPy optimizer."""
+
+from .optimizer import optimize
+from .protocol import SCHEMA, BridgeError, OptimizeRequest
+
+__all__ = ["SCHEMA", "BridgeError", "OptimizeRequest", "optimize"]

package/dist/training/dspy/spawnfile_training_dspy/checkpoint.py

@@ -0,0 +1,146 @@
+"""Pair upstream GEPA state with the host session and upstream sampler state.
+
+Only evaluator-owned directories are accepted by the TypeScript caller. Pickle
+is an upstream checkpoint format, never an agent-supplied dataset artifact.
+"""
+import base64
+import hashlib
+import json
+import os
+import random
+from collections import Counter
+from pathlib import Path
+
+from gepa.strategies.batch_sampler import EpochShuffledBatchSampler
+
+from .protocol import BridgeError, EvaluationResult
+from .transport import parse_json
+
+
+def _bytes(value):
+    return json.dumps(value, sort_keys=True, separators=(",", ":")).encode()
+
+
+def _tuples(value):
+    return tuple(_tuples(item) for item in value) if isinstance(value, list) else value
+
+
+class Checkpoint:
+    def __init__(self, session):
+        self.session = session
+        self.directory = Path(session.request.checkpoint.directory)
+        self.work = self.directory / "gepa-work"
+        self.path = self.directory / "optimizer.json"
+        self.sampler = EpochShuffledBatchSampler(1, random.Random(session.request.seed))
+        self.identity = self._identity(session.request.id, session.request.checkpoint.identity)
+        self.lineage = None
+        fork = session.request.checkpoint.fork
+        if fork:
+            parent, child = Path(fork.directory).resolve(), self.directory.resolve()
+            if (parent == child or parent in child.parents or child in parent.parents
+                    or fork.id == session.request.id):
+                raise BridgeError("checkpoint_mismatch", "checkpoint fork requires a distinct experiment and disjoint directory")
+        if self.path.exists():
+            self._restore()
+        elif fork:
+            self._import(fork)
+        else:
+            self.work.mkdir(mode=0o700, parents=True, exist_ok=True)
+            # Unpaired upstream files can only be an interrupted save. Never load them.
+            (self.work / "gepa_state.bin").unlink(missing_ok=True)
+
+    def _identity(self, experiment_id, execution):
+        request = self.session.request.model_dump(exclude={"checkpoint", "recovery"})
+        # Preserve legacy checkpoint identity bytes when population fields are absent.
+        for field in ("search", "editScope"):
+            if request[field] is None:
+                del request[field]
+        request["id"] = experiment_id
+        return hashlib.sha256(_bytes({
+            "request": request, "execution": execution, "versions": ["dspy3.3.1", "gepa0.1.4"],
+        })).hexdigest()
+
+    def _read(self, path, identity, digest=None):
+        try:
+            data = path.read_bytes()
+            if digest and "sha256:" + hashlib.sha256(data).hexdigest() != digest:
+                raise ValueError("parent bytes changed")
+            envelope = parse_json(data)
+            saved = envelope["payload"]
+            if (saved["identity"] != identity
+                    or hashlib.sha256(_bytes(saved)).hexdigest() != envelope["digest"]):
+                raise ValueError("payload identity or digest changed")
+            # Decode and validate before creating a child checkpoint or host work.
+            base64.b64decode(saved["gepa"], validate=True)
+            return saved
+        except (OSError, ValueError, TypeError, KeyError, BridgeError) as error:
+            raise BridgeError("checkpoint_mismatch", "optimizer checkpoint integrity or identity mismatch; re-prepare the training run with the installed bridge") from error
+
+    def _import(self, fork):
+        saved = self._read(Path(fork.directory) / "optimizer.json",
+                           self._identity(fork.id, fork.identity), fork.digest)
+        self.lineage = {"parent": fork.model_dump(), "previous": saved.get("lineage")}
+        self._apply(saved)
+        # Persist the new identity and lineage before even replaying a host callback.
+        self.on_state_saved(None)
+
+    def _restore(self):
+        saved = self._read(self.path, self.identity)
+        self.lineage = saved.get("lineage")
+        fork = self.session.request.checkpoint.fork
+        if ((self.lineage is None) != (fork is None)
+                or (fork and self.lineage.get("parent") != fork.model_dump())):
+            raise BridgeError("checkpoint_mismatch", "optimizer checkpoint fork lineage mismatch")
+        self._apply(saved)
+
+    def _apply(self, saved):
+        self.work.mkdir(mode=0o700, parents=True, exist_ok=True)
+        (self.work / "gepa_state.bin").write_bytes(base64.b64decode(saved["gepa"], validate=True))
+        state = saved["session"]
+        for name in ["evaluations", "proposals", "captures", "generations", "results", "proposal_records", "proposal_stopped"]:
+            setattr(self.session, name, state[name])
+        self.session.seed_replay = {
+            item["params"]["caseId"]: EvaluationResult.model_validate(item["result"])
+            for item in self.session.results[:len(self.session.request.validationCaseIds)]
+        }
+        if hasattr(self.session.host, "restore"):
+            self.session.host.restore(saved["callbacks"])
+        sample = saved["sampler"]
+        for key, value in sample.items():
+            if key == "rng":
+                self.sampler.rng.setstate(_tuples(value))
+            elif key == "id_freqs":
+                self.sampler.id_freqs = Counter(dict(value))
+            else:
+                setattr(self.sampler, key, value)
+
+    def on_state_saved(self, event):
+        if self.session.failure is not None:
+            return
+        state = self.work / "gepa_state.bin"
+        sample = {key: value for key, value in vars(self.sampler).items() if key not in {"rng", "id_freqs"}}
+        sample["rng"] = self.sampler.rng.getstate()
+        sample["id_freqs"] = list(self.sampler.id_freqs.items())
+        payload = {
+            "identity": self.identity, "gepa": base64.b64encode(state.read_bytes()).decode(),
+            "session": {name: getattr(self.session, name) for name in
+                        ["evaluations", "proposals", "captures", "generations", "results", "proposal_records", "proposal_stopped"]},
+            "sampler": sample,
+            "callbacks": self.session.host.checkpoint() if hasattr(self.session.host, "checkpoint") else {},
+        }
+        if self.lineage is not None:
+            payload["lineage"] = self.lineage
+        temporary = self.path.with_suffix(".tmp")
+        with temporary.open("wb") as output:
+            output.write(_bytes({"payload": payload, "digest": hashlib.sha256(_bytes(payload)).hexdigest()}))
+            output.flush()
+            os.fsync(output.fileno())
+        os.replace(temporary, self.path)
+        directory = os.open(self.directory, os.O_RDONLY)
+        try:
+            os.fsync(directory)
+        finally:
+            os.close(directory)
+
+    def kwargs(self):
+        return {"batch_sampler": self.sampler, "callbacks": [self]}

package/dist/training/dspy/spawnfile_training_dspy/legacy_wire_aliases.json

@@ -0,0 +1,90 @@
+{
+  "paideia.adjudication.v1": "spawnfile.training.adjudication.v1",
+  "paideia.atomic-grade.v1": "spawnfile.training.atomic-grade.v1",
+  "paideia.atomic-grader.v1": "spawnfile.training.atomic-grader.v1",
+  "paideia.atomic-grading-blob.v1": "spawnfile.training.atomic-grading-blob.v1",
+  "paideia.authored-judge.v1": "spawnfile.training.authored-judge.v1",
+  "paideia.bootstrap-failed.v1": "spawnfile.training.bootstrap-failed.v1",
+  "paideia.candidate.v1": "spawnfile.training.candidate.v1",
+  "paideia.check-templates.v1": "spawnfile.training.check-templates.v1",
+  "paideia.claude-attested-response.v3": "spawnfile.training.claude-attested-response.v3",
+  "paideia.cli-evaluation.v1": "spawnfile.training.cli-evaluation.v1",
+  "paideia.cli-failure.v1": "spawnfile.training.cli-failure.v1",
+  "paideia.codex-contained-response.v1": "spawnfile.training.codex-contained-response.v1",
+  "paideia.command-checkpoint.v1": "spawnfile.training.command-checkpoint.v1",
+  "paideia.command-execution.v1": "spawnfile.training.command-execution.v1",
+  "paideia.comparison-checkpoint.v1": "spawnfile.training.comparison-checkpoint.v1",
+  "paideia.comparison.v1": "spawnfile.training.comparison.v1",
+  "paideia.cost-config.v1": "spawnfile.training.cost-config.v1",
+  "paideia.cost-dataset.v1": "spawnfile.training.cost-dataset.v1",
+  "paideia.cost-record.v1": "spawnfile.training.cost-record.v1",
+  "paideia.cost-report.v1": "spawnfile.training.cost-report.v1",
+  "paideia.council.v6": "spawnfile.training.council.v6",
+  "paideia.council.v7": "spawnfile.training.council.v7",
+  "paideia.daimon-native.broker-requests.v1": "spawnfile.training.daimon-native.broker-requests.v1",
+  "paideia.daimon-native.broker-seal.v1": "spawnfile.training.daimon-native.broker-seal.v1",
+  "paideia.daimon-native.child-diagnostics.v1": "spawnfile.training.daimon-native.child-diagnostics.v1",
+  "paideia.daimon-native.child-grok-slot-preflight.v1": "spawnfile.training.daimon-native.child-grok-slot-preflight.v1",
+  "paideia.daimon-native.child-sandbox-preflight.v1": "spawnfile.training.daimon-native.child-sandbox-preflight.v1",
+  "paideia.daimon-native.evidence.v1": "spawnfile.training.daimon-native.evidence.v1",
+  "paideia.daimon-native.grok-broker-projection.v1": "spawnfile.training.daimon-native.grok-broker-projection.v1",
+  "paideia.daimon-native.grok-slot-freshness.v1": "spawnfile.training.daimon-native.grok-slot-freshness.v1",
+  "paideia.daimon-native.launch.v1": "spawnfile.training.daimon-native.launch.v1",
+  "paideia.daimon-native.launch.v2": "spawnfile.training.daimon-native.launch.v2",
+  "paideia.daimon-native.operator-capture.v1": "spawnfile.training.daimon-native.operator-capture.v1",
+  "paideia.daimon-native.policy-preflight.v1": "spawnfile.training.daimon-native.policy-preflight.v1",
+  "paideia.daimon-native.ready.v1": "spawnfile.training.daimon-native.ready.v1",
+  "paideia.daimon-native.sandbox-preflight.v1": "spawnfile.training.daimon-native.sandbox-preflight.v1",
+  "paideia.daimon-native.sandbox-projection.v1": "spawnfile.training.daimon-native.sandbox-projection.v1",
+  "paideia.daimon-native.start-failed.v1": "spawnfile.training.daimon-native.start-failed.v1",
+  "paideia.daimon-native.startup-diagnostic.v1": "spawnfile.training.daimon-native.startup-diagnostic.v1",
+  "paideia.daimon-native.stop.v1": "spawnfile.training.daimon-native.stop.v1",
+  "paideia.daimon-native.transcript.v1": "spawnfile.training.daimon-native.transcript.v1",
+  "paideia.daimon-native.trial.v1": "spawnfile.training.daimon-native.trial.v1",
+  "paideia.dataset.v1": "spawnfile.training.dataset.v1",
+  "paideia.dataset.v2": "spawnfile.training.dataset.v2",
+  "paideia.decomposition.v1": "spawnfile.training.decomposition.v1",
+  "paideia.dspy.v2": "spawnfile.training.dspy.v2",
+  "paideia.evaluator-request.v1": "spawnfile.training.evaluator-request.v1",
+  "paideia.evaluator-routes.v1": "spawnfile.training.evaluator-routes.v1",
+  "paideia.evidence-units.v1": "spawnfile.training.evidence-units.v1",
+  "paideia.external-input.v1": "spawnfile.training.external-input.v1",
+  "paideia.extraction.v6": "spawnfile.training.extraction.v6",
+  "paideia.extraction.v7": "spawnfile.training.extraction.v7",
+  "paideia.extraction.v8": "spawnfile.training.extraction.v8",
+  "paideia.extraction.v9": "spawnfile.training.extraction.v9",
+  "paideia.failure.v1": "spawnfile.training.failure.v1",
+  "paideia.grader-adjudication.v1": "spawnfile.training.grader-adjudication.v1",
+  "paideia.grader-calibration.v1": "spawnfile.training.grader-calibration.v1",
+  "paideia.grader-failure.v1": "spawnfile.training.grader-failure.v1",
+  "paideia.grader-inputs.v1": "spawnfile.training.grader-inputs.v1",
+  "paideia.grader-label-prompt.v1": "spawnfile.training.grader-label-prompt.v1",
+  "paideia.grader-prompt.v4": "spawnfile.training.grader-prompt.v4",
+  "paideia.grader.v1": "spawnfile.training.grader.v1",
+  "paideia.grading.v1": "spawnfile.training.grading.v1",
+  "paideia.grok-attested-response.v3": "spawnfile.training.grok-attested-response.v3",
+  "paideia.grok-broker-grant-response.v1": "spawnfile.training.grok-broker-grant-response.v1",
+  "paideia.invocation.v1": "spawnfile.training.invocation.v1",
+  "paideia.judgment-cache.v1": "spawnfile.training.judgment-cache.v1",
+  "paideia.judgment-cache.v3": "spawnfile.training.judgment-cache.v3",
+  "paideia.label-import.v1": "spawnfile.training.label-import.v1",
+  "paideia.labels.v1": "spawnfile.training.labels.v1",
+  "paideia.measurement-repair.v1": "spawnfile.training.measurement-repair.v1",
+  "paideia.ollama-attested-response.v2": "spawnfile.training.ollama-attested-response.v2",
+  "paideia.python-bridge.v1": "spawnfile.training.python-bridge.v1",
+  "paideia.references.v1": "spawnfile.training.references.v1",
+  "paideia.repair-parent.v1": "spawnfile.training.repair-parent.v1",
+  "paideia.rescore.v1": "spawnfile.training.rescore.v1",
+  "paideia.run.v1": "spawnfile.training.run.v1",
+  "paideia.startup-diagnostics.v1": "spawnfile.training.startup-diagnostics.v1",
+  "paideia.subject.returned-turn-totals.v1": "spawnfile.training.subject.returned-turn-totals.v1",
+  "paideia.training-cost-plan.v1": "spawnfile.training.training-cost-plan.v1",
+  "paideia.training-export.v1": "spawnfile.training.training-export.v1",
+  "paideia.training-moment-prepared.v1": "spawnfile.training.training-moment-prepared.v1",
+  "paideia.training.v1": "spawnfile.training.training.v1",
+  "paideia.traps.v1": "spawnfile.training.traps.v1",
+  "paideia.trial.v1": "spawnfile.training.trial.v1",
+  "paideia.viewer.v1": "spawnfile.training.viewer.v1",
+  "clank.paideia.python-bridge.v1": "spawnfile.training.python-bridge.v1",
+  "paideia.population": "training.population"
+}

package/dist/training/dspy/spawnfile_training_dspy/main.py

@@ -0,0 +1,36 @@
+"""One optimization per process. Protocol stdout; upstream diagnostics stderr."""
+
+import contextlib
+import sys
+from typing import BinaryIO, TextIO
+
+from pydantic import ValidationError
+
+from .optimizer import optimize
+from .protocol import SCHEMA, BridgeError, OptimizeRequest
+from .transport import JsonlHost, read_message, write_message
+
+
+def run(reader: BinaryIO, writer: TextIO, diagnostic: TextIO) -> int:
+    request_id = "unknown"
+    try:
+        request = OptimizeRequest.model_validate(read_message(reader))
+        request_id = request.id
+        with contextlib.redirect_stdout(diagnostic):
+            result = optimize(request, JsonlHost(reader, writer))
+        write_message(writer, {"schema": SCHEMA, "kind": "result", "id": request_id, **result})
+        return 0
+    except Exception as error:
+        if isinstance(error, BridgeError):
+            code, message = error.code, str(error)
+        elif isinstance(error, ValidationError):
+            code, message = "invalid_request", "optimization request violates the protocol"
+        else:
+            code, message = "optimizer_error", f"DSPy optimization failed ({type(error).__name__})"
+        write_message(writer, {"schema": SCHEMA, "kind": "error", "id": request_id,
+                               "error": {"code": code, "message": message}})
+        return 1
+
+
+def cli():
+    raise SystemExit(run(sys.stdin.buffer, sys.stdout, sys.stderr))

package/dist/training/dspy/spawnfile_training_dspy/native.py

@@ -0,0 +1,89 @@
+"""Ordinary DSPy Predict traces backed by one native trial of all components."""
+
+import json
+
+import dspy
+
+from .protocol import BridgeError
+from .session import Session
+from .transport import parse_json
+
+
+def encode_component(path: str, text: str) -> str:
+    # DSPy Signature.instructions uses inspect.cleandoc. Single-line JSON avoids
+    # removing Markdown indentation or the final LF from native source bytes.
+    return json.dumps({"path": path, "text": text}, ensure_ascii=True, separators=(",", ":"))
+
+
+def decode_component(instructions: str) -> tuple[str, str]:
+    value = parse_json(instructions)
+    if (not isinstance(value, dict) or set(value) != {"path", "text"}
+            or not isinstance(value["path"], str) or not isinstance(value["text"], str)):
+        raise BridgeError("invalid_candidate", "component instruction envelope is malformed")
+    return value["path"], value["text"]
+
+
+class NativePlaceholderLM(dspy.BaseLM):
+    """Satisfies Predict's type contract; the native adapter never invokes an LM."""
+
+    forward_contract = "typed_lm"
+
+    def __init__(self):
+        super().__init__("training/native-host", cache=False, num_retries=0)
+
+    def forward(self, request):
+        raise BridgeError("unexpected_model_call", "native prediction must use the host trial adapter")
+
+
+class NativeTrialAdapter(dspy.Adapter):
+    def __init__(self, session: Session, case_id: str, components: dict[str, str]):
+        super().__init__()
+        self.session, self.case_id, self.components = session, case_id, components
+        self.result = None
+
+    def __call__(self, lm, lm_kwargs, signature, demos, inputs):
+        if demos or lm_kwargs or inputs != {"case_id": self.case_id}:
+            raise BridgeError("unsupported_predictor", "native predictor accepts only its case reference")
+        path, text = decode_component(signature.instructions)
+        if self.components.get(path) != text:
+            raise BridgeError("invalid_candidate", "predictor instruction does not match the native bundle")
+        if self.result is None:
+            self.result = self.session.evaluate(self.case_id, self.components)
+        # Each component's trace cites the same genuine whole-program evaluation.
+        # This is component exposure, not a claim of per-file causal attribution.
+        return [self.result.model_dump()]
+
+
+class NativeProgram(dspy.Module):
+    def __init__(self, session: Session):
+        super().__init__()
+        self._session = session
+        for index, (path, text) in enumerate(sorted(session.request.components.items())):
+            signature = dspy.Signature(
+                "case_id: str -> output: str, score: float, feedback: str, captureId: str",
+                instructions=encode_component(path, text),
+            )
+            setattr(self, f"component_{index}", dspy.Predict(signature))
+
+    def components(self):
+        pairs = [decode_component(predictor.signature.instructions)
+                 for _, predictor in self.named_predictors()]
+        if len(dict(pairs)) != len(pairs):
+            raise BridgeError("invalid_candidate", "candidate repeats a component resource")
+        return dict(pairs)
+
+    def forward(self, case_id: str):
+        try:
+            adapter = NativeTrialAdapter(self._session, case_id, self.components())
+            with dspy.context(adapter=adapter, lm=NativePlaceholderLM()):
+                for _, predictor in self.named_predictors():
+                    prediction = predictor(case_id=case_id)
+            return prediction
+        except Exception as error:
+            # DSPy's evaluator deliberately converts ordinary Exceptions to scores.
+            # A latched bridge error prevents that from becoming a successful run.
+            raise self._session.latch(error) from error
+
+
+def metric(gold, prediction, trace=None, pred_name=None, pred_trace=None):
+    return dspy.Prediction(score=prediction.score, feedback=prediction.feedback)

package/dist/training/dspy/spawnfile_training_dspy/optimizer.py

@@ -0,0 +1,74 @@
+"""Pinned DSPy GEPA invocation; no alternative optimization algorithm."""
+
+from importlib.metadata import version
+
+import dspy
+
+from .checkpoint import Checkpoint
+from .native import NativeProgram, metric
+from .protocol import BridgeError, OptimizeRequest
+from .reflection import ComponentProposer, SubscriptionLM
+from .session import Session
+
+DSPY_VERSION = "3.3.1"
+GEPA_VERSION = "0.1.4"
+
+
+def optimize(request: OptimizeRequest, host):
+    if version("dspy") != DSPY_VERSION or version("gepa") != GEPA_VERSION:
+        raise BridgeError("version_mismatch", "install the pinned DSPy and GEPA releases")
+    if request.search is not None:
+        from .search.population import optimize_population
+        return optimize_population(request, host)
+    session = Session(request, host)
+    checkpoint = Checkpoint(session) if request.checkpoint else None
+    student = NativeProgram(session)
+    reflection_lm = SubscriptionLM(session)
+    optimizer = dspy.GEPA(
+        metric=metric,
+        max_metric_calls=max(1, request.budget.maxMetricCalls - session.extra_evaluations),
+        reflection_minibatch_size=None if checkpoint else 1,
+        reflection_lm=reflection_lm,
+        instruction_proposer=ComponentProposer(session),
+        component_selector="all",
+        candidate_selection_strategy="current_best",
+        use_merge=False,
+        num_threads=request.concurrency,
+        track_stats=True,
+        log_dir=str(checkpoint.work) if checkpoint else None,
+        seed=request.seed,
+        gepa_kwargs={"stop_callbacks": [session.should_stop],
+                     **(checkpoint.kwargs() if checkpoint else {}),
+                     "callbacks": [session, *([checkpoint] if checkpoint else [])]},
+    )
+    # The only generation route is reflection_lm. Native Predict overrides its
+    # adapter and LM in forward; no global DSPy model or teacher is consulted.
+    with dspy.context(lm=reflection_lm, track_usage=False, callbacks=[]):
+        selected = optimizer.compile(
+            student,
+            trainset=[dspy.Example(case_id=case_id).with_inputs("case_id")
+                      for case_id in request.trainCaseIds],
+            valset=[dspy.Example(case_id=case_id).with_inputs("case_id")
+                    for case_id in request.validationCaseIds],
+        )
+    session.healthy()
+    if checkpoint:
+        # GEPA writes its final state without dispatching on_state_saved. Pair
+        # that state with final benign-stop/history bookkeeping before returning.
+        checkpoint.on_state_saved(None)
+    stats = selected.detailed_results
+    return {
+        "components": selected.components(),
+        "provenance": {
+            "optimizer": "dspy.GEPA", "dspyVersion": DSPY_VERSION, "gepaVersion": GEPA_VERSION,
+            "seed": request.seed, "evaluations": session.evaluations, "proposals": session.proposals,
+            "selectedCandidateIndex": stats.best_idx,
+            "candidates": [
+                {"components": candidate.components(), "parents": stats.parents[index],
+                 "validationScore": stats.val_aggregate_scores[index],
+                 "discoveryEvaluations": stats.discovery_eval_counts[index]}
+                for index, candidate in enumerate(stats.candidates)
+            ],
+            "captures": session.captures, "generations": session.generations,
+        },
+    }