benchmax 0.2.3__tar.gz → 0.2.4__tar.gz

{benchmax-0.2.3 → benchmax-0.2.4}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: benchmax
-Version: 0.2.3
+Version: 0.2.4
 Summary: Platform-independent runtime for grouped LLM environments
 Author: benchmax Authors
 Classifier: Operating System :: OS Independent
@@ -31,6 +31,12 @@ all benchmax environments implement the same dataset and rollout contracts. choo
 
 most custom environments should extend `BaseEnv`. most Harbor users configure `HarborEnv` directly rather than subclassing it.
 
+## sft datasets
+
+supervised finetuning does not use an environment. construct a validated
+[`benchmax.sft.SftDataset`](src/benchmax/sft/README.md) from chat-format rows and upload it
+through the castform package; validation is strict and all-or-nothing.
+
 ## architecture
 
 an environment defines its dataset and how a group of rollouts runs against each example.

{benchmax-0.2.3 → benchmax-0.2.4}/README.md

@@ -16,6 +16,12 @@ all benchmax environments implement the same dataset and rollout contracts. choo
 
 most custom environments should extend `BaseEnv`. most Harbor users configure `HarborEnv` directly rather than subclassing it.
 
+## sft datasets
+
+supervised finetuning does not use an environment. construct a validated
+[`benchmax.sft.SftDataset`](src/benchmax/sft/README.md) from chat-format rows and upload it
+through the castform package; validation is strict and all-or-nothing.
+
 ## architecture
 
 an environment defines its dataset and how a group of rollouts runs against each example.

{benchmax-0.2.3 → benchmax-0.2.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "benchmax"
-version = "0.2.3"
+version = "0.2.4"
 description = "Platform-independent runtime for grouped LLM environments"
 readme = "README.md"
 authors = [{ name = "benchmax Authors" }]

benchmax-0.2.4/src/benchmax/sft/README.md

@@ -0,0 +1,59 @@
+# benchmax.sft
+
+the validated `benchmax-sft-v1` supervised-finetuning dataset artifact. construction is
+all-or-nothing: a dataset either satisfies the whole contract or raises with every ordered,
+line-aware issue.
+
+```python
+from benchmax.sft import SftDataset, SftDatasetError
+
+train = SftDataset.from_jsonl("train.jsonl")   # or SftDataset.from_rows(rows)
+payload = train.to_jsonl_bytes()               # canonical, deterministic bytes
+```
+
+## row contract
+
+one JSONL line per row. a row is a closed object: required `messages`, optional `tools`,
+optional `metadata`.
+
+```json
+{
+  "messages": [
+    {"role": "user", "content": "What is 2 + 2?"},
+    {"role": "assistant", "content": "4", "weight": 1}
+  ],
+  "tools": [],
+  "metadata": {"id": "optional producer identity"}
+}
+```
+
+- `system`/`user`: exactly `role` + non-empty string `content`.
+- `assistant`: optional string-or-null `content`, optional non-empty `tool_calls`,
+  optional integer `weight` `0 | 1` (omitted means `1`; `0` masks the turn from loss).
+  each assistant turn needs non-empty content or a tool call; each row needs at least one
+  assistant turn with effective weight `1`.
+- `tool`: exactly `role` + string `content` + non-empty `tool_call_id`. every tool call
+  gets exactly one result, in declaration order, before the next non-tool message.
+- tool definitions/calls are OpenAI-style function shapes, validated deeply; called
+  names must match a definition; `function.arguments` must decode as a JSON object.
+- `metadata` is an open finite JSON object (canonical size ≤ 64 KiB; `_castform_*` keys
+  are reserved for the runtime). rows are ≤ 1 MiB canonical, `messages` ≤ 1024 entries,
+  `tools` ≤ 128 unique functions.
+
+rejected outright: multimodal content parts, fractional weights, legacy
+prompt/completion shapes, duplicate JSON keys, lone surrogates, non-finite numbers,
+nesting deeper than 64 levels, and unknown keys anywhere outside `metadata` and
+`parameters`.
+
+## canonical bytes
+
+`to_jsonl_bytes()` is deterministic for equivalent inputs: UTF-8 without a BOM,
+`ensure_ascii=False`, sorted keys, compact separators, one trailing newline per row,
+original row order.
+
+## golden fixtures
+
+`packages/benchmax/tests/fixtures/sft_v1/` is the cross-repository contract consumed by
+the castform trainer: valid inputs with byte-exact canonical outputs, and invalid inputs
+with exact ordered diagnostics in `expected_issues.json`. change them only with a format
+version bump in mind.

benchmax-0.2.4/src/benchmax/sft/__init__.py

@@ -0,0 +1,36 @@
+"""Public `benchmax-sft-v1` supervised-finetuning dataset contract.
+
+One row is a closed JSON object with required ``messages``, optional ``tools``,
+and optional ``metadata``:
+
+- ``system``/``user`` messages have exactly ``role`` and non-empty string
+  ``content``.
+- ``assistant`` messages have optional string-or-null ``content``, optional
+  non-empty ``tool_calls``, and optional integer ``weight`` of ``0`` or ``1``
+  (omitted means ``1``); each needs non-empty content or at least one tool
+  call, and every row needs at least one assistant turn with effective
+  weight ``1``.
+- ``tool`` results have exactly ``role``, string ``content``, and non-empty
+  ``tool_call_id``; every tool call receives exactly one result, in
+  declaration order, before the next non-tool message.
+- OpenAI-style function tool definitions/calls are validated deeply; called
+  function names must match a definition.
+- ``metadata`` is an open finite JSON object (``_castform_``-prefixed keys are
+  reserved for the runtime).
+
+Validation is strict and all-or-nothing: images/audio/multimodal content
+parts, fractional weights, legacy prompt/completion shapes, duplicate JSON
+keys, lone surrogates, non-finite numbers, and nesting deeper than 64 levels
+are rejected with ordered, line-aware diagnostics. Canonical serialization is
+deterministic byte-for-byte for equivalent inputs.
+"""
+
+from benchmax.sft.dataset import SFT_DATASET_FORMAT, SftDataset
+from benchmax.sft.issues import SftDatasetError, SftIssue
+
+__all__ = [
+    "SFT_DATASET_FORMAT",
+    "SftDataset",
+    "SftDatasetError",
+    "SftIssue",
+]

benchmax-0.2.4/src/benchmax/sft/dataset.py

@@ -0,0 +1,187 @@
+"""The validated `benchmax-sft-v1` dataset artifact.
+
+:class:`SftDataset` is the single trust boundary for SFT training data: it is
+either fully valid or it does not exist. Both constructors run the complete
+strict contract — closed row schema, strict JSON rules, size limits — and raise
+:class:`~benchmax.sft.issues.SftDatasetError` with every ordered issue on any
+violation, including empty input.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+from collections.abc import Iterable, Iterator
+from pathlib import Path
+
+from benchmax.sft.issues import SftDatasetError, SftIssue
+from benchmax.sft.rows import MAX_ROW_BYTES, validate_row
+from benchmax.sft.strict_json import (
+    StrictJsonError,
+    canonical_json_bytes,
+    check_json_tree,
+    parse_strict_json,
+)
+
+__all__ = ["SFT_DATASET_FORMAT", "SftDataset"]
+
+SFT_DATASET_FORMAT = "benchmax-sft-v1"
+"""Format identifier used at the platform boundary for this row contract."""
+
+_INTERNAL = object()
+
+
+class SftDataset:
+    """Immutable, fully validated SFT training dataset.
+
+    Construct only through :meth:`from_jsonl` or :meth:`from_rows`; direct
+    instantiation is rejected so a partially valid dataset can never exist.
+    Row order is preserved from the input. :meth:`to_jsonl_bytes` returns
+    canonical bytes — UTF-8 without a BOM, ``ensure_ascii=False``, sorted keys,
+    compact separators, one trailing newline per row — deterministic for
+    equivalent input objects.
+    """
+
+    __slots__ = ("_row_bytes", "_rows")
+
+    def __init__(
+        self,
+        rows: tuple[dict[str, object], ...],
+        row_bytes: tuple[bytes, ...],
+        token: object = None,
+    ) -> None:
+        if token is not _INTERNAL:
+            raise TypeError("use SftDataset.from_jsonl(...) or SftDataset.from_rows(...)")
+        self._rows = rows
+        self._row_bytes = row_bytes
+
+    @classmethod
+    def from_jsonl(cls, path: str | os.PathLike[str]) -> SftDataset:
+        """Load and validate a JSONL file.
+
+        Whitespace-only physical lines are ignored while physical line numbers
+        are preserved in diagnostics. Raises
+        :class:`~benchmax.sft.issues.SftDatasetError` on any violation.
+        """
+
+        data = Path(path).read_bytes()
+        collector = _IssueCollector()
+        rows: list[dict[str, object]] = []
+        row_bytes: list[bytes] = []
+        row_index = 0
+        for line_number, raw_line in enumerate(data.split(b"\n"), start=1):
+            if raw_line.strip(b" \t\r\f\v") == b"":
+                continue
+            row = row_index
+            row_index += 1
+            if line_number == 1 and raw_line.startswith(b"\xef\xbb\xbf"):
+                collector.add(line_number, row, "$", "a UTF-8 BOM is not allowed")
+                continue
+            try:
+                text = raw_line.decode("utf-8")
+            except UnicodeDecodeError:
+                collector.add(line_number, row, "$", "line is not valid UTF-8")
+                continue
+            try:
+                parsed = parse_strict_json(text)
+            except StrictJsonError as error:
+                collector.add(line_number, row, "$", str(error))
+                continue
+            encoded = _process_row(parsed, collector.for_row(line_number, row))
+            if encoded is not None:
+                rows.append(json.loads(encoded.decode("utf-8")))
+                row_bytes.append(encoded)
+        if row_index == 0:
+            collector.add(None, None, "$", "dataset contains no rows")
+        collector.raise_if_any()
+        return cls(tuple(rows), tuple(row_bytes), _INTERNAL)
+
+    @classmethod
+    def from_rows(cls, rows: Iterable[dict[str, object]]) -> SftDataset:
+        """Validate in-memory row objects.
+
+        Rows must already use plain JSON-compatible Python types; anything else
+        is reported as an issue rather than coerced. Raises
+        :class:`~benchmax.sft.issues.SftDatasetError` on any violation.
+        """
+
+        collector = _IssueCollector()
+        stored: list[dict[str, object]] = []
+        row_bytes: list[bytes] = []
+        row_index = -1
+        for row_index, row in enumerate(iter(rows)):
+            encoded = _process_row(row, collector.for_row(None, row_index))
+            if encoded is not None:
+                # Round-trip through canonical bytes to decouple stored rows
+                # from caller-owned objects.
+                stored.append(json.loads(encoded.decode("utf-8")))
+                row_bytes.append(encoded)
+        if row_index < 0:
+            collector.add(None, None, "$", "dataset contains no rows")
+        collector.raise_if_any()
+        return cls(tuple(stored), tuple(row_bytes), _INTERNAL)
+
+    def to_jsonl_bytes(self) -> bytes:
+        """Serialize every row as canonical JSONL bytes in original order."""
+
+        return b"".join(encoded + b"\n" for encoded in self._row_bytes)
+
+    @property
+    def rows(self) -> tuple[dict[str, object], ...]:
+        """The validated rows in original order. Treat as read-only."""
+
+        return self._rows
+
+    def __len__(self) -> int:
+        return len(self._rows)
+
+    def __iter__(self) -> Iterator[dict[str, object]]:
+        return iter(self._rows)
+
+
+class _IssueCollector:
+    """Ordered issue sink shared by both constructors."""
+
+    def __init__(self) -> None:
+        self._issues: list[SftIssue] = []
+
+    def add(self, line: int | None, row: int | None, location: str, message: str) -> None:
+        self._issues.append(SftIssue(message=message, location=location, line=line, row=row))
+
+    def for_row(self, line: int | None, row: int | None) -> _RowSink:
+        return _RowSink(self, line, row)
+
+    def raise_if_any(self) -> None:
+        if self._issues:
+            raise SftDatasetError(self._issues)
+
+
+class _RowSink:
+    """Emit adapter binding one row's line/row coordinates."""
+
+    def __init__(self, collector: _IssueCollector, line: int | None, row: int | None) -> None:
+        self._collector = collector
+        self.line = line
+        self.row = row
+        self.count = 0
+
+    def __call__(self, location: str, message: str) -> None:
+        self.count += 1
+        self._collector.add(self.line, self.row, location, message)
+
+
+def _process_row(row: object, sink: _RowSink) -> bytes | None:
+    """Fully validate one row; return its canonical bytes only when clean."""
+
+    check_json_tree(row, "$", sink)
+    if type(row) is not dict:
+        sink("$", "row must be a JSON object")
+        return None
+    validate_row(row, sink)
+    if sink.count:
+        return None
+    encoded = canonical_json_bytes(row)
+    if len(encoded) > MAX_ROW_BYTES:
+        sink("$", f"row canonical size is {len(encoded)} bytes; the maximum is {MAX_ROW_BYTES}")
+        return None
+    return encoded

benchmax-0.2.4/src/benchmax/sft/issues.py

@@ -0,0 +1,73 @@
+"""Diagnostics for `benchmax-sft-v1` dataset validation.
+
+Validation is all-or-nothing: constructing an :class:`~benchmax.sft.SftDataset`
+either succeeds completely or raises :class:`SftDatasetError` carrying every
+:class:`SftIssue` found, in deterministic document order.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from dataclasses import dataclass
+
+__all__ = ["SftDatasetError", "SftIssue"]
+
+_PREVIEW_ISSUES = 20
+
+
+@dataclass(frozen=True, slots=True)
+class SftIssue:
+    """One validation problem at one location in the input.
+
+    ``location`` addresses the offending value inside the row using ``$`` for
+    the row object itself, e.g. ``$.messages[2].weight``. ``line`` is the
+    1-based physical JSONL line for :meth:`SftDataset.from_jsonl` input and
+    ``None`` for :meth:`SftDataset.from_rows` input. ``row`` is the 0-based
+    dataset row index, or ``None`` for dataset-level problems such as an empty
+    input.
+    """
+
+    message: str
+    location: str
+    line: int | None = None
+    row: int | None = None
+
+    def as_dict(self) -> dict[str, object]:
+        """Return the stable JSON-friendly shape used by golden fixtures."""
+
+        return {
+            "line": self.line,
+            "location": self.location,
+            "message": self.message,
+            "row": self.row,
+        }
+
+    def describe(self) -> str:
+        """Render a one-line human-readable description."""
+
+        origin: list[str] = []
+        if self.line is not None:
+            origin.append(f"line {self.line}")
+        if self.row is not None:
+            origin.append(f"row {self.row}")
+        prefix = f"{' '.join(origin)} " if origin else ""
+        return f"{prefix}{self.location}: {self.message}"
+
+
+class SftDatasetError(ValueError):
+    """Raised when input cannot form a valid `benchmax-sft-v1` dataset.
+
+    ``issues`` holds every problem found, ordered by input position and then by
+    a fixed within-row validation order, so equivalent inputs always produce an
+    identical issue sequence.
+    """
+
+    def __init__(self, issues: Sequence[SftIssue]) -> None:
+        self.issues: tuple[SftIssue, ...] = tuple(issues)
+        if not self.issues:
+            raise ValueError("SftDatasetError requires at least one issue")
+        lines = [f"invalid benchmax-sft-v1 dataset: {len(self.issues)} issue(s)"]
+        lines += [f"  {issue.describe()}" for issue in self.issues[:_PREVIEW_ISSUES]]
+        if len(self.issues) > _PREVIEW_ISSUES:
+            lines.append(f"  ... and {len(self.issues) - _PREVIEW_ISSUES} more")
+        super().__init__("\n".join(lines))