trainlib 0.3.1__tar.gz → 0.6.1__tar.gz

trainlib-0.6.1/PKG-INFO

@@ -0,0 +1,243 @@
+Metadata-Version: 2.4
+Name: trainlib
+Version: 0.6.1
+Summary: Minimal framework for ML modeling. Supports advanced dataset operations and streamlined training.
+License-Expression: MIT
+Project-URL: Homepage, https://doc.olog.io/trainlib
+Project-URL: Documentation, https://doc.olog.io/trainlib
+Project-URL: Repository, https://git.olog.io/olog/trainlib
+Project-URL: Issues, https://git.olog.io/olog/trainlib/issues
+Keywords: machine-learning
+Classifier: Programming Language :: Python
+Classifier: Operating System :: OS Independent
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: End Users/Desktop
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: torch
+Requires-Dist: pandas
+Requires-Dist: colorama>=0.4.6
+Requires-Dist: matplotlib>=3.10.8
+Requires-Dist: numpy>=2.4.1
+Requires-Dist: tensorboard>=2.20.0
+Requires-Dist: tqdm>=4.67.1
+Requires-Dist: rich>=13.7
+Requires-Dist: setuptools<=81.0.0
+Provides-Extra: dev
+Requires-Dist: ipykernel; extra == "dev"
+Provides-Extra: doc
+Requires-Dist: furo; extra == "doc"
+Requires-Dist: myst-parser; extra == "doc"
+Requires-Dist: sphinx; extra == "doc"
+Requires-Dist: sphinx-togglebutton; extra == "doc"
+Requires-Dist: sphinx-autodoc-typehints; extra == "doc"
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+
+# Overview
+Minimal framework for ML modeling, supporting advanced dataset operations and
+streamlined training workflows.
+    
+# Install
+The `trainlib` package can be installed from PyPI:
+
+```sh
+pip install trainlib
+```
+
+# Development
+- Initialize/synchronize the project with `uv sync`, creating a virtual
+  environment with base package dependencies.
+- Depending on needs, install the development dependencies with `uv sync
+  --extra dev`.
+
+## Testing
+- To run the unit tests, make sure to first have the test dependencies
+  installed with `uv sync --extra test`, then run `make test`.
+- For notebook testing, run `make install-kernel` to make the environment
+  available as a Jupyter kernel (to be selected when running notebooks).
+
+## Documentation
+- Install the documentation dependencies with `uv sync --extra doc`.
+- Run `make docs-build` (optionally preceded by `make docs-clean`), and serve
+  locally with `make docs-serve`.
+
+# Development remarks
+- Across `Trainer` / `Estimator` / `Dataset`, I've considered a
+  `ParamSpec`-based typing scheme to better orchestrate alignment in the
+  `Trainer.train()` loop, e.g., so we can statically check whether a dataset
+  appears to be fulfilling the argument requirements for the estimator's
+  `forward()` / `metrics()`  methods. Something like
+
+  ```py
+  class Estimator[**P](nn.Module):
+      def forward(
+          self,
+          input: Tensor,
+          *args: P.args,
+          **kwargs: P.kwargs,
+      ) -> Tensor:
+          ...
+
+  class Trainer[**P]:
+      def __init__(
+          self,
+          estimator: Estimator[P],
+          ...
+      ): ...
+  ```
+
+  might be how we begin threading signatures. But ensuring dataset items can
+  match `P` is challenging. You can consider a "packed" object where we
+  obfuscate passing data through `P`-signatures:
+
+  ```py
+  class PackedItem[**P]:
+      def __init__(self, *args: P.args, **kwargs: P.kwargs) -> None:
+          self._args = args
+          self._kwargs = kwargs
+  
+      def apply[R](self, func: Callable[P, R]) -> R:
+          return func(*self._args, **self._kwargs)
+  
+  
+  class BatchedDataset[U, R, I, **P](Dataset):
+      @abstractmethod
+      def _process_item_data(
+          self,
+          item_data: I,
+          item_index: int,
+      ) -> PackedItem[P]:
+          ...
+  
+      def __iter__(self) -> Iterator[PackedItem[P]]:
+          ...
+  ```
+
+  Meaningfully shaping those signatures is what remains, but you can't really
+  do this, not with typical type expression flexibility. For instance, if I'm
+  trying to appropriately type my base `TupleDataset`:
+
+  ```py
+  class SequenceDataset[I, **P](HomogenousDataset[int, I, I, P]):
+      ...
+  
+  class TupleDataset[I](SequenceDataset[tuple[I, ...], "?"]):
+      ...
+  ```
+
+  Here there's no way for me to shape a `ParamSpec` to indicate arbitrarily
+  many arguments of a fixed type (`I` in this case) to allow me to unpack my
+  item tuples into an appropriate `PackedItem` (since `ParamSpec` isn't exactly
+  controllable in that way; it's meant make a generic signature expressible as
+  a type variable, *not to express particular shapes* of the types in that
+  signature).
+
+  Until this (among other issues) becomes clearer, I'm setting up around a
+  simpler `TypedDict` type variable. We won't have particularly strong static
+  checks for item alignment inside `Trainer`, but this seems about as good as I
+  can get around the current infrastructure. 
+
+## Typing planning
+There are currently two package-wide typing strategies geared toward future
+Python version bumps and type system changes:
+
+- Using `Unpack` with a type variable with a `TypedDict` upper bound, rather
+  than with a `TypedDict` directly. This is commented on extensively in
+  `trainlib/estimator.py`, but the short version is we want our key objects to
+  remain generic in their method signatures when they are expected to operate
+  on session-specific structured batch data. For instance, you can imagine an
+  `Estimator`'s `forward` method accepting a typed batch like
+
+  ```py
+  def forward(self, batch: BatchKwargs): ...
+
+  # or
+
+  def forward(self, **kwargs: Unpack[Batchwargs]): ...
+  ```
+
+  These are both currently supported patterns. Problems arise, however, when we
+  want an inherited class to specifically operate on an inherited type of
+  `BatchKwargs`
+
+  ```py
+  class BatchKwargs(TypedDict):
+      inputs: Tensor
+
+  class ExtraBatchKwargs(BatchKwargs):
+      inputs: Tensor
+      labels: Tensor
+
+  class A(Estimator):
+      def forward(self, batch: BatchKwargs): ...
+
+  # want to require Estimator subclass handles both inputs and labels
+  class B(A):
+      def forward(self, batch: ExtraBatchKwargs): ...
+  ```
+
+  Note how `ExtraBatchKwargs` doesn't use `NotRequired` for the new `labels`
+  attribute; the intention here is to actually require that `B` should handle
+  *more attributes* than `A`. **That breaks LSP**, so this doesn't work. You'd
+  need to either break out the `Estimator` hierarchies, or decide that `B`
+  could have inputs typed as `BatchKwargs` and possibly implement some extra
+  handling to check if batches of type `ExtraBatchKwargs` are passed in (note
+  that passing in a batch of type `ExtraBatchKwargs` to `A.forward` is
+  perfectly fine) and supply labels. **Or...** you make the class generic in
+  the batch type, and during inheritance you can establish increasingly tight
+  upper bounds on the batch type to accomplish precisely the desired effect.
+  That is, you can formally require subclasses handle new batch attributes
+  without breaking LSP. This could look like:
+
+  ```py
+  class A[Kw: BatchKwargs](Estimator):
+      def forward(self, batch: Kw): ...
+
+  class B[Kw: ExtraBatchKwargs](A[Kw]):
+      def forward(self, batch: Kw): ...
+  ```
+
+  This formally allows `B` to inherit from `A` while depending on new,
+  non-optional attributes in its batch type; you can achieve that with refined
+  type upper bound in a generic subtype. So this pattern works *now*, but what
+  doesn't work is further *unpacking* `Kw` in this pattern into kwargs. That
+  is,
+
+  ```py
+  class A[Kw: BatchKwargs](Estimator):
+      def forward(self, **kwargs: Unpack[Kw]): ...
+
+  class B[Kw: ExtraBatchKwargs](A[Kw]):
+      def forward(self, **kwargs: Unpack[Kw]): ...
+  ```
+
+  As sound as this is (and it *is* sound, described in PEP 827), `Unpack` isn't
+  supported for type variables; it has to be used directly with `TypedDict`
+  references. I have the latter in place across the package anticipating
+  eventual support for this pattern, but things are subject to change. I may
+  simply toggle back to the currently supported pattern provided it's very
+  similar, and if/when PEP 827 is integrated, the change will be as simply as
+  adding back the `Unpack` statements and keyword expansions.
+- At the time of writing, `trainlib` attempts to remain compatible with Python
+  3.12. This means the convenient lazy evaluation of type annotations can't be
+  fully embraced across the package, but we use `__future__` where possible to
+  make the transition easy when eventually bumping to 3.14+ (since the imports
+  can just be removed at that time). For instance, class definitions that need
+  to reference the class type itself shouldn't use string annotations but
+  instead direct type references:
+
+  ```py
+  # not
+  class RecordIndex:
+      @overload
+      def __getitem__(self, key: slice) -> "RecordIndex": ...
+
+  # but
+  from __future__ import annotations
+
+  class RecordIndex:
+      @overload
+      def __getitem__(self, key: slice) -> RecordIndex: ...
+  ```

trainlib-0.6.1/README.md

@@ -0,0 +1,206 @@
+# Overview
+Minimal framework for ML modeling, supporting advanced dataset operations and
+streamlined training workflows.
+    
+# Install
+The `trainlib` package can be installed from PyPI:
+
+```sh
+pip install trainlib
+```
+
+# Development
+- Initialize/synchronize the project with `uv sync`, creating a virtual
+  environment with base package dependencies.
+- Depending on needs, install the development dependencies with `uv sync
+  --extra dev`.
+
+## Testing
+- To run the unit tests, make sure to first have the test dependencies
+  installed with `uv sync --extra test`, then run `make test`.
+- For notebook testing, run `make install-kernel` to make the environment
+  available as a Jupyter kernel (to be selected when running notebooks).
+
+## Documentation
+- Install the documentation dependencies with `uv sync --extra doc`.
+- Run `make docs-build` (optionally preceded by `make docs-clean`), and serve
+  locally with `make docs-serve`.
+
+# Development remarks
+- Across `Trainer` / `Estimator` / `Dataset`, I've considered a
+  `ParamSpec`-based typing scheme to better orchestrate alignment in the
+  `Trainer.train()` loop, e.g., so we can statically check whether a dataset
+  appears to be fulfilling the argument requirements for the estimator's
+  `forward()` / `metrics()`  methods. Something like
+
+  ```py
+  class Estimator[**P](nn.Module):
+      def forward(
+          self,
+          input: Tensor,
+          *args: P.args,
+          **kwargs: P.kwargs,
+      ) -> Tensor:
+          ...
+
+  class Trainer[**P]:
+      def __init__(
+          self,
+          estimator: Estimator[P],
+          ...
+      ): ...
+  ```
+
+  might be how we begin threading signatures. But ensuring dataset items can
+  match `P` is challenging. You can consider a "packed" object where we
+  obfuscate passing data through `P`-signatures:
+
+  ```py
+  class PackedItem[**P]:
+      def __init__(self, *args: P.args, **kwargs: P.kwargs) -> None:
+          self._args = args
+          self._kwargs = kwargs
+  
+      def apply[R](self, func: Callable[P, R]) -> R:
+          return func(*self._args, **self._kwargs)
+  
+  
+  class BatchedDataset[U, R, I, **P](Dataset):
+      @abstractmethod
+      def _process_item_data(
+          self,
+          item_data: I,
+          item_index: int,
+      ) -> PackedItem[P]:
+          ...
+  
+      def __iter__(self) -> Iterator[PackedItem[P]]:
+          ...
+  ```
+
+  Meaningfully shaping those signatures is what remains, but you can't really
+  do this, not with typical type expression flexibility. For instance, if I'm
+  trying to appropriately type my base `TupleDataset`:
+
+  ```py
+  class SequenceDataset[I, **P](HomogenousDataset[int, I, I, P]):
+      ...
+  
+  class TupleDataset[I](SequenceDataset[tuple[I, ...], "?"]):
+      ...
+  ```
+
+  Here there's no way for me to shape a `ParamSpec` to indicate arbitrarily
+  many arguments of a fixed type (`I` in this case) to allow me to unpack my
+  item tuples into an appropriate `PackedItem` (since `ParamSpec` isn't exactly
+  controllable in that way; it's meant make a generic signature expressible as
+  a type variable, *not to express particular shapes* of the types in that
+  signature).
+
+  Until this (among other issues) becomes clearer, I'm setting up around a
+  simpler `TypedDict` type variable. We won't have particularly strong static
+  checks for item alignment inside `Trainer`, but this seems about as good as I
+  can get around the current infrastructure. 
+
+## Typing planning
+There are currently two package-wide typing strategies geared toward future
+Python version bumps and type system changes:
+
+- Using `Unpack` with a type variable with a `TypedDict` upper bound, rather
+  than with a `TypedDict` directly. This is commented on extensively in
+  `trainlib/estimator.py`, but the short version is we want our key objects to
+  remain generic in their method signatures when they are expected to operate
+  on session-specific structured batch data. For instance, you can imagine an
+  `Estimator`'s `forward` method accepting a typed batch like
+
+  ```py
+  def forward(self, batch: BatchKwargs): ...
+
+  # or
+
+  def forward(self, **kwargs: Unpack[Batchwargs]): ...
+  ```
+
+  These are both currently supported patterns. Problems arise, however, when we
+  want an inherited class to specifically operate on an inherited type of
+  `BatchKwargs`
+
+  ```py
+  class BatchKwargs(TypedDict):
+      inputs: Tensor
+
+  class ExtraBatchKwargs(BatchKwargs):
+      inputs: Tensor
+      labels: Tensor
+
+  class A(Estimator):
+      def forward(self, batch: BatchKwargs): ...
+
+  # want to require Estimator subclass handles both inputs and labels
+  class B(A):
+      def forward(self, batch: ExtraBatchKwargs): ...
+  ```
+
+  Note how `ExtraBatchKwargs` doesn't use `NotRequired` for the new `labels`
+  attribute; the intention here is to actually require that `B` should handle
+  *more attributes* than `A`. **That breaks LSP**, so this doesn't work. You'd
+  need to either break out the `Estimator` hierarchies, or decide that `B`
+  could have inputs typed as `BatchKwargs` and possibly implement some extra
+  handling to check if batches of type `ExtraBatchKwargs` are passed in (note
+  that passing in a batch of type `ExtraBatchKwargs` to `A.forward` is
+  perfectly fine) and supply labels. **Or...** you make the class generic in
+  the batch type, and during inheritance you can establish increasingly tight
+  upper bounds on the batch type to accomplish precisely the desired effect.
+  That is, you can formally require subclasses handle new batch attributes
+  without breaking LSP. This could look like:
+
+  ```py
+  class A[Kw: BatchKwargs](Estimator):
+      def forward(self, batch: Kw): ...
+
+  class B[Kw: ExtraBatchKwargs](A[Kw]):
+      def forward(self, batch: Kw): ...
+  ```
+
+  This formally allows `B` to inherit from `A` while depending on new,
+  non-optional attributes in its batch type; you can achieve that with refined
+  type upper bound in a generic subtype. So this pattern works *now*, but what
+  doesn't work is further *unpacking* `Kw` in this pattern into kwargs. That
+  is,
+
+  ```py
+  class A[Kw: BatchKwargs](Estimator):
+      def forward(self, **kwargs: Unpack[Kw]): ...
+
+  class B[Kw: ExtraBatchKwargs](A[Kw]):
+      def forward(self, **kwargs: Unpack[Kw]): ...
+  ```
+
+  As sound as this is (and it *is* sound, described in PEP 827), `Unpack` isn't
+  supported for type variables; it has to be used directly with `TypedDict`
+  references. I have the latter in place across the package anticipating
+  eventual support for this pattern, but things are subject to change. I may
+  simply toggle back to the currently supported pattern provided it's very
+  similar, and if/when PEP 827 is integrated, the change will be as simply as
+  adding back the `Unpack` statements and keyword expansions.
+- At the time of writing, `trainlib` attempts to remain compatible with Python
+  3.12. This means the convenient lazy evaluation of type annotations can't be
+  fully embraced across the package, but we use `__future__` where possible to
+  make the transition easy when eventually bumping to 3.14+ (since the imports
+  can just be removed at that time). For instance, class definitions that need
+  to reference the class type itself shouldn't use string annotations but
+  instead direct type references:
+
+  ```py
+  # not
+  class RecordIndex:
+      @overload
+      def __getitem__(self, key: slice) -> "RecordIndex": ...
+
+  # but
+  from __future__ import annotations
+
+  class RecordIndex:
+      @overload
+      def __getitem__(self, key: slice) -> RecordIndex: ...
+  ```

{trainlib-0.3.1 → trainlib-0.6.1}/pyproject.toml

@@ -4,12 +4,10 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "trainlib"
-version = "0.3.1"
+version = "0.6.1"
 description = "Minimal framework for ML modeling. Supports advanced dataset operations and streamlined training."
-requires-python = ">=3.13"
-authors = [
-  { name="Sam Griesemer", email="git@olog.io" },
-]
+requires-python = ">=3.12"
+authors = []
 readme = "README.md"
 license = "MIT"
 keywords = [
@@ -25,15 +23,18 @@ classifiers = [
 ]
 dependencies = [
     "torch",
+    "pandas",
     "colorama>=0.4.6",
     "matplotlib>=3.10.8",
     "numpy>=2.4.1",
     "tensorboard>=2.20.0",
     "tqdm>=4.67.1",
+    "rich>=13.7",
     "setuptools<=81.0.0",  # here currently b/c tensorboard breaks @ v82.0.0
 ]
 
 [project.scripts]
+trainlib = "trainlib.__main__:main"
 symconf = "trainlib.__main__:main"
 
 [project.optional-dependencies]
@@ -61,6 +62,10 @@ Issues = "https://git.olog.io/olog/trainlib/issues"
 [tool.setuptools.packages.find]
 include = ["trainlib*"]
 
+[tool.pytest.ini_options]
+testpaths = ["test"]
+addopts = "--import-mode=importlib"
+
 # for static data files under package root
 # [tool.setuptools.package-data]
 # "<package>" = ["data/*.toml"]
@@ -68,8 +73,13 @@ include = ["trainlib*"]
 [tool.ruff]
 line-length = 79
 
+[tool.ruff.format]
+quote-style = "double"
+indent-style = "space"
+docstring-code-format = true
+
 [tool.ruff.lint]
-select = ["ANN", "E", "F", "UP", "B", "SIM", "I", "C4", "PERF"]
+select = ["ANN", "D", "E", "F", "UP", "B", "SIM", "I", "C4", "PERF"]
 
 [tool.ruff.lint.isort]
 length-sort = true
@@ -77,13 +87,14 @@ order-by-type = false
 force-sort-within-sections = false
 
 [tool.ruff.lint.per-file-ignores]
-"tests/**" = ["S101"]
+"test/**" = ["S101"]
 "**/__init__.py" = ["F401"]
 
-[tool.ruff.format]
-quote-style = "double"
-indent-style = "space"
-docstring-code-format = true
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+[tool.setuptools.package-data]
+"trainlib.utils" = ["custom.mplstyle"]
 
 [tool.uv.sources]
 torch = { index = "pytorch" }

trainlib-0.6.1/test/test_cli.py

@@ -0,0 +1,160 @@
+import json
+from copy import deepcopy
+from pathlib import Path
+
+import pytest
+
+from trainlib.cli import train, experiment, create_parser
+from trainlib.run import expand_runs
+from trainlib.trainer import Trainer
+from trainlib.cli.experiment import ExperimentConfig
+from trainlib.experiments.train import TrainExperiment
+
+EXAMPLES = Path(__file__).resolve().parents[1] / "example" / "cli"
+
+
+def test_cli_dispatches_non_training_experiment(tmp_path, monkeypatch):
+    from trainlib.run import Run
+    from trainlib.record import RecordIndex
+    from trainlib.experiment import Experiment
+    from trainlib.experiments import experiment_map
+
+    class Evaluation(Experiment[Run]):
+        def _execute(self, spec, directory, *, progress, console, output):
+            self.runstore.save(
+                directory, RecordIndex(), {"test": {"score": 1.0}}
+            )
+
+    monkeypatch.setitem(experiment_map, "test_evaluation", Evaluation)
+    result = experiment.run(
+        {
+            "experiment": "test_evaluation",
+            "output": str(tmp_path / "eval"),
+            "groups": {"raw": {}},
+            "seeds": [7],
+            "summary": {"plots": False, "resamples": 10},
+        },
+        quiet=True,
+    )
+    assert len(result["completed"]) == 1
+    assert "summary" in result
+
+
+def test_single_run_cli_loads_and_trains_without_experiment(
+    tmp_path: Path,
+) -> None:
+    config = json.loads((EXAMPLES / "run.json").read_text())
+    config["data"]["dataset"]["kwargs"]["num_samples"] = 12
+    config["estimator"]["kwargs"]["verbose"] = False
+    config["reporter"]["kwargs"].update(
+        progress=False,
+        console=False,
+        tensorboard=False,
+    )
+    config.update(max_epochs=1, chkpt_every=1)
+    path = tmp_path / "train.json"
+    path.write_text(json.dumps(config))
+    output = tmp_path / "single"
+
+    trainer = train.run_from_json(
+        parameters_file=str(path),
+        load_only=True,
+        output=output,
+    )
+    assert isinstance(trainer, Trainer)
+    assert not len(trainer.record_index)
+    assert not (output / "checkpoints").exists()
+
+    args = create_parser().parse_args(
+        [
+            "train",
+            "--parameters-file",
+            str(path),
+            "--output",
+            str(output),
+        ]
+    )
+    args.func(args)
+
+    assert (output / "config.json").is_file()
+    assert list((output / "checkpoints").rglob("*.pth"))
+    assert not (output / "experiment.json").exists()
+    assert not (output / "runs").exists()
+
+
+def test_group_seed_expansion_matches_explicit_runs_and_extends_stably(
+    tmp_path: Path,
+) -> None:
+    config = json.loads((EXAMPLES / "experiment.json").read_text())
+    original = deepcopy(config)
+    runs = expand_runs(config)
+    assert len(runs) == 9
+    assert list(runs)[:3] == ["mlp-small-1", "mlp-small-2", "mlp-small-3"]
+    assert runs["mlp-medium-2"]["group"] == "mlp-medium"
+    assert runs["mlp-medium-2"]["seed"] == 2
+    assert config == original
+
+    initial = TrainExperiment(tmp_path, runs, common=config["common"])
+    assert expand_runs({"runs": runs}) == runs
+    config["seeds"].append(4)
+    extended = TrainExperiment(
+        tmp_path,
+        expand_runs(config),
+        common=config["common"],
+    )
+    assert len(extended.runs) == 12
+    for name, spec in initial.runs.items():
+        assert extended.runs[name] == spec
+
+    runs["mlp-small-1"]["estimator"]["kwargs"]["hidden_dims"].append(99)
+    assert runs["mlp-small-2"]["estimator"]["kwargs"]["hidden_dims"] == [8]
+    assert original["groups"] == config["groups"]
+
+
+@pytest.mark.parametrize(
+    "config",
+    [
+        {"runs": {"one": {}}, "groups": {"one": {}}, "seeds": [1]},
+        {"groups": {"one": {}}},
+        {"groups": {"one": {}}, "seeds": [1, 1]},
+    ],
+)
+def test_experiment_rejects_ambiguous_or_incomplete_expansion(
+    config: ExperimentConfig,
+) -> None:
+    with pytest.raises(ValueError):
+        expand_runs(config)
+
+
+def test_compact_experiment_cli_writes_expanded_manifest_and_analysis(
+    tmp_path: Path,
+) -> None:
+    config = json.loads((EXAMPLES / "experiment.json").read_text())
+    config["groups"] = {"small": config["groups"]["mlp-small"]}
+    config["seeds"] = [1]
+    config["comparisons"] = {}
+    common = config["common"]
+    common["data"]["dataset"]["kwargs"]["num_samples"] = 12
+    common["reporter"]["kwargs"].update(tensorboard=False)
+    common.update(max_epochs=1, trace_graph=False)
+    config["summary"].update(plots=False, resamples=10)
+    path = tmp_path / "experiment.json"
+    path.write_text(json.dumps(config))
+    output = tmp_path / "experiment"
+
+    result = experiment.run_from_json(
+        parameters_file=str(path),
+        output=output,
+        quiet=True,
+    )
+
+    assert len(result["completed"]) == 1
+    assert result["failures"] == {}
+    assert result["summary"]["rows"]
+    manifest = json.loads((output / "experiment.json").read_text())
+    (spec,) = manifest["runs"].values()
+    assert spec["name"] == "small-1"
+    assert spec["group"] == "small"
+    assert spec["seed"] == 1
+    assert list(output.glob("runs/*/attempt-*/state.pt"))
+    assert (output / "analysis" / "summary.json").is_file()