statcatlib 0.13.0__tar.gz → 0.14.0__tar.gz

{statcatlib-0.13.0 → statcatlib-0.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: statcatlib
-Version: 0.13.0
+Version: 0.14.0
 Summary: Queryable open data catalog engine for DCAT-AP, StatDCAT-AP, CKAN, and SDMX
 Keywords: sdmx,dcat,open-data,catalog,duckdb,statistical-data,ckan
 Author: gabrielgellner
@@ -18,7 +18,7 @@ Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
 Requires-Dist: attrs>=26.1.0
 Requires-Dist: duckdb>=2.0.0
 Requires-Dist: httpx>=0.28
-Requires-Dist: sdmxlib>=0.84,<0.85
+Requires-Dist: sdmxlib>=0.85,<0.86
 Requires-Dist: lancedb>=0.39 ; extra == 'search'
 Requires-Dist: sentence-transformers>=3.0 ; extra == 'search'
 Requires-Dist: typer>=0.15 ; extra == 'search'

{statcatlib-0.13.0 → statcatlib-0.14.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "statcatlib"
-version = "0.13.0"
+version = "0.14.0"
 description = "Queryable open data catalog engine for DCAT-AP, StatDCAT-AP, CKAN, and SDMX"
 license = "Apache-2.0"
 license-files = ["LICENSE"]
@@ -29,7 +29,7 @@ dependencies = [
     "attrs>=26.1.0",
     "duckdb>=2.0.0",
     "httpx>=0.28",
-    "sdmxlib>=0.84,<0.85",
+    "sdmxlib>=0.85,<0.86",
 ]
 
 [[project.authors]]

{statcatlib-0.13.0 → statcatlib-0.14.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "statcatlib"
-version = "0.13.0"
+version = "0.14.0"
 description = "Queryable open data catalog engine for DCAT-AP, StatDCAT-AP, CKAN, and SDMX"
 license = "Apache-2.0"
 license-files = ["LICENSE"]
@@ -27,9 +27,9 @@ dependencies = [
     # by DuckDB 1.x, so by statcatlib 0.8 or earlier.
     "duckdb>=2.0.0",
     "httpx>=0.28",
-    # <0.85: every suite library is pre-1.0, so a minor bump may break
+    # <0.86: every suite library is pre-1.0, so a minor bump may break
     # (GH-67).
-    "sdmxlib>=0.84,<0.85",
+    "sdmxlib>=0.85,<0.86",
 ]
 
 [project.optional-dependencies]

{statcatlib-0.13.0 → statcatlib-0.14.0}/src/statcatlib/EGM.md

@@ -5,8 +5,8 @@
 `statcatlib` is the package root. `__init__.py` re-exports the public API of
 `model`, `sources` and `store`; `CatalogAdmin` from `admin.py`; `Catalog`, `Product`, `Edition`,
 `SealOutcomes` and `NoLiveEditionError` from `catalog.py`; the errors `InventoryConflictError`,
-`InventoryContentionError`, `InventoryRuleError`, `NonFastForwardError` and
-`CrossProductRuleError` from `store/inventory.py`, and `WorkingCopyExistsError`,
+`InventoryContentionError`, `InventoryRuleError`, `InventoryWriteRefusedError`,
+`NonFastForwardError` and `CrossProductRuleError` from `store/inventory.py`, and `WorkingCopyExistsError`,
 `WorkingCopyClosedError`, `SealCheckError`, `EditionCheckError` and `JournalError` from
 `store/working_copy.py`; `NotLoadedError` and `is_unloaded` from the store's
 sentinel module; and part of `expr` (listed below). It exposes `checks`,
@@ -94,8 +94,8 @@ and test catalogs (`testing.py`).
   `Edition`, all recorded in one version) and `refused` (product id to the
   error: `SealCheckError`, `NonFastForwardError` or another
   `InventoryRuleError` (`CrossProductRuleError` from the pre-check or the
-  record included), `ManifestError`, `EditionCheckError`, or
-  `InventoryContentionError`).
+  record included), `ManifestError`, `EditionCheckError`,
+  `InventoryContentionError`, or `InventoryWriteRefusedError`).
 - `Edition` is re-exported from `store/working_copy.py`.
 
 ### `admin.py`
@@ -211,7 +211,9 @@ read.
 - `verify_catalog(path) -> VerifyReport`: reads `FullInventory`; reports each
   root the read protocol skipped (`full.skipped`), each leaf that fails its
   hash or `validate_leaf`, and for every leaf entry a `manifest.json` whose
-  SHA-256 differs from the entry's, then runs
+  SHA-256 differs from the entry's, or, when it matches, each filled routing
+  field or artefact record `sdmxlib.inventory.verify_entry` finds differing,
+  with the leaf as the finding's file, then runs
   `check_edition(at="publish", structures=..., extensions={"statcat":
   check_catalog})`, with `structures=` from `_structures_for`: every recorded
   edition of every structure product for a data edition, the other
@@ -295,6 +297,17 @@ read.
   dataflow per table and the rows from `(seed, product_id)`, fills with
   `ingest_dataflows(..., partials=True)`, indexes `CL_GEO`'s codes with an
   `fts_only` `CodelistIndexPolicy` when lancedb is installed, and seals.
+  `codr_product` is `codr_working_copy(...).seal()`: `codr_working_copy`
+  takes the same arguments and returns the filled, indexed working copy,
+  unsealed. `codr_corpus(catalog, groups=(CorpusGroup(7000),), *, seed=0,
+  shape=..., embargoed=0.1, on_sealed=None) -> list[str]` seals each
+  `CorpusGroup(products, tables=1)`'s products, named by
+  `corpus_product_id` (`codr-<tables>t-<index:05d>`), through
+  `codr_product`, skipping products the Inventory records and discarding a
+  working copy a crash left; it seals the share `corpus_embargoed` picks (a
+  SplitMix64 draw per product id) between two `CatalogAdmin.set_embargo`
+  versions, turns the embargo off first and leaves it off, and returns the
+  products it sealed. `bench/corpus.py` (GH-124) drives it.
   `structure-classifications` holds `CL_CLASS_*`, `CL_UNIT` and the
   SDMX-agency `CL_OBS_STATUS`, so every codelist a table cuts has a
   structure home (A5). The draws are SplitMix64 seeded by SHA-256, so they

{statcatlib-0.13.0 → statcatlib-0.14.0}/src/statcatlib/__init__.py

@@ -93,6 +93,7 @@ from statcatlib.store.inventory import (
     InventoryConflictError,
     InventoryContentionError,
     InventoryRuleError,
+    InventoryWriteRefusedError,
     NonFastForwardError,
 )
 from statcatlib.store.sentinel import NotLoadedError, is_unloaded
@@ -159,6 +160,7 @@ __all__ = [
     "InventoryConflictError",
     "InventoryContentionError",
     "InventoryRuleError",
+    "InventoryWriteRefusedError",
     "JournalError",
     "LicenceDocument",
     "MicrodataDataset",

{statcatlib-0.13.0 → statcatlib-0.14.0}/src/statcatlib/catalog.py

@@ -37,7 +37,13 @@ from sdmxlib.edition import Finding, ManifestError, read_manifest
 from sdmxlib.inventory import CATALOG_NAME, LiveEdition, LiveInventory, decode_root, local_fetch, root_path
 
 from statcatlib.bundle import description_from_root
-from statcatlib.store.inventory import Change, InventoryContentionError, InventoryRuleError, InventoryWriter
+from statcatlib.store.inventory import (
+    Change,
+    InventoryContentionError,
+    InventoryRuleError,
+    InventoryWriter,
+    InventoryWriteRefusedError,
+)
 from statcatlib.store.store import VIEW_ATTRIBUTES, EditionView, IncompatibleCatalogError, _Store
 from statcatlib.store.working_copy import (
     Edition,
@@ -289,7 +295,7 @@ def _seal_ready(
     for wc in working_copies:
         try:
             wc._precheck(checks, batch=batch)
-        except (SealCheckError, InventoryRuleError, InventoryContentionError) as exc:
+        except (SealCheckError, InventoryRuleError, InventoryContentionError, InventoryWriteRefusedError) as exc:
             refused[wc.product_id] = exc
         else:
             ready.append(wc)
@@ -479,6 +485,12 @@ def _record_batch(
             msg = f"{exc}; the edition {done.wc.edition_id} is recorded at the next working_copy()"
             refused[done.wc.product_id] = InventoryContentionError(msg)
         return {}
+    except InventoryWriteRefusedError as exc:
+        # The journal entries stay, so a later working_copy() records each edition.
+        for done in sealed:
+            msg = f"{exc}; the edition {done.wc.edition_id} stays unrecorded until a working_copy() can record it"
+            refused[done.wc.product_id] = InventoryWriteRefusedError(msg)
+        return {}
     recorded: dict[str, Edition] = {}
     for done in sealed:
         exc = at_record.get(done.wc.product_id)

{statcatlib-0.13.0 → statcatlib-0.14.0}/src/statcatlib/cli/__init__.py

@@ -452,8 +452,8 @@ def verify_catalog(
     """Check a catalog's integrity: its Inventory and every edition it records.
 
     It reads the newest valid Inventory version and checks every leaf, every
-    manifest's SHA-256, every file each manifest lists, the shared ``data/``
-    files included, each partial against its structure edition, and each
+    manifest's SHA-256, each entry's routing fields and artefact records, every
+    file each manifest lists, the shared ``data/`` files included, each partial against its structure edition, and each
     structure edition's references (`statcatlib.diagnostics.verify_catalog`).
     It writes nothing. It exits 0 when everything checks, 1 on any finding,
     each printed with the file it names, and 2 when the catalog cannot be
@@ -539,7 +539,13 @@ def prune(
     try:
         admin = sc.CatalogAdmin.open(path)
         report = admin.prune(older_than=timedelta(days=older_than), reason=reason, dry_run=not apply)
-    except (IncompatibleCatalogError, sc.InventoryRuleError, sc.InventoryContentionError, ValueError) as exc:
+    except (
+        IncompatibleCatalogError,
+        sc.InventoryRuleError,
+        sc.InventoryContentionError,
+        sc.InventoryWriteRefusedError,
+        ValueError,
+    ) as exc:
         _say(f"Cannot prune the catalog: {exc}", style="red")
         raise typer.Exit(code=1) from None
     for product_id, why in report.skipped_products.items():

{statcatlib-0.13.0 → statcatlib-0.14.0}/src/statcatlib/diagnostics.py

@@ -5,7 +5,8 @@ Two functions, each run by a ``statcat`` command:
 - `verify_catalog` (``statcat verify-catalog PATH``) reads the newest valid
   Inventory version with `sdmxlib.inventory.FullInventory`. It checks every
   root the read protocol skipped, every leaf, and, for every entry the leaves
-  hold, the ``manifest_sha256`` the Inventory records. It then runs
+  hold, the ``manifest_sha256`` the Inventory records and the routing fields
+  and artefact records it holds. It then runs
   `sdmxlib.edition.check_edition` at the ``publish`` profile with
   `statcatlib.bundle.check_catalog`, which checks every file the manifest
   lists, the shared ``data/`` files included, each partial against the
@@ -33,7 +34,7 @@ from pathlib import Path
 from typing import TYPE_CHECKING, Literal
 
 from attrs import frozen
-from sdmxlib.edition import MANIFEST_NAME, ManifestError, check_edition, list_editions, read_journal
+from sdmxlib.edition import MANIFEST_NAME, ManifestError, check_edition, list_editions, read_journal, read_manifest
 from sdmxlib.inventory import (
     CATALOG_NAME,
     CURRENT_NAME,
@@ -41,6 +42,7 @@ from sdmxlib.inventory import (
     InventoryError,
     decode_current,
     validate_leaf,
+    verify_entry,
 )
 
 from statcatlib.bundle import STATCAT_SECTION, check_catalog
@@ -281,8 +283,23 @@ def _structures_for(full: FullInventory, root: Path, product_id: str) -> list[Pa
     return found
 
 
-def _verify_edition(root: Path, product_id: str, entry: LeafEntry, structures: list[Path]) -> list[VerifyFinding]:
-    """Check one recorded edition: its manifest's SHA-256, then `check_edition` at ``publish`` with *structures*."""
+def _routing_findings(leaf_file: str, entry: LeafEntry, edition_dir: Path) -> list[VerifyFinding]:
+    """Each filled routing field or artefact record of *entry* that differs from its manifest (`verify_entry`).
+
+    The caller has checked the manifest's SHA-256. A manifest that cannot be
+    parsed is left to `check_edition`, which reports it.
+    """
+    try:
+        manifest = read_manifest(edition_dir)
+    except ManifestError:
+        return []
+    return [VerifyFinding(file=leaf_file, detail=detail) for detail in verify_entry(entry, manifest)]
+
+
+def _verify_edition(
+    root: Path, product_id: str, leaf_file: str, entry: LeafEntry, structures: list[Path]
+) -> list[VerifyFinding]:
+    """Check one edition: its manifest's SHA-256, its entry's fields, then `check_edition` with *structures*."""
     edition_rel = f"products/{product_id}/{entry.edition_id}"
     edition_dir = root / edition_rel
     manifest_rel = f"{edition_rel}/{MANIFEST_NAME}"
@@ -293,6 +310,8 @@ def _verify_edition(root: Path, product_id: str, entry: LeafEntry, structures: l
     if digest != entry.manifest_sha256:
         detail = f"SHA-256 {digest}, the Inventory records {entry.manifest_sha256}"
         found.append(VerifyFinding(file=manifest_rel, detail=detail))
+    else:
+        found += _routing_findings(leaf_file, entry, edition_dir)
     report = check_edition(
         edition_dir, at="publish", structures=structures or None, extensions={STATCAT_SECTION: check_catalog}
     )
@@ -307,7 +326,9 @@ def verify_catalog(path: str | Path) -> VerifyReport:
     read protocol skipped is a finding. For each product it checks the leaf
     against its name and sdmxlib's leaf rules; for each entry of the leaf, in
     every state but retired, it compares ``manifest.json`` with the
-    ``manifest_sha256`` the Inventory records, then runs
+    ``manifest_sha256`` the Inventory records, compares the entry's filled
+    ``dataflows``, ``pull_bytes`` and ``artefacts`` with the manifest's
+    (`sdmxlib.inventory.verify_entry`; an entry without them is skipped), then runs
     ``check_edition(at="publish", extensions={"statcat": check_catalog})``,
     which checks the size and digests of every file the manifest lists,
     ``data/`` files included, the lake, the bindings and the ``statcat``
@@ -345,7 +366,7 @@ def verify_catalog(path: str | Path) -> VerifyReport:
         structures = _structures_for(full, root, product_id)
         for entry in leaf.editions:
             editions.append(f"{product_id}/{entry.edition_id}")
-            findings += _verify_edition(root, product_id, entry, structures)
+            findings += _verify_edition(root, product_id, ref.leaf, entry, structures)
     stale, unread = _stale(full, InventoryWriter(root).artefacts)
     findings += unread
     return VerifyReport(

{statcatlib-0.13.0 → statcatlib-0.14.0}/src/statcatlib/store/EGM.md

@@ -115,7 +115,9 @@ constants, `IncompatibleCatalogError`, `IncompatibleIndexError`, `Row` and
   `previous` and `manifest_sha256`.
 - `settle_journal(writer, product_dir, product_id, lifecycle, *,
   exclude=())` applies the journal rule catalog-wide (`_Repair` runs the
-  publish checks and the one commit); `record_journalled(...)` is its pure
+  publish checks and the one commit, with each edition's `routing_fields`);
+  `recorded_routing(fields, kind)` gives `record` both routing fields, and
+  `artefacts` for a structure edition only; `record_journalled(...)` is its pure
   part per product, taking each entry as `(JournalEntry, manifest_sha256,
   schema_version, kind)`.
 - `Planned(product_id, edition_id, leaf, read=None)` and
@@ -149,17 +151,23 @@ constants, `IncompatibleCatalogError`, `IncompatibleIndexError`, `Row` and
   reader the rules use; `cross_product.ManifestArtefacts(root)` by default),
   `catalog_document()`, `has_version()`, `full() -> FullInventory | None`,
   `head() -> Root | None`, `write_catalog_json(*, catalog_id=None)`,
-  `try_commit(base, leaves, *, retired=None, embargo=None, extensions=None,
-  kind, reason) -> Root` (no cross-product rule), `refusals(full, leaves, *,
+  `check_writable(base, *, now=None) -> CatalogDocument` (the two stage-1
+  refusals alone), `try_commit(base, leaves, *, retired=None, embargo=None,
+  extensions=None, built_from=None, now=None, kind, reason) -> Root` (no
+  cross-product rule), `refusals(full, leaves, *,
   artefacts=None) -> dict[str, CrossProductRuleError]`, and `commit(change,
   *, kind, reason) -> Root`, which runs `refusals` on each planned change and
   raises the first refusal by product id before writing.
-- `Change(leaves, retired=None, embargo=None, extensions=None)`: what
+- `Change(leaves, retired=None, embargo=None, extensions=None,
+  built_from=None)`: what
   `commit`'s *change* callable returns for a head.
 - `MAX_RACES = 8`. `producer()` returns
   `Producer(name="statcatlib", version=<installed version>)`.
 - Errors: `InventoryConflictError` (another writer created the version
   first), `InventoryContentionError` (`MAX_RACES` lost in a row),
+  `InventoryWriteRefusedError` (a subclass of sdmxlib's `InventoryError`: a
+  `catalog.json` at 2.x, or a base version more than 5 minutes ahead of this
+  machine's clock, named in the message),
   `InventoryRuleError(message, *, rule, product, edition)` with
   `InventoryRuleError.wrapping(exc, ...)` over sdmxlib's `TransitionError`,
   and its subclasses `NonFastForwardError` and `CrossProductRuleError(message,
@@ -184,7 +192,7 @@ Imports only sdmxlib, so `inventory.py` can import it.
   `manifest.json`, checks its SHA-256 (`ManifestError` otherwise) and caches
   by `(edition_id, sha)`; `StoreArtefacts(edition_id, read, *, fallback)`
   answers one working copy's edition with `read()` and the rest with
-  *fallback*; `manifest_records(manifest)` is the shared conversion.
+  *fallback*. The conversion is `sdmxlib.inventory.manifest_records`.
 - `StaleEdition` and `stale_editions(full, artefacts)`: each live or
   embargoed data edition and URN whose fingerprint (a partial's
   `partial_of`) no live or embargoed edition of its structure home holds.
@@ -307,7 +315,7 @@ Imports only sdmxlib, so `inventory.py` can import it.
   writer's journal repair, counts as recorded); clear the journal entry. The steps are split for
   `Catalog.seal`: `_precheck(checks)` (steps 0 to 2), `_seal_edition()`
   (steps 3 to 5, returning a private `_Sealed` with the edition, its
-  manifest's store schema and its `cut_partials`), `_next_leaf(full, sealed)` (the record's leaf;
+  manifest's store schema, its `routing_fields` and its `cut_partials`), `_next_leaf(full, sealed)` (the record's leaf, with `recorded_routing`;
   `InventoryRuleError` on a refusal), `_publish_check(structures)`,
   `_recorded()` and `_unrecorded(exc)` (clear the journal entry; the second
   also logs the WARNING, for an `InventoryRuleError` or an
@@ -343,7 +351,8 @@ Imports only sdmxlib, so `inventory.py` can import it.
   one whose check was given a journalled structure edition not recorded,
   is dropped. Any other refusal, and a failed check, clears the entry and
   logs a WARNING.
-  `InventoryContentionError` keeps the entry for the next working copy. An
+  `InventoryContentionError` and `InventoryWriteRefusedError` keep the entry
+  for a later working copy. An
   unreadable entry raises `JournalError`.
 - A refusal after the manifest (a rollback landing during the seal) raises
   `NonFastForwardError` from the record; the seal clears the journal entry,
@@ -358,7 +367,15 @@ Imports only sdmxlib, so `inventory.py` can import it.
   `FileExistsError` becomes `InventoryConflictError`; a leaf that exists already counts as written, since
   its name is the hash of its bytes. It writes leaves, then the root, then
   `CURRENT`, and replaces `CURRENT` only with a higher `seq` in the same
-  `restore_epoch`. `commit` re-reads the head and calls *change* again after
+  `restore_epoch`. Every root it writes carries `previous_root_sha256` and
+  `changes`, from root `seq - 1`'s stored bytes and the leaves of each product
+  whose pointer or tombstones change, also when `built_from` names an older
+  version; only those leaves are read. `try_commit` runs `check_writable`
+  first: `catalog.json` is decoded with `decoder="stage-1-writer"`, and
+  `next_created_at` refuses a base more than 5 minutes ahead. The seal's
+  `_precheck` runs it too, so a refused seal writes nothing and leaves the
+  working copy open. `catalog_document()` decodes as a reader does.
+  `commit` re-reads the head and calls *change* again after
   each `InventoryConflictError`, so every rule is evaluated against the
   version the commit lands on, and raises `InventoryContentionError` after
   `MAX_RACES`. A root records `catalog.json`'s `restore_epoch`, and its
@@ -401,9 +418,11 @@ Imports only sdmxlib, so `inventory.py` can import it.
 
 - Run: `just ci`
 - Unit tier only: `uv run pytest -m unit`
-- The Inventory writer: `tests/test_inventory_writer.py` (unit) and
+- The Inventory writer: `tests/test_inventory_writer.py` (unit),
   `tests/test_inventory_conformance.py` (integration, sdmxlib's conformance
-  suite). The cross-product rules: `tests/test_cross_product_rules.py` (unit,
+  suite and `run_history`) and `tests/test_inventory_history.py` (integration:
+  the chain over seals, a correction, a lost race, the fast base, an opted-in
+  catalog). The routing fields: `tests/test_inventory_routing.py`. The cross-product rules: `tests/test_cross_product_rules.py` (unit,
   `refusals` over built leaves) and `tests/test_cross_product.py`
   (integration: seals, batches, releases, rollbacks, stale editions,
   partials with no structure home, the committed sabotage test). Seals: `tests/test_seal_checks.py`, `tests/test_seal_crash.py`

{statcatlib-0.13.0 → statcatlib-0.14.0}/src/statcatlib/store/cross_product.py

@@ -48,14 +48,13 @@ from typing import TYPE_CHECKING, Literal
 
 from attrs import frozen
 from sdmxlib.edition import MANIFEST_NAME, ManifestError, read_manifest
-from sdmxlib.inventory import PartialRecord
+from sdmxlib.inventory import manifest_records
 
 if TYPE_CHECKING:
     from collections.abc import Callable, Iterable, Mapping
     from pathlib import Path
 
-    from sdmxlib.edition import Manifest
-    from sdmxlib.inventory import FullInventory, Kind, Leaf, LeafEntry, State
+    from sdmxlib.inventory import FullInventory, Kind, Leaf, LeafEntry, PartialRecord, State
 
 __all__ = [
     "ACTIVE",
@@ -69,7 +68,6 @@ __all__ = [
     "cited_structures",
     "homeless_partials",
     "incoming",
-    "manifest_records",
     "refusals",
     "stale_editions",
 ]
@@ -124,17 +122,6 @@ class _Held:
     fingerprint: str
 
 
-def manifest_records(manifest: Manifest) -> dict[str, PartialRecord]:
-    """A sealed edition's artefacts as the rules read them (see `Artefacts`)."""
-    found: dict[str, PartialRecord] = {}
-    for artefact in manifest.artefacts:
-        if not artefact.partial:
-            found[artefact.urn] = PartialRecord(fingerprint=artefact.fingerprint)
-        elif artefact.partial_of is not None:
-            found[artefact.urn] = PartialRecord(fingerprint=artefact.fingerprint, partial_of=artefact.partial_of)
-    return found
-
-
 class ManifestArtefacts:
     """The `Artefacts` reader over sealed manifests, ``products/<product>/<edition>/manifest.json``.
 

{statcatlib-0.13.0 → statcatlib-0.14.0}/src/statcatlib/store/inventory.py

@@ -3,7 +3,7 @@
 sdmxlib owns the Inventory's documents, their validation and the state
 transition rules (`sdmxlib.inventory`). statcatlib writes them, and this module
 is the only code that writes ``catalog.json``, ``inventory/`` and ``CURRENT``.
-It adds three things to sdmxlib's helpers:
+It adds these things to sdmxlib's helpers:
 
 - **exclusive create**: each document is written by
   `sdmxlib.storage.create_exclusive`, which stages the file, fsyncs it,
@@ -17,7 +17,14 @@ It adds three things to sdmxlib's helpers:
 - **the cross-product rules** (`statcatlib.store.cross_product`, Phase 3):
   `InventoryWriter.commit` refuses a version that breaks specification §6.5
   item 4, 7 or 8 for an entry that becomes live or embargoed in it, with
-  `CrossProductRuleError`.
+  `CrossProductRuleError`;
+- **the history fields** (Inventory 1.2, GH-125): every root carries
+  ``previous_root_sha256`` and ``changes``, computed from root ``seq - 1`` and
+  its leaves, whatever version a correction is built from, and again after a
+  lost race;
+- **the stage-1 refusals**: every commit decodes ``catalog.json`` as a writer
+  without the catalog lock, so it refuses a catalog that has opted in to expiry,
+  and refuses a base version more than 5 minutes ahead of this machine's clock.
 """
 
 from __future__ import annotations
@@ -39,6 +46,7 @@ from sdmxlib.inventory import (
     CATALOG_NAME,
     CURRENT_NAME,
     CatalogDocument,
+    ChangeItem,
     Current,
     Event,
     EventKind,
@@ -48,24 +56,26 @@ from sdmxlib.inventory import (
     Root,
     Tombstone,
     TransitionError,
+    change_list,
     decode_catalog,
     decode_current,
+    decode_leaf,
     encode,
     leaf_ref,
     local_fetch,
+    next_created_at,
     next_root,
     root_path,
     sha256_hex,
-    validate_root,
 )
 from sdmxlib.storage import create_exclusive
 
 from statcatlib.store import cross_product
 
 if TYPE_CHECKING:
-    from collections.abc import Callable, Mapping, Sequence
+    from collections.abc import Callable, Iterable, Mapping, Sequence
 
-    from sdmxlib.inventory import Rule
+    from sdmxlib.inventory import LeafRef, Rule
     from sdmxlib.storage import JsonValue
 
     from statcatlib.store.cross_product import Artefacts, CrossRule
@@ -77,6 +87,7 @@ __all__ = [
     "InventoryConflictError",
     "InventoryContentionError",
     "InventoryRuleError",
+    "InventoryWriteRefusedError",
     "InventoryWriter",
     "NonFastForwardError",
     "producer",
@@ -100,6 +111,18 @@ class InventoryContentionError(Exception):
     """A commit lost `MAX_RACES` races in a row to other writers, and gave up."""
 
 
+class InventoryWriteRefusedError(InventoryError):
+    """This writer must not write the next Inventory version, whatever the version holds.
+
+    The catalog has opted in to expiry (``catalog.json`` 2.x), which needs a
+    writer with the catalog lock, or the base version's ``created_at`` is more
+    than 5 minutes ahead of this machine's clock. The message names the
+    format, or the base version's time and this machine's. It stays refused
+    until the catalog or the clock changes, so a seal refused after its
+    manifest keeps its journal entry, as `InventoryContentionError` does.
+    """
+
+
 class InventoryRuleError(Exception):
     """A change sdmxlib's Inventory rules refuse, or a lifecycle change statcatlib refuses.
 
@@ -229,12 +252,16 @@ class Change:
         retired: The tombstones the next root holds; ``None`` keeps the head's.
         embargo: The next root's embargo setting; ``None`` keeps the head's.
         extensions: The next root's extension sections; ``None`` keeps the head's.
+        built_from: The older version a correction is built from
+            (``previous_inventory``); ``None`` builds from the head. The
+            history fields follow the head either way.
     """
 
     leaves: Mapping[str, Leaf]
     retired: Sequence[Tombstone] | None = None
     embargo: bool | None = None
     extensions: Mapping[str, Mapping[str, JsonValue]] | None = None
+    built_from: int | None = None
 
 
 def _fsync_dir(directory: Path) -> None:
@@ -313,7 +340,10 @@ class InventoryWriter:
     # -- reading ---------------------------------------------------------
 
     def catalog_document(self) -> CatalogDocument:
-        """``catalog.json``, decoded; a missing one raises `FileNotFoundError`, a malformed one `InventoryError`."""
+        """``catalog.json``, decoded as a reader decodes it, 2.x included.
+
+        A missing one raises `FileNotFoundError`, a malformed one `InventoryError`.
+        """
         return decode_catalog((self._root / CATALOG_NAME).read_bytes())
 
     def has_version(self) -> bool:
@@ -378,6 +408,81 @@ class InventoryWriter:
         create_exclusive(self._root / CATALOG_NAME, encode(document))
         return document
 
+    def check_writable(self, base: Root | None, *, now: datetime | None = None) -> CatalogDocument:
+        """Refuse a commit this writer must not make on top of *base*, and return ``catalog.json``.
+
+        Every commit runs it first, and a seal runs it before it writes
+        anything.
+
+        Args:
+            base: The newest version, or ``None`` for the catalog's first.
+            now: This machine's time; the clock when ``None``.
+
+        Returns:
+            ``catalog.json``, decoded as a writer without the catalog lock.
+
+        Raises:
+            InventoryWriteRefusedError: ``catalog.json`` is malformed or 2.x, or
+                *base* was created more than 5 minutes ahead of *now*; the
+                message names the format or the base version's time.
+
+        A missing ``catalog.json`` raises `FileNotFoundError`.
+        """
+        raw = (self._root / CATALOG_NAME).read_bytes()
+        try:
+            document = decode_catalog(raw, decoder="stage-1-writer")
+            next_created_at(base, now or datetime.now(UTC))
+        except InventoryError as exc:
+            raise InventoryWriteRefusedError(str(exc)) from exc
+        return document
+
+    def _stored_root(self, base: Root) -> bytes:
+        """The stored bytes of *base*; a root a restore removed means another writer moved the head."""
+        try:
+            return (self._root / root_path(base.seq)).read_bytes()
+        except FileNotFoundError as exc:
+            msg = f"Inventory version {base.seq} no longer exists"
+            raise InventoryConflictError(msg) from exc
+
+    def _base_leaves(
+        self, base: Root, products: Mapping[str, LeafRef], retired: Iterable[Tombstone]
+    ) -> dict[str, Leaf]:
+        """The leaves of *base* the change list reads: each product whose pointer or tombstones change.
+
+        Args:
+            base: The base version, root ``seq - 1``.
+            products: Every product's leaf pointer in the new version.
+            retired: The new version's tombstones.
+
+        Returns:
+            Those products' leaves in *base*, by product.
+
+        Raises:
+            InventoryError: A leaf is missing or malformed.
+        """
+        moved = {t.product_id for t in set(base.retired) ^ set(retired)}
+        moved |= {p for p, ref in base.products.items() if products.get(p) != ref}
+        found: dict[str, Leaf] = {}
+        for product_id in moved & base.products.keys():
+            ref = base.products[product_id]
+            try:
+                found[product_id] = decode_leaf((self._root / ref.leaf).read_bytes())
+            except FileNotFoundError as exc:
+                msg = f"{product_id}'s leaf {ref.leaf} in Inventory version {base.seq} is missing"
+                raise InventoryError(msg) from exc
+        return found
+
+    def _changes(
+        self,
+        base: Root | None,
+        leaves: Mapping[str, Leaf],
+        products: Mapping[str, LeafRef],
+        retired: Sequence[Tombstone],
+    ) -> tuple[ChangeItem, ...]:
+        """The change list of the version after *base*, which is root ``seq - 1`` whatever it is built from."""
+        base_leaves = {} if base is None else self._base_leaves(base, products, retired)
+        return change_list(base, base_leaves, products=products, leaves=leaves, retired=retired)
+
     def try_commit(
         self,
         base: Root | None,
@@ -386,11 +491,16 @@ class InventoryWriter:
         retired: Sequence[Tombstone] | None = None,
         embargo: bool | None = None,
         extensions: Mapping[str, Mapping[str, JsonValue]] | None = None,
+        built_from: int | None = None,
+        now: datetime | None = None,
         kind: EventKind,
         reason: str,
     ) -> Root:
         """Write the version after *base*, once: its changed leaves, its root, then ``CURRENT``.
 
+        The root carries the history fields, computed from *base*'s stored
+        bytes and its leaves, also when *built_from* names an older version.
+
         Args:
             base: The newest version, read before the change was computed, or
                 ``None`` for the catalog's first.
@@ -399,6 +509,9 @@ class InventoryWriter:
             retired: The tombstones; *base*'s when ``None``.
             embargo: The embargo setting; *base*'s when ``None``.
             extensions: The extension sections; *base*'s when ``None``.
+            built_from: The older version a correction is built from; *base*
+                when ``None``.
+            now: This machine's time; the clock when ``None``.
             kind: What the version records (``event.kind``).
             reason: Why, in the caller's words (``event.reason``).
 
@@ -408,25 +521,34 @@ class InventoryWriter:
         Raises:
             InventoryConflictError: Another writer created the version after
                 *base* first.
+
+        `check_writable` raises `InventoryWriteRefusedError` before anything is
+        written.
         """
+        now = now or datetime.now(UTC)
+        catalog = self.check_writable(base, now=now)
+        previous_bytes = None if base is None else self._stored_root(base)
         products = dict(base.products) if base is not None else {}
         for product_id, leaf in leaves.items():
             data, ref = leaf_ref(leaf)
             self._write_leaf(self._root / ref.leaf, data)
             products[product_id] = ref
+        tombstones = tuple(retired) if retired is not None else (base.retired if base is not None else ())
         root = next_root(
             base,
             products=products,
             event=Event(actor=_actor(), reason=reason, kind=kind),
             producer=producer(),
-            now=datetime.now(UTC),
-            retired=retired,
+            now=now,
+            retired=tombstones,
             embargo=embargo,
             # A root records catalog.json's restore_epoch when it is written.
-            restore_epoch=self.catalog_document().restore_epoch,
+            restore_epoch=catalog.restore_epoch,
+            built_from=built_from,
             extensions=extensions,
+            previous_bytes=previous_bytes,
+            changes=self._changes(base, leaves, products, tombstones),
         )
-        validate_root(root)
         data = encode(root)
         try:
             create_exclusive(self._root / root_path(root.seq), data)
@@ -466,7 +588,9 @@ class InventoryWriter:
         A version that breaks a cross-product rule raises the first refused
         product's `CrossProductRuleError`, by product id, and a manifest the
         rules cannot read raises `sdmxlib.edition.ManifestError`; neither
-        writes anything.
+        writes anything. `try_commit` raises `InventoryWriteRefusedError` for a
+        catalog or a base version this writer must not write on, and it
+        writes nothing either.
         """
         for _ in range(MAX_RACES):
             full = self.full()
@@ -479,6 +603,7 @@ class InventoryWriter:
                     retired=planned.retired,
                     embargo=planned.embargo,
                     extensions=planned.extensions,
+                    built_from=planned.built_from,
                     kind=kind,
                     reason=reason,
                 )

{statcatlib-0.13.0 → statcatlib-0.14.0}/src/statcatlib/store/working_copy.py

@@ -38,6 +38,7 @@ version:
 | The cross-product rules refuse it | Clear the entry, log a WARNING, leave the edition |
 | Its check was given a structure edition that is not recorded | Clear the entry, log a WARNING, leave the edition |
 | `InventoryContentionError` | Keep the entry, so the next working copy records it |
+| `InventoryWriteRefusedError` | Keep the entry, so a working copy records it once the catalog or the clock allows |
 | The entry cannot be read | Raise `JournalError` naming the file and ``statcat doctor`` |
 
 Another product's entry is acted on only when its edition has a manifest and
@@ -68,7 +69,7 @@ import sdmxlib.edition as _edition
 from attrs import frozen
 from sdmxlib import SdmxUrn
 from sdmxlib.edition import MANIFEST_NAME, Finding, ManifestError, canonical_json, check_edition
-from sdmxlib.inventory import FullInventory, Leaf, PartialRecord, TransitionError, head, record
+from sdmxlib.inventory import FullInventory, Leaf, PartialRecord, TransitionError, head, record, routing_fields
 from sdmxlib.storage.schema import SCHEMA_VERSION, meta_value
 
 from statcatlib.bundle import STATCAT_SECTION, catalog_section, check_catalog, structure_content
@@ -80,6 +81,7 @@ from statcatlib.store.inventory import (
     InventoryContentionError,
     InventoryRuleError,
     InventoryWriter,
+    InventoryWriteRefusedError,
     NonFastForwardError,
     producer,
 )
@@ -92,7 +94,7 @@ if TYPE_CHECKING:
     import duckdb
     from sdmxlib import Dataflow, EditionRegistry, MaintainableArtefact, RestRegistry, WriteSource
     from sdmxlib.edition import JournalEntry, Lifecycle, Manifest
-    from sdmxlib.inventory import Kind
+    from sdmxlib.inventory import Kind, RoutingFields
 
     from statcatlib.model.dataset import Dataset
     from statcatlib.model.skos import Concept, ConceptScheme
@@ -317,6 +319,24 @@ def _named(full: FullInventory | None, product_id: str, edition_id: str) -> bool
     return any(e.edition_id == edition_id for e in full.entries(product_id))
 
 
+def recorded_routing(fields: RoutingFields | None, kind: Kind) -> _Routing:
+    """What a leaf entry records of *fields*: both routing fields, and the artefact records of a structure edition.
+
+    Phase 5 extends the artefact records to data editions (GH-125 part 4).
+
+    Args:
+        fields: The edition's `sdmxlib.inventory.routing_fields`, or ``None``
+            to record none.
+        kind: The product's kind.
+
+    Returns:
+        ``record``'s ``dataflows``, ``pull_bytes`` and ``artefacts``.
+    """
+    if fields is None:
+        return None, None, None
+    return fields.dataflows, fields.pull_bytes, fields.artefacts if kind == "structure" else None
+
+
 def record_journalled(
     leaf: Leaf | None,
     entries: Sequence[tuple[JournalEntry, str, str, Kind]],
@@ -326,6 +346,7 @@ def record_journalled(
     embargo: bool,
     retired: Iterable[str],
     now: datetime,
+    routing: Mapping[str, RoutingFields] | None = None,
 ) -> tuple[Leaf | None, list[str], dict[str, TransitionError]]:
     """Record each journalled edition in *leaf*, by the journal rule.
 
@@ -338,6 +359,8 @@ def record_journalled(
         embargo: The head's embargo setting.
         retired: The edition ids the head tombstones for the product.
         now: The record time.
+        routing: Each edition's routing fields, by edition id; an edition
+            not in it is recorded without them.
 
     Returns:
         The next leaf (``None`` when nothing changed), the edition ids that are
@@ -350,6 +373,7 @@ def record_journalled(
     refused: dict[str, TransitionError] = {}
     gone = set(retired)
     for entry, sha, schema_version, kind in entries:
+        dataflows, pull_bytes, artefacts = recorded_routing((routing or {}).get(entry.edition_id), kind)
         try:
             current = record(
                 current,
@@ -363,6 +387,9 @@ def record_journalled(
                 embargo=embargo,
                 now=now,
                 retired=gone,
+                artefacts=artefacts,
+                dataflows=dataflows,
+                pull_bytes=pull_bytes,
             )
         except TransitionError as exc:
             held = next((e for e in (current.editions if current else ()) if e.edition_id == entry.edition_id), None)
@@ -410,8 +437,10 @@ def settle_journal(
 
     An entry of the product that cannot be read raises `JournalError`;
     another product's unreadable journal is left to that product. A record
-    that loses too many races raises `InventoryContentionError`, and the
-    entries are kept for the next working copy.
+    that loses too many races raises `InventoryContentionError`, a catalog or
+    base version the writer must not write on raises
+    `InventoryWriteRefusedError`, and either way the entries are kept for a
+    later working copy.
     """
     products_dir = product_dir.parent
     own = _read_journal(product_dir)
@@ -455,6 +484,10 @@ type _Journalled = tuple[JournalEntry, str, str, Kind]
 """A journal entry with its edition's ``manifest_sha256``, ``schema_version`` and product kind."""
 
 
+type _Routing = tuple[tuple[str, ...] | None, int | None, Mapping[str, PartialRecord] | None]
+"""``record``'s ``dataflows``, ``pull_bytes`` and ``artefacts``."""
+
+
 def _other_journals(products_dir: Path, product_id: str, exclude: Collection[str]) -> dict[str, list[JournalEntry]]:
     """The journal entries of every product but *product_id* and *exclude*; an unreadable journal is skipped."""
     found: dict[str, list[JournalEntry]] = {}
@@ -486,6 +519,11 @@ class _Repair:
         self._failed: dict[tuple[str, str], Exception] = {}
         self._refused: dict[tuple[str, str], Exception] = {}
         self._given: dict[tuple[str, str], frozenset[str]] = {}
+        self._routing = {
+            entry.edition_id: routing_fields(_edition.read_manifest(self._dir(product_id, entry.edition_id)))
+            for product_id, items in pending.items()
+            for entry, _, _, _ in items
+        }
         self._structures = self._journalled_structures()
         self._check(full)
 
@@ -550,6 +588,7 @@ class _Repair:
                 embargo=head_now.embargo if head_now is not None else False,
                 retired=retired,
                 now=datetime.now(UTC),
+                routing=self._routing,
             )
             for edition_id, exc in found.items():
                 self._refused[product_id, edition_id] = exc
@@ -594,6 +633,9 @@ class _Repair:
         except InventoryContentionError as exc:
             msg = f"{exc}; the journalled edition is recorded at the next working_copy()"
             raise InventoryContentionError(msg) from exc
+        except InventoryWriteRefusedError as exc:
+            msg = f"{exc}; the journalled edition stays unrecorded until a working_copy() can record it"
+            raise InventoryWriteRefusedError(msg) from exc
 
 
 class _NothingToRepairError(Exception):
@@ -773,11 +815,13 @@ class _Sealed:
     Attributes:
         edition: The edition.
         schema_version: Its store schema.
+        routing: Its manifest's routing fields, which the record writes.
         partials: The partials its manifest records a ``partial_of`` for.
     """
 
     edition: Edition
     schema_version: str
+    routing: RoutingFields
     partials: tuple[str, ...] = ()
 
 
@@ -1443,6 +1487,11 @@ class WorkingCopy:
                 disk unrecorded.
             InventoryContentionError: The record lost too many races; the
                 product's next ``working_copy()`` records the edition.
+            InventoryWriteRefusedError: The catalog has opted in to expiry, or
+                the newest Inventory version's time is more than 5 minutes
+                ahead of this machine's clock. Raised from the pre-check, it
+                leaves the working copy open; raised from the record, it keeps
+                the journal entry for a later ``working_copy()``.
 
         Before anything is written, a check's finding raises `SealCheckError`
         (for a structure product, also each dataflow, binding or partial it
@@ -1479,6 +1528,9 @@ class WorkingCopy:
         except InventoryContentionError as exc:
             msg = f"{exc}; the edition {self._edition_id} is recorded at the next working_copy()"
             raise InventoryContentionError(msg) from exc
+        except InventoryWriteRefusedError as exc:
+            msg = f"{exc}; the edition {self._edition_id} stays unrecorded until a working_copy() can record it"
+            raise InventoryWriteRefusedError(msg) from exc
         self._recorded()
         return sealed.edition
 
@@ -1501,8 +1553,12 @@ class WorkingCopy:
         Raises:
             SealCheckError: A check found something.
             NonFastForwardError: The base is no longer the live edition.
+
+        A catalog or a base version the writer must not write on raises
+        `InventoryWriteRefusedError` first.
         """
         store = self._live()
+        self._writer.check_writable(self._writer.head())
         settle_journal(
             self._writer,
             self._product_dir,
@@ -1702,7 +1758,12 @@ class WorkingCopy:
             previous=manifest.previous,
             manifest_sha256=manifest_sha256(edition_dir),
         )
-        return _Sealed(edition=edition, schema_version=manifest.schema_version, partials=cut_partials(manifest))
+        return _Sealed(
+            edition=edition,
+            schema_version=manifest.schema_version,
+            routing=routing_fields(manifest),
+            partials=cut_partials(manifest),
+        )
 
     def _next_leaf(self, full: FullInventory | None, sealed: _Sealed) -> Leaf:
         """Seal step 6's record: the product's next leaf in *full*, with *sealed* recorded.
@@ -1722,6 +1783,7 @@ class WorkingCopy:
         product_id, edition_id = self._product_id, self._edition_id
         leaf = full.leaf(product_id) if full is not None and product_id in full.products() else None
         retired = {t.edition_id for t in full.retired if t.product_id == product_id} if full else set()
+        dataflows, pull_bytes, artefacts = recorded_routing(sealed.routing, self._kind)
         try:
             return record(
                 leaf,
@@ -1735,6 +1797,9 @@ class WorkingCopy:
                 embargo=full.embargo if full is not None else False,
                 now=datetime.now(UTC),
                 retired=retired,
+                artefacts=artefacts,
+                dataflows=dataflows,
+                pull_bytes=pull_bytes,
             )
         except TransitionError as exc:
             raise InventoryRuleError.wrapping(exc, product=product_id, edition=edition_id) from exc

{statcatlib-0.13.0 → statcatlib-0.14.0}/src/statcatlib/testing.py

@@ -43,23 +43,30 @@ import sdmxlib as sl
 from attrs import field, frozen
 from sdmxlib.testing import synthetic_edition
 
+from statcatlib.admin import CatalogAdmin
 from statcatlib.catalog import Catalog
 from statcatlib.sources.sdmx import ingest_dataflows, prepare_sdmx
 from statcatlib.store.store import CodelistIndexPolicy
+from statcatlib.store.working_copy import WorkingCopyExistsError
 
 if TYPE_CHECKING:
-    from collections.abc import Generator, Sequence
+    from collections.abc import Callable, Generator, Sequence
 
     from sdmxlib.testing import Shape
 
-    from statcatlib.store.working_copy import Edition
+    from statcatlib.store.working_copy import Edition, WorkingCopy
 
 __all__ = [
     "CODR_AGENCY",
     "STRUCTURE_PRODUCTS",
     "CodrShape",
+    "CorpusGroup",
+    "codr_corpus",
     "codr_product",
     "codr_structures",
+    "codr_working_copy",
+    "corpus_embargoed",
+    "corpus_product_id",
     "synthetic_catalog",
     "temporary_catalog",
 ]
@@ -425,27 +432,29 @@ def codr_structures(catalog: Catalog, *, seed: int = 0, shape: CodrShape = _DEFA
     return dict(outcomes.sealed)
 
 
-def codr_product(
+def codr_working_copy(
     catalog: Catalog,
     product_id: str,
     *,
     tables: int = 1,
     seed: int = 0,
     shape: CodrShape = _DEFAULT_SHAPE,
-) -> Edition:
-    """Seal a CODR-shaped data product of *tables* tables into *catalog*, with its codelists cut to partials.
-
-    It seals the three structure products first (`codr_structures`) unless all
-    three are live. Each table ``n`` gets a data structure,
-    ``DSD_<product_id>_<n>`` (dimensions ``GEO`` and ``CLASS_1`` onward, then
-    ``TIME_PERIOD``; attributes ``UNIT_MEASURE`` and ``OBS_STATUS``; measure
-    ``OBS_VALUE``), a dataflow ``DF_<product_id>_<n>``, and
-    ``shape.observations`` rows over a share of each codelist's codes: 5 % of
-    the geography's leaves with their ancestors, a quarter of each
-    classification, two units and two statuses. `ingest_dataflows` fills the
-    product's working copy with ``partials=True``, and the seal records each
-    partial's ``partial_of``. No hierarchy association names a table, so
-    ``CL_GEO`` is cut. The search index takes the geography's codes with
+) -> WorkingCopy:
+    """Fill a CODR-shaped data product's working copy of *tables* tables, with its codelists cut to partials.
+
+    `codr_product` seals what this returns; a caller sealing several products
+    in one `Catalog.seal` takes the working copies from here. It seals the
+    three structure products first (`codr_structures`) unless all three are
+    live. Each table ``n`` gets a data structure, ``DSD_<product_id>_<n>``
+    (dimensions ``GEO`` and ``CLASS_1`` onward, then ``TIME_PERIOD``;
+    attributes ``UNIT_MEASURE`` and ``OBS_STATUS``; measure ``OBS_VALUE``), a
+    dataflow ``DF_<product_id>_<n>``, and ``shape.observations`` rows over a
+    share of each codelist's codes: 5 % of the geography's leaves with their
+    ancestors, a quarter of each classification, two units and two statuses.
+    `ingest_dataflows` fills the product's working copy with
+    ``partials=True``, so the seal records each partial's ``partial_of``. No
+    hierarchy association names a table, so ``CL_GEO`` is cut. When lancedb
+    is installed, the search index takes the geography's codes with
     ``CodelistIndexPolicy`` in ``fts_only`` mode.
 
     The structures depend on *seed* alone, and the rows on *seed* and
@@ -461,7 +470,7 @@ def codr_product(
         shape: The sizes.
 
     Returns:
-        The sealed edition.
+        The filled working copy, unsealed.
     """
     catalog.refresh()
     if not set(STRUCTURE_PRODUCTS) <= set(catalog.products()):
@@ -480,7 +489,154 @@ def codr_product(
         # The geography's codes go in the search index whatever their number,
         # without embeddings, as a CODR producer indexes them.
         wc.build_index(codelist_policies=[CodelistIndexPolicy(urns=(str(structures.geography.urn),))])
-    return wc.seal()
+    return wc
+
+
+def codr_product(
+    catalog: Catalog,
+    product_id: str,
+    *,
+    tables: int = 1,
+    seed: int = 0,
+    shape: CodrShape = _DEFAULT_SHAPE,
+) -> Edition:
+    """Seal a CODR-shaped data product of *tables* tables into *catalog*, with its codelists cut to partials.
+
+    `codr_working_copy` fills the working copy, and this seals it.
+
+    Args:
+        catalog: A writable catalog.
+        product_id: The data product.
+        tables: The tables, one dataflow each.
+        seed: Chooses the structures and the rows.
+        shape: The sizes.
+
+    Returns:
+        The sealed edition.
+    """
+    return codr_working_copy(catalog, product_id, tables=tables, seed=seed, shape=shape).seal()
+
+
+@frozen
+class CorpusGroup:
+    """One group of a `codr_corpus`: *products* data products of *tables* tables each.
+
+    Attributes:
+        products: The data products in the group.
+        tables: The tables of each, one dataflow each.
+    """
+
+    products: int
+    tables: int = 1
+
+    def __attrs_post_init__(self) -> None:
+        """Refuse a negative product count and a product of no table.
+
+        Raises:
+            ValueError: *products* is negative or *tables* is below 1.
+        """
+        if self.products < 0 or self.tables < 1:
+            msg = f"a corpus group holds zero or more products of one or more tables, not {self!r}"
+            raise ValueError(msg)
+
+
+def corpus_product_id(group: CorpusGroup, index: int) -> str:
+    """The id of the *index*-th product of *group*: ``codr-<tables>t-<index, five digits>``.
+
+    Args:
+        group: The group.
+        index: The product's place in the group, from 0.
+
+    Returns:
+        The product id.
+    """
+    return f"codr-{group.tables}t-{index:05d}"
+
+
+def corpus_embargoed(product_id: str, *, seed: int = 0, embargoed: float = 0.1) -> bool:
+    """Whether `codr_corpus` seals *product_id* under embargo: a seeded draw below the share *embargoed*.
+
+    Args:
+        product_id: The product.
+        seed: The corpus seed.
+        embargoed: The share of products sealed under embargo.
+
+    Returns:
+        Whether the product seals under embargo.
+    """
+    return _Draws(seed, f"embargo/{product_id}").below(1_000_000) < round(embargoed * 1_000_000)
+
+
+def _corpus_product(catalog: Catalog, product_id: str, *, tables: int, seed: int, shape: CodrShape) -> Edition:
+    """`codr_product`, after discarding an unsealed working copy a crash left."""
+    try:
+        return codr_product(catalog, product_id, tables=tables, seed=seed, shape=shape)
+    except WorkingCopyExistsError:
+        catalog.product(product_id).working_copy(resume=True).discard()
+        return codr_product(catalog, product_id, tables=tables, seed=seed, shape=shape)
+
+
+def codr_corpus(
+    catalog: Catalog,
+    groups: Sequence[CorpusGroup] = (CorpusGroup(7000),),
+    *,
+    seed: int = 0,
+    shape: CodrShape = _DEFAULT_SHAPE,
+    embargoed: float = 0.1,
+    on_sealed: Callable[[str, Edition], None] | None = None,
+) -> list[str]:
+    """Seal a corpus of CODR data products into *catalog*, with the structure products they are cut from.
+
+    Each group's products are named by `corpus_product_id` and built by
+    `codr_product` with *seed* and *shape*, so every product cuts its
+    codelists from the same structure editions, and each product's rows come
+    from its id. A product the Inventory records already, in any state, is
+    skipped, so a call with larger groups grows a corpus a smaller call
+    built, and a call after a crash resumes it, discarding the unsealed
+    working copy the crash left. A share *embargoed* of the products, chosen
+    by `corpus_embargoed`, seals with the catalog's embargo on, at the cost
+    of two ``embargo-change`` versions each, so the corpus holds live and
+    embargoed data entries. The embargo is off when the call returns.
+
+    Args:
+        catalog: A writable catalog.
+        groups: The groups, sealed in order.
+        seed: Chooses the structures and every product's rows.
+        shape: The sizes.
+        embargoed: The share of products sealed under embargo, from 0 to 1.
+        on_sealed: Called with each product id and edition after its seal.
+
+    Returns:
+        The products this call sealed, in order.
+
+    Raises:
+        ValueError: *embargoed* is outside 0 to 1.
+    """
+    if not 0 <= embargoed <= 1:
+        msg = f"embargoed is a share from 0 to 1, not {embargoed}"
+        raise ValueError(msg)
+    admin = CatalogAdmin.open(catalog._root)
+    if admin.embargo:
+        admin.set_embargo(on=False, reason="codr_corpus: seal the corpus with the embargo off")
+    recorded = set(admin.products())
+    sealed: list[str] = []
+    for group in groups:
+        for index in range(group.products):
+            product_id = corpus_product_id(group, index)
+            if product_id in recorded:
+                continue
+            under_embargo = corpus_embargoed(product_id, seed=seed, embargoed=embargoed)
+            if under_embargo:
+                admin.set_embargo(on=True, reason=f"codr_corpus: seal {product_id} under embargo")
+            try:
+                edition = _corpus_product(catalog, product_id, tables=group.tables, seed=seed, shape=shape)
+            finally:
+                if under_embargo:
+                    admin.set_embargo(on=False, reason=f"codr_corpus: {product_id} sealed")
+            sealed.append(product_id)
+            if on_sealed is not None:
+                on_sealed(product_id, edition)
+    return sealed
 
 
 def synthetic_catalog(path: str | Path, shape: Shape, *, product_id: str = "synthetic", seed: int = 0) -> Edition: