statcatlib 0.12.0__tar.gz → 0.13.0__tar.gz

{statcatlib-0.12.0 → statcatlib-0.13.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: statcatlib
-Version: 0.12.0
+Version: 0.13.0
 Summary: Queryable open data catalog engine for DCAT-AP, StatDCAT-AP, CKAN, and SDMX
 Keywords: sdmx,dcat,open-data,catalog,duckdb,statistical-data,ckan
 Author: gabrielgellner
@@ -18,7 +18,7 @@ Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
 Requires-Dist: attrs>=26.1.0
 Requires-Dist: duckdb>=2.0.0
 Requires-Dist: httpx>=0.28
-Requires-Dist: sdmxlib>=0.81.0,<0.82
+Requires-Dist: sdmxlib>=0.84,<0.85
 Requires-Dist: lancedb>=0.39 ; extra == 'search'
 Requires-Dist: sentence-transformers>=3.0 ; extra == 'search'
 Requires-Dist: typer>=0.15 ; extra == 'search'

{statcatlib-0.12.0 → statcatlib-0.13.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "statcatlib"
-version = "0.12.0"
+version = "0.13.0"
 description = "Queryable open data catalog engine for DCAT-AP, StatDCAT-AP, CKAN, and SDMX"
 license = "Apache-2.0"
 license-files = ["LICENSE"]
@@ -29,7 +29,7 @@ dependencies = [
     "attrs>=26.1.0",
     "duckdb>=2.0.0",
     "httpx>=0.28",
-    "sdmxlib>=0.81.0,<0.82",
+    "sdmxlib>=0.84,<0.85",
 ]
 
 [[project.authors]]

{statcatlib-0.12.0 → statcatlib-0.13.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "statcatlib"
-version = "0.12.0"
+version = "0.13.0"
 description = "Queryable open data catalog engine for DCAT-AP, StatDCAT-AP, CKAN, and SDMX"
 license = "Apache-2.0"
 license-files = ["LICENSE"]
@@ -27,9 +27,9 @@ dependencies = [
     # by DuckDB 1.x, so by statcatlib 0.8 or earlier.
     "duckdb>=2.0.0",
     "httpx>=0.28",
-    # <0.82: every suite library is pre-1.0, so a minor bump may break
+    # <0.85: every suite library is pre-1.0, so a minor bump may break
     # (GH-67).
-    "sdmxlib>=0.81.0,<0.82",
+    "sdmxlib>=0.84,<0.85",
 ]
 
 [project.optional-dependencies]

{statcatlib-0.12.0 → statcatlib-0.13.0}/src/statcatlib/EGM.md

@@ -5,9 +5,9 @@
 `statcatlib` is the package root. `__init__.py` re-exports the public API of
 `model`, `sources` and `store`; `CatalogAdmin` from `admin.py`; `Catalog`, `Product`, `Edition`,
 `SealOutcomes` and `NoLiveEditionError` from `catalog.py`; the errors `InventoryConflictError`,
-`InventoryContentionError`, `InventoryRuleError` and `NonFastForwardError`
-from `store/inventory.py`, and `WorkingCopyExistsError`,
-`WorkingCopyClosedError`, `SealCheckError` and `JournalError` from
+`InventoryContentionError`, `InventoryRuleError`, `NonFastForwardError` and
+`CrossProductRuleError` from `store/inventory.py`, and `WorkingCopyExistsError`,
+`WorkingCopyClosedError`, `SealCheckError`, `EditionCheckError` and `JournalError` from
 `store/working_copy.py`; `NotLoadedError` and `is_unloaded` from the store's
 sentinel module; and part of `expr` (listed below). It exposes `checks`,
 `query` and `urn` as submodules (`sc.checks`, `sc.query`, `sc.urn`). The
@@ -57,11 +57,25 @@ and test catalogs (`testing.py`).
   - `seal(working_copies, *, checks=()) -> SealOutcomes`: the batched seal.
     It refuses, with `ValueError`, a working copy of another catalog or two
     of one product, and raises `WorkingCopyClosedError` for a closed one,
-    before sealing anything. Then it runs `WorkingCopy._precheck` (journal,
-    checks, fast-forward) for every working copy, `_seal_edition` for each
-    that passed, and records every sealed edition in one
-    `InventoryWriter.commit`, leaving out each product whose `_next_leaf`
-    raises `InventoryRuleError`. `RuntimeError` on a catalog from `open`.
+    before sealing anything. `_seal_each` then runs the structure pass
+    (`_seal_ready`: `WorkingCopy._precheck(checks, batch=)` for each
+    structure working copy, whose journal repair leaves the batch's other
+    products out; `_cross_ready`, the cross-product pre-check of the pass's
+    working copies together through `precheck_refusals`; then
+    `_seal_edition` for each), the structure-check pass
+    (`_check_structures`: each structure edition's `_publish_check`, given
+    the batch's other structure editions, repeated until it drops none) and
+    the data pass (the same `_seal_ready`, its cross-product pre-check given
+    the checked structure editions as `Planned` records (`_fixed`), then
+    `_check_partials`: the `_publish_check` of each data edition whose
+    partials record a `partial_of`, given every structure product's editions
+    with the batch's structure editions in their place). `_record_batch`
+    records every sealed edition in one `InventoryWriter.commit`;
+    `_batch_leaves` leaves out each product whose `_next_leaf` raises
+    `InventoryRuleError`, each the cross-product rules refuse, and each
+    edition whose publish check was given a left-out one (`_drop_dependents`),
+    running the rules again until a pass drops none, and counts an edition a
+    journal repair recorded already as recorded. `RuntimeError` on a catalog from `open`.
   - `search` and `facets` raise `NotImplementedError` naming
     `product(id).view()`. `__getattr__` raises `AttributeError` naming
     `cat.product(id).view().<name>` for every name in
@@ -69,8 +83,9 @@ and test catalogs (`testing.py`).
 - `Product`: `product_id`; `view(*, embedding_model=None, query_prefix="")
   -> EditionView`; `editions() -> list[Edition]` (the live head, with the
   `previous` its manifest records; empty without one); `working_copy(*,
-  lifecycle="revisable", resume=False, embedding_model=None,
-  document_prefix="", query_prefix="") -> WorkingCopy`; `rollback()`, which
+  kind=None, lifecycle="revisable", resume=False, embedding_model=None,
+  document_prefix="", query_prefix="") -> WorkingCopy` (`kind` is `"data"`
+  or `"structure"`; `None` takes the recorded one, or `"data"`); `rollback()`, which
   raises `AttributeError` naming `CatalogAdmin.rollback`.
   `Catalog.product` is the only constructor.
 - `NoLiveEditionError` (`LookupError`): `view()` on a product with no live
@@ -78,7 +93,9 @@ and test catalogs (`testing.py`).
 - `SealOutcomes`: a frozen attrs value with `sealed` (product id to
   `Edition`, all recorded in one version) and `refused` (product id to the
   error: `SealCheckError`, `NonFastForwardError` or another
-  `InventoryRuleError`, `ManifestError`, or `InventoryContentionError`).
+  `InventoryRuleError` (`CrossProductRuleError` from the pre-check or the
+  record included), `ManifestError`, `EditionCheckError`, or
+  `InventoryContentionError`).
 - `Edition` is re-exported from `store/working_copy.py`.
 
 ### `admin.py`
@@ -111,13 +128,18 @@ are imported from `statcatlib.admin`.
   *, kind, reason)`, which applies one of sdmxlib's pure rules to the head's
   leaf with `datetime.now(UTC)` (the local clock, design decision P1) and
   wraps a `TransitionError` with `InventoryRuleError.wrapping`. A product the
-  head does not record raises `InventoryRuleError(rule="unknown")`.
+  head does not record raises `InventoryRuleError(rule="unknown")`. The
+  writer's `commit` runs the cross-product rules on a release or a rollback
+  and raises `CrossProductRuleError`; `admin.py` adds no rule of its own.
 - `prune(*, older_than, reason, dry_run=True) -> PruneReport` (design
   decision P7): refuses a blank *reason* or a window that is not positive
   (`ValueError`). It skips every product `_busy` names (an unsealed edition
   whose id is not tombstoned, a `journal/*.json` file, or an unreadable
   manifest), then runs three steps. `_prune_retire` retires superseded and
-  withdrawn entries whose `state_changed_at` is before the cutoff with
+  withdrawn entries whose `state_changed_at` is before the cutoff, except
+  the structure editions `_cited` returns (those a live or embargoed data
+  edition's `partial_of` cites, `cross_product.cited_structures`; every
+  structure edition when a manifest cannot be read), with
   sdmxlib's `retire`, in one `InventoryWriter.commit` of kind `retire`
   (`Change(leaves=..., retired=head tombstones + new)`), re-planned against
   each head and skipped when nothing qualifies. `_prune_dirs` deletes
@@ -150,16 +172,22 @@ are imported from `statcatlib.admin`.
 The module keeps its 0.11 name and describes editions. Imported from
 `statcatlib.bundle`; nothing here is re-exported from `statcatlib`:
 
-- `catalog_section(store: _Store) -> dict`: builds the `statcat` manifest
-  section (format 3: `format`, `kind`, `statcatlib_version`, `datasets`,
-  `search`) from a working copy's open store. `WorkingCopy.seal` passes it to
+- `catalog_section(store: _Store, *, kind="data") -> dict`: builds the
+  `statcat` manifest section (format 3: `format`, `kind`,
+  `statcatlib_version`, `datasets`, `search`) from a working copy's open
+  store. `WorkingCopy.seal` passes it to
   `sdmxlib.edition.seal(..., extensions={"statcat": ...})`. It raises
   `ValueError` when `search/` holds files and no `index.json`.
 - `check_catalog(root, manifest, reg, profile) -> list[Finding]`: the
   `statcat` check, with sdmxlib 0.80's `Check` signature, run as an extension
   of `sdmxlib.edition.check_edition`. It refuses a section format other than
   3, a `kind` outside `PRODUCT_KINDS`, and a `structure` edition whose
-  manifest binds a dataflow.
+  manifest binds a dataflow or whose store holds a dataflow, binding or
+  partial.
+- `structure_content(conn) -> list[Finding]`: one finding per `Dataflow`,
+  `DataflowBinding` and partial artefact in `sdmx.artefact`. `check_catalog`
+  runs it on a structure edition, and `WorkingCopy._precheck` on a
+  structure working copy's store.
 - `overlay_affects_search(overlay, types=None)`: lists overlay entries that
   change search results.
 - `description_to_json(description) -> dict[str, object]` and
@@ -172,7 +200,7 @@ The module keeps its 0.11 name and describes editions. Imported from
   of an Inventory root, or returns `None`.
 - `ROOT_SECTION_FORMAT = "3"`: the root section's `format_version`.
 - `STATCAT_SECTION = "statcat"`, `STATCAT_SECTION_FORMAT = 3`,
-  `PRODUCT_KINDS = ("data", "structure")`. Phase 2a seals `data` only.
+  `PRODUCT_KINDS = ("data", "structure")`.
 
 ### `diagnostics.py`
 
@@ -184,16 +212,27 @@ read.
   root the read protocol skipped (`full.skipped`), each leaf that fails its
   hash or `validate_leaf`, and for every leaf entry a `manifest.json` whose
   SHA-256 differs from the entry's, then runs
-  `check_edition(at="publish", extensions={"statcat": check_catalog})`.
+  `check_edition(at="publish", structures=..., extensions={"statcat":
+  check_catalog})`, with `structures=` from `_structures_for`: every recorded
+  edition of every structure product for a data edition, the other
+  structure products' live and embargoed editions for a structure edition,
+  `None` when empty. Last it lists the stale data editions
+  (`cross_product.stale_editions`, re-exported here as `StaleEdition`).
   `VerifyReport` has `root`, `seq` (`None` before the first version),
-  `editions` and `findings` (`VerifyFinding(file, detail)`, the file a path
-  from the catalog directory) and `ok`. A path that is no directory, a legacy
+  `editions`, `findings` (`VerifyFinding(file, detail)`, the file a path
+  from the catalog directory), `stale` (no finding; `ok` ignores it) and
+  `ok`. A path that is no directory, a legacy
   layout, a missing or malformed `catalog.json`, or no valid root raise
   `CatalogUnreadableError`.
 - `doctor(path) -> DoctorReport`: `layout` (`catalog`, `legacy`, `unknown`)
   and `items`, each a `DoctorItem(topic, level, detail)` with level `info` or
   `warning`; `warnings` and `about(topic)` filter them. Topics: `synced
-  folder`, `layout`, `catalog.json`, `CURRENT`, `inventory`, `working copy`
+  folder`, `layout`, `catalog.json`, `CURRENT`, `inventory`, `stale data
+  editions` (an item per stale data edition and URN: `info` for a live
+  edition, `warning` for an embargoed one), `partials with no structure
+  home` (a `warning` per live or embargoed data edition and partial whose
+  URN no live or embargoed structure edition holds full,
+  `cross_product.homeless_partials`), `working copy`
   (`list_editions`), `journal` (empty entries included), `unrecorded edition`
   (sealed, and neither named nor tombstoned by the newest version) and
   `leftover file` (`*.tmp` in `inventory/` and `.*.tmp` in the root).
@@ -242,6 +281,24 @@ read.
   product's directory, resumes it as a working copy, stores one
   `AggregateDataset` per dataflow (`prepare_sdmx(wc.registry)`), and seals.
   The seal builds the search index when `statcatlib[search]` is installed.
+- The CODR generator (design decision O6): `CodrShape(geography_codes=6000,
+  geography_depth=5, classification_codes=(900, 500, 300),
+  observations=2000)`; `STRUCTURE_PRODUCTS` (`structure-geo`,
+  `structure-classifications`, `structure-concepts`); `CODR_AGENCY`;
+  `codr_structures(catalog, *, seed=0, shape=...) -> dict[str, Edition]`,
+  which builds the structures from the seed alone (agency `CODR`, version
+  `1.0`, names `"<id> en"` and `"<id> fr"`) into a scratch `LocalRegistry`,
+  fills the three structure working copies by URN and seals them in one
+  `Catalog.seal`, raising `RuntimeError` on a refusal; and `codr_product(catalog,
+  product_id, *, tables=1, seed=0, shape=...) -> Edition`, which seals the
+  structure products first unless all three are live, writes one DSD and
+  dataflow per table and the rows from `(seed, product_id)`, fills with
+  `ingest_dataflows(..., partials=True)`, indexes `CL_GEO`'s codes with an
+  `fts_only` `CodelistIndexPolicy` when lancedb is installed, and seals.
+  `structure-classifications` holds `CL_CLASS_*`, `CL_UNIT` and the
+  SDMX-agency `CL_OBS_STATUS`, so every codelist a table cuts has a
+  structure home (A5). The draws are SplitMix64 seeded by SHA-256, so they
+  do not depend on Python's `random`.
 
 ## Invariants
 
@@ -275,14 +332,17 @@ read.
   lands on: a rollback whose `if_live` is no longer live, a release that is
   no longer a fast-forward, and a seal record computed before the embargo was
   turned on are each refused or recomputed.
-- `Product.view` hashes the live edition's `manifest.json` and raises
-  `IncompatibleCatalogError` when it differs from the `manifest_sha256` the
-  Inventory records. Every view a `Catalog` returns shares one
+- `Product.view` passes the Inventory's `manifest_sha256` to
+  `sdmxlib.LocalRegistry.edition`, which hashes the bytes of
+  `manifest.json` it parses. A missing, malformed or different manifest
+  raises sdmxlib's `ManifestError`, which `view` re-raises as
+  `IncompatibleCatalogError` naming `statcat verify-catalog`, chained. Every view a `Catalog` returns shares one
   `sdmxlib.EditionEngine`, and `Catalog.close` closes them all.
 - A seal check only reads: `seal` runs every check against `wc.view()`
   before writing anything, and any finding raises `SealCheckError`.
-- `Catalog.seal` runs every pre-check before any sdmxlib seal, and writes at
-  most one Inventory version; a batch whose products are all refused writes
+- `Catalog.seal` seals and checks the batch's structure products before
+  any data product, runs every data pre-check before any data seal, and
+  writes at most one Inventory version; a batch whose products are all refused writes
   none. A refusal in one product never stops another's. A crash during the
   batch leaves each sealed, unrecorded edition's journal entry, so its
   product's next `working_copy()` records it, in a version of its own.
@@ -300,7 +360,7 @@ read.
   at INFO, and writes and removes a `.statcat-link-probe.*` file and its hard
   link.
 - `Product.view` attaches the edition to the catalog's engine
-  (`LocalRegistry.edition`). `Product.working_copy` writes as
+  (`LocalRegistry.edition(path, engine=, manifest_sha256=)`). `Product.working_copy` writes as
   `store/EGM.md` describes.
 - `synthetic_catalog` writes a whole catalog: `catalog.json`, sdmxlib's
   synthetic edition, the catalog rows, the search index when lancedb is

{statcatlib-0.12.0 → statcatlib-0.13.0}/src/statcatlib/__init__.py

@@ -35,6 +35,7 @@ from statcatlib.sources import (
     dataset_from_dataflow,
     ingest_ckan,
     ingest_data,
+    ingest_dataflows,
     ingest_sdmx,
     ingest_sdmx_registry,
     prepare_sdmx,
@@ -88,6 +89,7 @@ from statcatlib.store import (
     xhtml_to_markdown,
 )
 from statcatlib.store.inventory import (
+    CrossProductRuleError,
     InventoryConflictError,
     InventoryContentionError,
     InventoryRuleError,
@@ -95,6 +97,7 @@ from statcatlib.store.inventory import (
 )
 from statcatlib.store.sentinel import NotLoadedError, is_unloaded
 from statcatlib.store.working_copy import (
+    EditionCheckError,
     JournalError,
     SealCheckError,
     WorkingCopyClosedError,
@@ -131,6 +134,7 @@ __all__ = [
     "ConceptsScope",
     "ContactPoint",
     "CountExpr",
+    "CrossProductRuleError",
     "DataService",
     "Dataset",
     "DatasetScope",
@@ -141,6 +145,7 @@ __all__ = [
     "DimensionsScope",
     "Distribution",
     "Edition",
+    "EditionCheckError",
     "EditionView",
     "EmbeddingMetadata",
     "ExistsExpr",
@@ -189,6 +194,7 @@ __all__ = [
     "each",
     "ingest_ckan",
     "ingest_data",
+    "ingest_dataflows",
     "ingest_sdmx",
     "ingest_sdmx_registry",
     "is_concept_in_scheme",

{statcatlib-0.12.0 → statcatlib-0.13.0}/src/statcatlib/admin.py

@@ -24,7 +24,14 @@ skewed times.
 2a S4). It also deletes files: the directories of editions tombstoned longer
 than its window, and ``data/`` files no manifest on disk lists. It never
 touches a product with a working copy or a seal journal entry, and deletes no
-Inventory root or leaf.
+Inventory root or leaf. It keeps a structure edition that a live or embargoed
+data edition's ``partial_of`` cites (specification §11.4), so a stale data
+edition's partials still verify.
+
+`CatalogAdmin.release` and `CatalogAdmin.rollback` make an edition live, so
+the Inventory writer runs the cross-product rules on their version
+(`statcatlib.store.cross_product`): a refusal raises `CrossProductRuleError`
+and writes nothing.
 """
 
 from __future__ import annotations
@@ -42,6 +49,7 @@ from sdmxlib.storage.schema import SCHEMA_VERSION
 
 from statcatlib.bundle import ROOT_SECTION_FORMAT, STATCAT_SECTION, description_to_json
 from statcatlib.catalog import _refuse_legacy_layout
+from statcatlib.store.cross_product import cited_structures
 from statcatlib.store.inventory import Change, InventoryRuleError, InventoryWriter, NonFastForwardError
 
 if TYPE_CHECKING:
@@ -146,8 +154,21 @@ def _busy(product_dir: Path, tombstoned: Collection[str]) -> str | None:
     return None
 
 
-def _retirable(full: FullInventory | None, cutoff: datetime, skipped: Collection[str]) -> dict[str, list[str]]:
-    """The superseded and withdrawn editions that entered their state before *cutoff*, by product."""
+def _retirable(
+    full: FullInventory | None, cutoff: datetime, skipped: Collection[str], kept: Collection[tuple[str, str]]
+) -> dict[str, list[str]]:
+    """The superseded and withdrawn editions that entered their state before *cutoff*, by product.
+
+    Args:
+        full: The head, or ``None``.
+        cutoff: The window's start; older entries retire.
+        skipped: The products prune leaves alone.
+        kept: ``(product, edition)`` pairs to keep: the structure editions
+            a live or embargoed data edition's ``partial_of`` cites.
+
+    Returns:
+        The edition ids to retire, by product.
+    """
     if full is None:
         return {}
     found: dict[str, list[str]] = {}
@@ -157,7 +178,9 @@ def _retirable(full: FullInventory | None, cutoff: datetime, skipped: Collection
         ids = [
             e.edition_id
             for e in full.entries(product_id)
-            if e.state in {"superseded", "withdrawn"} and e.state_changed_at < cutoff
+            if e.state in {"superseded", "withdrawn"}
+            and e.state_changed_at < cutoff
+            and (product_id, e.edition_id) not in kept
         ]
         if ids:
             found[product_id] = ids
@@ -331,6 +354,13 @@ class CatalogAdmin:
                 other than *if_live* (``rule="if-live"``), or *to* is not a
                 superseded edition (``rule="state"`` or ``"unknown"``).
 
+        An edition made live that breaks a cross-product rule raises
+        `CrossProductRuleError` and writes nothing: a data edition cut from a
+        structure edition that is neither live nor embargoed
+        (``rule="full-artefacts"``), or a structure edition holding a full
+        artefact another structure product holds live
+        (``rule="structure-home"``).
+
         An empty *reason* raises `ValueError`. It acts on the live edition at
         this sdmxlib's store schema; rolling back another schema's live
         edition arrives in Phase 4.
@@ -364,6 +394,15 @@ class CatalogAdmin:
         Returns:
             The Inventory version written (``event.kind`` ``release``).
 
+        A release that breaks a cross-product rule raises
+        `CrossProductRuleError` and writes nothing: a data edition whose
+        partial or full artefact matches only an embargoed structure edition
+        (``rule="publish-order"``; release that one first); a stale data
+        edition, or one whose full artefact with no structure home differs
+        from another data product's live copy (``rule="full-artefacts"``); a
+        structure edition holding a full artefact another structure product
+        holds live (``rule="structure-home"``).
+
         An edition built on an edition that is no longer live raises
         `NonFastForwardError`, whose message names both and the fix. An edition
         that is not recorded (``rule="unknown"``) or not embargoed
@@ -498,7 +537,11 @@ class CatalogAdmin:
         so is one with an edition whose manifest cannot be read. Prune
         deletes no Inventory root or leaf, never expires a tombstone, and
         never deletes an edition the Inventory does not tombstone, so a
-        live, embargoed or unrecorded edition stays. An edition retired in
+        live, embargoed or unrecorded edition stays. It never retires a
+        structure edition that a live or embargoed data edition's
+        ``partial_of`` cites (specification §11.4); when a manifest that
+        decision needs cannot be read, it retires no structure edition. An
+        edition retired in
         step 1 keeps its directory until a run after its tombstone has aged
         past the window.
 
@@ -596,12 +639,12 @@ class CatalogAdmin:
         """
         stones: tuple[Tombstone, ...] = () if full is None else full.root.retired
         if dry_run:
-            planned = _retirable(full, cutoff, skipped)
+            planned = _retirable(full, cutoff, skipped, self._cited(full))
             return [f"{p}/{e}" for p, ids in planned.items() for e in ids], None, stones
         retired: list[str] = []
 
         def change(head_now: FullInventory | None) -> Change:
-            planned = _retirable(head_now, cutoff, skipped)
+            planned = _retirable(head_now, cutoff, skipped, self._cited(head_now))
             if head_now is None or not planned:
                 raise _NothingToRetireError
             now = datetime.now(UTC)
@@ -622,6 +665,22 @@ class CatalogAdmin:
             return [], None, stones
         return retired, written.seq, written.retired
 
+    def _cited(self, full: FullInventory | None) -> set[tuple[str, str]]:
+        """The structure editions prune keeps: those a live or embargoed data edition's ``partial_of`` cites.
+
+        When a manifest the answer needs cannot be read, every structure
+        edition is kept.
+        """
+        try:
+            return cited_structures(full, self._writer.artefacts)
+        except ManifestError as exc:
+            log.warning("prune keeps every structure edition: a manifest cannot be read (%s)", exc)
+            if full is None:
+                return set()
+            return {
+                (p, e.edition_id) for p in full.products() if full.leaf(p).kind == "structure" for e in full.entries(p)
+            }
+
     def _prune_data(
         self, product_dir: Path, cutoff: datetime, gone: Collection[str], *, dry_run: bool, failed: dict[str, str]
     ) -> list[str]:

{statcatlib-0.12.0 → statcatlib-0.13.0}/src/statcatlib/bundle.py

@@ -39,6 +39,7 @@ if TYPE_CHECKING:
     import duckdb
     from sdmxlib import EditionRegistry
     from sdmxlib.edition import Profile
+    from sdmxlib.inventory import Kind
 
     from statcatlib.store.lance import LanceIndex, LanguageLayout
     from statcatlib.store.store import _Store
@@ -54,6 +55,7 @@ __all__ = [
     "description_from_root",
     "description_to_json",
     "overlay_affects_search",
+    "structure_content",
 ]
 
 #: The manifest's extension section statcatlib owns.
@@ -64,9 +66,9 @@ STATCAT_SECTION = "statcat"
 #: 3 added ``kind``, the product's kind (Phase 2a).
 STATCAT_SECTION_FORMAT = 3
 
-#: The product kinds format 3 names. Phase 2a seals ``data`` products only;
-#: ``structure`` is reserved for structure products (Phase 3), whose editions
-#: hold no observations, so `check_catalog` refuses a binding in one.
+#: The product kinds format 3 names. A ``structure`` product's editions hold
+#: full artefacts and no dataflow, binding or partial (`structure_content`), so
+#: `check_catalog` refuses each of those, and a manifest binding, in one.
 PRODUCT_KINDS: tuple[str, ...] = ("data", "structure")
 
 _SEARCH_DIRNAME = "search"
@@ -108,11 +110,10 @@ def _search_section(store: _Store, search_dir: Path) -> dict[str, object] | None
     }
 
 
-def catalog_section(store: _Store) -> dict[str, object]:
+def catalog_section(store: _Store, *, kind: Kind = "data") -> dict[str, object]:
     """Describe a working copy's edition, as its manifest's ``statcat`` section (format 3).
 
-    The section records the product's kind (``data``; Phase 2a seals no other),
-    the number of catalog datasets and, when a search index is built, its
+    The section records the product's kind, the number of catalog datasets and, when a search index is built, its
     index format version, embedding model, each language's row count and
     stemmer, and the annotations it was built from. `WorkingCopy.seal`
     computes it after rebuilding the index and before closing the store, and
@@ -120,6 +121,7 @@ def catalog_section(store: _Store) -> dict[str, object]:
 
     Args:
         store: The working copy's store, open.
+        kind: The product's kind, ``data`` or ``structure``.
 
     Returns:
         The section, ready for ``seal(extensions=...)``.
@@ -131,13 +133,42 @@ def catalog_section(store: _Store) -> dict[str, object]:
     search_dir = store._search_dir
     return {
         "format": STATCAT_SECTION_FORMAT,
-        "kind": "data",
+        "kind": kind,
         "statcatlib_version": metadata.version("statcatlib"),
         "datasets": row[0] if row else 0,
         "search": _search_section(store, search_dir) if search_dir is not None else None,
     }
 
 
+def structure_content(conn: duckdb.DuckDBPyConnection) -> list[Finding]:
+    """What a structure product's edition may not hold: each dataflow, binding and partial in its store.
+
+    A structure product holds full artefacts only. `check_catalog` runs this
+    on a sealed structure edition, and `WorkingCopy.seal` on a structure
+    working copy before anything is sealed.
+
+    Args:
+        conn: A connection whose current catalog is the edition's, so
+            ``sdmx.artefact`` names its table.
+
+    Returns:
+        One finding per such artefact, by URN.
+    """
+    rows = conn.execute(
+        """
+        SELECT urn, artefact_type, is_partial FROM sdmx.artefact
+        WHERE artefact_type IN ('Dataflow', 'DataflowBinding') OR is_partial
+        ORDER BY urn
+        """
+    ).fetchall()
+    findings: list[Finding] = []
+    for urn, artefact_type, is_partial in rows:
+        held = f"a partial {artefact_type}" if is_partial else f"a {artefact_type}"
+        detail = f"{held}, which a structure product does not hold: it holds full artefacts only"
+        findings.append(Finding(STATCAT_SECTION, urn, detail))
+    return findings
+
+
 def _check_catalog_rows(conn: duckdb.DuckDBPyConnection, section: Mapping[str, object]) -> Iterator[Finding]:
     """The catalog's datasets: counted as declared, each linked flow stored."""
     from statcatlib.store.schema import first_row  # noqa: PLC0415
@@ -244,7 +275,8 @@ def check_catalog(root: Path, manifest: Manifest, reg: EditionRegistry, profile:
       reads, and the manifest has a ``statcat`` section in a format this
       build reads;
     - the section's ``kind`` is one of `PRODUCT_KINDS`, and a ``structure``
-      edition binds no dataflow to observations;
+      edition binds no dataflow to observations and holds no dataflow,
+      binding or partial (`structure_content`);
     - the catalog holds the datasets the section counts, and every dataset
       linked to a dataflow names one the registry holds;
     - ``search/`` is present exactly when the section declares an index;
@@ -299,6 +331,7 @@ def check_catalog(root: Path, manifest: Manifest, reg: EditionRegistry, profile:
             Finding(STATCAT_SECTION, binding.dataflow, "a structure product's edition binds a dataflow to observations")
             for binding in manifest.bindings
         ]
+        findings += structure_content(conn)
     findings += [*_check_catalog_rows(conn, section), *_check_search(root, section.get("search"))]
     search = section.get("search")
     if isinstance(search, Mapping):