statcatlib 0.16.2__tar.gz → 0.17.0__tar.gz

{statcatlib-0.16.2 → statcatlib-0.17.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: statcatlib
-Version: 0.16.2
+Version: 0.17.0
 Summary: Queryable open data catalog engine for DCAT-AP, StatDCAT-AP, CKAN, and SDMX
 Keywords: sdmx,dcat,open-data,catalog,duckdb,statistical-data,ckan
 Author: gabrielgellner
@@ -18,7 +18,7 @@ Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
 Requires-Dist: attrs>=26.1.0
 Requires-Dist: duckdb>=2.0.0
 Requires-Dist: httpx>=0.28
-Requires-Dist: sdmxlib>=0.87,<0.88
+Requires-Dist: sdmxlib>=0.88,<0.89
 Requires-Dist: obstore>=0.12,<0.13 ; extra == 'blob'
 Requires-Dist: azure-identity>=1 ; extra == 'blob'
 Requires-Dist: typer>=0.15 ; extra == 'blob'

{statcatlib-0.16.2 → statcatlib-0.17.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "statcatlib"
-version = "0.16.2"
+version = "0.17.0"
 description = "Queryable open data catalog engine for DCAT-AP, StatDCAT-AP, CKAN, and SDMX"
 license = "Apache-2.0"
 license-files = ["LICENSE"]
@@ -29,7 +29,7 @@ dependencies = [
     "attrs>=26.1.0",
     "duckdb>=2.0.0",
     "httpx>=0.28",
-    "sdmxlib>=0.87,<0.88",
+    "sdmxlib>=0.88,<0.89",
 ]
 
 [[project.authors]]

{statcatlib-0.16.2 → statcatlib-0.17.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "statcatlib"
-version = "0.16.2"
+version = "0.17.0"
 description = "Queryable open data catalog engine for DCAT-AP, StatDCAT-AP, CKAN, and SDMX"
 license = "Apache-2.0"
 license-files = ["LICENSE"]
@@ -27,9 +27,9 @@ dependencies = [
     # by DuckDB 1.x, so by statcatlib 0.8 or earlier.
     "duckdb>=2.0.0",
     "httpx>=0.28",
-    # <0.88: every suite library is pre-1.0, so a minor bump may break
+    # <0.89: every suite library is pre-1.0, so a minor bump may break
     # (GH-67).
-    "sdmxlib>=0.87,<0.88",
+    "sdmxlib>=0.88,<0.89",
 ]
 
 [project.optional-dependencies]

{statcatlib-0.16.2 → statcatlib-0.17.0}/src/statcatlib/EGM.md

@@ -3,8 +3,8 @@
 ## Overview
 
 `statcatlib` is the package root. `__init__.py` re-exports the public API of
-`model`, `sources` and `store`; `CatalogAdmin` from `admin.py`; `Catalog`, `Product`, `Edition`,
-`SealOutcomes` and `NoLiveEditionError` from `catalog.py`; the errors `InventoryConflictError`,
+`model`, `sources` and `store`; `CatalogAdmin` from `admin.py`; `EmbargoReader` from `embargo.py`;
+`Catalog`, `Product`, `Edition`, `SealOutcomes` and `NoLiveEditionError` from `catalog.py`; the errors `InventoryConflictError`,
 `InventoryContentionError`, `InventoryRuleError`, `InventoryWriteRefusedError`,
 `NonFastForwardError` and `CrossProductRuleError` from `store/inventory.py`; `BlobLocation` and `TokenIdentity`
 from `store/blob.py`; `upload_edition`, `UploadReport` and `UploadCheckError` from `upload.py`; `WorkingCopyExistsError`,
@@ -14,10 +14,11 @@ sentinel module; and part of `expr` (listed below). It exposes `checks`,
 `query` and `urn` as submodules (`sc.checks`, `sc.query`, `sc.urn`). The
 top-level modules hold the parts that sit across subpackages: the catalog of
 products (`catalog.py`), the operational changes to its Inventory
-(`admin.py`), seal checks (`checks.py`), the `statcat` section of
+(`admin.py`), the reader of embargoed editions (`embargo.py`), seal checks (`checks.py`), the `statcat` section of
 an sdmxlib edition manifest (`bundle.py`), the integrity and support
 checks behind `statcat verify-catalog` and `statcat doctor`
-(`diagnostics.py`), the copy of a sealed edition to object storage (`upload.py`), composable query filters
+(`diagnostics.py`), the copy of a sealed edition to object storage (`upload.py`), the
+jobs configuration the release and GC jobs read (`jobs.py`), the GC job (`gc.py`), composable query filters
 (`query.py`), traversal expressions (`expr.py`), SDMX URN builders (`urn.py`)
 and test catalogs (`testing.py`).
 
@@ -88,7 +89,8 @@ and test catalogs (`testing.py`).
     `cat.product(id).view().<name>` for every name in
     `store.store.VIEW_ATTRIBUTES`.
 - `Product`: `product_id`; `view(*, embedding_model=None, query_prefix="")
-  -> EditionView`; `editions() -> list[Edition]` (the live head, with the
+  -> EditionView`, which calls `_open_view` with the live head and the stamps
+  directory `<root>/verified`; `editions() -> list[Edition]` (the live head, with the
   `previous` its manifest records; empty without one); `working_copy(*,
   kind=None, lifecycle="revisable", resume=False, embedding_model=None,
   document_prefix="", query_prefix="") -> WorkingCopy` (`kind` is `"data"`
@@ -97,6 +99,24 @@ and test catalogs (`testing.py`).
   `Catalog.product` is the only constructor.
 - `NoLiveEditionError` (`LookupError`): `view()` on a product with no live
   edition.
+- `_open_read_only(path, *, opener) -> _Opening`: `Catalog.open`'s checks and
+  its `_Opening`, shared with `EmbargoReader.open`; *opener* names the call in
+  messages.
+- `_open_view(engine, root, location, product_id, edition_id, *,
+  manifest_sha256, stamps, state, embedding_model, query_prefix) ->
+  tuple[EditionView, EditionRegistry, _Store]`: the one place an `EditionView`
+  of a recorded edition is built, shared by `Product.view` and
+  `EmbargoReader.view` (Phase 5 D12). *engine* is a callable that builds the
+  caller's engine on first use, so an edition refused before it opens builds none.
+  On local disk it attaches
+  `<root>/products/<product>/<edition>`; over a `BlobLocation`, `_checked_copy`
+  runs `_cached_edition` (the manifest first, then `pull_edition`), refuses an
+  edition with data files when the location holds no `secrets` (`ValueError`),
+  and runs `check_on_open(stamps=, references=bool(manifest.bindings),
+  extensions={"statcat": ...})`. Then `LocalRegistry.edition(...,
+  manifest_sha256=)`, its `ManifestError` re-raised as `IncompatibleCatalogError`,
+  and a read-only `_Store`. The caller keeps the registry and the store to close.
+  *state* names the entry's state in messages.
 - `SealOutcomes`: a frozen attrs value with `sealed` (product id to
   `Edition`, all recorded in one version) and `refused` (product id to the
   error: `SealCheckError`, `NonFastForwardError` or another
@@ -110,11 +130,15 @@ and test catalogs (`testing.py`).
 `CatalogAdmin` is exported from `statcatlib`; `EditionEntry` and `PruneReport`
 are imported from `statcatlib.admin`.
 
-- `CatalogAdmin.open(path)`: refuses a legacy layout
+- `CatalogAdmin.open(path, *, actor=None, clock=None)`: *path* is a catalog
+  directory or a `BlobLocation`. A directory: refuses a legacy layout
   (`catalog._refuse_legacy_layout`) and a path with no `catalog.json`
-  (`FileNotFoundError`), reads `catalog.json`, and holds an
-  `InventoryWriter`. It writes nothing until a change is made. Direct
-  construction raises `TypeError`.
+  (`FileNotFoundError`), and refuses `actor=` or `clock=` with `TypeError`. A
+  `BlobLocation`: builds `BlobInventoryStorage(location, actor=, clock=)`, and a
+  store with no `catalog.json` raises `FileNotFoundError`. Either way it reads
+  `catalog.json` and holds an `InventoryWriter`, and writes nothing until a
+  change is made. `repr` shows `storage.location`. Direct construction raises
+  `TypeError`.
 - Reads, each from a fresh `InventoryWriter.full()`: `embargo`,
   `products()` (every recorded product, any state) and
   `editions(product_id) -> list[EditionEntry]` (every entry the newest version
@@ -131,25 +155,28 @@ are imported from `statcatlib.admin`.
   `embargo-change`) and `describe(description, *, reason)` (sets
   `format_version` `"3"` and `description` in the root's `extensions.statcat`,
   keeping its other keys; kind `describe`).
+- `advance_current() -> int | None`: `InventoryWriter.advance_current`; it
+  writes `CURRENT` only.
 - Product changes go through `_commit_product(product_id, edition_id, step,
-  *, kind, reason)`, which applies one of sdmxlib's pure rules to the head's
-  leaf with `datetime.now(UTC)` (the local clock, design decision P1) and
-  wraps a `TransitionError` with `InventoryRuleError.wrapping`. A product the
+  *, kind, reason)`, which reads `storage.now()` once per head (the local
+  clock on disk, design decision P1; the storage clock on blob), applies one of
+  sdmxlib's pure rules to the head's leaf with it, returns `Change(now=)` with
+  the same time, and wraps a `TransitionError` with `InventoryRuleError.wrapping`. A product the
   head does not record raises `InventoryRuleError(rule="unknown")`. The
   writer's `commit` runs the cross-product rules on a release or a rollback
   and raises `CrossProductRuleError`; `admin.py` adds no rule of its own.
 - `prune(*, older_than, reason, dry_run=True) -> PruneReport` (design
-  decision P7): refuses a blank *reason* or a window that is not positive
-  (`ValueError`). It skips every product `_busy` names (an unsealed edition
+  decision P7): raises `TypeError` on a storage other than
+  `LocalInventoryStorage`, naming `statcat gc` and the release job; refuses a
+  blank *reason* or a window that is not positive (`ValueError`). Its cutoff
+  and tombstone time are `statcatlib.admin.datetime.now(UTC)`. It skips every product `_busy` names (an unsealed edition
   whose id is not tombstoned, a `journal/*.json` file, or an unreadable
-  manifest), then runs three steps. `_prune_retire` retires superseded and
-  withdrawn entries whose `state_changed_at` is before the cutoff, except
-  the structure editions `_cited` returns (those a live or embargoed data
-  edition's `partial_of` cites, `cross_product.cited_structures`; every
-  structure edition when a manifest cannot be read), with
-  sdmxlib's `retire`, in one `InventoryWriter.commit` of kind `retire`
-  (`Change(leaves=..., retired=head tombstones + new)`), re-planned against
-  each head and skipped when nothing qualifies. `_prune_dirs` deletes
+  manifest), then runs three steps. `_prune_retire` commits
+  `store.inventory.retire_change(head, cutoff, now=, artefacts=writer.artefacts,
+  skipped=)` in one `InventoryWriter.commit` of kind `retire`, re-planned
+  against each head; `None` writes nothing. A dry run plans the same change
+  against the head it read. `_new_tombstones` lists the tombstones the change
+  adds. `_prune_dirs` deletes
   `products/<p>/<edition_id>` for each tombstone whose `retired_at` is
   before the cutoff. `_prune_data` deletes each `data/` file no on-disk
   manifest lists (`sdmxlib.edition.edition_files`) whose mtime is before the
@@ -164,6 +191,122 @@ are imported from `statcatlib.admin`.
   `previous`, `state`, `manifest_sha256`, `schema_version`,
   `state_changed_at` and `release_at`.
 
+### `embargo.py`
+
+`EmbargoReader` (Phase 5 D12) is exported from `statcatlib`. statcat-serve's preview
+and pre-materialisation read through it (sdmxlib plan Q8, Q9).
+
+- `EmbargoReader.open(location: str | Path | BlobLocation) -> Self`: runs
+  `catalog._open_read_only(opener="EmbargoReader.open")`, so it refuses what
+  `Catalog.open` refuses, then `refresh()`. Direct construction raises `TypeError`
+  naming `open`.
+- `refresh() -> Self`: holds a `FullInventory` of its own over
+  `InventoryWriter(...).storage.read`, made once a version exists, and refreshes it.
+- `editions(*, states=("embargoed",)) -> list[EditionEntry]`: the entries in
+  *states* of the held version, by product id and then leaf order, through
+  `admin._edition_entry`; empty before the first version. A string, or a state
+  outside sdmxlib's `State`, raises `ValueError`.
+- `view(product_id, edition_id, *, embedding_model=None, query_prefix="") ->
+  EditionView`: an unrecorded product raises `InventoryRuleError(rule="unknown")`
+  (`admin._leaf`), as does an unrecorded edition; a `superseded` or `withdrawn`
+  entry raises `ValueError`, before anything under the edition is read. Otherwise
+  `catalog._open_view` on the reader's own `sdmxlib.EditionEngine`, with stamps in
+  `<root>/embargo-verified`.
+- `close()`: closes every view's store and registry, then the engine; idempotent.
+  A closed reader's methods and views raise `RuntimeError`.
+
+### `jobs.py`
+
+The jobs configuration (Phase 5 D7), imported from `statcatlib.jobs`; nothing is
+re-exported from `statcatlib`. Standard library and `attrs` only.
+
+- `load_jobs_config(path) -> JobsConfig`: reads the bytes, parses with
+  `tomllib`, decodes, and runs `RetentionPolicy.check()`. Every refusal raises
+  `JobsConfigError` (`ValueError`), its message starting with the path: an
+  unreadable file, not UTF-8, not TOML, an unknown key at any level (the top
+  level, `[retention]`, `[access]`, each allowlist entry, `[expiry]`), a value of
+  the wrong type (`bool` is no integer), a missing or other `format_version`
+  (`FORMAT_VERSION = 1`), a missing `[retention]` or required key, a bad duration
+  or instant, an `[expiry]` with `enabled = true` and no `not_before`,
+  `namespace`, `deployments` or `port`, a port outside 1 to 65535, or a failed
+  inequality.
+- `JobsConfig(retention, access, expiry, sha256, format_version=1)`: `access` is
+  `None` when the file has no `[access]`; `expiry` is `ExpiryPolicy()` (disabled)
+  when it has no `[expiry]`; `sha256` is the file's, which each run logs.
+- `RetentionPolicy(backup, gc_window, tombstone_window, inventory_window,
+  retire_after, pod_staleness, keep_roots=DEFAULT_KEEP_ROOTS (10),
+  upload_grace=DEFAULT_UPLOAD_GRACE (24 h))`, `durations()`, and `check()`,
+  which raises `JobsConfigError` naming both values for the first failure of:
+  every duration positive; `backup <= gc_window <= tombstone_window`;
+  `inventory_window >= gc_window + 1d`; `pod_staleness < inventory_window`;
+  `keep_roots >= 2`.
+- `AccessPolicy(storage_account, container, allowlist, sentinels=(), scopes=())`
+  with `Principal(object_id, name)`; `storage_account`, `container`,
+  `allowlist` and `sentinels` are required keys, `sentinels` may be empty.
+- `ExpiryPolicy(enabled=False, not_before=None, namespace=None, deployments=(),
+  port=None)`.
+- `parse_duration(text) -> timedelta` (`^[0-9]+[smhd]$`) and
+  `format_duration(value) -> str` (the largest whole unit, as messages write it).
+- `decode_registration(path, raw) -> Registration`: reads
+  `pending/<product_id>/<edition_id>.json` as the release job's fold and the GC job
+  read it. `ValueError` for a path of another shape, a product id that cannot name a
+  directory (`_PRODUCT_ID`, `store/blob.py`'s `_SAFE_ID` pattern), an edition id that
+  is no canonical UUIDv7, a document that is not one object with the keys
+  `edition_id` and `release_at` in that order, an `edition_id` other than the file
+  name's, a `release_at` that is neither `null` nor a zoned time, or bytes that differ
+  from `_pending_document` of their own decode.
+- `Registration(product_id, edition_id, release_at)` (frozen): `path`, and
+  `edition_time`, the UUIDv7's time, which the tombstone window is measured from.
+- `_pending_document(edition_id, release_at) -> bytes`: the `pending/` document's
+  bytes, the contract `docs/catalog-format.md` §1.5 states; `upload.py` writes with
+  it. `_stamp(moment)` writes a time in UTC to the millisecond with `Z`.
+- `_emit(logger, level, event, message)`: logs *message* with the JSON object *event*
+  as the record's `_EVENT_ATTRIBUTE` (`"statcat_event"`), the D14 event a job command
+  prints as one line on standard output.
+
+### `gc.py`
+
+The GC job (Phase 5 D10), imported from `statcatlib.gc`; nothing is re-exported from
+`statcatlib`. `statcat gc` runs it.
+
+- `collect_garbage(location, policy, *, storage=None, dry_run=False, forecast=None,
+  discard=()) -> GcReport`: runs `policy.check()` (`JobsConfigError` before any
+  read), parses each *discard* as `<product>/<edition>` (`ValueError`), reads
+  `storage.now()` once (`storage` defaults to `BlobInventoryStorage(location)`), then
+  `_Run`'s steps: `inventory()` (the head through `FullInventory`, then older roots
+  by `seq` until `keep_roots` are read and one created before `now - gc_window`,
+  never at or below `HISTORY.expired_through`; leaves by hash on `GC_THREADS`),
+  the discard refusal (`ValueError` naming `statcat admin cancel` for an edition the
+  head records or a root read keeps), `registrations()` (LIST `pending/`; a
+  registration of an edition the head records is `folded` once last modified
+  before the cutoff and is never read; any other is decoded with
+  `jobs.decode_registration`, and an unreadable one fails its product closed; one
+  whose `edition_time` is older than `tombstone_window` is `expired-registration`
+  and protects nothing), `manifests()` (every kept and registered manifest on
+  `GC_THREADS`; a kept one must match its `manifest_sha256`, a registered one must
+  name its path; a failure, or a kept tombstone with no entry read, fails the
+  product closed), `products()` (LIST `products/`: `data/` files no manifest lists,
+  older than `upload_grace`, outside failed-closed products, are `unreferenced`)
+  and `directories()` (`retired` for an edition tombstoned before the cutoff and not
+  kept; `discarded`; `gc.unregistered` for a directory nothing keeps or registers,
+  older than `upload_grace`). It logs every `gc.planned` line, then each
+  `gc.forecast_deviation`, then deletes unless `dry_run`, in three pooled phases:
+  registrations and `data/` files; each directory's `manifest.json` once its
+  registration, if any, is deleted; the rest once its manifest is deleted or absent.
+  With *forecast* only paths in both plans are deleted.
+- `GcReport(planned, deleted, failed, failed_closed, unregistered, deviations,
+  dry_run)` and `Deletion(path, reason, edition=None)`, frozen. `Reason` is
+  `retired`, `unreferenced`, `folded`, `expired-registration` or `discarded`.
+- `GcStoppedError(RuntimeError)`: no `catalog.json`, an opted-in catalog with no
+  `HISTORY`, an unreadable head, any root the reader skipped, a missing or invalid
+  root above `expired_through`, a missing or unhashed leaf. Nothing is deleted.
+- `_delete(location, path)` refuses (`ValueError`) any path outside `products/` and
+  `pending/`, and returns a `BlobStoreError`'s text in place of raising it.
+- Events go through `jobs._emit` on the `statcatlib.gc` logger: deletion log lines
+  (`event`, `path`, `reason`, `edition`, and `error` on `gc.delete_failed`) and D14
+  alerts (`event`, `severity`, `at`, `catalog`, and `product`, `edition`, `reason`,
+  `detail` when they apply).
+
 ### `checks.py`
 
 - `SealCheck`: the type `Callable[[EditionView], Iterable[Finding]]`, passed
@@ -179,16 +322,20 @@ are imported from `statcatlib.admin`.
 The module keeps its 0.11 name and describes editions. Imported from
 `statcatlib.bundle`; nothing here is re-exported from `statcatlib`:
 
-- `catalog_section(store: _Store, *, kind="data") -> dict`: builds the
-  `statcat` manifest section (format 3: `format`, `kind`,
+- `catalog_section(store: _Store, *, kind="data", lifecycle) -> dict`: builds the
+  `statcat` manifest section (format 3: `format`, `kind`, `lifecycle`,
   `statcatlib_version`, `datasets`, `search`) from a working copy's open
-  store. `WorkingCopy.seal` passes it to
+  store. `lifecycle` (`"final"` or `"revisable"`, from 0.17) is the one the
+  seal compacts by, which the release job's fold records for a product the
+  Inventory does not record (`store.working_copy.fold_lifecycle`). `WorkingCopy.seal` passes it to
   `sdmxlib.edition.seal(..., extensions={"statcat": ...})`. It raises
   `ValueError` when `search/` holds files and no `index.json`.
 - `check_catalog(root, manifest, reg, profile) -> list[Finding]`: the
   `statcat` check, with sdmxlib 0.80's `Check` signature, run as an extension
   of `sdmxlib.edition.check_edition`. It refuses a section format other than
-  3, a `kind` outside `PRODUCT_KINDS`, and a `structure` edition whose
+  3, a `kind` outside `PRODUCT_KINDS`, a `lifecycle` other than `final` or
+  `revisable` (an absent one passes: editions sealed before 0.17 have none),
+  and a `structure` edition whose
   manifest binds a dataflow or whose store holds a dataflow, binding or
   partial.
 - `structure_content(conn) -> list[Finding]`: one finding per `Dataflow`,
@@ -230,8 +377,8 @@ The module keeps its 0.11 name and describes editions. Imported from
 - `UploadCheckError(findings)` (`RuntimeError`): any finding, an oversize file, a
   path leaving its directory, a rotted local file, a differing manifest, or a
   differing `pending/` document.
-- `_pending_document(edition_id, release_at) -> bytes`: the `pending/` document's
-  bytes, the contract `docs/catalog-format.md` §1.5 states.
+- `_pending_document` comes from `jobs.py`, which the release job's fold and GC
+  decode against.
 - `_lookup(rest, manifest)`: adapts `rest.head` to sdmxlib's `Lookup`, asking once per
   file. `_store_path(manifest, entry)` normalises `../data/`, which `obstore` refuses.
 - `MAX_PUT_BYTES`: Put Blob's limit, 5,000 MiB.
@@ -387,12 +534,21 @@ read.
   store takes a `BlobLocation` the caller built. `catalog.py` and `store/blob.py` read
   no environment variable.
 - Only `CatalogAdmin` changes the Inventory, apart from a seal's record and
-  the journal repair. No attribute of `Catalog`, `Product` or `EditionView`
-  returns a `CatalogAdmin` or an embargoed, superseded or withdrawn edition.
+  the journal repair. It runs over a catalog directory or a `BlobLocation`,
+  `prune` on a directory only. No attribute of `Catalog`, `Product` or `EditionView`
+  returns a `CatalogAdmin`, an `EmbargoReader` (spec §6.4) or an embargoed,
+  superseded or withdrawn edition. `Catalog`'s reads (`products`, `Product.view`,
+  `Product.editions`) go through a `LiveInventory`.
 - Every `CatalogAdmin` rule is re-evaluated against the head each commit
   lands on: a rollback whose `if_live` is no longer live, a release that is
   no longer a fast-forward, and a seal record computed before the embargo was
   turned on are each refused or recomputed.
+- `EmbargoReader` writes nothing to the catalog. Over a `BlobLocation` it writes to its
+  `cache_dir` alone: the files of the embargoed and live editions it views, and their
+  receive stamps under `<cache_dir>/<catalog_id>/embargo-verified`. That `cache_dir`
+  holds embargoed bytes, so a deployment gives the reader its own. A superseded or
+  withdrawn edition, or one the Inventory does not record, is refused before any read
+  under it.
 - `Product.view` passes the Inventory's `manifest_sha256` to
   `sdmxlib.LocalRegistry.edition`, which hashes the bytes of
   `manifest.json` it parses. A missing, malformed or different manifest
@@ -409,6 +565,13 @@ read.
   none. A refusal in one product never stops another's. A crash during the
   batch leaves each sealed, unrecorded edition's journal entry, so its
   product's next `working_copy()` records it, in a version of its own.
+- `collect_garbage` deletes nothing when it cannot read the Inventory back through
+  the window (`GcStoppedError`), nothing under the `data/` of a product whose
+  manifest or registration it cannot read or verify, no file a kept or registered
+  manifest lists, no `data/` file younger than `upload_grace`, no edition directory
+  whose tombstone is within the window or that no Inventory version read records,
+  and nothing under `inventory/` or `history/`. A directory loses `manifest.json`
+  before any other file, and keeps every file when it keeps its manifest.
 - `CatalogAdmin.prune` never retires or deletes anything of a product with
   an unsealed edition or a seal journal entry, deletes only the directories
   of tombstoned editions, and deletes no Inventory root, leaf or tombstone.
@@ -423,21 +586,39 @@ read.
   at INFO, and writes and removes a `.statcat-link-probe.*` file and its hard
   link.
 - `Product.view` attaches the edition to the catalog's engine
-  (`LocalRegistry.edition(path, engine=, manifest_sha256=)`). `Product.working_copy` writes as
+  (`LocalRegistry.edition(path, engine=, manifest_sha256=)`), and
+  `EmbargoReader.view` to the reader's; over a `BlobLocation` each writes the pulled
+  edition and its stamp under its own `cache_dir`. `Product.working_copy` writes as
   `store/EGM.md` describes.
 - `synthetic_catalog` writes a whole catalog: `catalog.json`, sdmxlib's
   synthetic edition, the catalog rows, the search index when lancedb is
   installed, the manifest and Inventory version 1. `temporary_catalog`
   deletes its directory on exit.
 - Each `CatalogAdmin` change writes one Inventory version (a root, and for a
-  product change one leaf) and `CURRENT`. `prune(dry_run=False)` writes at
+  product change one leaf) and `CURRENT`; over a `BlobLocation` it sends one
+  anonymous `HEAD` for the storage clock unless given `clock=`.
+  `advance_current` writes `CURRENT` alone, or nothing. `prune(dry_run=False)` writes at
   most one, of kind `retire`, deletes edition directories and `data/` files,
   and logs each deleted path at INFO and each failed deletion at WARNING on
   the `statcatlib.admin` logger.
 - `diagnostics.verify_catalog` opens each edition in a DuckDB engine of
   `check_edition`'s own; neither it nor `diagnostics.doctor` writes a file.
-- These modules make no network calls.
+- These modules make no network calls, except `upload.py`, `admin.py`, `gc.py`,
+  `catalog.py` and `embargo.py`, which reach a store only through the `BlobLocation`
+  they are given. `jobs.py` reads
+  the one file it is given.
+- `gc.collect_garbage` writes nothing. It deletes under `products/` and `pending/`
+  only, unless `dry_run`, and logs each event on the `statcatlib.gc` logger at INFO
+  (`gc.planned`, `gc.deleted`), WARNING (`gc.failed_closed`, `gc.unregistered`,
+  `gc.forecast_deviation`) or ERROR (`gc.delete_failed`). It sends one anonymous
+  `HEAD` for the storage clock unless its storage has a `clock=`.
 
 ## Verification
 
 - Run: `just ci` (unit tier alone: `uv run pytest -m unit`).
+- `EmbargoReader`: `tests/test_embargo_reader.py` (a local catalog), the
+  `TestNoOperationalReach` and blob-extra tests in `tests/test_public_api.py`, and,
+  against Azurite, `tests/stores/test_embargo_reader.py`.
+- GC: `tests/test_gc.py` (every rule over a `MemoryStore`, with the helpers in
+  `tests/_gc.py`), `tests/test_cli_gc.py` and, against Azurite,
+  `tests/stores/test_gc.py`; `decode_registration` in `tests/test_jobs_config.py`.

{statcatlib-0.16.2 → statcatlib-0.17.0}/src/statcatlib/__init__.py

@@ -43,6 +43,7 @@ from statcatlib.sources import (
 from statcatlib import checks, query, urn
 from statcatlib.admin import CatalogAdmin
 from statcatlib.catalog import Catalog, Edition, NoLiveEditionError, Product, SealOutcomes
+from statcatlib.embargo import EmbargoReader
 from statcatlib.expr import (
     AliasedExpr,
     CountExpr,
@@ -151,6 +152,7 @@ __all__ = [
     "Edition",
     "EditionCheckError",
     "EditionView",
+    "EmbargoReader",
     "EmbeddingMetadata",
     "ExistsExpr",
     "Expr",