statcatlib 0.16.0__tar.gz → 0.16.2__tar.gz

{statcatlib-0.16.0 → statcatlib-0.16.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: statcatlib
-Version: 0.16.0
+Version: 0.16.2
 Summary: Queryable open data catalog engine for DCAT-AP, StatDCAT-AP, CKAN, and SDMX
 Keywords: sdmx,dcat,open-data,catalog,duckdb,statistical-data,ckan
 Author: gabrielgellner

{statcatlib-0.16.0 → statcatlib-0.16.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "statcatlib"
-version = "0.16.0"
+version = "0.16.2"
 description = "Queryable open data catalog engine for DCAT-AP, StatDCAT-AP, CKAN, and SDMX"
 license = "Apache-2.0"
 license-files = ["LICENSE"]

{statcatlib-0.16.0 → statcatlib-0.16.2}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "statcatlib"
-version = "0.16.0"
+version = "0.16.2"
 description = "Queryable open data catalog engine for DCAT-AP, StatDCAT-AP, CKAN, and SDMX"
 license = "Apache-2.0"
 license-files = ["LICENSE"]

{statcatlib-0.16.0 → statcatlib-0.16.2}/src/statcatlib/cli/EGM.md

@@ -39,7 +39,7 @@ Commands:
 | `statcat verify-catalog PATH` | `verify_catalog` | `diagnostics.verify_catalog(path)` | any finding, each printed as `FAIL <file>: <detail>`; exit 2 on `CatalogUnreadableError`. Each stale data edition prints as `STALE <StaleEdition>` and leaves the exit code alone |
 | `statcat doctor PATH` | `doctor` | `diagnostics.doctor(path)` | the report holds a warning (topic `stale data editions`: a stale embargoed data edition is one; a stale live edition is `info` and leaves the exit code alone; topic `partials with no structure home`: each item is one) |
 | `statcat prune PATH --older-than DAYS --reason TEXT [--apply]` | `prune` | `sc.CatalogAdmin.open(path).prune(older_than=timedelta(days=DAYS), reason=TEXT, dry_run=not apply)` | `catalog.json` is missing, the catalog cannot be opened, the retire commit is refused, or a deletion failed (each printed as `FAIL <path>: <error>`) |
-| `statcat upload EDITION_DIR AZ_URL [--release-at TIME]` | `upload` | `upload_edition(edition_dir, location, release_at=)` | a finding, each printed as `FAIL <file>: <detail>`, or a `ManifestError`, `BlobStoreError` or `ClientAuthenticationError`, printed as `ERROR <message>`; exit 2 on a usage error, a bad `AZ_URL`, `s3://`, no `AZURE_STORAGE_ACCOUNT_NAME`, a key, SAS, token or endpoint variable set, even empty (`_REFUSED_VARIABLES`), a naive time, or a missing `blob` extra |
+| `statcat upload EDITION_DIR AZ_URL [--release-at TIME]` | `upload` | `upload_edition(edition_dir, location, release_at=)` | a finding, each printed as `FAIL <file>: <detail>`, or a `ManifestError`, `BlobStoreError` or `ClientAuthenticationError`, printed as `ERROR <message>`; exit 2 on a usage error, a bad `AZ_URL`, `s3://`, no `AZURE_STORAGE_ACCOUNT_NAME`, a refused `obstore` variable set, even empty, in any spelling `obstore` reads (`_REFUSED_VARIABLES`, matched by `_refused_variables`: `AZURE_` then any case), a naive time, a missing `blob` extra, or an environment value `obstore` cannot parse, such as `AZURE_DISABLE_TAGGING=junk` (`GenericError` from the store build, before any request) |
 
 `build-index` options and their `WorkingCopy.build_index` keyword:
 
@@ -75,10 +75,13 @@ Private helpers that tests and later commands rely on:
   `statcatlib.cli` logger (currently unused).
 - `_usage(message)` — prints it and exits 2. `_release_time(raw)`, `_catalog_uri(raw)`
   and `_azure_account()` parse `upload`'s inputs, each through `_usage`.
-  `_azure_location(uri, account)` builds the store with `obstore.store.from_url`,
+  `_refused_variables()` lists the environment's names that `obstore` would parse as a
+  refused key, as spelled. `_azure_location(uri, account)` builds the store with `obstore.store.from_url`,
   `TokenIdentity(AzureCredentialProvider(DefaultAzureCredential()))` and the one-hour
-  timeout; `upload` is the only code in statcatlib that reads the environment for a
-  store.
+  timeout, and turns obstore's `GenericError` from that build into `_usage`, with the text from
+  `_store_build_message(exc)`: obstore's first line, and the `AZURE_DISABLE_TAGGING` spelling the
+  environment sets, since obstore 0.12.1's message omits the variable. `upload` is the only code in
+  statcatlib that reads the environment for a store.
 - `_parse_codelist_policy(raw)` — accepts only mode `fts_only`
   (`_ACCEPTED_CODELIST_MODES`); raises `typer.BadParameter` otherwise.
 - `_parse_reference_metadata_blend(raw)` — `max_chars` must be a positive

{statcatlib-0.16.0 → statcatlib-0.16.2}/src/statcatlib/cli/__init__.py

@@ -60,9 +60,12 @@ _DEFAULT_MODEL = "intfloat/multilingual-e5-base"
 
 _ACCEPTED_CODELIST_MODES = frozenset({"fts_only"})
 
-#: The `obstore` variables `statcat upload` refuses, as `_store/_azure.pyi` spells them. Keys and SAS break spec
-#: §10.1. A token writes under another identity than the one the upload records. An endpoint sends the writes to
-#: another host than the digest read.
+#: The `obstore` variables `statcat upload` refuses: every ``azure_`` spelling that object_store 0.14.2's
+#: `AzureConfigKey::from_str` accepts for these keys, the crate obstore 0.12.1 bundles. Spec §10.1 allows the
+#: publish job no account key or SAS. The credential provider's token wins beside the key, SAS and token variables
+#: on obstore 0.12.1, and a later obstore could change that order. An endpoint, the emulator or the Fabric
+#: endpoint sends the writes to another host than the digest read. Skipping the signature sends unsigned writes. A
+#: customer-provided key encrypts blobs that the digest read sends no key for.
 _REFUSED_VARIABLES = (
     "AZURE_STORAGE_ACCOUNT_KEY",
     "AZURE_STORAGE_ACCESS_KEY",
@@ -72,8 +75,16 @@ _REFUSED_VARIABLES = (
     "AZURE_STORAGE_TOKEN",
     "AZURE_STORAGE_ENDPOINT",
     "AZURE_ENDPOINT",
+    "AZURE_STORAGE_USE_EMULATOR",
+    "AZURE_USE_FABRIC_ENDPOINT",
+    "AZURE_SKIP_SIGNATURE",
+    "AZURE_STORAGE_ENCRYPTION_KEY",
 )
 
+#: The refused names as `MicrosoftAzureBuilder::from_env` compares them: it reads each variable whose name starts
+#: with ``AZURE_`` and parses the name lowercased.
+_REFUSED_KEYS = frozenset(name.lower() for name in _REFUSED_VARIABLES)
+
 #: How long one request may take: a 5,000 MiB Put at about 1.4 MB/s.
 _UPLOAD_TIMEOUT = "1h"
 
@@ -633,10 +644,24 @@ def _catalog_uri(raw: str) -> str:
     return f"az://{'/'.join(segments)}/"
 
 
+def _refused_variables() -> list[str]:
+    """The environment's names that `obstore` parses as a refused key, spelled as the environment spells them.
+
+    `MicrosoftAzureBuilder::from_env` lowercases ASCII letters only, so a name with any other character never
+    matches a key there, and is not refused here.
+    """
+    return sorted(
+        name for name in os.environ if name.startswith("AZURE_") and name.isascii() and name.lower() in _REFUSED_KEYS
+    )
+
+
 def _azure_account() -> str:
     """The storage account from the environment; a refused variable, even empty, or no account, exits 2."""
-    if found := [name for name in _REFUSED_VARIABLES if name in os.environ]:
-        _usage(f"upload acts under one identity at the account's URL; unset {', '.join(found)}")
+    if found := _refused_variables():
+        _usage(
+            "upload writes with its own credential, signed and with no customer-provided key, "
+            f"to the account's URL; unset {', '.join(found)}"
+        )
     account = os.environ.get("AZURE_STORAGE_ACCOUNT_NAME")
     if not account:
         _usage("set AZURE_STORAGE_ACCOUNT_NAME to the storage account's name")
@@ -644,22 +669,49 @@ def _azure_account() -> str:
 
 
 def _azure_location(uri: str, account: str) -> BlobLocation:
-    """The catalog at *uri*, reached with `DefaultAzureCredential`; a missing ``blob`` extra exits 2."""
+    """The catalog at *uri*, reached with `DefaultAzureCredential`.
+
+    A missing ``blob`` extra exits 2, and so does an environment value `obstore` cannot parse, such as
+    ``AZURE_DISABLE_TAGGING=junk``: obstore 0.12.1 raises `GenericError` from the store build, before any request.
+    """
     try:
         from azure.identity import DefaultAzureCredential  # noqa: PLC0415
         from obstore.auth.azure import AzureCredentialProvider  # noqa: PLC0415
+        from obstore.exceptions import GenericError  # noqa: PLC0415
         from obstore.store import from_url  # noqa: PLC0415
     except ImportError:
         _usage("upload needs the blob extra: uv add 'statcatlib[blob]' (or pip install 'statcatlib[blob]')")
-    store = from_url(
-        uri.removesuffix("/"),
-        account_name=account,
-        credential_provider=TokenIdentity(AzureCredentialProvider(DefaultAzureCredential())),
-        client_options={"timeout": _UPLOAD_TIMEOUT},
-    )
+    credential = TokenIdentity(AzureCredentialProvider(DefaultAzureCredential()))
+    try:
+        store = from_url(
+            uri.removesuffix("/"),
+            account_name=account,
+            credential_provider=credential,
+            client_options={"timeout": _UPLOAD_TIMEOUT},
+        )
+    except GenericError as exc:
+        _usage(_store_build_message(exc))
     return BlobLocation(uri, store)
 
 
+def _store_build_message(exc: Exception) -> str:
+    """The exit-2 text for a store build that failed on the environment.
+
+    obstore 0.12.1's message names the value and omits the variable, so the text names
+    ``AZURE_DISABLE_TAGGING`` when the environment sets it, in any case after ``AZURE_``.
+    """
+    reason = str(exc).splitlines()[0] if str(exc) else type(exc).__name__
+    message = f"obstore cannot build the store from the environment ({reason})"
+    tagging = sorted(
+        name
+        for name in os.environ
+        if name.startswith("AZURE_") and name.isascii() and name.lower() == "azure_disable_tagging"
+    )
+    if tagging:
+        message += f"; {', '.join(tagging)} takes true or false"
+    return message
+
+
 @app.command("upload")
 def upload(
     edition_dir: Path = typer.Argument(
@@ -684,11 +736,17 @@ def upload(
     release job. It exits 0 when the edition is uploaded, or already registered
     with the same bytes; 1 on any finding, each printed as ``FAIL <file>: <detail>``,
     or on a store or credential failure, printed as ``ERROR <message>``; and 2 on
-    a usage error, a missing ``blob`` extra, or any of these variables set, even
-    empty: ``AZURE_STORAGE_ACCOUNT_KEY``, ``AZURE_STORAGE_ACCESS_KEY``,
+    a usage error, a missing ``blob`` extra, an environment value ``obstore``
+    cannot parse, such as ``AZURE_DISABLE_TAGGING=junk``, or any of these
+    variables set, even empty: ``AZURE_STORAGE_ACCOUNT_KEY``,
+    ``AZURE_STORAGE_ACCESS_KEY``,
     ``AZURE_STORAGE_MASTER_KEY``, ``AZURE_STORAGE_SAS_KEY``,
     ``AZURE_STORAGE_SAS_TOKEN``, ``AZURE_STORAGE_TOKEN``,
-    ``AZURE_STORAGE_ENDPOINT`` and ``AZURE_ENDPOINT``.
+    ``AZURE_STORAGE_ENDPOINT``, ``AZURE_ENDPOINT``,
+    ``AZURE_STORAGE_USE_EMULATOR``, ``AZURE_USE_FABRIC_ENDPOINT``,
+    ``AZURE_SKIP_SIGNATURE`` and ``AZURE_STORAGE_ENCRYPTION_KEY``. Any case
+    after ``AZURE_`` counts, such as ``AZURE_storage_endpoint``, since
+    ``obstore`` reads that spelling too.
     """
     when = _release_time(release_at)
     uri = _catalog_uri(az_url)