argus-cortex 0.4.2__tar.gz → 0.4.4__tar.gz

{argus_cortex-0.4.2 → argus_cortex-0.4.4}/CHANGELOG.md

@@ -7,6 +7,46 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.4.4] - 2026-10-31
+
+### Fixed
+- **`SingGuardClient` keeps a longer writeup.** `to_result` clipped `explanation` at 4000
+  characters, which cut cascade/slow reasoning mid-sentence. The cap is now 16384, enough
+  for a 2048-token slow writeup plus a cascade preface.
+- **Re-recording an asset no longer erases what another stage recorded on it.** `record_asset`'s
+  upsert on `sha256` replaced `immich_id` and `metadata` outright, so argus-lens recording a caption
+  for an image argus-quarry had already recorded (licence, author, source) wiped that provenance,
+  and a call without an `immich_id` cleared the stored one. `metadata` now merges key by key
+  (`jsonb ||`: the new call's keys win, others are kept) and a `NULL` `immich_id` keeps the stored
+  one. `uri` is still last-write-wins: it is the asset's latest location. A caller that needs to
+  remove a metadata key can no longer do it through `record_asset`.
+
+## [0.4.3] - 2026-10-30
+
+### Added
+- **`argus_cortex.gpu`**: GPU tenancy shared across the stages, generalised
+  from argus-lens's `gpu` coordinator.
+  - `GpuCoordinator`: the protocol. `lease(caller=, min_vram_mb=)` is a
+    context manager that holds a GPU slot for its body. `GpuLeaseTimeout` is a
+    `TimeoutError`.
+  - `NullCoordinator`: no gating, and the default.
+  - `LeaseCoordinator`: an exclusive `flock` that every argus process on the
+    host takes, then, while still holding it, a wait until `free_vram_mb()`
+    reports `min_vram_mb`. Unlike lens's lease, `min_vram_mb` is honoured. NVML
+    counts every process on the card, so ComfyUI and Ollama are accounted for
+    without taking part. One deadline covers both waits. A probe that cannot
+    run (no driver, no `[gpu]` extra, a CPU host) lets the lease through.
+  - `free_vram_mb(device)`: NVML free memory in MB, or `None`.
+  - `build_coordinator()` / `coordinator_from_env()`: `ARGUS_GPU_COORDINATOR`
+    (`none` / `lease`), `ARGUS_GPU_LEASE_PATH`, `ARGUS_GPU_LEASE_TIMEOUT_S`
+    (300 s) and `ARGUS_GPU_DEVICE`. These are the names lens already reads. An
+    unusable number logs a warning and falls back to the default.
+  - The default lock is `<tempdir>/argus-gpu.lock`. Containers have their own
+    `/tmp`, so a shared lock needs `ARGUS_GPU_LEASE_PATH` on a volume every
+    GPU service mounts. The lock file is `chmod 0o666` after create so a
+    restrictive umask does not block other container UIDs from opening it.
+- `[gpu]` extra: `nvidia-ml-py`, imported only when probed.
+
 ## [0.4.2] - 2026-10-29
 
 ### Added
@@ -138,7 +178,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
     (`LocalBackend`, `RemoteBackend`, `RemoteProvider`, `with_retries`
     retrying only transient errors, `resolve_device`).
 
-[Unreleased]: https://github.com/smk762/argus-cortex/compare/v0.4.2...HEAD
+[Unreleased]: https://github.com/smk762/argus-cortex/compare/v0.4.4...HEAD
+[0.4.4]: https://github.com/smk762/argus-cortex/compare/v0.4.3...v0.4.4
+[0.4.3]: https://github.com/smk762/argus-cortex/compare/v0.4.2...v0.4.3
 [0.4.2]: https://github.com/smk762/argus-cortex/compare/v0.4.1...v0.4.2
 [0.4.1]: https://github.com/smk762/argus-cortex/compare/v0.4.0...v0.4.1
 [0.4.0]: https://github.com/smk762/argus-cortex/compare/v0.3.0...v0.4.0

{argus_cortex-0.4.2 → argus_cortex-0.4.4}/CLAUDE.md

@@ -18,6 +18,7 @@ Because everything here is imported by the downstream stages, **a change ripples
 - `wire.py` — the versioned wire-schema toolkit consumers build their contracts on: `make_versioned_base(field_name, current, majors)` (a Pydantic base that stamps + version-checks a field like `proof_version`/`manifest_version`), `check_version()` (the major-compatibility gate), `render_schema()`/`wire_schema()` (the canonical committed-schema string behind each consumer's `schema --check` CLI), and `VersionError`.
 - `backends.py` — the AI-backend contract: `Backend`/`LocalBackend`/`RemoteBackend` (point at a hosted service via `from_host(host, port)` or `from_provider(RemoteProvider.…)`), plus `with_retries()` and `resolve_device()`. httpx is lazy, behind `[remote]`.
 - `server.py` — the shared FastAPI/ASGI scaffolding other repos' micro-servers mount: `WriteGuard` (pure-ASGI method gate), the `constant_refuse()` (read-only/replay) and `cross_site_refuse()` (cross-origin write) predicates, `env_flag()`, and the `UNSAFE_METHODS`/`TRUTHY`/`FALSY` constants. Starlette is behind `[server]`.
+- `gpu.py` — GPU tenancy (from argus-lens): the `GpuCoordinator` protocol (`lease(caller=, min_vram_mb=)`), `NullCoordinator`, `LeaseCoordinator` (host-wide `flock` **then** a wait for NVML free VRAM, so tenants outside the lock like ComfyUI and Ollama are accounted for), `free_vram_mb()`, `coordinator_from_env()` (`ARGUS_GPU_*` — host-wide names on purpose: every process must agree). NVML lazily, behind `[gpu]`; a probe that cannot run grants the lease. **The lock file must be on a volume every GPU container mounts** — per-container `/tmp` coordinates nothing.
 - `xmp.py` — the suite's XMP sidecar read/merge/write (`write_xmp`/`read_xmp`/`merge_xmp`/`parse_xmp`, `XmpData`/`XmpRegions`, `XmpError`), used by lens (keywords/caption) and curator (faces, content rating). Stdlib `xml.etree` only. **Writes merge — latest wins per property, foreign properties preserved; keep that.** `None` = leave untouched, empty = remove. A DTD is refused and an unparseable sidecar raises rather than being overwritten. The `argus` namespace URI is a wire contract — don't change it.
 - `moderation/` — content-moderation clients over HTTP: `nudenet.py` (`NudeNetClient`, `NsfwResult`, `DEFAULT_EXPOSED_LABELS` — **the suite's one definition of nudity; don't widen it casually**, every stage's NSFW numbers move with it), `singguard.py` (`SingGuardClient` incl. text-only `check_text()`, `GuardResult`, `rule_matches`), `policies.py` (the named policy texts, the per-policy illegal-rule table and `mode()`: `default`/`sfw-*` are SFW — **a policy's text and its illegal letters change together**), `_http.py` (shared POST, proxy-free opener, `CheckFailed`/`ModerationConfigError`). Stdlib + pydantic; Pillow lazily, behind `[images]`. Never import NudeNet (AGPL) — HTTP only. URLs are passed in; stages own their env config.
 - `store/` — the optional stateful-services layer, opt-in per `CORTEX_*` env var: `lineage.py` (Phase 1 Postgres DAG, `open_lineage_store()`), `vector.py` (Phase 2 Qdrant, `open_vector_store()`), `blob.py` (Phase 3 MinIO/S3, `open_blob_store()`), over `config.py`/`models.py`/`errors.py`. Each `open_*()` returns a `Null*` no-op when its URL is unset. Drivers are behind `[postgres]`/`[qdrant]`/`[s3]`.

{argus_cortex-0.4.2 → argus_cortex-0.4.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: argus-cortex
-Version: 0.4.2
+Version: 0.4.4
 Summary: Shared foundation for the Argus suite: taxonomy, versioned wire-schema tooling, and the local/remote AI backend contract
 Project-URL: Repository, https://github.com/smk762/argus-cortex
 Author: smk762
@@ -20,6 +20,7 @@ Requires-Dist: certifi>=2026.10.19; extra == 'dev'
 Requires-Dist: httpx2>=2.7; extra == 'dev'
 Requires-Dist: httpx>=0.27; extra == 'dev'
 Requires-Dist: minio>=7.2; extra == 'dev'
+Requires-Dist: nvidia-ml-py>=12.535; extra == 'dev'
 Requires-Dist: pillow>=10.0; extra == 'dev'
 Requires-Dist: psycopg[binary,pool]>=3.1; extra == 'dev'
 Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
@@ -27,6 +28,8 @@ Requires-Dist: pytest>=8.0; extra == 'dev'
 Requires-Dist: qdrant-client>=1.10; extra == 'dev'
 Requires-Dist: ruff==0.15.16; extra == 'dev'
 Requires-Dist: starlette>=0.37; extra == 'dev'
+Provides-Extra: gpu
+Requires-Dist: nvidia-ml-py>=12.535; extra == 'gpu'
 Provides-Extra: images
 Requires-Dist: pillow>=10.0; extra == 'images'
 Provides-Extra: postgres
@@ -58,6 +61,7 @@ Part of the [Argus suite](https://github.com/smk762?tab=repositories&q=argus) (q
 - **`argus_cortex.server`** — shared write-guard + env-flag scaffolding for the suite's FastAPI micro-servers: `WriteGuard` (a pure-ASGI method-gate that never buffers streaming responses), the `constant_refuse()` / `cross_site_refuse()` predicates (read-only/replay mode and cross-site-write protection are two configs of the one guard), the `UNSAFE_METHODS` / `TRUTHY` / `FALSY` constants, and `env_flag()` (warns on a mistyped protection flag rather than silently disabling it). Starlette lives behind the `[server]` extra.
 - **`argus_cortex.xmp`** — the suite's one XMP sidecar implementation (hoisted from argus-lens): `write_xmp()` / `read_xmp()` / `merge_xmp()` / `parse_xmp()` over an `XmpData` model covering `dc` (keywords, description), `xmp` (Rating, Label), `lr` (hierarchical keywords), `Iptc4xmpExt` (PersonInImage), `mwg-rs` face/pet regions, and `argus:ContentRating`. Writes **merge**: each property you set replaces that property (latest write wins), everything else in the sidecar — Lightroom develop settings, GPS, another stage's fields — is preserved. `None` leaves a property alone, an empty value removes it. Writes are atomic; an unparseable existing sidecar raises `XmpError` instead of being clobbered. Stdlib only.
 - **`argus_cortex.moderation`** — the suite's content-moderation clients, over HTTP to servers the deployer runs (nothing here imports a detector): `NudeNetClient` (a NudeNet server's `/infer` → `NsfwResult`: detections, per-class maxima, and an **exposed score**) and `SingGuardClient` (a SingGuard server's `/api/guard/check` → `GuardResult`: verdict, the policy rules that fired, explanation; optional custom policy text). `DEFAULT_EXPOSED_LABELS` is the suite's **one definition of nudity**: the five exposed buttocks/breast/genitalia/anus classes — faces, covered parts, and bare feet, belly, armpits or male chest never count. Stdlib + pydantic; Pillow (for SingGuard's downscaling of large images, on by default) is behind `[images]`: without it `SingGuardClient(...)` raises `ModerationConfigError` at construction unless given `downscale=False`. A `user:password@` in a URL is sent as HTTP Basic auth. URLs are passed in, never read from the environment here: each stage owns its config. NudeNet itself is AGPL-3.0, which is why it stays behind an HTTP boundary.
+- **`argus_cortex.gpu`**: GPU tenancy, generalised from argus-lens. `coordinator_from_env()` returns a coordinator whose `lease(caller=, min_vram_mb=)` holds a GPU slot for a block of work. `none` (the default) grants every lease at once. `lease` takes an exclusive `flock` that every argus process on the host shares (one lock for the host, not per GPU), then waits until `free_vram_mb()` (NVML, behind `[gpu]`) reports `min_vram_mb` free across *every* process on the card, ComfyUI and Ollama included, which never take the lock. It raises `GpuLeaseTimeout` after `ARGUS_GPU_LEASE_TIMEOUT_S` (300 s). The VRAM check is a snapshot, not a reservation, and a probe that cannot run lets the lease through. Containers each have their own `/tmp`, so set `ARGUS_GPU_LEASE_PATH` to a file on a volume every GPU service mounts, or the lock coordinates nothing.
 - **`argus_cortex.store`** — the optional **stateful services layer** (the suite's persistent "memory"). External and opt-in — every store is configured by `CORTEX_*` env vars and no-ops when its URL is unset.
   - **Phase 1 — Postgres lineage store.** Persists the `source_asset → caption → human_edit → dataset_membership → training_run` DAG for LoRA reproducibility, edit capture, and the feedback loop. Backed by a psycopg connection **pool** (concurrent `asyncio.to_thread` calls are safe; a dropped connection is replaced), with a `transaction()` boundary for atomic multi-step writes. `open_lineage_store()` returns a no-op store when `CORTEX_PG_URL` is unset; psycopg lives behind the `[postgres]` extra.
   - **Phase 2 — Qdrant vector store.** Stores image + tag-set embeddings for retrieval-augmented few-shot and near-duplicate curation; put a `caption_id`/`asset_id` in a point's payload and a search hit joins straight back to the lineage. `open_vector_store()` no-ops when `CORTEX_QDRANT_URL` is unset; qdrant-client lives behind the `[qdrant]` extra. cortex stores vectors it's *handed* — computing embeddings is the caller's job.
@@ -119,6 +123,17 @@ policies.mode("adult-granular"), policies.mode("default")   # "nsfw", "sfw"
 
 Stages ask for a policy by name; `policies` holds each one's text, the rules that mean illegal content under it, and its SFW/NSFW mode (`default` and `sfw-*` are SFW).
 
+```python
+from argus_cortex.gpu import GpuLeaseTimeout, coordinator_from_env
+
+gpu = coordinator_from_env()   # ARGUS_GPU_COORDINATOR=lease, ARGUS_GPU_LEASE_PATH=/shared/argus-gpu.lock
+try:
+    with gpu.lease(caller="faces", min_vram_mb=1500):   # one argus job at a time, and 1.5 GB free
+        run_the_model()
+except GpuLeaseTimeout:
+    ...   # no slot in time: back off, or answer 503
+```
+
 Each `check()` raises `CheckFailed` (`NudeNetCheckFailed` / `GuardCheckFailed`) for a network error or an unusable answer, so a caller checking many images can record "unchecked" against the one image — and should treat unchecked as *not* safe. Bad configuration raises `ModerationConfigError`, whose message never echoes the URL.
 
 ```python

{argus_cortex-0.4.2 → argus_cortex-0.4.4}/README.md

@@ -12,6 +12,7 @@ Part of the [Argus suite](https://github.com/smk762?tab=repositories&q=argus) (q
 - **`argus_cortex.server`** — shared write-guard + env-flag scaffolding for the suite's FastAPI micro-servers: `WriteGuard` (a pure-ASGI method-gate that never buffers streaming responses), the `constant_refuse()` / `cross_site_refuse()` predicates (read-only/replay mode and cross-site-write protection are two configs of the one guard), the `UNSAFE_METHODS` / `TRUTHY` / `FALSY` constants, and `env_flag()` (warns on a mistyped protection flag rather than silently disabling it). Starlette lives behind the `[server]` extra.
 - **`argus_cortex.xmp`** — the suite's one XMP sidecar implementation (hoisted from argus-lens): `write_xmp()` / `read_xmp()` / `merge_xmp()` / `parse_xmp()` over an `XmpData` model covering `dc` (keywords, description), `xmp` (Rating, Label), `lr` (hierarchical keywords), `Iptc4xmpExt` (PersonInImage), `mwg-rs` face/pet regions, and `argus:ContentRating`. Writes **merge**: each property you set replaces that property (latest write wins), everything else in the sidecar — Lightroom develop settings, GPS, another stage's fields — is preserved. `None` leaves a property alone, an empty value removes it. Writes are atomic; an unparseable existing sidecar raises `XmpError` instead of being clobbered. Stdlib only.
 - **`argus_cortex.moderation`** — the suite's content-moderation clients, over HTTP to servers the deployer runs (nothing here imports a detector): `NudeNetClient` (a NudeNet server's `/infer` → `NsfwResult`: detections, per-class maxima, and an **exposed score**) and `SingGuardClient` (a SingGuard server's `/api/guard/check` → `GuardResult`: verdict, the policy rules that fired, explanation; optional custom policy text). `DEFAULT_EXPOSED_LABELS` is the suite's **one definition of nudity**: the five exposed buttocks/breast/genitalia/anus classes — faces, covered parts, and bare feet, belly, armpits or male chest never count. Stdlib + pydantic; Pillow (for SingGuard's downscaling of large images, on by default) is behind `[images]`: without it `SingGuardClient(...)` raises `ModerationConfigError` at construction unless given `downscale=False`. A `user:password@` in a URL is sent as HTTP Basic auth. URLs are passed in, never read from the environment here: each stage owns its config. NudeNet itself is AGPL-3.0, which is why it stays behind an HTTP boundary.
+- **`argus_cortex.gpu`**: GPU tenancy, generalised from argus-lens. `coordinator_from_env()` returns a coordinator whose `lease(caller=, min_vram_mb=)` holds a GPU slot for a block of work. `none` (the default) grants every lease at once. `lease` takes an exclusive `flock` that every argus process on the host shares (one lock for the host, not per GPU), then waits until `free_vram_mb()` (NVML, behind `[gpu]`) reports `min_vram_mb` free across *every* process on the card, ComfyUI and Ollama included, which never take the lock. It raises `GpuLeaseTimeout` after `ARGUS_GPU_LEASE_TIMEOUT_S` (300 s). The VRAM check is a snapshot, not a reservation, and a probe that cannot run lets the lease through. Containers each have their own `/tmp`, so set `ARGUS_GPU_LEASE_PATH` to a file on a volume every GPU service mounts, or the lock coordinates nothing.
 - **`argus_cortex.store`** — the optional **stateful services layer** (the suite's persistent "memory"). External and opt-in — every store is configured by `CORTEX_*` env vars and no-ops when its URL is unset.
   - **Phase 1 — Postgres lineage store.** Persists the `source_asset → caption → human_edit → dataset_membership → training_run` DAG for LoRA reproducibility, edit capture, and the feedback loop. Backed by a psycopg connection **pool** (concurrent `asyncio.to_thread` calls are safe; a dropped connection is replaced), with a `transaction()` boundary for atomic multi-step writes. `open_lineage_store()` returns a no-op store when `CORTEX_PG_URL` is unset; psycopg lives behind the `[postgres]` extra.
   - **Phase 2 — Qdrant vector store.** Stores image + tag-set embeddings for retrieval-augmented few-shot and near-duplicate curation; put a `caption_id`/`asset_id` in a point's payload and a search hit joins straight back to the lineage. `open_vector_store()` no-ops when `CORTEX_QDRANT_URL` is unset; qdrant-client lives behind the `[qdrant]` extra. cortex stores vectors it's *handed* — computing embeddings is the caller's job.
@@ -73,6 +74,17 @@ policies.mode("adult-granular"), policies.mode("default")   # "nsfw", "sfw"
 
 Stages ask for a policy by name; `policies` holds each one's text, the rules that mean illegal content under it, and its SFW/NSFW mode (`default` and `sfw-*` are SFW).
 
+```python
+from argus_cortex.gpu import GpuLeaseTimeout, coordinator_from_env
+
+gpu = coordinator_from_env()   # ARGUS_GPU_COORDINATOR=lease, ARGUS_GPU_LEASE_PATH=/shared/argus-gpu.lock
+try:
+    with gpu.lease(caller="faces", min_vram_mb=1500):   # one argus job at a time, and 1.5 GB free
+        run_the_model()
+except GpuLeaseTimeout:
+    ...   # no slot in time: back off, or answer 503
+```
+
 Each `check()` raises `CheckFailed` (`NudeNetCheckFailed` / `GuardCheckFailed`) for a network error or an unusable answer, so a caller checking many images can record "unchecked" against the one image — and should treat unchecked as *not* safe. Bad configuration raises `ModerationConfigError`, whose message never echoes the URL.
 
 ```python

{argus_cortex-0.4.2 → argus_cortex-0.4.4}/pyproject.toml

@@ -35,6 +35,9 @@ server = ["starlette>=0.37"]
 # Pillow, for SingGuardClient's downscaling of large images
 # (argus_cortex.moderation.singguard), imported only when it downscales.
 images = ["pillow>=10.0"]
+# NVML (nvidia-ml-py, imported as pynvml), for argus_cortex.gpu's free-VRAM probe;
+# imported only when probed. Without it the lease skips the VRAM wait.
+gpu = ["nvidia-ml-py>=12.535"]
 # The store drivers are in dev so each store's real client path (pool/enum/client
 # construction, error wrapping) is exercised in CI, not just via fakes.
 # starlette + httpx power the pure-ASGI guard tests (Starlette TestClient).
@@ -49,6 +52,7 @@ dev = [
     "httpx>=0.27",
     "httpx2>=2.7",
     "pillow>=10.0",
+    "nvidia-ml-py>=12.535",
     "certifi>=2026.10.19",
 ]
 

{argus_cortex-0.4.2 → argus_cortex-0.4.4}/src/argus_cortex/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.4.2'
-__version_tuple__ = version_tuple = (0, 4, 2)
+__version__ = version = '0.4.4'
+__version_tuple__ = version_tuple = (0, 4, 4)
 
 __commit_id__ = commit_id = None

argus_cortex-0.4.4/src/argus_cortex/gpu.py

@@ -0,0 +1,242 @@
+"""GPU tenancy shared across the suite: one heavy argus job on a GPU at a time,
+and only once the card has room for it.
+
+Generalised from argus-lens's ``gpu`` coordinator so every stage that runs a
+model on a GPU it shares (lens captioning, curator's faces, alongside ComfyUI
+and an Ollama-backed SingGuard) takes the same lease.
+
+* :class:`GpuCoordinator`: the protocol: ``lease(caller=, min_vram_mb=)`` is a
+  context manager that holds a GPU slot for its body.
+* :class:`NullCoordinator`: no gating, the default.
+* :class:`LeaseCoordinator`: an advisory file lock (POSIX ``flock``) taken by
+  every argus process on the host, then a wait until :func:`free_vram_mb` reports
+  ``min_vram_mb`` free. The lock orders argus jobs among themselves; the VRAM wait
+  is what accounts for tenants that never take the lock (ComfyUI, Ollama), since
+  NVML reports free memory across every process on the card.
+* :func:`coordinator_from_env`: ``ARGUS_GPU_COORDINATOR`` (``none``/``lease``),
+  ``ARGUS_GPU_LEASE_PATH``, ``ARGUS_GPU_LEASE_TIMEOUT_S``, ``ARGUS_GPU_DEVICE``.
+  The names are host-wide, not per stage: every process must agree on them.
+  One lock serialises every argus GPU job on the host; ``ARGUS_GPU_DEVICE``
+  only selects which card NVML probes.
+
+The VRAM check is a snapshot, not a reservation: a tenant outside the lock can
+still allocate between the check and the job. Each stage owns its footprint
+estimates (``min_vram_mb``); a probe that cannot run (no driver, no NVML, a CPU
+host) lets the lease through rather than blocking forever.
+
+Containers have their own ``/tmp``: the lock only coordinates processes that see
+the same file, so point ``ARGUS_GPU_LEASE_PATH`` at a volume every GPU service
+mounts. ``flock`` works across containers on one host kernel. NVML is behind the
+``[gpu]`` extra (``nvidia-ml-py``) and imported only when probed.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import logging
+import math
+import os
+import tempfile
+import threading
+import time
+from collections.abc import Callable, Iterator
+from typing import Protocol, runtime_checkable
+
+_log = logging.getLogger(__name__)
+
+DEFAULT_TIMEOUT_S = 300.0
+DEFAULT_POLL_S = 0.5
+COORDINATORS = ("none", "lease")
+
+_nvml_lock = threading.Lock()
+_nvml_ready: bool | None = None  # None: not tried yet; False: NVML cannot run here
+
+
+class GpuLeaseTimeout(TimeoutError):
+    """No GPU slot within the timeout: the lock stayed held, or the card never had room.
+
+    A ``TimeoutError``, so a caller can catch either, and tell "no GPU slot" from
+    a failed job and back off.
+    """
+
+
+@runtime_checkable
+class GpuCoordinator(Protocol):
+    """Gates a block of GPU work behind a lease."""
+
+    name: str
+
+    def lease(self, *, caller: str, min_vram_mb: int) -> contextlib.AbstractContextManager[None]:
+        """A context manager that holds a GPU slot for its body."""
+        ...
+
+
+class NullCoordinator:
+    """No gating: every lease is granted at once. The default."""
+
+    name = "none"
+
+    @contextlib.contextmanager
+    def lease(self, *, caller: str, min_vram_mb: int) -> Iterator[None]:
+        yield
+
+
+def free_vram_mb(device: int = 0) -> int | None:
+    """Free VRAM on NVML device *device*, in MB, across every process on it.
+
+    ``None`` when it cannot be read: no ``[gpu]`` extra, no driver, no such
+    device. NVML numbers devices as ``nvidia-smi`` does, which inside a container
+    means the GPUs the container was given.
+    """
+    global _nvml_ready
+    try:
+        import pynvml  # noqa: PLC0415 - optional, behind the [gpu] extra
+    except ImportError:
+        return None
+    with _nvml_lock:
+        if _nvml_ready is None:
+            try:
+                pynvml.nvmlInit()
+                _nvml_ready = True
+            except Exception:  # noqa: BLE001 - any NVML failure means "cannot probe"
+                _nvml_ready = False
+        if not _nvml_ready:
+            return None
+    try:
+        info = pynvml.nvmlDeviceGetMemoryInfo(pynvml.nvmlDeviceGetHandleByIndex(device))
+    except Exception:  # noqa: BLE001
+        return None
+    return max(0, int(info.free) // (1024 * 1024))
+
+
+def default_lock_path() -> str:
+    """``<tempdir>/argus-gpu.lock``: shared only by processes that share that directory."""
+    return os.path.join(tempfile.gettempdir(), "argus-gpu.lock")
+
+
+class LeaseCoordinator:
+    """One argus GPU job at a time on this host, started only when the card has room.
+
+    Takes an exclusive ``flock`` on *lock_path*, then (when *min_vram_mb* > 0)
+    waits until *probe* reports that much free, polling every *poll_s*. The lock
+    is held through the wait, so the next argus job cannot take the room first.
+    Both waits share one *timeout_s* deadline; past it :class:`GpuLeaseTimeout`
+    is raised and the lock released. A probe answering ``None`` grants the lease.
+    """
+
+    name = "lease"
+
+    def __init__(
+        self,
+        lock_path: str | None = None,
+        timeout_s: float = DEFAULT_TIMEOUT_S,
+        poll_s: float = DEFAULT_POLL_S,
+        device: int = 0,
+        probe: Callable[[int], int | None] = free_vram_mb,
+    ) -> None:
+        if not math.isfinite(timeout_s):
+            raise ValueError("timeout_s must be a finite number")
+        self.lock_path = lock_path or default_lock_path()
+        self.timeout_s = max(0.0, timeout_s)
+        self.poll_s = poll_s
+        self.device = device
+        self.probe = probe
+
+    @contextlib.contextmanager
+    def lease(self, *, caller: str, min_vram_mb: int) -> Iterator[None]:
+        try:
+            import fcntl  # noqa: PLC0415 - POSIX only
+        except ImportError as exc:  # pragma: no cover - non-POSIX host
+            raise RuntimeError("the 'lease' GPU coordinator needs POSIX fcntl; use 'none'") from exc
+
+        deadline = time.monotonic() + self.timeout_s
+        directory = os.path.dirname(self.lock_path)
+        if directory:
+            os.makedirs(directory, exist_ok=True)
+        # 0o666: services in different containers may run as different users.
+        # os.open's mode is masked by umask, so chmod the fd or the file is
+        # 0644/0600 and the next UID cannot open it.
+        fd = os.open(self.lock_path, os.O_CREAT | os.O_RDWR, 0o666)
+        try:
+            with contextlib.suppress(OSError):
+                os.fchmod(fd, 0o666)
+            while True:
+                try:
+                    fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
+                    break
+                except BlockingIOError:
+                    if time.monotonic() >= deadline:
+                        raise GpuLeaseTimeout(
+                            f"GPU lease not acquired within {self.timeout_s:g}s ({self.lock_path})"
+                        ) from None
+                    time.sleep(self.poll_s)
+            try:
+                self._wait_for_room(caller, min_vram_mb, deadline)
+                _log.debug("gpu lease acquired by %s (min %d MB, %s)", caller, min_vram_mb, self.lock_path)
+                yield
+            finally:
+                with contextlib.suppress(OSError):
+                    fcntl.flock(fd, fcntl.LOCK_UN)
+        finally:
+            os.close(fd)
+
+    def _wait_for_room(self, caller: str, min_vram_mb: int, deadline: float) -> None:
+        if min_vram_mb <= 0:
+            return
+        while True:
+            free = self.probe(self.device)
+            if free is None or free >= min_vram_mb:
+                return
+            if time.monotonic() >= deadline:
+                raise GpuLeaseTimeout(
+                    f"GPU {self.device} had {free} MB free, {caller} needs {min_vram_mb} MB, after {self.timeout_s:g}s"
+                )
+            time.sleep(self.poll_s)
+
+
+def build_coordinator(
+    name: str,
+    *,
+    lock_path: str | None = None,
+    timeout_s: float | None = None,
+    device: int = 0,
+) -> GpuCoordinator:
+    """A coordinator by name, one of :data:`COORDINATORS` (empty is ``none``)."""
+    key = (name or "none").strip().lower()
+    if key == "none":
+        return NullCoordinator()
+    if key == "lease":
+        return LeaseCoordinator(
+            lock_path=lock_path,
+            timeout_s=DEFAULT_TIMEOUT_S if timeout_s is None else timeout_s,
+            device=device,
+        )
+    raise ValueError(f"unknown GPU coordinator {name!r}; choose from: {', '.join(COORDINATORS)}")
+
+
+def _env_number(name: str, kind: type[float] | type[int]) -> float | int | None:
+    raw = os.environ.get(name, "").strip()
+    if not raw:
+        return None
+    try:
+        value = kind(raw)
+    except (ValueError, OverflowError):
+        value = -1
+    # inf/nan parse as float and never compare < 0, so the deadline never fires.
+    if isinstance(value, float) and not math.isfinite(value):
+        value = -1
+    if value < 0:
+        _log.warning("%s set to an unusable value %r; using the default", name, raw)
+        return None
+    return value
+
+
+def coordinator_from_env() -> GpuCoordinator:
+    """The coordinator the environment names; ``none`` when ``ARGUS_GPU_COORDINATOR`` is unset."""
+    device = _env_number("ARGUS_GPU_DEVICE", int)
+    return build_coordinator(
+        os.environ.get("ARGUS_GPU_COORDINATOR", "none"),
+        lock_path=os.environ.get("ARGUS_GPU_LEASE_PATH") or None,
+        timeout_s=_env_number("ARGUS_GPU_LEASE_TIMEOUT_S", float),
+        device=0 if device is None else int(device),
+    )

{argus_cortex-0.4.2 → argus_cortex-0.4.4}/src/argus_cortex/moderation/singguard.py

@@ -47,7 +47,9 @@ MAX_EDGE = 1536
 MAX_BYTES = 4 * 1024 * 1024
 _MIN_EDGE = 256  # below this, give up rather than send SingGuard a postage stamp
 _NEEDS_PILLOW = "downscaling images needs: pip install argus-cortex[images] (or pass downscale=False)"
-_EXPLANATION_MAX = 4000
+# Fits a 2048-token slow writeup plus a short cascade preface. 4000
+# cut recorded reasoning mid-sentence on cascade scans.
+_EXPLANATION_MAX = 16384
 _FIELD_MAX = 200
 
 

{argus_cortex-0.4.2 → argus_cortex-0.4.4}/src/argus_cortex/store/lineage.py

@@ -70,7 +70,13 @@ class LineageStore(Protocol):
         ...
 
     def record_asset(self, asset: SourceAsset) -> str | None:
-        """Insert (or reuse, by ``sha256``) a source asset; return its id."""
+        """Insert (or reuse, by ``sha256``) a source asset; return its id.
+
+        On reuse, ``uri`` becomes this call's; ``metadata`` merges into the
+        stored bag (this call's keys win); a ``None`` ``immich_id`` keeps the
+        stored one. So stages recording the same bytes add to the row rather
+        than erase each other's fields.
+        """
         ...
 
     def record_caption(self, asset_id: str, caption: Caption) -> str | None:
@@ -302,8 +308,12 @@ class PostgresLineageStore:
 
     def record_asset(self, asset: SourceAsset) -> str | None:
         # De-dupe on content hash: re-ingesting identical bytes returns the
-        # existing row (DO UPDATE ensures RETURNING fires even on conflict) and
-        # refreshes uri/immich_id/metadata to the latest call (last-write-wins).
+        # existing row (DO UPDATE ensures RETURNING fires even on conflict).
+        # Several stages record the same asset (quarry with provenance, lens
+        # with a caption), so a later call must not erase what an earlier one
+        # knew: uri is the latest location (last-write-wins), immich_id keeps
+        # the stored id when this call has none, and metadata merges key by
+        # key (jsonb ||: this call's keys win, the others are kept).
         # NOTE: dedup only works when sha256 is set — NULLs are distinct under the
         # UNIQUE index, so a hash-less asset inserts a fresh row every call.
         return self._insert(
@@ -312,8 +322,8 @@ class PostgresLineageStore:
             VALUES (%s, %s, %s, %s::jsonb)
             ON CONFLICT (sha256) DO UPDATE
                 SET uri = EXCLUDED.uri,
-                    immich_id = EXCLUDED.immich_id,
-                    metadata = EXCLUDED.metadata
+                    immich_id = COALESCE(EXCLUDED.immich_id, source_asset.immich_id),
+                    metadata = source_asset.metadata || EXCLUDED.metadata
             RETURNING id
             """,
             (asset.uri, asset.sha256, asset.immich_id, self._json(asset.metadata)),

argus_cortex-0.4.4/tests/test_gpu.py

@@ -0,0 +1,227 @@
+"""GPU coordination: the file lease, the VRAM wait, and configuration from the environment."""
+
+from __future__ import annotations
+
+import builtins
+import os
+import threading
+from pathlib import Path
+
+import pytest
+
+from argus_cortex import gpu
+
+
+@pytest.fixture(autouse=True)
+def _reset_nvml_ready(monkeypatch: pytest.MonkeyPatch) -> None:
+    monkeypatch.setattr(gpu, "_nvml_ready", None)
+
+
+def _lease(tmp_path: Path, probe=lambda device: None, timeout_s: float = 5.0) -> gpu.LeaseCoordinator:  # noqa: ANN001
+    return gpu.LeaseCoordinator(
+        lock_path=str(tmp_path / "locks" / "gpu.lock"), timeout_s=timeout_s, poll_s=0.01, probe=probe
+    )
+
+
+def test_null_coordinator_grants_at_once() -> None:
+    coordinator = gpu.NullCoordinator()
+    assert isinstance(coordinator, gpu.GpuCoordinator)
+    with coordinator.lease(caller="t", min_vram_mb=10**9):
+        pass
+
+
+def test_lock_file_is_world_writable_despite_umask(tmp_path: Path) -> None:
+    """os.open's mode is umask-masked; without fchmod the next container UID cannot open the lock."""
+    path = tmp_path / "locks" / "gpu.lock"
+    previous = os.umask(0o077)
+    try:
+        with gpu.LeaseCoordinator(lock_path=str(path), timeout_s=1, probe=lambda device: None).lease(
+            caller="a", min_vram_mb=0
+        ):
+            mode = path.stat().st_mode & 0o777
+    finally:
+        os.umask(previous)
+    assert mode == 0o666
+
+
+def test_lease_released_when_the_body_raises(tmp_path: Path) -> None:
+    with (
+        pytest.raises(RuntimeError, match="boom"),
+        _lease(tmp_path).lease(caller="a", min_vram_mb=0),
+    ):
+        raise RuntimeError("boom")
+    with _lease(tmp_path, timeout_s=0.05).lease(caller="b", min_vram_mb=0):
+        pass
+
+
+def test_lease_is_exclusive_and_released(tmp_path: Path) -> None:
+    """A second holder waits for the first; a holder that never lets go times the other out."""
+    holder, waiter = _lease(tmp_path), _lease(tmp_path, timeout_s=0.1)
+    assert isinstance(holder, gpu.GpuCoordinator)
+    with (
+        holder.lease(caller="a", min_vram_mb=0),
+        pytest.raises(gpu.GpuLeaseTimeout, match="not acquired"),
+        waiter.lease(caller="b", min_vram_mb=0),
+    ):
+        pass
+    with waiter.lease(caller="b", min_vram_mb=0):  # free again once the first exits
+        pass
+
+
+def test_lease_waits_for_the_holder_to_finish(tmp_path: Path) -> None:
+    order: list[str] = []
+    entered, release = threading.Event(), threading.Event()
+
+    def first() -> None:
+        with _lease(tmp_path).lease(caller="a", min_vram_mb=0):
+            entered.set()
+            release.wait(5)
+            order.append("a done")
+
+    thread = threading.Thread(target=first)
+    thread.start()
+    assert entered.wait(5)
+    threading.Timer(0.05, release.set).start()
+    with _lease(tmp_path).lease(caller="b", min_vram_mb=0):
+        order.append("b in")
+    thread.join(5)
+    assert order == ["a done", "b in"]
+
+
+def test_lease_waits_for_free_vram_while_holding_the_lock(tmp_path: Path) -> None:
+    readings = iter([1000, 2000, 9000])
+    seen: list[int] = []
+
+    def probe(device: int) -> int | None:
+        seen.append(device)
+        return next(readings)
+
+    coordinator = _lease(tmp_path, probe=probe)
+    # The lock is held during and after the wait: another argus job cannot take the room.
+    with (
+        coordinator.lease(caller="faces", min_vram_mb=4000),
+        pytest.raises(gpu.GpuLeaseTimeout),
+        _lease(tmp_path, timeout_s=0.05).lease(caller="lens", min_vram_mb=0),
+    ):
+        pass
+    assert seen == [0, 0, 0]
+
+
+def test_lease_times_out_when_the_card_never_has_room_and_lets_go(tmp_path: Path) -> None:
+    coordinator = _lease(tmp_path, probe=lambda device: 500, timeout_s=0.05)
+    with (
+        pytest.raises(gpu.GpuLeaseTimeout, match="500 MB free, faces needs 4000 MB"),
+        coordinator.lease(caller="faces", min_vram_mb=4000),
+    ):
+        pass
+    with _lease(tmp_path, timeout_s=0.05).lease(caller="next", min_vram_mb=0):  # the lock was released
+        pass
+
+
+def test_lease_proceeds_when_vram_cannot_be_read_or_is_not_needed(tmp_path: Path) -> None:
+    calls: list[int] = []
+    with _lease(tmp_path, probe=lambda device: calls.append(device)).lease(caller="cpu host", min_vram_mb=4000):
+        pass
+    assert calls == [0]  # asked once, got None, went ahead
+    with _lease(tmp_path, probe=lambda device: pytest.fail("probed")).lease(caller="tiny", min_vram_mb=0):
+        pass
+
+
+def test_free_vram_reads_bytes_for_the_asked_device(monkeypatch: pytest.MonkeyPatch) -> None:
+    import sys
+    import types
+
+    fake = types.ModuleType("pynvml")
+    seen: list[int] = []
+
+    class _Info:
+        free = 3 * 1024 * 1024 * 1024
+
+    def handle(index: int) -> str:
+        seen.append(index)
+        return f"dev{index}"
+
+    def memory_info(dev: str) -> _Info:
+        assert dev == "dev2"
+        return _Info()
+
+    fake.nvmlInit = lambda: None  # type: ignore[attr-defined]
+    fake.nvmlDeviceGetHandleByIndex = handle  # type: ignore[attr-defined]
+    fake.nvmlDeviceGetMemoryInfo = memory_info  # type: ignore[attr-defined]
+    monkeypatch.setitem(sys.modules, "pynvml", fake)
+    assert gpu.free_vram_mb(2) == 3072
+    assert seen == [2]
+
+
+def test_free_vram_is_none_without_nvml(monkeypatch: pytest.MonkeyPatch) -> None:
+    real_import = builtins.__import__
+
+    def no_pynvml(name: str, *args, **kwargs):  # noqa: ANN002, ANN003, ANN202
+        if name == "pynvml":
+            raise ImportError(name)
+        return real_import(name, *args, **kwargs)
+
+    monkeypatch.setattr(builtins, "__import__", no_pynvml)
+    assert gpu.free_vram_mb() is None
+
+
+def test_free_vram_is_none_when_nvml_cannot_start(monkeypatch: pytest.MonkeyPatch) -> None:
+    """No driver: NVML init fails once and is not retried on every probe."""
+    import sys
+    import types
+
+    attempts: list[int] = []
+    fake = types.ModuleType("pynvml")
+
+    def init() -> None:
+        attempts.append(1)
+        raise RuntimeError("NVML Shared Library Not Found")
+
+    fake.nvmlInit = init  # type: ignore[attr-defined]
+    monkeypatch.setitem(sys.modules, "pynvml", fake)
+    monkeypatch.setattr(gpu, "_nvml_ready", None)
+    assert gpu.free_vram_mb() is None and gpu.free_vram_mb() is None
+    assert attempts == [1]
+
+
+def test_build_coordinator() -> None:
+    assert isinstance(gpu.build_coordinator(""), gpu.NullCoordinator)
+    assert isinstance(gpu.build_coordinator(" NONE "), gpu.NullCoordinator)
+    lease = gpu.build_coordinator("lease", lock_path="/x/gpu.lock", timeout_s=7, device=1)
+    assert isinstance(lease, gpu.LeaseCoordinator)
+    assert (lease.lock_path, lease.timeout_s, lease.device) == ("/x/gpu.lock", 7, 1)
+    assert gpu.build_coordinator("lease").lock_path == gpu.default_lock_path()
+    with pytest.raises(ValueError, match="none, lease"):
+        gpu.build_coordinator("gothmog")
+    with pytest.raises(ValueError, match="finite"):
+        gpu.LeaseCoordinator(timeout_s=float("inf"))
+    with pytest.raises(ValueError, match="finite"):
+        gpu.LeaseCoordinator(timeout_s=float("nan"))
+
+
+def test_coordinator_from_env(monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture) -> None:
+    for name in ("ARGUS_GPU_COORDINATOR", "ARGUS_GPU_LEASE_PATH", "ARGUS_GPU_LEASE_TIMEOUT_S", "ARGUS_GPU_DEVICE"):
+        monkeypatch.delenv(name, raising=False)
+    assert isinstance(gpu.coordinator_from_env(), gpu.NullCoordinator)
+    monkeypatch.setenv("ARGUS_GPU_COORDINATOR", "lease")
+    monkeypatch.setenv("ARGUS_GPU_LEASE_PATH", "/shared/argus-gpu.lock")
+    monkeypatch.setenv("ARGUS_GPU_LEASE_TIMEOUT_S", "30")
+    monkeypatch.setenv("ARGUS_GPU_DEVICE", "1")
+    lease = gpu.coordinator_from_env()
+    assert (lease.lock_path, lease.timeout_s, lease.device) == ("/shared/argus-gpu.lock", 30.0, 1)
+    monkeypatch.setenv("ARGUS_GPU_LEASE_TIMEOUT_S", "soon")
+    monkeypatch.setenv("ARGUS_GPU_DEVICE", "-1")
+    with caplog.at_level("WARNING"):
+        lease = gpu.coordinator_from_env()
+    assert (lease.timeout_s, lease.device) == (gpu.DEFAULT_TIMEOUT_S, 0)
+    assert "ARGUS_GPU_LEASE_TIMEOUT_S" in caplog.text and "ARGUS_GPU_DEVICE" in caplog.text
+    caplog.clear()
+    monkeypatch.setenv("ARGUS_GPU_LEASE_TIMEOUT_S", "inf")
+    with caplog.at_level("WARNING"):
+        assert gpu.coordinator_from_env().timeout_s == gpu.DEFAULT_TIMEOUT_S
+    assert "ARGUS_GPU_LEASE_TIMEOUT_S" in caplog.text
+    caplog.clear()
+    monkeypatch.setenv("ARGUS_GPU_LEASE_TIMEOUT_S", "nan")
+    with caplog.at_level("WARNING"):
+        assert gpu.coordinator_from_env().timeout_s == gpu.DEFAULT_TIMEOUT_S
+    assert "ARGUS_GPU_LEASE_TIMEOUT_S" in caplog.text

{argus_cortex-0.4.2 → argus_cortex-0.4.4}/tests/test_moderation.py

@@ -253,6 +253,24 @@ def test_singguard_config() -> None:
         SingGuardClient("gopher://g")
 
 
+def test_to_result_keeps_a_long_explanation() -> None:
+    writeup = "Cascade: fast flagged this; the slow pass is the verdict.\n\n" + ("A posed adult. " * 400)
+    writeup = writeup.strip()
+    assert 4000 < len(writeup) < singguard._EXPLANATION_MAX
+    result = singguard.to_result(
+        {"safe": True, "verdict": "safe", "rule": "Safe", "explanation": writeup, "categories": []}
+    )
+    assert result.explanation == writeup
+
+
+def test_to_result_clips_explanation_at_max() -> None:
+    exact = "y" * singguard._EXPLANATION_MAX
+    over = exact + "z"
+    payload = {"safe": True, "verdict": "safe", "rule": "Safe", "categories": []}
+    assert singguard.to_result({**payload, "explanation": exact}).explanation == exact
+    assert singguard.to_result({**payload, "explanation": over}).explanation == exact
+
+
 def test_singguard_check(singguard_server) -> None:
     fake, url = singguard_server
     client = SingGuardClient(url, timeout=5, thinking="fast-slow", policy="### A. Minors")

{argus_cortex-0.4.2 → argus_cortex-0.4.4}/tests/test_store.py

@@ -206,9 +206,11 @@ def test_record_asset_returns_id_and_upserts_on_sha() -> None:
     sql, params = conn.executed[-1]
     assert "INSERT INTO source_asset" in sql
     assert "ON CONFLICT (sha256)" in sql
-    # on a dedup hit, uri/immich_id/metadata are all refreshed (not just uri)
-    assert "immich_id = EXCLUDED.immich_id" in sql
-    assert "metadata = EXCLUDED.metadata" in sql
+    # on a dedup hit uri is refreshed, but a call without an immich_id or a
+    # metadata key keeps the stored one (another stage may have recorded it)
+    assert "uri = EXCLUDED.uri" in sql
+    assert "immich_id = COALESCE(EXCLUDED.immich_id, source_asset.immich_id)" in sql
+    assert "metadata = source_asset.metadata || EXCLUDED.metadata" in sql
     # metadata is serialised to a JSON string for the ::jsonb cast
     assert params == ("immich://1", "abc", None, '{"k": "v"}')
 

{argus_cortex-0.4.2 → argus_cortex-0.4.4}/tests/test_store_pg_integration.py

@@ -81,6 +81,20 @@ def test_transaction_rolls_back_on_error(store: PostgresLineageStore) -> None:
     assert reused is not None
 
 
+def test_reupsert_merges_metadata_and_keeps_immich_id(store: PostgresLineageStore) -> None:
+    # quarry records provenance first; lens later records the same bytes with
+    # none: the provenance and immich id survive, the uri moves, shared keys update.
+    sha = uuid.uuid4().hex
+    first = store.record_asset(
+        SourceAsset(uri="quarry://a", sha256=sha, immich_id="im-1", metadata={"license": "cc0", "seen": 1})
+    )
+    again = store.record_asset(SourceAsset(uri="/data/images/a.png", sha256=sha, metadata={"seen": 2}))
+    assert again == first
+    with store._acquire() as conn:
+        row = conn.execute("SELECT uri, immich_id, metadata FROM source_asset WHERE id = %s", (first,)).fetchone()
+    assert row == ("/data/images/a.png", "im-1", {"license": "cc0", "seen": 2})
+
+
 def test_concurrent_writes_are_isolated(store: PostgresLineageStore) -> None:
     # Many concurrent record_asset calls each borrow their own pooled connection;
     # all succeed with distinct ids (no "another operation in progress").