pyuncommonkit 0.1.0__tar.gz

pyuncommonkit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Sean D. True
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pyuncommonkit-0.1.0/PKG-INFO

@@ -0,0 +1,200 @@
+Metadata-Version: 2.4
+Name: pyuncommonkit
+Version: 0.1.0
+Summary: Named shared-memory segments for numpy arrays: publish once, attach from any process.
+Keywords: numpy,npy,mmap,shared-memory,cache
+Author: Sean D. True
+Author-email: Sean D. True <sean.true@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Filesystems
+Requires-Dist: numpy>=1.26
+Requires-Dist: pytest>=8 ; extra == 'dev'
+Requires-Dist: ruff>=0.6 ; extra == 'dev'
+Requires-Python: >=3.11
+Project-URL: Homepage, https://github.com/seantrue/pycommonkit
+Project-URL: Repository, https://github.com/seantrue/pycommonkit
+Project-URL: Issues, https://github.com/seantrue/pycommonkit/issues
+Project-URL: Contract, https://github.com/seantrue/pycommonkit/blob/main/docs/contract.md
+Project-URL: Rust crate, https://crates.io/crates/commonkit
+Provides-Extra: dev
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/seantrue/pycommonkit/main/docs/images/commons-banner.jpg"
+       alt="Illustrated aerial view of farmland: square fields and center-pivot circles on a shared road grid"
+       width="100%">
+</p>
+
+# pycommonkit
+
+Named shared-memory segments for numpy arrays. One process decodes and
+publishes; every other process attaches the same physical pages.
+
+Installed as **`pyuncommonkit`**, imported as **`commonkit`** (the PyPI name `commonkit` belongs
+to an unrelated project, and PyPI refused `pycommonkit` as too similar to it). A Rust crate, `commonkit`, implementing the same on-disk contract is
+in this repository under `rust/` (`docs/rust-crate-plan.md`): keys, segments, the store and the
+reaper are done, and a cross-language conformance suite runs both on one tree.
+
+The name is for Fortran's **named `COMMON` block**. `COMMON /GRID/ X, Y` in one
+subroutine and the same name in another, compiled separately, give both the same
+storage: the linker joins them by the name alone, and no unit owns the block. The
+name is the whole interface. A commonkit key is the same idea: any process that
+can spell the key reaches the same bytes, without knowing who wrote them.
+
+Where the analogy stops is where commonkit had to be stricter:
+
+* **Across processes, not within one program.** A `COMMON` block is shared by the
+  program units of one executable; a segment is shared by separate processes.
+* **The layout travels with the data.** Each Fortran unit declares its own view
+  of a block, and a unit that declares it differently silently reads other
+  variables' bits. A segment carries its dtype, shape and order in its `.npy`
+  header, so every reader sees the same layout, and publishing a different array
+  under an existing key is refused loudly (`KeyCollision`).
+* **Written once, then read-only.** Any unit may assign into a `COMMON` block at
+  any time. A segment is written once, whole, and readers map it read-only.
+* **It may not be there.** A `COMMON` block exists for the whole run. A segment
+  exists only until the reaper needs the space, which is why every caller must be
+  able to recompute it (below).
+
+## The contract
+
+`docs/contract.md` (version 1) is the authority on the on-disk format: keys, layout, the `.npy`
+segment, publishing, the write-once rules, touch, the reaper, policy and counters. Python is the
+reference implementation and `conformance/vectors/` decides any disagreement
+(`conformance/generate.py` regenerates the fixtures byte for byte). Both the Python package and
+the Rust crate pass every vector.
+
+## Status
+
+See `docs/STATUS.md`. Nothing is published to PyPI or crates.io yet.
+
+## What it is
+
+* `SharedArrayStore` — publish/attach `.npy` segments under a root directory,
+  with admission gates, a TTL reaper and a byte budget.
+* `KeyScheme` — the addressing vocabulary for one *domain*. A scheme fixes the
+  domain, a version, the legal forms and the legal variants, and turns
+  `(digest, variant, form)` into both a key and a relative path.
+* `shared_store()` — the ONE store per host that every user shares. Its byte budget, TTL,
+  free-space floor, fsync and republish verification come only from the tree's `.policy.json`
+  (which wins), then `COMMONKIT_STORE_*` environment variables, then the defaults (macOS:
+  `~/Library/Caches/commonkit/store`, 2 GB, 900 s, 8 GB free), because the reaping process applies
+  its own policy to the whole tree; the root comes from the environment or the default. A caller
+  chooses only `mode` and the smallest array worth storing.
+* `TieredPixelCache` — an in-process L1 memo in front of a store, so a loop
+  that looks up the same digest repeatedly pays one attach rather than one per
+  call.
+
+## A cache, not a store of record
+
+**A missing segment is never an error, only a slower path.** The store promises
+that a key which is present names the right bytes. It never promises that a key
+is present. So every caller must be able to compute the value again when
+`attach` returns `None`, and must treat a `publish` that returns `False` as
+normal.
+
+A segment can be absent at any moment, for reasons the caller doesn't control:
+
+* **Its publish was declined.** Under pressure the store refuses rather than
+  blocks: the array was smaller than worth storing or larger than allowed, the
+  volume was below its free-space floor, or the tree was over its byte budget.
+  A decline is how the store survives back-pressure, not a fault.
+* **The reaper removed it.** Segments are evicted when they expire, least
+  recently used first to get back under the byte budget, and to restore the
+  free-space floor. The reaper doesn't know who might still want a segment, and
+  doesn't ask: no reference counts, no coordination between processes.
+* **It was torn.** A file shorter than its header promises, after a crash with
+  `fsync` off (the default) or damage from outside the contract, is a miss.
+* **The store is disabled.** The mode is `off`, or the root can't be created or
+  written: every attach misses and every publish declines.
+* **The tree is gone.** On Linux it lives in `/dev/shm` and does not survive a
+  reboot; any root can be cleared; another machine has its own tree.
+
+That one rule is what makes the rest cheap. The reaper can unlink freely,
+publish never blocks, writes need no durability, and a process that fails
+costs the others only time: the next reader recomputes.
+
+**What does raise.** Exactly two things, both bugs at the call site rather than
+conditions of the store: building a key that isn't valid for its scheme
+(`ValueError`), and publishing a *different* array under a key that already
+holds one (`KeyCollision`, a `ValueError` subclass), which means a key scheme or
+a digest is broken. Everything else (an absent root, a full filesystem, a
+reaped or torn segment, a hostile `.usage.json` or `.policy.json`) returns
+`None` or `False` and ticks a counter. The constructor never raises: an
+unusable root leaves the store `disabled` and every method a no-op.
+
+**Replacing a value under the same key.** A segment is written once: publishing
+the same array again only refreshes it, and a different array raises
+`KeyCollision`. A caller that must replace a value whose key hasn't changed (a
+recompute after a fix the key doesn't capture) calls `remove(key)` first, then
+publishes. `remove` only unlinks, so a reader still holding the old segment keeps
+its bytes; there is no in-use check.
+
+**A value that can't be recomputed doesn't belong here alone.** Keep its
+durable copy somewhere you own and use the store to share it fast. (A pinned
+subtree the reaper skips is a possible later addition; it isn't built.)
+
+**Callers never need `.usage.json`.** It is the reaper's record of how much the
+tree holds, read only by the byte-budget gate. When it is missing or stale,
+usage is unknown, that gate passes, and the free-space floor still applies;
+the next complete reap writes a fresh one.
+
+## Keys
+
+    {domain}:{version}:{form}:{digest}:{variant}
+
+and the path it names:
+
+    {root}/{form}/{digest[:2]}/{digest}.{variant}.npy
+
+Two-level sharding, so no directory holds a million entries. The `variant` axis
+is whatever distinguishes two renderings of the same source: for images it is
+orientation (`o0`, `o1`, `oa`), for audio it would be sample rate (`sr16000`).
+Schemes are registered by domain so `parse()` and `relpath()` can dispatch on a
+key alone, which is what lets one store hold several domains.
+
+## Tests
+
+    uv run --extra dev pytest -q          # includes the conformance vectors
+
+The suite addresses audio and video rather than images, to show the package is
+not pixel-specific. Two modules run end to end against real media and skip
+themselves when their tool is not on `PATH`:
+
+* `tests/test_sox.py` — `sox` synthesizes a sine wave and dumps it as raw
+  int16 (and float32) samples. The samples are published, attached back as a
+  read-only `np.memmap`, attached again from a separate process, and stored at
+  two sample rates as two variants (`sr16000`, `sr44100`) of one digest.
+* `tests/test_ffmpeg.py` — `ffmpeg` renders its `testsrc` pattern to
+  `rawvideo` frames, stored as one `(T, H, W, C)` array. The same checks, with
+  the frame rate as the variant (`fr10`, `fr25`), plus a clip over
+  `max_array_bytes` being declined whole rather than truncated.
+
+Both use their tool's built-in generators, so no media files are checked in.
+Both also exercise per-form dtypes: `dtypes` maps a *form* to the dtype it
+requires (`{"wav16k": "int16"}`), and a float dump is refused under the
+integer form and accepted under the float one. A form's name is only a label —
+the store enforces a dtype only for forms listed in `dtypes`.
+
+## Origin
+
+Lifted on 2026-10-27 from `seantrue/commonkit` (`main` at `4dbfc7d`, plus the contract and
+vectors from its `rust-crate-plan` branch at `2b6bcf9` and after), with a fresh history. That
+package was extracted from the pixel-cache layer of a private image-corpus
+application, where it ran in production first. The measurements in the module
+docstrings were taken there; the framework-coupled half (the settings wiring
+and the decode entry point) deliberately stayed behind.

pyuncommonkit-0.1.0/README.md

@@ -0,0 +1,165 @@
+<p align="center">
+  <img src="https://raw.githubusercontent.com/seantrue/pycommonkit/main/docs/images/commons-banner.jpg"
+       alt="Illustrated aerial view of farmland: square fields and center-pivot circles on a shared road grid"
+       width="100%">
+</p>
+
+# pycommonkit
+
+Named shared-memory segments for numpy arrays. One process decodes and
+publishes; every other process attaches the same physical pages.
+
+Installed as **`pyuncommonkit`**, imported as **`commonkit`** (the PyPI name `commonkit` belongs
+to an unrelated project, and PyPI refused `pycommonkit` as too similar to it). A Rust crate, `commonkit`, implementing the same on-disk contract is
+in this repository under `rust/` (`docs/rust-crate-plan.md`): keys, segments, the store and the
+reaper are done, and a cross-language conformance suite runs both on one tree.
+
+The name is for Fortran's **named `COMMON` block**. `COMMON /GRID/ X, Y` in one
+subroutine and the same name in another, compiled separately, give both the same
+storage: the linker joins them by the name alone, and no unit owns the block. The
+name is the whole interface. A commonkit key is the same idea: any process that
+can spell the key reaches the same bytes, without knowing who wrote them.
+
+Where the analogy stops is where commonkit had to be stricter:
+
+* **Across processes, not within one program.** A `COMMON` block is shared by the
+  program units of one executable; a segment is shared by separate processes.
+* **The layout travels with the data.** Each Fortran unit declares its own view
+  of a block, and a unit that declares it differently silently reads other
+  variables' bits. A segment carries its dtype, shape and order in its `.npy`
+  header, so every reader sees the same layout, and publishing a different array
+  under an existing key is refused loudly (`KeyCollision`).
+* **Written once, then read-only.** Any unit may assign into a `COMMON` block at
+  any time. A segment is written once, whole, and readers map it read-only.
+* **It may not be there.** A `COMMON` block exists for the whole run. A segment
+  exists only until the reaper needs the space, which is why every caller must be
+  able to recompute it (below).
+
+## The contract
+
+`docs/contract.md` (version 1) is the authority on the on-disk format: keys, layout, the `.npy`
+segment, publishing, the write-once rules, touch, the reaper, policy and counters. Python is the
+reference implementation and `conformance/vectors/` decides any disagreement
+(`conformance/generate.py` regenerates the fixtures byte for byte). Both the Python package and
+the Rust crate pass every vector.
+
+## Status
+
+See `docs/STATUS.md`. Nothing is published to PyPI or crates.io yet.
+
+## What it is
+
+* `SharedArrayStore` — publish/attach `.npy` segments under a root directory,
+  with admission gates, a TTL reaper and a byte budget.
+* `KeyScheme` — the addressing vocabulary for one *domain*. A scheme fixes the
+  domain, a version, the legal forms and the legal variants, and turns
+  `(digest, variant, form)` into both a key and a relative path.
+* `shared_store()` — the ONE store per host that every user shares. Its byte budget, TTL,
+  free-space floor, fsync and republish verification come only from the tree's `.policy.json`
+  (which wins), then `COMMONKIT_STORE_*` environment variables, then the defaults (macOS:
+  `~/Library/Caches/commonkit/store`, 2 GB, 900 s, 8 GB free), because the reaping process applies
+  its own policy to the whole tree; the root comes from the environment or the default. A caller
+  chooses only `mode` and the smallest array worth storing.
+* `TieredPixelCache` — an in-process L1 memo in front of a store, so a loop
+  that looks up the same digest repeatedly pays one attach rather than one per
+  call.
+
+## A cache, not a store of record
+
+**A missing segment is never an error, only a slower path.** The store promises
+that a key which is present names the right bytes. It never promises that a key
+is present. So every caller must be able to compute the value again when
+`attach` returns `None`, and must treat a `publish` that returns `False` as
+normal.
+
+A segment can be absent at any moment, for reasons the caller doesn't control:
+
+* **Its publish was declined.** Under pressure the store refuses rather than
+  blocks: the array was smaller than worth storing or larger than allowed, the
+  volume was below its free-space floor, or the tree was over its byte budget.
+  A decline is how the store survives back-pressure, not a fault.
+* **The reaper removed it.** Segments are evicted when they expire, least
+  recently used first to get back under the byte budget, and to restore the
+  free-space floor. The reaper doesn't know who might still want a segment, and
+  doesn't ask: no reference counts, no coordination between processes.
+* **It was torn.** A file shorter than its header promises, after a crash with
+  `fsync` off (the default) or damage from outside the contract, is a miss.
+* **The store is disabled.** The mode is `off`, or the root can't be created or
+  written: every attach misses and every publish declines.
+* **The tree is gone.** On Linux it lives in `/dev/shm` and does not survive a
+  reboot; any root can be cleared; another machine has its own tree.
+
+That one rule is what makes the rest cheap. The reaper can unlink freely,
+publish never blocks, writes need no durability, and a process that fails
+costs the others only time: the next reader recomputes.
+
+**What does raise.** Exactly two things, both bugs at the call site rather than
+conditions of the store: building a key that isn't valid for its scheme
+(`ValueError`), and publishing a *different* array under a key that already
+holds one (`KeyCollision`, a `ValueError` subclass), which means a key scheme or
+a digest is broken. Everything else (an absent root, a full filesystem, a
+reaped or torn segment, a hostile `.usage.json` or `.policy.json`) returns
+`None` or `False` and ticks a counter. The constructor never raises: an
+unusable root leaves the store `disabled` and every method a no-op.
+
+**Replacing a value under the same key.** A segment is written once: publishing
+the same array again only refreshes it, and a different array raises
+`KeyCollision`. A caller that must replace a value whose key hasn't changed (a
+recompute after a fix the key doesn't capture) calls `remove(key)` first, then
+publishes. `remove` only unlinks, so a reader still holding the old segment keeps
+its bytes; there is no in-use check.
+
+**A value that can't be recomputed doesn't belong here alone.** Keep its
+durable copy somewhere you own and use the store to share it fast. (A pinned
+subtree the reaper skips is a possible later addition; it isn't built.)
+
+**Callers never need `.usage.json`.** It is the reaper's record of how much the
+tree holds, read only by the byte-budget gate. When it is missing or stale,
+usage is unknown, that gate passes, and the free-space floor still applies;
+the next complete reap writes a fresh one.
+
+## Keys
+
+    {domain}:{version}:{form}:{digest}:{variant}
+
+and the path it names:
+
+    {root}/{form}/{digest[:2]}/{digest}.{variant}.npy
+
+Two-level sharding, so no directory holds a million entries. The `variant` axis
+is whatever distinguishes two renderings of the same source: for images it is
+orientation (`o0`, `o1`, `oa`), for audio it would be sample rate (`sr16000`).
+Schemes are registered by domain so `parse()` and `relpath()` can dispatch on a
+key alone, which is what lets one store hold several domains.
+
+## Tests
+
+    uv run --extra dev pytest -q          # includes the conformance vectors
+
+The suite addresses audio and video rather than images, to show the package is
+not pixel-specific. Two modules run end to end against real media and skip
+themselves when their tool is not on `PATH`:
+
+* `tests/test_sox.py` — `sox` synthesizes a sine wave and dumps it as raw
+  int16 (and float32) samples. The samples are published, attached back as a
+  read-only `np.memmap`, attached again from a separate process, and stored at
+  two sample rates as two variants (`sr16000`, `sr44100`) of one digest.
+* `tests/test_ffmpeg.py` — `ffmpeg` renders its `testsrc` pattern to
+  `rawvideo` frames, stored as one `(T, H, W, C)` array. The same checks, with
+  the frame rate as the variant (`fr10`, `fr25`), plus a clip over
+  `max_array_bytes` being declined whole rather than truncated.
+
+Both use their tool's built-in generators, so no media files are checked in.
+Both also exercise per-form dtypes: `dtypes` maps a *form* to the dtype it
+requires (`{"wav16k": "int16"}`), and a float dump is refused under the
+integer form and accepted under the float one. A form's name is only a label —
+the store enforces a dtype only for forms listed in `dtypes`.
+
+## Origin
+
+Lifted on 2026-10-27 from `seantrue/commonkit` (`main` at `4dbfc7d`, plus the contract and
+vectors from its `rust-crate-plan` branch at `2b6bcf9` and after), with a fresh history. That
+package was extracted from the pixel-cache layer of a private image-corpus
+application, where it ran in production first. The measurements in the module
+docstrings were taken there; the framework-coupled half (the settings wiring
+and the decode entry point) deliberately stayed behind.

pyuncommonkit-0.1.0/pyproject.toml

@@ -0,0 +1,59 @@
+[project]
+name = "pyuncommonkit"
+version = "0.1.0"
+description = "Named shared-memory segments for numpy arrays: publish once, attach from any process."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.11"
+dependencies = ["numpy>=1.26"]
+keywords = [
+    "numpy",
+    "npy",
+    "mmap",
+    "shared-memory",
+    "cache",
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Operating System :: MacOS",
+    "Operating System :: POSIX :: Linux",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Scientific/Engineering",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+    "Topic :: System :: Filesystems",
+]
+
+[[project.authors]]
+name = "Sean D. True"
+email = "sean.true@gmail.com"
+
+[project.urls]
+Homepage = "https://github.com/seantrue/pycommonkit"
+Repository = "https://github.com/seantrue/pycommonkit"
+Issues = "https://github.com/seantrue/pycommonkit/issues"
+Contract = "https://github.com/seantrue/pycommonkit/blob/main/docs/contract.md"
+"Rust crate" = "https://crates.io/crates/commonkit"
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8",
+    "ruff>=0.6",
+]
+
+[build-system]
+requires = ["uv_build>=0.11.21,<0.12.0"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+module-name = "commonkit"
+
+[tool.ruff]
+line-length = 100

pyuncommonkit-0.1.0/pyproject.toml.orig

@@ -0,0 +1,61 @@
+[project]
+name = "pyuncommonkit"
+version = "0.1.0"
+description = "Named shared-memory segments for numpy arrays: publish once, attach from any process."
+readme = "README.md"
+authors = [
+    { name = "Sean D. True", email = "sean.true@gmail.com" }
+]
+license = "MIT"
+license-files = ["LICENSE"]
+# 3.11, NOT 3.12: the model venvs this exists to serve are python 3.11, and
+# being importable there is the whole point of the package.
+requires-python = ">=3.11"
+dependencies = [
+    "numpy>=1.26",
+]
+keywords = ["numpy", "npy", "mmap", "shared-memory", "cache"]
+# No License :: classifier: PEP 639's `license` expression replaces it. Linux and macOS only:
+# Windows is supported in the contract (§12) but has no Python default root or CI yet.
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Operating System :: MacOS",
+    "Operating System :: POSIX :: Linux",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Scientific/Engineering",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+    "Topic :: System :: Filesystems",
+]
+
+[project.urls]
+Homepage = "https://github.com/seantrue/pycommonkit"
+Repository = "https://github.com/seantrue/pycommonkit"
+Issues = "https://github.com/seantrue/pycommonkit/issues"
+Contract = "https://github.com/seantrue/pycommonkit/blob/main/docs/contract.md"
+"Rust crate" = "https://crates.io/crates/commonkit"
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8",
+    "ruff>=0.6",
+]
+
+[build-system]
+requires = ["uv_build>=0.11.21,<0.12.0"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+# Published as pyuncommonkit, imported as commonkit, so no consumer's code changes; only its
+# dependency line does. PyPI `commonkit` belongs to an unrelated project, and PyPI refused
+# `pycommonkit` as too similar to it (2026-10-28).
+module-name = "commonkit"
+
+[tool.ruff]
+line-length = 100

pyuncommonkit-0.1.0/src/commonkit/__init__.py

@@ -0,0 +1,19 @@
+"""Named shared-memory segments for numpy arrays.
+
+The name is for Fortran's ``COMMON`` block: a named region several separately
+compiled programs agree to share. The name is the whole interface, and the
+segment outlives whoever wrote it.
+
+``store`` is the mechanism and carries the measured reasons behind it, ``keys``
+the addressing, ``cache`` the in-process memo that keeps a tight loop from
+paying an attach per call.
+"""
+from .cache import TieredPixelCache
+from .keys import KeyScheme, parse, register, relpath, scheme_for
+from .shared import shared_store
+from .store import KeyCollision, SharedArrayStore
+
+__all__ = [
+    "KeyCollision", "KeyScheme", "SharedArrayStore", "TieredPixelCache",
+    "parse", "register", "relpath", "scheme_for", "shared_store",
+]

pyuncommonkit-0.1.0/src/commonkit/cache.py

@@ -0,0 +1,125 @@
+"""Two-tier pixel cache: an in-process memo over the shared store.
+
+Why two tiers and not a straight replacement, in one measurement:
+
+The consumer's face-crop path calls its pixel accessor **twice per face**
+-- once for the base dimensions, once for the crop. An 18-face page is
+therefore 36 lookups. Today that is one 38 ms decode plus 35 free in-process
+hits. Against a shared store alone it would be 36 attaches at ~3.3 ms each =
+119 ms, **three times slower than doing nothing at all**. Attaching is cheap, but
+it is not free, and a loop that attaches per crop is the one access pattern that
+turns a cache into a pessimization.
+
+So L1 memoises the ATTACHED MAPPING, not a second copy of the pixels. The first
+lookup attaches; every later lookup in that process is a dict hit. The pixels
+still live once in shared memory and are still shared across processes -- L1 adds
+no bytes, it removes syscalls.
+
+Tier order: L1 in-process memo -> L2 shared store -> decode.
+"""
+from __future__ import annotations
+
+from collections import Counter
+
+import numpy as np
+
+from .store import KeyCollision
+
+
+class TieredPixelCache:
+    """Satisfies the consumer's ``PixelCache`` protocol.
+
+    ``l1`` is anything with ``get``/``set_array`` (the original consumer
+    passes an LRU array cache); ``l2`` is a
+    :class:`~commonkit.store.SharedArrayStore`. Either may be None,
+    which degrades to the other -- and with both None this is a no-op cache,
+    leaving its caller exactly as it would be with no cache at all.
+    """
+
+    def __init__(self, l1=None, l2=None):
+        self.l1 = l1
+        self.l2 = l2
+        self.stats: Counter = Counter()
+
+    # ``_base_array`` checks for this and prefers it over the bytes path.
+    def get(self, key: str):
+        """The PixelCache protocol: the array, or None."""
+        return self.get_with_source(key)[0]
+
+    def get_with_source(self, key: str):
+        """``(array, source)`` -- which TIER answered, or ``(None, "decode")``.
+
+        The return channel this class used to lack, and the lack of which was a
+        real defect rather than an inelegance. ``entry.decoded`` could not ask
+        what happened, so it inferred it from counter DELTAS -- reading
+        ``shm_hit`` off this object, which deliberately does not tick it because
+        the store already does. The inference silently defaulted to "decode",
+        so every cross-process attach was reported as a decode.
+
+        Answering directly deletes the inference instead of correcting it. It is
+        also the only thread-safe shape: this cache is a module-global shared
+        across a node's CONCURRENCY threads, so recording "what I did last" on
+        ``self`` would be a race between them.
+        """
+        if self.l1 is not None:
+            arr = self.l1.get(key)
+            if arr is not None:
+                self.stats["l1_hit"] += 1
+                return arr, "l1"
+        if self.l2 is not None:
+            arr = self.l2.attach(key)
+            if arr is not None:
+                # The store counts the hit (and the publish, below) itself.
+                # Counting it here too doubled both in drain_stats, which
+                # sums the tier's counters with the store's.
+                # Promote so a crop loop pays one attach, not one per crop.
+                if self.l1 is not None:
+                    try:
+                        self.l1.set_array(key, arr)
+                    except Exception:  # noqa: BLE001, S110 -- an L1 that can't take the array costs a later attach, never an error
+                        pass
+                return arr, "shm"
+        self.stats["miss"] += 1
+        return None, "decode"
+
+    def set_array(self, key: str, arr) -> None:
+        """Called by ``_base_array`` after a decode. Raises only :class:`KeyCollision`."""
+        if self.l1 is not None:
+            try:
+                self.l1.set_array(key, arr)
+            except Exception:  # noqa: BLE001, S110 -- an L1 that can't take the array costs a later decode, never an error
+                pass
+        if self.l2 is not None:
+            try:
+                if not self.l2.publish(key, arr):
+                    self.stats["shm_declined"] += 1
+            except KeyCollision:
+                # The one exception that must reach the caller (contract §6.4): a
+                # broken key scheme hidden here would serve two contents under one key.
+                raise
+            except Exception:                                  # noqa: BLE001
+                self.stats["shm_publish_failed"] += 1
+
+    def set(self, key: str, value: bytes) -> None:
+        """Bytes path, for protocol compliance. ``_base_array`` prefers
+        ``set_array`` and will not normally reach this."""
+        try:
+            import io
+            arr = np.load(io.BytesIO(value), allow_pickle=False)
+        except Exception:  # noqa: BLE001 -- undecodable bytes are a miss: the cache never raises
+            return
+        self.set_array(key, arr)    # outside the try, so a KeyCollision gets out
+
+    def drain_stats(self) -> Counter:
+        """Take and reset counters, including the store's own."""
+        out = Counter(self.stats)
+        self.stats.clear()
+        if self.l2 is not None:
+            try:
+                out.update(self.l2.drain_stats())
+            except Exception:  # noqa: BLE001, S110 -- a store whose stats fail still lets the tier's own counters out
+                pass
+        return out
+
+    def __repr__(self) -> str:
+        return f"<TieredPixelCache l1={self.l1!r} l2={self.l2!r}>"