pyuncommonkit 0.1.0__py3-none-any.whl

pyuncommonkit-0.1.0.dist-info/METADATA

@@ -0,0 +1,200 @@
+Metadata-Version: 2.4
+Name: pyuncommonkit
+Version: 0.1.0
+Summary: Named shared-memory segments for numpy arrays: publish once, attach from any process.
+Keywords: numpy,npy,mmap,shared-memory,cache
+Author: Sean D. True
+Author-email: Sean D. True <sean.true@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Filesystems
+Requires-Dist: numpy>=1.26
+Requires-Dist: pytest>=8 ; extra == 'dev'
+Requires-Dist: ruff>=0.6 ; extra == 'dev'
+Requires-Python: >=3.11
+Project-URL: Homepage, https://github.com/seantrue/pycommonkit
+Project-URL: Repository, https://github.com/seantrue/pycommonkit
+Project-URL: Issues, https://github.com/seantrue/pycommonkit/issues
+Project-URL: Contract, https://github.com/seantrue/pycommonkit/blob/main/docs/contract.md
+Project-URL: Rust crate, https://crates.io/crates/commonkit
+Provides-Extra: dev
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/seantrue/pycommonkit/main/docs/images/commons-banner.jpg"
+       alt="Illustrated aerial view of farmland: square fields and center-pivot circles on a shared road grid"
+       width="100%">
+</p>
+
+# pycommonkit
+
+Named shared-memory segments for numpy arrays. One process decodes and
+publishes; every other process attaches the same physical pages.
+
+Installed as **`pyuncommonkit`**, imported as **`commonkit`** (the PyPI name `commonkit` belongs
+to an unrelated project, and PyPI refused `pycommonkit` as too similar to it). A Rust crate, `commonkit`, implementing the same on-disk contract is
+in this repository under `rust/` (`docs/rust-crate-plan.md`): keys, segments, the store and the
+reaper are done, and a cross-language conformance suite runs both on one tree.
+
+The name is for Fortran's **named `COMMON` block**. `COMMON /GRID/ X, Y` in one
+subroutine and the same name in another, compiled separately, give both the same
+storage: the linker joins them by the name alone, and no unit owns the block. The
+name is the whole interface. A commonkit key is the same idea: any process that
+can spell the key reaches the same bytes, without knowing who wrote them.
+
+Where the analogy stops is where commonkit had to be stricter:
+
+* **Across processes, not within one program.** A `COMMON` block is shared by the
+  program units of one executable; a segment is shared by separate processes.
+* **The layout travels with the data.** Each Fortran unit declares its own view
+  of a block, and a unit that declares it differently silently reads other
+  variables' bits. A segment carries its dtype, shape and order in its `.npy`
+  header, so every reader sees the same layout, and publishing a different array
+  under an existing key is refused loudly (`KeyCollision`).
+* **Written once, then read-only.** Any unit may assign into a `COMMON` block at
+  any time. A segment is written once, whole, and readers map it read-only.
+* **It may not be there.** A `COMMON` block exists for the whole run. A segment
+  exists only until the reaper needs the space, which is why every caller must be
+  able to recompute it (below).
+
+## The contract
+
+`docs/contract.md` (version 1) is the authority on the on-disk format: keys, layout, the `.npy`
+segment, publishing, the write-once rules, touch, the reaper, policy and counters. Python is the
+reference implementation and `conformance/vectors/` decides any disagreement
+(`conformance/generate.py` regenerates the fixtures byte for byte). Both the Python package and
+the Rust crate pass every vector.
+
+## Status
+
+See `docs/STATUS.md`. Nothing is published to PyPI or crates.io yet.
+
+## What it is
+
+* `SharedArrayStore` — publish/attach `.npy` segments under a root directory,
+  with admission gates, a TTL reaper and a byte budget.
+* `KeyScheme` — the addressing vocabulary for one *domain*. A scheme fixes the
+  domain, a version, the legal forms and the legal variants, and turns
+  `(digest, variant, form)` into both a key and a relative path.
+* `shared_store()` — the ONE store per host that every user shares. Its byte budget, TTL,
+  free-space floor, fsync and republish verification come only from the tree's `.policy.json`
+  (which wins), then `COMMONKIT_STORE_*` environment variables, then the defaults (macOS:
+  `~/Library/Caches/commonkit/store`, 2 GB, 900 s, 8 GB free), because the reaping process applies
+  its own policy to the whole tree; the root comes from the environment or the default. A caller
+  chooses only `mode` and the smallest array worth storing.
+* `TieredPixelCache` — an in-process L1 memo in front of a store, so a loop
+  that looks up the same digest repeatedly pays one attach rather than one per
+  call.
+
+## A cache, not a store of record
+
+**A missing segment is never an error, only a slower path.** The store promises
+that a key which is present names the right bytes. It never promises that a key
+is present. So every caller must be able to compute the value again when
+`attach` returns `None`, and must treat a `publish` that returns `False` as
+normal.
+
+A segment can be absent at any moment, for reasons the caller doesn't control:
+
+* **Its publish was declined.** Under pressure the store refuses rather than
+  blocks: the array was smaller than worth storing or larger than allowed, the
+  volume was below its free-space floor, or the tree was over its byte budget.
+  A decline is how the store survives back-pressure, not a fault.
+* **The reaper removed it.** Segments are evicted when they expire, least
+  recently used first to get back under the byte budget, and to restore the
+  free-space floor. The reaper doesn't know who might still want a segment, and
+  doesn't ask: no reference counts, no coordination between processes.
+* **It was torn.** A file shorter than its header promises, after a crash with
+  `fsync` off (the default) or damage from outside the contract, is a miss.
+* **The store is disabled.** The mode is `off`, or the root can't be created or
+  written: every attach misses and every publish declines.
+* **The tree is gone.** On Linux it lives in `/dev/shm` and does not survive a
+  reboot; any root can be cleared; another machine has its own tree.
+
+That one rule is what makes the rest cheap. The reaper can unlink freely,
+publish never blocks, writes need no durability, and a process that fails
+costs the others only time: the next reader recomputes.
+
+**What does raise.** Exactly two things, both bugs at the call site rather than
+conditions of the store: building a key that isn't valid for its scheme
+(`ValueError`), and publishing a *different* array under a key that already
+holds one (`KeyCollision`, a `ValueError` subclass), which means a key scheme or
+a digest is broken. Everything else (an absent root, a full filesystem, a
+reaped or torn segment, a hostile `.usage.json` or `.policy.json`) returns
+`None` or `False` and ticks a counter. The constructor never raises: an
+unusable root leaves the store `disabled` and every method a no-op.
+
+**Replacing a value under the same key.** A segment is written once: publishing
+the same array again only refreshes it, and a different array raises
+`KeyCollision`. A caller that must replace a value whose key hasn't changed (a
+recompute after a fix the key doesn't capture) calls `remove(key)` first, then
+publishes. `remove` only unlinks, so a reader still holding the old segment keeps
+its bytes; there is no in-use check.
+
+**A value that can't be recomputed doesn't belong here alone.** Keep its
+durable copy somewhere you own and use the store to share it fast. (A pinned
+subtree the reaper skips is a possible later addition; it isn't built.)
+
+**Callers never need `.usage.json`.** It is the reaper's record of how much the
+tree holds, read only by the byte-budget gate. When it is missing or stale,
+usage is unknown, that gate passes, and the free-space floor still applies;
+the next complete reap writes a fresh one.
+
+## Keys
+
+    {domain}:{version}:{form}:{digest}:{variant}
+
+and the path it names:
+
+    {root}/{form}/{digest[:2]}/{digest}.{variant}.npy
+
+Two-level sharding, so no directory holds a million entries. The `variant` axis
+is whatever distinguishes two renderings of the same source: for images it is
+orientation (`o0`, `o1`, `oa`), for audio it would be sample rate (`sr16000`).
+Schemes are registered by domain so `parse()` and `relpath()` can dispatch on a
+key alone, which is what lets one store hold several domains.
+
+## Tests
+
+    uv run --extra dev pytest -q          # includes the conformance vectors
+
+The suite addresses audio and video rather than images, to show the package is
+not pixel-specific. Two modules run end to end against real media and skip
+themselves when their tool is not on `PATH`:
+
+* `tests/test_sox.py` — `sox` synthesizes a sine wave and dumps it as raw
+  int16 (and float32) samples. The samples are published, attached back as a
+  read-only `np.memmap`, attached again from a separate process, and stored at
+  two sample rates as two variants (`sr16000`, `sr44100`) of one digest.
+* `tests/test_ffmpeg.py` — `ffmpeg` renders its `testsrc` pattern to
+  `rawvideo` frames, stored as one `(T, H, W, C)` array. The same checks, with
+  the frame rate as the variant (`fr10`, `fr25`), plus a clip over
+  `max_array_bytes` being declined whole rather than truncated.
+
+Both use their tool's built-in generators, so no media files are checked in.
+Both also exercise per-form dtypes: `dtypes` maps a *form* to the dtype it
+requires (`{"wav16k": "int16"}`), and a float dump is refused under the
+integer form and accepted under the float one. A form's name is only a label —
+the store enforces a dtype only for forms listed in `dtypes`.
+
+## Origin
+
+Lifted on 2026-10-27 from `seantrue/commonkit` (`main` at `4dbfc7d`, plus the contract and
+vectors from its `rust-crate-plan` branch at `2b6bcf9` and after), with a fresh history. That
+package was extracted from the pixel-cache layer of a private image-corpus
+application, where it ran in production first. The measurements in the module
+docstrings were taken there; the framework-coupled half (the settings wiring
+and the decode entry point) deliberately stayed behind.

pyuncommonkit-0.1.0.dist-info/RECORD

@@ -0,0 +1,10 @@
+commonkit/__init__.py,sha256=lYuo5uXsaJqQIZ1aLzbKTSzztgoVxByxAtmYRXQ3Gcw,765
+commonkit/cache.py,sha256=zjyRS-RvTCwteaEty8L7SpClcIP2V9tl15TLmIsnBhg,5550
+commonkit/keys.py,sha256=0sA00GXOUVleTvKrXfkgaLDhjFt_hXXUiUUqQ_E7xgE,10239
+commonkit/read.py,sha256=VaP14S0K-nvPkgsIWSupbRHpwoTNqznuZGOT-LyZ7Gw,1964
+commonkit/shared.py,sha256=kaS3duagm0InnFYm907ABXQQFPhT0twI6JZ4qp2XIuQ,9205
+commonkit/store.py,sha256=BFDqTyPi4lGS9XV3uDmlAMHXFBurAbU2qikgRZ5P1mM,44008
+pyuncommonkit-0.1.0.dist-info/licenses/LICENSE,sha256=MQpVKT9wlLIb4SIMkLUdVcCwsdM_rIMZwdhqAFM9PfU,1069
+pyuncommonkit-0.1.0.dist-info/WHEEL,sha256=4OL6Foqnnp3xRY5wMkjgc25_i5YJC6dKsC6LPcjqEoU,80
+pyuncommonkit-0.1.0.dist-info/METADATA,sha256=ZmJWiGwdeUGGOaET_-fpWSZJ7Hs7Euc-3S-TsieB0Ok,10743
+pyuncommonkit-0.1.0.dist-info/RECORD,,

pyuncommonkit-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: uv 0.12.5
+Root-Is-Purelib: true
+Tag: py3-none-any

pyuncommonkit-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Sean D. True
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.