amber-lmdb 0.1.0__tar.gz

amber_lmdb-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,130 @@
+# Contributing
+
+Run `uv sync` to install the base package and development tools. The base test
+suite must run without PyTorch. Test tensor support separately with
+`uv sync --all-extras`, then run the suite again.
+
+Before a change is ready:
+
+```console
+uv run pytest
+uv run ruff check .
+uv run ruff format --check .
+uv run pyright
+uv run pyright --verifytypes amber_lmdb --ignoreexternal
+uv lock --check
+uv build
+```
+
+The type checks above need the optional dependencies installed. Write docstrings,
+comments, diagnostic messages and project documentation in English. Keep Unicode
+sample data in tests and compatibility fixtures unchanged.
+
+Public APIs must have annotations and documentation. Stored dictionary contents
+are intentionally opaque and use `Any`; omitted generic arguments are not an
+acceptable substitute.
+
+Keep read transactions bounded and close them before decoding owned bytes. Do not
+turn off LMDB locks, add global readahead changes or copy array bodies in
+header-only inspection. Whole-value validation must be explicit and honor its
+body/integrity-metadata copy budgets; unavailable or blocked codecs remain unverified.
+Checksum verification must finish after transaction closure and before decoders; only
+a reader built with `verify_checksums=False` may skip the digest, never its tag and
+length checks, and store-level verification must not honor that option. Header-only
+operations must not claim full-body integrity. Cooperative deletion requires retained
+shared OS locks for native environment lifetimes and a gate around acquisition,
+publication and retirement. Never replace or unlink the root gate or a lock file that
+a live generation may use; a generation's own lease file is removed only by
+deletion, under the root gate, after the generation has left both `versions` and
+`.trash`, so no acquisition can reach it. Dry runs create and remove nothing.
+
+Generation history is a safety contract. `HISTORY` is appended after every `CURRENT`
+switch (publication and `set_current`), and a writer never removes a generation that
+`HISTORY` lists: a root without `CURRENT` is refused with a hint, never repaired by
+deletion; only generations absent from an existing `HISTORY` count as unpublished.
+Readers pin the generation of their first read; `refresh()` releases it, `close()`
+is final, a dropped reader releases its reference through a finalizer, and pickling
+carries the pinned generation (an explicit `generation=` survives `refresh()`).
+Lazy opening must stay safe for threads. These are compatibility contracts, not
+implementation details. `cooperative=False` must remain read-only and must never
+create or take coordination locks or LMDB lock files.
+
+Key listing, counts, fingerprints and diffs read integrity metadata only; keep them
+free of body copies. An overlay must never return or count an entry that exists in
+more than one layer. Tuple keys starting with `"amber-lmdb"` are reserved for the
+manifest `("amber-lmdb", "manifest")` and array chunks: writers refuse them for user
+data, and fingerprints, diffs, listings and overlay conflict checks exclude them, so
+neither a manifest nor a chunk changes content identity on its own. A chunked header
+must list every chunk's envelope digest; readers compare each chunk's envelope with
+it even when `verify_checksums=False`, writers remove a chunked array's chunks
+whenever they replace or delete it, and `verify_store` checks the links both ways. A
+derived writer copies its base while holding the base's shared lease.
+
+Cache builders lock one of a fixed number of stripe files in `<directory>/.locks`;
+never unlink them. The stripe count is recorded once in `.locks/STRIPES` and every
+cache on the directory must use it; a different explicit count is an error, never a
+silent remapping. Cache identities are validated and canonicalized before hashing;
+tuples and non-string keys are refused rather than converted. `prune` removes an entry, or an abandoned temporary, only while
+holding its stripe without waiting, and a builder publishes only under its stripe, so
+a busy entry is skipped rather than removed. A lock timeout, a cooperative `cancel`
+and process isolation are different stops; none may publish partial output.
+`ValueLimits` may only lower the built-in decoding limits, and a node budget must
+raise rather than truncate. The typed-value decoder must keep the previous
+implementation's results and error messages; `tests/test_value_limits.py` compares
+them on valid and mutated nodes. Threaded digest verification must report the first
+corrupt entry in request order, like sequential verification. Progress callbacks run between pages, outside transactions, and a
+cancelled report must never be `ok`. Tensor bodies are copied once into a private
+bytearray that the returned tensor owns; never let a tensor alias LMDB memory or a
+caller's buffer. The same holds for arrays read with `writable=True` and for
+`read_as_tensor` on stored arrays.
+
+Compressed NumPy arrays use array format version 3 and name their codec in the
+header metadata; uncompressed arrays keep version 2, so older readers refuse
+compressed arrays instead of misreading them. A compressed chunked header lists every
+chunk's stored size before the digests. Decompression must be bounded by the size the
+header implies and must reject any other outcome; integrity digests cover the stored
+(compressed) bytes.
+
+Value writers take the value first and the key and index only as keywords. The
+command line prints one JSON document per command and keeps its exit codes: 0
+success, 1 findings, 2 usage errors, 3 the command could not complete (with an
+`{"error": ...}` document), 130 interrupted.
+
+The frozen fixture in `tests/fixtures/mmfabric_0_2_2.json` was generated by the
+untouched mmfabric 0.2.2 writer before extraction. Tests reconstruct a database from
+its raw key/value/tag bytes without using the current writer. Do not regenerate
+this fixture to make a format change pass. Legacy-format support is not a permanent
+requirement. A deliberate breaking format needs a new identifier, a stated rebuild
+or migration policy and separate fixtures/tests; it never authorizes automatic
+conversion or removal of application stores. The current marker is `amber-lmdb-v1`;
+old `mmfabric-lmdb-v2` stores are deliberately rejected. Rebuild from application
+sources into a separately reviewed output. Golden envelope metadata is in
+`tests/fixtures/amber_lmdb_integrity_v1.json` (SHA-256, version 1) and
+`tests/fixtures/amber_lmdb_integrity_v2.json` (pinned version 1 and 2 digests).
+These pinned digests are what make a store verifiable on any CPU; never regenerate
+them to make a change pass, and keep every published envelope version readable. See
+[formats and limits](guides/bounded-values-and-schemas.md) for the envelope contract.
+
+This repository must not import or depend on consumers such as mmfabric or
+dataset-cejc. Downstream compatibility modules must reuse this implementation,
+not copy codecs or create separate environment/custom-codec registries.
+The independence test blocks consumer imports while exercising the standalone API.
+Frozen historical fixtures remain evidence of their original producer, rather than
+being rewritten to match a new implementation.
+
+## Releasing
+
+The package is published on PyPI as `amber-lmdb` from
+https://github.com/Nekch0/amber-lmdb. A release bumps `version` in
+`pyproject.toml`, passes CI on Linux and Windows for the pushed commit, and is
+tagged `v<version>`. Build the wheel and source distribution from a clean checkout
+of that tag, install the wheel into a fresh environment and run the tests against
+it, then upload both files and attach them to a GitHub release. A version number
+is never reused, even after a deleted upload.
+
+Consumers replace local path sources with the published dependency only after both
+files are available from PyPI; do not publish consumer wheels that depend on an
+unavailable version.
+
+No application data, private metadata or generated benchmark results should be
+included in a distribution or source commit.

amber_lmdb-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Nekch0
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

amber_lmdb-0.1.0/PKG-INFO

@@ -0,0 +1,353 @@
+Metadata-Version: 2.4
+Name: amber-lmdb
+Version: 0.1.0
+Summary: Typed LMDB values, atomic generation publication, and bounded store inspection
+Author: Nekch0
+Author-email: Nekch0 <ptdmc.akdmc@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Typing :: Typed
+Requires-Dist: lmdb>=3.0.0,<4.0.0
+Requires-Dist: numpy>=2.4.6
+Requires-Dist: msgpack>=1.2.1
+Requires-Dist: frozendict>=2.4.7
+Requires-Dist: ml-dtypes>=0.5.4 ; extra == 'dtypes'
+Requires-Dist: torch>=2.12.1 ; extra == 'torch'
+Requires-Python: >=3.12
+Project-URL: Homepage, https://github.com/Nekch0/amber-lmdb
+Project-URL: Documentation, https://github.com/Nekch0/amber-lmdb/tree/main/guides
+Project-URL: Repository, https://github.com/Nekch0/amber-lmdb
+Project-URL: Issues, https://github.com/Nekch0/amber-lmdb/issues
+Provides-Extra: dtypes
+Provides-Extra: torch
+Description-Content-Type: text/markdown
+
+# amber-lmdb
+
+Store Python values and NumPy arrays by key in LMDB. Every write builds a complete
+new generation and publishes it with an atomic `CURRENT` switch; readers stay on
+the generation they opened until they ask for a newer one.
+
+Python 3.12 or later is required.
+
+## Install
+
+```console
+pip install amber-lmdb
+amber-lmdb --help
+```
+
+The base installation depends on LMDB, NumPy, MessagePack and frozendict. It does
+not install or import PyTorch. Install `amber-lmdb[torch]` for tensor conversion
+and `amber-lmdb[dtypes]` for NumPy extension dtypes that require ml-dtypes.
+
+## Concepts
+
+- **Dataset root**: the directory you pass as `path`. It holds a `FORMAT` marker,
+  the `versions/` directory with one subdirectory per generation, the `CURRENT`
+  pointer, the `HISTORY` log and `.coordination` lock files.
+- **Generation**: one complete LMDB database (`versions/v-<32 hex>`). A writer
+  stages a new generation beside the root and publishes it by rewriting `CURRENT`;
+  nothing is ever modified in place. `HISTORY` records every switch in order.
+- **Pinning**: a reader opens the generation `CURRENT` selects at its first read and
+  keeps reading it, even after newer publications, until `refresh()`. Old
+  generations stay on disk until you delete them explicitly.
+- **Keys**: a string, a path (stored in POSIX form) or a tuple of strings, plus an
+  optional index in `[0, 2**64)`. `"a"` and `("a",)` are different keys. All
+  entries of one key with indices form a sequence you can count and list.
+- **Writer tags and typed readers**: each entry records which writer stored it, and
+  each reader method accepts only matching tags, so a value comes back with the type
+  it was written with. `read` chooses the decoder from the tag.
+- **Integrity envelopes**: every entry has a digest bound to its key, tag and
+  length. Reads verify bodies before decoding; `verify_store` checks everything.
+
+## Write and read
+
+```python
+from pathlib import Path
+import numpy as np
+from amber_lmdb import LMDBReader, LMDBWriter
+
+path = Path("example.lmdb")
+with LMDBWriter(path) as writer:
+    writer.write_dict({"name": "example", "items": 2}, key="meta")
+    writer.write_ndarray(np.arange(8, dtype=np.float32).reshape(2, 4), key="features", index=0)
+    writer.write_primitive(3, key=("session", "count"))
+    # A successful exit commits the remaining writes and publishes the generation.
+
+with LMDBReader(path) as reader:
+    array = reader.read_as_ndarray("features", index=0)
+    records = reader.read_as_dict_many(["meta", "meta"])
+    snapshot = reader.get_snapshot()
+```
+
+Writers take the value first and the key and index as keywords (`key=`, `index=`),
+so a value and a key can never be swapped silently. Readers take the key first.
+`commit()` is optional; it only bounds the size of one transaction. Batch reads copy
+owned bytes during one bounded transaction and decode after it closes; single reads
+also close their transactions before returning.
+
+## Supported values
+
+| Writer | Values | Readers |
+|---|---|---|
+| `write_primitive` | `bool`, `int` (any size), `float`, `complex` | `read_as_primitive`, `read_as_any` |
+| `write_str` | `str` (UTF-8) | `read_as_str`, `read_as_str_many` |
+| `write_sequence` | `list`, `tuple` of supported values | `read_as_list`, `read_as_tuple`, `read_as_any` |
+| `write_set` | `set`, `frozenset` | `read_as_set`, `read_as_frozenset`, `read_as_any` |
+| `write_dict` | `dict`, `frozendict` | `read_as_dict`, `read_as_frozendict`, `read_as_any` |
+| `write_any` | the above nested in each other (arrays too), `bytes`, `None`, built-in types, registered classes | `read_as_any` |
+| `write_ndarray` | NumPy arrays of any non-object dtype, optionally chunked or compressed | `read_as_ndarray`, `read_as_tensor`, `read_array_rows` |
+| `write_tensor` | PyTorch tensors (CPU copy; device and gradients are dropped) | `read_as_tensor` |
+| `write` | chooses one of the writers above from the value's type | `read` |
+
+MessagePack values (everything except strings and arrays) are limited to 64 MiB
+encoded, 1,000,000 nodes and nesting depth 100 when written, so whatever is stored
+can be read back. NumPy scalars are refused by the value writers with a hint
+(`.item()` gives a Python number); `write_ndarray` stores them as 0-d arrays.
+Masked arrays are refused because their mask would be lost. Typed reads never
+convert: a stored tuple is not returned by `read_as_list`.
+
+Arrays come back read-only, owning the bytes copied from LMDB; pass
+`writable=True` for a writable array without a second copy. A `dtype=` conversion
+follows NumPy's `"same_kind"` casting by default, so float-to-integer truncation
+needs `casting="unsafe"`. `write_ndarray(..., compression="zlib" | "lzma" | "zstd")`
+compresses values (zstd needs Python 3.14); reads decompress transparently.
+
+## Update a store
+
+| Goal | How |
+|---|---|
+| Replace everything | `LMDBWriter(path, overwrite=True)` writes a new generation from scratch |
+| Change, add or delete some entries | `LMDBWriter(path, base="current")` copies the current generation, then `write_*`, `delete`, `delete_key` |
+| Add new features without rewriting | write a separate store and read both with `LMDBOverlayReader([base, features])` |
+| Fold layers into one store | `merge_stores([base, features], destination)` copies entries as stored |
+| Roll back | `set_current(path, "v-...")` or `amber-lmdb set-current PATH NAME` |
+| Free disk space | `prune_generations(path, keep_last=N)` or `delete_generations(path, names)`; dry runs by default |
+
+Existing readers keep their generation through all of these. Deletion requires
+`same_host=True` (CLI `--apply --same-host`): it is safe only when every access goes
+through this library on one host, and it refuses generations that a reader holds.
+A writer lock rejects a second concurrent writer to the same root, and `CURRENT`
+is never removed automatically: a root that lost `CURRENT` is refused until you
+restore it with `set_current`.
+
+## Readers, processes and threads
+
+A reader opens nothing until its first read, then pins that generation (see
+Concepts). `refresh()` releases it so the next read follows `CURRENT`; `close()`
+releases it for good, and a closed reader raises `ReaderClosedError`. A reader
+dropped without closing releases its reference when it is garbage collected.
+
+Readers can be shared by threads. Pickling a reader, as `DataLoader` workers do,
+carries its pinned generation, so every copy reads the same data; call
+`reader.get_snapshot()` before handing an unopened reader to workers so they all
+pin one generation. Copies open their own environments. See
+[PyTorch data loading](https://github.com/Nekch0/amber-lmdb/blob/main/guides/pytorch.md).
+
+Within a process, readers of one generation share one LMDB environment; its tuning
+(map size, readahead) follows the first opener. `LMDBReader(path, generation=...)`
+reads a named generation. `cooperative=False` reads stores on media nothing can
+change, such as read-only mounts or snapshots, without lock files; never use it
+while a writer or a cleanup may touch the store.
+
+## Choose an API for the task
+
+| Task | API | Boundary |
+|---|---|---|
+| Restore many values | `read_many`, `read_as_*_many`, `iter_read_pages` | Up to 2**20 items; byte budget 64 MiB by default, `max_bytes=None` removes it |
+| Bound a single read | `read(..., max_bytes=)` and every `read_as_*` | Refused before copying when the body is larger |
+| Inspect array shapes/dtypes | `read_array_headers_many` | No body copies; repeated headers share parsing within a call |
+| Read row windows of large arrays | `read_array_rows`, `read_array_rows_many`; `write_ndarray(..., chunk_rows=)` | Chunked arrays copy and verify only touched chunks |
+| Feed PyTorch | `read_as_tensor` (tensors and arrays), `as_tensor` | Writable tensors without the read-only warning |
+| Construct indexed records | `RecordTable` | A sequence of rows `0..count-1`; optional factory and schema |
+| Check mapping fields | `ValueSchema`, `RecordSchema` | Exact types or explicit shape/dtype/range/optional rules; no coercion |
+| Share a derived result | `DerivedIndexCache.get_or_build`, `lookup` | Caller identity/codec; `CacheResult.hit` distinguishes a stored `None` |
+| Keep a cache directory bounded | `DerivedIndexCache.prune`; CLI `cache-list`, `cache-prune` | Dry run by default; busy entries kept |
+| Bound decoding of untrusted values | `LMDBReader(..., value_limits=ValueLimits(...))`, `max_decoded_nodes=` | Lowers MessagePack limits only; arrays unaffected |
+| Verify large batches on idle cores | `LMDBReader(..., verify_threads=N)` | Bodies and chunks of at least 64 KiB; off by default |
+| Check, count or list keys | `contains`, `count`, `index_range`, `iter_indices`, `iter_keys` | Integrity metadata only; listing costs one seek per logical key |
+| Read several stores as one | `LMDBOverlayReader` | Nothing copied; an entry in more than one layer raises `OverlayConflictError` |
+| Manage generations | `inspect_generations`, `set_current`, `prune_generations`, `delete_generations`, `read_history` | Deletion is explicit, cooperative and same-host |
+| Diagnose a store | `inspect_store`, `verify_store`, `verify_physical` | Sampled, typed-value and physical coverage are different |
+| Identify or compare content | `fingerprint_store`, `diff_stores` | Integrity envelopes only; bodies are read only to resolve algorithm changes |
+| Summarize a generation at write time | `LMDBWriter(..., manifest=True)`, `reader.manifest()` | Opt-in; one reserved dictionary entry, excluded from fingerprints |
+| Follow or stop long scans | `progress=`, `cancel=` on scans | Called between pages; a cancelled `verify_store` report is never `ok` |
+
+Scans (`verify_store`, `array_statistics`, `inspect_store`, `fingerprint_store`,
+`diff_stores`) also take `generation=` and `cooperative=`.
+
+## Errors
+
+Every library error derives from `AmberLMDBError` and also from the built-in
+exception it refines, so existing `except KeyError:` code keeps working:
+`KeyNotFoundError` (`KeyError`, with `.key` and `.index`), `ReaderMismatchError`
+(`ValueError`, with the stored writer tag and the readers that accept it),
+`StoreNotFoundError` (`FileNotFoundError`), `ReaderClosedError`, `ReadLimitError`,
+`IntegrityError`, `OverlayConflictError`, `SourceChangedError` and
+`OperationCancelled`. Messages name keys in their decoded form, such as
+`key ('s01', 'pitch') index 3`.
+
+## Command line
+
+```console
+amber-lmdb list ./stores
+amber-lmdb inspect ./example.lmdb --meta --entries 20
+amber-lmdb doctor ./example.lmdb                 # quick sample
+amber-lmdb doctor ./example.lmdb --full --progress
+amber-lmdb stats ./example.lmdb
+amber-lmdb generations ./example.lmdb
+amber-lmdb set-current ./example.lmdb v-<32 hex>
+amber-lmdb delete-generations ./example.lmdb --keep-last 2   # preview
+amber-lmdb fingerprint ./example.lmdb --generation v-<32 hex>
+amber-lmdb diff ./example.lmdb ./example.lmdb --before-generation v-<32 hex>
+amber-lmdb manifest ./example.lmdb
+python -m amber_lmdb --version
+```
+
+Every command prints one JSON document. Exit codes: 0 success; 1 the command ran
+and found something (problems, differences, refused deletions, busy cache entries);
+2 invalid usage; 3 the command could not complete (missing or unreadable store,
+invalid layout, I/O error), with `{"error": {"type", "message"}}` on stdout and the
+message on stderr; 130 interrupted. `--read-only-media` and `--generation` select
+the reading mode and generation of the reading commands.
+
+`doctor` without `--full` checks the layout and a bounded sample of keys, tags and
+headers and reports `sample_ok`; it is **not** a full check. `doctor --full` visits
+every entry, verifies checksums and decodes values within the byte limits, and
+reports `ok` and `complete`. `--max-issues` caps listed issue details (with their
+decoded keys), never the number of checked entries. Pass the previous `next_after`
+token to `inspect --after` for another page; each call opens a fresh transaction.
+Metadata with non-string dictionary keys is shown as a `mapping` list of pairs;
+arrays are summarized by type, shape and dtype. The CLI never runs custom decoders.
+
+The same inspection is available in Python:
+
+```python
+from amber_lmdb import inspect_store
+
+info = inspect_store("example.lmdb", with_meta=True, entry_limit=20)
+# Select another unindexed dictionary key with metadata_key="description".
+print(info.snapshot, info.data_entries, info.samples)
+```
+
+`data_entries` excludes the integrity database reference that `entries` (LMDB's
+own count) includes. `used_bytes` counts allocated pages of the inspected
+generation; it is not the total disk footprint.
+
+## Backend engines and physical checks
+
+py-lmdb 3 is required. New generations default to LMDB 1.0; pass
+`LMDBWriter(..., lib_version=0)` for 0.9. Readers detect the existing file's engine.
+`get_backend_info()` / `amber-lmdb backend` report available engines, and
+`reader.get_lib_version()` reports the one actually opened.
+
+`verify_physical(path)` / `doctor PATH --physical` use the upstream offline Python
+page verifier without opening the native engine. Use a private immutable copy;
+lock files are not verified. `LMDBWriter(..., verify_on_publish=True)` optionally
+checks its exclusively owned staging data before publication. See
+[py-lmdb 3](https://github.com/Nekch0/amber-lmdb/blob/main/guides/lmdb-3.md) for engine formats, trust boundaries and cost.
+
+## Define records and named fields
+
+Use `ValueSchema` / `infer_schema` / `validate_schema` for exact field types.
+`RecordSchema` / `FieldSpec` add nullable/optional fields, exact-type unions,
+shape/dtype/range/length constraints and trusted cross-field validation; invalid
+specifications, such as a NumPy scalar type or an unknown dtype name, fail when the
+schema is built. `ValueView` reads declared fields under a namespace; `RecordTable`
+is a read-only sequence of indexed dictionaries (`len`, `table[i]`, slices,
+iteration in bounded pages) with an optional factory applied after transactions
+close. Neither API introduces padding, filtering, dataset session conventions or RNG.
+
+`DerivedIndexCache` shares caller-built results under an explicit identity: a
+str-keyed mapping of JSON values or a dataclass such as `LMDBSnapshot`; tuples and
+non-string keys are refused because JSON would change them. Builders share a fixed
+set of lock stripes whose count is recorded in the cache directory, and `prune`
+removes invalid, unwanted, expired or excess files safely.
+
+See [bounded values and schemas](https://github.com/Nekch0/amber-lmdb/blob/main/guides/bounded-values-and-schemas.md) for byte-aware
+streaming, field constraints and checksum coverage, [records and
+indexes](https://github.com/Nekch0/amber-lmdb/blob/main/guides/records-and-indexes.md) for runnable invoice examples, and
+[connection lifetimes](https://github.com/Nekch0/amber-lmdb/blob/main/guides/connection-lifecycle.md) for raw `open_environment`
+handles, generation retention and deletion.
+
+## Storage format compatibility
+
+This library has no dependency on a dataset framework or on its consumer packages.
+Applications supply their own schemas, keys, constructors and cache identities.
+
+The store marker is `amber-lmdb-v1\n`, with key/tag/length-bound integrity
+envelopes. New entries use BLAKE2b-256 (envelope version 2) by default;
+`LMDBWriter(..., checksum="sha256")` writes SHA-256 (version 1). Readers verify
+both, per entry. `LMDBReader(path, verify_checksums=False)` skips only the body
+digest for one reader, for stores whose operator takes responsibility that they are
+undamaged and unaltered; tags, lengths and decoders are still checked, and
+`verify_store`/`doctor` still verify. The historical `mmfabric-lmdb-v2` marker is
+rejected explicitly; there is no automatic migration. This library reads its own
+typed format, not arbitrary LMDB layouts.
+
+Compressed arrays use array format version 3; readers older than this release
+refuse them with an "unsupported array version" error instead of misreading them.
+Uncompressed arrays keep version 2. `HISTORY` is a JSON-lines file appended after
+each `CURRENT` switch; roots written before it existed are read normally and gain
+one at their next publication.
+
+Tuple keys whose first component is `"amber-lmdb"` are reserved: the optional
+manifest and the chunks of arrays written with `chunk_rows` live there. Writers refuse
+them for user data, and listings, fingerprints, diffs and overlay checks skip them.
+A chunked array's header entry lists its chunks' digests, so its own envelope covers
+the whole array and each chunk must carry exactly the listed digest.
+
+Custom codecs require explicit registration (`register_bytes_codec`, undone with
+`unregister_bytes_codec`) and `allow_custom=True` when reading. The CLI never enables
+custom decoding, and no module is imported from stored type names.
+
+## Scan arrays or validate all stored values
+
+`array_statistics(path)` scans headers and groups arrays by writer tag, dtype and
+rank. Axis minima are bounds, not necessarily real sample shapes; smallest and
+largest sample shapes are reported separately, with their keys.
+
+`verify_store(path)` visits all data and tag entries, decodes values after each
+short transaction, and detects malformed keys/tags/values and orphan tags. Values
+over the copy limit (`max_value_bytes`, 64 MiB by default), unavailable optional
+codecs and blocked custom decoders are reported as **unverified**, not certified.
+`report.ok` requires complete coverage and no detected errors. Checksums detect
+corruption, not authenticity. In-place updates abort a scan rather than mixing
+transactions. See [scanning stores](https://github.com/Nekch0/amber-lmdb/blob/main/guides/scanning-stores.md).
+
+## Operational limits
+
+Normal LMDB locks remain enabled. No network filesystem safety guarantee is made;
+the filesystem must satisfy LMDB's locking and memory-mapping requirements.
+Generation deletion assumes that every access cooperates through this library on
+one host; never use it for NFS or cross-host access. Metadata can contain
+transcripts or other restricted information; use `--meta` only when appropriate.
+Record definitions, index builders, selection and application semantics belong in
+the consumer.
+
+## Development
+
+Clone the repository and run `uv sync`. Type checking requires the optional
+dependencies, so use `uv sync --all-extras` before those checks. Keep a separate
+base test run without PyTorch.
+
+```console
+uv run pytest
+uv run ruff check .
+uv run ruff format --check .
+uv run pyright
+uv build
+```
+
+Tests with PyTorch skip when the optional dependency is absent. CI tests Python
+3.12, 3.13 and 3.14 without PyTorch, and tests the tensor extra separately.
+
+Use the [synthetic benchmark](https://github.com/Nekch0/amber-lmdb/blob/main/benchmarks/README.md) to compare batch sizes and
+engines without touching existing stores.
+
+See [CONTRIBUTING.md](https://github.com/Nekch0/amber-lmdb/blob/main/CONTRIBUTING.md) for compatibility and release checks.