foldlitedb 0.1.0__tar.gz

foldlitedb-0.1.0/DESIGN.md

@@ -0,0 +1,190 @@
+# Foldlite DB design
+
+Foldlite DB derives live views from immutable documents. Its computation model is
+inspired by CouchDB: map each document into keyed contributions, then reduce each
+key. Its persistence model is a Rust append-only document log with a disposable
+cache. Python exposes that engine directly. One compiled WebAssembly module
+contains the full map and reducer, and runs unchanged in the native host and a
+browser worker.
+
+## Separate facts from derived state
+
+The document log is the persistent source for this engine. Accepted documents are
+immutable. An identical replay is a no-op; conflicting reuse of an identity or
+source position is an error. Changes in a domain are new documents, rather than
+mutations of old observations.
+
+Document indexes, source checkpoints, group states, finalized results, and change
+history are derived from those facts and the exact view module. They can be
+reconstructed. A cache is useful only while its dependencies can be verified:
+format version, document-log identity, committed prefix and digest, and compiled
+module digest. Missing, corrupt, or incompatible cache data cannot silently
+become authoritative.
+
+Applications must back up the document log and computation artifacts, or retain
+an external authoritative document stream with stable identities and order.
+Deleting a cache is safe; deleting the only copy of the log is data loss. A
+reference version is insufficient to reproduce a lost reference snapshot; any
+reference data needed by a deterministic view must remain reproducible.
+
+## Native durability
+
+`native/engine` owns storage and Wasm execution. The path passed to `Database`
+is a log file; `.cache` and `.lock` siblings hold an optional cache checkpoint and
+an operating-system lock. One active owner may open a persistent log. `:memory:`
+uses the same computation without persistent files or durable acknowledgements.
+
+The log has an identity and checksummed, hash-linked frames containing accepted
+document batches. The source format does not contain a view-module dependency.
+An ingestion operation validates identities, source order, limits, mapping, and
+reduction before writing. The engine appends the complete frame and syncs it
+before acknowledging the batch, then exposes its already-computed derived state.
+A failed validation or view operation cannot commit a batch prefix. An I/O error
+can leave commit outcome uncertain, as with any durable append; reopen and retry
+using explicit document identities rather than assuming a failed acknowledgement
+means absence.
+
+Opening verifies the complete log. An incomplete final frame can be discarded
+as an interrupted append. A complete frame with corrupt checksums or invalid
+content is an error; the engine does not skip committed source data. Frame size
+is bounded to 8 MiB. The ordinary map-request limit is smaller and often limits a
+batch first.
+
+`checkpoint()` writes an atomic derived snapshot, and graceful `close()` also
+checkpoints. Checkpoints are not written after every ingestion. Encoded cache
+snapshots are limited to 1 GiB. Checkpoint errors, including on close, are reported;
+they do not undo already-synced document batches. A failed close still releases
+the native handle. A valid snapshot
+must match the log identity, exact module digest, and verified prefix. The host
+loads it and replays only the original suffix batches through Wasm. A missing,
+invalid, or different-module cache triggers a full rebuild from the log.
+
+A checkpoint is a performance hint: failure or absence must not change the
+meaning of accepted documents. Its checksum and verified log prefix detect
+accidental corruption and stale files; they are not an authenticity guarantee
+against an adversary who can rewrite local files and checksums.
+
+## Rebuild as the ordinary computation path
+
+Live ingestion and replay use the same map, grouping, ordered fold, and finalizer.
+Rebuild preserves accepted document order and original batch boundaries. That
+also preserves batch-dependent engine sequence allocation; a separate export and
+re-import with different boundaries need only preserve view state and results.
+
+`rebuild()` discards derived state, replays the log, creates a new cursor
+generation, and checkpoints it. Opening the same log with a changed module does
+the same reconstruction automatically. Neither operation rewrites documents.
+An unchanged valid cache preserves its generation across reopen; cursor consumers
+must obtain a new snapshot after reconstruction.
+
+Rebuild cost is a linear read of the log plus Wasm work for uncached documents
+and their emissions. A valid cache avoids Wasm execution for its verified prefix,
+but opening still reads that prefix to validate it. There is no unmeasured
+throughput promise. Current documents, indexes, grouped states, and change history
+grow in host memory; bounded per-call Wasm memory is not a total-memory bound.
+Compaction, incremental checkpoint formats, and multiple native views remain
+future work rather than implicit capabilities.
+
+## Identity and order
+
+The public input is a `Document(id, payload, position, source)`. Payloads are
+finite JSON. `add` can generate IDs and source positions; `add_many` resolves one
+atomic batch. Generated IDs are stable only after resolution, so an unnamed
+retry creates another observation. External sources can provide their own IDs
+and monotonically increasing positions with gaps. Positions start at zero when
+assigned automatically and are scoped to a source.
+
+An identical named append reuses its original position even after later records
+arrive. A supplied source position can also identify a replay without a supplied
+ID. Reusing an ID or position with different content fails. Previously unseen
+positions behind the source checkpoint fail. Source progress and document
+acceptance are one logical commit.
+
+Per-key reduction follows ingestion order, including within a batch. Source
+positions do not sort documents across sources, and timestamps do not reorder a
+fold. Applications choose domain order before ingestion. Maps receive payloads;
+identity, deduplication, positions, and transaction boundaries belong to the host.
+Any domain metadata a view needs belongs in its explicit payload or declared
+reproducible module data.
+
+## Portable computation
+
+A view exports `descriptor`, `map`, `identity`, `add`, `finalize`, and optionally
+`merge` through ABI v1. AssemblyScript is the first SDK; the ABI does not require
+that source language. A map returns zero or more JSON key/value contributions.
+The host partitions by canonical keys and supplies ordered contributions to the
+module's reducer.
+
+State is explicit. Fresh Wasm instances prevent module globals from carrying
+state between requests. Views must not depend on clocks, randomness, mutable
+external data, or request boundaries. Splitting one ordered contribution sequence
+across successive `add` calls must produce equivalent state. Implement `merge`
+only when independently covered compatible states can be combined correctly;
+an ordered lifecycle fold need not be mergeable.
+
+The exact module SHA-256 binds cached state to computation, state schema, compiler,
+and SDK runtime. Human-readable view and state versions explain changes but do
+not override a changed digest. Preserve the previous module to reproduce its
+previous result. Opening the same documents with another module is a new derived
+view and invalidates old cursors.
+
+Modules have no host imports. ABI requests and responses are limited to 256 KiB,
+linear memory to 16 MiB, and native execution to a fuel budget. Ingestion is
+bounded to 1024 documents and 4096 emissions per batch. JSON nesting is bounded
+to 64 levels. Python releases the GIL while the native engine runs. These limits
+isolate individual operations; they do not bound accumulated host storage.
+
+JSON numbers follow finite IEEE-754 double semantics in both hosts. Python
+rejects integers that cannot be represented exactly as doubles. Source positions
+and engine sequences stay within JavaScript's nonnegative safe integer range.
+Canonical object keys and numeric formatting agree across hosts. Floating-point
+addition is not associative, so arbitrary merge trees can change rounding.
+
+## Reads and cursor generations
+
+A snapshot identifies a database generation, module digest, and committed
+sequence. Bounded pages retain a stable prefix while ingestion continues. Change
+pages return complete resulting key states after a cursor, coalesced per key
+within each batch. Zero-emission documents still advance progress. Sequence is
+not a document count: inserted documents and changed group states allocate it.
+
+`rows()` and `documents()` pin their prefix on first iteration and release the
+engine between pages. No lock spans a yield into application code. The persistent
+file's ownership lock is retained for the database lifetime. Cursors are local
+continuation markers, not access credentials. Rebuild or module change starts a
+new generation and explicitly expires old cursors. Current history is retained;
+there is no compaction or reader-retention policy yet.
+
+## Python packaging and the browser
+
+`Database` and `View` are the primary Python types; `PortableDatabase` and
+`PortableView` are exact aliases. The package has one native storage and
+computation path. `native/python` provides PyO3 bindings, and maturin builds the
+extension into wheels. Source builds need Python 3.11+, Rust/Cargo 1.90+, and a C
+toolchain. Bundled modules need no Node; authoring uses Node 20+.
+
+The browser worker runs the same module bytes, with JavaScript owning grouping
+and in-memory storage. It does not run the Rust log engine and has no persistent
+browser storage yet. Export and replay resolved documents to preserve work.
+Worker deadlines, retained-byte limits, and document/group/history caps bound
+this preview's requests and retained data. A timeout discards its memory.
+
+## Acceptance and evidence
+
+Correctness requires immutable-identity conflicts, source-order enforcement,
+atomic batches, interrupted-tail recovery, complete-corruption rejection,
+single-owner locking, checkpoint validation, suffix replay, module replacement,
+and explicit rebuild. Rebuild and live execution must agree on structured keys,
+ordered state, and results. Snapshot/change continuity survives a valid cache
+reopen and fails explicitly after a new generation. Native and actual browser
+execution must agree for the same module and supported numerical inputs.
+
+Installation checks exercise built wheels without local compilers, and source
+distributions with build prerequisites. PyPI publication remains pending. Future
+performance evidence must record workload, fixture, batch size, documents,
+emissions, key cardinality, host/runtime, durability, log/cache bytes, peak memory,
+rebuild time, reopen validation time, and document-to-query latency. Run `python scripts/benchmark_database.py --documents 10000 --keys 16
+--batch-size 100 --live-documents 20 --output native-benchmark.json` to produce a
+reproducible local report; `--storage-root` selects the measured filesystem. The
+design promises reconstructibility and a clear cost model, rather than an
+unmeasured speedup.

foldlitedb-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 255BITS
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

foldlitedb-0.1.0/PKG-INFO

@@ -0,0 +1,239 @@
+Metadata-Version: 2.4
+Name: foldlitedb
+Version: 0.1.0
+Requires-Dist: playwright>=1.58,<2 ; extra == 'browser'
+Requires-Dist: build>=1,<2 ; extra == 'dev'
+Requires-Dist: maturin>=1.9,<2 ; extra == 'dev'
+Requires-Dist: pytest>=8,<10 ; extra == 'dev'
+Requires-Dist: pytest>=8,<10 ; extra == 'test'
+Provides-Extra: browser
+Provides-Extra: dev
+Provides-Extra: test
+License-File: LICENSE
+Summary: Immutable document logs and rebuildable views with Rust and portable WebAssembly.
+License-Expression: MIT
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
+Project-URL: Documentation, https://255bits.github.io/foldlitedb/
+Project-URL: Issues, https://github.com/255BITS/foldlitedb/issues
+Project-URL: Repository, https://github.com/255BITS/foldlitedb
+
+# Foldlite DB
+
+**Immutable documents. Rebuildable live views. Rust and WebAssembly.**
+
+Foldlite DB is a CouchDB-inspired map/reduce engine built around a simple rule:
+keep immutable documents, derive everything else. The Rust engine appends accepted
+document batches to a durable log. Indexes, source checkpoints, group states, and
+change history form a disposable cache that can be reconstructed from that log.
+There is one native implementation, exposed to Python; storage needs no database
+server or external database library.
+
+Write a view in AssemblyScript and compile it to WebAssembly. The same module
+maps and reduces documents in Rust or entirely in a browser worker. Try the
+[browser playground](https://255bits.github.io/foldlitedb/playground.html) or read
+the [view authoring guide](docs/portable.html).
+
+## Install
+
+Use Python 3.11+ to install the 0.1.0 release:
+
+```sh
+python -m pip install foldlitedb==0.1.0
+```
+
+Native wheels are provided for Linux x86-64 and ARM64, macOS Intel and Apple
+Silicon, and Windows x86-64. Installing a matching wheel needs neither Rust nor
+Node. The [hosted documentation](https://255bits.github.io/foldlitedb/) describes
+the API and bundled portable views.
+
+### Build from this checkout
+
+Building from source requires Python 3.11+, Rust/Cargo 1.90+, and a platform C
+compiler and linker. From the repository root:
+
+```sh
+python -m venv .venv
+. .venv/bin/activate
+python -m pip install --upgrade pip
+python -m pip install .
+python examples/portable_metrics.py
+```
+
+On Windows, activate with `.venv\Scripts\activate` and use an MSVC build
+environment. Node.js is unnecessary for installation and bundled views. To make
+a wheel, run `python -m pip wheel . --no-deps --wheel-dir dist`. Install that
+wheel on a matching platform with `python -m pip install /path/to/the-built.whl`;
+installing a built wheel needs neither Rust nor Node. See the
+[release guide](docs/releasing.html) for PyPI Trusted Publisher setup and the
+0.1.0 publication workflow.
+
+## Append documents, read a live view
+
+Start with a fresh `metrics.fold` file:
+
+```python
+from foldlitedb import Database, View
+
+view = View.bundled("metrics")
+with Database("metrics.fold", view) as db:
+    documents = db.add_many([
+        {"metric": "loss", "value": 4},
+        {"metric": "loss", "value": 8},
+    ])
+    assert list(db.rows())[0].result == {
+        "count": 2, "sum": 12, "min": 4, "max": 8, "mean": 6,
+    }
+    snapshot = db.query()
+    documents += (db.add({"metric": "loss", "value": 3}),)
+    assert db.changes(since=snapshot.cursor).rows[0].result["mean"] == 5
+    retained = tuple(db.documents())
+    db.checkpoint()
+```
+
+Rust executes both map and reduce through a bounded Wasm interpreter. `metrics`
+groups finite numbers by a string `metric`; `catalogue` demonstrates an ordered
+fold over structured events. Each database holds one view module. `Database` and
+`View` are the primary Python names; `PortableDatabase` and `PortableView` are
+aliases for the same types.
+
+Try the [telemetry collector](examples/telemetry.py) for bulk import, retries, and
+a persistent changes consumer, or the [catalogue example](examples/catalogue.py)
+for an ordered item lifecycle. If a payload produces no rows, inspect its map
+output with [`db.preview(payload)`](docs/api.html#preview).
+
+Every unnamed append creates a new observation. Give an observation an ID when
+identical retries should count once. IDs and positions are scoped to a source;
+conflicting reuse raises an error. Documents cannot be updated or deleted. Append
+new observations to represent changes, then let the view fold their history.
+
+`add_many` commits one atomic batch of at most 1024 documents. Its complete Wasm
+map request must fit 256 KiB, so larger payloads need smaller batches.
+`rows()` follows a stable snapshot in bounded pages; `query()` and
+`changes(since=...)` expose continuation cursors. See the
+[Python API](docs/api.html#database) for signatures and limits.
+
+## A log you keep; a cache you can discard
+
+For a persistent database, `metrics.fold` contains the immutable document log.
+`metrics.fold.cache` is an optional checkpoint; `metrics.fold.lock` coordinates
+one active owner. A batch is acknowledged after its log frame is synced. The
+engine validates and computes a batch before appending it, so a map or reducer
+failure does not commit a prefix.
+
+`checkpoint()` writes an atomic cache snapshot; closing cleanly does the same.
+On reopen, the engine verifies the log and matching cache prefix, then runs Wasm
+only for batches after the checkpoint. Missing, corrupt, or incompatible caches
+are rebuilt from the documents. `rebuild()` explicitly recomputes the view and
+starts a new cursor generation. Neither operation changes the source documents.
+
+The document log is independent of the view module. Opening it with different
+module bytes rebuilds the cache in place; retain the exact module if you need to
+reproduce its previous results. Close the database before backing up the log and
+view artifacts; Windows locks prevent other handles from reading an active log.
+The cache is replaceable; the only copy of a document log is not.
+
+Rebuild uses the original batch order. It scans the log linearly and runs each
+uncached batch through the same map/reduce path as live ingestion. A valid cache
+avoids repeated Wasm work, but reopening still verifies the complete log. Current
+indexes and history grow in memory with the data. Cache snapshots are limited to
+1 GiB of encoded checkpoint data; a checkpoint or close error reports a failure
+to save that accelerator, while committed log data remains intact. There is no
+claim of constant memory or measured rebuild throughput.
+
+To copy retained records into a fresh database:
+
+```python
+with Database("copy.fold", view) as copied:
+    copied.ingest(retained)
+    assert list(copied.rows())[0].result["mean"] == 5
+    before = [(r.key, r.state, r.result) for r in copied.rows()]
+    copied.rebuild()
+    assert [(r.key, r.state, r.result) for r in copied.rows()] == before
+```
+
+Use `ingest_many(records, batch_size=100)` for larger archives. **Iterate its
+results to perform the import**: each yielded result commits one separate bounded
+batch, and earlier batches remain committed if a later batch fails. See the
+[retry-safe source import](docs/api.html#bulk-import) for constructing records with
+upstream positions or stable IDs alone. Preserve record order; preserve original
+batch boundaries too if you need identical engine sequence values.
+
+## Author a view, run it on a website
+
+The [AssemblyScript SDK](views/README.md) provides JSON helpers and the Wasm ABI.
+Implement `map`, `identity`, `add`, and `finalize`; implement `merge` only for
+states that can be combined correctly. Inputs and state are explicit, and views
+must be deterministic. An ordered fold must give equivalent state when the same
+contributions are split across successive batches.
+
+Rebuild the two bundled examples with Node.js 20+:
+
+```sh
+npm ci --prefix views
+npm run build --prefix views
+npm test --prefix views
+npm run check --prefix views
+```
+
+The build writes modules and SHA-256 manifests into
+`src/foldlitedb/portable_assets/`. It builds the named `metrics` and `catalogue`
+examples. Compile a custom view and generate its manifest with:
+
+```sh
+npm run build:view --prefix views -- assembly/examples/my-view.ts --out-dir .build
+```
+
+Both paths are relative to `views/`. The
+[authoring guide](docs/portable.html#authoring) includes a complete custom view
+and the build command to generate its Wasm and digest manifest.
+
+`foldlitedb.portable_assets()` exposes compiled modules plus `client.js`,
+`worker.js`, and `host.js`. Copy them to a static site and import
+`BrowserDatabase` from `client.js`. The [browser example](docs/portable.html#browser)
+covers appends, snapshots, changes, export, and replay verification. JavaScript
+owns in-memory grouping and storage in the browser worker; the Rust log engine
+is not compiled into this browser build.
+
+Browser state is lost on reload or close. Export resolved documents first. The
+preview bounds documents, groups, history, and request sizes. Persistent browser
+storage, log/history compaction, and multiple views per native database remain
+future work. JSON numbers use IEEE-754 double precision; use strings for exact
+arbitrary-precision identifiers.
+
+## Development and documentation
+
+```sh
+python -m pip install '.[dev,browser]'
+python -m playwright install chromium
+python -m pytest
+python -m pytest tests/browser
+cargo test --locked --manifest-path native/Cargo.toml -p foldlitedb-portable-engine
+npm ci --prefix views
+npm test --prefix views
+npm run check --prefix views
+python scripts/build_docs.py
+python scripts/check_docs.py --execute --browser
+python -m http.server 8000 --directory _site
+```
+
+The docs checker executes annotated Python examples in fresh temporary
+directories and checks links, search, and desktop/mobile rendering in Chromium.
+Open `http://localhost:8000` for the built docs and playground. An authorized
+maintainer can publish checked generated docs with
+`python scripts/publish_docs.py`; building alone does not publish them.
+
+Measure the native path on your target filesystem with:
+
+```sh
+python scripts/benchmark_database.py --documents 10000 --keys 16 --batch-size 100 --live-documents 20 --output native-benchmark.json
+```
+
+Use `--storage-root /path/to/filesystem` to select the filesystem being measured.
+The report compares durable ingestion, checkpointing, cached reopen, suffix
+replay, and full reconstruction, while checking unchanged log bytes and matching
+results. Local measurements describe that fixture and host.
+
+See [DESIGN.md](DESIGN.md) for durability, replay, cache validation, and execution
+contracts.
+

foldlitedb-0.1.0/README.md

@@ -0,0 +1,218 @@
+# Foldlite DB
+
+**Immutable documents. Rebuildable live views. Rust and WebAssembly.**
+
+Foldlite DB is a CouchDB-inspired map/reduce engine built around a simple rule:
+keep immutable documents, derive everything else. The Rust engine appends accepted
+document batches to a durable log. Indexes, source checkpoints, group states, and
+change history form a disposable cache that can be reconstructed from that log.
+There is one native implementation, exposed to Python; storage needs no database
+server or external database library.
+
+Write a view in AssemblyScript and compile it to WebAssembly. The same module
+maps and reduces documents in Rust or entirely in a browser worker. Try the
+[browser playground](https://255bits.github.io/foldlitedb/playground.html) or read
+the [view authoring guide](docs/portable.html).
+
+## Install
+
+Use Python 3.11+ to install the 0.1.0 release:
+
+```sh
+python -m pip install foldlitedb==0.1.0
+```
+
+Native wheels are provided for Linux x86-64 and ARM64, macOS Intel and Apple
+Silicon, and Windows x86-64. Installing a matching wheel needs neither Rust nor
+Node. The [hosted documentation](https://255bits.github.io/foldlitedb/) describes
+the API and bundled portable views.
+
+### Build from this checkout
+
+Building from source requires Python 3.11+, Rust/Cargo 1.90+, and a platform C
+compiler and linker. From the repository root:
+
+```sh
+python -m venv .venv
+. .venv/bin/activate
+python -m pip install --upgrade pip
+python -m pip install .
+python examples/portable_metrics.py
+```
+
+On Windows, activate with `.venv\Scripts\activate` and use an MSVC build
+environment. Node.js is unnecessary for installation and bundled views. To make
+a wheel, run `python -m pip wheel . --no-deps --wheel-dir dist`. Install that
+wheel on a matching platform with `python -m pip install /path/to/the-built.whl`;
+installing a built wheel needs neither Rust nor Node. See the
+[release guide](docs/releasing.html) for PyPI Trusted Publisher setup and the
+0.1.0 publication workflow.
+
+## Append documents, read a live view
+
+Start with a fresh `metrics.fold` file:
+
+```python
+from foldlitedb import Database, View
+
+view = View.bundled("metrics")
+with Database("metrics.fold", view) as db:
+    documents = db.add_many([
+        {"metric": "loss", "value": 4},
+        {"metric": "loss", "value": 8},
+    ])
+    assert list(db.rows())[0].result == {
+        "count": 2, "sum": 12, "min": 4, "max": 8, "mean": 6,
+    }
+    snapshot = db.query()
+    documents += (db.add({"metric": "loss", "value": 3}),)
+    assert db.changes(since=snapshot.cursor).rows[0].result["mean"] == 5
+    retained = tuple(db.documents())
+    db.checkpoint()
+```
+
+Rust executes both map and reduce through a bounded Wasm interpreter. `metrics`
+groups finite numbers by a string `metric`; `catalogue` demonstrates an ordered
+fold over structured events. Each database holds one view module. `Database` and
+`View` are the primary Python names; `PortableDatabase` and `PortableView` are
+aliases for the same types.
+
+Try the [telemetry collector](examples/telemetry.py) for bulk import, retries, and
+a persistent changes consumer, or the [catalogue example](examples/catalogue.py)
+for an ordered item lifecycle. If a payload produces no rows, inspect its map
+output with [`db.preview(payload)`](docs/api.html#preview).
+
+Every unnamed append creates a new observation. Give an observation an ID when
+identical retries should count once. IDs and positions are scoped to a source;
+conflicting reuse raises an error. Documents cannot be updated or deleted. Append
+new observations to represent changes, then let the view fold their history.
+
+`add_many` commits one atomic batch of at most 1024 documents. Its complete Wasm
+map request must fit 256 KiB, so larger payloads need smaller batches.
+`rows()` follows a stable snapshot in bounded pages; `query()` and
+`changes(since=...)` expose continuation cursors. See the
+[Python API](docs/api.html#database) for signatures and limits.
+
+## A log you keep; a cache you can discard
+
+For a persistent database, `metrics.fold` contains the immutable document log.
+`metrics.fold.cache` is an optional checkpoint; `metrics.fold.lock` coordinates
+one active owner. A batch is acknowledged after its log frame is synced. The
+engine validates and computes a batch before appending it, so a map or reducer
+failure does not commit a prefix.
+
+`checkpoint()` writes an atomic cache snapshot; closing cleanly does the same.
+On reopen, the engine verifies the log and matching cache prefix, then runs Wasm
+only for batches after the checkpoint. Missing, corrupt, or incompatible caches
+are rebuilt from the documents. `rebuild()` explicitly recomputes the view and
+starts a new cursor generation. Neither operation changes the source documents.
+
+The document log is independent of the view module. Opening it with different
+module bytes rebuilds the cache in place; retain the exact module if you need to
+reproduce its previous results. Close the database before backing up the log and
+view artifacts; Windows locks prevent other handles from reading an active log.
+The cache is replaceable; the only copy of a document log is not.
+
+Rebuild uses the original batch order. It scans the log linearly and runs each
+uncached batch through the same map/reduce path as live ingestion. A valid cache
+avoids repeated Wasm work, but reopening still verifies the complete log. Current
+indexes and history grow in memory with the data. Cache snapshots are limited to
+1 GiB of encoded checkpoint data; a checkpoint or close error reports a failure
+to save that accelerator, while committed log data remains intact. There is no
+claim of constant memory or measured rebuild throughput.
+
+To copy retained records into a fresh database:
+
+```python
+with Database("copy.fold", view) as copied:
+    copied.ingest(retained)
+    assert list(copied.rows())[0].result["mean"] == 5
+    before = [(r.key, r.state, r.result) for r in copied.rows()]
+    copied.rebuild()
+    assert [(r.key, r.state, r.result) for r in copied.rows()] == before
+```
+
+Use `ingest_many(records, batch_size=100)` for larger archives. **Iterate its
+results to perform the import**: each yielded result commits one separate bounded
+batch, and earlier batches remain committed if a later batch fails. See the
+[retry-safe source import](docs/api.html#bulk-import) for constructing records with
+upstream positions or stable IDs alone. Preserve record order; preserve original
+batch boundaries too if you need identical engine sequence values.
+
+## Author a view, run it on a website
+
+The [AssemblyScript SDK](views/README.md) provides JSON helpers and the Wasm ABI.
+Implement `map`, `identity`, `add`, and `finalize`; implement `merge` only for
+states that can be combined correctly. Inputs and state are explicit, and views
+must be deterministic. An ordered fold must give equivalent state when the same
+contributions are split across successive batches.
+
+Rebuild the two bundled examples with Node.js 20+:
+
+```sh
+npm ci --prefix views
+npm run build --prefix views
+npm test --prefix views
+npm run check --prefix views
+```
+
+The build writes modules and SHA-256 manifests into
+`src/foldlitedb/portable_assets/`. It builds the named `metrics` and `catalogue`
+examples. Compile a custom view and generate its manifest with:
+
+```sh
+npm run build:view --prefix views -- assembly/examples/my-view.ts --out-dir .build
+```
+
+Both paths are relative to `views/`. The
+[authoring guide](docs/portable.html#authoring) includes a complete custom view
+and the build command to generate its Wasm and digest manifest.
+
+`foldlitedb.portable_assets()` exposes compiled modules plus `client.js`,
+`worker.js`, and `host.js`. Copy them to a static site and import
+`BrowserDatabase` from `client.js`. The [browser example](docs/portable.html#browser)
+covers appends, snapshots, changes, export, and replay verification. JavaScript
+owns in-memory grouping and storage in the browser worker; the Rust log engine
+is not compiled into this browser build.
+
+Browser state is lost on reload or close. Export resolved documents first. The
+preview bounds documents, groups, history, and request sizes. Persistent browser
+storage, log/history compaction, and multiple views per native database remain
+future work. JSON numbers use IEEE-754 double precision; use strings for exact
+arbitrary-precision identifiers.
+
+## Development and documentation
+
+```sh
+python -m pip install '.[dev,browser]'
+python -m playwright install chromium
+python -m pytest
+python -m pytest tests/browser
+cargo test --locked --manifest-path native/Cargo.toml -p foldlitedb-portable-engine
+npm ci --prefix views
+npm test --prefix views
+npm run check --prefix views
+python scripts/build_docs.py
+python scripts/check_docs.py --execute --browser
+python -m http.server 8000 --directory _site
+```
+
+The docs checker executes annotated Python examples in fresh temporary
+directories and checks links, search, and desktop/mobile rendering in Chromium.
+Open `http://localhost:8000` for the built docs and playground. An authorized
+maintainer can publish checked generated docs with
+`python scripts/publish_docs.py`; building alone does not publish them.
+
+Measure the native path on your target filesystem with:
+
+```sh
+python scripts/benchmark_database.py --documents 10000 --keys 16 --batch-size 100 --live-documents 20 --output native-benchmark.json
+```
+
+Use `--storage-root /path/to/filesystem` to select the filesystem being measured.
+The report compares durable ingestion, checkpointing, cached reopen, suffix
+replay, and full reconstruction, while checking unchanged log bytes and matching
+results. Local measurements describe that fixture and host.
+
+See [DESIGN.md](DESIGN.md) for durability, replay, cache validation, and execution
+contracts.