proto-db 2.0.0rc3__tar.gz

proto_db-2.0.0rc3/CHANGELOG.md

@@ -0,0 +1,224 @@
+# Changelog
+
+All notable changes to `proto_db` are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [2.0.0rc3] - 2026-11-13
+
+The first release candidate published on PyPI.
+- Python 3.12 is the minimum (D61): on 3.11, comparing deeply nested tuples counts against the recursion
+  limit, so a value near `MAX_VALUE_DEPTH` could raise a bare `RecursionError`. `pip` refuses to
+  install it on 3.11 (`requires-python >=3.12`).
+- Test suite: process-wide object counts replaced by per-test counts, a stress mode for the cluster tests
+  (`PROTO_STRESS`, safety asserts only), and fixes to tests that failed in a clean export or under load.
+- CI: a test matrix over 3.12, 3.13, 3.14 and 3.14t, and a `sphinx-build -W` documentation job.
+- Benchmarks: a 10^7-object inventory and the periodic cluster stress task, with result files under
+  `benchmarks/results/`.
+
+## [2.0.0] - unreleased
+
+`proto_db` 2.0 is a complete rebuild of the 0.1 code base under the design principles of
+`docs/design-principles.md`: everything stored is immutable, a transaction is an explicit object, and
+every safety argument holds under arbitrary delay and loss. There are no 0.1 data or API to preserve.
+The design is `docs/superpowers/specs/2026-11-01-protodb-v2-design.md`; every decision is in
+`docs/decisions.md` (D1-D61).
+
+### Added
+- **One value model** with a total order across types (`1 == 1.0 == Decimal(1)`, `True != 1`), used by
+  collections, indexes and queries identically (D2, D3); frozen `list` and `dict` values (D25).
+- **Atoms** with stable identity (`oid`), lazy thread-safe loading and msgpack serialisation (D10, D18, D21).
+- **Collections**, all persistent AVL trees and all indexable: `List`, `Dictionary` (key-ordered),
+  `HashDictionary`, `RepeatedKeysDictionary`, `Set`, `CountedSet`, and `DBObject` records (D8, D20-D27).
+- **Index framework**: `Index` base class with a documented hook contract, `KeySpec` (paths, the element,
+  registered extractors), `EqualityIndex`, `RangeIndex` and `VectorIndex` (exact, IVF-Flat, optional HNSW),
+  every mutator of every collection keeping its indexes current, and a reusable index contract
+  (`tests/contracts/index_contract.py`) for new kinds (D22, D29, D30, D39).
+- **Query layer**: expression terms, physical plans, one cost-based planner that never drops a predicate, a
+  LINQ front end (`from_collection`, `F`, `Agg`), explicit `PythonFilter` nodes, `explain()`, a query result
+  cache keyed by pointers, and `Pager`s with portable tokens over snapshots (D28, D36-D38, D56).
+- **Transactions**: snapshot isolation with read-set validation (absent roots included), `Database.run` with
+  bounded exponential back-off and jitter, per-key and per-element merge of unread collection roots,
+  `MutableObject` cells, group commit, `Database.history()` and read-only `Snapshot`s of any past version,
+  `ProtoOutcomeUnknownError` for commits whose outcome cannot be proved (D9, D11, D32-D35, D52).
+- **Storage v2**: `MemoryStorage`; `FileStorage` with append-only CRC-checked segments, atomic root
+  publication and `fsync` per commit by default (`durability="flush"` and `"none"` are explicit opt-outs);
+  `CloudStorage` over S3 or Google Cloud Storage with data strictly before root and a conditional root write;
+  `TieredStorage` (D4, D6, D19, D43, D46, D51).
+- **Cluster**: several writer processes through a peer coordinator (majority lease, write permissions, commit
+  by majority acknowledgement, bounded replication, learners, fencing), with the object store written after
+  the commit point (D44-D46, D50, D54, D55). `ObjectSpace.sync()` is a durability barrier; a barrier commit is
+  the linearizable read path (it adds a permanent database version per read).
+- **Decoded-atom cache** shared by the transactions and threads of a space, by pointer, bounded on loaded
+  atoms, with counters (`ObjectSpace.atom_cache_stats()`) that expose a thrashing working set (D53).
+- **Runtime services**: `parallel_scan` and `parallel_filter` (real speedup on free-threaded CPython),
+  `FSM`; **interop**: `to_arrow`, `to_parquet`, `scan_parquet` (D40-D42).
+- **Free-threading**: tested on CPython 3.14t with the GIL disabled and with the GIL.
+- **Performance suite** (`python -m benchmarks`): self-verifying benchmarks, with results by workload
+  profile in `docs/source/performance.rst` (D49).
+- **Documentation**: a rewritten Sphinx site (concepts, design pages, ten executable tutorials, API
+  reference) and a test that runs every tutorial code block (`tests/docs/test_tutorials.py`).
+- A single exception hierarchy rooted at `ProtoDBError` (D12), and a static check that forbids swallowed
+  exceptions and test-aware code in `proto_db/` (D16).
+
+### Changed
+- The public API is the set of names in `proto_db.__all__`, resolved lazily: `import proto_db` imports no
+  submodule or optional dependency and starts no thread (D47).
+- Dictionaries are key-ordered AVL trees (O(log n)), not a binary search over a positional list (D8).
+- A conflict raises `ProtoConflictError` and the caller retries; merging applies only to roots the transaction
+  did not read and only by replaying recorded operations, never by inferring intent from value types (D9, D33).
+- Strings are stored inline; the global `Literal` catalog is gone (D7).
+- Python floor 3.12 (D61; was 3.11 under D14); CI targets 3.12-3.14 and 3.14t. `six` and `wrapt` are no longer
+  dependencies (D14).
+- The package is split into subpackages: `core`, `storage`, `collections`, `indexing`, `query`, `cluster`,
+  `runtime`, `interop` (D15; the spec's `indexes` became `indexing`).
+
+- Storage format 4 (D58): exact tokens are canonical encodings, datetimes in an IANA zone are stored as their
+  UTC instant, map keys follow an explicit conflict rule, and the class of a built-in subclass is not persisted
+  (a `str` or `int` `Enum` is stored and read back as its value). Formats 2 and 3 are still read. Migration
+  cost: a key-group or vector index written by format 3 is rebuilt from its collection at the collection's
+  first change, inside that write transaction (an ivf_flat index retrains, an hnsw graph is rebuilt). A
+  format-3 map keyed by a zone datetime or by repeated NaNs is read as written but refused when written again
+  until the application converts those keys.
+- Polyglot freeze (D59, Task 10.P): a record of a type this process has no class for (another language's
+  type) loads as an `OpaqueAtom` that keeps its fields readable and is never rewritten, instead of raising
+  `ProtoCorruptionError`; root names starting with `__` are reserved for the format (`ProtoValidationError`);
+  the attribute-name errors explain the accessor rule. `python -m proto_db.tools.verify <dir>` decodes and
+  validates a whole file-storage space and dumps it in the conformance notation. The format specification
+  gains "Format stability" (the freeze point, the additive `@sym` and registry), "Atoms of unknown types",
+  the accessor rule and "A minimal external writer"; `tests/conformance/vectors.sha256` pins the vectors.
+  After the pre-tag review: a reference to an opaque atom may name its type differently from the record's
+  `t` (an alias only the owner knows) and keeps the name it was referenced by; writers write the lowest
+  sufficient format version (`WRITE_FORMAT_VERSION`, separate from the highest read, `FORMAT_VERSION`); the
+  verifier also reports non-canonical encodings and exits 2 only for an unreadable root; a field named like a
+  metaclass attribute (`mro`) is readable; a `date` or `timedelta` subclass is stored from its base fields,
+  never its `__reduce__`.
+- Format-3 data misfiled by built-in subclasses: format 3 wrote a `(str, Enum)` member (on every Python
+  version) or a subclass overriding `__str__`, `__float__` or `__bytes__` through that conversion
+  (`"Color.RED"`) but filed it by its base value (`"red"`). Collections holding such values cannot find them,
+  and a key-ordered Dictionary also loses other keys. `collection.validate()` now detects it
+  (`ProtoMisfiledError`, a `ProtoCorruptionError`) and `tx.set_root_object(name, collection.rebuilt())`
+  repairs it (entries refiled, indexes rebuilt; one key with two different values is refused, never dropped).
+- IVF-Flat training no longer depends on the members' oids (R128, 012db7f): a bulk build of the same vectors
+  with the same `seed` gives the same centroids and the same answers, in any order (an index grown by inserts
+  still trains on the members present at each retrain threshold, which depends on the insertion order). An `ivf_flat` index built before 012db7f
+  keeps its persisted centroids until it retrains; at its next retrain (or rebuild) it gets different
+  centroids than that same data gave before, and stable ones from then on. No format change.
+
+### Deprecated
+- The 0.1 exception names `ProtoBaseException`, `ProtoValidationException`, `ProtoUserException`,
+  `ProtoCorruptionException`, `ProtoNotSupportedException`, `ProtoNotAuthorizedException`,
+  `ProtoUnexpectedException` and `ProtoLockingException`: aliases of the `Proto*Error` classes, removed in 2.1
+  (D12).
+
+### Removed
+- The 0.1 implementation (`common`, `db_access`, the 0.1 storages, collections, queries, LINQ, vectors,
+  `parallel`, `fsm`, `arrow_bridge`, `hybrid_executor`, `exceptions`) and its test suite `proto_db/tests/`
+  (D47, D48; dropped tests: `docs/legacy/dropped-tests.md`). UDP voting is gone from the cloud storages (D5).
+
+### Fixed
+- Saving a chain of unsaved versions built in one transaction (the `previous` pattern) no longer raises
+  `RecursionError` past about 150 links; 10,000 versions save in one commit, and the verifier describes them
+  (R128).
+- Nested values: `proto_db` no longer writes values it cannot read back (B12, D60). The codec, `canonical()`,
+  the map-key, order-key, hash and equality helpers and the verifier walked values by recursion, so a value
+  past about 330 nested lists could be written but raised a bare `RecursionError` when read in a fresh
+  process (`verify` exited 1), and the limit moved with the Python stack in use. They now go on with an
+  explicit stack past 12 levels, and the format has a normative limit, `MAX_VALUE_DEPTH` = 1022 MessagePack
+  containers per value (format spec, "Value nesting depth"): a writer refuses a deeper value with
+  `ProtoValidationError` naming the field, a reader refuses deeper data with `ProtoCorruptionError`, at the
+  same count. No format version change (no conforming writer could produce deeper data); the encoded bytes
+  are unchanged. `proto_db` now requires msgpack 1.2.0 or later, the first release whose packer nests the 1024
+  containers a record at the limit needs, and refuses to import with an older one. The limit binds collection
+  and index nodes too: a Set, CountedSet, HashDictionary or RepeatedKeysDictionary, and an index (whose member
+  keys nest an unorderable element about four times deeper: 254 nested lists, 169 nested maps), refuse an
+  element they cannot store when it is added, naming the collection or the index and the effective limit,
+  never later at commit. The query result cache, `explain()` and `to_arrow` work at any depth too.
+  **Readers must accept records 1024 containers deep**: an implementation that stops at 509 levels is not
+  conforming from rc2.
+
+Defects of the 0.1 code base found by the 2026-11-01 review, none of which survives the rebuild: integer
+values merged as `+1` and special cases for keys named `counter` and `k`; commits that were not durable;
+indexes lost on the first write; queries that silently returned empty or partial results; about 280 broad
+`except` handlers hiding errors; two writers appending to one segment.
+
+### Known limitations
+- No `proto_db.tools.inspect` browser (the spec's section 6 named one); the formats are specified in
+  `docs/source/format-spec.rst`, `python -m proto_db.tools.verify` checks and dumps a space, and the operations
+  tutorial shows both.
+- No online-backup call; see the operations tutorial for what a copy of an open space contains.
+- Cluster fairness for eight or more contending writer processes, and for hot-key read-modify-write
+  transactions, is not guaranteed (open defect D-LIVE-2).
+- A read-only linearizable read (`read_barrier`) is proposed in the D46 amendment but not implemented; the
+  barrier commit is the supported path.
+
+## Before 2.0
+
+Entries from the 0.1 line, kept as written. They describe code that 2.0 removed.
+
+### Added (0.1 line, unreleased)
+- Optional parallel scanning module (proto_db.parallel) with adaptive chunking and a simple work-stealing scheduler.
+- Metrics hooks for per-worker and global stats (chunk latencies, steals, queue depths).
+- parallel_scan helper for easy opt-in usage without altering existing query code.
+- Documentation: user guide (Parallel Scans), API reference, and ADR.
+- Unit tests for adaptive chunking, work-stealing behavior, cancellation, and fallback correctness.
+
+### Changed
+- Attribute access style: migrated docs/tests/examples to prefer dot notation; clarified DBObject semantics and dynamic getattr in README.
+
+## [0.1.0] - 2023
+
+### Added
+
+#### Google Cloud Storage Support
+
+- Added `GoogleCloudClient` as an implementation of `CloudStorageClient` for Google Cloud Storage
+- Ensured that `CloudBlockProvider`, `CloudFileStorage`, and `CloudClusterFileStorage` work with both `S3Client` and
+  `GoogleCloudClient`
+- Added comprehensive test suites for Google Cloud Storage integration
+- Added usage examples for Google Cloud Storage in documentation
+
+#### Multiple Serialization Formats Support
+
+- Added a format indicator after the size header to specify the serialization format
+- Added support for JSON UTF-8 (the original format) and MessagePack (a new efficient binary format)
+- Maintained backward compatibility with existing data
+- Added new methods to support the new formats:
+    - `push_atom(atom: dict, format_type: int = FORMAT_JSON_UTF8) -> Future[AtomPointer]`
+    - `push_atom_msgpack(atom: dict) -> Future[AtomPointer]`
+    - `push_bytes(data: bytes, format_type: int = FORMAT_RAW_BINARY) -> Future[tuple[uuid.UUID, int]]`
+    - `push_bytes_msgpack(data: dict) -> Future[tuple[uuid.UUID, int]]`
+
+#### PyPI Package Support
+
+- Created `pyproject.toml` with package metadata, dependencies, and build system information
+- Added MIT License file
+- Created `MANIFEST.in` to include non-Python files in the package distribution
+- Added `test_install.py` script to test the local installation of the package
+- Created `PYPI_UPLOAD.md` with documentation on how to build and upload the package to PyPI
+- Updated README.md with installation instructions and license information
+
+#### Advanced Storage Options Documentation
+
+- Added comprehensive documentation for `ClusterFileStorage` and `CloudFileStorage`
+- Updated `__init__.py` to expose the advanced storage options
+- Added detailed use cases for the advanced storage options
+
+### Fixed
+
+#### CloudFileStorage Test Hanging Issue
+
+- Added an `uploader_running` flag to control the background uploader thread
+- Modified the `_background_uploader` method to check both the `state` and the `uploader_running` flag
+- Updated the `close` method to properly stop the background uploader thread and wait for it to finish
+- Added test verification to ensure the uploader thread is properly stopped
+
+### Changed
+
+- Completed the implementation of `S3Client` by adding missing abstract methods:
+    - `_init_client`: Initializes the S3 client using boto3 or falls back to a mock implementation
+    - `get_object`: Gets an object from S3
+    - `put_object`: Puts an object to S3
+    - `list_objects`: Lists objects in S3
+    - `delete_object`: Deletes an object from S3

proto_db-2.0.0rc3/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 ProtoBase Team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

proto_db-2.0.0rc3/MANIFEST.in

@@ -0,0 +1,16 @@
+include LICENSE
+include README.md
+include CHANGELOG.md
+include PYPI_UPLOAD.md
+
+# Exclude non-runtime large artifacts
+prune examples
+prune docs/build
+prune cloud_page_cache
+prune benchmark_data_small
+
+# Exclude benchmark outputs and misc
+global-exclude *.pyc __pycache__ *.pyo *.DS_Store
+global-exclude *.json
+
+recursive-include proto_db *.py

proto_db-2.0.0rc3/PKG-INFO

@@ -0,0 +1,315 @@
+Metadata-Version: 2.4
+Name: proto_db
+Version: 2.0.0rc3
+Summary: proto_db is an embedded, transactional, object-oriented database for Python.
+Author-email: Gustavo Marino <gamarino@numaes.com>
+License: MIT License
+        
+        Copyright (c) 2023 ProtoBase Team
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+Project-URL: Homepage, https://github.com/gamarino/ProtoDB
+Project-URL: Bug Tracker, https://github.com/gamarino/ProtoDB/issues
+Keywords: database,embedded,transactional,object-oriented,ACID,ORM-free,NoSQL
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: Free Threading
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: msgpack>=1.2.0
+Provides-Extra: parquet
+Requires-Dist: pyarrow; extra == "parquet"
+Provides-Extra: vectors
+Requires-Dist: numpy; extra == "vectors"
+Provides-Extra: hnsw
+Requires-Dist: numpy; extra == "hnsw"
+Requires-Dist: hnswlib; extra == "hnsw"
+Provides-Extra: s3
+Requires-Dist: boto3; extra == "s3"
+Provides-Extra: gcs
+Requires-Dist: google-cloud-storage; extra == "gcs"
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-timeout; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: hypothesis; extra == "dev"
+Requires-Dist: sphinx; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Dynamic: license-file
+
+# proto_db
+
+[![PyPI version](https://img.shields.io/pypi/v/proto_db.svg)](https://pypi.org/project/proto_db/)
+[![Python Version](https://img.shields.io/pypi/pyversions/proto_db.svg)](https://pypi.org/project/proto_db/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+[![Project Status](https://img.shields.io/badge/status-2.0.0.dev0-orange.svg)](#status)
+
+`proto_db` is an embedded, transactional, object-oriented database for Python. It stores immutable,
+persistent collections with secondary indexes, runs queries through an index-aware planner, and
+commits serializable transactions per root, all in-process, without a separate server.
+
+### Status
+
+`2.0.0.dev0`: the 2.0 rebuild, unreleased (design: `docs/superpowers/specs/2026-11-01-protodb-v2-design.md`,
+decisions: `docs/decisions.md`). The 0.1 implementation has been removed; the API below is the 2.0 API.
+Everything this page lists is implemented and covered by the test suite, which runs on CPython 3.14t
+(free-threaded) and with the GIL. The cluster is new: it is documented, tested and measured in this
+repository and has no production history. Open defects are listed where they apply (see the cluster item
+under Features). The performance suite has run (results by workload in `docs/source/performance.rst`);
+the API may still change before 2.0.0 is released.
+
+### A note on names
+
+The product is named after its import. The repository has its own name, and the sibling
+Smalltalk project keeps its name too:
+
+| | Name |
+|---|---|
+| The product (the library, the database) and its documentation | **`proto_db`** — `pip install proto_db`, `import proto_db` |
+| The repository | **protoBase** — `github.com/gamarino/ProtoDB` |
+| The Smalltalk sibling project | **protoBaseST** |
+
+Earlier documentation called the product *ProtoDB* or *ProtoBase*; both are the same
+product, now named `proto_db` everywhere so that the name matches the import. The old
+spelling survives in `ProtoBaseException`, a deprecated alias of `ProtoDBError` (the base
+of the exception hierarchy) that is removed in 2.1.
+
+## Design principles
+
+`proto_db` is built from a small set of principles. Every design decision cites them, and
+a change that contradicts one is treated as conceptual debt, resolved before it is built.
+If you read only one thing before the code, read
+[docs/design-principles.md](docs/design-principles.md).
+
+1. **Everything is immutable.** A pointer names one content forever. History is never lost,
+   caches keyed by pointer never go stale, and snapshots are stable.
+2. **A transaction is an explicit object.** It is not bound to a thread, and its commit
+   concerns only its own reads and writes.
+3. **Each node owns its work.** Commits are built where they happen, local contention is
+   resolved first, and replication is bounded.
+4. **Contention is resolved among the processes, never by the storage.** The storage only
+   fences, and the object store is written after the commit.
+5. **Safety never depends on time.** Timers drive liveness only, and delays can cause only
+   conservative failures.
+6. **Known techniques, honestly measured.** Established mechanisms are adopted and cited, and
+   every performance claim carries its measurement.
+7. **Correct before fast.**
+
+## Features
+
+- **Transactions**: snapshot isolation, read sets checked at commit (a conflict raises
+  `ProtoConflictError`; `Database.run` retries), per-key merges of collections written without being
+  read, named mutable cells (`MutableObject`), a version history per database.
+- **Values**: one value model with a total order across types (`1 == 1.0 == Decimal(1)`, `True != 1`),
+  used identically by collections, indexes and queries.
+- **Collections**: `List`, `Dictionary` (key-ordered), `HashDictionary`, `RepeatedKeysDictionary`,
+  `Set`, `CountedSet` and `DBObject` records, all immutable and persistent (every change returns a new
+  version; unchanged parts are shared).
+- **Indexes on every collection**: `EqualityIndex`, `RangeIndex`, `VectorIndex` (exact, IVF-Flat, or
+  HNSW when `hnswlib` is installed), and an `Index` base class for custom indexes. Indexes are
+  persisted with the collection and maintained by every mutator.
+- **Queries**: a LINQ front end (`from_collection`, `F`) and expression terms over physical plans; the
+  planner picks indexes by cost, never drops a predicate, and `explain()` shows the plan that runs.
+  Python lambdas are allowed and appear as an explicit `PythonFilter` node.
+- **Storage**: `MemoryStorage`; `FileStorage` (append-only segments, CRC-checked records, `fsync` per
+  commit by default); `CloudStorage` over S3 or Google Cloud Storage; `TieredStorage` and a cluster
+  coordinator for several writer processes. The cluster is new, with no production history, and its
+  guarantees assume that the storage devices honour `fsync`:
+  - *Power loss with the disks intact* (D50, Task 8.3f): a node accepts a root with ONE synced
+    acceptor-log entry that also holds the root's records, in every durability mode, so an acknowledged
+    commit survives power loss of any set of nodes.
+  - *Lease and ballots* (D46, Task 8.3e): every node syncs its acceptor log before replying, in every
+    durability mode, so across process crashes and power losses no term is granted twice, provided at
+    most one acceptor log is lost at a time (R48).
+  - *Loss of disks* (D54, Task 8.3g): replication is bounded. Three voters (configurable) order roots, the
+    other members are learners, and a commit's records are synced on the committer plus `f` data replicas
+    (one, with three voters), not on every member; the others keep them only in a bounded cache and fetch on
+    demand what they miss. Losing `f + 1` voters stops writes while learners keep serving reads; losing every
+    durable copy of a commit's records (`f + 1` disks) before the persister uploads them to the object store
+    loses that data, as no re-replication runs.
+  - *Fairness*: two to four contending writer processes take the lease in turn; at eight, and for hot-key
+    read-modify-write transactions, fairness is not yet guaranteed (open defect D-LIVE-2).
+  - *Reads*: a local read is a committed snapshot that may lag; a linearizable read is a barrier commit,
+    which adds a permanent version per read (see the cluster tutorial).
+- **Free-threading**: tested on CPython 3.14t (GIL disabled) and with the GIL; `parallel_scan` and
+  `parallel_filter` scale on free-threaded builds.
+- **Interop**: export to Apache Arrow and Parquet (`to_arrow`, `to_parquet`).
+
+## Installation
+
+Requirements: Python 3.12+.
+
+```bash
+pip install --pre proto_db
+```
+
+Until 2.0.0 is final, the published versions are release candidates, so `pip` needs `--pre` (or an
+explicit version such as `proto_db==2.0.0rc3`). Start with a single-node space: under heavy load the
+cluster reports outcomes as unknown, or times out, more often than it should (see the CHANGELOG's known
+limitations).
+
+Optional extras:
+
+| Extra | Installs | Needed for |
+|---|---|---|
+| `vectors` | numpy | `VectorIndex` |
+| `hnsw` | numpy, hnswlib | the HNSW vector backend (note: hnswlib re-enables the GIL on 3.14t) |
+| `parquet` | pyarrow | `to_arrow`, `to_parquet`, `scan_parquet` |
+| `s3` / `gcs` | boto3 / google-cloud-storage | `CloudStorage` with `S3ObjectStore` / `GCSObjectStore` |
+| `dev` | pytest, hypothesis, sphinx, build, twine, ... | development |
+
+`import proto_db` imports none of them and starts no thread; each optional package is imported when
+the feature that needs it is first used.
+
+## Quick start
+
+```python
+from proto_db import MemoryStorage, ObjectSpace
+
+space = ObjectSpace(MemoryStorage())
+db = space.new_database("example")
+
+# A transaction commits when the block ends without an exception.
+with db.new_transaction() as tr:
+    numbers = tr.new_list().append_last(1).append_last(2).append_last(3)
+    tr.set_root_object("numbers", numbers)
+    tr.set_root_object("ada", tr.new_object(name="Ada Lovelace", born=1815))
+
+tr = db.new_transaction()
+print(list(tr.get_root_object("numbers")))   # [1, 2, 3]
+print(tr.get_root_object("ada").name)        # Ada Lovelace
+tr.abort()
+space.close()
+```
+
+Use `FileStorage("/path/to/space")` instead of `MemoryStorage()` to persist the space;
+`durability="fsync"` (the default), `"flush"` or `"none"` selects what a returned commit survives.
+
+## Indexes and queries
+
+```python
+from proto_db import EqualityIndex, F, MemoryStorage, ObjectSpace, RangeIndex, from_collection
+
+space = ObjectSpace(MemoryStorage())
+db = space.new_database("people")
+with db.new_transaction() as tr:
+    people = tr.new_dictionary() \
+        .add_index(EqualityIndex("by_city", "city")) \
+        .add_index(RangeIndex("by_age", "age"))
+    for key, city, age in [("ann", "Paris", 25), ("bob", "Rome", 40), ("cid", "Paris", 33)]:
+        people = people.set_at(key, tr.new_object(name=key, city=city, age=age))
+    tr.set_root_object("people", people)
+
+tr = db.new_transaction()
+people = tr.get_root_object("people")
+query = from_collection(people).where((F.city == "Paris") & (F.age > 30))
+print(query.explain()["plan"])            # IndexLookup or IndexRange, chosen by cost
+print([p.name for p in query.to_list()])  # ['cid']
+tr.abort()
+space.close()
+```
+
+The LINQ front end works over plain iterables too:
+
+```python
+from proto_db import Agg, F, from_collection
+
+orders = [{"customer": 1, "total": 10.0}, {"customer": 2, "total": 7.5}, {"customer": 1, "total": 2.5}]
+totals = (from_collection(orders)
+          .group_by(F.customer, {"customer": Agg.key(), "spent": Agg.sum(F.total)})
+          .order_by(F.spent, descending=True))
+print(totals.to_list())  # [{'customer': 1, 'spent': 12.5}, {'customer': 2, 'spent': 7.5}]
+```
+
+## Concurrency
+
+```python
+import threading
+from proto_db import MemoryStorage, ObjectSpace
+
+space = ObjectSpace(MemoryStorage())
+db = space.new_database("counters")
+with db.new_transaction() as tr:
+    tr.set_root_object("hits", 0)
+
+def increment(tr):
+    tr.set_root_object("hits", tr.get_root_object("hits") + 1)
+
+# A read-modify-write that loses the race raises ProtoConflictError; run() retries it.
+threads = [threading.Thread(target=lambda: [db.run(increment, retries=100) for _ in range(50)])
+           for _ in range(4)]
+for t in threads:
+    t.start()
+for t in threads:
+    t.join()
+
+tr = db.new_transaction()
+print(tr.get_root_object("hits"))  # 200
+tr.abort()
+space.close()
+```
+
+## Examples
+
+Runnable, self-checking scripts live in `examples/` (see `examples/README.md`): getting started,
+file storage, indexed queries with `explain()`, LINQ queries, transactions under concurrency, and
+vector search. Each one runs in the test suite, as do the Python snippets of this README and every code
+block of the tutorials.
+
+## Documentation
+
+The Sphinx site is under `docs/source` (build it with `python -m sphinx -W -b html docs/source docs/build/html`):
+
+- **Tutorials**, each executed by the test suite: getting started, modelling data, indexing and querying,
+  concurrency and retries, vector search, writing a custom index, operations (durability, inspection,
+  backup), an ERP example ("as of" queries, an audit with no audit table, stable paginated reports), a CAD
+  example (reusing decoded objects across transactions) and a cluster tutorial (local against linearizable
+  reads).
+- **Design**: concepts, architecture, the concurrency and isolation model, the index framework, the query
+  planner, durability and recovery, objects across transactions, the storage format.
+- **Reference**: the API (generated from the docstrings of the `proto_db` package, whose module docstring
+  lists the public names) and the performance results by workload profile.
+- Design specification: `docs/superpowers/specs/2026-11-01-protodb-v2-design.md`; decision log:
+  `docs/decisions.md`; principles: `docs/design-principles.md`.
+
+The 0.1 pages are kept for reference in `docs/legacy/`.
+
+## Testing
+
+```bash
+pip install -e ".[dev]"
+python -m pytest            # runs tests/ (pytest's testpaths)
+```
+
+The suite has unit, contract, property-based (Hypothesis), integration, crash (fault injection
+and SIGKILL), concurrency and static-check layers. The primary target is the free-threaded build:
+`python3.14t -m pytest`, and once more with `-X gil=1`.
+
+## License
+
+MIT License. See LICENSE for details.