rubydb 0.1.4 → 0.1.5

data/docs/architecture/production-roadmap.md

@@ -1,82 +1,82 @@
-# RubyDB production roadmap
-
-## Current baseline
-
-The project has a meaningful repository skeleton and substantial design intent. It already includes modules for storage, indexing, transactions, MVCC, SQL parts, server, replication, and backup flows. However, the implementation still lacks enough end-to-end proof for durable, crash-safe operation.
-
-## Prioritized roadmap
-
-### Phase 0 — Repository stabilization
-
-- fix library loading and Ruby compatibility metadata
-- verify the project loads via `require "rubydb"`
-- establish a truthful project health baseline
-- add characterization tests for current public behavior
-
-### Phase 1 — Storage durability
-
-- define a stable on-disk page format and file format
-- add checksums, header validation, and corruption detection
-- ensure WAL durability precedes page flush semantics
-- add real restart tests and crash tests
-
-### Phase 2 — Transaction correctness
-
-- complete transaction lifecycle handling
-- prove commit/rollback semantics under restart
-- validate isolation semantics and visibility rules
-- add deadlock and lock manager tests
-
-### Phase 3 — Catalog and indexes
-
-- persist catalog metadata consistently
-- maintain indexes correctly during DML
-- verify index consistency after crash and rollback
-- document supported SQL compatibility targets honestly
-
-### Phase 4 — SQL execution and optimizer
-
-- validate end-to-end SQL execution
-- ensure binder, planner, and executor are integrated
-- add integration tests for basic DML and queries
-- implement `EXPLAIN` only with real, honest plan output
-
-### Phase 5 — Server and security
-
-- harden server lifecycle and malformed-client handling
-- enforce connection limits and timeouts
-- check authorization before execution
-- validate auth flows and access controls
-
-### Phase 6 — Backup, recovery, and replication
-
-- prove full backup + restore works from a clean environment
-- validate point-in-time or incremental recovery only when implemented correctly
-- only add replication once WAL recovery is demonstrably correct
-
-### Phase 7 — Monitoring and production readiness
-
-- add structured logging and health/readiness checks
-- expose operational metrics
-- create a final production-readiness audit with limitations and unsupported features
-
-## Hard gates before declaring production readiness
-
-RubyDB should not claim production readiness until it demonstrates all of the following:
-
-- updated data survives restart
-- crash recovery is validated with subprocess termination tests
-- transactions preserve isolation semantics
-- WAL ordering is durable and replayed correctly
-- index and table data stay consistent
-- authZ/authN is enforced over real operations
-- backup and restore work in a clean environment
-- server operations handle failures safely without whole-process crashes
-
-## Decision rule
-
-The repository must prefer correctness over feature breadth. A smaller well-tested engine is more valuable than a broad but unsafe one.
-
-## Long-term target
-
-The final target is not a fake "PostgreSQL clone" or a marketing demo. The target is a Ruby-first, SQLite-like development experience paired with durable, crash-safe relational database semantics that can be validated under real workloads.
+# RubyDB production roadmap
+
+## Current baseline
+
+The project has a meaningful repository skeleton and substantial design intent. It already includes modules for storage, indexing, transactions, MVCC, SQL parts, server, replication, and backup flows. However, the implementation still lacks enough end-to-end proof for durable, crash-safe operation.
+
+## Prioritized roadmap
+
+### Phase 0 — Repository stabilization
+
+- fix library loading and Ruby compatibility metadata
+- verify the project loads via `require "rubydb"`
+- establish a truthful project health baseline
+- add characterization tests for current public behavior
+
+### Phase 1 — Storage durability
+
+- define a stable on-disk page format and file format
+- add checksums, header validation, and corruption detection
+- ensure WAL durability precedes page flush semantics
+- add real restart tests and crash tests
+
+### Phase 2 — Transaction correctness
+
+- complete transaction lifecycle handling
+- prove commit/rollback semantics under restart
+- validate isolation semantics and visibility rules
+- add deadlock and lock manager tests
+
+### Phase 3 — Catalog and indexes
+
+- persist catalog metadata consistently
+- maintain indexes correctly during DML
+- verify index consistency after crash and rollback
+- document supported SQL compatibility targets honestly
+
+### Phase 4 — SQL execution and optimizer
+
+- validate end-to-end SQL execution
+- ensure binder, planner, and executor are integrated
+- add integration tests for basic DML and queries
+- implement `EXPLAIN` only with real, honest plan output
+
+### Phase 5 — Server and security
+
+- harden server lifecycle and malformed-client handling
+- enforce connection limits and timeouts
+- check authorization before execution
+- validate auth flows and access controls
+
+### Phase 6 — Backup, recovery, and replication
+
+- prove full backup + restore works from a clean environment
+- validate point-in-time or incremental recovery only when implemented correctly
+- only add replication once WAL recovery is demonstrably correct
+
+### Phase 7 — Monitoring and production readiness
+
+- add structured logging and health/readiness checks
+- expose operational metrics
+- create a final production-readiness audit with limitations and unsupported features
+
+## Hard gates before declaring production readiness
+
+RubyDB should not claim production readiness until it demonstrates all of the following:
+
+- updated data survives restart
+- crash recovery is validated with subprocess termination tests
+- transactions preserve isolation semantics
+- WAL ordering is durable and replayed correctly
+- index and table data stay consistent
+- authZ/authN is enforced over real operations
+- backup and restore work in a clean environment
+- server operations handle failures safely without whole-process crashes
+
+## Decision rule
+
+The repository must prefer correctness over feature breadth. A smaller well-tested engine is more valuable than a broad but unsafe one.
+
+## Long-term target
+
+The final target is not a fake "PostgreSQL clone" or a marketing demo. The target is a Ruby-first, SQLite-like development experience paired with durable, crash-safe relational database semantics that can be validated under real workloads.

data/docs/architecture/query-planner.md

@@ -1,20 +1,20 @@
-# Query planner
-
-The planner binds identifiers to catalog columns, validates expressions, and
-builds executable plans. It supports sequential/index scans, safe inner-join
-reordering, outer joins, filters, grouping, sorting, limits, set operations,
-subqueries, CTEs, and window operations in the documented SQL surface.
-
-Plans must preserve SQL null, ordering, grouping, and transaction visibility
-semantics. `EXPLAIN` reports the actual selected plan; it is not a performance
-promise. Use the benchmark and workload harnesses to establish capacity on the
-target hardware.
-
-## Plan review
-
-For every rewrite, compare the optimized and baseline results on empty,
-duplicate, null, and concurrent data. Include joins with unmatched rows,
-grouping, limits, and repeated execution with bound parameters. Measure plan
-selection, execution, lock wait, and I/O separately. A plan that is faster but
-changes cardinality or ordering is incorrect; add the minimized query as a
-regression spec before merging.
+# Query planner
+
+The planner binds identifiers to catalog columns, validates expressions, and
+builds executable plans. It supports sequential/index scans, safe inner-join
+reordering, outer joins, filters, grouping, sorting, limits, set operations,
+subqueries, CTEs, and window operations in the documented SQL surface.
+
+Plans must preserve SQL null, ordering, grouping, and transaction visibility
+semantics. `EXPLAIN` reports the actual selected plan; it is not a performance
+promise. Use the benchmark and workload harnesses to establish capacity on the
+target hardware.
+
+## Plan review
+
+For every rewrite, compare the optimized and baseline results on empty,
+duplicate, null, and concurrent data. Include joins with unmatched rows,
+grouping, limits, and repeated execution with bound parameters. Measure plan
+selection, execution, lock wait, and I/O separately. A plan that is faster but
+changes cardinality or ordering is incorrect; add the minimized query as a
+regression spec before merging.

data/docs/architecture/recovery.md

@@ -1,18 +1,18 @@
-# Recovery architecture
-
-Recovery validates the database format, page headers, checksums, metadata, and
-WAL before making state available. Valid committed WAL records may be replayed;
-malformed or ambiguous records fail closed and require operator investigation.
-
-Recovery procedures work from a preserved source or verified backup. Restore to
-a new directory, run dry-run and checksum checks, reopen the restored engine,
-compare schema/row evidence, and only then switch application traffic. See the
-[disaster recovery](../operations/disaster-recovery.md) procedure.
-
-## Failure handling
-
-Recovery is allowed to replay only validated, committed records. Truncated
-frames, invalid checksums, impossible LSNs, and incomplete metadata must stop
-startup with an actionable error. Preserve the original directory and WAL;
-repair or compaction experiments belong on a copy. The [debugging playbook](../debugging.md)
-lists the evidence to collect.
+# Recovery architecture
+
+Recovery validates the database format, page headers, checksums, metadata, and
+WAL before making state available. Valid committed WAL records may be replayed;
+malformed or ambiguous records fail closed and require operator investigation.
+
+Recovery procedures work from a preserved source or verified backup. Restore to
+a new directory, run dry-run and checksum checks, reopen the restored engine,
+compare schema/row evidence, and only then switch application traffic. See the
+[disaster recovery](../operations/disaster-recovery.md) procedure.
+
+## Failure handling
+
+Recovery is allowed to replay only validated, committed records. Truncated
+frames, invalid checksums, impossible LSNs, and incomplete metadata must stop
+startup with an actionable error. Preserve the original directory and WAL;
+repair or compaction experiments belong on a copy. The [debugging playbook](../debugging.md)
+lists the evidence to collect.

data/docs/architecture/sql-engine.md

@@ -1,12 +1,12 @@
-# SQL engine
-
-RubyDB implements a documented RubyDB SQL subset with a tested common
-SQLite-style application profile. Covered features include CRUD, constraints,
-joins, grouping/aggregates, transactions/savepoints, CTEs, subqueries, set
-operations, upserts, windows, indexes, views, and maintenance statements listed
-in [SQL compatibility](../sql/compatibility.md).
-
-This is not complete PostgreSQL, MySQL, or SQLite dialect/file-format
-compatibility. Unsupported syntax must fail explicitly. Applications migrating
-from another engine must run their own schema, query, migration, and error
-behavior suite.
+# SQL engine
+
+RubyDB implements a documented RubyDB SQL subset with a tested common
+SQLite-style application profile. Covered features include CRUD, constraints,
+joins, grouping/aggregates, transactions/savepoints, CTEs, subqueries, set
+operations, upserts, windows, indexes, views, and maintenance statements listed
+in [SQL compatibility](../sql/compatibility.md).
+
+This is not complete PostgreSQL, MySQL, or SQLite dialect/file-format
+compatibility. Unsupported syntax must fail explicitly. Applications migrating
+from another engine must run their own schema, query, migration, and error
+behavior suite.

data/docs/architecture/storage-engine.md

@@ -1,14 +1,14 @@
-# Storage engine
-
-The storage engine owns the database path, page manager, buffer pool, catalog,
-indexes, WAL, and recovery lifecycle. It serializes durable changes and exposes
-Ruby-native operations used by the SQL executor and adapters.
-
-Embedded ownership is exclusive and enforced with an operating-system lock.
-Multiple processes must use the server. On failure, callers should preserve the
-original directory and recover into a new destination; do not delete WAL or
-overwrite the source during investigation.
-
-Storage changes require reopen, crash, fault-injection, corruption, compaction,
-and backup/restore coverage. The [lessons learned](../lessons-learned.md) page
-explains why these are separate guarantees.
+# Storage engine
+
+The storage engine owns the database path, page manager, buffer pool, catalog,
+indexes, WAL, and recovery lifecycle. It serializes durable changes and exposes
+Ruby-native operations used by the SQL executor and adapters.
+
+Embedded ownership is exclusive and enforced with an operating-system lock.
+Multiple processes must use the server. On failure, callers should preserve the
+original directory and recover into a new destination; do not delete WAL or
+overwrite the source during investigation.
+
+Storage changes require reopen, crash, fault-injection, corruption, compaction,
+and backup/restore coverage. The [lessons learned](../lessons-learned.md) page
+explains why these are separate guarantees.

data/docs/architecture/transactions.md

@@ -1,10 +1,10 @@
-# Transactions
-
-Transactions group mutations into a commit or rollback boundary. WAL commit and
-flush ordering precede durable acknowledgement. MVCC determines visibility,
-while lock management protects conflicting writes and resolves wait-for cycles.
-
-Supported controls include `BEGIN`, `COMMIT`, `ROLLBACK`, savepoints, and
-rollback to savepoint. A deadlock victim is rolled back and must retry the whole
-application unit as appropriate. Test isolation and recovery together; a green
-single-thread transaction test does not prove concurrent correctness.
+# Transactions
+
+Transactions group mutations into a commit or rollback boundary. WAL commit and
+flush ordering precede durable acknowledgement. MVCC determines visibility,
+while lock management protects conflicting writes and resolves wait-for cycles.
+
+Supported controls include `BEGIN`, `COMMIT`, `ROLLBACK`, savepoints, and
+rollback to savepoint. A deadlock victim is rolled back and must retry the whole
+application unit as appropriate. Test isolation and recovery together; a green
+single-thread transaction test does not prove concurrent correctness.

data/docs/architecture/wal.md

@@ -1,28 +1,28 @@
-# RubyDB WAL format
-
-Each WAL segment is a sequence of independently framed records:
-
-```text
-4-byte unsigned big-endian payload length
-16-byte ASCII SHA-256 checksum prefix
-UTF-8 JSON payload
-```
-
-The checksum covers the JSON payload only. A record's LSN offset is the byte
-offset of its frame header within the segment, so readers can advance exactly
-one frame at a time. If the final frame is incomplete, recovery stops at that
-tail and retains all complete preceding records.
-
-WAL data is flushed and fsynced before the storage engine flushes data pages
-when synchronous WAL mode is enabled (the default). A successful `write` is
-therefore durable only after the WAL sync completes; callers must still use
-the engine's transaction/close APIs to coordinate higher-level durability.
-
-## Transaction recovery records
-
-Mutation records include the transaction ID. UPDATE records include the
-previous column values, and DELETE records include the previous row values and
-columns. Recovery replays mutations for transactions with a durable COMMIT
-record. Transactions that have a BEGIN but no COMMIT are undone in reverse WAL
-order. Recovery uses the existing row version for DELETE undo and does not
-create a replacement row with a new identifier.
+# RubyDB WAL format
+
+Each WAL segment is a sequence of independently framed records:
+
+```text
+4-byte unsigned big-endian payload length
+16-byte ASCII SHA-256 checksum prefix
+UTF-8 JSON payload
+```
+
+The checksum covers the JSON payload only. A record's LSN offset is the byte
+offset of its frame header within the segment, so readers can advance exactly
+one frame at a time. If the final frame is incomplete, recovery stops at that
+tail and retains all complete preceding records.
+
+WAL data is flushed and fsynced before the storage engine flushes data pages
+when synchronous WAL mode is enabled (the default). A successful `write` is
+therefore durable only after the WAL sync completes; callers must still use
+the engine's transaction/close APIs to coordinate higher-level durability.
+
+## Transaction recovery records
+
+Mutation records include the transaction ID. UPDATE records include the
+previous column values, and DELETE records include the previous row values and
+columns. Recovery replays mutations for transactions with a durable COMMIT
+record. Transactions that have a BEGIN but no COMMIT are undone in reverse WAL
+order. Recovery uses the existing row version for DELETE undo and does not
+create a replacement row with a new identifier.

data/docs/cli-cheatsheet.md

@@ -1,98 +1,98 @@
-# RubyDB CLI cheat sheet
-
-Replace paths and names before running. Put global options before the command.
-
-## Global
-
-```sh
-rubydb --help
-rubydb --version
-rubydb --config config/production.yml --env production --no-color status --json
-```
-
-## Database lifecycle
-
-```sh
-rubydb init --name app --dir data
-rubydb create --database data/app.rdb
-rubydb status --json
-rubydb doctor --quick --json
-rubydb inspect --database data/app.rdb --stats --wal
-rubydb drop --database data/app.rdb              # prompts
-rubydb drop --database data/app.rdb --force      # destructive
-```
-
-## Server
-
-```sh
-rubydb --env production start --host 127.0.0.1 --port 7432 \
-  --data-dir /var/lib/rubydb/data --log-dir /var/log/rubydb \
-  --pid-file /run/rubydb.pid
-rubydb stop --pid-file /run/rubydb.pid
-rubydb restart --pid-file /run/rubydb.pid
-```
-
-## SQL shell
-
-```sh
-rubydb shell --database app --table
-rubydb shell --database app --json
-```
-
-Inside shell: `.help` `.tables` `.table users` `.schema users` `.status`
-`.begin` `.commit` `.rollback` `.explain SELECT ...` `.exit`.
-
-## Migrations
-
-```sh
-rubydb migrate --database data/app.rdb --path db/migrate --dry-run
-rubydb migrate --database data/app.rdb --path db/migrate
-rubydb migrate --database data/app.rdb --path db/migrate --down --steps 1
-```
-
-## Backups and restore
-
-```sh
-rubydb backup --database data/app.rdb --dir backups --type full --compress
-rubydb backup --database data/app.rdb --dir backups --type incremental
-rubydb backup --database data/app.rdb --dir backups --type differential
-rubydb restore --dir backups --latest --dry-run
-rubydb restore --dir backups --latest
-rubydb restore --dir backups --backup NAME --point-in-time 2026-09-09T12:00:00Z
-```
-
-## Snapshots and branches
-
-```sh
-rubydb snapshot --database data/app.rdb --dir snapshots --name before-change
-rubydb snapshot --database data/app.rdb --dir snapshots --list
-rubydb branch --database data/app.rdb --branch-dir branches --list
-rubydb branch --database data/app.rdb --branch-dir branches --create feature --from main
-rubydb checkout --database data/app.rdb --branch-dir branches feature
-rubydb diff --database data/app.rdb --branch-dir branches main feature --summary
-rubydb merge --database data/app.rdb --branch-dir branches feature --into main
-```
-
-## Maintenance and release checks
-
-```sh
-rubydb vacuum --database data/app.rdb --dry-run
-rubydb vacuum --database data/app.rdb --full --analyze
-ruby scripts/durability_drill
-ruby scripts/replication_failover_drill
-ruby scripts/replication_network_failover_drill
-ruby scripts/release
-```
-
-## Safety rules
-
-- Back up and verify before `migrate`, `merge`, `checkout`, `vacuum --full`, or
-  any restore.
-- Treat `drop --force`, `create --force`, restore `--force`, and process `--force`
-  stop as destructive operations.
-- Preserve database, WAL, lock, metadata, logs, and configuration together.
-- Use one embedded owner; use server/client mode for multiple processes.
-- Never put passwords, tokens, private keys, or API keys in commands committed
-  to source control or in captured output.
-- A successful CLI exit is not a universal production certification; retain the
-  workload, restore, failover, CI, and security evidence.
+# RubyDB CLI cheat sheet
+
+Replace paths and names before running. Put global options before the command.
+
+## Global
+
+```sh
+rubydb --help
+rubydb --version
+rubydb --config config/production.yml --env production --no-color status --json
+```
+
+## Database lifecycle
+
+```sh
+rubydb init --name app --dir data
+rubydb create --database data/app.rdb
+rubydb status --json
+rubydb doctor --quick --json
+rubydb inspect --database data/app.rdb --stats --wal
+rubydb drop --database data/app.rdb              # prompts
+rubydb drop --database data/app.rdb --force      # destructive
+```
+
+## Server
+
+```sh
+rubydb --env production start --host 127.0.0.1 --port 7432 \
+  --data-dir /var/lib/rubydb/data --log-dir /var/log/rubydb \
+  --pid-file /run/rubydb.pid
+rubydb stop --pid-file /run/rubydb.pid
+rubydb restart --pid-file /run/rubydb.pid
+```
+
+## SQL shell
+
+```sh
+rubydb shell --database app --table
+rubydb shell --database app --json
+```
+
+Inside shell: `.help` `.tables` `.table users` `.schema users` `.status`
+`.begin` `.commit` `.rollback` `.explain SELECT ...` `.exit`.
+
+## Migrations
+
+```sh
+rubydb migrate --database data/app.rdb --path db/migrate --dry-run
+rubydb migrate --database data/app.rdb --path db/migrate
+rubydb migrate --database data/app.rdb --path db/migrate --down --steps 1
+```
+
+## Backups and restore
+
+```sh
+rubydb backup --database data/app.rdb --dir backups --type full --compress
+rubydb backup --database data/app.rdb --dir backups --type incremental
+rubydb backup --database data/app.rdb --dir backups --type differential
+rubydb restore --dir backups --latest --dry-run
+rubydb restore --dir backups --latest
+rubydb restore --dir backups --backup NAME --point-in-time 2026-09-09T12:00:00Z
+```
+
+## Snapshots and branches
+
+```sh
+rubydb snapshot --database data/app.rdb --dir snapshots --name before-change
+rubydb snapshot --database data/app.rdb --dir snapshots --list
+rubydb branch --database data/app.rdb --branch-dir branches --list
+rubydb branch --database data/app.rdb --branch-dir branches --create feature --from main
+rubydb checkout --database data/app.rdb --branch-dir branches feature
+rubydb diff --database data/app.rdb --branch-dir branches main feature --summary
+rubydb merge --database data/app.rdb --branch-dir branches feature --into main
+```
+
+## Maintenance and release checks
+
+```sh
+rubydb vacuum --database data/app.rdb --dry-run
+rubydb vacuum --database data/app.rdb --full --analyze
+ruby scripts/durability_drill
+ruby scripts/replication_failover_drill
+ruby scripts/replication_network_failover_drill
+ruby scripts/release
+```
+
+## Safety rules
+
+- Back up and verify before `migrate`, `merge`, `checkout`, `vacuum --full`, or
+  any restore.
+- Treat `drop --force`, `create --force`, restore `--force`, and process `--force`
+  stop as destructive operations.
+- Preserve database, WAL, lock, metadata, logs, and configuration together.
+- Use one embedded owner; use server/client mode for multiple processes.
+- Never put passwords, tokens, private keys, or API keys in commands committed
+  to source control or in captured output.
+- A successful CLI exit is not a universal production certification; retain the
+  workload, restore, failover, CI, and security evidence.