rubydb 0.1.0

data/docs/getting-started/first-query.md

@@ -0,0 +1,13 @@
+# First query
+
+RubyDB executes the documented SQL subset through its lexer, parser, planner,
+and executor:
+
+```ruby
+rows = engine.execute("SELECT id, email FROM accounts WHERE id = 1 ORDER BY id LIMIT 10")
+puts rows.inspect
+```
+
+Use the Rails connection or client binding APIs for external values. Do not
+interpolate untrusted input into SQL. See [SQL compatibility](../sql/compatibility.md)
+for supported statements and explicit boundaries.

data/docs/getting-started/installation.md

@@ -0,0 +1,19 @@
+# Installation
+
+RubyDB requires Ruby 3.3 or newer in the current support policy.
+
+```sh
+gem install rubydb --pre
+```
+
+From a checkout:
+
+```sh
+bundle install
+bundle exec rspec
+```
+
+For Rails, install the adapter from `adapters/activerecord` and use the
+configuration documented in `docs/rails/database-yml.md`. Pin RubyDB and Ruby
+versions in production and test the exact gem on a restored staging backup
+before deployment.

data/docs/getting-started/local-to-production.md

@@ -0,0 +1,300 @@
+# From local development to production
+
+This guide is for a beginner starting with a Ruby or Rails application. It
+shows two valid deployment choices:
+
+1. develop with RubyDB locally and deploy with PostgreSQL; or
+2. develop with RubyDB locally and keep RubyDB in production.
+
+The application code can be similar in both cases, but the database is not
+interchangeable by copying files. Choose the production database before
+launch, run the application’s migrations and query tests against it, and make
+backups before importing important data.
+
+## Which path should I choose?
+
+| Goal | Local database | Production database | Recommended when |
+| --- | --- | --- | --- |
+| Learn/build quickly | RubyDB embedded | PostgreSQL | You want the broadest hosting and Rails ecosystem |
+| Keep one Ruby-native database | RubyDB embedded | RubyDB server | Your SQL/workload fits RubyDB and you can operate the server |
+| Prototype only | RubyDB embedded | RubyDB embedded | One process owns the path and data is non-critical |
+
+For a public or business-critical application, PostgreSQL is the safer default
+until RubyDB has been validated against your complete workload, backup/restore
+process, concurrency, and failover requirements. RubyDB is not a complete
+PostgreSQL or SQLite replacement.
+
+## Part 1: local Rails development with RubyDB
+
+### 1. Add the gems
+
+In the Rails application `Gemfile`:
+
+```ruby
+gem "rubydb"
+gem "rubydb-activerecord"
+gem "pg" # Keep this if PostgreSQL is a possible production target.
+```
+
+Run:
+
+```sh
+bundle install
+```
+
+### 2. Configure development and test
+
+Use an embedded path for local development. Only the Rails process should own
+each path:
+
+```yaml
+default: &default
+  adapter: rubydb
+  embedded: true
+  database: <%= Rails.root.join("tmp/rubydb_development.rdb") %>
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS", "5") %>
+
+development:
+  <<: *default
+
+test:
+  <<: *default
+  database: <%= Rails.root.join("tmp/rubydb_test.rdb") %>
+```
+
+Use separate development and test paths. Do not put the database under a
+source-controlled directory. Do not let a web server, job worker, and console
+open the same embedded path at the same time.
+
+### 3. Create the schema and run the app
+
+```sh
+bin/rails db:create
+bin/rails db:migrate
+bin/rails db:seed       # only if your seed data is safe to recreate
+bin/rails test
+bin/rails server
+```
+
+Exercise real application flows: sign-up, login, CRUD, joins, eager loading,
+background jobs, file metadata, transactions, and error handling. Do not test
+only that the server boots.
+
+### 4. Use RubyDB in a regular Ruby app
+
+Local single-process code can use the embedded engine:
+
+```ruby
+require "rubydb"
+
+engine = RubyDB::Storage::Engine.new("tmp/app.rdb")
+begin
+  engine.execute("CREATE TABLE IF NOT EXISTS events (id INTEGER PRIMARY KEY, name TEXT)")
+  engine.execute("INSERT INTO events (name) VALUES (?)", ["boot"])
+  p engine.execute("SELECT * FROM events")
+ensure
+  engine.close
+end
+```
+
+For multiple processes, use the client/server connection described below.
+
+## Part 2A: deploy the same app on PostgreSQL
+
+This path keeps RubyDB for local development and uses PostgreSQL in production.
+It is a database migration, not a file copy.
+
+### 1. Configure the production Rails block
+
+```yaml
+production:
+  adapter: postgresql
+  url: <%= ENV.fetch("DATABASE_URL") %>
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS", "5") %>
+```
+
+The hosting provider supplies a PostgreSQL URL such as:
+
+```text
+postgresql://app_user:password@postgres.example.com:5432/my_app
+```
+
+Keep `DATABASE_URL` in the provider’s secret settings. Never commit it.
+
+### 2. Review the schema and SQL
+
+Before switching, review RubyDB-specific behavior:
+
+* integer primary-key/autoincrement behavior;
+* `BOOLEAN`, `JSON`, `UUID`, dates, timestamps, and decimals;
+* indexes, foreign keys, check constraints, and defaults;
+* `NULL`, ordering, grouping, joins, upserts, and functions;
+* migrations that alter populated tables; and
+* raw SQL, extensions, pragmas, and database-specific functions.
+
+Run `db:schema:dump` and inspect the generated schema. Do not assume every
+RubyDB schema statement is valid PostgreSQL syntax. Fix migrations or use
+adapter-specific migrations deliberately and document the difference.
+
+### 3. Transfer data safely
+
+RubyDB `.rdb` files cannot be opened by PostgreSQL. A safe small-dataset flow is:
+
+1. stop writes to the RubyDB application;
+2. create a verified RubyDB backup;
+3. create the PostgreSQL database and run reviewed migrations;
+4. export each table in a deterministic order, including parent tables first;
+5. transform types and IDs explicitly;
+6. import into PostgreSQL using bound/escaped data or PostgreSQL `COPY`;
+7. compare row counts, checksums, foreign keys, indexes, and business totals;
+8. run the application test and smoke suite against PostgreSQL; and
+9. switch traffic only after the verification report is approved.
+
+For large or live data, use a purpose-built migration process with checkpoints,
+retries, idempotency, and a cutover plan. Do not write a one-off script that
+silently skips a row or converts an unknown type to text.
+
+### 4. Test production configuration before cutover
+
+```sh
+RAILS_ENV=production DATABASE_URL='postgresql://...' bin/rails db:migrate
+RAILS_ENV=production DATABASE_URL='postgresql://...' bin/rails db:seed
+RAILS_ENV=production DATABASE_URL='postgresql://...' bin/rails test
+```
+
+Run this against a restored staging database, not the only production copy.
+Keep the RubyDB backup until PostgreSQL row counts and application behavior
+have been accepted.
+
+## Part 2B: deploy RubyDB in production
+
+This path runs one managed RubyDB server and connects Rails/Ruby processes to it
+over the RubyDB protocol. The application never opens the server’s data file.
+
+### 1. Provision a database host
+
+Install a pinned RubyDB release, create a dedicated service account, and attach
+persistent storage. The data directory must survive process restarts and
+deploys. Keep backups on a separate system or failure domain.
+
+```sh
+gem install rubydb -v 0.1.0
+install -d -o rubydb -g rubydb -m 0700 /var/lib/rubydb/data
+install -d -o rubydb -g rubydb -m 0750 /var/log/rubydb
+```
+
+Use the complete setup in the [production operations guide](../operations/production-guide.md)
+for service supervision, TLS, authentication, resource limits, backups, and
+monitoring.
+
+### 2. Start the RubyDB server
+
+Use a reviewed production configuration with WAL, authentication, TLS, and
+bounded resources:
+
+```sh
+rubydb --config /etc/rubydb/production.yml --env production start
+```
+
+Keep port `7432` on a private network. Verify the server before connecting the
+application:
+
+```sh
+rubydb --config /etc/rubydb/production.yml --env production status --json
+rubydb --config /etc/rubydb/production.yml --env production doctor --json
+```
+
+### 3. Put one RubyDB URL in the application environment
+
+RubyDB supports its own URL format. `rubydbs` enables TLS:
+
+```text
+RUBYDB_URL=rubydbs://app_user:URL_ENCODED_PASSWORD@db.internal.example:7432/my_app?verify_peer=true&ca_file=%2Fetc%2Frubydb%2Fca.crt
+```
+
+Special characters in usernames and passwords must be percent-encoded. Store
+the complete value in a secret manager. Do not print it during deploys.
+
+### 4. Configure Rails
+
+```yaml
+production:
+  adapter: rubydb
+  embedded: false
+  url: <%= ENV.fetch("RUBYDB_URL") %>
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS", "5") %>
+```
+
+Run the migration once from a controlled release job:
+
+```sh
+RAILS_ENV=production bundle exec rails db:migrate
+RAILS_ENV=production bundle exec rails runner 'puts User.count'
+```
+
+Do not run migrations simultaneously from every web process. Confirm that the
+Rails pool fits below the RubyDB server connection limit with room for workers,
+monitoring, and administration.
+
+### 5. Configure regular Ruby
+
+```ruby
+require "rubydb"
+
+client = RubyDB::Client::Client.new(url: ENV.fetch("RUBYDB_URL"))
+begin
+  result = client.query("SELECT 1")
+  puts result.to_hash
+ensure
+  client.disconnect
+end
+```
+
+Use idempotency keys for retried writes. A timeout or broken connection does
+not prove that a write was rolled back; check the operation outcome before
+retrying a non-idempotent request.
+
+### 6. Perform a production smoke test
+
+Before routing users:
+
+```sh
+RAILS_ENV=production bundle exec rails db:migrate:status
+RAILS_ENV=production bundle exec rails runner 'puts User.limit(1).to_a.inspect'
+rubydb --config /etc/rubydb/production.yml --env production doctor --json
+```
+
+Then test one authenticated read, one create/update/delete transaction, one
+background job, one backup, and one restore into a separate directory. Confirm
+logs, metrics, disk alerts, WAL/checkpoint state, and rollback ownership.
+
+## Render-style cloud deployment
+
+On a platform such as Render, use a private service for the RubyDB server and a
+web service for Rails. Attach a persistent disk to the RubyDB service and set
+`RUBYDB_URL` on the Rails service. Keep both services in the same region and
+use the private hostname. A platform’s default filesystem is often ephemeral;
+verify the provider’s storage behavior before using it for database files.
+
+A single persistent disk normally means one RubyDB primary instance. Do not
+assume that increasing the web-service instance count creates database
+high availability. Maintain external backups and validate failover separately.
+
+## Final beginner checklist
+
+Before calling the app production-ready, confirm:
+
+* the production database choice is written down;
+* all application SQL and migrations pass against that database;
+* credentials are environment/secret-manager values, not source files;
+* RubyDB uses server mode when more than one process connects;
+* TLS, authentication, private networking, and least privilege are enabled;
+* a verified backup can be restored on another directory or host;
+* monitoring and alerts are visible to an on-call owner;
+* connection, lock, request, and shutdown timeouts are bounded; and
+* rollback, data migration, and incident procedures have been rehearsed.
+
+For deeper details, continue with [Rails compatibility](../rails/compatibility-guide.md),
+[production operations](../operations/production-guide.md),
+[troubleshooting](../troubleshooting.md), and [debugging](../debugging.md).
+

data/docs/getting-started/quickstart.md

@@ -0,0 +1,17 @@
+# Quick start
+
+Create a local database in one owning Ruby process:
+
+```ruby
+require "rubydb"
+
+engine = RubyDB::Storage::Engine.new("tmp/quickstart.rdb")
+engine.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT NOT NULL)")
+engine.execute("INSERT INTO users (id, name) VALUES (1, 'Aldane')")
+p engine.execute("SELECT * FROM users")
+engine.close
+```
+
+For a full walkthrough see [first database](first-database.md) and
+[first query](first-query.md). For multiple processes, use the server/client
+topology and do not share an embedded path.

data/docs/getting-started/rails.md

@@ -0,0 +1,16 @@
+# Rails quick start
+
+The runnable application under `examples/rails_app` demonstrates a real Rails
+migration, model, query, and server:
+
+```sh
+cd examples/rails_app
+bundle install
+bundle exec ruby bin/rails db:migrate
+bundle exec ruby bin/rails server -b 127.0.0.1 -p 3001
+```
+
+Use the ActiveRecord adapter only within its tested feature surface. For a
+production Rails deployment, use server mode for multiple processes, protect
+TLS/authentication secrets, back up before migrations, and run the exact Rails
+matrix and populated-schema checks described in `docs/rails/production.md`.

data/docs/hardening_backlog.md

@@ -0,0 +1,93 @@
+# Remaining production work
+
+Completed checkpoints: exclusive embedded engine ownership, duplicate-open
+rejection, release after process exit, initialization failure cleanup, fail-closed
+metadata and recovery startup, atomic metadata replacement, and deterministic
+background-maintenance shutdown.
+
+The following work remains open; passing the regression suite does not certify
+these capabilities.
+
+1. Server concurrency: cancellation and sustained load with latency percentiles.
+   Client deadlines are checked before execution and propagated into the query
+   executor, which checks during long read phases. Engine transaction state is
+   now scoped per client connection thread, and concurrent commit/rollback
+   behavior is covered. Multi-process durability is covered; wire-level
+   cancellation is now request-scoped and cooperative: a client can send
+   a cancel frame while the connection reader remains active, and the server
+   acknowledges it only for the active request. Mixed transaction reference-log
+   and restart validation remain open.
+2. Persistence: fault-test disk-full and interrupted checkpoints/schema changes;
+   index metadata load and write errors now fail visibly, and failed schema
+   publications roll back in-memory state. Metadata and index catalogs are
+   published through unique temporary files with flush/fsync/atomic rename.
+   Commit acknowledgements now expose durable versus uncertain WAL state and
+   recovery-required post-WAL flush failures. Storage accepts an explicit
+   `io_fault_injector` hook for page writes, file extension/truncation, and
+   sync operations; those paths now verify typed failures and descriptor
+   cleanup. Compaction now reads the engine's actual record-header layout,
+   observes dirty buffer-pool pages, and is covered through reopen validation.
+   Real filesystem quota and power-loss tests remain environment work.
+3. SQL correctness: ambiguous identifiers and broader dialect-specific edge
+   cases remain open. Boolean false values, `IS NULL`, NULL comparison behavior, and
+   outer-join NULL extension now have regression coverage. Non-recursive CTEs,
+   subqueries, set operations, targeted `ON CONFLICT DO UPDATE`, and
+   ranking/partition window functions, explicit `ROWS` window frames, bounded
+   recursive CTEs, dependency-aware inner-join reordering, and targetless
+   conflict updates using primary/unique definitions are implemented; multi-row
+   `VALUES` sources are now atomic when executed outside a caller transaction.
+   Broader dialect upsert forms and statistics-driven plan costing remain open.
+4. Replication: synchronized acknowledgements and promotion with an explicit
+   recovery point. Engine commits now package all committed row changes into a
+   single replication envelope after the local WAL commit point. Primary
+   connections bootstrap the catalog before row replay, including empty replicas. TCP input now
+   uses bounded newline framing and replay positions are fsynced before ack;
+   replica engine mutation entry points are read-only except for internal replay.
+   Promotion now rejects any received/replayed LSN gap and can require a
+   caller-supplied recovery point. Active primary engine mutations validate the
+   fencing lease before writing. Replication shutdown now closes established
+   replica sockets, and a reconnect/catch-up partition test verifies that the
+   replica resumes from the durable log. A process-level failover drill now
+   kills and replaces an independent primary, advances the fencing epoch from
+   another process, rejects the stale writer, and verifies replica catch-up.
+   A reproducible TCP fault-proxy drill now exercises live partition, healing,
+   catch-up, fencing, and promotion while the primary remains alive. True
+   multi-host partitions, independent fencing infrastructure, and split-brain
+   recovery still require deployment validation before adding automatic election.
+5. Rails: populated migration round trips, eager loading, nested associations,
+   and live adapter coverage are present. CI now exercises Rails 7.1, 7.2, and
+   8.0 against the adapter; connection-pool behavior and each version's
+   compatibility result still require hosted-run evidence. Migration
+   tracking now uses stable content checksums and fails closed for changed or
+   missing applied migrations. Native and ActiveRecord schema dumps now
+   round-trip automatic/custom primary-key modes, defaults, and indexes through
+   live engines.
+6. Operations: backup manifest writes now use durable atomic publication, live
+   engines flush WAL/storage before physical backup, and a scheduled restore
+   drill reopens restored files. Upgrade tests, measured resource limits,
+   alerting, and full security review remain open. Replication peers now
+   support constant-time shared-token authentication when configured; TLS and
+   credential rotation procedures still require deployment validation.
+
+7. Release engineering: cross-platform Ruby 3.3/3.4 CI, Rails 7.1/7.2/8.0
+   adapter jobs, and a deterministic bounded fuzz safety workflow are wired
+   into GitHub Actions. CI enforces 25% line and 20% branch coverage (the
+   current audit measured 62.0% line and 32.84% branch). Release provenance
+   signing is enabled, RubyGems gem-level signing requires protected
+   key/certificate paths, tag/changelog preflight fails closed, dependency
+   audit runs weekly, and tag releases can publish generated GitHub release
+   notes. A maintainer must still provision the RubyGems signing secrets and
+   review generated notes before publication.
+
+Deployment tests must record the commit, platform, workload and measured
+results. Keep untested features marked as unvalidated.
+
+## Latest documentation checkpoint
+
+The current repository also ships a tested common SQLite-style compatibility
+profile, a durability release gate, Prometheus alert rules, a multi-node Docker
+topology manifest, a production runbook, and a release checklist. The lessons
+from these hardening changes are collected in
+[`docs/lessons-learned.md`](lessons-learned.md). These artifacts improve
+repeatability but do not replace hosted multi-host, physical-filesystem, or
+independent security validation.

data/docs/lessons-learned.md

@@ -0,0 +1,112 @@
+# RubyDB lessons learned
+
+This page records the engineering lessons from hardening RubyDB. Each lesson is
+paired with the rule it creates and the validation that protects it. New
+features should update this page when they change an operational guarantee.
+
+## Correctness before breadth
+
+An implemented parser branch, adapter method, or replication class is not a
+production feature until it is exercised end to end. A narrow, explicit
+compatibility contract is safer than accepting syntax with incomplete
+semantics.
+
+Rule: document supported behavior and fail closed for unsupported behavior.
+Validation: integration specs and the SQL compatibility documents must agree.
+
+## Durability is a sequence, not a boolean
+
+“Write succeeded” and “data is durable” are different outcomes. WAL ordering,
+flush errors, atomic metadata publication, checksums, recovery, and restore must
+be tested together. A successful in-memory mutation cannot hide a failed
+publication or sync.
+
+Rule: never report a durable acknowledgement before the required WAL and file
+operations succeed. Preserve uncertain state for recovery instead of guessing.
+Validation: filesystem fault injection, crash recovery, compaction/reopen, and
+backup/restore drills.
+
+## Ownership must be explicit
+
+An embedded file cannot safely be opened by multiple independent application
+processes merely because each process has a mutex. The operating-system lock and
+the server boundary are part of the data-safety model.
+
+Rule: one embedded owner; multiple processes use the server/client topology.
+Validation: duplicate-open rejection, process workloads, and connection-limit
+tests.
+
+## Replication needs fencing before election
+
+Replica catch-up alone does not prevent two writers. Promotion requires a
+known recovery point, durable fencing, stale-writer rejection, and operator
+evidence. Automatic election without an independent fencing authority can turn
+a network partition into split-brain writes.
+
+Rule: keep automatic election disabled until multi-host fencing is independently
+validated in the deployment environment.
+Validation: process replacement, TCP partition/healing, stale-primary rejection,
+and a real multi-host drill.
+
+## Cancellation must reach the wire and the executor
+
+A client-side timeout that only stops waiting does not stop database work. The
+server must keep reading control frames, mark the active request cancelled, and
+the executor must check cooperatively during long operations.
+
+Rule: test cancellation as an in-flight protocol event, not as a mocked timeout.
+Validation: the production soak sends a real cancel frame and checks the server
+request lifecycle.
+
+## Deadlock recovery belongs to transaction ownership
+
+Detecting a cycle is insufficient if the chosen victim is not rolled back and
+its locks are not released. The lock manager and transaction manager must share
+the same victim identity and lifecycle.
+
+Rule: every resolved deadlock has a victim, rollback, lock release, and a
+retryable error for the victim.
+Validation: the two-transaction cycle specs and the production soak.
+
+## Adapters are compatibility products
+
+ActiveRecord compatibility includes schema introspection, quoted identifiers,
+binds, associations, eager loading, migrations, schema dumps, connection pools,
+and error behavior. Passing one CRUD example is not Rails compatibility.
+
+Rule: test the adapter against every supported Rails version and a populated
+schema, not only an empty database.
+Validation: the Rails matrix and adapter integration suite for Rails 7.1, 7.2,
+and 8.0.
+
+## Operations are part of the implementation
+
+Backups, restore destinations, monitoring thresholds, certificate rotation,
+secret storage, upgrade rollback, incident preservation, and release evidence
+are not afterthoughts. Operators need safe commands and clear stop conditions.
+
+Rule: every destructive or irreversible operation must have a dry run, a backup
+boundary, and a documented recovery path.
+Validation: the production runbook, restore drill, alert rules, and release
+preflight.
+
+## Evidence must be reproducible
+
+Test counts without the commit, runtime, platform, workload, seed, and result
+artifact are difficult to trust or reproduce.
+
+Rule: archive machine-readable workload output and record commit, Ruby/Rails
+versions, OS, resources, seed, latency percentiles, recovery time, and RPO/RTO.
+Validation: CI artifacts, bounded fuzz seeds, benchmark output, and release
+checklists.
+
+## Security claims require a separate review
+
+Passing authentication and TLS tests does not review deployment identity,
+filesystem permissions, secret lifecycle, dependency exposure, or incident
+response.
+
+Rule: treat the repository security suite as a baseline and require an
+independent review before sensitive or regulated workloads.
+Validation: dependency audit, CodeQL, TLS/auth tests, deployment review, and
+credential/certificate rotation drills.

data/docs/operations/backups.md

@@ -0,0 +1,33 @@
+# Backup policy
+
+Backups are only useful after they have been verified and restored. Keep the
+database file, WAL chain, backup manifest, and checksum together. Store a copy
+outside the database host and encrypt it with the organization’s approved
+archive tooling.
+
+## Create and verify
+
+Stop writes for a maintenance backup when possible, then run:
+
+```sh
+rubydb backup --dir /var/lib/rubydb/backups
+rubydb restore --latest --dry-run --dir /var/lib/rubydb/backups
+```
+
+The backup implementation flushes the WAL and storage before copying files and
+publishes its manifest atomically. A successful command is not a substitute for
+the scheduled restore drill in `scripts/restore_drill`.
+
+## Retention and deletion
+
+Use a documented retention schedule such as daily backups for 14 days, weekly
+backups for 12 weeks, and monthly backups for the required compliance period.
+Delete only verified, expired backup sets; never delete a base backup while an
+incremental or differential chain depends on it.
+
+## Recovery point checks
+
+Record the backup name, manifest checksum, database version, page size, WAL
+position, and verification result in the deployment record. Before restoring,
+confirm that the selected backup is complete and that its destination is a new,
+inactive directory.

data/docs/operations/disaster-recovery.md

@@ -0,0 +1,31 @@
+# Disaster recovery
+
+Define application-specific recovery objectives before deployment:
+
+- RPO: the maximum acceptable committed data loss.
+- RTO: the maximum time to restore service.
+- retention: how long backups and incident copies must be kept.
+
+RubyDB’s verified recovery path is restore to a new directory followed by an
+engine reopen and row/catalog verification. It does not silently repair
+unknown corruption or perform automatic cross-region failover.
+
+## Recovery procedure
+
+1. Declare the incident and stop application writes to the affected database.
+2. Preserve the original data directory, WAL, logs, configuration, and process
+   information. Work from a copied incident set.
+3. Select the newest backup whose manifest and checksum verify within the RPO.
+4. Restore into a new directory and run `rubydb restore --dry-run` before
+   opening it.
+5. Open the restored database, run migrations/status checks, and execute the
+   application smoke and integrity queries.
+6. Switch the application to the restored directory only after validation.
+7. Preserve the failed directory until the incident review is complete.
+
+Run `ruby scripts/restore_drill` in staging on the same filesystem and RubyDB
+release family used in production. Record measured restore time, recovered WAL
+position, row counts, and any gaps against RTO/RPO.
+
+Do not remove WAL, run vacuum, force promotion, or overwrite the source during
+an active corruption investigation.

data/docs/operations/failover.md

@@ -0,0 +1,30 @@
+# Failover procedure
+
+Automatic election is intentionally disabled until fencing and partition
+testing are complete. Failover is an operator-controlled procedure.
+
+## Promote a replica safely
+
+1. Confirm the primary is stopped or fenced and cannot accept writes.
+2. Confirm the candidate replica is `SYNCED`, or is `DISCONNECTED` only after
+   it was previously synchronized, and that received and replayed WAL
+   positions are equal.
+3. Compare the candidate replay position with the incident recovery point.
+4. Promote only after the candidate passes the configured fencing checks.
+5. Point clients at the promoted node and run read/write smoke queries.
+6. Keep the old primary isolated until its data and fencing epoch are reviewed.
+
+Promotion must fail closed when a replica is lagging, has never replayed a
+durable position, or is missing a required recovery point. A disconnected but
+caught-up replica is eligible only for explicit operator promotion after the
+old primary is stopped or fenced. Never force promotion to hide a WAL gap.
+
+## Rejoin the old primary
+
+Treat the old primary as a stale writer. Preserve its files, compare WAL and
+catalog state, and rebuild or re-bootstrap it as a replica if divergence cannot
+be proven absent. Re-enable it for writes only after a new fencing lease and a
+clean recovery validation.
+
+Record the old and new fencing epochs, replica LSNs, promotion time, observed
+lag, and application error rate in the incident record.