rubydb 0.1.4 → 0.1.6

data/lessons/02-local-development.md

@@ -0,0 +1,121 @@
+# Lesson 2: repeatable local development
+
+The fastest safe workflow is to make local setup disposable and repeatable.
+Keep database files under `tmp/` or another ignored directory, use separate
+development and test paths, and run migrations from source control.
+
+## Rails application setup
+
+Add the RubyDB gems to a Rails application. Pin versions in a real application
+after testing them; the versions below match the published example used by the
+RubyDB project at the time this lesson was written.
+
+```ruby
+# Gemfile
+gem "rubydb", "0.1.6"
+gem "rubydb-activerecord", "0.1.3"
+gem "pg" # Keep this when production may use PostgreSQL.
+```
+
+Install dependencies:
+
+```sh
+bundle install
+```
+
+Configure development and test with separate embedded files:
+
+```yaml
+# config/database.yml
+default: &default
+  adapter: rubydb
+  embedded: true
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS", "5") %>
+
+development:
+  <<: *default
+  database: <%= Rails.root.join("tmp/rubydb_development.rdb") %>
+
+test:
+  <<: *default
+  database: <%= Rails.root.join("tmp/rubydb_test.rdb") %>
+```
+
+Run the schema and tests:
+
+```sh
+bin/rails db:create
+bin/rails db:migrate
+bin/rails test
+bin/rails server
+```
+
+`db:create` is harmless only when it targets a disposable local path. Never
+point it at a production database and never commit `.rdb` files.
+
+## A regular Ruby smoke program
+
+This is a complete single-process example using the public storage API:
+
+```ruby
+# script/rubydb_smoke.rb
+require "rubydb"
+
+path = "tmp/rubydb_smoke.rdb"
+engine = RubyDB::Storage::Engine.new(path)
+begin
+  engine.execute("CREATE TABLE IF NOT EXISTS events (id INTEGER PRIMARY KEY, name TEXT NOT NULL)")
+  engine.execute("INSERT INTO events (name) VALUES ('boot')")
+  rows = engine.execute("SELECT id, name FROM events ORDER BY id")
+  puts rows.inspect
+ensure
+  engine.close
+end
+```
+
+Run it with:
+
+```sh
+bundle exec ruby script/rubydb_smoke.rb
+```
+
+For a multi-process application, replace the embedded engine with a RubyDB
+server and `RubyDB::Client::Client`. Do not let a web process and a worker
+open the same embedded path independently.
+
+## Local PostgreSQL when it is the production target
+
+Use the same application code against a local PostgreSQL instance when you
+intend to deploy PostgreSQL. This catches dialect, type, index, and migration
+differences early:
+
+```sh
+docker run --name rubydb-lesson-postgres \
+  --env POSTGRES_PASSWORD=devpassword \
+  --env POSTGRES_DB=lesson_app \
+  --publish 5432:5432 \
+  --detach postgres:16
+```
+
+Set the local connection string in your shell or an ignored `.env` file:
+
+```sh
+DATABASE_URL=postgresql://postgres:devpassword@127.0.0.1:5432/lesson_app
+bin/rails db:migrate
+bin/rails test
+```
+
+Do not put that password in Git. Stop and remove this disposable container
+only after confirming it is the lesson container:
+
+```sh
+docker stop rubydb-lesson-postgres
+docker rm rubydb-lesson-postgres
+```
+
+## Checkpoint
+
+The checkpoint passes when a new developer can clone the project, run
+`bundle install`, migrate an empty database, run the tests, and reproduce the
+smoke query without manually editing a database file. Continue with [lesson 3](03-embedded-rubydb.md)
+to understand the limits of embedded mode.

data/lessons/03-embedded-rubydb.md

@@ -0,0 +1,99 @@
+# Lesson 3: use RubyDB embedded deliberately
+
+Embedded mode puts the database engine in the application process. It is
+simple, fast to start, and useful for local development, command-line tools,
+small internal utilities, and a service with one database-owning process.
+
+It is not a shared file protocol. Two independent processes must not open the
+same `.rdb` path concurrently. A Rails web server with several worker
+processes, or a web server plus a job worker, should use RubyDB server/client
+mode instead.
+
+## A safe embedded boundary
+
+```text
+one Ruby process
+    |
+    +-- RubyDB::Storage::Engine
+            |
+            +-- tmp/service.rdb or /var/lib/service/service.rdb
+```
+
+Keep the path outside the source tree and make its owner explicit:
+
+```ruby
+require "rubydb"
+
+database_path = ENV.fetch("RUBYDB_DATABASE", "tmp/service.rdb")
+engine = RubyDB::Storage::Engine.new(database_path)
+
+begin
+  engine.execute(<<~SQL)
+    CREATE TABLE IF NOT EXISTS jobs (
+      id INTEGER PRIMARY KEY,
+      state TEXT NOT NULL,
+      created_at TIMESTAMP
+    )
+  SQL
+
+  engine.execute("INSERT INTO jobs (state, created_at) VALUES ('queued', CURRENT_TIMESTAMP)")
+  p engine.execute("SELECT id, state FROM jobs ORDER BY id")
+ensure
+  engine.close
+end
+```
+
+Use bound values for data. Do not build SQL by interpolating request
+parameters, usernames, or search terms.
+
+## Rails embedded configuration
+
+```yaml
+development:
+  adapter: rubydb
+  embedded: true
+  database: <%= ENV.fetch("RUBYDB_DATABASE", Rails.root.join("tmp/development.rdb")) %>
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS", "5") %>
+```
+
+For tests, use a different path. For a one-process production service, put the
+database on persistent storage, set restrictive filesystem permissions, and
+ensure the service manager starts only one owner. If the deployment scales the
+service horizontally, embedded mode is the wrong topology.
+
+## Inspect and back up the file
+
+The repository CLI has read-only inspection, verified backup, restore, and
+doctor commands:
+
+```sh
+rubydb doctor --quick --json
+rubydb inspect --database tmp/service.rdb --stats --wal
+rubydb backup --database tmp/service.rdb --dir tmp/backups --type full --compress
+```
+
+Check the installed command’s help before automating a version-specific option:
+
+```sh
+rubydb doctor --help
+rubydb backup --help
+rubydb restore --help
+```
+
+Keep the backup directory on a different disk or host in production. A second
+copy on the same failed disk is not a recovery plan.
+
+## What embedded mode does not solve
+
+Embedded mode does not provide a network endpoint, cross-host failover,
+automatic leader election, a connection pool shared by processes, or a managed
+cloud backup. Those are deployment responsibilities. If the workload needs
+them, move to server/client mode or choose a managed PostgreSQL service.
+
+## Checkpoint
+
+Prove that the owner rule is enforceable: document the process that opens the
+file, the persistent path, the backup destination, and the restore operator.
+Then deliberately run the application with two processes in a disposable test
+environment and confirm your deployment prevents shared-file access. Continue
+to [lesson 4](04-rails-complex-apps.md) for Rails query and migration validation.

data/lessons/04-rails-complex-apps.md

@@ -0,0 +1,138 @@
+# Lesson 4: Rails and complex application code
+
+RubyDB can be useful for a Rails application when the application stays within
+the adapter’s tested surface. Complex Rails code is still possible, but every
+important query and migration must be exercised against the exact database
+topology you will deploy.
+
+## Pin the adapter and configure the boundary
+
+```ruby
+# Gemfile
+gem "rubydb", "0.1.6"
+gem "rubydb-activerecord", "0.1.3"
+```
+
+Embedded development configuration:
+
+```yaml
+# config/database.yml
+development:
+  adapter: rubydb
+  embedded: true
+  database: <%= Rails.root.join("tmp/development.rdb") %>
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS", "5") %>
+
+test:
+  adapter: rubydb
+  embedded: true
+  database: <%= Rails.root.join("tmp/test.rdb") %>
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS", "5") %>
+```
+
+For server mode, use one connection URL supplied by the deployment:
+
+```yaml
+production:
+  adapter: rubydb
+  embedded: false
+  url: <%= ENV.fetch("RUBYDB_URL") %>
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS", "5") %>
+```
+
+The RubyDB adapter URL is `rubydb://` or TLS-enabled `rubydbs://`; it is not a
+PostgreSQL `DATABASE_URL`.
+
+## Associations, joins, and eager loading
+
+Start with ordinary Rails models:
+
+```ruby
+class Account < ApplicationRecord
+  has_many :projects, dependent: :destroy
+end
+
+class Project < ApplicationRecord
+  belongs_to :account
+  has_many :tasks, dependent: :destroy
+
+  scope :active, -> { where(status: "active") }
+end
+
+class Task < ApplicationRecord
+  belongs_to :project
+end
+```
+
+Exercise both the SQL shape and the object-loading behavior:
+
+```ruby
+accounts = Account
+  .joins(:projects)
+  .merge(Project.active)
+  .where(projects: { archived: false })
+  .includes(projects: :tasks)
+  .distinct
+  .order(:name)
+
+accounts.each do |account|
+  account.projects.each do |project|
+    puts [account.name, project.name, project.tasks.size].join(" | ")
+  end
+end
+```
+
+This example is a validation target, not a promise that every Arel variation
+or database-specific query will work. Add request or model tests that assert
+the result set, duplicate behavior, `NULL` behavior, ordering, and query count.
+
+## Transactions and migrations
+
+Keep a transaction around a business operation and make retry behavior
+explicit:
+
+```ruby
+ApplicationRecord.transaction do
+  project = Project.create!(account: account, name: "Billing", status: "active")
+  project.tasks.create!(title: "Verify invoice", state: "open")
+end
+```
+
+A migration must be safe on an empty and populated database:
+
+```ruby
+class AddStateToTasks < ActiveRecord::Migration[7.2]
+  def change
+    add_column :tasks, :state, :string, null: false, default: "open"
+    add_index :tasks, [:project_id, :state]
+  end
+end
+```
+
+For a large table, test the migration duration, lock behavior, disk usage, and
+rollback boundary on a restored copy. Do not assume a migration that succeeds
+on an empty local file is safe during live traffic.
+
+## The compatibility test matrix
+
+Run the application suite for every supported combination, including the
+database actually used in production:
+
+```sh
+bundle exec rails db:drop db:create db:migrate
+bundle exec rails test
+bundle exec rails db:schema:dump
+```
+
+Repeat that sequence with RubyDB embedded, RubyDB server/client, and
+PostgreSQL when all three are supported. Compare schema dumps and test
+business behavior, not merely process exit codes. Rails versions and adapter
+versions should be pinned in CI; the project’s compatibility guide is the
+source of truth for the currently tested matrix.
+
+## Checkpoint
+
+The checkpoint passes when your suite covers joins, eager loading, nested
+associations, transactions, constraints, indexes, a populated-table
+migration, schema dump/load, and error handling. Continue to [lesson 5](05-rubydb-production-server.md)
+to run the same application through a private RubyDB server.

data/lessons/05-rubydb-production-server.md

@@ -0,0 +1,237 @@
+# Lesson 5: RubyDB server production setup
+
+Server mode puts one RubyDB process in charge of the data directory and lets
+Ruby or Rails clients connect over the RubyDB protocol. This is the correct
+RubyDB topology when several application processes need one database. The
+application must never open the server’s `.rdb` files directly.
+
+## Provision the service
+
+Use a dedicated service account and persistent storage. The commands below
+assume a Unix-like host; adapt ownership commands to your platform:
+
+```sh
+gem install rubydb -v 0.1.6
+install -d -o rubydb -g rubydb -m 0700 /var/lib/rubydb/data
+install -d -o rubydb -g rubydb -m 0750 /var/log/rubydb
+install -d -o rubydb -g rubydb -m 0700 /etc/rubydb
+```
+
+Keep `/var/lib/rubydb` on persistent storage and place backups in a separate
+failure domain. Restrict the database port to the application network.
+
+## Configuration and startup
+
+Create `/etc/rubydb/production.yml` from the repository’s production template.
+Supply credentials and TLS paths through a secret manager or protected
+deployment environment. A production configuration needs WAL, durable storage,
+authentication, TLS, and bounded resources.
+
+Start the supervised foreground process:
+
+```sh
+rubydb --config /etc/rubydb/production.yml --env production start
+```
+
+For a development smoke server, the CLI also accepts explicit settings:
+
+```sh
+rubydb --env production start \
+  --host 127.0.0.1 \
+  --port 7432 \
+  --data-dir /var/lib/rubydb/data \
+  --log-dir /var/log/rubydb \
+  --pid-file /run/rubydb.pid
+```
+
+Use systemd, a container supervisor, or the platform’s process manager to
+restart the process and preserve logs. Do not let an orchestrator launch two
+writers against one embedded path.
+
+## Straight-to-production deployment path
+
+Use this path when RubyDB is the chosen production database for a bounded
+service. It works for a Rails app or a regular Ruby app. For a massive shared
+application, use the PostgreSQL path in [lesson 6](06-postgresql-massive-apps.md)
+instead.
+
+### 1. Prepare the database host
+
+The database host needs a persistent local volume, a dedicated service account,
+and a private network route from the application. Run these commands as an
+administrator and replace paths only after verifying the target host:
+
+```sh
+gem install rubydb -v 0.1.6
+useradd --system --home-dir /var/lib/rubydb --shell /usr/sbin/nologin rubydb
+install -d -o rubydb -g rubydb -m 0700 /var/lib/rubydb/data
+install -d -o rubydb -g rubydb -m 0750 /var/log/rubydb
+install -d -o root -g rubydb -m 0750 /etc/rubydb
+```
+
+Copy the reviewed `config/production.yml` from the RubyDB repository to
+`/etc/rubydb/production.yml`. Keep the database directory and WAL on approved
+persistent storage. Keep backups on another host or failure domain.
+
+### 2. Install secrets and TLS material
+
+Use a CA-issued certificate for a real deployment. Do not use a self-signed
+development certificate for public traffic. Inject these values through a
+secret manager, protected service environment, or equivalent mechanism:
+
+```text
+RUBYDB_USERNAME=app_rw
+RUBYDB_PASSWORD=generated-secret
+RUBYDB_SSL_ENABLED=true
+RUBYDB_SSL_CERT_FILE=/etc/rubydb/tls/server.crt
+RUBYDB_SSL_KEY_FILE=/etc/rubydb/tls/server.key
+RUBYDB_SSL_CA_FILE=/etc/rubydb/tls/ca.crt
+RUBYDB_SSL_VERIFY_PEER=true
+```
+
+Restrict the private key and environment file to the service. Do not put
+secrets in Git, Docker images, process arguments, shell history, or logs.
+Allow port `7432` only from the application and administration networks.
+
+### 3. Start and verify RubyDB
+
+Start the foreground process under a service manager such as systemd:
+
+```sh
+sudo -u rubydb env RUBYDB_USERNAME=app_rw RUBYDB_PASSWORD='from-secret-store' \
+  rubydb --config /etc/rubydb/production.yml --env production start
+```
+
+Verify the server from the application network:
+
+```sh
+rubydb --config /etc/rubydb/production.yml --env production status --json
+rubydb --config /etc/rubydb/production.yml --env production doctor --json
+```
+
+Run an authenticated TLS query using the same URL that the app will use. The
+password below is intentionally a secret-manager value, not a real credential:
+
+```sh
+RUBYDB_URL='rubydbs://app_rw:URL_ENCODED_PASSWORD@db.internal:7432/app?verify_peer=true&ca_file=%2Fetc%2Frubydb%2Ftls%2Fca.crt' \
+  ruby -rrubydb -e 'c=RubyDB::Client::Client.new(url: ENV.fetch("RUBYDB_URL")); p c.query("SELECT 1").to_hash; c.disconnect'
+```
+
+### 4. Deploy Rails or Ruby
+
+For Rails, use server mode and inject the URL:
+
+```yaml
+# config/database.yml
+production:
+  adapter: rubydb
+  embedded: false
+  url: <%= ENV.fetch("RUBYDB_URL") %>
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS", "5") %>
+```
+
+For a regular Ruby service:
+
+```ruby
+require "rubydb"
+
+client = RubyDB::Client::Client.new(url: ENV.fetch("RUBYDB_URL"))
+begin
+  result = client.query("SELECT 1")
+  puts result.to_hash
+ensure
+  client.disconnect
+end
+```
+
+Deploy the application artifact with the pinned RubyDB gems, then run schema
+migrations once from a controlled release job:
+
+```sh
+RAILS_ENV=production bundle exec rails db:migrate
+RAILS_ENV=production bundle exec rails db:migrate:status
+RAILS_ENV=production bundle exec rails runner 'puts ApplicationRecord.connection.select_value("SELECT 1")'
+```
+
+The web and worker processes connect to `RUBYDB_URL`; they never mount or open
+the server’s database directory.
+
+### 5. Back up before accepting traffic
+
+Create and verify a full backup, then perform a restore into an inactive path:
+
+```sh
+rubydb backup --database /var/lib/rubydb/data/app.rdb \
+  --dir /var/backups/rubydb --type full --compress
+rubydb restore --database /var/lib/rubydb/restore-check.rdb \
+  --dir /var/backups/rubydb --latest --dry-run
+rubydb restore --database /var/lib/rubydb/restore-check.rdb \
+  --dir /var/backups/rubydb --latest --force
+```
+
+Complete an actual isolated restore and run representative reads before the
+first public request. Record the backup checksum, restore duration, RPO, RTO,
+and the operator responsible.
+
+### 6. Release traffic gradually
+
+Start with a canary or a small percentage of traffic. Verify one read, one
+write transaction, one background job, and one application error path. Watch
+latency, errors, active connections, WAL/checkpoint growth, memory, and free
+disk. Keep the previous app artifact and verified backup until the rollback
+window closes.
+
+This path makes RubyDB usable in production for a validated bounded workload;
+it does not provide automatic multi-host high availability by itself. If the
+service needs automatic election, cross-host failover, or PostgreSQL-specific
+SQL, stop and complete the corresponding validation or choose PostgreSQL.
+
+## Rails connection
+
+```yaml
+# config/database.yml
+production:
+  adapter: rubydb
+  embedded: false
+  url: <%= ENV.fetch("RUBYDB_URL") %>
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS", "5") %>
+```
+
+Use TLS verification in the URL:
+
+```text
+rubydbs://app_user:URL_ENCODED_PASSWORD@db.internal.example:7432/app?verify_peer=true&ca_file=%2Fetc%2Frubydb%2Fca.crt
+```
+
+Inject `RUBYDB_URL` from a secret manager. Percent-encode special characters
+in credentials, and never print the complete URL. The server must separately
+define its authentication credentials, certificate, private key, and CA.
+
+## Smoke test before traffic
+
+```sh
+rubydb --config /etc/rubydb/production.yml --env production status --json
+rubydb --config /etc/rubydb/production.yml --env production doctor --quick --json
+RAILS_ENV=production bundle exec rails db:migrate:status
+RAILS_ENV=production bundle exec rails runner 'puts ApplicationRecord.connection.select_value("SELECT 1")'
+```
+
+Then exercise one authenticated read, one transaction that creates and updates
+data, one background job, and one verified backup/restore drill. Capture the
+RubyDB version, application release, configuration checksum, and test output.
+
+## Cloud deployment shape
+
+On a platform such as Render, use a private database service with a persistent
+disk and a separate web service. Put the private hostname in `RUBYDB_URL` and
+keep both services in the same region. A web-service scale-out is not database
+high availability; maintain external backups and validate a failover design
+before relying on it.
+
+## Checkpoint
+
+The checkpoint passes when multiple app processes connect over TLS, writes are
+visible to another client, migrations run once from a release job, readiness
+checks are monitored, and a restore has been tested into a new directory.
+Continue to [lesson 6](06-postgresql-massive-apps.md) to decide when PostgreSQL
+is the better production choice.

data/lessons/06-postgresql-massive-apps.md

@@ -0,0 +1,96 @@
+# Lesson 6: PostgreSQL for large applications
+
+For a massive or business-critical application, PostgreSQL is usually the
+safer system of record. It has a mature ecosystem of managed services,
+replication, backup tooling, connection poolers, extensions, observability,
+and Rails deployment experience. Choose it when many app instances, large
+tables, broad SQL compatibility, or a large operations team are requirements.
+
+RubyDB can still be valuable during development or as a bounded service. The
+choice is architectural: do not make one database responsible for a workload
+whose concurrency, failover, or SQL requirements it has not passed.
+
+## Rails configuration
+
+Add the PostgreSQL adapter:
+
+```ruby
+# Gemfile
+gem "pg"
+```
+
+Configure production with the provider’s secret connection URL:
+
+```yaml
+# config/database.yml
+production:
+  url: <%= ENV.fetch("DATABASE_URL") %>
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS", "5") %>
+  checkout_timeout: <%= ENV.fetch("DB_CHECKOUT_TIMEOUT", "5") %>
+```
+
+The URL is normally shaped like this; use a generated password rather than
+the sample credentials:
+
+```text
+postgresql://app_user:URL_ENCODED_PASSWORD@postgres.internal:5432/app_production
+```
+
+Run migrations from one controlled release job:
+
+```sh
+RAILS_ENV=production bundle exec rails db:migrate
+RAILS_ENV=production bundle exec rails db:migrate:status
+```
+
+Do not run `db:migrate` concurrently from every web instance.
+
+## Local parity
+
+Use a pinned PostgreSQL major version locally and in CI:
+
+```sh
+docker run --name lesson-postgres \
+  --env POSTGRES_PASSWORD=devpassword \
+  --env POSTGRES_DB=app_development \
+  --publish 5432:5432 \
+  --detach postgres:16
+
+DATABASE_URL=postgresql://postgres:devpassword@127.0.0.1:5432/app_development \
+  bundle exec rails db:migrate
+```
+
+The password is for a disposable local container only. Use secret storage in
+CI and production.
+
+## Connection pool sizing
+
+Each Rails process can open up to its configured pool size. A first estimate is
+`web_processes * pool_size + worker_pools + admin_headroom`, but the database
+provider’s connection limit and actual workload decide the safe value. Measure
+queue time and query latency. A pool setting that is larger than the database
+limit causes timeouts rather than more throughput.
+
+For larger deployments, evaluate a PostgreSQL-aware connection pooler and your
+provider’s read-replica strategy. Test transactions, prepared statements,
+failover behavior, and session settings with the pooler before production.
+
+## SQL and data features
+
+PostgreSQL is appropriate when the application relies on PostgreSQL-specific
+types, functions, extensions, advanced indexing, row-level locking, or complex
+query plans. Keep those choices explicit in migrations and tests. RubyDB’s
+adapter is not a promise that PostgreSQL SQL can run unchanged on RubyDB.
+
+If the application starts on RubyDB and moves to PostgreSQL, run reviewed
+migrations on a new PostgreSQL database, export and transform data explicitly,
+compare row counts and business totals, and test the cutover. An environment
+variable changes where new connections go; it does not convert an `.rdb` file.
+
+## Checkpoint
+
+The checkpoint passes when the production Rails build can create a PostgreSQL
+database, apply migrations once, run the full application test suite, take a
+provider-supported backup, restore a staging copy, and demonstrate the
+connection pool remains below the provider limit. Continue to [lesson 7](07-hybrid-microservices.md)
+for a hybrid design using both databases responsibly.