rubydb 0.1.4 → 0.1.6

data/lessons/07-hybrid-microservices.md

@@ -0,0 +1,179 @@
+# Lesson 7: a hybrid microservice architecture
+
+A practical RubyDB architecture is to keep the large shared business system
+on PostgreSQL and use RubyDB for a small service with a narrow responsibility.
+Examples include a local catalog, a bounded document/index service, an
+internal workflow, or a tenant-isolated tool whose SQL and recovery needs have
+been validated.
+
+## Give each service ownership
+
+```text
+Rails monolith / public API
+    |
+    +--> PostgreSQL: users, billing, orders, reporting
+    |
+    +--> RubyDB service API: bounded internal records
+                 |
+                 +--> one RubyDB server and persistent data directory
+```
+
+The services communicate through an API or an event contract. They do not
+share an embedded file and they do not write directly into each other’s tables.
+Each service owns its migrations, credentials, backups, alerts, and recovery
+runbook.
+
+## Ruby client for a service
+
+```ruby
+# app/services/catalog_store.rb
+require "rubydb"
+
+class CatalogStore
+  def initialize(url: ENV.fetch("RUBYDB_URL"))
+    @client = RubyDB::Client::Client.new(url: url)
+  end
+
+  def find(code)
+    result = @client.query(
+      "SELECT code, title FROM catalog_items WHERE code = ?",
+      [code]
+    )
+    result.to_a.first
+  end
+
+  def close
+    @client.disconnect
+  end
+end
+```
+
+Use the actual client API in the version pinned by the service and add tests
+for connection failures, timeouts, duplicate requests, and empty results. In a
+long-running app, put client lifecycle management in the application’s
+dependency/container layer and close it during shutdown.
+
+## Python services with the RubyDB adapter
+
+Python applications connect to RubyDB server mode through the published
+`rubydb-python` DB-API 2.0 adapter. The Python process must not open an
+embedded `.rdb` file. Put the server URL in a secret-managed environment
+variable:
+
+```powershell
+$env:RUBYDB_URL = "rubydbs://service_user:URL_ENCODED_PASSWORD@rubydb.internal:7432/orders?verify_peer=true&ca_file=%2Fetc%2Frubydb%2Ftls%2Fca.crt"
+python -m pip install rubydb-python
+```
+
+Use parameterized queries and a bounded pool in workers:
+
+```python
+import os
+from rubydb import ConnectionPool
+
+pool = ConnectionPool(os.environ["RUBYDB_URL"], min_size=1, max_size=8)
+try:
+    with pool.connection() as connection:
+        with connection.cursor() as cursor:
+            cursor.execute(
+                "SELECT id, status FROM jobs WHERE account_id = ?",
+                [account_id],
+            )
+            rows = cursor.fetchall()
+finally:
+    pool.close()
+```
+
+The adapter is synchronous DB-API code. In an async framework such as Flaxon,
+run database calls in a worker thread so a slow query does not block the event
+loop:
+
+```python
+import asyncio
+from rubydb import connect
+
+async def load_jobs(url):
+    def query():
+        with connect(url, timeout=5) as db:
+            with db.cursor() as cursor:
+                cursor.execute("SELECT id, status FROM jobs ORDER BY id")
+                return cursor.fetchall()
+
+    return await asyncio.to_thread(query)
+```
+
+Run the complete examples in `examples/python_flask` and
+`examples/python_flaxon`. Both examples use real RubyDB TCP traffic and have
+live integration tests; they are intentionally small starting points, not a
+replacement for application-specific authorization, migrations, backups,
+timeouts, monitoring, and load testing.
+
+## Node.js and TypeScript services
+
+Node services use the `rubydb-node` package over the same RubyDB server
+protocol:
+
+```sh
+npm install rubydb-node
+```
+
+```ts
+import { connect } from "rubydb-node";
+
+const db = await connect(process.env.RUBYDB_URL!);
+try {
+  const result = await db.query(
+    "SELECT id, state FROM jobs WHERE account_id = ?",
+    [accountId],
+  );
+  console.log(result.rows);
+} finally {
+  await db.close();
+}
+```
+
+Use `ConnectionPool` for concurrent workers, keep the pool bounded per process,
+and use `rubydbs://` with peer verification in production. The package is
+TypeScript-first, supports prepared statements, transactions, timeouts with
+wire cancellation, and does not access embedded database files. See
+`adapters/rubydb/README.md` for the full Node release and operations boundary.
+
+## A small Rails service
+
+```yaml
+# service/config/database.yml
+production:
+  adapter: rubydb
+  embedded: false
+  url: <%= ENV.fetch("RUBYDB_URL") %>
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS", "5") %>
+```
+
+Keep the API idempotent. A client timeout can happen after the server commits
+a write, so a retry must use an idempotency key or first check the operation’s
+result. For cross-service workflows, record an outbox/event in the owning
+system and design consumers to tolerate duplicate delivery.
+
+## What belongs where
+
+Keep users, payments, orders, and cross-tenant reporting in PostgreSQL when
+they need shared relational consistency and broad analytical tooling. Keep
+RubyDB data that can be independently backed up, restored, migrated, and
+reconciled. Do not split a single atomic business transaction across the two
+databases unless you have designed and tested a distributed workflow.
+
+## Failure and deployment rules
+
+* Deploy the RubyDB service with a persistent volume and one server owner.
+* Make the service private; clients use TLS and least-privilege credentials.
+* Set bounded connection and request timeouts and expose a useful health probe.
+* Retry only idempotent operations, with backoff and a maximum attempt count.
+* Maintain a PostgreSQL and RubyDB restore drill independently.
+* Version the API/event contract before changing either database schema.
+
+## Checkpoint
+
+The checkpoint passes when each data set has one owner, the API can tolerate a
+restarted database service, duplicate requests do not create duplicate business
+records, and the two systems can be restored independently. Continue to [lesson 8](08-migrations-backups-recovery.md)
+for migration and recovery drills.

data/lessons/08-migrations-backups-recovery.md

@@ -0,0 +1,86 @@
+# Lesson 8: migrations, backups, and recovery
+
+Production data work is a controlled change to a recoverable system. A green
+migration on an empty database is not enough. Test populated tables, rollback
+boundaries, disk growth, indexes, locks, and application compatibility.
+
+## RubyDB migration workflow
+
+Preview and apply a migration against a disposable or staging copy:
+
+```sh
+rubydb doctor --quick --json
+rubydb backup --database data/app.rdb --dir backups --type full --compress
+rubydb migrate --database data/app.rdb --path db/migrate --dry-run
+rubydb migrate --database data/app.rdb --path db/migrate
+```
+
+Run the migration once from a release job, not once per web process. Keep the
+pre-migration backup and record the database version, application revision,
+backup checksum, operator, and start/end times.
+
+## Verified RubyDB backup and restore
+
+Create a full backup with verification enabled, then dry-run the restore:
+
+```sh
+rubydb backup --database data/app.rdb --dir backups --type full --compress
+rubydb restore --dir backups --latest --dry-run
+rubydb restore --database restored/app.rdb --dir backups --latest
+```
+
+If a release depends on incremental or differential backups, retain the entire
+required chain and its manifest. Keep backups away from the database disk and
+test that a new host can read them. Restore into a new inactive path; do not
+overwrite the only source with `--force`.
+
+After restoring, verify both structure and meaning:
+
+```sh
+rubydb inspect --database restored/app.rdb --stats --wal
+rubydb shell --database restored/app.rdb --json
+```
+
+Run schema checks, row counts, foreign-key checks, representative application
+queries, and business totals. Record the achieved RPO and RTO rather than
+assuming the command’s success proves recoverability.
+
+## PostgreSQL backup and restore
+
+For PostgreSQL, use the provider’s managed backup/PITR feature where possible
+and rehearse an independent logical backup:
+
+```sh
+pg_dump --format=custom --file=tmp/app-staging.dump "$DATABASE_URL"
+createdb app_restore
+pg_restore --clean --if-exists --dbname=app_restore tmp/app-staging.dump
+```
+
+The restore target must be isolated from production. Validate extensions,
+roles, ownership, sequences, indexes, and application behavior after restore.
+For a managed service, follow its documented restore and point-in-time
+procedure instead of assuming local `createdb` access exists.
+
+## Corruption and interrupted writes
+
+When storage or recovery is suspect:
+
+1. stop application writes and preserve the database, WAL, metadata, config,
+   and logs together;
+2. copy the evidence to a separate incident location;
+3. run `doctor --quick` and `inspect --stats --wal` on the copy;
+4. restore the latest verified backup into a new path;
+5. compare schema, row counts, checksums, and business totals; and
+6. document what data was lost, replayed, or manually reconciled.
+
+Do not “repair” corruption by deleting files, dropping tables, or running full
+vacuum on the only copy. Use the repository’s durability and recovery drills
+to rehearse interrupted checkpoints, full disks, corrupted files, compaction,
+and restore at scale before an incident.
+
+## Checkpoint
+
+The checkpoint passes when an operator who did not create the backup can restore
+it on another path, prove the data is usable, and state measured RPO/RTO. Keep
+the report with the release evidence. Continue to [lesson 9](09-observability-security-scale.md)
+for operations and security.

data/lessons/09-observability-security-scale.md

@@ -0,0 +1,87 @@
+# Lesson 9: observability, security, and scale
+
+Production readiness is an operating discipline. A database is ready only when
+the team can see failures, limit damage, rotate credentials, and make a safe
+recovery decision under pressure.
+
+## Protect the connection
+
+For RubyDB server mode, use a private network, password authentication, and
+TLS with peer verification:
+
+```text
+rubydbs://app_user:URL_ENCODED_PASSWORD@db.internal.example:7432/app?verify_peer=true&ca_file=%2Fetc%2Frubydb%2Fca.crt
+```
+
+Store `RUBYDB_URL`, `RUBYDB_PASSWORD`, certificates, and private keys in the
+deployment secret manager. Do not commit them, put them in a Docker image, or
+print them in health checks. Give the application only the database privileges
+it needs and use a separate migration/admin identity.
+
+For PostgreSQL, use `DATABASE_URL` from the provider’s secret store, TLS
+settings required by that provider, least-privilege roles, and rotated
+credentials. Review role grants after every schema or service change.
+
+## Set limits before load
+
+Bound every layer that can wait:
+
+```yaml
+# config/database.yml
+production:
+  adapter: rubydb
+  embedded: false
+  url: <%= ENV.fetch("RUBYDB_URL") %>
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS", "5") %>
+  checkout_timeout: <%= ENV.fetch("DB_CHECKOUT_TIMEOUT", "5") %>
+```
+
+Also set application request/job timeouts, connection idle limits, server
+connection limits, maximum request body sizes, and process memory/CPU limits.
+Choose values from measurements. A timeout should fail a request cleanly and
+produce a useful event; it should not cause an unsafe blind retry.
+
+## Monitor signals that lead to incidents
+
+Collect metrics and structured logs for:
+
+* request and query latency by operation, including p95 and p99;
+* error, timeout, cancellation, retry, and deadlock counts;
+* active connections, pool wait time, and transaction age;
+* WAL/checkpoint growth, database size, free disk, and compaction/vacuum time;
+* backup age, backup verification result, restore drill age, and RPO/RTO; and
+* process restarts, readiness failures, replication lag, and failover events.
+
+The RubyDB CLI is useful evidence in an operator check:
+
+```sh
+rubydb --env production status --json
+rubydb --env production doctor --quick --json
+rubydb inspect --database data/app.rdb --stats --wal
+```
+
+Turn those outputs into alerts with thresholds and an owner. A green process
+status is not the same as a healthy application: include a real application
+query and a write/read smoke test in staging and deployment verification.
+
+## Load and concurrency validation
+
+Start with a reproducible workload, then increase concurrency gradually:
+
+```sh
+RUBYDB_WORKLOAD_THREADS=16 RUBYDB_WORKLOAD_OPERATIONS=10000 \
+  ruby benchmarks/concurrent_workload.rb
+```
+
+Use the repository’s documented workload script and record its output. Measure
+throughput, latency, failures, cancellations, timeouts, deadlocks, memory, WAL,
+and disk use. Run long enough to expose leaks and queue growth. Test separate
+processes and separate hosts for server mode; an in-process benchmark does not
+prove network behavior.
+
+## Checkpoint
+
+The checkpoint passes when alerts have thresholds and owners, secrets can be
+rotated without source changes, resource limits are documented, and a sustained
+test produces a baseline with no unexplained errors or unbounded growth.
+Continue to [lesson 10](10-release-readiness.md) for the release gate.

data/lessons/10-release-readiness.md

@@ -0,0 +1,192 @@
+# Lesson 10: release readiness
+
+The final checkpoint is evidence, not optimism. A production release should
+identify the exact RubyDB and adapter versions, supported Ruby/Rails/OS matrix,
+tested SQL surface, backup artifact, restore result, load baseline, security
+review, and rollback owner.
+
+## Run repository checks
+
+From the RubyDB repository:
+
+```sh
+bundle install
+bundle exec rspec
+bundle exec rake
+git diff --check
+```
+
+Run the Rails adapter suite from its directory and repeat it for every Rails
+and Ruby version you claim to support:
+
+```sh
+cd adapters/activerecord
+bundle install
+bundle exec rspec
+```
+
+Add your application’s complex query, migration, schema dump/load, concurrency,
+and failure tests to CI. A passing library suite does not certify an arbitrary
+application.
+
+## Release a gem safely
+
+Review the project’s release instructions and run the preflight with a version
+that has not already been published:
+
+```sh
+RUBYDB_RELEASE_VERSION=0.1.6 ruby scripts/release
+```
+
+On PowerShell, use:
+
+```powershell
+$env:RUBYDB_RELEASE_VERSION = "0.1.6"
+ruby scripts/release
+```
+
+The release script builds the gem and writes a checksum. Check the artifact
+locally before publishing:
+
+```sh
+gem specification pkg/rubydb-0.1.6.gem
+gem install pkg/rubydb-0.1.6.gem --local
+ruby -rrubydb -e 'puts RubyDB::VERSION'
+```
+
+Publishing requires a RubyGems API key or trusted publishing setup configured
+on the release machine. The local script publishes only when both the explicit
+publish flag and secret are present; never commit the secret:
+
+```sh
+RUBYDB_RELEASE_VERSION=0.1.6 \
+RUBYDB_PUBLISH=1 \
+GEM_HOST_API_KEY="YOUR_RUBYGEMS_API_KEY" \
+ruby scripts/release
+```
+
+On Windows PowerShell:
+
+```powershell
+$env:RUBYDB_RELEASE_VERSION = "0.1.6"
+$env:RUBYDB_PUBLISH = "1"
+$env:GEM_HOST_API_KEY = "YOUR_RUBYGEMS_API_KEY"
+ruby scripts/release
+```
+
+Prefer the repository’s signed GitHub Actions release workflow for a public
+release. Store signing keys and RubyGems secrets only in protected secret
+storage; do not put them in the repository or a checked-in `.env` file.
+
+Release the adapter separately when its version changes, update the changelog,
+tag the source commit, and publish the checksums and supported-version notes.
+
+## Publish the Python adapter to PyPI
+
+The Python adapter is a separate distribution named `rubydb-python`; publishing
+the Ruby gem does not publish this package. Build it from the adapter directory
+and validate both distribution formats before upload:
+
+```powershell
+cd adapters/python
+python -m pip install --upgrade build twine
+python -m build
+python -m twine check dist/*
+```
+
+Prefer PyPI Trusted Publishing from CI. For a local upload, use a short-lived,
+scope-limited PyPI token through the environment or Twine's prompt. Never
+commit a token:
+
+```powershell
+$env:TWINE_USERNAME = "__token__"
+$env:TWINE_PASSWORD = (Get-Clipboard).Trim()
+python -m twine upload dist/*
+Remove-Item Env:TWINE_PASSWORD
+```
+
+After upload, verify the package from a clean environment and run the live
+adapter tests against a RubyDB server:
+
+```powershell
+python -m venv .venv-clean
+.venv-clean\Scripts\Activate.ps1
+python -m pip install rubydb-python
+$env:RUBYDB_URL = "rubydbs://service_user:password@127.0.0.1:7432/rubydb"
+python -m unittest discover -s adapters/python/tests -v
+```
+
+The package provides DB-API 2.0 access to RubyDB server mode. It is not a
+PostgreSQL driver and does not make PostgreSQL SQL portable to RubyDB. Pin the
+adapter and server versions together, use TLS in production, and keep the
+application's migration and rollback procedure under version control.
+
+## Build and publish the Node adapter
+
+The Node adapter is a separate public npm package named `rubydb-node`. The
+literal `node/rubydb` is not a valid npm name because npm reserves `/` for
+scoped packages such as `@scope/package`.
+
+```powershell
+cd adapters/rubydb
+npm ci
+npm test
+npm run publish:check
+npm publish --access public
+```
+
+Use npm Trusted Publishing from CI or a protected npm token. Never commit an
+`.npmrc` containing credentials. The package's live test runs against a real
+RubyDB server when `RUBYDB_URL` is set:
+
+```powershell
+$env:RUBYDB_URL = "rubydb://rubydb@127.0.0.1:7432/rubydb"
+npm test
+```
+
+The package is a Node.js/TypeScript RubyDB client, not a PostgreSQL driver. Pin
+the npm client and RubyDB server versions together and validate the target
+application's SQL, retry, TLS, migration, backup, and failover behavior.
+
+## Deployment gate
+
+For a direct RubyDB production deployment, follow [lesson 5](05-rubydb-production-server.md)
+from top to bottom before this gate. For a massive Rails application, follow
+[lesson 6](06-postgresql-massive-apps.md) and keep RubyDB at a separate service
+boundary.
+
+Do not promote until all of these have an owner and a recorded result:
+
+* application tests pass against the production database topology;
+* migrations pass on empty and populated staging data;
+* a verified backup restores on another path or host;
+* load tests cover concurrency, timeouts, cancellation, and resource limits;
+* multi-process client/server tests cover restart and network failure;
+* failover and fencing behavior is validated if high availability is claimed;
+* TLS, secrets, least privilege, certificate rotation, and audit logging are
+  reviewed;
+* dashboards and alerts page an on-call person;
+* rollback, upgrade, and data-reconciliation procedures are rehearsed; and
+* the README and compatibility guide state what is supported and what is not.
+
+## A sensible first production architecture
+
+For a large Rails product, use managed PostgreSQL for the main application and
+deploy RubyDB only for an independently owned microservice whose workload has
+passed the lessons above. For a small internal or single-owner service, RubyDB
+server mode can be reasonable when its SQL, concurrency, recovery, and
+operational limits are accepted. Do not call either architecture universally
+compatible without workload evidence.
+
+## Final checkpoint
+
+The journey is complete when a new operator can deploy the exact release,
+verify a real query, observe health and capacity, restore data, and explain the
+rollback path without relying on the author’s laptop. Keep the evidence with
+the release and repeat the drills after major RubyDB, Rails, schema, or hosting
+changes.
+
+Continue using the repository’s [CLI guide](../docs/cli.md), [production
+operations guide](../docs/operations/production-guide.md), [Rails compatibility
+guide](../docs/rails/compatibility-guide.md), and [SQL compatibility
+guide](../docs/sql/compatibility-guide.md) as the detailed references.