rubydb 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.gitattributes +20 -0
- data/.github/ISSUE_TEMPLATE/bug.yml +0 -0
- data/.github/ISSUE_TEMPLATE/feature.yml +0 -0
- data/.github/ISSUE_TEMPLATE/performance.yml +0 -0
- data/.github/ISSUE_TEMPLATE/security.yml +0 -0
- data/.github/PULL_REQUEST_TEMPLATE.md +15 -0
- data/.github/dependabot.yml +0 -0
- data/.github/workflows/benchmark.yml +26 -0
- data/.github/workflows/compatibility.yml +63 -0
- data/.github/workflows/fuzz.yml +33 -0
- data/.github/workflows/lint.yml +21 -0
- data/.github/workflows/operations.yml +24 -0
- data/.github/workflows/production-validation.yml +111 -0
- data/.github/workflows/release.yml +77 -0
- data/.github/workflows/security.yml +37 -0
- data/.github/workflows/test.yml +26 -0
- data/.github/workflows/workload.yml +58 -0
- data/.gitignore +43 -0
- data/.rspec +4 -0
- data/.rubocop.yml +44 -0
- data/.ruby-version +1 -0
- data/.standard.yml +14 -0
- data/ARCHITECTURE.md +21 -0
- data/CHANGELOG.md +31 -0
- data/CODE_OF_CONDUCT.md +13 -0
- data/CONTRIBUTING.md +29 -0
- data/GOVERNANCE.md +16 -0
- data/Gemfile +17 -0
- data/Gemfile.lock +71 -0
- data/LICENSE +60 -0
- data/README.md +162 -0
- data/ROADMAP.md +27 -0
- data/Rakefile +71 -0
- data/SECURITY.md +54 -0
- data/SUPPORT.md +14 -0
- data/adapters/activerecord/Gemfile +11 -0
- data/adapters/activerecord/README.md +9 -0
- data/adapters/activerecord/lib/active_record/connection_adapters/rubydb_adapter.rb +887 -0
- data/adapters/activerecord/rubydb-activerecord.gemspec +27 -0
- data/adapters/activerecord/spec/rubydb_adapter_integration_spec.rb +151 -0
- data/adapters/ruby/README.md +18 -0
- data/adapters/sequel/README.md +11 -0
- data/bin/rubydb +19 -0
- data/bin/rubydb-server +126 -0
- data/config/development.yml +0 -0
- data/config/monitoring/prometheus-alerts.yml +39 -0
- data/config/production.yml +36 -0
- data/config/test.yml +0 -0
- data/docs/README.md +71 -0
- data/docs/architecture/concurrency.md +14 -0
- data/docs/architecture/current-state.md +125 -0
- data/docs/architecture/execution-engine.md +25 -0
- data/docs/architecture/indexes.md +19 -0
- data/docs/architecture/mvcc.md +19 -0
- data/docs/architecture/overview.md +13 -0
- data/docs/architecture/pages.md +11 -0
- data/docs/architecture/production-roadmap.md +82 -0
- data/docs/architecture/query-planner.md +20 -0
- data/docs/architecture/recovery.md +18 -0
- data/docs/architecture/sql-engine.md +12 -0
- data/docs/architecture/storage-engine.md +14 -0
- data/docs/architecture/transactions.md +10 -0
- data/docs/architecture/wal.md +28 -0
- data/docs/cli-cheatsheet.md +98 -0
- data/docs/cli.md +275 -0
- data/docs/contributing/architecture.md +9 -0
- data/docs/contributing/benchmarking.md +14 -0
- data/docs/contributing/development.md +16 -0
- data/docs/contributing/release-process.md +49 -0
- data/docs/contributing/testing.md +16 -0
- data/docs/debugging.md +229 -0
- data/docs/developer/branching.md +10 -0
- data/docs/developer/database-diff.md +10 -0
- data/docs/developer/local-development.md +17 -0
- data/docs/developer/snapshots.md +9 -0
- data/docs/developer/temporal-data.md +10 -0
- data/docs/developer-guide.md +297 -0
- data/docs/getting-started/first-database.md +16 -0
- data/docs/getting-started/first-query.md +13 -0
- data/docs/getting-started/installation.md +19 -0
- data/docs/getting-started/local-to-production.md +300 -0
- data/docs/getting-started/quickstart.md +17 -0
- data/docs/getting-started/rails.md +16 -0
- data/docs/hardening_backlog.md +93 -0
- data/docs/lessons-learned.md +112 -0
- data/docs/operations/backups.md +33 -0
- data/docs/operations/disaster-recovery.md +31 -0
- data/docs/operations/failover.md +30 -0
- data/docs/operations/monitoring.md +25 -0
- data/docs/operations/production-guide.md +295 -0
- data/docs/operations/production-runbook.md +45 -0
- data/docs/operations/replication.md +33 -0
- data/docs/operations/restore.md +6 -0
- data/docs/operations/runbook.md +34 -0
- data/docs/operations/upgrades.md +14 -0
- data/docs/operations/workload-testing.md +17 -0
- data/docs/production-readiness.md +118 -0
- data/docs/production_validation.md +150 -0
- data/docs/rails/active-record.md +11 -0
- data/docs/rails/compatibility-guide.md +90 -0
- data/docs/rails/database-yml.md +92 -0
- data/docs/rails/installation.md +17 -0
- data/docs/rails/migrations.md +17 -0
- data/docs/rails/production.md +82 -0
- data/docs/rails/troubleshooting.md +18 -0
- data/docs/release.md +59 -0
- data/docs/server/architecture.md +10 -0
- data/docs/server/authentication.md +10 -0
- data/docs/server/configuration.md +16 -0
- data/docs/server/connection-pooling.md +10 -0
- data/docs/server/deployment.md +10 -0
- data/docs/server/protocol.md +12 -0
- data/docs/sql/compatibility-guide.md +82 -0
- data/docs/sql/compatibility.md +39 -0
- data/docs/sql/data-types.md +10 -0
- data/docs/sql/functions.md +9 -0
- data/docs/sql/joins.md +9 -0
- data/docs/sql/operators.md +9 -0
- data/docs/sql/sqlite-compatibility.md +21 -0
- data/docs/sql/syntax.md +10 -0
- data/docs/sql/transactions.md +10 -0
- data/docs/troubleshooting.md +244 -0
- data/exe/rubydb +23 -0
- data/lib/rubydb/backup/archive.rb +334 -0
- data/lib/rubydb/backup/backup.rb +401 -0
- data/lib/rubydb/backup/incremental.rb +353 -0
- data/lib/rubydb/backup/restore.rb +290 -0
- data/lib/rubydb/backup/snapshot.rb +267 -0
- data/lib/rubydb/backup/verification.rb +279 -0
- data/lib/rubydb/branching/branch.rb +181 -0
- data/lib/rubydb/branching/branch_manager.rb +311 -0
- data/lib/rubydb/branching/branch_metadata.rb +140 -0
- data/lib/rubydb/branching/checkout.rb +166 -0
- data/lib/rubydb/branching/copy_on_write.rb +272 -0
- data/lib/rubydb/branching/diff.rb +138 -0
- data/lib/rubydb/branching/merge.rb +285 -0
- data/lib/rubydb/build_info.rb +15 -0
- data/lib/rubydb/catalog/catalog.rb +391 -0
- data/lib/rubydb/catalog/column.rb +112 -0
- data/lib/rubydb/catalog/constraint.rb +180 -0
- data/lib/rubydb/catalog/database.rb +184 -0
- data/lib/rubydb/catalog/index.rb +97 -0
- data/lib/rubydb/catalog/schema.rb +103 -0
- data/lib/rubydb/catalog/sequence.rb +90 -0
- data/lib/rubydb/catalog/system_catalog.rb +698 -0
- data/lib/rubydb/catalog/table.rb +178 -0
- data/lib/rubydb/catalog/trigger.rb +102 -0
- data/lib/rubydb/catalog/view.rb +66 -0
- data/lib/rubydb/cli/application.rb +163 -0
- data/lib/rubydb/cli/commands/backup.rb +81 -0
- data/lib/rubydb/cli/commands/branch.rb +72 -0
- data/lib/rubydb/cli/commands/checkout.rb +54 -0
- data/lib/rubydb/cli/commands/create.rb +58 -0
- data/lib/rubydb/cli/commands/diff.rb +77 -0
- data/lib/rubydb/cli/commands/doctor.rb +74 -0
- data/lib/rubydb/cli/commands/drop.rb +57 -0
- data/lib/rubydb/cli/commands/init.rb +102 -0
- data/lib/rubydb/cli/commands/inspect.rb +95 -0
- data/lib/rubydb/cli/commands/merge.rb +63 -0
- data/lib/rubydb/cli/commands/migrate.rb +62 -0
- data/lib/rubydb/cli/commands/restart.rb +39 -0
- data/lib/rubydb/cli/commands/restore.rb +121 -0
- data/lib/rubydb/cli/commands/shell.rb +365 -0
- data/lib/rubydb/cli/commands/snapshot.rb +79 -0
- data/lib/rubydb/cli/commands/start.rb +82 -0
- data/lib/rubydb/cli/commands/status.rb +92 -0
- data/lib/rubydb/cli/commands/stop.rb +47 -0
- data/lib/rubydb/cli/commands/vacuum.rb +58 -0
- data/lib/rubydb/cli/formatter.rb +221 -0
- data/lib/rubydb/cli/output.rb +168 -0
- data/lib/rubydb/client/client.rb +304 -0
- data/lib/rubydb/client/connection.rb +415 -0
- data/lib/rubydb/client/connection_pool.rb +168 -0
- data/lib/rubydb/client/connection_url.rb +96 -0
- data/lib/rubydb/client/prepared_statement.rb +60 -0
- data/lib/rubydb/client/result.rb +123 -0
- data/lib/rubydb/client/statement.rb +52 -0
- data/lib/rubydb/client/transaction.rb +130 -0
- data/lib/rubydb/concurrency/concurrency.rb +19 -0
- data/lib/rubydb/concurrency/deadlock_detector.rb +150 -0
- data/lib/rubydb/concurrency/latch.rb +101 -0
- data/lib/rubydb/concurrency/lock_graph.rb +165 -0
- data/lib/rubydb/concurrency/mutex.rb +183 -0
- data/lib/rubydb/concurrency/rw_lock.rb +180 -0
- data/lib/rubydb/concurrency/scheduler.rb +250 -0
- data/lib/rubydb/concurrency/worker_pool.rb +143 -0
- data/lib/rubydb/configuration/config.rb +170 -0
- data/lib/rubydb/configuration/defaults.rb +179 -0
- data/lib/rubydb/configuration/environment.rb +152 -0
- data/lib/rubydb/configuration/parser.rb +185 -0
- data/lib/rubydb/configuration/validation.rb +221 -0
- data/lib/rubydb/constants.rb +74 -0
- data/lib/rubydb/constraints/check.rb +181 -0
- data/lib/rubydb/constraints/constraint.rb +101 -0
- data/lib/rubydb/constraints/foreign_key.rb +130 -0
- data/lib/rubydb/constraints/not_null.rb +64 -0
- data/lib/rubydb/constraints/primary_key.rb +99 -0
- data/lib/rubydb/constraints/unique.rb +108 -0
- data/lib/rubydb/constraints/validator.rb +350 -0
- data/lib/rubydb/errors/authentication_error.rb +10 -0
- data/lib/rubydb/errors/authorization_error.rb +23 -0
- data/lib/rubydb/errors/client_error.rb +10 -0
- data/lib/rubydb/errors/configuration_error.rb +10 -0
- data/lib/rubydb/errors/connection_error.rb +10 -0
- data/lib/rubydb/errors/constraint_error.rb +23 -0
- data/lib/rubydb/errors/corruption_error.rb +10 -0
- data/lib/rubydb/errors/database_error.rb +10 -0
- data/lib/rubydb/errors/error.rb +20 -0
- data/lib/rubydb/errors/execution_error.rb +10 -0
- data/lib/rubydb/errors/parser_error.rb +10 -0
- data/lib/rubydb/errors/recovery_error.rb +10 -0
- data/lib/rubydb/errors/replication_error.rb +10 -0
- data/lib/rubydb/errors/server_error.rb +6 -0
- data/lib/rubydb/errors/storage_error.rb +10 -0
- data/lib/rubydb/errors/transaction_error.rb +10 -0
- data/lib/rubydb/execution/aggregate_executor.rb +138 -0
- data/lib/rubydb/execution/delete_executor.rb +112 -0
- data/lib/rubydb/execution/distinct_executor.rb +135 -0
- data/lib/rubydb/execution/executor.rb +1192 -0
- data/lib/rubydb/execution/expression.rb +193 -0
- data/lib/rubydb/execution/index_scan.rb +142 -0
- data/lib/rubydb/execution/insert_executor.rb +217 -0
- data/lib/rubydb/execution/join_executor.rb +249 -0
- data/lib/rubydb/execution/limit_executor.rb +85 -0
- data/lib/rubydb/execution/optimizer.rb +215 -0
- data/lib/rubydb/execution/plan.rb +361 -0
- data/lib/rubydb/execution/planner.rb +546 -0
- data/lib/rubydb/execution/predicate.rb +235 -0
- data/lib/rubydb/execution/scan.rb +49 -0
- data/lib/rubydb/execution/sequential_scan.rb +63 -0
- data/lib/rubydb/execution/sort_executor.rb +185 -0
- data/lib/rubydb/execution/update_executor.rb +162 -0
- data/lib/rubydb/functions/aggregate.rb +70 -0
- data/lib/rubydb/functions/date_functions.rb +278 -0
- data/lib/rubydb/functions/function.rb +85 -0
- data/lib/rubydb/functions/json_functions.rb +215 -0
- data/lib/rubydb/functions/numeric_functions.rb +346 -0
- data/lib/rubydb/functions/scalar.rb +52 -0
- data/lib/rubydb/functions/string_functions.rb +383 -0
- data/lib/rubydb/functions/system_functions.rb +246 -0
- data/lib/rubydb/history/as_of.rb +238 -0
- data/lib/rubydb/history/change.rb +105 -0
- data/lib/rubydb/history/history.rb +131 -0
- data/lib/rubydb/history/history_manager.rb +229 -0
- data/lib/rubydb/history/temporal_query.rb +202 -0
- data/lib/rubydb/history/timeline.rb +144 -0
- data/lib/rubydb/indexes/btree.rb +186 -0
- data/lib/rubydb/indexes/btree_cursor.rb +258 -0
- data/lib/rubydb/indexes/btree_node.rb +385 -0
- data/lib/rubydb/indexes/hash_index.rb +150 -0
- data/lib/rubydb/indexes/index.rb +71 -0
- data/lib/rubydb/indexes/index_manager.rb +406 -0
- data/lib/rubydb/indexes/index_scan.rb +470 -0
- data/lib/rubydb/migrations/migration.rb +254 -0
- data/lib/rubydb/migrations/migration_lock.rb +146 -0
- data/lib/rubydb/migrations/migration_manager.rb +176 -0
- data/lib/rubydb/migrations/migration_version.rb +71 -0
- data/lib/rubydb/migrations/schema_diff.rb +211 -0
- data/lib/rubydb/migrations/schema_version.rb +64 -0
- data/lib/rubydb/monitoring/events.rb +160 -0
- data/lib/rubydb/monitoring/health.rb +222 -0
- data/lib/rubydb/monitoring/logger.rb +193 -0
- data/lib/rubydb/monitoring/metrics.rb +359 -0
- data/lib/rubydb/monitoring/performance.rb +176 -0
- data/lib/rubydb/monitoring/statistics.rb +170 -0
- data/lib/rubydb/mvcc/garbage_collector.rb +199 -0
- data/lib/rubydb/mvcc/mvcc.rb +16 -0
- data/lib/rubydb/mvcc/snapshot.rb +147 -0
- data/lib/rubydb/mvcc/vacuum.rb +180 -0
- data/lib/rubydb/mvcc/version.rb +106 -0
- data/lib/rubydb/mvcc/version_store.rb +398 -0
- data/lib/rubydb/mvcc/visibility.rb +109 -0
- data/lib/rubydb/protocol/capabilities.rb +125 -0
- data/lib/rubydb/protocol/decoder.rb +145 -0
- data/lib/rubydb/protocol/encoder.rb +136 -0
- data/lib/rubydb/protocol/handshake.rb +305 -0
- data/lib/rubydb/protocol/message.rb +121 -0
- data/lib/rubydb/protocol/protocol.rb +277 -0
- data/lib/rubydb/protocol/version.rb +54 -0
- data/lib/rubydb/rails/adapter.rb +239 -0
- data/lib/rubydb/rails/connection.rb +314 -0
- data/lib/rubydb/rails/database_statements.rb +122 -0
- data/lib/rubydb/rails/migration.rb +131 -0
- data/lib/rubydb/rails/quoting.rb +109 -0
- data/lib/rubydb/rails/result.rb +120 -0
- data/lib/rubydb/rails/schema_statements.rb +339 -0
- data/lib/rubydb/rails/transaction.rb +105 -0
- data/lib/rubydb/rails/type.rb +126 -0
- data/lib/rubydb/recovery/checkpoint.rb +257 -0
- data/lib/rubydb/recovery/consistency.rb +467 -0
- data/lib/rubydb/recovery/corruption_detector.rb +5 -0
- data/lib/rubydb/recovery/crash_recovery.rb +387 -0
- data/lib/rubydb/recovery/recovery_manager.rb +206 -0
- data/lib/rubydb/recovery/redo.rb +237 -0
- data/lib/rubydb/recovery/undo.rb +206 -0
- data/lib/rubydb/replication/failover.rb +5 -0
- data/lib/rubydb/replication/fencing.rb +63 -0
- data/lib/rubydb/replication/primary.rb +450 -0
- data/lib/rubydb/replication/replica.rb +384 -0
- data/lib/rubydb/replication/replication_log.rb +200 -0
- data/lib/rubydb/replication/replication_manager.rb +308 -0
- data/lib/rubydb/replication/replication_slot.rb +295 -0
- data/lib/rubydb/replication/replication_stream.rb +201 -0
- data/lib/rubydb/rubydb.rb +560 -0
- data/lib/rubydb/security/access_control.rb +254 -0
- data/lib/rubydb/security/audit_log.rb +213 -0
- data/lib/rubydb/security/authentication.rb +302 -0
- data/lib/rubydb/security/authorization.rb +282 -0
- data/lib/rubydb/security/credentials.rb +196 -0
- data/lib/rubydb/security/password.rb +215 -0
- data/lib/rubydb/security/permissions.rb +74 -0
- data/lib/rubydb/security/role.rb +101 -0
- data/lib/rubydb/security/user.rb +86 -0
- data/lib/rubydb/server/connection.rb +366 -0
- data/lib/rubydb/server/connection_pool.rb +193 -0
- data/lib/rubydb/server/lifecycle.rb +228 -0
- data/lib/rubydb/server/listener.rb +136 -0
- data/lib/rubydb/server/request_handler.rb +276 -0
- data/lib/rubydb/server/server.rb +364 -0
- data/lib/rubydb/server/session.rb +369 -0
- data/lib/rubydb/server/worker.rb +210 -0
- data/lib/rubydb/server/worker_pool.rb +168 -0
- data/lib/rubydb/sql/ast/alter_table.rb +169 -0
- data/lib/rubydb/sql/ast/begin_transaction.rb +47 -0
- data/lib/rubydb/sql/ast/commit.rb +37 -0
- data/lib/rubydb/sql/ast/constraint.rb +83 -0
- data/lib/rubydb/sql/ast/create_database.rb +41 -0
- data/lib/rubydb/sql/ast/create_index.rb +61 -0
- data/lib/rubydb/sql/ast/create_schema.rb +52 -0
- data/lib/rubydb/sql/ast/create_table.rb +187 -0
- data/lib/rubydb/sql/ast/delete.rb +54 -0
- data/lib/rubydb/sql/ast/drop_database.rb +41 -0
- data/lib/rubydb/sql/ast/drop_index.rb +41 -0
- data/lib/rubydb/sql/ast/drop_schema.rb +49 -0
- data/lib/rubydb/sql/ast/drop_table.rb +49 -0
- data/lib/rubydb/sql/ast/explain.rb +64 -0
- data/lib/rubydb/sql/ast/expression.rb +604 -0
- data/lib/rubydb/sql/ast/insert.rb +82 -0
- data/lib/rubydb/sql/ast/node.rb +42 -0
- data/lib/rubydb/sql/ast/rollback.rb +63 -0
- data/lib/rubydb/sql/ast/savepoint.rb +59 -0
- data/lib/rubydb/sql/ast/select.rb +88 -0
- data/lib/rubydb/sql/ast/set_operation.rb +20 -0
- data/lib/rubydb/sql/ast/trigger.rb +29 -0
- data/lib/rubydb/sql/ast/update.rb +88 -0
- data/lib/rubydb/sql/ast/vacuum.rb +19 -0
- data/lib/rubydb/sql/ast/view.rb +32 -0
- data/lib/rubydb/sql/ast/with.rb +32 -0
- data/lib/rubydb/sql/grammar.rb +86 -0
- data/lib/rubydb/sql/keywords.rb +156 -0
- data/lib/rubydb/sql/lexer.rb +214 -0
- data/lib/rubydb/sql/operators.rb +100 -0
- data/lib/rubydb/sql/parser.rb +1185 -0
- data/lib/rubydb/sql/planner/analyzer.rb +302 -0
- data/lib/rubydb/sql/planner/binder.rb +550 -0
- data/lib/rubydb/sql/planner/type_checker.rb +431 -0
- data/lib/rubydb/sql/token.rb +210 -0
- data/lib/rubydb/storage/buffer_frame.rb +44 -0
- data/lib/rubydb/storage/buffer_pool.rb +155 -0
- data/lib/rubydb/storage/database_lock.rb +74 -0
- data/lib/rubydb/storage/deserializer.rb +342 -0
- data/lib/rubydb/storage/engine.rb +2338 -0
- data/lib/rubydb/storage/file_manager.rb +187 -0
- data/lib/rubydb/storage/free_space_map.rb +81 -0
- data/lib/rubydb/storage/page.rb +94 -0
- data/lib/rubydb/storage/page_allocator.rb +855 -0
- data/lib/rubydb/storage/page_header.rb +67 -0
- data/lib/rubydb/storage/page_manager.rb +131 -0
- data/lib/rubydb/storage/record.rb +58 -0
- data/lib/rubydb/storage/row.rb +78 -0
- data/lib/rubydb/storage/serializer.rb +51 -0
- data/lib/rubydb/storage/storage_layout.rb +151 -0
- data/lib/rubydb/storage/storage_manager.rb +114 -0
- data/lib/rubydb/storage/tuple.rb +461 -0
- data/lib/rubydb/storage/visibility_map.rb +984 -0
- data/lib/rubydb/transactions/commit_manager.rb +220 -0
- data/lib/rubydb/transactions/isolation.rb +98 -0
- data/lib/rubydb/transactions/lock.rb +76 -0
- data/lib/rubydb/transactions/lock_manager.rb +362 -0
- data/lib/rubydb/transactions/savepoint.rb +143 -0
- data/lib/rubydb/transactions/transaction.rb +215 -0
- data/lib/rubydb/transactions/transaction_id.rb +84 -0
- data/lib/rubydb/transactions/transaction_log.rb +257 -0
- data/lib/rubydb/transactions/transaction_manager.rb +435 -0
- data/lib/rubydb/types/bigint.rb +36 -0
- data/lib/rubydb/types/blob.rb +37 -0
- data/lib/rubydb/types/boolean.rb +34 -0
- data/lib/rubydb/types/date.rb +39 -0
- data/lib/rubydb/types/decimal.rb +48 -0
- data/lib/rubydb/types/float.rb +34 -0
- data/lib/rubydb/types/integer.rb +36 -0
- data/lib/rubydb/types/json.rb +41 -0
- data/lib/rubydb/types/null.rb +34 -0
- data/lib/rubydb/types/smallint.rb +36 -0
- data/lib/rubydb/types/text.rb +37 -0
- data/lib/rubydb/types/time.rb +46 -0
- data/lib/rubydb/types/timestamp.rb +39 -0
- data/lib/rubydb/types/type.rb +119 -0
- data/lib/rubydb/types/uuid.rb +47 -0
- data/lib/rubydb/types/varchar.rb +37 -0
- data/lib/rubydb/version.rb +33 -0
- data/lib/rubydb/wal/archive.rb +193 -0
- data/lib/rubydb/wal/checkpoint.rb +183 -0
- data/lib/rubydb/wal/lsn.rb +94 -0
- data/lib/rubydb/wal/reader.rb +260 -0
- data/lib/rubydb/wal/record.rb +105 -0
- data/lib/rubydb/wal/segment.rb +193 -0
- data/lib/rubydb/wal/wal.rb +452 -0
- data/lib/rubydb/wal/writer.rb +236 -0
- data/lib/rubydb.rb +7 -0
- data/packaging/docker/Dockerfile +36 -0
- data/packaging/docker/docker-compose.failover.yml +43 -0
- data/packaging/docker/entrypoint.sh +15 -0
- data/packaging/homebrew/rubydb.rb +19 -0
- data/packaging/systemd/rubydb.service +22 -0
- data/rubydb.gemspec +57 -0
- data/scripts/benchmark +7 -0
- data/scripts/durability_drill +37 -0
- data/scripts/format +0 -0
- data/scripts/fuzz +63 -0
- data/scripts/lint +0 -0
- data/scripts/release +50 -0
- data/scripts/release_check +43 -0
- data/scripts/replication_failover_drill +250 -0
- data/scripts/replication_network_failover_drill +255 -0
- data/scripts/restore_drill +45 -0
- data/scripts/security +0 -0
- data/scripts/setup +0 -0
- data/scripts/test +0 -0
- data/scripts/test-chaos +0 -0
- data/scripts/test-crash +0 -0
- data/scripts/test-integration +0 -0
- data/scripts/test-unit +0 -0
- data/visibility_map.json +1 -0
- metadata +538 -0
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# Releasing RubyDB to RubyGems
|
|
2
|
+
|
|
3
|
+
1. Update `CHANGELOG.md`, bump the semantic version, and confirm the Ruby/Rails support matrix for the release. The version must have a top-level `## <version>` changelog entry; the release preflight fails closed when it is missing.
|
|
4
|
+
2. Run the complete verification suite and workload test. The release workflow independently builds and validates the gem on a version tag.
|
|
5
|
+
3. Create a RubyGems API key with the minimum scope needed to push this gem. Store it as the `RUBYGEMS_API_KEY` GitHub Actions secret or in RubyGems' protected credentials file; never commit it.
|
|
6
|
+
4. Create and push an annotated `v<version>` tag. The compatibility workflow
|
|
7
|
+
must pass its SimpleCov line-coverage gate, and the release workflow
|
|
8
|
+
publishes only when that tag's version matches `RubyDB::VERSION` and the
|
|
9
|
+
secret is available.
|
|
10
|
+
5. To build, verify, and publish manually:
|
|
11
|
+
|
|
12
|
+
```sh
|
|
13
|
+
RUBYDB_PUBLISH=1 GEM_HOST_API_KEY=<RubyGems API key> ruby scripts/release
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
Without `RUBYDB_PUBLISH=1`, `ruby scripts/release` only rebuilds and verifies the gem/checksum. Manual publication must also set `RUBYDB_RELEASE_VERSION=<version>`; tagged CI obtains that identity from `GITHUB_REF_NAME`.
|
|
17
|
+
|
|
18
|
+
The tag workflow also creates a signed GitHub build-provenance attestation for
|
|
19
|
+
the exact gem artifact. Verify that attestation in the repository's Actions or
|
|
20
|
+
Releases UI before distributing the package; the SHA-512 file remains available
|
|
21
|
+
for an independent byte-for-byte check.
|
|
22
|
+
|
|
23
|
+
For the tag release workflow, provision base64-encoded
|
|
24
|
+
`RUBYDB_GEM_SIGNING_KEY_B64` and `RUBYDB_GEM_CERT_B64` repository secrets. The
|
|
25
|
+
release job materializes them only on the ephemeral runner and passes the
|
|
26
|
+
protected paths to `gem build`; it never stores them in the repository. GitHub
|
|
27
|
+
release notes are generated from the tag history after publication, so review
|
|
28
|
+
the generated release before announcing it.
|
|
29
|
+
|
|
30
|
+
The tag workflow fails closed when the signing secrets or RubyGems publication
|
|
31
|
+
key are missing. Local unsigned builds remain available through `rake build`
|
|
32
|
+
and are not publication artifacts.
|
|
33
|
+
|
|
34
|
+
Pull requests also run the supported Ruby 3.3/3.4 matrix on Linux, macOS, and
|
|
35
|
+
Windows, plus the ActiveRecord adapter suite on Rails 7.1, 7.2, and 8.0. A scheduled bounded fuzz job runs
|
|
36
|
+
the SQL parser, WAL, storage, transaction, and query-engine fuzzers. Increase
|
|
37
|
+
`RUBYDB_FUZZ_ITERATIONS` locally when investigating a failure, retaining the
|
|
38
|
+
reported `RUBYDB_FUZZ_SEED` for reproduction.
|
|
39
|
+
|
|
40
|
+
The scheduled operations workflow runs `ruby scripts/restore_drill`, which
|
|
41
|
+
creates a live backup, verifies its manifest/checksums, restores it into a
|
|
42
|
+
separate directory, and reopens the restored database before succeeding.
|
|
43
|
+
|
|
44
|
+
After publication, install the exact released version in a clean environment and run a smoke test:
|
|
45
|
+
|
|
46
|
+
```sh
|
|
47
|
+
gem install rubydb --version 0.1.0
|
|
48
|
+
ruby -e "require 'rubydb'; puts RubyDB::VERSION"
|
|
49
|
+
```
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Testing guide
|
|
2
|
+
|
|
3
|
+
The test layers are complementary:
|
|
4
|
+
|
|
5
|
+
- unit specs validate isolated algorithms and invariants;
|
|
6
|
+
- integration specs run the real parser, planner, executor, storage, server,
|
|
7
|
+
client, adapter, or replication path;
|
|
8
|
+
- chaos/fault specs inject write, crash, corruption, and network failures;
|
|
9
|
+
- workload scripts measure concurrency, latency, cancellation, capacity, and
|
|
10
|
+
durable reopen behavior;
|
|
11
|
+
- CI covers supported Ruby/Rails/OS combinations, security scans, fuzzing, and
|
|
12
|
+
release preflight.
|
|
13
|
+
|
|
14
|
+
Run `bundle exec rspec` for the complete local gate. Preserve the random seed
|
|
15
|
+
when reproducing failures. A passing local suite does not replace hosted
|
|
16
|
+
multi-host, physical-filesystem, or independent security validation.
|
data/docs/debugging.md
ADDED
|
@@ -0,0 +1,229 @@
|
|
|
1
|
+
# Debugging RubyDB
|
|
2
|
+
|
|
3
|
+
This playbook is for finding defects without destroying the evidence needed
|
|
4
|
+
to recover data. Use it in development and staging first. Production
|
|
5
|
+
diagnostics must follow the application’s privacy, access, and change-control
|
|
6
|
+
policies.
|
|
7
|
+
|
|
8
|
+
## Debugging principles
|
|
9
|
+
|
|
10
|
+
1. Reproduce on a copy or a new temporary directory.
|
|
11
|
+
2. Identify the layer: API, session, protocol, parser, planner, executor,
|
|
12
|
+
transaction, storage, WAL, filesystem, or deployment.
|
|
13
|
+
3. Reduce the input while preserving the failure.
|
|
14
|
+
4. Record versions, configuration names, seed, timing, and request IDs.
|
|
15
|
+
5. Add a regression test before changing behavior.
|
|
16
|
+
|
|
17
|
+
A database that returns an error is often safer than one that silently repairs,
|
|
18
|
+
retries, or acknowledges unknown state. Keep fail-closed behavior intact while
|
|
19
|
+
diagnosing.
|
|
20
|
+
|
|
21
|
+
## Reproducible diagnostic baseline
|
|
22
|
+
|
|
23
|
+
```sh
|
|
24
|
+
git rev-parse HEAD
|
|
25
|
+
ruby -v
|
|
26
|
+
bundle exec ruby -Ilib exe/rubydb --version
|
|
27
|
+
bundle exec rspec --format progress
|
|
28
|
+
git diff --check
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
Use a fresh directory for each reproduction. Save the command and, for random
|
|
32
|
+
or concurrent tests, the seed:
|
|
33
|
+
|
|
34
|
+
```sh
|
|
35
|
+
RUBYDB_FUZZ_SEED=12345 RUBYDB_FUZZ_ITERATIONS=1000 ruby scripts/fuzz
|
|
36
|
+
bundle exec rspec spec/path/to/failing_spec.rb:42 --format doc
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
## CLI inspection
|
|
40
|
+
|
|
41
|
+
The CLI is the first diagnostic interface:
|
|
42
|
+
|
|
43
|
+
```sh
|
|
44
|
+
bundle exec ruby -Ilib exe/rubydb status --config config/rubydb.yml
|
|
45
|
+
bundle exec ruby -Ilib exe/rubydb doctor --config config/rubydb.yml
|
|
46
|
+
bundle exec ruby -Ilib exe/rubydb inspect --config config/rubydb.yml
|
|
47
|
+
bundle exec ruby -Ilib exe/rubydb shell --config config/rubydb.yml
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Use `status` for ownership, lifecycle, and health; `doctor` for non-destructive
|
|
51
|
+
checks and safe repairs; and `inspect` for metadata, schema, WAL, and storage
|
|
52
|
+
facts. Capture JSON output when the command supports it so an incident can be
|
|
53
|
+
compared over time. Read [CLI guide](cli.md) before using maintenance commands.
|
|
54
|
+
|
|
55
|
+
## Logging and safe verbosity
|
|
56
|
+
|
|
57
|
+
Use the normal structured logs first. Increase verbosity only for a bounded
|
|
58
|
+
reproduction. `--verbose` is useful for CLI execution, while `RUBYDB_DEBUG=1`
|
|
59
|
+
enables development backtraces in command wrappers. Debug logs can contain SQL
|
|
60
|
+
shapes, paths, identifiers, and timing; they must be access-controlled and
|
|
61
|
+
sanitized before sharing.
|
|
62
|
+
|
|
63
|
+
When adding logs, include stable fields such as request ID, transaction ID,
|
|
64
|
+
connection ID, LSN, operation, duration, and outcome. Never log passwords,
|
|
65
|
+
SCRAM secrets, bearer tokens, private keys, or unredacted customer values.
|
|
66
|
+
|
|
67
|
+
## Layer isolation
|
|
68
|
+
|
|
69
|
+
### API versus engine
|
|
70
|
+
|
|
71
|
+
Run the same operation through the direct Ruby API and the server/client path.
|
|
72
|
+
If only server mode fails, inspect protocol framing, authentication, session
|
|
73
|
+
state, serialization, and timeout handling. If both fail, reduce to engine SQL
|
|
74
|
+
and storage.
|
|
75
|
+
|
|
76
|
+
### Parser versus executor
|
|
77
|
+
|
|
78
|
+
First parse/bind a statement without committing data. Then execute it against a
|
|
79
|
+
minimal schema. A parser failure should identify token position and expected
|
|
80
|
+
construct. A binder failure should identify parameter count/type context. An
|
|
81
|
+
executor failure should preserve transaction rollback and identify the table,
|
|
82
|
+
index, or constraint involved.
|
|
83
|
+
|
|
84
|
+
### Planner versus semantics
|
|
85
|
+
|
|
86
|
+
Compare an optimized plan with a simple scan where possible. Verify row IDs,
|
|
87
|
+
duplicates, `NULL`, ordering, grouping, and snapshot visibility. Optimizer
|
|
88
|
+
changes require equivalence tests, not only performance numbers.
|
|
89
|
+
|
|
90
|
+
### Storage versus filesystem
|
|
91
|
+
|
|
92
|
+
Test the storage operation with a real temporary filesystem first. Then inject
|
|
93
|
+
failures at write, flush, rename, allocation, and close boundaries. A Ruby
|
|
94
|
+
exception around a filesystem call is not equivalent to a process termination
|
|
95
|
+
after the filesystem accepted the write.
|
|
96
|
+
|
|
97
|
+
## Transaction and deadlock diagnosis
|
|
98
|
+
|
|
99
|
+
Capture transaction lifecycle events: begin, snapshot, lock wait, lock grant,
|
|
100
|
+
savepoint, statement error, rollback, commit request, durable commit, and
|
|
101
|
+
connection close. For a deadlock, draw a wait-for graph from transaction IDs
|
|
102
|
+
and locks. Verify the selected victim’s before-images were applied and that
|
|
103
|
+
all locks and snapshots were released.
|
|
104
|
+
|
|
105
|
+
For a timeout or cancellation, answer three questions:
|
|
106
|
+
|
|
107
|
+
1. Did the server stop executing the request?
|
|
108
|
+
2. Did the transaction commit, roll back, or remain unknown?
|
|
109
|
+
3. Were connection, lock, snapshot, and temporary resources released?
|
|
110
|
+
|
|
111
|
+
Do not retry a write with an unknown outcome unless it is idempotent or the
|
|
112
|
+
application can query the request/transaction outcome.
|
|
113
|
+
|
|
114
|
+
## WAL and recovery diagnosis
|
|
115
|
+
|
|
116
|
+
Preserve the complete database directory, WAL, metadata, and service logs.
|
|
117
|
+
Record file sizes, modification times, checksums, checkpoint LSN, last
|
|
118
|
+
acknowledged LSN, and the process termination reason. Run inspection and
|
|
119
|
+
restore against a copy. Compare:
|
|
120
|
+
|
|
121
|
+
```text
|
|
122
|
+
checkpoint LSN <= durable WAL end LSN
|
|
123
|
+
last acknowledged commit <= durable WAL end LSN
|
|
124
|
+
restored checksum == manifest checksum
|
|
125
|
+
replayed transaction set == committed transaction set
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
For corruption, stop writes and escalate rather than repeatedly reopening the
|
|
129
|
+
original. For a compaction issue, compare row counts, indexes, checksums, and
|
|
130
|
+
reopen behavior before and after compaction on a copy.
|
|
131
|
+
|
|
132
|
+
## Wire and protocol debugging
|
|
133
|
+
|
|
134
|
+
Use a local test endpoint and sanitized packet/frame logging. Validate one
|
|
135
|
+
request at a time: handshake, authentication, capability negotiation, query,
|
|
136
|
+
result frames, cancellation, and close. Check frame length, request ID,
|
|
137
|
+
sequence, status, and error payload. Test partial reads and writes because a
|
|
138
|
+
single `read` or `write` is not guaranteed to transfer a complete frame.
|
|
139
|
+
|
|
140
|
+
For an in-flight cancellation race, log the request ID and server state at
|
|
141
|
+
cancel receipt, executor stop, transaction decision, and response emission.
|
|
142
|
+
The client must distinguish cancellation accepted, cancellation too late, and
|
|
143
|
+
unknown connection loss.
|
|
144
|
+
|
|
145
|
+
## Rails debugging
|
|
146
|
+
|
|
147
|
+
Enable Rails SQL logging in a non-production reproduction and redact bind
|
|
148
|
+
values before sharing. Compare the generated SQL with a direct RubyDB query.
|
|
149
|
+
For adapter bugs, create the smallest model and migration that shows the
|
|
150
|
+
problem, then test both a fresh schema and a populated table. Inspect:
|
|
151
|
+
|
|
152
|
+
* quoting and bind parameter order;
|
|
153
|
+
* transaction/savepoint boundaries;
|
|
154
|
+
* affected rows and last-insert ID;
|
|
155
|
+
* schema introspection and default values;
|
|
156
|
+
* pool checkout/checkin and leaked transactions; and
|
|
157
|
+
* exception class and retry behavior.
|
|
158
|
+
|
|
159
|
+
Use the Rails example under `examples/rails_app` as a smoke harness before
|
|
160
|
+
reproducing inside a large application.
|
|
161
|
+
|
|
162
|
+
## Ruby-level tools
|
|
163
|
+
|
|
164
|
+
For a focused local reproduction, Ruby’s standard tools are usually enough:
|
|
165
|
+
|
|
166
|
+
```sh
|
|
167
|
+
RUBYOPT="-d" bundle exec rspec spec/path/to/failing_spec.rb
|
|
168
|
+
bundle exec ruby -w -Ilib path/to/reproduction.rb
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
`TracePoint` can observe method calls and exceptions without modifying the
|
|
172
|
+
engine. Use it only in a short-lived reproduction because tracing changes
|
|
173
|
+
timing and can invalidate concurrency conclusions. Capture thread backtraces
|
|
174
|
+
when a process appears hung, and include the thread roles (acceptor, worker,
|
|
175
|
+
checkpoint, replication, application).
|
|
176
|
+
|
|
177
|
+
## Performance debugging
|
|
178
|
+
|
|
179
|
+
Separate CPU, lock, I/O, and queue time. Record p50/p95/p99 latency, throughput,
|
|
180
|
+
errors, WAL growth, checkpoint duration, memory, file descriptors, and active
|
|
181
|
+
transactions. Change one variable per run and repeat enough times to expose
|
|
182
|
+
variance. A faster benchmark with weaker durability is not an equivalent
|
|
183
|
+
optimization.
|
|
184
|
+
|
|
185
|
+
Use:
|
|
186
|
+
|
|
187
|
+
```sh
|
|
188
|
+
RUBYDB_BENCHMARK_ITERATIONS=100 ruby -Ilib benchmarks/basic_workload.rb
|
|
189
|
+
ruby benchmarks/concurrent_workload.rb
|
|
190
|
+
ruby scripts/production_soak
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
Keep benchmark artifacts out of commits unless they are intentional fixtures.
|
|
194
|
+
|
|
195
|
+
## Fuzzing and property failures
|
|
196
|
+
|
|
197
|
+
Save the seed, generated SQL/input, Ruby version, commit, and failing database
|
|
198
|
+
directory. Reduce the number of operations while preserving the seed. Check
|
|
199
|
+
the invariant: no crash, no invalid state, rollback equivalence, parser
|
|
200
|
+
round-trip, or index/table agreement. Add the minimized case as a deterministic
|
|
201
|
+
spec, then keep the fuzz run as a secondary guard.
|
|
202
|
+
|
|
203
|
+
## What not to do
|
|
204
|
+
|
|
205
|
+
Do not delete WAL or lock files, edit database bytes manually, disable checksum
|
|
206
|
+
validation, run repair on the only copy, force a replica promotion without
|
|
207
|
+
fencing, or publish sanitized logs that still contain secrets. These actions
|
|
208
|
+
can turn a diagnosable incident into irreversible data loss or a security
|
|
209
|
+
incident.
|
|
210
|
+
|
|
211
|
+
## Diagnostic report template
|
|
212
|
+
|
|
213
|
+
```text
|
|
214
|
+
Summary:
|
|
215
|
+
First observed (UTC):
|
|
216
|
+
RubyDB version/commit:
|
|
217
|
+
Ruby/Rails/OS/filesystem:
|
|
218
|
+
Topology and ownership mode:
|
|
219
|
+
Configuration names/checksum:
|
|
220
|
+
Command or request shape:
|
|
221
|
+
Request/transaction/LSN IDs:
|
|
222
|
+
Expected result:
|
|
223
|
+
Actual result:
|
|
224
|
+
Reproduction and seed:
|
|
225
|
+
Logs/metrics/checksums:
|
|
226
|
+
Actions already taken:
|
|
227
|
+
Data impact and current containment:
|
|
228
|
+
```
|
|
229
|
+
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
# Branching
|
|
2
|
+
|
|
3
|
+
RubyDB branches represent database snapshots and development lines in the
|
|
4
|
+
engine. Create and inspect branches with the CLI, verify the target state, and
|
|
5
|
+
retain a backup before merging or checking out a branch containing important
|
|
6
|
+
data.
|
|
7
|
+
|
|
8
|
+
Branch operations are not a substitute for backups or replication. Test branch
|
|
9
|
+
diff, merge, checkout, conflict handling, and reopen behavior before using them
|
|
10
|
+
in an operational workflow.
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
# Database diff
|
|
2
|
+
|
|
3
|
+
Database diff compares schema and supported state between database snapshots or
|
|
4
|
+
branches. Use it to review migration impact and branch changes before applying
|
|
5
|
+
them. A diff is evidence for review, not an automatic guarantee that every
|
|
6
|
+
application query remains compatible.
|
|
7
|
+
|
|
8
|
+
For production changes, take a verified backup, inspect the diff, apply it to a
|
|
9
|
+
staging restore with representative data, run migrations and smoke queries, and
|
|
10
|
+
retain the original for rollback.
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
# Local development
|
|
2
|
+
|
|
3
|
+
```sh
|
|
4
|
+
bundle install
|
|
5
|
+
bundle exec rspec
|
|
6
|
+
bundle exec rubocop
|
|
7
|
+
```
|
|
8
|
+
|
|
9
|
+
Use `tmp/` or `Dir.mktmpdir` for databases. The server/client examples are
|
|
10
|
+
appropriate when testing process boundaries. Do not open one embedded path from
|
|
11
|
+
multiple processes. Use the reported fuzz seed and commit when reproducing a
|
|
12
|
+
failure.
|
|
13
|
+
|
|
14
|
+
Start with the [developer guide](../developer-guide.md). For failures, use a
|
|
15
|
+
fresh copy, preserve WAL and metadata, run the narrowest spec first, then the
|
|
16
|
+
full suite. The [debugging playbook](../debugging.md) describes safe logging,
|
|
17
|
+
thread dumps, protocol isolation, and diagnostic reports.
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
# Snapshots
|
|
2
|
+
|
|
3
|
+
Snapshots capture a consistent RubyDB state for inspection, backup validation,
|
|
4
|
+
branching, or staging. Create snapshots through the supported CLI/API and keep
|
|
5
|
+
the snapshot with its metadata and checksum.
|
|
6
|
+
|
|
7
|
+
Validate a snapshot by opening it in a separate directory. Do not treat a
|
|
8
|
+
snapshot as a replacement for an independently verified backup or replication.
|
|
9
|
+
Keep retention and deletion policies explicit for production data.
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
# Temporal data
|
|
2
|
+
|
|
3
|
+
MVCC keeps row versions long enough for active transaction visibility and safe
|
|
4
|
+
vacuum. Readers observe a transaction-consistent view according to the selected
|
|
5
|
+
isolation behavior; uncommitted changes are not published to other readers.
|
|
6
|
+
|
|
7
|
+
Long-running transactions retain old versions and can increase storage. Monitor
|
|
8
|
+
transaction age and vacuum/compaction duration, and terminate or redesign stale
|
|
9
|
+
work before it affects the workload. Test temporal behavior with restart and
|
|
10
|
+
rollback, not only a single read.
|
|
@@ -0,0 +1,297 @@
|
|
|
1
|
+
# RubyDB developer guide
|
|
2
|
+
|
|
3
|
+
This is the implementation guide for contributors who need to change RubyDB
|
|
4
|
+
safely. It explains the repository, the request path, the storage invariants,
|
|
5
|
+
the test strategy, and the debugging workflow. Read it together with the
|
|
6
|
+
topic specifications under `spec/`; the specifications are the behavioral
|
|
7
|
+
contract, while this guide explains how the pieces fit together.
|
|
8
|
+
|
|
9
|
+
## 1. Scope and support promise
|
|
10
|
+
|
|
11
|
+
RubyDB is a Ruby-native relational database with two ownership modes:
|
|
12
|
+
|
|
13
|
+
* embedded mode, where one Ruby process owns a database directory; and
|
|
14
|
+
* server mode, where the server owns the directory and application processes
|
|
15
|
+
use the client protocol.
|
|
16
|
+
|
|
17
|
+
The supported surface is the behavior exercised by the test suite and listed
|
|
18
|
+
in [SQL compatibility](sql/compatibility.md). Similar syntax does not imply
|
|
19
|
+
complete PostgreSQL, MySQL, or SQLite compatibility. A change that broadens
|
|
20
|
+
syntax must also specify its semantics, errors, types, transactions, and
|
|
21
|
+
adapter behavior.
|
|
22
|
+
|
|
23
|
+
## 2. Repository orientation
|
|
24
|
+
|
|
25
|
+
The important directories are:
|
|
26
|
+
|
|
27
|
+
| Path | Responsibility |
|
|
28
|
+
| --- | --- |
|
|
29
|
+
| `lib/rubydb` | Engine, SQL, storage, transactions, server, client, adapters |
|
|
30
|
+
| `spec` | Unit, integration, protocol, SQL, recovery, and adapter contracts |
|
|
31
|
+
| `benchmarks` | Repeatable throughput and concurrency measurements |
|
|
32
|
+
| `scripts` | Soak tests, durability drills, fuzzing, and release checks |
|
|
33
|
+
| `config` | Example configuration and monitoring rules |
|
|
34
|
+
| `examples` | Runnable Ruby and Rails applications |
|
|
35
|
+
| `docs` | User, operator, developer, and compatibility documentation |
|
|
36
|
+
| `packaging` | Container and deployment examples |
|
|
37
|
+
| `exe` and `bin` | Public command-line entry points |
|
|
38
|
+
|
|
39
|
+
Start with `lib/rubydb.rb` and follow the public API into the engine. For a
|
|
40
|
+
feature, locate the nearest existing spec before editing implementation code.
|
|
41
|
+
Avoid adding a second abstraction when an existing subsystem already owns the
|
|
42
|
+
invariant.
|
|
43
|
+
|
|
44
|
+
## 3. Local setup
|
|
45
|
+
|
|
46
|
+
Use a supported Ruby version from the project CI matrix and install the locked
|
|
47
|
+
dependencies:
|
|
48
|
+
|
|
49
|
+
```sh
|
|
50
|
+
bundle install
|
|
51
|
+
bundle exec rspec
|
|
52
|
+
bundle exec rubocop
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Use a temporary database for experiments. Never use a production path in a
|
|
56
|
+
test or run a destructive command against an unknown directory:
|
|
57
|
+
|
|
58
|
+
```ruby
|
|
59
|
+
require "tmpdir"
|
|
60
|
+
require "rubydb"
|
|
61
|
+
|
|
62
|
+
Dir.mktmpdir("rubydb-dev-") do |dir|
|
|
63
|
+
engine = RubyDB::Storage::Engine.new(File.join(dir, "dev.rdb"))
|
|
64
|
+
engine.execute("CREATE TABLE items (id INTEGER PRIMARY KEY, name TEXT)")
|
|
65
|
+
engine.close
|
|
66
|
+
end
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
The server/client path is required when testing process boundaries. Multiple
|
|
70
|
+
independent embedded owners must never open the same database directory.
|
|
71
|
+
|
|
72
|
+
## 4. The request lifecycle
|
|
73
|
+
|
|
74
|
+
A typical SQL request follows this sequence:
|
|
75
|
+
|
|
76
|
+
```text
|
|
77
|
+
Ruby API or client
|
|
78
|
+
-> connection/session and authentication
|
|
79
|
+
-> SQL tokenizer/parser
|
|
80
|
+
-> binder and type/parameter validation
|
|
81
|
+
-> planner and executor
|
|
82
|
+
-> transaction/MVCC read or write set
|
|
83
|
+
-> table/index mutation
|
|
84
|
+
-> WAL append and durable commit
|
|
85
|
+
-> result encoding and protocol response
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
Keep failures at the layer that owns them. Parser errors should not be turned
|
|
89
|
+
into storage errors. A failed WAL append must not report a committed schema or
|
|
90
|
+
row change. A client timeout must not silently convert an unknown commit into
|
|
91
|
+
a rollback; the caller must be able to determine whether a request was
|
|
92
|
+
committed before retrying.
|
|
93
|
+
|
|
94
|
+
## 5. Storage and durability invariants
|
|
95
|
+
|
|
96
|
+
The storage directory contains data pages, metadata, WAL state, and recovery
|
|
97
|
+
artifacts. Changes must preserve these invariants:
|
|
98
|
+
|
|
99
|
+
1. A page has valid framing and checksum before it is trusted.
|
|
100
|
+
2. WAL records are validated before replay and are applied in LSN order.
|
|
101
|
+
3. A commit acknowledgement is emitted only after the configured durability
|
|
102
|
+
point has succeeded.
|
|
103
|
+
4. Schema publication and its dependent indexes are visible atomically.
|
|
104
|
+
5. Recovery is idempotent: replaying a committed record does not duplicate a
|
|
105
|
+
row or corrupt an index.
|
|
106
|
+
6. Torn, truncated, or corrupted input fails closed with a useful error.
|
|
107
|
+
7. Temporary files are not mistaken for a completed checkpoint or backup.
|
|
108
|
+
|
|
109
|
+
When changing page layout or record encoding, update the format specification,
|
|
110
|
+
versioning/upgrade path, compatibility tests, and recovery fixtures. Do not
|
|
111
|
+
silently reinterpret old bytes. If a format cannot be read, return an explicit
|
|
112
|
+
upgrade or corruption error and preserve the original files for diagnosis.
|
|
113
|
+
|
|
114
|
+
## 6. WAL, checkpoints, and recovery
|
|
115
|
+
|
|
116
|
+
The WAL is the source of truth between checkpoints. A checkpoint copies safe
|
|
117
|
+
state to durable pages and advances the recovery boundary only after all
|
|
118
|
+
required data and metadata have been flushed. Recovery should:
|
|
119
|
+
|
|
120
|
+
1. open the directory read-only where possible;
|
|
121
|
+
2. validate metadata and the last known checkpoint;
|
|
122
|
+
3. scan WAL frames, stopping only at a valid end boundary;
|
|
123
|
+
4. reject checksum, sequence, length, or transaction inconsistencies;
|
|
124
|
+
5. replay committed work and discard incomplete transactions;
|
|
125
|
+
6. rebuild or validate indexes before accepting writes; and
|
|
126
|
+
7. publish a recovery result with the replayed LSN and warnings.
|
|
127
|
+
|
|
128
|
+
Tests must cover normal reopen, a process terminated during a write, an
|
|
129
|
+
interrupted checkpoint, truncated WAL, invalid checksums, missing metadata,
|
|
130
|
+
full-disk behavior, and restore into a new directory. Fault injection belongs
|
|
131
|
+
around filesystem calls, not only around Ruby methods, because failures occur
|
|
132
|
+
at `fsync`, rename, allocation, and close boundaries.
|
|
133
|
+
|
|
134
|
+
## 7. Transactions, MVCC, and locking
|
|
135
|
+
|
|
136
|
+
Every statement executes in a transaction context, whether it is explicit or
|
|
137
|
+
implicit. The context owns the snapshot, read view, write set, lock state,
|
|
138
|
+
savepoints, and commit result. A transaction must not leak locks or snapshots
|
|
139
|
+
when it raises, times out, is cancelled, or loses its connection.
|
|
140
|
+
|
|
141
|
+
MVCC readers use a stable visibility point. Writers create new versions and
|
|
142
|
+
retain the before-image needed by active readers and rollback. Vacuum may
|
|
143
|
+
reclaim a version only after the global safe point has passed it. A new feature
|
|
144
|
+
must define behavior for:
|
|
145
|
+
|
|
146
|
+
* read committed, repeatable read, and serializable transactions;
|
|
147
|
+
* concurrent update of the same row;
|
|
148
|
+
* unique and foreign-key conflicts;
|
|
149
|
+
* deadlock detection and victim rollback;
|
|
150
|
+
* lock wait timeouts and request cancellation; and
|
|
151
|
+
* commit acknowledgement followed by client disconnect.
|
|
152
|
+
|
|
153
|
+
Deadlock resolution must abort a complete victim transaction, release all of
|
|
154
|
+
its locks, and leave other transactions able to progress. Never fix a deadlock
|
|
155
|
+
by globally disabling locking or by releasing a lock without undoing the
|
|
156
|
+
corresponding write set.
|
|
157
|
+
|
|
158
|
+
## 8. SQL implementation workflow
|
|
159
|
+
|
|
160
|
+
For a new statement or expression:
|
|
161
|
+
|
|
162
|
+
1. write the syntax and semantic contract in `spec/sql`;
|
|
163
|
+
2. add parser acceptance and rejection examples;
|
|
164
|
+
3. add binder/type/nullability behavior;
|
|
165
|
+
4. add planner and executor tests;
|
|
166
|
+
5. test transaction, constraint, index, and error interactions;
|
|
167
|
+
6. test the Ruby API and server protocol path; and
|
|
168
|
+
7. test the ActiveRecord-generated SQL when the feature is adapter-visible.
|
|
169
|
+
|
|
170
|
+
Prefer parameter binding to string interpolation. Every expression needs
|
|
171
|
+
defined behavior for `NULL`, booleans, numeric coercion, text comparison,
|
|
172
|
+
collation, and invalid input. Every DDL operation needs idempotence or an
|
|
173
|
+
explicit error contract. Every optimizer rewrite must have a semantic
|
|
174
|
+
equivalence test against the non-optimized execution path.
|
|
175
|
+
|
|
176
|
+
The documented common SQLite-style profile is intentionally narrower than
|
|
177
|
+
SQLite itself. Do not label a feature “SQLite compatible” until its syntax,
|
|
178
|
+
results, types, error behavior, and migration behavior are tested.
|
|
179
|
+
|
|
180
|
+
## 9. Tables, indexes, and constraints
|
|
181
|
+
|
|
182
|
+
Table mutations and index mutations are one logical operation. If an index
|
|
183
|
+
write fails, the statement must fail visibly and the transaction must either
|
|
184
|
+
roll back or retain a recoverable pending state; it must not acknowledge a row
|
|
185
|
+
that cannot be found by a required index.
|
|
186
|
+
|
|
187
|
+
For each index type, test empty and populated tables, duplicate keys, `NULL`,
|
|
188
|
+
deep splits, reopen, recovery replay, deletion/merge, and concurrent readers.
|
|
189
|
+
For each constraint, test direct SQL, prepared parameters, ActiveRecord
|
|
190
|
+
inserts/updates, rollback, and the exact error class/message contract where
|
|
191
|
+
callers may depend on it.
|
|
192
|
+
|
|
193
|
+
## 10. Server, client, and wire protocol
|
|
194
|
+
|
|
195
|
+
The server is the single owner of the database directory. A client session
|
|
196
|
+
handles authentication, capability negotiation, request IDs, deadlines,
|
|
197
|
+
cancellation, result framing, and connection shutdown. Protocol changes must
|
|
198
|
+
be backward-compatible or carry an explicit protocol version and rejection
|
|
199
|
+
path.
|
|
200
|
+
|
|
201
|
+
Wire cancellation is part of correctness. A cancelled request must stop
|
|
202
|
+
execution at safe checkpoints, release transaction resources, and return a
|
|
203
|
+
definitive cancellation response. If cancellation races with commit, the
|
|
204
|
+
server must preserve the commit outcome and expose enough request identity for
|
|
205
|
+
the client to query status rather than blindly retrying.
|
|
206
|
+
|
|
207
|
+
Test malformed lengths, unknown message types, duplicate request IDs, partial
|
|
208
|
+
frames, client disconnects, server shutdown, timeout races, authentication
|
|
209
|
+
failure, TLS negotiation, and cancellation during a long scan.
|
|
210
|
+
|
|
211
|
+
## 11. Rails and adapter work
|
|
212
|
+
|
|
213
|
+
The ActiveRecord adapter translates Rails schema and query APIs into RubyDB
|
|
214
|
+
SQL. Adapter code must preserve Rails expectations for quoting, bind
|
|
215
|
+
parameters, affected-row counts, last-insert IDs, transactions, savepoints,
|
|
216
|
+
schema introspection, migration versions, and exceptions.
|
|
217
|
+
|
|
218
|
+
When changing adapter behavior, run the example app and test at least:
|
|
219
|
+
|
|
220
|
+
* `where`, ordering, limits, scopes, joins, and aggregate relations;
|
|
221
|
+
* eager loading, nested associations, and inverse relationships;
|
|
222
|
+
* connection pools with multiple threads;
|
|
223
|
+
* schema dump/load and populated-table migrations;
|
|
224
|
+
* rollback, retry, and migration checksum behavior; and
|
|
225
|
+
* every supported Ruby/Rails combination in CI.
|
|
226
|
+
|
|
227
|
+
The adapter is not proof of complete SQLite or PostgreSQL compatibility. An
|
|
228
|
+
application should run its own generated-SQL and migration suite before a
|
|
229
|
+
cutover.
|
|
230
|
+
|
|
231
|
+
## 12. Testing pyramid
|
|
232
|
+
|
|
233
|
+
Use the smallest test that proves the invariant:
|
|
234
|
+
|
|
235
|
+
* unit specs for parsing, encoding, types, and isolated algorithms;
|
|
236
|
+
* component specs for storage, WAL, indexes, transactions, and protocol;
|
|
237
|
+
* integration specs for engine-to-SQL and server-to-client behavior;
|
|
238
|
+
* process specs for crash recovery, pooling, and failover;
|
|
239
|
+
* workload tests for sustained concurrency and resource limits; and
|
|
240
|
+
* example applications for real Rails and Ruby workflows.
|
|
241
|
+
|
|
242
|
+
Useful commands include:
|
|
243
|
+
|
|
244
|
+
```sh
|
|
245
|
+
bundle exec rspec
|
|
246
|
+
bundle exec rspec spec/sql spec/storage spec/transactions
|
|
247
|
+
ruby scripts/durability_drill
|
|
248
|
+
ruby scripts/production_soak
|
|
249
|
+
ruby scripts/fuzz
|
|
250
|
+
RUBYDB_BENCHMARK_ITERATIONS=100 ruby -Ilib benchmarks/basic_workload.rb
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
Record the Ruby version, commit, OS, command, seed, database configuration,
|
|
254
|
+
and artifact paths for every non-trivial run. A green unit suite does not
|
|
255
|
+
replace a restore drill, multi-process soak, or deployment test.
|
|
256
|
+
|
|
257
|
+
## 13. Debugging a failing change
|
|
258
|
+
|
|
259
|
+
Start with [Debugging RubyDB](debugging.md). Reproduce on a fresh temporary
|
|
260
|
+
directory, reduce the schema and SQL, rerun with the reported seed, and save
|
|
261
|
+
logs plus the WAL metadata. Compare embedded and server/client execution when
|
|
262
|
+
ownership or protocol is suspected. Use [Troubleshooting](troubleshooting.md)
|
|
263
|
+
for operator symptoms and evidence-preserving recovery steps.
|
|
264
|
+
|
|
265
|
+
Never “repair” a failing test by deleting WAL, disabling checksums, loosening a
|
|
266
|
+
constraint, or turning off synchronization. Preserve the failing directory,
|
|
267
|
+
copy it, and investigate the copy.
|
|
268
|
+
|
|
269
|
+
## 14. Performance and capacity work
|
|
270
|
+
|
|
271
|
+
Benchmark one variable at a time and report latency percentiles, throughput,
|
|
272
|
+
concurrency, payload shape, cache size, WAL/checkpoint settings, and storage
|
|
273
|
+
medium. Track p50, p95, p99, error rate, lock wait time, WAL growth, checkpoint
|
|
274
|
+
duration, memory, file descriptors, and CPU.
|
|
275
|
+
|
|
276
|
+
Capacity limits are workload-specific. A benchmark result is not a guarantee
|
|
277
|
+
for a Rails application with different indexes, query shapes, or connection
|
|
278
|
+
pool settings. Add a regression threshold only after the benchmark is stable
|
|
279
|
+
across repeated runs and environments.
|
|
280
|
+
|
|
281
|
+
## 15. Safe contribution checklist
|
|
282
|
+
|
|
283
|
+
Before opening a pull request:
|
|
284
|
+
|
|
285
|
+
1. Explain the invariant and compatibility impact.
|
|
286
|
+
2. Add focused tests, including failure paths.
|
|
287
|
+
3. Run the relevant focused suite and the full suite.
|
|
288
|
+
4. Run formatting/lint and `git diff --check`.
|
|
289
|
+
5. Update the appropriate spec and user/operator documentation.
|
|
290
|
+
6. Note migrations, format changes, recovery implications, and rollback plan.
|
|
291
|
+
7. Include benchmark or soak evidence for hot-path and concurrency changes.
|
|
292
|
+
8. Remove debug output, secrets, temporary data, and generated artifacts.
|
|
293
|
+
|
|
294
|
+
See [Contributing](../CONTRIBUTING.md), [testing](contributing/testing.md),
|
|
295
|
+
and [release process](contributing/release-process.md) for the repository
|
|
296
|
+
workflow.
|
|
297
|
+
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# First database
|
|
2
|
+
|
|
3
|
+
```ruby
|
|
4
|
+
require "rubydb"
|
|
5
|
+
|
|
6
|
+
path = "tmp/first-database.rdb"
|
|
7
|
+
engine = RubyDB::Storage::Engine.new(path)
|
|
8
|
+
engine.execute("CREATE TABLE accounts (id INTEGER PRIMARY KEY, email TEXT UNIQUE NOT NULL)")
|
|
9
|
+
engine.execute("INSERT INTO accounts (id, email) VALUES (1, 'ada@example.test')")
|
|
10
|
+
puts engine.execute("SELECT id, email FROM accounts").inspect
|
|
11
|
+
engine.close
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
The engine creates durable storage and WAL state under the selected path. Close
|
|
15
|
+
the engine cleanly and keep the database, WAL, and lock files together. A single
|
|
16
|
+
embedded path has one owner; use a server for multiple processes.
|