rubydb 0.1.4 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.github/PULL_REQUEST_TEMPLATE.md +15 -15
- data/.github/workflows/benchmark.yml +26 -26
- data/.github/workflows/compatibility.yml +63 -63
- data/.github/workflows/fuzz.yml +33 -33
- data/.github/workflows/lint.yml +21 -21
- data/.github/workflows/operations.yml +24 -24
- data/.github/workflows/production-validation.yml +111 -111
- data/.github/workflows/release.yml +77 -77
- data/.github/workflows/security.yml +39 -37
- data/.github/workflows/test.yml +26 -26
- data/.github/workflows/workload.yml +58 -58
- data/.gitignore +16 -5
- data/.rubocop.yml +50 -44
- data/.standard.yml +9 -14
- data/ARCHITECTURE.md +21 -21
- data/CHANGELOG.md +57 -27
- data/CODE_OF_CONDUCT.md +13 -13
- data/CONTRIBUTING.md +29 -29
- data/GOVERNANCE.md +16 -16
- data/Gemfile +18 -17
- data/Gemfile.lock +125 -71
- data/README.md +168 -12
- data/ROADMAP.md +27 -27
- data/Rakefile +76 -71
- data/SECURITY.md +54 -54
- data/SUPPORT.md +14 -14
- data/accelerator/bin/SHA256SUMS +6 -0
- data/accelerator/bin/rubydb-accelerator-darwin-amd64 +0 -0
- data/accelerator/bin/rubydb-accelerator-darwin-arm64 +0 -0
- data/accelerator/bin/rubydb-accelerator-linux-amd64 +0 -0
- data/accelerator/bin/rubydb-accelerator-linux-arm64 +0 -0
- data/accelerator/bin/rubydb-accelerator-windows-amd64.exe +0 -0
- data/accelerator/bin/rubydb-accelerator-windows-arm64.exe +0 -0
- data/accelerator/cmd/rubydb-accelerator/main.go +11 -0
- data/accelerator/go.mod +3 -0
- data/accelerator/internal/execution/aggregate.go +94 -0
- data/accelerator/internal/execution/distinct.go +22 -0
- data/accelerator/internal/execution/filter.go +73 -0
- data/accelerator/internal/execution/join.go +79 -0
- data/accelerator/internal/execution/operators.go +167 -0
- data/accelerator/internal/execution/scan.go +20 -0
- data/accelerator/internal/execution/sort.go +62 -0
- data/accelerator/internal/execution/types.go +136 -0
- data/accelerator/internal/execution/value.go +67 -0
- data/accelerator/internal/memory/arena.go +47 -0
- data/accelerator/internal/memory/reuse.go +22 -0
- data/accelerator/internal/metrics/registry.go +67 -0
- data/accelerator/internal/parallel/bounded_queue.go +56 -0
- data/accelerator/internal/parallel/scheduler.go +47 -0
- data/accelerator/internal/parallel/worker_pool.go +53 -0
- data/accelerator/internal/protocol/cancellation.go +48 -0
- data/accelerator/internal/protocol/columnar.go +263 -0
- data/accelerator/internal/protocol/frame.go +187 -0
- data/accelerator/internal/runtime/worker.go +521 -0
- data/accelerator/internal/storage/page_reader.go +81 -0
- data/accelerator/internal/storage/snapshot_scan.go +539 -0
- data/accelerator/internal/wal/checksum.go +13 -0
- data/accelerator/internal/wal/compression.go +41 -0
- data/accelerator/internal/wal/group_commit.go +24 -0
- data/accelerator/internal/wal/record_encoder.go +40 -0
- data/adapters/activerecord/Gemfile +11 -11
- data/adapters/activerecord/README.md +8 -3
- data/adapters/activerecord/lib/active_record/connection_adapters/rubydb_adapter.rb +881 -879
- data/adapters/activerecord/rubydb-activerecord.gemspec +21 -21
- data/adapters/activerecord/spec/rubydb_adapter_integration_spec.rb +143 -143
- data/adapters/ruby/README.md +18 -18
- data/adapters/sequel/README.md +11 -11
- data/config/monitoring/prometheus-alerts.yml +39 -39
- data/config/production.yml +36 -36
- data/docs/README.md +77 -71
- data/docs/architecture/concurrency.md +14 -14
- data/docs/architecture/current-state.md +125 -125
- data/docs/architecture/execution-engine.md +25 -25
- data/docs/architecture/go-accelerator.md +179 -0
- data/docs/architecture/indexes.md +19 -19
- data/docs/architecture/mvcc.md +19 -19
- data/docs/architecture/overview.md +13 -13
- data/docs/architecture/pages.md +11 -11
- data/docs/architecture/production-roadmap.md +82 -82
- data/docs/architecture/query-planner.md +20 -20
- data/docs/architecture/recovery.md +18 -18
- data/docs/architecture/sql-engine.md +12 -12
- data/docs/architecture/storage-engine.md +14 -14
- data/docs/architecture/transactions.md +10 -10
- data/docs/architecture/wal.md +28 -28
- data/docs/cli-cheatsheet.md +98 -98
- data/docs/cli.md +299 -275
- data/docs/contributing/architecture.md +9 -9
- data/docs/contributing/benchmarking.md +30 -14
- data/docs/contributing/development.md +16 -16
- data/docs/contributing/release-process.md +49 -49
- data/docs/contributing/testing.md +16 -16
- data/docs/debugging.md +229 -229
- data/docs/developer/branching.md +10 -10
- data/docs/developer/database-diff.md +10 -10
- data/docs/developer/local-development.md +49 -17
- data/docs/developer/snapshots.md +9 -9
- data/docs/developer/temporal-data.md +10 -10
- data/docs/developer-guide.md +297 -297
- data/docs/getting-started/first-database.md +16 -16
- data/docs/getting-started/first-query.md +13 -13
- data/docs/getting-started/installation.md +19 -19
- data/docs/getting-started/local-to-production.md +300 -300
- data/docs/getting-started/quickstart.md +17 -17
- data/docs/getting-started/rails.md +16 -16
- data/docs/hardening_backlog.md +93 -93
- data/docs/lessons-learned.md +112 -112
- data/docs/operations/backups.md +33 -33
- data/docs/operations/disaster-recovery.md +31 -31
- data/docs/operations/failover.md +30 -30
- data/docs/operations/monitoring.md +25 -25
- data/docs/operations/production-guide.md +295 -295
- data/docs/operations/production-runbook.md +45 -45
- data/docs/operations/replication.md +33 -33
- data/docs/operations/restore.md +6 -6
- data/docs/operations/runbook.md +34 -34
- data/docs/operations/upgrades.md +14 -14
- data/docs/operations/workload-testing.md +17 -17
- data/docs/production-readiness.md +118 -118
- data/docs/production_validation.md +150 -150
- data/docs/rails/active-record.md +11 -11
- data/docs/rails/compatibility-guide.md +90 -90
- data/docs/rails/database-yml.md +92 -92
- data/docs/rails/installation.md +17 -17
- data/docs/rails/migrations.md +17 -17
- data/docs/rails/production.md +82 -82
- data/docs/rails/troubleshooting.md +18 -18
- data/docs/release.md +25 -25
- data/docs/server/architecture.md +10 -10
- data/docs/server/authentication.md +10 -10
- data/docs/server/configuration.md +16 -16
- data/docs/server/connection-pooling.md +10 -10
- data/docs/server/deployment.md +10 -10
- data/docs/server/protocol.md +12 -12
- data/docs/sql/compatibility-guide.md +82 -82
- data/docs/sql/compatibility.md +39 -39
- data/docs/sql/data-types.md +10 -10
- data/docs/sql/functions.md +9 -9
- data/docs/sql/joins.md +9 -9
- data/docs/sql/operators.md +9 -9
- data/docs/sql/sqlite-compatibility.md +21 -21
- data/docs/sql/syntax.md +10 -10
- data/docs/sql/transactions.md +10 -10
- data/docs/troubleshooting.md +244 -244
- data/lessons/01-foundations.md +73 -0
- data/lessons/02-local-development.md +121 -0
- data/lessons/03-embedded-rubydb.md +99 -0
- data/lessons/04-rails-complex-apps.md +138 -0
- data/lessons/05-rubydb-production-server.md +237 -0
- data/lessons/06-postgresql-massive-apps.md +96 -0
- data/lessons/07-hybrid-microservices.md +179 -0
- data/lessons/08-migrations-backups-recovery.md +86 -0
- data/lessons/09-observability-security-scale.md +87 -0
- data/lessons/10-release-readiness.md +192 -0
- data/lessons/11-community-adapter.md +323 -0
- data/lessons/12-rails-ecommerce-pressure.md +263 -0
- data/lib/rubydb/accelerator/client.rb +451 -0
- data/lib/rubydb/accelerator/error.rb +22 -0
- data/lib/rubydb/accelerator/manager.rb +606 -0
- data/lib/rubydb/accelerator.rb +13 -0
- data/lib/rubydb/backup/archive.rb +332 -334
- data/lib/rubydb/backup/backup.rb +400 -401
- data/lib/rubydb/backup/incremental.rb +349 -353
- data/lib/rubydb/backup/restore.rb +289 -290
- data/lib/rubydb/backup/snapshot.rb +265 -267
- data/lib/rubydb/backup/verification.rb +276 -279
- data/lib/rubydb/branching/branch.rb +181 -181
- data/lib/rubydb/branching/branch_manager.rb +307 -311
- data/lib/rubydb/branching/branch_metadata.rb +140 -140
- data/lib/rubydb/branching/checkout.rb +165 -166
- data/lib/rubydb/branching/copy_on_write.rb +272 -272
- data/lib/rubydb/branching/diff.rb +137 -138
- data/lib/rubydb/branching/merge.rb +282 -285
- data/lib/rubydb/build_info.rb +15 -15
- data/lib/rubydb/catalog/catalog.rb +391 -391
- data/lib/rubydb/catalog/column.rb +112 -112
- data/lib/rubydb/catalog/constraint.rb +180 -180
- data/lib/rubydb/catalog/database.rb +184 -184
- data/lib/rubydb/catalog/index.rb +97 -97
- data/lib/rubydb/catalog/schema.rb +103 -103
- data/lib/rubydb/catalog/sequence.rb +90 -90
- data/lib/rubydb/catalog/system_catalog.rb +698 -698
- data/lib/rubydb/catalog/table.rb +178 -178
- data/lib/rubydb/catalog/trigger.rb +102 -102
- data/lib/rubydb/catalog/view.rb +66 -66
- data/lib/rubydb/cli/application.rb +168 -163
- data/lib/rubydb/cli/commands/accelerator.rb +72 -0
- data/lib/rubydb/cli/commands/backup.rb +80 -81
- data/lib/rubydb/cli/commands/branch.rb +72 -72
- data/lib/rubydb/cli/commands/checkout.rb +54 -54
- data/lib/rubydb/cli/commands/create.rb +58 -58
- data/lib/rubydb/cli/commands/diff.rb +76 -77
- data/lib/rubydb/cli/commands/doctor.rb +77 -74
- data/lib/rubydb/cli/commands/drop.rb +57 -57
- data/lib/rubydb/cli/commands/init.rb +101 -102
- data/lib/rubydb/cli/commands/inspect.rb +95 -95
- data/lib/rubydb/cli/commands/merge.rb +63 -63
- data/lib/rubydb/cli/commands/migrate.rb +62 -62
- data/lib/rubydb/cli/commands/restart.rb +42 -39
- data/lib/rubydb/cli/commands/restore.rb +121 -121
- data/lib/rubydb/cli/commands/shell.rb +365 -365
- data/lib/rubydb/cli/commands/snapshot.rb +79 -79
- data/lib/rubydb/cli/commands/start.rb +88 -82
- data/lib/rubydb/cli/commands/status.rb +96 -92
- data/lib/rubydb/cli/commands/stop.rb +47 -47
- data/lib/rubydb/cli/commands/vacuum.rb +58 -58
- data/lib/rubydb/cli/formatter.rb +221 -221
- data/lib/rubydb/cli/output.rb +168 -168
- data/lib/rubydb/client/client.rb +309 -304
- data/lib/rubydb/client/connection.rb +429 -415
- data/lib/rubydb/client/connection_pool.rb +168 -168
- data/lib/rubydb/client/connection_url.rb +96 -96
- data/lib/rubydb/client/prepared_statement.rb +60 -60
- data/lib/rubydb/client/result.rb +127 -123
- data/lib/rubydb/client/statement.rb +52 -52
- data/lib/rubydb/client/transaction.rb +130 -130
- data/lib/rubydb/concurrency/concurrency.rb +19 -19
- data/lib/rubydb/concurrency/deadlock_detector.rb +148 -150
- data/lib/rubydb/concurrency/latch.rb +101 -101
- data/lib/rubydb/concurrency/lock_graph.rb +163 -165
- data/lib/rubydb/concurrency/mutex.rb +181 -183
- data/lib/rubydb/concurrency/rw_lock.rb +180 -180
- data/lib/rubydb/concurrency/scheduler.rb +248 -250
- data/lib/rubydb/concurrency/worker_pool.rb +145 -143
- data/lib/rubydb/configuration/config.rb +170 -170
- data/lib/rubydb/configuration/defaults.rb +191 -179
- data/lib/rubydb/configuration/environment.rb +152 -152
- data/lib/rubydb/configuration/parser.rb +185 -185
- data/lib/rubydb/configuration/validation.rb +228 -221
- data/lib/rubydb/constants.rb +74 -74
- data/lib/rubydb/constraints/check.rb +181 -181
- data/lib/rubydb/constraints/constraint.rb +101 -101
- data/lib/rubydb/constraints/foreign_key.rb +130 -130
- data/lib/rubydb/constraints/not_null.rb +64 -64
- data/lib/rubydb/constraints/primary_key.rb +99 -99
- data/lib/rubydb/constraints/unique.rb +106 -108
- data/lib/rubydb/constraints/validator.rb +349 -350
- data/lib/rubydb/errors/authentication_error.rb +10 -10
- data/lib/rubydb/errors/authorization_error.rb +23 -23
- data/lib/rubydb/errors/client_error.rb +10 -10
- data/lib/rubydb/errors/configuration_error.rb +10 -10
- data/lib/rubydb/errors/connection_error.rb +10 -10
- data/lib/rubydb/errors/constraint_error.rb +23 -23
- data/lib/rubydb/errors/corruption_error.rb +10 -10
- data/lib/rubydb/errors/database_error.rb +10 -10
- data/lib/rubydb/errors/error.rb +20 -20
- data/lib/rubydb/errors/execution_error.rb +10 -10
- data/lib/rubydb/errors/parser_error.rb +10 -10
- data/lib/rubydb/errors/recovery_error.rb +10 -10
- data/lib/rubydb/errors/replication_error.rb +10 -10
- data/lib/rubydb/errors/server_error.rb +6 -6
- data/lib/rubydb/errors/storage_error.rb +10 -10
- data/lib/rubydb/errors/transaction_error.rb +10 -10
- data/lib/rubydb/execution/accelerator_dispatch.rb +30 -0
- data/lib/rubydb/execution/aggregate_executor.rb +134 -138
- data/lib/rubydb/execution/cost_model.rb +72 -0
- data/lib/rubydb/execution/delete_executor.rb +110 -112
- data/lib/rubydb/execution/distinct_executor.rb +131 -135
- data/lib/rubydb/execution/executor.rb +1544 -1188
- data/lib/rubydb/execution/expression.rb +191 -193
- data/lib/rubydb/execution/index_scan.rb +142 -142
- data/lib/rubydb/execution/insert_executor.rb +215 -217
- data/lib/rubydb/execution/join_executor.rb +243 -249
- data/lib/rubydb/execution/limit_executor.rb +83 -85
- data/lib/rubydb/execution/operator_selection.rb +57 -0
- data/lib/rubydb/execution/optimizer.rb +227 -215
- data/lib/rubydb/execution/physical_plan.rb +47 -0
- data/lib/rubydb/execution/plan.rb +355 -353
- data/lib/rubydb/execution/planner.rb +508 -536
- data/lib/rubydb/execution/predicate.rb +235 -235
- data/lib/rubydb/execution/scan.rb +49 -49
- data/lib/rubydb/execution/sequential_scan.rb +63 -63
- data/lib/rubydb/execution/sort_executor.rb +194 -185
- data/lib/rubydb/execution/update_executor.rb +160 -162
- data/lib/rubydb/functions/aggregate.rb +70 -70
- data/lib/rubydb/functions/date_functions.rb +274 -278
- data/lib/rubydb/functions/function.rb +85 -85
- data/lib/rubydb/functions/json_functions.rb +231 -215
- data/lib/rubydb/functions/numeric_functions.rb +346 -346
- data/lib/rubydb/functions/scalar.rb +52 -52
- data/lib/rubydb/functions/string_functions.rb +383 -383
- data/lib/rubydb/functions/system_functions.rb +258 -246
- data/lib/rubydb/history/as_of.rb +238 -238
- data/lib/rubydb/history/change.rb +105 -105
- data/lib/rubydb/history/history.rb +131 -131
- data/lib/rubydb/history/history_manager.rb +228 -229
- data/lib/rubydb/history/temporal_query.rb +202 -202
- data/lib/rubydb/history/timeline.rb +144 -144
- data/lib/rubydb/indexes/btree.rb +215 -186
- data/lib/rubydb/indexes/btree_cursor.rb +258 -258
- data/lib/rubydb/indexes/btree_node.rb +384 -385
- data/lib/rubydb/indexes/hash_index.rb +150 -150
- data/lib/rubydb/indexes/index.rb +71 -71
- data/lib/rubydb/indexes/index_manager.rb +408 -406
- data/lib/rubydb/indexes/index_scan.rb +466 -470
- data/lib/rubydb/migrations/migration.rb +253 -254
- data/lib/rubydb/migrations/migration_lock.rb +146 -146
- data/lib/rubydb/migrations/migration_manager.rb +187 -176
- data/lib/rubydb/migrations/migration_version.rb +71 -71
- data/lib/rubydb/migrations/schema_diff.rb +211 -211
- data/lib/rubydb/migrations/schema_version.rb +64 -64
- data/lib/rubydb/monitoring/events.rb +155 -160
- data/lib/rubydb/monitoring/health.rb +216 -222
- data/lib/rubydb/monitoring/logger.rb +188 -193
- data/lib/rubydb/monitoring/metrics.rb +363 -359
- data/lib/rubydb/monitoring/performance.rb +176 -176
- data/lib/rubydb/monitoring/statistics.rb +168 -170
- data/lib/rubydb/mvcc/garbage_collector.rb +199 -199
- data/lib/rubydb/mvcc/mvcc.rb +16 -16
- data/lib/rubydb/mvcc/snapshot.rb +146 -147
- data/lib/rubydb/mvcc/vacuum.rb +180 -180
- data/lib/rubydb/mvcc/version.rb +106 -106
- data/lib/rubydb/mvcc/version_store.rb +396 -398
- data/lib/rubydb/mvcc/visibility.rb +107 -109
- data/lib/rubydb/protocol/capabilities.rb +125 -125
- data/lib/rubydb/protocol/decoder.rb +142 -145
- data/lib/rubydb/protocol/encoder.rb +131 -136
- data/lib/rubydb/protocol/handshake.rb +306 -305
- data/lib/rubydb/protocol/message.rb +121 -121
- data/lib/rubydb/protocol/parameter_binder.rb +101 -0
- data/lib/rubydb/protocol/protocol.rb +276 -277
- data/lib/rubydb/protocol/version.rb +54 -54
- data/lib/rubydb/rails/adapter.rb +245 -239
- data/lib/rubydb/rails/connection.rb +312 -314
- data/lib/rubydb/rails/database_statements.rb +122 -122
- data/lib/rubydb/rails/migration.rb +131 -131
- data/lib/rubydb/rails/quoting.rb +109 -109
- data/lib/rubydb/rails/result.rb +117 -117
- data/lib/rubydb/rails/schema_statements.rb +339 -339
- data/lib/rubydb/rails/transaction.rb +105 -105
- data/lib/rubydb/rails/type.rb +126 -126
- data/lib/rubydb/recovery/checkpoint.rb +261 -257
- data/lib/rubydb/recovery/consistency.rb +457 -467
- data/lib/rubydb/recovery/corruption_detector.rb +5 -5
- data/lib/rubydb/recovery/crash_recovery.rb +381 -387
- data/lib/rubydb/recovery/recovery_manager.rb +204 -206
- data/lib/rubydb/recovery/redo.rb +235 -237
- data/lib/rubydb/recovery/undo.rb +204 -206
- data/lib/rubydb/replication/failover.rb +5 -5
- data/lib/rubydb/replication/fencing.rb +63 -63
- data/lib/rubydb/replication/primary.rb +461 -450
- data/lib/rubydb/replication/replica.rb +382 -384
- data/lib/rubydb/replication/replication_log.rb +194 -200
- data/lib/rubydb/replication/replication_manager.rb +307 -308
- data/lib/rubydb/replication/replication_slot.rb +293 -295
- data/lib/rubydb/replication/replication_stream.rb +198 -201
- data/lib/rubydb/rubydb.rb +570 -560
- data/lib/rubydb/security/access_control.rb +252 -254
- data/lib/rubydb/security/audit_log.rb +209 -213
- data/lib/rubydb/security/authentication.rb +302 -302
- data/lib/rubydb/security/authorization.rb +282 -282
- data/lib/rubydb/security/credentials.rb +192 -196
- data/lib/rubydb/security/password.rb +205 -215
- data/lib/rubydb/security/permissions.rb +74 -74
- data/lib/rubydb/security/role.rb +99 -101
- data/lib/rubydb/security/user.rb +86 -86
- data/lib/rubydb/server/connection.rb +383 -366
- data/lib/rubydb/server/connection_pool.rb +193 -193
- data/lib/rubydb/server/lifecycle.rb +227 -228
- data/lib/rubydb/server/listener.rb +139 -136
- data/lib/rubydb/server/request_handler.rb +277 -276
- data/lib/rubydb/server/server.rb +363 -364
- data/lib/rubydb/server/session.rb +416 -369
- data/lib/rubydb/server/worker.rb +206 -210
- data/lib/rubydb/server/worker_pool.rb +168 -168
- data/lib/rubydb/sql/ast/alter_table.rb +169 -169
- data/lib/rubydb/sql/ast/begin_transaction.rb +47 -47
- data/lib/rubydb/sql/ast/commit.rb +37 -37
- data/lib/rubydb/sql/ast/constraint.rb +92 -83
- data/lib/rubydb/sql/ast/create_database.rb +41 -41
- data/lib/rubydb/sql/ast/create_index.rb +61 -61
- data/lib/rubydb/sql/ast/create_schema.rb +52 -52
- data/lib/rubydb/sql/ast/create_table.rb +187 -187
- data/lib/rubydb/sql/ast/delete.rb +54 -54
- data/lib/rubydb/sql/ast/drop_database.rb +41 -41
- data/lib/rubydb/sql/ast/drop_index.rb +41 -41
- data/lib/rubydb/sql/ast/drop_schema.rb +49 -49
- data/lib/rubydb/sql/ast/drop_table.rb +49 -49
- data/lib/rubydb/sql/ast/explain.rb +64 -64
- data/lib/rubydb/sql/ast/expression.rb +617 -604
- data/lib/rubydb/sql/ast/insert.rb +66 -66
- data/lib/rubydb/sql/ast/node.rb +42 -42
- data/lib/rubydb/sql/ast/rollback.rb +63 -63
- data/lib/rubydb/sql/ast/savepoint.rb +59 -59
- data/lib/rubydb/sql/ast/select.rb +88 -88
- data/lib/rubydb/sql/ast/set_operation.rb +22 -20
- data/lib/rubydb/sql/ast/trigger.rb +35 -29
- data/lib/rubydb/sql/ast/update.rb +88 -88
- data/lib/rubydb/sql/ast/vacuum.rb +19 -19
- data/lib/rubydb/sql/ast/view.rb +38 -32
- data/lib/rubydb/sql/ast/with.rb +32 -32
- data/lib/rubydb/sql/grammar.rb +86 -86
- data/lib/rubydb/sql/keywords.rb +156 -156
- data/lib/rubydb/sql/lexer.rb +209 -214
- data/lib/rubydb/sql/operators.rb +100 -100
- data/lib/rubydb/sql/parser.rb +1167 -1170
- data/lib/rubydb/sql/planner/analyzer.rb +283 -302
- data/lib/rubydb/sql/planner/binder.rb +537 -550
- data/lib/rubydb/sql/planner/type_checker.rb +427 -431
- data/lib/rubydb/sql/token.rb +210 -210
- data/lib/rubydb/storage/buffer_frame.rb +44 -44
- data/lib/rubydb/storage/buffer_pool.rb +155 -155
- data/lib/rubydb/storage/database_lock.rb +74 -74
- data/lib/rubydb/storage/deserializer.rb +332 -342
- data/lib/rubydb/storage/engine.rb +2409 -2330
- data/lib/rubydb/storage/file_manager.rb +191 -187
- data/lib/rubydb/storage/free_space_map.rb +79 -81
- data/lib/rubydb/storage/page.rb +92 -94
- data/lib/rubydb/storage/page_allocator.rb +852 -855
- data/lib/rubydb/storage/page_header.rb +63 -67
- data/lib/rubydb/storage/page_manager.rb +127 -131
- data/lib/rubydb/storage/record.rb +58 -58
- data/lib/rubydb/storage/row.rb +78 -78
- data/lib/rubydb/storage/serializer.rb +51 -51
- data/lib/rubydb/storage/snapshot_reader.rb +167 -0
- data/lib/rubydb/storage/storage_layout.rb +151 -151
- data/lib/rubydb/storage/storage_manager.rb +114 -114
- data/lib/rubydb/storage/tuple.rb +458 -461
- data/lib/rubydb/storage/visibility_map.rb +964 -973
- data/lib/rubydb/transactions/commit_manager.rb +219 -220
- data/lib/rubydb/transactions/isolation.rb +98 -98
- data/lib/rubydb/transactions/lock.rb +76 -76
- data/lib/rubydb/transactions/lock_manager.rb +359 -362
- data/lib/rubydb/transactions/savepoint.rb +142 -143
- data/lib/rubydb/transactions/transaction.rb +214 -215
- data/lib/rubydb/transactions/transaction_id.rb +84 -84
- data/lib/rubydb/transactions/transaction_log.rb +256 -257
- data/lib/rubydb/transactions/transaction_manager.rb +434 -435
- data/lib/rubydb/types/bigint.rb +36 -36
- data/lib/rubydb/types/blob.rb +37 -37
- data/lib/rubydb/types/boolean.rb +34 -34
- data/lib/rubydb/types/date.rb +39 -39
- data/lib/rubydb/types/decimal.rb +48 -48
- data/lib/rubydb/types/float.rb +34 -34
- data/lib/rubydb/types/integer.rb +36 -36
- data/lib/rubydb/types/json.rb +41 -41
- data/lib/rubydb/types/null.rb +34 -34
- data/lib/rubydb/types/smallint.rb +36 -36
- data/lib/rubydb/types/text.rb +37 -37
- data/lib/rubydb/types/time.rb +46 -46
- data/lib/rubydb/types/timestamp.rb +39 -39
- data/lib/rubydb/types/type.rb +119 -119
- data/lib/rubydb/types/uuid.rb +47 -47
- data/lib/rubydb/types/varchar.rb +37 -37
- data/lib/rubydb/version.rb +32 -32
- data/lib/rubydb/wal/archive.rb +207 -193
- data/lib/rubydb/wal/checkpoint.rb +181 -183
- data/lib/rubydb/wal/lsn.rb +94 -94
- data/lib/rubydb/wal/reader.rb +259 -260
- data/lib/rubydb/wal/record.rb +105 -105
- data/lib/rubydb/wal/segment.rb +193 -193
- data/lib/rubydb/wal/wal.rb +481 -452
- data/lib/rubydb/wal/writer.rb +236 -236
- data/lib/rubydb.rb +7 -7
- data/packaging/docker/docker-compose.failover.yml +43 -43
- data/packaging/homebrew/rubydb.rb +19 -19
- data/rubydb.gemspec +70 -57
- data/scripts/benchmark +7 -7
- data/scripts/build_accelerator +49 -0
- data/scripts/durability_drill +37 -37
- data/scripts/fuzz +63 -63
- data/scripts/release +77 -42
- data/scripts/release_check +43 -43
- data/scripts/replication_failover_drill +268 -250
- data/scripts/replication_network_failover_drill +287 -255
- data/scripts/restore_drill +45 -45
- data/scripts/security +45 -0
- metadata +102 -1
data/docs/developer-guide.md
CHANGED
|
@@ -1,297 +1,297 @@
|
|
|
1
|
-
# RubyDB developer guide
|
|
2
|
-
|
|
3
|
-
This is the implementation guide for contributors who need to change RubyDB
|
|
4
|
-
safely. It explains the repository, the request path, the storage invariants,
|
|
5
|
-
the test strategy, and the debugging workflow. Read it together with the
|
|
6
|
-
topic specifications under `spec/`; the specifications are the behavioral
|
|
7
|
-
contract, while this guide explains how the pieces fit together.
|
|
8
|
-
|
|
9
|
-
## 1. Scope and support promise
|
|
10
|
-
|
|
11
|
-
RubyDB is a Ruby-native relational database with two ownership modes:
|
|
12
|
-
|
|
13
|
-
* embedded mode, where one Ruby process owns a database directory; and
|
|
14
|
-
* server mode, where the server owns the directory and application processes
|
|
15
|
-
use the client protocol.
|
|
16
|
-
|
|
17
|
-
The supported surface is the behavior exercised by the test suite and listed
|
|
18
|
-
in [SQL compatibility](sql/compatibility.md). Similar syntax does not imply
|
|
19
|
-
complete PostgreSQL, MySQL, or SQLite compatibility. A change that broadens
|
|
20
|
-
syntax must also specify its semantics, errors, types, transactions, and
|
|
21
|
-
adapter behavior.
|
|
22
|
-
|
|
23
|
-
## 2. Repository orientation
|
|
24
|
-
|
|
25
|
-
The important directories are:
|
|
26
|
-
|
|
27
|
-
| Path | Responsibility |
|
|
28
|
-
| --- | --- |
|
|
29
|
-
| `lib/rubydb` | Engine, SQL, storage, transactions, server, client, adapters |
|
|
30
|
-
| `spec` | Unit, integration, protocol, SQL, recovery, and adapter contracts |
|
|
31
|
-
| `benchmarks` | Repeatable throughput and concurrency measurements |
|
|
32
|
-
| `scripts` | Soak tests, durability drills, fuzzing, and release checks |
|
|
33
|
-
| `config` | Example configuration and monitoring rules |
|
|
34
|
-
| `examples` | Runnable Ruby and Rails applications |
|
|
35
|
-
| `docs` | User, operator, developer, and compatibility documentation |
|
|
36
|
-
| `packaging` | Container and deployment examples |
|
|
37
|
-
| `exe` and `bin` | Public command-line entry points |
|
|
38
|
-
|
|
39
|
-
Start with `lib/rubydb.rb` and follow the public API into the engine. For a
|
|
40
|
-
feature, locate the nearest existing spec before editing implementation code.
|
|
41
|
-
Avoid adding a second abstraction when an existing subsystem already owns the
|
|
42
|
-
invariant.
|
|
43
|
-
|
|
44
|
-
## 3. Local setup
|
|
45
|
-
|
|
46
|
-
Use a supported Ruby version from the project CI matrix and install the locked
|
|
47
|
-
dependencies:
|
|
48
|
-
|
|
49
|
-
```sh
|
|
50
|
-
bundle install
|
|
51
|
-
bundle exec rspec
|
|
52
|
-
bundle exec rubocop
|
|
53
|
-
```
|
|
54
|
-
|
|
55
|
-
Use a temporary database for experiments. Never use a production path in a
|
|
56
|
-
test or run a destructive command against an unknown directory:
|
|
57
|
-
|
|
58
|
-
```ruby
|
|
59
|
-
require "tmpdir"
|
|
60
|
-
require "rubydb"
|
|
61
|
-
|
|
62
|
-
Dir.mktmpdir("rubydb-dev-") do |dir|
|
|
63
|
-
engine = RubyDB::Storage::Engine.new(File.join(dir, "dev.rdb"))
|
|
64
|
-
engine.execute("CREATE TABLE items (id INTEGER PRIMARY KEY, name TEXT)")
|
|
65
|
-
engine.close
|
|
66
|
-
end
|
|
67
|
-
```
|
|
68
|
-
|
|
69
|
-
The server/client path is required when testing process boundaries. Multiple
|
|
70
|
-
independent embedded owners must never open the same database directory.
|
|
71
|
-
|
|
72
|
-
## 4. The request lifecycle
|
|
73
|
-
|
|
74
|
-
A typical SQL request follows this sequence:
|
|
75
|
-
|
|
76
|
-
```text
|
|
77
|
-
Ruby API or client
|
|
78
|
-
-> connection/session and authentication
|
|
79
|
-
-> SQL tokenizer/parser
|
|
80
|
-
-> binder and type/parameter validation
|
|
81
|
-
-> planner and executor
|
|
82
|
-
-> transaction/MVCC read or write set
|
|
83
|
-
-> table/index mutation
|
|
84
|
-
-> WAL append and durable commit
|
|
85
|
-
-> result encoding and protocol response
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
Keep failures at the layer that owns them. Parser errors should not be turned
|
|
89
|
-
into storage errors. A failed WAL append must not report a committed schema or
|
|
90
|
-
row change. A client timeout must not silently convert an unknown commit into
|
|
91
|
-
a rollback; the caller must be able to determine whether a request was
|
|
92
|
-
committed before retrying.
|
|
93
|
-
|
|
94
|
-
## 5. Storage and durability invariants
|
|
95
|
-
|
|
96
|
-
The storage directory contains data pages, metadata, WAL state, and recovery
|
|
97
|
-
artifacts. Changes must preserve these invariants:
|
|
98
|
-
|
|
99
|
-
1. A page has valid framing and checksum before it is trusted.
|
|
100
|
-
2. WAL records are validated before replay and are applied in LSN order.
|
|
101
|
-
3. A commit acknowledgement is emitted only after the configured durability
|
|
102
|
-
point has succeeded.
|
|
103
|
-
4. Schema publication and its dependent indexes are visible atomically.
|
|
104
|
-
5. Recovery is idempotent: replaying a committed record does not duplicate a
|
|
105
|
-
row or corrupt an index.
|
|
106
|
-
6. Torn, truncated, or corrupted input fails closed with a useful error.
|
|
107
|
-
7. Temporary files are not mistaken for a completed checkpoint or backup.
|
|
108
|
-
|
|
109
|
-
When changing page layout or record encoding, update the format specification,
|
|
110
|
-
versioning/upgrade path, compatibility tests, and recovery fixtures. Do not
|
|
111
|
-
silently reinterpret old bytes. If a format cannot be read, return an explicit
|
|
112
|
-
upgrade or corruption error and preserve the original files for diagnosis.
|
|
113
|
-
|
|
114
|
-
## 6. WAL, checkpoints, and recovery
|
|
115
|
-
|
|
116
|
-
The WAL is the source of truth between checkpoints. A checkpoint copies safe
|
|
117
|
-
state to durable pages and advances the recovery boundary only after all
|
|
118
|
-
required data and metadata have been flushed. Recovery should:
|
|
119
|
-
|
|
120
|
-
1. open the directory read-only where possible;
|
|
121
|
-
2. validate metadata and the last known checkpoint;
|
|
122
|
-
3. scan WAL frames, stopping only at a valid end boundary;
|
|
123
|
-
4. reject checksum, sequence, length, or transaction inconsistencies;
|
|
124
|
-
5. replay committed work and discard incomplete transactions;
|
|
125
|
-
6. rebuild or validate indexes before accepting writes; and
|
|
126
|
-
7. publish a recovery result with the replayed LSN and warnings.
|
|
127
|
-
|
|
128
|
-
Tests must cover normal reopen, a process terminated during a write, an
|
|
129
|
-
interrupted checkpoint, truncated WAL, invalid checksums, missing metadata,
|
|
130
|
-
full-disk behavior, and restore into a new directory. Fault injection belongs
|
|
131
|
-
around filesystem calls, not only around Ruby methods, because failures occur
|
|
132
|
-
at `fsync`, rename, allocation, and close boundaries.
|
|
133
|
-
|
|
134
|
-
## 7. Transactions, MVCC, and locking
|
|
135
|
-
|
|
136
|
-
Every statement executes in a transaction context, whether it is explicit or
|
|
137
|
-
implicit. The context owns the snapshot, read view, write set, lock state,
|
|
138
|
-
savepoints, and commit result. A transaction must not leak locks or snapshots
|
|
139
|
-
when it raises, times out, is cancelled, or loses its connection.
|
|
140
|
-
|
|
141
|
-
MVCC readers use a stable visibility point. Writers create new versions and
|
|
142
|
-
retain the before-image needed by active readers and rollback. Vacuum may
|
|
143
|
-
reclaim a version only after the global safe point has passed it. A new feature
|
|
144
|
-
must define behavior for:
|
|
145
|
-
|
|
146
|
-
* read committed, repeatable read, and serializable transactions;
|
|
147
|
-
* concurrent update of the same row;
|
|
148
|
-
* unique and foreign-key conflicts;
|
|
149
|
-
* deadlock detection and victim rollback;
|
|
150
|
-
* lock wait timeouts and request cancellation; and
|
|
151
|
-
* commit acknowledgement followed by client disconnect.
|
|
152
|
-
|
|
153
|
-
Deadlock resolution must abort a complete victim transaction, release all of
|
|
154
|
-
its locks, and leave other transactions able to progress. Never fix a deadlock
|
|
155
|
-
by globally disabling locking or by releasing a lock without undoing the
|
|
156
|
-
corresponding write set.
|
|
157
|
-
|
|
158
|
-
## 8. SQL implementation workflow
|
|
159
|
-
|
|
160
|
-
For a new statement or expression:
|
|
161
|
-
|
|
162
|
-
1. write the syntax and semantic contract in `spec/sql`;
|
|
163
|
-
2. add parser acceptance and rejection examples;
|
|
164
|
-
3. add binder/type/nullability behavior;
|
|
165
|
-
4. add planner and executor tests;
|
|
166
|
-
5. test transaction, constraint, index, and error interactions;
|
|
167
|
-
6. test the Ruby API and server protocol path; and
|
|
168
|
-
7. test the ActiveRecord-generated SQL when the feature is adapter-visible.
|
|
169
|
-
|
|
170
|
-
Prefer parameter binding to string interpolation. Every expression needs
|
|
171
|
-
defined behavior for `NULL`, booleans, numeric coercion, text comparison,
|
|
172
|
-
collation, and invalid input. Every DDL operation needs idempotence or an
|
|
173
|
-
explicit error contract. Every optimizer rewrite must have a semantic
|
|
174
|
-
equivalence test against the non-optimized execution path.
|
|
175
|
-
|
|
176
|
-
The documented common SQLite-style profile is intentionally narrower than
|
|
177
|
-
SQLite itself. Do not label a feature “SQLite compatible” until its syntax,
|
|
178
|
-
results, types, error behavior, and migration behavior are tested.
|
|
179
|
-
|
|
180
|
-
## 9. Tables, indexes, and constraints
|
|
181
|
-
|
|
182
|
-
Table mutations and index mutations are one logical operation. If an index
|
|
183
|
-
write fails, the statement must fail visibly and the transaction must either
|
|
184
|
-
roll back or retain a recoverable pending state; it must not acknowledge a row
|
|
185
|
-
that cannot be found by a required index.
|
|
186
|
-
|
|
187
|
-
For each index type, test empty and populated tables, duplicate keys, `NULL`,
|
|
188
|
-
deep splits, reopen, recovery replay, deletion/merge, and concurrent readers.
|
|
189
|
-
For each constraint, test direct SQL, prepared parameters, ActiveRecord
|
|
190
|
-
inserts/updates, rollback, and the exact error class/message contract where
|
|
191
|
-
callers may depend on it.
|
|
192
|
-
|
|
193
|
-
## 10. Server, client, and wire protocol
|
|
194
|
-
|
|
195
|
-
The server is the single owner of the database directory. A client session
|
|
196
|
-
handles authentication, capability negotiation, request IDs, deadlines,
|
|
197
|
-
cancellation, result framing, and connection shutdown. Protocol changes must
|
|
198
|
-
be backward-compatible or carry an explicit protocol version and rejection
|
|
199
|
-
path.
|
|
200
|
-
|
|
201
|
-
Wire cancellation is part of correctness. A cancelled request must stop
|
|
202
|
-
execution at safe checkpoints, release transaction resources, and return a
|
|
203
|
-
definitive cancellation response. If cancellation races with commit, the
|
|
204
|
-
server must preserve the commit outcome and expose enough request identity for
|
|
205
|
-
the client to query status rather than blindly retrying.
|
|
206
|
-
|
|
207
|
-
Test malformed lengths, unknown message types, duplicate request IDs, partial
|
|
208
|
-
frames, client disconnects, server shutdown, timeout races, authentication
|
|
209
|
-
failure, TLS negotiation, and cancellation during a long scan.
|
|
210
|
-
|
|
211
|
-
## 11. Rails and adapter work
|
|
212
|
-
|
|
213
|
-
The ActiveRecord adapter translates Rails schema and query APIs into RubyDB
|
|
214
|
-
SQL. Adapter code must preserve Rails expectations for quoting, bind
|
|
215
|
-
parameters, affected-row counts, last-insert IDs, transactions, savepoints,
|
|
216
|
-
schema introspection, migration versions, and exceptions.
|
|
217
|
-
|
|
218
|
-
When changing adapter behavior, run the example app and test at least:
|
|
219
|
-
|
|
220
|
-
* `where`, ordering, limits, scopes, joins, and aggregate relations;
|
|
221
|
-
* eager loading, nested associations, and inverse relationships;
|
|
222
|
-
* connection pools with multiple threads;
|
|
223
|
-
* schema dump/load and populated-table migrations;
|
|
224
|
-
* rollback, retry, and migration checksum behavior; and
|
|
225
|
-
* every supported Ruby/Rails combination in CI.
|
|
226
|
-
|
|
227
|
-
The adapter is not proof of complete SQLite or PostgreSQL compatibility. An
|
|
228
|
-
application should run its own generated-SQL and migration suite before a
|
|
229
|
-
cutover.
|
|
230
|
-
|
|
231
|
-
## 12. Testing pyramid
|
|
232
|
-
|
|
233
|
-
Use the smallest test that proves the invariant:
|
|
234
|
-
|
|
235
|
-
* unit specs for parsing, encoding, types, and isolated algorithms;
|
|
236
|
-
* component specs for storage, WAL, indexes, transactions, and protocol;
|
|
237
|
-
* integration specs for engine-to-SQL and server-to-client behavior;
|
|
238
|
-
* process specs for crash recovery, pooling, and failover;
|
|
239
|
-
* workload tests for sustained concurrency and resource limits; and
|
|
240
|
-
* example applications for real Rails and Ruby workflows.
|
|
241
|
-
|
|
242
|
-
Useful commands include:
|
|
243
|
-
|
|
244
|
-
```sh
|
|
245
|
-
bundle exec rspec
|
|
246
|
-
bundle exec rspec spec/sql spec/storage spec/transactions
|
|
247
|
-
ruby scripts/durability_drill
|
|
248
|
-
ruby scripts/production_soak
|
|
249
|
-
ruby scripts/fuzz
|
|
250
|
-
RUBYDB_BENCHMARK_ITERATIONS=100 ruby -Ilib benchmarks/basic_workload.rb
|
|
251
|
-
```
|
|
252
|
-
|
|
253
|
-
Record the Ruby version, commit, OS, command, seed, database configuration,
|
|
254
|
-
and artifact paths for every non-trivial run. A green unit suite does not
|
|
255
|
-
replace a restore drill, multi-process soak, or deployment test.
|
|
256
|
-
|
|
257
|
-
## 13. Debugging a failing change
|
|
258
|
-
|
|
259
|
-
Start with [Debugging RubyDB](debugging.md). Reproduce on a fresh temporary
|
|
260
|
-
directory, reduce the schema and SQL, rerun with the reported seed, and save
|
|
261
|
-
logs plus the WAL metadata. Compare embedded and server/client execution when
|
|
262
|
-
ownership or protocol is suspected. Use [Troubleshooting](troubleshooting.md)
|
|
263
|
-
for operator symptoms and evidence-preserving recovery steps.
|
|
264
|
-
|
|
265
|
-
Never “repair” a failing test by deleting WAL, disabling checksums, loosening a
|
|
266
|
-
constraint, or turning off synchronization. Preserve the failing directory,
|
|
267
|
-
copy it, and investigate the copy.
|
|
268
|
-
|
|
269
|
-
## 14. Performance and capacity work
|
|
270
|
-
|
|
271
|
-
Benchmark one variable at a time and report latency percentiles, throughput,
|
|
272
|
-
concurrency, payload shape, cache size, WAL/checkpoint settings, and storage
|
|
273
|
-
medium. Track p50, p95, p99, error rate, lock wait time, WAL growth, checkpoint
|
|
274
|
-
duration, memory, file descriptors, and CPU.
|
|
275
|
-
|
|
276
|
-
Capacity limits are workload-specific. A benchmark result is not a guarantee
|
|
277
|
-
for a Rails application with different indexes, query shapes, or connection
|
|
278
|
-
pool settings. Add a regression threshold only after the benchmark is stable
|
|
279
|
-
across repeated runs and environments.
|
|
280
|
-
|
|
281
|
-
## 15. Safe contribution checklist
|
|
282
|
-
|
|
283
|
-
Before opening a pull request:
|
|
284
|
-
|
|
285
|
-
1. Explain the invariant and compatibility impact.
|
|
286
|
-
2. Add focused tests, including failure paths.
|
|
287
|
-
3. Run the relevant focused suite and the full suite.
|
|
288
|
-
4. Run formatting/lint and `git diff --check`.
|
|
289
|
-
5. Update the appropriate spec and user/operator documentation.
|
|
290
|
-
6. Note migrations, format changes, recovery implications, and rollback plan.
|
|
291
|
-
7. Include benchmark or soak evidence for hot-path and concurrency changes.
|
|
292
|
-
8. Remove debug output, secrets, temporary data, and generated artifacts.
|
|
293
|
-
|
|
294
|
-
See [Contributing](../CONTRIBUTING.md), [testing](contributing/testing.md),
|
|
295
|
-
and [release process](contributing/release-process.md) for the repository
|
|
296
|
-
workflow.
|
|
297
|
-
|
|
1
|
+
# RubyDB developer guide
|
|
2
|
+
|
|
3
|
+
This is the implementation guide for contributors who need to change RubyDB
|
|
4
|
+
safely. It explains the repository, the request path, the storage invariants,
|
|
5
|
+
the test strategy, and the debugging workflow. Read it together with the
|
|
6
|
+
topic specifications under `spec/`; the specifications are the behavioral
|
|
7
|
+
contract, while this guide explains how the pieces fit together.
|
|
8
|
+
|
|
9
|
+
## 1. Scope and support promise
|
|
10
|
+
|
|
11
|
+
RubyDB is a Ruby-native relational database with two ownership modes:
|
|
12
|
+
|
|
13
|
+
* embedded mode, where one Ruby process owns a database directory; and
|
|
14
|
+
* server mode, where the server owns the directory and application processes
|
|
15
|
+
use the client protocol.
|
|
16
|
+
|
|
17
|
+
The supported surface is the behavior exercised by the test suite and listed
|
|
18
|
+
in [SQL compatibility](sql/compatibility.md). Similar syntax does not imply
|
|
19
|
+
complete PostgreSQL, MySQL, or SQLite compatibility. A change that broadens
|
|
20
|
+
syntax must also specify its semantics, errors, types, transactions, and
|
|
21
|
+
adapter behavior.
|
|
22
|
+
|
|
23
|
+
## 2. Repository orientation
|
|
24
|
+
|
|
25
|
+
The important directories are:
|
|
26
|
+
|
|
27
|
+
| Path | Responsibility |
|
|
28
|
+
| --- | --- |
|
|
29
|
+
| `lib/rubydb` | Engine, SQL, storage, transactions, server, client, adapters |
|
|
30
|
+
| `spec` | Unit, integration, protocol, SQL, recovery, and adapter contracts |
|
|
31
|
+
| `benchmarks` | Repeatable throughput and concurrency measurements |
|
|
32
|
+
| `scripts` | Soak tests, durability drills, fuzzing, and release checks |
|
|
33
|
+
| `config` | Example configuration and monitoring rules |
|
|
34
|
+
| `examples` | Runnable Ruby and Rails applications |
|
|
35
|
+
| `docs` | User, operator, developer, and compatibility documentation |
|
|
36
|
+
| `packaging` | Container and deployment examples |
|
|
37
|
+
| `exe` and `bin` | Public command-line entry points |
|
|
38
|
+
|
|
39
|
+
Start with `lib/rubydb.rb` and follow the public API into the engine. For a
|
|
40
|
+
feature, locate the nearest existing spec before editing implementation code.
|
|
41
|
+
Avoid adding a second abstraction when an existing subsystem already owns the
|
|
42
|
+
invariant.
|
|
43
|
+
|
|
44
|
+
## 3. Local setup
|
|
45
|
+
|
|
46
|
+
Use a supported Ruby version from the project CI matrix and install the locked
|
|
47
|
+
dependencies:
|
|
48
|
+
|
|
49
|
+
```sh
|
|
50
|
+
bundle install
|
|
51
|
+
bundle exec rspec
|
|
52
|
+
bundle exec rubocop
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Use a temporary database for experiments. Never use a production path in a
|
|
56
|
+
test or run a destructive command against an unknown directory:
|
|
57
|
+
|
|
58
|
+
```ruby
|
|
59
|
+
require "tmpdir"
|
|
60
|
+
require "rubydb"
|
|
61
|
+
|
|
62
|
+
Dir.mktmpdir("rubydb-dev-") do |dir|
|
|
63
|
+
engine = RubyDB::Storage::Engine.new(File.join(dir, "dev.rdb"))
|
|
64
|
+
engine.execute("CREATE TABLE items (id INTEGER PRIMARY KEY, name TEXT)")
|
|
65
|
+
engine.close
|
|
66
|
+
end
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
The server/client path is required when testing process boundaries. Multiple
|
|
70
|
+
independent embedded owners must never open the same database directory.
|
|
71
|
+
|
|
72
|
+
## 4. The request lifecycle
|
|
73
|
+
|
|
74
|
+
A typical SQL request follows this sequence:
|
|
75
|
+
|
|
76
|
+
```text
|
|
77
|
+
Ruby API or client
|
|
78
|
+
-> connection/session and authentication
|
|
79
|
+
-> SQL tokenizer/parser
|
|
80
|
+
-> binder and type/parameter validation
|
|
81
|
+
-> planner and executor
|
|
82
|
+
-> transaction/MVCC read or write set
|
|
83
|
+
-> table/index mutation
|
|
84
|
+
-> WAL append and durable commit
|
|
85
|
+
-> result encoding and protocol response
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
Keep failures at the layer that owns them. Parser errors should not be turned
|
|
89
|
+
into storage errors. A failed WAL append must not report a committed schema or
|
|
90
|
+
row change. A client timeout must not silently convert an unknown commit into
|
|
91
|
+
a rollback; the caller must be able to determine whether a request was
|
|
92
|
+
committed before retrying.
|
|
93
|
+
|
|
94
|
+
## 5. Storage and durability invariants
|
|
95
|
+
|
|
96
|
+
The storage directory contains data pages, metadata, WAL state, and recovery
|
|
97
|
+
artifacts. Changes must preserve these invariants:
|
|
98
|
+
|
|
99
|
+
1. A page has valid framing and checksum before it is trusted.
|
|
100
|
+
2. WAL records are validated before replay and are applied in LSN order.
|
|
101
|
+
3. A commit acknowledgement is emitted only after the configured durability
|
|
102
|
+
point has succeeded.
|
|
103
|
+
4. Schema publication and its dependent indexes are visible atomically.
|
|
104
|
+
5. Recovery is idempotent: replaying a committed record does not duplicate a
|
|
105
|
+
row or corrupt an index.
|
|
106
|
+
6. Torn, truncated, or corrupted input fails closed with a useful error.
|
|
107
|
+
7. Temporary files are not mistaken for a completed checkpoint or backup.
|
|
108
|
+
|
|
109
|
+
When changing page layout or record encoding, update the format specification,
|
|
110
|
+
versioning/upgrade path, compatibility tests, and recovery fixtures. Do not
|
|
111
|
+
silently reinterpret old bytes. If a format cannot be read, return an explicit
|
|
112
|
+
upgrade or corruption error and preserve the original files for diagnosis.
|
|
113
|
+
|
|
114
|
+
## 6. WAL, checkpoints, and recovery
|
|
115
|
+
|
|
116
|
+
The WAL is the source of truth between checkpoints. A checkpoint copies safe
|
|
117
|
+
state to durable pages and advances the recovery boundary only after all
|
|
118
|
+
required data and metadata have been flushed. Recovery should:
|
|
119
|
+
|
|
120
|
+
1. open the directory read-only where possible;
|
|
121
|
+
2. validate metadata and the last known checkpoint;
|
|
122
|
+
3. scan WAL frames, stopping only at a valid end boundary;
|
|
123
|
+
4. reject checksum, sequence, length, or transaction inconsistencies;
|
|
124
|
+
5. replay committed work and discard incomplete transactions;
|
|
125
|
+
6. rebuild or validate indexes before accepting writes; and
|
|
126
|
+
7. publish a recovery result with the replayed LSN and warnings.
|
|
127
|
+
|
|
128
|
+
Tests must cover normal reopen, a process terminated during a write, an
|
|
129
|
+
interrupted checkpoint, truncated WAL, invalid checksums, missing metadata,
|
|
130
|
+
full-disk behavior, and restore into a new directory. Fault injection belongs
|
|
131
|
+
around filesystem calls, not only around Ruby methods, because failures occur
|
|
132
|
+
at `fsync`, rename, allocation, and close boundaries.
|
|
133
|
+
|
|
134
|
+
## 7. Transactions, MVCC, and locking
|
|
135
|
+
|
|
136
|
+
Every statement executes in a transaction context, whether it is explicit or
|
|
137
|
+
implicit. The context owns the snapshot, read view, write set, lock state,
|
|
138
|
+
savepoints, and commit result. A transaction must not leak locks or snapshots
|
|
139
|
+
when it raises, times out, is cancelled, or loses its connection.
|
|
140
|
+
|
|
141
|
+
MVCC readers use a stable visibility point. Writers create new versions and
|
|
142
|
+
retain the before-image needed by active readers and rollback. Vacuum may
|
|
143
|
+
reclaim a version only after the global safe point has passed it. A new feature
|
|
144
|
+
must define behavior for:
|
|
145
|
+
|
|
146
|
+
* read committed, repeatable read, and serializable transactions;
|
|
147
|
+
* concurrent update of the same row;
|
|
148
|
+
* unique and foreign-key conflicts;
|
|
149
|
+
* deadlock detection and victim rollback;
|
|
150
|
+
* lock wait timeouts and request cancellation; and
|
|
151
|
+
* commit acknowledgement followed by client disconnect.
|
|
152
|
+
|
|
153
|
+
Deadlock resolution must abort a complete victim transaction, release all of
|
|
154
|
+
its locks, and leave other transactions able to progress. Never fix a deadlock
|
|
155
|
+
by globally disabling locking or by releasing a lock without undoing the
|
|
156
|
+
corresponding write set.
|
|
157
|
+
|
|
158
|
+
## 8. SQL implementation workflow
|
|
159
|
+
|
|
160
|
+
For a new statement or expression:
|
|
161
|
+
|
|
162
|
+
1. write the syntax and semantic contract in `spec/sql`;
|
|
163
|
+
2. add parser acceptance and rejection examples;
|
|
164
|
+
3. add binder/type/nullability behavior;
|
|
165
|
+
4. add planner and executor tests;
|
|
166
|
+
5. test transaction, constraint, index, and error interactions;
|
|
167
|
+
6. test the Ruby API and server protocol path; and
|
|
168
|
+
7. test the ActiveRecord-generated SQL when the feature is adapter-visible.
|
|
169
|
+
|
|
170
|
+
Prefer parameter binding to string interpolation. Every expression needs
|
|
171
|
+
defined behavior for `NULL`, booleans, numeric coercion, text comparison,
|
|
172
|
+
collation, and invalid input. Every DDL operation needs idempotence or an
|
|
173
|
+
explicit error contract. Every optimizer rewrite must have a semantic
|
|
174
|
+
equivalence test against the non-optimized execution path.
|
|
175
|
+
|
|
176
|
+
The documented common SQLite-style profile is intentionally narrower than
|
|
177
|
+
SQLite itself. Do not label a feature “SQLite compatible” until its syntax,
|
|
178
|
+
results, types, error behavior, and migration behavior are tested.
|
|
179
|
+
|
|
180
|
+
## 9. Tables, indexes, and constraints
|
|
181
|
+
|
|
182
|
+
Table mutations and index mutations are one logical operation. If an index
|
|
183
|
+
write fails, the statement must fail visibly and the transaction must either
|
|
184
|
+
roll back or retain a recoverable pending state; it must not acknowledge a row
|
|
185
|
+
that cannot be found by a required index.
|
|
186
|
+
|
|
187
|
+
For each index type, test empty and populated tables, duplicate keys, `NULL`,
|
|
188
|
+
deep splits, reopen, recovery replay, deletion/merge, and concurrent readers.
|
|
189
|
+
For each constraint, test direct SQL, prepared parameters, ActiveRecord
|
|
190
|
+
inserts/updates, rollback, and the exact error class/message contract where
|
|
191
|
+
callers may depend on it.
|
|
192
|
+
|
|
193
|
+
## 10. Server, client, and wire protocol
|
|
194
|
+
|
|
195
|
+
The server is the single owner of the database directory. A client session
|
|
196
|
+
handles authentication, capability negotiation, request IDs, deadlines,
|
|
197
|
+
cancellation, result framing, and connection shutdown. Protocol changes must
|
|
198
|
+
be backward-compatible or carry an explicit protocol version and rejection
|
|
199
|
+
path.
|
|
200
|
+
|
|
201
|
+
Wire cancellation is part of correctness. A cancelled request must stop
|
|
202
|
+
execution at safe checkpoints, release transaction resources, and return a
|
|
203
|
+
definitive cancellation response. If cancellation races with commit, the
|
|
204
|
+
server must preserve the commit outcome and expose enough request identity for
|
|
205
|
+
the client to query status rather than blindly retrying.
|
|
206
|
+
|
|
207
|
+
Test malformed lengths, unknown message types, duplicate request IDs, partial
|
|
208
|
+
frames, client disconnects, server shutdown, timeout races, authentication
|
|
209
|
+
failure, TLS negotiation, and cancellation during a long scan.
|
|
210
|
+
|
|
211
|
+
## 11. Rails and adapter work
|
|
212
|
+
|
|
213
|
+
The ActiveRecord adapter translates Rails schema and query APIs into RubyDB
|
|
214
|
+
SQL. Adapter code must preserve Rails expectations for quoting, bind
|
|
215
|
+
parameters, affected-row counts, last-insert IDs, transactions, savepoints,
|
|
216
|
+
schema introspection, migration versions, and exceptions.
|
|
217
|
+
|
|
218
|
+
When changing adapter behavior, run the example app and test at least:
|
|
219
|
+
|
|
220
|
+
* `where`, ordering, limits, scopes, joins, and aggregate relations;
|
|
221
|
+
* eager loading, nested associations, and inverse relationships;
|
|
222
|
+
* connection pools with multiple threads;
|
|
223
|
+
* schema dump/load and populated-table migrations;
|
|
224
|
+
* rollback, retry, and migration checksum behavior; and
|
|
225
|
+
* every supported Ruby/Rails combination in CI.
|
|
226
|
+
|
|
227
|
+
The adapter is not proof of complete SQLite or PostgreSQL compatibility. An
|
|
228
|
+
application should run its own generated-SQL and migration suite before a
|
|
229
|
+
cutover.
|
|
230
|
+
|
|
231
|
+
## 12. Testing pyramid
|
|
232
|
+
|
|
233
|
+
Use the smallest test that proves the invariant:
|
|
234
|
+
|
|
235
|
+
* unit specs for parsing, encoding, types, and isolated algorithms;
|
|
236
|
+
* component specs for storage, WAL, indexes, transactions, and protocol;
|
|
237
|
+
* integration specs for engine-to-SQL and server-to-client behavior;
|
|
238
|
+
* process specs for crash recovery, pooling, and failover;
|
|
239
|
+
* workload tests for sustained concurrency and resource limits; and
|
|
240
|
+
* example applications for real Rails and Ruby workflows.
|
|
241
|
+
|
|
242
|
+
Useful commands include:
|
|
243
|
+
|
|
244
|
+
```sh
|
|
245
|
+
bundle exec rspec
|
|
246
|
+
bundle exec rspec spec/sql spec/storage spec/transactions
|
|
247
|
+
ruby scripts/durability_drill
|
|
248
|
+
ruby scripts/production_soak
|
|
249
|
+
ruby scripts/fuzz
|
|
250
|
+
RUBYDB_BENCHMARK_ITERATIONS=100 ruby -Ilib benchmarks/basic_workload.rb
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
Record the Ruby version, commit, OS, command, seed, database configuration,
|
|
254
|
+
and artifact paths for every non-trivial run. A green unit suite does not
|
|
255
|
+
replace a restore drill, multi-process soak, or deployment test.
|
|
256
|
+
|
|
257
|
+
## 13. Debugging a failing change
|
|
258
|
+
|
|
259
|
+
Start with [Debugging RubyDB](debugging.md). Reproduce on a fresh temporary
|
|
260
|
+
directory, reduce the schema and SQL, rerun with the reported seed, and save
|
|
261
|
+
logs plus the WAL metadata. Compare embedded and server/client execution when
|
|
262
|
+
ownership or protocol is suspected. Use [Troubleshooting](troubleshooting.md)
|
|
263
|
+
for operator symptoms and evidence-preserving recovery steps.
|
|
264
|
+
|
|
265
|
+
Never “repair” a failing test by deleting WAL, disabling checksums, loosening a
|
|
266
|
+
constraint, or turning off synchronization. Preserve the failing directory,
|
|
267
|
+
copy it, and investigate the copy.
|
|
268
|
+
|
|
269
|
+
## 14. Performance and capacity work
|
|
270
|
+
|
|
271
|
+
Benchmark one variable at a time and report latency percentiles, throughput,
|
|
272
|
+
concurrency, payload shape, cache size, WAL/checkpoint settings, and storage
|
|
273
|
+
medium. Track p50, p95, p99, error rate, lock wait time, WAL growth, checkpoint
|
|
274
|
+
duration, memory, file descriptors, and CPU.
|
|
275
|
+
|
|
276
|
+
Capacity limits are workload-specific. A benchmark result is not a guarantee
|
|
277
|
+
for a Rails application with different indexes, query shapes, or connection
|
|
278
|
+
pool settings. Add a regression threshold only after the benchmark is stable
|
|
279
|
+
across repeated runs and environments.
|
|
280
|
+
|
|
281
|
+
## 15. Safe contribution checklist
|
|
282
|
+
|
|
283
|
+
Before opening a pull request:
|
|
284
|
+
|
|
285
|
+
1. Explain the invariant and compatibility impact.
|
|
286
|
+
2. Add focused tests, including failure paths.
|
|
287
|
+
3. Run the relevant focused suite and the full suite.
|
|
288
|
+
4. Run formatting/lint and `git diff --check`.
|
|
289
|
+
5. Update the appropriate spec and user/operator documentation.
|
|
290
|
+
6. Note migrations, format changes, recovery implications, and rollback plan.
|
|
291
|
+
7. Include benchmark or soak evidence for hot-path and concurrency changes.
|
|
292
|
+
8. Remove debug output, secrets, temporary data, and generated artifacts.
|
|
293
|
+
|
|
294
|
+
See [Contributing](../CONTRIBUTING.md), [testing](contributing/testing.md),
|
|
295
|
+
and [release process](contributing/release-process.md) for the repository
|
|
296
|
+
workflow.
|
|
297
|
+
|
|
@@ -1,16 +1,16 @@
|
|
|
1
|
-
# First database
|
|
2
|
-
|
|
3
|
-
```ruby
|
|
4
|
-
require "rubydb"
|
|
5
|
-
|
|
6
|
-
path = "tmp/first-database.rdb"
|
|
7
|
-
engine = RubyDB::Storage::Engine.new(path)
|
|
8
|
-
engine.execute("CREATE TABLE accounts (id INTEGER PRIMARY KEY, email TEXT UNIQUE NOT NULL)")
|
|
9
|
-
engine.execute("INSERT INTO accounts (id, email) VALUES (1, 'ada@example.test')")
|
|
10
|
-
puts engine.execute("SELECT id, email FROM accounts").inspect
|
|
11
|
-
engine.close
|
|
12
|
-
```
|
|
13
|
-
|
|
14
|
-
The engine creates durable storage and WAL state under the selected path. Close
|
|
15
|
-
the engine cleanly and keep the database, WAL, and lock files together. A single
|
|
16
|
-
embedded path has one owner; use a server for multiple processes.
|
|
1
|
+
# First database
|
|
2
|
+
|
|
3
|
+
```ruby
|
|
4
|
+
require "rubydb"
|
|
5
|
+
|
|
6
|
+
path = "tmp/first-database.rdb"
|
|
7
|
+
engine = RubyDB::Storage::Engine.new(path)
|
|
8
|
+
engine.execute("CREATE TABLE accounts (id INTEGER PRIMARY KEY, email TEXT UNIQUE NOT NULL)")
|
|
9
|
+
engine.execute("INSERT INTO accounts (id, email) VALUES (1, 'ada@example.test')")
|
|
10
|
+
puts engine.execute("SELECT id, email FROM accounts").inspect
|
|
11
|
+
engine.close
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
The engine creates durable storage and WAL state under the selected path. Close
|
|
15
|
+
the engine cleanly and keep the database, WAL, and lock files together. A single
|
|
16
|
+
embedded path has one owner; use a server for multiple processes.
|
|
@@ -1,13 +1,13 @@
|
|
|
1
|
-
# First query
|
|
2
|
-
|
|
3
|
-
RubyDB executes the documented SQL subset through its lexer, parser, planner,
|
|
4
|
-
and executor:
|
|
5
|
-
|
|
6
|
-
```ruby
|
|
7
|
-
rows = engine.execute("SELECT id, email FROM accounts WHERE id = 1 ORDER BY id LIMIT 10")
|
|
8
|
-
puts rows.inspect
|
|
9
|
-
```
|
|
10
|
-
|
|
11
|
-
Use the Rails connection or client binding APIs for external values. Do not
|
|
12
|
-
interpolate untrusted input into SQL. See [SQL compatibility](../sql/compatibility.md)
|
|
13
|
-
for supported statements and explicit boundaries.
|
|
1
|
+
# First query
|
|
2
|
+
|
|
3
|
+
RubyDB executes the documented SQL subset through its lexer, parser, planner,
|
|
4
|
+
and executor:
|
|
5
|
+
|
|
6
|
+
```ruby
|
|
7
|
+
rows = engine.execute("SELECT id, email FROM accounts WHERE id = 1 ORDER BY id LIMIT 10")
|
|
8
|
+
puts rows.inspect
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
Use the Rails connection or client binding APIs for external values. Do not
|
|
12
|
+
interpolate untrusted input into SQL. See [SQL compatibility](../sql/compatibility.md)
|
|
13
|
+
for supported statements and explicit boundaries.
|