rubydb 0.1.4 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.github/PULL_REQUEST_TEMPLATE.md +15 -15
- data/.github/workflows/benchmark.yml +26 -26
- data/.github/workflows/compatibility.yml +63 -63
- data/.github/workflows/fuzz.yml +33 -33
- data/.github/workflows/lint.yml +21 -21
- data/.github/workflows/operations.yml +24 -24
- data/.github/workflows/production-validation.yml +111 -111
- data/.github/workflows/release.yml +77 -77
- data/.github/workflows/security.yml +39 -37
- data/.github/workflows/test.yml +26 -26
- data/.github/workflows/workload.yml +58 -58
- data/.gitignore +16 -5
- data/.rubocop.yml +50 -44
- data/.standard.yml +9 -14
- data/ARCHITECTURE.md +21 -21
- data/CHANGELOG.md +57 -27
- data/CODE_OF_CONDUCT.md +13 -13
- data/CONTRIBUTING.md +29 -29
- data/GOVERNANCE.md +16 -16
- data/Gemfile +18 -17
- data/Gemfile.lock +125 -71
- data/README.md +168 -12
- data/ROADMAP.md +27 -27
- data/Rakefile +76 -71
- data/SECURITY.md +54 -54
- data/SUPPORT.md +14 -14
- data/accelerator/bin/SHA256SUMS +6 -0
- data/accelerator/bin/rubydb-accelerator-darwin-amd64 +0 -0
- data/accelerator/bin/rubydb-accelerator-darwin-arm64 +0 -0
- data/accelerator/bin/rubydb-accelerator-linux-amd64 +0 -0
- data/accelerator/bin/rubydb-accelerator-linux-arm64 +0 -0
- data/accelerator/bin/rubydb-accelerator-windows-amd64.exe +0 -0
- data/accelerator/bin/rubydb-accelerator-windows-arm64.exe +0 -0
- data/accelerator/cmd/rubydb-accelerator/main.go +11 -0
- data/accelerator/go.mod +3 -0
- data/accelerator/internal/execution/aggregate.go +94 -0
- data/accelerator/internal/execution/distinct.go +22 -0
- data/accelerator/internal/execution/filter.go +73 -0
- data/accelerator/internal/execution/join.go +79 -0
- data/accelerator/internal/execution/operators.go +167 -0
- data/accelerator/internal/execution/scan.go +20 -0
- data/accelerator/internal/execution/sort.go +62 -0
- data/accelerator/internal/execution/types.go +136 -0
- data/accelerator/internal/execution/value.go +67 -0
- data/accelerator/internal/memory/arena.go +47 -0
- data/accelerator/internal/memory/reuse.go +22 -0
- data/accelerator/internal/metrics/registry.go +67 -0
- data/accelerator/internal/parallel/bounded_queue.go +56 -0
- data/accelerator/internal/parallel/scheduler.go +47 -0
- data/accelerator/internal/parallel/worker_pool.go +53 -0
- data/accelerator/internal/protocol/cancellation.go +48 -0
- data/accelerator/internal/protocol/columnar.go +263 -0
- data/accelerator/internal/protocol/frame.go +187 -0
- data/accelerator/internal/runtime/worker.go +521 -0
- data/accelerator/internal/storage/page_reader.go +81 -0
- data/accelerator/internal/storage/snapshot_scan.go +539 -0
- data/accelerator/internal/wal/checksum.go +13 -0
- data/accelerator/internal/wal/compression.go +41 -0
- data/accelerator/internal/wal/group_commit.go +24 -0
- data/accelerator/internal/wal/record_encoder.go +40 -0
- data/adapters/activerecord/Gemfile +11 -11
- data/adapters/activerecord/README.md +8 -3
- data/adapters/activerecord/lib/active_record/connection_adapters/rubydb_adapter.rb +881 -879
- data/adapters/activerecord/rubydb-activerecord.gemspec +21 -21
- data/adapters/activerecord/spec/rubydb_adapter_integration_spec.rb +143 -143
- data/adapters/ruby/README.md +18 -18
- data/adapters/sequel/README.md +11 -11
- data/config/monitoring/prometheus-alerts.yml +39 -39
- data/config/production.yml +36 -36
- data/docs/README.md +77 -71
- data/docs/architecture/concurrency.md +14 -14
- data/docs/architecture/current-state.md +125 -125
- data/docs/architecture/execution-engine.md +25 -25
- data/docs/architecture/go-accelerator.md +179 -0
- data/docs/architecture/indexes.md +19 -19
- data/docs/architecture/mvcc.md +19 -19
- data/docs/architecture/overview.md +13 -13
- data/docs/architecture/pages.md +11 -11
- data/docs/architecture/production-roadmap.md +82 -82
- data/docs/architecture/query-planner.md +20 -20
- data/docs/architecture/recovery.md +18 -18
- data/docs/architecture/sql-engine.md +12 -12
- data/docs/architecture/storage-engine.md +14 -14
- data/docs/architecture/transactions.md +10 -10
- data/docs/architecture/wal.md +28 -28
- data/docs/cli-cheatsheet.md +98 -98
- data/docs/cli.md +299 -275
- data/docs/contributing/architecture.md +9 -9
- data/docs/contributing/benchmarking.md +30 -14
- data/docs/contributing/development.md +16 -16
- data/docs/contributing/release-process.md +49 -49
- data/docs/contributing/testing.md +16 -16
- data/docs/debugging.md +229 -229
- data/docs/developer/branching.md +10 -10
- data/docs/developer/database-diff.md +10 -10
- data/docs/developer/local-development.md +49 -17
- data/docs/developer/snapshots.md +9 -9
- data/docs/developer/temporal-data.md +10 -10
- data/docs/developer-guide.md +297 -297
- data/docs/getting-started/first-database.md +16 -16
- data/docs/getting-started/first-query.md +13 -13
- data/docs/getting-started/installation.md +19 -19
- data/docs/getting-started/local-to-production.md +300 -300
- data/docs/getting-started/quickstart.md +17 -17
- data/docs/getting-started/rails.md +16 -16
- data/docs/hardening_backlog.md +93 -93
- data/docs/lessons-learned.md +112 -112
- data/docs/operations/backups.md +33 -33
- data/docs/operations/disaster-recovery.md +31 -31
- data/docs/operations/failover.md +30 -30
- data/docs/operations/monitoring.md +25 -25
- data/docs/operations/production-guide.md +295 -295
- data/docs/operations/production-runbook.md +45 -45
- data/docs/operations/replication.md +33 -33
- data/docs/operations/restore.md +6 -6
- data/docs/operations/runbook.md +34 -34
- data/docs/operations/upgrades.md +14 -14
- data/docs/operations/workload-testing.md +17 -17
- data/docs/production-readiness.md +118 -118
- data/docs/production_validation.md +150 -150
- data/docs/rails/active-record.md +11 -11
- data/docs/rails/compatibility-guide.md +90 -90
- data/docs/rails/database-yml.md +92 -92
- data/docs/rails/installation.md +17 -17
- data/docs/rails/migrations.md +17 -17
- data/docs/rails/production.md +82 -82
- data/docs/rails/troubleshooting.md +18 -18
- data/docs/release.md +25 -25
- data/docs/server/architecture.md +10 -10
- data/docs/server/authentication.md +10 -10
- data/docs/server/configuration.md +16 -16
- data/docs/server/connection-pooling.md +10 -10
- data/docs/server/deployment.md +10 -10
- data/docs/server/protocol.md +12 -12
- data/docs/sql/compatibility-guide.md +82 -82
- data/docs/sql/compatibility.md +39 -39
- data/docs/sql/data-types.md +10 -10
- data/docs/sql/functions.md +9 -9
- data/docs/sql/joins.md +9 -9
- data/docs/sql/operators.md +9 -9
- data/docs/sql/sqlite-compatibility.md +21 -21
- data/docs/sql/syntax.md +10 -10
- data/docs/sql/transactions.md +10 -10
- data/docs/troubleshooting.md +244 -244
- data/lessons/01-foundations.md +73 -0
- data/lessons/02-local-development.md +121 -0
- data/lessons/03-embedded-rubydb.md +99 -0
- data/lessons/04-rails-complex-apps.md +138 -0
- data/lessons/05-rubydb-production-server.md +237 -0
- data/lessons/06-postgresql-massive-apps.md +96 -0
- data/lessons/07-hybrid-microservices.md +179 -0
- data/lessons/08-migrations-backups-recovery.md +86 -0
- data/lessons/09-observability-security-scale.md +87 -0
- data/lessons/10-release-readiness.md +192 -0
- data/lessons/11-community-adapter.md +323 -0
- data/lessons/12-rails-ecommerce-pressure.md +263 -0
- data/lib/rubydb/accelerator/client.rb +451 -0
- data/lib/rubydb/accelerator/error.rb +22 -0
- data/lib/rubydb/accelerator/manager.rb +606 -0
- data/lib/rubydb/accelerator.rb +13 -0
- data/lib/rubydb/backup/archive.rb +332 -334
- data/lib/rubydb/backup/backup.rb +400 -401
- data/lib/rubydb/backup/incremental.rb +349 -353
- data/lib/rubydb/backup/restore.rb +289 -290
- data/lib/rubydb/backup/snapshot.rb +265 -267
- data/lib/rubydb/backup/verification.rb +276 -279
- data/lib/rubydb/branching/branch.rb +181 -181
- data/lib/rubydb/branching/branch_manager.rb +307 -311
- data/lib/rubydb/branching/branch_metadata.rb +140 -140
- data/lib/rubydb/branching/checkout.rb +165 -166
- data/lib/rubydb/branching/copy_on_write.rb +272 -272
- data/lib/rubydb/branching/diff.rb +137 -138
- data/lib/rubydb/branching/merge.rb +282 -285
- data/lib/rubydb/build_info.rb +15 -15
- data/lib/rubydb/catalog/catalog.rb +391 -391
- data/lib/rubydb/catalog/column.rb +112 -112
- data/lib/rubydb/catalog/constraint.rb +180 -180
- data/lib/rubydb/catalog/database.rb +184 -184
- data/lib/rubydb/catalog/index.rb +97 -97
- data/lib/rubydb/catalog/schema.rb +103 -103
- data/lib/rubydb/catalog/sequence.rb +90 -90
- data/lib/rubydb/catalog/system_catalog.rb +698 -698
- data/lib/rubydb/catalog/table.rb +178 -178
- data/lib/rubydb/catalog/trigger.rb +102 -102
- data/lib/rubydb/catalog/view.rb +66 -66
- data/lib/rubydb/cli/application.rb +168 -163
- data/lib/rubydb/cli/commands/accelerator.rb +72 -0
- data/lib/rubydb/cli/commands/backup.rb +80 -81
- data/lib/rubydb/cli/commands/branch.rb +72 -72
- data/lib/rubydb/cli/commands/checkout.rb +54 -54
- data/lib/rubydb/cli/commands/create.rb +58 -58
- data/lib/rubydb/cli/commands/diff.rb +76 -77
- data/lib/rubydb/cli/commands/doctor.rb +77 -74
- data/lib/rubydb/cli/commands/drop.rb +57 -57
- data/lib/rubydb/cli/commands/init.rb +101 -102
- data/lib/rubydb/cli/commands/inspect.rb +95 -95
- data/lib/rubydb/cli/commands/merge.rb +63 -63
- data/lib/rubydb/cli/commands/migrate.rb +62 -62
- data/lib/rubydb/cli/commands/restart.rb +42 -39
- data/lib/rubydb/cli/commands/restore.rb +121 -121
- data/lib/rubydb/cli/commands/shell.rb +365 -365
- data/lib/rubydb/cli/commands/snapshot.rb +79 -79
- data/lib/rubydb/cli/commands/start.rb +88 -82
- data/lib/rubydb/cli/commands/status.rb +96 -92
- data/lib/rubydb/cli/commands/stop.rb +47 -47
- data/lib/rubydb/cli/commands/vacuum.rb +58 -58
- data/lib/rubydb/cli/formatter.rb +221 -221
- data/lib/rubydb/cli/output.rb +168 -168
- data/lib/rubydb/client/client.rb +309 -304
- data/lib/rubydb/client/connection.rb +429 -415
- data/lib/rubydb/client/connection_pool.rb +168 -168
- data/lib/rubydb/client/connection_url.rb +96 -96
- data/lib/rubydb/client/prepared_statement.rb +60 -60
- data/lib/rubydb/client/result.rb +127 -123
- data/lib/rubydb/client/statement.rb +52 -52
- data/lib/rubydb/client/transaction.rb +130 -130
- data/lib/rubydb/concurrency/concurrency.rb +19 -19
- data/lib/rubydb/concurrency/deadlock_detector.rb +148 -150
- data/lib/rubydb/concurrency/latch.rb +101 -101
- data/lib/rubydb/concurrency/lock_graph.rb +163 -165
- data/lib/rubydb/concurrency/mutex.rb +181 -183
- data/lib/rubydb/concurrency/rw_lock.rb +180 -180
- data/lib/rubydb/concurrency/scheduler.rb +248 -250
- data/lib/rubydb/concurrency/worker_pool.rb +145 -143
- data/lib/rubydb/configuration/config.rb +170 -170
- data/lib/rubydb/configuration/defaults.rb +191 -179
- data/lib/rubydb/configuration/environment.rb +152 -152
- data/lib/rubydb/configuration/parser.rb +185 -185
- data/lib/rubydb/configuration/validation.rb +228 -221
- data/lib/rubydb/constants.rb +74 -74
- data/lib/rubydb/constraints/check.rb +181 -181
- data/lib/rubydb/constraints/constraint.rb +101 -101
- data/lib/rubydb/constraints/foreign_key.rb +130 -130
- data/lib/rubydb/constraints/not_null.rb +64 -64
- data/lib/rubydb/constraints/primary_key.rb +99 -99
- data/lib/rubydb/constraints/unique.rb +106 -108
- data/lib/rubydb/constraints/validator.rb +349 -350
- data/lib/rubydb/errors/authentication_error.rb +10 -10
- data/lib/rubydb/errors/authorization_error.rb +23 -23
- data/lib/rubydb/errors/client_error.rb +10 -10
- data/lib/rubydb/errors/configuration_error.rb +10 -10
- data/lib/rubydb/errors/connection_error.rb +10 -10
- data/lib/rubydb/errors/constraint_error.rb +23 -23
- data/lib/rubydb/errors/corruption_error.rb +10 -10
- data/lib/rubydb/errors/database_error.rb +10 -10
- data/lib/rubydb/errors/error.rb +20 -20
- data/lib/rubydb/errors/execution_error.rb +10 -10
- data/lib/rubydb/errors/parser_error.rb +10 -10
- data/lib/rubydb/errors/recovery_error.rb +10 -10
- data/lib/rubydb/errors/replication_error.rb +10 -10
- data/lib/rubydb/errors/server_error.rb +6 -6
- data/lib/rubydb/errors/storage_error.rb +10 -10
- data/lib/rubydb/errors/transaction_error.rb +10 -10
- data/lib/rubydb/execution/accelerator_dispatch.rb +30 -0
- data/lib/rubydb/execution/aggregate_executor.rb +134 -138
- data/lib/rubydb/execution/cost_model.rb +72 -0
- data/lib/rubydb/execution/delete_executor.rb +110 -112
- data/lib/rubydb/execution/distinct_executor.rb +131 -135
- data/lib/rubydb/execution/executor.rb +1544 -1188
- data/lib/rubydb/execution/expression.rb +191 -193
- data/lib/rubydb/execution/index_scan.rb +142 -142
- data/lib/rubydb/execution/insert_executor.rb +215 -217
- data/lib/rubydb/execution/join_executor.rb +243 -249
- data/lib/rubydb/execution/limit_executor.rb +83 -85
- data/lib/rubydb/execution/operator_selection.rb +57 -0
- data/lib/rubydb/execution/optimizer.rb +227 -215
- data/lib/rubydb/execution/physical_plan.rb +47 -0
- data/lib/rubydb/execution/plan.rb +355 -353
- data/lib/rubydb/execution/planner.rb +508 -536
- data/lib/rubydb/execution/predicate.rb +235 -235
- data/lib/rubydb/execution/scan.rb +49 -49
- data/lib/rubydb/execution/sequential_scan.rb +63 -63
- data/lib/rubydb/execution/sort_executor.rb +194 -185
- data/lib/rubydb/execution/update_executor.rb +160 -162
- data/lib/rubydb/functions/aggregate.rb +70 -70
- data/lib/rubydb/functions/date_functions.rb +274 -278
- data/lib/rubydb/functions/function.rb +85 -85
- data/lib/rubydb/functions/json_functions.rb +231 -215
- data/lib/rubydb/functions/numeric_functions.rb +346 -346
- data/lib/rubydb/functions/scalar.rb +52 -52
- data/lib/rubydb/functions/string_functions.rb +383 -383
- data/lib/rubydb/functions/system_functions.rb +258 -246
- data/lib/rubydb/history/as_of.rb +238 -238
- data/lib/rubydb/history/change.rb +105 -105
- data/lib/rubydb/history/history.rb +131 -131
- data/lib/rubydb/history/history_manager.rb +228 -229
- data/lib/rubydb/history/temporal_query.rb +202 -202
- data/lib/rubydb/history/timeline.rb +144 -144
- data/lib/rubydb/indexes/btree.rb +215 -186
- data/lib/rubydb/indexes/btree_cursor.rb +258 -258
- data/lib/rubydb/indexes/btree_node.rb +384 -385
- data/lib/rubydb/indexes/hash_index.rb +150 -150
- data/lib/rubydb/indexes/index.rb +71 -71
- data/lib/rubydb/indexes/index_manager.rb +408 -406
- data/lib/rubydb/indexes/index_scan.rb +466 -470
- data/lib/rubydb/migrations/migration.rb +253 -254
- data/lib/rubydb/migrations/migration_lock.rb +146 -146
- data/lib/rubydb/migrations/migration_manager.rb +187 -176
- data/lib/rubydb/migrations/migration_version.rb +71 -71
- data/lib/rubydb/migrations/schema_diff.rb +211 -211
- data/lib/rubydb/migrations/schema_version.rb +64 -64
- data/lib/rubydb/monitoring/events.rb +155 -160
- data/lib/rubydb/monitoring/health.rb +216 -222
- data/lib/rubydb/monitoring/logger.rb +188 -193
- data/lib/rubydb/monitoring/metrics.rb +363 -359
- data/lib/rubydb/monitoring/performance.rb +176 -176
- data/lib/rubydb/monitoring/statistics.rb +168 -170
- data/lib/rubydb/mvcc/garbage_collector.rb +199 -199
- data/lib/rubydb/mvcc/mvcc.rb +16 -16
- data/lib/rubydb/mvcc/snapshot.rb +146 -147
- data/lib/rubydb/mvcc/vacuum.rb +180 -180
- data/lib/rubydb/mvcc/version.rb +106 -106
- data/lib/rubydb/mvcc/version_store.rb +396 -398
- data/lib/rubydb/mvcc/visibility.rb +107 -109
- data/lib/rubydb/protocol/capabilities.rb +125 -125
- data/lib/rubydb/protocol/decoder.rb +142 -145
- data/lib/rubydb/protocol/encoder.rb +131 -136
- data/lib/rubydb/protocol/handshake.rb +306 -305
- data/lib/rubydb/protocol/message.rb +121 -121
- data/lib/rubydb/protocol/parameter_binder.rb +101 -0
- data/lib/rubydb/protocol/protocol.rb +276 -277
- data/lib/rubydb/protocol/version.rb +54 -54
- data/lib/rubydb/rails/adapter.rb +245 -239
- data/lib/rubydb/rails/connection.rb +312 -314
- data/lib/rubydb/rails/database_statements.rb +122 -122
- data/lib/rubydb/rails/migration.rb +131 -131
- data/lib/rubydb/rails/quoting.rb +109 -109
- data/lib/rubydb/rails/result.rb +117 -117
- data/lib/rubydb/rails/schema_statements.rb +339 -339
- data/lib/rubydb/rails/transaction.rb +105 -105
- data/lib/rubydb/rails/type.rb +126 -126
- data/lib/rubydb/recovery/checkpoint.rb +261 -257
- data/lib/rubydb/recovery/consistency.rb +457 -467
- data/lib/rubydb/recovery/corruption_detector.rb +5 -5
- data/lib/rubydb/recovery/crash_recovery.rb +381 -387
- data/lib/rubydb/recovery/recovery_manager.rb +204 -206
- data/lib/rubydb/recovery/redo.rb +235 -237
- data/lib/rubydb/recovery/undo.rb +204 -206
- data/lib/rubydb/replication/failover.rb +5 -5
- data/lib/rubydb/replication/fencing.rb +63 -63
- data/lib/rubydb/replication/primary.rb +461 -450
- data/lib/rubydb/replication/replica.rb +382 -384
- data/lib/rubydb/replication/replication_log.rb +194 -200
- data/lib/rubydb/replication/replication_manager.rb +307 -308
- data/lib/rubydb/replication/replication_slot.rb +293 -295
- data/lib/rubydb/replication/replication_stream.rb +198 -201
- data/lib/rubydb/rubydb.rb +570 -560
- data/lib/rubydb/security/access_control.rb +252 -254
- data/lib/rubydb/security/audit_log.rb +209 -213
- data/lib/rubydb/security/authentication.rb +302 -302
- data/lib/rubydb/security/authorization.rb +282 -282
- data/lib/rubydb/security/credentials.rb +192 -196
- data/lib/rubydb/security/password.rb +205 -215
- data/lib/rubydb/security/permissions.rb +74 -74
- data/lib/rubydb/security/role.rb +99 -101
- data/lib/rubydb/security/user.rb +86 -86
- data/lib/rubydb/server/connection.rb +383 -366
- data/lib/rubydb/server/connection_pool.rb +193 -193
- data/lib/rubydb/server/lifecycle.rb +227 -228
- data/lib/rubydb/server/listener.rb +139 -136
- data/lib/rubydb/server/request_handler.rb +277 -276
- data/lib/rubydb/server/server.rb +363 -364
- data/lib/rubydb/server/session.rb +416 -369
- data/lib/rubydb/server/worker.rb +206 -210
- data/lib/rubydb/server/worker_pool.rb +168 -168
- data/lib/rubydb/sql/ast/alter_table.rb +169 -169
- data/lib/rubydb/sql/ast/begin_transaction.rb +47 -47
- data/lib/rubydb/sql/ast/commit.rb +37 -37
- data/lib/rubydb/sql/ast/constraint.rb +92 -83
- data/lib/rubydb/sql/ast/create_database.rb +41 -41
- data/lib/rubydb/sql/ast/create_index.rb +61 -61
- data/lib/rubydb/sql/ast/create_schema.rb +52 -52
- data/lib/rubydb/sql/ast/create_table.rb +187 -187
- data/lib/rubydb/sql/ast/delete.rb +54 -54
- data/lib/rubydb/sql/ast/drop_database.rb +41 -41
- data/lib/rubydb/sql/ast/drop_index.rb +41 -41
- data/lib/rubydb/sql/ast/drop_schema.rb +49 -49
- data/lib/rubydb/sql/ast/drop_table.rb +49 -49
- data/lib/rubydb/sql/ast/explain.rb +64 -64
- data/lib/rubydb/sql/ast/expression.rb +617 -604
- data/lib/rubydb/sql/ast/insert.rb +66 -66
- data/lib/rubydb/sql/ast/node.rb +42 -42
- data/lib/rubydb/sql/ast/rollback.rb +63 -63
- data/lib/rubydb/sql/ast/savepoint.rb +59 -59
- data/lib/rubydb/sql/ast/select.rb +88 -88
- data/lib/rubydb/sql/ast/set_operation.rb +22 -20
- data/lib/rubydb/sql/ast/trigger.rb +35 -29
- data/lib/rubydb/sql/ast/update.rb +88 -88
- data/lib/rubydb/sql/ast/vacuum.rb +19 -19
- data/lib/rubydb/sql/ast/view.rb +38 -32
- data/lib/rubydb/sql/ast/with.rb +32 -32
- data/lib/rubydb/sql/grammar.rb +86 -86
- data/lib/rubydb/sql/keywords.rb +156 -156
- data/lib/rubydb/sql/lexer.rb +209 -214
- data/lib/rubydb/sql/operators.rb +100 -100
- data/lib/rubydb/sql/parser.rb +1167 -1170
- data/lib/rubydb/sql/planner/analyzer.rb +283 -302
- data/lib/rubydb/sql/planner/binder.rb +537 -550
- data/lib/rubydb/sql/planner/type_checker.rb +427 -431
- data/lib/rubydb/sql/token.rb +210 -210
- data/lib/rubydb/storage/buffer_frame.rb +44 -44
- data/lib/rubydb/storage/buffer_pool.rb +155 -155
- data/lib/rubydb/storage/database_lock.rb +74 -74
- data/lib/rubydb/storage/deserializer.rb +332 -342
- data/lib/rubydb/storage/engine.rb +2409 -2330
- data/lib/rubydb/storage/file_manager.rb +191 -187
- data/lib/rubydb/storage/free_space_map.rb +79 -81
- data/lib/rubydb/storage/page.rb +92 -94
- data/lib/rubydb/storage/page_allocator.rb +852 -855
- data/lib/rubydb/storage/page_header.rb +63 -67
- data/lib/rubydb/storage/page_manager.rb +127 -131
- data/lib/rubydb/storage/record.rb +58 -58
- data/lib/rubydb/storage/row.rb +78 -78
- data/lib/rubydb/storage/serializer.rb +51 -51
- data/lib/rubydb/storage/snapshot_reader.rb +167 -0
- data/lib/rubydb/storage/storage_layout.rb +151 -151
- data/lib/rubydb/storage/storage_manager.rb +114 -114
- data/lib/rubydb/storage/tuple.rb +458 -461
- data/lib/rubydb/storage/visibility_map.rb +964 -973
- data/lib/rubydb/transactions/commit_manager.rb +219 -220
- data/lib/rubydb/transactions/isolation.rb +98 -98
- data/lib/rubydb/transactions/lock.rb +76 -76
- data/lib/rubydb/transactions/lock_manager.rb +359 -362
- data/lib/rubydb/transactions/savepoint.rb +142 -143
- data/lib/rubydb/transactions/transaction.rb +214 -215
- data/lib/rubydb/transactions/transaction_id.rb +84 -84
- data/lib/rubydb/transactions/transaction_log.rb +256 -257
- data/lib/rubydb/transactions/transaction_manager.rb +434 -435
- data/lib/rubydb/types/bigint.rb +36 -36
- data/lib/rubydb/types/blob.rb +37 -37
- data/lib/rubydb/types/boolean.rb +34 -34
- data/lib/rubydb/types/date.rb +39 -39
- data/lib/rubydb/types/decimal.rb +48 -48
- data/lib/rubydb/types/float.rb +34 -34
- data/lib/rubydb/types/integer.rb +36 -36
- data/lib/rubydb/types/json.rb +41 -41
- data/lib/rubydb/types/null.rb +34 -34
- data/lib/rubydb/types/smallint.rb +36 -36
- data/lib/rubydb/types/text.rb +37 -37
- data/lib/rubydb/types/time.rb +46 -46
- data/lib/rubydb/types/timestamp.rb +39 -39
- data/lib/rubydb/types/type.rb +119 -119
- data/lib/rubydb/types/uuid.rb +47 -47
- data/lib/rubydb/types/varchar.rb +37 -37
- data/lib/rubydb/version.rb +32 -32
- data/lib/rubydb/wal/archive.rb +207 -193
- data/lib/rubydb/wal/checkpoint.rb +181 -183
- data/lib/rubydb/wal/lsn.rb +94 -94
- data/lib/rubydb/wal/reader.rb +259 -260
- data/lib/rubydb/wal/record.rb +105 -105
- data/lib/rubydb/wal/segment.rb +193 -193
- data/lib/rubydb/wal/wal.rb +481 -452
- data/lib/rubydb/wal/writer.rb +236 -236
- data/lib/rubydb.rb +7 -7
- data/packaging/docker/docker-compose.failover.yml +43 -43
- data/packaging/homebrew/rubydb.rb +19 -19
- data/rubydb.gemspec +70 -57
- data/scripts/benchmark +7 -7
- data/scripts/build_accelerator +49 -0
- data/scripts/durability_drill +37 -37
- data/scripts/fuzz +63 -63
- data/scripts/release +77 -42
- data/scripts/release_check +43 -43
- data/scripts/replication_failover_drill +268 -250
- data/scripts/replication_network_failover_drill +287 -255
- data/scripts/restore_drill +45 -45
- data/scripts/security +45 -0
- metadata +102 -1
|
@@ -1,16 +1,16 @@
|
|
|
1
|
-
# Development guide
|
|
2
|
-
|
|
3
|
-
Install dependencies with `bundle install`. Run focused specs while developing,
|
|
4
|
-
then run `bundle exec rspec` and `bundle exec rubocop` before opening a pull
|
|
5
|
-
request. Keep test databases in temporary directories and close engines in
|
|
6
|
-
`ensure` blocks.
|
|
7
|
-
|
|
8
|
-
Never use production data or secrets in local tests. Changes that affect a
|
|
9
|
-
stored format, SQL behavior, protocol, migration, or release process must update
|
|
10
|
-
the corresponding documentation and changelog.
|
|
11
|
-
|
|
12
|
-
For the complete change workflow, read the [developer guide](../developer-guide.md),
|
|
13
|
-
[debugging playbook](../debugging.md), and [testing guide](testing.md). Every
|
|
14
|
-
bug fix should include a regression test and an explanation of its invariant.
|
|
15
|
-
Run fault, concurrency, or recovery tests when the change crosses a durable
|
|
16
|
-
boundary; a unit test alone is not sufficient evidence.
|
|
1
|
+
# Development guide
|
|
2
|
+
|
|
3
|
+
Install dependencies with `bundle install`. Run focused specs while developing,
|
|
4
|
+
then run `bundle exec rspec` and `bundle exec rubocop` before opening a pull
|
|
5
|
+
request. Keep test databases in temporary directories and close engines in
|
|
6
|
+
`ensure` blocks.
|
|
7
|
+
|
|
8
|
+
Never use production data or secrets in local tests. Changes that affect a
|
|
9
|
+
stored format, SQL behavior, protocol, migration, or release process must update
|
|
10
|
+
the corresponding documentation and changelog.
|
|
11
|
+
|
|
12
|
+
For the complete change workflow, read the [developer guide](../developer-guide.md),
|
|
13
|
+
[debugging playbook](../debugging.md), and [testing guide](testing.md). Every
|
|
14
|
+
bug fix should include a regression test and an explanation of its invariant.
|
|
15
|
+
Run fault, concurrency, or recovery tests when the change crosses a durable
|
|
16
|
+
boundary; a unit test alone is not sufficient evidence.
|
|
@@ -1,49 +1,49 @@
|
|
|
1
|
-
# Releasing RubyDB to RubyGems
|
|
2
|
-
|
|
3
|
-
1. Update `CHANGELOG.md`, bump the semantic version, and confirm the Ruby/Rails support matrix for the release. The version must have a top-level `## <version>` changelog entry; the release preflight fails closed when it is missing.
|
|
4
|
-
2. Run the complete verification suite and workload test. The release workflow independently builds and validates the gem on a version tag.
|
|
5
|
-
3. Create a RubyGems API key with the minimum scope needed to push this gem. Store it as the `RUBYGEMS_API_KEY` GitHub Actions secret or in RubyGems' protected credentials file; never commit it.
|
|
6
|
-
4. Create and push an annotated `v<version>` tag. The compatibility workflow
|
|
7
|
-
must pass its SimpleCov line-coverage gate, and the release workflow
|
|
8
|
-
publishes only when that tag's version matches `RubyDB::VERSION` and the
|
|
9
|
-
secret is available.
|
|
10
|
-
5. To build, verify, and publish manually:
|
|
11
|
-
|
|
12
|
-
```sh
|
|
13
|
-
RUBYDB_PUBLISH=1 GEM_HOST_API_KEY=<RubyGems API key> ruby scripts/release
|
|
14
|
-
```
|
|
15
|
-
|
|
16
|
-
Without `RUBYDB_PUBLISH=1`, `ruby scripts/release` only rebuilds and verifies the gem/checksum. Manual publication must also set `RUBYDB_RELEASE_VERSION=<version>`; tagged CI obtains that identity from `GITHUB_REF_NAME`.
|
|
17
|
-
|
|
18
|
-
The tag workflow also creates a signed GitHub build-provenance attestation for
|
|
19
|
-
the exact gem artifact. Verify that attestation in the repository's Actions or
|
|
20
|
-
Releases UI before distributing the package; the SHA-512 file remains available
|
|
21
|
-
for an independent byte-for-byte check.
|
|
22
|
-
|
|
23
|
-
For the tag release workflow, provision base64-encoded
|
|
24
|
-
`RUBYDB_GEM_SIGNING_KEY_B64` and `RUBYDB_GEM_CERT_B64` repository secrets. The
|
|
25
|
-
release job materializes them only on the ephemeral runner and passes the
|
|
26
|
-
protected paths to `gem build`; it never stores them in the repository. GitHub
|
|
27
|
-
release notes are generated from the tag history after publication, so review
|
|
28
|
-
the generated release before announcing it.
|
|
29
|
-
|
|
30
|
-
The tag workflow fails closed when the signing secrets or RubyGems publication
|
|
31
|
-
key are missing. Local unsigned builds remain available through `rake build`
|
|
32
|
-
and are not publication artifacts.
|
|
33
|
-
|
|
34
|
-
Pull requests also run the supported Ruby 3.3/3.4 matrix on Linux, macOS, and
|
|
35
|
-
Windows, plus the ActiveRecord adapter suite on Rails 7.1, 7.2, and 8.0. A scheduled bounded fuzz job runs
|
|
36
|
-
the SQL parser, WAL, storage, transaction, and query-engine fuzzers. Increase
|
|
37
|
-
`RUBYDB_FUZZ_ITERATIONS` locally when investigating a failure, retaining the
|
|
38
|
-
reported `RUBYDB_FUZZ_SEED` for reproduction.
|
|
39
|
-
|
|
40
|
-
The scheduled operations workflow runs `ruby scripts/restore_drill`, which
|
|
41
|
-
creates a live backup, verifies its manifest/checksums, restores it into a
|
|
42
|
-
separate directory, and reopens the restored database before succeeding.
|
|
43
|
-
|
|
44
|
-
After publication, install the exact released version in a clean environment and run a smoke test:
|
|
45
|
-
|
|
46
|
-
```sh
|
|
47
|
-
gem install rubydb --version 0.1.0
|
|
48
|
-
ruby -e "require 'rubydb'; puts RubyDB::VERSION"
|
|
49
|
-
```
|
|
1
|
+
# Releasing RubyDB to RubyGems
|
|
2
|
+
|
|
3
|
+
1. Update `CHANGELOG.md`, bump the semantic version, and confirm the Ruby/Rails support matrix for the release. The version must have a top-level `## <version>` changelog entry; the release preflight fails closed when it is missing.
|
|
4
|
+
2. Run the complete verification suite and workload test. The release workflow independently builds and validates the gem on a version tag.
|
|
5
|
+
3. Create a RubyGems API key with the minimum scope needed to push this gem. Store it as the `RUBYGEMS_API_KEY` GitHub Actions secret or in RubyGems' protected credentials file; never commit it.
|
|
6
|
+
4. Create and push an annotated `v<version>` tag. The compatibility workflow
|
|
7
|
+
must pass its SimpleCov line-coverage gate, and the release workflow
|
|
8
|
+
publishes only when that tag's version matches `RubyDB::VERSION` and the
|
|
9
|
+
secret is available.
|
|
10
|
+
5. To build, verify, and publish manually:
|
|
11
|
+
|
|
12
|
+
```sh
|
|
13
|
+
RUBYDB_PUBLISH=1 GEM_HOST_API_KEY=<RubyGems API key> ruby scripts/release
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
Without `RUBYDB_PUBLISH=1`, `ruby scripts/release` only rebuilds and verifies the gem/checksum. Manual publication must also set `RUBYDB_RELEASE_VERSION=<version>`; tagged CI obtains that identity from `GITHUB_REF_NAME`.
|
|
17
|
+
|
|
18
|
+
The tag workflow also creates a signed GitHub build-provenance attestation for
|
|
19
|
+
the exact gem artifact. Verify that attestation in the repository's Actions or
|
|
20
|
+
Releases UI before distributing the package; the SHA-512 file remains available
|
|
21
|
+
for an independent byte-for-byte check.
|
|
22
|
+
|
|
23
|
+
For the tag release workflow, provision base64-encoded
|
|
24
|
+
`RUBYDB_GEM_SIGNING_KEY_B64` and `RUBYDB_GEM_CERT_B64` repository secrets. The
|
|
25
|
+
release job materializes them only on the ephemeral runner and passes the
|
|
26
|
+
protected paths to `gem build`; it never stores them in the repository. GitHub
|
|
27
|
+
release notes are generated from the tag history after publication, so review
|
|
28
|
+
the generated release before announcing it.
|
|
29
|
+
|
|
30
|
+
The tag workflow fails closed when the signing secrets or RubyGems publication
|
|
31
|
+
key are missing. Local unsigned builds remain available through `rake build`
|
|
32
|
+
and are not publication artifacts.
|
|
33
|
+
|
|
34
|
+
Pull requests also run the supported Ruby 3.3/3.4 matrix on Linux, macOS, and
|
|
35
|
+
Windows, plus the ActiveRecord adapter suite on Rails 7.1, 7.2, and 8.0. A scheduled bounded fuzz job runs
|
|
36
|
+
the SQL parser, WAL, storage, transaction, and query-engine fuzzers. Increase
|
|
37
|
+
`RUBYDB_FUZZ_ITERATIONS` locally when investigating a failure, retaining the
|
|
38
|
+
reported `RUBYDB_FUZZ_SEED` for reproduction.
|
|
39
|
+
|
|
40
|
+
The scheduled operations workflow runs `ruby scripts/restore_drill`, which
|
|
41
|
+
creates a live backup, verifies its manifest/checksums, restores it into a
|
|
42
|
+
separate directory, and reopens the restored database before succeeding.
|
|
43
|
+
|
|
44
|
+
After publication, install the exact released version in a clean environment and run a smoke test:
|
|
45
|
+
|
|
46
|
+
```sh
|
|
47
|
+
gem install rubydb --version 0.1.0
|
|
48
|
+
ruby -e "require 'rubydb'; puts RubyDB::VERSION"
|
|
49
|
+
```
|
|
@@ -1,16 +1,16 @@
|
|
|
1
|
-
# Testing guide
|
|
2
|
-
|
|
3
|
-
The test layers are complementary:
|
|
4
|
-
|
|
5
|
-
- unit specs validate isolated algorithms and invariants;
|
|
6
|
-
- integration specs run the real parser, planner, executor, storage, server,
|
|
7
|
-
client, adapter, or replication path;
|
|
8
|
-
- chaos/fault specs inject write, crash, corruption, and network failures;
|
|
9
|
-
- workload scripts measure concurrency, latency, cancellation, capacity, and
|
|
10
|
-
durable reopen behavior;
|
|
11
|
-
- CI covers supported Ruby/Rails/OS combinations, security scans, fuzzing, and
|
|
12
|
-
release preflight.
|
|
13
|
-
|
|
14
|
-
Run `bundle exec rspec` for the complete local gate. Preserve the random seed
|
|
15
|
-
when reproducing failures. A passing local suite does not replace hosted
|
|
16
|
-
multi-host, physical-filesystem, or independent security validation.
|
|
1
|
+
# Testing guide
|
|
2
|
+
|
|
3
|
+
The test layers are complementary:
|
|
4
|
+
|
|
5
|
+
- unit specs validate isolated algorithms and invariants;
|
|
6
|
+
- integration specs run the real parser, planner, executor, storage, server,
|
|
7
|
+
client, adapter, or replication path;
|
|
8
|
+
- chaos/fault specs inject write, crash, corruption, and network failures;
|
|
9
|
+
- workload scripts measure concurrency, latency, cancellation, capacity, and
|
|
10
|
+
durable reopen behavior;
|
|
11
|
+
- CI covers supported Ruby/Rails/OS combinations, security scans, fuzzing, and
|
|
12
|
+
release preflight.
|
|
13
|
+
|
|
14
|
+
Run `bundle exec rspec` for the complete local gate. Preserve the random seed
|
|
15
|
+
when reproducing failures. A passing local suite does not replace hosted
|
|
16
|
+
multi-host, physical-filesystem, or independent security validation.
|
data/docs/debugging.md
CHANGED
|
@@ -1,229 +1,229 @@
|
|
|
1
|
-
# Debugging RubyDB
|
|
2
|
-
|
|
3
|
-
This playbook is for finding defects without destroying the evidence needed
|
|
4
|
-
to recover data. Use it in development and staging first. Production
|
|
5
|
-
diagnostics must follow the application’s privacy, access, and change-control
|
|
6
|
-
policies.
|
|
7
|
-
|
|
8
|
-
## Debugging principles
|
|
9
|
-
|
|
10
|
-
1. Reproduce on a copy or a new temporary directory.
|
|
11
|
-
2. Identify the layer: API, session, protocol, parser, planner, executor,
|
|
12
|
-
transaction, storage, WAL, filesystem, or deployment.
|
|
13
|
-
3. Reduce the input while preserving the failure.
|
|
14
|
-
4. Record versions, configuration names, seed, timing, and request IDs.
|
|
15
|
-
5. Add a regression test before changing behavior.
|
|
16
|
-
|
|
17
|
-
A database that returns an error is often safer than one that silently repairs,
|
|
18
|
-
retries, or acknowledges unknown state. Keep fail-closed behavior intact while
|
|
19
|
-
diagnosing.
|
|
20
|
-
|
|
21
|
-
## Reproducible diagnostic baseline
|
|
22
|
-
|
|
23
|
-
```sh
|
|
24
|
-
git rev-parse HEAD
|
|
25
|
-
ruby -v
|
|
26
|
-
bundle exec ruby -Ilib exe/rubydb --version
|
|
27
|
-
bundle exec rspec --format progress
|
|
28
|
-
git diff --check
|
|
29
|
-
```
|
|
30
|
-
|
|
31
|
-
Use a fresh directory for each reproduction. Save the command and, for random
|
|
32
|
-
or concurrent tests, the seed:
|
|
33
|
-
|
|
34
|
-
```sh
|
|
35
|
-
RUBYDB_FUZZ_SEED=12345 RUBYDB_FUZZ_ITERATIONS=1000 ruby scripts/fuzz
|
|
36
|
-
bundle exec rspec spec/path/to/failing_spec.rb:42 --format doc
|
|
37
|
-
```
|
|
38
|
-
|
|
39
|
-
## CLI inspection
|
|
40
|
-
|
|
41
|
-
The CLI is the first diagnostic interface:
|
|
42
|
-
|
|
43
|
-
```sh
|
|
44
|
-
bundle exec ruby -Ilib exe/rubydb status --config config/rubydb.yml
|
|
45
|
-
bundle exec ruby -Ilib exe/rubydb doctor --config config/rubydb.yml
|
|
46
|
-
bundle exec ruby -Ilib exe/rubydb inspect --config config/rubydb.yml
|
|
47
|
-
bundle exec ruby -Ilib exe/rubydb shell --config config/rubydb.yml
|
|
48
|
-
```
|
|
49
|
-
|
|
50
|
-
Use `status` for ownership, lifecycle, and health; `doctor` for non-destructive
|
|
51
|
-
checks and safe repairs; and `inspect` for metadata, schema, WAL, and storage
|
|
52
|
-
facts. Capture JSON output when the command supports it so an incident can be
|
|
53
|
-
compared over time. Read [CLI guide](cli.md) before using maintenance commands.
|
|
54
|
-
|
|
55
|
-
## Logging and safe verbosity
|
|
56
|
-
|
|
57
|
-
Use the normal structured logs first. Increase verbosity only for a bounded
|
|
58
|
-
reproduction. `--verbose` is useful for CLI execution, while `RUBYDB_DEBUG=1`
|
|
59
|
-
enables development backtraces in command wrappers. Debug logs can contain SQL
|
|
60
|
-
shapes, paths, identifiers, and timing; they must be access-controlled and
|
|
61
|
-
sanitized before sharing.
|
|
62
|
-
|
|
63
|
-
When adding logs, include stable fields such as request ID, transaction ID,
|
|
64
|
-
connection ID, LSN, operation, duration, and outcome. Never log passwords,
|
|
65
|
-
SCRAM secrets, bearer tokens, private keys, or unredacted customer values.
|
|
66
|
-
|
|
67
|
-
## Layer isolation
|
|
68
|
-
|
|
69
|
-
### API versus engine
|
|
70
|
-
|
|
71
|
-
Run the same operation through the direct Ruby API and the server/client path.
|
|
72
|
-
If only server mode fails, inspect protocol framing, authentication, session
|
|
73
|
-
state, serialization, and timeout handling. If both fail, reduce to engine SQL
|
|
74
|
-
and storage.
|
|
75
|
-
|
|
76
|
-
### Parser versus executor
|
|
77
|
-
|
|
78
|
-
First parse/bind a statement without committing data. Then execute it against a
|
|
79
|
-
minimal schema. A parser failure should identify token position and expected
|
|
80
|
-
construct. A binder failure should identify parameter count/type context. An
|
|
81
|
-
executor failure should preserve transaction rollback and identify the table,
|
|
82
|
-
index, or constraint involved.
|
|
83
|
-
|
|
84
|
-
### Planner versus semantics
|
|
85
|
-
|
|
86
|
-
Compare an optimized plan with a simple scan where possible. Verify row IDs,
|
|
87
|
-
duplicates, `NULL`, ordering, grouping, and snapshot visibility. Optimizer
|
|
88
|
-
changes require equivalence tests, not only performance numbers.
|
|
89
|
-
|
|
90
|
-
### Storage versus filesystem
|
|
91
|
-
|
|
92
|
-
Test the storage operation with a real temporary filesystem first. Then inject
|
|
93
|
-
failures at write, flush, rename, allocation, and close boundaries. A Ruby
|
|
94
|
-
exception around a filesystem call is not equivalent to a process termination
|
|
95
|
-
after the filesystem accepted the write.
|
|
96
|
-
|
|
97
|
-
## Transaction and deadlock diagnosis
|
|
98
|
-
|
|
99
|
-
Capture transaction lifecycle events: begin, snapshot, lock wait, lock grant,
|
|
100
|
-
savepoint, statement error, rollback, commit request, durable commit, and
|
|
101
|
-
connection close. For a deadlock, draw a wait-for graph from transaction IDs
|
|
102
|
-
and locks. Verify the selected victim’s before-images were applied and that
|
|
103
|
-
all locks and snapshots were released.
|
|
104
|
-
|
|
105
|
-
For a timeout or cancellation, answer three questions:
|
|
106
|
-
|
|
107
|
-
1. Did the server stop executing the request?
|
|
108
|
-
2. Did the transaction commit, roll back, or remain unknown?
|
|
109
|
-
3. Were connection, lock, snapshot, and temporary resources released?
|
|
110
|
-
|
|
111
|
-
Do not retry a write with an unknown outcome unless it is idempotent or the
|
|
112
|
-
application can query the request/transaction outcome.
|
|
113
|
-
|
|
114
|
-
## WAL and recovery diagnosis
|
|
115
|
-
|
|
116
|
-
Preserve the complete database directory, WAL, metadata, and service logs.
|
|
117
|
-
Record file sizes, modification times, checksums, checkpoint LSN, last
|
|
118
|
-
acknowledged LSN, and the process termination reason. Run inspection and
|
|
119
|
-
restore against a copy. Compare:
|
|
120
|
-
|
|
121
|
-
```text
|
|
122
|
-
checkpoint LSN <= durable WAL end LSN
|
|
123
|
-
last acknowledged commit <= durable WAL end LSN
|
|
124
|
-
restored checksum == manifest checksum
|
|
125
|
-
replayed transaction set == committed transaction set
|
|
126
|
-
```
|
|
127
|
-
|
|
128
|
-
For corruption, stop writes and escalate rather than repeatedly reopening the
|
|
129
|
-
original. For a compaction issue, compare row counts, indexes, checksums, and
|
|
130
|
-
reopen behavior before and after compaction on a copy.
|
|
131
|
-
|
|
132
|
-
## Wire and protocol debugging
|
|
133
|
-
|
|
134
|
-
Use a local test endpoint and sanitized packet/frame logging. Validate one
|
|
135
|
-
request at a time: handshake, authentication, capability negotiation, query,
|
|
136
|
-
result frames, cancellation, and close. Check frame length, request ID,
|
|
137
|
-
sequence, status, and error payload. Test partial reads and writes because a
|
|
138
|
-
single `read` or `write` is not guaranteed to transfer a complete frame.
|
|
139
|
-
|
|
140
|
-
For an in-flight cancellation race, log the request ID and server state at
|
|
141
|
-
cancel receipt, executor stop, transaction decision, and response emission.
|
|
142
|
-
The client must distinguish cancellation accepted, cancellation too late, and
|
|
143
|
-
unknown connection loss.
|
|
144
|
-
|
|
145
|
-
## Rails debugging
|
|
146
|
-
|
|
147
|
-
Enable Rails SQL logging in a non-production reproduction and redact bind
|
|
148
|
-
values before sharing. Compare the generated SQL with a direct RubyDB query.
|
|
149
|
-
For adapter bugs, create the smallest model and migration that shows the
|
|
150
|
-
problem, then test both a fresh schema and a populated table. Inspect:
|
|
151
|
-
|
|
152
|
-
* quoting and bind parameter order;
|
|
153
|
-
* transaction/savepoint boundaries;
|
|
154
|
-
* affected rows and last-insert ID;
|
|
155
|
-
* schema introspection and default values;
|
|
156
|
-
* pool checkout/checkin and leaked transactions; and
|
|
157
|
-
* exception class and retry behavior.
|
|
158
|
-
|
|
159
|
-
Use the Rails example under `examples/rails_app` as a smoke harness before
|
|
160
|
-
reproducing inside a large application.
|
|
161
|
-
|
|
162
|
-
## Ruby-level tools
|
|
163
|
-
|
|
164
|
-
For a focused local reproduction, Ruby’s standard tools are usually enough:
|
|
165
|
-
|
|
166
|
-
```sh
|
|
167
|
-
RUBYOPT="-d" bundle exec rspec spec/path/to/failing_spec.rb
|
|
168
|
-
bundle exec ruby -w -Ilib path/to/reproduction.rb
|
|
169
|
-
```
|
|
170
|
-
|
|
171
|
-
`TracePoint` can observe method calls and exceptions without modifying the
|
|
172
|
-
engine. Use it only in a short-lived reproduction because tracing changes
|
|
173
|
-
timing and can invalidate concurrency conclusions. Capture thread backtraces
|
|
174
|
-
when a process appears hung, and include the thread roles (acceptor, worker,
|
|
175
|
-
checkpoint, replication, application).
|
|
176
|
-
|
|
177
|
-
## Performance debugging
|
|
178
|
-
|
|
179
|
-
Separate CPU, lock, I/O, and queue time. Record p50/p95/p99 latency, throughput,
|
|
180
|
-
errors, WAL growth, checkpoint duration, memory, file descriptors, and active
|
|
181
|
-
transactions. Change one variable per run and repeat enough times to expose
|
|
182
|
-
variance. A faster benchmark with weaker durability is not an equivalent
|
|
183
|
-
optimization.
|
|
184
|
-
|
|
185
|
-
Use:
|
|
186
|
-
|
|
187
|
-
```sh
|
|
188
|
-
RUBYDB_BENCHMARK_ITERATIONS=100 ruby -Ilib benchmarks/basic_workload.rb
|
|
189
|
-
ruby benchmarks/concurrent_workload.rb
|
|
190
|
-
ruby scripts/production_soak
|
|
191
|
-
```
|
|
192
|
-
|
|
193
|
-
Keep benchmark artifacts out of commits unless they are intentional fixtures.
|
|
194
|
-
|
|
195
|
-
## Fuzzing and property failures
|
|
196
|
-
|
|
197
|
-
Save the seed, generated SQL/input, Ruby version, commit, and failing database
|
|
198
|
-
directory. Reduce the number of operations while preserving the seed. Check
|
|
199
|
-
the invariant: no crash, no invalid state, rollback equivalence, parser
|
|
200
|
-
round-trip, or index/table agreement. Add the minimized case as a deterministic
|
|
201
|
-
spec, then keep the fuzz run as a secondary guard.
|
|
202
|
-
|
|
203
|
-
## What not to do
|
|
204
|
-
|
|
205
|
-
Do not delete WAL or lock files, edit database bytes manually, disable checksum
|
|
206
|
-
validation, run repair on the only copy, force a replica promotion without
|
|
207
|
-
fencing, or publish sanitized logs that still contain secrets. These actions
|
|
208
|
-
can turn a diagnosable incident into irreversible data loss or a security
|
|
209
|
-
incident.
|
|
210
|
-
|
|
211
|
-
## Diagnostic report template
|
|
212
|
-
|
|
213
|
-
```text
|
|
214
|
-
Summary:
|
|
215
|
-
First observed (UTC):
|
|
216
|
-
RubyDB version/commit:
|
|
217
|
-
Ruby/Rails/OS/filesystem:
|
|
218
|
-
Topology and ownership mode:
|
|
219
|
-
Configuration names/checksum:
|
|
220
|
-
Command or request shape:
|
|
221
|
-
Request/transaction/LSN IDs:
|
|
222
|
-
Expected result:
|
|
223
|
-
Actual result:
|
|
224
|
-
Reproduction and seed:
|
|
225
|
-
Logs/metrics/checksums:
|
|
226
|
-
Actions already taken:
|
|
227
|
-
Data impact and current containment:
|
|
228
|
-
```
|
|
229
|
-
|
|
1
|
+
# Debugging RubyDB
|
|
2
|
+
|
|
3
|
+
This playbook is for finding defects without destroying the evidence needed
|
|
4
|
+
to recover data. Use it in development and staging first. Production
|
|
5
|
+
diagnostics must follow the application’s privacy, access, and change-control
|
|
6
|
+
policies.
|
|
7
|
+
|
|
8
|
+
## Debugging principles
|
|
9
|
+
|
|
10
|
+
1. Reproduce on a copy or a new temporary directory.
|
|
11
|
+
2. Identify the layer: API, session, protocol, parser, planner, executor,
|
|
12
|
+
transaction, storage, WAL, filesystem, or deployment.
|
|
13
|
+
3. Reduce the input while preserving the failure.
|
|
14
|
+
4. Record versions, configuration names, seed, timing, and request IDs.
|
|
15
|
+
5. Add a regression test before changing behavior.
|
|
16
|
+
|
|
17
|
+
A database that returns an error is often safer than one that silently repairs,
|
|
18
|
+
retries, or acknowledges unknown state. Keep fail-closed behavior intact while
|
|
19
|
+
diagnosing.
|
|
20
|
+
|
|
21
|
+
## Reproducible diagnostic baseline
|
|
22
|
+
|
|
23
|
+
```sh
|
|
24
|
+
git rev-parse HEAD
|
|
25
|
+
ruby -v
|
|
26
|
+
bundle exec ruby -Ilib exe/rubydb --version
|
|
27
|
+
bundle exec rspec --format progress
|
|
28
|
+
git diff --check
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
Use a fresh directory for each reproduction. Save the command and, for random
|
|
32
|
+
or concurrent tests, the seed:
|
|
33
|
+
|
|
34
|
+
```sh
|
|
35
|
+
RUBYDB_FUZZ_SEED=12345 RUBYDB_FUZZ_ITERATIONS=1000 ruby scripts/fuzz
|
|
36
|
+
bundle exec rspec spec/path/to/failing_spec.rb:42 --format doc
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
## CLI inspection
|
|
40
|
+
|
|
41
|
+
The CLI is the first diagnostic interface:
|
|
42
|
+
|
|
43
|
+
```sh
|
|
44
|
+
bundle exec ruby -Ilib exe/rubydb status --config config/rubydb.yml
|
|
45
|
+
bundle exec ruby -Ilib exe/rubydb doctor --config config/rubydb.yml
|
|
46
|
+
bundle exec ruby -Ilib exe/rubydb inspect --config config/rubydb.yml
|
|
47
|
+
bundle exec ruby -Ilib exe/rubydb shell --config config/rubydb.yml
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Use `status` for ownership, lifecycle, and health; `doctor` for non-destructive
|
|
51
|
+
checks and safe repairs; and `inspect` for metadata, schema, WAL, and storage
|
|
52
|
+
facts. Capture JSON output when the command supports it so an incident can be
|
|
53
|
+
compared over time. Read [CLI guide](cli.md) before using maintenance commands.
|
|
54
|
+
|
|
55
|
+
## Logging and safe verbosity
|
|
56
|
+
|
|
57
|
+
Use the normal structured logs first. Increase verbosity only for a bounded
|
|
58
|
+
reproduction. `--verbose` is useful for CLI execution, while `RUBYDB_DEBUG=1`
|
|
59
|
+
enables development backtraces in command wrappers. Debug logs can contain SQL
|
|
60
|
+
shapes, paths, identifiers, and timing; they must be access-controlled and
|
|
61
|
+
sanitized before sharing.
|
|
62
|
+
|
|
63
|
+
When adding logs, include stable fields such as request ID, transaction ID,
|
|
64
|
+
connection ID, LSN, operation, duration, and outcome. Never log passwords,
|
|
65
|
+
SCRAM secrets, bearer tokens, private keys, or unredacted customer values.
|
|
66
|
+
|
|
67
|
+
## Layer isolation
|
|
68
|
+
|
|
69
|
+
### API versus engine
|
|
70
|
+
|
|
71
|
+
Run the same operation through the direct Ruby API and the server/client path.
|
|
72
|
+
If only server mode fails, inspect protocol framing, authentication, session
|
|
73
|
+
state, serialization, and timeout handling. If both fail, reduce to engine SQL
|
|
74
|
+
and storage.
|
|
75
|
+
|
|
76
|
+
### Parser versus executor
|
|
77
|
+
|
|
78
|
+
First parse/bind a statement without committing data. Then execute it against a
|
|
79
|
+
minimal schema. A parser failure should identify token position and expected
|
|
80
|
+
construct. A binder failure should identify parameter count/type context. An
|
|
81
|
+
executor failure should preserve transaction rollback and identify the table,
|
|
82
|
+
index, or constraint involved.
|
|
83
|
+
|
|
84
|
+
### Planner versus semantics
|
|
85
|
+
|
|
86
|
+
Compare an optimized plan with a simple scan where possible. Verify row IDs,
|
|
87
|
+
duplicates, `NULL`, ordering, grouping, and snapshot visibility. Optimizer
|
|
88
|
+
changes require equivalence tests, not only performance numbers.
|
|
89
|
+
|
|
90
|
+
### Storage versus filesystem
|
|
91
|
+
|
|
92
|
+
Test the storage operation with a real temporary filesystem first. Then inject
|
|
93
|
+
failures at write, flush, rename, allocation, and close boundaries. A Ruby
|
|
94
|
+
exception around a filesystem call is not equivalent to a process termination
|
|
95
|
+
after the filesystem accepted the write.
|
|
96
|
+
|
|
97
|
+
## Transaction and deadlock diagnosis
|
|
98
|
+
|
|
99
|
+
Capture transaction lifecycle events: begin, snapshot, lock wait, lock grant,
|
|
100
|
+
savepoint, statement error, rollback, commit request, durable commit, and
|
|
101
|
+
connection close. For a deadlock, draw a wait-for graph from transaction IDs
|
|
102
|
+
and locks. Verify the selected victim’s before-images were applied and that
|
|
103
|
+
all locks and snapshots were released.
|
|
104
|
+
|
|
105
|
+
For a timeout or cancellation, answer three questions:
|
|
106
|
+
|
|
107
|
+
1. Did the server stop executing the request?
|
|
108
|
+
2. Did the transaction commit, roll back, or remain unknown?
|
|
109
|
+
3. Were connection, lock, snapshot, and temporary resources released?
|
|
110
|
+
|
|
111
|
+
Do not retry a write with an unknown outcome unless it is idempotent or the
|
|
112
|
+
application can query the request/transaction outcome.
|
|
113
|
+
|
|
114
|
+
## WAL and recovery diagnosis
|
|
115
|
+
|
|
116
|
+
Preserve the complete database directory, WAL, metadata, and service logs.
|
|
117
|
+
Record file sizes, modification times, checksums, checkpoint LSN, last
|
|
118
|
+
acknowledged LSN, and the process termination reason. Run inspection and
|
|
119
|
+
restore against a copy. Compare:
|
|
120
|
+
|
|
121
|
+
```text
|
|
122
|
+
checkpoint LSN <= durable WAL end LSN
|
|
123
|
+
last acknowledged commit <= durable WAL end LSN
|
|
124
|
+
restored checksum == manifest checksum
|
|
125
|
+
replayed transaction set == committed transaction set
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
For corruption, stop writes and escalate rather than repeatedly reopening the
|
|
129
|
+
original. For a compaction issue, compare row counts, indexes, checksums, and
|
|
130
|
+
reopen behavior before and after compaction on a copy.
|
|
131
|
+
|
|
132
|
+
## Wire and protocol debugging
|
|
133
|
+
|
|
134
|
+
Use a local test endpoint and sanitized packet/frame logging. Validate one
|
|
135
|
+
request at a time: handshake, authentication, capability negotiation, query,
|
|
136
|
+
result frames, cancellation, and close. Check frame length, request ID,
|
|
137
|
+
sequence, status, and error payload. Test partial reads and writes because a
|
|
138
|
+
single `read` or `write` is not guaranteed to transfer a complete frame.
|
|
139
|
+
|
|
140
|
+
For an in-flight cancellation race, log the request ID and server state at
|
|
141
|
+
cancel receipt, executor stop, transaction decision, and response emission.
|
|
142
|
+
The client must distinguish cancellation accepted, cancellation too late, and
|
|
143
|
+
unknown connection loss.
|
|
144
|
+
|
|
145
|
+
## Rails debugging
|
|
146
|
+
|
|
147
|
+
Enable Rails SQL logging in a non-production reproduction and redact bind
|
|
148
|
+
values before sharing. Compare the generated SQL with a direct RubyDB query.
|
|
149
|
+
For adapter bugs, create the smallest model and migration that shows the
|
|
150
|
+
problem, then test both a fresh schema and a populated table. Inspect:
|
|
151
|
+
|
|
152
|
+
* quoting and bind parameter order;
|
|
153
|
+
* transaction/savepoint boundaries;
|
|
154
|
+
* affected rows and last-insert ID;
|
|
155
|
+
* schema introspection and default values;
|
|
156
|
+
* pool checkout/checkin and leaked transactions; and
|
|
157
|
+
* exception class and retry behavior.
|
|
158
|
+
|
|
159
|
+
Use the Rails example under `examples/rails_app` as a smoke harness before
|
|
160
|
+
reproducing inside a large application.
|
|
161
|
+
|
|
162
|
+
## Ruby-level tools
|
|
163
|
+
|
|
164
|
+
For a focused local reproduction, Ruby’s standard tools are usually enough:
|
|
165
|
+
|
|
166
|
+
```sh
|
|
167
|
+
RUBYOPT="-d" bundle exec rspec spec/path/to/failing_spec.rb
|
|
168
|
+
bundle exec ruby -w -Ilib path/to/reproduction.rb
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
`TracePoint` can observe method calls and exceptions without modifying the
|
|
172
|
+
engine. Use it only in a short-lived reproduction because tracing changes
|
|
173
|
+
timing and can invalidate concurrency conclusions. Capture thread backtraces
|
|
174
|
+
when a process appears hung, and include the thread roles (acceptor, worker,
|
|
175
|
+
checkpoint, replication, application).
|
|
176
|
+
|
|
177
|
+
## Performance debugging
|
|
178
|
+
|
|
179
|
+
Separate CPU, lock, I/O, and queue time. Record p50/p95/p99 latency, throughput,
|
|
180
|
+
errors, WAL growth, checkpoint duration, memory, file descriptors, and active
|
|
181
|
+
transactions. Change one variable per run and repeat enough times to expose
|
|
182
|
+
variance. A faster benchmark with weaker durability is not an equivalent
|
|
183
|
+
optimization.
|
|
184
|
+
|
|
185
|
+
Use:
|
|
186
|
+
|
|
187
|
+
```sh
|
|
188
|
+
RUBYDB_BENCHMARK_ITERATIONS=100 ruby -Ilib benchmarks/basic_workload.rb
|
|
189
|
+
ruby benchmarks/concurrent_workload.rb
|
|
190
|
+
ruby scripts/production_soak
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
Keep benchmark artifacts out of commits unless they are intentional fixtures.
|
|
194
|
+
|
|
195
|
+
## Fuzzing and property failures
|
|
196
|
+
|
|
197
|
+
Save the seed, generated SQL/input, Ruby version, commit, and failing database
|
|
198
|
+
directory. Reduce the number of operations while preserving the seed. Check
|
|
199
|
+
the invariant: no crash, no invalid state, rollback equivalence, parser
|
|
200
|
+
round-trip, or index/table agreement. Add the minimized case as a deterministic
|
|
201
|
+
spec, then keep the fuzz run as a secondary guard.
|
|
202
|
+
|
|
203
|
+
## What not to do
|
|
204
|
+
|
|
205
|
+
Do not delete WAL or lock files, edit database bytes manually, disable checksum
|
|
206
|
+
validation, run repair on the only copy, force a replica promotion without
|
|
207
|
+
fencing, or publish sanitized logs that still contain secrets. These actions
|
|
208
|
+
can turn a diagnosable incident into irreversible data loss or a security
|
|
209
|
+
incident.
|
|
210
|
+
|
|
211
|
+
## Diagnostic report template
|
|
212
|
+
|
|
213
|
+
```text
|
|
214
|
+
Summary:
|
|
215
|
+
First observed (UTC):
|
|
216
|
+
RubyDB version/commit:
|
|
217
|
+
Ruby/Rails/OS/filesystem:
|
|
218
|
+
Topology and ownership mode:
|
|
219
|
+
Configuration names/checksum:
|
|
220
|
+
Command or request shape:
|
|
221
|
+
Request/transaction/LSN IDs:
|
|
222
|
+
Expected result:
|
|
223
|
+
Actual result:
|
|
224
|
+
Reproduction and seed:
|
|
225
|
+
Logs/metrics/checksums:
|
|
226
|
+
Actions already taken:
|
|
227
|
+
Data impact and current containment:
|
|
228
|
+
```
|
|
229
|
+
|
data/docs/developer/branching.md
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
|
-
# Branching
|
|
2
|
-
|
|
3
|
-
RubyDB branches represent database snapshots and development lines in the
|
|
4
|
-
engine. Create and inspect branches with the CLI, verify the target state, and
|
|
5
|
-
retain a backup before merging or checking out a branch containing important
|
|
6
|
-
data.
|
|
7
|
-
|
|
8
|
-
Branch operations are not a substitute for backups or replication. Test branch
|
|
9
|
-
diff, merge, checkout, conflict handling, and reopen behavior before using them
|
|
10
|
-
in an operational workflow.
|
|
1
|
+
# Branching
|
|
2
|
+
|
|
3
|
+
RubyDB branches represent database snapshots and development lines in the
|
|
4
|
+
engine. Create and inspect branches with the CLI, verify the target state, and
|
|
5
|
+
retain a backup before merging or checking out a branch containing important
|
|
6
|
+
data.
|
|
7
|
+
|
|
8
|
+
Branch operations are not a substitute for backups or replication. Test branch
|
|
9
|
+
diff, merge, checkout, conflict handling, and reopen behavior before using them
|
|
10
|
+
in an operational workflow.
|