rubydb 0.1.4 → 0.1.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (469) hide show
  1. checksums.yaml +4 -4
  2. data/.github/PULL_REQUEST_TEMPLATE.md +15 -15
  3. data/.github/workflows/benchmark.yml +26 -26
  4. data/.github/workflows/compatibility.yml +63 -63
  5. data/.github/workflows/fuzz.yml +33 -33
  6. data/.github/workflows/lint.yml +21 -21
  7. data/.github/workflows/operations.yml +24 -24
  8. data/.github/workflows/production-validation.yml +111 -111
  9. data/.github/workflows/release.yml +77 -77
  10. data/.github/workflows/security.yml +39 -37
  11. data/.github/workflows/test.yml +26 -26
  12. data/.github/workflows/workload.yml +58 -58
  13. data/.gitignore +16 -5
  14. data/.rubocop.yml +50 -44
  15. data/.standard.yml +9 -14
  16. data/ARCHITECTURE.md +21 -21
  17. data/CHANGELOG.md +57 -27
  18. data/CODE_OF_CONDUCT.md +13 -13
  19. data/CONTRIBUTING.md +29 -29
  20. data/GOVERNANCE.md +16 -16
  21. data/Gemfile +18 -17
  22. data/Gemfile.lock +125 -71
  23. data/README.md +168 -12
  24. data/ROADMAP.md +27 -27
  25. data/Rakefile +76 -71
  26. data/SECURITY.md +54 -54
  27. data/SUPPORT.md +14 -14
  28. data/accelerator/bin/SHA256SUMS +6 -0
  29. data/accelerator/bin/rubydb-accelerator-darwin-amd64 +0 -0
  30. data/accelerator/bin/rubydb-accelerator-darwin-arm64 +0 -0
  31. data/accelerator/bin/rubydb-accelerator-linux-amd64 +0 -0
  32. data/accelerator/bin/rubydb-accelerator-linux-arm64 +0 -0
  33. data/accelerator/bin/rubydb-accelerator-windows-amd64.exe +0 -0
  34. data/accelerator/bin/rubydb-accelerator-windows-arm64.exe +0 -0
  35. data/accelerator/cmd/rubydb-accelerator/main.go +11 -0
  36. data/accelerator/go.mod +3 -0
  37. data/accelerator/internal/execution/aggregate.go +94 -0
  38. data/accelerator/internal/execution/distinct.go +22 -0
  39. data/accelerator/internal/execution/filter.go +73 -0
  40. data/accelerator/internal/execution/join.go +79 -0
  41. data/accelerator/internal/execution/operators.go +167 -0
  42. data/accelerator/internal/execution/scan.go +20 -0
  43. data/accelerator/internal/execution/sort.go +62 -0
  44. data/accelerator/internal/execution/types.go +136 -0
  45. data/accelerator/internal/execution/value.go +67 -0
  46. data/accelerator/internal/memory/arena.go +47 -0
  47. data/accelerator/internal/memory/reuse.go +22 -0
  48. data/accelerator/internal/metrics/registry.go +67 -0
  49. data/accelerator/internal/parallel/bounded_queue.go +56 -0
  50. data/accelerator/internal/parallel/scheduler.go +47 -0
  51. data/accelerator/internal/parallel/worker_pool.go +53 -0
  52. data/accelerator/internal/protocol/cancellation.go +48 -0
  53. data/accelerator/internal/protocol/columnar.go +263 -0
  54. data/accelerator/internal/protocol/frame.go +187 -0
  55. data/accelerator/internal/runtime/worker.go +521 -0
  56. data/accelerator/internal/storage/page_reader.go +81 -0
  57. data/accelerator/internal/storage/snapshot_scan.go +539 -0
  58. data/accelerator/internal/wal/checksum.go +13 -0
  59. data/accelerator/internal/wal/compression.go +41 -0
  60. data/accelerator/internal/wal/group_commit.go +24 -0
  61. data/accelerator/internal/wal/record_encoder.go +40 -0
  62. data/adapters/activerecord/Gemfile +11 -11
  63. data/adapters/activerecord/README.md +8 -3
  64. data/adapters/activerecord/lib/active_record/connection_adapters/rubydb_adapter.rb +881 -879
  65. data/adapters/activerecord/rubydb-activerecord.gemspec +21 -21
  66. data/adapters/activerecord/spec/rubydb_adapter_integration_spec.rb +143 -143
  67. data/adapters/ruby/README.md +18 -18
  68. data/adapters/sequel/README.md +11 -11
  69. data/config/monitoring/prometheus-alerts.yml +39 -39
  70. data/config/production.yml +36 -36
  71. data/docs/README.md +77 -71
  72. data/docs/architecture/concurrency.md +14 -14
  73. data/docs/architecture/current-state.md +125 -125
  74. data/docs/architecture/execution-engine.md +25 -25
  75. data/docs/architecture/go-accelerator.md +179 -0
  76. data/docs/architecture/indexes.md +19 -19
  77. data/docs/architecture/mvcc.md +19 -19
  78. data/docs/architecture/overview.md +13 -13
  79. data/docs/architecture/pages.md +11 -11
  80. data/docs/architecture/production-roadmap.md +82 -82
  81. data/docs/architecture/query-planner.md +20 -20
  82. data/docs/architecture/recovery.md +18 -18
  83. data/docs/architecture/sql-engine.md +12 -12
  84. data/docs/architecture/storage-engine.md +14 -14
  85. data/docs/architecture/transactions.md +10 -10
  86. data/docs/architecture/wal.md +28 -28
  87. data/docs/cli-cheatsheet.md +98 -98
  88. data/docs/cli.md +299 -275
  89. data/docs/contributing/architecture.md +9 -9
  90. data/docs/contributing/benchmarking.md +30 -14
  91. data/docs/contributing/development.md +16 -16
  92. data/docs/contributing/release-process.md +49 -49
  93. data/docs/contributing/testing.md +16 -16
  94. data/docs/debugging.md +229 -229
  95. data/docs/developer/branching.md +10 -10
  96. data/docs/developer/database-diff.md +10 -10
  97. data/docs/developer/local-development.md +49 -17
  98. data/docs/developer/snapshots.md +9 -9
  99. data/docs/developer/temporal-data.md +10 -10
  100. data/docs/developer-guide.md +297 -297
  101. data/docs/getting-started/first-database.md +16 -16
  102. data/docs/getting-started/first-query.md +13 -13
  103. data/docs/getting-started/installation.md +19 -19
  104. data/docs/getting-started/local-to-production.md +300 -300
  105. data/docs/getting-started/quickstart.md +17 -17
  106. data/docs/getting-started/rails.md +16 -16
  107. data/docs/hardening_backlog.md +93 -93
  108. data/docs/lessons-learned.md +112 -112
  109. data/docs/operations/backups.md +33 -33
  110. data/docs/operations/disaster-recovery.md +31 -31
  111. data/docs/operations/failover.md +30 -30
  112. data/docs/operations/monitoring.md +25 -25
  113. data/docs/operations/production-guide.md +295 -295
  114. data/docs/operations/production-runbook.md +45 -45
  115. data/docs/operations/replication.md +33 -33
  116. data/docs/operations/restore.md +6 -6
  117. data/docs/operations/runbook.md +34 -34
  118. data/docs/operations/upgrades.md +14 -14
  119. data/docs/operations/workload-testing.md +17 -17
  120. data/docs/production-readiness.md +118 -118
  121. data/docs/production_validation.md +150 -150
  122. data/docs/rails/active-record.md +11 -11
  123. data/docs/rails/compatibility-guide.md +90 -90
  124. data/docs/rails/database-yml.md +92 -92
  125. data/docs/rails/installation.md +17 -17
  126. data/docs/rails/migrations.md +17 -17
  127. data/docs/rails/production.md +82 -82
  128. data/docs/rails/troubleshooting.md +18 -18
  129. data/docs/release.md +25 -25
  130. data/docs/server/architecture.md +10 -10
  131. data/docs/server/authentication.md +10 -10
  132. data/docs/server/configuration.md +16 -16
  133. data/docs/server/connection-pooling.md +10 -10
  134. data/docs/server/deployment.md +10 -10
  135. data/docs/server/protocol.md +12 -12
  136. data/docs/sql/compatibility-guide.md +82 -82
  137. data/docs/sql/compatibility.md +39 -39
  138. data/docs/sql/data-types.md +10 -10
  139. data/docs/sql/functions.md +9 -9
  140. data/docs/sql/joins.md +9 -9
  141. data/docs/sql/operators.md +9 -9
  142. data/docs/sql/sqlite-compatibility.md +21 -21
  143. data/docs/sql/syntax.md +10 -10
  144. data/docs/sql/transactions.md +10 -10
  145. data/docs/troubleshooting.md +244 -244
  146. data/lessons/01-foundations.md +73 -0
  147. data/lessons/02-local-development.md +121 -0
  148. data/lessons/03-embedded-rubydb.md +99 -0
  149. data/lessons/04-rails-complex-apps.md +138 -0
  150. data/lessons/05-rubydb-production-server.md +237 -0
  151. data/lessons/06-postgresql-massive-apps.md +96 -0
  152. data/lessons/07-hybrid-microservices.md +179 -0
  153. data/lessons/08-migrations-backups-recovery.md +86 -0
  154. data/lessons/09-observability-security-scale.md +87 -0
  155. data/lessons/10-release-readiness.md +192 -0
  156. data/lessons/11-community-adapter.md +323 -0
  157. data/lessons/12-rails-ecommerce-pressure.md +263 -0
  158. data/lib/rubydb/accelerator/client.rb +451 -0
  159. data/lib/rubydb/accelerator/error.rb +22 -0
  160. data/lib/rubydb/accelerator/manager.rb +606 -0
  161. data/lib/rubydb/accelerator.rb +13 -0
  162. data/lib/rubydb/backup/archive.rb +332 -334
  163. data/lib/rubydb/backup/backup.rb +400 -401
  164. data/lib/rubydb/backup/incremental.rb +349 -353
  165. data/lib/rubydb/backup/restore.rb +289 -290
  166. data/lib/rubydb/backup/snapshot.rb +265 -267
  167. data/lib/rubydb/backup/verification.rb +276 -279
  168. data/lib/rubydb/branching/branch.rb +181 -181
  169. data/lib/rubydb/branching/branch_manager.rb +307 -311
  170. data/lib/rubydb/branching/branch_metadata.rb +140 -140
  171. data/lib/rubydb/branching/checkout.rb +165 -166
  172. data/lib/rubydb/branching/copy_on_write.rb +272 -272
  173. data/lib/rubydb/branching/diff.rb +137 -138
  174. data/lib/rubydb/branching/merge.rb +282 -285
  175. data/lib/rubydb/build_info.rb +15 -15
  176. data/lib/rubydb/catalog/catalog.rb +391 -391
  177. data/lib/rubydb/catalog/column.rb +112 -112
  178. data/lib/rubydb/catalog/constraint.rb +180 -180
  179. data/lib/rubydb/catalog/database.rb +184 -184
  180. data/lib/rubydb/catalog/index.rb +97 -97
  181. data/lib/rubydb/catalog/schema.rb +103 -103
  182. data/lib/rubydb/catalog/sequence.rb +90 -90
  183. data/lib/rubydb/catalog/system_catalog.rb +698 -698
  184. data/lib/rubydb/catalog/table.rb +178 -178
  185. data/lib/rubydb/catalog/trigger.rb +102 -102
  186. data/lib/rubydb/catalog/view.rb +66 -66
  187. data/lib/rubydb/cli/application.rb +168 -163
  188. data/lib/rubydb/cli/commands/accelerator.rb +72 -0
  189. data/lib/rubydb/cli/commands/backup.rb +80 -81
  190. data/lib/rubydb/cli/commands/branch.rb +72 -72
  191. data/lib/rubydb/cli/commands/checkout.rb +54 -54
  192. data/lib/rubydb/cli/commands/create.rb +58 -58
  193. data/lib/rubydb/cli/commands/diff.rb +76 -77
  194. data/lib/rubydb/cli/commands/doctor.rb +77 -74
  195. data/lib/rubydb/cli/commands/drop.rb +57 -57
  196. data/lib/rubydb/cli/commands/init.rb +101 -102
  197. data/lib/rubydb/cli/commands/inspect.rb +95 -95
  198. data/lib/rubydb/cli/commands/merge.rb +63 -63
  199. data/lib/rubydb/cli/commands/migrate.rb +62 -62
  200. data/lib/rubydb/cli/commands/restart.rb +42 -39
  201. data/lib/rubydb/cli/commands/restore.rb +121 -121
  202. data/lib/rubydb/cli/commands/shell.rb +365 -365
  203. data/lib/rubydb/cli/commands/snapshot.rb +79 -79
  204. data/lib/rubydb/cli/commands/start.rb +88 -82
  205. data/lib/rubydb/cli/commands/status.rb +96 -92
  206. data/lib/rubydb/cli/commands/stop.rb +47 -47
  207. data/lib/rubydb/cli/commands/vacuum.rb +58 -58
  208. data/lib/rubydb/cli/formatter.rb +221 -221
  209. data/lib/rubydb/cli/output.rb +168 -168
  210. data/lib/rubydb/client/client.rb +309 -304
  211. data/lib/rubydb/client/connection.rb +429 -415
  212. data/lib/rubydb/client/connection_pool.rb +168 -168
  213. data/lib/rubydb/client/connection_url.rb +96 -96
  214. data/lib/rubydb/client/prepared_statement.rb +60 -60
  215. data/lib/rubydb/client/result.rb +127 -123
  216. data/lib/rubydb/client/statement.rb +52 -52
  217. data/lib/rubydb/client/transaction.rb +130 -130
  218. data/lib/rubydb/concurrency/concurrency.rb +19 -19
  219. data/lib/rubydb/concurrency/deadlock_detector.rb +148 -150
  220. data/lib/rubydb/concurrency/latch.rb +101 -101
  221. data/lib/rubydb/concurrency/lock_graph.rb +163 -165
  222. data/lib/rubydb/concurrency/mutex.rb +181 -183
  223. data/lib/rubydb/concurrency/rw_lock.rb +180 -180
  224. data/lib/rubydb/concurrency/scheduler.rb +248 -250
  225. data/lib/rubydb/concurrency/worker_pool.rb +145 -143
  226. data/lib/rubydb/configuration/config.rb +170 -170
  227. data/lib/rubydb/configuration/defaults.rb +191 -179
  228. data/lib/rubydb/configuration/environment.rb +152 -152
  229. data/lib/rubydb/configuration/parser.rb +185 -185
  230. data/lib/rubydb/configuration/validation.rb +228 -221
  231. data/lib/rubydb/constants.rb +74 -74
  232. data/lib/rubydb/constraints/check.rb +181 -181
  233. data/lib/rubydb/constraints/constraint.rb +101 -101
  234. data/lib/rubydb/constraints/foreign_key.rb +130 -130
  235. data/lib/rubydb/constraints/not_null.rb +64 -64
  236. data/lib/rubydb/constraints/primary_key.rb +99 -99
  237. data/lib/rubydb/constraints/unique.rb +106 -108
  238. data/lib/rubydb/constraints/validator.rb +349 -350
  239. data/lib/rubydb/errors/authentication_error.rb +10 -10
  240. data/lib/rubydb/errors/authorization_error.rb +23 -23
  241. data/lib/rubydb/errors/client_error.rb +10 -10
  242. data/lib/rubydb/errors/configuration_error.rb +10 -10
  243. data/lib/rubydb/errors/connection_error.rb +10 -10
  244. data/lib/rubydb/errors/constraint_error.rb +23 -23
  245. data/lib/rubydb/errors/corruption_error.rb +10 -10
  246. data/lib/rubydb/errors/database_error.rb +10 -10
  247. data/lib/rubydb/errors/error.rb +20 -20
  248. data/lib/rubydb/errors/execution_error.rb +10 -10
  249. data/lib/rubydb/errors/parser_error.rb +10 -10
  250. data/lib/rubydb/errors/recovery_error.rb +10 -10
  251. data/lib/rubydb/errors/replication_error.rb +10 -10
  252. data/lib/rubydb/errors/server_error.rb +6 -6
  253. data/lib/rubydb/errors/storage_error.rb +10 -10
  254. data/lib/rubydb/errors/transaction_error.rb +10 -10
  255. data/lib/rubydb/execution/accelerator_dispatch.rb +30 -0
  256. data/lib/rubydb/execution/aggregate_executor.rb +134 -138
  257. data/lib/rubydb/execution/cost_model.rb +72 -0
  258. data/lib/rubydb/execution/delete_executor.rb +110 -112
  259. data/lib/rubydb/execution/distinct_executor.rb +131 -135
  260. data/lib/rubydb/execution/executor.rb +1544 -1188
  261. data/lib/rubydb/execution/expression.rb +191 -193
  262. data/lib/rubydb/execution/index_scan.rb +142 -142
  263. data/lib/rubydb/execution/insert_executor.rb +215 -217
  264. data/lib/rubydb/execution/join_executor.rb +243 -249
  265. data/lib/rubydb/execution/limit_executor.rb +83 -85
  266. data/lib/rubydb/execution/operator_selection.rb +57 -0
  267. data/lib/rubydb/execution/optimizer.rb +227 -215
  268. data/lib/rubydb/execution/physical_plan.rb +47 -0
  269. data/lib/rubydb/execution/plan.rb +355 -353
  270. data/lib/rubydb/execution/planner.rb +508 -536
  271. data/lib/rubydb/execution/predicate.rb +235 -235
  272. data/lib/rubydb/execution/scan.rb +49 -49
  273. data/lib/rubydb/execution/sequential_scan.rb +63 -63
  274. data/lib/rubydb/execution/sort_executor.rb +194 -185
  275. data/lib/rubydb/execution/update_executor.rb +160 -162
  276. data/lib/rubydb/functions/aggregate.rb +70 -70
  277. data/lib/rubydb/functions/date_functions.rb +274 -278
  278. data/lib/rubydb/functions/function.rb +85 -85
  279. data/lib/rubydb/functions/json_functions.rb +231 -215
  280. data/lib/rubydb/functions/numeric_functions.rb +346 -346
  281. data/lib/rubydb/functions/scalar.rb +52 -52
  282. data/lib/rubydb/functions/string_functions.rb +383 -383
  283. data/lib/rubydb/functions/system_functions.rb +258 -246
  284. data/lib/rubydb/history/as_of.rb +238 -238
  285. data/lib/rubydb/history/change.rb +105 -105
  286. data/lib/rubydb/history/history.rb +131 -131
  287. data/lib/rubydb/history/history_manager.rb +228 -229
  288. data/lib/rubydb/history/temporal_query.rb +202 -202
  289. data/lib/rubydb/history/timeline.rb +144 -144
  290. data/lib/rubydb/indexes/btree.rb +215 -186
  291. data/lib/rubydb/indexes/btree_cursor.rb +258 -258
  292. data/lib/rubydb/indexes/btree_node.rb +384 -385
  293. data/lib/rubydb/indexes/hash_index.rb +150 -150
  294. data/lib/rubydb/indexes/index.rb +71 -71
  295. data/lib/rubydb/indexes/index_manager.rb +408 -406
  296. data/lib/rubydb/indexes/index_scan.rb +466 -470
  297. data/lib/rubydb/migrations/migration.rb +253 -254
  298. data/lib/rubydb/migrations/migration_lock.rb +146 -146
  299. data/lib/rubydb/migrations/migration_manager.rb +187 -176
  300. data/lib/rubydb/migrations/migration_version.rb +71 -71
  301. data/lib/rubydb/migrations/schema_diff.rb +211 -211
  302. data/lib/rubydb/migrations/schema_version.rb +64 -64
  303. data/lib/rubydb/monitoring/events.rb +155 -160
  304. data/lib/rubydb/monitoring/health.rb +216 -222
  305. data/lib/rubydb/monitoring/logger.rb +188 -193
  306. data/lib/rubydb/monitoring/metrics.rb +363 -359
  307. data/lib/rubydb/monitoring/performance.rb +176 -176
  308. data/lib/rubydb/monitoring/statistics.rb +168 -170
  309. data/lib/rubydb/mvcc/garbage_collector.rb +199 -199
  310. data/lib/rubydb/mvcc/mvcc.rb +16 -16
  311. data/lib/rubydb/mvcc/snapshot.rb +146 -147
  312. data/lib/rubydb/mvcc/vacuum.rb +180 -180
  313. data/lib/rubydb/mvcc/version.rb +106 -106
  314. data/lib/rubydb/mvcc/version_store.rb +396 -398
  315. data/lib/rubydb/mvcc/visibility.rb +107 -109
  316. data/lib/rubydb/protocol/capabilities.rb +125 -125
  317. data/lib/rubydb/protocol/decoder.rb +142 -145
  318. data/lib/rubydb/protocol/encoder.rb +131 -136
  319. data/lib/rubydb/protocol/handshake.rb +306 -305
  320. data/lib/rubydb/protocol/message.rb +121 -121
  321. data/lib/rubydb/protocol/parameter_binder.rb +101 -0
  322. data/lib/rubydb/protocol/protocol.rb +276 -277
  323. data/lib/rubydb/protocol/version.rb +54 -54
  324. data/lib/rubydb/rails/adapter.rb +245 -239
  325. data/lib/rubydb/rails/connection.rb +312 -314
  326. data/lib/rubydb/rails/database_statements.rb +122 -122
  327. data/lib/rubydb/rails/migration.rb +131 -131
  328. data/lib/rubydb/rails/quoting.rb +109 -109
  329. data/lib/rubydb/rails/result.rb +117 -117
  330. data/lib/rubydb/rails/schema_statements.rb +339 -339
  331. data/lib/rubydb/rails/transaction.rb +105 -105
  332. data/lib/rubydb/rails/type.rb +126 -126
  333. data/lib/rubydb/recovery/checkpoint.rb +261 -257
  334. data/lib/rubydb/recovery/consistency.rb +457 -467
  335. data/lib/rubydb/recovery/corruption_detector.rb +5 -5
  336. data/lib/rubydb/recovery/crash_recovery.rb +381 -387
  337. data/lib/rubydb/recovery/recovery_manager.rb +204 -206
  338. data/lib/rubydb/recovery/redo.rb +235 -237
  339. data/lib/rubydb/recovery/undo.rb +204 -206
  340. data/lib/rubydb/replication/failover.rb +5 -5
  341. data/lib/rubydb/replication/fencing.rb +63 -63
  342. data/lib/rubydb/replication/primary.rb +461 -450
  343. data/lib/rubydb/replication/replica.rb +382 -384
  344. data/lib/rubydb/replication/replication_log.rb +194 -200
  345. data/lib/rubydb/replication/replication_manager.rb +307 -308
  346. data/lib/rubydb/replication/replication_slot.rb +293 -295
  347. data/lib/rubydb/replication/replication_stream.rb +198 -201
  348. data/lib/rubydb/rubydb.rb +570 -560
  349. data/lib/rubydb/security/access_control.rb +252 -254
  350. data/lib/rubydb/security/audit_log.rb +209 -213
  351. data/lib/rubydb/security/authentication.rb +302 -302
  352. data/lib/rubydb/security/authorization.rb +282 -282
  353. data/lib/rubydb/security/credentials.rb +192 -196
  354. data/lib/rubydb/security/password.rb +205 -215
  355. data/lib/rubydb/security/permissions.rb +74 -74
  356. data/lib/rubydb/security/role.rb +99 -101
  357. data/lib/rubydb/security/user.rb +86 -86
  358. data/lib/rubydb/server/connection.rb +383 -366
  359. data/lib/rubydb/server/connection_pool.rb +193 -193
  360. data/lib/rubydb/server/lifecycle.rb +227 -228
  361. data/lib/rubydb/server/listener.rb +139 -136
  362. data/lib/rubydb/server/request_handler.rb +277 -276
  363. data/lib/rubydb/server/server.rb +363 -364
  364. data/lib/rubydb/server/session.rb +416 -369
  365. data/lib/rubydb/server/worker.rb +206 -210
  366. data/lib/rubydb/server/worker_pool.rb +168 -168
  367. data/lib/rubydb/sql/ast/alter_table.rb +169 -169
  368. data/lib/rubydb/sql/ast/begin_transaction.rb +47 -47
  369. data/lib/rubydb/sql/ast/commit.rb +37 -37
  370. data/lib/rubydb/sql/ast/constraint.rb +92 -83
  371. data/lib/rubydb/sql/ast/create_database.rb +41 -41
  372. data/lib/rubydb/sql/ast/create_index.rb +61 -61
  373. data/lib/rubydb/sql/ast/create_schema.rb +52 -52
  374. data/lib/rubydb/sql/ast/create_table.rb +187 -187
  375. data/lib/rubydb/sql/ast/delete.rb +54 -54
  376. data/lib/rubydb/sql/ast/drop_database.rb +41 -41
  377. data/lib/rubydb/sql/ast/drop_index.rb +41 -41
  378. data/lib/rubydb/sql/ast/drop_schema.rb +49 -49
  379. data/lib/rubydb/sql/ast/drop_table.rb +49 -49
  380. data/lib/rubydb/sql/ast/explain.rb +64 -64
  381. data/lib/rubydb/sql/ast/expression.rb +617 -604
  382. data/lib/rubydb/sql/ast/insert.rb +66 -66
  383. data/lib/rubydb/sql/ast/node.rb +42 -42
  384. data/lib/rubydb/sql/ast/rollback.rb +63 -63
  385. data/lib/rubydb/sql/ast/savepoint.rb +59 -59
  386. data/lib/rubydb/sql/ast/select.rb +88 -88
  387. data/lib/rubydb/sql/ast/set_operation.rb +22 -20
  388. data/lib/rubydb/sql/ast/trigger.rb +35 -29
  389. data/lib/rubydb/sql/ast/update.rb +88 -88
  390. data/lib/rubydb/sql/ast/vacuum.rb +19 -19
  391. data/lib/rubydb/sql/ast/view.rb +38 -32
  392. data/lib/rubydb/sql/ast/with.rb +32 -32
  393. data/lib/rubydb/sql/grammar.rb +86 -86
  394. data/lib/rubydb/sql/keywords.rb +156 -156
  395. data/lib/rubydb/sql/lexer.rb +209 -214
  396. data/lib/rubydb/sql/operators.rb +100 -100
  397. data/lib/rubydb/sql/parser.rb +1167 -1170
  398. data/lib/rubydb/sql/planner/analyzer.rb +283 -302
  399. data/lib/rubydb/sql/planner/binder.rb +537 -550
  400. data/lib/rubydb/sql/planner/type_checker.rb +427 -431
  401. data/lib/rubydb/sql/token.rb +210 -210
  402. data/lib/rubydb/storage/buffer_frame.rb +44 -44
  403. data/lib/rubydb/storage/buffer_pool.rb +155 -155
  404. data/lib/rubydb/storage/database_lock.rb +74 -74
  405. data/lib/rubydb/storage/deserializer.rb +332 -342
  406. data/lib/rubydb/storage/engine.rb +2409 -2330
  407. data/lib/rubydb/storage/file_manager.rb +191 -187
  408. data/lib/rubydb/storage/free_space_map.rb +79 -81
  409. data/lib/rubydb/storage/page.rb +92 -94
  410. data/lib/rubydb/storage/page_allocator.rb +852 -855
  411. data/lib/rubydb/storage/page_header.rb +63 -67
  412. data/lib/rubydb/storage/page_manager.rb +127 -131
  413. data/lib/rubydb/storage/record.rb +58 -58
  414. data/lib/rubydb/storage/row.rb +78 -78
  415. data/lib/rubydb/storage/serializer.rb +51 -51
  416. data/lib/rubydb/storage/snapshot_reader.rb +167 -0
  417. data/lib/rubydb/storage/storage_layout.rb +151 -151
  418. data/lib/rubydb/storage/storage_manager.rb +114 -114
  419. data/lib/rubydb/storage/tuple.rb +458 -461
  420. data/lib/rubydb/storage/visibility_map.rb +964 -973
  421. data/lib/rubydb/transactions/commit_manager.rb +219 -220
  422. data/lib/rubydb/transactions/isolation.rb +98 -98
  423. data/lib/rubydb/transactions/lock.rb +76 -76
  424. data/lib/rubydb/transactions/lock_manager.rb +359 -362
  425. data/lib/rubydb/transactions/savepoint.rb +142 -143
  426. data/lib/rubydb/transactions/transaction.rb +214 -215
  427. data/lib/rubydb/transactions/transaction_id.rb +84 -84
  428. data/lib/rubydb/transactions/transaction_log.rb +256 -257
  429. data/lib/rubydb/transactions/transaction_manager.rb +434 -435
  430. data/lib/rubydb/types/bigint.rb +36 -36
  431. data/lib/rubydb/types/blob.rb +37 -37
  432. data/lib/rubydb/types/boolean.rb +34 -34
  433. data/lib/rubydb/types/date.rb +39 -39
  434. data/lib/rubydb/types/decimal.rb +48 -48
  435. data/lib/rubydb/types/float.rb +34 -34
  436. data/lib/rubydb/types/integer.rb +36 -36
  437. data/lib/rubydb/types/json.rb +41 -41
  438. data/lib/rubydb/types/null.rb +34 -34
  439. data/lib/rubydb/types/smallint.rb +36 -36
  440. data/lib/rubydb/types/text.rb +37 -37
  441. data/lib/rubydb/types/time.rb +46 -46
  442. data/lib/rubydb/types/timestamp.rb +39 -39
  443. data/lib/rubydb/types/type.rb +119 -119
  444. data/lib/rubydb/types/uuid.rb +47 -47
  445. data/lib/rubydb/types/varchar.rb +37 -37
  446. data/lib/rubydb/version.rb +32 -32
  447. data/lib/rubydb/wal/archive.rb +207 -193
  448. data/lib/rubydb/wal/checkpoint.rb +181 -183
  449. data/lib/rubydb/wal/lsn.rb +94 -94
  450. data/lib/rubydb/wal/reader.rb +259 -260
  451. data/lib/rubydb/wal/record.rb +105 -105
  452. data/lib/rubydb/wal/segment.rb +193 -193
  453. data/lib/rubydb/wal/wal.rb +481 -452
  454. data/lib/rubydb/wal/writer.rb +236 -236
  455. data/lib/rubydb.rb +7 -7
  456. data/packaging/docker/docker-compose.failover.yml +43 -43
  457. data/packaging/homebrew/rubydb.rb +19 -19
  458. data/rubydb.gemspec +70 -57
  459. data/scripts/benchmark +7 -7
  460. data/scripts/build_accelerator +49 -0
  461. data/scripts/durability_drill +37 -37
  462. data/scripts/fuzz +63 -63
  463. data/scripts/release +77 -42
  464. data/scripts/release_check +43 -43
  465. data/scripts/replication_failover_drill +268 -250
  466. data/scripts/replication_network_failover_drill +287 -255
  467. data/scripts/restore_drill +45 -45
  468. data/scripts/security +45 -0
  469. metadata +102 -1
@@ -1,297 +1,297 @@
1
- # RubyDB developer guide
2
-
3
- This is the implementation guide for contributors who need to change RubyDB
4
- safely. It explains the repository, the request path, the storage invariants,
5
- the test strategy, and the debugging workflow. Read it together with the
6
- topic specifications under `spec/`; the specifications are the behavioral
7
- contract, while this guide explains how the pieces fit together.
8
-
9
- ## 1. Scope and support promise
10
-
11
- RubyDB is a Ruby-native relational database with two ownership modes:
12
-
13
- * embedded mode, where one Ruby process owns a database directory; and
14
- * server mode, where the server owns the directory and application processes
15
- use the client protocol.
16
-
17
- The supported surface is the behavior exercised by the test suite and listed
18
- in [SQL compatibility](sql/compatibility.md). Similar syntax does not imply
19
- complete PostgreSQL, MySQL, or SQLite compatibility. A change that broadens
20
- syntax must also specify its semantics, errors, types, transactions, and
21
- adapter behavior.
22
-
23
- ## 2. Repository orientation
24
-
25
- The important directories are:
26
-
27
- | Path | Responsibility |
28
- | --- | --- |
29
- | `lib/rubydb` | Engine, SQL, storage, transactions, server, client, adapters |
30
- | `spec` | Unit, integration, protocol, SQL, recovery, and adapter contracts |
31
- | `benchmarks` | Repeatable throughput and concurrency measurements |
32
- | `scripts` | Soak tests, durability drills, fuzzing, and release checks |
33
- | `config` | Example configuration and monitoring rules |
34
- | `examples` | Runnable Ruby and Rails applications |
35
- | `docs` | User, operator, developer, and compatibility documentation |
36
- | `packaging` | Container and deployment examples |
37
- | `exe` and `bin` | Public command-line entry points |
38
-
39
- Start with `lib/rubydb.rb` and follow the public API into the engine. For a
40
- feature, locate the nearest existing spec before editing implementation code.
41
- Avoid adding a second abstraction when an existing subsystem already owns the
42
- invariant.
43
-
44
- ## 3. Local setup
45
-
46
- Use a supported Ruby version from the project CI matrix and install the locked
47
- dependencies:
48
-
49
- ```sh
50
- bundle install
51
- bundle exec rspec
52
- bundle exec rubocop
53
- ```
54
-
55
- Use a temporary database for experiments. Never use a production path in a
56
- test or run a destructive command against an unknown directory:
57
-
58
- ```ruby
59
- require "tmpdir"
60
- require "rubydb"
61
-
62
- Dir.mktmpdir("rubydb-dev-") do |dir|
63
- engine = RubyDB::Storage::Engine.new(File.join(dir, "dev.rdb"))
64
- engine.execute("CREATE TABLE items (id INTEGER PRIMARY KEY, name TEXT)")
65
- engine.close
66
- end
67
- ```
68
-
69
- The server/client path is required when testing process boundaries. Multiple
70
- independent embedded owners must never open the same database directory.
71
-
72
- ## 4. The request lifecycle
73
-
74
- A typical SQL request follows this sequence:
75
-
76
- ```text
77
- Ruby API or client
78
- -> connection/session and authentication
79
- -> SQL tokenizer/parser
80
- -> binder and type/parameter validation
81
- -> planner and executor
82
- -> transaction/MVCC read or write set
83
- -> table/index mutation
84
- -> WAL append and durable commit
85
- -> result encoding and protocol response
86
- ```
87
-
88
- Keep failures at the layer that owns them. Parser errors should not be turned
89
- into storage errors. A failed WAL append must not report a committed schema or
90
- row change. A client timeout must not silently convert an unknown commit into
91
- a rollback; the caller must be able to determine whether a request was
92
- committed before retrying.
93
-
94
- ## 5. Storage and durability invariants
95
-
96
- The storage directory contains data pages, metadata, WAL state, and recovery
97
- artifacts. Changes must preserve these invariants:
98
-
99
- 1. A page has valid framing and checksum before it is trusted.
100
- 2. WAL records are validated before replay and are applied in LSN order.
101
- 3. A commit acknowledgement is emitted only after the configured durability
102
- point has succeeded.
103
- 4. Schema publication and its dependent indexes are visible atomically.
104
- 5. Recovery is idempotent: replaying a committed record does not duplicate a
105
- row or corrupt an index.
106
- 6. Torn, truncated, or corrupted input fails closed with a useful error.
107
- 7. Temporary files are not mistaken for a completed checkpoint or backup.
108
-
109
- When changing page layout or record encoding, update the format specification,
110
- versioning/upgrade path, compatibility tests, and recovery fixtures. Do not
111
- silently reinterpret old bytes. If a format cannot be read, return an explicit
112
- upgrade or corruption error and preserve the original files for diagnosis.
113
-
114
- ## 6. WAL, checkpoints, and recovery
115
-
116
- The WAL is the source of truth between checkpoints. A checkpoint copies safe
117
- state to durable pages and advances the recovery boundary only after all
118
- required data and metadata have been flushed. Recovery should:
119
-
120
- 1. open the directory read-only where possible;
121
- 2. validate metadata and the last known checkpoint;
122
- 3. scan WAL frames, stopping only at a valid end boundary;
123
- 4. reject checksum, sequence, length, or transaction inconsistencies;
124
- 5. replay committed work and discard incomplete transactions;
125
- 6. rebuild or validate indexes before accepting writes; and
126
- 7. publish a recovery result with the replayed LSN and warnings.
127
-
128
- Tests must cover normal reopen, a process terminated during a write, an
129
- interrupted checkpoint, truncated WAL, invalid checksums, missing metadata,
130
- full-disk behavior, and restore into a new directory. Fault injection belongs
131
- around filesystem calls, not only around Ruby methods, because failures occur
132
- at `fsync`, rename, allocation, and close boundaries.
133
-
134
- ## 7. Transactions, MVCC, and locking
135
-
136
- Every statement executes in a transaction context, whether it is explicit or
137
- implicit. The context owns the snapshot, read view, write set, lock state,
138
- savepoints, and commit result. A transaction must not leak locks or snapshots
139
- when it raises, times out, is cancelled, or loses its connection.
140
-
141
- MVCC readers use a stable visibility point. Writers create new versions and
142
- retain the before-image needed by active readers and rollback. Vacuum may
143
- reclaim a version only after the global safe point has passed it. A new feature
144
- must define behavior for:
145
-
146
- * read committed, repeatable read, and serializable transactions;
147
- * concurrent update of the same row;
148
- * unique and foreign-key conflicts;
149
- * deadlock detection and victim rollback;
150
- * lock wait timeouts and request cancellation; and
151
- * commit acknowledgement followed by client disconnect.
152
-
153
- Deadlock resolution must abort a complete victim transaction, release all of
154
- its locks, and leave other transactions able to progress. Never fix a deadlock
155
- by globally disabling locking or by releasing a lock without undoing the
156
- corresponding write set.
157
-
158
- ## 8. SQL implementation workflow
159
-
160
- For a new statement or expression:
161
-
162
- 1. write the syntax and semantic contract in `spec/sql`;
163
- 2. add parser acceptance and rejection examples;
164
- 3. add binder/type/nullability behavior;
165
- 4. add planner and executor tests;
166
- 5. test transaction, constraint, index, and error interactions;
167
- 6. test the Ruby API and server protocol path; and
168
- 7. test the ActiveRecord-generated SQL when the feature is adapter-visible.
169
-
170
- Prefer parameter binding to string interpolation. Every expression needs
171
- defined behavior for `NULL`, booleans, numeric coercion, text comparison,
172
- collation, and invalid input. Every DDL operation needs idempotence or an
173
- explicit error contract. Every optimizer rewrite must have a semantic
174
- equivalence test against the non-optimized execution path.
175
-
176
- The documented common SQLite-style profile is intentionally narrower than
177
- SQLite itself. Do not label a feature “SQLite compatible” until its syntax,
178
- results, types, error behavior, and migration behavior are tested.
179
-
180
- ## 9. Tables, indexes, and constraints
181
-
182
- Table mutations and index mutations are one logical operation. If an index
183
- write fails, the statement must fail visibly and the transaction must either
184
- roll back or retain a recoverable pending state; it must not acknowledge a row
185
- that cannot be found by a required index.
186
-
187
- For each index type, test empty and populated tables, duplicate keys, `NULL`,
188
- deep splits, reopen, recovery replay, deletion/merge, and concurrent readers.
189
- For each constraint, test direct SQL, prepared parameters, ActiveRecord
190
- inserts/updates, rollback, and the exact error class/message contract where
191
- callers may depend on it.
192
-
193
- ## 10. Server, client, and wire protocol
194
-
195
- The server is the single owner of the database directory. A client session
196
- handles authentication, capability negotiation, request IDs, deadlines,
197
- cancellation, result framing, and connection shutdown. Protocol changes must
198
- be backward-compatible or carry an explicit protocol version and rejection
199
- path.
200
-
201
- Wire cancellation is part of correctness. A cancelled request must stop
202
- execution at safe checkpoints, release transaction resources, and return a
203
- definitive cancellation response. If cancellation races with commit, the
204
- server must preserve the commit outcome and expose enough request identity for
205
- the client to query status rather than blindly retrying.
206
-
207
- Test malformed lengths, unknown message types, duplicate request IDs, partial
208
- frames, client disconnects, server shutdown, timeout races, authentication
209
- failure, TLS negotiation, and cancellation during a long scan.
210
-
211
- ## 11. Rails and adapter work
212
-
213
- The ActiveRecord adapter translates Rails schema and query APIs into RubyDB
214
- SQL. Adapter code must preserve Rails expectations for quoting, bind
215
- parameters, affected-row counts, last-insert IDs, transactions, savepoints,
216
- schema introspection, migration versions, and exceptions.
217
-
218
- When changing adapter behavior, run the example app and test at least:
219
-
220
- * `where`, ordering, limits, scopes, joins, and aggregate relations;
221
- * eager loading, nested associations, and inverse relationships;
222
- * connection pools with multiple threads;
223
- * schema dump/load and populated-table migrations;
224
- * rollback, retry, and migration checksum behavior; and
225
- * every supported Ruby/Rails combination in CI.
226
-
227
- The adapter is not proof of complete SQLite or PostgreSQL compatibility. An
228
- application should run its own generated-SQL and migration suite before a
229
- cutover.
230
-
231
- ## 12. Testing pyramid
232
-
233
- Use the smallest test that proves the invariant:
234
-
235
- * unit specs for parsing, encoding, types, and isolated algorithms;
236
- * component specs for storage, WAL, indexes, transactions, and protocol;
237
- * integration specs for engine-to-SQL and server-to-client behavior;
238
- * process specs for crash recovery, pooling, and failover;
239
- * workload tests for sustained concurrency and resource limits; and
240
- * example applications for real Rails and Ruby workflows.
241
-
242
- Useful commands include:
243
-
244
- ```sh
245
- bundle exec rspec
246
- bundle exec rspec spec/sql spec/storage spec/transactions
247
- ruby scripts/durability_drill
248
- ruby scripts/production_soak
249
- ruby scripts/fuzz
250
- RUBYDB_BENCHMARK_ITERATIONS=100 ruby -Ilib benchmarks/basic_workload.rb
251
- ```
252
-
253
- Record the Ruby version, commit, OS, command, seed, database configuration,
254
- and artifact paths for every non-trivial run. A green unit suite does not
255
- replace a restore drill, multi-process soak, or deployment test.
256
-
257
- ## 13. Debugging a failing change
258
-
259
- Start with [Debugging RubyDB](debugging.md). Reproduce on a fresh temporary
260
- directory, reduce the schema and SQL, rerun with the reported seed, and save
261
- logs plus the WAL metadata. Compare embedded and server/client execution when
262
- ownership or protocol is suspected. Use [Troubleshooting](troubleshooting.md)
263
- for operator symptoms and evidence-preserving recovery steps.
264
-
265
- Never “repair” a failing test by deleting WAL, disabling checksums, loosening a
266
- constraint, or turning off synchronization. Preserve the failing directory,
267
- copy it, and investigate the copy.
268
-
269
- ## 14. Performance and capacity work
270
-
271
- Benchmark one variable at a time and report latency percentiles, throughput,
272
- concurrency, payload shape, cache size, WAL/checkpoint settings, and storage
273
- medium. Track p50, p95, p99, error rate, lock wait time, WAL growth, checkpoint
274
- duration, memory, file descriptors, and CPU.
275
-
276
- Capacity limits are workload-specific. A benchmark result is not a guarantee
277
- for a Rails application with different indexes, query shapes, or connection
278
- pool settings. Add a regression threshold only after the benchmark is stable
279
- across repeated runs and environments.
280
-
281
- ## 15. Safe contribution checklist
282
-
283
- Before opening a pull request:
284
-
285
- 1. Explain the invariant and compatibility impact.
286
- 2. Add focused tests, including failure paths.
287
- 3. Run the relevant focused suite and the full suite.
288
- 4. Run formatting/lint and `git diff --check`.
289
- 5. Update the appropriate spec and user/operator documentation.
290
- 6. Note migrations, format changes, recovery implications, and rollback plan.
291
- 7. Include benchmark or soak evidence for hot-path and concurrency changes.
292
- 8. Remove debug output, secrets, temporary data, and generated artifacts.
293
-
294
- See [Contributing](../CONTRIBUTING.md), [testing](contributing/testing.md),
295
- and [release process](contributing/release-process.md) for the repository
296
- workflow.
297
-
1
+ # RubyDB developer guide
2
+
3
+ This is the implementation guide for contributors who need to change RubyDB
4
+ safely. It explains the repository, the request path, the storage invariants,
5
+ the test strategy, and the debugging workflow. Read it together with the
6
+ topic specifications under `spec/`; the specifications are the behavioral
7
+ contract, while this guide explains how the pieces fit together.
8
+
9
+ ## 1. Scope and support promise
10
+
11
+ RubyDB is a Ruby-native relational database with two ownership modes:
12
+
13
+ * embedded mode, where one Ruby process owns a database directory; and
14
+ * server mode, where the server owns the directory and application processes
15
+ use the client protocol.
16
+
17
+ The supported surface is the behavior exercised by the test suite and listed
18
+ in [SQL compatibility](sql/compatibility.md). Similar syntax does not imply
19
+ complete PostgreSQL, MySQL, or SQLite compatibility. A change that broadens
20
+ syntax must also specify its semantics, errors, types, transactions, and
21
+ adapter behavior.
22
+
23
+ ## 2. Repository orientation
24
+
25
+ The important directories are:
26
+
27
+ | Path | Responsibility |
28
+ | --- | --- |
29
+ | `lib/rubydb` | Engine, SQL, storage, transactions, server, client, adapters |
30
+ | `spec` | Unit, integration, protocol, SQL, recovery, and adapter contracts |
31
+ | `benchmarks` | Repeatable throughput and concurrency measurements |
32
+ | `scripts` | Soak tests, durability drills, fuzzing, and release checks |
33
+ | `config` | Example configuration and monitoring rules |
34
+ | `examples` | Runnable Ruby and Rails applications |
35
+ | `docs` | User, operator, developer, and compatibility documentation |
36
+ | `packaging` | Container and deployment examples |
37
+ | `exe` and `bin` | Public command-line entry points |
38
+
39
+ Start with `lib/rubydb.rb` and follow the public API into the engine. For a
40
+ feature, locate the nearest existing spec before editing implementation code.
41
+ Avoid adding a second abstraction when an existing subsystem already owns the
42
+ invariant.
43
+
44
+ ## 3. Local setup
45
+
46
+ Use a supported Ruby version from the project CI matrix and install the locked
47
+ dependencies:
48
+
49
+ ```sh
50
+ bundle install
51
+ bundle exec rspec
52
+ bundle exec rubocop
53
+ ```
54
+
55
+ Use a temporary database for experiments. Never use a production path in a
56
+ test or run a destructive command against an unknown directory:
57
+
58
+ ```ruby
59
+ require "tmpdir"
60
+ require "rubydb"
61
+
62
+ Dir.mktmpdir("rubydb-dev-") do |dir|
63
+ engine = RubyDB::Storage::Engine.new(File.join(dir, "dev.rdb"))
64
+ engine.execute("CREATE TABLE items (id INTEGER PRIMARY KEY, name TEXT)")
65
+ engine.close
66
+ end
67
+ ```
68
+
69
+ The server/client path is required when testing process boundaries. Multiple
70
+ independent embedded owners must never open the same database directory.
71
+
72
+ ## 4. The request lifecycle
73
+
74
+ A typical SQL request follows this sequence:
75
+
76
+ ```text
77
+ Ruby API or client
78
+ -> connection/session and authentication
79
+ -> SQL tokenizer/parser
80
+ -> binder and type/parameter validation
81
+ -> planner and executor
82
+ -> transaction/MVCC read or write set
83
+ -> table/index mutation
84
+ -> WAL append and durable commit
85
+ -> result encoding and protocol response
86
+ ```
87
+
88
+ Keep failures at the layer that owns them. Parser errors should not be turned
89
+ into storage errors. A failed WAL append must not report a committed schema or
90
+ row change. A client timeout must not silently convert an unknown commit into
91
+ a rollback; the caller must be able to determine whether a request was
92
+ committed before retrying.
93
+
94
+ ## 5. Storage and durability invariants
95
+
96
+ The storage directory contains data pages, metadata, WAL state, and recovery
97
+ artifacts. Changes must preserve these invariants:
98
+
99
+ 1. A page has valid framing and checksum before it is trusted.
100
+ 2. WAL records are validated before replay and are applied in LSN order.
101
+ 3. A commit acknowledgement is emitted only after the configured durability
102
+ point has succeeded.
103
+ 4. Schema publication and its dependent indexes are visible atomically.
104
+ 5. Recovery is idempotent: replaying a committed record does not duplicate a
105
+ row or corrupt an index.
106
+ 6. Torn, truncated, or corrupted input fails closed with a useful error.
107
+ 7. Temporary files are not mistaken for a completed checkpoint or backup.
108
+
109
+ When changing page layout or record encoding, update the format specification,
110
+ versioning/upgrade path, compatibility tests, and recovery fixtures. Do not
111
+ silently reinterpret old bytes. If a format cannot be read, return an explicit
112
+ upgrade or corruption error and preserve the original files for diagnosis.
113
+
114
+ ## 6. WAL, checkpoints, and recovery
115
+
116
+ The WAL is the source of truth between checkpoints. A checkpoint copies safe
117
+ state to durable pages and advances the recovery boundary only after all
118
+ required data and metadata have been flushed. Recovery should:
119
+
120
+ 1. open the directory read-only where possible;
121
+ 2. validate metadata and the last known checkpoint;
122
+ 3. scan WAL frames, stopping only at a valid end boundary;
123
+ 4. reject checksum, sequence, length, or transaction inconsistencies;
124
+ 5. replay committed work and discard incomplete transactions;
125
+ 6. rebuild or validate indexes before accepting writes; and
126
+ 7. publish a recovery result with the replayed LSN and warnings.
127
+
128
+ Tests must cover normal reopen, a process terminated during a write, an
129
+ interrupted checkpoint, truncated WAL, invalid checksums, missing metadata,
130
+ full-disk behavior, and restore into a new directory. Fault injection belongs
131
+ around filesystem calls, not only around Ruby methods, because failures occur
132
+ at `fsync`, rename, allocation, and close boundaries.
133
+
134
+ ## 7. Transactions, MVCC, and locking
135
+
136
+ Every statement executes in a transaction context, whether it is explicit or
137
+ implicit. The context owns the snapshot, read view, write set, lock state,
138
+ savepoints, and commit result. A transaction must not leak locks or snapshots
139
+ when it raises, times out, is cancelled, or loses its connection.
140
+
141
+ MVCC readers use a stable visibility point. Writers create new versions and
142
+ retain the before-image needed by active readers and rollback. Vacuum may
143
+ reclaim a version only after the global safe point has passed it. A new feature
144
+ must define behavior for:
145
+
146
+ * read committed, repeatable read, and serializable transactions;
147
+ * concurrent update of the same row;
148
+ * unique and foreign-key conflicts;
149
+ * deadlock detection and victim rollback;
150
+ * lock wait timeouts and request cancellation; and
151
+ * commit acknowledgement followed by client disconnect.
152
+
153
+ Deadlock resolution must abort a complete victim transaction, release all of
154
+ its locks, and leave other transactions able to progress. Never fix a deadlock
155
+ by globally disabling locking or by releasing a lock without undoing the
156
+ corresponding write set.
157
+
158
+ ## 8. SQL implementation workflow
159
+
160
+ For a new statement or expression:
161
+
162
+ 1. write the syntax and semantic contract in `spec/sql`;
163
+ 2. add parser acceptance and rejection examples;
164
+ 3. add binder/type/nullability behavior;
165
+ 4. add planner and executor tests;
166
+ 5. test transaction, constraint, index, and error interactions;
167
+ 6. test the Ruby API and server protocol path; and
168
+ 7. test the ActiveRecord-generated SQL when the feature is adapter-visible.
169
+
170
+ Prefer parameter binding to string interpolation. Every expression needs
171
+ defined behavior for `NULL`, booleans, numeric coercion, text comparison,
172
+ collation, and invalid input. Every DDL operation needs idempotence or an
173
+ explicit error contract. Every optimizer rewrite must have a semantic
174
+ equivalence test against the non-optimized execution path.
175
+
176
+ The documented common SQLite-style profile is intentionally narrower than
177
+ SQLite itself. Do not label a feature “SQLite compatible” until its syntax,
178
+ results, types, error behavior, and migration behavior are tested.
179
+
180
+ ## 9. Tables, indexes, and constraints
181
+
182
+ Table mutations and index mutations are one logical operation. If an index
183
+ write fails, the statement must fail visibly and the transaction must either
184
+ roll back or retain a recoverable pending state; it must not acknowledge a row
185
+ that cannot be found by a required index.
186
+
187
+ For each index type, test empty and populated tables, duplicate keys, `NULL`,
188
+ deep splits, reopen, recovery replay, deletion/merge, and concurrent readers.
189
+ For each constraint, test direct SQL, prepared parameters, ActiveRecord
190
+ inserts/updates, rollback, and the exact error class/message contract where
191
+ callers may depend on it.
192
+
193
+ ## 10. Server, client, and wire protocol
194
+
195
+ The server is the single owner of the database directory. A client session
196
+ handles authentication, capability negotiation, request IDs, deadlines,
197
+ cancellation, result framing, and connection shutdown. Protocol changes must
198
+ be backward-compatible or carry an explicit protocol version and rejection
199
+ path.
200
+
201
+ Wire cancellation is part of correctness. A cancelled request must stop
202
+ execution at safe checkpoints, release transaction resources, and return a
203
+ definitive cancellation response. If cancellation races with commit, the
204
+ server must preserve the commit outcome and expose enough request identity for
205
+ the client to query status rather than blindly retrying.
206
+
207
+ Test malformed lengths, unknown message types, duplicate request IDs, partial
208
+ frames, client disconnects, server shutdown, timeout races, authentication
209
+ failure, TLS negotiation, and cancellation during a long scan.
210
+
211
+ ## 11. Rails and adapter work
212
+
213
+ The ActiveRecord adapter translates Rails schema and query APIs into RubyDB
214
+ SQL. Adapter code must preserve Rails expectations for quoting, bind
215
+ parameters, affected-row counts, last-insert IDs, transactions, savepoints,
216
+ schema introspection, migration versions, and exceptions.
217
+
218
+ When changing adapter behavior, run the example app and test at least:
219
+
220
+ * `where`, ordering, limits, scopes, joins, and aggregate relations;
221
+ * eager loading, nested associations, and inverse relationships;
222
+ * connection pools with multiple threads;
223
+ * schema dump/load and populated-table migrations;
224
+ * rollback, retry, and migration checksum behavior; and
225
+ * every supported Ruby/Rails combination in CI.
226
+
227
+ The adapter is not proof of complete SQLite or PostgreSQL compatibility. An
228
+ application should run its own generated-SQL and migration suite before a
229
+ cutover.
230
+
231
+ ## 12. Testing pyramid
232
+
233
+ Use the smallest test that proves the invariant:
234
+
235
+ * unit specs for parsing, encoding, types, and isolated algorithms;
236
+ * component specs for storage, WAL, indexes, transactions, and protocol;
237
+ * integration specs for engine-to-SQL and server-to-client behavior;
238
+ * process specs for crash recovery, pooling, and failover;
239
+ * workload tests for sustained concurrency and resource limits; and
240
+ * example applications for real Rails and Ruby workflows.
241
+
242
+ Useful commands include:
243
+
244
+ ```sh
245
+ bundle exec rspec
246
+ bundle exec rspec spec/sql spec/storage spec/transactions
247
+ ruby scripts/durability_drill
248
+ ruby scripts/production_soak
249
+ ruby scripts/fuzz
250
+ RUBYDB_BENCHMARK_ITERATIONS=100 ruby -Ilib benchmarks/basic_workload.rb
251
+ ```
252
+
253
+ Record the Ruby version, commit, OS, command, seed, database configuration,
254
+ and artifact paths for every non-trivial run. A green unit suite does not
255
+ replace a restore drill, multi-process soak, or deployment test.
256
+
257
+ ## 13. Debugging a failing change
258
+
259
+ Start with [Debugging RubyDB](debugging.md). Reproduce on a fresh temporary
260
+ directory, reduce the schema and SQL, rerun with the reported seed, and save
261
+ logs plus the WAL metadata. Compare embedded and server/client execution when
262
+ ownership or protocol is suspected. Use [Troubleshooting](troubleshooting.md)
263
+ for operator symptoms and evidence-preserving recovery steps.
264
+
265
+ Never “repair” a failing test by deleting WAL, disabling checksums, loosening a
266
+ constraint, or turning off synchronization. Preserve the failing directory,
267
+ copy it, and investigate the copy.
268
+
269
+ ## 14. Performance and capacity work
270
+
271
+ Benchmark one variable at a time and report latency percentiles, throughput,
272
+ concurrency, payload shape, cache size, WAL/checkpoint settings, and storage
273
+ medium. Track p50, p95, p99, error rate, lock wait time, WAL growth, checkpoint
274
+ duration, memory, file descriptors, and CPU.
275
+
276
+ Capacity limits are workload-specific. A benchmark result is not a guarantee
277
+ for a Rails application with different indexes, query shapes, or connection
278
+ pool settings. Add a regression threshold only after the benchmark is stable
279
+ across repeated runs and environments.
280
+
281
+ ## 15. Safe contribution checklist
282
+
283
+ Before opening a pull request:
284
+
285
+ 1. Explain the invariant and compatibility impact.
286
+ 2. Add focused tests, including failure paths.
287
+ 3. Run the relevant focused suite and the full suite.
288
+ 4. Run formatting/lint and `git diff --check`.
289
+ 5. Update the appropriate spec and user/operator documentation.
290
+ 6. Note migrations, format changes, recovery implications, and rollback plan.
291
+ 7. Include benchmark or soak evidence for hot-path and concurrency changes.
292
+ 8. Remove debug output, secrets, temporary data, and generated artifacts.
293
+
294
+ See [Contributing](../CONTRIBUTING.md), [testing](contributing/testing.md),
295
+ and [release process](contributing/release-process.md) for the repository
296
+ workflow.
297
+
@@ -1,16 +1,16 @@
1
- # First database
2
-
3
- ```ruby
4
- require "rubydb"
5
-
6
- path = "tmp/first-database.rdb"
7
- engine = RubyDB::Storage::Engine.new(path)
8
- engine.execute("CREATE TABLE accounts (id INTEGER PRIMARY KEY, email TEXT UNIQUE NOT NULL)")
9
- engine.execute("INSERT INTO accounts (id, email) VALUES (1, 'ada@example.test')")
10
- puts engine.execute("SELECT id, email FROM accounts").inspect
11
- engine.close
12
- ```
13
-
14
- The engine creates durable storage and WAL state under the selected path. Close
15
- the engine cleanly and keep the database, WAL, and lock files together. A single
16
- embedded path has one owner; use a server for multiple processes.
1
+ # First database
2
+
3
+ ```ruby
4
+ require "rubydb"
5
+
6
+ path = "tmp/first-database.rdb"
7
+ engine = RubyDB::Storage::Engine.new(path)
8
+ engine.execute("CREATE TABLE accounts (id INTEGER PRIMARY KEY, email TEXT UNIQUE NOT NULL)")
9
+ engine.execute("INSERT INTO accounts (id, email) VALUES (1, 'ada@example.test')")
10
+ puts engine.execute("SELECT id, email FROM accounts").inspect
11
+ engine.close
12
+ ```
13
+
14
+ The engine creates durable storage and WAL state under the selected path. Close
15
+ the engine cleanly and keep the database, WAL, and lock files together. A single
16
+ embedded path has one owner; use a server for multiple processes.
@@ -1,13 +1,13 @@
1
- # First query
2
-
3
- RubyDB executes the documented SQL subset through its lexer, parser, planner,
4
- and executor:
5
-
6
- ```ruby
7
- rows = engine.execute("SELECT id, email FROM accounts WHERE id = 1 ORDER BY id LIMIT 10")
8
- puts rows.inspect
9
- ```
10
-
11
- Use the Rails connection or client binding APIs for external values. Do not
12
- interpolate untrusted input into SQL. See [SQL compatibility](../sql/compatibility.md)
13
- for supported statements and explicit boundaries.
1
+ # First query
2
+
3
+ RubyDB executes the documented SQL subset through its lexer, parser, planner,
4
+ and executor:
5
+
6
+ ```ruby
7
+ rows = engine.execute("SELECT id, email FROM accounts WHERE id = 1 ORDER BY id LIMIT 10")
8
+ puts rows.inspect
9
+ ```
10
+
11
+ Use the Rails connection or client binding APIs for external values. Do not
12
+ interpolate untrusted input into SQL. See [SQL compatibility](../sql/compatibility.md)
13
+ for supported statements and explicit boundaries.