rubydb 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (436) hide show
  1. checksums.yaml +7 -0
  2. data/.gitattributes +20 -0
  3. data/.github/ISSUE_TEMPLATE/bug.yml +0 -0
  4. data/.github/ISSUE_TEMPLATE/feature.yml +0 -0
  5. data/.github/ISSUE_TEMPLATE/performance.yml +0 -0
  6. data/.github/ISSUE_TEMPLATE/security.yml +0 -0
  7. data/.github/PULL_REQUEST_TEMPLATE.md +15 -0
  8. data/.github/dependabot.yml +0 -0
  9. data/.github/workflows/benchmark.yml +26 -0
  10. data/.github/workflows/compatibility.yml +63 -0
  11. data/.github/workflows/fuzz.yml +33 -0
  12. data/.github/workflows/lint.yml +21 -0
  13. data/.github/workflows/operations.yml +24 -0
  14. data/.github/workflows/production-validation.yml +111 -0
  15. data/.github/workflows/release.yml +77 -0
  16. data/.github/workflows/security.yml +37 -0
  17. data/.github/workflows/test.yml +26 -0
  18. data/.github/workflows/workload.yml +58 -0
  19. data/.gitignore +43 -0
  20. data/.rspec +4 -0
  21. data/.rubocop.yml +44 -0
  22. data/.ruby-version +1 -0
  23. data/.standard.yml +14 -0
  24. data/ARCHITECTURE.md +21 -0
  25. data/CHANGELOG.md +31 -0
  26. data/CODE_OF_CONDUCT.md +13 -0
  27. data/CONTRIBUTING.md +29 -0
  28. data/GOVERNANCE.md +16 -0
  29. data/Gemfile +17 -0
  30. data/Gemfile.lock +71 -0
  31. data/LICENSE +60 -0
  32. data/README.md +162 -0
  33. data/ROADMAP.md +27 -0
  34. data/Rakefile +71 -0
  35. data/SECURITY.md +54 -0
  36. data/SUPPORT.md +14 -0
  37. data/adapters/activerecord/Gemfile +11 -0
  38. data/adapters/activerecord/README.md +9 -0
  39. data/adapters/activerecord/lib/active_record/connection_adapters/rubydb_adapter.rb +887 -0
  40. data/adapters/activerecord/rubydb-activerecord.gemspec +27 -0
  41. data/adapters/activerecord/spec/rubydb_adapter_integration_spec.rb +151 -0
  42. data/adapters/ruby/README.md +18 -0
  43. data/adapters/sequel/README.md +11 -0
  44. data/bin/rubydb +19 -0
  45. data/bin/rubydb-server +126 -0
  46. data/config/development.yml +0 -0
  47. data/config/monitoring/prometheus-alerts.yml +39 -0
  48. data/config/production.yml +36 -0
  49. data/config/test.yml +0 -0
  50. data/docs/README.md +71 -0
  51. data/docs/architecture/concurrency.md +14 -0
  52. data/docs/architecture/current-state.md +125 -0
  53. data/docs/architecture/execution-engine.md +25 -0
  54. data/docs/architecture/indexes.md +19 -0
  55. data/docs/architecture/mvcc.md +19 -0
  56. data/docs/architecture/overview.md +13 -0
  57. data/docs/architecture/pages.md +11 -0
  58. data/docs/architecture/production-roadmap.md +82 -0
  59. data/docs/architecture/query-planner.md +20 -0
  60. data/docs/architecture/recovery.md +18 -0
  61. data/docs/architecture/sql-engine.md +12 -0
  62. data/docs/architecture/storage-engine.md +14 -0
  63. data/docs/architecture/transactions.md +10 -0
  64. data/docs/architecture/wal.md +28 -0
  65. data/docs/cli-cheatsheet.md +98 -0
  66. data/docs/cli.md +275 -0
  67. data/docs/contributing/architecture.md +9 -0
  68. data/docs/contributing/benchmarking.md +14 -0
  69. data/docs/contributing/development.md +16 -0
  70. data/docs/contributing/release-process.md +49 -0
  71. data/docs/contributing/testing.md +16 -0
  72. data/docs/debugging.md +229 -0
  73. data/docs/developer/branching.md +10 -0
  74. data/docs/developer/database-diff.md +10 -0
  75. data/docs/developer/local-development.md +17 -0
  76. data/docs/developer/snapshots.md +9 -0
  77. data/docs/developer/temporal-data.md +10 -0
  78. data/docs/developer-guide.md +297 -0
  79. data/docs/getting-started/first-database.md +16 -0
  80. data/docs/getting-started/first-query.md +13 -0
  81. data/docs/getting-started/installation.md +19 -0
  82. data/docs/getting-started/local-to-production.md +300 -0
  83. data/docs/getting-started/quickstart.md +17 -0
  84. data/docs/getting-started/rails.md +16 -0
  85. data/docs/hardening_backlog.md +93 -0
  86. data/docs/lessons-learned.md +112 -0
  87. data/docs/operations/backups.md +33 -0
  88. data/docs/operations/disaster-recovery.md +31 -0
  89. data/docs/operations/failover.md +30 -0
  90. data/docs/operations/monitoring.md +25 -0
  91. data/docs/operations/production-guide.md +295 -0
  92. data/docs/operations/production-runbook.md +45 -0
  93. data/docs/operations/replication.md +33 -0
  94. data/docs/operations/restore.md +6 -0
  95. data/docs/operations/runbook.md +34 -0
  96. data/docs/operations/upgrades.md +14 -0
  97. data/docs/operations/workload-testing.md +17 -0
  98. data/docs/production-readiness.md +118 -0
  99. data/docs/production_validation.md +150 -0
  100. data/docs/rails/active-record.md +11 -0
  101. data/docs/rails/compatibility-guide.md +90 -0
  102. data/docs/rails/database-yml.md +92 -0
  103. data/docs/rails/installation.md +17 -0
  104. data/docs/rails/migrations.md +17 -0
  105. data/docs/rails/production.md +82 -0
  106. data/docs/rails/troubleshooting.md +18 -0
  107. data/docs/release.md +59 -0
  108. data/docs/server/architecture.md +10 -0
  109. data/docs/server/authentication.md +10 -0
  110. data/docs/server/configuration.md +16 -0
  111. data/docs/server/connection-pooling.md +10 -0
  112. data/docs/server/deployment.md +10 -0
  113. data/docs/server/protocol.md +12 -0
  114. data/docs/sql/compatibility-guide.md +82 -0
  115. data/docs/sql/compatibility.md +39 -0
  116. data/docs/sql/data-types.md +10 -0
  117. data/docs/sql/functions.md +9 -0
  118. data/docs/sql/joins.md +9 -0
  119. data/docs/sql/operators.md +9 -0
  120. data/docs/sql/sqlite-compatibility.md +21 -0
  121. data/docs/sql/syntax.md +10 -0
  122. data/docs/sql/transactions.md +10 -0
  123. data/docs/troubleshooting.md +244 -0
  124. data/exe/rubydb +23 -0
  125. data/lib/rubydb/backup/archive.rb +334 -0
  126. data/lib/rubydb/backup/backup.rb +401 -0
  127. data/lib/rubydb/backup/incremental.rb +353 -0
  128. data/lib/rubydb/backup/restore.rb +290 -0
  129. data/lib/rubydb/backup/snapshot.rb +267 -0
  130. data/lib/rubydb/backup/verification.rb +279 -0
  131. data/lib/rubydb/branching/branch.rb +181 -0
  132. data/lib/rubydb/branching/branch_manager.rb +311 -0
  133. data/lib/rubydb/branching/branch_metadata.rb +140 -0
  134. data/lib/rubydb/branching/checkout.rb +166 -0
  135. data/lib/rubydb/branching/copy_on_write.rb +272 -0
  136. data/lib/rubydb/branching/diff.rb +138 -0
  137. data/lib/rubydb/branching/merge.rb +285 -0
  138. data/lib/rubydb/build_info.rb +15 -0
  139. data/lib/rubydb/catalog/catalog.rb +391 -0
  140. data/lib/rubydb/catalog/column.rb +112 -0
  141. data/lib/rubydb/catalog/constraint.rb +180 -0
  142. data/lib/rubydb/catalog/database.rb +184 -0
  143. data/lib/rubydb/catalog/index.rb +97 -0
  144. data/lib/rubydb/catalog/schema.rb +103 -0
  145. data/lib/rubydb/catalog/sequence.rb +90 -0
  146. data/lib/rubydb/catalog/system_catalog.rb +698 -0
  147. data/lib/rubydb/catalog/table.rb +178 -0
  148. data/lib/rubydb/catalog/trigger.rb +102 -0
  149. data/lib/rubydb/catalog/view.rb +66 -0
  150. data/lib/rubydb/cli/application.rb +163 -0
  151. data/lib/rubydb/cli/commands/backup.rb +81 -0
  152. data/lib/rubydb/cli/commands/branch.rb +72 -0
  153. data/lib/rubydb/cli/commands/checkout.rb +54 -0
  154. data/lib/rubydb/cli/commands/create.rb +58 -0
  155. data/lib/rubydb/cli/commands/diff.rb +77 -0
  156. data/lib/rubydb/cli/commands/doctor.rb +74 -0
  157. data/lib/rubydb/cli/commands/drop.rb +57 -0
  158. data/lib/rubydb/cli/commands/init.rb +102 -0
  159. data/lib/rubydb/cli/commands/inspect.rb +95 -0
  160. data/lib/rubydb/cli/commands/merge.rb +63 -0
  161. data/lib/rubydb/cli/commands/migrate.rb +62 -0
  162. data/lib/rubydb/cli/commands/restart.rb +39 -0
  163. data/lib/rubydb/cli/commands/restore.rb +121 -0
  164. data/lib/rubydb/cli/commands/shell.rb +365 -0
  165. data/lib/rubydb/cli/commands/snapshot.rb +79 -0
  166. data/lib/rubydb/cli/commands/start.rb +82 -0
  167. data/lib/rubydb/cli/commands/status.rb +92 -0
  168. data/lib/rubydb/cli/commands/stop.rb +47 -0
  169. data/lib/rubydb/cli/commands/vacuum.rb +58 -0
  170. data/lib/rubydb/cli/formatter.rb +221 -0
  171. data/lib/rubydb/cli/output.rb +168 -0
  172. data/lib/rubydb/client/client.rb +304 -0
  173. data/lib/rubydb/client/connection.rb +415 -0
  174. data/lib/rubydb/client/connection_pool.rb +168 -0
  175. data/lib/rubydb/client/connection_url.rb +96 -0
  176. data/lib/rubydb/client/prepared_statement.rb +60 -0
  177. data/lib/rubydb/client/result.rb +123 -0
  178. data/lib/rubydb/client/statement.rb +52 -0
  179. data/lib/rubydb/client/transaction.rb +130 -0
  180. data/lib/rubydb/concurrency/concurrency.rb +19 -0
  181. data/lib/rubydb/concurrency/deadlock_detector.rb +150 -0
  182. data/lib/rubydb/concurrency/latch.rb +101 -0
  183. data/lib/rubydb/concurrency/lock_graph.rb +165 -0
  184. data/lib/rubydb/concurrency/mutex.rb +183 -0
  185. data/lib/rubydb/concurrency/rw_lock.rb +180 -0
  186. data/lib/rubydb/concurrency/scheduler.rb +250 -0
  187. data/lib/rubydb/concurrency/worker_pool.rb +143 -0
  188. data/lib/rubydb/configuration/config.rb +170 -0
  189. data/lib/rubydb/configuration/defaults.rb +179 -0
  190. data/lib/rubydb/configuration/environment.rb +152 -0
  191. data/lib/rubydb/configuration/parser.rb +185 -0
  192. data/lib/rubydb/configuration/validation.rb +221 -0
  193. data/lib/rubydb/constants.rb +74 -0
  194. data/lib/rubydb/constraints/check.rb +181 -0
  195. data/lib/rubydb/constraints/constraint.rb +101 -0
  196. data/lib/rubydb/constraints/foreign_key.rb +130 -0
  197. data/lib/rubydb/constraints/not_null.rb +64 -0
  198. data/lib/rubydb/constraints/primary_key.rb +99 -0
  199. data/lib/rubydb/constraints/unique.rb +108 -0
  200. data/lib/rubydb/constraints/validator.rb +350 -0
  201. data/lib/rubydb/errors/authentication_error.rb +10 -0
  202. data/lib/rubydb/errors/authorization_error.rb +23 -0
  203. data/lib/rubydb/errors/client_error.rb +10 -0
  204. data/lib/rubydb/errors/configuration_error.rb +10 -0
  205. data/lib/rubydb/errors/connection_error.rb +10 -0
  206. data/lib/rubydb/errors/constraint_error.rb +23 -0
  207. data/lib/rubydb/errors/corruption_error.rb +10 -0
  208. data/lib/rubydb/errors/database_error.rb +10 -0
  209. data/lib/rubydb/errors/error.rb +20 -0
  210. data/lib/rubydb/errors/execution_error.rb +10 -0
  211. data/lib/rubydb/errors/parser_error.rb +10 -0
  212. data/lib/rubydb/errors/recovery_error.rb +10 -0
  213. data/lib/rubydb/errors/replication_error.rb +10 -0
  214. data/lib/rubydb/errors/server_error.rb +6 -0
  215. data/lib/rubydb/errors/storage_error.rb +10 -0
  216. data/lib/rubydb/errors/transaction_error.rb +10 -0
  217. data/lib/rubydb/execution/aggregate_executor.rb +138 -0
  218. data/lib/rubydb/execution/delete_executor.rb +112 -0
  219. data/lib/rubydb/execution/distinct_executor.rb +135 -0
  220. data/lib/rubydb/execution/executor.rb +1192 -0
  221. data/lib/rubydb/execution/expression.rb +193 -0
  222. data/lib/rubydb/execution/index_scan.rb +142 -0
  223. data/lib/rubydb/execution/insert_executor.rb +217 -0
  224. data/lib/rubydb/execution/join_executor.rb +249 -0
  225. data/lib/rubydb/execution/limit_executor.rb +85 -0
  226. data/lib/rubydb/execution/optimizer.rb +215 -0
  227. data/lib/rubydb/execution/plan.rb +361 -0
  228. data/lib/rubydb/execution/planner.rb +546 -0
  229. data/lib/rubydb/execution/predicate.rb +235 -0
  230. data/lib/rubydb/execution/scan.rb +49 -0
  231. data/lib/rubydb/execution/sequential_scan.rb +63 -0
  232. data/lib/rubydb/execution/sort_executor.rb +185 -0
  233. data/lib/rubydb/execution/update_executor.rb +162 -0
  234. data/lib/rubydb/functions/aggregate.rb +70 -0
  235. data/lib/rubydb/functions/date_functions.rb +278 -0
  236. data/lib/rubydb/functions/function.rb +85 -0
  237. data/lib/rubydb/functions/json_functions.rb +215 -0
  238. data/lib/rubydb/functions/numeric_functions.rb +346 -0
  239. data/lib/rubydb/functions/scalar.rb +52 -0
  240. data/lib/rubydb/functions/string_functions.rb +383 -0
  241. data/lib/rubydb/functions/system_functions.rb +246 -0
  242. data/lib/rubydb/history/as_of.rb +238 -0
  243. data/lib/rubydb/history/change.rb +105 -0
  244. data/lib/rubydb/history/history.rb +131 -0
  245. data/lib/rubydb/history/history_manager.rb +229 -0
  246. data/lib/rubydb/history/temporal_query.rb +202 -0
  247. data/lib/rubydb/history/timeline.rb +144 -0
  248. data/lib/rubydb/indexes/btree.rb +186 -0
  249. data/lib/rubydb/indexes/btree_cursor.rb +258 -0
  250. data/lib/rubydb/indexes/btree_node.rb +385 -0
  251. data/lib/rubydb/indexes/hash_index.rb +150 -0
  252. data/lib/rubydb/indexes/index.rb +71 -0
  253. data/lib/rubydb/indexes/index_manager.rb +406 -0
  254. data/lib/rubydb/indexes/index_scan.rb +470 -0
  255. data/lib/rubydb/migrations/migration.rb +254 -0
  256. data/lib/rubydb/migrations/migration_lock.rb +146 -0
  257. data/lib/rubydb/migrations/migration_manager.rb +176 -0
  258. data/lib/rubydb/migrations/migration_version.rb +71 -0
  259. data/lib/rubydb/migrations/schema_diff.rb +211 -0
  260. data/lib/rubydb/migrations/schema_version.rb +64 -0
  261. data/lib/rubydb/monitoring/events.rb +160 -0
  262. data/lib/rubydb/monitoring/health.rb +222 -0
  263. data/lib/rubydb/monitoring/logger.rb +193 -0
  264. data/lib/rubydb/monitoring/metrics.rb +359 -0
  265. data/lib/rubydb/monitoring/performance.rb +176 -0
  266. data/lib/rubydb/monitoring/statistics.rb +170 -0
  267. data/lib/rubydb/mvcc/garbage_collector.rb +199 -0
  268. data/lib/rubydb/mvcc/mvcc.rb +16 -0
  269. data/lib/rubydb/mvcc/snapshot.rb +147 -0
  270. data/lib/rubydb/mvcc/vacuum.rb +180 -0
  271. data/lib/rubydb/mvcc/version.rb +106 -0
  272. data/lib/rubydb/mvcc/version_store.rb +398 -0
  273. data/lib/rubydb/mvcc/visibility.rb +109 -0
  274. data/lib/rubydb/protocol/capabilities.rb +125 -0
  275. data/lib/rubydb/protocol/decoder.rb +145 -0
  276. data/lib/rubydb/protocol/encoder.rb +136 -0
  277. data/lib/rubydb/protocol/handshake.rb +305 -0
  278. data/lib/rubydb/protocol/message.rb +121 -0
  279. data/lib/rubydb/protocol/protocol.rb +277 -0
  280. data/lib/rubydb/protocol/version.rb +54 -0
  281. data/lib/rubydb/rails/adapter.rb +239 -0
  282. data/lib/rubydb/rails/connection.rb +314 -0
  283. data/lib/rubydb/rails/database_statements.rb +122 -0
  284. data/lib/rubydb/rails/migration.rb +131 -0
  285. data/lib/rubydb/rails/quoting.rb +109 -0
  286. data/lib/rubydb/rails/result.rb +120 -0
  287. data/lib/rubydb/rails/schema_statements.rb +339 -0
  288. data/lib/rubydb/rails/transaction.rb +105 -0
  289. data/lib/rubydb/rails/type.rb +126 -0
  290. data/lib/rubydb/recovery/checkpoint.rb +257 -0
  291. data/lib/rubydb/recovery/consistency.rb +467 -0
  292. data/lib/rubydb/recovery/corruption_detector.rb +5 -0
  293. data/lib/rubydb/recovery/crash_recovery.rb +387 -0
  294. data/lib/rubydb/recovery/recovery_manager.rb +206 -0
  295. data/lib/rubydb/recovery/redo.rb +237 -0
  296. data/lib/rubydb/recovery/undo.rb +206 -0
  297. data/lib/rubydb/replication/failover.rb +5 -0
  298. data/lib/rubydb/replication/fencing.rb +63 -0
  299. data/lib/rubydb/replication/primary.rb +450 -0
  300. data/lib/rubydb/replication/replica.rb +384 -0
  301. data/lib/rubydb/replication/replication_log.rb +200 -0
  302. data/lib/rubydb/replication/replication_manager.rb +308 -0
  303. data/lib/rubydb/replication/replication_slot.rb +295 -0
  304. data/lib/rubydb/replication/replication_stream.rb +201 -0
  305. data/lib/rubydb/rubydb.rb +560 -0
  306. data/lib/rubydb/security/access_control.rb +254 -0
  307. data/lib/rubydb/security/audit_log.rb +213 -0
  308. data/lib/rubydb/security/authentication.rb +302 -0
  309. data/lib/rubydb/security/authorization.rb +282 -0
  310. data/lib/rubydb/security/credentials.rb +196 -0
  311. data/lib/rubydb/security/password.rb +215 -0
  312. data/lib/rubydb/security/permissions.rb +74 -0
  313. data/lib/rubydb/security/role.rb +101 -0
  314. data/lib/rubydb/security/user.rb +86 -0
  315. data/lib/rubydb/server/connection.rb +366 -0
  316. data/lib/rubydb/server/connection_pool.rb +193 -0
  317. data/lib/rubydb/server/lifecycle.rb +228 -0
  318. data/lib/rubydb/server/listener.rb +136 -0
  319. data/lib/rubydb/server/request_handler.rb +276 -0
  320. data/lib/rubydb/server/server.rb +364 -0
  321. data/lib/rubydb/server/session.rb +369 -0
  322. data/lib/rubydb/server/worker.rb +210 -0
  323. data/lib/rubydb/server/worker_pool.rb +168 -0
  324. data/lib/rubydb/sql/ast/alter_table.rb +169 -0
  325. data/lib/rubydb/sql/ast/begin_transaction.rb +47 -0
  326. data/lib/rubydb/sql/ast/commit.rb +37 -0
  327. data/lib/rubydb/sql/ast/constraint.rb +83 -0
  328. data/lib/rubydb/sql/ast/create_database.rb +41 -0
  329. data/lib/rubydb/sql/ast/create_index.rb +61 -0
  330. data/lib/rubydb/sql/ast/create_schema.rb +52 -0
  331. data/lib/rubydb/sql/ast/create_table.rb +187 -0
  332. data/lib/rubydb/sql/ast/delete.rb +54 -0
  333. data/lib/rubydb/sql/ast/drop_database.rb +41 -0
  334. data/lib/rubydb/sql/ast/drop_index.rb +41 -0
  335. data/lib/rubydb/sql/ast/drop_schema.rb +49 -0
  336. data/lib/rubydb/sql/ast/drop_table.rb +49 -0
  337. data/lib/rubydb/sql/ast/explain.rb +64 -0
  338. data/lib/rubydb/sql/ast/expression.rb +604 -0
  339. data/lib/rubydb/sql/ast/insert.rb +82 -0
  340. data/lib/rubydb/sql/ast/node.rb +42 -0
  341. data/lib/rubydb/sql/ast/rollback.rb +63 -0
  342. data/lib/rubydb/sql/ast/savepoint.rb +59 -0
  343. data/lib/rubydb/sql/ast/select.rb +88 -0
  344. data/lib/rubydb/sql/ast/set_operation.rb +20 -0
  345. data/lib/rubydb/sql/ast/trigger.rb +29 -0
  346. data/lib/rubydb/sql/ast/update.rb +88 -0
  347. data/lib/rubydb/sql/ast/vacuum.rb +19 -0
  348. data/lib/rubydb/sql/ast/view.rb +32 -0
  349. data/lib/rubydb/sql/ast/with.rb +32 -0
  350. data/lib/rubydb/sql/grammar.rb +86 -0
  351. data/lib/rubydb/sql/keywords.rb +156 -0
  352. data/lib/rubydb/sql/lexer.rb +214 -0
  353. data/lib/rubydb/sql/operators.rb +100 -0
  354. data/lib/rubydb/sql/parser.rb +1185 -0
  355. data/lib/rubydb/sql/planner/analyzer.rb +302 -0
  356. data/lib/rubydb/sql/planner/binder.rb +550 -0
  357. data/lib/rubydb/sql/planner/type_checker.rb +431 -0
  358. data/lib/rubydb/sql/token.rb +210 -0
  359. data/lib/rubydb/storage/buffer_frame.rb +44 -0
  360. data/lib/rubydb/storage/buffer_pool.rb +155 -0
  361. data/lib/rubydb/storage/database_lock.rb +74 -0
  362. data/lib/rubydb/storage/deserializer.rb +342 -0
  363. data/lib/rubydb/storage/engine.rb +2338 -0
  364. data/lib/rubydb/storage/file_manager.rb +187 -0
  365. data/lib/rubydb/storage/free_space_map.rb +81 -0
  366. data/lib/rubydb/storage/page.rb +94 -0
  367. data/lib/rubydb/storage/page_allocator.rb +855 -0
  368. data/lib/rubydb/storage/page_header.rb +67 -0
  369. data/lib/rubydb/storage/page_manager.rb +131 -0
  370. data/lib/rubydb/storage/record.rb +58 -0
  371. data/lib/rubydb/storage/row.rb +78 -0
  372. data/lib/rubydb/storage/serializer.rb +51 -0
  373. data/lib/rubydb/storage/storage_layout.rb +151 -0
  374. data/lib/rubydb/storage/storage_manager.rb +114 -0
  375. data/lib/rubydb/storage/tuple.rb +461 -0
  376. data/lib/rubydb/storage/visibility_map.rb +984 -0
  377. data/lib/rubydb/transactions/commit_manager.rb +220 -0
  378. data/lib/rubydb/transactions/isolation.rb +98 -0
  379. data/lib/rubydb/transactions/lock.rb +76 -0
  380. data/lib/rubydb/transactions/lock_manager.rb +362 -0
  381. data/lib/rubydb/transactions/savepoint.rb +143 -0
  382. data/lib/rubydb/transactions/transaction.rb +215 -0
  383. data/lib/rubydb/transactions/transaction_id.rb +84 -0
  384. data/lib/rubydb/transactions/transaction_log.rb +257 -0
  385. data/lib/rubydb/transactions/transaction_manager.rb +435 -0
  386. data/lib/rubydb/types/bigint.rb +36 -0
  387. data/lib/rubydb/types/blob.rb +37 -0
  388. data/lib/rubydb/types/boolean.rb +34 -0
  389. data/lib/rubydb/types/date.rb +39 -0
  390. data/lib/rubydb/types/decimal.rb +48 -0
  391. data/lib/rubydb/types/float.rb +34 -0
  392. data/lib/rubydb/types/integer.rb +36 -0
  393. data/lib/rubydb/types/json.rb +41 -0
  394. data/lib/rubydb/types/null.rb +34 -0
  395. data/lib/rubydb/types/smallint.rb +36 -0
  396. data/lib/rubydb/types/text.rb +37 -0
  397. data/lib/rubydb/types/time.rb +46 -0
  398. data/lib/rubydb/types/timestamp.rb +39 -0
  399. data/lib/rubydb/types/type.rb +119 -0
  400. data/lib/rubydb/types/uuid.rb +47 -0
  401. data/lib/rubydb/types/varchar.rb +37 -0
  402. data/lib/rubydb/version.rb +33 -0
  403. data/lib/rubydb/wal/archive.rb +193 -0
  404. data/lib/rubydb/wal/checkpoint.rb +183 -0
  405. data/lib/rubydb/wal/lsn.rb +94 -0
  406. data/lib/rubydb/wal/reader.rb +260 -0
  407. data/lib/rubydb/wal/record.rb +105 -0
  408. data/lib/rubydb/wal/segment.rb +193 -0
  409. data/lib/rubydb/wal/wal.rb +452 -0
  410. data/lib/rubydb/wal/writer.rb +236 -0
  411. data/lib/rubydb.rb +7 -0
  412. data/packaging/docker/Dockerfile +36 -0
  413. data/packaging/docker/docker-compose.failover.yml +43 -0
  414. data/packaging/docker/entrypoint.sh +15 -0
  415. data/packaging/homebrew/rubydb.rb +19 -0
  416. data/packaging/systemd/rubydb.service +22 -0
  417. data/rubydb.gemspec +57 -0
  418. data/scripts/benchmark +7 -0
  419. data/scripts/durability_drill +37 -0
  420. data/scripts/format +0 -0
  421. data/scripts/fuzz +63 -0
  422. data/scripts/lint +0 -0
  423. data/scripts/release +50 -0
  424. data/scripts/release_check +43 -0
  425. data/scripts/replication_failover_drill +250 -0
  426. data/scripts/replication_network_failover_drill +255 -0
  427. data/scripts/restore_drill +45 -0
  428. data/scripts/security +0 -0
  429. data/scripts/setup +0 -0
  430. data/scripts/test +0 -0
  431. data/scripts/test-chaos +0 -0
  432. data/scripts/test-crash +0 -0
  433. data/scripts/test-integration +0 -0
  434. data/scripts/test-unit +0 -0
  435. data/visibility_map.json +1 -0
  436. metadata +538 -0
@@ -0,0 +1,49 @@
1
+ # Releasing RubyDB to RubyGems
2
+
3
+ 1. Update `CHANGELOG.md`, bump the semantic version, and confirm the Ruby/Rails support matrix for the release. The version must have a top-level `## <version>` changelog entry; the release preflight fails closed when it is missing.
4
+ 2. Run the complete verification suite and workload test. The release workflow independently builds and validates the gem on a version tag.
5
+ 3. Create a RubyGems API key with the minimum scope needed to push this gem. Store it as the `RUBYGEMS_API_KEY` GitHub Actions secret or in RubyGems' protected credentials file; never commit it.
6
+ 4. Create and push an annotated `v<version>` tag. The compatibility workflow
7
+ must pass its SimpleCov line-coverage gate, and the release workflow
8
+ publishes only when that tag's version matches `RubyDB::VERSION` and the
9
+ secret is available.
10
+ 5. To build, verify, and publish manually:
11
+
12
+ ```sh
13
+ RUBYDB_PUBLISH=1 GEM_HOST_API_KEY=<RubyGems API key> ruby scripts/release
14
+ ```
15
+
16
+ Without `RUBYDB_PUBLISH=1`, `ruby scripts/release` only rebuilds and verifies the gem/checksum. Manual publication must also set `RUBYDB_RELEASE_VERSION=<version>`; tagged CI obtains that identity from `GITHUB_REF_NAME`.
17
+
18
+ The tag workflow also creates a signed GitHub build-provenance attestation for
19
+ the exact gem artifact. Verify that attestation in the repository's Actions or
20
+ Releases UI before distributing the package; the SHA-512 file remains available
21
+ for an independent byte-for-byte check.
22
+
23
+ For the tag release workflow, provision base64-encoded
24
+ `RUBYDB_GEM_SIGNING_KEY_B64` and `RUBYDB_GEM_CERT_B64` repository secrets. The
25
+ release job materializes them only on the ephemeral runner and passes the
26
+ protected paths to `gem build`; it never stores them in the repository. GitHub
27
+ release notes are generated from the tag history after publication, so review
28
+ the generated release before announcing it.
29
+
30
+ The tag workflow fails closed when the signing secrets or RubyGems publication
31
+ key are missing. Local unsigned builds remain available through `rake build`
32
+ and are not publication artifacts.
33
+
34
+ Pull requests also run the supported Ruby 3.3/3.4 matrix on Linux, macOS, and
35
+ Windows, plus the ActiveRecord adapter suite on Rails 7.1, 7.2, and 8.0. A scheduled bounded fuzz job runs
36
+ the SQL parser, WAL, storage, transaction, and query-engine fuzzers. Increase
37
+ `RUBYDB_FUZZ_ITERATIONS` locally when investigating a failure, retaining the
38
+ reported `RUBYDB_FUZZ_SEED` for reproduction.
39
+
40
+ The scheduled operations workflow runs `ruby scripts/restore_drill`, which
41
+ creates a live backup, verifies its manifest/checksums, restores it into a
42
+ separate directory, and reopens the restored database before succeeding.
43
+
44
+ After publication, install the exact released version in a clean environment and run a smoke test:
45
+
46
+ ```sh
47
+ gem install rubydb --version 0.1.0
48
+ ruby -e "require 'rubydb'; puts RubyDB::VERSION"
49
+ ```
@@ -0,0 +1,16 @@
1
+ # Testing guide
2
+
3
+ The test layers are complementary:
4
+
5
+ - unit specs validate isolated algorithms and invariants;
6
+ - integration specs run the real parser, planner, executor, storage, server,
7
+ client, adapter, or replication path;
8
+ - chaos/fault specs inject write, crash, corruption, and network failures;
9
+ - workload scripts measure concurrency, latency, cancellation, capacity, and
10
+ durable reopen behavior;
11
+ - CI covers supported Ruby/Rails/OS combinations, security scans, fuzzing, and
12
+ release preflight.
13
+
14
+ Run `bundle exec rspec` for the complete local gate. Preserve the random seed
15
+ when reproducing failures. A passing local suite does not replace hosted
16
+ multi-host, physical-filesystem, or independent security validation.
data/docs/debugging.md ADDED
@@ -0,0 +1,229 @@
1
+ # Debugging RubyDB
2
+
3
+ This playbook is for finding defects without destroying the evidence needed
4
+ to recover data. Use it in development and staging first. Production
5
+ diagnostics must follow the application’s privacy, access, and change-control
6
+ policies.
7
+
8
+ ## Debugging principles
9
+
10
+ 1. Reproduce on a copy or a new temporary directory.
11
+ 2. Identify the layer: API, session, protocol, parser, planner, executor,
12
+ transaction, storage, WAL, filesystem, or deployment.
13
+ 3. Reduce the input while preserving the failure.
14
+ 4. Record versions, configuration names, seed, timing, and request IDs.
15
+ 5. Add a regression test before changing behavior.
16
+
17
+ A database that returns an error is often safer than one that silently repairs,
18
+ retries, or acknowledges unknown state. Keep fail-closed behavior intact while
19
+ diagnosing.
20
+
21
+ ## Reproducible diagnostic baseline
22
+
23
+ ```sh
24
+ git rev-parse HEAD
25
+ ruby -v
26
+ bundle exec ruby -Ilib exe/rubydb --version
27
+ bundle exec rspec --format progress
28
+ git diff --check
29
+ ```
30
+
31
+ Use a fresh directory for each reproduction. Save the command and, for random
32
+ or concurrent tests, the seed:
33
+
34
+ ```sh
35
+ RUBYDB_FUZZ_SEED=12345 RUBYDB_FUZZ_ITERATIONS=1000 ruby scripts/fuzz
36
+ bundle exec rspec spec/path/to/failing_spec.rb:42 --format doc
37
+ ```
38
+
39
+ ## CLI inspection
40
+
41
+ The CLI is the first diagnostic interface:
42
+
43
+ ```sh
44
+ bundle exec ruby -Ilib exe/rubydb status --config config/rubydb.yml
45
+ bundle exec ruby -Ilib exe/rubydb doctor --config config/rubydb.yml
46
+ bundle exec ruby -Ilib exe/rubydb inspect --config config/rubydb.yml
47
+ bundle exec ruby -Ilib exe/rubydb shell --config config/rubydb.yml
48
+ ```
49
+
50
+ Use `status` for ownership, lifecycle, and health; `doctor` for non-destructive
51
+ checks and safe repairs; and `inspect` for metadata, schema, WAL, and storage
52
+ facts. Capture JSON output when the command supports it so an incident can be
53
+ compared over time. Read [CLI guide](cli.md) before using maintenance commands.
54
+
55
+ ## Logging and safe verbosity
56
+
57
+ Use the normal structured logs first. Increase verbosity only for a bounded
58
+ reproduction. `--verbose` is useful for CLI execution, while `RUBYDB_DEBUG=1`
59
+ enables development backtraces in command wrappers. Debug logs can contain SQL
60
+ shapes, paths, identifiers, and timing; they must be access-controlled and
61
+ sanitized before sharing.
62
+
63
+ When adding logs, include stable fields such as request ID, transaction ID,
64
+ connection ID, LSN, operation, duration, and outcome. Never log passwords,
65
+ SCRAM secrets, bearer tokens, private keys, or unredacted customer values.
66
+
67
+ ## Layer isolation
68
+
69
+ ### API versus engine
70
+
71
+ Run the same operation through the direct Ruby API and the server/client path.
72
+ If only server mode fails, inspect protocol framing, authentication, session
73
+ state, serialization, and timeout handling. If both fail, reduce to engine SQL
74
+ and storage.
75
+
76
+ ### Parser versus executor
77
+
78
+ First parse/bind a statement without committing data. Then execute it against a
79
+ minimal schema. A parser failure should identify token position and expected
80
+ construct. A binder failure should identify parameter count/type context. An
81
+ executor failure should preserve transaction rollback and identify the table,
82
+ index, or constraint involved.
83
+
84
+ ### Planner versus semantics
85
+
86
+ Compare an optimized plan with a simple scan where possible. Verify row IDs,
87
+ duplicates, `NULL`, ordering, grouping, and snapshot visibility. Optimizer
88
+ changes require equivalence tests, not only performance numbers.
89
+
90
+ ### Storage versus filesystem
91
+
92
+ Test the storage operation with a real temporary filesystem first. Then inject
93
+ failures at write, flush, rename, allocation, and close boundaries. A Ruby
94
+ exception around a filesystem call is not equivalent to a process termination
95
+ after the filesystem accepted the write.
96
+
97
+ ## Transaction and deadlock diagnosis
98
+
99
+ Capture transaction lifecycle events: begin, snapshot, lock wait, lock grant,
100
+ savepoint, statement error, rollback, commit request, durable commit, and
101
+ connection close. For a deadlock, draw a wait-for graph from transaction IDs
102
+ and locks. Verify the selected victim’s before-images were applied and that
103
+ all locks and snapshots were released.
104
+
105
+ For a timeout or cancellation, answer three questions:
106
+
107
+ 1. Did the server stop executing the request?
108
+ 2. Did the transaction commit, roll back, or remain unknown?
109
+ 3. Were connection, lock, snapshot, and temporary resources released?
110
+
111
+ Do not retry a write with an unknown outcome unless it is idempotent or the
112
+ application can query the request/transaction outcome.
113
+
114
+ ## WAL and recovery diagnosis
115
+
116
+ Preserve the complete database directory, WAL, metadata, and service logs.
117
+ Record file sizes, modification times, checksums, checkpoint LSN, last
118
+ acknowledged LSN, and the process termination reason. Run inspection and
119
+ restore against a copy. Compare:
120
+
121
+ ```text
122
+ checkpoint LSN <= durable WAL end LSN
123
+ last acknowledged commit <= durable WAL end LSN
124
+ restored checksum == manifest checksum
125
+ replayed transaction set == committed transaction set
126
+ ```
127
+
128
+ For corruption, stop writes and escalate rather than repeatedly reopening the
129
+ original. For a compaction issue, compare row counts, indexes, checksums, and
130
+ reopen behavior before and after compaction on a copy.
131
+
132
+ ## Wire and protocol debugging
133
+
134
+ Use a local test endpoint and sanitized packet/frame logging. Validate one
135
+ request at a time: handshake, authentication, capability negotiation, query,
136
+ result frames, cancellation, and close. Check frame length, request ID,
137
+ sequence, status, and error payload. Test partial reads and writes because a
138
+ single `read` or `write` is not guaranteed to transfer a complete frame.
139
+
140
+ For an in-flight cancellation race, log the request ID and server state at
141
+ cancel receipt, executor stop, transaction decision, and response emission.
142
+ The client must distinguish cancellation accepted, cancellation too late, and
143
+ unknown connection loss.
144
+
145
+ ## Rails debugging
146
+
147
+ Enable Rails SQL logging in a non-production reproduction and redact bind
148
+ values before sharing. Compare the generated SQL with a direct RubyDB query.
149
+ For adapter bugs, create the smallest model and migration that shows the
150
+ problem, then test both a fresh schema and a populated table. Inspect:
151
+
152
+ * quoting and bind parameter order;
153
+ * transaction/savepoint boundaries;
154
+ * affected rows and last-insert ID;
155
+ * schema introspection and default values;
156
+ * pool checkout/checkin and leaked transactions; and
157
+ * exception class and retry behavior.
158
+
159
+ Use the Rails example under `examples/rails_app` as a smoke harness before
160
+ reproducing inside a large application.
161
+
162
+ ## Ruby-level tools
163
+
164
+ For a focused local reproduction, Ruby’s standard tools are usually enough:
165
+
166
+ ```sh
167
+ RUBYOPT="-d" bundle exec rspec spec/path/to/failing_spec.rb
168
+ bundle exec ruby -w -Ilib path/to/reproduction.rb
169
+ ```
170
+
171
+ `TracePoint` can observe method calls and exceptions without modifying the
172
+ engine. Use it only in a short-lived reproduction because tracing changes
173
+ timing and can invalidate concurrency conclusions. Capture thread backtraces
174
+ when a process appears hung, and include the thread roles (acceptor, worker,
175
+ checkpoint, replication, application).
176
+
177
+ ## Performance debugging
178
+
179
+ Separate CPU, lock, I/O, and queue time. Record p50/p95/p99 latency, throughput,
180
+ errors, WAL growth, checkpoint duration, memory, file descriptors, and active
181
+ transactions. Change one variable per run and repeat enough times to expose
182
+ variance. A faster benchmark with weaker durability is not an equivalent
183
+ optimization.
184
+
185
+ Use:
186
+
187
+ ```sh
188
+ RUBYDB_BENCHMARK_ITERATIONS=100 ruby -Ilib benchmarks/basic_workload.rb
189
+ ruby benchmarks/concurrent_workload.rb
190
+ ruby scripts/production_soak
191
+ ```
192
+
193
+ Keep benchmark artifacts out of commits unless they are intentional fixtures.
194
+
195
+ ## Fuzzing and property failures
196
+
197
+ Save the seed, generated SQL/input, Ruby version, commit, and failing database
198
+ directory. Reduce the number of operations while preserving the seed. Check
199
+ the invariant: no crash, no invalid state, rollback equivalence, parser
200
+ round-trip, or index/table agreement. Add the minimized case as a deterministic
201
+ spec, then keep the fuzz run as a secondary guard.
202
+
203
+ ## What not to do
204
+
205
+ Do not delete WAL or lock files, edit database bytes manually, disable checksum
206
+ validation, run repair on the only copy, force a replica promotion without
207
+ fencing, or publish sanitized logs that still contain secrets. These actions
208
+ can turn a diagnosable incident into irreversible data loss or a security
209
+ incident.
210
+
211
+ ## Diagnostic report template
212
+
213
+ ```text
214
+ Summary:
215
+ First observed (UTC):
216
+ RubyDB version/commit:
217
+ Ruby/Rails/OS/filesystem:
218
+ Topology and ownership mode:
219
+ Configuration names/checksum:
220
+ Command or request shape:
221
+ Request/transaction/LSN IDs:
222
+ Expected result:
223
+ Actual result:
224
+ Reproduction and seed:
225
+ Logs/metrics/checksums:
226
+ Actions already taken:
227
+ Data impact and current containment:
228
+ ```
229
+
@@ -0,0 +1,10 @@
1
+ # Branching
2
+
3
+ RubyDB branches represent database snapshots and development lines in the
4
+ engine. Create and inspect branches with the CLI, verify the target state, and
5
+ retain a backup before merging or checking out a branch containing important
6
+ data.
7
+
8
+ Branch operations are not a substitute for backups or replication. Test branch
9
+ diff, merge, checkout, conflict handling, and reopen behavior before using them
10
+ in an operational workflow.
@@ -0,0 +1,10 @@
1
+ # Database diff
2
+
3
+ Database diff compares schema and supported state between database snapshots or
4
+ branches. Use it to review migration impact and branch changes before applying
5
+ them. A diff is evidence for review, not an automatic guarantee that every
6
+ application query remains compatible.
7
+
8
+ For production changes, take a verified backup, inspect the diff, apply it to a
9
+ staging restore with representative data, run migrations and smoke queries, and
10
+ retain the original for rollback.
@@ -0,0 +1,17 @@
1
+ # Local development
2
+
3
+ ```sh
4
+ bundle install
5
+ bundle exec rspec
6
+ bundle exec rubocop
7
+ ```
8
+
9
+ Use `tmp/` or `Dir.mktmpdir` for databases. The server/client examples are
10
+ appropriate when testing process boundaries. Do not open one embedded path from
11
+ multiple processes. Use the reported fuzz seed and commit when reproducing a
12
+ failure.
13
+
14
+ Start with the [developer guide](../developer-guide.md). For failures, use a
15
+ fresh copy, preserve WAL and metadata, run the narrowest spec first, then the
16
+ full suite. The [debugging playbook](../debugging.md) describes safe logging,
17
+ thread dumps, protocol isolation, and diagnostic reports.
@@ -0,0 +1,9 @@
1
+ # Snapshots
2
+
3
+ Snapshots capture a consistent RubyDB state for inspection, backup validation,
4
+ branching, or staging. Create snapshots through the supported CLI/API and keep
5
+ the snapshot with its metadata and checksum.
6
+
7
+ Validate a snapshot by opening it in a separate directory. Do not treat a
8
+ snapshot as a replacement for an independently verified backup or replication.
9
+ Keep retention and deletion policies explicit for production data.
@@ -0,0 +1,10 @@
1
+ # Temporal data
2
+
3
+ MVCC keeps row versions long enough for active transaction visibility and safe
4
+ vacuum. Readers observe a transaction-consistent view according to the selected
5
+ isolation behavior; uncommitted changes are not published to other readers.
6
+
7
+ Long-running transactions retain old versions and can increase storage. Monitor
8
+ transaction age and vacuum/compaction duration, and terminate or redesign stale
9
+ work before it affects the workload. Test temporal behavior with restart and
10
+ rollback, not only a single read.
@@ -0,0 +1,297 @@
1
+ # RubyDB developer guide
2
+
3
+ This is the implementation guide for contributors who need to change RubyDB
4
+ safely. It explains the repository, the request path, the storage invariants,
5
+ the test strategy, and the debugging workflow. Read it together with the
6
+ topic specifications under `spec/`; the specifications are the behavioral
7
+ contract, while this guide explains how the pieces fit together.
8
+
9
+ ## 1. Scope and support promise
10
+
11
+ RubyDB is a Ruby-native relational database with two ownership modes:
12
+
13
+ * embedded mode, where one Ruby process owns a database directory; and
14
+ * server mode, where the server owns the directory and application processes
15
+ use the client protocol.
16
+
17
+ The supported surface is the behavior exercised by the test suite and listed
18
+ in [SQL compatibility](sql/compatibility.md). Similar syntax does not imply
19
+ complete PostgreSQL, MySQL, or SQLite compatibility. A change that broadens
20
+ syntax must also specify its semantics, errors, types, transactions, and
21
+ adapter behavior.
22
+
23
+ ## 2. Repository orientation
24
+
25
+ The important directories are:
26
+
27
+ | Path | Responsibility |
28
+ | --- | --- |
29
+ | `lib/rubydb` | Engine, SQL, storage, transactions, server, client, adapters |
30
+ | `spec` | Unit, integration, protocol, SQL, recovery, and adapter contracts |
31
+ | `benchmarks` | Repeatable throughput and concurrency measurements |
32
+ | `scripts` | Soak tests, durability drills, fuzzing, and release checks |
33
+ | `config` | Example configuration and monitoring rules |
34
+ | `examples` | Runnable Ruby and Rails applications |
35
+ | `docs` | User, operator, developer, and compatibility documentation |
36
+ | `packaging` | Container and deployment examples |
37
+ | `exe` and `bin` | Public command-line entry points |
38
+
39
+ Start with `lib/rubydb.rb` and follow the public API into the engine. For a
40
+ feature, locate the nearest existing spec before editing implementation code.
41
+ Avoid adding a second abstraction when an existing subsystem already owns the
42
+ invariant.
43
+
44
+ ## 3. Local setup
45
+
46
+ Use a supported Ruby version from the project CI matrix and install the locked
47
+ dependencies:
48
+
49
+ ```sh
50
+ bundle install
51
+ bundle exec rspec
52
+ bundle exec rubocop
53
+ ```
54
+
55
+ Use a temporary database for experiments. Never use a production path in a
56
+ test or run a destructive command against an unknown directory:
57
+
58
+ ```ruby
59
+ require "tmpdir"
60
+ require "rubydb"
61
+
62
+ Dir.mktmpdir("rubydb-dev-") do |dir|
63
+ engine = RubyDB::Storage::Engine.new(File.join(dir, "dev.rdb"))
64
+ engine.execute("CREATE TABLE items (id INTEGER PRIMARY KEY, name TEXT)")
65
+ engine.close
66
+ end
67
+ ```
68
+
69
+ The server/client path is required when testing process boundaries. Multiple
70
+ independent embedded owners must never open the same database directory.
71
+
72
+ ## 4. The request lifecycle
73
+
74
+ A typical SQL request follows this sequence:
75
+
76
+ ```text
77
+ Ruby API or client
78
+ -> connection/session and authentication
79
+ -> SQL tokenizer/parser
80
+ -> binder and type/parameter validation
81
+ -> planner and executor
82
+ -> transaction/MVCC read or write set
83
+ -> table/index mutation
84
+ -> WAL append and durable commit
85
+ -> result encoding and protocol response
86
+ ```
87
+
88
+ Keep failures at the layer that owns them. Parser errors should not be turned
89
+ into storage errors. A failed WAL append must not report a committed schema or
90
+ row change. A client timeout must not silently convert an unknown commit into
91
+ a rollback; the caller must be able to determine whether a request was
92
+ committed before retrying.
93
+
94
+ ## 5. Storage and durability invariants
95
+
96
+ The storage directory contains data pages, metadata, WAL state, and recovery
97
+ artifacts. Changes must preserve these invariants:
98
+
99
+ 1. A page has valid framing and checksum before it is trusted.
100
+ 2. WAL records are validated before replay and are applied in LSN order.
101
+ 3. A commit acknowledgement is emitted only after the configured durability
102
+ point has succeeded.
103
+ 4. Schema publication and its dependent indexes are visible atomically.
104
+ 5. Recovery is idempotent: replaying a committed record does not duplicate a
105
+ row or corrupt an index.
106
+ 6. Torn, truncated, or corrupted input fails closed with a useful error.
107
+ 7. Temporary files are not mistaken for a completed checkpoint or backup.
108
+
109
+ When changing page layout or record encoding, update the format specification,
110
+ versioning/upgrade path, compatibility tests, and recovery fixtures. Do not
111
+ silently reinterpret old bytes. If a format cannot be read, return an explicit
112
+ upgrade or corruption error and preserve the original files for diagnosis.
113
+
114
+ ## 6. WAL, checkpoints, and recovery
115
+
116
+ The WAL is the source of truth between checkpoints. A checkpoint copies safe
117
+ state to durable pages and advances the recovery boundary only after all
118
+ required data and metadata have been flushed. Recovery should:
119
+
120
+ 1. open the directory read-only where possible;
121
+ 2. validate metadata and the last known checkpoint;
122
+ 3. scan WAL frames, stopping only at a valid end boundary;
123
+ 4. reject checksum, sequence, length, or transaction inconsistencies;
124
+ 5. replay committed work and discard incomplete transactions;
125
+ 6. rebuild or validate indexes before accepting writes; and
126
+ 7. publish a recovery result with the replayed LSN and warnings.
127
+
128
+ Tests must cover normal reopen, a process terminated during a write, an
129
+ interrupted checkpoint, truncated WAL, invalid checksums, missing metadata,
130
+ full-disk behavior, and restore into a new directory. Fault injection belongs
131
+ around filesystem calls, not only around Ruby methods, because failures occur
132
+ at `fsync`, rename, allocation, and close boundaries.
133
+
134
+ ## 7. Transactions, MVCC, and locking
135
+
136
+ Every statement executes in a transaction context, whether it is explicit or
137
+ implicit. The context owns the snapshot, read view, write set, lock state,
138
+ savepoints, and commit result. A transaction must not leak locks or snapshots
139
+ when it raises, times out, is cancelled, or loses its connection.
140
+
141
+ MVCC readers use a stable visibility point. Writers create new versions and
142
+ retain the before-image needed by active readers and rollback. Vacuum may
143
+ reclaim a version only after the global safe point has passed it. A new feature
144
+ must define behavior for:
145
+
146
+ * read committed, repeatable read, and serializable transactions;
147
+ * concurrent update of the same row;
148
+ * unique and foreign-key conflicts;
149
+ * deadlock detection and victim rollback;
150
+ * lock wait timeouts and request cancellation; and
151
+ * commit acknowledgement followed by client disconnect.
152
+
153
+ Deadlock resolution must abort a complete victim transaction, release all of
154
+ its locks, and leave other transactions able to progress. Never fix a deadlock
155
+ by globally disabling locking or by releasing a lock without undoing the
156
+ corresponding write set.
157
+
158
+ ## 8. SQL implementation workflow
159
+
160
+ For a new statement or expression:
161
+
162
+ 1. write the syntax and semantic contract in `spec/sql`;
163
+ 2. add parser acceptance and rejection examples;
164
+ 3. add binder/type/nullability behavior;
165
+ 4. add planner and executor tests;
166
+ 5. test transaction, constraint, index, and error interactions;
167
+ 6. test the Ruby API and server protocol path; and
168
+ 7. test the ActiveRecord-generated SQL when the feature is adapter-visible.
169
+
170
+ Prefer parameter binding to string interpolation. Every expression needs
171
+ defined behavior for `NULL`, booleans, numeric coercion, text comparison,
172
+ collation, and invalid input. Every DDL operation needs idempotence or an
173
+ explicit error contract. Every optimizer rewrite must have a semantic
174
+ equivalence test against the non-optimized execution path.
175
+
176
+ The documented common SQLite-style profile is intentionally narrower than
177
+ SQLite itself. Do not label a feature “SQLite compatible” until its syntax,
178
+ results, types, error behavior, and migration behavior are tested.
179
+
180
+ ## 9. Tables, indexes, and constraints
181
+
182
+ Table mutations and index mutations are one logical operation. If an index
183
+ write fails, the statement must fail visibly and the transaction must either
184
+ roll back or retain a recoverable pending state; it must not acknowledge a row
185
+ that cannot be found by a required index.
186
+
187
+ For each index type, test empty and populated tables, duplicate keys, `NULL`,
188
+ deep splits, reopen, recovery replay, deletion/merge, and concurrent readers.
189
+ For each constraint, test direct SQL, prepared parameters, ActiveRecord
190
+ inserts/updates, rollback, and the exact error class/message contract where
191
+ callers may depend on it.
192
+
193
+ ## 10. Server, client, and wire protocol
194
+
195
+ The server is the single owner of the database directory. A client session
196
+ handles authentication, capability negotiation, request IDs, deadlines,
197
+ cancellation, result framing, and connection shutdown. Protocol changes must
198
+ be backward-compatible or carry an explicit protocol version and rejection
199
+ path.
200
+
201
+ Wire cancellation is part of correctness. A cancelled request must stop
202
+ execution at safe checkpoints, release transaction resources, and return a
203
+ definitive cancellation response. If cancellation races with commit, the
204
+ server must preserve the commit outcome and expose enough request identity for
205
+ the client to query status rather than blindly retrying.
206
+
207
+ Test malformed lengths, unknown message types, duplicate request IDs, partial
208
+ frames, client disconnects, server shutdown, timeout races, authentication
209
+ failure, TLS negotiation, and cancellation during a long scan.
210
+
211
+ ## 11. Rails and adapter work
212
+
213
+ The ActiveRecord adapter translates Rails schema and query APIs into RubyDB
214
+ SQL. Adapter code must preserve Rails expectations for quoting, bind
215
+ parameters, affected-row counts, last-insert IDs, transactions, savepoints,
216
+ schema introspection, migration versions, and exceptions.
217
+
218
+ When changing adapter behavior, run the example app and test at least:
219
+
220
+ * `where`, ordering, limits, scopes, joins, and aggregate relations;
221
+ * eager loading, nested associations, and inverse relationships;
222
+ * connection pools with multiple threads;
223
+ * schema dump/load and populated-table migrations;
224
+ * rollback, retry, and migration checksum behavior; and
225
+ * every supported Ruby/Rails combination in CI.
226
+
227
+ The adapter is not proof of complete SQLite or PostgreSQL compatibility. An
228
+ application should run its own generated-SQL and migration suite before a
229
+ cutover.
230
+
231
+ ## 12. Testing pyramid
232
+
233
+ Use the smallest test that proves the invariant:
234
+
235
+ * unit specs for parsing, encoding, types, and isolated algorithms;
236
+ * component specs for storage, WAL, indexes, transactions, and protocol;
237
+ * integration specs for engine-to-SQL and server-to-client behavior;
238
+ * process specs for crash recovery, pooling, and failover;
239
+ * workload tests for sustained concurrency and resource limits; and
240
+ * example applications for real Rails and Ruby workflows.
241
+
242
+ Useful commands include:
243
+
244
+ ```sh
245
+ bundle exec rspec
246
+ bundle exec rspec spec/sql spec/storage spec/transactions
247
+ ruby scripts/durability_drill
248
+ ruby scripts/production_soak
249
+ ruby scripts/fuzz
250
+ RUBYDB_BENCHMARK_ITERATIONS=100 ruby -Ilib benchmarks/basic_workload.rb
251
+ ```
252
+
253
+ Record the Ruby version, commit, OS, command, seed, database configuration,
254
+ and artifact paths for every non-trivial run. A green unit suite does not
255
+ replace a restore drill, multi-process soak, or deployment test.
256
+
257
+ ## 13. Debugging a failing change
258
+
259
+ Start with [Debugging RubyDB](debugging.md). Reproduce on a fresh temporary
260
+ directory, reduce the schema and SQL, rerun with the reported seed, and save
261
+ logs plus the WAL metadata. Compare embedded and server/client execution when
262
+ ownership or protocol is suspected. Use [Troubleshooting](troubleshooting.md)
263
+ for operator symptoms and evidence-preserving recovery steps.
264
+
265
+ Never “repair” a failing test by deleting WAL, disabling checksums, loosening a
266
+ constraint, or turning off synchronization. Preserve the failing directory,
267
+ copy it, and investigate the copy.
268
+
269
+ ## 14. Performance and capacity work
270
+
271
+ Benchmark one variable at a time and report latency percentiles, throughput,
272
+ concurrency, payload shape, cache size, WAL/checkpoint settings, and storage
273
+ medium. Track p50, p95, p99, error rate, lock wait time, WAL growth, checkpoint
274
+ duration, memory, file descriptors, and CPU.
275
+
276
+ Capacity limits are workload-specific. A benchmark result is not a guarantee
277
+ for a Rails application with different indexes, query shapes, or connection
278
+ pool settings. Add a regression threshold only after the benchmark is stable
279
+ across repeated runs and environments.
280
+
281
+ ## 15. Safe contribution checklist
282
+
283
+ Before opening a pull request:
284
+
285
+ 1. Explain the invariant and compatibility impact.
286
+ 2. Add focused tests, including failure paths.
287
+ 3. Run the relevant focused suite and the full suite.
288
+ 4. Run formatting/lint and `git diff --check`.
289
+ 5. Update the appropriate spec and user/operator documentation.
290
+ 6. Note migrations, format changes, recovery implications, and rollback plan.
291
+ 7. Include benchmark or soak evidence for hot-path and concurrency changes.
292
+ 8. Remove debug output, secrets, temporary data, and generated artifacts.
293
+
294
+ See [Contributing](../CONTRIBUTING.md), [testing](contributing/testing.md),
295
+ and [release process](contributing/release-process.md) for the repository
296
+ workflow.
297
+
@@ -0,0 +1,16 @@
1
+ # First database
2
+
3
+ ```ruby
4
+ require "rubydb"
5
+
6
+ path = "tmp/first-database.rdb"
7
+ engine = RubyDB::Storage::Engine.new(path)
8
+ engine.execute("CREATE TABLE accounts (id INTEGER PRIMARY KEY, email TEXT UNIQUE NOT NULL)")
9
+ engine.execute("INSERT INTO accounts (id, email) VALUES (1, 'ada@example.test')")
10
+ puts engine.execute("SELECT id, email FROM accounts").inspect
11
+ engine.close
12
+ ```
13
+
14
+ The engine creates durable storage and WAL state under the selected path. Close
15
+ the engine cleanly and keep the database, WAL, and lock files together. A single
16
+ embedded path has one owner; use a server for multiple processes.