benodb 0.17.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (614) hide show
  1. benodb-0.17.0/.cargo/config.toml +3 -0
  2. benodb-0.17.0/.dockerignore +17 -0
  3. benodb-0.17.0/.env.example +51 -0
  4. benodb-0.17.0/.gitattributes +1 -0
  5. benodb-0.17.0/.gitignore +150 -0
  6. benodb-0.17.0/.instructions.md +67 -0
  7. benodb-0.17.0/.mailmap +6 -0
  8. benodb-0.17.0/AGENTS.md +117 -0
  9. benodb-0.17.0/BenoDB.png +0 -0
  10. benodb-0.17.0/CHANGELOG.md +3756 -0
  11. benodb-0.17.0/CONTRIBUTING.md +279 -0
  12. benodb-0.17.0/Cargo.lock +9804 -0
  13. benodb-0.17.0/Cargo.toml +259 -0
  14. benodb-0.17.0/Dockerfile +110 -0
  15. benodb-0.17.0/LICENSE-APACHE +201 -0
  16. benodb-0.17.0/LICENSE-MIT +21 -0
  17. benodb-0.17.0/NOTICE +22 -0
  18. benodb-0.17.0/PKG-INFO +1253 -0
  19. benodb-0.17.0/README.md +1204 -0
  20. benodb-0.17.0/SECURITY.md +124 -0
  21. benodb-0.17.0/THIRDPARTY_NOTICES.md +109 -0
  22. benodb-0.17.0/audit.toml +48 -0
  23. benodb-0.17.0/benodb-gpu-ann/Cargo.toml +44 -0
  24. benodb-0.17.0/benodb-gpu-ann/LICENSE-APACHE +201 -0
  25. benodb-0.17.0/benodb-gpu-ann/LICENSE-MIT +21 -0
  26. benodb-0.17.0/benodb-gpu-ann/NOTICE +13 -0
  27. benodb-0.17.0/benodb-gpu-ann/src/backend/cpu.rs +142 -0
  28. benodb-0.17.0/benodb-gpu-ann/src/backend/cuda.rs +334 -0
  29. benodb-0.17.0/benodb-gpu-ann/src/backend/metal.rs +172 -0
  30. benodb-0.17.0/benodb-gpu-ann/src/backend/mod.rs +216 -0
  31. benodb-0.17.0/benodb-gpu-ann/src/backend/nvrtc.rs +542 -0
  32. benodb-0.17.0/benodb-gpu-ann/src/backend/wgpu.rs +211 -0
  33. benodb-0.17.0/benodb-gpu-ann/src/cagra.rs +578 -0
  34. benodb-0.17.0/benodb-gpu-ann/src/e_means.rs +608 -0
  35. benodb-0.17.0/benodb-gpu-ann/src/fastscan.rs +392 -0
  36. benodb-0.17.0/benodb-gpu-ann/src/hnsw.rs +533 -0
  37. benodb-0.17.0/benodb-gpu-ann/src/ivf_fastscan.rs +279 -0
  38. benodb-0.17.0/benodb-gpu-ann/src/ivf_flat.rs +245 -0
  39. benodb-0.17.0/benodb-gpu-ann/src/kernels/cuda/batch_distance.cu +165 -0
  40. benodb-0.17.0/benodb-gpu-ann/src/kernels/cuda/cosine_distance.cu +61 -0
  41. benodb-0.17.0/benodb-gpu-ann/src/kernels/cuda/hamming_distance.cu +43 -0
  42. benodb-0.17.0/benodb-gpu-ann/src/kernels/cuda/hamming_packed.cu +28 -0
  43. benodb-0.17.0/benodb-gpu-ann/src/kernels/cuda/inner_product.cu +40 -0
  44. benodb-0.17.0/benodb-gpu-ann/src/kernels/cuda/jaccard_distance.cu +64 -0
  45. benodb-0.17.0/benodb-gpu-ann/src/kernels/cuda/jaccard_packed.cu +41 -0
  46. benodb-0.17.0/benodb-gpu-ann/src/kernels/cuda/kmeans_assignment.cu +33 -0
  47. benodb-0.17.0/benodb-gpu-ann/src/kernels/cuda/l1_distance.cu +40 -0
  48. benodb-0.17.0/benodb-gpu-ann/src/kernels/cuda/l2_distance.cu +40 -0
  49. benodb-0.17.0/benodb-gpu-ann/src/kernels/metal/cosine_distance.metal +41 -0
  50. benodb-0.17.0/benodb-gpu-ann/src/kernels/metal/hamming_distance.metal +31 -0
  51. benodb-0.17.0/benodb-gpu-ann/src/kernels/metal/hamming_packed.metal +26 -0
  52. benodb-0.17.0/benodb-gpu-ann/src/kernels/metal/inner_product.metal +29 -0
  53. benodb-0.17.0/benodb-gpu-ann/src/kernels/metal/jaccard_distance.metal +43 -0
  54. benodb-0.17.0/benodb-gpu-ann/src/kernels/metal/jaccard_packed.metal +30 -0
  55. benodb-0.17.0/benodb-gpu-ann/src/kernels/metal/kmeans_assignment.metal +41 -0
  56. benodb-0.17.0/benodb-gpu-ann/src/kernels/metal/l1_distance.metal +29 -0
  57. benodb-0.17.0/benodb-gpu-ann/src/kernels/metal/l2_distance.metal +30 -0
  58. benodb-0.17.0/benodb-gpu-ann/src/kernels/wgsl/wgpu_binary_kernel.wgsl +46 -0
  59. benodb-0.17.0/benodb-gpu-ann/src/kernels/wgsl/wgpu_kernel.wgsl +96 -0
  60. benodb-0.17.0/benodb-gpu-ann/src/kmeans.rs +194 -0
  61. benodb-0.17.0/benodb-gpu-ann/src/lib.rs +272 -0
  62. benodb-0.17.0/benodb-gpu-ann/src/metric.rs +33 -0
  63. benodb-0.17.0/benodb-gpu-ann/tests/test_cagra.rs +173 -0
  64. benodb-0.17.0/benodb-gpu-ann/tests/test_cross_backend.rs +249 -0
  65. benodb-0.17.0/benodb-gpu-ann/tests/test_edge_cases.rs +256 -0
  66. benodb-0.17.0/benodb-gpu-ann/tests/test_fuzz_no_panic.rs +120 -0
  67. benodb-0.17.0/benodb-gpu-ann/tests/test_hnsw.rs +163 -0
  68. benodb-0.17.0/benodb-gpu-ann/tests/test_ivf_fastscan.rs +81 -0
  69. benodb-0.17.0/benodb-gpu-ann/tests/test_ivf_flat.rs +101 -0
  70. benodb-0.17.0/benodb-gpu-ann/tests/test_parity_all_algorithms.rs +164 -0
  71. benodb-0.17.0/benodb-gpu-ann/tests/test_soak.rs +116 -0
  72. benodb-0.17.0/blog/articles/2026-10-22_graph_rag_howto.ipynb +2030 -0
  73. benodb-0.17.0/blog/articles/2026-10-27_graph_rag_howto.ipynb +2036 -0
  74. benodb-0.17.0/book.toml +10 -0
  75. benodb-0.17.0/build-connectors.sh +187 -0
  76. benodb-0.17.0/build.rs +58 -0
  77. benodb-0.17.0/clippy.toml +17 -0
  78. benodb-0.17.0/dbt-benodb/README.md +82 -0
  79. benodb-0.17.0/dbt-benodb/dbt/adapters/benodb/__init__.py +12 -0
  80. benodb-0.17.0/dbt-benodb/dbt/adapters/benodb/__version__.py +57 -0
  81. benodb-0.17.0/dbt-benodb/dbt/adapters/benodb/connections.py +206 -0
  82. benodb-0.17.0/dbt-benodb/dbt/adapters/benodb/impl.py +96 -0
  83. benodb-0.17.0/dbt-benodb/dbt/include/benodb/__init__.py +3 -0
  84. benodb-0.17.0/dbt-benodb/dbt/include/benodb/dbt_project.yml +10 -0
  85. benodb-0.17.0/dbt-benodb/dbt/include/benodb/macros/adapters.sql +90 -0
  86. benodb-0.17.0/dbt-benodb/dbt/include/benodb/macros/ddl.sql +89 -0
  87. benodb-0.17.0/dbt-benodb/dbt/include/benodb/macros/graph.sql +503 -0
  88. benodb-0.17.0/dbt-benodb/dbt/include/benodb/macros/json.sql +115 -0
  89. benodb-0.17.0/dbt-benodb/dbt/include/benodb/macros/materializations/incremental.sql +108 -0
  90. benodb-0.17.0/dbt-benodb/dbt/include/benodb/macros/materializations/table.sql +19 -0
  91. benodb-0.17.0/dbt-benodb/dbt/include/benodb/macros/search.sql +55 -0
  92. benodb-0.17.0/dbt-benodb/dbt/include/benodb/macros/vector.sql +349 -0
  93. benodb-0.17.0/dbt-benodb/pyproject.toml +53 -0
  94. benodb-0.17.0/dbt-benodb/setup.py +10 -0
  95. benodb-0.17.0/deny.toml +116 -0
  96. benodb-0.17.0/deploy/observability/README.md +74 -0
  97. benodb-0.17.0/deploy/observability/docker-compose.observability.yml +54 -0
  98. benodb-0.17.0/deploy/observability/grafana/dashboards/benodb.json +162 -0
  99. benodb-0.17.0/deploy/observability/grafana/provisioning/dashboards/dashboards.yml +13 -0
  100. benodb-0.17.0/deploy/observability/grafana/provisioning/datasources/prometheus.yml +12 -0
  101. benodb-0.17.0/deploy/observability/otel-collector.yaml +32 -0
  102. benodb-0.17.0/deploy/observability/prometheus.yml +17 -0
  103. benodb-0.17.0/dml_copy.rs +693 -0
  104. benodb-0.17.0/docker-compose-hive.yml +104 -0
  105. benodb-0.17.0/docker-compose-rustfs-nessie.yml +135 -0
  106. benodb-0.17.0/docker-compose.production.yml +121 -0
  107. benodb-0.17.0/docker-compose.yml +153 -0
  108. benodb-0.17.0/docs/.nojekyll +0 -0
  109. benodb-0.17.0/docs/ADMIN_CLI.md +59 -0
  110. benodb-0.17.0/docs/ARCHITECTURE.md +98 -0
  111. benodb-0.17.0/docs/BENCHMARKING.md +652 -0
  112. benodb-0.17.0/docs/BENCHMARKING_PLAN.md +455 -0
  113. benodb-0.17.0/docs/CATALOG_USAGE.md +429 -0
  114. benodb-0.17.0/docs/CONCURRENCY.md +58 -0
  115. benodb-0.17.0/docs/CONFIGURATION.md +721 -0
  116. benodb-0.17.0/docs/EMEANS_SPECIFICATION.md +191 -0
  117. benodb-0.17.0/docs/GPU_SETUP_GUIDE.md +483 -0
  118. benodb-0.17.0/docs/GRAPH_RAG_EDGE_TABLES.md +374 -0
  119. benodb-0.17.0/docs/ICEBERG_COMPATIBILITY.md +75 -0
  120. benodb-0.17.0/docs/ICEBERG_STORAGE.md +211 -0
  121. benodb-0.17.0/docs/ICEBERG_V2_V3_API.md +249 -0
  122. benodb-0.17.0/docs/INDEX.md +76 -0
  123. benodb-0.17.0/docs/INSTALLATION.md +734 -0
  124. benodb-0.17.0/docs/MONITORING.md +150 -0
  125. benodb-0.17.0/docs/MUSIQUE_GRAPHRAG.md +128 -0
  126. benodb-0.17.0/docs/NO_PANIC_POLICY.md +146 -0
  127. benodb-0.17.0/docs/OPENSEARCH_COMPATIBILITY.md +310 -0
  128. benodb-0.17.0/docs/PGVECTOR_SQL_GUIDE.md +735 -0
  129. benodb-0.17.0/docs/PRODUCTION_READINESS_ASSESSMENT.md +136 -0
  130. benodb-0.17.0/docs/PYTHON_VECTOR_API.md +1071 -0
  131. benodb-0.17.0/docs/QDRANT_COMPATIBILITY.md +160 -0
  132. benodb-0.17.0/docs/QUICKSTART.md +262 -0
  133. benodb-0.17.0/docs/RECOVERY_RUNBOOK.md +101 -0
  134. benodb-0.17.0/docs/RESOURCE_LIMITS.md +197 -0
  135. benodb-0.17.0/docs/ROADMAP.md +474 -0
  136. benodb-0.17.0/docs/SUPPORT_MATRIX.md +127 -0
  137. benodb-0.17.0/docs/architecture/ADR_001_TIGHTENED_INDEX_OVERLAY_CONTRACT.md +546 -0
  138. benodb-0.17.0/docs/architecture/ADR_002_GLOBAL_VECTOR_CLUSTERING.md +170 -0
  139. benodb-0.17.0/docs/architecture/ADR_003_GRAPH_RAG_OUT_OF_CORE_MEMORY.md +85 -0
  140. benodb-0.17.0/docs/architecture/ADR_004_OBSERVABILITY_PIPELINE.md +145 -0
  141. benodb-0.17.0/docs/architecture/ADR_005_SQL_DDL_AND_MAINTENANCE_SURFACE.md +289 -0
  142. benodb-0.17.0/docs/architecture/ADR_006_TRINO_TABLE_FUNCTION_PUSHDOWN.md +409 -0
  143. benodb-0.17.0/docs/architecture/ADR_007_NATIVE_CONNECTOR_ARCHITECTURE_AND_UDF_EXPOSURE.md +169 -0
  144. benodb-0.17.0/docs/architecture/ADR_008_PENDING_COMMIT_STATE_MACHINE.md +135 -0
  145. benodb-0.17.0/docs/architecture/ADR_009_CATALOG_PARITY_AND_CAS_HARDENING.md +120 -0
  146. benodb-0.17.0/docs/architecture/ADR_010_ICEBERG_V3_DELETION_VECTORS_AND_CROSS_ENGINE_CONFORMANCE.md +57 -0
  147. benodb-0.17.0/docs/architecture/README.md +97 -0
  148. benodb-0.17.0/docs/governance/DEPENDENCY_RISK.md +183 -0
  149. benodb-0.17.0/docs/integrations/DBT.md +239 -0
  150. benodb-0.17.0/docs/integrations/FLIGHT_SQL.md +143 -0
  151. benodb-0.17.0/docs/integrations/ICEBERG.md +184 -0
  152. benodb-0.17.0/docs/integrations/JAVA_JNI.md +34 -0
  153. benodb-0.17.0/docs/integrations/MCP.md +141 -0
  154. benodb-0.17.0/docs/integrations/PYTHON.md +170 -0
  155. benodb-0.17.0/docs/integrations/README.md +27 -0
  156. benodb-0.17.0/docs/integrations/SEARCH_GATEWAY.md +116 -0
  157. benodb-0.17.0/docs/integrations/SNOWFLAKE.md +172 -0
  158. benodb-0.17.0/docs/integrations/SPARK.md +401 -0
  159. benodb-0.17.0/docs/integrations/TRINO.md +408 -0
  160. benodb-0.17.0/docs/integrations/TRINO_VERSION_BANDS.md +299 -0
  161. benodb-0.17.0/docs/requirements.txt +10 -0
  162. benodb-0.17.0/docs/source/_static/BenoDB.png +0 -0
  163. benodb-0.17.0/docs/source/_static/custom.css +11 -0
  164. benodb-0.17.0/docs/source/api/python.rst +40 -0
  165. benodb-0.17.0/docs/source/api/rust.rst +11 -0
  166. benodb-0.17.0/docs/source/conf.py +104 -0
  167. benodb-0.17.0/docs/source/index.rst +119 -0
  168. benodb-0.17.0/docs/tutorials/01_AI_RETRIEVAL_MCP.md +121 -0
  169. benodb-0.17.0/docs/tutorials/02_JSON_DOCUMENT_RETRIEVAL.md +76 -0
  170. benodb-0.17.0/docs/tutorials/03_GRAPH_TRAVERSAL.md +72 -0
  171. benodb-0.17.0/hive-config/hive-site.xml +75 -0
  172. benodb-0.17.0/plans/OUT_OF_CORE_SPEEDUP_ANALYSIS.md +353 -0
  173. benodb-0.17.0/plans/production_readiness_plan.md +829 -0
  174. benodb-0.17.0/proptest-regressions/core/index/gpu.txt +9 -0
  175. benodb-0.17.0/proptest-regressions/core/sql/vector_literal.txt +10 -0
  176. benodb-0.17.0/proptest-regressions/core/sql/vector_udf.txt +8 -0
  177. benodb-0.17.0/pyproject.toml +93 -0
  178. benodb-0.17.0/python/benodb/__init__.py +4387 -0
  179. benodb-0.17.0/python/benodb/cli.py +225 -0
  180. benodb-0.17.0/python/benodb/embeddings.py +100 -0
  181. benodb-0.17.0/run_test.py +6 -0
  182. benodb-0.17.0/rust-toolchain.toml +11 -0
  183. benodb-0.17.0/scripts/ai_pr_review.py +207 -0
  184. benodb-0.17.0/scripts/build_demo_dataset.py +370 -0
  185. benodb-0.17.0/scripts/build_three_tier.py +700 -0
  186. benodb-0.17.0/scripts/check_docs_links.py +77 -0
  187. benodb-0.17.0/scripts/clean_scratch.sh +83 -0
  188. benodb-0.17.0/scripts/download_wiki_dumps.py +131 -0
  189. benodb-0.17.0/scripts/git-hooks/pre-push +27 -0
  190. benodb-0.17.0/scripts/inject_canonical.py +46 -0
  191. benodb-0.17.0/scripts/install_hooks.sh +18 -0
  192. benodb-0.17.0/scripts/no_panic_baseline.txt +1 -0
  193. benodb-0.17.0/scripts/no_panic_check.sh +144 -0
  194. benodb-0.17.0/scripts/pre_release_check.sh +283 -0
  195. benodb-0.17.0/scripts/pre_release_soak.sh +103 -0
  196. benodb-0.17.0/scripts/prepare_demo.py +652 -0
  197. benodb-0.17.0/scripts/prepare_docs.sh +80 -0
  198. benodb-0.17.0/scripts/release_teardown.py +507 -0
  199. benodb-0.17.0/scripts/repro_hnsw_crash.py +92 -0
  200. benodb-0.17.0/scripts/services/README.md +76 -0
  201. benodb-0.17.0/scripts/services/benodb-search-runner.sh +13 -0
  202. benodb-0.17.0/scripts/services/benodb-search.conf +35 -0
  203. benodb-0.17.0/scripts/services/benodb-search.service +16 -0
  204. benodb-0.17.0/scripts/services/com.benodb.search.plist +21 -0
  205. benodb-0.17.0/scripts/services/install.sh +75 -0
  206. benodb-0.17.0/scripts/services/uninstall.sh +49 -0
  207. benodb-0.17.0/spark-benodb/README.md +144 -0
  208. benodb-0.17.0/spark-benodb/pom.xml +306 -0
  209. benodb-0.17.0/spark-benodb/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister +1 -0
  210. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/BenoDBArrowIpc.scala +213 -0
  211. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/BenoDBArrowUtils.scala +207 -0
  212. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/BenoDBArrowWriter.scala +81 -0
  213. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/BenoDBCatalogFunctions.scala +503 -0
  214. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/BenoDBDataFrameOps.scala +56 -0
  215. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/BenoDBJson.scala +133 -0
  216. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/BenoDBQueryTable.scala +74 -0
  217. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/BenoDBRowLevelOperation.scala +296 -0
  218. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/BenoDBScanBuilder.scala +428 -0
  219. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/BenoDBTable.scala +78 -0
  220. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/BenoDBTableCatalog.scala +127 -0
  221. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/BenoDBWriteBuilder.scala +145 -0
  222. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/DefaultSource.scala +67 -0
  223. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/functions/BenoDBFunctions.scala +86 -0
  224. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/gpu/GpuContextResolver.scala +71 -0
  225. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/jni/BenoDBJNIBridge.scala +210 -0
  226. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/procedures/BenoDBTableResolver.scala +90 -0
  227. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/vector/SparseVector.scala +38 -0
  228. benodb-0.17.0/spark-benodb/src/main/scala/com/benodb/spark/vector/VectorDistance.scala +169 -0
  229. benodb-0.17.0/spark-benodb/src/main/spark-3/com/benodb/spark/BenoDBCatalog.scala +58 -0
  230. benodb-0.17.0/spark-benodb/src/main/spark-3/com/benodb/spark/procedures/AddIndexProcedure.scala +45 -0
  231. benodb-0.17.0/spark-benodb/src/main/spark-3/com/benodb/spark/procedures/BuildIndexProcedure.scala +36 -0
  232. benodb-0.17.0/spark-benodb/src/main/spark-3/com/benodb/spark/procedures/CompactTableProcedure.scala +36 -0
  233. benodb-0.17.0/spark-benodb/src/main/spark-3/com/benodb/spark/procedures/DropIndexProcedure.scala +42 -0
  234. benodb-0.17.0/spark-benodb/src/main/spark-3/com/benodb/spark/procedures/DropPrimaryKeyProcedure.scala +39 -0
  235. benodb-0.17.0/spark-benodb/src/main/spark-3/com/benodb/spark/procedures/RebuildClusterIndexProcedure.scala +47 -0
  236. benodb-0.17.0/spark-benodb/src/main/spark-3/com/benodb/spark/procedures/RebuildIndexProcedure.scala +40 -0
  237. benodb-0.17.0/spark-benodb/src/main/spark-3/com/benodb/spark/procedures/RegionalDriftProcedure.scala +50 -0
  238. benodb-0.17.0/spark-benodb/src/main/spark-3/com/benodb/spark/procedures/SetPrimaryKeyProcedure.scala +39 -0
  239. benodb-0.17.0/spark-benodb/src/main/spark-3/com/benodb/spark/procedures/ShowIndexesProcedure.scala +78 -0
  240. benodb-0.17.0/spark-benodb/src/main/spark-3/com/benodb/spark/procedures/WarmupProcedure.scala +38 -0
  241. benodb-0.17.0/spark-benodb/src/main/spark-4/com/benodb/spark/BenoDBCatalog.scala +80 -0
  242. benodb-0.17.0/spark-benodb/src/main/spark-4/com/benodb/spark/procedures/AddIndexProcedure.scala +42 -0
  243. benodb-0.17.0/spark-benodb/src/main/spark-4/com/benodb/spark/procedures/BenoDBProcedureScan.scala +25 -0
  244. benodb-0.17.0/spark-benodb/src/main/spark-4/com/benodb/spark/procedures/BuildIndexProcedure.scala +31 -0
  245. benodb-0.17.0/spark-benodb/src/main/spark-4/com/benodb/spark/procedures/CompactTableProcedure.scala +30 -0
  246. benodb-0.17.0/spark-benodb/src/main/spark-4/com/benodb/spark/procedures/DropIndexProcedure.scala +36 -0
  247. benodb-0.17.0/spark-benodb/src/main/spark-4/com/benodb/spark/procedures/DropPrimaryKeyProcedure.scala +31 -0
  248. benodb-0.17.0/spark-benodb/src/main/spark-4/com/benodb/spark/procedures/RebuildClusterIndexProcedure.scala +42 -0
  249. benodb-0.17.0/spark-benodb/src/main/spark-4/com/benodb/spark/procedures/RebuildIndexProcedure.scala +34 -0
  250. benodb-0.17.0/spark-benodb/src/main/spark-4/com/benodb/spark/procedures/RegionalDriftProcedure.scala +68 -0
  251. benodb-0.17.0/spark-benodb/src/main/spark-4/com/benodb/spark/procedures/SetPrimaryKeyProcedure.scala +31 -0
  252. benodb-0.17.0/spark-benodb/src/main/spark-4/com/benodb/spark/procedures/ShowIndexesProcedure.scala +87 -0
  253. benodb-0.17.0/spark-benodb/src/main/spark-4/com/benodb/spark/procedures/WarmupProcedure.scala +33 -0
  254. benodb-0.17.0/spark-benodb/src/test/java/com/benodb/spark/BenoDBJavaSparkTest.java +112 -0
  255. benodb-0.17.0/spark-benodb/src/test/scala/com/benodb/spark/BenoDBSparkWarehouseTest.scala +325 -0
  256. benodb-0.17.0/spark-benodb/src/test/scala/com/benodb/spark/BenoDBSqlTopNTest.scala +61 -0
  257. benodb-0.17.0/spark-benodb/src/test/scala/com/benodb/spark/BenoDBTableResolverTest.scala +71 -0
  258. benodb-0.17.0/spark-benodb/src/test/scala/com/benodb/spark/GpuContextResolverTest.scala +28 -0
  259. benodb-0.17.0/spark-benodb/src/test/scala/com/benodb/spark/SparkProcedureCatalogTest.scala +88 -0
  260. benodb-0.17.0/spark-benodb/src/test/scala/com/benodb/spark/VectorSearchIntegrationTest.scala +105 -0
  261. benodb-0.17.0/src/bin/bdb-admin.rs +113 -0
  262. benodb-0.17.0/src/bin/bdb.rs +367 -0
  263. benodb-0.17.0/src/bin/bench_concurrency_correctness.rs +643 -0
  264. benodb-0.17.0/src/bin/bench_recovery.rs +179 -0
  265. benodb-0.17.0/src/bin/bench_throttling.rs +601 -0
  266. benodb-0.17.0/src/bin/check_table_paths.rs +41 -0
  267. benodb-0.17.0/src/bin/fix_table_paths.rs +117 -0
  268. benodb-0.17.0/src/bin/gateway.rs +259 -0
  269. benodb-0.17.0/src/bin/iceberg_rest.rs +1141 -0
  270. benodb-0.17.0/src/bin/ingest_wikipedia.rs +320 -0
  271. benodb-0.17.0/src/bin/reindex_inverted.rs +53 -0
  272. benodb-0.17.0/src/bin/relocate_table.rs +307 -0
  273. benodb-0.17.0/src/bin/setup_test_data.rs +39 -0
  274. benodb-0.17.0/src/bin/verify_bm25.rs +40 -0
  275. benodb-0.17.0/src/bin/verify_layered_indexing.rs +254 -0
  276. benodb-0.17.0/src/core/algorithms/communities.rs +308 -0
  277. benodb-0.17.0/src/core/algorithms/connected_components.rs +288 -0
  278. benodb-0.17.0/src/core/algorithms/frontier.rs +172 -0
  279. benodb-0.17.0/src/core/algorithms/label_propagation.rs +128 -0
  280. benodb-0.17.0/src/core/algorithms/mod.rs +5 -0
  281. benodb-0.17.0/src/core/algorithms/topological_sort.rs +97 -0
  282. benodb-0.17.0/src/core/auth.rs +359 -0
  283. benodb-0.17.0/src/core/cache.rs +1328 -0
  284. benodb-0.17.0/src/core/catalog/common.rs +584 -0
  285. benodb-0.17.0/src/core/catalog/config.rs +54 -0
  286. benodb-0.17.0/src/core/catalog/glue.rs +562 -0
  287. benodb-0.17.0/src/core/catalog/hive.rs +507 -0
  288. benodb-0.17.0/src/core/catalog/jdbc.rs +635 -0
  289. benodb-0.17.0/src/core/catalog/memory.rs +282 -0
  290. benodb-0.17.0/src/core/catalog/mod.rs +583 -0
  291. benodb-0.17.0/src/core/catalog/nessie.rs +637 -0
  292. benodb-0.17.0/src/core/catalog/policy.rs +1075 -0
  293. benodb-0.17.0/src/core/catalog/rest.rs +1076 -0
  294. benodb-0.17.0/src/core/catalog/unity.rs +637 -0
  295. benodb-0.17.0/src/core/clustering.rs +209 -0
  296. benodb-0.17.0/src/core/compaction.rs +948 -0
  297. benodb-0.17.0/src/core/config.rs +7 -0
  298. benodb-0.17.0/src/core/embeddings.rs +179 -0
  299. benodb-0.17.0/src/core/error.rs +400 -0
  300. benodb-0.17.0/src/core/fault_injection.rs +232 -0
  301. benodb-0.17.0/src/core/ffi.rs +3064 -0
  302. benodb-0.17.0/src/core/iceberg/delete.rs +876 -0
  303. benodb-0.17.0/src/core/iceberg/manifest.rs +530 -0
  304. benodb-0.17.0/src/core/iceberg/mod.rs +67 -0
  305. benodb-0.17.0/src/core/iceberg/schema.rs +244 -0
  306. benodb-0.17.0/src/core/iceberg/transform.rs +446 -0
  307. benodb-0.17.0/src/core/iceberg/types.rs +100 -0
  308. benodb-0.17.0/src/core/iceberg/value.rs +233 -0
  309. benodb-0.17.0/src/core/iceberg/writer.rs +1443 -0
  310. benodb-0.17.0/src/core/index/analyzer.rs +82 -0
  311. benodb-0.17.0/src/core/index/benoquant.rs +2154 -0
  312. benodb-0.17.0/src/core/index/bm25.rs +142 -0
  313. benodb-0.17.0/src/core/index/build_graph.rs +130 -0
  314. benodb-0.17.0/src/core/index/build_inverted.rs +812 -0
  315. benodb-0.17.0/src/core/index/build_vector.rs +157 -0
  316. benodb-0.17.0/src/core/index/cluster_index.rs +5246 -0
  317. benodb-0.17.0/src/core/index/csr_graph.rs +338 -0
  318. benodb-0.17.0/src/core/index/cuda/batch_distance.cu +240 -0
  319. benodb-0.17.0/src/core/index/cuda/cosine_distance.cu +61 -0
  320. benodb-0.17.0/src/core/index/cuda/hamming_distance.cu +43 -0
  321. benodb-0.17.0/src/core/index/cuda/hamming_packed.cu +28 -0
  322. benodb-0.17.0/src/core/index/cuda/inner_product.cu +43 -0
  323. benodb-0.17.0/src/core/index/cuda/jaccard_distance.cu +64 -0
  324. benodb-0.17.0/src/core/index/cuda/jaccard_packed.cu +41 -0
  325. benodb-0.17.0/src/core/index/cuda/kmeans_assignment.cu +33 -0
  326. benodb-0.17.0/src/core/index/cuda/l1_distance.cu +40 -0
  327. benodb-0.17.0/src/core/index/cuda/l2_distance.cu +40 -0
  328. benodb-0.17.0/src/core/index/distance.rs +982 -0
  329. benodb-0.17.0/src/core/index/e_means.rs +188 -0
  330. benodb-0.17.0/src/core/index/global_centroids.rs +182 -0
  331. benodb-0.17.0/src/core/index/gpu.rs +2296 -0
  332. benodb-0.17.0/src/core/index/hnsw_ivf.rs +3834 -0
  333. benodb-0.17.0/src/core/index/hnsw_rs/LICENSE-APACHE +201 -0
  334. benodb-0.17.0/src/core/index/hnsw_rs/LICENSE-MIT +21 -0
  335. benodb-0.17.0/src/core/index/hnsw_rs/annhdf5.rs +218 -0
  336. benodb-0.17.0/src/core/index/hnsw_rs/api.rs +86 -0
  337. benodb-0.17.0/src/core/index/hnsw_rs/arrow_hnsw.rs +462 -0
  338. benodb-0.17.0/src/core/index/hnsw_rs/arrow_ipc.rs +311 -0
  339. benodb-0.17.0/src/core/index/hnsw_rs/dist.rs +443 -0
  340. benodb-0.17.0/src/core/index/hnsw_rs/flatten.rs +129 -0
  341. benodb-0.17.0/src/core/index/hnsw_rs/hnsw.rs +1953 -0
  342. benodb-0.17.0/src/core/index/hnsw_rs/hnswio.rs +860 -0
  343. benodb-0.17.0/src/core/index/hnsw_rs/mod.rs +54 -0
  344. benodb-0.17.0/src/core/index/hnsw_rs/prelude.rs +5 -0
  345. benodb-0.17.0/src/core/index/hnsw_rs/test.rs +292 -0
  346. benodb-0.17.0/src/core/index/ivf.rs +671 -0
  347. benodb-0.17.0/src/core/index/json_path.rs +482 -0
  348. benodb-0.17.0/src/core/index/memory.rs +511 -0
  349. benodb-0.17.0/src/core/index/mod.rs +288 -0
  350. benodb-0.17.0/src/core/index/mps/cosine_distance.metal +41 -0
  351. benodb-0.17.0/src/core/index/mps/hamming_distance.metal +31 -0
  352. benodb-0.17.0/src/core/index/mps/hamming_packed.metal +26 -0
  353. benodb-0.17.0/src/core/index/mps/inner_product.metal +31 -0
  354. benodb-0.17.0/src/core/index/mps/jaccard_distance.metal +43 -0
  355. benodb-0.17.0/src/core/index/mps/jaccard_packed.metal +30 -0
  356. benodb-0.17.0/src/core/index/mps/kmeans_assignment.metal +41 -0
  357. benodb-0.17.0/src/core/index/mps/l1_distance.metal +29 -0
  358. benodb-0.17.0/src/core/index/mps/l2_distance.metal +30 -0
  359. benodb-0.17.0/src/core/index/nvrtc.rs +641 -0
  360. benodb-0.17.0/src/core/index/pq.rs +864 -0
  361. benodb-0.17.0/src/core/index/sq.rs +993 -0
  362. benodb-0.17.0/src/core/index/tokenizer.rs +96 -0
  363. benodb-0.17.0/src/core/index/turboquant.rs +6 -0
  364. benodb-0.17.0/src/core/index/vector_router.rs +898 -0
  365. benodb-0.17.0/src/core/index/wgpu_binary_kernel.wgsl +46 -0
  366. benodb-0.17.0/src/core/index/wgpu_kernel.wgsl +98 -0
  367. benodb-0.17.0/src/core/jni_util.rs +158 -0
  368. benodb-0.17.0/src/core/lock.rs +327 -0
  369. benodb-0.17.0/src/core/maintenance.rs +372 -0
  370. benodb-0.17.0/src/core/manifest/manager/commit.rs +1415 -0
  371. benodb-0.17.0/src/core/manifest/manager/load.rs +619 -0
  372. benodb-0.17.0/src/core/manifest/manager/partition.rs +251 -0
  373. benodb-0.17.0/src/core/manifest/manager/schema.rs +337 -0
  374. benodb-0.17.0/src/core/manifest/manager.rs +270 -0
  375. benodb-0.17.0/src/core/manifest/mod.rs +5 -0
  376. benodb-0.17.0/src/core/manifest/types.rs +1892 -0
  377. benodb-0.17.0/src/core/memory.rs +275 -0
  378. benodb-0.17.0/src/core/merge.rs +769 -0
  379. benodb-0.17.0/src/core/metadata.rs +617 -0
  380. benodb-0.17.0/src/core/mod.rs +52 -0
  381. benodb-0.17.0/src/core/nessie.rs +124 -0
  382. benodb-0.17.0/src/core/planner.rs +2445 -0
  383. benodb-0.17.0/src/core/puffin.rs +1212 -0
  384. benodb-0.17.0/src/core/query.rs +1507 -0
  385. benodb-0.17.0/src/core/reader/delete.rs +483 -0
  386. benodb-0.17.0/src/core/reader/filter.rs +1642 -0
  387. benodb-0.17.0/src/core/reader/mod.rs +553 -0
  388. benodb-0.17.0/src/core/reader/scan.rs +3405 -0
  389. benodb-0.17.0/src/core/reader/stats.rs +172 -0
  390. benodb-0.17.0/src/core/resources.rs +519 -0
  391. benodb-0.17.0/src/core/search/mod.rs +140 -0
  392. benodb-0.17.0/src/core/search/rrf.rs +73 -0
  393. benodb-0.17.0/src/core/segment.rs +2292 -0
  394. benodb-0.17.0/src/core/sql/catalog_ddl.rs +1445 -0
  395. benodb-0.17.0/src/core/sql/graph_udf/all_shortest_paths.rs +241 -0
  396. benodb-0.17.0/src/core/sql/graph_udf/betweenness_centrality.rs +229 -0
  397. benodb-0.17.0/src/core/sql/graph_udf/closeness_centrality.rs +170 -0
  398. benodb-0.17.0/src/core/sql/graph_udf/connected_components.rs +157 -0
  399. benodb-0.17.0/src/core/sql/graph_udf/degree_centrality.rs +164 -0
  400. benodb-0.17.0/src/core/sql/graph_udf/drift_search.rs +1058 -0
  401. benodb-0.17.0/src/core/sql/graph_udf/graph_neighbors.rs +193 -0
  402. benodb-0.17.0/src/core/sql/graph_udf/graph_table_functions.rs +1051 -0
  403. benodb-0.17.0/src/core/sql/graph_udf/graph_view.rs +1350 -0
  404. benodb-0.17.0/src/core/sql/graph_udf/jaccard_coefficient.rs +175 -0
  405. benodb-0.17.0/src/core/sql/graph_udf/label_propagation.rs +125 -0
  406. benodb-0.17.0/src/core/sql/graph_udf/leiden_communities.rs +380 -0
  407. benodb-0.17.0/src/core/sql/graph_udf/louvain_communities.rs +332 -0
  408. benodb-0.17.0/src/core/sql/graph_udf/mod.rs +80 -0
  409. benodb-0.17.0/src/core/sql/graph_udf/modularity.rs +332 -0
  410. benodb-0.17.0/src/core/sql/graph_udf/pagerank.rs +289 -0
  411. benodb-0.17.0/src/core/sql/graph_udf/personalized_pagerank.rs +461 -0
  412. benodb-0.17.0/src/core/sql/graph_udf/preferential_attachment.rs +201 -0
  413. benodb-0.17.0/src/core/sql/graph_udf/shortest_path.rs +216 -0
  414. benodb-0.17.0/src/core/sql/graph_udf/strongly_connected_components.rs +257 -0
  415. benodb-0.17.0/src/core/sql/graph_udf/subgraph.rs +562 -0
  416. benodb-0.17.0/src/core/sql/graph_udf/triangle_count.rs +146 -0
  417. benodb-0.17.0/src/core/sql/literal/binary.rs +192 -0
  418. benodb-0.17.0/src/core/sql/literal/dense.rs +92 -0
  419. benodb-0.17.0/src/core/sql/literal/mod.rs +1614 -0
  420. benodb-0.17.0/src/core/sql/literal/sparse.rs +110 -0
  421. benodb-0.17.0/src/core/sql/merge_into.rs +874 -0
  422. benodb-0.17.0/src/core/sql/mod.rs +1152 -0
  423. benodb-0.17.0/src/core/sql/optimizer/config.rs +340 -0
  424. benodb-0.17.0/src/core/sql/optimizer/index_join.rs +90 -0
  425. benodb-0.17.0/src/core/sql/optimizer/vector_search/mod.rs +84 -0
  426. benodb-0.17.0/src/core/sql/optimizer/vector_search/plan_detection.rs +109 -0
  427. benodb-0.17.0/src/core/sql/optimizer/vector_search/plan_rewriter.rs +118 -0
  428. benodb-0.17.0/src/core/sql/optimizer/vector_search/sort_expr_parser.rs +357 -0
  429. benodb-0.17.0/src/core/sql/optimizer.rs +14 -0
  430. benodb-0.17.0/src/core/sql/partition_rewriter.rs +38 -0
  431. benodb-0.17.0/src/core/sql/pgvector_rewriter.rs +258 -0
  432. benodb-0.17.0/src/core/sql/physical_plan/index_join.rs +385 -0
  433. benodb-0.17.0/src/core/sql/physical_plan/vector_merge.rs +230 -0
  434. benodb-0.17.0/src/core/sql/physical_plan/vector_scan.rs +342 -0
  435. benodb-0.17.0/src/core/sql/physical_plan.rs +346 -0
  436. benodb-0.17.0/src/core/sql/session.rs +496 -0
  437. benodb-0.17.0/src/core/sql/subscribe_table_function.rs +222 -0
  438. benodb-0.17.0/src/core/sql/udf/aggregate.rs +501 -0
  439. benodb-0.17.0/src/core/sql/udf/distance.rs +821 -0
  440. benodb-0.17.0/src/core/sql/udf/json.rs +669 -0
  441. benodb-0.17.0/src/core/sql/udf/mod.rs +184 -0
  442. benodb-0.17.0/src/core/sql/udf/sparse.rs +399 -0
  443. benodb-0.17.0/src/core/sql/udf/text.rs +216 -0
  444. benodb-0.17.0/src/core/sql/udf/transform.rs +765 -0
  445. benodb-0.17.0/src/core/sql/udf/vector_stats.rs +646 -0
  446. benodb-0.17.0/src/core/sql/vector_literal.rs +5 -0
  447. benodb-0.17.0/src/core/sql/vector_operators.rs +452 -0
  448. benodb-0.17.0/src/core/sql/vector_udf/aggregate.rs +356 -0
  449. benodb-0.17.0/src/core/sql/vector_udf/distance.rs +522 -0
  450. benodb-0.17.0/src/core/sql/vector_udf/mod.rs +6 -0
  451. benodb-0.17.0/src/core/sql/vector_udf/sparse.rs +242 -0
  452. benodb-0.17.0/src/core/sql/vector_udf/transform.rs +515 -0
  453. benodb-0.17.0/src/core/sql/vector_udf.rs +104 -0
  454. benodb-0.17.0/src/core/storage.rs +593 -0
  455. benodb-0.17.0/src/core/table/builder.rs +585 -0
  456. benodb-0.17.0/src/core/table/catalog.rs +605 -0
  457. benodb-0.17.0/src/core/table/coordinator.rs +336 -0
  458. benodb-0.17.0/src/core/table/detached.rs +745 -0
  459. benodb-0.17.0/src/core/table/fluent.rs +96 -0
  460. benodb-0.17.0/src/core/table/graph.rs +1549 -0
  461. benodb-0.17.0/src/core/table/graph_metadata.rs +337 -0
  462. benodb-0.17.0/src/core/table/index_config.rs +1379 -0
  463. benodb-0.17.0/src/core/table/ingest.rs +902 -0
  464. benodb-0.17.0/src/core/table/maintenance.rs +1752 -0
  465. benodb-0.17.0/src/core/table/merge.rs +169 -0
  466. benodb-0.17.0/src/core/table/mod.rs +945 -0
  467. benodb-0.17.0/src/core/table/preload.rs +1038 -0
  468. benodb-0.17.0/src/core/table/primary_key.rs +876 -0
  469. benodb-0.17.0/src/core/table/properties.rs +592 -0
  470. benodb-0.17.0/src/core/table/read.rs +2323 -0
  471. benodb-0.17.0/src/core/table/schema.rs +369 -0
  472. benodb-0.17.0/src/core/table/stage.rs +552 -0
  473. benodb-0.17.0/src/core/table/state.rs +289 -0
  474. benodb-0.17.0/src/core/table/stats.rs +1166 -0
  475. benodb-0.17.0/src/core/table/subscribe.rs +283 -0
  476. benodb-0.17.0/src/core/table/tests.rs +548 -0
  477. benodb-0.17.0/src/core/table/write.rs +2045 -0
  478. benodb-0.17.0/src/core/telemetry.rs +122 -0
  479. benodb-0.17.0/src/core/wal.rs +1815 -0
  480. benodb-0.17.0/src/lib.rs +450 -0
  481. benodb-0.17.0/src/python/catalog/glue.rs +155 -0
  482. benodb-0.17.0/src/python/catalog/hive.rs +100 -0
  483. benodb-0.17.0/src/python/catalog/jdbc.rs +110 -0
  484. benodb-0.17.0/src/python/catalog/memory.rs +104 -0
  485. benodb-0.17.0/src/python/catalog/mod.rs +241 -0
  486. benodb-0.17.0/src/python/catalog/nessie.rs +108 -0
  487. benodb-0.17.0/src/python/catalog/rest.rs +182 -0
  488. benodb-0.17.0/src/python/catalog/s3tables.rs +152 -0
  489. benodb-0.17.0/src/python/catalog/unity.rs +120 -0
  490. benodb-0.17.0/src/python/drift_bindings.rs +56 -0
  491. benodb-0.17.0/src/python/graph.rs +276 -0
  492. benodb-0.17.0/src/python/helpers.rs +875 -0
  493. benodb-0.17.0/src/python/manifest.rs +62 -0
  494. benodb-0.17.0/src/python/mod.rs +82 -0
  495. benodb-0.17.0/src/python/schema.rs +226 -0
  496. benodb-0.17.0/src/python/session.rs +130 -0
  497. benodb-0.17.0/src/python/stats.rs +103 -0
  498. benodb-0.17.0/src/python/subscribe.rs +119 -0
  499. benodb-0.17.0/src/python/table.rs +3735 -0
  500. benodb-0.17.0/src/python_distance.rs +1684 -0
  501. benodb-0.17.0/src/python_gpu_context.rs +215 -0
  502. benodb-0.17.0/src/telemetry/metrics.rs +345 -0
  503. benodb-0.17.0/src/telemetry/mod.rs +5 -0
  504. benodb-0.17.0/src/telemetry/otlp.rs +310 -0
  505. benodb-0.17.0/src/telemetry/tracing.rs +123 -0
  506. benodb-0.17.0/tests/all_types_index_test.rs +157 -0
  507. benodb-0.17.0/tests/ann_latency_probe.rs +128 -0
  508. benodb-0.17.0/tests/bin/generate_iceberg_manifests.rs +322 -0
  509. benodb-0.17.0/tests/bin/verify_iceberg_read_check.rs +81 -0
  510. benodb-0.17.0/tests/data/start_nessie.sh +7 -0
  511. benodb-0.17.0/tests/datafusion_rust_test.rs +112 -0
  512. benodb-0.17.0/tests/gpu_test_helpers.rs +16 -0
  513. benodb-0.17.0/tests/integration_test_csr_graph.rs +101 -0
  514. benodb-0.17.0/tests/integration_test_graph_mode.rs +272 -0
  515. benodb-0.17.0/tests/integration_test_hnsw_ivf_native.rs +389 -0
  516. benodb-0.17.0/tests/integration_test_hybrid_search_batches.rs +218 -0
  517. benodb-0.17.0/tests/integration_test_iceberg_zero_copy.rs +701 -0
  518. benodb-0.17.0/tests/integration_test_scan_splits.rs +97 -0
  519. benodb-0.17.0/tests/integration_test_sql_multi.rs +100 -0
  520. benodb-0.17.0/tests/oracle_fixture.rs +82 -0
  521. benodb-0.17.0/tests/python/verify_docstrings.py +109 -0
  522. benodb-0.17.0/tests/python/verify_fluent_api.py +167 -0
  523. benodb-0.17.0/tests/python/verify_unified_ingest.py +78 -0
  524. benodb-0.17.0/tests/read_manifest.rs +161 -0
  525. benodb-0.17.0/tests/schema_evolution_test.rs +78 -0
  526. benodb-0.17.0/tests/scratch.rs +22 -0
  527. benodb-0.17.0/tests/soak.rs +358 -0
  528. benodb-0.17.0/tests/stability.rs +70 -0
  529. benodb-0.17.0/tests/stress/README.md +44 -0
  530. benodb-0.17.0/tests/stress/conftest.py +92 -0
  531. benodb-0.17.0/tests/verify_admin_cli.sh +20 -0
  532. benodb-0.17.0/tests/verify_all_algos.py +84 -0
  533. benodb-0.17.0/tests/verify_catalog_commit.rs +84 -0
  534. benodb-0.17.0/tests/verify_compliance.rs +23 -0
  535. benodb-0.17.0/tests/verify_delete_correctness.rs +184 -0
  536. benodb-0.17.0/tests/verify_iceberg_python_delete.sh +90 -0
  537. benodb-0.17.0/tests/verify_iceberg_rest.sh +39 -0
  538. benodb-0.17.0/tests/verify_iceberg_rest_create.sh +52 -0
  539. benodb-0.17.0/tests/verify_iceberg_rest_delete.sh +108 -0
  540. benodb-0.17.0/tests/verify_iceberg_rest_remove_index.sh +135 -0
  541. benodb-0.17.0/tests/verify_iceberg_rest_update.sh +66 -0
  542. benodb-0.17.0/tests/verify_index_recovery.rs +101 -0
  543. benodb-0.17.0/tests/verify_metadata_creation.rs +41 -0
  544. benodb-0.17.0/tests/verify_mor_reads.rs +132 -0
  545. benodb-0.17.0/tests/verify_mor_writes.rs +91 -0
  546. benodb-0.17.0/tests/verify_partition_transforms.rs +81 -0
  547. benodb-0.17.0/tests/verify_partitioned_writes.rs +105 -0
  548. benodb-0.17.0/tests/verify_puffin_index.sh +85 -0
  549. benodb-0.17.0/tests/verify_rest_updates.sh +47 -0
  550. benodb-0.17.0/tests/verify_schema_compat.rs +415 -0
  551. benodb-0.17.0/trino-benodb/README.md +204 -0
  552. benodb-0.17.0/trino-benodb/etc/catalog/benodb.properties +9 -0
  553. benodb-0.17.0/trino-benodb/etc/catalog/glue_catalog.properties +13 -0
  554. benodb-0.17.0/trino-benodb/etc/catalog/iceberg.properties +15 -0
  555. benodb-0.17.0/trino-benodb/etc/catalog/iceberg_hive.properties +11 -0
  556. benodb-0.17.0/trino-benodb/etc/catalog/memory.properties +1 -0
  557. benodb-0.17.0/trino-benodb/etc/catalog/postgres.properties +4 -0
  558. benodb-0.17.0/trino-benodb/etc/config.properties +9 -0
  559. benodb-0.17.0/trino-benodb/etc/entrypoint.sh +45 -0
  560. benodb-0.17.0/trino-benodb/etc/jvm.config +11 -0
  561. benodb-0.17.0/trino-benodb/etc/node.properties +3 -0
  562. benodb-0.17.0/trino-benodb/pom.xml +264 -0
  563. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/AbstractBenoDBMetadata.java +672 -0
  564. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/AbstractBenoDBPageSinkProvider.java +62 -0
  565. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBArrowConverter.java +222 -0
  566. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBColumnHandle.java +85 -0
  567. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBConnectorFactory.java +211 -0
  568. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBInsertTableHandle.java +62 -0
  569. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBJNIBridge.java +149 -0
  570. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBMergeSink.java +186 -0
  571. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBMergeTableHandle.java +63 -0
  572. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBOutputTableHandle.java +62 -0
  573. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBPageSink.java +84 -0
  574. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBPageSource.java +228 -0
  575. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBPageSourceProvider.java +35 -0
  576. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBPlugin.java +24 -0
  577. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBProcedures.java +177 -0
  578. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBScalarFunctions.java +329 -0
  579. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBSplit.java +121 -0
  580. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBSplitManager.java +134 -0
  581. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBTableHandle.java +114 -0
  582. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/BenoDBTableUri.java +82 -0
  583. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/GraphRagSearchFunction.java +238 -0
  584. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/GraphRagSql.java +79 -0
  585. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/GraphSql.java +168 -0
  586. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/GraphTableFunction.java +159 -0
  587. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/GraphTableFunctionHandle.java +66 -0
  588. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/HybridSearchFunction.java +211 -0
  589. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/HybridSearchHandle.java +118 -0
  590. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/KeywordSearchFunction.java +185 -0
  591. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/KeywordSearchHandle.java +94 -0
  592. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/KeywordSearchSql.java +77 -0
  593. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/ListIndexesFunction.java +112 -0
  594. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/ListIndexesPageSource.java +113 -0
  595. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/ListIndexesSplit.java +45 -0
  596. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/ListIndexesTableHandle.java +62 -0
  597. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/RegionalDriftFunction.java +301 -0
  598. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/RegionalDriftPageSource.java +130 -0
  599. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/VectorSearchFunction.java +186 -0
  600. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/VectorSearchHandle.java +98 -0
  601. benodb-0.17.0/trino-benodb/src/main/java/com/benodb/trino/VectorSearchSql.java +142 -0
  602. benodb-0.17.0/trino-benodb/src/main/java-trino-d/com/benodb/trino/BenoDBMetadata.java +40 -0
  603. benodb-0.17.0/trino-benodb/src/main/java-trino-d/com/benodb/trino/BenoDBPageSinkProvider.java +34 -0
  604. benodb-0.17.0/trino-benodb/src/main/java-trino-e/com/benodb/trino/BenoDBMetadata.java +42 -0
  605. benodb-0.17.0/trino-benodb/src/main/java-trino-e/com/benodb/trino/BenoDBPageSinkProvider.java +36 -0
  606. benodb-0.17.0/trino-benodb/src/test/java/com/benodb/trino/KeywordSearchSqlTest.java +87 -0
  607. benodb-0.17.0/trino-benodb/src/test/java/com/benodb/trino/TestBenoDBQueries.java +552 -0
  608. benodb-0.17.0/trino-benodb/src/test/java/com/benodb/trino/TestBenoDBScalarFunctions.java +80 -0
  609. benodb-0.17.0/trino-benodb/src/test/java/com/benodb/trino/TestCrossCatalogJoin.java +124 -0
  610. benodb-0.17.0/trino-benodb/src/test/java/com/benodb/trino/TestFileGranularitySplits.java +91 -0
  611. benodb-0.17.0/trino-benodb/src/test/java/com/benodb/trino/TestGraphSqlAndTableUri.java +52 -0
  612. benodb-0.17.0/trino-benodb/src/test/java/com/benodb/trino/TrinoConnectorTest.java +374 -0
  613. benodb-0.17.0/trino-benodb/src/test/java/com/benodb/trino/VectorSearchSqlTest.java +126 -0
  614. benodb-0.17.0/wix/main.wxs +399 -0
@@ -0,0 +1,3 @@
1
+ [build]
2
+ jobs = 4
3
+
@@ -0,0 +1,17 @@
1
+ target/
2
+ .git/
3
+ .github/
4
+ .venv/
5
+ venv/
6
+ *.db
7
+ *.parquet
8
+ *.log
9
+ test_*
10
+ rag_db/
11
+ news_db/
12
+ demo/
13
+ benchmarks/**/data/
14
+ benchmarks/**/results/
15
+ __pycache__/
16
+ *.pyc
17
+ !target/release/bsdb-search
@@ -0,0 +1,51 @@
1
+ # BenoStreamDB Environment Configuration
2
+ # Copy this file to .env and fill in your credentials.
3
+ # DO NOT commit .env to version control.
4
+
5
+ # -------------------------------------------------------
6
+ # RustFS (S3-compatible object storage)
7
+ # -------------------------------------------------------
8
+ RUSTFS_ROOT_USER=rustfsadmin
9
+ RUSTFS_ROOT_PASSWORD=rustfsadmin
10
+
11
+ # -------------------------------------------------------
12
+ # Nessie PostgreSQL backend
13
+ # -------------------------------------------------------
14
+ NESSIE_POSTGRES_DB=nessie
15
+ NESSIE_POSTGRES_USER=nessie
16
+ NESSIE_POSTGRES_PASSWORD=nessie
17
+
18
+ # -------------------------------------------------------
19
+ # S3 credentials (used by Nessie catalog and Trino)
20
+ # These should match the RustFS root user/password above
21
+ # unless you are using a separate S3-compatible service.
22
+ # -------------------------------------------------------
23
+ S3_ACCESS_KEY=rustfsadmin
24
+ S3_SECRET_KEY=rustfsadmin
25
+ S3_ENDPOINT=http://rustfs:9000
26
+ S3_REGION=us-east-1
27
+
28
+ # -------------------------------------------------------
29
+ # Trino S3 credentials (for iceberg/benostreamdb catalogs)
30
+ # -------------------------------------------------------
31
+ TRINO_S3_ACCESS_KEY=rustfsadmin
32
+ TRINO_S3_SECRET_KEY=rustfsadmin
33
+
34
+ # -------------------------------------------------------
35
+ # Trino PostgreSQL catalog credentials
36
+ # -------------------------------------------------------
37
+ TRINO_PG_CONNECTION_PASSWORD=indie
38
+
39
+ # -------------------------------------------------------
40
+ # External database connection (used by Trino entrypoint)
41
+ # -------------------------------------------------------
42
+ PG_CONN_RW=
43
+
44
+ # -------------------------------------------------------
45
+ # API keys (add as needed for your deployment)
46
+ # -------------------------------------------------------
47
+ # AWS_ACCESS_KEY_ID=
48
+ # AWS_SECRET_ACCESS_KEY=
49
+ # AZURE_STORAGE_ACCOUNT=
50
+ # AZURE_STORAGE_KEY=
51
+ # GCS_SERVICE_ACCOUNT_KEY=
@@ -0,0 +1 @@
1
+ *.so binary=true
@@ -0,0 +1,150 @@
1
+ # ── Rust ────────────────────────────────────────────────────────────────────
2
+ target/
3
+ **/*.rs.bk
4
+
5
+ # ── Python ──────────────────────────────────────────────────────────────────
6
+ __pycache__/
7
+ *.py[cod]
8
+ *$py.class
9
+ .venv/
10
+ .venv_iceberg/
11
+ .venv-cuda-shims/
12
+ venv/
13
+ venv_verify/
14
+ venv_demo/
15
+ test_venv/
16
+ !venv/requirements.txt
17
+ !venv/.gitkeep
18
+ *.egg-info/
19
+
20
+ # Python packaging build output (setuptools / maturin / `python -m build`)
21
+ dist/
22
+ build/
23
+
24
+ # Compiled Python extension (built by `maturin develop`)
25
+ python/**/*.so
26
+ python/**/*.pyd
27
+ python/**/*.dylib
28
+
29
+ # ── IDE / editor / tooling ──────────────────────────────────────────────────
30
+ .vscode/
31
+ .idea/
32
+ .metals/
33
+ .bloop/
34
+ .continue/
35
+ .qwen/
36
+ .agents/
37
+ AGENTS.md
38
+ .clinerules
39
+
40
+ # ── Compiled native binaries / debug symbols ────────────────────────────────
41
+ *.dylib
42
+ *.dSYM/
43
+ *.so
44
+ *.dll
45
+ *.exe
46
+
47
+ # ── BenoStream data artifacts ───────────────────────────────────────────────
48
+ *.parquet
49
+ *.idx
50
+ *.hnsw.graph
51
+ *.hnsw.data
52
+ *.del
53
+ _manifest/
54
+ _metadata/
55
+ test_admin_table/
56
+ test_explain_db/
57
+
58
+ # ── Local toolchain downloads (Maven/JDK fetched by build-connectors.sh) ────
59
+ .java/
60
+ .maven/
61
+
62
+ # ── Local vector DBs, scratch data & generated artifacts ────────────────────
63
+ data/
64
+ docker/data/
65
+ benchmarks/competitive/data/
66
+ hsdbg_bench_data/
67
+ benchmarks/scratch_*/
68
+ # Downloaded Zilliz/HippoRAG comparison corpora (large, reproducible from upstream)
69
+ benchmarks/musique_corpus.json
70
+ benchmarks/hippo_openie.json
71
+ benchmarks/zilliz_manifest.json
72
+ dist_archive/
73
+ dist_streaming/
74
+
75
+ # Pre-release soak gate outputs
76
+ soak-report.md
77
+ soak.log
78
+
79
+ # ── Documentation build output (Sphinx) ─────────────────────────────────────
80
+ docs/build/
81
+
82
+ # ── Build / test logs & scratch ─────────────────────────────────────────────
83
+ # Run logs are written here (see docs/BENCHMARKING.md), never committed.
84
+ logs/
85
+ *.log
86
+ *.pid
87
+ *.patch
88
+ build_errors*.txt
89
+ rust_test_output*.log
90
+ python_test_output*.log
91
+ python_feature_test_output*.log
92
+ test_output*.txt
93
+ *_output.txt
94
+
95
+ # ── Test caches ─────────────────────────────────────────────────────────────
96
+ .hypothesis/
97
+ .pytest_cache/
98
+ .ipynb_checkpoints/
99
+
100
+ # ── Local secrets — never commit ────────────────────────────────────────────
101
+ .streamlit/secrets.toml
102
+ .streamlit/config.toml
103
+ .env
104
+ .env.local
105
+ groq_api_key.txt
106
+
107
+ # ── Ad-hoc local scripts & scratch (not part of the repo) ───────────────────
108
+ scratch*.rs
109
+ debug_read*.py
110
+ patch_*.py
111
+ patch_*.sh
112
+ fix_*.py
113
+ run_*.sh
114
+ # Exceptions: these are benchmark fixtures, not ad-hoc scratch scripts. The bench
115
+ # image COPYs run_bench.sh, and the docs/quick-starts invoke the other three, so a
116
+ # clean clone could not build bsdb-bench-runner or follow the guide without them.
117
+ !benchmarks/competitors/run_bench.sh
118
+ !benchmarks/run_musique_docker.sh
119
+ !benchmarks/run_musique_entrypoint.sh
120
+ !benchmarks/run_neurips_docker.sh
121
+ bench_*.py
122
+
123
+ # ── Private / commercial planning (not part of the public repo) ─────────────
124
+ business_plan/
125
+ benolabs/
126
+ target-jni/
127
+
128
+ # ── Unintended large files / test artifacts ────────────────────────
129
+ test_db/
130
+ test_ingestion_db/
131
+ news_db/
132
+ scifact_news_db/
133
+ connector-artifacts/
134
+ test_c
135
+ test_prefetch
136
+ scratch_mem_test
137
+
138
+ # Blog article drafts (unpublished)
139
+ blog/articles/drafts/
140
+ # Quarantined pre-sweep benchmark artifacts (see benchmarks/run_sweep.py).
141
+ benchmarks/results/stale/
142
+ # Per-section quarantine archives of superseded competitor runs. They are the
143
+ # provenance trail for a report, not an input to it: the sweep reads
144
+ # `results/*.json` and the generator never looks inside `stale/`.
145
+ benchmarks/competitors/results/stale/
146
+
147
+ # Generated by scripts/prepare_docs.sh for the Sphinx build.
148
+ /docs/source/guides/
149
+ /docs/source/integrations/
150
+ /docs/source/tutorials/
@@ -0,0 +1,67 @@
1
+ # BenoStreamDB Development Guidelines
2
+
3
+ **Foundation:** These guidelines are rooted in [The Pragmatic Programmer](https://pragprog.com/) principles.
4
+
5
+ ## Core BenoStreamDB Principles
6
+
7
+ ### Code Quality
8
+ - **Always attempt to fix warnings if possible** - Don't ignore compiler warnings; resolve them during implementation (Aligns with: Don't Live with Broken Windows #4)
9
+ - Use proper error handling instead of silencing warnings
10
+ - Maintain clean build output (zero clippy warnings)
11
+ - Type safety is paramount
12
+ - Refactor early and often as code is written
13
+
14
+ ### Versioning & Releases
15
+ - Version numbers are injected from git tags during build
16
+ - `Cargo.toml` and `pyproject.toml` use dynamic versioning from `build.rs`
17
+ - Tag format: `v0.X.Y` (e.g., `v0.1.2`)
18
+ - All releases should be tagged and pushed with comprehensive release notes
19
+
20
+ ### Licensing & Attribution
21
+ - Apache 2.0 / MIT dual license
22
+ - Properly attribute optimizations borrowed from iceberg-rust or other Apache-licensed projects
23
+ - Maintain THIRDPARTY_NOTICES.md with all third-party code usage
24
+ - Include inline code comments for borrowed patterns
25
+
26
+ ### API Design
27
+ - Use native Rust names (PascalCase for types: `Cosine`, `L2`, `InnerProduct`)
28
+ - Python API uses lowercase for user-facing functions: `metric="cosine"`
29
+ - Support both primary parameter names and backward-compatible aliases
30
+ - Document all vector distance functions and index parameters
31
+
32
+ ### Testing & Validation
33
+ - Unit tests required for new optimizer rules and query optimizations (Aligns with: Design to Test #20, Test Your Software #21)
34
+ - Explain() output should show correct index usage and query strategy
35
+ - Test suite must pass before any release tagging (Aligns with: Make Quality a Requirements Issue #7)
36
+ - Performance benchmarks are part of testing, not optional extras
37
+ - Integration tests for query planning changes
38
+ - Run full test suite before tagging releases
39
+ - Verify explain() output shows correct index detection
40
+
41
+ ### Packaging
42
+ - Exclude build artifacts, virtual environments from Cargo.toml (Aligns with: Make It Easy to Reuse #11)
43
+ - Ensure PyPI builds succeed without external dependencies
44
+ - Test `maturin develop --release` before release tagging
45
+ - Dynamic versioning from git tags (build.rs injection)
46
+
47
+ ## Current Focus Areas
48
+
49
+ ### Hybrid Query Optimization
50
+ - Support multiple vector columns with ordering
51
+ - Compound primary key detection for automatic hybrid search
52
+ - Vector search always runs after scalar filters (push-down)
53
+ - Proper explain() output showing both scalar and vector execution
54
+ - Use tracer bullets: Implement end-to-end before perfecting details (Aligns with: Use Tracer Bullets #14)
55
+
56
+ ### Iceberg-Rust Optimizations
57
+ - LIMIT pushdown to file scanning layer
58
+ - Row group skipping based on column statistics
59
+ - Manifest metadata caching for performance
60
+ - Predicate pushdown improvements
61
+ - Properly attribute all borrowed code patterns (Aligns with: Communicate! #9, Make It Easy to Reuse #11)
62
+
63
+ ### Observability & Debugging
64
+ - Structured tracing via `tracing` crate (no println!) (Aligns with: Program Close to Problem Domain #16)
65
+ - Proper log levels (debug, info, warn, error)
66
+ - Explain query plans for user visibility (Aligns with: Use Tracer Bullets #14)
67
+ - Document query strategy through explain() output
benodb-0.17.0/.mailmap ADDED
@@ -0,0 +1,6 @@
1
+ # Normalize every historical author/committer to the project owner.
2
+ Rick Albright <rla3rd@gmail.com> <rick@hyperstreamdb.com>
3
+ Rick Albright <rla3rd@gmail.com> <ralbright@verityplatform.com>
4
+ Rick Albright <rla3rd@gmail.com> <ralbright@google.com>
5
+ Rick Albright <rla3rd@gmail.com> <rla3rd@users.noreply.github.com>
6
+ Rick Albright <rla3rd@gmail.com> <rla3rd@gmail.com>
@@ -0,0 +1,117 @@
1
+ # BenoStreamDB Agent Steering & Release Guidelines
2
+
3
+ All AI agents, pairing assistants, and contributors operating in this repository must strictly adhere to the following release and verification engineering rules.
4
+
5
+ ---
6
+
7
+ ## 1. Mandatory Pre-Release Steering Gate
8
+
9
+ **NEVER tag, release, or push commits to GitHub without executing the local pre-release suite:**
10
+
11
+ ```bash
12
+ ./scripts/pre_release_check.sh
13
+ ```
14
+
15
+ This script mirrors all required gates from `.github/workflows/ci.yml` and `.github/workflows/release.yml`:
16
+ 1. `cargo fmt --check` (Rust code style & formatting)
17
+ 2. `cargo clippy --all-targets` (Clippy zero-warning check under Rust 1.98.0)
18
+ 3. `cargo check --all-targets` & `cargo check --bins` (Compilation of library, tests, and binaries)
19
+ 4. `cargo doc --no-deps --features java` (Documentation link integrity)
20
+ 5. `version-check` (Single-source version assertion across all connectors, POMs, and pyproject files)
21
+ 6. Python regression tests (`tests/python/test_vector_string_conformance.py`)
22
+
23
+ If any check fails, **stop immediately and fix the underlying issue locally**. Never push code expecting CI to catch it.
24
+
25
+ ---
26
+
27
+ ## 2. Git Pre-Push Hook
28
+
29
+ The repository provides a tracked git pre-push hook in `scripts/git-hooks/pre-push`.
30
+ Install it in any local checkout:
31
+
32
+ ```bash
33
+ ./scripts/install_hooks.sh
34
+ ```
35
+
36
+ The hook automatically intercepts `git push` (for both branches and tags) and blocks the push if `./scripts/pre_release_check.sh` fails. **Never bypass this hook using `--no-verify`.**
37
+
38
+ ---
39
+
40
+ ## 3. Rust Formatting & Lints
41
+
42
+ - Always run `cargo fmt` after editing any Rust file in `src/`, `server/`, or `contrib/`.
43
+ - Ensure Rust code compiles cleanly with Clippy on toolchain `1.98.0` (`rustup run 1.98.0 cargo clippy --all-targets`).
44
+
45
+ ---
46
+
47
+ ## 4. Single Source of Truth for Versions
48
+
49
+ The single authoritative version is defined in:
50
+ ```toml
51
+ # Cargo.toml
52
+ [workspace.package]
53
+ version = "X.Y.Z"
54
+ ```
55
+
56
+ When bumping versions for any release:
57
+ 1. Update `[workspace.package].version` in `Cargo.toml`.
58
+ 2. Update inter-crate dependency pins in:
59
+ - `server/flight_sql/Cargo.toml` (`benodb = { version = "X.Y.Z", ... }`)
60
+ - `contrib/benodb-search/Cargo.toml` (`benodb = { version = "X.Y.Z", ... }`)
61
+ 3. Update connector metadata:
62
+ - `spark-benodb/pom.xml` (`<revision>X.Y.Z</revision>`)
63
+ - `trino-benodb/pom.xml` (`<revision>X.Y.Z</revision>`)
64
+ - `dbt-benodb/pyproject.toml` (`benodb>=X.Y.Z`)
65
+ - `dbt-benodb/dbt/include/benodb/dbt_project.yml` (`version: X.Y.Z`)
66
+ 4. Verify dynamic versioning in all Python packages (`pyproject.toml`, `server/flight_sql/pyproject.toml`, `contrib/benodb-search/pyproject.toml`, `contrib/benodb-mcp/pyproject.toml`).
67
+ 5. Run `./scripts/pre_release_check.sh` to assert full synchronization.
68
+
69
+ ---
70
+
71
+ ## 5. In-Process SQL & Registration
72
+
73
+ - `Table.sql(query)` binds the table directly to the default alias `'t'` (`SELECT ... FROM t`).
74
+ - To query tables by named identifiers (`SELECT ... FROM events`), register the table in an in-process `Session`:
75
+ ```python
76
+ session = bdb.Session()
77
+ session.register("events", table)
78
+ df = session.sql("SELECT * FROM events ...")
79
+ ```
80
+
81
+ ---
82
+
83
+ ## 6. Authorial Voice for Documentation, Papers, and Release Collateral
84
+
85
+ Every word of prose committed to this repository follows the authorial voice of Richard Albright: the README, `docs/`, `CHANGELOG.md`, blog posts, benchmark reports, academic papers, and release notes. The authoritative guide is `../benolabs/.agents/skills/authorial-voice/SKILL.md` and the always-on steering rule is `../benolabs/.agents/rules/authorial_voice.md`. When those files and this summary disagree, read the guide, then ask.
86
+
87
+ ### 6.1 Core Pillars
88
+
89
+ 1. **Dual audience clarity.** Assume two readers at once: an engineer who will run the code, and a technical decision maker who signs the infrastructure bill. Open each major section with a plain statement of what happens and why it matters in operational terms (cost, memory ceilings, cluster count, vendor freedom), then follow immediately with the schema, the query, the benchmark controls, or the math.
90
+ 2. **Empirical, never promotional.** Every claim carries a number, a mechanism, or a physical constraint. If a sentence could survive with the numbers deleted, it is an advertisement and does not belong.
91
+ 3. **Systems grounded mechanics.** Name the actual components: Apache Iceberg, Apache Parquet, Puffin sidecars, Apache DataFusion, Arrow `RowSelection`, AVX2 kernels, cgroup limits. "The storage layer" is a failure to look.
92
+ 4. **Honest about trade offs, negative results, and boundaries.** State what the system does not do, where performance inverts, and which scale is out of scope. A reader trusts a paper that concedes a loss more than one that reports only wins.
93
+ 5. **Technical marketing woven throughout.** Never confine persuasion to the top of a document. Put it in section transitions, architecture comparisons, and benchmark callouts, and make it argue through architectural contrast rather than adjectives.
94
+
95
+ ### 6.2 Punctuation and Mechanical Rules
96
+
97
+ - **No em dashes.** Do not join words with `---` in LaTeX or an em dash in prose. Split the sentence, use a colon, use parentheses, or use a contrastive connective. The construction reads as machine written.
98
+ - **No hyphenated compound modifiers between words.** Write `two stage`, `high dimensional`, `in process`, `single node`, `read only`, `build time`, `float point` compounds as separate words or restructure the sentence. Numeric and named compounds keep their hyphens (`32-KB`, `128-D`, `512-MiB`, `Walsh-Hadamard`, `Johnson-Lindenstrauss`, `deletion-vector-v1`, `--strict-hardware`). The tell the reader notices is not the hyphen itself, it is the pile of coined compounds standing in for a verb.
99
+ - **No tilde approximations.** Do not write `~GB`, `~1.3 MB`, or `$\sim 40$`. Print the figure the harness produced, state the bound, or describe the order of magnitude in words. The LaTeX tie in `512~MiB` is spacing and stays.
100
+ - **Define every acronym at first use.** Expand it in the abstract and again in the body before the abbreviation appears: Fast Walsh-Hadamard Transform (FWHT), maximum inner product search (MIPS), 8 bit scalar quantization (SQ8), Hierarchical Navigable Small World (HNSW), Product Quantization (PQ). A reader who meets `FWHT` cold stops reading.
101
+ - **One name per concept.** Do not alternate between `TQ8` and `BQ8` for the same quantizer, or between `OOM` and `out of memory` inside one section.
102
+ - **Active voice, finding first.** `We ingest the corpus and partition chunks by section boundaries`, not `The corpus is ingested and chunks are partitioned`. State the result, then the mechanism or measurement that supports it.
103
+
104
+ ### 6.3 Forbidden Vocabulary and Replacements
105
+
106
+ | Avoid | Use |
107
+ |---|---|
108
+ | "seamless", "effortless", "magical" | Zero copy Arrow record batching; `pip install benodb` with no external service |
109
+ | "blazing fast", "revolutionary", "game changing" | Sub millisecond retrieval; AVX2 SIMD distance kernels at 1,700 QPS on eight cores |
110
+ | "massive I/O reduction" | 32 KB Parquet pages with Arrow `RowSelection` cutting rerank I/O from 512 MB to 1.3 MB |
111
+ | "next generation database" | Vector and graph engine over Apache Iceberg Puffin sidecars |
112
+ | "best in class vector search" | HNSW with BenoQuant TQ8 isotropic scalar quantization |
113
+ | "massive cost savings" | Eliminates the three system tax and the multi hop network round trips |
114
+
115
+ ### 6.4 Scope Discipline for Release Text
116
+
117
+ State what a release does and does not contain before describing what it improves. Name the boundary explicitly, for example that the engine is single node with no distributed consensus, no cross-table multi statement transactions, and no automated failover, and that 100M vectors is the empirical single node boundary at a 4.8 times working set to RAM ratio. A preview note that hides its own scope is worse than no note.
Binary file