motedb-python 0.12.8__tar.gz → 0.13.0__tar.gz

{motedb_python-0.12.8 → motedb_python-0.13.0}/.gitignore

@@ -33,3 +33,4 @@ benchmark_results/
 __pycache__/
 *.pyc
 bindings/python/dist/
+_native.abi3.so

{motedb_python-0.12.8 → motedb_python-0.13.0}/CHANGELOG.md

@@ -1,5 +1,104 @@
 # Changelog
 
+## [0.13.0] — 2026-10-11
+
+- **🔑 BLOB 类型(全链路)**: `payload BLOB` 列 — 第 8 个 ColumnType。
+  - 字面量: SQLite 兼容 `X'0aff00'`(奇数位/非 hex 报错); Python `bytes`/
+    `bytearray` 参数与取回; CLI 十六进制预览; 16 MiB 上限
+  - 存储: 复用 Text 段字节布局 + 独立 `ColumnTypeTag::Blob`(读侧按 schema
+    包装 `Value::Blob`, 非 UTF-8 字节零损耗); row_format 变长编解码;
+    非定长批/缓存/点查/扫描/缓冲各读路径补齐
+  - 语义: 字节等值(`=`/`IS NULL`)、`LENGTH(blob)`=字节数(SQLite 语义)
+  - **已知缺口(v0.13.0, 测试内注释标注)**: BLOB 列上的标量函数在部分
+    执行器路径(含 VECTOR 混合表/`WHERE` 内函数)与 `ORDER BY` 输出下返回
+    NULL — 裸投影/等值/时间窗扫描/参数全稳
+- **🔑 MCAP 机器人数据导入(具身工作流入场券)**:
+  - `motedb-cli import mcap <db> <file> [--include s] [--exclude s] [--prefix p]`
+    与 Python `db.import_mcap(path, include=…, prefix=…, decode={topic: fn})`
+  - 每 topic 一表: `_uid INTEGER PRIMARY KEY`((channel<<40)|seq, 天然去重)、
+    `log_time/publish_time TIMESTAMP`(自动时间索引, ns→µs)、`data BLOB`
+    (原始消息字节)。解码边界在 Python 钩子(返回 dict → 类型化列, 可产
+    VECTOR 列) — 引擎不碰 ROS2 CDR/protobuf 语义
+  - 重放幂等: `INSERT OR IGNORE` + `_uid` 去重(kill -9 后重跑即续)
+  - 依赖: 官方 `mcap` crate 0.25(只读, 经 cargo-deny)
+- **🔑 词法平方级修复(BLOB 测试实抓)**: `Lexer::current_utf8_char` 每字符
+  对剩余输入跑 `str::from_utf8` 全量校验 — 任何 16 MiB 字符串/X'' 字面量
+  解析耗时 ~1h; 改为仅解码首码点(decode_first_codepoint), 行为不变
+- **🔑 RRF 混合检索 + bm25()/snippet()**:
+  - `FROM RRF((子查询1), (子查询2)[, k = n][, rrk = n][, w1..w4 = f])` 表
+    函数: 2..=4 路排名列表的 Reciprocal Rank Fusion( score = Σ wᵢ/(rrk+rankᵢ),
+    默认 k=10/rrk=60/均权), 输出按分数降序、值升序 tie-break(确定性可对拍),
+    经 `rows_to_union_statement` 内联为派生表 → 外层 WHERE/ORDER BY 走常规机制
+  - `bm25(col)` 作为 `BM25_SCORE` 的别名接入 MATCH 快路径: `ORDER BY
+    bm25(body)` 与 `ORDER BY score DESC` 等价, LIMIT 下取 ranked top-k
+  - `snippet(text, query[, width=64])`: FTS5 风格摘录(首命中窗口 +
+    `<b>` 高亮 + 省略号), 纯字符索引实现
+  - **顺带修复**: FROM 为非普通表(JOIN/派生表/TableFn)且 ORDER BY+LIMIT
+    的查询被 optimizer 的 "unknown" 占位表名绊倒(FullScan 计划臂现按
+    FROM 形状回退通用机制)
+  - 测试: test_rrf_hybrid.rs(3) — 引擎融合 vs 测试内参照公式逐 id 对拍、
+    命名参数(w1=0 退化为单路)、形状错误、bm25 排序等价、snippet 窗口/高亮
+- 文档对齐: README SQL Support 勘误(WITH RECURSIVE/窗口函数 0.12 已实现
+  但写着不支持)+ BLOB/MCAP 能力与已知缺口如实标注
+- 测试: test_blob_type.rs(4)、test_mcap_import.rs(合成 bag 回环: 布局/
+  字节保真/钩子/过滤/重放幂等/重开)、bindings/python/tests/
+  test_mcap_import.py(固定 fixture bag 端到端)、examples/gen_test_bag.rs
+- 顺延(下版): RRF 混合检索 + bm25()/snippet()(设计已定: docs/design/
+  2026-10-embodied-roadmap.md); MCAP 断点续导清单表(`_mcap_imports`)
+
+## [0.12.9] — 2026-10-10
+
+- **🔑 崩溃恢复审计三连修(0.12.8 stale 检测的边界补全)**:
+  - **重建不清空旧索引 → 统计膨胀**: 恢复重建在已加载的(可能滞后的)
+    索引上**追加**而非重置 —— `batch_insert` 不幂等(total_docs 无条件
+    自增、`PostingList::add` 对已有 doc 抬 TF), 一次重建即 total_docs
+    翻倍 + 词频加倍(BM25 静默漂移, MATCH 成员正确所以压测没抓到)。
+    修复: 重建前硬重置(删除 text_{name}.fts.d 与 dict.d 目录、新建空
+    索引, 与 load 路径同构), 重建后 total_docs 精确等于活行探针实测
+    80/80。回归: tests/test_fts_recovery_staleness.rs(UPDATE/DELETE/
+    NULL 混合 + 标记契约 + 恢复幂等)
+  - **NULL 文本行 → 每次全量重建**: NULL 不入索引是常态, 但 0.12.8 的
+    `indexed < table_rows` 谓词把含 NULL 的表判为永久 stale → 每次
+    open 全量重建(且触发上面的追加膨胀)。修复: 引入**干净关闭标记**
+    `fts_indexes.fresh` —— Drop 时 flush 全部文本索引(有界锁等待,
+    builder 卡死则跳过标记而非卡死 Drop)并原子写标记; open 起始消费
+    标记, 标记在位 → 干净重开零校验零重建, 标记缺失(崩溃证据)→ 才
+    跑计数校验。干净重开实测零重建
+  - **墓碑丢失方向 → MATCH 幻影**: crash 前 DELETE 的内存 tombstone
+    未落盘 → 磁盘索引 total_docs **大于**表行数, 旧 `<` 谓词不触发
+    重建 → posting 残留已删行, `COUNT(*) WHERE MATCH` 快路径按索引
+    计数会多计。修复: 谓词改 `indexed != table_rows`(双向); 实证中
+    干净关闭/WAL 回放通常兜底此窗口, `!=` 为纵深防御。回归:
+    test_crash_recovery.py 扩展奇数轮 kill@DELCKPT(删除后 checkpoint
+    即杀), 断言 MATCH 与活行精确一致
+- 详查结论: UPDATE 后多 segment 无同 key 双活(重建计数精确); 文本
+  索引磁盘产物全部位于 fts.d/dict.d 两目录(重置完整)
+- **WAL 截断点 durability 对齐(纵深防御)**: checkpoint_impl 与 vacuum
+  在 `wal.checkpoint_all()` 截断前新增有界(2s try-write)FTS flush。
+  原理: UPDATE 的 term 级墓碑(deleted_term_docs)纯内存, 计数谓词
+  (total_docs 净变化为零)天然看不见内容级漂移; 截断 WAL 会移除本可
+  回放重放的索引变更。实测该窗口被 WAL 回放兜住(不 flush 崩溃结果
+  也正确), 此为原理性对齐而非实测缺陷修复; 锁竞争时跳过(不劣于
+  修复前)
+- **🔑 在线 backup 竞态修复(全套件并行实抓, 上项改动的连带暴露)**:
+  backup 拷贝与"已 in-flight 的段交换"竞态时 NotFound 只重试 3×50ms
+  — 全套件并行负载下逃逸(backup_to 报 Io NotFound)。且 auto-checkpoint
+  持 is_flushing 的临界区因上面的 FTS flush 变长, backup 的 CAS 冲突
+  从"立即报错"概率上升。修复: 拷贝重试 3×50ms → 10×100ms; is_flushing
+  CAS 冲突内部重试 ~2.5s 后才上抛 "retry backup"。因果验证: 去 FTS
+  flush 的全套件 exit 0, 带上后 2/2 失败, 加宽重试后收敛
+- **标记契约加固(二轮自查)**:
+  - **重置顺序(Windows 关键)**: `TextFTSIndex::new` 是 create-or-OPEN,
+    会加载残留文件; 旧索引对象持着 fts.d 内文件句柄时 Windows 删除目录
+    因共享冲突失败 → new() 会加载旧状态、追加膨胀复活。修复: 先从 map
+    移除旧条目(释放句柄)再删目录再新建(macOS 上不可测, 代码序保证)
+  - **盖章归属与僵尸句柄**: close() 释放 flock 后对象可存活到 GC ——
+    僵尸 Drop 在新实例 open 之后再盖章 = 给新会话中途状态谎报"干净"。
+    修复: close() 在**持锁期间**(release_lock 之前)盖章(Python 主路径
+    即时生效, 不依赖 GC); Drop 路径仅在仍持锁(从未 close)时盖章
+  - **标记持久化**: fsync 文件 + rename + unix 目录 fsync, 标记严格
+    排在被 flush 的索引数据之后(断电不出现"标记在、数据不在")
+
 ## [0.12.8] — 2026-10-09
 
 - **🔑 kill -9 崩溃后全文索引静默滞后修复(PyPI 包压测实抓)**: 崩溃发生在

{motedb_python-0.12.8 → motedb_python-0.13.0}/Cargo.lock

@@ -100,12 +100,24 @@ dependencies = [
  "rustversion",
 ]
 
+[[package]]
+name = "array-init"
+version = "2.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3d62b7694a562cdf5a74227903507c56ab2cc8bdd1f781ed5cb4cf9c9f810bfc"
+
 [[package]]
 name = "autocfg"
 version = "1.5.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
 
+[[package]]
+name = "bimap"
+version = "0.6.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "230c5f1ca6a325a32553f8640d31ac9b49f2411e901e427570154868b46da4f7"
+
 [[package]]
 name = "bincode"
 version = "1.3.3"
@@ -115,6 +127,30 @@ dependencies = [
  "serde",
 ]
 
+[[package]]
+name = "binrw"
+version = "0.15.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6ad120d555272286c1017d25165ab8bd74806f13fc85b258484ec7e4ce75458f"
+dependencies = [
+ "array-init",
+ "binrw_derive",
+ "bytemuck",
+]
+
+[[package]]
+name = "binrw_derive"
+version = "0.15.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6df92e0e9baae4dc82c7bad7715ca40c0a5c71539057bf2ea04a5c29c980410b"
+dependencies = [
+ "either",
+ "owo-colors",
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
 [[package]]
 name = "bitflags"
 version = "1.3.2"
@@ -327,6 +363,40 @@ version = "0.2.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5"
 
+[[package]]
+name = "darling"
+version = "0.21.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9cdf337090841a411e2a7f3deb9187445851f91b309c0c0a29e05f74a00a48c0"
+dependencies = [
+ "darling_core",
+ "darling_macro",
+]
+
+[[package]]
+name = "darling_core"
+version = "0.21.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1247195ecd7e3c85f83c8d2a366e4210d588e802133e1e355180a9870b517ea4"
+dependencies = [
+ "fnv",
+ "ident_case",
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
+[[package]]
+name = "darling_macro"
+version = "0.21.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d38308df82d1080de0afee5d069fa14b0326a88c14f15c5ccda35b4a6c414c81"
+dependencies = [
+ "darling_core",
+ "quote",
+ "syn",
+]
+
 [[package]]
 name = "dary_heap"
 version = "0.3.9"
@@ -384,6 +454,27 @@ version = "1.16.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e"
 
+[[package]]
+name = "enumset"
+version = "1.1.14"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ccc5801fd11762e24d1e420d01d2ac518f2a2ca4329d4fbb6639f2412b6204e0"
+dependencies = [
+ "enumset_derive",
+]
+
+[[package]]
+name = "enumset_derive"
+version = "0.15.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4bd536557b58c682b217b8fb199afdff47cd3eff260623f19e77074eb073d63a"
+dependencies = [
+ "darling",
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
 [[package]]
 name = "env_filter"
 version = "2.0.0"
@@ -447,6 +538,12 @@ version = "0.1.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
 
+[[package]]
+name = "fnv"
+version = "1.0.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
+
 [[package]]
 name = "foldhash"
 version = "0.2.0"
@@ -571,6 +668,12 @@ version = "0.5.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c"
 
+[[package]]
+name = "ident_case"
+version = "1.0.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39"
+
 [[package]]
 name = "include-flate"
 version = "0.3.3"
@@ -783,6 +886,24 @@ dependencies = [
  "hashbrown 0.17.1",
 ]
 
+[[package]]
+name = "mcap"
+version = "0.25.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b215e79631de2a19ed76fda13cd52950c3057b487a2f41d9dc85c4079b969c40"
+dependencies = [
+ "bimap",
+ "binrw",
+ "byteorder",
+ "crc32fast",
+ "enumset",
+ "log",
+ "num_cpus",
+ "paste",
+ "static_assertions",
+ "thiserror 2.0.18",
+]
+
 [[package]]
 name = "memchr"
 version = "2.8.2"
@@ -800,7 +921,7 @@ dependencies = [
 
 [[package]]
 name = "motedb"
-version = "0.12.8"
+version = "0.13.0"
 dependencies = [
  "arc-swap",
  "bincode",
@@ -813,6 +934,7 @@ dependencies = [
  "libc",
  "log",
  "lru",
+ "mcap",
  "memmap2",
  "parking_lot",
  "phf",
@@ -848,6 +970,16 @@ dependencies = [
  "autocfg",
 ]
 
+[[package]]
+name = "num_cpus"
+version = "1.17.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "91df4bbde75afed763b708b7eee1e8e7651e02d97f6d5dd763e89367e957b23b"
+dependencies = [
+ "hermit-abi",
+ "libc",
+]
+
 [[package]]
 name = "once_cell"
 version = "1.21.4"
@@ -866,6 +998,12 @@ version = "11.1.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "d6790f58c7ff633d8771f42965289203411a5e5c68388703c06e14f24770b41e"
 
+[[package]]
+name = "owo-colors"
+version = "4.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "13c45bb4a6ae1280ec0803b1ef9d3455eb50f01efbbe1447ab020f1d54fba9d8"
+
 [[package]]
 name = "parking_lot"
 version = "0.12.5"
@@ -1282,6 +1420,12 @@ version = "1.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "1b6b67fb9a61334225b5b790716f609cd58395f895b3fe8b328786812a40bc3b"
 
+[[package]]
+name = "static_assertions"
+version = "1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f"
+
 [[package]]
 name = "strsim"
 version = "0.11.1"

{motedb_python-0.12.8 → motedb_python-0.13.0}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "motedb"
-version = "0.12.8"
+version = "0.13.0"
 edition = "2021"
 rust-version = "1.87"
 description = "AI-native embedded multimodal database for embodied intelligence (robots, AR glasses, industrial arms)."
@@ -41,6 +41,7 @@ name = "motedb-cli"
 path = "src/bin/motedb-cli.rs"
 
 [dependencies]
+mcap = { version = "0.25.0", default-features = false }
 # Memory-mapped file I/O (used by: spatial_hybrid, diskann/sst)
 memmap2 = "0.9"
 

{motedb_python-0.12.8 → motedb_python-0.13.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: motedb-python
-Version: 0.12.8
+Version: 0.13.0
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research

{motedb_python-0.12.8 → motedb_python-0.13.0}/README.md

@@ -160,6 +160,32 @@ See [`examples/logging.rs`](examples/logging.rs) for a runnable demo.
 
 - 🔍 **运维自检** — `db.doctor()` / `motedb-cli doctor <path>`：表布局、内存预算、索引覆盖率、磁盘占用逐项 PASS/WARN 报告。
 
+### Hybrid search (new in v0.13.1)
+
+```sql
+SELECT id FROM RRF(
+  (SELECT id FROM docs WHERE MATCH(body, 'grasp failure') ORDER BY bm25(body) LIMIT 100),
+  (SELECT id FROM docs ORDER BY emb <-> ? LIMIT 100)
+);
+```
+
+One-SQL BM25 ⊕ vector fusion (Reciprocal Rank Fusion, `k`/`rrk`/`w1..w4`
+tunable). `snippet(body, 'grasp failure', 80)` renders FTS5-style excerpts
+with `<b>` highlights.
+
+### Robotics data on-ramp (new in v0.13)
+
+```bash
+motedb import mcap robot.mote recording.mcap        # CLI
+# 或 Python:
+db.import_mcap("recording.mcap", decode={"/joint_states": my_decoder})
+```
+
+One table per topic (`_uid` PK, `log_time`/`publish_time` TIMESTAMP + auto
+index, raw message `data BLOB`); a per-topic Python decode hook adds typed
+columns (e.g. decoded `emb VECTOR`) at import time. Re-imports are no-ops
+(`_uid` dedup) — crash-safe resumes.
+
 ### Multimodal
 
 | Modality | Index | Query |
@@ -253,8 +279,11 @@ EXISTS]` / `DROP INDEX`, `ALTER TABLE` (`ADD COLUMN`, `AUTO_INCREMENT = N`),
 - Aggregates: `COUNT`, `SUM`, `AVG`, `MIN`, `MAX`, `STDDEV`, `VARIANCE`
 - `UNION` / `UNION ALL`
 - `CASE WHEN ... THEN ... ELSE ... END`
-- `WITH` / Common Table Expressions (non-recursive; `WITH name [(cols)] AS
-  (SELECT ...), ... <main query>`)
+- `WITH` / Common Table Expressions incl. **`WITH RECURSIVE`** (semi-naive
+  fixed-point, UNION ALL bodies, chained CTEs)
+- Window functions: `ROW_NUMBER`, `RANK`, `DENSE_RANK`, `NTILE`, `LAG`,
+  `LEAD`, `FIRST_VALUE`, `LAST_VALUE` + aggregates `OVER (PARTITION BY …
+  ORDER BY …)` (RANGE peer-group default frame)
 - Scalar functions: `UPPER`/`LOWER`/`LENGTH`/`TRIM`/`SUBSTR`/`REPLACE`/`CONCAT` (skips NULL args, SQLite/Postgres semantics; `||` propagates NULL)/`INSTR`/`COALESCE`/`ROUND` (binary-exact decimal rounding)/`ABS`/arithmetic
 - Multimodal predicates: `MATCH(col) AGAINST('q')` (BM25 ranked FTS;
   multi-word default is AND, explicit `a OR b` unions — FTS5-compatible),
@@ -263,9 +292,15 @@ EXISTS]` / `DROP INDEX`, `ALTER TABLE` (`ADD COLUMN`, `AUTO_INCREMENT = N`),
 - Transactions: `BEGIN` / `COMMIT` / `ROLLBACK`, savepoints, read-your-writes
   visibility inside a transaction
 
-**Not yet supported:** `WITH RECURSIVE` (the keyword is accepted but
-self-referencing CTEs error out), `DECIMAL`/`DATE`/`BLOB` types, window
-functions, and cross-statement server-side cursors.
+**BLOB type:** `payload BLOB` columns with SQLite-style `X'0aff00'` hex
+literals and Python `bytes` parameters (byte-exact round-trip, 16 MiB
+cap). Known v0.13 gaps: scalar functions over BLOB columns
+(`LENGTH(payload)`) and BLOB output under `ORDER BY` are not yet wired
+through all executor paths — bare projection, equality (`WHERE payload =
+X'…'`), time-window scans and parameters are solid.
+
+**Not yet supported:** `DECIMAL`/`DATE` types, cross-statement server-side
+cursors, and explicit window frames (`ROWS`/`RANGE` specs).
 
 ## Documentation