motedb-python 0.12.9__tar.gz → 0.14.0__tar.gz

{motedb_python-0.12.9 → motedb_python-0.14.0}/CHANGELOG.md

@@ -1,5 +1,98 @@
 # Changelog
 
+## [0.14.0] — 2026-10-12
+
+- **🔑 ASOF JOIN(W8, v0.14)**: 机器人帧↔状态对齐原语
+  - `t1 ASOF JOIN t2 ON key = key AND right_ts <= left_ts`(也接受
+    `left_ts >= right_ts` 等价形)— 每左行取同键下 ts 不超过它的最新右行
+  - `LEFT ASOF JOIN` 保留无匹配左行(右侧 NULL);可选键等值(缺省=全局桶)
+  - 实现: 右表按 (key → ts 升序) 建桶, 左行二分 partition_point, O(N·logM)
+  - 基建修复: **隐式别名吞 ASOF**(`fr ASOF JOIN jt` 曾解析为
+    fr{alias:"ASOF"} + Inner JOIN → 静默 0 行); normalize 操作数交换时
+    镜像不等号; 前向形式(left_ts <= right_ts)v1 明确报错
+  - 测试: test_asof_join.rs(4) — 对齐/左保留/反向形式与错误/30×40 抖动
+    时间戳 vs 测试内暴力参照逐行对拍
+- **Perf Gate 新形状**: +blob_scan(0.30x)/rrf_fusion(0.04x)/
+  traj_sim_topk(1.79x) 三个 v0.14 查询形状的比率预算(基线=全表扫描)
+- **🔑 数据集快照(W7, v0.14)**:
+  - `COMMIT DATASET 'name' [NOTE '…']` — 钉住全部用户表的当前段集
+    (段 = 不可变文件,快照即段 id 清单,近乎免费);清单持久化在
+    `_dataset_commits` 表(dataset/table/seg_id/note/committed_at)
+  - **GC 引用保护**: ColSegmentStore 增加 gc_filter(三个构造点统一安装:
+    get_or_create Vacant/Occupied 补装/open 直接构造点)—— sync_manifest
+    对被钉住的段**延迟物理删除**(留在 pending_gc 反复复查);reopen 时
+    非 active 段的清理同样改为延迟(修复:原先直接删除会吃掉被钉段,
+    kill 测试实抓)
+  - `EXPORT DATASET 'name' TO 'path'` — 把快照物化为独立新库(清单段:
+    内存句柄或按 `{id:010}.sst` 直开 Segment::open;逐活行写入新库,
+    同 schema 含 EPISODE BY;索引在目标库按需重建);导出末尾 checkpoint
+    保证即时持久化与一致首读
+  - `DROP DATASET [IF EXISTS] 'name'` — 释放钉(原生 delete_rows_batch,
+    引用集全量重建)
+  - 测试: test_datasets.rs(2) — 快照隔离(commit 前后变更不串)、
+    reopen+VACUUM 后导出仍完整(GC 保护实抓:open 路径绕过 filter 安装)、
+    DROP/IF EXISTS/未知数据集错误
+- **🔑 Episode 数据模型(W6, v0.14 第一块)**:
+  - `CREATE EPISODE TABLE … EPISODE BY (key_col, ts_col) GAP '5s'|'100ms'|
+    microseconds` — 列存在性校验, 配置持久化在 TableSchema(serde 兼容旧库)
+  - `FROM EPISODES(t)` 表函数: 按 key 分组 → ts 排序 → gap 分片, 每片段一行
+    (`<key>`, `t0`, `t1`, `n_rows`), 且每个 VECTOR 列派生 `traj_<col>` —
+    统一 64 采样线性插值重采样的展平轨迹(64×dim)。Episode 为**派生数据**
+    (分片是 key/ts/gap 的纯函数) — 无注册表需持久化, 重开一致性免费
+  - `traj_sim(traj_col, demo)` — 64×dim DTW 距离(demo 任意采样数, 内部
+    重采样到 64); W5 spike 决议的两段式架构(无物理倒排, PAA 短名单化
+    为后续优化), evaluator/positional/simple 三求值表接线
+  - 基建: 表函数参数支持裸表名 + 字符串字面量; UNION 合成支持 Vector
+    字面量(EPISODES 内联通道, 递归 CTE 路径保持标量限制)
+  - 测试: test_episodes.rs(3) — 分片正确性/形状、traj_sim vs 测试内
+    暴力 DTW 对拍、重开耐久
+
+## [0.13.0] — 2026-10-11
+
+- **🔑 BLOB 类型(全链路)**: `payload BLOB` 列 — 第 8 个 ColumnType。
+  - 字面量: SQLite 兼容 `X'0aff00'`(奇数位/非 hex 报错); Python `bytes`/
+    `bytearray` 参数与取回; CLI 十六进制预览; 16 MiB 上限
+  - 存储: 复用 Text 段字节布局 + 独立 `ColumnTypeTag::Blob`(读侧按 schema
+    包装 `Value::Blob`, 非 UTF-8 字节零损耗); row_format 变长编解码;
+    非定长批/缓存/点查/扫描/缓冲各读路径补齐
+  - 语义: 字节等值(`=`/`IS NULL`)、`LENGTH(blob)`=字节数(SQLite 语义)
+  - **已知缺口(v0.13.0, 测试内注释标注)**: BLOB 列上的标量函数在部分
+    执行器路径(含 VECTOR 混合表/`WHERE` 内函数)与 `ORDER BY` 输出下返回
+    NULL — 裸投影/等值/时间窗扫描/参数全稳
+- **🔑 MCAP 机器人数据导入(具身工作流入场券)**:
+  - `motedb-cli import mcap <db> <file> [--include s] [--exclude s] [--prefix p]`
+    与 Python `db.import_mcap(path, include=…, prefix=…, decode={topic: fn})`
+  - 每 topic 一表: `_uid INTEGER PRIMARY KEY`((channel<<40)|seq, 天然去重)、
+    `log_time/publish_time TIMESTAMP`(自动时间索引, ns→µs)、`data BLOB`
+    (原始消息字节)。解码边界在 Python 钩子(返回 dict → 类型化列, 可产
+    VECTOR 列) — 引擎不碰 ROS2 CDR/protobuf 语义
+  - 重放幂等: `INSERT OR IGNORE` + `_uid` 去重(kill -9 后重跑即续)
+  - 依赖: 官方 `mcap` crate 0.25(只读, 经 cargo-deny)
+- **🔑 词法平方级修复(BLOB 测试实抓)**: `Lexer::current_utf8_char` 每字符
+  对剩余输入跑 `str::from_utf8` 全量校验 — 任何 16 MiB 字符串/X'' 字面量
+  解析耗时 ~1h; 改为仅解码首码点(decode_first_codepoint), 行为不变
+- **🔑 RRF 混合检索 + bm25()/snippet()**:
+  - `FROM RRF((子查询1), (子查询2)[, k = n][, rrk = n][, w1..w4 = f])` 表
+    函数: 2..=4 路排名列表的 Reciprocal Rank Fusion( score = Σ wᵢ/(rrk+rankᵢ),
+    默认 k=10/rrk=60/均权), 输出按分数降序、值升序 tie-break(确定性可对拍),
+    经 `rows_to_union_statement` 内联为派生表 → 外层 WHERE/ORDER BY 走常规机制
+  - `bm25(col)` 作为 `BM25_SCORE` 的别名接入 MATCH 快路径: `ORDER BY
+    bm25(body)` 与 `ORDER BY score DESC` 等价, LIMIT 下取 ranked top-k
+  - `snippet(text, query[, width=64])`: FTS5 风格摘录(首命中窗口 +
+    `<b>` 高亮 + 省略号), 纯字符索引实现
+  - **顺带修复**: FROM 为非普通表(JOIN/派生表/TableFn)且 ORDER BY+LIMIT
+    的查询被 optimizer 的 "unknown" 占位表名绊倒(FullScan 计划臂现按
+    FROM 形状回退通用机制)
+  - 测试: test_rrf_hybrid.rs(3) — 引擎融合 vs 测试内参照公式逐 id 对拍、
+    命名参数(w1=0 退化为单路)、形状错误、bm25 排序等价、snippet 窗口/高亮
+- 文档对齐: README SQL Support 勘误(WITH RECURSIVE/窗口函数 0.12 已实现
+  但写着不支持)+ BLOB/MCAP 能力与已知缺口如实标注
+- 测试: test_blob_type.rs(4)、test_mcap_import.rs(合成 bag 回环: 布局/
+  字节保真/钩子/过滤/重放幂等/重开)、bindings/python/tests/
+  test_mcap_import.py(固定 fixture bag 端到端)、examples/gen_test_bag.rs
+- 顺延(下版): RRF 混合检索 + bm25()/snippet()(设计已定: docs/design/
+  2026-10-embodied-roadmap.md); MCAP 断点续导清单表(`_mcap_imports`)
+
 ## [0.12.9] — 2026-10-10
 
 - **🔑 崩溃恢复审计三连修(0.12.8 stale 检测的边界补全)**:

{motedb_python-0.12.9 → motedb_python-0.14.0}/Cargo.lock

@@ -100,12 +100,24 @@ dependencies = [
  "rustversion",
 ]
 
+[[package]]
+name = "array-init"
+version = "2.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3d62b7694a562cdf5a74227903507c56ab2cc8bdd1f781ed5cb4cf9c9f810bfc"
+
 [[package]]
 name = "autocfg"
 version = "1.5.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
 
+[[package]]
+name = "bimap"
+version = "0.6.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "230c5f1ca6a325a32553f8640d31ac9b49f2411e901e427570154868b46da4f7"
+
 [[package]]
 name = "bincode"
 version = "1.3.3"
@@ -115,6 +127,30 @@ dependencies = [
  "serde",
 ]
 
+[[package]]
+name = "binrw"
+version = "0.15.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6ad120d555272286c1017d25165ab8bd74806f13fc85b258484ec7e4ce75458f"
+dependencies = [
+ "array-init",
+ "binrw_derive",
+ "bytemuck",
+]
+
+[[package]]
+name = "binrw_derive"
+version = "0.15.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6df92e0e9baae4dc82c7bad7715ca40c0a5c71539057bf2ea04a5c29c980410b"
+dependencies = [
+ "either",
+ "owo-colors",
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
 [[package]]
 name = "bitflags"
 version = "1.3.2"
@@ -327,6 +363,40 @@ version = "0.2.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5"
 
+[[package]]
+name = "darling"
+version = "0.21.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9cdf337090841a411e2a7f3deb9187445851f91b309c0c0a29e05f74a00a48c0"
+dependencies = [
+ "darling_core",
+ "darling_macro",
+]
+
+[[package]]
+name = "darling_core"
+version = "0.21.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1247195ecd7e3c85f83c8d2a366e4210d588e802133e1e355180a9870b517ea4"
+dependencies = [
+ "fnv",
+ "ident_case",
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
+[[package]]
+name = "darling_macro"
+version = "0.21.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d38308df82d1080de0afee5d069fa14b0326a88c14f15c5ccda35b4a6c414c81"
+dependencies = [
+ "darling_core",
+ "quote",
+ "syn",
+]
+
 [[package]]
 name = "dary_heap"
 version = "0.3.9"
@@ -384,6 +454,27 @@ version = "1.16.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e"
 
+[[package]]
+name = "enumset"
+version = "1.1.14"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ccc5801fd11762e24d1e420d01d2ac518f2a2ca4329d4fbb6639f2412b6204e0"
+dependencies = [
+ "enumset_derive",
+]
+
+[[package]]
+name = "enumset_derive"
+version = "0.15.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4bd536557b58c682b217b8fb199afdff47cd3eff260623f19e77074eb073d63a"
+dependencies = [
+ "darling",
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
 [[package]]
 name = "env_filter"
 version = "2.0.0"
@@ -447,6 +538,12 @@ version = "0.1.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
 
+[[package]]
+name = "fnv"
+version = "1.0.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
+
 [[package]]
 name = "foldhash"
 version = "0.2.0"
@@ -571,6 +668,12 @@ version = "0.5.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c"
 
+[[package]]
+name = "ident_case"
+version = "1.0.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39"
+
 [[package]]
 name = "include-flate"
 version = "0.3.3"
@@ -783,6 +886,24 @@ dependencies = [
  "hashbrown 0.17.1",
 ]
 
+[[package]]
+name = "mcap"
+version = "0.25.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b215e79631de2a19ed76fda13cd52950c3057b487a2f41d9dc85c4079b969c40"
+dependencies = [
+ "bimap",
+ "binrw",
+ "byteorder",
+ "crc32fast",
+ "enumset",
+ "log",
+ "num_cpus",
+ "paste",
+ "static_assertions",
+ "thiserror 2.0.18",
+]
+
 [[package]]
 name = "memchr"
 version = "2.8.2"
@@ -800,7 +921,7 @@ dependencies = [
 
 [[package]]
 name = "motedb"
-version = "0.12.9"
+version = "0.14.0"
 dependencies = [
  "arc-swap",
  "bincode",
@@ -813,6 +934,7 @@ dependencies = [
  "libc",
  "log",
  "lru",
+ "mcap",
  "memmap2",
  "parking_lot",
  "phf",
@@ -848,6 +970,16 @@ dependencies = [
  "autocfg",
 ]
 
+[[package]]
+name = "num_cpus"
+version = "1.17.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "91df4bbde75afed763b708b7eee1e8e7651e02d97f6d5dd763e89367e957b23b"
+dependencies = [
+ "hermit-abi",
+ "libc",
+]
+
 [[package]]
 name = "once_cell"
 version = "1.21.4"
@@ -866,6 +998,12 @@ version = "11.1.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "d6790f58c7ff633d8771f42965289203411a5e5c68388703c06e14f24770b41e"
 
+[[package]]
+name = "owo-colors"
+version = "4.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "13c45bb4a6ae1280ec0803b1ef9d3455eb50f01efbbe1447ab020f1d54fba9d8"
+
 [[package]]
 name = "parking_lot"
 version = "0.12.5"
@@ -1282,6 +1420,12 @@ version = "1.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "1b6b67fb9a61334225b5b790716f609cd58395f895b3fe8b328786812a40bc3b"
 
+[[package]]
+name = "static_assertions"
+version = "1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f"
+
 [[package]]
 name = "strsim"
 version = "0.11.1"

{motedb_python-0.12.9 → motedb_python-0.14.0}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "motedb"
-version = "0.12.9"
+version = "0.14.0"
 edition = "2021"
 rust-version = "1.87"
 description = "AI-native embedded multimodal database for embodied intelligence (robots, AR glasses, industrial arms)."
@@ -41,6 +41,7 @@ name = "motedb-cli"
 path = "src/bin/motedb-cli.rs"
 
 [dependencies]
+mcap = { version = "0.25.0", default-features = false }
 # Memory-mapped file I/O (used by: spatial_hybrid, diskann/sst)
 memmap2 = "0.9"
 

{motedb_python-0.12.9 → motedb_python-0.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: motedb-python
-Version: 0.12.9
+Version: 0.14.0
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research

{motedb_python-0.12.9 → motedb_python-0.14.0}/README.md

@@ -160,6 +160,76 @@ See [`examples/logging.rs`](examples/logging.rs) for a runnable demo.
 
 - 🔍 **运维自检** — `db.doctor()` / `motedb-cli doctor <path>`：表布局、内存预算、索引覆盖率、磁盘占用逐项 PASS/WARN 报告。
 
+### Hybrid search (new in v0.13.1)
+
+```sql
+SELECT id FROM RRF(
+  (SELECT id FROM docs WHERE MATCH(body, 'grasp failure') ORDER BY bm25(body) LIMIT 100),
+  (SELECT id FROM docs ORDER BY emb <-> ? LIMIT 100)
+);
+```
+
+One-SQL BM25 ⊕ vector fusion (Reciprocal Rank Fusion, `k`/`rrk`/`w1..w4`
+tunable). `snippet(body, 'grasp failure', 80)` renders FTS5-style excerpts
+with `<b>` highlights.
+
+### Frame↔state alignment: ASOF JOIN (v0.14 work-in-progress)
+
+```sql
+-- For each frame, the latest joint-state at-or-before its timestamp (per session):
+SELECT f.fname, j.q FROM frames f
+ASOF JOIN joints j ON f.session = j.session AND j.ts <= f.ts;
+-- LEFT ASOF JOIN keeps frames with no prior state (NULL right side).
+```
+
+### Episode model + trajectory search (v0.14 work-in-progress)
+
+```sql
+CREATE EPISODE TABLE teleop (
+  session TEXT, ts TIMESTAMP, joint VECTOR(7), gripper FLOAT
+) EPISODE BY (session, ts) GAP '5s';
+
+-- One row per episode: bounds, row count, and a 64-sample resampled
+-- trajectory per VECTOR column (flattened 64×dim).
+SELECT session, t0, t1, n_rows FROM EPISODES(teleop);
+
+-- Trajectory similarity (DTW over the resampled series; two-stage
+-- PAA-shortlist design validated in docs/design/2026-10-traj-spike.md):
+SELECT session, t0 FROM EPISODES(teleop)
+ORDER BY traj_sim(traj_joint, [0.0, 0.1, …]) LIMIT 10;
+```
+
+Episodes are **derived data** — segmentation is a pure function of
+`(key, ts, gap)`, so there is no registry to persist or invalidate and
+reopen consistency is free.
+
+### Dataset snapshots (v0.14 work-in-progress)
+
+```sql
+COMMIT DATASET 'fixpos-v3' NOTE 'after gripper fix';
+-- … table keeps evolving (compaction GC will not touch pinned segments) …
+EXPORT DATASET 'fixpos-v3' TO '/data/fixpos-v3.mote';  -- materialized copy
+DROP DATASET 'fixpos-v3';                              -- release the pin
+```
+
+A snapshot is just the pinned segment list (segments are immutable files)
+— near-free to take, persisted in `_dataset_commits`, and compaction
+physically defers deletion of pinned files until the pin is dropped.
+EXPORT materializes the pinned state into a fresh standalone database.
+
+### Robotics data on-ramp (new in v0.13)
+
+```bash
+motedb import mcap robot.mote recording.mcap        # CLI
+# 或 Python:
+db.import_mcap("recording.mcap", decode={"/joint_states": my_decoder})
+```
+
+One table per topic (`_uid` PK, `log_time`/`publish_time` TIMESTAMP + auto
+index, raw message `data BLOB`); a per-topic Python decode hook adds typed
+columns (e.g. decoded `emb VECTOR`) at import time. Re-imports are no-ops
+(`_uid` dedup) — crash-safe resumes.
+
 ### Multimodal
 
 | Modality | Index | Query |
@@ -253,8 +323,11 @@ EXISTS]` / `DROP INDEX`, `ALTER TABLE` (`ADD COLUMN`, `AUTO_INCREMENT = N`),
 - Aggregates: `COUNT`, `SUM`, `AVG`, `MIN`, `MAX`, `STDDEV`, `VARIANCE`
 - `UNION` / `UNION ALL`
 - `CASE WHEN ... THEN ... ELSE ... END`
-- `WITH` / Common Table Expressions (non-recursive; `WITH name [(cols)] AS
-  (SELECT ...), ... <main query>`)
+- `WITH` / Common Table Expressions incl. **`WITH RECURSIVE`** (semi-naive
+  fixed-point, UNION ALL bodies, chained CTEs)
+- Window functions: `ROW_NUMBER`, `RANK`, `DENSE_RANK`, `NTILE`, `LAG`,
+  `LEAD`, `FIRST_VALUE`, `LAST_VALUE` + aggregates `OVER (PARTITION BY …
+  ORDER BY …)` (RANGE peer-group default frame)
 - Scalar functions: `UPPER`/`LOWER`/`LENGTH`/`TRIM`/`SUBSTR`/`REPLACE`/`CONCAT` (skips NULL args, SQLite/Postgres semantics; `||` propagates NULL)/`INSTR`/`COALESCE`/`ROUND` (binary-exact decimal rounding)/`ABS`/arithmetic
 - Multimodal predicates: `MATCH(col) AGAINST('q')` (BM25 ranked FTS;
   multi-word default is AND, explicit `a OR b` unions — FTS5-compatible),
@@ -263,9 +336,15 @@ EXISTS]` / `DROP INDEX`, `ALTER TABLE` (`ADD COLUMN`, `AUTO_INCREMENT = N`),
 - Transactions: `BEGIN` / `COMMIT` / `ROLLBACK`, savepoints, read-your-writes
   visibility inside a transaction
 
-**Not yet supported:** `WITH RECURSIVE` (the keyword is accepted but
-self-referencing CTEs error out), `DECIMAL`/`DATE`/`BLOB` types, window
-functions, and cross-statement server-side cursors.
+**BLOB type:** `payload BLOB` columns with SQLite-style `X'0aff00'` hex
+literals and Python `bytes` parameters (byte-exact round-trip, 16 MiB
+cap). Known v0.13 gaps: scalar functions over BLOB columns
+(`LENGTH(payload)`) and BLOB output under `ORDER BY` are not yet wired
+through all executor paths — bare projection, equality (`WHERE payload =
+X'…'`), time-window scans and parameters are solid.
+
+**Not yet supported:** `DECIMAL`/`DATE` types, cross-statement server-side
+cursors, and explicit window frames (`ROWS`/`RANGE` specs).
 
 ## Documentation