@1-/scan 0.1.3 → 0.1.5

package/README.md

@@ -5,16 +5,16 @@
 <a id="en"></a>
 # @1-/scan : Incrementally scan directory files and track metadata in SQLite
 
-Incrementally scans directory files, compares file sizes and modification times to detect changes, synchronizes metadata to an SQLite database, and returns updated relative paths.
+Incrementally scans directory files, compares file sizes and modification times to detect changes, synchronizes metadata to SQLite database, and returns updated relative paths.
 
 ## Features
 
-- **Incremental Scanning**: Detects and processes only new, modified, or deleted files, avoiding redundant file system operations.
+- **Incremental Scanning**: Processes only new, modified, or deleted files, avoiding redundant file system operations.
 - **Key Optimization**: Stores relative paths within 16 bytes directly as raw bytes; hashes longer paths to 16-byte MD5 digests to optimize database index space and query performance.
 - **Metadata Compression**: Compresses file sizes and modification times using Varint (variable-length byte) encoding.
-- **Transactional Integrity**: Packages updates and deletions in a single database transaction to guarantee consistency.
-- **Flexible Filtering**: Supports custom ignore callback functions to filter specific files and directories.
-- **Native Database**: Integrates Bun's native `bun:sqlite` module, eliminating external database driver dependencies.
+- **Transactional Integrity**: Packages updates and deletions in database transactions to guarantee consistency.
+- **File Filtering**: Supports custom ignore callback functions to filter files and directories.
+- **Native Database**: Integrates Bun native `bun:sqlite` module, eliminating external database driver dependencies.
 
 ## Usage
 
@@ -24,29 +24,65 @@ Incrementally scans directory files, compares file sizes and modification times
 import scan from "@1-/scan";
 
 const dir = "./data";
-const dbPath = "./scan_record.db";
+const db_path = "./scan_record.db";
 
-// Scan directory and sync metadata to SQLite, returning modified relative paths
-const updatedPaths = await scan(dir, dbPath);
-console.log("Updated files:", updatedPaths);
+// Scan directory and sync metadata to SQLite, returning modified relative paths and upsert function
+const [updated_paths, upsert] = await scan(dir, db_path);
+
+// Auto-close database when exiting scope
+using _upsert = upsert;
+
+console.log("Updated files:", updated_paths);
+
+// Update scanned file metadata in database
+for (const rel_path of updated_paths) {
+  await upsert(rel_path);
+}
 ```
 
 ### Scan with Ignore Filter
 
 ```javascript
-import { FILE } from "@1-/walk";
 import scan from "@1-/scan";
 
 const dir = "./data";
-const dbPath = "./scan_record.db";
+const db_path = "./scan_record.db";
 
 // Ignore temporary files and specific configurations
-const ignore = (kind, relPath) => {
-  return relPath.startsWith("temp/") || relPath === "config.json";
+const ignore = (kind, rel_path) => {
+  return rel_path.startsWith("temp/") || rel_path === "config.json";
 };
 
-const updatedPaths = await scan(dir, dbPath, ignore);
-console.log("Synced. Updated files:", updatedPaths);
+const [updated_paths, upsert] = await scan(dir, db_path, ignore);
+using _upsert = upsert;
+
+console.log("Synced. Updated files:", updated_paths);
+
+for (const rel_path of updated_paths) {
+  await upsert(rel_path);
+}
+```
+
+### Bulk Storage Module Usage
+
+```javascript
+import save from "@1-/scan/save.js";
+import sqlite from "@1-/scan/sqlite.js";
+
+const db = sqlite("./scan_record.db");
+
+// Bulk update and delete metadata
+save(
+  db,
+  [
+    ["file.txt", new Uint8Array([1, 2, 3]), 123, 1620000000]
+  ],
+  [
+    new Uint8Array([4, 5, 6])
+  ]
+);
+
+db.close();
 ```
 
 ## Design Ideas
@@ -55,24 +91,25 @@ The main entry orchestrates independent modules to execute the incremental scann
 
 ```mermaid
 graph TD
-    Entry["_.js (Entry Point)"] -->|1. Initialize Connection| Sqlite[sqlite.js]
-    Entry -->|2. Load Existing Records| Load[load.js]
-    Entry -->|3. Walk & Compare Files| DirWalk[dirWalk.js]
+    Entry["_.js (Entry Point)"] -->|1. Initialize Connection| Sqlite["sqlite.js"]
+    Entry -->|2. Load Existing Records| Load["load.js"]
+    Entry -->|3. Walk & Compare Files| DirWalk["dirWalk.js"]
     DirWalk -->|Invoke| Walk["@1-/walk/walkRelIgnore"]
-    DirWalk -->|Process Path Keys| Hash[hash.js]
-    Entry -->|4. Persist Changes| Save[save.js]
-    Save -->|Transaction Wrapper| Trans[trans.js]
+    DirWalk -->|Process Path Keys| Hash["hash.js"]
+    Entry -->|4. Delete Absent & Return Upsert| Trans["trans.js"]
+    Save["save.js (Independent Sync Helper)"] -->|Transaction Wrapper| Trans
 ```
 
-1. **Initialize Connection (`sqlite.js`)**: Opens the SQLite database connection and configures automatic connection disposal.
-2. **Load Records (`load.js`)**: Automatically creates the schema if missing, retrieves existing file hashes, sizes, and modification times, and reconstructs the reference set in memory.
-3. **Walk & Compare (`dirWalk.js`)**: Traverses the directory structure recursively. Paths are transformed into 16-byte keys via `hash.js`. File attributes are encoded using `@3-/vb` and compared against database records to identify additions and modifications.
-4. **Persist Changes (`save.js`)**: Executes bulk inserts and deletions in a single transaction via `trans.js` to update database state.
+1. **Initialize Connection (`sqlite.js`)**: Opens SQLite database connection and configures automatic connection disposal.
+2. **Load Records (`load.js`)**: Automatically creates `scanMtimeLen` table if missing, retrieves existing file hashes, sizes, and modification times, and reconstructs reference set in memory.
+3. **Walk & Compare (`dirWalk.js`)**: Traverses directory structure recursively. Paths are transformed into 16-byte keys via `hash.js`. File attributes are encoded using `@3-/vb` and compared against database records to identify additions and modifications.
+4. **Delete & Return Upsert**: Uses `trans.js` to execute transaction-safe deletions for deleted files, and returns modified relative paths and an `upsert` function so that caller can update database records.
+5. **Independent Sync Helper (`save.js`)**: Exported independent module to execute bulk updates and deletions in transactions.
 
 ## Tech Stack
 
 - **Bun**: Runtime environment and test framework.
-- **Bun SQLite**: Native high-performance SQLite engine built into Bun.
+- **Bun SQLite**: Native SQLite engine built into Bun.
 - **@1-/walk**: Directory walker with ignore support.
 - **@3-/vb**: Variable-length byte (Varint) encoder and decoder.
 - **@3-/binmap / @3-/binset**: Memory-efficient collections designed for binary keys.
@@ -82,14 +119,14 @@ graph TD
 ```
 .
 ├── src
-│   ├── _.js          # Entry point coordinating scanning and synchronization
+│   ├── _.js          # Entry point coordinating scanning and returning upsert helper
 │   ├── dirWalk.js    # Directory traverser comparing file metadata
 │   ├── hash.js       # Hashing helper mapping paths to 16-byte keys
 │   ├── load.js       # Database loader initializing schema and loading records
-│   ├── save.js       # Writer executing bulk updates and deletions
+│   ├── save.js       # Independent helper executing bulk updates and deletions
 │   ├── sqlite.js     # Connection manager instantiating SQLite database
 │   └── trans.js      # Transaction wrapper providing rollback mechanism
-└── tests             # Test suites
+└── tests             # Test directory
 ```
 
 ## History
@@ -104,16 +141,16 @@ To conserve disk space and reduce I/O overhead, SQLite utilizes Varint (variable
 <a id="zh"></a>
 # @1-/scan : 增量扫描目录文件并使用 SQLite 记录元数据
 
-增量扫描目录文件，通过比对文件大小和修改时间检测变更，并同步至 SQLite 数据库中，最终返回有更新的相对路径列表。
+增量扫描目录文件，比对大小与修改时间以检测变更，同步元数据至 SQLite 数据库，返回发生变更之相对路径列表。
 
 ## 功能介绍
 
-- **增量扫描**：仅处理新增、修改或删除的文件，避免冗余的文件系统读写，提升同步速度。
-- **路径压缩**：当相对路径长度小于等于 16 字节时保留原始字节；超出 16 字节则转换为 16 字节 MD5 值作为数据库主键，优化索引空间与查询性能。
-- **元数据压缩**：使用 Varint（可变字节整型）编码方式压缩存储文件大小和修改时间。
-- **事务安全**：将更新与删除操作合并在单个数据库事务中执行，确保数据一致性。
-- **灵活过滤**：支持通过自定义回调函数过滤指定类型的文件与目录。
-- **原生依赖**：基于 Bun 内置 `bun:sqlite` 模块，无需额外安装或编译数据库驱动。
+- **增量扫描**：处理新增、修改或删除之文件，避免冗余文件系统读写，提升同步效率。
+- **路径压缩**：相对路径长度不大于 16 字节时保留原始字节；超出 16 字节则转换为 16 字节 MD5 值作为主键，优化索引空间与查询性能。
+- **元数据压缩**：使用 Varint（可变字节整型）编码方式压缩存储文件大小与修改时间。
+- **事务安全**：将更新与删除操作合并在数据库事务中执行，确保数据一致性。
+- **文件过滤**：支持自定义过滤函数以排除特定文件与目录。
+- **原生依赖**：基于 Bun 内置 `bun:sqlite` 模块，免去安装与编译数据库驱动步骤。
 
 ## 使用演示
 
@@ -123,55 +160,92 @@ To conserve disk space and reduce I/O overhead, SQLite utilizes Varint (variable
 import scan from "@1-/scan";
 
 const dir = "./data";
-const dbPath = "./scan_record.db";
+const db_path = "./scan_record.db";
 
-// 扫描目录并同步至 SQLite，返回发生变更的相对路径列表
-const updatedPaths = await scan(dir, dbPath);
-console.log("更新文件列表：", updatedPaths);
+// 扫描目录并同步至 SQLite，返回发生变更之相对路径列表与更新函数
+const [updated_paths, upsert] = await scan(dir, db_path);
+
+// 退出作用域时自动关闭数据库
+using _upsert = upsert;
+
+console.log("更新文件列表：", updated_paths);
+
+// 更新已处理文件元数据至数据库
+for (const rel_path of updated_paths) {
+  await upsert(rel_path);
+}
 ```
 
-### 带有忽略规则的扫描
+### 过滤规则扫描
 
 ```javascript
-import { FILE } from "@1-/walk";
 import scan from "@1-/scan";
 
 const dir = "./data";
-const dbPath = "./scan_record.db";
+const db_path = "./scan_record.db";
 
-// 忽略特定文件或目录
-const ignore = (kind, relPath) => {
-  return relPath.startsWith("temp/") || relPath === "config.json";
+// 过滤临时文件与特定配置
+const ignore = (kind, rel_path) => {
+  return rel_path.startsWith("temp/") || rel_path === "config.json";
 };
 
-const updatedPaths = await scan(dir, dbPath, ignore);
-console.log("已同步，更新列表：", updatedPaths);
+const [updated_paths, upsert] = await scan(dir, db_path, ignore);
+using _upsert = upsert;
+
+console.log("已同步，更新列表：", updated_paths);
+
+for (const rel_path of updated_paths) {
+  await upsert(rel_path);
+}
+```
+
+### 批量存储模块使用
+
+```javascript
+import save from "@1-/scan/save.js";
+import sqlite from "@1-/scan/sqlite.js";
+
+const db = sqlite("./scan_record.db");
+
+// 批量更新与删除元数据
+save(
+  db,
+  [
+    ["file.txt", new Uint8Array([1, 2, 3]), 123, 1620000000]
+  ],
+  [
+    new Uint8Array([4, 5, 6])
+  ]
+);
+
+db.close();
 ```
 
 ## 设计思路
 
-系统主入口调用各个独立模块完成增量扫描与数据同步流程。
+系统主入口调度各独立模块完成增量扫描与数据同步。
 
 ```mermaid
 graph TD
-    Entry["_.js (主入口)"] -->|1. 初始化连接| Sqlite[sqlite.js]
-    Entry -->|2. 加载已有记录| Load[load.js]
-    Entry -->|3. 扫描文件系统并对比| DirWalk[dirWalk.js]
+    Entry["_.js (主入口)"] -->|1. 初始化连接| Sqlite["sqlite.js"]
+    Entry -->|2. 加载已有记录| Load["load.js"]
+    Entry -->|3. 扫描文件系统并对比| DirWalk["dirWalk.js"]
     DirWalk -->|调用| Walk["@1-/walk/walkRelIgnore"]
-    DirWalk -->|处理路径键| Hash[hash.js]
-    Entry -->|4. 持久化数据变更| Save[save.js]
-    Save -->|事务保障| Trans[trans.js]
+    DirWalk -->|处理路径键| Hash["hash.js"]
+    Entry -->|4. 删除失效记录并返回更新函数| Trans["trans.js"]
+    Save["save.js (独立批量存储模块)"] -->|事务保障| Trans
 ```
 
 1. **初始化连接 (`sqlite.js`)**：打开 SQLite 数据库，并配置自动释放连接机制。
-2. **加载记录 (`load.js`)**：若表不存在则自动创建，读取已记录的文件哈希、大小及修改时间，在内存中还原比对集合。
+2. **加载记录 (`load.js`)**：若数据表 `scanMtimeLen` 不存在则自动创建，读取已记录的文件哈希、大小及修改时间，在内存中还原比对集合。
 3. **文件系统扫描 (`dirWalk.js`)**：递归遍历目录，利用 `hash.js` 将路径映射为 16 字节键。对比当前文件与数据库元数据（利用 `@3-/vb` 进行压缩状态对比），筛选出新增和修改的文件。
-4. **数据存储 (`save.js`)**：使用 `trans.js` 开启事务，将需要删除的无效记录及需要更新的元数据批量写入 SQLite 数据库。
+4. **删除与返回更新函数**：使用 `trans.js` 开启事务，批量删除已被移除的记录，并返回变更的相对路径列表与 `upsert` 函数，供调用者持久化数据。
+5. **独立批量存储模块 (`save.js`)**：供外部调用的独立工具模块，用于在事务中批量写入与删除。
 
 ## 技术栈
 
-- **Bun**：JavaScript 运行时及测试框架。
-- **Bun SQLite**：内置的轻量级、高性能 SQLite 实现。
+- **Bun**：JavaScript 运行时与测试框架。
+- **Bun SQLite**：内置 SQLite 实现。
 - **@1-/walk**：支持过滤规则的目录递归遍历工具。
 - **@3-/vb**：Varint（可变字节）编码与解码器。
 - **@3-/binmap / @3-/binset**：针对二进制键优化的 Map 和 Set 容器。
@@ -181,19 +255,19 @@ graph TD
 ```
 .
 ├── src
-│   ├── _.js          # 核心流程控制器，调度各模块完成增量同步
+│   ├── _.js          # 核心流程控制器，调度各模块并返回变更及更新函数
 │   ├── dirWalk.js    # 遍历目录并比对元数据，输出变更队列
 │   ├── hash.js       # 将文件相对路径编码或计算为固定 16 字节键
 │   ├── load.js       # 查询数据库现有记录，若数据表缺失则执行初始化
-│   ├── save.js       # 执行批量写入与删除操作
+│   ├── save.js       # 独立导出的批量持久化与删除辅助函数
 │   ├── sqlite.js     # 创建并配置 SQLite 数据库实例
 │   └── trans.js      # 封装 SQLite 事务，提供异常回滚机制
-└── tests             # 单元测试模块
+└── tests             # 单元测试目录
 ```
 
 ## 历史故事
 
-SQLite 的诞生与军事应用密切相关。2000 年，D. Richard Hipp 在为美国海军陆战队设计导弹驱逐舰板载损害控制系统软件时，遇到商业数据库由于配置复杂、日常需要专业维护且一旦连接丢失便会导致整个软件瘫痪的问题。Hipp 随即着手设计了一套无需任何独立服务器、零配置且直接对本地文件进行读写的嵌入式数据库，这便是 SQLite。
+SQLite 的诞生源自海军军工项目。2000 年，D. Richard Hipp 为美国海军陆战队设计导弹驱逐舰板载损害控制软件时，遭遇商业数据库因配置复杂、日常维护繁琐且连接丢失即导致系统瘫痪之痛点。Hipp 随后设计出免服务器配置、直接读写本地文件之嵌入式数据库，即 SQLite。
 
-为极限节约磁盘空间和降低读写延迟，SQLite 广泛应用了 Varint（可变字节整型）编码。在这种编码下，数值较小的整数（如常见的文件大小、序列号）仅占用 1 个字节，只有大数值才会占用更多字节。本项目中对文件大小和修改时间采用同样的压缩设计，从而秉承了 SQLite 极致节约空间与高效率的系统设计哲学。
+为了节省磁盘空间与降低读写延迟，SQLite 广泛应用了 Varint（可变字节整型）编码。在这种编码下，数值较小的整数仅占用 1 字节，只有大数值才会占用更多字节。本项目中对文件大小和修改时间采用同样的压缩设计，秉承了 SQLite 节省空间与高效之设计哲学。
 ../doc/zh/about.md

package/_.js

@@ -3,25 +3,39 @@ import vbE from "@3-/vb/vbE.js";
 import sqlite from "./sqlite.js";
 import load from "./load.js";
 import dirWalk from "./dirWalk.js";
-import save from "./save.js";
+import { stat } from "node:fs/promises";
+import { join } from "node:path";
+import int from "@3-/int";
+import hash from "./hash.js";
+import trans from "./trans.js";
 
 export default async (dir, db_path, ignore) => {
-  using db = sqlite(db_path);
-  const existing = new BinMap(),
+  const db = sqlite(db_path),
+    existing = new BinMap(),
     db_rows = load(db);
 
-  for (const row of db_rows) {
-    existing.set(row.hash, vbE([row.size, row.mtime]));
-  }
+  db_rows.forEach(({ hash, size, mtime }) => existing.set(hash, vbE([size, mtime])));
 
   const [scanned, to_update] = await dirWalk(dir, existing, ignore),
-    to_delete = [];
-  for (const row of db_rows) {
-    if (!scanned.has(row.hash)) {
-      to_delete.push(row.hash);
-    }
+    to_delete = db_rows.filter(({ hash }) => !scanned.has(hash)).map(({ hash }) => hash);
+
+  if (to_delete.length > 0) {
+    trans(db, () => {
+      const del = db.prepare("DELETE FROM scanMtimeLen WHERE hash=?");
+      to_delete.forEach((h) => del.run(h));
+    });
   }
 
-  save(db, to_update, to_delete);
-  return to_update.map(([rel_path]) => rel_path);
+  const insert = db.prepare("INSERT OR REPLACE INTO scanMtimeLen(hash,size,mtime)VALUES(?,?,?)"),
+    upsert = async (rel_path) => {
+      const fp = join(dir, rel_path),
+        { size, mtimeMs } = await stat(fp),
+        mtime = int(mtimeMs),
+        h = hash(rel_path);
+      insert.run(h, size, mtime);
+    };
+
+  upsert[Symbol.dispose] = () => db.close();
+
+  return [to_update.map(([rel_path]) => rel_path), upsert];
 };

package/dirWalk.js

@@ -13,7 +13,7 @@ export default async (dir, existing, ignore) => {
     to_update = [];
 
   await walkRelIgnore(dir, async (kind, rel_path) => {
-    if (ignore && ignore(kind, rel_path)) {
+    if (ignore && ignore(kind, rel_path) === false) {
       return false;
     }
     if (kind === FILE) {

package/hash.js

@@ -3,5 +3,5 @@ import utf8e from "@3-/utf8/utf8e.js";
 
 export default (str) => {
   const buf = utf8e(str);
-  return buf.length <= 16 ? buf : createHash("md5").update(buf).digest();
+  return buf.length <= 16 ? buf : new Uint8Array(createHash("md5").update(buf).digest());
 };

package/load.js

@@ -2,10 +2,10 @@ const SQLITE_ERROR = 1;
 
 export default (db) => {
   try {
-    return db.prepare("SELECT hash,size,mtime FROM file").all();
+    return db.prepare("SELECT hash,size,mtime FROM scanMtimeLen").all();
   } catch (err) {
     if (err.errno === SQLITE_ERROR) {
-      db.exec("CREATE TABLE file(hash PRIMARY KEY,size INT UNSIGNED,mtime INT UNSIGNED)");
+      db.exec("CREATE TABLE scanMtimeLen(hash PRIMARY KEY,size INT UNSIGNED,mtime INT UNSIGNED)");
       return [];
     }
     throw err;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@1-/scan",
-  "version": "0.1.3",
+  "version": "0.1.5",
   "description": "Incrementally scan directory files and track metadata in SQLite / 增量扫描目录文件并使用 SQLite 记录元数据",
   "keywords": [
     "scan",

package/save.js

@@ -4,16 +4,14 @@ export default (db, to_update, to_delete) => {
   if (to_update.length > 0 || to_delete.length > 0) {
     trans(db, () => {
       if (to_update.length > 0) {
-        const insert = db.prepare("INSERT OR REPLACE INTO file(hash,size,mtime)VALUES(?,?,?)");
-        for (const [_, h, size, mtime] of to_update) {
-          insert.run(h, size, mtime);
-        }
+        const insert = db.prepare(
+          "INSERT OR REPLACE INTO scanMtimeLen(hash,size,mtime)VALUES(?,?,?)",
+        );
+        to_update.forEach(([_, h, size, mtime]) => insert.run(h, size, mtime));
       }
       if (to_delete.length > 0) {
-        const del = db.prepare("DELETE FROM file WHERE hash=?");
-        for (const h of to_delete) {
-          del.run(h);
-        }
+        const del = db.prepare("DELETE FROM scanMtimeLen WHERE hash=?");
+        to_delete.forEach((h) => del.run(h));
       }
     });
   }