@1-/scan 0.1.6 → 0.1.7

package/README.md

@@ -7,16 +7,15 @@
 
 Incrementally scans directory files, compares file sizes and modification times to detect changes, synchronizes metadata to SQLite database, and returns updated relative paths.
 
-## Features
+## 1. Features
 
 - **Incremental Scanning**: Compares file sizes and modification times to process only new, modified, or deleted files, avoiding redundant read/write operations.
 - **Key Optimization**: Stores relative paths within 16 bytes directly as raw bytes; hashes longer paths to 16-byte MD5 digests to optimize database index space and query performance.
-- **Metadata Compression**: Compresses file sizes and modification times using Varint (variable-length byte) encoding.
+- **Memory Efficiency**: Utilizes BinMap and BinSet to store binary keys in memory, avoiding string decoding overhead and reducing memory footprint.
 - **Transactional Integrity**: Performs updates and deletions within database transactions to guarantee consistency.
-- **File Filtering**: Supports custom ignore callback functions to filter files and directories.
-- **Native Database**: Integrates Bun native `bun:sqlite` module, eliminating external database driver dependencies.
+- **Configuration-free**: Built-in automatic table initialization and connection management via @1-/sqlite.
 
-## Usage
+## 2. Usage
 
 ### Basic Incremental Scan
 
@@ -25,12 +24,13 @@ import scan from "@1-/scan";
 
 const dir = "./data";
 const db_path = "./scan_record.db";
+const files = ["file1.txt", "file2.txt"];
 
-// Scan directory and sync metadata to SQLite, returning modified relative paths and upsert function
-const [updated_paths, upsert] = await scan(dir, db_path);
+// Scan file list and sync metadata to SQLite, returning modified relative paths and upsert function
+const [updated_paths, upsert] = await scan(dir, db_path, files);
 
 // Auto-close database when exiting scope
-using _upsert = upsert;
+using _ = upsert;
 
 console.log("Updated files:", updated_paths);
 
@@ -40,35 +40,11 @@ for (const rel_path of updated_paths) {
 }
 ```
 
-### Scan with Ignore Filter
-
-```javascript
-import scan from "@1-/scan";
-
-const dir = "./data";
-const db_path = "./scan_record.db";
-
-// Ignore temporary files and specific configurations
-const ignore = (kind, rel_path) => {
-  return rel_path.startsWith("temp/") || rel_path === "config.json";
-};
-
-const [updated_paths, upsert] = await scan(dir, db_path, ignore);
-using _upsert = upsert;
-
-console.log("Synced. Updated files:", updated_paths);
-
-// Update scanned file metadata in database
-for (const rel_path of updated_paths) {
-  await upsert(rel_path);
-}
-```
-
-### Bulk Storage Module Usage
+### Bulk Storage Module
 
 ```javascript
 import save from "@1-/scan/save.js";
-import sqlite from "@1-/scan/sqlite.js";
+import sqlite from "@1-/sqlite";
 
 const db = sqlite("./scan_record.db");
 
@@ -78,46 +54,41 @@ save(db, [["file.txt", new Uint8Array([1, 2, 3]), 123, 1620000000]], [new Uint8A
 db.close();
 ```
 
-## Design Ideas
+## 3. Design Ideas
 
-The main entry orchestrates independent modules to execute the incremental scanning and synchronization flow.
+The entry point orchestrates independent modules to execute the incremental scanning and synchronization flow.
 
-![](https://i-01.eu.org/KDY9Lax79XwPBKztEqhU5A)
+![](https://i-01.eu.org/5UJ3b_gajqcCgVKaX5rqjw)
 
-1. **Initialize Connection (`sqlite.js`)**: Opens SQLite database connection and configures automatic connection disposal.
-2. **Load Records (`load.js`)**: Automatically creates `scanMtimeLen` table if missing, retrieves existing file hashes, sizes, and modification times, and reconstructs reference set in memory.
-3. **Walk & Compare (`dirWalk.js`)**: Traverses directory structure recursively. Paths are transformed into 16-byte keys via `hash.js`. File attributes are encoded using `@3-/vb` and compared against database records to identify additions and modifications.
-4. **Delete & Return Upsert**: Uses `trans.js` to execute transaction-safe deletions for deleted files, and returns modified relative paths and an `upsert` function so that caller can update database records.
-5. **Independent Sync Helper (`save.js`)**: Exported independent module to execute bulk updates and deletions in transactions.
+1. **Initialize Connection**: Invokes `@1-/sqlite` to open the SQLite database.
+2. **Load Records**: `load.js` checks if the `scanMtimeLen` table exists and creates it if missing. It retrieves stored hashes, sizes, and modification times, and reconstructs the memory mapping.
+3. **Compare Metadata**: Iterates over the input file list, mapping paths to 16-byte binary keys via `@1-/hash`. It adds files with size or modification time mismatches to the update list.
+4. **Delete and Return**: Deletes absent records in a transaction and returns the changed path list along with the `upsert` function for external persistence.
 
-## Tech Stack
+## 4. Tech Stack
 
 - **Bun**: Runtime environment and test framework.
-- **Bun SQLite**: Native SQLite engine built into Bun.
-- **@1-/walk**: Directory walker with ignore support.
+- **@1-/sqlite**: Database connection management and transaction wrapper.
+- **@1-/hash**: Length-bounded MD5 hash utility.
 - **@3-/vb**: Variable-length byte (Varint) encoder and decoder.
-- **@3-/binmap / @3-/binset**: Memory-efficient collections designed for binary keys.
+- **@3-/binmap / @3-/binset**: Rust and WebAssembly binary key containers.
 
-## Code Structure
+## 5. Code Structure
 
-```
+```text
 .
 ├── src
-│   ├── _.js          # Entry point coordinating scanning and returning upsert helper
-│   ├── dirWalk.js    # Directory traverser comparing file metadata
-│   ├── hash.js       # Hashing helper mapping paths to 16-byte keys
-│   ├── load.js       # Database loader initializing schema and loading records
-│   ├── save.js       # Independent helper executing bulk updates and deletions
-│   ├── sqlite.js     # Connection manager instantiating SQLite database
-│   └── trans.js      # Transaction wrapper providing rollback mechanism
-└── tests             # Test directory
+│   ├── _.js          # Core flow controller
+│   ├── load.js       # Table schema initialization and loader
+│   └── save.js       # Bulk update and delete helper
+└── tests             # Unit tests
 ```
 
-## History
+## 6. History
 
-SQLite was created by D. Richard Hipp in 2000 while designing board software for US Navy guided-missile destroyers. The system originally depended on a commercial database that required constant database administration; a connection loss could stall the entire damage control application. To resolve this vulnerability, Hipp designed a serverless, zero-configuration embedded database that directly reads and writes local files—marking the birth of SQLite.
+SQLite was created by D. Richard Hipp in 2000 while designing board software for guided-missile destroyers. The system originally depended on a commercial database that required constant database administration; a connection loss could stall the entire damage control application. Hipp designed a serverless, zero-configuration embedded database that directly reads and writes local files, marking the birth of SQLite.
 
-To conserve disk space and reduce I/O overhead, SQLite utilizes Varint (variable-length integer) encoding for metadata storage. Under this scheme, small integers consume only 1 byte, while larger numbers scale dynamically. This library inherits that design philosophy, compressing file metadata into varints before storing it, ensuring minimal footprint and high sync performance.
+To conserve space and reduce latency, SQLite utilizes Varint (variable-length integer) encoding for metadata storage. Under this scheme, small integers consume only 1 byte, while larger numbers scale dynamically. This library inherits that design philosophy, compressing file metadata into varints for memory storage to ensure minimal footprint and high synchronization performance.
 ## About
 
 This library is developed by [WebC.site](https://webc.site).
@@ -129,18 +100,17 @@ This library is developed by [WebC.site](https://webc.site).
 <a id="zh"></a>
 # @1-/scan : 增量扫描目录文件并使用 SQLite 记录元数据
 
-增量扫描目录文件，通过比对大小与修改时间检测变更，同步元数据至 SQLite 数据库，返回发生变更之相对路径列表。
+增量扫描目录文件，比对大小与修改时间检测变更，同步元数据至 SQLite 数据库，返回发生变更的相对路径列表。
 
-## 功能介绍
+## 1. 功能介绍
 
-- **增量扫描**：比对文件大小与修改时间，仅对新增、修改或删除之文件执行操作，避免冗余读写，提升效率。
-- **路径压缩**：当相对路径长度不大于 16 字节时保留原始字节；超出 16 字节则转换为 16 字节 MD5 值作为主键，优化索引空间与查询性能。
-- **元数据压缩**：使用 Varint（可变字节整型）编码方式压缩存储文件大小与修改时间。
-- **事务安全**：更新与删除操作合并在数据库事务中执行，确保数据一致性。
-- **过滤规则**：支持传入自定义过滤函数，按需排除特定文件与目录。
-- **原生依赖**：基于 Bun 内置 `bun:sqlite` 模块，无需额外安装或编译数据库驱动。
+- **增量扫描**：比对大小与修改时间，过滤未变更文件，减少磁盘读写。
+- **路径键优化**：路径长度不大于 16 字节时存储原始字节，超出 16 字节则转换为 16 字节 MD5 值，优化索引空间与查询性能。
+- **内存优化**：在内存中使用 BinMap 与 BinSet 存储二进制哈希键，避免字符串解码开销，降低内存占用。
+- **事务保障**：变更及删除操作合并在数据库事务中执行，确保数据一致性。
+- **免除配置**：基于 @1-/sqlite，内置表初始化与连接管理，开箱即用。
 
-## 使用演示
+## 2. 使用演示
 
 ### 基础增量扫描
 
@@ -149,12 +119,13 @@ import scan from "@1-/scan";
 
 const dir = "./data";
 const db_path = "./scan_record.db";
+const files = ["file1.txt", "file2.txt"];
 
-// 扫描目录并同步至 SQLite，返回发生变更之相对路径列表与更新函数
-const [updated_paths, upsert] = await scan(dir, db_path);
+// 扫描文件列表并同步至 SQLite，返回发生变更的相对路径列表与更新函数
+const [updated_paths, upsert] = await scan(dir, db_path, files);
 
 // 退出作用域时自动关闭数据库
-using _upsert = upsert;
+using _ = upsert;
 
 console.log("更新文件列表：", updated_paths);
 
@@ -164,34 +135,11 @@ for (const rel_path of updated_paths) {
 }
 ```
 
-### 过滤规则扫描
-
-```javascript
-import scan from "@1-/scan";
-
-const dir = "./data";
-const db_path = "./scan_record.db";
-
-// 过滤临时文件与特定配置
-const ignore = (kind, rel_path) => {
-  return rel_path.startsWith("temp/") || rel_path === "config.json";
-};
-
-const [updated_paths, upsert] = await scan(dir, db_path, ignore);
-using _upsert = upsert;
-
-console.log("已同步，更新列表：", updated_paths);
-
-for (const rel_path of updated_paths) {
-  await upsert(rel_path);
-}
-```
-
-### 批量存储模块使用
+### 独立批量存储模块
 
 ```javascript
 import save from "@1-/scan/save.js";
-import sqlite from "@1-/scan/sqlite.js";
+import sqlite from "@1-/sqlite";
 
 const db = sqlite("./scan_record.db");
 
@@ -201,46 +149,41 @@ save(db, [["file.txt", new Uint8Array([1, 2, 3]), 123, 1620000000]], [new Uint8A
 db.close();
 ```
 
-## 设计思路
+## 3. 设计思路
 
-系统主入口调度各独立模块完成增量扫描与数据同步。
+系统调度各模块完成增量扫描与数据同步。
 
-![](https://i-01.eu.org/o9xJHQ8BDxiBtfjhCcTUtQ)
+![](https://i-01.eu.org/qBVoq3Bhu1yHqHFfBcBTJg)
 
-1. **初始化连接 (`sqlite.js`)**：打开 SQLite 数据库，并配置自动释放连接机制。
-2. **加载记录 (`load.js`)**：若数据表 `scanMtimeLen` 不存在则自动创建，读取已记录的文件哈希、大小及修改时间，在内存中还原比对集合。
-3. **文件系统扫描 (`dirWalk.js`)**：递归遍历目录，利用 `hash.js` 将路径映射为 16 字节键。对比当前文件与数据库元数据（利用 `@3-/vb` 进行压缩状态对比），筛选出新增和修改的文件。
-4. **删除与返回更新函数**：使用 `trans.js` 开启事务，批量删除已被移除的记录，并返回变更的相对路径列表与 `upsert` 函数，供调用者持久化数据。
-5. **独立批量存储模块 (`save.js`)**：供外部调用的独立工具模块，用于在事务中批量写入与删除。
+1. **初始化连接**：调用 `@1-/sqlite` 打开 SQLite 数据库。
+2. **加载记录**：`load.js` 检查 `scanMtimeLen` 表是否存在，若不存在则创建。读取已记录的哈希、大小及修改时间，在内存中恢复已存在的文件映射。
+3. **元数据比对**：遍历输入的文件列表，利用 `@1-/hash` 将路径映射为 16 字节二进制键。若大小或修改时间不一致，则加入变更列表。
+4. **删除与返回**：使用事务批量删除已被移除的记录（已存在于数据库中但不在本次扫描列表内的文件），并返回变更路径列表与 `upsert` 函数，供外部持久化。
 
-## 技术栈
+## 4. 技术栈
 
 - **Bun**：JavaScript 运行时与测试框架。
-- **Bun SQLite**：内置 SQLite 实现。
-- **@1-/walk**：支持过滤规则的目录递归遍历工具。
-- **@3-/vb**：Varint（可变字节）编码与解码器。
-- **@3-/binmap / @3-/binset**：针对二进制键优化的 Map 和 Set 容器。
+- **@1-/sqlite**：SQLite 连接管理与事务封装。
+- **@1-/hash**：长度限制的 MD5 哈希工具。
+- **@3-/vb**：Varint 变长整型编码与解码器。
+- **@3-/binmap / @3-/binset**：基于 Rust 与 WebAssembly 的高效二进制键容器。
 
-## 代码结构
+## 5. 代码结构
 
-```
+```text
 .
 ├── src
-│   ├── _.js          # 核心流程控制器，调度各模块并返回变更及更新函数
-│   ├── dirWalk.js    # 遍历目录并比对元数据，输出变更队列
-│   ├── hash.js       # 将文件相对路径编码或计算为固定 16 字节键
-│   ├── load.js       # 查询数据库现有记录，若数据表缺失则执行初始化
-│   ├── save.js       # 独立导出的批量持久化与删除辅助函数
-│   ├── sqlite.js     # 创建并配置 SQLite 数据库实例
-│   └── trans.js      # 封装 SQLite 事务，提供异常回滚机制
-└── tests             # 单元测试目录
+│   ├── _.js          # 核心控制流程
+│   ├── load.js       # 元数据表初始化与加载
+│   └── save.js       # 批量更新与删除工具
+└── tests             # 单元测试
 ```
 
-## 历史故事
+## 6. 历史故事
 
-SQLite 的诞生源自海军军工项目。2000 年，D. Richard Hipp 为美国海军陆战队设计导弹驱逐舰板载损害控制软件时，遭遇商业数据库因配置复杂、日常维护繁琐且连接丢失即导致系统瘫痪之痛点。Hipp 随后设计出免服务器配置、直接读写本地文件之嵌入式数据库，即 SQLite。
+SQLite 的诞生源自导弹驱逐舰板载损害控制软件项目。2000 年，D. Richard Hipp 为美国海军设计该系统时，遭遇商业数据库因配置复杂、无法承受断连和崩溃之痛点。Hipp 随后设计出免服务器配置、直接读写本地文件之嵌入式数据库，即 SQLite。
 
-为了节省磁盘空间与降低读写延迟，SQLite 广泛应用了 Varint（可变字节整型）编码。在这种编码下，数值较小的整数仅占用 1 字节，只有大数值才会占用更多字节。本项目中对文件大小和修改时间采用同样的压缩设计，秉承了 SQLite 节省空间与高效之设计哲学。
+为节省空间与降低读写延迟，SQLite 广泛应用了 Varint（可变字节整型）编码。在这种编码下，小整数仅占用 1 字节，只有大数值才占用更多字节。本项目在内存存储设计中对文件大小与修改时间采用同样的压缩设计，契合 SQLite 节省空间与高效之设计哲学。
 ## 关于
 
 本库由 [WebC.site](https://webc.site) 开发。

package/_.js

@@ -1,41 +1,59 @@
+import sqlite from "@1-/sqlite";
+import tx from "@1-/sqlite/tx.js";
 import { BinMap } from "@3-/binmap";
 import vbE from "@3-/vb/vbE.js";
-import sqlite from "./sqlite.js";
 import load from "./load.js";
-import dirWalk from "./dirWalk.js";
-import { stat } from "node:fs/promises";
+import { stat as fsStat } from "node:fs/promises";
 import { join } from "node:path";
 import int from "@3-/int";
-import hash from "./hash.js";
-import trans from "./trans.js";
+import strmd5 from "@1-/hash/strmd5.js";
+import { BinSet } from "@3-/binset";
+import u8eq from "@3-/u8/u8eq.js";
 
-export default async (dir, db_path, ignore) => {
+const stat = async (dir, rel_path) => {
+  const { size, mtimeMs: mtime_ms } = await fsStat(join(dir, rel_path));
+  return [size, int(mtime_ms), strmd5(rel_path)];
+};
+
+export default async (dir, db_path, files) => {
   const db = sqlite(db_path),
     existing = new BinMap(),
-    db_rows = load(db);
+    db_rows = load(db),
+    scanned = new BinSet(),
+    update = [];
 
   db_rows.forEach(({ hash, size, mtime }) => existing.set(hash, vbE([size, mtime])));
 
-  const [scanned, to_update] = await dirWalk(dir, existing, ignore),
-    to_delete = db_rows.filter(({ hash }) => !scanned.has(hash)).map(({ hash }) => hash);
+  for (const rel_path of files) {
+    try {
+      const [size, mtime, hash] = await stat(dir, rel_path),
+        val = existing.get(hash);
 
-  if (to_delete.length > 0) {
-    trans(db, () => {
-      const del = db.prepare("DELETE FROM scanMtimeLen WHERE hash=?");
-      to_delete.forEach((h) => del.run(h));
-    });
+      scanned.add(hash);
+
+      if (!val || !u8eq(val, vbE([size, mtime]))) {
+        update.push(rel_path);
+      }
+    } catch {}
   }
 
-  const insert = db.prepare("INSERT OR REPLACE INTO scanMtimeLen(hash,size,mtime)VALUES(?,?,?)"),
+  const rm = db_rows.filter(({ hash }) => !scanned.has(hash)).map(({ hash }) => hash),
+    insert = db.prepare("INSERT OR REPLACE INTO scanMtimeLen(hash,size,mtime)VALUES(?,?,?)"),
     upsert = async (rel_path) => {
-      const fp = join(dir, rel_path),
-        { size, mtimeMs } = await stat(fp),
-        mtime = int(mtimeMs),
-        h = hash(rel_path);
-      insert.run(h, size, mtime);
+      try {
+        const [size, mtime, hash] = await stat(dir, rel_path);
+        insert.run(hash, size, mtime);
+      } catch {}
     };
 
+  if (rm.length > 0) {
+    tx(db, () => {
+      const del = db.prepare("DELETE FROM scanMtimeLen WHERE hash=?");
+      rm.forEach((hash) => del.run(hash));
+    });
+  }
+
   upsert[Symbol.dispose] = () => db.close();
 
-  return [to_update.map(([rel_path]) => rel_path), upsert];
+  return [update, upsert];
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@1-/scan",
-  "version": "0.1.6",
+  "version": "0.1.7",
   "description": "Incrementally scan directory files and track metadata in SQLite / 增量扫描目录文件并使用 SQLite 记录元数据",
   "keywords": [
     "directory",
@@ -21,8 +21,9 @@
     ".": "./_.js",
     "./*": "./*"
   },
-  "dependencies": {
-    "@1-/walk": "^0.1.1",
+  "peerDependencies": {
+    "@1-/hash": "^0.1.0",
+    "@1-/sqlite": "^0.1.0",
     "@3-/binmap": "^0.1.20",
     "@3-/binset": "^0.1.6",
     "@3-/int": "^0.1.1",

package/save.js

@@ -1,17 +1,17 @@
-import trans from "./trans.js";
+import tx from "@1-/sqlite/tx.js";
 
-export default (db, to_update, to_delete) => {
-  if (to_update.length > 0 || to_delete.length > 0) {
-    trans(db, () => {
-      if (to_update.length > 0) {
+export default (db, update, rm) => {
+  if (update.length > 0 || rm.length > 0) {
+    tx(db, () => {
+      if (update.length > 0) {
         const insert = db.prepare(
           "INSERT OR REPLACE INTO scanMtimeLen(hash,size,mtime)VALUES(?,?,?)",
         );
-        to_update.forEach(([_, h, size, mtime]) => insert.run(h, size, mtime));
+        update.forEach(([_, h, size, mtime]) => insert.run(h, size, mtime));
       }
-      if (to_delete.length > 0) {
+      if (rm.length > 0) {
         const del = db.prepare("DELETE FROM scanMtimeLen WHERE hash=?");
-        to_delete.forEach((h) => del.run(h));
+        rm.forEach((h) => del.run(h));
       }
     });
   }

package/dirWalk.js

@@ -1,35 +0,0 @@
-import { stat } from "node:fs/promises";
-import { join } from "node:path";
-import { FILE } from "@1-/walk";
-import walkRelIgnore from "@1-/walk/walkRelIgnore.js";
-import { BinSet } from "@3-/binset";
-import u8eq from "@3-/u8/u8eq.js";
-import vbE from "@3-/vb/vbE.js";
-import int from "@3-/int";
-import hash from "./hash.js";
-
-export default async (dir, existing, ignore) => {
-  const scanned = new BinSet(),
-    to_update = [];
-
-  await walkRelIgnore(dir, async (kind, rel_path) => {
-    if (ignore && ignore(kind, rel_path) === false) {
-      return false;
-    }
-    if (kind === FILE) {
-      const { size, mtimeMs } = await stat(join(dir, rel_path)),
-        mtime = int(mtimeMs),
-        h = hash(rel_path);
-
-      scanned.add(h);
-
-      const val = existing.get(h);
-      if (val && u8eq(val, vbE([size, mtime]))) {
-        return;
-      }
-      to_update.push([rel_path, h, size, mtime]);
-    }
-  });
-
-  return [scanned, to_update];
-};

package/hash.js

@@ -1,7 +0,0 @@
-import { createHash } from "node:crypto";
-import utf8e from "@3-/utf8/utf8e.js";
-
-export default (str) => {
-  const buf = utf8e(str);
-  return buf.length <= 16 ? buf : new Uint8Array(createHash("md5").update(buf).digest());
-};

package/sqlite.js

@@ -1,7 +0,0 @@
-import { Database } from "bun:sqlite";
-
-export default (db_path) => {
-  const db = new Database(db_path);
-  db[Symbol.dispose] = () => db.close();
-  return db;
-};

package/trans.js

@@ -1,11 +0,0 @@
-export default (db, run) => {
-  db.exec("BEGIN");
-  try {
-    const res = run();
-    db.exec("COMMIT");
-    return res;
-  } catch (err) {
-    db.exec("ROLLBACK");
-    throw err;
-  }
-};