@1-/scan 0.1.3 → 0.1.4

package/README.md

@@ -5,7 +5,7 @@
 <a id="en"></a>
 # @1-/scan : Incrementally scan directory files and track metadata in SQLite
 
-Incrementally scans directory files, compares file sizes and modification times to detect changes, synchronizes metadata to an SQLite database, and returns updated relative paths.
+Incrementally scans directory files, compares file sizes and modification times to detect changes, synchronizes metadata to SQLite database, and returns updated relative paths.
 
 ## Features
 
@@ -14,7 +14,7 @@ Incrementally scans directory files, compares file sizes and modification times
 - **Metadata Compression**: Compresses file sizes and modification times using Varint (variable-length byte) encoding.
 - **Transactional Integrity**: Packages updates and deletions in a single database transaction to guarantee consistency.
 - **Flexible Filtering**: Supports custom ignore callback functions to filter specific files and directories.
-- **Native Database**: Integrates Bun's native `bun:sqlite` module, eliminating external database driver dependencies.
+- **Native Database**: Integrates Bun native `bun:sqlite` module, eliminating external database driver dependencies.
 
 ## Usage
 
@@ -24,29 +24,43 @@ Incrementally scans directory files, compares file sizes and modification times
 import scan from "@1-/scan";
 
 const dir = "./data";
-const dbPath = "./scan_record.db";
+const db_path = "./scan_record.db";
 
-// Scan directory and sync metadata to SQLite, returning modified relative paths
-const updatedPaths = await scan(dir, dbPath);
-console.log("Updated files:", updatedPaths);
+// Scan directory and sync metadata to SQLite, returning modified relative paths and upsert function
+const [updated_paths, upsert] = await scan(dir, db_path);
+
+// Auto-close database when exiting scope
+using _upsert = upsert;
+
+console.log("Updated files:", updated_paths);
+
+// Update scanned file metadata in database
+for (const rel_path of updated_paths) {
+  await upsert(rel_path);
+}
 ```
 
 ### Scan with Ignore Filter
 
 ```javascript
-import { FILE } from "@1-/walk";
 import scan from "@1-/scan";
 
 const dir = "./data";
-const dbPath = "./scan_record.db";
+const db_path = "./scan_record.db";
 
 // Ignore temporary files and specific configurations
-const ignore = (kind, relPath) => {
-  return relPath.startsWith("temp/") || relPath === "config.json";
+const ignore = (kind, rel_path) => {
+  return rel_path.startsWith("temp/") || rel_path === "config.json";
 };
 
-const updatedPaths = await scan(dir, dbPath, ignore);
-console.log("Synced. Updated files:", updatedPaths);
+const [updated_paths, upsert] = await scan(dir, db_path, ignore);
+using _upsert = upsert;
+
+console.log("Synced. Updated files:", updated_paths);
+
+for (const rel_path of updated_paths) {
+  await upsert(rel_path);
+}
 ```
 
 ## Design Ideas
@@ -55,19 +69,20 @@ The main entry orchestrates independent modules to execute the incremental scann
 
 ```mermaid
 graph TD
-    Entry["_.js (Entry Point)"] -->|1. Initialize Connection| Sqlite[sqlite.js]
-    Entry -->|2. Load Existing Records| Load[load.js]
-    Entry -->|3. Walk & Compare Files| DirWalk[dirWalk.js]
+    Entry["_.js (Entry Point)"] -->|1. Initialize Connection| Sqlite["sqlite.js"]
+    Entry -->|2. Load Existing Records| Load["load.js"]
+    Entry -->|3. Walk & Compare Files| DirWalk["dirWalk.js"]
     DirWalk -->|Invoke| Walk["@1-/walk/walkRelIgnore"]
-    DirWalk -->|Process Path Keys| Hash[hash.js]
-    Entry -->|4. Persist Changes| Save[save.js]
-    Save -->|Transaction Wrapper| Trans[trans.js]
+    DirWalk -->|Process Path Keys| Hash["hash.js"]
+    Entry -->|4. Delete Absent & Return Upsert| Trans["trans.js"]
+    Save["save.js (Independent Sync Helper)"] -->|Transaction Wrapper| Trans
 ```
 
-1. **Initialize Connection (`sqlite.js`)**: Opens the SQLite database connection and configures automatic connection disposal.
-2. **Load Records (`load.js`)**: Automatically creates the schema if missing, retrieves existing file hashes, sizes, and modification times, and reconstructs the reference set in memory.
-3. **Walk & Compare (`dirWalk.js`)**: Traverses the directory structure recursively. Paths are transformed into 16-byte keys via `hash.js`. File attributes are encoded using `@3-/vb` and compared against database records to identify additions and modifications.
-4. **Persist Changes (`save.js`)**: Executes bulk inserts and deletions in a single transaction via `trans.js` to update database state.
+1. **Initialize Connection (`sqlite.js`)**: Opens SQLite database connection and configures automatic connection disposal.
+2. **Load Records (`load.js`)**: Automatically creates schema if missing, retrieves existing file hashes, sizes, and modification times, and reconstructs reference set in memory.
+3. **Walk & Compare (`dirWalk.js`)**: Traverses directory structure recursively. Paths are transformed into 16-byte keys via `hash.js`. File attributes are encoded using `@3-/vb` and compared against database records to identify additions and modifications.
+4. **Delete & Return Upsert**: Uses `trans.js` to execute transaction-safe deletions for deleted files, and returns modified relative paths and an `upsert` function so that caller can update database records.
+5. **Independent Sync Helper (`save.js`)**: Exported independent module to execute bulk inserts and deletions in a single transaction.
 
 ## Tech Stack
 
@@ -82,11 +97,11 @@ graph TD
 ```
 .
 ├── src
-│   ├── _.js          # Entry point coordinating scanning and synchronization
+│   ├── _.js          # Entry point coordinating scanning and returning upsert helper
 │   ├── dirWalk.js    # Directory traverser comparing file metadata
 │   ├── hash.js       # Hashing helper mapping paths to 16-byte keys
 │   ├── load.js       # Database loader initializing schema and loading records
-│   ├── save.js       # Writer executing bulk updates and deletions
+│   ├── save.js       # Independent helper executing bulk updates and deletions
 │   ├── sqlite.js     # Connection manager instantiating SQLite database
 │   └── trans.js      # Transaction wrapper providing rollback mechanism
 └── tests             # Test suites
@@ -123,29 +138,43 @@ To conserve disk space and reduce I/O overhead, SQLite utilizes Varint (variable
 import scan from "@1-/scan";
 
 const dir = "./data";
-const dbPath = "./scan_record.db";
+const db_path = "./scan_record.db";
 
-// 扫描目录并同步至 SQLite，返回发生变更的相对路径列表
-const updatedPaths = await scan(dir, dbPath);
-console.log("更新文件列表：", updatedPaths);
+// 扫描目录并同步至 SQLite，返回发生变更的相对路径列表与更新函数
+const [updated_paths, upsert] = await scan(dir, db_path);
+
+// 退出作用域时自动关闭数据库
+using _upsert = upsert;
+
+console.log("更新文件列表：", updated_paths);
+
+// 更新已处理文件的元数据至数据库
+for (const rel_path of updated_paths) {
+  await upsert(rel_path);
+}
 ```
 
 ### 带有忽略规则的扫描
 
 ```javascript
-import { FILE } from "@1-/walk";
 import scan from "@1-/scan";
 
 const dir = "./data";
-const dbPath = "./scan_record.db";
+const db_path = "./scan_record.db";
 
 // 忽略特定文件或目录
-const ignore = (kind, relPath) => {
-  return relPath.startsWith("temp/") || relPath === "config.json";
+const ignore = (kind, rel_path) => {
+  return rel_path.startsWith("temp/") || rel_path === "config.json";
 };
 
-const updatedPaths = await scan(dir, dbPath, ignore);
-console.log("已同步，更新列表：", updatedPaths);
+const [updated_paths, upsert] = await scan(dir, db_path, ignore);
+using _upsert = upsert;
+
+console.log("已同步，更新列表：", updated_paths);
+
+for (const rel_path of updated_paths) {
+  await upsert(rel_path);
+}
 ```
 
 ## 设计思路
@@ -154,19 +183,20 @@ console.log("已同步，更新列表：", updatedPaths);
 
 ```mermaid
 graph TD
-    Entry["_.js (主入口)"] -->|1. 初始化连接| Sqlite[sqlite.js]
-    Entry -->|2. 加载已有记录| Load[load.js]
-    Entry -->|3. 扫描文件系统并对比| DirWalk[dirWalk.js]
+    Entry["_.js (主入口)"] -->|1. 初始化连接| Sqlite["sqlite.js"]
+    Entry -->|2. 加载已有记录| Load["load.js"]
+    Entry -->|3. 扫描文件系统并对比| DirWalk["dirWalk.js"]
     DirWalk -->|调用| Walk["@1-/walk/walkRelIgnore"]
-    DirWalk -->|处理路径键| Hash[hash.js]
-    Entry -->|4. 持久化数据变更| Save[save.js]
-    Save -->|事务保障| Trans[trans.js]
+    DirWalk -->|处理路径键| Hash["hash.js"]
+    Entry -->|4. 删除失效记录并返回更新函数| Trans["trans.js"]
+    Save["save.js (独立批量存储辅助模块)"] -->|事务保障| Trans
 ```
 
 1. **初始化连接 (`sqlite.js`)**：打开 SQLite 数据库，并配置自动释放连接机制。
 2. **加载记录 (`load.js`)**：若表不存在则自动创建，读取已记录的文件哈希、大小及修改时间，在内存中还原比对集合。
 3. **文件系统扫描 (`dirWalk.js`)**：递归遍历目录，利用 `hash.js` 将路径映射为 16 字节键。对比当前文件与数据库元数据（利用 `@3-/vb` 进行压缩状态对比），筛选出新增和修改的文件。
-4. **数据存储 (`save.js`)**：使用 `trans.js` 开启事务，将需要删除的无效记录及需要更新的元数据批量写入 SQLite 数据库。
+4. **删除与返回更新函数**：使用 `trans.js` 开启事务，批量删除已被移除的无效记录，并返回变更的相对路径列表与 `upsert` 函数，供调用者按需持久化数据。
+5. **独立批量存储辅助模块 (`save.js`)**：导出的独立工具模块，用于在单个事务中一次性批量写入与删除。
 
 ## 技术栈
 
@@ -181,11 +211,11 @@ graph TD
 ```
 .
 ├── src
-│   ├── _.js          # 核心流程控制器，调度各模块完成增量同步
+│   ├── _.js          # 核心流程控制器，调度各模块并返回变更及更新函数
 │   ├── dirWalk.js    # 遍历目录并比对元数据，输出变更队列
 │   ├── hash.js       # 将文件相对路径编码或计算为固定 16 字节键
 │   ├── load.js       # 查询数据库现有记录，若数据表缺失则执行初始化
-│   ├── save.js       # 执行批量写入与删除操作
+│   ├── save.js       # 独立导出的批量持久化与删除辅助函数
 │   ├── sqlite.js     # 创建并配置 SQLite 数据库实例
 │   └── trans.js      # 封装 SQLite 事务，提供异常回滚机制
 └── tests             # 单元测试模块
@@ -195,5 +225,5 @@ graph TD
 
 SQLite 的诞生与军事应用密切相关。2000 年，D. Richard Hipp 在为美国海军陆战队设计导弹驱逐舰板载损害控制系统软件时，遇到商业数据库由于配置复杂、日常需要专业维护且一旦连接丢失便会导致整个软件瘫痪的问题。Hipp 随即着手设计了一套无需任何独立服务器、零配置且直接对本地文件进行读写的嵌入式数据库，这便是 SQLite。
 
-为极限节约磁盘空间和降低读写延迟，SQLite 广泛应用了 Varint（可变字节整型）编码。在这种编码下，数值较小的整数（如常见的文件大小、序列号）仅占用 1 个字节，只有大数值才会占用更多字节。本项目中对文件大小和修改时间采用同样的压缩设计，从而秉承了 SQLite 极致节约空间与高效率的系统设计哲学。
+为极限节约磁盘空间 and 降低读写延迟，SQLite 广泛应用了 Varint（可变字节整型）编码。在这种编码下，数值较小的整数（如常见的文件大小、序列号）仅占用 1 个字节，只有大数值才会占用更多字节。本项目中对文件大小和修改时间采用同样的压缩设计，从而秉承了 SQLite 极致节约空间与高效率的系统设计哲学。
 ../doc/zh/about.md

package/_.js

@@ -3,25 +3,39 @@ import vbE from "@3-/vb/vbE.js";
 import sqlite from "./sqlite.js";
 import load from "./load.js";
 import dirWalk from "./dirWalk.js";
-import save from "./save.js";
+import { stat } from "node:fs/promises";
+import { join } from "node:path";
+import int from "@3-/int";
+import hash from "./hash.js";
+import trans from "./trans.js";
 
 export default async (dir, db_path, ignore) => {
-  using db = sqlite(db_path);
-  const existing = new BinMap(),
+  const db = sqlite(db_path),
+    existing = new BinMap(),
     db_rows = load(db);
 
-  for (const row of db_rows) {
-    existing.set(row.hash, vbE([row.size, row.mtime]));
-  }
+  db_rows.forEach(({ hash, size, mtime }) => existing.set(hash, vbE([size, mtime])));
 
   const [scanned, to_update] = await dirWalk(dir, existing, ignore),
-    to_delete = [];
-  for (const row of db_rows) {
-    if (!scanned.has(row.hash)) {
-      to_delete.push(row.hash);
-    }
+    to_delete = db_rows.filter(({ hash }) => !scanned.has(hash)).map(({ hash }) => hash);
+
+  if (to_delete.length > 0) {
+    trans(db, () => {
+      const del = db.prepare("DELETE FROM file WHERE hash=?");
+      to_delete.forEach((h) => del.run(h));
+    });
   }
 
-  save(db, to_update, to_delete);
-  return to_update.map(([rel_path]) => rel_path);
+  const insert = db.prepare("INSERT OR REPLACE INTO file(hash,size,mtime)VALUES(?,?,?)"),
+    upsert = async (rel_path) => {
+      const fp = join(dir, rel_path),
+        { size, mtimeMs } = await stat(fp),
+        mtime = int(mtimeMs),
+        h = hash(rel_path);
+      insert.run(h, size, mtime);
+    };
+
+  upsert[Symbol.dispose] = () => db.close();
+
+  return [to_update.map(([rel_path]) => rel_path), upsert];
 };

package/dirWalk.js

@@ -13,7 +13,7 @@ export default async (dir, existing, ignore) => {
     to_update = [];
 
   await walkRelIgnore(dir, async (kind, rel_path) => {
-    if (ignore && ignore(kind, rel_path)) {
+    if (ignore && ignore(kind, rel_path) === false) {
       return false;
     }
     if (kind === FILE) {

package/hash.js

@@ -3,5 +3,5 @@ import utf8e from "@3-/utf8/utf8e.js";
 
 export default (str) => {
   const buf = utf8e(str);
-  return buf.length <= 16 ? buf : createHash("md5").update(buf).digest();
+  return buf.length <= 16 ? buf : new Uint8Array(createHash("md5").update(buf).digest());
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@1-/scan",
-  "version": "0.1.3",
+  "version": "0.1.4",
   "description": "Incrementally scan directory files and track metadata in SQLite / 增量扫描目录文件并使用 SQLite 记录元数据",
   "keywords": [
     "scan",

package/save.js

@@ -5,15 +5,11 @@ export default (db, to_update, to_delete) => {
     trans(db, () => {
       if (to_update.length > 0) {
         const insert = db.prepare("INSERT OR REPLACE INTO file(hash,size,mtime)VALUES(?,?,?)");
-        for (const [_, h, size, mtime] of to_update) {
-          insert.run(h, size, mtime);
-        }
+        to_update.forEach(([_, h, size, mtime]) => insert.run(h, size, mtime));
       }
       if (to_delete.length > 0) {
         const del = db.prepare("DELETE FROM file WHERE hash=?");
-        for (const h of to_delete) {
-          del.run(h);
-        }
+        to_delete.forEach((h) => del.run(h));
       }
     });
   }