@1-/scan 0.1.7 → 0.1.8

package/README.md

@@ -3,17 +3,17 @@
 ---
 
 <a id="en"></a>
-# @1-/scan : Incrementally scan directory files and track metadata in SQLite
+# @1-/scan : SQLite-backed incremental directory scanner
 
-Incrementally scans directory files, compares file sizes and modification times to detect changes, synchronizes metadata to SQLite database, and returns updated relative paths.
+Incrementally scans directory files, compares file sizes and modification times to detect changes, synchronizes metadata to SQLite database, and returns list of changed relative paths.
 
 ## 1. Features
 
-- **Incremental Scanning**: Compares file sizes and modification times to process only new, modified, or deleted files, avoiding redundant read/write operations.
-- **Key Optimization**: Stores relative paths within 16 bytes directly as raw bytes; hashes longer paths to 16-byte MD5 digests to optimize database index space and query performance.
-- **Memory Efficiency**: Utilizes BinMap and BinSet to store binary keys in memory, avoiding string decoding overhead and reducing memory footprint.
-- **Transactional Integrity**: Performs updates and deletions within database transactions to guarantee consistency.
-- **Configuration-free**: Built-in automatic table initialization and connection management via @1-/sqlite.
+- **Incremental Scan**: Compares size and modification time, filtering unchanged files to reduce disk I/O.
+- **Key Length Optimization**: Stores raw bytes for paths up to 16 bytes. Converts longer paths into 16-byte MD5 hashes to optimize database index space and query performance.
+- **Memory Optimization**: Uses BinMap and BinSet to store binary keys in memory, avoiding string decoding overhead and reducing memory footprint.
+- **Transactional Integrity**: Performs metadata updates and deletions in database transactions to ensure consistency.
+- **Zero Configuration**: Integrates @1-/sqlite to initialize database schema and manage database connections automatically.
 
 ## 2. Usage
 
@@ -26,10 +26,10 @@ const dir = "./data";
 const db_path = "./scan_record.db";
 const files = ["file1.txt", "file2.txt"];
 
-// Scan file list and sync metadata to SQLite, returning modified relative paths and upsert function
+// Scan file list, sync metadata to SQLite, return changed relative paths and upsert function
 const [updated_paths, upsert] = await scan(dir, db_path, files);
 
-// Auto-close database when exiting scope
+// Close database automatically when exiting scope
 using _ = upsert;
 
 console.log("Updated files:", updated_paths);
@@ -54,16 +54,16 @@ save(db, [["file.txt", new Uint8Array([1, 2, 3]), 123, 1620000000]], [new Uint8A
 db.close();
 ```
 
-## 3. Design Ideas
+## 3. Design
 
-The entry point orchestrates independent modules to execute the incremental scanning and synchronization flow.
+Main entry orchestrates modules to scan directories and synchronize metadata.
 
-![](https://i-01.eu.org/5UJ3b_gajqcCgVKaX5rqjw)
+![](https://fastly.jsdelivr.net/gh/webc-fs/-@-a/YhYlWMxhnYfN4bgT4FrA.svg)
 
-1. **Initialize Connection**: Invokes `@1-/sqlite` to open the SQLite database.
-2. **Load Records**: `load.js` checks if the `scanMtimeLen` table exists and creates it if missing. It retrieves stored hashes, sizes, and modification times, and reconstructs the memory mapping.
-3. **Compare Metadata**: Iterates over the input file list, mapping paths to 16-byte binary keys via `@1-/hash`. It adds files with size or modification time mismatches to the update list.
-4. **Delete and Return**: Deletes absent records in a transaction and returns the changed path list along with the `upsert` function for external persistence.
+1. **Initialize Connection**: Calls `@1-/sqlite` to open SQLite database.
+2. **Load Records**: `load.js` checks `scanMtimeLen` table, creates it if missing. Reads stored hashes, sizes, and modification times to restore memory mappings.
+3. **Compare Files**: Iterates over input file list. Maps paths to 16-byte binary keys via `@1-/hash`. Adds files with mismatched size or modification time to update list.
+4. **Delete and Return**: Deletes absent or unscanned records in transaction. Returns changed paths list and `upsert` function for persistence.
 
 ## 4. Tech Stack
 
@@ -86,7 +86,7 @@ The entry point orchestrates independent modules to execute the incremental scan
 
 ## 6. History
 
-SQLite was created by D. Richard Hipp in 2000 while designing board software for guided-missile destroyers. The system originally depended on a commercial database that required constant database administration; a connection loss could stall the entire damage control application. Hipp designed a serverless, zero-configuration embedded database that directly reads and writes local files, marking the birth of SQLite.
+SQLite was created by D. Richard Hipp in 2000 while designing board software for guided-missile destroyers. The system originally depended on commercial database that required constant database administration; connection loss could stall the entire damage control application. Hipp designed serverless, zero-configuration embedded database that directly reads and writes local files, marking the birth of SQLite.
 
 To conserve space and reduce latency, SQLite utilizes Varint (variable-length integer) encoding for metadata storage. Under this scheme, small integers consume only 1 byte, while larger numbers scale dynamically. This library inherits that design philosophy, compressing file metadata into varints for memory storage to ensure minimal footprint and high synchronization performance.
 ## About
@@ -98,17 +98,17 @@ This library is developed by [WebC.site](https://webc.site).
 ---
 
 <a id="zh"></a>
-# @1-/scan : 增量扫描目录文件并使用 SQLite 记录元数据
+# @1-/scan : 基于 SQLite 的目录增量扫描工具
 
-增量扫描目录文件，比对大小与修改时间检测变更，同步元数据至 SQLite 数据库，返回发生变更的相对路径列表。
+增量扫描目录文件，比对文件大小与修改时间检测变更，同步元数据至 SQLite 数据库，返回已变更相对路径列表。
 
 ## 1. 功能介绍
 
 - **增量扫描**：比对大小与修改时间，过滤未变更文件，减少磁盘读写。
-- **路径键优化**：路径长度不大于 16 字节时存储原始字节，超出 16 字节则转换为 16 字节 MD5 值，优化索引空间与查询性能。
-- **内存优化**：在内存中使用 BinMap 与 BinSet 存储二进制哈希键，避免字符串解码开销，降低内存占用。
-- **事务保障**：变更及删除操作合并在数据库事务中执行，确保数据一致性。
-- **免除配置**：基于 @1-/sqlite，内置表初始化与连接管理，开箱即用。
+- **键长优化**：路径长度不大于 16 字节存储原始字节，超出 16 字节转换为 16 字节 MD5 值，优化索引空间与查询性能。
+- **内存优化**：使用 BinMap 与 BinSet 存储二进制键，避免字符串解码，降低内存占用。
+- **事务保障**：元数据变更与删除操作合并在数据库事务中执行，确保数据一致性。
+- **零配置**：集成 @1-/sqlite，自动初始化数据库表结构，管理数据库连接。
 
 ## 2. 使用演示
 
@@ -121,15 +121,15 @@ const dir = "./data";
 const db_path = "./scan_record.db";
 const files = ["file1.txt", "file2.txt"];
 
-// 扫描文件列表并同步至 SQLite，返回发生变更的相对路径列表与更新函数
+// 扫描文件列表并同步至 SQLite，返回已变更的相对路径列表与更新函数
 const [updated_paths, upsert] = await scan(dir, db_path, files);
 
-// 退出作用域时自动关闭数据库
+// 退出作用域自动关闭数据库
 using _ = upsert;
 
 console.log("更新文件列表：", updated_paths);
 
-// 更新已处理文件元数据至数据库
+// 更新已处理文件的元数据至数据库
 for (const rel_path of updated_paths) {
   await upsert(rel_path);
 }
@@ -151,14 +151,14 @@ db.close();
 
 ## 3. 设计思路
 
-系统调度各模块完成增量扫描与数据同步。
+主入口调度各模块，协作完成目录扫描与数据同步。
 
-![](https://i-01.eu.org/qBVoq3Bhu1yHqHFfBcBTJg)
+![](https://fastly.jsdelivr.net/gh/webc-fs/-@e2/-ae7ErmK8GpR3UFyhddA.svg)
 
 1. **初始化连接**：调用 `@1-/sqlite` 打开 SQLite 数据库。
-2. **加载记录**：`load.js` 检查 `scanMtimeLen` 表是否存在，若不存在则创建。读取已记录的哈希、大小及修改时间，在内存中恢复已存在的文件映射。
-3. **元数据比对**：遍历输入的文件列表，利用 `@1-/hash` 将路径映射为 16 字节二进制键。若大小或修改时间不一致，则加入变更列表。
-4. **删除与返回**：使用事务批量删除已被移除的记录（已存在于数据库中但不在本次扫描列表内的文件），并返回变更路径列表与 `upsert` 函数，供外部持久化。
+2. **加载记录**：`load.js` 检查 `scanMtimeLen` 表，表不存在则自动创建。读取已记录哈希、大小及修改时间，恢复内存映射。
+3. **比对文件**：遍历输入文件列表，利用 `@1-/hash` 将路径映射为 16 字节二进制键。若大小或修改时间不一致，则加入变更列表。
+4. **删除与返回**：在事务中批量删除已被物理移除或不再扫描记录。返回变更路径列表与 `upsert` 函数，供外部持久化。
 
 ## 4. 技术栈
 
@@ -168,7 +168,7 @@ db.close();
 - **@3-/vb**：Varint 变长整型编码与解码器。
 - **@3-/binmap / @3-/binset**：基于 Rust 与 WebAssembly 的高效二进制键容器。
 
-## 5. 代码结构
+## 5. Code Structure
 
 ```text
 .

package/_.js

@@ -9,51 +9,66 @@ import int from "@3-/int";
 import strmd5 from "@1-/hash/strmd5.js";
 import { BinSet } from "@3-/binset";
 import u8eq from "@3-/u8/u8eq.js";
+import { availableParallelism } from "node:os";
+import pLimit from "@3-/plimit";
 
 const stat = async (dir, rel_path) => {
-  const { size, mtimeMs: mtime_ms } = await fsStat(join(dir, rel_path));
-  return [size, int(mtime_ms), strmd5(rel_path)];
-};
+    const { size, mtimeMs: mtime_ms } = await fsStat(join(dir, rel_path));
+    return [size, int(mtime_ms), strmd5(rel_path)];
+  },
+  scanFiles = async (dir, files, existing, limit) => {
+    const scanned = new BinSet(),
+      update = [];
+    await Promise.all(
+      files.map((rel_path) =>
+        limit(async () => {
+          try {
+            const [size, mtime, hash] = await stat(dir, rel_path),
+              val = existing.get(hash);
+
+            scanned.add(hash);
+
+            if (!val || !u8eq(val, vbE([size, mtime]))) {
+              update.push(rel_path);
+            }
+          } catch {}
+        }),
+      ),
+    );
+    return [scanned, update];
+  },
+  rmHashes = (db, rm) => {
+    if (rm.length > 0) {
+      tx(db, () => {
+        const del = db.prepare("DELETE FROM scanMtimeLen WHERE hash=?");
+        rm.forEach((hash) => del.run(hash));
+      });
+    }
+  },
+  newUpsert = (db, dir) => {
+    const insert = db.prepare("INSERT OR REPLACE INTO scanMtimeLen(hash,size,mtime)VALUES(?,?,?)"),
+      upsert = async (rel_path) => {
+        try {
+          const [size, mtime, hash] = await stat(dir, rel_path);
+          insert.run(hash, size, mtime);
+        } catch {}
+      };
+    upsert[Symbol.dispose] = () => db.close();
+    return upsert;
+  };
 
 export default async (dir, db_path, files) => {
   const db = sqlite(db_path),
     existing = new BinMap(),
     db_rows = load(db),
-    scanned = new BinSet(),
-    update = [];
+    limit = pLimit(availableParallelism());
 
   db_rows.forEach(({ hash, size, mtime }) => existing.set(hash, vbE([size, mtime])));
 
-  for (const rel_path of files) {
-    try {
-      const [size, mtime, hash] = await stat(dir, rel_path),
-        val = existing.get(hash);
-
-      scanned.add(hash);
-
-      if (!val || !u8eq(val, vbE([size, mtime]))) {
-        update.push(rel_path);
-      }
-    } catch {}
-  }
-
-  const rm = db_rows.filter(({ hash }) => !scanned.has(hash)).map(({ hash }) => hash),
-    insert = db.prepare("INSERT OR REPLACE INTO scanMtimeLen(hash,size,mtime)VALUES(?,?,?)"),
-    upsert = async (rel_path) => {
-      try {
-        const [size, mtime, hash] = await stat(dir, rel_path);
-        insert.run(hash, size, mtime);
-      } catch {}
-    };
-
-  if (rm.length > 0) {
-    tx(db, () => {
-      const del = db.prepare("DELETE FROM scanMtimeLen WHERE hash=?");
-      rm.forEach((hash) => del.run(hash));
-    });
-  }
-
-  upsert[Symbol.dispose] = () => db.close();
-
-  return [update, upsert];
+  const [scanned, update] = await scanFiles(dir, files, existing, limit),
+    rm = db_rows.filter(({ hash }) => !scanned.has(hash)).map(({ hash }) => hash);
+
+  rmHashes(db, rm);
+
+  return [update, newUpsert(db, dir)];
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@1-/scan",
-  "version": "0.1.7",
+  "version": "0.1.8",
   "description": "Incrementally scan directory files and track metadata in SQLite / 增量扫描目录文件并使用 SQLite 记录元数据",
   "keywords": [
     "directory",
@@ -23,10 +23,11 @@
   },
   "peerDependencies": {
     "@1-/hash": "^0.1.0",
-    "@1-/sqlite": "^0.1.0",
+    "@1-/sqlite": "^0.1.1",
     "@3-/binmap": "^0.1.20",
     "@3-/binset": "^0.1.6",
     "@3-/int": "^0.1.1",
+    "@3-/plimit": "^0.1.3",
     "@3-/u8": "^0.1.2",
     "@3-/utf8": "^0.1.1",
     "@3-/vb": "^0.1.6"