@1-/scan 0.1.2 → 0.1.4

package/README.md

@@ -5,73 +5,113 @@
 <a id="en"></a>
 # @1-/scan : Incrementally scan directory files and track metadata in SQLite
 
-Incrementally scans directory files, tracks file sizes and modification times, and synchronizes status into an SQLite database using Bun's native SQLite driver (`bun:sqlite`), returning an array of updated relative paths.
+Incrementally scans directory files, compares file sizes and modification times to detect changes, synchronizes metadata to SQLite database, and returns updated relative paths.
 
 ## Features
 
-- Incremental Scanning: Detects and updates only new, modified, or deleted files, avoiding redundant file system operations.
-- Space-Efficient Storage: Employs Varint compression to serialize and compare file sizes and modification times.
-- Smart Path Key: Stores relative paths not exceeding 16 bytes as raw binary to preserve readability, while hashing longer paths to 16-byte MD5 digests to optimize index performance.
-- Database Synchronization: Synchronizes updates and deletions in a single atomic transaction.
-- Ignore Pattern Support: Integrates ignore rules dynamically during traversal.
-- Native SQLite: Leverages Bun's native, high-performance `bun:sqlite` engine, eliminating external build dependencies.
+- **Incremental Scanning**: Detects and processes only new, modified, or deleted files, avoiding redundant file system operations.
+- **Key Optimization**: Stores relative paths within 16 bytes directly as raw bytes; hashes longer paths to 16-byte MD5 digests to optimize database index space and query performance.
+- **Metadata Compression**: Compresses file sizes and modification times using Varint (variable-length byte) encoding.
+- **Transactional Integrity**: Packages updates and deletions in a single database transaction to guarantee consistency.
+- **Flexible Filtering**: Supports custom ignore callback functions to filter specific files and directories.
+- **Native Database**: Integrates Bun native `bun:sqlite` module, eliminating external database driver dependencies.
 
 ## Usage
 
+### Basic Incremental Scan
+
 ```javascript
 import scan from "@1-/scan";
 
-const dir = "./src";
-const dbPath = "./files.db";
+const dir = "./data";
+const db_path = "./scan_record.db";
+
+// Scan directory and sync metadata to SQLite, returning modified relative paths and upsert function
+const [updated_paths, upsert] = await scan(dir, db_path);
+
+// Auto-close database when exiting scope
+using _upsert = upsert;
+
+console.log("Updated files:", updated_paths);
 
-// Scan directory and sync records into SQLite, returning an array of updated relative paths
-const updatedPaths = await scan(dir, dbPath);
-console.log(updatedPaths);
+// Update scanned file metadata in database
+for (const rel_path of updated_paths) {
+  await upsert(rel_path);
+}
+```
+
+### Scan with Ignore Filter
+
+```javascript
+import scan from "@1-/scan";
+
+const dir = "./data";
+const db_path = "./scan_record.db";
+
+// Ignore temporary files and specific configurations
+const ignore = (kind, rel_path) => {
+  return rel_path.startsWith("temp/") || rel_path === "config.json";
+};
+
+const [updated_paths, upsert] = await scan(dir, db_path, ignore);
+using _upsert = upsert;
+
+console.log("Synced. Updated files:", updated_paths);
+
+for (const rel_path of updated_paths) {
+  await upsert(rel_path);
+}
 ```
 
 ## Design Ideas
 
-Execution flow of modules:
+The main entry orchestrates independent modules to execute the incremental scanning and synchronization flow.
 
 ```mermaid
 graph TD
-    Entry["_.js (Main)"] -->|Open database| Sqlite[sqlite.js]
-    Entry -->|Load existing records| Load[load.js]
-    Entry -->|Walk and compare| DirWalk[dirWalk.js]
-    DirWalk -->|Traverse files| Walk["@1-/walk/walkRelIgnore"]
-    DirWalk -->|Optimize keys| Hash[hash.js]
-    Entry -->|Apply modifications| Save[save.js]
-    Save -->|Wrap transaction| Trans[trans.js]
+    Entry["_.js (Entry Point)"] -->|1. Initialize Connection| Sqlite["sqlite.js"]
+    Entry -->|2. Load Existing Records| Load["load.js"]
+    Entry -->|3. Walk & Compare Files| DirWalk["dirWalk.js"]
+    DirWalk -->|Invoke| Walk["@1-/walk/walkRelIgnore"]
+    DirWalk -->|Process Path Keys| Hash["hash.js"]
+    Entry -->|4. Delete Absent & Return Upsert| Trans["trans.js"]
+    Save["save.js (Independent Sync Helper)"] -->|Transaction Wrapper| Trans
 ```
 
+1. **Initialize Connection (`sqlite.js`)**: Opens SQLite database connection and configures automatic connection disposal.
+2. **Load Records (`load.js`)**: Automatically creates schema if missing, retrieves existing file hashes, sizes, and modification times, and reconstructs reference set in memory.
+3. **Walk & Compare (`dirWalk.js`)**: Traverses directory structure recursively. Paths are transformed into 16-byte keys via `hash.js`. File attributes are encoded using `@3-/vb` and compared against database records to identify additions and modifications.
+4. **Delete & Return Upsert**: Uses `trans.js` to execute transaction-safe deletions for deleted files, and returns modified relative paths and an `upsert` function so that caller can update database records.
+5. **Independent Sync Helper (`save.js`)**: Exported independent module to execute bulk inserts and deletions in a single transaction.
+
 ## Tech Stack
 
-- Bun: Runtime and test runner
-- Bun SQLite: Bun's built-in high-performance SQLite engine
-- `@1-/walk`: Directory walker with ignore support
-- `@3-/vb`: Variable-length byte encoder
-- `@3-/binmap` / `@3-/binset`: Efficient binary collection structures
+- **Bun**: Runtime environment and test framework.
+- **Bun SQLite**: Native high-performance SQLite engine built into Bun.
+- **@1-/walk**: Directory walker with ignore support.
+- **@3-/vb**: Variable-length byte (Varint) encoder and decoder.
+- **@3-/binmap / @3-/binset**: Memory-efficient collections designed for binary keys.
 
 ## Directory Structure
 
 ```
 .
 ├── src
-│   ├── _.js          # Entry point orchestrating the scanning and sync process
-│   ├── dirWalk.js    # Recursively scans files and filters modified ones
-│   ├── load.js       # Retrieves database records and initializes schema
-│   ├── save.js       # Performs bulk database inserts and deletes
-│   ├── hash.js       # Processes path keys into raw bytes or MD5 digests
-│   ├── sqlite.js     # Manages SQLite database connection and disposal
-│   └── trans.js      # Wraps operations inside an SQL transaction
+│   ├── _.js          # Entry point coordinating scanning and returning upsert helper
+│   ├── dirWalk.js    # Directory traverser comparing file metadata
+│   ├── hash.js       # Hashing helper mapping paths to 16-byte keys
+│   ├── load.js       # Database loader initializing schema and loading records
+│   ├── save.js       # Independent helper executing bulk updates and deletions
+│   ├── sqlite.js     # Connection manager instantiating SQLite database
+│   └── trans.js      # Transaction wrapper providing rollback mechanism
 └── tests             # Test suites
 ```
 
 ## History
 
-SQLite was designed in 2000 by D. Richard Hipp while working on a US Navy damage control system. The application originally relied on an Informix database, which required extensive database administration. Hipp designed SQLite to be a serverless, self-contained library requiring zero configuration, allowing the software to function reliably even when database services were unavailable.
+SQLite was created by D. Richard Hipp in 2000 while designing board software for US Navy guided-missile destroyers. The system originally depended on a commercial database that required constant database administration; a connection loss could stall the entire damage control application. To resolve this vulnerability, Hipp designed a serverless, zero-configuration embedded database that directly reads and writes local files—marking the birth of SQLite.
 
-To optimize space inside the database file, SQLite internally uses variable-length integers (Varints) to compress metadata. This project adopts similar techniques—compressing file size and modification time into varints before storage—inheriting the SQLite philosophy of minimalism and space efficiency for local file synchronization.
+To conserve disk space and reduce I/O overhead, SQLite utilizes Varint (variable-length integer) encoding for metadata storage. Under this scheme, small integers consume only 1 byte, while larger numbers scale dynamically. This library inherits that design philosophy, compressing file metadata into varints before storing it, ensuring minimal footprint and high sync performance.
 ../doc/en/about.md
 
 ---
@@ -79,71 +119,111 @@ To optimize space inside the database file, SQLite internally uses variable-leng
 <a id="zh"></a>
 # @1-/scan : 增量扫描目录文件并使用 SQLite 记录元数据
 
-基于 Bun 原生的高性能内置 SQLite 数据库（`bun:sqlite`）增量扫描目录，比对并同步文件大小与修改时间，并返回有变更的相对路径数组。
+增量扫描目录文件，通过比对文件大小和修改时间检测变更，并同步至 SQLite 数据库中，最终返回有更新的相对路径列表。
 
 ## 功能介绍
 
-- 增量扫描：仅处理新增、修改或删除的文件，避免冗余文件操作。
-- 紧凑存储：使用可变字节码（Varint）压缩技术比对并保存文件大小和修改时间。
-- 路径映射：相对路径长度不大于 16 字节时保留原始字节，大于 16 字节时计算为 16 字节 MD5，优化数据库索引。
-- 事务同步：更新与删除操作合并至单次数据库事务，确保一致性。
-- 规则过滤：基于 `@1-/walk` 的忽略规则过滤特定文件与目录。
-- 原生数据库：使用 Bun 内置的 `bun:sqlite`，性能优异且无需安装或编译外部依赖。
+- **增量扫描**：仅处理新增、修改或删除的文件，避免冗余的文件系统读写，提升同步速度。
+- **路径压缩**：当相对路径长度小于等于 16 字节时保留原始字节；超出 16 字节则转换为 16 字节 MD5 值作为数据库主键，优化索引空间与查询性能。
+- **元数据压缩**：使用 Varint（可变字节整型）编码方式压缩存储文件大小和修改时间。
+- **事务安全**：将更新与删除操作合并在单个数据库事务中执行，确保数据一致性。
+- **灵活过滤**：支持通过自定义回调函数过滤指定类型的文件与目录。
+- **原生依赖**：基于 Bun 内置 `bun:sqlite` 模块，无需额外安装或编译数据库驱动。
 
 ## 使用演示
 
+### 基础增量扫描
+
 ```javascript
 import scan from "@1-/scan";
 
-const dir = "./src";
-const dbPath = "./files.db";
+const dir = "./data";
+const db_path = "./scan_record.db";
+
+// 扫描目录并同步至 SQLite，返回发生变更的相对路径列表与更新函数
+const [updated_paths, upsert] = await scan(dir, db_path);
+
+// 退出作用域时自动关闭数据库
+using _upsert = upsert;
+
+console.log("更新文件列表：", updated_paths);
 
-// 扫描目录并同步至 SQLite 数据库，返回有变更的相对路径数组
-const updatedPaths = await scan(dir, dbPath);
-console.log(updatedPaths);
+// 更新已处理文件的元数据至数据库
+for (const rel_path of updated_paths) {
+  await upsert(rel_path);
+}
+```
+
+### 带有忽略规则的扫描
+
+```javascript
+import scan from "@1-/scan";
+
+const dir = "./data";
+const db_path = "./scan_record.db";
+
+// 忽略特定文件或目录
+const ignore = (kind, rel_path) => {
+  return rel_path.startsWith("temp/") || rel_path === "config.json";
+};
+
+const [updated_paths, upsert] = await scan(dir, db_path, ignore);
+using _upsert = upsert;
+
+console.log("已同步，更新列表：", updated_paths);
+
+for (const rel_path of updated_paths) {
+  await upsert(rel_path);
+}
 ```
 
 ## 设计思路
 
-模块调用流程：
+系统主入口调用各个独立模块完成增量扫描与数据同步流程。
 
 ```mermaid
 graph TD
-    Entry["_.js (主入口)"] -->|打开数据库| Sqlite[sqlite.js]
-    Entry -->|载入已有记录| Load[load.js]
-    Entry -->|遍历并比对| DirWalk[dirWalk.js]
-    DirWalk -->|扫描文件系统| Walk["@1-/walk/walkRelIgnore"]
-    DirWalk -->|计算路径哈希| Hash[hash.js]
-    Entry -->|写入变更数据| Save[save.js]
-    Save -->|执行事务控制| Trans[trans.js]
+    Entry["_.js (主入口)"] -->|1. 初始化连接| Sqlite["sqlite.js"]
+    Entry -->|2. 加载已有记录| Load["load.js"]
+    Entry -->|3. 扫描文件系统并对比| DirWalk["dirWalk.js"]
+    DirWalk -->|调用| Walk["@1-/walk/walkRelIgnore"]
+    DirWalk -->|处理路径键| Hash["hash.js"]
+    Entry -->|4. 删除失效记录并返回更新函数| Trans["trans.js"]
+    Save["save.js (独立批量存储辅助模块)"] -->|事务保障| Trans
 ```
 
+1. **初始化连接 (`sqlite.js`)**：打开 SQLite 数据库，并配置自动释放连接机制。
+2. **加载记录 (`load.js`)**：若表不存在则自动创建，读取已记录的文件哈希、大小及修改时间，在内存中还原比对集合。
+3. **文件系统扫描 (`dirWalk.js`)**：递归遍历目录，利用 `hash.js` 将路径映射为 16 字节键。对比当前文件与数据库元数据（利用 `@3-/vb` 进行压缩状态对比），筛选出新增和修改的文件。
+4. **删除与返回更新函数**：使用 `trans.js` 开启事务，批量删除已被移除的无效记录，并返回变更的相对路径列表与 `upsert` 函数，供调用者按需持久化数据。
+5. **独立批量存储辅助模块 (`save.js`)**：导出的独立工具模块，用于在单个事务中一次性批量写入与删除。
+
 ## 技术栈
 
-- Bun：运行环境与测试工具
-- Bun SQLite：Bun 内置的高性能 SQLite 模块
-- `@1-/walk`：支持忽略规则的目录遍历工具
-- `@3-/vb`：可变长度整型编码器
-- `@3-/binmap` / `@3-/binset`：二进制哈希键容器
+- **Bun**：JavaScript 运行时及测试框架。
+- **Bun SQLite**：内置的轻量级、高性能 SQLite 实现。
+- **@1-/walk**：支持过滤规则的目录递归遍历工具。
+- **@3-/vb**：Varint（可变字节）编码与解码器。
+- **@3-/binmap / @3-/binset**：针对二进制键优化的 Map 和 Set 容器。
 
 ## 目录结构
 
 ```
 .
 ├── src
-│   ├── _.js          # 主入口，统筹扫描与同步逻辑
-│   ├── dirWalk.js    # 递归遍历目录，比对筛选出变更文件
-│   ├── load.js       # 读取数据库中全部记录，初始化数据表
-│   ├── save.js       # 事务内执行批量插入与删除
-│   ├── hash.js       # 计算相对路径哈希值或保留原始字节
-│   ├── sqlite.js     # 管理 SQLite 数据库连接及资源释放
-│   └── trans.js      # 封装数据库事务控制
-└── tests             # 测试目录
+│   ├── _.js          # 核心流程控制器，调度各模块并返回变更及更新函数
+│   ├── dirWalk.js    # 遍历目录并比对元数据，输出变更队列
+│   ├── hash.js       # 将文件相对路径编码或计算为固定 16 字节键
+│   ├── load.js       # 查询数据库现有记录，若数据表缺失则执行初始化
+│   ├── save.js       # 独立导出的批量持久化与删除辅助函数
+│   ├── sqlite.js     # 创建并配置 SQLite 数据库实例
+│   └── trans.js      # 封装 SQLite 事务，提供异常回滚机制
+└── tests             # 单元测试模块
 ```
 
 ## 历史故事
 
-SQLite 由 D. Richard Hipp 于 2000 年为驱逐舰控制系统编写。当时系统采用的商业数据库需要繁琐的管理，且一旦故障系统便无法运行。Hipp 设计出无服务器、零配置且直接读写单文件的 SQLite。
+SQLite 的诞生与军事应用密切相关。2000 年，D. Richard Hipp 在为美国海军陆战队设计导弹驱逐舰板载损害控制系统软件时，遇到商业数据库由于配置复杂、日常需要专业维护且一旦连接丢失便会导致整个软件瘫痪的问题。Hipp 随即着手设计了一套无需任何独立服务器、零配置且直接对本地文件进行读写的嵌入式数据库，这便是 SQLite。
 
-为节约存储空间，SQLite 内部采用可变长度整数（Varint）编码。本项目同样引入 Varint 压缩算法，对文件大小与修改时间做编码后再比对存储，延续了 SQLite 追求性能与紧凑空间的优良传统。
+为极限节约磁盘空间 and 降低读写延迟，SQLite 广泛应用了 Varint（可变字节整型）编码。在这种编码下，数值较小的整数（如常见的文件大小、序列号）仅占用 1 个字节，只有大数值才会占用更多字节。本项目中对文件大小和修改时间采用同样的压缩设计，从而秉承了 SQLite 极致节约空间与高效率的系统设计哲学。
 ../doc/zh/about.md

package/_.js

@@ -3,25 +3,39 @@ import vbE from "@3-/vb/vbE.js";
 import sqlite from "./sqlite.js";
 import load from "./load.js";
 import dirWalk from "./dirWalk.js";
-import save from "./save.js";
+import { stat } from "node:fs/promises";
+import { join } from "node:path";
+import int from "@3-/int";
+import hash from "./hash.js";
+import trans from "./trans.js";
 
-export default async (dir, db_path) => {
-  using db = sqlite(db_path);
-  const existing = new BinMap(),
+export default async (dir, db_path, ignore) => {
+  const db = sqlite(db_path),
+    existing = new BinMap(),
     db_rows = load(db);
 
-  for (const row of db_rows) {
-    existing.set(row.hash, vbE([row.size, row.mtime]));
-  }
+  db_rows.forEach(({ hash, size, mtime }) => existing.set(hash, vbE([size, mtime])));
+
+  const [scanned, to_update] = await dirWalk(dir, existing, ignore),
+    to_delete = db_rows.filter(({ hash }) => !scanned.has(hash)).map(({ hash }) => hash);
 
-  const [scanned, to_update] = await dirWalk(dir, existing),
-    to_delete = [];
-  for (const row of db_rows) {
-    if (!scanned.has(row.hash)) {
-      to_delete.push(row.hash);
-    }
+  if (to_delete.length > 0) {
+    trans(db, () => {
+      const del = db.prepare("DELETE FROM file WHERE hash=?");
+      to_delete.forEach((h) => del.run(h));
+    });
   }
 
-  save(db, to_update, to_delete);
-  return to_update.map(([rel_path]) => rel_path);
+  const insert = db.prepare("INSERT OR REPLACE INTO file(hash,size,mtime)VALUES(?,?,?)"),
+    upsert = async (rel_path) => {
+      const fp = join(dir, rel_path),
+        { size, mtimeMs } = await stat(fp),
+        mtime = int(mtimeMs),
+        h = hash(rel_path);
+      insert.run(h, size, mtime);
+    };
+
+  upsert[Symbol.dispose] = () => db.close();
+
+  return [to_update.map(([rel_path]) => rel_path), upsert];
 };

package/dirWalk.js

@@ -8,11 +8,14 @@ import vbE from "@3-/vb/vbE.js";
 import int from "@3-/int";
 import hash from "./hash.js";
 
-export default async (dir, existing) => {
+export default async (dir, existing, ignore) => {
   const scanned = new BinSet(),
     to_update = [];
 
   await walkRelIgnore(dir, async (kind, rel_path) => {
+    if (ignore && ignore(kind, rel_path) === false) {
+      return false;
+    }
     if (kind === FILE) {
       const { size, mtimeMs } = await stat(join(dir, rel_path)),
         mtime = int(mtimeMs),

package/hash.js

@@ -3,5 +3,5 @@ import utf8e from "@3-/utf8/utf8e.js";
 
 export default (str) => {
   const buf = utf8e(str);
-  return buf.length <= 16 ? buf : createHash("md5").update(buf).digest();
+  return buf.length <= 16 ? buf : new Uint8Array(createHash("md5").update(buf).digest());
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@1-/scan",
-  "version": "0.1.2",
+  "version": "0.1.4",
   "description": "Incrementally scan directory files and track metadata in SQLite / 增量扫描目录文件并使用 SQLite 记录元数据",
   "keywords": [
     "scan",

package/save.js

@@ -5,15 +5,11 @@ export default (db, to_update, to_delete) => {
     trans(db, () => {
       if (to_update.length > 0) {
         const insert = db.prepare("INSERT OR REPLACE INTO file(hash,size,mtime)VALUES(?,?,?)");
-        for (const [_, h, size, mtime] of to_update) {
-          insert.run(h, size, mtime);
-        }
+        to_update.forEach(([_, h, size, mtime]) => insert.run(h, size, mtime));
       }
       if (to_delete.length > 0) {
         const del = db.prepare("DELETE FROM file WHERE hash=?");
-        for (const h of to_delete) {
-          del.run(h);
-        }
+        to_delete.forEach((h) => del.run(h));
       }
     });
   }