@1-/scan 0.1.2 → 0.1.3

package/README.md

@@ -5,73 +5,98 @@
 <a id="en"></a>
 # @1-/scan : Incrementally scan directory files and track metadata in SQLite
 
-Incrementally scans directory files, tracks file sizes and modification times, and synchronizes status into an SQLite database using Bun's native SQLite driver (`bun:sqlite`), returning an array of updated relative paths.
+Incrementally scans directory files, compares file sizes and modification times to detect changes, synchronizes metadata to an SQLite database, and returns updated relative paths.
 
 ## Features
 
-- Incremental Scanning: Detects and updates only new, modified, or deleted files, avoiding redundant file system operations.
-- Space-Efficient Storage: Employs Varint compression to serialize and compare file sizes and modification times.
-- Smart Path Key: Stores relative paths not exceeding 16 bytes as raw binary to preserve readability, while hashing longer paths to 16-byte MD5 digests to optimize index performance.
-- Database Synchronization: Synchronizes updates and deletions in a single atomic transaction.
-- Ignore Pattern Support: Integrates ignore rules dynamically during traversal.
-- Native SQLite: Leverages Bun's native, high-performance `bun:sqlite` engine, eliminating external build dependencies.
+- **Incremental Scanning**: Detects and processes only new, modified, or deleted files, avoiding redundant file system operations.
+- **Key Optimization**: Stores relative paths within 16 bytes directly as raw bytes; hashes longer paths to 16-byte MD5 digests to optimize database index space and query performance.
+- **Metadata Compression**: Compresses file sizes and modification times using Varint (variable-length byte) encoding.
+- **Transactional Integrity**: Packages updates and deletions in a single database transaction to guarantee consistency.
+- **Flexible Filtering**: Supports custom ignore callback functions to filter specific files and directories.
+- **Native Database**: Integrates Bun's native `bun:sqlite` module, eliminating external database driver dependencies.
 
 ## Usage
 
+### Basic Incremental Scan
+
 ```javascript
 import scan from "@1-/scan";
 
-const dir = "./src";
-const dbPath = "./files.db";
+const dir = "./data";
+const dbPath = "./scan_record.db";
 
-// Scan directory and sync records into SQLite, returning an array of updated relative paths
+// Scan directory and sync metadata to SQLite, returning modified relative paths
 const updatedPaths = await scan(dir, dbPath);
-console.log(updatedPaths);
+console.log("Updated files:", updatedPaths);
+```
+
+### Scan with Ignore Filter
+
+```javascript
+import { FILE } from "@1-/walk";
+import scan from "@1-/scan";
+
+const dir = "./data";
+const dbPath = "./scan_record.db";
+
+// Ignore temporary files and specific configurations
+const ignore = (kind, relPath) => {
+  return relPath.startsWith("temp/") || relPath === "config.json";
+};
+
+const updatedPaths = await scan(dir, dbPath, ignore);
+console.log("Synced. Updated files:", updatedPaths);
 ```
 
 ## Design Ideas
 
-Execution flow of modules:
+The main entry orchestrates independent modules to execute the incremental scanning and synchronization flow.
 
 ```mermaid
 graph TD
-    Entry["_.js (Main)"] -->|Open database| Sqlite[sqlite.js]
-    Entry -->|Load existing records| Load[load.js]
-    Entry -->|Walk and compare| DirWalk[dirWalk.js]
-    DirWalk -->|Traverse files| Walk["@1-/walk/walkRelIgnore"]
-    DirWalk -->|Optimize keys| Hash[hash.js]
-    Entry -->|Apply modifications| Save[save.js]
-    Save -->|Wrap transaction| Trans[trans.js]
+    Entry["_.js (Entry Point)"] -->|1. Initialize Connection| Sqlite[sqlite.js]
+    Entry -->|2. Load Existing Records| Load[load.js]
+    Entry -->|3. Walk & Compare Files| DirWalk[dirWalk.js]
+    DirWalk -->|Invoke| Walk["@1-/walk/walkRelIgnore"]
+    DirWalk -->|Process Path Keys| Hash[hash.js]
+    Entry -->|4. Persist Changes| Save[save.js]
+    Save -->|Transaction Wrapper| Trans[trans.js]
 ```
 
+1. **Initialize Connection (`sqlite.js`)**: Opens the SQLite database connection and configures automatic connection disposal.
+2. **Load Records (`load.js`)**: Automatically creates the schema if missing, retrieves existing file hashes, sizes, and modification times, and reconstructs the reference set in memory.
+3. **Walk & Compare (`dirWalk.js`)**: Traverses the directory structure recursively. Paths are transformed into 16-byte keys via `hash.js`. File attributes are encoded using `@3-/vb` and compared against database records to identify additions and modifications.
+4. **Persist Changes (`save.js`)**: Executes bulk inserts and deletions in a single transaction via `trans.js` to update database state.
+
 ## Tech Stack
 
-- Bun: Runtime and test runner
-- Bun SQLite: Bun's built-in high-performance SQLite engine
-- `@1-/walk`: Directory walker with ignore support
-- `@3-/vb`: Variable-length byte encoder
-- `@3-/binmap` / `@3-/binset`: Efficient binary collection structures
+- **Bun**: Runtime environment and test framework.
+- **Bun SQLite**: Native high-performance SQLite engine built into Bun.
+- **@1-/walk**: Directory walker with ignore support.
+- **@3-/vb**: Variable-length byte (Varint) encoder and decoder.
+- **@3-/binmap / @3-/binset**: Memory-efficient collections designed for binary keys.
 
 ## Directory Structure
 
 ```
 .
 ├── src
-│   ├── _.js          # Entry point orchestrating the scanning and sync process
-│   ├── dirWalk.js    # Recursively scans files and filters modified ones
-│   ├── load.js       # Retrieves database records and initializes schema
-│   ├── save.js       # Performs bulk database inserts and deletes
-│   ├── hash.js       # Processes path keys into raw bytes or MD5 digests
-│   ├── sqlite.js     # Manages SQLite database connection and disposal
-│   └── trans.js      # Wraps operations inside an SQL transaction
+│   ├── _.js          # Entry point coordinating scanning and synchronization
+│   ├── dirWalk.js    # Directory traverser comparing file metadata
+│   ├── hash.js       # Hashing helper mapping paths to 16-byte keys
+│   ├── load.js       # Database loader initializing schema and loading records
+│   ├── save.js       # Writer executing bulk updates and deletions
+│   ├── sqlite.js     # Connection manager instantiating SQLite database
+│   └── trans.js      # Transaction wrapper providing rollback mechanism
 └── tests             # Test suites
 ```
 
 ## History
 
-SQLite was designed in 2000 by D. Richard Hipp while working on a US Navy damage control system. The application originally relied on an Informix database, which required extensive database administration. Hipp designed SQLite to be a serverless, self-contained library requiring zero configuration, allowing the software to function reliably even when database services were unavailable.
+SQLite was created by D. Richard Hipp in 2000 while designing board software for US Navy guided-missile destroyers. The system originally depended on a commercial database that required constant database administration; a connection loss could stall the entire damage control application. To resolve this vulnerability, Hipp designed a serverless, zero-configuration embedded database that directly reads and writes local files—marking the birth of SQLite.
 
-To optimize space inside the database file, SQLite internally uses variable-length integers (Varints) to compress metadata. This project adopts similar techniques—compressing file size and modification time into varints before storage—inheriting the SQLite philosophy of minimalism and space efficiency for local file synchronization.
+To conserve disk space and reduce I/O overhead, SQLite utilizes Varint (variable-length integer) encoding for metadata storage. Under this scheme, small integers consume only 1 byte, while larger numbers scale dynamically. This library inherits that design philosophy, compressing file metadata into varints before storing it, ensuring minimal footprint and high sync performance.
 ../doc/en/about.md
 
 ---
@@ -79,71 +104,96 @@ To optimize space inside the database file, SQLite internally uses variable-leng
 <a id="zh"></a>
 # @1-/scan : 增量扫描目录文件并使用 SQLite 记录元数据
 
-基于 Bun 原生的高性能内置 SQLite 数据库（`bun:sqlite`）增量扫描目录，比对并同步文件大小与修改时间，并返回有变更的相对路径数组。
+增量扫描目录文件，通过比对文件大小和修改时间检测变更，并同步至 SQLite 数据库中，最终返回有更新的相对路径列表。
 
 ## 功能介绍
 
-- 增量扫描：仅处理新增、修改或删除的文件，避免冗余文件操作。
-- 紧凑存储：使用可变字节码（Varint）压缩技术比对并保存文件大小和修改时间。
-- 路径映射：相对路径长度不大于 16 字节时保留原始字节，大于 16 字节时计算为 16 字节 MD5，优化数据库索引。
-- 事务同步：更新与删除操作合并至单次数据库事务，确保一致性。
-- 规则过滤：基于 `@1-/walk` 的忽略规则过滤特定文件与目录。
-- 原生数据库：使用 Bun 内置的 `bun:sqlite`，性能优异且无需安装或编译外部依赖。
+- **增量扫描**：仅处理新增、修改或删除的文件，避免冗余的文件系统读写，提升同步速度。
+- **路径压缩**：当相对路径长度小于等于 16 字节时保留原始字节；超出 16 字节则转换为 16 字节 MD5 值作为数据库主键，优化索引空间与查询性能。
+- **元数据压缩**：使用 Varint（可变字节整型）编码方式压缩存储文件大小和修改时间。
+- **事务安全**：将更新与删除操作合并在单个数据库事务中执行，确保数据一致性。
+- **灵活过滤**：支持通过自定义回调函数过滤指定类型的文件与目录。
+- **原生依赖**：基于 Bun 内置 `bun:sqlite` 模块，无需额外安装或编译数据库驱动。
 
 ## 使用演示
 
+### 基础增量扫描
+
 ```javascript
 import scan from "@1-/scan";
 
-const dir = "./src";
-const dbPath = "./files.db";
+const dir = "./data";
+const dbPath = "./scan_record.db";
 
-// 扫描目录并同步至 SQLite 数据库，返回有变更的相对路径数组
+// 扫描目录并同步至 SQLite，返回发生变更的相对路径列表
 const updatedPaths = await scan(dir, dbPath);
-console.log(updatedPaths);
+console.log("更新文件列表：", updatedPaths);
+```
+
+### 带有忽略规则的扫描
+
+```javascript
+import { FILE } from "@1-/walk";
+import scan from "@1-/scan";
+
+const dir = "./data";
+const dbPath = "./scan_record.db";
+
+// 忽略特定文件或目录
+const ignore = (kind, relPath) => {
+  return relPath.startsWith("temp/") || relPath === "config.json";
+};
+
+const updatedPaths = await scan(dir, dbPath, ignore);
+console.log("已同步，更新列表：", updatedPaths);
 ```
 
 ## 设计思路
 
-模块调用流程：
+系统主入口调用各个独立模块完成增量扫描与数据同步流程。
 
 ```mermaid
 graph TD
-    Entry["_.js (主入口)"] -->|打开数据库| Sqlite[sqlite.js]
-    Entry -->|载入已有记录| Load[load.js]
-    Entry -->|遍历并比对| DirWalk[dirWalk.js]
-    DirWalk -->|扫描文件系统| Walk["@1-/walk/walkRelIgnore"]
-    DirWalk -->|计算路径哈希| Hash[hash.js]
-    Entry -->|写入变更数据| Save[save.js]
-    Save -->|执行事务控制| Trans[trans.js]
+    Entry["_.js (主入口)"] -->|1. 初始化连接| Sqlite[sqlite.js]
+    Entry -->|2. 加载已有记录| Load[load.js]
+    Entry -->|3. 扫描文件系统并对比| DirWalk[dirWalk.js]
+    DirWalk -->|调用| Walk["@1-/walk/walkRelIgnore"]
+    DirWalk -->|处理路径键| Hash[hash.js]
+    Entry -->|4. 持久化数据变更| Save[save.js]
+    Save -->|事务保障| Trans[trans.js]
 ```
 
+1. **初始化连接 (`sqlite.js`)**：打开 SQLite 数据库，并配置自动释放连接机制。
+2. **加载记录 (`load.js`)**：若表不存在则自动创建，读取已记录的文件哈希、大小及修改时间，在内存中还原比对集合。
+3. **文件系统扫描 (`dirWalk.js`)**：递归遍历目录，利用 `hash.js` 将路径映射为 16 字节键。对比当前文件与数据库元数据（利用 `@3-/vb` 进行压缩状态对比），筛选出新增和修改的文件。
+4. **数据存储 (`save.js`)**：使用 `trans.js` 开启事务，将需要删除的无效记录及需要更新的元数据批量写入 SQLite 数据库。
+
 ## 技术栈
 
-- Bun：运行环境与测试工具
-- Bun SQLite：Bun 内置的高性能 SQLite 模块
-- `@1-/walk`：支持忽略规则的目录遍历工具
-- `@3-/vb`：可变长度整型编码器
-- `@3-/binmap` / `@3-/binset`：二进制哈希键容器
+- **Bun**：JavaScript 运行时及测试框架。
+- **Bun SQLite**：内置的轻量级、高性能 SQLite 实现。
+- **@1-/walk**：支持过滤规则的目录递归遍历工具。
+- **@3-/vb**：Varint（可变字节）编码与解码器。
+- **@3-/binmap / @3-/binset**：针对二进制键优化的 Map 和 Set 容器。
 
 ## 目录结构
 
 ```
 .
 ├── src
-│   ├── _.js          # 主入口，统筹扫描与同步逻辑
-│   ├── dirWalk.js    # 递归遍历目录，比对筛选出变更文件
-│   ├── load.js       # 读取数据库中全部记录，初始化数据表
-│   ├── save.js       # 事务内执行批量插入与删除
-│   ├── hash.js       # 计算相对路径哈希值或保留原始字节
-│   ├── sqlite.js     # 管理 SQLite 数据库连接及资源释放
-│   └── trans.js      # 封装数据库事务控制
-└── tests             # 测试目录
+│   ├── _.js          # 核心流程控制器，调度各模块完成增量同步
+│   ├── dirWalk.js    # 遍历目录并比对元数据，输出变更队列
+│   ├── hash.js       # 将文件相对路径编码或计算为固定 16 字节键
+│   ├── load.js       # 查询数据库现有记录，若数据表缺失则执行初始化
+│   ├── save.js       # 执行批量写入与删除操作
+│   ├── sqlite.js     # 创建并配置 SQLite 数据库实例
+│   └── trans.js      # 封装 SQLite 事务，提供异常回滚机制
+└── tests             # 单元测试模块
 ```
 
 ## 历史故事
 
-SQLite 由 D. Richard Hipp 于 2000 年为驱逐舰控制系统编写。当时系统采用的商业数据库需要繁琐的管理，且一旦故障系统便无法运行。Hipp 设计出无服务器、零配置且直接读写单文件的 SQLite。
+SQLite 的诞生与军事应用密切相关。2000 年，D. Richard Hipp 在为美国海军陆战队设计导弹驱逐舰板载损害控制系统软件时，遇到商业数据库由于配置复杂、日常需要专业维护且一旦连接丢失便会导致整个软件瘫痪的问题。Hipp 随即着手设计了一套无需任何独立服务器、零配置且直接对本地文件进行读写的嵌入式数据库，这便是 SQLite。
 
-为节约存储空间，SQLite 内部采用可变长度整数（Varint）编码。本项目同样引入 Varint 压缩算法，对文件大小与修改时间做编码后再比对存储，延续了 SQLite 追求性能与紧凑空间的优良传统。
+为极限节约磁盘空间和降低读写延迟，SQLite 广泛应用了 Varint（可变字节整型）编码。在这种编码下，数值较小的整数（如常见的文件大小、序列号）仅占用 1 个字节，只有大数值才会占用更多字节。本项目中对文件大小和修改时间采用同样的压缩设计，从而秉承了 SQLite 极致节约空间与高效率的系统设计哲学。
 ../doc/zh/about.md

package/_.js

@@ -5,7 +5,7 @@ import load from "./load.js";
 import dirWalk from "./dirWalk.js";
 import save from "./save.js";
 
-export default async (dir, db_path) => {
+export default async (dir, db_path, ignore) => {
   using db = sqlite(db_path);
   const existing = new BinMap(),
     db_rows = load(db);
@@ -14,7 +14,7 @@ export default async (dir, db_path) => {
     existing.set(row.hash, vbE([row.size, row.mtime]));
   }
 
-  const [scanned, to_update] = await dirWalk(dir, existing),
+  const [scanned, to_update] = await dirWalk(dir, existing, ignore),
     to_delete = [];
   for (const row of db_rows) {
     if (!scanned.has(row.hash)) {

package/dirWalk.js

@@ -8,11 +8,14 @@ import vbE from "@3-/vb/vbE.js";
 import int from "@3-/int";
 import hash from "./hash.js";
 
-export default async (dir, existing) => {
+export default async (dir, existing, ignore) => {
   const scanned = new BinSet(),
     to_update = [];
 
   await walkRelIgnore(dir, async (kind, rel_path) => {
+    if (ignore && ignore(kind, rel_path)) {
+      return false;
+    }
     if (kind === FILE) {
       const { size, mtimeMs } = await stat(join(dir, rel_path)),
         mtime = int(mtimeMs),

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@1-/scan",
-  "version": "0.1.2",
+  "version": "0.1.3",
   "description": "Incrementally scan directory files and track metadata in SQLite / 增量扫描目录文件并使用 SQLite 记录元数据",
   "keywords": [
     "scan",