@1-/scan 0.1.4 → 0.1.6

package/README.md

@@ -9,11 +9,11 @@ Incrementally scans directory files, compares file sizes and modification times
 
 ## Features
 
-- **Incremental Scanning**: Detects and processes only new, modified, or deleted files, avoiding redundant file system operations.
+- **Incremental Scanning**: Compares file sizes and modification times to process only new, modified, or deleted files, avoiding redundant read/write operations.
 - **Key Optimization**: Stores relative paths within 16 bytes directly as raw bytes; hashes longer paths to 16-byte MD5 digests to optimize database index space and query performance.
 - **Metadata Compression**: Compresses file sizes and modification times using Varint (variable-length byte) encoding.
-- **Transactional Integrity**: Packages updates and deletions in a single database transaction to guarantee consistency.
-- **Flexible Filtering**: Supports custom ignore callback functions to filter specific files and directories.
+- **Transactional Integrity**: Performs updates and deletions within database transactions to guarantee consistency.
+- **File Filtering**: Supports custom ignore callback functions to filter files and directories.
 - **Native Database**: Integrates Bun native `bun:sqlite` module, eliminating external database driver dependencies.
 
 ## Usage
@@ -58,41 +58,47 @@ using _upsert = upsert;
 
 console.log("Synced. Updated files:", updated_paths);
 
+// Update scanned file metadata in database
 for (const rel_path of updated_paths) {
   await upsert(rel_path);
 }
 ```
 
+### Bulk Storage Module Usage
+
+```javascript
+import save from "@1-/scan/save.js";
+import sqlite from "@1-/scan/sqlite.js";
+
+const db = sqlite("./scan_record.db");
+
+// Bulk update and delete metadata
+save(db, [["file.txt", new Uint8Array([1, 2, 3]), 123, 1620000000]], [new Uint8Array([4, 5, 6])]);
+
+db.close();
+```
+
 ## Design Ideas
 
 The main entry orchestrates independent modules to execute the incremental scanning and synchronization flow.
 
-```mermaid
-graph TD
-    Entry["_.js (Entry Point)"] -->|1. Initialize Connection| Sqlite["sqlite.js"]
-    Entry -->|2. Load Existing Records| Load["load.js"]
-    Entry -->|3. Walk & Compare Files| DirWalk["dirWalk.js"]
-    DirWalk -->|Invoke| Walk["@1-/walk/walkRelIgnore"]
-    DirWalk -->|Process Path Keys| Hash["hash.js"]
-    Entry -->|4. Delete Absent & Return Upsert| Trans["trans.js"]
-    Save["save.js (Independent Sync Helper)"] -->|Transaction Wrapper| Trans
-```
+![](https://i-01.eu.org/KDY9Lax79XwPBKztEqhU5A)
 
 1. **Initialize Connection (`sqlite.js`)**: Opens SQLite database connection and configures automatic connection disposal.
-2. **Load Records (`load.js`)**: Automatically creates schema if missing, retrieves existing file hashes, sizes, and modification times, and reconstructs reference set in memory.
+2. **Load Records (`load.js`)**: Automatically creates `scanMtimeLen` table if missing, retrieves existing file hashes, sizes, and modification times, and reconstructs reference set in memory.
 3. **Walk & Compare (`dirWalk.js`)**: Traverses directory structure recursively. Paths are transformed into 16-byte keys via `hash.js`. File attributes are encoded using `@3-/vb` and compared against database records to identify additions and modifications.
 4. **Delete & Return Upsert**: Uses `trans.js` to execute transaction-safe deletions for deleted files, and returns modified relative paths and an `upsert` function so that caller can update database records.
-5. **Independent Sync Helper (`save.js`)**: Exported independent module to execute bulk inserts and deletions in a single transaction.
+5. **Independent Sync Helper (`save.js`)**: Exported independent module to execute bulk updates and deletions in transactions.
 
 ## Tech Stack
 
 - **Bun**: Runtime environment and test framework.
-- **Bun SQLite**: Native high-performance SQLite engine built into Bun.
+- **Bun SQLite**: Native SQLite engine built into Bun.
 - **@1-/walk**: Directory walker with ignore support.
 - **@3-/vb**: Variable-length byte (Varint) encoder and decoder.
 - **@3-/binmap / @3-/binset**: Memory-efficient collections designed for binary keys.
 
-## Directory Structure
+## Code Structure
 
 ```
 .
@@ -104,7 +110,7 @@ graph TD
 │   ├── save.js       # Independent helper executing bulk updates and deletions
 │   ├── sqlite.js     # Connection manager instantiating SQLite database
 │   └── trans.js      # Transaction wrapper providing rollback mechanism
-└── tests             # Test suites
+└── tests             # Test directory
 ```
 
 ## History
@@ -112,22 +118,26 @@ graph TD
 SQLite was created by D. Richard Hipp in 2000 while designing board software for US Navy guided-missile destroyers. The system originally depended on a commercial database that required constant database administration; a connection loss could stall the entire damage control application. To resolve this vulnerability, Hipp designed a serverless, zero-configuration embedded database that directly reads and writes local files—marking the birth of SQLite.
 
 To conserve disk space and reduce I/O overhead, SQLite utilizes Varint (variable-length integer) encoding for metadata storage. Under this scheme, small integers consume only 1 byte, while larger numbers scale dynamically. This library inherits that design philosophy, compressing file metadata into varints before storing it, ensuring minimal footprint and high sync performance.
-../doc/en/about.md
+## About
+
+This library is developed by [WebC.site](https://webc.site).
+
+[WebC.site](https://webc.site): A new paradigm of web development for AI
 
 ---
 
 <a id="zh"></a>
 # @1-/scan : 增量扫描目录文件并使用 SQLite 记录元数据
 
-增量扫描目录文件，通过比对文件大小和修改时间检测变更，并同步至 SQLite 数据库中，最终返回有更新的相对路径列表。
+增量扫描目录文件，通过比对大小与修改时间检测变更，同步元数据至 SQLite 数据库，返回发生变更之相对路径列表。
 
 ## 功能介绍
 
-- **增量扫描**：仅处理新增、修改或删除的文件，避免冗余的文件系统读写，提升同步速度。
-- **路径压缩**：当相对路径长度小于等于 16 字节时保留原始字节；超出 16 字节则转换为 16 字节 MD5 值作为数据库主键，优化索引空间与查询性能。
-- **元数据压缩**：使用 Varint（可变字节整型）编码方式压缩存储文件大小和修改时间。
-- **事务安全**：将更新与删除操作合并在单个数据库事务中执行，确保数据一致性。
-- **灵活过滤**：支持通过自定义回调函数过滤指定类型的文件与目录。
+- **增量扫描**：比对文件大小与修改时间，仅对新增、修改或删除之文件执行操作，避免冗余读写，提升效率。
+- **路径压缩**：当相对路径长度不大于 16 字节时保留原始字节；超出 16 字节则转换为 16 字节 MD5 值作为主键，优化索引空间与查询性能。
+- **元数据压缩**：使用 Varint（可变字节整型）编码方式压缩存储文件大小与修改时间。
+- **事务安全**：更新与删除操作合并在数据库事务中执行，确保数据一致性。
+- **过滤规则**：支持传入自定义过滤函数，按需排除特定文件与目录。
 - **原生依赖**：基于 Bun 内置 `bun:sqlite` 模块，无需额外安装或编译数据库驱动。
 
 ## 使用演示
@@ -140,7 +150,7 @@ import scan from "@1-/scan";
 const dir = "./data";
 const db_path = "./scan_record.db";
 
-// 扫描目录并同步至 SQLite，返回发生变更的相对路径列表与更新函数
+// 扫描目录并同步至 SQLite，返回发生变更之相对路径列表与更新函数
 const [updated_paths, upsert] = await scan(dir, db_path);
 
 // 退出作用域时自动关闭数据库
@@ -148,13 +158,13 @@ using _upsert = upsert;
 
 console.log("更新文件列表：", updated_paths);
 
-// 更新已处理文件的元数据至数据库
+// 更新已处理文件元数据至数据库
 for (const rel_path of updated_paths) {
   await upsert(rel_path);
 }
 ```
 
-### 带有忽略规则的扫描
+### 过滤规则扫描
 
 ```javascript
 import scan from "@1-/scan";
@@ -162,7 +172,7 @@ import scan from "@1-/scan";
 const dir = "./data";
 const db_path = "./scan_record.db";
 
-// 忽略特定文件或目录
+// 过滤临时文件与特定配置
 const ignore = (kind, rel_path) => {
   return rel_path.startsWith("temp/") || rel_path === "config.json";
 };
@@ -177,36 +187,41 @@ for (const rel_path of updated_paths) {
 }
 ```
 
-## 设计思路
+### 批量存储模块使用
+
+```javascript
+import save from "@1-/scan/save.js";
+import sqlite from "@1-/scan/sqlite.js";
 
-系统主入口调用各个独立模块完成增量扫描与数据同步流程。
-
-```mermaid
-graph TD
-    Entry["_.js (主入口)"] -->|1. 初始化连接| Sqlite["sqlite.js"]
-    Entry -->|2. 加载已有记录| Load["load.js"]
-    Entry -->|3. 扫描文件系统并对比| DirWalk["dirWalk.js"]
-    DirWalk -->|调用| Walk["@1-/walk/walkRelIgnore"]
-    DirWalk -->|处理路径键| Hash["hash.js"]
-    Entry -->|4. 删除失效记录并返回更新函数| Trans["trans.js"]
-    Save["save.js (独立批量存储辅助模块)"] -->|事务保障| Trans
+const db = sqlite("./scan_record.db");
+
+// 批量更新与删除元数据
+save(db, [["file.txt", new Uint8Array([1, 2, 3]), 123, 1620000000]], [new Uint8Array([4, 5, 6])]);
+
+db.close();
 ```
 
+## 设计思路
+
+系统主入口调度各独立模块完成增量扫描与数据同步。
+
+![](https://i-01.eu.org/o9xJHQ8BDxiBtfjhCcTUtQ)
+
 1. **初始化连接 (`sqlite.js`)**：打开 SQLite 数据库，并配置自动释放连接机制。
-2. **加载记录 (`load.js`)**：若表不存在则自动创建，读取已记录的文件哈希、大小及修改时间，在内存中还原比对集合。
+2. **加载记录 (`load.js`)**：若数据表 `scanMtimeLen` 不存在则自动创建，读取已记录的文件哈希、大小及修改时间，在内存中还原比对集合。
 3. **文件系统扫描 (`dirWalk.js`)**：递归遍历目录，利用 `hash.js` 将路径映射为 16 字节键。对比当前文件与数据库元数据（利用 `@3-/vb` 进行压缩状态对比），筛选出新增和修改的文件。
-4. **删除与返回更新函数**：使用 `trans.js` 开启事务，批量删除已被移除的无效记录，并返回变更的相对路径列表与 `upsert` 函数，供调用者按需持久化数据。
-5. **独立批量存储辅助模块 (`save.js`)**：导出的独立工具模块，用于在单个事务中一次性批量写入与删除。
+4. **删除与返回更新函数**：使用 `trans.js` 开启事务，批量删除已被移除的记录，并返回变更的相对路径列表与 `upsert` 函数，供调用者持久化数据。
+5. **独立批量存储模块 (`save.js`)**：供外部调用的独立工具模块，用于在事务中批量写入与删除。
 
 ## 技术栈
 
-- **Bun**：JavaScript 运行时及测试框架。
-- **Bun SQLite**：内置的轻量级、高性能 SQLite 实现。
+- **Bun**：JavaScript 运行时与测试框架。
+- **Bun SQLite**：内置 SQLite 实现。
 - **@1-/walk**：支持过滤规则的目录递归遍历工具。
 - **@3-/vb**：Varint（可变字节）编码与解码器。
 - **@3-/binmap / @3-/binset**：针对二进制键优化的 Map 和 Set 容器。
 
-## 目录结构
+## 代码结构
 
 ```
 .
@@ -218,12 +233,16 @@ graph TD
 │   ├── save.js       # 独立导出的批量持久化与删除辅助函数
 │   ├── sqlite.js     # 创建并配置 SQLite 数据库实例
 │   └── trans.js      # 封装 SQLite 事务，提供异常回滚机制
-└── tests             # 单元测试模块
+└── tests             # 单元测试目录
 ```
 
 ## 历史故事
 
-SQLite 的诞生与军事应用密切相关。2000 年，D. Richard Hipp 在为美国海军陆战队设计导弹驱逐舰板载损害控制系统软件时，遇到商业数据库由于配置复杂、日常需要专业维护且一旦连接丢失便会导致整个软件瘫痪的问题。Hipp 随即着手设计了一套无需任何独立服务器、零配置且直接对本地文件进行读写的嵌入式数据库，这便是 SQLite。
+SQLite 的诞生源自海军军工项目。2000 年，D. Richard Hipp 为美国海军陆战队设计导弹驱逐舰板载损害控制软件时，遭遇商业数据库因配置复杂、日常维护繁琐且连接丢失即导致系统瘫痪之痛点。Hipp 随后设计出免服务器配置、直接读写本地文件之嵌入式数据库，即 SQLite。
+
+为了节省磁盘空间与降低读写延迟，SQLite 广泛应用了 Varint（可变字节整型）编码。在这种编码下，数值较小的整数仅占用 1 字节，只有大数值才会占用更多字节。本项目中对文件大小和修改时间采用同样的压缩设计，秉承了 SQLite 节省空间与高效之设计哲学。
+## 关于
+
+本库由 [WebC.site](https://webc.site) 开发。
 
-为极限节约磁盘空间 and 降低读写延迟，SQLite 广泛应用了 Varint（可变字节整型）编码。在这种编码下，数值较小的整数（如常见的文件大小、序列号）仅占用 1 个字节，只有大数值才会占用更多字节。本项目中对文件大小和修改时间采用同样的压缩设计，从而秉承了 SQLite 极致节约空间与高效率的系统设计哲学。
-../doc/zh/about.md
+[WebC.site](https://webc.site) : 面向人工智能的网站开发新范式

package/_.js

@@ -21,12 +21,12 @@ export default async (dir, db_path, ignore) => {
 
   if (to_delete.length > 0) {
     trans(db, () => {
-      const del = db.prepare("DELETE FROM file WHERE hash=?");
+      const del = db.prepare("DELETE FROM scanMtimeLen WHERE hash=?");
       to_delete.forEach((h) => del.run(h));
     });
   }
 
-  const insert = db.prepare("INSERT OR REPLACE INTO file(hash,size,mtime)VALUES(?,?,?)"),
+  const insert = db.prepare("INSERT OR REPLACE INTO scanMtimeLen(hash,size,mtime)VALUES(?,?,?)"),
     upsert = async (rel_path) => {
       const fp = join(dir, rel_path),
         { size, mtimeMs } = await stat(fp),

package/load.js

@@ -2,10 +2,10 @@ const SQLITE_ERROR = 1;
 
 export default (db) => {
   try {
-    return db.prepare("SELECT hash,size,mtime FROM file").all();
+    return db.prepare("SELECT hash,size,mtime FROM scanMtimeLen").all();
   } catch (err) {
     if (err.errno === SQLITE_ERROR) {
-      db.exec("CREATE TABLE file(hash PRIMARY KEY,size INT UNSIGNED,mtime INT UNSIGNED)");
+      db.exec("CREATE TABLE scanMtimeLen(hash PRIMARY KEY,size INT UNSIGNED,mtime INT UNSIGNED)");
       return [];
     }
     throw err;

package/package.json

@@ -1,13 +1,13 @@
 {
   "name": "@1-/scan",
-  "version": "0.1.4",
+  "version": "0.1.6",
   "description": "Incrementally scan directory files and track metadata in SQLite / 增量扫描目录文件并使用 SQLite 记录元数据",
   "keywords": [
-    "scan",
-    "incremental",
-    "sqlite",
     "directory",
-    "metadata"
+    "incremental",
+    "metadata",
+    "scan",
+    "sqlite"
   ],
   "homepage": "https://github.com/webc-site/npm/tree/main/scan",
   "license": "MulanPSL-2.0",
@@ -22,7 +22,7 @@
     "./*": "./*"
   },
   "dependencies": {
-    "@1-/walk": "^0.1.0",
+    "@1-/walk": "^0.1.1",
     "@3-/binmap": "^0.1.20",
     "@3-/binset": "^0.1.6",
     "@3-/int": "^0.1.1",

package/save.js

@@ -4,11 +4,13 @@ export default (db, to_update, to_delete) => {
   if (to_update.length > 0 || to_delete.length > 0) {
     trans(db, () => {
       if (to_update.length > 0) {
-        const insert = db.prepare("INSERT OR REPLACE INTO file(hash,size,mtime)VALUES(?,?,?)");
+        const insert = db.prepare(
+          "INSERT OR REPLACE INTO scanMtimeLen(hash,size,mtime)VALUES(?,?,?)",
+        );
         to_update.forEach(([_, h, size, mtime]) => insert.run(h, size, mtime));
       }
       if (to_delete.length > 0) {
-        const del = db.prepare("DELETE FROM file WHERE hash=?");
+        const del = db.prepare("DELETE FROM scanMtimeLen WHERE hash=?");
         to_delete.forEach((h) => del.run(h));
       }
     });