@1-/scan 0.1.7 → 0.1.9

package/README.md

@@ -3,17 +3,17 @@
 ---
 
 <a id="en"></a>
-# @1-/scan : Incrementally scan directory files and track metadata in SQLite
+# @1-/scan : SQLite-backed incremental directory file scanner
 
-Incrementally scans directory files, compares file sizes and modification times to detect changes, synchronizes metadata to SQLite database, and returns updated relative paths.
+Incrementally scans directory files, compares file sizes and modification times to detect changes, synchronizes metadata to SQLite database, and returns list of changed relative paths.
 
 ## 1. Features
 
-- **Incremental Scanning**: Compares file sizes and modification times to process only new, modified, or deleted files, avoiding redundant read/write operations.
-- **Key Optimization**: Stores relative paths within 16 bytes directly as raw bytes; hashes longer paths to 16-byte MD5 digests to optimize database index space and query performance.
-- **Memory Efficiency**: Utilizes BinMap and BinSet to store binary keys in memory, avoiding string decoding overhead and reducing memory footprint.
-- **Transactional Integrity**: Performs updates and deletions within database transactions to guarantee consistency.
-- **Configuration-free**: Built-in automatic table initialization and connection management via @1-/sqlite.
+- **Incremental Scan**: Compares size and modification time, filtering unchanged files to reduce disk I/O.
+- **Key Length Optimization**: Stores raw bytes for paths up to 16 bytes. Converts longer paths into 16-byte MD5 hashes to optimize database index space and query performance.
+- **Memory Optimization**: Uses BinMap and BinSet to store binary keys in memory, avoiding string decoding overhead and reducing memory footprint.
+- **Transactional Integrity**: Performs metadata updates and deletions in database transactions to ensure consistency.
+- **Auto Configuration**: Integrates @1-/sqlite to initialize database schema and manage database connections automatically, updating .gitignore when new database is detected.
 
 ## 2. Usage
 
@@ -26,10 +26,10 @@ const dir = "./data";
 const db_path = "./scan_record.db";
 const files = ["file1.txt", "file2.txt"];
 
-// Scan file list and sync metadata to SQLite, returning modified relative paths and upsert function
+// Scan file list, sync metadata to SQLite, return changed relative paths and upsert function
 const [updated_paths, upsert] = await scan(dir, db_path, files);
 
-// Auto-close database when exiting scope
+// Close database automatically when exiting scope
 using _ = upsert;
 
 console.log("Updated files:", updated_paths);
@@ -54,16 +54,16 @@ save(db, [["file.txt", new Uint8Array([1, 2, 3]), 123, 1620000000]], [new Uint8A
 db.close();
 ```
 
-## 3. Design Ideas
+## 3. Design
 
-The entry point orchestrates independent modules to execute the incremental scanning and synchronization flow.
+Main entry orchestrates modules to scan directories and synchronize metadata.
 
-![](https://i-01.eu.org/5UJ3b_gajqcCgVKaX5rqjw)
+![](https://fastly.jsdelivr.net/gh/webc-fs/-@K0/4mOzfZ1TGQuIAaily4Jw.svg)
 
-1. **Initialize Connection**: Invokes `@1-/sqlite` to open the SQLite database.
-2. **Load Records**: `load.js` checks if the `scanMtimeLen` table exists and creates it if missing. It retrieves stored hashes, sizes, and modification times, and reconstructs the memory mapping.
-3. **Compare Metadata**: Iterates over the input file list, mapping paths to 16-byte binary keys via `@1-/hash`. It adds files with size or modification time mismatches to the update list.
-4. **Delete and Return**: Deletes absent records in a transaction and returns the changed path list along with the `upsert` function for external persistence.
+1. **Initialize Connection**: Calls `@1-/sqlite` to open SQLite database. Updates `.gitignore` in the database directory if the database is newly created to prevent tracking.
+2. **Load Records**: `load.js` checks and creates `scanMtimeLen` table. Reads stored hashes, sizes, and modification times to restore memory mappings inside `BinMap`.
+3. **Compare Files**: `scan.js` iterates over input file list, calling `stat.js` for metadata and utilizing `@1-/hash` to map paths to 16-byte binary keys. Adds files with mismatched size or modification time to change list.
+4. **Delete and Return**: `rm.js` deletes absent or unscanned records in transaction. Returns changed paths list and `upsert` function (provided by `upsert.js`) for persistence, supporting automatic resource disposal.
 
 ## 4. Tech Stack
 
@@ -78,15 +78,19 @@ The entry point orchestrates independent modules to execute the incremental scan
 ```text
 .
 ├── src
-│   ├── _.js          # Core flow controller
-│   ├── load.js       # Table schema initialization and loader
-│   └── save.js       # Bulk update and delete helper
+│   ├── _.js          # Core controller flow
+│   ├── load.js       # Table initialization and loading
+│   ├── rm.js         # Batch deletion of metadata
+│   ├── save.js       # Batch storage and updates
+│   ├── scan.js       # Scans and compares files
+│   ├── stat.js       # Retrieves file metadata and path hash
+│   └── upsert.js     # Single-record updates and auto-dispose
 └── tests             # Unit tests
 ```
 
 ## 6. History
 
-SQLite was created by D. Richard Hipp in 2000 while designing board software for guided-missile destroyers. The system originally depended on a commercial database that required constant database administration; a connection loss could stall the entire damage control application. Hipp designed a serverless, zero-configuration embedded database that directly reads and writes local files, marking the birth of SQLite.
+SQLite was created by D. Richard Hipp in 2000 while designing board software for guided-missile destroyers. The system originally depended on commercial database that required constant database administration; connection loss could stall the entire damage control application. Hipp designed serverless, zero-configuration embedded database that directly reads and writes local files, marking the birth of SQLite.
 
 To conserve space and reduce latency, SQLite utilizes Varint (variable-length integer) encoding for metadata storage. Under this scheme, small integers consume only 1 byte, while larger numbers scale dynamically. This library inherits that design philosophy, compressing file metadata into varints for memory storage to ensure minimal footprint and high synchronization performance.
 ## About
@@ -98,17 +102,17 @@ This library is developed by [WebC.site](https://webc.site).
 ---
 
 <a id="zh"></a>
-# @1-/scan : 增量扫描目录文件并使用 SQLite 记录元数据
+# @1-/scan : 基于 SQLite 的目录文件增量扫描器
 
-增量扫描目录文件，比对大小与修改时间检测变更，同步元数据至 SQLite 数据库，返回发生变更的相对路径列表。
+增量扫描目录文件，比对文件大小与修改时间检测变更，同步元数据至 SQLite 数据库，返回已变更相对路径列表。
 
 ## 1. 功能介绍
 
 - **增量扫描**：比对大小与修改时间，过滤未变更文件，减少磁盘读写。
-- **路径键优化**：路径长度不大于 16 字节时存储原始字节，超出 16 字节则转换为 16 字节 MD5 值，优化索引空间与查询性能。
-- **内存优化**：在内存中使用 BinMap 与 BinSet 存储二进制哈希键，避免字符串解码开销，降低内存占用。
-- **事务保障**：变更及删除操作合并在数据库事务中执行，确保数据一致性。
-- **免除配置**：基于 @1-/sqlite，内置表初始化与连接管理，开箱即用。
+- **键长优化**：路径长度不大于 16 字节时存储原始字节，超出 16 字节转换为 16 字节 MD5 值，优化索引空间与查询性能。
+- **内存优化**：使用 BinMap 与 BinSet 存储二进制键，避免字符串解码，降低内存占用。
+- **事务保障**：元数据变更与删除操作合并在数据库事务中执行，确保数据一致性。
+- **自动配置**：集成 @1-/sqlite，自动初始化数据库表结构，并在检测到新数据库时自动更新 .gitignore。
 
 ## 2. 使用演示
 
@@ -121,15 +125,15 @@ const dir = "./data";
 const db_path = "./scan_record.db";
 const files = ["file1.txt", "file2.txt"];
 
-// 扫描文件列表并同步至 SQLite，返回发生变更的相对路径列表与更新函数
+// 扫描文件列表并同步至 SQLite，返回已变更的相对路径列表与更新函数
 const [updated_paths, upsert] = await scan(dir, db_path, files);
 
-// 退出作用域时自动关闭数据库
+// 退出作用域自动关闭数据库
 using _ = upsert;
 
 console.log("更新文件列表：", updated_paths);
 
-// 更新已处理文件元数据至数据库
+// 更新已处理文件的元数据至数据库
 for (const rel_path of updated_paths) {
   await upsert(rel_path);
 }
@@ -151,14 +155,14 @@ db.close();
 
 ## 3. 设计思路
 
-系统调度各模块完成增量扫描与数据同步。
+主入口调度各模块，协作完成目录扫描与数据同步。
 
-![](https://i-01.eu.org/qBVoq3Bhu1yHqHFfBcBTJg)
+![](https://fastly.jsdelivr.net/gh/webc-fs/-@EB/fMlrHP9fXkA64lXr2r2Q.svg)
 
-1. **初始化连接**：调用 `@1-/sqlite` 打开 SQLite 数据库。
-2. **加载记录**：`load.js` 检查 `scanMtimeLen` 表是否存在，若不存在则创建。读取已记录的哈希、大小及修改时间，在内存中恢复已存在的文件映射。
-3. **元数据比对**：遍历输入的文件列表，利用 `@1-/hash` 将路径映射为 16 字节二进制键。若大小或修改时间不一致，则加入变更列表。
-4. **删除与返回**：使用事务批量删除已被移除的记录（已存在于数据库中但不在本次扫描列表内的文件），并返回变更路径列表与 `upsert` 函数，供外部持久化。
+1. **初始化连接**：调用 `@1-/sqlite` 打开 SQLite 数据库。若数据库为新创建，自动更新所在目录的 `.gitignore` 阻断提交。
+2. **加载记录**：`load.js` 检查并创建 `scanMtimeLen` 表。读取已记录的哈希、大小及修改时间，恢复至内存映射 `BinMap` 中。
+3. **比对文件**：`scan.js` 遍历输入文件列表，调用 `stat.js` 获取元数据，并利用 `@1-/hash` 将路径映射为 16 字节二进制键。比对大小或修改时间，差异项归入变更列表。
+4. **删除与返回**：`rm.js` 在事务中批量删除物理移除或不再扫描的记录。返回变更路径列表与 `upsert` 函数（`upsert.js` 提供），用以更新数据库，支持自动释放。
 
 ## 4. 技术栈
 
@@ -175,7 +179,11 @@ db.close();
 ├── src
 │   ├── _.js          # 核心控制流程
 │   ├── load.js       # 元数据表初始化与加载
-│   └── save.js       # 批量更新与删除工具
+│   ├── rm.js         # 批量删除元数据
+│   ├── save.js       # 批量存储元数据
+│   ├── scan.js       # 扫描与比对文件
+│   ├── stat.js       # 获取文件元数据及路径哈希
+│   └── upsert.js     # 逐个更新与自动关闭
 └── tests             # 单元测试
 ```
 

package/_.js

@@ -1,59 +1,31 @@
 import sqlite from "@1-/sqlite";
-import tx from "@1-/sqlite/tx.js";
 import { BinMap } from "@3-/binmap";
 import vbE from "@3-/vb/vbE.js";
+import { availableParallelism } from "node:os";
+import pLimit from "@3-/plimit";
+import upsertGitignore from "@1-/upsert_gitignore";
+import { existsSync } from "node:fs";
+import { join, dirname, basename } from "node:path";
 import load from "./load.js";
-import { stat as fsStat } from "node:fs/promises";
-import { join } from "node:path";
-import int from "@3-/int";
-import strmd5 from "@1-/hash/strmd5.js";
-import { BinSet } from "@3-/binset";
-import u8eq from "@3-/u8/u8eq.js";
-
-const stat = async (dir, rel_path) => {
-  const { size, mtimeMs: mtime_ms } = await fsStat(join(dir, rel_path));
-  return [size, int(mtime_ms), strmd5(rel_path)];
-};
+import scan from "./scan.js";
+import rm from "./rm.js";
+import upsert from "./upsert.js";
 
 export default async (dir, db_path, files) => {
+  if (!existsSync(db_path)) {
+    upsertGitignore(join(dirname(db_path), ".gitignore"), basename(db_path));
+  }
   const db = sqlite(db_path),
     existing = new BinMap(),
     db_rows = load(db),
-    scanned = new BinSet(),
-    update = [];
+    limit = pLimit(availableParallelism());
 
   db_rows.forEach(({ hash, size, mtime }) => existing.set(hash, vbE([size, mtime])));
 
-  for (const rel_path of files) {
-    try {
-      const [size, mtime, hash] = await stat(dir, rel_path),
-        val = existing.get(hash);
-
-      scanned.add(hash);
-
-      if (!val || !u8eq(val, vbE([size, mtime]))) {
-        update.push(rel_path);
-      }
-    } catch {}
-  }
-
-  const rm = db_rows.filter(({ hash }) => !scanned.has(hash)).map(({ hash }) => hash),
-    insert = db.prepare("INSERT OR REPLACE INTO scanMtimeLen(hash,size,mtime)VALUES(?,?,?)"),
-    upsert = async (rel_path) => {
-      try {
-        const [size, mtime, hash] = await stat(dir, rel_path);
-        insert.run(hash, size, mtime);
-      } catch {}
-    };
-
-  if (rm.length > 0) {
-    tx(db, () => {
-      const del = db.prepare("DELETE FROM scanMtimeLen WHERE hash=?");
-      rm.forEach((hash) => del.run(hash));
-    });
-  }
+  const [scanned, update] = await scan(dir, files, existing, limit),
+    rm_hashes = db_rows.filter(({ hash }) => !scanned.has(hash)).map(({ hash }) => hash);
 
-  upsert[Symbol.dispose] = () => db.close();
+  rm(db, rm_hashes);
 
-  return [update, upsert];
+  return [update, upsert(db, dir)];
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@1-/scan",
-  "version": "0.1.7",
+  "version": "0.1.9",
   "description": "Incrementally scan directory files and track metadata in SQLite / 增量扫描目录文件并使用 SQLite 记录元数据",
   "keywords": [
     "directory",
@@ -23,12 +23,14 @@
   },
   "peerDependencies": {
     "@1-/hash": "^0.1.0",
-    "@1-/sqlite": "^0.1.0",
+    "@1-/sqlite": "^0.1.1",
     "@3-/binmap": "^0.1.20",
     "@3-/binset": "^0.1.6",
     "@3-/int": "^0.1.1",
+    "@3-/plimit": "^0.1.3",
     "@3-/u8": "^0.1.2",
     "@3-/utf8": "^0.1.1",
-    "@3-/vb": "^0.1.6"
+    "@3-/vb": "^0.1.6",
+    "@1-/upsert_gitignore": "^0.1.3"
   }
 }

package/rm.js

@@ -0,0 +1,10 @@
+import tx from "@1-/sqlite/tx.js";
+
+export default (db, rm) => {
+  if (rm.length > 0) {
+    tx(db, () => {
+      const del = db.prepare("DELETE FROM scanMtimeLen WHERE hash=?");
+      rm.forEach((hash) => del.run(hash));
+    });
+  }
+};

package/scan.js

@@ -0,0 +1,26 @@
+import { BinSet } from "@3-/binset";
+import u8eq from "@3-/u8/u8eq.js";
+import vbE from "@3-/vb/vbE.js";
+import stat from "./stat.js";
+
+export default async (dir, files, existing, limit) => {
+  const scanned = new BinSet(),
+    update = [];
+  await Promise.all(
+    files.map((rel_path) =>
+      limit(async () => {
+        try {
+          const [size, mtime, hash] = await stat(dir, rel_path),
+            val = existing.get(hash);
+
+          scanned.add(hash);
+
+          if (!val || !u8eq(val, vbE([size, mtime]))) {
+            update.push(rel_path);
+          }
+        } catch {}
+      }),
+    ),
+  );
+  return [scanned, update];
+};

package/stat.js

@@ -0,0 +1,9 @@
+import { stat as fsStat } from "node:fs/promises";
+import { join } from "node:path";
+import int from "@3-/int";
+import strmd5 from "@1-/hash/strmd5.js";
+
+export default async (dir, rel_path) => {
+  const { size, mtimeMs: mtime_ms } = await fsStat(join(dir, rel_path));
+  return [size, int(mtime_ms), strmd5(rel_path)];
+};

package/upsert.js

@@ -0,0 +1,13 @@
+import stat from "./stat.js";
+
+export default (db, dir) => {
+  const insert = db.prepare("INSERT OR REPLACE INTO scanMtimeLen(hash,size,mtime)VALUES(?,?,?)"),
+    upsert = async (rel_path) => {
+      try {
+        const [size, mtime, hash] = await stat(dir, rel_path);
+        insert.run(hash, size, mtime);
+      } catch {}
+    };
+  upsert[Symbol.dispose] = () => db.close();
+  return upsert;
+};