@1-/scan 0.1.8 → 0.1.9

package/README.md

@@ -3,7 +3,7 @@
 ---
 
 <a id="en"></a>
-# @1-/scan : SQLite-backed incremental directory scanner
+# @1-/scan : SQLite-backed incremental directory file scanner
 
 Incrementally scans directory files, compares file sizes and modification times to detect changes, synchronizes metadata to SQLite database, and returns list of changed relative paths.
 
@@ -13,7 +13,7 @@ Incrementally scans directory files, compares file sizes and modification times
 - **Key Length Optimization**: Stores raw bytes for paths up to 16 bytes. Converts longer paths into 16-byte MD5 hashes to optimize database index space and query performance.
 - **Memory Optimization**: Uses BinMap and BinSet to store binary keys in memory, avoiding string decoding overhead and reducing memory footprint.
 - **Transactional Integrity**: Performs metadata updates and deletions in database transactions to ensure consistency.
-- **Zero Configuration**: Integrates @1-/sqlite to initialize database schema and manage database connections automatically.
+- **Auto Configuration**: Integrates @1-/sqlite to initialize database schema and manage database connections automatically, updating .gitignore when new database is detected.
 
 ## 2. Usage
 
@@ -58,12 +58,12 @@ db.close();
 
 Main entry orchestrates modules to scan directories and synchronize metadata.
 
-![](https://fastly.jsdelivr.net/gh/webc-fs/-@-a/YhYlWMxhnYfN4bgT4FrA.svg)
+![](https://fastly.jsdelivr.net/gh/webc-fs/-@K0/4mOzfZ1TGQuIAaily4Jw.svg)
 
-1. **Initialize Connection**: Calls `@1-/sqlite` to open SQLite database.
-2. **Load Records**: `load.js` checks `scanMtimeLen` table, creates it if missing. Reads stored hashes, sizes, and modification times to restore memory mappings.
-3. **Compare Files**: Iterates over input file list. Maps paths to 16-byte binary keys via `@1-/hash`. Adds files with mismatched size or modification time to update list.
-4. **Delete and Return**: Deletes absent or unscanned records in transaction. Returns changed paths list and `upsert` function for persistence.
+1. **Initialize Connection**: Calls `@1-/sqlite` to open SQLite database. Updates `.gitignore` in the database directory if the database is newly created to prevent tracking.
+2. **Load Records**: `load.js` checks and creates `scanMtimeLen` table. Reads stored hashes, sizes, and modification times to restore memory mappings inside `BinMap`.
+3. **Compare Files**: `scan.js` iterates over input file list, calling `stat.js` for metadata and utilizing `@1-/hash` to map paths to 16-byte binary keys. Adds files with mismatched size or modification time to change list.
+4. **Delete and Return**: `rm.js` deletes absent or unscanned records in transaction. Returns changed paths list and `upsert` function (provided by `upsert.js`) for persistence, supporting automatic resource disposal.
 
 ## 4. Tech Stack
 
@@ -78,9 +78,13 @@ Main entry orchestrates modules to scan directories and synchronize metadata.
 ```text
 .
 ├── src
-│   ├── _.js          # Core flow controller
-│   ├── load.js       # Table schema initialization and loader
-│   └── save.js       # Bulk update and delete helper
+│   ├── _.js          # Core controller flow
+│   ├── load.js       # Table initialization and loading
+│   ├── rm.js         # Batch deletion of metadata
+│   ├── save.js       # Batch storage and updates
+│   ├── scan.js       # Scans and compares files
+│   ├── stat.js       # Retrieves file metadata and path hash
+│   └── upsert.js     # Single-record updates and auto-dispose
 └── tests             # Unit tests
 ```
 
@@ -98,17 +102,17 @@ This library is developed by [WebC.site](https://webc.site).
 ---
 
 <a id="zh"></a>
-# @1-/scan : 基于 SQLite 的目录增量扫描工具
+# @1-/scan : 基于 SQLite 的目录文件增量扫描器
 
 增量扫描目录文件，比对文件大小与修改时间检测变更，同步元数据至 SQLite 数据库，返回已变更相对路径列表。
 
 ## 1. 功能介绍
 
 - **增量扫描**：比对大小与修改时间，过滤未变更文件，减少磁盘读写。
-- **键长优化**：路径长度不大于 16 字节存储原始字节，超出 16 字节转换为 16 字节 MD5 值，优化索引空间与查询性能。
+- **键长优化**：路径长度不大于 16 字节时存储原始字节，超出 16 字节转换为 16 字节 MD5 值，优化索引空间与查询性能。
 - **内存优化**：使用 BinMap 与 BinSet 存储二进制键，避免字符串解码，降低内存占用。
 - **事务保障**：元数据变更与删除操作合并在数据库事务中执行，确保数据一致性。
-- **零配置**：集成 @1-/sqlite，自动初始化数据库表结构，管理数据库连接。
+- **自动配置**：集成 @1-/sqlite，自动初始化数据库表结构，并在检测到新数据库时自动更新 .gitignore。
 
 ## 2. 使用演示
 
@@ -153,12 +157,12 @@ db.close();
 
 主入口调度各模块，协作完成目录扫描与数据同步。
 
-![](https://fastly.jsdelivr.net/gh/webc-fs/-@e2/-ae7ErmK8GpR3UFyhddA.svg)
+![](https://fastly.jsdelivr.net/gh/webc-fs/-@EB/fMlrHP9fXkA64lXr2r2Q.svg)
 
-1. **初始化连接**：调用 `@1-/sqlite` 打开 SQLite 数据库。
-2. **加载记录**：`load.js` 检查 `scanMtimeLen` 表，表不存在则自动创建。读取已记录哈希、大小及修改时间，恢复内存映射。
-3. **比对文件**：遍历输入文件列表，利用 `@1-/hash` 将路径映射为 16 字节二进制键。若大小或修改时间不一致，则加入变更列表。
-4. **删除与返回**：在事务中批量删除已被物理移除或不再扫描记录。返回变更路径列表与 `upsert` 函数，供外部持久化。
+1. **初始化连接**：调用 `@1-/sqlite` 打开 SQLite 数据库。若数据库为新创建，自动更新所在目录的 `.gitignore` 阻断提交。
+2. **加载记录**：`load.js` 检查并创建 `scanMtimeLen` 表。读取已记录的哈希、大小及修改时间，恢复至内存映射 `BinMap` 中。
+3. **比对文件**：`scan.js` 遍历输入文件列表，调用 `stat.js` 获取元数据，并利用 `@1-/hash` 将路径映射为 16 字节二进制键。比对大小或修改时间，差异项归入变更列表。
+4. **删除与返回**：`rm.js` 在事务中批量删除物理移除或不再扫描的记录。返回变更路径列表与 `upsert` 函数（`upsert.js` 提供），用以更新数据库，支持自动释放。
 
 ## 4. 技术栈
 
@@ -168,14 +172,18 @@ db.close();
 - **@3-/vb**：Varint 变长整型编码与解码器。
 - **@3-/binmap / @3-/binset**：基于 Rust 与 WebAssembly 的高效二进制键容器。
 
-## 5. Code Structure
+## 5. 代码结构
 
 ```text
 .
 ├── src
 │   ├── _.js          # 核心控制流程
 │   ├── load.js       # 元数据表初始化与加载
-│   └── save.js       # 批量更新与删除工具
+│   ├── rm.js         # 批量删除元数据
+│   ├── save.js       # 批量存储元数据
+│   ├── scan.js       # 扫描与比对文件
+│   ├── stat.js       # 获取文件元数据及路径哈希
+│   └── upsert.js     # 逐个更新与自动关闭
 └── tests             # 单元测试
 ```
 

package/_.js

@@ -1,63 +1,20 @@
 import sqlite from "@1-/sqlite";
-import tx from "@1-/sqlite/tx.js";
 import { BinMap } from "@3-/binmap";
 import vbE from "@3-/vb/vbE.js";
-import load from "./load.js";
-import { stat as fsStat } from "node:fs/promises";
-import { join } from "node:path";
-import int from "@3-/int";
-import strmd5 from "@1-/hash/strmd5.js";
-import { BinSet } from "@3-/binset";
-import u8eq from "@3-/u8/u8eq.js";
 import { availableParallelism } from "node:os";
 import pLimit from "@3-/plimit";
-
-const stat = async (dir, rel_path) => {
-    const { size, mtimeMs: mtime_ms } = await fsStat(join(dir, rel_path));
-    return [size, int(mtime_ms), strmd5(rel_path)];
-  },
-  scanFiles = async (dir, files, existing, limit) => {
-    const scanned = new BinSet(),
-      update = [];
-    await Promise.all(
-      files.map((rel_path) =>
-        limit(async () => {
-          try {
-            const [size, mtime, hash] = await stat(dir, rel_path),
-              val = existing.get(hash);
-
-            scanned.add(hash);
-
-            if (!val || !u8eq(val, vbE([size, mtime]))) {
-              update.push(rel_path);
-            }
-          } catch {}
-        }),
-      ),
-    );
-    return [scanned, update];
-  },
-  rmHashes = (db, rm) => {
-    if (rm.length > 0) {
-      tx(db, () => {
-        const del = db.prepare("DELETE FROM scanMtimeLen WHERE hash=?");
-        rm.forEach((hash) => del.run(hash));
-      });
-    }
-  },
-  newUpsert = (db, dir) => {
-    const insert = db.prepare("INSERT OR REPLACE INTO scanMtimeLen(hash,size,mtime)VALUES(?,?,?)"),
-      upsert = async (rel_path) => {
-        try {
-          const [size, mtime, hash] = await stat(dir, rel_path);
-          insert.run(hash, size, mtime);
-        } catch {}
-      };
-    upsert[Symbol.dispose] = () => db.close();
-    return upsert;
-  };
+import upsertGitignore from "@1-/upsert_gitignore";
+import { existsSync } from "node:fs";
+import { join, dirname, basename } from "node:path";
+import load from "./load.js";
+import scan from "./scan.js";
+import rm from "./rm.js";
+import upsert from "./upsert.js";
 
 export default async (dir, db_path, files) => {
+  if (!existsSync(db_path)) {
+    upsertGitignore(join(dirname(db_path), ".gitignore"), basename(db_path));
+  }
   const db = sqlite(db_path),
     existing = new BinMap(),
     db_rows = load(db),
@@ -65,10 +22,10 @@ export default async (dir, db_path, files) => {
 
   db_rows.forEach(({ hash, size, mtime }) => existing.set(hash, vbE([size, mtime])));
 
-  const [scanned, update] = await scanFiles(dir, files, existing, limit),
-    rm = db_rows.filter(({ hash }) => !scanned.has(hash)).map(({ hash }) => hash);
+  const [scanned, update] = await scan(dir, files, existing, limit),
+    rm_hashes = db_rows.filter(({ hash }) => !scanned.has(hash)).map(({ hash }) => hash);
 
-  rmHashes(db, rm);
+  rm(db, rm_hashes);
 
-  return [update, newUpsert(db, dir)];
+  return [update, upsert(db, dir)];
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@1-/scan",
-  "version": "0.1.8",
+  "version": "0.1.9",
   "description": "Incrementally scan directory files and track metadata in SQLite / 增量扫描目录文件并使用 SQLite 记录元数据",
   "keywords": [
     "directory",
@@ -30,6 +30,7 @@
     "@3-/plimit": "^0.1.3",
     "@3-/u8": "^0.1.2",
     "@3-/utf8": "^0.1.1",
-    "@3-/vb": "^0.1.6"
+    "@3-/vb": "^0.1.6",
+    "@1-/upsert_gitignore": "^0.1.3"
   }
 }

package/rm.js

@@ -0,0 +1,10 @@
+import tx from "@1-/sqlite/tx.js";
+
+export default (db, rm) => {
+  if (rm.length > 0) {
+    tx(db, () => {
+      const del = db.prepare("DELETE FROM scanMtimeLen WHERE hash=?");
+      rm.forEach((hash) => del.run(hash));
+    });
+  }
+};

package/scan.js

@@ -0,0 +1,26 @@
+import { BinSet } from "@3-/binset";
+import u8eq from "@3-/u8/u8eq.js";
+import vbE from "@3-/vb/vbE.js";
+import stat from "./stat.js";
+
+export default async (dir, files, existing, limit) => {
+  const scanned = new BinSet(),
+    update = [];
+  await Promise.all(
+    files.map((rel_path) =>
+      limit(async () => {
+        try {
+          const [size, mtime, hash] = await stat(dir, rel_path),
+            val = existing.get(hash);
+
+          scanned.add(hash);
+
+          if (!val || !u8eq(val, vbE([size, mtime]))) {
+            update.push(rel_path);
+          }
+        } catch {}
+      }),
+    ),
+  );
+  return [scanned, update];
+};

package/stat.js

@@ -0,0 +1,9 @@
+import { stat as fsStat } from "node:fs/promises";
+import { join } from "node:path";
+import int from "@3-/int";
+import strmd5 from "@1-/hash/strmd5.js";
+
+export default async (dir, rel_path) => {
+  const { size, mtimeMs: mtime_ms } = await fsStat(join(dir, rel_path));
+  return [size, int(mtime_ms), strmd5(rel_path)];
+};

package/upsert.js

@@ -0,0 +1,13 @@
+import stat from "./stat.js";
+
+export default (db, dir) => {
+  const insert = db.prepare("INSERT OR REPLACE INTO scanMtimeLen(hash,size,mtime)VALUES(?,?,?)"),
+    upsert = async (rel_path) => {
+      try {
+        const [size, mtime, hash] = await stat(dir, rel_path);
+        insert.run(hash, size, mtime);
+      } catch {}
+    };
+  upsert[Symbol.dispose] = () => db.close();
+  return upsert;
+};