@1-/scan 0.1.9 → 0.1.10

package/README.md

@@ -5,6 +5,17 @@
 <a id="en"></a>
 # @1-/scan : SQLite-backed incremental directory file scanner
 
+- [@1-/scan : SQLite-backed incremental directory file scanner](#1-scan-sqlite-backed-incremental-directory-file-scanner)
+  - [1. Features](#1-features)
+  - [2. Usage](#2-usage)
+    - [Basic Incremental Scan](#basic-incremental-scan)
+    - [Bulk Storage Module](#bulk-storage-module)
+  - [3. Design](#3-design)
+  - [4. Tech Stack](#4-tech-stack)
+  - [5. Code Structure](#5-code-structure)
+  - [6. History](#6-history)
+  - [About](#about)
+
 Incrementally scans directory files, compares file sizes and modification times to detect changes, synchronizes metadata to SQLite database, and returns list of changed relative paths.
 
 ## 1. Features
@@ -23,11 +34,11 @@ Incrementally scans directory files, compares file sizes and modification times
 import scan from "@1-/scan";
 
 const dir = "./data";
-const db_path = "./scan_record.db";
+const db_dir = "./db";
 const files = ["file1.txt", "file2.txt"];
 
 // Scan file list, sync metadata to SQLite, return changed relative paths and upsert function
-const [updated_paths, upsert] = await scan(dir, db_path, files);
+const [updated_paths, upsert] = await scan(dir, db_dir, files);
 
 // Close database automatically when exiting scope
 using _ = upsert;
@@ -58,10 +69,10 @@ db.close();
 
 Main entry orchestrates modules to scan directories and synchronize metadata.
 
-![](https://fastly.jsdelivr.net/gh/webc-fs/-@K0/4mOzfZ1TGQuIAaily4Jw.svg)
+![](https://fastly.jsdelivr.net/gh/webc-fs/-@PE/plI9PPJK-kcEku6PWdAg.svg)
 
 1. **Initialize Connection**: Calls `@1-/sqlite` to open SQLite database. Updates `.gitignore` in the database directory if the database is newly created to prevent tracking.
-2. **Load Records**: `load.js` checks and creates `scanMtimeLen` table. Reads stored hashes, sizes, and modification times to restore memory mappings inside `BinMap`.
+2. **Load Records**: `load.js` checks and creates `scan` table. Reads stored paths, sizes, and modification times to restore memory mappings inside `BinMap`.
 3. **Compare Files**: `scan.js` iterates over input file list, calling `stat.js` for metadata and utilizing `@1-/hash` to map paths to 16-byte binary keys. Adds files with mismatched size or modification time to change list.
 4. **Delete and Return**: `rm.js` deletes absent or unscanned records in transaction. Returns changed paths list and `upsert` function (provided by `upsert.js`) for persistence, supporting automatic resource disposal.
 
@@ -93,17 +104,30 @@ Main entry orchestrates modules to scan directories and synchronize metadata.
 SQLite was created by D. Richard Hipp in 2000 while designing board software for guided-missile destroyers. The system originally depended on commercial database that required constant database administration; connection loss could stall the entire damage control application. Hipp designed serverless, zero-configuration embedded database that directly reads and writes local files, marking the birth of SQLite.
 
 To conserve space and reduce latency, SQLite utilizes Varint (variable-length integer) encoding for metadata storage. Under this scheme, small integers consume only 1 byte, while larger numbers scale dynamically. This library inherits that design philosophy, compressing file metadata into varints for memory storage to ensure minimal footprint and high synchronization performance.
+
 ## About
 
 This library is developed by [WebC.site](https://webc.site).
 
 [WebC.site](https://webc.site): A new paradigm of web development for AI
 
+
 ---
 
 <a id="zh"></a>
 # @1-/scan : 基于 SQLite 的目录文件增量扫描器
 
+- [@1-/scan : 基于 SQLite 的目录文件增量扫描器](#1-scan-基于-sqlite-的目录文件增量扫描器)
+  - [1. 功能介绍](#1-功能介绍)
+  - [2. 使用演示](#2-使用演示)
+    - [基础增量扫描](#基础增量扫描)
+    - [独立批量存储模块](#独立批量存储模块)
+  - [3. 设计思路](#3-设计思路)
+  - [4. 技术栈](#4-技术栈)
+  - [5. 代码结构](#5-代码结构)
+  - [6. 历史故事](#6-历史故事)
+  - [关于](#关于)
+
 增量扫描目录文件，比对文件大小与修改时间检测变更，同步元数据至 SQLite 数据库，返回已变更相对路径列表。
 
 ## 1. 功能介绍
@@ -122,11 +146,11 @@ This library is developed by [WebC.site](https://webc.site).
 import scan from "@1-/scan";
 
 const dir = "./data";
-const db_path = "./scan_record.db";
+const db_dir = "./db";
 const files = ["file1.txt", "file2.txt"];
 
 // 扫描文件列表并同步至 SQLite，返回已变更的相对路径列表与更新函数
-const [updated_paths, upsert] = await scan(dir, db_path, files);
+const [updated_paths, upsert] = await scan(dir, db_dir, files);
 
 // 退出作用域自动关闭数据库
 using _ = upsert;
@@ -157,10 +181,10 @@ db.close();
 
 主入口调度各模块，协作完成目录扫描与数据同步。
 
-![](https://fastly.jsdelivr.net/gh/webc-fs/-@EB/fMlrHP9fXkA64lXr2r2Q.svg)
+![](https://fastly.jsdelivr.net/gh/webc-fs/-@Bb/xl59c_lSLj5HJBw9gJ3g.svg)
 
 1. **初始化连接**：调用 `@1-/sqlite` 打开 SQLite 数据库。若数据库为新创建，自动更新所在目录的 `.gitignore` 阻断提交。
-2. **加载记录**：`load.js` 检查并创建 `scanMtimeLen` 表。读取已记录的哈希、大小及修改时间，恢复至内存映射 `BinMap` 中。
+2. **加载记录**：`load.js` 检查并创建 `scan` 表。读取已记录的路径、大小及修改时间，恢复至内存映射 `BinMap` 中。
 3. **比对文件**：`scan.js` 遍历输入文件列表，调用 `stat.js` 获取元数据，并利用 `@1-/hash` 将路径映射为 16 字节二进制键。比对大小或修改时间，差异项归入变更列表。
 4. **删除与返回**：`rm.js` 在事务中批量删除物理移除或不再扫描的记录。返回变更路径列表与 `upsert` 函数（`upsert.js` 提供），用以更新数据库，支持自动释放。
 
@@ -192,8 +216,10 @@ db.close();
 SQLite 的诞生源自导弹驱逐舰板载损害控制软件项目。2000 年，D. Richard Hipp 为美国海军设计该系统时，遭遇商业数据库因配置复杂、无法承受断连和崩溃之痛点。Hipp 随后设计出免服务器配置、直接读写本地文件之嵌入式数据库，即 SQLite。
 
 为节省空间与降低读写延迟，SQLite 广泛应用了 Varint（可变字节整型）编码。在这种编码下，小整数仅占用 1 字节，只有大数值才占用更多字节。本项目在内存存储设计中对文件大小与修改时间采用同样的压缩设计，契合 SQLite 节省空间与高效之设计哲学。
+
 ## 关于
 
 本库由 [WebC.site](https://webc.site) 开发。
 
 [WebC.site](https://webc.site) : 面向人工智能的网站开发新范式
+

package/_.js

@@ -1,31 +1,32 @@
-import sqlite from "@1-/sqlite";
-import { BinMap } from "@3-/binmap";
-import vbE from "@3-/vb/vbE.js";
 import { availableParallelism } from "node:os";
 import pLimit from "@3-/plimit";
-import upsertGitignore from "@1-/upsert_gitignore";
-import { existsSync } from "node:fs";
-import { join, dirname, basename } from "node:path";
-import load from "./load.js";
+import dbInit from "./dbInit.js";
+import { mtime, md5 } from "./load.js";
 import scan from "./scan.js";
 import rm from "./rm.js";
 import upsert from "./upsert.js";
 
-export default async (dir, db_path, files) => {
-  if (!existsSync(db_path)) {
-    upsertGitignore(join(dirname(db_path), ".gitignore"), basename(db_path));
-  }
-  const db = sqlite(db_path),
-    existing = new BinMap(),
-    db_rows = load(db),
-    limit = pLimit(availableParallelism());
+/*
+扫描指定目录下文件列表，比对缓存并做清理，返回更新列表和 upsert 存储函数
+dir: 扫描的目标目录
+db_dir: 数据库存放目录
+files: 待扫描的文件列表
+返回值: [update, upsert]
+  update: 发生变动需要更新的相对路径列表
+  upsert: 用于将新扫描记录保存至数据库的 dispose 异步函数
+*/
+export default async (dir, db_dir, files) => {
+  const [db_mtime, db_md5] = dbInit(db_dir),
+    existing_mtime = mtime(db_mtime),
+    existing_md5 = md5(db_md5),
+    limit = pLimit(availableParallelism()),
+    [scanned, update] = await scan(dir, files, existing_mtime, existing_md5, limit, db_mtime),
+    rmPaths = (existing) => [...existing.keys()].filter((path) => !scanned.has(path));
 
-  db_rows.forEach(({ hash, size, mtime }) => existing.set(hash, vbE([size, mtime])));
+  [
+    [db_mtime, existing_mtime],
+    [db_md5, existing_md5],
+  ].forEach(([db, existing]) => rm(db, "scan", rmPaths(existing)));
 
-  const [scanned, update] = await scan(dir, files, existing, limit),
-    rm_hashes = db_rows.filter(({ hash }) => !scanned.has(hash)).map(({ hash }) => hash);
-
-  rm(db, rm_hashes);
-
-  return [update, upsert(db, dir)];
+  return [update, upsert(db_mtime, db_md5, dir)];
 };

package/dbInit.js

@@ -0,0 +1,30 @@
+import sqlite from "@1-/sqlite";
+import upsertGitignore from "@1-/upsert_gitignore";
+import { existsSync } from "node:fs";
+import { join, basename } from "node:path";
+
+/*
+初始化/打开修改时间和 md5 的 sqlite 数据库，并将数据库文件加入 gitignore
+db_dir: 数据库存放目录
+返回值: [db_mtime, db_md5]
+*/
+export default (db_dir) => {
+  const li = ["mtime", "md5"],
+    path_li = li.map((x) => join(db_dir, x + ".sqlite"));
+
+  if (path_li.some((x) => !existsSync(x))) {
+    upsertGitignore(
+      join(db_dir, ".gitignore"),
+      path_li.map((x) => basename(x)),
+    );
+  }
+
+  const [db_mtime, db_md5] = path_li.map((x) => sqlite(x));
+
+  db_mtime.exec(
+    "CREATE TABLE IF NOT EXISTS scan(path PRIMARY KEY,size INT UNSIGNED,mtime INT UNSIGNED)",
+  );
+  db_md5.exec("CREATE TABLE IF NOT EXISTS scan(path PRIMARY KEY,md5 BINARY(16))");
+
+  return [db_mtime, db_md5];
+};

package/load.js

@@ -1,13 +1,22 @@
-const SQLITE_ERROR = 1;
+import { BinMap } from "@3-/binmap";
+import vbE from "@3-/vb/vbE.js";
 
-export default (db) => {
-  try {
-    return db.prepare("SELECT hash,size,mtime FROM scanMtimeLen").all();
-  } catch (err) {
-    if (err.errno === SQLITE_ERROR) {
-      db.exec("CREATE TABLE scanMtimeLen(hash PRIMARY KEY,size INT UNSIGNED,mtime INT UNSIGNED)");
-      return [];
-    }
-    throw err;
-  }
-};
+/*
+分别从修改时间数据库和 md5 数据库中加载记录，并创建内存 Map 缓存
+db: 数据库实例
+返回值: 包含记录的 Map 缓存
+*/
+export const mtime = (db) => {
+    const existing = new BinMap();
+    db.query("SELECT path,size,mtime FROM scan")
+      .all()
+      .forEach(({ path, size, mtime }) => existing.set(path, vbE([size, mtime])));
+    return existing;
+  },
+  md5 = (db) => {
+    const existing = new BinMap();
+    db.query("SELECT path,md5 FROM scan")
+      .all()
+      .forEach(({ path, md5 }) => existing.set(path, md5));
+    return existing;
+  };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@1-/scan",
-  "version": "0.1.9",
+  "version": "0.1.10",
   "description": "Incrementally scan directory files and track metadata in SQLite / 增量扫描目录文件并使用 SQLite 记录元数据",
   "keywords": [
     "directory",
@@ -21,16 +21,16 @@
     ".": "./_.js",
     "./*": "./*"
   },
-  "peerDependencies": {
+  "dependencies": {
     "@1-/hash": "^0.1.0",
+    "@1-/md5": "^0.1.1",
     "@1-/sqlite": "^0.1.1",
-    "@3-/binmap": "^0.1.20",
-    "@3-/binset": "^0.1.6",
+    "@1-/upsert_gitignore": "^0.1.6",
+    "@3-/binmap": "^0.1.21",
+    "@3-/binset": "^0.1.7",
     "@3-/int": "^0.1.1",
     "@3-/plimit": "^0.1.3",
     "@3-/u8": "^0.1.2",
-    "@3-/utf8": "^0.1.1",
-    "@3-/vb": "^0.1.6",
-    "@1-/upsert_gitignore": "^0.1.3"
+    "@3-/vb": "^0.1.6"
   }
 }

package/rm.js

@@ -1,10 +1,16 @@
 import tx from "@1-/sqlite/tx.js";
 
-export default (db, rm) => {
-  if (rm.length > 0) {
+/*
+批量删除记录
+db: sqlite 实例
+table: 数据表名
+rm: 待删除的路径 path 数组
+*/
+export default (db, table, rm) => {
+  if (rm.length) {
     tx(db, () => {
-      const del = db.prepare("DELETE FROM scanMtimeLen WHERE hash=?");
-      rm.forEach((hash) => del.run(hash));
+      const del = db.prepare("DELETE FROM " + table + " WHERE path=?");
+      rm.forEach((path) => del.run(path));
     });
   }
 };

package/scan.js

@@ -1,21 +1,47 @@
 import { BinSet } from "@3-/binset";
 import u8eq from "@3-/u8/u8eq.js";
 import vbE from "@3-/vb/vbE.js";
+import { join } from "node:path";
 import stat from "./stat.js";
+import pathMd5 from "@1-/md5/pathMd5.js";
 
-export default async (dir, files, existing, limit) => {
+/*
+扫描指定目录下的文件，并与已有记录进行对比
+dir: 扫描目录
+files: 相对路径列表
+existing: 已有的元数据 Map
+existing_md5: 已有的 md5 Map
+limit: 并发限制器
+db_mtime: 修改时间数据库实例
+返回值: [scanned, update]
+  scanned: 已扫描文件的 path 集合
+  update: 需要更新或新增的文件相对路径列表
+*/
+export default async (dir, files, existing, existing_md5, limit, db_mtime) => {
   const scanned = new BinSet(),
-    update = [];
+    update = [],
+    update_mtime = db_mtime.query("INSERT OR REPLACE INTO scan(path,size,mtime)VALUES(?,?,?)");
+
   await Promise.all(
     files.map((rel_path) =>
       limit(async () => {
         try {
-          const [size, mtime, hash] = await stat(dir, rel_path),
-            val = existing.get(hash);
+          const [size, mtime, path] = await stat(dir, rel_path),
+            val = existing.get(path);
 
-          scanned.add(hash);
+          scanned.add(path);
 
-          if (!val || !u8eq(val, vbE([size, mtime]))) {
+          if (!val) {
+            update.push(rel_path);
+          } else if (!u8eq(val, vbE([size, mtime]))) {
+            const old_md5 = existing_md5.get(path);
+            if (old_md5) {
+              const cur_md5 = await pathMd5(join(dir, rel_path));
+              if (u8eq(old_md5, cur_md5)) {
+                update_mtime.run(path, size, mtime);
+                return;
+              }
+            }
             update.push(rel_path);
           }
         } catch {}

package/stat.js

@@ -3,6 +3,12 @@ import { join } from "node:path";
 import int from "@3-/int";
 import strmd5 from "@1-/hash/strmd5.js";
 
+/*
+获取文件的大小、修改时间（秒）以及路径哈希
+dir: 基础目录
+rel_path: 相对路径
+返回值: [大小, 修改时间, 路径哈希]
+*/
 export default async (dir, rel_path) => {
   const { size, mtimeMs: mtime_ms } = await fsStat(join(dir, rel_path));
   return [size, int(mtime_ms), strmd5(rel_path)];

package/upsert.js

@@ -1,13 +1,28 @@
+import { join } from "node:path";
 import stat from "./stat.js";
+import pathMd5 from "@1-/md5/pathMd5.js";
 
-export default (db, dir) => {
-  const insert = db.prepare("INSERT OR REPLACE INTO scanMtimeLen(hash,size,mtime)VALUES(?,?,?)"),
+/*
+创建用于插入/更新文件元数据（大小、修改时间、md5）的函数，并在资源释放时关闭数据库
+db_mtime: 修改时间数据库实例
+db_md5: md5 数据库实例
+dir: 扫描目录
+返回值: upsert 异步函数 (带有 [Symbol.dispose] 方法)
+*/
+export default (db_mtime, db_md5, dir) => {
+  const insert_mtime = db_mtime.query("INSERT OR REPLACE INTO scan(path,size,mtime)VALUES(?,?,?)"),
+    insert_md5 = db_md5.query("INSERT OR REPLACE INTO scan(path,md5)VALUES(?,?)"),
     upsert = async (rel_path) => {
       try {
-        const [size, mtime, hash] = await stat(dir, rel_path);
-        insert.run(hash, size, mtime);
+        const [size, mtime, path] = await stat(dir, rel_path),
+          md5 = await pathMd5(join(dir, rel_path));
+        insert_mtime.run(path, size, mtime);
+        insert_md5.run(path, md5);
       } catch {}
     };
-  upsert[Symbol.dispose] = () => db.close();
+  upsert[Symbol.dispose] = () => {
+    db_mtime.close();
+    db_md5.close();
+  };
   return upsert;
 };

package/save.js

@@ -1,18 +0,0 @@
-import tx from "@1-/sqlite/tx.js";
-
-export default (db, update, rm) => {
-  if (update.length > 0 || rm.length > 0) {
-    tx(db, () => {
-      if (update.length > 0) {
-        const insert = db.prepare(
-          "INSERT OR REPLACE INTO scanMtimeLen(hash,size,mtime)VALUES(?,?,?)",
-        );
-        update.forEach(([_, h, size, mtime]) => insert.run(h, size, mtime));
-      }
-      if (rm.length > 0) {
-        const del = db.prepare("DELETE FROM scanMtimeLen WHERE hash=?");
-        rm.forEach((h) => del.run(h));
-      }
-    });
-  }
-};