@1-/scan 0.1.0

package/README.md

@@ -0,0 +1,145 @@
+[English](#en) | [中文](#zh)
+
+---
+
+<a id="en"></a>
+# @1-/scan : Incrementally scan directory files and track metadata in SQLite
+
+Incrementally scans directory files, tracks file sizes and modification times, and synchronizes status into a SQLite database.
+
+## Features
+
+- **Incremental Scanning**: Detects and updates only new, modified, or deleted files, avoiding redundant file system operations
+- **Space-Efficient Storage**: Employs Varint compression (`@3-/vb`) to serialize and compare file sizes and modification times
+- **Smart Path Key**: Stores short relative paths (≤ 16 bytes) as raw binary to preserve readability, while hashing longer paths to 16-byte MD5 digests to optimize index performance
+- **Database Synchronization**: Synchronizes updates and deletions in a single atomic transaction
+- **Ignore Pattern Support**: Integrates ignore rules dynamically during traversal
+
+## Usage
+
+```javascript
+import scan from "@1-/scan";
+
+const directoryPath = "./src";
+const sqliteDbPath = "./files.db";
+
+// Scan directory and sync records into SQLite
+await scan(directoryPath, sqliteDbPath);
+```
+
+## Design Ideas
+
+Execution flow of modules:
+
+```mermaid
+graph TD
+    Entry["_.js (Main)"] -->|Open database| Sqlite[sqlite.js]
+    Entry -->|Load existing records| FileAll[fileAll.js]
+    Entry -->|Walk and compare| DirWalk[dirWalk.js]
+    DirWalk -->|Traverse files| Walk["@1-/walk/walkRelIgnore"]
+    DirWalk -->|Optimize keys| Hash[hash.js]
+    Entry -->|Apply modifications| FileWrite[fileWrite.js]
+    FileWrite -->|Wrap transaction| Trans[trans.js]
+```
+
+## Tech Stack
+
+- **Bun**: Runtime and test runner
+- **SQLite**: Local relational database engine
+- **@1-/walk**: Directory walker with ignore support
+- **@3-/vb**: Variable-length byte encoder
+- **@3-/binmap** / **@3-/binset**: Efficient binary collection structures
+
+## Directory Structure
+
+```
+.
+├── src
+│   ├── _.js          # Entry point orchestrating the scanning and sync process
+│   ├── dirWalk.js    # Recursively scans files and filters modified ones
+│   ├── fileAll.js    # Retrieves database records and initializes schema
+│   ├── fileWrite.js  # Performs bulk database inserts and deletes
+│   ├── hash.js       # Processes path keys into raw bytes or MD5 digests
+│   ├── sqlite.js     # Manages SQLite database connection and disposal
+│   └── trans.js      # Wraps operations inside an SQL transaction
+└── tests             # Test suites
+```
+
+## History
+
+SQLite was designed in 2000 by D. Richard Hipp while working on a US Navy damage control system. The application originally relied on an Informix database, which required extensive database administration. Hipp designed SQLite to be a serverless, self-contained library requiring zero configuration, allowing the software to function reliably even when database services were unavailable.
+
+To optimize space inside the database file, SQLite internally uses variable-length integers (Varints) to compress metadata. This project adopts similar techniques—compressing file size and modification time into varints before storage—inheriting the SQLite philosophy of minimalism and space efficiency for local file synchronization.
+../doc/en/about.md
+
+---
+
+<a id="zh"></a>
+# @1-/scan : 增量扫描目录文件并使用 SQLite 记录元数据
+
+增量扫描指定目录，比对并同步文件大小与修改时间，将记录存入 SQLite 数据库。
+
+## 功能介绍
+
+- **增量扫描**：仅处理新增、修改或删除的文件，避免冗余文件操作
+- **紧凑存储**：使用可变字节码（Varint）压缩技术（`@3-/vb`）比对并保存文件大小和修改时间
+- **智能哈希**：短相对路径（≤ 16 字节）保留原始字节，长相对路径哈希为 16 字节 MD5，优化数据库索引效率
+- **事务同步**：更新与删除操作合并至单次数据库事务，确保一致性
+- **规则过滤**：基于 `@1-/walk` 的忽略规则过滤特定文件与目录
+
+## 使用演示
+
+```javascript
+import scan from "@1-/scan";
+
+const directoryPath = "./src";
+const sqliteDbPath = "./files.db";
+
+// 扫描目录并同步至 SQLite 数据库
+await scan(directoryPath, sqliteDbPath);
+```
+
+## 设计思路
+
+模块调用流程如下：
+
+```mermaid
+graph TD
+    Entry["_.js (主入口)"] -->|打开数据库| Sqlite[sqlite.js]
+    Entry -->|载入已有记录| FileAll[fileAll.js]
+    Entry -->|遍历并比对| DirWalk[dirWalk.js]
+    DirWalk -->|扫描文件系统| Walk["@1-/walk/walkRelIgnore"]
+    DirWalk -->|计算路径哈希| Hash[hash.js]
+    Entry -->|写入变更数据| FileWrite[fileWrite.js]
+    FileWrite -->|执行事务控制| Trans[trans.js]
+```
+
+## 技术栈
+
+- **Bun**: 运行环境与测试工具
+- **SQLite**: 本地关系型数据库
+- **@1-/walk**: 支持忽略规则的目录遍历工具
+- **@3-/vb**: 可变长度整型编码器
+- **@3-/binmap** / **@3-/binset**: 二进制哈希键容器
+
+## 目录结构
+
+```
+.
+├── src
+│   ├── _.js          # 主入口，统筹扫描与同步逻辑
+│   ├── dirWalk.js    # 递归遍历目录，比对筛选出变更文件
+│   ├── fileAll.js    # 读取数据库中全部记录，初始化数据表
+│   ├── fileWrite.js  # 事务内执行批量插入与删除
+│   ├── hash.js       # 计算相对路径哈希值或保留原始字节
+│   ├── sqlite.js     # 管理 SQLite 数据库连接及资源释放
+│   └── trans.js      # 封装数据库事务控制
+└── tests             # 测试目录
+```
+
+## 历史故事
+
+SQLite 由 D. Richard Hipp 于 2000 年为美国海军驱逐舰的控制系统编写。当时系统采用的商业数据库需繁琐的系统管理，一旦数据库故障系统便无法运行。Hipp 因而设计出无服务器、零配置且直接读写单文件的 SQLite。
+
+为极限节约存储空间，SQLite 内部大量采用可变长度整数（Varint）编码。本项目同样引入 Varint 压缩算法，对文件大小与修改时间做高效编码后再作比对存储，延续了 SQLite 追求极致性能与紧凑空间的优良传统。
+../doc/zh/about.md

package/_.js

@@ -0,0 +1,26 @@
+import { BinMap } from "@3-/binmap";
+import vbE from "@3-/vb/vbE.js";
+import sqlite from "./sqlite.js";
+import fileAll from "./fileAll.js";
+import dirWalk from "./dirWalk.js";
+import fileWrite from "./fileWrite.js";
+
+export default async (dir, db_path) => {
+  using db = sqlite(db_path);
+  const existing = new BinMap(),
+    db_rows = fileAll(db);
+
+  for (const row of db_rows) {
+    existing.set(row.hash, vbE([row.size, row.mtime]));
+  }
+
+  const [scanned, to_update] = await dirWalk(dir, existing),
+    to_delete = [];
+  for (const row of db_rows) {
+    if (!scanned.has(row.hash)) {
+      to_delete.push(row.hash);
+    }
+  }
+
+  fileWrite(db, to_update, to_delete);
+};

package/dirWalk.js

@@ -0,0 +1,32 @@
+import { stat } from "node:fs/promises";
+import { join } from "node:path";
+import { FILE } from "@1-/walk";
+import walkRelIgnore from "@1-/walk/walkRelIgnore.js";
+import { BinSet } from "@3-/binset";
+import u8eq from "@3-/u8/u8eq.js";
+import vbE from "@3-/vb/vbE.js";
+import int from "@3-/int";
+import hash from "./hash.js";
+
+export default async (dir, existing) => {
+  const scanned = new BinSet(),
+    to_update = [];
+
+  await walkRelIgnore(dir, async (kind, rel_path) => {
+    if (kind === FILE) {
+      const { size, mtimeMs } = await stat(join(dir, rel_path)),
+        mtime = int(mtimeMs),
+        h = hash(rel_path);
+
+      scanned.add(h);
+
+      const val = existing.get(h);
+      if (val && u8eq(val, vbE([size, mtime]))) {
+        return;
+      }
+      to_update.push([h, size, mtime]);
+    }
+  });
+
+  return [scanned, to_update];
+};

package/fileAll.js

@@ -0,0 +1,13 @@
+const SQLITE_ERROR = 1;
+
+export default (db) => {
+  try {
+    return db.prepare("SELECT hash,size,mtime FROM file").all();
+  } catch (err) {
+    if (err.errcode === SQLITE_ERROR) {
+      db.exec("CREATE TABLE file(hash PRIMARY KEY,size INT UNSIGNED,mtime INT UNSIGNED)");
+      return [];
+    }
+    throw err;
+  }
+};

package/fileWrite.js

@@ -0,0 +1,20 @@
+import trans from "./trans.js";
+
+export default (db, to_update, to_delete) => {
+  if (to_update.length > 0 || to_delete.length > 0) {
+    trans(db, () => {
+      if (to_update.length > 0) {
+        const insert = db.prepare("INSERT OR REPLACE INTO file(hash,size,mtime)VALUES(?,?,?)");
+        for (const record of to_update) {
+          insert.run(...record);
+        }
+      }
+      if (to_delete.length > 0) {
+        const del = db.prepare("DELETE FROM file WHERE hash=?");
+        for (const h of to_delete) {
+          del.run(h);
+        }
+      }
+    });
+  }
+};

package/hash.js

@@ -0,0 +1,7 @@
+import { createHash } from "node:crypto";
+import utf8e from "@3-/utf8/utf8e.js";
+
+export default (str) => {
+  const buf = utf8e(str);
+  return buf.length <= 16 ? buf : createHash("md5").update(buf).digest();
+};

package/package.json

@@ -0,0 +1,33 @@
+{
+  "name": "@1-/scan",
+  "version": "0.1.0",
+  "description": "Incrementally scan directory files and track metadata in SQLite / 增量扫描目录文件并使用 SQLite 记录元数据",
+  "keywords": [
+    "scan",
+    "incremental",
+    "sqlite",
+    "directory",
+    "metadata"
+  ],
+  "homepage": "https://github.com/webc-site/npm/tree/main/scan",
+  "license": "MulanPSL-2.0",
+  "author": "x-at-01@googlegroups.com",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/webc-site/npm.git"
+  },
+  "type": "module",
+  "exports": {
+    ".": "./_.js",
+    "./*": "./*"
+  },
+  "dependencies": {
+    "@1-/walk": "^0.1.0",
+    "@3-/binmap": "^0.1.20",
+    "@3-/binset": "^0.1.6",
+    "@3-/int": "^0.1.1",
+    "@3-/u8": "^0.1.2",
+    "@3-/utf8": "^0.1.1",
+    "@3-/vb": "^0.1.6"
+  }
+}

package/sqlite.js

@@ -0,0 +1,7 @@
+const { DatabaseSync } = await import("node:sqlite");
+
+export default (db_path) => {
+  const db = new DatabaseSync(db_path);
+  db[Symbol.dispose] = () => db.close();
+  return db;
+};

package/trans.js

@@ -0,0 +1,11 @@
+export default (db, run) => {
+  db.exec("BEGIN");
+  try {
+    const res = run();
+    db.exec("COMMIT");
+    return res;
+  } catch (err) {
+    db.exec("ROLLBACK");
+    throw err;
+  }
+};