@1-/scan 0.1.5 → 0.1.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +25 -50
  2. package/package.json +6 -6
package/README.md CHANGED
@@ -9,10 +9,10 @@ Incrementally scans directory files, compares file sizes and modification times
9
9
 
10
10
  ## Features
11
11
 
12
- - **Incremental Scanning**: Processes only new, modified, or deleted files, avoiding redundant file system operations.
12
+ - **Incremental Scanning**: Compares file sizes and modification times to process only new, modified, or deleted files, avoiding redundant read/write operations.
13
13
  - **Key Optimization**: Stores relative paths within 16 bytes directly as raw bytes; hashes longer paths to 16-byte MD5 digests to optimize database index space and query performance.
14
14
  - **Metadata Compression**: Compresses file sizes and modification times using Varint (variable-length byte) encoding.
15
- - **Transactional Integrity**: Packages updates and deletions in database transactions to guarantee consistency.
15
+ - **Transactional Integrity**: Performs updates and deletions within database transactions to guarantee consistency.
16
16
  - **File Filtering**: Supports custom ignore callback functions to filter files and directories.
17
17
  - **Native Database**: Integrates Bun native `bun:sqlite` module, eliminating external database driver dependencies.
18
18
 
@@ -58,6 +58,7 @@ using _upsert = upsert;
58
58
 
59
59
  console.log("Synced. Updated files:", updated_paths);
60
60
 
61
+ // Update scanned file metadata in database
61
62
  for (const rel_path of updated_paths) {
62
63
  await upsert(rel_path);
63
64
  }
@@ -72,15 +73,7 @@ import sqlite from "@1-/scan/sqlite.js";
72
73
  const db = sqlite("./scan_record.db");
73
74
 
74
75
  // Bulk update and delete metadata
75
- save(
76
- db,
77
- [
78
- ["file.txt", new Uint8Array([1, 2, 3]), 123, 1620000000]
79
- ],
80
- [
81
- new Uint8Array([4, 5, 6])
82
- ]
83
- );
76
+ save(db, [["file.txt", new Uint8Array([1, 2, 3]), 123, 1620000000]], [new Uint8Array([4, 5, 6])]);
84
77
 
85
78
  db.close();
86
79
  ```
@@ -89,16 +82,7 @@ db.close();
89
82
 
90
83
  The main entry orchestrates independent modules to execute the incremental scanning and synchronization flow.
91
84
 
92
- ```mermaid
93
- graph TD
94
- Entry["_.js (Entry Point)"] -->|1. Initialize Connection| Sqlite["sqlite.js"]
95
- Entry -->|2. Load Existing Records| Load["load.js"]
96
- Entry -->|3. Walk & Compare Files| DirWalk["dirWalk.js"]
97
- DirWalk -->|Invoke| Walk["@1-/walk/walkRelIgnore"]
98
- DirWalk -->|Process Path Keys| Hash["hash.js"]
99
- Entry -->|4. Delete Absent & Return Upsert| Trans["trans.js"]
100
- Save["save.js (Independent Sync Helper)"] -->|Transaction Wrapper| Trans
101
- ```
85
+ ![](https://i-01.eu.org/KDY9Lax79XwPBKztEqhU5A)
102
86
 
103
87
  1. **Initialize Connection (`sqlite.js`)**: Opens SQLite database connection and configures automatic connection disposal.
104
88
  2. **Load Records (`load.js`)**: Automatically creates `scanMtimeLen` table if missing, retrieves existing file hashes, sizes, and modification times, and reconstructs reference set in memory.
@@ -114,7 +98,7 @@ graph TD
114
98
  - **@3-/vb**: Variable-length byte (Varint) encoder and decoder.
115
99
  - **@3-/binmap / @3-/binset**: Memory-efficient collections designed for binary keys.
116
100
 
117
- ## Directory Structure
101
+ ## Code Structure
118
102
 
119
103
  ```
120
104
  .
@@ -134,23 +118,27 @@ graph TD
134
118
  SQLite was created by D. Richard Hipp in 2000 while designing board software for US Navy guided-missile destroyers. The system originally depended on a commercial database that required constant database administration; a connection loss could stall the entire damage control application. To resolve this vulnerability, Hipp designed a serverless, zero-configuration embedded database that directly reads and writes local files—marking the birth of SQLite.
135
119
 
136
120
  To conserve disk space and reduce I/O overhead, SQLite utilizes Varint (variable-length integer) encoding for metadata storage. Under this scheme, small integers consume only 1 byte, while larger numbers scale dynamically. This library inherits that design philosophy, compressing file metadata into varints before storing it, ensuring minimal footprint and high sync performance.
137
- ../doc/en/about.md
121
+ ## About
122
+
123
+ This library is developed by [WebC.site](https://webc.site).
124
+
125
+ [WebC.site](https://webc.site): A new paradigm of web development for AI
138
126
 
139
127
  ---
140
128
 
141
129
  <a id="zh"></a>
142
130
  # @1-/scan : 增量扫描目录文件并使用 SQLite 记录元数据
143
131
 
144
- 增量扫描目录文件,比对大小与修改时间以检测变更,同步元数据至 SQLite 数据库,返回发生变更之相对路径列表。
132
+ 增量扫描目录文件,通过比对大小与修改时间检测变更,同步元数据至 SQLite 数据库,返回发生变更之相对路径列表。
145
133
 
146
134
  ## 功能介绍
147
135
 
148
- - **增量扫描**:处理新增、修改或删除之文件,避免冗余文件系统读写,提升同步效率。
149
- - **路径压缩**:相对路径长度不大于 16 字节时保留原始字节;超出 16 字节则转换为 16 字节 MD5 值作为主键,优化索引空间与查询性能。
136
+ - **增量扫描**:比对文件大小与修改时间,仅对新增、修改或删除之文件执行操作,避免冗余读写,提升效率。
137
+ - **路径压缩**:当相对路径长度不大于 16 字节时保留原始字节;超出 16 字节则转换为 16 字节 MD5 值作为主键,优化索引空间与查询性能。
150
138
  - **元数据压缩**:使用 Varint(可变字节整型)编码方式压缩存储文件大小与修改时间。
151
- - **事务安全**:将更新与删除操作合并在数据库事务中执行,确保数据一致性。
152
- - **文件过滤**:支持自定义过滤函数以排除特定文件与目录。
153
- - **原生依赖**:基于 Bun 内置 `bun:sqlite` 模块,免去安装与编译数据库驱动步骤。
139
+ - **事务安全**:更新与删除操作合并在数据库事务中执行,确保数据一致性。
140
+ - **过滤规则**:支持传入自定义过滤函数,按需排除特定文件与目录。
141
+ - **原生依赖**:基于 Bun 内置 `bun:sqlite` 模块,无需额外安装或编译数据库驱动。
154
142
 
155
143
  ## 使用演示
156
144
 
@@ -208,15 +196,7 @@ import sqlite from "@1-/scan/sqlite.js";
208
196
  const db = sqlite("./scan_record.db");
209
197
 
210
198
  // 批量更新与删除元数据
211
- save(
212
- db,
213
- [
214
- ["file.txt", new Uint8Array([1, 2, 3]), 123, 1620000000]
215
- ],
216
- [
217
- new Uint8Array([4, 5, 6])
218
- ]
219
- );
199
+ save(db, [["file.txt", new Uint8Array([1, 2, 3]), 123, 1620000000]], [new Uint8Array([4, 5, 6])]);
220
200
 
221
201
  db.close();
222
202
  ```
@@ -225,16 +205,7 @@ db.close();
225
205
 
226
206
  系统主入口调度各独立模块完成增量扫描与数据同步。
227
207
 
228
- ```mermaid
229
- graph TD
230
- Entry["_.js (主入口)"] -->|1. 初始化连接| Sqlite["sqlite.js"]
231
- Entry -->|2. 加载已有记录| Load["load.js"]
232
- Entry -->|3. 扫描文件系统并对比| DirWalk["dirWalk.js"]
233
- DirWalk -->|调用| Walk["@1-/walk/walkRelIgnore"]
234
- DirWalk -->|处理路径键| Hash["hash.js"]
235
- Entry -->|4. 删除失效记录并返回更新函数| Trans["trans.js"]
236
- Save["save.js (独立批量存储模块)"] -->|事务保障| Trans
237
- ```
208
+ ![](https://i-01.eu.org/o9xJHQ8BDxiBtfjhCcTUtQ)
238
209
 
239
210
  1. **初始化连接 (`sqlite.js`)**:打开 SQLite 数据库,并配置自动释放连接机制。
240
211
  2. **加载记录 (`load.js`)**:若数据表 `scanMtimeLen` 不存在则自动创建,读取已记录的文件哈希、大小及修改时间,在内存中还原比对集合。
@@ -250,7 +221,7 @@ graph TD
250
221
  - **@3-/vb**:Varint(可变字节)编码与解码器。
251
222
  - **@3-/binmap / @3-/binset**:针对二进制键优化的 Map 和 Set 容器。
252
223
 
253
- ## 目录结构
224
+ ## 代码结构
254
225
 
255
226
  ```
256
227
  .
@@ -270,4 +241,8 @@ graph TD
270
241
  SQLite 的诞生源自海军军工项目。2000 年,D. Richard Hipp 为美国海军陆战队设计导弹驱逐舰板载损害控制软件时,遭遇商业数据库因配置复杂、日常维护繁琐且连接丢失即导致系统瘫痪之痛点。Hipp 随后设计出免服务器配置、直接读写本地文件之嵌入式数据库,即 SQLite。
271
242
 
272
243
  为了节省磁盘空间与降低读写延迟,SQLite 广泛应用了 Varint(可变字节整型)编码。在这种编码下,数值较小的整数仅占用 1 字节,只有大数值才会占用更多字节。本项目中对文件大小和修改时间采用同样的压缩设计,秉承了 SQLite 节省空间与高效之设计哲学。
273
- ../doc/zh/about.md
244
+ ## 关于
245
+
246
+ 本库由 [WebC.site](https://webc.site) 开发。
247
+
248
+ [WebC.site](https://webc.site) : 面向人工智能的网站开发新范式
package/package.json CHANGED
@@ -1,13 +1,13 @@
1
1
  {
2
2
  "name": "@1-/scan",
3
- "version": "0.1.5",
3
+ "version": "0.1.6",
4
4
  "description": "Incrementally scan directory files and track metadata in SQLite / 增量扫描目录文件并使用 SQLite 记录元数据",
5
5
  "keywords": [
6
- "scan",
7
- "incremental",
8
- "sqlite",
9
6
  "directory",
10
- "metadata"
7
+ "incremental",
8
+ "metadata",
9
+ "scan",
10
+ "sqlite"
11
11
  ],
12
12
  "homepage": "https://github.com/webc-site/npm/tree/main/scan",
13
13
  "license": "MulanPSL-2.0",
@@ -22,7 +22,7 @@
22
22
  "./*": "./*"
23
23
  },
24
24
  "dependencies": {
25
- "@1-/walk": "^0.1.0",
25
+ "@1-/walk": "^0.1.1",
26
26
  "@3-/binmap": "^0.1.20",
27
27
  "@3-/binset": "^0.1.6",
28
28
  "@3-/int": "^0.1.1",