@1-/scan 0.1.5 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +25 -50
- package/package.json +6 -6
package/README.md
CHANGED
|
@@ -9,10 +9,10 @@ Incrementally scans directory files, compares file sizes and modification times
|
|
|
9
9
|
|
|
10
10
|
## Features
|
|
11
11
|
|
|
12
|
-
- **Incremental Scanning**:
|
|
12
|
+
- **Incremental Scanning**: Compares file sizes and modification times to process only new, modified, or deleted files, avoiding redundant read/write operations.
|
|
13
13
|
- **Key Optimization**: Stores relative paths within 16 bytes directly as raw bytes; hashes longer paths to 16-byte MD5 digests to optimize database index space and query performance.
|
|
14
14
|
- **Metadata Compression**: Compresses file sizes and modification times using Varint (variable-length byte) encoding.
|
|
15
|
-
- **Transactional Integrity**:
|
|
15
|
+
- **Transactional Integrity**: Performs updates and deletions within database transactions to guarantee consistency.
|
|
16
16
|
- **File Filtering**: Supports custom ignore callback functions to filter files and directories.
|
|
17
17
|
- **Native Database**: Integrates Bun native `bun:sqlite` module, eliminating external database driver dependencies.
|
|
18
18
|
|
|
@@ -58,6 +58,7 @@ using _upsert = upsert;
|
|
|
58
58
|
|
|
59
59
|
console.log("Synced. Updated files:", updated_paths);
|
|
60
60
|
|
|
61
|
+
// Update scanned file metadata in database
|
|
61
62
|
for (const rel_path of updated_paths) {
|
|
62
63
|
await upsert(rel_path);
|
|
63
64
|
}
|
|
@@ -72,15 +73,7 @@ import sqlite from "@1-/scan/sqlite.js";
|
|
|
72
73
|
const db = sqlite("./scan_record.db");
|
|
73
74
|
|
|
74
75
|
// Bulk update and delete metadata
|
|
75
|
-
save(
|
|
76
|
-
db,
|
|
77
|
-
[
|
|
78
|
-
["file.txt", new Uint8Array([1, 2, 3]), 123, 1620000000]
|
|
79
|
-
],
|
|
80
|
-
[
|
|
81
|
-
new Uint8Array([4, 5, 6])
|
|
82
|
-
]
|
|
83
|
-
);
|
|
76
|
+
save(db, [["file.txt", new Uint8Array([1, 2, 3]), 123, 1620000000]], [new Uint8Array([4, 5, 6])]);
|
|
84
77
|
|
|
85
78
|
db.close();
|
|
86
79
|
```
|
|
@@ -89,16 +82,7 @@ db.close();
|
|
|
89
82
|
|
|
90
83
|
The main entry orchestrates independent modules to execute the incremental scanning and synchronization flow.
|
|
91
84
|
|
|
92
|
-
|
|
93
|
-
graph TD
|
|
94
|
-
Entry["_.js (Entry Point)"] -->|1. Initialize Connection| Sqlite["sqlite.js"]
|
|
95
|
-
Entry -->|2. Load Existing Records| Load["load.js"]
|
|
96
|
-
Entry -->|3. Walk & Compare Files| DirWalk["dirWalk.js"]
|
|
97
|
-
DirWalk -->|Invoke| Walk["@1-/walk/walkRelIgnore"]
|
|
98
|
-
DirWalk -->|Process Path Keys| Hash["hash.js"]
|
|
99
|
-
Entry -->|4. Delete Absent & Return Upsert| Trans["trans.js"]
|
|
100
|
-
Save["save.js (Independent Sync Helper)"] -->|Transaction Wrapper| Trans
|
|
101
|
-
```
|
|
85
|
+

|
|
102
86
|
|
|
103
87
|
1. **Initialize Connection (`sqlite.js`)**: Opens SQLite database connection and configures automatic connection disposal.
|
|
104
88
|
2. **Load Records (`load.js`)**: Automatically creates `scanMtimeLen` table if missing, retrieves existing file hashes, sizes, and modification times, and reconstructs reference set in memory.
|
|
@@ -114,7 +98,7 @@ graph TD
|
|
|
114
98
|
- **@3-/vb**: Variable-length byte (Varint) encoder and decoder.
|
|
115
99
|
- **@3-/binmap / @3-/binset**: Memory-efficient collections designed for binary keys.
|
|
116
100
|
|
|
117
|
-
##
|
|
101
|
+
## Code Structure
|
|
118
102
|
|
|
119
103
|
```
|
|
120
104
|
.
|
|
@@ -134,23 +118,27 @@ graph TD
|
|
|
134
118
|
SQLite was created by D. Richard Hipp in 2000 while designing board software for US Navy guided-missile destroyers. The system originally depended on a commercial database that required constant database administration; a connection loss could stall the entire damage control application. To resolve this vulnerability, Hipp designed a serverless, zero-configuration embedded database that directly reads and writes local files—marking the birth of SQLite.
|
|
135
119
|
|
|
136
120
|
To conserve disk space and reduce I/O overhead, SQLite utilizes Varint (variable-length integer) encoding for metadata storage. Under this scheme, small integers consume only 1 byte, while larger numbers scale dynamically. This library inherits that design philosophy, compressing file metadata into varints before storing it, ensuring minimal footprint and high sync performance.
|
|
137
|
-
|
|
121
|
+
## About
|
|
122
|
+
|
|
123
|
+
This library is developed by [WebC.site](https://webc.site).
|
|
124
|
+
|
|
125
|
+
[WebC.site](https://webc.site): A new paradigm of web development for AI
|
|
138
126
|
|
|
139
127
|
---
|
|
140
128
|
|
|
141
129
|
<a id="zh"></a>
|
|
142
130
|
# @1-/scan : 增量扫描目录文件并使用 SQLite 记录元数据
|
|
143
131
|
|
|
144
|
-
|
|
132
|
+
增量扫描目录文件,通过比对大小与修改时间检测变更,同步元数据至 SQLite 数据库,返回发生变更之相对路径列表。
|
|
145
133
|
|
|
146
134
|
## 功能介绍
|
|
147
135
|
|
|
148
|
-
-
|
|
149
|
-
-
|
|
136
|
+
- **增量扫描**:比对文件大小与修改时间,仅对新增、修改或删除之文件执行操作,避免冗余读写,提升效率。
|
|
137
|
+
- **路径压缩**:当相对路径长度不大于 16 字节时保留原始字节;超出 16 字节则转换为 16 字节 MD5 值作为主键,优化索引空间与查询性能。
|
|
150
138
|
- **元数据压缩**:使用 Varint(可变字节整型)编码方式压缩存储文件大小与修改时间。
|
|
151
|
-
-
|
|
152
|
-
-
|
|
153
|
-
- **原生依赖**:基于 Bun 内置 `bun:sqlite`
|
|
139
|
+
- **事务安全**:更新与删除操作合并在数据库事务中执行,确保数据一致性。
|
|
140
|
+
- **过滤规则**:支持传入自定义过滤函数,按需排除特定文件与目录。
|
|
141
|
+
- **原生依赖**:基于 Bun 内置 `bun:sqlite` 模块,无需额外安装或编译数据库驱动。
|
|
154
142
|
|
|
155
143
|
## 使用演示
|
|
156
144
|
|
|
@@ -208,15 +196,7 @@ import sqlite from "@1-/scan/sqlite.js";
|
|
|
208
196
|
const db = sqlite("./scan_record.db");
|
|
209
197
|
|
|
210
198
|
// 批量更新与删除元数据
|
|
211
|
-
save(
|
|
212
|
-
db,
|
|
213
|
-
[
|
|
214
|
-
["file.txt", new Uint8Array([1, 2, 3]), 123, 1620000000]
|
|
215
|
-
],
|
|
216
|
-
[
|
|
217
|
-
new Uint8Array([4, 5, 6])
|
|
218
|
-
]
|
|
219
|
-
);
|
|
199
|
+
save(db, [["file.txt", new Uint8Array([1, 2, 3]), 123, 1620000000]], [new Uint8Array([4, 5, 6])]);
|
|
220
200
|
|
|
221
201
|
db.close();
|
|
222
202
|
```
|
|
@@ -225,16 +205,7 @@ db.close();
|
|
|
225
205
|
|
|
226
206
|
系统主入口调度各独立模块完成增量扫描与数据同步。
|
|
227
207
|
|
|
228
|
-
|
|
229
|
-
graph TD
|
|
230
|
-
Entry["_.js (主入口)"] -->|1. 初始化连接| Sqlite["sqlite.js"]
|
|
231
|
-
Entry -->|2. 加载已有记录| Load["load.js"]
|
|
232
|
-
Entry -->|3. 扫描文件系统并对比| DirWalk["dirWalk.js"]
|
|
233
|
-
DirWalk -->|调用| Walk["@1-/walk/walkRelIgnore"]
|
|
234
|
-
DirWalk -->|处理路径键| Hash["hash.js"]
|
|
235
|
-
Entry -->|4. 删除失效记录并返回更新函数| Trans["trans.js"]
|
|
236
|
-
Save["save.js (独立批量存储模块)"] -->|事务保障| Trans
|
|
237
|
-
```
|
|
208
|
+

|
|
238
209
|
|
|
239
210
|
1. **初始化连接 (`sqlite.js`)**:打开 SQLite 数据库,并配置自动释放连接机制。
|
|
240
211
|
2. **加载记录 (`load.js`)**:若数据表 `scanMtimeLen` 不存在则自动创建,读取已记录的文件哈希、大小及修改时间,在内存中还原比对集合。
|
|
@@ -250,7 +221,7 @@ graph TD
|
|
|
250
221
|
- **@3-/vb**:Varint(可变字节)编码与解码器。
|
|
251
222
|
- **@3-/binmap / @3-/binset**:针对二进制键优化的 Map 和 Set 容器。
|
|
252
223
|
|
|
253
|
-
##
|
|
224
|
+
## 代码结构
|
|
254
225
|
|
|
255
226
|
```
|
|
256
227
|
.
|
|
@@ -270,4 +241,8 @@ graph TD
|
|
|
270
241
|
SQLite 的诞生源自海军军工项目。2000 年,D. Richard Hipp 为美国海军陆战队设计导弹驱逐舰板载损害控制软件时,遭遇商业数据库因配置复杂、日常维护繁琐且连接丢失即导致系统瘫痪之痛点。Hipp 随后设计出免服务器配置、直接读写本地文件之嵌入式数据库,即 SQLite。
|
|
271
242
|
|
|
272
243
|
为了节省磁盘空间与降低读写延迟,SQLite 广泛应用了 Varint(可变字节整型)编码。在这种编码下,数值较小的整数仅占用 1 字节,只有大数值才会占用更多字节。本项目中对文件大小和修改时间采用同样的压缩设计,秉承了 SQLite 节省空间与高效之设计哲学。
|
|
273
|
-
|
|
244
|
+
## 关于
|
|
245
|
+
|
|
246
|
+
本库由 [WebC.site](https://webc.site) 开发。
|
|
247
|
+
|
|
248
|
+
[WebC.site](https://webc.site) : 面向人工智能的网站开发新范式
|
package/package.json
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@1-/scan",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.6",
|
|
4
4
|
"description": "Incrementally scan directory files and track metadata in SQLite / 增量扫描目录文件并使用 SQLite 记录元数据",
|
|
5
5
|
"keywords": [
|
|
6
|
-
"scan",
|
|
7
|
-
"incremental",
|
|
8
|
-
"sqlite",
|
|
9
6
|
"directory",
|
|
10
|
-
"
|
|
7
|
+
"incremental",
|
|
8
|
+
"metadata",
|
|
9
|
+
"scan",
|
|
10
|
+
"sqlite"
|
|
11
11
|
],
|
|
12
12
|
"homepage": "https://github.com/webc-site/npm/tree/main/scan",
|
|
13
13
|
"license": "MulanPSL-2.0",
|
|
@@ -22,7 +22,7 @@
|
|
|
22
22
|
"./*": "./*"
|
|
23
23
|
},
|
|
24
24
|
"dependencies": {
|
|
25
|
-
"@1-/walk": "^0.1.
|
|
25
|
+
"@1-/walk": "^0.1.1",
|
|
26
26
|
"@3-/binmap": "^0.1.20",
|
|
27
27
|
"@3-/binset": "^0.1.6",
|
|
28
28
|
"@3-/int": "^0.1.1",
|