@1-/scan 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +34 -32
- package/_.js +1 -0
- package/dirWalk.js +1 -1
- package/fileWrite.js +2 -2
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -9,22 +9,23 @@ Incrementally scans directory files, tracks file sizes and modification times, a
|
|
|
9
9
|
|
|
10
10
|
## Features
|
|
11
11
|
|
|
12
|
-
-
|
|
13
|
-
-
|
|
14
|
-
-
|
|
15
|
-
-
|
|
16
|
-
-
|
|
12
|
+
- Incremental Scanning: Detects and updates only new, modified, or deleted files, avoiding redundant file system operations.
|
|
13
|
+
- Space-Efficient Storage: Employs Varint compression to serialize and compare file sizes and modification times.
|
|
14
|
+
- Smart Path Key: Stores relative paths not exceeding 16 bytes as raw binary to preserve readability, while hashing longer paths to 16-byte MD5 digests to optimize index performance.
|
|
15
|
+
- Database Synchronization: Synchronizes updates and deletions in a single atomic transaction.
|
|
16
|
+
- Ignore Pattern Support: Integrates ignore rules dynamically during traversal.
|
|
17
17
|
|
|
18
18
|
## Usage
|
|
19
19
|
|
|
20
20
|
```javascript
|
|
21
21
|
import scan from "@1-/scan";
|
|
22
22
|
|
|
23
|
-
const
|
|
24
|
-
const
|
|
23
|
+
const dir = "./src";
|
|
24
|
+
const dbPath = "./files.db";
|
|
25
25
|
|
|
26
|
-
// Scan directory and sync records into SQLite
|
|
27
|
-
await scan(
|
|
26
|
+
// Scan directory and sync records into SQLite, returning an array of updated relative paths
|
|
27
|
+
const updatedPaths = await scan(dir, dbPath);
|
|
28
|
+
console.log(updatedPaths);
|
|
28
29
|
```
|
|
29
30
|
|
|
30
31
|
## Design Ideas
|
|
@@ -44,11 +45,11 @@ graph TD
|
|
|
44
45
|
|
|
45
46
|
## Tech Stack
|
|
46
47
|
|
|
47
|
-
-
|
|
48
|
-
-
|
|
49
|
-
-
|
|
50
|
-
-
|
|
51
|
-
-
|
|
48
|
+
- Bun: Runtime and test runner
|
|
49
|
+
- SQLite: Local relational database engine
|
|
50
|
+
- `@1-/walk`: Directory walker with ignore support
|
|
51
|
+
- `@3-/vb`: Variable-length byte encoder
|
|
52
|
+
- `@3-/binmap` / `@3-/binset`: Efficient binary collection structures
|
|
52
53
|
|
|
53
54
|
## Directory Structure
|
|
54
55
|
|
|
@@ -77,31 +78,32 @@ To optimize space inside the database file, SQLite internally uses variable-leng
|
|
|
77
78
|
<a id="zh"></a>
|
|
78
79
|
# @1-/scan : 增量扫描目录文件并使用 SQLite 记录元数据
|
|
79
80
|
|
|
80
|
-
|
|
81
|
+
增量扫描目录,比对并同步文件大小与修改时间,记录存入 SQLite 数据库。
|
|
81
82
|
|
|
82
83
|
## 功能介绍
|
|
83
84
|
|
|
84
|
-
-
|
|
85
|
-
-
|
|
86
|
-
-
|
|
87
|
-
-
|
|
88
|
-
-
|
|
85
|
+
- 增量扫描:仅处理新增、修改或删除的文件,避免冗余文件操作。
|
|
86
|
+
- 紧凑存储:使用可变字节码(Varint)压缩技术比对并保存文件大小和修改时间。
|
|
87
|
+
- 路径映射:相对路径长度不大于 16 字节时保留原始字节,大于 16 字节时计算为 16 字节 MD5,优化数据库索引。
|
|
88
|
+
- 事务同步:更新与删除操作合并至单次数据库事务,确保一致性。
|
|
89
|
+
- 规则过滤:基于 `@1-/walk` 的忽略规则过滤特定文件与目录。
|
|
89
90
|
|
|
90
91
|
## 使用演示
|
|
91
92
|
|
|
92
93
|
```javascript
|
|
93
94
|
import scan from "@1-/scan";
|
|
94
95
|
|
|
95
|
-
const
|
|
96
|
-
const
|
|
96
|
+
const dir = "./src";
|
|
97
|
+
const dbPath = "./files.db";
|
|
97
98
|
|
|
98
|
-
// 扫描目录并同步至 SQLite
|
|
99
|
-
await scan(
|
|
99
|
+
// 扫描目录并同步至 SQLite 数据库,返回有变更的相对路径数组
|
|
100
|
+
const updatedPaths = await scan(dir, dbPath);
|
|
101
|
+
console.log(updatedPaths);
|
|
100
102
|
```
|
|
101
103
|
|
|
102
104
|
## 设计思路
|
|
103
105
|
|
|
104
|
-
|
|
106
|
+
模块调用流程:
|
|
105
107
|
|
|
106
108
|
```mermaid
|
|
107
109
|
graph TD
|
|
@@ -116,11 +118,11 @@ graph TD
|
|
|
116
118
|
|
|
117
119
|
## 技术栈
|
|
118
120
|
|
|
119
|
-
-
|
|
120
|
-
-
|
|
121
|
-
-
|
|
122
|
-
-
|
|
123
|
-
-
|
|
121
|
+
- Bun:运行环境与测试工具
|
|
122
|
+
- SQLite:本地关系型数据库
|
|
123
|
+
- `@1-/walk`:支持忽略规则的目录遍历工具
|
|
124
|
+
- `@3-/vb`:可变长度整型编码器
|
|
125
|
+
- `@3-/binmap` / `@3-/binset`:二进制哈希键容器
|
|
124
126
|
|
|
125
127
|
## 目录结构
|
|
126
128
|
|
|
@@ -139,7 +141,7 @@ graph TD
|
|
|
139
141
|
|
|
140
142
|
## 历史故事
|
|
141
143
|
|
|
142
|
-
SQLite 由 D. Richard Hipp 于 2000
|
|
144
|
+
SQLite 由 D. Richard Hipp 于 2000 年为驱逐舰控制系统编写。当时系统采用的商业数据库需要繁琐的管理,且一旦故障系统便无法运行。Hipp 设计出无服务器、零配置且直接读写单文件的 SQLite。
|
|
143
145
|
|
|
144
|
-
|
|
146
|
+
为节约存储空间,SQLite 内部采用可变长度整数(Varint)编码。本项目同样引入 Varint 压缩算法,对文件大小与修改时间做编码后再比对存储,延续了 SQLite 追求性能与紧凑空间的优良传统。
|
|
145
147
|
../doc/zh/about.md
|
package/_.js
CHANGED
package/dirWalk.js
CHANGED
package/fileWrite.js
CHANGED
|
@@ -5,8 +5,8 @@ export default (db, to_update, to_delete) => {
|
|
|
5
5
|
trans(db, () => {
|
|
6
6
|
if (to_update.length > 0) {
|
|
7
7
|
const insert = db.prepare("INSERT OR REPLACE INTO file(hash,size,mtime)VALUES(?,?,?)");
|
|
8
|
-
for (const
|
|
9
|
-
insert.run(
|
|
8
|
+
for (const [_, h, size, mtime] of to_update) {
|
|
9
|
+
insert.run(h, size, mtime);
|
|
10
10
|
}
|
|
11
11
|
}
|
|
12
12
|
if (to_delete.length > 0) {
|