pixivflow 2.20.4 → 2.21.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en.md +180 -53
- package/README.md +113 -59
- package/dist/commands/SchedulerCommand.js +66 -0
- package/dist/commands/SchedulerIdleLifecycle.d.ts +17 -0
- package/dist/commands/SchedulerIdleLifecycle.js +3 -1
- package/dist/commands/scheduler-runtime.d.ts +8 -0
- package/dist/commands/scheduler-runtime.js +121 -94
- package/dist/config/types.d.ts +26 -0
- package/dist/config/validation.js +21 -0
- package/dist/delivery/types.d.ts +15 -0
- package/dist/notification/NotificationPolicy.d.ts +2 -0
- package/dist/notification/NotificationPolicy.js +18 -1
- package/dist/package.json +1 -1
- package/dist/scheduler/MultiScheduleManager.d.ts +12 -1
- package/dist/scheduler/MultiScheduleManager.js +46 -50
- package/dist/scheduler/RecoveryPolicy.d.ts +45 -0
- package/dist/scheduler/RecoveryPolicy.js +63 -0
- package/dist/scheduler/ResourceAdmission.d.ts +68 -0
- package/dist/scheduler/ResourceAdmission.js +83 -0
- package/dist/scheduler/ScheduleTriggerServer.d.ts +16 -0
- package/dist/scheduler/ScheduleTriggerServer.js +55 -0
- package/dist/scheduler/Scheduler.d.ts +19 -2
- package/dist/scheduler/Scheduler.js +12 -3
- package/dist/scheduler/SlotCoordinator.d.ts +20 -0
- package/dist/scheduler/SlotCoordinator.js +44 -2
- package/dist/scheduler/TargetOutcome.d.ts +31 -0
- package/dist/scheduler/TargetOutcome.js +114 -0
- package/dist/storage/DatabaseMigration.js +17 -0
- package/dist/storage/repositories/SlotRepository.d.ts +29 -0
- package/dist/storage/repositories/SlotRepository.js +28 -2
- package/dist/version.js +1 -1
- package/dist/webui/package.json +1 -1
- package/package.json +2 -2
package/README.en.md
CHANGED
|
@@ -2,39 +2,51 @@
|
|
|
2
2
|
|
|
3
3
|
**Language / 语言:** [中文](README.md) · English
|
|
4
4
|
|
|
5
|
-
Pixiv
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
5
|
+
**Pixiv downloader, filter and automatic collection tool.**
|
|
6
|
+
|
|
7
|
+
Download a single Pixiv artwork (illustration, novel, ugoira) directly, or batch-collect
|
|
8
|
+
by tag, ranking, publish date and bookmark count, and let the scheduler keep collecting
|
|
9
|
+
on a cron. Results stay on your disk, or get delivered reliably over HTTP to another
|
|
10
|
+
service — downstreams are optional, PixivFlow alone covers
|
|
11
|
+
discover → filter → download → save.
|
|
9
12
|
|
|
10
13
|
[](https://www.npmjs.com/package/pixivflow)
|
|
11
14
|
[](https://nodejs.org/)
|
|
12
15
|
[](LICENSE)
|
|
13
16
|
[](https://redtidev1918.github.io/PixivFlow/)
|
|
14
17
|
|
|
15
|
-
##
|
|
18
|
+
## Typical scenarios
|
|
16
19
|
|
|
17
|
-
|
|
20
|
+
**1. Download one link.** Paste any Pixiv link — artwork, novel, series, or user
|
|
21
|
+
profile are all recognized:
|
|
18
22
|
|
|
19
23
|
```bash
|
|
20
|
-
|
|
21
|
-
pixivflow --help
|
|
24
|
+
pixivflow download --url https://www.pixiv.net/artworks/123456789
|
|
22
25
|
```
|
|
23
26
|
|
|
24
|
-
|
|
25
|
-
|
|
27
|
+
**2. Batch download by conditions.** Define what to collect in your config (tag,
|
|
28
|
+
ranking, date range, minimum bookmarks) and run it in one pass. Downloaded items are
|
|
29
|
+
tracked in SQLite and skipped, so re-running never re-downloads. See
|
|
30
|
+
[Filtering and targets](#filtering-and-targets).
|
|
26
31
|
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
npm install
|
|
31
|
-
npm run build
|
|
32
|
-
```
|
|
32
|
+
**3. Collect on a schedule and deliver.** Run it under cron: discover, download, and
|
|
33
|
+
optionally hand files to another service — the local copy is deleted only after the
|
|
34
|
+
receiver confirms.
|
|
33
35
|
|
|
34
|
-
|
|
36
|
+
```text
|
|
37
|
+
Pixiv ──► PixivFlow ──┬──► local, permanent (persistent)
|
|
38
|
+
└──► HTTP delivery (cache) ──► TelePost / any compatible service
|
|
39
|
+
```
|
|
35
40
|
|
|
36
41
|
## Quick start
|
|
37
42
|
|
|
43
|
+
Requires Node.js 22.13 or later; use a currently supported LTS release in production.
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
npm install -g pixivflow
|
|
47
|
+
pixivflow --help
|
|
48
|
+
```
|
|
49
|
+
|
|
38
50
|
Sign in to your Pixiv account (creates OAuth credentials, one time):
|
|
39
51
|
|
|
40
52
|
```bash
|
|
@@ -56,22 +68,27 @@ pixivflow download
|
|
|
56
68
|
pixivflow scheduler # long-running cron collection
|
|
57
69
|
```
|
|
58
70
|
|
|
59
|
-
|
|
71
|
+
Prefer not to hand-write config? The interactive wizard `pixivflow setup` generates it.
|
|
72
|
+
For a GUI, run `pixivflow webui` (frontend:
|
|
73
|
+
[pixivflow-webui](https://github.com/redtidev1918/pixivflow-webui)).
|
|
60
74
|
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
fully validates the new snapshot before swapping the entire cron table. An
|
|
66
|
-
invalid edit leaves the previous schedules running. In-flight work finishes
|
|
67
|
-
on its old snapshot; the next run sees the new one.
|
|
75
|
+
Downloading Pixiv ugoira additionally needs `python3` and `ffmpeg`: frames are
|
|
76
|
+
composited into a looping GIF by per-frame delay, ready to be delivered as an animation.
|
|
77
|
+
The official Docker image includes both — see
|
|
78
|
+
[configuration notes](docs/CONFIG.md#pixiv-动图ugoira).
|
|
68
79
|
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
80
|
+
Build from source:
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
git clone https://github.com/redtidev1918/PixivFlow.git
|
|
84
|
+
cd PixivFlow
|
|
85
|
+
npm install
|
|
86
|
+
npm run build
|
|
87
|
+
```
|
|
73
88
|
|
|
74
|
-
|
|
89
|
+
Termux / Android: see [TERMUX_INSTALL.md](docs/TERMUX_INSTALL.md).
|
|
90
|
+
|
|
91
|
+
## Filtering and targets
|
|
75
92
|
|
|
76
93
|
Define what to collect in the `targets` section of your config. Conditions
|
|
77
94
|
combine:
|
|
@@ -86,30 +103,44 @@ combine:
|
|
|
86
103
|
|
|
87
104
|
Downloaded items are tracked in a SQLite database and skipped automatically;
|
|
88
105
|
files that exist without a database record are reconciled, so the two never
|
|
89
|
-
conflict.
|
|
106
|
+
conflict. Illustration tasks with `mode: "topic"` keep a bounded hotness candidate
|
|
107
|
+
pool: if the top item is already downloaded, the next undownloaded one is promoted by
|
|
108
|
+
hotness instead of the run coming back empty.
|
|
109
|
+
|
|
110
|
+
## Persistent and cache delivery modes
|
|
90
111
|
|
|
91
|
-
|
|
112
|
+
Each target (one tag / schedule) can use one of two storage modes:
|
|
92
113
|
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
114
|
+
- **`persistent`** (the default): keep files forever.
|
|
115
|
+
- **`cache`**: send files to a named delivery target and delete the local copy only
|
|
116
|
+
after the receiver confirms — saves disk.
|
|
117
|
+
|
|
118
|
+
A "delivery target" is just configuration: which URL to POST to and which fields to
|
|
119
|
+
send. It is not bound to any service and can point at any compatible HTTP endpoint;
|
|
120
|
+
[TelePost](https://github.com/redtidev1918/TelePost) and
|
|
121
|
+
[telepress](https://github.com/redtidev1918/telepress) are example downstreams. Example:
|
|
97
122
|
|
|
98
123
|
```json
|
|
99
124
|
{
|
|
100
125
|
"delivery": {
|
|
126
|
+
"outboxRetryBaseMs": 300000,
|
|
127
|
+
"outboxRetryMaxMs": 21600000,
|
|
101
128
|
"targets": {
|
|
102
129
|
"sharing-api": {
|
|
103
130
|
"type": "httpMultipart",
|
|
104
|
-
"url": "https://example
|
|
105
|
-
"readinessUrl": "https://example
|
|
106
|
-
"notificationUrl": "https://example
|
|
131
|
+
"url": "https://your-domain.example/api/bot1/v1/submissions",
|
|
132
|
+
"readinessUrl": "https://your-domain.example/ready",
|
|
133
|
+
"notificationUrl": "https://your-domain.example/api/bot1/v1/notifications",
|
|
107
134
|
"headers": { "Authorization": "Bearer ${SHARING_TOKEN}" },
|
|
108
135
|
"fileField": "files",
|
|
109
136
|
"fields": { "title": "{{title}}" },
|
|
110
|
-
"success": { "statuses": [201], "jsonPath": "ok", "equals": true }
|
|
137
|
+
"success": { "statuses": [201], "jsonPath": "ok", "equals": true },
|
|
138
|
+
"arrayFormat": "comma",
|
|
139
|
+
"maxAttempts": 3,
|
|
140
|
+
"retryDelayMs": 2000
|
|
111
141
|
}
|
|
112
|
-
}
|
|
142
|
+
},
|
|
143
|
+
"deleteAfterDelivery": true
|
|
113
144
|
},
|
|
114
145
|
"targets": [
|
|
115
146
|
{ "type": "illustration", "tag": "archive", "storageMode": "persistent" },
|
|
@@ -126,20 +157,89 @@ merely translates an HTTP multipart submission API into configuration:
|
|
|
126
157
|
}
|
|
127
158
|
```
|
|
128
159
|
|
|
129
|
-
Headers and URLs accept
|
|
130
|
-
illustration delivery also sends an optional, one-to-one Pixiv preview in the
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
notifications
|
|
135
|
-
|
|
160
|
+
Headers and URLs accept `${ENV_NAME}` interpolation (never hard-code tokens). Cache-mode
|
|
161
|
+
illustration delivery also sends an optional, one-to-one Pixiv preview in the `previews`
|
|
162
|
+
multipart field while retaining the original as the authoritative artifact; a generic
|
|
163
|
+
receiver may ignore that field.
|
|
164
|
+
|
|
165
|
+
Operational notifications can point `notificationUrl` at an
|
|
166
|
+
[Apprise API](docs/APPRISE.md) instance, which fans out to Email, Telegram, Discord,
|
|
167
|
+
ntfy and more; PixivFlow does not implement those notification protocols itself.
|
|
168
|
+
|
|
169
|
+
- The `url` above is any compatible HTTP submission endpoint; the example uses
|
|
170
|
+
TelePost's `/api/botN/v1/submissions` (put the `tp_...` from `/gen_token` into
|
|
171
|
+
`SHARING_TOKEN` — that is the example service's own auth scheme).
|
|
172
|
+
- The same target can point at [telepress](https://github.com/redtidev1918/telepress)'s
|
|
173
|
+
`/publish/gallery` to publish illustrations as a Telegra.ph gallery, see the
|
|
174
|
+
"Telegraph (telegra.ph) gallery upload" section of [CONFIG.md](docs/CONFIG.md).
|
|
175
|
+
|
|
176
|
+
## Automation and reliability
|
|
177
|
+
|
|
178
|
+
### Multiple schedules and atomic hot reload
|
|
179
|
+
|
|
180
|
+
`schedules[]` hosts independently timed target groups in one Node process.
|
|
181
|
+
Plans share Pixiv authentication, SQLite, and file services, while a bounded
|
|
182
|
+
serial queue prevents overlapping downloads from producing memory spikes. This runs
|
|
183
|
+
comfortably on small machines — measured with `topic` discovery, selection and
|
|
184
|
+
download inside a 256 MB cgroup: peak RSS ≈ 106 MB, heapUsed ≈ 33 MB, no OOM
|
|
185
|
+
(see [DOCKER.md](docs/DOCKER.md)).
|
|
186
|
+
The active config is watched by default: replace it over SSH and PixivFlow
|
|
187
|
+
fully validates the new snapshot before swapping the entire cron table. An
|
|
188
|
+
invalid edit leaves the previous schedules running. In-flight work finishes
|
|
189
|
+
on its old snapshot; the next run sees the new one.
|
|
190
|
+
|
|
191
|
+
```json
|
|
192
|
+
{
|
|
193
|
+
"scheduler": { "enabled": false, "cron": "0 3 * * *" },
|
|
194
|
+
"schedules": [
|
|
195
|
+
{ "id": "bot1", "enabled": true, "cron": "10 5 * * *", "targetIds": ["bot1-art", "bot1-novel"] },
|
|
196
|
+
{ "id": "bot2", "enabled": true, "cron": "30 5 * * *", "targetIds": ["bot2-art", "bot2-novel"] }
|
|
197
|
+
],
|
|
198
|
+
"targets": [
|
|
199
|
+
{ "id": "bot1-art", "type": "illustration", "mode": "ranking", "rankingDate": "YESTERDAY" },
|
|
200
|
+
{ "id": "bot1-novel", "type": "novel", "mode": "ranking", "rankingDate": "YESTERDAY" }
|
|
201
|
+
]
|
|
202
|
+
}
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
Legacy single-`scheduler` configs remain supported. `schedules`, `targets`,
|
|
206
|
+
`delivery`, and `download` are hot-reloadable; changes to `pixiv`, `network`,
|
|
207
|
+
or `storage` require a process restart. See the ready-to-edit
|
|
208
|
+
[`config/fly-two-bots.example.json`](config/fly-two-bots.example.json) template.
|
|
136
209
|
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
210
|
+
### Durable delivery outbox
|
|
211
|
+
|
|
212
|
+
Delivery is transactional (a SQLite outbox): at-least-once execution with
|
|
213
|
+
effectively-once visible effects. Every external side effect (one content delivery,
|
|
214
|
+
one notification) becomes a row with an idempotency key and a row-level lease; an
|
|
215
|
+
independent worker pumps it right away and retries with exponential backoff (5 minutes
|
|
216
|
+
up to 6 hours by default), and rows that exceed `maxAttempts` go dead and are recorded
|
|
217
|
+
as `failed` in the delivery ledger. After a crash or machine hang, expired `processing`
|
|
218
|
+
leases are taken over by the next process, which retries the same idempotent intent — a
|
|
219
|
+
downstream converges via `idempotent_replay` (same key, lost ACK) or
|
|
220
|
+
`duplicate_existing` (historical duplicate), so the channel still shows exactly one
|
|
221
|
+
message. Legacy file-based `delivery-outbox/*.json` manifests are migrated into SQLite
|
|
222
|
+
once at startup, idempotently. "Nothing to publish today" notifications share that table
|
|
223
|
+
but are pumped independently, so a temporarily unavailable review endpoint cannot block
|
|
224
|
+
content delivery.
|
|
225
|
+
|
|
226
|
+
When `readinessUrl` is configured, a non-2xx readiness response returns the row to
|
|
227
|
+
pending without incrementing its attempt count — strictly separate from `/live`'s
|
|
228
|
+
"process is alive" meaning. Dead letters are recovered through first-class CLI:
|
|
141
229
|
|
|
142
|
-
|
|
230
|
+
```bash
|
|
231
|
+
pixivflow outbox list --status dead
|
|
232
|
+
pixivflow outbox inspect <id>
|
|
233
|
+
pixivflow outbox retry <id> # only accepts dead rows, keeps the idempotency key
|
|
234
|
+
pixivflow outbox retry --dead
|
|
235
|
+
pixivflow outbox cancel <id> # cancels rows that have not run yet
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
`run-once` re-runs the download plan; it is not an outbox replay, and you should not
|
|
239
|
+
hand-edit SQLite's `next_attempt_at`. For long-running hosts, use `pixivflow doctor`
|
|
240
|
+
(stuck slot/outbox leases, pending deliveries, dead rows; `--repair` converges) and
|
|
241
|
+
`pixivflow reconcile` (record downstream-confirmed historical duplicates in the delivery
|
|
242
|
+
ledger; dry-run by default).
|
|
143
243
|
|
|
144
244
|
## Common commands
|
|
145
245
|
|
|
@@ -149,16 +249,31 @@ Interactive configuration wizard: `pixivflow setup`.
|
|
|
149
249
|
| `pixivflow download --url <url>` | Direct download via URL |
|
|
150
250
|
| `pixivflow random` | Random popular artwork |
|
|
151
251
|
| `pixivflow scheduler` | Start scheduled jobs |
|
|
252
|
+
| `pixivflow webui` | Start the WebUI |
|
|
152
253
|
| `pixivflow config` | Manage config (view / edit / backup / restore) |
|
|
153
254
|
| `pixivflow status` | Download stats and recent records |
|
|
154
255
|
| `pixivflow health` | Health check: config, directories, connectivity |
|
|
256
|
+
| `pixivflow doctor` | Reliability check: stuck slot/outbox leases, pending deliveries, dead rows; `--repair` converges |
|
|
257
|
+
| `pixivflow reconcile` | Record downstream-confirmed historical duplicates (dry-run by default, `--repair` writes) |
|
|
258
|
+
| `pixivflow outbox` | List, inspect, replay dead letters, or cancel durable intents that have not run |
|
|
155
259
|
| `pixivflow tags discover <seed>` | Discover related tags (Pixiv autocomplete + tag co-occurrence); lists candidates only |
|
|
156
260
|
| `pixivflow tags apply <manifest> --target <id> --select <tag1,tag2>` | Atomically write chosen tags into config after manual confirmation, then hot-reload |
|
|
261
|
+
| `pixivflow topic resolve <topic>` | Inspect the derived tag space for a topic (`--type illustration\|novel`, `--refresh`) |
|
|
262
|
+
| `pixivflow topic test <topic> --date YESTERDAY` | Dry-run the candidates and Top N for one day, without downloading |
|
|
157
263
|
|
|
158
264
|
`tags discover` calls the Pixiv autocomplete endpoint and samples recent illustrations/novels to count co-occurring tags, caching results for 7 days; it **never** changes active plans. After reviewing candidates, run `tags apply` to explicitly select tags: it validates the whole config, writes a backup and atomically replaces the file so a running scheduler hot-reloads it.
|
|
159
265
|
|
|
160
266
|
More commands in [USAGE.md](docs/USAGE.md); see the [migration guide](docs/MIGRATION.md) for upgrading from v1 to v2.
|
|
161
267
|
|
|
268
|
+
## Deployment
|
|
269
|
+
|
|
270
|
+
- **Docker / long-running server**: see [DOCKER.md](docs/DOCKER.md).
|
|
271
|
+
- **Android / Termux**: see [TERMUX_INSTALL.md](docs/TERMUX_INSTALL.md).
|
|
272
|
+
- **Composing with TelePost**: PixivFlow and TelePost are both usable on their own.
|
|
273
|
+
Only if you want to deploy the two together as one workflow do you need the
|
|
274
|
+
[pixivflow-telepost-deploy](https://github.com/redtidev1918/pixivflow-telepost-deploy)
|
|
275
|
+
deployment and operations kit.
|
|
276
|
+
|
|
162
277
|
## Documentation
|
|
163
278
|
|
|
164
279
|
Full tutorial site: <https://redtidev1918.github.io/PixivFlow/>
|
|
@@ -179,6 +294,18 @@ Full tutorial site: <https://redtidev1918.github.io/PixivFlow/>
|
|
|
179
294
|
|
|
180
295
|
Chinese version: [README.md](README.md).
|
|
181
296
|
|
|
297
|
+
## Related projects
|
|
298
|
+
|
|
299
|
+
PixivFlow is fully standalone. These are the related projects in the same author's
|
|
300
|
+
ecosystem, and what each one owns:
|
|
301
|
+
|
|
302
|
+
| Project | What it is | When you need it |
|
|
303
|
+
| --- | --- | --- |
|
|
304
|
+
| [TelePost](https://github.com/redtidev1918/TelePost) | Telegram channel submission, moderation and automated publishing platform | When you want downloads to land in a Telegram channel for human review before publishing — configure it as a delivery downstream. This is an optional composition; PixivFlow does not depend on it |
|
|
305
|
+
| [pixivflow-telepost-deploy](https://github.com/redtidev1918/pixivflow-telepost-deploy) | Deployment and operations kit for PixivFlow + TelePost (Docker / VPS / cloud) | When you want to deploy and operate both projects together. Running PixivFlow alone does not need it |
|
|
306
|
+
| [pixivflow-webui](https://github.com/redtidev1918/pixivflow-webui) | WebUI frontend for PixivFlow | When you want a GUI to manage downloads and schedules |
|
|
307
|
+
| [pixiv-token-getter](https://github.com/redtidev1918/pixiv-token-getter) | PKCE OAuth login library and CLI (`ptg`) | PixivFlow's login dependency; also usable on its own to obtain Pixiv tokens |
|
|
308
|
+
|
|
182
309
|
## Feedback
|
|
183
310
|
|
|
184
311
|
Bugs and feature requests go to
|
package/README.md
CHANGED
|
@@ -2,42 +2,46 @@
|
|
|
2
2
|
|
|
3
3
|
**语言 / Language:** 中文 · [English](README.en.md)
|
|
4
4
|
|
|
5
|
-
Pixiv
|
|
6
|
-
|
|
7
|
-
|
|
5
|
+
**Pixiv 下载、筛选与自动收集工具。**
|
|
6
|
+
|
|
7
|
+
可以直接下载单个 Pixiv 作品(插画、小说、动图),也可以按标签、热度、日期和收藏数
|
|
8
|
+
批量筛选,并通过 scheduler 定时自动收集。结果既能永久保存在本地,也能按需通过 HTTP
|
|
9
|
+
可靠交付给其他服务——下游是可选的,PixivFlow 自己就能跑完「发现 → 筛选 → 下载 → 保存」
|
|
10
|
+
的完整链路。
|
|
8
11
|
|
|
9
12
|
[](https://www.npmjs.com/package/pixivflow)
|
|
10
13
|
[](https://nodejs.org/)
|
|
11
14
|
[](LICENSE)
|
|
12
15
|
[](https://redtidev1918.github.io/PixivFlow/)
|
|
13
16
|
|
|
14
|
-
##
|
|
17
|
+
## 典型场景
|
|
15
18
|
|
|
16
|
-
|
|
17
|
-
下载 Pixiv 动图(ugoira)还需要 `python3` 和 `ffmpeg`:程序按逐帧延迟合成循环 GIF,
|
|
18
|
-
可直接交给 TelePost 作为动画发送。官方 Docker 镜像已包含两者,详见 [配置说明](docs/CONFIG.md#pixiv-动图ugoira)。
|
|
19
|
+
**1. 下载一个链接。** 直接粘贴任意 Pixiv 链接——插画、小说、系列、用户主页都能识别:
|
|
19
20
|
|
|
20
21
|
```bash
|
|
21
|
-
|
|
22
|
-
pixivflow --help
|
|
22
|
+
pixivflow download --url https://www.pixiv.net/artworks/123456789
|
|
23
23
|
```
|
|
24
24
|
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
[
|
|
28
|
-
—— 一套配置,支持国内/海外、有/无公网 IP、VPS/Fly.io 任意场景。
|
|
29
|
-
从源码构建:
|
|
25
|
+
**2. 按条件批量下载。** 在配置里定义要收集什么(标签、榜单、发布日期、收藏数下限),
|
|
26
|
+
一次跑完;已下载的作品由 SQLite 记录并自动跳过,重复运行不会重复拉取。
|
|
27
|
+
见[筛选与下载目标](#筛选与下载目标)。
|
|
30
28
|
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
29
|
+
**3. 定时自动收集并交付。** 用 cron 长期挂机:定时发现、下载,再按需把内容投递给
|
|
30
|
+
其他服务——对方确认收到后才删除本地副本。
|
|
31
|
+
|
|
32
|
+
```text
|
|
33
|
+
Pixiv ──► PixivFlow ──┬──► 本地永久保存(persistent)
|
|
34
|
+
└──► HTTP 交付(cache)──► TelePost / 其他兼容服务
|
|
36
35
|
```
|
|
37
36
|
|
|
38
|
-
|
|
37
|
+
## 快速开始
|
|
39
38
|
|
|
40
|
-
|
|
39
|
+
需要 Node.js 22.13 或更高版本;生产环境请使用仍受支持的 LTS。
|
|
40
|
+
|
|
41
|
+
```bash
|
|
42
|
+
npm install -g pixivflow
|
|
43
|
+
pixivflow --help
|
|
44
|
+
```
|
|
41
45
|
|
|
42
46
|
登录 Pixiv 账号(生成 OAuth 凭据,只需一次):
|
|
43
47
|
|
|
@@ -59,35 +63,25 @@ pixivflow download
|
|
|
59
63
|
pixivflow scheduler # 按 cron 配置长期挂机自动收集
|
|
60
64
|
```
|
|
61
65
|
|
|
62
|
-
|
|
66
|
+
不想手写配置?运行交互式向导 `pixivflow setup` 一步步生成。图形界面用
|
|
67
|
+
`pixivflow webui`(前端见 [pixivflow-webui](https://github.com/redtidev1918/pixivflow-webui))。
|
|
63
68
|
|
|
64
|
-
`
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
稳定运行,峰值 RSS ≈ 106 MB、heapUsed ≈ 33 MB,无 OOM,见 [DOCKER.md](docs/DOCKER.md))。配置文件默认被监听,SSH/同步工具替换文件后会先完整
|
|
68
|
-
校验,再一次性替换全部调度项;无效 JSON、错误 Cron 或未知 target id 不会破坏
|
|
69
|
-
当前运行中的计划。正在执行的任务继续使用旧快照,下一次任务使用新快照。
|
|
69
|
+
下载 Pixiv 动图(ugoira)还需要 `python3` 和 `ffmpeg`:程序按逐帧延迟合成循环 GIF,
|
|
70
|
+
可直接作为动画交付给下游。官方 Docker 镜像已包含两者,详见
|
|
71
|
+
[配置说明](docs/CONFIG.md#pixiv-动图ugoira)。
|
|
70
72
|
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
"targets": [
|
|
79
|
-
{ "id": "bot1-art", "type": "illustration", "mode": "ranking", "rankingDate": "YESTERDAY" },
|
|
80
|
-
{ "id": "bot1-novel", "type": "novel", "mode": "ranking", "rankingDate": "YESTERDAY" }
|
|
81
|
-
]
|
|
82
|
-
}
|
|
73
|
+
从源码构建:
|
|
74
|
+
|
|
75
|
+
```bash
|
|
76
|
+
git clone https://github.com/redtidev1918/PixivFlow.git
|
|
77
|
+
cd PixivFlow
|
|
78
|
+
npm install
|
|
79
|
+
npm run build
|
|
83
80
|
```
|
|
84
81
|
|
|
85
|
-
|
|
86
|
-
连接或路径,修改后需要重启;`schedules`、`targets`、`delivery`、`download` 可以
|
|
87
|
-
热重载。完整双 Bot 缓存投递模板见
|
|
88
|
-
[`config/fly-two-bots.example.json`](config/fly-two-bots.example.json)。
|
|
82
|
+
Termux / Android 环境见 [TERMUX_INSTALL.md](docs/TERMUX_INSTALL.md)。
|
|
89
83
|
|
|
90
|
-
##
|
|
84
|
+
## 筛选与下载目标
|
|
91
85
|
|
|
92
86
|
在配置文件的 `targets` 中定义要收集的内容,多个条件可以组合:
|
|
93
87
|
|
|
@@ -103,7 +97,7 @@ pixivflow scheduler # 按 cron 配置长期挂机自动收集
|
|
|
103
97
|
两者互不冲突。`mode: "topic"` 的插画任务会保留一个有界热度候选池:同一发布日期
|
|
104
98
|
重复执行时若第一名已经下载,会按热度自动递补下一部未下载作品,而不是空跑。
|
|
105
99
|
|
|
106
|
-
|
|
100
|
+
## 本地留存与缓存交付
|
|
107
101
|
|
|
108
102
|
每个 target(一个 tag / 计划)有两种保存方式:
|
|
109
103
|
|
|
@@ -111,7 +105,9 @@ pixivflow scheduler # 按 cron 配置长期挂机自动收集
|
|
|
111
105
|
- **`cache`**:下载后投给一个「交付目标」(比如投稿机器人),对方确认收到后才删本地文件,省磁盘。
|
|
112
106
|
|
|
113
107
|
「交付目标」就是一段配置:告诉 PixivFlow 把文件 POST 到哪个地址、带哪些字段。
|
|
114
|
-
|
|
108
|
+
它不绑定具体服务,可指向任意兼容的 HTTP 接口;
|
|
109
|
+
[TelePost](https://github.com/redtidev1918/TelePost) 与
|
|
110
|
+
[telepress](https://github.com/redtidev1918/telepress) 只是示例下游。示例:
|
|
115
111
|
|
|
116
112
|
```json
|
|
117
113
|
{
|
|
@@ -119,12 +115,12 @@ pixivflow scheduler # 按 cron 配置长期挂机自动收集
|
|
|
119
115
|
"outboxRetryBaseMs": 300000,
|
|
120
116
|
"outboxRetryMaxMs": 21600000,
|
|
121
117
|
"targets": {
|
|
122
|
-
"
|
|
118
|
+
"sharing-api": {
|
|
123
119
|
"type": "httpMultipart",
|
|
124
120
|
"url": "https://your-domain.example/api/bot1/v1/submissions",
|
|
125
121
|
"readinessUrl": "https://your-domain.example/ready",
|
|
126
122
|
"notificationUrl": "https://your-domain.example/api/bot1/v1/notifications",
|
|
127
|
-
"headers": { "Authorization": "Bearer ${
|
|
123
|
+
"headers": { "Authorization": "Bearer ${SHARING_TOKEN}" },
|
|
128
124
|
"fileField": "files",
|
|
129
125
|
"fields": { "title": "{{title}}" },
|
|
130
126
|
"success": { "statuses": [201], "jsonPath": "ok", "equals": true },
|
|
@@ -142,7 +138,7 @@ pixivflow scheduler # 按 cron 配置长期挂机自动收集
|
|
|
142
138
|
"tag": "更新",
|
|
143
139
|
"storageMode": "cache",
|
|
144
140
|
"delivery": {
|
|
145
|
-
"target": "
|
|
141
|
+
"target": "sharing-api",
|
|
146
142
|
"fields": { "tags": ["公告", "更新"], "anonymous": false }
|
|
147
143
|
}
|
|
148
144
|
}
|
|
@@ -154,6 +150,49 @@ pixivflow scheduler # 按 cron 配置长期挂机自动收集
|
|
|
154
150
|
插画 cache 投递会在 multipart 的 `previews` 字段携带 Pixiv 的低分辨率预览(与 `files`
|
|
155
151
|
一一对应),原图仍是权威素材;通用接收端可以忽略该可选字段。
|
|
156
152
|
|
|
153
|
+
运维通知可直接把 `notificationUrl` 指向 [Apprise API](docs/APPRISE.md),由 Apprise 统一发送
|
|
154
|
+
Email、Telegram、Discord、ntfy 等渠道;PixivFlow 不实现这些通知协议。
|
|
155
|
+
|
|
156
|
+
- 上面的 `url` 指向任意兼容的 HTTP 投稿接口;示例里用的是 TelePost 的
|
|
157
|
+
`/api/botN/v1/submissions`(把 `/gen_token` 得到的 `tp_...` 放进 `SHARING_TOKEN` 即可,
|
|
158
|
+
这是示例服务自己的鉴权方式)。
|
|
159
|
+
- 同一目标也可指向 [telepress](https://github.com/redtidev1918/telepress) 的 `/publish/gallery`,
|
|
160
|
+
把插画自动发布成 Telegra.ph 相册,见 [CONFIG.md](docs/CONFIG.md) 的
|
|
161
|
+
「Telegraph(telegra.ph)相册上传」。
|
|
162
|
+
|
|
163
|
+
## 自动化与可靠性
|
|
164
|
+
|
|
165
|
+
### 单进程多计划与配置热重载
|
|
166
|
+
|
|
167
|
+
`schedules[]` 可以为不同 target 组设置各自的 Cron。所有计划由一个 Node
|
|
168
|
+
进程托管,共享 Pixiv 客户端、SQLite 与文件服务;执行阶段使用有界串行队列,
|
|
169
|
+
适合 512 MiB 小内存机器(实测:`topic` 发现/采集/下载全程在 256 MB cgroup 限制下
|
|
170
|
+
稳定运行,峰值 RSS ≈ 106 MB、heapUsed ≈ 33 MB,无 OOM,见 [DOCKER.md](docs/DOCKER.md))。
|
|
171
|
+
配置文件默认被监听,SSH/同步工具替换文件后会先完整校验,再一次性替换全部调度项;
|
|
172
|
+
无效 JSON、错误 Cron 或未知 target id 不会破坏当前运行中的计划。正在执行的任务
|
|
173
|
+
继续使用旧快照,下一次任务使用新快照。
|
|
174
|
+
|
|
175
|
+
```json
|
|
176
|
+
{
|
|
177
|
+
"scheduler": { "enabled": false, "cron": "0 3 * * *" },
|
|
178
|
+
"schedules": [
|
|
179
|
+
{ "id": "bot1", "enabled": true, "cron": "10 5 * * *", "targetIds": ["bot1-art", "bot1-novel"] },
|
|
180
|
+
{ "id": "bot2", "enabled": true, "cron": "30 5 * * *", "targetIds": ["bot2-art", "bot2-novel"] }
|
|
181
|
+
],
|
|
182
|
+
"targets": [
|
|
183
|
+
{ "id": "bot1-art", "type": "illustration", "mode": "ranking", "rankingDate": "YESTERDAY" },
|
|
184
|
+
{ "id": "bot1-novel", "type": "novel", "mode": "ranking", "rankingDate": "YESTERDAY" }
|
|
185
|
+
]
|
|
186
|
+
}
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
旧的单 `scheduler` 配置继续兼容。`pixiv`、`network`、`storage` 涉及长生命周期
|
|
190
|
+
连接或路径,修改后需要重启;`schedules`、`targets`、`delivery`、`download` 可以
|
|
191
|
+
热重载。完整双 Bot 缓存投递模板见
|
|
192
|
+
[`config/fly-two-bots.example.json`](config/fly-two-bots.example.json)。
|
|
193
|
+
|
|
194
|
+
### 投递事务发件箱
|
|
195
|
+
|
|
157
196
|
投递是**事务发件箱(SQLite outbox)**式的,at-least-once 执行、effectively-once
|
|
158
197
|
可见效果:每个外部副作用(一次内容投递、一条通知)在 `outbox` 表落一行,带幂等键和
|
|
159
198
|
行级租约,独立 worker 立即泵送、指数退避重试(默认 5 分钟起步、最长 6 小时),超过
|
|
@@ -176,15 +215,9 @@ pixivflow outbox cancel <id> # 只取消尚未执行的 row
|
|
|
176
215
|
```
|
|
177
216
|
|
|
178
217
|
`run-once` 会重新执行下载计划,不等价于 outbox replay;不要手改 SQLite 的
|
|
179
|
-
`next_attempt_at
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
Email、Telegram、Discord、ntfy 等渠道;PixivFlow 不实现这些通知协议。
|
|
183
|
-
|
|
184
|
-
- 上面的示例是投稿给一个 HTTP 接口:把 `/gen_token` 得到的 `tp_...` 放进 `TG_SUBMIT_TOKEN` 即可(这是示例服务自己的鉴权方式)。
|
|
185
|
-
- 同一目标也可指向 [telepress](https://github.com/redtidev1918/telepress) 的 `/publish/gallery`,把插画自动发布成 Telegra.ph 相册,见 [CONFIG.md](docs/CONFIG.md) 的「Telegraph(telegra.ph)相册上传」。
|
|
186
|
-
|
|
187
|
-
不想手写配置?运行交互式向导 `pixivflow setup` 一步步生成。
|
|
218
|
+
`next_attempt_at`。长期运行环境的体检与收敛用 `pixivflow doctor`(卡住的 slot/outbox
|
|
219
|
+
租约、pending 投递、dead 行,`--repair` 收敛)与 `pixivflow reconcile`(把下游已确认的
|
|
220
|
+
历史重复登记进投递账本,默认 dry-run)。
|
|
188
221
|
|
|
189
222
|
## 常用命令
|
|
190
223
|
|
|
@@ -194,6 +227,7 @@ Email、Telegram、Discord、ntfy 等渠道;PixivFlow 不实现这些通知协
|
|
|
194
227
|
| `pixivflow download --url <url>` | 通过 URL 直接下载 |
|
|
195
228
|
| `pixivflow random` | 随机下载热门作品 |
|
|
196
229
|
| `pixivflow scheduler` | 启动定时任务 |
|
|
230
|
+
| `pixivflow webui` | 启动 WebUI |
|
|
197
231
|
| `pixivflow config` | 配置管理(查看 / 编辑 / 备份 / 恢复) |
|
|
198
232
|
| `pixivflow status` | 下载统计与最近记录 |
|
|
199
233
|
| `pixivflow health` | 健康检查:配置、目录可写性、连通性 |
|
|
@@ -209,6 +243,15 @@ Email、Telegram、Discord、ntfy 等渠道;PixivFlow 不实现这些通知协
|
|
|
209
243
|
|
|
210
244
|
其他用法见 [USAGE.md](docs/USAGE.md);从 v1 升级到 v2 见 [迁移指南](docs/MIGRATION.md)。
|
|
211
245
|
|
|
246
|
+
## 部署
|
|
247
|
+
|
|
248
|
+
- **Docker / 服务器长期挂机**:见 [DOCKER.md](docs/DOCKER.md)。
|
|
249
|
+
- **Android / Termux**:见 [TERMUX_INSTALL.md](docs/TERMUX_INSTALL.md)。
|
|
250
|
+
- **与 TelePost 组合部署**:PixivFlow 与 TelePost 都可以独立使用;只有当你希望把两者
|
|
251
|
+
组合成一套完整工作流时,才需要
|
|
252
|
+
[pixivflow-telepost-deploy](https://github.com/redtidev1918/pixivflow-telepost-deploy)
|
|
253
|
+
这个部署与运维套件。
|
|
254
|
+
|
|
212
255
|
## 文档
|
|
213
256
|
|
|
214
257
|
完整教程站点:<https://redtidev1918.github.io/PixivFlow/>
|
|
@@ -229,6 +272,17 @@ Email、Telegram、Discord、ntfy 等渠道;PixivFlow 不实现这些通知协
|
|
|
229
272
|
|
|
230
273
|
English version: [README.en.md](README.en.md).
|
|
231
274
|
|
|
275
|
+
## 相关项目
|
|
276
|
+
|
|
277
|
+
PixivFlow 可以完全独立使用。下面是同一作者生态里与它相关的项目,以及各自负责什么:
|
|
278
|
+
|
|
279
|
+
| 项目 | 是什么 | 什么时候需要 |
|
|
280
|
+
| --- | --- | --- |
|
|
281
|
+
| [TelePost](https://github.com/redtidev1918/TelePost) | Telegram 频道投稿、审核与自动化发布平台 | 想把下载结果投进 Telegram 频道、先人工审核再发布时,把它配成 delivery 下游即可。这只是可选组合,PixivFlow 不依赖它 |
|
|
282
|
+
| [pixivflow-telepost-deploy](https://github.com/redtidev1918/pixivflow-telepost-deploy) | PixivFlow + TelePost 的部署与运维套件(Docker / VPS / 云平台) | 想一次性把上面两个项目部署并运维起来时。只跑 PixivFlow 不需要它 |
|
|
283
|
+
| [pixivflow-webui](https://github.com/redtidev1918/pixivflow-webui) | PixivFlow 的 WebUI 前端 | 想用图形界面管理下载与计划 |
|
|
284
|
+
| [pixiv-token-getter](https://github.com/redtidev1918/pixiv-token-getter) | PKCE OAuth 登录库与 CLI(`ptg`) | PixivFlow 的登录依赖;也可以单独用于获取 Pixiv token |
|
|
285
|
+
|
|
232
286
|
## 问题反馈
|
|
233
287
|
|
|
234
288
|
Bug 与功能建议请提交到
|