pixivflow 2.20.5 → 2.21.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en.md +179 -56
- package/README.md +110 -57
- package/dist/commands/SchedulerCommand.js +63 -0
- package/dist/commands/SchedulerIdleLifecycle.d.ts +8 -0
- package/dist/commands/SchedulerIdleLifecycle.js +2 -1
- package/dist/commands/scheduler-runtime.js +17 -0
- package/dist/config/types.d.ts +26 -0
- package/dist/config/validation.js +21 -0
- package/dist/delivery/types.d.ts +15 -0
- package/dist/notification/NotificationPolicy.d.ts +2 -0
- package/dist/notification/NotificationPolicy.js +18 -1
- package/dist/package.json +1 -1
- package/dist/scheduler/MultiScheduleManager.d.ts +12 -1
- package/dist/scheduler/MultiScheduleManager.js +46 -50
- package/dist/scheduler/RecoveryPolicy.d.ts +45 -0
- package/dist/scheduler/RecoveryPolicy.js +63 -0
- package/dist/scheduler/ResourceAdmission.d.ts +68 -0
- package/dist/scheduler/ResourceAdmission.js +83 -0
- package/dist/scheduler/ScheduleTriggerServer.d.ts +16 -0
- package/dist/scheduler/ScheduleTriggerServer.js +55 -0
- package/dist/scheduler/Scheduler.d.ts +19 -2
- package/dist/scheduler/Scheduler.js +12 -3
- package/dist/scheduler/SlotCoordinator.d.ts +20 -0
- package/dist/scheduler/SlotCoordinator.js +44 -2
- package/dist/scheduler/TargetOutcome.d.ts +31 -0
- package/dist/scheduler/TargetOutcome.js +114 -0
- package/dist/storage/DatabaseMigration.js +17 -0
- package/dist/storage/repositories/SlotRepository.d.ts +29 -0
- package/dist/storage/repositories/SlotRepository.js +28 -2
- package/dist/version.js +1 -1
- package/dist/webui/package.json +1 -1
- package/package.json +2 -2
package/README.en.md
CHANGED
|
@@ -2,41 +2,51 @@
|
|
|
2
2
|
|
|
3
3
|
**Language / 语言:** [中文](README.md) · English
|
|
4
4
|
|
|
5
|
-
Pixiv
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
5
|
+
**Pixiv downloader, filter and automatic collection tool.**
|
|
6
|
+
|
|
7
|
+
Download a single Pixiv artwork (illustration, novel, ugoira) directly, or batch-collect
|
|
8
|
+
by tag, ranking, publish date and bookmark count, and let the scheduler keep collecting
|
|
9
|
+
on a cron. Results stay on your disk, or get delivered reliably over HTTP to another
|
|
10
|
+
service — downstreams are optional, PixivFlow alone covers
|
|
11
|
+
discover → filter → download → save.
|
|
9
12
|
|
|
10
13
|
[](https://www.npmjs.com/package/pixivflow)
|
|
11
14
|
[](https://nodejs.org/)
|
|
12
15
|
[](LICENSE)
|
|
13
16
|
[](https://redtidev1918.github.io/PixivFlow/)
|
|
14
17
|
|
|
15
|
-
##
|
|
18
|
+
## Typical scenarios
|
|
16
19
|
|
|
17
|
-
|
|
20
|
+
**1. Download one link.** Paste any Pixiv link — artwork, novel, series, or user
|
|
21
|
+
profile are all recognized:
|
|
18
22
|
|
|
19
23
|
```bash
|
|
20
|
-
|
|
21
|
-
pixivflow --help
|
|
24
|
+
pixivflow download --url https://www.pixiv.net/artworks/123456789
|
|
22
25
|
```
|
|
23
26
|
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
27
|
+
**2. Batch download by conditions.** Define what to collect in your config (tag,
|
|
28
|
+
ranking, date range, minimum bookmarks) and run it in one pass. Downloaded items are
|
|
29
|
+
tracked in SQLite and skipped, so re-running never re-downloads. See
|
|
30
|
+
[Filtering and targets](#filtering-and-targets).
|
|
28
31
|
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
npm install
|
|
33
|
-
npm run build
|
|
34
|
-
```
|
|
32
|
+
**3. Collect on a schedule and deliver.** Run it under cron: discover, download, and
|
|
33
|
+
optionally hand files to another service — the local copy is deleted only after the
|
|
34
|
+
receiver confirms.
|
|
35
35
|
|
|
36
|
-
|
|
36
|
+
```text
|
|
37
|
+
Pixiv ──► PixivFlow ──┬──► local, permanent (persistent)
|
|
38
|
+
└──► HTTP delivery (cache) ──► TelePost / any compatible service
|
|
39
|
+
```
|
|
37
40
|
|
|
38
41
|
## Quick start
|
|
39
42
|
|
|
43
|
+
Requires Node.js 22.13 or later; use a currently supported LTS release in production.
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
npm install -g pixivflow
|
|
47
|
+
pixivflow --help
|
|
48
|
+
```
|
|
49
|
+
|
|
40
50
|
Sign in to your Pixiv account (creates OAuth credentials, one time):
|
|
41
51
|
|
|
42
52
|
```bash
|
|
@@ -58,22 +68,27 @@ pixivflow download
|
|
|
58
68
|
pixivflow scheduler # long-running cron collection
|
|
59
69
|
```
|
|
60
70
|
|
|
61
|
-
|
|
71
|
+
Prefer not to hand-write config? The interactive wizard `pixivflow setup` generates it.
|
|
72
|
+
For a GUI, run `pixivflow webui` (frontend:
|
|
73
|
+
[pixivflow-webui](https://github.com/redtidev1918/pixivflow-webui)).
|
|
62
74
|
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
fully validates the new snapshot before swapping the entire cron table. An
|
|
68
|
-
invalid edit leaves the previous schedules running. In-flight work finishes
|
|
69
|
-
on its old snapshot; the next run sees the new one.
|
|
75
|
+
Downloading Pixiv ugoira additionally needs `python3` and `ffmpeg`: frames are
|
|
76
|
+
composited into a looping GIF by per-frame delay, ready to be delivered as an animation.
|
|
77
|
+
The official Docker image includes both — see
|
|
78
|
+
[configuration notes](docs/CONFIG.md#pixiv-动图ugoira).
|
|
70
79
|
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
80
|
+
Build from source:
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
git clone https://github.com/redtidev1918/PixivFlow.git
|
|
84
|
+
cd PixivFlow
|
|
85
|
+
npm install
|
|
86
|
+
npm run build
|
|
87
|
+
```
|
|
75
88
|
|
|
76
|
-
|
|
89
|
+
Termux / Android: see [TERMUX_INSTALL.md](docs/TERMUX_INSTALL.md).
|
|
90
|
+
|
|
91
|
+
## Filtering and targets
|
|
77
92
|
|
|
78
93
|
Define what to collect in the `targets` section of your config. Conditions
|
|
79
94
|
combine:
|
|
@@ -88,32 +103,44 @@ combine:
|
|
|
88
103
|
|
|
89
104
|
Downloaded items are tracked in a SQLite database and skipped automatically;
|
|
90
105
|
files that exist without a database record are reconciled, so the two never
|
|
91
|
-
conflict.
|
|
106
|
+
conflict. Illustration tasks with `mode: "topic"` keep a bounded hotness candidate
|
|
107
|
+
pool: if the top item is already downloaded, the next undownloaded one is promoted by
|
|
108
|
+
hotness instead of the run coming back empty.
|
|
109
|
+
|
|
110
|
+
## Persistent and cache delivery modes
|
|
92
111
|
|
|
93
|
-
|
|
112
|
+
Each target (one tag / schedule) can use one of two storage modes:
|
|
94
113
|
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
114
|
+
- **`persistent`** (the default): keep files forever.
|
|
115
|
+
- **`cache`**: send files to a named delivery target and delete the local copy only
|
|
116
|
+
after the receiver confirms — saves disk.
|
|
117
|
+
|
|
118
|
+
A "delivery target" is just configuration: which URL to POST to and which fields to
|
|
119
|
+
send. It is not bound to any service and can point at any compatible HTTP endpoint;
|
|
98
120
|
[TelePost](https://github.com/redtidev1918/TelePost) and
|
|
99
|
-
[telepress](https://github.com/redtidev1918/telepress) are example downstreams
|
|
100
|
-
not dependencies. This example merely translates an HTTP multipart submission API into configuration:
|
|
121
|
+
[telepress](https://github.com/redtidev1918/telepress) are example downstreams. Example:
|
|
101
122
|
|
|
102
123
|
```json
|
|
103
124
|
{
|
|
104
125
|
"delivery": {
|
|
126
|
+
"outboxRetryBaseMs": 300000,
|
|
127
|
+
"outboxRetryMaxMs": 21600000,
|
|
105
128
|
"targets": {
|
|
106
129
|
"sharing-api": {
|
|
107
130
|
"type": "httpMultipart",
|
|
108
|
-
"url": "https://example
|
|
109
|
-
"readinessUrl": "https://example
|
|
110
|
-
"notificationUrl": "https://example
|
|
131
|
+
"url": "https://your-domain.example/api/bot1/v1/submissions",
|
|
132
|
+
"readinessUrl": "https://your-domain.example/ready",
|
|
133
|
+
"notificationUrl": "https://your-domain.example/api/bot1/v1/notifications",
|
|
111
134
|
"headers": { "Authorization": "Bearer ${SHARING_TOKEN}" },
|
|
112
135
|
"fileField": "files",
|
|
113
136
|
"fields": { "title": "{{title}}" },
|
|
114
|
-
"success": { "statuses": [201], "jsonPath": "ok", "equals": true }
|
|
137
|
+
"success": { "statuses": [201], "jsonPath": "ok", "equals": true },
|
|
138
|
+
"arrayFormat": "comma",
|
|
139
|
+
"maxAttempts": 3,
|
|
140
|
+
"retryDelayMs": 2000
|
|
115
141
|
}
|
|
116
|
-
}
|
|
142
|
+
},
|
|
143
|
+
"deleteAfterDelivery": true
|
|
117
144
|
},
|
|
118
145
|
"targets": [
|
|
119
146
|
{ "type": "illustration", "tag": "archive", "storageMode": "persistent" },
|
|
@@ -130,20 +157,89 @@ not dependencies. This example merely translates an HTTP multipart submission AP
|
|
|
130
157
|
}
|
|
131
158
|
```
|
|
132
159
|
|
|
133
|
-
Headers and URLs accept
|
|
134
|
-
illustration delivery also sends an optional, one-to-one Pixiv preview in the
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
notifications
|
|
139
|
-
|
|
160
|
+
Headers and URLs accept `${ENV_NAME}` interpolation (never hard-code tokens). Cache-mode
|
|
161
|
+
illustration delivery also sends an optional, one-to-one Pixiv preview in the `previews`
|
|
162
|
+
multipart field while retaining the original as the authoritative artifact; a generic
|
|
163
|
+
receiver may ignore that field.
|
|
164
|
+
|
|
165
|
+
Operational notifications can point `notificationUrl` at an
|
|
166
|
+
[Apprise API](docs/APPRISE.md) instance, which fans out to Email, Telegram, Discord,
|
|
167
|
+
ntfy and more; PixivFlow does not implement those notification protocols itself.
|
|
168
|
+
|
|
169
|
+
- The `url` above is any compatible HTTP submission endpoint; the example uses
|
|
170
|
+
TelePost's `/api/botN/v1/submissions` (put the `tp_...` from `/gen_token` into
|
|
171
|
+
`SHARING_TOKEN` — that is the example service's own auth scheme).
|
|
172
|
+
- The same target can point at [telepress](https://github.com/redtidev1918/telepress)'s
|
|
173
|
+
`/publish/gallery` to publish illustrations as a Telegra.ph gallery, see the
|
|
174
|
+
"Telegraph (telegra.ph) gallery upload" section of [CONFIG.md](docs/CONFIG.md).
|
|
175
|
+
|
|
176
|
+
## Automation and reliability
|
|
177
|
+
|
|
178
|
+
### Multiple schedules and atomic hot reload
|
|
179
|
+
|
|
180
|
+
`schedules[]` hosts independently timed target groups in one Node process.
|
|
181
|
+
Plans share Pixiv authentication, SQLite, and file services, while a bounded
|
|
182
|
+
serial queue prevents overlapping downloads from producing memory spikes. This runs
|
|
183
|
+
comfortably on small machines — measured with `topic` discovery, selection and
|
|
184
|
+
download inside a 256 MB cgroup: peak RSS ≈ 106 MB, heapUsed ≈ 33 MB, no OOM
|
|
185
|
+
(see [DOCKER.md](docs/DOCKER.md)).
|
|
186
|
+
The active config is watched by default: replace it over SSH and PixivFlow
|
|
187
|
+
fully validates the new snapshot before swapping the entire cron table. An
|
|
188
|
+
invalid edit leaves the previous schedules running. In-flight work finishes
|
|
189
|
+
on its old snapshot; the next run sees the new one.
|
|
190
|
+
|
|
191
|
+
```json
|
|
192
|
+
{
|
|
193
|
+
"scheduler": { "enabled": false, "cron": "0 3 * * *" },
|
|
194
|
+
"schedules": [
|
|
195
|
+
{ "id": "bot1", "enabled": true, "cron": "10 5 * * *", "targetIds": ["bot1-art", "bot1-novel"] },
|
|
196
|
+
{ "id": "bot2", "enabled": true, "cron": "30 5 * * *", "targetIds": ["bot2-art", "bot2-novel"] }
|
|
197
|
+
],
|
|
198
|
+
"targets": [
|
|
199
|
+
{ "id": "bot1-art", "type": "illustration", "mode": "ranking", "rankingDate": "YESTERDAY" },
|
|
200
|
+
{ "id": "bot1-novel", "type": "novel", "mode": "ranking", "rankingDate": "YESTERDAY" }
|
|
201
|
+
]
|
|
202
|
+
}
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
Legacy single-`scheduler` configs remain supported. `schedules`, `targets`,
|
|
206
|
+
`delivery`, and `download` are hot-reloadable; changes to `pixiv`, `network`,
|
|
207
|
+
or `storage` require a process restart. See the ready-to-edit
|
|
208
|
+
[`config/fly-two-bots.example.json`](config/fly-two-bots.example.json) template.
|
|
140
209
|
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
210
|
+
### Durable delivery outbox
|
|
211
|
+
|
|
212
|
+
Delivery is transactional (a SQLite outbox): at-least-once execution with
|
|
213
|
+
effectively-once visible effects. Every external side effect (one content delivery,
|
|
214
|
+
one notification) becomes a row with an idempotency key and a row-level lease; an
|
|
215
|
+
independent worker pumps it right away and retries with exponential backoff (5 minutes
|
|
216
|
+
up to 6 hours by default), and rows that exceed `maxAttempts` go dead and are recorded
|
|
217
|
+
as `failed` in the delivery ledger. After a crash or machine hang, expired `processing`
|
|
218
|
+
leases are taken over by the next process, which retries the same idempotent intent — a
|
|
219
|
+
downstream converges via `idempotent_replay` (same key, lost ACK) or
|
|
220
|
+
`duplicate_existing` (historical duplicate), so the channel still shows exactly one
|
|
221
|
+
message. Legacy file-based `delivery-outbox/*.json` manifests are migrated into SQLite
|
|
222
|
+
once at startup, idempotently. "Nothing to publish today" notifications share that table
|
|
223
|
+
but are pumped independently, so a temporarily unavailable review endpoint cannot block
|
|
224
|
+
content delivery.
|
|
225
|
+
|
|
226
|
+
When `readinessUrl` is configured, a non-2xx readiness response returns the row to
|
|
227
|
+
pending without incrementing its attempt count — strictly separate from `/live`'s
|
|
228
|
+
"process is alive" meaning. Dead letters are recovered through first-class CLI:
|
|
145
229
|
|
|
146
|
-
|
|
230
|
+
```bash
|
|
231
|
+
pixivflow outbox list --status dead
|
|
232
|
+
pixivflow outbox inspect <id>
|
|
233
|
+
pixivflow outbox retry <id> # only accepts dead rows, keeps the idempotency key
|
|
234
|
+
pixivflow outbox retry --dead
|
|
235
|
+
pixivflow outbox cancel <id> # cancels rows that have not run yet
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
`run-once` re-runs the download plan; it is not an outbox replay, and you should not
|
|
239
|
+
hand-edit SQLite's `next_attempt_at`. For long-running hosts, use `pixivflow doctor`
|
|
240
|
+
(stuck slot/outbox leases, pending deliveries, dead rows; `--repair` converges) and
|
|
241
|
+
`pixivflow reconcile` (record downstream-confirmed historical duplicates in the delivery
|
|
242
|
+
ledger; dry-run by default).
|
|
147
243
|
|
|
148
244
|
## Common commands
|
|
149
245
|
|
|
@@ -153,16 +249,31 @@ Interactive configuration wizard: `pixivflow setup`.
|
|
|
153
249
|
| `pixivflow download --url <url>` | Direct download via URL |
|
|
154
250
|
| `pixivflow random` | Random popular artwork |
|
|
155
251
|
| `pixivflow scheduler` | Start scheduled jobs |
|
|
252
|
+
| `pixivflow webui` | Start the WebUI |
|
|
156
253
|
| `pixivflow config` | Manage config (view / edit / backup / restore) |
|
|
157
254
|
| `pixivflow status` | Download stats and recent records |
|
|
158
255
|
| `pixivflow health` | Health check: config, directories, connectivity |
|
|
256
|
+
| `pixivflow doctor` | Reliability check: stuck slot/outbox leases, pending deliveries, dead rows; `--repair` converges |
|
|
257
|
+
| `pixivflow reconcile` | Record downstream-confirmed historical duplicates (dry-run by default, `--repair` writes) |
|
|
258
|
+
| `pixivflow outbox` | List, inspect, replay dead letters, or cancel durable intents that have not run |
|
|
159
259
|
| `pixivflow tags discover <seed>` | Discover related tags (Pixiv autocomplete + tag co-occurrence); lists candidates only |
|
|
160
260
|
| `pixivflow tags apply <manifest> --target <id> --select <tag1,tag2>` | Atomically write chosen tags into config after manual confirmation, then hot-reload |
|
|
261
|
+
| `pixivflow topic resolve <topic>` | Inspect the derived tag space for a topic (`--type illustration\|novel`, `--refresh`) |
|
|
262
|
+
| `pixivflow topic test <topic> --date YESTERDAY` | Dry-run the candidates and Top N for one day, without downloading |
|
|
161
263
|
|
|
162
264
|
`tags discover` calls the Pixiv autocomplete endpoint and samples recent illustrations/novels to count co-occurring tags, caching results for 7 days; it **never** changes active plans. After reviewing candidates, run `tags apply` to explicitly select tags: it validates the whole config, writes a backup and atomically replaces the file so a running scheduler hot-reloads it.
|
|
163
265
|
|
|
164
266
|
More commands in [USAGE.md](docs/USAGE.md); see the [migration guide](docs/MIGRATION.md) for upgrading from v1 to v2.
|
|
165
267
|
|
|
268
|
+
## Deployment
|
|
269
|
+
|
|
270
|
+
- **Docker / long-running server**: see [DOCKER.md](docs/DOCKER.md).
|
|
271
|
+
- **Android / Termux**: see [TERMUX_INSTALL.md](docs/TERMUX_INSTALL.md).
|
|
272
|
+
- **Composing with TelePost**: PixivFlow and TelePost are both usable on their own.
|
|
273
|
+
Only if you want to deploy the two together as one workflow do you need the
|
|
274
|
+
[pixivflow-telepost-deploy](https://github.com/redtidev1918/pixivflow-telepost-deploy)
|
|
275
|
+
deployment and operations kit.
|
|
276
|
+
|
|
166
277
|
## Documentation
|
|
167
278
|
|
|
168
279
|
Full tutorial site: <https://redtidev1918.github.io/PixivFlow/>
|
|
@@ -183,6 +294,18 @@ Full tutorial site: <https://redtidev1918.github.io/PixivFlow/>
|
|
|
183
294
|
|
|
184
295
|
Chinese version: [README.md](README.md).
|
|
185
296
|
|
|
297
|
+
## Related projects
|
|
298
|
+
|
|
299
|
+
PixivFlow is fully standalone. These are the related projects in the same author's
|
|
300
|
+
ecosystem, and what each one owns:
|
|
301
|
+
|
|
302
|
+
| Project | What it is | When you need it |
|
|
303
|
+
| --- | --- | --- |
|
|
304
|
+
| [TelePost](https://github.com/redtidev1918/TelePost) | Telegram channel submission, moderation and automated publishing platform | When you want downloads to land in a Telegram channel for human review before publishing — configure it as a delivery downstream. This is an optional composition; PixivFlow does not depend on it |
|
|
305
|
+
| [pixivflow-telepost-deploy](https://github.com/redtidev1918/pixivflow-telepost-deploy) | Deployment and operations kit for PixivFlow + TelePost (Docker / VPS / cloud) | When you want to deploy and operate both projects together. Running PixivFlow alone does not need it |
|
|
306
|
+
| [pixivflow-webui](https://github.com/redtidev1918/pixivflow-webui) | WebUI frontend for PixivFlow | When you want a GUI to manage downloads and schedules |
|
|
307
|
+
| [pixiv-token-getter](https://github.com/redtidev1918/pixiv-token-getter) | PKCE OAuth login library and CLI (`ptg`) | PixivFlow's login dependency; also usable on its own to obtain Pixiv tokens |
|
|
308
|
+
|
|
186
309
|
## Feedback
|
|
187
310
|
|
|
188
311
|
Bugs and feature requests go to
|
package/README.md
CHANGED
|
@@ -2,41 +2,46 @@
|
|
|
2
2
|
|
|
3
3
|
**语言 / Language:** 中文 · [English](README.en.md)
|
|
4
4
|
|
|
5
|
-
Pixiv
|
|
6
|
-
|
|
7
|
-
|
|
5
|
+
**Pixiv 下载、筛选与自动收集工具。**
|
|
6
|
+
|
|
7
|
+
可以直接下载单个 Pixiv 作品(插画、小说、动图),也可以按标签、热度、日期和收藏数
|
|
8
|
+
批量筛选,并通过 scheduler 定时自动收集。结果既能永久保存在本地,也能按需通过 HTTP
|
|
9
|
+
可靠交付给其他服务——下游是可选的,PixivFlow 自己就能跑完「发现 → 筛选 → 下载 → 保存」
|
|
10
|
+
的完整链路。
|
|
8
11
|
|
|
9
12
|
[](https://www.npmjs.com/package/pixivflow)
|
|
10
13
|
[](https://nodejs.org/)
|
|
11
14
|
[](LICENSE)
|
|
12
15
|
[](https://redtidev1918.github.io/PixivFlow/)
|
|
13
16
|
|
|
14
|
-
##
|
|
17
|
+
## 典型场景
|
|
15
18
|
|
|
16
|
-
|
|
17
|
-
下载 Pixiv 动图(ugoira)还需要 `python3` 和 `ffmpeg`:程序按逐帧延迟合成循环 GIF,
|
|
18
|
-
可直接交给 TelePost 作为动画发送。官方 Docker 镜像已包含两者,详见 [配置说明](docs/CONFIG.md#pixiv-动图ugoira)。
|
|
19
|
+
**1. 下载一个链接。** 直接粘贴任意 Pixiv 链接——插画、小说、系列、用户主页都能识别:
|
|
19
20
|
|
|
20
21
|
```bash
|
|
21
|
-
|
|
22
|
-
pixivflow --help
|
|
22
|
+
pixivflow download --url https://www.pixiv.net/artworks/123456789
|
|
23
23
|
```
|
|
24
24
|
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
从源码构建:
|
|
25
|
+
**2. 按条件批量下载。** 在配置里定义要收集什么(标签、榜单、发布日期、收藏数下限),
|
|
26
|
+
一次跑完;已下载的作品由 SQLite 记录并自动跳过,重复运行不会重复拉取。
|
|
27
|
+
见[筛选与下载目标](#筛选与下载目标)。
|
|
29
28
|
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
29
|
+
**3. 定时自动收集并交付。** 用 cron 长期挂机:定时发现、下载,再按需把内容投递给
|
|
30
|
+
其他服务——对方确认收到后才删除本地副本。
|
|
31
|
+
|
|
32
|
+
```text
|
|
33
|
+
Pixiv ──► PixivFlow ──┬──► 本地永久保存(persistent)
|
|
34
|
+
└──► HTTP 交付(cache)──► TelePost / 其他兼容服务
|
|
35
35
|
```
|
|
36
36
|
|
|
37
|
-
|
|
37
|
+
## 快速开始
|
|
38
38
|
|
|
39
|
-
|
|
39
|
+
需要 Node.js 22.13 或更高版本;生产环境请使用仍受支持的 LTS。
|
|
40
|
+
|
|
41
|
+
```bash
|
|
42
|
+
npm install -g pixivflow
|
|
43
|
+
pixivflow --help
|
|
44
|
+
```
|
|
40
45
|
|
|
41
46
|
登录 Pixiv 账号(生成 OAuth 凭据,只需一次):
|
|
42
47
|
|
|
@@ -58,35 +63,25 @@ pixivflow download
|
|
|
58
63
|
pixivflow scheduler # 按 cron 配置长期挂机自动收集
|
|
59
64
|
```
|
|
60
65
|
|
|
61
|
-
|
|
66
|
+
不想手写配置?运行交互式向导 `pixivflow setup` 一步步生成。图形界面用
|
|
67
|
+
`pixivflow webui`(前端见 [pixivflow-webui](https://github.com/redtidev1918/pixivflow-webui))。
|
|
62
68
|
|
|
63
|
-
`
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
稳定运行,峰值 RSS ≈ 106 MB、heapUsed ≈ 33 MB,无 OOM,见 [DOCKER.md](docs/DOCKER.md))。配置文件默认被监听,SSH/同步工具替换文件后会先完整
|
|
67
|
-
校验,再一次性替换全部调度项;无效 JSON、错误 Cron 或未知 target id 不会破坏
|
|
68
|
-
当前运行中的计划。正在执行的任务继续使用旧快照,下一次任务使用新快照。
|
|
69
|
+
下载 Pixiv 动图(ugoira)还需要 `python3` 和 `ffmpeg`:程序按逐帧延迟合成循环 GIF,
|
|
70
|
+
可直接作为动画交付给下游。官方 Docker 镜像已包含两者,详见
|
|
71
|
+
[配置说明](docs/CONFIG.md#pixiv-动图ugoira)。
|
|
69
72
|
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
"targets": [
|
|
78
|
-
{ "id": "bot1-art", "type": "illustration", "mode": "ranking", "rankingDate": "YESTERDAY" },
|
|
79
|
-
{ "id": "bot1-novel", "type": "novel", "mode": "ranking", "rankingDate": "YESTERDAY" }
|
|
80
|
-
]
|
|
81
|
-
}
|
|
73
|
+
从源码构建:
|
|
74
|
+
|
|
75
|
+
```bash
|
|
76
|
+
git clone https://github.com/redtidev1918/PixivFlow.git
|
|
77
|
+
cd PixivFlow
|
|
78
|
+
npm install
|
|
79
|
+
npm run build
|
|
82
80
|
```
|
|
83
81
|
|
|
84
|
-
|
|
85
|
-
连接或路径,修改后需要重启;`schedules`、`targets`、`delivery`、`download` 可以
|
|
86
|
-
热重载。完整双 Bot 缓存投递模板见
|
|
87
|
-
[`config/fly-two-bots.example.json`](config/fly-two-bots.example.json)。
|
|
82
|
+
Termux / Android 环境见 [TERMUX_INSTALL.md](docs/TERMUX_INSTALL.md)。
|
|
88
83
|
|
|
89
|
-
##
|
|
84
|
+
## 筛选与下载目标
|
|
90
85
|
|
|
91
86
|
在配置文件的 `targets` 中定义要收集的内容,多个条件可以组合:
|
|
92
87
|
|
|
@@ -102,7 +97,7 @@ pixivflow scheduler # 按 cron 配置长期挂机自动收集
|
|
|
102
97
|
两者互不冲突。`mode: "topic"` 的插画任务会保留一个有界热度候选池:同一发布日期
|
|
103
98
|
重复执行时若第一名已经下载,会按热度自动递补下一部未下载作品,而不是空跑。
|
|
104
99
|
|
|
105
|
-
|
|
100
|
+
## 本地留存与缓存交付
|
|
106
101
|
|
|
107
102
|
每个 target(一个 tag / 计划)有两种保存方式:
|
|
108
103
|
|
|
@@ -120,12 +115,12 @@ pixivflow scheduler # 按 cron 配置长期挂机自动收集
|
|
|
120
115
|
"outboxRetryBaseMs": 300000,
|
|
121
116
|
"outboxRetryMaxMs": 21600000,
|
|
122
117
|
"targets": {
|
|
123
|
-
"
|
|
118
|
+
"sharing-api": {
|
|
124
119
|
"type": "httpMultipart",
|
|
125
120
|
"url": "https://your-domain.example/api/bot1/v1/submissions",
|
|
126
121
|
"readinessUrl": "https://your-domain.example/ready",
|
|
127
122
|
"notificationUrl": "https://your-domain.example/api/bot1/v1/notifications",
|
|
128
|
-
"headers": { "Authorization": "Bearer ${
|
|
123
|
+
"headers": { "Authorization": "Bearer ${SHARING_TOKEN}" },
|
|
129
124
|
"fileField": "files",
|
|
130
125
|
"fields": { "title": "{{title}}" },
|
|
131
126
|
"success": { "statuses": [201], "jsonPath": "ok", "equals": true },
|
|
@@ -143,7 +138,7 @@ pixivflow scheduler # 按 cron 配置长期挂机自动收集
|
|
|
143
138
|
"tag": "更新",
|
|
144
139
|
"storageMode": "cache",
|
|
145
140
|
"delivery": {
|
|
146
|
-
"target": "
|
|
141
|
+
"target": "sharing-api",
|
|
147
142
|
"fields": { "tags": ["公告", "更新"], "anonymous": false }
|
|
148
143
|
}
|
|
149
144
|
}
|
|
@@ -155,6 +150,49 @@ pixivflow scheduler # 按 cron 配置长期挂机自动收集
|
|
|
155
150
|
插画 cache 投递会在 multipart 的 `previews` 字段携带 Pixiv 的低分辨率预览(与 `files`
|
|
156
151
|
一一对应),原图仍是权威素材;通用接收端可以忽略该可选字段。
|
|
157
152
|
|
|
153
|
+
运维通知可直接把 `notificationUrl` 指向 [Apprise API](docs/APPRISE.md),由 Apprise 统一发送
|
|
154
|
+
Email、Telegram、Discord、ntfy 等渠道;PixivFlow 不实现这些通知协议。
|
|
155
|
+
|
|
156
|
+
- 上面的 `url` 指向任意兼容的 HTTP 投稿接口;示例里用的是 TelePost 的
|
|
157
|
+
`/api/botN/v1/submissions`(把 `/gen_token` 得到的 `tp_...` 放进 `SHARING_TOKEN` 即可,
|
|
158
|
+
这是示例服务自己的鉴权方式)。
|
|
159
|
+
- 同一目标也可指向 [telepress](https://github.com/redtidev1918/telepress) 的 `/publish/gallery`,
|
|
160
|
+
把插画自动发布成 Telegra.ph 相册,见 [CONFIG.md](docs/CONFIG.md) 的
|
|
161
|
+
「Telegraph(telegra.ph)相册上传」。
|
|
162
|
+
|
|
163
|
+
## 自动化与可靠性
|
|
164
|
+
|
|
165
|
+
### 单进程多计划与配置热重载
|
|
166
|
+
|
|
167
|
+
`schedules[]` 可以为不同 target 组设置各自的 Cron。所有计划由一个 Node
|
|
168
|
+
进程托管,共享 Pixiv 客户端、SQLite 与文件服务;执行阶段使用有界串行队列,
|
|
169
|
+
适合 512 MiB 小内存机器(实测:`topic` 发现/采集/下载全程在 256 MB cgroup 限制下
|
|
170
|
+
稳定运行,峰值 RSS ≈ 106 MB、heapUsed ≈ 33 MB,无 OOM,见 [DOCKER.md](docs/DOCKER.md))。
|
|
171
|
+
配置文件默认被监听,SSH/同步工具替换文件后会先完整校验,再一次性替换全部调度项;
|
|
172
|
+
无效 JSON、错误 Cron 或未知 target id 不会破坏当前运行中的计划。正在执行的任务
|
|
173
|
+
继续使用旧快照,下一次任务使用新快照。
|
|
174
|
+
|
|
175
|
+
```json
|
|
176
|
+
{
|
|
177
|
+
"scheduler": { "enabled": false, "cron": "0 3 * * *" },
|
|
178
|
+
"schedules": [
|
|
179
|
+
{ "id": "bot1", "enabled": true, "cron": "10 5 * * *", "targetIds": ["bot1-art", "bot1-novel"] },
|
|
180
|
+
{ "id": "bot2", "enabled": true, "cron": "30 5 * * *", "targetIds": ["bot2-art", "bot2-novel"] }
|
|
181
|
+
],
|
|
182
|
+
"targets": [
|
|
183
|
+
{ "id": "bot1-art", "type": "illustration", "mode": "ranking", "rankingDate": "YESTERDAY" },
|
|
184
|
+
{ "id": "bot1-novel", "type": "novel", "mode": "ranking", "rankingDate": "YESTERDAY" }
|
|
185
|
+
]
|
|
186
|
+
}
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
旧的单 `scheduler` 配置继续兼容。`pixiv`、`network`、`storage` 涉及长生命周期
|
|
190
|
+
连接或路径,修改后需要重启;`schedules`、`targets`、`delivery`、`download` 可以
|
|
191
|
+
热重载。完整双 Bot 缓存投递模板见
|
|
192
|
+
[`config/fly-two-bots.example.json`](config/fly-two-bots.example.json)。
|
|
193
|
+
|
|
194
|
+
### 投递事务发件箱
|
|
195
|
+
|
|
158
196
|
投递是**事务发件箱(SQLite outbox)**式的,at-least-once 执行、effectively-once
|
|
159
197
|
可见效果:每个外部副作用(一次内容投递、一条通知)在 `outbox` 表落一行,带幂等键和
|
|
160
198
|
行级租约,独立 worker 立即泵送、指数退避重试(默认 5 分钟起步、最长 6 小时),超过
|
|
@@ -177,15 +215,9 @@ pixivflow outbox cancel <id> # 只取消尚未执行的 row
|
|
|
177
215
|
```
|
|
178
216
|
|
|
179
217
|
`run-once` 会重新执行下载计划,不等价于 outbox replay;不要手改 SQLite 的
|
|
180
|
-
`next_attempt_at
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
Email、Telegram、Discord、ntfy 等渠道;PixivFlow 不实现这些通知协议。
|
|
184
|
-
|
|
185
|
-
- 上面的示例是投稿给一个 HTTP 接口:把 `/gen_token` 得到的 `tp_...` 放进 `TG_SUBMIT_TOKEN` 即可(这是示例服务自己的鉴权方式)。
|
|
186
|
-
- 同一目标也可指向 [telepress](https://github.com/redtidev1918/telepress) 的 `/publish/gallery`,把插画自动发布成 Telegra.ph 相册,见 [CONFIG.md](docs/CONFIG.md) 的「Telegraph(telegra.ph)相册上传」。
|
|
187
|
-
|
|
188
|
-
不想手写配置?运行交互式向导 `pixivflow setup` 一步步生成。
|
|
218
|
+
`next_attempt_at`。长期运行环境的体检与收敛用 `pixivflow doctor`(卡住的 slot/outbox
|
|
219
|
+
租约、pending 投递、dead 行,`--repair` 收敛)与 `pixivflow reconcile`(把下游已确认的
|
|
220
|
+
历史重复登记进投递账本,默认 dry-run)。
|
|
189
221
|
|
|
190
222
|
## 常用命令
|
|
191
223
|
|
|
@@ -195,6 +227,7 @@ Email、Telegram、Discord、ntfy 等渠道;PixivFlow 不实现这些通知协
|
|
|
195
227
|
| `pixivflow download --url <url>` | 通过 URL 直接下载 |
|
|
196
228
|
| `pixivflow random` | 随机下载热门作品 |
|
|
197
229
|
| `pixivflow scheduler` | 启动定时任务 |
|
|
230
|
+
| `pixivflow webui` | 启动 WebUI |
|
|
198
231
|
| `pixivflow config` | 配置管理(查看 / 编辑 / 备份 / 恢复) |
|
|
199
232
|
| `pixivflow status` | 下载统计与最近记录 |
|
|
200
233
|
| `pixivflow health` | 健康检查:配置、目录可写性、连通性 |
|
|
@@ -210,6 +243,15 @@ Email、Telegram、Discord、ntfy 等渠道;PixivFlow 不实现这些通知协
|
|
|
210
243
|
|
|
211
244
|
其他用法见 [USAGE.md](docs/USAGE.md);从 v1 升级到 v2 见 [迁移指南](docs/MIGRATION.md)。
|
|
212
245
|
|
|
246
|
+
## 部署
|
|
247
|
+
|
|
248
|
+
- **Docker / 服务器长期挂机**:见 [DOCKER.md](docs/DOCKER.md)。
|
|
249
|
+
- **Android / Termux**:见 [TERMUX_INSTALL.md](docs/TERMUX_INSTALL.md)。
|
|
250
|
+
- **与 TelePost 组合部署**:PixivFlow 与 TelePost 都可以独立使用;只有当你希望把两者
|
|
251
|
+
组合成一套完整工作流时,才需要
|
|
252
|
+
[pixivflow-telepost-deploy](https://github.com/redtidev1918/pixivflow-telepost-deploy)
|
|
253
|
+
这个部署与运维套件。
|
|
254
|
+
|
|
213
255
|
## 文档
|
|
214
256
|
|
|
215
257
|
完整教程站点:<https://redtidev1918.github.io/PixivFlow/>
|
|
@@ -230,6 +272,17 @@ Email、Telegram、Discord、ntfy 等渠道;PixivFlow 不实现这些通知协
|
|
|
230
272
|
|
|
231
273
|
English version: [README.en.md](README.en.md).
|
|
232
274
|
|
|
275
|
+
## 相关项目
|
|
276
|
+
|
|
277
|
+
PixivFlow 可以完全独立使用。下面是同一作者生态里与它相关的项目,以及各自负责什么:
|
|
278
|
+
|
|
279
|
+
| 项目 | 是什么 | 什么时候需要 |
|
|
280
|
+
| --- | --- | --- |
|
|
281
|
+
| [TelePost](https://github.com/redtidev1918/TelePost) | Telegram 频道投稿、审核与自动化发布平台 | 想把下载结果投进 Telegram 频道、先人工审核再发布时,把它配成 delivery 下游即可。这只是可选组合,PixivFlow 不依赖它 |
|
|
282
|
+
| [pixivflow-telepost-deploy](https://github.com/redtidev1918/pixivflow-telepost-deploy) | PixivFlow + TelePost 的部署与运维套件(Docker / VPS / 云平台) | 想一次性把上面两个项目部署并运维起来时。只跑 PixivFlow 不需要它 |
|
|
283
|
+
| [pixivflow-webui](https://github.com/redtidev1918/pixivflow-webui) | PixivFlow 的 WebUI 前端 | 想用图形界面管理下载与计划 |
|
|
284
|
+
| [pixiv-token-getter](https://github.com/redtidev1918/pixiv-token-getter) | PKCE OAuth 登录库与 CLI(`ptg`) | PixivFlow 的登录依赖;也可以单独用于获取 Pixiv token |
|
|
285
|
+
|
|
233
286
|
## 问题反馈
|
|
234
287
|
|
|
235
288
|
Bug 与功能建议请提交到
|