pixivflow 3.2.0 → 3.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (58) hide show
  1. package/README.md +26 -1
  2. package/dist/commands/SchedulerCommand.js +31 -51
  3. package/dist/commands/scheduler-runtime.js +47 -7
  4. package/dist/config/defaults.d.ts +2 -0
  5. package/dist/config/defaults.js +6 -0
  6. package/dist/config/types.d.ts +20 -0
  7. package/dist/config/validation.js +11 -0
  8. package/dist/delivery/DeliveryDispatcher.d.ts +14 -0
  9. package/dist/delivery/DeliveryDispatcher.js +14 -0
  10. package/dist/delivery/EventCallbackDelivery.d.ts +54 -0
  11. package/dist/delivery/EventCallbackDelivery.js +89 -0
  12. package/dist/delivery/OutboxWorker.d.ts +11 -0
  13. package/dist/delivery/OutboxWorker.js +51 -1
  14. package/dist/package.json +1 -1
  15. package/dist/scheduler/CandidateSearchParams.d.ts +76 -0
  16. package/dist/scheduler/CandidateSearchParams.js +94 -0
  17. package/dist/scheduler/JobCancellation.d.ts +36 -0
  18. package/dist/scheduler/JobCancellation.js +73 -0
  19. package/dist/scheduler/JobEventStream.d.ts +94 -0
  20. package/dist/scheduler/JobEventStream.js +251 -0
  21. package/dist/scheduler/JobFacade.d.ts +298 -0
  22. package/dist/scheduler/JobFacade.js +622 -0
  23. package/dist/scheduler/JobProjection.d.ts +67 -0
  24. package/dist/scheduler/JobProjection.js +32 -0
  25. package/dist/scheduler/JobView.d.ts +34 -0
  26. package/dist/scheduler/JobView.js +82 -0
  27. package/dist/scheduler/ManualJobAdmission.d.ts +126 -0
  28. package/dist/scheduler/ManualJobAdmission.js +310 -0
  29. package/dist/scheduler/ManualJobService.d.ts +57 -0
  30. package/dist/scheduler/ManualJobService.js +91 -0
  31. package/dist/scheduler/ManualRefetchAdapter.d.ts +26 -0
  32. package/dist/scheduler/ManualRefetchAdapter.js +41 -0
  33. package/dist/scheduler/MultiScheduleManager.d.ts +17 -0
  34. package/dist/scheduler/MultiScheduleManager.js +68 -6
  35. package/dist/scheduler/ProtocolErrors.d.ts +68 -0
  36. package/dist/scheduler/ProtocolErrors.js +94 -0
  37. package/dist/scheduler/ScheduleTriggerServer.d.ts +56 -7
  38. package/dist/scheduler/ScheduleTriggerServer.js +166 -1
  39. package/dist/scheduler/SlotBusinessStatus.d.ts +5 -1
  40. package/dist/scheduler/SlotBusinessStatus.js +6 -0
  41. package/dist/scheduler/SlotCoordinator.d.ts +49 -5
  42. package/dist/scheduler/SlotCoordinator.js +89 -22
  43. package/dist/scheduler/StallSweep.d.ts +65 -0
  44. package/dist/scheduler/StallSweep.js +105 -0
  45. package/dist/scheduler/TargetOutcome.d.ts +39 -1
  46. package/dist/scheduler/TargetOutcome.js +51 -1
  47. package/dist/scheduler/ledger-time.d.ts +23 -0
  48. package/dist/scheduler/ledger-time.js +37 -0
  49. package/dist/storage/DatabaseMigration.js +48 -0
  50. package/dist/storage/repositories/DeliveryRepository.d.ts +6 -0
  51. package/dist/storage/repositories/DeliveryRepository.js +13 -0
  52. package/dist/storage/repositories/OutboxRepository.d.ts +73 -1
  53. package/dist/storage/repositories/OutboxRepository.js +142 -0
  54. package/dist/storage/repositories/SlotRepository.d.ts +34 -0
  55. package/dist/storage/repositories/SlotRepository.js +61 -2
  56. package/dist/version.js +1 -1
  57. package/dist/webui/package.json +1 -1
  58. package/package.json +1 -1
package/README.md CHANGED
@@ -257,6 +257,31 @@ pixivflow outbox cancel <id> # 只取消尚未执行的 row
257
257
  租约、pending 投递、dead 行,`--repair` 收敛)与 `pixivflow reconcile`
258
258
  (把下游已确认的历史重复登记进投递账本,默认 dry-run)。
259
259
 
260
+ ### 面向编排方的 Job API(Workflow Protocol v1)
261
+
262
+ PixivFlow 只做「内容采集与处理引擎」:发现、筛选、下载、元数据与媒体处理、候选生成。
263
+ **审核 / 替换 / 队列 / 发布等业务语义不在这里实现**,由编排方(如 TelePost)负责。
264
+ 两者之间通过一套稳定的 **Workflow Protocol v1** 用通用 Job 面交互,编排方不互调
265
+ 内部接口、也不往请求里塞业务字段。scheduler 对外暴露:
266
+
267
+ | 端点 | 作用 |
268
+ | --- | --- |
269
+ | `GET /capabilities` | 能力发现:如实声明 `protocol_version` 与 `features`(`idempotency` / `cancel` / `events` / ...),能力声明必须诚实,没声明就是没有 |
270
+ | `POST /jobs` | 提交一个采集任务(如 `candidate_search`):`idempotency_key` 幂等,携带消费者声明的 `callback_url` 时事件会推送到该地址 |
271
+ | `GET /jobs` · `GET /jobs/{job_id}` | 按幂等键或按 id 查作业状态(含 `status`、`created_at`、`heartbeat_at`、`deadline_at`、`result`/`error`) |
272
+ | `GET /jobs/{job_id}/events` · `POST /jobs/{job_id}/events/ack` | 事件流与对账:`?after=` 续拉、`unacked` 计数、单调游标 ack;回调只是加速通道,漏送也能从这里补齐 |
273
+
274
+ **作业生命周期可信、可查,不再永久静默。** 每个作业都是持久化的、带完整状态机
275
+ (`queued → running → succeeded|failed|expired|cancelled`)、30 秒心跳、活性预算、
276
+ 启动恢复与终态事件;终态必有 `result` 或 `error`。编排方可以随时查询作业、订阅事件、
277
+ 对账补拉——再也回不到「点了重抓之后什么反应都没有」的那种状态。写入方与读取方都受
278
+ [外包层](docs/architecture/delivery-runtime.md) 与仓库内 `protocol/v1/`(schema、夹具
279
+ 与验收清单)约束;规范与跨仓部署见
280
+ [pixivflow-telepost-deploy](https://github.com/redtidev1918/pixivflow-telepost-deploy)
281
+ 的 [Workflow Protocol 文档](https://github.com/redtidev1918/pixivflow-telepost-deploy/blob/main/docs/architecture/workflow-protocol.md)。
282
+ 旧的 `/internal/targets/:targetId/refetch` 仍作为字节兼容 shim 可用,与 Job 面共享同一
283
+ 身份空间,但新集成请走 Job 面 / `POST /jobs`。
284
+
260
285
  ## 常用命令
261
286
 
262
287
  | 命令 | 说明 |
@@ -316,7 +341,7 @@ PixivFlow 可以完全独立使用。下面是同一作者生态里与它相关
316
341
 
317
342
  | 项目 | 是什么 | 什么时候需要 |
318
343
  | --- | --- | --- |
319
- | [TelePost](https://github.com/redtidev1918/TelePost) | Telegram 频道投稿、审核与自动化发布平台 | 想把下载结果投进 Telegram 频道、先人工审核再发布时,把它配成 delivery 下游即可。这只是可选组合,PixivFlow 不依赖它 |
344
+ | [TelePost](https://github.com/redtidev1918/TelePost) | Telegram 频道投稿、审核与自动化发布平台 | 作为投递下游接收下载结果,或作为**编排方**通过 [Workflow Protocol v1](#面向编排方的-job-apiworkflow-protocol-v1) 驱动 PixivFlow 提交候选查找作业(审核重抓即走此路)。这只是可选组合,PixivFlow 不依赖它 |
320
345
  | [pixivflow-telepost-deploy](https://github.com/redtidev1918/pixivflow-telepost-deploy) | PixivFlow + TelePost 的部署与运维套件(Docker / VPS / 云平台) | 想一次性把上面两个项目部署并运维起来时。只跑 PixivFlow 不需要它 |
321
346
  | [pixivflow-webui](https://github.com/redtidev1918/pixivflow-webui) | PixivFlow 的 WebUI 前端 | 想用图形界面管理下载与计划 |
322
347
  | [pixivflow-desktop](https://github.com/redtidev1918/pixivflow-desktop) | PixivFlow 的官方桌面客户端(macOS / Windows / Linux),内置本仓库运行时与 WebUI | 想要双击即用的原生应用、不想自己装 Node 或起服务时。业务逻辑仍在本仓库,桌面端只负责启动、守护与打包 |
@@ -10,6 +10,9 @@ const config_1 = require("../config");
10
10
  const MultiScheduleManager_1 = require("../scheduler/MultiScheduleManager");
11
11
  const ScheduleTriggerServer_1 = require("../scheduler/ScheduleTriggerServer");
12
12
  const SlotCoordinator_1 = require("../scheduler/SlotCoordinator");
13
+ const ManualJobAdmission_1 = require("../scheduler/ManualJobAdmission");
14
+ const ManualJobService_1 = require("../scheduler/ManualJobService");
15
+ const ManualRefetchAdapter_1 = require("../scheduler/ManualRefetchAdapter");
13
16
  const schedules_1 = require("../scheduler/schedules");
14
17
  const targetRoutes_1 = require("../delivery/targetRoutes");
15
18
  const scheduler_runtime_1 = require("./scheduler-runtime");
@@ -82,6 +85,22 @@ class SchedulerCommand extends Command_1.BaseCommand {
82
85
  const coordinator = new SlotCoordinator_1.SlotCoordinator(runtime.database);
83
86
  const resolveConfig = () => manager['activeConfig'] ?? runtime.config;
84
87
  const findPlan = (cfg, scheduleId) => cfg.schedules?.find((s) => s.id === scheduleId && s.enabled !== false);
88
+ // The ONE admission path for consumer-initiated candidate search. Both
89
+ // the legacy refetch endpoint (below) and the generic protocol job
90
+ // surface (`jobs`, below) translate their request into this call, so
91
+ // they share one identity space, one target resolution and one durable
92
+ // work item per idempotency key.
93
+ const admission = new ManualJobAdmission_1.ManualJobAdmission({
94
+ database: runtime.database,
95
+ coordinator,
96
+ config: resolveConfig,
97
+ admit: (planId, slot, targetId) => manager.triggerSchedule(planId, { slot, onlyTarget: targetId, triggerSource: 'manual' }),
98
+ });
99
+ const jobService = new ManualJobService_1.ManualJobService({
100
+ database: runtime.database,
101
+ config: resolveConfig,
102
+ admission,
103
+ });
85
104
  // Durable dispatch. The trigger endpoint records the occurrence FIRST,
86
105
  // then hands it to the shared scheduler and answers immediately. It must
87
106
  // never await the download itself: a 10-40 minute run outlives the
@@ -159,57 +178,18 @@ class SchedulerCommand extends Command_1.BaseCommand {
159
178
  };
160
179
  },
161
180
  drainOutbox: () => runtime.drainOutbox(),
162
- refetch: async (targetId, requestId, correlationId) => {
163
- const cfg = resolveConfig();
164
- const plans = (cfg.schedules ?? []).filter((plan) => plan.enabled !== false && (0, schedules_1.selectScheduleTargets)(cfg.targets, plan).some((target) => target.id === targetId));
165
- if (plans.length === 0)
166
- throw new Error('unknown target');
167
- if (plans.length !== 1)
168
- throw new Error('ambiguous target');
169
- const plan = plans[0];
170
- const target = (0, schedules_1.selectScheduleTargets)(cfg.targets, plan).find((item) => item.id === targetId);
171
- const deliveryName = (0, targetRoutes_1.primaryDeliveryName)(target);
172
- const delivery = deliveryName ? cfg.delivery?.targets?.[deliveryName] : undefined;
173
- if (delivery?.type !== 'httpMultipart' || !delivery.refetchOutcomeUrl?.trim()) {
174
- throw new Error('refetch outcome endpoint not configured');
175
- }
176
- if ((target.delivery?.fields?.refetch_request_id ?? delivery.fields?.refetch_request_id) !== '{{refetchRequestId}}') {
177
- throw new Error('refetch_request_id delivery field not configured');
178
- }
179
- const now = new Date();
180
- const date = new Intl.DateTimeFormat('en-CA', {
181
- timeZone: plan.timezone ?? 'UTC', year: 'numeric', month: '2-digit', day: '2-digit',
182
- }).format(now);
183
- const slot = {
184
- slotId: `${plan.id}@manual-${requestId.toLowerCase()}`,
185
- scheduleId: plan.id,
186
- occurrenceAt: now.getTime(),
187
- occurrenceDate: date,
188
- occurrenceLabel: 'manual',
189
- timezone: plan.timezone ?? 'UTC',
190
- triggerSource: 'manual',
191
- slotName: '审核群重抓',
192
- slotDate: date,
193
- manualRequestId: requestId,
194
- correlationId: correlationId || undefined,
195
- };
196
- const existing = runtime.database.slots.getSlot(slot.slotId);
197
- if (existing && (existing.scheduleId !== plan.id || existing.targetIds.length !== 1 || existing.targetIds[0] !== targetId)) {
198
- throw new Error('ambiguous target');
199
- }
200
- const prepared = coordinator.prepare(slot, plan, [target]);
201
- if (prepared.alreadyCompleted)
202
- return { slotId: slot.slotId, disposition: 'already_completed' };
203
- const started = manager.triggerSchedule(plan.id, { slot, onlyTarget: targetId, triggerSource: 'manual' });
204
- return { slotId: slot.slotId, disposition: started ? 'accepted' : 'queued' };
205
- },
206
- refetchStatus: (targetId, requestId) => {
207
- const slot = runtime.database.slots.findManualSlot(requestId, targetId);
208
- const cell = slot && runtime.database.slots.getCell(slot.id, targetId);
209
- return slot && cell
210
- ? { requestId, slotId: slot.id, state: cell.status, slotStatus: slot.status }
211
- : null;
212
- },
181
+ /**
182
+ * The generic Workflow Protocol v1 job surface (`/capabilities`,
183
+ * `/jobs`). It answers from the same ledger as the legacy endpoints
184
+ * and submits through the same admission path, so a job submitted
185
+ * here IS the work item a legacy refetch would have created.
186
+ */
187
+ jobs: jobService,
188
+ // Legacy adapters (protocol §3.1): the historic refetch shape over
189
+ // the same shared admission, so `requestId` and `idempotency_key`
190
+ // are one identity space.
191
+ refetch: (0, ManualRefetchAdapter_1.legacyRefetchSubmit)(admission),
192
+ refetchStatus: (0, ManualRefetchAdapter_1.legacyRefetchStatus)(runtime.database),
213
193
  /**
214
194
  * Manual recovery of a FAILED target (§manual-recovery). Distinct
215
195
  * from refetch: there is no review chain to replace and no
@@ -61,12 +61,14 @@ const token_maintenance_1 = require("../utils/token-maintenance");
61
61
  const schedules_1 = require("../scheduler/schedules");
62
62
  const SlotCoordinator_1 = require("../scheduler/SlotCoordinator");
63
63
  const RecoveryPolicy_1 = require("../scheduler/RecoveryPolicy");
64
+ const CandidateSearchParams_1 = require("../scheduler/CandidateSearchParams");
64
65
  const targetRoutes_1 = require("../delivery/targetRoutes");
65
66
  const DeliveryLedgerPort_1 = require("../delivery/DeliveryLedgerPort");
66
67
  const OutboxWorker_1 = require("../delivery/OutboxWorker");
67
68
  const settleDeliveryTerminal_1 = require("../delivery/settleDeliveryTerminal");
68
69
  const LegacyOutboxMigration_1 = require("../delivery/LegacyOutboxMigration");
69
70
  const NotificationPolicy_1 = require("../notification/NotificationPolicy");
71
+ const JobEventStream_1 = require("../scheduler/JobEventStream");
70
72
  const node_crypto_1 = require("node:crypto");
71
73
  const placeholders_1 = require("../config/placeholders");
72
74
  const logger_1 = require("../logger");
@@ -293,8 +295,20 @@ async function createSchedulerRuntime(configPathArg) {
293
295
  // the long-running scheduler daemon; run-once drains explicitly before exit.
294
296
  const deliveryDispatcher = new DeliveryDispatcher_1.DeliveryDispatcher(config.delivery, buildProxyUrl(config.network));
295
297
  const notificationPolicy = new NotificationPolicy_1.NotificationPolicy(database, config);
298
+ // Consumer-facing job events (§events). The stream is a projection of the same
299
+ // `delivery_events` log; this sweep repairs a declared `callback_url` whose
300
+ // terminal event was lost with the process, exactly like
301
+ // reconcileScheduleSummaries repairs a terminal slot that was never notified.
302
+ const jobEvents = new JobEventStream_1.JobEventStream({
303
+ database,
304
+ config: () => config,
305
+ now: () => Date.now(),
306
+ });
296
307
  const outboxWorker = new OutboxWorker_1.OutboxWorker(database, deliveryDispatcher, {
297
- beforeDrain: () => notificationPolicy.reconcileScheduleSummaries(),
308
+ beforeDrain: () => {
309
+ notificationPolicy.reconcileScheduleSummaries();
310
+ jobEvents.reconcileOutstanding();
311
+ },
298
312
  retryBaseMs: config.delivery?.outboxRetryBaseMs,
299
313
  retryMaxMs: config.delivery?.outboxRetryMaxMs,
300
314
  // A confirmed ACK settles the owning Slot cell (submitted / duplicate /
@@ -342,6 +356,30 @@ async function createSchedulerRuntime(configPathArg) {
342
356
  targets: targets.map((t) => t.id),
343
357
  });
344
358
  }
359
+ // Occurrence-scoped RETRIEVAL view of a generic `candidate_search` job (§6).
360
+ // Same pattern as the recovery policy above: a pure map over THIS
361
+ // occurrence's target snapshot, never a write to the global config. A slot
362
+ // with no stored view (every scheduled occurrence, and every slot written
363
+ // before the column existed) keeps the configured behaviour untouched.
364
+ const requestedSearch = (0, CandidateSearchParams_1.parseCandidateSearchParamsJson)(providedSlot?.paramsJson);
365
+ if (requestedSearch) {
366
+ targets = targets.map((target) => (0, CandidateSearchParams_1.applyCandidateSearchParams)(target, requestedSearch));
367
+ logger_1.logger.info('Generic job retrieval view applied to this occurrence', {
368
+ scheduleId: schedule.id,
369
+ slot: providedSlot?.slotId,
370
+ targets: targets.map((t) => t.id),
371
+ });
372
+ }
373
+ // `constraints.exclude` is a run-level duplicate filter, so it joins the
374
+ // history the runner already refuses to select from instead of inventing a
375
+ // second exclusion mechanism.
376
+ const requestedExclusions = requestedSearch ? (0, CandidateSearchParams_1.excludedWorkIdsFromParams)(requestedSearch) : null;
377
+ const excludedWorkIds = requestedExclusions
378
+ ? {
379
+ illustration: [...(options.excludedWorkIds?.illustration ?? []), ...requestedExclusions.illustration],
380
+ novel: [...(options.excludedWorkIds?.novel ?? []), ...requestedExclusions.novel],
381
+ }
382
+ : options.excludedWorkIds;
345
383
  if (targets.length === 0) {
346
384
  logger_1.logger.warn('Scheduled plan has no selected targets; skipping', {
347
385
  scheduleId: schedule.id,
@@ -362,6 +400,9 @@ async function createSchedulerRuntime(configPathArg) {
362
400
  * overwrite the terminal `failed` record the abandon path wrote).
363
401
  */
364
402
  let slotAbandoned = false;
403
+ // This run's lease identity. It is passed to `finish` so a rollup can tell
404
+ // "my own live lease" from "another worker is on this slot right now".
405
+ const runOwner = `run-${process.pid}-${(0, node_crypto_1.randomUUID)().slice(0, 8)}`;
365
406
  if (!adhoc) {
366
407
  if (providedSlot) {
367
408
  slotCtx = providedSlot;
@@ -380,7 +421,6 @@ async function createSchedulerRuntime(configPathArg) {
380
421
  slotCtx = resolved.context;
381
422
  }
382
423
  const activeSlot = slotCtx;
383
- const runOwner = `run-${process.pid}-${(0, node_crypto_1.randomUUID)().slice(0, 8)}`;
384
424
  const prepared = coordinator.prepare(slotCtx, schedule, targets);
385
425
  if (prepared.alreadyCompleted && !onlyTarget) {
386
426
  logger_1.logger.info('Slot already terminal; nothing to do', {
@@ -474,7 +514,7 @@ async function createSchedulerRuntime(configPathArg) {
474
514
  let runTargets = selected.map((p) => p.target);
475
515
  if (slotCtx && runTargets.length === 0) {
476
516
  logger_1.logger.info('All slot cells already complete', { slot: slotCtx.slotId });
477
- coordinator.finish(slotCtx, schedule, targets);
517
+ coordinator.finish(slotCtx, schedule, targets, { leaseOwner: runOwner });
478
518
  for (const target of targets)
479
519
  if (target.id)
480
520
  notificationPolicy.noteTerminalRefetchCell(slotCtx.slotId, target.id);
@@ -540,8 +580,8 @@ async function createSchedulerRuntime(configPathArg) {
540
580
  const manager = new DownloadManager_1.DownloadManager(scoped, pixivClient, database, fileService);
541
581
  if (targetExecutionContexts)
542
582
  manager.setTargetExecutionContexts(targetExecutionContexts);
543
- if (options.excludedWorkIds)
544
- manager.setProcessedWorkIds(options.excludedWorkIds);
583
+ if (excludedWorkIds)
584
+ manager.setProcessedWorkIds(excludedWorkIds);
545
585
  manager.setTargetOutcomeHook(outcomeHook);
546
586
  if (scheduleSlot) {
547
587
  manager.slotContext = {
@@ -670,7 +710,7 @@ async function createSchedulerRuntime(configPathArg) {
670
710
  // - the process is going away (`shutdown`). Deliberately leave the Slot
671
711
  // non-terminal so recovery resumes the same occurrence after restart.
672
712
  if (slotCtx && shouldTerminaliseAbortedSlot(activeAbortOrigin, slotAbandoned)) {
673
- coordinator.finish(slotCtx, schedule, targets);
713
+ coordinator.finish(slotCtx, schedule, targets, { leaseOwner: runOwner });
674
714
  for (const target of targets)
675
715
  if (target.id)
676
716
  notificationPolicy.noteTerminalRefetchCell(slotCtx.slotId, target.id);
@@ -691,7 +731,7 @@ async function createSchedulerRuntime(configPathArg) {
691
731
  }
692
732
  const duration = Math.round((Date.now() - startTime) / 1000);
693
733
  if (slotCtx && !slotAbandoned) {
694
- const summary = coordinator.finish(slotCtx, schedule, targets);
734
+ const summary = coordinator.finish(slotCtx, schedule, targets, { leaseOwner: runOwner });
695
735
  for (const target of targets)
696
736
  if (target.id)
697
737
  notificationPolicy.noteTerminalRefetchCell(slotCtx.slotId, target.id);
@@ -24,6 +24,8 @@ export declare const DEFAULT_CONFIG: {
24
24
  readonly watchConfig: true;
25
25
  readonly reloadDebounceMs: 500;
26
26
  readonly queueLimit: 8;
27
+ readonly queuedTimeoutMs: number;
28
+ readonly stallTimeoutMs: number;
27
29
  readonly trigger: {
28
30
  readonly port: 8090;
29
31
  readonly host: "0.0.0.0";
@@ -27,6 +27,12 @@ exports.DEFAULT_CONFIG = {
27
27
  watchConfig: true,
28
28
  reloadDebounceMs: 500,
29
29
  queueLimit: 8,
30
+ // Liveness budgets (§liveness). Generous on purpose: production runs take
31
+ // 10-40 minutes and are sometimes queued hours behind a busy account, so a
32
+ // slot is only terminalised when its LEASE is dead and it stopped
33
+ // progressing — never merely because it is slow.
34
+ queuedTimeoutMs: 30 * 60 * 1000,
35
+ stallTimeoutMs: 15 * 60 * 1000,
30
36
  trigger: {
31
37
  port: 8090,
32
38
  host: '0.0.0.0',
@@ -551,6 +551,26 @@ export interface SchedulerRuntimeConfig {
551
551
  * One pending run per plan is retained; extra ticks are skipped.
552
552
  */
553
553
  queueLimit?: number;
554
+ /**
555
+ * Liveness budget for admitted work that never started (§liveness). A Slot
556
+ * that is still `pending` with no live lease after this long is terminalised
557
+ * as `queued_too_long` (its cells fail with that reason) instead of being
558
+ * re-dispatched forever and reporting an eternal "unfinished" job. Values
559
+ * below one minute are ignored (the sweep then uses its default) and warned
560
+ * about at config validation; nothing here is fatal.
561
+ * Default: 1800000ms (30 minutes).
562
+ */
563
+ queuedTimeoutMs?: number;
564
+ /**
565
+ * Liveness budget for a claimed run that stopped progressing (§liveness). A
566
+ * `running` Slot whose lease expired AND whose last heartbeat is older than
567
+ * this is terminalised as `stalled_no_heartbeat`. A LIVE lease is never
568
+ * eligible, so a genuinely long search is never killed. Values below one
569
+ * minute are ignored (the sweep then uses its default) and warned about at
570
+ * config validation; nothing here is fatal.
571
+ * Default: 900000ms (15 minutes).
572
+ */
573
+ stallTimeoutMs?: number;
554
574
  /**
555
575
  * External-mode run-to-completion lifecycle. When true, the daemon exits the
556
576
  * process once its OWN durable ledger says there is nothing left to do, so a
@@ -639,6 +639,17 @@ function validateConfig(config, location, databasePath) {
639
639
  if (queueLimit !== undefined && (!Number.isInteger(queueLimit) || queueLimit < 0 || queueLimit > 100)) {
640
640
  errors.push('schedulerRuntime.queueLimit: Must be an integer between 0 and 100');
641
641
  }
642
+ // Liveness budgets (§liveness): a WARNING, never an error. A bad value must
643
+ // not stop the daemon from starting; the sweep clamps to its own default so
644
+ // the behaviour stays safe either way.
645
+ for (const key of ['queuedTimeoutMs', 'stallTimeoutMs']) {
646
+ const value = rt[key];
647
+ if (value === undefined)
648
+ continue;
649
+ if (typeof value !== 'number' || !Number.isInteger(value) || value < 60000) {
650
+ warnings.push(`schedulerRuntime.${key}: Should be an integer of at least 60000 (milliseconds); the liveness sweep uses its default instead`);
651
+ }
652
+ }
642
653
  if (rt.exitWhenIdle !== undefined && typeof rt.exitWhenIdle !== 'boolean') {
643
654
  errors.push('schedulerRuntime.exitWhenIdle: Must be a boolean');
644
655
  }
@@ -1,10 +1,12 @@
1
1
  import { DeliveryConfig } from '../config';
2
+ import { EventCallbackPayload } from './EventCallbackDelivery';
2
3
  import { ReadinessProbeResult } from './HttpMultipartDelivery';
3
4
  import { DeliveryNotificationRequest, DeliveryRequest, DeliveryResult } from './types';
4
5
  /** Resolves named delivery targets without coupling the outbox to a provider. */
5
6
  export declare class DeliveryDispatcher {
6
7
  private readonly config;
7
8
  private readonly proxyUrl?;
9
+ private readonly eventCallbacks;
8
10
  constructor(config: DeliveryConfig | undefined, proxyUrl?: string | undefined);
9
11
  hasTarget(name: string): boolean;
10
12
  isReady(name: string): Promise<boolean>;
@@ -12,5 +14,17 @@ export declare class DeliveryDispatcher {
12
14
  readinessProbe(name: string): Promise<ReadinessProbeResult>;
13
15
  deliver(name: string, request: DeliveryRequest): Promise<DeliveryResult>;
14
16
  notify(name: string, request: DeliveryNotificationRequest): Promise<DeliveryResult>;
17
+ /**
18
+ * Deliver one `$defs/Event` to a Task's declared `callback_url`.
19
+ *
20
+ * This does NOT go through a named delivery target: the endpoint is chosen by
21
+ * the consumer per Task, not by our config. It is still a delivery in every
22
+ * other sense — the caller is the outbox worker, so retry, backoff,
23
+ * dead-lettering and the idempotency index all apply unchanged.
24
+ */
25
+ deliverEventCallback(url: string, request: {
26
+ payload: EventCallbackPayload;
27
+ idempotencyKey: string;
28
+ }): Promise<number>;
15
29
  }
16
30
  //# sourceMappingURL=DeliveryDispatcher.d.ts.map
@@ -2,6 +2,7 @@
2
2
  Object.defineProperty(exports, "__esModule", { value: true });
3
3
  exports.DeliveryDispatcher = void 0;
4
4
  const errors_1 = require("../utils/errors");
5
+ const EventCallbackDelivery_1 = require("./EventCallbackDelivery");
5
6
  const HttpMultipartDelivery_1 = require("./HttpMultipartDelivery");
6
7
  const TelegramReviewDelivery_1 = require("./TelegramReviewDelivery");
7
8
  const WebhookDelivery_1 = require("./WebhookDelivery");
@@ -9,9 +10,11 @@ const WebhookDelivery_1 = require("./WebhookDelivery");
9
10
  class DeliveryDispatcher {
10
11
  config;
11
12
  proxyUrl;
13
+ eventCallbacks;
12
14
  constructor(config, proxyUrl) {
13
15
  this.config = config;
14
16
  this.proxyUrl = proxyUrl;
17
+ this.eventCallbacks = new EventCallbackDelivery_1.EventCallbackDelivery(proxyUrl);
15
18
  }
16
19
  hasTarget(name) {
17
20
  return Boolean(this.config?.targets?.[name]);
@@ -67,6 +70,17 @@ class DeliveryDispatcher {
67
70
  }
68
71
  return new HttpMultipartDelivery_1.HttpMultipartDelivery(target, this.proxyUrl).notifyOnce(request);
69
72
  }
73
+ /**
74
+ * Deliver one `$defs/Event` to a Task's declared `callback_url`.
75
+ *
76
+ * This does NOT go through a named delivery target: the endpoint is chosen by
77
+ * the consumer per Task, not by our config. It is still a delivery in every
78
+ * other sense — the caller is the outbox worker, so retry, backoff,
79
+ * dead-lettering and the idempotency index all apply unchanged.
80
+ */
81
+ async deliverEventCallback(url, request) {
82
+ return this.eventCallbacks.deliver({ url, payload: request.payload, idempotencyKey: request.idempotencyKey });
83
+ }
70
84
  }
71
85
  exports.DeliveryDispatcher = DeliveryDispatcher;
72
86
  //# sourceMappingURL=DeliveryDispatcher.js.map
@@ -0,0 +1,54 @@
1
+ /**
2
+ * Deliver one `$defs/Event` to the `callback_url` an accepted `$defs/Task`
3
+ * declared.
4
+ *
5
+ * At-least-once, exactly like every other delivery in this repo: the caller is
6
+ * the EXISTING `OutboxWorker`, which owns retry, backoff and dead-lettering. So
7
+ * this module performs exactly ONE HTTP POST and reports the outcome as a thrown
8
+ * error whose `status` lets `classifyError` decide whether the mistake was
9
+ * retryable (5xx / 429 -> retry, 4xx -> dead-letter). Transport errors are
10
+ * rethrown untouched, which keeps them retryable by construction.
11
+ *
12
+ * A separate delivery path is deliberately NOT introduced: the callback rides
13
+ * the same outbox, the same `idx_outbox_key` idempotency index and the same
14
+ * `delivery_events` audit log as outcome delivery.
15
+ */
16
+ export interface EventCallbackPayload {
17
+ /** The `$defs/Event`. The POST body is exactly this object. */
18
+ event: unknown;
19
+ job_id: string;
20
+ /**
21
+ * Correlation hint read by `OutboxWorker.contextCorrelation` so the outbox
22
+ * audit rows for this callback are linked to the job's slot.
23
+ */
24
+ context?: {
25
+ slotId?: string;
26
+ };
27
+ }
28
+ export declare const EVENT_CALLBACK_TIMEOUT_MS = 30000;
29
+ /**
30
+ * A callback endpoint answered with a non-2xx status. `status` is carried as an
31
+ * own enumerable property because `Error.message` is not a machine-readable
32
+ * contract — the outbox hands it to `classifyError(error, status)`.
33
+ */
34
+ export declare class EventCallbackError extends Error {
35
+ readonly status: number;
36
+ constructor(message: string, status: number);
37
+ }
38
+ export declare class EventCallbackDelivery {
39
+ private readonly proxyUrl?;
40
+ private readonly timeoutMs;
41
+ private readonly dispatcher?;
42
+ constructor(proxyUrl?: string | undefined, timeoutMs?: number);
43
+ /**
44
+ * One POST of one event. Resolves with the HTTP status on success; throws
45
+ * `EventCallbackError` on a non-2xx status and the raw transport error when
46
+ * the request never completed.
47
+ */
48
+ deliver(request: {
49
+ url: string;
50
+ payload: EventCallbackPayload;
51
+ idempotencyKey: string;
52
+ }): Promise<number>;
53
+ }
54
+ //# sourceMappingURL=EventCallbackDelivery.d.ts.map
@@ -0,0 +1,89 @@
1
+ "use strict";
2
+ /**
3
+ * Deliver one `$defs/Event` to the `callback_url` an accepted `$defs/Task`
4
+ * declared.
5
+ *
6
+ * At-least-once, exactly like every other delivery in this repo: the caller is
7
+ * the EXISTING `OutboxWorker`, which owns retry, backoff and dead-lettering. So
8
+ * this module performs exactly ONE HTTP POST and reports the outcome as a thrown
9
+ * error whose `status` lets `classifyError` decide whether the mistake was
10
+ * retryable (5xx / 429 -> retry, 4xx -> dead-letter). Transport errors are
11
+ * rethrown untouched, which keeps them retryable by construction.
12
+ *
13
+ * A separate delivery path is deliberately NOT introduced: the callback rides
14
+ * the same outbox, the same `idx_outbox_key` idempotency index and the same
15
+ * `delivery_events` audit log as outcome delivery.
16
+ */
17
+ Object.defineProperty(exports, "__esModule", { value: true });
18
+ exports.EventCallbackDelivery = exports.EventCallbackError = exports.EVENT_CALLBACK_TIMEOUT_MS = void 0;
19
+ exports.EVENT_CALLBACK_TIMEOUT_MS = 30_000;
20
+ /**
21
+ * A callback endpoint answered with a non-2xx status. `status` is carried as an
22
+ * own enumerable property because `Error.message` is not a machine-readable
23
+ * contract — the outbox hands it to `classifyError(error, status)`.
24
+ */
25
+ class EventCallbackError extends Error {
26
+ status;
27
+ constructor(message, status) {
28
+ super(message);
29
+ this.name = 'EventCallbackError';
30
+ this.status = status;
31
+ }
32
+ }
33
+ exports.EventCallbackError = EventCallbackError;
34
+ class EventCallbackDelivery {
35
+ proxyUrl;
36
+ timeoutMs;
37
+ dispatcher;
38
+ constructor(proxyUrl, timeoutMs = exports.EVENT_CALLBACK_TIMEOUT_MS) {
39
+ this.proxyUrl = proxyUrl;
40
+ this.timeoutMs = timeoutMs;
41
+ if (proxyUrl) {
42
+ // Same egress path (and same undici ProxyAgent wiring) as every other
43
+ // outbound delivery.
44
+ const { ProxyAgent } = require('undici');
45
+ this.dispatcher = new ProxyAgent(proxyUrl);
46
+ }
47
+ }
48
+ /**
49
+ * One POST of one event. Resolves with the HTTP status on success; throws
50
+ * `EventCallbackError` on a non-2xx status and the raw transport error when
51
+ * the request never completed.
52
+ */
53
+ async deliver(request) {
54
+ const controller = new AbortController();
55
+ const timer = setTimeout(() => controller.abort(), this.timeoutMs);
56
+ timer.unref?.();
57
+ const options = {
58
+ method: 'POST',
59
+ headers: {
60
+ 'Content-Type': 'application/json',
61
+ 'X-PixivFlow-Delivery': 'job-event',
62
+ // The consumer deduplicates on the event_id; the outbox deduplicates on
63
+ // its own key. Sending both is deliberate: either side can converge.
64
+ 'X-Idempotency-Key': request.idempotencyKey,
65
+ },
66
+ body: JSON.stringify(request.payload.event),
67
+ signal: controller.signal,
68
+ };
69
+ if (this.dispatcher)
70
+ options.dispatcher = this.dispatcher;
71
+ try {
72
+ const response = await fetch(request.url, options);
73
+ // Drain (and bound) the body so the socket can be reused/closed.
74
+ await response.text().catch(() => '');
75
+ if (response.status >= 200 && response.status < 300)
76
+ return response.status;
77
+ // 409 means the consumer already recorded this event_id — idempotent
78
+ // convergence, not a failure.
79
+ if (response.status === 409)
80
+ return response.status;
81
+ throw new EventCallbackError(`job event callback answered HTTP ${response.status}`, response.status);
82
+ }
83
+ finally {
84
+ clearTimeout(timer);
85
+ }
86
+ }
87
+ }
88
+ exports.EventCallbackDelivery = EventCallbackDelivery;
89
+ //# sourceMappingURL=EventCallbackDelivery.js.map
@@ -81,6 +81,17 @@ export declare class OutboxWorker {
81
81
  private pump;
82
82
  /** Probe via structured readinessProbe when available; boolean fallback stays compatible. */
83
83
  private checkReadiness;
84
+ /**
85
+ * Honour a consumer cancel.
86
+ *
87
+ * A job cancelled while a delivery attempt was in flight (or waiting to
88
+ * retry) must not be delivered afterwards: the ledger's cell already carries
89
+ * the terminal `cancelled_by_consumer` verdict, so the intent behind this row
90
+ * is void. The row is dead-lettered rather than retried — no amount of
91
+ * retrying can make a cancelled delivery correct — and the skip is logged so
92
+ * the operator can see what did NOT happen.
93
+ */
94
+ private stopCancelledWork;
84
95
  /** Record at most one deferral per outbox row per 60s (cold-start poll guard). */
85
96
  private recordDeferred;
86
97
  /** Pull correlation fields out of payload.context without trusting its shape. */