@animastor/orchestration 0.1.7 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +156 -0
- package/package.json +3 -2
- package/src/index.js +1 -0
- package/src/orchestration/orchestrator.js +18 -8
- package/src/orchestration/scene-callbacks.js +55 -3
- package/src/orchestration/scene-orchestrator.js +91 -14
- package/src/runtime/active-scenes-index.js +18 -2
- package/src/runtime/command-gate.js +270 -0
- package/src/runtime/dispatch-engine.js +322 -6
- package/src/runtime/dispatch-outcomes.js +124 -0
- package/src/runtime/hub-cancel-port.js +6 -0
- package/src/runtime/persistence-port.js +6 -0
- package/src/runtime/progress-events-port.js +6 -0
- package/src/runtime/reconciliation-engine.js +206 -172
- package/src/runtime/runtime-scheduler.js +88 -61
- package/src/runtime/scene-window.js +51 -2
- package/test/capacity-deferral.test.js +265 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.3.0 (2026-10-31) — генерация только по явной команде
|
|
4
|
+
|
|
5
|
+
Правило владельца: GPU-генерация запускается только кнопками «аудио» /
|
|
6
|
+
«картинки» / «всё»; никаких автозапусков следующих стадий и фоновых
|
|
7
|
+
запусков; задачи только на включённые слои. Основание — разбор
|
|
8
|
+
`reports/2026-10-31/auto-image-task-ba-incident.md` (§6, §8). Описание:
|
|
9
|
+
animastor-docs `architecture/COMMAND_ONLY_GENERATION.md`.
|
|
10
|
+
|
|
11
|
+
**Инвариант:** сцена×стадия попадает в active index / dispatch только при
|
|
12
|
+
активной задаче реестра (создана явной командой, стадия = type задачи), при
|
|
13
|
+
включённом слое и без отмены (cancel flag / tombstone).
|
|
14
|
+
|
|
15
|
+
### Added
|
|
16
|
+
|
|
17
|
+
- `runtime/command-gate.js` — единственная точка вычисления инварианта
|
|
18
|
+
(`getAuthorizedStages`, `isStageAuthorized`), force-набор сцена×стадия,
|
|
19
|
+
`rollbackGeneratingToDirty`, `blockStageNoWorker`. Экспорт
|
|
20
|
+
`runtime.commandGate`.
|
|
21
|
+
- ProgressEventsPort: операция `blockSceneStage` (хост помечает команду
|
|
22
|
+
blocked в реестре и PG).
|
|
23
|
+
|
|
24
|
+
### Changed (поведение)
|
|
25
|
+
|
|
26
|
+
- `active-scenes-index.addActiveScene` — запись в active index только при
|
|
27
|
+
авторизованной стадии; иначе `{ added:false, denied:true }`.
|
|
28
|
+
- `runtime-scheduler.shouldScheduleAssets` — стадии только по гейту; убраны
|
|
29
|
+
fallback на layer-config без задачи и `isInFlight`-override.
|
|
30
|
+
- `runtime-scheduler.tick/attemptDispatch` — force только для сцен×стадий
|
|
31
|
+
команды (ключ `animastor:force-dispatch-stages:<book>`, вместо книжного
|
|
32
|
+
`animastor:force-dispatch:<book>`); `no_worker_available` → стадия blocked
|
|
33
|
+
(GENERATING → DIRTY, задача blocked), без автоповтора.
|
|
34
|
+
- `markVersionStaleDirty` — только помечает stale, в работу не ставит.
|
|
35
|
+
- `orchestrator.resetScenes` — пишет force-набор вместо книжного флага.
|
|
36
|
+
- `scene-orchestrator.dispatchStage` — NEW-сцена переводит в PENDING только
|
|
37
|
+
диспатчимую стадию; при `no_worker_available` откат в DIRTY, не PENDING.
|
|
38
|
+
- `reconciliation-engine`:
|
|
39
|
+
- удалены `PROGRESS_TO_IMAGE` / `PROGRESS_TO_VIDEO`; partial build — только
|
|
40
|
+
диагностика;
|
|
41
|
+
- `REGENERATE_MISSING_ASSET`, `MOVE_TO_PENDING`, `RELEASE_STALE_LEASE`,
|
|
42
|
+
перезапуск аудио в `RECOVER_ORPHAN_ASSETS` — только под активной командой
|
|
43
|
+
стадии, иначе диагностика; осиротевший GENERATING без команды → DIRTY;
|
|
44
|
+
- C5 вызывает `deps.interruptIncompleteSessions` (возобновления нет);
|
|
45
|
+
- C8 `rebuildWorkList` пересобирает только факты asset-state, PENDING и
|
|
46
|
+
active index не пишет.
|
|
47
|
+
- `scene-window.startScene` — без авторизованной стадии сцену не стартует;
|
|
48
|
+
`trySlideWindowOnComplete` — за флагом хоста `windowAutoNextEnabled()`
|
|
49
|
+
(по умолчанию выкл.).
|
|
50
|
+
- `dispatch-engine.repairOrphanGeneratingStates` — опция `target: 'dirty'`.
|
|
51
|
+
|
|
52
|
+
## 0.2.0 (2026-10-27)
|
|
53
|
+
|
|
54
|
+
Version-line split for `stage143a-foundation-phase1`: the branch line moves
|
|
55
|
+
`0.1.6 → 0.2.0` so it can never collide with, or be shadowed by, the hotfix
|
|
56
|
+
line. **No functional change** — the content is the `0.1.6` code below (D1 +
|
|
57
|
+
A1 already in this branch).
|
|
58
|
+
|
|
59
|
+
**The hotfix line is separate:** `0.1.7` (branch `hotfix/d1-d2-a1`) is cut
|
|
60
|
+
from `0.1.4`, not from this line, and carries only D1 + the A1 version-gate
|
|
61
|
+
fallback (D2 and the host side of A1 live in `backend`). `^0.1.x` therefore
|
|
62
|
+
resolves within the `0.1.x` line, while this branch's backend depends on
|
|
63
|
+
`^0.2.0`.
|
|
64
|
+
|
|
65
|
+
## 0.1.7 (2026-10-27) — hotfix line (содержимое входит в 0.2.0)
|
|
66
|
+
|
|
67
|
+
Hotfix line (branch `hotfix/d1-d2-a1`, base `78e8f1ae` = `0.1.4`): only the
|
|
68
|
+
D1 and A1-gate fixes are carried in this package. It is a **separate line from
|
|
69
|
+
the `0.1.6` entry of the `stage143a-foundation-phase1` branch** — this package
|
|
70
|
+
is `0.1.4` + the two fixes, not a descendant of `0.1.5`/`0.1.6`.
|
|
71
|
+
|
|
72
|
+
**In this package:** D1, A1-gate. **In `backend` (not here):** D2 and the host
|
|
73
|
+
side of A1 (`scene-assets-repo.getAssetNullBuild` + runtime adapter + port op).
|
|
74
|
+
|
|
75
|
+
### Fixed
|
|
76
|
+
|
|
77
|
+
- `orchestration/orchestrator.js` (D1, stage 146) — the
|
|
78
|
+
`require('@animastor/generation').artifactNaming` binding lived INSIDE
|
|
79
|
+
`beginStage()`, but `resetScenes()` also reads it in the stale-PNG/preview
|
|
80
|
+
pre-delete branch (step 6). Any regeneration that passed `cleanPngUnitIds`
|
|
81
|
+
therefore threw `ReferenceError: artifactNaming is not defined`, and
|
|
82
|
+
`/regenerate` returned HTTP 500. The binding now sits at module scope, next
|
|
83
|
+
to the other module-level requires; `beginStage()` keeps its own
|
|
84
|
+
`dispatchEngine` lazy require. Behavior is otherwise unchanged.
|
|
85
|
+
- `orchestration/orchestrator.js` (`completeStage` version gate, A1 /
|
|
86
|
+
stage 146 §5.4) — when `sceneAssets.getAsset(..., buildId)` found no row for
|
|
87
|
+
THIS build, the gate went fail-closed (DIRTY) even though the key's only row
|
|
88
|
+
was a legacy `build_id IS NULL` row: `getAsset` filters `build_id = $5`, and
|
|
89
|
+
`markReady` — the only writer able to stamp `build_id`
|
|
90
|
+
(`buildIdFromArtifactPath`) — runs BEHIND the gate. The scene then
|
|
91
|
+
re-dispatched every 5 s tick (`cache_hit` → callback → `stale` → DIRTY)
|
|
92
|
+
forever. The gate now performs exactly ONE additional read of the SAME key
|
|
93
|
+
with `build_id IS NULL` (never another build's row, never "latest any row")
|
|
94
|
+
through the PersistencePort op `sceneAssets.getAssetNullBuild`, applies the
|
|
95
|
+
SAME version rules (`scene_content_version`, plus
|
|
96
|
+
`scene_audio_config_version` for audio), and lets `markReady` stamp
|
|
97
|
+
`build_id` as before. A NULL row whose version is below the scene version
|
|
98
|
+
still fails the gate (DIRTY), so the loop converges after the next real
|
|
99
|
+
dispatch instead of spinning. PG errors stay fail-closed.
|
|
100
|
+
|
|
101
|
+
### Shipped by backend, not by this package
|
|
102
|
+
|
|
103
|
+
- D2 — `/api/v1/debug/runtime/*` middleware now calls `next()`
|
|
104
|
+
(`backend/src/routes/debug-routes.cjs`).
|
|
105
|
+
- A1 host side — `sceneAssets.getAssetNullBuild` implementation and delegation
|
|
106
|
+
(`backend/src/storage/postgres/repositories/scene-assets-repo.js`,
|
|
107
|
+
`backend/src/storage/runtime-persistence-adapter.js`) plus the op
|
|
108
|
+
declaration in `runtime/persistence-port.js`.
|
|
109
|
+
|
|
110
|
+
## 0.1.6 (2026-10-26)
|
|
111
|
+
|
|
112
|
+
Stage 146 — D1 fix: `artifactNaming` hoisted to module scope.
|
|
113
|
+
Stage 147 — A1 fix: version-gate sees the legacy `build_id IS NULL` row.
|
|
114
|
+
|
|
115
|
+
### Fixed
|
|
116
|
+
|
|
117
|
+
- `orchestration/orchestrator.js` — the `require('@animastor/generation')
|
|
118
|
+
.artifactNaming` binding lived INSIDE `beginStage()`, but
|
|
119
|
+
`resetScenes()` also reads it in the stale-PNG/preview pre-delete branch
|
|
120
|
+
(step 6). Any regeneration that passed `cleanPngUnitIds` therefore threw
|
|
121
|
+
`ReferenceError: artifactNaming is not defined`, and `/regenerate`
|
|
122
|
+
returned HTTP 500. The binding now sits at module scope, next to the
|
|
123
|
+
other module-level requires; `beginStage()` keeps its own
|
|
124
|
+
`dispatchEngine` lazy require. Behavior is otherwise unchanged.
|
|
125
|
+
- `orchestration/orchestrator.js` (`completeStage` version gate, A1 /
|
|
126
|
+
stage146 §5.4) — when `sceneAssets.getAsset(..., buildId)` found no row
|
|
127
|
+
for THIS build, the gate went fail-closed (DIRTY) even though the key's
|
|
128
|
+
only row was a legacy `build_id IS NULL` row: `getAsset` filters
|
|
129
|
+
`build_id = $5`, and `markReady` — the only writer able to stamp
|
|
130
|
+
`build_id` (`buildIdFromArtifactPath`) — runs BEHIND the gate. The scene
|
|
131
|
+
then re-dispatched every 5 s tick (`cache_hit` → callback → `stale` →
|
|
132
|
+
DIRTY) forever. The gate now performs exactly ONE additional read of the
|
|
133
|
+
SAME key with `build_id IS NULL` (never another build's row, never
|
|
134
|
+
"latest any row") through the same PersistencePort op
|
|
135
|
+
`sceneAssets.getAssetNullBuild`, applies the SAME version rules
|
|
136
|
+
(`scene_content_version`, plus `scene_audio_config_version` for audio),
|
|
137
|
+
and lets `markReady` stamp `build_id` as before. A NULL row whose
|
|
138
|
+
version is below the scene version still fails the gate (DIRTY) — the
|
|
139
|
+
correct version arrives from `stampSceneVersionsForDispatch` (dispatch
|
|
140
|
+
step 5.5 stamps ALL rows of the key, incl. `build_id IS NULL`, before
|
|
141
|
+
every generation/cache_hit), so the loop converges after the next real
|
|
142
|
+
dispatch instead of spinning. PG errors stay fail-closed.
|
|
143
|
+
|
|
144
|
+
### Changed (host adapter, A1)
|
|
145
|
+
|
|
146
|
+
- `backend/src/storage/postgres/repositories/scene-assets-repo.js` — new
|
|
147
|
+
`getAssetNullBuild` (same key, `build_id IS NULL`, latest NULL row).
|
|
148
|
+
- `backend/src/storage/runtime-persistence-adapter.js` — delegates the new
|
|
149
|
+
op; `runtime/persistence-port.js` — declares `sceneAssets.getAssetNullBuild`
|
|
150
|
+
in `OPS` (no SQL crosses the port).
|
|
151
|
+
|
|
152
|
+
## 0.1.5 (2026-10-26)
|
|
153
|
+
|
|
154
|
+
Stage 143a — versioned-pointer readers + atomic publication. See the
|
|
155
|
+
player 0.1.2/0.1.3 entries (same stage) for the flag-gated IU-media
|
|
156
|
+
contour this package complements.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@animastor/orchestration",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.3.0",
|
|
4
4
|
"description": "Animastor Orchestration — the generation orchestration/runtime contour of the Animastor backend, physically extracted from the host (§32.30): the unified generation lifecycle owner (orchestrator facade, stage executor, scene callbacks/restoration), the dispatch engine and its lease/quota/retry managers, the reconciliation engine and counter reconciliation, the scene window/scheduler/worker-health, the runtime-result emitter/consumer, and the eight O-2..O-10 port contracts (Persistence, SceneData, PlaceholderAudio, ProgressEvents, AudioFsm, VideoFsm, HubCancel, LayerConfig). The package reaches the host (runtime config slices, Redis-domain state layer, media facades, the S-5 seam registry, the artifact root, the PW-2 workspace resolver) ONLY through the composition-root host-bindings seam — no backend/src require, no storage/book/routes/services, no Redis, no PG, no Express inside.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"animastor",
|
|
@@ -18,7 +18,8 @@
|
|
|
18
18
|
"src/",
|
|
19
19
|
"test/",
|
|
20
20
|
"README.md",
|
|
21
|
-
"LICENSE"
|
|
21
|
+
"LICENSE",
|
|
22
|
+
"CHANGELOG.md"
|
|
22
23
|
],
|
|
23
24
|
"scripts": {
|
|
24
25
|
"test": "mocha --exit --require test/test-bindings.js test/*.test.js"
|
package/src/index.js
CHANGED
|
@@ -75,6 +75,7 @@ const EAGER_EXPORTS = {
|
|
|
75
75
|
const RUNTIME_NS = {
|
|
76
76
|
scheduler: './runtime/runtime-scheduler',
|
|
77
77
|
activeScenes: './runtime/active-scenes-index',
|
|
78
|
+
commandGate: './runtime/command-gate',
|
|
78
79
|
reconciliation: './runtime/reconciliation-engine',
|
|
79
80
|
dispatch: './runtime/dispatch-engine',
|
|
80
81
|
leaseManager: './runtime/lease-manager',
|
|
@@ -465,7 +465,7 @@ async function reconcile(redis, bookId, chapterId, sceneId) {
|
|
|
465
465
|
// options.readdToActiveIndex — вернуть сцены планировщику после reset (default true)
|
|
466
466
|
//
|
|
467
467
|
// Что делает:
|
|
468
|
-
// 1. force
|
|
468
|
+
// 1. force-набор: только сцены×стадии этой регенерации
|
|
469
469
|
// 2. событие SCENE_RESET в journal
|
|
470
470
|
// 3. удаление из active index
|
|
471
471
|
// 4. снятие dispatch leases только для сбрасываемых стадий
|
|
@@ -497,11 +497,18 @@ async function resetScenes(redis, bookId, buildId, scenes, layerCfg, options = {
|
|
|
497
497
|
const scopeDisplay = `${scope}${chapterId ? '/' + chapterId : ''}${sceneId ? '/' + sceneId : ''}`;
|
|
498
498
|
log(`[RESET-SCENES] ${bookId}: ${scenes.length} scenes, scope=${scopeDisplay}`);
|
|
499
499
|
|
|
500
|
-
// 1. Force
|
|
501
|
-
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
);
|
|
500
|
+
// 1. Force set (T1: TTL из TIMEOUTS). Command-only generation: force
|
|
501
|
+
// covers EXACTLY the scene×stage pairs reset here (dirty_layers ∩ enabled
|
|
502
|
+
// layers of the request) — never the whole book (incident 2026-10-30:
|
|
503
|
+
// one-scene audio regenerate force-dispatched an unrelated image stage).
|
|
504
|
+
const commandGate = require('../runtime/command-gate');
|
|
505
|
+
await commandGate.addForceStages(redis, bookId, scenes.map(ds => ({
|
|
506
|
+
chapter_id: ds.chapter_id,
|
|
507
|
+
scene_id: ds.scene_id,
|
|
508
|
+
stages: (ds.dirty_layers || mediaRegistry.listMediaTypes()).filter(stage =>
|
|
509
|
+
layerCfg?.[`${stage}_enabled`] !== false
|
|
510
|
+
),
|
|
511
|
+
})), runtimeConfig.TIMEOUTS.FORCE_DISPATCH_TTL_S);
|
|
505
512
|
|
|
506
513
|
// 2. Event journal — start
|
|
507
514
|
await journal.appendSceneEvent(redis, bookId, chapterId || bookId, sceneId || 'all',
|
|
@@ -601,10 +608,13 @@ async function resetScenes(redis, bookId, buildId, scenes, layerCfg, options = {
|
|
|
601
608
|
// Selective generation can defer this until its task registry is ready.
|
|
602
609
|
// Гарантированно выполняется даже при ошибке markDirty (шаг 8).
|
|
603
610
|
if (readdToActiveIndex) {
|
|
611
|
+
// Command-gated write: only scenes with an authorized stage enter.
|
|
612
|
+
let readded = 0;
|
|
604
613
|
for (const ds of scenes) {
|
|
605
|
-
await scheduler.addSceneToActiveIndex(redis, bookId, ds.chapter_id, ds.scene_id);
|
|
614
|
+
const r = await scheduler.addSceneToActiveIndex(redis, bookId, ds.chapter_id, ds.scene_id);
|
|
615
|
+
if (r && !r.denied) readded++;
|
|
606
616
|
}
|
|
607
|
-
log(`[RESET-SCENES] ${bookId}: ${scenes.length} scenes re-added to active index`);
|
|
617
|
+
log(`[RESET-SCENES] ${bookId}: ${readded}/${scenes.length} scenes re-added to active index (command-gated)`);
|
|
608
618
|
} else {
|
|
609
619
|
log(`[RESET-SCENES] ${bookId}: activation deferred for ${scenes.length} scenes`);
|
|
610
620
|
}
|
|
@@ -277,9 +277,17 @@ async function handleImageCompleted(redis, bookId, chapterId, sceneId, buildId)
|
|
|
277
277
|
// T1: возвращает { ok, retryable, reason, artifact }
|
|
278
278
|
log(`IMAGE_CALLBACK: ${bookId}/${chapterId}/${sceneId}`);
|
|
279
279
|
|
|
280
|
-
|
|
280
|
+
// Stage 143a: legacy canonical scan first (unchanged behavior, also the
|
|
281
|
+
// path every host stub knows), then — flag on and scan empty — the
|
|
282
|
+
// published pointer of this scene build as the completion proof.
|
|
283
|
+
let sceneImage = image.resolveCanonicalSceneImage(
|
|
281
284
|
artifactRoot(), buildId, bookId, chapterId, sceneId
|
|
282
285
|
);
|
|
286
|
+
if (!sceneImage && typeof image.resolveVersionedSceneProof === 'function') {
|
|
287
|
+
sceneImage = await image.resolveVersionedSceneProof(
|
|
288
|
+
artifactRoot(), buildId, bookId, chapterId, sceneId
|
|
289
|
+
);
|
|
290
|
+
}
|
|
283
291
|
|
|
284
292
|
// Per-asset check
|
|
285
293
|
const assetStates = await state.getAssetStates(redis, bookId, chapterId, sceneId);
|
|
@@ -334,7 +342,28 @@ async function handleImageCompleted(redis, bookId, chapterId, sceneId, buildId)
|
|
|
334
342
|
const stillPending = [];
|
|
335
343
|
for (const uid of dirtyIds) {
|
|
336
344
|
const pngPath = path.join(buildDir, artifactNaming.sceneImageName(bookId, chapterId, sceneId, uid));
|
|
337
|
-
if (
|
|
345
|
+
if (fs.existsSync(pngPath)) continue;
|
|
346
|
+
// ── Stage 143a (phase 2a · N1) ─────────────────────
|
|
347
|
+
// A versioned-only build never writes the canonical file, so
|
|
348
|
+
// the published pointer is the completion proof for this
|
|
349
|
+
// unit. Without it the dirty marker is never cleared and the
|
|
350
|
+
// unit is force-regenerated on EVERY reconcile pass (infinite
|
|
351
|
+
// regeneration loop with the flag on).
|
|
352
|
+
// Flag OFF ⇒ the facade answers null with zero PG calls → the
|
|
353
|
+
// legacy check above decides, exactly as before.
|
|
354
|
+
// A pointer-store ERROR keeps the unit pending (never cleared
|
|
355
|
+
// on an unknown verdict) and is logged.
|
|
356
|
+
let provenByPointer = false;
|
|
357
|
+
try {
|
|
358
|
+
if (image && typeof image.resolveIuImagePath === 'function') {
|
|
359
|
+
const published = await image.resolveIuImagePath(
|
|
360
|
+
artifactRoot(), buildId, bookId, chapterId, sceneId, uid);
|
|
361
|
+
provenByPointer = !!(published && published.status === 'ok');
|
|
362
|
+
}
|
|
363
|
+
} catch (pgErr) {
|
|
364
|
+
warn(`[DIRTY-UNITS] pointer proof unavailable for ${uid}: ${pgErr.message} — keeping it pending`);
|
|
365
|
+
}
|
|
366
|
+
if (!provenByPointer) {
|
|
338
367
|
stillPending.push(uid);
|
|
339
368
|
}
|
|
340
369
|
}
|
|
@@ -343,8 +372,31 @@ async function handleImageCompleted(redis, bookId, chapterId, sceneId, buildId)
|
|
|
343
372
|
log(`[DIRTY-UNITS-CLEARED] ${bookId}/${chapterId}/${sceneId}: all ${dirtyIds.length} dirty unit(s) completed, cleared`);
|
|
344
373
|
try {
|
|
345
374
|
const iuPrefix = artifactNaming.iuImagePrefix(bookId, chapterId, sceneId);
|
|
375
|
+
const scenePrefix = artifactNaming.scenePrefix(bookId, chapterId, sceneId);
|
|
346
376
|
const allFiles = fs.existsSync(buildDir) ? fs.readdirSync(buildDir) : [];
|
|
347
|
-
|
|
377
|
+
// ── Stage 143a (phase 2a · N2) ─────────────────
|
|
378
|
+
// Ready units = canonical file names ∪ published pointer
|
|
379
|
+
// ids (versioned files are invisible to this scan). The
|
|
380
|
+
// counter is a DISPLAY source (consumers clamp it), so a
|
|
381
|
+
// pointer-store error here only logs and leaves the
|
|
382
|
+
// legacy count — never a readiness decision.
|
|
383
|
+
const readyIds = new Set();
|
|
384
|
+
for (const f of allFiles) {
|
|
385
|
+
if (f.startsWith(iuPrefix) && f.endsWith('.png')) {
|
|
386
|
+
readyIds.add(f.startsWith(`${scenePrefix}_`)
|
|
387
|
+
? f.slice(scenePrefix.length + 1, -4)
|
|
388
|
+
: f);
|
|
389
|
+
}
|
|
390
|
+
}
|
|
391
|
+
if (image && typeof image.listPublishedIuIdsForScene === 'function') {
|
|
392
|
+
try {
|
|
393
|
+
const publishedIds = await image.listPublishedIuIdsForScene(buildId, bookId, chapterId, sceneId);
|
|
394
|
+
for (const id of publishedIds) readyIds.add(String(id));
|
|
395
|
+
} catch (pgErr) {
|
|
396
|
+
warn(`[IU-COUNTER-RESET] published ids unavailable for ${bookId}/${chapterId}/${sceneId}: ${pgErr.message} — counting canonical files only`);
|
|
397
|
+
}
|
|
398
|
+
}
|
|
399
|
+
const iuFileCount = readyIds.size;
|
|
348
400
|
if (iuFileCount > 0) {
|
|
349
401
|
const progKey = `animastor:iu-progress:${bookId}:${chapterId}:${sceneId}:image`;
|
|
350
402
|
await redis.set(progKey, String(iuFileCount), 'EX', 14400);
|
|
@@ -13,10 +13,15 @@ const chunks = lazyHostBinding('media.audio.chunks');
|
|
|
13
13
|
// orchestrator never requires gpu-dispatcher or the workflow connector.
|
|
14
14
|
const provider = require('@animastor/generation').comfyuiProvider;
|
|
15
15
|
const runtimeScheduler = require('../runtime/runtime-scheduler');
|
|
16
|
+
// Stage 118 (M3): the shared outcome taxonomy — the dispatch engine (runtime
|
|
17
|
+
// tier) and the stage executors (orchestration tier) must classify a
|
|
18
|
+
// "nothing was enqueued" outcome IDENTICALLY.
|
|
19
|
+
const { classifyNotSentReasons } = require('../runtime/dispatch-outcomes');
|
|
16
20
|
// O-2: persistence arrives ONLY through the PersistencePort (host adapter:
|
|
17
21
|
// storage/runtime-persistence-adapter) — the scene-assets repository left
|
|
18
22
|
// this file. SQL lives host-side.
|
|
19
23
|
const persist = require('../runtime/persistence-port').persist;
|
|
24
|
+
const isPersistencePortWired = require('../runtime/persistence-port').isPersistencePortWired;
|
|
20
25
|
// O-3: scene content arrives ONLY through the SceneDataPort (host adapter:
|
|
21
26
|
// storage/scene-data-adapter) — the Book Model facade (../book →
|
|
22
27
|
// @animastor/vbook-runtime) left this file. Aliased `sceneDataPort` because
|
|
@@ -63,19 +68,21 @@ const EXECUTORS = {
|
|
|
63
68
|
video: executeVideoDispatch,
|
|
64
69
|
};
|
|
65
70
|
|
|
66
|
-
async function startScene(redis, scene, loadedBook, buildId) {
|
|
71
|
+
async function startScene(redis, scene, loadedBook, buildId, stage = 'audio') {
|
|
67
72
|
const bookId = scene.book_id;
|
|
68
73
|
const chapterId = scene.chapter_id;
|
|
69
74
|
const sceneId = scene.scene_id;
|
|
70
75
|
|
|
71
|
-
log(`START SCENE: ${bookId}/${chapterId}/${sceneId}`);
|
|
76
|
+
log(`START SCENE: ${bookId}/${chapterId}/${sceneId} (stage=${stage})`);
|
|
72
77
|
|
|
73
|
-
await logEvent(redis, scene, 'SCENE_STARTED',
|
|
78
|
+
await logEvent(redis, scene, 'SCENE_STARTED', `${stage}_pending`, {
|
|
74
79
|
buildId, bookId, chapterId, sceneId
|
|
75
80
|
});
|
|
76
81
|
|
|
77
|
-
// T7: через фасад — единый владелец PENDING
|
|
78
|
-
|
|
82
|
+
// T7: через фасад — единый владелец PENDING. Command-only generation:
|
|
83
|
+
// only the stage being dispatched becomes PENDING — a NEW scene asked for
|
|
84
|
+
// `image` must not get an unrequested `audio` PENDING.
|
|
85
|
+
await setScenePending(redis, bookId, chapterId, sceneId, stage, buildId);
|
|
79
86
|
await runtimeScheduler.addSceneToActiveIndex(redis, bookId, chapterId, sceneId);
|
|
80
87
|
log(`ADDED TO ACTIVE: ${bookId}/${chapterId}/${sceneId}`);
|
|
81
88
|
|
|
@@ -300,8 +307,22 @@ async function executeAudioDispatch(redis, scene, loadedBook, buildId, dispatchI
|
|
|
300
307
|
const completion = await completeStage(redis, bookId, chapterId, sceneId, 'audio', buildId, dispatchId);
|
|
301
308
|
return { dispatched: false, jobs: 0, completed: completion.completed, reason: 'no_segments' };
|
|
302
309
|
} else {
|
|
303
|
-
|
|
304
|
-
|
|
310
|
+
// Stage 116 (D3) / Stage 118 (B3, M3): nothing was enqueued — carry
|
|
311
|
+
// the precise reason (no live worker / everything already queued /
|
|
312
|
+
// ambiguous POST) so the dispatch engine can apply a BOUNDED deferral
|
|
313
|
+
// instead of a fatal cancel.
|
|
314
|
+
//
|
|
315
|
+
// Stage 118 (B3): the merged (pure-dialogue) path and the per-segment
|
|
316
|
+
// path must produce the SAME classification — the merged branch used
|
|
317
|
+
// to leave the reason empty, so a pure-dialogue scene kept the
|
|
318
|
+
// 5-second retry loop. `deferReason` is computed by the host (audio
|
|
319
|
+
// generation) for EVERY path; the fallback below keeps the historic
|
|
320
|
+
// 'no_jobs_generated' semantics when the host could not classify it.
|
|
321
|
+
const deferReason = (result && result.reason === 'no_jobs_accepted' && result.deferReason)
|
|
322
|
+
? result.deferReason
|
|
323
|
+
: 'no_jobs_generated';
|
|
324
|
+
warn(`AUDIO_DISPATCH: generateSceneAudio did not send jobs for ${bookId}/${chapterId}/${sceneId} (${deferReason})`);
|
|
325
|
+
return { dispatched: false, jobs: 0, reason: deferReason, retryable: true };
|
|
305
326
|
}
|
|
306
327
|
}
|
|
307
328
|
|
|
@@ -352,8 +373,11 @@ async function executeImageDispatch(redis, scene, loadedBook, buildId, dispatchI
|
|
|
352
373
|
|
|
353
374
|
// T3: generateSceneIUImages возвращает количество отправленных jobs
|
|
354
375
|
let jobsSent = 0;
|
|
376
|
+
// Stage 116 (D3): the result is needed AFTER the try/catch (the deferral
|
|
377
|
+
// reason) — it must be visible outside the block.
|
|
378
|
+
let genResult = null;
|
|
355
379
|
try {
|
|
356
|
-
|
|
380
|
+
genResult = await image.generateSceneIUImages(
|
|
357
381
|
redis,
|
|
358
382
|
sceneData,
|
|
359
383
|
bookData,
|
|
@@ -377,6 +401,16 @@ async function executeImageDispatch(redis, scene, loadedBook, buildId, dispatchI
|
|
|
377
401
|
return { dispatched: true, jobs: jobsSent, reason: null };
|
|
378
402
|
}
|
|
379
403
|
|
|
404
|
+
// Stage 116 (D3) / Stage 118 (M3): the executor sent NOTHING. That is
|
|
405
|
+
// not a lost task and not a fatal cancellation — it is a bounded
|
|
406
|
+
// deferral. The precise reason (no live worker / everything already
|
|
407
|
+
// queued / ambiguous POST) is carried through so the deferral can be
|
|
408
|
+
// classified by the shared taxonomy (runtime/dispatch-outcomes).
|
|
409
|
+
if (genResult && genResult.deferReason) {
|
|
410
|
+
warn(`IMAGE_DISPATCH: ${bookId}/${chapterId}/${sceneId} — nothing enqueued (${genResult.deferReason}); deferring`);
|
|
411
|
+
return { dispatched: false, jobs: 0, reason: genResult.deferReason, retryable: true };
|
|
412
|
+
}
|
|
413
|
+
|
|
380
414
|
log(`IMAGE_DISPATCH: ${bookId}/${chapterId}/${sceneId} — no jobs sent (cache hit or empty units)`);
|
|
381
415
|
return { dispatched: false, jobs: 0, reason: 'no_jobs_sent' };
|
|
382
416
|
}
|
|
@@ -551,15 +585,41 @@ async function executeVideoDispatch(redis, scene, loadedBook, buildId, dispatchI
|
|
|
551
585
|
await videoOrch.setWaitingChunks(redis, bookId, chapterId, sceneId);
|
|
552
586
|
|
|
553
587
|
let sentCount = 0;
|
|
588
|
+
// Stage 116 (D3) / Stage 118 (M3): collect WHY a job was not sent — the
|
|
589
|
+
// dominant reason becomes the deferral reason (bounded backoff) instead
|
|
590
|
+
// of the generic 'send_failed'.
|
|
591
|
+
const notSentReasons = new Map();
|
|
554
592
|
for (const jobSpec of jobsToSend) {
|
|
555
593
|
try {
|
|
556
|
-
|
|
594
|
+
let sendResult = await provider.generate(jobSpec);
|
|
595
|
+
// Stage 122 (N2): the group file was probed absent above (a cached
|
|
596
|
+
// group never reaches this loop), so an `already_completed` verdict
|
|
597
|
+
// is by construction an ORPHANED completion marker. Free it and
|
|
598
|
+
// retry ONCE so the video regenerates instead of looping on a
|
|
599
|
+
// 900 s deferral with the scene stuck in the active index (audit
|
|
600
|
+
// 121 · N2). The release rides the persistence port — the
|
|
601
|
+
// orchestration tier cannot require the backend helper directly.
|
|
602
|
+
// Skipped when the port is unwired (harnesses that don't exercise
|
|
603
|
+
// orphan cleanup); production always wires it in backend.cjs.
|
|
604
|
+
if (isPersistencePortWired()) {
|
|
605
|
+
let released = false;
|
|
606
|
+
try {
|
|
607
|
+
released = await persist('idem.releaseTaskDoneIfOrphan')(redis, sendResult);
|
|
608
|
+
} catch (_) { released = false; }
|
|
609
|
+
if (released) {
|
|
610
|
+
log(`VIDEO_DISPATCH: orphaned completion marker released for ${jobSpec.job_id} — regenerating`);
|
|
611
|
+
sendResult = await provider.generate(jobSpec);
|
|
612
|
+
}
|
|
613
|
+
}
|
|
557
614
|
if (sendResult.sent) {
|
|
558
615
|
sentCount++;
|
|
559
616
|
} else {
|
|
560
|
-
|
|
617
|
+
const reason = sendResult.reason || 'send_failed';
|
|
618
|
+
notSentReasons.set(reason, (notSentReasons.get(reason) || 0) + 1);
|
|
619
|
+
warn(`VIDEO_DISPATCH: Hub rejected job ${jobSpec.job_id}: ${sendResult.error || reason}`);
|
|
561
620
|
}
|
|
562
621
|
} catch (sendErr) {
|
|
622
|
+
notSentReasons.set('send_failed', (notSentReasons.get('send_failed') || 0) + 1);
|
|
563
623
|
warn(`VIDEO_DISPATCH: provider generate failed for job ${jobSpec.job_id}: ${sendErr.message}`);
|
|
564
624
|
}
|
|
565
625
|
}
|
|
@@ -569,8 +629,13 @@ async function executeVideoDispatch(redis, scene, loadedBook, buildId, dispatchI
|
|
|
569
629
|
return { dispatched: true, jobs: sentCount, reason: null };
|
|
570
630
|
}
|
|
571
631
|
|
|
572
|
-
|
|
573
|
-
|
|
632
|
+
// Stage 116 (D3) / Stage 118 (M3): the shared taxonomy decides whether
|
|
633
|
+
// the outcome is a bounded deferral. EVERY reason must be deferrable —
|
|
634
|
+
// a single real failure (send_failed) keeps the historic 'send_failed'
|
|
635
|
+
// semantics (an immediate retry on the next tick).
|
|
636
|
+
const deferReason = classifyNotSentReasons(notSentReasons) || 'send_failed';
|
|
637
|
+
warn(`VIDEO_DISPATCH: ${bookId}/${chapterId}/${sceneId}: nothing enqueued (${deferReason})`);
|
|
638
|
+
return { dispatched: false, jobs: 0, reason: deferReason, retryable: true };
|
|
574
639
|
}
|
|
575
640
|
|
|
576
641
|
// T4: dispatchStage принимает dispatchId (из dispatch-engine) и передаёт executors
|
|
@@ -589,7 +654,7 @@ async function dispatchStage(redis, scene, loadedBook, buildId, overrideStage, d
|
|
|
589
654
|
);
|
|
590
655
|
if (isNewScene) {
|
|
591
656
|
log(`NEW SCENE: ${bookId}/${chapterId}/${sceneId} (asset-state)`);
|
|
592
|
-
await startScene(redis, scene, loadedBook, buildId);
|
|
657
|
+
await startScene(redis, scene, loadedBook, buildId, overrideStage || 'audio');
|
|
593
658
|
}
|
|
594
659
|
|
|
595
660
|
if (!overrideStage) {
|
|
@@ -610,7 +675,19 @@ async function dispatchStage(redis, scene, loadedBook, buildId, overrideStage, d
|
|
|
610
675
|
result = await executor(redis, scene, loadedBook, buildId, dispatchId);
|
|
611
676
|
|
|
612
677
|
result = result || { dispatched: false, reason: 'no_result', jobs: 0 };
|
|
613
|
-
if (!result.dispatched && !result.completed) {
|
|
678
|
+
if (!result.dispatched && !result.completed && result.reason === 'no_worker_available') {
|
|
679
|
+
// Command-only generation: no live worker of this type → the stage is
|
|
680
|
+
// BLOCKED, not re-queued. GENERATING → DIRTY only (never PENDING): the
|
|
681
|
+
// scheduler marks the command blocked and nothing retries it until a
|
|
682
|
+
// new command arrives (incident 2026-10-30: endless no_worker loop).
|
|
683
|
+
const rollback = await require('../runtime/command-gate').rollbackGeneratingToDirty(
|
|
684
|
+
redis, bookId, chapterId, sceneId, overrideStage, 'no_worker_available');
|
|
685
|
+
if (!rollback.changed && rollback.reason !== 'state_dirty' && rollback.reason !== 'state_ready') {
|
|
686
|
+
warn(`DISPATCH_ROLLBACK_FAILED: ${bookId}/${chapterId}/${sceneId} ${overrideStage}: ${rollback.reason} — blocked stage not released`);
|
|
687
|
+
result.rollbackFailed = true;
|
|
688
|
+
result.rollbackReason = rollback.reason;
|
|
689
|
+
}
|
|
690
|
+
} else if (!result.dispatched && !result.completed) {
|
|
614
691
|
// FSM-валидный rollback (GENERATING → DIRTY → PENDING). Прямой
|
|
615
692
|
// generating→pending запрещён FSM; раньше здесь вызывался setScenePending,
|
|
616
693
|
// rejection молча проглатывался — и сцена навсегда оставалась в GENERATING
|
|
@@ -33,12 +33,28 @@ async function getActiveCount(redis) {
|
|
|
33
33
|
}
|
|
34
34
|
|
|
35
35
|
/**
|
|
36
|
-
* Add scene to active index
|
|
36
|
+
* Add scene to active index — COMMAND-GATED (generation only by explicit
|
|
37
|
+
* command, see ./command-gate). The scene enters the index only when at
|
|
38
|
+
* least one of its stages is authorized: an ACTIVE task-registry task of that
|
|
39
|
+
* stage targets the scene, the layer is enabled and the book is not
|
|
40
|
+
* cancelled. Otherwise nothing is written and `{ added:false, denied:true }`
|
|
41
|
+
* is returned with the per-stage denial reasons.
|
|
42
|
+
*
|
|
43
|
+
* This is the ONE write path into the index (scheduler wrapper, regenerate,
|
|
44
|
+
* reconcile, scene-window, host services and the player all reach it), so
|
|
45
|
+
* no background path can queue work without a command.
|
|
37
46
|
*/
|
|
38
47
|
async function addActiveScene(redis, bookId, chapterId, sceneId) {
|
|
39
48
|
const sceneKey = `${bookId}:${chapterId}:${sceneId}`;
|
|
49
|
+
// Lazy: command-gate → scene-window → this module (acyclic at load time).
|
|
50
|
+
const auth = await require('./command-gate').getAuthorizedStages(redis, bookId, chapterId, sceneId);
|
|
51
|
+
if (auth.stages.size === 0) {
|
|
52
|
+
const reasons = require('./command-gate').describeDenied(auth.denied);
|
|
53
|
+
console.log(`[ACTIVE-INDEX] GATE_DENY ${sceneKey}: no command-authorized stage (${reasons}) — not added`);
|
|
54
|
+
return { added: false, denied: true, sceneKey, reasons: auth.denied };
|
|
55
|
+
}
|
|
40
56
|
const added = await redis.sadd(ACTIVE_SCENES_KEY, sceneKey);
|
|
41
|
-
return { added: added > 0, sceneKey };
|
|
57
|
+
return { added: added > 0, sceneKey, stages: [...auth.stages] };
|
|
42
58
|
}
|
|
43
59
|
|
|
44
60
|
/**
|