@animastor/orchestration 0.3.1 → 0.3.4

package/CHANGELOG.md

@@ -1,5 +1,86 @@
 # Changelog
 
+## 0.3.4 (2026-11-04) — 0.3.3 + «ready» аудио только при коммите GPU
+
+Одна публикация вместо двух: 0.3.3 (уборка хаба при `/regenerate`, см. ниже) **не
+публиковался** в npm отдельно и входит в 0.3.4 целиком. Основание:
+`reports/2026-11-04/audio-placeholder-ready.md` (сцена `sc-4bb85e02` — тишина-заглушка
+стала `ready` без GPU), интеграция: `reports/2026-11-04/integration-A-B-C.md`.
+Описание инварианта: animastor-docs `02-orchestration/AUDIO_READY_SEMANTICS.md`.
+
+### Changed (поведение)
+
+- `scene-orchestrator.executeAudioDispatch`: `AUDIO_ALREADY_DONE` пропускает работу только
+  если хост подтверждает коммит (`audio.isSceneAudioCommitted(...).ready === true`:
+  канонический файл без маркера заглушки **и** чанки сцены на диске). Иначе — лог
+  `AUDIO_ALREADY_DONE rejected … DONE without a committed audio chunk`, FSM-состояние
+  удаляется, стадия идёт обычным путём (dispatch на GPU). То же — в safety-net ветке
+  «stale `DONE`». Пробник недоступен или упал → считается «не закоммичено» (fail-closed:
+  лишняя генерация безопасна, пропуск — нет).
+- `scene-callbacks.handleAudioCompleted`: проверка после завершения — `isSceneAudioCommitted`
+  (legacy `isSceneAudioReady` только если хост не дал нового пробника); в
+  `replacePlaceholderWithRealAudio` передаётся `{ committed: true }` — хост без этого
+  доказательства не поднимает запись в `ready`.
+
+### Host contract (новое, обратно совместимо)
+
+- `media.audio.isSceneAudioCommitted(buildId, bookId, chapterId, sceneId, expectedCount|null)`
+  → `{ ready, reason, chunks, path }`. Backend ≥ интеграции A+B его даёт; на старом хосте
+  пакет откатывается к file-only проверке.
+- `placeholderAudio.replacePlaceholderWithRealAudio(…, realDuration, options)` — 7-й аргумент
+  `options.committed`.
+
+Инварианты 0.3.3 (sweep только своих dispatch-id, fail-closed book-wide sweep) не менялись;
+fence, command-gate и retry-политика — тоже.
+
+## 0.3.3 (2026-11-04, не публиковался — вошёл в 0.3.4) — /regenerate не сносит чужие задачи на хабе
+
+Основание: `reports/2026-11-04/gpu-generation-stall.md` §5.3, §9.1 — `/regenerate`
+одной сцены (audio) закончился `clearHubDispatches([], { book_id })` →
+`DELETE /queue/clear?book_id=`; хаб удалил running-запись image-задачи другой
+сцены (результат отклонён 409 и потерян) и queued-задачу (восстановлена
+stale-lease recovery через 7 мин). Отчёт исправления:
+`reports/2026-11-04/regenerate-sweep-fix.md`. Описание: animastor-docs
+`architecture/COMMAND_ONLY_GENERATION.md` §«Уборка хаба при /regenerate».
+
+### Changed (поведение)
+
+- `dispatch-engine.clearHubDispatches`: book-scoped форма (пустой список id +
+  `book_id`) — **fail-closed**: запрос `?book_id=` уходит только при явном
+  `allowBookSweep: true`; иначе ни одного запроса, warn
+  `book-wide GPU hub sweep … REFUSED`, результат `{requested:0, cleared:0,
+  failed:0, refused:1}`. Per-id форма не менялась.
+- `orchestrator.resetScenes` возвращает `dispatchIds` — dispatch-id,
+  отменённые этим reset'ом (только сбрасываемые сцена×стадия); вызывающий
+  повторно убирает на хабе только их.
+
+Fence, command-gate и retry-политика не меняются.
+
+## 0.3.2 (2026-11-02) — эпоха dispatch в audio-orch, без ложного FAILED
+
+Основание: `reports/2026-11-02/gpu-e2e-installer-025-logs.md` §6.1, §8 п.3, п.5 —
+состояние audio-orch отменённой попытки (11:09) пережило отмену, новый
+dispatch (12:25) унаследовал его `started_at`, и watchdog через 28 с после
+dispatch'а пометил здоровый прогон FAILED (`chunk_stall_timeout:4619s`), затем
+`[FAIL-STAGE] Rejected … missing_dispatch_id` и `[INVARIANT] 1 audio-orch violations`.
+Описание: animastor-docs `architecture/COMMAND_ONLY_GENERATION.md` §«Эпоха dispatch».
+
+### Changed (поведение)
+
+- `scene-orchestrator.executeAudioDispatch` открывает новую эпоху на каждом
+  dispatch: `setWaitingChunks(…, { dispatchId, buildId, expectedCount })`.
+  Хост пишет `started_at=now`, `dispatch_id`, `expected_count`,
+  `chunks_received=0` и убирает `fail_reason`/`failed_at`/`last_chunk_at`
+  прошлой попытки. Хост без поддержки 5-го аргумента ведёт себя как раньше.
+- `reconciliation-engine.checkStalledAudioScenes`: запись проваливается, только
+  если её `dispatch_id` равен живому `dispatch-meta` сцены. Запись без
+  `dispatch_id` или другого dispatch — `stale_state_ignored` (одна info-строка
+  на запись, не FAILED). `failWaitingScene` получает `dispatchId` записи.
+- `orchestrator.failStage` без `dispatchId` → `{failed:false,
+  reason:'stale_state_ignored'}` (info), а не `Rejected … missing_dispatch_id`.
+
+Fence, command-gate (Stage 155) и retry-политика не меняются.
+
 ## 0.3.1 (2026-11-01) — результат GPU обрабатывается один раз
 
 Основание: `reports/2026-11-01/gpu-audio-image-run-analysis.md`, P1 — каждый

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@animastor/orchestration",
-    "version": "0.3.1",
+    "version": "0.3.4",
     "description": "Animastor Orchestration — the generation orchestration/runtime contour of the Animastor backend, physically extracted from the host (§32.30): the unified generation lifecycle owner (orchestrator facade, stage executor, scene callbacks/restoration), the dispatch engine and its lease/quota/retry managers, the reconciliation engine and counter reconciliation, the scene window/scheduler/worker-health, the runtime-result emitter/consumer, and the eight O-2..O-10 port contracts (Persistence, SceneData, PlaceholderAudio, ProgressEvents, AudioFsm, VideoFsm, HubCancel, LayerConfig). The package reaches the host (runtime config slices, Redis-domain state layer, media facades, the S-5 seam registry, the artifact root, the PW-2 workspace resolver) ONLY through the composition-root host-bindings seam — no backend/src require, no storage/book/routes/services, no Redis, no PG, no Express inside.",
     "keywords": [
         "animastor",

package/src/orchestration/orchestrator.js

@@ -288,6 +288,14 @@ async function failStage(redis, bookId, chapterId, sceneId, stage, buildId, reas
         throw new Error(`orchestrator.failStage: unknown stage '${stage}'`);
     }
 
+    // GPU E2E 2026-11-02 (§6.1): a fail signal without dispatch_id comes
+    // from a legacy / stale record (no epoch) — it can never fail the live
+    // dispatch. A regular no-op, not a reject.
+    if (!dispatchId || typeof dispatchId !== 'string') {
+        log(`[FAIL-STAGE] stale_state_ignored ${bookId}/${chapterId}/${sceneId} ${stage}: no dispatch_id (reason=${reason})`);
+        return { failed: false, reason: 'stale_state_ignored' };
+    }
+
     const identity = await dispatchEngine.verifyDispatchIdentity(
         redis,
         bookId,
@@ -491,7 +499,7 @@ async function resetScenes(redis, bookId, buildId, scenes, layerCfg, options = {
     
     if (!scenes || scenes.length === 0) {
         log('[RESET-SCENES] No scenes to reset');
-        return { marked: 0, reset_scenes: 0 };
+        return { marked: 0, reset_scenes: 0, dispatchIds: [] };
     }
 
     const scopeDisplay = `${scope}${chapterId ? '/' + chapterId : ''}${sceneId ? '/' + sceneId : ''}`;
@@ -627,7 +635,10 @@ async function resetScenes(redis, bookId, buildId, scenes, layerCfg, options = {
         { scenes_count: scenes.length, marked: marked.marked }
     ).catch(() => {});
 
-    return { ...marked, reset_scenes: scenes.length };
+    // 0.3.3 (reports/2026-11-04/regenerate-sweep-fix.md): the dispatch ids
+    // cancelled by THIS reset — the only hub entries a caller may sweep again
+    // (a book-wide sweep would remove other scenes' in-flight jobs).
+    return { ...marked, reset_scenes: scenes.length, dispatchIds: [...new Set(cancellation.dispatchIds || [])] };
 }
 
 // ── S-5: pure FSM-writer ownership moved to the state layer ──

package/src/orchestration/scene-callbacks.js

@@ -109,9 +109,18 @@ async function handleAudioCompleted(redis, bookId, chapterId, sceneId, buildId)
         return { ok: false, retryable: false, reason: 'invalid_asset_state' };
     }
 
-    const isReady = await audio.isSceneAudioReady(buildId, bookId, chapterId, sceneId, require('music-metadata'));
+    // Ready means "a fenced GPU commit of this generation produced the
+    // chunks" — NOT "a file exists at the canonical path". A silent
+    // placeholder is a valid 10.85 s MP3 and passed the old file-only check,
+    // which is how sc-4bb85e02 reached READY with silence on 2026-11-04.
+    // The legacy probe remains only for hosts that have not wired the
+    // committed-audio probe yet.
+    const committedProbe = typeof audio.isSceneAudioCommitted === 'function'
+        ? await audio.isSceneAudioCommitted(buildId, bookId, chapterId, sceneId, null)
+        : { ready: await audio.isSceneAudioReady(buildId, bookId, chapterId, sceneId, require('music-metadata')), reason: 'legacy_file_check', chunks: [] };
+    const isReady = !!(committedProbe && committedProbe.ready === true);
     if (!isReady) {
-        error(`Audio not ready after completion: ${bookId}/${chapterId}/${sceneId}`);
+        error(`Audio not ready after completion: ${bookId}/${chapterId}/${sceneId} (reason=${committedProbe && committedProbe.reason}, chunks=[${(committedProbe && committedProbe.chunks || []).join(',')}])`);
         const scene = { book_id: bookId, chapter_id: chapterId, scene_id: sceneId };
         await logEvent(redis, scene, 'AUDIO_FAILED', 'audio_ready', {
             reason: 'not_ready_validation'
@@ -149,8 +158,11 @@ async function handleAudioCompleted(redis, bookId, chapterId, sceneId, buildId)
             const metadata = await mm.parseFile(audioPath);
             realDuration = metadata.format.duration || 0;
         } catch {}
+        // The commit proof travels down: the placeholder service refuses to
+        // upgrade a record to 'ready' without it (2026-11-04 regression).
         await placeholderAudioOp('replacePlaceholderWithRealAudio')(
-            bookId, chapterId, sceneId, buildId, audioPath, realDuration
+            bookId, chapterId, sceneId, buildId, audioPath, realDuration,
+            { committed: isReady }
         );
     } catch (err) {
         warn(`Failed to replace placeholder audio: ${err.message}`);

package/src/orchestration/scene-orchestrator.js

@@ -53,6 +53,28 @@ const orchestrator = require('./orchestrator');
 // S-2: media registry for executor dispatch
 const mediaRegistry = require('@animastor/generation').mediaRegistry;
 
+/**
+ * Fenced-commit probe for the audio FSM guards.
+ *
+ * A DONE record is only authoritative when a real GPU commit backs it. On
+ * 2026-11-04 the already_ready fast-track wrote DONE for a scene that had no
+ * committed chunk at all, and AUDIO_ALREADY_DONE then swallowed every later
+ * "audio" command. Returns false when the probe is unavailable (fail closed
+ * for the guard — regenerating is safe, skipping is not).
+ */
+async function hasCommittedAudio(buildId, bookId, chapterId, sceneId, expectedCount) {
+    try {
+        if (typeof audio.isSceneAudioCommitted === 'function') {
+            const probe = await audio.isSceneAudioCommitted(buildId, bookId, chapterId, sceneId, expectedCount || null);
+            return !!(probe && probe.ready === true);
+        }
+        return await audio.isSceneAudioReady(buildId, bookId, chapterId, sceneId);
+    } catch (err) {
+        warn(`AUDIO_READY_PROBE failed for ${bookId}/${chapterId}/${sceneId}: ${err.message}`);
+        return false;
+    }
+}
+
 // ======================================================
 // SCENE ORCHESTRATOR
 // ======================================================
@@ -211,8 +233,16 @@ async function executeAudioDispatch(redis, scene, loadedBook, buildId, dispatchI
     const orchState = await audioFsmOp('getState')(redis, bookId, chapterId, sceneId);
     if (orchState) {
         if (orchState.phase === audioFsmPhases().DONE) {
-            log(`AUDIO_ALREADY_DONE: ${bookId}/${chapterId}/${sceneId} — skipping dispatch`);
-            return { dispatched: false, jobs: 0, completed: true, reason: 'already_done' };
+            // DONE is only authoritative with a committed chunk behind it.
+            if (await hasCommittedAudio(buildId, bookId, chapterId, sceneId, orchState.expected_count)) {
+                log(`AUDIO_ALREADY_DONE: ${bookId}/${chapterId}/${sceneId} — skipping dispatch`);
+                return { dispatched: false, jobs: 0, completed: true, reason: 'already_done' };
+            }
+            log(
+                `AUDIO_ALREADY_DONE rejected: ${bookId}/${chapterId}/${sceneId} — DONE without a committed audio chunk ` +
+                `(expected=${orchState.expected_count || '?'}), resetting state to regenerate`
+            );
+            await audioFsmOp('deleteState')(redis, bookId, chapterId, sceneId);
         }
     }
 
@@ -234,8 +264,11 @@ async function executeAudioDispatch(redis, scene, loadedBook, buildId, dispatchI
         const stalePhase = transResult.from;
         // DONE — дополнительный safety net (выше уже должен был отсечься)
         if (stalePhase === audioFsmPhases().DONE) {
-            warn(`AUDIO_ORCH: DONE guard prevented stale reset for ${bookId}/${chapterId}/${sceneId}`);
-            return { dispatched: false, jobs: 0, completed: true, reason: 'already_done' };
+            if (await hasCommittedAudio(buildId, bookId, chapterId, sceneId, orchState && orchState.expected_count)) {
+                warn(`AUDIO_ORCH: DONE guard prevented stale reset for ${bookId}/${chapterId}/${sceneId}`);
+                return { dispatched: false, jobs: 0, completed: true, reason: 'already_done' };
+            }
+            warn(`AUDIO_ORCH: stale DONE without a committed chunk for ${bookId}/${chapterId}/${sceneId} — resetting to PLACEHOLDER_READY`);
         }
         // WAITING_CHUNKS / MERGING (stale recovery): с A2 timeout formula lease
         // переживает watchdog — если дошли до stale recovery, значит lease истёк
@@ -280,7 +313,16 @@ async function executeAudioDispatch(redis, scene, loadedBook, buildId, dispatchI
     // jobs, иначе первый чанк может вернуться быстрее чем setWaitingChunks выполнится,
     // и completeChunk увидит фазу GENERATING → early return → чанк на диске есть,
     // но merge не запускается → retry exhausting → failStage → re-dispatch → цикл.
-    await audioFsmOp('setWaitingChunks')(redis, bookId, chapterId, sceneId);
+    //
+    // GPU E2E 2026-11-02 (§6.1): every dispatch opens a NEW audio-orch epoch
+    // (started_at=now, dispatch_id, expected_count; fail_reason/failed_at/
+    // last_chunk_at of an earlier or cancelled attempt are dropped). The stall
+    // watchdog compares this dispatch_id with the live dispatch metadata.
+    await audioFsmOp('setWaitingChunks')(redis, bookId, chapterId, sceneId, {
+        dispatchId,
+        buildId,
+        expectedCount: segments.buildSegments(sceneData).length,
+    });
 
     const result = await audio.generateSceneAudio(redis, sceneData, bookData, buildId, bookId, dispatchId);
 

package/src/runtime/dispatch-engine.js

@@ -1623,8 +1623,11 @@ async function clearLeasesForScenes(redis, bookId, scenes) {
  *
  * The contract now is:
  *   - ids present        → one `?dispatch_id=` DELETE per id (unchanged);
- *   - no ids + book_id   → exactly ONE `?book_id=` DELETE (the whole-book sweep
+ *   - no ids + book_id + `allowBookSweep: true`
+ *                        → exactly ONE `?book_id=` DELETE (the whole-book sweep
  *                          the hub already supports and cache/delete paths use);
+ *   - no ids + book_id WITHOUT `allowBookSweep` (0.3.3, fail-closed)
+ *                        → NO request, a warning, `{requested:0, refused:1}`;
  *   - neither            → NO request at all (never a fake HTTP call).
  *
  * A book-scoped sweep is checked against the hub's echoed `scope.book_id`:
@@ -1659,6 +1662,19 @@ async function clearHubDispatches(dispatchIds, options = {}) {
     // ── Book-scoped sweep (no explicit dispatch ids) ──────────────────────
     // Exactly one request; the hub filters by record.book_id so only this
     // book's queue/running/result entries are removed.
+    //
+    // 0.3.3 FAIL-CLOSED (reports/2026-11-04/regenerate-sweep-fix.md): a
+    // book-wide sweep removes EVERY in-flight job of the book — incident
+    // 2026-11-04: a one-scene audio /regenerate swept a running image job
+    // of another scene (result rejected 409, lost) and a queued one (back
+    // only after 7 min of stale-lease recovery). It is issued ONLY when the
+    // caller states it explicitly (`allowBookSweep: true` — an explicit
+    // cancellation / full restart of the whole book / book deletion).
+    // Otherwise: no request at all, a warning, `refused: 1`.
+    if (ids.length === 0 && bookId && options.allowBookSweep !== true) {
+        reportWarning(`${context}: book-wide GPU hub sweep for ${bookId} REFUSED (no dispatch ids, allowBookSweep not set) — nothing swept`);
+        return { requested: 0, cleared: 0, failed: 0, refused: 1 };
+    }
     if (ids.length === 0 && bookId) {
         try {
             const response = await fetchImpl(

package/src/runtime/reconciliation-engine.js

@@ -473,6 +473,33 @@ async function checkPartialBuilds(redis, bookId, chapterId, sceneId) {
 // Важно: watchdog НЕ конкурирует с gpu-hub timeout — они ловят
 // разные сценарии (стоп прогресса vs мёртвый воркер).
 
+// GPU E2E 2026-11-02 (§6.1) — dispatch epoch of an audio-orch record.
+// current=true only when the record carries a dispatch_id AND it equals the
+// live dispatch metadata of the scene's audio stage. A metadata read error is
+// "cannot prove" → not current (the watchdog never fails on a guess).
+async function _audioDispatchEpoch(redis, bookId, chapterId, sceneId, orchState) {
+    if (!orchState || !orchState.dispatch_id) return { current: false, reason: 'no_dispatch_id' };
+    let meta = null;
+    try {
+        meta = await require('./dispatch-engine').getDispatchMetadata(redis, bookId, chapterId, sceneId, 'audio');
+    } catch (_) {
+        return { current: false, reason: 'dispatch_metadata_unreadable' };
+    }
+    if (!meta || !meta.dispatch_id) return { current: false, reason: 'no_active_dispatch' };
+    if (meta.dispatch_id !== orchState.dispatch_id) return { current: false, reason: 'other_dispatch' };
+    return { current: true, reason: 'current_dispatch' };
+}
+
+// One info line per stale record (scene + its epoch), not one per cycle.
+const _staleAudioOrchLogged = new Set();
+function _logStaleAudioOrchOnce(sceneRef, orchState, reason) {
+    const k = `${sceneRef}|${orchState.dispatch_id || ''}|${orchState.started_at || ''}`;
+    if (_staleAudioOrchLogged.has(k)) return;
+    if (_staleAudioOrchLogged.size > 5000) _staleAudioOrchLogged.clear();
+    _staleAudioOrchLogged.add(k);
+    log(`[STALLED-AUDIO] ${sceneRef} — stale_state_ignored (${reason}): the record is not of the live dispatch, not failing it`);
+}
+
 async function checkStalledAudioScenes(redis, deps) {
     // O-7: the audio FSM is consumed through the AudioFsmPort (host adapter:
     // storage/audio-fsm-adapter). The pre-O-7 optional-load semantics are
@@ -513,12 +540,24 @@ async function checkStalledAudioScenes(redis, deps) {
             const age = Date.now() - threshold;
             if (age < AUDIO_CHUNK_STALL_MS) continue;
 
+            // GPU E2E 2026-11-02 (§6.1): the record must belong to the LIVE
+            // dispatch. A record without dispatch_id (legacy / pre-epoch) or
+            // of another dispatch is a stale epoch — failing it marked a
+            // healthy run FAILED (`chunk_stall_timeout:4619s`). It is ignored
+            // (one info line per record), never failed.
+            const epoch = await _audioDispatchEpoch(redis, bookId, chapterId, sceneId, orchState);
+            if (!epoch.current) {
+                _logStaleAudioOrchOnce(`${bookId}/${chapterId}/${sceneId}`, orchState, epoch.reason);
+                continue;
+            }
+
             const reason = `chunk_stall_timeout:${Math.round(age / 1000)}s`;
-            warn(`[STALLED-AUDIO] ${bookId}/${chapterId}/${sceneId} — ${reason} (last_chunk_at=${!!lastChunkAt} started_at=${!!startedAt})`);
+            warn(`[STALLED-AUDIO] ${bookId}/${chapterId}/${sceneId} — ${reason} (last_chunk_at=${!!lastChunkAt} started_at=${!!startedAt} dispatch=${String(orchState.dispatch_id).slice(0, 24)})`);
 
             try {
                 await audioOrch.failWaitingScene(redis, bookId, chapterId, sceneId, buildId, reason, {
-                    orchestrator: deps.orchestrator
+                    orchestrator: deps.orchestrator,
+                    dispatchId: orchState.dispatch_id,
                 });
                 stalled++;
             } catch (err) {
@@ -551,7 +590,7 @@ async function checkStalledAudioScenes(redis, deps) {
                     warn(`[STALLED-AUDIO] completeChunk recovery failed: ${err.message}`);
                     try {
                         await audioOrch.failWaitingScene(redis, bookId, chapterId, sceneId, buildId,
-                            `recovery_fallback:${err.message}`, { orchestrator: deps.orchestrator });
+                            `recovery_fallback:${err.message}`, { orchestrator: deps.orchestrator, dispatchId: orchState.dispatch_id });
                         stalled++;
                     } catch (fsErr) {
                         warn(`[STALLED-AUDIO] failWaitingScene fallback failed: ${fsErr.message}`);
@@ -561,7 +600,7 @@ async function checkStalledAudioScenes(redis, deps) {
                 warn(`[STALLED-AUDIO] ${bookId}/${chapterId}/${sceneId} — ${presentCount}/${expectedCount} on disk, no last_chunk_at. Partial chunks, failing scene.`);
                 try {
                     await audioOrch.failWaitingScene(redis, bookId, chapterId, sceneId, buildId,
-                        `partial_chunks:${presentCount}/${expectedCount}_no_timestamps`, { orchestrator: deps.orchestrator });
+                        `partial_chunks:${presentCount}/${expectedCount}_no_timestamps`, { orchestrator: deps.orchestrator, dispatchId: orchState.dispatch_id });
                     stalled++;
                 } catch (err) {
                     warn(`[STALLED-AUDIO] failWaitingScene failed: ${err.message}`);