@animastor/orchestration 0.3.2 → 0.3.4

package/CHANGELOG.md

@@ -1,5 +1,61 @@
 # Changelog
 
+## 0.3.4 (2026-11-04) — 0.3.3 + «ready» аудио только при коммите GPU
+
+Одна публикация вместо двух: 0.3.3 (уборка хаба при `/regenerate`, см. ниже) **не
+публиковался** в npm отдельно и входит в 0.3.4 целиком. Основание:
+`reports/2026-11-04/audio-placeholder-ready.md` (сцена `sc-4bb85e02` — тишина-заглушка
+стала `ready` без GPU), интеграция: `reports/2026-11-04/integration-A-B-C.md`.
+Описание инварианта: animastor-docs `02-orchestration/AUDIO_READY_SEMANTICS.md`.
+
+### Changed (поведение)
+
+- `scene-orchestrator.executeAudioDispatch`: `AUDIO_ALREADY_DONE` пропускает работу только
+  если хост подтверждает коммит (`audio.isSceneAudioCommitted(...).ready === true`:
+  канонический файл без маркера заглушки **и** чанки сцены на диске). Иначе — лог
+  `AUDIO_ALREADY_DONE rejected … DONE without a committed audio chunk`, FSM-состояние
+  удаляется, стадия идёт обычным путём (dispatch на GPU). То же — в safety-net ветке
+  «stale `DONE`». Пробник недоступен или упал → считается «не закоммичено» (fail-closed:
+  лишняя генерация безопасна, пропуск — нет).
+- `scene-callbacks.handleAudioCompleted`: проверка после завершения — `isSceneAudioCommitted`
+  (legacy `isSceneAudioReady` только если хост не дал нового пробника); в
+  `replacePlaceholderWithRealAudio` передаётся `{ committed: true }` — хост без этого
+  доказательства не поднимает запись в `ready`.
+
+### Host contract (новое, обратно совместимо)
+
+- `media.audio.isSceneAudioCommitted(buildId, bookId, chapterId, sceneId, expectedCount|null)`
+  → `{ ready, reason, chunks, path }`. Backend ≥ интеграции A+B его даёт; на старом хосте
+  пакет откатывается к file-only проверке.
+- `placeholderAudio.replacePlaceholderWithRealAudio(…, realDuration, options)` — 7-й аргумент
+  `options.committed`.
+
+Инварианты 0.3.3 (sweep только своих dispatch-id, fail-closed book-wide sweep) не менялись;
+fence, command-gate и retry-политика — тоже.
+
+## 0.3.3 (2026-11-04, не публиковался — вошёл в 0.3.4) — /regenerate не сносит чужие задачи на хабе
+
+Основание: `reports/2026-11-04/gpu-generation-stall.md` §5.3, §9.1 — `/regenerate`
+одной сцены (audio) закончился `clearHubDispatches([], { book_id })` →
+`DELETE /queue/clear?book_id=`; хаб удалил running-запись image-задачи другой
+сцены (результат отклонён 409 и потерян) и queued-задачу (восстановлена
+stale-lease recovery через 7 мин). Отчёт исправления:
+`reports/2026-11-04/regenerate-sweep-fix.md`. Описание: animastor-docs
+`architecture/COMMAND_ONLY_GENERATION.md` §«Уборка хаба при /regenerate».
+
+### Changed (поведение)
+
+- `dispatch-engine.clearHubDispatches`: book-scoped форма (пустой список id +
+  `book_id`) — **fail-closed**: запрос `?book_id=` уходит только при явном
+  `allowBookSweep: true`; иначе ни одного запроса, warn
+  `book-wide GPU hub sweep … REFUSED`, результат `{requested:0, cleared:0,
+  failed:0, refused:1}`. Per-id форма не менялась.
+- `orchestrator.resetScenes` возвращает `dispatchIds` — dispatch-id,
+  отменённые этим reset'ом (только сбрасываемые сцена×стадия); вызывающий
+  повторно убирает на хабе только их.
+
+Fence, command-gate и retry-политика не меняются.
+
 ## 0.3.2 (2026-11-02) — эпоха dispatch в audio-orch, без ложного FAILED
 
 Основание: `reports/2026-11-02/gpu-e2e-installer-025-logs.md` §6.1, §8 п.3, п.5 —

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@animastor/orchestration",
-    "version": "0.3.2",
+    "version": "0.3.4",
     "description": "Animastor Orchestration — the generation orchestration/runtime contour of the Animastor backend, physically extracted from the host (§32.30): the unified generation lifecycle owner (orchestrator facade, stage executor, scene callbacks/restoration), the dispatch engine and its lease/quota/retry managers, the reconciliation engine and counter reconciliation, the scene window/scheduler/worker-health, the runtime-result emitter/consumer, and the eight O-2..O-10 port contracts (Persistence, SceneData, PlaceholderAudio, ProgressEvents, AudioFsm, VideoFsm, HubCancel, LayerConfig). The package reaches the host (runtime config slices, Redis-domain state layer, media facades, the S-5 seam registry, the artifact root, the PW-2 workspace resolver) ONLY through the composition-root host-bindings seam — no backend/src require, no storage/book/routes/services, no Redis, no PG, no Express inside.",
     "keywords": [
         "animastor",

package/src/orchestration/orchestrator.js

@@ -499,7 +499,7 @@ async function resetScenes(redis, bookId, buildId, scenes, layerCfg, options = {
     
     if (!scenes || scenes.length === 0) {
         log('[RESET-SCENES] No scenes to reset');
-        return { marked: 0, reset_scenes: 0 };
+        return { marked: 0, reset_scenes: 0, dispatchIds: [] };
     }
 
     const scopeDisplay = `${scope}${chapterId ? '/' + chapterId : ''}${sceneId ? '/' + sceneId : ''}`;
@@ -635,7 +635,10 @@ async function resetScenes(redis, bookId, buildId, scenes, layerCfg, options = {
         { scenes_count: scenes.length, marked: marked.marked }
     ).catch(() => {});
 
-    return { ...marked, reset_scenes: scenes.length };
+    // 0.3.3 (reports/2026-11-04/regenerate-sweep-fix.md): the dispatch ids
+    // cancelled by THIS reset — the only hub entries a caller may sweep again
+    // (a book-wide sweep would remove other scenes' in-flight jobs).
+    return { ...marked, reset_scenes: scenes.length, dispatchIds: [...new Set(cancellation.dispatchIds || [])] };
 }
 
 // ── S-5: pure FSM-writer ownership moved to the state layer ──

package/src/orchestration/scene-callbacks.js

@@ -109,9 +109,18 @@ async function handleAudioCompleted(redis, bookId, chapterId, sceneId, buildId)
         return { ok: false, retryable: false, reason: 'invalid_asset_state' };
     }
 
-    const isReady = await audio.isSceneAudioReady(buildId, bookId, chapterId, sceneId, require('music-metadata'));
+    // Ready means "a fenced GPU commit of this generation produced the
+    // chunks" — NOT "a file exists at the canonical path". A silent
+    // placeholder is a valid 10.85 s MP3 and passed the old file-only check,
+    // which is how sc-4bb85e02 reached READY with silence on 2026-11-04.
+    // The legacy probe remains only for hosts that have not wired the
+    // committed-audio probe yet.
+    const committedProbe = typeof audio.isSceneAudioCommitted === 'function'
+        ? await audio.isSceneAudioCommitted(buildId, bookId, chapterId, sceneId, null)
+        : { ready: await audio.isSceneAudioReady(buildId, bookId, chapterId, sceneId, require('music-metadata')), reason: 'legacy_file_check', chunks: [] };
+    const isReady = !!(committedProbe && committedProbe.ready === true);
     if (!isReady) {
-        error(`Audio not ready after completion: ${bookId}/${chapterId}/${sceneId}`);
+        error(`Audio not ready after completion: ${bookId}/${chapterId}/${sceneId} (reason=${committedProbe && committedProbe.reason}, chunks=[${(committedProbe && committedProbe.chunks || []).join(',')}])`);
         const scene = { book_id: bookId, chapter_id: chapterId, scene_id: sceneId };
         await logEvent(redis, scene, 'AUDIO_FAILED', 'audio_ready', {
             reason: 'not_ready_validation'
@@ -149,8 +158,11 @@ async function handleAudioCompleted(redis, bookId, chapterId, sceneId, buildId)
             const metadata = await mm.parseFile(audioPath);
             realDuration = metadata.format.duration || 0;
         } catch {}
+        // The commit proof travels down: the placeholder service refuses to
+        // upgrade a record to 'ready' without it (2026-11-04 regression).
         await placeholderAudioOp('replacePlaceholderWithRealAudio')(
-            bookId, chapterId, sceneId, buildId, audioPath, realDuration
+            bookId, chapterId, sceneId, buildId, audioPath, realDuration,
+            { committed: isReady }
         );
     } catch (err) {
         warn(`Failed to replace placeholder audio: ${err.message}`);

package/src/orchestration/scene-orchestrator.js

@@ -53,6 +53,28 @@ const orchestrator = require('./orchestrator');
 // S-2: media registry for executor dispatch
 const mediaRegistry = require('@animastor/generation').mediaRegistry;
 
+/**
+ * Fenced-commit probe for the audio FSM guards.
+ *
+ * A DONE record is only authoritative when a real GPU commit backs it. On
+ * 2026-11-04 the already_ready fast-track wrote DONE for a scene that had no
+ * committed chunk at all, and AUDIO_ALREADY_DONE then swallowed every later
+ * "audio" command. Returns false when the probe is unavailable (fail closed
+ * for the guard — regenerating is safe, skipping is not).
+ */
+async function hasCommittedAudio(buildId, bookId, chapterId, sceneId, expectedCount) {
+    try {
+        if (typeof audio.isSceneAudioCommitted === 'function') {
+            const probe = await audio.isSceneAudioCommitted(buildId, bookId, chapterId, sceneId, expectedCount || null);
+            return !!(probe && probe.ready === true);
+        }
+        return await audio.isSceneAudioReady(buildId, bookId, chapterId, sceneId);
+    } catch (err) {
+        warn(`AUDIO_READY_PROBE failed for ${bookId}/${chapterId}/${sceneId}: ${err.message}`);
+        return false;
+    }
+}
+
 // ======================================================
 // SCENE ORCHESTRATOR
 // ======================================================
@@ -211,8 +233,16 @@ async function executeAudioDispatch(redis, scene, loadedBook, buildId, dispatchI
     const orchState = await audioFsmOp('getState')(redis, bookId, chapterId, sceneId);
     if (orchState) {
         if (orchState.phase === audioFsmPhases().DONE) {
-            log(`AUDIO_ALREADY_DONE: ${bookId}/${chapterId}/${sceneId} — skipping dispatch`);
-            return { dispatched: false, jobs: 0, completed: true, reason: 'already_done' };
+            // DONE is only authoritative with a committed chunk behind it.
+            if (await hasCommittedAudio(buildId, bookId, chapterId, sceneId, orchState.expected_count)) {
+                log(`AUDIO_ALREADY_DONE: ${bookId}/${chapterId}/${sceneId} — skipping dispatch`);
+                return { dispatched: false, jobs: 0, completed: true, reason: 'already_done' };
+            }
+            log(
+                `AUDIO_ALREADY_DONE rejected: ${bookId}/${chapterId}/${sceneId} — DONE without a committed audio chunk ` +
+                `(expected=${orchState.expected_count || '?'}), resetting state to regenerate`
+            );
+            await audioFsmOp('deleteState')(redis, bookId, chapterId, sceneId);
         }
     }
 
@@ -234,8 +264,11 @@ async function executeAudioDispatch(redis, scene, loadedBook, buildId, dispatchI
         const stalePhase = transResult.from;
         // DONE — дополнительный safety net (выше уже должен был отсечься)
         if (stalePhase === audioFsmPhases().DONE) {
-            warn(`AUDIO_ORCH: DONE guard prevented stale reset for ${bookId}/${chapterId}/${sceneId}`);
-            return { dispatched: false, jobs: 0, completed: true, reason: 'already_done' };
+            if (await hasCommittedAudio(buildId, bookId, chapterId, sceneId, orchState && orchState.expected_count)) {
+                warn(`AUDIO_ORCH: DONE guard prevented stale reset for ${bookId}/${chapterId}/${sceneId}`);
+                return { dispatched: false, jobs: 0, completed: true, reason: 'already_done' };
+            }
+            warn(`AUDIO_ORCH: stale DONE without a committed chunk for ${bookId}/${chapterId}/${sceneId} — resetting to PLACEHOLDER_READY`);
         }
         // WAITING_CHUNKS / MERGING (stale recovery): с A2 timeout formula lease
         // переживает watchdog — если дошли до stale recovery, значит lease истёк

package/src/runtime/dispatch-engine.js

@@ -1623,8 +1623,11 @@ async function clearLeasesForScenes(redis, bookId, scenes) {
  *
  * The contract now is:
  *   - ids present        → one `?dispatch_id=` DELETE per id (unchanged);
- *   - no ids + book_id   → exactly ONE `?book_id=` DELETE (the whole-book sweep
+ *   - no ids + book_id + `allowBookSweep: true`
+ *                        → exactly ONE `?book_id=` DELETE (the whole-book sweep
  *                          the hub already supports and cache/delete paths use);
+ *   - no ids + book_id WITHOUT `allowBookSweep` (0.3.3, fail-closed)
+ *                        → NO request, a warning, `{requested:0, refused:1}`;
  *   - neither            → NO request at all (never a fake HTTP call).
  *
  * A book-scoped sweep is checked against the hub's echoed `scope.book_id`:
@@ -1659,6 +1662,19 @@ async function clearHubDispatches(dispatchIds, options = {}) {
     // ── Book-scoped sweep (no explicit dispatch ids) ──────────────────────
     // Exactly one request; the hub filters by record.book_id so only this
     // book's queue/running/result entries are removed.
+    //
+    // 0.3.3 FAIL-CLOSED (reports/2026-11-04/regenerate-sweep-fix.md): a
+    // book-wide sweep removes EVERY in-flight job of the book — incident
+    // 2026-11-04: a one-scene audio /regenerate swept a running image job
+    // of another scene (result rejected 409, lost) and a queued one (back
+    // only after 7 min of stale-lease recovery). It is issued ONLY when the
+    // caller states it explicitly (`allowBookSweep: true` — an explicit
+    // cancellation / full restart of the whole book / book deletion).
+    // Otherwise: no request at all, a warning, `refused: 1`.
+    if (ids.length === 0 && bookId && options.allowBookSweep !== true) {
+        reportWarning(`${context}: book-wide GPU hub sweep for ${bookId} REFUSED (no dispatch ids, allowBookSweep not set) — nothing swept`);
+        return { requested: 0, cleared: 0, failed: 0, refused: 1 };
+    }
     if (ids.length === 0 && bookId) {
         try {
             const response = await fetchImpl(