@goodandready/dsh-moa 0.2.35 → 0.2.37

package/README.md

@@ -170,7 +170,7 @@ Seamlessly retains synthesized baseline code from prior turns while pruning inte
 `report_generation_enabled` generates comprehensive Markdown & JSON reports detailing candidate metrics, agreement scores, test gate results, and cost breakdowns via `GET /dsh-moa/runs/:id/report`.
 
 ### 18. Graceful Degradation & Local Fallback Resilience
-When `local_fallback_enabled: true`, candidates that fail due to network outages or rate limits (429/500) automatically recover using configured local models (Ollama / MiniPC).
+When `local_fallback_enabled: true`, candidates that fail due to network outages, timeouts, or rate limits (429/500) automatically recover using configured local models (Ollama / MiniPC). Each fallback model operates with an independent execution budget and an isolated controller linked to the parent turn signal, preventing cloud timeouts from starving fallback models. Additionally, terminal errors and aborted streams are strictly classified, ensuring reliable error propagation and failover.
 
 ### 6. Live Canvas 1-Click Preview (optional)
 If `@goodandready/dsh-live-canvas` is installed in the same profile, `dsh-moa` pushes the promoted HTML file to the Live Canvas REST contract (`POST /dsh-live-canvas/api/preview`, served by the same harness webServer) and appends a one-click preview link (`/dsh-live-canvas/sandbox/<id>`) to the answer. Without the plugin the step is skipped silently — no errors in the log, no dead links.

package/README.ru.md

@@ -170,7 +170,7 @@ graph TD
 При включении `report_generation_enabled` генерируется подробный аналитический отчет с матрицей моделей, временами задержек, оценками судей, расходами и результатами тестов. Отчет доступен по API `GET /dsh-moa/runs/:id/report?format=md`.
 
 ### 18. Плавная деградация и локальный фоллбэк
-Опция `local_fallback_enabled` позволяет задать резервные локальные модели (Ollama, vLLM, MiniPC), к которым плагин бесшовно обращается при сбоях сетевых API или превышении лимитов rate limit (429/500).
+Опция `local_fallback_enabled` позволяет задать резервные локальные модели (Ollama, vLLM, MiniPC), к которым плагин бесшовно обращается при сбоях сетевых API, таймаутах или превышении лимитов rate limit (429/500). Каждая фоллбэк-модель запускается с независимым бюджетом времени и изолированным контроллером отмены, связанным с сигналом основного хода, предотвращая исчерпание времени ожидания из-за задержек облачных провайдеров. Стриминговые сбои и терминальные ошибки классифицируются типизированными исключениями с гарантированной защитой от позднего промоушна.
 
 ### 6. Live Canvas 1-клик предпросмотр (опционально)
 Если в профиле установлен `@goodandready/dsh-live-canvas`, `dsh-moa` отправляет промоученный HTML-файл в REST-контракт Live Canvas (`POST /dsh-live-canvas/api/preview`, тот же webServer харнесса) и добавляет к ответу ссылку на предпросмотр (`/dsh-live-canvas/sandbox/<id>`). Без плагина шаг пропускается тихо — без ошибок в журнале и без битых ссылок.

package/README.zh.md

@@ -142,6 +142,9 @@ graph TD
 ### 8. 提升前 Git 快照检查点 (`dsh-time-machine`)
 在将胜出候选者的文件提升并覆盖至项目根目录前，`dsh-moa` 自动调用本地 `dsh-time-machine` 建立影子 Git 检查点 (`moa-pre-promotion: candidate-N`)。如未安装则静默回退，不阻断提升流程。
 
+### 18. 平滑降级与本地容灾恢复 (Local Fallback Resilience)
+当 `local_fallback_enabled: true` 时，因网络中断、超时或速率限制（429/500）而失败的候选模型会自动使用配置的本地模型（Ollama / MiniPC）恢复。每个后备模型都拥有独立的超时执行配额与关联主 turn signal 的隔离取消控制器，防止云端超时耗尽本地恢复时间。同时严格检测终止错误与流中断，确保安全容灾且杜绝已取消 turn 的延迟文件提升。
+
 ### 6. Live Canvas 一键预览（可选）
 若同一 profile 中安装了 `@goodandready/dsh-live-canvas`，`dsh-moa` 会将提升后的 HTML 文件推送到 Live Canvas 的 REST 契约（`POST /dsh-live-canvas/api/preview`，由同一 harness webServer 提供服务），并在回答中附上一键预览链接（`/dsh-live-canvas/sandbox/<id>`）。未安装该插件时此步骤静默跳过——日志无报错，也不会出现死链接。
 

package/lib/moa-candidates.js

@@ -99,6 +99,7 @@ export async function runReferencesParallel(references, messages, options = {},
     }
 
     const runOne = async () => {
+      let candidateTimer = null
       try {
         const callPromise = callWithTransientRetry(
           callLlm,
@@ -115,14 +116,12 @@ export async function runReferencesParallel(references, messages, options = {},
         )
 
         const timeoutPromise = new Promise((_, reject) => {
-          const timer = setTimeout(() => {
+          candidateTimer = setTimeout(() => {
             bestEffort('candidate timeout abort', () => {
               abortControllers[i].abort(new Error(`Timeout after ${Math.round(timeoutMs / 1000)}s`))
             })
             reject(new Error(`Timeout after ${Math.round(timeoutMs / 1000)}s`))
           }, timeoutMs)
-          timer.unref?.()
-          abortControllers[i].signal.addEventListener('abort', () => clearTimeout(timer), { once: true })
         })
 
         const res = await Promise.race([callPromise, timeoutPromise])
@@ -153,16 +152,51 @@ export async function runReferencesParallel(references, messages, options = {},
           onProgress(`✅ *Candidate ${i + 1}/${total} (${label}, T=${effectiveTemperature}) finished${costStr}* | Live total: ~\$${currentTotalCost.toFixed(4)}\n`)
         }
       } catch (err) {
+        if (candidateTimer) clearTimeout(candidateTimer)
+
+        // If turn was aborted by caller, do NOT run fallbacks
+        if (options.signal?.aborted) {
+          finishedCount++
+          results[i] = {
+            index: i + 1,
+            slot,
+            label,
+            role_persona: role,
+            temperature: effectiveTemperature,
+            text: '[aborted]',
+            usage: { inputTokens: 0, outputTokens: 0, totalTokens: 0, costUsd: 0 },
+            costUsd: 0,
+            ok: false,
+            error: 'Turn aborted',
+          }
+          return
+        }
+
         // Feature 10: Local fallback models if online candidate fails
         let fallbackRecovered = false
         if (options.local_fallback_enabled && Array.isArray(options.local_fallback_models) && options.local_fallback_models.length > 0) {
           for (const fbSlot of options.local_fallback_models) {
+            if (options.signal?.aborted) break
             const fbLabel = slotLabel(fbSlot)
+            const fbAbortCtrl = new AbortController()
+            let fbTimer = null
+            let removeParentAbort = null
+
+            if (options.signal) {
+              const onParentAbort = () => fbAbortCtrl.abort(options.signal.reason || new Error('Turn aborted'))
+              options.signal.addEventListener('abort', onParentAbort, { once: true })
+              removeParentAbort = () => options.signal.removeEventListener('abort', onParentAbort)
+            }
+
+            const fbTimeoutMs = typeof options.fallbackTimeoutMs === 'number' && options.fallbackTimeoutMs > 0
+              ? options.fallbackTimeoutMs
+              : Math.max(15000, timeoutMs)
+
             try {
               if (typeof onProgress === 'function') {
                 onProgress(`🔄 *Candidate ${i + 1} (${label}) failed, attempting local fallback to ${fbLabel}...*\n`)
               }
-              const fbRes = await callWithTransientRetry(
+              const fbCallPromise = callWithTransientRetry(
                 callLlm,
                 {
                   provider: fbSlot.provider,
@@ -170,11 +204,22 @@ export async function runReferencesParallel(references, messages, options = {},
                   messages: slotMessages,
                   temperature: effectiveTemperature,
                   maxTokens: options.maxTokens ?? 4096,
-                  timeoutMs,
-                  signal: abortControllers[i].signal,
+                  timeoutMs: fbTimeoutMs,
+                  signal: fbAbortCtrl.signal,
                 },
                 0
               )
+
+              const fbTimeoutPromise = new Promise((_, reject) => {
+                fbTimer = setTimeout(() => {
+                  bestEffort('fallback timeout abort', () => {
+                    fbAbortCtrl.abort(new Error(`Timeout after ${Math.round(fbTimeoutMs / 1000)}s`))
+                  })
+                  reject(new Error(`Timeout after ${Math.round(fbTimeoutMs / 1000)}s`))
+                }, fbTimeoutMs)
+              })
+
+              const fbRes = await Promise.race([fbCallPromise, fbTimeoutPromise])
               const fbText = typeof fbRes === 'string' ? fbRes : (fbRes?.content || fbRes?.text || '')
               const fbUsage = (typeof fbRes === 'object' && fbRes?.usage) ? fbRes.usage : {
                 inputTokens: Math.max(1, Math.round(slotMessages.map((m) => m.content).join('').length / 4)),
@@ -202,6 +247,9 @@ export async function runReferencesParallel(references, messages, options = {},
               break
             } catch (fbErr) {
               logger.warn(`[dsh-moa] Fallback ${fbLabel} also failed:`, fbErr?.message || fbErr)
+            } finally {
+              if (fbTimer) clearTimeout(fbTimer)
+              if (removeParentAbort) removeParentAbort()
             }
           }
         }

package/lib/moa-llm.js

@@ -1,17 +1,49 @@
+function combineSignals(signal, timeoutSignal) {
+  if (!signal) return timeoutSignal
+  if (typeof AbortSignal.any === 'function') {
+    try {
+      return AbortSignal.any([signal, timeoutSignal])
+    } catch {
+      // fallback if signal is not an instance of AbortSignal
+    }
+  }
+  const combinedCtrl = new AbortController()
+  const onAbort = () => {
+    const reason = signal.aborted ? (signal.reason || new Error('Turn aborted')) : timeoutSignal.reason
+    combinedCtrl.abort(reason)
+  }
+  if (signal.aborted || timeoutSignal.aborted) {
+    onAbort()
+    return combinedCtrl.signal
+  }
+  signal.addEventListener?.('abort', onAbort, { once: true })
+  timeoutSignal.addEventListener?.('abort', onAbort, { once: true })
+  return combinedCtrl.signal
+}
+
 /**
  * Unified LLM Dispatcher using ctx.llm.prepareCall / stream
  */
 
 export function createLlmCaller(ctx) {
-  return async ({ provider, model, messages, temperature = 0.6, maxTokens = 4096, timeoutMs, onStreamDelta }) => {
+  return async ({ provider, model, messages, temperature = 0.6, maxTokens = 4096, timeoutMs, onStreamDelta, signal, stop, reasoningEffort }) => {
     if (!ctx.llm) {
       throw new Error('ctx.llm is not available in cordis context')
     }
 
+    const callConfig = {
+      provider,
+      model,
+      ...(temperature !== undefined ? { temperature } : {}),
+      ...(maxTokens !== undefined ? { maxTokens } : {}),
+      ...(stop !== undefined ? { stop } : {}),
+      ...(reasoningEffort !== undefined ? { reasoningEffort } : {}),
+    }
+
     let prep
     try {
-      // DSH 0.1.2-rc.1 API expects { provider, model } object
-      prep = await ctx.llm.prepareCall({ provider, model })
+      // DSH 0.2.0-rc.2 API expects LlmCallConfig object, supports signal
+      prep = await ctx.llm.prepareCall(callConfig, signal)
     } catch (err) {
       if (typeof ctx.llm.prepareCall === 'function') {
         try {
@@ -27,23 +59,33 @@ export function createLlmCaller(ctx) {
       throw new Error(`LLM provider/model ${provider}:${model} could not be prepared`)
     }
 
+    if (signal?.aborted) {
+      throw (signal.reason || new Error('Turn aborted'))
+    }
+
     const effectiveTimeoutMs = typeof timeoutMs === 'number' && timeoutMs > 0 ? timeoutMs : 120000
     const abortCtrl = new AbortController()
     const timeoutId = setTimeout(() => abortCtrl.abort(new Error(`LLM call timeout after ${Math.round(effectiveTimeoutMs / 1000)}s`)), effectiveTimeoutMs)
     timeoutId.unref?.()
+    const combinedSignal = combineSignals(signal, abortCtrl.signal)
 
     try {
+      const streamConfig = prep.config || callConfig
       const stream = prep.stream({
+        ...streamConfig,
         messages,
-        temperature,
-        maxTokens,
-        signal: abortCtrl.signal,
+        temperature: streamConfig.temperature ?? temperature,
+        maxTokens: streamConfig.maxTokens ?? maxTokens,
+        signal: combinedSignal,
       })
 
       let fullText = ''
       let usageInfo = null
 
       for await (const chunk of stream) {
+        if (combinedSignal?.aborted) {
+          throw (combinedSignal.reason || new Error('Turn aborted'))
+        }
         if (chunk.type === 'text-delta' && typeof chunk.text === 'string') {
           fullText += chunk.text
           if (typeof onStreamDelta === 'function') {
@@ -53,9 +95,35 @@ export function createLlmCaller(ctx) {
           if (!fullText) fullText = chunk.block.text
         } else if (chunk.type === 'usage' && chunk.usage) {
           usageInfo = chunk.usage
+        } else if (chunk.type === 'finish') {
+          const reason = chunk.reason
+          if (reason && (reason.kind === 'error' || reason.kind === 'aborted')) {
+            const failure = reason.failure || {}
+            const errMsg = failure.message || failure.error?.message || reason.message || reason.error?.message || `LLM stream finished with ${reason.kind}`
+            const err = new Error(errMsg)
+            err.code = failure.code || reason.code || (reason.kind === 'error' ? 'LLM_ERROR' : 'ABORT_ERR')
+            if (failure.cause) err.cause = failure.cause
+            throw err
+          }
+        } else if (chunk.type === 'error') {
+          const errMsg = chunk.error?.message || chunk.message || 'LLM stream error'
+          const err = new Error(errMsg)
+          err.code = chunk.code || chunk.error?.code || 'LLM_ERROR'
+          if (chunk.error?.cause) err.cause = chunk.error.cause
+          throw err
         }
       }
 
+      if (combinedSignal?.aborted) {
+        throw (combinedSignal.reason || new Error('Turn aborted'))
+      }
+
+      if (!fullText.trim()) {
+        const err = new Error('LLM call returned empty response')
+        err.code = 'EMPTY_RESPONSE'
+        throw err
+      }
+
       clearTimeout(timeoutId)
       return {
         content: fullText,

package/lib/moa-runner.js

@@ -207,6 +207,11 @@ export async function runMoAPipeline({
     }
   }
 
+  if (signal?.aborted) {
+    await cleanMoaWorkspaces(cwd)
+    return { error: new Error('Turn aborted') }
+  }
+
   // 5b. Test Execution Gate
   await executeTestGateForCandidates({
     cwd,
@@ -292,6 +297,10 @@ export async function runMoAPipeline({
 
   // 7. Fast Mode (1 candidate, bypass judge)
   if (isFastMode) {
+    if (signal?.aborted) {
+      await cleanMoaWorkspaces(cwd)
+      return { error: new Error('Turn aborted') }
+    }
     const single = successfulRefs[0]
     let promotedFiles = []
     if (single.files?.length > 0) {
@@ -348,6 +357,11 @@ export async function runMoAPipeline({
     }
   }
 
+  if (signal?.aborted) {
+    await cleanMoaWorkspaces(cwd)
+    return { error: new Error('Turn aborted') }
+  }
+
   // 7b. Consilium Round 2 (Peer Critique) if enabled
   if (isPeerCritiqueEnabled && successfulRefs.length > 1) {
     if (typeof onProgress === 'function') {
@@ -404,6 +418,11 @@ export async function runMoAPipeline({
 
   // 7c. Feature 2: Multi-Judge Panel & Consensus Voting
   let multiJudgeResult = null
+  if (signal?.aborted) {
+    await cleanMoaWorkspaces(cwd)
+    return { error: new Error('Turn aborted') }
+  }
+
   if (preset?.multi_judge_enabled && successfulRefs.length > 1) {
     multiJudgeResult = await executeMultiJudgePanel({
       judges: (preset.judge_models && preset.judge_models.length > 0) ? preset.judge_models : [primaryJudge],
@@ -488,6 +507,11 @@ export async function runMoAPipeline({
       successfulRefs.map((r, i) => `### Candidate ${i + 1} (${r.label})\n${r.text}`).join('\n\n')
   }
 
+  if (signal?.aborted) {
+    await cleanMoaWorkspaces(cwd)
+    return { error: new Error('Turn aborted') }
+  }
+
   // 9. Evaluate winner & promote files
   const winningIndex = (multiJudgeResult?.consensus && multiJudgeResult?.winningCandidateIndex)
     ? multiJudgeResult.winningCandidateIndex

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@goodandready/dsh-moa",
-  "version": "0.2.35",
+  "version": "0.2.37",
   "description": "Mixture of Agents (MoA) plugin for DeepSeek Harness with /moa slash command",
   "type": "module",
   "main": "lib/index.js",