@goodandready/dsh-moa 0.2.11 → 0.2.12

package/README.md

@@ -199,6 +199,7 @@ dsh-moa:
 | `presets[].quorum_enabled` | `boolean` | `false` | Straggler mitigation: proceed with synthesis once >= 60% candidates respond |
 | `presets[].grace_period_sec` | `number` | `10` | Grace period in seconds to wait for stragglers after quorum is reached |
 | `presets[].aggregator_fallbacks` | `array` | `[]` | Ordered fallback judge models tried if primary aggregator encounters transient errors |
+| `presets[].blind_evaluation` | `boolean` | `false` | Anonymize candidate model names for the judge/curator to eliminate family/brand bias |
 | `presets[].reference_timeout_sec` | `number` | `60` | Per-candidate execution timeout in seconds |
 | `presets[].aggregator_timeout_sec` | `number` | `180` | Aggregator/judge synthesis timeout in seconds |
 | `presets[].reference_temperature` / `.aggregator_temperature` | `number` | `0.6` / `0.4` | Sampling temperatures for proposers and judge |

package/docs/README.ru.md

@@ -199,6 +199,7 @@ dsh-moa:
 | `presets[].quorum_enabled` | `boolean` | `false` | Защита от зависших моделей (stragglers): запуск синтеза при ответе от >= 60% кандидатов |
 | `presets[].grace_period_sec` | `number` | `10` | Грейс-период (в секундах) ожидания оставшихся моделей после достижения кворума |
 | `presets[].aggregator_fallbacks` | `array` | `[]` | Список запасных моделей-судей при сбоях основной модели агрегатора |
+| `presets[].blind_evaluation` | `boolean` | `false` | Обезличивание имен кандидатов («Candidate 1», «Candidate 2») для исключения предвзятости судьи |
 | `presets[].reference_timeout_sec` | `number` | `60` | Таймаут опроса каждого кандидата в секундах |
 | `presets[].aggregator_timeout_sec` | `number` | `180` | Таймаут синтеза решения судьей в секундах |
 | `presets[].reference_temperature` / `.aggregator_temperature` | `number` | `0.6` / `0.4` | Температуры сэмплирования советников и судьи |

package/docs/design/DESIGN.md

@@ -58,6 +58,7 @@
 - 2026-09-10 — Унификация UI с дизайн-стандартом `dsh-clinebot` (статусные чипы в header, секционные карточки, дизайн-токены `--dsw-alias-*`, глубокий аудит устойчивости).
 - 2026-09-11 — English-canonical пользовательские строки (сервер и клиент); ru-перевод предоставляет translation-плагин, собственный ru-дубль из пакета удалён. Причина: стандарт DSH (dsh-plugin-authoring); пересмотр — только по явному решению владельца.
 - 2026-09-11 — Заглушка Live Canvas удалена (фабрикация `http://localhost:3000/preview/...`); заявления README сняты. Реальная интеграция с `dsh-live-canvas` — отдельная задача (Gitea #46). Changed: прежний пункт про автопревью больше не действует.
+- 2026-09-12 — Реализован режим слепого судейства (`blind_evaluation: boolean`), гарантированное зеркалирование языка запроса (Language Mirroring Guard в промптах судьи и куратора), интерактивный лидерборд моделей (винрейт, запуски, победы, средняя стоимость) в секции аналитики UI и раздельные настройки таймаутов кандидатов и судьи в карточке настроек (Gitea Issue #57).
 - 2026-09-11 — Статусный бейдж карточки отражает фактический `GET /dsh-moa/status` (online / offline / disabled); телеметрия Total Runs / Avg Run Cost берётся из `GET /dsh-moa/history`. Причина: карточка не должна показывать состояния, которые она не проверяла.
 - 2026-09-11 — Переключатель `enabled` в карточке сохраняется через `settingsScope`/REST и влияет на `/moa`-turn и `POST /dsh-moa/run`.
 - 2026-09-11 (вечер) — Интеграция с Live Canvas реализована через собственный REST-контракт `@goodandready/dsh-live-canvas` (`POST /dsh-live-canvas/api/preview`, self-call на порт хоста `ctx.webServer.port`) с тихой деградацией при отсутствии плагина (любая ошибка → ответ без preview-ссылки). Заменяет прежнее решение об удалении фабрикованной заглушки: ссылка `/dsh-live-canvas/sandbox/<id>` теперь создаётся реальной песочницей.

package/lib/client.js

@@ -55,6 +55,18 @@ window.__ModuleLoader__.load({
       'aggregator.curator_hint': 'When enabled: the curator highlights the finest components of each candidate, applies the antipatterns rubric, and advises which agent model should assemble the solution.',
       'aggregator.stream_label': 'Live Stream Curator/Judge Thinking',
       'aggregator.stream_hint': 'Streams aggregator tokens directly to chat in real time for zero-latency initial response.',
+      'aggregator.blind_label': 'Blind Review (Anonymize Candidates for Judge)',
+      'aggregator.blind_hint': 'When enabled: candidate model names and providers are hidden from the judge prompt to prevent family/brand bias.',
+      'aggregator.timeout_label': 'Judge synthesis timeout (sec):',
+      'proposers.timeout_label': 'Candidate timeout (sec):',
+      'leaderboard.title': '🏆 Model Win-Rate Leaderboard',
+      'leaderboard.desc': 'Performance metrics calculated from recorded MoA run verdicts.',
+      'leaderboard.model': 'Model',
+      'leaderboard.runs': 'Runs',
+      'leaderboard.wins': 'Wins',
+      'leaderboard.winrate': 'Win Rate',
+      'leaderboard.cost': 'Avg Cost',
+      'leaderboard.empty': 'No historical runs recorded yet.',
       'aggregator.fallbacks_title': 'Fallback Judges Chain',
       'aggregator.fallbacks_desc': 'Sequential backup models invoked automatically if the primary judge experiences rate limits or outages.',
       'aggregator.add_fallback_btn': '+ Add Fallback Judge',
@@ -448,6 +460,7 @@ window.__ModuleLoader__.load({
       setEnabled,
       hostStatus,
       stats,
+      leaderboard = [],
       handleSave,
       saveStatus,
       reload,
@@ -831,6 +844,39 @@ window.__ModuleLoader__.load({
                 { style: { fontSize: 12, color: 'var(--dsw-alias-label-secondary)', marginLeft: 22 } },
                 t('aggregator.stream_hint')
               ),
+              React.createElement(
+                'label',
+                { style: { display: 'flex', alignItems: 'center', gap: 8, cursor: 'pointer', fontSize: 13, fontWeight: 500, marginTop: 8 } },
+                React.createElement('input', {
+                  type: 'checkbox',
+                  checked: Boolean(currentPreset.blind_evaluation),
+                  onChange: (e) => updateCurrentPreset((p) => ({ ...p, blind_evaluation: e.target.checked })),
+                  style: { accentColor: 'var(--dsw-alias-state-brand-primary, var(--dsw-alias-label-primary))', cursor: 'pointer' },
+                }),
+                t('aggregator.blind_label')
+              ),
+              React.createElement(
+                'div',
+                { style: { fontSize: 12, color: 'var(--dsw-alias-label-secondary)', marginLeft: 22 } },
+                t('aggregator.blind_hint')
+              ),
+              React.createElement(
+                'div',
+                { style: { display: 'flex', alignItems: 'center', gap: 8, marginTop: 8 } },
+                React.createElement('span', { style: { fontSize: 12, color: 'var(--dsw-alias-label-secondary)' } }, t('aggregator.timeout_label')),
+                React.createElement('input', {
+                  type: 'number',
+                  className: 'moa-input',
+                  style: { width: 80, height: 28, padding: '0 6px', fontSize: 12 },
+                  min: 30,
+                  max: 600,
+                  value: currentPreset.aggregator_timeout_sec ?? 180,
+                  onChange: (e) => {
+                    const val = parseInt(e.target.value, 10)
+                    updateCurrentPreset((p) => ({ ...p, aggregator_timeout_sec: isNaN(val) ? 180 : val }))
+                  },
+                })
+              ),
               /* Fallback Judges Chain */
               React.createElement(
                 'div',
@@ -968,6 +1014,23 @@ window.__ModuleLoader__.load({
                   updateCurrentPreset((p) => ({ ...p, grace_period_sec: isNaN(val) ? 10 : val }))
                 },
               })
+            ),
+            React.createElement(
+              'div',
+              { style: { display: 'flex', alignItems: 'center', gap: 8, marginTop: 4 } },
+              React.createElement('span', { style: { fontSize: 12, color: 'var(--dsw-alias-label-secondary)' } }, t('proposers.timeout_label')),
+              React.createElement('input', {
+                type: 'number',
+                className: 'moa-input',
+                style: { width: 70, height: 28, padding: '0 6px', fontSize: 12 },
+                min: 10,
+                max: 300,
+                value: currentPreset.reference_timeout_sec ?? 60,
+                onChange: (e) => {
+                  const val = parseInt(e.target.value, 10)
+                  updateCurrentPreset((p) => ({ ...p, reference_timeout_sec: isNaN(val) ? 60 : val }))
+                },
+              })
             )
           ),
           React.createElement(
@@ -1072,6 +1135,48 @@ window.__ModuleLoader__.load({
               React.createElement('div', { className: 'moa-stat-val' }, String(currentRefs.length)),
               React.createElement('div', { className: 'moa-stat-lbl' }, t('stats.active_models'))
             )
+          ),
+          Array.isArray(leaderboard) && leaderboard.length > 0 && React.createElement(
+            'div',
+            { style: { marginTop: 12, display: 'flex', flexDirection: 'column', gap: 6 } },
+            React.createElement('div', { style: { fontSize: 13, fontWeight: 600, color: 'var(--dsw-alias-label-primary)' } }, t('leaderboard.title')),
+            React.createElement('div', { style: { fontSize: 12, color: 'var(--dsw-alias-label-secondary)', marginTop: -4 } }, t('leaderboard.desc')),
+            React.createElement(
+              'div',
+              { style: { overflowX: 'auto', border: '1px solid var(--dsw-alias-border-l2)', borderRadius: 8 } },
+              React.createElement(
+                'table',
+                { style: { width: '100%', borderCollapse: 'collapse', fontSize: 12, textAlign: 'left' } },
+                React.createElement(
+                  'thead',
+                  { style: { background: 'var(--dsw-alias-bg-layer-2)', borderBottom: '1px solid var(--dsw-alias-border-l2)' } },
+                  React.createElement(
+                    'tr',
+                    null,
+                    React.createElement('th', { style: { padding: '6px 10px' } }, t('leaderboard.model')),
+                    React.createElement('th', { style: { padding: '6px 8px' } }, t('leaderboard.runs')),
+                    React.createElement('th', { style: { padding: '6px 8px' } }, t('leaderboard.wins')),
+                    React.createElement('th', { style: { padding: '6px 8px' } }, t('leaderboard.winrate')),
+                    React.createElement('th', { style: { padding: '6px 10px' } }, t('leaderboard.cost'))
+                  )
+                ),
+                React.createElement(
+                  'tbody',
+                  null,
+                  leaderboard.map((m, mIdx) =>
+                    React.createElement(
+                      'tr',
+                      { key: mIdx, style: { borderBottom: mIdx < leaderboard.length - 1 ? '1px solid var(--dsw-alias-border-l2)' : 'none' } },
+                      React.createElement('td', { style: { padding: '6px 10px', fontWeight: 500, color: 'var(--dsw-alias-label-primary)' } }, m.modelKey || m.model),
+                      React.createElement('td', { style: { padding: '6px 8px', color: 'var(--dsw-alias-label-secondary)' } }, String(m.runs)),
+                      React.createElement('td', { style: { padding: '6px 8px', color: 'var(--dsw-alias-state-success-primary)' } }, String(m.wins)),
+                      React.createElement('td', { style: { padding: '6px 8px', fontWeight: 600 } }, (m.winRate ?? 0) + '%'),
+                      React.createElement('td', { style: { padding: '6px 10px', color: 'var(--dsw-alias-label-secondary)' } }, (m.avgCostUsd > 0) ? ('$' + m.avgCostUsd.toFixed(4)) : 'Free')
+                    )
+                  )
+                )
+              )
+            )
           )
         ),
 
@@ -1151,6 +1256,7 @@ window.__ModuleLoader__.load({
       const [enabled, setEnabled] = React.useState(true)
       const [hostStatus, setHostStatus] = React.useState('connecting')
       const [stats, setStats] = React.useState({ totalRuns: null, avgCostUsd: null })
+      const [leaderboard, setLeaderboard] = React.useState([])
 
       React.useEffect(() => {
         if (!scope || !snapshot) return
@@ -1182,6 +1288,15 @@ window.__ModuleLoader__.load({
             setHostStatus('offline')
           })
 
+        fetch('/dsh-moa/leaderboard', { cache: 'no-store' })
+          .then((r) => r.json())
+          .then((data) => {
+            if (data && data.ok && Array.isArray(data.models)) {
+              setLeaderboard(data.models.slice(0, 10))
+            }
+          })
+          .catch(() => {})
+
         fetch('/dsh-moa/history?limit=100', { cache: 'no-store' })
           .then((r) => r.json())
           .then((data) => {
@@ -1290,6 +1405,7 @@ window.__ModuleLoader__.load({
         setEnabled,
         hostStatus,
         stats,
+        leaderboard,
         handleSave,
         saveStatus,
         reload,

package/lib/index.js

@@ -58,6 +58,7 @@ export const PresetSchema = z.object({
   aggregator_temperature: z.number().default(0.4),
   reference_timeout_sec: z.number().default(60),
   aggregator_timeout_sec: z.number().default(180),
+  blind_evaluation: z.boolean().default(false),
   max_tokens: z.number().default(4096),
   judge_criteria: z.string().default(''),
 })

package/lib/moa-prompts.js

@@ -33,6 +33,9 @@ Penalize and strictly downgrade candidates exhibiting any of the following flaws
 8. 🚫 Context Amnesia & Regressions:
    - Dropping or breaking previously functioning project features while adding new code.`
 
+export const LANGUAGE_MIRRORING_DIRECTIVE = `### 🌐 Language Mirroring Requirement
+CRITICAL: You MUST write your entire analysis, reasoning, verdict, explanations and instructions in the EXACT SAME LANGUAGE as the user's prompt (e.g. Russian if the prompt is written in Russian, English if in English). Do NOT switch or translate to English unless explicitly requested by the user.`
+
 /**
  * Formats a provider + model slot into a readable string key.
  */
@@ -141,15 +144,19 @@ The user gave the task:
 The advisors proposed the following decision points and clarifications:
 ${joined}
 
+${LANGUAGE_MIRRORING_DIRECTIVE}
+
 Your task is to synthesize a single, compact, friendly and structured questionnaire (2-4 questions) in the same language as the user's prompt.
 Each question must offer 2-3 concrete recommended answer options (e.g.: 1. Format: single-file HTML/JS or React? 2. Style: minimalism, iOS or neubrutalism?).
 At the end, add a note that the user can answer briefly (e.g.: "1, 2, dark theme") or trust the defaults.`
 }
 
 /**
- * Builds the curator synthesis prompt with component analysis and model recommendation.
+ * Builds the curator synthesis prompt with component analysis, model recommendation, and optional blind review.
  */
-export function buildCuratorSynthesisPrompt(userPrompt, referenceOutputs = [], judgeCriteria = '') {
+export function buildCuratorSynthesisPrompt(userPrompt, referenceOutputs = [], judgeCriteria = '', options = {}) {
+  const isBlind = Boolean(options.blindEvaluation)
+
   const joined = referenceOutputs
     .map((r, i) => {
       const fileSummary = (r.files && r.files.length > 0)
@@ -159,7 +166,10 @@ export function buildCuratorSynthesisPrompt(userPrompt, referenceOutputs = [], j
       if (referenceOutputs.length >= 3 && textContent.length > 3000) {
         textContent = stripOrSummarizeCode(textContent)
       }
-      return `Candidate ${i + 1} — ${r.label}:${fileSummary}\n${textContent}`
+      const header = isBlind
+        ? `Candidate ${i + 1}:${fileSummary}`
+        : `Candidate ${i + 1} — ${r.label}:${fileSummary}`
+      return `${header}\n${textContent}`
     })
     .join('\n\n')
 
@@ -178,8 +188,10 @@ ${joined}
 
 ${ANTIPATTERNS_RUBRIC}
 
+${LANGUAGE_MIRRORING_DIRECTIVE}
+
 Instructions:
-Your response MUST be structured into three clear parts (respond in the same language as the user's prompt, e.g. Russian):
+Your response MUST be structured into three clear parts:
 
 ### 1. 🔍 Curator Analysis & Component Breakdown
 - For EACH candidate, provide:
@@ -189,7 +201,7 @@ Your response MUST be structured into three clear parts (respond in the same lan
 
 ### 2. 🧩 Assembly Recipe & Recommended Master Assembler
 - Recommend the best single agent model to assemble and finalize the solution:
-  RECOMMENDED_ASSEMBLER: <number from 1 to N> (<provider:model>)
+  RECOMMENDED_ASSEMBLER: <number from 1 to N> ${isBlind ? '' : '(<provider:model>)'}
 - State the machine winner index marker for file promotion:
   WINNER_CANDIDATE_INDEX: <number from 1 to N>
 - Provide the exact blueprint / instructions for combining the best pieces into a unified deliverable.
@@ -200,13 +212,15 @@ Your response MUST be structured into three clear parts (respond in the same lan
 }
 
 /**
- * Builds the judge synthesis prompt for standard or curator mode.
+ * Builds the judge synthesis prompt for standard or curator mode with optional blind evaluation.
  */
 export function buildSynthesisPrompt(userPrompt, referenceOutputs = [], judgeCriteria = '', options = {}) {
   if (options.curatorSynthesis) {
-    return buildCuratorSynthesisPrompt(userPrompt, referenceOutputs, judgeCriteria)
+    return buildCuratorSynthesisPrompt(userPrompt, referenceOutputs, judgeCriteria, options)
   }
 
+  const isBlind = Boolean(options.blindEvaluation)
+
   const joined = referenceOutputs
     .map((r, i) => {
       const fileSummary = (r.files && r.files.length > 0)
@@ -216,7 +230,10 @@ export function buildSynthesisPrompt(userPrompt, referenceOutputs = [], judgeCri
       if (referenceOutputs.length >= 3 && textContent.length > 3000) {
         textContent = stripOrSummarizeCode(textContent)
       }
-      return `Reference ${i + 1} — ${r.label}:${fileSummary}\n${textContent}`
+      const header = isBlind
+        ? `Reference ${i + 1}:${fileSummary}`
+        : `Reference ${i + 1} — ${r.label}:${fileSummary}`
+      return `${header}\n${textContent}`
     })
     .join('\n\n')
 
@@ -234,11 +251,13 @@ ${joined}
 
 ${ANTIPATTERNS_RUBRIC}
 
+${LANGUAGE_MIRRORING_DIRECTIVE}
+
 Instructions:
-Your response MUST be structured into three clear parts (respond in the same language as the user's prompt, e.g. Russian):
+Your response MUST be structured into three clear parts:
 
 ### 1. ⚖️ Judge verdict and comparative analysis
-- **Winner**: clearly name the model and candidate number (e.g. "Winner: Candidate 1 (opencode-go:deepseek-v4-flash)" or "Reference 1 (label) is chosen").
+- **Winner**: clearly name the chosen candidate (e.g. "Winner: Candidate 1" or "Reference 1 is chosen").
 - Always add the machine winner-selection marker:
   WINNER_CANDIDATE_INDEX: <number from 1 to N>
 - **Why this choice**: compare code, architecture, strengths, weaknesses and reliability of all candidates in detail.

package/lib/moa-runner.js

@@ -327,6 +327,7 @@ export async function runMoAPipeline({
   const candidateRetries = 1
   const refTimeoutSec = typeof preset?.reference_timeout_sec === 'number' ? preset.reference_timeout_sec : 60
   const aggTimeoutSec = typeof preset?.aggregator_timeout_sec === 'number' ? preset.aggregator_timeout_sec : 180
+  const isBlindEvaluation = Boolean(preset?.blind_evaluation)
 
   // 2. Collect project context for refinement tasks
   const isRefinement = isRefinementTask(userPrompt)
@@ -531,7 +532,10 @@ export async function runMoAPipeline({
   }
 
   // 8. Synthesis phase via primary judge or fallback chain
-  const synthesisPrompt = buildSynthesisPrompt(userPrompt, referenceOutputs, judgeCriteria, { curatorSynthesis: isCuratorSynthesis })
+  const synthesisPrompt = buildSynthesisPrompt(userPrompt, referenceOutputs, judgeCriteria, {
+    curatorSynthesis: isCuratorSynthesis,
+    blindEvaluation: isBlindEvaluation,
+  })
   let synthesizedText = ''
   let aggUsage = { inputTokens: 0, outputTokens: 0, totalTokens: 0, costUsd: 0 }
   let chosenJudge = primaryJudge

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@goodandready/dsh-moa",
-  "version": "0.2.11",
+  "version": "0.2.12",
   "description": "Mixture of Agents (MoA) plugin for DeepSeek Harness with /moa slash command",
   "type": "module",
   "main": "lib/index.js",