aiforcecli-chat 0.1.0

package/tools/scorecard/generated/example/scorecard-report.md

@@ -0,0 +1,147 @@
+# Generated Scorecard Report
+
+Generated: 2026-06-16T20:38:21.186Z
+Snapshots: 3
+Mapped models: 6
+Unmapped rows: 69
+Invalid rows: 0
+
+## Snapshot Files
+
+- tools\scorecard\snapshots\example.normalized.example.json
+- tools\scorecard\snapshots\live.aider_polyglot.json
+- tools\scorecard\snapshots\live.terminal_bench_2_1.json
+
+## Scores
+
+### antigravity:gemini-3.1-pro
+
+| Task | Score | Confidence | Evidence Weight | Top Evidence |
+| --- | ---: | ---: | ---: | --- |
+| bugfix | 73.3% | 21.4% | 0.5456 | terminal_bench/accuracy (Gemini CLI Gemini 3.1 Pro) |
+| feature | 74.2% | 19.9% | 0.4954 | terminal_bench/accuracy (Gemini CLI Gemini 3.1 Pro) |
+| refactor | 75.7% | 19.5% | 0.484 | aider_polyglot/pass_rate_2 (gemini-2.5-pro-preview-06-05 32k think) |
+| test | 73.6% | 16.1% | 0.3831 | terminal_bench/accuracy (Gemini CLI Gemini 3.1 Pro) |
+| docs | 79.8% | 4.6% | 0.0969 | aider_polyglot/pass_rate_2 (gemini-2.5-pro-preview-06-05 32k think) |
+| security | 72.1% | 13.3% | 0.3056 | terminal_bench/accuracy (Gemini CLI Gemini 3.1 Pro) |
+| perf | 72.5% | 15.9% | 0.3791 | terminal_bench/accuracy (Gemini CLI Gemini 3.1 Pro) |
+| general | 72.3% | 24% | 0.6305 | terminal_bench/accuracy (Gemini CLI Gemini 3.1 Pro) |
+
+### claude-code:opus
+
+| Task | Score | Confidence | Evidence Weight | Top Evidence |
+| --- | ---: | ---: | ---: | --- |
+| bugfix | 74.5% | 14.1% | 0.3276 | terminal_bench/accuracy (Claude Code Claude Opus 4.8) |
+| feature | 74.5% | 11.3% | 0.2548 | terminal_bench/accuracy (Claude Code Claude Opus 4.8) |
+| refactor | 74.5% | 8.3% | 0.182 | terminal_bench/accuracy (Claude Code Claude Opus 4.8) |
+| test | 74.5% | 9.8% | 0.2184 | terminal_bench/accuracy (Claude Code Claude Opus 4.8) |
+| security | 74.5% | 9.8% | 0.2184 | terminal_bench/accuracy (Claude Code Claude Opus 4.8) |
+| perf | 74.5% | 11.3% | 0.2548 | terminal_bench/accuracy (Claude Code Claude Opus 4.8) |
+| general | 74.5% | 17.9% | 0.4368 | terminal_bench/accuracy (Claude Code Claude Opus 4.8) |
+
+### claude-code:sonnet
+
+| Task | Score | Confidence | Evidence Weight | Top Evidence |
+| --- | ---: | ---: | ---: | --- |
+| bugfix | 71% | 38.7% | 1.2626 | swebench_verified/resolved (Claude Sonnet) |
+| feature | 69.9% | 25.3% | 0.6779 | aider_polyglot/percent_correct (Claude Sonnet) |
+| refactor | 70.2% | 32.1% | 0.9465 | swebench_verified/resolved (Claude Sonnet) |
+| test | 70% | 19.2% | 0.4766 | aider_polyglot/percent_correct (Claude Sonnet) |
+| docs | 68% | 6.3% | 0.1341 | aider_polyglot/percent_correct (Claude Sonnet) |
+| security | 70.6% | 15.6% | 0.3693 | swebench_verified/resolved (Claude Sonnet) |
+| perf | 70.2% | 17.5% | 0.4229 | swebench_verified/resolved (Claude Sonnet) |
+| general | 70.4% | 26.4% | 0.7184 | swebench_verified/resolved (Claude Sonnet) |
+
+### codex:gpt-5.4
+
+| Task | Score | Confidence | Evidence Weight | Top Evidence |
+| --- | ---: | ---: | ---: | --- |
+| bugfix | 86.7% | 5.2% | 0.1095 | aider_polyglot/pass_rate_2 (gpt-5 medium) |
+| feature | 86.7% | 5.6% | 0.1186 | aider_polyglot/pass_rate_2 (gpt-5 medium) |
+| refactor | 86.7% | 6.8% | 0.146 | aider_polyglot/pass_rate_2 (gpt-5 medium) |
+| test | 86.7% | 3.9% | 0.0821 | aider_polyglot/pass_rate_2 (gpt-5 medium) |
+| docs | 86.7% | 2.2% | 0.0456 | aider_polyglot/pass_rate_2 (gpt-5 medium) |
+| security | 86.7% | 2.2% | 0.0456 | aider_polyglot/pass_rate_2 (gpt-5 medium) |
+| perf | 86.7% | 3.1% | 0.0639 | aider_polyglot/pass_rate_2 (gpt-5 medium) |
+| general | 86.7% | 4.8% | 0.1004 | aider_polyglot/pass_rate_2 (gpt-5 medium) |
+
+### codex:gpt-5.4-mini
+
+| Task | Score | Confidence | Evidence Weight | Top Evidence |
+| --- | ---: | ---: | ---: | --- |
+| bugfix | 77.4% | 10.6% | 0.2364 | evalplus/pass_at_1 (gpt-5.4-mini) |
+| feature | 76.2% | 16.6% | 0.3978 | evalplus/pass_at_1 (gpt-5.4-mini) |
+| refactor | 77.9% | 12% | 0.2729 | aider_polyglot/pass_rate_2 (gpt-5 low) |
+| test | 75.4% | 18% | 0.4375 | evalplus/pass_at_1 (gpt-5.4-mini) |
+| docs | 77.5% | 4.6% | 0.0964 | evalplus/pass_at_1 (gpt-5.4-mini) |
+| security | 76.7% | 5.7% | 0.1218 | evalplus/pass_at_1 (gpt-5.4-mini) |
+| perf | 76.8% | 7.6% | 0.1654 | evalplus/pass_at_1 (gpt-5.4-mini) |
+| general | 76.2% | 14.1% | 0.3288 | evalplus/pass_at_1 (gpt-5.4-mini) |
+
+### codex:gpt-5.5
+
+| Task | Score | Confidence | Evidence Weight | Top Evidence |
+| --- | ---: | ---: | ---: | --- |
+| bugfix | 85.3% | 11.7% | 0.2636 | terminal_bench/accuracy (Codex CLI GPT-5.5) |
+| feature | 85.7% | 10.6% | 0.2382 | terminal_bench/accuracy (Codex CLI GPT-5.5) |
+| refactor | 86.3% | 10.4% | 0.231 | aider_polyglot/pass_rate_2 (gpt-5 high) |
+| test | 85.4% | 8.5% | 0.1848 | terminal_bench/accuracy (Codex CLI GPT-5.5) |
+| docs | 88% | 2.2% | 0.0453 | aider_polyglot/pass_rate_2 (gpt-5 high) |
+| security | 84.8% | 6.9% | 0.1486 | terminal_bench/accuracy (Codex CLI GPT-5.5) |
+| perf | 85% | 8.4% | 0.1839 | terminal_bench/accuracy (Codex CLI GPT-5.5) |
+| general | 84.9% | 13.3% | 0.3062 | terminal_bench/accuracy (Codex CLI GPT-5.5) |
+
+## Unmapped Models
+
+- o3-pro high from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- o3 high from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- grok-4 high from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- o3 high + gpt-4.1 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- o3 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- DeepSeek-V3.2-Exp Reasoner from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- Gemini 2.5 Pro Preview 03-25 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- claude-opus-4-20250514 32k thinking from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- o4-mini high from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- DeepSeek R1 0528 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- claude-opus-4-20250514 no think from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- DeepSeek-V3.2-Exp Chat from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- claude-3-7-sonnet-20250219 32k thinking tokens from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- DeepSeek R1 + claude-3-5-sonnet-20241022 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- o1-2024-12-17 high from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- claude-sonnet-4-20250514 32k thinking from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- claude-3-7-sonnet-20250219 no thinking from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- o3-mini high from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- Qwen3 235B A22B diff, no think, Alibaba API from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- Kimi K2 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- DeepSeek R1 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- claude-sonnet-4-20250514 no thinking from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- gemini-2.5-flash-preview-05-20 24k think from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- DeepSeek V3 0324 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- Quasar Alpha from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- o3-mini medium from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- Grok 3 Beta from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- Optimus Alpha from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- gpt-4.1 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- claude-3-5-sonnet-20241022 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- Grok 3 Mini Beta high from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- DeepSeek Chat V3 prev from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- gemini-2.5-flash-preview-04-17 default from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- chatgpt-4o-latest 2025-03-29 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- gpt-4.5-preview from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- gemini-2.5-flash-preview-05-20 no think from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- gpt-oss-120b high from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- Qwen3 32B from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- gemini-exp-1206 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- Gemini 2.0 Pro exp-02-05 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- Grok 3 Mini Beta low from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- o1-mini-2024-09-12 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- gpt-4.1-mini from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- claude-3-5-haiku-20241022 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- chatgpt-4o-latest 2025-02-15 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- QwQ-32B + Qwen 2.5 Coder Instruct from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- gpt-4o-2024-08-06 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- gemini-2.0-flash-exp from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- qwen-max-2025-01-25 from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- QwQ-32B from aider_polyglot in tools\scorecard\snapshots\live.aider_polyglot.json
+- ...19 more
+

package/tools/scorecard/generated/example/scorecard.compact.json

@@ -0,0 +1,61 @@
+{
+  "antigravity:gemini-3.1-pro": {
+    "bugfix": 0.7331,
+    "feature": 0.7424,
+    "refactor": 0.7574,
+    "test": 0.7364,
+    "docs": 0.7981,
+    "security": 0.7208,
+    "perf": 0.7254,
+    "general": 0.7231
+  },
+  "claude-code:opus": {
+    "bugfix": 0.7454,
+    "feature": 0.7454,
+    "refactor": 0.7454,
+    "test": 0.7454,
+    "security": 0.7454,
+    "perf": 0.7454,
+    "general": 0.7454
+  },
+  "claude-code:sonnet": {
+    "bugfix": 0.7098,
+    "feature": 0.6994,
+    "refactor": 0.7019,
+    "test": 0.6997,
+    "docs": 0.68,
+    "security": 0.7055,
+    "perf": 0.7022,
+    "general": 0.7036
+  },
+  "codex:gpt-5.4": {
+    "bugfix": 0.867,
+    "feature": 0.867,
+    "refactor": 0.867,
+    "test": 0.867,
+    "docs": 0.867,
+    "security": 0.867,
+    "perf": 0.867,
+    "general": 0.867
+  },
+  "codex:gpt-5.4-mini": {
+    "bugfix": 0.7738,
+    "feature": 0.7618,
+    "refactor": 0.7791,
+    "test": 0.7537,
+    "docs": 0.7746,
+    "security": 0.7673,
+    "perf": 0.7682,
+    "general": 0.7623
+  },
+  "codex:gpt-5.5": {
+    "bugfix": 0.8528,
+    "feature": 0.8566,
+    "refactor": 0.8627,
+    "test": 0.8541,
+    "docs": 0.88,
+    "security": 0.8478,
+    "perf": 0.8497,
+    "general": 0.8488
+  }
+}