dsh-super-code 0.3.9 → 0.3.11
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en.md +36 -11
- package/README.md +36 -11
- package/client/index.js +168 -64
- package/cordis.patch.yml +3 -7
- package/docs/Changelog.md +59 -0
- package/docs/Compatibility.md +51 -3
- package/docs/EvaluationResults.md +21 -5
- package/docs/UsageGuide.md +48 -9
- package/docs/en/Changelog.md +59 -0
- package/docs/en/Compatibility.md +51 -3
- package/docs/en/EvaluationResults.md +21 -5
- package/docs/en/UsageGuide.md +48 -9
- package/lib/connection/directories.d.ts.map +1 -1
- package/lib/connection/directories.js +14 -5
- package/lib/connection/directories.js.map +1 -1
- package/lib/connection/guidance.d.ts.map +1 -1
- package/lib/connection/guidance.js +7 -0
- package/lib/connection/guidance.js.map +1 -1
- package/lib/connection/http.d.ts.map +1 -1
- package/lib/connection/http.js +59 -30
- package/lib/connection/http.js.map +1 -1
- package/lib/connection/service.d.ts +3 -0
- package/lib/connection/service.d.ts.map +1 -1
- package/lib/connection/service.js +82 -33
- package/lib/connection/service.js.map +1 -1
- package/lib/connection/store.d.ts +2 -0
- package/lib/connection/store.d.ts.map +1 -1
- package/lib/connection/store.js +95 -30
- package/lib/connection/store.js.map +1 -1
- package/lib/connection/tasks.d.ts +6 -3
- package/lib/connection/tasks.d.ts.map +1 -1
- package/lib/connection/tasks.js +125 -53
- package/lib/connection/tasks.js.map +1 -1
- package/lib/connection/types.d.ts +1 -0
- package/lib/connection/types.d.ts.map +1 -1
- package/lib/connection/types.js.map +1 -1
- package/lib/core/deletion-command.d.ts.map +1 -1
- package/lib/core/deletion-command.js +84 -23
- package/lib/core/deletion-command.js.map +1 -1
- package/lib/core/windows-runtime.d.ts +16 -0
- package/lib/core/windows-runtime.d.ts.map +1 -0
- package/lib/core/windows-runtime.js +123 -0
- package/lib/core/windows-runtime.js.map +1 -0
- package/lib/dsh/budget-settings.d.ts +2 -0
- package/lib/dsh/budget-settings.d.ts.map +1 -1
- package/lib/dsh/budget-settings.js +9 -3
- package/lib/dsh/budget-settings.js.map +1 -1
- package/lib/dsh/budgets.d.ts +2 -0
- package/lib/dsh/budgets.d.ts.map +1 -1
- package/lib/dsh/budgets.js +6 -0
- package/lib/dsh/budgets.js.map +1 -1
- package/lib/dsh/compaction.d.ts.map +1 -1
- package/lib/dsh/compaction.js +1 -0
- package/lib/dsh/compaction.js.map +1 -1
- package/lib/dsh/connection-history.d.ts.map +1 -1
- package/lib/dsh/connection-history.js +3 -0
- package/lib/dsh/connection-history.js.map +1 -1
- package/lib/dsh/connection-host.js +1 -1
- package/lib/dsh/connection-host.js.map +1 -1
- package/lib/dsh/connection.d.ts.map +1 -1
- package/lib/dsh/connection.js +24 -3
- package/lib/dsh/connection.js.map +1 -1
- package/lib/dsh/deletion-protection.d.ts.map +1 -1
- package/lib/dsh/deletion-protection.js +18 -10
- package/lib/dsh/deletion-protection.js.map +1 -1
- package/lib/dsh/index.d.ts.map +1 -1
- package/lib/dsh/index.js +6 -0
- package/lib/dsh/index.js.map +1 -1
- package/lib/dsh/official-budgets.d.ts.map +1 -1
- package/lib/dsh/official-budgets.js +15 -3
- package/lib/dsh/official-budgets.js.map +1 -1
- package/lib/dsh/official-windows-shell.d.ts +4 -0
- package/lib/dsh/official-windows-shell.d.ts.map +1 -0
- package/lib/dsh/official-windows-shell.js +128 -0
- package/lib/dsh/official-windows-shell.js.map +1 -0
- package/lib/dsh/preset-declaration.d.ts +3 -0
- package/lib/dsh/preset-declaration.d.ts.map +1 -1
- package/lib/dsh/preset-declaration.js +68 -45
- package/lib/dsh/preset-declaration.js.map +1 -1
- package/lib/dsh/preset-installation.d.ts +10 -7
- package/lib/dsh/preset-installation.d.ts.map +1 -1
- package/lib/dsh/preset-installation.js +66 -30
- package/lib/dsh/preset-installation.js.map +1 -1
- package/lib/dsh/preset-metadata.d.ts.map +1 -1
- package/lib/dsh/preset-metadata.js +2 -1
- package/lib/dsh/preset-metadata.js.map +1 -1
- package/lib/dsh/preset-recovery-runtime.d.ts +76 -0
- package/lib/dsh/preset-recovery-runtime.d.ts.map +1 -0
- package/lib/dsh/preset-recovery-runtime.js +322 -0
- package/lib/dsh/preset-recovery-runtime.js.map +1 -0
- package/lib/dsh/preset-recovery.d.ts +14 -0
- package/lib/dsh/preset-recovery.d.ts.map +1 -0
- package/lib/dsh/preset-recovery.js +140 -0
- package/lib/dsh/preset-recovery.js.map +1 -0
- package/lib/dsh/ptc-budget.d.ts.map +1 -1
- package/lib/dsh/ptc-budget.js +20 -10
- package/lib/dsh/ptc-budget.js.map +1 -1
- package/lib/dsh/windows-shell.d.ts +53 -0
- package/lib/dsh/windows-shell.d.ts.map +1 -0
- package/lib/dsh/windows-shell.js +196 -0
- package/lib/dsh/windows-shell.js.map +1 -0
- package/lib/long-memory.d.ts.map +1 -1
- package/lib/long-memory.js +3 -2
- package/lib/long-memory.js.map +1 -1
- package/lib/super-agent-usage.d.ts +2 -0
- package/lib/super-agent-usage.d.ts.map +1 -1
- package/lib/super-agent-usage.js +16 -11
- package/lib/super-agent-usage.js.map +1 -1
- package/lib/task-feedback-projection.d.ts +11 -0
- package/lib/task-feedback-projection.d.ts.map +1 -1
- package/lib/task-feedback-projection.js +37 -12
- package/lib/task-feedback-projection.js.map +1 -1
- package/package.json +21 -16
- package/presets/super-code/agent.cordis.yml +3 -7
- package/runtime/windows/README.md +45 -0
- package/runtime/windows/arm64-busybox.exe.gz +0 -0
- package/runtime/windows/bash.exe.gz +0 -0
- package/runtime/windows/grep.exe.gz +0 -0
- package/runtime/windows/licenses/BusyBox-LICENSE.txt +348 -0
- package/runtime/windows/licenses/GPL-3.0.txt +674 -0
- package/runtime/windows/licenses/LGPL-3.0.txt +165 -0
- package/runtime/windows/licenses/MSYS-COPYING.LIB +165 -0
- package/runtime/windows/licenses/MSYS-COPYING.RUNTIME +674 -0
- package/runtime/windows/licenses/PCRE-LICENCE.txt +93 -0
- package/runtime/windows/licenses/ncurses-COPYING.txt +29 -0
- package/runtime/windows/manifest.json +236 -0
- package/runtime/windows/msys-2.0.dll.gz +0 -0
- package/runtime/windows/msys-iconv-2.dll.gz +0 -0
- package/runtime/windows/msys-intl-8.dll.gz +0 -0
- package/runtime/windows/msys-pcre-1.dll.gz +0 -0
- package/runtime/windows/sources/busybox-w32-FRP-6354-g5f09fb446.tgz +0 -0
- package/runtime/windows/sources.json +45 -0
- package/runtime/windows/x64-busybox.exe.gz +0 -0
package/README.en.md
CHANGED
|
@@ -17,7 +17,7 @@ Tested on version `0.3.5`. [Full results and limitations](docs/en/EvaluationResu
|
|
|
17
17
|
|
|
18
18
|
## Install
|
|
19
19
|
|
|
20
|
-
Supports Harness from `0.1.5-alpha.1
|
|
20
|
+
Supports Harness from `0.1.5-alpha.1`; see [compatibility](docs/en/Compatibility.md) for tested Web, official Desktop, and NEXT versions.
|
|
21
21
|
|
|
22
22
|
**Desktop**: open **Plugins → Add plugin** in the left sidebar, enter `dsh-super-code`, then click **Install** and **Enable now** (on older versions, reload or restart when prompted). Desktop includes its own runtime; Node.js is not required.
|
|
23
23
|
|
|
@@ -40,11 +40,19 @@ Desktop: enter the full path of the package in "Package name or address", then c
|
|
|
40
40
|
Web: run the following command (use the actual file name):
|
|
41
41
|
|
|
42
42
|
```bash
|
|
43
|
-
dsh plugin --profile web add ./dsh-super-code-0.3.
|
|
43
|
+
dsh plugin --profile web add ./dsh-super-code-0.3.11.tgz
|
|
44
44
|
```
|
|
45
45
|
|
|
46
46
|
</details>
|
|
47
47
|
|
|
48
|
+
### Windows commands
|
|
49
|
+
|
|
50
|
+
Bundles GNU Bash, the required MSYS DLLs, BusyBox, and GNU grep, so Git does not need to be installed just to run Bash commands. Developer tools such as Git, Node, and Python still need separate installation. macOS and Linux keep the host environment.
|
|
51
|
+
|
|
52
|
+
Under **Settings → Super Code → Preset configuration → Command environment**, bundled Bash defaults on and **Also apply to official presets** defaults off. Enabling the latter covers Standard, PTC, and Create; Minimal retains its persistent terminal. Switches are editable only on Windows hosts, and are gray with a hover tooltip on other platforms.
|
|
53
|
+
|
|
54
|
+
In confined environments, Bash commands require DSH approval for that command. See the [usage guide](docs/en/UsageGuide.md#windows-command-environment) for configuration and compatibility limits.
|
|
55
|
+
|
|
48
56
|
## Usage
|
|
49
57
|
|
|
50
58
|
1. Open **Settings → Agent Presets** and select **Super Code** (identifier `super-code`).
|
|
@@ -62,7 +70,7 @@ Select **Super Code** in the right sidebar. It contains three sections:
|
|
|
62
70
|
- **Collaborative execution**: the Agent execution tree. Click a node to view details.
|
|
63
71
|
- **Memory**: view, add, edit, and delete project memory and global preferences.
|
|
64
72
|
|
|
65
|
-
|
|
73
|
+
Exactly one of the three sections stays open. Tasks opens by default; clicking its heading keeps it expanded.
|
|
66
74
|
|
|
67
75
|
[](docs/images/task-status.png)
|
|
68
76
|
|
|
@@ -78,6 +86,7 @@ Screenshots use the current UI with sample data to illustrate the plugin.
|
|
|
78
86
|
| Layered memory | Current task, project lessons, and global preferences are stored separately, each with its source |
|
|
79
87
|
| Runtime budgets | Uses the better-performing legacy compression budgets by default; adjustable in Settings |
|
|
80
88
|
| Deletion protection | Directs Agents to use recoverable `trash` by default; can switch to strict confirmation |
|
|
89
|
+
| Windows commands | Bundled Bash and common utilities, optionally applied to official presets |
|
|
81
90
|
| MCP connection | Allows external MCP clients to delegate tasks to DSH. Off by default |
|
|
82
91
|
| Frontend Skill | Loaded on demand for UI work to check alignment, hit areas, and browser behavior |
|
|
83
92
|
|
|
@@ -85,9 +94,9 @@ See the [usage guide](docs/en/UsageGuide.md).
|
|
|
85
94
|
|
|
86
95
|
## Recent releases
|
|
87
96
|
|
|
97
|
+
- **0.3.11**: Fixed numerous bugs.
|
|
98
|
+
- **0.3.10**: Added bundled Bash on Windows and an official-preset switch, with legacy-host compatibility and tooltip fixes.
|
|
88
99
|
- **0.3.9**: Polished Settings and the task sidebar, improved MCP conversation control, fixed PTC child-tool approvals, and verified rc.4 Web compatibility.
|
|
89
|
-
- **0.3.8**: Added MCP connection; external clients can delegate, inspect, continue, and cancel DSH conversations. Added a Home overview in Settings.
|
|
90
|
-
- **0.3.7**: Added three-level deletion protection (Relaxed by default). Added Task status to the sidebar.
|
|
91
100
|
|
|
92
101
|
[Full changelog](docs/en/Changelog.md)
|
|
93
102
|
|
|
@@ -95,21 +104,37 @@ Versions follow `major.performance.maintenance`: a change in the second number i
|
|
|
95
104
|
|
|
96
105
|
## Evaluation
|
|
97
106
|
|
|
107
|
+
### Plugin comparison
|
|
108
|
+
|
|
109
|
+
Full SWE-bench Pro hard-100 results, sorted by tasks passed. All runs use Harness `0.2.1-rc.2`, `deepseek-v4.1-flash-1017`, high reasoning effort, and legacy budgets. Each group reuses its environment continuously across tasks.
|
|
110
|
+
|
|
111
|
+
| Configuration | Passed/valid | Mean total tokens (M) | Mean time (min) | Mean model calls | Mean tool calls | Cache rate |
|
|
112
|
+
|---|---:|---:|---:|---:|---:|---:|
|
|
113
|
+
| Super Code 0.3.5 | **51/100** | 6.435 | 24.751 | 76.660 | 75.840 | 98.927% |
|
|
114
|
+
| Official Standard | 50/100 | 8.030 | 17.198 | 91.990 | 113.880 | 99.167% |
|
|
115
|
+
| Unmodified dsh-ops 0.4.4 · Standard | 49/100 | 7.099 | 23.132 | 82.500 | 102.360 | 99.062% |
|
|
116
|
+
| Official Standard + billion-context 0.1.209 | 45/100 | 6.359 | 18.471 | 108.950 | 131.140 | 97.654% |
|
|
117
|
+
| Unmodified dsh-ops 0.4.4 · PTC | 44/100 | 7.322 | 23.695 | 81.490 | 80.960 | 99.056% |
|
|
118
|
+
|
|
119
|
+
M means one million tokens, including reasoning tokens. Time includes some in-run service waits. These additional results are recorded separately from the opening comparison and the historical budget experiments below. [Full results and methodology](docs/en/EvaluationResults.md#pro100-continuous-environment-comparison)
|
|
120
|
+
|
|
98
121
|
### Method
|
|
99
122
|
|
|
100
123
|
- **Task set**: SWE-bench Pro hard-100, graded with the official grader. Tuning uses only 33 hard tasks selected from the SWE-bench dev split; hard-100 is used only to evaluate frozen candidates and investigate issues, never for tuning, to avoid overfitting to the evaluation tasks.
|
|
101
|
-
- **
|
|
102
|
-
- **Controlled variables**: same host
|
|
103
|
-
- **Metrics**: tasks passed,
|
|
124
|
+
- **Baselines**: the official Standard preset and third-party plugin configurations. In the historical official-preset comparison on the old host, Standard + high passed 51 tasks and Minimal + high passed 46, with Standard using 23.5% fewer tokens per task, so Standard was selected as the baseline.
|
|
125
|
+
- **Controlled variables**: comparisons on the same host keep model, reasoning effort, and budgets fixed while comparing preset and plugin combinations. Cross-host and cross-model experiments are identified separately.
|
|
126
|
+
- **Metrics**: tasks passed, mean total tokens, time, model and tool calls, and cache rate. The plugin comparison table above includes some in-run service waits; the historical budget experiments below use net time, excluding failed requests and retry waits.
|
|
104
127
|
|
|
105
|
-
###
|
|
128
|
+
### Historical process
|
|
106
129
|
|
|
107
130
|
1. **Old-host baseline** (DSH `0.1.5-rc.2`, `deepseek-v4.1-flash`, high): the Super Code `0.3.0` candidate passed 56 tasks, the official Standard preset 51, and Minimal 46. Relative to Minimal, time fell by 47.6% and tokens by 44.9%.
|
|
108
131
|
2. **Regression after the host upgrade**: after upgrading to DSH `0.2.1-rc.2` and `deepseek-v4.1-flash-1017`, under default budgets Standard dropped to 46 and Super Code `0.3.4` to 41, below Standard. Investigation found the new host had tightened context budgets: the compression threshold fell from 800,000 to 550,464 tokens, and retained verbatim context from 160,000 to 98,560 tokens.
|
|
109
132
|
3. **Retest with legacy budgets**: both sides used legacy budgets. Standard stayed at 46 with 14.2% more tokens; Super Code `0.3.5` rose to 49 with only 0.4% more tokens and 19.4% less net time. Based on this, the plugin uses legacy budgets by default from `0.3.6`.
|
|
110
133
|
4. **Cross-model check** (`glm-5.3-flash`, max): Super Code `0.3.4` passed 45 tasks and native ZCode 29. Super Code used less net time per task (32.9 vs 52.3 min) and more tokens (5.50 M vs 3.61 M).
|
|
111
134
|
|
|
112
|
-
###
|
|
135
|
+
### Historical summary
|
|
136
|
+
|
|
137
|
+
These are earlier evaluation runs and are not pooled with the plugin comparison results above.
|
|
113
138
|
|
|
114
139
|
| Comparison | Passed | Efficiency |
|
|
115
140
|
|---|---|---|
|
|
@@ -123,7 +148,7 @@ Versions follow `major.performance.maintenance`: a change in the second number i
|
|
|
123
148
|
|
|
124
149
|
- Each configuration was run once. The same task can pass in one run and fail in another; run-to-run variance for a fixed configuration has not been estimated.
|
|
125
150
|
- The default-budget and legacy-budget runs used different plugin versions (`0.3.4` → `0.3.5`), so the gain cannot be attributed entirely to budgets.
|
|
126
|
-
- `0.3.6` – `0.3.
|
|
151
|
+
- `0.3.6` – `0.3.11` have not been re-evaluated on the 100 tasks.
|
|
127
152
|
- The cross-model comparison is between two complete systems and cannot isolate the plugin's contribution.
|
|
128
153
|
|
|
129
154
|
[Results and methodology](docs/en/EvaluationResults.md) · [All result tables](eval/results/swebench-pro-final-100/ResultsTables.md)
|
package/README.md
CHANGED
|
@@ -17,7 +17,7 @@
|
|
|
17
17
|
|
|
18
18
|
## 安装
|
|
19
19
|
|
|
20
|
-
支持 Harness `0.1.5-alpha.1`
|
|
20
|
+
支持 Harness `0.1.5-alpha.1` 起的版本;Web、官方桌面版和 NEXT 的实测范围见[兼容性记录](docs/Compatibility.md)。
|
|
21
21
|
|
|
22
22
|
**桌面版**:在左侧打开「插件 → 添加插件」,输入 `dsh-super-code`,依次点击「安装」和「立即启用」(旧版本请按提示重载或重启)。桌面版内置运行环境,无需安装 Node.js。
|
|
23
23
|
|
|
@@ -40,11 +40,19 @@ dsh plugin --profile web add dsh-super-code
|
|
|
40
40
|
Web 版:运行以下命令(文件名以实际下载的为准):
|
|
41
41
|
|
|
42
42
|
```bash
|
|
43
|
-
dsh plugin --profile web add ./dsh-super-code-0.3.
|
|
43
|
+
dsh plugin --profile web add ./dsh-super-code-0.3.11.tgz
|
|
44
44
|
```
|
|
45
45
|
|
|
46
46
|
</details>
|
|
47
47
|
|
|
48
|
+
### Windows 命令支持
|
|
49
|
+
|
|
50
|
+
内置 GNU Bash、必要的 MSYS DLL、BusyBox 和 GNU grep,无需为了运行 Bash 命令额外安装 Git。Git、Node、Python 等开发工具仍需自行安装;macOS、Linux 沿用宿主环境。
|
|
51
|
+
|
|
52
|
+
在「设置 → Super Code 模式 → 预设配置 → 命令环境」中,内置 Bash 默认开启,「同时应用到官方预设」默认关闭。开启后可用于官方标准、PTC、创建模式;极简模式保留持久终端。仅 Windows 宿主可修改开关,其他平台显示为灰色并提供悬浮提示。
|
|
53
|
+
|
|
54
|
+
受限环境下,Bash 命令需要通过 DSH 对该条命令的审批。配置和兼容边界见[使用说明](docs/UsageGuide.md#windows-命令环境)。
|
|
55
|
+
|
|
48
56
|
## 使用
|
|
49
57
|
|
|
50
58
|
1. 打开「设置 → Agent 预设」,选择 **Super Code 模式**(标识符 `super-code`)。
|
|
@@ -62,7 +70,7 @@ dsh plugin --profile web add ./dsh-super-code-0.3.9.tgz
|
|
|
62
70
|
- **协作执行**:Agent 执行树,点击节点可查看详情。
|
|
63
71
|
- **记忆**:查看、新增、编辑和删除项目记忆与全局偏好。
|
|
64
72
|
|
|
65
|
-
|
|
73
|
+
三个部分始终只展开一项,默认显示任务;再次点击当前标题会保持展开。
|
|
66
74
|
|
|
67
75
|
[](docs/images/task-status.png)
|
|
68
76
|
|
|
@@ -78,6 +86,7 @@ dsh plugin --profile web add ./dsh-super-code-0.3.9.tgz
|
|
|
78
86
|
| 分层记忆 | 当前任务、项目经验和全局偏好分别存储,每条记录均标注来源 |
|
|
79
87
|
| 运行预算 | 默认使用效果更好的旧压缩预算,可在设置中调整 |
|
|
80
88
|
| 删除保护 | 默认引导 Agent 使用可恢复的 `trash`,可切换为严格确认 |
|
|
89
|
+
| Windows 命令环境 | 内置 Bash 与常用命令,可选应用到官方预设 |
|
|
81
90
|
| MCP 连接 | 允许外部 MCP 客户端将任务委派给 DSH 执行,默认关闭 |
|
|
82
91
|
| 前端 Skill | 开发页面时按需加载,检查对齐、点击区域和浏览器表现 |
|
|
83
92
|
|
|
@@ -85,9 +94,9 @@ dsh plugin --profile web add ./dsh-super-code-0.3.9.tgz
|
|
|
85
94
|
|
|
86
95
|
## 最近更新
|
|
87
96
|
|
|
97
|
+
- **0.3.11**:修复了大量 bug。
|
|
98
|
+
- **0.3.10**:新增 Windows 内置 Bash 和官方预设开关,修复旧宿主兼容与提示气泡样式。
|
|
88
99
|
- **0.3.9**:美化设置和任务侧栏,完善 MCP 对话控制并修复 PTC 子工具审批,验证 rc.4 Web 兼容性。
|
|
89
|
-
- **0.3.8**:新增 MCP 连接,外部客户端可委派、查看、继续和取消 DSH 对话;设置页新增首页概览。
|
|
90
|
-
- **0.3.7**:新增三档删除保护(默认宽松);侧边栏新增任务状态。
|
|
91
100
|
|
|
92
101
|
[完整版本记录](docs/Changelog.md)
|
|
93
102
|
|
|
@@ -95,21 +104,37 @@ dsh plugin --profile web add ./dsh-super-code-0.3.9.tgz
|
|
|
95
104
|
|
|
96
105
|
## 评测
|
|
97
106
|
|
|
107
|
+
### 插件方案对比
|
|
108
|
+
|
|
109
|
+
SWE-bench Pro hard-100 完整成绩,按通过数量降序排列。统一使用 Harness `0.2.1-rc.2`、`deepseek-v4.1-flash-1017`、high、旧预算,各组内部连续复用环境。
|
|
110
|
+
|
|
111
|
+
| 方案 | 通过/有效 | 平均总 Token(M) | 平均耗时(分钟) | 平均模型调用 | 平均工具调用 | 缓存率 |
|
|
112
|
+
|---|---:|---:|---:|---:|---:|---:|
|
|
113
|
+
| Super Code 0.3.5 | **51/100** | 6.435 | 24.751 | 76.660 | 75.840 | 98.927% |
|
|
114
|
+
| 官方标准 | 50/100 | 8.030 | 17.198 | 91.990 | 113.880 | 99.167% |
|
|
115
|
+
| 原版 dsh-ops 0.4.4 · 标准 | 49/100 | 7.099 | 23.132 | 82.500 | 102.360 | 99.062% |
|
|
116
|
+
| 官方标准+billion-context 0.1.209 | 45/100 | 6.359 | 18.471 | 108.950 | 131.140 | 97.654% |
|
|
117
|
+
| 原版 dsh-ops 0.4.4 · PTC | 44/100 | 7.322 | 23.695 | 81.490 | 80.960 | 99.056% |
|
|
118
|
+
|
|
119
|
+
M 为百万 Token,包含思考 Token;耗时包含部分运行内服务等待。这批追加成绩与开头及下方历史预算实验分开记录。[完整结果与口径](docs/EvaluationResults.md)
|
|
120
|
+
|
|
98
121
|
### 评测方法
|
|
99
122
|
|
|
100
123
|
- **题集**:SWE-bench Pro hard-100,使用官方评分器。调优仅使用 SWE-bench dev 中筛选的 33 道困难题;hard-100 只用于评测冻结后的候选版本和排查问题,不参与调优,以避免对评测题过拟合。
|
|
101
|
-
-
|
|
102
|
-
-
|
|
103
|
-
-
|
|
124
|
+
- **对照**:官方标准预设及第三方插件方案。历史官方预设对比中,旧宿主的标准 + high 通过 51 题,minimal + high 通过 46 题,且标准每题 token 少 23.5%,因此选用标准预设作为基线。
|
|
125
|
+
- **控制变量**:同宿主对比保持模型、思考等级和预算一致,对比不同预设及插件组合;跨宿主、跨模型的实验单独说明。
|
|
126
|
+
- **指标**:通过数、平均总 Token、耗时、模型与工具调用次数、缓存率。上述插件方案对比表的耗时包含部分运行内服务等待;下方历史预算实验使用净耗时,扣除失败请求和重试等待。
|
|
104
127
|
|
|
105
|
-
###
|
|
128
|
+
### 历史评测过程
|
|
106
129
|
|
|
107
130
|
1. **旧宿主基线**(DSH `0.1.5-rc.2`,`deepseek-v4.1-flash`,high):Super Code `0.3.0` 候选通过 56 题,官方标准预设 51 题,minimal 46 题。相对 minimal,耗时减少 47.6%,token 减少 44.9%。
|
|
108
131
|
2. **宿主升级后效果下降**:升级到 DSH `0.2.1-rc.2` 和 `deepseek-v4.1-flash-1017` 后,在默认预算下标准降至 46 题,Super Code `0.3.4` 降至 41 题,低于标准。排查发现新宿主收紧了上下文预算:压缩阈值由 800,000 降至 550,464 token,保留原文由 160,000 降至 98,560 token。
|
|
109
132
|
3. **恢复旧预算后复测**:双方均使用旧预算。标准仍为 46 题,token 增加 14.2%;Super Code `0.3.5` 提升至 49 题,token 仅增加 0.4%,净耗时减少 19.4%。据此,`0.3.6` 起插件默认使用旧预算。
|
|
110
133
|
4. **跨模型验证**(`glm-5.3-flash`,max):Super Code `0.3.4` 通过 45 题,原生 ZCode 通过 29 题;Super Code 每题净耗时更少(32.9 vs 52.3 分钟),token 更多(550 万 vs 361 万)。
|
|
111
134
|
|
|
112
|
-
###
|
|
135
|
+
### 历史结果汇总
|
|
136
|
+
|
|
137
|
+
以下为较早的评测批次,不与上述插件方案对比的成绩合并计算。
|
|
113
138
|
|
|
114
139
|
| 对比 | 通过 | 效率 |
|
|
115
140
|
|---|---|---|
|
|
@@ -123,7 +148,7 @@ dsh plugin --profile web add ./dsh-super-code-0.3.9.tgz
|
|
|
123
148
|
|
|
124
149
|
- 每组配置只运行一轮。同一道题在不同轮次中结果可能不同,目前尚未估计固定配置下的随机波动。
|
|
125
150
|
- 默认预算与旧预算两组的插件版本不同(`0.3.4` → `0.3.5`),收益不能全部归因于预算。
|
|
126
|
-
- `0.3.6` ~ `0.3.
|
|
151
|
+
- `0.3.6` ~ `0.3.11` 未重新运行 100 题评测。
|
|
127
152
|
- 跨模型对比是两套完整系统之间的比较,无法单独分离插件的贡献。
|
|
128
153
|
|
|
129
154
|
[评测结果与口径](docs/EvaluationResults.md) · [完整成绩表](eval/results/swebench-pro-final-100/ResultsTables.md)
|