@mengruo/dsh-vision-toolkit 0.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (216) hide show
  1. package/LICENSE +21 -0
  2. package/README.i18n.yaml +6 -0
  3. package/README.md +356 -0
  4. package/README.zh.md +354 -0
  5. package/assets/community-group-qr.png +0 -0
  6. package/assets/dsh-conversation-artifact.png +0 -0
  7. package/assets/dsh-conversation-image-qa-top.png +0 -0
  8. package/assets/dsh-conversation-image-qa.png +0 -0
  9. package/assets/dsh-conversation-pixel-diff.png +0 -0
  10. package/assets/dsh-conversation-screenshot-debugging-top.png +0 -0
  11. package/assets/dsh-conversation-screenshot-debugging.png +0 -0
  12. package/assets/dsh-conversation-tool-call.png +0 -0
  13. package/assets/dsh-conversation-vision-trace.png +0 -0
  14. package/assets/dsh-view-example.png +0 -0
  15. package/assets/hero-v2.png +0 -0
  16. package/assets/logo_aihubmix.png +0 -0
  17. package/assets/python-bootstrap.json +48 -0
  18. package/assets/skill/SKILL.md +329 -0
  19. package/assets/skill/UPSTREAM.json +71 -0
  20. package/assets/skill/references/gui.md +88 -0
  21. package/assets/skill/references/long-screenshot-ocr.md +77 -0
  22. package/assets/skill/references/restore-graphic.md +84 -0
  23. package/assets/skill/references/restore-structure.md +45 -0
  24. package/assets/skill/references/restore-ui.md +202 -0
  25. package/assets/social-preview.png +0 -0
  26. package/assets/upstream/README.md +18 -0
  27. package/assets/upstream/focus-hint-comparison-1.webp +0 -0
  28. package/assets/upstream/focus-hint-comparison-2.webp +0 -0
  29. package/assets/upstream/infographic-reference.webp +0 -0
  30. package/assets/upstream/infographic-result.webp +0 -0
  31. package/assets/upstream/ui-fast-restore-reference.webp +0 -0
  32. package/assets/upstream/ui-fast-restore-result.webp +0 -0
  33. package/assets/upstream/ui-result.webp +0 -0
  34. package/assets/upstream/ui-sketch.webp +0 -0
  35. package/assets/vision-model-test.png +0 -0
  36. package/assets/vision-settings.png +0 -0
  37. package/assets/wechat-reward.png +0 -0
  38. package/cordis.patch.yml +6 -0
  39. package/docs/aihubmix-gemini-vision.i18n.yaml +6 -0
  40. package/docs/aihubmix-gemini-vision.md +138 -0
  41. package/docs/aihubmix-gemini-vision.zh.md +138 -0
  42. package/docs/assets/aihubmix-api-keys.png +0 -0
  43. package/docs/assets/aihubmix-create-key.png +0 -0
  44. package/docs/assets/aihubmix-free-vision-model.png +0 -0
  45. package/docs/assets/aihubmix-home.png +0 -0
  46. package/docs/assets/aihubmix-sign-up.png +0 -0
  47. package/docs/assets/groq-console-home.png +0 -0
  48. package/docs/assets/groq-console-keys-login.png +0 -0
  49. package/docs/assets/groq-docs-quickstart.png +0 -0
  50. package/docs/assets/groq-docs-qwen3.6.png +0 -0
  51. package/docs/assets/groq-docs-vision.png +0 -0
  52. package/docs/assets/vision-settings.png +0 -0
  53. package/docs/dsh-desktop-install.md +67 -0
  54. package/docs/dsh-desktop-install.zh.md +67 -0
  55. package/docs/groq-qwen3.6-vision.i18n.yaml +6 -0
  56. package/docs/groq-qwen3.6-vision.md +236 -0
  57. package/docs/groq-qwen3.6-vision.zh.md +236 -0
  58. package/docs/python-runtime.i18n.yaml +6 -0
  59. package/docs/python-runtime.md +89 -0
  60. package/docs/python-runtime.zh.md +89 -0
  61. package/docs/requirements-traceability/README.i18n.yaml +6 -0
  62. package/docs/requirements-traceability/README.md +75 -0
  63. package/docs/requirements-traceability/README.zh.md +75 -0
  64. package/examples/ui-restoration/README.i18n.yaml +6 -0
  65. package/examples/ui-restoration/README.md +70 -0
  66. package/examples/ui-restoration/README.zh.md +70 -0
  67. package/examples/ui-restoration/assets/final-heatmap.png +0 -0
  68. package/examples/ui-restoration/assets/final-report.json +83 -0
  69. package/examples/ui-restoration/assets/implementation.png +0 -0
  70. package/examples/ui-restoration/assets/initial-heatmap.png +0 -0
  71. package/examples/ui-restoration/assets/initial-report.json +83 -0
  72. package/examples/ui-restoration/assets/initial.png +0 -0
  73. package/examples/ui-restoration/assets/metrics.json +12 -0
  74. package/examples/ui-restoration/assets/reference.png +0 -0
  75. package/examples/ui-restoration/implementation.html +94 -0
  76. package/examples/ui-restoration/initial.html +57 -0
  77. package/lib/artifact-access.js +369 -0
  78. package/lib/artifact-access.js.map +1 -0
  79. package/lib/artifacts.js +56 -0
  80. package/lib/artifacts.js.map +1 -0
  81. package/lib/client.js +2099 -0
  82. package/lib/client.js.map +1 -0
  83. package/lib/config.js +172 -0
  84. package/lib/config.js.map +1 -0
  85. package/lib/defaults.js +6 -0
  86. package/lib/defaults.js.map +1 -0
  87. package/lib/errors.js +56 -0
  88. package/lib/errors.js.map +1 -0
  89. package/lib/evidence-cache.js +350 -0
  90. package/lib/evidence-cache.js.map +1 -0
  91. package/lib/exposure.js +247 -0
  92. package/lib/exposure.js.map +1 -0
  93. package/lib/image-input-variants.js +882 -0
  94. package/lib/image-input-variants.js.map +1 -0
  95. package/lib/index.js +105 -0
  96. package/lib/index.js.map +1 -0
  97. package/lib/paste-images.js +217 -0
  98. package/lib/paste-images.js.map +1 -0
  99. package/lib/paths.js +354 -0
  100. package/lib/paths.js.map +1 -0
  101. package/lib/plugin-update.js +1003 -0
  102. package/lib/plugin-update.js.map +1 -0
  103. package/lib/runtime-install.js +964 -0
  104. package/lib/runtime-install.js.map +1 -0
  105. package/lib/runtime-manager.js +133 -0
  106. package/lib/runtime-manager.js.map +1 -0
  107. package/lib/runtime.js +1678 -0
  108. package/lib/runtime.js.map +1 -0
  109. package/lib/skill.js +25 -0
  110. package/lib/skill.js.map +1 -0
  111. package/lib/tools.js +549 -0
  112. package/lib/tools.js.map +1 -0
  113. package/lib/types/artifact-access.d.ts +61 -0
  114. package/lib/types/artifact-access.d.ts.map +1 -0
  115. package/lib/types/artifacts.d.ts +42 -0
  116. package/lib/types/artifacts.d.ts.map +1 -0
  117. package/lib/types/client/display-config.d.ts +24 -0
  118. package/lib/types/client/display-config.d.ts.map +1 -0
  119. package/lib/types/client/index.d.ts +350 -0
  120. package/lib/types/client/index.d.ts.map +1 -0
  121. package/lib/types/client/model-variants-hider.d.ts +40 -0
  122. package/lib/types/client/model-variants-hider.d.ts.map +1 -0
  123. package/lib/types/client/paste-images.d.ts +126 -0
  124. package/lib/types/client/paste-images.d.ts.map +1 -0
  125. package/lib/types/config.d.ts +123 -0
  126. package/lib/types/config.d.ts.map +1 -0
  127. package/lib/types/defaults.d.ts +6 -0
  128. package/lib/types/defaults.d.ts.map +1 -0
  129. package/lib/types/errors.d.ts +35 -0
  130. package/lib/types/errors.d.ts.map +1 -0
  131. package/lib/types/evidence-cache.d.ts +91 -0
  132. package/lib/types/evidence-cache.d.ts.map +1 -0
  133. package/lib/types/exposure.d.ts +50 -0
  134. package/lib/types/exposure.d.ts.map +1 -0
  135. package/lib/types/image-input-variants.d.ts +144 -0
  136. package/lib/types/image-input-variants.d.ts.map +1 -0
  137. package/lib/types/index.d.ts +19 -0
  138. package/lib/types/index.d.ts.map +1 -0
  139. package/lib/types/paste-images.d.ts +73 -0
  140. package/lib/types/paste-images.d.ts.map +1 -0
  141. package/lib/types/paths.d.ts +117 -0
  142. package/lib/types/paths.d.ts.map +1 -0
  143. package/lib/types/plugin-update.d.ts +119 -0
  144. package/lib/types/plugin-update.d.ts.map +1 -0
  145. package/lib/types/runtime-install.d.ts +92 -0
  146. package/lib/types/runtime-install.d.ts.map +1 -0
  147. package/lib/types/runtime-manager.d.ts +60 -0
  148. package/lib/types/runtime-manager.d.ts.map +1 -0
  149. package/lib/types/runtime.d.ts +412 -0
  150. package/lib/types/runtime.d.ts.map +1 -0
  151. package/lib/types/skill.d.ts +14 -0
  152. package/lib/types/skill.d.ts.map +1 -0
  153. package/lib/types/tools.d.ts +35 -0
  154. package/lib/types/tools.d.ts.map +1 -0
  155. package/lib/types/upstream.d.ts +233 -0
  156. package/lib/types/upstream.d.ts.map +1 -0
  157. package/lib/types/version.d.ts +15 -0
  158. package/lib/types/version.d.ts.map +1 -0
  159. package/lib/types/web-request.d.ts +11 -0
  160. package/lib/types/web-request.d.ts.map +1 -0
  161. package/lib/types/web.d.ts +118 -0
  162. package/lib/types/web.d.ts.map +1 -0
  163. package/lib/upstream.js +860 -0
  164. package/lib/upstream.js.map +1 -0
  165. package/lib/version.js +18 -0
  166. package/lib/version.js.map +1 -0
  167. package/lib/web-request.js +29 -0
  168. package/lib/web-request.js.map +1 -0
  169. package/lib/web.js +462 -0
  170. package/lib/web.js.map +1 -0
  171. package/package.json +149 -0
  172. package/patches/vision-tools-dsh.patch +970 -0
  173. package/runtime/requirements.lock +3 -0
  174. package/src/artifact-access.ts +386 -0
  175. package/src/artifacts.ts +85 -0
  176. package/src/client/display-config.ts +62 -0
  177. package/src/client/index.tsx +1580 -0
  178. package/src/client/model-variants-hider.ts +159 -0
  179. package/src/client/paste-images.tsx +747 -0
  180. package/src/config.ts +290 -0
  181. package/src/defaults.ts +5 -0
  182. package/src/errors.ts +62 -0
  183. package/src/evidence-cache.ts +418 -0
  184. package/src/exposure.ts +265 -0
  185. package/src/image-input-variants.ts +996 -0
  186. package/src/index.ts +141 -0
  187. package/src/paste-images.ts +286 -0
  188. package/src/paths.ts +383 -0
  189. package/src/plugin-update.ts +1143 -0
  190. package/src/runtime-install.ts +1161 -0
  191. package/src/runtime-manager.ts +174 -0
  192. package/src/runtime.ts +2189 -0
  193. package/src/skill.ts +35 -0
  194. package/src/tools.ts +691 -0
  195. package/src/upstream.ts +1078 -0
  196. package/src/version.ts +37 -0
  197. package/src/web-request.ts +27 -0
  198. package/src/web.ts +594 -0
  199. package/vendor/agent-vision-toolkit/CHANGELOG.md +30 -0
  200. package/vendor/agent-vision-toolkit/LICENSE +21 -0
  201. package/vendor/agent-vision-toolkit/README.md +405 -0
  202. package/vendor/agent-vision-toolkit/UPSTREAM_MANIFEST.json +94 -0
  203. package/vendor/agent-vision-toolkit/bin/crop +90 -0
  204. package/vendor/agent-vision-toolkit/bin/detect +13 -0
  205. package/vendor/agent-vision-toolkit/bin/glance +93 -0
  206. package/vendor/agent-vision-toolkit/bin/ground +13 -0
  207. package/vendor/agent-vision-toolkit/bin/trace +129 -0
  208. package/vendor/agent-vision-toolkit/detect.py +60 -0
  209. package/vendor/agent-vision-toolkit/ground.py +267 -0
  210. package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/dominant_colors.py +224 -0
  211. package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/extract_fg.py +278 -0
  212. package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/html_shot.py +420 -0
  213. package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/long_screenshot_ocr.py +1245 -0
  214. package/vendor/agent-vision-toolkit/skills/vision-tools/scripts/pixel_diff.py +88 -0
  215. package/vendor/agent-vision-toolkit/tests/test_vision_client.py +450 -0
  216. package/vendor/agent-vision-toolkit/vision_client.py +310 -0
package/src/version.ts ADDED
@@ -0,0 +1,37 @@
1
+ /**
2
+ * Plugin and pinned upstream version facts. The upstream snapshot is fixed at
3
+ * build time and recorded in package.json's `dsh.visionToolkit` metadata so
4
+ * the runtime, tool results, and docs all report the same source commit.
5
+ * @module dsh-vision-toolkit/version
6
+ */
7
+
8
+ import { readFileSync } from 'node:fs'
9
+
10
+ interface VisionToolkitPackageMetadata {
11
+ version: string
12
+ dsh?: {
13
+ visionToolkit?: {
14
+ upstreamRepository?: string
15
+ upstreamVersion?: string
16
+ upstreamCommit?: string
17
+ }
18
+ }
19
+ }
20
+
21
+ const metadata = JSON.parse(
22
+ readFileSync(new URL('../package.json', import.meta.url), 'utf8'),
23
+ ) as VisionToolkitPackageMetadata
24
+
25
+ const visionToolkit = metadata.dsh?.visionToolkit
26
+
27
+ /** Plugin package version. */
28
+ export const PLUGIN_VERSION = metadata.version
29
+
30
+ /** Pinned upstream repository URL. */
31
+ export const UPSTREAM_REPOSITORY = visionToolkit?.upstreamRepository ?? 'https://github.com/Anionex/agent-vision-toolkit'
32
+
33
+ /** Pinned upstream release tag. */
34
+ export const UPSTREAM_VERSION = visionToolkit?.upstreamVersion ?? 'v0.1.0+snapshot.8ccf9d7'
35
+
36
+ /** Pinned upstream source commit. */
37
+ export const UPSTREAM_COMMIT = visionToolkit?.upstreamCommit ?? '8ccf9d7d89bc395ad7dd19dde840401a5b84623d'
@@ -0,0 +1,27 @@
1
+ import type { IncomingMessage } from 'node:http'
2
+
3
+ /**
4
+ * Accept a request only from the DSH Web application's origin. Method-agnostic:
5
+ * the same fence guards state-changing POSTs and policy GETs.
6
+ * @param req - the incoming request whose headers carry the origin evidence.
7
+ * @returns whether the request may be answered.
8
+ */
9
+ export function sameOriginRequest(req: IncomingMessage): boolean {
10
+ const fetchSite = req.headers['sec-fetch-site']
11
+ if (fetchSite === 'cross-site') return false
12
+ const origin = req.headers.origin
13
+ if (origin === undefined) return fetchSite === 'same-origin' || fetchSite === 'same-site' || fetchSite === 'none'
14
+ const host = req.headers.host
15
+ if (host === undefined) return false
16
+ try {
17
+ const parsed = new URL(origin)
18
+ return (parsed.protocol === 'http:' || parsed.protocol === 'https:') && parsed.host === host
19
+ } catch {
20
+ return false
21
+ }
22
+ }
23
+
24
+ /** Accept state-changing requests only from the DSH Web application's origin. */
25
+ export function sameOriginPost(req: IncomingMessage): boolean {
26
+ return sameOriginRequest(req)
27
+ }
package/src/web.ts ADDED
@@ -0,0 +1,594 @@
1
+ /**
2
+ * Optional Web-profile routes: signed Artifact delivery plus a same-origin
3
+ * Settings/health endpoint. The browser never receives credential values and
4
+ * connection tests run only after an explicit POST action.
5
+ * @module dsh-vision-toolkit/web
6
+ */
7
+
8
+ import type { IncomingMessage, ServerResponse } from 'node:http'
9
+ import type { Context } from '@deepseek-ai/cordis'
10
+ import type { CredentialInfo, CredentialRef } from '@deepseek-ai/dsh-credentials'
11
+ import { credentialRef } from '@deepseek-ai/dsh-credentials'
12
+ import { SettingsConflictError, type SettingsDescriptor } from '@deepseek-ai/dsh-settings'
13
+ // Type-only import activates the optional webServer Context declaration.
14
+ import type {} from '@deepseek-ai/dsh-host-webserver'
15
+ import { ArtifactAccessController, ARTIFACT_ROUTE_PREFIX } from './artifact-access.ts'
16
+ import {
17
+ PastedImageBackend,
18
+ PASTE_IMAGES_ROUTE,
19
+ PASTE_POLICY_ROUTE,
20
+ type PasteSelectionQuery,
21
+ type PasteVerdict,
22
+ } from './paste-images.ts'
23
+ import {
24
+ resolveConfig,
25
+ isBuiltInFreeVisionProvider,
26
+ VISION_TOOLKIT_SETTINGS_NAMESPACE,
27
+ type ResolvedVisionToolkitConfig,
28
+ type VisionToolkitConfig,
29
+ } from './config.ts'
30
+ import type { VisionToolkitHealthResult } from './runtime.ts'
31
+ import {
32
+ PluginUpdateError,
33
+ VisionToolkitPluginUpdateService,
34
+ type PluginUpdateCapability,
35
+ type PluginUpdateCheck,
36
+ type PluginUpdateResult,
37
+ } from './plugin-update.ts'
38
+ import {
39
+ VisionToolkitRuntimeManager,
40
+ type PreparedRuntimeGeneration,
41
+ type RuntimeManagerStatus,
42
+ } from './runtime-manager.ts'
43
+ import { PLUGIN_VERSION, UPSTREAM_COMMIT, UPSTREAM_REPOSITORY, UPSTREAM_VERSION } from './version.ts'
44
+ import { sameOriginPost, sameOriginRequest } from './web-request.ts'
45
+
46
+ /** Exact route used by the browser Settings page. */
47
+ export const SETTINGS_ROUTE = '/_dsh/vision-toolkit/settings'
48
+
49
+ /** Same-origin route used by the browser client to read display-mode flags. */
50
+ export const DISPLAY_CONFIG_ROUTE = '/_dsh/vision-toolkit/display-config'
51
+
52
+ /** Public Settings snapshot; credential values are deliberately impossible here. */
53
+ export interface VisionToolkitSettingsSnapshot {
54
+ schemaVersion: 1
55
+ writable: boolean
56
+ settings: {
57
+ value: VisionToolkitConfig
58
+ user?: unknown
59
+ base?: unknown
60
+ revision: number
61
+ applies: 'live'
62
+ }
63
+ credential: {
64
+ ref: string
65
+ configured: boolean
66
+ source?: string
67
+ writable: boolean
68
+ }
69
+ runtime: RuntimeManagerStatus
70
+ release: {
71
+ pluginVersion: string
72
+ upstreamRepository: string
73
+ upstreamVersion: string
74
+ upstreamCommit: string
75
+ update: PluginUpdateCapability
76
+ }
77
+ artifactRouteAvailable: boolean
78
+ }
79
+
80
+ interface SaveRequest {
81
+ action: 'save'
82
+ expectedRevision: number
83
+ value: VisionToolkitConfig
84
+ }
85
+
86
+ interface HealthRequest {
87
+ action: 'health'
88
+ testConnection: boolean
89
+ testModel: boolean
90
+ }
91
+
92
+ interface CredentialRequest {
93
+ action: 'credential'
94
+ expectedRevision: number
95
+ ref: CredentialRef
96
+ value: string
97
+ }
98
+
99
+ interface CheckUpdateRequest {
100
+ action: 'check-update'
101
+ }
102
+
103
+ interface ApplyUpdateRequest {
104
+ action: 'apply-update'
105
+ expectedVersion: string
106
+ }
107
+
108
+ type SettingsRequest = SaveRequest | HealthRequest | CredentialRequest | CheckUpdateRequest | ApplyUpdateRequest
109
+
110
+ interface JsonError {
111
+ ok: false
112
+ error: { code: string; message: string }
113
+ }
114
+
115
+ interface JsonSuccess<T> {
116
+ ok: true
117
+ value: T
118
+ }
119
+
120
+ type JsonResponse<T> = JsonSuccess<T> | JsonError
121
+
122
+ /** Minimal runtime-manager face used by the Web route and its tests. */
123
+ export interface WebRuntimeManager {
124
+ readonly ready: boolean
125
+ current(): ReturnType<VisionToolkitRuntimeManager['current']>
126
+ prepareCandidate(raw: VisionToolkitConfig): Promise<PreparedRuntimeGeneration>
127
+ activateCandidate(candidate: PreparedRuntimeGeneration): void
128
+ recordFailure(error: unknown): void
129
+ status(): RuntimeManagerStatus
130
+ }
131
+
132
+ /** Minimal self-update face used by the Web route and its tests. */
133
+ export interface WebPluginUpdater {
134
+ configureWebServer?(host: string, port: number): void
135
+ capability(): Promise<PluginUpdateCapability>
136
+ check(): Promise<PluginUpdateCheck>
137
+ installAndRestart(expectedVersion: string): Promise<PluginUpdateResult>
138
+ }
139
+
140
+ /** Callback invoked when a Settings save makes the first runtime available. */
141
+ export type RuntimeActivated = () => void
142
+
143
+ function isRecord(value: unknown): value is Record<string, unknown> {
144
+ return typeof value === 'object' && value !== null && !Array.isArray(value)
145
+ }
146
+
147
+ class CredentialReferenceConflictError extends Error {}
148
+
149
+ function descriptorOf(ctx: Context): SettingsDescriptor {
150
+ const descriptor = ctx.settings.describe().find(row => row.ns === VISION_TOOLKIT_SETTINGS_NAMESPACE)
151
+ if (descriptor === undefined) throw new Error('vision-toolkit Settings namespace is not registered')
152
+ return descriptor
153
+ }
154
+
155
+ function responseJson<T>(res: ServerResponse, status: number, body: JsonResponse<T>): void {
156
+ const bytes = Buffer.from(JSON.stringify(body))
157
+ res.setHeader('Content-Type', 'application/json; charset=utf-8')
158
+ res.setHeader('Content-Length', String(bytes.length))
159
+ res.setHeader('Cache-Control', 'no-store')
160
+ res.setHeader('X-Content-Type-Options', 'nosniff')
161
+ res.setHeader('Content-Security-Policy', "default-src 'none'; frame-ancestors 'none'")
162
+ res.writeHead(status)
163
+ res.end(bytes)
164
+ }
165
+
166
+ function requestError(res: ServerResponse, status: number, code: string, message: string): void {
167
+ responseJson(res, status, { ok: false, error: { code, message } })
168
+ }
169
+
170
+ async function readJson(req: IncomingMessage, maxBytes = 64 * 1024): Promise<unknown> {
171
+ const contentType = req.headers['content-type']?.split(';', 1)[0]?.trim().toLowerCase()
172
+ if (contentType !== 'application/json') throw new TypeError('Content-Type must be application/json')
173
+ const chunks: Buffer[] = []
174
+ let bytes = 0
175
+ for await (const chunk of req) {
176
+ const part = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)
177
+ bytes += part.length
178
+ if (bytes > maxBytes) throw new RangeError(`request body exceeds ${maxBytes} bytes`)
179
+ chunks.push(part)
180
+ }
181
+ if (chunks.length === 0) throw new TypeError('request body is empty')
182
+ return JSON.parse(Buffer.concat(chunks).toString('utf8')) as unknown
183
+ }
184
+
185
+ function parseRequest(value: unknown): SettingsRequest {
186
+ if (!isRecord(value) || typeof value.action !== 'string') throw new TypeError('request action is required')
187
+ if (value.action === 'health') {
188
+ if (typeof value.testConnection !== 'boolean') throw new TypeError('health.testConnection must be boolean')
189
+ const testModel = value.testModel === undefined ? false : value.testModel
190
+ if (typeof testModel !== 'boolean') throw new TypeError('health.testModel must be boolean')
191
+ if (testModel && !value.testConnection) throw new TypeError('health.testModel requires health.testConnection')
192
+ return { action: 'health', testConnection: value.testConnection, testModel }
193
+ }
194
+ if (value.action === 'save') {
195
+ if (!Number.isSafeInteger(value.expectedRevision) || (value.expectedRevision as number) < 0) {
196
+ throw new TypeError('save.expectedRevision must be a non-negative integer')
197
+ }
198
+ if (!isRecord(value.value)) throw new TypeError('save.value must be an object')
199
+ return {
200
+ action: 'save',
201
+ expectedRevision: value.expectedRevision as number,
202
+ value: value.value as VisionToolkitConfig,
203
+ }
204
+ }
205
+ if (value.action === 'credential') {
206
+ if (!Number.isSafeInteger(value.expectedRevision) || (value.expectedRevision as number) < 0) {
207
+ throw new TypeError('credential.expectedRevision must be a non-negative integer')
208
+ }
209
+ if (typeof value.ref !== 'string') throw new TypeError('credential.ref must be a string')
210
+ if (typeof value.value !== 'string') throw new TypeError('credential.value must be a string')
211
+ const secret = value.value.trim()
212
+ if (secret.length === 0) throw new TypeError('API key cannot be blank')
213
+ const first = secret[0]
214
+ const quoted = secret.length > 1 && (first === '"' || first === '\'' || first === '`') && secret.endsWith(first)
215
+ const environmentLine = /^[A-Z][A-Z0-9_]*=[^=]/u.test(secret)
216
+ if (quoted || environmentLine || !/^[\x21-\x7E]+$/u.test(secret)) {
217
+ throw new TypeError('paste only the API key, without a variable name, quotes, spaces, or line breaks')
218
+ }
219
+ return {
220
+ action: 'credential',
221
+ expectedRevision: value.expectedRevision as number,
222
+ ref: credentialRef(value.ref),
223
+ value: secret,
224
+ }
225
+ }
226
+ if (value.action === 'check-update') return { action: 'check-update' }
227
+ if (value.action === 'apply-update') {
228
+ if (typeof value.expectedVersion !== 'string' || value.expectedVersion.trim().length === 0) {
229
+ throw new TypeError('apply-update.expectedVersion must be a non-empty string')
230
+ }
231
+ return { action: 'apply-update', expectedVersion: value.expectedVersion.trim() }
232
+ }
233
+ throw new TypeError(`unsupported action: ${value.action}`)
234
+ }
235
+
236
+ function publicMessage(error: unknown): string {
237
+ if (error instanceof Error) return error.message
238
+ return String(error)
239
+ }
240
+
241
+ /** Same-origin Settings and health handler. */
242
+ export class VisionToolkitWebBackend {
243
+ private readonly updater: WebPluginUpdater
244
+
245
+ constructor(
246
+ private readonly ctx: Context,
247
+ private readonly manager: WebRuntimeManager,
248
+ private readonly artifacts: ArtifactAccessController,
249
+ private readonly onRuntimeActivated: RuntimeActivated,
250
+ updater?: WebPluginUpdater,
251
+ ) {
252
+ this.updater = updater ?? new VisionToolkitPluginUpdateService(ctx, PLUGIN_VERSION, {
253
+ runtimeReady: () => this.manager.status().ready,
254
+ })
255
+ }
256
+
257
+ /** Supply the active listener address before the Settings route becomes reachable. */
258
+ configureWebServer(host: string, port: number): void {
259
+ this.updater.configureWebServer?.(host, port)
260
+ }
261
+
262
+ private async credential(config: ResolvedVisionToolkitConfig): Promise<CredentialInfo> {
263
+ if (isBuiltInFreeVisionProvider(config.provider)) {
264
+ return { configured: true, source: 'built-in-free', writable: false }
265
+ }
266
+ return this.ctx.credentials.describe(credentialRef(String(config.provider.credential)))
267
+ }
268
+
269
+ /** Build the current settings/runtime/credential snapshot without secrets. */
270
+ async snapshot(): Promise<VisionToolkitSettingsSnapshot> {
271
+ const descriptor = descriptorOf(this.ctx)
272
+ const value = descriptor.value as VisionToolkitConfig
273
+ const resolved = resolveConfig(value)
274
+ const credential = await this.credential(resolved)
275
+ const update = await this.updater.capability()
276
+ return {
277
+ schemaVersion: 1,
278
+ writable: this.ctx.settings.writable,
279
+ settings: {
280
+ value,
281
+ ...(descriptor.user === undefined ? {} : { user: descriptor.user }),
282
+ ...(descriptor.base === undefined ? {} : { base: descriptor.base }),
283
+ revision: descriptor.revision,
284
+ applies: 'live',
285
+ },
286
+ credential: {
287
+ ref: String(resolved.provider.credential),
288
+ configured: credential.configured,
289
+ ...(credential.source === undefined ? {} : { source: credential.source }),
290
+ writable: credential.writable,
291
+ },
292
+ runtime: this.manager.status(),
293
+ release: {
294
+ pluginVersion: PLUGIN_VERSION,
295
+ upstreamRepository: UPSTREAM_REPOSITORY,
296
+ upstreamVersion: UPSTREAM_VERSION,
297
+ upstreamCommit: UPSTREAM_COMMIT,
298
+ update,
299
+ },
300
+ artifactRouteAvailable: this.artifacts.routeAvailable,
301
+ }
302
+ }
303
+
304
+ private async save(request: SaveRequest): Promise<VisionToolkitSettingsSnapshot> {
305
+ if (!this.ctx.settings.writable) throw new Error('settings provider is read-only')
306
+ let candidate: PreparedRuntimeGeneration
307
+ try {
308
+ candidate = await this.manager.prepareCandidate(request.value)
309
+ } catch (error) {
310
+ this.manager.recordFailure(error)
311
+ throw error
312
+ }
313
+ await this.ctx.settings.replace(
314
+ VISION_TOOLKIT_SETTINGS_NAMESPACE,
315
+ request.value as object,
316
+ request.expectedRevision,
317
+ )
318
+ this.manager.activateCandidate(candidate)
319
+ this.onRuntimeActivated()
320
+ return this.snapshot()
321
+ }
322
+
323
+ private async saveCredential(request: CredentialRequest): Promise<VisionToolkitSettingsSnapshot> {
324
+ const descriptor = descriptorOf(this.ctx)
325
+ if (descriptor.revision !== request.expectedRevision) {
326
+ throw new SettingsConflictError(
327
+ VISION_TOOLKIT_SETTINGS_NAMESPACE,
328
+ request.expectedRevision,
329
+ descriptor.revision,
330
+ )
331
+ }
332
+ const resolved = resolveConfig(descriptor.value as VisionToolkitConfig)
333
+ const currentRef = credentialRef(String(resolved.provider.credential))
334
+ if (currentRef !== request.ref) {
335
+ throw new CredentialReferenceConflictError(
336
+ `credential reference changed from "${request.ref}" to "${currentRef}"; reload Settings and try again`,
337
+ )
338
+ }
339
+ if (isBuiltInFreeVisionProvider(resolved.provider)) {
340
+ throw new Error('The built-in free vision provider does not accept a user API key')
341
+ }
342
+ await this.ctx.credentials.set(currentRef, request.value)
343
+ return this.snapshot()
344
+ }
345
+
346
+ private async health(request: HealthRequest, req: IncomingMessage): Promise<VisionToolkitHealthResult> {
347
+ if (!this.manager.ready) throw new Error('runtime is not ready; fix Settings and save a valid configuration first')
348
+ const controller = new AbortController()
349
+ const abort = (): void => { controller.abort() }
350
+ req.once('aborted', abort)
351
+ req.socket.once('close', abort)
352
+ try {
353
+ const runtime = this.manager.current()
354
+ // Use the prepared runtime home instead of the host process cwd.
355
+ return await runtime.health(request.testConnection, {
356
+ signal: controller.signal,
357
+ workspace: runtime.upstreamVersion.runtimeHome,
358
+ sessionId: 'vision-toolkit-settings',
359
+ }, request.testModel)
360
+ } finally {
361
+ req.off('aborted', abort)
362
+ req.socket.off('close', abort)
363
+ }
364
+ }
365
+
366
+ /** Handle the exact Settings route. */
367
+ async handle(req: IncomingMessage, res: ServerResponse): Promise<void> {
368
+ if (req.method === 'GET') {
369
+ try {
370
+ responseJson(res, 200, { ok: true, value: await this.snapshot() })
371
+ } catch (error) {
372
+ this.ctx.logger.warn('dsh-vision-toolkit Settings snapshot failed: %s', publicMessage(error))
373
+ requestError(res, 503, 'settings-unavailable', 'Vision Toolkit Settings are unavailable')
374
+ }
375
+ return
376
+ }
377
+ if (req.method !== 'POST') {
378
+ res.setHeader('Allow', 'GET, POST')
379
+ requestError(res, 405, 'method-not-allowed', 'Use GET or POST')
380
+ return
381
+ }
382
+ if (!sameOriginPost(req)) {
383
+ requestError(res, 403, 'origin-rejected', 'The request must originate from this DSH Web application')
384
+ return
385
+ }
386
+ let parsed: SettingsRequest
387
+ try {
388
+ parsed = parseRequest(await readJson(req))
389
+ } catch (error) {
390
+ requestError(res, error instanceof RangeError ? 413 : 400, 'invalid-request', publicMessage(error))
391
+ return
392
+ }
393
+ try {
394
+ switch (parsed.action) {
395
+ case 'health':
396
+ responseJson(res, 200, { ok: true, value: await this.health(parsed, req) })
397
+ break
398
+ case 'save':
399
+ responseJson(res, 200, { ok: true, value: await this.save(parsed) })
400
+ break
401
+ case 'credential':
402
+ responseJson(res, 200, { ok: true, value: await this.saveCredential(parsed) })
403
+ break
404
+ case 'check-update':
405
+ responseJson(res, 200, { ok: true, value: await this.updater.check() })
406
+ break
407
+ case 'apply-update':
408
+ responseJson(res, 200, { ok: true, value: await this.updater.installAndRestart(parsed.expectedVersion) })
409
+ break
410
+ }
411
+ } catch (error) {
412
+ const settingsConflict = error instanceof SettingsConflictError
413
+ const credentialConflict = error instanceof CredentialReferenceConflictError
414
+ const updateError = error instanceof PluginUpdateError
415
+ const code = settingsConflict
416
+ ? 'settings-conflict'
417
+ : credentialConflict
418
+ ? 'credential-conflict'
419
+ : updateError
420
+ ? error.code
421
+ : parsed.action === 'health'
422
+ ? 'health-failed'
423
+ : parsed.action === 'credential'
424
+ ? 'credential-rejected'
425
+ : 'settings-rejected'
426
+ const updateConflict = updateError && ['update-in-progress', 'update-stale', 'update-unavailable', 'already-current'].includes(error.code)
427
+ const updateGateway = updateError && error.code === 'update-check-failed'
428
+ const status = settingsConflict || credentialConflict || updateConflict
429
+ ? 409
430
+ : parsed.action === 'health'
431
+ ? 503
432
+ : updateGateway
433
+ ? 502
434
+ : updateError
435
+ ? 500
436
+ : 400
437
+ this.ctx.logger.warn('dsh-vision-toolkit Web action=%s failed: %s', parsed.action, publicMessage(error))
438
+ requestError(res, status, code, publicMessage(error))
439
+ }
440
+ }
441
+ }
442
+
443
+ /**
444
+ * Same-origin policy handler for the paste route: whether the browser should
445
+ * take a paste over into workspace paths, or let it flow natively after an
446
+ * optional automatic switch to the image-input variant. The optional `model`
447
+ * query carries the model-selector label the client currently shows; the
448
+ * optional `provider`/`modelId`/`reasoningEffort` queries carry the exact
449
+ * route the client read from the live model catalog, which the resolver
450
+ * prefers (a label alone cannot pick a provider). Unresolvable routes answer
451
+ * native — the safe default.
452
+ * @param resolve - resolves one live Session's paste verdict.
453
+ * @returns the HTTP handler.
454
+ */
455
+ export function createPastePolicyHandler(
456
+ resolve: (sessionId: string, selection?: PasteSelectionQuery, modelLabel?: string) => Promise<PasteVerdict>,
457
+ ): (req: IncomingMessage, res: ServerResponse) => void {
458
+ return (req, res) => {
459
+ void (async () => {
460
+ try {
461
+ if (req.method !== 'GET') {
462
+ requestError(res, 405, 'method-not-allowed', 'Use GET')
463
+ return
464
+ }
465
+ if (!sameOriginRequest(req)) {
466
+ requestError(res, 403, 'origin-rejected', 'The request must originate from this DSH Web application')
467
+ return
468
+ }
469
+ let sessionId: string
470
+ let modelLabel: string | undefined
471
+ let selection: PasteSelectionQuery | undefined
472
+ try {
473
+ const url = new URL(req.url ?? PASTE_POLICY_ROUTE, 'http://dsh.internal')
474
+ const sessions = url.searchParams.getAll('sessionId')
475
+ if (sessions.length !== 1 || sessions[0] === undefined || sessions[0] === '') {
476
+ throw new TypeError('sessionId is required exactly once')
477
+ }
478
+ sessionId = sessions[0]!
479
+ const models = url.searchParams.getAll('model')
480
+ if (models.length > 1) throw new TypeError('model may be given at most once')
481
+ modelLabel = models[0]
482
+ const providers = url.searchParams.getAll('provider')
483
+ if (providers.length > 1) throw new TypeError('provider may be given at most once')
484
+ const modelIds = url.searchParams.getAll('modelId')
485
+ if (modelIds.length > 1) throw new TypeError('modelId may be given at most once')
486
+ const efforts = url.searchParams.getAll('reasoningEffort')
487
+ if (efforts.length > 1) throw new TypeError('reasoningEffort may be given at most once')
488
+ const provider = providers[0]
489
+ const modelId = modelIds[0]
490
+ if (provider !== undefined && modelId !== undefined && provider !== '' && modelId !== '') {
491
+ selection = {
492
+ provider,
493
+ model: modelId,
494
+ ...(efforts[0] === undefined || efforts[0] === '' ? {} : { reasoningEffort: efforts[0] }),
495
+ }
496
+ }
497
+ } catch (error) {
498
+ requestError(res, 400, 'invalid-request', publicMessage(error))
499
+ return
500
+ }
501
+ const verdict = await resolve(sessionId, selection, modelLabel)
502
+ responseJson(res, 200, { ok: true, value: verdict })
503
+ } catch (error) {
504
+ requestError(res, 500, 'policy-failed', publicMessage(error))
505
+ }
506
+ })()
507
+ }
508
+ }
509
+
510
+ /**
511
+ * Same-origin display-config handler: exposes whether transparent routing is
512
+ * active. The paste integration uses it to choose its notice text; the model
513
+ * selector hides upstream twins synchronously from DOM display names and does
514
+ * not depend on this route.
515
+ * @param getDisplayConfig - resolves the current display-mode flags.
516
+ * @returns the HTTP handler.
517
+ */
518
+ export function createDisplayConfigHandler(
519
+ getDisplayConfig: () => { hidden: boolean },
520
+ ): (req: IncomingMessage, res: ServerResponse) => void {
521
+ return (req, res) => {
522
+ try {
523
+ if (req.method !== 'GET') {
524
+ requestError(res, 405, 'method-not-allowed', 'Use GET')
525
+ return
526
+ }
527
+ if (!sameOriginRequest(req)) {
528
+ requestError(res, 403, 'origin-rejected', 'The request must originate from this DSH Web application')
529
+ return
530
+ }
531
+ responseJson(res, 200, { ok: true, value: getDisplayConfig() })
532
+ } catch (error) {
533
+ requestError(res, 500, 'display-config-failed', publicMessage(error))
534
+ }
535
+ }
536
+ }
537
+
538
+ /**
539
+ * Attach optional Web routes whenever a webServer service is present.
540
+ * @param ctx - plugin context owning route effects.
541
+ * @param backend - Settings handler.
542
+ * @param artifacts - signed Artifact handler.
543
+ * @param pastedImages - pasted-image workspace handler.
544
+ * @param pastePolicy - paste-policy verdict resolver (sessionId, selection, modelLabel).
545
+ * @param getDisplayConfig - resolves display-mode flags for the browser client.
546
+ */
547
+ export function installVisionToolkitWeb(
548
+ ctx: Context,
549
+ backend: VisionToolkitWebBackend,
550
+ artifacts: ArtifactAccessController,
551
+ pastedImages: PastedImageBackend,
552
+ pastePolicy: (sessionId: string, selection?: PasteSelectionQuery, modelLabel?: string) => Promise<PasteVerdict>,
553
+ getDisplayConfig: () => { hidden: boolean },
554
+ ): void {
555
+ ctx.inject(['webServer'], (webCtx) => {
556
+ webCtx.effect(() => {
557
+ backend.configureWebServer(webCtx.webServer.host, webCtx.webServer.port)
558
+ const detach = artifacts.attachRoute()
559
+ const disposeArtifact = webCtx.webServer.register({
560
+ kind: 'prefix',
561
+ path: ARTIFACT_ROUTE_PREFIX,
562
+ handler: (req, res) => artifacts.handle(req, res),
563
+ })
564
+ const disposeSettings = webCtx.webServer.register({
565
+ kind: 'exact',
566
+ path: SETTINGS_ROUTE,
567
+ handler: (req, res) => backend.handle(req, res),
568
+ })
569
+ const disposePasteImages = webCtx.webServer.register({
570
+ kind: 'exact',
571
+ path: PASTE_IMAGES_ROUTE,
572
+ handler: (req, res) => pastedImages.handle(req, res),
573
+ })
574
+ const disposePastePolicy = webCtx.webServer.register({
575
+ kind: 'exact',
576
+ path: PASTE_POLICY_ROUTE,
577
+ handler: createPastePolicyHandler(pastePolicy),
578
+ })
579
+ const disposeDisplayConfig = webCtx.webServer.register({
580
+ kind: 'exact',
581
+ path: DISPLAY_CONFIG_ROUTE,
582
+ handler: createDisplayConfigHandler(getDisplayConfig),
583
+ })
584
+ return () => {
585
+ disposeDisplayConfig()
586
+ disposePastePolicy()
587
+ disposePasteImages()
588
+ disposeSettings()
589
+ disposeArtifact()
590
+ detach()
591
+ }
592
+ }, 'dsh-vision-toolkit: Web routes')
593
+ })
594
+ }
@@ -0,0 +1,30 @@
1
+ # Changelog
2
+
3
+ All notable user-facing changes to agent-vision-toolkit are documented in this file.
4
+
5
+ ## [Unreleased]
6
+
7
+ ### Added
8
+
9
+ - Let the shared Python vision client call either Chat Completions or Responses APIs, including optional reasoning effort and explicit `store: false` data handling.
10
+ - Add native Anthropic Messages requests with protocol-specific authentication, image sources, optional thinking control, and text-block response extraction.
11
+ - Rewrite OpenAI Chat Completions `image_url` blocks through the existing vision-description pipeline with a host-neutral channel note.
12
+
13
+ ### Fixed
14
+
15
+ - Send a browser-compatible, configurable User-Agent from the shared Python vision client so Cloudflare-backed OpenAI-compatible endpoints do not reject the default `Python-urllib` signature.
16
+ - Honor `Retry-After` and retry Anthropic 529 overload responses.
17
+ - Allow trusted internal endpoints with self-signed certificates or MITM proxies to opt out of TLS certificate verification through `VISION_SSL_VERIFY`, while preserving verification by default.
18
+
19
+ ## [0.1.0] - 2026-08-07
20
+
21
+ ### Added
22
+
23
+ - Five vision CLIs — `glance`, `ground`, `detect`, `trace`, and `crop` — plus the `vision-tools` agent skill.
24
+ - Optional seamless integration: a local proxy for Codex and Claude Code, and single-file native extensions for Pi, Oh My Pi, and OpenCode.
25
+ - Pasted-image and tool-fetched image support with task-aware focus hints, parallel multi-image descriptions, per-request caching, and honest failure notes.
26
+ - Vision playbooks for long-screenshot OCR, UI restoration, graphic restoration, structure recovery, and GUI operation.
27
+ - Community contribution, conduct, support, and security policies.
28
+ - Structured issue forms and a pull request template.
29
+ - GitHub funding configuration and continuous integration checks.
30
+ - A bilingual funding policy and sponsorship-use statement.