@mokuyoaxis/dsh-iris 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.dsh/skills/iris-compose-media/SKILL.md +114 -0
- package/.dsh/skills/iris-compose-media/agents/openai.yaml +4 -0
- package/.dsh/skills/iris-verify-ui/SKILL.md +127 -0
- package/CHANGELOG.md +53 -0
- package/LICENSE +21 -0
- package/README.md +177 -0
- package/cordis.patch.yml +3 -0
- package/docs/ROADMAP.md +72 -0
- package/docs/assets/iris-wordmark.svg +25 -0
- package/docs/file-access-across-environments.md +78 -0
- package/lib/actions.js +930 -0
- package/lib/adapters.js +504 -0
- package/lib/api.js +467 -0
- package/lib/capability.js +103 -0
- package/lib/client.d.ts +5 -0
- package/lib/client.js +1187 -0
- package/lib/config.js +384 -0
- package/lib/guard.js +73 -0
- package/lib/index.js +1028 -0
- package/lib/locate.js +94 -0
- package/lib/media-probe.js +169 -0
- package/lib/media.js +172 -0
- package/lib/models.js +128 -0
- package/lib/ocr.js +80 -0
- package/lib/pixels.js +140 -0
- package/lib/render.js +138 -0
- package/lib/summarize.js +111 -0
- package/lib/tasks.js +315 -0
- package/lib/vision.js +171 -0
- package/package.json +95 -0
- package/user_guide.md +212 -0
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: iris-compose-media
|
|
3
|
+
description: "Compose two or more dsh-iris tools into goal-driven image, video, speech, transcription, and multimodal workflows. Use when a request requires analyzing supplied media and creating a derivative, generating then reviewing an image, turning a video summary into narration, preparing image and audio inputs for video, or coordinating asynchronous Iris tasks. Do not use for a single direct media operation. 用于看图后绘图、生成后自检、视频总结配旁白、图像加语音生成视频等多步骤媒体任务。"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Compose Media with Iris
|
|
7
|
+
|
|
8
|
+
Turn a media goal into the shortest reliable chain of Iris tools. Preserve useful attachments and paths between steps, keep paid generation bounded, and distinguish an accepted background task from a finished artifact.
|
|
9
|
+
|
|
10
|
+
## Decide whether to use this skill
|
|
11
|
+
|
|
12
|
+
Use this skill only when the request needs two or more dependent Iris operations. Typical triggers include:
|
|
13
|
+
|
|
14
|
+
- analyze an image, then create a new image from the findings;
|
|
15
|
+
- generate an image, inspect it, and revise only if requested;
|
|
16
|
+
- summarize a video, then write and synthesize narration;
|
|
17
|
+
- generate or reuse a still image, synthesize speech, then create an S2V video;
|
|
18
|
+
- transcribe or OCR source media, then turn its content into another artifact.
|
|
19
|
+
|
|
20
|
+
For one direct operation such as drawing, describing, OCR, transcription, speech synthesis, or video generation, do not use this skill; call the matching tool directly. For screenshot comparison or frontend visual acceptance, use `iris-verify-ui` instead.
|
|
21
|
+
|
|
22
|
+
Ask one concise question only when a missing source, target format, voice, aspect ratio, or intended audience would materially change the result. Otherwise make a conservative assumption and state it in the result.
|
|
23
|
+
|
|
24
|
+
## Plan the chain
|
|
25
|
+
|
|
26
|
+
Before calling tools, identify:
|
|
27
|
+
|
|
28
|
+
1. the source media and how Iris can read it;
|
|
29
|
+
2. the requested final artifact;
|
|
30
|
+
3. the minimum dependent tool chain;
|
|
31
|
+
4. which steps can incur generation cost;
|
|
32
|
+
5. whether a downstream step needs a file path, attachment ID, text, or completed task.
|
|
33
|
+
|
|
34
|
+
Do not add an analysis, transcription, or review step merely because a tool exists. Parallelize only independent read-only work; dependent media operations must remain ordered.
|
|
35
|
+
|
|
36
|
+
## Resolve media inputs
|
|
37
|
+
|
|
38
|
+
- Use `iris_look_at_image` for an absolute image path visible to the host.
|
|
39
|
+
- Use `iris_relook_attachment` for an image attachment already present in the session or produced by Iris.
|
|
40
|
+
- Use `iris_long_ocr` when exact text from an image matters more than a visual description.
|
|
41
|
+
- Video and audio tools require host-visible absolute `video_path` or `audio_path` values. A browser-local path, `content://` URI, or ordinary web URL is not a host path; request an upload or exported host path.
|
|
42
|
+
- Keep every returned attachment ID, task ID, and output path that a later step needs. Do not invent or reconstruct identifiers.
|
|
43
|
+
|
|
44
|
+
An attachment returned by `iris_draw_image` can be passed as `first_frame_attachment_id` to `iris_generate_video`. An existing host image uses `first_frame_path`. The local path returned by `iris_speak_text` can be passed as `audio_path` for S2V.
|
|
45
|
+
|
|
46
|
+
## Choose a workflow
|
|
47
|
+
|
|
48
|
+
### Analyze an image, then draw
|
|
49
|
+
|
|
50
|
+
1. Inspect the source with `iris_look_at_image` or `iris_relook_attachment`. Ask specifically about the subject, composition, palette, style, visible text, and the transformation requested by the user.
|
|
51
|
+
2. Convert the observations into a self-contained generation prompt. Separate facts visible in the source from requested changes; preserve explicit constraints and exact text.
|
|
52
|
+
3. Call `iris_draw_image` once for each requested artifact.
|
|
53
|
+
4. If the user requested verification or iteration, inspect the generated attachment with `iris_relook_attachment` against the original requirements. Regenerate only for a concrete mismatch.
|
|
54
|
+
|
|
55
|
+
### Generate a still, then animate it
|
|
56
|
+
|
|
57
|
+
1. Create the still with `iris_draw_image`, reuse an Iris-produced first-frame attachment, or obtain a host-visible absolute path for the supplied image. Do not pass an ordinary session attachment as `first_frame_attachment_id`.
|
|
58
|
+
2. Write a motion prompt describing camera movement, subject motion, timing, and what must remain stable. Do not merely repeat the still-image prompt.
|
|
59
|
+
3. Call `iris_generate_video` with `first_frame_attachment_id` or `first_frame_path` for i2v. Use `size` and `duration` only when appropriate for the selected t2v/i2v model.
|
|
60
|
+
4. If the call returns a background task ID, use `iris_task_status` before claiming that the video exists.
|
|
61
|
+
|
|
62
|
+
### Summarize a video, then narrate it
|
|
63
|
+
|
|
64
|
+
1. Call `iris_media_summarize` with the user's question and `transcribe=true` unless they want a visual-only summary.
|
|
65
|
+
2. Do not also call `iris_video_frames` or `iris_transcribe_audio` by default: `iris_media_summarize` already samples frames and optionally transcribes the audio track.
|
|
66
|
+
3. Turn the summary into narration suited to the requested audience and length. Do not present inferred details as visible facts.
|
|
67
|
+
4. Call `iris_speak_text` and return the summary, contact-sheet attachment, and saved audio path.
|
|
68
|
+
|
|
69
|
+
Use `iris_video_frames` separately only when the user explicitly needs individual frames or a custom frame-level workflow. Use `iris_transcribe_audio` separately when the full transcript itself is required or when the input is an audio file rather than a video-summary task.
|
|
70
|
+
|
|
71
|
+
### Create an S2V talking video
|
|
72
|
+
|
|
73
|
+
1. Resolve or generate a suitable first-frame portrait.
|
|
74
|
+
2. Use an existing host audio file, or call `iris_speak_text` and capture its saved absolute path.
|
|
75
|
+
3. Confirm that the audio is a clear human voice in WAV or MP3 format, smaller than 15 MB, and shorter than 20 seconds. Shorten the script and synthesize it again if the generated speech exceeds the limit.
|
|
76
|
+
4. Call `iris_generate_video` with an S2V model, `first_frame_attachment_id` or `first_frame_path`, `audio_path`, and `resolution` of `480P` or `720P`. Do not pass t2v/i2v-only `size` or `duration` controls.
|
|
77
|
+
5. Track a background result with `iris_task_status`; do not submit a duplicate while it is running.
|
|
78
|
+
|
|
79
|
+
### Turn recognized content into a new artifact
|
|
80
|
+
|
|
81
|
+
Use `iris_long_ocr` for image text or `iris_transcribe_audio` for audio speech, then structure the recognized content for the requested output. Feed only the necessary, checked facts into `iris_draw_image`, `iris_speak_text`, or `iris_generate_video`. Preserve names, numbers, and quoted wording exactly when accuracy matters.
|
|
82
|
+
|
|
83
|
+
## Control cost and iteration
|
|
84
|
+
|
|
85
|
+
- Make one generation attempt per requested artifact by default.
|
|
86
|
+
- A request to create an artifact authorizes that requested generation; do not ask again solely because it may incur provider cost.
|
|
87
|
+
- Run at most 2 generation attempts per artifact unless the user explicitly requests a larger iterative loop.
|
|
88
|
+
- Review does not automatically authorize regeneration. If the user asked only for one result, report a detected mismatch and offer the next step.
|
|
89
|
+
- Do not probe models or create speculative alternatives as part of the workflow.
|
|
90
|
+
- Honor an explicit `providerId::modelId` override. Otherwise use the configured capability assignment and its ordered failover list.
|
|
91
|
+
|
|
92
|
+
Iris generation failover covers upload, submission, and synchronous-generation failures only. Once a remote service accepts an asynchronous task, that task is bound to its provider. If polling, waiting, or download later fails, query `iris_task_status` and report the state; never resubmit automatically, because that can duplicate work and billing.
|
|
93
|
+
|
|
94
|
+
## Handle tasks, cancellation, and failures
|
|
95
|
+
|
|
96
|
+
- Continue immediately when a tool returns a completed attachment, text, or local path.
|
|
97
|
+
- When a tool reports a background task, wait or query with `iris_task_status` only when the next step depends on its output.
|
|
98
|
+
- If the user cancels, stop the chain. Do not launch downstream generation after cancellation, even if an earlier artifact later completes.
|
|
99
|
+
- If an attachment cannot be found, ask the user to re-upload it or provide a host path.
|
|
100
|
+
- If `ffmpeg` or `ffprobe` is unavailable, video frames and video summarization cannot run. Explain the missing optional dependency; do not claim that visual video analysis completed.
|
|
101
|
+
- If vision is unavailable but the user's creation prompt is already complete, skip source analysis and proceed only when that still satisfies the request.
|
|
102
|
+
- Preserve partial outputs. Report which step succeeded, which step failed, and whether any accepted task remains active.
|
|
103
|
+
|
|
104
|
+
## Report the result
|
|
105
|
+
|
|
106
|
+
Return a compact workflow record containing:
|
|
107
|
+
|
|
108
|
+
1. the completed chain and any material assumptions;
|
|
109
|
+
2. attachment IDs and host paths needed to use the outputs;
|
|
110
|
+
3. task IDs and their last known status;
|
|
111
|
+
4. any omitted or failed step and its effect on the final artifact;
|
|
112
|
+
5. whether further generation would require another attempt.
|
|
113
|
+
|
|
114
|
+
Never describe a queued or running task as complete. Never hide that a summary omitted audio, that a review was skipped, or that the final artifact differs from the request.
|
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: iris-verify-ui
|
|
3
|
+
description: "Verify web UIs, HTML, and interface screenshots with dsh-iris through repeatable capture, semantic inspection, element location, cropping, pixel-diff analysis, and bounded rechecks. Use when comparing a UI with a reference image, locating visual regressions, validating frontend changes, or matching an implementation to a design. 用于对照参考图检查界面、定位视觉偏差、验证前端修改或比较两张 UI 截图。"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Verify UI with Iris
|
|
7
|
+
|
|
8
|
+
Use vision models to understand and locate problems. Use deterministic pixel tools to measure them. Establish comparable capture conditions before editing, then recheck under the same conditions. Do not declare success from visual impression alone.
|
|
9
|
+
|
|
10
|
+
## Choose the shortest path
|
|
11
|
+
|
|
12
|
+
Select a path from the evidence provided:
|
|
13
|
+
|
|
14
|
+
- Current and reference screenshots are both available: compare them directly; do not recapture.
|
|
15
|
+
- Raw HTML and a reference screenshot are available: capture the current version with `iris_html_screenshot`.
|
|
16
|
+
- Editable page source is available but no current screenshot exists: use the project's browser or screenshot workflow first. `iris_html_screenshot` accepts an HTML string; it cannot navigate an arbitrary URL.
|
|
17
|
+
- Only one screenshot is available and no reference or acceptance criteria exist: perform a semantic review and state that comparative pixel verification is not possible.
|
|
18
|
+
- The request is only to describe, OCR, or generate media: do not use this skill. Use the matching Iris tool or a media-composition skill.
|
|
19
|
+
|
|
20
|
+
Ask one concise question only when a missing reference, target state, theme, or viewport would materially change the result. Make conservative assumptions otherwise and record them in the verdict.
|
|
21
|
+
|
|
22
|
+
## Prepare the inputs
|
|
23
|
+
|
|
24
|
+
1. Identify each image source:
|
|
25
|
+
- Host absolute path: pass `image_path`.
|
|
26
|
+
- Current-session or Iris-produced attachment: pass `attachment_id`.
|
|
27
|
+
- Browser-local path, `content://` URI, or ordinary web URL: do not assume the host can read it. Ask for an upload or a host-accessible file.
|
|
28
|
+
2. Record the comparison conditions: viewport, screenshot dimensions, device pixel ratio, theme, fonts, zoom, scroll position, and dynamic-content state.
|
|
29
|
+
3. Freeze time, animation, cursor, random content, loading state, and scrollbars where practical.
|
|
30
|
+
4. Require matching image dimensions for reliable UI comparison. If dimensions differ, recapture before comparing; do not let automatic stretching hide a layout error.
|
|
31
|
+
|
|
32
|
+
## Run the verification loop
|
|
33
|
+
|
|
34
|
+
### 1. Establish the baseline
|
|
35
|
+
|
|
36
|
+
State the current image, reference image, target area, and acceptance criteria. Follow a threshold supplied by the user. If no threshold exists, do not invent a universal passing percentage.
|
|
37
|
+
|
|
38
|
+
When raw HTML needs capture, call `iris_html_screenshot` with the same HTML wrapper, `fullPage` value, and size parameters for every iteration. Account for these constraints:
|
|
39
|
+
|
|
40
|
+
- `width` and `height` set a minimum container size; they do not control the browser viewport.
|
|
41
|
+
- The page runs in an opaque-origin CSP sandbox. Scripts, external network access, form submission, and remote resources are unavailable.
|
|
42
|
+
- Inline required styles, fonts, and images, or use data URLs.
|
|
43
|
+
- The shared browser may briefly show the temporary tab.
|
|
44
|
+
- `iris_html_screenshot` is not concurrency-safe. Capture multiple states sequentially.
|
|
45
|
+
|
|
46
|
+
### 2. Inspect semantics
|
|
47
|
+
|
|
48
|
+
Inspect the current and reference images separately. Call `iris_look_at_image` for a host path and `iris_relook_attachment` for an attachment.
|
|
49
|
+
|
|
50
|
+
Each vision call sees only one image. Ask the same structured questions for both images:
|
|
51
|
+
|
|
52
|
+
- List visible components, relative positions, and hierarchy by top, middle, and bottom regions.
|
|
53
|
+
- Identify clipping, overflow, misalignment, unusual spacing, typography, and color issues.
|
|
54
|
+
- Record the target element's position relative to its container and neighboring elements.
|
|
55
|
+
|
|
56
|
+
Compare the two observation sets and prioritize the differences. Treat vision answers as diagnostic leads, not pixel evidence.
|
|
57
|
+
|
|
58
|
+
### 3. Measure the whole image
|
|
59
|
+
|
|
60
|
+
Call `iris_pixel_diff` with the current and reference images. Record:
|
|
61
|
+
|
|
62
|
+
- diff ratio
|
|
63
|
+
- changed-pixel count and normalized dimensions
|
|
64
|
+
- worst grid regions
|
|
65
|
+
- heatmap attachment ID
|
|
66
|
+
|
|
67
|
+
Use the result as UI regression evidence only when image dimensions and render conditions match. The tool ignores alpha and counts RGB differences above its noise threshold. It stretches mismatched images to the smaller dimensions and downsizes when the longest side exceeds 1024 pixels. A low normalized diff therefore does not prove that the original large images are pixel-identical.
|
|
68
|
+
|
|
69
|
+
### 4. Locate and verify local regions
|
|
70
|
+
|
|
71
|
+
When a component must be located, call `iris_locate` separately on the current and reference images:
|
|
72
|
+
|
|
73
|
+
- Compare the two bounding boxes to detect position and size changes.
|
|
74
|
+
- If it returns `found=false`, retry once with a more specific target. If it still fails, report uncertain location; never fabricate coordinates.
|
|
75
|
+
- Treat the model-generated bounding box as a hypothesis and cross-check it against the screenshots and heatmap.
|
|
76
|
+
|
|
77
|
+
Use `iris_crop` when local evidence is needed:
|
|
78
|
+
|
|
79
|
+
- For a same-coordinate comparison, use identical `left`, `top`, `width`, and `height` on both images.
|
|
80
|
+
- When an element moved, report bounding-box geometry first. Do not crop different shapes and then use a stretched diff to hide the movement.
|
|
81
|
+
- Keep any added margin within image boundaries.
|
|
82
|
+
|
|
83
|
+
Run `iris_pixel_diff` on the two crop attachments to verify whether the problem is concentrated in the expected component.
|
|
84
|
+
|
|
85
|
+
### 5. Modify and recheck
|
|
86
|
+
|
|
87
|
+
Edit source files only when the task authorizes changes. Fix the highest-impact deterministic issue first, keep the patch focused, and preserve existing user changes.
|
|
88
|
+
|
|
89
|
+
After each modification:
|
|
90
|
+
|
|
91
|
+
1. Capture again under the baseline conditions.
|
|
92
|
+
2. Rerun the whole-image diff.
|
|
93
|
+
3. Recheck the previous worst regions.
|
|
94
|
+
4. Record before-and-after diff ratios and bounding-box changes.
|
|
95
|
+
5. Confirm that the fix did not move the difference elsewhere.
|
|
96
|
+
|
|
97
|
+
Run at most 3 modify-and-recheck rounds by default. Stop as soon as the user's criteria pass. Also stop when one round gives no improvement, capture conditions cannot be aligned, or remaining differences come from the environment. Explain the stopping reason instead of looping indefinitely.
|
|
98
|
+
|
|
99
|
+
## Assign a verdict
|
|
100
|
+
|
|
101
|
+
Use exactly one verdict:
|
|
102
|
+
|
|
103
|
+
- **Pass**: all objective requirements are satisfied and the evidence supports the conclusion.
|
|
104
|
+
- **Partial pass**: the primary goal is satisfied, but localized differences or environment noise remain.
|
|
105
|
+
- **Indeterminate**: a reference is missing, capture conditions differ, a required tool is unavailable, or evidence is insufficient.
|
|
106
|
+
|
|
107
|
+
For a “pixel-identical” claim, require matching dimensions and matching render conditions. When the longest side exceeds 1024 pixels, `iris_pixel_diff` downsizes the comparison and cannot by itself prove original-resolution identity.
|
|
108
|
+
|
|
109
|
+
## Degrade safely
|
|
110
|
+
|
|
111
|
+
- `dsh-builtin-browser` unavailable: ask for a current screenshot and continue with image comparison.
|
|
112
|
+
- Vision model unavailable: skip semantic inspection and model location; continue with known coordinates, crops, and pixel diff.
|
|
113
|
+
- `iris_locate` unstable: narrow the target using the heatmap and known layout; do not retry blindly.
|
|
114
|
+
- HTML depends on scripts or remote resources: create a static offline-renderable version, or use the project's existing screenshot workflow.
|
|
115
|
+
- Reference contains dynamic data: freeze or mask dynamic regions. If that is impossible, report them separately as noise.
|
|
116
|
+
|
|
117
|
+
## Report the result
|
|
118
|
+
|
|
119
|
+
Include:
|
|
120
|
+
|
|
121
|
+
1. Conditions: inputs, dimensions, theme, capture method, and known noise sources.
|
|
122
|
+
2. Changes: what changed and why.
|
|
123
|
+
3. Evidence: before-and-after diff, worst regions, key bounding boxes, and relevant attachments.
|
|
124
|
+
4. Verdict: Pass, Partial pass, or Indeterminate.
|
|
125
|
+
5. Remaining work: anything requiring human judgment or a real-browser check.
|
|
126
|
+
|
|
127
|
+
Do not finish with only “looks correct.” Do not report precision that the evidence cannot support.
|
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# 变更记录
|
|
2
|
+
|
|
3
|
+
本文件记录 dsh-iris 的用户可见变化。
|
|
4
|
+
|
|
5
|
+
## [Unreleased]
|
|
6
|
+
|
|
7
|
+
当前没有尚未发布的用户可见变更。
|
|
8
|
+
|
|
9
|
+
## [0.1.0] - 2026-09-04
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- 增加文生图、文生视频、图生视频和 S2V 数字人视频能力。
|
|
14
|
+
- 增加文本转语音和独立的音频转写能力。
|
|
15
|
+
- 增加看图问答、附件重看、长图 OCR、目标定位、图片裁剪和像素差异分析。
|
|
16
|
+
- 增加视频抽帧和多模态视频摘要;缺少 `ffmpeg` 时只影响相关工具。
|
|
17
|
+
- 增加 Iris 工作台及悬浮泡泡入口,可管理供应商、能力分配、文件输入和任务历史。
|
|
18
|
+
- 增加浏览器上传与会话附件选择,支持在不同文件系统之间传递输入文件。
|
|
19
|
+
- 增加异步任务跟踪、重启恢复、状态事件推送和带授权令牌的音视频播放。
|
|
20
|
+
- 为图生视频和 S2V 动作补充回归测试。
|
|
21
|
+
- 增加仓库级 `iris-verify-ui` 与 `iris-compose-media` Skills:前者执行可重复的 UI 视觉验收,后者编排多步骤媒体工作流。
|
|
22
|
+
- 增加面向安装、供应商、模型池、能力分配和文件输入的 `user_guide.md`。
|
|
23
|
+
|
|
24
|
+
### Changed
|
|
25
|
+
|
|
26
|
+
- npm 发布包改用 `@mokuyoaxis/dsh-iris`,避开已被其他项目占用的无 scope 包名;运行时插件 ID 和工具名保持不变。
|
|
27
|
+
- 模型引用升级为 `providerId::modelId`,避免不同供应商下的同名模型发生碰撞。
|
|
28
|
+
- 图像、视频、语音、转写和视觉能力分别配置,不再复用含义不同的能力槽位。
|
|
29
|
+
- 工作台和 Agent 的视频生成统一使用同一套动作实现和参数校验。
|
|
30
|
+
- 生成类故障转移只发生在上传、提交或同步生成失败时;远端受理后不再自动重提。
|
|
31
|
+
- 供应商更新采用字段级合并,未提交的字段会保留原值。
|
|
32
|
+
- npm 包显式包含 README、用户指南、变更记录和 `docs/`,Node.js 版本要求调整为 `>=20.9.0`。
|
|
33
|
+
|
|
34
|
+
### Fixed
|
|
35
|
+
|
|
36
|
+
- 修复新版 Qwen Image、Wan 2.6/2.7 和 Z-Image 被错误发送到旧文生图端点的问题,并兼容新版图片结果结构。
|
|
37
|
+
- 纯图像编辑模型不再被自动分配给文生图能力。
|
|
38
|
+
- 修复文件转写使用错误模型与请求字段、未下载 `transcription_url` 结果的问题。
|
|
39
|
+
- 修复局部更新供应商配置时覆盖已有配置的问题。
|
|
40
|
+
- 修复旧任务附件在索引裁剪或重启后无法正确查找的问题。
|
|
41
|
+
- 修复上传中断遗留临时文件、过期上传未及时回收和文件权限不一致的问题。
|
|
42
|
+
- 修复取消信号未完整传递到上传、生成等待和后台轮询的问题。
|
|
43
|
+
- 修复 SSE 测试依赖固定等待时间而偶发失败的问题。
|
|
44
|
+
- 修复工作台与 Agent 视频参数和返回结果不一致的问题。
|
|
45
|
+
|
|
46
|
+
### Security
|
|
47
|
+
|
|
48
|
+
- HTML 截图页面改为不具备同源权限的沙箱环境,页面脚本无法访问 Iris 或宿主 API。
|
|
49
|
+
- 上传采用流式限额、临时 `.part` 文件和原子落盘,失败时会清理未完成文件。
|
|
50
|
+
- 修改状态的路由拒绝明确的跨站请求,媒体文件继续使用随机能力令牌访问。
|
|
51
|
+
|
|
52
|
+
[Unreleased]: https://github.com/mokuyoaxis/dsh-iris/compare/v0.1.0...HEAD
|
|
53
|
+
[0.1.0]: https://github.com/mokuyoaxis/dsh-iris/releases/tag/v0.1.0
|
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2025 Messia
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,177 @@
|
|
|
1
|
+
<p align="center">
|
|
2
|
+
<img src="docs/assets/iris-wordmark.svg" width="520" alt="IRIS 彩色字标">
|
|
3
|
+
</p>
|
|
4
|
+
|
|
5
|
+
<h1 align="center">dsh-iris</h1>
|
|
6
|
+
|
|
7
|
+
<p align="center"><strong>DeepSeek Harness 的多供应商媒体与视觉工具箱</strong></p>
|
|
8
|
+
|
|
9
|
+
<p align="center">
|
|
10
|
+
<a href="#deepseek-harness-适配"><img alt="DeepSeek Harness compatible" src="https://img.shields.io/badge/DeepSeek%20Harness-compatible-4D6BFE.svg?style=flat-square"></a>
|
|
11
|
+
<a href="CHANGELOG.md"><img alt="Version 0.1.0" src="https://img.shields.io/badge/version-0.1.0-7C5CFC.svg?style=flat-square"></a>
|
|
12
|
+
<a href="https://nodejs.org/"><img alt="Node.js 20.9 or newer" src="https://img.shields.io/badge/Node.js-%3E%3D20.9-339933.svg?style=flat-square&logo=nodedotjs&logoColor=white"></a>
|
|
13
|
+
<a href="#开发与验证"><img alt="npm test" src="https://img.shields.io/badge/tests-npm%20test-2EA44F.svg?style=flat-square"></a>
|
|
14
|
+
<a href="LICENSE"><img alt="MIT License" src="https://img.shields.io/badge/license-MIT-1689FF.svg?style=flat-square"></a>
|
|
15
|
+
</p>
|
|
16
|
+
|
|
17
|
+
dsh-iris 为 Agent 和 Iris 工作台提供图像、视频、语音与视觉理解能力。它可以连接 DashScope 百炼及 OpenAI Images 兼容服务;完整配置和任务管理集中在 Iris 工作台,右下角的 Iris 泡泡提供快捷入口。
|
|
18
|
+
|
|
19
|
+
项目目前处于早期版本,接口和配置格式仍可能随版本迭代调整。
|
|
20
|
+
|
|
21
|
+
## 最快开始
|
|
22
|
+
|
|
23
|
+
已经安装 DeepSeek Harness 且 `pnpm` 在 PATH 中时,把 Iris 加入 Web profile:
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
dsh plugin --profile web add @mokuyoaxis/dsh-iris
|
|
27
|
+
dsh web
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
从当前源码目录试用时,把第一条命令中的 `@mokuyoaxis/dsh-iris` 换成 `.`。插件会在下一次 DSH 启动时装载,Web UI 默认位于 `http://127.0.0.1:3080`。
|
|
31
|
+
|
|
32
|
+
首次启动后,打开“设置 → Iris 工作台 → + 添加供应商”,填入 DashScope Base URL 和 API Key,再点“发现模型”。单供应商可以先使用自动能力分配。完整步骤、OpenAI Images 兼容配置和故障转移说明见[用户指南](user_guide.md)。
|
|
33
|
+
|
|
34
|
+
配置完成后,最小指令可以是:
|
|
35
|
+
|
|
36
|
+
```text
|
|
37
|
+
使用 iris_draw_image:画一只坐在蓝色窗边的白猫。
|
|
38
|
+
总结我刚上传的这张图片。
|
|
39
|
+
用 Iris 总结 /absolute/path/demo.mp4,再把摘要合成为语音。
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
DSH 的 profile 与插件命令由[官方安装说明](https://github.com/deepseek-ai/deepseek-harness/blob/master/docs/user/develop/basic/publish.md)定义。生成图片、视频或语音可能产生供应商费用。
|
|
43
|
+
|
|
44
|
+
## 功能概览
|
|
45
|
+
|
|
46
|
+
- 图像生成、文生视频、图生视频和 S2V 数字人视频
|
|
47
|
+
- 文本转语音与音频转写
|
|
48
|
+
- 看图问答、长图 OCR、目标定位、裁剪和像素差异分析
|
|
49
|
+
- 视频抽帧与多模态内容摘要
|
|
50
|
+
- 异步任务跟踪、重启恢复和带授权令牌的媒体播放
|
|
51
|
+
- 多供应商模型池,以及按 `供应商 + 模型` 分配能力
|
|
52
|
+
- 浏览器上传、会话附件和宿主路径三种文件来源
|
|
53
|
+
|
|
54
|
+
## 使用前准备
|
|
55
|
+
|
|
56
|
+
- 满足所用 DeepSeek Harness 版本要求的 Node.js;dsh-iris 自身最低为 20.9
|
|
57
|
+
- 一个可用的 DeepSeek Harness 环境,以及 PATH 中的 `pnpm`
|
|
58
|
+
- 至少一个受支持的媒体或视觉服务供应商
|
|
59
|
+
- 使用视频抽帧和视频摘要时,需要系统提供 `ffmpeg` 与 `ffprobe`
|
|
60
|
+
|
|
61
|
+
安装插件后,在 Iris 工作台中添加供应商并为所需能力分配模型。供应商密钥只保存在宿主侧,界面和接口不会返回完整密钥。
|
|
62
|
+
|
|
63
|
+
## DeepSeek Harness 适配
|
|
64
|
+
|
|
65
|
+
dsh-iris 按 DeepSeek Harness 插件形态提供服务端与 Web 客户端入口:服务端注册 14 个 Agent 工具,并复用宿主的工具、路由和生命周期服务;客户端通过 DSH 模块加载器接入设置页、会话输入区和全局悬浮层。插件不会启动独立服务或额外监听端口。
|
|
66
|
+
|
|
67
|
+
当前自动化测试覆盖插件装载、工具注册、客户端槽位和路由行为。由于项目尚未声明 DSH 的最低或最高版本,兼容徽章表示项目已按当前 DSH 插件接口实现,并不代表官方认证或对所有历史版本的承诺。
|
|
68
|
+
|
|
69
|
+
## 工具
|
|
70
|
+
|
|
71
|
+
### 生成、语音与任务
|
|
72
|
+
|
|
73
|
+
| 工具 | 用途 |
|
|
74
|
+
|---|---|
|
|
75
|
+
| `iris_draw_image` | 根据提示词生成图片 |
|
|
76
|
+
| `iris_generate_video` | 生成文生视频、图生视频或 S2V 数字人视频 |
|
|
77
|
+
| `iris_speak_text` | 将文本合成为语音 |
|
|
78
|
+
| `iris_transcribe_audio` | 将音频转写为文本 |
|
|
79
|
+
| `iris_task_status` | 查询单个任务或最近任务的状态和产物 |
|
|
80
|
+
|
|
81
|
+
### 视觉与图像处理
|
|
82
|
+
|
|
83
|
+
| 工具 | 用途 |
|
|
84
|
+
|---|---|
|
|
85
|
+
| `iris_look_at_image` | 对图片提问并获得视觉模型回答 |
|
|
86
|
+
| `iris_relook_attachment` | 使用新问题重新查看会话中的图片附件 |
|
|
87
|
+
| `iris_long_ocr` | 分块识别长截图或长图文字 |
|
|
88
|
+
| `iris_locate` | 定位图片中的目标并返回像素坐标 |
|
|
89
|
+
| `iris_crop` | 按像素区域裁剪图片 |
|
|
90
|
+
| `iris_pixel_diff` | 比较两张图片并生成差异统计和热力图 |
|
|
91
|
+
| `iris_html_screenshot` | 在受限环境中将 HTML 字符串渲染为截图 |
|
|
92
|
+
|
|
93
|
+
### 视频处理
|
|
94
|
+
|
|
95
|
+
| 工具 | 用途 |
|
|
96
|
+
|---|---|
|
|
97
|
+
| `iris_video_frames` | 从视频中均匀抽取画面帧 |
|
|
98
|
+
| `iris_media_summarize` | 结合画面帧和可选音频转写生成视频摘要 |
|
|
99
|
+
|
|
100
|
+
## 文件输入
|
|
101
|
+
|
|
102
|
+
Iris 支持三种文件来源,推荐顺序如下:
|
|
103
|
+
|
|
104
|
+
1. **浏览器上传**:适合本地文件和跨环境访问,单文件上限为 64 MB。
|
|
105
|
+
2. **会话附件**:适合复用当前会话中已经上传或生成的媒体。
|
|
106
|
+
3. **宿主路径**:适合宿主可以直接读取的大文件。这里填写的是宿主进程看到的路径,不一定等同于浏览器所在系统的路径。
|
|
107
|
+
|
|
108
|
+
上传内容保存在 `$DSH_HOME/iris/v1/uploads/`,默认保留 7 天。不同系统下的路径选择建议见[文件访问与跨环境](docs/file-access-across-environments.md)。
|
|
109
|
+
|
|
110
|
+
## 模型分配与故障转移
|
|
111
|
+
|
|
112
|
+
能力分配使用 `providerId::modelId` 作为模型身份。同名模型如果来自不同供应商或不同账号,会被视为两个独立选项。
|
|
113
|
+
|
|
114
|
+
生成类能力可以配置多个候选模型。Iris 只会在上传、提交或同步生成阶段失败时尝试下一个候选项;远端服务一旦受理任务,就不会自动重新提交,以免产生重复任务或重复计费。已经受理的异步任务由任务系统继续跟踪。
|
|
115
|
+
|
|
116
|
+
转写是独立的 `transcribe` 能力,不会占用 TTS 或视觉能力的模型配置。
|
|
117
|
+
|
|
118
|
+
## 数据与任务
|
|
119
|
+
|
|
120
|
+
运行数据默认位于 `$DSH_HOME/iris/v1/`:
|
|
121
|
+
|
|
122
|
+
| 位置 | 内容 |
|
|
123
|
+
|---|---|
|
|
124
|
+
| `providers.json` | 供应商和能力分配,文件权限为 0600 |
|
|
125
|
+
| `tasks.json` | 任务元数据和附件索引,最多保留 500 条 |
|
|
126
|
+
| `outputs/` | 生成和处理后的媒体文件 |
|
|
127
|
+
| `uploads/` | 浏览器上传的临时输入副本 |
|
|
128
|
+
|
|
129
|
+
异步任务会在后台轮询。插件重启后,可以继续接管仍在远端执行的任务。取消信号会传递到本地等待与轮询流程;是否能取消远端计算,取决于供应商接口。
|
|
130
|
+
|
|
131
|
+
音频和视频通过带随机令牌的 Iris 媒体链接访问。图片会尽量转存为 DSH 持久附件,方便在会话中继续使用。
|
|
132
|
+
|
|
133
|
+
## 安全说明
|
|
134
|
+
|
|
135
|
+
- 供应商密钥不会通过状态接口返回。
|
|
136
|
+
- 修改状态的 HTTP 请求会检查跨站来源,降低第三方页面触发付费操作的风险。
|
|
137
|
+
- HTML 截图在不具备同源权限的沙箱页面中渲染,脚本和外部网络默认不可用。
|
|
138
|
+
- 媒体链接使用随机能力令牌,文件路径只从任务记录解析。
|
|
139
|
+
- Iris 不提供独立账号体系,多用户隔离和访问控制由 DeepSeek Harness 部署负责。
|
|
140
|
+
|
|
141
|
+
## 已知限制
|
|
142
|
+
|
|
143
|
+
- `sharp` 包含原生组件;主要支持 glibc Linux、Windows 和 macOS。裸 Termux 等非标准运行环境可能需要额外处理。
|
|
144
|
+
- 视频抽帧和视频摘要依赖系统安装的 `ffmpeg` 与 `ffprobe`。
|
|
145
|
+
- HTML 截图不加载远程脚本、字体或图片;需要的资源应先内联。
|
|
146
|
+
- WSL、容器和远程部署中,浏览器路径通常不能直接交给宿主读取,请优先上传文件。
|
|
147
|
+
- 供应商返回的模型清单只是候选集合,模型是否真正支持某项能力仍以实际调用为准。
|
|
148
|
+
|
|
149
|
+
## Agent Skills
|
|
150
|
+
|
|
151
|
+
源码和 npm 包附带两个仓库级预览 Skill:
|
|
152
|
+
|
|
153
|
+
| Skill | 适用场景 |
|
|
154
|
+
|---|---|
|
|
155
|
+
| [`iris-verify-ui`](.dsh/skills/iris-verify-ui/SKILL.md) | 组合截图、语义检查、元素定位、裁剪和像素比较,完成有证据的 UI 验收 |
|
|
156
|
+
| [`iris-compose-media`](.dsh/skills/iris-compose-media/SKILL.md) | 组合两个以上 Iris 工具,完成看图后绘图、图片转视频、视频总结配旁白或 S2V 等媒体工作流 |
|
|
157
|
+
|
|
158
|
+
当 DSH 会话以本仓库为项目目录且启用了文件系统 Skill 服务时,可以直接发现它们。当前版本尚未在插件启用时自动向其他项目注册这些 Skill;14 个 Iris 工具不受影响。
|
|
159
|
+
|
|
160
|
+
## 开发与验证
|
|
161
|
+
|
|
162
|
+
在项目目录运行:
|
|
163
|
+
|
|
164
|
+
npm test
|
|
165
|
+
|
|
166
|
+
测试覆盖配置合并、模型身份、任务生命周期、生成动作、HTTP/SSE 路由、客户端交互和安全边界。项目采用 ES Modules,不需要构建步骤。
|
|
167
|
+
|
|
168
|
+
## 文档
|
|
169
|
+
|
|
170
|
+
- [用户指南](user_guide.md)
|
|
171
|
+
- [变更记录](CHANGELOG.md)
|
|
172
|
+
- [路线图](docs/ROADMAP.md)
|
|
173
|
+
- [文件访问与跨环境](docs/file-access-across-environments.md)
|
|
174
|
+
|
|
175
|
+
## License
|
|
176
|
+
|
|
177
|
+
[MIT](LICENSE)
|
package/cordis.patch.yml
ADDED
package/docs/ROADMAP.md
ADDED
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
# 路线图
|
|
2
|
+
|
|
3
|
+
dsh-iris 已具备完整的媒体生成、视觉处理和任务管理主流程。接下来的重点是跨平台验证、发布体验和少量高价值增强,而不是继续扩张工具数量。
|
|
4
|
+
|
|
5
|
+
变更详情见[变更记录](../CHANGELOG.md),完整工具清单见[README](../README.md)。
|
|
6
|
+
|
|
7
|
+
## 当前状态
|
|
8
|
+
|
|
9
|
+
现有功能包括:
|
|
10
|
+
|
|
11
|
+
- 图像、视频、语音生成和音频转写
|
|
12
|
+
- 看图问答、OCR、定位、裁剪与像素比较
|
|
13
|
+
- 视频抽帧和多模态摘要
|
|
14
|
+
- 多供应商模型池与复合模型身份
|
|
15
|
+
- 浏览器上传、会话附件和宿主路径输入
|
|
16
|
+
- 异步任务恢复、取消传播、SSE 状态更新和授权媒体链接
|
|
17
|
+
- 面向 Agent 与用户的统一视频生成实现
|
|
18
|
+
- 两个仓库级 Agent Skills:UI 视觉验收与多步骤媒体编排
|
|
19
|
+
|
|
20
|
+
当前包版本为 `0.1.0`,尚未建立正式发布标签。
|
|
21
|
+
|
|
22
|
+
## 下一步
|
|
23
|
+
|
|
24
|
+
### 0.2.0 发布准备
|
|
25
|
+
|
|
26
|
+
- 在原生 Windows、WSL 和常规 Linux 环境完成安装与文件输入冒烟测试。
|
|
27
|
+
- 在全新 DeepSeek Harness 环境验证插件装载、客户端入口、工具注册和路由注册。
|
|
28
|
+
- 检查 npm 包内容、公开文档链接和最低 Node.js 版本。
|
|
29
|
+
- 整理版本号、发布日期和升级说明后再创建正式发布。
|
|
30
|
+
|
|
31
|
+
### 使用体验
|
|
32
|
+
|
|
33
|
+
- 根据真实使用反馈改进文件选择器、任务历史和错误提示。
|
|
34
|
+
- 评估按媒体类型调整上传上限,兼顾大视频与远程部署。
|
|
35
|
+
- 在真实 DSH 会话验证 `iris-compose-media` 的触发与停止边界,再决定两个仓库级 Skills 的自动注册方式。
|
|
36
|
+
|
|
37
|
+
### 能力增强
|
|
38
|
+
|
|
39
|
+
以下方向会按需求和供应商支持情况安排:
|
|
40
|
+
|
|
41
|
+
- 流式语音合成
|
|
42
|
+
- 批量任务和可控并发
|
|
43
|
+
- 更丰富的图片比较指标,例如 SSIM、色彩空间和 EXIF 检查
|
|
44
|
+
- 最近使用文件和更易发现的历史产物入口
|
|
45
|
+
|
|
46
|
+
## 兼容性与限制
|
|
47
|
+
|
|
48
|
+
- Node.js 最低版本为 20.9。
|
|
49
|
+
- 图片处理依赖 `sharp`,非标准 libc 或移动终端环境可能没有可用的预编译包。
|
|
50
|
+
- 视频抽帧与视频摘要依赖 `ffmpeg` 和 `ffprobe`。
|
|
51
|
+
- WSL、远程浏览器和容器部署应优先使用上传或会话附件,避免混淆浏览器路径与宿主路径。
|
|
52
|
+
- HTML 截图运行在离线沙箱中,远程资源和页面脚本不可用。
|
|
53
|
+
- 模型发现只能列出候选项,具体能力需要通过实际请求确认。
|
|
54
|
+
|
|
55
|
+
## 质量要求
|
|
56
|
+
|
|
57
|
+
计划中的改动应满足以下条件:
|
|
58
|
+
|
|
59
|
+
- `npm test` 通过,并为新增行为补充有针对性的测试。
|
|
60
|
+
- 不泄露供应商密钥、宿主绝对路径或未授权媒体内容。
|
|
61
|
+
- 不在远端任务受理后自动重提生成请求。
|
|
62
|
+
- 不让局部配置更新覆盖未修改字段。
|
|
63
|
+
- 在支持的平台上保持 Agent 工具与工作台行为一致。
|
|
64
|
+
- 发布前检查 npm 包内容和公开文档中的相对链接。
|
|
65
|
+
|
|
66
|
+
## 暂不计划
|
|
67
|
+
|
|
68
|
+
- 在插件内实现账号、权限或多租户系统。
|
|
69
|
+
- 接管 DeepSeek Harness 的通用聊天模型配置。
|
|
70
|
+
- 启动独立后台服务或额外监听端口。
|
|
71
|
+
- 自动探测全部模型能力并触发可能计费的请求。
|
|
72
|
+
- 仅为未来可能使用的供应商预建没有调用方的适配层。
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" width="720" height="200" viewBox="0 0 720 200" role="img" aria-labelledby="title desc">
|
|
2
|
+
<title id="title">IRIS</title>
|
|
3
|
+
<desc id="desc">IRIS wordmark in a blue, violet, magenta, and coral gradient</desc>
|
|
4
|
+
<defs>
|
|
5
|
+
<linearGradient id="wordmark" x1="62" y1="44" x2="658" y2="158" gradientUnits="userSpaceOnUse">
|
|
6
|
+
<stop offset="0" stop-color="#18B7FF"/>
|
|
7
|
+
<stop offset="0.28" stop-color="#4F7CFF"/>
|
|
8
|
+
<stop offset="0.54" stop-color="#8758FF"/>
|
|
9
|
+
<stop offset="0.78" stop-color="#D946EF"/>
|
|
10
|
+
<stop offset="1" stop-color="#FF718F"/>
|
|
11
|
+
</linearGradient>
|
|
12
|
+
<linearGradient id="accent" x1="196" y1="176" x2="542" y2="176" gradientUnits="userSpaceOnUse">
|
|
13
|
+
<stop stop-color="#18B7FF" stop-opacity="0"/>
|
|
14
|
+
<stop offset="0.25" stop-color="#4F7CFF"/>
|
|
15
|
+
<stop offset="0.72" stop-color="#D946EF"/>
|
|
16
|
+
<stop offset="1" stop-color="#FF718F" stop-opacity="0"/>
|
|
17
|
+
</linearGradient>
|
|
18
|
+
<filter id="soft-glow" x="-15%" y="-40%" width="130%" height="180%">
|
|
19
|
+
<feGaussianBlur stdDeviation="10"/>
|
|
20
|
+
</filter>
|
|
21
|
+
</defs>
|
|
22
|
+
<text x="360" y="143" text-anchor="middle" fill="url(#wordmark)" font-family="Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif" font-size="132" font-weight="800" letter-spacing="18">IRIS</text>
|
|
23
|
+
<ellipse cx="369" cy="175" rx="205" ry="5" fill="url(#accent)" opacity="0.28" filter="url(#soft-glow)"/>
|
|
24
|
+
<path d="M196 175 C286 183 452 183 542 175" fill="none" stroke="url(#accent)" stroke-width="3" stroke-linecap="round"/>
|
|
25
|
+
</svg>
|