dsh-gemini-multimodal 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 RealAlexandreAI
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md ADDED
@@ -0,0 +1,73 @@
1
+ <p align="center">
2
+ <img src="assets/readme/hero.svg" alt="dsh-gemini-multimodal — eyes and ears for DeepSeek Harness" width="100%">
3
+ </p>
4
+
5
+ # dsh-gemini-multimodal
6
+
7
+ Gives DeepSeek Harness **multimodal perception**: the harness model (e.g. DeepSeek) is text-only, so this plugin delegates seeing/hearing to **Gemini** — image/audio/video/document understanding, transcription, and image generation.
8
+
9
+ [English](README.md) · [中文](README.zh.md)
10
+
11
+ ## Why
12
+
13
+ DeepSeek models are text-only. When a task involves a screenshot, a chart, a voice memo, a video clip, or a PDF, this plugin lets the agent actually look and listen — and keep the judgment in the main model.
14
+
15
+ ## Two providers (pick one)
16
+
17
+ | provider | what it needs | when to use |
18
+ |---|---|---|
19
+ | `gemini_api` | a Gemini API key from **aistudio.google.com** | default; direct REST, fast, one HTTP call |
20
+ | `antigravity_cli` | local [`agy`](https://antigravity.google) CLI, signed in with your Google account | no key to manage; uses your Antigravity/Google One quota |
21
+
22
+ **Where to get the key (gemini_api):** go to `aistudio.google.com` → sign in with your Google account → **Get API key** → Create API key. Keys start with `AIza...` or `AQ.` — both work. Pro/Google One subscribers get the same free-tier API quota (image generation is rate-limited on the free tier).
23
+
24
+ **antigravity_cli setup:** `curl -fsSL https://antigravity.google/cli/install.sh | bash`, then run `agy` once and sign in. No key needed.
25
+
26
+ ## Quick start
27
+
28
+ ```sh
29
+ dsh plugin --profile web add dsh-gemini-multimodal
30
+ ```
31
+
32
+ Configure in your profile/settings layer (optional — **zero-config**: with no config at all it defaults to `antigravity_cli` and works if `agy` is installed):
33
+
34
+ ```yaml
35
+ - id: gemini-multimodal
36
+ name: dsh-gemini-multimodal
37
+ config:
38
+ provider: gemini_api # optional; default antigravity_cli
39
+ api_key: <your gemini key> # only for gemini_api
40
+ # output_dir: /path/for/images # generated images land here (default OS temp)
41
+ ```
42
+
43
+ ## Tools
44
+
45
+ | tool | what it does |
46
+ |---|---|
47
+ | `media_understand` | analyze an image / audio / video / URL (OCR, charts, UI, spoken words) |
48
+ | `media_transcribe` | verbatim text of audio/video, timestamps where useful |
49
+ | `image_generate` | text prompt → PNG saved locally (free-tier image quota is low; 429s possible) |
50
+ | `read_document` | summarize or answer questions about PDF / Office / text files |
51
+
52
+ ## Security notes (antigravity_cli)
53
+
54
+ Headless `agy -p` cannot prompt for permissions, so the plugin passes `--dangerously-skip-permissions` by default. That grants the agent full tool access for that run. To tighten: set `skip_permissions: false` and add allow-rules in `~/.gemini/antigravity-cli/settings.json` (e.g. `read_file(...)`, deny `shell`). Prefer `gemini_api` when you need a narrower trust boundary.
55
+
56
+ ## Privacy
57
+
58
+ - `gemini_api`: media is sent inline (base64) to Google's Gemini API — nothing stored by the plugin, key lives in your config only.
59
+ - `antigravity_cli`: media is read by the local agent and sent to Google per the agy flow.
60
+ - Nothing is logged by this plugin.
61
+
62
+ ## Development
63
+
64
+ ```bash
65
+ npm install
66
+ npm run typecheck
67
+ npm test # mime mapping / agy args / smoke (no network)
68
+ npm run build
69
+ ```
70
+
71
+ ## License
72
+
73
+ MIT
package/README.zh.md ADDED
@@ -0,0 +1,67 @@
1
+ # dsh-gemini-multimodal
2
+
3
+ 给 DeepSeek Harness 装上**多模态感知**:harness 主模型(如 DeepSeek)是纯文本的,这个插件把"看/听"外包给 **Gemini** —— 图片/音频/视频/文档理解、转录、图片生成。
4
+
5
+ ## 为什么
6
+
7
+ DeepSeek 模型只支持文本。当任务涉及截图、图表、语音、视频片段或 PDF 时,这个插件让 agent 真的能看能听——判断仍由主模型负责。
8
+
9
+ ## 两种 provider(选一个)
10
+
11
+ | provider | 需要什么 | 什么时候用 |
12
+ |---|---|---|
13
+ | `gemini_api` | **aistudio.google.com** 申请的 Gemini API key | 默认;直接 REST,快,一次 HTTP |
14
+ | `antigravity_cli` | 本地 [`agy`](https://antigravity.google) CLI,用 Google 账号登录 | 不用管 key;消耗 Antigravity/Google One 额度 |
15
+
16
+ **key 去哪搞(gemini_api):** 打开 `aistudio.google.com` → 用你的 Google 账号登录 → **Get API key** → Create API key。key 以 `AIza...` 或 `AQ.` 开头,两种都行。Pro/Google One 订阅同样走 API 免费层(图片生成在免费层限流)。
17
+
18
+ **antigravity_cli 配置:** `curl -fsSL https://antigravity.google/cli/install.sh | bash`,然后跑一次 `agy` 并登录,无需 key。
19
+
20
+ ## 快速开始
21
+
22
+ ```sh
23
+ dsh plugin --profile web add dsh-gemini-multimodal
24
+ ```
25
+
26
+ 在 profile/settings 层配置(**可省略——零配置**默认走 `antigravity_cli`,本地装了 agy 就能直接用):
27
+
28
+ ```yaml
29
+ - id: gemini-multimodal
30
+ name: dsh-gemini-multimodal
31
+ config:
32
+ provider: gemini_api # 可省略;默认 antigravity_cli
33
+ api_key: <你的 gemini key> # 仅 gemini_api 需要
34
+ # output_dir: /图片输出目录 # 生成图存放处(默认系统临时目录)
35
+ ```
36
+
37
+ ## 工具
38
+
39
+ | 工具 | 作用 |
40
+ |---|---|
41
+ | `media_understand` | 分析图片/音频/视频/URL(OCR、图表、界面、语音内容) |
42
+ | `media_transcribe` | 音频/视频逐字转录(带时间戳) |
43
+ | `image_generate` | 文本 → PNG 存本地(免费层出图配额低,可能 429) |
44
+ | `read_document` | 总结/问答 PDF、Office、文本文件 |
45
+
46
+ ## 安全说明(antigravity_cli)
47
+
48
+ headless `agy -p` 无法弹窗审批,插件默认带 `--dangerously-skip-permissions`(该次运行 agent 有完整工具权限)。要收紧:设 `skip_permissions: false`,并在 `~/.gemini/antigravity-cli/settings.json` 加 allow 规则(如 `read_file(...)`、拒绝 `shell`)。需要更小信任边界时优先用 `gemini_api`。
49
+
50
+ ## 隐私
51
+
52
+ - `gemini_api`:媒体以 base64 内联发给 Google Gemini API——插件不落盘,key 只在你配置文件里。
53
+ - `antigravity_cli`:媒体由本地 agent 读取,按 agy 流程发给 Google。
54
+ - 本插件不写任何日志。
55
+
56
+ ## 开发
57
+
58
+ ```bash
59
+ npm install
60
+ npm run typecheck
61
+ npm test # mime 映射 / agy 参数 / smoke(无网络)
62
+ npm run build
63
+ ```
64
+
65
+ ## License
66
+
67
+ MIT
@@ -0,0 +1,12 @@
1
+ # Plugin bundle for dsh-gemini-multimodal.
2
+ # Add this bundle to a profile to expose Gemini multimodal tools.
3
+ # Configure the provider in the profile/settings layer:
4
+ #
5
+ # - id: gemini-multimodal
6
+ # name: 'dsh-gemini-multimodal'
7
+ # config:
8
+ # provider: gemini_api # or antigravity_cli
9
+ # api_key: <your gemini key> # required for gemini_api
10
+ - insert:
11
+ - id: gemini-multimodal
12
+ name: 'dsh-gemini-multimodal'
@@ -0,0 +1,20 @@
1
+ import { Context } from '@deepseek-ai/cordis';
2
+ import z from '@deepseek-ai/schemastery';
3
+ export declare const name = "gemini-multimodal";
4
+ export declare const inject: string[];
5
+ export interface Config {
6
+ /** Backend: `gemini_api` (REST + key) or `antigravity_cli` (local agy).
7
+ * Optional — defaults to antigravity_cli (zero-config: local agy, no key). */
8
+ provider?: string;
9
+ /** Gemini API key — required only when provider is gemini_api. */
10
+ api_key?: string;
11
+ /** Image generation model (default gemini-2.5-flash-image). */
12
+ image_model?: string;
13
+ /** Where generated images land (default OS temp). */
14
+ output_dir?: string;
15
+ /** agy headless: allow full tool access (default true). Set false to rely
16
+ * on permissions.allow rules in ~/.gemini/antigravity-cli/settings.json. */
17
+ skip_permissions?: boolean;
18
+ }
19
+ export declare const Config: z<Config>;
20
+ export declare function apply(ctx: Context, config: Config): void;
package/dist/index.js ADDED
@@ -0,0 +1,102 @@
1
+ // dsh-gemini-multimodal — DeepSeek Harness (Cordis) plugin.
2
+ //
3
+ // Multimodal tools for dsh: the harness model (e.g. DeepSeek) is text-only, so
4
+ // media understanding / transcription / document reading / image generation
5
+ // are delegated to Gemini. Two interchangeable providers:
6
+ // - `gemini_api` : direct Gemini REST, needs an API key
7
+ // (aistudio.google.com — AQ./AIza keys both work)
8
+ // - `antigravity_cli` : local `agy` binary on PATH (Google account sign-in,
9
+ // no key needed)
10
+ //
11
+ // Tools:
12
+ // media_understand(source, question?) image/audio/video/URL → analysis
13
+ // media_transcribe(source) audio/video → verbatim text
14
+ // image_generate(prompt) text → PNG (saved locally)
15
+ // read_document(path, question?) PDF/office/text → summary/answer
16
+ //
17
+ // Privacy: credentials come from plugin config only, never logged.
18
+ import { defineTool } from '@deepseek-ai/dsh-tools';
19
+ import z from '@deepseek-ai/schemastery';
20
+ import { runProvider } from './providers.js';
21
+ export const name = 'gemini-multimodal';
22
+ export const inject = ['tools', 'systemPrompt'];
23
+ export const Config = z.object({
24
+ provider: z.string().description('Backend: gemini_api (REST + key) or antigravity_cli (local agy). Default: antigravity_cli'),
25
+ api_key: z.string().description('Gemini API key (aistudio.google.com). Required for gemini_api provider.'),
26
+ image_model: z.string().description('Image generation model (default gemini-2.5-flash-image)'),
27
+ output_dir: z.string().description('Directory for generated images (default OS temp)'),
28
+ skip_permissions: z.boolean().description('agy headless: allow full tool access (default true)'),
29
+ });
30
+ function toolResultText(value) {
31
+ return typeof value === 'string' ? value : JSON.stringify(value);
32
+ }
33
+ const OUTPUT = {
34
+ schema: { type: 'json' },
35
+ render: (_args, value) => [{ type: 'text', text: toolResultText(value) }],
36
+ };
37
+ export function apply(ctx, config) {
38
+ // Zero-config friendly: no provider configured -> tools default to
39
+ // antigravity_cli (local agy) and surface a setup hint on first call if the
40
+ // CLI is missing. Always loads, never throws on missing config.
41
+ ctx.systemPrompt.section({
42
+ name: 'tool:gemini-multimodal',
43
+ order: 116,
44
+ text: 'You can delegate perception to Gemini (multimodal): media_understand for ' +
45
+ 'images/audio/video/URLs, media_transcribe for verbatim audio/video text, ' +
46
+ 'image_generate to create images, read_document for PDF/office files. Use ' +
47
+ 'them when the task needs eyes or ears that the main model lacks.',
48
+ });
49
+ const run = (action, arg, exec) => runProvider({
50
+ provider: config.provider,
51
+ apiKey: config.api_key,
52
+ imageModel: config.image_model,
53
+ outputDir: config.output_dir,
54
+ skipPermissions: config.skip_permissions,
55
+ signal: exec.signal,
56
+ }, action, arg);
57
+ const register = (tool) => {
58
+ ctx.tools.register(defineTool(tool));
59
+ };
60
+ register({
61
+ name: 'media_understand',
62
+ description: 'Understand an image, audio, video, or remote URL via Gemini. Returns a text analysis (content, OCR, charts, UI, spoken words). Accepts a local path or http(s) URL.',
63
+ parameters: {
64
+ source: { type: 'string', required: true, description: 'Local file path or http(s) URL of the media' },
65
+ question: { type: 'string', description: 'Optional specific question to answer about the media' },
66
+ },
67
+ output: OUTPUT,
68
+ isConcurrencySafe: () => true,
69
+ execute: (args, exec) => run('understand', { source: String(args.source ?? ''), question: args.question ? String(args.question) : undefined }, exec),
70
+ });
71
+ register({
72
+ name: 'media_transcribe',
73
+ description: 'Transcribe an audio or video file to verbatim text (with timestamps where useful) via Gemini. Accepts a local path or http(s) URL.',
74
+ parameters: {
75
+ source: { type: 'string', required: true, description: 'Local audio/video path or http(s) URL' },
76
+ },
77
+ output: OUTPUT,
78
+ isConcurrencySafe: () => true,
79
+ execute: (args, exec) => run('transcribe', { source: String(args.source ?? '') }, exec),
80
+ });
81
+ register({
82
+ name: 'image_generate',
83
+ description: 'Generate an image from a text prompt (Gemini Nano Banana). Saves a PNG locally and returns the file path. Note: free-tier image quota is very low (429s are common).',
84
+ parameters: {
85
+ prompt: { type: 'string', required: true, description: 'Image description / prompt' },
86
+ },
87
+ output: OUTPUT,
88
+ isConcurrencySafe: () => true,
89
+ execute: (args, exec) => run('image_generate', { prompt: String(args.prompt ?? '') }, exec),
90
+ });
91
+ register({
92
+ name: 'read_document',
93
+ description: 'Read a PDF, Office, or text document via Gemini and summarize it or answer a question about it. Accepts a local path.',
94
+ parameters: {
95
+ source: { type: 'string', required: true, description: 'Local document path (pdf/docx/txt/md/csv/...) or URL' },
96
+ question: { type: 'string', description: 'Optional specific question about the document' },
97
+ },
98
+ output: OUTPUT,
99
+ isConcurrencySafe: () => true,
100
+ execute: (args, exec) => run('read_document', { source: String(args.source ?? ''), question: args.question ? String(args.question) : undefined }, exec),
101
+ });
102
+ }
@@ -0,0 +1,37 @@
1
+ export type ProviderName = 'gemini_api' | 'antigravity_cli';
2
+ export interface ProviderOptions {
3
+ /** Backend. Optional — defaults to antigravity_cli (zero-config). */
4
+ provider?: ProviderName;
5
+ /** API key — required only for `gemini_api`. */
6
+ apiKey?: string;
7
+ /** Image generation model (default gemini-2.5-flash-image / Nano Banana). */
8
+ imageModel?: string;
9
+ /** Directory for generated images (default OS temp). */
10
+ outputDir?: string;
11
+ /** agy headless: pass --dangerously-skip-permissions (default true). */
12
+ skipPermissions?: boolean;
13
+ /** Cooperative cancellation from the tool executor. */
14
+ signal?: AbortSignal;
15
+ }
16
+ export interface ProviderResult {
17
+ ok: boolean;
18
+ /** Human-readable output (description / transcript / analysis). */
19
+ text?: string;
20
+ /** Local path when a file artifact (image) was produced. */
21
+ imagePath?: string;
22
+ error?: string;
23
+ }
24
+ export declare function guessMimeType(pathOrUrl: string): string;
25
+ /** Low-level Gemini generateContent call. Exported for tests/tools. */
26
+ export declare function geminiGenerate(apiKey: string, model: string, parts: unknown[], signal?: AbortSignal): Promise<{
27
+ text?: string;
28
+ inlineImage?: Uint8Array;
29
+ }>;
30
+ /** Build the agy argv for headless mode. Exported for tests. */
31
+ export declare function agyArgs(opts: ProviderOptions, prompt: string): string[];
32
+ export type MediaAction = 'understand' | 'transcribe' | 'read_document' | 'image_generate';
33
+ export declare function runProvider(opts: ProviderOptions, action: MediaAction, arg: {
34
+ source?: string;
35
+ question?: string;
36
+ prompt?: string;
37
+ }): Promise<ProviderResult>;
@@ -0,0 +1,237 @@
1
+ // dsh-gemini-multimodal — providers.
2
+ //
3
+ // Two interchangeable backends:
4
+ // gemini_api — direct Gemini REST (generateContent) with an API key
5
+ // from aistudio.google.com (AQ. / AIza keys both work).
6
+ // antigravity_cli — local Antigravity CLI (`agy` on PATH), no key needed
7
+ // once signed in. Headless mode requires permission
8
+ // bypass for tool access (see skipPermissions).
9
+ //
10
+ // Privacy: credentials come from plugin config only; nothing is logged.
11
+ import { readFileSync, mkdirSync, writeFileSync } from 'fs';
12
+ import { extname, join } from 'path';
13
+ import { tmpdir } from 'os';
14
+ import { spawn } from 'child_process';
15
+ const GEMINI_BASE = 'https://generativelanguage.googleapis.com/v1beta';
16
+ const DEFAULT_MODEL = 'gemini-3-flash-preview';
17
+ const DEFAULT_IMAGE_MODEL = 'gemini-2.5-flash-image';
18
+ const MAX_INLINE_BYTES = 19 * 1024 * 1024; // Gemini inline_data limit ~20MB
19
+ const REQUEST_TIMEOUT_MS = 120_000;
20
+ const MIME_BY_EXT = {
21
+ png: 'image/png',
22
+ jpg: 'image/jpeg',
23
+ jpeg: 'image/jpeg',
24
+ webp: 'image/webp',
25
+ gif: 'image/gif',
26
+ bmp: 'image/bmp',
27
+ heic: 'image/heic',
28
+ mp4: 'video/mp4',
29
+ mov: 'video/quicktime',
30
+ webm: 'video/webm',
31
+ mkv: 'video/x-matroska',
32
+ m4v: 'video/x-m4v',
33
+ mp3: 'audio/mpeg',
34
+ wav: 'audio/wav',
35
+ m4a: 'audio/mp4',
36
+ flac: 'audio/flac',
37
+ ogg: 'audio/ogg',
38
+ opus: 'audio/opus',
39
+ pdf: 'application/pdf',
40
+ txt: 'text/plain',
41
+ md: 'text/markdown',
42
+ csv: 'text/csv',
43
+ json: 'application/json',
44
+ };
45
+ export function guessMimeType(pathOrUrl) {
46
+ const clean = pathOrUrl.split('?')[0];
47
+ const ext = extname(clean).slice(1).toLowerCase();
48
+ return MIME_BY_EXT[ext] ?? 'application/octet-stream';
49
+ }
50
+ function isRemote(source) {
51
+ return /^https?:\/\//i.test(source);
52
+ }
53
+ /** Fetch a remote file into memory; local paths are read from disk. */
54
+ async function loadBytes(source, signal) {
55
+ if (isRemote(source)) {
56
+ const res = await fetch(source, { signal });
57
+ if (!res.ok)
58
+ throw new Error(`failed to download ${source}: HTTP ${res.status}`);
59
+ return new Uint8Array(await res.arrayBuffer());
60
+ }
61
+ const data = readFileSync(source);
62
+ if (data.byteLength > MAX_INLINE_BYTES) {
63
+ throw new Error(`file too large for inline upload (${data.byteLength} bytes > ${MAX_INLINE_BYTES})`);
64
+ }
65
+ return data;
66
+ }
67
+ // ---------------------------------------------------------------------------
68
+ // gemini_api — direct REST
69
+ // ---------------------------------------------------------------------------
70
+ /** Low-level Gemini generateContent call. Exported for tests/tools. */
71
+ export async function geminiGenerate(apiKey, model, parts, signal) {
72
+ const url = `${GEMINI_BASE}/models/${model}:generateContent`;
73
+ const body = { contents: [{ parts }] };
74
+ const res = await fetch(url, {
75
+ method: 'POST',
76
+ headers: {
77
+ 'Content-Type': 'application/json',
78
+ 'x-goog-api-key': apiKey,
79
+ },
80
+ body: JSON.stringify(body),
81
+ signal: AbortSignal.any([AbortSignal.timeout(REQUEST_TIMEOUT_MS), ...(signal ? [signal] : [])]),
82
+ });
83
+ if (!res.ok) {
84
+ const detail = (await res.text()).slice(0, 300);
85
+ const msg = detail.match(/"message"\s*:\s*"([^"]+)"/)?.[1] ?? detail;
86
+ const hint = res.status === 429
87
+ ? ' (quota exceeded — image generation free tier is very low; retry later or enable billing at aistudio.google.com)'
88
+ : res.status === 400
89
+ ? ' (bad request — unsupported media type or model?)'
90
+ : '';
91
+ throw new Error(`Gemini HTTP ${res.status}: ${msg}${hint}`);
92
+ }
93
+ const data = (await res.json());
94
+ const parts0 = data.candidates?.[0]?.content?.parts ?? [];
95
+ const text = parts0.map((p) => p.text ?? '').join('');
96
+ const inline = parts0.find((p) => p.inlineData?.data);
97
+ return { text, inlineImage: inline?.inlineData?.data ? Uint8Array.from(Buffer.from(inline.inlineData.data, 'base64')) : undefined };
98
+ }
99
+ async function geminiUnderstand(opts, source, question) {
100
+ try {
101
+ const bytes = await loadBytes(source, opts.signal);
102
+ const b64 = Buffer.from(bytes).toString('base64');
103
+ const parts = [
104
+ { inlineData: { mimeType: guessMimeType(source), data: b64 } },
105
+ { text: question || 'Describe this media in detail (content, text, and any key facts).' },
106
+ ];
107
+ const { text } = await geminiGenerate(opts.apiKey, DEFAULT_MODEL, parts, opts.signal);
108
+ return { ok: true, text: text ?? '(empty response)' };
109
+ }
110
+ catch (e) {
111
+ return { ok: false, error: e instanceof Error ? e.message : String(e) };
112
+ }
113
+ }
114
+ async function geminiTranscribe(opts, source) {
115
+ try {
116
+ const bytes = await loadBytes(source, opts.signal);
117
+ const b64 = Buffer.from(bytes).toString('base64');
118
+ const parts = [
119
+ { inlineData: { mimeType: guessMimeType(source), data: b64 } },
120
+ { text: 'Transcribe this audio/video to text verbatim, with timestamps where useful.' },
121
+ ];
122
+ const { text } = await geminiGenerate(opts.apiKey, DEFAULT_MODEL, parts, opts.signal);
123
+ return { ok: true, text: text ?? '(empty response)' };
124
+ }
125
+ catch (e) {
126
+ return { ok: false, error: e instanceof Error ? e.message : String(e) };
127
+ }
128
+ }
129
+ async function geminiReadDocument(opts, path, question) {
130
+ try {
131
+ const bytes = await loadBytes(path, opts.signal);
132
+ const b64 = Buffer.from(bytes).toString('base64');
133
+ const parts = [
134
+ { inlineData: { mimeType: guessMimeType(path), data: b64 } },
135
+ { text: question || 'Summarize this document: main points, structure, and anything actionable.' },
136
+ ];
137
+ const { text } = await geminiGenerate(opts.apiKey, DEFAULT_MODEL, parts, opts.signal);
138
+ return { ok: true, text: text ?? '(empty response)' };
139
+ }
140
+ catch (e) {
141
+ return { ok: false, error: e instanceof Error ? e.message : String(e) };
142
+ }
143
+ }
144
+ async function geminiGenerateImage(opts, prompt) {
145
+ try {
146
+ const model = opts.imageModel ?? DEFAULT_IMAGE_MODEL;
147
+ const { inlineImage } = await geminiGenerate(opts.apiKey, model, [{ text: prompt }], opts.signal);
148
+ if (!inlineImage)
149
+ return { ok: false, error: 'no image returned by the model' };
150
+ const dir = opts.outputDir ?? join(tmpdir(), 'dsh-gemini-multimodal');
151
+ mkdirSync(dir, { recursive: true });
152
+ const file = join(dir, `gemini-${Date.now()}-${Math.random().toString(36).slice(2, 8)}.png`);
153
+ writeFileSync(file, inlineImage);
154
+ return { ok: true, text: `Image saved to ${file}`, imagePath: file };
155
+ }
156
+ catch (e) {
157
+ return { ok: false, error: e instanceof Error ? e.message : String(e) };
158
+ }
159
+ }
160
+ // ---------------------------------------------------------------------------
161
+ // antigravity_cli — local `agy` on PATH
162
+ // ---------------------------------------------------------------------------
163
+ /** Build the agy argv for headless mode. Exported for tests. */
164
+ export function agyArgs(opts, prompt) {
165
+ const args = ['-p', prompt];
166
+ if (opts.skipPermissions !== false)
167
+ args.push('--dangerously-skip-permissions');
168
+ return args;
169
+ }
170
+ function runAgy(opts, prompt) {
171
+ return new Promise((resolve) => {
172
+ const child = spawn('agy', agyArgs(opts, prompt), { stdio: ['ignore', 'pipe', 'pipe'] });
173
+ let out = '';
174
+ let err = '';
175
+ const timer = setTimeout(() => child.kill('SIGTERM'), REQUEST_TIMEOUT_MS + 30_000);
176
+ child.stdout?.on('data', (d) => (out += String(d)));
177
+ child.stderr?.on('data', (d) => (err += String(d)));
178
+ child.on('error', () => {
179
+ clearTimeout(timer);
180
+ resolve({
181
+ ok: false,
182
+ error: 'agy not found on PATH. Install: curl -fsSL https://antigravity.google/cli/install.sh | bash, ' +
183
+ 'then run `agy` once and sign in (no key needed). ' +
184
+ 'Alternatively set provider: gemini_api + api_key (aistudio.google.com).',
185
+ });
186
+ });
187
+ child.on('close', (code) => {
188
+ clearTimeout(timer);
189
+ const text = out.trim();
190
+ if (!text)
191
+ resolve({ ok: false, error: err.trim() || `agy exited with code ${code}` });
192
+ else
193
+ resolve({ ok: true, text });
194
+ });
195
+ });
196
+ }
197
+ function agyUnderstand(opts, source, question) {
198
+ return runAgy(opts, `Analyze this media (${source}) and answer: ${question || 'describe its content in detail.'}`);
199
+ }
200
+ function agyTranscribe(opts, source) {
201
+ return runAgy(opts, `Transcribe this media file (${source}) to text verbatim, with timestamps where useful.`);
202
+ }
203
+ function agyReadDocument(opts, path, question) {
204
+ return runAgy(opts, `Read the document at ${path} and ${question ? `answer: ${question}` : 'summarize its main points.'}`);
205
+ }
206
+ async function agyGenerateImage(opts, prompt) {
207
+ const dir = opts.outputDir ?? join(tmpdir(), 'dsh-gemini-multimodal');
208
+ mkdirSync(dir, { recursive: true });
209
+ const r = await runAgy(opts, `Generate an image of: ${prompt}. Save it as a PNG file inside ${dir} and reply with the full saved file path only.`);
210
+ if (!r.ok)
211
+ return r;
212
+ const m = r.text?.match(/([\w./~-]+\.png)/i);
213
+ return m ? { ok: true, text: `Image saved to ${m[1]}`, imagePath: m[1] } : r;
214
+ }
215
+ export function runProvider(opts, action, arg) {
216
+ // No provider configured -> default to antigravity_cli so the plugin works
217
+ // with zero config (local agy, no key).
218
+ const provider = opts.provider ?? 'antigravity_cli';
219
+ if (provider === 'gemini_api') {
220
+ if (!opts.apiKey) {
221
+ return Promise.resolve({ ok: false, error: 'provider gemini_api needs api_key — get one at aistudio.google.com, or switch to antigravity_cli (local agy, no key).' });
222
+ }
223
+ switch (action) {
224
+ case 'understand': return geminiUnderstand(opts, arg.source, arg.question ?? '');
225
+ case 'transcribe': return geminiTranscribe(opts, arg.source);
226
+ case 'read_document': return geminiReadDocument(opts, arg.source, arg.question ?? '');
227
+ case 'image_generate': return geminiGenerateImage(opts, arg.prompt);
228
+ }
229
+ }
230
+ // antigravity_cli
231
+ switch (action) {
232
+ case 'understand': return agyUnderstand(opts, arg.source, arg.question ?? '');
233
+ case 'transcribe': return agyTranscribe(opts, arg.source);
234
+ case 'read_document': return agyReadDocument(opts, arg.source, arg.question ?? '');
235
+ case 'image_generate': return agyGenerateImage(opts, arg.prompt);
236
+ }
237
+ }
package/package.json ADDED
@@ -0,0 +1,54 @@
1
+ {
2
+ "name": "dsh-gemini-multimodal",
3
+ "version": "0.1.0",
4
+ "description": "DeepSeek Harness plugin: multimodal tools (image/audio/video/document understanding, transcription, image generation) via Gemini API or the local Antigravity CLI.",
5
+ "type": "module",
6
+ "main": "dist/index.js",
7
+ "exports": {
8
+ ".": "./dist/index.js"
9
+ },
10
+ "files": [
11
+ "dist",
12
+ "cordis.patch.yml",
13
+ "README.md",
14
+ "LICENSE"
15
+ ],
16
+ "scripts": {
17
+ "build": "tsc -p tsconfig.json",
18
+ "test": "node --import tsx --test tests/*.mjs",
19
+ "typecheck": "tsc --noEmit"
20
+ },
21
+ "keywords": [
22
+ "dsh",
23
+ "dsh-plugin",
24
+ "deepseek-harness",
25
+ "cordis",
26
+ "gemini",
27
+ "antigravity",
28
+ "multimodal",
29
+ "vision"
30
+ ],
31
+ "license": "MIT",
32
+ "engines": {
33
+ "node": ">=20.0.0"
34
+ },
35
+ "dependencies": {
36
+ "@deepseek-ai/cordis": "^4.0.1",
37
+ "@deepseek-ai/dsh-tools": "^0.0.1-rc.1",
38
+ "@deepseek-ai/schemastery": "^3.18.0"
39
+ },
40
+ "devDependencies": {
41
+ "@types/node": "^22.0.0",
42
+ "tsx": "^4.19.0",
43
+ "typescript": "^5.5.0"
44
+ },
45
+ "dsh": {
46
+ "bundle": {
47
+ "patch": "./cordis.patch.yml"
48
+ }
49
+ },
50
+ "repository": {
51
+ "type": "git",
52
+ "url": "https://github.com/RealAlexandreAI/dsh-gemini-multimodal"
53
+ }
54
+ }