dsh-audiogen 0.3.4 → 0.4.0

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "dsh-audiogen",
   "description": "AI audio generation plugin for the dsh web GUI: multi-vendor TTS/music/sound-effect channels (OpenAI-compatible, ElevenLabs, MiniMax, Stability AI and custom), per-channel model/voice catalogs, Agent tool and a sidebar AI 音频 panel.",
-  "version": "0.3.4",
+  "version": "0.4.0",
   "type": "module",
   "main": "lib/index.js",
   "exports": {

package/skills/design/SKILL.md

@@ -7,7 +7,13 @@
 - 帮助用户把音色或音效需求细化成可复用的描述，供后续 TTS/music/sfx 生成。
 - 输出不建议直接生成音频，而是给出音色参数建议和可用的模型/音色清单。
 
+## 面板/工具的音色设计（voice_design）
+- 渠道支持：MiniMax（POST /v1/voice_design，prompt + preview_text）与 ElevenLabs（POST /v1/text-to-voice/design）。
+- MiniMax 参数：`prompt`（音色描述）、`preview_text`（试听文本，任意长度）。
+- ElevenLabs 参数：`voice_description`（音色描述）；试听文本 `preview_text` 需 100-1000 字符，过短时自动 `auto_generate_text`；响应含 `previews[].audio_base_64` 与 `generated_voice_id`（可后续在 TTS 中复用该 voice_id）。
+- 面板音色设计模式顶部可切换「厂商 / 渠道」。
+
 ## 流程
 1. 询问目标风格、音域、情绪、适用场景。
 2. 生成结构化的音色描述（如温暖复古合成器、未来感 UI 提示音）。
-3. 在后续生成中复用该描述，必要时通过 `generate_audio` 试听。
+3. 在后续生成中复用该描述，必要时通过 `generate_audio`（mode=voice_design）试听。

package/skills/music/SKILL.md

@@ -37,3 +37,40 @@
 ## 常见错误
 - `lyrics-required`：MiniMax 音乐生成需要歌词，或开启纯音乐。
 - `HTTP 400` 且含 `2013`：上游参数不合法，检查 lyrics / audio_setting 枚举。
+
+## ElevenLabs Music（POST /v1/music，模型 music_v1 / music_v2）
+
+| 字段 | 工具/面板参数 | 说明 |
+| --- | --- | --- |
+| model_id | model（music_v2 等） | 官方枚举：music_v1 / music_v2，默认 music_v1 |
+| prompt | prompt | 音乐/歌词主题描述（不能与 composition_plan 同用；引擎用 prompt） |
+| music_length_ms | duration（秒，自动×1000） | 3000ms - 600000ms（3s-600s），超出自动收敛区间 |
+| lyrics_text | lyrics（歌词） | 选填歌词文本 |
+| force_instrumental | is_instrumental（纯音乐） | true 保证无演唱（无词） |
+| seed / generation_mode / finetune_* | — | 高级字段，暂未透出 |
+
+> 响应为音频字节流（audio/*，常为 mp3）。请求同时携带 `xi-api-key` 与 `Authorization: Bearer`，以兼容 New API 类网关（官方站任一头即可）。
+
+## Stability Stable Audio（官方 v2beta，multipart/form-data）
+
+官方端点（文本到音频，TTS 描述 / 音乐 / 音效统一走该接口，不同模型参数不同）：
+- `POST /v2beta/audio/stable-audio/text-to-audio` → 模型 `stable-audio-3`（202 异步，随后轮询 GET /v2beta/audio/results/{id}）
+- `POST /v2beta/audio/stable-audio-2/text-to-audio` → 模型 `stable-audio-2` / `stable-audio-2.5`（同步返回音频）
+
+| 字段 | 工具/面板参数 | 说明 |
+| --- | --- | --- |
+| prompt | prompt | 必填，描述性提示词（乐器/情绪/风格/体裁，≤10000 字符） |
+| model | model | stable-audio-3 / stable-audio-2.5 / stable-audio-2 |
+| duration | duration | 秒数：3 ≤380（默认 190）；2/2.5 ≤190（默认 190） |
+| seed | seed | 0-4294967294，默认 0=随机；同参数同 seed 可复现 |
+| steps | steps | 采样步数：2 → 30-100（默认 50）；2.5/3 → 4-8（默认 8） |
+| cfg_scale | cfg_scale | 1-25：2 默认 7，2.5/3 默认 1；越高越贴提示词 |
+| output_format | format | mp3 / wav |
+
+> 引擎按模型自动收敛步数/时长区间；渠道 preset/apiUrl 含 `stability` 或模型名以 `stable-audio-` 开头即走官方协议（自定义渠道同样适用）。
+
+### 双通道（自动选择）
+- **官方 v2beta**：apiUrl 为 `https://api.stability.ai`（含 `/v2beta`、`/v2beta/audio` 形态）→ multipart 原生端点（2/2.5 同步、3 异步轮询）。
+- **OpenAI 兼容网关**：apiUrl 以 `/v1` 结尾或含 `/audio/speech`（如 New API）→ `POST {apiUrl}/audio/speech`，JSON：
+  `{ "model": "stable-audio-2.5", "input": "<prompt>", "output_format": "mp3", "duration": 30, "seed": 0, "steps": 8, "cfg_scale": 1 }`（网关把该模型映射到 Stable 上游）。
+- 一方返回 `404 Invalid URL`（未路由）时自动换另一方重试；参数在两种通道均按模型收敛（duration/seed/steps/cfg_scale/output_format）。

package/skills/sfx/SKILL.md

@@ -2,15 +2,29 @@
 
 ## 触发
 - `/audio:sfx <描述>`
-- 用户说“生成一个音效 / 提示音 / 環境音”
+- 用户说“生成一个音效 / 提示音 / 环境音”
 
 ## 参数
 - prompt: 必填，音效描述
-- model: 可选，已配置模型
-- duration: 可选，秒数
+- model: 可选，已配置模型（ElevenLabs：eleven_text_to_sound_v2）
+- duration: 可选，秒数（ElevenLabs 0.5-30，留空则自动）
+- loop: 可选，ElevenLabs 无缝循环音效（仅 eleven_text_to_sound_v2）
+- prompt_influence: 可选，ElevenLabs 提示词影响度 0-1（默认 0.3，越高越贴近提示词、越少随机）
 - format: 可选
 
 ## 流程
 1. 确认已配置音效生成渠道。
 2. 调用 `generate_audio`，mode=sfx。
 3. 将音频 URL 返回。
+
+## ElevenLabs Sound Generation（POST /v1/sound-generation）
+
+| 字段 | 工具/面板参数 | 说明 |
+| --- | --- | --- |
+| text | prompt | 必填，转换为音效的文本/描述 |
+| model_id | model | 官方枚举：eleven_text_to_sound_v2（默认） |
+| duration_seconds | duration | 0.5-30 秒；留空由提示词推算最优时长 |
+| loop | loop | 是否生成平滑循环音效（仅 eleven_text_to_sound_v2） |
+| prompt_influence | prompt_influence | 0-1，默认 0.3；越高越贴提示词，越低越多样 |
+
+> 响应为 audio/mpeg 二进制。请求同时携带 `xi-api-key` 与 `Authorization: Bearer`，兼容 New API 类网关。

package/src/agent-audio-tools.ts

@@ -9,14 +9,15 @@ import { defineTool } from '@deepseek-ai/dsh-tools'
 import { randomUUID } from 'node:crypto'
 import type { AudioChannel } from './audio-engine.ts'
 import { generateAudio, AudioGenError } from './audio-engine.ts'
-import { appendHistory, saveAudioFile } from './audio-store.ts'
-import type { AudioMode, GenerateAudioRequest } from './protocol.ts'
+import { appendHistory, saveAudioFile, saveToLibrary, listLibrary } from './audio-store.ts'
+import type { AudioMode, GenerateAudioRequest, LibraryType } from './protocol.ts'
 
 export interface AgentAudioToolConfig {
   enabled: boolean
   allowAgentAudioGeneration: boolean
   channels: AudioChannel[]
   defaultChannelId: string
+  autoSaveToLibrary: boolean
 }
 
 interface AgentAudioRef {
@@ -27,12 +28,19 @@ interface AgentAudioRef {
   voiceId?: string
 }
 
+/** Internal: the persisted file name, needed for library copies. */
+interface SavedAudioRef extends AgentAudioRef {
+  file: string
+}
+
 interface AgentAudioResult {
   status: string
   message: string
   mode: AudioMode
   model: string
   audio: AgentAudioRef[]
+  /** Resource-library entry ids when the audio was saved to the library. */
+  resources?: string[]
   error?: string
 }
 
@@ -57,6 +65,7 @@ const resultSchema = {
     mode: { type: 'string', required: true, enum: ['tts', 'music', 'sfx', 'voice_design'] },
     model: { type: 'string', required: true },
     audio: { type: 'array', required: true, items: audioRefSchema },
+    resources: { type: 'array', items: { type: 'string' } },
     error: { type: 'string' },
   },
 } as const
@@ -96,11 +105,17 @@ function ensureConfigured(config: AgentAudioToolConfig): void {
   if (!usable) throw new AudioGenError('Audio API credentials are not configured. Open Settings > Plugins > AI Audio, add a channel and fill its API URL and API key.', 'audio-api-not-configured')
 }
 
+/** Library type from the generation mode, with an explicit override. */
+function libraryTypeOf(mode: AudioMode, override: unknown): LibraryType {
+  if (override === 'voice' || override === 'music' || override === 'sfx' || override === 'tts') return override
+  if (mode === 'voice_design') return 'voice'
+  return mode
+}
+
 /** Register the Agent audio tool. */
 export function registerAgentAudioTools(ctx: Context, resolve: () => AgentAudioToolConfig): () => void {
   const disposer = ctx.tools.register(defineTool({
-    name: 'generate_audio',
-    description: 'Generate audio with the configured audio provider. Supports text-to-speech, music generation, sound effects and MiniMax voice design. The tool call waits for the upstream result and returns same-origin audio URLs; pass those URLs to the user for playback or download. If multiple models are configured, first ask the user which one to use or pass model explicitly.',
+    name: 'generate_audio',    description: 'Generate audio with the configured audio provider. Supports text-to-speech, music generation, sound effects and voice design (MiniMax /v1/voice_design, ElevenLabs /v1/text-to-voice/design). The tool call waits for the upstream result and returns same-origin audio URLs; pass those URLs to the user for playback or download. If multiple models are configured, first ask the user which one to use or pass model explicitly.',
     parameters: {
       prompt: { type: 'string', required: true, description: 'For tts, the text to speak. For music/sfx, a descriptive prompt.' },
       mode: { type: 'string', enum: ['tts', 'music', 'sfx', 'voice_design'], description: 'Generation mode. Defaults to tts.' },
@@ -111,6 +126,11 @@ export function registerAgentAudioTools(ctx: Context, resolve: () => AgentAudioT
       duration: { type: 'number', description: 'Requested duration in seconds for music/sfx.' },
       lyrics: { type: 'string', description: 'Lyrics for music generation (MiniMax music-3.0/music-cover). Required unless is_instrumental is true. Split verses with an empty line.' },
       is_instrumental: { type: 'boolean', description: 'Generate purely instrumental music without vocals/lyrics (MiniMax is_instrumental). When true, lyrics may be omitted.' },
+      loop: { type: 'boolean', description: 'Create a seamlessly looping sound effect (ElevenLabs sound generation loop, only for eleven_text_to_sound_v2).' },
+      prompt_influence: { type: 'number', description: 'Sound effect prompt influence 0-1 (ElevenLabs prompt_influence, default 0.3): higher follows the prompt more closely, lower is more variable.' },
+      seed: { type: 'integer', description: 'Stable Audio random seed 0-4294967294 (default 0 = random); same seed yields reproducible audio.' },
+      steps: { type: 'integer', description: 'Stable Audio sampling steps, model-dependent: stable-audio-2 30-100, stable-audio-2.5/3 4-8 (out-of-range auto-clamped).' },
+      cfg_scale: { type: 'number', description: 'Stable Audio prompt adherence 1-25 (stable-audio-2 default 7, 2.5/3 default 1); higher follows the prompt more strictly.' },
       format: { type: 'string', description: 'Output format such as mp3 or wav. MiniMax music supports mp3/wav/pcm.' },
       // ---- MiniMax TTS only (ignored by other providers) ----
       emotion: { type: 'string', description: 'MiniMax TTS emotion, e.g. happy/sad/angry/nervous/fearful/bored (voice_setting.emotion).' },
@@ -150,6 +170,11 @@ export function registerAgentAudioTools(ctx: Context, resolve: () => AgentAudioT
         },
         description: 'MiniMax TTS dual-voice blend weights (timbre_weights).',
       },
+      // ---- resource library ----
+      save_to_library: { type: 'boolean', description: 'Save the generated audio into the local resource library after success. Also enabled globally by the "auto save to library" setting; pass false to skip a single run.' },
+      library_name: { type: 'string', description: 'Resource name in the library. Defaults to the prompt.' },
+      library_type: { type: 'string', enum: ['voice', 'music', 'sfx', 'tts'], description: 'Resource type in the library. Defaults to the generation mode (voice_design → voice).' },
+      library_tags: { type: 'array', items: { type: 'string' }, description: 'Tags for the library resource.' },
     },
     output: {
       schema: resultSchema,
@@ -199,6 +224,11 @@ export function registerAgentAudioTools(ctx: Context, resolve: () => AgentAudioT
         ...(typeof args.duration === 'number' ? { duration: args.duration } : {}),
         ...(typeof args.lyrics === 'string' && args.lyrics.trim() !== '' ? { lyrics: args.lyrics.trim() } : {}),
         ...(typeof args.is_instrumental === 'boolean' ? { isInstrumental: args.is_instrumental } : {}),
+        ...(typeof args.loop === 'boolean' ? { loop: args.loop } : {}),
+        ...(typeof args.prompt_influence === 'number' && Number.isFinite(args.prompt_influence) ? { promptInfluence: args.prompt_influence } : {}),
+        ...(typeof args.seed === 'number' && Number.isFinite(args.seed) ? { seed: args.seed } : {}),
+        ...(typeof args.steps === 'number' && Number.isFinite(args.steps) ? { steps: args.steps } : {}),
+        ...(typeof args.cfg_scale === 'number' && Number.isFinite(args.cfg_scale) ? { cfgScale: args.cfg_scale } : {}),
         ...(typeof args.format === 'string' && args.format.trim() !== '' ? { format: args.format.trim() } : {}),
         // ---- MiniMax TTS 专属字段 ----
         ...(typeof args.emotion === 'string' && args.emotion.trim() !== '' ? { emotion: args.emotion.trim() } : {}),
@@ -222,13 +252,22 @@ export function registerAgentAudioTools(ctx: Context, resolve: () => AgentAudioT
       try {
         const outputs = await generateAudio(picked.channel, request, exec.signal)
         const audio: AgentAudioRef[] = []
+        const saved: SavedAudioRef[] = []
         for (const [index, output] of outputs.entries()) {
-          const saved = await saveAudioFile(output.data, output.mime, `generated-${index + 1}`)
+          const stored = await saveAudioFile(output.data, output.mime, `generated-${index + 1}`)
+          saved.push({
+            id: stored.id,
+            url: `/api/dsh-audiogen/audio/${encodeURIComponent(stored.file)}`,
+            file: stored.file,
+            mime: stored.mime,
+            bytes: stored.bytes,
+            ...(output.voiceId === undefined ? {} : { voiceId: output.voiceId }),
+          })
           audio.push({
-            id: saved.id,
-            url: `/api/dsh-audiogen/audio/${encodeURIComponent(saved.file)}`,
-            mime: saved.mime,
-            bytes: saved.bytes,
+            id: stored.id,
+            url: `/api/dsh-audiogen/audio/${encodeURIComponent(stored.file)}`,
+            mime: stored.mime,
+            bytes: stored.bytes,
             ...(output.voiceId === undefined ? {} : { voiceId: output.voiceId }),
           })
         }
@@ -244,25 +283,60 @@ export function registerAgentAudioTools(ctx: Context, resolve: () => AgentAudioT
             ...(request.duration === undefined ? {} : { duration: request.duration }),
             ...(request.format === undefined ? {} : { format: request.format }),
             audio: outputs.map((output, index) => ({
-              id: audio[index]!.id,
+              id: saved[index]!.id,
+              file: saved[index]!.file,
               b64: Buffer.from(output.data).toString('base64'),
-              mime: audio[index]!.mime,
-              bytes: audio[index]!.bytes,
-              url: audio[index]!.url,
+              mime: saved[index]!.mime,
+              bytes: saved[index]!.bytes,
+              url: saved[index]!.url,
               ...(output.voiceId === undefined ? {} : { voiceId: output.voiceId }),
             })),
             channelId: picked.channel.id,
             channel: picked.channel.name,
+            params: { ...request },
           })
         } catch {
           // History is best-effort and must not fail the agent tool.
         }
+        // ---- 资源库保存：显式参数优先；设置自动入库时可用 false 跳过 ----
+        const wantSave = args.save_to_library === true || (config.autoSaveToLibrary && args.save_to_library !== false)
+        let resources: string[] | undefined
+        if (wantSave) {
+          try {
+            const entry = await saveToLibrary({
+              audioFiles: saved.map(item => ({
+                id: item.id,
+                file: item.file,
+                mime: item.mime,
+                ...(item.voiceId === undefined ? {} : { voiceId: item.voiceId }),
+              })),
+              type: libraryTypeOf(request.mode, args.library_type),
+              ...(typeof args.library_name === 'string' && args.library_name.trim() !== '' ? { name: args.library_name.trim() } : {}),
+              ...(Array.isArray(args.library_tags) ? { tags: args.library_tags.filter((tag): tag is string => typeof tag === 'string' && tag.trim() !== '').map(tag => tag.trim()) } : {}),
+              provenance: {
+                mode: request.mode,
+                prompt: request.prompt,
+                channel: picked.channel.name,
+                channelId: picked.channel.id,
+                apiUrl: picked.channel.apiUrl,
+                model: picked.alias,
+                upstream: picked.upstream,
+                ...(request.voice === undefined ? {} : { voice: request.voice }),
+                params: { ...request },
+              },
+            })
+            resources = [entry.id]
+          } catch {
+            // library-save is best-effort; generation already succeeded.
+          }
+        }
         return {
           status: 'completed',
           message: 'Audio generation completed. The audio files can be played/downloaded from the returned URLs.',
           mode: request.mode,
           model: picked.alias,
           audio,
+          ...(resources === undefined ? {} : { resources }),
         }
       } catch (error) {
         if (exec.signal?.aborted === true) throw error
@@ -277,5 +351,76 @@ export function registerAgentAudioTools(ctx: Context, resolve: () => AgentAudioT
       }
     },
   }))
-  return disposer
+
+  const searchDisposer = ctx.tools.register(defineTool({
+    name: 'search_audio_library',
+    description: 'Search curated audio resources in the local resource library (voice / music / sfx / tts). Returns matching resources with type, category, name, tags, full provenance (channel, model, voiceId, prompt) and same-origin audio URLs the user can play. Use it before generating to reuse an existing voice, music bed or sound effect instead of generating a new one.',
+    parameters: {
+      type: { type: 'string', enum: ['voice', 'music', 'sfx', 'tts'], description: 'Filter by resource type.' },
+      category: { type: 'string', description: 'Filter by category (voice: male/female/custom; tts: the speaking voice key).' },
+      keyword: { type: 'string', description: 'Search name, tags, prompt and model.' },
+    },
+    output: {
+      schema: {
+        type: 'object',
+        additionalProperties: false,
+        properties: {
+          status: { type: 'string', required: true, enum: ['ok'] },
+          count: { type: 'integer', required: true },
+          entries: {
+            type: 'array', required: true,
+            items: {
+              type: 'object',
+              additionalProperties: false,
+              properties: {
+                id: { type: 'string', required: true },
+                name: { type: 'string', required: true },
+                type: { type: 'string', required: true, enum: ['voice', 'music', 'sfx', 'tts'] },
+                category: { type: 'string' },
+                tags: { type: 'array', items: { type: 'string' }, required: true },
+                prompt: { type: 'string', required: true },
+                model: { type: 'string' },
+                channel: { type: 'string' },
+                voiceId: { type: 'string' },
+                urls: { type: 'array', items: { type: 'string' }, required: true },
+              },
+            },
+          },
+        },
+      } as const,
+      render: (_args, value) => [{ type: 'text', text: JSON.stringify(value) }],
+    },
+    isConcurrencySafe: () => true,
+    async execute(args) {
+      const keyword = typeof args.keyword === 'string' ? args.keyword.trim().toLowerCase() : ''
+      const wantedType = args.type === 'voice' || args.type === 'music' || args.type === 'sfx' || args.type === 'tts' ? args.type : undefined
+      const wantedCategory = typeof args.category === 'string' && args.category.trim() !== '' ? args.category.trim() : undefined
+      const all = await listLibrary()
+      const entries = all.filter(entry => {
+        if (wantedType !== undefined && entry.type !== wantedType) return false
+        if (wantedCategory !== undefined && (entry.category ?? '') !== wantedCategory) return false
+        if (keyword !== '') {
+          const haystack = [entry.name, ...entry.tags, entry.provenance.prompt, entry.provenance.model ?? '', entry.provenance.channel ?? ''].join(' ').toLowerCase()
+          if (!haystack.includes(keyword)) return false
+        }
+        return true
+      }).slice(0, 30).map(entry => ({
+        id: entry.id,
+        name: entry.name,
+        type: entry.type,
+        ...(entry.category === undefined ? {} : { category: entry.category }),
+        tags: entry.tags,
+        prompt: entry.provenance.prompt,
+        ...(entry.provenance.model === undefined ? {} : { model: entry.provenance.model }),
+        ...(entry.provenance.channel === undefined ? {} : { channel: entry.provenance.channel }),
+        ...(entry.provenance.voiceId === undefined ? {} : { voiceId: entry.provenance.voiceId }),
+        urls: entry.files.map(file => file.url),
+      }))
+      return { status: 'ok' as const, count: entries.length, entries }
+    },
+  }))
+  return () => {
+    disposer()
+    searchDisposer()
+  }
 }