openclacky 1.5.18 → 1.5.19

data/lib/clacky/default_skills/media-gen/SKILL.md

@@ -1,8 +1,8 @@
 ---
 name: media-gen
 name_zh: 媒体生成
-description_zh: 生成或编辑图片、视频，以及文字转语音，适用于封面、海报、插画、配音等。
-description: 'Generate or edit images, videos, or audio (text-to-speech) in the current task. Use whenever the user asks to create/generate/produce or edit/modify a picture / image / illustration / cover / poster / icon / artwork, a video / clip / animation, or speech / voiceover / narration / TTS — e.g. generate image, draw, design a cover, edit this image, change the background, text-to-video, generate speech; 画一张, 配图, 编辑图片, 改图, 换背景, 做个视频, 配音, 文字转语音. Also use when a document (slides, poster, README hero) needs an inline image.'
+description_zh: 生成或编辑图片、视频和音频，适用于封面、海报、插画、配音、环境音等。
+description: 'Generate or edit images, videos, or audio in the current task. Use whenever the user asks to create/generate/produce or edit/modify a picture / image / illustration / cover / poster / icon / artwork, a video / clip / animation, or speech / voiceover / narration / TTS / ambient audio / soundscapes — e.g. generate image, draw, design a cover, edit this image, change the background, text-to-video, generate speech; 画一张, 配图, 编辑图片, 改图, 换背景, 做个视频, 配音, 文字转语音, 环境音. Also use when a document (slides, poster, README hero) needs an inline image.'
 disable-model-invocation: false
 user-invocable: true
 always-show: true
@@ -71,6 +71,12 @@ curl -s -X POST http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/ima
 - If a call fails with `400 / INVALID_ARGUMENT`, drop the `aspect_ratio` field and retry once before reporting the error.
 - If a call fails with `unknown image model` (400), the configured model name isn't recognized by its backend — tell the user to fix the model name in the settings page; do NOT guess another name and retry.
 
+### Content-safety refusal (content_filter)
+
+If a call fails with no image and mentions `content_filter` / `safety` / `blocked` (or HTTP `422`), the prompt was blocked. Don't resend as-is — **rephrase the prompt** (soften the sensitive part, keep the intent) and retry, up to 3 times, checking each returned image. Still blocked? Stop and ask the user to adjust.
+
+
+
 ### Request fields
 
 | Field          | Required | Values                              | Notes |
@@ -82,6 +88,11 @@ curl -s -X POST http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/ima
 | `image`        | no       | file path / base64 / data URL       | A single input image to **edit**. Triggers image-edit mode (see below). |
 | `images`       | no       | array of the above                  | Multiple input images for a multi-image edit. Takes precedence over `image`. |
 
+### Input / reference image format
+
+Images passed as `image` / `images` must be **PNG, JPEG, or WebP**. Other formats (SVG, GIF, BMP, TIFF, …) are rejected upstream and fail the call — convert them to one of the supported formats first.
+
+
 ### Editing an existing image
 
 To edit instead of generate from scratch, pass the existing image as `image`
@@ -313,14 +324,16 @@ Rules & caveats:
   chaining is the practical option today; Veo's native server-side `extend`
   (148s) is not wired into this endpoint yet.
 
-### Seedance (Volcengine Ark) — multimodal video
+### Seedance — multimodal video
 
 When the configured `type=video` model is a ByteDance **Doubao Seedance**
-model on Volcengine Ark (its Base URL is under `*.volces.com`, e.g.
-`https://ark.cn-beijing.volces.com/api/v3`), the **same**
-`POST /api/media/video` endpoint drives it. No separate endpoint — the server
-routes by Base URL automatically. Seedance adds richer inputs on top of the
-common fields; all are optional and only apply to Seedance:
+model, the **same** `POST /api/media/video` endpoint drives it. No separate
+endpoint — the server routes by Base URL automatically. Seedance adds richer
+inputs on top of the common fields (all optional, Seedance-only), split below
+into a **common** set that works on any Seedance gateway and an **Ark-only**
+set that takes effect only on the native Volcengine Ark transport (Base URL
+under `*.volces.com`). On any other OpenAI-compatible Seedance gateway, stick
+to the common fields.
 
 > **Cost gate — ask before EVERY generation.** Resolution is the main driver
 > of Seedance's price (4k costs far more than 720p). So **once you've confirmed
@@ -337,22 +350,34 @@ common fields; all are optional and only apply to Seedance:
 > **When editing or continuing/extending an existing video, default to that
 > source video's resolution — never silently upgrade it (e.g. don't turn a
 > 720p source into a 4k render).** If the user gave no answer and you didn't
-> ask, the server pins `720p`. **These Seedance-only fields (`resolution`,
-> `generate_audio`, `watermark`, `seed`, `first_frame`, `last_frame`,
-> `reference_*`) have NO effect on Veo or Qwen/DashScope backends — never send
-> them unless the Base URL is `*.volces.com`.**
+> ask, the server pins `720p`. **These Seedance-only fields have NO effect on
+> Veo or Qwen/DashScope backends — never send them there.** (And within
+> Seedance, the Ark-only table additionally requires a `*.volces.com` Base
+> URL.)
+
+**Common fields** — work on any Seedance gateway (native Ark or an
+OpenAI-compatible host):
 
 | Field              | Values                                   | Notes |
 |--------------------|------------------------------------------|-------|
 | `aspect_ratio`     | `landscape`/`portrait`/`square`, or a raw Ark ratio like `16:9`, `9:16`, `4:3`, `3:4`, `21:9`, `adaptive` | Raw ratios pass through unchanged. |
 | `duration_seconds` | integer, or `-1`                         | `-1` lets the model pick the length (Seedance 2.0 / 1.5 Pro). |
+| `first_frame`      | media ref (see below)                    | First frame → image-to-video. |
+| `reference_images` | array of media refs (0–9)                | Reference images. |
+
+**Ark-only fields** — only take effect on the native Volcengine Ark transport
+(`video.base_url` under `*.volces.com`). On any other OpenAI-compatible
+Seedance gateway, `resolution` / `generate_audio` / `watermark` / `seed` are
+silently ignored, and `last_frame` / `reference_videos` / `reference_audios`
+are rejected outright — so don't send any of them there:
+
+| Field              | Values                                   | Notes |
+|--------------------|------------------------------------------|-------|
 | `resolution`       | `480p` / `720p` / `1080p` / `4k`         | **Defaults to `720p` when omitted** (cost control). Ask the user before every generation — never reuse a prior answer. See the cost gate above. Model-dependent; unsupported values are rejected upstream. |
 | `generate_audio`   | `true` / `false`                         | Seedance 2.0 / 1.5 Pro can synthesize a synced audio track. |
 | `watermark`        | `true` / `false`                         | |
 | `seed`             | integer                                  | Reproducibility. |
-| `first_frame`      | media ref (see below)                    | First frame → image-to-video. |
 | `last_frame`       | media ref                                | Together with `first_frame` → first+last-frame video. |
-| `reference_images` | array of media refs (0–9)                | Reference images. |
 | `reference_videos` | array of media refs (0–3)                | Reference videos. |
 | `reference_audios` | array of media refs (0–3)                | Reference audio (background music / voice). |
 
@@ -482,6 +507,12 @@ done
   polling and tell the user the task is still rendering in the background,
   give them the `task_id`, and let them check again later — do NOT resubmit.
 
+## Generating ambient audio
+
+Ambient audio includes rain, ocean waves, wind, and café background
+sounds. Generate it programmatically, verify the audio, and provide
+it for playback or download.
+
 ## Generating speech (Gemini TTS)
 
 The same `/api/media/` namespace serves text-to-speech. The user must

data/lib/clacky/locales/en.rb

@@ -6,7 +6,7 @@ module Clacky
       "phase.skill_evolution"           => "Reflecting on this task",
       "phase.memory_update"             => "Updating long-term memory",
       "llm.error.insufficient_credit"   => "Insufficient credit, please top up your account to continue",
-      "llm.error.rate_limit_400"        => "Rate limit or service issue, retrying...",
+      "llm.error.rate_limit_400"        => "Too many requests or model temporarily unavailable. Please switch to another model and retry",
       "llm.error.invalid_api_key"       => "API key is invalid or expired, please update it in Settings",
       "llm.error.403.model_not_allowed" => "Free trial only supports the or-gemini-3-8-flash model. Switch to that model, or top up to use other models",
       "llm.error.403.api_key_revoked"   => "API key has been revoked, please generate a new one",

data/lib/clacky/locales/zh.rb

@@ -6,7 +6,7 @@ module Clacky
       "phase.skill_evolution"           => "正在复盘本次任务",
       "phase.memory_update"             => "正在更新长期记忆",
       "llm.error.insufficient_credit"   => "API Key 账户余额不足，请前往控制台充值后继续使用",
-      "llm.error.rate_limit_400"        => "请求频率过高或服务暂时不可用，正在重试...",
+      "llm.error.rate_limit_400"        => "请求频率过高或模型暂时不可用，请暂时切换其他模型重试",
       "llm.error.invalid_api_key"       => "API Key 无效或已过期，请到设置中重新配置",
       "llm.error.403.model_not_allowed" => "免费试用仅支持 or-gemini-3-8-flash 模型，请切换到该模型，或充值后使用其他模型",
       "llm.error.403.api_key_revoked"   => "API 密钥已被撤销，请前往控制台重新生成",

data/lib/clacky/media/openai_compat.rb

@@ -160,7 +160,10 @@ module Clacky
         )
       end
 
-      def generate_video(prompt:, aspect_ratio: DEFAULT_ASPECT, duration_seconds: nil, output_dir: nil, image: nil, **_kwargs)
+      def generate_video(prompt:, aspect_ratio: DEFAULT_ASPECT, duration_seconds: nil, output_dir: nil,
+                          image: nil, first_frame: nil, last_frame: nil,
+                          reference_images: nil, reference_videos: nil, reference_audios: nil,
+                          **_kwargs)
         provider_id = Clacky::Providers.find_by_base_url(@base_url) || "custom"
         aspect      = VIDEO_ASPECTS.include?(aspect_ratio) ? aspect_ratio : DEFAULT_ASPECT
         duration    = duration_seconds.to_i
@@ -179,8 +182,42 @@ module Clacky
           )
         end
 
+        # The OpenAI-compatible video endpoint has two independent image channels:
+        #   • the first frame (exact anchor), carried as `image` — Veo-style
+        #     image-to-video, and the only thing this facade honored before.
+        #   • reference images (non-exact subject/identity/style guidance),
+        #     carried as `input_references` — forwarded to Seedance 2.x's
+        #     reference-to-video. `last_frame` and video/audio references are
+        #     NOT representable on this facade (only the native Volcengine Ark
+        #     path supports them); callers wanting those must use an Ark model.
+        # A bare `first_frame` (no explicit `image`) acts as the first frame.
+        first_frame_image = normalize_video_first_frame(image, first_frame)
+        begin
+          references = normalize_video_references(reference_images)
+        rescue ArgumentError => e
+          return video_error_response(
+            error: e.message,
+            error_type: "invalid_argument", provider: provider_id, prompt: prompt, aspect_ratio: aspect
+          )
+        end
+        unsupported = []
+        unsupported << "last_frame" unless to_list(last_frame).empty?
+        unsupported << "reference_videos" unless to_list(reference_videos).empty?
+        unsupported << "reference_audios" unless to_list(reference_audios).empty?
+        unless unsupported.empty?
+          return video_error_response(
+            error: "#{unsupported.join('/')} #{unsupported.length == 1 ? 'is' : 'are'} only " \
+                   "supported by the native Volcengine (Seedance Ark) video backend, not by this " \
+                   "OpenAI-compatible video model '#{@model}'. Use first_frame and/or " \
+                   "reference_images here, or configure a *.volces.com video model for " \
+                   "last_frame / reference_videos / reference_audios.",
+            error_type: "invalid_argument", provider: provider_id, prompt: prompt, aspect_ratio: aspect
+          )
+        end
+
         payload = { model: @model, prompt: prompt, aspect_ratio: aspect, duration_seconds: duration }
-        payload[:image] = image if image.is_a?(Hash) && image["b64_json"]
+        payload[:image] = first_frame_image unless first_frame_image.nil?
+        payload[:input_references] = references unless references.empty?
 
         begin
           response = video_connection.post("videos/generations") do |req|
@@ -547,6 +584,56 @@ module Clacky
         end
       end
 
+      # Coerce one video media ref into the first-frame image hash the gateway's
+      # /videos endpoint expects: { "b64_json" => "<base64 or data URL>",
+      # "mime_type" => "image/..." }. An explicit `image` (already that hash)
+      # wins; otherwise a `first_frame` string (file path / data URL / bare
+      # base64) or hash is accepted. Returns nil when neither is given.
+      private def normalize_video_first_frame(image, first_frame)
+        if image.is_a?(Hash) && !image["b64_json"].to_s.strip.empty?
+          return image
+        end
+        source = first_frame
+        source = first_frame.first if first_frame.is_a?(Array) && !first_frame.empty?
+        return nil if source.nil?
+
+        if source.is_a?(Hash)
+          return source unless source["b64_json"].to_s.strip.empty?
+          return nil
+        end
+        s = source.to_s.strip
+        return nil if s.empty?
+        { "b64_json" => to_data_url(s) }
+      end
+
+      # Normalise reference images into the gateway `input_references` array:
+      # a list of data-URL / https strings. Accepts a single value or an array,
+      # where each entry is a file path, a data URL, a bare base64 string, an
+      # http(s) URL, or a { "b64_json", "mime_type" } hash. Raises ArgumentError
+      # on an undecodable local input (same contract as normalize_input_images).
+      private def normalize_video_references(reference_images)
+        to_list(reference_images).filter_map do |item|
+          if item.is_a?(Hash)
+            b64 = item["b64_json"].to_s.strip
+            next if b64.empty?
+            next b64 if b64.start_with?("data:")
+            mime = item["mime_type"].to_s.strip
+            mime = "image/png" if mime.empty?
+            next "data:#{mime};base64,#{b64}"
+          end
+          s = item.to_s.strip
+          next if s.empty?
+          next s if s.start_with?("http://", "https://")
+          to_data_url(s)
+        end
+      end
+
+      # Coerce a nil / scalar / array value into an array, dropping a nil.
+      private def to_list(value)
+        return [] if value.nil?
+        value.is_a?(Array) ? value : [value]
+      end
+
       private def to_data_url(input)
         return input if input.start_with?("data:")
 

data/lib/clacky/message_format/bedrock.rb

@@ -88,15 +88,18 @@ module Clacky
           body[:toolConfig] = { tools: tools.map { |t| to_api_tool(t) } }
         end
 
-        extra = additional_fields_for_effort(reasoning_effort)
+        extra = additional_fields_for_effort(reasoning_effort, model)
         body[:additionalModelRequestFields] = extra if extra
 
         body
       end
 
-      private_class_method def self.additional_fields_for_effort(effort)
+      private_class_method def self.additional_fields_for_effort(effort, model = nil)
         return nil if effort.nil? || effort.to_s.empty?
         return nil unless %w[low medium high xhigh max].include?(effort.to_s)
+        # Converse accepts thinking/output_config only for the Anthropic family;
+        # other Converse models (Z.AI GLM, DeepSeek) reject unknown fields.
+        return nil unless model.to_s.downcase.include?("claude")
         {
           thinking: { type: "adaptive" },
           output_config: { effort: effort.to_s }

data/lib/clacky/providers.rb

@@ -36,23 +36,28 @@ module Clacky
         "default_model" => "auto",
         "models" => [
           "auto",
-          "abs-gpt-6-astra",
-          "abs-gpt-6-sol",
-          "abs-gpt-6-luna",
-          "abs-gpt-5.6-sol",
-          "abs-gpt-5.6-terra",
-          "abs-gpt-5.6-luna",
+          "dsk-deepseek-flash",
+          "dsk-deepseek-v4-pro",
+          "oc-glm-5.3",
+          "oc-kimi-k3",
+          "oc-minimax-m2.7",
           "abs-claude-fable-5-1",
           "abs-claude-fable-5",
           "abs-claude-opus-5-5",
           "abs-claude-opus-5",
           "abs-claude-opus-4-8",
           "abs-claude-opus-4-7",
+          "abs-claude-sonnet-5-5",
           "abs-claude-sonnet-5",
           "abs-claude-sonnet-4-6",
           "abs-claude-haiku-4-5",
-          "dsk-deepseek-flash",
-          "dsk-deepseek-v4-pro",
+          "abs-gpt-6-astra",
+          "abs-gpt-6.1-sol",
+          "abs-gpt-6-sol",
+          "abs-gpt-6-luna",
+          "abs-gpt-5.6-sol",
+          "abs-gpt-5.6-terra",
+          "abs-gpt-5.6-luna",
           "or-gemini-3-1-pro",
           "or-gemini-3-8-flash",
           "or-gemini-3-7-flash"
@@ -141,9 +146,17 @@ module Clacky
         # images uploaded for text-only models must be downgraded to disk
         # refs. Gemini 3.1 Pro keeps the provider-default vision=true
         # (it accepts image/audio/video input natively via OpenRouter).
+        # oc-glm-5.3 is likewise text-only: its upstream rejects any content
+        # block other than `type: "text"`, so images must never be sent
+        # inline. oc-minimax-m2.7 is also text-only, but its upstream accepts
+        # the image block and silently ignores it — a worse failure mode,
+        # since the model then answers as if no image had been attached.
+        # oc-kimi-k3 is natively multimodal and keeps vision=true.
         "model_capabilities" => {
           "dsk-deepseek-flash"  => { "vision" => true }.freeze,
-          "dsk-deepseek-v4-pro" => { "vision" => false }.freeze
+          "dsk-deepseek-v4-pro" => { "vision" => false }.freeze,
+          "oc-glm-5.3"          => { "vision" => false }.freeze,
+          "oc-minimax-m2.7"      => { "vision" => false }.freeze
         }.freeze,
         # Bedrock GPT models (abs-gpt-*) are served through the OpenAI
         # Responses API — their Chat Completions endpoint rejects function
@@ -166,6 +179,7 @@ module Clacky
           "abs-claude-opus-5"     => "abs-claude-haiku-4-5",
           "abs-claude-opus-4-8"   => "abs-claude-haiku-4-5",
           "abs-claude-opus-4-7"   => "abs-claude-haiku-4-5",
+          "abs-claude-sonnet-5-5" => "abs-claude-haiku-4-5",
           "abs-claude-sonnet-5"   => "abs-claude-haiku-4-5",
           "abs-claude-sonnet-4-6" => "abs-claude-haiku-4-5",
           "abs-gpt-6-astra"       => "abs-gpt-6-luna",
@@ -182,6 +196,7 @@ module Clacky
           "abs-claude-fable-5"    => "abs-claude-opus-5",
           "abs-claude-opus-5-5"   => "abs-claude-opus-5",
           "abs-claude-opus-5"     => "abs-claude-opus-4-8",
+          "abs-claude-sonnet-5-5" => "abs-claude-sonnet-5",
           "abs-claude-sonnet-5"   => "abs-claude-sonnet-4-6"
         },
         # Selectable endpoints exposed in the Base URL dropdown. The gateway
@@ -515,27 +530,29 @@ module Clacky
         "name" => "MiMo (Xiaomi)",
         "base_url" => "https://api.xiaomimimo.com/v1",
         "api" => "openai-completions",
-        "default_model" => "mimo-v2.5-pro",
-        # The MiMo-V2 family (mimo-v2-pro / mimo-v2-omni) was retired on
-        # 2026-06-30 and the model ids are no longer accepted by the API. The
-        # current lineup is the V2.5 series:
-        #   - mimo-v2.5-pro: text reasoning flagship, no vision
-        #   - mimo-v2.5: native omni-modal (image/video/audio/text), vision-capable
-        # Source: https://platform.xiaomimimo.com/docs/zh-CN/model
-        "models" => ["mimo-v2.5-pro", "mimo-v2.5"],
-        "capabilities" => { "vision" => false }.freeze,
+        "default_model" => "mimo-v2.6-pro",
+        # The V2.6 chat models are both native omni-modal models. Keep the V2.5
+        # ids available during Xiaomi's migration window; V2.5-Pro remains the
+        # only text-only model in this preset.
+        #   - mimo-v2.6-pro: flagship omni-modal reasoning model
+        #   - mimo-v2.6-flash: efficient omni-modal reasoning model
+        #   - mimo-v2.5-pro: legacy text reasoning flagship, no vision
+        #   - mimo-v2.5: legacy native omni-modal model
+        # Source: https://mimo.mi.com/docs/zh-CN/updates/model
+        "models" => ["mimo-v2.6-pro", "mimo-v2.6-flash", "mimo-v2.5-pro", "mimo-v2.5"],
+        "capabilities" => { "vision" => true }.freeze,
         "model_capabilities" => {
-          "mimo-v2.5" => { "vision" => true }.freeze
+          "mimo-v2.5-pro" => { "vision" => false }.freeze
         }.freeze,
-        "default_ocr_model" => "mimo-v2.5",
-        # Xiaomi serves the same V2.5 lineup from two billing endpoints: the
+        "default_ocr_model" => "mimo-v2.6-flash",
+        # Xiaomi serves the same chat lineup from two billing endpoints: the
         # pay-as-you-go API (api.xiaomimimo.com) and the Token Plan subscription
         # endpoint (token-plan-cn.xiaomimimo.com). Both accept identical model
         # ids and share one capability profile, so a single preset with
         # endpoint_variants recognises both. Without this, users on the Token
         # Plan endpoint fell through to "unknown provider" and the conservative
         # vision=true default applied to every model — sending images to the
-        # text-only mimo-v2.5-pro, which rejects image input.
+        # text-only legacy mimo-v2.5-pro, which rejects image input.
         "endpoint_variants" => [
           { "label" => "Pay-as-you-go", "label_key" => "settings.models.baseurl.variant.mimo_payg",        "base_url" => "https://api.xiaomimimo.com/v1",          "region" => "cn" }.freeze,
           { "label" => "Token Plan",    "label_key" => "settings.models.baseurl.variant.mimo_token_plan", "base_url" => "https://token-plan-cn.xiaomimimo.com/v1", "region" => "cn" }.freeze
@@ -791,17 +808,19 @@ module Clacky
     # Per-model maximum output token limits.
     #
     # The Agent global default (@max_tokens = 16_384) is tuned for the lowest
-    # common denominator. Strong reasoning models (GLM-5.2, Kimi-K3, MiMo-V2.5)
-    # support 64K–128K output but are artificially throttled to 16K, causing
-    # truncation on long reasoning chains and code generation.
+    # common denominator. Strong reasoning models (GLM-5.2, Kimi-K3, MiniMax-M2.7,
+    # MiMo-V2.5/V2.6) support 64K–128K output but are artificially throttled to
+    # 16K, causing truncation on long reasoning chains and code generation.
     #
     # Entries are matched top-to-bottom; the first match wins. Models not
     # listed here fall back to the global default.
     MODEL_MAX_OUTPUT = [
       { pattern: /glm/i,           limit: 65_536 }, # GLM-5.2: 128K output ceiling; 64K ample for reasoning+answer
       { pattern: /kimi-k3/i,       limit: 65_536 }, # Kimi K3: max_completion_tokens=131072 (max 1M); 64K ample
-      { pattern: /mimo-v2\.5-pro/i, limit: 65_536 }, # MiMo-V2.5-Pro: max_completion_tokens=131072; 64K ample
-      { pattern: /mimo/i,           limit: 32_768 }, # MiMo-V2.5: max_completion_tokens=32768; full default ceiling
+      { pattern: /minimax-m2\.7/i, limit: 65_536 }, # MiniMax M2.7: max_tokens ceiling 196608; 64K ample
+      { pattern: /\Amimo-v2\.6-(?:pro|flash)\z/i, limit: 65_536 }, # MiMo-V2.6: max_completion_tokens=131072; 64K ample
+      { pattern: /mimo-v2\.5-pro/i,                limit: 65_536 }, # MiMo-V2.5-Pro: max_completion_tokens=131072; 64K ample
+      { pattern: /mimo/i,                          limit: 32_768 }, # MiMo-V2.5: max_completion_tokens=32768; full default ceiling
       { pattern: /gemini/i,        limit: 65_536 }  # Gemini 3.x: 64K max output ceiling
     ].freeze