openclacky 1.2.8 → 1.2.10

data/lib/clacky/default_skills/media-gen/SKILL.md

@@ -0,0 +1,128 @@
+---
+name: media-gen
+description: 'Generate images (and later videos / audio) inside the current task. Use this skill whenever the user asks to create, generate, or produce a picture / image / illustration / cover / poster / icon / artwork — including phrases like 生成图片, 画一张, 做封面, 来张配图, generate image, make a picture, draw, create artwork, design a cover. Also use when building documents (slides, PPT, posters, marketing pages, README hero shots) where an image is needed inline. Routes calls through the local Clacky HTTP server, which uses the user-configured `type=image` model — you do NOT need to know which provider; the server handles it.'
+disable-model-invocation: false
+user-invocable: true
+---
+
+# media-gen
+
+Generate images on demand by calling the local Clacky HTTP server, which dispatches to whichever image-generation model the user configured (`type=image` in their model settings).
+
+## Endpoint
+
+```
+POST http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/image
+GET  http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/types
+```
+
+## Step 1 — Verify a backend is configured
+
+Before generating anything, confirm the user has a `type=image` model set up:
+
+```bash
+curl -s http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/types
+```
+
+If the response shows `image.configured = false`, stop and tell the user:
+
+> 还没有配置生图模型。请打开 Clacky 设置页 → 添加模型 → 类型选 `image`（推荐 `or-gemini-3-pro-image` 或 `or-gpt-image-1`）。配好后再让我生图。
+
+Do NOT try to fall back to `terminal` + a hand-written `curl https://api.openai.com/...` — that bypasses the user's configured backend and won't be billed correctly.
+
+## Step 2 — Generate the image
+
+```bash
+curl -s -X POST http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/image \
+  -H "Content-Type: application/json" \
+  -d '{
+    "prompt": "A clean, modern hero illustration for a tech startup landing page. Soft gradient background, abstract geometric shapes in blue and purple, minimal style, 4K quality.",
+    "aspect_ratio": "landscape"
+  }'
+```
+
+### Request fields
+
+| Field          | Required | Values                              | Notes |
+|----------------|----------|-------------------------------------|-------|
+| `prompt`       | yes      | string                              | Be detailed and concrete. See prompt tips below. |
+| `aspect_ratio` | no       | `landscape` / `square` / `portrait` | Defaults to `landscape`. |
+| `output_dir`   | no       | absolute path                       | Defaults to the current working directory. The image is saved under `<output_dir>/assets/generated/`. |
+
+### Response shape (success)
+
+```json
+{
+  "success": true,
+  "image": "/abs/path/to/working_dir/assets/generated/img_20260525_011820_a1b2c3d4.png",
+  "model": "or-gemini-3-pro-image",
+  "provider": "openclacky",
+  "prompt": "A clean, modern hero illustration ...",
+  "aspect_ratio": "landscape",
+  "size": "1536x1024",
+  "usage": {
+    "prompt_tokens": 50,
+    "completion_tokens": 4500,
+    "cache_read_tokens": 0,
+    "cache_write_tokens": 0,
+    "total_tokens": 4550
+  }
+}
+```
+
+The `image` field is an absolute path on disk. To embed it in markdown, slides, or HTML, convert it to a path relative to the document you're writing.
+
+`usage` may be absent when the configured backend doesn't return token counts. Treat it as optional.
+
+### Response shape (failure)
+
+```json
+{
+  "success": false,
+  "image": null,
+  "error": "Upstream 401: Invalid API key",
+  "error_type": "api_error",
+  "model": "...",
+  "provider": "..."
+}
+```
+
+Common `error_type` values: `not_configured`, `auth_required`, `network_error`, `api_error`, `empty_response`. Tell the user the error plainly; if it's `auth_required` or `api_error 401/403`, point them at settings to fix the api_key.
+
+## Step 3 — Show the image
+
+`Read` does NOT show the image to the user — it only feeds it into your own context. To make the user actually see it, write a markdown tag in your reply:
+
+```markdown
+![](file:///abs/path/from/response.png)
+```
+
+Take the `image` field from the response and prefix `file://` (three slashes, since the path is absolute).
+
+If you're also embedding it in a document (README, PPT, etc.), use a relative path: `![](./assets/generated/xxx.png)`.
+
+## Prompt writing tips
+
+A good image prompt has 4 layers, in this order:
+
+1. **Subject** — what is in the image, concretely. ("a golden retriever puppy", "a stylized icon of a rocket")
+2. **Style / medium** — photo / illustration / 3D render / watercolor / flat vector / line art
+3. **Composition / lighting** — close-up / wide shot / overhead / soft natural light / dramatic backlight
+4. **Mood / palette** — minimal / playful / corporate / pastel / high-contrast monochrome
+
+For PPT / slide decks specifically:
+- Hero / cover slides: `aspect_ratio: landscape`, prompt should emphasise "clean", "minimal", "negative space" so text overlays well
+- Section dividers: `aspect_ratio: landscape`, abstract or pattern-style works better than literal subjects
+- Inline figures: `aspect_ratio: square` or `portrait`, more literal subject is fine
+
+When the user gives a vague request like "给我配张图", ask one clarifying question (subject? style?) before calling the API — costs real money per image.
+
+## When NOT to use this skill
+
+- The user asks to **edit** an existing image (this skill is text-to-image only today)
+- The user wants a **diagram / chart** with specific data — use a charting library (matplotlib, mermaid, etc.) instead; image gen is for illustrations, not data viz
+- The user asks for **screenshots** of real software — use the browser tool
+
+## Future modalities
+
+The same `/api/media/` namespace will gain `video` and `audio` endpoints. The pattern is identical: the user configures `type=video` / `type=audio` models in settings, this skill (or its successor) calls the matching endpoint.

data/lib/clacky/idle_compression_timer.rb

@@ -32,6 +32,7 @@ module Clacky
       @timer_thread    = nil
       @compress_thread = nil
       @mutex           = Mutex.new
+      @shutdown        = false
     end
 
     # Start (or restart) the idle timer.
@@ -39,24 +40,35 @@ module Clacky
     def start
       cancel # reset any existing timer
 
-      @timer_thread = Thread.new do
-        Thread.current.name = "idle-compression-timer"
-        sleep IDLE_DELAY
-
-        # Register @compress_thread inside the mutex BEFORE the thread starts running,
-        # so cancel() can always find and interrupt it even if it fires immediately.
-        compress_thread = nil
-        @mutex.synchronize do
-          compress_thread = Thread.new do
-            Thread.current.name = "idle-compression-work"
-            run_compression
+      @mutex.synchronize do
+        return false if @shutdown
+
+        @timer_thread = Thread.new do
+          Thread.current.name = "idle-compression-timer"
+          sleep IDLE_DELAY
+          next if shutdown?
+
+          # Register @compress_thread inside the mutex BEFORE the thread starts running,
+          # so cancel() can always find and interrupt it even if it fires immediately.
+          compress_thread = nil
+          @mutex.synchronize do
+            unless @shutdown
+              compress_thread = Thread.new do
+                Thread.current.name = "idle-compression-work"
+                run_compression
+              end
+              @compress_thread = compress_thread
+            end
           end
-          @compress_thread = compress_thread
-        end
 
-        compress_thread.join
-        @mutex.synchronize { @compress_thread = nil; @timer_thread = nil }
+          compress_thread&.join
+          @mutex.synchronize { @compress_thread = nil; @timer_thread = nil }
+        end
       end
+      true
+    rescue ThreadError => e
+      log("Idle compression timer could not start: #{e.message}", level: :debug)
+      false
     end
 
     # Cancel the timer and any in-progress compression.
@@ -81,6 +93,13 @@ module Clacky
       compress_thread_to_join&.join(5)
     end
 
+    # Permanently stop this timer. Used during application shutdown so
+    # background agent-thread ensure blocks cannot create new timer threads.
+    def shutdown
+      @mutex.synchronize { @shutdown = true }
+      cancel
+    end
+
     # True if the timer or compression is currently active.
     def active?
       @mutex.synchronize { @timer_thread&.alive? || @compress_thread&.alive? }
@@ -94,6 +113,10 @@ module Clacky
       @mutex.synchronize { @compress_thread&.alive? || false }
     end
 
+    def shutdown?
+      @mutex.synchronize { @shutdown }
+    end
+
     private def run_compression
       success = @agent.trigger_idle_compression
 

data/lib/clacky/media/base.rb

@@ -0,0 +1,68 @@
+# frozen_string_literal: true
+
+require "fileutils"
+require "base64"
+require "securerandom"
+
+module Clacky
+  module Media
+    # Abstract base for media (image / video / audio) generation providers.
+    #
+    # Subclasses implement #generate_image (and later #generate_video,
+    # #generate_audio). The base class supplies the uniform success/error
+    # response shape and the on-disk persistence helper, mirroring the
+    # design used by Hermes' image_gen_provider so the surface stays
+    # learnable across modalities.
+    class Base
+      # @param model_entry [Hash] one entry from AgentConfig#models — must
+      #   include "model", "base_url", "api_key" keys.
+      def initialize(model_entry)
+        @model_entry = model_entry
+        @model       = model_entry["model"]
+        @base_url    = model_entry["base_url"]
+        @api_key     = model_entry["api_key"]
+      end
+
+      # @return [Hash] either success_response(...) or error_response(...)
+      def generate_image(prompt:, aspect_ratio: "landscape", output_dir: nil, **_kwargs)
+        raise NotImplementedError, "#{self.class.name} must implement #generate_image"
+      end
+
+      # Persist a base64-encoded image under <output_dir>/assets/generated/.
+      # Returns the absolute path on disk.
+      private def save_b64_image(b64_data, output_dir:, prefix: "img", extension: "png")
+        target_dir = File.join(output_dir, "assets", "generated")
+        FileUtils.mkdir_p(target_dir)
+        ts    = Time.now.strftime("%Y%m%d_%H%M%S")
+        short = SecureRandom.hex(4)
+        path  = File.join(target_dir, "#{prefix}_#{ts}_#{short}.#{extension}")
+        File.binwrite(path, Base64.decode64(b64_data))
+        path
+      end
+
+      private def success_response(image:, prompt:, aspect_ratio:, provider:, extra: {})
+        {
+          "success"      => true,
+          "image"        => image,
+          "model"        => @model,
+          "prompt"       => prompt,
+          "aspect_ratio" => aspect_ratio,
+          "provider"     => provider
+        }.merge(extra)
+      end
+
+      private def error_response(error:, error_type: "provider_error", provider: "", prompt: "", aspect_ratio: "landscape")
+        {
+          "success"      => false,
+          "image"        => nil,
+          "error"        => error,
+          "error_type"   => error_type,
+          "model"        => @model,
+          "prompt"       => prompt,
+          "aspect_ratio" => aspect_ratio,
+          "provider"     => provider
+        }
+      end
+    end
+  end
+end

data/lib/clacky/media/gemini.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+require "faraday"
+require "json"
+require_relative "base"
+
+module Clacky
+  module Media
+    # Native Google Gemini image generation adapter.
+    #
+    # Reserved for users who configure a direct Google AI Studio base_url
+    # (e.g. https://generativelanguage.googleapis.com) with a raw Google API
+    # key. The official endpoints are:
+    #   POST /v1beta/models/<model>:generateContent  — image-out via Gemini
+    #   POST /v1beta/models/<model>:predict          — Imagen
+    # with x-goog-api-key auth, contents[].parts[] request schema, and
+    # candidates[].content.parts[].inlineData response schema. Completely
+    # different from the OpenAI /v1/images/generations contract.
+    #
+    # Today every shipping path (openclacky gateway, OpenRouter) wraps Gemini
+    # behind an OpenAI-compatible facade, so OpenAICompat handles them and
+    # this class is intentionally a stub. We surface a clear error rather
+    # than silently 404 against Google's actual host.
+    class Gemini < Base
+      def generate_image(prompt:, aspect_ratio: "landscape", output_dir: nil, **_kwargs)
+        error_response(
+          error: "Direct Google AI Studio (generativelanguage.googleapis.com) image generation is not yet supported. Use the openclacky or OpenRouter gateway instead — set base_url to https://api.openclacky.com or https://openrouter.ai/api/v1 and pick a Gemini image model (e.g. or-gemini-3-pro-image, google/gemini-3-pro-image-preview).",
+          error_type: "not_implemented",
+          provider: "gemini-direct",
+          prompt: prompt,
+          aspect_ratio: aspect_ratio
+        )
+      end
+    end
+  end
+end

data/lib/clacky/media/generator.rb

@@ -0,0 +1,78 @@
+# frozen_string_literal: true
+
+require_relative "openai_compat"
+require_relative "gemini"
+
+module Clacky
+  module Media
+    # Top-level dispatcher: takes an AgentConfig and a request, picks the
+    # right provider class based on the configured image model's base_url,
+    # and delegates.
+    #
+    # Adding a new modality (video / audio) means:
+    #   1. add a generate_<modality> method here that resolves the correct
+    #      type=<modality> entry and class
+    #   2. add a provider class under lib/clacky/media/ implementing the call
+    class Generator
+      # Hosts that speak the native Google AI Studio API instead of an
+      # OpenAI-compatible facade. Matched as a substring against the
+      # configured base_url so any regional / staging variant is caught.
+      GOOGLE_NATIVE_HOSTS = [
+        "generativelanguage.googleapis.com",
+        "aiplatform.googleapis.com"
+      ].freeze
+
+      # @param agent_config [Clacky::AgentConfig]
+      def initialize(agent_config)
+        @agent_config = agent_config
+      end
+
+      # @return [Hash, nil] the type=image model entry, or nil if not configured
+      def image_model_entry
+        @agent_config.find_model_by_type("image")
+      end
+
+      def generate_image(prompt:, aspect_ratio: "landscape", output_dir: nil, **kwargs)
+        entry = image_model_entry
+        if entry.nil?
+          return {
+            "success"    => false,
+            "image"      => nil,
+            "error"      => "No image model configured. Add a model with type=image in settings.",
+            "error_type" => "not_configured",
+            "provider"   => "",
+            "model"      => "",
+            "prompt"     => prompt
+          }
+        end
+
+        provider = build_provider_for(entry)
+        provider.generate_image(
+          prompt: prompt,
+          aspect_ratio: aspect_ratio,
+          output_dir: output_dir,
+          **kwargs
+        )
+      end
+
+      # Pick the adapter class for a media model entry.
+      #
+      # Routing rules:
+      #   • base_url points directly at a Google AI Studio host → Gemini
+      #     (native /v1beta/models/<m>:generateContent schema).
+      #   • everything else → OpenAICompat. This covers OpenAI itself, the
+      #     openclacky gateway, OpenRouter, and any third-party proxy that
+      #     re-exposes Gemini / Imagen / DALL-E behind /v1/images/generations.
+      #     OpenAICompat#generate_image branches internally on model id to
+      #     drop OpenAI-only params (size) when talking to Gemini families.
+      private def build_provider_for(entry)
+        url = entry["base_url"].to_s
+        if GOOGLE_NATIVE_HOSTS.any? { |host| url.include?(host) }
+          Gemini.new(entry)
+        else
+          OpenAICompat.new(entry)
+        end
+      end
+    end
+  end
+end

data/lib/clacky/media/openai_compat.rb

@@ -0,0 +1,168 @@
+# frozen_string_literal: true
+
+require "faraday"
+require "json"
+require_relative "base"
+
+module Clacky
+  module Media
+    # OpenAI-compatible image generation provider.
+    #
+    # Talks to POST <base_url>/images/generations with the standard OpenAI
+    # request shape. Handles three providers under one class because they
+    # all expose the same endpoint: OpenAI, OpenRouter, and the openclacky
+    # platform gateway. Provider-specific quirks (model id naming, billing)
+    # live in PRESETS, not here.
+    class OpenAICompat < Base
+      ASPECT_TO_SIZE = {
+        "landscape" => "1536x1024",
+        "square"    => "1024x1024",
+        "portrait"  => "1024x1536"
+      }.freeze
+
+      DEFAULT_ASPECT = "landscape"
+
+      def generate_image(prompt:, aspect_ratio: DEFAULT_ASPECT, output_dir: nil, n: 1, **_kwargs)
+        provider_id = Clacky::Providers.find_by_base_url(@base_url) || "custom"
+        aspect      = ASPECT_TO_SIZE.key?(aspect_ratio) ? aspect_ratio : DEFAULT_ASPECT
+        size        = ASPECT_TO_SIZE[aspect]
+
+        if prompt.to_s.strip.empty?
+          return error_response(
+            error: "Prompt is required and must be a non-empty string",
+            error_type: "invalid_argument",
+            provider: provider_id,
+            aspect_ratio: aspect
+          )
+        end
+
+        if @api_key.to_s.empty?
+          return error_response(
+            error: "api_key not configured for image model '#{@model}'",
+            error_type: "auth_required",
+            provider: provider_id,
+            prompt: prompt,
+            aspect_ratio: aspect
+          )
+        end
+
+        payload = { model: @model, n: n }
+        if gemini_family?(@model)
+          # Gemini image models (routed via openclacky / openrouter gateway)
+          # don't accept the OpenAI `size` parameter — they infer aspect from
+          # the prompt text. Embedding a hint keeps the user's aspect choice
+          # honoured without breaking the gateway request validator.
+          payload[:prompt] = "#{prompt}\n\n[aspect: #{aspect}]"
+        else
+          payload[:prompt] = prompt
+          payload[:size]   = size
+        end
+
+        begin
+          response = connection.post("images/generations") do |req|
+            req.headers["Content-Type"]  = "application/json"
+            req.headers["Authorization"] = "Bearer #{@api_key}"
+            req.body = JSON.generate(payload)
+          end
+        rescue Faraday::Error => e
+          return error_response(
+            error: "HTTP request failed: #{e.message}",
+            error_type: "network_error",
+            provider: provider_id,
+            prompt: prompt,
+            aspect_ratio: aspect
+          )
+        end
+
+        unless response.success?
+          return error_response(
+            error: "Upstream #{response.status}: #{truncate(response.body, 500)}",
+            error_type: "api_error",
+            provider: provider_id,
+            prompt: prompt,
+            aspect_ratio: aspect
+          )
+        end
+
+        body = parse_json(response.body)
+        return error_response(
+          error: "Invalid JSON response from upstream",
+          error_type: "invalid_response",
+          provider: provider_id,
+          prompt: prompt,
+          aspect_ratio: aspect
+        ) unless body.is_a?(Hash)
+
+        data = body["data"] || []
+        first = data.first
+        if first.nil?
+          return error_response(
+            error: "Upstream returned no image data",
+            error_type: "empty_response",
+            provider: provider_id,
+            prompt: prompt,
+            aspect_ratio: aspect
+          )
+        end
+
+        image_ref =
+          if first["b64_json"]
+            save_b64_image(first["b64_json"], output_dir: output_dir || Dir.pwd, prefix: "img")
+          elsif first["url"]
+            first["url"]
+          end
+
+        if image_ref.nil?
+          return error_response(
+            error: "Response contained neither b64_json nor url",
+            error_type: "empty_response",
+            provider: provider_id,
+            prompt: prompt,
+            aspect_ratio: aspect
+          )
+        end
+
+        success_response(
+          image: image_ref,
+          prompt: prompt,
+          aspect_ratio: aspect,
+          provider: provider_id,
+          extra: {
+            "size"     => size,
+            "usage"    => body["usage"],
+            "cost_usd" => body["cost_usd"]
+          }.compact
+        )
+      end
+
+      private def connection
+        Faraday.new(url: normalized_base_url) do |f|
+          f.options.timeout      = 240
+          f.options.open_timeout = 10
+        end
+      end
+
+      private def gemini_family?(model_name)
+        model_name.to_s.match?(/gemini|imagen/i)
+      end
+
+      # base_url is taken verbatim from PRESETS (each provider already
+      # includes the API version segment when needed). We only ensure a
+      # trailing slash so Faraday's relative-path join behaves.
+      private def normalized_base_url
+        "#{@base_url.to_s.chomp("/")}/"
+      end
+
+      private def parse_json(body)
+        JSON.parse(body)
+      rescue JSON::ParserError
+        nil
+      end
+
+      private def truncate(str, max)
+        s = str.to_s
+        s.length > max ? "#{s[0, max]}..." : s
+      end
+    end
+  end
+end