openclacky 1.3.11 → 1.4.0

data/lib/clacky/default_skills/media-gen/SKILL.md

@@ -60,7 +60,8 @@ curl -s -X POST http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/ima
   -d '{
     "prompt": "A clean, modern hero illustration for a tech startup landing page. Soft gradient background, abstract geometric shapes in blue and purple, minimal style, 4K quality.",
     "aspect_ratio": "landscape",
-    "output_dir": "'"$(pwd)"'"
+    "output_dir": "'"$(pwd)"'",
+    "session_id": "<%= session_id %>"
   }'
 ```
 
@@ -75,6 +76,7 @@ curl -s -X POST http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/ima
 | `prompt`       | yes      | string                              | Be detailed and concrete. See prompt tips below. |
 | `aspect_ratio` | no       | `landscape` / `square` / `portrait` | Defaults to `landscape`. |
 | `output_dir`   | yes      | absolute path                       | Always pass `$(pwd)` so files land in the current session workspace. The image is saved under `<output_dir>/assets/generated/`. |
+| `session_id`   | yes      | string                              | Current Clacky session ID. Always pass the rendered value shown in the request example. |
 | `image`        | no       | file path / base64 / data URL       | A single input image to **edit**. Triggers image-edit mode (see below). |
 | `images`       | no       | array of the above                  | Multiple input images for a multi-image edit. Takes precedence over `image`. |
 
@@ -90,7 +92,8 @@ curl -s -X POST http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/ima
   -H "Content-Type: application/json" \
   -d '{
     "prompt": "change the background to a starry night sky, keep the cat unchanged",
-    "image": "/abs/path/to/input.png"
+    "image": "/abs/path/to/input.png",
+    "session_id": "<%= session_id %>"
   }'
 ```
 
@@ -206,7 +209,8 @@ curl -s -X POST http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/vid
     "prompt": "A cinematic drone shot flying over a misty mountain range at sunrise, golden light, 4K.",
     "aspect_ratio": "landscape",
     "duration_seconds": 8,
-    "output_dir": "'"$(pwd)"'"
+    "output_dir": "'"$(pwd)"'",
+    "session_id": "<%= session_id %>"
   }'
 ```
 
@@ -217,6 +221,7 @@ curl -s -X POST http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/vid
 | `duration_seconds` | no       | 4–8                             | Defaults to 8. |
 | `image`            | no       | `{ "b64_json": "...", "mime_type": "image/png" }` | Optional first frame for image-to-video. |
 | `output_dir`       | yes      | absolute path                   | Always pass `$(pwd)` so files land in the current session workspace. MP4 saved under `<output_dir>/assets/generated/`. |
+| `session_id`       | yes      | string                          | Current Clacky session ID. Always pass the rendered value shown in the request example. |
 
 ### Response (success)
 
@@ -278,11 +283,11 @@ Workflow for an N-segment continuous video:
    writes a ready-to-send JSON file:
    ```bash
    "$SEQ" payload /tmp/seg2.json /tmp/seg1_last.jpg 8 landscape "$OUT_DIR" \
-     "Continuing the same scene, the camera keeps pushing forward…"
+     "Continuing the same scene, the camera keeps pushing forward…" "<%= session_id %>"
    curl -s -X POST .../api/media/video -H "Content-Type: application/json" \
      --data @/tmp/seg2.json
    ```
-   (`payload <out.json> <frame> <duration_seconds> <aspect_ratio> <output_dir> <prompt>`)
+   (`payload <out.json> <frame> <duration_seconds> <aspect_ratio> <output_dir> <prompt> [session_id]`)
 5. **Repeat** steps 3–4 for each subsequent segment, always chaining off the
    *previous* segment's last frame.
 6. **Stitch** all clips in order into one file:
@@ -305,6 +310,175 @@ Rules & caveats:
   chaining is the practical option today; Veo's native server-side `extend`
   (148s) is not wired into this endpoint yet.
 
+### Seedance (Volcengine Ark) — multimodal video
+
+When the configured `type=video` model is a ByteDance **Doubao Seedance**
+model on Volcengine Ark (its Base URL is under `*.volces.com`, e.g.
+`https://ark.cn-beijing.volces.com/api/v3`), the **same**
+`POST /api/media/video` endpoint drives it. No separate endpoint — the server
+routes by Base URL automatically. Seedance adds richer inputs on top of the
+common fields; all are optional and only apply to Seedance:
+
+> **Cost gate — ask before EVERY generation.** Resolution is the main driver
+> of Seedance's price (4k costs far more than 720p). So **once you've confirmed
+> via `GET /api/media/types` that the `type=video` Base URL is under
+> `*.volces.com`, you MUST ask the user which resolution they want before EACH
+> AND EVERY billable call** — this covers not just a brand-new clip but also
+> editing, multimodal reference, and extending/continuing an existing video
+> (they all cost the same as a fresh render). Offer `480p` / `720p` / `1080p` /
+> `4k` and state the default is `720p`. Only after they answer (or explicitly
+> say "use the default") do you proceed, passing their choice as `resolution`.
+> **Ask again every single time — a resolution the user picked for one clip is
+> NEVER carried over to the next generation. Do not assume, do not reuse a
+> prior answer, do not batch. One generation = one fresh resolution question.**
+> **When editing or continuing/extending an existing video, default to that
+> source video's resolution — never silently upgrade it (e.g. don't turn a
+> 720p source into a 4k render).** If the user gave no answer and you didn't
+> ask, the server pins `720p`. **These Seedance-only fields (`resolution`,
+> `generate_audio`, `watermark`, `seed`, `first_frame`, `last_frame`,
+> `reference_*`) have NO effect on Veo or Qwen/DashScope backends — never send
+> them unless the Base URL is `*.volces.com`.**
+
+| Field              | Values                                   | Notes |
+|--------------------|------------------------------------------|-------|
+| `aspect_ratio`     | `landscape`/`portrait`/`square`, or a raw Ark ratio like `16:9`, `9:16`, `4:3`, `3:4`, `21:9`, `adaptive` | Raw ratios pass through unchanged. |
+| `duration_seconds` | integer, or `-1`                         | `-1` lets the model pick the length (Seedance 2.0 / 1.5 Pro). |
+| `resolution`       | `480p` / `720p` / `1080p` / `4k`         | **Defaults to `720p` when omitted** (cost control). Ask the user before every generation — never reuse a prior answer. See the cost gate above. Model-dependent; unsupported values are rejected upstream. |
+| `generate_audio`   | `true` / `false`                         | Seedance 2.0 / 1.5 Pro can synthesize a synced audio track. |
+| `watermark`        | `true` / `false`                         | |
+| `seed`             | integer                                  | Reproducibility. |
+| `first_frame`      | media ref (see below)                    | First frame → image-to-video. |
+| `last_frame`       | media ref                                | Together with `first_frame` → first+last-frame video. |
+| `reference_images` | array of media refs (0–9)                | Reference images. |
+| `reference_videos` | array of media refs (0–3)                | Reference videos. |
+| `reference_audios` | array of media refs (0–3)                | Reference audio (background music / voice). |
+
+**Which fields for which task** — Seedance covers six capabilities; pick the
+fields by intent, and never mix the two families below:
+
+| Task | What you want | Fields to send |
+|------|---------------|----------------|
+| Text-to-video | a clip from a prompt only | `prompt` (no media) |
+| Image-to-video (first frame) | animate a still image forward | `first_frame` |
+| Image-to-video (first + last frame) | interpolate between two stills | `first_frame` + `last_frame` |
+| Multimodal generation | new clip guided by reference images/videos/audio | `reference_images` / `reference_videos` / `reference_audios` |
+| **Edit an existing video** | replace/add/remove/repaint something *inside* a given video | `reference_videos: [<the video to edit>]` (+ optional `reference_images` / `reference_audios`) + a prompt describing the edit |
+| **Extend / continue a video** | prepend/append or stitch clips into one | `reference_videos: [<clip1>, <clip2>, ...]` (up to 3) + a prompt describing the join |
+
+> **🚫 Hard rule — the two families are mutually exclusive.**
+> `first_frame`/`last_frame` **cannot** be combined with any `reference_*`
+> field; Ark rejects the request. If the user wants to **edit or extend an
+> existing video, that is a `reference_videos` task — do NOT fall back to
+> extracting a frame and using `first_frame`** (that produces a brand-new clip
+> and silently loses the "edit the original" intent). The server also enforces
+> this and returns a clear `invalid_argument` error if you mix them.
+
+A **media ref** may be:
+- a public `http(s)://` URL, or a `data:` URL, or
+- a local file path (the server reads and base64-encodes it), or
+- a `{ "b64_json": "...", "mime_type": "image/png" }` hash.
+
+Note: audio cannot be sent alone — pair it with at least one image or video.
+Prefer passing large videos/audios as public URLs; base64-encoding a big local
+file can exceed upstream size limits.
+
+Example — **first + last frame** (image-to-video, no `reference_*`):
+
+```bash
+curl -s -X POST http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/video \
+  -H "Content-Type: application/json" \
+  -d '{
+    "prompt": "First-person POV, a hand raises a cup of fruit tea toward the camera, bright and refreshing lighting",
+    "aspect_ratio": "9:16",
+    "duration_seconds": 8,
+    "resolution": "720p",
+    "first_frame": "'"$(pwd)"'/assets/frame_first.jpg",
+    "last_frame": "'"$(pwd)"'/assets/frame_last.jpg",
+    "output_dir": "'"$(pwd)"'",
+    "session_id": "<%= session_id %>"
+  }'
+```
+
+Example — **edit an existing video** (replace/add/remove something inside it;
+uses `reference_videos`, NOT `first_frame`):
+
+```bash
+curl -s -X POST http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/video \
+  -H "Content-Type: application/json" \
+  -d '{
+    "prompt": "Add a small wooden fishing boat with a warm lantern drifting slowly across the lake in the foreground, keep everything else unchanged",
+    "resolution": "720p",
+    "reference_videos": ["'"$(pwd)"'/assets/original.mp4"],
+    "output_dir": "'"$(pwd)"'",
+    "session_id": "<%= session_id %>"
+  }'
+```
+
+**Seedance is asynchronous — POST only submits, it does NOT return the video.**
+Unlike Veo (which blocks and returns the mp4 in one call), the Seedance POST
+returns immediately with a task id:
+
+```json
+{ "success": true, "status": "submitted", "task_id": "cgt-2024...-xxxx", "provider": "volcengine" }
+```
+
+`status: "submitted"` means the render is now running on Volcengine's servers
+and **is already being billed** — it does NOT mean it is done. You MUST now
+poll for completion: sleep ~15 seconds, then query the status endpoint, and
+repeat until it is `succeeded` (or `failed`):
+
+```bash
+curl -s "http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/video/status?task_id=cgt-2024...-xxxx&output_dir=$(pwd)&session_id=<%= session_id %>"
+```
+
+Status responses:
+
+```json
+{ "success": true,  "status": "running" }                         // keep polling
+{ "success": true,  "status": "succeeded", "video": "/abs/path.mp4" }  // done — this is the file
+{ "success": false, "status": "failed", "error": "..." }          // give up, report to user
+```
+
+Only once you receive `status: "succeeded"` and the absolute `video` path may
+you present the result to the user. Do NOT end your turn while the task is
+still `submitted`/`running` — the user is waiting for the finished video.
+
+A minimal poll loop:
+
+```bash
+TASK_ID=$(curl -s -X POST http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/video \
+  -H "Content-Type: application/json" \
+  -d '{"prompt":"...","resolution":"720p","output_dir":"'"$(pwd)"'","session_id":"<%= session_id %>"}' \
+  | sed -n 's/.*"task_id"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p')
+
+while true; do
+  sleep 15
+  RESP=$(curl -s "http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/video/status?task_id=${TASK_ID}&output_dir=$(pwd)&session_id=<%= session_id %>")
+  STATUS=$(echo "$RESP" | sed -n 's/.*"status"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p')
+  echo "poll: $STATUS"
+  [ "$STATUS" = "succeeded" ] && echo "$RESP" && break
+  [ "$STATUS" = "failed" ] && echo "$RESP" && break
+done
+```
+
+**Hard rules — a broken version of this once doubled a user's bill:**
+
+- ❌ **Never POST the same generation twice.** Once you have a `task_id`, the
+  only valid next action is polling `/api/media/video/status`. A slow render
+  is not a failed one.
+- ⚠️ **A timeout or error is NOT proof the task failed.** The task keeps
+  running and billing on Volcengine's side. Always query the status endpoint
+  to find out the real state before doing anything else — never resubmit.
+- ❌ **Never kill the poll to "cancel" the job.** Killing your curl/session
+  does not stop the Volcengine task; it keeps running and billing. A running
+  task also cannot be deleted upstream.
+- ❌ **Never bypass `/api/media/*` to call Volcengine's native API directly.**
+  All submission and status checks must go through this server (it meters
+  cost). There is no reason to touch the raw Ark API.
+- ⏱️ If polling exceeds ~15 minutes and status is still `running`, stop
+  polling and tell the user the task is still rendering in the background,
+  give them the `task_id`, and let them check again later — do NOT resubmit.
+
 ## Generating speech (Gemini TTS)
 
 The same `/api/media/` namespace serves text-to-speech. The user must
@@ -328,7 +502,8 @@ curl -s -X POST http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/aud
   -d '{
     "input": "Hello and welcome to openclacky. Today we will explore...",
     "voice": "Kore",
-    "output_dir": "'"$(pwd)"'"
+    "output_dir": "'"$(pwd)"'",
+    "session_id": "<%= session_id %>"
   }'
 ```
 
@@ -337,6 +512,7 @@ curl -s -X POST http://${CLACKY_SERVER_HOST}:${CLACKY_SERVER_PORT}/api/media/aud
 | `input`      | yes      | string                          | The text to speak. Plain prose works best; you can prefix with style cues like "Say cheerfully:" or "In a calm tone:". |
 | `voice`      | no       | string voice name               | Defaults to `Kore`. Common Gemini voices: `Kore`, `Puck`, `Charon`, `Fenrir`, `Aoede`. |
 | `output_dir` | yes      | absolute path                   | Always pass `$(pwd)` so files land in the current session workspace. WAV saved under `<output_dir>/assets/generated/`. |
+| `session_id` | yes      | string                          | Current Clacky session ID. Always pass the rendered value shown in the request example. |
 
 Generation typically takes 2–10 seconds depending on length. The request
 blocks until the WAV is ready.
@@ -388,4 +564,3 @@ Same shape and `error_type` values as image generation, but with `"audio": null`
 - Voice consistency: Gemini TTS does not currently support voice cloning;
   use the same `voice` name across calls in one project to keep the
   narrator consistent.
-

data/lib/clacky/default_skills/media-gen/scripts/video_seq.sh

@@ -10,7 +10,7 @@
 # Subcommands:
 #   lastframe  <video.mp4> <out.jpg>           extract the final frame (JPEG by default)
 #   tob64      <image>                          print base64 (no newlines) to stdout
-#   payload    <out.json> <frame.jpg> <dur> <aspect> <output_dir> <prompt>
+#   payload    <out.json> <frame.jpg> <dur> <aspect> <output_dir> <prompt> [session_id]
 #                                               build an image-to-video JSON body
 #                                               for `curl --data @out.json`
 #   concat     <out.mp4> <clip1.mp4> [clip2 …]  losslessly join clips in order
@@ -43,7 +43,7 @@ cmd_tob64() {
 # Build the image-to-video request body as a file so curl can send it with
 # `--data @file`, avoiding "Argument list too long" from inlining base64.
 cmd_payload() {
-  local out="$1" frame="$2" dur="$3" aspect="$4" odir="$5" prompt="$6"
+  local out="$1" frame="$2" dur="$3" aspect="$4" odir="$5" prompt="$6" session_id="${7:-}"
   [[ -f "$frame" ]] || die "no such frame: $frame"
   need ffprobe
   local mime b64
@@ -53,7 +53,7 @@ cmd_payload() {
   esac
   b64="$(base64 < "$frame" | tr -d '\n')"
   FRAME_B64="$b64" FRAME_MIME="$mime" P_PROMPT="$prompt" P_DUR="$dur" \
-  P_ASPECT="$aspect" P_ODIR="$odir" python3 - "$out" <<'PY'
+  P_ASPECT="$aspect" P_ODIR="$odir" P_SESSION_ID="$session_id" python3 - "$out" <<'PY'
 import json, os, sys
 body = {
   "prompt": os.environ["P_PROMPT"],
@@ -62,6 +62,8 @@ body = {
   "output_dir": os.environ["P_ODIR"],
   "image": {"b64_json": os.environ["FRAME_B64"], "mime_type": os.environ["FRAME_MIME"]},
 }
+if os.environ["P_SESSION_ID"]:
+  body["session_id"] = os.environ["P_SESSION_ID"]
 open(sys.argv[1], "w").write(json.dumps(body))
 PY
   [[ -f "$out" ]] || die "failed to write payload"

data/lib/clacky/extension/packager.rb

@@ -17,6 +17,9 @@ module Clacky
     MANIFEST     = "ext.yml"
     MAX_ZIP_SIZE = 50 * 1024 * 1024
 
+    # Platform metadata that leaks in from the developer's OS; never ship it.
+    SYSTEM_METADATA = [".DS_Store", "__MACOSX", "Thumbs.db", "desktop.ini"].freeze
+
     Result = Struct.new(:ext_id, :path, :units, keyword_init: true)
 
     class Error < StandardError; end
@@ -111,6 +114,8 @@ module Clacky
             next if base == "." || base == ".."
 
             rel = relative(container_dir, abs)
+            next if system_metadata?(rel)
+
             entry = File.join(File.basename(container_dir), rel)
             if File.directory?(abs)
               zip.mkdir(entry) unless zip.find_entry(entry)
@@ -121,6 +126,12 @@ module Clacky
         end
       end
 
+      # True if any path segment is a platform metadata file/dir (e.g. a nested
+      # agents/.DS_Store or a whole __MACOSX/ tree).
+      private def system_metadata?(rel)
+        rel.split(File::SEPARATOR).any? { |seg| SYSTEM_METADATA.include?(seg) }
+      end
+
       private def local_zip_for(source, tmp)
         src = source.to_s
         if src.match?(%r{\Ahttps?://})

data/lib/clacky/extension/verifier.rb

@@ -19,7 +19,7 @@ module Clacky
     KNOWN_TOP_KEYS    = %w[id name title description version origin author homepage license public license_required keywords contributes].freeze
     KNOWN_CONTRIBUTES = %w[panels api skills agents channels patches hooks].freeze
 
-    PANEL_KEYS   = %w[id title title_zh description description_zh view order attach].freeze
+    PANEL_KEYS   = %w[id title title_zh description description_zh view order attach entry_points].freeze
     API_KEYS     = %w[id handler].freeze
     SKILL_KEYS   = %w[id dir protected].freeze
     AGENT_KEYS   = %w[id title title_zh description description_zh order prompt panels skills avatar].freeze

data/lib/clacky/media/generator.rb

@@ -3,6 +3,7 @@
 require_relative "openai_compat"
 require_relative "gemini"
 require_relative "dashscope"
+require_relative "volcengine"
 
 module Clacky
   module Media
@@ -34,6 +35,13 @@ module Clacky
         "aliyuncs.com"
       ].freeze
 
+      # Hosts that speak Volcengine Ark's native async video-task API
+      # (ByteDance Doubao Seedance) instead of an OpenAI-compatible facade.
+      # Matched as a substring so regional variants are caught.
+      VOLCENGINE_NATIVE_HOSTS = [
+        "volces.com"
+      ].freeze
+
       # @param agent_config [Clacky::AgentConfig]
       def initialize(agent_config)
         @agent_config = agent_config
@@ -101,6 +109,34 @@ module Clacky
         )
       end
 
+      def video_status(task_id:, output_dir: nil)
+        entry = video_model_entry
+        if entry.nil?
+          return {
+            "success"    => false,
+            "status"     => "failed",
+            "error"      => "No video model configured. Add a model with type=video in settings.",
+            "error_type" => "not_configured",
+            "provider"   => "",
+            "model"      => ""
+          }
+        end
+
+        provider = build_provider_for(entry)
+        unless provider.respond_to?(:video_status)
+          return {
+            "success"    => false,
+            "status"     => "failed",
+            "error"      => "Status queries are only supported for Volcengine (Seedance) video; this provider generates synchronously.",
+            "error_type" => "unsupported",
+            "provider"   => "",
+            "model"      => ""
+          }
+        end
+
+        provider.video_status(task_id: task_id, output_dir: output_dir)
+      end
+
       def generate_speech(input:, voice: nil, output_dir: nil, **kwargs)
         entry = audio_model_entry
         if entry.nil?
@@ -185,6 +221,9 @@ module Clacky
       #     DashScope (native /api/v1/.../multimodal-generation schema for
       #     Qwen-Image). Third-party aggregators re-exposing qwen-image behind
       #     an OpenAI-compatible facade are NOT on aliyuncs.com and fall through.
+      #   • base_url points at a Volcengine Ark host (*.volces.com) →
+      #     Volcengine (native async /api/v3/contents/generations/tasks schema
+      #     for Doubao Seedance video).
       #   • everything else → OpenAICompat. This covers OpenAI itself, the
       #     openclacky gateway, OpenRouter, and any third-party proxy that
       #     re-exposes Gemini / Imagen / DALL-E behind /v1/images/generations.
@@ -196,6 +235,8 @@ module Clacky
           Gemini.new(entry)
         elsif DASHSCOPE_NATIVE_HOSTS.any? { |host| url.include?(host) }
           DashScope.new(entry)
+        elsif VOLCENGINE_NATIVE_HOSTS.any? { |host| url.include?(host) }
+          Volcengine.new(entry)
         else
           OpenAICompat.new(entry)
         end