@renoise/plugin 0.2.1

package/skills/gemini-gen/SKILL.md

@@ -0,0 +1,236 @@
+---
+name: gemini-gen
+description: >
+  Visual understanding and multimodal analysis via Gemini 3.1 Pro.
+  Use when you need to understand, analyze, or extract information from images or videos:
+  analyze product photos, extract video scripts/dialogue, understand video content for replication,
+  compare visual assets, OCR/text extraction from images, describe scenes for prompt writing,
+  extract style/color/composition from reference footage.
+  Do NOT use for generating images or videos — use renoise-gen instead.
+allowed-tools: Bash, Read
+metadata:
+  author: renoise
+  version: 0.2.0
+  category: ai-foundation
+  tags: [vision, analysis, multimodal, gemini]
+---
+
+# Gemini Gen — Visual Understanding & Multimodal Analysis
+
+Gemini 3.1 Pro via Renoise gateway. Zero npm dependencies, native `fetch` only.
+
+## When to Use
+
+| Scenario                        | Example                                                                                            |
+| ------------------------------- | -------------------------------------------------------------------------------------------------- |
+| **Analyze product photos**      | Extract type, color, material, selling points, brand tone from product images                      |
+| **Understand video content**    | "What happens in this video?", summarize scenes, identify actions and objects                      |
+| **Extract scripts from video**  | Watch a reference video → output timestamped dialogue, scene descriptions, camera movements        |
+| **Replicate a video style**     | Analyze a reference clip → extract visual style, pacing, transitions, color grading for recreation |
+| **Compare visual assets**       | Side-by-side analysis of two product photos, before/after comparison                               |
+| **OCR / text extraction**       | Read text from screenshots, packaging, signage in images                                           |
+| **Describe scenes for prompts** | Look at a reference image → write a detailed prompt for `renoise-gen` to recreate the style        |
+| **Content review**              | Check if generated output matches the creative brief                                               |
+
+## When NOT to Use
+
+- **Generating images** → use `renoise-gen` with `nano-banana-2`
+- **Generating videos** → use `renoise-gen` with `renoise-2.0`
+- **Uploading large files** → use `file-upload` first, then pass the URL here
+
+## Authentication
+
+Use environment variable `RENOISE_API_KEY`. Get one at: https://www.renoise.ai
+
+## API Endpoint
+
+```
+POST https://renoise.ai/api/public/llm/proxy/v1beta/models/{model}:generateContent?key={RENOISE_API_KEY}
+```
+
+Default model: `gemini-3.1-pro`
+
+## Request Format
+
+```json
+{
+  "contents": [
+    {
+      "role": "user",
+      "parts": [{ "text": "your prompt here" }]
+    }
+  ],
+  "generationConfig": {
+    "temperature": 1.0,
+    "maxOutputTokens": 8192
+  }
+}
+```
+
+## Multimodal Input (images & videos)
+
+Gemini supports images and videos as inline base64 parts alongside text. Multiple files can be sent in the same request.
+
+### Image Input
+
+```json
+{
+  "contents": [
+    {
+      "parts": [
+        {
+          "inlineData": {
+            "mimeType": "image/jpeg",
+            "data": "<base64-encoded-data>"
+          },
+          "mediaResolution": { "level": "media_resolution_high" }
+        },
+        { "text": "Describe this image" }
+      ]
+    }
+  ]
+}
+```
+
+`mediaResolution` controls token allocation per image/frame:
+
+| Level                         | Image Tokens | Video Frame Tokens |
+| ----------------------------- | ------------ | ------------------ |
+| `media_resolution_low`        | 280          | 70                 |
+| `media_resolution_medium`     | 560          | 140                |
+| `media_resolution_high`       | 840          | 210                |
+| `media_resolution_ultra_high` | 1120         | 280                |
+
+Default: `media_resolution_medium`. Use `high` or `ultra_high` for detail-critical analysis (product photos, fine text). Use `low` for bulk/batch processing.
+
+### Video Input
+
+Same format — just use a video MIME type. Inline base64 has a **20MB limit**. For larger videos, use the dedicated file upload skill.
+
+```json
+{
+  "contents": [
+    {
+      "parts": [
+        {
+          "inlineData": {
+            "mimeType": "video/mp4",
+            "data": "<base64-encoded-data>"
+          },
+          "mediaResolution": { "level": "media_resolution_low" }
+        },
+        { "text": "Summarize what happens in this video" }
+      ]
+    }
+  ]
+}
+```
+
+> **Tip**: Use `media_resolution_low` for videos to reduce token consumption — video has many frames.
+
+### Multiple Files
+
+Send multiple images/videos in one request by adding more `inlineData` parts:
+
+```json
+{
+  "contents": [
+    {
+      "parts": [
+        {
+          "inlineData": { "mimeType": "image/jpeg", "data": "<base64-image-1>" }
+        },
+        {
+          "inlineData": { "mimeType": "image/png", "data": "<base64-image-2>" }
+        },
+        { "text": "Compare these two product photos" }
+      ]
+    }
+  ]
+}
+```
+
+### Large Files (> 20MB)
+
+Inline base64 has a **20MB limit**. For larger files, use the dedicated file upload skill to get a file URI, then reference it:
+
+```json
+{
+  "contents": [
+    {
+      "parts": [
+        {
+          "fileData": {
+            "mimeType": "video/mp4",
+            "fileUri": "<uploaded-file-uri>"
+          }
+        },
+        { "text": "Analyze this video" }
+      ]
+    }
+  ]
+}
+```
+
+## Supported MIME Types
+
+| Extension  | MIME Type       | Max Inline |
+| ---------- | --------------- | ---------- |
+| .jpg/.jpeg | image/jpeg      | 20MB       |
+| .png       | image/png       | 20MB       |
+| .webp      | image/webp      | 20MB       |
+| .gif       | image/gif       | 20MB       |
+| .mp4       | video/mp4       | 20MB       |
+| .mov       | video/quicktime | 20MB       |
+| .webm      | video/webm      | 20MB       |
+
+## Response Parsing
+
+```javascript
+const data = await response.json();
+const text = data.candidates[0].content.parts[0].text;
+```
+
+## Error Handling
+
+- 400: Bad request (check prompt format)
+- 403: Invalid API key
+- 429: Rate limited (wait and retry)
+- 500: Server error (retry with backoff)
+
+## CLI Script
+
+`${CLAUDE_SKILL_DIR}/scripts/gemini.mjs` — zero-dependency Node.js script, other skills can directly call it.
+
+```bash
+# Text only
+node ${CLAUDE_SKILL_DIR}/scripts/gemini.mjs "Explain quantum computing"
+
+# Analyze an image (high resolution for product detail)
+node ${CLAUDE_SKILL_DIR}/scripts/gemini.mjs --file photo.jpg --resolution high "Describe this product"
+
+# Analyze a video (low resolution to save tokens)
+node ${CLAUDE_SKILL_DIR}/scripts/gemini.mjs --file clip.mp4 --resolution low "Summarize this clip"
+
+# Multiple images
+node ${CLAUDE_SKILL_DIR}/scripts/gemini.mjs --file a.jpg --file b.jpg "Compare these two"
+
+# Uploaded file URI (from file upload skill, for files > 20MB)
+node ${CLAUDE_SKILL_DIR}/scripts/gemini.mjs --file-uri "<uri>" --file-mime video/mp4 "Analyze this video"
+
+# JSON output mode
+node ${CLAUDE_SKILL_DIR}/scripts/gemini.mjs --json "Return a JSON object with name and age"
+```
+
+### Options
+
+| Flag                   | Default          | Description                                          |
+| ---------------------- | ---------------- | ---------------------------------------------------- |
+| `--file <path>`        | —                | Attach local file (repeatable)                       |
+| `--file-uri <uri>`     | —                | Attach uploaded file by URI (requires `--file-mime`) |
+| `--file-mime <mime>`   | —                | MIME type for `--file-uri`                           |
+| `--resolution <level>` | `medium`         | `low` / `medium` / `high` / `ultra_high`             |
+| `--model <name>`       | `gemini-3.1-pro` | Model name                                           |
+| `--temperature <n>`    | `1.0`            | Temperature                                          |
+| `--max-tokens <n>`     | `8192`           | Max output tokens                                    |
+| `--json`               | off              | Request JSON response format                         |

package/skills/gemini-gen/scripts/gemini.mjs

@@ -0,0 +1,220 @@
+#!/usr/bin/env node
+
+/**
+ * Gemini API client via Renoise gateway.
+ * Zero npm dependencies — uses native fetch.
+ *
+ * Usage:
+ *   # Text only
+ *   node gemini.mjs "Explain quantum computing"
+ *
+ *   # With image(s)
+ *   node gemini.mjs --file photo.jpg "Describe this product"
+ *   node gemini.mjs --file a.jpg --file b.jpg "Compare these two"
+ *
+ *   # With video
+ *   node gemini.mjs --file clip.mp4 --resolution low "Summarize this clip"
+ *
+ *   # With uploaded file URI (from file upload skill)
+ *   node gemini.mjs --file-uri "https://...fileUri" --file-mime video/mp4 "Analyze this video"
+ *
+ *   # JSON output mode
+ *   node gemini.mjs --json "Return a JSON object with name and age"
+ *
+ * Options:
+ *   --file <path>         Attach a local file (image/video). Repeatable.
+ *   --file-uri <uri>      Attach an uploaded file by URI. Requires --file-mime.
+ *   --file-mime <mime>     MIME type for --file-uri.
+ *   --resolution <level>  Media resolution: low|medium|high|ultra_high (default: medium)
+ *   --model <name>        Model name (default: gemini-3.1-pro)
+ *   --temperature <n>     Temperature (default: 1.0)
+ *   --max-tokens <n>      Max output tokens (default: 8192)
+ *   --json                Request JSON response format
+ *
+ * Environment:
+ *   RENOISE_API_KEY       Required. Get one at https://www.renoise.ai
+ */
+
+import fs from "fs/promises";
+import path from "path";
+
+// --- Auth ---
+const RENOISE_API_KEY = process.env.RENOISE_API_KEY;
+if (!RENOISE_API_KEY) {
+  console.error(
+    "RENOISE_API_KEY not set. Get one at: https://www.renoise.ai"
+  );
+  process.exit(1);
+}
+
+// --- MIME detection ---
+const MIME_MAP = {
+  ".jpg": "image/jpeg",
+  ".jpeg": "image/jpeg",
+  ".png": "image/png",
+  ".webp": "image/webp",
+  ".gif": "image/gif",
+  ".mp4": "video/mp4",
+  ".mov": "video/quicktime",
+  ".webm": "video/webm",
+};
+
+function getMimeType(filePath) {
+  return (
+    MIME_MAP[path.extname(filePath).toLowerCase()] ?? "application/octet-stream"
+  );
+}
+
+// --- Resolution mapping ---
+const RESOLUTION_MAP = {
+  low: "media_resolution_low",
+  medium: "media_resolution_medium",
+  high: "media_resolution_high",
+  ultra_high: "media_resolution_ultra_high",
+};
+
+// --- Parse args ---
+function parseArgs(argv) {
+  const files = [];
+  let fileUri = null;
+  let fileMime = null;
+  let resolution = "medium";
+  let model = "gemini-3.1-pro";
+  let temperature = 1.0;
+  let maxTokens = 8192;
+  let json = false;
+  const textParts = [];
+
+  for (let i = 0; i < argv.length; i++) {
+    switch (argv[i]) {
+      case "--file":
+        files.push(argv[++i]);
+        break;
+      case "--file-uri":
+        fileUri = argv[++i];
+        break;
+      case "--file-mime":
+        fileMime = argv[++i];
+        break;
+      case "--resolution":
+        resolution = argv[++i];
+        break;
+      case "--model":
+        model = argv[++i];
+        break;
+      case "--temperature":
+        temperature = parseFloat(argv[++i]);
+        break;
+      case "--max-tokens":
+        maxTokens = parseInt(argv[++i], 10);
+        break;
+      case "--json":
+        json = true;
+        break;
+      default:
+        textParts.push(argv[i]);
+    }
+  }
+
+  return {
+    files,
+    fileUri,
+    fileMime,
+    resolution,
+    model,
+    temperature,
+    maxTokens,
+    json,
+    prompt: textParts.join(" "),
+  };
+}
+
+// --- Build parts ---
+async function buildParts(opts) {
+  const parts = [];
+  const resLevel = RESOLUTION_MAP[opts.resolution] ?? RESOLUTION_MAP.medium;
+
+  // Local files → inline base64
+  for (const filePath of opts.files) {
+    const data = await fs.readFile(filePath);
+    parts.push({
+      inlineData: {
+        mimeType: getMimeType(filePath),
+        data: data.toString("base64"),
+      },
+      mediaResolution: { level: resLevel },
+    });
+  }
+
+  // Uploaded file URI
+  if (opts.fileUri) {
+    parts.push({
+      fileData: {
+        mimeType: opts.fileMime ?? "application/octet-stream",
+        fileUri: opts.fileUri,
+      },
+    });
+  }
+
+  // Text prompt (always last)
+  if (opts.prompt) {
+    parts.push({ text: opts.prompt });
+  }
+
+  return parts;
+}
+
+// --- Main ---
+async function main() {
+  const opts = parseArgs(process.argv.slice(2));
+
+  if (!opts.prompt && opts.files.length === 0 && !opts.fileUri) {
+    console.error(
+      "Usage: node gemini.mjs [--file <path>] [--resolution low|medium|high|ultra_high] <prompt>"
+    );
+    process.exit(1);
+  }
+
+  const endpoint = `https://renoise.ai/api/public/llm/proxy/v1beta/models/${opts.model}:generateContent?key=${RENOISE_API_KEY}`;
+
+  const parts = await buildParts(opts);
+
+  const body = {
+    contents: [{ role: "user", parts }],
+    generationConfig: {
+      temperature: opts.temperature,
+      maxOutputTokens: opts.maxTokens,
+    },
+  };
+
+  if (opts.json) {
+    body.generationConfig.responseMimeType = "application/json";
+  }
+
+  const res = await fetch(endpoint, {
+    method: "POST",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify(body),
+  });
+
+  if (!res.ok) {
+    const errText = await res.text();
+    console.error(`Gemini API error ${res.status}: ${errText}`);
+    process.exit(1);
+  }
+
+  const data = await res.json();
+  const text = data.candidates?.[0]?.content?.parts?.[0]?.text;
+
+  if (!text) {
+    console.error("No text in response:", JSON.stringify(data, null, 2));
+    process.exit(1);
+  }
+
+  console.log(text);
+}
+
+main().catch((err) => {
+  console.error("ERROR:", err.message);
+  process.exit(1);
+});

package/skills/product-sheet-generate/SKILL.md

@@ -0,0 +1,75 @@
+---
+name: product-sheet-generate
+description: >
+  Generates multi-angle Product Design Sheet from product photos using Renoise
+  nano-banana-2. Shows front/back/side views, color palette, materials, and
+  construction details. Use when user says "product design sheet", "product
+  reference image", "multi-angle product view", or needs a visual reference
+  for video production. Do NOT use for scene backgrounds or video generation.
+allowed-tools: Bash, Read
+metadata:
+  author: renoise
+  version: 0.2.0
+  category: video-production
+  tags: [product, design, renoise]
+---
+
+# Product Design Sheet Generation
+
+Generate a comprehensive Product Design Sheet image from product photos using Renoise `nano-banana-2` model.
+
+## Arguments
+
+- First argument — Product images directory or single image path (required)
+- Second argument — Output directory (required, e.g. `output/foam-roller-20260313-1420/analysis`)
+
+## What is a Product Design Sheet?
+
+A single image containing:
+- Multiple angle views (front, back, side, three-quarter, bottom)
+- Color palette with Pantone or hex references
+- Material and texture callouts
+- Construction details and hardware close-ups
+- Proportions and measurements
+- Interior views (if applicable)
+
+## Instructions
+
+1. List images in the input path to confirm they exist.
+
+2. Upload each product image as a material:
+
+```bash
+node ${CLAUDE_SKILL_DIR}/../renoise-gen/renoise-cli.mjs material upload "<image_path>"
+```
+
+Note the material ID from each upload output (e.g. `#42`).
+
+3. Write an English prompt for the design sheet. Example:
+
+> Based on the provided product reference photos, generate a professional Product Design Sheet on a single clean white background. Include: multiple angle views (front, back, side, three-quarter, top), color palette swatches, material and texture close-ups, construction detail callouts, and relative proportions. Layout: clean organized grid. Photorealistic rendering. No cartoons or illustrations.
+
+Add `Do NOT include any text, labels, titles, annotations, or typography.` if user wants no-text version.
+
+4. Generate the design sheet, referencing the uploaded materials:
+
+```bash
+node ${CLAUDE_SKILL_DIR}/../renoise-gen/renoise-cli.mjs task generate \
+  --model nano-banana-2 \
+  --prompt "<english_prompt>" \
+  --materials "<id1>:ref_image,<id2>:image1,<id3>:image2" \
+  --resolution 2k \
+  --ratio 1:1
+```
+
+Assign material roles: first image as `ref_image`, additional images as `image1`, `image2`, etc.
+
+5. The command will output an `imageUrl` when complete. Download the image:
+
+```bash
+curl -sL "<imageUrl>" -o "<output_dir>/product_design_sheet.png"
+```
+
+6. Verify the output image was created and print file size.
+
+7. Show the generated image to the user for approval. If not satisfactory, adjust the prompt and regenerate.