@kolbo/mcp 1.21.0 → 1.22.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +15 -15
- package/package.json +2 -1
- package/skill/SKILL.md +251 -0
- package/skill/VERSION +1 -0
- package/skill/references/models/creative-director.md +106 -0
- package/skill/references/models/gpt-image.md +111 -0
- package/skill/references/models/html-presentation.md +139 -0
- package/skill/references/models/landing-page.md +135 -0
- package/skill/references/models/music.md +120 -0
- package/skill/references/models/nano-banana.md +97 -0
- package/skill/references/models/prompt-copilot.md +133 -0
- package/skill/references/models/seedance.md +90 -0
- package/skill/references/models/veo.md +110 -0
- package/skill/references/models/visual-code.md +80 -0
- package/skill/references/workflows/app-builder.md +41 -0
- package/skill/references/workflows/cost-and-validation.md +138 -0
- package/skill/references/workflows/dtc-ads.md +126 -0
- package/skill/references/workflows/marketing-studio.md +157 -0
- package/skill/references/workflows/marketplace-cards.md +146 -0
- package/skill/references/workflows/media-library.md +76 -0
- package/skill/references/workflows/product-photoshoot.md +199 -0
- package/skill/references/workflows/production-log.md +155 -0
- package/skill/references/workflows/research-first.md +174 -0
- package/skill/references/workflows/transcription.md +163 -0
- package/skill/references/workflows/troubleshooting.md +73 -0
- package/skill/references/workflows/visual-dna.md +233 -0
- package/src/install.js +29 -0
|
@@ -0,0 +1,155 @@
|
|
|
1
|
+
# Production Log — `.kolbo/production.md`
|
|
2
|
+
|
|
3
|
+
Load this file when starting a multi-step production, or before any continuation of prior media work ("edit", "redo", "the same character", `@name` references, "scene N").
|
|
4
|
+
|
|
5
|
+
## Why It Exists
|
|
6
|
+
|
|
7
|
+
Every URL, id, and brief produced by a Kolbo MCP tool MUST be recorded in `.kolbo/production.md` in the user's workspace. This file — not chat history — is your source of truth for prior artifacts: URLs scattered across `tool_result` blobs are unreliable to re-scan and disappear entirely on context compaction.
|
|
8
|
+
|
|
9
|
+
## When to READ it
|
|
10
|
+
|
|
11
|
+
Read `.kolbo/production.md` **before** acting on any of these signals:
|
|
12
|
+
- "edit", "animate", "combine", "redo", "polish", "fix", "regenerate"
|
|
13
|
+
- "the same character / scene / image / video / sound", "that X", "scene N", "the rainy one", etc.
|
|
14
|
+
- `@name` references for Visual DNA
|
|
15
|
+
- Any continuation of prior media work ("now make scene 3")
|
|
16
|
+
|
|
17
|
+
If the file is missing and the user is referencing prior media, ask the user — do not guess from chat.
|
|
18
|
+
|
|
19
|
+
## When to WRITE to it
|
|
20
|
+
|
|
21
|
+
**Immediately after every successful generation tool call**, before your next tool call or your final reply. The runtime will inject a reminder after generation tool results — treat that as a hard rule, not a suggestion.
|
|
22
|
+
|
|
23
|
+
Tools that REQUIRE logging:
|
|
24
|
+
- `generate_image`, `generate_image_edit`, `edit_image`
|
|
25
|
+
- `generate_video`, `generate_video_from_image`, `generate_video_from_video`, `edit_video`
|
|
26
|
+
- `generate_elements`, `generate_first_last_frame`, `generate_lipsync`
|
|
27
|
+
- `generate_music`, `generate_sound`, `generate_speech`
|
|
28
|
+
- `generate_3d`, `generate_creative_director`
|
|
29
|
+
- `create_visual_dna`, `upload_media`
|
|
30
|
+
|
|
31
|
+
Tools that do NOT log: `list_*`, `get_*`, `check_credits`, `chat_*`, `transcribe_audio` (read-only / discovery).
|
|
32
|
+
|
|
33
|
+
## File creation — pick the right tool to avoid the "must Read first" error
|
|
34
|
+
|
|
35
|
+
`Edit` refuses to overwrite a file unless you've `Read` it first in the same session. Pick by file state:
|
|
36
|
+
|
|
37
|
+
| State | Tool |
|
|
38
|
+
|---|---|
|
|
39
|
+
| File **does not exist** (typical first turn) | `Write` with the full stub below |
|
|
40
|
+
| File **exists** | `Read` first, then `Edit` |
|
|
41
|
+
| Not sure | `Read` first; on ENOENT, fall back to `Write` |
|
|
42
|
+
|
|
43
|
+
Stub for first creation:
|
|
44
|
+
|
|
45
|
+
```md
|
|
46
|
+
<!-- .kolbo/production.md — agent-managed media artifact registry.
|
|
47
|
+
User may hand-edit; agent must Read-before-Edit to reconcile. -->
|
|
48
|
+
|
|
49
|
+
# Production Log
|
|
50
|
+
|
|
51
|
+
## 🎯 Now
|
|
52
|
+
|
|
53
|
+
**Brief:** <paraphrase of user's overall goal in 1-3 sentences>
|
|
54
|
+
**Now working on:** <the immediate next step>
|
|
55
|
+
**Last updated:** <ISO date>
|
|
56
|
+
|
|
57
|
+
---
|
|
58
|
+
|
|
59
|
+
## Production: <name from user's request, slugified human label>
|
|
60
|
+
|
|
61
|
+
### Cast
|
|
62
|
+
### Visual DNA
|
|
63
|
+
### Scenes
|
|
64
|
+
### Audio
|
|
65
|
+
### Final
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Subsections (`### Cast` etc.) are **suggested defaults**, not required. Adapt: a logo set has `### Logos`, an album has `### Tracks`, a 3D render has `### Models`. Leave empty subsections out of the file when you create entries.
|
|
69
|
+
|
|
70
|
+
## Entry shape
|
|
71
|
+
|
|
72
|
+
One bullet per artifact. Write the label **the way the user would reference it next time** ("the rainy one"), not the model's raw output.
|
|
73
|
+
|
|
74
|
+
```md
|
|
75
|
+
### Cast
|
|
76
|
+
- **Maya** — female, 30, urban photographer, leather jacket
|
|
77
|
+
- portrait: https://...characters/maya.png (nano-banana-2, 2026-05-13)
|
|
78
|
+
- visual DNA: vdna_8f2c (@maya)
|
|
79
|
+
|
|
80
|
+
### Scenes
|
|
81
|
+
1. **Coffee shop morning** — Maya at counter, soft light, wide shot
|
|
82
|
+
- still: https://...scenes/01-coffee.png (flux-2-pro, 2026-05-13)
|
|
83
|
+
- video: (pending)
|
|
84
|
+
2. **Rainy street walk** — neon reflections, slow dolly
|
|
85
|
+
- still: https://...scenes/02-rain.png (flux-2-pro, 2026-05-13)
|
|
86
|
+
- video: https://...videos/02-rain.mp4 (kling-2, 2026-05-13)
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
## Header rewrite rule (Manus pattern — IMPORTANT)
|
|
90
|
+
|
|
91
|
+
The `## 🎯 Now` block at the top of the file is **rewritten every turn** to keep the brief + current step near the model's recency window. Body sections (everything below the first `---`) are **append-only**.
|
|
92
|
+
|
|
93
|
+
When a user request supersedes a previous artifact (e.g., "redo scene 2 with more rain"), do not delete the old entry. Mark it `(superseded YYYY-MM-DD)` and place the new entry beneath:
|
|
94
|
+
|
|
95
|
+
```md
|
|
96
|
+
2. **Rainy street walk** — neon reflections, slow dolly
|
|
97
|
+
- still: https://...scenes/02-rain.png (superseded 2026-05-13)
|
|
98
|
+
- still: https://...scenes/02-rain-v2.png (flux-2-pro, 2026-05-13)
|
|
99
|
+
- video: https://...videos/02-rain-v2.mp4 (kling-2, 2026-05-13)
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
## Rules
|
|
103
|
+
|
|
104
|
+
1. **First touch `Write`, subsequent touches `Read` → `Edit`** (see "File creation" above). If `Edit` fails on exact-match, `Read` again — the user may have hand-edited.
|
|
105
|
+
2. **Plain English labels** — write what the user would call it.
|
|
106
|
+
3. **Append-only body.** Only the `## 🎯 Now` header is rewritten. Never delete artifact entries; mark them `(superseded)` instead.
|
|
107
|
+
4. **Do not log failures.** Only successful generations.
|
|
108
|
+
5. **Resolve user references via the log, not chat history.** If the user says "scene 3," use the URL the log says is scene 3, even if a later tool_result mentioned a different URL.
|
|
109
|
+
6. **One file per workspace.** Multiple concurrent productions go under separate `## Production: <name>` headings inside the same file.
|
|
110
|
+
|
|
111
|
+
## Bulk Generation Entry Shape
|
|
112
|
+
|
|
113
|
+
For batch runs (50-item UGC sets, etc.), persist every `generation_id` (even for failures) — required for `get_generation_status` recovery and cross-session dedupe.
|
|
114
|
+
|
|
115
|
+
```md
|
|
116
|
+
12. ✅ Asian F 24, bedroom, hype POV
|
|
117
|
+
- generation_id: gen_8a2c…
|
|
118
|
+
- url: https://…
|
|
119
|
+
- model: seedance-2 · 720p · 10s · sound-on
|
|
120
|
+
- generated: 2026-05-14T07:42Z
|
|
121
|
+
13. ❌ Latino M 31, gym
|
|
122
|
+
- generation_id: gen_ff19…
|
|
123
|
+
- error: 429 Too many generation requests
|
|
124
|
+
- retry_after: 2026-05-14T07:43Z
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
## Always log the resolution / duration / sound choices
|
|
128
|
+
|
|
129
|
+
Production-log entries should include the resolution and (for video) duration + sound state alongside the URL, so the user can see what they paid for:
|
|
130
|
+
|
|
131
|
+
```md
|
|
132
|
+
- still: https://...01-coffee.png (flux-2-pro · 1K, 2026-05-14)
|
|
133
|
+
- video: https://...02-rain.mp4 (kling-2 · 1080p · 5s · sound-off, 2026-05-14)
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
## Production Log vs TodoWrite
|
|
137
|
+
|
|
138
|
+
Use both — different jobs:
|
|
139
|
+
|
|
140
|
+
| | `.kolbo/production.md` | `TodoWrite` |
|
|
141
|
+
|---|---|---|
|
|
142
|
+
| Purpose | Durable artifact registry | Ephemeral step plan |
|
|
143
|
+
| Lifetime | Persists across sessions / compaction | Per turn / per request |
|
|
144
|
+
| Content | URLs, ids, briefs | "Do X, then Y, then Z" |
|
|
145
|
+
| Example | `still: https://...01-coffee.png` | `Generate visual DNA for Maya` |
|
|
146
|
+
|
|
147
|
+
## Real Cost Quoting
|
|
148
|
+
|
|
149
|
+
Every generation now returns `credits_used` (multiplier-adjusted total) and `credits_breakdown` (per-model attribution). **Log `credits_used` to `.kolbo/production.md`, not `base × count`.**
|
|
150
|
+
|
|
151
|
+
```json
|
|
152
|
+
{ "credits_used": 12, "credits_breakdown": [{ "model": "nano-banana-2", "base": 8, "final": 12, ... }], "urls": [...] }
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
When the user asks "how much did I spend?" → call `get_session_usage` for the real, multiplier-adjusted session total + per-tool + per-model breakdowns (same numbers as the desktop bottom-bar counter).
|
|
@@ -0,0 +1,174 @@
|
|
|
1
|
+
# Research-First Creative — Scrape Before Generating
|
|
2
|
+
|
|
3
|
+
Load this file when the brief involves a real brand, product URL, audience, or market — especially for ads, marketing creative, or anything tied to identity / brand palette / on-image text.
|
|
4
|
+
|
|
5
|
+
## Why
|
|
6
|
+
|
|
7
|
+
When the user gives you a **product URL, brand reference, or "make X for Y audience" brief**, don't jump straight to prompts. Spend one turn researching first — the cost of a single research turn is far less than 10 mis-aimed generations.
|
|
8
|
+
|
|
9
|
+
## When to do research-first
|
|
10
|
+
|
|
11
|
+
- Any URL appears in the brief (product page, landing page, brand site)
|
|
12
|
+
- The brief names a brand, product, or company you don't already have context on
|
|
13
|
+
- The brief targets a specific audience / language / market with conventions you should respect (Hebrew/Israeli, Japanese, Gen-Z TikTok, B2B SaaS, luxury, etc.)
|
|
14
|
+
- The brief explicitly says "research" / "תחקור" / "look up" / "find examples" / "check best practices"
|
|
15
|
+
|
|
16
|
+
## How to research (parallel calls in one response)
|
|
17
|
+
|
|
18
|
+
Fire these IN PARALLEL — they're independent reads:
|
|
19
|
+
|
|
20
|
+
1. **`WebSearch`** for prompt-engineering patterns specific to the chosen model. **The model name in the search query MUST be the literal model the user named** — never substitute a generic / default / "popular" model. If the user said "nano banana 2", search for `"nano banana 2" prompt …`, NOT `"flux" prompt …` or `"midjourney" prompt …`. The same HARD RULE that applies to *calling* the named model applies to *researching* it. Examples (replace `<model>` with the user's exact wording):
|
|
21
|
+
- `"<model>" prompt engineering ad image text rendering`
|
|
22
|
+
- `"<model>" hex color font specification advertising prompt`
|
|
23
|
+
- `"<model>" hebrew text RTL rendering` (or any user-named language)
|
|
24
|
+
2. **`WebSearch`** for the audience / market design conventions:
|
|
25
|
+
- `<audience> advertising design trends <year>`
|
|
26
|
+
- `<language> typography <use case> RTL/LTR best practices`
|
|
27
|
+
3. **`WebFetch`** the product URL with a precise extraction prompt (see below).
|
|
28
|
+
4. (Optional) `WebSearch` for competitor / reference visuals to set bar.
|
|
29
|
+
|
|
30
|
+
## Extracting the product page (WebFetch prompt template)
|
|
31
|
+
|
|
32
|
+
Don't ask WebFetch a vague "what is this page" — ask for structured extraction:
|
|
33
|
+
|
|
34
|
+
```
|
|
35
|
+
Extract from this page, in compact bullets:
|
|
36
|
+
1. Product name + one-line value proposition.
|
|
37
|
+
2. 3–5 concrete capabilities/benefits (user-facing language).
|
|
38
|
+
3. All product hero / screenshot image URLs visible in the page.
|
|
39
|
+
4. Brand color hex codes — pull from inline `style=`, `<style>` tags, or
|
|
40
|
+
linked CSS, ignoring generic UI defaults (#fff/#000). Identify which
|
|
41
|
+
color plays which role (primary CTA, headline text, background, accent).
|
|
42
|
+
5. Brand voice signals (tone, target user, formality).
|
|
43
|
+
6. Any explicit fonts named in CSS or visible.
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
## Re-host every external image via `upload_media`
|
|
47
|
+
|
|
48
|
+
The bulk-API rule applies: external URLs in `reference_images` / `source_images` / `image_url` cause **400 Bad Request**. Pipeline:
|
|
49
|
+
|
|
50
|
+
1. `Bash: curl -fsSL "<external-url>" -o /tmp/<name>.<ext>` (or use WebFetch where it returns the binary)
|
|
51
|
+
2. `mcp__kolbo__upload_media` with the local file → returns Kolbo CDN URL
|
|
52
|
+
3. Use the returned CDN URL in any subsequent generation call
|
|
53
|
+
4. Log both URLs in the production log (so the user can trace provenance)
|
|
54
|
+
|
|
55
|
+
## Synthesizing the research
|
|
56
|
+
|
|
57
|
+
In the production log create:
|
|
58
|
+
```md
|
|
59
|
+
### Research notes
|
|
60
|
+
- Prompt patterns for <model>: …
|
|
61
|
+
- Audience conventions: …
|
|
62
|
+
|
|
63
|
+
### Product brief
|
|
64
|
+
- Name: …
|
|
65
|
+
- Value prop: …
|
|
66
|
+
- Capabilities: …, …, …
|
|
67
|
+
|
|
68
|
+
### Brand palette
|
|
69
|
+
- primary: #...
|
|
70
|
+
- accent: #...
|
|
71
|
+
- text: #...
|
|
72
|
+
- bg: #...
|
|
73
|
+
|
|
74
|
+
### Re-hosted assets
|
|
75
|
+
- hero_1: <kolbo CDN url> (from <original url>)
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
## Persist as a Reusable Brand Kit
|
|
79
|
+
|
|
80
|
+
After research, **persist the brand-identity bits into a reusable file** at `.kolbo/brand-kits/<slug>.md` so future generations in any session can read it instead of re-scraping. This is the durable, cross-session record (the production log is per-production; brand kits are per-brand).
|
|
81
|
+
|
|
82
|
+
**Slug rule:** lowercase, single token, derived from the domain (`drinkolipop` from `drinkolipop.com`, `acme_skincare` from `acme-skincare.io`). Strip TLD and hyphens.
|
|
83
|
+
|
|
84
|
+
**Brand kit file schema** (use `Write` for first creation; `Read → Edit` for updates):
|
|
85
|
+
|
|
86
|
+
```md
|
|
87
|
+
<!-- .kolbo/brand-kits/<slug>.md — agent-managed brand identity registry.
|
|
88
|
+
Reusable across all generations for this brand. Read first before scraping. -->
|
|
89
|
+
|
|
90
|
+
# Brand Kit: <Brand Name>
|
|
91
|
+
|
|
92
|
+
url: https://...
|
|
93
|
+
fetched: 2026-05-23
|
|
94
|
+
last_refreshed: 2026-05-23
|
|
95
|
+
|
|
96
|
+
## Identity
|
|
97
|
+
brand_name: ...
|
|
98
|
+
tagline: ...
|
|
99
|
+
business_overview: <1-2 sentences>
|
|
100
|
+
industry: <fashion | beauty | food | beverage | electronics | saas | ...>
|
|
101
|
+
|
|
102
|
+
## Visuals
|
|
103
|
+
logo_url: <kolbo CDN url, re-hosted via upload_media>
|
|
104
|
+
primary_color: #...
|
|
105
|
+
accent_color: #...
|
|
106
|
+
text_color: #...
|
|
107
|
+
bg_color: #...
|
|
108
|
+
fonts:
|
|
109
|
+
headline: <font name>
|
|
110
|
+
body: <font name>
|
|
111
|
+
mono: <font name, optional>
|
|
112
|
+
|
|
113
|
+
## Voice & Audience
|
|
114
|
+
tone: <playful | refined | technical | bold | warm | ...>
|
|
115
|
+
target_user: <one-line persona>
|
|
116
|
+
formality: <casual | professional | luxury>
|
|
117
|
+
|
|
118
|
+
## Hero Assets (re-hosted via upload_media)
|
|
119
|
+
- hero_1: <kolbo CDN url> (from <original url>)
|
|
120
|
+
- hero_2: <kolbo CDN url> (from <original url>)
|
|
121
|
+
- product_1: <kolbo CDN url> (from <original url>)
|
|
122
|
+
|
|
123
|
+
## Notes
|
|
124
|
+
- Any brand-specific gotchas, design rules, do-not-show items
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
### When to READ a brand kit
|
|
128
|
+
|
|
129
|
+
Before any generation tied to a known brand — **always** Read first:
|
|
130
|
+
|
|
131
|
+
- User mentions a brand by name ("make a Pinterest pin for OliPop")
|
|
132
|
+
- User pastes a brand URL again ("make ads for drinkolipop.com")
|
|
133
|
+
- User says "use the same brand as last time" / "match our brand"
|
|
134
|
+
- Any DTC ad / product photoshoot / marketplace card request where the brand is implicit
|
|
135
|
+
|
|
136
|
+
If `.kolbo/brand-kits/<slug>.md` exists, **skip the research-first workflow entirely** for the brand-extraction parts (palette, logo, fonts, voice). Still do `WebSearch` for fresh prompt-engineering patterns or audience trends if needed.
|
|
137
|
+
|
|
138
|
+
If it doesn't exist and there's a URL, run the full research workflow above and **end by persisting the brand kit**.
|
|
139
|
+
|
|
140
|
+
### When to UPDATE a brand kit
|
|
141
|
+
|
|
142
|
+
- User explicitly says "the brand updated their colors / logo / fonts"
|
|
143
|
+
- Brand kit is >90 days old AND the user is starting a major campaign (refresh recommended, not mandatory)
|
|
144
|
+
- Generation results look "off-brand" and palette is suspect — refresh to verify
|
|
145
|
+
|
|
146
|
+
Update by `Read → Edit`. Bump `last_refreshed`. Keep older asset URLs in place (they still work) and append new ones.
|
|
147
|
+
|
|
148
|
+
### Brand-kit reuse downstream
|
|
149
|
+
|
|
150
|
+
The other workflow files consume the brand kit:
|
|
151
|
+
|
|
152
|
+
- `workflows/dtc-ads.md` — pulls palette + fonts + logo into every ad prompt
|
|
153
|
+
- `workflows/product-photoshoot.md` — bakes hex codes into prompts; uses logo as `reference_images[0]`
|
|
154
|
+
- `workflows/marketplace-cards.md` — palette + fonts critical for A+ module consistency
|
|
155
|
+
- `workflows/marketing-studio.md` — voice/tone shapes UGC presenter dialogue; palette shapes any branded overlays
|
|
156
|
+
|
|
157
|
+
Always cite the brand-kit slug in the production log so reviewers can trace which kit drove a given generation.
|
|
158
|
+
|
|
159
|
+
## Building prompts informed by the research
|
|
160
|
+
|
|
161
|
+
When generating ad / marketing creative based on this research:
|
|
162
|
+
|
|
163
|
+
- **Exact hex codes for every color** — `#FF4D2E` not "orange". Match brand palette.
|
|
164
|
+
- **On-image text in literal double quotes** — `"שלום עולם"` not `Hebrew greeting`. Specify language and direction (RTL/LTR) when non-English.
|
|
165
|
+
- **Per text element**: position, font weight, point size, color hex, alignment.
|
|
166
|
+
- **Forbid uninvited additions** — explicitly tell the model: NO captions, NO subtitles, NO watermarks, NO extra text beyond what's specified. Same rule as UGC defaults.
|
|
167
|
+
- **Use research findings to shape composition** — e.g. if research said "Israeli social ads favor bold contrast and minimal copy", reflect that.
|
|
168
|
+
- Always **approve the concept + sample prompts with the user** before firing the full batch when the batch is ≥4 ads or the user said "approve first".
|
|
169
|
+
|
|
170
|
+
## Skipping research is OK when…
|
|
171
|
+
|
|
172
|
+
- User gave no URL, no brand, no audience-specific signal — pure creative ("make a sunset")
|
|
173
|
+
- User said "skip research" / "just generate" / "I have the prompt ready"
|
|
174
|
+
- The brief is for a single quick draft
|
|
@@ -0,0 +1,163 @@
|
|
|
1
|
+
# Transcription & Video/Audio Analysis
|
|
2
|
+
|
|
3
|
+
Load this file when the user wants to transcribe audio/video, get SRT subtitles, or analyze video/audio content. For image analysis, use native vision (no tool needed) — see "Image analysis" at bottom.
|
|
4
|
+
|
|
5
|
+
## Decision Tree
|
|
6
|
+
|
|
7
|
+
You have three routes. The right one depends on the file profile — pick before calling any tool.
|
|
8
|
+
|
|
9
|
+
```
|
|
10
|
+
Image (jpg/png/webp)? → Read directly (native vision, up to 10 per pass)
|
|
11
|
+
File >100MB OR >15 min OR dialogue-dense? → HYBRID (transcribe + ffmpeg frames + Read + your synthesis)
|
|
12
|
+
User wants the transcript/SRT as deliverable? → transcribe_audio, return the URLs
|
|
13
|
+
Precise answer about one specific frame? → ffmpeg that frame → Read
|
|
14
|
+
Otherwise (short/medium video, mixed content) → upload_media → chat_send_message (Gemini native)
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
## Why `upload_media` → chat is **not** always the default
|
|
18
|
+
|
|
19
|
+
Gemini-via-chat processes frames + motion + audio in one pass and is the simplest route when it works. But it has three known failure surfaces — recognize them and pivot to the hybrid path:
|
|
20
|
+
|
|
21
|
+
1. **>100MB upload cap.** Hard limit; the upload won't succeed. No option but to split with ffmpeg or go hybrid.
|
|
22
|
+
2. **Long-form decay** (rough threshold: 15–20 min). Even when it fits, attention degrades — shallow or hallucinated answers on the back half of the file.
|
|
23
|
+
3. **Transcription-dense laziness.** Lectures, interviews, podcasts, anything where speech is the substance: chat models summarize aggressively, paraphrase quotes wrong, or silently skip stretches. Always transcribe these first to get the actual words, then add visuals only if they matter.
|
|
24
|
+
|
|
25
|
+
## The hybrid path (workaround for all three failures)
|
|
26
|
+
|
|
27
|
+
```
|
|
28
|
+
1. transcribe_audio({ source }) → text, srt_url, word_by_word_srt_url, duration
|
|
29
|
+
2. Read the transcript text from the tool output directly
|
|
30
|
+
3. Pick 3–8 timestamps from the SRT where visuals actually matter
|
|
31
|
+
4. ffmpeg -ss <ts> -i <file> -frames:v 1 <frame.jpg> (one extract per timestamp)
|
|
32
|
+
5. Read each frame with native vision (up to ~10 frames per analysis pass)
|
|
33
|
+
6. Synthesize from transcript + frames + the user's question
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
This is usually **cheaper** than chat for long files — transcription is per-minute, ffmpeg + Read are free — and produces stronger answers on dialogue-heavy material because you have the complete text, not a model's summary of it.
|
|
37
|
+
|
|
38
|
+
For media >30 min (past the transcription cap), split with ffmpeg into ~25-min chunks, transcribe each, concatenate.
|
|
39
|
+
|
|
40
|
+
## Transcribe-as-deliverable vs transcribe-as-input
|
|
41
|
+
|
|
42
|
+
| Request pattern | Action |
|
|
43
|
+
|---|---|
|
|
44
|
+
| "Transcribe this" / "give me an SRT" / "I need word-by-word timing" / "make subtitles" | Run `transcribe_audio`, return the URL(s). The transcript IS the deliverable. |
|
|
45
|
+
| "What did they say about X?" / "Summarize this meeting" / "Find the part where they mention Y" | Run `transcribe_audio` to *get* the text → **you** read/summarize/search. Transcript is a means, not the answer. |
|
|
46
|
+
|
|
47
|
+
## `transcribe_audio` — tool details
|
|
48
|
+
|
|
49
|
+
- `source`: URL or absolute local path.
|
|
50
|
+
- **Audio**: mp3, wav, m4a, flac, aac. **Video** (audio track extracted): mp4, mov, webm, mkv, avi, m4v.
|
|
51
|
+
- **30-minute hard cap.** Longer → split with ffmpeg first.
|
|
52
|
+
- Returns:
|
|
53
|
+
- `text` — full transcript, plain.
|
|
54
|
+
- `srt_url` — grouped SRT (~12 words per line, up to 2 lines per subtitle). Use this for normal subtitle delivery.
|
|
55
|
+
- `word_by_word_srt_url` — one word per cue with millisecond-precise start/end (ElevenLabs Scribe v2). Use **only** when downstream is animation (Remotion captions, after-effects karaoke, precise speech-aligned cuts). Noise for normal subtitle workflows.
|
|
56
|
+
- `txt_url` — plain text file.
|
|
57
|
+
- `duration` — seconds.
|
|
58
|
+
- Cost: per-minute (`model.credit × duration_minutes`). Run `check_credits` before transcribing very long files.
|
|
59
|
+
- Read-only / discovery — does NOT trigger the `.kolbo/production.md` log nudge. If the user wants the transcript saved as a durable artifact, `Write` it to a workspace file, not the production log.
|
|
60
|
+
|
|
61
|
+
## `upload_media` → `chat_send_message` — tool details
|
|
62
|
+
|
|
63
|
+
- `upload_media({ source: "/absolute/local/path/file.mp4" })` → returns `{ url, thumbnail_url, ... }`. **Use `url`** (the CDN URL); ignore `thumbnail_url` (preview JPG only).
|
|
64
|
+
- `chat_send_message({ message, media_urls: [url] })`:
|
|
65
|
+
- `media_urls` is **mandatory** — the model only sees the file if you pass the CDN URL here. Always an array.
|
|
66
|
+
- **Omit `model`** — Smart Select auto-routes to Gemini when media is detected.
|
|
67
|
+
- Sessions do NOT remember media between messages. On retry: reuse the same CDN URL (no re-upload), but always pass `media_urls` again.
|
|
68
|
+
- Batch / many short videos cost-sensitively: `list_models` for the cheapest Gemini, pass it explicitly.
|
|
69
|
+
|
|
70
|
+
## Image analysis — never via chat
|
|
71
|
+
|
|
72
|
+
You have native vision. **Always `Read` images directly** (you handle up to 10 per pass). Do not `upload_media` + chat for images unless the user explicitly names a specific Kolbo chat model. Don't extract frames from images either — they're already viewable.
|
|
73
|
+
|
|
74
|
+
**NEVER ask the user which path to use — diagnose from the file profile and pick.**
|
|
75
|
+
|
|
76
|
+
## ⚠️ Batching Media in Chat Messages (CRITICAL)
|
|
77
|
+
|
|
78
|
+
**Send ALL media in ONE `chat_send_message` call.** `media_urls` accepts up to **10 URLs**. Each separate chat call counts toward rate limits — splitting trips "Too many generation requests."
|
|
79
|
+
|
|
80
|
+
```
|
|
81
|
+
# Step 1: parallel uploads (one response)
|
|
82
|
+
upload_media({ source: "video1.mp4" }) → url1
|
|
83
|
+
... (up to 10)
|
|
84
|
+
|
|
85
|
+
# Step 2: ONE chat call with all URLs
|
|
86
|
+
chat_send_message({ message: "Analyze all 5 videos...", media_urls: [url1, url2, ...] })
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
On 429: wait 60s, retry the same chat call — reuse the CDN URLs, do not re-upload.
|
|
90
|
+
|
|
91
|
+
**Never:** pass a local path in `media_urls` (CDN URLs only); use a transcription `.txt` URL as a video URL; construct a CDN URL yourself; split media across multiple chat calls.
|
|
92
|
+
|
|
93
|
+
## Analyzing the source before a chained generation — when it's worth it
|
|
94
|
+
|
|
95
|
+
Before feeding a media asset into another generation tool (`generate_image_edit`, `edit_image`, `generate_video_from_image`, `generate_first_last_frame`, `generate_video_from_video`, `edit_video`, `generate_elements`, `generate_lipsync`), think about whether you actually *know* what's in the source. If you don't, analyze it first so the next prompt can reference concrete details instead of generic adjectives.
|
|
96
|
+
|
|
97
|
+
**Analyze first when:**
|
|
98
|
+
|
|
99
|
+
- The source is **old** — more than a few turns back, or pulled via `list_media` / `get_media` from earlier in the project. Context has drifted; you likely don't remember the specifics.
|
|
100
|
+
- The source was **user-provided without a description** — they pasted a URL or uploaded a file but didn't say what it shows.
|
|
101
|
+
- The previous prompt was **vague** ("make something pretty", "a cool shot") — the output details matter and you don't know them.
|
|
102
|
+
- The chain step needs to **preserve specific details** the original prompt didn't pin down (exact pose, color of a prop, lighting direction, audio room tone, etc.).
|
|
103
|
+
- Source is a **video or audio** going into elements / video-from-video / lipsync — motion direction, pacing, voice characteristics, and ambient bed drive the next prompt and can't be guessed from a URL.
|
|
104
|
+
|
|
105
|
+
**Skip analysis when:**
|
|
106
|
+
|
|
107
|
+
- You **just generated** the asset in the same conversation with a precise prompt — that prompt *is* the spec. Re-analyzing wastes credits.
|
|
108
|
+
- The edit is **mechanical** — "remove background", "brighten 10%", "loop to 5 seconds", "crop to 1:1". The source content doesn't matter.
|
|
109
|
+
- The user already **described what's in it** in this turn.
|
|
110
|
+
|
|
111
|
+
Default to skipping unless one of the "analyze first" cases applies — an analysis-per-step habit on long chains burns credits and latency without adding signal.
|
|
112
|
+
|
|
113
|
+
**How to analyze (pick by media type):**
|
|
114
|
+
|
|
115
|
+
| Source media | How |
|
|
116
|
+
|---|---|
|
|
117
|
+
| Image (URL or local) | Your native vision — view it directly. No `chat_send_message` round-trip needed. |
|
|
118
|
+
| Video / Audio | `chat_send_message({ message: "Describe...", media_urls: [url] })`. Batch up to 10 URLs in **one** call (see batching rule above). Omit `model` so Smart Select routes to Gemini vision. |
|
|
119
|
+
|
|
120
|
+
**What the analysis should extract** (use whatever is relevant for the next step's prompt):
|
|
121
|
+
|
|
122
|
+
- **Subject** — pose, expression, framing (head-and-shoulders / full body / wide).
|
|
123
|
+
- **Wardrobe & props** — exact colors, materials, distinguishing items.
|
|
124
|
+
- **Scene & environment** — location, time of day, weather, background depth.
|
|
125
|
+
- **Lighting & color palette** — dominant temperature, key/fill direction, contrast, color grade.
|
|
126
|
+
- **Camera** — angle, focal length feel (wide / portrait), depth-of-field.
|
|
127
|
+
- **Motion** (videos only) — direction, speed, camera move (push-in, pan, locked), what changes between first and last frame.
|
|
128
|
+
- **Audio** (videos/audio only) — voice characteristics, ambient bed, speech pace, music tempo/mood.
|
|
129
|
+
- **Anything that already looks wrong** — artifacts, blurred faces, wrong fingers, blown highlights, audio glitches — note these so the next prompt either fixes them (edit) or doesn't preserve them (elements/video).
|
|
130
|
+
|
|
131
|
+
**Then write the next prompt with concrete references**, not generic adjectives. Example for an image-to-video chain:
|
|
132
|
+
|
|
133
|
+
Bad — generic, no analysis:
|
|
134
|
+
```
|
|
135
|
+
prompt: "Animate this image with a slow push-in"
|
|
136
|
+
image_url: <generated still>
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
Good — analyzed first, prompt names the specifics:
|
|
140
|
+
```
|
|
141
|
+
prompt: "Slow 4-second dolly-in toward @maya's face from the medium shot;
|
|
142
|
+
the warm golden-hour rim light on her left shoulder stays
|
|
143
|
+
consistent; the wind moves the leaves behind her gently to the
|
|
144
|
+
right. Camera locked, no shake. Subject does not turn — she keeps
|
|
145
|
+
the half-smile and direct eye contact from the still."
|
|
146
|
+
image_url: <generated still>
|
|
147
|
+
visual_dna_ids: ["vdna_8f2c"] // maya
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
The point is **not** to dump an essay into the prompt — it's to make sure every concrete detail the next model needs to preserve (or change) is named, so the chain doesn't lose continuity across steps.
|
|
151
|
+
|
|
152
|
+
**Production-log tie-in:** when you analyze a generated still/clip, write a one-line description into `.kolbo/production.md` next to the URL — that way the next chained step can read the log instead of re-analyzing.
|
|
153
|
+
|
|
154
|
+
## Image Analysis Detail (when the user uploads images)
|
|
155
|
+
|
|
156
|
+
When the user shares an image and asks about it:
|
|
157
|
+
|
|
158
|
+
- **Analyze thoroughly**: describe composition, subjects, colors, lighting, style, text/signage, setting, mood, visible objects, and any embedded information (charts, diagrams, screenshots).
|
|
159
|
+
- **Reference specific regions** when helpful: "top-left corner", "in the foreground", "the figure on the right".
|
|
160
|
+
- **Extract text verbatim** when asked (OCR-style requests are fine).
|
|
161
|
+
- **Cannot identify real people.** Describe hair, clothing, pose, expression, and apparent role — but never name a specific individual, even a well-known public figure. If the user insists, decline and offer to describe instead.
|
|
162
|
+
- **Copyrighted content**: summarize and reference, don't reproduce verbatim large chunks.
|
|
163
|
+
- If the user wants an **edit** based on the analysis, hand off to `generate_image_edit` (visual edit) or `generate_video_from_image` (motion).
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
# Troubleshooting
|
|
2
|
+
|
|
3
|
+
Load this file when the user hits an auth error, MCP tools aren't responding, or they're being rate-limited.
|
|
4
|
+
|
|
5
|
+
## "API key is invalid or expired"
|
|
6
|
+
|
|
7
|
+
This usually means the CLI is sending a key to the wrong API endpoint.
|
|
8
|
+
|
|
9
|
+
**Common cause — whitelabel overlap:** if the user previously used regular `kolbo` and then switched to a whitelabel/partner CLI (e.g. `sapir`), the old API key may still be cached against the main Kolbo API. Running `kolbo` instead of the branded command (`sapir`) overwrites the MCP config with the wrong endpoint.
|
|
10
|
+
|
|
11
|
+
**Fix:** tell the user to re-authenticate with their branded CLI command:
|
|
12
|
+
```
|
|
13
|
+
sapir auth login
|
|
14
|
+
```
|
|
15
|
+
(Replace `sapir` with their actual CLI command.)
|
|
16
|
+
|
|
17
|
+
Then **restart the editor/session** so the MCP picks up the new key and endpoint.
|
|
18
|
+
|
|
19
|
+
**Important:** whitelabel users must always use their branded CLI command (e.g. `sapir`), not `kolbo`, to keep the MCP pointed at the correct API.
|
|
20
|
+
|
|
21
|
+
## MCP tools not responding or not found
|
|
22
|
+
|
|
23
|
+
If Kolbo tools timeout or aren't listed, the MCP server may not be wired. Tell the user to run:
|
|
24
|
+
```
|
|
25
|
+
<their-cli-command> auth login
|
|
26
|
+
```
|
|
27
|
+
This re-wires the MCP configuration automatically. Then restart the session.
|
|
28
|
+
|
|
29
|
+
## "Rate limited" (429 errors)
|
|
30
|
+
|
|
31
|
+
Wait 60s for the window to reset, retry only the failed calls. For batch image work prefer `generate_creative_director` over multiple `generate_image` calls. Full rate-limit details + retry sequence: see SKILL.md "Rate Limiting & Batch Generation".
|
|
32
|
+
|
|
33
|
+
## Failure envelope from `get_generation_status`
|
|
34
|
+
|
|
35
|
+
When a generation fails, `get_generation_status` returns a structured `failure` field alongside `error`:
|
|
36
|
+
|
|
37
|
+
```json
|
|
38
|
+
{
|
|
39
|
+
"state": "failed",
|
|
40
|
+
"error": "The input or output was flagged as sensitive…",
|
|
41
|
+
"failure": {
|
|
42
|
+
"message": "The input or output was flagged as sensitive…",
|
|
43
|
+
"category": "content_policy",
|
|
44
|
+
"code": "CONTENT_FLAGGED_SENSITIVE",
|
|
45
|
+
"retryable": false,
|
|
46
|
+
"severity": "error",
|
|
47
|
+
"provider": "kie-nano-banana"
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
Branch on `failure.category` / `failure.retryable`:
|
|
53
|
+
|
|
54
|
+
- `category === "content_policy"` (or `code === "CONTENT_FLAGGED_SENSITIVE"`) → **do not retry the same prompt**. Tell the user the model refused, suggest a less explicit phrasing or a Visual DNA fallback. Log to `.kolbo/production.md` Failures section with the exact reason.
|
|
55
|
+
- `category === "auth"` or `code === "[KOLBO_AUTH_EXPIRED]"` → surface the reconnect flow, don't auto-retry.
|
|
56
|
+
- `retryable === true` (transient: network, rate limit, provider 5xx) → retry once with the same payload after a short pause. If it fails again, surface to user.
|
|
57
|
+
- `retryable === false` and unknown category → surface the raw `message` to the user, don't retry.
|
|
58
|
+
|
|
59
|
+
## Kolbo Code Documentation
|
|
60
|
+
|
|
61
|
+
Full public documentation for Kolbo Code (the CLI you are running inside) lives at **[docs.kolbo.ai/docs/kolbo-code](https://docs.kolbo.ai/docs/kolbo-code)**. If the user asks about installation, authentication, voice input, supported languages, commands, or how to uninstall, point them to the matching page below rather than guessing:
|
|
62
|
+
|
|
63
|
+
| Topic | Path |
|
|
64
|
+
|-------|------|
|
|
65
|
+
| Overview & quick links | `/docs/kolbo-code` |
|
|
66
|
+
| Installation (npm / bun / brew / scoop / choco) | `/docs/kolbo-code/installation` |
|
|
67
|
+
| Sign in with Kolbo (device-code OAuth) | `/docs/kolbo-code/authentication` |
|
|
68
|
+
| Push-to-talk voice input (hold `space`) | `/docs/kolbo-code/voice-input` |
|
|
69
|
+
| 12 supported UI languages + RTL | `/docs/kolbo-code/languages` |
|
|
70
|
+
| Full CLI command reference | `/docs/kolbo-code/commands` |
|
|
71
|
+
| Uninstall + cleanup | `/docs/kolbo-code/uninstall` |
|
|
72
|
+
|
|
73
|
+
The MDX sources are in the `kolbo-docs` repo under `content/docs/kolbo-code/`. When the user's question has a concrete answer in one of those pages, cite the path and summarize — do not invent new instructions.
|