@inneranimalmedia/agentsam-sdk 2.6.4 → 2.6.7

package/skills/grok-video2dsprite/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 0x0funky
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/skills/grok-video2dsprite/SKILL.md

@@ -0,0 +1,236 @@
+---
+name: video2dsprite
+description: >
+  Grok Build only. Turn a 2D character still into denser animation sprites via
+  imagine_text_to_image base → imagine_image_to_video (6s/10s run-in-place) → ffmpeg
+  frames → magenta chroma-key → dense sampled strips/grids/GIFs. Use when the
+  user wants video-to-sprite, smoother run/walk cycles, or denser intermediate
+  poses. Prefer generate2dsprite for crisp production pixel sheets. Triggers on
+  "video to sprite", "imagine_image_to_video sprite", "dense walk cycle", "smooth run
+  animation from video".
+metadata:
+  short-description: "Video→dense sprites (imagine_image_to_video + chroma postprocess)"
+user-invocable: false
+---
+
+# Video2dsprite (Grok Build only)
+
+Convert a **base 2D character image** into **dense animation sprites** using Grok Build's native video tools.
+
+## App-builder / Grok environment
+
+| Item | Value |
+| --- | --- |
+| Skill dir | `.grok/skills/video2dsprite/` |
+| Scripts | `python3 .grok/skills/video2dsprite/scripts/video2dsprite.py …` |
+| Video tools | `imagine_image_to_video` — animate one base `file_path` (verify present) |
+| Inspect | `read_file` on stills/frames; report paths for videos |
+| Deps | ffmpeg + Pillow + numpy (preinstalled in app-builder image) |
+| Output home | `assets/sprites/video2dsprite/<name>/` under `/workspace` |
+| Default | Prefer **`generate2dsprite`** for production heroes; this is the denser-motion path |
+
+```text
+base still → imagine_image_to_video (in-place motion) → extract frames → chroma key → sample/normalize → strip / grid / GIF
+```
+
+## Platform gate (read first)
+
+| Runtime | Supported? |
+| --- | --- |
+| **Grok Build** (xAI) | **Yes** — requires an image generator + an image→video tool |
+| Codex / Claude / other agents | **No** — they lack Grok video tools. Tell the user this skill is Grok Build only and offer `$generate2dsprite` instead |
+
+**Gate on the capability, not the exact tool name.** Image generation appears
+as `imagine_text_to_image` / `imagine_image_to_image`; video as `imagine_image_to_video`
+or (legacy) `generate_video`. Use whichever pair your tool list has. Stop and
+explain only when you have no image→video tool at all. Do not fake motion with
+code-drawn frames.
+
+This skill is an **optional denser-motion path**. It does **not** replace `$generate2dsprite`:
+
+| Use `$generate2dsprite` when… | Use `$video2dsprite` when… |
+| --- | --- |
+| Crisp pixel sheets, fixed grids, identity-critical heroes | User wants denser intermediate poses / smoother feeling loops |
+| Attack/cast body sheets, prop packs, engine atlases | Experimenting with video-sourced run/walk/idle motion |
+| Production default for most game sprites | User explicitly asks for video → frames → sprites |
+
+Video softens pixels, drifts identity, and leaves chroma fringes. Always QC; for production heroes, prefer `$generate2dsprite` unless the user wants the video look.
+
+## Parameters
+
+Infer from the user request:
+
+- `subject`: character / creature description, or path to existing still
+- `action`: `run` | `walk` | `idle` | `attack` | custom motion phrase
+- `view`: usually `side` (side-scroller). `topdown` is harder — warn and keep camera locked
+- `duration`: `6` (default) or `10` seconds
+- `frame_counts`: which denser sets to export, default `8,16,24,48`
+- `cell_size`: output sprite cell, default `128`
+- `anchor`: `feet` (default for side locomotion) | `center`
+- `bg`: solid `#FF00FF` (required for chroma)
+- `name`: output slug
+- `out_dir`: working folder (default `./sprites/video2dsprite/<name>/` or project-relative)
+
+## Agent rules
+
+1. **Grok-only.** Refuse on non-Grok runtimes with a short explanation + `$generate2dsprite` alternative.
+2. **Still → video, never text-to-video alone.** Stage frame 1 as a clean still with `imagine_text_to_image` (from a prompt, or from a reference `file_path`). Then call `imagine_image_to_video` with that still's `file_path`.
+3. **In-place motion.** Prompt for run/walk **in place** facing a fixed direction. No camera pan, no background scroll, no scene change. Subject stays roughly centered.
+4. **Solid magenta background** on the base and preserved in the video prompt (`#FF00FF` / pure magenta). Required for flood-fill chroma.
+5. **Do not invent art with PIL/Canvas.** Base art comes from `imagine_text_to_image` or a user/local still. Scripts only postprocess.
+6. **Do not put experimental outputs into the game** unless the user asks to integrate.
+7. **Prefer one locomotion cycle for game use.** Dense sample across a full 6s multi-cycle clip is fine for previews; for engine sheets, optionally re-sample a single cycle (12–16 frames) after visual QC.
+8. **Report absolute paths** of video, cleaned frames, strips, and preview GIFs when done.
+
+## Workflow
+
+### 1. Plan
+
+Pick the smallest useful run:
+
+- Side-view run/walk loop → this skill
+- Multi-action hero kit → still use `$generate2dsprite` per action; only use video for locomotion if requested
+- FX / projectile / prop packs → `$generate2dsprite`, not video
+
+Create:
+
+```text
+<out_dir>/
+  base/
+  video/
+  frames-raw/
+  frames-clean/
+  sprite/          # default 8-frame set + denser x16/x24/x48
+  prompt-used.txt
+  pipeline-meta.json
+  README.txt
+```
+
+### 2. Build the base still
+
+Options:
+
+- **A. Existing sprite:** read the image, composite onto solid `#FF00FF` if needed, then pass that sandbox path to `imagine_image_to_image` / `imagine_image_to_video`
+- **B. New character:** `imagine_text_to_image` with solid magenta background, full body, side view, centered
+- **C. Match reference:** `imagine_image_to_image` with the reference `file_path`, moving it onto magenta and preserving identity
+
+Base requirements:
+
+- Full body visible, generous magenta margin
+- Side view for run/walk (profile or 3/4 side), feet near bottom third
+- Same art style as the rest of the project when a reference exists
+- No text, UI, watermark, or second character
+
+Copy the returned `file_path` and save as
+`<out_dir>/base/<name>-base.png`; keep the base `file_path` for the video call.
+
+Write the exact image prompt into `prompt-used.txt`.
+
+### 3. Animate with `imagine_image_to_video`
+
+Call Grok `imagine_image_to_video`:
+
+- source path: `<base still file_path>` (sandbox path returned by T2I / I2I)
+- `duration`: `6` (default) or `10`
+- `resolution_name`: `480p` unless user asks `720p`
+- `prompt`: one short present-tense shot (see [references/prompt-rules.md](references/prompt-rules.md))
+
+Mandatory motion constraints in the prompt:
+
+- Subject runs/walks **in place** (treadmill style)
+- Camera **locked** — no pan, zoom, or orbit
+- Background stays **flat solid magenta**
+- Identity, costume, palette stable for the whole shot
+- Single continuous action only
+
+Copy the returned video `file_path` to
+`<out_dir>/video/<name>-<duration>s.mp4`.
+
+If video tools are unavailable, stop (platform gate).
+
+### 4. Extract + chroma + sample (local script)
+
+Run the processor (ffmpeg + Pillow + numpy):
+
+```bash
+python3 .grok/skills/video2dsprite/scripts/video2dsprite.py process \
+  --video <out_dir>/video/<name>-6s.mp4 \
+  --out-dir <out_dir> \
+  --name <name> \
+  --frame-counts 8,16,24,48 \
+  --cell-size 128 \
+  --body-height 100 \
+  --foot-y 118 \
+  --fps 0
+```
+
+Notes:
+
+- `--fps 0` = extract every decoded frame (use source fps)
+- Magenta flood-fill from corners + despill
+- Even sampling for each count in `--frame-counts`
+- Feet-normalized cells, horizontal strip, grid, loop GIF per count
+
+Optional: only re-sample denser sets from existing cleaned frames:
+
+```bash
+python3 .grok/skills/video2dsprite/scripts/video2dsprite.py sample \
+  --clean-dir <out_dir>/frames-clean \
+  --out-dir <out_dir> \
+  --frame-counts 16,24,48 \
+  --cell-size 128
+```
+
+### 5. QC
+
+Visually check:
+
+- [ ] Preview GIF loops without huge pops
+- [ ] Magenta gone (no solid pink blocks); fringe acceptable or re-key
+- [ ] Feet stay on a stable baseline (no hop from bad crop)
+- [ ] Identity roughly stable (face/clothes not morphing every frame)
+- [ ] Action is in-place (not sliding out of frame)
+- [ ] For game use: pick one count (often **16 or 24**) or cut one true cycle
+
+If identity drifts hard or pixels are too soft, fall back to `$generate2dsprite` for production sheets and keep the video set as motion reference only.
+
+### 6. Deliver
+
+Report paths only (unless user asked to wire into a game):
+
+- Video: `video/*.mp4`
+- Dense sprites: `sprite/x16|x24|x48/`
+- Strips / grids / GIFs: `sprite/run-strip-N.png`, `run-grid-N.png`, `run-preview-N.gif`
+- Meta: `pipeline-meta.json`
+
+Do **not** modify game code unless requested.
+
+## Defaults
+
+- Duration: **6s**
+- Action: **side run in place**, facing right
+- Export counts: **8, 16, 24, 48**
+- Cell: **128²**, body height ~100, feet at y≈118
+- Background: **#FF00FF**
+- Prefer single-asset `imagine_image_to_video` over multi-ref (compose multi-ref with `imagine_text_to_image` first if needed)
+
+## Tradeoffs (tell the user once)
+
+**Pros:** denser intermediates → often feels smoother than 4–8 discrete gen poses.  
+**Cons:** softer pixels, identity drift, chroma fringe, multi-cycle 6s clips are not a single perfect loop, heavier assets.  
+**Rule of thumb:** 8→16→24 usually gains smoothness; 48 is often diminishing returns; 145 raw frames are for sampling, not all for runtime.
+
+## Resources
+
+- [references/prompt-rules.md](references/prompt-rules.md) — base still + video prompts
+- [references/pipeline.md](references/pipeline.md) — folder layout, ffmpeg, sampling strategy
+- [scripts/video2dsprite.py](scripts/video2dsprite.py) — extract, chroma, normalize, export
+
+## Relationship to other skills
+
+- `$generate2dsprite` — primary sheet pipeline (Codex + Grok when image gen exists)
+- `$generate2dmap` — maps; not used here
+- `$video2dsprite` — **Grok Build exclusive** motion densification path
+- `$game-asset-core` / `$game-animation-frames` — loop/flip-test/motion laws and
+  engine-ready defaults; still use this skill’s scripts for sandbox execution
+  (magenta base + chroma), not a freeform background color

package/skills/grok-video2dsprite/SOURCE.md

@@ -0,0 +1,6 @@
+# Source
+
+Vendored from [agent-sprite-forge](https://github.com/0x0funky/agent-sprite-forge) (`53dce6055984c610d833e77887939cbd0fb1c92b`).
+MIT License — see `LICENSE`.
+
+Adapted for Grok Build / app-builder sandbox (image tool paths, `read_file` for image inspection, workspace-relative script paths).

package/skills/grok-video2dsprite/agentsam.skill.json

@@ -0,0 +1,25 @@
+{
+  "schema": "agentsam.skill.v1",
+  "id": "grok-video2dsprite",
+  "name": "Grok \u00b7 video2dsprite",
+  "description": "Video\u2192dense sprites (imagine_image_to_video + chroma postprocess)",
+  "icon": "skill",
+  "slash": {
+    "suggested": "/grok-video2dsprite"
+  },
+  "instructions": {
+    "source": "local_file",
+    "ref": "SKILL.md"
+  },
+  "execution": {
+    "mode": "turn",
+    "tools": [],
+    "operations": [],
+    "capabilities": []
+  },
+  "provenance": {
+    "source": "apps/local-studio/.grok/skills",
+    "vendor": "grok",
+    "status": "imported"
+  }
+}

package/skills/grok-video2dsprite/references/pipeline.md

@@ -0,0 +1,90 @@
+# Video2dsprite pipeline
+
+## End-to-end
+
+```text
+1. base still          imagine_text_to_image / existing PNG on #FF00FF
+2. video               imagine_image_to_video  (6s default, 10s optional)
+3. frames-raw          ffmpeg decode all frames (or fixed fps)
+4. frames-clean        magenta flood-fill + light despill → RGBA
+5. sample              even indices for N in {8,16,24,48}
+6. normalize           crop alpha bbox → scale body height → feet line
+7. export              sprite_XX.png, strip, grid, preview GIF
+8. meta                pipeline-meta.json + README.txt
+```
+
+## Suggested folder layout
+
+```text
+<out_dir>/
+  base/<name>-base.png
+  video/<name>-6s.mp4
+  frames-raw/frame_0001.png ...
+  frames-clean/clean_0000.png ...
+  sprite/
+    sprite_01.png ...          # default small set if requested
+    x16/sprite_01.png ...
+    x24/...
+    x48/...
+    run-strip-8.png
+    run-strip-16.png
+    run-preview-24.gif
+    ...
+  prompt-used.txt
+  pipeline-meta.json
+  README.txt
+```
+
+## ffmpeg extract
+
+Processor shells out to `ffmpeg` when available:
+
+```bash
+ffmpeg -y -i video.mp4 -vsync 0 frames-raw/frame_%04d.png
+```
+
+If ffmpeg is missing, fail with a clear install message. Do not invent frames.
+
+## Chroma key
+
+1. Treat near-magenta pixels as key candidates (hue distance + high magenta channel).
+2. Flood-fill from image corners so interior magenta-ish costume bits are less likely to vanish.
+3. Despill residual pink fringes toward neutral/transparent.
+4. Write RGBA PNGs.
+
+## Sampling
+
+Even spacing including first and last:
+
+```text
+idx[i] = round(i * (total - 1) / (want - 1))  for i in 0..want-1
+```
+
+Export multiple `want` values in one run so the user can compare smoothness vs softness.
+
+## Normalize (feet anchor)
+
+1. Alpha bbox of cleaned frame
+2. Scale so content height ≈ `body_height` (default 100 in a 128 cell)
+3. Paste so bottom of content sits at `foot_y` (default 118)
+4. Center horizontally
+
+For `center` anchor, place bbox center at cell center instead.
+
+## GIF duration defaults
+
+| Frames | ms / frame (approx) |
+| --- | --- |
+| 8 | 80 |
+| 16 | 60 |
+| 24 | 40 |
+| 48 | 25 |
+
+Goal: roughly 0.6–1.2s visual loop for previews (not necessarily matching source video realtime).
+
+## When not to use this pipeline
+
+- Need hard pixel edges and fixed multi-row grids → `$generate2dsprite`
+- Map props / tilesets → `$generate2dmap` + `$generate2dsprite`
+- Non-Grok agent without `imagine_image_to_video`
+- User wants production-perfect hero kit with many actions — video path is locomotion experiment first

package/skills/grok-video2dsprite/references/prompt-rules.md

@@ -0,0 +1,63 @@
+# Video2dsprite prompt rules
+
+## Base still (`imagine_text_to_image`)
+
+Required:
+
+- Solid flat background `#FF00FF` (pure magenta), no gradient, no floor shadow if possible
+- Full body, centered, generous margin on all sides
+- Side or 3/4-side view for run/walk
+- Same scale and costume the game already uses when a reference exists
+- No text, UI, watermark, speech bubbles, second character
+
+Good base pattern:
+
+```text
+Side-view full-body 2D game sprite of <subject>, <style>, standing ready pose,
+facing right, centered in frame, feet near lower third, solid flat magenta
+background #FF00FF only, no ground, no shadow, no text, crisp readable silhouette.
+```
+
+If matching a project sprite: use `imagine_image_to_image` with the existing frame's `file_path` and only change pose/background to magenta if needed. Prefer compositing a known good frame onto magenta in code when the art already exists.
+
+## Video (`imagine_image_to_video`)
+
+Write **one short present-tense shot** (1–2 sentences). Constraints:
+
+| Do | Don't |
+| --- | --- |
+| Run/walk **in place** (treadmill) | Travel across the screen |
+| Locked camera | Pan, zoom, orbit, handheld |
+| Keep solid magenta background | Scenic BG, ground scroll, particles filling frame |
+| Single continuous action | Combo attacks + movement + camera |
+| Stable identity/clothes | Costume change mid-clip |
+
+Run example:
+
+```text
+The same chibi ninja character runs in place facing right with a classic side-scroller
+stride, arms trailing slightly back, body centered, camera locked, solid flat magenta
+background only.
+```
+
+Walk / idle examples:
+
+```text
+The character walks in place facing right with a steady side-view walk cycle, camera locked, solid magenta background.
+```
+
+```text
+The character idles in place with a subtle breathing bob and weight shift, camera locked, solid magenta background.
+```
+
+Attack (use carefully — identity drift is higher):
+
+```text
+The character performs a single short punch combo in place facing right, camera locked, solid magenta background, no screen-filling FX.
+```
+
+## Sampling guidance (post-video)
+
+- Export multiple densities: 8 / 16 / 24 / 48 for comparison GIFs
+- For engine integration: pick one smooth cycle (~12–16 frames) after watching previews
+- If the 6s clip contains multiple run cycles, denser even sampling across the whole clip can look like a long multi-cycle animation — that is OK for previews, but for a game loop re-sample one cycle region manually if needed