@lalalic/markcut 3.1.1 → 3.2.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lalalic/markcut",
-  "version": "3.1.1",
+  "version": "3.2.0",
   "description": "Markdown-to-video engine. Describe scenes in markdown, get a rendered video.",
   "bin": {
     "markcut": "bin/markcut"
@@ -65,5 +65,9 @@
     "@types/react-dom": "^19.0.0",
     "typescript": "^5.6.0",
     "vitest": "^2.1.0"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/lalalic/markcut.git"
   }
 }

package/skills/markcut/SKILL.md

@@ -18,6 +18,13 @@ Everything video is a **stream tree** described with markdown. see [docs/markdow
 
 - Use `scene` nodes to organize your video. Scenes can nest inside other scenes.
 - Use `description`, `scene.instruction`, `script`, `image|video.prompt` to structure your video content.
+- **If a scene has a `script` (or `audio`) plus one primary visual, the visual MUST be `isBackground:true`.** Without it the visual plays only its own duration (3s default for images) while narration continues — the rest of the scene is a black screen. Scene duration follows the audio, not the image.
+
+```md
+## scene-1
+- image prompt:"..." isBackground:true   # fills the whole scene, narration decides duration
+- script "..."
+```
 
 see [docs/markdown-descriptive.md](docs/markdown-descriptive.md) for full details.
 
@@ -71,15 +78,12 @@ npx @lalalic/markcut spots --waypoints "lat,lng;lat,lng" # discover POIs along a
 
 ---
 
-## self verification
-some common issues (photo or video can't be displayed, audio missing), take below actions to verify
-### preview
-- take screenshot for some key frames in player, and understand image to verify intent
-
-### final video
-- screenshot some key frames, and understand image to verify intent
-- stt the final video audio, and verify if vtt result is correct
-
+## review
+use the review contract defined in [./review.md](./review.md) to guide the review process.
+review as early as possible in the video production process to catch issues before they propagate.
+* review md file
+* review compiled.json
+* review the rendered video
 
 ## Reference
 
@@ -89,8 +93,7 @@ some common issues (photo or video can't be displayed, audio missing), take belo
 | Route / vlog map clips (effects, spots, examples) | [docs/map-dynamic-camera.md](docs/map-dynamic-camera.md) |
 | Built-in components & common npm packages | [docs/components.md](docs/components.md) |
 | Sound effects | [docs/sound-effects.md](docs/sound-effects.md) |
-
-
+| Review contract | [./review.md](./review.md) |
 
 ## Built-in Components
 
@@ -98,12 +101,6 @@ Built-in components available via `@lalalic/markcut/components`. See [docs/compo
 
 
 ## Golden rule
-- always check stream start and duration to avoid
-  - audio cut off
-  - video cut off
-  - subtitle mismatch
-  - sync issues between audio, video, and subtitles
-according to the content change. rm `.markcut` will cause all content to be regenerated, which is time consuming and wasteful.
 - put all manual assets in `assets` folder, such as bgm, logo, watermark, etc. don't put them in `.markcut` folder, which is auto generated and will be deleted when `markcut clean` command is run.
 
 ### Don'ts

package/skills/markcut/docs/markdown-descriptive.md

@@ -705,7 +705,7 @@ tween(#000, #FFF)             — color interpolation
 # Best practices
 - Markcut engine will automatically determine the duration of background streams, video, script audio. Don't set duration for them if no special requirements (e.g., speed up/down on purpose).
 - global streams (BGM, Logo, Lip-sync video,...) should be set on root level, and set `isBackground:true` to let it loop to fill the whole video duration.
-- Set `isBackground:true` for static vision content, such as image, when audio is playing
+- **Set `isBackground:true` for static vision content (image) whenever the scene has audio/script.** Scene duration is driven by the audio (which can be much longer than the image's default 3s); without `isBackground` the image ends early and the scene tail plays over a black screen while narration continues.
 - adjust audio start time with `start` property to avoid audio cut when transition effect is applied on vision scene
 
 

package/skills/markcut/review.md

@@ -0,0 +1,480 @@
+# Markcut Skill — Agent Video Review Prompt
+
+You are an expert reviewer evaluating a video generated by an LLM using the Markcut skill.
+
+Your goal is to determine whether the **skill enabled the LLM to make good video-authoring decisions** and whether the resulting video satisfies the user's intent.
+
+You will receive:
+
+- **USER_REQUEST** — the original user's request
+- **VIDEO_MD** — the Markdown generated by the agent
+- **VIDEO** — the rendered video
+
+Review the rendered video as the primary artifact. Use the request to understand intent and the Markdown to understand authoring decisions.
+
+## Review workflow
+
+1. Read the user's request and identify the purpose, audience, platform, duration, style, and required information.
+2. Read the generated Markdown and understand its scenes, media, narration, timing, layouts, subtitles, and effects.
+3. Watch the **entire rendered video** or compiled.json before making a final judgment if available.
+4. Compare the video with the user's intent.
+5. Identify only meaningful, observable problems.
+6. For each problem, determine whether it is caused by AUTHORING, CONTENT, DESIGN, ENGINE, ENVIRONMENT, or is UNKNOWN.
+7. For AUTHORING problems, provide a concrete rule that could improve the Markcut skill.
+
+Do not judge the Markdown alone. A technically valid Markdown file can still produce a poor video.
+
+Do not invent problems. Do not report personal stylistic preferences. Prefer a small number of high-confidence, actionable findings over a long list of subjective observations.
+
+---
+
+# Critical Markcut Checks
+
+The following checks are especially important because they represent common failures observed when agents generate Markcut video Markdown.
+
+## 1. No dark scene tail
+
+**Problem:** A scene becomes black near the end while narration continues.
+
+Typical symptom:
+
+- the scene image/video ends before the scene narration;
+- the final frames of the scene are black or empty;
+- narration continues over the black tail.
+
+This is a high-priority timing/authoring problem because the viewer sees an unintended blank screen while still hearing narration.
+
+**Preferred authoring pattern:** When a scene has one primary visual that should remain visible for the whole scene, make the scene image a background using `isBackground` so it plays for the entire scene.
+
+**Review question:**
+
+> Does the visual remain intentionally present for the entire narration, especially at the end of the scene?
+
+If the narration continues after the visible content has ended, report the issue.
+
+Typical classification:
+
+- `TIMING-001` or `TIMING-003`
+- source: `AUTHORING`
+
+Do not report this if the black/empty tail is clearly intentional and appropriate.
+
+## 2. Every scene should have an intentional visual
+
+**Problem:** A whole scene is black because the scene contains narration but no image/video/component that produces visible content.
+
+Typical symptom:
+
+```md
+## Scene
+
+- script: ...
+```
+
+with no visual content.
+
+For normal video scenes, the default expectation is that narration should have a meaningful visual supporting it.
+
+**Preferred authoring pattern:** Add an image prompt when appropriate, for example:
+
+```md
+- image prompt: "..."
+```
+
+The prompt should describe a visual that supports the narration, not merely repeat the narration.
+
+**Review question:**
+
+> Is any scene unintentionally blank because the agent forgot to provide visual content?
+
+Typical classification:
+
+- `VISUAL-009` or `STRUCTURE-001`
+- source: `AUTHORING`
+
+Do not flag intentionally audio-only content when the user's request or video style clearly calls for it.
+
+## 3. Visuals actually display
+
+**Problem:** The Markdown contains a visual reference, but the rendered video shows white, empty, missing, or broken content.
+
+Common causes:
+
+- 404 media source;
+- invalid URL;
+- absolute local path that is not portable;
+- media path resolved relative to the wrong directory;
+- generated asset does not exist;
+- stale/corrupt cached asset.
+
+**Preferred authoring pattern:** Media paths should normally be portable and relative to the Markdown file's folder, for example:
+
+```md
+- image src: assets/example.jpg
+```
+
+rather than relying on an environment-specific absolute path.
+
+**Review question:**
+
+> Does every referenced visual actually appear correctly in the rendered video?
+
+If the Markdown requests a visual and the rendered result is blank/white/404, report it.
+
+Typical classification:
+
+- `VISUAL-002` or `VISUAL-008`
+- source: `ENGINE` if the path is reasonable but Markcut fails to render it;
+- source: `AUTHORING` if the agent generated an invalid/non-portable path;
+- source: `ENVIRONMENT` if the resource genuinely cannot be accessed in the evaluation environment.
+
+## 4. Audio must not be cut off
+
+**Problem:** The final part of the narration is missing or truncated.
+
+Typical symptom:
+
+- the script contains more text than the generated speech;
+- STT/transcription ends before the script ends;
+- the last sentence or words are missing from the audio;
+- the scene appears visually complete but the spoken narration is cut off.
+
+**Preferred authoring principle:** Apply the same timing principle as the scene-tail check: visual and scene duration must accommodate the complete narration.
+
+**Review question:**
+
+> Does the generated audio contain the complete intended narration, including the final words of the script?
+
+If the rendered audio cuts off the script tail, report it.
+
+Typical classification:
+
+- `AUDIO-006` or `TIMING-001`
+- source: `AUTHORING` when scene timing/authoring causes the truncation;
+- source: `ENGINE` when reasonable Markdown is rendered incorrectly.
+
+## 5. Subtitles must match narration
+
+**Problem:** Subtitles are missing, stale, or no longer correspond to the actual narration.
+
+Check for:
+
+- no subtitles when subtitles are expected;
+- subtitle cues missing from the narration;
+- stale VTT generated from an earlier script;
+- subtitle text different from the current script/audio;
+- subtitle timing that clearly does not match speech.
+
+**Preferred resolution:** Regenerate stale VTT/subtitle data whenever the narration changes. Do not preserve cached subtitle data that corresponds to an older script.
+
+**Review question:**
+
+> Do the subtitles represent the narration that the viewer actually hears?
+
+Typical classification:
+
+- `SUBTITLE-003` or `SUBTITLE-004`
+- source: `AUTHORING` when stale/generated artifacts are knowingly reused;
+- source: `ENGINE` when Markcut fails to regenerate or synchronize subtitles correctly.
+
+## 6. Content must match user intent
+
+**Problem:** The video technically works, but the imagery or content does not communicate what the user asked for.
+
+Examples:
+
+- narration says "a child learning to ride a bicycle" but the image is unrelated;
+- a financial explanation uses generic unrelated office imagery;
+- the visual suggests a different meaning from the narration;
+- an important concept is never visually represented;
+- the generated story drifts away from the requested topic.
+
+This is generally **advisory** rather than a rendering failure.
+
+**Preferred resolution:** Improve the image/media prompt, scene decomposition, or narration. If the incorrect media is caused by a stale generated asset, invalidate the cache and regenerate it.
+
+**Review question:**
+
+> If a viewer watches the visuals while listening to the narration, do the two communicate the same intended idea?
+
+Typical classification:
+
+- `VISUAL-001`, `VISUAL-008`, or `VISUAL-009`
+- source: `AUTHORING` or `CONTENT`
+
+Do not fail a case merely because the image is metaphorical rather than literal, provided it supports the intended message.
+
+---
+
+# General Review Dimensions
+
+Score each applicable dimension from 1 to 5.
+
+## Storytelling / Communication
+
+Check:
+
+- Is there a clear purpose?
+- Is the opening effective?
+- Does each scene move the message forward?
+- Is the information ordered logically?
+- Is the main message understandable?
+- Is there unnecessary repetition?
+- Does the ending provide an appropriate conclusion or payoff?
+
+## Narration / Visual Alignment
+
+Check whether visuals support the narration.
+
+A visual does not need to literally illustrate every word. Conceptual visuals, examples, diagrams, and metaphors are acceptable when they improve understanding.
+
+## Visual Quality
+
+Check:
+
+- readability;
+- cropping;
+- overlap;
+- empty space;
+- visual clutter;
+- media relevance;
+- repetition;
+- excessive effects;
+- overall coherence.
+
+## Timing / Pacing
+
+Check:
+
+- narration vs visual duration;
+- static visuals that last too long;
+- visuals that disappear too early;
+- important content shown too briefly;
+- awkward cuts;
+- excessive transitions;
+- unnatural pacing.
+
+## Audio
+
+Check:
+
+- narration speed;
+- pronunciation;
+- unnatural pauses;
+- missing/truncated narration;
+- volume consistency;
+- background music competing with speech.
+
+## Subtitles
+
+Check:
+
+- presence when expected;
+- synchronization;
+- stale content;
+- readability;
+- line breaks;
+- density;
+- visual overlap.
+
+## Platform Suitability
+
+If the user specifies a platform, judge against it.
+
+For short-form video, pay particular attention to the hook, pacing, mobile readability, and visual changes.
+
+For tutorials, pay particular attention to step ordering, demonstrations, and sufficient time to understand each step.
+
+Do not impose platform-specific requirements when the user did not specify a platform.
+
+## Markcut Authoring Quality
+
+Evaluate whether the agent uses Markcut appropriately.
+
+Look for:
+
+- poor scene decomposition;
+- poor duration choices;
+- unnecessary complexity;
+- unnecessary components/effects;
+- poor media instructions;
+- narration packed into one visual beat;
+- missing visuals;
+- non-portable media references.
+
+Prefer the simplest Markcut structure that produces a strong result.
+
+---
+
+# Issue Taxonomy
+
+Use these IDs when applicable.
+
+### STORY
+
+- `STORY-001` — weak-hook
+- `STORY-002` — no-progression
+- `STORY-003` — repetitive-content
+- `STORY-004` — unclear-message
+- `STORY-005` — weak-ending
+- `STORY-006` — unnecessary-introduction
+- `STORY-007` — poor-information-order
+
+### VISUAL
+
+- `VISUAL-001` — narration-visual-mismatch
+- `VISUAL-002` — missing-or-broken-visual
+- `VISUAL-003` — repetitive-visual
+- `VISUAL-004` — text-too-small
+- `VISUAL-005` — text-overflow
+- `VISUAL-006` — excessive-empty-space
+- `VISUAL-007` — visual-clutter
+- `VISUAL-008` — poor-media-selection
+- `VISUAL-009` — insufficient-visual-explanation
+
+### TIMING
+
+- `TIMING-001` — narration-too-long-for-scene
+- `TIMING-002` — narration-too-short-for-scene
+- `TIMING-003` — static-visual-too-long
+- `TIMING-004` — visual-too-short
+- `TIMING-005` — awkward-cut
+- `TIMING-006` — transition-too-long
+- `TIMING-007` — excessive-transitions
+- `TIMING-008` — poor-pacing
+
+### AUDIO
+
+- `AUDIO-001` — narration-too-fast
+- `AUDIO-002` — narration-too-slow
+- `AUDIO-003` — unnatural-narration
+- `AUDIO-004` — awkward-pause
+- `AUDIO-005` — music-too-loud
+- `AUDIO-006` — narration-cut-off
+
+### SUBTITLE
+
+- `SUBTITLE-001` — too-dense
+- `SUBTITLE-002` — bad-line-break
+- `SUBTITLE-003` — timing-mismatch
+- `SUBTITLE-004` — missing-or-stale-subtitles
+- `SUBTITLE-005` — visual-overlap
+
+### STRUCTURE
+
+- `STRUCTURE-001` — missing-content
+- `STRUCTURE-002` — unnecessary-scene
+- `STRUCTURE-003` — wrong-order
+- `STRUCTURE-004` — inappropriate-layout
+- `STRUCTURE-005` — poor-scene-decomposition
+
+### AUTHORING
+
+- `AUTHORING-001` — unnecessary-complexity
+- `AUTHORING-002` — poor-duration-selection
+- `AUTHORING-003` — poor-media-instruction
+- `AUTHORING-004` — unnecessary-component
+- `AUTHORING-005` — unnecessary-effect
+- `AUTHORING-006` — non-portable-media-path
+- `AUTHORING-007` — stale-generated-artifact
+
+### ENGINE
+
+Use only when the Markdown contains a reasonable instruction and Markcut appears not to implement it correctly.
+
+---
+
+# Severity
+
+- `critical` — unusable or fails the core purpose
+- `high` — substantially damages viewer experience
+- `medium` — noticeable and worth fixing
+- `low` — minor improvement or polish
+
+---
+
+# Evidence Rules
+
+Every issue must contain concrete evidence.
+
+Reference, when possible:
+
+- scene name;
+- approximate timestamp;
+- narration;
+- visible content;
+- duration;
+- subtitle behavior;
+- Markdown instruction.
+
+Do not invent exact timestamps. Use approximate timestamps when necessary.
+
+Do not report the same underlying problem multiple times. Report the root problem and mention related symptoms in the evidence.
+
+---
+
+# Output Format
+
+Return the following structure.
+
+## Overall
+
+```yaml
+score: <1-5>
+summary: <short assessment>
+```
+
+## Scores
+
+```yaml
+storytelling: <1-5>
+visual_quality: <1-5>
+narration_visual_alignment: <1-5>
+timing_pacing: <1-5>
+audio: <1-5>
+subtitles: <1-5>
+platform_suitability: <1-5>
+markcut_authoring: <1-5>
+```
+
+Use the average of applicable dimensions for the overall score.
+
+## Issues
+
+For each meaningful issue:
+
+```yaml
+- id: VISUAL-001
+  severity: medium
+  source: AUTHORING
+  scene: "Solution"
+  timestamp: "approximately 00:18-00:25"
+  evidence: "..."
+  impact: "..."
+  recommendation: "..."
+  confidence: high
+```
+
+If there are no meaningful issues:
+
+```yaml
+issues: []
+```
+
+## Strengths
+
+List the most important things the video does well.
+
+## Priority Improvements
+
+List the three most valuable improvements, in priority order.
+
+---
+
+# Final Principle
+
+Answer this question above all others:
+
+> **If a user installed the Markcut skill and asked an LLM to create this video, did the skill enable the LLM to make good decisions?**
+
+Evaluate the actual viewer experience, use the Markdown to understand authoring intent, distinguish authoring mistakes from engine failures, and provide actionable evidence-based feedback.

package/tests/evals/README.md

@@ -0,0 +1,41 @@
+# markcut storyboard evals
+
+DeepEval suite that scores an agent authoring markcut storyboards from a
+video brief, using the markcut skill.
+
+## Components
+
+| File | Purpose |
+| --- | --- |
+| `openrouter_model.py` | Token-capped OpenRouter LLM wrapper (free-tier friendly, with model fallback rotation) |
+| `gen_dataset.py` | Generates goldens from scratch via the Synthesizer |
+| `dataset.json` | The golden dataset (committed, editable) |
+| `storyboard_app.py` | Traced app: runs `pi` CLI with the markcut skill to author a storyboard |
+| `metrics.py` | Judge metrics: TaskCompletion, Storyboard Format GEval, Story Narrative GEval |
+| `test_storyboard.py` | pytest traced single-turn evals |
+
+## Regenerate the dataset
+
+```bash
+.venv-evals/bin/python tests/evals/gen_dataset.py
+```
+
+## Run the evals
+
+```bash
+cd tests/evals && ../../.venv-evals/bin/deepeval test run test_storyboard.py \
+  --identifier "iterating-on-storyboard-authoring-round-1"
+```
+
+Each test runs the pi agent on the golden's video brief and asserts
+trace-level metrics. Failures typically indicate the agent violated a
+skill rule (e.g. `duration:` on scripted scenes, missing
+`isBackground:true`) or produced weak narrative structure.
+
+## Notes
+
+- Evaluation model runs on OpenRouter free models; the wrapper caps
+  max_tokens at 16000 to fit the workspace key's daily budget and
+  rotates through fallback models on rate limits.
+- Traces are local (Confident AI not enabled); view latest results via
+  `deepeval view --latest` offline report in `.deepeval/`.

package/tests/evals/dataset.json

@@ -0,0 +1,170 @@
+[
+    {
+        "input": "30s cinematic coastal drive vlog w/ voiceover detailing scenic landmarks & geography",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Detail how teaser reveals specs, competitive edge, and launch strategy.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Draft a 2‑speaker dialogue explaining quantum computing, with visual analogy and real‑world app.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Develop a 5‑min pasta recipe paired with lo‑fi background music for enhanced cooking experience",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "How did 1960s Ford integrate sustainability into its corporate brand story?",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Create a cinematic sci-fi thriller trailer, <2 min, featuring dystopian cityscape and AI antagonist",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Compare walkthrough of modern luxury villa vs historic estate: design, amenities, layout.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Compare high-energy cuts vs. slow cinematic pacing for fitness motivation shorts.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Produce a short historical documentary snippet on the Indus. Rev., focusing on inventions",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Compare dashboard walkthroughs in SW A and B tutorials, noting UI similarities and differences.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Compare a 15‑s skincare ad with strong CTA vs a 30‑s ad: effectiveness, engagement, conversion.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Describe how pressure adaptations let deep-sea creatures survive extreme conditions in the clip.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "30s cinematic coastal drive vlog w/ voiceover detailing scenic landmarks & geography",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Detail how teaser reveals specs, competitive edge, and launch strategy.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Draft a 2‑speaker dialogue explaining quantum computing, with visual analogy and real‑world app.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Develop a 5‑min pasta recipe paired with lo‑fi background music for enhanced cooking experience",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "How did 1960s Ford integrate sustainability into its corporate brand story?",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Create a cinematic sci-fi thriller trailer, <2 min, featuring dystopian cityscape and AI antagonist",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Compare walkthrough of modern luxury villa vs historic estate: design, amenities, layout.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Compare high-energy cuts vs. slow cinematic pacing for fitness motivation shorts.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Produce a short historical documentary snippet on the Indus. Rev., focusing on inventions",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Compare dashboard walkthroughs in SW A and B tutorials, noting UI similarities and differences.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Compare a 15‑s skincare ad with strong CTA vs a 30‑s ad: effectiveness, engagement, conversion.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Describe how pressure adaptations let deep-sea creatures survive extreme conditions in the clip.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    }
+]

package/tests/evals/gen_dataset.py

@@ -0,0 +1,63 @@
+"""Generate the markcut storyboard goldens with a token-capped OpenRouter model.
+
+Run:
+  OPENROUTER_API_KEY=... .venv-evals/bin/python tests/evals/gen_dataset.py
+"""
+
+import os
+
+from deepeval.synthesizer import Synthesizer
+from deepeval.synthesizer.config import StylingConfig
+
+from openrouter_model import OpenRouterLLM
+
+SCENARIO = (
+    "Video creators and AI agents using the markcut markdown-to-video skill "
+    "to author video storyboards from a brief. markcut uses a markdown "
+    "descriptive format: '## scene' headings, '- image prompt:...' bullets, "
+    "script \"...\" narration, isBackground:true for visuals under scripted "
+    "scenes, map stream type with view/tween camera moves for route vlogs, "
+    "built-in components, TTS/TTI/STT media pipeline, and golden rules such "
+    "as never set duration on scripted scenes, keep manual assets in the "
+    "assets/ folder, and review md then compiled json then rendered video."
+)
+TASK = (
+    "Act as a storyboard author: given a video brief, produce a markcut "
+    "storyboard in markdown descriptive format that follows the skill's "
+    "scene/isBackground/duration rules and narrative structure (hook, "
+    "conflict, resolution, emotion, call to action)."
+)
+INPUT_FORMAT = (
+    "A short video brief, e.g. 'a 30s travel vlog of a coastal route with "
+    "narration', 'a product launch teaser with stats', 'a two-speaker "
+    "dialogue explainer', 'a recipe short with background music'."
+)
+EXPECTED_OUTPUT_FORMAT = (
+    "A complete markcut storyboard markdown using ## scene headings, "
+    "- image/script bullets, correct isBackground usage, and no manual "
+    "durations on scripted scenes."
+)
+
+
+def main():
+    model = OpenRouterLLM()
+    synthesizer = Synthesizer(
+        model=model,
+        styling_config=StylingConfig(
+            scenario=SCENARIO,
+            task=TASK,
+            input_format=INPUT_FORMAT,
+            expected_output_format=EXPECTED_OUTPUT_FORMAT,
+        ),
+    )
+    goldens = synthesizer.generate_goldens_from_scratch(num_goldens=12)
+    synthesizer.save_as(
+        file_type="json",
+        directory="tests/evals",
+        file_name="dataset",
+    )
+    print(f"Saved {len(goldens)} goldens to tests/evals/dataset.json")
+
+
+if __name__ == "__main__":
+    main()

package/tests/evals/metrics.py

@@ -0,0 +1,70 @@
+"""DeepEval metrics for the markcut storyboard eval suite.
+
+Judge model is the token-capped OpenRouter wrapper so the whole suite
+runs on the workspace's free OpenRouter key.
+"""
+
+import sys
+import os
+
+sys.path.insert(0, os.path.dirname(__file__))
+
+from deepeval.metrics import GEval, TaskCompletionMetric
+from deepeval.test_case import LLMTestCaseParams
+
+from openrouter_model import OpenRouterLLM
+
+
+def _judge():
+    return OpenRouterLLM(temperature=0.0)
+
+
+# Trace-level: did the agent finish the storyboard task at all?
+TASK_COMPLETION_METRICS = [
+    TaskCompletionMetric(model=_judge(), threshold=0.7),
+]
+
+# Is the storyboard valid markcut markdown following the skill's rules?
+STORYBOARD_FORMAT_METRIC = GEval(
+    name="Storyboard Format",
+    criteria=(
+        "Evaluate whether the actual output is a valid markcut storyboard "
+        "in markdown descriptive format. It must: use '## scene' headings "
+        "for scenes; use '- image prompt:\"...\"' or similar bullet syntax "
+        "for visuals; include narration via script \"...\"; set "
+        "isBackground:true on the primary visual of any scene that has a "
+        "script or audio (otherwise the scene plays black); and NOT set "
+        "manual durations on scripted scenes since the resolver derives "
+        "duration from the audio script."
+    ),
+    evaluation_params=[
+        LLMTestCaseParams.INPUT,
+        LLMTestCaseParams.ACTUAL_OUTPUT,
+    ],
+    model=_judge(),
+    threshold=0.7,
+)
+
+# Does the storyboard tell a compelling viral story?
+STORY_NARRATIVE_METRIC = GEval(
+    name="Story Narrative",
+    criteria=(
+        "Evaluate whether the storyboard tells a compelling short video "
+        "story. It should contain a hook (why watch), conflict or "
+        "challenge, resolution, emotional beats, a call to action, and "
+        "ideally an open ending where appropriate. Score low if it is a "
+        "flat list of scenes with no narrative structure."
+    ),
+    evaluation_params=[
+        LLMTestCaseParams.INPUT,
+        LLMTestCaseParams.ACTUAL_OUTPUT,
+    ],
+    model=_judge(),
+    threshold=0.6,
+)
+
+SINGLE_TURN_TRACE_METRICS = [
+    TASK_COMPLETION_METRICS[0],
+    STORYBOARD_FORMAT_METRIC,
+    STORY_NARRATIVE_METRIC,
+]

package/tests/evals/openrouter_model.py

@@ -0,0 +1,143 @@
+"""OpenRouter-backed DeepEval LLM with a capped max_tokens budget.
+
+The workspace OPENROUTER_API_KEY has a daily credit cap that rejects
+requests asking for 65536 max_tokens, so we wrap the OpenAI-compatible
+client ourselves and ask for far fewer output tokens.
+"""
+
+import os
+from typing import Optional, List
+
+from deepeval.models import DeepEvalBaseLLM
+from deepeval.synthesizer.schema import (
+    Response as _ResponseSchema,
+    SyntheticData,
+    SyntheticDataList,
+)
+from deepeval.metrics.utils import trimAndLoadJson
+
+MAX_OUTPUT_TOKENS = 16000
+
+# Free-tier models are flaky; rotate through candidates on rate limits.
+FALLBACK_MODELS = [
+    "google/gemma-4-31b-it:free",
+    "nvidia/nemotron-3-super-120b-a12b:free",
+    "inclusionai/ling-3.0-flash-sante:free",
+    "minimax/minimax-m2.7:free",
+]
+
+
+class OpenRouterLLM(DeepEvalBaseLLM):
+    def __init__(
+        self,
+        model: str = "google/gemma-4-31b-it:free",
+        temperature: float = 0.0,
+        **kwargs,
+    ):
+        self.model_id = model
+        self.temperature = temperature
+        super().__init__(model, **kwargs)
+
+    def load_model(self):
+        from openai import OpenAI
+
+        return OpenAI(
+            api_key=os.environ["OPENROUTER_API_KEY"],
+            base_url="https://openrouter.ai/api/v1",
+        )
+
+    def generate(
+        self,
+        prompt: str,
+        schema=None,
+        max_output_tokens: int = MAX_OUTPUT_TOKENS,
+    ) -> str:
+        client = self.load_model()
+        if schema is not None:
+            # Sending the raw JSON schema makes weak models echo it back;
+            # describe the shape in prose instead.
+            if schema is SyntheticDataList:
+                shape_desc = (
+                    'a JSON object {"data": [{"input": "<brief text>"}]} '
+                    "with as many items as requested"
+                )
+            elif schema is _ResponseSchema:
+                shape_desc = 'a JSON object {"response": "<your answer>"}'
+            else:
+                shape_desc = (
+                    "a single JSON object with exactly these keys: "
+                    f"{list(schema.model_fields.keys())}"
+                )
+            prompt = (
+                "You are a JSON generator. Respond with ONLY one valid JSON "
+                "value - no prose, no markdown fences, no commentary, and "
+                "never repeat the schema itself. The JSON must be exactly "
+                f"{shape_desc}.\n\nTask:\n{prompt}"
+            )
+
+        import time
+
+        text = None
+        last_err = None
+        for model_id in [self.model_id] + [
+            m for m in FALLBACK_MODELS if m != self.model_id
+        ]:
+            for attempt in range(3):
+                try:
+                    response = client.chat.completions.create(
+                        model=model_id,
+                        messages=[{"role": "user", "content": prompt}],
+                        temperature=self.temperature,
+                        max_tokens=min(max_output_tokens, MAX_OUTPUT_TOKENS),
+                    )
+                    text = response.choices[0].message.content or ""
+                    break
+                except Exception as err:  # rate limits, upstream 429s
+                    last_err = err
+                    time.sleep(5 * (attempt + 1))
+            if text:
+                break
+        if not text:
+            raise last_err
+        if schema is None:
+            return text
+        # Parse the JSON the prompt asked for into the requested schema
+        try:
+            data = trimAndLoadJson(text, self)
+        except ValueError:
+            with open("/tmp/deepeval_last_output.txt", "w") as f:
+                f.write(text)
+            raise
+        if schema is SyntheticDataList:
+            return SyntheticDataList(
+                data=[SyntheticData(**item) for item in data["data"]]
+            )
+        if schema is _ResponseSchema and isinstance(data, str):
+            return _ResponseSchema(response=data)
+        return schema(**data)
+
+    async def a_generate(
+        self,
+        prompt: str,
+        schema=None,
+        max_output_tokens: int = MAX_OUTPUT_TOKENS,
+    ) -> str:
+        return self.generate(prompt, schema, max_output_tokens)
+
+    def get_model_name(self) -> str:
+        return self.model_id
+
+    def supports_json_mode(self) -> bool:
+        return False
+
+    def supports_structured_outputs(self) -> bool:
+        return False
+
+    def supports_log_probs(self) -> Optional[bool]:
+        return False
+
+    def supports_multimodal(self) -> bool:
+        return False
+
+    def batch_generate(self, prompts: List[str]) -> List[str]:
+        return [self.generate(p) for p in prompts]

package/tests/evals/storyboard_app.py

@@ -0,0 +1,55 @@
+"""Traced markcut storyboard authoring app.
+
+Runs the pi agent CLI with the markcut skill against a video brief and
+returns the produced storyboard markdown. The @observe decorator makes
+the run a DeepEval agent trace so traced single-turn evals can score it.
+"""
+
+import os
+import subprocess
+import tempfile
+
+from deepeval.tracing import observe
+
+MARKCUT_SKILL = os.path.abspath(
+    os.path.join(
+        os.path.dirname(__file__), "..", "..", "skills", "markcut", "SKILL.md"
+    )
+)
+
+
+def _read_skill() -> str:
+    with open(MARKCUT_SKILL) as f:
+        return f.read()
+
+
+@observe(type="agent", name="markcut-storyboard-author")
+def run_traced_storyboard(brief: str) -> str:
+    """Author a markcut storyboard markdown for the given brief."""
+    with tempfile.TemporaryDirectory() as workdir:
+        prompt = (
+            "You are a video storyboard author using the markcut skill.\n"
+            f"Video brief: {brief}\n\n"
+            "Write the storyboard as a markcut markdown file named "
+            "storyboard.md in the current directory. Follow the skill's "
+            "rules strictly. When done, print ONLY the final storyboard "
+            "markdown to stdout with no extra commentary."
+        )
+        result = subprocess.run(
+            [
+                "pi",
+                "-p",
+                "--no-session",
+                "--skill", MARKCUT_SKILL,
+                prompt,
+            ],
+            capture_output=True,
+            text=True,
+            timeout=600,
+            cwd=workdir,
+        )
+        storyboard_path = os.path.join(workdir, "storyboard.md")
+        if os.path.exists(storyboard_path):
+            with open(storyboard_path) as f:
+                return f.read()
+        return result.stdout

package/tests/evals/test_storyboard.py

@@ -0,0 +1,32 @@
+"""Traced single-turn evals for the markcut storyboard authoring agent.
+
+Each golden's input is a video brief; the app runs the pi agent CLI with
+the markcut skill to author a storyboard markdown, traced with @observe.
+
+Run:
+  cd tests/evals && ../../.venv-evals/bin/deepeval test run test_storyboard.py
+"""
+
+import sys
+import os
+
+sys.path.insert(0, os.path.dirname(__file__))
+
+import pytest
+from deepeval import assert_test
+from deepeval.dataset import EvaluationDataset, Golden
+
+from metrics import SINGLE_TURN_TRACE_METRICS
+from storyboard_app import run_traced_storyboard
+
+
+dataset = EvaluationDataset()
+dataset.add_goldens_from_json_file(
+    file_path=os.path.join(os.path.dirname(__file__), "dataset.json")
+)
+
+
+@pytest.mark.parametrize("golden", dataset.goldens)
+def test_storyboard_authoring(golden: Golden):
+    run_traced_storyboard(golden.input)
+    assert_test(golden=golden, metrics=SINGLE_TURN_TRACE_METRICS)

package/.vscode/settings.json

@@ -1,3 +0,0 @@
-{
-    "copilot-infinite.discord.enabled": true
-}