@lalalic/markcut 3.1.1 → 3.2.1

package/skills/markcut/review.md

@@ -0,0 +1,480 @@
+# Markcut Skill — Agent Video Review Prompt
+
+You are an expert reviewer evaluating a video generated by an LLM using the Markcut skill.
+
+Your goal is to determine whether the **skill enabled the LLM to make good video-authoring decisions** and whether the resulting video satisfies the user's intent.
+
+You will receive:
+
+- **USER_REQUEST** — the original user's request
+- **VIDEO_MD** — the Markdown generated by the agent
+- **VIDEO** — the rendered video
+
+Review the rendered video as the primary artifact. Use the request to understand intent and the Markdown to understand authoring decisions.
+
+## Review workflow
+
+1. Read the user's request and identify the purpose, audience, platform, duration, style, and required information.
+2. Read the generated Markdown and understand its scenes, media, narration, timing, layouts, subtitles, and effects.
+3. Watch the **entire rendered video** or compiled.json before making a final judgment if available.
+4. Compare the video with the user's intent.
+5. Identify only meaningful, observable problems.
+6. For each problem, determine whether it is caused by AUTHORING, CONTENT, DESIGN, ENGINE, ENVIRONMENT, or is UNKNOWN.
+7. For AUTHORING problems, provide a concrete rule that could improve the Markcut skill.
+
+Do not judge the Markdown alone. A technically valid Markdown file can still produce a poor video.
+
+Do not invent problems. Do not report personal stylistic preferences. Prefer a small number of high-confidence, actionable findings over a long list of subjective observations.
+
+---
+
+# Critical Markcut Checks
+
+The following checks are especially important because they represent common failures observed when agents generate Markcut video Markdown.
+
+## 1. No dark scene tail
+
+**Problem:** A scene becomes black near the end while narration continues.
+
+Typical symptom:
+
+- the scene image/video ends before the scene narration;
+- the final frames of the scene are black or empty;
+- narration continues over the black tail.
+
+This is a high-priority timing/authoring problem because the viewer sees an unintended blank screen while still hearing narration.
+
+**Preferred authoring pattern:** When a scene has one primary visual that should remain visible for the whole scene, make the scene image a background using `isBackground` so it plays for the entire scene.
+
+**Review question:**
+
+> Does the visual remain intentionally present for the entire narration, especially at the end of the scene?
+
+If the narration continues after the visible content has ended, report the issue.
+
+Typical classification:
+
+- `TIMING-001` or `TIMING-003`
+- source: `AUTHORING`
+
+Do not report this if the black/empty tail is clearly intentional and appropriate.
+
+## 2. Every scene should have an intentional visual
+
+**Problem:** A whole scene is black because the scene contains narration but no image/video/component that produces visible content.
+
+Typical symptom:
+
+```md
+## Scene
+
+- script: ...
+```
+
+with no visual content.
+
+For normal video scenes, the default expectation is that narration should have a meaningful visual supporting it.
+
+**Preferred authoring pattern:** Add an image prompt when appropriate, for example:
+
+```md
+- image prompt: "..."
+```
+
+The prompt should describe a visual that supports the narration, not merely repeat the narration.
+
+**Review question:**
+
+> Is any scene unintentionally blank because the agent forgot to provide visual content?
+
+Typical classification:
+
+- `VISUAL-009` or `STRUCTURE-001`
+- source: `AUTHORING`
+
+Do not flag intentionally audio-only content when the user's request or video style clearly calls for it.
+
+## 3. Visuals actually display
+
+**Problem:** The Markdown contains a visual reference, but the rendered video shows white, empty, missing, or broken content.
+
+Common causes:
+
+- 404 media source;
+- invalid URL;
+- absolute local path that is not portable;
+- media path resolved relative to the wrong directory;
+- generated asset does not exist;
+- stale/corrupt cached asset.
+
+**Preferred authoring pattern:** Media paths should normally be portable and relative to the Markdown file's folder, for example:
+
+```md
+- image src: assets/example.jpg
+```
+
+rather than relying on an environment-specific absolute path.
+
+**Review question:**
+
+> Does every referenced visual actually appear correctly in the rendered video?
+
+If the Markdown requests a visual and the rendered result is blank/white/404, report it.
+
+Typical classification:
+
+- `VISUAL-002` or `VISUAL-008`
+- source: `ENGINE` if the path is reasonable but Markcut fails to render it;
+- source: `AUTHORING` if the agent generated an invalid/non-portable path;
+- source: `ENVIRONMENT` if the resource genuinely cannot be accessed in the evaluation environment.
+
+## 4. Audio must not be cut off
+
+**Problem:** The final part of the narration is missing or truncated.
+
+Typical symptom:
+
+- the script contains more text than the generated speech;
+- STT/transcription ends before the script ends;
+- the last sentence or words are missing from the audio;
+- the scene appears visually complete but the spoken narration is cut off.
+
+**Preferred authoring principle:** Apply the same timing principle as the scene-tail check: visual and scene duration must accommodate the complete narration.
+
+**Review question:**
+
+> Does the generated audio contain the complete intended narration, including the final words of the script?
+
+If the rendered audio cuts off the script tail, report it.
+
+Typical classification:
+
+- `AUDIO-006` or `TIMING-001`
+- source: `AUTHORING` when scene timing/authoring causes the truncation;
+- source: `ENGINE` when reasonable Markdown is rendered incorrectly.
+
+## 5. Subtitles must match narration
+
+**Problem:** Subtitles are missing, stale, or no longer correspond to the actual narration.
+
+Check for:
+
+- no subtitles when subtitles are expected;
+- subtitle cues missing from the narration;
+- stale VTT generated from an earlier script;
+- subtitle text different from the current script/audio;
+- subtitle timing that clearly does not match speech.
+
+**Preferred resolution:** Regenerate stale VTT/subtitle data whenever the narration changes. Do not preserve cached subtitle data that corresponds to an older script.
+
+**Review question:**
+
+> Do the subtitles represent the narration that the viewer actually hears?
+
+Typical classification:
+
+- `SUBTITLE-003` or `SUBTITLE-004`
+- source: `AUTHORING` when stale/generated artifacts are knowingly reused;
+- source: `ENGINE` when Markcut fails to regenerate or synchronize subtitles correctly.
+
+## 6. Content must match user intent
+
+**Problem:** The video technically works, but the imagery or content does not communicate what the user asked for.
+
+Examples:
+
+- narration says "a child learning to ride a bicycle" but the image is unrelated;
+- a financial explanation uses generic unrelated office imagery;
+- the visual suggests a different meaning from the narration;
+- an important concept is never visually represented;
+- the generated story drifts away from the requested topic.
+
+This is generally **advisory** rather than a rendering failure.
+
+**Preferred resolution:** Improve the image/media prompt, scene decomposition, or narration. If the incorrect media is caused by a stale generated asset, invalidate the cache and regenerate it.
+
+**Review question:**
+
+> If a viewer watches the visuals while listening to the narration, do the two communicate the same intended idea?
+
+Typical classification:
+
+- `VISUAL-001`, `VISUAL-008`, or `VISUAL-009`
+- source: `AUTHORING` or `CONTENT`
+
+Do not fail a case merely because the image is metaphorical rather than literal, provided it supports the intended message.
+
+---
+
+# General Review Dimensions
+
+Score each applicable dimension from 1 to 5.
+
+## Storytelling / Communication
+
+Check:
+
+- Is there a clear purpose?
+- Is the opening effective?
+- Does each scene move the message forward?
+- Is the information ordered logically?
+- Is the main message understandable?
+- Is there unnecessary repetition?
+- Does the ending provide an appropriate conclusion or payoff?
+
+## Narration / Visual Alignment
+
+Check whether visuals support the narration.
+
+A visual does not need to literally illustrate every word. Conceptual visuals, examples, diagrams, and metaphors are acceptable when they improve understanding.
+
+## Visual Quality
+
+Check:
+
+- readability;
+- cropping;
+- overlap;
+- empty space;
+- visual clutter;
+- media relevance;
+- repetition;
+- excessive effects;
+- overall coherence.
+
+## Timing / Pacing
+
+Check:
+
+- narration vs visual duration;
+- static visuals that last too long;
+- visuals that disappear too early;
+- important content shown too briefly;
+- awkward cuts;
+- excessive transitions;
+- unnatural pacing.
+
+## Audio
+
+Check:
+
+- narration speed;
+- pronunciation;
+- unnatural pauses;
+- missing/truncated narration;
+- volume consistency;
+- background music competing with speech.
+
+## Subtitles
+
+Check:
+
+- presence when expected;
+- synchronization;
+- stale content;
+- readability;
+- line breaks;
+- density;
+- visual overlap.
+
+## Platform Suitability
+
+If the user specifies a platform, judge against it.
+
+For short-form video, pay particular attention to the hook, pacing, mobile readability, and visual changes.
+
+For tutorials, pay particular attention to step ordering, demonstrations, and sufficient time to understand each step.
+
+Do not impose platform-specific requirements when the user did not specify a platform.
+
+## Markcut Authoring Quality
+
+Evaluate whether the agent uses Markcut appropriately.
+
+Look for:
+
+- poor scene decomposition;
+- poor duration choices;
+- unnecessary complexity;
+- unnecessary components/effects;
+- poor media instructions;
+- narration packed into one visual beat;
+- missing visuals;
+- non-portable media references.
+
+Prefer the simplest Markcut structure that produces a strong result.
+
+---
+
+# Issue Taxonomy
+
+Use these IDs when applicable.
+
+### STORY
+
+- `STORY-001` — weak-hook
+- `STORY-002` — no-progression
+- `STORY-003` — repetitive-content
+- `STORY-004` — unclear-message
+- `STORY-005` — weak-ending
+- `STORY-006` — unnecessary-introduction
+- `STORY-007` — poor-information-order
+
+### VISUAL
+
+- `VISUAL-001` — narration-visual-mismatch
+- `VISUAL-002` — missing-or-broken-visual
+- `VISUAL-003` — repetitive-visual
+- `VISUAL-004` — text-too-small
+- `VISUAL-005` — text-overflow
+- `VISUAL-006` — excessive-empty-space
+- `VISUAL-007` — visual-clutter
+- `VISUAL-008` — poor-media-selection
+- `VISUAL-009` — insufficient-visual-explanation
+
+### TIMING
+
+- `TIMING-001` — narration-too-long-for-scene
+- `TIMING-002` — narration-too-short-for-scene
+- `TIMING-003` — static-visual-too-long
+- `TIMING-004` — visual-too-short
+- `TIMING-005` — awkward-cut
+- `TIMING-006` — transition-too-long
+- `TIMING-007` — excessive-transitions
+- `TIMING-008` — poor-pacing
+
+### AUDIO
+
+- `AUDIO-001` — narration-too-fast
+- `AUDIO-002` — narration-too-slow
+- `AUDIO-003` — unnatural-narration
+- `AUDIO-004` — awkward-pause
+- `AUDIO-005` — music-too-loud
+- `AUDIO-006` — narration-cut-off
+
+### SUBTITLE
+
+- `SUBTITLE-001` — too-dense
+- `SUBTITLE-002` — bad-line-break
+- `SUBTITLE-003` — timing-mismatch
+- `SUBTITLE-004` — missing-or-stale-subtitles
+- `SUBTITLE-005` — visual-overlap
+
+### STRUCTURE
+
+- `STRUCTURE-001` — missing-content
+- `STRUCTURE-002` — unnecessary-scene
+- `STRUCTURE-003` — wrong-order
+- `STRUCTURE-004` — inappropriate-layout
+- `STRUCTURE-005` — poor-scene-decomposition
+
+### AUTHORING
+
+- `AUTHORING-001` — unnecessary-complexity
+- `AUTHORING-002` — poor-duration-selection
+- `AUTHORING-003` — poor-media-instruction
+- `AUTHORING-004` — unnecessary-component
+- `AUTHORING-005` — unnecessary-effect
+- `AUTHORING-006` — non-portable-media-path
+- `AUTHORING-007` — stale-generated-artifact
+
+### ENGINE
+
+Use only when the Markdown contains a reasonable instruction and Markcut appears not to implement it correctly.
+
+---
+
+# Severity
+
+- `critical` — unusable or fails the core purpose
+- `high` — substantially damages viewer experience
+- `medium` — noticeable and worth fixing
+- `low` — minor improvement or polish
+
+---
+
+# Evidence Rules
+
+Every issue must contain concrete evidence.
+
+Reference, when possible:
+
+- scene name;
+- approximate timestamp;
+- narration;
+- visible content;
+- duration;
+- subtitle behavior;
+- Markdown instruction.
+
+Do not invent exact timestamps. Use approximate timestamps when necessary.
+
+Do not report the same underlying problem multiple times. Report the root problem and mention related symptoms in the evidence.
+
+---
+
+# Output Format
+
+Return the following structure.
+
+## Overall
+
+```yaml
+score: <1-5>
+summary: <short assessment>
+```
+
+## Scores
+
+```yaml
+storytelling: <1-5>
+visual_quality: <1-5>
+narration_visual_alignment: <1-5>
+timing_pacing: <1-5>
+audio: <1-5>
+subtitles: <1-5>
+platform_suitability: <1-5>
+markcut_authoring: <1-5>
+```
+
+Use the average of applicable dimensions for the overall score.
+
+## Issues
+
+For each meaningful issue:
+
+```yaml
+- id: VISUAL-001
+  severity: medium
+  source: AUTHORING
+  scene: "Solution"
+  timestamp: "approximately 00:18-00:25"
+  evidence: "..."
+  impact: "..."
+  recommendation: "..."
+  confidence: high
+```
+
+If there are no meaningful issues:
+
+```yaml
+issues: []
+```
+
+## Strengths
+
+List the most important things the video does well.
+
+## Priority Improvements
+
+List the three most valuable improvements, in priority order.
+
+---
+
+# Final Principle
+
+Answer this question above all others:
+
+> **If a user installed the Markcut skill and asked an LLM to create this video, did the skill enable the LLM to make good decisions?**
+
+Evaluate the actual viewer experience, use the Markdown to understand authoring intent, distinguish authoring mistakes from engine failures, and provide actionable evidence-based feedback.

package/tests/evals/README.md

@@ -0,0 +1,41 @@
+# markcut storyboard evals
+
+DeepEval suite that scores an agent authoring markcut storyboards from a
+video brief, using the markcut skill.
+
+## Components
+
+| File | Purpose |
+| --- | --- |
+| `openrouter_model.py` | Token-capped OpenRouter LLM wrapper (free-tier friendly, with model fallback rotation) |
+| `gen_dataset.py` | Generates goldens from scratch via the Synthesizer |
+| `dataset.json` | The golden dataset (committed, editable) |
+| `storyboard_app.py` | Traced app: runs `pi` CLI with the markcut skill to author a storyboard |
+| `metrics.py` | Judge metrics: TaskCompletion, Storyboard Format GEval, Story Narrative GEval |
+| `test_storyboard.py` | pytest traced single-turn evals |
+
+## Regenerate the dataset
+
+```bash
+.venv-evals/bin/python tests/evals/gen_dataset.py
+```
+
+## Run the evals
+
+```bash
+cd tests/evals && ../../.venv-evals/bin/deepeval test run test_storyboard.py \
+  --identifier "iterating-on-storyboard-authoring-round-1"
+```
+
+Each test runs the pi agent on the golden's video brief and asserts
+trace-level metrics. Failures typically indicate the agent violated a
+skill rule (e.g. `duration:` on scripted scenes, missing
+`isBackground:true`) or produced weak narrative structure.
+
+## Notes
+
+- Evaluation model runs on OpenRouter free models; the wrapper caps
+  max_tokens at 16000 to fit the workspace key's daily budget and
+  rotates through fallback models on rate limits.
+- Traces are local (Confident AI not enabled); view latest results via
+  `deepeval view --latest` offline report in `.deepeval/`.

package/tests/evals/dataset.json

@@ -0,0 +1,170 @@
+[
+    {
+        "input": "30s cinematic coastal drive vlog w/ voiceover detailing scenic landmarks & geography",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Detail how teaser reveals specs, competitive edge, and launch strategy.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Draft a 2‑speaker dialogue explaining quantum computing, with visual analogy and real‑world app.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Develop a 5‑min pasta recipe paired with lo‑fi background music for enhanced cooking experience",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "How did 1960s Ford integrate sustainability into its corporate brand story?",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Create a cinematic sci-fi thriller trailer, <2 min, featuring dystopian cityscape and AI antagonist",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Compare walkthrough of modern luxury villa vs historic estate: design, amenities, layout.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Compare high-energy cuts vs. slow cinematic pacing for fitness motivation shorts.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Produce a short historical documentary snippet on the Indus. Rev., focusing on inventions",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Compare dashboard walkthroughs in SW A and B tutorials, noting UI similarities and differences.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Compare a 15‑s skincare ad with strong CTA vs a 30‑s ad: effectiveness, engagement, conversion.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Describe how pressure adaptations let deep-sea creatures survive extreme conditions in the clip.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "30s cinematic coastal drive vlog w/ voiceover detailing scenic landmarks & geography",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Detail how teaser reveals specs, competitive edge, and launch strategy.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Draft a 2‑speaker dialogue explaining quantum computing, with visual analogy and real‑world app.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Develop a 5‑min pasta recipe paired with lo‑fi background music for enhanced cooking experience",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "How did 1960s Ford integrate sustainability into its corporate brand story?",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Create a cinematic sci-fi thriller trailer, <2 min, featuring dystopian cityscape and AI antagonist",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Compare walkthrough of modern luxury villa vs historic estate: design, amenities, layout.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Compare high-energy cuts vs. slow cinematic pacing for fitness motivation shorts.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Produce a short historical documentary snippet on the Indus. Rev., focusing on inventions",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Compare dashboard walkthroughs in SW A and B tutorials, noting UI similarities and differences.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Compare a 15‑s skincare ad with strong CTA vs a 30‑s ad: effectiveness, engagement, conversion.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    },
+    {
+        "input": "Describe how pressure adaptations let deep-sea creatures survive extreme conditions in the clip.",
+        "actual_output": null,
+        "expected_output": null,
+        "context": null,
+        "source_file": null
+    }
+]