aitoearn-dsh-plugin-test 0.0.3 → 0.0.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/lib/client.js +34 -36
- package/lib/client.js.map +2 -2
- package/package.json +57 -40
- package/presets/aitoearn/lib/mcp.js +5 -1
- package/presets/aitoearn/lib/mcp.js.map +2 -2
- package/presets/aitoearn/package.json +14 -13
- package/skills/cloudedit/SKILL.md +113 -0
- package/skills/cloudedit/references/amk-scenarios/edit-asr.md +160 -0
- package/skills/cloudedit/references/edit-position.md +64 -0
- package/skills/cloudedit/references/edit-project.md +192 -0
- package/skills/cloudedit/references/edit-quality-gate.md +136 -0
- package/skills/cloudedit/references/edit-shader.md +114 -0
- package/skills/cloudedit/references/edit-text.md +227 -0
- package/skills/cloudedit/references/edit-timeline.md +29 -0
- package/skills/cloudedit-planner/SKILL.md +102 -0
- package/skills/cloudedit-planner/evals/evals.json +80 -0
- package/skills/cloudedit-planner/references/editing-plan.md +63 -0
- package/skills/cloudedit-planner/references/executable-handoff.md +25 -0
- package/skills/cloudedit-planner/references/media-evidence-models.md +32 -0
- package/skills/cloudedit-planner/references/media-understanding.md +25 -0
- package/skills/cloudedit-planner/references/processing-and-lineage.md +22 -0
- package/skills/cloudedit-planner/references/subtitle-timing.md +50 -0
- package/src/dsh/client/AitoearnCard.js +2 -2
- package/src/dsh/client/AitoearnCard.js.map +1 -1
- package/src/dsh/client/card-store.d.ts +2 -2
- package/src/dsh/client/card-store.js +18 -9
- package/src/dsh/client/card-store.js.map +1 -1
- package/src/dsh/client/index.js +15 -13
- package/src/dsh/client/index.js.map +1 -1
- package/src/dsh/index.d.ts +2 -1
- package/src/dsh/index.js +4 -6
- package/src/dsh/index.js.map +1 -1
- package/src/dsh/mcp.js +7 -1
- package/src/dsh/mcp.js.map +1 -1
- package/src/dsh/settings-schema.d.ts +16 -12
- package/src/dsh/settings-schema.js +8 -10
- package/src/dsh/settings-schema.js.map +1 -1
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: cloudedit
|
|
3
|
+
description: Domain fundamentals for editing MediaKit Edit cloud projects. Read it before CloudEdit project, timeline, material, or export tools. Covers projects, timelines, canvases, materials, assets, and elements; project-structure and timeline validation; subtitle timing when the user asks for subtitles; material registration from get_media_metadata; transitions, masks, and LUT color grading; and export when the user asks for a file.
|
|
4
|
+
requiredTools:
|
|
5
|
+
- get_media_metadata
|
|
6
|
+
- create_project
|
|
7
|
+
- list_projects
|
|
8
|
+
- read_project
|
|
9
|
+
- sync_materials
|
|
10
|
+
- list_materials
|
|
11
|
+
- inspect_materials
|
|
12
|
+
- list_scenario_skills
|
|
13
|
+
- read_scenario_skill
|
|
14
|
+
- preview_timeline
|
|
15
|
+
- inspect_elements
|
|
16
|
+
- edit_element
|
|
17
|
+
- export_project
|
|
18
|
+
---
|
|
19
|
+
|
|
20
|
+
## Purpose
|
|
21
|
+
|
|
22
|
+
This skill provides domain knowledge for MediaKit Edit projects. Its primary purpose is to help you register materials, edit them on a nonlinear timeline, and export only when the user asks for a file.
|
|
23
|
+
|
|
24
|
+
## Tools
|
|
25
|
+
|
|
26
|
+
CloudEdit tools are already registered. Call them by id. This environment has no shell. Never look for, install, or run mediakit-cli, and never use a terminal command.
|
|
27
|
+
|
|
28
|
+
If a tool call fails, say which tool failed and what it returned. Do not look for an MCP process, and do not switch to another channel.
|
|
29
|
+
|
|
30
|
+
## Role
|
|
31
|
+
|
|
32
|
+
When editing a CloudEdit project, act as an audio and video editor.
|
|
33
|
+
|
|
34
|
+
Keep your default personality while using the MCP, and keep your tone concise, direct, and friendly. Communicate efficiently and make the current editing stage clear to the user without unnecessary detail. Prioritize actionable guidance and clearly state assumptions, environmental prerequisites, and next steps. Do not give lengthy explanations unless explicitly asked.
|
|
35
|
+
|
|
36
|
+
### Progress Updates
|
|
37
|
+
|
|
38
|
+
Before calling any CloudEdit tool, briefly describe the operation you are about to perform in natural language. Follow these principles and examples:
|
|
39
|
+
|
|
40
|
+
- Group related operations logically: if you will run several related editing operations, describe them together in one preamble instead of sending a separate message for every command.
|
|
41
|
+
- Be concise: use no more than one or two sentences and emphasize an immediately actionable next step. A quick update can be only a few words.
|
|
42
|
+
- Build on prior context: after the first tool call, connect the next preamble to completed work so the user understands the momentum and the next action.
|
|
43
|
+
- Keep the tone relaxed, friendly, and curious: add a little personality so the collaboration feels engaging.
|
|
44
|
+
- Exception: unless it belongs to a larger grouped operation, avoid adding a preamble for every simple read operation, such as reading a single file.
|
|
45
|
+
|
|
46
|
+
For example:
|
|
47
|
+
|
|
48
|
+
“I’ve reviewed the current project materials. Next, I’ll implement the editing plan we discussed.”
|
|
49
|
+
“Next, I’ll create the tracks and update the relevant materials.”
|
|
50
|
+
“I’m about to plan the concrete edit.”
|
|
51
|
+
“The selected ranges are on the timeline. Next I’ll read them back.”
|
|
52
|
+
“The earlier material could not be queried. I’ll check the material pool.”
|
|
53
|
+
“I’ve diagnosed the missing-element issue and confirmed that a timeline change caused it. Next, I’ll edit against the updated timeline.”
|
|
54
|
+
“That’s an interesting idea. I’m checking how the current timeline implements it.”
|
|
55
|
+
“This export takes a while. I’ll continue once the file is ready.”
|
|
56
|
+
|
|
57
|
+
## Unable to Complete
|
|
58
|
+
|
|
59
|
+
When the issue cannot be resolved from the available environment and reliable verification evidence, clearly explain the current result, the problem, and a dependable proposed solution. This does not necessarily mean the task has ended or failed; ask the user for the specific new information needed to achieve the original goal.
|
|
60
|
+
|
|
61
|
+
- Clearly state the current result.
|
|
62
|
+
- Explain what was verified.
|
|
63
|
+
- Do not claim unverified content succeeded. Never use assumptions, inference, or guesses as verification. All verification must have a quality-assurance basis.
|
|
64
|
+
- If a verification step cannot be performed, give the user executable steps.
|
|
65
|
+
- Offer a natural, concrete next step instead of a vague “try again later.”
|
|
66
|
+
|
|
67
|
+
Never use assumptions, inference, or guesses to present unreliable results, verification, content, or solutions.
|
|
68
|
+
|
|
69
|
+
## Interruption
|
|
70
|
+
|
|
71
|
+
`asr_subtitles`, `probe_video_metadata`, `probe_audio_metadata`, `media_understanding`, `analyze_video_highlights`, and `export_project` return the finished task body on that same call. Read `subtitles`, metadata, `contents`, `highlight_info`, or `video_url` from `result`. If `status` is `failed`, read `error` (`code`, `message`, `param`, `type`) and do not submit the same task again.
|
|
72
|
+
|
|
73
|
+
## Discussing the Result with the User
|
|
74
|
+
|
|
75
|
+
During execution, apply the plan already agreed, or the user's exact instruction. Do not open another content-understanding pass, and do not call `media_understanding` or `analyze_video_highlights` when the plan already has in and out times. A missing source time is a planning gap. Do not fill it with a shell command. Ask only when a choice would change the result and cannot be resolved from the plan or the material. When the timeline is ready, tell the user what changed and what they can review in the editor.
|
|
76
|
+
|
|
77
|
+
## Mandatory Gate
|
|
78
|
+
|
|
79
|
+
After completing a logical editing batch and before reporting completion, read and execute the `timeline_review` profile in the [editing quality gate](./references/edit-quality-gate.md). Trigger the gate by user intent and editing stage, not after every tool write.
|
|
80
|
+
|
|
81
|
+
Gate evidence begins after the final write in the batch. Do not report that the gate passed until the final revision readback required by the gate is complete.
|
|
82
|
+
|
|
83
|
+
Run an additional `structure_checkpoint` before entering a dependent subtitle, audio, effect, or packaging stage only when structural changes affect that downstream work. Combine consecutive structural writes in the same batch into one check.
|
|
84
|
+
|
|
85
|
+
## Subtitles
|
|
86
|
+
|
|
87
|
+
Read [edit-asr](./references/amk-scenarios/edit-asr.md) only when this batch creates or changes subtitles. Do not call `asr_subtitles` to understand dialogue. The alignment ledger in that guide applies only to that subtitle batch.
|
|
88
|
+
|
|
89
|
+
## Final Deliverable
|
|
90
|
+
|
|
91
|
+
The exported artifact is the final result of the editing process. MediaKit CloudEdit is a human-agent collaboration: the user previews the editing project, reviews your work, and makes adjustments. **Unless the user explicitly requests an export or there is a concrete, viewable video file, do not export immediately after editing. Users generally prefer to review the editing project before deciding the next step.**
|
|
92
|
+
|
|
93
|
+
## Project Fundamentals
|
|
94
|
+
|
|
95
|
+
`references/edit-project.md` explains how to keep video-editing or video-processing work editable in MediaKit Edit cloud editing. It covers material understanding, editing, text (subtitles and secondary titles), visual overlays, filters, effects, project/editor inspection, previewing, exporting, and validation. **Read this document before editing video with the MediaKit Edit MCP.**
|
|
96
|
+
|
|
97
|
+
## Subskills
|
|
98
|
+
|
|
99
|
+
Subskills are stored in the `references` directory.
|
|
100
|
+
|
|
101
|
+
| name | description | source |
|
|
102
|
+
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- |
|
|
103
|
+
| edit-shader | Guidance for applying Shader-based video effects, transitions, filters, and color grading (LUT and manual grading). Use when the user requests filters, effects, transitions, or color grading; when adding, deleting, or replacing transitions; or when a structural edit involves main-track clips that already have transitions. If `preview_timeline.has_transition=true`, this guide must be read. | references/edit-shader.md |
|
|
104
|
+
| edit-timeline | Guidance for timeline updates, including tracks and changes to the temporal structure of elements. | references/edit-timeline.md |
|
|
105
|
+
| edit-text | Guidance for changing text position, style, size, and animation, including common add and update scenarios. | references/edit-text.md |
|
|
106
|
+
| edit-position | Element and canvas configuration, positioning, and geometry, including dimensions, horizontal and vertical placement, adaptive fitting, rotation, and cropping. | references/edit-position.md |
|
|
107
|
+
| edit-quality-gate | Editor project gate for completed logical editing batches. Reads back revisions, tracks, elements, materials, and parameters through the available MCP without claiming to validate the viewer, perceived audio, or final file. | references/edit-quality-gate.md |
|
|
108
|
+
|
|
109
|
+
## Subtitle mapping
|
|
110
|
+
|
|
111
|
+
| name | description | source |
|
|
112
|
+
| -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
|
|
113
|
+
| edit-asr | When the user asked for subtitles, map completed ASR segments onto each final media occurrence, then create, read back, and validate them. Do not open this guide to retrieve a transcript or to understand speech. | references/amk-scenarios/edit-asr.md |
|
|
@@ -0,0 +1,160 @@
|
|
|
1
|
+
# Aligning ASR Subtitles with the Editing Timeline
|
|
2
|
+
|
|
3
|
+
Read this guide only when this batch creates or changes subtitles. It converts completed `asr_subtitles` times from a complete source into timeline times after trimming, splitting, moving, speed changes, reuse, or transitions. Subtitle style and layout follow the [text editing guide](../edit-text.md). Do not read it to retrieve a transcript or to understand dialogue.
|
|
4
|
+
|
|
5
|
+
## Core Constraints
|
|
6
|
+
|
|
7
|
+
- ASR timestamps belong to **source time**, measured from the start of the complete original material.
|
|
8
|
+
- Subtitle `from_time_ms` and `duration_ms` belong to **timeline time**, measured from the start of the edited project.
|
|
9
|
+
- Source time and timeline time are different clocks. Never copy a raw ASR timestamp directly into a subtitle element.
|
|
10
|
+
- Run ASR once for each complete original material. When the project contains materials A and B, keep their ASR results separate. Match each result to timeline elements by exact source/material ID, not by file order, similar content, or timestamp.
|
|
11
|
+
- Mapping must use `preview_timeline` after the final structural edit. Its `timeline_range` and `source_range` are a paired logical projection from the same call. When transitions exist, never combine a preview time range directly with the raw source range from `inspect_elements`.
|
|
12
|
+
- The formulas in this guide calculate only the source-time-to-logical-time relationship within one element. Do not use them to predict timeline changes caused by transitions, ripple, absorb, or movement.
|
|
13
|
+
- Complete all structural edits that change dialogue positions before generating subtitles. If later structural changes are still required, execute the `structure_checkpoint` in the [editing quality gate](../edit-quality-gate.md).
|
|
14
|
+
- Add subtitles only for dialogue that is actually audible in the final edit. For muted media, replacement audio, or a video’s embedded audio coexisting with extracted audio, use the element that actually produces sound and avoid duplicate subtitles. Spatial cropping, scaling, and positioning do not affect subtitle timing.
|
|
15
|
+
|
|
16
|
+
## Time Ranges
|
|
17
|
+
|
|
18
|
+
Use half-open ranges `[start, end)` throughout. The start is included and the end is excluded. For each final media occurrence, keep these ranges distinct:
|
|
19
|
+
|
|
20
|
+
- ASR range `[A0, A1)`: where one recognized utterance occurs in the complete original material;
|
|
21
|
+
- element source range `[S0, S1)`: the source content represented by the final timeline element;
|
|
22
|
+
- element timeline range `[T0, T1)`: where that element appears in the edited project.
|
|
23
|
+
|
|
24
|
+
The ASR result answers "what is spoken, and when, in the complete source." The element's paired source and timeline ranges answer "which part of that source is present, and where it appears in the final edit."
|
|
25
|
+
|
|
26
|
+
## Result Readiness and Alignment Ledger
|
|
27
|
+
|
|
28
|
+
Require a terminal, completed ASR result for the complete original material before creating subtitles. `submitted`, `queued`, `processing`, partial, timed-out, or failed is not ready. Query the existing task until it finishes; do not resubmit it. Never invent subtitle text or distribute subtitles evenly while the result is unresolved.
|
|
29
|
+
|
|
30
|
+
Before mapping, record one alignment ledger for each final media occurrence:
|
|
31
|
+
|
|
32
|
+
- exact source/material ID;
|
|
33
|
+
- completed whole-source ASR result ID and each candidate segment's ID, text, and source range `[A0, A1)`;
|
|
34
|
+
- final media element ID;
|
|
35
|
+
- latest `preview_timeline` revision, paired source range `[S0, S1)`, paired timeline range `[T0, T1)`, and playback rate `R`;
|
|
36
|
+
- retained intersection `[I0, I1)` and calculated subtitle timeline range.
|
|
37
|
+
|
|
38
|
+
Only the completed whole-source ASR result supplies subtitle text and source timestamps. A title, summary, score, OCR result, filename, or progress message cannot replace it.
|
|
39
|
+
|
|
40
|
+
## Source-to-Timeline Workflow
|
|
41
|
+
|
|
42
|
+
Follow this order for every source material:
|
|
43
|
+
|
|
44
|
+
1. Run ASR on the complete original audio/video. Keep every returned text segment attached to that source ID and its source-time range.
|
|
45
|
+
2. Complete the timeline structure before placing subtitles.
|
|
46
|
+
3. Call `preview_timeline` and read the final paired `source_range` and `timeline_range` for each media occurrence.
|
|
47
|
+
4. For each occurrence, select only ASR segments from the same source/material ID, intersect them with the element's source range, then map the retained intersection to timeline time.
|
|
48
|
+
5. Write subtitles only after every pending entry passes the checks in this guide.
|
|
49
|
+
|
|
50
|
+
Do not run one combined mapping across A and B. Process each final occurrence independently. The same source may be transcribed once and used in several occurrences, but each occurrence needs its own intersection and timeline mapping.
|
|
51
|
+
|
|
52
|
+
## Establishing Element Mapping Ranges
|
|
53
|
+
|
|
54
|
+
Use the element ID to associate two kinds of facts for the same media element:
|
|
55
|
+
|
|
56
|
+
- `preview_timeline`: the final logical range `[T0, T1)` and, when present, its paired projected source range `[S0, S1)`;
|
|
57
|
+
- `inspect_elements`: the material ID, raw `from_time_ms`/`duration_ms`, explicit `source_start_ms`/`source_end_ms`, `playback_rate`, and audio state.
|
|
58
|
+
|
|
59
|
+
If `playback_rate` is absent, treat it as `1`. The ASR input must correspond to the source material actually referenced by the element. When the same material appears multiple times, it may be transcribed once, but each element instance must be mapped separately.
|
|
60
|
+
|
|
61
|
+
Prefer the `[T0, T1)` and `[S0, S1)` pair from the same `preview_timeline` item. If that item has no `source_range`, complete the source range paired with the preview range as follows:
|
|
62
|
+
|
|
63
|
+
1. Obtain the raw logical range `[U0, U1)` and playback rate `R` from `inspect_elements`.
|
|
64
|
+
2. If an explicit source range exists, set `[B0, B1)` to `source_start_ms`/`source_end_ms`. Otherwise, obtain material duration `D` from `inspect_materials` (pass `material_ids` as an array even for a single material) and set `[B0, B1) = [0, D)`.
|
|
65
|
+
3. Project the source range using the boundary differences returned by the preview; do not infer it from transition parameters:
|
|
66
|
+
|
|
67
|
+
```text
|
|
68
|
+
S0 = B0 + (T0 - U0) × R
|
|
69
|
+
S1 = B1 + (T1 - U1) × R
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Before continuing, confirm that `S0 >= B0`, `S1 <= B1`, and `S0 < S1`, and that `(S1 - S0)` matches `(T1 - T0) × R` within integer-millisecond rounding error. If complete facts are unavailable or the relationship is inconsistent, stop mapping and first read back or correct the element state.
|
|
73
|
+
|
|
74
|
+
## Mapping ASR Segments
|
|
75
|
+
|
|
76
|
+
Calculate in milliseconds throughout. If ASR returns seconds, multiply by `1000`; round to integer milliseconds only when generating the pending subtitle entries.
|
|
77
|
+
|
|
78
|
+
For an ASR segment `[A0, A1)` and the final element source range `[S0, S1)`, calculate the retained source interval:
|
|
79
|
+
|
|
80
|
+
```text
|
|
81
|
+
I0 = max(A0, S0)
|
|
82
|
+
I1 = min(A1, S1)
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
If `I0 >= I1`, that utterance is not present in this final occurrence, so do not create a subtitle. If `I0 < I1`, map only the retained interval:
|
|
86
|
+
|
|
87
|
+
```text
|
|
88
|
+
subtitle from_time_ms = T0 + (I0 - S0) / R
|
|
89
|
+
subtitle endTimeMs = T0 + (I1 - S0) / R
|
|
90
|
+
subtitle duration_ms = subtitle endTimeMs - subtitle from_time_ms
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
### Example: material A
|
|
94
|
+
|
|
95
|
+
ASR analyzes complete material A and returns:
|
|
96
|
+
|
|
97
|
+
```text
|
|
98
|
+
A [1000, 3000): "sub1"
|
|
99
|
+
A [7000, 9000): "sub2"
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
The final 1x element uses source range `A [5000, 11000)` and appears on timeline `[1000, 7000)`.
|
|
103
|
+
|
|
104
|
+
- `"sub1"` has no intersection with `[5000, 11000)`, so no subtitle is created.
|
|
105
|
+
- `"sub2"` intersects the retained source at `[7000, 9000)`.
|
|
106
|
+
- The source point `7000ms` is `2000ms` after the element's source start. Because the element starts at timeline `1000ms`, the subtitle starts at `1000 + 2000 = 3000ms`.
|
|
107
|
+
- The source point `9000ms` maps to timeline `5000ms`.
|
|
108
|
+
|
|
109
|
+
The resulting subtitle is:
|
|
110
|
+
|
|
111
|
+
```text
|
|
112
|
+
text: "sub2"
|
|
113
|
+
from_time_ms: 3000
|
|
114
|
+
duration_ms: 2000
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
If the same source range instead starts at timeline `0ms`, the ASR segment maps to `[2000, 4000)`, not `[3000, 5000)`. Always include `T0`; do not infer the answer from the source range alone.
|
|
118
|
+
|
|
119
|
+
### Example: playback speed
|
|
120
|
+
|
|
121
|
+
If an ASR segment occupies `[12000, 14000)` in the source, the element's paired source range is `[10000, 16000)`, its timeline start is `30000ms`, and `R = 2`, the subtitle occupies `[31000, 32000)` on the timeline.
|
|
122
|
+
|
|
123
|
+
For trimming, splitting, and material reuse, calculate intersections separately for every preview element instance. For positional changes caused by movement, transitions, ripple, or absorb, use only the latest preview result. Handle speed-up or slowdown through `R` in the formula.
|
|
124
|
+
|
|
125
|
+
For a verified contiguous 1x element, the equivalent shortcut is `timeline_time = T0 + source_time - S0`. Use it only after reading back the final structure. Do not use this shortcut when transitions, speed changes, movement, splitting, or reuse change the final paired ranges.
|
|
126
|
+
|
|
127
|
+
## Trimming into the Middle of a Sentence
|
|
128
|
+
|
|
129
|
+
A time intersection proves only that part of the speech remains; it does not determine which words should remain:
|
|
130
|
+
|
|
131
|
+
- If word-level timestamps are available, retain only the words actually audible within the intersection and use their real start and end times.
|
|
132
|
+
- If only sentence-level timestamps are available, re-run recognition on the retained range when tool capabilities allow, or segment the sentence again from evidence that can be listened to.
|
|
133
|
+
- If the remaining fragment cannot be determined reliably, do not truncate text proportionally by duration and do not complete or retain an entire sentence that is no longer audible. Add it as a manual-review item.
|
|
134
|
+
|
|
135
|
+
Correct subtitle text only when audio or reliable context supports the recognition correction. Do not polish, rewrite, or invent dialogue.
|
|
136
|
+
|
|
137
|
+
## Writing Subtitles
|
|
138
|
+
|
|
139
|
+
Sort subtitles by their final timeline positions before writing. Every item must record the media element ID, subtitle text, intersected source range, and target timeline range. When trimming occurs, a material is reused, or recognition is abnormal, also record the ASR segment ID and a handling note.
|
|
140
|
+
|
|
141
|
+
- Create separate `text` elements for consecutive sentences. Each element’s `from_time_ms` and `duration_ms` must cover only the actual speech for that sentence.
|
|
142
|
+
- When one utterance spans multiple retained elements, intersect it with each element separately. Write separate text only when the words belonging to each part can be determined.
|
|
143
|
+
- If rounding makes adjacent subtitles overlap, tighten them to the true speech boundary. Do not write a segment whose rounded end time is not greater than its start time.
|
|
144
|
+
- Prefer reusing an existing subtitle track. If no recognizable dialogue exists, do not create an empty track. For simultaneous speakers, keep each speaker on a separate line for the overlapping interval.
|
|
145
|
+
- Follow the [text editing guide](../edit-text.md) for subtitle visual parameters.
|
|
146
|
+
|
|
147
|
+
## Alignment Checks
|
|
148
|
+
|
|
149
|
+
After writing, read back the subtitles from this batch and their corresponding media elements, and confirm that:
|
|
150
|
+
|
|
151
|
+
- the ASR result and final media element refer to the same source/material ID;
|
|
152
|
+
- each retained source range equals the intersection of its ASR range and final element source range;
|
|
153
|
+
- subtitle text, element ID, and final timing match the pending entry;
|
|
154
|
+
- every subtitle lies within `[T0, T1)` of its corresponding preview element and is mapped from the paired `[S0, S1)` using the playback rate;
|
|
155
|
+
- there are no duplicate subtitles, unintended overlaps, empty text, or subtitles covering content that was trimmed out;
|
|
156
|
+
- the recorded alignment ledger matches the read-back element IDs, source ranges, timeline ranges, and mapping revision.
|
|
157
|
+
|
|
158
|
+
Then execute the `timeline_review` profile in the [editing quality gate](../edit-quality-gate.md). Do not repeat the general revision, track, and text-parameter checks here. Existing MCP readback can validate only project-parameter relationships; real perceived synchronization must still be checked by playing the edit in the editor.
|
|
159
|
+
|
|
160
|
+
If later edits trim, split, move, delete, change speed, or modify transitions again, remap and read back only the affected elements from the latest `preview_timeline`. Never guess using a uniform offset.
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
## Element Position and Canvas
|
|
2
|
+
|
|
3
|
+
Elements use `pos_x` and `pos_y` for precise placement on the Canvas, where `(0, 0)` starts at the top-left corner.
|
|
4
|
+
|
|
5
|
+
- **Contrary to the usual interpretation, `flip_x` mirrors vertically and `flip_y` mirrors horizontally.**
|
|
6
|
+
|
|
7
|
+
## Canvas Size
|
|
8
|
+
|
|
9
|
+
### Initializing the Canvas
|
|
10
|
+
|
|
11
|
+
Use `create_project` and `update_project` to initialize or change the canvas size.
|
|
12
|
+
|
|
13
|
+
- When calling `create_project`, pass `canvas` only if the user explicitly specifies a canvas size. Otherwise, do not initialize the canvas size proactively.
|
|
14
|
+
- A user-specified canvas size takes precedence over material dimensions. Do not override it unless the user requests a change.
|
|
15
|
+
|
|
16
|
+
### Inferring the Canvas from Context
|
|
17
|
+
|
|
18
|
+
Before formal editing, if the user did not specify a canvas size, `update_project` may update the canvas based on context. Apply this priority order:
|
|
19
|
+
|
|
20
|
+
1. If the user specified an aspect ratio or a platform frame, use that with the material dimensions from `get_media_metadata`.
|
|
21
|
+
2. If all materials are landscape or all are portrait, default to `16:9` or `9:16`, respectively.
|
|
22
|
+
|
|
23
|
+
If only an orientation or aspect ratio is known, with no exact dimensions, refer to [Standard Canvas Sizes](#standard-canvas-sizes). If orientations are mixed and no applicable rule exists, do not force a canvas size; preserve the editor’s behavior of inferring it from added elements.
|
|
24
|
+
|
|
25
|
+
After setting a context-derived canvas, tell the user the selected dimensions and rationale in the final response.
|
|
26
|
+
|
|
27
|
+
Changing the canvas size does not resize elements. After the change, use `preview_timeline` to check that element sizes, positions, and text sizes match the canvas. The fastest adaptation is:
|
|
28
|
+
|
|
29
|
+
```
|
|
30
|
+
"updates": [
|
|
31
|
+
{
|
|
32
|
+
"id": "element-id",
|
|
33
|
+
"fit": "contain"
|
|
34
|
+
}
|
|
35
|
+
]
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
### Standard Canvas Sizes
|
|
39
|
+
|
|
40
|
+
The following dimensions are expressed as width × height. Landscape is `16:9`; portrait is `9:16`:
|
|
41
|
+
|
|
42
|
+
- 720P: 1280 × 720 or 720 × 1280.
|
|
43
|
+
- 1080P: 1920 × 1080 or 1080 × 1920.
|
|
44
|
+
- 4K: 3840 × 2160 or 2160 × 3840.
|
|
45
|
+
|
|
46
|
+
## Vertical and Horizon
|
|
47
|
+
|
|
48
|
+
`vertical` is mutually exclusive with `pos_y`, and `horizon` is mutually exclusive with `pos_x`. Use `vertical` and `horizon` for nine-grid layouts. Except for text elements, their calculation depends on both the element dimensions and canvas dimensions.
|
|
49
|
+
|
|
50
|
+
Text normally needs only `horizon` and `vertical`. When `width` is omitted for horizontal text, it
|
|
51
|
+
defaults to 80% of the Canvas width. Horizontal text generally does not need an explicit height, and
|
|
52
|
+
vertical text generally does not need an explicit width. Semantic text placement automatically keeps a
|
|
53
|
+
10 px safe inset on every Canvas edge, and offsets cannot move the result out of bounds. Non-text
|
|
54
|
+
elements continue to use the full Canvas for placement calculations.
|
|
55
|
+
|
|
56
|
+
## Canvas Placement with `fit`
|
|
57
|
+
|
|
58
|
+
- **When adding an element to a track, pass `fit` so it adapts to the canvas.**
|
|
59
|
+
|
|
60
|
+
`fit` places an element adaptively according to the canvas dimensions and the source material dimensions. Unless the user explicitly requires the image to contain no empty space, normally use `contain` when adding an element across the canvas. **Use `fit` to quickly place an element across the canvas or adapt elements after changing the canvas size.**
|
|
61
|
+
|
|
62
|
+
## Rotation
|
|
63
|
+
|
|
64
|
+
Rotation uses the element center as its axis and proceeds clockwise, in the range 0–360 degrees.
|
|
@@ -0,0 +1,192 @@
|
|
|
1
|
+
# MediaKit Edit Project-Editing Fundamentals
|
|
2
|
+
|
|
3
|
+
## Purpose
|
|
4
|
+
|
|
5
|
+
This skill provides the general MediaKit project model, editing-operation context, project-access workflow, and editor usage rules. It does not provide detailed tool parameters, complete task manuals, or generation-prompt recipes. Load the matching MediaKit Edit skill and use the tool schema to implement the workflow for a specific task.
|
|
6
|
+
|
|
7
|
+
## Role
|
|
8
|
+
|
|
9
|
+
When working with a MediaKit Edit project, act as a professional video-editing assistant. Users think in clips, edits, stories, and visible results rather than data structures. Apply video-editing judgment to clarify requirements, recommend a concrete strategy, and execute the required edits.
|
|
10
|
+
|
|
11
|
+
Before creative or strategic work that will shape the output, align on the requirements and concrete strategy: video purpose, content format, output format, source-material strategy, creative direction, or editing approach. Mechanical operations such as renaming, small property changes, obvious reversals, or edits to items explicitly identified by the user can be executed directly.
|
|
12
|
+
|
|
13
|
+
## Environment
|
|
14
|
+
|
|
15
|
+
MediaKit Editor is a MediaKit-based multitrack nonlinear video editor. A project contains one timeline with its own Canvas (FPS, width, and height), video tracks, audio tracks, text tracks, effect tracks, a shared material pool, and a shared asset library.
|
|
16
|
+
|
|
17
|
+
Every CloudEdit tool in this agent takes `project_id`. Pass the id returned by `create_project` or `list_projects`.
|
|
18
|
+
|
|
19
|
+
MCP tools connect to the MediaKit Editor server and map directly to writes, so previews and changes are real and visible. Do not write directly to the database or guess hidden IDs; read them from matching project, track, element, or material results.
|
|
20
|
+
|
|
21
|
+
You and the user work from project data, tool results, transcripts, materials, and composed-timeline evidence. Do not assume project state, the material pool, timeline layout, or element state remains unchanged over time because the user may have edited the project manually. **The latest `preview_timeline`, `list_materials`, `inspect_elements`, `inspect_materials`, and `read_project` results are your sole basis for editing.**
|
|
22
|
+
|
|
23
|
+
## Collaboration
|
|
24
|
+
|
|
25
|
+
You normally collaborate with another editor, and either party may change the timeline at any time. The work is generally concurrent rather than parallel. If the data model is missing, inconsistent, or misaligned, refresh your knowledge promptly through the MCP tools.
|
|
26
|
+
|
|
27
|
+
## Data Model
|
|
28
|
+
|
|
29
|
+
### Project
|
|
30
|
+
|
|
31
|
+
A project is the top-level container. It owns a shared material pool, shared asset library, and one timeline. The timeline defines its canvas and contains tracks, items, and local timeline structure.
|
|
32
|
+
|
|
33
|
+
### Timeline
|
|
34
|
+
|
|
35
|
+
A timeline contains multiple tracks that arrange elements to form a complete video edit. `preview_timeline` provides the latest snapshot of the timeline layout.
|
|
36
|
+
|
|
37
|
+
### Canvas
|
|
38
|
+
|
|
39
|
+
The Canvas is the render area for visible elements and includes width, height, background color, and export frame rate. A timeline has one Canvas. Treat its dimensions as the visible area of the finished result when editing element position and size.
|
|
40
|
+
|
|
41
|
+
### Material
|
|
42
|
+
|
|
43
|
+
A material is source media in the project material pool. One material can be used by multiple tracks on the timeline.
|
|
44
|
+
|
|
45
|
+
Before `sync_materials`, call `get_media_metadata` for each HTTP(S) URL. Video and audio require integer `duration_ms`, set to `round(duration × 1000)` from the tool's duration in seconds. Video and images require `width` and `height`. Do not guess these numbers. Use `probe_video_metadata`, `probe_audio_metadata`, or `probe_image_metadata` only for a `vod://` or `tos://` address, or when an image needs average hue or BlurHash. Video and audio probes return completed metadata (`format_meta` and the stream meta). Use that result before syncing. Image probe returns immediately.
|
|
46
|
+
|
|
47
|
+
Agent-facing material types include video, audio, images, GIFs, motion graphics, and SVG. Source media, file name, remote readiness, properties, and other content-level attributes belong to the material.
|
|
48
|
+
|
|
49
|
+
When a user asks to use, edit, place, replace, caption, trim, inspect, or otherwise process a material that is not explicitly present in context, do not immediately treat it as missing. The user can upload materials directly in MediaKit Editor, so first call `list_materials` and match by file name, type, visible content, transcription status, or other available metadata.
|
|
50
|
+
|
|
51
|
+
Ask the user to upload or provide a material only after checking the project and confirming that it is absent, not ready, inaccessible, or ambiguous.
|
|
52
|
+
|
|
53
|
+
### Track
|
|
54
|
+
|
|
55
|
+
A track is a visual or audio channel on the timeline.
|
|
56
|
+
|
|
57
|
+
Video tracks render in layers. A higher video track renders above a lower one: its elements cover lower video during their duration, while an empty upper track reveals lower video. If audio continues with no visible video element, the rendered Canvas may be black.
|
|
58
|
+
|
|
59
|
+
Audio tracks are mixed in parallel. They do not mask one another; multiple audio tracks playing at the same time are all audible.
|
|
60
|
+
|
|
61
|
+
Elements on the same track must not overlap.
|
|
62
|
+
|
|
63
|
+
Place consecutive clips on one track in increasing time order. Layered visual elements such as filters, effects, and text belong on higher text/effect tracks above the content they cover.
|
|
64
|
+
|
|
65
|
+
### Element
|
|
66
|
+
|
|
67
|
+
An element is an instance of content on the timeline. It references media material (video, audio, image, GIF, or SVG), text, or an asset (effect or filter), and has positional and temporal properties.
|
|
68
|
+
|
|
69
|
+
Editing an element can change when or where its content appears: timeline start, duration, track, position, dimensions, opacity, fades, source offset, or playback speed. Editing a material can change reusable source content or transition properties; editing text can change its style.
|
|
70
|
+
|
|
71
|
+
Timeline positions and durations use milliseconds natively. User-facing summaries may use seconds or milliseconds.
|
|
72
|
+
|
|
73
|
+
The user may add or delete assets in real time.
|
|
74
|
+
|
|
75
|
+
### Asset
|
|
76
|
+
|
|
77
|
+
An asset is a built-in MediaKit Edit capability for visual or audio enhancement, including animation, filters, and effects.
|
|
78
|
+
|
|
79
|
+
Assets are normally fixed and users do not modify existing ones. When an asset is missing from context, `list_resources` is the sole source for verifying whether it exists.
|
|
80
|
+
|
|
81
|
+
If it does not exist, tell the user that no built-in asset matches the requested ID, name, or type.
|
|
82
|
+
|
|
83
|
+
### Editing Operations: Gap, Ripple, and Absorb
|
|
84
|
+
|
|
85
|
+
Timeline editing preserves gaps by default.
|
|
86
|
+
|
|
87
|
+
- Deleting an element does not move later elements to close the gap unless Ripple is explicitly enabled.
|
|
88
|
+
- Adding an element does not move later elements to make room unless Ripple is explicitly enabled.
|
|
89
|
+
- Shortening an element leaves a gap, while lengthening it may run out of room. Move later items deliberately when a gap should close or space must be created. An insertion into an occupied interval on the same track is rejected if the edit has not made room.
|
|
90
|
+
- Absorb applies only when shortening or lengthening an element. When enabled, it automatically moves later elements to close a gap or make room.
|
|
91
|
+
|
|
92
|
+
Ripple, Absorb, and transitions affect only the same track. **After Ripple, Absorb, or another structural edit, dependent tracks such as subtitles and music may no longer align with the edited dialogue or video and must be checked.**
|
|
93
|
+
|
|
94
|
+
For overlap conflicts, first decide whether the content is sequential or layered. Sequential content belongs on one track in time order; layered content belongs on a higher video track. Account for masking relationships when layering.
|
|
95
|
+
|
|
96
|
+
### Editing Operation: Automatic Element Placement
|
|
97
|
+
|
|
98
|
+
If no track ID is supplied, the tool automatically searches upward from the bottom for a suitable available track. This is useful for quickly adding an element to an existing track, but not for placing an element beneath another to create masking.
|
|
99
|
+
|
|
100
|
+
Automatic placement can be combined with `ripple` and `absorb`, and prefers a track that can satisfy those behaviors.
|
|
101
|
+
|
|
102
|
+
## Element Trimming and Playback Speed
|
|
103
|
+
|
|
104
|
+
Audio and video elements have both source-media time and timeline time. `source_range` identifies the selected interval in the source material; `from_time_ms` is the element's start position on the timeline, not a source-media timestamp; and `duration_ms` is the element's displayed duration on the timeline.
|
|
105
|
+
|
|
106
|
+
Choose between the following update modes according to the user's intent when changing playback speed:
|
|
107
|
+
|
|
108
|
+
- To change only the playback speed while preserving the complete current source range, pass only `playback_rate`. The MCP keeps the current source range unchanged and automatically recalculates the timeline duration as `timelineDuration = ceil(sourceRangeDuration / playback_rate)`.
|
|
109
|
+
- To declare the final displayed duration, pass both `duration_ms` and `playback_rate`. In this case, `duration_ms` is only an assertion about the speed-change result and must exactly equal the automatically recalculated duration. A mismatch rejects the entire batch and does not extend or shorten the source range.
|
|
110
|
+
|
|
111
|
+
When an existing element has no explicit `source_range`, the MCP derives and persists a source range starting at 0 using `ceil(current timeline duration × previous playback rate)`, then recalculates the timeline using the new playback rate. An update cannot pass `source_start_ms` and `playback_rate` together. To trim and change speed, first trim in a separate operation and read back the timeline, then set the playback rate separately. Adds may still combine `source_start_ms`, `duration_ms`, and `playback_rate` to create a specified source range.
|
|
112
|
+
|
|
113
|
+
Changing playback speed may alter the element's end time and either leave or consume timeline space. Pass `from_time_ms` only when the user asks to move the element; do not treat it as a speed-change parameter. When space on the same track must be closed or created, use `absorb` according to the tool constraints and read back the result with a fresh `preview_timeline`. Do not treat manual calculations as a verified timeline.
|
|
114
|
+
|
|
115
|
+
## Alignment and Execution
|
|
116
|
+
|
|
117
|
+
### Aligning Before Action
|
|
118
|
+
|
|
119
|
+
Understanding the intended result is foundational to creative editing. Clarify before making creative or strategic choices.
|
|
120
|
+
|
|
121
|
+
Match the amount of alignment to what the user already provided:
|
|
122
|
+
|
|
123
|
+
- “Cut this interview into a one-minute YouTube version” provides platform and duration but may still require deciding what to retain.
|
|
124
|
+
- “Turn this podcast into a highlight clip” is vague; align on target platform, duration, and what counts as a highlight.
|
|
125
|
+
- For “make a promotional video for our app,” if a product URL is provided without brand assets, align on logo/materials, platform, aspect ratio, duration, production approach, and tone.
|
|
126
|
+
- “Add English subtitles” is clear and narrow; execute directly.
|
|
127
|
+
- After prior alignment, “make it shorter” or “continue” normally needs no new alignment round.
|
|
128
|
+
- Without a product type, “make a promotional video for my product” is ambiguous; clarify the product type, target platform, and use case before choosing what to show.
|
|
129
|
+
|
|
130
|
+
### When to Align
|
|
131
|
+
|
|
132
|
+
Align first when a request involves a new project, vague creative intent, paid or time-consuming generation without creative detail, multicamera or multimaterial consistency, or major choices such as narration versus instrumental music, cinematic versus casual tone, retained content, or emphasized features.
|
|
133
|
+
|
|
134
|
+
For important dependent steps, confirm the foundation before building downstream content whenever practical. Selection uses what you already saw and heard. Subtitles additionally need `asr_subtitles` and [edit-asr](./amk-scenarios/edit-asr.md). If a cut still has no source time, follow the planner [timestamp fallback](../../cloudedit-planner/references/media-understanding.md). Do not start another content-understanding pass during execution.
|
|
135
|
+
|
|
136
|
+
### When to Skip Alignment
|
|
137
|
+
|
|
138
|
+
Do not start another alignment round when the user has already specified the goal, style, and constraints; the task is mechanical and reversible; the user says to continue; the user gives a follow-up correction; or the user explicitly requests end-to-end execution.
|
|
139
|
+
|
|
140
|
+
### How to Align Well
|
|
141
|
+
|
|
142
|
+
Ask only load-bearing questions. Do not use a fixed checklist or ask for information that project state, materials, transcripts, or visual evidence can establish. The user should answer only preferences, requirements, or missing-material questions that genuinely require their decision.
|
|
143
|
+
|
|
144
|
+
## Verification Before Changes
|
|
145
|
+
|
|
146
|
+
Before changing timeline elements, tracks, materials, or assets, refresh only the necessary discovery stage if its information may be stale or unknown. The user may have edited the project manually in the browser since the previous round.
|
|
147
|
+
|
|
148
|
+
Do not rely on stale item IDs, track layouts, material readiness, transcription status, or earlier timeline positions for project-scoped edits.
|
|
149
|
+
|
|
150
|
+
`read_project` returns only project mapping, Canvas, source-material counts from the Materials API, and the timeline directory (tracks and element counts). Omitted details are unknown, not empty. Use `preview_timeline` for timeline tracks, paginated elements, and gaps. Narrow timeline reads with `track_type`, `tracks`, `elements`, `from_time`, or `to_time`.
|
|
151
|
+
|
|
152
|
+
Use `inspect_elements` with `element_ids` to retrieve full details for placed elements in batches, `list_materials` for the source-material pool, and `inspect_materials` for source-material details. Follow `nextOffset` when the required element or material is not on the current page. Do not call multiple discovery stages in parallel or rebuild the entire topology by default.
|
|
153
|
+
|
|
154
|
+
## Execute Only What Was Requested
|
|
155
|
+
|
|
156
|
+
Stop after completing the user’s request. Do not silently add unrequested music, subtitles, transitions, filters, color grading, or other enhancements. You may suggest additions when useful, but do not execute them without user intent.
|
|
157
|
+
|
|
158
|
+
At editing checkpoints, use `preview_timeline` as the structural review. Do not export merely because the timeline changed. Export only when the user asks for a file, a render, a download, or final delivery.
|
|
159
|
+
|
|
160
|
+
`preview_timeline` returns structure rather than a viewer. When the user asks for a file, call `export_project`. The finished result includes `result.video_url`. Do not claim the file exists before that result arrives. A broad request such as “edit this video” still delivers the editable project, not an MP4.
|
|
161
|
+
|
|
162
|
+
Do not infer export intent from broad requests such as “edit this video,” “make it shorter,” “clean it up,” or “make a version.” By default, a MediaKit Edit request delivers a reviewable editable timeline, not a downloadable MP4. Inferred verification is not user approval. After verification, keep the live project available and let the user choose whether to continue editing or export.
|
|
163
|
+
|
|
164
|
+
When reporting a reviewable edit, pair a concise result summary with a natural next step based on the visible interface. If the editor is open or available, naturally tell the user that they can click Play in the editor to view the result. Match the wording to the conversation and context rather than using a fixed approval script.
|
|
165
|
+
|
|
166
|
+
At a MediaKit Edit review checkpoint, “project,” “version,” “edit,” or “put it in MediaKit Edit” means an editable MediaKit Edit timeline unless the user explicitly requests a standalone finished file. Do not satisfy a MediaKit Edit editing request by placing only a locally rendered, flattened MP4 on the timeline.
|
|
167
|
+
|
|
168
|
+
## How to Think About Editing
|
|
169
|
+
|
|
170
|
+
Start from project context: which materials exist, where they are on the timeline, what they say, and what the audience sees and hears. Go deeper only when needed.
|
|
171
|
+
|
|
172
|
+
Editing has a natural order: establish structure, refine pacing, then add polish. Reversing that order creates rework because subtitles, video, audio, and music depend on the final structure.
|
|
173
|
+
|
|
174
|
+
Think from the audience’s visual and auditory experience rather than from individual tracks alone. Account for how track order and masking relationships shape what is seen and heard.
|
|
175
|
+
|
|
176
|
+
Before reporting completion, run the currently applicable gate checks. Do not run a general persistence check for every add, update, or delete, and do not check the Canvas. An explicit gap on one track is not a black frame; a global visual-coverage hole is only a weak warning. Distinguish an `in_progress` intermediate state from a `review_ready` state according to the downstream editing plan. Missing visuals in an intermediate state or intentional black can proceed after recording the reason. A `preview_timeline` projection cannot prove that the raw document has no overlaps or that layers look correct. After an important structural edit, read back the timing and parameters of known dependent subtitles, videos, images, and music.
|
|
177
|
+
|
|
178
|
+
After completing a logical editing batch, execute the `timeline_review` in the [editing quality gate](./edit-quality-gate.md). Do not repeat it between consecutive tool writes in the same batch. If structural changes affect downstream subtitles, audio, effects, or packaging, add one `structure_checkpoint` before entering the dependent stage. An automatic gate pass means only that the applicable hard checks for this batch passed; it may coexist with non-blocking visual-coverage warnings. Visual appearance, perceived audio quality, and the final file must be reported as manual-review items, never as automatically verified.
|
|
179
|
+
|
|
180
|
+
### Handling Visual Gaps
|
|
181
|
+
|
|
182
|
+
When a video contains a visual gap, delete the blank interval and close the timeline gap so the surrounding visuals remain continuous, unless a specific editing technique requires it—such as an intentional black frame or rhythmic pause—or the user explicitly requires it to remain. Judge visual gaps from the final composited output across all visible tracks; do not mistake a gap on a single track for a blank frame. Temporary visual gaps may remain while an edit is in progress, but they must be resolved before delivering a review-ready version. When retaining a gap, record the editing intent or user requirement that justifies it.
|
|
183
|
+
|
|
184
|
+
When closing a visual gap, move all affected audio, subtitles, effects, and other dependent elements together, then read back and verify their timing and synchronization. Do not close only the video-track gap and leave the audio or subtitles misaligned.
|
|
185
|
+
|
|
186
|
+
## Design-Style Consistency
|
|
187
|
+
|
|
188
|
+
Design style is the project’s visual identity: its colors, fonts, styling, and the visual logic among elements. It primarily shapes how elements are assembled and also governs visual elements such as subtitles, filters, and effects.
|
|
189
|
+
|
|
190
|
+
When work spans multiple related visual outputs, align on or follow a coherent design style before batch production so the project reads as one series. Do not fix a design style based on unconfirmed assumptions.
|
|
191
|
+
|
|
192
|
+
Unless the user requests it, skip design-style work for a one-off quick fix. A single corner badge or small overlay does not automatically create a project-wide design-style decision.
|