@minicor/mcp-server 4.7.0 → 4.9.0

package/skills/general/blueprint-vm-walkthrough.md

@@ -0,0 +1,162 @@
+---
+name: blueprint-vm-walkthrough
+description: Run a guided VM walkthrough session that teaches a blueprint - the user narrates a process on a live VM while you capture screenshots, ground UI boundaries, and persist everything as blueprint library entries (walkthrough + screenshots + screen-maps + anchors). Use when a user wants to walk you through a process on a VM to build or enrich a blueprint.
+category: general
+tags: [blueprint, walkthrough, vm, screen-map, anchor, onboarding]
+priority: 10
+---
+
+# Blueprint VM Walkthrough
+
+A walkthrough is a conversational session over a live VM: the user (or you)
+drives the target application step by step while you capture the screens,
+ground the UI elements, and persist the whole session into the blueprint's
+library. The output feeds `blueprint_synthesize` — the walkthrough transcript
+is context, and the screen-maps/anchors/screenshots it produces are
+runtime-referenceable resources (`entry://<id>#<fragment>`).
+
+**Persistence contract: create early, update often.** The walkthrough entry is
+your session store — there is no other one. Create it the moment the session
+starts and PATCH it after EVERY step so a crash or disconnect loses at most
+one step.
+
+## Step 0: Preflight
+
+1. `vm_list` — find a deployed VM with a live agent. If MDS isn't running,
+   `start_mds_on_vm`; if none exists, `deploy_vm`.
+2. `vm_connect` (local MCP) or carry `vmId + workspaceId` on every `vm_*` call
+   (hosted MCP). The same hosted-MCP rule applies to every `blueprint_*` call:
+   `workspaceId` is a required parameter on all of them, so pass it alongside
+   `blueprintId` in each example below.
+3. Confirm the blueprint: `blueprint_get` (or `blueprint_create` for a fresh
+   one). Note the `blueprintId` — every persistence call needs it.
+4. Tell the user where to watch: the onboarding studio's VM viewer streams the
+   same VM's screen, so they can follow along (and drive) while you work.
+
+## Step 1: Open the walkthrough entry IMMEDIATELY
+
+```
+blueprint_add_entry blueprintId=<id> kind=walkthrough
+  title="Walkthrough: <process name> (<date>)"
+  content={ "transcript": [], "filmstrip": [], "artifactRefs": [] }
+```
+
+Save the returned `entryId` — you will PATCH this entry after every step
+(`blueprint_add_entry entryId=<entryId> content=<updated WalkthroughContent>`).
+
+`WalkthroughContent` shape:
+
+- `transcript`: `[{ role: "agent" | "user", text, at }]` — the conversation, oldest first
+- `filmstrip`: `[{ ref, caption, at, click?: {x, y}, stepHint? }]` — captured screens, oldest first
+- `artifactRefs`: `entry://` refs of sibling entries the session produced (screen-maps, anchors)
+
+## Step 2: The per-step loop
+
+For each step the user narrates ("now I open the invoice screen and click
+Export"):
+
+### 2a. Drive
+
+The user drives on the studio viewer, or you drive via `vm_execute_script`
+(pyautogui for desktop, CDP for browsers). Verify the screen settled with
+`vm_screenshot` before capturing.
+
+### 2b. Capture — screenshot into the library
+
+The VM PUTs the screenshot to blob storage itself (no image bytes through the
+MCP):
+
+1. `blueprint_entry_upload_url blueprintId=<id> fileName="step-<n>-<screen>.png"`
+   → `{ uploadUrl, blobRef }`
+2. Capture + upload FROM the VM:
+
+```python
+# vm_execute_script
+from PIL import ImageGrab
+import io, requests
+
+img = ImageGrab.grab()
+buf = io.BytesIO()
+img.save(buf, format="PNG")
+r = requests.put(
+    "<uploadUrl>",
+    data=buf.getvalue(),
+    headers={"Content-Type": "image/png"},
+)
+print(r.status_code)  # expect 200
+```
+
+3. `blueprint_add_entry blueprintId=<id> kind=screenshot title="Step <n>: <screen>"
+   blobRef=<blobRef> fileName=... contentType="image/png"` → screenshot `entryId`
+4. Append to the walkthrough's `filmstrip`:
+   `{ "ref": "entry://<screenshotEntryId>", "caption": "<what happened>", "click": {"x": .., "y": ..} }`
+   (record `click` when the step was a click — the coordinates you or the user used).
+
+### 2c. Ground — element boundaries
+
+- `vm_inspect_ui` mode `element_tree` (or `element_at_point` for one control)
+  — the element RECTS are the boundary source of truth.
+- The user dictates names and flags conversationally ("that's the amount
+  field, mask it in recordings") → accumulate boundaries for this screen:
+  `{ name, rect: {x, y, w, h}, flags: { verify?, viewport?, redact? }, note? }`.
+- Click order matters: keep the boundaries array in the order the process
+  touches them — reference order IS the grounding contract.
+
+### 2d. Transcript + persist
+
+Append the exchange to `transcript` (both the user's narration and your
+descriptions), then PATCH the walkthrough entry:
+
+```
+blueprint_add_entry blueprintId=<id> entryId=<walkthroughEntryId>
+  content=<full updated WalkthroughContent>
+```
+
+Then move to the next step.
+
+## Step 3: Anchors (per stable screen)
+
+For each screen the automation must recognize deterministically, pick 2-4
+distinctive regions (logos, fixed labels, unique chrome — NOT data that
+changes per record). Verify each region is legible at native resolution with
+`vm_screenshot_region`, then persist:
+
+```
+blueprint_add_entry blueprintId=<id> kind=anchor title="Anchor: <screen name>"
+  content={ "name": "<screen>", "threshold": 0.9,
+            "imageRef": "entry://<screenshotEntryId>",
+            "points": [{ "x":.., "y":.., "w":.., "h":.., "label": "<region>" }] }
+```
+
+## Step 4: Finalize
+
+1. One `screen-map` entry per named screen, from the boundaries gathered in 2c:
+
+```
+blueprint_add_entry blueprintId=<id> kind=screen-map title="Screen map: <screen>"
+  content={ "pages": [{ "name": "<screen>",
+                        "screenshotRef": "entry://<screenshotEntryId>",
+                        "boundaries": [ ... ] }] }
+```
+
+   Steps reference boundaries as `entry://<screenMapEntryId>#<page>.<boundary>`.
+2. Collect every produced `entry://` ref into the walkthrough's
+   `artifactRefs` and make the final PATCH.
+3. Close the loop: the library moved, so the spec is stale —
+   `blueprint_synthesize blueprintId=<id>`, then review/apply the proposal and
+   `blueprint_build`. `resolve_blueprint_state` confirms where you are.
+
+## Known issues
+
+- Upload URLs are GCS signed URLs: a plain PUT with the file's `Content-Type`
+  is the whole contract (success is a 200). Don't add storage-specific headers
+  like Azure's `x-ms-blob-type` — GCS ignores stray extension headers, but
+  they're wrong for this backend and will confuse debugging.
+- Upload URLs are short-lived — request one per screenshot, right before the PUT.
+- `blueprint_entry_upload_url` returning blocked (blob storage unconfigured):
+  fall back to text-only capture — describe each screen in the transcript and
+  skip filmstrip refs; the walkthrough is still valuable synthesis context.
+- On a broker (multi-session) VM, every `vm_execute_script` / `vm_screenshot`
+  needs `targetUser`.
+- Don't batch persistence "for later" — the create-early-update-often entry IS
+  the crash recovery.