stillmade-sdk 0.1.42 → 0.1.43

package/MODULES.md

@@ -57,13 +57,18 @@ against the manifest's port types.
 | `progress(value, label)` | Reports progress from 0 to 1 with a short label. Each report extends the run limit. |
 | `log(...values)` | Adds a line to the run log (up to 500 lines). |
 | `asset(path)` | A copy of one of the module's own files, such as a `.wasm` build, as an `ArrayBuffer`. |
-| `files.read(ref)` | The bytes, as an `ArrayBuffer`, of a saved image, video, audio, asset or file (a PDF, text, CSV, Markdown or JSON file on a `file` input) the Block received as input, or that StillMade made in this run: files it wrote, transform results and generated media. Media the person kept on their device (a `sm-media://device/...` address in a project Step) reads the same way, without leaving the device. Other files cannot be read, and transforms, hosted calls and actions accept only these references too; a hosted call or action gets a cloud copy of device media, made once per version. |
+| `files.read(ref)` | The bytes, as an `ArrayBuffer`, of a saved image, video, audio, asset or file (a PDF, text, CSV, Markdown or JSON file on a `file` input) the Block received as input, or that StillMade made in this run: files it wrote, transform results and generated media. Media the person kept on their device (a `sm-media://device/...` address in a project Step) reads the same way, without leaving the device. Other files cannot be read, and transforms, hosted calls and actions accept only these references too; a hosted call or action gets a cloud copy of device media, made once per version. A reference is the file its `kind`, `assetId`, `versionId` and `url` name, so `{kind, assetId, versionId, url}` rebuilt from a `media.generate` result item (without its `key`) is the same file; any other file is refused with `MODULE_FILE_READ`, by `stillmade-block test --mock`, `ready` and `gate` as by StillMade. |
 | `files.write(data, {mediaType, name})` | Saves PNG, JPEG, WebP, GIF, MP4, WebM, MP3, WAV, M4A or OGG media, or a PDF, TXT, CSV, Markdown, JSON or ZIP file (`application/pdf`, `text/plain`, `text/csv`, `text/markdown`, `application/json`, `application/zip`; a `{kind: 'file'}` reference), to the runner's library and returns a reference you can return as an output. At most 64 files per run. |
 | `media.transform(request)` | On-device media work done by StillMade: decode audio to samples, extract or loudness-normalise audio, cut, join or re-encode video, and take thumbnails. Results are saved like `files.write`. See [Media processing](#media-processing). |
-| `actions.run(action)` | Runs a declared portable action. `media.generate` asks the person to approve one StillMade credit maximum for the batch; `stock.search` runs directly. |
+| `actions.run(action)` | Runs a declared portable action. `media.generate` opens one approval for its whole batch, naming the most it can cost in StillMade credits; `stock.search` runs directly. |
 | `hosted.run(operation, request)` | Calls a hosted operation the manifest lists in `permissions.capabilities` (for example `text.generate`, `image.generate`, `audio.speech`, `web.research`). Each call is quoted and approved like a capability Block and paid with StillMade credits. |
-| `net.fetch(url, {method, headers, body})` | Calls one of the origins declared in `permissions.network` through StillMade, which adds the person's own key or OAuth token. Returns a standard `Response`. |
+| `net.fetch(url, {method, headers, body})` | Calls one of the origins declared in `permissions.network` through StillMade, which adds the person's own key or OAuth token. Returns a standard `Response`. See [Outside APIs](/docs/build/outside-apis). |
 | `net.connect(url, {protocols})` | Opens a live WebSocket to the `wss://` form of a declared origin and resolves with the open socket. No key is added. See [Live connections](#live-connections). |
+| `connections.call(appId, operationId, input)` | Calls an operation of a published adapter declared in `permissions.connections`, with the person's own account. Returns the checked JSON result. |
+| `storage.get/set/remove/list/info(...)` | The Block's own cloud storage in the current project: JSON values up to 1 MiB under folder-like keys. Kept across runs, devices and the Block's versions. See [Project storage, reads and assets](/docs/build/project-storage). |
+| `storage.putFile(key, data, {mediaType})` / `storage.getFile(key)` | Files up to 64 MiB in the same storage; `getFile` returns `{key, mediaType, bytes, data}` with `data` as an `ArrayBuffer`, or `null`. |
+| `project.read(fields?)` | A copy of the current project limited to the fields the manifest may read (`project.read` plus `context.<field>.read`), such as `timeline`, `shots` or `assets`. |
+| `assets.create(file, {name})` / `assets.appendVersion(asset, file)` | Adds a saved image, video or audio file to the project's media as a new asset (`asset.create`), or as the newest version of an asset the Block received, read or created (`asset.version.append`). |
 
 Hosted calls take the operation's request fields, for example:
 
@@ -82,19 +87,158 @@ saved video, whose own sound it transcribes, so there is no need to extract the
 audio first. `media.analyze` returns `{schemaVersion: 1, items}`, one item per
 picture or video, and for one video takes an optional `range: {start, end}` in
 seconds, at most 12 apart, to look closely at that part. `image.generate` takes `prompt` and optional
-`references`, one to eight saved images the new picture keeps. Look at pictures with `media.analyze`:
+`references`, one to eight saved images the new picture keeps. `video.generate` takes `prompt` and optional
+`firstFrame` (a saved image the video starts on, for image-to-video), `lastFrame` (one it ends on; needs
+`firstFrame`), `negativePrompt` (up to 2,000 characters) and `settings`; a model named in `settings` must take
+those frames (Kling 3.0 Turbo and Wan 2.6 Flash take no last frame; Wan 2.6 Flash and Wan 2.7 need a first frame).
+For a vertical or square clip, set `settings.aspectRatio` (`'9:16'`, `'1:1'`) and name no model: the default
+model (Vidu Q3 Turbo) takes no aspect ratio, so StillMade uses the cheapest model that makes that shape with the
+frames and `duration` given (Wan 2.6 Flash with a first frame, else Veo 3.1 Lite, else Seedance 2.0 Mini; see
+[Generation models](/docs/reference/generation-models)). A model you name is kept, and one that cannot make the shape
+is refused. Look at pictures with `media.analyze`:
 StillMade credits pay for it, while `image.describe` runs only on the person's
 own OpenRouter key. `web.fetch` takes `url` and `instruction`;
 `web.research` takes `topic`; music and sound effects take `prompt` and
 `durationSec`.
 
+### Actions (`permissions.actions`)
+
+`actions.run` runs only the actions listed in `permissions.actions`, the same list a
+JavaScript Block declares: `"actions": ["media.generate"]` (or `"stock.search"`). A
+module that runs actions and calls no hosted operation leaves
+`permissions.capabilities` out; an empty list means the same.
+
+```js
+const { value, failures } = await stillmade.actions.run({
+  schemaVersion: 1, kind: 'stillmade.media-action', operation: 'media.generate',
+  resultPort: null,
+  request: { items: [
+    { key: 'front', operation: 'image', prompt: `The front of ${input.address} at dusk`,
+      settings: { model: 'nano-banana-pro', aspectRatio: '16:9' } },
+    { key: 'tour', operation: 'speech', prompt: input.script },
+  ] },
+});
+```
+
+The action is the object a JavaScript Block returns, with every key present; its
+items, settings and limits are in [Generate media from
+JavaScript](/docs/build/generate-media). Pass `resultPort: null`: no
+output port receives the media, `actions.run` returns it to your code and `run()`
+returns what the Block outputs. Before anything is generated StillMade opens one
+approval for the whole batch with the most it can cost in StillMade credits; credits
+the batch does not use come back when it closes. Each `actions.run` call is its own
+approval, so put every item one run needs in one batch: one clip per photo, the
+voice and the music are one `media.generate` call of up to 50 items (StillMade makes
+four at a time), never a call per photo, and never `hosted.run('video.generate')` once
+per photo, which asks again from the second call. Video items take the same
+`settings` as `video.generate` above, `aspectRatio` included. `actions.run` resolves with:
+
+- `value`: the items that were made, in request order, each
+  `{key, kind, assetId, versionId, url, duration?}` with `kind` `image`, `video` or `audio`
+  (speech, music and sound effects). `duration` is the length in seconds of a
+  video, music or sound effect item: the length it asked for (`settings.duration`,
+  `settings.durationSec`, or the model's default). A speech item has none, since
+  its length is what the words take: decode it with
+  `media.transform({operation: 'audio.decode', source})`, whose result has
+  `duration`, before timing a cut to it. Return them as outputs, or pass them to
+  `files.read`, `media.transform` or a hosted call.
+- `failures`: one `{key, message}` for each item that was not made; those items are
+  not in `value`.
+
+It throws when no item was made, when the person declines (`MODULE_ACTION_DECLINED`)
+and when the action is not declared (`MODULE_PERMISSION`). `stock.search` resolves
+with `{results}`. With `--mock` or a `tests/mocks.json`, `media.generate` answers
+with stand-in media and no failures; record your own answer under
+`actions["media.generate"]`.
+
+## A check, then the expensive run, on one page
+
+A job the person fills in and then gets a result from is one Block on one page
+(builder contract, "One job per Step"): the inputs, a cheap check before the
+expensive run, and the result. A listing video writes the script and voices a
+few seconds of it first, and makes the video only once the person has read the
+script and heard the voice. Build that as one module Block with a custom view,
+never as two Blocks or two Steps.
+
+The manifest takes the mode and the checked values as optional inputs, and the
+outputs the check leaves out are optional (a run may leave out any output with
+`"required": false`). `action` names the main button, the expensive run:
+
+```json
+"inputs": {
+  "listingPhotos": { "type": "image[]", "primary": true, "description": "Photos of the home, in the order of the tour." },
+  "homeFacts": { "type": "text", "description": "Address, rooms and what stands out." },
+  "make": { "type": "text", "required": false, "default": "video", "description": "check: the script and a voice sample; video: the tour." },
+  "script": { "type": "script", "required": false, "description": "The checked script, as the person left it." }
+},
+"outputs": {
+  "tour": { "type": "video", "primary": true, "required": false },
+  "script": { "type": "script", "required": false },
+  "voiceSample": { "type": "audio", "required": false }
+},
+"action": { "label": "Generate tour", "running": "Generating tour…" },
+"permissions": { "capabilities": ["text.generate", "audio.speech"] }
+```
+
+The module does only the cheap work when asked for the check:
+
+```js
+export default async function run(input, stillmade) {
+  const write = async () => (await stillmade.hosted.run('text.generate', {
+    prompt: `A 40-second listing tour script from these facts: ${input.homeFacts}`,
+    schema: { type: 'object', required: ['script'], properties: { script: { type: 'string' } } },
+  })).script;
+  if (input.make === 'check') {
+    const script = await write();
+    const voiceSample = await stillmade.hosted.run('audio.speech', { text: script.split(/(?<=[.!?])\s/)[0] });
+    return { script, voiceSample };
+  }
+  const script = input.script || await write();
+  const voice = await stillmade.hosted.run('audio.speech', { text: script });
+  const { duration } = await stillmade.media.transform({ operation: 'audio.decode', source: voice });
+  const each = duration / input.listingPhotos.length;
+  const tour = await stillmade.media.transform({
+    operation: 'timeline.render', aspect: '9:16',
+    clips: input.listingPhotos.map((photo, index) => ({ source: photo, start: index * each, duration: each, motion: 'zoom-in' })),
+    audio: [{ source: voice, start: 0, duration }],
+  });
+  return { tour, script };
+}
+```
+
+The view runs it twice with `StillMade.run`, which takes the inputs the view
+sets (StillMade keeps them on the Step and fills the rest): `{make: 'check'}`
+first ("Write the script"), then `{make: 'video', script}` with the script the
+person checked or edited. One `sm-primary` at a time: the check until there is
+a script, then the expensive run, labelled with `manifest.action`, while
+"Write again" stays a quiet button. The result's place
+(`sm-stage data-ratio="9:16"`) is drawn before either run; the script and the sample sit in it or beside the inputs
+until the video replaces them. Each run starts empty: return the script again
+with the video so it stays on the Step. StillMade saves the inputs a view sets,
+so the mode last pressed stays on the Step; until the view first sets it, a run
+started elsewhere (Chat, an automation) takes the default and makes the video.
+
+Each run is priced and approved by StillMade on its own. Beside the Block's
+page StillMade shows what each hosted call costs at the Step's saved or default
+model ("Write · 1 credit · Make voiceover · 3 credits"), with an (i) saying that
+pressing the Block's button is the approval. A run started by the person's own
+click or key in the view then runs each hosted operation once without a dialog,
+up to that price. StillMade checks the price with the provider first and opens
+its approval dialog, with the quote, only when the quote is higher, when the run
+calls the same operation a second time, or when the run did not come from the
+person's press (in a preview outside a project every call shows the dialog;
+`stillmade-block dev` answers with stand-ins and spends nothing). The view shows
+no prices. `media.transform`, `timeline.render` included, runs on the device
+and costs nothing; `actions.run` `media.generate` always asks first.
+
 ## Media processing
 
 `stillmade.media.transform(request)` asks StillMade to do heavy media work on
 the person's device: Web Audio for audio, and the same renderer as Final
 Render for video (WebCodecs in the browser, the native renderer on desktop).
 It never runs on StillMade's servers and costs no credits. A transform reads
-only files the Block received, wrote or generated in this run, and saves its
+only files the Block received, wrote or generated in this run (including media
+from `actions.run` `media.generate`; see `files.read` above), and saves its
 result to the runner's library, so you can return it as an output or pass it
 to the next transform.
 
@@ -117,7 +261,9 @@ from?, transition?: {type, duration?}, look?, mute?, motion?, fit?, focus?}` (a
 image or video; a video keeps its own sound unless `mute`; `motion` is a slow
 camera move over the clip: `zoom-in`, `zoom-out`, `pan-left`, `pan-right`,
 `pan-up` or `pan-down`, which always keeps the picture filling the frame), each
-audio item `{source, start, duration, from?, volume?}`, and each caption
+audio item `{source, start, duration, from?, volume?}` (`volume` is a percentage of the
+recording's own level, 0 to 200: 100, the default, plays it as recorded, 50 at half
+the level, 0 silent; put music under a voice at about 20 to 30), and each caption
 `{text, start, end, words?, position?, highlight?, animation?, size?}`, where
 `words` (`[{text, start, end}]`) highlights each word as it is spoken and `size`
 is the text height in pixels of the rendered frame (16 to 200; default 44, so