@tanstack/ai 0.41.0 → 0.43.0

package/skills/ai-core/locks/SKILL.md

@@ -0,0 +1,143 @@
+---
+name: ai-core/locks
+description: >
+  LockStore, InMemoryLockStore, LocksCapability and withLocks for
+  multi-instance coordination in TanStack AI. Ships in @tanstack/ai — NOT in
+  @tanstack/ai-persistence. Separate from AIPersistence state stores — not a
+  stores key, not composable. InMemoryLockStore vs a distributed (e.g.
+  Cloudflare Durable Object) lock, lease recovery, AbortSignal in critical
+  sections. Use when sandbox or other middleware needs cross-worker mutual
+  exclusion — NOT for storing messages/runs (use withPersistence).
+type: sub-skill
+library: tanstack-ai
+library_version: '0.42.0'
+sources:
+  - 'TanStack/ai:docs/advanced/locks.md'
+  - 'TanStack/ai:packages/ai/src/activities/chat/middleware/locks.ts'
+---
+
+# Locks (coordination — not persistence)
+
+> **Dependency note:** This skill builds on ai-core and ai-core/middleware.
+> `withLocks` is a ChatMiddleware that provides a capability. Locks are **not**
+> part of `AIPersistence.stores` and are **not** composed with
+> `composePersistence` — they ship in `@tanstack/ai`, independent of
+> `@tanstack/ai-persistence`.
+
+## Why separate?
+
+State stores answer "what is durable chat data?"  
+Locks answer "who may run this critical section right now?"
+
+`withPersistence` does **not** automatically lock a whole turn. Take a
+per-thread (or other) lock yourself when multi-writer races matter.
+
+## Wire locks
+
+```ts
+import { withLocks, InMemoryLockStore } from '@tanstack/ai/locks'
+
+middleware: [
+  withLocks(new InMemoryLockStore()), // single process
+]
+```
+
+Alongside persistence — optional, locks do not require it:
+
+```ts
+import { withLocks, InMemoryLockStore } from '@tanstack/ai/locks'
+import { withPersistence } from '@tanstack/ai-persistence'
+
+middleware: [withPersistence(persistence), withLocks(new InMemoryLockStore())]
+```
+
+`withLocks` provides `LocksCapability` for downstream middleware (e.g.
+sandbox). Order: usually state first, locks alongside or after depending on
+who consumes the capability.
+
+## The contract
+
+```ts
+interface LockStore {
+  withLock<T>(key: string, fn: (signal: AbortSignal) => Promise<T>): Promise<T>
+}
+```
+
+`InMemoryLockStore` ships in **`@tanstack/ai/locks`**: a per-key promise chain,
+correct **within a single process only**. Multi-instance deployments need a
+distributed implementation — you write it. The Cloudflare Durable Object recipe
+is in **ai-persistence/build-cloudflare-adapter** (`@tanstack/ai-persistence`).
+
+Type your own store with `defineLock` (autocomplete, no `: LockStore`
+annotation), then hand it to `withLocks`. Acquire the key, run `fn`, release when
+`fn` settles:
+
+```ts
+import { defineLock, withLocks } from '@tanstack/ai/locks'
+import { acquire } from './my-lock-backend'
+
+const locks = defineLock({
+  async withLock(key, fn) {
+    const { release, signal } = await acquire(key)
+    try {
+      return await fn(signal)
+    } finally {
+      release()
+    }
+  },
+})
+
+middleware: [withLocks(locks)]
+```
+
+## Lease semantics
+
+A good `LockStore`:
+
+- Serializes owners per key,
+- Uses **leases** (or equivalent) so a crashed owner cannot block forever,
+- Passes an `AbortSignal` into the critical section via `withLock`; when the
+  lease is lost, abort so work stops starting external mutations.
+
+Callbacks must honor the signal and pass it to cancellable dependencies.
+`InMemoryLockStore` never aborts its signal — within one process, ownership
+cannot be lost.
+
+## Capability identity
+
+The `'locks'` capability token lives in `@tanstack/ai/locks`. Capability identity
+is by **object reference**, so one shared token means a `withLocks` in the chain
+reaches `withSandbox` automatically.
+
+## Common mistakes
+
+### HIGH: Importing locks from `@tanstack/ai-persistence`
+
+They are not exported there. Use `@tanstack/ai`.
+
+### HIGH: Putting `locks` on `AIPersistence.stores`
+
+Not supported. `stores` accepts only `messages`, `runs`, `interrupts`,
+`metadata` — never `locks`. Use `withLocks`.
+
+### HIGH: Passing `locks` to `composePersistence` overrides
+
+Same rejection, at the override layer. Locks are not state.
+
+### HIGH: Passing `'locks'` to the conformance testkit's `skip`
+
+`skip` accepts only chat state store keys. The suite does not cover locks
+at all — test lease expiry and abort separately.
+
+### HIGH: `InMemoryLockStore` across multiple processes
+
+No mutual exclusion between machines — use a distributed lock store.
+
+### MEDIUM: Ignoring lease abort
+
+Continuing work after losing the lease races other owners.
+
+## Cross-references
+
+- See also: **ai-core/middleware/SKILL.md** -- the middleware chain and capability plumbing
+- See also: **`@tanstack/ai-persistence` skills** (`skills/ai-persistence/SKILL.md` in that package) -- `ai-persistence/server` (state middleware) and `ai-persistence/build-cloudflare-adapter` (Durable Object lock recipe)

package/skills/ai-core/media-generation/SKILL.md

@@ -2,16 +2,16 @@
 name: ai-core/media-generation
 description: >
   Image, audio, video, speech (TTS), and transcription generation using
-  activity-specific adapters: generateImage() with openaiImage/geminiImage,
+  activity-specific adapters: generateImage() with openaiImage/geminiImage/byteplusImage,
   generateAudio() with geminiAudio/falAudio, generateVideo() with async
-  polling (openaiVideo/geminiVideo/grokVideo/falVideo, per-model typed
-  durations), generateSpeech() with openaiSpeech, generateTranscription()
-  with openaiTranscription. React hooks: useGenerateImage, useGenerateAudio,
+  polling (openaiVideo/geminiVideo/grokVideo/falVideo/byteplusVideo, per-model typed
+  durations), generateSpeech() with openaiSpeech/byteplusSpeech, generateTranscription()
+  with openaiTranscription/byteplusTranscription. React hooks: useGenerateImage, useGenerateAudio,
   useGenerateSpeech, useTranscription, useGenerateVideo.
   TanStack Start server function integration with toServerSentEventsResponse.
 type: sub-skill
 library: tanstack-ai
-library_version: '0.10.0'
+library_version: '0.42.0'
 sources:
   - 'TanStack/ai:docs/media/generations.md'
   - 'TanStack/ai:docs/media/generation-hooks.md'
@@ -150,8 +150,16 @@ function ImageGenerator() {
 ### 1. Image Generation
 
 Supported adapters: `openaiImage` (dall-e-2, dall-e-3, gpt-image-1,
-gpt-image-1-mini, gpt-image-2) and `geminiImage` (gemini-3.1-flash-image-preview,
-gemini-3.1-flash-lite-image, imagen-4.0-generate-001, etc.).
+gpt-image-1-mini, gpt-image-2), `geminiImage` (gemini-3.1-flash-image-preview,
+gemini-3.1-flash-lite-image, imagen-4.0-generate-001, etc.) and `byteplusImage`
+(Seedream — `seedream-4-0-250828`, `seedream-4-5-251128`, the 5.0 family).
+
+> **Seedream quirks:** `watermark` defaults to **`true`** (pass
+> `modelOptions: { watermark: false }` for a clean image), `size` is a token
+> (`'1K'` | `'2K'` | `'4K'`) **or** explicit `'2048x2048'` pixels but never a
+> mix, and `numberOfImages` is an **upper bound** — Seedream has no `n`, so it
+> maps onto group-image mode and the model may return fewer. Reads
+> `ARK_API_KEY`.
 
 ```typescript
 import { generateImage } from '@tanstack/ai'
@@ -333,7 +341,19 @@ const { generate, result, isLoading } = useGenerateAudio({
 
 ### 3. Text-to-Speech
 
-Adapter: `openaiSpeech` (tts-1, tts-1-hd, gpt-4o-audio-preview).
+Adapters: `openaiSpeech` (tts-1, tts-1-hd, gpt-4o-audio-preview) and
+`byteplusSpeech` (`seed-audio-1.0`).
+
+> **BytePlus Seed Speech is a separate product from ModelArk** — it reads
+> **`BYTEPLUS_VOICE_API_KEY`**, not `ARK_API_KEY`, and an Ark key there fails
+> with `45000010 Invalid X-Api-Key`. Output is capped at **120 seconds**.
+> There is no top-level `speaker` field — `voice` is sent as
+> `references: [{ speaker }]`, and `modelOptions.references` **replaces** that
+> array rather than merging, so passing `references` for voice cloning silently
+> drops `voice`. Voice ids ending `_uranus_bigtts` are TTS 2.0,
+> `_mars_bigtts` / `_moon_bigtts` are TTS 1.0, and `*_emo_v2_*` are the 1.0
+> voices that accept emotion tags. Formats: `wav`, `mp3`, `pcm`, `ogg_opus`;
+> `watermark` is also available on `modelOptions`.
 
 ```typescript
 import { generateSpeech } from '@tanstack/ai'
@@ -367,8 +387,10 @@ const { generate, result, isLoading } = useGenerateSpeech({
 
 ### 4. Audio Transcription
 
-Adapter: `openaiTranscription` (whisper-1, gpt-4o-transcribe,
-gpt-4o-mini-transcribe, gpt-4o-transcribe-diarize).
+Adapters: `openaiTranscription` (whisper-1, gpt-4o-transcribe,
+gpt-4o-mini-transcribe, gpt-4o-transcribe-diarize) and `byteplusTranscription`
+(`seed-asr` — synchronous, no polling; audio up to 2 hours / 100 MB; also reads
+**`BYTEPLUS_VOICE_API_KEY`**).
 
 > **Capturing audio in the browser:** Use `useAudioRecorder` from `@tanstack/ai-react` to record directly in the browser, then pass the recording as the `audio` input to `generate()`, or use `recording.part` as a prompt part in chat/generation calls. No transcoding or extra dependencies required — the recorder returns the native browser format (`audio/webm` or `audio/mp4`). For transcription, wrap it as a `data:` URL so the provider gets the real content type; passing raw `recording.base64` makes the adapter assume `audio/mpeg` and mislabel the webm/mp4 bytes.
 >
@@ -517,7 +539,20 @@ durations 4/8/12s, single `input_reference` image prompt part), `grokVideo(...)`
 (`grok-imagine-video` does text-to-video + image-to-video; `grok-imagine-video-1.5` is
 image-to-video only — needs an `image` prompt part as the starting frame, text-only throws;
 aspect-ratio size template like `'16:9_720p'`, integer durations 1-15s, reports
-`usage.unitsBilled` seconds and exact `usage.cost`), and `falVideo(...)` (hosted models, see cost tracking below).
+`usage.unitsBilled` seconds and exact `usage.cost`), `byteplusVideo(...)` (Seedance —
+aspect-ratio size template like `'16:9_720p'`, durations 4-15s on the 2.0 family,
+4-12s on 1.5-pro, 2-12s on the 1.0-pro models; reads `ARK_API_KEY`), and
+`falVideo(...)` (hosted models, see cost tracking below).
+
+> **Seedance option applicability is per model and enforced server-side** —
+> Ark returns a 400 for an inapplicable field rather than ignoring it.
+> `service_tier` / `camera_fixed` are Seedance 1.x only, `frames` is
+> 1-0-pro + 1-0-pro-fast only, `draft` is 1-5-pro only, `priority` is the 2.0
+> family only, and `duration: -1` works on 2.0 + 1-5-pro. There is no 2K tier
+> on any model and `4k` exists only on `dreamina-seedance-2-0-260128`.
+> **Video URLs expire 24 hours after the task completes** (task record kept 7
+> days). Seedance is also reachable via `falVideo` — `byteplusVideo` is the
+> direct-to-BytePlus path.
 
 Client hook with job tracking:
 
@@ -563,6 +598,94 @@ if (result.usage?.unitsBilled != null) {
 For video, the units arrive with the completed result: `getVideoJobStatus()`
 returns `usage` and emits a `video:usage` devtools event when fal reports it.
 
+### 7. Durable persistence (job lifecycle + artifact bytes)
+
+To make generations survive a server restart and be re-served later, add
+`withGenerationPersistence` from `@tanstack/ai-persistence` as generation
+middleware. It requires `stores.generationRuns` (a `GenerationRunStore`, keyed on
+the run's own `runId`, with a required `threadId` naming the stable slot the run
+fills — that is what a client hydrates by) and, when you also pass an `stores.artifacts` +
+`stores.blobs` **pair** (both or neither), it persists the generated media bytes
+at blob key `artifacts/<runId>/<artifactId>` with an `ArtifactRecord` per file.
+`memoryPersistence()` ships all three for dev/tests.
+
+```typescript
+import { generateImage, toServerSentEventsResponse } from '@tanstack/ai'
+import { openaiImage } from '@tanstack/ai-openai'
+import {
+  withGenerationPersistence,
+  memoryPersistence,
+  retrieveArtifact,
+  retrieveBlob,
+  reconstructGeneration,
+} from '@tanstack/ai-persistence'
+
+const persistence = memoryPersistence() // swap for your DB/object-store adapter
+
+export async function POST(req: Request) {
+  const { prompt, threadId } = await req.json()
+  return toServerSentEventsResponse(
+    generateImage({
+      adapter: openaiImage('gpt-image-1'),
+      prompt,
+      threadId, // the slot recorded on the job + artifacts
+      stream: true,
+      middleware: [
+        withGenerationPersistence(persistence, {
+          // Stamp a durable app-origin serve URL (the GET route below) onto
+          // each persisted artifact ref, and rewrite the live result's media to
+          // it. Both live and restored results then render from your origin.
+          artifactUrl: (ref) => `/api/artifacts?id=${ref.artifactId}`,
+        }),
+      ],
+    }),
+  )
+}
+
+// Serve the stored bytes back (GET /api/artifacts?id=…):
+export async function GET(req: Request) {
+  const id = new URL(req.url).searchParams.get('id') ?? ''
+  const record = await retrieveArtifact(persistence, id)
+  if (!record) return new Response('Not found', { status: 404 })
+  const blob = await retrieveBlob(persistence, record)
+  if (!blob?.body) return new Response('Not found', { status: 404 })
+  return new Response(blob.body, {
+    headers: { 'content-type': record.mimeType },
+  })
+}
+```
+
+That route is enough for images. **Video needs `Range`**: seeking a `<video>`
+is built on `206` / `Content-Range`, and Safari refuses to play a source that
+ignores `Range` at all. Resolve the header against `record.size` (`416` when it
+does not fit), pass `retrieveBlob(persistence, record, { range })`, and answer
+`206` from the returned `blob.range` plus `accept-ranges: bytes`. The full
+route is in the persistence docs under **Serve video: honour `Range`**.
+
+On the client, `persistence` is **boolean only**: `persistence: true` hydrates
+the last generation for the thread on mount, via the connection's
+`hydrateGeneration` handler backed by a `reconstructGeneration` GET route. There
+is no storage-adapter mode, so nothing about a generation is cached in the
+browser.
+
+**`persistence: true` requires a stable `threadId`**, and it is a type error to
+set one without the other. That is the generation's scope, the slot successive
+runs fill (e.g. `video-9-start-frame`), not a link to a chat. `id` is the
+devtools label only and never a persistence key.
+
+The hooks are transparent (like `useChat`): a reload repaints `status` /
+`result` / `error`, not a separate `resumeSnapshot`. Because the `artifactUrl`
+above stamps a durable URL onto each ref (carried on `result.artifacts`), the
+restored `result` rebuilds its media from those refs and serves from your own
+origin. Without byte storage, a reload restores `status` / `error` and `result`
+stays `null`.
+
+- Building the R2/D1-backed byte stores for a Cloudflare Worker:
+  **ai-persistence/build-cloudflare-artifact-store**.
+- Store contracts, `composePersistence`, and the wiring end-to-end:
+  `docs/persistence/generation-persistence.md` and the
+  `ai-core/client-persistence` sub-skill.
+
 ---
 
 ## Common Hook API
@@ -577,7 +700,16 @@ All generation hooks return the same shape:
 | `error`     | `Error \| undefined`       | Current error                                    |
 | `status`    | `GenerationClientState`    | `'idle' \| 'generating' \| 'success' \| 'error'` |
 | `stop`      | `() => void`               | Abort current generation                         |
-| `reset`     | `() => void`               | Clear state, return to idle                      |
+| `reset`     | `() => void`               | Clear state and the in-memory snapshot           |
+| `runId`     | `string \| null`           | Id of the job WHILE it runs; null when idle      |
+
+The hook is **transparent**, mirroring `useChat`: there is no `resumeSnapshot`,
+`resumeState`, `pendingArtifacts`, or `resultArtifacts` field. Hooks also accept
+`persistence: true` plus a stable `threadId`: on mount the client hydrates the
+last run for that scope from the server and repaints the **normal** `status` /
+`result` / `error` fields, so the last run survives a reload (metadata only,
+never media bytes; `result`'s media returns only with server byte storage +
+`artifactUrl`). See `ai-core/client-persistence` for details.
 
 Provide either `connection` (streaming SSE transport) or `fetcher`
 (direct async call / server function returning `Response`). Use `onResult`