@durable-streams/client 0.2.0 → 0.2.2

package/skills/reading-streams/SKILL.md

@@ -0,0 +1,247 @@
+---
+name: reading-streams
+description: >
+  All stream reading patterns for @durable-streams/client. stream() function,
+  DurableStream.stream(), LiveMode (false, true, "long-poll", "sse"),
+  StreamResponse state machine, .json(), .text(), .jsonStream(), .textStream(),
+  .subscribeJson(), .subscribeBytes(), .subscribeText(), SSE resilience with
+  auto-fallback to long-poll, visibility-based pause, binary SSE base64
+  auto-decode, dynamic headers for auth token refresh, backoff config,
+  StreamErrorHandler onError for error recovery.
+type: core
+library: durable-streams
+library_version: "0.2.1"
+requires:
+  - getting-started
+sources:
+  - "durable-streams/durable-streams:packages/client/src/stream-api.ts"
+  - "durable-streams/durable-streams:packages/client/src/response.ts"
+  - "durable-streams/durable-streams:packages/client/src/types.ts"
+  - "durable-streams/durable-streams:packages/client/src/fetch.ts"
+---
+
+This skill builds on durable-streams/getting-started. Read it first for setup and offset basics.
+
+# Durable Streams — Reading Streams
+
+Use `stream()` for read-only access (fetch-like API). Use `DurableStream.stream()`
+when you already have a `DurableStream` handle for read/write operations. Both
+return a `StreamResponse` with identical consumption methods.
+
+## Setup
+
+```typescript
+import { stream } from "@durable-streams/client"
+
+// Catch-up read (returns all existing data, then stops)
+const res = await stream<{ event: string; userId: string }>({
+  url: "https://your-server.com/v1/stream/my-stream",
+  offset: "-1",
+  live: false,
+})
+const items = await res.json()
+```
+
+## Core Patterns
+
+### Live modes
+
+```typescript
+import { stream } from "@durable-streams/client"
+
+// Catch-up only — stop at end of existing data
+const catchUp = await stream({ url, offset: "-1", live: false })
+
+// Auto-select best transport (SSE for JSON, long-poll for binary)
+const auto = await stream({ url, offset: "-1", live: true })
+
+// Explicit long-poll
+const longPoll = await stream({ url, offset: "-1", live: "long-poll" })
+
+// Explicit SSE
+const sse = await stream({ url, offset: "-1", live: "sse" })
+```
+
+### Dynamic headers for auth token refresh
+
+Header functions are called **per-request**, allowing token refresh during long-lived live streams:
+
+```typescript
+import { stream } from "@durable-streams/client"
+
+const res = await stream({
+  url: "https://your-server.com/v1/stream/my-stream",
+  offset: "-1",
+  live: true,
+  headers: {
+    Authorization: async () => `Bearer ${await getAccessToken()}`,
+  },
+})
+```
+
+### Error recovery with onError
+
+```typescript
+import { stream } from "@durable-streams/client"
+
+const res = await stream({
+  url: "https://your-server.com/v1/stream/my-stream",
+  offset: "-1",
+  live: true,
+  onError: (error) => {
+    if (error.status === 401) {
+      // Refresh auth and retry with new headers
+      return { headers: { Authorization: `Bearer ${newToken}` } }
+    }
+    if (error.status === 404) {
+      return // Stop retrying (void = propagate error)
+    }
+    return {} // Retry with same params
+  },
+})
+```
+
+### SSE resilience with auto-fallback
+
+```typescript
+import { stream } from "@durable-streams/client"
+
+const res = await stream({
+  url: "https://your-server.com/v1/stream/my-stream",
+  offset: "-1",
+  live: "sse",
+  sseResilience: {
+    minConnectionDuration: 1000, // Connections under 1s are "short"
+    maxShortConnections: 3, // Fall back after 3 short connections
+  },
+})
+```
+
+## Common Mistakes
+
+### CRITICAL Not saving offset for resumption
+
+Wrong:
+
+```typescript
+res.subscribeJson((batch) => {
+  processItems(batch.items)
+  // offset not saved!
+})
+```
+
+Correct:
+
+```typescript
+res.subscribeJson((batch) => {
+  processItems(batch.items)
+  saveCheckpoint(batch.offset)
+})
+```
+
+The whole point of durable streams is resumability. Without persisting the offset, you lose the ability to resume after disconnect.
+
+Source: README.md resume from offset section
+
+### HIGH Using .json() on non-JSON content type streams
+
+Wrong:
+
+```typescript
+// Stream created with contentType: "text/plain"
+const res = await stream({ url, offset: "-1", live: false })
+const data = await res.json() // throws DurableStreamError!
+```
+
+Correct:
+
+```typescript
+const res = await stream({ url, offset: "-1", live: false })
+const text = await res.text()
+```
+
+`.json()`, `.jsonStream()`, and `.subscribeJson()` only work on JSON-mode streams (`contentType: "application/json"`). Use `.text()` or `.body()` for other content types.
+
+Source: packages/client/src/response.ts
+
+### HIGH Ignoring onError handler for live streams
+
+Wrong:
+
+```typescript
+const res = await stream({ url, offset: "-1", live: true })
+// No onError — auth failures retry forever with exponential backoff
+```
+
+Correct:
+
+```typescript
+const res = await stream({
+  url,
+  offset: "-1",
+  live: true,
+  onError: (error) => {
+    if (error.status === 401) return // Stop retrying
+    return {} // Retry for transient errors
+  },
+})
+```
+
+Without `onError`, permanent errors (401, 403) silently retry forever with exponential backoff.
+
+Source: packages/client/src/types.ts StreamErrorHandler
+
+### HIGH Returning void from onError to retry
+
+Wrong:
+
+```typescript
+onError: (error) => {
+  console.log("retrying...")
+  // Returns undefined — error propagates instead of retrying!
+}
+```
+
+Correct:
+
+```typescript
+onError: (error) => {
+  console.log("retrying...")
+  return {} // Return an object to signal retry
+}
+```
+
+The `onError` handler must return an object (`{}` or `{ headers, params }`) to signal retry. Returning `void`/`undefined` propagates the error.
+
+Source: packages/client/src/types.ts RetryOpts
+
+### MEDIUM Using HTTP instead of HTTPS in browser
+
+Wrong:
+
+```typescript
+const res = await stream({ url: "http://api.example.com/v1/stream/my-stream" })
+```
+
+Correct:
+
+```typescript
+const res = await stream({ url: "https://api.example.com/v1/stream/my-stream" })
+```
+
+HTTP/1.1 in browsers limits to ~6 concurrent connections per origin. With multiple live streams, this can freeze the app.
+
+Source: packages/client/src/utils.ts warnIfUsingHttpInBrowser
+
+## References
+
+- [StreamResponse consumption methods](references/stream-response-methods.md)
+
+## See also
+
+- [getting-started](../getting-started/SKILL.md) — Basic setup and offset concepts
+- [stream-db](../../../state/skills/stream-db/SKILL.md) — StreamDB uses stream reading internally
+
+## Version
+
+Targets @durable-streams/client v0.2.1.

package/skills/reading-streams/references/stream-response-methods.md

@@ -0,0 +1,133 @@
+# StreamResponse Consumption Methods
+
+A `StreamResponse<TJson>` provides three families of consumption methods.
+Choose one per response — calling a second method throws `ALREADY_CONSUMED`.
+
+## Promise-Based (await for complete result)
+
+Use for catch-up reads (`live: false`) or when you want all data at once.
+
+### `.json(): Promise<TJson[]>`
+
+Returns all items as a parsed JSON array. JSON-mode streams only.
+
+```typescript
+const res = await stream<{ id: string }>({ url, offset: "-1", live: false })
+const items = await res.json()
+// items: Array<{ id: string }>
+```
+
+### `.text(): Promise<string>`
+
+Returns the complete body as a string. Works with any content type.
+
+```typescript
+const res = await stream({ url, offset: "-1", live: false })
+const text = await res.text()
+```
+
+### `.body(): Promise<Uint8Array>`
+
+Returns the complete body as raw bytes.
+
+```typescript
+const res = await stream({ url, offset: "-1", live: false })
+const bytes = await res.body()
+```
+
+## ReadableStream-Based (async iteration)
+
+Use for processing items one at a time as they arrive.
+
+### `.jsonStream(): ReadableStream<TJson>`
+
+Yields individual JSON items. JSON-mode streams only.
+
+```typescript
+const res = await stream<{ event: string }>({ url, offset: "-1", live: true })
+for await (const item of res.jsonStream()) {
+  console.log(item.event)
+}
+```
+
+### `.textStream(): ReadableStream<string>`
+
+Yields text chunks as they arrive.
+
+```typescript
+const res = await stream({ url, offset: "-1", live: true })
+for await (const chunk of res.textStream()) {
+  process.stdout.write(chunk)
+}
+```
+
+### `.bodyStream(): ReadableStream<Uint8Array>`
+
+Yields raw byte chunks.
+
+```typescript
+const res = await stream({ url, offset: "-1", live: true })
+for await (const chunk of res.bodyStream()) {
+  processBytes(chunk)
+}
+```
+
+## Subscriber-Based (callback with offset tracking)
+
+Use for live subscriptions where you need the offset for checkpointing.
+
+### `.subscribeJson(callback): () => void`
+
+Calls back with batches of JSON items and offset.
+
+```typescript
+const res = await stream<{ event: string }>({ url, offset: "-1", live: true })
+res.subscribeJson(async (batch) => {
+  for (const item of batch.items) {
+    console.log(item.event)
+  }
+  saveCheckpoint(batch.offset)
+})
+```
+
+### `.subscribeText(callback): () => void`
+
+Calls back with text data and offset.
+
+```typescript
+const res = await stream({ url, offset: "-1", live: true })
+res.subscribeText(async (chunk) => {
+  appendToUI(chunk.text)
+  saveCheckpoint(chunk.offset)
+})
+```
+
+### `.subscribeBytes(callback): () => void`
+
+Calls back with byte data and offset.
+
+```typescript
+const res = await stream({ url, offset: "-1", live: true })
+res.subscribeBytes(async (chunk) => {
+  processBytes(chunk.data)
+  saveCheckpoint(chunk.offset)
+})
+```
+
+## Properties
+
+| Property       | Type                  | Description                               |
+| -------------- | --------------------- | ----------------------------------------- |
+| `offset`       | `string`              | Current offset (updates after each chunk) |
+| `contentType`  | `string \| undefined` | Content type from stream creation         |
+| `live`         | `LiveMode`            | The live mode for this session            |
+| `startOffset`  | `string`              | The offset this session started from      |
+| `upToDate`     | `boolean`             | Whether caught up with all existing data  |
+| `streamClosed` | `boolean`             | Whether the stream is permanently closed  |
+| `closed`       | `Promise<void>`       | Resolves when the stream session ends     |
+
+## Methods
+
+| Method     | Description               |
+| ---------- | ------------------------- |
+| `cancel()` | Cancel the stream session |

package/skills/server-deployment/SKILL.md

@@ -0,0 +1,211 @@
+---
+name: server-deployment
+description: >
+  Running durable stream servers. DurableStreamTestServer for development
+  (Node.js, @durable-streams/server, not for production), Caddy plugin for
+  production with Caddyfile configuration, data_dir for file-backed persistence,
+  max_file_handles tuning, long_poll_timeout, server binary downloads for macOS
+  Linux Windows, @durable-streams/cli tool setup, conformance test runner.
+type: lifecycle
+library: durable-streams
+library_version: "0.2.1"
+sources:
+  - "durable-streams/durable-streams:packages/caddy-plugin/README.md"
+  - "durable-streams/durable-streams:packages/server/README.md"
+  - "durable-streams/durable-streams:packages/cli/README.md"
+---
+
+# Durable Streams — Server Deployment
+
+Two server options: a Node.js development server for prototyping and a
+Caddy-based production server with file persistence and CDN support.
+
+## Setup
+
+### Development server (Node.js)
+
+```typescript
+import { DurableStreamTestServer } from "@durable-streams/server"
+
+const server = new DurableStreamTestServer({
+  port: 4437,
+  host: "127.0.0.1",
+})
+
+await server.start()
+console.log(`Dev server running on ${server.url}`)
+```
+
+### Production server (Caddy binary)
+
+Download the binary from [GitHub releases](https://github.com/durable-streams/durable-streams/releases) for your platform.
+
+```bash
+# Start the server
+./durable-streams-server run --config Caddyfile
+```
+
+Caddyfile configuration:
+
+```
+:8787 {
+  route /v1/stream/* {
+    durable_streams {
+      data_dir ./data
+      max_file_handles 200
+      long_poll_timeout 60s
+    }
+  }
+}
+```
+
+## Core Patterns
+
+### CLI for testing
+
+```bash
+# Install CLI
+npm install -g @durable-streams/cli
+
+# Set server URL
+export STREAM_URL=http://localhost:4437
+
+# Create, write, read
+durable-stream create my-stream
+durable-stream write my-stream "Hello, world!"
+durable-stream read my-stream
+```
+
+### Running conformance tests
+
+```bash
+# Against your server
+npx @durable-streams/server-conformance-tests --run http://localhost:4437
+
+# Watch mode for development
+npx @durable-streams/server-conformance-tests --watch src http://localhost:4437
+```
+
+### Programmatic dev server with stream creation
+
+```typescript
+import { DurableStreamTestServer } from "@durable-streams/server"
+import { DurableStream } from "@durable-streams/client"
+
+const server = new DurableStreamTestServer({ port: 0 }) // Random port
+await server.start()
+
+// Streams must be created before clients can connect
+await DurableStream.create({
+  url: `${server.url}/v1/stream/my-app`,
+  contentType: "application/json",
+})
+
+console.log(`Dev server running on ${server.url}`)
+```
+
+### Programmatic dev server in tests
+
+```typescript
+import { DurableStreamTestServer } from "@durable-streams/server"
+
+const server = new DurableStreamTestServer({ port: 0 }) // Random port
+await server.start()
+
+// Run your tests against server.url
+
+await server.stop()
+```
+
+### Caddy configuration options
+
+| Option              | Default            | Description                            |
+| ------------------- | ------------------ | -------------------------------------- |
+| `data_dir`          | (none — in-memory) | Directory for file-backed persistence  |
+| `max_file_handles`  | 100                | Max concurrent open file handles       |
+| `long_poll_timeout` | 60s                | How long to hold long-poll connections |
+
+## Common Mistakes
+
+### CRITICAL Using the Node.js dev server in production
+
+Wrong:
+
+```typescript
+// production deployment
+import { DurableStreamTestServer } from "@durable-streams/server"
+const server = new DurableStreamTestServer({ port: 4437 })
+```
+
+Correct:
+
+```bash
+# Use the Caddy plugin binary
+./durable-streams-server run --config Caddyfile
+```
+
+`DurableStreamTestServer` is explicitly not for production. It uses in-memory storage, has no CDN integration, and is single-process only.
+
+Source: packages/server/README.md
+
+### CRITICAL Not configuring data_dir for persistence
+
+Wrong:
+
+```
+:8787 {
+  route /v1/stream/* {
+    durable_streams
+  }
+}
+```
+
+Correct:
+
+```
+:8787 {
+  route /v1/stream/* {
+    durable_streams {
+      data_dir ./data
+      max_file_handles 200
+    }
+  }
+}
+```
+
+Without `data_dir`, the Caddy plugin uses in-memory storage. Server restarts lose all data.
+
+Source: packages/caddy-plugin/README.md
+
+### MEDIUM Setting max_file_handles too low for production
+
+Wrong:
+
+```
+durable_streams {
+  data_dir ./data
+  # default 100 handles — fine for dev, low for production
+}
+```
+
+Correct:
+
+```
+durable_streams {
+  data_dir ./data
+  max_file_handles 500  # Tune based on active stream count
+}
+```
+
+Default is 100 file handles. High-throughput deployments with many concurrent streams can exhaust the pool, causing latency spikes.
+
+Source: packages/caddy-plugin/store/filepool.go
+
+## See also
+
+- [getting-started](../getting-started/SKILL.md) — Connect a client to your server
+- [go-to-production](../go-to-production/SKILL.md) — CDN caching, TTL, and HTTPS for production
+
+## Version
+
+Targets durable-streams-server v0.2.1.