@durable-streams/client 0.2.0 → 0.2.2

package/skills/writing-data/SKILL.md

@@ -0,0 +1,311 @@
+---
+name: writing-data
+description: >
+  Writing data to durable streams. DurableStream.create() with contentType,
+  DurableStream.append() for simple writes, IdempotentProducer for
+  high-throughput exactly-once delivery with autoClaim, fire-and-forget
+  append(), flush(), close(), StaleEpochError handling, JSON mode vs byte
+  stream mode, stream closure. Load when writing, producing, or appending
+  data to a durable stream.
+type: core
+library: durable-streams
+library_version: "0.2.1"
+requires:
+  - getting-started
+sources:
+  - "durable-streams/durable-streams:packages/client/src/stream.ts"
+  - "durable-streams/durable-streams:packages/client/src/idempotent-producer.ts"
+  - "durable-streams/durable-streams:packages/client/src/types.ts"
+---
+
+This skill builds on durable-streams/getting-started. Read it first for setup and offset basics.
+
+# Durable Streams — Writing Data
+
+Two write APIs: `DurableStream.append()` for simple writes, `IdempotentProducer`
+for sustained high-throughput writes with exactly-once delivery. Use the producer
+for anything beyond a few one-off appends.
+
+## Setup
+
+```typescript
+import {
+  DurableStream,
+  IdempotentProducer,
+  StaleEpochError,
+} from "@durable-streams/client"
+
+// Create a JSON-mode stream
+const handle = await DurableStream.create({
+  url: "https://your-server.com/v1/stream/my-stream",
+  contentType: "application/json",
+})
+
+// Set up IdempotentProducer for reliable writes
+const producer = new IdempotentProducer(handle, "my-service", {
+  autoClaim: true,
+  onError: (err) => {
+    if (err instanceof StaleEpochError) {
+      console.log("Another producer took over")
+    } else {
+      console.error("Write error:", err)
+    }
+  },
+})
+
+// Fire-and-forget writes — automatically batched and deduplicated
+for (const event of events) {
+  producer.append(JSON.stringify(event))
+}
+
+// Ensure all pending writes are delivered
+await producer.flush()
+await producer.close()
+```
+
+## Core Patterns
+
+### Simple writes with append()
+
+For writing a few items and waiting for completion, `append()` is straightforward:
+
+```typescript
+import { DurableStream } from "@durable-streams/client"
+
+const handle = await DurableStream.create({
+  url: "https://your-server.com/v1/stream/events",
+  contentType: "application/json",
+})
+
+// Each append waits for server confirmation
+await handle.append({ type: "order.created", orderId: "123" })
+await handle.append({ type: "order.paid", orderId: "123" })
+```
+
+### High-throughput writes with IdempotentProducer
+
+For sustained writes, the producer batches, pipelines, and deduplicates automatically:
+
+```typescript
+import { DurableStream, IdempotentProducer } from "@durable-streams/client"
+
+const handle = await DurableStream.create({
+  url: "https://your-server.com/v1/stream/tokens",
+  contentType: "text/plain",
+})
+
+const producer = new IdempotentProducer(handle, "llm-worker", {
+  autoClaim: true,
+  lingerMs: 10, // Batch window (default: 5ms)
+  maxBatchBytes: 65536, // Max batch size (default: 1MB)
+  maxInFlight: 4, // Concurrent HTTP requests (default: 5)
+  onError: (err) => console.error(err),
+})
+
+for await (const token of llm.stream(prompt)) {
+  producer.append(token) // Fire-and-forget — don't await
+}
+
+await producer.flush() // Wait for all batches to land
+await producer.close() // Clean up
+```
+
+### Closing a stream
+
+Mark a stream as permanently closed (no more writes accepted):
+
+```typescript
+// Close with optional final message
+await handle.close({ body: JSON.stringify({ type: "stream.complete" }) })
+
+// Or close without a final message
+await handle.close()
+```
+
+### Byte stream mode with custom framing
+
+For non-JSON streams, use your own framing (e.g., newline-delimited JSON):
+
+```typescript
+const handle = await DurableStream.create({
+  url: "https://your-server.com/v1/stream/logs",
+  contentType: "text/plain",
+})
+
+const producer = new IdempotentProducer(handle, "logger", { autoClaim: true })
+
+producer.append(JSON.stringify({ level: "info", msg: "started" }) + "\n")
+producer.append(JSON.stringify({ level: "error", msg: "failed" }) + "\n")
+
+await producer.flush()
+```
+
+## Common Mistakes
+
+### CRITICAL Using raw append() for sustained writes
+
+Wrong:
+
+```typescript
+const handle = await DurableStream.create({
+  url,
+  contentType: "application/json",
+})
+for (const event of events) {
+  await handle.append(JSON.stringify(event)) // No dedup, no batching, sequential
+}
+```
+
+Correct:
+
+```typescript
+const handle = await DurableStream.create({
+  url,
+  contentType: "application/json",
+})
+const producer = new IdempotentProducer(handle, "my-service", {
+  autoClaim: true,
+  onError: (err) => console.error(err),
+})
+for (const event of events) {
+  producer.append(JSON.stringify(event)) // Fire-and-forget, batched, deduplicated
+}
+await producer.flush()
+```
+
+Raw `append()` has no deduplication — on retry after network error, data may be duplicated. `IdempotentProducer` handles batching, pipelining, and exactly-once delivery.
+
+Source: packages/client/src/idempotent-producer.ts
+
+### CRITICAL Not calling flush() before shutdown
+
+Wrong:
+
+```typescript
+for (const event of events) {
+  producer.append(event)
+}
+// Process exits — pending batch lost!
+```
+
+Correct:
+
+```typescript
+for (const event of events) {
+  producer.append(event)
+}
+await producer.flush()
+await producer.close()
+```
+
+`IdempotentProducer` batches writes. Without `flush()`, pending messages in the buffer are lost when the process exits.
+
+Source: packages/client/src/idempotent-producer.ts
+
+### HIGH Awaiting each producer.append() call
+
+Wrong:
+
+```typescript
+for (const event of events) {
+  await producer.append(JSON.stringify(event)) // Defeats pipelining!
+}
+```
+
+Correct:
+
+```typescript
+for (const event of events) {
+  producer.append(JSON.stringify(event)) // Fire-and-forget
+}
+await producer.flush() // Wait for all to complete
+```
+
+`append()` is fire-and-forget by design. Awaiting it serializes every write and defeats batching and pipelining. Errors go to the `onError` callback.
+
+Source: packages/client/src/idempotent-producer.ts
+
+### HIGH Passing objects to append instead of strings
+
+Wrong:
+
+```typescript
+producer.append({ event: "user.created" }) // throws!
+```
+
+Correct:
+
+```typescript
+producer.append(JSON.stringify({ event: "user.created" }))
+```
+
+`IdempotentProducer.append()` accepts only `string` or `Uint8Array` — it does **not** auto-serialize objects, even for JSON-mode streams. Always call `JSON.stringify()` before appending.
+
+Source: packages/client/src/idempotent-producer.ts
+
+### HIGH Not handling StaleEpochError for multi-worker scenarios
+
+Wrong:
+
+```typescript
+const producer = new IdempotentProducer(handle, "worker", {
+  onError: (err) => console.error(err), // Logs but doesn't stop
+})
+```
+
+Correct:
+
+```typescript
+let fenced = false
+const producer = new IdempotentProducer(handle, "worker", {
+  onError: (err) => {
+    if (err instanceof StaleEpochError) {
+      fenced = true // Stop producing
+    }
+  },
+})
+
+for await (const token of source) {
+  if (fenced) break
+  producer.append(token)
+}
+```
+
+When another producer claims a higher epoch, the current producer gets fenced. Without checking, your write loop continues appending to a dead producer.
+
+Source: packages/client/src/idempotent-producer.ts
+
+### HIGH Manually managing epoch numbers
+
+Wrong:
+
+```typescript
+const producer = new IdempotentProducer(handle, "worker", { epoch: 2 })
+```
+
+Correct:
+
+```typescript
+const producer = new IdempotentProducer(handle, "worker", {
+  autoClaim: true, // Handles epoch transitions automatically
+})
+```
+
+Epoch and sequence management is an internal protocol detail. Use `autoClaim: true` and let the producer handle transitions automatically.
+
+Source: maintainer guidance
+
+### HIGH Tension: Simplicity vs. reliability in writes
+
+This skill's patterns conflict with getting-started. Getting-started simplicity favors `append()` for quick demos, but production code should use `IdempotentProducer` for exactly-once guarantees. Agents optimizing for a quick setup may use `append()` and miss deduplication and batching.
+
+See also: durable-streams/getting-started/SKILL.md § Common Mistakes
+
+## See also
+
+- [getting-started](../getting-started/SKILL.md) — Basic stream creation and reading
+- [go-to-production](../go-to-production/SKILL.md) — TTL configuration and production concerns
+
+## Version
+
+Targets @durable-streams/client v0.2.1.

package/src/constants.ts

@@ -26,6 +26,12 @@ export const STREAM_CURSOR_HEADER = `Stream-Cursor`
  */
 export const STREAM_UP_TO_DATE_HEADER = `Stream-Up-To-Date`
 
+/**
+ * Response/request header indicating stream is closed (EOF).
+ * When present with value "true", the stream is permanently closed.
+ */
+export const STREAM_CLOSED_HEADER = `Stream-Closed`
+
 // ============================================================================
 // Request Headers
 // ============================================================================
@@ -97,6 +103,11 @@ export const LIVE_QUERY_PARAM = `live`
  */
 export const CURSOR_QUERY_PARAM = `cursor`
 
+/**
+ * Response header indicating SSE data encoding (e.g., base64 for binary streams).
+ */
+export const STREAM_SSE_DATA_ENCODING_HEADER = `stream-sse-data-encoding`
+
 // ============================================================================
 // SSE Control Event Fields (camelCase per PROTOCOL.md Section 5.7)
 // ============================================================================
@@ -113,13 +124,19 @@ export const SSE_OFFSET_FIELD = `streamNextOffset`
  */
 export const SSE_CURSOR_FIELD = `streamCursor`
 
+/**
+ * SSE control event field for stream closed state.
+ * Note: Different from HTTP header name (camelCase vs Header-Case).
+ */
+export const SSE_CLOSED_FIELD = `streamClosed`
+
 // ============================================================================
 // Internal Constants
 // ============================================================================
 
 /**
- * Content types that support SSE mode.
- * SSE is only valid for text/* or application/json streams.
+ * Content types that are natively compatible with SSE (UTF-8 text).
+ * Binary content types are also supported via automatic base64 encoding.
  */
 export const SSE_COMPATIBLE_CONTENT_TYPES: ReadonlyArray<string> = [
   `text/`,

package/src/error.ts

@@ -178,6 +178,26 @@ export class MissingStreamUrlError extends Error {
   }
 }
 
+/**
+ * Error thrown when attempting to append to a closed stream.
+ */
+export class StreamClosedError extends DurableStreamError {
+  readonly code = `STREAM_CLOSED` as const
+  readonly status = 409
+  readonly streamClosed = true
+
+  /**
+   * The final offset of the stream, if available from the response.
+   */
+  readonly finalOffset?: string
+
+  constructor(url?: string, finalOffset?: string) {
+    super(`Cannot append to closed stream`, `STREAM_CLOSED`, 409, url)
+    this.name = `StreamClosedError`
+    this.finalOffset = finalOffset
+  }
+}
+
 /**
  * Error thrown when signal option is invalid.
  */

package/src/idempotent-producer.ts

@@ -17,11 +17,12 @@ import {
   PRODUCER_ID_HEADER,
   PRODUCER_RECEIVED_SEQ_HEADER,
   PRODUCER_SEQ_HEADER,
+  STREAM_CLOSED_HEADER,
   STREAM_OFFSET_HEADER,
 } from "./constants"
 import type { queueAsPromised } from "fastq"
 import type { DurableStream } from "./stream"
-import type { IdempotentProducerOptions, Offset } from "./types"
+import type { CloseResult, IdempotentProducerOptions, Offset } from "./types"
 
 /**
  * Error thrown when a producer's epoch is stale (zombie fencing).
@@ -138,6 +139,8 @@ export class IdempotentProducer {
   readonly #queue: queueAsPromised<BatchTask>
   readonly #maxInFlight: number
   #closed = false
+  #closeResult: CloseResult | null = null
+  #pendingFinalMessage?: Uint8Array | string
 
   // When autoClaim is true, we must wait for the first batch to complete
   // before allowing pipelining (to know what epoch was claimed)
@@ -318,11 +321,17 @@ export class IdempotentProducer {
   }
 
   /**
-   * Flush pending messages and close the producer.
+   * Stop the producer without closing the underlying stream.
    *
-   * After calling close(), further append() calls will throw.
+   * Use this when you want to:
+   * - Hand off writing to another producer
+   * - Keep the stream open for future writes
+   * - Stop this producer but not signal EOF to readers
+   *
+   * Flushes any pending messages before detaching.
+   * After calling detach(), further append() calls will throw.
    */
-  async close(): Promise<void> {
+  async detach(): Promise<void> {
     if (this.#closed) return
 
     this.#closed = true
@@ -330,8 +339,138 @@ export class IdempotentProducer {
     try {
       await this.flush()
     } catch {
-      // Ignore errors during close
+      // Ignore errors during detach
+    }
+  }
+
+  /**
+   * Flush pending messages and close the underlying stream (EOF).
+   *
+   * This is the typical way to end a producer session. It:
+   * 1. Flushes all pending messages
+   * 2. Optionally appends a final message
+   * 3. Closes the stream (no further appends permitted)
+   *
+   * **Idempotent**: Unlike `DurableStream.close({ body })`, this method is
+   * idempotent even with a final message because it uses producer headers
+   * for deduplication. Safe to retry on network failures.
+   *
+   * @param finalMessage - Optional final message to append atomically with close
+   * @returns CloseResult with the final offset
+   */
+  async close(finalMessage?: Uint8Array | string): Promise<CloseResult> {
+    if (this.#closed) {
+      // Already closed - return cached result for idempotency
+      if (this.#closeResult) {
+        return this.#closeResult
+      }
+      // Retry path: flush() threw on a previous attempt, so we need to re-run
+      // the entire close sequence with the stored finalMessage
+      await this.flush()
+      const result = await this.#doClose(this.#pendingFinalMessage)
+      this.#closeResult = result
+      return result
+    }
+
+    this.#closed = true
+
+    // Store finalMessage for retry safety (if flush() throws, we can retry)
+    this.#pendingFinalMessage = finalMessage
+
+    // Flush pending messages first
+    await this.flush()
+
+    // Close the stream with optional final message
+    const result = await this.#doClose(finalMessage)
+    this.#closeResult = result
+    return result
+  }
+
+  /**
+   * Actually close the stream with optional final message.
+   * Uses producer headers for idempotency.
+   */
+  async #doClose(finalMessage?: Uint8Array | string): Promise<CloseResult> {
+    const contentType = this.#stream.contentType ?? `application/octet-stream`
+    const isJson = normalizeContentType(contentType) === `application/json`
+
+    // Build body if final message is provided
+    let body: BodyInit | undefined
+    if (finalMessage !== undefined) {
+      const bodyBytes =
+        typeof finalMessage === `string`
+          ? new TextEncoder().encode(finalMessage)
+          : finalMessage
+
+      if (isJson) {
+        // For JSON mode, wrap in array
+        const jsonStr = new TextDecoder().decode(bodyBytes)
+        body = `[${jsonStr}]`
+      } else {
+        body = bodyBytes as unknown as BodyInit
+      }
+    }
+
+    // Capture the sequence number for this request (for retry safety)
+    // We only increment #nextSeq after a successful response
+    const seqForThisRequest = this.#nextSeq
+
+    // Build headers with producer info and Stream-Closed
+    const headers: Record<string, string> = {
+      "content-type": contentType,
+      [PRODUCER_ID_HEADER]: this.#producerId,
+      [PRODUCER_EPOCH_HEADER]: this.#epoch.toString(),
+      [PRODUCER_SEQ_HEADER]: seqForThisRequest.toString(),
+      [STREAM_CLOSED_HEADER]: `true`,
+    }
+
+    const response = await this.#fetchClient(this.#stream.url, {
+      method: `POST`,
+      headers,
+      body,
+      signal: this.#signal,
+    })
+
+    // Handle 204 (duplicate close - idempotent success)
+    if (response.status === 204) {
+      // Only increment seq on success (retry-safe)
+      this.#nextSeq = seqForThisRequest + 1
+      const finalOffset = response.headers.get(STREAM_OFFSET_HEADER) ?? ``
+      return { finalOffset }
+    }
+
+    // Handle success
+    if (response.status === 200) {
+      // Only increment seq on success (retry-safe)
+      this.#nextSeq = seqForThisRequest + 1
+      const finalOffset = response.headers.get(STREAM_OFFSET_HEADER) ?? ``
+      return { finalOffset }
     }
+
+    // Handle errors
+    if (response.status === 403) {
+      // Stale epoch
+      const currentEpochStr = response.headers.get(PRODUCER_EPOCH_HEADER)
+      const currentEpoch = currentEpochStr
+        ? parseInt(currentEpochStr, 10)
+        : this.#epoch
+
+      if (this.#autoClaim) {
+        // Auto-claim: retry with epoch+1
+        const newEpoch = currentEpoch + 1
+        this.#epoch = newEpoch
+        // Reset sequence for new epoch - set to 0 so the recursive call uses seq 0
+        // (the first operation in a new epoch should be seq 0)
+        this.#nextSeq = 0
+        return this.#doClose(finalMessage)
+      }
+
+      throw new StaleEpochError(currentEpoch)
+    }
+
+    // Other errors
+    const error = await FetchError.fromResponse(response, this.#stream.url)
+    throw error
   }
 
   /**

package/src/index.ts

@@ -65,6 +65,10 @@ export type {
   HeadResult,
   LegacyLiveMode,
 
+  // Close types
+  CloseResult,
+  CloseOptions,
+
   // Idempotent producer types
   IdempotentProducerOptions,
   IdempotentAppendResult,
@@ -89,6 +93,7 @@ export {
   DurableStreamError,
   MissingStreamUrlError,
   InvalidSignalError,
+  StreamClosedError,
 } from "./error"
 
 // ============================================================================
@@ -110,6 +115,7 @@ export {
   STREAM_OFFSET_HEADER,
   STREAM_CURSOR_HEADER,
   STREAM_UP_TO_DATE_HEADER,
+  STREAM_CLOSED_HEADER,
   STREAM_SEQ_HEADER,
   STREAM_TTL_HEADER,
   STREAM_EXPIRES_AT_HEADER,
@@ -117,6 +123,7 @@ export {
   LIVE_QUERY_PARAM,
   CURSOR_QUERY_PARAM,
   SSE_COMPATIBLE_CONTENT_TYPES,
+  SSE_CLOSED_FIELD,
   DURABLE_STREAM_PROTOCOL_QUERY_PARAMS,
   // Idempotent producer headers
   PRODUCER_ID_HEADER,